<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2305.14627] ALCE: An Automatic Benchmark for Large Language Model Generations with Citations</title><meta property="og:description" content="Large language models (LLMs) have emerged as a widely-used tool for information seeking, but their generated outputs are prone to hallucination. In this work, our aim is to allow LLMs to generate text with citations,
i…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="ALCE: An Automatic Benchmark for Large Language Model Generations with Citations">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="ALCE: An Automatic Benchmark for Large Language Model Generations with Citations">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2305.14627">

<!--Generated on Thu Feb 29 05:56:16 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">ALCE: An Automatic Benchmark for
<br class="ltx_break">Large Language Model Generations with <span id="id1.id1" class="ltx_text ltx_font_italic">Citations</span>
</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Tianyu Gao Howard Yen Jiatong Yu Danqi Chen 
<br class="ltx_break">Department of Computer Science
&amp; Princeton Language and Intelligence
<br class="ltx_break">Princeton University 
<br class="ltx_break"><span id="id2.1.id1" class="ltx_text ltx_font_typewriter">{tianyug,hyen,jiatongy,danqic}@cs.princeton.edu</span>
</span></span>
</div>

<h1 class="ltx_title ltx_title_document">Can Large Language Models Cite Precisely?
<br class="ltx_break">A Benchmark for Automatic Evaluation and Modeling</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Tianyu Gao Howard Yen Jiatong Yu Danqi Chen 
<br class="ltx_break">Department of Computer Science
&amp; Princeton Language and Intelligence
<br class="ltx_break">Princeton University 
<br class="ltx_break"><span id="id2.1.id1" class="ltx_text ltx_font_typewriter">{tianyug,hyen,jiatongy,danqic}@cs.princeton.edu</span>
</span></span>
</div>

<h1 class="ltx_title ltx_title_document">Enabling Large Language Models to Generate Text with Citations</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Tianyu Gao Howard Yen Jiatong Yu Danqi Chen 
<br class="ltx_break">Department of Computer Science
&amp; Princeton Language and Intelligence
<br class="ltx_break">Princeton University 
<br class="ltx_break"><span id="id2.1.id1" class="ltx_text ltx_font_typewriter">{tianyug,hyen,jiatongy,danqic}@cs.princeton.edu</span>
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id3.id1" class="ltx_p">Large language models (LLMs) have emerged as a widely-used tool for information seeking, but their generated outputs are prone to hallucination. In this work, our aim is to allow LLMs to generate text with <span id="id3.id1.1" class="ltx_text ltx_font_italic">citations</span>,
improving their factual correctness and verifiability.
Existing work mainly relies on commercial search engines and human evaluation, making it challenging to reproduce and compare different modeling approaches.
We propose <span id="id3.id1.2" class="ltx_text ltx_font_bold">ALCE</span>, the first benchmark for <span id="id3.id1.3" class="ltx_text ltx_font_bold">A</span>utomatic <span id="id3.id1.4" class="ltx_text ltx_font_bold">L</span>LMs’ <span id="id3.id1.5" class="ltx_text ltx_font_bold">C</span>itation <span id="id3.id1.6" class="ltx_text ltx_font_bold">E</span>valuation.
ALCE collects a diverse set of questions and retrieval corpora
and requires building end-to-end systems to retrieve supporting evidence and generate answers with citations.
We develop automatic metrics along three dimensions—fluency, correctness, and citation quality—and
demonstrate their strong correlation with human judgements.
Our experiments with state-of-the-art LLMs and novel prompting strategies show that current systems have considerable room for improvement—For example, on the ELI5 dataset,
even the best models lack complete citation support 50% of the time.
Our analyses further highlight promising future directions, including
developing better retrievers,
advancing long-context LLMs,
and improving the ability to synthesize information from multiple sources.<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>Our code and data are available at <a target="_blank" href="https://github.com/princeton-nlp/ALCE" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/princeton-nlp/ALCE</a>.</span></span></span></p>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Large language models (LLMs; <cite class="ltx_cite ltx_citemacro_citep">Brown et al., <a href="#bib.bib4" title="" class="ltx_ref">2020</a>; OpenAI, <a href="#bib.bib29" title="" class="ltx_ref">2023</a></cite>) have gained increasing popularity as a tool for information seeking.
While they generate engaging and coherent responses,
their outputs are prone to hallucination and often contain factually incorrect information <cite class="ltx_cite ltx_citemacro_cite">Ji et al. (<a href="#bib.bib15" title="" class="ltx_ref">2023</a>)</cite>.
This makes it harder for users to trust and verify LLM-generated outputs without any supporting evidence.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2305.14627/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_img_landscape" width="452" height="257" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>The task setup of ALCE. Given a question,
the system generates text while providing <span id="S1.F1.3.1" class="ltx_text ltx_font_italic">citing passages</span> from a large retrieval corpus. Each statement may contain multiple citations (e.g., <span id="S1.F1.4.2" class="ltx_text ltx_font_typewriter">[1][2]</span>). </figcaption>
</figure>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">In this work, we study a new generation paradigm for LLMs, in which
we require LLMs to provide <span id="S1.p2.1.1" class="ltx_text ltx_font_italic">citations</span> to one or a few text passages for any statement they generate (Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>).
Incorporating citations brings several benefits:
(1) users can easily verify LLMs’ claims with the provided citations;
(2) LLMs can generate text that faithfully follows cited passages, which has the promise to improve correctness and alleviate hallucination.</p>
</div>
<figure id="S1.T1" class="ltx_table">
<table id="S1.T1.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S1.T1.1.1.1" class="ltx_tr">
<th id="S1.T1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="S1.T1.1.1.1.1.1" class="ltx_text ltx_font_bold">Dataset</span></th>
<th id="S1.T1.1.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="S1.T1.1.1.1.2.1" class="ltx_text ltx_font_bold">Corpus (#passages)</span></th>
<th id="S1.T1.1.1.1.3" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="S1.T1.1.1.1.3.1" class="ltx_text ltx_font_bold">Question type</span></th>
<th id="S1.T1.1.1.1.4" class="ltx_td ltx_align_justify ltx_align_top ltx_th ltx_th_column ltx_border_tt">
<span id="S1.T1.1.1.1.4.1" class="ltx_inline-block ltx_align_top">
<span id="S1.T1.1.1.1.4.1.1" class="ltx_p" style="width:216.8pt;"><span id="S1.T1.1.1.1.4.1.1.1" class="ltx_text ltx_font_bold">Example</span></span>
</span>
</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S1.T1.1.2.1" class="ltx_tr">
<td id="S1.T1.1.2.1.1" class="ltx_td ltx_align_left ltx_border_t">ASQA</td>
<td id="S1.T1.1.2.1.2" class="ltx_td ltx_align_left ltx_border_t">Wikipedia (21M)</td>
<td id="S1.T1.1.2.1.3" class="ltx_td ltx_align_left ltx_border_t">Factoid (ambiguous)</td>
<td id="S1.T1.1.2.1.4" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S1.T1.1.2.1.4.1" class="ltx_inline-block ltx_align_top">
<span id="S1.T1.1.2.1.4.1.1" class="ltx_p" style="width:216.8pt;"><span id="S1.T1.1.2.1.4.1.1.1" class="ltx_text ltx_font_bold">Q</span>: When did the US break away from England?</span>
</span>
</td>
</tr>
<tr id="S1.T1.1.3.2" class="ltx_tr">
<td id="S1.T1.1.3.2.1" class="ltx_td"></td>
<td id="S1.T1.1.3.2.2" class="ltx_td"></td>
<td id="S1.T1.1.3.2.3" class="ltx_td"></td>
<td id="S1.T1.1.3.2.4" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S1.T1.1.3.2.4.1" class="ltx_inline-block ltx_align_top">
<span id="S1.T1.1.3.2.4.1.1" class="ltx_p" style="width:216.8pt;"><span id="S1.T1.1.3.2.4.1.1.1" class="ltx_text ltx_font_bold">A</span>: The US declared independence on July 2, 1776 <span id="S1.T1.1.3.2.4.1.1.2" class="ltx_text" style="color:#000080;">[1][2]</span> … The Treaty of Paris was later signed on September 3, 1783 <span id="S1.T1.1.3.2.4.1.1.3" class="ltx_text" style="color:#000080;">[3]</span>.</span>
</span>
</td>
</tr>
<tr id="S1.T1.1.4.3" class="ltx_tr">
<td id="S1.T1.1.4.3.1" class="ltx_td ltx_align_left">QAMPARI</td>
<td id="S1.T1.1.4.3.2" class="ltx_td ltx_align_left">Wikipedia (21M)</td>
<td id="S1.T1.1.4.3.3" class="ltx_td ltx_align_left">Factoid (list)</td>
<td id="S1.T1.1.4.3.4" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S1.T1.1.4.3.4.1" class="ltx_inline-block ltx_align_top">
<span id="S1.T1.1.4.3.4.1.1" class="ltx_p" style="width:216.8pt;"><span id="S1.T1.1.4.3.4.1.1.1" class="ltx_text ltx_font_bold">Q</span>: Which films have Gong Li as a member of their cast?</span>
</span>
</td>
</tr>
<tr id="S1.T1.1.5.4" class="ltx_tr">
<td id="S1.T1.1.5.4.1" class="ltx_td"></td>
<td id="S1.T1.1.5.4.2" class="ltx_td"></td>
<td id="S1.T1.1.5.4.3" class="ltx_td"></td>
<td id="S1.T1.1.5.4.4" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S1.T1.1.5.4.4.1" class="ltx_inline-block ltx_align_top">
<span id="S1.T1.1.5.4.4.1.1" class="ltx_p" style="width:216.8pt;"><span id="S1.T1.1.5.4.4.1.1.1" class="ltx_text ltx_font_bold">A</span>: The Story of Qiu Ju <span id="S1.T1.1.5.4.4.1.1.2" class="ltx_text" style="color:#000080;">[1]</span>, Farewell My Concubine <span id="S1.T1.1.5.4.4.1.1.3" class="ltx_text" style="color:#000080;">[2]</span>, The Monkey King 2 <span id="S1.T1.1.5.4.4.1.1.4" class="ltx_text" style="color:#000080;">[3]</span>, Mulan <span id="S1.T1.1.5.4.4.1.1.5" class="ltx_text" style="color:#000080;">[3]</span>, Saturday Fiction <span id="S1.T1.1.5.4.4.1.1.6" class="ltx_text" style="color:#000080;">[3]</span> …</span>
</span>
</td>
</tr>
<tr id="S1.T1.1.6.5" class="ltx_tr">
<td id="S1.T1.1.6.5.1" class="ltx_td ltx_align_left">ELI5</td>
<td id="S1.T1.1.6.5.2" class="ltx_td ltx_align_left">Sphere (899M)</td>
<td id="S1.T1.1.6.5.3" class="ltx_td ltx_align_left">Why/How/What</td>
<td id="S1.T1.1.6.5.4" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S1.T1.1.6.5.4.1" class="ltx_inline-block ltx_align_top">
<span id="S1.T1.1.6.5.4.1.1" class="ltx_p" style="width:216.8pt;"><span id="S1.T1.1.6.5.4.1.1.1" class="ltx_text ltx_font_bold">Q</span>: How do student loans affect getting a mortgage?</span>
</span>
</td>
</tr>
<tr id="S1.T1.1.7.6" class="ltx_tr">
<td id="S1.T1.1.7.6.1" class="ltx_td ltx_border_bb"></td>
<td id="S1.T1.1.7.6.2" class="ltx_td ltx_border_bb"></td>
<td id="S1.T1.1.7.6.3" class="ltx_td ltx_border_bb"></td>
<td id="S1.T1.1.7.6.4" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="S1.T1.1.7.6.4.1" class="ltx_inline-block ltx_align_top">
<span id="S1.T1.1.7.6.4.1.1" class="ltx_p" style="width:216.8pt;"><span id="S1.T1.1.7.6.4.1.1.1" class="ltx_text ltx_font_bold">A</span>: Student loans can affect the debt to income ratio <span id="S1.T1.1.7.6.4.1.1.2" class="ltx_text" style="color:#000080;">[1]</span>, which is a key factor in determining the amount that … <span id="S1.T1.1.7.6.4.1.1.3" class="ltx_text" style="color:#000080;">[2][3]</span></span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>The three datasets used in our ALCE benchmark. These datasets cover a wide range of question types
and the corresponding corpora span from Wikipedia to
Web-scale document collection. </figcaption>
</figure>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">Multiple commercial systems have adopted this paradigm: Bing Chat<span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span><a target="_blank" href="https://www.bing.com/new" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.bing.com/new</a></span></span></span> and perplexity.ai<span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span><a target="_blank" href="https://www.perplexity.ai" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.perplexity.ai</a></span></span></span>
respond to user questions in natural language with references to Web pages.
<cite class="ltx_cite ltx_citemacro_citet">Nakano et al. (<a href="#bib.bib27" title="" class="ltx_ref">2021</a>); Menick et al. (<a href="#bib.bib25" title="" class="ltx_ref">2022</a>)</cite> share a similar motivation, but they mainly experiment with commercial search engines and closed-source models, making their results difficult to evaluate.
Retrieval-augmented LMs <cite class="ltx_cite ltx_citemacro_citep">(Borgeaud et al., <a href="#bib.bib2" title="" class="ltx_ref">2022</a>; Izacard et al., <a href="#bib.bib14" title="" class="ltx_ref">2022</a>)</cite>
incorporate retrieved passages during both training and inference,
but do not guarantee faithfulness to retrieved passages or explicitly provide citations.
Additionally,
previous studies mostly rely on human evaluation <cite class="ltx_cite ltx_citemacro_cite">Nakano et al. (<a href="#bib.bib27" title="" class="ltx_ref">2021</a>); Menick et al. (<a href="#bib.bib25" title="" class="ltx_ref">2022</a>); Liu et al. (<a href="#bib.bib24" title="" class="ltx_ref">2023</a>)</cite>, which is expensive and difficult to reproduce.
We argue that the absence of automated evaluation
hinders the advances of such systems.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">We present <span id="S1.p4.1.1" class="ltx_text ltx_font_bold">ALCE</span>, the first reproducible benchmark for <span id="S1.p4.1.2" class="ltx_text ltx_font_italic">automatically</span> evaluating LLMs’ generations with citations.
ALCE assumes a natural-language question and a retrieval corpus, and requires building end-to-end systems to retrieve relevant passages from the corpus, generate a response to the question, and cite corresponding supporting passages.
We compile three datasets that cover different types of questions and corpora—ASQA <cite class="ltx_cite ltx_citemacro_cite">Stelmakh et al. (<a href="#bib.bib44" title="" class="ltx_ref">2022</a>)</cite>, QAMPARI <cite class="ltx_cite ltx_citemacro_cite">Rubin et al. (<a href="#bib.bib39" title="" class="ltx_ref">2022</a>)</cite>, and ELI5 <cite class="ltx_cite ltx_citemacro_cite">Fan et al. (<a href="#bib.bib6" title="" class="ltx_ref">2019</a>)</cite>—as shown in Table <a href="#S1.T1" title="Table 1 ‣ 1 Introduction ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.
Different from previous benchmarks <cite class="ltx_cite ltx_citemacro_citep">(Lee et al., <a href="#bib.bib23" title="" class="ltx_ref">2019</a>; Bohnet et al., <a href="#bib.bib1" title="" class="ltx_ref">2022</a>)</cite>,
ALCE evaluates long-text generation,
focusing on automatically evaluating citation quality,
and allows citing <span id="S1.p4.1.3" class="ltx_text ltx_font_italic">multiple</span> passages for individual statements.</p>
</div>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.1" class="ltx_p">We design automatic evaluation methods in three dimensions:
<span id="S1.p5.1.1" class="ltx_text ltx_font_bold">fluency</span>, <span id="S1.p5.1.2" class="ltx_text ltx_font_bold">correctness</span>, and <span id="S1.p5.1.3" class="ltx_text ltx_font_bold">citation quality</span>.
Specifically,
we use MAUVE <cite class="ltx_cite ltx_citemacro_citep">(Pillutla et al., <a href="#bib.bib34" title="" class="ltx_ref">2021</a>)</cite> to measure fluency,
propose tailored correctness metrics for each dataset, and
adopt a natural language inference (NLI) model <cite class="ltx_cite ltx_citemacro_citep">(Honovich et al., <a href="#bib.bib12" title="" class="ltx_ref">2022</a>)</cite> to measure citation quality.
We showcase how the three dimensions together contribute to a robust evaluation,
preventing systems from exploiting shortcuts. Additionally, we conduct human evaluation and demonstrate a strong correlation with our automatic metrics.</p>
</div>
<div id="S1.p6" class="ltx_para">
<p id="S1.p6.1" class="ltx_p">We experiment on multiple systems with state-of-the-art LLMs and retrievers and also propose novel prompting strategies to synthesize retrieved text into text generation.
Although all systems are capable of providing fluent and coherent responses,
there remains substantial room for improvement in terms of correctness and citation quality: For example, on the ELI5 dataset,
around 50% generations of our ChatGPT and GPT-4 baselines are not fully supported by the cited passages.
Additionally, we find that
(1) a closed-book model (generating answers without accessing any retrieved documents) with post-hoc citing achieves good correctness but much worse citation quality;
(2) although interactive retrieval approaches <cite class="ltx_cite ltx_citemacro_citep">(Yao et al., <a href="#bib.bib51" title="" class="ltx_ref">2023</a>; Schick et al., <a href="#bib.bib40" title="" class="ltx_ref">2023</a>)</cite> offer more flexibility in when/what to retrieve, they do not improve the performance on this challenging benchmark;
(3) summarizing the retrieved passages in a shorter text improves correctness but not citation quality;
(4) reranking multiple generations boosts citation quality measured by human evaluation;
(5) incorporating more retrieved passages in context does not help ChatGPT but improves GPT-4 performance.</p>
</div>
<div id="S1.p7" class="ltx_para">
<p id="S1.p7.1" class="ltx_p">Our extensive analyses
highlight three major challenges of building LLMs to generate text with citations:
(1)
the retrieval quality is crucial to the final performance and has substantial room for improvement;
(2)
LLMs’ limited context window
restricts the number of passages they can incorporate;
(3)
current LLMs struggle to synthesize multiple documents in context
without being distracted by irrelevant ones, although better instruction tuning brings significant improvement.
These challenges pose promising research directions for
developing better systems integrating retrieval and LLMs.</p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Task Setup and Datasets</h2>

<div id="S2.p1" class="ltx_para">
<p id="S2.p1.9" class="ltx_p">Our task is formalized as follows:
Given a query <math id="S2.p1.1.m1.1" class="ltx_Math" alttext="q" display="inline"><semantics id="S2.p1.1.m1.1a"><mi id="S2.p1.1.m1.1.1" xref="S2.p1.1.m1.1.1.cmml">q</mi><annotation-xml encoding="MathML-Content" id="S2.p1.1.m1.1b"><ci id="S2.p1.1.m1.1.1.cmml" xref="S2.p1.1.m1.1.1">𝑞</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.1.m1.1c">q</annotation></semantics></math> and a corpus of text passages <math id="S2.p1.2.m2.1" class="ltx_Math" alttext="\mathcal{D}" display="inline"><semantics id="S2.p1.2.m2.1a"><mi class="ltx_font_mathcaligraphic" id="S2.p1.2.m2.1.1" xref="S2.p1.2.m2.1.1.cmml">𝒟</mi><annotation-xml encoding="MathML-Content" id="S2.p1.2.m2.1b"><ci id="S2.p1.2.m2.1.1.cmml" xref="S2.p1.2.m2.1.1">𝒟</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.2.m2.1c">\mathcal{D}</annotation></semantics></math>,
the system is required to return an output <math id="S2.p1.3.m3.1" class="ltx_Math" alttext="\mathcal{S}" display="inline"><semantics id="S2.p1.3.m3.1a"><mi class="ltx_font_mathcaligraphic" id="S2.p1.3.m3.1.1" xref="S2.p1.3.m3.1.1.cmml">𝒮</mi><annotation-xml encoding="MathML-Content" id="S2.p1.3.m3.1b"><ci id="S2.p1.3.m3.1.1.cmml" xref="S2.p1.3.m3.1.1">𝒮</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.3.m3.1c">\mathcal{S}</annotation></semantics></math>, which consists of <math id="S2.p1.4.m4.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S2.p1.4.m4.1a"><mi id="S2.p1.4.m4.1.1" xref="S2.p1.4.m4.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S2.p1.4.m4.1b"><ci id="S2.p1.4.m4.1.1.cmml" xref="S2.p1.4.m4.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.4.m4.1c">n</annotation></semantics></math> statements <math id="S2.p1.5.m5.1" class="ltx_Math" alttext="s_{1}" display="inline"><semantics id="S2.p1.5.m5.1a"><msub id="S2.p1.5.m5.1.1" xref="S2.p1.5.m5.1.1.cmml"><mi id="S2.p1.5.m5.1.1.2" xref="S2.p1.5.m5.1.1.2.cmml">s</mi><mn id="S2.p1.5.m5.1.1.3" xref="S2.p1.5.m5.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.p1.5.m5.1b"><apply id="S2.p1.5.m5.1.1.cmml" xref="S2.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S2.p1.5.m5.1.1.1.cmml" xref="S2.p1.5.m5.1.1">subscript</csymbol><ci id="S2.p1.5.m5.1.1.2.cmml" xref="S2.p1.5.m5.1.1.2">𝑠</ci><cn type="integer" id="S2.p1.5.m5.1.1.3.cmml" xref="S2.p1.5.m5.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.5.m5.1c">s_{1}</annotation></semantics></math>, …, <math id="S2.p1.6.m6.1" class="ltx_Math" alttext="s_{n}" display="inline"><semantics id="S2.p1.6.m6.1a"><msub id="S2.p1.6.m6.1.1" xref="S2.p1.6.m6.1.1.cmml"><mi id="S2.p1.6.m6.1.1.2" xref="S2.p1.6.m6.1.1.2.cmml">s</mi><mi id="S2.p1.6.m6.1.1.3" xref="S2.p1.6.m6.1.1.3.cmml">n</mi></msub><annotation-xml encoding="MathML-Content" id="S2.p1.6.m6.1b"><apply id="S2.p1.6.m6.1.1.cmml" xref="S2.p1.6.m6.1.1"><csymbol cd="ambiguous" id="S2.p1.6.m6.1.1.1.cmml" xref="S2.p1.6.m6.1.1">subscript</csymbol><ci id="S2.p1.6.m6.1.1.2.cmml" xref="S2.p1.6.m6.1.1.2">𝑠</ci><ci id="S2.p1.6.m6.1.1.3.cmml" xref="S2.p1.6.m6.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.6.m6.1c">s_{n}</annotation></semantics></math>,
and each statement <math id="S2.p1.7.m7.1" class="ltx_Math" alttext="s_{i}" display="inline"><semantics id="S2.p1.7.m7.1a"><msub id="S2.p1.7.m7.1.1" xref="S2.p1.7.m7.1.1.cmml"><mi id="S2.p1.7.m7.1.1.2" xref="S2.p1.7.m7.1.1.2.cmml">s</mi><mi id="S2.p1.7.m7.1.1.3" xref="S2.p1.7.m7.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.p1.7.m7.1b"><apply id="S2.p1.7.m7.1.1.cmml" xref="S2.p1.7.m7.1.1"><csymbol cd="ambiguous" id="S2.p1.7.m7.1.1.1.cmml" xref="S2.p1.7.m7.1.1">subscript</csymbol><ci id="S2.p1.7.m7.1.1.2.cmml" xref="S2.p1.7.m7.1.1.2">𝑠</ci><ci id="S2.p1.7.m7.1.1.3.cmml" xref="S2.p1.7.m7.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.7.m7.1c">s_{i}</annotation></semantics></math> cites a list of passages <math id="S2.p1.8.m8.7" class="ltx_Math" alttext="\mathcal{C}_{i}=\{c_{i,1},c_{i,2},\ldots\}" display="inline"><semantics id="S2.p1.8.m8.7a"><mrow id="S2.p1.8.m8.7.7" xref="S2.p1.8.m8.7.7.cmml"><msub id="S2.p1.8.m8.7.7.4" xref="S2.p1.8.m8.7.7.4.cmml"><mi class="ltx_font_mathcaligraphic" id="S2.p1.8.m8.7.7.4.2" xref="S2.p1.8.m8.7.7.4.2.cmml">𝒞</mi><mi id="S2.p1.8.m8.7.7.4.3" xref="S2.p1.8.m8.7.7.4.3.cmml">i</mi></msub><mo id="S2.p1.8.m8.7.7.3" xref="S2.p1.8.m8.7.7.3.cmml">=</mo><mrow id="S2.p1.8.m8.7.7.2.2" xref="S2.p1.8.m8.7.7.2.3.cmml"><mo stretchy="false" id="S2.p1.8.m8.7.7.2.2.3" xref="S2.p1.8.m8.7.7.2.3.cmml">{</mo><msub id="S2.p1.8.m8.6.6.1.1.1" xref="S2.p1.8.m8.6.6.1.1.1.cmml"><mi id="S2.p1.8.m8.6.6.1.1.1.2" xref="S2.p1.8.m8.6.6.1.1.1.2.cmml">c</mi><mrow id="S2.p1.8.m8.2.2.2.4" xref="S2.p1.8.m8.2.2.2.3.cmml"><mi id="S2.p1.8.m8.1.1.1.1" xref="S2.p1.8.m8.1.1.1.1.cmml">i</mi><mo id="S2.p1.8.m8.2.2.2.4.1" xref="S2.p1.8.m8.2.2.2.3.cmml">,</mo><mn id="S2.p1.8.m8.2.2.2.2" xref="S2.p1.8.m8.2.2.2.2.cmml">1</mn></mrow></msub><mo id="S2.p1.8.m8.7.7.2.2.4" xref="S2.p1.8.m8.7.7.2.3.cmml">,</mo><msub id="S2.p1.8.m8.7.7.2.2.2" xref="S2.p1.8.m8.7.7.2.2.2.cmml"><mi id="S2.p1.8.m8.7.7.2.2.2.2" xref="S2.p1.8.m8.7.7.2.2.2.2.cmml">c</mi><mrow id="S2.p1.8.m8.4.4.2.4" xref="S2.p1.8.m8.4.4.2.3.cmml"><mi id="S2.p1.8.m8.3.3.1.1" xref="S2.p1.8.m8.3.3.1.1.cmml">i</mi><mo id="S2.p1.8.m8.4.4.2.4.1" xref="S2.p1.8.m8.4.4.2.3.cmml">,</mo><mn id="S2.p1.8.m8.4.4.2.2" xref="S2.p1.8.m8.4.4.2.2.cmml">2</mn></mrow></msub><mo id="S2.p1.8.m8.7.7.2.2.5" xref="S2.p1.8.m8.7.7.2.3.cmml">,</mo><mi mathvariant="normal" id="S2.p1.8.m8.5.5" xref="S2.p1.8.m8.5.5.cmml">…</mi><mo stretchy="false" id="S2.p1.8.m8.7.7.2.2.6" xref="S2.p1.8.m8.7.7.2.3.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.p1.8.m8.7b"><apply id="S2.p1.8.m8.7.7.cmml" xref="S2.p1.8.m8.7.7"><eq id="S2.p1.8.m8.7.7.3.cmml" xref="S2.p1.8.m8.7.7.3"></eq><apply id="S2.p1.8.m8.7.7.4.cmml" xref="S2.p1.8.m8.7.7.4"><csymbol cd="ambiguous" id="S2.p1.8.m8.7.7.4.1.cmml" xref="S2.p1.8.m8.7.7.4">subscript</csymbol><ci id="S2.p1.8.m8.7.7.4.2.cmml" xref="S2.p1.8.m8.7.7.4.2">𝒞</ci><ci id="S2.p1.8.m8.7.7.4.3.cmml" xref="S2.p1.8.m8.7.7.4.3">𝑖</ci></apply><set id="S2.p1.8.m8.7.7.2.3.cmml" xref="S2.p1.8.m8.7.7.2.2"><apply id="S2.p1.8.m8.6.6.1.1.1.cmml" xref="S2.p1.8.m8.6.6.1.1.1"><csymbol cd="ambiguous" id="S2.p1.8.m8.6.6.1.1.1.1.cmml" xref="S2.p1.8.m8.6.6.1.1.1">subscript</csymbol><ci id="S2.p1.8.m8.6.6.1.1.1.2.cmml" xref="S2.p1.8.m8.6.6.1.1.1.2">𝑐</ci><list id="S2.p1.8.m8.2.2.2.3.cmml" xref="S2.p1.8.m8.2.2.2.4"><ci id="S2.p1.8.m8.1.1.1.1.cmml" xref="S2.p1.8.m8.1.1.1.1">𝑖</ci><cn type="integer" id="S2.p1.8.m8.2.2.2.2.cmml" xref="S2.p1.8.m8.2.2.2.2">1</cn></list></apply><apply id="S2.p1.8.m8.7.7.2.2.2.cmml" xref="S2.p1.8.m8.7.7.2.2.2"><csymbol cd="ambiguous" id="S2.p1.8.m8.7.7.2.2.2.1.cmml" xref="S2.p1.8.m8.7.7.2.2.2">subscript</csymbol><ci id="S2.p1.8.m8.7.7.2.2.2.2.cmml" xref="S2.p1.8.m8.7.7.2.2.2.2">𝑐</ci><list id="S2.p1.8.m8.4.4.2.3.cmml" xref="S2.p1.8.m8.4.4.2.4"><ci id="S2.p1.8.m8.3.3.1.1.cmml" xref="S2.p1.8.m8.3.3.1.1">𝑖</ci><cn type="integer" id="S2.p1.8.m8.4.4.2.2.cmml" xref="S2.p1.8.m8.4.4.2.2">2</cn></list></apply><ci id="S2.p1.8.m8.5.5.cmml" xref="S2.p1.8.m8.5.5">…</ci></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.8.m8.7c">\mathcal{C}_{i}=\{c_{i,1},c_{i,2},\ldots\}</annotation></semantics></math><span id="footnote4" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">4</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">4</sup><span class="ltx_tag ltx_tag_note">4</span>In practice, we allow at most 3 citations for each statement as more citations usually do not help.</span></span></span>, where <math id="S2.p1.9.m9.2" class="ltx_Math" alttext="c_{i,j}\in\mathcal{D}" display="inline"><semantics id="S2.p1.9.m9.2a"><mrow id="S2.p1.9.m9.2.3" xref="S2.p1.9.m9.2.3.cmml"><msub id="S2.p1.9.m9.2.3.2" xref="S2.p1.9.m9.2.3.2.cmml"><mi id="S2.p1.9.m9.2.3.2.2" xref="S2.p1.9.m9.2.3.2.2.cmml">c</mi><mrow id="S2.p1.9.m9.2.2.2.4" xref="S2.p1.9.m9.2.2.2.3.cmml"><mi id="S2.p1.9.m9.1.1.1.1" xref="S2.p1.9.m9.1.1.1.1.cmml">i</mi><mo id="S2.p1.9.m9.2.2.2.4.1" xref="S2.p1.9.m9.2.2.2.3.cmml">,</mo><mi id="S2.p1.9.m9.2.2.2.2" xref="S2.p1.9.m9.2.2.2.2.cmml">j</mi></mrow></msub><mo id="S2.p1.9.m9.2.3.1" xref="S2.p1.9.m9.2.3.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S2.p1.9.m9.2.3.3" xref="S2.p1.9.m9.2.3.3.cmml">𝒟</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.p1.9.m9.2b"><apply id="S2.p1.9.m9.2.3.cmml" xref="S2.p1.9.m9.2.3"><in id="S2.p1.9.m9.2.3.1.cmml" xref="S2.p1.9.m9.2.3.1"></in><apply id="S2.p1.9.m9.2.3.2.cmml" xref="S2.p1.9.m9.2.3.2"><csymbol cd="ambiguous" id="S2.p1.9.m9.2.3.2.1.cmml" xref="S2.p1.9.m9.2.3.2">subscript</csymbol><ci id="S2.p1.9.m9.2.3.2.2.cmml" xref="S2.p1.9.m9.2.3.2.2">𝑐</ci><list id="S2.p1.9.m9.2.2.2.3.cmml" xref="S2.p1.9.m9.2.2.2.4"><ci id="S2.p1.9.m9.1.1.1.1.cmml" xref="S2.p1.9.m9.1.1.1.1">𝑖</ci><ci id="S2.p1.9.m9.2.2.2.2.cmml" xref="S2.p1.9.m9.2.2.2.2">𝑗</ci></list></apply><ci id="S2.p1.9.m9.2.3.3.cmml" xref="S2.p1.9.m9.2.3.3">𝒟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.9.m9.2c">c_{i,j}\in\mathcal{D}</annotation></semantics></math>.
In this work,
we segment LLMs’ output into statements by sentence boundaries.<span id="footnote5" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">5</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">5</sup><span class="ltx_tag ltx_tag_note">5</span>
QAMPARI requires a list as the answer, and we choose each entity in the generated list as a statement.</span></span></span>
While LLMs may include sentences that do not require a citation, such as “<span id="S2.p1.9.1" class="ltx_text ltx_font_italic">I’m happy to help</span>”,
we observe that almost all sentences that LLMs output provide valuable information and require citations, similar to findings in
<cite class="ltx_cite ltx_citemacro_citet">Liu et al. (<a href="#bib.bib24" title="" class="ltx_ref">2023</a>)</cite>.
In this work, citations are enclosed by box brackets such as <span id="S2.p1.9.2" class="ltx_text ltx_font_typewriter">[1][2]</span>.</p>
</div>
<div id="S2.p2" class="ltx_para">
<p id="S2.p2.1" class="ltx_p">We divide the corpus <math id="S2.p2.1.m1.1" class="ltx_Math" alttext="\mathcal{D}" display="inline"><semantics id="S2.p2.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S2.p2.1.m1.1.1" xref="S2.p2.1.m1.1.1.cmml">𝒟</mi><annotation-xml encoding="MathML-Content" id="S2.p2.1.m1.1b"><ci id="S2.p2.1.m1.1.1.cmml" xref="S2.p2.1.m1.1.1">𝒟</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p2.1.m1.1c">\mathcal{D}</annotation></semantics></math> into 100-word passages following previous works on open-domain question answering <cite class="ltx_cite ltx_citemacro_cite">Karpukhin et al. (<a href="#bib.bib18" title="" class="ltx_ref">2020</a>); Petroni et al. (<a href="#bib.bib32" title="" class="ltx_ref">2021</a>); Piktus et al. (<a href="#bib.bib33" title="" class="ltx_ref">2021</a>)</cite>,
in contrast to
commercial systems like Bing Chat, which cite entire Web pages. We take 100-word passages because
it is easier for humans to verify, and allows for more retrieved passages to fit in LLMs’ limited context.</p>
</div>
<div id="S2.p3" class="ltx_para">
<p id="S2.p3.1" class="ltx_p">We choose QA datasets so that
(1) they contain factual questions, in which references are important;
(2) questions require long-text answers that cover multiple aspects; (3) answering the questions requires synthesizing multiple sources.
We select three datasets (Table <a href="#S1.T1" title="Table 1 ‣ 1 Introduction ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>) and introduce them below. See §<a href="#A2" title="Appendix B Dataset Statistics ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a> for additional statistics.</p>
</div>
<div id="S2.p4" class="ltx_para">
<p id="S2.p4.1" class="ltx_p"><span id="S2.p4.1.1" class="ltx_text ltx_font_bold">ASQA</span> <cite class="ltx_cite ltx_citemacro_cite">Stelmakh et al. (<a href="#bib.bib44" title="" class="ltx_ref">2022</a>)</cite> is a long-form factoid dataset.
As shown in Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>,
each question is an ambiguous question from AmbigQA <cite class="ltx_cite ltx_citemacro_cite">Min et al. (<a href="#bib.bib26" title="" class="ltx_ref">2020</a>)</cite> that requires multiple short answers to cover different aspects,
and the dataset provides a long-form answer that covers all short answers.
Since most questions can be answered by Wikipedia, we use the 2018-12-20 Wikipedia snapshot as <math id="S2.p4.1.m1.1" class="ltx_Math" alttext="\mathcal{D}" display="inline"><semantics id="S2.p4.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S2.p4.1.m1.1.1" xref="S2.p4.1.m1.1.1.cmml">𝒟</mi><annotation-xml encoding="MathML-Content" id="S2.p4.1.m1.1b"><ci id="S2.p4.1.m1.1.1.cmml" xref="S2.p4.1.m1.1.1">𝒟</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p4.1.m1.1c">\mathcal{D}</annotation></semantics></math>.</p>
</div>
<div id="S2.p5" class="ltx_para">
<p id="S2.p5.1" class="ltx_p"><span id="S2.p5.1.1" class="ltx_text ltx_font_bold">QAMPARI</span> <cite class="ltx_cite ltx_citemacro_cite">Rubin et al. (<a href="#bib.bib39" title="" class="ltx_ref">2022</a>)</cite>
is a factoid QA dataset constructed from Wikipedia,
where the answer is a list of entities that are drawn from different passages.
Same as ASQA, we use the 2018-12-20 Wikipedia as the corpus.</p>
</div>
<div id="S2.p6" class="ltx_para">
<p id="S2.p6.1" class="ltx_p"><span id="S2.p6.1.1" class="ltx_text ltx_font_bold">ELI5</span> <cite class="ltx_cite ltx_citemacro_cite">Fan et al. (<a href="#bib.bib6" title="" class="ltx_ref">2019</a>)</cite>
is a long-form QA dataset built on the Reddit forum “Explain Like I’m Five”.<span id="footnote6" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">6</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">6</sup><span class="ltx_tag ltx_tag_note">6</span><a target="_blank" href="https://www.reddit.com/r/explainlikeimfive/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.reddit.com/r/explainlikeimfive/</a></span></span></span> Most ELI5 questions are how/why/what questions that require long answers and multiple passages as evidence.
Due to the diverse topics discussed in the questions, we use
Sphere <cite class="ltx_cite ltx_citemacro_cite">Piktus et al. (<a href="#bib.bib33" title="" class="ltx_ref">2021</a>)</cite>—a filtered version of Common Crawl<span id="footnote7" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">7</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">7</sup><span class="ltx_tag ltx_tag_note">7</span><a target="_blank" href="https://commoncrawl.org" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://commoncrawl.org</a>. We also filter out any Web pages from Reddit. </span></span></span>—as the corpus. The ELI5 dataset is widely used in related work due to its challenging nature <cite class="ltx_cite ltx_citemacro_cite">Nakano et al. (<a href="#bib.bib27" title="" class="ltx_ref">2021</a>); Menick et al. (<a href="#bib.bib25" title="" class="ltx_ref">2022</a>); Liu et al. (<a href="#bib.bib24" title="" class="ltx_ref">2023</a>)</cite>.</p>
</div>
<div id="S2.p7" class="ltx_para">
<p id="S2.p7.1" class="ltx_p">We randomly select 1,000 examples from the development set of each dataset for ALCE.
Our benchmark primarily assesses the citation capabilities of existing LLMs and does not provide training data,
as
there are no available examples that provide supervision for citations
in these datasets.</p>
</div>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Automatic Evaluation</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.1" class="ltx_p">Our benchmark measures the following three dimensions of system responses:</p>
<ul id="S3.I1" class="ltx_itemize">
<li id="S3.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I1.i1.p1" class="ltx_para">
<p id="S3.I1.i1.p1.1" class="ltx_p"><span id="S3.I1.i1.p1.1.1" class="ltx_text ltx_font_bold">Fluency</span>: whether the model’s generated text is fluent and coherent.</p>
</div>
</li>
<li id="S3.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I1.i2.p1" class="ltx_para">
<p id="S3.I1.i2.p1.1" class="ltx_p"><span id="S3.I1.i2.p1.1.1" class="ltx_text ltx_font_bold">Correctness</span>: whether the answer is accurate and covers all aspects of interest.</p>
</div>
</li>
<li id="S3.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I1.i3.p1" class="ltx_para">
<p id="S3.I1.i3.p1.1" class="ltx_p"><span id="S3.I1.i3.p1.1.1" class="ltx_text ltx_font_bold">Citation quality</span>: whether the answer is well supported by the cited passages and no irrelevant passages are cited.</p>
</div>
</li>
</ul>
<p id="S3.p1.2" class="ltx_p">In the following, we present automatic metrics for each dimension
and discuss why the combination of the three metrics provides a robust evaluation.</p>
</div>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Fluency</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.1" class="ltx_p">We use MAUVE <cite class="ltx_cite ltx_citemacro_cite">Pillutla et al. (<a href="#bib.bib34" title="" class="ltx_ref">2021</a>)</cite> to evaluate the fluency of the output (§<a href="#A3" title="Appendix C Implementation Details ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>).
We deploy MAUVE for ASQA and ELI5
and omit it for QAMPARI,
as QAMPARI
only requires a list of short answers as the response
and LLMs consistently adhere to the format in our experiments.
As MAUVE is sensitive to output length and text style, and most LLMs are capable of producing fluent text,
we mainly employ it as a sanity check as long as the MAUVE scores are high enough.</p>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Correctness</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.1" class="ltx_p">Our objective is to measure the informativeness and utility of the generation to the question.
<cite class="ltx_cite ltx_citemacro_citet">Liu et al. (<a href="#bib.bib24" title="" class="ltx_ref">2023</a>)</cite> propose to directly evaluate <span id="S3.SS2.p1.1.1" class="ltx_text ltx_font_italic">perceived utility</span>
by humans, a process difficult to automate.
Therefore, we use correctness—whether the response is accurate compared to a ground truth answer—as a proxy.
Evaluating the correctness of long-form generation is a challenging task <cite class="ltx_cite ltx_citemacro_citep">(Krishna et al., <a href="#bib.bib21" title="" class="ltx_ref">2021</a>)</cite>,
and we describe our strategy for each dataset below.
Figure <a href="#S3.F2" title="Figure 2 ‣ 3.2 Correctness ‣ 3 Automatic Evaluation ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> illustrates the metrics and we include additional implementation details in §<a href="#A3" title="Appendix C Implementation Details ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>.</p>
</div>
<div id="S3.SS2.p2" class="ltx_para">
<p id="S3.SS2.p2.1" class="ltx_p">For <span id="S3.SS2.p2.1.1" class="ltx_text ltx_font_bold">ASQA</span>, we follow <cite class="ltx_cite ltx_citemacro_citet">Stelmakh et al. (<a href="#bib.bib44" title="" class="ltx_ref">2022</a>)</cite> and
calculate the recall of correct short answers
by checking whether the short answers (provided by the dataset) are exact substrings of the generation (<span id="S3.SS2.p2.1.2" class="ltx_text ltx_font_italic">exact match recall</span>; EM recall).</p>
</div>
<div id="S3.SS2.p3" class="ltx_para">
<p id="S3.SS2.p3.1" class="ltx_p">For <span id="S3.SS2.p3.1.1" class="ltx_text ltx_font_bold">QAMPARI</span>, we follow <cite class="ltx_cite ltx_citemacro_citet">Rubin et al. (<a href="#bib.bib39" title="" class="ltx_ref">2022</a>)</cite> and calculate the <span id="S3.SS2.p3.1.2" class="ltx_text ltx_font_italic">precision</span> and <span id="S3.SS2.p3.1.3" class="ltx_text ltx_font_italic">recall</span> of
the model prediction,
by checking the exact match to the gold answer list.
We add one additional adjustment:
considering that users often want to know only a few example answers of
the question, our evaluation considers recall to be 100% if the prediction includes at least 5 correct answers (<span id="S3.SS2.p3.1.4" class="ltx_text ltx_font_italic">recall-5</span>).</p>
</div>
<div id="S3.SS2.p4" class="ltx_para">
<p id="S3.SS2.p4.1" class="ltx_p">Unlike ASQA and QAMPARI, the <span id="S3.SS2.p4.1.1" class="ltx_text ltx_font_bold">ELI5</span> dataset does not provide short entity answers.
<cite class="ltx_cite ltx_citemacro_citet">Fan et al. (<a href="#bib.bib6" title="" class="ltx_ref">2019</a>)</cite> use ROUGE for evaluation, which does not reflect the correctness well (<cite class="ltx_cite ltx_citemacro_citep">Krishna et al., <a href="#bib.bib21" title="" class="ltx_ref">2021</a></cite>; §<a href="#A1" title="Appendix A Generating Claims for ELI5 ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a>).
Inspired by works in summarization evaluation <cite class="ltx_cite ltx_citemacro_cite">Zhang and Bansal (<a href="#bib.bib53" title="" class="ltx_ref">2021</a>); Kamoi et al. (<a href="#bib.bib17" title="" class="ltx_ref">2023</a>); Wang et al. (<a href="#bib.bib49" title="" class="ltx_ref">2020</a>)</cite>,
we use InstructGPT (<span id="S3.SS2.p4.1.2" class="ltx_text ltx_font_typewriter">text-davinci-003</span>; <cite class="ltx_cite ltx_citemacro_citep">Ouyang et al., <a href="#bib.bib30" title="" class="ltx_ref">2022</a></cite>) to generate three “sub-claims”.
Then we use TRUE<span id="footnote8" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">8</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">8</sup><span class="ltx_tag ltx_tag_note">8</span><a target="_blank" href="https://huggingface.co/google/t5_xxl_true_nli_mixture" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/google/t5_xxl_true_nli_mixture</a>. Details in §<a href="#A3" title="Appendix C Implementation Details ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>.</span></span></span> <cite class="ltx_cite ltx_citemacro_cite">Honovich et al. (<a href="#bib.bib12" title="" class="ltx_ref">2022</a>)</cite>, a T5-11B <cite class="ltx_cite ltx_citemacro_citep">(Raffel et al., <a href="#bib.bib36" title="" class="ltx_ref">2020</a>)</cite> model fine-tuned on a collection of natural language inference (NLI) datasets, to check whether the model output entails the sub-claims (<span id="S3.SS2.p4.1.3" class="ltx_text ltx_font_italic">claim recall</span>). TRUE targets factual correctness and has been used by previous works in similar context <cite class="ltx_cite ltx_citemacro_citep">(Bohnet et al., <a href="#bib.bib1" title="" class="ltx_ref">2022</a>; Gao et al., <a href="#bib.bib8" title="" class="ltx_ref">2023</a>)</cite>.
We demonstrate that claim recall provides a more accurate measure of correctness than existing metrics (more details in §<a href="#A1" title="Appendix A Generating Claims for ELI5 ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a>).</p>
</div>
<figure id="S3.F2" class="ltx_figure"><img src="/html/2305.14627/assets/x2.png" id="S3.F2.g1" class="ltx_graphics ltx_centering ltx_img_square" width="433" height="522" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>Evaluation of correctness (details in §<a href="#S3.SS2" title="3.2 Correctness ‣ 3 Automatic Evaluation ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a>).</figcaption>
</figure>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Citation Quality</h3>

<div id="S3.SS3.p1" class="ltx_para">
<p id="S3.SS3.p1.1" class="ltx_p">We evaluate citation qualities using two metrics:
(1) <span id="S3.SS3.p1.1.1" class="ltx_text ltx_font_italic">citation recall</span>, which determines if the output is entirely supported by cited passages, and
(2) <span id="S3.SS3.p1.1.2" class="ltx_text ltx_font_italic">citation precision</span>, which identifies any irrelevant citations. Although we prioritize citation recall as it entails a well-supported and truthful answer,
enhancing precision is crucial for better user satisfaction,
reducing the need for human review of extraneous passages.
Figure <a href="#S3.F3" title="Figure 3 ‣ 3.3 Citation Quality ‣ 3 Automatic Evaluation ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> provides an illustrated example.</p>
</div>
<figure id="S3.F3" class="ltx_figure"><img src="/html/2305.14627/assets/x3.png" id="S3.F3.g1" class="ltx_graphics ltx_centering ltx_img_square" width="452" height="528" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>Evaluation of citation quality (details in §<a href="#S3.SS3" title="3.3 Citation Quality ‣ 3 Automatic Evaluation ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3</span></a>). We use
an NLI model to verify whether a statement is supported by its citations. </figcaption>
</figure>
<div id="S3.SS3.p2" class="ltx_para">
<p id="S3.SS3.p2.1" class="ltx_p">We use the NLI model TRUE <cite class="ltx_cite ltx_citemacro_citep">(Honovich et al., <a href="#bib.bib12" title="" class="ltx_ref">2022</a>)</cite> again
to automatically examine whether the cited passages entail the model generation.
We conduct human evaluation (§<a href="#S6" title="6 Human Evaluation ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>) to demonstrate strong human correlation of our metric.</p>
</div>
<section id="S3.SS3.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Citation recall.</h4>

<div id="S3.SS3.SSS0.Px1.p1" class="ltx_para">
<p id="S3.SS3.SSS0.Px1.p1.12" class="ltx_p">We calculate the citation recall of <span id="S3.SS3.SSS0.Px1.p1.12.1" class="ltx_text ltx_font_italic">each statement</span> (0 or 1) and average
over all statements in the model response. For each statement <math id="S3.SS3.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="s_{i}" display="inline"><semantics id="S3.SS3.SSS0.Px1.p1.1.m1.1a"><msub id="S3.SS3.SSS0.Px1.p1.1.m1.1.1" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.cmml"><mi id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.cmml">s</mi><mi id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p1.1.m1.1b"><apply id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2">𝑠</ci><ci id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p1.1.m1.1c">s_{i}</annotation></semantics></math>, its citation recall is <math id="S3.SS3.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S3.SS3.SSS0.Px1.p1.2.m2.1a"><mn id="S3.SS3.SSS0.Px1.p1.2.m2.1.1" xref="S3.SS3.SSS0.Px1.p1.2.m2.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p1.2.m2.1b"><cn type="integer" id="S3.SS3.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.2.m2.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p1.2.m2.1c">1</annotation></semantics></math> if and only if there is at least one citation (<math id="S3.SS3.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="\mathcal{C}_{i}\neq\emptyset" display="inline"><semantics id="S3.SS3.SSS0.Px1.p1.3.m3.1a"><mrow id="S3.SS3.SSS0.Px1.p1.3.m3.1.1" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1.cmml"><msub id="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2.2" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2.2.cmml">𝒞</mi><mi id="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2.3" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2.3.cmml">i</mi></msub><mo id="S3.SS3.SSS0.Px1.p1.3.m3.1.1.1" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1.1.cmml">≠</mo><mi mathvariant="normal" id="S3.SS3.SSS0.Px1.p1.3.m3.1.1.3" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1.3.cmml">∅</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p1.3.m3.1b"><apply id="S3.SS3.SSS0.Px1.p1.3.m3.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1"><neq id="S3.SS3.SSS0.Px1.p1.3.m3.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1.1"></neq><apply id="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2.1.cmml" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2.2.cmml" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2.2">𝒞</ci><ci id="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2.3.cmml" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2.3">𝑖</ci></apply><emptyset id="S3.SS3.SSS0.Px1.p1.3.m3.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1.3"></emptyset></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p1.3.m3.1c">\mathcal{C}_{i}\neq\emptyset</annotation></semantics></math>) and
<math id="S3.SS3.SSS0.Px1.p1.4.m4.2" class="ltx_Math" alttext="\phi(\text{concat}(\mathcal{C}_{i}),s_{i})=1" display="inline"><semantics id="S3.SS3.SSS0.Px1.p1.4.m4.2a"><mrow id="S3.SS3.SSS0.Px1.p1.4.m4.2.2" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.cmml"><mrow id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.cmml"><mi id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.4" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.4.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.3" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.3.cmml">​</mo><mrow id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.3.cmml"><mo stretchy="false" id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2.3" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.3.cmml">(</mo><mrow id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.cmml"><mtext id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.3" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.3a.cmml">concat</mtext><mo lspace="0em" rspace="0em" id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.2" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.1.1" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.1.1.2" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.1.1.1.cmml">(</mo><msub id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.1.1.1" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.1.1.1.2" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.1.1.1.2.cmml">𝒞</mi><mi id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.1.1.1.3" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.1.1.3" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2.4" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.3.cmml">,</mo><msub id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2.2" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2.2.cmml"><mi id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2.2.2" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2.2.2.cmml">s</mi><mi id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2.2.3" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2.2.3.cmml">i</mi></msub><mo stretchy="false" id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2.5" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.3.cmml">)</mo></mrow></mrow><mo id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.3" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.3.cmml">=</mo><mn id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.4" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.4.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p1.4.m4.2b"><apply id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2"><eq id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.3.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.3"></eq><apply id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2"><times id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.3.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.3"></times><ci id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.4.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.4">italic-ϕ</ci><interval closure="open" id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.3.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2"><apply id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1"><times id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.2"></times><ci id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.3a.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.3"><mtext id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.3">concat</mtext></ci><apply id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.1.1.1.2">𝒞</ci><ci id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.1.1.1.3">𝑖</ci></apply></apply><apply id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2.2.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2.2.1.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2.2.2.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2.2.2">𝑠</ci><ci id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2.2.3.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2.2.3">𝑖</ci></apply></interval></apply><cn type="integer" id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.4.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.4">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p1.4.m4.2c">\phi(\text{concat}(\mathcal{C}_{i}),s_{i})=1</annotation></semantics></math>,
where
<math id="S3.SS3.SSS0.Px1.p1.5.m5.2" class="ltx_Math" alttext="\phi(\text{premise},\text{hypothesis})" display="inline"><semantics id="S3.SS3.SSS0.Px1.p1.5.m5.2a"><mrow id="S3.SS3.SSS0.Px1.p1.5.m5.2.3" xref="S3.SS3.SSS0.Px1.p1.5.m5.2.3.cmml"><mi id="S3.SS3.SSS0.Px1.p1.5.m5.2.3.2" xref="S3.SS3.SSS0.Px1.p1.5.m5.2.3.2.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S3.SS3.SSS0.Px1.p1.5.m5.2.3.1" xref="S3.SS3.SSS0.Px1.p1.5.m5.2.3.1.cmml">​</mo><mrow id="S3.SS3.SSS0.Px1.p1.5.m5.2.3.3.2" xref="S3.SS3.SSS0.Px1.p1.5.m5.2.3.3.1.cmml"><mo stretchy="false" id="S3.SS3.SSS0.Px1.p1.5.m5.2.3.3.2.1" xref="S3.SS3.SSS0.Px1.p1.5.m5.2.3.3.1.cmml">(</mo><mtext id="S3.SS3.SSS0.Px1.p1.5.m5.1.1" xref="S3.SS3.SSS0.Px1.p1.5.m5.1.1a.cmml">premise</mtext><mo id="S3.SS3.SSS0.Px1.p1.5.m5.2.3.3.2.2" xref="S3.SS3.SSS0.Px1.p1.5.m5.2.3.3.1.cmml">,</mo><mtext id="S3.SS3.SSS0.Px1.p1.5.m5.2.2" xref="S3.SS3.SSS0.Px1.p1.5.m5.2.2a.cmml">hypothesis</mtext><mo stretchy="false" id="S3.SS3.SSS0.Px1.p1.5.m5.2.3.3.2.3" xref="S3.SS3.SSS0.Px1.p1.5.m5.2.3.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p1.5.m5.2b"><apply id="S3.SS3.SSS0.Px1.p1.5.m5.2.3.cmml" xref="S3.SS3.SSS0.Px1.p1.5.m5.2.3"><times id="S3.SS3.SSS0.Px1.p1.5.m5.2.3.1.cmml" xref="S3.SS3.SSS0.Px1.p1.5.m5.2.3.1"></times><ci id="S3.SS3.SSS0.Px1.p1.5.m5.2.3.2.cmml" xref="S3.SS3.SSS0.Px1.p1.5.m5.2.3.2">italic-ϕ</ci><interval closure="open" id="S3.SS3.SSS0.Px1.p1.5.m5.2.3.3.1.cmml" xref="S3.SS3.SSS0.Px1.p1.5.m5.2.3.3.2"><ci id="S3.SS3.SSS0.Px1.p1.5.m5.1.1a.cmml" xref="S3.SS3.SSS0.Px1.p1.5.m5.1.1"><mtext id="S3.SS3.SSS0.Px1.p1.5.m5.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.5.m5.1.1">premise</mtext></ci><ci id="S3.SS3.SSS0.Px1.p1.5.m5.2.2a.cmml" xref="S3.SS3.SSS0.Px1.p1.5.m5.2.2"><mtext id="S3.SS3.SSS0.Px1.p1.5.m5.2.2.cmml" xref="S3.SS3.SSS0.Px1.p1.5.m5.2.2">hypothesis</mtext></ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p1.5.m5.2c">\phi(\text{premise},\text{hypothesis})</annotation></semantics></math>
is the NLI model that outputs <math id="S3.SS3.SSS0.Px1.p1.6.m6.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S3.SS3.SSS0.Px1.p1.6.m6.1a"><mn id="S3.SS3.SSS0.Px1.p1.6.m6.1.1" xref="S3.SS3.SSS0.Px1.p1.6.m6.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p1.6.m6.1b"><cn type="integer" id="S3.SS3.SSS0.Px1.p1.6.m6.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.6.m6.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p1.6.m6.1c">1</annotation></semantics></math> if the premise entails the hypothesis, and <math id="S3.SS3.SSS0.Px1.p1.7.m7.1" class="ltx_Math" alttext="0" display="inline"><semantics id="S3.SS3.SSS0.Px1.p1.7.m7.1a"><mn id="S3.SS3.SSS0.Px1.p1.7.m7.1.1" xref="S3.SS3.SSS0.Px1.p1.7.m7.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p1.7.m7.1b"><cn type="integer" id="S3.SS3.SSS0.Px1.p1.7.m7.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.7.m7.1.1">0</cn></annotation-xml></semantics></math> otherwise;
<math id="S3.SS3.SSS0.Px1.p1.8.m8.1" class="ltx_Math" alttext="\text{concat}(\mathcal{C}_{i})" display="inline"><semantics id="S3.SS3.SSS0.Px1.p1.8.m8.1a"><mrow id="S3.SS3.SSS0.Px1.p1.8.m8.1.1" xref="S3.SS3.SSS0.Px1.p1.8.m8.1.1.cmml"><mtext id="S3.SS3.SSS0.Px1.p1.8.m8.1.1.3" xref="S3.SS3.SSS0.Px1.p1.8.m8.1.1.3a.cmml">concat</mtext><mo lspace="0em" rspace="0em" id="S3.SS3.SSS0.Px1.p1.8.m8.1.1.2" xref="S3.SS3.SSS0.Px1.p1.8.m8.1.1.2.cmml">​</mo><mrow id="S3.SS3.SSS0.Px1.p1.8.m8.1.1.1.1" xref="S3.SS3.SSS0.Px1.p1.8.m8.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS3.SSS0.Px1.p1.8.m8.1.1.1.1.2" xref="S3.SS3.SSS0.Px1.p1.8.m8.1.1.1.1.1.cmml">(</mo><msub id="S3.SS3.SSS0.Px1.p1.8.m8.1.1.1.1.1" xref="S3.SS3.SSS0.Px1.p1.8.m8.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.SSS0.Px1.p1.8.m8.1.1.1.1.1.2" xref="S3.SS3.SSS0.Px1.p1.8.m8.1.1.1.1.1.2.cmml">𝒞</mi><mi id="S3.SS3.SSS0.Px1.p1.8.m8.1.1.1.1.1.3" xref="S3.SS3.SSS0.Px1.p1.8.m8.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S3.SS3.SSS0.Px1.p1.8.m8.1.1.1.1.3" xref="S3.SS3.SSS0.Px1.p1.8.m8.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p1.8.m8.1b"><apply id="S3.SS3.SSS0.Px1.p1.8.m8.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.8.m8.1.1"><times id="S3.SS3.SSS0.Px1.p1.8.m8.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p1.8.m8.1.1.2"></times><ci id="S3.SS3.SSS0.Px1.p1.8.m8.1.1.3a.cmml" xref="S3.SS3.SSS0.Px1.p1.8.m8.1.1.3"><mtext id="S3.SS3.SSS0.Px1.p1.8.m8.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p1.8.m8.1.1.3">concat</mtext></ci><apply id="S3.SS3.SSS0.Px1.p1.8.m8.1.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.8.m8.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p1.8.m8.1.1.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.8.m8.1.1.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px1.p1.8.m8.1.1.1.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p1.8.m8.1.1.1.1.1.2">𝒞</ci><ci id="S3.SS3.SSS0.Px1.p1.8.m8.1.1.1.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p1.8.m8.1.1.1.1.1.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p1.8.m8.1c">\text{concat}(\mathcal{C}_{i})</annotation></semantics></math> concatenates all passages in <math id="S3.SS3.SSS0.Px1.p1.9.m9.1" class="ltx_Math" alttext="\mathcal{C}_{i}" display="inline"><semantics id="S3.SS3.SSS0.Px1.p1.9.m9.1a"><msub id="S3.SS3.SSS0.Px1.p1.9.m9.1.1" xref="S3.SS3.SSS0.Px1.p1.9.m9.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.SSS0.Px1.p1.9.m9.1.1.2" xref="S3.SS3.SSS0.Px1.p1.9.m9.1.1.2.cmml">𝒞</mi><mi id="S3.SS3.SSS0.Px1.p1.9.m9.1.1.3" xref="S3.SS3.SSS0.Px1.p1.9.m9.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p1.9.m9.1b"><apply id="S3.SS3.SSS0.Px1.p1.9.m9.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.9.m9.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p1.9.m9.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.9.m9.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px1.p1.9.m9.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p1.9.m9.1.1.2">𝒞</ci><ci id="S3.SS3.SSS0.Px1.p1.9.m9.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p1.9.m9.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p1.9.m9.1c">\mathcal{C}_{i}</annotation></semantics></math> together (details in §<a href="#A3" title="Appendix C Implementation Details ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>).
The NLI evaluation is in accordance with the <span id="S3.SS3.SSS0.Px1.p1.12.2" class="ltx_text ltx_font_italic">attributable to identified sources</span> (AIS) framework <cite class="ltx_cite ltx_citemacro_cite">Rashkin et al. (<a href="#bib.bib38" title="" class="ltx_ref">2023</a>)</cite>:
<math id="S3.SS3.SSS0.Px1.p1.10.m10.2" class="ltx_Math" alttext="\phi(\text{concat}(\mathcal{C}_{i}),s_{i})=1" display="inline"><semantics id="S3.SS3.SSS0.Px1.p1.10.m10.2a"><mrow id="S3.SS3.SSS0.Px1.p1.10.m10.2.2" xref="S3.SS3.SSS0.Px1.p1.10.m10.2.2.cmml"><mrow id="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2" xref="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.cmml"><mi id="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.4" xref="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.4.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.3" xref="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.3.cmml">​</mo><mrow id="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.2.2" xref="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.2.3.cmml"><mo stretchy="false" id="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.2.2.3" xref="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.2.3.cmml">(</mo><mrow id="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1" xref="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.cmml"><mtext id="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.3" xref="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.3a.cmml">concat</mtext><mo lspace="0em" rspace="0em" id="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.2" xref="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.1.1" xref="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.1.1.2" xref="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.1.1.1.cmml">(</mo><msub id="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.1.1.1" xref="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.1.1.1.2" xref="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.1.1.1.2.cmml">𝒞</mi><mi id="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.1.1.1.3" xref="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.1.1.3" xref="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.2.2.4" xref="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.2.3.cmml">,</mo><msub id="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.2.2.2" xref="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.2.2.2.cmml"><mi id="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.2.2.2.2" xref="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.2.2.2.2.cmml">s</mi><mi id="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.2.2.2.3" xref="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.2.2.2.3.cmml">i</mi></msub><mo stretchy="false" id="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.2.2.5" xref="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.2.3.cmml">)</mo></mrow></mrow><mo id="S3.SS3.SSS0.Px1.p1.10.m10.2.2.3" xref="S3.SS3.SSS0.Px1.p1.10.m10.2.2.3.cmml">=</mo><mn id="S3.SS3.SSS0.Px1.p1.10.m10.2.2.4" xref="S3.SS3.SSS0.Px1.p1.10.m10.2.2.4.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p1.10.m10.2b"><apply id="S3.SS3.SSS0.Px1.p1.10.m10.2.2.cmml" xref="S3.SS3.SSS0.Px1.p1.10.m10.2.2"><eq id="S3.SS3.SSS0.Px1.p1.10.m10.2.2.3.cmml" xref="S3.SS3.SSS0.Px1.p1.10.m10.2.2.3"></eq><apply id="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.cmml" xref="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2"><times id="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.3.cmml" xref="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.3"></times><ci id="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.4.cmml" xref="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.4">italic-ϕ</ci><interval closure="open" id="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.2.3.cmml" xref="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.2.2"><apply id="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1"><times id="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.2"></times><ci id="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.3a.cmml" xref="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.3"><mtext id="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.3">concat</mtext></ci><apply id="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.1.1.1.2">𝒞</ci><ci id="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.1.1.1.3">𝑖</ci></apply></apply><apply id="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.2.2.2.cmml" xref="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.2.2.2.1.cmml" xref="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.2.2.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.2.2.2.2.cmml" xref="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.2.2.2.2">𝑠</ci><ci id="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.2.2.2.3.cmml" xref="S3.SS3.SSS0.Px1.p1.10.m10.2.2.2.2.2.2.3">𝑖</ci></apply></interval></apply><cn type="integer" id="S3.SS3.SSS0.Px1.p1.10.m10.2.2.4.cmml" xref="S3.SS3.SSS0.Px1.p1.10.m10.2.2.4">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p1.10.m10.2c">\phi(\text{concat}(\mathcal{C}_{i}),s_{i})=1</annotation></semantics></math> implies that
<math id="S3.SS3.SSS0.Px1.p1.11.m11.1" class="ltx_Math" alttext="s_{i}" display="inline"><semantics id="S3.SS3.SSS0.Px1.p1.11.m11.1a"><msub id="S3.SS3.SSS0.Px1.p1.11.m11.1.1" xref="S3.SS3.SSS0.Px1.p1.11.m11.1.1.cmml"><mi id="S3.SS3.SSS0.Px1.p1.11.m11.1.1.2" xref="S3.SS3.SSS0.Px1.p1.11.m11.1.1.2.cmml">s</mi><mi id="S3.SS3.SSS0.Px1.p1.11.m11.1.1.3" xref="S3.SS3.SSS0.Px1.p1.11.m11.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p1.11.m11.1b"><apply id="S3.SS3.SSS0.Px1.p1.11.m11.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.11.m11.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p1.11.m11.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.11.m11.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px1.p1.11.m11.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p1.11.m11.1.1.2">𝑠</ci><ci id="S3.SS3.SSS0.Px1.p1.11.m11.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p1.11.m11.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p1.11.m11.1c">s_{i}</annotation></semantics></math> is true based solely on <math id="S3.SS3.SSS0.Px1.p1.12.m12.1" class="ltx_Math" alttext="\text{concat}(\mathcal{C}_{i})" display="inline"><semantics id="S3.SS3.SSS0.Px1.p1.12.m12.1a"><mrow id="S3.SS3.SSS0.Px1.p1.12.m12.1.1" xref="S3.SS3.SSS0.Px1.p1.12.m12.1.1.cmml"><mtext id="S3.SS3.SSS0.Px1.p1.12.m12.1.1.3" xref="S3.SS3.SSS0.Px1.p1.12.m12.1.1.3a.cmml">concat</mtext><mo lspace="0em" rspace="0em" id="S3.SS3.SSS0.Px1.p1.12.m12.1.1.2" xref="S3.SS3.SSS0.Px1.p1.12.m12.1.1.2.cmml">​</mo><mrow id="S3.SS3.SSS0.Px1.p1.12.m12.1.1.1.1" xref="S3.SS3.SSS0.Px1.p1.12.m12.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS3.SSS0.Px1.p1.12.m12.1.1.1.1.2" xref="S3.SS3.SSS0.Px1.p1.12.m12.1.1.1.1.1.cmml">(</mo><msub id="S3.SS3.SSS0.Px1.p1.12.m12.1.1.1.1.1" xref="S3.SS3.SSS0.Px1.p1.12.m12.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.SSS0.Px1.p1.12.m12.1.1.1.1.1.2" xref="S3.SS3.SSS0.Px1.p1.12.m12.1.1.1.1.1.2.cmml">𝒞</mi><mi id="S3.SS3.SSS0.Px1.p1.12.m12.1.1.1.1.1.3" xref="S3.SS3.SSS0.Px1.p1.12.m12.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S3.SS3.SSS0.Px1.p1.12.m12.1.1.1.1.3" xref="S3.SS3.SSS0.Px1.p1.12.m12.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p1.12.m12.1b"><apply id="S3.SS3.SSS0.Px1.p1.12.m12.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.12.m12.1.1"><times id="S3.SS3.SSS0.Px1.p1.12.m12.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p1.12.m12.1.1.2"></times><ci id="S3.SS3.SSS0.Px1.p1.12.m12.1.1.3a.cmml" xref="S3.SS3.SSS0.Px1.p1.12.m12.1.1.3"><mtext id="S3.SS3.SSS0.Px1.p1.12.m12.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p1.12.m12.1.1.3">concat</mtext></ci><apply id="S3.SS3.SSS0.Px1.p1.12.m12.1.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.12.m12.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p1.12.m12.1.1.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.12.m12.1.1.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px1.p1.12.m12.1.1.1.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p1.12.m12.1.1.1.1.1.2">𝒞</ci><ci id="S3.SS3.SSS0.Px1.p1.12.m12.1.1.1.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p1.12.m12.1.1.1.1.1.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p1.12.m12.1c">\text{concat}(\mathcal{C}_{i})</annotation></semantics></math>.</p>
</div>
</section>
<section id="S3.SS3.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Citation precision.</h4>

<div id="S3.SS3.SSS0.Px2.p1" class="ltx_para">
<p id="S3.SS3.SSS0.Px2.p1.1" class="ltx_p">Our citation precision evaluation detects citations that are irrelevant,
but it does not require citing a minimal set.
We follow this design because human writing often cites redundant sources to enhance credibility;
human readers may also appreciate multiple citations, especially when it pertains to critical claims such as medical advice.</p>
</div>
<div id="S3.SS3.SSS0.Px2.p2" class="ltx_para">
<p id="S3.SS3.SSS0.Px2.p2.6" class="ltx_p">We calculate the citation precision for <span id="S3.SS3.SSS0.Px2.p2.6.1" class="ltx_text ltx_font_italic">each citation</span> (0 or 1) and
average over all citations in the response. We first define if a citation is “irrelevant”.
Intuitively, a citation <math id="S3.SS3.SSS0.Px2.p2.1.m1.2" class="ltx_Math" alttext="c_{i,j}" display="inline"><semantics id="S3.SS3.SSS0.Px2.p2.1.m1.2a"><msub id="S3.SS3.SSS0.Px2.p2.1.m1.2.3" xref="S3.SS3.SSS0.Px2.p2.1.m1.2.3.cmml"><mi id="S3.SS3.SSS0.Px2.p2.1.m1.2.3.2" xref="S3.SS3.SSS0.Px2.p2.1.m1.2.3.2.cmml">c</mi><mrow id="S3.SS3.SSS0.Px2.p2.1.m1.2.2.2.4" xref="S3.SS3.SSS0.Px2.p2.1.m1.2.2.2.3.cmml"><mi id="S3.SS3.SSS0.Px2.p2.1.m1.1.1.1.1" xref="S3.SS3.SSS0.Px2.p2.1.m1.1.1.1.1.cmml">i</mi><mo id="S3.SS3.SSS0.Px2.p2.1.m1.2.2.2.4.1" xref="S3.SS3.SSS0.Px2.p2.1.m1.2.2.2.3.cmml">,</mo><mi id="S3.SS3.SSS0.Px2.p2.1.m1.2.2.2.2" xref="S3.SS3.SSS0.Px2.p2.1.m1.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p2.1.m1.2b"><apply id="S3.SS3.SSS0.Px2.p2.1.m1.2.3.cmml" xref="S3.SS3.SSS0.Px2.p2.1.m1.2.3"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p2.1.m1.2.3.1.cmml" xref="S3.SS3.SSS0.Px2.p2.1.m1.2.3">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p2.1.m1.2.3.2.cmml" xref="S3.SS3.SSS0.Px2.p2.1.m1.2.3.2">𝑐</ci><list id="S3.SS3.SSS0.Px2.p2.1.m1.2.2.2.3.cmml" xref="S3.SS3.SSS0.Px2.p2.1.m1.2.2.2.4"><ci id="S3.SS3.SSS0.Px2.p2.1.m1.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.1.m1.1.1.1.1">𝑖</ci><ci id="S3.SS3.SSS0.Px2.p2.1.m1.2.2.2.2.cmml" xref="S3.SS3.SSS0.Px2.p2.1.m1.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p2.1.m1.2c">c_{i,j}</annotation></semantics></math> is “irrelevant” if (a) <math id="S3.SS3.SSS0.Px2.p2.2.m2.2" class="ltx_Math" alttext="c_{i,j}" display="inline"><semantics id="S3.SS3.SSS0.Px2.p2.2.m2.2a"><msub id="S3.SS3.SSS0.Px2.p2.2.m2.2.3" xref="S3.SS3.SSS0.Px2.p2.2.m2.2.3.cmml"><mi id="S3.SS3.SSS0.Px2.p2.2.m2.2.3.2" xref="S3.SS3.SSS0.Px2.p2.2.m2.2.3.2.cmml">c</mi><mrow id="S3.SS3.SSS0.Px2.p2.2.m2.2.2.2.4" xref="S3.SS3.SSS0.Px2.p2.2.m2.2.2.2.3.cmml"><mi id="S3.SS3.SSS0.Px2.p2.2.m2.1.1.1.1" xref="S3.SS3.SSS0.Px2.p2.2.m2.1.1.1.1.cmml">i</mi><mo id="S3.SS3.SSS0.Px2.p2.2.m2.2.2.2.4.1" xref="S3.SS3.SSS0.Px2.p2.2.m2.2.2.2.3.cmml">,</mo><mi id="S3.SS3.SSS0.Px2.p2.2.m2.2.2.2.2" xref="S3.SS3.SSS0.Px2.p2.2.m2.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p2.2.m2.2b"><apply id="S3.SS3.SSS0.Px2.p2.2.m2.2.3.cmml" xref="S3.SS3.SSS0.Px2.p2.2.m2.2.3"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p2.2.m2.2.3.1.cmml" xref="S3.SS3.SSS0.Px2.p2.2.m2.2.3">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p2.2.m2.2.3.2.cmml" xref="S3.SS3.SSS0.Px2.p2.2.m2.2.3.2">𝑐</ci><list id="S3.SS3.SSS0.Px2.p2.2.m2.2.2.2.3.cmml" xref="S3.SS3.SSS0.Px2.p2.2.m2.2.2.2.4"><ci id="S3.SS3.SSS0.Px2.p2.2.m2.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.2.m2.1.1.1.1">𝑖</ci><ci id="S3.SS3.SSS0.Px2.p2.2.m2.2.2.2.2.cmml" xref="S3.SS3.SSS0.Px2.p2.2.m2.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p2.2.m2.2c">c_{i,j}</annotation></semantics></math> itself cannot support <math id="S3.SS3.SSS0.Px2.p2.3.m3.1" class="ltx_Math" alttext="s_{i}" display="inline"><semantics id="S3.SS3.SSS0.Px2.p2.3.m3.1a"><msub id="S3.SS3.SSS0.Px2.p2.3.m3.1.1" xref="S3.SS3.SSS0.Px2.p2.3.m3.1.1.cmml"><mi id="S3.SS3.SSS0.Px2.p2.3.m3.1.1.2" xref="S3.SS3.SSS0.Px2.p2.3.m3.1.1.2.cmml">s</mi><mi id="S3.SS3.SSS0.Px2.p2.3.m3.1.1.3" xref="S3.SS3.SSS0.Px2.p2.3.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p2.3.m3.1b"><apply id="S3.SS3.SSS0.Px2.p2.3.m3.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p2.3.m3.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.3.m3.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p2.3.m3.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p2.3.m3.1.1.2">𝑠</ci><ci id="S3.SS3.SSS0.Px2.p2.3.m3.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p2.3.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p2.3.m3.1c">s_{i}</annotation></semantics></math> and (b) removing <math id="S3.SS3.SSS0.Px2.p2.4.m4.2" class="ltx_Math" alttext="c_{i,j}" display="inline"><semantics id="S3.SS3.SSS0.Px2.p2.4.m4.2a"><msub id="S3.SS3.SSS0.Px2.p2.4.m4.2.3" xref="S3.SS3.SSS0.Px2.p2.4.m4.2.3.cmml"><mi id="S3.SS3.SSS0.Px2.p2.4.m4.2.3.2" xref="S3.SS3.SSS0.Px2.p2.4.m4.2.3.2.cmml">c</mi><mrow id="S3.SS3.SSS0.Px2.p2.4.m4.2.2.2.4" xref="S3.SS3.SSS0.Px2.p2.4.m4.2.2.2.3.cmml"><mi id="S3.SS3.SSS0.Px2.p2.4.m4.1.1.1.1" xref="S3.SS3.SSS0.Px2.p2.4.m4.1.1.1.1.cmml">i</mi><mo id="S3.SS3.SSS0.Px2.p2.4.m4.2.2.2.4.1" xref="S3.SS3.SSS0.Px2.p2.4.m4.2.2.2.3.cmml">,</mo><mi id="S3.SS3.SSS0.Px2.p2.4.m4.2.2.2.2" xref="S3.SS3.SSS0.Px2.p2.4.m4.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p2.4.m4.2b"><apply id="S3.SS3.SSS0.Px2.p2.4.m4.2.3.cmml" xref="S3.SS3.SSS0.Px2.p2.4.m4.2.3"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p2.4.m4.2.3.1.cmml" xref="S3.SS3.SSS0.Px2.p2.4.m4.2.3">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p2.4.m4.2.3.2.cmml" xref="S3.SS3.SSS0.Px2.p2.4.m4.2.3.2">𝑐</ci><list id="S3.SS3.SSS0.Px2.p2.4.m4.2.2.2.3.cmml" xref="S3.SS3.SSS0.Px2.p2.4.m4.2.2.2.4"><ci id="S3.SS3.SSS0.Px2.p2.4.m4.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.4.m4.1.1.1.1">𝑖</ci><ci id="S3.SS3.SSS0.Px2.p2.4.m4.2.2.2.2.cmml" xref="S3.SS3.SSS0.Px2.p2.4.m4.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p2.4.m4.2c">c_{i,j}</annotation></semantics></math> does not affect the rest of the citations to support <math id="S3.SS3.SSS0.Px2.p2.5.m5.1" class="ltx_Math" alttext="s_{i}" display="inline"><semantics id="S3.SS3.SSS0.Px2.p2.5.m5.1a"><msub id="S3.SS3.SSS0.Px2.p2.5.m5.1.1" xref="S3.SS3.SSS0.Px2.p2.5.m5.1.1.cmml"><mi id="S3.SS3.SSS0.Px2.p2.5.m5.1.1.2" xref="S3.SS3.SSS0.Px2.p2.5.m5.1.1.2.cmml">s</mi><mi id="S3.SS3.SSS0.Px2.p2.5.m5.1.1.3" xref="S3.SS3.SSS0.Px2.p2.5.m5.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p2.5.m5.1b"><apply id="S3.SS3.SSS0.Px2.p2.5.m5.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p2.5.m5.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.5.m5.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p2.5.m5.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p2.5.m5.1.1.2">𝑠</ci><ci id="S3.SS3.SSS0.Px2.p2.5.m5.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p2.5.m5.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p2.5.m5.1c">s_{i}</annotation></semantics></math>. Formally, <math id="S3.SS3.SSS0.Px2.p2.6.m6.2" class="ltx_Math" alttext="c_{i,j}" display="inline"><semantics id="S3.SS3.SSS0.Px2.p2.6.m6.2a"><msub id="S3.SS3.SSS0.Px2.p2.6.m6.2.3" xref="S3.SS3.SSS0.Px2.p2.6.m6.2.3.cmml"><mi id="S3.SS3.SSS0.Px2.p2.6.m6.2.3.2" xref="S3.SS3.SSS0.Px2.p2.6.m6.2.3.2.cmml">c</mi><mrow id="S3.SS3.SSS0.Px2.p2.6.m6.2.2.2.4" xref="S3.SS3.SSS0.Px2.p2.6.m6.2.2.2.3.cmml"><mi id="S3.SS3.SSS0.Px2.p2.6.m6.1.1.1.1" xref="S3.SS3.SSS0.Px2.p2.6.m6.1.1.1.1.cmml">i</mi><mo id="S3.SS3.SSS0.Px2.p2.6.m6.2.2.2.4.1" xref="S3.SS3.SSS0.Px2.p2.6.m6.2.2.2.3.cmml">,</mo><mi id="S3.SS3.SSS0.Px2.p2.6.m6.2.2.2.2" xref="S3.SS3.SSS0.Px2.p2.6.m6.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p2.6.m6.2b"><apply id="S3.SS3.SSS0.Px2.p2.6.m6.2.3.cmml" xref="S3.SS3.SSS0.Px2.p2.6.m6.2.3"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p2.6.m6.2.3.1.cmml" xref="S3.SS3.SSS0.Px2.p2.6.m6.2.3">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p2.6.m6.2.3.2.cmml" xref="S3.SS3.SSS0.Px2.p2.6.m6.2.3.2">𝑐</ci><list id="S3.SS3.SSS0.Px2.p2.6.m6.2.2.2.3.cmml" xref="S3.SS3.SSS0.Px2.p2.6.m6.2.2.2.4"><ci id="S3.SS3.SSS0.Px2.p2.6.m6.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.6.m6.1.1.1.1">𝑖</ci><ci id="S3.SS3.SSS0.Px2.p2.6.m6.2.2.2.2.cmml" xref="S3.SS3.SSS0.Px2.p2.6.m6.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p2.6.m6.2c">c_{i,j}</annotation></semantics></math> is “irrelevant” if and only if</p>
<ol id="S3.I2" class="ltx_enumerate">
<li id="S3.I2.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">(a)</span> 
<div id="S3.I2.i1.p1" class="ltx_para">
<p id="S3.I2.i1.p1.1" class="ltx_p"><math id="S3.I2.i1.p1.1.m1.4" class="ltx_Math" alttext="\phi(c_{i,j},s_{i})=0" display="inline"><semantics id="S3.I2.i1.p1.1.m1.4a"><mrow id="S3.I2.i1.p1.1.m1.4.4" xref="S3.I2.i1.p1.1.m1.4.4.cmml"><mrow id="S3.I2.i1.p1.1.m1.4.4.2" xref="S3.I2.i1.p1.1.m1.4.4.2.cmml"><mi id="S3.I2.i1.p1.1.m1.4.4.2.4" xref="S3.I2.i1.p1.1.m1.4.4.2.4.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S3.I2.i1.p1.1.m1.4.4.2.3" xref="S3.I2.i1.p1.1.m1.4.4.2.3.cmml">​</mo><mrow id="S3.I2.i1.p1.1.m1.4.4.2.2.2" xref="S3.I2.i1.p1.1.m1.4.4.2.2.3.cmml"><mo stretchy="false" id="S3.I2.i1.p1.1.m1.4.4.2.2.2.3" xref="S3.I2.i1.p1.1.m1.4.4.2.2.3.cmml">(</mo><msub id="S3.I2.i1.p1.1.m1.3.3.1.1.1.1" xref="S3.I2.i1.p1.1.m1.3.3.1.1.1.1.cmml"><mi id="S3.I2.i1.p1.1.m1.3.3.1.1.1.1.2" xref="S3.I2.i1.p1.1.m1.3.3.1.1.1.1.2.cmml">c</mi><mrow id="S3.I2.i1.p1.1.m1.2.2.2.4" xref="S3.I2.i1.p1.1.m1.2.2.2.3.cmml"><mi id="S3.I2.i1.p1.1.m1.1.1.1.1" xref="S3.I2.i1.p1.1.m1.1.1.1.1.cmml">i</mi><mo id="S3.I2.i1.p1.1.m1.2.2.2.4.1" xref="S3.I2.i1.p1.1.m1.2.2.2.3.cmml">,</mo><mi id="S3.I2.i1.p1.1.m1.2.2.2.2" xref="S3.I2.i1.p1.1.m1.2.2.2.2.cmml">j</mi></mrow></msub><mo id="S3.I2.i1.p1.1.m1.4.4.2.2.2.4" xref="S3.I2.i1.p1.1.m1.4.4.2.2.3.cmml">,</mo><msub id="S3.I2.i1.p1.1.m1.4.4.2.2.2.2" xref="S3.I2.i1.p1.1.m1.4.4.2.2.2.2.cmml"><mi id="S3.I2.i1.p1.1.m1.4.4.2.2.2.2.2" xref="S3.I2.i1.p1.1.m1.4.4.2.2.2.2.2.cmml">s</mi><mi id="S3.I2.i1.p1.1.m1.4.4.2.2.2.2.3" xref="S3.I2.i1.p1.1.m1.4.4.2.2.2.2.3.cmml">i</mi></msub><mo stretchy="false" id="S3.I2.i1.p1.1.m1.4.4.2.2.2.5" xref="S3.I2.i1.p1.1.m1.4.4.2.2.3.cmml">)</mo></mrow></mrow><mo id="S3.I2.i1.p1.1.m1.4.4.3" xref="S3.I2.i1.p1.1.m1.4.4.3.cmml">=</mo><mn id="S3.I2.i1.p1.1.m1.4.4.4" xref="S3.I2.i1.p1.1.m1.4.4.4.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.I2.i1.p1.1.m1.4b"><apply id="S3.I2.i1.p1.1.m1.4.4.cmml" xref="S3.I2.i1.p1.1.m1.4.4"><eq id="S3.I2.i1.p1.1.m1.4.4.3.cmml" xref="S3.I2.i1.p1.1.m1.4.4.3"></eq><apply id="S3.I2.i1.p1.1.m1.4.4.2.cmml" xref="S3.I2.i1.p1.1.m1.4.4.2"><times id="S3.I2.i1.p1.1.m1.4.4.2.3.cmml" xref="S3.I2.i1.p1.1.m1.4.4.2.3"></times><ci id="S3.I2.i1.p1.1.m1.4.4.2.4.cmml" xref="S3.I2.i1.p1.1.m1.4.4.2.4">italic-ϕ</ci><interval closure="open" id="S3.I2.i1.p1.1.m1.4.4.2.2.3.cmml" xref="S3.I2.i1.p1.1.m1.4.4.2.2.2"><apply id="S3.I2.i1.p1.1.m1.3.3.1.1.1.1.cmml" xref="S3.I2.i1.p1.1.m1.3.3.1.1.1.1"><csymbol cd="ambiguous" id="S3.I2.i1.p1.1.m1.3.3.1.1.1.1.1.cmml" xref="S3.I2.i1.p1.1.m1.3.3.1.1.1.1">subscript</csymbol><ci id="S3.I2.i1.p1.1.m1.3.3.1.1.1.1.2.cmml" xref="S3.I2.i1.p1.1.m1.3.3.1.1.1.1.2">𝑐</ci><list id="S3.I2.i1.p1.1.m1.2.2.2.3.cmml" xref="S3.I2.i1.p1.1.m1.2.2.2.4"><ci id="S3.I2.i1.p1.1.m1.1.1.1.1.cmml" xref="S3.I2.i1.p1.1.m1.1.1.1.1">𝑖</ci><ci id="S3.I2.i1.p1.1.m1.2.2.2.2.cmml" xref="S3.I2.i1.p1.1.m1.2.2.2.2">𝑗</ci></list></apply><apply id="S3.I2.i1.p1.1.m1.4.4.2.2.2.2.cmml" xref="S3.I2.i1.p1.1.m1.4.4.2.2.2.2"><csymbol cd="ambiguous" id="S3.I2.i1.p1.1.m1.4.4.2.2.2.2.1.cmml" xref="S3.I2.i1.p1.1.m1.4.4.2.2.2.2">subscript</csymbol><ci id="S3.I2.i1.p1.1.m1.4.4.2.2.2.2.2.cmml" xref="S3.I2.i1.p1.1.m1.4.4.2.2.2.2.2">𝑠</ci><ci id="S3.I2.i1.p1.1.m1.4.4.2.2.2.2.3.cmml" xref="S3.I2.i1.p1.1.m1.4.4.2.2.2.2.3">𝑖</ci></apply></interval></apply><cn type="integer" id="S3.I2.i1.p1.1.m1.4.4.4.cmml" xref="S3.I2.i1.p1.1.m1.4.4.4">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I2.i1.p1.1.m1.4c">\phi(c_{i,j},s_{i})=0</annotation></semantics></math>, AND</p>
</div>
</li>
<li id="S3.I2.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">(b)</span> 
<div id="S3.I2.i2.p1" class="ltx_para">
<p id="S3.I2.i2.p1.1" class="ltx_p"><math id="S3.I2.i2.p1.1.m1.4" class="ltx_Math" alttext="\phi(\text{concat}(\mathcal{C}_{i}\setminus\{c_{i,j}\}),s_{i})=1" display="inline"><semantics id="S3.I2.i2.p1.1.m1.4a"><mrow id="S3.I2.i2.p1.1.m1.4.4" xref="S3.I2.i2.p1.1.m1.4.4.cmml"><mrow id="S3.I2.i2.p1.1.m1.4.4.2" xref="S3.I2.i2.p1.1.m1.4.4.2.cmml"><mi id="S3.I2.i2.p1.1.m1.4.4.2.4" xref="S3.I2.i2.p1.1.m1.4.4.2.4.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S3.I2.i2.p1.1.m1.4.4.2.3" xref="S3.I2.i2.p1.1.m1.4.4.2.3.cmml">​</mo><mrow id="S3.I2.i2.p1.1.m1.4.4.2.2.2" xref="S3.I2.i2.p1.1.m1.4.4.2.2.3.cmml"><mo stretchy="false" id="S3.I2.i2.p1.1.m1.4.4.2.2.2.3" xref="S3.I2.i2.p1.1.m1.4.4.2.2.3.cmml">(</mo><mrow id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.cmml"><mtext id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.3" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.3a.cmml">concat</mtext><mo lspace="0em" rspace="0em" id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.2" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.2.cmml">​</mo><mrow id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.2" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.cmml"><msub id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.3" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.3.2" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.3.2.cmml">𝒞</mi><mi id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.3.3" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.3.3.cmml">i</mi></msub><mo id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.2" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.2.cmml">∖</mo><mrow id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.1.1" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.1.1.2" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.1.2.cmml">{</mo><msub id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.1.1.1" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.1.1.1.2" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.1.1.1.2.cmml">c</mi><mrow id="S3.I2.i2.p1.1.m1.2.2.2.4" xref="S3.I2.i2.p1.1.m1.2.2.2.3.cmml"><mi id="S3.I2.i2.p1.1.m1.1.1.1.1" xref="S3.I2.i2.p1.1.m1.1.1.1.1.cmml">i</mi><mo id="S3.I2.i2.p1.1.m1.2.2.2.4.1" xref="S3.I2.i2.p1.1.m1.2.2.2.3.cmml">,</mo><mi id="S3.I2.i2.p1.1.m1.2.2.2.2" xref="S3.I2.i2.p1.1.m1.2.2.2.2.cmml">j</mi></mrow></msub><mo stretchy="false" id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.1.1.3" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.1.2.cmml">}</mo></mrow></mrow><mo stretchy="false" id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.3" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.I2.i2.p1.1.m1.4.4.2.2.2.4" xref="S3.I2.i2.p1.1.m1.4.4.2.2.3.cmml">,</mo><msub id="S3.I2.i2.p1.1.m1.4.4.2.2.2.2" xref="S3.I2.i2.p1.1.m1.4.4.2.2.2.2.cmml"><mi id="S3.I2.i2.p1.1.m1.4.4.2.2.2.2.2" xref="S3.I2.i2.p1.1.m1.4.4.2.2.2.2.2.cmml">s</mi><mi id="S3.I2.i2.p1.1.m1.4.4.2.2.2.2.3" xref="S3.I2.i2.p1.1.m1.4.4.2.2.2.2.3.cmml">i</mi></msub><mo stretchy="false" id="S3.I2.i2.p1.1.m1.4.4.2.2.2.5" xref="S3.I2.i2.p1.1.m1.4.4.2.2.3.cmml">)</mo></mrow></mrow><mo id="S3.I2.i2.p1.1.m1.4.4.3" xref="S3.I2.i2.p1.1.m1.4.4.3.cmml">=</mo><mn id="S3.I2.i2.p1.1.m1.4.4.4" xref="S3.I2.i2.p1.1.m1.4.4.4.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.I2.i2.p1.1.m1.4b"><apply id="S3.I2.i2.p1.1.m1.4.4.cmml" xref="S3.I2.i2.p1.1.m1.4.4"><eq id="S3.I2.i2.p1.1.m1.4.4.3.cmml" xref="S3.I2.i2.p1.1.m1.4.4.3"></eq><apply id="S3.I2.i2.p1.1.m1.4.4.2.cmml" xref="S3.I2.i2.p1.1.m1.4.4.2"><times id="S3.I2.i2.p1.1.m1.4.4.2.3.cmml" xref="S3.I2.i2.p1.1.m1.4.4.2.3"></times><ci id="S3.I2.i2.p1.1.m1.4.4.2.4.cmml" xref="S3.I2.i2.p1.1.m1.4.4.2.4">italic-ϕ</ci><interval closure="open" id="S3.I2.i2.p1.1.m1.4.4.2.2.3.cmml" xref="S3.I2.i2.p1.1.m1.4.4.2.2.2"><apply id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.cmml" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1"><times id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.2.cmml" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.2"></times><ci id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.3a.cmml" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.3"><mtext id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.3.cmml" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.3">concat</mtext></ci><apply id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.cmml" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1"><setdiff id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.2.cmml" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.2"></setdiff><apply id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.3.cmml" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.3.1.cmml" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.3.2.cmml" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.3.2">𝒞</ci><ci id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.3.3.cmml" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.3.3">𝑖</ci></apply><set id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.1.2.cmml" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.1.1"><apply id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.I2.i2.p1.1.m1.3.3.1.1.1.1.1.1.1.1.1.1.2">𝑐</ci><list id="S3.I2.i2.p1.1.m1.2.2.2.3.cmml" xref="S3.I2.i2.p1.1.m1.2.2.2.4"><ci id="S3.I2.i2.p1.1.m1.1.1.1.1.cmml" xref="S3.I2.i2.p1.1.m1.1.1.1.1">𝑖</ci><ci id="S3.I2.i2.p1.1.m1.2.2.2.2.cmml" xref="S3.I2.i2.p1.1.m1.2.2.2.2">𝑗</ci></list></apply></set></apply></apply><apply id="S3.I2.i2.p1.1.m1.4.4.2.2.2.2.cmml" xref="S3.I2.i2.p1.1.m1.4.4.2.2.2.2"><csymbol cd="ambiguous" id="S3.I2.i2.p1.1.m1.4.4.2.2.2.2.1.cmml" xref="S3.I2.i2.p1.1.m1.4.4.2.2.2.2">subscript</csymbol><ci id="S3.I2.i2.p1.1.m1.4.4.2.2.2.2.2.cmml" xref="S3.I2.i2.p1.1.m1.4.4.2.2.2.2.2">𝑠</ci><ci id="S3.I2.i2.p1.1.m1.4.4.2.2.2.2.3.cmml" xref="S3.I2.i2.p1.1.m1.4.4.2.2.2.2.3">𝑖</ci></apply></interval></apply><cn type="integer" id="S3.I2.i2.p1.1.m1.4.4.4.cmml" xref="S3.I2.i2.p1.1.m1.4.4.4">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I2.i2.p1.1.m1.4c">\phi(\text{concat}(\mathcal{C}_{i}\setminus\{c_{i,j}\}),s_{i})=1</annotation></semantics></math>.</p>
</div>
</li>
</ol>
<p id="S3.SS3.SSS0.Px2.p2.19" class="ltx_p"><math id="S3.SS3.SSS0.Px2.p2.7.m1.2" class="ltx_Math" alttext="c_{i,j}" display="inline"><semantics id="S3.SS3.SSS0.Px2.p2.7.m1.2a"><msub id="S3.SS3.SSS0.Px2.p2.7.m1.2.3" xref="S3.SS3.SSS0.Px2.p2.7.m1.2.3.cmml"><mi id="S3.SS3.SSS0.Px2.p2.7.m1.2.3.2" xref="S3.SS3.SSS0.Px2.p2.7.m1.2.3.2.cmml">c</mi><mrow id="S3.SS3.SSS0.Px2.p2.7.m1.2.2.2.4" xref="S3.SS3.SSS0.Px2.p2.7.m1.2.2.2.3.cmml"><mi id="S3.SS3.SSS0.Px2.p2.7.m1.1.1.1.1" xref="S3.SS3.SSS0.Px2.p2.7.m1.1.1.1.1.cmml">i</mi><mo id="S3.SS3.SSS0.Px2.p2.7.m1.2.2.2.4.1" xref="S3.SS3.SSS0.Px2.p2.7.m1.2.2.2.3.cmml">,</mo><mi id="S3.SS3.SSS0.Px2.p2.7.m1.2.2.2.2" xref="S3.SS3.SSS0.Px2.p2.7.m1.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p2.7.m1.2b"><apply id="S3.SS3.SSS0.Px2.p2.7.m1.2.3.cmml" xref="S3.SS3.SSS0.Px2.p2.7.m1.2.3"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p2.7.m1.2.3.1.cmml" xref="S3.SS3.SSS0.Px2.p2.7.m1.2.3">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p2.7.m1.2.3.2.cmml" xref="S3.SS3.SSS0.Px2.p2.7.m1.2.3.2">𝑐</ci><list id="S3.SS3.SSS0.Px2.p2.7.m1.2.2.2.3.cmml" xref="S3.SS3.SSS0.Px2.p2.7.m1.2.2.2.4"><ci id="S3.SS3.SSS0.Px2.p2.7.m1.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.7.m1.1.1.1.1">𝑖</ci><ci id="S3.SS3.SSS0.Px2.p2.7.m1.2.2.2.2.cmml" xref="S3.SS3.SSS0.Px2.p2.7.m1.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p2.7.m1.2c">c_{i,j}</annotation></semantics></math> has a precision of 1 if <math id="S3.SS3.SSS0.Px2.p2.8.m2.1" class="ltx_Math" alttext="s_{i}" display="inline"><semantics id="S3.SS3.SSS0.Px2.p2.8.m2.1a"><msub id="S3.SS3.SSS0.Px2.p2.8.m2.1.1" xref="S3.SS3.SSS0.Px2.p2.8.m2.1.1.cmml"><mi id="S3.SS3.SSS0.Px2.p2.8.m2.1.1.2" xref="S3.SS3.SSS0.Px2.p2.8.m2.1.1.2.cmml">s</mi><mi id="S3.SS3.SSS0.Px2.p2.8.m2.1.1.3" xref="S3.SS3.SSS0.Px2.p2.8.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p2.8.m2.1b"><apply id="S3.SS3.SSS0.Px2.p2.8.m2.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.8.m2.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p2.8.m2.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.8.m2.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p2.8.m2.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p2.8.m2.1.1.2">𝑠</ci><ci id="S3.SS3.SSS0.Px2.p2.8.m2.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p2.8.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p2.8.m2.1c">s_{i}</annotation></semantics></math> has recall<math id="S3.SS3.SSS0.Px2.p2.9.m3.1" class="ltx_Math" alttext="=" display="inline"><semantics id="S3.SS3.SSS0.Px2.p2.9.m3.1a"><mo id="S3.SS3.SSS0.Px2.p2.9.m3.1.1" xref="S3.SS3.SSS0.Px2.p2.9.m3.1.1.cmml">=</mo><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p2.9.m3.1b"><eq id="S3.SS3.SSS0.Px2.p2.9.m3.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.9.m3.1.1"></eq></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p2.9.m3.1c">=</annotation></semantics></math><math id="S3.SS3.SSS0.Px2.p2.10.m4.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S3.SS3.SSS0.Px2.p2.10.m4.1a"><mn id="S3.SS3.SSS0.Px2.p2.10.m4.1.1" xref="S3.SS3.SSS0.Px2.p2.10.m4.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p2.10.m4.1b"><cn type="integer" id="S3.SS3.SSS0.Px2.p2.10.m4.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.10.m4.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p2.10.m4.1c">1</annotation></semantics></math> and <math id="S3.SS3.SSS0.Px2.p2.11.m5.2" class="ltx_Math" alttext="c_{i,j}" display="inline"><semantics id="S3.SS3.SSS0.Px2.p2.11.m5.2a"><msub id="S3.SS3.SSS0.Px2.p2.11.m5.2.3" xref="S3.SS3.SSS0.Px2.p2.11.m5.2.3.cmml"><mi id="S3.SS3.SSS0.Px2.p2.11.m5.2.3.2" xref="S3.SS3.SSS0.Px2.p2.11.m5.2.3.2.cmml">c</mi><mrow id="S3.SS3.SSS0.Px2.p2.11.m5.2.2.2.4" xref="S3.SS3.SSS0.Px2.p2.11.m5.2.2.2.3.cmml"><mi id="S3.SS3.SSS0.Px2.p2.11.m5.1.1.1.1" xref="S3.SS3.SSS0.Px2.p2.11.m5.1.1.1.1.cmml">i</mi><mo id="S3.SS3.SSS0.Px2.p2.11.m5.2.2.2.4.1" xref="S3.SS3.SSS0.Px2.p2.11.m5.2.2.2.3.cmml">,</mo><mi id="S3.SS3.SSS0.Px2.p2.11.m5.2.2.2.2" xref="S3.SS3.SSS0.Px2.p2.11.m5.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p2.11.m5.2b"><apply id="S3.SS3.SSS0.Px2.p2.11.m5.2.3.cmml" xref="S3.SS3.SSS0.Px2.p2.11.m5.2.3"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p2.11.m5.2.3.1.cmml" xref="S3.SS3.SSS0.Px2.p2.11.m5.2.3">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p2.11.m5.2.3.2.cmml" xref="S3.SS3.SSS0.Px2.p2.11.m5.2.3.2">𝑐</ci><list id="S3.SS3.SSS0.Px2.p2.11.m5.2.2.2.3.cmml" xref="S3.SS3.SSS0.Px2.p2.11.m5.2.2.2.4"><ci id="S3.SS3.SSS0.Px2.p2.11.m5.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.11.m5.1.1.1.1">𝑖</ci><ci id="S3.SS3.SSS0.Px2.p2.11.m5.2.2.2.2.cmml" xref="S3.SS3.SSS0.Px2.p2.11.m5.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p2.11.m5.2c">c_{i,j}</annotation></semantics></math> is not irrelevant.
For example (Figure <a href="#S3.F3" title="Figure 3 ‣ 3.3 Citation Quality ‣ 3 Automatic Evaluation ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>), when <math id="S3.SS3.SSS0.Px2.p2.12.m6.1" class="ltx_Math" alttext="s_{3}" display="inline"><semantics id="S3.SS3.SSS0.Px2.p2.12.m6.1a"><msub id="S3.SS3.SSS0.Px2.p2.12.m6.1.1" xref="S3.SS3.SSS0.Px2.p2.12.m6.1.1.cmml"><mi id="S3.SS3.SSS0.Px2.p2.12.m6.1.1.2" xref="S3.SS3.SSS0.Px2.p2.12.m6.1.1.2.cmml">s</mi><mn id="S3.SS3.SSS0.Px2.p2.12.m6.1.1.3" xref="S3.SS3.SSS0.Px2.p2.12.m6.1.1.3.cmml">3</mn></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p2.12.m6.1b"><apply id="S3.SS3.SSS0.Px2.p2.12.m6.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.12.m6.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p2.12.m6.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.12.m6.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p2.12.m6.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p2.12.m6.1.1.2">𝑠</ci><cn type="integer" id="S3.SS3.SSS0.Px2.p2.12.m6.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p2.12.m6.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p2.12.m6.1c">s_{3}</annotation></semantics></math> cites three references <span id="S3.SS3.SSS0.Px2.p2.19.1" class="ltx_text ltx_font_typewriter">[2][4][5]</span> and recall<math id="S3.SS3.SSS0.Px2.p2.13.m7.1" class="ltx_Math" alttext="=" display="inline"><semantics id="S3.SS3.SSS0.Px2.p2.13.m7.1a"><mo id="S3.SS3.SSS0.Px2.p2.13.m7.1.1" xref="S3.SS3.SSS0.Px2.p2.13.m7.1.1.cmml">=</mo><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p2.13.m7.1b"><eq id="S3.SS3.SSS0.Px2.p2.13.m7.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.13.m7.1.1"></eq></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p2.13.m7.1c">=</annotation></semantics></math><math id="S3.SS3.SSS0.Px2.p2.14.m8.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S3.SS3.SSS0.Px2.p2.14.m8.1a"><mn id="S3.SS3.SSS0.Px2.p2.14.m8.1.1" xref="S3.SS3.SSS0.Px2.p2.14.m8.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p2.14.m8.1b"><cn type="integer" id="S3.SS3.SSS0.Px2.p2.14.m8.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.14.m8.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p2.14.m8.1c">1</annotation></semantics></math>,
<span id="S3.SS3.SSS0.Px2.p2.19.2" class="ltx_text ltx_font_typewriter">[2]</span> is “irrelevant” if <math id="S3.SS3.SSS0.Px2.p2.15.m9.2" class="ltx_Math" alttext="\phi(\texttt{[2]},s_{3})=0" display="inline"><semantics id="S3.SS3.SSS0.Px2.p2.15.m9.2a"><mrow id="S3.SS3.SSS0.Px2.p2.15.m9.2.2" xref="S3.SS3.SSS0.Px2.p2.15.m9.2.2.cmml"><mrow id="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1" xref="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.cmml"><mi id="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.3" xref="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.3.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.2" xref="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.2.cmml">​</mo><mrow id="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.1.1" xref="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.1.2.cmml"><mo stretchy="false" id="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.1.1.2" xref="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.1.2.cmml">(</mo><mtext class="ltx_mathvariant_monospace" id="S3.SS3.SSS0.Px2.p2.15.m9.1.1" xref="S3.SS3.SSS0.Px2.p2.15.m9.1.1a.cmml">[2]</mtext><mo id="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.1.1.3" xref="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.1.2.cmml">,</mo><msub id="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.1.1.1" xref="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.1.1.1.cmml"><mi id="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.1.1.1.2" xref="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.1.1.1.2.cmml">s</mi><mn id="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.1.1.1.3" xref="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.1.1.1.3.cmml">3</mn></msub><mo stretchy="false" id="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.1.1.4" xref="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.1.2.cmml">)</mo></mrow></mrow><mo id="S3.SS3.SSS0.Px2.p2.15.m9.2.2.2" xref="S3.SS3.SSS0.Px2.p2.15.m9.2.2.2.cmml">=</mo><mn id="S3.SS3.SSS0.Px2.p2.15.m9.2.2.3" xref="S3.SS3.SSS0.Px2.p2.15.m9.2.2.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p2.15.m9.2b"><apply id="S3.SS3.SSS0.Px2.p2.15.m9.2.2.cmml" xref="S3.SS3.SSS0.Px2.p2.15.m9.2.2"><eq id="S3.SS3.SSS0.Px2.p2.15.m9.2.2.2.cmml" xref="S3.SS3.SSS0.Px2.p2.15.m9.2.2.2"></eq><apply id="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.cmml" xref="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1"><times id="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.2.cmml" xref="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.2"></times><ci id="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.3.cmml" xref="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.3">italic-ϕ</ci><interval closure="open" id="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.1.1"><ci id="S3.SS3.SSS0.Px2.p2.15.m9.1.1a.cmml" xref="S3.SS3.SSS0.Px2.p2.15.m9.1.1"><mtext class="ltx_mathvariant_monospace" id="S3.SS3.SSS0.Px2.p2.15.m9.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.15.m9.1.1">[2]</mtext></ci><apply id="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.1.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.1.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.1.1.1.2">𝑠</ci><cn type="integer" id="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.1.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p2.15.m9.2.2.1.1.1.1.3">3</cn></apply></interval></apply><cn type="integer" id="S3.SS3.SSS0.Px2.p2.15.m9.2.2.3.cmml" xref="S3.SS3.SSS0.Px2.p2.15.m9.2.2.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p2.15.m9.2c">\phi(\texttt{[2]},s_{3})=0</annotation></semantics></math> and <math id="S3.SS3.SSS0.Px2.p2.16.m10.2" class="ltx_Math" alttext="\phi(\texttt{[4][5]},s_{3})=1" display="inline"><semantics id="S3.SS3.SSS0.Px2.p2.16.m10.2a"><mrow id="S3.SS3.SSS0.Px2.p2.16.m10.2.2" xref="S3.SS3.SSS0.Px2.p2.16.m10.2.2.cmml"><mrow id="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1" xref="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.cmml"><mi id="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.3" xref="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.3.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.2" xref="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.2.cmml">​</mo><mrow id="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.1.1" xref="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.1.2.cmml"><mo stretchy="false" id="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.1.1.2" xref="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.1.2.cmml">(</mo><mtext class="ltx_mathvariant_monospace" id="S3.SS3.SSS0.Px2.p2.16.m10.1.1" xref="S3.SS3.SSS0.Px2.p2.16.m10.1.1a.cmml">[4][5]</mtext><mo id="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.1.1.3" xref="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.1.2.cmml">,</mo><msub id="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.1.1.1" xref="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.1.1.1.cmml"><mi id="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.1.1.1.2" xref="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.1.1.1.2.cmml">s</mi><mn id="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.1.1.1.3" xref="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.1.1.1.3.cmml">3</mn></msub><mo stretchy="false" id="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.1.1.4" xref="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.1.2.cmml">)</mo></mrow></mrow><mo id="S3.SS3.SSS0.Px2.p2.16.m10.2.2.2" xref="S3.SS3.SSS0.Px2.p2.16.m10.2.2.2.cmml">=</mo><mn id="S3.SS3.SSS0.Px2.p2.16.m10.2.2.3" xref="S3.SS3.SSS0.Px2.p2.16.m10.2.2.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p2.16.m10.2b"><apply id="S3.SS3.SSS0.Px2.p2.16.m10.2.2.cmml" xref="S3.SS3.SSS0.Px2.p2.16.m10.2.2"><eq id="S3.SS3.SSS0.Px2.p2.16.m10.2.2.2.cmml" xref="S3.SS3.SSS0.Px2.p2.16.m10.2.2.2"></eq><apply id="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.cmml" xref="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1"><times id="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.2.cmml" xref="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.2"></times><ci id="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.3.cmml" xref="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.3">italic-ϕ</ci><interval closure="open" id="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.1.1"><ci id="S3.SS3.SSS0.Px2.p2.16.m10.1.1a.cmml" xref="S3.SS3.SSS0.Px2.p2.16.m10.1.1"><mtext class="ltx_mathvariant_monospace" id="S3.SS3.SSS0.Px2.p2.16.m10.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.16.m10.1.1">[4][5]</mtext></ci><apply id="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.1.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.1.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.1.1.1.2">𝑠</ci><cn type="integer" id="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.1.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p2.16.m10.2.2.1.1.1.1.3">3</cn></apply></interval></apply><cn type="integer" id="S3.SS3.SSS0.Px2.p2.16.m10.2.2.3.cmml" xref="S3.SS3.SSS0.Px2.p2.16.m10.2.2.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p2.16.m10.2c">\phi(\texttt{[4][5]},s_{3})=1</annotation></semantics></math>.
For condition (b) to work, we set recall<math id="S3.SS3.SSS0.Px2.p2.17.m11.1" class="ltx_Math" alttext="=" display="inline"><semantics id="S3.SS3.SSS0.Px2.p2.17.m11.1a"><mo id="S3.SS3.SSS0.Px2.p2.17.m11.1.1" xref="S3.SS3.SSS0.Px2.p2.17.m11.1.1.cmml">=</mo><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p2.17.m11.1b"><eq id="S3.SS3.SSS0.Px2.p2.17.m11.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.17.m11.1.1"></eq></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p2.17.m11.1c">=</annotation></semantics></math><math id="S3.SS3.SSS0.Px2.p2.18.m12.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S3.SS3.SSS0.Px2.p2.18.m12.1a"><mn id="S3.SS3.SSS0.Px2.p2.18.m12.1.1" xref="S3.SS3.SSS0.Px2.p2.18.m12.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p2.18.m12.1b"><cn type="integer" id="S3.SS3.SSS0.Px2.p2.18.m12.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.18.m12.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p2.18.m12.1c">1</annotation></semantics></math> as a prerequisite for precision<math id="S3.SS3.SSS0.Px2.p2.19.m13.1" class="ltx_Math" alttext="=1" display="inline"><semantics id="S3.SS3.SSS0.Px2.p2.19.m13.1a"><mrow id="S3.SS3.SSS0.Px2.p2.19.m13.1.1" xref="S3.SS3.SSS0.Px2.p2.19.m13.1.1.cmml"><mi id="S3.SS3.SSS0.Px2.p2.19.m13.1.1.2" xref="S3.SS3.SSS0.Px2.p2.19.m13.1.1.2.cmml"></mi><mo id="S3.SS3.SSS0.Px2.p2.19.m13.1.1.1" xref="S3.SS3.SSS0.Px2.p2.19.m13.1.1.1.cmml">=</mo><mn id="S3.SS3.SSS0.Px2.p2.19.m13.1.1.3" xref="S3.SS3.SSS0.Px2.p2.19.m13.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p2.19.m13.1b"><apply id="S3.SS3.SSS0.Px2.p2.19.m13.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.19.m13.1.1"><eq id="S3.SS3.SSS0.Px2.p2.19.m13.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.19.m13.1.1.1"></eq><csymbol cd="latexml" id="S3.SS3.SSS0.Px2.p2.19.m13.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p2.19.m13.1.1.2">absent</csymbol><cn type="integer" id="S3.SS3.SSS0.Px2.p2.19.m13.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p2.19.m13.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p2.19.m13.1c">=1</annotation></semantics></math>. Note that this algorithm overlooks the scenario when one citation partially supports the statement.
We discuss the details in §<a href="#A5" title="Appendix E Citation Recall Discussion ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">E</span></a>.</p>
</div>
</section>
</section>
<section id="S3.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.4 </span>ALCE is Robust to Shortcut Cases</h3>

<div id="S3.SS4.p1" class="ltx_para">
<p id="S3.SS4.p1.1" class="ltx_p">We showcase how the ALCE evaluation is robust to two possible shortcuts in §<a href="#A4" title="Appendix D ALCE Catches Shortcut Cases ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">D</span></a>:
(1) using the top-1 retrieved passage as the response and citing itself,
and (2) using the first two sentences of the top-1 passage.
Both cases have almost-perfect citation scores, but (1) has low fluency due to its unnaturally long length compared to human answers, and (2) has low correctness due to low coverage.</p>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Modeling</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.1" class="ltx_p">In this section,
we discuss three major modeling components for an ALCE system—retrieval, synthesis, and post-editing.</p>
</div>
<figure id="S4.T2" class="ltx_table">
<table id="S4.T2.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S4.T2.1.1.1" class="ltx_tr">
<td id="S4.T2.1.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="S4.T2.1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T2.1.1.1.1.1.1" class="ltx_p" style="width:407.6pt;"><span id="S4.T2.1.1.1.1.1.1.1" class="ltx_text ltx_font_typewriter">Instruction: Write an accurate, engaging, and concise answer for ...</span></span>
</span>
</td>
</tr>
<tr id="S4.T2.1.2.2" class="ltx_tr">
<td id="S4.T2.1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T2.1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T2.1.2.2.1.1.1" class="ltx_p" style="width:407.6pt;"></span>
</span>
</td>
</tr>
<tr id="S4.T2.1.3.3" class="ltx_tr">
<td id="S4.T2.1.3.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T2.1.3.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T2.1.3.3.1.1.1" class="ltx_p" style="width:407.6pt;"><span id="S4.T2.1.3.3.1.1.1.1" class="ltx_text ltx_font_typewriter">
<span id="S4.T2.1.3.3.1.1.1.1.1" class="ltx_text" style="color:#BF8040;">&lt;Retrieve for the question&gt;</span></span></span>
</span>
</td>
</tr>
<tr id="S4.T2.1.4.4" class="ltx_tr">
<td id="S4.T2.1.4.4.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T2.1.4.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T2.1.4.4.1.1.1" class="ltx_p" style="width:407.6pt;"><span id="S4.T2.1.4.4.1.1.1.1" class="ltx_text ltx_font_typewriter">Document [1](Title: American Decolonization) ...</span></span>
</span>
</td>
</tr>
<tr id="S4.T2.1.5.5" class="ltx_tr">
<td id="S4.T2.1.5.5.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T2.1.5.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T2.1.5.5.1.1.1" class="ltx_p" style="width:407.6pt;"><span id="S4.T2.1.5.5.1.1.1.1" class="ltx_text ltx_font_typewriter">Document [2](Title: Decolonization) ...</span></span>
</span>
</td>
</tr>
<tr id="S4.T2.1.6.6" class="ltx_tr">
<td id="S4.T2.1.6.6.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T2.1.6.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T2.1.6.6.1.1.1" class="ltx_p" style="width:407.6pt;"><span id="S4.T2.1.6.6.1.1.1.1" class="ltx_text ltx_font_typewriter">Document [3](Title: American Revolution) ...</span></span>
</span>
</td>
</tr>
<tr id="S4.T2.1.7.7" class="ltx_tr">
<td id="S4.T2.1.7.7.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T2.1.7.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T2.1.7.7.1.1.1" class="ltx_p" style="width:407.6pt;"><span id="S4.T2.1.7.7.1.1.1.1" class="ltx_text ltx_font_typewriter">...</span></span>
</span>
</td>
</tr>
<tr id="S4.T2.1.8.8" class="ltx_tr">
<td id="S4.T2.1.8.8.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T2.1.8.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T2.1.8.8.1.1.1" class="ltx_p" style="width:407.6pt;"></span>
</span>
</td>
</tr>
<tr id="S4.T2.1.9.9" class="ltx_tr">
<td id="S4.T2.1.9.9.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T2.1.9.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T2.1.9.9.1.1.1" class="ltx_p" style="width:407.6pt;"><span id="S4.T2.1.9.9.1.1.1.1" class="ltx_text ltx_font_typewriter">
Question: When did US break away from England?</span></span>
</span>
</td>
</tr>
<tr id="S4.T2.1.10.10" class="ltx_tr">
<td id="S4.T2.1.10.10.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="S4.T2.1.10.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T2.1.10.10.1.1.1" class="ltx_p" style="width:407.6pt;"><span id="S4.T2.1.10.10.1.1.1.1" class="ltx_text ltx_font_typewriter">Answer: <span id="S4.T2.1.10.10.1.1.1.1.1" class="ltx_text" style="color:#0000FF;"> The United States took the first step towards gaining independence ... <span id="S4.T2.1.10.10.1.1.1.1.1.1" class="ltx_text ltx_font_serif ltx_font_bold ltx_framed ltx_framed_underline">[1][2]</span>. The Treaty of Paris was later signed ... <span id="S4.T2.1.10.10.1.1.1.1.1.2" class="ltx_text ltx_font_serif ltx_font_bold ltx_framed ltx_framed_underline">[3]</span></span>.</span></span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 2: </span>
An example of our <span id="S4.T2.6.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> method. Different colors represent <span id="S4.T2.7.2" class="ltx_text ltx_font_typewriter">prompt</span>, <span id="S4.T2.8.3" class="ltx_text ltx_font_typewriter" style="color:#0000FF;"> model generation</span>, and <span id="S4.T2.9.4" class="ltx_text ltx_font_typewriter" style="color:#BF8040;"> &lt;actions&gt;</span>.
We also provide two in-context demonstrations before the test example.
</figcaption>
</figure>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Retrieval</h3>

<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.1" class="ltx_p">We explore simple, off-the-shelf retrievers.
We use dense retrievers for Wikipedia, including GTR <cite class="ltx_cite ltx_citemacro_cite">Ni et al. (<a href="#bib.bib28" title="" class="ltx_ref">2022</a>)</cite> and DPR <cite class="ltx_cite ltx_citemacro_cite">Karpukhin et al. (<a href="#bib.bib18" title="" class="ltx_ref">2020</a>)</cite>;
we use
BM25 for Sphere. For each question,
we
retrieve the top-100 passages.</p>
</div>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Synthesis</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.1" class="ltx_p">We focus on how to
prompt an LLM to
interact with the retriever, and
synthesize and cite the evidence (without fine-tuning internal parameters).
One noteworthy challenge is that existing LLMs all have limited context window and thus
can only fit a handful of passages.</p>
</div>
<section id="S4.SS2.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">
<span id="S4.SS2.SSS0.Px1.1.1" class="ltx_text ltx_font_smallcaps">Vanilla</span>.</h4>

<div id="S4.SS2.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS2.SSS0.Px1.p1.1" class="ltx_p">We simply provide the model with the top-<math id="S4.SS2.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S4.SS2.SSS0.Px1.p1.1.m1.1a"><mi id="S4.SS2.SSS0.Px1.p1.1.m1.1.1" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p1.1.m1.1b"><ci id="S4.SS2.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p1.1.m1.1c">k</annotation></semantics></math><span id="footnote9" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">9</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">9</sup><span class="ltx_tag ltx_tag_note">9</span>We can fit at most <math id="footnote9.m1.1" class="ltx_Math" alttext="k=3" display="inline"><semantics id="footnote9.m1.1b"><mrow id="footnote9.m1.1.1" xref="footnote9.m1.1.1.cmml"><mi id="footnote9.m1.1.1.2" xref="footnote9.m1.1.1.2.cmml">k</mi><mo id="footnote9.m1.1.1.1" xref="footnote9.m1.1.1.1.cmml">=</mo><mn id="footnote9.m1.1.1.3" xref="footnote9.m1.1.1.3.cmml">3</mn></mrow><annotation-xml encoding="MathML-Content" id="footnote9.m1.1c"><apply id="footnote9.m1.1.1.cmml" xref="footnote9.m1.1.1"><eq id="footnote9.m1.1.1.1.cmml" xref="footnote9.m1.1.1.1"></eq><ci id="footnote9.m1.1.1.2.cmml" xref="footnote9.m1.1.1.2">𝑘</ci><cn type="integer" id="footnote9.m1.1.1.3.cmml" xref="footnote9.m1.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote9.m1.1d">k=3</annotation></semantics></math> for models with 2K window and at most <math id="footnote9.m2.1" class="ltx_Math" alttext="k=5" display="inline"><semantics id="footnote9.m2.1b"><mrow id="footnote9.m2.1.1" xref="footnote9.m2.1.1.cmml"><mi id="footnote9.m2.1.1.2" xref="footnote9.m2.1.1.2.cmml">k</mi><mo id="footnote9.m2.1.1.1" xref="footnote9.m2.1.1.1.cmml">=</mo><mn id="footnote9.m2.1.1.3" xref="footnote9.m2.1.1.3.cmml">5</mn></mrow><annotation-xml encoding="MathML-Content" id="footnote9.m2.1c"><apply id="footnote9.m2.1.1.cmml" xref="footnote9.m2.1.1"><eq id="footnote9.m2.1.1.1.cmml" xref="footnote9.m2.1.1.1"></eq><ci id="footnote9.m2.1.1.2.cmml" xref="footnote9.m2.1.1.2">𝑘</ci><cn type="integer" id="footnote9.m2.1.1.3.cmml" xref="footnote9.m2.1.1.3">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote9.m2.1d">k=5</annotation></semantics></math> for models with 4K context window.</span></span></span> passages and instruct the model to cite accordingly (Table <a href="#S4.T2" title="Table 2 ‣ 4 Modeling ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>).
We also use in-context learning <cite class="ltx_cite ltx_citemacro_cite">Brown et al. (<a href="#bib.bib4" title="" class="ltx_ref">2020</a>)</cite> and prepend two demonstrations.
The complete instruction is in Table <a href="#A8.T23" title="Table 23 ‣ Appendix H Prompts ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">23</span></a>.</p>
</div>
</section>
<section id="S4.SS2.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">
<span id="S4.SS2.SSS0.Px2.1.1" class="ltx_text ltx_font_smallcaps">Summ</span>/<span id="S4.SS2.SSS0.Px2.2.2" class="ltx_text ltx_font_smallcaps">Snippet</span>.</h4>

<div id="S4.SS2.SSS0.Px2.p1" class="ltx_para">
<p id="S4.SS2.SSS0.Px2.p1.1" class="ltx_p">With a 4K context window, we can at most safely fit <math id="S4.SS2.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="k=5" display="inline"><semantics id="S4.SS2.SSS0.Px2.p1.1.m1.1a"><mrow id="S4.SS2.SSS0.Px2.p1.1.m1.1.1" xref="S4.SS2.SSS0.Px2.p1.1.m1.1.1.cmml"><mi id="S4.SS2.SSS0.Px2.p1.1.m1.1.1.2" xref="S4.SS2.SSS0.Px2.p1.1.m1.1.1.2.cmml">k</mi><mo id="S4.SS2.SSS0.Px2.p1.1.m1.1.1.1" xref="S4.SS2.SSS0.Px2.p1.1.m1.1.1.1.cmml">=</mo><mn id="S4.SS2.SSS0.Px2.p1.1.m1.1.1.3" xref="S4.SS2.SSS0.Px2.p1.1.m1.1.1.3.cmml">5</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px2.p1.1.m1.1b"><apply id="S4.SS2.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.1.1"><eq id="S4.SS2.SSS0.Px2.p1.1.m1.1.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.1.1.1"></eq><ci id="S4.SS2.SSS0.Px2.p1.1.m1.1.1.2.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.1.1.2">𝑘</ci><cn type="integer" id="S4.SS2.SSS0.Px2.p1.1.m1.1.1.3.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.1.1.3">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px2.p1.1.m1.1c">k=5</annotation></semantics></math> passages. As shown in Figure <a href="#S5.F4" title="Figure 4 ‣ 5.2 Comparison of Different LLMs ‣ 5 Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, top-5 retrieved passages can only cover 56.8% percent of the answers in ASQA.</p>
</div>
<div id="S4.SS2.SSS0.Px2.p2" class="ltx_para">
<p id="S4.SS2.SSS0.Px2.p2.1" class="ltx_p">To tackle this limitation, we propose to provide <span id="S4.SS2.SSS0.Px2.p2.1.1" class="ltx_text ltx_font_italic">summaries</span> or <span id="S4.SS2.SSS0.Px2.p2.1.2" class="ltx_text ltx_font_italic">snippets</span> of passages instead of the full text (summaries are abstractive but snippets are spans from passages).
We acquire summaries and snippets by prompting ChatGPT with instructions (prompts in Table <a href="#A8.T25" title="Table 25 ‣ Appendix H Prompts ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">25</span></a> and <a href="#A8.T26" title="Table 26 ‣ Appendix H Prompts ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">26</span></a>).<span id="footnote10" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">10</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">10</sup><span class="ltx_tag ltx_tag_note">10</span>We also query ChatGPT whether the passage is relevant to the question, and
filter out passages that are “irrelevant”.</span></span></span>
Then we replace all passages with summaries/snippets.
Summaries or snippets significantly reduce the passage length, allowing for more passages to fit in:
for ASQA, they reduce passage length by 6<math id="S4.SS2.SSS0.Px2.p2.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S4.SS2.SSS0.Px2.p2.1.m1.1a"><mo id="S4.SS2.SSS0.Px2.p2.1.m1.1.1" xref="S4.SS2.SSS0.Px2.p2.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px2.p2.1.m1.1b"><times id="S4.SS2.SSS0.Px2.p2.1.m1.1.1.cmml" xref="S4.SS2.SSS0.Px2.p2.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px2.p2.1.m1.1c">\times</annotation></semantics></math> on average.</p>
</div>
<div id="S4.SS2.SSS0.Px2.p3" class="ltx_para">
<p id="S4.SS2.SSS0.Px2.p3.1" class="ltx_p">Though <span id="S4.SS2.SSS0.Px2.p3.1.1" class="ltx_text ltx_font_smallcaps">Summ</span>/<span id="S4.SS2.SSS0.Px2.p3.1.2" class="ltx_text ltx_font_smallcaps">Snippet</span> allows for more retrieved passages,
they are lossy compressions.
To alleviate this problem,
we propose <span id="S4.SS2.SSS0.Px2.p3.1.3" class="ltx_text ltx_font_smallcaps">Interact</span>, an interactive prompting scheme
to allow the model to check the full text of certain passages. At each step, the model can execute one of three actions:
(1) “<span id="S4.SS2.SSS0.Px2.p3.1.4" class="ltx_text ltx_font_typewriter">Check: Document [1][2]</span>” to check the full text of the corresponding documents;
(2) “<span id="S4.SS2.SSS0.Px2.p3.1.5" class="ltx_text ltx_font_typewriter">Output:</span>” to output a statement of the answer;
(3) “<span id="S4.SS2.SSS0.Px2.p3.1.6" class="ltx_text ltx_font_typewriter">End.</span>” to end the generation.
§<a href="#A3" title="Appendix C Implementation Details ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a> provides more details.</p>
</div>
<figure id="S4.T3" class="ltx_table">
<table id="S4.T3.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S4.T3.1.1.1" class="ltx_tr">
<td id="S4.T3.1.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="S4.T3.1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T3.1.1.1.1.1.1" class="ltx_p" style="width:407.6pt;"><span id="S4.T3.1.1.1.1.1.1.1" class="ltx_text ltx_font_typewriter">Instruction: ...</span></span>
</span>
</td>
</tr>
<tr id="S4.T3.1.2.2" class="ltx_tr">
<td id="S4.T3.1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T3.1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T3.1.2.2.1.1.1" class="ltx_p" style="width:407.6pt;"></span>
</span>
</td>
</tr>
<tr id="S4.T3.1.3.3" class="ltx_tr">
<td id="S4.T3.1.3.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T3.1.3.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T3.1.3.3.1.1.1" class="ltx_p" style="width:407.6pt;"><span id="S4.T3.1.3.3.1.1.1.1" class="ltx_text ltx_font_typewriter">
<span id="S4.T3.1.3.3.1.1.1.1.1" class="ltx_text" style="color:#BF8040;">&lt;Retrieve for question ‘‘...’’&gt;</span></span></span>
</span>
</td>
</tr>
<tr id="S4.T3.1.4.4" class="ltx_tr">
<td id="S4.T3.1.4.4.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T3.1.4.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T3.1.4.4.1.1.1" class="ltx_p" style="width:407.6pt;"><span id="S4.T3.1.4.4.1.1.1.1" class="ltx_text ltx_font_typewriter">Question: When did US break away from England?</span></span>
</span>
</td>
</tr>
<tr id="S4.T3.1.5.5" class="ltx_tr">
<td id="S4.T3.1.5.5.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T3.1.5.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T3.1.5.5.1.1.1" class="ltx_p" style="width:407.6pt;"><span id="S4.T3.1.5.5.1.1.1.1" class="ltx_text ltx_font_typewriter" style="color:#0000FF;">Search: Declaration of Independence</span></span>
</span>
</td>
</tr>
<tr id="S4.T3.1.6.6" class="ltx_tr">
<td id="S4.T3.1.6.6.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T3.1.6.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T3.1.6.6.1.1.1" class="ltx_p" style="width:407.6pt;"><span id="S4.T3.1.6.6.1.1.1.1" class="ltx_text ltx_font_typewriter" style="color:#BF8040;">&lt;Search the query among the top-100 passages&gt;</span></span>
</span>
</td>
</tr>
<tr id="S4.T3.1.7.7" class="ltx_tr">
<td id="S4.T3.1.7.7.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T3.1.7.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T3.1.7.7.1.1.1" class="ltx_p" style="width:407.6pt;"><span id="S4.T3.1.7.7.1.1.1.1" class="ltx_text ltx_font_typewriter">Document [1](Title: ...) ...</span></span>
</span>
</td>
</tr>
<tr id="S4.T3.1.8.8" class="ltx_tr">
<td id="S4.T3.1.8.8.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T3.1.8.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T3.1.8.8.1.1.1" class="ltx_p" style="width:407.6pt;"><span id="S4.T3.1.8.8.1.1.1.1" class="ltx_text ltx_font_typewriter" style="color:#0000FF;">Output: The United States ... <span id="S4.T3.1.8.8.1.1.1.1.1" class="ltx_text ltx_font_serif ltx_font_bold ltx_framed ltx_framed_underline">[1]</span>.</span></span>
</span>
</td>
</tr>
<tr id="S4.T3.1.9.9" class="ltx_tr">
<td id="S4.T3.1.9.9.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T3.1.9.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T3.1.9.9.1.1.1" class="ltx_p" style="width:407.6pt;"><span id="S4.T3.1.9.9.1.1.1.1" class="ltx_text ltx_font_typewriter" style="color:#BF8040;">&lt;Remove Document [1] from context&gt;</span></span>
</span>
</td>
</tr>
<tr id="S4.T3.1.10.10" class="ltx_tr">
<td id="S4.T3.1.10.10.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T3.1.10.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T3.1.10.10.1.1.1" class="ltx_p" style="width:407.6pt;"><span id="S4.T3.1.10.10.1.1.1.1" class="ltx_text ltx_font_typewriter" style="color:#0000FF;">Search: Treaty of Paris</span></span>
</span>
</td>
</tr>
<tr id="S4.T3.1.11.11" class="ltx_tr">
<td id="S4.T3.1.11.11.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T3.1.11.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T3.1.11.11.1.1.1" class="ltx_p" style="width:407.6pt;"><span id="S4.T3.1.11.11.1.1.1.1" class="ltx_text ltx_font_typewriter" style="color:#BF8040;">&lt;Search the query among the top-100 passages&gt;</span></span>
</span>
</td>
</tr>
<tr id="S4.T3.1.12.12" class="ltx_tr">
<td id="S4.T3.1.12.12.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T3.1.12.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T3.1.12.12.1.1.1" class="ltx_p" style="width:407.6pt;"><span id="S4.T3.1.12.12.1.1.1.1" class="ltx_text ltx_font_typewriter">Document [3](Title: ...) ...</span></span>
</span>
</td>
</tr>
<tr id="S4.T3.1.13.13" class="ltx_tr">
<td id="S4.T3.1.13.13.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T3.1.13.13.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T3.1.13.13.1.1.1" class="ltx_p" style="width:407.6pt;"><span id="S4.T3.1.13.13.1.1.1.1" class="ltx_text ltx_font_typewriter" style="color:#0000FF;">Output: The Treaty of Paris ... <span id="S4.T3.1.13.13.1.1.1.1.1" class="ltx_text ltx_font_serif ltx_font_bold ltx_framed ltx_framed_underline">[3]</span>.</span></span>
</span>
</td>
</tr>
<tr id="S4.T3.1.14.14" class="ltx_tr">
<td id="S4.T3.1.14.14.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T3.1.14.14.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T3.1.14.14.1.1.1" class="ltx_p" style="width:407.6pt;"><span id="S4.T3.1.14.14.1.1.1.1" class="ltx_text ltx_font_typewriter" style="color:#BF8040;">&lt;Remove Document [3] from context&gt;</span></span>
</span>
</td>
</tr>
<tr id="S4.T3.1.15.15" class="ltx_tr">
<td id="S4.T3.1.15.15.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="S4.T3.1.15.15.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T3.1.15.15.1.1.1" class="ltx_p" style="width:407.6pt;"><span id="S4.T3.1.15.15.1.1.1.1" class="ltx_text ltx_font_typewriter" style="color:#0000FF;">End.</span></span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 3: </span>
An example of <span id="S4.T3.3.1" class="ltx_text ltx_font_smallcaps">InlineSearch</span>.
</figcaption>
</figure>
</section>
<section id="S4.SS2.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">
<span id="S4.SS2.SSS0.Px3.1.1" class="ltx_text ltx_font_smallcaps">InlineSearch</span>.</h4>

<div id="S4.SS2.SSS0.Px3.p1" class="ltx_para">
<p id="S4.SS2.SSS0.Px3.p1.1" class="ltx_p">The above methods all display retrieval results at the beginning.
In <span id="S4.SS2.SSS0.Px3.p1.1.1" class="ltx_text ltx_font_smallcaps">InlineSearch</span>,
we allow LLMs to call “search” during the generation process <cite class="ltx_cite ltx_citemacro_citep">(Yao et al., <a href="#bib.bib51" title="" class="ltx_ref">2023</a>; Press et al., <a href="#bib.bib35" title="" class="ltx_ref">2022</a>; Jiang et al., <a href="#bib.bib16" title="" class="ltx_ref">2023</a>)</cite>.
At each step, the model can execute one of three actions:
“<span id="S4.SS2.SSS0.Px3.p1.1.2" class="ltx_text ltx_font_typewriter">Search: {query}</span>” to search among the top-100 passages<span id="footnote11" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">11</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">11</sup><span class="ltx_tag ltx_tag_note">11</span>We do not search over the entire corpus because <span id="footnote11.1" class="ltx_text ltx_font_typewriter">{query}</span> may leave out certain context in the question and searching among the already-retrieved passages gives better results.</span></span></span> by using GTR; the “<span id="S4.SS2.SSS0.Px3.p1.1.3" class="ltx_text ltx_font_typewriter">Output</span>” and “<span id="S4.SS2.SSS0.Px3.p1.1.4" class="ltx_text ltx_font_typewriter">End</span>” actions are the same as <span id="S4.SS2.SSS0.Px3.p1.1.5" class="ltx_text ltx_font_smallcaps">Interact</span>.
For each “<span id="S4.SS2.SSS0.Px3.p1.1.6" class="ltx_text ltx_font_typewriter">Search</span>” action, we display the best retrieved passage in the context.
The passage is removed after one action to save context space.
Table <a href="#S4.T3" title="Table 3 ‣ Summ/Snippet. ‣ 4.2 Synthesis ‣ 4 Modeling ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> shows an example.</p>
</div>
</section>
<section id="S4.SS2.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">
<span id="S4.SS2.SSS0.Px4.1.1" class="ltx_text ltx_font_smallcaps">ClosedBook</span>.</h4>

<div id="S4.SS2.SSS0.Px4.p1" class="ltx_para">
<p id="S4.SS2.SSS0.Px4.p1.1" class="ltx_p">We also add a simple closed-book baseline, where the model is only prompted with the instruction and the question,
without any retrieved passages provided.
Consequently, this variant does not cite any evidences.</p>
</div>
</section>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Post-editing</h3>

<div id="S4.SS3.p1" class="ltx_para">
<p id="S4.SS3.p1.1" class="ltx_p">In this section we discuss two strategies for refining the output to further improve its quality.</p>
</div>
<section id="S4.SS3.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">
<span id="S4.SS3.SSS0.Px1.1.1" class="ltx_text ltx_font_smallcaps">Rerank</span>.</h4>

<div id="S4.SS3.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS3.SSS0.Px1.p1.1" class="ltx_p">We randomly sample <math id="S4.SS3.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="n_{\text{sample}}=4" display="inline"><semantics id="S4.SS3.SSS0.Px1.p1.1.m1.1a"><mrow id="S4.SS3.SSS0.Px1.p1.1.m1.1.1" xref="S4.SS3.SSS0.Px1.p1.1.m1.1.1.cmml"><msub id="S4.SS3.SSS0.Px1.p1.1.m1.1.1.2" xref="S4.SS3.SSS0.Px1.p1.1.m1.1.1.2.cmml"><mi id="S4.SS3.SSS0.Px1.p1.1.m1.1.1.2.2" xref="S4.SS3.SSS0.Px1.p1.1.m1.1.1.2.2.cmml">n</mi><mtext id="S4.SS3.SSS0.Px1.p1.1.m1.1.1.2.3" xref="S4.SS3.SSS0.Px1.p1.1.m1.1.1.2.3a.cmml">sample</mtext></msub><mo id="S4.SS3.SSS0.Px1.p1.1.m1.1.1.1" xref="S4.SS3.SSS0.Px1.p1.1.m1.1.1.1.cmml">=</mo><mn id="S4.SS3.SSS0.Px1.p1.1.m1.1.1.3" xref="S4.SS3.SSS0.Px1.p1.1.m1.1.1.3.cmml">4</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px1.p1.1.m1.1b"><apply id="S4.SS3.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S4.SS3.SSS0.Px1.p1.1.m1.1.1"><eq id="S4.SS3.SSS0.Px1.p1.1.m1.1.1.1.cmml" xref="S4.SS3.SSS0.Px1.p1.1.m1.1.1.1"></eq><apply id="S4.SS3.SSS0.Px1.p1.1.m1.1.1.2.cmml" xref="S4.SS3.SSS0.Px1.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S4.SS3.SSS0.Px1.p1.1.m1.1.1.2.1.cmml" xref="S4.SS3.SSS0.Px1.p1.1.m1.1.1.2">subscript</csymbol><ci id="S4.SS3.SSS0.Px1.p1.1.m1.1.1.2.2.cmml" xref="S4.SS3.SSS0.Px1.p1.1.m1.1.1.2.2">𝑛</ci><ci id="S4.SS3.SSS0.Px1.p1.1.m1.1.1.2.3a.cmml" xref="S4.SS3.SSS0.Px1.p1.1.m1.1.1.2.3"><mtext mathsize="70%" id="S4.SS3.SSS0.Px1.p1.1.m1.1.1.2.3.cmml" xref="S4.SS3.SSS0.Px1.p1.1.m1.1.1.2.3">sample</mtext></ci></apply><cn type="integer" id="S4.SS3.SSS0.Px1.p1.1.m1.1.1.3.cmml" xref="S4.SS3.SSS0.Px1.p1.1.m1.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px1.p1.1.m1.1c">n_{\text{sample}}=4</annotation></semantics></math> responses for each question, and select the best response using the automatic <span id="S4.SS3.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_italic">citation recall</span> score.
we expect <span id="S4.SS3.SSS0.Px1.p1.1.2" class="ltx_text ltx_font_smallcaps">Rerank</span> to improve the citation quality.</p>
</div>
</section>
<section id="S4.SS3.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">
<span id="S4.SS3.SSS0.Px2.1.1" class="ltx_text ltx_font_smallcaps">PostCite</span>.</h4>

<div id="S4.SS3.SSS0.Px2.p1" class="ltx_para">
<p id="S4.SS3.SSS0.Px2.p1.1" class="ltx_p">For each statement, we find the best matching passage among the top-100 retrieved passages using GTR and cite it.
We combine this with <span id="S4.SS3.SSS0.Px2.p1.1.1" class="ltx_text ltx_font_smallcaps">ClosedBook</span> in our experiments.</p>
</div>
</section>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Experiments</h2>

<figure id="S5.T4" class="ltx_table">
<div id="S5.T4.5" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:424.9pt;height:480.5pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(64.8pt,-73.3pt) scale(1.43876422245418,1.43876422245418) ;">
<table id="S5.T4.5.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S5.T4.5.1.1.1" class="ltx_tr">
<th id="S5.T4.5.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="S5.T4.5.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S5.T4.5.1.1.1.2.1" class="ltx_text ltx_font_bold">Fluency</span></th>
<th id="S5.T4.5.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S5.T4.5.1.1.1.3.1" class="ltx_text ltx_font_bold">Correct.</span></th>
<th id="S5.T4.5.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2"><span id="S5.T4.5.1.1.1.4.1" class="ltx_text ltx_font_bold">Citation</span></th>
</tr>
<tr id="S5.T4.5.1.2.2" class="ltx_tr">
<th id="S5.T4.5.1.2.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="S5.T4.5.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">(MAUVE)</th>
<th id="S5.T4.5.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">(EM Rec.)</th>
<th id="S5.T4.5.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Rec.</th>
<th id="S5.T4.5.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Prec.</th>
</tr>
<tr id="S5.T4.5.1.3.3" class="ltx_tr" style="background-color:#ECECEC;">
<th id="S5.T4.5.1.3.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_t" colspan="5"><span id="S5.T4.5.1.3.3.1.1" class="ltx_text ltx_font_bold" style="background-color:#ECECEC;">ChatGPT</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S5.T4.5.1.4.1" class="ltx_tr">
<th id="S5.T4.5.1.4.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S5.T4.5.1.4.1.1.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (5-psg)</th>
<td id="S5.T4.5.1.4.1.2" class="ltx_td ltx_align_center">66.6</td>
<td id="S5.T4.5.1.4.1.3" class="ltx_td ltx_align_center">40.4</td>
<td id="S5.T4.5.1.4.1.4" class="ltx_td ltx_align_center">73.6</td>
<td id="S5.T4.5.1.4.1.5" class="ltx_td ltx_align_center">72.5</td>
</tr>
<tr id="S5.T4.5.1.5.2" class="ltx_tr">
<th id="S5.T4.5.1.5.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">w/ <span id="S5.T4.5.1.5.2.1.1" class="ltx_text ltx_font_smallcaps">Rerank</span>
</th>
<td id="S5.T4.5.1.5.2.2" class="ltx_td ltx_align_center">77.0</td>
<td id="S5.T4.5.1.5.2.3" class="ltx_td ltx_align_center">40.2</td>
<td id="S5.T4.5.1.5.2.4" class="ltx_td ltx_align_center"><span id="S5.T4.5.1.5.2.4.1" class="ltx_text ltx_font_bold">84.8</span></td>
<td id="S5.T4.5.1.5.2.5" class="ltx_td ltx_align_center"><span id="S5.T4.5.1.5.2.5.1" class="ltx_text ltx_font_bold">81.6</span></td>
</tr>
<tr id="S5.T4.5.1.6.3" class="ltx_tr">
<th id="S5.T4.5.1.6.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S5.T4.5.1.6.3.1.1" class="ltx_text ltx_font_smallcaps">Summ</span> (10-psg)</th>
<td id="S5.T4.5.1.6.3.2" class="ltx_td ltx_align_center">70.0</td>
<td id="S5.T4.5.1.6.3.3" class="ltx_td ltx_align_center"><span id="S5.T4.5.1.6.3.3.1" class="ltx_text ltx_font_bold">43.3</span></td>
<td id="S5.T4.5.1.6.3.4" class="ltx_td ltx_align_center">68.9</td>
<td id="S5.T4.5.1.6.3.5" class="ltx_td ltx_align_center">61.8</td>
</tr>
<tr id="S5.T4.5.1.7.4" class="ltx_tr">
<th id="S5.T4.5.1.7.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">w/ <span id="S5.T4.5.1.7.4.1.1" class="ltx_text ltx_font_smallcaps">Interact</span>
</th>
<td id="S5.T4.5.1.7.4.2" class="ltx_td ltx_align_center">69.0</td>
<td id="S5.T4.5.1.7.4.3" class="ltx_td ltx_align_center">39.1</td>
<td id="S5.T4.5.1.7.4.4" class="ltx_td ltx_align_center">73.4</td>
<td id="S5.T4.5.1.7.4.5" class="ltx_td ltx_align_center">66.5</td>
</tr>
<tr id="S5.T4.5.1.8.5" class="ltx_tr">
<th id="S5.T4.5.1.8.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S5.T4.5.1.8.5.1.1" class="ltx_text ltx_font_smallcaps">Snippet</span> (10-psg)</th>
<td id="S5.T4.5.1.8.5.2" class="ltx_td ltx_align_center">69.8</td>
<td id="S5.T4.5.1.8.5.3" class="ltx_td ltx_align_center">41.4</td>
<td id="S5.T4.5.1.8.5.4" class="ltx_td ltx_align_center">65.3</td>
<td id="S5.T4.5.1.8.5.5" class="ltx_td ltx_align_center">57.4</td>
</tr>
<tr id="S5.T4.5.1.9.6" class="ltx_tr">
<th id="S5.T4.5.1.9.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S5.T4.5.1.9.6.1.1" class="ltx_text ltx_font_smallcaps">InlineSearch</span></th>
<td id="S5.T4.5.1.9.6.2" class="ltx_td ltx_align_center">58.7</td>
<td id="S5.T4.5.1.9.6.3" class="ltx_td ltx_align_center">32.4</td>
<td id="S5.T4.5.1.9.6.4" class="ltx_td ltx_align_center">58.3</td>
<td id="S5.T4.5.1.9.6.5" class="ltx_td ltx_align_center">58.2</td>
</tr>
<tr id="S5.T4.5.1.10.7" class="ltx_tr">
<th id="S5.T4.5.1.10.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S5.T4.5.1.10.7.1.1" class="ltx_text ltx_font_smallcaps">ClosedBook</span></th>
<td id="S5.T4.5.1.10.7.2" class="ltx_td ltx_align_center">52.7</td>
<td id="S5.T4.5.1.10.7.3" class="ltx_td ltx_align_center">38.3</td>
<td id="S5.T4.5.1.10.7.4" class="ltx_td ltx_align_center">26.7</td>
<td id="S5.T4.5.1.10.7.5" class="ltx_td ltx_align_center">26.7</td>
</tr>
<tr id="S5.T4.5.1.11.8" class="ltx_tr" style="background-color:#ECECEC;">
<th id="S5.T4.5.1.11.8.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" colspan="5"><span id="S5.T4.5.1.11.8.1.1" class="ltx_text ltx_font_bold" style="background-color:#ECECEC;">GPT-4<span id="S5.T4.5.1.11.8.1.1.1" class="ltx_text ltx_font_medium"> (<span id="S5.T4.5.1.11.8.1.1.1.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> prompting)</span></span></th>
</tr>
<tr id="S5.T4.5.1.12.9" class="ltx_tr">
<th id="S5.T4.5.1.12.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">GPT-4 (5-psg)</th>
<td id="S5.T4.5.1.12.9.2" class="ltx_td ltx_align_center">67.1</td>
<td id="S5.T4.5.1.12.9.3" class="ltx_td ltx_align_center">41.3</td>
<td id="S5.T4.5.1.12.9.4" class="ltx_td ltx_align_center">68.5</td>
<td id="S5.T4.5.1.12.9.5" class="ltx_td ltx_align_center">75.6</td>
</tr>
<tr id="S5.T4.5.1.13.10" class="ltx_tr">
<th id="S5.T4.5.1.13.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">GPT-4 (20-psg)</th>
<td id="S5.T4.5.1.13.10.2" class="ltx_td ltx_align_center">64.9</td>
<td id="S5.T4.5.1.13.10.3" class="ltx_td ltx_align_center">44.4</td>
<td id="S5.T4.5.1.13.10.4" class="ltx_td ltx_align_center">73.0</td>
<td id="S5.T4.5.1.13.10.5" class="ltx_td ltx_align_center">76.5</td>
</tr>
<tr id="S5.T4.5.1.14.11" class="ltx_tr" style="background-color:#ECECEC;">
<th id="S5.T4.5.1.14.11.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" colspan="5"><span id="S5.T4.5.1.14.11.1.1" class="ltx_text ltx_font_bold" style="background-color:#ECECEC;">LLaMA<span id="S5.T4.5.1.14.11.1.1.1" class="ltx_text ltx_font_medium"> (<span id="S5.T4.5.1.14.11.1.1.1.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> prompting)</span></span></th>
</tr>
<tr id="S5.T4.5.1.15.12" class="ltx_tr">
<th id="S5.T4.5.1.15.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-13B (3-psg)</th>
<td id="S5.T4.5.1.15.12.2" class="ltx_td ltx_align_center">68.4</td>
<td id="S5.T4.5.1.15.12.3" class="ltx_td ltx_align_center">26.9</td>
<td id="S5.T4.5.1.15.12.4" class="ltx_td ltx_align_center">10.6</td>
<td id="S5.T4.5.1.15.12.5" class="ltx_td ltx_align_center">15.4</td>
</tr>
<tr id="S5.T4.5.1.16.13" class="ltx_tr">
<th id="S5.T4.5.1.16.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Vicuna-13B (3-psg)</th>
<td id="S5.T4.5.1.16.13.2" class="ltx_td ltx_align_center">82.6</td>
<td id="S5.T4.5.1.16.13.3" class="ltx_td ltx_align_center">31.9</td>
<td id="S5.T4.5.1.16.13.4" class="ltx_td ltx_align_center">51.1</td>
<td id="S5.T4.5.1.16.13.5" class="ltx_td ltx_align_center">50.1</td>
</tr>
<tr id="S5.T4.5.1.17.14" class="ltx_tr">
<th id="S5.T4.5.1.17.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Chat-13B (5-psg)</th>
<td id="S5.T4.5.1.17.14.2" class="ltx_td ltx_align_center">72.4</td>
<td id="S5.T4.5.1.17.14.3" class="ltx_td ltx_align_center">35.2</td>
<td id="S5.T4.5.1.17.14.4" class="ltx_td ltx_align_center">38.4</td>
<td id="S5.T4.5.1.17.14.5" class="ltx_td ltx_align_center">39.4</td>
</tr>
<tr id="S5.T4.5.1.18.15" class="ltx_tr">
<th id="S5.T4.5.1.18.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">Chat-70B (5-psg)</th>
<td id="S5.T4.5.1.18.15.2" class="ltx_td ltx_align_center ltx_border_bb">88.3</td>
<td id="S5.T4.5.1.18.15.3" class="ltx_td ltx_align_center ltx_border_bb">41.5</td>
<td id="S5.T4.5.1.18.15.4" class="ltx_td ltx_align_center ltx_border_bb">62.9</td>
<td id="S5.T4.5.1.18.15.5" class="ltx_td ltx_align_center ltx_border_bb">61.3</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 4: </span>
Experiments on ASQA.
For <span id="S5.T4.8.1" class="ltx_text ltx_font_smallcaps">ClosedBook</span>, we use <span id="S5.T4.9.2" class="ltx_text ltx_font_smallcaps">PostCite</span> to get citations.
<math id="S5.T4.3.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S5.T4.3.m1.1b"><mi id="S5.T4.3.m1.1.1" xref="S5.T4.3.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S5.T4.3.m1.1c"><ci id="S5.T4.3.m1.1.1.cmml" xref="S5.T4.3.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.T4.3.m1.1d">k</annotation></semantics></math>-psg: putting top-<math id="S5.T4.4.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S5.T4.4.m2.1b"><mi id="S5.T4.4.m2.1.1" xref="S5.T4.4.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S5.T4.4.m2.1c"><ci id="S5.T4.4.m2.1.1.cmml" xref="S5.T4.4.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.T4.4.m2.1d">k</annotation></semantics></math> passages from the retrieval results into the context.
Chat-13B and Chat-70B refer to LLaMA-2-Chat.
</figcaption>
</figure>
<figure id="S5.T5" class="ltx_table">
<div id="S5.T5.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:424.9pt;height:560.9pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(86.0pt,-113.5pt) scale(1.67948653235355,1.67948653235355) ;">
<table id="S5.T5.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S5.T5.1.1.1.1" class="ltx_tr">
<th id="S5.T5.1.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="S5.T5.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2"><span id="S5.T5.1.1.1.1.2.1" class="ltx_text ltx_font_bold">Correctness</span></th>
<th id="S5.T5.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2"><span id="S5.T5.1.1.1.1.3.1" class="ltx_text ltx_font_bold">Citation</span></th>
</tr>
<tr id="S5.T5.1.1.2.2" class="ltx_tr">
<th id="S5.T5.1.1.2.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="S5.T5.1.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Rec.-5</th>
<th id="S5.T5.1.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Prec.</th>
<th id="S5.T5.1.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Rec.</th>
<th id="S5.T5.1.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Prec.</th>
</tr>
<tr id="S5.T5.1.1.3.3" class="ltx_tr" style="background-color:#ECECEC;">
<th id="S5.T5.1.1.3.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_t" colspan="5"><span id="S5.T5.1.1.3.3.1.1" class="ltx_text ltx_font_bold" style="background-color:#ECECEC;">ChatGPT</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S5.T5.1.1.4.1" class="ltx_tr">
<th id="S5.T5.1.1.4.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S5.T5.1.1.4.1.1.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (5-psg)</th>
<td id="S5.T5.1.1.4.1.2" class="ltx_td ltx_align_center">20.8</td>
<td id="S5.T5.1.1.4.1.3" class="ltx_td ltx_align_center">20.8</td>
<td id="S5.T5.1.1.4.1.4" class="ltx_td ltx_align_center">20.5</td>
<td id="S5.T5.1.1.4.1.5" class="ltx_td ltx_align_center">20.9</td>
</tr>
<tr id="S5.T5.1.1.5.2" class="ltx_tr">
<th id="S5.T5.1.1.5.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">w/ <span id="S5.T5.1.1.5.2.1.1" class="ltx_text ltx_font_smallcaps">Rerank</span>
</th>
<td id="S5.T5.1.1.5.2.2" class="ltx_td ltx_align_center">22.8</td>
<td id="S5.T5.1.1.5.2.3" class="ltx_td ltx_align_center">21.4</td>
<td id="S5.T5.1.1.5.2.4" class="ltx_td ltx_align_center">21.2</td>
<td id="S5.T5.1.1.5.2.5" class="ltx_td ltx_align_center">21.4</td>
</tr>
<tr id="S5.T5.1.1.6.3" class="ltx_tr">
<th id="S5.T5.1.1.6.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S5.T5.1.1.6.3.1.1" class="ltx_text ltx_font_smallcaps">Summ</span> (10-psg)</th>
<td id="S5.T5.1.1.6.3.2" class="ltx_td ltx_align_center">23.6</td>
<td id="S5.T5.1.1.6.3.3" class="ltx_td ltx_align_center">21.2</td>
<td id="S5.T5.1.1.6.3.4" class="ltx_td ltx_align_center"><span id="S5.T5.1.1.6.3.4.1" class="ltx_text ltx_font_bold">23.6</span></td>
<td id="S5.T5.1.1.6.3.5" class="ltx_td ltx_align_center"><span id="S5.T5.1.1.6.3.5.1" class="ltx_text ltx_font_bold">25.7</span></td>
</tr>
<tr id="S5.T5.1.1.7.4" class="ltx_tr">
<th id="S5.T5.1.1.7.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S5.T5.1.1.7.4.1.1" class="ltx_text ltx_font_smallcaps">Snippet</span> (10-psg)</th>
<td id="S5.T5.1.1.7.4.2" class="ltx_td ltx_align_center">24.5</td>
<td id="S5.T5.1.1.7.4.3" class="ltx_td ltx_align_center">21.5</td>
<td id="S5.T5.1.1.7.4.4" class="ltx_td ltx_align_center">22.9</td>
<td id="S5.T5.1.1.7.4.5" class="ltx_td ltx_align_center">24.9</td>
</tr>
<tr id="S5.T5.1.1.8.5" class="ltx_tr">
<th id="S5.T5.1.1.8.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">w/ <span id="S5.T5.1.1.8.5.1.1" class="ltx_text ltx_font_smallcaps">Interact</span>
</th>
<td id="S5.T5.1.1.8.5.2" class="ltx_td ltx_align_center">21.9</td>
<td id="S5.T5.1.1.8.5.3" class="ltx_td ltx_align_center"><span id="S5.T5.1.1.8.5.3.1" class="ltx_text ltx_font_bold">23.0</span></td>
<td id="S5.T5.1.1.8.5.4" class="ltx_td ltx_align_center">21.9</td>
<td id="S5.T5.1.1.8.5.5" class="ltx_td ltx_align_center">23.4</td>
</tr>
<tr id="S5.T5.1.1.9.6" class="ltx_tr">
<th id="S5.T5.1.1.9.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S5.T5.1.1.9.6.1.1" class="ltx_text ltx_font_smallcaps">InlineSearch</span></th>
<td id="S5.T5.1.1.9.6.2" class="ltx_td ltx_align_center">17.2</td>
<td id="S5.T5.1.1.9.6.3" class="ltx_td ltx_align_center">20.4</td>
<td id="S5.T5.1.1.9.6.4" class="ltx_td ltx_align_center">14.9</td>
<td id="S5.T5.1.1.9.6.5" class="ltx_td ltx_align_center">14.9</td>
</tr>
<tr id="S5.T5.1.1.10.7" class="ltx_tr">
<th id="S5.T5.1.1.10.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S5.T5.1.1.10.7.1.1" class="ltx_text ltx_font_smallcaps">ClosedBook</span></th>
<td id="S5.T5.1.1.10.7.2" class="ltx_td ltx_align_center"><span id="S5.T5.1.1.10.7.2.1" class="ltx_text ltx_font_bold">32.9</span></td>
<td id="S5.T5.1.1.10.7.3" class="ltx_td ltx_align_center">19.8</td>
<td id="S5.T5.1.1.10.7.4" class="ltx_td ltx_align_center">10.0</td>
<td id="S5.T5.1.1.10.7.5" class="ltx_td ltx_align_center">10.0</td>
</tr>
<tr id="S5.T5.1.1.11.8" class="ltx_tr" style="background-color:#ECECEC;">
<th id="S5.T5.1.1.11.8.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" colspan="5"><span id="S5.T5.1.1.11.8.1.1" class="ltx_text ltx_font_bold" style="background-color:#ECECEC;">GPT-4<span id="S5.T5.1.1.11.8.1.1.1" class="ltx_text ltx_font_medium"> (<span id="S5.T5.1.1.11.8.1.1.1.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> prompting)</span></span></th>
</tr>
<tr id="S5.T5.1.1.12.9" class="ltx_tr">
<th id="S5.T5.1.1.12.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">GPT-4 (5-psg)</th>
<td id="S5.T5.1.1.12.9.2" class="ltx_td ltx_align_center">22.2</td>
<td id="S5.T5.1.1.12.9.3" class="ltx_td ltx_align_center">25.0</td>
<td id="S5.T5.1.1.12.9.4" class="ltx_td ltx_align_center">25.9</td>
<td id="S5.T5.1.1.12.9.5" class="ltx_td ltx_align_center">27.0</td>
</tr>
<tr id="S5.T5.1.1.13.10" class="ltx_tr">
<th id="S5.T5.1.1.13.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">GPT-4 (20-psg)</th>
<td id="S5.T5.1.1.13.10.2" class="ltx_td ltx_align_center">29.6</td>
<td id="S5.T5.1.1.13.10.3" class="ltx_td ltx_align_center">26.2</td>
<td id="S5.T5.1.1.13.10.4" class="ltx_td ltx_align_center">27.4</td>
<td id="S5.T5.1.1.13.10.5" class="ltx_td ltx_align_center">28.5</td>
</tr>
<tr id="S5.T5.1.1.14.11" class="ltx_tr" style="background-color:#ECECEC;">
<th id="S5.T5.1.1.14.11.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" colspan="5"><span id="S5.T5.1.1.14.11.1.1" class="ltx_text ltx_font_bold" style="background-color:#ECECEC;">LLaMA<span id="S5.T5.1.1.14.11.1.1.1" class="ltx_text ltx_font_medium"> (<span id="S5.T5.1.1.14.11.1.1.1.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> prompting)</span></span></th>
</tr>
<tr id="S5.T5.1.1.15.12" class="ltx_tr">
<th id="S5.T5.1.1.15.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-13B (3-psg)</th>
<td id="S5.T5.1.1.15.12.2" class="ltx_td ltx_align_center">9.7</td>
<td id="S5.T5.1.1.15.12.3" class="ltx_td ltx_align_center">9.1</td>
<td id="S5.T5.1.1.15.12.4" class="ltx_td ltx_align_center">6.7</td>
<td id="S5.T5.1.1.15.12.5" class="ltx_td ltx_align_center">7.1</td>
</tr>
<tr id="S5.T5.1.1.16.13" class="ltx_tr">
<th id="S5.T5.1.1.16.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Vicuna-13B (5-psg)</th>
<td id="S5.T5.1.1.16.13.2" class="ltx_td ltx_align_center">14.0</td>
<td id="S5.T5.1.1.16.13.3" class="ltx_td ltx_align_center">15.9</td>
<td id="S5.T5.1.1.16.13.4" class="ltx_td ltx_align_center">12.5</td>
<td id="S5.T5.1.1.16.13.5" class="ltx_td ltx_align_center">13.4</td>
</tr>
<tr id="S5.T5.1.1.17.14" class="ltx_tr">
<th id="S5.T5.1.1.17.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Chat-13B (5-psg)</th>
<td id="S5.T5.1.1.17.14.2" class="ltx_td ltx_align_center">21.1</td>
<td id="S5.T5.1.1.17.14.3" class="ltx_td ltx_align_center">18.2</td>
<td id="S5.T5.1.1.17.14.4" class="ltx_td ltx_align_center">9.6</td>
<td id="S5.T5.1.1.17.14.5" class="ltx_td ltx_align_center">9.7</td>
</tr>
<tr id="S5.T5.1.1.18.15" class="ltx_tr">
<th id="S5.T5.1.1.18.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">Chat-70B (5-psg)</th>
<td id="S5.T5.1.1.18.15.2" class="ltx_td ltx_align_center ltx_border_bb">21.8</td>
<td id="S5.T5.1.1.18.15.3" class="ltx_td ltx_align_center ltx_border_bb">18.4</td>
<td id="S5.T5.1.1.18.15.4" class="ltx_td ltx_align_center ltx_border_bb">15.1</td>
<td id="S5.T5.1.1.18.15.5" class="ltx_td ltx_align_center ltx_border_bb">15.6</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 5: </span>
Experiments on QAMPARI.
“Rec.-5”: we set the recall to be 100% if the prediction includes at least 5 correct answers.
</figcaption>
</figure>
<figure id="S5.T6" class="ltx_table">
<div id="S5.T6.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:424.9pt;height:498.3pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(70.0pt,-82.1pt) scale(1.4917984480054,1.4917984480054) ;">
<table id="S5.T6.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S5.T6.1.1.1.1" class="ltx_tr">
<th id="S5.T6.1.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="S5.T6.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S5.T6.1.1.1.1.2.1" class="ltx_text ltx_font_bold">Fluency</span></th>
<th id="S5.T6.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S5.T6.1.1.1.1.3.1" class="ltx_text ltx_font_bold">Correct.</span></th>
<th id="S5.T6.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2"><span id="S5.T6.1.1.1.1.4.1" class="ltx_text ltx_font_bold">Citation</span></th>
</tr>
<tr id="S5.T6.1.1.2.2" class="ltx_tr">
<th id="S5.T6.1.1.2.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="S5.T6.1.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">(MAUVE)</th>
<th id="S5.T6.1.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">(Claim)</th>
<th id="S5.T6.1.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Rec.</th>
<th id="S5.T6.1.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Prec.</th>
</tr>
<tr id="S5.T6.1.1.3.3" class="ltx_tr" style="background-color:#ECECEC;">
<th id="S5.T6.1.1.3.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_t" colspan="5"><span id="S5.T6.1.1.3.3.1.1" class="ltx_text ltx_font_bold" style="background-color:#ECECEC;">ChatGPT</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S5.T6.1.1.4.1" class="ltx_tr">
<th id="S5.T6.1.1.4.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S5.T6.1.1.4.1.1.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (5-psg)</th>
<td id="S5.T6.1.1.4.1.2" class="ltx_td ltx_align_center">57.2</td>
<td id="S5.T6.1.1.4.1.3" class="ltx_td ltx_align_center">12.0</td>
<td id="S5.T6.1.1.4.1.4" class="ltx_td ltx_align_center">51.1</td>
<td id="S5.T6.1.1.4.1.5" class="ltx_td ltx_align_center">50.0</td>
</tr>
<tr id="S5.T6.1.1.5.2" class="ltx_tr">
<th id="S5.T6.1.1.5.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">w/ <span id="S5.T6.1.1.5.2.1.1" class="ltx_text ltx_font_smallcaps">Rerank</span>
</th>
<td id="S5.T6.1.1.5.2.2" class="ltx_td ltx_align_center">56.1</td>
<td id="S5.T6.1.1.5.2.3" class="ltx_td ltx_align_center">11.4</td>
<td id="S5.T6.1.1.5.2.4" class="ltx_td ltx_align_center"><span id="S5.T6.1.1.5.2.4.1" class="ltx_text ltx_font_bold">69.3</span></td>
<td id="S5.T6.1.1.5.2.5" class="ltx_td ltx_align_center"><span id="S5.T6.1.1.5.2.5.1" class="ltx_text ltx_font_bold">67.8</span></td>
</tr>
<tr id="S5.T6.1.1.6.3" class="ltx_tr">
<th id="S5.T6.1.1.6.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S5.T6.1.1.6.3.1.1" class="ltx_text ltx_font_smallcaps">Summ</span> (10-psg)</th>
<td id="S5.T6.1.1.6.3.2" class="ltx_td ltx_align_center">40.3</td>
<td id="S5.T6.1.1.6.3.3" class="ltx_td ltx_align_center">12.5</td>
<td id="S5.T6.1.1.6.3.4" class="ltx_td ltx_align_center">51.5</td>
<td id="S5.T6.1.1.6.3.5" class="ltx_td ltx_align_center">48.2</td>
</tr>
<tr id="S5.T6.1.1.7.4" class="ltx_tr">
<th id="S5.T6.1.1.7.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S5.T6.1.1.7.4.1.1" class="ltx_text ltx_font_smallcaps">Snippet</span> (10-psg)</th>
<td id="S5.T6.1.1.7.4.2" class="ltx_td ltx_align_center">62.9</td>
<td id="S5.T6.1.1.7.4.3" class="ltx_td ltx_align_center">14.3</td>
<td id="S5.T6.1.1.7.4.4" class="ltx_td ltx_align_center">50.4</td>
<td id="S5.T6.1.1.7.4.5" class="ltx_td ltx_align_center">45.0</td>
</tr>
<tr id="S5.T6.1.1.8.5" class="ltx_tr">
<th id="S5.T6.1.1.8.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">w/ <span id="S5.T6.1.1.8.5.1.1" class="ltx_text ltx_font_smallcaps">Interact</span>
</th>
<td id="S5.T6.1.1.8.5.2" class="ltx_td ltx_align_center">68.0</td>
<td id="S5.T6.1.1.8.5.3" class="ltx_td ltx_align_center">13.3</td>
<td id="S5.T6.1.1.8.5.4" class="ltx_td ltx_align_center">47.8</td>
<td id="S5.T6.1.1.8.5.5" class="ltx_td ltx_align_center">45.0</td>
</tr>
<tr id="S5.T6.1.1.9.6" class="ltx_tr">
<th id="S5.T6.1.1.9.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S5.T6.1.1.9.6.1.1" class="ltx_text ltx_font_smallcaps">InlineSearch</span></th>
<td id="S5.T6.1.1.9.6.2" class="ltx_td ltx_align_center">49.7</td>
<td id="S5.T6.1.1.9.6.3" class="ltx_td ltx_align_center">13.4</td>
<td id="S5.T6.1.1.9.6.4" class="ltx_td ltx_align_center">45.6</td>
<td id="S5.T6.1.1.9.6.5" class="ltx_td ltx_align_center">43.7</td>
</tr>
<tr id="S5.T6.1.1.10.7" class="ltx_tr">
<th id="S5.T6.1.1.10.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S5.T6.1.1.10.7.1.1" class="ltx_text ltx_font_smallcaps">ClosedBook</span></th>
<td id="S5.T6.1.1.10.7.2" class="ltx_td ltx_align_center">32.6</td>
<td id="S5.T6.1.1.10.7.3" class="ltx_td ltx_align_center"><span id="S5.T6.1.1.10.7.3.1" class="ltx_text ltx_font_bold">18.6</span></td>
<td id="S5.T6.1.1.10.7.4" class="ltx_td ltx_align_center">15.5</td>
<td id="S5.T6.1.1.10.7.5" class="ltx_td ltx_align_center">15.5</td>
</tr>
<tr id="S5.T6.1.1.11.8" class="ltx_tr" style="background-color:#ECECEC;">
<th id="S5.T6.1.1.11.8.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" colspan="5"><span id="S5.T6.1.1.11.8.1.1" class="ltx_text ltx_font_bold" style="background-color:#ECECEC;">GPT-4<span id="S5.T6.1.1.11.8.1.1.1" class="ltx_text ltx_font_medium"> (<span id="S5.T6.1.1.11.8.1.1.1.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> prompting)</span></span></th>
</tr>
<tr id="S5.T6.1.1.12.9" class="ltx_tr">
<th id="S5.T6.1.1.12.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">GPT-4 (5-psg)</th>
<td id="S5.T6.1.1.12.9.2" class="ltx_td ltx_align_center">38.4</td>
<td id="S5.T6.1.1.12.9.3" class="ltx_td ltx_align_center">14.2</td>
<td id="S5.T6.1.1.12.9.4" class="ltx_td ltx_align_center">44.0</td>
<td id="S5.T6.1.1.12.9.5" class="ltx_td ltx_align_center">50.1</td>
</tr>
<tr id="S5.T6.1.1.13.10" class="ltx_tr">
<th id="S5.T6.1.1.13.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">GPT-4 (20-psg)</th>
<td id="S5.T6.1.1.13.10.2" class="ltx_td ltx_align_center">41.5</td>
<td id="S5.T6.1.1.13.10.3" class="ltx_td ltx_align_center">18.3</td>
<td id="S5.T6.1.1.13.10.4" class="ltx_td ltx_align_center">48.5</td>
<td id="S5.T6.1.1.13.10.5" class="ltx_td ltx_align_center">53.4</td>
</tr>
<tr id="S5.T6.1.1.14.11" class="ltx_tr" style="background-color:#ECECEC;">
<th id="S5.T6.1.1.14.11.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" colspan="5"><span id="S5.T6.1.1.14.11.1.1" class="ltx_text ltx_font_bold" style="background-color:#ECECEC;">LLaMA<span id="S5.T6.1.1.14.11.1.1.1" class="ltx_text ltx_font_medium"> (<span id="S5.T6.1.1.14.11.1.1.1.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> prompting)</span></span></th>
</tr>
<tr id="S5.T6.1.1.15.12" class="ltx_tr">
<th id="S5.T6.1.1.15.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-13B (3-psg)</th>
<td id="S5.T6.1.1.15.12.2" class="ltx_td ltx_align_center">50.0</td>
<td id="S5.T6.1.1.15.12.3" class="ltx_td ltx_align_center">3.9</td>
<td id="S5.T6.1.1.15.12.4" class="ltx_td ltx_align_center">3.1</td>
<td id="S5.T6.1.1.15.12.5" class="ltx_td ltx_align_center">5.3</td>
</tr>
<tr id="S5.T6.1.1.16.13" class="ltx_tr">
<th id="S5.T6.1.1.16.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Vicuna-13B (3-psg)</th>
<td id="S5.T6.1.1.16.13.2" class="ltx_td ltx_align_center">58.2</td>
<td id="S5.T6.1.1.16.13.3" class="ltx_td ltx_align_center">10.0</td>
<td id="S5.T6.1.1.16.13.4" class="ltx_td ltx_align_center">15.6</td>
<td id="S5.T6.1.1.16.13.5" class="ltx_td ltx_align_center">19.6</td>
</tr>
<tr id="S5.T6.1.1.17.14" class="ltx_tr">
<th id="S5.T6.1.1.17.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Chat-13B (5-psg)</th>
<td id="S5.T6.1.1.17.14.2" class="ltx_td ltx_align_center">34.7</td>
<td id="S5.T6.1.1.17.14.3" class="ltx_td ltx_align_center">13.4</td>
<td id="S5.T6.1.1.17.14.4" class="ltx_td ltx_align_center">17.3</td>
<td id="S5.T6.1.1.17.14.5" class="ltx_td ltx_align_center">15.8</td>
</tr>
<tr id="S5.T6.1.1.18.15" class="ltx_tr">
<th id="S5.T6.1.1.18.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">Chat-70B (5-psg)</th>
<td id="S5.T6.1.1.18.15.2" class="ltx_td ltx_align_center ltx_border_bb">38.6</td>
<td id="S5.T6.1.1.18.15.3" class="ltx_td ltx_align_center ltx_border_bb">12.8</td>
<td id="S5.T6.1.1.18.15.4" class="ltx_td ltx_align_center ltx_border_bb">38.3</td>
<td id="S5.T6.1.1.18.15.5" class="ltx_td ltx_align_center ltx_border_bb">37.9</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 6: </span>
Experiments on ELI5. We use <span id="S5.T6.3.1" class="ltx_text ltx_font_italic">claim recall</span> for the correctness evaluation.
Chat-13B and Chat-70B refer to LLaMA-2-Chat.
</figcaption>
</figure>
<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">We describe experiment details in §<a href="#A3" title="Appendix C Implementation Details ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>.
We use ChatGPT (<span id="S5.p1.1.1" class="ltx_text ltx_font_typewriter">gpt-3.5-turbo-0301</span>) with a 4K context window for most main experiments and ablations.
We also report results with ChatGPT-16K (<span id="S5.p1.1.2" class="ltx_text ltx_font_typewriter">gpt- 3.5-turbo-16k-0613</span>) and GPT-4 (<span id="S5.p1.1.3" class="ltx_text ltx_font_typewriter">gpt-4-0613</span>; 8K context window).
For open-source models, we test LLaMA <cite class="ltx_cite ltx_citemacro_citep">(Touvron et al., <a href="#bib.bib47" title="" class="ltx_ref">2023a</a>)</cite> and
its instruction-tuned versions, including
Alpaca <cite class="ltx_cite ltx_citemacro_citep">(Taori et al., <a href="#bib.bib45" title="" class="ltx_ref">2023</a>)</cite>, Vicuna <cite class="ltx_cite ltx_citemacro_citep">(Chiang et al., <a href="#bib.bib5" title="" class="ltx_ref">2023</a>)</cite>, and Oasst <cite class="ltx_cite ltx_citemacro_citep">(Köpf et al., <a href="#bib.bib22" title="" class="ltx_ref">2023</a>)</cite>.
They all have a 2K context window.
We use short instructions for LLaMA (Table <a href="#A8.T24" title="Table 24 ‣ Appendix H Prompts ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">24</span></a>) to save context budget.
Additionally, we test LLaMA-2-Chat, which were also trained to follow instructions <cite class="ltx_cite ltx_citemacro_cite">Touvron et al. (<a href="#bib.bib48" title="" class="ltx_ref">2023b</a>)</cite>. These models have a context window of 4K tokens, which allows for 5 passages per question.</p>
</div>
<section id="S5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.1 </span>Main Results</h3>

<div id="S5.SS1.p1" class="ltx_para">
<p id="S5.SS1.p1.1" class="ltx_p">We present the main results on three datasets in
Table <a href="#S5.T4" title="Table 4 ‣ 5 Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, <a href="#S5.T5" title="Table 5 ‣ 5 Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, and <a href="#S5.T6" title="Table 6 ‣ 5 Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> respectively (full results in §<a href="#A7.SS6" title="G.6 Main Results ‣ Appendix G More Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">G.6</span></a>). We first note that all models achieve good fluency scores (except some models on ELI5 mainly due to their longer generations).
We summarize the main takeaways from the experiments below.</p>
</div>
<section id="S5.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">
<span id="S5.SS1.SSS0.Px1.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">Vanilla</span> achieves strong performance.</h4>

<div id="S5.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="S5.SS1.SSS0.Px1.p1.1" class="ltx_p">Despite its simplicity,
<span id="S5.SS1.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (putting retrieved passages in context)
achieves close-to-the-best performance among all prompting strategies.</p>
</div>
</section>
<section id="S5.SS1.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Using summaries or snippets improves correctness.</h4>

<div id="S5.SS1.SSS0.Px2.p1" class="ltx_para">
<p id="S5.SS1.SSS0.Px2.p1.1" class="ltx_p">We see a universal trend that
<span id="S5.SS1.SSS0.Px2.p1.1.1" class="ltx_text ltx_font_smallcaps">Summ</span> or <span id="S5.SS1.SSS0.Px2.p1.1.2" class="ltx_text ltx_font_smallcaps">Snippet</span>
improves correctness,
though
on ASQA and ELI5, such an improvement comes at a cost of citation quality
due to the lossy compression. Combining <span id="S5.SS1.SSS0.Px2.p1.1.3" class="ltx_text ltx_font_smallcaps">Interact</span> with <span id="S5.SS1.SSS0.Px2.p1.1.4" class="ltx_text ltx_font_smallcaps">Summ</span>/<span id="S5.SS1.SSS0.Px2.p1.1.5" class="ltx_text ltx_font_smallcaps">Snippet</span> does not bring improvement,
and we hypothesize that
checking the full passages offers limited benefit and
current LLMs are not proficient in an interactive usage.</p>
</div>
</section>
<section id="S5.SS1.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Retrieving text on the fly does not improve performance.</h4>

<div id="S5.SS1.SSS0.Px3.p1" class="ltx_para">
<p id="S5.SS1.SSS0.Px3.p1.1" class="ltx_p">All datasets show that
<span id="S5.SS1.SSS0.Px3.p1.1.1" class="ltx_text ltx_font_smallcaps">Vanilla</span>
outperforms
<span id="S5.SS1.SSS0.Px3.p1.1.2" class="ltx_text ltx_font_smallcaps">InlineSearch</span> on citation quality (and on correctness for ASQA and ELI5).
By manually examining the examples,
we find that
it is challenging to ask detailed questions without seeing any passages.
To improve <span id="S5.SS1.SSS0.Px3.p1.1.3" class="ltx_text ltx_font_smallcaps">InlineSearch</span>,
one may need to provide more context about the questions in advance or
encourage the model to call retrievers with more detailed and diverse queries.</p>
</div>
</section>
<section id="S5.SS1.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">
<span id="S5.SS1.SSS0.Px4.1.1" class="ltx_text ltx_font_smallcaps">Rerank</span> boosts citation quality.</h4>

<div id="S5.SS1.SSS0.Px4.p1" class="ltx_para">
<p id="S5.SS1.SSS0.Px4.p1.1" class="ltx_p">We observe that <span id="S5.SS1.SSS0.Px4.p1.1.1" class="ltx_text ltx_font_smallcaps">Rerank</span>
leads to consistent improvement in citation quality (on ASQA and ELI5).
As the automatic scores may be biased in <span id="S5.SS1.SSS0.Px4.p1.1.2" class="ltx_text ltx_font_smallcaps">Rerank</span>,
we also conduct human evaluation (§<a href="#S6" title="6 Human Evaluation ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>)
and verify its effectiveness.</p>
</div>
</section>
<section id="S5.SS1.SSS0.Px5" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">
<span id="S5.SS1.SSS0.Px5.1.1" class="ltx_text ltx_font_smallcaps">ClosedBook</span>+<span id="S5.SS1.SSS0.Px5.2.2" class="ltx_text ltx_font_smallcaps">PostCite</span> delivers strong correctness but poor citation quality.</h4>

<div id="S5.SS1.SSS0.Px5.p1" class="ltx_para">
<p id="S5.SS1.SSS0.Px5.p1.1" class="ltx_p"><span id="S5.SS1.SSS0.Px5.p1.1.1" class="ltx_text ltx_font_smallcaps">ClosedBook</span> outperforms <span id="S5.SS1.SSS0.Px5.p1.1.2" class="ltx_text ltx_font_smallcaps">Vanilla</span> in correctness on ELI5 and QAMPARI, and has only a 2% gap on ASQA.
However,
<span id="S5.SS1.SSS0.Px5.p1.1.3" class="ltx_text ltx_font_smallcaps">ClosedBook</span> cannot provide any citation;
when combined with <span id="S5.SS1.SSS0.Px5.p1.1.4" class="ltx_text ltx_font_smallcaps">PostCite</span>,
the citation quality remains inadequate. For instance,
citation recall of <span id="S5.SS1.SSS0.Px5.p1.1.5" class="ltx_text ltx_font_smallcaps">ClosedBook</span>+<span id="S5.SS1.SSS0.Px5.p1.1.6" class="ltx_text ltx_font_smallcaps">PostCite</span> is lower than <span id="S5.SS1.SSS0.Px5.p1.1.7" class="ltx_text ltx_font_smallcaps">Vanilla</span> by 47% on ASQA.</p>
</div>
<div id="S5.SS1.SSS0.Px5.p2" class="ltx_para">
<p id="S5.SS1.SSS0.Px5.p2.1" class="ltx_p">To understand why <span id="S5.SS1.SSS0.Px5.p2.1.1" class="ltx_text ltx_font_smallcaps">ClosedBook</span> achieves better correctness and why <span id="S5.SS1.SSS0.Px5.p2.1.2" class="ltx_text ltx_font_smallcaps">PostCite</span> cannot deliver satisfying citation quality,
we manually examine model outputs and find that:
(1) open-book models are easily distracted by irrelevant passages and generate responses with lower correctness,
a phenomenon also observed by <cite class="ltx_cite ltx_citemacro_citet">Shi et al. (<a href="#bib.bib42" title="" class="ltx_ref">2023</a>)</cite>;
(2) <span id="S5.SS1.SSS0.Px5.p2.1.3" class="ltx_text ltx_font_smallcaps">ClosedBook</span> often generates texts that are correct but not similar to any retrieved passages,
making it difficult to match a citation post-hoc.</p>
</div>
</section>
<section id="S5.SS1.SSS0.Px6" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">GPT-4 brings limited improvement but is better at using long context.</h4>

<div id="S5.SS1.SSS0.Px6.p1" class="ltx_para">
<p id="S5.SS1.SSS0.Px6.p1.1" class="ltx_p">We evaluate GPT-4 with <span id="S5.SS1.SSS0.Px6.p1.1.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> and different numbers of passages (more results in §<a href="#A7.SS6" title="G.6 Main Results ‣ Appendix G More Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">G.6</span></a>).
GPT-4 brings consistent (but limited) improvement on correctness, but often at a cost of citation quality.
GPT-4 can also incorporate more passages due to its longer context window,
which boosts both correctness and citation quality.
On the contrary,
including more passages with ChatGPT-16K does not improve the results (Table <a href="#S5.T7" title="Table 7 ‣ 5.3 Retrieval Analysis ‣ 5 Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>),
suggesting that processing more passages is non-trivial and GPT-4 is better at synthesizing information from its long context than ChatGPT.</p>
</div>
</section>
</section>
<section id="S5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.2 </span>Comparison of Different LLMs</h3>

<figure id="S5.F4" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="S5.F4.1" class="ltx_block ltx_figure_panel ltx_minipage ltx_align_center ltx_align_bottom" style="width:286.2pt;">
<img src="/html/2305.14627/assets/x4.png" id="S5.F4.1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="148" alt="Refer to caption">
</div>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="S5.F4.4" class="ltx_inline-block ltx_figure_panel ltx_minipage ltx_align_center ltx_align_bottom ltx_transformed_outer" style="width:143.1pt;height:225.6pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(93.8pt,-48.8pt) scale(1.76238261493942,1.76238261493942) ;">
<table id="S5.F4.4.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S5.F4.4.1.1.1" class="ltx_tr">
<th id="S5.F4.4.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="S5.F4.4.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S5.F4.4.1.1.1.2.1" class="ltx_text ltx_font_bold">Fluency</span></th>
<th id="S5.F4.4.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S5.F4.4.1.1.1.3.1" class="ltx_text ltx_font_bold">Correct.</span></th>
<th id="S5.F4.4.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2"><span id="S5.F4.4.1.1.1.4.1" class="ltx_text ltx_font_bold">Citation</span></th>
</tr>
<tr id="S5.F4.4.1.2.2" class="ltx_tr">
<th id="S5.F4.4.1.2.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="S5.F4.4.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">(MAUVE)</th>
<th id="S5.F4.4.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">(EM)</th>
<th id="S5.F4.4.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Rec.</th>
<th id="S5.F4.4.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Prec.</th>
</tr>
<tr id="S5.F4.4.1.3.3" class="ltx_tr" style="background-color:#ECECEC;">
<th id="S5.F4.4.1.3.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" colspan="5"><span id="S5.F4.4.1.3.3.1.1" class="ltx_text ltx_font_bold" style="background-color:#ECECEC;">ChatGPT<span id="S5.F4.4.1.3.3.1.1.1" class="ltx_text ltx_font_medium"> (<span id="S5.F4.4.1.3.3.1.1.1.1" class="ltx_text ltx_font_smallcaps">Vanilla</span>)</span></span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S5.F4.4.1.4.1" class="ltx_tr">
<th id="S5.F4.4.1.4.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">DPR (5-psg)</th>
<td id="S5.F4.4.1.4.1.2" class="ltx_td ltx_align_center">61.8</td>
<td id="S5.F4.4.1.4.1.3" class="ltx_td ltx_align_center">36.1</td>
<td id="S5.F4.4.1.4.1.4" class="ltx_td ltx_align_center">65.0</td>
<td id="S5.F4.4.1.4.1.5" class="ltx_td ltx_align_center">65.6</td>
</tr>
<tr id="S5.F4.4.1.5.2" class="ltx_tr">
<th id="S5.F4.4.1.5.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">GTR (1-psg)</th>
<td id="S5.F4.4.1.5.2.2" class="ltx_td ltx_align_center">69.5</td>
<td id="S5.F4.4.1.5.2.3" class="ltx_td ltx_align_center">38.4</td>
<td id="S5.F4.4.1.5.2.4" class="ltx_td ltx_align_center">56.0</td>
<td id="S5.F4.4.1.5.2.5" class="ltx_td ltx_align_center">64.0</td>
</tr>
<tr id="S5.F4.4.1.6.3" class="ltx_tr">
<th id="S5.F4.4.1.6.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">GTR (3-psg)</th>
<td id="S5.F4.4.1.6.3.2" class="ltx_td ltx_align_center">66.6</td>
<td id="S5.F4.4.1.6.3.3" class="ltx_td ltx_align_center">39.6</td>
<td id="S5.F4.4.1.6.3.4" class="ltx_td ltx_align_center">72.8</td>
<td id="S5.F4.4.1.6.3.5" class="ltx_td ltx_align_center"><span id="S5.F4.4.1.6.3.5.1" class="ltx_text ltx_font_bold">73.9</span></td>
</tr>
<tr id="S5.F4.4.1.7.4" class="ltx_tr">
<th id="S5.F4.4.1.7.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">GTR (5-psg)</th>
<td id="S5.F4.4.1.7.4.2" class="ltx_td ltx_align_center ltx_border_bb">66.6</td>
<td id="S5.F4.4.1.7.4.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.F4.4.1.7.4.3.1" class="ltx_text ltx_font_bold">40.4</span></td>
<td id="S5.F4.4.1.7.4.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.F4.4.1.7.4.4.1" class="ltx_text ltx_font_bold">73.6</span></td>
<td id="S5.F4.4.1.7.4.5" class="ltx_td ltx_align_center ltx_border_bb">72.5</td>
</tr>
</tbody>
</table>
</span></div>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>
Retrieval recall@<math id="S5.F4.3.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S5.F4.3.m1.1b"><mi id="S5.F4.3.m1.1.1" xref="S5.F4.3.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S5.F4.3.m1.1c"><ci id="S5.F4.3.m1.1.1.cmml" xref="S5.F4.3.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.F4.3.m1.1d">k</annotation></semantics></math> on ASQA (<span id="S5.F4.9.1" class="ltx_text ltx_font_italic">EM recall</span>), QAMPARI (<span id="S5.F4.10.2" class="ltx_text ltx_font_italic">recall-5</span>), and ELI5 (<span id="S5.F4.11.3" class="ltx_text ltx_font_italic">claim recall</span>).
Retrieval recall serves as an upper bound for model performance,
and we compare them with two models’ correctness results in the figure (dashed lines):
“Vanilla (5-psg)” is ChatGPT <span id="S5.F4.12.4" class="ltx_text ltx_font_smallcaps">Vanilla</span> with top-5 passages in context;
“Oracle” is the same model except that it uses 5 gold passages (§<a href="#A7.SS1" title="G.1 Retrieval Analysis ‣ Appendix G More Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">G.1</span></a>), whose recall matches Recall@100 on all three datasets. </figcaption>
</figure>
<div id="S5.SS2.p1" class="ltx_para">
<p id="S5.SS2.p1.1" class="ltx_p">Table <a href="#S5.T7" title="Table 7 ‣ 5.3 Retrieval Analysis ‣ 5 Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a> compares
different LLMs on ASQA using <span id="S5.SS2.p1.1.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (more results in §<a href="#A7.SS6" title="G.6 Main Results ‣ Appendix G More Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">G.6</span></a>).
Notably, instruction-tuned models (Vicuna-13B and LLaMA-2-Chat)
outperform the original LLaMA models in correctness and
considerably enhance the citation quality.
We observe that
while the original LLaMA models are able to copy facts from the context,
they struggle with accurately citing the sources or simply do not cite.
Notably, the best open-source model, LLaMA-2-70B-Chat, achieves comparable correctness score as the OpenAI models, but still lags behind in citation quality.</p>
</div>
</section>
<section id="S5.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.3 </span>Retrieval Analysis</h3>

<figure id="S5.T7" class="ltx_table">
<div id="S5.T7.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:424.9pt;height:497.5pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(68.1pt,-79.8pt) scale(1.47195482598643,1.47195482598643) ;">
<table id="S5.T7.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S5.T7.1.1.1.1" class="ltx_tr">
<th id="S5.T7.1.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="S5.T7.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S5.T7.1.1.1.1.2.1" class="ltx_text ltx_font_bold">Fluency</span></th>
<th id="S5.T7.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S5.T7.1.1.1.1.3.1" class="ltx_text ltx_font_bold">Correct.</span></th>
<th id="S5.T7.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2"><span id="S5.T7.1.1.1.1.4.1" class="ltx_text ltx_font_bold">Citation</span></th>
</tr>
<tr id="S5.T7.1.1.2.2" class="ltx_tr">
<th id="S5.T7.1.1.2.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="S5.T7.1.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">(MAUVE)</th>
<th id="S5.T7.1.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">(EM Rec.)</th>
<th id="S5.T7.1.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Rec.</th>
<th id="S5.T7.1.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Prec.</th>
</tr>
<tr id="S5.T7.1.1.3.3" class="ltx_tr" style="background-color:#ECECEC;">
<th id="S5.T7.1.1.3.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" colspan="5"><span id="S5.T7.1.1.3.3.1.1" class="ltx_text ltx_font_bold" style="background-color:#ECECEC;">Open-source<span id="S5.T7.1.1.3.3.1.1.1" class="ltx_text ltx_font_medium"> (max #tokens=2K-4K)</span></span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S5.T7.1.1.4.1" class="ltx_tr">
<th id="S5.T7.1.1.4.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-13B (3-psg)</th>
<td id="S5.T7.1.1.4.1.2" class="ltx_td ltx_align_center">68.4</td>
<td id="S5.T7.1.1.4.1.3" class="ltx_td ltx_align_center">26.9</td>
<td id="S5.T7.1.1.4.1.4" class="ltx_td ltx_align_center">10.6</td>
<td id="S5.T7.1.1.4.1.5" class="ltx_td ltx_align_center">15.4</td>
</tr>
<tr id="S5.T7.1.1.5.2" class="ltx_tr">
<th id="S5.T7.1.1.5.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Vicuna-13B (3-psg)</th>
<td id="S5.T7.1.1.5.2.2" class="ltx_td ltx_align_center">82.6</td>
<td id="S5.T7.1.1.5.2.3" class="ltx_td ltx_align_center">31.9</td>
<td id="S5.T7.1.1.5.2.4" class="ltx_td ltx_align_center">51.1</td>
<td id="S5.T7.1.1.5.2.5" class="ltx_td ltx_align_center">50.1</td>
</tr>
<tr id="S5.T7.1.1.6.3" class="ltx_tr">
<th id="S5.T7.1.1.6.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Chat-13B (5-psg)</th>
<td id="S5.T7.1.1.6.3.2" class="ltx_td ltx_align_center">72.4</td>
<td id="S5.T7.1.1.6.3.3" class="ltx_td ltx_align_center">35.2</td>
<td id="S5.T7.1.1.6.3.4" class="ltx_td ltx_align_center">38.4</td>
<td id="S5.T7.1.1.6.3.5" class="ltx_td ltx_align_center">39.4</td>
</tr>
<tr id="S5.T7.1.1.7.4" class="ltx_tr">
<th id="S5.T7.1.1.7.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Chat-70B (5-psg)</th>
<td id="S5.T7.1.1.7.4.2" class="ltx_td ltx_align_center">88.3</td>
<td id="S5.T7.1.1.7.4.3" class="ltx_td ltx_align_center">41.5</td>
<td id="S5.T7.1.1.7.4.4" class="ltx_td ltx_align_center">62.9</td>
<td id="S5.T7.1.1.7.4.5" class="ltx_td ltx_align_center">61.3</td>
</tr>
<tr id="S5.T7.1.1.8.5" class="ltx_tr" style="background-color:#ECECEC;">
<th id="S5.T7.1.1.8.5.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" colspan="5"><span id="S5.T7.1.1.8.5.1.1" class="ltx_text ltx_font_bold" style="background-color:#ECECEC;">ChatGPT<span id="S5.T7.1.1.8.5.1.1.1" class="ltx_text ltx_font_medium"> (max #tokens=4K)</span></span></th>
</tr>
<tr id="S5.T7.1.1.9.6" class="ltx_tr">
<th id="S5.T7.1.1.9.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">ChatGPT (3-psg)</th>
<td id="S5.T7.1.1.9.6.2" class="ltx_td ltx_align_center">66.6</td>
<td id="S5.T7.1.1.9.6.3" class="ltx_td ltx_align_center">39.6</td>
<td id="S5.T7.1.1.9.6.4" class="ltx_td ltx_align_center">72.8</td>
<td id="S5.T7.1.1.9.6.5" class="ltx_td ltx_align_center">73.9</td>
</tr>
<tr id="S5.T7.1.1.10.7" class="ltx_tr">
<th id="S5.T7.1.1.10.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">ChatGPT (5-psg)</th>
<td id="S5.T7.1.1.10.7.2" class="ltx_td ltx_align_center">66.6</td>
<td id="S5.T7.1.1.10.7.3" class="ltx_td ltx_align_center">40.4</td>
<td id="S5.T7.1.1.10.7.4" class="ltx_td ltx_align_center">73.6</td>
<td id="S5.T7.1.1.10.7.5" class="ltx_td ltx_align_center">72.5</td>
</tr>
<tr id="S5.T7.1.1.11.8" class="ltx_tr" style="background-color:#ECECEC;">
<th id="S5.T7.1.1.11.8.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" colspan="5"><span id="S5.T7.1.1.11.8.1.1" class="ltx_text ltx_font_bold" style="background-color:#ECECEC;">ChatGPT-16K<span id="S5.T7.1.1.11.8.1.1.1" class="ltx_text ltx_font_medium"> (max #tokens=16K)</span></span></th>
</tr>
<tr id="S5.T7.1.1.12.9" class="ltx_tr">
<th id="S5.T7.1.1.12.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">ChatGPT (5-psg)</th>
<td id="S5.T7.1.1.12.9.2" class="ltx_td ltx_align_center">60.3</td>
<td id="S5.T7.1.1.12.9.3" class="ltx_td ltx_align_center">36.1</td>
<td id="S5.T7.1.1.12.9.4" class="ltx_td ltx_align_center">76.2</td>
<td id="S5.T7.1.1.12.9.5" class="ltx_td ltx_align_center">76.5</td>
</tr>
<tr id="S5.T7.1.1.13.10" class="ltx_tr">
<th id="S5.T7.1.1.13.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">ChatGPT (10-psg)</th>
<td id="S5.T7.1.1.13.10.2" class="ltx_td ltx_align_center">56.3</td>
<td id="S5.T7.1.1.13.10.3" class="ltx_td ltx_align_center">36.7</td>
<td id="S5.T7.1.1.13.10.4" class="ltx_td ltx_align_center">75.3</td>
<td id="S5.T7.1.1.13.10.5" class="ltx_td ltx_align_center">75.0</td>
</tr>
<tr id="S5.T7.1.1.14.11" class="ltx_tr">
<th id="S5.T7.1.1.14.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">ChatGPT (20-psg)</th>
<td id="S5.T7.1.1.14.11.2" class="ltx_td ltx_align_center">56.7</td>
<td id="S5.T7.1.1.14.11.3" class="ltx_td ltx_align_center">36.1</td>
<td id="S5.T7.1.1.14.11.4" class="ltx_td ltx_align_center">73.7</td>
<td id="S5.T7.1.1.14.11.5" class="ltx_td ltx_align_center">73.5</td>
</tr>
<tr id="S5.T7.1.1.15.12" class="ltx_tr" style="background-color:#ECECEC;">
<th id="S5.T7.1.1.15.12.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" colspan="5"><span id="S5.T7.1.1.15.12.1.1" class="ltx_text ltx_font_bold" style="background-color:#ECECEC;">GPT-4<span id="S5.T7.1.1.15.12.1.1.1" class="ltx_text ltx_font_medium"> (max #tokens=8K)</span></span></th>
</tr>
<tr id="S5.T7.1.1.16.13" class="ltx_tr">
<th id="S5.T7.1.1.16.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">GPT-4 (5-psg)</th>
<td id="S5.T7.1.1.16.13.2" class="ltx_td ltx_align_center">67.1</td>
<td id="S5.T7.1.1.16.13.3" class="ltx_td ltx_align_center">41.3</td>
<td id="S5.T7.1.1.16.13.4" class="ltx_td ltx_align_center">68.5</td>
<td id="S5.T7.1.1.16.13.5" class="ltx_td ltx_align_center">75.6</td>
</tr>
<tr id="S5.T7.1.1.17.14" class="ltx_tr">
<th id="S5.T7.1.1.17.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">GPT-4 (10-psg)</th>
<td id="S5.T7.1.1.17.14.2" class="ltx_td ltx_align_center">71.5</td>
<td id="S5.T7.1.1.17.14.3" class="ltx_td ltx_align_center">43.1</td>
<td id="S5.T7.1.1.17.14.4" class="ltx_td ltx_align_center">72.0</td>
<td id="S5.T7.1.1.17.14.5" class="ltx_td ltx_align_center">75.5</td>
</tr>
<tr id="S5.T7.1.1.18.15" class="ltx_tr">
<th id="S5.T7.1.1.18.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">GPT-4 (20-psg)</th>
<td id="S5.T7.1.1.18.15.2" class="ltx_td ltx_align_center ltx_border_bb">64.9</td>
<td id="S5.T7.1.1.18.15.3" class="ltx_td ltx_align_center ltx_border_bb">44.4</td>
<td id="S5.T7.1.1.18.15.4" class="ltx_td ltx_align_center ltx_border_bb">73.0</td>
<td id="S5.T7.1.1.18.15.5" class="ltx_td ltx_align_center ltx_border_bb">76.5</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 7: </span>
Comparison of different LLMs on ASQA (GTR+<span id="S5.T7.3.1" class="ltx_text ltx_font_smallcaps">Vanilla</span>). LLaMA-13B and Vicuna-13B have a context limit of 2,048 tokens, and thus can only use a short version of instructions and at most top-3 passages.
Chat-13B and Chat-70B refer to LLaMA-2-Chat.
</figcaption>
</figure>
<div id="S5.SS3.p1" class="ltx_para">
<p id="S5.SS3.p1.1" class="ltx_p">The retrieval results play a crucial role to the correctness and the citation quality.
Figure <a href="#S5.F4" title="Figure 4 ‣ 5.2 Comparison of Different LLMs ‣ 5 Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> presents
the retrieval recall@<math id="S5.SS3.p1.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S5.SS3.p1.1.m1.1a"><mi id="S5.SS3.p1.1.m1.1.1" xref="S5.SS3.p1.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S5.SS3.p1.1.m1.1b"><ci id="S5.SS3.p1.1.m1.1.1.cmml" xref="S5.SS3.p1.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p1.1.m1.1c">k</annotation></semantics></math> with different datasets and retrievers.
As the number of passages increases, retrieval recall steadily improves.
Additionally, Figure <a href="#S5.F4" title="Figure 4 ‣ 5.2 Comparison of Different LLMs ‣ 5 Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> shows
the correctness performance of two models:
(1) ChatGPT <span id="S5.SS3.p1.1.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> with top-5 passages (our primary baseline);
(2) an oracle version of the same model employing
5 gold passages (§<a href="#A7.SS1" title="G.1 Retrieval Analysis ‣ Appendix G More Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">G.1</span></a>; the 5 gold passages match the retrieval recall@100).
Notably,
both models’ correctness lags behind the corresponding retrieval recall (except for ELI5 top-5).
The discrepancy suggests that
despite the presence of accurate answers in context,
LLMs struggle to utilize them in their outputs.</p>
</div>
<div id="S5.SS3.p2" class="ltx_para">
<p id="S5.SS3.p2.1" class="ltx_p">We compare the impact of different retrievers and different numbers of passages to LLMs.
Figure <a href="#S5.F4" title="Figure 4 ‣ 5.2 Comparison of Different LLMs ‣ 5 Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> (right)
shows that
GTR outperforms DPR in both correctness and citation quality,
emphasizing the importance of deploying better retrievers.
Contrary to the retrieval recall trend in Figure <a href="#S5.F4" title="Figure 4 ‣ 5.2 Comparison of Different LLMs ‣ 5 Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>,
more passages in context do not yield substantial improvement for ChatGPT.
Specifically, correctness plateaus at top-1 passage and citation quality plateaus at top-3.
GPT-4 (Table <a href="#S5.T7" title="Table 7 ‣ 5.3 Retrieval Analysis ‣ 5 Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>) exhibits an increasing trend with more passages,
but the improvement is not proportional to the retrieval performance.
This indicates the limited ability of LLMs in utilizing multiple passages within context.</p>
</div>
</section>
<section id="S5.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.4 </span>Other Ablations</h3>

<div id="S5.SS4.p1" class="ltx_para">
<p id="S5.SS4.p1.1" class="ltx_p">We provide additional ablations in §<a href="#A7" title="Appendix G More Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">G</span></a>.
In summary, we find that
(1)
using comprehensive instructions
enhances the citation quality of instruction-tuned models (§<a href="#A7.SS2" title="G.2 Effect of Instructions ‣ Appendix G More Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">G.2</span></a>);
(2)
including at least one demonstration improves the performance (§<a href="#A7.SS3" title="G.3 Effect of Demonstrations ‣ Appendix G More Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">G.3</span></a>);
(3) fine-tuned models (FiD; <cite class="ltx_cite ltx_citemacro_citep">Izacard and Grave, <a href="#bib.bib13" title="" class="ltx_ref">2021</a></cite>) with <span id="S5.SS4.p1.1.1" class="ltx_text ltx_font_smallcaps">PostCite</span> lag behind LLMs in both correctness and citation quality and fail to generalize (§<a href="#A7.SS4" title="G.4 Fine-tuned Models ‣ Appendix G More Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">G.4</span></a>).</p>
</div>
</section>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Human Evaluation</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">To verify that our automatic evaluation correlates with human judgement,
we conduct human evaluation on selected models and request workers
to judge model generations on three dimensions similar to <cite class="ltx_cite ltx_citemacro_citet">Liu et al. (<a href="#bib.bib24" title="" class="ltx_ref">2023</a>)</cite>—(1) utility: a 1-to-5 score indicating whether the generation helps answer the question; (2) citation recall: the annotator is given a sentence and all passages that the sentence cited, and is asked to judge whether the passages fully support the sentence; (3) citation precision: given a sentence and one of its citations, the annotator is asked to judge whether the citation “fully supports”, “partially supports”, or “does not support” the sentence.
Each citation gets a precision score 1 if the output sentence has a citation recall of 1 and this citation at least “partially supports” it.
See Appendix <a href="#A6" title="Appendix F Human Evaluation ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">F</span></a> for more details.</p>
</div>
<figure id="S6.T8" class="ltx_table">
<div id="S6.T8.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:424.9pt;height:158.5pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(91.8pt,-34.2pt) scale(1.76107545758693,1.76107545758693) ;">
<table id="S6.T8.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S6.T8.1.1.1.1" class="ltx_tr">
<th id="S6.T8.1.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="S6.T8.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2"><span id="S6.T8.1.1.1.1.2.1" class="ltx_text ltx_font_bold">Human scores</span></th>
<th id="S6.T8.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2"><span id="S6.T8.1.1.1.1.3.1" class="ltx_text ltx_font_bold">ALCE scores</span></th>
</tr>
<tr id="S6.T8.1.1.2.2" class="ltx_tr">
<th id="S6.T8.1.1.2.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="S6.T8.1.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Rec.</th>
<th id="S6.T8.1.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Prec.</th>
<th id="S6.T8.1.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Rec.</th>
<th id="S6.T8.1.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Prec.</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S6.T8.1.1.3.1" class="ltx_tr">
<th id="S6.T8.1.1.3.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">ChatGPT <span id="S6.T8.1.1.3.1.1.1" class="ltx_text ltx_font_smallcaps">Vanilla</span>
</th>
<td id="S6.T8.1.1.3.1.2" class="ltx_td ltx_align_center ltx_border_t">74.7</td>
<td id="S6.T8.1.1.3.1.3" class="ltx_td ltx_align_center ltx_border_t">76.6</td>
<td id="S6.T8.1.1.3.1.4" class="ltx_td ltx_align_center ltx_border_t">75.3</td>
<td id="S6.T8.1.1.3.1.5" class="ltx_td ltx_align_center ltx_border_t">74.4</td>
</tr>
<tr id="S6.T8.1.1.4.2" class="ltx_tr">
<th id="S6.T8.1.1.4.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">w/ <span id="S6.T8.1.1.4.2.1.1" class="ltx_text ltx_font_smallcaps">Rerank</span>
</th>
<td id="S6.T8.1.1.4.2.2" class="ltx_td ltx_align_center"><span id="S6.T8.1.1.4.2.2.1" class="ltx_text ltx_font_bold">79.3</span></td>
<td id="S6.T8.1.1.4.2.3" class="ltx_td ltx_align_center"><span id="S6.T8.1.1.4.2.3.1" class="ltx_text ltx_font_bold">81.9</span></td>
<td id="S6.T8.1.1.4.2.4" class="ltx_td ltx_align_center"><span id="S6.T8.1.1.4.2.4.1" class="ltx_text ltx_font_bold">83.9</span></td>
<td id="S6.T8.1.1.4.2.5" class="ltx_td ltx_align_center"><span id="S6.T8.1.1.4.2.5.1" class="ltx_text ltx_font_bold">80.8</span></td>
</tr>
<tr id="S6.T8.1.1.5.3" class="ltx_tr">
<th id="S6.T8.1.1.5.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">Vicuna-13B <span id="S6.T8.1.1.5.3.1.1" class="ltx_text ltx_font_smallcaps">Vanilla</span>
</th>
<td id="S6.T8.1.1.5.3.2" class="ltx_td ltx_align_center ltx_border_bb">51.6</td>
<td id="S6.T8.1.1.5.3.3" class="ltx_td ltx_align_center ltx_border_bb">51.5</td>
<td id="S6.T8.1.1.5.3.4" class="ltx_td ltx_align_center ltx_border_bb">50.3</td>
<td id="S6.T8.1.1.5.3.5" class="ltx_td ltx_align_center ltx_border_bb">50.1</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 8: </span>
Human citation quality evaluation vs. ALCE citation quality evaluation on ASQA.
</figcaption>
</figure>
<figure id="S6.T9" class="ltx_table">
<div id="S6.T9.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:424.9pt;height:158.5pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(91.8pt,-34.2pt) scale(1.76107545758693,1.76107545758693) ;">
<table id="S6.T9.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S6.T9.1.1.1.1" class="ltx_tr">
<th id="S6.T9.1.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="S6.T9.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2"><span id="S6.T9.1.1.1.1.2.1" class="ltx_text ltx_font_bold">Human scores</span></th>
<th id="S6.T9.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2"><span id="S6.T9.1.1.1.1.3.1" class="ltx_text ltx_font_bold">ALCE scores</span></th>
</tr>
<tr id="S6.T9.1.1.2.2" class="ltx_tr">
<th id="S6.T9.1.1.2.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="S6.T9.1.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Rec.</th>
<th id="S6.T9.1.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Prec.</th>
<th id="S6.T9.1.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Rec.</th>
<th id="S6.T9.1.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Prec.</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S6.T9.1.1.3.1" class="ltx_tr">
<th id="S6.T9.1.1.3.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">ChatGPT <span id="S6.T9.1.1.3.1.1.1" class="ltx_text ltx_font_smallcaps">Vanilla</span>
</th>
<td id="S6.T9.1.1.3.1.2" class="ltx_td ltx_align_center ltx_border_t">50.8</td>
<td id="S6.T9.1.1.3.1.3" class="ltx_td ltx_align_center ltx_border_t">52.4</td>
<td id="S6.T9.1.1.3.1.4" class="ltx_td ltx_align_center ltx_border_t">52.8</td>
<td id="S6.T9.1.1.3.1.5" class="ltx_td ltx_align_center ltx_border_t">50.4</td>
</tr>
<tr id="S6.T9.1.1.4.2" class="ltx_tr">
<th id="S6.T9.1.1.4.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">w/ <span id="S6.T9.1.1.4.2.1.1" class="ltx_text ltx_font_smallcaps">Rerank</span>
</th>
<td id="S6.T9.1.1.4.2.2" class="ltx_td ltx_align_center"><span id="S6.T9.1.1.4.2.2.1" class="ltx_text ltx_font_bold">59.7</span></td>
<td id="S6.T9.1.1.4.2.3" class="ltx_td ltx_align_center"><span id="S6.T9.1.1.4.2.3.1" class="ltx_text ltx_font_bold">60.6</span></td>
<td id="S6.T9.1.1.4.2.4" class="ltx_td ltx_align_center"><span id="S6.T9.1.1.4.2.4.1" class="ltx_text ltx_font_bold">63.0</span></td>
<td id="S6.T9.1.1.4.2.5" class="ltx_td ltx_align_center"><span id="S6.T9.1.1.4.2.5.1" class="ltx_text ltx_font_bold">60.6</span></td>
</tr>
<tr id="S6.T9.1.1.5.3" class="ltx_tr">
<th id="S6.T9.1.1.5.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">Vicuna-13B <span id="S6.T9.1.1.5.3.1.1" class="ltx_text ltx_font_smallcaps">Vanilla</span>
</th>
<td id="S6.T9.1.1.5.3.2" class="ltx_td ltx_align_center ltx_border_bb">13.4</td>
<td id="S6.T9.1.1.5.3.3" class="ltx_td ltx_align_center ltx_border_bb">19.2</td>
<td id="S6.T9.1.1.5.3.4" class="ltx_td ltx_align_center ltx_border_bb">13.6</td>
<td id="S6.T9.1.1.5.3.5" class="ltx_td ltx_align_center ltx_border_bb">18.1</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 9: </span>
Human citation quality evaluation vs. ALCE citation quality evaluation on ELI5.
</figcaption>
</figure>
<section id="S6.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Model outputs score high utility.</h4>

<div id="S6.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S6.SS0.SSS0.Px1.p1.4" class="ltx_p">The utility scores do not differ significantly between models, ranging <math id="S6.SS0.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="3.7" display="inline"><semantics id="S6.SS0.SSS0.Px1.p1.1.m1.1a"><mn id="S6.SS0.SSS0.Px1.p1.1.m1.1.1" xref="S6.SS0.SSS0.Px1.p1.1.m1.1.1.cmml">3.7</mn><annotation-xml encoding="MathML-Content" id="S6.SS0.SSS0.Px1.p1.1.m1.1b"><cn type="float" id="S6.SS0.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S6.SS0.SSS0.Px1.p1.1.m1.1.1">3.7</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS0.SSS0.Px1.p1.1.m1.1c">3.7</annotation></semantics></math>-<math id="S6.SS0.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="3.9" display="inline"><semantics id="S6.SS0.SSS0.Px1.p1.2.m2.1a"><mn id="S6.SS0.SSS0.Px1.p1.2.m2.1.1" xref="S6.SS0.SSS0.Px1.p1.2.m2.1.1.cmml">3.9</mn><annotation-xml encoding="MathML-Content" id="S6.SS0.SSS0.Px1.p1.2.m2.1b"><cn type="float" id="S6.SS0.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S6.SS0.SSS0.Px1.p1.2.m2.1.1">3.9</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS0.SSS0.Px1.p1.2.m2.1c">3.9</annotation></semantics></math> for ASQA and <math id="S6.SS0.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="3.5" display="inline"><semantics id="S6.SS0.SSS0.Px1.p1.3.m3.1a"><mn id="S6.SS0.SSS0.Px1.p1.3.m3.1.1" xref="S6.SS0.SSS0.Px1.p1.3.m3.1.1.cmml">3.5</mn><annotation-xml encoding="MathML-Content" id="S6.SS0.SSS0.Px1.p1.3.m3.1b"><cn type="float" id="S6.SS0.SSS0.Px1.p1.3.m3.1.1.cmml" xref="S6.SS0.SSS0.Px1.p1.3.m3.1.1">3.5</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS0.SSS0.Px1.p1.3.m3.1c">3.5</annotation></semantics></math>-<math id="S6.SS0.SSS0.Px1.p1.4.m4.1" class="ltx_Math" alttext="3.6" display="inline"><semantics id="S6.SS0.SSS0.Px1.p1.4.m4.1a"><mn id="S6.SS0.SSS0.Px1.p1.4.m4.1.1" xref="S6.SS0.SSS0.Px1.p1.4.m4.1.1.cmml">3.6</mn><annotation-xml encoding="MathML-Content" id="S6.SS0.SSS0.Px1.p1.4.m4.1b"><cn type="float" id="S6.SS0.SSS0.Px1.p1.4.m4.1.1.cmml" xref="S6.SS0.SSS0.Px1.p1.4.m4.1.1">3.6</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS0.SSS0.Px1.p1.4.m4.1c">3.6</annotation></semantics></math> for ELI5.
Upon inspection, all tested models are mostly able to output fluent answers that are related to the question, despite differences in factual correctness.</p>
</div>
</section>
<section id="S6.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Our automatic evaluation of citation quality strongly correlates with human judgements.</h4>

<div id="S6.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S6.SS0.SSS0.Px2.p1.4" class="ltx_p">As shown in Table <a href="#S6.T8" title="Table 8 ‣ 6 Human Evaluation ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a> (ASQA) and Table <a href="#S6.T9" title="Table 9 ‣ 6 Human Evaluation ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a> (ELI5),
the relative rankings induced by human and our automatic metrics are consistent.
The absolute citation scores from human and ALCE are very close except for <span id="S6.SS0.SSS0.Px2.p1.4.1" class="ltx_text ltx_font_smallcaps">Rerank</span> (which uses the automated citation recall for reranking).
This suggests that an improvement on ALCE citation metrics translates to improvement on human preferences. Furthermore, the Cohen’s kappa coefficient between
human and ALCE suggests substantial agreement for citation recall (<math id="S6.SS0.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="0.698" display="inline"><semantics id="S6.SS0.SSS0.Px2.p1.1.m1.1a"><mn id="S6.SS0.SSS0.Px2.p1.1.m1.1.1" xref="S6.SS0.SSS0.Px2.p1.1.m1.1.1.cmml">0.698</mn><annotation-xml encoding="MathML-Content" id="S6.SS0.SSS0.Px2.p1.1.m1.1b"><cn type="float" id="S6.SS0.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S6.SS0.SSS0.Px2.p1.1.m1.1.1">0.698</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS0.SSS0.Px2.p1.1.m1.1c">0.698</annotation></semantics></math>) and moderate agreement for citation precision (<math id="S6.SS0.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="0.525" display="inline"><semantics id="S6.SS0.SSS0.Px2.p1.2.m2.1a"><mn id="S6.SS0.SSS0.Px2.p1.2.m2.1.1" xref="S6.SS0.SSS0.Px2.p1.2.m2.1.1.cmml">0.525</mn><annotation-xml encoding="MathML-Content" id="S6.SS0.SSS0.Px2.p1.2.m2.1b"><cn type="float" id="S6.SS0.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S6.SS0.SSS0.Px2.p1.2.m2.1.1">0.525</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS0.SSS0.Px2.p1.2.m2.1c">0.525</annotation></semantics></math>).
We also show in §<a href="#A7.SS5" title="G.5 More Human Evaluation ‣ Appendix G More Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">G.5</span></a> that our automatic evaluation achieves high accuracy when treating human annotations as gold labels
(<math id="S6.SS0.SSS0.Px2.p1.3.m3.1" class="ltx_Math" alttext="85.1\%" display="inline"><semantics id="S6.SS0.SSS0.Px2.p1.3.m3.1a"><mrow id="S6.SS0.SSS0.Px2.p1.3.m3.1.1" xref="S6.SS0.SSS0.Px2.p1.3.m3.1.1.cmml"><mn id="S6.SS0.SSS0.Px2.p1.3.m3.1.1.2" xref="S6.SS0.SSS0.Px2.p1.3.m3.1.1.2.cmml">85.1</mn><mo id="S6.SS0.SSS0.Px2.p1.3.m3.1.1.1" xref="S6.SS0.SSS0.Px2.p1.3.m3.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S6.SS0.SSS0.Px2.p1.3.m3.1b"><apply id="S6.SS0.SSS0.Px2.p1.3.m3.1.1.cmml" xref="S6.SS0.SSS0.Px2.p1.3.m3.1.1"><csymbol cd="latexml" id="S6.SS0.SSS0.Px2.p1.3.m3.1.1.1.cmml" xref="S6.SS0.SSS0.Px2.p1.3.m3.1.1.1">percent</csymbol><cn type="float" id="S6.SS0.SSS0.Px2.p1.3.m3.1.1.2.cmml" xref="S6.SS0.SSS0.Px2.p1.3.m3.1.1.2">85.1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS0.SSS0.Px2.p1.3.m3.1c">85.1\%</annotation></semantics></math> for citation recall and <math id="S6.SS0.SSS0.Px2.p1.4.m4.1" class="ltx_Math" alttext="77.6\%" display="inline"><semantics id="S6.SS0.SSS0.Px2.p1.4.m4.1a"><mrow id="S6.SS0.SSS0.Px2.p1.4.m4.1.1" xref="S6.SS0.SSS0.Px2.p1.4.m4.1.1.cmml"><mn id="S6.SS0.SSS0.Px2.p1.4.m4.1.1.2" xref="S6.SS0.SSS0.Px2.p1.4.m4.1.1.2.cmml">77.6</mn><mo id="S6.SS0.SSS0.Px2.p1.4.m4.1.1.1" xref="S6.SS0.SSS0.Px2.p1.4.m4.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S6.SS0.SSS0.Px2.p1.4.m4.1b"><apply id="S6.SS0.SSS0.Px2.p1.4.m4.1.1.cmml" xref="S6.SS0.SSS0.Px2.p1.4.m4.1.1"><csymbol cd="latexml" id="S6.SS0.SSS0.Px2.p1.4.m4.1.1.1.cmml" xref="S6.SS0.SSS0.Px2.p1.4.m4.1.1.1">percent</csymbol><cn type="float" id="S6.SS0.SSS0.Px2.p1.4.m4.1.1.2.cmml" xref="S6.SS0.SSS0.Px2.p1.4.m4.1.1.2">77.6</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS0.SSS0.Px2.p1.4.m4.1c">77.6\%</annotation></semantics></math> for citation precision).</p>
</div>
</section>
</section>
<section id="S7" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">7 </span>Related Work</h2>

<section id="S7.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Evaluating citations.</h4>

<div id="S7.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S7.SS0.SSS0.Px1.p1.1" class="ltx_p">Generating text with citations is closely related to attribution.
<cite class="ltx_cite ltx_citemacro_citet">Rashkin et al. (<a href="#bib.bib38" title="" class="ltx_ref">2023</a>)</cite> define the “attributable to identified sources” (AIS) score to measure how faithful a generated text is to its sources.
<cite class="ltx_cite ltx_citemacro_citet">Bohnet et al. (<a href="#bib.bib1" title="" class="ltx_ref">2022</a>)</cite> apply AIS scores on a single-document short-answer QA dataset.
<cite class="ltx_cite ltx_citemacro_citet">Honovich et al. (<a href="#bib.bib12" title="" class="ltx_ref">2022</a>); Yue et al. (<a href="#bib.bib52" title="" class="ltx_ref">2023</a>)</cite> study automatic evaluations for the AIS score.
A concurrent work <cite class="ltx_cite ltx_citemacro_citep">(Liu et al., <a href="#bib.bib24" title="" class="ltx_ref">2023</a>)</cite>
conduct human evaluation on commercial generative search engines to
examine their citation qualities.</p>
</div>
<div id="S7.SS0.SSS0.Px1.p2" class="ltx_para">
<p id="S7.SS0.SSS0.Px1.p2.1" class="ltx_p">Scientific citation text generation <cite class="ltx_cite ltx_citemacro_citep">(Funkquist et al., <a href="#bib.bib7" title="" class="ltx_ref">2022</a>)</cite> is a related task to ALCE where the model is provided the papers-to-cite and context and is required to recover the citing text.
It is different from ALCE
as all citations are provided
and the model only needs to perform the summarization.</p>
</div>
</section>
<section id="S7.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Retrieval-augmented LMs.</h4>

<div id="S7.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S7.SS0.SSS0.Px2.p1.1" class="ltx_p">Many studies have explored augmenting LMs with
externally retrieved information.
<cite class="ltx_cite ltx_citemacro_citet">Guu et al. (<a href="#bib.bib9" title="" class="ltx_ref">2020</a>); Borgeaud et al. (<a href="#bib.bib2" title="" class="ltx_ref">2022</a>); Izacard et al. (<a href="#bib.bib14" title="" class="ltx_ref">2022</a>)</cite>
pre-train language models with retrieved passages,
while <cite class="ltx_cite ltx_citemacro_citet">Khandelwal et al. (<a href="#bib.bib19" title="" class="ltx_ref">2020</a>); Zhong et al. (<a href="#bib.bib55" title="" class="ltx_ref">2022</a>)</cite> augment
LLMs’ output by interpolating it with a <math id="S7.SS0.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S7.SS0.SSS0.Px2.p1.1.m1.1a"><mi id="S7.SS0.SSS0.Px2.p1.1.m1.1.1" xref="S7.SS0.SSS0.Px2.p1.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S7.SS0.SSS0.Px2.p1.1.m1.1b"><ci id="S7.SS0.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S7.SS0.SSS0.Px2.p1.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S7.SS0.SSS0.Px2.p1.1.m1.1c">k</annotation></semantics></math>NN module;
though none of them explicitly provide citations to the retrieved sources.
Other works prompt or fine-tune LLMs to
“retrieve on-the-fly” <cite class="ltx_cite ltx_citemacro_citep">(Parisi et al., <a href="#bib.bib31" title="" class="ltx_ref">2022</a>; Schick et al., <a href="#bib.bib40" title="" class="ltx_ref">2023</a>; Shuster et al., <a href="#bib.bib43" title="" class="ltx_ref">2022</a>; Jiang et al., <a href="#bib.bib16" title="" class="ltx_ref">2023</a>; Yao et al., <a href="#bib.bib51" title="" class="ltx_ref">2023</a>; Press et al., <a href="#bib.bib35" title="" class="ltx_ref">2022</a>)</cite>,
which offers flexibility of when and what to search.
<cite class="ltx_cite ltx_citemacro_citet">Gao et al. (<a href="#bib.bib8" title="" class="ltx_ref">2023</a>); He et al. (<a href="#bib.bib10" title="" class="ltx_ref">2022</a>)</cite> propose to first generate text without accessing external documents and then
retrieve relevant documents and revise the generation to be consistent.</p>
</div>
<div id="S7.SS0.SSS0.Px2.p2" class="ltx_para">
<p id="S7.SS0.SSS0.Px2.p2.1" class="ltx_p">Among previous explorations, <cite class="ltx_cite ltx_citemacro_citet">Nakano et al. (<a href="#bib.bib27" title="" class="ltx_ref">2021</a>); Menick et al. (<a href="#bib.bib25" title="" class="ltx_ref">2022</a>)</cite>
are the closest to our setting, where
LLMs are trained to answer questions while providing citations.
However, they do not explore retrieval strategies and simply use commercial search engines, which are not reproducible, and their models and training data are closed-source.
To the best of our knowledge,
we are the first to implement end-to-end systems that retrieve, synthesize, and cite documents with LLMs.</p>
</div>
</section>
</section>
<section id="S8" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">8 </span>Conclusion</h2>

<div id="S8.p1" class="ltx_para">
<p id="S8.p1.1" class="ltx_p">We propose ALCE, the first automatic benchmark for evaluating LLM generations with citations.
We deploy automatic metrics to measure fluency, correctness, and citation quality, and verify their efficacy via human evaluation.
We explore a variety of strategies for incorporating citations in LLMs
and demonstrate that current systems have considerable room for improvement on ALCE.</p>
</div>
<div id="S8.p2" class="ltx_para">
<p id="S8.p2.1" class="ltx_p">Our experiments highlight a number of promising research directions, including
(1) enhancing retrieval and refining retrieval integrations in LLMs,
(2) developing long-context LLMs, and
(3) advancing LLMs’ ability to synthesize multiple sources.
What’s even more intriguing is that these research proposals extend beyond the ALCE setup (for example, long-context LLMs have numerous exciting applications), and
ALCE can serve as a valuable testbed for their development.</p>
</div>
</section>
<section id="Sx1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Limitations</h2>

<div id="Sx1.p1" class="ltx_para">
<p id="Sx1.p1.1" class="ltx_p">Our evaluation still has room for improvement:
(1) MAUVE is found to be sensitive to output length and may provide unstable results;
(2) for the ELI5’s correctness evaluation, the automatically generated claims may not cover all possible answers due to the open-ended nature of the questions;
(3) our citation quality evaluation is limited by the accuracy of the NLI model; for citation precision, the NLI model cannot detect the case of “partially support” and thus
leads to a lower citation precision score than the human evaluation.</p>
</div>
<div id="Sx1.p2" class="ltx_para">
<p id="Sx1.p2.1" class="ltx_p">Although we believe our curated datasets closely resemble the distribution of real-world user questions,
we acknowledge that they do not cover more challenging scenarios, such as
multi-hop reasoning, math reasoning, and code completion.</p>
</div>
<div id="Sx1.p3" class="ltx_para">
<p id="Sx1.p3.1" class="ltx_p">In our experiments, we focus on prompting LLMs without updating their model weights.
Training a model directly to incorporate citations remains challenging due to the lack of supervised data.
However, we observe that certain human-instruction datasets
contain examples similar to our task setup.
We leave the exploration of training LLMs to generate citations for future work.</p>
</div>
</section>
<section id="Sx2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Acknowledgments</h2>

<div id="Sx2.p1" class="ltx_para">
<p id="Sx2.p1.1" class="ltx_p">We appreciate the helpful feedback from the members of the Princeton NLP group. We thank Alexander Wettig, Nelson Liu, Tianyi Zhang, Yu Meng, Sadhika Malladi, Yangsibo Huang, Zhiyuan Zeng, and Dan Friedman for the valuable discussion.
We thank Surge AI (especially Anna Folinsky and Edwin Chen)
for their support with the human evaluation.
Tianyu Gao is supported by an IBM PhD Fellowship.
This research is supported by an NSF CAREER award (IIS-2239290), a Sloan Research Fellowship,
and Microsoft Azure credits through the “Accelerate Foundation Models Academic Research” Initiative.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bohnet et al. (2022)</span>
<span class="ltx_bibblock">
Bernd Bohnet, Vinh Q Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, et al. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/pdf/2212.08037.pdf" title="" class="ltx_ref ltx_href">Attributed question answering: Evaluation and modeling for attributed large language models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2212.08037</em>.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Borgeaud et al. (2022)</span>
<span class="ltx_bibblock">
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://proceedings.mlr.press/v162/borgeaud22a.html" title="" class="ltx_ref ltx_href">Improving language models by retrieving from trillions of tokens</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning (ICML)</em>, volume 162, pages 2206–2240.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bowman et al. (2015)</span>
<span class="ltx_bibblock">
Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://aclanthology.org/D15-1075.pdf" title="" class="ltx_ref ltx_href">A large annotated corpus for learning natural language inference</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">Empirical Methods in Natural Language Processing (EMNLP)</em>.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Brown et al. (2020)</span>
<span class="ltx_bibblock">
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://papers.nips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html" title="" class="ltx_ref ltx_href">Language models are few-shot learners</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems (NeurIPS)</em>.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chiang et al. (2023)</span>
<span class="ltx_bibblock">
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://lmsys.org/blog/2023-03-30-vicuna/" title="" class="ltx_ref ltx_href">Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality</a>.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Fan et al. (2019)</span>
<span class="ltx_bibblock">
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/P19-1346" title="" class="ltx_ref ltx_href">ELI5: Long form question answering</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">Association for Computational Linguistics (ACL)</em>, pages 3558–3567.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Funkquist et al. (2022)</span>
<span class="ltx_bibblock">
Martin Funkquist, Ilia Kuznetsov, Yufang Hou, and Iryna Gurevych. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/pdf/2212.09577" title="" class="ltx_ref ltx_href">CiteBench: A benchmark for Scientific Citation Text Generation</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2212.09577</em>.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gao et al. (2023)</span>
<span class="ltx_bibblock">
Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, et al. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/pdf/2210.08726.pdf" title="" class="ltx_ref ltx_href">RARR: Researching and revising what language models say, using language models</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">Association for Computational Linguistics (ACL)</em>.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Guu et al. (2020)</span>
<span class="ltx_bibblock">
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/pdf/2002.08909.pdf" title="" class="ltx_ref ltx_href">REALM: Retrieval-augmented language model pre-training</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning (ICML)</em>.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">He et al. (2022)</span>
<span class="ltx_bibblock">
Hangfeng He, Hongming Zhang, and Dan Roth. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/pdf/2301.00303.pdf" title="" class="ltx_ref ltx_href">Rethinking with retrieval: Faithful large language model inference</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2301.00303</em>.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Holtzman et al. (2020)</span>
<span class="ltx_bibblock">
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=rygGQyrFvH" title="" class="ltx_ref ltx_href">The curious case of neural text degeneration</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations (ICLR)</em>.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Honovich et al. (2022)</span>
<span class="ltx_bibblock">
Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2022.naacl-main.287" title="" class="ltx_ref ltx_href">TRUE: Re-evaluating factual consistency evaluation</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)</em>, pages 3905–3920.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Izacard and Grave (2021)</span>
<span class="ltx_bibblock">
Gautier Izacard and Edouard Grave. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2021.eacl-main.74" title="" class="ltx_ref ltx_href">Leveraging passage retrieval with generative models for open domain question answering</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume</em>, pages 874–880, Online. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Izacard et al. (2022)</span>
<span class="ltx_bibblock">
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/pdf/2208.03299.pdf" title="" class="ltx_ref ltx_href">Atlas: Few-shot learning with retrieval augmented language models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2208.03299</em>.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ji et al. (2023)</span>
<span class="ltx_bibblock">
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://dl.acm.org/doi/pdf/10.1145/3571730" title="" class="ltx_ref ltx_href">Survey of hallucination in natural language generation</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">ACM Computing Surveys</em>, 55(12):1–38.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jiang et al. (2023)</span>
<span class="ltx_bibblock">
Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/pdf/2305.06983" title="" class="ltx_ref ltx_href">Active retrieval augmented generation</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.06983</em>.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kamoi et al. (2023)</span>
<span class="ltx_bibblock">
Ryo Kamoi, Tanya Goyal, Juan Diego Rodriguez, and Greg Durrett. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2303.01432" title="" class="ltx_ref ltx_href">WiCE: Real-World Entailment for Claims in Wikipedia</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2303.01432</em>.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Karpukhin et al. (2020)</span>
<span class="ltx_bibblock">
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2020.emnlp-main.550" title="" class="ltx_ref ltx_href">Dense passage retrieval for open-domain question answering</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)</em>, pages 6769–6781, Online. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Khandelwal et al. (2020)</span>
<span class="ltx_bibblock">
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=HklBjCEKvH" title="" class="ltx_ref ltx_href">Generalization through memorization: Nearest neighbor language models</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations (ICLR)</em>.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Khot et al. (2018)</span>
<span class="ltx_bibblock">
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://ojs.aaai.org/index.php/AAAI/article/download/12022/11881" title="" class="ltx_ref ltx_href">Scitail: A textual entailment dataset from science question answering</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">Conference on Artificial Intelligence (AAAI)</em>, volume 32.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Krishna et al. (2021)</span>
<span class="ltx_bibblock">
Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2021.naacl-main.393" title="" class="ltx_ref ltx_href">Hurdles to progress in long-form question answering</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies</em>, pages 4940–4957, Online. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Köpf et al. (2023)</span>
<span class="ltx_bibblock">
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2304.07327" title="" class="ltx_ref ltx_href">Openassistant conversations – democratizing large language model alignment</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2304.07327</em>.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lee et al. (2019)</span>
<span class="ltx_bibblock">
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/P19-1612" title="" class="ltx_ref ltx_href">Latent retrieval for weakly supervised open domain question answering</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">Association for Computational Linguistics (ACL)</em>, pages 6086–6096.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2023)</span>
<span class="ltx_bibblock">
Nelson F Liu, Tianyi Zhang, and Percy Liang. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/pdf/2304.09848.pdf" title="" class="ltx_ref ltx_href">Evaluating verifiability in generative search engines</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2304.09848</em>.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Menick et al. (2022)</span>
<span class="ltx_bibblock">
Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, et al. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/pdf/2203.11147.pdf" title="" class="ltx_ref ltx_href">Teaching language models to support answers with verified quotes</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2203.11147</em>.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Min et al. (2020)</span>
<span class="ltx_bibblock">
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2020.emnlp-main.466" title="" class="ltx_ref ltx_href">AmbigQA: Answering ambiguous open-domain questions</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)</em>, pages 5783–5797, Online. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nakano et al. (2021)</span>
<span class="ltx_bibblock">
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/pdf/2112.09332.pdf" title="" class="ltx_ref ltx_href">WebGPT: Browser-assisted question-answering with human feedback</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2112.09332</em>.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ni et al. (2022)</span>
<span class="ltx_bibblock">
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, and Yinfei Yang. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://aclanthology.org/2022.emnlp-main.669" title="" class="ltx_ref ltx_href">Large dual encoders are generalizable retrievers</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">Empirical Methods in Natural Language Processing (EMNLP)</em>, pages 9844–9855.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">OpenAI (2023)</span>
<span class="ltx_bibblock">
OpenAI. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2303.08774" title="" class="ltx_ref ltx_href">GPT-4 Technical Report</a>.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ouyang et al. (2022)</span>
<span class="ltx_bibblock">
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf" title="" class="ltx_ref ltx_href">Training language models to follow instructions with human feedback</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems (NeurIPS)</em>, 35:27730–27744.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Parisi et al. (2022)</span>
<span class="ltx_bibblock">
Aaron Parisi, Yao Zhao, and Noah Fiedel. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/pdf/2205.12255.pdf" title="" class="ltx_ref ltx_href">TALM: Tool augmented language models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2205.12255</em>.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Petroni et al. (2021)</span>
<span class="ltx_bibblock">
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2021.naacl-main.200" title="" class="ltx_ref ltx_href">KILT: a benchmark for knowledge intensive language tasks</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies</em>, pages 2523–2544, Online. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Piktus et al. (2021)</span>
<span class="ltx_bibblock">
Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Dmytro Okhonko, Samuel Broscheit, Gautier Izacard, Patrick Lewis, Barlas Oğuz, Edouard Grave, Wen-tau Yih, et al. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/pdf/2112.09924.pdf" title="" class="ltx_ref ltx_href">The Web Is Your Oyster–Knowledge-Intensive NLP against a Very Large Web Corpus</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2112.09924</em>.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Pillutla et al. (2021)</span>
<span class="ltx_bibblock">
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=Tqx7nJp7PR" title="" class="ltx_ref ltx_href">MAUVE: Measuring the gap between neural text and human text using divergence frontiers</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Press et al. (2022)</span>
<span class="ltx_bibblock">
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/pdf/2210.03350.pdf" title="" class="ltx_ref ltx_href">Measuring and narrowing the compositionality gap in language models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2210.03350</em>.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Raffel et al. (2020)</span>
<span class="ltx_bibblock">
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://jmlr.org/papers/v21/20-074.html" title="" class="ltx_ref ltx_href">Exploring the limits of transfer learning with a unified text-to-text Transformer</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">The Journal of Machine Learning Research (JMLR)</em>, 21(140).

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rajpurkar et al. (2016)</span>
<span class="ltx_bibblock">
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/D16-1264" title="" class="ltx_ref ltx_href">SQuAD: 100,000+ questions for machine comprehension of text</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">Empirical Methods in Natural Language Processing (EMNLP)</em>, pages 2383–2392.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rashkin et al. (2023)</span>
<span class="ltx_bibblock">
Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.1162/coli_a_00486" title="" class="ltx_ref ltx_href">Measuring Attribution in Natural Language Generation Models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">Computational Linguistics</em>, pages 1–64.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rubin et al. (2022)</span>
<span class="ltx_bibblock">
Samuel Joseph Amouyal Ohad Rubin, Ori Yoran, Tomer Wolfson, Jonathan Herzig, and Jonathan Berant. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2205.12665" title="" class="ltx_ref ltx_href">QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2205.12665</em>.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Schick et al. (2023)</span>
<span class="ltx_bibblock">
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/pdf/2302.04761.pdf" title="" class="ltx_ref ltx_href">Toolformer: Language models can teach themselves to use tools</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2302.04761</em>.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Schuster et al. (2021)</span>
<span class="ltx_bibblock">
Tal Schuster, Adam Fisch, and Regina Barzilay. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2021.naacl-main.52" title="" class="ltx_ref ltx_href">Get your vitamin C! robust fact verification with contrastive evidence</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)</em>, pages 624–643.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shi et al. (2023)</span>
<span class="ltx_bibblock">
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/pdf/2302.00093.pdf?trk=public_post_comment-text" title="" class="ltx_ref ltx_href">Large language models can be easily distracted by irrelevant context</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning (ICML)</em>.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shuster et al. (2022)</span>
<span class="ltx_bibblock">
Kurt Shuster, Mojtaba Komeili, Leonard Adolphs, Stephen Roller, Arthur Szlam, and Jason Weston. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://aclanthology.org/2022.findings-emnlp.27" title="" class="ltx_ref ltx_href">Language models that seek for knowledge: Modular search &amp; generation for dialogue and prompt completion</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">Findings of the Association for Computational Linguistics: EMNLP 2022</em>, pages 373–393.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Stelmakh et al. (2022)</span>
<span class="ltx_bibblock">
Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2022.emnlp-main.566" title="" class="ltx_ref ltx_href">ASQA: Factoid questions meet long-form answers</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib44.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing</em>, pages 8273–8288, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Taori et al. (2023)</span>
<span class="ltx_bibblock">
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/tatsu-lab/stanford_alpaca" title="" class="ltx_ref ltx_href">Stanford Alpaca: An Instruction-following LLaMA model</a>.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Thorne et al. (2018)</span>
<span class="ltx_bibblock">
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/N18-1074" title="" class="ltx_ref ltx_href">FEVER: a large-scale dataset for fact extraction and VERification</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib46.1.1" class="ltx_emph ltx_font_italic">North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)</em>, pages 809–819.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. (2023a)</span>
<span class="ltx_bibblock">
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/pdf/2302.13971.pdf" title="" class="ltx_ref ltx_href">LLaMA: Open and Efficient Foundation Language Models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib47.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2302.13971</em>.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. (2023b)</span>
<span class="ltx_bibblock">
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2307.09288" title="" class="ltx_ref ltx_href">Llama 2: Open foundation and fine-tuned chat models</a>.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2020)</span>
<span class="ltx_bibblock">
Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2020.acl-main.450" title="" class="ltx_ref ltx_href">Asking and answering questions to evaluate the factual consistency of summaries</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib49.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics</em>, pages 5008–5020, Online. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Williams et al. (2018)</span>
<span class="ltx_bibblock">
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://aclanthology.org/N18-1101.pdf" title="" class="ltx_ref ltx_href">A broad-coverage challenge corpus for sentence understanding through inference</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib50.1.1" class="ltx_emph ltx_font_italic">North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)</em>.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yao et al. (2023)</span>
<span class="ltx_bibblock">
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=WE_vluYUL-X" title="" class="ltx_ref ltx_href">React: Synergizing reasoning and acting in language models</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib51.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations (ICLR)</em>.

</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yue et al. (2023)</span>
<span class="ltx_bibblock">
Xiang Yue, Boshi Wang, Kai Zhang, Ziru Chen, Yu Su, and Huan Sun. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/pdf/2305.06311" title="" class="ltx_ref ltx_href">Automatic evaluation of attribution by large language models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib52.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.06311</em>.

</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang and Bansal (2021)</span>
<span class="ltx_bibblock">
Shiyue Zhang and Mohit Bansal. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2021.emnlp-main.531" title="" class="ltx_ref ltx_href">Finding a balanced degree of automation for summary evaluation</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib53.1.1" class="ltx_emph ltx_font_italic">Empirical Methods in Natural Language Processing (EMNLP)</em>, pages 6617–6632.

</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2019)</span>
<span class="ltx_bibblock">
Yuan Zhang, Jason Baldridge, and Luheng He. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/N19-1131" title="" class="ltx_ref ltx_href">PAWS: Paraphrase adversaries from word scrambling</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib54.1.1" class="ltx_emph ltx_font_italic">North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)</em>, pages 1298–1308.

</span>
</li>
<li id="bib.bib55" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhong et al. (2022)</span>
<span class="ltx_bibblock">
Zexuan Zhong, Tao Lei, and Danqi Chen. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://aclanthology.org/2022.emnlp-main.382" title="" class="ltx_ref ltx_href">Training language models with memory augmentation</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib55.1.1" class="ltx_emph ltx_font_italic">Empirical Methods in Natural Language Processing (EMNLP)</em>, pages 5657–5673.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Generating Claims for ELI5</h2>

<div id="A1.p1" class="ltx_para">
<p id="A1.p1.1" class="ltx_p">We elect not to use ROUGE-L as our main correctness metrics since it does not account for the different ways of expressing the same answer and it can be easily gamed <cite class="ltx_cite ltx_citemacro_cite">Krishna et al. (<a href="#bib.bib21" title="" class="ltx_ref">2021</a>)</cite>.
We further illustrate this issue in Table <a href="#A1.T10" title="Table 10 ‣ Appendix A Generating Claims for ELI5 ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">10</span></a>.
A system can easily achieve high ROUGE-L score by retrieving and returning the top passage from a BM25 index.
However, the claims evaluation metric does not reward this approach since the output often lacks different aspects of the answers.</p>
</div>
<figure id="A1.T10" class="ltx_table">
<table id="A1.T10.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T10.1.1.1" class="ltx_tr">
<th id="A1.T10.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="A1.T10.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">ROUGE-L</th>
<th id="A1.T10.1.1.1.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt">Claim recall</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T10.1.2.1" class="ltx_tr">
<th id="A1.T10.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">ChatGPT <span id="A1.T10.1.2.1.1.1" class="ltx_text ltx_font_smallcaps">Vanilla</span>
</th>
<td id="A1.T10.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t">20.6</td>
<td id="A1.T10.1.2.1.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">12.0</td>
</tr>
<tr id="A1.T10.1.3.2" class="ltx_tr">
<th id="A1.T10.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">ChatGPT <span id="A1.T10.1.3.2.1.1" class="ltx_text ltx_font_smallcaps">Oracle</span>
</th>
<td id="A1.T10.1.3.2.2" class="ltx_td ltx_align_center">21.2</td>
<td id="A1.T10.1.3.2.3" class="ltx_td ltx_nopad_r ltx_align_center">21.3</td>
</tr>
<tr id="A1.T10.1.4.3" class="ltx_tr">
<th id="A1.T10.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMa-13B <span id="A1.T10.1.4.3.1.1" class="ltx_text ltx_font_smallcaps">Vanilla</span>
</th>
<td id="A1.T10.1.4.3.2" class="ltx_td ltx_align_center">16.2</td>
<td id="A1.T10.1.4.3.3" class="ltx_td ltx_nopad_r ltx_align_center">3.9</td>
</tr>
<tr id="A1.T10.1.5.4" class="ltx_tr">
<th id="A1.T10.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">Top-1 passage</th>
<td id="A1.T10.1.5.4.2" class="ltx_td ltx_align_center ltx_border_bb">19.1</td>
<td id="A1.T10.1.5.4.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb">3.0</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 10: </span>
Comparison between ROUGE-L and claim recall scores on ELI5.
</figcaption>
</figure>
<div id="A1.p2" class="ltx_para">
<p id="A1.p2.1" class="ltx_p">Instead, we leverage the original answers
to generate sub-claims
and use them to serve as an estimate of the different aspects of the answers that we expect the model to cover.
This approach is inspired by works in summarization evaluation and claim verification <cite class="ltx_cite ltx_citemacro_cite">Zhang and Bansal (<a href="#bib.bib53" title="" class="ltx_ref">2021</a>); Kamoi et al. (<a href="#bib.bib17" title="" class="ltx_ref">2023</a>); Wang et al. (<a href="#bib.bib49" title="" class="ltx_ref">2020</a>)</cite>.</p>
</div>
<div id="A1.p3" class="ltx_para">
<p id="A1.p3.1" class="ltx_p">Specifically, we use <span id="A1.p3.1.1" class="ltx_text ltx_font_typewriter">text-davinci-003</span> to generate the sub-claims.
We first manually annotate three question and answer pairs from the original ELI5 training set with 3 sub-claims each.
Then, we prompt <span id="A1.p3.1.2" class="ltx_text ltx_font_typewriter">text-davinci-003</span> with these pairs as demonstrations.
The full prompt with an example is shown in Table <a href="#A7.T22" title="Table 22 ‣ G.7 Open-source Models ‣ Appendix G More Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">22</span></a>.</p>
</div>
<section id="A1.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">InstructGPT generates coherent and faithful sub-claims.</h4>

<div id="A1.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="A1.SS0.SSS0.Px1.p1.7" class="ltx_p">To ensure that the generated sub-claims are of good quality, we manually inspect a random sample of 40 answers and their generated sub-claims (totaling to 120 sub-claims).
For each sub-claim, we assign a score of <math id="A1.SS0.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="1" display="inline"><semantics id="A1.SS0.SSS0.Px1.p1.1.m1.1a"><mn id="A1.SS0.SSS0.Px1.p1.1.m1.1.1" xref="A1.SS0.SSS0.Px1.p1.1.m1.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="A1.SS0.SSS0.Px1.p1.1.m1.1b"><cn type="integer" id="A1.SS0.SSS0.Px1.p1.1.m1.1.1.cmml" xref="A1.SS0.SSS0.Px1.p1.1.m1.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS0.SSS0.Px1.p1.1.m1.1c">1</annotation></semantics></math> if it is relevant to the question and faithful to the facts presented in the ground truth, and <math id="A1.SS0.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="0" display="inline"><semantics id="A1.SS0.SSS0.Px1.p1.2.m2.1a"><mn id="A1.SS0.SSS0.Px1.p1.2.m2.1.1" xref="A1.SS0.SSS0.Px1.p1.2.m2.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="A1.SS0.SSS0.Px1.p1.2.m2.1b"><cn type="integer" id="A1.SS0.SSS0.Px1.p1.2.m2.1.1.cmml" xref="A1.SS0.SSS0.Px1.p1.2.m2.1.1">0</cn></annotation-xml></semantics></math> otherwise.
We found that <math id="A1.SS0.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="112" display="inline"><semantics id="A1.SS0.SSS0.Px1.p1.3.m3.1a"><mn id="A1.SS0.SSS0.Px1.p1.3.m3.1.1" xref="A1.SS0.SSS0.Px1.p1.3.m3.1.1.cmml">112</mn><annotation-xml encoding="MathML-Content" id="A1.SS0.SSS0.Px1.p1.3.m3.1b"><cn type="integer" id="A1.SS0.SSS0.Px1.p1.3.m3.1.1.cmml" xref="A1.SS0.SSS0.Px1.p1.3.m3.1.1">112</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS0.SSS0.Px1.p1.3.m3.1c">112</annotation></semantics></math> out of the <math id="A1.SS0.SSS0.Px1.p1.4.m4.1" class="ltx_Math" alttext="120" display="inline"><semantics id="A1.SS0.SSS0.Px1.p1.4.m4.1a"><mn id="A1.SS0.SSS0.Px1.p1.4.m4.1.1" xref="A1.SS0.SSS0.Px1.p1.4.m4.1.1.cmml">120</mn><annotation-xml encoding="MathML-Content" id="A1.SS0.SSS0.Px1.p1.4.m4.1b"><cn type="integer" id="A1.SS0.SSS0.Px1.p1.4.m4.1.1.cmml" xref="A1.SS0.SSS0.Px1.p1.4.m4.1.1">120</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS0.SSS0.Px1.p1.4.m4.1c">120</annotation></semantics></math> (<math id="A1.SS0.SSS0.Px1.p1.5.m5.1" class="ltx_Math" alttext="93.33\%" display="inline"><semantics id="A1.SS0.SSS0.Px1.p1.5.m5.1a"><mrow id="A1.SS0.SSS0.Px1.p1.5.m5.1.1" xref="A1.SS0.SSS0.Px1.p1.5.m5.1.1.cmml"><mn id="A1.SS0.SSS0.Px1.p1.5.m5.1.1.2" xref="A1.SS0.SSS0.Px1.p1.5.m5.1.1.2.cmml">93.33</mn><mo id="A1.SS0.SSS0.Px1.p1.5.m5.1.1.1" xref="A1.SS0.SSS0.Px1.p1.5.m5.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.SS0.SSS0.Px1.p1.5.m5.1b"><apply id="A1.SS0.SSS0.Px1.p1.5.m5.1.1.cmml" xref="A1.SS0.SSS0.Px1.p1.5.m5.1.1"><csymbol cd="latexml" id="A1.SS0.SSS0.Px1.p1.5.m5.1.1.1.cmml" xref="A1.SS0.SSS0.Px1.p1.5.m5.1.1.1">percent</csymbol><cn type="float" id="A1.SS0.SSS0.Px1.p1.5.m5.1.1.2.cmml" xref="A1.SS0.SSS0.Px1.p1.5.m5.1.1.2">93.33</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS0.SSS0.Px1.p1.5.m5.1c">93.33\%</annotation></semantics></math>) sub-claims received a score of <math id="A1.SS0.SSS0.Px1.p1.6.m6.1" class="ltx_Math" alttext="1" display="inline"><semantics id="A1.SS0.SSS0.Px1.p1.6.m6.1a"><mn id="A1.SS0.SSS0.Px1.p1.6.m6.1.1" xref="A1.SS0.SSS0.Px1.p1.6.m6.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="A1.SS0.SSS0.Px1.p1.6.m6.1b"><cn type="integer" id="A1.SS0.SSS0.Px1.p1.6.m6.1.1.cmml" xref="A1.SS0.SSS0.Px1.p1.6.m6.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS0.SSS0.Px1.p1.6.m6.1c">1</annotation></semantics></math>, meaning that our generated sub-claims are of high quality and faithful to the ground truth.
Furthermore, the average number of words in the generated sub-claims is <math id="A1.SS0.SSS0.Px1.p1.7.m7.1" class="ltx_Math" alttext="14" display="inline"><semantics id="A1.SS0.SSS0.Px1.p1.7.m7.1a"><mn id="A1.SS0.SSS0.Px1.p1.7.m7.1.1" xref="A1.SS0.SSS0.Px1.p1.7.m7.1.1.cmml">14</mn><annotation-xml encoding="MathML-Content" id="A1.SS0.SSS0.Px1.p1.7.m7.1b"><cn type="integer" id="A1.SS0.SSS0.Px1.p1.7.m7.1.1.cmml" xref="A1.SS0.SSS0.Px1.p1.7.m7.1.1">14</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS0.SSS0.Px1.p1.7.m7.1c">14</annotation></semantics></math> words, and they are typically just one sentence long.
This is aligned with the intent behind the metric—to capture short factual claims made by the original answer.</p>
</div>
</section>
<section id="A1.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">NLI model accurately predicts the entailment of sub-claims.</h4>

<div id="A1.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="A1.SS0.SSS0.Px2.p1.1" class="ltx_p">We further analyze our sub-claim evaluation metrics by checking the error rate of the final prediction of the NLI model.
To this end, we first manually annotate the entailment scores between 40 outputs and their sub-claims (in total of 120 pairs; these are the same questions from the previous analysis).
We then use the NLI model to obtain the entailment scores for the output and sub-claims.
Using the human annotations as the ground truth label, we found that the NLI model achieved an accuracy of <math id="A1.SS0.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="80.0\%" display="inline"><semantics id="A1.SS0.SSS0.Px2.p1.1.m1.1a"><mrow id="A1.SS0.SSS0.Px2.p1.1.m1.1.1" xref="A1.SS0.SSS0.Px2.p1.1.m1.1.1.cmml"><mn id="A1.SS0.SSS0.Px2.p1.1.m1.1.1.2" xref="A1.SS0.SSS0.Px2.p1.1.m1.1.1.2.cmml">80.0</mn><mo id="A1.SS0.SSS0.Px2.p1.1.m1.1.1.1" xref="A1.SS0.SSS0.Px2.p1.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.SS0.SSS0.Px2.p1.1.m1.1b"><apply id="A1.SS0.SSS0.Px2.p1.1.m1.1.1.cmml" xref="A1.SS0.SSS0.Px2.p1.1.m1.1.1"><csymbol cd="latexml" id="A1.SS0.SSS0.Px2.p1.1.m1.1.1.1.cmml" xref="A1.SS0.SSS0.Px2.p1.1.m1.1.1.1">percent</csymbol><cn type="float" id="A1.SS0.SSS0.Px2.p1.1.m1.1.1.2.cmml" xref="A1.SS0.SSS0.Px2.p1.1.m1.1.1.2">80.0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS0.SSS0.Px2.p1.1.m1.1c">80.0\%</annotation></semantics></math>.</p>
</div>
</section>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Dataset Statistics</h2>

<div id="A2.p1" class="ltx_para">
<p id="A2.p1.1" class="ltx_p">For ASQA, human answers have an average length of 65 words.
For QAMPARI, each question has on average 13 answers.
For ELI5, human answers have an average length of 131 words.</p>
</div>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Implementation Details</h2>

<section id="A3.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">NLI model.</h4>

<div id="A3.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="A3.SS0.SSS0.Px1.p1.1" class="ltx_p">We use the version of TRUE model from <a target="_blank" href="https://huggingface.co/google/t5_xxl_true_nli_mixture" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/google/t5_xxl_true_nli_mixture</a>, which is trained on
SNLI <cite class="ltx_cite ltx_citemacro_citep">(Bowman et al., <a href="#bib.bib3" title="" class="ltx_ref">2015</a>)</cite>,
MNLI <cite class="ltx_cite ltx_citemacro_citep">(Williams et al., <a href="#bib.bib50" title="" class="ltx_ref">2018</a>)</cite>,
Fever <cite class="ltx_cite ltx_citemacro_citep">(Thorne et al., <a href="#bib.bib46" title="" class="ltx_ref">2018</a>)</cite>,
Scitail <cite class="ltx_cite ltx_citemacro_citep">(Khot et al., <a href="#bib.bib20" title="" class="ltx_ref">2018</a>)</cite>,
PAWS <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al., <a href="#bib.bib54" title="" class="ltx_ref">2019</a>)</cite>,
and VitaminC <cite class="ltx_cite ltx_citemacro_citep">(Schuster et al., <a href="#bib.bib41" title="" class="ltx_ref">2021</a>)</cite>.
This model uses the following prompt: “<span id="A3.SS0.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_typewriter">premise: {PREMISE} hypothesis: {}</span>”
and outputs “<span id="A3.SS0.SSS0.Px1.p1.1.2" class="ltx_text ltx_font_typewriter">1</span>” if the premise entails the hypothesis.
We format each passage (when used as premise)
by the format of “<span id="A3.SS0.SSS0.Px1.p1.1.3" class="ltx_text ltx_font_typewriter">Title: {TITLE}\n{TEXT}</span>” and concatenate all passages with “<span id="A3.SS0.SSS0.Px1.p1.1.4" class="ltx_text ltx_font_typewriter">\n</span>” as a separator.</p>
</div>
</section>
<section id="A3.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">MAUVE.</h4>

<div id="A3.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="A3.SS0.SSS0.Px2.p1.1" class="ltx_p">When running MAUVE, we concatenate the question and the model output (or human answer) by space.
We truncate both the references and the model generations to 100 words,
as we found MAUVE results are unstable beyond this length for ELI5 (this is due to that ELI5 has a lot of extremely long human answers).</p>
</div>
</section>
<section id="A3.SS0.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Exact match for ASQA and QAMPARI.</h4>

<div id="A3.SS0.SSS0.Px3.p1" class="ltx_para">
<p id="A3.SS0.SSS0.Px3.p1.1" class="ltx_p">Both ASQA and QAMPARI provide aliases for their short answers.
We normalize the response and the short answers similarly to <cite class="ltx_cite ltx_citemacro_citet">Rajpurkar et al. (<a href="#bib.bib37" title="" class="ltx_ref">2016</a>)</cite>
and report the score with the best-matching aliases.
For ASQA, <cite class="ltx_cite ltx_citemacro_citet">Stelmakh et al. (<a href="#bib.bib44" title="" class="ltx_ref">2022</a>)</cite> also propose a QA-based evaluation which we found to be not as stable, and thus we do not report it in our paper.</p>
</div>
</section>
<section id="A3.SS0.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Output truncation.</h4>

<div id="A3.SS0.SSS0.Px4.p1" class="ltx_para">
<p id="A3.SS0.SSS0.Px4.p1.1" class="ltx_p">Before evaluation,
we truncate model output by new lines,
as non-instruction-tuned models may generate more content after new lines that are irrelevant.</p>
</div>
</section>
<section id="A3.SS0.SSS0.Px5" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">
<span id="A3.SS0.SSS0.Px5.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">Interact</span>.</h4>

<div id="A3.SS0.SSS0.Px5.p1" class="ltx_para">
<p id="A3.SS0.SSS0.Px5.p1.1" class="ltx_p">Empirically, we found that models tend to execute too many consecutive “<span id="A3.SS0.SSS0.Px5.p1.1.1" class="ltx_text ltx_font_typewriter">check</span>” actions, so we force the model to always “<span id="A3.SS0.SSS0.Px5.p1.1.2" class="ltx_text ltx_font_typewriter">output</span>” after each “<span id="A3.SS0.SSS0.Px5.p1.1.3" class="ltx_text ltx_font_typewriter">check</span>”.
We limit the maximum number of passages to check as 3 to avoid exceeding the length limit.
The full passages are removed from the context after one action to save context space.
Table <a href="#A8.T27" title="Table 27 ‣ Appendix H Prompts ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">27</span></a> provides an example for <span id="A3.SS0.SSS0.Px5.p1.1.4" class="ltx_text ltx_font_smallcaps">Interact</span>.</p>
</div>
</section>
<section id="A3.SS0.SSS0.Px6" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Main experiments.</h4>

<div id="A3.SS0.SSS0.Px6.p1" class="ltx_para">
<p id="A3.SS0.SSS0.Px6.p1.1" class="ltx_p">For all experiments except ChatGPT <span id="A3.SS0.SSS0.Px6.p1.1.1" class="ltx_text ltx_font_smallcaps">Rerank</span>,
we run each model three times with different seeds and each time we sample two demonstrations from a pool of four.
We report the averaged scores for all experiments in the main paper
and we report the standard deviations in Appendix <a href="#A7.SS6" title="G.6 Main Results ‣ Appendix G More Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">G.6</span></a>.</p>
</div>
</section>
<section id="A3.SS0.SSS0.Px7" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Decoding methods.</h4>

<div id="A3.SS0.SSS0.Px7.p1" class="ltx_para">
<p id="A3.SS0.SSS0.Px7.p1.2" class="ltx_p">Based on preliminary experiments we choose the following decoding methods: For ChatGPT and GPT-4, we use sampling with temperature <math id="A3.SS0.SSS0.Px7.p1.1.m1.1" class="ltx_Math" alttext="0.5" display="inline"><semantics id="A3.SS0.SSS0.Px7.p1.1.m1.1a"><mn id="A3.SS0.SSS0.Px7.p1.1.m1.1.1" xref="A3.SS0.SSS0.Px7.p1.1.m1.1.1.cmml">0.5</mn><annotation-xml encoding="MathML-Content" id="A3.SS0.SSS0.Px7.p1.1.m1.1b"><cn type="float" id="A3.SS0.SSS0.Px7.p1.1.m1.1.1.cmml" xref="A3.SS0.SSS0.Px7.p1.1.m1.1.1">0.5</cn></annotation-xml><annotation encoding="application/x-tex" id="A3.SS0.SSS0.Px7.p1.1.m1.1c">0.5</annotation></semantics></math>; for all open-source models, we use Nucleus sampling <cite class="ltx_cite ltx_citemacro_citep">(Holtzman et al., <a href="#bib.bib11" title="" class="ltx_ref">2020</a>)</cite> and set <math id="A3.SS0.SSS0.Px7.p1.2.m2.1" class="ltx_Math" alttext="\text{top\_p}=0.95" display="inline"><semantics id="A3.SS0.SSS0.Px7.p1.2.m2.1a"><mrow id="A3.SS0.SSS0.Px7.p1.2.m2.1.1" xref="A3.SS0.SSS0.Px7.p1.2.m2.1.1.cmml"><mtext id="A3.SS0.SSS0.Px7.p1.2.m2.1.1.2" xref="A3.SS0.SSS0.Px7.p1.2.m2.1.1.2a.cmml">top_p</mtext><mo id="A3.SS0.SSS0.Px7.p1.2.m2.1.1.1" xref="A3.SS0.SSS0.Px7.p1.2.m2.1.1.1.cmml">=</mo><mn id="A3.SS0.SSS0.Px7.p1.2.m2.1.1.3" xref="A3.SS0.SSS0.Px7.p1.2.m2.1.1.3.cmml">0.95</mn></mrow><annotation-xml encoding="MathML-Content" id="A3.SS0.SSS0.Px7.p1.2.m2.1b"><apply id="A3.SS0.SSS0.Px7.p1.2.m2.1.1.cmml" xref="A3.SS0.SSS0.Px7.p1.2.m2.1.1"><eq id="A3.SS0.SSS0.Px7.p1.2.m2.1.1.1.cmml" xref="A3.SS0.SSS0.Px7.p1.2.m2.1.1.1"></eq><ci id="A3.SS0.SSS0.Px7.p1.2.m2.1.1.2a.cmml" xref="A3.SS0.SSS0.Px7.p1.2.m2.1.1.2"><mtext id="A3.SS0.SSS0.Px7.p1.2.m2.1.1.2.cmml" xref="A3.SS0.SSS0.Px7.p1.2.m2.1.1.2">top_p</mtext></ci><cn type="float" id="A3.SS0.SSS0.Px7.p1.2.m2.1.1.3.cmml" xref="A3.SS0.SSS0.Px7.p1.2.m2.1.1.3">0.95</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.SS0.SSS0.Px7.p1.2.m2.1c">\text{top\_p}=0.95</annotation></semantics></math>.</p>
</div>
</section>
</section>
<section id="A4" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>ALCE Catches Shortcut Cases</h2>

<figure id="A4.T11" class="ltx_table">
<div id="A4.T11.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:424.9pt;height:146.2pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(81.6pt,-28.1pt) scale(1.62399472143652,1.62399472143652) ;">
<table id="A4.T11.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A4.T11.1.1.1.1" class="ltx_tr">
<th id="A4.T11.1.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="A4.T11.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A4.T11.1.1.1.1.2.1" class="ltx_text ltx_font_bold">Fluency</span></th>
<th id="A4.T11.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A4.T11.1.1.1.1.3.1" class="ltx_text ltx_font_bold">Correct.</span></th>
<th id="A4.T11.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2"><span id="A4.T11.1.1.1.1.4.1" class="ltx_text ltx_font_bold">Citation</span></th>
</tr>
<tr id="A4.T11.1.1.2.2" class="ltx_tr">
<th id="A4.T11.1.1.2.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="A4.T11.1.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">(MAUVE)</th>
<th id="A4.T11.1.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">(EM Rec.)</th>
<th id="A4.T11.1.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Rec.</th>
<th id="A4.T11.1.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Prec.</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A4.T11.1.1.3.1" class="ltx_tr">
<th id="A4.T11.1.1.3.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">ChatGPT</th>
<td id="A4.T11.1.1.3.1.2" class="ltx_td ltx_align_center ltx_border_t">66.6</td>
<td id="A4.T11.1.1.3.1.3" class="ltx_td ltx_align_center ltx_border_t">40.4</td>
<td id="A4.T11.1.1.3.1.4" class="ltx_td ltx_align_center ltx_border_t">73.6</td>
<td id="A4.T11.1.1.3.1.5" class="ltx_td ltx_align_center ltx_border_t">63.0</td>
</tr>
<tr id="A4.T11.1.1.4.2" class="ltx_tr">
<th id="A4.T11.1.1.4.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Top-1 passage</th>
<td id="A4.T11.1.1.4.2.2" class="ltx_td ltx_align_center"><span id="A4.T11.1.1.4.2.2.1" class="ltx_text" style="color:#FF0000;">20.8</span></td>
<td id="A4.T11.1.1.4.2.3" class="ltx_td ltx_align_center">35.1</td>
<td id="A4.T11.1.1.4.2.4" class="ltx_td ltx_align_center">99.4</td>
<td id="A4.T11.1.1.4.2.5" class="ltx_td ltx_align_center">99.4</td>
</tr>
<tr id="A4.T11.1.1.5.3" class="ltx_tr">
<th id="A4.T11.1.1.5.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">First 2 sents</th>
<td id="A4.T11.1.1.5.3.2" class="ltx_td ltx_align_center ltx_border_bb">67.2</td>
<td id="A4.T11.1.1.5.3.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="A4.T11.1.1.5.3.3.1" class="ltx_text" style="color:#FF0000;">18.9</span></td>
<td id="A4.T11.1.1.5.3.4" class="ltx_td ltx_align_center ltx_border_bb">98.7</td>
<td id="A4.T11.1.1.5.3.5" class="ltx_td ltx_align_center ltx_border_bb">98.7</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 11: </span>
ASQA cheating cases. “ChatGPT”: the ChatGPT <span id="A4.T11.3.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> model with GTR-retrieved top-5 passages.
“Top-1 passage”: use the top-1 retrieved passage as the response.
“First 2 sents”: use the first 2 sentences of the top-1 retrieved passage.
</figcaption>
</figure>
<div id="A4.p1" class="ltx_para">
<p id="A4.p1.1" class="ltx_p">Table <a href="#A4.T11" title="Table 11 ‣ Appendix D ALCE Catches Shortcut Cases ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a> demonstrates the experiments to show that ALCE
is robust to shortcut cases.
Using the top-1 passages or first two sentences of the top-1 passages
induces almost perfect citation quality,
but fluency and correctness are dramatically lower.</p>
</div>
</section>
<section id="A5" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix E </span>Citation Recall Discussion</h2>

<div id="A5.p1" class="ltx_para">
<p id="A5.p1.13" class="ltx_p">Our citation precision evaluation cannot detect a citation that partially supports the statement and hence will falsely penalize it.
Consider a statement <math id="A5.p1.1.m1.1" class="ltx_Math" alttext="s_{3}" display="inline"><semantics id="A5.p1.1.m1.1a"><msub id="A5.p1.1.m1.1.1" xref="A5.p1.1.m1.1.1.cmml"><mi id="A5.p1.1.m1.1.1.2" xref="A5.p1.1.m1.1.1.2.cmml">s</mi><mn id="A5.p1.1.m1.1.1.3" xref="A5.p1.1.m1.1.1.3.cmml">3</mn></msub><annotation-xml encoding="MathML-Content" id="A5.p1.1.m1.1b"><apply id="A5.p1.1.m1.1.1.cmml" xref="A5.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.p1.1.m1.1.1.1.cmml" xref="A5.p1.1.m1.1.1">subscript</csymbol><ci id="A5.p1.1.m1.1.1.2.cmml" xref="A5.p1.1.m1.1.1.2">𝑠</ci><cn type="integer" id="A5.p1.1.m1.1.1.3.cmml" xref="A5.p1.1.m1.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p1.1.m1.1c">s_{3}</annotation></semantics></math> and its citations <span id="A5.p1.13.1" class="ltx_text ltx_font_typewriter">[2][4][5]</span>:
if <span id="A5.p1.13.2" class="ltx_text ltx_font_typewriter">[2]</span> entails partial information of <math id="A5.p1.2.m2.1" class="ltx_Math" alttext="s_{3}" display="inline"><semantics id="A5.p1.2.m2.1a"><msub id="A5.p1.2.m2.1.1" xref="A5.p1.2.m2.1.1.cmml"><mi id="A5.p1.2.m2.1.1.2" xref="A5.p1.2.m2.1.1.2.cmml">s</mi><mn id="A5.p1.2.m2.1.1.3" xref="A5.p1.2.m2.1.1.3.cmml">3</mn></msub><annotation-xml encoding="MathML-Content" id="A5.p1.2.m2.1b"><apply id="A5.p1.2.m2.1.1.cmml" xref="A5.p1.2.m2.1.1"><csymbol cd="ambiguous" id="A5.p1.2.m2.1.1.1.cmml" xref="A5.p1.2.m2.1.1">subscript</csymbol><ci id="A5.p1.2.m2.1.1.2.cmml" xref="A5.p1.2.m2.1.1.2">𝑠</ci><cn type="integer" id="A5.p1.2.m2.1.1.3.cmml" xref="A5.p1.2.m2.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p1.2.m2.1c">s_{3}</annotation></semantics></math> that <span id="A5.p1.13.3" class="ltx_text ltx_font_typewriter">[4][5]</span> also entails,
<span id="A5.p1.13.4" class="ltx_text ltx_font_typewriter">[2]</span> will be counted as “irrelevant” while it should not be penalized.
<cite class="ltx_cite ltx_citemacro_citet">Liu et al. (<a href="#bib.bib24" title="" class="ltx_ref">2023</a>)</cite> conduct human evaluation on citation precision in a different way:
For each citation,
they ask annotators to judge whether the citation (1) fully support, (2) partially support, or (3) does not support <math id="A5.p1.3.m3.1" class="ltx_Math" alttext="s_{i}" display="inline"><semantics id="A5.p1.3.m3.1a"><msub id="A5.p1.3.m3.1.1" xref="A5.p1.3.m3.1.1.cmml"><mi id="A5.p1.3.m3.1.1.2" xref="A5.p1.3.m3.1.1.2.cmml">s</mi><mi id="A5.p1.3.m3.1.1.3" xref="A5.p1.3.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="A5.p1.3.m3.1b"><apply id="A5.p1.3.m3.1.1.cmml" xref="A5.p1.3.m3.1.1"><csymbol cd="ambiguous" id="A5.p1.3.m3.1.1.1.cmml" xref="A5.p1.3.m3.1.1">subscript</csymbol><ci id="A5.p1.3.m3.1.1.2.cmml" xref="A5.p1.3.m3.1.1.2">𝑠</ci><ci id="A5.p1.3.m3.1.1.3.cmml" xref="A5.p1.3.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p1.3.m3.1c">s_{i}</annotation></semantics></math>.
One citation <math id="A5.p1.4.m4.2" class="ltx_Math" alttext="c_{i,j}" display="inline"><semantics id="A5.p1.4.m4.2a"><msub id="A5.p1.4.m4.2.3" xref="A5.p1.4.m4.2.3.cmml"><mi id="A5.p1.4.m4.2.3.2" xref="A5.p1.4.m4.2.3.2.cmml">c</mi><mrow id="A5.p1.4.m4.2.2.2.4" xref="A5.p1.4.m4.2.2.2.3.cmml"><mi id="A5.p1.4.m4.1.1.1.1" xref="A5.p1.4.m4.1.1.1.1.cmml">i</mi><mo id="A5.p1.4.m4.2.2.2.4.1" xref="A5.p1.4.m4.2.2.2.3.cmml">,</mo><mi id="A5.p1.4.m4.2.2.2.2" xref="A5.p1.4.m4.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.p1.4.m4.2b"><apply id="A5.p1.4.m4.2.3.cmml" xref="A5.p1.4.m4.2.3"><csymbol cd="ambiguous" id="A5.p1.4.m4.2.3.1.cmml" xref="A5.p1.4.m4.2.3">subscript</csymbol><ci id="A5.p1.4.m4.2.3.2.cmml" xref="A5.p1.4.m4.2.3.2">𝑐</ci><list id="A5.p1.4.m4.2.2.2.3.cmml" xref="A5.p1.4.m4.2.2.2.4"><ci id="A5.p1.4.m4.1.1.1.1.cmml" xref="A5.p1.4.m4.1.1.1.1">𝑖</ci><ci id="A5.p1.4.m4.2.2.2.2.cmml" xref="A5.p1.4.m4.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p1.4.m4.2c">c_{i,j}</annotation></semantics></math> is precise if
(a) <math id="A5.p1.5.m5.2" class="ltx_Math" alttext="c_{i,j}" display="inline"><semantics id="A5.p1.5.m5.2a"><msub id="A5.p1.5.m5.2.3" xref="A5.p1.5.m5.2.3.cmml"><mi id="A5.p1.5.m5.2.3.2" xref="A5.p1.5.m5.2.3.2.cmml">c</mi><mrow id="A5.p1.5.m5.2.2.2.4" xref="A5.p1.5.m5.2.2.2.3.cmml"><mi id="A5.p1.5.m5.1.1.1.1" xref="A5.p1.5.m5.1.1.1.1.cmml">i</mi><mo id="A5.p1.5.m5.2.2.2.4.1" xref="A5.p1.5.m5.2.2.2.3.cmml">,</mo><mi id="A5.p1.5.m5.2.2.2.2" xref="A5.p1.5.m5.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.p1.5.m5.2b"><apply id="A5.p1.5.m5.2.3.cmml" xref="A5.p1.5.m5.2.3"><csymbol cd="ambiguous" id="A5.p1.5.m5.2.3.1.cmml" xref="A5.p1.5.m5.2.3">subscript</csymbol><ci id="A5.p1.5.m5.2.3.2.cmml" xref="A5.p1.5.m5.2.3.2">𝑐</ci><list id="A5.p1.5.m5.2.2.2.3.cmml" xref="A5.p1.5.m5.2.2.2.4"><ci id="A5.p1.5.m5.1.1.1.1.cmml" xref="A5.p1.5.m5.1.1.1.1">𝑖</ci><ci id="A5.p1.5.m5.2.2.2.2.cmml" xref="A5.p1.5.m5.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p1.5.m5.2c">c_{i,j}</annotation></semantics></math> fully supports <math id="A5.p1.6.m6.1" class="ltx_Math" alttext="s_{i}" display="inline"><semantics id="A5.p1.6.m6.1a"><msub id="A5.p1.6.m6.1.1" xref="A5.p1.6.m6.1.1.cmml"><mi id="A5.p1.6.m6.1.1.2" xref="A5.p1.6.m6.1.1.2.cmml">s</mi><mi id="A5.p1.6.m6.1.1.3" xref="A5.p1.6.m6.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="A5.p1.6.m6.1b"><apply id="A5.p1.6.m6.1.1.cmml" xref="A5.p1.6.m6.1.1"><csymbol cd="ambiguous" id="A5.p1.6.m6.1.1.1.cmml" xref="A5.p1.6.m6.1.1">subscript</csymbol><ci id="A5.p1.6.m6.1.1.2.cmml" xref="A5.p1.6.m6.1.1.2">𝑠</ci><ci id="A5.p1.6.m6.1.1.3.cmml" xref="A5.p1.6.m6.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p1.6.m6.1c">s_{i}</annotation></semantics></math> or
(b) <math id="A5.p1.7.m7.1" class="ltx_Math" alttext="\mathcal{C}_{i}" display="inline"><semantics id="A5.p1.7.m7.1a"><msub id="A5.p1.7.m7.1.1" xref="A5.p1.7.m7.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A5.p1.7.m7.1.1.2" xref="A5.p1.7.m7.1.1.2.cmml">𝒞</mi><mi id="A5.p1.7.m7.1.1.3" xref="A5.p1.7.m7.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="A5.p1.7.m7.1b"><apply id="A5.p1.7.m7.1.1.cmml" xref="A5.p1.7.m7.1.1"><csymbol cd="ambiguous" id="A5.p1.7.m7.1.1.1.cmml" xref="A5.p1.7.m7.1.1">subscript</csymbol><ci id="A5.p1.7.m7.1.1.2.cmml" xref="A5.p1.7.m7.1.1.2">𝒞</ci><ci id="A5.p1.7.m7.1.1.3.cmml" xref="A5.p1.7.m7.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p1.7.m7.1c">\mathcal{C}_{i}</annotation></semantics></math> fully supports <math id="A5.p1.8.m8.1" class="ltx_Math" alttext="s_{i}" display="inline"><semantics id="A5.p1.8.m8.1a"><msub id="A5.p1.8.m8.1.1" xref="A5.p1.8.m8.1.1.cmml"><mi id="A5.p1.8.m8.1.1.2" xref="A5.p1.8.m8.1.1.2.cmml">s</mi><mi id="A5.p1.8.m8.1.1.3" xref="A5.p1.8.m8.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="A5.p1.8.m8.1b"><apply id="A5.p1.8.m8.1.1.cmml" xref="A5.p1.8.m8.1.1"><csymbol cd="ambiguous" id="A5.p1.8.m8.1.1.1.cmml" xref="A5.p1.8.m8.1.1">subscript</csymbol><ci id="A5.p1.8.m8.1.1.2.cmml" xref="A5.p1.8.m8.1.1.2">𝑠</ci><ci id="A5.p1.8.m8.1.1.3.cmml" xref="A5.p1.8.m8.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p1.8.m8.1c">s_{i}</annotation></semantics></math>, <math id="A5.p1.9.m9.2" class="ltx_Math" alttext="c_{i,j}" display="inline"><semantics id="A5.p1.9.m9.2a"><msub id="A5.p1.9.m9.2.3" xref="A5.p1.9.m9.2.3.cmml"><mi id="A5.p1.9.m9.2.3.2" xref="A5.p1.9.m9.2.3.2.cmml">c</mi><mrow id="A5.p1.9.m9.2.2.2.4" xref="A5.p1.9.m9.2.2.2.3.cmml"><mi id="A5.p1.9.m9.1.1.1.1" xref="A5.p1.9.m9.1.1.1.1.cmml">i</mi><mo id="A5.p1.9.m9.2.2.2.4.1" xref="A5.p1.9.m9.2.2.2.3.cmml">,</mo><mi id="A5.p1.9.m9.2.2.2.2" xref="A5.p1.9.m9.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.p1.9.m9.2b"><apply id="A5.p1.9.m9.2.3.cmml" xref="A5.p1.9.m9.2.3"><csymbol cd="ambiguous" id="A5.p1.9.m9.2.3.1.cmml" xref="A5.p1.9.m9.2.3">subscript</csymbol><ci id="A5.p1.9.m9.2.3.2.cmml" xref="A5.p1.9.m9.2.3.2">𝑐</ci><list id="A5.p1.9.m9.2.2.2.3.cmml" xref="A5.p1.9.m9.2.2.2.4"><ci id="A5.p1.9.m9.1.1.1.1.cmml" xref="A5.p1.9.m9.1.1.1.1">𝑖</ci><ci id="A5.p1.9.m9.2.2.2.2.cmml" xref="A5.p1.9.m9.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p1.9.m9.2c">c_{i,j}</annotation></semantics></math> partially supports <math id="A5.p1.10.m10.1" class="ltx_Math" alttext="s_{i}" display="inline"><semantics id="A5.p1.10.m10.1a"><msub id="A5.p1.10.m10.1.1" xref="A5.p1.10.m10.1.1.cmml"><mi id="A5.p1.10.m10.1.1.2" xref="A5.p1.10.m10.1.1.2.cmml">s</mi><mi id="A5.p1.10.m10.1.1.3" xref="A5.p1.10.m10.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="A5.p1.10.m10.1b"><apply id="A5.p1.10.m10.1.1.cmml" xref="A5.p1.10.m10.1.1"><csymbol cd="ambiguous" id="A5.p1.10.m10.1.1.1.cmml" xref="A5.p1.10.m10.1.1">subscript</csymbol><ci id="A5.p1.10.m10.1.1.2.cmml" xref="A5.p1.10.m10.1.1.2">𝑠</ci><ci id="A5.p1.10.m10.1.1.3.cmml" xref="A5.p1.10.m10.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p1.10.m10.1c">s_{i}</annotation></semantics></math>, and no <math id="A5.p1.11.m11.1" class="ltx_Math" alttext="c\in\mathcal{C}_{i}" display="inline"><semantics id="A5.p1.11.m11.1a"><mrow id="A5.p1.11.m11.1.1" xref="A5.p1.11.m11.1.1.cmml"><mi id="A5.p1.11.m11.1.1.2" xref="A5.p1.11.m11.1.1.2.cmml">c</mi><mo id="A5.p1.11.m11.1.1.1" xref="A5.p1.11.m11.1.1.1.cmml">∈</mo><msub id="A5.p1.11.m11.1.1.3" xref="A5.p1.11.m11.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="A5.p1.11.m11.1.1.3.2" xref="A5.p1.11.m11.1.1.3.2.cmml">𝒞</mi><mi id="A5.p1.11.m11.1.1.3.3" xref="A5.p1.11.m11.1.1.3.3.cmml">i</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="A5.p1.11.m11.1b"><apply id="A5.p1.11.m11.1.1.cmml" xref="A5.p1.11.m11.1.1"><in id="A5.p1.11.m11.1.1.1.cmml" xref="A5.p1.11.m11.1.1.1"></in><ci id="A5.p1.11.m11.1.1.2.cmml" xref="A5.p1.11.m11.1.1.2">𝑐</ci><apply id="A5.p1.11.m11.1.1.3.cmml" xref="A5.p1.11.m11.1.1.3"><csymbol cd="ambiguous" id="A5.p1.11.m11.1.1.3.1.cmml" xref="A5.p1.11.m11.1.1.3">subscript</csymbol><ci id="A5.p1.11.m11.1.1.3.2.cmml" xref="A5.p1.11.m11.1.1.3.2">𝒞</ci><ci id="A5.p1.11.m11.1.1.3.3.cmml" xref="A5.p1.11.m11.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p1.11.m11.1c">c\in\mathcal{C}_{i}</annotation></semantics></math> alone fully supports <math id="A5.p1.12.m12.1" class="ltx_Math" alttext="s_{i}" display="inline"><semantics id="A5.p1.12.m12.1a"><msub id="A5.p1.12.m12.1.1" xref="A5.p1.12.m12.1.1.cmml"><mi id="A5.p1.12.m12.1.1.2" xref="A5.p1.12.m12.1.1.2.cmml">s</mi><mi id="A5.p1.12.m12.1.1.3" xref="A5.p1.12.m12.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="A5.p1.12.m12.1b"><apply id="A5.p1.12.m12.1.1.cmml" xref="A5.p1.12.m12.1.1"><csymbol cd="ambiguous" id="A5.p1.12.m12.1.1.1.cmml" xref="A5.p1.12.m12.1.1">subscript</csymbol><ci id="A5.p1.12.m12.1.1.2.cmml" xref="A5.p1.12.m12.1.1.2">𝑠</ci><ci id="A5.p1.12.m12.1.1.3.cmml" xref="A5.p1.12.m12.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p1.12.m12.1c">s_{i}</annotation></semantics></math>.
This evaluation solved the corner case we mentioned in the main paper (one citation partially supports the claim but is identified as “irrelevant”).
However, it is challenging to conduct such evaluation automatically,
as there is no existing model that can judge whether a citation “partially” supports a claim.
We also explore prompting ChatGPT to conduct such a task, which yields poor results.
We defer it to future work to collect supervised data to train a better <math id="A5.p1.13.m13.1" class="ltx_Math" alttext="\phi" display="inline"><semantics id="A5.p1.13.m13.1a"><mi id="A5.p1.13.m13.1.1" xref="A5.p1.13.m13.1.1.cmml">ϕ</mi><annotation-xml encoding="MathML-Content" id="A5.p1.13.m13.1b"><ci id="A5.p1.13.m13.1.1.cmml" xref="A5.p1.13.m13.1.1">italic-ϕ</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.p1.13.m13.1c">\phi</annotation></semantics></math> that can detect “partial support”.</p>
</div>
</section>
<section id="A6" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix F </span>Human Evaluation</h2>

<div id="A6.p1" class="ltx_para">
<p id="A6.p1.1" class="ltx_p">We employ Surge AI (<a target="_blank" href="https://www.surgehq.ai/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.surgehq.ai/</a>) for our human evaluation.
The average pay to workers is 20 USD per hour.
We randomly sample 100 examples from ASQA and ELI5 and annotate outputs of selected models: ChatGPT <span id="A6.p1.1.1" class="ltx_text ltx_font_smallcaps">Vanilla</span>, ChatGPT <span id="A6.p1.1.2" class="ltx_text ltx_font_smallcaps">Rerank</span>, and Vicuna-13B <span id="A6.p1.1.3" class="ltx_text ltx_font_smallcaps">Vanilla</span>.</p>
</div>
<section id="A6.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">F.1 </span>Utility</h3>

<div id="A6.SS1.p1" class="ltx_para">
<p id="A6.SS1.p1.3" class="ltx_p">To check if the model output is useful to downstream users, we measure the utility of the response <math id="A6.SS1.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{S}" display="inline"><semantics id="A6.SS1.p1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="A6.SS1.p1.1.m1.1.1" xref="A6.SS1.p1.1.m1.1.1.cmml">𝒮</mi><annotation-xml encoding="MathML-Content" id="A6.SS1.p1.1.m1.1b"><ci id="A6.SS1.p1.1.m1.1.1.cmml" xref="A6.SS1.p1.1.m1.1.1">𝒮</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS1.p1.1.m1.1c">\mathcal{S}</annotation></semantics></math>.
We first show the query <math id="A6.SS1.p1.2.m2.1" class="ltx_Math" alttext="q" display="inline"><semantics id="A6.SS1.p1.2.m2.1a"><mi id="A6.SS1.p1.2.m2.1.1" xref="A6.SS1.p1.2.m2.1.1.cmml">q</mi><annotation-xml encoding="MathML-Content" id="A6.SS1.p1.2.m2.1b"><ci id="A6.SS1.p1.2.m2.1.1.cmml" xref="A6.SS1.p1.2.m2.1.1">𝑞</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS1.p1.2.m2.1c">q</annotation></semantics></math> and model response <math id="A6.SS1.p1.3.m3.1" class="ltx_Math" alttext="\mathcal{S}" display="inline"><semantics id="A6.SS1.p1.3.m3.1a"><mi class="ltx_font_mathcaligraphic" id="A6.SS1.p1.3.m3.1.1" xref="A6.SS1.p1.3.m3.1.1.cmml">𝒮</mi><annotation-xml encoding="MathML-Content" id="A6.SS1.p1.3.m3.1b"><ci id="A6.SS1.p1.3.m3.1.1.cmml" xref="A6.SS1.p1.3.m3.1.1">𝒮</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS1.p1.3.m3.1c">\mathcal{S}</annotation></semantics></math> to the worker and ask them to rate their agreement with
the statement "The response is a helpful and informative answer to the query" on a Likert scale of 1-5,
corresponding to <span id="A6.SS1.p1.3.1" class="ltx_text ltx_font_italic">Strongly Disagree</span>, <span id="A6.SS1.p1.3.2" class="ltx_text ltx_font_italic">Disagree</span>, <span id="A6.SS1.p1.3.3" class="ltx_text ltx_font_italic">Neutral</span>, <span id="A6.SS1.p1.3.4" class="ltx_text ltx_font_italic">Agree</span>, and <span id="A6.SS1.p1.3.5" class="ltx_text ltx_font_italic">Strongly Agree</span>.</p>
</div>
</section>
<section id="A6.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">F.2 </span>Citation Recall</h3>

<div id="A6.SS2.p1" class="ltx_para">
<p id="A6.SS2.p1.3" class="ltx_p">The annotators are shown the question <math id="A6.SS2.p1.1.m1.1" class="ltx_Math" alttext="q" display="inline"><semantics id="A6.SS2.p1.1.m1.1a"><mi id="A6.SS2.p1.1.m1.1.1" xref="A6.SS2.p1.1.m1.1.1.cmml">q</mi><annotation-xml encoding="MathML-Content" id="A6.SS2.p1.1.m1.1b"><ci id="A6.SS2.p1.1.m1.1.1.cmml" xref="A6.SS2.p1.1.m1.1.1">𝑞</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.p1.1.m1.1c">q</annotation></semantics></math>, the statement <math id="A6.SS2.p1.2.m2.1" class="ltx_Math" alttext="s_{i}" display="inline"><semantics id="A6.SS2.p1.2.m2.1a"><msub id="A6.SS2.p1.2.m2.1.1" xref="A6.SS2.p1.2.m2.1.1.cmml"><mi id="A6.SS2.p1.2.m2.1.1.2" xref="A6.SS2.p1.2.m2.1.1.2.cmml">s</mi><mi id="A6.SS2.p1.2.m2.1.1.3" xref="A6.SS2.p1.2.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="A6.SS2.p1.2.m2.1b"><apply id="A6.SS2.p1.2.m2.1.1.cmml" xref="A6.SS2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="A6.SS2.p1.2.m2.1.1.1.cmml" xref="A6.SS2.p1.2.m2.1.1">subscript</csymbol><ci id="A6.SS2.p1.2.m2.1.1.2.cmml" xref="A6.SS2.p1.2.m2.1.1.2">𝑠</ci><ci id="A6.SS2.p1.2.m2.1.1.3.cmml" xref="A6.SS2.p1.2.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.p1.2.m2.1c">s_{i}</annotation></semantics></math>, and all of its citations <math id="A6.SS2.p1.3.m3.1" class="ltx_Math" alttext="\mathcal{C}_{i}" display="inline"><semantics id="A6.SS2.p1.3.m3.1a"><msub id="A6.SS2.p1.3.m3.1.1" xref="A6.SS2.p1.3.m3.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A6.SS2.p1.3.m3.1.1.2" xref="A6.SS2.p1.3.m3.1.1.2.cmml">𝒞</mi><mi id="A6.SS2.p1.3.m3.1.1.3" xref="A6.SS2.p1.3.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="A6.SS2.p1.3.m3.1b"><apply id="A6.SS2.p1.3.m3.1.1.cmml" xref="A6.SS2.p1.3.m3.1.1"><csymbol cd="ambiguous" id="A6.SS2.p1.3.m3.1.1.1.cmml" xref="A6.SS2.p1.3.m3.1.1">subscript</csymbol><ci id="A6.SS2.p1.3.m3.1.1.2.cmml" xref="A6.SS2.p1.3.m3.1.1.2">𝒞</ci><ci id="A6.SS2.p1.3.m3.1.1.3.cmml" xref="A6.SS2.p1.3.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS2.p1.3.m3.1c">\mathcal{C}_{i}</annotation></semantics></math>, and they rate if the joint set of citations fully support the statement (recall=1) or if they do not support all the claims (recall=0).
We calculate the overall recall score
for the generation by taking an average of all the statements’ recall scores.</p>
</div>
</section>
<section id="A6.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">F.3 </span>Citation Precision</h3>

<div id="A6.SS3.p1" class="ltx_para">
<p id="A6.SS3.p1.11" class="ltx_p">We show the question <math id="A6.SS3.p1.1.m1.1" class="ltx_Math" alttext="q" display="inline"><semantics id="A6.SS3.p1.1.m1.1a"><mi id="A6.SS3.p1.1.m1.1.1" xref="A6.SS3.p1.1.m1.1.1.cmml">q</mi><annotation-xml encoding="MathML-Content" id="A6.SS3.p1.1.m1.1b"><ci id="A6.SS3.p1.1.m1.1.1.cmml" xref="A6.SS3.p1.1.m1.1.1">𝑞</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS3.p1.1.m1.1c">q</annotation></semantics></math> and a pair of a statement <math id="A6.SS3.p1.2.m2.1" class="ltx_Math" alttext="s_{i}" display="inline"><semantics id="A6.SS3.p1.2.m2.1a"><msub id="A6.SS3.p1.2.m2.1.1" xref="A6.SS3.p1.2.m2.1.1.cmml"><mi id="A6.SS3.p1.2.m2.1.1.2" xref="A6.SS3.p1.2.m2.1.1.2.cmml">s</mi><mi id="A6.SS3.p1.2.m2.1.1.3" xref="A6.SS3.p1.2.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="A6.SS3.p1.2.m2.1b"><apply id="A6.SS3.p1.2.m2.1.1.cmml" xref="A6.SS3.p1.2.m2.1.1"><csymbol cd="ambiguous" id="A6.SS3.p1.2.m2.1.1.1.cmml" xref="A6.SS3.p1.2.m2.1.1">subscript</csymbol><ci id="A6.SS3.p1.2.m2.1.1.2.cmml" xref="A6.SS3.p1.2.m2.1.1.2">𝑠</ci><ci id="A6.SS3.p1.2.m2.1.1.3.cmml" xref="A6.SS3.p1.2.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS3.p1.2.m2.1c">s_{i}</annotation></semantics></math> and one of its citation <math id="A6.SS3.p1.3.m3.2" class="ltx_Math" alttext="c_{i,j}\in\mathcal{C}_{i}" display="inline"><semantics id="A6.SS3.p1.3.m3.2a"><mrow id="A6.SS3.p1.3.m3.2.3" xref="A6.SS3.p1.3.m3.2.3.cmml"><msub id="A6.SS3.p1.3.m3.2.3.2" xref="A6.SS3.p1.3.m3.2.3.2.cmml"><mi id="A6.SS3.p1.3.m3.2.3.2.2" xref="A6.SS3.p1.3.m3.2.3.2.2.cmml">c</mi><mrow id="A6.SS3.p1.3.m3.2.2.2.4" xref="A6.SS3.p1.3.m3.2.2.2.3.cmml"><mi id="A6.SS3.p1.3.m3.1.1.1.1" xref="A6.SS3.p1.3.m3.1.1.1.1.cmml">i</mi><mo id="A6.SS3.p1.3.m3.2.2.2.4.1" xref="A6.SS3.p1.3.m3.2.2.2.3.cmml">,</mo><mi id="A6.SS3.p1.3.m3.2.2.2.2" xref="A6.SS3.p1.3.m3.2.2.2.2.cmml">j</mi></mrow></msub><mo id="A6.SS3.p1.3.m3.2.3.1" xref="A6.SS3.p1.3.m3.2.3.1.cmml">∈</mo><msub id="A6.SS3.p1.3.m3.2.3.3" xref="A6.SS3.p1.3.m3.2.3.3.cmml"><mi class="ltx_font_mathcaligraphic" id="A6.SS3.p1.3.m3.2.3.3.2" xref="A6.SS3.p1.3.m3.2.3.3.2.cmml">𝒞</mi><mi id="A6.SS3.p1.3.m3.2.3.3.3" xref="A6.SS3.p1.3.m3.2.3.3.3.cmml">i</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="A6.SS3.p1.3.m3.2b"><apply id="A6.SS3.p1.3.m3.2.3.cmml" xref="A6.SS3.p1.3.m3.2.3"><in id="A6.SS3.p1.3.m3.2.3.1.cmml" xref="A6.SS3.p1.3.m3.2.3.1"></in><apply id="A6.SS3.p1.3.m3.2.3.2.cmml" xref="A6.SS3.p1.3.m3.2.3.2"><csymbol cd="ambiguous" id="A6.SS3.p1.3.m3.2.3.2.1.cmml" xref="A6.SS3.p1.3.m3.2.3.2">subscript</csymbol><ci id="A6.SS3.p1.3.m3.2.3.2.2.cmml" xref="A6.SS3.p1.3.m3.2.3.2.2">𝑐</ci><list id="A6.SS3.p1.3.m3.2.2.2.3.cmml" xref="A6.SS3.p1.3.m3.2.2.2.4"><ci id="A6.SS3.p1.3.m3.1.1.1.1.cmml" xref="A6.SS3.p1.3.m3.1.1.1.1">𝑖</ci><ci id="A6.SS3.p1.3.m3.2.2.2.2.cmml" xref="A6.SS3.p1.3.m3.2.2.2.2">𝑗</ci></list></apply><apply id="A6.SS3.p1.3.m3.2.3.3.cmml" xref="A6.SS3.p1.3.m3.2.3.3"><csymbol cd="ambiguous" id="A6.SS3.p1.3.m3.2.3.3.1.cmml" xref="A6.SS3.p1.3.m3.2.3.3">subscript</csymbol><ci id="A6.SS3.p1.3.m3.2.3.3.2.cmml" xref="A6.SS3.p1.3.m3.2.3.3.2">𝒞</ci><ci id="A6.SS3.p1.3.m3.2.3.3.3.cmml" xref="A6.SS3.p1.3.m3.2.3.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS3.p1.3.m3.2c">c_{i,j}\in\mathcal{C}_{i}</annotation></semantics></math>
to the annotator.
We ask the annotator if the citation <span id="A6.SS3.p1.11.1" class="ltx_text ltx_font_italic">fully supports</span>, <span id="A6.SS3.p1.11.2" class="ltx_text ltx_font_italic">partially supports</span>, or <span id="A6.SS3.p1.11.3" class="ltx_text ltx_font_italic">does not support</span> the factual claims in <math id="A6.SS3.p1.4.m4.1" class="ltx_Math" alttext="s_{i}" display="inline"><semantics id="A6.SS3.p1.4.m4.1a"><msub id="A6.SS3.p1.4.m4.1.1" xref="A6.SS3.p1.4.m4.1.1.cmml"><mi id="A6.SS3.p1.4.m4.1.1.2" xref="A6.SS3.p1.4.m4.1.1.2.cmml">s</mi><mi id="A6.SS3.p1.4.m4.1.1.3" xref="A6.SS3.p1.4.m4.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="A6.SS3.p1.4.m4.1b"><apply id="A6.SS3.p1.4.m4.1.1.cmml" xref="A6.SS3.p1.4.m4.1.1"><csymbol cd="ambiguous" id="A6.SS3.p1.4.m4.1.1.1.cmml" xref="A6.SS3.p1.4.m4.1.1">subscript</csymbol><ci id="A6.SS3.p1.4.m4.1.1.2.cmml" xref="A6.SS3.p1.4.m4.1.1.2">𝑠</ci><ci id="A6.SS3.p1.4.m4.1.1.3.cmml" xref="A6.SS3.p1.4.m4.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS3.p1.4.m4.1c">s_{i}</annotation></semantics></math>.
Citation <math id="A6.SS3.p1.5.m5.2" class="ltx_Math" alttext="c_{i,j}" display="inline"><semantics id="A6.SS3.p1.5.m5.2a"><msub id="A6.SS3.p1.5.m5.2.3" xref="A6.SS3.p1.5.m5.2.3.cmml"><mi id="A6.SS3.p1.5.m5.2.3.2" xref="A6.SS3.p1.5.m5.2.3.2.cmml">c</mi><mrow id="A6.SS3.p1.5.m5.2.2.2.4" xref="A6.SS3.p1.5.m5.2.2.2.3.cmml"><mi id="A6.SS3.p1.5.m5.1.1.1.1" xref="A6.SS3.p1.5.m5.1.1.1.1.cmml">i</mi><mo id="A6.SS3.p1.5.m5.2.2.2.4.1" xref="A6.SS3.p1.5.m5.2.2.2.3.cmml">,</mo><mi id="A6.SS3.p1.5.m5.2.2.2.2" xref="A6.SS3.p1.5.m5.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A6.SS3.p1.5.m5.2b"><apply id="A6.SS3.p1.5.m5.2.3.cmml" xref="A6.SS3.p1.5.m5.2.3"><csymbol cd="ambiguous" id="A6.SS3.p1.5.m5.2.3.1.cmml" xref="A6.SS3.p1.5.m5.2.3">subscript</csymbol><ci id="A6.SS3.p1.5.m5.2.3.2.cmml" xref="A6.SS3.p1.5.m5.2.3.2">𝑐</ci><list id="A6.SS3.p1.5.m5.2.2.2.3.cmml" xref="A6.SS3.p1.5.m5.2.2.2.4"><ci id="A6.SS3.p1.5.m5.1.1.1.1.cmml" xref="A6.SS3.p1.5.m5.1.1.1.1">𝑖</ci><ci id="A6.SS3.p1.5.m5.2.2.2.2.cmml" xref="A6.SS3.p1.5.m5.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS3.p1.5.m5.2c">c_{i,j}</annotation></semantics></math> has a citation precision of <math id="A6.SS3.p1.6.m6.1" class="ltx_Math" alttext="1" display="inline"><semantics id="A6.SS3.p1.6.m6.1a"><mn id="A6.SS3.p1.6.m6.1.1" xref="A6.SS3.p1.6.m6.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="A6.SS3.p1.6.m6.1b"><cn type="integer" id="A6.SS3.p1.6.m6.1.1.cmml" xref="A6.SS3.p1.6.m6.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="A6.SS3.p1.6.m6.1c">1</annotation></semantics></math> if <math id="A6.SS3.p1.7.m7.1" class="ltx_Math" alttext="s_{i}" display="inline"><semantics id="A6.SS3.p1.7.m7.1a"><msub id="A6.SS3.p1.7.m7.1.1" xref="A6.SS3.p1.7.m7.1.1.cmml"><mi id="A6.SS3.p1.7.m7.1.1.2" xref="A6.SS3.p1.7.m7.1.1.2.cmml">s</mi><mi id="A6.SS3.p1.7.m7.1.1.3" xref="A6.SS3.p1.7.m7.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="A6.SS3.p1.7.m7.1b"><apply id="A6.SS3.p1.7.m7.1.1.cmml" xref="A6.SS3.p1.7.m7.1.1"><csymbol cd="ambiguous" id="A6.SS3.p1.7.m7.1.1.1.cmml" xref="A6.SS3.p1.7.m7.1.1">subscript</csymbol><ci id="A6.SS3.p1.7.m7.1.1.2.cmml" xref="A6.SS3.p1.7.m7.1.1.2">𝑠</ci><ci id="A6.SS3.p1.7.m7.1.1.3.cmml" xref="A6.SS3.p1.7.m7.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS3.p1.7.m7.1c">s_{i}</annotation></semantics></math> has a recall of <math id="A6.SS3.p1.8.m8.1" class="ltx_Math" alttext="1" display="inline"><semantics id="A6.SS3.p1.8.m8.1a"><mn id="A6.SS3.p1.8.m8.1.1" xref="A6.SS3.p1.8.m8.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="A6.SS3.p1.8.m8.1b"><cn type="integer" id="A6.SS3.p1.8.m8.1.1.cmml" xref="A6.SS3.p1.8.m8.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="A6.SS3.p1.8.m8.1c">1</annotation></semantics></math>, and
<math id="A6.SS3.p1.9.m9.2" class="ltx_Math" alttext="c_{i,j}" display="inline"><semantics id="A6.SS3.p1.9.m9.2a"><msub id="A6.SS3.p1.9.m9.2.3" xref="A6.SS3.p1.9.m9.2.3.cmml"><mi id="A6.SS3.p1.9.m9.2.3.2" xref="A6.SS3.p1.9.m9.2.3.2.cmml">c</mi><mrow id="A6.SS3.p1.9.m9.2.2.2.4" xref="A6.SS3.p1.9.m9.2.2.2.3.cmml"><mi id="A6.SS3.p1.9.m9.1.1.1.1" xref="A6.SS3.p1.9.m9.1.1.1.1.cmml">i</mi><mo id="A6.SS3.p1.9.m9.2.2.2.4.1" xref="A6.SS3.p1.9.m9.2.2.2.3.cmml">,</mo><mi id="A6.SS3.p1.9.m9.2.2.2.2" xref="A6.SS3.p1.9.m9.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A6.SS3.p1.9.m9.2b"><apply id="A6.SS3.p1.9.m9.2.3.cmml" xref="A6.SS3.p1.9.m9.2.3"><csymbol cd="ambiguous" id="A6.SS3.p1.9.m9.2.3.1.cmml" xref="A6.SS3.p1.9.m9.2.3">subscript</csymbol><ci id="A6.SS3.p1.9.m9.2.3.2.cmml" xref="A6.SS3.p1.9.m9.2.3.2">𝑐</ci><list id="A6.SS3.p1.9.m9.2.2.2.3.cmml" xref="A6.SS3.p1.9.m9.2.2.2.4"><ci id="A6.SS3.p1.9.m9.1.1.1.1.cmml" xref="A6.SS3.p1.9.m9.1.1.1.1">𝑖</ci><ci id="A6.SS3.p1.9.m9.2.2.2.2.cmml" xref="A6.SS3.p1.9.m9.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS3.p1.9.m9.2c">c_{i,j}</annotation></semantics></math> fully or partially supports <math id="A6.SS3.p1.10.m10.1" class="ltx_Math" alttext="s_{i}" display="inline"><semantics id="A6.SS3.p1.10.m10.1a"><msub id="A6.SS3.p1.10.m10.1.1" xref="A6.SS3.p1.10.m10.1.1.cmml"><mi id="A6.SS3.p1.10.m10.1.1.2" xref="A6.SS3.p1.10.m10.1.1.2.cmml">s</mi><mi id="A6.SS3.p1.10.m10.1.1.3" xref="A6.SS3.p1.10.m10.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="A6.SS3.p1.10.m10.1b"><apply id="A6.SS3.p1.10.m10.1.1.cmml" xref="A6.SS3.p1.10.m10.1.1"><csymbol cd="ambiguous" id="A6.SS3.p1.10.m10.1.1.1.cmml" xref="A6.SS3.p1.10.m10.1.1">subscript</csymbol><ci id="A6.SS3.p1.10.m10.1.1.2.cmml" xref="A6.SS3.p1.10.m10.1.1.2">𝑠</ci><ci id="A6.SS3.p1.10.m10.1.1.3.cmml" xref="A6.SS3.p1.10.m10.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS3.p1.10.m10.1c">s_{i}</annotation></semantics></math>.
Finally, we take an average of precision scores of all citations in the statement <math id="A6.SS3.p1.11.m11.1" class="ltx_Math" alttext="\mathcal{S}" display="inline"><semantics id="A6.SS3.p1.11.m11.1a"><mi class="ltx_font_mathcaligraphic" id="A6.SS3.p1.11.m11.1.1" xref="A6.SS3.p1.11.m11.1.1.cmml">𝒮</mi><annotation-xml encoding="MathML-Content" id="A6.SS3.p1.11.m11.1b"><ci id="A6.SS3.p1.11.m11.1.1.cmml" xref="A6.SS3.p1.11.m11.1.1">𝒮</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.SS3.p1.11.m11.1c">\mathcal{S}</annotation></semantics></math> to obtain the citation precision score.</p>
</div>
<figure id="A6.T12" class="ltx_table">
<table id="A6.T12.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A6.T12.1.1.1" class="ltx_tr">
<th id="A6.T12.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="A6.T12.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">R@1</th>
<th id="A6.T12.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">R@3</th>
<th id="A6.T12.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">R@5</th>
<th id="A6.T12.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">R@20</th>
<th id="A6.T12.1.1.1.6" class="ltx_td ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt">R@100</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A6.T12.1.2.1" class="ltx_tr">
<th id="A6.T12.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">DPR</th>
<td id="A6.T12.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t">29.6</td>
<td id="A6.T12.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t">44.5</td>
<td id="A6.T12.1.2.1.4" class="ltx_td ltx_align_center ltx_border_t">51.5</td>
<td id="A6.T12.1.2.1.5" class="ltx_td ltx_align_center ltx_border_t">64.6</td>
<td id="A6.T12.1.2.1.6" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">74.1</td>
</tr>
<tr id="A6.T12.1.3.2" class="ltx_tr">
<th id="A6.T12.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">GTR</th>
<td id="A6.T12.1.3.2.2" class="ltx_td ltx_align_center">35.1</td>
<td id="A6.T12.1.3.2.3" class="ltx_td ltx_align_center">50.7</td>
<td id="A6.T12.1.3.2.4" class="ltx_td ltx_align_center">56.8</td>
<td id="A6.T12.1.3.2.5" class="ltx_td ltx_align_center">70.3</td>
<td id="A6.T12.1.3.2.6" class="ltx_td ltx_nopad_r ltx_align_center">78.4</td>
</tr>
<tr id="A6.T12.1.4.3" class="ltx_tr">
<th id="A6.T12.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">Oracle</th>
<td id="A6.T12.1.4.3.2" class="ltx_td ltx_align_center ltx_border_bb">63.8</td>
<td id="A6.T12.1.4.3.3" class="ltx_td ltx_align_center ltx_border_bb">72.8</td>
<td id="A6.T12.1.4.3.4" class="ltx_td ltx_align_center ltx_border_bb">78.4</td>
<td id="A6.T12.1.4.3.5" class="ltx_td ltx_align_center ltx_border_bb">-</td>
<td id="A6.T12.1.4.3.6" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb">-</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 12: </span>Retrieval results for ASQA (EM recall).</figcaption>
</figure>
<figure id="A6.T13" class="ltx_table">
<table id="A6.T13.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A6.T13.1.1.1" class="ltx_tr">
<th id="A6.T13.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="A6.T13.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">R@1</th>
<th id="A6.T13.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">R@3</th>
<th id="A6.T13.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">R@5</th>
<th id="A6.T13.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">R@20</th>
<th id="A6.T13.1.1.1.6" class="ltx_td ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt">R@100</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A6.T13.1.2.1" class="ltx_tr">
<th id="A6.T13.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">DPR</th>
<td id="A6.T13.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t">6.7</td>
<td id="A6.T13.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t">13.5</td>
<td id="A6.T13.1.2.1.4" class="ltx_td ltx_align_center ltx_border_t">17.6</td>
<td id="A6.T13.1.2.1.5" class="ltx_td ltx_align_center ltx_border_t">30.4</td>
<td id="A6.T13.1.2.1.6" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">47.6</td>
</tr>
<tr id="A6.T13.1.3.2" class="ltx_tr">
<th id="A6.T13.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">GTR</th>
<td id="A6.T13.1.3.2.2" class="ltx_td ltx_align_center">14.6</td>
<td id="A6.T13.1.3.2.3" class="ltx_td ltx_align_center">24.7</td>
<td id="A6.T13.1.3.2.4" class="ltx_td ltx_align_center">31.6</td>
<td id="A6.T13.1.3.2.5" class="ltx_td ltx_align_center">49.7</td>
<td id="A6.T13.1.3.2.6" class="ltx_td ltx_nopad_r ltx_align_center">65.6</td>
</tr>
<tr id="A6.T13.1.4.3" class="ltx_tr">
<th id="A6.T13.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">Oracle</th>
<td id="A6.T13.1.4.3.2" class="ltx_td ltx_align_center ltx_border_bb">44.3</td>
<td id="A6.T13.1.4.3.3" class="ltx_td ltx_align_center ltx_border_bb">58.7</td>
<td id="A6.T13.1.4.3.4" class="ltx_td ltx_align_center ltx_border_bb">65.6</td>
<td id="A6.T13.1.4.3.5" class="ltx_td ltx_align_center ltx_border_bb">-</td>
<td id="A6.T13.1.4.3.6" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb">-</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 13: </span>Retrieval results for QAMPARI (recall-5).</figcaption>
</figure>
<figure id="A6.T14" class="ltx_table">
<table id="A6.T14.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A6.T14.1.1.1" class="ltx_tr">
<th id="A6.T14.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="A6.T14.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">R@1</th>
<th id="A6.T14.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">R@3</th>
<th id="A6.T14.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">R@5</th>
<th id="A6.T14.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">R@20</th>
<th id="A6.T14.1.1.1.6" class="ltx_td ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt">R@100</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A6.T14.1.2.1" class="ltx_tr">
<th id="A6.T14.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">BM25</th>
<td id="A6.T14.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t">3.0</td>
<td id="A6.T14.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t">6.6</td>
<td id="A6.T14.1.2.1.4" class="ltx_td ltx_align_center ltx_border_t">9.6</td>
<td id="A6.T14.1.2.1.5" class="ltx_td ltx_align_center ltx_border_t">19.3</td>
<td id="A6.T14.1.2.1.6" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">31.8</td>
</tr>
<tr id="A6.T14.1.3.2" class="ltx_tr">
<th id="A6.T14.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">Oracle</th>
<td id="A6.T14.1.3.2.2" class="ltx_td ltx_align_center ltx_border_bb">25.3</td>
<td id="A6.T14.1.3.2.3" class="ltx_td ltx_align_center ltx_border_bb">29.7</td>
<td id="A6.T14.1.3.2.4" class="ltx_td ltx_align_center ltx_border_bb">31.8</td>
<td id="A6.T14.1.3.2.5" class="ltx_td ltx_align_center ltx_border_bb">-</td>
<td id="A6.T14.1.3.2.6" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb">-</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 14: </span>Retrieval results for ELI5 (claim recall).</figcaption>
</figure>
</section>
</section>
<section id="A7" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix G </span>More Experiments</h2>

<figure id="A7.T15" class="ltx_table">
<div id="A7.T15.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:424.9pt;height:139.2pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(75.1pt,-24.6pt) scale(1.54681766534833,1.54681766534833) ;">
<table id="A7.T15.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A7.T15.1.1.1.1" class="ltx_tr">
<th id="A7.T15.1.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="A7.T15.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A7.T15.1.1.1.1.2.1" class="ltx_text ltx_font_bold">Fluency</span></th>
<th id="A7.T15.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A7.T15.1.1.1.1.3.1" class="ltx_text ltx_font_bold">Correct.</span></th>
<th id="A7.T15.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2"><span id="A7.T15.1.1.1.1.4.1" class="ltx_text ltx_font_bold">Citation</span></th>
</tr>
<tr id="A7.T15.1.1.2.2" class="ltx_tr">
<th id="A7.T15.1.1.2.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="A7.T15.1.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">(MAUVE)</th>
<th id="A7.T15.1.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">(EM Rec.)</th>
<th id="A7.T15.1.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Rec.</th>
<th id="A7.T15.1.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Prec.</th>
</tr>
<tr id="A7.T15.1.1.3.3" class="ltx_tr" style="background-color:#ECECEC;">
<th id="A7.T15.1.1.3.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" colspan="5"><span id="A7.T15.1.1.3.3.1.1" class="ltx_text ltx_font_bold" style="background-color:#ECECEC;">ChatGPT<span id="A7.T15.1.1.3.3.1.1.1" class="ltx_text ltx_font_medium"> (<span id="A7.T15.1.1.3.3.1.1.1.1" class="ltx_text ltx_font_smallcaps">Vanilla</span>, 5-doc)</span></span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A7.T15.1.1.4.1" class="ltx_tr">
<th id="A7.T15.1.1.4.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Short instruction</th>
<td id="A7.T15.1.1.4.1.2" class="ltx_td ltx_align_center">64.1</td>
<td id="A7.T15.1.1.4.1.3" class="ltx_td ltx_align_center">39.5</td>
<td id="A7.T15.1.1.4.1.4" class="ltx_td ltx_align_center">69.6</td>
<td id="A7.T15.1.1.4.1.5" class="ltx_td ltx_align_center">73.2</td>
</tr>
<tr id="A7.T15.1.1.5.2" class="ltx_tr">
<th id="A7.T15.1.1.5.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">Full instruction</th>
<td id="A7.T15.1.1.5.2.2" class="ltx_td ltx_align_center ltx_border_bb">66.6</td>
<td id="A7.T15.1.1.5.2.3" class="ltx_td ltx_align_center ltx_border_bb">40.4</td>
<td id="A7.T15.1.1.5.2.4" class="ltx_td ltx_align_center ltx_border_bb">73.6</td>
<td id="A7.T15.1.1.5.2.5" class="ltx_td ltx_align_center ltx_border_bb">72.5</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 15: </span>
Effect of different instructions on ASQA.
</figcaption>
</figure>
<figure id="A7.T16" class="ltx_table">
<div id="A7.T16.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:424.9pt;height:190.6pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(92.1pt,-41.3pt) scale(1.76488563008856,1.76488563008856) ;">
<table id="A7.T16.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A7.T16.1.1.1.1" class="ltx_tr">
<th id="A7.T16.1.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="A7.T16.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A7.T16.1.1.1.1.2.1" class="ltx_text ltx_font_bold">Fluency</span></th>
<th id="A7.T16.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A7.T16.1.1.1.1.3.1" class="ltx_text ltx_font_bold">Correct.</span></th>
<th id="A7.T16.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2"><span id="A7.T16.1.1.1.1.4.1" class="ltx_text ltx_font_bold">Citation</span></th>
</tr>
<tr id="A7.T16.1.1.2.2" class="ltx_tr">
<th id="A7.T16.1.1.2.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="A7.T16.1.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">(MAUVE)</th>
<th id="A7.T16.1.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">(EM)</th>
<th id="A7.T16.1.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Rec.</th>
<th id="A7.T16.1.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Prec.</th>
</tr>
<tr id="A7.T16.1.1.3.3" class="ltx_tr" style="background-color:#ECECEC;">
<th id="A7.T16.1.1.3.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" colspan="5"><span id="A7.T16.1.1.3.3.1.1" class="ltx_text ltx_font_bold" style="background-color:#ECECEC;">ChatGPT<span id="A7.T16.1.1.3.3.1.1.1" class="ltx_text ltx_font_medium"> (<span id="A7.T16.1.1.3.3.1.1.1.1" class="ltx_text ltx_font_smallcaps">Vanilla</span>)</span></span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A7.T16.1.1.4.1" class="ltx_tr">
<th id="A7.T16.1.1.4.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">#demo = 0</th>
<td id="A7.T16.1.1.4.1.2" class="ltx_td ltx_align_center">74.5</td>
<td id="A7.T16.1.1.4.1.3" class="ltx_td ltx_align_center">41.9</td>
<td id="A7.T16.1.1.4.1.4" class="ltx_td ltx_align_center">69.3</td>
<td id="A7.T16.1.1.4.1.5" class="ltx_td ltx_align_center">73.4</td>
</tr>
<tr id="A7.T16.1.1.5.2" class="ltx_tr">
<th id="A7.T16.1.1.5.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">#demo = 1</th>
<td id="A7.T16.1.1.5.2.2" class="ltx_td ltx_align_center">68.9</td>
<td id="A7.T16.1.1.5.2.3" class="ltx_td ltx_align_center">39.8</td>
<td id="A7.T16.1.1.5.2.4" class="ltx_td ltx_align_center">74.6</td>
<td id="A7.T16.1.1.5.2.5" class="ltx_td ltx_align_center">73.2</td>
</tr>
<tr id="A7.T16.1.1.6.3" class="ltx_tr">
<th id="A7.T16.1.1.6.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">#demo = 2</th>
<td id="A7.T16.1.1.6.3.2" class="ltx_td ltx_align_center ltx_border_bb">66.6</td>
<td id="A7.T16.1.1.6.3.3" class="ltx_td ltx_align_center ltx_border_bb">40.4</td>
<td id="A7.T16.1.1.6.3.4" class="ltx_td ltx_align_center ltx_border_bb">73.6</td>
<td id="A7.T16.1.1.6.3.5" class="ltx_td ltx_align_center ltx_border_bb">72.5</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 16: </span>
Different demonstrations on ASQA.
</figcaption>
</figure>
<figure id="A7.T17" class="ltx_table">
<div id="A7.T17.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:424.9pt;height:113.6pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(77.8pt,-20.8pt) scale(1.57825536899082,1.57825536899082) ;">
<table id="A7.T17.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A7.T17.1.1.1.1" class="ltx_tr">
<th id="A7.T17.1.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="A7.T17.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A7.T17.1.1.1.1.2.1" class="ltx_text ltx_font_bold">Fluency</span></th>
<th id="A7.T17.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A7.T17.1.1.1.1.3.1" class="ltx_text ltx_font_bold">Correct.</span></th>
<th id="A7.T17.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2"><span id="A7.T17.1.1.1.1.4.1" class="ltx_text ltx_font_bold">Citation</span></th>
</tr>
<tr id="A7.T17.1.1.2.2" class="ltx_tr">
<th id="A7.T17.1.1.2.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="A7.T17.1.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">(MAUVE)</th>
<th id="A7.T17.1.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">(EM Rec.)</th>
<th id="A7.T17.1.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Rec.</th>
<th id="A7.T17.1.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Prec.</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A7.T17.1.1.3.1" class="ltx_tr">
<th id="A7.T17.1.1.3.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">ChatGPT</th>
<td id="A7.T17.1.1.3.1.2" class="ltx_td ltx_align_center ltx_border_t">66.6</td>
<td id="A7.T17.1.1.3.1.3" class="ltx_td ltx_align_center ltx_border_t">40.4</td>
<td id="A7.T17.1.1.3.1.4" class="ltx_td ltx_align_center ltx_border_t">73.6</td>
<td id="A7.T17.1.1.3.1.5" class="ltx_td ltx_align_center ltx_border_t">72.5</td>
</tr>
<tr id="A7.T17.1.1.4.2" class="ltx_tr">
<th id="A7.T17.1.1.4.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">FiD + <span id="A7.T17.1.1.4.2.1.1" class="ltx_text ltx_font_smallcaps">PostCite</span>
</th>
<td id="A7.T17.1.1.4.2.2" class="ltx_td ltx_align_center ltx_border_bb">75.8</td>
<td id="A7.T17.1.1.4.2.3" class="ltx_td ltx_align_center ltx_border_bb">28.4</td>
<td id="A7.T17.1.1.4.2.4" class="ltx_td ltx_align_center ltx_border_bb">58.1</td>
<td id="A7.T17.1.1.4.2.5" class="ltx_td ltx_align_center ltx_border_bb">58.0</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 17: </span>
Comparison of Fusion-in-Decoder with ChatGPT on ASQA.
Both models use top-5 GTR passages.
</figcaption>
</figure>
<figure id="A7.T18" class="ltx_table">
<div id="A7.T18.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:424.9pt;height:118.2pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(83.1pt,-23.1pt) scale(1.64230053571116,1.64230053571116) ;">
<table id="A7.T18.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A7.T18.1.1.1.1" class="ltx_tr">
<th id="A7.T18.1.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="A7.T18.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A7.T18.1.1.1.1.2.1" class="ltx_text ltx_font_bold">Fluency</span></th>
<th id="A7.T18.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A7.T18.1.1.1.1.3.1" class="ltx_text ltx_font_bold">Correct.</span></th>
<th id="A7.T18.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2"><span id="A7.T18.1.1.1.1.4.1" class="ltx_text ltx_font_bold">Citation</span></th>
</tr>
<tr id="A7.T18.1.1.2.2" class="ltx_tr">
<th id="A7.T18.1.1.2.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="A7.T18.1.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">(MAUVE)</th>
<th id="A7.T18.1.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">(Claim)</th>
<th id="A7.T18.1.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Rec.</th>
<th id="A7.T18.1.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Prec.</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A7.T18.1.1.3.1" class="ltx_tr">
<th id="A7.T18.1.1.3.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">ChatGPT</th>
<td id="A7.T18.1.1.3.1.2" class="ltx_td ltx_align_center ltx_border_t">57.2</td>
<td id="A7.T18.1.1.3.1.3" class="ltx_td ltx_align_center ltx_border_t">12.0</td>
<td id="A7.T18.1.1.3.1.4" class="ltx_td ltx_align_center ltx_border_t">51.1</td>
<td id="A7.T18.1.1.3.1.5" class="ltx_td ltx_align_center ltx_border_t">50.0</td>
</tr>
<tr id="A7.T18.1.1.4.2" class="ltx_tr">
<th id="A7.T18.1.1.4.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">FiD + <span id="A7.T18.1.1.4.2.1.1" class="ltx_text ltx_font_smallcaps">PostCite</span>
</th>
<td id="A7.T18.1.1.4.2.2" class="ltx_td ltx_align_center ltx_border_bb">25.2</td>
<td id="A7.T18.1.1.4.2.3" class="ltx_td ltx_align_center ltx_border_bb">4.4</td>
<td id="A7.T18.1.1.4.2.4" class="ltx_td ltx_align_center ltx_border_bb">39.3</td>
<td id="A7.T18.1.1.4.2.5" class="ltx_td ltx_align_center ltx_border_bb">39.3</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 18: </span>
Comparison of Fusion-in-Decoder with ChatGPT on ELI5.
Both models use top-5 GTR passages.
</figcaption>
</figure>
<section id="A7.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">G.1 </span>Retrieval Analysis</h3>

<section id="A7.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Oracle.</h4>

<div id="A7.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="A7.SS1.SSS0.Px1.p1.1" class="ltx_p">Since the original datasets do not contain gold passages
at the same granularity level as our setting (100-word passages),
we approximate gold passages by running the following algorithm on the top-100 retrieved passages.
We first calculate the recall score for each passage. Then, we sort the passages using their recall score and take the top 5 passages as our initial oracle set.
Finally, we iterate through all passages that were not initially in the oracle set and try to replace the passages in the oracle set in a greedy fashion: we calculate the change in the recall score of the oracle set for every possible replacement and proceed with the replacement that results in the largest recall improvement. The set of 5 oracle passages were able to match the recall scores of the top-100 retrieved passages.</p>
</div>
</section>
<section id="A7.SS1.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Detailed retrieval results.</h4>

<div id="A7.SS1.SSS0.Px2.p1" class="ltx_para">
<p id="A7.SS1.SSS0.Px2.p1.1" class="ltx_p">We show detailed retrieval results in Tables <a href="#A6.T12" title="Table 12 ‣ F.3 Citation Precision ‣ Appendix F Human Evaluation ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">12</span></a>, <a href="#A6.T13" title="Table 13 ‣ F.3 Citation Precision ‣ Appendix F Human Evaluation ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">13</span></a>, and <a href="#A6.T14" title="Table 14 ‣ F.3 Citation Precision ‣ Appendix F Human Evaluation ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">14</span></a>.</p>
</div>
</section>
</section>
<section id="A7.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">G.2 </span>Effect of Instructions</h3>

<div id="A7.SS2.p1" class="ltx_para">
<p id="A7.SS2.p1.1" class="ltx_p">Table <a href="#A7.T15" title="Table 15 ‣ Appendix G More Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">15</span></a> shows results of using a full instruction (Table <a href="#A8.T23" title="Table 23 ‣ Appendix H Prompts ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">23</span></a>) and a short version of the instruction (Table <a href="#A8.T24" title="Table 24 ‣ Appendix H Prompts ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">24</span></a>).
We see that the full version induces stronger correctness and citation recall, while the two instructions lead to similar citation precision.</p>
</div>
</section>
<section id="A7.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">G.3 </span>Effect of Demonstrations</h3>

<div id="A7.SS3.p1" class="ltx_para">
<p id="A7.SS3.p1.1" class="ltx_p">Table <a href="#A7.T16" title="Table 16 ‣ Appendix G More Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">16</span></a>
shows results on effect of different numbers of demonstrations.
We see that
numbers of demonstrations do not affect ChatGPT’s correctness but
using at least one demonstration
ensures high citation recall.
For the original LLaMA model, Table <a href="#A7.T16" title="Table 16 ‣ Appendix G More Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">16</span></a> shows the trend that more demonstrations lead to better performance.</p>
</div>
</section>
<section id="A7.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">G.4 </span>Fine-tuned Models</h3>

<div id="A7.SS4.p1" class="ltx_para">
<p id="A7.SS4.p1.1" class="ltx_p">To better understand the differences between fine-tuned models and prompted large language models, we train state-of-the-art question answering model, Fusion-in-Decoder (FiD; <cite class="ltx_cite ltx_citemacro_citet">Izacard and Grave (<a href="#bib.bib13" title="" class="ltx_ref">2021</a>)</cite>), and evaluate it in conjunction with <span id="A7.SS4.p1.1.1" class="ltx_text ltx_font_smallcaps">PostCite</span>.
Due to the lack of training data with citation annotation, we first train a T5-base FiD model for 5 epochs on the ASQA training set with a batch size of 64 and a learning rate of 1e-4.
During evaluation, we use <span id="A7.SS4.p1.1.2" class="ltx_text ltx_font_smallcaps">PostCite</span> to add citations to the output.
We also use <math id="A7.SS4.p1.1.m1.1" class="ltx_Math" alttext="k=5" display="inline"><semantics id="A7.SS4.p1.1.m1.1a"><mrow id="A7.SS4.p1.1.m1.1.1" xref="A7.SS4.p1.1.m1.1.1.cmml"><mi id="A7.SS4.p1.1.m1.1.1.2" xref="A7.SS4.p1.1.m1.1.1.2.cmml">k</mi><mo id="A7.SS4.p1.1.m1.1.1.1" xref="A7.SS4.p1.1.m1.1.1.1.cmml">=</mo><mn id="A7.SS4.p1.1.m1.1.1.3" xref="A7.SS4.p1.1.m1.1.1.3.cmml">5</mn></mrow><annotation-xml encoding="MathML-Content" id="A7.SS4.p1.1.m1.1b"><apply id="A7.SS4.p1.1.m1.1.1.cmml" xref="A7.SS4.p1.1.m1.1.1"><eq id="A7.SS4.p1.1.m1.1.1.1.cmml" xref="A7.SS4.p1.1.m1.1.1.1"></eq><ci id="A7.SS4.p1.1.m1.1.1.2.cmml" xref="A7.SS4.p1.1.m1.1.1.2">𝑘</ci><cn type="integer" id="A7.SS4.p1.1.m1.1.1.3.cmml" xref="A7.SS4.p1.1.m1.1.1.3">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A7.SS4.p1.1.m1.1c">k=5</annotation></semantics></math> passages during both training and evaluation of the FiD model.</p>
</div>
<div id="A7.SS4.p2" class="ltx_para">
<p id="A7.SS4.p2.1" class="ltx_p">Then, we evaluate this model on both ASQA (in-domain) and ELI5 (out-of-domain), and the results can be found in Tables <a href="#A7.T17" title="Table 17 ‣ Appendix G More Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">17</span></a> and <a href="#A7.T18" title="Table 18 ‣ Appendix G More Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">18</span></a>.
Note that this is not a direct comparison, as ALCE assumes only evaluation data available and uses only few-shot data for prompting.
As the results show, the FiD baseline still significantly lags behind prompting ChatGPT in both correctness and citation quality (even though it is trained on 4000+ examples).
When tested on another dataset (ELI5), FiD performs even worse, showing that it is challenging to solve the problem by fine-tuning a small pre-trained model.</p>
</div>
</section>
<section id="A7.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">G.5 </span>More Human Evaluation</h3>

<div id="A7.SS5.p1" class="ltx_para">
<p id="A7.SS5.p1.6" class="ltx_p">We evaluate the accuracy of our automatic metrics by treating the human annotations as gold labels.
For citation recall, ALCE achieves an accuracy of <math id="A7.SS5.p1.1.m1.1" class="ltx_Math" alttext="85.1\%" display="inline"><semantics id="A7.SS5.p1.1.m1.1a"><mrow id="A7.SS5.p1.1.m1.1.1" xref="A7.SS5.p1.1.m1.1.1.cmml"><mn id="A7.SS5.p1.1.m1.1.1.2" xref="A7.SS5.p1.1.m1.1.1.2.cmml">85.1</mn><mo id="A7.SS5.p1.1.m1.1.1.1" xref="A7.SS5.p1.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="A7.SS5.p1.1.m1.1b"><apply id="A7.SS5.p1.1.m1.1.1.cmml" xref="A7.SS5.p1.1.m1.1.1"><csymbol cd="latexml" id="A7.SS5.p1.1.m1.1.1.1.cmml" xref="A7.SS5.p1.1.m1.1.1.1">percent</csymbol><cn type="float" id="A7.SS5.p1.1.m1.1.1.2.cmml" xref="A7.SS5.p1.1.m1.1.1.2">85.1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A7.SS5.p1.1.m1.1c">85.1\%</annotation></semantics></math>; for citation precision, ALCE has an accuracy of <math id="A7.SS5.p1.2.m2.1" class="ltx_Math" alttext="77.6\%" display="inline"><semantics id="A7.SS5.p1.2.m2.1a"><mrow id="A7.SS5.p1.2.m2.1.1" xref="A7.SS5.p1.2.m2.1.1.cmml"><mn id="A7.SS5.p1.2.m2.1.1.2" xref="A7.SS5.p1.2.m2.1.1.2.cmml">77.6</mn><mo id="A7.SS5.p1.2.m2.1.1.1" xref="A7.SS5.p1.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="A7.SS5.p1.2.m2.1b"><apply id="A7.SS5.p1.2.m2.1.1.cmml" xref="A7.SS5.p1.2.m2.1.1"><csymbol cd="latexml" id="A7.SS5.p1.2.m2.1.1.1.cmml" xref="A7.SS5.p1.2.m2.1.1.1">percent</csymbol><cn type="float" id="A7.SS5.p1.2.m2.1.1.2.cmml" xref="A7.SS5.p1.2.m2.1.1.2">77.6</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A7.SS5.p1.2.m2.1c">77.6\%</annotation></semantics></math>.
Regarding detecting insufficient citations, ALCE has a recall of <math id="A7.SS5.p1.3.m3.1" class="ltx_Math" alttext="82.3\%" display="inline"><semantics id="A7.SS5.p1.3.m3.1a"><mrow id="A7.SS5.p1.3.m3.1.1" xref="A7.SS5.p1.3.m3.1.1.cmml"><mn id="A7.SS5.p1.3.m3.1.1.2" xref="A7.SS5.p1.3.m3.1.1.2.cmml">82.3</mn><mo id="A7.SS5.p1.3.m3.1.1.1" xref="A7.SS5.p1.3.m3.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="A7.SS5.p1.3.m3.1b"><apply id="A7.SS5.p1.3.m3.1.1.cmml" xref="A7.SS5.p1.3.m3.1.1"><csymbol cd="latexml" id="A7.SS5.p1.3.m3.1.1.1.cmml" xref="A7.SS5.p1.3.m3.1.1.1">percent</csymbol><cn type="float" id="A7.SS5.p1.3.m3.1.1.2.cmml" xref="A7.SS5.p1.3.m3.1.1.2">82.3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A7.SS5.p1.3.m3.1c">82.3\%</annotation></semantics></math> and a precision of <math id="A7.SS5.p1.4.m4.1" class="ltx_Math" alttext="84.2\%" display="inline"><semantics id="A7.SS5.p1.4.m4.1a"><mrow id="A7.SS5.p1.4.m4.1.1" xref="A7.SS5.p1.4.m4.1.1.cmml"><mn id="A7.SS5.p1.4.m4.1.1.2" xref="A7.SS5.p1.4.m4.1.1.2.cmml">84.2</mn><mo id="A7.SS5.p1.4.m4.1.1.1" xref="A7.SS5.p1.4.m4.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="A7.SS5.p1.4.m4.1b"><apply id="A7.SS5.p1.4.m4.1.1.cmml" xref="A7.SS5.p1.4.m4.1.1"><csymbol cd="latexml" id="A7.SS5.p1.4.m4.1.1.1.cmml" xref="A7.SS5.p1.4.m4.1.1.1">percent</csymbol><cn type="float" id="A7.SS5.p1.4.m4.1.1.2.cmml" xref="A7.SS5.p1.4.m4.1.1.2">84.2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A7.SS5.p1.4.m4.1c">84.2\%</annotation></semantics></math>;
regarding detecting “irrelevant” citations, ALCE has a recall of <math id="A7.SS5.p1.5.m5.1" class="ltx_Math" alttext="75.6\%" display="inline"><semantics id="A7.SS5.p1.5.m5.1a"><mrow id="A7.SS5.p1.5.m5.1.1" xref="A7.SS5.p1.5.m5.1.1.cmml"><mn id="A7.SS5.p1.5.m5.1.1.2" xref="A7.SS5.p1.5.m5.1.1.2.cmml">75.6</mn><mo id="A7.SS5.p1.5.m5.1.1.1" xref="A7.SS5.p1.5.m5.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="A7.SS5.p1.5.m5.1b"><apply id="A7.SS5.p1.5.m5.1.1.cmml" xref="A7.SS5.p1.5.m5.1.1"><csymbol cd="latexml" id="A7.SS5.p1.5.m5.1.1.1.cmml" xref="A7.SS5.p1.5.m5.1.1.1">percent</csymbol><cn type="float" id="A7.SS5.p1.5.m5.1.1.2.cmml" xref="A7.SS5.p1.5.m5.1.1.2">75.6</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A7.SS5.p1.5.m5.1c">75.6\%</annotation></semantics></math> and a precision of <math id="A7.SS5.p1.6.m6.1" class="ltx_Math" alttext="66.1\%" display="inline"><semantics id="A7.SS5.p1.6.m6.1a"><mrow id="A7.SS5.p1.6.m6.1.1" xref="A7.SS5.p1.6.m6.1.1.cmml"><mn id="A7.SS5.p1.6.m6.1.1.2" xref="A7.SS5.p1.6.m6.1.1.2.cmml">66.1</mn><mo id="A7.SS5.p1.6.m6.1.1.1" xref="A7.SS5.p1.6.m6.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="A7.SS5.p1.6.m6.1b"><apply id="A7.SS5.p1.6.m6.1.1.cmml" xref="A7.SS5.p1.6.m6.1.1"><csymbol cd="latexml" id="A7.SS5.p1.6.m6.1.1.1.cmml" xref="A7.SS5.p1.6.m6.1.1.1">percent</csymbol><cn type="float" id="A7.SS5.p1.6.m6.1.1.2.cmml" xref="A7.SS5.p1.6.m6.1.1.2">66.1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A7.SS5.p1.6.m6.1c">66.1\%</annotation></semantics></math>—ALCE is effective in detecting “irrelevant” citations, but
due to the limitation of the NLI model (cannot detect “partial support”), it has a relatively high false positive rate.</p>
</div>
</section>
<section id="A7.SS6" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">G.6 </span>Main Results</h3>

<div id="A7.SS6.p1" class="ltx_para">
<p id="A7.SS6.p1.1" class="ltx_p">We show full results of our experiments along with the standard deviation
in Tables <a href="#A7.T19" title="Table 19 ‣ G.7 Open-source Models ‣ Appendix G More Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">19</span></a>, <a href="#A7.T20" title="Table 20 ‣ G.7 Open-source Models ‣ Appendix G More Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">20</span></a>, and <a href="#A7.T21" title="Table 21 ‣ G.7 Open-source Models ‣ Appendix G More Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">21</span></a>.
We repeat all experiments with three different random seeds.
However, for ChatGPT <span id="A7.SS6.p1.1.1" class="ltx_text ltx_font_smallcaps">Rerank</span>, we use only one seeded run since each run repeats the generation step four times, and more experiments would incur significant costs. Similarly, due to the cost of running ChatGPT-16K and GPT-4, we only use one seeded run for each model.</p>
</div>
</section>
<section id="A7.SS7" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">G.7 </span>Open-source Models</h3>

<div id="A7.SS7.p1" class="ltx_para">
<p id="A7.SS7.p1.1" class="ltx_p">In addition to the open-source models discussed in the main text, we also show the results of LLaMA-7B, Alpaca-7B, Vicuna-7B, LLaMA-33B, Oasst-33B, and Stable Beluga 2 in the Tables <a href="#A7.T19" title="Table 19 ‣ G.7 Open-source Models ‣ Appendix G More Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">19</span></a>, <a href="#A7.T20" title="Table 20 ‣ G.7 Open-source Models ‣ Appendix G More Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">20</span></a>, and <a href="#A7.T21" title="Table 21 ‣ G.7 Open-source Models ‣ Appendix G More Experiments ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">21</span></a>.
For selected models, we also tested them using approaches beyond <span id="A7.SS7.p1.1.1" class="ltx_text ltx_font_smallcaps">Vanilla</span>.</p>
</div>
<div id="A7.SS7.p2" class="ltx_para">
<p id="A7.SS7.p2.1" class="ltx_p">Although open-source models generally lag behind the state-of-the-art models (i.e. GPT-4) in both correctness and citation quality, the largest instruction-following models (i.e. LLaMA-2-70B-Chat and Stable Beluga 2) can sometimes achieve competitive correctness to the SoTA.</p>
</div>
<div id="A7.SS7.p3" class="ltx_para">
<p id="A7.SS7.p3.1" class="ltx_p">Furthermore, we found that the open-source models follow a similar trend between different approaches as ChatGPT. Specifically, using <span id="A7.SS7.p3.1.1" class="ltx_text ltx_font_smallcaps">Summ</span> and <span id="A7.SS7.p3.1.2" class="ltx_text ltx_font_smallcaps">Snippet</span> improves correctness and <span id="A7.SS7.p3.1.3" class="ltx_text ltx_font_smallcaps">Rerank</span> boosts citation quality.</p>
</div>
<figure id="A7.T19" class="ltx_table">
<table id="A7.T19.246" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A7.T19.246.247.1" class="ltx_tr">
<th id="A7.T19.246.247.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<td id="A7.T19.246.247.1.2" class="ltx_td ltx_align_center ltx_border_tt"><span id="A7.T19.246.247.1.2.1" class="ltx_text ltx_font_bold">Fluency</span></td>
<td id="A7.T19.246.247.1.3" class="ltx_td ltx_align_center ltx_border_tt"><span id="A7.T19.246.247.1.3.1" class="ltx_text ltx_font_bold">Correct.</span></td>
<td id="A7.T19.246.247.1.4" class="ltx_td ltx_align_center ltx_border_tt" colspan="2"><span id="A7.T19.246.247.1.4.1" class="ltx_text ltx_font_bold">Citation</span></td>
<td id="A7.T19.246.247.1.5" class="ltx_td ltx_border_tt"></td>
<td id="A7.T19.246.247.1.6" class="ltx_td ltx_border_tt"></td>
</tr>
<tr id="A7.T19.246.248.2" class="ltx_tr">
<th id="A7.T19.246.248.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="A7.T19.246.248.2.2" class="ltx_td ltx_align_center ltx_border_t">(MAUVE)</td>
<td id="A7.T19.246.248.2.3" class="ltx_td ltx_align_center ltx_border_t">(EM Rec.)</td>
<td id="A7.T19.246.248.2.4" class="ltx_td ltx_align_center ltx_border_t">Rec.</td>
<td id="A7.T19.246.248.2.5" class="ltx_td ltx_align_center ltx_border_t">Prec.</td>
<td id="A7.T19.246.248.2.6" class="ltx_td ltx_align_center">ROUGE-L</td>
<td id="A7.T19.246.248.2.7" class="ltx_td ltx_align_center">Length</td>
</tr>
<tr id="A7.T19.246.249.3" class="ltx_tr" style="background-color:#E9E9E9;">
<th id="A7.T19.246.249.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" colspan="7"><span id="A7.T19.246.249.3.1.1" class="ltx_text ltx_font_bold" style="background-color:#E9E9E9;">ChatGPT</span></th>
</tr>
<tr id="A7.T19.6.6" class="ltx_tr">
<th id="A7.T19.6.6.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T19.6.6.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (5-psg)</th>
<td id="A7.T19.1.1.1" class="ltx_td ltx_align_center">66.8 <sub id="A7.T19.1.1.1.1" class="ltx_sub"><span id="A7.T19.1.1.1.1.1" class="ltx_text ltx_font_italic">(2.0)</span></sub>
</td>
<td id="A7.T19.2.2.2" class="ltx_td ltx_align_center">40.4 <sub id="A7.T19.2.2.2.1" class="ltx_sub"><span id="A7.T19.2.2.2.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
<td id="A7.T19.3.3.3" class="ltx_td ltx_align_center">73.6 <sub id="A7.T19.3.3.3.1" class="ltx_sub"><span id="A7.T19.3.3.3.1.1" class="ltx_text ltx_font_italic">(1.1)</span></sub>
</td>
<td id="A7.T19.4.4.4" class="ltx_td ltx_align_center">72.5 <sub id="A7.T19.4.4.4.1" class="ltx_sub"><span id="A7.T19.4.4.4.1.1" class="ltx_text ltx_font_italic">(1.8)</span></sub>
</td>
<td id="A7.T19.5.5.5" class="ltx_td ltx_align_center">37.0 <sub id="A7.T19.5.5.5.1" class="ltx_sub"><span id="A7.T19.5.5.5.1.1" class="ltx_text ltx_font_italic">(0.4)</span></sub>
</td>
<td id="A7.T19.6.6.6" class="ltx_td ltx_align_center">40.0 <sub id="A7.T19.6.6.6.1" class="ltx_sub"><span id="A7.T19.6.6.6.1.1" class="ltx_text ltx_font_italic">(3.1)</span></sub>
</td>
</tr>
<tr id="A7.T19.12.12" class="ltx_tr">
<th id="A7.T19.12.12.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">w/ <span id="A7.T19.12.12.7.1" class="ltx_text ltx_font_smallcaps">Rerank</span>
</th>
<td id="A7.T19.7.7.1" class="ltx_td ltx_align_center">77.0 <sub id="A7.T19.7.7.1.1" class="ltx_sub"><span id="A7.T19.7.7.1.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.8.8.2" class="ltx_td ltx_align_center">40.2 <sub id="A7.T19.8.8.2.1" class="ltx_sub"><span id="A7.T19.8.8.2.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.9.9.3" class="ltx_td ltx_align_center">84.8 <sub id="A7.T19.9.9.3.1" class="ltx_sub"><span id="A7.T19.9.9.3.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.10.10.4" class="ltx_td ltx_align_center">81.6 <sub id="A7.T19.10.10.4.1" class="ltx_sub"><span id="A7.T19.10.10.4.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.11.11.5" class="ltx_td ltx_align_center">36.9 <sub id="A7.T19.11.11.5.1" class="ltx_sub"><span id="A7.T19.11.11.5.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.12.12.6" class="ltx_td ltx_align_center">40.8 <sub id="A7.T19.12.12.6.1" class="ltx_sub"><span id="A7.T19.12.12.6.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
</tr>
<tr id="A7.T19.18.18" class="ltx_tr">
<th id="A7.T19.18.18.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T19.18.18.7.1" class="ltx_text ltx_font_smallcaps">Summ</span> (10-psg)</th>
<td id="A7.T19.13.13.1" class="ltx_td ltx_align_center">70.0 <sub id="A7.T19.13.13.1.1" class="ltx_sub"><span id="A7.T19.13.13.1.1.1" class="ltx_text ltx_font_italic">(1.2)</span></sub>
</td>
<td id="A7.T19.14.14.2" class="ltx_td ltx_align_center">43.3 <sub id="A7.T19.14.14.2.1" class="ltx_sub"><span id="A7.T19.14.14.2.1.1" class="ltx_text ltx_font_italic">(0.8)</span></sub>
</td>
<td id="A7.T19.15.15.3" class="ltx_td ltx_align_center">68.8 <sub id="A7.T19.15.15.3.1" class="ltx_sub"><span id="A7.T19.15.15.3.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
<td id="A7.T19.16.16.4" class="ltx_td ltx_align_center">61.8 <sub id="A7.T19.16.16.4.1" class="ltx_sub"><span id="A7.T19.16.16.4.1.1" class="ltx_text ltx_font_italic">(1.1)</span></sub>
</td>
<td id="A7.T19.17.17.5" class="ltx_td ltx_align_center">36.9 <sub id="A7.T19.17.17.5.1" class="ltx_sub"><span id="A7.T19.17.17.5.1.1" class="ltx_text ltx_font_italic">(0.2)</span></sub>
</td>
<td id="A7.T19.18.18.6" class="ltx_td ltx_align_center">49.8 <sub id="A7.T19.18.18.6.1" class="ltx_sub"><span id="A7.T19.18.18.6.1.1" class="ltx_text ltx_font_italic">(4.3)</span></sub>
</td>
</tr>
<tr id="A7.T19.24.24" class="ltx_tr">
<th id="A7.T19.24.24.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">w/ <span id="A7.T19.24.24.7.1" class="ltx_text ltx_font_smallcaps">Interact</span>
</th>
<td id="A7.T19.19.19.1" class="ltx_td ltx_align_center">69.0 <sub id="A7.T19.19.19.1.1" class="ltx_sub"><span id="A7.T19.19.19.1.1.1" class="ltx_text ltx_font_italic">(2.7)</span></sub>
</td>
<td id="A7.T19.20.20.2" class="ltx_td ltx_align_center">39.1 <sub id="A7.T19.20.20.2.1" class="ltx_sub"><span id="A7.T19.20.20.2.1.1" class="ltx_text ltx_font_italic">(0.5)</span></sub>
</td>
<td id="A7.T19.21.21.3" class="ltx_td ltx_align_center">73.4 <sub id="A7.T19.21.21.3.1" class="ltx_sub"><span id="A7.T19.21.21.3.1.1" class="ltx_text ltx_font_italic">(0.2)</span></sub>
</td>
<td id="A7.T19.22.22.4" class="ltx_td ltx_align_center">66.5 <sub id="A7.T19.22.22.4.1" class="ltx_sub"><span id="A7.T19.22.22.4.1.1" class="ltx_text ltx_font_italic">(4.9)</span></sub>
</td>
<td id="A7.T19.23.23.5" class="ltx_td ltx_align_center">35.7 <sub id="A7.T19.23.23.5.1" class="ltx_sub"><span id="A7.T19.23.23.5.1.1" class="ltx_text ltx_font_italic">(0.2)</span></sub>
</td>
<td id="A7.T19.24.24.6" class="ltx_td ltx_align_center">34.0 <sub id="A7.T19.24.24.6.1" class="ltx_sub"><span id="A7.T19.24.24.6.1.1" class="ltx_text ltx_font_italic">(0.9)</span></sub>
</td>
</tr>
<tr id="A7.T19.30.30" class="ltx_tr">
<th id="A7.T19.30.30.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T19.30.30.7.1" class="ltx_text ltx_font_smallcaps">Snippet</span> (10-psg)</th>
<td id="A7.T19.25.25.1" class="ltx_td ltx_align_center">69.8 <sub id="A7.T19.25.25.1.1" class="ltx_sub"><span id="A7.T19.25.25.1.1.1" class="ltx_text ltx_font_italic">(2.5)</span></sub>
</td>
<td id="A7.T19.26.26.2" class="ltx_td ltx_align_center">41.4 <sub id="A7.T19.26.26.2.1" class="ltx_sub"><span id="A7.T19.26.26.2.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
<td id="A7.T19.27.27.3" class="ltx_td ltx_align_center">65.3 <sub id="A7.T19.27.27.3.1" class="ltx_sub"><span id="A7.T19.27.27.3.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
<td id="A7.T19.28.28.4" class="ltx_td ltx_align_center">57.4 <sub id="A7.T19.28.28.4.1" class="ltx_sub"><span id="A7.T19.28.28.4.1.1" class="ltx_text ltx_font_italic">(0.9)</span></sub>
</td>
<td id="A7.T19.29.29.5" class="ltx_td ltx_align_center">36.4 <sub id="A7.T19.29.29.5.1" class="ltx_sub"><span id="A7.T19.29.29.5.1.1" class="ltx_text ltx_font_italic">(0.4)</span></sub>
</td>
<td id="A7.T19.30.30.6" class="ltx_td ltx_align_center">43.0 <sub id="A7.T19.30.30.6.1" class="ltx_sub"><span id="A7.T19.30.30.6.1.1" class="ltx_text ltx_font_italic">(3.5)</span></sub>
</td>
</tr>
<tr id="A7.T19.36.36" class="ltx_tr">
<th id="A7.T19.36.36.7" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A7.T19.36.36.7.1" class="ltx_text ltx_font_smallcaps">InlineSearch</span></th>
<td id="A7.T19.31.31.1" class="ltx_td ltx_align_center">58.7 <sub id="A7.T19.31.31.1.1" class="ltx_sub"><span id="A7.T19.31.31.1.1.1" class="ltx_text ltx_font_italic">(1.3)</span></sub>
</td>
<td id="A7.T19.32.32.2" class="ltx_td ltx_align_center">32.4 <sub id="A7.T19.32.32.2.1" class="ltx_sub"><span id="A7.T19.32.32.2.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
<td id="A7.T19.33.33.3" class="ltx_td ltx_align_center">58.3 <sub id="A7.T19.33.33.3.1" class="ltx_sub"><span id="A7.T19.33.33.3.1.1" class="ltx_text ltx_font_italic">(1.3)</span></sub>
</td>
<td id="A7.T19.34.34.4" class="ltx_td ltx_align_center">58.3 <sub id="A7.T19.34.34.4.1" class="ltx_sub"><span id="A7.T19.34.34.4.1.1" class="ltx_text ltx_font_italic">(1.3)</span></sub>
</td>
<td id="A7.T19.35.35.5" class="ltx_td ltx_align_center">58.2 <sub id="A7.T19.35.35.5.1" class="ltx_sub"><span id="A7.T19.35.35.5.1.1" class="ltx_text ltx_font_italic">(1.1)</span></sub>
</td>
<td id="A7.T19.36.36.6" class="ltx_td ltx_align_center">23.7 <sub id="A7.T19.36.36.6.1" class="ltx_sub"><span id="A7.T19.36.36.6.1.1" class="ltx_text ltx_font_italic">(1.1)</span></sub>
</td>
</tr>
<tr id="A7.T19.42.42" class="ltx_tr">
<th id="A7.T19.42.42.7" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A7.T19.42.42.7.1" class="ltx_text ltx_font_smallcaps">ClosedBook</span></th>
<td id="A7.T19.37.37.1" class="ltx_td ltx_align_center">52.7 <sub id="A7.T19.37.37.1.1" class="ltx_sub"><span id="A7.T19.37.37.1.1.1" class="ltx_text ltx_font_italic">(4.9)</span></sub>
</td>
<td id="A7.T19.38.38.2" class="ltx_td ltx_align_center">38.2 <sub id="A7.T19.38.38.2.1" class="ltx_sub"><span id="A7.T19.38.38.2.1.1" class="ltx_text ltx_font_italic">(0.1)</span></sub>
</td>
<td id="A7.T19.39.39.3" class="ltx_td ltx_align_center">26.7 <sub id="A7.T19.39.39.3.1" class="ltx_sub"><span id="A7.T19.39.39.3.1.1" class="ltx_text ltx_font_italic">(1.1)</span></sub>
</td>
<td id="A7.T19.40.40.4" class="ltx_td ltx_align_center">26.7 <sub id="A7.T19.40.40.4.1" class="ltx_sub"><span id="A7.T19.40.40.4.1.1" class="ltx_text ltx_font_italic">(1.1)</span></sub>
</td>
<td id="A7.T19.41.41.5" class="ltx_td ltx_align_center">37.1 <sub id="A7.T19.41.41.5.1" class="ltx_sub"><span id="A7.T19.41.41.5.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
<td id="A7.T19.42.42.6" class="ltx_td ltx_align_center">61.1 <sub id="A7.T19.42.42.6.1" class="ltx_sub"><span id="A7.T19.42.42.6.1.1" class="ltx_text ltx_font_italic">(4.5)</span></sub>
</td>
</tr>
<tr id="A7.T19.48.48" class="ltx_tr">
<th id="A7.T19.48.48.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T19.48.48.7.1" class="ltx_text ltx_font_smallcaps">Oracle</span>(5-psg)</th>
<td id="A7.T19.43.43.1" class="ltx_td ltx_align_center">64.4 <sub id="A7.T19.43.43.1.1" class="ltx_sub"><span id="A7.T19.43.43.1.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
<td id="A7.T19.44.44.2" class="ltx_td ltx_align_center">48.9 <sub id="A7.T19.44.44.2.1" class="ltx_sub"><span id="A7.T19.44.44.2.1.1" class="ltx_text ltx_font_italic">(1.2)</span></sub>
</td>
<td id="A7.T19.45.45.3" class="ltx_td ltx_align_center">74.5 <sub id="A7.T19.45.45.3.1" class="ltx_sub"><span id="A7.T19.45.45.3.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
<td id="A7.T19.46.46.4" class="ltx_td ltx_align_center">72.7 <sub id="A7.T19.46.46.4.1" class="ltx_sub"><span id="A7.T19.46.46.4.1.1" class="ltx_text ltx_font_italic">(1.0)</span></sub>
</td>
<td id="A7.T19.47.47.5" class="ltx_td ltx_align_center">38.2 <sub id="A7.T19.47.47.5.1" class="ltx_sub"><span id="A7.T19.47.47.5.1.1" class="ltx_text ltx_font_italic">(1.0)</span></sub>
</td>
<td id="A7.T19.48.48.6" class="ltx_td ltx_align_center">37.4 <sub id="A7.T19.48.48.6.1" class="ltx_sub"><span id="A7.T19.48.48.6.1.1" class="ltx_text ltx_font_italic">(3.0)</span></sub>
</td>
</tr>
<tr id="A7.T19.246.250.4" class="ltx_tr">
<th id="A7.T19.246.250.4.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" colspan="7"><span id="A7.T19.246.250.4.1.1" class="ltx_text ltx_font_bold" style="background-color:#E9E9E9;">ChatGPT-16K</span></th>
</tr>
<tr id="A7.T19.54.54" class="ltx_tr">
<th id="A7.T19.54.54.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T19.54.54.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (5-psg)</th>
<td id="A7.T19.49.49.1" class="ltx_td ltx_align_center">60.3 <sub id="A7.T19.49.49.1.1" class="ltx_sub"><span id="A7.T19.49.49.1.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.50.50.2" class="ltx_td ltx_align_center">36.1 <sub id="A7.T19.50.50.2.1" class="ltx_sub"><span id="A7.T19.50.50.2.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.51.51.3" class="ltx_td ltx_align_center">76.2 <sub id="A7.T19.51.51.3.1" class="ltx_sub"><span id="A7.T19.51.51.3.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.52.52.4" class="ltx_td ltx_align_center">76.5 <sub id="A7.T19.52.52.4.1" class="ltx_sub"><span id="A7.T19.52.52.4.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.53.53.5" class="ltx_td ltx_align_center">36.2 <sub id="A7.T19.53.53.5.1" class="ltx_sub"><span id="A7.T19.53.53.5.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.54.54.6" class="ltx_td ltx_align_center">24.7 <sub id="A7.T19.54.54.6.1" class="ltx_sub"><span id="A7.T19.54.54.6.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
</tr>
<tr id="A7.T19.60.60" class="ltx_tr">
<th id="A7.T19.60.60.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T19.60.60.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (10-psg)</th>
<td id="A7.T19.55.55.1" class="ltx_td ltx_align_center">56.3 <sub id="A7.T19.55.55.1.1" class="ltx_sub"><span id="A7.T19.55.55.1.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.56.56.2" class="ltx_td ltx_align_center">36.7 <sub id="A7.T19.56.56.2.1" class="ltx_sub"><span id="A7.T19.56.56.2.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.57.57.3" class="ltx_td ltx_align_center">75.3 <sub id="A7.T19.57.57.3.1" class="ltx_sub"><span id="A7.T19.57.57.3.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.58.58.4" class="ltx_td ltx_align_center">75.0 <sub id="A7.T19.58.58.4.1" class="ltx_sub"><span id="A7.T19.58.58.4.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.59.59.5" class="ltx_td ltx_align_center">35.6 <sub id="A7.T19.59.59.5.1" class="ltx_sub"><span id="A7.T19.59.59.5.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.60.60.6" class="ltx_td ltx_align_center">23.5 <sub id="A7.T19.60.60.6.1" class="ltx_sub"><span id="A7.T19.60.60.6.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
</tr>
<tr id="A7.T19.66.66" class="ltx_tr">
<th id="A7.T19.66.66.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T19.66.66.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (20-psg)</th>
<td id="A7.T19.61.61.1" class="ltx_td ltx_align_center">56.7 <sub id="A7.T19.61.61.1.1" class="ltx_sub"><span id="A7.T19.61.61.1.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.62.62.2" class="ltx_td ltx_align_center">36.1 <sub id="A7.T19.62.62.2.1" class="ltx_sub"><span id="A7.T19.62.62.2.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.63.63.3" class="ltx_td ltx_align_center">73.7 <sub id="A7.T19.63.63.3.1" class="ltx_sub"><span id="A7.T19.63.63.3.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.64.64.4" class="ltx_td ltx_align_center">73.5 <sub id="A7.T19.64.64.4.1" class="ltx_sub"><span id="A7.T19.64.64.4.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.65.65.5" class="ltx_td ltx_align_center">35.5 <sub id="A7.T19.65.65.5.1" class="ltx_sub"><span id="A7.T19.65.65.5.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.66.66.6" class="ltx_td ltx_align_center">23.1 <sub id="A7.T19.66.66.6.1" class="ltx_sub"><span id="A7.T19.66.66.6.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
</tr>
<tr id="A7.T19.246.251.5" class="ltx_tr">
<th id="A7.T19.246.251.5.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" colspan="7"><span id="A7.T19.246.251.5.1.1" class="ltx_text ltx_font_bold" style="background-color:#E9E9E9;">GPT-4</span></th>
</tr>
<tr id="A7.T19.72.72" class="ltx_tr">
<th id="A7.T19.72.72.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T19.72.72.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (5-psg)</th>
<td id="A7.T19.67.67.1" class="ltx_td ltx_align_center">67.1 <sub id="A7.T19.67.67.1.1" class="ltx_sub"><span id="A7.T19.67.67.1.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.68.68.2" class="ltx_td ltx_align_center">41.3 <sub id="A7.T19.68.68.2.1" class="ltx_sub"><span id="A7.T19.68.68.2.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.69.69.3" class="ltx_td ltx_align_center">68.5 <sub id="A7.T19.69.69.3.1" class="ltx_sub"><span id="A7.T19.69.69.3.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.70.70.4" class="ltx_td ltx_align_center">75.6 <sub id="A7.T19.70.70.4.1" class="ltx_sub"><span id="A7.T19.70.70.4.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.71.71.5" class="ltx_td ltx_align_center">39.2 <sub id="A7.T19.71.71.5.1" class="ltx_sub"><span id="A7.T19.71.71.5.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.72.72.6" class="ltx_td ltx_align_center">31.8 <sub id="A7.T19.72.72.6.1" class="ltx_sub"><span id="A7.T19.72.72.6.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
</tr>
<tr id="A7.T19.78.78" class="ltx_tr">
<th id="A7.T19.78.78.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T19.78.78.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (10-psg)</th>
<td id="A7.T19.73.73.1" class="ltx_td ltx_align_center">71.5 <sub id="A7.T19.73.73.1.1" class="ltx_sub"><span id="A7.T19.73.73.1.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.74.74.2" class="ltx_td ltx_align_center">43.1 <sub id="A7.T19.74.74.2.1" class="ltx_sub"><span id="A7.T19.74.74.2.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.75.75.3" class="ltx_td ltx_align_center">72.0 <sub id="A7.T19.75.75.3.1" class="ltx_sub"><span id="A7.T19.75.75.3.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.76.76.4" class="ltx_td ltx_align_center">75.5 <sub id="A7.T19.76.76.4.1" class="ltx_sub"><span id="A7.T19.76.76.4.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.77.77.5" class="ltx_td ltx_align_center">39.7 <sub id="A7.T19.77.77.5.1" class="ltx_sub"><span id="A7.T19.77.77.5.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.78.78.6" class="ltx_td ltx_align_center">33.8 <sub id="A7.T19.78.78.6.1" class="ltx_sub"><span id="A7.T19.78.78.6.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
</tr>
<tr id="A7.T19.84.84" class="ltx_tr">
<th id="A7.T19.84.84.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T19.84.84.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (20-psg)</th>
<td id="A7.T19.79.79.1" class="ltx_td ltx_align_center">64.9 <sub id="A7.T19.79.79.1.1" class="ltx_sub"><span id="A7.T19.79.79.1.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.80.80.2" class="ltx_td ltx_align_center">44.4 <sub id="A7.T19.80.80.2.1" class="ltx_sub"><span id="A7.T19.80.80.2.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.81.81.3" class="ltx_td ltx_align_center">73.0 <sub id="A7.T19.81.81.3.1" class="ltx_sub"><span id="A7.T19.81.81.3.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.82.82.4" class="ltx_td ltx_align_center">76.5 <sub id="A7.T19.82.82.4.1" class="ltx_sub"><span id="A7.T19.82.82.4.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.83.83.5" class="ltx_td ltx_align_center">40.1 <sub id="A7.T19.83.83.5.1" class="ltx_sub"><span id="A7.T19.83.83.5.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T19.84.84.6" class="ltx_td ltx_align_center">34.3 <sub id="A7.T19.84.84.6.1" class="ltx_sub"><span id="A7.T19.84.84.6.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
</tr>
<tr id="A7.T19.246.252.6" class="ltx_tr">
<th id="A7.T19.246.252.6.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" colspan="7"><span id="A7.T19.246.252.6.1.1" class="ltx_text ltx_font_bold" style="background-color:#E9E9E9;">Open-source</span></th>
</tr>
<tr id="A7.T19.90.90" class="ltx_tr">
<th id="A7.T19.90.90.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-7B <span id="A7.T19.90.90.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (3-psg)</th>
<td id="A7.T19.85.85.1" class="ltx_td ltx_align_center">69.8 <sub id="A7.T19.85.85.1.1" class="ltx_sub"><span id="A7.T19.85.85.1.1.1" class="ltx_text ltx_font_italic">(2.0)</span></sub>
</td>
<td id="A7.T19.86.86.2" class="ltx_td ltx_align_center">22.6 <sub id="A7.T19.86.86.2.1" class="ltx_sub"><span id="A7.T19.86.86.2.1.1" class="ltx_text ltx_font_italic">(0.9)</span></sub>
</td>
<td id="A7.T19.87.87.3" class="ltx_td ltx_align_center">6.2 <sub id="A7.T19.87.87.3.1" class="ltx_sub"><span id="A7.T19.87.87.3.1.1" class="ltx_text ltx_font_italic">(2.7)</span></sub>
</td>
<td id="A7.T19.88.88.4" class="ltx_td ltx_align_center">9.2 <sub id="A7.T19.88.88.4.1" class="ltx_sub"><span id="A7.T19.88.88.4.1.1" class="ltx_text ltx_font_italic">(2.9)</span></sub>
</td>
<td id="A7.T19.89.89.5" class="ltx_td ltx_align_center">29.1 <sub id="A7.T19.89.89.5.1" class="ltx_sub"><span id="A7.T19.89.89.5.1.1" class="ltx_text ltx_font_italic">(0.2)</span></sub>
</td>
<td id="A7.T19.90.90.6" class="ltx_td ltx_align_center">61.3 <sub id="A7.T19.90.90.6.1" class="ltx_sub"><span id="A7.T19.90.90.6.1.1" class="ltx_text ltx_font_italic">(14.3)</span></sub>
</td>
</tr>
<tr id="A7.T19.96.96" class="ltx_tr">
<th id="A7.T19.96.96.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">Alpaca-7B <span id="A7.T19.96.96.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (3-psg)</th>
<td id="A7.T19.91.91.1" class="ltx_td ltx_align_center">84.2 <sub id="A7.T19.91.91.1.1" class="ltx_sub"><span id="A7.T19.91.91.1.1.1" class="ltx_text ltx_font_italic">(2.7)</span></sub>
</td>
<td id="A7.T19.92.92.2" class="ltx_td ltx_align_center">32.1 <sub id="A7.T19.92.92.2.1" class="ltx_sub"><span id="A7.T19.92.92.2.1.1" class="ltx_text ltx_font_italic">(1.7)</span></sub>
</td>
<td id="A7.T19.93.93.3" class="ltx_td ltx_align_center">12.3 <sub id="A7.T19.93.93.3.1" class="ltx_sub"><span id="A7.T19.93.93.3.1.1" class="ltx_text ltx_font_italic">(7.2)</span></sub>
</td>
<td id="A7.T19.94.94.4" class="ltx_td ltx_align_center">14.1 <sub id="A7.T19.94.94.4.1" class="ltx_sub"><span id="A7.T19.94.94.4.1.1" class="ltx_text ltx_font_italic">(7.0)</span></sub>
</td>
<td id="A7.T19.95.95.5" class="ltx_td ltx_align_center">33.1 <sub id="A7.T19.95.95.5.1" class="ltx_sub"><span id="A7.T19.95.95.5.1.1" class="ltx_text ltx_font_italic">(0.8)</span></sub>
</td>
<td id="A7.T19.96.96.6" class="ltx_td ltx_align_center">51.7 <sub id="A7.T19.96.96.6.1" class="ltx_sub"><span id="A7.T19.96.96.6.1.1" class="ltx_text ltx_font_italic">(12.8)</span></sub>
</td>
</tr>
<tr id="A7.T19.102.102" class="ltx_tr">
<th id="A7.T19.102.102.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">Vicuna-7B <span id="A7.T19.102.102.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (3-psg)</th>
<td id="A7.T19.97.97.1" class="ltx_td ltx_align_center">82.9 <sub id="A7.T19.97.97.1.1" class="ltx_sub"><span id="A7.T19.97.97.1.1.1" class="ltx_text ltx_font_italic">(5.0)</span></sub>
</td>
<td id="A7.T19.98.98.2" class="ltx_td ltx_align_center">34.6 <sub id="A7.T19.98.98.2.1" class="ltx_sub"><span id="A7.T19.98.98.2.1.1" class="ltx_text ltx_font_italic">(0.7)</span></sub>
</td>
<td id="A7.T19.99.99.3" class="ltx_td ltx_align_center">40.3 <sub id="A7.T19.99.99.3.1" class="ltx_sub"><span id="A7.T19.99.99.3.1.1" class="ltx_text ltx_font_italic">(0.5)</span></sub>
</td>
<td id="A7.T19.100.100.4" class="ltx_td ltx_align_center">42.6 <sub id="A7.T19.100.100.4.1" class="ltx_sub"><span id="A7.T19.100.100.4.1.1" class="ltx_text ltx_font_italic">(1.0)</span></sub>
</td>
<td id="A7.T19.101.101.5" class="ltx_td ltx_align_center">35.9 <sub id="A7.T19.101.101.5.1" class="ltx_sub"><span id="A7.T19.101.101.5.1.1" class="ltx_text ltx_font_italic">(0.7)</span></sub>
</td>
<td id="A7.T19.102.102.6" class="ltx_td ltx_align_center">48.9 <sub id="A7.T19.102.102.6.1" class="ltx_sub"><span id="A7.T19.102.102.6.1.1" class="ltx_text ltx_font_italic">(6.6)</span></sub>
</td>
</tr>
<tr id="A7.T19.108.108" class="ltx_tr">
<th id="A7.T19.108.108.7" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">LLaMA-13B <span id="A7.T19.108.108.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (3-psg)</th>
<td id="A7.T19.103.103.1" class="ltx_td ltx_align_center ltx_border_t">68.4 <sub id="A7.T19.103.103.1.1" class="ltx_sub"><span id="A7.T19.103.103.1.1.1" class="ltx_text ltx_font_italic">(6.4)</span></sub>
</td>
<td id="A7.T19.104.104.2" class="ltx_td ltx_align_center ltx_border_t">26.9 <sub id="A7.T19.104.104.2.1" class="ltx_sub"><span id="A7.T19.104.104.2.1.1" class="ltx_text ltx_font_italic">(0.4)</span></sub>
</td>
<td id="A7.T19.105.105.3" class="ltx_td ltx_align_center ltx_border_t">10.6 <sub id="A7.T19.105.105.3.1" class="ltx_sub"><span id="A7.T19.105.105.3.1.1" class="ltx_text ltx_font_italic">(4.7)</span></sub>
</td>
<td id="A7.T19.106.106.4" class="ltx_td ltx_align_center ltx_border_t">15.4 <sub id="A7.T19.106.106.4.1" class="ltx_sub"><span id="A7.T19.106.106.4.1.1" class="ltx_text ltx_font_italic">(5.2)</span></sub>
</td>
<td id="A7.T19.107.107.5" class="ltx_td ltx_align_center ltx_border_t">29.8 <sub id="A7.T19.107.107.5.1" class="ltx_sub"><span id="A7.T19.107.107.5.1.1" class="ltx_text ltx_font_italic">(0.5)</span></sub>
</td>
<td id="A7.T19.108.108.6" class="ltx_td ltx_align_center ltx_border_t">67.1 <sub id="A7.T19.108.108.6.1" class="ltx_sub"><span id="A7.T19.108.108.6.1.1" class="ltx_text ltx_font_italic">(19.1)</span></sub>
</td>
</tr>
<tr id="A7.T19.114.114" class="ltx_tr">
<th id="A7.T19.114.114.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">w/ <span id="A7.T19.114.114.7.1" class="ltx_text ltx_font_smallcaps">Rerank</span>
</th>
<td id="A7.T19.109.109.1" class="ltx_td ltx_align_center">60.9 <sub id="A7.T19.109.109.1.1" class="ltx_sub"><span id="A7.T19.109.109.1.1.1" class="ltx_text ltx_font_italic">(14.5)</span></sub>
</td>
<td id="A7.T19.110.110.2" class="ltx_td ltx_align_center">25.2 <sub id="A7.T19.110.110.2.1" class="ltx_sub"><span id="A7.T19.110.110.2.1.1" class="ltx_text ltx_font_italic">(2.5)</span></sub>
</td>
<td id="A7.T19.111.111.3" class="ltx_td ltx_align_center">28.1 <sub id="A7.T19.111.111.3.1" class="ltx_sub"><span id="A7.T19.111.111.3.1.1" class="ltx_text ltx_font_italic">(9.3)</span></sub>
</td>
<td id="A7.T19.112.112.4" class="ltx_td ltx_align_center">37.0 <sub id="A7.T19.112.112.4.1" class="ltx_sub"><span id="A7.T19.112.112.4.1.1" class="ltx_text ltx_font_italic">(7.2)</span></sub>
</td>
<td id="A7.T19.113.113.5" class="ltx_td ltx_align_center">27.9 <sub id="A7.T19.113.113.5.1" class="ltx_sub"><span id="A7.T19.113.113.5.1.1" class="ltx_text ltx_font_italic">(2.4)</span></sub>
</td>
<td id="A7.T19.114.114.6" class="ltx_td ltx_align_center">50.5 <sub id="A7.T19.114.114.6.1" class="ltx_sub"><span id="A7.T19.114.114.6.1.1" class="ltx_text ltx_font_italic">(14.3)</span></sub>
</td>
</tr>
<tr id="A7.T19.120.120" class="ltx_tr">
<th id="A7.T19.120.120.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-13B <span id="A7.T19.120.120.7.1" class="ltx_text ltx_font_smallcaps">Summ</span> (10-psg)</th>
<td id="A7.T19.115.115.1" class="ltx_td ltx_align_center">76.8 <sub id="A7.T19.115.115.1.1" class="ltx_sub"><span id="A7.T19.115.115.1.1.1" class="ltx_text ltx_font_italic">(4.7)</span></sub>
</td>
<td id="A7.T19.116.116.2" class="ltx_td ltx_align_center">33.3 <sub id="A7.T19.116.116.2.1" class="ltx_sub"><span id="A7.T19.116.116.2.1.1" class="ltx_text ltx_font_italic">(0.7)</span></sub>
</td>
<td id="A7.T19.117.117.3" class="ltx_td ltx_align_center">19.6 <sub id="A7.T19.117.117.3.1" class="ltx_sub"><span id="A7.T19.117.117.3.1.1" class="ltx_text ltx_font_italic">(3.9)</span></sub>
</td>
<td id="A7.T19.118.118.4" class="ltx_td ltx_align_center">23.7 <sub id="A7.T19.118.118.4.1" class="ltx_sub"><span id="A7.T19.118.118.4.1.1" class="ltx_text ltx_font_italic">(4.7)</span></sub>
</td>
<td id="A7.T19.119.119.5" class="ltx_td ltx_align_center">32.1 <sub id="A7.T19.119.119.5.1" class="ltx_sub"><span id="A7.T19.119.119.5.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
<td id="A7.T19.120.120.6" class="ltx_td ltx_align_center">54.4 <sub id="A7.T19.120.120.6.1" class="ltx_sub"><span id="A7.T19.120.120.6.1.1" class="ltx_text ltx_font_italic">(1.5)</span></sub>
</td>
</tr>
<tr id="A7.T19.126.126" class="ltx_tr">
<th id="A7.T19.126.126.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-13B <span id="A7.T19.126.126.7.1" class="ltx_text ltx_font_smallcaps">Snippet</span> (10-psg)</th>
<td id="A7.T19.121.121.1" class="ltx_td ltx_align_center">72.0 <sub id="A7.T19.121.121.1.1" class="ltx_sub"><span id="A7.T19.121.121.1.1.1" class="ltx_text ltx_font_italic">(0.8)</span></sub>
</td>
<td id="A7.T19.122.122.2" class="ltx_td ltx_align_center">31.3 <sub id="A7.T19.122.122.2.1" class="ltx_sub"><span id="A7.T19.122.122.2.1.1" class="ltx_text ltx_font_italic">(1.1)</span></sub>
</td>
<td id="A7.T19.123.123.3" class="ltx_td ltx_align_center">18.2 <sub id="A7.T19.123.123.3.1" class="ltx_sub"><span id="A7.T19.123.123.3.1.1" class="ltx_text ltx_font_italic">(3.1)</span></sub>
</td>
<td id="A7.T19.124.124.4" class="ltx_td ltx_align_center">21.1 <sub id="A7.T19.124.124.4.1" class="ltx_sub"><span id="A7.T19.124.124.4.1.1" class="ltx_text ltx_font_italic">(3.6)</span></sub>
</td>
<td id="A7.T19.125.125.5" class="ltx_td ltx_align_center">30.8 <sub id="A7.T19.125.125.5.1" class="ltx_sub"><span id="A7.T19.125.125.5.1.1" class="ltx_text ltx_font_italic">(0.4)</span></sub>
</td>
<td id="A7.T19.126.126.6" class="ltx_td ltx_align_center">50.5 <sub id="A7.T19.126.126.6.1" class="ltx_sub"><span id="A7.T19.126.126.6.1.1" class="ltx_text ltx_font_italic">(4.5)</span></sub>
</td>
</tr>
<tr id="A7.T19.132.132" class="ltx_tr">
<th id="A7.T19.132.132.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-13B <span id="A7.T19.132.132.7.1" class="ltx_text ltx_font_smallcaps">Oracle</span> (3-psg)</th>
<td id="A7.T19.127.127.1" class="ltx_td ltx_align_center">69.5 <sub id="A7.T19.127.127.1.1" class="ltx_sub"><span id="A7.T19.127.127.1.1.1" class="ltx_text ltx_font_italic">(11.4)</span></sub>
</td>
<td id="A7.T19.128.128.2" class="ltx_td ltx_align_center">34.3 <sub id="A7.T19.128.128.2.1" class="ltx_sub"><span id="A7.T19.128.128.2.1.1" class="ltx_text ltx_font_italic">(0.9)</span></sub>
</td>
<td id="A7.T19.129.129.3" class="ltx_td ltx_align_center">10.8 <sub id="A7.T19.129.129.3.1" class="ltx_sub"><span id="A7.T19.129.129.3.1.1" class="ltx_text ltx_font_italic">(4.9)</span></sub>
</td>
<td id="A7.T19.130.130.4" class="ltx_td ltx_align_center">15.8 <sub id="A7.T19.130.130.4.1" class="ltx_sub"><span id="A7.T19.130.130.4.1.1" class="ltx_text ltx_font_italic">(5.9)</span></sub>
</td>
<td id="A7.T19.131.131.5" class="ltx_td ltx_align_center">30.6 <sub id="A7.T19.131.131.5.1" class="ltx_sub"><span id="A7.T19.131.131.5.1.1" class="ltx_text ltx_font_italic">(0.1)</span></sub>
</td>
<td id="A7.T19.132.132.6" class="ltx_td ltx_align_center">67.3 <sub id="A7.T19.132.132.6.1" class="ltx_sub"><span id="A7.T19.132.132.6.1.1" class="ltx_text ltx_font_italic">(17.9)</span></sub>
</td>
</tr>
<tr id="A7.T19.138.138" class="ltx_tr">
<th id="A7.T19.138.138.7" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Vicuna-13B <span id="A7.T19.138.138.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (3-psg)</th>
<td id="A7.T19.133.133.1" class="ltx_td ltx_align_center ltx_border_t">82.6 <sub id="A7.T19.133.133.1.1" class="ltx_sub"><span id="A7.T19.133.133.1.1.1" class="ltx_text ltx_font_italic">(9.4)</span></sub>
</td>
<td id="A7.T19.134.134.2" class="ltx_td ltx_align_center ltx_border_t">31.9 <sub id="A7.T19.134.134.2.1" class="ltx_sub"><span id="A7.T19.134.134.2.1.1" class="ltx_text ltx_font_italic">(3.9)</span></sub>
</td>
<td id="A7.T19.135.135.3" class="ltx_td ltx_align_center ltx_border_t">51.1 <sub id="A7.T19.135.135.3.1" class="ltx_sub"><span id="A7.T19.135.135.3.1.1" class="ltx_text ltx_font_italic">(1.4)</span></sub>
</td>
<td id="A7.T19.136.136.4" class="ltx_td ltx_align_center ltx_border_t">50.1 <sub id="A7.T19.136.136.4.1" class="ltx_sub"><span id="A7.T19.136.136.4.1.1" class="ltx_text ltx_font_italic">(2.5)</span></sub>
</td>
<td id="A7.T19.137.137.5" class="ltx_td ltx_align_center ltx_border_t">34.9 <sub id="A7.T19.137.137.5.1" class="ltx_sub"><span id="A7.T19.137.137.5.1.1" class="ltx_text ltx_font_italic">(1.3)</span></sub>
</td>
<td id="A7.T19.138.138.6" class="ltx_td ltx_align_center ltx_border_t">39.1 <sub id="A7.T19.138.138.6.1" class="ltx_sub"><span id="A7.T19.138.138.6.1.1" class="ltx_text ltx_font_italic">(6.6)</span></sub>
</td>
</tr>
<tr id="A7.T19.144.144" class="ltx_tr">
<th id="A7.T19.144.144.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">w/ <span id="A7.T19.144.144.7.1" class="ltx_text ltx_font_smallcaps">Rerank</span>
</th>
<td id="A7.T19.139.139.1" class="ltx_td ltx_align_center">73.5 <sub id="A7.T19.139.139.1.1" class="ltx_sub"><span id="A7.T19.139.139.1.1.1" class="ltx_text ltx_font_italic">(2.1)</span></sub>
</td>
<td id="A7.T19.140.140.2" class="ltx_td ltx_align_center">32.9 <sub id="A7.T19.140.140.2.1" class="ltx_sub"><span id="A7.T19.140.140.2.1.1" class="ltx_text ltx_font_italic">(1.3)</span></sub>
</td>
<td id="A7.T19.141.141.3" class="ltx_td ltx_align_center">71.9 <sub id="A7.T19.141.141.3.1" class="ltx_sub"><span id="A7.T19.141.141.3.1.1" class="ltx_text ltx_font_italic">(1.9)</span></sub>
</td>
<td id="A7.T19.142.142.4" class="ltx_td ltx_align_center">65.4 <sub id="A7.T19.142.142.4.1" class="ltx_sub"><span id="A7.T19.142.142.4.1.1" class="ltx_text ltx_font_italic">(1.5)</span></sub>
</td>
<td id="A7.T19.143.143.5" class="ltx_td ltx_align_center">34.6 <sub id="A7.T19.143.143.5.1" class="ltx_sub"><span id="A7.T19.143.143.5.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
<td id="A7.T19.144.144.6" class="ltx_td ltx_align_center">35.7 <sub id="A7.T19.144.144.6.1" class="ltx_sub"><span id="A7.T19.144.144.6.1.1" class="ltx_text ltx_font_italic">(4.2)</span></sub>
</td>
</tr>
<tr id="A7.T19.150.150" class="ltx_tr">
<th id="A7.T19.150.150.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">Vicuna-13B <span id="A7.T19.150.150.7.1" class="ltx_text ltx_font_smallcaps">Summ</span> (10-psg)</th>
<td id="A7.T19.145.145.1" class="ltx_td ltx_align_center">67.7 <sub id="A7.T19.145.145.1.1" class="ltx_sub"><span id="A7.T19.145.145.1.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
<td id="A7.T19.146.146.2" class="ltx_td ltx_align_center">43.2 <sub id="A7.T19.146.146.2.1" class="ltx_sub"><span id="A7.T19.146.146.2.1.1" class="ltx_text ltx_font_italic">(0.1)</span></sub>
</td>
<td id="A7.T19.147.147.3" class="ltx_td ltx_align_center">52.7 <sub id="A7.T19.147.147.3.1" class="ltx_sub"><span id="A7.T19.147.147.3.1.1" class="ltx_text ltx_font_italic">(2.6)</span></sub>
</td>
<td id="A7.T19.148.148.4" class="ltx_td ltx_align_center">50.0 <sub id="A7.T19.148.148.4.1" class="ltx_sub"><span id="A7.T19.148.148.4.1.1" class="ltx_text ltx_font_italic">(2.1)</span></sub>
</td>
<td id="A7.T19.149.149.5" class="ltx_td ltx_align_center">36.7 <sub id="A7.T19.149.149.5.1" class="ltx_sub"><span id="A7.T19.149.149.5.1.1" class="ltx_text ltx_font_italic">(0.2)</span></sub>
</td>
<td id="A7.T19.150.150.6" class="ltx_td ltx_align_center">66.0 <sub id="A7.T19.150.150.6.1" class="ltx_sub"><span id="A7.T19.150.150.6.1.1" class="ltx_text ltx_font_italic">(1.2)</span></sub>
</td>
</tr>
<tr id="A7.T19.156.156" class="ltx_tr">
<th id="A7.T19.156.156.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">Vicuna-13B <span id="A7.T19.156.156.7.1" class="ltx_text ltx_font_smallcaps">Snippet</span> (10-psg)</th>
<td id="A7.T19.151.151.1" class="ltx_td ltx_align_center">81.4 <sub id="A7.T19.151.151.1.1" class="ltx_sub"><span id="A7.T19.151.151.1.1.1" class="ltx_text ltx_font_italic">(3.0)</span></sub>
</td>
<td id="A7.T19.152.152.2" class="ltx_td ltx_align_center">42.1 <sub id="A7.T19.152.152.2.1" class="ltx_sub"><span id="A7.T19.152.152.2.1.1" class="ltx_text ltx_font_italic">(1.2)</span></sub>
</td>
<td id="A7.T19.153.153.3" class="ltx_td ltx_align_center">53.4 <sub id="A7.T19.153.153.3.1" class="ltx_sub"><span id="A7.T19.153.153.3.1.1" class="ltx_text ltx_font_italic">(1.9)</span></sub>
</td>
<td id="A7.T19.154.154.4" class="ltx_td ltx_align_center">48.7 <sub id="A7.T19.154.154.4.1" class="ltx_sub"><span id="A7.T19.154.154.4.1.1" class="ltx_text ltx_font_italic">(1.6)</span></sub>
</td>
<td id="A7.T19.155.155.5" class="ltx_td ltx_align_center">36.9 <sub id="A7.T19.155.155.5.1" class="ltx_sub"><span id="A7.T19.155.155.5.1.1" class="ltx_text ltx_font_italic">(0.4)</span></sub>
</td>
<td id="A7.T19.156.156.6" class="ltx_td ltx_align_center">61.2 <sub id="A7.T19.156.156.6.1" class="ltx_sub"><span id="A7.T19.156.156.6.1.1" class="ltx_text ltx_font_italic">(7.4)</span></sub>
</td>
</tr>
<tr id="A7.T19.162.162" class="ltx_tr">
<th id="A7.T19.162.162.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">Vicuna-13B <span id="A7.T19.162.162.7.1" class="ltx_text ltx_font_smallcaps">Oracle</span> (3-psg)</th>
<td id="A7.T19.157.157.1" class="ltx_td ltx_align_center">72.9 <sub id="A7.T19.157.157.1.1" class="ltx_sub"><span id="A7.T19.157.157.1.1.1" class="ltx_text ltx_font_italic">(3.5)</span></sub>
</td>
<td id="A7.T19.158.158.2" class="ltx_td ltx_align_center">42.5 <sub id="A7.T19.158.158.2.1" class="ltx_sub"><span id="A7.T19.158.158.2.1.1" class="ltx_text ltx_font_italic">(1.6)</span></sub>
</td>
<td id="A7.T19.159.159.3" class="ltx_td ltx_align_center">52.2 <sub id="A7.T19.159.159.3.1" class="ltx_sub"><span id="A7.T19.159.159.3.1.1" class="ltx_text ltx_font_italic">(0.8)</span></sub>
</td>
<td id="A7.T19.160.160.4" class="ltx_td ltx_align_center">50.7 <sub id="A7.T19.160.160.4.1" class="ltx_sub"><span id="A7.T19.160.160.4.1.1" class="ltx_text ltx_font_italic">(1.6)</span></sub>
</td>
<td id="A7.T19.161.161.5" class="ltx_td ltx_align_center">36.5 <sub id="A7.T19.161.161.5.1" class="ltx_sub"><span id="A7.T19.161.161.5.1.1" class="ltx_text ltx_font_italic">(0.9)</span></sub>
</td>
<td id="A7.T19.162.162.6" class="ltx_td ltx_align_center">38.7 <sub id="A7.T19.162.162.6.1" class="ltx_sub"><span id="A7.T19.162.162.6.1.1" class="ltx_text ltx_font_italic">(3.5)</span></sub>
</td>
</tr>
<tr id="A7.T19.168.168" class="ltx_tr">
<th id="A7.T19.168.168.7" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">LLaMA-33B <span id="A7.T19.168.168.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (3-psg)</th>
<td id="A7.T19.163.163.1" class="ltx_td ltx_align_center ltx_border_t">83.7 <sub id="A7.T19.163.163.1.1" class="ltx_sub"><span id="A7.T19.163.163.1.1.1" class="ltx_text ltx_font_italic">(5.4)</span></sub>
</td>
<td id="A7.T19.164.164.2" class="ltx_td ltx_align_center ltx_border_t">31.0 <sub id="A7.T19.164.164.2.1" class="ltx_sub"><span id="A7.T19.164.164.2.1.1" class="ltx_text ltx_font_italic">(0.8)</span></sub>
</td>
<td id="A7.T19.165.165.3" class="ltx_td ltx_align_center ltx_border_t">19.5 <sub id="A7.T19.165.165.3.1" class="ltx_sub"><span id="A7.T19.165.165.3.1.1" class="ltx_text ltx_font_italic">(5.3)</span></sub>
</td>
<td id="A7.T19.166.166.4" class="ltx_td ltx_align_center ltx_border_t">23.0 <sub id="A7.T19.166.166.4.1" class="ltx_sub"><span id="A7.T19.166.166.4.1.1" class="ltx_text ltx_font_italic">(5.3)</span></sub>
</td>
<td id="A7.T19.167.167.5" class="ltx_td ltx_align_center ltx_border_t">32.3 <sub id="A7.T19.167.167.5.1" class="ltx_sub"><span id="A7.T19.167.167.5.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
<td id="A7.T19.168.168.6" class="ltx_td ltx_align_center ltx_border_t">44.1 <sub id="A7.T19.168.168.6.1" class="ltx_sub"><span id="A7.T19.168.168.6.1.1" class="ltx_text ltx_font_italic">(9.3)</span></sub>
</td>
</tr>
<tr id="A7.T19.174.174" class="ltx_tr">
<th id="A7.T19.174.174.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">w/ <span id="A7.T19.174.174.7.1" class="ltx_text ltx_font_smallcaps">Rerank</span>
</th>
<td id="A7.T19.169.169.1" class="ltx_td ltx_align_center">82.1 <sub id="A7.T19.169.169.1.1" class="ltx_sub"><span id="A7.T19.169.169.1.1.1" class="ltx_text ltx_font_italic">(3.0)</span></sub>
</td>
<td id="A7.T19.170.170.2" class="ltx_td ltx_align_center">31.3 <sub id="A7.T19.170.170.2.1" class="ltx_sub"><span id="A7.T19.170.170.2.1.1" class="ltx_text ltx_font_italic">(1.1)</span></sub>
</td>
<td id="A7.T19.171.171.3" class="ltx_td ltx_align_center">41.3 <sub id="A7.T19.171.171.3.1" class="ltx_sub"><span id="A7.T19.171.171.3.1.1" class="ltx_text ltx_font_italic">(6.4)</span></sub>
</td>
<td id="A7.T19.172.172.4" class="ltx_td ltx_align_center">44.7 <sub id="A7.T19.172.172.4.1" class="ltx_sub"><span id="A7.T19.172.172.4.1.1" class="ltx_text ltx_font_italic">(5.5)</span></sub>
</td>
<td id="A7.T19.173.173.5" class="ltx_td ltx_align_center">32.5 <sub id="A7.T19.173.173.5.1" class="ltx_sub"><span id="A7.T19.173.173.5.1.1" class="ltx_text ltx_font_italic">(0.9)</span></sub>
</td>
<td id="A7.T19.174.174.6" class="ltx_td ltx_align_center">39.4 <sub id="A7.T19.174.174.6.1" class="ltx_sub"><span id="A7.T19.174.174.6.1.1" class="ltx_text ltx_font_italic">(8.0)</span></sub>
</td>
</tr>
<tr id="A7.T19.180.180" class="ltx_tr">
<th id="A7.T19.180.180.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-33B <span id="A7.T19.180.180.7.1" class="ltx_text ltx_font_smallcaps">Summ</span> (10-psg)</th>
<td id="A7.T19.175.175.1" class="ltx_td ltx_align_center">72.0 <sub id="A7.T19.175.175.1.1" class="ltx_sub"><span id="A7.T19.175.175.1.1.1" class="ltx_text ltx_font_italic">(3.0)</span></sub>
</td>
<td id="A7.T19.176.176.2" class="ltx_td ltx_align_center">33.1 <sub id="A7.T19.176.176.2.1" class="ltx_sub"><span id="A7.T19.176.176.2.1.1" class="ltx_text ltx_font_italic">(1.9)</span></sub>
</td>
<td id="A7.T19.177.177.3" class="ltx_td ltx_align_center">34.7 <sub id="A7.T19.177.177.3.1" class="ltx_sub"><span id="A7.T19.177.177.3.1.1" class="ltx_text ltx_font_italic">(5.8)</span></sub>
</td>
<td id="A7.T19.178.178.4" class="ltx_td ltx_align_center">35.2 <sub id="A7.T19.178.178.4.1" class="ltx_sub"><span id="A7.T19.178.178.4.1.1" class="ltx_text ltx_font_italic">(6.0)</span></sub>
</td>
<td id="A7.T19.179.179.5" class="ltx_td ltx_align_center">31.1 <sub id="A7.T19.179.179.5.1" class="ltx_sub"><span id="A7.T19.179.179.5.1.1" class="ltx_text ltx_font_italic">(0.8)</span></sub>
</td>
<td id="A7.T19.180.180.6" class="ltx_td ltx_align_center">43.7 <sub id="A7.T19.180.180.6.1" class="ltx_sub"><span id="A7.T19.180.180.6.1.1" class="ltx_text ltx_font_italic">(5.0)</span></sub>
</td>
</tr>
<tr id="A7.T19.186.186" class="ltx_tr">
<th id="A7.T19.186.186.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-33B <span id="A7.T19.186.186.7.1" class="ltx_text ltx_font_smallcaps">Snippet</span> (10-psg)</th>
<td id="A7.T19.181.181.1" class="ltx_td ltx_align_center">70.8 <sub id="A7.T19.181.181.1.1" class="ltx_sub"><span id="A7.T19.181.181.1.1.1" class="ltx_text ltx_font_italic">(3.1)</span></sub>
</td>
<td id="A7.T19.182.182.2" class="ltx_td ltx_align_center">30.9 <sub id="A7.T19.182.182.2.1" class="ltx_sub"><span id="A7.T19.182.182.2.1.1" class="ltx_text ltx_font_italic">(1.4)</span></sub>
</td>
<td id="A7.T19.183.183.3" class="ltx_td ltx_align_center">31.4 <sub id="A7.T19.183.183.3.1" class="ltx_sub"><span id="A7.T19.183.183.3.1.1" class="ltx_text ltx_font_italic">(4.2)</span></sub>
</td>
<td id="A7.T19.184.184.4" class="ltx_td ltx_align_center">31.5 <sub id="A7.T19.184.184.4.1" class="ltx_sub"><span id="A7.T19.184.184.4.1.1" class="ltx_text ltx_font_italic">(5.3)</span></sub>
</td>
<td id="A7.T19.185.185.5" class="ltx_td ltx_align_center">30.1 <sub id="A7.T19.185.185.5.1" class="ltx_sub"><span id="A7.T19.185.185.5.1.1" class="ltx_text ltx_font_italic">(0.7)</span></sub>
</td>
<td id="A7.T19.186.186.6" class="ltx_td ltx_align_center">42.8 <sub id="A7.T19.186.186.6.1" class="ltx_sub"><span id="A7.T19.186.186.6.1.1" class="ltx_text ltx_font_italic">(3.6)</span></sub>
</td>
</tr>
<tr id="A7.T19.192.192" class="ltx_tr">
<th id="A7.T19.192.192.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-33B <span id="A7.T19.192.192.7.1" class="ltx_text ltx_font_smallcaps">Oracle</span> (3-psg)</th>
<td id="A7.T19.187.187.1" class="ltx_td ltx_align_center">82.6 <sub id="A7.T19.187.187.1.1" class="ltx_sub"><span id="A7.T19.187.187.1.1.1" class="ltx_text ltx_font_italic">(7.1)</span></sub>
</td>
<td id="A7.T19.188.188.2" class="ltx_td ltx_align_center">39.3 <sub id="A7.T19.188.188.2.1" class="ltx_sub"><span id="A7.T19.188.188.2.1.1" class="ltx_text ltx_font_italic">(2.9)</span></sub>
</td>
<td id="A7.T19.189.189.3" class="ltx_td ltx_align_center">20.2 <sub id="A7.T19.189.189.3.1" class="ltx_sub"><span id="A7.T19.189.189.3.1.1" class="ltx_text ltx_font_italic">(6.2)</span></sub>
</td>
<td id="A7.T19.190.190.4" class="ltx_td ltx_align_center">23.9 <sub id="A7.T19.190.190.4.1" class="ltx_sub"><span id="A7.T19.190.190.4.1.1" class="ltx_text ltx_font_italic">(6.3)</span></sub>
</td>
<td id="A7.T19.191.191.5" class="ltx_td ltx_align_center">33.1 <sub id="A7.T19.191.191.5.1" class="ltx_sub"><span id="A7.T19.191.191.5.1.1" class="ltx_text ltx_font_italic">(0.9)</span></sub>
</td>
<td id="A7.T19.192.192.6" class="ltx_td ltx_align_center">42.0 <sub id="A7.T19.192.192.6.1" class="ltx_sub"><span id="A7.T19.192.192.6.1.1" class="ltx_text ltx_font_italic">(9.3)</span></sub>
</td>
</tr>
<tr id="A7.T19.198.198" class="ltx_tr">
<th id="A7.T19.198.198.7" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Oasst-33B <span id="A7.T19.198.198.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (3-psg)</th>
<td id="A7.T19.193.193.1" class="ltx_td ltx_align_center ltx_border_t">82.9 <sub id="A7.T19.193.193.1.1" class="ltx_sub"><span id="A7.T19.193.193.1.1.1" class="ltx_text ltx_font_italic">(2.7)</span></sub>
</td>
<td id="A7.T19.194.194.2" class="ltx_td ltx_align_center ltx_border_t">34.8 <sub id="A7.T19.194.194.2.1" class="ltx_sub"><span id="A7.T19.194.194.2.1.1" class="ltx_text ltx_font_italic">(1.5)</span></sub>
</td>
<td id="A7.T19.195.195.3" class="ltx_td ltx_align_center ltx_border_t">36.2 <sub id="A7.T19.195.195.3.1" class="ltx_sub"><span id="A7.T19.195.195.3.1.1" class="ltx_text ltx_font_italic">(1.7)</span></sub>
</td>
<td id="A7.T19.196.196.4" class="ltx_td ltx_align_center ltx_border_t">38.3 <sub id="A7.T19.196.196.4.1" class="ltx_sub"><span id="A7.T19.196.196.4.1.1" class="ltx_text ltx_font_italic">(2.7)</span></sub>
</td>
<td id="A7.T19.197.197.5" class="ltx_td ltx_align_center ltx_border_t">35.5 <sub id="A7.T19.197.197.5.1" class="ltx_sub"><span id="A7.T19.197.197.5.1.1" class="ltx_text ltx_font_italic">(0.7)</span></sub>
</td>
<td id="A7.T19.198.198.6" class="ltx_td ltx_align_center ltx_border_t">45.2 <sub id="A7.T19.198.198.6.1" class="ltx_sub"><span id="A7.T19.198.198.6.1.1" class="ltx_text ltx_font_italic">(6.3)</span></sub>
</td>
</tr>
<tr id="A7.T19.204.204" class="ltx_tr">
<th id="A7.T19.204.204.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">w/ <span id="A7.T19.204.204.7.1" class="ltx_text ltx_font_smallcaps">Rerank</span>
</th>
<td id="A7.T19.199.199.1" class="ltx_td ltx_align_center">83.2 <sub id="A7.T19.199.199.1.1" class="ltx_sub"><span id="A7.T19.199.199.1.1.1" class="ltx_text ltx_font_italic">(2.4)</span></sub>
</td>
<td id="A7.T19.200.200.2" class="ltx_td ltx_align_center">35.1 <sub id="A7.T19.200.200.2.1" class="ltx_sub"><span id="A7.T19.200.200.2.1.1" class="ltx_text ltx_font_italic">(1.4)</span></sub>
</td>
<td id="A7.T19.201.201.3" class="ltx_td ltx_align_center">66.7 <sub id="A7.T19.201.201.3.1" class="ltx_sub"><span id="A7.T19.201.201.3.1.1" class="ltx_text ltx_font_italic">(0.2)</span></sub>
</td>
<td id="A7.T19.202.202.4" class="ltx_td ltx_align_center">64.3 <sub id="A7.T19.202.202.4.1" class="ltx_sub"><span id="A7.T19.202.202.4.1.1" class="ltx_text ltx_font_italic">(1.0)</span></sub>
</td>
<td id="A7.T19.203.203.5" class="ltx_td ltx_align_center">35.0 <sub id="A7.T19.203.203.5.1" class="ltx_sub"><span id="A7.T19.203.203.5.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
<td id="A7.T19.204.204.6" class="ltx_td ltx_align_center">41.8 <sub id="A7.T19.204.204.6.1" class="ltx_sub"><span id="A7.T19.204.204.6.1.1" class="ltx_text ltx_font_italic">(6.0)</span></sub>
</td>
</tr>
<tr id="A7.T19.210.210" class="ltx_tr">
<th id="A7.T19.210.210.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">Oasst-33B <span id="A7.T19.210.210.7.1" class="ltx_text ltx_font_smallcaps">Summ</span> (10-psg)</th>
<td id="A7.T19.205.205.1" class="ltx_td ltx_align_center">74.3 <sub id="A7.T19.205.205.1.1" class="ltx_sub"><span id="A7.T19.205.205.1.1.1" class="ltx_text ltx_font_italic">(4.6)</span></sub>
</td>
<td id="A7.T19.206.206.2" class="ltx_td ltx_align_center">40.9 <sub id="A7.T19.206.206.2.1" class="ltx_sub"><span id="A7.T19.206.206.2.1.1" class="ltx_text ltx_font_italic">(1.1)</span></sub>
</td>
<td id="A7.T19.207.207.3" class="ltx_td ltx_align_center">45.5 <sub id="A7.T19.207.207.3.1" class="ltx_sub"><span id="A7.T19.207.207.3.1.1" class="ltx_text ltx_font_italic">(1.9)</span></sub>
</td>
<td id="A7.T19.208.208.4" class="ltx_td ltx_align_center">44.0 <sub id="A7.T19.208.208.4.1" class="ltx_sub"><span id="A7.T19.208.208.4.1.1" class="ltx_text ltx_font_italic">(2.9)</span></sub>
</td>
<td id="A7.T19.209.209.5" class="ltx_td ltx_align_center">35.8 <sub id="A7.T19.209.209.5.1" class="ltx_sub"><span id="A7.T19.209.209.5.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
<td id="A7.T19.210.210.6" class="ltx_td ltx_align_center">54.3 <sub id="A7.T19.210.210.6.1" class="ltx_sub"><span id="A7.T19.210.210.6.1.1" class="ltx_text ltx_font_italic">(4.8)</span></sub>
</td>
</tr>
<tr id="A7.T19.216.216" class="ltx_tr">
<th id="A7.T19.216.216.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">Oasst-33B <span id="A7.T19.216.216.7.1" class="ltx_text ltx_font_smallcaps">Snippet</span> (10-psg)</th>
<td id="A7.T19.211.211.1" class="ltx_td ltx_align_center">79.3 <sub id="A7.T19.211.211.1.1" class="ltx_sub"><span id="A7.T19.211.211.1.1.1" class="ltx_text ltx_font_italic">(1.0)</span></sub>
</td>
<td id="A7.T19.212.212.2" class="ltx_td ltx_align_center">40.1 <sub id="A7.T19.212.212.2.1" class="ltx_sub"><span id="A7.T19.212.212.2.1.1" class="ltx_text ltx_font_italic">(0.9)</span></sub>
</td>
<td id="A7.T19.213.213.3" class="ltx_td ltx_align_center">45.0 <sub id="A7.T19.213.213.3.1" class="ltx_sub"><span id="A7.T19.213.213.3.1.1" class="ltx_text ltx_font_italic">(1.3)</span></sub>
</td>
<td id="A7.T19.214.214.4" class="ltx_td ltx_align_center">43.3 <sub id="A7.T19.214.214.4.1" class="ltx_sub"><span id="A7.T19.214.214.4.1.1" class="ltx_text ltx_font_italic">(2.2)</span></sub>
</td>
<td id="A7.T19.215.215.5" class="ltx_td ltx_align_center">35.8 <sub id="A7.T19.215.215.5.1" class="ltx_sub"><span id="A7.T19.215.215.5.1.1" class="ltx_text ltx_font_italic">(0.2)</span></sub>
</td>
<td id="A7.T19.216.216.6" class="ltx_td ltx_align_center">50.9 <sub id="A7.T19.216.216.6.1" class="ltx_sub"><span id="A7.T19.216.216.6.1.1" class="ltx_text ltx_font_italic">(4.1)</span></sub>
</td>
</tr>
<tr id="A7.T19.222.222" class="ltx_tr">
<th id="A7.T19.222.222.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">Oasst-33B <span id="A7.T19.222.222.7.1" class="ltx_text ltx_font_smallcaps">Oracle</span> (3-psg)</th>
<td id="A7.T19.217.217.1" class="ltx_td ltx_align_center">85.1 <sub id="A7.T19.217.217.1.1" class="ltx_sub"><span id="A7.T19.217.217.1.1.1" class="ltx_text ltx_font_italic">(2.8)</span></sub>
</td>
<td id="A7.T19.218.218.2" class="ltx_td ltx_align_center">44.3 <sub id="A7.T19.218.218.2.1" class="ltx_sub"><span id="A7.T19.218.218.2.1.1" class="ltx_text ltx_font_italic">(2.4)</span></sub>
</td>
<td id="A7.T19.219.219.3" class="ltx_td ltx_align_center">37.0 <sub id="A7.T19.219.219.3.1" class="ltx_sub"><span id="A7.T19.219.219.3.1.1" class="ltx_text ltx_font_italic">(1.0)</span></sub>
</td>
<td id="A7.T19.220.220.4" class="ltx_td ltx_align_center">39.6 <sub id="A7.T19.220.220.4.1" class="ltx_sub"><span id="A7.T19.220.220.4.1.1" class="ltx_text ltx_font_italic">(1.5)</span></sub>
</td>
<td id="A7.T19.221.221.5" class="ltx_td ltx_align_center">36.5 <sub id="A7.T19.221.221.5.1" class="ltx_sub"><span id="A7.T19.221.221.5.1.1" class="ltx_text ltx_font_italic">(1.1)</span></sub>
</td>
<td id="A7.T19.222.222.6" class="ltx_td ltx_align_center">44.2 <sub id="A7.T19.222.222.6.1" class="ltx_sub"><span id="A7.T19.222.222.6.1.1" class="ltx_text ltx_font_italic">(5.8)</span></sub>
</td>
</tr>
<tr id="A7.T19.228.228" class="ltx_tr">
<th id="A7.T19.228.228.7" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">LLaMA-2-7B-Chat <span id="A7.T19.228.228.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (5-psg)</th>
<td id="A7.T19.223.223.1" class="ltx_td ltx_align_center ltx_border_t">80.1 <sub id="A7.T19.223.223.1.1" class="ltx_sub"><span id="A7.T19.223.223.1.1.1" class="ltx_text ltx_font_italic">(6.5)</span></sub>
</td>
<td id="A7.T19.224.224.2" class="ltx_td ltx_align_center ltx_border_t">33.9 <sub id="A7.T19.224.224.2.1" class="ltx_sub"><span id="A7.T19.224.224.2.1.1" class="ltx_text ltx_font_italic">(2.1)</span></sub>
</td>
<td id="A7.T19.225.225.3" class="ltx_td ltx_align_center ltx_border_t">50.9 <sub id="A7.T19.225.225.3.1" class="ltx_sub"><span id="A7.T19.225.225.3.1.1" class="ltx_text ltx_font_italic">(4.5)</span></sub>
</td>
<td id="A7.T19.226.226.4" class="ltx_td ltx_align_center ltx_border_t">47.5 <sub id="A7.T19.226.226.4.1" class="ltx_sub"><span id="A7.T19.226.226.4.1.1" class="ltx_text ltx_font_italic">(3.7)</span></sub>
</td>
<td id="A7.T19.227.227.5" class="ltx_td ltx_align_center ltx_border_t">35.1 <sub id="A7.T19.227.227.5.1" class="ltx_sub"><span id="A7.T19.227.227.5.1.1" class="ltx_text ltx_font_italic">(0.9)</span></sub>
</td>
<td id="A7.T19.228.228.6" class="ltx_td ltx_align_center ltx_border_t">42.3 <sub id="A7.T19.228.228.6.1" class="ltx_sub"><span id="A7.T19.228.228.6.1.1" class="ltx_text ltx_font_italic">(10.1)</span></sub>
</td>
</tr>
<tr id="A7.T19.234.234" class="ltx_tr">
<th id="A7.T19.234.234.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-2-13B-Chat <span id="A7.T19.234.234.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (5-psg)</th>
<td id="A7.T19.229.229.1" class="ltx_td ltx_align_center">72.4 <sub id="A7.T19.229.229.1.1" class="ltx_sub"><span id="A7.T19.229.229.1.1.1" class="ltx_text ltx_font_italic">(6.3)</span></sub>
</td>
<td id="A7.T19.230.230.2" class="ltx_td ltx_align_center">35.2 <sub id="A7.T19.230.230.2.1" class="ltx_sub"><span id="A7.T19.230.230.2.1.1" class="ltx_text ltx_font_italic">(1.2)</span></sub>
</td>
<td id="A7.T19.231.231.3" class="ltx_td ltx_align_center">38.4 <sub id="A7.T19.231.231.3.1" class="ltx_sub"><span id="A7.T19.231.231.3.1.1" class="ltx_text ltx_font_italic">(5.9)</span></sub>
</td>
<td id="A7.T19.232.232.4" class="ltx_td ltx_align_center">39.4 <sub id="A7.T19.232.232.4.1" class="ltx_sub"><span id="A7.T19.232.232.4.1.1" class="ltx_text ltx_font_italic">(4.8)</span></sub>
</td>
<td id="A7.T19.233.233.5" class="ltx_td ltx_align_center">35.8 <sub id="A7.T19.233.233.5.1" class="ltx_sub"><span id="A7.T19.233.233.5.1.1" class="ltx_text ltx_font_italic">(0.9)</span></sub>
</td>
<td id="A7.T19.234.234.6" class="ltx_td ltx_align_center">38.0 <sub id="A7.T19.234.234.6.1" class="ltx_sub"><span id="A7.T19.234.234.6.1.1" class="ltx_text ltx_font_italic">(6.4)</span></sub>
</td>
</tr>
<tr id="A7.T19.240.240" class="ltx_tr">
<th id="A7.T19.240.240.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-2-70B-Chat <span id="A7.T19.240.240.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (5-psg)</th>
<td id="A7.T19.235.235.1" class="ltx_td ltx_align_center">88.3 <sub id="A7.T19.235.235.1.1" class="ltx_sub"><span id="A7.T19.235.235.1.1.1" class="ltx_text ltx_font_italic">(4.1)</span></sub>
</td>
<td id="A7.T19.236.236.2" class="ltx_td ltx_align_center">41.5 <sub id="A7.T19.236.236.2.1" class="ltx_sub"><span id="A7.T19.236.236.2.1.1" class="ltx_text ltx_font_italic">(0.8)</span></sub>
</td>
<td id="A7.T19.237.237.3" class="ltx_td ltx_align_center">62.9 <sub id="A7.T19.237.237.3.1" class="ltx_sub"><span id="A7.T19.237.237.3.1.1" class="ltx_text ltx_font_italic">(1.4)</span></sub>
</td>
<td id="A7.T19.238.238.4" class="ltx_td ltx_align_center">61.3 <sub id="A7.T19.238.238.4.1" class="ltx_sub"><span id="A7.T19.238.238.4.1.1" class="ltx_text ltx_font_italic">(2.1)</span></sub>
</td>
<td id="A7.T19.239.239.5" class="ltx_td ltx_align_center">37.1 <sub id="A7.T19.239.239.5.1" class="ltx_sub"><span id="A7.T19.239.239.5.1.1" class="ltx_text ltx_font_italic">(0.4)</span></sub>
</td>
<td id="A7.T19.240.240.6" class="ltx_td ltx_align_center">52.9 <sub id="A7.T19.240.240.6.1" class="ltx_sub"><span id="A7.T19.240.240.6.1.1" class="ltx_text ltx_font_italic">(9.5)</span></sub>
</td>
</tr>
<tr id="A7.T19.246.246" class="ltx_tr">
<th id="A7.T19.246.246.7" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_t">Stable Beluga 2 <span id="A7.T19.246.246.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (5-psg)</th>
<td id="A7.T19.241.241.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">59.7 <sub id="A7.T19.241.241.1.1" class="ltx_sub"><span id="A7.T19.241.241.1.1.1" class="ltx_text ltx_font_italic">(3.3)</span></sub>
</td>
<td id="A7.T19.242.242.2" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">37.2 <sub id="A7.T19.242.242.2.1" class="ltx_sub"><span id="A7.T19.242.242.2.1.1" class="ltx_text ltx_font_italic">(1.0)</span></sub>
</td>
<td id="A7.T19.243.243.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">63.6 <sub id="A7.T19.243.243.3.1" class="ltx_sub"><span id="A7.T19.243.243.3.1.1" class="ltx_text ltx_font_italic">(0.7)</span></sub>
</td>
<td id="A7.T19.244.244.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">63.5 <sub id="A7.T19.244.244.4.1" class="ltx_sub"><span id="A7.T19.244.244.4.1.1" class="ltx_text ltx_font_italic">(0.1)</span></sub>
</td>
<td id="A7.T19.245.245.5" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">35.3 <sub id="A7.T19.245.245.5.1" class="ltx_sub"><span id="A7.T19.245.245.5.1.1" class="ltx_text ltx_font_italic">(0.7)</span></sub>
</td>
<td id="A7.T19.246.246.6" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">23.8 <sub id="A7.T19.246.246.6.1" class="ltx_sub"><span id="A7.T19.246.246.6.1.1" class="ltx_text ltx_font_italic">(3.2)</span></sub>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 19: </span>
ASQA full results. </figcaption>
</figure>
<figure id="A7.T20" class="ltx_table">
<table id="A7.T20.205" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A7.T20.205.206.1" class="ltx_tr">
<th id="A7.T20.205.206.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<td id="A7.T20.205.206.1.2" class="ltx_td ltx_align_center ltx_border_tt" colspan="2"><span id="A7.T20.205.206.1.2.1" class="ltx_text ltx_font_bold">Correctness</span></td>
<td id="A7.T20.205.206.1.3" class="ltx_td ltx_align_center ltx_border_tt" colspan="2"><span id="A7.T20.205.206.1.3.1" class="ltx_text ltx_font_bold">Citation</span></td>
<td id="A7.T20.205.206.1.4" class="ltx_td ltx_border_tt"></td>
</tr>
<tr id="A7.T20.205.207.2" class="ltx_tr">
<th id="A7.T20.205.207.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="A7.T20.205.207.2.2" class="ltx_td ltx_align_center ltx_border_t">Rec.-5</td>
<td id="A7.T20.205.207.2.3" class="ltx_td ltx_align_center ltx_border_t">Prec.</td>
<td id="A7.T20.205.207.2.4" class="ltx_td ltx_align_center ltx_border_t">Rec.</td>
<td id="A7.T20.205.207.2.5" class="ltx_td ltx_align_center ltx_border_t">Prec.</td>
<td id="A7.T20.205.207.2.6" class="ltx_td ltx_align_center">Num Pred.</td>
</tr>
<tr id="A7.T20.205.208.3" class="ltx_tr" style="background-color:#EBEBEB;">
<th id="A7.T20.205.208.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" colspan="6"><span id="A7.T20.205.208.3.1.1" class="ltx_text ltx_font_bold" style="background-color:#EBEBEB;">ChatGPT</span></th>
</tr>
<tr id="A7.T20.5.5" class="ltx_tr">
<th id="A7.T20.5.5.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T20.5.5.6.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (5-psg)</th>
<td id="A7.T20.1.1.1" class="ltx_td ltx_align_center">20.8 <sub id="A7.T20.1.1.1.1" class="ltx_sub"><span id="A7.T20.1.1.1.1.1" class="ltx_text ltx_font_italic">(2.2)</span></sub>
</td>
<td id="A7.T20.2.2.2" class="ltx_td ltx_align_center">20.8 <sub id="A7.T20.2.2.2.1" class="ltx_sub"><span id="A7.T20.2.2.2.1.1" class="ltx_text ltx_font_italic">(0.2)</span></sub>
</td>
<td id="A7.T20.3.3.3" class="ltx_td ltx_align_center">20.5 <sub id="A7.T20.3.3.3.1" class="ltx_sub"><span id="A7.T20.3.3.3.1.1" class="ltx_text ltx_font_italic">(0.7)</span></sub>
</td>
<td id="A7.T20.4.4.4" class="ltx_td ltx_align_center">20.9 <sub id="A7.T20.4.4.4.1" class="ltx_sub"><span id="A7.T20.4.4.4.1.1" class="ltx_text ltx_font_italic">(0.7)</span></sub>
</td>
<td id="A7.T20.5.5.5" class="ltx_td ltx_align_center">5.0 <sub id="A7.T20.5.5.5.1" class="ltx_sub"><span id="A7.T20.5.5.5.1.1" class="ltx_text ltx_font_italic">(0.5)</span></sub>
</td>
</tr>
<tr id="A7.T20.10.10" class="ltx_tr">
<th id="A7.T20.10.10.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">w/ <span id="A7.T20.10.10.6.1" class="ltx_text ltx_font_smallcaps">Rerank</span>
</th>
<td id="A7.T20.6.6.1" class="ltx_td ltx_align_center">22.8 <sub id="A7.T20.6.6.1.1" class="ltx_sub"><span id="A7.T20.6.6.1.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T20.7.7.2" class="ltx_td ltx_align_center">21.4 <sub id="A7.T20.7.7.2.1" class="ltx_sub"><span id="A7.T20.7.7.2.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T20.8.8.3" class="ltx_td ltx_align_center">21.2 <sub id="A7.T20.8.8.3.1" class="ltx_sub"><span id="A7.T20.8.8.3.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T20.9.9.4" class="ltx_td ltx_align_center">21.4 <sub id="A7.T20.9.9.4.1" class="ltx_sub"><span id="A7.T20.9.9.4.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T20.10.10.5" class="ltx_td ltx_align_center">5.4 <sub id="A7.T20.10.10.5.1" class="ltx_sub"><span id="A7.T20.10.10.5.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
</tr>
<tr id="A7.T20.15.15" class="ltx_tr">
<th id="A7.T20.15.15.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T20.15.15.6.1" class="ltx_text ltx_font_smallcaps">Summ</span> (10-psg)</th>
<td id="A7.T20.11.11.1" class="ltx_td ltx_align_center">23.6 <sub id="A7.T20.11.11.1.1" class="ltx_sub"><span id="A7.T20.11.11.1.1.1" class="ltx_text ltx_font_italic">(0.9)</span></sub>
</td>
<td id="A7.T20.12.12.2" class="ltx_td ltx_align_center">21.2 <sub id="A7.T20.12.12.2.1" class="ltx_sub"><span id="A7.T20.12.12.2.1.1" class="ltx_text ltx_font_italic">(0.5)</span></sub>
</td>
<td id="A7.T20.13.13.3" class="ltx_td ltx_align_center">23.6 <sub id="A7.T20.13.13.3.1" class="ltx_sub"><span id="A7.T20.13.13.3.1.1" class="ltx_text ltx_font_italic">(0.7)</span></sub>
</td>
<td id="A7.T20.14.14.4" class="ltx_td ltx_align_center">25.7 <sub id="A7.T20.14.14.4.1" class="ltx_sub"><span id="A7.T20.14.14.4.1.1" class="ltx_text ltx_font_italic">(0.8)</span></sub>
</td>
<td id="A7.T20.15.15.5" class="ltx_td ltx_align_center">6.7 <sub id="A7.T20.15.15.5.1" class="ltx_sub"><span id="A7.T20.15.15.5.1.1" class="ltx_text ltx_font_italic">(0.4)</span></sub>
</td>
</tr>
<tr id="A7.T20.20.20" class="ltx_tr">
<th id="A7.T20.20.20.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T20.20.20.6.1" class="ltx_text ltx_font_smallcaps">Snippet</span> (10-psg)</th>
<td id="A7.T20.16.16.1" class="ltx_td ltx_align_center">24.5 <sub id="A7.T20.16.16.1.1" class="ltx_sub"><span id="A7.T20.16.16.1.1.1" class="ltx_text ltx_font_italic">(1.4)</span></sub>
</td>
<td id="A7.T20.17.17.2" class="ltx_td ltx_align_center">21.5 <sub id="A7.T20.17.17.2.1" class="ltx_sub"><span id="A7.T20.17.17.2.1.1" class="ltx_text ltx_font_italic">(1.8)</span></sub>
</td>
<td id="A7.T20.18.18.3" class="ltx_td ltx_align_center">22.9 <sub id="A7.T20.18.18.3.1" class="ltx_sub"><span id="A7.T20.18.18.3.1.1" class="ltx_text ltx_font_italic">(1.6)</span></sub>
</td>
<td id="A7.T20.19.19.4" class="ltx_td ltx_align_center">24.9 <sub id="A7.T20.19.19.4.1" class="ltx_sub"><span id="A7.T20.19.19.4.1.1" class="ltx_text ltx_font_italic">(0.4)</span></sub>
</td>
<td id="A7.T20.20.20.5" class="ltx_td ltx_align_center">7.2 <sub id="A7.T20.20.20.5.1" class="ltx_sub"><span id="A7.T20.20.20.5.1.1" class="ltx_text ltx_font_italic">(0.9)</span></sub>
</td>
</tr>
<tr id="A7.T20.25.25" class="ltx_tr">
<th id="A7.T20.25.25.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">w/ <span id="A7.T20.25.25.6.1" class="ltx_text ltx_font_smallcaps">Interact</span>
</th>
<td id="A7.T20.21.21.1" class="ltx_td ltx_align_center">21.9 <sub id="A7.T20.21.21.1.1" class="ltx_sub"><span id="A7.T20.21.21.1.1.1" class="ltx_text ltx_font_italic">(0.9)</span></sub>
</td>
<td id="A7.T20.22.22.2" class="ltx_td ltx_align_center">23.0 <sub id="A7.T20.22.22.2.1" class="ltx_sub"><span id="A7.T20.22.22.2.1.1" class="ltx_text ltx_font_italic">(0.4)</span></sub>
</td>
<td id="A7.T20.23.23.3" class="ltx_td ltx_align_center">21.9 <sub id="A7.T20.23.23.3.1" class="ltx_sub"><span id="A7.T20.23.23.3.1.1" class="ltx_text ltx_font_italic">(1.2)</span></sub>
</td>
<td id="A7.T20.24.24.4" class="ltx_td ltx_align_center">23.4 <sub id="A7.T20.24.24.4.1" class="ltx_sub"><span id="A7.T20.24.24.4.1.1" class="ltx_text ltx_font_italic">(0.9)</span></sub>
</td>
<td id="A7.T20.25.25.5" class="ltx_td ltx_align_center">6.7 <sub id="A7.T20.25.25.5.1" class="ltx_sub"><span id="A7.T20.25.25.5.1.1" class="ltx_text ltx_font_italic">(0.4)</span></sub>
</td>
</tr>
<tr id="A7.T20.30.30" class="ltx_tr">
<th id="A7.T20.30.30.6" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A7.T20.30.30.6.1" class="ltx_text ltx_font_smallcaps">InlineSearch</span></th>
<td id="A7.T20.26.26.1" class="ltx_td ltx_align_center">17.2 <sub id="A7.T20.26.26.1.1" class="ltx_sub"><span id="A7.T20.26.26.1.1.1" class="ltx_text ltx_font_italic">(1.1)</span></sub>
</td>
<td id="A7.T20.27.27.2" class="ltx_td ltx_align_center">20.4 <sub id="A7.T20.27.27.2.1" class="ltx_sub"><span id="A7.T20.27.27.2.1.1" class="ltx_text ltx_font_italic">(0.8)</span></sub>
</td>
<td id="A7.T20.28.28.3" class="ltx_td ltx_align_center">14.9 <sub id="A7.T20.28.28.3.1" class="ltx_sub"><span id="A7.T20.28.28.3.1.1" class="ltx_text ltx_font_italic">(0.8)</span></sub>
</td>
<td id="A7.T20.29.29.4" class="ltx_td ltx_align_center">14.9 <sub id="A7.T20.29.29.4.1" class="ltx_sub"><span id="A7.T20.29.29.4.1.1" class="ltx_text ltx_font_italic">(0.8)</span></sub>
</td>
<td id="A7.T20.30.30.5" class="ltx_td ltx_align_center">6.7 <sub id="A7.T20.30.30.5.1" class="ltx_sub"><span id="A7.T20.30.30.5.1.1" class="ltx_text ltx_font_italic">(0.2)</span></sub>
</td>
</tr>
<tr id="A7.T20.35.35" class="ltx_tr">
<th id="A7.T20.35.35.6" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A7.T20.35.35.6.1" class="ltx_text ltx_font_smallcaps">ClosedBook</span></th>
<td id="A7.T20.31.31.1" class="ltx_td ltx_align_center">32.9 <sub id="A7.T20.31.31.1.1" class="ltx_sub"><span id="A7.T20.31.31.1.1.1" class="ltx_text ltx_font_italic">(1.1)</span></sub>
</td>
<td id="A7.T20.32.32.2" class="ltx_td ltx_align_center">19.8 <sub id="A7.T20.32.32.2.1" class="ltx_sub"><span id="A7.T20.32.32.2.1.1" class="ltx_text ltx_font_italic">(1.6)</span></sub>
</td>
<td id="A7.T20.33.33.3" class="ltx_td ltx_align_center">10.0 <sub id="A7.T20.33.33.3.1" class="ltx_sub"><span id="A7.T20.33.33.3.1.1" class="ltx_text ltx_font_italic">(0.4)</span></sub>
</td>
<td id="A7.T20.34.34.4" class="ltx_td ltx_align_center">10.0 <sub id="A7.T20.34.34.4.1" class="ltx_sub"><span id="A7.T20.34.34.4.1.1" class="ltx_text ltx_font_italic">(0.4)</span></sub>
</td>
<td id="A7.T20.35.35.5" class="ltx_td ltx_align_center">17.0 <sub id="A7.T20.35.35.5.1" class="ltx_sub"><span id="A7.T20.35.35.5.1.1" class="ltx_text ltx_font_italic">(2.9)</span></sub>
</td>
</tr>
<tr id="A7.T20.40.40" class="ltx_tr">
<th id="A7.T20.40.40.6" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A7.T20.40.40.6.1" class="ltx_text ltx_font_smallcaps">Oracle</span></th>
<td id="A7.T20.36.36.1" class="ltx_td ltx_align_center">37.0 <sub id="A7.T20.36.36.1.1" class="ltx_sub"><span id="A7.T20.36.36.1.1.1" class="ltx_text ltx_font_italic">(3.1)</span></sub>
</td>
<td id="A7.T20.37.37.2" class="ltx_td ltx_align_center">36.9 <sub id="A7.T20.37.37.2.1" class="ltx_sub"><span id="A7.T20.37.37.2.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
<td id="A7.T20.38.38.3" class="ltx_td ltx_align_center">24.1 <sub id="A7.T20.38.38.3.1" class="ltx_sub"><span id="A7.T20.38.38.3.1.1" class="ltx_text ltx_font_italic">(1.2)</span></sub>
</td>
<td id="A7.T20.39.39.4" class="ltx_td ltx_align_center">24.6 <sub id="A7.T20.39.39.4.1" class="ltx_sub"><span id="A7.T20.39.39.4.1.1" class="ltx_text ltx_font_italic">(1.3)</span></sub>
</td>
<td id="A7.T20.40.40.5" class="ltx_td ltx_align_center">5.3 <sub id="A7.T20.40.40.5.1" class="ltx_sub"><span id="A7.T20.40.40.5.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
</tr>
<tr id="A7.T20.205.209.4" class="ltx_tr">
<th id="A7.T20.205.209.4.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" colspan="6"><span id="A7.T20.205.209.4.1.1" class="ltx_text ltx_font_bold" style="background-color:#E9E9E9;">ChatGPT-16K</span></th>
</tr>
<tr id="A7.T20.45.45" class="ltx_tr">
<th id="A7.T20.45.45.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T20.45.45.6.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (5-psg)</th>
<td id="A7.T20.41.41.1" class="ltx_td ltx_align_center">21.1 <sub id="A7.T20.41.41.1.1" class="ltx_sub"><span id="A7.T20.41.41.1.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T20.42.42.2" class="ltx_td ltx_align_center">22.0 <sub id="A7.T20.42.42.2.1" class="ltx_sub"><span id="A7.T20.42.42.2.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T20.43.43.3" class="ltx_td ltx_align_center">20.7 <sub id="A7.T20.43.43.3.1" class="ltx_sub"><span id="A7.T20.43.43.3.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T20.44.44.4" class="ltx_td ltx_align_center">21.2 <sub id="A7.T20.44.44.4.1" class="ltx_sub"><span id="A7.T20.44.44.4.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T20.45.45.5" class="ltx_td ltx_align_center">4.9 <sub id="A7.T20.45.45.5.1" class="ltx_sub"><span id="A7.T20.45.45.5.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
</tr>
<tr id="A7.T20.50.50" class="ltx_tr">
<th id="A7.T20.50.50.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T20.50.50.6.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (10-psg)</th>
<td id="A7.T20.46.46.1" class="ltx_td ltx_align_center">23.4 <sub id="A7.T20.46.46.1.1" class="ltx_sub"><span id="A7.T20.46.46.1.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T20.47.47.2" class="ltx_td ltx_align_center">21.9 <sub id="A7.T20.47.47.2.1" class="ltx_sub"><span id="A7.T20.47.47.2.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T20.48.48.3" class="ltx_td ltx_align_center">21.6 <sub id="A7.T20.48.48.3.1" class="ltx_sub"><span id="A7.T20.48.48.3.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T20.49.49.4" class="ltx_td ltx_align_center">22.0 <sub id="A7.T20.49.49.4.1" class="ltx_sub"><span id="A7.T20.49.49.4.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T20.50.50.5" class="ltx_td ltx_align_center">5.7 <sub id="A7.T20.50.50.5.1" class="ltx_sub"><span id="A7.T20.50.50.5.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
</tr>
<tr id="A7.T20.55.55" class="ltx_tr">
<th id="A7.T20.55.55.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T20.55.55.6.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (20-psg)</th>
<td id="A7.T20.51.51.1" class="ltx_td ltx_align_center">26.4 <sub id="A7.T20.51.51.1.1" class="ltx_sub"><span id="A7.T20.51.51.1.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T20.52.52.2" class="ltx_td ltx_align_center">21.1 <sub id="A7.T20.52.52.2.1" class="ltx_sub"><span id="A7.T20.52.52.2.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T20.53.53.3" class="ltx_td ltx_align_center">19.4 <sub id="A7.T20.53.53.3.1" class="ltx_sub"><span id="A7.T20.53.53.3.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T20.54.54.4" class="ltx_td ltx_align_center">19.7 <sub id="A7.T20.54.54.4.1" class="ltx_sub"><span id="A7.T20.54.54.4.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T20.55.55.5" class="ltx_td ltx_align_center">7.6 <sub id="A7.T20.55.55.5.1" class="ltx_sub"><span id="A7.T20.55.55.5.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
</tr>
<tr id="A7.T20.205.210.5" class="ltx_tr">
<th id="A7.T20.205.210.5.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" colspan="6"><span id="A7.T20.205.210.5.1.1" class="ltx_text ltx_font_bold" style="background-color:#E9E9E9;">GPT-4</span></th>
</tr>
<tr id="A7.T20.60.60" class="ltx_tr">
<th id="A7.T20.60.60.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T20.60.60.6.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (5-psg)</th>
<td id="A7.T20.56.56.1" class="ltx_td ltx_align_center">22.2 <sub id="A7.T20.56.56.1.1" class="ltx_sub"><span id="A7.T20.56.56.1.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T20.57.57.2" class="ltx_td ltx_align_center">25.0 <sub id="A7.T20.57.57.2.1" class="ltx_sub"><span id="A7.T20.57.57.2.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T20.58.58.3" class="ltx_td ltx_align_center">25.9 <sub id="A7.T20.58.58.3.1" class="ltx_sub"><span id="A7.T20.58.58.3.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T20.59.59.4" class="ltx_td ltx_align_center">27.0 <sub id="A7.T20.59.59.4.1" class="ltx_sub"><span id="A7.T20.59.59.4.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T20.60.60.5" class="ltx_td ltx_align_center">4.4 <sub id="A7.T20.60.60.5.1" class="ltx_sub"><span id="A7.T20.60.60.5.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
</tr>
<tr id="A7.T20.65.65" class="ltx_tr">
<th id="A7.T20.65.65.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T20.65.65.6.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (10-psg)</th>
<td id="A7.T20.61.61.1" class="ltx_td ltx_align_center">26.8 <sub id="A7.T20.61.61.1.1" class="ltx_sub"><span id="A7.T20.61.61.1.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T20.62.62.2" class="ltx_td ltx_align_center">25.1 <sub id="A7.T20.62.62.2.1" class="ltx_sub"><span id="A7.T20.62.62.2.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T20.63.63.3" class="ltx_td ltx_align_center">26.2 <sub id="A7.T20.63.63.3.1" class="ltx_sub"><span id="A7.T20.63.63.3.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T20.64.64.4" class="ltx_td ltx_align_center">27.2 <sub id="A7.T20.64.64.4.1" class="ltx_sub"><span id="A7.T20.64.64.4.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T20.65.65.5" class="ltx_td ltx_align_center">5.7 <sub id="A7.T20.65.65.5.1" class="ltx_sub"><span id="A7.T20.65.65.5.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
</tr>
<tr id="A7.T20.70.70" class="ltx_tr">
<th id="A7.T20.70.70.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T20.70.70.6.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (20-psg)</th>
<td id="A7.T20.66.66.1" class="ltx_td ltx_align_center">29.6 <sub id="A7.T20.66.66.1.1" class="ltx_sub"><span id="A7.T20.66.66.1.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T20.67.67.2" class="ltx_td ltx_align_center">26.2 <sub id="A7.T20.67.67.2.1" class="ltx_sub"><span id="A7.T20.67.67.2.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T20.68.68.3" class="ltx_td ltx_align_center">27.4 <sub id="A7.T20.68.68.3.1" class="ltx_sub"><span id="A7.T20.68.68.3.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T20.69.69.4" class="ltx_td ltx_align_center">28.5 <sub id="A7.T20.69.69.4.1" class="ltx_sub"><span id="A7.T20.69.69.4.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T20.70.70.5" class="ltx_td ltx_align_center">6.8 <sub id="A7.T20.70.70.5.1" class="ltx_sub"><span id="A7.T20.70.70.5.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
</tr>
<tr id="A7.T20.205.211.6" class="ltx_tr" style="background-color:#EBEBEB;">
<th id="A7.T20.205.211.6.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" colspan="6"><span id="A7.T20.205.211.6.1.1" class="ltx_text ltx_font_bold" style="background-color:#EBEBEB;">Open-source</span></th>
</tr>
<tr id="A7.T20.75.75" class="ltx_tr">
<th id="A7.T20.75.75.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-7B <span id="A7.T20.75.75.6.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (3-psg)</th>
<td id="A7.T20.71.71.1" class="ltx_td ltx_align_center">7.8 <sub id="A7.T20.71.71.1.1" class="ltx_sub"><span id="A7.T20.71.71.1.1.1" class="ltx_text ltx_font_italic">(3.4)</span></sub>
</td>
<td id="A7.T20.72.72.2" class="ltx_td ltx_align_center">7.4 <sub id="A7.T20.72.72.2.1" class="ltx_sub"><span id="A7.T20.72.72.2.1.1" class="ltx_text ltx_font_italic">(2.7)</span></sub>
</td>
<td id="A7.T20.73.73.3" class="ltx_td ltx_align_center">5.1 <sub id="A7.T20.73.73.3.1" class="ltx_sub"><span id="A7.T20.73.73.3.1.1" class="ltx_text ltx_font_italic">(0.5)</span></sub>
</td>
<td id="A7.T20.74.74.4" class="ltx_td ltx_align_center">5.7 <sub id="A7.T20.74.74.4.1" class="ltx_sub"><span id="A7.T20.74.74.4.1.1" class="ltx_text ltx_font_italic">(0.8)</span></sub>
</td>
<td id="A7.T20.75.75.5" class="ltx_td ltx_align_center">5.7 <sub id="A7.T20.75.75.5.1" class="ltx_sub"><span id="A7.T20.75.75.5.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
</tr>
<tr id="A7.T20.80.80" class="ltx_tr">
<th id="A7.T20.80.80.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">Alpaca-7B <span id="A7.T20.80.80.6.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (3-psg)</th>
<td id="A7.T20.76.76.1" class="ltx_td ltx_align_center">9.4 <sub id="A7.T20.76.76.1.1" class="ltx_sub"><span id="A7.T20.76.76.1.1.1" class="ltx_text ltx_font_italic">(3.7)</span></sub>
</td>
<td id="A7.T20.77.77.2" class="ltx_td ltx_align_center">9.5 <sub id="A7.T20.77.77.2.1" class="ltx_sub"><span id="A7.T20.77.77.2.1.1" class="ltx_text ltx_font_italic">(3.6)</span></sub>
</td>
<td id="A7.T20.78.78.3" class="ltx_td ltx_align_center">6.4 <sub id="A7.T20.78.78.3.1" class="ltx_sub"><span id="A7.T20.78.78.3.1.1" class="ltx_text ltx_font_italic">(0.5)</span></sub>
</td>
<td id="A7.T20.79.79.4" class="ltx_td ltx_align_center">6.8 <sub id="A7.T20.79.79.4.1" class="ltx_sub"><span id="A7.T20.79.79.4.1.1" class="ltx_text ltx_font_italic">(0.5)</span></sub>
</td>
<td id="A7.T20.80.80.5" class="ltx_td ltx_align_center">5.1 <sub id="A7.T20.80.80.5.1" class="ltx_sub"><span id="A7.T20.80.80.5.1.1" class="ltx_text ltx_font_italic">(0.1)</span></sub>
</td>
</tr>
<tr id="A7.T20.85.85" class="ltx_tr">
<th id="A7.T20.85.85.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">Vicuna-7B <span id="A7.T20.85.85.6.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (3-psg)</th>
<td id="A7.T20.81.81.1" class="ltx_td ltx_align_center">11.3 <sub id="A7.T20.81.81.1.1" class="ltx_sub"><span id="A7.T20.81.81.1.1.1" class="ltx_text ltx_font_italic">(1.4)</span></sub>
</td>
<td id="A7.T20.82.82.2" class="ltx_td ltx_align_center">13.3 <sub id="A7.T20.82.82.2.1" class="ltx_sub"><span id="A7.T20.82.82.2.1.1" class="ltx_text ltx_font_italic">(2.3)</span></sub>
</td>
<td id="A7.T20.83.83.3" class="ltx_td ltx_align_center">10.1 <sub id="A7.T20.83.83.3.1" class="ltx_sub"><span id="A7.T20.83.83.3.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
<td id="A7.T20.84.84.4" class="ltx_td ltx_align_center">10.9 <sub id="A7.T20.84.84.4.1" class="ltx_sub"><span id="A7.T20.84.84.4.1.1" class="ltx_text ltx_font_italic">(0.5)</span></sub>
</td>
<td id="A7.T20.85.85.5" class="ltx_td ltx_align_center">3.9 <sub id="A7.T20.85.85.5.1" class="ltx_sub"><span id="A7.T20.85.85.5.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
</tr>
<tr id="A7.T20.90.90" class="ltx_tr">
<th id="A7.T20.90.90.6" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">LLaMA-13B <span id="A7.T20.90.90.6.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (3-psg)</th>
<td id="A7.T20.86.86.1" class="ltx_td ltx_align_center ltx_border_t">9.7 <sub id="A7.T20.86.86.1.1" class="ltx_sub"><span id="A7.T20.86.86.1.1.1" class="ltx_text ltx_font_italic">(3.6)</span></sub>
</td>
<td id="A7.T20.87.87.2" class="ltx_td ltx_align_center ltx_border_t">9.1 <sub id="A7.T20.87.87.2.1" class="ltx_sub"><span id="A7.T20.87.87.2.1.1" class="ltx_text ltx_font_italic">(3.1)</span></sub>
</td>
<td id="A7.T20.88.88.3" class="ltx_td ltx_align_center ltx_border_t">6.7 <sub id="A7.T20.88.88.3.1" class="ltx_sub"><span id="A7.T20.88.88.3.1.1" class="ltx_text ltx_font_italic">(0.9)</span></sub>
</td>
<td id="A7.T20.89.89.4" class="ltx_td ltx_align_center ltx_border_t">7.1 <sub id="A7.T20.89.89.4.1" class="ltx_sub"><span id="A7.T20.89.89.4.1.1" class="ltx_text ltx_font_italic">(0.9)</span></sub>
</td>
<td id="A7.T20.90.90.5" class="ltx_td ltx_align_center ltx_border_t">5.9 <sub id="A7.T20.90.90.5.1" class="ltx_sub"><span id="A7.T20.90.90.5.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
</tr>
<tr id="A7.T20.95.95" class="ltx_tr">
<th id="A7.T20.95.95.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">w/ <span id="A7.T20.95.95.6.1" class="ltx_text ltx_font_smallcaps">Rerank</span>
</th>
<td id="A7.T20.91.91.1" class="ltx_td ltx_align_center">10.0 <sub id="A7.T20.91.91.1.1" class="ltx_sub"><span id="A7.T20.91.91.1.1.1" class="ltx_text ltx_font_italic">(3.3)</span></sub>
</td>
<td id="A7.T20.92.92.2" class="ltx_td ltx_align_center">10.7 <sub id="A7.T20.92.92.2.1" class="ltx_sub"><span id="A7.T20.92.92.2.1.1" class="ltx_text ltx_font_italic">(3.3)</span></sub>
</td>
<td id="A7.T20.93.93.3" class="ltx_td ltx_align_center">9.9 <sub id="A7.T20.93.93.3.1" class="ltx_sub"><span id="A7.T20.93.93.3.1.1" class="ltx_text ltx_font_italic">(1.2)</span></sub>
</td>
<td id="A7.T20.94.94.4" class="ltx_td ltx_align_center">10.2 <sub id="A7.T20.94.94.4.1" class="ltx_sub"><span id="A7.T20.94.94.4.1.1" class="ltx_text ltx_font_italic">(1.1)</span></sub>
</td>
<td id="A7.T20.95.95.5" class="ltx_td ltx_align_center">5.4 <sub id="A7.T20.95.95.5.1" class="ltx_sub"><span id="A7.T20.95.95.5.1.1" class="ltx_text ltx_font_italic">(0.5)</span></sub>
</td>
</tr>
<tr id="A7.T20.100.100" class="ltx_tr">
<th id="A7.T20.100.100.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-13B <span id="A7.T20.100.100.6.1" class="ltx_text ltx_font_smallcaps">Summ</span> (10-psg)</th>
<td id="A7.T20.96.96.1" class="ltx_td ltx_align_center">14.8 <sub id="A7.T20.96.96.1.1" class="ltx_sub"><span id="A7.T20.96.96.1.1.1" class="ltx_text ltx_font_italic">(2.5)</span></sub>
</td>
<td id="A7.T20.97.97.2" class="ltx_td ltx_align_center">12.6 <sub id="A7.T20.97.97.2.1" class="ltx_sub"><span id="A7.T20.97.97.2.1.1" class="ltx_text ltx_font_italic">(1.5)</span></sub>
</td>
<td id="A7.T20.98.98.3" class="ltx_td ltx_align_center">7.4 <sub id="A7.T20.98.98.3.1" class="ltx_sub"><span id="A7.T20.98.98.3.1.1" class="ltx_text ltx_font_italic">(0.5)</span></sub>
</td>
<td id="A7.T20.99.99.4" class="ltx_td ltx_align_center">8.0 <sub id="A7.T20.99.99.4.1" class="ltx_sub"><span id="A7.T20.99.99.4.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
<td id="A7.T20.100.100.5" class="ltx_td ltx_align_center">8.1 <sub id="A7.T20.100.100.5.1" class="ltx_sub"><span id="A7.T20.100.100.5.1.1" class="ltx_text ltx_font_italic">(0.9)</span></sub>
</td>
</tr>
<tr id="A7.T20.105.105" class="ltx_tr">
<th id="A7.T20.105.105.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-13B <span id="A7.T20.105.105.6.1" class="ltx_text ltx_font_smallcaps">Snippet</span> (10-psg)</th>
<td id="A7.T20.101.101.1" class="ltx_td ltx_align_center">17.7 <sub id="A7.T20.101.101.1.1" class="ltx_sub"><span id="A7.T20.101.101.1.1.1" class="ltx_text ltx_font_italic">(1.4)</span></sub>
</td>
<td id="A7.T20.102.102.2" class="ltx_td ltx_align_center">15.7 <sub id="A7.T20.102.102.2.1" class="ltx_sub"><span id="A7.T20.102.102.2.1.1" class="ltx_text ltx_font_italic">(0.9)</span></sub>
</td>
<td id="A7.T20.103.103.3" class="ltx_td ltx_align_center">8.8 <sub id="A7.T20.103.103.3.1" class="ltx_sub"><span id="A7.T20.103.103.3.1.1" class="ltx_text ltx_font_italic">(0.7)</span></sub>
</td>
<td id="A7.T20.104.104.4" class="ltx_td ltx_align_center">9.9 <sub id="A7.T20.104.104.4.1" class="ltx_sub"><span id="A7.T20.104.104.4.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
<td id="A7.T20.105.105.5" class="ltx_td ltx_align_center">8.2 <sub id="A7.T20.105.105.5.1" class="ltx_sub"><span id="A7.T20.105.105.5.1.1" class="ltx_text ltx_font_italic">(0.4)</span></sub>
</td>
</tr>
<tr id="A7.T20.110.110" class="ltx_tr">
<th id="A7.T20.110.110.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-13B <span id="A7.T20.110.110.6.1" class="ltx_text ltx_font_smallcaps">Oracle</span> (3-psg)</th>
<td id="A7.T20.106.106.1" class="ltx_td ltx_align_center">16.8 <sub id="A7.T20.106.106.1.1" class="ltx_sub"><span id="A7.T20.106.106.1.1.1" class="ltx_text ltx_font_italic">(6.6)</span></sub>
</td>
<td id="A7.T20.107.107.2" class="ltx_td ltx_align_center">15.4 <sub id="A7.T20.107.107.2.1" class="ltx_sub"><span id="A7.T20.107.107.2.1.1" class="ltx_text ltx_font_italic">(5.6)</span></sub>
</td>
<td id="A7.T20.108.108.3" class="ltx_td ltx_align_center">7.7 <sub id="A7.T20.108.108.3.1" class="ltx_sub"><span id="A7.T20.108.108.3.1.1" class="ltx_text ltx_font_italic">(1.0)</span></sub>
</td>
<td id="A7.T20.109.109.4" class="ltx_td ltx_align_center">8.3 <sub id="A7.T20.109.109.4.1" class="ltx_sub"><span id="A7.T20.109.109.4.1.1" class="ltx_text ltx_font_italic">(1.1)</span></sub>
</td>
<td id="A7.T20.110.110.5" class="ltx_td ltx_align_center">5.7 <sub id="A7.T20.110.110.5.1" class="ltx_sub"><span id="A7.T20.110.110.5.1.1" class="ltx_text ltx_font_italic">(0.7)</span></sub>
</td>
</tr>
<tr id="A7.T20.115.115" class="ltx_tr">
<th id="A7.T20.115.115.6" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Vicuna-13B <span id="A7.T20.115.115.6.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (3-psg)</th>
<td id="A7.T20.111.111.1" class="ltx_td ltx_align_center ltx_border_t">14.0 <sub id="A7.T20.111.111.1.1" class="ltx_sub"><span id="A7.T20.111.111.1.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
<td id="A7.T20.112.112.2" class="ltx_td ltx_align_center ltx_border_t">15.9 <sub id="A7.T20.112.112.2.1" class="ltx_sub"><span id="A7.T20.112.112.2.1.1" class="ltx_text ltx_font_italic">(1.7)</span></sub>
</td>
<td id="A7.T20.113.113.3" class="ltx_td ltx_align_center ltx_border_t">12.5 <sub id="A7.T20.113.113.3.1" class="ltx_sub"><span id="A7.T20.113.113.3.1.1" class="ltx_text ltx_font_italic">(0.8)</span></sub>
</td>
<td id="A7.T20.114.114.4" class="ltx_td ltx_align_center ltx_border_t">13.4 <sub id="A7.T20.114.114.4.1" class="ltx_sub"><span id="A7.T20.114.114.4.1.1" class="ltx_text ltx_font_italic">(0.7)</span></sub>
</td>
<td id="A7.T20.115.115.5" class="ltx_td ltx_align_center ltx_border_t">4.7 <sub id="A7.T20.115.115.5.1" class="ltx_sub"><span id="A7.T20.115.115.5.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
</tr>
<tr id="A7.T20.120.120" class="ltx_tr">
<th id="A7.T20.120.120.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">w/ <span id="A7.T20.120.120.6.1" class="ltx_text ltx_font_smallcaps">Rerank</span>
</th>
<td id="A7.T20.116.116.1" class="ltx_td ltx_align_center">13.0 <sub id="A7.T20.116.116.1.1" class="ltx_sub"><span id="A7.T20.116.116.1.1.1" class="ltx_text ltx_font_italic">(0.7)</span></sub>
</td>
<td id="A7.T20.117.117.2" class="ltx_td ltx_align_center">17.2 <sub id="A7.T20.117.117.2.1" class="ltx_sub"><span id="A7.T20.117.117.2.1.1" class="ltx_text ltx_font_italic">(2.2)</span></sub>
</td>
<td id="A7.T20.118.118.3" class="ltx_td ltx_align_center">17.3 <sub id="A7.T20.118.118.3.1" class="ltx_sub"><span id="A7.T20.118.118.3.1.1" class="ltx_text ltx_font_italic">(0.8)</span></sub>
</td>
<td id="A7.T20.119.119.4" class="ltx_td ltx_align_center">17.7 <sub id="A7.T20.119.119.4.1" class="ltx_sub"><span id="A7.T20.119.119.4.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
<td id="A7.T20.120.120.5" class="ltx_td ltx_align_center">4.4 <sub id="A7.T20.120.120.5.1" class="ltx_sub"><span id="A7.T20.120.120.5.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
</tr>
<tr id="A7.T20.125.125" class="ltx_tr">
<th id="A7.T20.125.125.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">Vicuna-13B <span id="A7.T20.125.125.6.1" class="ltx_text ltx_font_smallcaps">Summ</span> (10-psg)</th>
<td id="A7.T20.121.121.1" class="ltx_td ltx_align_center">21.1 <sub id="A7.T20.121.121.1.1" class="ltx_sub"><span id="A7.T20.121.121.1.1.1" class="ltx_text ltx_font_italic">(1.4)</span></sub>
</td>
<td id="A7.T20.122.122.2" class="ltx_td ltx_align_center">17.1 <sub id="A7.T20.122.122.2.1" class="ltx_sub"><span id="A7.T20.122.122.2.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
<td id="A7.T20.123.123.3" class="ltx_td ltx_align_center">15.7 <sub id="A7.T20.123.123.3.1" class="ltx_sub"><span id="A7.T20.123.123.3.1.1" class="ltx_text ltx_font_italic">(0.2)</span></sub>
</td>
<td id="A7.T20.124.124.4" class="ltx_td ltx_align_center">17.8 <sub id="A7.T20.124.124.4.1" class="ltx_sub"><span id="A7.T20.124.124.4.1.1" class="ltx_text ltx_font_italic">(0.1)</span></sub>
</td>
<td id="A7.T20.125.125.5" class="ltx_td ltx_align_center">6.9 <sub id="A7.T20.125.125.5.1" class="ltx_sub"><span id="A7.T20.125.125.5.1.1" class="ltx_text ltx_font_italic">(0.7)</span></sub>
</td>
</tr>
<tr id="A7.T20.130.130" class="ltx_tr">
<th id="A7.T20.130.130.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">Vicuna-13B <span id="A7.T20.130.130.6.1" class="ltx_text ltx_font_smallcaps">Snippet</span> (10-psg)</th>
<td id="A7.T20.126.126.1" class="ltx_td ltx_align_center">21.9 <sub id="A7.T20.126.126.1.1" class="ltx_sub"><span id="A7.T20.126.126.1.1.1" class="ltx_text ltx_font_italic">(0.8)</span></sub>
</td>
<td id="A7.T20.127.127.2" class="ltx_td ltx_align_center">18.2 <sub id="A7.T20.127.127.2.1" class="ltx_sub"><span id="A7.T20.127.127.2.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
<td id="A7.T20.128.128.3" class="ltx_td ltx_align_center">16.8 <sub id="A7.T20.128.128.3.1" class="ltx_sub"><span id="A7.T20.128.128.3.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
<td id="A7.T20.129.129.4" class="ltx_td ltx_align_center">19.7 <sub id="A7.T20.129.129.4.1" class="ltx_sub"><span id="A7.T20.129.129.4.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
<td id="A7.T20.130.130.5" class="ltx_td ltx_align_center">7.5 <sub id="A7.T20.130.130.5.1" class="ltx_sub"><span id="A7.T20.130.130.5.1.1" class="ltx_text ltx_font_italic">(0.4)</span></sub>
</td>
</tr>
<tr id="A7.T20.135.135" class="ltx_tr">
<th id="A7.T20.135.135.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">Vicuna-13B <span id="A7.T20.135.135.6.1" class="ltx_text ltx_font_smallcaps">Oracle</span> (3-psg)</th>
<td id="A7.T20.131.131.1" class="ltx_td ltx_align_center">25.9 <sub id="A7.T20.131.131.1.1" class="ltx_sub"><span id="A7.T20.131.131.1.1.1" class="ltx_text ltx_font_italic">(1.6)</span></sub>
</td>
<td id="A7.T20.132.132.2" class="ltx_td ltx_align_center">28.4 <sub id="A7.T20.132.132.2.1" class="ltx_sub"><span id="A7.T20.132.132.2.1.1" class="ltx_text ltx_font_italic">(2.6)</span></sub>
</td>
<td id="A7.T20.133.133.3" class="ltx_td ltx_align_center">15.8 <sub id="A7.T20.133.133.3.1" class="ltx_sub"><span id="A7.T20.133.133.3.1.1" class="ltx_text ltx_font_italic">(1.4)</span></sub>
</td>
<td id="A7.T20.134.134.4" class="ltx_td ltx_align_center">16.8 <sub id="A7.T20.134.134.4.1" class="ltx_sub"><span id="A7.T20.134.134.4.1.1" class="ltx_text ltx_font_italic">(1.4)</span></sub>
</td>
<td id="A7.T20.135.135.5" class="ltx_td ltx_align_center">4.9 <sub id="A7.T20.135.135.5.1" class="ltx_sub"><span id="A7.T20.135.135.5.1.1" class="ltx_text ltx_font_italic">(0.5)</span></sub>
</td>
</tr>
<tr id="A7.T20.140.140" class="ltx_tr">
<th id="A7.T20.140.140.6" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">LLaMA-33B <span id="A7.T20.140.140.6.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (3-psg)</th>
<td id="A7.T20.136.136.1" class="ltx_td ltx_align_center ltx_border_t">14.7 <sub id="A7.T20.136.136.1.1" class="ltx_sub"><span id="A7.T20.136.136.1.1.1" class="ltx_text ltx_font_italic">(3.3)</span></sub>
</td>
<td id="A7.T20.137.137.2" class="ltx_td ltx_align_center ltx_border_t">12.0 <sub id="A7.T20.137.137.2.1" class="ltx_sub"><span id="A7.T20.137.137.2.1.1" class="ltx_text ltx_font_italic">(2.2)</span></sub>
</td>
<td id="A7.T20.138.138.3" class="ltx_td ltx_align_center ltx_border_t">7.9 <sub id="A7.T20.138.138.3.1" class="ltx_sub"><span id="A7.T20.138.138.3.1.1" class="ltx_text ltx_font_italic">(0.7)</span></sub>
</td>
<td id="A7.T20.139.139.4" class="ltx_td ltx_align_center ltx_border_t">8.3 <sub id="A7.T20.139.139.4.1" class="ltx_sub"><span id="A7.T20.139.139.4.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
<td id="A7.T20.140.140.5" class="ltx_td ltx_align_center ltx_border_t">7.2 <sub id="A7.T20.140.140.5.1" class="ltx_sub"><span id="A7.T20.140.140.5.1.1" class="ltx_text ltx_font_italic">(0.7)</span></sub>
</td>
</tr>
<tr id="A7.T20.145.145" class="ltx_tr">
<th id="A7.T20.145.145.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">w/ <span id="A7.T20.145.145.6.1" class="ltx_text ltx_font_smallcaps">Rerank</span>
</th>
<td id="A7.T20.141.141.1" class="ltx_td ltx_align_center">14.0 <sub id="A7.T20.141.141.1.1" class="ltx_sub"><span id="A7.T20.141.141.1.1.1" class="ltx_text ltx_font_italic">(3.4)</span></sub>
</td>
<td id="A7.T20.142.142.2" class="ltx_td ltx_align_center">13.9 <sub id="A7.T20.142.142.2.1" class="ltx_sub"><span id="A7.T20.142.142.2.1.1" class="ltx_text ltx_font_italic">(2.6)</span></sub>
</td>
<td id="A7.T20.143.143.3" class="ltx_td ltx_align_center">10.7 <sub id="A7.T20.143.143.3.1" class="ltx_sub"><span id="A7.T20.143.143.3.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
<td id="A7.T20.144.144.4" class="ltx_td ltx_align_center">11.1 <sub id="A7.T20.144.144.4.1" class="ltx_sub"><span id="A7.T20.144.144.4.1.1" class="ltx_text ltx_font_italic">(0.5)</span></sub>
</td>
<td id="A7.T20.145.145.5" class="ltx_td ltx_align_center">6.4 <sub id="A7.T20.145.145.5.1" class="ltx_sub"><span id="A7.T20.145.145.5.1.1" class="ltx_text ltx_font_italic">(0.7)</span></sub>
</td>
</tr>
<tr id="A7.T20.150.150" class="ltx_tr">
<th id="A7.T20.150.150.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-33B <span id="A7.T20.150.150.6.1" class="ltx_text ltx_font_smallcaps">Summ</span> (10-psg)</th>
<td id="A7.T20.146.146.1" class="ltx_td ltx_align_center">19.0 <sub id="A7.T20.146.146.1.1" class="ltx_sub"><span id="A7.T20.146.146.1.1.1" class="ltx_text ltx_font_italic">(1.9)</span></sub>
</td>
<td id="A7.T20.147.147.2" class="ltx_td ltx_align_center">14.8 <sub id="A7.T20.147.147.2.1" class="ltx_sub"><span id="A7.T20.147.147.2.1.1" class="ltx_text ltx_font_italic">(0.8)</span></sub>
</td>
<td id="A7.T20.148.148.3" class="ltx_td ltx_align_center">12.5 <sub id="A7.T20.148.148.3.1" class="ltx_sub"><span id="A7.T20.148.148.3.1.1" class="ltx_text ltx_font_italic">(0.2)</span></sub>
</td>
<td id="A7.T20.149.149.4" class="ltx_td ltx_align_center">15.0 <sub id="A7.T20.149.149.4.1" class="ltx_sub"><span id="A7.T20.149.149.4.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
<td id="A7.T20.150.150.5" class="ltx_td ltx_align_center">7.6 <sub id="A7.T20.150.150.5.1" class="ltx_sub"><span id="A7.T20.150.150.5.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
</tr>
<tr id="A7.T20.155.155" class="ltx_tr">
<th id="A7.T20.155.155.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-33B <span id="A7.T20.155.155.6.1" class="ltx_text ltx_font_smallcaps">Snippet</span> (10-psg)</th>
<td id="A7.T20.151.151.1" class="ltx_td ltx_align_center">19.6 <sub id="A7.T20.151.151.1.1" class="ltx_sub"><span id="A7.T20.151.151.1.1.1" class="ltx_text ltx_font_italic">(1.1)</span></sub>
</td>
<td id="A7.T20.152.152.2" class="ltx_td ltx_align_center">15.7 <sub id="A7.T20.152.152.2.1" class="ltx_sub"><span id="A7.T20.152.152.2.1.1" class="ltx_text ltx_font_italic">(0.1)</span></sub>
</td>
<td id="A7.T20.153.153.3" class="ltx_td ltx_align_center">12.8 <sub id="A7.T20.153.153.3.1" class="ltx_sub"><span id="A7.T20.153.153.3.1.1" class="ltx_text ltx_font_italic">(1.1)</span></sub>
</td>
<td id="A7.T20.154.154.4" class="ltx_td ltx_align_center">15.2 <sub id="A7.T20.154.154.4.1" class="ltx_sub"><span id="A7.T20.154.154.4.1.1" class="ltx_text ltx_font_italic">(1.2)</span></sub>
</td>
<td id="A7.T20.155.155.5" class="ltx_td ltx_align_center">7.8 <sub id="A7.T20.155.155.5.1" class="ltx_sub"><span id="A7.T20.155.155.5.1.1" class="ltx_text ltx_font_italic">(0.5)</span></sub>
</td>
</tr>
<tr id="A7.T20.160.160" class="ltx_tr">
<th id="A7.T20.160.160.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-33B <span id="A7.T20.160.160.6.1" class="ltx_text ltx_font_smallcaps">Oracle</span> (3-psg)</th>
<td id="A7.T20.156.156.1" class="ltx_td ltx_align_center">23.9 <sub id="A7.T20.156.156.1.1" class="ltx_sub"><span id="A7.T20.156.156.1.1.1" class="ltx_text ltx_font_italic">(6.9)</span></sub>
</td>
<td id="A7.T20.157.157.2" class="ltx_td ltx_align_center">20.3 <sub id="A7.T20.157.157.2.1" class="ltx_sub"><span id="A7.T20.157.157.2.1.1" class="ltx_text ltx_font_italic">(5.2)</span></sub>
</td>
<td id="A7.T20.158.158.3" class="ltx_td ltx_align_center">9.8 <sub id="A7.T20.158.158.3.1" class="ltx_sub"><span id="A7.T20.158.158.3.1.1" class="ltx_text ltx_font_italic">(1.2)</span></sub>
</td>
<td id="A7.T20.159.159.4" class="ltx_td ltx_align_center">10.4 <sub id="A7.T20.159.159.4.1" class="ltx_sub"><span id="A7.T20.159.159.4.1.1" class="ltx_text ltx_font_italic">(1.2)</span></sub>
</td>
<td id="A7.T20.160.160.5" class="ltx_td ltx_align_center">6.8 <sub id="A7.T20.160.160.5.1" class="ltx_sub"><span id="A7.T20.160.160.5.1.1" class="ltx_text ltx_font_italic">(0.9)</span></sub>
</td>
</tr>
<tr id="A7.T20.165.165" class="ltx_tr">
<th id="A7.T20.165.165.6" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Oasst-33B <span id="A7.T20.165.165.6.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (3-psg)</th>
<td id="A7.T20.161.161.1" class="ltx_td ltx_align_center ltx_border_t">15.5 <sub id="A7.T20.161.161.1.1" class="ltx_sub"><span id="A7.T20.161.161.1.1.1" class="ltx_text ltx_font_italic">(1.5)</span></sub>
</td>
<td id="A7.T20.162.162.2" class="ltx_td ltx_align_center ltx_border_t">14.9 <sub id="A7.T20.162.162.2.1" class="ltx_sub"><span id="A7.T20.162.162.2.1.1" class="ltx_text ltx_font_italic">(1.4)</span></sub>
</td>
<td id="A7.T20.163.163.3" class="ltx_td ltx_align_center ltx_border_t">9.0 <sub id="A7.T20.163.163.3.1" class="ltx_sub"><span id="A7.T20.163.163.3.1.1" class="ltx_text ltx_font_italic">(1.6)</span></sub>
</td>
<td id="A7.T20.164.164.4" class="ltx_td ltx_align_center ltx_border_t">10.1 <sub id="A7.T20.164.164.4.1" class="ltx_sub"><span id="A7.T20.164.164.4.1.1" class="ltx_text ltx_font_italic">(1.8)</span></sub>
</td>
<td id="A7.T20.165.165.5" class="ltx_td ltx_align_center ltx_border_t">5.6 <sub id="A7.T20.165.165.5.1" class="ltx_sub"><span id="A7.T20.165.165.5.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
</tr>
<tr id="A7.T20.170.170" class="ltx_tr">
<th id="A7.T20.170.170.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">w/ <span id="A7.T20.170.170.6.1" class="ltx_text ltx_font_smallcaps">Rerank</span>
</th>
<td id="A7.T20.166.166.1" class="ltx_td ltx_align_center">14.1 <sub id="A7.T20.166.166.1.1" class="ltx_sub"><span id="A7.T20.166.166.1.1.1" class="ltx_text ltx_font_italic">(1.1)</span></sub>
</td>
<td id="A7.T20.167.167.2" class="ltx_td ltx_align_center">15.8 <sub id="A7.T20.167.167.2.1" class="ltx_sub"><span id="A7.T20.167.167.2.1.1" class="ltx_text ltx_font_italic">(1.0)</span></sub>
</td>
<td id="A7.T20.168.168.3" class="ltx_td ltx_align_center">15.0 <sub id="A7.T20.168.168.3.1" class="ltx_sub"><span id="A7.T20.168.168.3.1.1" class="ltx_text ltx_font_italic">(1.6)</span></sub>
</td>
<td id="A7.T20.169.169.4" class="ltx_td ltx_align_center">15.9 <sub id="A7.T20.169.169.4.1" class="ltx_sub"><span id="A7.T20.169.169.4.1.1" class="ltx_text ltx_font_italic">(1.6)</span></sub>
</td>
<td id="A7.T20.170.170.5" class="ltx_td ltx_align_center">4.7 <sub id="A7.T20.170.170.5.1" class="ltx_sub"><span id="A7.T20.170.170.5.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
</tr>
<tr id="A7.T20.175.175" class="ltx_tr">
<th id="A7.T20.175.175.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">Oasst-33B <span id="A7.T20.175.175.6.1" class="ltx_text ltx_font_smallcaps">Summ</span> (10-psg)</th>
<td id="A7.T20.171.171.1" class="ltx_td ltx_align_center">21.0 <sub id="A7.T20.171.171.1.1" class="ltx_sub"><span id="A7.T20.171.171.1.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
<td id="A7.T20.172.172.2" class="ltx_td ltx_align_center">17.5 <sub id="A7.T20.172.172.2.1" class="ltx_sub"><span id="A7.T20.172.172.2.1.1" class="ltx_text ltx_font_italic">(1.0)</span></sub>
</td>
<td id="A7.T20.173.173.3" class="ltx_td ltx_align_center">12.9 <sub id="A7.T20.173.173.3.1" class="ltx_sub"><span id="A7.T20.173.173.3.1.1" class="ltx_text ltx_font_italic">(1.2)</span></sub>
</td>
<td id="A7.T20.174.174.4" class="ltx_td ltx_align_center">16.6 <sub id="A7.T20.174.174.4.1" class="ltx_sub"><span id="A7.T20.174.174.4.1.1" class="ltx_text ltx_font_italic">(1.2)</span></sub>
</td>
<td id="A7.T20.175.175.5" class="ltx_td ltx_align_center">7.1 <sub id="A7.T20.175.175.5.1" class="ltx_sub"><span id="A7.T20.175.175.5.1.1" class="ltx_text ltx_font_italic">(0.4)</span></sub>
</td>
</tr>
<tr id="A7.T20.180.180" class="ltx_tr">
<th id="A7.T20.180.180.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">Oasst-33B <span id="A7.T20.180.180.6.1" class="ltx_text ltx_font_smallcaps">Snippet</span> (10-psg)</th>
<td id="A7.T20.176.176.1" class="ltx_td ltx_align_center">22.0 <sub id="A7.T20.176.176.1.1" class="ltx_sub"><span id="A7.T20.176.176.1.1.1" class="ltx_text ltx_font_italic">(0.4)</span></sub>
</td>
<td id="A7.T20.177.177.2" class="ltx_td ltx_align_center">17.4 <sub id="A7.T20.177.177.2.1" class="ltx_sub"><span id="A7.T20.177.177.2.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
<td id="A7.T20.178.178.3" class="ltx_td ltx_align_center">13.6 <sub id="A7.T20.178.178.3.1" class="ltx_sub"><span id="A7.T20.178.178.3.1.1" class="ltx_text ltx_font_italic">(1.7)</span></sub>
</td>
<td id="A7.T20.179.179.4" class="ltx_td ltx_align_center">17.7 <sub id="A7.T20.179.179.4.1" class="ltx_sub"><span id="A7.T20.179.179.4.1.1" class="ltx_text ltx_font_italic">(1.6)</span></sub>
</td>
<td id="A7.T20.180.180.5" class="ltx_td ltx_align_center">7.5 <sub id="A7.T20.180.180.5.1" class="ltx_sub"><span id="A7.T20.180.180.5.1.1" class="ltx_text ltx_font_italic">(0.1)</span></sub>
</td>
</tr>
<tr id="A7.T20.185.185" class="ltx_tr">
<th id="A7.T20.185.185.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">Oasst-33B <span id="A7.T20.185.185.6.1" class="ltx_text ltx_font_smallcaps">Oracle</span> (3-psg)</th>
<td id="A7.T20.181.181.1" class="ltx_td ltx_align_center">26.9 <sub id="A7.T20.181.181.1.1" class="ltx_sub"><span id="A7.T20.181.181.1.1.1" class="ltx_text ltx_font_italic">(3.7)</span></sub>
</td>
<td id="A7.T20.182.182.2" class="ltx_td ltx_align_center">26.0 <sub id="A7.T20.182.182.2.1" class="ltx_sub"><span id="A7.T20.182.182.2.1.1" class="ltx_text ltx_font_italic">(3.3)</span></sub>
</td>
<td id="A7.T20.183.183.3" class="ltx_td ltx_align_center">11.7 <sub id="A7.T20.183.183.3.1" class="ltx_sub"><span id="A7.T20.183.183.3.1.1" class="ltx_text ltx_font_italic">(1.0)</span></sub>
</td>
<td id="A7.T20.184.184.4" class="ltx_td ltx_align_center">12.9 <sub id="A7.T20.184.184.4.1" class="ltx_sub"><span id="A7.T20.184.184.4.1.1" class="ltx_text ltx_font_italic">(1.2)</span></sub>
</td>
<td id="A7.T20.185.185.5" class="ltx_td ltx_align_center">5.6 <sub id="A7.T20.185.185.5.1" class="ltx_sub"><span id="A7.T20.185.185.5.1.1" class="ltx_text ltx_font_italic">(0.4)</span></sub>
</td>
</tr>
<tr id="A7.T20.190.190" class="ltx_tr">
<th id="A7.T20.190.190.6" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">LLaMA-2-7B-Chat <span id="A7.T20.190.190.6.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (5-psg)</th>
<td id="A7.T20.186.186.1" class="ltx_td ltx_align_center ltx_border_t">16.2 <sub id="A7.T20.186.186.1.1" class="ltx_sub"><span id="A7.T20.186.186.1.1.1" class="ltx_text ltx_font_italic">(1.3)</span></sub>
</td>
<td id="A7.T20.187.187.2" class="ltx_td ltx_align_center ltx_border_t">15.3 <sub id="A7.T20.187.187.2.1" class="ltx_sub"><span id="A7.T20.187.187.2.1.1" class="ltx_text ltx_font_italic">(1.6)</span></sub>
</td>
<td id="A7.T20.188.188.3" class="ltx_td ltx_align_center ltx_border_t">10.6 <sub id="A7.T20.188.188.3.1" class="ltx_sub"><span id="A7.T20.188.188.3.1.1" class="ltx_text ltx_font_italic">(0.9)</span></sub>
</td>
<td id="A7.T20.189.189.4" class="ltx_td ltx_align_center ltx_border_t">10.9 <sub id="A7.T20.189.189.4.1" class="ltx_sub"><span id="A7.T20.189.189.4.1.1" class="ltx_text ltx_font_italic">(1.0)</span></sub>
</td>
<td id="A7.T20.190.190.5" class="ltx_td ltx_align_center ltx_border_t">5.5 <sub id="A7.T20.190.190.5.1" class="ltx_sub"><span id="A7.T20.190.190.5.1.1" class="ltx_text ltx_font_italic">(0.0)</span></sub>
</td>
</tr>
<tr id="A7.T20.195.195" class="ltx_tr">
<th id="A7.T20.195.195.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-2-13B-Chat <span id="A7.T20.195.195.6.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (5-psg)</th>
<td id="A7.T20.191.191.1" class="ltx_td ltx_align_center">21.1 <sub id="A7.T20.191.191.1.1" class="ltx_sub"><span id="A7.T20.191.191.1.1.1" class="ltx_text ltx_font_italic">(0.9)</span></sub>
</td>
<td id="A7.T20.192.192.2" class="ltx_td ltx_align_center">18.2 <sub id="A7.T20.192.192.2.1" class="ltx_sub"><span id="A7.T20.192.192.2.1.1" class="ltx_text ltx_font_italic">(0.5)</span></sub>
</td>
<td id="A7.T20.193.193.3" class="ltx_td ltx_align_center">9.6 <sub id="A7.T20.193.193.3.1" class="ltx_sub"><span id="A7.T20.193.193.3.1.1" class="ltx_text ltx_font_italic">(1.5)</span></sub>
</td>
<td id="A7.T20.194.194.4" class="ltx_td ltx_align_center">9.7 <sub id="A7.T20.194.194.4.1" class="ltx_sub"><span id="A7.T20.194.194.4.1.1" class="ltx_text ltx_font_italic">(1.5)</span></sub>
</td>
<td id="A7.T20.195.195.5" class="ltx_td ltx_align_center">6.5 <sub id="A7.T20.195.195.5.1" class="ltx_sub"><span id="A7.T20.195.195.5.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
</tr>
<tr id="A7.T20.200.200" class="ltx_tr">
<th id="A7.T20.200.200.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-2-70B-Chat <span id="A7.T20.200.200.6.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (5-psg)</th>
<td id="A7.T20.196.196.1" class="ltx_td ltx_align_center">21.8 <sub id="A7.T20.196.196.1.1" class="ltx_sub"><span id="A7.T20.196.196.1.1.1" class="ltx_text ltx_font_italic">(0.7)</span></sub>
</td>
<td id="A7.T20.197.197.2" class="ltx_td ltx_align_center">18.4 <sub id="A7.T20.197.197.2.1" class="ltx_sub"><span id="A7.T20.197.197.2.1.1" class="ltx_text ltx_font_italic">(0.1)</span></sub>
</td>
<td id="A7.T20.198.198.3" class="ltx_td ltx_align_center">15.1 <sub id="A7.T20.198.198.3.1" class="ltx_sub"><span id="A7.T20.198.198.3.1.1" class="ltx_text ltx_font_italic">(1.2)</span></sub>
</td>
<td id="A7.T20.199.199.4" class="ltx_td ltx_align_center">15.6 <sub id="A7.T20.199.199.4.1" class="ltx_sub"><span id="A7.T20.199.199.4.1.1" class="ltx_text ltx_font_italic">(1.3)</span></sub>
</td>
<td id="A7.T20.200.200.5" class="ltx_td ltx_align_center">7.1 <sub id="A7.T20.200.200.5.1" class="ltx_sub"><span id="A7.T20.200.200.5.1.1" class="ltx_text ltx_font_italic">(0.2)</span></sub>
</td>
</tr>
<tr id="A7.T20.205.205" class="ltx_tr">
<th id="A7.T20.205.205.6" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_t">Stable Beluga 2</th>
<td id="A7.T20.201.201.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">19.7 <sub id="A7.T20.201.201.1.1" class="ltx_sub"><span id="A7.T20.201.201.1.1.1" class="ltx_text ltx_font_italic">(0.7)</span></sub>
</td>
<td id="A7.T20.202.202.2" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">21.2 <sub id="A7.T20.202.202.2.1" class="ltx_sub"><span id="A7.T20.202.202.2.1.1" class="ltx_text ltx_font_italic">(1.4)</span></sub>
</td>
<td id="A7.T20.203.203.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">18.6 <sub id="A7.T20.203.203.3.1" class="ltx_sub"><span id="A7.T20.203.203.3.1.1" class="ltx_text ltx_font_italic">(1.2)</span></sub>
</td>
<td id="A7.T20.204.204.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">20.7 <sub id="A7.T20.204.204.4.1" class="ltx_sub"><span id="A7.T20.204.204.4.1.1" class="ltx_text ltx_font_italic">(1.1)</span></sub>
</td>
<td id="A7.T20.205.205.5" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">5.1 <sub id="A7.T20.205.205.5.1" class="ltx_sub"><span id="A7.T20.205.205.5.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 20: </span>
QAMPARI full results.
</figcaption>
</figure>
<figure id="A7.T21" class="ltx_table">
<table id="A7.T21.246" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A7.T21.246.247.1" class="ltx_tr">
<th id="A7.T21.246.247.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<td id="A7.T21.246.247.1.2" class="ltx_td ltx_align_center ltx_border_tt"><span id="A7.T21.246.247.1.2.1" class="ltx_text ltx_font_bold">Fluency</span></td>
<td id="A7.T21.246.247.1.3" class="ltx_td ltx_align_center ltx_border_tt"><span id="A7.T21.246.247.1.3.1" class="ltx_text ltx_font_bold">Correct.</span></td>
<td id="A7.T21.246.247.1.4" class="ltx_td ltx_align_center ltx_border_tt" colspan="2"><span id="A7.T21.246.247.1.4.1" class="ltx_text ltx_font_bold">Citation</span></td>
<td id="A7.T21.246.247.1.5" class="ltx_td ltx_border_tt"></td>
<td id="A7.T21.246.247.1.6" class="ltx_td ltx_border_tt"></td>
</tr>
<tr id="A7.T21.246.248.2" class="ltx_tr">
<th id="A7.T21.246.248.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="A7.T21.246.248.2.2" class="ltx_td ltx_align_center ltx_border_t">(MAUVE)</td>
<td id="A7.T21.246.248.2.3" class="ltx_td ltx_align_center ltx_border_t">(Claim)</td>
<td id="A7.T21.246.248.2.4" class="ltx_td ltx_align_center ltx_border_t">Rec.</td>
<td id="A7.T21.246.248.2.5" class="ltx_td ltx_align_center ltx_border_t">Prec.</td>
<td id="A7.T21.246.248.2.6" class="ltx_td ltx_align_center">ROUGE-L</td>
<td id="A7.T21.246.248.2.7" class="ltx_td ltx_align_center">Length</td>
</tr>
<tr id="A7.T21.246.249.3" class="ltx_tr" style="background-color:#E9E9E9;">
<th id="A7.T21.246.249.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" colspan="7"><span id="A7.T21.246.249.3.1.1" class="ltx_text ltx_font_bold" style="background-color:#E9E9E9;">ChatGPT</span></th>
</tr>
<tr id="A7.T21.6.6" class="ltx_tr">
<th id="A7.T21.6.6.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T21.6.6.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (5-psg)</th>
<td id="A7.T21.1.1.1" class="ltx_td ltx_align_center">57.2 <sub id="A7.T21.1.1.1.1" class="ltx_sub"><span id="A7.T21.1.1.1.1.1" class="ltx_text ltx_font_italic">(1.6)</span></sub>
</td>
<td id="A7.T21.2.2.2" class="ltx_td ltx_align_center">12.0 <sub id="A7.T21.2.2.2.1" class="ltx_sub"><span id="A7.T21.2.2.2.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
<td id="A7.T21.3.3.3" class="ltx_td ltx_align_center">51.1 <sub id="A7.T21.3.3.3.1" class="ltx_sub"><span id="A7.T21.3.3.3.1.1" class="ltx_text ltx_font_italic">(4.2)</span></sub>
</td>
<td id="A7.T21.4.4.4" class="ltx_td ltx_align_center">50.0 <sub id="A7.T21.4.4.4.1" class="ltx_sub"><span id="A7.T21.4.4.4.1.1" class="ltx_text ltx_font_italic">(4.8)</span></sub>
</td>
<td id="A7.T21.5.5.5" class="ltx_td ltx_align_center">20.6 <sub id="A7.T21.5.5.5.1" class="ltx_sub"><span id="A7.T21.5.5.5.1.1" class="ltx_text ltx_font_italic">(0.2)</span></sub>
</td>
<td id="A7.T21.6.6.6" class="ltx_td ltx_align_center">91.5 <sub id="A7.T21.6.6.6.1" class="ltx_sub"><span id="A7.T21.6.6.6.1.1" class="ltx_text ltx_font_italic">(6.5)</span></sub>
</td>
</tr>
<tr id="A7.T21.12.12" class="ltx_tr">
<th id="A7.T21.12.12.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">w/ <span id="A7.T21.12.12.7.1" class="ltx_text ltx_font_smallcaps">Rerank</span>
</th>
<td id="A7.T21.7.7.1" class="ltx_td ltx_align_center">56.1 <sub id="A7.T21.7.7.1.1" class="ltx_sub"><span id="A7.T21.7.7.1.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.8.8.2" class="ltx_td ltx_align_center">11.4 <sub id="A7.T21.8.8.2.1" class="ltx_sub"><span id="A7.T21.8.8.2.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.9.9.3" class="ltx_td ltx_align_center">69.3 <sub id="A7.T21.9.9.3.1" class="ltx_sub"><span id="A7.T21.9.9.3.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.10.10.4" class="ltx_td ltx_align_center">67.8 <sub id="A7.T21.10.10.4.1" class="ltx_sub"><span id="A7.T21.10.10.4.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.11.11.5" class="ltx_td ltx_align_center">20.3 <sub id="A7.T21.11.11.5.1" class="ltx_sub"><span id="A7.T21.11.11.5.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.12.12.6" class="ltx_td ltx_align_center">103.4 <sub id="A7.T21.12.12.6.1" class="ltx_sub"><span id="A7.T21.12.12.6.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
</tr>
<tr id="A7.T21.18.18" class="ltx_tr">
<th id="A7.T21.18.18.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T21.18.18.7.1" class="ltx_text ltx_font_smallcaps">Summ</span> (10-psg)</th>
<td id="A7.T21.13.13.1" class="ltx_td ltx_align_center">40.2 <sub id="A7.T21.13.13.1.1" class="ltx_sub"><span id="A7.T21.13.13.1.1.1" class="ltx_text ltx_font_italic">(1.2)</span></sub>
</td>
<td id="A7.T21.14.14.2" class="ltx_td ltx_align_center">12.5 <sub id="A7.T21.14.14.2.1" class="ltx_sub"><span id="A7.T21.14.14.2.1.1" class="ltx_text ltx_font_italic">(0.2)</span></sub>
</td>
<td id="A7.T21.15.15.3" class="ltx_td ltx_align_center">51.5 <sub id="A7.T21.15.15.3.1" class="ltx_sub"><span id="A7.T21.15.15.3.1.1" class="ltx_text ltx_font_italic">(1.1)</span></sub>
</td>
<td id="A7.T21.16.16.4" class="ltx_td ltx_align_center">48.2 <sub id="A7.T21.16.16.4.1" class="ltx_sub"><span id="A7.T21.16.16.4.1.1" class="ltx_text ltx_font_italic">(2.0)</span></sub>
</td>
<td id="A7.T21.17.17.5" class="ltx_td ltx_align_center">20.3 <sub id="A7.T21.17.17.5.1" class="ltx_sub"><span id="A7.T21.17.17.5.1.1" class="ltx_text ltx_font_italic">(0.1)</span></sub>
</td>
<td id="A7.T21.18.18.6" class="ltx_td ltx_align_center">90.0 <sub id="A7.T21.18.18.6.1" class="ltx_sub"><span id="A7.T21.18.18.6.1.1" class="ltx_text ltx_font_italic">(6.6)</span></sub>
</td>
</tr>
<tr id="A7.T21.24.24" class="ltx_tr">
<th id="A7.T21.24.24.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T21.24.24.7.1" class="ltx_text ltx_font_smallcaps">Snippet</span> (10-psg)</th>
<td id="A7.T21.19.19.1" class="ltx_td ltx_align_center">62.9 <sub id="A7.T21.19.19.1.1" class="ltx_sub"><span id="A7.T21.19.19.1.1.1" class="ltx_text ltx_font_italic">(2.2)</span></sub>
</td>
<td id="A7.T21.20.20.2" class="ltx_td ltx_align_center">14.3 <sub id="A7.T21.20.20.2.1" class="ltx_sub"><span id="A7.T21.20.20.2.1.1" class="ltx_text ltx_font_italic">(0.1)</span></sub>
</td>
<td id="A7.T21.21.21.3" class="ltx_td ltx_align_center">50.4 <sub id="A7.T21.21.21.3.1" class="ltx_sub"><span id="A7.T21.21.21.3.1.1" class="ltx_text ltx_font_italic">(1.1)</span></sub>
</td>
<td id="A7.T21.22.22.4" class="ltx_td ltx_align_center">45.0 <sub id="A7.T21.22.22.4.1" class="ltx_sub"><span id="A7.T21.22.22.4.1.1" class="ltx_text ltx_font_italic">(2.w)</span></sub>
</td>
<td id="A7.T21.23.23.5" class="ltx_td ltx_align_center">21.0 <sub id="A7.T21.23.23.5.1" class="ltx_sub"><span id="A7.T21.23.23.5.1.1" class="ltx_text ltx_font_italic">(0.1)</span></sub>
</td>
<td id="A7.T21.24.24.6" class="ltx_td ltx_align_center">100.0 <sub id="A7.T21.24.24.6.1" class="ltx_sub"><span id="A7.T21.24.24.6.1.1" class="ltx_text ltx_font_italic">(6.8)</span></sub>
</td>
</tr>
<tr id="A7.T21.30.30" class="ltx_tr">
<th id="A7.T21.30.30.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">w/ <span id="A7.T21.30.30.7.1" class="ltx_text ltx_font_smallcaps">Interact</span>
</th>
<td id="A7.T21.25.25.1" class="ltx_td ltx_align_center">68.0 <sub id="A7.T21.25.25.1.1" class="ltx_sub"><span id="A7.T21.25.25.1.1.1" class="ltx_text ltx_font_italic">(5.8)</span></sub>
</td>
<td id="A7.T21.26.26.2" class="ltx_td ltx_align_center">13.3 <sub id="A7.T21.26.26.2.1" class="ltx_sub"><span id="A7.T21.26.26.2.1.1" class="ltx_text ltx_font_italic">(0.5)</span></sub>
</td>
<td id="A7.T21.27.27.3" class="ltx_td ltx_align_center">47.8 <sub id="A7.T21.27.27.3.1" class="ltx_sub"><span id="A7.T21.27.27.3.1.1" class="ltx_text ltx_font_italic">(3.3)</span></sub>
</td>
<td id="A7.T21.28.28.4" class="ltx_td ltx_align_center">45.0 <sub id="A7.T21.28.28.4.1" class="ltx_sub"><span id="A7.T21.28.28.4.1.1" class="ltx_text ltx_font_italic">(3.1)</span></sub>
</td>
<td id="A7.T21.29.29.5" class="ltx_td ltx_align_center">20.1 <sub id="A7.T21.29.29.5.1" class="ltx_sub"><span id="A7.T21.29.29.5.1.1" class="ltx_text ltx_font_italic">(0.2)</span></sub>
</td>
<td id="A7.T21.30.30.6" class="ltx_td ltx_align_center">99.8 <sub id="A7.T21.30.30.6.1" class="ltx_sub"><span id="A7.T21.30.30.6.1.1" class="ltx_text ltx_font_italic">(6.1)</span></sub>
</td>
</tr>
<tr id="A7.T21.36.36" class="ltx_tr">
<th id="A7.T21.36.36.7" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A7.T21.36.36.7.1" class="ltx_text ltx_font_smallcaps">InlineSearch</span></th>
<td id="A7.T21.31.31.1" class="ltx_td ltx_align_center">49.7 <sub id="A7.T21.31.31.1.1" class="ltx_sub"><span id="A7.T21.31.31.1.1.1" class="ltx_text ltx_font_italic">(4.6)</span></sub>
</td>
<td id="A7.T21.32.32.2" class="ltx_td ltx_align_center">13.4 <sub id="A7.T21.32.32.2.1" class="ltx_sub"><span id="A7.T21.32.32.2.1.1" class="ltx_text ltx_font_italic">(1.1)</span></sub>
</td>
<td id="A7.T21.33.33.3" class="ltx_td ltx_align_center">45.6 <sub id="A7.T21.33.33.3.1" class="ltx_sub"><span id="A7.T21.33.33.3.1.1" class="ltx_text ltx_font_italic">(2.5)</span></sub>
</td>
<td id="A7.T21.34.34.4" class="ltx_td ltx_align_center">43.7 <sub id="A7.T21.34.34.4.1" class="ltx_sub"><span id="A7.T21.34.34.4.1.1" class="ltx_text ltx_font_italic">(3.9)</span></sub>
</td>
<td id="A7.T21.35.35.5" class="ltx_td ltx_align_center">20.4 <sub id="A7.T21.35.35.5.1" class="ltx_sub"><span id="A7.T21.35.35.5.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
<td id="A7.T21.36.36.6" class="ltx_td ltx_align_center">103.0 <sub id="A7.T21.36.36.6.1" class="ltx_sub"><span id="A7.T21.36.36.6.1.1" class="ltx_text ltx_font_italic">(18.1)</span></sub>
</td>
</tr>
<tr id="A7.T21.42.42" class="ltx_tr">
<th id="A7.T21.42.42.7" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A7.T21.42.42.7.1" class="ltx_text ltx_font_smallcaps">ClosedBook</span></th>
<td id="A7.T21.37.37.1" class="ltx_td ltx_align_center">32.6 <sub id="A7.T21.37.37.1.1" class="ltx_sub"><span id="A7.T21.37.37.1.1.1" class="ltx_text ltx_font_italic">(1.1)</span></sub>
</td>
<td id="A7.T21.38.38.2" class="ltx_td ltx_align_center">18.6 <sub id="A7.T21.38.38.2.1" class="ltx_sub"><span id="A7.T21.38.38.2.1.1" class="ltx_text ltx_font_italic">(0.5)</span></sub>
</td>
<td id="A7.T21.39.39.3" class="ltx_td ltx_align_center">15.4 <sub id="A7.T21.39.39.3.1" class="ltx_sub"><span id="A7.T21.39.39.3.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
<td id="A7.T21.40.40.4" class="ltx_td ltx_align_center">15.4 <sub id="A7.T21.40.40.4.1" class="ltx_sub"><span id="A7.T21.40.40.4.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
<td id="A7.T21.41.41.5" class="ltx_td ltx_align_center">22.8 <sub id="A7.T21.41.41.5.1" class="ltx_sub"><span id="A7.T21.41.41.5.1.1" class="ltx_text ltx_font_italic">(0.1)</span></sub>
</td>
<td id="A7.T21.42.42.6" class="ltx_td ltx_align_center">108.3 <sub id="A7.T21.42.42.6.1" class="ltx_sub"><span id="A7.T21.42.42.6.1.1" class="ltx_text ltx_font_italic">(8.9)</span></sub>
</td>
</tr>
<tr id="A7.T21.48.48" class="ltx_tr">
<th id="A7.T21.48.48.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T21.48.48.7.1" class="ltx_text ltx_font_smallcaps">Oracle</span> (5-psg)</th>
<td id="A7.T21.43.43.1" class="ltx_td ltx_align_center">59.4 <sub id="A7.T21.43.43.1.1" class="ltx_sub"><span id="A7.T21.43.43.1.1.1" class="ltx_text ltx_font_italic">(4.1)</span></sub>
</td>
<td id="A7.T21.44.44.2" class="ltx_td ltx_align_center">21.3 <sub id="A7.T21.44.44.2.1" class="ltx_sub"><span id="A7.T21.44.44.2.1.1" class="ltx_text ltx_font_italic">(0.2)</span></sub>
</td>
<td id="A7.T21.45.45.3" class="ltx_td ltx_align_center">57.8 <sub id="A7.T21.45.45.3.1" class="ltx_sub"><span id="A7.T21.45.45.3.1.1" class="ltx_text ltx_font_italic">(3.7)</span></sub>
</td>
<td id="A7.T21.46.46.4" class="ltx_td ltx_align_center">56.0 <sub id="A7.T21.46.46.4.1" class="ltx_sub"><span id="A7.T21.46.46.4.1.1" class="ltx_text ltx_font_italic">(3.8)</span></sub>
</td>
<td id="A7.T21.47.47.5" class="ltx_td ltx_align_center">21.2 <sub id="A7.T21.47.47.5.1" class="ltx_sub"><span id="A7.T21.47.47.5.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
<td id="A7.T21.48.48.6" class="ltx_td ltx_align_center">93.0 <sub id="A7.T21.48.48.6.1" class="ltx_sub"><span id="A7.T21.48.48.6.1.1" class="ltx_text ltx_font_italic">(7.8)</span></sub>
</td>
</tr>
<tr id="A7.T21.246.250.4" class="ltx_tr" style="background-color:#E9E9E9;">
<th id="A7.T21.246.250.4.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" colspan="7"><span id="A7.T21.246.250.4.1.1" class="ltx_text ltx_font_bold" style="background-color:#E9E9E9;">ChatGPT-16K</span></th>
</tr>
<tr id="A7.T21.54.54" class="ltx_tr">
<th id="A7.T21.54.54.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T21.54.54.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (5-psg)</th>
<td id="A7.T21.49.49.1" class="ltx_td ltx_align_center">31.6 <sub id="A7.T21.49.49.1.1" class="ltx_sub"><span id="A7.T21.49.49.1.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.50.50.2" class="ltx_td ltx_align_center">14.4 <sub id="A7.T21.50.50.2.1" class="ltx_sub"><span id="A7.T21.50.50.2.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.51.51.3" class="ltx_td ltx_align_center">44.6 <sub id="A7.T21.51.51.3.1" class="ltx_sub"><span id="A7.T21.51.51.3.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.52.52.4" class="ltx_td ltx_align_center">44.1 <sub id="A7.T21.52.52.4.1" class="ltx_sub"><span id="A7.T21.52.52.4.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.53.53.5" class="ltx_td ltx_align_center">21.4 <sub id="A7.T21.53.53.5.1" class="ltx_sub"><span id="A7.T21.53.53.5.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.54.54.6" class="ltx_td ltx_align_center">87.6 <sub id="A7.T21.54.54.6.1" class="ltx_sub"><span id="A7.T21.54.54.6.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
</tr>
<tr id="A7.T21.60.60" class="ltx_tr">
<th id="A7.T21.60.60.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T21.60.60.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (10-psg)</th>
<td id="A7.T21.55.55.1" class="ltx_td ltx_align_center">26.6 <sub id="A7.T21.55.55.1.1" class="ltx_sub"><span id="A7.T21.55.55.1.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.56.56.2" class="ltx_td ltx_align_center">14.4 <sub id="A7.T21.56.56.2.1" class="ltx_sub"><span id="A7.T21.56.56.2.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.57.57.3" class="ltx_td ltx_align_center">45.5 <sub id="A7.T21.57.57.3.1" class="ltx_sub"><span id="A7.T21.57.57.3.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.58.58.4" class="ltx_td ltx_align_center">43.3 <sub id="A7.T21.58.58.4.1" class="ltx_sub"><span id="A7.T21.58.58.4.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.59.59.5" class="ltx_td ltx_align_center">21.5 <sub id="A7.T21.59.59.5.1" class="ltx_sub"><span id="A7.T21.59.59.5.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.60.60.6" class="ltx_td ltx_align_center">87.5 <sub id="A7.T21.60.60.6.1" class="ltx_sub"><span id="A7.T21.60.60.6.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
</tr>
<tr id="A7.T21.66.66" class="ltx_tr">
<th id="A7.T21.66.66.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T21.66.66.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (20-psg)</th>
<td id="A7.T21.61.61.1" class="ltx_td ltx_align_center">31.6 <sub id="A7.T21.61.61.1.1" class="ltx_sub"><span id="A7.T21.61.61.1.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.62.62.2" class="ltx_td ltx_align_center">15.9 <sub id="A7.T21.62.62.2.1" class="ltx_sub"><span id="A7.T21.62.62.2.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.63.63.3" class="ltx_td ltx_align_center">43.4 <sub id="A7.T21.63.63.3.1" class="ltx_sub"><span id="A7.T21.63.63.3.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.64.64.4" class="ltx_td ltx_align_center">40.9 <sub id="A7.T21.64.64.4.1" class="ltx_sub"><span id="A7.T21.64.64.4.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.65.65.5" class="ltx_td ltx_align_center">21.7 <sub id="A7.T21.65.65.5.1" class="ltx_sub"><span id="A7.T21.65.65.5.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.66.66.6" class="ltx_td ltx_align_center">92.6 <sub id="A7.T21.66.66.6.1" class="ltx_sub"><span id="A7.T21.66.66.6.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
</tr>
<tr id="A7.T21.246.251.5" class="ltx_tr">
<th id="A7.T21.246.251.5.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" colspan="7"><span id="A7.T21.246.251.5.1.1" class="ltx_text ltx_font_bold" style="background-color:#E9E9E9;">GPT-4</span></th>
</tr>
<tr id="A7.T21.72.72" class="ltx_tr">
<th id="A7.T21.72.72.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T21.72.72.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (5-psg)</th>
<td id="A7.T21.67.67.1" class="ltx_td ltx_align_center">38.4 <sub id="A7.T21.67.67.1.1" class="ltx_sub"><span id="A7.T21.67.67.1.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.68.68.2" class="ltx_td ltx_align_center">14.2 <sub id="A7.T21.68.68.2.1" class="ltx_sub"><span id="A7.T21.68.68.2.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.69.69.3" class="ltx_td ltx_align_center">44.0 <sub id="A7.T21.69.69.3.1" class="ltx_sub"><span id="A7.T21.69.69.3.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.70.70.4" class="ltx_td ltx_align_center">50.1 <sub id="A7.T21.70.70.4.1" class="ltx_sub"><span id="A7.T21.70.70.4.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.71.71.5" class="ltx_td ltx_align_center">20.6 <sub id="A7.T21.71.71.5.1" class="ltx_sub"><span id="A7.T21.71.71.5.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.72.72.6" class="ltx_td ltx_align_center">79.6 <sub id="A7.T21.72.72.6.1" class="ltx_sub"><span id="A7.T21.72.72.6.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
</tr>
<tr id="A7.T21.78.78" class="ltx_tr">
<th id="A7.T21.78.78.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T21.78.78.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (10-psg)</th>
<td id="A7.T21.73.73.1" class="ltx_td ltx_align_center">39.9 <sub id="A7.T21.73.73.1.1" class="ltx_sub"><span id="A7.T21.73.73.1.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.74.74.2" class="ltx_td ltx_align_center">15.7 <sub id="A7.T21.74.74.2.1" class="ltx_sub"><span id="A7.T21.74.74.2.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.75.75.3" class="ltx_td ltx_align_center">49.5 <sub id="A7.T21.75.75.3.1" class="ltx_sub"><span id="A7.T21.75.75.3.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.76.76.4" class="ltx_td ltx_align_center">54.2 <sub id="A7.T21.76.76.4.1" class="ltx_sub"><span id="A7.T21.76.76.4.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.77.77.5" class="ltx_td ltx_align_center">21.2 <sub id="A7.T21.77.77.5.1" class="ltx_sub"><span id="A7.T21.77.77.5.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.78.78.6" class="ltx_td ltx_align_center">88.2 <sub id="A7.T21.78.78.6.1" class="ltx_sub"><span id="A7.T21.78.78.6.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
</tr>
<tr id="A7.T21.84.84" class="ltx_tr">
<th id="A7.T21.84.84.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T21.84.84.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (20-psg)</th>
<td id="A7.T21.79.79.1" class="ltx_td ltx_align_center">41.5 <sub id="A7.T21.79.79.1.1" class="ltx_sub"><span id="A7.T21.79.79.1.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.80.80.2" class="ltx_td ltx_align_center">18.3 <sub id="A7.T21.80.80.2.1" class="ltx_sub"><span id="A7.T21.80.80.2.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.81.81.3" class="ltx_td ltx_align_center">48.5 <sub id="A7.T21.81.81.3.1" class="ltx_sub"><span id="A7.T21.81.81.3.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.82.82.4" class="ltx_td ltx_align_center">53.4 <sub id="A7.T21.82.82.4.1" class="ltx_sub"><span id="A7.T21.82.82.4.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.83.83.5" class="ltx_td ltx_align_center">22.2 <sub id="A7.T21.83.83.5.1" class="ltx_sub"><span id="A7.T21.83.83.5.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
<td id="A7.T21.84.84.6" class="ltx_td ltx_align_center">97.0 <sub id="A7.T21.84.84.6.1" class="ltx_sub"><span id="A7.T21.84.84.6.1.1" class="ltx_text ltx_font_italic">(-)</span></sub>
</td>
</tr>
<tr id="A7.T21.246.252.6" class="ltx_tr">
<th id="A7.T21.246.252.6.1" class="ltx_td ltx_align_center ltx_th ltx_th_row" colspan="7"><span id="A7.T21.246.252.6.1.1" class="ltx_text ltx_font_bold" style="background-color:#E9E9E9;">Open-source</span></th>
</tr>
<tr id="A7.T21.90.90" class="ltx_tr">
<th id="A7.T21.90.90.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-7B <span id="A7.T21.90.90.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (3-psg)</th>
<td id="A7.T21.85.85.1" class="ltx_td ltx_align_center">28.6 <sub id="A7.T21.85.85.1.1" class="ltx_sub"><span id="A7.T21.85.85.1.1.1" class="ltx_text ltx_font_italic">(17.9)</span></sub>
</td>
<td id="A7.T21.86.86.2" class="ltx_td ltx_align_center">1.6 <sub id="A7.T21.86.86.2.1" class="ltx_sub"><span id="A7.T21.86.86.2.1.1" class="ltx_text ltx_font_italic">(0.9)</span></sub>
</td>
<td id="A7.T21.87.87.3" class="ltx_td ltx_align_center">1.2 <sub id="A7.T21.87.87.3.1" class="ltx_sub"><span id="A7.T21.87.87.3.1.1" class="ltx_text ltx_font_italic">(0.0)</span></sub>
</td>
<td id="A7.T21.88.88.4" class="ltx_td ltx_align_center">2.7 <sub id="A7.T21.88.88.4.1" class="ltx_sub"><span id="A7.T21.88.88.4.1.1" class="ltx_text ltx_font_italic">(0.1)</span></sub>
</td>
<td id="A7.T21.89.89.5" class="ltx_td ltx_align_center">12.2 <sub id="A7.T21.89.89.5.1" class="ltx_sub"><span id="A7.T21.89.89.5.1.1" class="ltx_text ltx_font_italic">(1.3)</span></sub>
</td>
<td id="A7.T21.90.90.6" class="ltx_td ltx_align_center">46.9 <sub id="A7.T21.90.90.6.1" class="ltx_sub"><span id="A7.T21.90.90.6.1.1" class="ltx_text ltx_font_italic">(1.2)</span></sub>
</td>
</tr>
<tr id="A7.T21.96.96" class="ltx_tr">
<th id="A7.T21.96.96.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">Alpaca-7B <span id="A7.T21.96.96.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (3-psg)</th>
<td id="A7.T21.91.91.1" class="ltx_td ltx_align_center">45.9 <sub id="A7.T21.91.91.1.1" class="ltx_sub"><span id="A7.T21.91.91.1.1.1" class="ltx_text ltx_font_italic">(5.3)</span></sub>
</td>
<td id="A7.T21.92.92.2" class="ltx_td ltx_align_center">9.2 <sub id="A7.T21.92.92.2.1" class="ltx_sub"><span id="A7.T21.92.92.2.1.1" class="ltx_text ltx_font_italic">(0.1)</span></sub>
</td>
<td id="A7.T21.93.93.3" class="ltx_td ltx_align_center">4.5 <sub id="A7.T21.93.93.3.1" class="ltx_sub"><span id="A7.T21.93.93.3.1.1" class="ltx_text ltx_font_italic">(1.6)</span></sub>
</td>
<td id="A7.T21.94.94.4" class="ltx_td ltx_align_center">5.2 <sub id="A7.T21.94.94.4.1" class="ltx_sub"><span id="A7.T21.94.94.4.1.1" class="ltx_text ltx_font_italic">(1.9)</span></sub>
</td>
<td id="A7.T21.95.95.5" class="ltx_td ltx_align_center">18.8 <sub id="A7.T21.95.95.5.1" class="ltx_sub"><span id="A7.T21.95.95.5.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
<td id="A7.T21.96.96.6" class="ltx_td ltx_align_center">67.1 <sub id="A7.T21.96.96.6.1" class="ltx_sub"><span id="A7.T21.96.96.6.1.1" class="ltx_text ltx_font_italic">(1.2)</span></sub>
</td>
</tr>
<tr id="A7.T21.102.102" class="ltx_tr">
<th id="A7.T21.102.102.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">Vicuna-7B <span id="A7.T21.102.102.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (3-psg)</th>
<td id="A7.T21.97.97.1" class="ltx_td ltx_align_center">43.2 <sub id="A7.T21.97.97.1.1" class="ltx_sub"><span id="A7.T21.97.97.1.1.1" class="ltx_text ltx_font_italic">(3.9)</span></sub>
</td>
<td id="A7.T21.98.98.2" class="ltx_td ltx_align_center">10.0 <sub id="A7.T21.98.98.2.1" class="ltx_sub"><span id="A7.T21.98.98.2.1.1" class="ltx_text ltx_font_italic">(0.5)</span></sub>
</td>
<td id="A7.T21.99.99.3" class="ltx_td ltx_align_center">12.6 <sub id="A7.T21.99.99.3.1" class="ltx_sub"><span id="A7.T21.99.99.3.1.1" class="ltx_text ltx_font_italic">(2.3)</span></sub>
</td>
<td id="A7.T21.100.100.4" class="ltx_td ltx_align_center">16.3 <sub id="A7.T21.100.100.4.1" class="ltx_sub"><span id="A7.T21.100.100.4.1.1" class="ltx_text ltx_font_italic">(2.6)</span></sub>
</td>
<td id="A7.T21.101.101.5" class="ltx_td ltx_align_center">19.1 <sub id="A7.T21.101.101.5.1" class="ltx_sub"><span id="A7.T21.101.101.5.1.1" class="ltx_text ltx_font_italic">(0.4)</span></sub>
</td>
<td id="A7.T21.102.102.6" class="ltx_td ltx_align_center">68.7 <sub id="A7.T21.102.102.6.1" class="ltx_sub"><span id="A7.T21.102.102.6.1.1" class="ltx_text ltx_font_italic">(2.0)</span></sub>
</td>
</tr>
<tr id="A7.T21.108.108" class="ltx_tr">
<th id="A7.T21.108.108.7" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">LLaMA-13B <span id="A7.T21.108.108.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (3-psg)</th>
<td id="A7.T21.103.103.1" class="ltx_td ltx_align_center ltx_border_t">50.0 <sub id="A7.T21.103.103.1.1" class="ltx_sub"><span id="A7.T21.103.103.1.1.1" class="ltx_text ltx_font_italic">(2.0)</span></sub>
</td>
<td id="A7.T21.104.104.2" class="ltx_td ltx_align_center ltx_border_t">3.9 <sub id="A7.T21.104.104.2.1" class="ltx_sub"><span id="A7.T21.104.104.2.1.1" class="ltx_text ltx_font_italic">(0.4)</span></sub>
</td>
<td id="A7.T21.105.105.3" class="ltx_td ltx_align_center ltx_border_t">3.1 <sub id="A7.T21.105.105.3.1" class="ltx_sub"><span id="A7.T21.105.105.3.1.1" class="ltx_text ltx_font_italic">(0.9)</span></sub>
</td>
<td id="A7.T21.106.106.4" class="ltx_td ltx_align_center ltx_border_t">5.3 <sub id="A7.T21.106.106.4.1" class="ltx_sub"><span id="A7.T21.106.106.4.1.1" class="ltx_text ltx_font_italic">(1.3)</span></sub>
</td>
<td id="A7.T21.107.107.5" class="ltx_td ltx_align_center ltx_border_t">16.1 <sub id="A7.T21.107.107.5.1" class="ltx_sub"><span id="A7.T21.107.107.5.1.1" class="ltx_text ltx_font_italic">(0.5)</span></sub>
</td>
<td id="A7.T21.108.108.6" class="ltx_td ltx_align_center ltx_border_t">63.3 <sub id="A7.T21.108.108.6.1" class="ltx_sub"><span id="A7.T21.108.108.6.1.1" class="ltx_text ltx_font_italic">(2.0)</span></sub>
</td>
</tr>
<tr id="A7.T21.114.114" class="ltx_tr">
<th id="A7.T21.114.114.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">w/ <span id="A7.T21.114.114.7.1" class="ltx_text ltx_font_smallcaps">Rerank</span>
</th>
<td id="A7.T21.109.109.1" class="ltx_td ltx_align_center">46.7 <sub id="A7.T21.109.109.1.1" class="ltx_sub"><span id="A7.T21.109.109.1.1.1" class="ltx_text ltx_font_italic">(2.9)</span></sub>
</td>
<td id="A7.T21.110.110.2" class="ltx_td ltx_align_center">4.3 <sub id="A7.T21.110.110.2.1" class="ltx_sub"><span id="A7.T21.110.110.2.1.1" class="ltx_text ltx_font_italic">(0.4)</span></sub>
</td>
<td id="A7.T21.111.111.3" class="ltx_td ltx_align_center">9.7 <sub id="A7.T21.111.111.3.1" class="ltx_sub"><span id="A7.T21.111.111.3.1.1" class="ltx_text ltx_font_italic">(2.1)</span></sub>
</td>
<td id="A7.T21.112.112.4" class="ltx_td ltx_align_center">15.0 <sub id="A7.T21.112.112.4.1" class="ltx_sub"><span id="A7.T21.112.112.4.1.1" class="ltx_text ltx_font_italic">(2.2)</span></sub>
</td>
<td id="A7.T21.113.113.5" class="ltx_td ltx_align_center">16.1 <sub id="A7.T21.113.113.5.1" class="ltx_sub"><span id="A7.T21.113.113.5.1.1" class="ltx_text ltx_font_italic">(0.7)</span></sub>
</td>
<td id="A7.T21.114.114.6" class="ltx_td ltx_align_center">63.0 <sub id="A7.T21.114.114.6.1" class="ltx_sub"><span id="A7.T21.114.114.6.1.1" class="ltx_text ltx_font_italic">(2.3)</span></sub>
</td>
</tr>
<tr id="A7.T21.120.120" class="ltx_tr">
<th id="A7.T21.120.120.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-13B <span id="A7.T21.120.120.7.1" class="ltx_text ltx_font_smallcaps">Summ</span> (10-psg)</th>
<td id="A7.T21.115.115.1" class="ltx_td ltx_align_center">28.6 <sub id="A7.T21.115.115.1.1" class="ltx_sub"><span id="A7.T21.115.115.1.1.1" class="ltx_text ltx_font_italic">(1.8)</span></sub>
</td>
<td id="A7.T21.116.116.2" class="ltx_td ltx_align_center">2.9 <sub id="A7.T21.116.116.2.1" class="ltx_sub"><span id="A7.T21.116.116.2.1.1" class="ltx_text ltx_font_italic">(0.1)</span></sub>
</td>
<td id="A7.T21.117.117.3" class="ltx_td ltx_align_center">2.5 <sub id="A7.T21.117.117.3.1" class="ltx_sub"><span id="A7.T21.117.117.3.1.1" class="ltx_text ltx_font_italic">(0.8)</span></sub>
</td>
<td id="A7.T21.118.118.4" class="ltx_td ltx_align_center">3.8 <sub id="A7.T21.118.118.4.1" class="ltx_sub"><span id="A7.T21.118.118.4.1.1" class="ltx_text ltx_font_italic">(0.8)</span></sub>
</td>
<td id="A7.T21.119.119.5" class="ltx_td ltx_align_center">8.5 <sub id="A7.T21.119.119.5.1" class="ltx_sub"><span id="A7.T21.119.119.5.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
<td id="A7.T21.120.120.6" class="ltx_td ltx_align_center">33.1 <sub id="A7.T21.120.120.6.1" class="ltx_sub"><span id="A7.T21.120.120.6.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
</tr>
<tr id="A7.T21.126.126" class="ltx_tr">
<th id="A7.T21.126.126.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-13B <span id="A7.T21.126.126.7.1" class="ltx_text ltx_font_smallcaps">Snippet</span> (10-psg)</th>
<td id="A7.T21.121.121.1" class="ltx_td ltx_align_center">48.4 <sub id="A7.T21.121.121.1.1" class="ltx_sub"><span id="A7.T21.121.121.1.1.1" class="ltx_text ltx_font_italic">(3.1)</span></sub>
</td>
<td id="A7.T21.122.122.2" class="ltx_td ltx_align_center">5.7 <sub id="A7.T21.122.122.2.1" class="ltx_sub"><span id="A7.T21.122.122.2.1.1" class="ltx_text ltx_font_italic">(0.9)</span></sub>
</td>
<td id="A7.T21.123.123.3" class="ltx_td ltx_align_center">5.8 <sub id="A7.T21.123.123.3.1" class="ltx_sub"><span id="A7.T21.123.123.3.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
<td id="A7.T21.124.124.4" class="ltx_td ltx_align_center">7.6 <sub id="A7.T21.124.124.4.1" class="ltx_sub"><span id="A7.T21.124.124.4.1.1" class="ltx_text ltx_font_italic">(0.9)</span></sub>
</td>
<td id="A7.T21.125.125.5" class="ltx_td ltx_align_center">15.1 <sub id="A7.T21.125.125.5.1" class="ltx_sub"><span id="A7.T21.125.125.5.1.1" class="ltx_text ltx_font_italic">(1.1)</span></sub>
</td>
<td id="A7.T21.126.126.6" class="ltx_td ltx_align_center">60.2 <sub id="A7.T21.126.126.6.1" class="ltx_sub"><span id="A7.T21.126.126.6.1.1" class="ltx_text ltx_font_italic">(3.2)</span></sub>
</td>
</tr>
<tr id="A7.T21.132.132" class="ltx_tr">
<th id="A7.T21.132.132.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-13B <span id="A7.T21.132.132.7.1" class="ltx_text ltx_font_smallcaps">Oracle</span> (3-psg)</th>
<td id="A7.T21.127.127.1" class="ltx_td ltx_align_center">49.5 <sub id="A7.T21.127.127.1.1" class="ltx_sub"><span id="A7.T21.127.127.1.1.1" class="ltx_text ltx_font_italic">(2.4)</span></sub>
</td>
<td id="A7.T21.128.128.2" class="ltx_td ltx_align_center">6.4 <sub id="A7.T21.128.128.2.1" class="ltx_sub"><span id="A7.T21.128.128.2.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
<td id="A7.T21.129.129.3" class="ltx_td ltx_align_center">3.7 <sub id="A7.T21.129.129.3.1" class="ltx_sub"><span id="A7.T21.129.129.3.1.1" class="ltx_text ltx_font_italic">(0.7)</span></sub>
</td>
<td id="A7.T21.130.130.4" class="ltx_td ltx_align_center">6.5 <sub id="A7.T21.130.130.4.1" class="ltx_sub"><span id="A7.T21.130.130.4.1.1" class="ltx_text ltx_font_italic">(1.0)</span></sub>
</td>
<td id="A7.T21.131.131.5" class="ltx_td ltx_align_center">16.8 <sub id="A7.T21.131.131.5.1" class="ltx_sub"><span id="A7.T21.131.131.5.1.1" class="ltx_text ltx_font_italic">(0.5)</span></sub>
</td>
<td id="A7.T21.132.132.6" class="ltx_td ltx_align_center">64.5 <sub id="A7.T21.132.132.6.1" class="ltx_sub"><span id="A7.T21.132.132.6.1.1" class="ltx_text ltx_font_italic">(1.7)</span></sub>
</td>
</tr>
<tr id="A7.T21.138.138" class="ltx_tr">
<th id="A7.T21.138.138.7" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Vicuna-13B <span id="A7.T21.138.138.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (3-psg)</th>
<td id="A7.T21.133.133.1" class="ltx_td ltx_align_center ltx_border_t">58.2 <sub id="A7.T21.133.133.1.1" class="ltx_sub"><span id="A7.T21.133.133.1.1.1" class="ltx_text ltx_font_italic">(25.1)</span></sub>
</td>
<td id="A7.T21.134.134.2" class="ltx_td ltx_align_center ltx_border_t">10.0 <sub id="A7.T21.134.134.2.1" class="ltx_sub"><span id="A7.T21.134.134.2.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
<td id="A7.T21.135.135.3" class="ltx_td ltx_align_center ltx_border_t">15.6 <sub id="A7.T21.135.135.3.1" class="ltx_sub"><span id="A7.T21.135.135.3.1.1" class="ltx_text ltx_font_italic">(2.2)</span></sub>
</td>
<td id="A7.T21.136.136.4" class="ltx_td ltx_align_center ltx_border_t">19.6 <sub id="A7.T21.136.136.4.1" class="ltx_sub"><span id="A7.T21.136.136.4.1.1" class="ltx_text ltx_font_italic">(2.0)</span></sub>
</td>
<td id="A7.T21.137.137.5" class="ltx_td ltx_align_center ltx_border_t">19.1 <sub id="A7.T21.137.137.5.1" class="ltx_sub"><span id="A7.T21.137.137.5.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
<td id="A7.T21.138.138.6" class="ltx_td ltx_align_center ltx_border_t">69.6 <sub id="A7.T21.138.138.6.1" class="ltx_sub"><span id="A7.T21.138.138.6.1.1" class="ltx_text ltx_font_italic">(0.6)</span></sub>
</td>
</tr>
<tr id="A7.T21.144.144" class="ltx_tr">
<th id="A7.T21.144.144.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">w/ <span id="A7.T21.144.144.7.1" class="ltx_text ltx_font_smallcaps">Rerank</span>
</th>
<td id="A7.T21.139.139.1" class="ltx_td ltx_align_center">45.9 <sub id="A7.T21.139.139.1.1" class="ltx_sub"><span id="A7.T21.139.139.1.1.1" class="ltx_text ltx_font_italic">(4.3)</span></sub>
</td>
<td id="A7.T21.140.140.2" class="ltx_td ltx_align_center">9.2 <sub id="A7.T21.140.140.2.1" class="ltx_sub"><span id="A7.T21.140.140.2.1.1" class="ltx_text ltx_font_italic">(0.0)</span></sub>
</td>
<td id="A7.T21.141.141.3" class="ltx_td ltx_align_center">31.7 <sub id="A7.T21.141.141.3.1" class="ltx_sub"><span id="A7.T21.141.141.3.1.1" class="ltx_text ltx_font_italic">(2.9)</span></sub>
</td>
<td id="A7.T21.142.142.4" class="ltx_td ltx_align_center">38.2 <sub id="A7.T21.142.142.4.1" class="ltx_sub"><span id="A7.T21.142.142.4.1.1" class="ltx_text ltx_font_italic">(1.6)</span></sub>
</td>
<td id="A7.T21.143.143.5" class="ltx_td ltx_align_center">18.6 <sub id="A7.T21.143.143.5.1" class="ltx_sub"><span id="A7.T21.143.143.5.1.1" class="ltx_text ltx_font_italic">(0.5)</span></sub>
</td>
<td id="A7.T21.144.144.6" class="ltx_td ltx_align_center">69.7 <sub id="A7.T21.144.144.6.1" class="ltx_sub"><span id="A7.T21.144.144.6.1.1" class="ltx_text ltx_font_italic">(1.0)</span></sub>
</td>
</tr>
<tr id="A7.T21.150.150" class="ltx_tr">
<th id="A7.T21.150.150.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">Vicuna-13B <span id="A7.T21.150.150.7.1" class="ltx_text ltx_font_smallcaps">Summ</span> (10-psg)</th>
<td id="A7.T21.145.145.1" class="ltx_td ltx_align_center">22.4 <sub id="A7.T21.145.145.1.1" class="ltx_sub"><span id="A7.T21.145.145.1.1.1" class="ltx_text ltx_font_italic">(3.0)</span></sub>
</td>
<td id="A7.T21.146.146.2" class="ltx_td ltx_align_center">4.9 <sub id="A7.T21.146.146.2.1" class="ltx_sub"><span id="A7.T21.146.146.2.1.1" class="ltx_text ltx_font_italic">(0.1)</span></sub>
</td>
<td id="A7.T21.147.147.3" class="ltx_td ltx_align_center">9.7 <sub id="A7.T21.147.147.3.1" class="ltx_sub"><span id="A7.T21.147.147.3.1.1" class="ltx_text ltx_font_italic">(1.3)</span></sub>
</td>
<td id="A7.T21.148.148.4" class="ltx_td ltx_align_center">12.2 <sub id="A7.T21.148.148.4.1" class="ltx_sub"><span id="A7.T21.148.148.4.1.1" class="ltx_text ltx_font_italic">(1.2)</span></sub>
</td>
<td id="A7.T21.149.149.5" class="ltx_td ltx_align_center">9.3 <sub id="A7.T21.149.149.5.1" class="ltx_sub"><span id="A7.T21.149.149.5.1.1" class="ltx_text ltx_font_italic">(0.4)</span></sub>
</td>
<td id="A7.T21.150.150.6" class="ltx_td ltx_align_center">33.0 <sub id="A7.T21.150.150.6.1" class="ltx_sub"><span id="A7.T21.150.150.6.1.1" class="ltx_text ltx_font_italic">(3.7)</span></sub>
</td>
</tr>
<tr id="A7.T21.156.156" class="ltx_tr">
<th id="A7.T21.156.156.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">Vicuna-13B <span id="A7.T21.156.156.7.1" class="ltx_text ltx_font_smallcaps">Snippet</span> (10-psg)</th>
<td id="A7.T21.151.151.1" class="ltx_td ltx_align_center">48.1 <sub id="A7.T21.151.151.1.1" class="ltx_sub"><span id="A7.T21.151.151.1.1.1" class="ltx_text ltx_font_italic">(5.3)</span></sub>
</td>
<td id="A7.T21.152.152.2" class="ltx_td ltx_align_center">11.2 <sub id="A7.T21.152.152.2.1" class="ltx_sub"><span id="A7.T21.152.152.2.1.1" class="ltx_text ltx_font_italic">(1.4)</span></sub>
</td>
<td id="A7.T21.153.153.3" class="ltx_td ltx_align_center">27.2 <sub id="A7.T21.153.153.3.1" class="ltx_sub"><span id="A7.T21.153.153.3.1.1" class="ltx_text ltx_font_italic">(3.6)</span></sub>
</td>
<td id="A7.T21.154.154.4" class="ltx_td ltx_align_center">27.9 <sub id="A7.T21.154.154.4.1" class="ltx_sub"><span id="A7.T21.154.154.4.1.1" class="ltx_text ltx_font_italic">(1.9)</span></sub>
</td>
<td id="A7.T21.155.155.5" class="ltx_td ltx_align_center">18.4 <sub id="A7.T21.155.155.5.1" class="ltx_sub"><span id="A7.T21.155.155.5.1.1" class="ltx_text ltx_font_italic">(1.9)</span></sub>
</td>
<td id="A7.T21.156.156.6" class="ltx_td ltx_align_center">76.8 <sub id="A7.T21.156.156.6.1" class="ltx_sub"><span id="A7.T21.156.156.6.1.1" class="ltx_text ltx_font_italic">(8.7)</span></sub>
</td>
</tr>
<tr id="A7.T21.162.162" class="ltx_tr">
<th id="A7.T21.162.162.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">Vicuna-13B <span id="A7.T21.162.162.7.1" class="ltx_text ltx_font_smallcaps">Oracle</span> (3-psg)</th>
<td id="A7.T21.157.157.1" class="ltx_td ltx_align_center">41.6 <sub id="A7.T21.157.157.1.1" class="ltx_sub"><span id="A7.T21.157.157.1.1.1" class="ltx_text ltx_font_italic">(3.1)</span></sub>
</td>
<td id="A7.T21.158.158.2" class="ltx_td ltx_align_center">17.1 <sub id="A7.T21.158.158.2.1" class="ltx_sub"><span id="A7.T21.158.158.2.1.1" class="ltx_text ltx_font_italic">(0.4)</span></sub>
</td>
<td id="A7.T21.159.159.3" class="ltx_td ltx_align_center">20.2 <sub id="A7.T21.159.159.3.1" class="ltx_sub"><span id="A7.T21.159.159.3.1.1" class="ltx_text ltx_font_italic">(3.0)</span></sub>
</td>
<td id="A7.T21.160.160.4" class="ltx_td ltx_align_center">26.5 <sub id="A7.T21.160.160.4.1" class="ltx_sub"><span id="A7.T21.160.160.4.1.1" class="ltx_text ltx_font_italic">(3.0)</span></sub>
</td>
<td id="A7.T21.161.161.5" class="ltx_td ltx_align_center">20.0 <sub id="A7.T21.161.161.5.1" class="ltx_sub"><span id="A7.T21.161.161.5.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
<td id="A7.T21.162.162.6" class="ltx_td ltx_align_center">72.0 <sub id="A7.T21.162.162.6.1" class="ltx_sub"><span id="A7.T21.162.162.6.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
</tr>
<tr id="A7.T21.168.168" class="ltx_tr">
<th id="A7.T21.168.168.7" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">LLaMA-33B <span id="A7.T21.168.168.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (3-psg)</th>
<td id="A7.T21.163.163.1" class="ltx_td ltx_align_center ltx_border_t">58.8 <sub id="A7.T21.163.163.1.1" class="ltx_sub"><span id="A7.T21.163.163.1.1.1" class="ltx_text ltx_font_italic">(4.3)</span></sub>
</td>
<td id="A7.T21.164.164.2" class="ltx_td ltx_align_center ltx_border_t">6.2 <sub id="A7.T21.164.164.2.1" class="ltx_sub"><span id="A7.T21.164.164.2.1.1" class="ltx_text ltx_font_italic">(0.0)</span></sub>
</td>
<td id="A7.T21.165.165.3" class="ltx_td ltx_align_center ltx_border_t">9.3 <sub id="A7.T21.165.165.3.1" class="ltx_sub"><span id="A7.T21.165.165.3.1.1" class="ltx_text ltx_font_italic">(3.0)</span></sub>
</td>
<td id="A7.T21.166.166.4" class="ltx_td ltx_align_center ltx_border_t">12.1 <sub id="A7.T21.166.166.4.1" class="ltx_sub"><span id="A7.T21.166.166.4.1.1" class="ltx_text ltx_font_italic">(4.2)</span></sub>
</td>
<td id="A7.T21.167.167.5" class="ltx_td ltx_align_center ltx_border_t">16.9 <sub id="A7.T21.167.167.5.1" class="ltx_sub"><span id="A7.T21.167.167.5.1.1" class="ltx_text ltx_font_italic">(0.2)</span></sub>
</td>
<td id="A7.T21.168.168.6" class="ltx_td ltx_align_center ltx_border_t">60.0 <sub id="A7.T21.168.168.6.1" class="ltx_sub"><span id="A7.T21.168.168.6.1.1" class="ltx_text ltx_font_italic">(1.3)</span></sub>
</td>
</tr>
<tr id="A7.T21.174.174" class="ltx_tr">
<th id="A7.T21.174.174.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">w/ <span id="A7.T21.174.174.7.1" class="ltx_text ltx_font_smallcaps">Rerank</span>
</th>
<td id="A7.T21.169.169.1" class="ltx_td ltx_align_center">65.9 <sub id="A7.T21.169.169.1.1" class="ltx_sub"><span id="A7.T21.169.169.1.1.1" class="ltx_text ltx_font_italic">(2.5)</span></sub>
</td>
<td id="A7.T21.170.170.2" class="ltx_td ltx_align_center">6.0 <sub id="A7.T21.170.170.2.1" class="ltx_sub"><span id="A7.T21.170.170.2.1.1" class="ltx_text ltx_font_italic">(0.7)</span></sub>
</td>
<td id="A7.T21.171.171.3" class="ltx_td ltx_align_center">22.5 <sub id="A7.T21.171.171.3.1" class="ltx_sub"><span id="A7.T21.171.171.3.1.1" class="ltx_text ltx_font_italic">(5.2)</span></sub>
</td>
<td id="A7.T21.172.172.4" class="ltx_td ltx_align_center">26.1 <sub id="A7.T21.172.172.4.1" class="ltx_sub"><span id="A7.T21.172.172.4.1.1" class="ltx_text ltx_font_italic">(6.9)</span></sub>
</td>
<td id="A7.T21.173.173.5" class="ltx_td ltx_align_center">17.5 <sub id="A7.T21.173.173.5.1" class="ltx_sub"><span id="A7.T21.173.173.5.1.1" class="ltx_text ltx_font_italic">(0.4)</span></sub>
</td>
<td id="A7.T21.174.174.6" class="ltx_td ltx_align_center">61.0 <sub id="A7.T21.174.174.6.1" class="ltx_sub"><span id="A7.T21.174.174.6.1.1" class="ltx_text ltx_font_italic">(1.2)</span></sub>
</td>
</tr>
<tr id="A7.T21.180.180" class="ltx_tr">
<th id="A7.T21.180.180.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-33B <span id="A7.T21.180.180.7.1" class="ltx_text ltx_font_smallcaps">Summ</span> (10-psg)</th>
<td id="A7.T21.175.175.1" class="ltx_td ltx_align_center">23.3 <sub id="A7.T21.175.175.1.1" class="ltx_sub"><span id="A7.T21.175.175.1.1.1" class="ltx_text ltx_font_italic">(2.0)</span></sub>
</td>
<td id="A7.T21.176.176.2" class="ltx_td ltx_align_center">3.0 <sub id="A7.T21.176.176.2.1" class="ltx_sub"><span id="A7.T21.176.176.2.1.1" class="ltx_text ltx_font_italic">(0.2)</span></sub>
</td>
<td id="A7.T21.177.177.3" class="ltx_td ltx_align_center">6.2 <sub id="A7.T21.177.177.3.1" class="ltx_sub"><span id="A7.T21.177.177.3.1.1" class="ltx_text ltx_font_italic">(0.5)</span></sub>
</td>
<td id="A7.T21.178.178.4" class="ltx_td ltx_align_center">8.2 <sub id="A7.T21.178.178.4.1" class="ltx_sub"><span id="A7.T21.178.178.4.1.1" class="ltx_text ltx_font_italic">(0.7)</span></sub>
</td>
<td id="A7.T21.179.179.5" class="ltx_td ltx_align_center">7.5 <sub id="A7.T21.179.179.5.1" class="ltx_sub"><span id="A7.T21.179.179.5.1.1" class="ltx_text ltx_font_italic">(0.4)</span></sub>
</td>
<td id="A7.T21.180.180.6" class="ltx_td ltx_align_center">26.2 <sub id="A7.T21.180.180.6.1" class="ltx_sub"><span id="A7.T21.180.180.6.1.1" class="ltx_text ltx_font_italic">(2.3)</span></sub>
</td>
</tr>
<tr id="A7.T21.186.186" class="ltx_tr">
<th id="A7.T21.186.186.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-33B <span id="A7.T21.186.186.7.1" class="ltx_text ltx_font_smallcaps">Snippet</span> (10-psg)</th>
<td id="A7.T21.181.181.1" class="ltx_td ltx_align_center">53.2 <sub id="A7.T21.181.181.1.1" class="ltx_sub"><span id="A7.T21.181.181.1.1.1" class="ltx_text ltx_font_italic">(4.0)</span></sub>
</td>
<td id="A7.T21.182.182.2" class="ltx_td ltx_align_center">7.4 <sub id="A7.T21.182.182.2.1" class="ltx_sub"><span id="A7.T21.182.182.2.1.1" class="ltx_text ltx_font_italic">(1.3)</span></sub>
</td>
<td id="A7.T21.183.183.3" class="ltx_td ltx_align_center">13.7 <sub id="A7.T21.183.183.3.1" class="ltx_sub"><span id="A7.T21.183.183.3.1.1" class="ltx_text ltx_font_italic">(0.5)</span></sub>
</td>
<td id="A7.T21.184.184.4" class="ltx_td ltx_align_center">15.1 <sub id="A7.T21.184.184.4.1" class="ltx_sub"><span id="A7.T21.184.184.4.1.1" class="ltx_text ltx_font_italic">(0.4)</span></sub>
</td>
<td id="A7.T21.185.185.5" class="ltx_td ltx_align_center">14.4 <sub id="A7.T21.185.185.5.1" class="ltx_sub"><span id="A7.T21.185.185.5.1.1" class="ltx_text ltx_font_italic">(1.7)</span></sub>
</td>
<td id="A7.T21.186.186.6" class="ltx_td ltx_align_center">53.3 <sub id="A7.T21.186.186.6.1" class="ltx_sub"><span id="A7.T21.186.186.6.1.1" class="ltx_text ltx_font_italic">(8.5)</span></sub>
</td>
</tr>
<tr id="A7.T21.192.192" class="ltx_tr">
<th id="A7.T21.192.192.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-33B <span id="A7.T21.192.192.7.1" class="ltx_text ltx_font_smallcaps">Oracle</span> (3-psg)</th>
<td id="A7.T21.187.187.1" class="ltx_td ltx_align_center">63.7 <sub id="A7.T21.187.187.1.1" class="ltx_sub"><span id="A7.T21.187.187.1.1.1" class="ltx_text ltx_font_italic">(2.8)</span></sub>
</td>
<td id="A7.T21.188.188.2" class="ltx_td ltx_align_center">11.4 <sub id="A7.T21.188.188.2.1" class="ltx_sub"><span id="A7.T21.188.188.2.1.1" class="ltx_text ltx_font_italic">(0.5)</span></sub>
</td>
<td id="A7.T21.189.189.3" class="ltx_td ltx_align_center">11.9 <sub id="A7.T21.189.189.3.1" class="ltx_sub"><span id="A7.T21.189.189.3.1.1" class="ltx_text ltx_font_italic">(2.6)</span></sub>
</td>
<td id="A7.T21.190.190.4" class="ltx_td ltx_align_center">15.4 <sub id="A7.T21.190.190.4.1" class="ltx_sub"><span id="A7.T21.190.190.4.1.1" class="ltx_text ltx_font_italic">(3.6)</span></sub>
</td>
<td id="A7.T21.191.191.5" class="ltx_td ltx_align_center">17.9 <sub id="A7.T21.191.191.5.1" class="ltx_sub"><span id="A7.T21.191.191.5.1.1" class="ltx_text ltx_font_italic">(0.2)</span></sub>
</td>
<td id="A7.T21.192.192.6" class="ltx_td ltx_align_center">61.7 <sub id="A7.T21.192.192.6.1" class="ltx_sub"><span id="A7.T21.192.192.6.1.1" class="ltx_text ltx_font_italic">(2.6)</span></sub>
</td>
</tr>
<tr id="A7.T21.198.198" class="ltx_tr">
<th id="A7.T21.198.198.7" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Oasst-33B <span id="A7.T21.198.198.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (3-psg)</th>
<td id="A7.T21.193.193.1" class="ltx_td ltx_align_center ltx_border_t">46.8 <sub id="A7.T21.193.193.1.1" class="ltx_sub"><span id="A7.T21.193.193.1.1.1" class="ltx_text ltx_font_italic">(7.6)</span></sub>
</td>
<td id="A7.T21.194.194.2" class="ltx_td ltx_align_center ltx_border_t">9.5 <sub id="A7.T21.194.194.2.1" class="ltx_sub"><span id="A7.T21.194.194.2.1.1" class="ltx_text ltx_font_italic">(0.2)</span></sub>
</td>
<td id="A7.T21.195.195.3" class="ltx_td ltx_align_center ltx_border_t">16.0 <sub id="A7.T21.195.195.3.1" class="ltx_sub"><span id="A7.T21.195.195.3.1.1" class="ltx_text ltx_font_italic">(2.5)</span></sub>
</td>
<td id="A7.T21.196.196.4" class="ltx_td ltx_align_center ltx_border_t">21.6 <sub id="A7.T21.196.196.4.1" class="ltx_sub"><span id="A7.T21.196.196.4.1.1" class="ltx_text ltx_font_italic">(3.5)</span></sub>
</td>
<td id="A7.T21.197.197.5" class="ltx_td ltx_align_center ltx_border_t">18.6 <sub id="A7.T21.197.197.5.1" class="ltx_sub"><span id="A7.T21.197.197.5.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
<td id="A7.T21.198.198.6" class="ltx_td ltx_align_center ltx_border_t">67.8 <sub id="A7.T21.198.198.6.1" class="ltx_sub"><span id="A7.T21.198.198.6.1.1" class="ltx_text ltx_font_italic">(1.5)</span></sub>
</td>
</tr>
<tr id="A7.T21.204.204" class="ltx_tr">
<th id="A7.T21.204.204.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">w/ <span id="A7.T21.204.204.7.1" class="ltx_text ltx_font_smallcaps">Rerank</span>
</th>
<td id="A7.T21.199.199.1" class="ltx_td ltx_align_center">52.1 <sub id="A7.T21.199.199.1.1" class="ltx_sub"><span id="A7.T21.199.199.1.1.1" class="ltx_text ltx_font_italic">(6.1)</span></sub>
</td>
<td id="A7.T21.200.200.2" class="ltx_td ltx_align_center">8.5 <sub id="A7.T21.200.200.2.1" class="ltx_sub"><span id="A7.T21.200.200.2.1.1" class="ltx_text ltx_font_italic">(0.5)</span></sub>
</td>
<td id="A7.T21.201.201.3" class="ltx_td ltx_align_center">34.4 <sub id="A7.T21.201.201.3.1" class="ltx_sub"><span id="A7.T21.201.201.3.1.1" class="ltx_text ltx_font_italic">(2.9)</span></sub>
</td>
<td id="A7.T21.202.202.4" class="ltx_td ltx_align_center">41.5 <sub id="A7.T21.202.202.4.1" class="ltx_sub"><span id="A7.T21.202.202.4.1.1" class="ltx_text ltx_font_italic">(2.5)</span></sub>
</td>
<td id="A7.T21.203.203.5" class="ltx_td ltx_align_center">18.2 <sub id="A7.T21.203.203.5.1" class="ltx_sub"><span id="A7.T21.203.203.5.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
<td id="A7.T21.204.204.6" class="ltx_td ltx_align_center">67.0 <sub id="A7.T21.204.204.6.1" class="ltx_sub"><span id="A7.T21.204.204.6.1.1" class="ltx_text ltx_font_italic">(1.5)</span></sub>
</td>
</tr>
<tr id="A7.T21.210.210" class="ltx_tr">
<th id="A7.T21.210.210.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">Oasst-33B <span id="A7.T21.210.210.7.1" class="ltx_text ltx_font_smallcaps">Summ</span> (10-psg)</th>
<td id="A7.T21.205.205.1" class="ltx_td ltx_align_center">24.8 <sub id="A7.T21.205.205.1.1" class="ltx_sub"><span id="A7.T21.205.205.1.1.1" class="ltx_text ltx_font_italic">(2.8)</span></sub>
</td>
<td id="A7.T21.206.206.2" class="ltx_td ltx_align_center">3.9 <sub id="A7.T21.206.206.2.1" class="ltx_sub"><span id="A7.T21.206.206.2.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
<td id="A7.T21.207.207.3" class="ltx_td ltx_align_center">12.3 <sub id="A7.T21.207.207.3.1" class="ltx_sub"><span id="A7.T21.207.207.3.1.1" class="ltx_text ltx_font_italic">(0.2)</span></sub>
</td>
<td id="A7.T21.208.208.4" class="ltx_td ltx_align_center">16.3 <sub id="A7.T21.208.208.4.1" class="ltx_sub"><span id="A7.T21.208.208.4.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
<td id="A7.T21.209.209.5" class="ltx_td ltx_align_center">9.1 <sub id="A7.T21.209.209.5.1" class="ltx_sub"><span id="A7.T21.209.209.5.1.1" class="ltx_text ltx_font_italic">(0.3)</span></sub>
</td>
<td id="A7.T21.210.210.6" class="ltx_td ltx_align_center">31.6 <sub id="A7.T21.210.210.6.1" class="ltx_sub"><span id="A7.T21.210.210.6.1.1" class="ltx_text ltx_font_italic">(1.5)</span></sub>
</td>
</tr>
<tr id="A7.T21.216.216" class="ltx_tr">
<th id="A7.T21.216.216.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">Oasst-33B <span id="A7.T21.216.216.7.1" class="ltx_text ltx_font_smallcaps">Snippet</span> (10-psg)</th>
<td id="A7.T21.211.211.1" class="ltx_td ltx_align_center">50.7 <sub id="A7.T21.211.211.1.1" class="ltx_sub"><span id="A7.T21.211.211.1.1.1" class="ltx_text ltx_font_italic">(4.6)</span></sub>
</td>
<td id="A7.T21.212.212.2" class="ltx_td ltx_align_center">10.7 <sub id="A7.T21.212.212.2.1" class="ltx_sub"><span id="A7.T21.212.212.2.1.1" class="ltx_text ltx_font_italic">(1.2)</span></sub>
</td>
<td id="A7.T21.213.213.3" class="ltx_td ltx_align_center">25.8 <sub id="A7.T21.213.213.3.1" class="ltx_sub"><span id="A7.T21.213.213.3.1.1" class="ltx_text ltx_font_italic">(3.3)</span></sub>
</td>
<td id="A7.T21.214.214.4" class="ltx_td ltx_align_center">26.7 <sub id="A7.T21.214.214.4.1" class="ltx_sub"><span id="A7.T21.214.214.4.1.1" class="ltx_text ltx_font_italic">(2.3)</span></sub>
</td>
<td id="A7.T21.215.215.5" class="ltx_td ltx_align_center">17.8 <sub id="A7.T21.215.215.5.1" class="ltx_sub"><span id="A7.T21.215.215.5.1.1" class="ltx_text ltx_font_italic">(1.8)</span></sub>
</td>
<td id="A7.T21.216.216.6" class="ltx_td ltx_align_center">69.6 <sub id="A7.T21.216.216.6.1" class="ltx_sub"><span id="A7.T21.216.216.6.1.1" class="ltx_text ltx_font_italic">(8.6)</span></sub>
</td>
</tr>
<tr id="A7.T21.222.222" class="ltx_tr">
<th id="A7.T21.222.222.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">Oasst-33B <span id="A7.T21.222.222.7.1" class="ltx_text ltx_font_smallcaps">Oracle</span> (3-psg)</th>
<td id="A7.T21.217.217.1" class="ltx_td ltx_align_center">50.7 <sub id="A7.T21.217.217.1.1" class="ltx_sub"><span id="A7.T21.217.217.1.1.1" class="ltx_text ltx_font_italic">(12.1)</span></sub>
</td>
<td id="A7.T21.218.218.2" class="ltx_td ltx_align_center">15.8 <sub id="A7.T21.218.218.2.1" class="ltx_sub"><span id="A7.T21.218.218.2.1.1" class="ltx_text ltx_font_italic">(0.1)</span></sub>
</td>
<td id="A7.T21.219.219.3" class="ltx_td ltx_align_center">20.8 <sub id="A7.T21.219.219.3.1" class="ltx_sub"><span id="A7.T21.219.219.3.1.1" class="ltx_text ltx_font_italic">(2.8)</span></sub>
</td>
<td id="A7.T21.220.220.4" class="ltx_td ltx_align_center">28.0 <sub id="A7.T21.220.220.4.1" class="ltx_sub"><span id="A7.T21.220.220.4.1.1" class="ltx_text ltx_font_italic">(3.2)</span></sub>
</td>
<td id="A7.T21.221.221.5" class="ltx_td ltx_align_center">19.4 <sub id="A7.T21.221.221.5.1" class="ltx_sub"><span id="A7.T21.221.221.5.1.1" class="ltx_text ltx_font_italic">(0.1)</span></sub>
</td>
<td id="A7.T21.222.222.6" class="ltx_td ltx_align_center">70.3 <sub id="A7.T21.222.222.6.1" class="ltx_sub"><span id="A7.T21.222.222.6.1.1" class="ltx_text ltx_font_italic">(1.1)</span></sub>
</td>
</tr>
<tr id="A7.T21.228.228" class="ltx_tr">
<th id="A7.T21.228.228.7" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">LLaMA-2-7B-Chat <span id="A7.T21.228.228.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (5-psg)</th>
<td id="A7.T21.223.223.1" class="ltx_td ltx_align_center ltx_border_t">27.8 <sub id="A7.T21.223.223.1.1" class="ltx_sub"><span id="A7.T21.223.223.1.1.1" class="ltx_text ltx_font_italic">(3.0)</span></sub>
</td>
<td id="A7.T21.224.224.2" class="ltx_td ltx_align_center ltx_border_t">10.9 <sub id="A7.T21.224.224.2.1" class="ltx_sub"><span id="A7.T21.224.224.2.1.1" class="ltx_text ltx_font_italic">(0.2)</span></sub>
</td>
<td id="A7.T21.225.225.3" class="ltx_td ltx_align_center ltx_border_t">19.8 <sub id="A7.T21.225.225.3.1" class="ltx_sub"><span id="A7.T21.225.225.3.1.1" class="ltx_text ltx_font_italic">(1.2)</span></sub>
</td>
<td id="A7.T21.226.226.4" class="ltx_td ltx_align_center ltx_border_t">15.0 <sub id="A7.T21.226.226.4.1" class="ltx_sub"><span id="A7.T21.226.226.4.1.1" class="ltx_text ltx_font_italic">(1.4)</span></sub>
</td>
<td id="A7.T21.227.227.5" class="ltx_td ltx_align_center ltx_border_t">20.5 <sub id="A7.T21.227.227.5.1" class="ltx_sub"><span id="A7.T21.227.227.5.1.1" class="ltx_text ltx_font_italic">(0.2)</span></sub>
</td>
<td id="A7.T21.228.228.6" class="ltx_td ltx_align_center ltx_border_t">87.8 <sub id="A7.T21.228.228.6.1" class="ltx_sub"><span id="A7.T21.228.228.6.1.1" class="ltx_text ltx_font_italic">(8.1)</span></sub>
</td>
</tr>
<tr id="A7.T21.234.234" class="ltx_tr">
<th id="A7.T21.234.234.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-2-13B-Chat <span id="A7.T21.234.234.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (5-psg)</th>
<td id="A7.T21.229.229.1" class="ltx_td ltx_align_center">34.7 <sub id="A7.T21.229.229.1.1" class="ltx_sub"><span id="A7.T21.229.229.1.1.1" class="ltx_text ltx_font_italic">(1.5)</span></sub>
</td>
<td id="A7.T21.230.230.2" class="ltx_td ltx_align_center">13.4 <sub id="A7.T21.230.230.2.1" class="ltx_sub"><span id="A7.T21.230.230.2.1.1" class="ltx_text ltx_font_italic">(0.4)</span></sub>
</td>
<td id="A7.T21.231.231.3" class="ltx_td ltx_align_center">17.3 <sub id="A7.T21.231.231.3.1" class="ltx_sub"><span id="A7.T21.231.231.3.1.1" class="ltx_text ltx_font_italic">(1.3)</span></sub>
</td>
<td id="A7.T21.232.232.4" class="ltx_td ltx_align_center">15.8 <sub id="A7.T21.232.232.4.1" class="ltx_sub"><span id="A7.T21.232.232.4.1.1" class="ltx_text ltx_font_italic">(1.4)</span></sub>
</td>
<td id="A7.T21.233.233.5" class="ltx_td ltx_align_center">20.9 <sub id="A7.T21.233.233.5.1" class="ltx_sub"><span id="A7.T21.233.233.5.1.1" class="ltx_text ltx_font_italic">(0.2)</span></sub>
</td>
<td id="A7.T21.234.234.6" class="ltx_td ltx_align_center">88.3 <sub id="A7.T21.234.234.6.1" class="ltx_sub"><span id="A7.T21.234.234.6.1.1" class="ltx_text ltx_font_italic">(6.3)</span></sub>
</td>
</tr>
<tr id="A7.T21.240.240" class="ltx_tr">
<th id="A7.T21.240.240.7" class="ltx_td ltx_align_left ltx_th ltx_th_row">LLaMA-2-70B-Chat <span id="A7.T21.240.240.7.1" class="ltx_text ltx_font_smallcaps">Vanilla</span> (5-psg)</th>
<td id="A7.T21.235.235.1" class="ltx_td ltx_align_center">38.6 <sub id="A7.T21.235.235.1.1" class="ltx_sub"><span id="A7.T21.235.235.1.1.1" class="ltx_text ltx_font_italic">(4.8)</span></sub>
</td>
<td id="A7.T21.236.236.2" class="ltx_td ltx_align_center">12.8 <sub id="A7.T21.236.236.2.1" class="ltx_sub"><span id="A7.T21.236.236.2.1.1" class="ltx_text ltx_font_italic">(1.0)</span></sub>
</td>
<td id="A7.T21.237.237.3" class="ltx_td ltx_align_center">38.3 <sub id="A7.T21.237.237.3.1" class="ltx_sub"><span id="A7.T21.237.237.3.1.1" class="ltx_text ltx_font_italic">(2.4)</span></sub>
</td>
<td id="A7.T21.238.238.4" class="ltx_td ltx_align_center">37.9 <sub id="A7.T21.238.238.4.1" class="ltx_sub"><span id="A7.T21.238.238.4.1.1" class="ltx_text ltx_font_italic">(1.9)</span></sub>
</td>
<td id="A7.T21.239.239.5" class="ltx_td ltx_align_center">21.3 <sub id="A7.T21.239.239.5.1" class="ltx_sub"><span id="A7.T21.239.239.5.1.1" class="ltx_text ltx_font_italic">(0.1)</span></sub>
</td>
<td id="A7.T21.240.240.6" class="ltx_td ltx_align_center">110.8 <sub id="A7.T21.240.240.6.1" class="ltx_sub"><span id="A7.T21.240.240.6.1.1" class="ltx_text ltx_font_italic">(5.6)</span></sub>
</td>
</tr>
<tr id="A7.T21.246.246" class="ltx_tr">
<th id="A7.T21.246.246.7" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_t">StableBeluga2</th>
<td id="A7.T21.241.241.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">33.0 <sub id="A7.T21.241.241.1.1" class="ltx_sub"><span id="A7.T21.241.241.1.1.1" class="ltx_text ltx_font_italic">(4.4)</span></sub>
</td>
<td id="A7.T21.242.242.2" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">14.0 <sub id="A7.T21.242.242.2.1" class="ltx_sub"><span id="A7.T21.242.242.2.1.1" class="ltx_text ltx_font_italic">(0.5)</span></sub>
</td>
<td id="A7.T21.243.243.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">27.9 <sub id="A7.T21.243.243.3.1" class="ltx_sub"><span id="A7.T21.243.243.3.1.1" class="ltx_text ltx_font_italic">(2.7)</span></sub>
</td>
<td id="A7.T21.244.244.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">29.0 <sub id="A7.T21.244.244.4.1" class="ltx_sub"><span id="A7.T21.244.244.4.1.1" class="ltx_text ltx_font_italic">(2.7)</span></sub>
</td>
<td id="A7.T21.245.245.5" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">20.6 <sub id="A7.T21.245.245.5.1" class="ltx_sub"><span id="A7.T21.245.245.5.1.1" class="ltx_text ltx_font_italic">(0.2)</span></sub>
</td>
<td id="A7.T21.246.246.6" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">75.6 <sub id="A7.T21.246.246.6.1" class="ltx_sub"><span id="A7.T21.246.246.6.1.1" class="ltx_text ltx_font_italic">(5.0)</span></sub>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 21: </span>
ELI5 full results.
</figcaption>
</figure>
<figure id="A7.T22" class="ltx_table">
<table id="A7.T22.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A7.T22.1.1.1" class="ltx_tr">
<td id="A7.T22.1.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="A7.T22.1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T22.1.1.1.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A7.T22.1.1.1.1.1.1.1" class="ltx_text ltx_font_typewriter">Read the original question and passage, and generate 3 additional claims that are supported by the passage and answer the question.</span></span>
</span>
</td>
</tr>
<tr id="A7.T22.1.2.2" class="ltx_tr">
<td id="A7.T22.1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T22.1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T22.1.2.2.1.1.1" class="ltx_p" style="width:424.9pt;"></span>
</span>
</td>
</tr>
<tr id="A7.T22.1.3.3" class="ltx_tr">
<td id="A7.T22.1.3.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T22.1.3.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T22.1.3.3.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A7.T22.1.3.3.1.1.1.1" class="ltx_text ltx_font_typewriter">Original question: What’s the difference between Shia vs. Sunni Islam?</span></span>
</span>
</td>
</tr>
<tr id="A7.T22.1.4.4" class="ltx_tr">
<td id="A7.T22.1.4.4.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T22.1.4.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T22.1.4.4.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A7.T22.1.4.4.1.1.1.1" class="ltx_text ltx_font_typewriter">Passage: The main difference between Shia and Sunni Muslim is related to ideological heritage and issues of leadership. This difference is first formed after the death of the Prophet Muhammad in 632 A.D. The ideological practice of the Sunni branch strictly follows Prophet Muhammad and his teachings, while the Shia branch follows Prophet Muhammad’s son-in-law Ali. Nowadays, Sunni and Shia are the major branches of Islam.</span></span>
</span>
</td>
</tr>
<tr id="A7.T22.1.5.5" class="ltx_tr">
<td id="A7.T22.1.5.5.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T22.1.5.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T22.1.5.5.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A7.T22.1.5.5.1.1.1.1" class="ltx_text ltx_font_typewriter">Claim 1: The major branches of Islam are Sunni and Shia.</span></span>
</span>
</td>
</tr>
<tr id="A7.T22.1.6.6" class="ltx_tr">
<td id="A7.T22.1.6.6.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T22.1.6.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T22.1.6.6.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A7.T22.1.6.6.1.1.1.1" class="ltx_text ltx_font_typewriter">Claim 2: Prophet Muhammad died in 632 A.D.</span></span>
</span>
</td>
</tr>
<tr id="A7.T22.1.7.7" class="ltx_tr">
<td id="A7.T22.1.7.7.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T22.1.7.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T22.1.7.7.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A7.T22.1.7.7.1.1.1.1" class="ltx_text ltx_font_typewriter">Claim 3: The ideological practice of the Sunni branch strictly follows Prophet Muhammad and his teachings.</span></span>
</span>
</td>
</tr>
<tr id="A7.T22.1.8.8" class="ltx_tr">
<td id="A7.T22.1.8.8.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T22.1.8.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T22.1.8.8.1.1.1" class="ltx_p" style="width:424.9pt;"></span>
</span>
</td>
</tr>
<tr id="A7.T22.1.9.9" class="ltx_tr">
<td id="A7.T22.1.9.9.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T22.1.9.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T22.1.9.9.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A7.T22.1.9.9.1.1.1.1" class="ltx_text ltx_font_typewriter">Original question: What causes Bi-polar disorder?</span></span>
</span>
</td>
</tr>
<tr id="A7.T22.1.10.10" class="ltx_tr">
<td id="A7.T22.1.10.10.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T22.1.10.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T22.1.10.10.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A7.T22.1.10.10.1.1.1.1" class="ltx_text ltx_font_typewriter">Passage: Bipolar disorder is an emotional disorder that causes extreme mood swings between excitement and depression. The spectrum of mood swing may span from days to months. We are still not certain of the exact factors that cause such disorder, but genetics is considered a major factor.</span></span>
</span>
</td>
</tr>
<tr id="A7.T22.1.11.11" class="ltx_tr">
<td id="A7.T22.1.11.11.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T22.1.11.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T22.1.11.11.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A7.T22.1.11.11.1.1.1.1" class="ltx_text ltx_font_typewriter">Claim 1: One symptom of Bi-polar disorder is extreme mood swings between excitement and depression.</span></span>
</span>
</td>
</tr>
<tr id="A7.T22.1.12.12" class="ltx_tr">
<td id="A7.T22.1.12.12.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T22.1.12.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T22.1.12.12.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A7.T22.1.12.12.1.1.1.1" class="ltx_text ltx_font_typewriter">Claim 2: Genetics could be one of the major factors that causes Bi-polar disorder.</span></span>
</span>
</td>
</tr>
<tr id="A7.T22.1.13.13" class="ltx_tr">
<td id="A7.T22.1.13.13.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T22.1.13.13.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T22.1.13.13.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A7.T22.1.13.13.1.1.1.1" class="ltx_text ltx_font_typewriter">Claim 3: The mood swing from Bi-polar disorder can last days to months.</span></span>
</span>
</td>
</tr>
<tr id="A7.T22.1.14.14" class="ltx_tr">
<td id="A7.T22.1.14.14.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T22.1.14.14.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T22.1.14.14.1.1.1" class="ltx_p" style="width:424.9pt;"></span>
</span>
</td>
</tr>
<tr id="A7.T22.1.15.15" class="ltx_tr">
<td id="A7.T22.1.15.15.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T22.1.15.15.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T22.1.15.15.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A7.T22.1.15.15.1.1.1.1" class="ltx_text ltx_font_typewriter">Original question: How do we hear differences in sound besides volume and pitch?</span></span>
</span>
</td>
</tr>
<tr id="A7.T22.1.16.16" class="ltx_tr">
<td id="A7.T22.1.16.16.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T22.1.16.16.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T22.1.16.16.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A7.T22.1.16.16.1.1.1.1" class="ltx_text ltx_font_typewriter">Passage: Pitch refers to the frequency of soundwave, and volumn refers to the amplitude of the soundwave. Besides volumn and pitch, we can also tell the difference between sounds based on the tone of sound. For example, we can differentiate the sound of different instruments based on the tone of the sounds.</span></span>
</span>
</td>
</tr>
<tr id="A7.T22.1.17.17" class="ltx_tr">
<td id="A7.T22.1.17.17.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T22.1.17.17.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T22.1.17.17.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A7.T22.1.17.17.1.1.1.1" class="ltx_text ltx_font_typewriter">Claim 1: Volume of sound is the amplitude of the soundwave.</span></span>
</span>
</td>
</tr>
<tr id="A7.T22.1.18.18" class="ltx_tr">
<td id="A7.T22.1.18.18.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T22.1.18.18.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T22.1.18.18.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A7.T22.1.18.18.1.1.1.1" class="ltx_text ltx_font_typewriter">Claim 2: Pitch is the frequency of soundwave.</span></span>
</span>
</td>
</tr>
<tr id="A7.T22.1.19.19" class="ltx_tr">
<td id="A7.T22.1.19.19.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T22.1.19.19.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T22.1.19.19.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A7.T22.1.19.19.1.1.1.1" class="ltx_text ltx_font_typewriter">Claim 3: We can use the tone of the sounds to differentiate the sound of different instruments.</span></span>
</span>
</td>
</tr>
<tr id="A7.T22.1.20.20" class="ltx_tr">
<td id="A7.T22.1.20.20.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T22.1.20.20.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T22.1.20.20.1.1.1" class="ltx_p" style="width:424.9pt;"></span>
</span>
</td>
</tr>
<tr id="A7.T22.1.21.21" class="ltx_tr">
<td id="A7.T22.1.21.21.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T22.1.21.21.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T22.1.21.21.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A7.T22.1.21.21.1.1.1.1" class="ltx_text ltx_font_typewriter">
<span id="A7.T22.1.21.21.1.1.1.1.1" class="ltx_text" style="color:#BF8040;">Original question: How are we able to discern whether a sound is coming from in front of us or behind us?</span></span></span>
</span>
</td>
</tr>
<tr id="A7.T22.1.22.22" class="ltx_tr">
<td id="A7.T22.1.22.22.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T22.1.22.22.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T22.1.22.22.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A7.T22.1.22.22.1.1.1.1" class="ltx_text ltx_font_typewriter">
<span id="A7.T22.1.22.22.1.1.1.1.1" class="ltx_text" style="color:#BF8040;">Passage: There are multiple explanations for why we can localize sounds. One explanation is that sounds travelling to the corresponding side of one’s ear will be slightly louder. Another explanation is that there is a slight difference in the hitting time to one’s left and right ear based on the sound’s direction. However, these explanation means that when a sound is exactly in front of someone or exactly behind someone, he or she can not tell the difference.</span></span></span>
</span>
</td>
</tr>
<tr id="A7.T22.1.23.23" class="ltx_tr">
<td id="A7.T22.1.23.23.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T22.1.23.23.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T22.1.23.23.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A7.T22.1.23.23.1.1.1.1" class="ltx_text ltx_font_typewriter">
<span id="A7.T22.1.23.23.1.1.1.1.1" class="ltx_text" style="color:#BF8040;">Claim 1:</span> <span id="A7.T22.1.23.23.1.1.1.1.2" class="ltx_text" style="color:#0000FF;"> We can localize sounds by recognizing that the sound travelling to the corresponding side of one’s ear will be slightly louder.</span></span></span>
</span>
</td>
</tr>
<tr id="A7.T22.1.24.24" class="ltx_tr">
<td id="A7.T22.1.24.24.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T22.1.24.24.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T22.1.24.24.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A7.T22.1.24.24.1.1.1.1" class="ltx_text ltx_font_typewriter">
<span id="A7.T22.1.24.24.1.1.1.1.1" class="ltx_text" style="color:#0000FF;">Claim 2: We can also localize sounds by recognizing the difference in hitting time to one’s left and right ear based on the sound’s direction.</span></span></span>
</span>
</td>
</tr>
<tr id="A7.T22.1.25.25" class="ltx_tr">
<td id="A7.T22.1.25.25.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A7.T22.1.25.25.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T22.1.25.25.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A7.T22.1.25.25.1.1.1.1" class="ltx_text ltx_font_typewriter">
<span id="A7.T22.1.25.25.1.1.1.1.1" class="ltx_text" style="color:#0000FF;">Claim 3: We cannot tell the difference between a sound that is exactly in front of us or exactly behind us.</span></span></span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 22: </span>
Prompt used to generate the sub-claims for ELI5 questions.
<span id="A7.T22.4.1" class="ltx_text" style="color:#0000FF;"> Blue text is model generation.</span>
<span id="A7.T22.5.2" class="ltx_text" style="color:#BF8040;"> Brown text is the ELI5 example that we want to generate sub-claims for.</span>
We construct the prompt by manually writing the sub-claims for three questions from the training set.
</figcaption>
</figure>
</section>
</section>
<section id="A8" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix H </span>Prompts</h2>

<div id="A8.p1" class="ltx_para">
<p id="A8.p1.1" class="ltx_p">We show detailed prompts used in our paper in Tables <a href="#A8.T23" title="Table 23 ‣ Appendix H Prompts ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">23</span></a>, <a href="#A8.T24" title="Table 24 ‣ Appendix H Prompts ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">24</span></a>, <a href="#A8.T25" title="Table 25 ‣ Appendix H Prompts ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">25</span></a>, <a href="#A8.T26" title="Table 26 ‣ Appendix H Prompts ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">26</span></a>, <a href="#A8.T27" title="Table 27 ‣ Appendix H Prompts ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">27</span></a>, <a href="#A8.T28" title="Table 28 ‣ Appendix H Prompts ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">28</span></a>, and <a href="#A8.T29" title="Table 29 ‣ Appendix H Prompts ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">29</span></a>.</p>
</div>
<figure id="A8.T23" class="ltx_table">
<table id="A8.T23.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A8.T23.1.1.1" class="ltx_tr">
<td id="A8.T23.1.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb ltx_border_tt">
<span id="A8.T23.1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T23.1.1.1.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T23.1.1.1.1.1.1.1" class="ltx_text ltx_font_typewriter">Instruction: Write an accurate, engaging, and concise answer for the given question using only the provided search results (some of which might be irrelevant) and cite them properly. Use an unbiased and journalistic tone. Always cite for any factual claim. When citing several search results, use [1][2][3]. Cite at least one document and at most three documents in each sentence. If multiple documents support the sentence, only cite a minimum sufficient subset of the documents.</span></span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 23: </span>
Instruction for <span id="A8.T23.3.1" class="ltx_text ltx_font_smallcaps">Vanilla</span>.
</figcaption>
</figure>
<figure id="A8.T24" class="ltx_table">
<table id="A8.T24.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A8.T24.1.1.1" class="ltx_tr">
<td id="A8.T24.1.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb ltx_border_tt">
<span id="A8.T24.1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T24.1.1.1.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T24.1.1.1.1.1.1.1" class="ltx_text ltx_font_typewriter">Instruction: Write a high-quality answer for the given question using only the provided search results and cite them properly using [1][2][3].</span></span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 24: </span>
Short instruction for <span id="A8.T24.3.1" class="ltx_text ltx_font_smallcaps">Vanilla</span>.
</figcaption>
</figure>
<figure id="A8.T25" class="ltx_table">
<table id="A8.T25.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A8.T25.1.1.1" class="ltx_tr">
<td id="A8.T25.1.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="A8.T25.1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T25.1.1.1.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T25.1.1.1.1.1.1.1" class="ltx_text ltx_font_typewriter">Summarize the following document within 50 words with the question of interest "{QUESTION}" Return "irrelevant" if the document is irrelevant to the question. Try to keep all the important dates, numbers, and names.</span></span>
</span>
</td>
</tr>
<tr id="A8.T25.1.2.2" class="ltx_tr">
<td id="A8.T25.1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A8.T25.1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T25.1.2.2.1.1.1" class="ltx_p" style="width:411.9pt;"></span>
</span>
</td>
</tr>
<tr id="A8.T25.1.3.3" class="ltx_tr">
<td id="A8.T25.1.3.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A8.T25.1.3.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T25.1.3.3.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T25.1.3.3.1.1.1.1" class="ltx_text ltx_font_typewriter">Title: {TITLE}</span></span>
</span>
</td>
</tr>
<tr id="A8.T25.1.4.4" class="ltx_tr">
<td id="A8.T25.1.4.4.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A8.T25.1.4.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T25.1.4.4.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T25.1.4.4.1.1.1.1" class="ltx_text ltx_font_typewriter">Text: {TEXT}</span></span>
</span>
</td>
</tr>
<tr id="A8.T25.1.5.5" class="ltx_tr">
<td id="A8.T25.1.5.5.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A8.T25.1.5.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T25.1.5.5.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T25.1.5.5.1.1.1.1" class="ltx_text ltx_font_typewriter">Summary:</span></span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 25: </span>
Prompts for <span id="A8.T25.3.1" class="ltx_text ltx_font_smallcaps">Summ</span>.
</figcaption>
</figure>
<figure id="A8.T26" class="ltx_table">
<table id="A8.T26.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A8.T26.1.1.1" class="ltx_tr">
<td id="A8.T26.1.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="A8.T26.1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T26.1.1.1.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T26.1.1.1.1.1.1.1" class="ltx_text ltx_font_typewriter">Given the follow passage and the question "{QUESTION}", extract a useful span from the passage that can answer the question. Resolve all the coreference issues to make the extracted span understandable standalone. If the passage is not helpful for answering the question, return "irrelevant".</span></span>
</span>
</td>
</tr>
<tr id="A8.T26.1.2.2" class="ltx_tr">
<td id="A8.T26.1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A8.T26.1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T26.1.2.2.1.1.1" class="ltx_p" style="width:411.9pt;"></span>
</span>
</td>
</tr>
<tr id="A8.T26.1.3.3" class="ltx_tr">
<td id="A8.T26.1.3.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A8.T26.1.3.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T26.1.3.3.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T26.1.3.3.1.1.1.1" class="ltx_text ltx_font_typewriter">Title: {TITLE}</span></span>
</span>
</td>
</tr>
<tr id="A8.T26.1.4.4" class="ltx_tr">
<td id="A8.T26.1.4.4.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A8.T26.1.4.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T26.1.4.4.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T26.1.4.4.1.1.1.1" class="ltx_text ltx_font_typewriter">Text: {TEXT}</span></span>
</span>
</td>
</tr>
<tr id="A8.T26.1.5.5" class="ltx_tr">
<td id="A8.T26.1.5.5.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A8.T26.1.5.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T26.1.5.5.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T26.1.5.5.1.1.1.1" class="ltx_text ltx_font_typewriter">Extracted span:</span></span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 26: </span>
Prompts for <span id="A8.T26.3.1" class="ltx_text ltx_font_smallcaps">Snippet</span>.
</figcaption>
</figure>
<figure id="A8.T27" class="ltx_table">
<table id="A8.T27.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A8.T27.1.1.1" class="ltx_tr">
<td id="A8.T27.1.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="A8.T27.1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T27.1.1.1.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T27.1.1.1.1.1.1.1" class="ltx_text ltx_font_typewriter">Instruction: Write an accurate, engaging, and concise answer for the given question using only the provided search results and cite them properly. Use an unbiased and journalistic tone. Always cite for any factual claim.</span></span>
</span>
</td>
</tr>
<tr id="A8.T27.1.2.2" class="ltx_tr">
<td id="A8.T27.1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A8.T27.1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T27.1.2.2.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T27.1.2.2.1.1.1.1" class="ltx_text ltx_font_typewriter">You are provided summaries/snippets of the search results. You can use "Check: Document [1][2]" to check the corresponding full documents (you should only check relevant documents and you can at most check 3 documents at a time) and use "Output:" to output a sentence in the answer. In the answer, cite properly by using [1][2][3]. Cite at least one document and at most three documents in each sentence. If multiple documents support the sentence, only cite a minimum sufficient subset of the documents. Use "End" to end the generation.</span></span>
</span>
</td>
</tr>
<tr id="A8.T27.1.3.3" class="ltx_tr">
<td id="A8.T27.1.3.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A8.T27.1.3.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T27.1.3.3.1.1.1" class="ltx_p" style="width:411.9pt;"></span>
</span>
</td>
</tr>
<tr id="A8.T27.1.4.4" class="ltx_tr">
<td id="A8.T27.1.4.4.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A8.T27.1.4.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T27.1.4.4.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T27.1.4.4.1.1.1.1" class="ltx_text ltx_font_typewriter">
<span id="A8.T27.1.4.4.1.1.1.1.1" class="ltx_text" style="color:#BF8040;">&lt;Retrieve for question ‘‘...’’&gt;</span></span></span>
</span>
</td>
</tr>
<tr id="A8.T27.1.5.5" class="ltx_tr">
<td id="A8.T27.1.5.5.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A8.T27.1.5.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T27.1.5.5.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T27.1.5.5.1.1.1.1" class="ltx_text ltx_font_typewriter" style="color:#BF8040;">&lt;Get summaries/snippets for the passages and delete those that are ‘‘irrelevant’’&gt;</span></span>
</span>
</td>
</tr>
<tr id="A8.T27.1.6.6" class="ltx_tr">
<td id="A8.T27.1.6.6.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A8.T27.1.6.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T27.1.6.6.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T27.1.6.6.1.1.1.1" class="ltx_text ltx_font_typewriter">Document [1](Title: ...) {SUMMARY OR SNIPPET}</span></span>
</span>
</td>
</tr>
<tr id="A8.T27.1.7.7" class="ltx_tr">
<td id="A8.T27.1.7.7.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A8.T27.1.7.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T27.1.7.7.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T27.1.7.7.1.1.1.1" class="ltx_text ltx_font_typewriter">...</span></span>
</span>
</td>
</tr>
<tr id="A8.T27.1.8.8" class="ltx_tr">
<td id="A8.T27.1.8.8.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A8.T27.1.8.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T27.1.8.8.1.1.1" class="ltx_p" style="width:411.9pt;"></span>
</span>
</td>
</tr>
<tr id="A8.T27.1.9.9" class="ltx_tr">
<td id="A8.T27.1.9.9.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A8.T27.1.9.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T27.1.9.9.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T27.1.9.9.1.1.1.1" class="ltx_text ltx_font_typewriter">
Question: When did US break away from England?</span></span>
</span>
</td>
</tr>
<tr id="A8.T27.1.10.10" class="ltx_tr">
<td id="A8.T27.1.10.10.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A8.T27.1.10.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T27.1.10.10.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T27.1.10.10.1.1.1.1" class="ltx_text ltx_font_typewriter" style="color:#0000FF;">Check: Document [1][2]</span></span>
</span>
</td>
</tr>
<tr id="A8.T27.1.11.11" class="ltx_tr">
<td id="A8.T27.1.11.11.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A8.T27.1.11.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T27.1.11.11.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T27.1.11.11.1.1.1.1" class="ltx_text ltx_font_typewriter">Document [1] {FULL TEXT}</span></span>
</span>
</td>
</tr>
<tr id="A8.T27.1.12.12" class="ltx_tr">
<td id="A8.T27.1.12.12.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A8.T27.1.12.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T27.1.12.12.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T27.1.12.12.1.1.1.1" class="ltx_text ltx_font_typewriter">Document [2] {FULL TEXT}</span></span>
</span>
</td>
</tr>
<tr id="A8.T27.1.13.13" class="ltx_tr">
<td id="A8.T27.1.13.13.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A8.T27.1.13.13.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T27.1.13.13.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T27.1.13.13.1.1.1.1" class="ltx_text ltx_font_typewriter" style="color:#0000FF;">Output: The United States ... <span id="A8.T27.1.13.13.1.1.1.1.1" class="ltx_text ltx_font_serif ltx_font_bold ltx_framed ltx_framed_underline">[1]</span> ... <span id="A8.T27.1.13.13.1.1.1.1.2" class="ltx_text ltx_font_serif ltx_font_bold ltx_framed ltx_framed_underline">[2]</span>.</span></span>
</span>
</td>
</tr>
<tr id="A8.T27.1.14.14" class="ltx_tr">
<td id="A8.T27.1.14.14.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A8.T27.1.14.14.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T27.1.14.14.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T27.1.14.14.1.1.1.1" class="ltx_text ltx_font_typewriter" style="color:#BF8040;">&lt;Remove the full text of [1][2] from context&gt;</span></span>
</span>
</td>
</tr>
<tr id="A8.T27.1.15.15" class="ltx_tr">
<td id="A8.T27.1.15.15.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A8.T27.1.15.15.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T27.1.15.15.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T27.1.15.15.1.1.1.1" class="ltx_text ltx_font_typewriter" style="color:#0000FF;">Check: Document [3]</span></span>
</span>
</td>
</tr>
<tr id="A8.T27.1.16.16" class="ltx_tr">
<td id="A8.T27.1.16.16.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A8.T27.1.16.16.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T27.1.16.16.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T27.1.16.16.1.1.1.1" class="ltx_text ltx_font_typewriter">Document [3] {FULL TEXT}</span></span>
</span>
</td>
</tr>
<tr id="A8.T27.1.17.17" class="ltx_tr">
<td id="A8.T27.1.17.17.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A8.T27.1.17.17.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T27.1.17.17.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T27.1.17.17.1.1.1.1" class="ltx_text ltx_font_typewriter" style="color:#0000FF;">Output: The Treaty of Paris ... <span id="A8.T27.1.17.17.1.1.1.1.1" class="ltx_text ltx_font_serif ltx_font_bold ltx_framed ltx_framed_underline">[3]</span>.</span></span>
</span>
</td>
</tr>
<tr id="A8.T27.1.18.18" class="ltx_tr">
<td id="A8.T27.1.18.18.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A8.T27.1.18.18.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T27.1.18.18.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T27.1.18.18.1.1.1.1" class="ltx_text ltx_font_typewriter" style="color:#BF8040;">&lt;Remove the full text of [3] from context&gt;</span></span>
</span>
</td>
</tr>
<tr id="A8.T27.1.19.19" class="ltx_tr">
<td id="A8.T27.1.19.19.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A8.T27.1.19.19.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T27.1.19.19.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T27.1.19.19.1.1.1.1" class="ltx_text ltx_font_typewriter" style="color:#0000FF;">End.</span></span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 27: </span>
An example for <span id="A8.T27.3.1" class="ltx_text ltx_font_smallcaps">Interact</span>.
</figcaption>
</figure>
<figure id="A8.T28" class="ltx_table">
<table id="A8.T28.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A8.T28.1.1.1" class="ltx_tr">
<td id="A8.T28.1.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="A8.T28.1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T28.1.1.1.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T28.1.1.1.1.1.1.1" class="ltx_text ltx_font_typewriter">Instruction: Write an accurate, engaging, and concise answer for the given question using only the provided search results and cite them properly. Use an unbiased and journalistic tone. Always cite for any factual claim.</span></span>
</span>
</td>
</tr>
<tr id="A8.T28.1.2.2" class="ltx_tr">
<td id="A8.T28.1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A8.T28.1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T28.1.2.2.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T28.1.2.2.1.1.1.1" class="ltx_text ltx_font_typewriter">You can use "Search: key words" to check the most relevant document’s full text and use "Output:" to output a sentence in the answer. In the answer, cite properly by using [1][2][3]. Cite at least one document and at most three documents in each sentence. If multiple documents support the sentence, only cite a minimum sufficient subset of the documents. Use "End" to end the generation.</span></span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 28: </span>
Instruction for <span id="A8.T28.3.1" class="ltx_text ltx_font_smallcaps">InlineSearch</span>.
</figcaption>
</figure>
<figure id="A8.T29" class="ltx_table">
<table id="A8.T29.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A8.T29.1.1.1" class="ltx_tr">
<td id="A8.T29.1.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb ltx_border_tt">
<span id="A8.T29.1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T29.1.1.1.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A8.T29.1.1.1.1.1.1.1" class="ltx_text ltx_font_typewriter">Instruction: Write an accurate, engaging, and concise answer for the given question. Use an unbiased and journalistic tone.</span></span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 29: </span>
Instruction for <span id="A8.T29.3.1" class="ltx_text ltx_font_smallcaps">ClosedBook</span>.
</figcaption>
</figure>
</section>
<section id="A9" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix I </span>Examples</h2>

<div id="A9.p1" class="ltx_para">
<p id="A9.p1.1" class="ltx_p">In Tables <a href="#A9.T30" title="Table 30 ‣ Appendix I Examples ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">30</span></a> and <a href="#A9.T31" title="Table 31 ‣ Appendix I Examples ‣ ALCE: An Automatic Benchmark for Large Language Model Generations with Citations" class="ltx_ref"><span class="ltx_text ltx_ref_tag">31</span></a> we show some examples of questions and model generated outputs.</p>
</div>
<figure id="A9.T30" class="ltx_table">
<table id="A9.T30.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A9.T30.1.1.1" class="ltx_tr">
<td id="A9.T30.1.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="A9.T30.1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T30.1.1.1.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A9.T30.1.1.1.1.1.1.1" class="ltx_text ltx_font_typewriter">Instruction: Write an accurate, engaging, and concise answer for ...</span></span>
</span>
</td>
</tr>
<tr id="A9.T30.1.2.2" class="ltx_tr">
<td id="A9.T30.1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A9.T30.1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T30.1.2.2.1.1.1" class="ltx_p" style="width:411.9pt;"></span>
</span>
</td>
</tr>
<tr id="A9.T30.1.3.3" class="ltx_tr">
<td id="A9.T30.1.3.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A9.T30.1.3.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T30.1.3.3.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A9.T30.1.3.3.1.1.1.1" class="ltx_text ltx_font_typewriter">
Document [1](Title: How to Treat and Prevent Food Poisoning - MsPrepper) just a typical gastro upset. Salmonella is most commonly caused by eating undercooked or raw foods like eggs or meat. You know how your mom always warned you not to eat raw cookie dough? This is why. Most people do eat cookie dough and they are fine, but salmonella is a risk. If you do contract salmonella, you could start to feel bad within in a couple of hours after eating contaminated food, and sometimes it could take a day or two. Common symptoms are nausea and vomiting, loose stools (sometimes bloody), flu like symptoms, and stomach cramps. To treat</span></span>
</span>
</td>
</tr>
<tr id="A9.T30.1.4.4" class="ltx_tr">
<td id="A9.T30.1.4.4.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A9.T30.1.4.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T30.1.4.4.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A9.T30.1.4.4.1.1.1.1" class="ltx_text ltx_font_typewriter">Document [2](Title: FDA Issues Warning About Eating Raw Cookie Dough, But Not For Salmonella Risks) FDA Issues Warning About Eating Raw Cookie Dough, But Not For Salmonella Risks Used to licking the spoon or placating yourself with full-on chunks of raw cookie dough? The Food and Drug Administration issued a warning on Tuesday that strongly advises against continuing the habit. The agency asserted that consuming raw batter of any kind, whether for bread, cookies or pizza, could make a person sick. While you may have been warned in the past against eating raw dough due to the risk of contracting salmonella from raw eggs, the FDA is citing raw flour as the culprit for a</span></span>
</span>
</td>
</tr>
<tr id="A9.T30.1.5.5" class="ltx_tr">
<td id="A9.T30.1.5.5.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A9.T30.1.5.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T30.1.5.5.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A9.T30.1.5.5.1.1.1.1" class="ltx_text ltx_font_typewriter">Document [3](Title: It’s Probably OK to Eat Raw Cookie Dough — As Long As You’re Smart About It - The Crux - Very Top Secret Information) First, when most people think about health risks and cookie dough, they think about raw egg. Eggs can be contaminated with salmonella bacteria, and food safety recommendations encourage people to cook eggs until the white and yolk are firm in order to kill any bacteria. However, anyone making cookies can do things to reduce this risk by using pasteurized egg products. When my kids and I make cookie dough, we never use regular eggs. Instead, we use shell eggs that have been pasteurized to kill any harmful bacteria without actually cooking the egg itself. (A great public health innovation, if</span></span>
</span>
</td>
</tr>
<tr id="A9.T30.1.6.6" class="ltx_tr">
<td id="A9.T30.1.6.6.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A9.T30.1.6.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T30.1.6.6.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A9.T30.1.6.6.1.1.1.1" class="ltx_text ltx_font_typewriter">Document [4](Title: How Dangerous Is It to Eat Raw Cookie Dough? | Men’s Health) Can Eating Raw Cookie Dough Really Make You Sick? Scientists reveal the truth about this supposedly dangerous delicacy By Katherine Dempsey There are few things more tempting in life than eyeing a bowl of cookie dough and deciding whether or not to stick your finger in for a scoop. It’s a bit like playing Russian roulette. You could get lucky and enjoy the delicious dough without conseqence, but there’s always the risk of getting serously sick with a food-borne illness. That’s because multiple ingredients within the dough could be contaminated by pathogens such as Salmonella and E. coli, says Soohyoun</span></span>
</span>
</td>
</tr>
<tr id="A9.T30.1.7.7" class="ltx_tr">
<td id="A9.T30.1.7.7.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A9.T30.1.7.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T30.1.7.7.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A9.T30.1.7.7.1.1.1.1" class="ltx_text ltx_font_typewriter">Document [5](Title: How Dangerous Is It to Eat Raw Cookie Dough? | Men’s Health) is usually pasteurized, so it’s not likely that the egg would make you sick. However, other ingredients in the dough could potentially harbor pathogens. Experts say that a prime suspect in a 2009 E. coli outbreak linked to prepackaged cookie dough was actually the flour. Nuts and chocolate have also been linked to Salmonella outbreaks. Bottom line: You’re better off skipping raw cookie dough. At least you’ll have killer cookies as a consolation. The article How Bad Is It To Eat Raw Cookie Dough? originally ran on Prevention.com The 9 Foods Most Likely to Make You Sick Mmm, Just Don’t</span></span>
</span>
</td>
</tr>
<tr id="A9.T30.1.8.8" class="ltx_tr">
<td id="A9.T30.1.8.8.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A9.T30.1.8.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T30.1.8.8.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A9.T30.1.8.8.1.1.1.1" class="ltx_text ltx_font_typewriter">...</span></span>
</span>
</td>
</tr>
<tr id="A9.T30.1.9.9" class="ltx_tr">
<td id="A9.T30.1.9.9.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A9.T30.1.9.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T30.1.9.9.1.1.1" class="ltx_p" style="width:411.9pt;"></span>
</span>
</td>
</tr>
<tr id="A9.T30.1.10.10" class="ltx_tr">
<td id="A9.T30.1.10.10.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A9.T30.1.10.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T30.1.10.10.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A9.T30.1.10.10.1.1.1.1" class="ltx_text ltx_font_typewriter">
Question: Why is it bad to eat cookie dough for risk of salmonella but things like Cookie Dough Bites are ok?</span></span>
</span>
</td>
</tr>
<tr id="A9.T30.1.11.11" class="ltx_tr">
<td id="A9.T30.1.11.11.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A9.T30.1.11.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T30.1.11.11.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A9.T30.1.11.11.1.1.1.1" class="ltx_text ltx_font_typewriter">Answer: <span id="A9.T30.1.11.11.1.1.1.1.1" class="ltx_text" style="color:#0000FF;"> Raw cookie dough is not recommended to be eaten due to the risk of salmonella <span id="A9.T30.1.11.11.1.1.1.1.1.1" class="ltx_text ltx_font_serif ltx_font_bold ltx_framed ltx_framed_underline">[1][2]</span>. Eating raw flour is also a risk for food poisoning <span id="A9.T30.1.11.11.1.1.1.1.1.2" class="ltx_text ltx_font_serif ltx_font_bold ltx_framed ltx_framed_underline">[2]</span>. The egg in cookie dough is usually pasteurized, so it’s not likely that the egg would make you sick, but other ingredients in the dough could potentially harbor pathogens <span id="A9.T30.1.11.11.1.1.1.1.1.3" class="ltx_text ltx_font_serif ltx_font_bold ltx_framed ltx_framed_underline">[4][5]</span>. However, prepackaged cookie dough like Cookie Dough Bites is safe to eat because the dough is made with pasteurized egg products and heat-treated flour <span id="A9.T30.1.11.11.1.1.1.1.1.4" class="ltx_text ltx_font_serif ltx_font_bold ltx_framed ltx_framed_underline">[2][3]</span>.</span>.</span></span>
</span>
</td>
</tr>
<tr id="A9.T30.1.12.12" class="ltx_tr">
<td id="A9.T30.1.12.12.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A9.T30.1.12.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T30.1.12.12.1.1.1" class="ltx_p" style="width:411.9pt;"></span>
</span>
</td>
</tr>
<tr id="A9.T30.1.13.13" class="ltx_tr">
<td id="A9.T30.1.13.13.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A9.T30.1.13.13.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T30.1.13.13.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A9.T30.1.13.13.1.1.1.1" class="ltx_text ltx_font_typewriter" style="color:#FF8000;">Ground truth claims:</span></span>
</span>
</td>
</tr>
<tr id="A9.T30.1.14.14" class="ltx_tr">
<td id="A9.T30.1.14.14.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A9.T30.1.14.14.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T30.1.14.14.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A9.T30.1.14.14.1.1.1.1" class="ltx_text ltx_font_typewriter" style="color:#FF8000;">1. Cookie Dough Bites are safe to eat since they do not contain eggs.</span></span>
</span>
</td>
</tr>
<tr id="A9.T30.1.15.15" class="ltx_tr">
<td id="A9.T30.1.15.15.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A9.T30.1.15.15.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T30.1.15.15.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A9.T30.1.15.15.1.1.1.1" class="ltx_text ltx_font_typewriter" style="color:#FF8000;">2. Salmonella is a bacteria found in eggs.</span></span>
</span>
</td>
</tr>
<tr id="A9.T30.1.16.16" class="ltx_tr">
<td id="A9.T30.1.16.16.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A9.T30.1.16.16.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T30.1.16.16.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A9.T30.1.16.16.1.1.1.1" class="ltx_text ltx_font_typewriter" style="color:#FF8000;">3. Eating raw cookie dough carries the risk of salmonella infection.</span></span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 30: </span>
ELI5 example 1.
</figcaption>
</figure>
<figure id="A9.T31" class="ltx_table">
<table id="A9.T31.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A9.T31.1.1.1" class="ltx_tr">
<td id="A9.T31.1.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="A9.T31.1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T31.1.1.1.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A9.T31.1.1.1.1.1.1.1" class="ltx_text ltx_font_typewriter">Instruction: Write an accurate, engaging, and concise answer for ...</span></span>
</span>
</td>
</tr>
<tr id="A9.T31.1.2.2" class="ltx_tr">
<td id="A9.T31.1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A9.T31.1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T31.1.2.2.1.1.1" class="ltx_p" style="width:424.9pt;"></span>
</span>
</td>
</tr>
<tr id="A9.T31.1.3.3" class="ltx_tr">
<td id="A9.T31.1.3.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A9.T31.1.3.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T31.1.3.3.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A9.T31.1.3.3.1.1.1.1" class="ltx_text ltx_font_typewriter">
Document [1](Title: Is Snapchat really worth $19 billion? - CSMonitor.com) reporting that the Los Angeles-based company is aiming to raise $500 million at a valuation of $16 billion to $19 billion, making it the third most highly valued tech start-up backed by venture capitalists. The Chinese handset maker Xiaomi is valued at $45 billion, while Uber is estimated to be valued at about $40 billion, according to data from CB Insights. Read MoreVC investment hits $86B thanks to Uber, Xiaomi Snapchat was valued at $10 billion in August, according to a Dow Jones report. Some of its investors from previous rounds include Benchmark, Lightspeed Venture Partners and Kleiner Perkins Caufield</span></span>
</span>
</td>
</tr>
<tr id="A9.T31.1.4.4" class="ltx_tr">
<td id="A9.T31.1.4.4.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A9.T31.1.4.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T31.1.4.4.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A9.T31.1.4.4.1.1.1.1" class="ltx_text ltx_font_typewriter">Document [2](Title: What Are Venture Capital Investments? – DollarsAndSense.my) Ever wondered how highly valued technology giants like Google and Facebook were able to grow so fast and pay their employees so well in such a short amount of time, or how still growing start-ups like Uber are able to lose 1.2 billion US dollars in just the first half of this year alone and still command a valuation upwards of 50 billion US dollars? The answer lies with a special category of investment activity known as venture capital. Venture capitalists are professional investors who invest in a number of highly scalable high-risk technology ventures hoping to make a multi-fold</span></span>
</span>
</td>
</tr>
<tr id="A9.T31.1.5.5" class="ltx_tr">
<td id="A9.T31.1.5.5.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A9.T31.1.5.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T31.1.5.5.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A9.T31.1.5.5.1.1.1.1" class="ltx_text ltx_font_typewriter">Document [3](Title: Opinion | What Dara Khosrowshahi Must Do to Save Uber - The New York Times) at a discount. These are troubling signs. Every start-up must one day fulfill the market’s demand that it turn a profit, but Uber has never figured out how to do that. While ride sharing in some form will probably survive, it’s more likely that without some drastic changes, Uber won’t be around in three to five years. Mr. Khosrowshahi must avoid the mistakes of his predecessor by accepting that “pivots” (Silicon Valley-speak for the desperate changes troubled companies make to reassure their venture capitalist funders) are not the answer. None of the pivots Mr. Kalanick tried — like on-demand delivery</span></span>
</span>
</td>
</tr>
<tr id="A9.T31.1.6.6" class="ltx_tr">
<td id="A9.T31.1.6.6.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A9.T31.1.6.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T31.1.6.6.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A9.T31.1.6.6.1.1.1.1" class="ltx_text ltx_font_typewriter">Document [4](Title: Snapchat raising funding round at $19 billion valuation: Report) Snapchat raising funding round at $19 billion valuation: Report The mobile messaging app Snapchat is said to be seeking funding at a $19 billion valuation, according to a report, citing a person with knowledge of the matter. Bloomberg is reporting that the Los Angeles-based company is aiming to raise $500 million at a valuation of $16 billion to $19 billion, making it the third most highly valued tech start-up backed by venture capitalists. The Chinese handset maker Xiaomi is valued at $45 billion, while Uber is estimated to be valued at about $40 billion, according to data from CB Insights.</span></span>
</span>
</td>
</tr>
<tr id="A9.T31.1.7.7" class="ltx_tr">
<td id="A9.T31.1.7.7.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A9.T31.1.7.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T31.1.7.7.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A9.T31.1.7.7.1.1.1.1" class="ltx_text ltx_font_typewriter">Document [5](Title: Unicorns And Wall Street | MoneyTips) not start showing profits soon. Even Uber, with its massive $62-billion-plus valuation, will test investor’s patience if it does not start showing signs of a profit in the next few years. Cracks are starting to show in terms of pre-IPO devaluations. Dropbox and Snapchat have seen a nearly one-quarter drop in their pre-IPO valuations, while Square’s pre-IPO value has dropped by approximately one-third. Larger investors and venture capitalists are sending a message to the unicorns: Outline a solid path to profits and start producing them soon, or expect your valuation to be slashed. Diving ahead with an IPO is not</span></span>
</span>
</td>
</tr>
<tr id="A9.T31.1.8.8" class="ltx_tr">
<td id="A9.T31.1.8.8.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A9.T31.1.8.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T31.1.8.8.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A9.T31.1.8.8.1.1.1.1" class="ltx_text ltx_font_typewriter">...</span></span>
</span>
</td>
</tr>
<tr id="A9.T31.1.9.9" class="ltx_tr">
<td id="A9.T31.1.9.9.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A9.T31.1.9.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T31.1.9.9.1.1.1" class="ltx_p" style="width:424.9pt;"></span>
</span>
</td>
</tr>
<tr id="A9.T31.1.10.10" class="ltx_tr">
<td id="A9.T31.1.10.10.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A9.T31.1.10.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T31.1.10.10.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A9.T31.1.10.10.1.1.1.1" class="ltx_text ltx_font_typewriter">
Question: How are firms like snapchat, uber etc valued so highly while still not making a profit? Do venture capitalists not expect some form of repayment within a number of years?</span></span>
</span>
</td>
</tr>
<tr id="A9.T31.1.11.11" class="ltx_tr">
<td id="A9.T31.1.11.11.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A9.T31.1.11.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T31.1.11.11.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A9.T31.1.11.11.1.1.1.1" class="ltx_text ltx_font_typewriter">Answer: <span id="A9.T31.1.11.11.1.1.1.1.1" class="ltx_text" style="color:#0000FF;"> Venture capitalists invest in a number of highly scalable high-risk technology ventures hoping to make a multi-fold return on their investment in a short amount of time <span id="A9.T31.1.11.11.1.1.1.1.1.1" class="ltx_text ltx_font_serif ltx_font_bold ltx_framed ltx_framed_underline">[2]</span>. Highly valued companies like Snapchat, Uber, and Xiaomi are able to grow so fast and command high valuations because they have the potential to generate significant revenue in the future <span id="A9.T31.1.11.11.1.1.1.1.1.2" class="ltx_text ltx_font_serif ltx_font_bold ltx_framed ltx_framed_underline">[2][4]</span>. Venture capitalists expect a return on their investment within a number of years, but they also invest in the long-term potential of the company <span id="A9.T31.1.11.11.1.1.1.1.1.3" class="ltx_text ltx_font_serif ltx_font_bold ltx_framed ltx_framed_underline">[2]</span>. It is important to note that every start-up must eventually turn a profit, but some start-ups like Uber have not yet figured out how to do that <span id="A9.T31.1.11.11.1.1.1.1.1.4" class="ltx_text ltx_font_serif ltx_font_bold ltx_framed ltx_framed_underline">[3][5]</span>.</span></span></span>
</span>
</td>
</tr>
<tr id="A9.T31.1.12.12" class="ltx_tr">
<td id="A9.T31.1.12.12.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A9.T31.1.12.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T31.1.12.12.1.1.1" class="ltx_p" style="width:424.9pt;"></span>
</span>
</td>
</tr>
<tr id="A9.T31.1.13.13" class="ltx_tr">
<td id="A9.T31.1.13.13.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A9.T31.1.13.13.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T31.1.13.13.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A9.T31.1.13.13.1.1.1.1" class="ltx_text ltx_font_typewriter" style="color:#FF8000;">Ground truth claims:</span></span>
</span>
</td>
</tr>
<tr id="A9.T31.1.14.14" class="ltx_tr">
<td id="A9.T31.1.14.14.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A9.T31.1.14.14.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T31.1.14.14.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A9.T31.1.14.14.1.1.1.1" class="ltx_text ltx_font_typewriter" style="color:#FF8000;">1. Firms like Snapchat and Uber need to establish their brand and amass users before introducing ads.</span></span>
</span>
</td>
</tr>
<tr id="A9.T31.1.15.15" class="ltx_tr">
<td id="A9.T31.1.15.15.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A9.T31.1.15.15.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T31.1.15.15.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A9.T31.1.15.15.1.1.1.1" class="ltx_text ltx_font_typewriter" style="color:#FF8000;">2. Introducing ads too early can deter potential users.</span></span>
</span>
</td>
</tr>
<tr id="A9.T31.1.16.16" class="ltx_tr">
<td id="A9.T31.1.16.16.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A9.T31.1.16.16.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T31.1.16.16.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A9.T31.1.16.16.1.1.1.1" class="ltx_text ltx_font_typewriter" style="color:#FF8000;">3. Uber is reinvesting a lot of money to make their service better.</span></span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 31: </span>
ELI5 example 2.
</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2305.14626" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2305.14627" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2305.14627">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2305.14627" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2305.14628" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Thu Feb 29 05:56:16 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2305.14627
Large language models (LLMs)  have emerged as a widely-used tool for information seeking, but their generated outputs are prone to hallucination. In this work, our aim is to allow LLMs to generate text withcitations,
improving their factual correctness and verifiability.
Existing work mainly relies on commercial search engines and human evaluation, making it challenging to reproduce and compare different modeling approaches.
We proposeALCE, the first benchmark forAutomaticLLMs’CitationEvaluation.
ALCE collects a diverse set of questions and retrieval corpora
and requires building end-to-end systems to retrieve supporting evidence and generate answers with citations.
We develop automatic metrics along three dimensions—fluency, correctness, and citation quality—and
demonstrate their strong correlation with human judgements.
Our experiments with state-of-the-art LLMs and novel prompting strategies show that current systems have considerable room for improvement—For example, on the ELI5 dataset,
even the best models lack complete citation support 50% of the time.
Our analyses further highlight promising future directions, including
developing better retrievers,
advancing long-context LLMs,
and improving the ability to synthesize information from multiple sources.111Our code and data are available athttps://github.com/princeton-nlp/ALCE.
abstract
ALCE: An Automatic Benchmark forLarge Language Model Generations withCitations
Abstract
2305.14627
1IntroductionLarge language models (LLMs;)  have gained increasing popularity as a tool for information seeking.
While they generate engaging and coherent responses,
their outputs are prone to hallucination and often contain factually incorrect information.
This makes it harder for users to trust and verify LLM-generated outputs without any supporting evidence.Figure 1:The task setup of ALCE. Given a question,
the system generates text while providingciting passagesfrom a large retrieval corpus. Each statement may contain multiple citations (e.g.,[1][2]) .In this work, we study a new generation paradigm for LLMs, in which
we require LLMs to providecitationsto one or a few text passages for any statement they generate (Figure1) .
Incorporating citations brings several benefits:
(1)  users can easily verify LLMs’ claims with the provided citations;
(2)  LLMs can generate text that faithfully follows cited passages, which has the promise to improve correctness and alleviate hallucination.DatasetCorpus (#passages) Question typeExampleASQAWikipedia (21M) Factoid (ambiguous) Q: When did the US break away from England?A: The US declared independence on July 2, 1776[1][2]… The Treaty of Paris was later signed on September 3, 1783[3].QAMPARIWikipedia (21M) Factoid (list) Q: Which films have Gong Li as a member of their cast?A: The Story of Qiu Ju[1], Farewell My Concubine[2], The Monkey King 2[3], Mulan[3], Saturday Fiction[3]…ELI5Sphere (899M) Why/How/WhatQ: How do student loans affect getting a mortgage?A: Student loans can affect the debt to income ratio[1], which is a key factor in determining the amount that …[2][3]Table 1:The three datasets used in our ALCE benchmark. These datasets cover a wide range of question types
and the corresponding corpora span from Wikipedia to
Web-scale document collection.Multiple commercial systems have adopted this paradigm: Bing Chat222https://www.bing.com/newand perplexity.ai333https://www.perplexity.airespond to user questions in natural language with references to Web pages.share a similar motivation, but they mainly experiment with commercial search engines and closed-source models, making their results difficult to evaluate.
Retrieval-augmented LMsincorporate retrieved passages during both training and inference,
but do not guarantee faithfulness to retrieved passages or explicitly provide citations.
Additionally,
previous studies mostly rely on human evaluation, which is expensive and difficult to reproduce.
We argue that the absence of automated evaluation
hinders the advances of such systems.We presentALCE, the first reproducible benchmark forautomaticallyevaluating LLMs’ generations with citations.
ALCE assumes a natural-language question and a retrieval corpus, and requires building end-to-end systems to retrieve relevant passages from the corpus, generate a response to the question, and cite corresponding supporting passages.
We compile three datasets that cover different types of questions and corpora—ASQA, QAMPARI, and ELI5—as shown in Table1.
Different from previous benchmarks,
ALCE evaluates long-text generation,
focusing on automatically evaluating citation quality,
and allows citingmultiplepassages for individual statements.We design automatic evaluation methods in three dimensions:fluency,correctness, andcitation quality.
Specifically,
we use MAUVEto measure fluency,
propose tailored correctness metrics for each dataset, and
adopt a natural language inference (NLI)  modelto measure citation quality.
We showcase how the three dimensions together contribute to a robust evaluation,
preventing systems from exploiting shortcuts. Additionally, we conduct human evaluation and demonstrate a strong correlation with our automatic metrics.We experiment on multiple systems with state-of-the-art LLMs and retrievers and also propose novel prompting strategies to synthesize retrieved text into text generation.
Although all systems are capable of providing fluent and coherent responses,
there remains substantial room for improvement in terms of correctness and citation quality: For example, on the ELI5 dataset,
around 50% generations of our ChatGPT and GPT-4 baselines are not fully supported by the cited passages.
Additionally, we find that
(1)  a closed-book model (generating answers without accessing any retrieved documents)  with post-hoc citing achieves good correctness but much worse citation quality;
(2)  although interactive retrieval approachesoffer more flexibility in when/what to retrieve, they do not improve the performance on this challenging benchmark;
(3)  summarizing the retrieved passages in a shorter text improves correctness but not citation quality;
(4)  reranking multiple generations boosts citation quality measured by human evaluation;
(5)  incorporating more retrieved passages in context does not help ChatGPT but improves GPT-4 performance.Our extensive analyses
highlight three major challenges of building LLMs to generate text with citations:
(1) 
the retrieval quality is crucial to the final performance and has substantial room for improvement;
(2) 
LLMs’ limited context window
restricts the number of passages they can incorporate;
(3) 
current LLMs struggle to synthesize multiple documents in context
without being distracted by irrelevant ones, although better instruction tuning brings significant improvement.
These challenges pose promising research directions for
developing better systems integrating retrieval and LLMs.
introduction
ALCE: An Automatic Benchmark forLarge Language Model Generations withCitations
1Introduction
2305.14627
2Task Setup and DatasetsOur task is formalized as follows:
Given a queryand a corpus of text passages,
the system is required to return an output, which consists ofstatements, …,,
and each statementcites a list of passages444In practice, we allow at most 3 citations for each statement as more citations usually do not help., where.
In this work,
we segment LLMs’ output into statements by sentence boundaries.555QAMPARI requires a list as the answer, and we choose each entity in the generated list as a statement.While LLMs may include sentences that do not require a citation, such as “I’m happy to help”,
we observe that almost all sentences that LLMs output provide valuable information and require citations, similar to findings in.
In this work, citations are enclosed by box brackets such as[1][2].We divide the corpusinto 100-word passages following previous works on open-domain question answering,
in contrast to
commercial systems like Bing Chat, which cite entire Web pages. We take 100-word passages because
it is easier for humans to verify, and allows for more retrieved passages to fit in LLMs’ limited context.We choose QA datasets so that
(1)  they contain factual questions, in which references are important;
(2)  questions require long-text answers that cover multiple aspects; (3)  answering the questions requires synthesizing multiple sources.
We select three datasets (Table1)  and introduce them below. See §Bfor additional statistics.ASQAis a long-form factoid dataset.
As shown in Figure1,
each question is an ambiguous question from AmbigQAthat requires multiple short answers to cover different aspects,
and the dataset provides a long-form answer that covers all short answers.
Since most questions can be answered by Wikipedia, we use the 2018-12-20 Wikipedia snapshot as.QAMPARIis a factoid QA dataset constructed from Wikipedia,
where the answer is a list of entities that are drawn from different passages.
Same as ASQA, we use the 2018-12-20 Wikipedia as the corpus.ELI5is a long-form QA dataset built on the Reddit forum “Explain Like I’m Five”.666https://www.reddit.com/r/explainlikeimfive/Most ELI5 questions are how/why/what questions that require long answers and multiple passages as evidence.
Due to the diverse topics discussed in the questions, we use
Sphere—a filtered version of Common Crawl777https://commoncrawl.org. We also filter out any Web pages from Reddit.—as the corpus. The ELI5 dataset is widely used in related work due to its challenging nature.We randomly select 1,000 examples from the development set of each dataset for ALCE.
Our benchmark primarily assesses the citation capabilities of existing LLMs and does not provide training data,
as
there are no available examples that provide supervision for citations
in these datasets.
body
ALCE: An Automatic Benchmark forLarge Language Model Generations withCitations
2Task Setup and Datasets
2305.14627
3Automatic EvaluationOur benchmark measures the following three dimensions of system responses:•Fluency: whether the model’s generated text is fluent and coherent.•Correctness: whether the answer is accurate and covers all aspects of interest.•Citation quality: whether the answer is well supported by the cited passages and no irrelevant passages are cited.In the following, we present automatic metrics for each dimension
and discuss why the combination of the three metrics provides a robust evaluation.3.1FluencyWe use MAUVEto evaluate the fluency of the output (§C) .
We deploy MAUVE for ASQA and ELI5
and omit it for QAMPARI,
as QAMPARI
only requires a list of short answers as the response
and LLMs consistently adhere to the format in our experiments.
As MAUVE is sensitive to output length and text style, and most LLMs are capable of producing fluent text,
we mainly employ it as a sanity check as long as the MAUVE scores are high enough.3.2CorrectnessOur objective is to measure the informativeness and utility of the generation to the question.propose to directly evaluateperceived utilityby humans, a process difficult to automate.
Therefore, we use correctness—whether the response is accurate compared to a ground truth answer—as a proxy.
Evaluating the correctness of long-form generation is a challenging task,
and we describe our strategy for each dataset below.
Figure2illustrates the metrics and we include additional implementation details in §C.ForASQA, we followand
calculate the recall of correct short answers
by checking whether the short answers (provided by the dataset)  are exact substrings of the generation (exact match recall; EM recall) .ForQAMPARI, we followand calculate theprecisionandrecallof
the model prediction,
by checking the exact match to the gold answer list.
We add one additional adjustment:
considering that users often want to know only a few example answers of
the question, our evaluation considers recall to be 100% if the prediction includes at least 5 correct answers (recall-5) .Unlike ASQA and QAMPARI, theELI5dataset does not provide short entity answers.use ROUGE for evaluation, which does not reflect the correctness well (; §A) .
Inspired by works in summarization evaluation,
we use InstructGPT (text-davinci-003;)  to generate three “sub-claims”.
Then we use TRUE888https://huggingface.co/google/t5_xxl_true_nli_mixture. Details in §C., a T5-11Bmodel fine-tuned on a collection of natural language inference (NLI)  datasets, to check whether the model output entails the sub-claims (claim recall) . TRUE targets factual correctness and has been used by previous works in similar context.
We demonstrate that claim recall provides a more accurate measure of correctness than existing metrics (more details in §A) .Figure 2:Evaluation of correctness (details in §3.2) .3.3Citation QualityWe evaluate citation qualities using two metrics:
(1) citation recall, which determines if the output is entirely supported by cited passages, and
(2) citation precision, which identifies any irrelevant citations. Although we prioritize citation recall as it entails a well-supported and truthful answer,
enhancing precision is crucial for better user satisfaction,
reducing the need for human review of extraneous passages.
Figure3provides an illustrated example.Figure 3:Evaluation of citation quality (details in §3.3) . We use
an NLI model to verify whether a statement is supported by its citations.We use the NLI model TRUEagain
to automatically examine whether the cited passages entail the model generation.
We conduct human evaluation (§6)  to demonstrate strong human correlation of our metric.Citation recall.We calculate the citation recall ofeach statement(0 or 1)  and average
over all statements in the model response. For each statement, its citation recall isif and only if there is at least one citation ()  and,
whereis the NLI model that outputsif the premise entails the hypothesis, andotherwise;concatenates all passages intogether (details in §C) .
The NLI evaluation is in accordance with theattributable to identified sources(AIS)  framework:implies thatis true based solely on.Citation precision.Our citation precision evaluation detects citations that are irrelevant,
but it does not require citing a minimal set.
We follow this design because human writing often cites redundant sources to enhance credibility;
human readers may also appreciate multiple citations, especially when it pertains to critical claims such as medical advice.We calculate the citation precision foreach citation(0 or 1)  and
average over all citations in the response. We first define if a citation is “irrelevant”.
Intuitively, a citationis “irrelevant” if (a) itself cannot supportand (b)  removingdoes not affect the rest of the citations to support. Formally,is “irrelevant” if and only if(a) , AND(b) .has a precision of 1 ifhas recallandis not irrelevant.
For example (Figure3) , whencites three references[2][4][5]and recall,[2]is “irrelevant” ifand.
For condition (b)  to work, we set recallas a prerequisite for precision. Note that this algorithm overlooks the scenario when one citation partially supports the statement.
We discuss the details in §E.3.4ALCE is Robust to Shortcut CasesWe showcase how the ALCE evaluation is robust to two possible shortcuts in §D:
(1)  using the top-1 retrieved passage as the response and citing itself,
and (2)  using the first two sentences of the top-1 passage.
Both cases have almost-perfect citation scores, but (1)  has low fluency due to its unnaturally long length compared to human answers, and (2)  has low correctness due to low coverage.
body
ALCE: An Automatic Benchmark forLarge Language Model Generations withCitations
3Automatic Evaluation
2305.14627
4ModelingIn this section,
we discuss three major modeling components for an ALCE system—retrieval, synthesis, and post-editing.Instruction: Write an accurate, engaging, and concise answer for ...<Retrieve for the question>Document [1](Title: American Decolonization)  ...Document [2](Title: Decolonization)  ...Document [3](Title: American Revolution)  ......Question: When did US break away from England?Answer:The United States took the first step towards gaining independence ...[1][2]. The Treaty of Paris was later signed ...[3].Table 2:An example of ourVanillamethod. Different colors representprompt,model generation, and<actions>.
We also provide two in-context demonstrations before the test example.4.1RetrievalWe explore simple, off-the-shelf retrievers.
We use dense retrievers for Wikipedia, including GTRand DPR;
we use
BM25 for Sphere. For each question,
we
retrieve the top-100 passages.4.2SynthesisWe focus on how to
prompt an LLM to
interact with the retriever, and
synthesize and cite the evidence (without fine-tuning internal parameters) .
One noteworthy challenge is that existing LLMs all have limited context window and thus
can only fit a handful of passages.Vanilla.We simply provide the model with the top-999We can fit at mostfor models with 2K window and at mostfor models with 4K context window.passages and instruct the model to cite accordingly (Table2) .
We also use in-context learningand prepend two demonstrations.
The complete instruction is in Table23.Summ/Snippet.With a 4K context window, we can at most safely fitpassages. As shown in Figure4, top-5 retrieved passages can only cover 56.8% percent of the answers in ASQA.To tackle this limitation, we propose to providesummariesorsnippetsof passages instead of the full text (summaries are abstractive but snippets are spans from passages) .
We acquire summaries and snippets by prompting ChatGPT with instructions (prompts in Table25and26) .101010We also query ChatGPT whether the passage is relevant to the question, and
filter out passages that are “irrelevant”.Then we replace all passages with summaries/snippets.
Summaries or snippets significantly reduce the passage length, allowing for more passages to fit in:
for ASQA, they reduce passage length by 6on average.ThoughSumm/Snippetallows for more retrieved passages,
they are lossy compressions.
To alleviate this problem,
we proposeInteract, an interactive prompting scheme
to allow the model to check the full text of certain passages. At each step, the model can execute one of three actions:
(1)  “Check: Document [1][2]” to check the full text of the corresponding documents;
(2)  “Output:” to output a statement of the answer;
(3)  “End.” to end the generation.
§Cprovides more details.Instruction: ...<Retrieve for question ‘‘...’’>Question: When did US break away from England?Search: Declaration of Independence<Search the query among the top-100 passages>Document [1](Title: ...)  ...Output: The United States ...[1].<Remove Document [1] from context>Search: Treaty of Paris<Search the query among the top-100 passages>Document [3](Title: ...)  ...Output: The Treaty of Paris ...[3].<Remove Document [3] from context>End.Table 3:An example ofInlineSearch.InlineSearch.The above methods all display retrieval results at the beginning.
InInlineSearch,
we allow LLMs to call “search” during the generation process.
At each step, the model can execute one of three actions:
“Search: {query}” to search among the top-100 passages111111We do not search over the entire corpus because{query}may leave out certain context in the question and searching among the already-retrieved passages gives better results.by using GTR; the “Output” and “End” actions are the same asInteract.
For each “Search” action, we display the best retrieved passage in the context.
The passage is removed after one action to save context space.
Table3shows an example.ClosedBook.We also add a simple closed-book baseline, where the model is only prompted with the instruction and the question,
without any retrieved passages provided.
Consequently, this variant does not cite any evidences.4.3Post-editingIn this section we discuss two strategies for refining the output to further improve its quality.Rerank.We randomly sampleresponses for each question, and select the best response using the automaticcitation recallscore.
we expectRerankto improve the citation quality.PostCite.For each statement, we find the best matching passage among the top-100 retrieved passages using GTR and cite it.
We combine this withClosedBookin our experiments.
body
ALCE: An Automatic Benchmark forLarge Language Model Generations withCitations
4Modeling
2305.14627
5ExperimentsFluencyCorrect.Citation(MAUVE) (EM Rec.) Rec.Prec.ChatGPTVanilla(5-psg) 66.640.473.672.5w/Rerank77.040.284.881.6Summ(10-psg) 70.043.368.961.8w/Interact69.039.173.466.5Snippet(10-psg) 69.841.465.357.4InlineSearch58.732.458.358.2ClosedBook52.738.326.726.7GPT-4(Vanillaprompting) GPT-4 (5-psg) 67.141.368.575.6GPT-4 (20-psg) 64.944.473.076.5LLaMA(Vanillaprompting) LLaMA-13B (3-psg) 68.426.910.615.4Vicuna-13B (3-psg) 82.631.951.150.1Chat-13B (5-psg) 72.435.238.439.4Chat-70B (5-psg) 88.341.562.961.3Table 4:Experiments on ASQA.
ForClosedBook, we usePostCiteto get citations.-psg: putting top-passages from the retrieval results into the context.
Chat-13B and Chat-70B refer to LLaMA-2-Chat.CorrectnessCitationRec.-5Prec.Rec.Prec.ChatGPTVanilla(5-psg) 20.820.820.520.9w/Rerank22.821.421.221.4Summ(10-psg) 23.621.223.625.7Snippet(10-psg) 24.521.522.924.9w/Interact21.923.021.923.4InlineSearch17.220.414.914.9ClosedBook32.919.810.010.0GPT-4(Vanillaprompting) GPT-4 (5-psg) 22.225.025.927.0GPT-4 (20-psg) 29.626.227.428.5LLaMA(Vanillaprompting) LLaMA-13B (3-psg) 9.79.16.77.1Vicuna-13B (5-psg) 14.015.912.513.4Chat-13B (5-psg) 21.118.29.69.7Chat-70B (5-psg) 21.818.415.115.6Table 5:Experiments on QAMPARI.
“Rec.-5”: we set the recall to be 100% if the prediction includes at least 5 correct answers.FluencyCorrect.Citation(MAUVE) (Claim) Rec.Prec.ChatGPTVanilla(5-psg) 57.212.051.150.0w/Rerank56.111.469.367.8Summ(10-psg) 40.312.551.548.2Snippet(10-psg) 62.914.350.445.0w/Interact68.013.347.845.0InlineSearch49.713.445.643.7ClosedBook32.618.615.515.5GPT-4(Vanillaprompting) GPT-4 (5-psg) 38.414.244.050.1GPT-4 (20-psg) 41.518.348.553.4LLaMA(Vanillaprompting) LLaMA-13B (3-psg) 50.03.93.15.3Vicuna-13B (3-psg) 58.210.015.619.6Chat-13B (5-psg) 34.713.417.315.8Chat-70B (5-psg) 38.612.838.337.9Table 6:Experiments on ELI5. We useclaim recallfor the correctness evaluation.
Chat-13B and Chat-70B refer to LLaMA-2-Chat.We describe experiment details in §C.
We use ChatGPT (gpt-3.5-turbo-0301)  with a 4K context window for most main experiments and ablations.
We also report results with ChatGPT-16K (gpt- 3.5-turbo-16k-0613)  and GPT-4 (gpt-4-0613; 8K context window) .
For open-source models, we test LLaMAand
its instruction-tuned versions, including
Alpaca, Vicuna, and Oasst.
They all have a 2K context window.
We use short instructions for LLaMA (Table24)  to save context budget.
Additionally, we test LLaMA-2-Chat, which were also trained to follow instructions. These models have a context window of 4K tokens, which allows for 5 passages per question.5.1Main ResultsWe present the main results on three datasets in
Table4,5, and6respectively (full results in §G.6) . We first note that all models achieve good fluency scores (except some models on ELI5 mainly due to their longer generations) .
We summarize the main takeaways from the experiments below.Vanillaachieves strong performance.Despite its simplicity,Vanilla(putting retrieved passages in context) 
achieves close-to-the-best performance among all prompting strategies.Using summaries or snippets improves correctness.We see a universal trend thatSummorSnippetimproves correctness,
though
on ASQA and ELI5, such an improvement comes at a cost of citation quality
due to the lossy compression. CombiningInteractwithSumm/Snippetdoes not bring improvement,
and we hypothesize that
checking the full passages offers limited benefit and
current LLMs are not proficient in an interactive usage.Retrieving text on the fly does not improve performance.All datasets show thatVanillaoutperformsInlineSearchon citation quality (and on correctness for ASQA and ELI5) .
By manually examining the examples,
we find that
it is challenging to ask detailed questions without seeing any passages.
To improveInlineSearch,
one may need to provide more context about the questions in advance or
encourage the model to call retrievers with more detailed and diverse queries.Rerankboosts citation quality.We observe thatRerankleads to consistent improvement in citation quality (on ASQA and ELI5) .
As the automatic scores may be biased inRerank,
we also conduct human evaluation (§6) 
and verify its effectiveness.ClosedBook+PostCitedelivers strong correctness but poor citation quality.ClosedBookoutperformsVanillain correctness on ELI5 and QAMPARI, and has only a 2% gap on ASQA.
However,ClosedBookcannot provide any citation;
when combined withPostCite,
the citation quality remains inadequate. For instance,
citation recall ofClosedBook+PostCiteis lower thanVanillaby 47% on ASQA.To understand whyClosedBookachieves better correctness and whyPostCitecannot deliver satisfying citation quality,
we manually examine model outputs and find that:
(1)  open-book models are easily distracted by irrelevant passages and generate responses with lower correctness,
a phenomenon also observed by;
(2) ClosedBookoften generates texts that are correct but not similar to any retrieved passages,
making it difficult to match a citation post-hoc.GPT-4 brings limited improvement but is better at using long context.We evaluate GPT-4 withVanillaand different numbers of passages (more results in §G.6) .
GPT-4 brings consistent (but limited)  improvement on correctness, but often at a cost of citation quality.
GPT-4 can also incorporate more passages due to its longer context window,
which boosts both correctness and citation quality.
On the contrary,
including more passages with ChatGPT-16K does not improve the results (Table7) ,
suggesting that processing more passages is non-trivial and GPT-4 is better at synthesizing information from its long context than ChatGPT.5.2Comparison of Different LLMsFluencyCorrect.Citation(MAUVE) (EM) Rec.Prec.ChatGPT(Vanilla) DPR (5-psg) 61.836.165.065.6GTR (1-psg) 69.538.456.064.0GTR (3-psg) 66.639.672.873.9GTR (5-psg) 66.640.473.672.5Figure 4:Retrieval recall@on ASQA (EM recall) , QAMPARI (recall-5) , and ELI5 (claim recall) .
Retrieval recall serves as an upper bound for model performance,
and we compare them with two models’ correctness results in the figure (dashed lines) :
“Vanilla (5-psg) ” is ChatGPTVanillawith top-5 passages in context;
“Oracle” is the same model except that it uses 5 gold passages (§G.1) , whose recall matches Recall@100 on all three datasets.Table7compares
different LLMs on ASQA usingVanilla(more results in §G.6) .
Notably, instruction-tuned models (Vicuna-13B and LLaMA-2-Chat) 
outperform the original LLaMA models in correctness and
considerably enhance the citation quality.
We observe that
while the original LLaMA models are able to copy facts from the context,
they struggle with accurately citing the sources or simply do not cite.
Notably, the best open-source model, LLaMA-2-70B-Chat, achieves comparable correctness score as the OpenAI models, but still lags behind in citation quality.5.3Retrieval AnalysisFluencyCorrect.Citation(MAUVE) (EM Rec.) Rec.Prec.Open-source(max #tokens=2K-4K) LLaMA-13B (3-psg) 68.426.910.615.4Vicuna-13B (3-psg) 82.631.951.150.1Chat-13B (5-psg) 72.435.238.439.4Chat-70B (5-psg) 88.341.562.961.3ChatGPT(max #tokens=4K) ChatGPT (3-psg) 66.639.672.873.9ChatGPT (5-psg) 66.640.473.672.5ChatGPT-16K(max #tokens=16K) ChatGPT (5-psg) 60.336.176.276.5ChatGPT (10-psg) 56.336.775.375.0ChatGPT (20-psg) 56.736.173.773.5GPT-4(max #tokens=8K) GPT-4 (5-psg) 67.141.368.575.6GPT-4 (10-psg) 71.543.172.075.5GPT-4 (20-psg) 64.944.473.076.5Table 7:Comparison of different LLMs on ASQA (GTR+Vanilla) . LLaMA-13B and Vicuna-13B have a context limit of 2,048 tokens, and thus can only use a short version of instructions and at most top-3 passages.
Chat-13B and Chat-70B refer to LLaMA-2-Chat.The retrieval results play a crucial role to the correctness and the citation quality.
Figure4presents
the retrieval recall@with different datasets and retrievers.
As the number of passages increases, retrieval recall steadily improves.
Additionally, Figure4shows
the correctness performance of two models:
(1)  ChatGPTVanillawith top-5 passages (our primary baseline) ;
(2)  an oracle version of the same model employing
5 gold passages (§G.1; the 5 gold passages match the retrieval recall@100) .
Notably,
both models’ correctness lags behind the corresponding retrieval recall (except for ELI5 top-5) .
The discrepancy suggests that
despite the presence of accurate answers in context,
LLMs struggle to utilize them in their outputs.We compare the impact of different retrievers and different numbers of passages to LLMs.
Figure4(right) 
shows that
GTR outperforms DPR in both correctness and citation quality,
emphasizing the importance of deploying better retrievers.
Contrary to the retrieval recall trend in Figure4,
more passages in context do not yield substantial improvement for ChatGPT.
Specifically, correctness plateaus at top-1 passage and citation quality plateaus at top-3.
GPT-4 (Table7)  exhibits an increasing trend with more passages,
but the improvement is not proportional to the retrieval performance.
This indicates the limited ability of LLMs in utilizing multiple passages within context.5.4Other AblationsWe provide additional ablations in §G.
In summary, we find that
(1) 
using comprehensive instructions
enhances the citation quality of instruction-tuned models (§G.2) ;
(2) 
including at least one demonstration improves the performance (§G.3) ;
(3)  fine-tuned models (FiD;)  withPostCitelag behind LLMs in both correctness and citation quality and fail to generalize (§G.4) .
body
ALCE: An Automatic Benchmark forLarge Language Model Generations withCitations
5Experiments
2305.14627
6Human EvaluationTo verify that our automatic evaluation correlates with human judgement,
we conduct human evaluation on selected models and request workers
to judge model generations on three dimensions similar to—(1)  utility: a 1-to-5 score indicating whether the generation helps answer the question; (2)  citation recall: the annotator is given a sentence and all passages that the sentence cited, and is asked to judge whether the passages fully support the sentence; (3)  citation precision: given a sentence and one of its citations, the annotator is asked to judge whether the citation “fully supports”, “partially supports”, or “does not support” the sentence.
Each citation gets a precision score 1 if the output sentence has a citation recall of 1 and this citation at least “partially supports” it.
See AppendixFfor more details.Human scoresALCE scoresRec.Prec.Rec.Prec.ChatGPTVanilla74.776.675.374.4w/Rerank79.381.983.980.8Vicuna-13BVanilla51.651.550.350.1Table 8:Human citation quality evaluation vs. ALCE citation quality evaluation on ASQA.Human scoresALCE scoresRec.Prec.Rec.Prec.ChatGPTVanilla50.852.452.850.4w/Rerank59.760.663.060.6Vicuna-13BVanilla13.419.213.618.1Table 9:Human citation quality evaluation vs. ALCE citation quality evaluation on ELI5.Model outputs score high utility.The utility scores do not differ significantly between models, ranging-for ASQA and-for ELI5.
Upon inspection, all tested models are mostly able to output fluent answers that are related to the question, despite differences in factual correctness.Our automatic evaluation of citation quality strongly correlates with human judgements.As shown in Table8(ASQA)  and Table9(ELI5) ,
the relative rankings induced by human and our automatic metrics are consistent.
The absolute citation scores from human and ALCE are very close except forRerank(which uses the automated citation recall for reranking) .
This suggests that an improvement on ALCE citation metrics translates to improvement on human preferences. Furthermore, the Cohen’s kappa coefficient between
human and ALCE suggests substantial agreement for citation recall ()  and moderate agreement for citation precision () .
We also show in §G.5that our automatic evaluation achieves high accuracy when treating human annotations as gold labels
(for citation recall andfor citation precision) .
body
ALCE: An Automatic Benchmark forLarge Language Model Generations withCitations
6Human Evaluation
2305.14627
7Related WorkEvaluating citations.Generating text with citations is closely related to attribution.define the “attributable to identified sources” (AIS)  score to measure how faithful a generated text is to its sources.apply AIS scores on a single-document short-answer QA dataset.study automatic evaluations for the AIS score.
A concurrent workconduct human evaluation on commercial generative search engines to
examine their citation qualities.Scientific citation text generationis a related task to ALCE where the model is provided the papers-to-cite and context and is required to recover the citing text.
It is different from ALCE
as all citations are provided
and the model only needs to perform the summarization.Retrieval-augmented LMs.Many studies have explored augmenting LMs with
externally retrieved information.pre-train language models with retrieved passages,
whileaugment
LLMs’ output by interpolating it with aNN module;
though none of them explicitly provide citations to the retrieved sources.
Other works prompt or fine-tune LLMs to
“retrieve on-the-fly”,
which offers flexibility of when and what to search.propose to first generate text without accessing external documents and then
retrieve relevant documents and revise the generation to be consistent.Among previous explorations,are the closest to our setting, where
LLMs are trained to answer questions while providing citations.
However, they do not explore retrieval strategies and simply use commercial search engines, which are not reproducible, and their models and training data are closed-source.
To the best of our knowledge,
we are the first to implement end-to-end systems that retrieve, synthesize, and cite documents with LLMs.
body
ALCE: An Automatic Benchmark forLarge Language Model Generations withCitations
7Related Work
2305.14627
8ConclusionWe propose ALCE, the first automatic benchmark for evaluating LLM generations with citations.
We deploy automatic metrics to measure fluency, correctness, and citation quality, and verify their efficacy via human evaluation.
We explore a variety of strategies for incorporating citations in LLMs
and demonstrate that current systems have considerable room for improvement on ALCE.Our experiments highlight a number of promising research directions, including
(1)  enhancing retrieval and refining retrieval integrations in LLMs,
(2)  developing long-context LLMs, and
(3)  advancing LLMs’ ability to synthesize multiple sources.
What’s even more intriguing is that these research proposals extend beyond the ALCE setup (for example, long-context LLMs have numerous exciting applications) , and
ALCE can serve as a valuable testbed for their development.
body
ALCE: An Automatic Benchmark forLarge Language Model Generations withCitations
8Conclusion
2305.14627
LimitationsOur evaluation still has room for improvement:
(1)  MAUVE is found to be sensitive to output length and may provide unstable results;
(2)  for the ELI5’s correctness evaluation, the automatically generated claims may not cover all possible answers due to the open-ended nature of the questions;
(3)  our citation quality evaluation is limited by the accuracy of the NLI model; for citation precision, the NLI model cannot detect the case of “partially support” and thus
leads to a lower citation precision score than the human evaluation.Although we believe our curated datasets closely resemble the distribution of real-world user questions,
we acknowledge that they do not cover more challenging scenarios, such as
multi-hop reasoning, math reasoning, and code completion.In our experiments, we focus on prompting LLMs without updating their model weights.
Training a model directly to incorporate citations remains challenging due to the lack of supervised data.
However, we observe that certain human-instruction datasets
contain examples similar to our task setup.
We leave the exploration of training LLMs to generate citations for future work.
body
ALCE: An Automatic Benchmark forLarge Language Model Generations withCitations
Limitations
2305.14627
AcknowledgmentsWe appreciate the helpful feedback from the members of the Princeton NLP group. We thank Alexander Wettig, Nelson Liu, Tianyi Zhang, Yu Meng, Sadhika Malladi, Yangsibo Huang, Zhiyuan Zeng, and Dan Friedman for the valuable discussion.
We thank Surge AI (especially Anna Folinsky and Edwin Chen) 
for their support with the human evaluation.
Tianyu Gao is supported by an IBM PhD Fellowship.
This research is supported by an NSF CAREER award (IIS-2239290) , a Sloan Research Fellowship,
and Microsoft Azure credits through the “Accelerate Foundation Models Academic Research” Initiative.
conclusion
ALCE: An Automatic Benchmark forLarge Language Model Generations withCitations
Acknowledgments