<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2406.13121] Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?</title><meta property="og:description" content="Long-context language models (LCLMs) have the potential to revolutionize our approach to tasks traditionally reliant on external tools like retrieval systems or databases.
Leveraging LCLMs’ ability to natively ingest a…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2406.13121">

<!--Generated on Fri Jul  5 18:14:27 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">Can Long-Context Language Models Subsume
<br class="ltx_break">Retrieval, RAG, SQL, and More?</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname"><span id="id1.1.id1" class="ltx_text ltx_font_bold">Jinhyuk Lee</span>  <span id="id2.2.id2" class="ltx_text ltx_font_bold">Anthony Chen<span id="footnotex1" class="ltx_note ltx_role_footnotemark"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_note_type">footnotemark: </span><span class="ltx_tag ltx_tag_note"><span id="footnotex1.1.1.1" class="ltx_text ltx_font_medium">1</span></span></span></span></span></span>  <span id="id3.3.id3" class="ltx_text ltx_font_bold">Zhuyun Dai<span id="footnotex2" class="ltx_note ltx_role_footnotemark"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_note_type">footnotemark: </span><span class="ltx_tag ltx_tag_note"><span id="footnotex2.1.1.1" class="ltx_text ltx_font_medium">1</span></span></span></span></span></span> 
<br class="ltx_break">
<span id="id4.4.id4" class="ltx_text ltx_font_bold">Dheeru Dua</span>  <span id="id5.5.id5" class="ltx_text ltx_font_bold">Devendra Singh Sachan</span>  <span id="id6.6.id6" class="ltx_text ltx_font_bold">Michael Boratko</span>  <span id="id7.7.id7" class="ltx_text ltx_font_bold">Yi Luan</span> 
<br class="ltx_break">
<span id="id8.8.id8" class="ltx_text ltx_font_bold">Sébastien M. R. Arnold</span>  <span id="id9.9.id9" class="ltx_text ltx_font_bold">Vincent Perot</span>  <span id="id10.10.id10" class="ltx_text ltx_font_bold">Siddharth Dalmia</span>  <span id="id11.11.id11" class="ltx_text ltx_font_bold">Hexiang Hu</span> 
<br class="ltx_break">
<span id="id12.12.id12" class="ltx_text ltx_font_bold">Xudong Lin</span>  <span id="id13.13.id13" class="ltx_text ltx_font_bold">Panupong Pasupat</span>  <span id="id14.14.id14" class="ltx_text ltx_font_bold">Aida Amini</span>  <span id="id15.15.id15" class="ltx_text ltx_font_bold">Jeremy R. Cole</span> 
<br class="ltx_break">
<span id="id16.16.id16" class="ltx_text ltx_font_bold">Sebastian Riedel</span>  <span id="id17.17.id17" class="ltx_text ltx_font_bold">Iftekhar Naim</span>  <span id="id18.18.id18" class="ltx_text ltx_font_bold">Ming-Wei Chang</span>  <span id="id19.19.id19" class="ltx_text ltx_font_bold">Kelvin Guu
<br class="ltx_break"></span>
Google DeepMind
</span><span class="ltx_author_notes">Lead contributors.</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id20.id1" class="ltx_p">Long-context language models (LCLMs) have the potential to revolutionize our approach to tasks traditionally reliant on external tools like retrieval systems or databases.
Leveraging LCLMs’ ability to natively ingest and process entire corpora of information offers numerous advantages.
It enhances user-friendliness by eliminating the need for specialized knowledge of tools, provides robust end-to-end modeling that minimizes cascading errors in complex pipelines, and allows for the application of sophisticated prompting techniques across the entire system.
To assess this paradigm shift, we introduce <span id="id20.id1.1" class="ltx_text ltx_font_smallcaps">LOFT</span>, a benchmark of real-world tasks requiring context up to millions of tokens designed to evaluate LCLMs’ performance on in-context retrieval and reasoning.
Our findings reveal LCLMs’ surprising ability to rival state-of-the-art retrieval and RAG systems, despite never having been explicitly trained for these tasks.
However, LCLMs still face challenges in areas like compositional reasoning that are required in SQL-like tasks.
Notably, prompting strategies significantly influence performance, emphasizing the need for continued research as context lengths grow.
Overall, <span id="id20.id1.2" class="ltx_text ltx_font_smallcaps">LOFT</span> provides a rigorous testing ground for LCLMs, showcasing their potential to supplant existing paradigms and tackle novel tasks as model capabilities scale.<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>The <span id="footnote1.1" class="ltx_text ltx_font_smallcaps">LOFT</span> benchmark is available at <a target="_blank" href="https://github.com/google-deepmind/loft" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/google-deepmind/loft</a>.</span></span></span>

<span id="S0.F1" class="ltx_figure"><img src="/html/2406.13121/assets/x1.png" id="S0.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="456" height="272" alt="Refer to caption">
<span class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>

An overview of the <span id="S0.F1.3.1" class="ltx_text ltx_font_smallcaps">LOFT</span> benchmark, made of six tasks which measure LCLMs’ ability to do in-context retrieval, reasoning, and many-shot learning on corpora up to millions of tokens.
We compare the performance of LCLMs against specialized models (<span id="S0.F1.4.2" class="ltx_text ltx_font_italic">e.g.,</span> CLIP for visual retrieval), which often rely on complex task-specific fine-tuning or pipelining.
Unlike specialized models, we show how LCLMs can simplify various tasks through Corpus-in-Context Prompting (<a href="#S3" title="3 Corpus-in-Context Prompting ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">3</span></a>).
</span>
</span></p>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Long-context language models (LCLMs) <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib9" title="" class="ltx_ref">9</a>, <a href="#bib.bib21" title="" class="ltx_ref">21</a>, <a href="#bib.bib45" title="" class="ltx_ref">45</a>, <a href="#bib.bib4" title="" class="ltx_ref">4</a>, <a href="#bib.bib52" title="" class="ltx_ref">52</a>]</cite> hold the promise of reshaping artificial intelligence by enabling entirely new tasks and applications while eliminating the reliance on tools and complex pipelines previously necessary due to context length limitations <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib22" title="" class="ltx_ref">22</a>, <a href="#bib.bib34" title="" class="ltx_ref">34</a>]</cite>.
By consolidating complex pipelines into a unified model, LCLMs ameliorate issues like cascading errors <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib8" title="" class="ltx_ref">8</a>]</cite> and cumbersome optimization <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib30" title="" class="ltx_ref">30</a>, <a href="#bib.bib61" title="" class="ltx_ref">61</a>]</cite>, offering a streamlined end-to-end approach to model development.
Moreover, techniques such as adding instructions <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib27" title="" class="ltx_ref">27</a>, <a href="#bib.bib59" title="" class="ltx_ref">59</a>, <a href="#bib.bib14" title="" class="ltx_ref">14</a>]</cite>, incorporating few-shot examples <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib12" title="" class="ltx_ref">12</a>]</cite>, and leveraging demonstrations via chain-of-thought prompting <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib44" title="" class="ltx_ref">44</a>, <a href="#bib.bib60" title="" class="ltx_ref">60</a>]</cite> can be seamlessly integrated to optimize LCLMs for the task at hand.</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">However, realizing the full potential of LCLMs requires rigorous evaluation on truly long-context tasks useful in real-world applications.
Existing benchmarks fall short in this regard, relying on synthetic tasks like the popular “needle-in-haystack” <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib25" title="" class="ltx_ref">25</a>, <a href="#bib.bib32" title="" class="ltx_ref">32</a>]</cite> or fixed-length datasets that fail to keep pace with the evolving definition of “long-context” <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib6" title="" class="ltx_ref">6</a>]</cite>.
Critically, existing evaluations do not adequately stress-test LCLMs on any paradigm-shifting tasks.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">To address this, we introduce the <span id="S1.p3.1.1" class="ltx_text ltx_font_bold">Lo</span>ng-Context <span id="S1.p3.1.2" class="ltx_text ltx_font_bold">F</span>ron<span id="S1.p3.1.3" class="ltx_text ltx_font_bold">t</span>iers (<span id="S1.p3.1.4" class="ltx_text ltx_font_smallcaps">LOFT</span>) benchmark, a suite of six tasks consisting of 35 datasets which span text, visual, and audio modalities, to push LCLMs to their limits and gauge their real-world impact.
Unlike previous benchmarks, <span id="S1.p3.1.5" class="ltx_text ltx_font_smallcaps">LOFT</span> allows for automatic creation of increasing context lengths. While the current version extends to one million tokens, it can easily be extended further to tens of millions, ensuring rigorous evaluation as LCLMs continue to scale.
<span id="S1.p3.1.6" class="ltx_text ltx_font_smallcaps">LOFT</span> focuses on the following areas where LCLMs have the potential for disruption:</p>
</div>
<div id="S1.p4" class="ltx_para">
<ul id="S1.I1" class="ltx_itemize">
<li id="S1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i1.p1" class="ltx_para">
<p id="S1.I1.i1.p1.1" class="ltx_p"><span id="S1.I1.i1.p1.1.1" class="ltx_text ltx_font_bold">Retrieval</span>:
LCLMs can directly ingest and retrieve information from a corpus, eliminating the need for separate dual-encoder models <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib26" title="" class="ltx_ref">26</a>, <a href="#bib.bib43" title="" class="ltx_ref">43</a>, <a href="#bib.bib31" title="" class="ltx_ref">31</a>, <a href="#bib.bib47" title="" class="ltx_ref">47</a>]</cite>. This addresses long-standing challenges in retrieval systems such as multi-hop reasoning, instruction following, and few-shot task adaptation.
We assess retrieval performance across text, visual, and audio modalities.</p>
</div>
</li>
<li id="S1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i2.p1" class="ltx_para">
<p id="S1.I1.i2.p1.1" class="ltx_p"><span id="S1.I1.i2.p1.1.1" class="ltx_text ltx_font_bold">Retrieval-Augmented Generation (RAG)</span>:
LCLMs simplify RAG pipelines by directly reasoning over a corpus, overcoming challenges like query decomposition <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib46" title="" class="ltx_ref">46</a>]</cite> and mitigating cascading errors due to retrieval misses <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib8" title="" class="ltx_ref">8</a>, <a href="#bib.bib38" title="" class="ltx_ref">38</a>]</cite>.</p>
</div>
</li>
<li id="S1.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i3.p1" class="ltx_para">
<p id="S1.I1.i3.p1.1" class="ltx_p"><span id="S1.I1.i3.p1.1.1" class="ltx_text ltx_font_bold">SQL</span>:
We explore LCLMs’ capacity to process entire databases as text, enabling natural language database querying and bypassing conversion to a formal query language like SQL <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib69" title="" class="ltx_ref">69</a>]</cite>.
This potentially enables more expressive querying and handling of noisy or mixed-structured data. Importantly, it can also be seen as a case study representing the ability of LCLMs to subsume other types of structured data and complicated formal languages to query them, such as knowledge graphs that often require bespoke solutions.</p>
</div>
</li>
<li id="S1.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i4.p1" class="ltx_para">
<p id="S1.I1.i4.p1.1" class="ltx_p"><span id="S1.I1.i4.p1.1.1" class="ltx_text ltx_font_bold">Many-Shot ICL</span>:
LCLMs can scale the number of examples from the tens in the traditional in-context learning setup to hundreds or thousands <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib65" title="" class="ltx_ref">65</a>, <a href="#bib.bib10" title="" class="ltx_ref">10</a>]</cite>, removing the need to find the optimal set of few-shot examples to use <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib39" title="" class="ltx_ref">39</a>]</cite>.</p>
</div>
</li>
</ul>
</div>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.1" class="ltx_p">The <span id="S1.p5.1.1" class="ltx_text ltx_font_smallcaps">LOFT</span> benchmark opens up a novel line of research on long-context prompting, which we introduce as Corpus-in-Context (CiC) Prompting (<a href="#S3" title="3 Corpus-in-Context Prompting ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">3</span></a>). Using this approach, we evaluate Gemini 1.5 Pro <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib52" title="" class="ltx_ref">52</a>]</cite>, GPT-4o <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib45" title="" class="ltx_ref">45</a>]</cite>, and Claude 3 Opus <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib4" title="" class="ltx_ref">4</a>]</cite> on <span id="S1.p5.1.2" class="ltx_text ltx_font_smallcaps">LOFT</span>. Figure <a href="#S0.F1" title="Figure 1 ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> summarizes the performance of these LCLMs and specialized models that were carefully hand-optimized for each task, showcasing how LCLMs can tackle <span id="S1.p5.1.3" class="ltx_text ltx_font_smallcaps">LOFT</span> tasks without specialized pipelines.</p>
</div>
<div id="S1.p6" class="ltx_para">
<p id="S1.p6.1" class="ltx_p">Our evaluation on <span id="S1.p6.1.1" class="ltx_text ltx_font_smallcaps">LOFT</span> reveals several key insights when comparing state-of-the-art LCLMs with specialized, task-specific models. At the 128k token level, the largest size comparable across all models, LCLMs rival the performance of Gecko <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib31" title="" class="ltx_ref">31</a>]</cite>, a leading textual retrieval system. Notably, Gemini <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib52" title="" class="ltx_ref">52</a>]</cite> also surpasses strong multi-modal retrieval models such as CLIP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib47" title="" class="ltx_ref">47</a>]</cite>. However, LCLMs lag significantly on complex multi-hop compositional reasoning tasks, indicating substantial room for improvement.
Furthermore, rigorous ablations reveal large performance variance depending on prompting strategies such as chain-of-thought reasoning, underscoring the need for further research to enhance LCLMs robustness and instructability.
Taken together, our results on <span id="S1.p6.1.2" class="ltx_text ltx_font_smallcaps">LOFT</span> demonstrate that LCLMs can match the performance of many specialized models, while also revealing ample headroom for improvement in robust long-context reasoning as context windows continue to scale.</p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span><span id="S2.1.1" class="ltx_text ltx_font_smallcaps">LOFT</span>: A 1 Million+ Token Long-Context Benchmark</h2>

<figure id="S2.T1" class="ltx_table">
<div id="S2.T1.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:398.9pt;height:458.7pt;vertical-align:-0.7pt;"><span class="ltx_transformed_inner" style="transform:translate(-93.0pt,106.7pt) scale(0.682124084448158,0.682124084448158) ;">
<table id="S2.T1.1.1" class="ltx_tabular ltx_align_middle">
<tr id="S2.T1.1.1.1" class="ltx_tr">
<td id="S2.T1.1.1.1.1" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S2.T1.1.1.1.1.1" class="ltx_text ltx_font_bold">Task</span></td>
<td id="S2.T1.1.1.1.2" class="ltx_td ltx_align_left ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S2.T1.1.1.1.2.1" class="ltx_text ltx_font_bold">Dataset</span></td>
<td id="S2.T1.1.1.1.3" class="ltx_td ltx_align_left ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S2.T1.1.1.1.3.1" class="ltx_text ltx_font_bold">Description</span></td>
<td id="S2.T1.1.1.1.4" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;">
<span id="S2.T1.1.1.1.4.1" class="ltx_text"></span><span id="S2.T1.1.1.1.4.2" class="ltx_text ltx_font_bold"> <span id="S2.T1.1.1.1.4.2.1" class="ltx_text">
<span id="S2.T1.1.1.1.4.2.1.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T1.1.1.1.4.2.1.1.1" class="ltx_tr">
<span id="S2.T1.1.1.1.4.2.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Avg. Cand.</span></span>
<span id="S2.T1.1.1.1.4.2.1.1.2" class="ltx_tr">
<span id="S2.T1.1.1.1.4.2.1.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Length</span></span>
</span></span><span id="S2.T1.1.1.1.4.2.2" class="ltx_text"></span></span>
</td>
<td id="S2.T1.1.1.1.5" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;">
<span id="S2.T1.1.1.1.5.1" class="ltx_text"></span><span id="S2.T1.1.1.1.5.2" class="ltx_text ltx_font_bold"> <span id="S2.T1.1.1.1.5.2.1" class="ltx_text">
<span id="S2.T1.1.1.1.5.2.1.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T1.1.1.1.5.2.1.1.1" class="ltx_tr">
<span id="S2.T1.1.1.1.5.2.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"># Cand.</span></span>
<span id="S2.T1.1.1.1.5.2.1.1.2" class="ltx_tr">
<span id="S2.T1.1.1.1.5.2.1.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">(128k)</span></span>
</span></span><span id="S2.T1.1.1.1.5.2.2" class="ltx_text"></span></span>
</td>
<td id="S2.T1.1.1.1.6" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;">
<span id="S2.T1.1.1.1.6.1" class="ltx_text"></span><span id="S2.T1.1.1.1.6.2" class="ltx_text ltx_font_bold"> <span id="S2.T1.1.1.1.6.2.1" class="ltx_text">
<span id="S2.T1.1.1.1.6.2.1.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T1.1.1.1.6.2.1.1.1" class="ltx_tr">
<span id="S2.T1.1.1.1.6.2.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Candidates</span></span>
</span></span><span id="S2.T1.1.1.1.6.2.2" class="ltx_text"></span></span>
</td>
<td id="S2.T1.1.1.1.7" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S2.T1.1.1.1.7.1" class="ltx_text ltx_font_bold">Input</span></td>
<td id="S2.T1.1.1.1.8" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S2.T1.1.1.1.8.1" class="ltx_text ltx_font_bold">Target</span></td>
</tr>
<tr id="S2.T1.1.1.2" class="ltx_tr">
<td id="S2.T1.1.1.2.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="13"><span id="S2.T1.1.1.2.1.1" class="ltx_text ltx_font_bold"><span id="S2.T1.1.1.2.1.1.1" class="ltx_text"></span> <span id="S2.T1.1.1.2.1.1.2" class="ltx_text">
<span id="S2.T1.1.1.2.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T1.1.1.2.1.1.2.1.1" class="ltx_tr">
<span id="S2.T1.1.1.2.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Text</span></span>
<span id="S2.T1.1.1.2.1.1.2.1.2" class="ltx_tr">
<span id="S2.T1.1.1.2.1.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Retrieval</span></span>
</span></span> <span id="S2.T1.1.1.2.1.1.3" class="ltx_text"></span></span></td>
<td id="S2.T1.1.1.2.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">ArguAna</td>
<td id="S2.T1.1.1.2.3" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">Argument Retrieval</td>
<td id="S2.T1.1.1.2.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">196</td>
<td id="S2.T1.1.1.2.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">531</td>
<td id="S2.T1.1.1.2.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="13"><span id="S2.T1.1.1.2.6.1" class="ltx_text">Passages</span></td>
<td id="S2.T1.1.1.2.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="13"><span id="S2.T1.1.1.2.7.1" class="ltx_text">Query</span></td>
<td id="S2.T1.1.1.2.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="13"><span id="S2.T1.1.1.2.8.1" class="ltx_text">Passage ID(s)</span></td>
</tr>
<tr id="S2.T1.1.1.3" class="ltx_tr">
<td id="S2.T1.1.1.3.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">FEVER</td>
<td id="S2.T1.1.1.3.2" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">Fact Checking</td>
<td id="S2.T1.1.1.3.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">176</td>
<td id="S2.T1.1.1.3.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">588</td>
</tr>
<tr id="S2.T1.1.1.4" class="ltx_tr">
<td id="S2.T1.1.1.4.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">FIQA</td>
<td id="S2.T1.1.1.4.2" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">Question Answering</td>
<td id="S2.T1.1.1.4.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">196</td>
<td id="S2.T1.1.1.4.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">531</td>
</tr>
<tr id="S2.T1.1.1.5" class="ltx_tr">
<td id="S2.T1.1.1.5.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">MS MARCO</td>
<td id="S2.T1.1.1.5.2" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">Web Search</td>
<td id="S2.T1.1.1.5.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">77</td>
<td id="S2.T1.1.1.5.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">1,174</td>
</tr>
<tr id="S2.T1.1.1.6" class="ltx_tr">
<td id="S2.T1.1.1.6.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">NQ</td>
<td id="S2.T1.1.1.6.2" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">Question Answering</td>
<td id="S2.T1.1.1.6.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">110</td>
<td id="S2.T1.1.1.6.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">883</td>
</tr>
<tr id="S2.T1.1.1.7" class="ltx_tr">
<td id="S2.T1.1.1.7.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">Quora</td>
<td id="S2.T1.1.1.7.2" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">Duplication Detection</td>
<td id="S2.T1.1.1.7.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">14</td>
<td id="S2.T1.1.1.7.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">3,306</td>
</tr>
<tr id="S2.T1.1.1.8" class="ltx_tr">
<td id="S2.T1.1.1.8.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">SciFact</td>
<td id="S2.T1.1.1.8.2" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">Citation Prediction</td>
<td id="S2.T1.1.1.8.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">301</td>
<td id="S2.T1.1.1.8.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">357</td>
</tr>
<tr id="S2.T1.1.1.9" class="ltx_tr">
<td id="S2.T1.1.1.9.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">Touché-2020</td>
<td id="S2.T1.1.1.9.2" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">Argument Retrieval</td>
<td id="S2.T1.1.1.9.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">330</td>
<td id="S2.T1.1.1.9.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">329</td>
</tr>
<tr id="S2.T1.1.1.10" class="ltx_tr">
<td id="S2.T1.1.1.10.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">TopiOCQA</td>
<td id="S2.T1.1.1.10.2" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">Multi-turn QA</td>
<td id="S2.T1.1.1.10.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">149</td>
<td id="S2.T1.1.1.10.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">680</td>
</tr>
<tr id="S2.T1.1.1.11" class="ltx_tr">
<td id="S2.T1.1.1.11.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">HotPotQA</td>
<td id="S2.T1.1.1.11.2" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">Multi-hop QA</td>
<td id="S2.T1.1.1.11.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">74</td>
<td id="S2.T1.1.1.11.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">1,222</td>
</tr>
<tr id="S2.T1.1.1.12" class="ltx_tr">
<td id="S2.T1.1.1.12.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">MuSiQue</td>
<td id="S2.T1.1.1.12.2" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">Multi-hop QA</td>
<td id="S2.T1.1.1.12.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">120</td>
<td id="S2.T1.1.1.12.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">824</td>
</tr>
<tr id="S2.T1.1.1.13" class="ltx_tr">
<td id="S2.T1.1.1.13.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">QAMPARI</td>
<td id="S2.T1.1.1.13.2" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">Multi-target QA</td>
<td id="S2.T1.1.1.13.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">132</td>
<td id="S2.T1.1.1.13.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">755</td>
</tr>
<tr id="S2.T1.1.1.14" class="ltx_tr">
<td id="S2.T1.1.1.14.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">QUEST</td>
<td id="S2.T1.1.1.14.2" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">Multi-target QA</td>
<td id="S2.T1.1.1.14.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">328</td>
<td id="S2.T1.1.1.14.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">328</td>
</tr>
<tr id="S2.T1.1.1.15" class="ltx_tr">
<td id="S2.T1.1.1.15.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="4"><span id="S2.T1.1.1.15.1.1" class="ltx_text ltx_font_bold"><span id="S2.T1.1.1.15.1.1.1" class="ltx_text"></span> <span id="S2.T1.1.1.15.1.1.2" class="ltx_text">
<span id="S2.T1.1.1.15.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T1.1.1.15.1.1.2.1.1" class="ltx_tr">
<span id="S2.T1.1.1.15.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Visual</span></span>
<span id="S2.T1.1.1.15.1.1.2.1.2" class="ltx_tr">
<span id="S2.T1.1.1.15.1.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Retrieval</span></span>
</span></span> <span id="S2.T1.1.1.15.1.1.3" class="ltx_text"></span></span></td>
<td id="S2.T1.1.1.15.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">Flickr30k</td>
<td id="S2.T1.1.1.15.3" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">Image Retrieval</td>
<td id="S2.T1.1.1.15.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">258</td>
<td id="S2.T1.1.1.15.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">440</td>
<td id="S2.T1.1.1.15.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S2.T1.1.1.15.6.1" class="ltx_text">Images</span></td>
<td id="S2.T1.1.1.15.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S2.T1.1.1.15.7.1" class="ltx_text">Text Query</span></td>
<td id="S2.T1.1.1.15.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S2.T1.1.1.15.8.1" class="ltx_text">Image ID</span></td>
</tr>
<tr id="S2.T1.1.1.16" class="ltx_tr">
<td id="S2.T1.1.1.16.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">MS COCO</td>
<td id="S2.T1.1.1.16.2" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">Image Retrieval</td>
<td id="S2.T1.1.1.16.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">258</td>
<td id="S2.T1.1.1.16.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">440</td>
<td id="S2.T1.1.1.16.5" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S2.T1.1.1.16.5.1" class="ltx_text">Images</span></td>
<td id="S2.T1.1.1.16.6" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S2.T1.1.1.16.6.1" class="ltx_text">Text Query</span></td>
<td id="S2.T1.1.1.16.7" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S2.T1.1.1.16.7.1" class="ltx_text">Image ID</span></td>
</tr>
<tr id="S2.T1.1.1.17" class="ltx_tr">
<td id="S2.T1.1.1.17.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">OVEN</td>
<td id="S2.T1.1.1.17.2" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">Image-text Retrieval</td>
<td id="S2.T1.1.1.17.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">278</td>
<td id="S2.T1.1.1.17.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">448</td>
<td id="S2.T1.1.1.17.5" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Images+Texts</td>
<td id="S2.T1.1.1.17.6" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Image+Text Query</td>
<td id="S2.T1.1.1.17.7" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Wikipedia ID</td>
</tr>
<tr id="S2.T1.1.1.18" class="ltx_tr">
<td id="S2.T1.1.1.18.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">MSR-VTT</td>
<td id="S2.T1.1.1.18.2" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">Video Retrieval</td>
<td id="S2.T1.1.1.18.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">774</td>
<td id="S2.T1.1.1.18.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">140</td>
<td id="S2.T1.1.1.18.5" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Videos</td>
<td id="S2.T1.1.1.18.6" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Text Query</td>
<td id="S2.T1.1.1.18.7" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Video ID</td>
</tr>
<tr id="S2.T1.1.1.19" class="ltx_tr">
<td id="S2.T1.1.1.19.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="5"><span id="S2.T1.1.1.19.1.1" class="ltx_text ltx_font_bold"><span id="S2.T1.1.1.19.1.1.1" class="ltx_text"></span> <span id="S2.T1.1.1.19.1.1.2" class="ltx_text">
<span id="S2.T1.1.1.19.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T1.1.1.19.1.1.2.1.1" class="ltx_tr">
<span id="S2.T1.1.1.19.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Audio</span></span>
<span id="S2.T1.1.1.19.1.1.2.1.2" class="ltx_tr">
<span id="S2.T1.1.1.19.1.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Retrieval</span></span>
</span></span> <span id="S2.T1.1.1.19.1.1.3" class="ltx_text"></span></span></td>
<td id="S2.T1.1.1.19.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">FLEURS-en</td>
<td id="S2.T1.1.1.19.3" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="5"><span id="S2.T1.1.1.19.3.1" class="ltx_text"><span id="S2.T1.1.1.19.3.1.1" class="ltx_text"></span> <span id="S2.T1.1.1.19.3.1.2" class="ltx_text">
<span id="S2.T1.1.1.19.3.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T1.1.1.19.3.1.2.1.1" class="ltx_tr">
<span id="S2.T1.1.1.19.3.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Audio Retrieval</span></span>
</span></span> <span id="S2.T1.1.1.19.3.1.3" class="ltx_text"></span></span></td>
<td id="S2.T1.1.1.19.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">249</td>
<td id="S2.T1.1.1.19.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">428</td>
<td id="S2.T1.1.1.19.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="5"><span id="S2.T1.1.1.19.6.1" class="ltx_text"><span id="S2.T1.1.1.19.6.1.1" class="ltx_text"></span> <span id="S2.T1.1.1.19.6.1.2" class="ltx_text">
<span id="S2.T1.1.1.19.6.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T1.1.1.19.6.1.2.1.1" class="ltx_tr">
<span id="S2.T1.1.1.19.6.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Speech</span></span>
</span></span> <span id="S2.T1.1.1.19.6.1.3" class="ltx_text"></span></span></td>
<td id="S2.T1.1.1.19.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="5"><span id="S2.T1.1.1.19.7.1" class="ltx_text">Text Query</span></td>
<td id="S2.T1.1.1.19.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="5"><span id="S2.T1.1.1.19.8.1" class="ltx_text"><span id="S2.T1.1.1.19.8.1.1" class="ltx_text"></span> <span id="S2.T1.1.1.19.8.1.2" class="ltx_text">
<span id="S2.T1.1.1.19.8.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T1.1.1.19.8.1.2.1.1" class="ltx_tr">
<span id="S2.T1.1.1.19.8.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Speech ID</span></span>
</span></span> <span id="S2.T1.1.1.19.8.1.3" class="ltx_text"></span></span></td>
</tr>
<tr id="S2.T1.1.1.20" class="ltx_tr">
<td id="S2.T1.1.1.20.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">FLEURS-es</td>
<td id="S2.T1.1.1.20.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">315</td>
<td id="S2.T1.1.1.20.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">343</td>
</tr>
<tr id="S2.T1.1.1.21" class="ltx_tr">
<td id="S2.T1.1.1.21.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">FLEURS-fr</td>
<td id="S2.T1.1.1.21.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">259</td>
<td id="S2.T1.1.1.21.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">412</td>
</tr>
<tr id="S2.T1.1.1.22" class="ltx_tr">
<td id="S2.T1.1.1.22.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">FLEURS-hi</td>
<td id="S2.T1.1.1.22.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">292</td>
<td id="S2.T1.1.1.22.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">369</td>
</tr>
<tr id="S2.T1.1.1.23" class="ltx_tr">
<td id="S2.T1.1.1.23.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">FLEURS-zh</td>
<td id="S2.T1.1.1.23.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">291</td>
<td id="S2.T1.1.1.23.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">370</td>
</tr>
<tr id="S2.T1.1.1.24" class="ltx_tr">
<td id="S2.T1.1.1.24.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="6"><span id="S2.T1.1.1.24.1.1" class="ltx_text ltx_font_bold"><span id="S2.T1.1.1.24.1.1.1" class="ltx_text"></span> <span id="S2.T1.1.1.24.1.1.2" class="ltx_text">
<span id="S2.T1.1.1.24.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T1.1.1.24.1.1.2.1.1" class="ltx_tr">
<span id="S2.T1.1.1.24.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">RAG</span></span>
</span></span> <span id="S2.T1.1.1.24.1.1.3" class="ltx_text"></span></span></td>
<td id="S2.T1.1.1.24.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">NQ</td>
<td id="S2.T1.1.1.24.3" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">Question Answering</td>
<td id="S2.T1.1.1.24.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">110</td>
<td id="S2.T1.1.1.24.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">883</td>
<td id="S2.T1.1.1.24.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="6"><span id="S2.T1.1.1.24.6.1" class="ltx_text">Passages</span></td>
<td id="S2.T1.1.1.24.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="6"><span id="S2.T1.1.1.24.7.1" class="ltx_text">Question</span></td>
<td id="S2.T1.1.1.24.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="6"><span id="S2.T1.1.1.24.8.1" class="ltx_text">Answer(s)</span></td>
</tr>
<tr id="S2.T1.1.1.25" class="ltx_tr">
<td id="S2.T1.1.1.25.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">TopiOCQA</td>
<td id="S2.T1.1.1.25.2" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">Multi-turn QA</td>
<td id="S2.T1.1.1.25.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">149</td>
<td id="S2.T1.1.1.25.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">680</td>
</tr>
<tr id="S2.T1.1.1.26" class="ltx_tr">
<td id="S2.T1.1.1.26.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">HotPotQA</td>
<td id="S2.T1.1.1.26.2" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">Multi-hop QA</td>
<td id="S2.T1.1.1.26.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">74</td>
<td id="S2.T1.1.1.26.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">1,222</td>
</tr>
<tr id="S2.T1.1.1.27" class="ltx_tr">
<td id="S2.T1.1.1.27.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">MuSiQue</td>
<td id="S2.T1.1.1.27.2" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">Multi-hop QA</td>
<td id="S2.T1.1.1.27.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">120</td>
<td id="S2.T1.1.1.27.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">824</td>
</tr>
<tr id="S2.T1.1.1.28" class="ltx_tr">
<td id="S2.T1.1.1.28.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">QAMPARI</td>
<td id="S2.T1.1.1.28.2" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">Multi-target QA</td>
<td id="S2.T1.1.1.28.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">132</td>
<td id="S2.T1.1.1.28.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">755</td>
</tr>
<tr id="S2.T1.1.1.29" class="ltx_tr">
<td id="S2.T1.1.1.29.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">QUEST</td>
<td id="S2.T1.1.1.29.2" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">Multi-target QA</td>
<td id="S2.T1.1.1.29.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">328</td>
<td id="S2.T1.1.1.29.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">328</td>
</tr>
<tr id="S2.T1.1.1.30" class="ltx_tr">
<td id="S2.T1.1.1.30.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="2"><span id="S2.T1.1.1.30.1.1" class="ltx_text ltx_font_bold">SQL</span></td>
<td id="S2.T1.1.1.30.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">Spider</td>
<td id="S2.T1.1.1.30.3" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">Single-turn SQL</td>
<td id="S2.T1.1.1.30.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">111k</td>
<td id="S2.T1.1.1.30.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">1</td>
<td id="S2.T1.1.1.30.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="2"><span id="S2.T1.1.1.30.6.1" class="ltx_text"><span id="S2.T1.1.1.30.6.1.1" class="ltx_text"></span> <span id="S2.T1.1.1.30.6.1.2" class="ltx_text">
<span id="S2.T1.1.1.30.6.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T1.1.1.30.6.1.2.1.1" class="ltx_tr">
<span id="S2.T1.1.1.30.6.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">SQL</span></span>
<span id="S2.T1.1.1.30.6.1.2.1.2" class="ltx_tr">
<span id="S2.T1.1.1.30.6.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Database</span></span>
</span></span> <span id="S2.T1.1.1.30.6.1.3" class="ltx_text"></span></span></td>
<td id="S2.T1.1.1.30.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="2"><span id="S2.T1.1.1.30.7.1" class="ltx_text">Question</span></td>
<td id="S2.T1.1.1.30.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="2"><span id="S2.T1.1.1.30.8.1" class="ltx_text"><span id="S2.T1.1.1.30.8.1.1" class="ltx_text"></span> <span id="S2.T1.1.1.30.8.1.2" class="ltx_text">
<span id="S2.T1.1.1.30.8.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T1.1.1.30.8.1.2.1.1" class="ltx_tr">
<span id="S2.T1.1.1.30.8.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Answer</span></span>
</span></span> <span id="S2.T1.1.1.30.8.1.3" class="ltx_text"></span></span></td>
</tr>
<tr id="S2.T1.1.1.31" class="ltx_tr">
<td id="S2.T1.1.1.31.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">SParC</td>
<td id="S2.T1.1.1.31.2" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">Multi-turn SQL</td>
<td id="S2.T1.1.1.31.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">111k</td>
<td id="S2.T1.1.1.31.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">1</td>
</tr>
<tr id="S2.T1.1.1.32" class="ltx_tr">
<td id="S2.T1.1.1.32.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="5"><span id="S2.T1.1.1.32.1.1" class="ltx_text ltx_font_bold"><span id="S2.T1.1.1.32.1.1.1" class="ltx_text"></span> <span id="S2.T1.1.1.32.1.1.2" class="ltx_text">
<span id="S2.T1.1.1.32.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T1.1.1.32.1.1.2.1.1" class="ltx_tr">
<span id="S2.T1.1.1.32.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Many-Shot</span></span>
<span id="S2.T1.1.1.32.1.1.2.1.2" class="ltx_tr">
<span id="S2.T1.1.1.32.1.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">ICL</span></span>
</span></span> <span id="S2.T1.1.1.32.1.1.3" class="ltx_text"></span></span></td>
<td id="S2.T1.1.1.32.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">BBH-date</td>
<td id="S2.T1.1.1.32.3" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S2.T1.1.1.32.3.1" class="ltx_text">Multiple-choice QA</span></td>
<td id="S2.T1.1.1.32.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">131</td>
<td id="S2.T1.1.1.32.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">150</td>
<td id="S2.T1.1.1.32.6" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="5"><span id="S2.T1.1.1.32.6.1" class="ltx_text"><span id="S2.T1.1.1.32.6.1.1" class="ltx_text"></span> <span id="S2.T1.1.1.32.6.1.2" class="ltx_text">
<span id="S2.T1.1.1.32.6.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T1.1.1.32.6.1.2.1.1" class="ltx_tr">
<span id="S2.T1.1.1.32.6.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Training</span></span>
<span id="S2.T1.1.1.32.6.1.2.1.2" class="ltx_tr">
<span id="S2.T1.1.1.32.6.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Examples</span></span>
</span></span> <span id="S2.T1.1.1.32.6.1.3" class="ltx_text"></span></span></td>
<td id="S2.T1.1.1.32.7" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="5"><span id="S2.T1.1.1.32.7.1" class="ltx_text">Question</span></td>
<td id="S2.T1.1.1.32.8" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="5"><span id="S2.T1.1.1.32.8.1" class="ltx_text"><span id="S2.T1.1.1.32.8.1.1" class="ltx_text"></span> <span id="S2.T1.1.1.32.8.1.2" class="ltx_text">
<span id="S2.T1.1.1.32.8.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S2.T1.1.1.32.8.1.2.1.1" class="ltx_tr">
<span id="S2.T1.1.1.32.8.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Answer</span></span>
</span></span> <span id="S2.T1.1.1.32.8.1.3" class="ltx_text"></span></span></td>
</tr>
<tr id="S2.T1.1.1.33" class="ltx_tr">
<td id="S2.T1.1.1.33.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">BBH-salient</td>
<td id="S2.T1.1.1.33.2" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">Multiple-choice QA</td>
<td id="S2.T1.1.1.33.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">246</td>
<td id="S2.T1.1.1.33.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">104</td>
</tr>
<tr id="S2.T1.1.1.34" class="ltx_tr">
<td id="S2.T1.1.1.34.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">BBH-tracking7</td>
<td id="S2.T1.1.1.34.2" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">Multiple-choice QA</td>
<td id="S2.T1.1.1.34.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">205</td>
<td id="S2.T1.1.1.34.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">123</td>
</tr>
<tr id="S2.T1.1.1.35" class="ltx_tr">
<td id="S2.T1.1.1.35.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">BBH-web</td>
<td id="S2.T1.1.1.35.2" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">Multiple-choice QA</td>
<td id="S2.T1.1.1.35.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">43</td>
<td id="S2.T1.1.1.35.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">150</td>
</tr>
<tr id="S2.T1.1.1.36" class="ltx_tr">
<td id="S2.T1.1.1.36.1" class="ltx_td ltx_align_left ltx_border_bb" style="padding-left:3.0pt;padding-right:3.0pt;">LIB-dialogue</td>
<td id="S2.T1.1.1.36.2" class="ltx_td ltx_align_left ltx_border_bb" style="padding-left:3.0pt;padding-right:3.0pt;">Classification</td>
<td id="S2.T1.1.1.36.3" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:3.0pt;padding-right:3.0pt;">266</td>
<td id="S2.T1.1.1.36.4" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:3.0pt;padding-right:3.0pt;">274</td>
</tr>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>

Tasks and datasets in the <span id="S2.T1.5.1" class="ltx_text ltx_font_smallcaps">LOFT</span> benchmark. <span id="S2.T1.6.2" class="ltx_text ltx_font_smallcaps">LOFT</span> has 6 types of tasks, 4 modalities, and 35 datasets in total.
For each dataset, we show the average length of the candidates (Avg. Cand. Length) as well as the number of candidates (# Cand) in the 128k version of <span id="S2.T1.7.3" class="ltx_text ltx_font_smallcaps">LOFT</span>.
More details on the datasets are available in <a href="#A1" title="Appendix A LOFT Dataset Creation ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Appendices</span> <span class="ltx_text ltx_ref_tag">A</span></a>, <a href="#A2" title="Appendix B Datasets Processing Details ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a> and <a href="#A3" title="Appendix C Detailed Statistics ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>.
</figcaption>
</figure>
<div id="S2.p1" class="ltx_para">
<p id="S2.p1.1" class="ltx_p">The <span id="S2.p1.1.1" class="ltx_text ltx_font_smallcaps">LOFT</span> benchmark aims to cover a wide range of real-world applications where LCLMs can be employed. These tasks range from retrieving relevant documents for a query to extracting compositional information from databases. <a href="#S2.T1" title="In 2 LOFT: A 1 Million+ Token Long-Context Benchmark ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">1</span></a> lists all tasks and their corresponding datasets.</p>
</div>
<div id="S2.p2" class="ltx_para">
<p id="S2.p2.1" class="ltx_p">For each dataset in all tasks, we sample up to 100 test queries, 5 few-shot queries, and 10 development queries.
To test how LCLMs perform when scaling the number of tokens in their context, we create <span id="S2.p2.1.1" class="ltx_text ltx_font_smallcaps">LOFT</span> with three different context length limits, namely 32k<span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span>Since the gold documents of 100 test queries alone often exceed 32k tokens, we do not include test queries for the 32k version. We report the development set performance for 32k instead.</span></span></span>, 128k, and 1M.
<span id="S2.p2.1.2" class="ltx_text ltx_font_smallcaps">LOFT</span> currently supports 1 million token contexts, but with the increasing capabilities of state-of-the-art LCLMs, our method allows <span id="S2.p2.1.3" class="ltx_text ltx_font_smallcaps">LOFT</span> to easily scale to larger context lengths (e.g., 1 billion tokens) in the future.
To allow testing the same set of queries over different context lengths, we process each dataset to have the same evaluation queries across different context lengths (except for SQL as detailed below).

<br class="ltx_break"></p>
</div>
<section id="S2.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Retrieval &amp; RAG</h4>

<div id="S2.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S2.SS0.SSS0.Px1.p1.1" class="ltx_p">We include diverse text retrieval and RAG datasets, covering heterogeneous retrieval tasks from BEIR <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib53" title="" class="ltx_ref">53</a>]</cite>, multi-turn conversational QA <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib2" title="" class="ltx_ref">2</a>]</cite>, multi-hop QA <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib63" title="" class="ltx_ref">63</a>, <a href="#bib.bib55" title="" class="ltx_ref">55</a>]</cite>, as well as multi-target QA that requires set operations such as unions or differences <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib3" title="" class="ltx_ref">3</a>, <a href="#bib.bib41" title="" class="ltx_ref">41</a>]</cite>.
For retrieval, we also include multimodal datasets, covering image, video, and audio <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib64" title="" class="ltx_ref">64</a>, <a href="#bib.bib62" title="" class="ltx_ref">62</a>, <a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite>.</p>
</div>
<figure id="S2.F2" class="ltx_figure"><img src="/html/2406.13121/assets/x2.png" id="S2.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="156" height="100" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>

<span id="S2.F2.2.1" class="ltx_text ltx_font_bold">Corpus creation</span> for retrieval and RAG.
Given a set of test queries, we use their associated gold passages and other random passages to form the corpus.
</figcaption>
</figure>
<div id="S2.SS0.SSS0.Px1.p2" class="ltx_para">
<p id="S2.SS0.SSS0.Px1.p2.1" class="ltx_p">All queries in each retrieval and RAG dataset share a single corpus, mimicking real retrieval applications. To create this shared corpus, we first include all gold passages from few-shot, development and the test queries, and then sample passages uniformly until reaching the desired context size (Figure <a href="#S2.F2" title="Figure 2 ‣ Retrieval &amp; RAG ‣ 2 LOFT: A 1 Million+ Token Long-Context Benchmark ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>). This construction ensures smaller corpora (e.g., 128k) are subsets of larger ones (e.g., 1M).
Gold and random passages are shuffled to avoid positional biases.
For fair comparison, specialized retriever models also use the same corpora for the evaluation.</p>
</div>
</section>
<section id="S2.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Many-shot ICL</h4>

<div id="S2.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S2.SS0.SSS0.Px2.p1.1" class="ltx_p">We adapt datasets from Big-Bench Hard (BBH) <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib10" title="" class="ltx_ref">10</a>, <a href="#bib.bib49" title="" class="ltx_ref">49</a>]</cite> and LongICLBench (LIB) <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib65" title="" class="ltx_ref">65</a>, <a href="#bib.bib35" title="" class="ltx_ref">35</a>]</cite> to evaluate many-shot in-context learning (ICL) capabilities.
Similar to retrieval, we construct shared many-shot ICL contexts, ensuring training examples in smaller contexts are included in larger ones.
Since all of the many-shot ICL datasets are classification tasks, we guarantee that each class is represented at least once.</p>
</div>
</section>
<section id="S2.SS0.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">SQL</h4>

<div id="S2.SS0.SSS0.Px3.p1" class="ltx_para">
<p id="S2.SS0.SSS0.Px3.p1.1" class="ltx_p">We evaluate SQL-like reasoning on Spider, a single-turn text-to-SQL dataset <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib67" title="" class="ltx_ref">67</a>]</cite>, and SparC, its multi-turn variant <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib68" title="" class="ltx_ref">68</a>]</cite>.
The corpus for each query is an associated database of one or more tables. To construct the corpus for each context length, we must select databases such that all of its tables are no larger than that context length. We do this by selecting the largest databases that will fit into that context. This necessarily means that the databases for the 1M token setting would not fit into the smaller context lengths. Therefore, unlike most of the other tasks that share a corpus, the query sets differ across <span id="S2.SS0.SSS0.Px3.p1.1.1" class="ltx_text ltx_font_smallcaps">LOFT</span> sizes.</p>
</div>
<div id="S2.SS0.SSS0.Px3.p2" class="ltx_para">
<p id="S2.SS0.SSS0.Px3.p2.2" class="ltx_p">Given a maximum context length of <math id="S2.SS0.SSS0.Px3.p2.1.m1.3" class="ltx_Math" alttext="N\in\{\text{32k},\text{128k},\text{1M}\}" display="inline"><semantics id="S2.SS0.SSS0.Px3.p2.1.m1.3a"><mrow id="S2.SS0.SSS0.Px3.p2.1.m1.3.4" xref="S2.SS0.SSS0.Px3.p2.1.m1.3.4.cmml"><mi id="S2.SS0.SSS0.Px3.p2.1.m1.3.4.2" xref="S2.SS0.SSS0.Px3.p2.1.m1.3.4.2.cmml">N</mi><mo id="S2.SS0.SSS0.Px3.p2.1.m1.3.4.1" xref="S2.SS0.SSS0.Px3.p2.1.m1.3.4.1.cmml">∈</mo><mrow id="S2.SS0.SSS0.Px3.p2.1.m1.3.4.3.2" xref="S2.SS0.SSS0.Px3.p2.1.m1.3.4.3.1.cmml"><mo stretchy="false" id="S2.SS0.SSS0.Px3.p2.1.m1.3.4.3.2.1" xref="S2.SS0.SSS0.Px3.p2.1.m1.3.4.3.1.cmml">{</mo><mtext id="S2.SS0.SSS0.Px3.p2.1.m1.1.1" xref="S2.SS0.SSS0.Px3.p2.1.m1.1.1a.cmml">32k</mtext><mo id="S2.SS0.SSS0.Px3.p2.1.m1.3.4.3.2.2" xref="S2.SS0.SSS0.Px3.p2.1.m1.3.4.3.1.cmml">,</mo><mtext id="S2.SS0.SSS0.Px3.p2.1.m1.2.2" xref="S2.SS0.SSS0.Px3.p2.1.m1.2.2a.cmml">128k</mtext><mo id="S2.SS0.SSS0.Px3.p2.1.m1.3.4.3.2.3" xref="S2.SS0.SSS0.Px3.p2.1.m1.3.4.3.1.cmml">,</mo><mtext id="S2.SS0.SSS0.Px3.p2.1.m1.3.3" xref="S2.SS0.SSS0.Px3.p2.1.m1.3.3a.cmml">1M</mtext><mo stretchy="false" id="S2.SS0.SSS0.Px3.p2.1.m1.3.4.3.2.4" xref="S2.SS0.SSS0.Px3.p2.1.m1.3.4.3.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px3.p2.1.m1.3b"><apply id="S2.SS0.SSS0.Px3.p2.1.m1.3.4.cmml" xref="S2.SS0.SSS0.Px3.p2.1.m1.3.4"><in id="S2.SS0.SSS0.Px3.p2.1.m1.3.4.1.cmml" xref="S2.SS0.SSS0.Px3.p2.1.m1.3.4.1"></in><ci id="S2.SS0.SSS0.Px3.p2.1.m1.3.4.2.cmml" xref="S2.SS0.SSS0.Px3.p2.1.m1.3.4.2">𝑁</ci><set id="S2.SS0.SSS0.Px3.p2.1.m1.3.4.3.1.cmml" xref="S2.SS0.SSS0.Px3.p2.1.m1.3.4.3.2"><ci id="S2.SS0.SSS0.Px3.p2.1.m1.1.1a.cmml" xref="S2.SS0.SSS0.Px3.p2.1.m1.1.1"><mtext id="S2.SS0.SSS0.Px3.p2.1.m1.1.1.cmml" xref="S2.SS0.SSS0.Px3.p2.1.m1.1.1">32k</mtext></ci><ci id="S2.SS0.SSS0.Px3.p2.1.m1.2.2a.cmml" xref="S2.SS0.SSS0.Px3.p2.1.m1.2.2"><mtext id="S2.SS0.SSS0.Px3.p2.1.m1.2.2.cmml" xref="S2.SS0.SSS0.Px3.p2.1.m1.2.2">128k</mtext></ci><ci id="S2.SS0.SSS0.Px3.p2.1.m1.3.3a.cmml" xref="S2.SS0.SSS0.Px3.p2.1.m1.3.3"><mtext id="S2.SS0.SSS0.Px3.p2.1.m1.3.3.cmml" xref="S2.SS0.SSS0.Px3.p2.1.m1.3.3">1M</mtext></ci></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px3.p2.1.m1.3c">N\in\{\text{32k},\text{128k},\text{1M}\}</annotation></semantics></math>, we create a corpus up to a size of <math id="S2.SS0.SSS0.Px3.p2.2.m2.1" class="ltx_Math" alttext="0.9N" display="inline"><semantics id="S2.SS0.SSS0.Px3.p2.2.m2.1a"><mrow id="S2.SS0.SSS0.Px3.p2.2.m2.1.1" xref="S2.SS0.SSS0.Px3.p2.2.m2.1.1.cmml"><mn id="S2.SS0.SSS0.Px3.p2.2.m2.1.1.2" xref="S2.SS0.SSS0.Px3.p2.2.m2.1.1.2.cmml">0.9</mn><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px3.p2.2.m2.1.1.1" xref="S2.SS0.SSS0.Px3.p2.2.m2.1.1.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px3.p2.2.m2.1.1.3" xref="S2.SS0.SSS0.Px3.p2.2.m2.1.1.3.cmml">N</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px3.p2.2.m2.1b"><apply id="S2.SS0.SSS0.Px3.p2.2.m2.1.1.cmml" xref="S2.SS0.SSS0.Px3.p2.2.m2.1.1"><times id="S2.SS0.SSS0.Px3.p2.2.m2.1.1.1.cmml" xref="S2.SS0.SSS0.Px3.p2.2.m2.1.1.1"></times><cn type="float" id="S2.SS0.SSS0.Px3.p2.2.m2.1.1.2.cmml" xref="S2.SS0.SSS0.Px3.p2.2.m2.1.1.2">0.9</cn><ci id="S2.SS0.SSS0.Px3.p2.2.m2.1.1.3.cmml" xref="S2.SS0.SSS0.Px3.p2.2.m2.1.1.3">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px3.p2.2.m2.1c">0.9N</annotation></semantics></math>, to account for differences in tokenizers, as well as to reserve room for instructions and formatting, which will be explained in more detail in <a href="#S3" title="3 Corpus-in-Context Prompting ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">3</span></a>.
Please refer to <a href="#A1" title="Appendix A LOFT Dataset Creation ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Appendix</span> <span class="ltx_text ltx_ref_tag">A</span></a> for more details about how the datasets are selected for each task.</p>
</div>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Corpus-in-Context Prompting</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.1" class="ltx_p">Traditionally, utilizing large corpora of passages, data tables, or training examples requires specialized recipes or systems.
LCLMs now enable direct ingestion and processing of entire corpora within their context window. This unlocks a novel prompting-based approach for solving new and existing tasks, which we call <span id="S3.p1.1.1" class="ltx_text ltx_font_bold">C</span>orpus-<span id="S3.p1.1.2" class="ltx_text ltx_font_bold">i</span>n-<span id="S3.p1.1.3" class="ltx_text ltx_font_bold">C</span>ontext prompting (<span id="S3.p1.1.4" class="ltx_text ltx_font_bold">CiC</span>, pronounced "seek").</p>
</div>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Prompt Design</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.1" class="ltx_p">CiC prompting effectively combines established prompting strategies, tailoring them to leverage the unique capabilities of LCLMs for learning, retrieving and reasoning over in-context corpora. <a href="#S3.F3" title="In Query Formatting ‣ 3.1 Prompt Design ‣ 3 Corpus-in-Context Prompting ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">3</span></a> illustrates our key design choices, whose effectiveness is rigorously evaluated through extensive ablation studies in <a href="#S5" title="5 CiC Prompt Ablations ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">5</span></a>.</p>
</div>
<section id="S3.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Instructions</h4>

<div id="S3.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="S3.SS1.SSS0.Px1.p1.1" class="ltx_p">We first provide task-specific instructions to guide the LCLM’s behaviors <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib27" title="" class="ltx_ref">27</a>, <a href="#bib.bib59" title="" class="ltx_ref">59</a>, <a href="#bib.bib14" title="" class="ltx_ref">14</a>]</cite>.
As an example, for the retrieval task shown in <a href="#S3.F3" title="In Query Formatting ‣ 3.1 Prompt Design ‣ 3 Corpus-in-Context Prompting ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">3</span></a>, we ask the model to read the corpus carefully and find relevant documents to answer the question.</p>
</div>
</section>
<section id="S3.SS1.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Corpus Formatting</h4>

<div id="S3.SS1.SSS0.Px2.p1" class="ltx_para">
<p id="S3.SS1.SSS0.Px2.p1.1" class="ltx_p">We then insert the entire corpus into the prompt.
Each candidate (e.g., passage, image, audio) in a corpus is assigned a unique identifier (ID) that can be referenced as needed for that task.<span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span>Original candidate IDs from each dataset were not used as they inadvertently disclose the gold documents.</span></span></span> For instance, in retrieval tasks, the LCLMs must output the correct candidate IDs.
The structure of the corpus significantly impacts retrieval performance.
Careful formatting, such as putting document IDs both before and after the passage in text retrieval, can mitigate the effects of causal attention in decoder-only LCLMs and enhance retrieval accuracy.</p>
</div>
</section>
<section id="S3.SS1.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Few-Shot Examples</h4>

<div id="S3.SS1.SSS0.Px3.p1" class="ltx_para">
<p id="S3.SS1.SSS0.Px3.p1.1" class="ltx_p">Providing a limited number of demonstrations helps the LCLM grasp the desired response format and improves task accuracy <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib12" title="" class="ltx_ref">12</a>]</cite>. Unlike common approaches where few-shot examples are independent, we ground all examples to the same corpus, aiming to teach the model to also learn more details about the specific corpus it needs to use.<span id="footnote4" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">4</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">4</sup><span class="ltx_tag ltx_tag_note">4</span>As we will see in <a href="#S4" title="4 LOFT Tasks and Primary Results ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">4</span></a>, carefully positioning the answers of these few-shot examples can guide the model’s attention to areas where it is typically weaker, mitigating ”dead zones” in attention distribution.</span></span></span>
To facilitate automated evaluation, answers within each few-shot example are formatted as a list (e.g., "Final Answer: [54, 0]" in <a href="#S3.F3" title="In Query Formatting ‣ 3.1 Prompt Design ‣ 3 Corpus-in-Context Prompting ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">3</span></a>), thus guiding the model to generate responses in a similar structure that can be readily parsed and compared against ground truth labels.
Finally, each few-shot example is accompanied by a Chain-of-Thought reasoning  <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib44" title="" class="ltx_ref">44</a>, <a href="#bib.bib60" title="" class="ltx_ref">60</a>]</cite>.
We find adding Chain-of-Thought reasoning helps the most on tasks requiring complex multi-hop compositional reasoning.</p>
</div>
</section>
<section id="S3.SS1.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Query Formatting</h4>

<div id="S3.SS1.SSS0.Px4.p1" class="ltx_para">
<p id="S3.SS1.SSS0.Px4.p1.1" class="ltx_p">The query to be evaluated is formatted similar to the few-shot examples. For multi-turn datasets, we prepend previous query turns and model outputs to the current query turn, ensuring that the model’s generation is conditioned on its prior responses.
Based on our query formatting, LCLMs generate tokens that are parsed into the final answer.</p>
</div>
<figure id="S3.F3" class="ltx_figure"><img src="/html/2406.13121/assets/x3.png" id="S3.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="201" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>

<span id="S3.F3.2.1" class="ltx_text ltx_font_bold">Example of Corpus-in-Context Prompting</span> for retrieval.
CiC prompting leverages large language models’ capacity to follow instructions, leverage few-shot examples, and benefit from reasoning demonstrations to retrieve and reason over large corpora provided in context.
</figcaption>
</figure>
<div id="S3.SS1.SSS0.Px4.p2" class="ltx_para">
<p id="S3.SS1.SSS0.Px4.p2.1" class="ltx_p"><a href="#A4" title="Appendix D Prompt Design ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Appendix</span> <span class="ltx_text ltx_ref_tag">D</span></a> provides examples of instructions for all datasets in <span id="S3.SS1.SSS0.Px4.p2.1.1" class="ltx_text ltx_font_smallcaps">LOFT</span>, along with details on the chain-of-thought reasonings used in this paper.</p>
</div>
</section>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Design Consideration</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.1" class="ltx_p">Given the variation in instructions, format, and tokenizers across different CiC prompting techniques, the resulting context lengths can differ substantially.
To accommodate this diversity, we allocate ample space for prompt customization, as detailed in <a href="#S2" title="2 LOFT: A 1 Million+ Token Long-Context Benchmark ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">2</span></a>.
To ensure a fair comparison among LCLMs, we strongly recommend that for each maximum context length of <span id="S3.SS2.p1.1.1" class="ltx_text ltx_font_smallcaps">LOFT</span> (e.g., 32k or 128k), the model still uses only the corpus and examples present in that version. We also recommend to evaluate models on the maximum size that can fit into their context length without truncating the corpus or any of the individual examples.</p>
</div>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Discussion on Efficiency</h3>

<div id="S3.SS3.p1" class="ltx_para">
<p id="S3.SS3.p1.1" class="ltx_p">Encoding a one million token context can be slow and computationally expensive.
One key advantage of CiC prompting is its compatibility with prefix-caching in autoregressive language models <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib20" title="" class="ltx_ref">20</a>]</cite> as the query appears at the end of the prompt.
This means <em id="S3.SS3.p1.1.1" class="ltx_emph ltx_font_italic">the corpus only needs to be encoded once</em>, similar to the indexing process in traditional information retrieval.
As demonstrated in <a href="#S5" title="5 CiC Prompt Ablations ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">5</span></a>, encoding the corpus as the prefix in this way does not lead to a performance drop in LCLMs.</p>
</div>
<figure id="S3.T2" class="ltx_table">
<div id="S3.T2.4" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:364.2pt;height:724.7pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-7.9pt,15.7pt) scale(0.958585673335038,0.958585673335038) ;">
<table id="S3.T2.4.4" class="ltx_tabular ltx_align_middle">
<tr id="S3.T2.4.4.5" class="ltx_tr">
<td id="S3.T2.4.4.5.1" class="ltx_td ltx_border_tt"></td>
<td id="S3.T2.4.4.5.2" class="ltx_td ltx_align_left ltx_border_tt"><span id="S3.T2.4.4.5.2.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Dataset</span></td>
<td id="S3.T2.4.4.5.3" class="ltx_td ltx_align_center ltx_border_tt"><span id="S3.T2.4.4.5.3.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Gemini 1.5 Pro</span></td>
<td id="S3.T2.4.4.5.4" class="ltx_td ltx_align_center ltx_border_tt"><span id="S3.T2.4.4.5.4.1" class="ltx_text ltx_font_bold" style="font-size:90%;">GPT-4o</span></td>
<td id="S3.T2.4.4.5.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt"><span id="S3.T2.4.4.5.5.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Claude 3 Opus</span></td>
<td id="S3.T2.4.4.5.6" class="ltx_td ltx_align_center ltx_border_tt"><span id="S3.T2.4.4.5.6.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Specialized</span></td>
</tr>
<tr id="S3.T2.4.4.6" class="ltx_tr">
<td id="S3.T2.4.4.6.1" class="ltx_td ltx_align_center ltx_border_t" rowspan="14"><span id="S3.T2.4.4.6.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;"><span id="S3.T2.4.4.6.1.1.1" class="ltx_text"></span> <span id="S3.T2.4.4.6.1.1.2" class="ltx_text">
<span id="S3.T2.4.4.6.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T2.4.4.6.1.1.2.1.1" class="ltx_tr">
<span id="S3.T2.4.4.6.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Text</span></span>
<span id="S3.T2.4.4.6.1.1.2.1.2" class="ltx_tr">
<span id="S3.T2.4.4.6.1.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">Retrieval</span></span>
</span></span> <span id="S3.T2.4.4.6.1.1.3" class="ltx_text"></span></span></td>
<td id="S3.T2.4.4.6.2" class="ltx_td ltx_align_left ltx_border_t"><span id="S3.T2.4.4.6.2.1" class="ltx_text" style="font-size:90%;">ArguAna</span></td>
<td id="S3.T2.4.4.6.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T2.4.4.6.3.1" class="ltx_text" style="font-size:90%;">0.84</span></td>
<td id="S3.T2.4.4.6.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T2.4.4.6.4.1" class="ltx_text" style="font-size:90%;">0.85</span></td>
<td id="S3.T2.4.4.6.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="S3.T2.4.4.6.5.1" class="ltx_text" style="font-size:90%;">0.74</span></td>
<td id="S3.T2.4.4.6.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T2.4.4.6.6.1" class="ltx_text" style="font-size:90%;">0.75</span></td>
</tr>
<tr id="S3.T2.4.4.7" class="ltx_tr">
<td id="S3.T2.4.4.7.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.7.1.1" class="ltx_text" style="font-size:90%;">FEVER</span></td>
<td id="S3.T2.4.4.7.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.7.2.1" class="ltx_text" style="font-size:90%;">0.98</span></td>
<td id="S3.T2.4.4.7.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.7.3.1" class="ltx_text" style="font-size:90%;">0.96</span></td>
<td id="S3.T2.4.4.7.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.7.4.1" class="ltx_text" style="font-size:90%;">0.94</span></td>
<td id="S3.T2.4.4.7.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.7.5.1" class="ltx_text" style="font-size:90%;">0.97</span></td>
</tr>
<tr id="S3.T2.4.4.8" class="ltx_tr">
<td id="S3.T2.4.4.8.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.8.1.1" class="ltx_text" style="font-size:90%;">FIQA</span></td>
<td id="S3.T2.4.4.8.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.8.2.1" class="ltx_text" style="font-size:90%;">0.79</span></td>
<td id="S3.T2.4.4.8.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.8.3.1" class="ltx_text" style="font-size:90%;">0.82</span></td>
<td id="S3.T2.4.4.8.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.8.4.1" class="ltx_text" style="font-size:90%;">0.61</span></td>
<td id="S3.T2.4.4.8.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.8.5.1" class="ltx_text" style="font-size:90%;">0.83</span></td>
</tr>
<tr id="S3.T2.4.4.9" class="ltx_tr">
<td id="S3.T2.4.4.9.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.9.1.1" class="ltx_text" style="font-size:90%;">MS MARCO</span></td>
<td id="S3.T2.4.4.9.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.9.2.1" class="ltx_text" style="font-size:90%;">0.95</span></td>
<td id="S3.T2.4.4.9.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.9.3.1" class="ltx_text" style="font-size:90%;">0.87</span></td>
<td id="S3.T2.4.4.9.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.9.4.1" class="ltx_text" style="font-size:90%;">0.93</span></td>
<td id="S3.T2.4.4.9.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.9.5.1" class="ltx_text" style="font-size:90%;">0.97</span></td>
</tr>
<tr id="S3.T2.4.4.10" class="ltx_tr">
<td id="S3.T2.4.4.10.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.10.1.1" class="ltx_text" style="font-size:90%;">NQ</span></td>
<td id="S3.T2.4.4.10.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.10.2.1" class="ltx_text" style="font-size:90%;">1.00</span></td>
<td id="S3.T2.4.4.10.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.10.3.1" class="ltx_text" style="font-size:90%;">0.99</span></td>
<td id="S3.T2.4.4.10.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.10.4.1" class="ltx_text" style="font-size:90%;">0.96</span></td>
<td id="S3.T2.4.4.10.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.10.5.1" class="ltx_text" style="font-size:90%;">0.99</span></td>
</tr>
<tr id="S3.T2.4.4.11" class="ltx_tr">
<td id="S3.T2.4.4.11.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.11.1.1" class="ltx_text" style="font-size:90%;">Quora</span></td>
<td id="S3.T2.4.4.11.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.11.2.1" class="ltx_text" style="font-size:90%;">0.93</span></td>
<td id="S3.T2.4.4.11.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.11.3.1" class="ltx_text" style="font-size:90%;">0.93</span></td>
<td id="S3.T2.4.4.11.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.11.4.1" class="ltx_text" style="font-size:90%;">0.94</span></td>
<td id="S3.T2.4.4.11.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.11.5.1" class="ltx_text" style="font-size:90%;">1.00</span></td>
</tr>
<tr id="S3.T2.4.4.12" class="ltx_tr">
<td id="S3.T2.4.4.12.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.12.1.1" class="ltx_text" style="font-size:90%;">SciFact</span></td>
<td id="S3.T2.4.4.12.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.12.2.1" class="ltx_text" style="font-size:90%;">0.88</span></td>
<td id="S3.T2.4.4.12.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.12.3.1" class="ltx_text" style="font-size:90%;">0.88</span></td>
<td id="S3.T2.4.4.12.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.12.4.1" class="ltx_text" style="font-size:90%;">0.73</span></td>
<td id="S3.T2.4.4.12.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.12.5.1" class="ltx_text" style="font-size:90%;">0.85</span></td>
</tr>
<tr id="S3.T2.4.4.13" class="ltx_tr">
<td id="S3.T2.4.4.13.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.13.1.1" class="ltx_text" style="font-size:90%;">Touché-2020</span></td>
<td id="S3.T2.4.4.13.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.13.2.1" class="ltx_text" style="font-size:90%;">0.91</span></td>
<td id="S3.T2.4.4.13.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.13.3.1" class="ltx_text" style="font-size:90%;">0.88</span></td>
<td id="S3.T2.4.4.13.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.13.4.1" class="ltx_text" style="font-size:90%;">0.71</span></td>
<td id="S3.T2.4.4.13.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.13.5.1" class="ltx_text" style="font-size:90%;">0.88</span></td>
</tr>
<tr id="S3.T2.4.4.14" class="ltx_tr">
<td id="S3.T2.4.4.14.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.14.1.1" class="ltx_text" style="font-size:90%;">TopiOCQA</span></td>
<td id="S3.T2.4.4.14.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.14.2.1" class="ltx_text" style="font-size:90%;">0.49</span></td>
<td id="S3.T2.4.4.14.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.14.3.1" class="ltx_text" style="font-size:90%;">0.30</span></td>
<td id="S3.T2.4.4.14.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.14.4.1" class="ltx_text" style="font-size:90%;">0.42</span></td>
<td id="S3.T2.4.4.14.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.14.5.1" class="ltx_text" style="font-size:90%;">0.36</span></td>
</tr>
<tr id="S3.T2.1.1.1" class="ltx_tr">
<td id="S3.T2.1.1.1.1" class="ltx_td ltx_align_left">
<span id="S3.T2.1.1.1.1.1" class="ltx_text" style="font-size:90%;">HotPotQA</span><sup id="S3.T2.1.1.1.1.2" class="ltx_sup"><span id="S3.T2.1.1.1.1.2.1" class="ltx_text" style="font-size:90%;">†</span></sup>
</td>
<td id="S3.T2.1.1.1.2" class="ltx_td ltx_align_center"><span id="S3.T2.1.1.1.2.1" class="ltx_text" style="font-size:90%;">0.90</span></td>
<td id="S3.T2.1.1.1.3" class="ltx_td ltx_align_center"><span id="S3.T2.1.1.1.3.1" class="ltx_text" style="font-size:90%;">0.82</span></td>
<td id="S3.T2.1.1.1.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.1.1.1.4.1" class="ltx_text" style="font-size:90%;">0.83</span></td>
<td id="S3.T2.1.1.1.5" class="ltx_td ltx_align_center"><span id="S3.T2.1.1.1.5.1" class="ltx_text" style="font-size:90%;">0.92</span></td>
</tr>
<tr id="S3.T2.2.2.2" class="ltx_tr">
<td id="S3.T2.2.2.2.1" class="ltx_td ltx_align_left">
<span id="S3.T2.2.2.2.1.1" class="ltx_text" style="font-size:90%;">MuSiQue</span><sup id="S3.T2.2.2.2.1.2" class="ltx_sup"><span id="S3.T2.2.2.2.1.2.1" class="ltx_text" style="font-size:90%;">†</span></sup>
</td>
<td id="S3.T2.2.2.2.2" class="ltx_td ltx_align_center"><span id="S3.T2.2.2.2.2.1" class="ltx_text" style="font-size:90%;">0.42</span></td>
<td id="S3.T2.2.2.2.3" class="ltx_td ltx_align_center"><span id="S3.T2.2.2.2.3.1" class="ltx_text" style="font-size:90%;">0.10</span></td>
<td id="S3.T2.2.2.2.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.2.2.2.4.1" class="ltx_text" style="font-size:90%;">0.27</span></td>
<td id="S3.T2.2.2.2.5" class="ltx_td ltx_align_center"><span id="S3.T2.2.2.2.5.1" class="ltx_text" style="font-size:90%;">0.29</span></td>
</tr>
<tr id="S3.T2.3.3.3" class="ltx_tr">
<td id="S3.T2.3.3.3.1" class="ltx_td ltx_align_left">
<span id="S3.T2.3.3.3.1.1" class="ltx_text" style="font-size:90%;">QAMPARI</span><sup id="S3.T2.3.3.3.1.2" class="ltx_sup"><span id="S3.T2.3.3.3.1.2.1" class="ltx_text" style="font-size:90%;">†</span></sup>
</td>
<td id="S3.T2.3.3.3.2" class="ltx_td ltx_align_center"><span id="S3.T2.3.3.3.2.1" class="ltx_text" style="font-size:90%;">0.61</span></td>
<td id="S3.T2.3.3.3.3" class="ltx_td ltx_align_center"><span id="S3.T2.3.3.3.3.1" class="ltx_text" style="font-size:90%;">0.18</span></td>
<td id="S3.T2.3.3.3.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.3.3.3.4.1" class="ltx_text" style="font-size:90%;">0.20</span></td>
<td id="S3.T2.3.3.3.5" class="ltx_td ltx_align_center"><span id="S3.T2.3.3.3.5.1" class="ltx_text" style="font-size:90%;">0.57</span></td>
</tr>
<tr id="S3.T2.4.4.4" class="ltx_tr">
<td id="S3.T2.4.4.4.1" class="ltx_td ltx_align_left">
<span id="S3.T2.4.4.4.1.1" class="ltx_text" style="font-size:90%;">QUEST</span><sup id="S3.T2.4.4.4.1.2" class="ltx_sup"><span id="S3.T2.4.4.4.1.2.1" class="ltx_text" style="font-size:90%;">†</span></sup>
</td>
<td id="S3.T2.4.4.4.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.4.2.1" class="ltx_text" style="font-size:90%;">0.30</span></td>
<td id="S3.T2.4.4.4.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.4.3.1" class="ltx_text" style="font-size:90%;">0.19</span></td>
<td id="S3.T2.4.4.4.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.4.4.1" class="ltx_text" style="font-size:90%;">0.18</span></td>
<td id="S3.T2.4.4.4.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.4.5.1" class="ltx_text" style="font-size:90%;">0.54</span></td>
</tr>
<tr id="S3.T2.4.4.15" class="ltx_tr">
<td id="S3.T2.4.4.15.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.15.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Average</span></td>
<td id="S3.T2.4.4.15.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.15.2.1" class="ltx_text ltx_font_bold" style="font-size:90%;">0.77</span></td>
<td id="S3.T2.4.4.15.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.15.3.1" class="ltx_text" style="font-size:90%;">0.67</span></td>
<td id="S3.T2.4.4.15.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.15.4.1" class="ltx_text" style="font-size:90%;">0.65</span></td>
<td id="S3.T2.4.4.15.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.15.5.1" class="ltx_text" style="font-size:90%;">0.76</span></td>
</tr>
<tr id="S3.T2.4.4.16" class="ltx_tr">
<td id="S3.T2.4.4.16.1" class="ltx_td ltx_align_center ltx_border_t" rowspan="5"><span id="S3.T2.4.4.16.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;"><span id="S3.T2.4.4.16.1.1.1" class="ltx_text"></span> <span id="S3.T2.4.4.16.1.1.2" class="ltx_text">
<span id="S3.T2.4.4.16.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T2.4.4.16.1.1.2.1.1" class="ltx_tr">
<span id="S3.T2.4.4.16.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Visual</span></span>
<span id="S3.T2.4.4.16.1.1.2.1.2" class="ltx_tr">
<span id="S3.T2.4.4.16.1.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">Retrieval</span></span>
</span></span> <span id="S3.T2.4.4.16.1.1.3" class="ltx_text"></span></span></td>
<td id="S3.T2.4.4.16.2" class="ltx_td ltx_align_left ltx_border_t"><span id="S3.T2.4.4.16.2.1" class="ltx_text" style="font-size:90%;">Flickr30k</span></td>
<td id="S3.T2.4.4.16.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T2.4.4.16.3.1" class="ltx_text" style="font-size:90%;">0.84</span></td>
<td id="S3.T2.4.4.16.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T2.4.4.16.4.1" class="ltx_text" style="font-size:90%;">0.65</span></td>
<td id="S3.T2.4.4.16.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="S3.T2.4.4.16.5.1" class="ltx_text" style="font-size:90%;">-</span></td>
<td id="S3.T2.4.4.16.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T2.4.4.16.6.1" class="ltx_text" style="font-size:90%;">0.75</span></td>
</tr>
<tr id="S3.T2.4.4.17" class="ltx_tr">
<td id="S3.T2.4.4.17.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.17.1.1" class="ltx_text" style="font-size:90%;">MS COCO</span></td>
<td id="S3.T2.4.4.17.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.17.2.1" class="ltx_text" style="font-size:90%;">0.77</span></td>
<td id="S3.T2.4.4.17.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.17.3.1" class="ltx_text" style="font-size:90%;">0.44</span></td>
<td id="S3.T2.4.4.17.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.17.4.1" class="ltx_text" style="font-size:90%;">-</span></td>
<td id="S3.T2.4.4.17.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.17.5.1" class="ltx_text" style="font-size:90%;">0.66</span></td>
</tr>
<tr id="S3.T2.4.4.18" class="ltx_tr">
<td id="S3.T2.4.4.18.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.18.1.1" class="ltx_text" style="font-size:90%;">MSR-VTT</span></td>
<td id="S3.T2.4.4.18.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.18.2.1" class="ltx_text" style="font-size:90%;">0.76</span></td>
<td id="S3.T2.4.4.18.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.18.3.1" class="ltx_text" style="font-size:90%;">0.72</span></td>
<td id="S3.T2.4.4.18.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.18.4.1" class="ltx_text" style="font-size:90%;">-</span></td>
<td id="S3.T2.4.4.18.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.18.5.1" class="ltx_text" style="font-size:90%;">0.64</span></td>
</tr>
<tr id="S3.T2.4.4.19" class="ltx_tr">
<td id="S3.T2.4.4.19.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.19.1.1" class="ltx_text" style="font-size:90%;">OVEN</span></td>
<td id="S3.T2.4.4.19.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.19.2.1" class="ltx_text" style="font-size:90%;">0.93</span></td>
<td id="S3.T2.4.4.19.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.19.3.1" class="ltx_text" style="font-size:90%;">0.89</span></td>
<td id="S3.T2.4.4.19.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.19.4.1" class="ltx_text" style="font-size:90%;">-</span></td>
<td id="S3.T2.4.4.19.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.19.5.1" class="ltx_text" style="font-size:90%;">0.79</span></td>
</tr>
<tr id="S3.T2.4.4.20" class="ltx_tr">
<td id="S3.T2.4.4.20.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.20.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Average</span></td>
<td id="S3.T2.4.4.20.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.20.2.1" class="ltx_text ltx_font_bold" style="font-size:90%;">0.83</span></td>
<td id="S3.T2.4.4.20.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.20.3.1" class="ltx_text" style="font-size:90%;">0.68</span></td>
<td id="S3.T2.4.4.20.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.20.4.1" class="ltx_text" style="font-size:90%;">-</span></td>
<td id="S3.T2.4.4.20.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.20.5.1" class="ltx_text" style="font-size:90%;">0.71</span></td>
</tr>
<tr id="S3.T2.4.4.21" class="ltx_tr">
<td id="S3.T2.4.4.21.1" class="ltx_td ltx_align_center ltx_border_t" rowspan="6"><span id="S3.T2.4.4.21.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;"><span id="S3.T2.4.4.21.1.1.1" class="ltx_text"></span> <span id="S3.T2.4.4.21.1.1.2" class="ltx_text">
<span id="S3.T2.4.4.21.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T2.4.4.21.1.1.2.1.1" class="ltx_tr">
<span id="S3.T2.4.4.21.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Audio</span></span>
<span id="S3.T2.4.4.21.1.1.2.1.2" class="ltx_tr">
<span id="S3.T2.4.4.21.1.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">Retrieval</span></span>
</span></span> <span id="S3.T2.4.4.21.1.1.3" class="ltx_text"></span></span></td>
<td id="S3.T2.4.4.21.2" class="ltx_td ltx_align_left ltx_border_t"><span id="S3.T2.4.4.21.2.1" class="ltx_text" style="font-size:90%;">FLEURS-en</span></td>
<td id="S3.T2.4.4.21.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T2.4.4.21.3.1" class="ltx_text" style="font-size:90%;">1.00</span></td>
<td id="S3.T2.4.4.21.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T2.4.4.21.4.1" class="ltx_text" style="font-size:90%;">-</span></td>
<td id="S3.T2.4.4.21.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="S3.T2.4.4.21.5.1" class="ltx_text" style="font-size:90%;">-</span></td>
<td id="S3.T2.4.4.21.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T2.4.4.21.6.1" class="ltx_text" style="font-size:90%;">0.98</span></td>
</tr>
<tr id="S3.T2.4.4.22" class="ltx_tr">
<td id="S3.T2.4.4.22.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.22.1.1" class="ltx_text" style="font-size:90%;">FLEURS-es</span></td>
<td id="S3.T2.4.4.22.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.22.2.1" class="ltx_text" style="font-size:90%;">0.99</span></td>
<td id="S3.T2.4.4.22.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.22.3.1" class="ltx_text" style="font-size:90%;">-</span></td>
<td id="S3.T2.4.4.22.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.22.4.1" class="ltx_text" style="font-size:90%;">-</span></td>
<td id="S3.T2.4.4.22.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.22.5.1" class="ltx_text" style="font-size:90%;">0.99</span></td>
</tr>
<tr id="S3.T2.4.4.23" class="ltx_tr">
<td id="S3.T2.4.4.23.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.23.1.1" class="ltx_text" style="font-size:90%;">FLEURS-fr</span></td>
<td id="S3.T2.4.4.23.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.23.2.1" class="ltx_text" style="font-size:90%;">1.00</span></td>
<td id="S3.T2.4.4.23.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.23.3.1" class="ltx_text" style="font-size:90%;">-</span></td>
<td id="S3.T2.4.4.23.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.23.4.1" class="ltx_text" style="font-size:90%;">-</span></td>
<td id="S3.T2.4.4.23.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.23.5.1" class="ltx_text" style="font-size:90%;">1.00</span></td>
</tr>
<tr id="S3.T2.4.4.24" class="ltx_tr">
<td id="S3.T2.4.4.24.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.24.1.1" class="ltx_text" style="font-size:90%;">FLEURS-hi</span></td>
<td id="S3.T2.4.4.24.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.24.2.1" class="ltx_text" style="font-size:90%;">1.00</span></td>
<td id="S3.T2.4.4.24.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.24.3.1" class="ltx_text" style="font-size:90%;">-</span></td>
<td id="S3.T2.4.4.24.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.24.4.1" class="ltx_text" style="font-size:90%;">-</span></td>
<td id="S3.T2.4.4.24.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.24.5.1" class="ltx_text" style="font-size:90%;">0.74</span></td>
</tr>
<tr id="S3.T2.4.4.25" class="ltx_tr">
<td id="S3.T2.4.4.25.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.25.1.1" class="ltx_text" style="font-size:90%;">FLEURS-zh</span></td>
<td id="S3.T2.4.4.25.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.25.2.1" class="ltx_text" style="font-size:90%;">1.00</span></td>
<td id="S3.T2.4.4.25.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.25.3.1" class="ltx_text" style="font-size:90%;">-</span></td>
<td id="S3.T2.4.4.25.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.25.4.1" class="ltx_text" style="font-size:90%;">-</span></td>
<td id="S3.T2.4.4.25.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.25.5.1" class="ltx_text" style="font-size:90%;">1.00</span></td>
</tr>
<tr id="S3.T2.4.4.26" class="ltx_tr">
<td id="S3.T2.4.4.26.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.26.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Average</span></td>
<td id="S3.T2.4.4.26.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.26.2.1" class="ltx_text ltx_font_bold" style="font-size:90%;">1.00</span></td>
<td id="S3.T2.4.4.26.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.26.3.1" class="ltx_text" style="font-size:90%;">-</span></td>
<td id="S3.T2.4.4.26.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.26.4.1" class="ltx_text" style="font-size:90%;">-</span></td>
<td id="S3.T2.4.4.26.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.26.5.1" class="ltx_text" style="font-size:90%;">0.94</span></td>
</tr>
<tr id="S3.T2.4.4.27" class="ltx_tr">
<td id="S3.T2.4.4.27.1" class="ltx_td ltx_align_center ltx_border_t" rowspan="7"><span id="S3.T2.4.4.27.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;"><span id="S3.T2.4.4.27.1.1.1" class="ltx_text"></span> <span id="S3.T2.4.4.27.1.1.2" class="ltx_text">
<span id="S3.T2.4.4.27.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T2.4.4.27.1.1.2.1.1" class="ltx_tr">
<span id="S3.T2.4.4.27.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">RAG</span></span>
</span></span> <span id="S3.T2.4.4.27.1.1.3" class="ltx_text"></span></span></td>
<td id="S3.T2.4.4.27.2" class="ltx_td ltx_align_left ltx_border_t"><span id="S3.T2.4.4.27.2.1" class="ltx_text" style="font-size:90%;">NQ</span></td>
<td id="S3.T2.4.4.27.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T2.4.4.27.3.1" class="ltx_text" style="font-size:90%;">0.84</span></td>
<td id="S3.T2.4.4.27.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T2.4.4.27.4.1" class="ltx_text" style="font-size:90%;">0.89</span></td>
<td id="S3.T2.4.4.27.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="S3.T2.4.4.27.5.1" class="ltx_text" style="font-size:90%;">0.85</span></td>
<td id="S3.T2.4.4.27.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T2.4.4.27.6.1" class="ltx_text" style="font-size:90%;">0.71</span></td>
</tr>
<tr id="S3.T2.4.4.28" class="ltx_tr">
<td id="S3.T2.4.4.28.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.28.1.1" class="ltx_text" style="font-size:90%;">TopiOCQA</span></td>
<td id="S3.T2.4.4.28.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.28.2.1" class="ltx_text" style="font-size:90%;">0.34</span></td>
<td id="S3.T2.4.4.28.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.28.3.1" class="ltx_text" style="font-size:90%;">0.33</span></td>
<td id="S3.T2.4.4.28.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.28.4.1" class="ltx_text" style="font-size:90%;">0.37</span></td>
<td id="S3.T2.4.4.28.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.28.5.1" class="ltx_text" style="font-size:90%;">0.35</span></td>
</tr>
<tr id="S3.T2.4.4.29" class="ltx_tr">
<td id="S3.T2.4.4.29.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.29.1.1" class="ltx_text" style="font-size:90%;">HotPotQA</span></td>
<td id="S3.T2.4.4.29.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.29.2.1" class="ltx_text" style="font-size:90%;">0.75</span></td>
<td id="S3.T2.4.4.29.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.29.3.1" class="ltx_text" style="font-size:90%;">0.72</span></td>
<td id="S3.T2.4.4.29.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.29.4.1" class="ltx_text" style="font-size:90%;">0.74</span></td>
<td id="S3.T2.4.4.29.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.29.5.1" class="ltx_text" style="font-size:90%;">0.70</span></td>
</tr>
<tr id="S3.T2.4.4.30" class="ltx_tr">
<td id="S3.T2.4.4.30.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.30.1.1" class="ltx_text" style="font-size:90%;">MuSiQue</span></td>
<td id="S3.T2.4.4.30.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.30.2.1" class="ltx_text" style="font-size:90%;">0.55</span></td>
<td id="S3.T2.4.4.30.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.30.3.1" class="ltx_text" style="font-size:90%;">0.47</span></td>
<td id="S3.T2.4.4.30.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.30.4.1" class="ltx_text" style="font-size:90%;">0.45</span></td>
<td id="S3.T2.4.4.30.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.30.5.1" class="ltx_text" style="font-size:90%;">0.45</span></td>
</tr>
<tr id="S3.T2.4.4.31" class="ltx_tr">
<td id="S3.T2.4.4.31.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.31.1.1" class="ltx_text" style="font-size:90%;">QAMPARI</span></td>
<td id="S3.T2.4.4.31.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.31.2.1" class="ltx_text" style="font-size:90%;">0.44</span></td>
<td id="S3.T2.4.4.31.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.31.3.1" class="ltx_text" style="font-size:90%;">0.27</span></td>
<td id="S3.T2.4.4.31.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.31.4.1" class="ltx_text" style="font-size:90%;">0.25</span></td>
<td id="S3.T2.4.4.31.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.31.5.1" class="ltx_text" style="font-size:90%;">0.55</span></td>
</tr>
<tr id="S3.T2.4.4.32" class="ltx_tr">
<td id="S3.T2.4.4.32.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.32.1.1" class="ltx_text" style="font-size:90%;">QUEST</span></td>
<td id="S3.T2.4.4.32.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.32.2.1" class="ltx_text" style="font-size:90%;">0.28</span></td>
<td id="S3.T2.4.4.32.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.32.3.1" class="ltx_text" style="font-size:90%;">0.20</span></td>
<td id="S3.T2.4.4.32.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.32.4.1" class="ltx_text" style="font-size:90%;">0.15</span></td>
<td id="S3.T2.4.4.32.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.32.5.1" class="ltx_text" style="font-size:90%;">0.35</span></td>
</tr>
<tr id="S3.T2.4.4.33" class="ltx_tr">
<td id="S3.T2.4.4.33.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.33.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Average</span></td>
<td id="S3.T2.4.4.33.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.33.2.1" class="ltx_text ltx_font_bold" style="font-size:90%;">0.53</span></td>
<td id="S3.T2.4.4.33.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.33.3.1" class="ltx_text" style="font-size:90%;">0.48</span></td>
<td id="S3.T2.4.4.33.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.33.4.1" class="ltx_text" style="font-size:90%;">0.47</span></td>
<td id="S3.T2.4.4.33.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.33.5.1" class="ltx_text" style="font-size:90%;">0.52</span></td>
</tr>
<tr id="S3.T2.4.4.34" class="ltx_tr">
<td id="S3.T2.4.4.34.1" class="ltx_td ltx_align_center ltx_border_t" rowspan="3"><span id="S3.T2.4.4.34.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">SQL</span></td>
<td id="S3.T2.4.4.34.2" class="ltx_td ltx_align_left ltx_border_t"><span id="S3.T2.4.4.34.2.1" class="ltx_text" style="font-size:90%;">Spider</span></td>
<td id="S3.T2.4.4.34.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T2.4.4.34.3.1" class="ltx_text" style="font-size:90%;">0.40</span></td>
<td id="S3.T2.4.4.34.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T2.4.4.34.4.1" class="ltx_text" style="font-size:90%;">0.14</span></td>
<td id="S3.T2.4.4.34.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="S3.T2.4.4.34.5.1" class="ltx_text" style="font-size:90%;">0.19</span></td>
<td id="S3.T2.4.4.34.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T2.4.4.34.6.1" class="ltx_text" style="font-size:90%;">0.74</span></td>
</tr>
<tr id="S3.T2.4.4.35" class="ltx_tr">
<td id="S3.T2.4.4.35.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.35.1.1" class="ltx_text" style="font-size:90%;">SParC</span></td>
<td id="S3.T2.4.4.35.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.35.2.1" class="ltx_text" style="font-size:90%;">0.36</span></td>
<td id="S3.T2.4.4.35.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.35.3.1" class="ltx_text" style="font-size:90%;">0.13</span></td>
<td id="S3.T2.4.4.35.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.35.4.1" class="ltx_text" style="font-size:90%;">0.21</span></td>
<td id="S3.T2.4.4.35.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.35.5.1" class="ltx_text" style="font-size:90%;">0.55</span></td>
</tr>
<tr id="S3.T2.4.4.36" class="ltx_tr">
<td id="S3.T2.4.4.36.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.36.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Average</span></td>
<td id="S3.T2.4.4.36.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.36.2.1" class="ltx_text" style="font-size:90%;">0.38</span></td>
<td id="S3.T2.4.4.36.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.36.3.1" class="ltx_text" style="font-size:90%;">0.13</span></td>
<td id="S3.T2.4.4.36.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.36.4.1" class="ltx_text" style="font-size:90%;">0.20</span></td>
<td id="S3.T2.4.4.36.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.36.5.1" class="ltx_text ltx_font_bold" style="font-size:90%;">0.65</span></td>
</tr>
<tr id="S3.T2.4.4.37" class="ltx_tr">
<td id="S3.T2.4.4.37.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" rowspan="6"><span id="S3.T2.4.4.37.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;"><span id="S3.T2.4.4.37.1.1.1" class="ltx_text"></span> <span id="S3.T2.4.4.37.1.1.2" class="ltx_text">
<span id="S3.T2.4.4.37.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T2.4.4.37.1.1.2.1.1" class="ltx_tr">
<span id="S3.T2.4.4.37.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Many-Shot</span></span>
<span id="S3.T2.4.4.37.1.1.2.1.2" class="ltx_tr">
<span id="S3.T2.4.4.37.1.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">ICL</span></span>
</span></span> <span id="S3.T2.4.4.37.1.1.3" class="ltx_text"></span></span></td>
<td id="S3.T2.4.4.37.2" class="ltx_td ltx_align_left ltx_border_t"><span id="S3.T2.4.4.37.2.1" class="ltx_text" style="font-size:90%;">BBH-date</span></td>
<td id="S3.T2.4.4.37.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T2.4.4.37.3.1" class="ltx_text" style="font-size:90%;">0.88</span></td>
<td id="S3.T2.4.4.37.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T2.4.4.37.4.1" class="ltx_text" style="font-size:90%;">0.81</span></td>
<td id="S3.T2.4.4.37.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="S3.T2.4.4.37.5.1" class="ltx_text" style="font-size:90%;">0.92</span></td>
<td id="S3.T2.4.4.37.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T2.4.4.37.6.1" class="ltx_text" style="font-size:90%;">-</span></td>
</tr>
<tr id="S3.T2.4.4.38" class="ltx_tr">
<td id="S3.T2.4.4.38.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.38.1.1" class="ltx_text" style="font-size:90%;">BBH-salient</span></td>
<td id="S3.T2.4.4.38.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.38.2.1" class="ltx_text" style="font-size:90%;">0.78</span></td>
<td id="S3.T2.4.4.38.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.38.3.1" class="ltx_text" style="font-size:90%;">0.64</span></td>
<td id="S3.T2.4.4.38.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.38.4.1" class="ltx_text" style="font-size:90%;">0.69</span></td>
<td id="S3.T2.4.4.38.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.38.5.1" class="ltx_text" style="font-size:90%;">-</span></td>
</tr>
<tr id="S3.T2.4.4.39" class="ltx_tr">
<td id="S3.T2.4.4.39.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.39.1.1" class="ltx_text" style="font-size:90%;">BBH-tracking7</span></td>
<td id="S3.T2.4.4.39.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.39.2.1" class="ltx_text" style="font-size:90%;">0.33</span></td>
<td id="S3.T2.4.4.39.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.39.3.1" class="ltx_text" style="font-size:90%;">0.81</span></td>
<td id="S3.T2.4.4.39.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.39.4.1" class="ltx_text" style="font-size:90%;">0.54</span></td>
<td id="S3.T2.4.4.39.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.39.5.1" class="ltx_text" style="font-size:90%;">-</span></td>
</tr>
<tr id="S3.T2.4.4.40" class="ltx_tr">
<td id="S3.T2.4.4.40.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.40.1.1" class="ltx_text" style="font-size:90%;">BBH-web</span></td>
<td id="S3.T2.4.4.40.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.40.2.1" class="ltx_text" style="font-size:90%;">0.67</span></td>
<td id="S3.T2.4.4.40.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.40.3.1" class="ltx_text" style="font-size:90%;">0.57</span></td>
<td id="S3.T2.4.4.40.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.40.4.1" class="ltx_text" style="font-size:90%;">0.83</span></td>
<td id="S3.T2.4.4.40.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.40.5.1" class="ltx_text" style="font-size:90%;">-</span></td>
</tr>
<tr id="S3.T2.4.4.41" class="ltx_tr">
<td id="S3.T2.4.4.41.1" class="ltx_td ltx_align_left"><span id="S3.T2.4.4.41.1.1" class="ltx_text" style="font-size:90%;">LIB-dialogue</span></td>
<td id="S3.T2.4.4.41.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.41.2.1" class="ltx_text" style="font-size:90%;">0.76</span></td>
<td id="S3.T2.4.4.41.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.41.3.1" class="ltx_text" style="font-size:90%;">0.67</span></td>
<td id="S3.T2.4.4.41.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T2.4.4.41.4.1" class="ltx_text" style="font-size:90%;">0.72</span></td>
<td id="S3.T2.4.4.41.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.4.41.5.1" class="ltx_text" style="font-size:90%;">-</span></td>
</tr>
<tr id="S3.T2.4.4.42" class="ltx_tr">
<td id="S3.T2.4.4.42.1" class="ltx_td ltx_align_left ltx_border_bb"><span id="S3.T2.4.4.42.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Average</span></td>
<td id="S3.T2.4.4.42.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T2.4.4.42.2.1" class="ltx_text" style="font-size:90%;">0.68</span></td>
<td id="S3.T2.4.4.42.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T2.4.4.42.3.1" class="ltx_text" style="font-size:90%;">0.70</span></td>
<td id="S3.T2.4.4.42.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r"><span id="S3.T2.4.4.42.4.1" class="ltx_text ltx_font_bold" style="font-size:90%;">0.74</span></td>
<td id="S3.T2.4.4.42.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T2.4.4.42.5.1" class="ltx_text" style="font-size:90%;">-</span></td>
</tr>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering" style="font-size:90%;"><span class="ltx_tag ltx_tag_table">Table 2: </span>

<span id="S3.T2.20.1" class="ltx_text ltx_font_bold">Main Results on <span id="S3.T2.20.1.1" class="ltx_text ltx_font_smallcaps">LOFT</span> 128k context test set</span>.
We show performances of three LCLMs as well as specialized models that rely on task-specific fine-tuning or pipelining.
For the evaluation metrics: text, visual, and audio retrieval use Recall@1; RAG uses subspan exact match; SQL uses accuracy; and many-shot ICL uses classification accuracy.
<sup id="S3.T2.21.2" class="ltx_sup">†</sup>: For retrieval with multiple gold targets, MRecall@<math id="S3.T2.9.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.T2.9.m2.1b"><mi id="S3.T2.9.m2.1.1" xref="S3.T2.9.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.T2.9.m2.1c"><ci id="S3.T2.9.m2.1.1.cmml" xref="S3.T2.9.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.9.m2.1d">k</annotation></semantics></math> (<math id="S3.T2.10.m3.4" class="ltx_Math" alttext="k=2,5,5,3" display="inline"><semantics id="S3.T2.10.m3.4b"><mrow id="S3.T2.10.m3.4.5" xref="S3.T2.10.m3.4.5.cmml"><mi id="S3.T2.10.m3.4.5.2" xref="S3.T2.10.m3.4.5.2.cmml">k</mi><mo id="S3.T2.10.m3.4.5.1" xref="S3.T2.10.m3.4.5.1.cmml">=</mo><mrow id="S3.T2.10.m3.4.5.3.2" xref="S3.T2.10.m3.4.5.3.1.cmml"><mn id="S3.T2.10.m3.1.1" xref="S3.T2.10.m3.1.1.cmml">2</mn><mo id="S3.T2.10.m3.4.5.3.2.1" xref="S3.T2.10.m3.4.5.3.1.cmml">,</mo><mn id="S3.T2.10.m3.2.2" xref="S3.T2.10.m3.2.2.cmml">5</mn><mo id="S3.T2.10.m3.4.5.3.2.2" xref="S3.T2.10.m3.4.5.3.1.cmml">,</mo><mn id="S3.T2.10.m3.3.3" xref="S3.T2.10.m3.3.3.cmml">5</mn><mo id="S3.T2.10.m3.4.5.3.2.3" xref="S3.T2.10.m3.4.5.3.1.cmml">,</mo><mn id="S3.T2.10.m3.4.4" xref="S3.T2.10.m3.4.4.cmml">3</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.T2.10.m3.4c"><apply id="S3.T2.10.m3.4.5.cmml" xref="S3.T2.10.m3.4.5"><eq id="S3.T2.10.m3.4.5.1.cmml" xref="S3.T2.10.m3.4.5.1"></eq><ci id="S3.T2.10.m3.4.5.2.cmml" xref="S3.T2.10.m3.4.5.2">𝑘</ci><list id="S3.T2.10.m3.4.5.3.1.cmml" xref="S3.T2.10.m3.4.5.3.2"><cn type="integer" id="S3.T2.10.m3.1.1.cmml" xref="S3.T2.10.m3.1.1">2</cn><cn type="integer" id="S3.T2.10.m3.2.2.cmml" xref="S3.T2.10.m3.2.2">5</cn><cn type="integer" id="S3.T2.10.m3.3.3.cmml" xref="S3.T2.10.m3.3.3">5</cn><cn type="integer" id="S3.T2.10.m3.4.4.cmml" xref="S3.T2.10.m3.4.4">3</cn></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.10.m3.4d">k=2,5,5,3</annotation></semantics></math> in order) is employed as described in <a href="#A1" title="Appendix A LOFT Dataset Creation ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Appendix</span> <span class="ltx_text ltx_ref_tag">A</span></a>.
</figcaption>
</figure>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span><span id="S4.1.1" class="ltx_text ltx_font_smallcaps">LOFT</span> Tasks and Primary Results</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.2" class="ltx_p">We evaluate three state-of-the-art LCLMs on <span id="S4.p1.2.1" class="ltx_text ltx_font_smallcaps">LOFT</span>: Google’s <span id="S4.p1.2.2" class="ltx_text ltx_font_bold">Gemini 1.5 Pro</span> <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib52" title="" class="ltx_ref">52</a>]</cite>, OpenAI’s <span id="S4.p1.2.3" class="ltx_text ltx_font_bold">GPT-4o</span> <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib45" title="" class="ltx_ref">45</a>]</cite>, and Anthropic’s <span id="S4.p1.2.4" class="ltx_text ltx_font_bold">Claude 3 Opus</span> <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib4" title="" class="ltx_ref">4</a>]</cite>.
We evaluate these models because their APIs support most of the modalities.
Their maximum context lengths are 2M, 128k, and 200k tokens, respectively.
We use their official APIs<span id="footnote5" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">5</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">5</sup><span class="ltx_tag ltx_tag_note">5</span><a target="_blank" href="https://ai.google.dev/gemini-api" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://ai.google.dev/gemini-api</a></span></span></span><sup id="S4.p1.2.5" class="ltx_sup">,</sup><span id="footnote6" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">6</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">6</sup><span class="ltx_tag ltx_tag_note">6</span><a target="_blank" href="https://platform.openai.com/docs/models/gpt-4o" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://platform.openai.com/docs/models/gpt-4o</a></span></span></span><sup id="S4.p1.2.6" class="ltx_sup">,</sup><span id="footnote7" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">7</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">7</sup><span class="ltx_tag ltx_tag_note">7</span><a target="_blank" href="https://www.anthropic.com/api" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.anthropic.com/api</a></span></span></span> for the evaluation.
Their prompts were chosen based on their performance on the development queries over the 128k token context.
While evaluating, a small number of API calls can be blocked due to various reasons such as errors in the safety filters; these examples are treated as incorrect.
In all <span id="S4.p1.2.7" class="ltx_text ltx_font_smallcaps">LOFT</span> tasks, <span id="S4.p1.2.8" class="ltx_text ltx_font_bold">LCLMs without any task-specific fine-tuning are benchmarked against specialized models that have undergone extensive fine-tuning or pipelining for the target task</span> and are therefore limited to that specific domain.
We select each specialized model that exemplifies recent task-specific advancements.</p>
</div>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Text Retrieval</h3>

<figure id="S4.F4" class="ltx_figure"><img src="/html/2406.13121/assets/x4.png" id="S4.F4.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="161" height="119" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>

<span id="S4.F4.2.1" class="ltx_text ltx_font_bold">Positional Analysis</span>.
We vary gold document positions of queries within the corpus (0% = beginning, 100% = end).
</figcaption>
</figure>
<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.1" class="ltx_p">We adopt Gecko <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib31" title="" class="ltx_ref">31</a>]</cite>, a state-of-the-art dual encoder as the specialized model for the retrieval task.
Gecko is fine-tuned on extensive text retrieval and similarity tasks.
To ensure fair comparison, we use exactly the same corpus to test both the LCLMs and Gecko, instead of using the full retrieval corpora or published results.</p>
</div>
<section id="S4.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Results</h4>

<div id="S4.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS1.SSS0.Px1.p1.1" class="ltx_p"><a href="#S3.T2" title="In 3.3 Discussion on Efficiency ‣ 3 Corpus-in-Context Prompting ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">2</span></a> demonstrates that Gemini 1.5 Pro performs comparably to Gecko at 128k context length.
Other LCLMs also perform surprisingly well.
This is notable, as LCLMs have not undergone specialized contrastive learning for retrieval. While LCLMs’s performance does degrade when scaling the corpus to millions of tokens (<a href="#S4.F5" title="In Results ‣ 4.1 Text Retrieval ‣ 4 LOFT Tasks and Primary Results ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">5</span></a>), the parity at 128k still suggests the potential of LCLMs to be used for retrieval tasks.</p>
</div>
<figure id="S4.F5" class="ltx_figure"><img src="/html/2406.13121/assets/x5.png" id="S4.F5.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="119" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 5: </span>

<span id="S4.F5.2.1" class="ltx_text ltx_font_bold">Scaling results</span> of LCLMs compared to each specialized model by scaling the corpus size from 32k to 1 million tokens.
Results are averaged over all constituent datasets in each task.
</figcaption>
</figure>
</section>
<section id="S4.SS1.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Positional Analysis</h4>

<div id="S4.SS1.SSS0.Px2.p1" class="ltx_para">
<p id="S4.SS1.SSS0.Px2.p1.1" class="ltx_p">To better understand the performance of LCLMs on longer context lengths, we investigate how the positioning of the gold document affects retrieval performance, examining the effect of the gold document for the test query and for the few shot examples within the prompt <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib37" title="" class="ltx_ref">37</a>]</cite>.</p>
</div>
<div id="S4.SS1.SSS0.Px2.p2" class="ltx_para">
<p id="S4.SS1.SSS0.Px2.p2.1" class="ltx_p"><a href="#S4.F4" title="In 4.1 Text Retrieval ‣ 4 LOFT Tasks and Primary Results ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">4</span></a> reveals that performance drops as the gold documents of the test queries are moved towards the end of the corpus, suggesting reduced attention in later sections of the prompt.
Conversely, placing the gold documents of few-shot queries at the end improves recall, indicating their ability to mitigate attention weaknesses in this region.
Co-locating gold documents of few-shot and test queries consistently boosts performance. This is perhaps unsurprising, as it gives the model information about where to look for the answer; however, it also indicates that the model does indeed pay special attention to the locations where the gold documents for the few-shot examples are placed, regardless of where they are in the corpus. This offers a promising approach to overcome performance degradation in large corpora.
Per-dataset analysis is provided in <a href="#A5" title="Appendix E Positional Analysis Detailed Results ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Appendix</span> <span class="ltx_text ltx_ref_tag">E</span></a>.</p>
</div>
</section>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Visual Retrieval</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.1" class="ltx_p">We employ CLIP-L/14 <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib47" title="" class="ltx_ref">47</a>]</cite>, a widely used text-to-image retrieval model, as our specialized model. For Flickr30k and MS-COCO, CLIP performs text-to-image retrieval. For MSR-VTT, it performs text-to-video retrieval by averaging scores across frames. For OVEN, due to the lack of suitable open-source image-to-text models, we approximate image-to-text retrieval by using CLIP’s text-to-image retrieval.
Evaluation of Claude 3 Opus on this task was not feasible due to the current limitation of 20 images per API request.</p>
</div>
<section id="S4.SS2.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Results</h4>

<div id="S4.SS2.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS2.SSS0.Px1.p1.1" class="ltx_p">Gemini 1.5 Pro outperforms GPT-4o across all four visual benchmarks (<a href="#S3.T2" title="In 3.3 Discussion on Efficiency ‣ 3 Corpus-in-Context Prompting ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">2</span></a>).
Notably, as shown in Figure <a href="#S4.F5" title="Figure 5 ‣ Results ‣ 4.1 Text Retrieval ‣ 4 LOFT Tasks and Primary Results ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, Gemini 1.5 Pro maintains a performance advantage over CLIP across all visual benchmarks and context lengths.</p>
</div>
</section>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Audio Retrieval</h3>

<div id="S4.SS3.p1" class="ltx_para">
<p id="S4.SS3.p1.1" class="ltx_p">We choose PaLM 2 DE <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib19" title="" class="ltx_ref">19</a>]</cite> as a specialized model, which is a dual-encoder trained to maximize the similarity between audio and their transcription and has achieved previous state-of-the-art on the FLEURS datasets.
Currently, GPT-4o and Claude 3 Opus do not support audio input.</p>
</div>
<section id="S4.SS3.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Results</h4>

<div id="S4.SS3.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS3.SSS0.Px1.p1.1" class="ltx_p">Gemini 1.5 Pro demonstrates comparable performance to PaLM 2 DE across all 5 languages (<a href="#S3.T2" title="In 3.3 Discussion on Efficiency ‣ 3 Corpus-in-Context Prompting ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">2</span></a>). We notice that Gemini 1.5 Pro notably surpasses PaLM 2 DE in Hindi; this advantage likely stems from differences in pre-training data between Gemini and PaLM. <a href="#S4.F5" title="In Results ‣ 4.1 Text Retrieval ‣ 4 LOFT Tasks and Primary Results ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">5</span></a> further confirms Gemini 1.5 Pro’s robust performance across various context length, highlighting the current capabilities of LCLMs while also indicating the need for more challenging audio datasets.</p>
</div>
</section>
</section>
<section id="S4.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.4 </span>RAG</h3>

<div id="S4.SS4.p1" class="ltx_para">
<p id="S4.SS4.p1.1" class="ltx_p">We set up a retrieve-and-read RAG pipeline as a specialized model, using Gecko <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib31" title="" class="ltx_ref">31</a>]</cite> to retrieve the top-40 documents which are then put into the context of Gemini 1.5 Pro and used to generate the answer conditioned on the question and the retrieved documents.</p>
</div>
<section id="S4.SS4.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Results</h4>

<div id="S4.SS4.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS4.SSS0.Px1.p1.1" class="ltx_p"><a href="#S3.T2" title="In 3.3 Discussion on Efficiency ‣ 3 Corpus-in-Context Prompting ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">2</span></a> demonstrates that Gemini 1.5 Pro, with the entire corpus in context, outperforms the RAG pipeline on multi-hop datasets (HotpotQA and MusiQue). This is because LCLMs can reason over multiple passages in the context window using Chain-of-Thought <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib60" title="" class="ltx_ref">60</a>]</cite>, a capability that RAG pipelines typically lack unless they have a separate module for planning and reasoning.</p>
</div>
<div id="S4.SS4.SSS0.Px1.p2" class="ltx_para">
<p id="S4.SS4.SSS0.Px1.p2.1" class="ltx_p">However, a specialized retriever like Gecko excels at ranking all topically relevant passages from a corpus, enabling it to identify a comprehensive set of passages covering all answers. This proves particularly beneficial for multi-target datasets, such as QUEST and QAMPARI.</p>
</div>
<div id="S4.SS4.SSS0.Px1.p3" class="ltx_para">
<p id="S4.SS4.SSS0.Px1.p3.1" class="ltx_p"><a href="#S4.F5" title="In Results ‣ 4.1 Text Retrieval ‣ 4 LOFT Tasks and Primary Results ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">5</span></a> reveals that while LCLMs match RAG performance at 128k compared to a pipeline, performance drops at 1M corresponding to the drop found in LCLM text retrieval performance.</p>
</div>
<figure id="S4.T3" class="ltx_table ltx_align_floatright">
<table id="S4.T3.1" class="ltx_tabular ltx_align_middle">
<tr id="S4.T3.1.1" class="ltx_tr">
<td id="S4.T3.1.1.1" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T3.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Dataset</span></td>
<td id="S4.T3.1.1.2" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T3.1.1.2.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Dev (32k)</span></td>
<td id="S4.T3.1.1.3" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T3.1.1.3.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Test (128k)</span></td>
</tr>
<tr id="S4.T3.1.2" class="ltx_tr">
<td id="S4.T3.1.2.1" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T3.1.2.1.1" class="ltx_text" style="font-size:90%;">NQ</span></td>
<td id="S4.T3.1.2.2" class="ltx_td ltx_align_center ltx_border_t">
<span id="S4.T3.1.2.2.1" class="ltx_text" style="font-size:90%;">0.60 </span><span id="S4.T3.1.2.2.2" class="ltx_text" style="font-size:90%;color:#FF0000;">(-0.10)</span>
</td>
<td id="S4.T3.1.2.3" class="ltx_td ltx_align_center ltx_border_t">
<span id="S4.T3.1.2.3.1" class="ltx_text" style="font-size:90%;">0.37 </span><span id="S4.T3.1.2.3.2" class="ltx_text" style="font-size:90%;color:#FF0000;">(-0.47)</span>
</td>
</tr>
<tr id="S4.T3.1.3" class="ltx_tr">
<td id="S4.T3.1.3.1" class="ltx_td ltx_align_center"><span id="S4.T3.1.3.1.1" class="ltx_text" style="font-size:90%;">HotPotQA</span></td>
<td id="S4.T3.1.3.2" class="ltx_td ltx_align_center">
<span id="S4.T3.1.3.2.1" class="ltx_text" style="font-size:90%;">0.60 </span><span id="S4.T3.1.3.2.2" class="ltx_text" style="font-size:90%;color:#FF0000;">(-0.30)</span>
</td>
<td id="S4.T3.1.3.3" class="ltx_td ltx_align_center">
<span id="S4.T3.1.3.3.1" class="ltx_text" style="font-size:90%;">0.33 </span><span id="S4.T3.1.3.3.2" class="ltx_text" style="font-size:90%;color:#FF0000;">(-0.42)</span>
</td>
</tr>
<tr id="S4.T3.1.4" class="ltx_tr">
<td id="S4.T3.1.4.1" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T3.1.4.1.1" class="ltx_text" style="font-size:90%;">MuSiQue</span></td>
<td id="S4.T3.1.4.2" class="ltx_td ltx_align_center ltx_border_bb">
<span id="S4.T3.1.4.2.1" class="ltx_text" style="font-size:90%;">0.20 </span><span id="S4.T3.1.4.2.2" class="ltx_text" style="font-size:90%;color:#FF0000;">(-0.60)</span>
</td>
<td id="S4.T3.1.4.3" class="ltx_td ltx_align_center ltx_border_bb">
<span id="S4.T3.1.4.3.1" class="ltx_text" style="font-size:90%;">0.10 </span><span id="S4.T3.1.4.3.2" class="ltx_text" style="font-size:90%;color:#FF0000;">(-0.45)</span>
</td>
</tr>
</table>
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_table">Table 3: </span><span id="S4.T3.5.1" class="ltx_text ltx_font_bold">Gemini’s closed-book performance on RAG</span>.
Red indicates the performance difference compared to the CiC prompting.</figcaption>
</figure>
</section>
<section id="S4.SS4.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Closed-Book Ablations</h4>

<div id="S4.SS4.SSS0.Px2.p1" class="ltx_para">
<p id="S4.SS4.SSS0.Px2.p1.1" class="ltx_p">To further probe LCLMs capabilities, we conduct closed-book ablations on Gemini 1.5 Pro. In this setting, we remove the corpus from the context to assess LCLM performance based solely on parametric knowledge <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib33" title="" class="ltx_ref">33</a>, <a href="#bib.bib38" title="" class="ltx_ref">38</a>]</cite>.
<a href="#S4.T3" title="In Results ‣ 4.4 RAG ‣ 4 LOFT Tasks and Primary Results ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">3</span></a> presents the results, revealing that the closed-book performance significantly lags behind the long-context and specialized models.
This underscores the tested models’ effectiveness in leveraging the external corpus to enhance its reasoning capabilities.</p>
</div>
</section>
</section>
<section id="S4.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.5 </span>SQL-Like Compositional Reasoning</h3>

<figure id="S4.F6" class="ltx_figure"><img src="/html/2406.13121/assets/x6.png" id="S4.F6.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="157" height="119" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 6: </span>
<span id="S4.F6.6.1" class="ltx_text ltx_font_bold">SQL Reasoning Analysis</span>.
We bin Spider queries by operators in their SQL query and report binned Gemini performance.
We group <span id="S4.F6.7.2" class="ltx_text ltx_font_typewriter">min</span> and <span id="S4.F6.8.3" class="ltx_text ltx_font_typewriter">max</span> into a bin and <span id="S4.F6.9.4" class="ltx_text ltx_font_typewriter">&gt;</span> and <span id="S4.F6.10.5" class="ltx_text ltx_font_typewriter">&lt;</span> into another bin.
</figcaption>
</figure>
<div id="S4.SS5.p1" class="ltx_para">
<p id="S4.SS5.p1.1" class="ltx_p">The traditional SQL pipeline uses a trained semantic parser to translate the natural language input into a SQL query.
Then, a separate SQL interpreter is used to execute the SQL query over the database.
As a specialized model, we use DAIL-SQL <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib18" title="" class="ltx_ref">18</a>]</cite> for the semantic parser, which prompts an LLM to provide the SQL query.
We adapt DAIL-SQL by replacing its LLM with Gemini 1.5 Pro and using a fixed set of few-shot examples.</p>
</div>
<section id="S4.SS5.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Results</h4>

<div id="S4.SS5.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS5.SSS0.Px1.p1.1" class="ltx_p">Results in <a href="#S3.T2" title="In 3.3 Discussion on Efficiency ‣ 3 Corpus-in-Context Prompting ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">2</span></a> show that LCLMs achieve reasonable performance, though they are significantly behind the specialized pipeline. This reveals substantial headroom to enhance the compositional reasoning capabilities of LCLMs.</p>
</div>
<div id="S4.SS5.SSS0.Px1.p2" class="ltx_para">
<p id="S4.SS5.SSS0.Px1.p2.1" class="ltx_p">Lower performance than the specialized pipeline on the SQL task is unsurprising, given the decades of work on semantic parsing and SQL.
Nonetheless, this performance still reveals the potential for the model to handle complex structured data with no task-specific tuning.</p>
</div>
</section>
<section id="S4.SS5.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Reasoning Analysis</h4>

<div id="S4.SS5.SSS0.Px2.p1" class="ltx_para">
<p id="S4.SS5.SSS0.Px2.p1.1" class="ltx_p">To gain insights into the short-comings of LCLMs in complex compositional reasoning, we categorize queries based on the operators in the gold SQL queries and measure Gemini 1.5 Pro’s performance for each operator.
<a href="#S4.F6" title="In 4.5 SQL-Like Compositional Reasoning ‣ 4 LOFT Tasks and Primary Results ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">6</span></a> shows that averaging is the most difficult operation while counting is relatively easy.
Moreover, we find that reasoning over equality is considerably easier than reasoning over inequality.</p>
</div>
</section>
</section>
<section id="S4.SS6" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.6 </span>Many-Shot ICL</h3>

<figure id="S4.F7" class="ltx_figure"><img src="/html/2406.13121/assets/x7.png" id="S4.F7.g1" class="ltx_graphics ltx_centering ltx_img_square" width="156" height="166" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 7: </span>

<span id="S4.F7.2.1" class="ltx_text ltx_font_bold">ICL Performance</span> as we scale the percentage of examples used up to 100%.
</figcaption>
</figure>
<section id="S4.SS6.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Results</h4>

<div id="S4.SS6.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS6.SSS0.Px1.p1.1" class="ltx_p"><a href="#S3.T2" title="In 3.3 Discussion on Efficiency ‣ 3 Corpus-in-Context Prompting ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">2</span></a> compares accuracy for each of the LCLMs across all ICL datasets.
For BBH, we report the accuracy on 32k, which is the maximum context length available.
Gemini 1.5 Pro outperforms GPT-4o on all benchmarks, except for BBH-tracking7 where Gemini performs surprisingly poorly.
On average, Claude 3 Opus achieves the best performance among LCLMs on this task.</p>
</div>
</section>
<section id="S4.SS6.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Scaling Many Shot ICL</h4>

<div id="S4.SS6.SSS0.Px2.p1" class="ltx_para">
<p id="S4.SS6.SSS0.Px2.p1.1" class="ltx_p"><a href="#S4.F7" title="In 4.6 Many-Shot ICL ‣ 4 LOFT Tasks and Primary Results ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">7</span></a> illustrates the impact of increasing the number of examples in Gemini.
In LIB-dialog, accuracy improves monotonically with more examples.
In contrast, results on BBH are mixed.
Knowledge-intensive tasks like BBH-date and BBH-salient see monotonic improvements similar to LIB-dialog, while reasoning-intensive tasks like BBH-tracking7 and BBH-web do not benefit.
These results suggest that more complicated tasks may see an earlier limit in how much models can learn from scaling the number of in-context examples.</p>
</div>
</section>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>CiC Prompt Ablations</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">We conduct ablations over the different facets of the CiC Prompt. Examples of the prompt for each ablation can be found in <a href="#A6" title="Appendix F Ablated Prompt Examples ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Appendix</span> <span class="ltx_text ltx_ref_tag">F</span></a>.
For the ablations, we evaluate Gemini 1.5 Pro on the 128k version of <span id="S5.p1.1.1" class="ltx_text ltx_font_smallcaps">LOFT</span>.</p>
</div>
<figure id="S5.T4" class="ltx_table">
<div id="S5.T4.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:205.7pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-76.6pt,36.3pt) scale(0.738956139928437,0.738956139928437) ;">
<table id="S5.T4.1.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S5.T4.1.1.2.1" class="ltx_tr">
<th id="S5.T4.1.1.2.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_tt">
<span id="S5.T4.1.1.2.1.1.1" class="ltx_text"></span> <span id="S5.T4.1.1.2.1.1.2" class="ltx_text">
<span id="S5.T4.1.1.2.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S5.T4.1.1.2.1.1.2.1.1" class="ltx_tr">
<span id="S5.T4.1.1.2.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S5.T4.1.1.2.1.1.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Task</span></span></span>
<span id="S5.T4.1.1.2.1.1.2.1.2" class="ltx_tr">
<span id="S5.T4.1.1.2.1.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S5.T4.1.1.2.1.1.2.1.2.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">(Metric)</span></span></span>
</span></span><span id="S5.T4.1.1.2.1.1.3" class="ltx_text"></span></th>
<td id="S5.T4.1.1.2.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt"><span id="S5.T4.1.1.2.1.2.1" class="ltx_text ltx_font_bold">Dataset</span></td>
<th id="S5.T4.1.1.2.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_tt">
<span id="S5.T4.1.1.2.1.3.1" class="ltx_text"></span> <span id="S5.T4.1.1.2.1.3.2" class="ltx_text">
<span id="S5.T4.1.1.2.1.3.2.1" class="ltx_tabular ltx_align_middle">
<span id="S5.T4.1.1.2.1.3.2.1.1" class="ltx_tr">
<span id="S5.T4.1.1.2.1.3.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S5.T4.1.1.2.1.3.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Best</span></span></span>
<span id="S5.T4.1.1.2.1.3.2.1.2" class="ltx_tr">
<span id="S5.T4.1.1.2.1.3.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S5.T4.1.1.2.1.3.2.1.2.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Prompt</span></span></span>
</span></span><span id="S5.T4.1.1.2.1.3.3" class="ltx_text"></span></th>
<th id="S5.T4.1.1.2.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_tt">
<span id="S5.T4.1.1.2.1.4.1" class="ltx_text"></span> <span id="S5.T4.1.1.2.1.4.2" class="ltx_text">
<span id="S5.T4.1.1.2.1.4.2.1" class="ltx_tabular ltx_align_middle">
<span id="S5.T4.1.1.2.1.4.2.1.1" class="ltx_tr">
<span id="S5.T4.1.1.2.1.4.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S5.T4.1.1.2.1.4.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Generic</span></span></span>
<span id="S5.T4.1.1.2.1.4.2.1.2" class="ltx_tr">
<span id="S5.T4.1.1.2.1.4.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S5.T4.1.1.2.1.4.2.1.2.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Instruction</span></span></span>
</span></span><span id="S5.T4.1.1.2.1.4.3" class="ltx_text"></span></th>
<th id="S5.T4.1.1.2.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_tt">
<span id="S5.T4.1.1.2.1.5.1" class="ltx_text"></span> <span id="S5.T4.1.1.2.1.5.2" class="ltx_text">
<span id="S5.T4.1.1.2.1.5.2.1" class="ltx_tabular ltx_align_middle">
<span id="S5.T4.1.1.2.1.5.2.1.1" class="ltx_tr">
<span id="S5.T4.1.1.2.1.5.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S5.T4.1.1.2.1.5.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Query at</span></span></span>
<span id="S5.T4.1.1.2.1.5.2.1.2" class="ltx_tr">
<span id="S5.T4.1.1.2.1.5.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S5.T4.1.1.2.1.5.2.1.2.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Beginning</span></span></span>
</span></span><span id="S5.T4.1.1.2.1.5.3" class="ltx_text"></span></th>
<th id="S5.T4.1.1.2.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_tt">
<span id="S5.T4.1.1.2.1.6.1" class="ltx_text"></span> <span id="S5.T4.1.1.2.1.6.2" class="ltx_text">
<span id="S5.T4.1.1.2.1.6.2.1" class="ltx_tabular ltx_align_middle">
<span id="S5.T4.1.1.2.1.6.2.1.1" class="ltx_tr">
<span id="S5.T4.1.1.2.1.6.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S5.T4.1.1.2.1.6.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Alphanu-</span></span></span>
<span id="S5.T4.1.1.2.1.6.2.1.2" class="ltx_tr">
<span id="S5.T4.1.1.2.1.6.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S5.T4.1.1.2.1.6.2.1.2.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">meric IDs</span></span></span>
</span></span><span id="S5.T4.1.1.2.1.6.3" class="ltx_text"></span></th>
<th id="S5.T4.1.1.2.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_tt">
<span id="S5.T4.1.1.2.1.7.1" class="ltx_text"></span> <span id="S5.T4.1.1.2.1.7.2" class="ltx_text">
<span id="S5.T4.1.1.2.1.7.2.1" class="ltx_tabular ltx_align_middle">
<span id="S5.T4.1.1.2.1.7.2.1.1" class="ltx_tr">
<span id="S5.T4.1.1.2.1.7.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S5.T4.1.1.2.1.7.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Titles</span></span></span>
<span id="S5.T4.1.1.2.1.7.2.1.2" class="ltx_tr">
<span id="S5.T4.1.1.2.1.7.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S5.T4.1.1.2.1.7.2.1.2.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Only</span></span></span>
</span></span><span id="S5.T4.1.1.2.1.7.3" class="ltx_text"></span></th>
<th id="S5.T4.1.1.2.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_tt">
<span id="S5.T4.1.1.2.1.8.1" class="ltx_text"></span> <span id="S5.T4.1.1.2.1.8.2" class="ltx_text">
<span id="S5.T4.1.1.2.1.8.2.1" class="ltx_tabular ltx_align_middle">
<span id="S5.T4.1.1.2.1.8.2.1.1" class="ltx_tr">
<span id="S5.T4.1.1.2.1.8.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S5.T4.1.1.2.1.8.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Without</span></span></span>
<span id="S5.T4.1.1.2.1.8.2.1.2" class="ltx_tr">
<span id="S5.T4.1.1.2.1.8.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S5.T4.1.1.2.1.8.2.1.2.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">ID Echo</span></span></span>
</span></span><span id="S5.T4.1.1.2.1.8.3" class="ltx_text"></span></th>
<th id="S5.T4.1.1.2.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_tt">
<span id="S5.T4.1.1.2.1.9.1" class="ltx_text"></span> <span id="S5.T4.1.1.2.1.9.2" class="ltx_text">
<span id="S5.T4.1.1.2.1.9.2.1" class="ltx_tabular ltx_align_middle">
<span id="S5.T4.1.1.2.1.9.2.1.1" class="ltx_tr">
<span id="S5.T4.1.1.2.1.9.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S5.T4.1.1.2.1.9.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Corpus in</span></span></span>
<span id="S5.T4.1.1.2.1.9.2.1.2" class="ltx_tr">
<span id="S5.T4.1.1.2.1.9.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S5.T4.1.1.2.1.9.2.1.2.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Each Few-shot</span></span></span>
</span></span><span id="S5.T4.1.1.2.1.9.3" class="ltx_text"></span></th>
<th id="S5.T4.1.1.2.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_tt">
<span id="S5.T4.1.1.2.1.10.1" class="ltx_text"></span> <span id="S5.T4.1.1.2.1.10.2" class="ltx_text">
<span id="S5.T4.1.1.2.1.10.2.1" class="ltx_tabular ltx_align_middle">
<span id="S5.T4.1.1.2.1.10.2.1.1" class="ltx_tr">
<span id="S5.T4.1.1.2.1.10.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S5.T4.1.1.2.1.10.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Without</span></span></span>
<span id="S5.T4.1.1.2.1.10.2.1.2" class="ltx_tr">
<span id="S5.T4.1.1.2.1.10.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S5.T4.1.1.2.1.10.2.1.2.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">CoT</span></span></span>
</span></span><span id="S5.T4.1.1.2.1.10.3" class="ltx_text"></span></th>
</tr>
<tr id="S5.T4.1.1.3.2" class="ltx_tr">
<td id="S5.T4.1.1.3.2.1" class="ltx_td ltx_align_center ltx_border_t" rowspan="7"><span id="S5.T4.1.1.3.2.1.1" class="ltx_text ltx_font_bold"><span id="S5.T4.1.1.3.2.1.1.1" class="ltx_text"></span> <span id="S5.T4.1.1.3.2.1.1.2" class="ltx_text">
<span id="S5.T4.1.1.3.2.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S5.T4.1.1.3.2.1.1.2.1.1" class="ltx_tr">
<span id="S5.T4.1.1.3.2.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Text</span></span>
<span id="S5.T4.1.1.3.2.1.1.2.1.2" class="ltx_tr">
<span id="S5.T4.1.1.3.2.1.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">Retrieval</span></span>
</span></span> <span id="S5.T4.1.1.3.2.1.1.3" class="ltx_text"></span></span></td>
<td id="S5.T4.1.1.3.2.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">ArguAna</td>
<td id="S5.T4.1.1.3.2.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">0.84</td>
<td id="S5.T4.1.1.3.2.4" class="ltx_td ltx_align_center ltx_border_t">0.76</td>
<td id="S5.T4.1.1.3.2.5" class="ltx_td ltx_align_center ltx_border_t">0.72</td>
<td id="S5.T4.1.1.3.2.6" class="ltx_td ltx_align_center ltx_border_t">0.81</td>
<td id="S5.T4.1.1.3.2.7" class="ltx_td ltx_align_center ltx_border_t">-</td>
<td id="S5.T4.1.1.3.2.8" class="ltx_td ltx_align_center ltx_border_t">0.78</td>
<td id="S5.T4.1.1.3.2.9" class="ltx_td ltx_align_center ltx_border_t">0.62</td>
<td id="S5.T4.1.1.3.2.10" class="ltx_td ltx_align_center ltx_border_t">0.79</td>
</tr>
<tr id="S5.T4.1.1.4.3" class="ltx_tr">
<td id="S5.T4.1.1.4.3.1" class="ltx_td ltx_align_center ltx_border_r">FIQA</td>
<td id="S5.T4.1.1.4.3.2" class="ltx_td ltx_align_center ltx_border_r">0.79</td>
<td id="S5.T4.1.1.4.3.3" class="ltx_td ltx_align_center">0.77</td>
<td id="S5.T4.1.1.4.3.4" class="ltx_td ltx_align_center">0.58</td>
<td id="S5.T4.1.1.4.3.5" class="ltx_td ltx_align_center">0.75</td>
<td id="S5.T4.1.1.4.3.6" class="ltx_td ltx_align_center">-</td>
<td id="S5.T4.1.1.4.3.7" class="ltx_td ltx_align_center">0.76</td>
<td id="S5.T4.1.1.4.3.8" class="ltx_td ltx_align_center">0.78</td>
<td id="S5.T4.1.1.4.3.9" class="ltx_td ltx_align_center">0.85</td>
</tr>
<tr id="S5.T4.1.1.5.4" class="ltx_tr">
<td id="S5.T4.1.1.5.4.1" class="ltx_td ltx_align_center ltx_border_r">NQ</td>
<td id="S5.T4.1.1.5.4.2" class="ltx_td ltx_align_center ltx_border_r">1.00</td>
<td id="S5.T4.1.1.5.4.3" class="ltx_td ltx_align_center">0.98</td>
<td id="S5.T4.1.1.5.4.4" class="ltx_td ltx_align_center">0.98</td>
<td id="S5.T4.1.1.5.4.5" class="ltx_td ltx_align_center">0.99</td>
<td id="S5.T4.1.1.5.4.6" class="ltx_td ltx_align_center">0.91</td>
<td id="S5.T4.1.1.5.4.7" class="ltx_td ltx_align_center">1.00</td>
<td id="S5.T4.1.1.5.4.8" class="ltx_td ltx_align_center">1.00</td>
<td id="S5.T4.1.1.5.4.9" class="ltx_td ltx_align_center">1.00</td>
</tr>
<tr id="S5.T4.1.1.6.5" class="ltx_tr">
<td id="S5.T4.1.1.6.5.1" class="ltx_td ltx_align_center ltx_border_r">SciFact</td>
<td id="S5.T4.1.1.6.5.2" class="ltx_td ltx_align_center ltx_border_r">0.88</td>
<td id="S5.T4.1.1.6.5.3" class="ltx_td ltx_align_center">0.88</td>
<td id="S5.T4.1.1.6.5.4" class="ltx_td ltx_align_center">0.81</td>
<td id="S5.T4.1.1.6.5.5" class="ltx_td ltx_align_center">0.90</td>
<td id="S5.T4.1.1.6.5.6" class="ltx_td ltx_align_center">0.84</td>
<td id="S5.T4.1.1.6.5.7" class="ltx_td ltx_align_center">0.87</td>
<td id="S5.T4.1.1.6.5.8" class="ltx_td ltx_align_center">0.78</td>
<td id="S5.T4.1.1.6.5.9" class="ltx_td ltx_align_center">0.90</td>
</tr>
<tr id="S5.T4.1.1.7.6" class="ltx_tr">
<td id="S5.T4.1.1.7.6.1" class="ltx_td ltx_align_center ltx_border_r">MuSiQue</td>
<td id="S5.T4.1.1.7.6.2" class="ltx_td ltx_align_center ltx_border_r">0.49</td>
<td id="S5.T4.1.1.7.6.3" class="ltx_td ltx_align_center">0.44</td>
<td id="S5.T4.1.1.7.6.4" class="ltx_td ltx_align_center">0.19</td>
<td id="S5.T4.1.1.7.6.5" class="ltx_td ltx_align_center">0.44</td>
<td id="S5.T4.1.1.7.6.6" class="ltx_td ltx_align_center">0.10</td>
<td id="S5.T4.1.1.7.6.7" class="ltx_td ltx_align_center">0.36</td>
<td id="S5.T4.1.1.7.6.8" class="ltx_td ltx_align_center">0.35</td>
<td id="S5.T4.1.1.7.6.9" class="ltx_td ltx_align_center">0.43</td>
</tr>
<tr id="S5.T4.1.1.8.7" class="ltx_tr">
<td id="S5.T4.1.1.8.7.1" class="ltx_td ltx_align_center ltx_border_r">QAMPARI</td>
<td id="S5.T4.1.1.8.7.2" class="ltx_td ltx_align_center ltx_border_r">0.61</td>
<td id="S5.T4.1.1.8.7.3" class="ltx_td ltx_align_center">0.61</td>
<td id="S5.T4.1.1.8.7.4" class="ltx_td ltx_align_center">0.49</td>
<td id="S5.T4.1.1.8.7.5" class="ltx_td ltx_align_center">0.54</td>
<td id="S5.T4.1.1.8.7.6" class="ltx_td ltx_align_center">0.09</td>
<td id="S5.T4.1.1.8.7.7" class="ltx_td ltx_align_center">0.49</td>
<td id="S5.T4.1.1.8.7.8" class="ltx_td ltx_align_center">0.35</td>
<td id="S5.T4.1.1.8.7.9" class="ltx_td ltx_align_center">0.43</td>
</tr>
<tr id="S5.T4.1.1.9.8" class="ltx_tr">
<td id="S5.T4.1.1.9.8.1" class="ltx_td ltx_align_center ltx_border_r">QUEST</td>
<td id="S5.T4.1.1.9.8.2" class="ltx_td ltx_align_center ltx_border_r">0.28</td>
<td id="S5.T4.1.1.9.8.3" class="ltx_td ltx_align_center">0.28</td>
<td id="S5.T4.1.1.9.8.4" class="ltx_td ltx_align_center">0.22</td>
<td id="S5.T4.1.1.9.8.5" class="ltx_td ltx_align_center">0.30</td>
<td id="S5.T4.1.1.9.8.6" class="ltx_td ltx_align_center">0.05</td>
<td id="S5.T4.1.1.9.8.7" class="ltx_td ltx_align_center">0.27</td>
<td id="S5.T4.1.1.9.8.8" class="ltx_td ltx_align_center">0.22</td>
<td id="S5.T4.1.1.9.8.9" class="ltx_td ltx_align_center">0.30</td>
</tr>
<tr id="S5.T4.1.1.10.9" class="ltx_tr">
<td id="S5.T4.1.1.10.9.1" class="ltx_td ltx_align_center ltx_border_t" rowspan="4"><span id="S5.T4.1.1.10.9.1.1" class="ltx_text ltx_font_bold"><span id="S5.T4.1.1.10.9.1.1.1" class="ltx_text"></span> <span id="S5.T4.1.1.10.9.1.1.2" class="ltx_text">
<span id="S5.T4.1.1.10.9.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S5.T4.1.1.10.9.1.1.2.1.1" class="ltx_tr">
<span id="S5.T4.1.1.10.9.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">RAG</span></span>
</span></span> <span id="S5.T4.1.1.10.9.1.1.3" class="ltx_text"></span></span></td>
<td id="S5.T4.1.1.10.9.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">MuSiQue</td>
<td id="S5.T4.1.1.10.9.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">0.55</td>
<td id="S5.T4.1.1.10.9.4" class="ltx_td ltx_align_center ltx_border_t">0.57</td>
<td id="S5.T4.1.1.10.9.5" class="ltx_td ltx_align_center ltx_border_t">0.39</td>
<td id="S5.T4.1.1.10.9.6" class="ltx_td ltx_align_center ltx_border_t">0.52</td>
<td id="S5.T4.1.1.10.9.7" class="ltx_td ltx_align_center ltx_border_t">0.23</td>
<td id="S5.T4.1.1.10.9.8" class="ltx_td ltx_align_center ltx_border_t">0.55</td>
<td id="S5.T4.1.1.10.9.9" class="ltx_td ltx_align_center ltx_border_t">0.78</td>
<td id="S5.T4.1.1.10.9.10" class="ltx_td ltx_align_center ltx_border_t">0.50</td>
</tr>
<tr id="S5.T4.1.1.11.10" class="ltx_tr">
<td id="S5.T4.1.1.11.10.1" class="ltx_td ltx_align_center ltx_border_r">NQ</td>
<td id="S5.T4.1.1.11.10.2" class="ltx_td ltx_align_center ltx_border_r">0.84</td>
<td id="S5.T4.1.1.11.10.3" class="ltx_td ltx_align_center">0.81</td>
<td id="S5.T4.1.1.11.10.4" class="ltx_td ltx_align_center">0.76</td>
<td id="S5.T4.1.1.11.10.5" class="ltx_td ltx_align_center">0.80</td>
<td id="S5.T4.1.1.11.10.6" class="ltx_td ltx_align_center">0.39</td>
<td id="S5.T4.1.1.11.10.7" class="ltx_td ltx_align_center">0.83</td>
<td id="S5.T4.1.1.11.10.8" class="ltx_td ltx_align_center">0.83</td>
<td id="S5.T4.1.1.11.10.9" class="ltx_td ltx_align_center">0.82</td>
</tr>
<tr id="S5.T4.1.1.12.11" class="ltx_tr">
<td id="S5.T4.1.1.12.11.1" class="ltx_td ltx_align_center ltx_border_r">QAMPARI</td>
<td id="S5.T4.1.1.12.11.2" class="ltx_td ltx_align_center ltx_border_r">0.44</td>
<td id="S5.T4.1.1.12.11.3" class="ltx_td ltx_align_center">0.42</td>
<td id="S5.T4.1.1.12.11.4" class="ltx_td ltx_align_center">0.35</td>
<td id="S5.T4.1.1.12.11.5" class="ltx_td ltx_align_center">0.42</td>
<td id="S5.T4.1.1.12.11.6" class="ltx_td ltx_align_center">0.09</td>
<td id="S5.T4.1.1.12.11.7" class="ltx_td ltx_align_center">0.36</td>
<td id="S5.T4.1.1.12.11.8" class="ltx_td ltx_align_center">0.36</td>
<td id="S5.T4.1.1.12.11.9" class="ltx_td ltx_align_center">0.33</td>
</tr>
<tr id="S5.T4.1.1.13.12" class="ltx_tr">
<td id="S5.T4.1.1.13.12.1" class="ltx_td ltx_align_center ltx_border_r">QUEST</td>
<td id="S5.T4.1.1.13.12.2" class="ltx_td ltx_align_center ltx_border_r">0.28</td>
<td id="S5.T4.1.1.13.12.3" class="ltx_td ltx_align_center">0.33</td>
<td id="S5.T4.1.1.13.12.4" class="ltx_td ltx_align_center">0.17</td>
<td id="S5.T4.1.1.13.12.5" class="ltx_td ltx_align_center">0.28</td>
<td id="S5.T4.1.1.13.12.6" class="ltx_td ltx_align_center">0.02</td>
<td id="S5.T4.1.1.13.12.7" class="ltx_td ltx_align_center">0.25</td>
<td id="S5.T4.1.1.13.12.8" class="ltx_td ltx_align_center">0.27</td>
<td id="S5.T4.1.1.13.12.9" class="ltx_td ltx_align_center">0.32</td>
</tr>
<tr id="S5.T4.1.1.14.13" class="ltx_tr">
<td id="S5.T4.1.1.14.13.1" class="ltx_td ltx_border_t"></td>
<td id="S5.T4.1.1.14.13.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="S5.T4.1.1.14.13.2.1" class="ltx_text ltx_font_bold">Average</span></td>
<td id="S5.T4.1.1.14.13.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">0.64</td>
<td id="S5.T4.1.1.14.13.4" class="ltx_td ltx_align_center ltx_border_t">0.62</td>
<td id="S5.T4.1.1.14.13.5" class="ltx_td ltx_align_center ltx_border_t">0.51</td>
<td id="S5.T4.1.1.14.13.6" class="ltx_td ltx_align_center ltx_border_t">0.61</td>
<td id="S5.T4.1.1.14.13.7" class="ltx_td ltx_align_center ltx_border_t">0.30</td>
<td id="S5.T4.1.1.14.13.8" class="ltx_td ltx_align_center ltx_border_t">0.59</td>
<td id="S5.T4.1.1.14.13.9" class="ltx_td ltx_align_center ltx_border_t">0.55</td>
<td id="S5.T4.1.1.14.13.10" class="ltx_td ltx_align_center ltx_border_t">0.61</td>
</tr>
<tr id="S5.T4.1.1.1" class="ltx_tr">
<td id="S5.T4.1.1.1.2" class="ltx_td ltx_border_bb"></td>
<td id="S5.T4.1.1.1.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r"><span id="S5.T4.1.1.1.1.1" class="ltx_text ltx_font_bold" style="color:#FF0000;">(<math id="S5.T4.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\Delta" display="inline"><semantics id="S5.T4.1.1.1.1.1.m1.1a"><mi mathcolor="#FF0000" mathvariant="normal" id="S5.T4.1.1.1.1.1.m1.1.1" xref="S5.T4.1.1.1.1.1.m1.1.1.cmml">Δ</mi><annotation-xml encoding="MathML-Content" id="S5.T4.1.1.1.1.1.m1.1b"><ci id="S5.T4.1.1.1.1.1.m1.1.1.cmml" xref="S5.T4.1.1.1.1.1.m1.1.1">Δ</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.T4.1.1.1.1.1.m1.1c">\Delta</annotation></semantics></math>)</span></td>
<td id="S5.T4.1.1.1.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r">-</td>
<td id="S5.T4.1.1.1.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T4.1.1.1.4.1" class="ltx_text" style="color:#FF0000;">(-0.02)</span></td>
<td id="S5.T4.1.1.1.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T4.1.1.1.5.1" class="ltx_text" style="color:#FF0000;">(-0.13)</span></td>
<td id="S5.T4.1.1.1.6" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T4.1.1.1.6.1" class="ltx_text" style="color:#FF0000;">(-0.03)</span></td>
<td id="S5.T4.1.1.1.7" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T4.1.1.1.7.1" class="ltx_text" style="color:#FF0000;">(-0.30)</span></td>
<td id="S5.T4.1.1.1.8" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T4.1.1.1.8.1" class="ltx_text" style="color:#FF0000;">(-0.05)</span></td>
<td id="S5.T4.1.1.1.9" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T4.1.1.1.9.1" class="ltx_text" style="color:#FF0000;">(-0.09)</span></td>
<td id="S5.T4.1.1.1.10" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T4.1.1.1.10.1" class="ltx_text" style="color:#FF0000;">(-0.03)</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 4: </span>Ablation results of Gemini 1.5 Pro on different tasks in <span id="S5.T4.4.1" class="ltx_text ltx_font_smallcaps">LOFT</span> at 128k context length.
Starting from our best prompt format (used in the rest of the experiments), individual facets of the corpus, query, and instruction are ablated to surface their relative effect on quality.
Since ArguAna and FIQA do not have any title for each passage, we report the average and its difference without these two datasets for the <span id="S5.T4.5.2" class="ltx_text ltx_font_typewriter">Titles Only</span> ablation.
</figcaption>
</figure>
<div id="S5.p2" class="ltx_para">
<p id="S5.p2.1" class="ltx_p">The ablations show the effectiveness of our CiC prompting design.
Removing task-specific instructions (<span id="S5.p2.1.1" class="ltx_text ltx_font_typewriter">Generic Instruction</span>) or Chain-of-Thought reasoning (<span id="S5.p2.1.2" class="ltx_text ltx_font_typewriter">Without CoT</span>) both lead to worse performance. We also observe a performance decrease for <span id="S5.p2.1.3" class="ltx_text ltx_font_typewriter">Corpus in Each Few-Shot</span>. In this setting, instead of using a shared corpus, each few-shot example has its own small corpus consisting of nine random passages and one gold passage. This performance degradation could be either because the few-shot examples help the model attend to the test corpus or because the few-shot task becomes much easier than the evaluation task.</p>
</div>
<div id="S5.p3" class="ltx_para">
<p id="S5.p3.1" class="ltx_p">Placing the query at the beginning of the prompt instead of at the end (<span id="S5.p3.1.1" class="ltx_text ltx_font_typewriter">Query at Beginning</span>) led to a significant and consistent performance decrease across tasks and models.
This result suggests that prefix-caching actually works better than encoding the corpus conditioned on each query, which would be much more expensive.</p>
</div>
<figure id="S5.F8" class="ltx_figure"><img src="/html/2406.13121/assets/x8.png" id="S5.F8.g1" class="ltx_graphics ltx_centering ltx_img_square" width="156" height="178" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 8: </span>

<span id="S5.F8.2.1" class="ltx_text ltx_font_bold">Effect of the number of few-shot examples</span>.
The performance increases with the number of few-shot examples.
</figcaption>
</figure>
<div id="S5.p4" class="ltx_para">
<p id="S5.p4.1" class="ltx_p">The presentation of document IDs also affects performance. In particular, replacing monotonic numerical IDs with random (<span id="S5.p4.1.1" class="ltx_text ltx_font_typewriter">Alphanumeric IDs</span>) negatively impacts performance in most datasets. This could possibly be due to way in which numbers are tokenized, with fewer tokens for certain numbers.
Only placing the IDs at the front of the document instead of at the front and the back (<span id="S5.p4.1.2" class="ltx_text ltx_font_typewriter">Without ID Echo</span>) also resulted in a 5% performance drop, confirming that repeating text can compensate for missing context in autoregressive language models  <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib48" title="" class="ltx_ref">48</a>]</cite>.</p>
</div>
<div id="S5.p5" class="ltx_para">
<p id="S5.p5.1" class="ltx_p">To test if the model is simply using parametric knowledge instead of grounding on the context, we remove the document content and simply keep the document title and ID in the corpus (<span id="S5.p5.1.1" class="ltx_text ltx_font_typewriter">Title Only</span>). In this case, it would imply the model is able to perform well because it has already seen all of the datasets we are evaluating on during training. Across all experiments, this ablation significantly degraded performance, indicating the model indeed relies on the provided context.</p>
</div>
<div id="S5.p6" class="ltx_para">
<p id="S5.p6.1" class="ltx_p">We also study how the number of few-shot examples in the prompt affects quality.
The results can be found in <a href="#S5.F8" title="In 5 CiC Prompt Ablations ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">8</span></a>.
Increasing the number of examples improves the quality on the retrieval task, from 0.76 at zero-shot to 0.81 at 5-shots.
Finally, qualitative analysis on the model outputs is provided in <a href="#A7" title="Appendix G Qualitative Analysis of Long-Context Wins and Losses ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Appendix</span> <span class="ltx_text ltx_ref_tag">G</span></a>.</p>
</div>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Related Work</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">Evaluating LCLMs remains a challenge due to the limitations of existing benchmarks.
Many popular datasets and methods rely on synthetic tasks <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib50" title="" class="ltx_ref">50</a>]</cite> such as the popular "Needle-in-A-Haystack" retrieval <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib25" title="" class="ltx_ref">25</a>]</cite> or its extension to multi-hop QA <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib32" title="" class="ltx_ref">32</a>]</cite>.
While these evaluations are also scalable to arbitrarily long contexts, they do not fully capture the nuances of real-world retrieval or reasoning tasks <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib23" title="" class="ltx_ref">23</a>]</cite>.
Some recent benchmarks leverage existing NLP datasets for tasks such as extreme summarization and multi-document QA <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib6" title="" class="ltx_ref">6</a>]</cite>.
However, these tasks lack the dynamic scaling capabilities of synthetic benchmarks, which makes them difficult to adapt to very long contexts.</p>
</div>
<div id="S6.p2" class="ltx_para">
<p id="S6.p2.1" class="ltx_p">LongAlpaca <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib13" title="" class="ltx_ref">13</a>]</cite> and LongBench-Chat <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib5" title="" class="ltx_ref">5</a>]</cite> evaluate instruction-following under long context settings but contain relatively low task diversity and no examples beyond 100k context length. Similar to <span id="S6.p2.1.1" class="ltx_text ltx_font_smallcaps">LOFT</span>, Ada-LEval <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib58" title="" class="ltx_ref">58</a>]</cite> proposes a length-adaptable benchmark; however, their tasks are somewhat synthetic and may not resemble real-world applications.</p>
</div>
<div id="S6.p3" class="ltx_para">
<p id="S6.p3.1" class="ltx_p">Also related to our work is <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib37" title="" class="ltx_ref">37</a>]</cite>, which applies LCLMs to long-context QA using the top retrieved documents from Natural Questions, similar to specialized models for RAG in <span id="S6.p3.1.1" class="ltx_text ltx_font_smallcaps">LOFT</span>.
They find that LCLMs lose recall when relevant information is placed in the middle of the context.
However, their analysis is limited to contexts that are under 10k tokens.
We extend this type of evaluation of LCLMs to context lengths of up to 1M tokens in addition to using multiple modalities and additional forms of reasoning. Finally, our work connects to the field of generative retrieval  <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib16" title="" class="ltx_ref">16</a>, <a href="#bib.bib51" title="" class="ltx_ref">51</a>]</cite>, where models are trained to memorize and generate retrieval targets. Our research offers an alternative approach where the retrieval corpus is directly provided as context, eliminating task specific training.</p>
</div>
</section>
<section id="S7" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">7 </span>Conclusion</h2>

<div id="S7.p1" class="ltx_para">
<p id="S7.p1.1" class="ltx_p">As language models improve and scale, their ability to retrieve and reason over increasingly long context will unlock unprecedented use-cases.
To measure this progress, we introduce <span id="S7.p1.1.1" class="ltx_text ltx_font_smallcaps">LOFT</span>, the Long Context Frontiers benchmark.
<span id="S7.p1.1.2" class="ltx_text ltx_font_smallcaps">LOFT</span> is a suite of tasks that rigorously assesses LCLMs on tasks ripe for a paradigm shift: retrieval, retrieval-augmented generation, SQL-like reasoning, and in-context learning.
<span id="S7.p1.1.3" class="ltx_text ltx_font_smallcaps">LOFT</span> provides dynamic scaling of context lengths of up to 1 million tokens, ensuring that evaluations remain relevant as LCLMs continue to evolve. The tasks can be further scaled up with minimal effort to 1 billion tokens as context lengths continue to increase.
Initial findings demonstrate that despite having never been trained to do retrieval, LCLMs have retrieval capabilities rivaling task-specific hand-crafted SOTA retrieval systems.
Nevertheless, there remains considerable room for advancement in long-context reasoning, particularly as models gain access to even longer context windows.
We believe that <span id="S7.p1.1.4" class="ltx_text ltx_font_smallcaps">LOFT</span> provides a fertile testing ground for measuring progress in long-context modeling.</p>
</div>
<section id="S7.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Limitations</h4>

<div id="S7.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S7.SS0.SSS0.Px1.p1.5" class="ltx_p">Our experiments were constrained by the computational resources and financial costs associated with utilizing LCLMs.
The entire LOFT 128k test sets contain around <math id="S7.SS0.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="35\text{ datasets}\times 100\text{ prompts}\times 128\text{k}\text{ tokens}=448\text{M}" display="inline"><semantics id="S7.SS0.SSS0.Px1.p1.1.m1.1a"><mrow id="S7.SS0.SSS0.Px1.p1.1.m1.1.1" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.cmml"><mrow id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.cmml"><mrow id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.cmml"><mrow id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.cmml"><mrow id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2.cmml"><mrow id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2.2" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2.2.cmml"><mn id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2.2.2" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2.2.2.cmml">35</mn><mo lspace="0em" rspace="0em" id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2.2.1" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2.2.1.cmml">​</mo><mtext id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2.2.3" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2.2.3a.cmml"> datasets</mtext></mrow><mo lspace="0.222em" rspace="0.222em" id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2.1" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2.1.cmml">×</mo><mn id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2.3" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2.3.cmml">100</mn></mrow><mo lspace="0em" rspace="0em" id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.1" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.1.cmml">​</mo><mtext id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.3" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.3a.cmml"> prompts</mtext></mrow><mo lspace="0.222em" rspace="0.222em" id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.1" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.1.cmml">×</mo><mn id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.3" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.3.cmml">128</mn></mrow><mo lspace="0em" rspace="0em" id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.1" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.1.cmml">​</mo><mrow id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.3" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.3c.cmml"><mtext id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.3a" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.3c.cmml">k</mtext><mtext id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.3b" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.3c.cmml"> tokens</mtext></mrow></mrow><mo id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.1" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.1.cmml">=</mo><mrow id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.3" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.3.cmml"><mn id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.3.2" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.3.2.cmml">448</mn><mo lspace="0em" rspace="0em" id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.3.1" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.3.1.cmml">​</mo><mtext id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.3.3" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.3.3a.cmml">M</mtext></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.SS0.SSS0.Px1.p1.1.m1.1b"><apply id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1"><eq id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.1.cmml" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.1"></eq><apply id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.cmml" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2"><times id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.1.cmml" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.1"></times><apply id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.cmml" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2"><times id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.1.cmml" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.1"></times><apply id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.cmml" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2"><times id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.1.cmml" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.1"></times><apply id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2.cmml" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2"><times id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2.1.cmml" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2.1"></times><apply id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2.2.cmml" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2.2"><times id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2.2.1.cmml" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2.2.1"></times><cn type="integer" id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2.2.2.cmml" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2.2.2">35</cn><ci id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2.2.3a.cmml" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2.2.3"><mtext id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2.2.3.cmml" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2.2.3"> datasets</mtext></ci></apply><cn type="integer" id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2.3.cmml" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.2.3">100</cn></apply><ci id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.3a.cmml" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.3"><mtext id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.3.cmml" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.2.3"> prompts</mtext></ci></apply><cn type="integer" id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.3.cmml" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.3">128</cn></apply><ci id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.3c.cmml" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.3"><mrow id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.3.cmml" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.3"><mtext id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.3a.cmml" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.3">k</mtext><mtext id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.3b.cmml" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.2.3"> tokens</mtext></mrow></ci></apply><apply id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.3.cmml" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.3"><times id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.3.1.cmml" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.3.1"></times><cn type="integer" id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.3.2.cmml" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.3.2">448</cn><ci id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.3.3a.cmml" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.3.3"><mtext id="S7.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.cmml" xref="S7.SS0.SSS0.Px1.p1.1.m1.1.1.3.3">M</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS0.SSS0.Px1.p1.1.m1.1c">35\text{ datasets}\times 100\text{ prompts}\times 128\text{k}\text{ tokens}=448\text{M}</annotation></semantics></math> input tokens, which cost <math id="S7.SS0.SSS0.Px1.p1.2.m2.2" class="ltx_Math" alttext="\$1,568" display="inline"><semantics id="S7.SS0.SSS0.Px1.p1.2.m2.2a"><mrow id="S7.SS0.SSS0.Px1.p1.2.m2.2.2.1" xref="S7.SS0.SSS0.Px1.p1.2.m2.2.2.2.cmml"><mrow id="S7.SS0.SSS0.Px1.p1.2.m2.2.2.1.1" xref="S7.SS0.SSS0.Px1.p1.2.m2.2.2.1.1.cmml"><mo rspace="0.167em" id="S7.SS0.SSS0.Px1.p1.2.m2.2.2.1.1.1" xref="S7.SS0.SSS0.Px1.p1.2.m2.2.2.1.1.1.cmml">$</mo><mn id="S7.SS0.SSS0.Px1.p1.2.m2.2.2.1.1.2" xref="S7.SS0.SSS0.Px1.p1.2.m2.2.2.1.1.2.cmml">1</mn></mrow><mo id="S7.SS0.SSS0.Px1.p1.2.m2.2.2.1.2" xref="S7.SS0.SSS0.Px1.p1.2.m2.2.2.2.cmml">,</mo><mn id="S7.SS0.SSS0.Px1.p1.2.m2.1.1" xref="S7.SS0.SSS0.Px1.p1.2.m2.1.1.cmml">568</mn></mrow><annotation-xml encoding="MathML-Content" id="S7.SS0.SSS0.Px1.p1.2.m2.2b"><list id="S7.SS0.SSS0.Px1.p1.2.m2.2.2.2.cmml" xref="S7.SS0.SSS0.Px1.p1.2.m2.2.2.1"><apply id="S7.SS0.SSS0.Px1.p1.2.m2.2.2.1.1.cmml" xref="S7.SS0.SSS0.Px1.p1.2.m2.2.2.1.1"><csymbol cd="latexml" id="S7.SS0.SSS0.Px1.p1.2.m2.2.2.1.1.1.cmml" xref="S7.SS0.SSS0.Px1.p1.2.m2.2.2.1.1.1">currency-dollar</csymbol><cn type="integer" id="S7.SS0.SSS0.Px1.p1.2.m2.2.2.1.1.2.cmml" xref="S7.SS0.SSS0.Px1.p1.2.m2.2.2.1.1.2">1</cn></apply><cn type="integer" id="S7.SS0.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S7.SS0.SSS0.Px1.p1.2.m2.1.1">568</cn></list></annotation-xml><annotation encoding="application/x-tex" id="S7.SS0.SSS0.Px1.p1.2.m2.2c">\$1,568</annotation></semantics></math> for Gemini 1.5 Pro, <math id="S7.SS0.SSS0.Px1.p1.3.m3.2" class="ltx_Math" alttext="\$2,240" display="inline"><semantics id="S7.SS0.SSS0.Px1.p1.3.m3.2a"><mrow id="S7.SS0.SSS0.Px1.p1.3.m3.2.2.1" xref="S7.SS0.SSS0.Px1.p1.3.m3.2.2.2.cmml"><mrow id="S7.SS0.SSS0.Px1.p1.3.m3.2.2.1.1" xref="S7.SS0.SSS0.Px1.p1.3.m3.2.2.1.1.cmml"><mo rspace="0.167em" id="S7.SS0.SSS0.Px1.p1.3.m3.2.2.1.1.1" xref="S7.SS0.SSS0.Px1.p1.3.m3.2.2.1.1.1.cmml">$</mo><mn id="S7.SS0.SSS0.Px1.p1.3.m3.2.2.1.1.2" xref="S7.SS0.SSS0.Px1.p1.3.m3.2.2.1.1.2.cmml">2</mn></mrow><mo id="S7.SS0.SSS0.Px1.p1.3.m3.2.2.1.2" xref="S7.SS0.SSS0.Px1.p1.3.m3.2.2.2.cmml">,</mo><mn id="S7.SS0.SSS0.Px1.p1.3.m3.1.1" xref="S7.SS0.SSS0.Px1.p1.3.m3.1.1.cmml">240</mn></mrow><annotation-xml encoding="MathML-Content" id="S7.SS0.SSS0.Px1.p1.3.m3.2b"><list id="S7.SS0.SSS0.Px1.p1.3.m3.2.2.2.cmml" xref="S7.SS0.SSS0.Px1.p1.3.m3.2.2.1"><apply id="S7.SS0.SSS0.Px1.p1.3.m3.2.2.1.1.cmml" xref="S7.SS0.SSS0.Px1.p1.3.m3.2.2.1.1"><csymbol cd="latexml" id="S7.SS0.SSS0.Px1.p1.3.m3.2.2.1.1.1.cmml" xref="S7.SS0.SSS0.Px1.p1.3.m3.2.2.1.1.1">currency-dollar</csymbol><cn type="integer" id="S7.SS0.SSS0.Px1.p1.3.m3.2.2.1.1.2.cmml" xref="S7.SS0.SSS0.Px1.p1.3.m3.2.2.1.1.2">2</cn></apply><cn type="integer" id="S7.SS0.SSS0.Px1.p1.3.m3.1.1.cmml" xref="S7.SS0.SSS0.Px1.p1.3.m3.1.1">240</cn></list></annotation-xml><annotation encoding="application/x-tex" id="S7.SS0.SSS0.Px1.p1.3.m3.2c">\$2,240</annotation></semantics></math> for GPT-4o, and <math id="S7.SS0.SSS0.Px1.p1.4.m4.2" class="ltx_Math" alttext="\$6,720" display="inline"><semantics id="S7.SS0.SSS0.Px1.p1.4.m4.2a"><mrow id="S7.SS0.SSS0.Px1.p1.4.m4.2.2.1" xref="S7.SS0.SSS0.Px1.p1.4.m4.2.2.2.cmml"><mrow id="S7.SS0.SSS0.Px1.p1.4.m4.2.2.1.1" xref="S7.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.cmml"><mo rspace="0.167em" id="S7.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.1" xref="S7.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.1.cmml">$</mo><mn id="S7.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.2" xref="S7.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.2.cmml">6</mn></mrow><mo id="S7.SS0.SSS0.Px1.p1.4.m4.2.2.1.2" xref="S7.SS0.SSS0.Px1.p1.4.m4.2.2.2.cmml">,</mo><mn id="S7.SS0.SSS0.Px1.p1.4.m4.1.1" xref="S7.SS0.SSS0.Px1.p1.4.m4.1.1.cmml">720</mn></mrow><annotation-xml encoding="MathML-Content" id="S7.SS0.SSS0.Px1.p1.4.m4.2b"><list id="S7.SS0.SSS0.Px1.p1.4.m4.2.2.2.cmml" xref="S7.SS0.SSS0.Px1.p1.4.m4.2.2.1"><apply id="S7.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.cmml" xref="S7.SS0.SSS0.Px1.p1.4.m4.2.2.1.1"><csymbol cd="latexml" id="S7.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.1.cmml" xref="S7.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.1">currency-dollar</csymbol><cn type="integer" id="S7.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.2.cmml" xref="S7.SS0.SSS0.Px1.p1.4.m4.2.2.1.1.2">6</cn></apply><cn type="integer" id="S7.SS0.SSS0.Px1.p1.4.m4.1.1.cmml" xref="S7.SS0.SSS0.Px1.p1.4.m4.1.1">720</cn></list></annotation-xml><annotation encoding="application/x-tex" id="S7.SS0.SSS0.Px1.p1.4.m4.2c">\$6,720</annotation></semantics></math> for Claude 3 Opus at the time of writing. To reduce costs, we also release dev sets, which are 10x smaller and can be evaluated with around <math id="S7.SS0.SSS0.Px1.p1.5.m5.1" class="ltx_Math" alttext="\$200" display="inline"><semantics id="S7.SS0.SSS0.Px1.p1.5.m5.1a"><mrow id="S7.SS0.SSS0.Px1.p1.5.m5.1.1" xref="S7.SS0.SSS0.Px1.p1.5.m5.1.1.cmml"><mo rspace="0.167em" id="S7.SS0.SSS0.Px1.p1.5.m5.1.1.1" xref="S7.SS0.SSS0.Px1.p1.5.m5.1.1.1.cmml">$</mo><mn id="S7.SS0.SSS0.Px1.p1.5.m5.1.1.2" xref="S7.SS0.SSS0.Px1.p1.5.m5.1.1.2.cmml">200</mn></mrow><annotation-xml encoding="MathML-Content" id="S7.SS0.SSS0.Px1.p1.5.m5.1b"><apply id="S7.SS0.SSS0.Px1.p1.5.m5.1.1.cmml" xref="S7.SS0.SSS0.Px1.p1.5.m5.1.1"><csymbol cd="latexml" id="S7.SS0.SSS0.Px1.p1.5.m5.1.1.1.cmml" xref="S7.SS0.SSS0.Px1.p1.5.m5.1.1.1">currency-dollar</csymbol><cn type="integer" id="S7.SS0.SSS0.Px1.p1.5.m5.1.1.2.cmml" xref="S7.SS0.SSS0.Px1.p1.5.m5.1.1.2">200</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS0.SSS0.Px1.p1.5.m5.1c">\$200</annotation></semantics></math> using Gemini 1.5 Pro or GPT-4o. We also expect LLM API prices to decrease over time.
Another limitation of this work is that we focused on evaluating the quality of LCLMs, and leave efficiency considerations for future work. We could not measure the efficiency improvements from prefix caching <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib20" title="" class="ltx_ref">20</a>]</cite> due to API constraints at the time of writing. Without caching, the Gemini 1.5 Pro API has a median latency of roughly four seconds for 32k input tokens, twelve seconds for 128k input tokens, and 100 seconds for 1 million input tokens. This speed is likely slower than specialized retrievers or SQL databases; the promising quality results on <span id="S7.SS0.SSS0.Px1.p1.5.1" class="ltx_text ltx_font_smallcaps">LOFT</span> encourage further investigation into optimizing LCLMs efficiency.
Additionally, our retrieval and RAG tasks was limited to 1 million tokens, which still leaves a large gap from real-world applications that may involve several million or even billions of documents.</p>
</div>
</section>
</section>
<section id="Sx1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Acknowledgement</h2>

<div id="Sx1.p1" class="ltx_para">
<p id="Sx1.p1.1" class="ltx_p">We are grateful to Frank Palma Gomez for providing the audio retrieval model, and to Raoul de Liedekerke and Miteyan Patel for their invaluable help with the API-based model evaluation. We thank Nicholas Monath, Raphael Hoffmann, Slav Petrov, Urvashi Khandelwal, and Kristina Toutanova for their thoughtful discussions and insightful feedback to the paper.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[1]</span>
<span class="ltx_bibblock">
Vaibhav Adlakha, Parishad BehnamGhader, Xing Han Lu, Nicholas Meade, and Siva
Reddy.

</span>
<span class="ltx_bibblock">Evaluating Correctness and Faithfulness of Instruction-Following
Models for Question Answering.

</span>
<span class="ltx_bibblock"><span id="bib.bib1.1.1" class="ltx_text ltx_font_italic">Transactions of the Association for Computational Linguistics</span>,
12:681–699, 2024.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[2]</span>
<span class="ltx_bibblock">
Vaibhav Adlakha, Shehzaad Dhuliawala, Kaheer Suleman, Harm de Vries, and Siva
Reddy.

</span>
<span class="ltx_bibblock">TopiOCQA: Open-domain conversational question answering with
topic switching.

</span>
<span class="ltx_bibblock"><span id="bib.bib2.1.1" class="ltx_text ltx_font_italic">Transactions of the Association for Computational Linguistics</span>,
10:468–483, 2022.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[3]</span>
<span class="ltx_bibblock">
Samuel Joseph Amouyal, Tomer Wolfson, Ohad Rubin, Ori Yoran, Jonathan Herzig,
and Jonathan Berant.

</span>
<span class="ltx_bibblock">Qampari: A benchmark for open-domain questions with many answers.

</span>
<span class="ltx_bibblock">In <span id="bib.bib3.1.1" class="ltx_text ltx_font_italic">IEEE Games Entertainment Media Conference</span>, 2022.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[4]</span>
<span class="ltx_bibblock">
Anthropic.

</span>
<span class="ltx_bibblock">The claude 3 model family: Opus, sonnet, haiku.

</span>
<span class="ltx_bibblock"><span id="bib.bib4.1.1" class="ltx_text ltx_font_italic">Claude-3 Model Card</span>, 2024.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[5]</span>
<span class="ltx_bibblock">
Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao
Dong, and Juanzi Li.

</span>
<span class="ltx_bibblock">Longalign: A recipe for long context alignment of large language
models, 2024.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[6]</span>
<span class="ltx_bibblock">
Yushi Bai, Xin Lv, Jiajie Zhang, Hong Lyu, Jiankai Tang, Zhidian Huang,
Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and
Juanzi Li.

</span>
<span class="ltx_bibblock">Longbench: A bilingual, multitask benchmark for long context
understanding.

</span>
<span class="ltx_bibblock"><span id="bib.bib6.1.1" class="ltx_text ltx_font_italic">ArXiv preprint</span>, abs/2308.14508, 2023.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[7]</span>
<span class="ltx_bibblock">
Payal Bajaj, Daniel Fernando Campos, Nick Craswell, Li Deng, Jianfeng Gao,
Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Minh
Nguyen, Mir Rosenberg, Xia Song, Alina Mihaela Stoica, Saurabh Tiwary, and
Tong Wang.

</span>
<span class="ltx_bibblock">Ms marco: A human generated machine reading comprehension dataset.

</span>
<span class="ltx_bibblock"><span id="bib.bib7.1.1" class="ltx_text ltx_font_italic">arXiv: Computation and Language</span>, 2016.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[8]</span>
<span class="ltx_bibblock">
Scott Barnett, Stefanus Kurniawan, Srikanth Thudumu, Zach Brannelly, and
Mohamed Abdelrazek.

</span>
<span class="ltx_bibblock">Seven failure points when engineering a retrieval augmented
generation system.

</span>
<span class="ltx_bibblock"><span id="bib.bib8.1.1" class="ltx_text ltx_font_italic">ArXiv preprint</span>, abs/2401.05856, 2024.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[9]</span>
<span class="ltx_bibblock">
Iz Beltagy, Matthew E Peters, and Arman Cohan.

</span>
<span class="ltx_bibblock">Longformer: The long-document transformer.

</span>
<span class="ltx_bibblock"><span id="bib.bib9.1.1" class="ltx_text ltx_font_italic">ArXiv preprint</span>, abs/2004.05150, 2020.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[10]</span>
<span class="ltx_bibblock">
BIG bench authors.

</span>
<span class="ltx_bibblock">Beyond the imitation game: Quantifying and extrapolating the
capabilities of language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib10.1.1" class="ltx_text ltx_font_italic">Transactions on Machine Learning Research</span>, 2023.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[11]</span>
<span class="ltx_bibblock">
Alexander Bondarenko, Maik Fröbe, Meriem Beloucif, Lukas Gienapp, Yamen
Ajjour, Alexander Panchenko, Christian Biemann, Benno Stein, Henning
Wachsmuth, Martin Potthast, and Matthias Hagen.

</span>
<span class="ltx_bibblock">Overview of touché 2020: Argument retrieval.

</span>
<span class="ltx_bibblock">In <span id="bib.bib11.1.1" class="ltx_text ltx_font_italic">Conference and Labs of the Evaluation Forum</span>, 2020.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[12]</span>
<span class="ltx_bibblock">
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan,
Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda
Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom
Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens
Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott
Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec
Radford, Ilya Sutskever, and Dario Amodei.

</span>
<span class="ltx_bibblock">Language models are few-shot learners.

</span>
<span class="ltx_bibblock">In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell,
Maria-Florina Balcan, and Hsuan-Tien Lin, editors, <span id="bib.bib12.1.1" class="ltx_text ltx_font_italic">Advances in
Neural Information Processing Systems 33: Annual Conference on Neural
Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020,
virtual</span>, 2020.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[13]</span>
<span class="ltx_bibblock">
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and
Jiaya Jia.

</span>
<span class="ltx_bibblock">Longlora: Efficient fine-tuning of long-context large language
models, 2024.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[14]</span>
<span class="ltx_bibblock">
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus,
Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson,
Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha
Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter,
Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew
Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam
Roberts, Denny Zhou, Quoc V. Le, and Jason Wei.

</span>
<span class="ltx_bibblock">Scaling instruction-finetuned language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib14.1.1" class="ltx_text ltx_font_italic">Journal of Machine Learning Research</span>, 25(70):1–53, 2024.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[15]</span>
<span class="ltx_bibblock">
Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth
Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna.

</span>
<span class="ltx_bibblock">Fleurs: Few-shot learning evaluation of universal representations of
speech.

</span>
<span class="ltx_bibblock"><span id="bib.bib15.1.1" class="ltx_text ltx_font_italic">2022 IEEE Spoken Language Technology Workshop (SLT)</span>, pages
798–805, 2022.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[16]</span>
<span class="ltx_bibblock">
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni.

</span>
<span class="ltx_bibblock">Autoregressive entity retrieval.

</span>
<span class="ltx_bibblock"><span id="bib.bib16.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2010.00904</span>, 2020.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[17]</span>
<span class="ltx_bibblock">
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and
Matt Gardner.

</span>
<span class="ltx_bibblock">DROP: A reading comprehension benchmark requiring discrete
reasoning over paragraphs.

</span>
<span class="ltx_bibblock">In <span id="bib.bib17.1.1" class="ltx_text ltx_font_italic">Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers)</span>, pages 2368–2378,
Minneapolis, Minnesota, 2019. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[18]</span>
<span class="ltx_bibblock">
Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and
Jingren Zhou.

</span>
<span class="ltx_bibblock">Text-to-sql empowered by large language models: A benchmark
evaluation.

</span>
<span class="ltx_bibblock"><span id="bib.bib18.1.1" class="ltx_text ltx_font_italic">ArXiv preprint</span>, abs/2308.15363, 2023.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[19]</span>
<span class="ltx_bibblock">
Frank Palma Gomez, Ramon Sanabria, Yun-hsuan Sung, Daniel Cer, Siddharth
Dalmia, and Gustavo Hernandez Abrego.

</span>
<span class="ltx_bibblock">Transforming llms into cross-modal and cross-lingual
retrievalsystems.

</span>
<span class="ltx_bibblock">In <span id="bib.bib19.1.1" class="ltx_text ltx_font_italic">Proceedings of the 21st International Conference on Spoken
Language Translation (IWSLT 2024)</span>, 2024.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[20]</span>
<span class="ltx_bibblock">
Google.

</span>
<span class="ltx_bibblock">Context caching guide.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://ai.google.dev/gemini-api/docs/caching" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://ai.google.dev/gemini-api/docs/caching</a>, 2024.

</span>
<span class="ltx_bibblock">Accessed: 2024-06-05.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[21]</span>
<span class="ltx_bibblock">
Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan
Sung, and Yinfei Yang.

</span>
<span class="ltx_bibblock">LongT5: Efficient text-to-text transformer for long sequences.

</span>
<span class="ltx_bibblock">In <span id="bib.bib21.1.1" class="ltx_text ltx_font_italic">Findings of the Association for Computational Linguistics:
NAACL 2022</span>, pages 724–736, Seattle, United States, 2022. Association for
Computational Linguistics.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[22]</span>
<span class="ltx_bibblock">
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang.

</span>
<span class="ltx_bibblock">Realm: retrieval-augmented language model pre-training.

</span>
<span class="ltx_bibblock">In <span id="bib.bib22.1.1" class="ltx_text ltx_font_italic">Proceedings of the 37th International Conference on Machine
Learning</span>, ICML’20. JMLR.org, 2020.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[23]</span>
<span class="ltx_bibblock">
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei
Jia, and Boris Ginsburg.

</span>
<span class="ltx_bibblock">Ruler: What’s the real context size of your long-context language
models?

</span>
<span class="ltx_bibblock"><span id="bib.bib23.1.1" class="ltx_text ltx_font_italic">ArXiv</span>, 2024.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[24]</span>
<span class="ltx_bibblock">
Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee,
Kristina Toutanova, and Ming-Wei Chang.

</span>
<span class="ltx_bibblock">Open-domain visual entity recognition: Towards recognizing millions
of wikipedia entities.

</span>
<span class="ltx_bibblock"><span id="bib.bib24.1.1" class="ltx_text ltx_font_italic">2023 IEEE/CVF International Conference on Computer Vision
(ICCV)</span>, pages 12031–12041, 2023.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[25]</span>
<span class="ltx_bibblock">
Greg Kamradt.

</span>
<span class="ltx_bibblock">Needle in a haystack - pressure testing llms, 2023.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[26]</span>
<span class="ltx_bibblock">
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey
Edunov, Danqi Chen, and Wen-tau Yih.

</span>
<span class="ltx_bibblock">Dense passage retrieval for open-domain question answering.

</span>
<span class="ltx_bibblock">In <span id="bib.bib26.1.1" class="ltx_text ltx_font_italic">Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP)</span>, pages 6769–6781, Online, 2020.
Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[27]</span>
<span class="ltx_bibblock">
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke
Iwasawa.

</span>
<span class="ltx_bibblock">Large language models are zero-shot reasoners.

</span>
<span class="ltx_bibblock">In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh,
editors, <span id="bib.bib27.1.1" class="ltx_text ltx_font_italic">Advances in Neural Information Processing Systems</span>, volume 35,
pages 22199–22213. Curran Associates, Inc., 2022.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[28]</span>
<span class="ltx_bibblock">
Taku Kudo and John Richardson.

</span>
<span class="ltx_bibblock">SentencePiece: A simple and language independent subword
tokenizer and detokenizer for neural text processing.

</span>
<span class="ltx_bibblock">In <span id="bib.bib28.1.1" class="ltx_text ltx_font_italic">Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing: System Demonstrations</span>, pages 66–71, Brussels,
Belgium, 2018. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[29]</span>
<span class="ltx_bibblock">
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur
Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin,
Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang,
Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov.

</span>
<span class="ltx_bibblock">Natural questions: A benchmark for question answering research.

</span>
<span class="ltx_bibblock"><span id="bib.bib29.1.1" class="ltx_text ltx_font_italic">Transactions of the Association for Computational Linguistics</span>,
7:452–466, 2019.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[30]</span>
<span class="ltx_bibblock">
Haejun Lee, Akhil Kedia, Jongwon Lee, Ashwin Paranjape, Christopher Manning,
and Kyoung-Gu Woo.

</span>
<span class="ltx_bibblock">You only need one model for open-domain question answering.

</span>
<span class="ltx_bibblock">In <span id="bib.bib30.1.1" class="ltx_text ltx_font_italic">Proceedings of the 2022 Conference on Empirical Methods in
Natural Language Processing</span>, pages 3047–3060, Abu Dhabi, United Arab
Emirates, 2022. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[31]</span>
<span class="ltx_bibblock">
Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R. Cole,
Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, Yi Luan, Sai Meher Karthik
Duddu, Gustavo Hernández Abrego, Weiqiang Shi, Nithi Gupta, Aditya
Kusupati, Prateek Jain, Siddhartha R. Jonnalagadda, Ming-Wei Chang, and
Iftekhar Naim.

</span>
<span class="ltx_bibblock">Gecko: Versatile text embeddings distilled from large language
models.

</span>
<span class="ltx_bibblock"><span id="bib.bib31.1.1" class="ltx_text ltx_font_italic">ArXiv preprint</span>, abs/2403.20327, 2024.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[32]</span>
<span class="ltx_bibblock">
Mosh Levy, Alon Jacoby, and Yoav Goldberg.

</span>
<span class="ltx_bibblock">Same task, more tokens: the impact of input length on the reasoning
performance of large language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib32.1.1" class="ltx_text ltx_font_italic">ArXiv preprint</span>, abs/2402.14848, 2024.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[33]</span>
<span class="ltx_bibblock">
Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel.

</span>
<span class="ltx_bibblock">Question and answer test-train overlap in open-domain question
answering datasets.

</span>
<span class="ltx_bibblock">In <span id="bib.bib33.1.1" class="ltx_text ltx_font_italic">Proceedings of the 16th Conference of the European Chapter of
the Association for Computational Linguistics: Main Volume</span>, pages
1000–1008, Online, 2021. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[34]</span>
<span class="ltx_bibblock">
Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir
Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih,
Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela.

</span>
<span class="ltx_bibblock">Retrieval-augmented generation for knowledge-intensive NLP tasks.

</span>
<span class="ltx_bibblock">In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell,
Maria-Florina Balcan, and Hsuan-Tien Lin, editors, <span id="bib.bib34.1.1" class="ltx_text ltx_font_italic">Advances in
Neural Information Processing Systems 33: Annual Conference on Neural
Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020,
virtual</span>, 2020.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[35]</span>
<span class="ltx_bibblock">
Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen.

</span>
<span class="ltx_bibblock">Long-context llms struggle with long in-context learning.

</span>
<span class="ltx_bibblock"><span id="bib.bib35.1.1" class="ltx_text ltx_font_italic">ArXiv preprint</span>, abs/2404.02060, 2024.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[36]</span>
<span class="ltx_bibblock">
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva
Ramanan, Piotr Dollár, and C Lawrence Zitnick.

</span>
<span class="ltx_bibblock">Microsoft coco: Common objects in context.

</span>
<span class="ltx_bibblock">In <span id="bib.bib36.1.1" class="ltx_text ltx_font_italic">Computer Vision–ECCV 2014: 13th European Conference, Zurich,
Switzerland, September 6-12, 2014, Proceedings, Part V 13</span>, pages 740–755.
Springer, 2014.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[37]</span>
<span class="ltx_bibblock">
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua,
Fabio Petroni, and Percy Liang.

</span>
<span class="ltx_bibblock">Lost in the middle: How language models use long contexts.

</span>
<span class="ltx_bibblock"><span id="bib.bib37.1.1" class="ltx_text ltx_font_italic">Transactions of the Association for Computational Linguistics</span>,
12:157–173, 2023.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[38]</span>
<span class="ltx_bibblock">
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois,
and Sameer Singh.

</span>
<span class="ltx_bibblock">Entity-based knowledge conflicts in question answering.

</span>
<span class="ltx_bibblock">In <span id="bib.bib38.1.1" class="ltx_text ltx_font_italic">Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing</span>, pages 7052–7063, Online and Punta Cana,
Dominican Republic, 2021. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[39]</span>
<span class="ltx_bibblock">
Man Luo, Xin Xu, Zhuyun Dai, Panupong Pasupat, Mehran Kazemi, Chitta Baral,
Vaiva Imbrasaite, and Vincent Zhao.

</span>
<span class="ltx_bibblock">Dr.icl: Demonstration-retrieved in-context learning.

</span>
<span class="ltx_bibblock"><span id="bib.bib39.1.1" class="ltx_text ltx_font_italic">ArXiv preprint</span>, abs/2305.14128, 2023.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[40]</span>
<span class="ltx_bibblock">
Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross
McDermott, Manel Zarrouk, and Alexandra Balahur.

</span>
<span class="ltx_bibblock">Www’18 open challenge: Financial opinion mining and question
answering.

</span>
<span class="ltx_bibblock"><span id="bib.bib40.1.1" class="ltx_text ltx_font_italic">Companion Proceedings of the The Web Conference 2018</span>, 2018.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[41]</span>
<span class="ltx_bibblock">
Chaitanya Malaviya, Peter Shaw, Ming-Wei Chang, Kenton Lee, and Kristina
Toutanova.

</span>
<span class="ltx_bibblock">QUEST: A retrieval dataset of entity-seeking queries with implicit
set operations.

</span>
<span class="ltx_bibblock">In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, <span id="bib.bib41.1.1" class="ltx_text ltx_font_italic">Proceedings of the 61st Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers)</span>, pages 14032–14047, Toronto, Canada,
July 2023. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[42]</span>
<span class="ltx_bibblock">
Sewon Min, Kenton Lee, Ming-Wei Chang, Kristina Toutanova, and Hannaneh
Hajishirzi.

</span>
<span class="ltx_bibblock">Joint passage ranking for diverse multi-answer retrieval.

</span>
<span class="ltx_bibblock">In <span id="bib.bib42.1.1" class="ltx_text ltx_font_italic">Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing</span>, pages 6997–7008, Online and Punta Cana,
Dominican Republic, 2021. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[43]</span>
<span class="ltx_bibblock">
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma,
Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, and Yinfei Yang.

</span>
<span class="ltx_bibblock">Large dual encoders are generalizable retrievers.

</span>
<span class="ltx_bibblock">In <span id="bib.bib43.1.1" class="ltx_text ltx_font_italic">Proceedings of the 2022 Conference on Empirical Methods in
Natural Language Processing</span>, pages 9844–9855, Abu Dhabi, United Arab
Emirates, 2022. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[44]</span>
<span class="ltx_bibblock">
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob
Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David
Luan, Charles Sutton, and Augustus Odena.

</span>
<span class="ltx_bibblock">Show your work: Scratchpads for intermediate computation with
language models.

</span>
<span class="ltx_bibblock">In <span id="bib.bib44.1.1" class="ltx_text ltx_font_italic">Deep Learning for Code Workshop</span>, 2022.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[45]</span>
<span class="ltx_bibblock">
OpenAI.

</span>
<span class="ltx_bibblock">Gpt-4 technical report.

</span>
<span class="ltx_bibblock"><span id="bib.bib45.1.1" class="ltx_text ltx_font_italic">ArXiv</span>, 2023.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[46]</span>
<span class="ltx_bibblock">
Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela.

</span>
<span class="ltx_bibblock">Unsupervised question decomposition for question answering.

</span>
<span class="ltx_bibblock">In <span id="bib.bib46.1.1" class="ltx_text ltx_font_italic">Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP)</span>, pages 8864–8880, Online, 2020.
Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[47]</span>
<span class="ltx_bibblock">
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,
Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
Gretchen Krueger, and Ilya Sutskever.

</span>
<span class="ltx_bibblock">Learning transferable visual models from natural language
supervision.

</span>
<span class="ltx_bibblock">In Marina Meila and Tong Zhang, editors, <span id="bib.bib47.1.1" class="ltx_text ltx_font_italic">Proceedings of the 38th
International Conference on Machine Learning, ICML 2021, 18-24 July 2021,
Virtual Event</span>, volume 139 of <span id="bib.bib47.2.2" class="ltx_text ltx_font_italic">Proceedings of Machine Learning Research</span>,
pages 8748–8763. PMLR, 2021.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[48]</span>
<span class="ltx_bibblock">
Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, and Aditi
Raghunathan.

</span>
<span class="ltx_bibblock">Repetition improves language model embeddings.

</span>
<span class="ltx_bibblock"><span id="bib.bib48.1.1" class="ltx_text ltx_font_italic">ArXiv preprint</span>, abs/2402.15449, 2024.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[49]</span>
<span class="ltx_bibblock">
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay,
Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason
Wei.

</span>
<span class="ltx_bibblock">Challenging BIG-bench tasks and whether chain-of-thought can solve
them.

</span>
<span class="ltx_bibblock">In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, <span id="bib.bib49.1.1" class="ltx_text ltx_font_italic">Findings of the Association for Computational Linguistics: ACL 2023</span>, pages
13003–13051, Toronto, Canada, July 2023. Association for Computational
Linguistics.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[50]</span>
<span class="ltx_bibblock">
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham,
Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler.

</span>
<span class="ltx_bibblock">Long range arena : A benchmark for efficient transformers.

</span>
<span class="ltx_bibblock">In <span id="bib.bib50.1.1" class="ltx_text ltx_font_italic">9th International Conference on Learning Representations,
ICLR 2021, Virtual Event, Austria, May 3-7, 2021</span>. OpenReview.net, 2021.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[51]</span>
<span class="ltx_bibblock">
Yi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen
Qin, Kai Hui, Zhe Zhao, Jai Gupta, et al.

</span>
<span class="ltx_bibblock">Transformer memory as a differentiable search index.

</span>
<span class="ltx_bibblock"><span id="bib.bib51.1.1" class="ltx_text ltx_font_italic">Advances in Neural Information Processing Systems</span>,
35:21831–21843, 2022.

</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[52]</span>
<span class="ltx_bibblock">
Gemini Team.

</span>
<span class="ltx_bibblock">Gemini 1.5: Unlocking multimodal understanding across millions of
tokens of context.

</span>
<span class="ltx_bibblock"><span id="bib.bib52.1.1" class="ltx_text ltx_font_italic">ArXiv preprint</span>, abs/2403.05530, 2024.

</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[53]</span>
<span class="ltx_bibblock">
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and
Iryna Gurevych.

</span>
<span class="ltx_bibblock">Beir: A heterogeneous benchmark for zero-shot evaluation of
information retrieval models.

</span>
<span class="ltx_bibblock">In J. Vanschoren and S. Yeung, editors, <span id="bib.bib53.1.1" class="ltx_text ltx_font_italic">Proceedings of the
Neural Information Processing Systems Track on Datasets and Benchmarks</span>,
volume 1, 2021.

</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[54]</span>
<span class="ltx_bibblock">
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal.

</span>
<span class="ltx_bibblock">FEVER: a large-scale dataset for fact extraction and
VERification.

</span>
<span class="ltx_bibblock">In <span id="bib.bib54.1.1" class="ltx_text ltx_font_italic">Proceedings of the 2018 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long Papers)</span>, pages 809–819, New Orleans,
Louisiana, 2018. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib55" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[55]</span>
<span class="ltx_bibblock">
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal.

</span>
<span class="ltx_bibblock">MuSiQue: Multihop questions via single-hop question
composition.

</span>
<span class="ltx_bibblock"><span id="bib.bib55.1.1" class="ltx_text ltx_font_italic">Transactions of the Association for Computational Linguistics</span>,
10:539–554, 2022.

</span>
</li>
<li id="bib.bib56" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[56]</span>
<span class="ltx_bibblock">
Henning Wachsmuth, Shahbaz Syed, and Benno Stein.

</span>
<span class="ltx_bibblock">Retrieval of the best counterargument without prior topic knowledge.

</span>
<span class="ltx_bibblock">In <span id="bib.bib56.1.1" class="ltx_text ltx_font_italic">Proceedings of the 56th Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers)</span>, pages 241–251,
Melbourne, Australia, 2018. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib57" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[57]</span>
<span class="ltx_bibblock">
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman
Cohan, and Hannaneh Hajishirzi.

</span>
<span class="ltx_bibblock">Fact or fiction: Verifying scientific claims.

</span>
<span class="ltx_bibblock">In <span id="bib.bib57.1.1" class="ltx_text ltx_font_italic">Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP)</span>, pages 7534–7550, Online, 2020.
Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib58" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[58]</span>
<span class="ltx_bibblock">
Chonghua Wang, Haodong Duan, Songyang Zhang, Dahua Lin, and Kai Chen.

</span>
<span class="ltx_bibblock">Ada-leval: Evaluating long-context llms with length-adaptable
benchmarks, 2024.

</span>
</li>
<li id="bib.bib59" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[59]</span>
<span class="ltx_bibblock">
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian
Lester, Nan Du, Andrew M. Dai, and Quoc V. Le.

</span>
<span class="ltx_bibblock">Finetuned language models are zero-shot learners.

</span>
<span class="ltx_bibblock">In <span id="bib.bib59.1.1" class="ltx_text ltx_font_italic">The Tenth International Conference on Learning
Representations, ICLR 2022, Virtual Event, April 25-29, 2022</span>.
OpenReview.net, 2022.

</span>
</li>
<li id="bib.bib60" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[60]</span>
<span class="ltx_bibblock">
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi,
F. Xia, Quoc Le, and Denny Zhou.

</span>
<span class="ltx_bibblock">Chain of thought prompting elicits reasoning in large language
models.

</span>
<span class="ltx_bibblock"><span id="bib.bib60.1.1" class="ltx_text ltx_font_italic">ArXiv preprint</span>, abs/2201.11903, 2022.

</span>
</li>
<li id="bib.bib61" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[61]</span>
<span class="ltx_bibblock">
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett,
Junaid Ahmed, and Arnold Overwijk.

</span>
<span class="ltx_bibblock">Approximate nearest neighbor negative contrastive learning for dense
text retrieval.

</span>
<span class="ltx_bibblock">In <span id="bib.bib61.1.1" class="ltx_text ltx_font_italic">9th International Conference on Learning Representations,
ICLR 2021, Virtual Event, Austria, May 3-7, 2021</span>. OpenReview.net, 2021.

</span>
</li>
<li id="bib.bib62" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[62]</span>
<span class="ltx_bibblock">
Jun Xu, Tao Mei, Ting Yao, and Yong Rui.

</span>
<span class="ltx_bibblock">MSR-VTT: A large video description dataset for bridging video and
language.

</span>
<span class="ltx_bibblock">In <span id="bib.bib62.1.1" class="ltx_text ltx_font_italic">2016 IEEE Conference on Computer Vision and Pattern
Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016</span>, pages
5288–5296. IEEE Computer Society, 2016.

</span>
</li>
<li id="bib.bib63" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[63]</span>
<span class="ltx_bibblock">
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan
Salakhutdinov, and Christopher D. Manning.

</span>
<span class="ltx_bibblock">HotpotQA: A dataset for diverse, explainable multi-hop question
answering.

</span>
<span class="ltx_bibblock">In <span id="bib.bib63.1.1" class="ltx_text ltx_font_italic">Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing</span>, pages 2369–2380, Brussels, Belgium, 2018.
Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib64" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[64]</span>
<span class="ltx_bibblock">
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier.

</span>
<span class="ltx_bibblock">From image descriptions to visual denotations: New similarity metrics
for semantic inference over event descriptions.

</span>
<span class="ltx_bibblock"><span id="bib.bib64.1.1" class="ltx_text ltx_font_italic">Transactions of the Association for Computational Linguistics</span>,
2:67–78, 2014.

</span>
</li>
<li id="bib.bib65" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[65]</span>
<span class="ltx_bibblock">
Dian Yu, Kai Sun, Claire Cardie, and Dong Yu.

</span>
<span class="ltx_bibblock">Dialogue-based relation extraction.

</span>
<span class="ltx_bibblock">In <span id="bib.bib65.1.1" class="ltx_text ltx_font_italic">Proceedings of the 58th Annual Meeting of the Association for
Computational Linguistics</span>, pages 4927–4940, Online, 2020. Association for
Computational Linguistics.

</span>
</li>
<li id="bib.bib66" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[66]</span>
<span class="ltx_bibblock">
Dian Yu, Kai Sun, Claire Cardie, and Dong Yu.

</span>
<span class="ltx_bibblock">Dialogue-based relation extraction.

</span>
<span class="ltx_bibblock">In <span id="bib.bib66.1.1" class="ltx_text ltx_font_italic">Proceedings of the 58th Annual Meeting of the Association for
Computational Linguistics</span>, pages 4927–4940, Online, 2020. Association for
Computational Linguistics.

</span>
</li>
<li id="bib.bib67" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[67]</span>
<span class="ltx_bibblock">
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James
Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev.

</span>
<span class="ltx_bibblock">Spider: A large-scale human-labeled dataset for complex and
cross-domain semantic parsing and text-to-SQL task.

</span>
<span class="ltx_bibblock">In <span id="bib.bib67.1.1" class="ltx_text ltx_font_italic">Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing</span>, pages 3911–3921, Brussels, Belgium, 2018.
Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib68" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[68]</span>
<span class="ltx_bibblock">
Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li,
Heyang Er, Irene Li, Bo Pang, Tao Chen, Emily Ji, Shreya Dixit, David
Proctor, Sungrok Shim, Jonathan Kraft, Vincent Zhang, Caiming Xiong, Richard
Socher, and Dragomir Radev.

</span>
<span class="ltx_bibblock">SParC: Cross-domain semantic parsing in context.

</span>
<span class="ltx_bibblock">In <span id="bib.bib68.1.1" class="ltx_text ltx_font_italic">Proceedings of the 57th Annual Meeting of the Association for
Computational Linguistics</span>, pages 4511–4523, Florence, Italy, 2019.
Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib69" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[69]</span>
<span class="ltx_bibblock">
Victor Zhong, Caiming Xiong, and Richard Socher.

</span>
<span class="ltx_bibblock">Seq2SQL: Generating structured queries from natural language using
reinforcement learning.

</span>
<span class="ltx_bibblock"><span id="bib.bib69.1.1" class="ltx_text ltx_font_italic">ArXiv preprint</span>, abs/1709.00103, 2017.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<div class="ltx_pagination ltx_role_newpage"></div>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span><span id="A1.1.1" class="ltx_text ltx_font_smallcaps">LOFT</span> Dataset Creation</h2>

<section id="A1.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.1 </span>Dataset Selection</h3>

<section id="A1.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Text Retrieval &amp; RAG</h4>

<div id="A1.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="A1.SS1.SSS0.Px1.p1.6" class="ltx_p">We test single-document retrieval on a representative subset of the BEIR benchmark <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib53" title="" class="ltx_ref">53</a>]</cite>, prioritizing datasets with high-quality ground truth labels <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib56" title="" class="ltx_ref">56</a>, <a href="#bib.bib54" title="" class="ltx_ref">54</a>, <a href="#bib.bib40" title="" class="ltx_ref">40</a>, <a href="#bib.bib7" title="" class="ltx_ref">7</a>, <a href="#bib.bib29" title="" class="ltx_ref">29</a>, <a href="#bib.bib57" title="" class="ltx_ref">57</a>]</cite>.
We also include TopiOCQA <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib2" title="" class="ltx_ref">2</a>]</cite>, which is a multi-turn conversational retrieval dataset.
We measure performance on single-document retrieval using Recall@1.
Additionally, we test multi-document retrieval on HotPotQA <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib63" title="" class="ltx_ref">63</a>]</cite>, MuSiQue <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib55" title="" class="ltx_ref">55</a>]</cite>, QAMPARI <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib41" title="" class="ltx_ref">41</a>]</cite>, where a set of documents must be retrieved to answer the query.
The evaluation metric for multi-document retrieval is MRecall@<math id="A1.SS1.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A1.SS1.SSS0.Px1.p1.1.m1.1a"><mi id="A1.SS1.SSS0.Px1.p1.1.m1.1.1" xref="A1.SS1.SSS0.Px1.p1.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A1.SS1.SSS0.Px1.p1.1.m1.1b"><ci id="A1.SS1.SSS0.Px1.p1.1.m1.1.1.cmml" xref="A1.SS1.SSS0.Px1.p1.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.SSS0.Px1.p1.1.m1.1c">k</annotation></semantics></math> <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib42" title="" class="ltx_ref">42</a>]</cite>, which gives a score of 1.0 if all <math id="A1.SS1.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A1.SS1.SSS0.Px1.p1.2.m2.1a"><mi id="A1.SS1.SSS0.Px1.p1.2.m2.1.1" xref="A1.SS1.SSS0.Px1.p1.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A1.SS1.SSS0.Px1.p1.2.m2.1b"><ci id="A1.SS1.SSS0.Px1.p1.2.m2.1.1.cmml" xref="A1.SS1.SSS0.Px1.p1.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.SSS0.Px1.p1.2.m2.1c">k</annotation></semantics></math> gold set items are retrieved in top-<math id="A1.SS1.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A1.SS1.SSS0.Px1.p1.3.m3.1a"><mi id="A1.SS1.SSS0.Px1.p1.3.m3.1.1" xref="A1.SS1.SSS0.Px1.p1.3.m3.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A1.SS1.SSS0.Px1.p1.3.m3.1b"><ci id="A1.SS1.SSS0.Px1.p1.3.m3.1.1.cmml" xref="A1.SS1.SSS0.Px1.p1.3.m3.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.SSS0.Px1.p1.3.m3.1c">k</annotation></semantics></math> and 0.0 otherwise.
When creating the <span id="A1.SS1.SSS0.Px1.p1.6.1" class="ltx_text ltx_font_smallcaps">LOFT</span> version of the multi-document retrieval datasets, we limit the number of relevant documents per query to <math id="A1.SS1.SSS0.Px1.p1.4.m4.1" class="ltx_Math" alttext="k=2" display="inline"><semantics id="A1.SS1.SSS0.Px1.p1.4.m4.1a"><mrow id="A1.SS1.SSS0.Px1.p1.4.m4.1.1" xref="A1.SS1.SSS0.Px1.p1.4.m4.1.1.cmml"><mi id="A1.SS1.SSS0.Px1.p1.4.m4.1.1.2" xref="A1.SS1.SSS0.Px1.p1.4.m4.1.1.2.cmml">k</mi><mo id="A1.SS1.SSS0.Px1.p1.4.m4.1.1.1" xref="A1.SS1.SSS0.Px1.p1.4.m4.1.1.1.cmml">=</mo><mn id="A1.SS1.SSS0.Px1.p1.4.m4.1.1.3" xref="A1.SS1.SSS0.Px1.p1.4.m4.1.1.3.cmml">2</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS1.SSS0.Px1.p1.4.m4.1b"><apply id="A1.SS1.SSS0.Px1.p1.4.m4.1.1.cmml" xref="A1.SS1.SSS0.Px1.p1.4.m4.1.1"><eq id="A1.SS1.SSS0.Px1.p1.4.m4.1.1.1.cmml" xref="A1.SS1.SSS0.Px1.p1.4.m4.1.1.1"></eq><ci id="A1.SS1.SSS0.Px1.p1.4.m4.1.1.2.cmml" xref="A1.SS1.SSS0.Px1.p1.4.m4.1.1.2">𝑘</ci><cn type="integer" id="A1.SS1.SSS0.Px1.p1.4.m4.1.1.3.cmml" xref="A1.SS1.SSS0.Px1.p1.4.m4.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.SSS0.Px1.p1.4.m4.1c">k=2</annotation></semantics></math>, 5, 5, and 3 for HotPotQA, MuSiQue, QAMPARI, and QUEST, respectively, and the corresponding <math id="A1.SS1.SSS0.Px1.p1.5.m5.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A1.SS1.SSS0.Px1.p1.5.m5.1a"><mi id="A1.SS1.SSS0.Px1.p1.5.m5.1.1" xref="A1.SS1.SSS0.Px1.p1.5.m5.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A1.SS1.SSS0.Px1.p1.5.m5.1b"><ci id="A1.SS1.SSS0.Px1.p1.5.m5.1.1.cmml" xref="A1.SS1.SSS0.Px1.p1.5.m5.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.SSS0.Px1.p1.5.m5.1c">k</annotation></semantics></math>’s are used for MRecall@<math id="A1.SS1.SSS0.Px1.p1.6.m6.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A1.SS1.SSS0.Px1.p1.6.m6.1a"><mi id="A1.SS1.SSS0.Px1.p1.6.m6.1.1" xref="A1.SS1.SSS0.Px1.p1.6.m6.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A1.SS1.SSS0.Px1.p1.6.m6.1b"><ci id="A1.SS1.SSS0.Px1.p1.6.m6.1.1.cmml" xref="A1.SS1.SSS0.Px1.p1.6.m6.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.SSS0.Px1.p1.6.m6.1c">k</annotation></semantics></math> (e.g. HotPotQA uses MRecall@2).</p>
</div>
<div id="A1.SS1.SSS0.Px1.p2" class="ltx_para">
<p id="A1.SS1.SSS0.Px1.p2.1" class="ltx_p">Our RAG task contains subsets of retrieval datasets, which have phrase-level answer annotations: Natural Questions, TopiOCQA, HotPotQA, MuSiQue, QAMPARI, and QUEST.
We use subspan exact match (EM) <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib1" title="" class="ltx_ref">1</a>]</cite> for evaluating performance of all the datasets.
In case of multi-answer datasets (i.e. QAMPARI, QUEST), we first match predicted answers to gold standard answers based on whether they overlap <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib17" title="" class="ltx_ref">17</a>]</cite> via linear sum assignment algorithm. We then give full credit if every gold answer has a perfect match with aligned predicted answers.</p>
</div>
</section>
<section id="A1.SS1.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Visual Retrieval</h4>

<div id="A1.SS1.SSS0.Px2.p1" class="ltx_para">
<p id="A1.SS1.SSS0.Px2.p1.1" class="ltx_p">We employ four diverse visual benchmarks: Flickr30k <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib64" title="" class="ltx_ref">64</a>]</cite> and MSCOCO <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib36" title="" class="ltx_ref">36</a>]</cite> for text-to-image retrieval; MSR-VTT <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib62" title="" class="ltx_ref">62</a>]</cite> for text-to-video retrieval (sampling 3 frames per video); and OVEN <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib24" title="" class="ltx_ref">24</a>]</cite> using the entity split for image-text retrieval where both queries and retrieval targets consist of image-text pairs.
All images are resized to 512x512 and performance is assessed using Recall@1 for all datasets.</p>
</div>
</section>
<section id="A1.SS1.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Audio Retrieval</h4>

<div id="A1.SS1.SSS0.Px3.p1" class="ltx_para">
<p id="A1.SS1.SSS0.Px3.p1.1" class="ltx_p">We utilize a subset of the multilingual FLEURS dataset <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite>, focusing on the five most spoken languages<span id="footnote8" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">8</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">8</sup><span class="ltx_tag ltx_tag_note">8</span><a target="_blank" href="https://en.wikipedia.org/wiki/List_of_languages_by_total_number_of_speakers" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://en.wikipedia.org/wiki/List_of_languages_by_total_number_of_speakers</a></span></span></span>: English (en), Hindi (hi), Chinese (zh), Spanish (es), and French (fr).
Recall@1 is employed as the evaluation metric, given the single gold target.</p>
</div>
</section>
<section id="A1.SS1.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">SQL</h4>

<div id="A1.SS1.SSS0.Px4.p1" class="ltx_para">
<p id="A1.SS1.SSS0.Px4.p1.1" class="ltx_p">We evaluate SQL-like reasoning on Spider, a single-turn text-to-SQL dataset <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib67" title="" class="ltx_ref">67</a>]</cite>, and SparC, its multi-turn variant <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib68" title="" class="ltx_ref">68</a>]</cite>.
The input contains the database tables serialized as CSV and the natural language question.
The model is allowed to perform reasoning in natural language before giving the final answer,
which must be formatted in a Markdown code block.
The extracted answers are evaluated against the execution results of the gold SQL queries.
For SparC, the multi-turn questions are provided one-by-one in a conversational format, and credit is awarded only when the answers of all steps are correct.</p>
</div>
</section>
<section id="A1.SS1.SSS0.Px5" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Many-shot ICL</h4>

<div id="A1.SS1.SSS0.Px5.p1" class="ltx_para">
<p id="A1.SS1.SSS0.Px5.p1.1" class="ltx_p">We investigate LCLMs’ many-shot ICL capabilities by repurposing datasets from Big Bench Hard (BBH) <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib10" title="" class="ltx_ref">10</a>, <a href="#bib.bib49" title="" class="ltx_ref">49</a>]</cite> and LongICLBench (LIB) <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib66" title="" class="ltx_ref">66</a>, <a href="#bib.bib35" title="" class="ltx_ref">35</a>]</cite> to fit a many-shot ICL setting, focusing on multi-class classification tasks.
The first set of datasets is drawn from Big-Bench Hard and includes: <span id="A1.SS1.SSS0.Px5.p1.1.1" class="ltx_text ltx_font_typewriter">date_understanding</span> (BBH-date), <span id="A1.SS1.SSS0.Px5.p1.1.2" class="ltx_text ltx_font_typewriter">salient_error_translation_detection</span> (BBH-salient), <span id="A1.SS1.SSS0.Px5.p1.1.3" class="ltx_text ltx_font_typewriter">tracking_shuffled_objects_seven_objects</span> (BBH-tracking7), and <span id="A1.SS1.SSS0.Px5.p1.1.4" class="ltx_text ltx_font_typewriter">web_of_lies</span> (BBH-web), each with up to 150 examples for prompting and up to 7 classes.
Unlike other <span id="A1.SS1.SSS0.Px5.p1.1.5" class="ltx_text ltx_font_smallcaps">LOFT</span> tasks, the full corpus fits within 32k tokens which leads us to also create variants from 2k to 32k context lengths.
We use accuracy as our metric for Big Bench Hard.
We also evaluate with DialogRE <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib66" title="" class="ltx_ref">66</a>]</cite>, a dialogue-based relation classification dataset with 36 relation labels.
We follow the LongICLBench format but use accuracy as our metric.</p>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</section>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Datasets Processing Details</h2>

<section id="A2.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Content Filtering</h4>

<div id="A2.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="A2.SS0.SSS0.Px1.p1.1" class="ltx_p">The language model APIs often block inputs with potentially harmful contents. When creating <span id="A2.SS0.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_smallcaps">LOFT</span>, we tried to remove such contents from textual and visual inputs. Our filtering was done using a classifier as well as a keyword-based filtering. Despite our best effort, some API calls still refused to provide answers, which we treated as incorrect in our evaluation.</p>
</div>
</section>
<section id="A2.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Tokenization</h4>

<div id="A2.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="A2.SS0.SSS0.Px2.p1.1" class="ltx_p">To measure the size of a corpus, we count the number of tokens returned by the SentencePiece tokenizer <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib28" title="" class="ltx_ref">28</a>]</cite>.</p>
</div>
</section>
<section id="A2.SS0.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Links to Dataset Sources</h4>

<div id="A2.SS0.SSS0.Px3.p1" class="ltx_para">
<p id="A2.SS0.SSS0.Px3.p1.1" class="ltx_p"><span id="A2.SS0.SSS0.Px3.p1.1.1" class="ltx_text ltx_font_smallcaps">LOFT</span> repurposes existing datasets for evaluating LCLMs.
Here are the links to the original datasets used in <span id="A2.SS0.SSS0.Px3.p1.1.2" class="ltx_text ltx_font_smallcaps">LOFT</span>.</p>
<ul id="A2.I1" class="ltx_itemize">
<li id="A2.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A2.I1.i1.p1" class="ltx_para">
<p id="A2.I1.i1.p1.1" class="ltx_p">Text Retrieval - BEIR <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib53" title="" class="ltx_ref">53</a>]</cite> (ArguAna <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib56" title="" class="ltx_ref">56</a>]</cite>, FEVER <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib54" title="" class="ltx_ref">54</a>]</cite>, FIQA <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib40" title="" class="ltx_ref">40</a>]</cite>, MS MARCO <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib7" title="" class="ltx_ref">7</a>]</cite>, NQ <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib29" title="" class="ltx_ref">29</a>]</cite>, Quora, SciFact <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib57" title="" class="ltx_ref">57</a>]</cite>, Touché-2020 <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib11" title="" class="ltx_ref">11</a>]</cite>, HotPotQA <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib63" title="" class="ltx_ref">63</a>]</cite>): <a target="_blank" href="https://github.com/beir-cellar/beir" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/beir-cellar/beir</a></p>
</div>
</li>
<li id="A2.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A2.I1.i2.p1" class="ltx_para">
<p id="A2.I1.i2.p1.1" class="ltx_p">Text Retrieval - TopiOCQA <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib2" title="" class="ltx_ref">2</a>]</cite>: <a target="_blank" href="https://github.com/McGill-NLP/topiocqa" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/McGill-NLP/topiocqa</a></p>
</div>
</li>
<li id="A2.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A2.I1.i3.p1" class="ltx_para">
<p id="A2.I1.i3.p1.1" class="ltx_p">Text Retrieval - MuSiQue <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib55" title="" class="ltx_ref">55</a>]</cite>: <a target="_blank" href="https://allenai.org/data/musique" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://allenai.org/data/musique</a></p>
</div>
</li>
<li id="A2.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A2.I1.i4.p1" class="ltx_para">
<p id="A2.I1.i4.p1.1" class="ltx_p">Text Retrieval - QAMPARI <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib3" title="" class="ltx_ref">3</a>]</cite>: <a target="_blank" href="https://github.com/samsam3232/qampari" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/samsam3232/qampari</a></p>
</div>
</li>
<li id="A2.I1.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A2.I1.i5.p1" class="ltx_para">
<p id="A2.I1.i5.p1.1" class="ltx_p">Text Retrieval - QUEST <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib41" title="" class="ltx_ref">41</a>]</cite>: <a target="_blank" href="https://github.com/google-research/language/tree/master/language/quest" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/google-research/language/tree/master/language/quest</a></p>
</div>
</li>
<li id="A2.I1.i6" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A2.I1.i6.p1" class="ltx_para">
<p id="A2.I1.i6.p1.1" class="ltx_p">Visual Retrieval - Flickr30k <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib64" title="" class="ltx_ref">64</a>]</cite>: <a target="_blank" href="https://www.kaggle.com/datasets/hsankesara/flickr-image-dataset" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.kaggle.com/datasets/hsankesara/flickr-image-dataset</a></p>
</div>
</li>
<li id="A2.I1.i7" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A2.I1.i7.p1" class="ltx_para">
<p id="A2.I1.i7.p1.1" class="ltx_p">Visual Retrieval - MS COCO <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib36" title="" class="ltx_ref">36</a>]</cite>: <a target="_blank" href="https://cocodataset.org" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://cocodataset.org</a></p>
</div>
</li>
<li id="A2.I1.i8" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A2.I1.i8.p1" class="ltx_para">
<p id="A2.I1.i8.p1.1" class="ltx_p">Visual Retrieval - OVEN <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib24" title="" class="ltx_ref">24</a>]</cite>: <a target="_blank" href="https://github.com/open-vision-language/oven" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/open-vision-language/oven</a></p>
</div>
</li>
<li id="A2.I1.i9" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A2.I1.i9.p1" class="ltx_para">
<p id="A2.I1.i9.p1.1" class="ltx_p">Visual Retrieval - MSR-VTT <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib62" title="" class="ltx_ref">62</a>]</cite>: <a target="_blank" href="https://cove.thecvf.com/datasets/839" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://cove.thecvf.com/datasets/839</a></p>
</div>
</li>
<li id="A2.I1.i10" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A2.I1.i10.p1" class="ltx_para">
<p id="A2.I1.i10.p1.1" class="ltx_p">Audio Retrieval - FLEURS <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite>: <a target="_blank" href="https://huggingface.co/datasets/google/fleurs" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/datasets/google/fleurs</a></p>
</div>
</li>
<li id="A2.I1.i11" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A2.I1.i11.p1" class="ltx_para">
<p id="A2.I1.i11.p1.1" class="ltx_p">RAG - Same as Text Retrieval</p>
</div>
</li>
<li id="A2.I1.i12" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A2.I1.i12.p1" class="ltx_para">
<p id="A2.I1.i12.p1.1" class="ltx_p">SQL - Spider <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib67" title="" class="ltx_ref">67</a>]</cite>: <a target="_blank" href="https://yale-lily.github.io/spider" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://yale-lily.github.io/spider</a></p>
</div>
</li>
<li id="A2.I1.i13" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A2.I1.i13.p1" class="ltx_para">
<p id="A2.I1.i13.p1.1" class="ltx_p">SQL - SparC <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib68" title="" class="ltx_ref">68</a>]</cite>: <a target="_blank" href="https://yale-lily.github.io/sparc" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://yale-lily.github.io/sparc</a></p>
</div>
</li>
<li id="A2.I1.i14" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A2.I1.i14.p1" class="ltx_para">
<p id="A2.I1.i14.p1.1" class="ltx_p">Many-Shot ICL - Big-Bench Hard <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib10" title="" class="ltx_ref">10</a>, <a href="#bib.bib49" title="" class="ltx_ref">49</a>]</cite>: <a target="_blank" href="https://github.com/suzgunmirac/BIG-Bench-Hard" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/suzgunmirac/BIG-Bench-Hard</a></p>
</div>
</li>
<li id="A2.I1.i15" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A2.I1.i15.p1" class="ltx_para">
<p id="A2.I1.i15.p1.1" class="ltx_p">Many-Shot ICL - LongICLBench <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib66" title="" class="ltx_ref">66</a>, <a href="#bib.bib35" title="" class="ltx_ref">35</a>]</cite>: <a target="_blank" href="https://github.com/TIGER-AI-Lab/LongICLBench" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/TIGER-AI-Lab/LongICLBench</a></p>
</div>
</li>
</ul>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Detailed Statistics</h2>

<div id="A3.p1" class="ltx_para">
<p id="A3.p1.1" class="ltx_p">In <a href="#A3.T5" title="In Appendix C Detailed Statistics ‣ Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">5</span></a>, we show detailed statistics of the <span id="A3.p1.1.1" class="ltx_text ltx_font_smallcaps">LOFT</span> benchmark.</p>
</div>
<figure id="A3.T5" class="ltx_table">
<div id="A3.T5.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:411.9pt;height:577.6pt;vertical-align:-0.9pt;"><span class="ltx_transformed_inner" style="transform:translate(-33.8pt,47.3pt) scale(0.858997365010355,0.858997365010355) ;">
<table id="A3.T5.1.1" class="ltx_tabular ltx_align_middle">
<tr id="A3.T5.1.1.1" class="ltx_tr">
<td id="A3.T5.1.1.1.1" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="A3.T5.1.1.1.1.1" class="ltx_text ltx_font_bold">Task</span></td>
<td id="A3.T5.1.1.1.2" class="ltx_td ltx_align_left ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="A3.T5.1.1.1.2.1" class="ltx_text ltx_font_bold">Dataset</span></td>
<td id="A3.T5.1.1.1.3" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;">
<span id="A3.T5.1.1.1.3.1" class="ltx_text"></span> <span id="A3.T5.1.1.1.3.2" class="ltx_text">
<span id="A3.T5.1.1.1.3.2.1" class="ltx_tabular ltx_align_middle">
<span id="A3.T5.1.1.1.3.2.1.1" class="ltx_tr">
<span id="A3.T5.1.1.1.3.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="A3.T5.1.1.1.3.2.1.1.1.1" class="ltx_text ltx_font_bold"># Queries</span></span></span>
<span id="A3.T5.1.1.1.3.2.1.2" class="ltx_tr">
<span id="A3.T5.1.1.1.3.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">(Few-shot / Development / Test)</span></span>
</span></span><span id="A3.T5.1.1.1.3.3" class="ltx_text"></span></td>
<td id="A3.T5.1.1.1.4" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;">
<span id="A3.T5.1.1.1.4.1" class="ltx_text"></span> <span id="A3.T5.1.1.1.4.2" class="ltx_text">
<span id="A3.T5.1.1.1.4.2.1" class="ltx_tabular ltx_align_middle">
<span id="A3.T5.1.1.1.4.2.1.1" class="ltx_tr">
<span id="A3.T5.1.1.1.4.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="A3.T5.1.1.1.4.2.1.1.1.1" class="ltx_text ltx_font_bold">Supported</span></span></span>
<span id="A3.T5.1.1.1.4.2.1.2" class="ltx_tr">
<span id="A3.T5.1.1.1.4.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="A3.T5.1.1.1.4.2.1.2.1.1" class="ltx_text ltx_font_bold">Context Length</span></span></span>
</span></span><span id="A3.T5.1.1.1.4.3" class="ltx_text"></span></td>
<td id="A3.T5.1.1.1.5" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="A3.T5.1.1.1.5.1" class="ltx_text ltx_font_bold"># Candidates</span></td>
</tr>
<tr id="A3.T5.1.1.2" class="ltx_tr">
<td id="A3.T5.1.1.2.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="13"><span id="A3.T5.1.1.2.1.1" class="ltx_text ltx_font_bold"><span id="A3.T5.1.1.2.1.1.1" class="ltx_text"></span> <span id="A3.T5.1.1.2.1.1.2" class="ltx_text">
<span id="A3.T5.1.1.2.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="A3.T5.1.1.2.1.1.2.1.1" class="ltx_tr">
<span id="A3.T5.1.1.2.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Text</span></span>
<span id="A3.T5.1.1.2.1.1.2.1.2" class="ltx_tr">
<span id="A3.T5.1.1.2.1.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Retrieval</span></span>
</span></span> <span id="A3.T5.1.1.2.1.1.3" class="ltx_text"></span></span></td>
<td id="A3.T5.1.1.2.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">ArguAna</td>
<td id="A3.T5.1.1.2.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">5 / 10 / 100</td>
<td id="A3.T5.1.1.2.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k / 1M</td>
<td id="A3.T5.1.1.2.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">123 / 531 / 3,891</td>
</tr>
<tr id="A3.T5.1.1.3" class="ltx_tr">
<td id="A3.T5.1.1.3.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">FEVER</td>
<td id="A3.T5.1.1.3.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">5 / 10 / 100</td>
<td id="A3.T5.1.1.3.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k / 1M</td>
<td id="A3.T5.1.1.3.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">154 / 588 / 6,031</td>
</tr>
<tr id="A3.T5.1.1.4" class="ltx_tr">
<td id="A3.T5.1.1.4.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">FIQA</td>
<td id="A3.T5.1.1.4.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">5 / 10 / 100</td>
<td id="A3.T5.1.1.4.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k / 1M</td>
<td id="A3.T5.1.1.4.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">148 / 531 / 4,471</td>
</tr>
<tr id="A3.T5.1.1.5" class="ltx_tr">
<td id="A3.T5.1.1.5.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">MS MARCO</td>
<td id="A3.T5.1.1.5.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">5 / 10 / 100</td>
<td id="A3.T5.1.1.5.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k / 1M</td>
<td id="A3.T5.1.1.5.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">302 / 1,174 / 9,208</td>
</tr>
<tr id="A3.T5.1.1.6" class="ltx_tr">
<td id="A3.T5.1.1.6.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">NQ</td>
<td id="A3.T5.1.1.6.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">5 / 10 / 100</td>
<td id="A3.T5.1.1.6.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k / 1M</td>
<td id="A3.T5.1.1.6.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">214 / 883 / 6,999</td>
</tr>
<tr id="A3.T5.1.1.7" class="ltx_tr">
<td id="A3.T5.1.1.7.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">Quora</td>
<td id="A3.T5.1.1.7.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">5 / 10 / 100</td>
<td id="A3.T5.1.1.7.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k / 1M</td>
<td id="A3.T5.1.1.7.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">820 / 3,306 / 25,755</td>
</tr>
<tr id="A3.T5.1.1.8" class="ltx_tr">
<td id="A3.T5.1.1.8.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">SciFact</td>
<td id="A3.T5.1.1.8.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">5 / 10 / 100</td>
<td id="A3.T5.1.1.8.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k / 1M</td>
<td id="A3.T5.1.1.8.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">86 / 357 / 2,753</td>
</tr>
<tr id="A3.T5.1.1.9" class="ltx_tr">
<td id="A3.T5.1.1.9.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">Touché-2020</td>
<td id="A3.T5.1.1.9.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">5 / 10 / 34</td>
<td id="A3.T5.1.1.9.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k / 1M</td>
<td id="A3.T5.1.1.9.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">77 / 329 / 2,843</td>
</tr>
<tr id="A3.T5.1.1.10" class="ltx_tr">
<td id="A3.T5.1.1.10.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">TopiOCQA</td>
<td id="A3.T5.1.1.10.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">5 / 10 / 100</td>
<td id="A3.T5.1.1.10.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k / 1M</td>
<td id="A3.T5.1.1.10.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">170 / 680 / 5,379</td>
</tr>
<tr id="A3.T5.1.1.11" class="ltx_tr">
<td id="A3.T5.1.1.11.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">HotPotQA</td>
<td id="A3.T5.1.1.11.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">5 / 10 / 100</td>
<td id="A3.T5.1.1.11.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k / 1M</td>
<td id="A3.T5.1.1.11.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">319 / 1,222 / 10,005</td>
</tr>
<tr id="A3.T5.1.1.12" class="ltx_tr">
<td id="A3.T5.1.1.12.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">MuSiQue</td>
<td id="A3.T5.1.1.12.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">5 / 10 / 100</td>
<td id="A3.T5.1.1.12.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k / 1M</td>
<td id="A3.T5.1.1.12.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">210 / 824 / 6,650</td>
</tr>
<tr id="A3.T5.1.1.13" class="ltx_tr">
<td id="A3.T5.1.1.13.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">QAMPARI</td>
<td id="A3.T5.1.1.13.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">5 / 10 / 100</td>
<td id="A3.T5.1.1.13.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k / 1M</td>
<td id="A3.T5.1.1.13.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">186 / 755 / 5,878</td>
</tr>
<tr id="A3.T5.1.1.14" class="ltx_tr">
<td id="A3.T5.1.1.14.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">QUEST</td>
<td id="A3.T5.1.1.14.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">5 / 10 / 100</td>
<td id="A3.T5.1.1.14.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k / 1M</td>
<td id="A3.T5.1.1.14.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">87 / 328 / 2,858</td>
</tr>
<tr id="A3.T5.1.1.15" class="ltx_tr">
<td id="A3.T5.1.1.15.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="4"><span id="A3.T5.1.1.15.1.1" class="ltx_text ltx_font_bold"><span id="A3.T5.1.1.15.1.1.1" class="ltx_text"></span> <span id="A3.T5.1.1.15.1.1.2" class="ltx_text">
<span id="A3.T5.1.1.15.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="A3.T5.1.1.15.1.1.2.1.1" class="ltx_tr">
<span id="A3.T5.1.1.15.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Visual</span></span>
<span id="A3.T5.1.1.15.1.1.2.1.2" class="ltx_tr">
<span id="A3.T5.1.1.15.1.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Retrieval</span></span>
</span></span> <span id="A3.T5.1.1.15.1.1.3" class="ltx_text"></span></span></td>
<td id="A3.T5.1.1.15.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">Flickr30k</td>
<td id="A3.T5.1.1.15.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">5 / 10 / 100</td>
<td id="A3.T5.1.1.15.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k</td>
<td id="A3.T5.1.1.15.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">115 / 440</td>
</tr>
<tr id="A3.T5.1.1.16" class="ltx_tr">
<td id="A3.T5.1.1.16.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">MS COCO</td>
<td id="A3.T5.1.1.16.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">5 / 10 / 100</td>
<td id="A3.T5.1.1.16.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k / 1M</td>
<td id="A3.T5.1.1.16.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">115 / 440 / 3,448</td>
</tr>
<tr id="A3.T5.1.1.17" class="ltx_tr">
<td id="A3.T5.1.1.17.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">OVEN</td>
<td id="A3.T5.1.1.17.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">5 / 10 / 100</td>
<td id="A3.T5.1.1.17.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k / 1M</td>
<td id="A3.T5.1.1.17.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">110 / 448 / 3475</td>
</tr>
<tr id="A3.T5.1.1.18" class="ltx_tr">
<td id="A3.T5.1.1.18.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">MSR-VTT</td>
<td id="A3.T5.1.1.18.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">5 / 10 / 100</td>
<td id="A3.T5.1.1.18.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k / 1M</td>
<td id="A3.T5.1.1.18.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">35 / 140 / 1,101</td>
</tr>
<tr id="A3.T5.1.1.19" class="ltx_tr">
<td id="A3.T5.1.1.19.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="5"><span id="A3.T5.1.1.19.1.1" class="ltx_text ltx_font_bold"><span id="A3.T5.1.1.19.1.1.1" class="ltx_text"></span> <span id="A3.T5.1.1.19.1.1.2" class="ltx_text">
<span id="A3.T5.1.1.19.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="A3.T5.1.1.19.1.1.2.1.1" class="ltx_tr">
<span id="A3.T5.1.1.19.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Audio</span></span>
<span id="A3.T5.1.1.19.1.1.2.1.2" class="ltx_tr">
<span id="A3.T5.1.1.19.1.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Retrieval</span></span>
</span></span> <span id="A3.T5.1.1.19.1.1.3" class="ltx_text"></span></span></td>
<td id="A3.T5.1.1.19.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">FLEURS-en</td>
<td id="A3.T5.1.1.19.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">5 / 10 / 100</td>
<td id="A3.T5.1.1.19.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k</td>
<td id="A3.T5.1.1.19.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">104 / 428</td>
</tr>
<tr id="A3.T5.1.1.20" class="ltx_tr">
<td id="A3.T5.1.1.20.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">FLEURS-es</td>
<td id="A3.T5.1.1.20.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">5 / 10 / 100</td>
<td id="A3.T5.1.1.20.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k</td>
<td id="A3.T5.1.1.20.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">77 / 343</td>
</tr>
<tr id="A3.T5.1.1.21" class="ltx_tr">
<td id="A3.T5.1.1.21.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">FLEURS-fr</td>
<td id="A3.T5.1.1.21.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">5 / 10 / 100</td>
<td id="A3.T5.1.1.21.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k</td>
<td id="A3.T5.1.1.21.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">94 / 412</td>
</tr>
<tr id="A3.T5.1.1.22" class="ltx_tr">
<td id="A3.T5.1.1.22.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">FLEURS-hi</td>
<td id="A3.T5.1.1.22.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">5 / 10 / 100</td>
<td id="A3.T5.1.1.22.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k</td>
<td id="A3.T5.1.1.22.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">83 / 369</td>
</tr>
<tr id="A3.T5.1.1.23" class="ltx_tr">
<td id="A3.T5.1.1.23.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">FLEURS-zh</td>
<td id="A3.T5.1.1.23.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">5 / 10 / 100</td>
<td id="A3.T5.1.1.23.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k</td>
<td id="A3.T5.1.1.23.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">85 / 370</td>
</tr>
<tr id="A3.T5.1.1.24" class="ltx_tr">
<td id="A3.T5.1.1.24.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="6"><span id="A3.T5.1.1.24.1.1" class="ltx_text ltx_font_bold"><span id="A3.T5.1.1.24.1.1.1" class="ltx_text"></span> <span id="A3.T5.1.1.24.1.1.2" class="ltx_text">
<span id="A3.T5.1.1.24.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="A3.T5.1.1.24.1.1.2.1.1" class="ltx_tr">
<span id="A3.T5.1.1.24.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">RAG</span></span>
</span></span> <span id="A3.T5.1.1.24.1.1.3" class="ltx_text"></span></span></td>
<td id="A3.T5.1.1.24.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">NQ</td>
<td id="A3.T5.1.1.24.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">5 / 10 / 100</td>
<td id="A3.T5.1.1.24.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k / 1M</td>
<td id="A3.T5.1.1.24.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">214 / 883 / 6,999</td>
</tr>
<tr id="A3.T5.1.1.25" class="ltx_tr">
<td id="A3.T5.1.1.25.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">TopiOCQA</td>
<td id="A3.T5.1.1.25.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">5 / 10 / 100</td>
<td id="A3.T5.1.1.25.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k / 1M</td>
<td id="A3.T5.1.1.25.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">170 / 680 / 5,379</td>
</tr>
<tr id="A3.T5.1.1.26" class="ltx_tr">
<td id="A3.T5.1.1.26.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">HotPotQA</td>
<td id="A3.T5.1.1.26.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">5 / 10 / 100</td>
<td id="A3.T5.1.1.26.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k / 1M</td>
<td id="A3.T5.1.1.26.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">319 / 1,222 / 10,005</td>
</tr>
<tr id="A3.T5.1.1.27" class="ltx_tr">
<td id="A3.T5.1.1.27.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">MuSiQue</td>
<td id="A3.T5.1.1.27.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">5 / 10 / 100</td>
<td id="A3.T5.1.1.27.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k / 1M</td>
<td id="A3.T5.1.1.27.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">210 / 824 / 6,650</td>
</tr>
<tr id="A3.T5.1.1.28" class="ltx_tr">
<td id="A3.T5.1.1.28.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">QAMPARI</td>
<td id="A3.T5.1.1.28.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">5 / 10 / 100</td>
<td id="A3.T5.1.1.28.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k / 1M</td>
<td id="A3.T5.1.1.28.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">186 / 755 / 5,878</td>
</tr>
<tr id="A3.T5.1.1.29" class="ltx_tr">
<td id="A3.T5.1.1.29.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">QUEST</td>
<td id="A3.T5.1.1.29.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">5 / 10 / 100</td>
<td id="A3.T5.1.1.29.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k / 1M</td>
<td id="A3.T5.1.1.29.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">87 / 328 / 2,858</td>
</tr>
<tr id="A3.T5.1.1.30" class="ltx_tr">
<td id="A3.T5.1.1.30.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="2"><span id="A3.T5.1.1.30.1.1" class="ltx_text ltx_font_bold">SQL</span></td>
<td id="A3.T5.1.1.30.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">Spider</td>
<td id="A3.T5.1.1.30.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">1 / 10 / 100</td>
<td id="A3.T5.1.1.30.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k / 1M</td>
<td id="A3.T5.1.1.30.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">1 / 1 / 1</td>
</tr>
<tr id="A3.T5.1.1.31" class="ltx_tr">
<td id="A3.T5.1.1.31.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">SParC</td>
<td id="A3.T5.1.1.31.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">1 / 10 / 100</td>
<td id="A3.T5.1.1.31.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k / 1M</td>
<td id="A3.T5.1.1.31.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">1 / 1 / 1</td>
</tr>
<tr id="A3.T5.1.1.32" class="ltx_tr">
<td id="A3.T5.1.1.32.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="5"><span id="A3.T5.1.1.32.1.1" class="ltx_text ltx_font_bold"><span id="A3.T5.1.1.32.1.1.1" class="ltx_text"></span> <span id="A3.T5.1.1.32.1.1.2" class="ltx_text">
<span id="A3.T5.1.1.32.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="A3.T5.1.1.32.1.1.2.1.1" class="ltx_tr">
<span id="A3.T5.1.1.32.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Many-Shot</span></span>
<span id="A3.T5.1.1.32.1.1.2.1.2" class="ltx_tr">
<span id="A3.T5.1.1.32.1.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">ICL</span></span>
</span></span> <span id="A3.T5.1.1.32.1.1.3" class="ltx_text"></span></span></td>
<td id="A3.T5.1.1.32.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">BBH-date</td>
<td id="A3.T5.1.1.32.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">- / 10 / 90</td>
<td id="A3.T5.1.1.32.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">32k</td>
<td id="A3.T5.1.1.32.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">150</td>
</tr>
<tr id="A3.T5.1.1.33" class="ltx_tr">
<td id="A3.T5.1.1.33.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">BBH-salient</td>
<td id="A3.T5.1.1.33.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">- / 10 / 90</td>
<td id="A3.T5.1.1.33.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32k</td>
<td id="A3.T5.1.1.33.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">104</td>
</tr>
<tr id="A3.T5.1.1.34" class="ltx_tr">
<td id="A3.T5.1.1.34.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">BBH-tracking7</td>
<td id="A3.T5.1.1.34.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">- / 10 / 90</td>
<td id="A3.T5.1.1.34.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32k</td>
<td id="A3.T5.1.1.34.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">123</td>
</tr>
<tr id="A3.T5.1.1.35" class="ltx_tr">
<td id="A3.T5.1.1.35.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">BBH-web</td>
<td id="A3.T5.1.1.35.2" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">- / 10 / 90</td>
<td id="A3.T5.1.1.35.3" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32k</td>
<td id="A3.T5.1.1.35.4" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">150</td>
</tr>
<tr id="A3.T5.1.1.36" class="ltx_tr">
<td id="A3.T5.1.1.36.1" class="ltx_td ltx_align_left ltx_border_bb" style="padding-left:3.0pt;padding-right:3.0pt;">LIB-dialogue</td>
<td id="A3.T5.1.1.36.2" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:3.0pt;padding-right:3.0pt;">- / 10 / 100</td>
<td id="A3.T5.1.1.36.3" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:3.0pt;padding-right:3.0pt;">32k / 128k / 1M</td>
<td id="A3.T5.1.1.36.4" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:3.0pt;padding-right:3.0pt;">61 / 274 / 1,059</td>
</tr>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 5: </span>

Tasks and datasets in the <span id="A3.T5.3.1" class="ltx_text ltx_font_smallcaps">LOFT</span> benchmark.
We show the number of queries per each split and and supported context lengths for each dataset.
</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A4" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>Prompt Design</h2>

<section id="A4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">D.1 </span>Dataset Instructions</h3>

<figure id="A4.T6" class="ltx_table">
<div id="A4.T6.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:745.1pt;vertical-align:-1.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-6.4pt,10.9pt) scale(0.971462553369776,0.971462553369776) ;">
<table id="A4.T6.1.1" class="ltx_tabular ltx_align_middle">
<tr id="A4.T6.1.1.1" class="ltx_tr">
<td id="A4.T6.1.1.1.1" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="A4.T6.1.1.1.1.1" class="ltx_text ltx_font_bold">Dataset</span></td>
<td id="A4.T6.1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;">
<span id="A4.T6.1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A4.T6.1.1.1.2.1.1" class="ltx_p" style="width:350.0pt;"><span id="A4.T6.1.1.1.2.1.1.1" class="ltx_text ltx_font_bold">Instruction</span></span>
</span>
</td>
</tr>
<tr id="A4.T6.1.1.2" class="ltx_tr">
<td id="A4.T6.1.1.2.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" colspan="2"><span id="A4.T6.1.1.2.1.1" class="ltx_text ltx_font_bold">Text Retrieval</span></td>
</tr>
<tr id="A4.T6.1.1.3" class="ltx_tr">
<td id="A4.T6.1.1.3.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="A4.T6.1.1.3.1.1" class="ltx_text"><span id="A4.T6.1.1.3.1.1.1" class="ltx_text"></span> <span id="A4.T6.1.1.3.1.1.2" class="ltx_text">
<span id="A4.T6.1.1.3.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="A4.T6.1.1.3.1.1.2.1.1" class="ltx_tr">
<span id="A4.T6.1.1.3.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">ArguAna</span></span>
</span></span> <span id="A4.T6.1.1.3.1.1.3" class="ltx_text"></span></span></td>
<td id="A4.T6.1.1.3.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">
<span id="A4.T6.1.1.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A4.T6.1.1.3.2.1.1" class="ltx_p" style="width:350.0pt;">You will be given a list of statements. You need to read carefully and understand all of them. Then you will be given a claim, and your goal is to find all statements from the list that can counterargue the claim.</span>
</span>
</td>
</tr>
<tr id="A4.T6.1.1.4" class="ltx_tr">
<td id="A4.T6.1.1.4.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="A4.T6.1.1.4.1.1" class="ltx_text"><span id="A4.T6.1.1.4.1.1.1" class="ltx_text"></span> <span id="A4.T6.1.1.4.1.1.2" class="ltx_text">
<span id="A4.T6.1.1.4.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="A4.T6.1.1.4.1.1.2.1.1" class="ltx_tr">
<span id="A4.T6.1.1.4.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">FEVER</span></span>
<span id="A4.T6.1.1.4.1.1.2.1.2" class="ltx_tr">
<span id="A4.T6.1.1.4.1.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Scifact</span></span>
</span></span> <span id="A4.T6.1.1.4.1.1.3" class="ltx_text"></span></span></td>
<td id="A4.T6.1.1.4.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">
<span id="A4.T6.1.1.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="A4.T6.1.1.4.2.1.1" class="ltx_p" style="width:350.0pt;">You will be given a list of passages. You need to read carefully and understand all of them. Then you will be given a claim, and your goal is to find all passages from the list that can help verify the claim as true of false.</span>
</span>
</td>
</tr>
<tr id="A4.T6.1.1.5" class="ltx_tr">
<td id="A4.T6.1.1.5.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="A4.T6.1.1.5.1.1" class="ltx_text"><span id="A4.T6.1.1.5.1.1.1" class="ltx_text"></span> <span id="A4.T6.1.1.5.1.1.2" class="ltx_text">
<span id="A4.T6.1.1.5.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="A4.T6.1.1.5.1.1.2.1.1" class="ltx_tr">
<span id="A4.T6.1.1.5.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">FIQA</span></span>
<span id="A4.T6.1.1.5.1.1.2.1.2" class="ltx_tr">
<span id="A4.T6.1.1.5.1.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">MS MARCO</span></span>
<span id="A4.T6.1.1.5.1.1.2.1.3" class="ltx_tr">
<span id="A4.T6.1.1.5.1.1.2.1.3.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">NQ, TopiOCQA</span></span>
</span></span> <span id="A4.T6.1.1.5.1.1.3" class="ltx_text"></span></span></td>
<td id="A4.T6.1.1.5.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">
<span id="A4.T6.1.1.5.2.1" class="ltx_inline-block ltx_align_top">
<span id="A4.T6.1.1.5.2.1.1" class="ltx_p" style="width:350.0pt;">You will be given a list of documents. You need to read carefully and understand all of them. Then you will be given a query, and your goal is to find all documents from the list that can help answer the query.</span>
</span>
</td>
</tr>
<tr id="A4.T6.1.1.6" class="ltx_tr">
<td id="A4.T6.1.1.6.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="A4.T6.1.1.6.1.1" class="ltx_text"><span id="A4.T6.1.1.6.1.1.1" class="ltx_text"></span> <span id="A4.T6.1.1.6.1.1.2" class="ltx_text">
<span id="A4.T6.1.1.6.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="A4.T6.1.1.6.1.1.2.1.1" class="ltx_tr">
<span id="A4.T6.1.1.6.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Quora</span></span>
</span></span> <span id="A4.T6.1.1.6.1.1.3" class="ltx_text"></span></span></td>
<td id="A4.T6.1.1.6.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">
<span id="A4.T6.1.1.6.2.1" class="ltx_inline-block ltx_align_top">
<span id="A4.T6.1.1.6.2.1.1" class="ltx_p" style="width:350.0pt;">You will be given a list of questions. You need to read carefully and understand all of them. Then you will be given a new question, and your goal is to find all questions from the list that are near duplicates of the new question.</span>
</span>
</td>
</tr>
<tr id="A4.T6.1.1.7" class="ltx_tr">
<td id="A4.T6.1.1.7.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="A4.T6.1.1.7.1.1" class="ltx_text"><span id="A4.T6.1.1.7.1.1.1" class="ltx_text"></span> <span id="A4.T6.1.1.7.1.1.2" class="ltx_text">
<span id="A4.T6.1.1.7.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="A4.T6.1.1.7.1.1.2.1.1" class="ltx_tr">
<span id="A4.T6.1.1.7.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Touché-2020</span></span>
</span></span> <span id="A4.T6.1.1.7.1.1.3" class="ltx_text"></span></span></td>
<td id="A4.T6.1.1.7.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">
<span id="A4.T6.1.1.7.2.1" class="ltx_inline-block ltx_align_top">
<span id="A4.T6.1.1.7.2.1.1" class="ltx_p" style="width:350.0pt;">You will be given a list of arguments. You need to read carefully and understand all of them. Then you will be given a controversial debating topic, and your goal is to find arguments from the list that’s relevant to the topic.</span>
</span>
</td>
</tr>
<tr id="A4.T6.1.1.8" class="ltx_tr">
<td id="A4.T6.1.1.8.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="A4.T6.1.1.8.1.1" class="ltx_text"><span id="A4.T6.1.1.8.1.1.1" class="ltx_text"></span> <span id="A4.T6.1.1.8.1.1.2" class="ltx_text">
<span id="A4.T6.1.1.8.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="A4.T6.1.1.8.1.1.2.1.1" class="ltx_tr">
<span id="A4.T6.1.1.8.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">HotPotQA</span></span>
<span id="A4.T6.1.1.8.1.1.2.1.2" class="ltx_tr">
<span id="A4.T6.1.1.8.1.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">MuSiQue</span></span>
<span id="A4.T6.1.1.8.1.1.2.1.3" class="ltx_tr">
<span id="A4.T6.1.1.8.1.1.2.1.3.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">QAMPARI</span></span>
<span id="A4.T6.1.1.8.1.1.2.1.4" class="ltx_tr">
<span id="A4.T6.1.1.8.1.1.2.1.4.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">QUEST</span></span>
</span></span> <span id="A4.T6.1.1.8.1.1.3" class="ltx_text"></span></span></td>
<td id="A4.T6.1.1.8.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">
<span id="A4.T6.1.1.8.2.1" class="ltx_inline-block ltx_align_top">
<span id="A4.T6.1.1.8.2.1.1" class="ltx_p" style="width:350.0pt;">You will be given a list of documents. You need to read carefully and understand all of them. Then you will be given a query that may require you to use 1 or more documents to find the answer. Your goal is to find all documents from the list that can help answer the query.</span>
</span>
</td>
</tr>
<tr id="A4.T6.1.1.9" class="ltx_tr">
<td id="A4.T6.1.1.9.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" colspan="2"><span id="A4.T6.1.1.9.1.1" class="ltx_text ltx_font_bold">Visual Retrieval</span></td>
</tr>
<tr id="A4.T6.1.1.10" class="ltx_tr">
<td id="A4.T6.1.1.10.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="A4.T6.1.1.10.1.1" class="ltx_text"><span id="A4.T6.1.1.10.1.1.1" class="ltx_text"></span> <span id="A4.T6.1.1.10.1.1.2" class="ltx_text">
<span id="A4.T6.1.1.10.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="A4.T6.1.1.10.1.1.2.1.1" class="ltx_tr">
<span id="A4.T6.1.1.10.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Flickr30k</span></span>
<span id="A4.T6.1.1.10.1.1.2.1.2" class="ltx_tr">
<span id="A4.T6.1.1.10.1.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">MS COCO</span></span>
</span></span> <span id="A4.T6.1.1.10.1.1.3" class="ltx_text"></span></span></td>
<td id="A4.T6.1.1.10.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">
<span id="A4.T6.1.1.10.2.1" class="ltx_inline-block ltx_align_top">
<span id="A4.T6.1.1.10.2.1.1" class="ltx_p" style="width:350.0pt;">You will be given a list of images. You need to carefully watch all of them. Then you will be given a new sentence, and your goal is to find most relevant image from the list for the given sentence.</span>
</span>
</td>
</tr>
<tr id="A4.T6.1.1.11" class="ltx_tr">
<td id="A4.T6.1.1.11.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="A4.T6.1.1.11.1.1" class="ltx_text"><span id="A4.T6.1.1.11.1.1.1" class="ltx_text"></span> <span id="A4.T6.1.1.11.1.1.2" class="ltx_text">
<span id="A4.T6.1.1.11.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="A4.T6.1.1.11.1.1.2.1.1" class="ltx_tr">
<span id="A4.T6.1.1.11.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">OVEN</span></span>
</span></span> <span id="A4.T6.1.1.11.1.1.3" class="ltx_text"></span></span></td>
<td id="A4.T6.1.1.11.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">
<span id="A4.T6.1.1.11.2.1" class="ltx_inline-block ltx_align_top">
<span id="A4.T6.1.1.11.2.1.1" class="ltx_p" style="width:350.0pt;">You will be given a list of Wikipedia entries which contains Wikipedia ID, Title and Description image. You need to carefully watch all of them. Then you will be given a input image and a question related to the image, and your goal is to find most relevant Wikipedia entry from the list that can be used to best answer the question.</span>
</span>
</td>
</tr>
<tr id="A4.T6.1.1.12" class="ltx_tr">
<td id="A4.T6.1.1.12.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="A4.T6.1.1.12.1.1" class="ltx_text"><span id="A4.T6.1.1.12.1.1.1" class="ltx_text"></span> <span id="A4.T6.1.1.12.1.1.2" class="ltx_text">
<span id="A4.T6.1.1.12.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="A4.T6.1.1.12.1.1.2.1.1" class="ltx_tr">
<span id="A4.T6.1.1.12.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">MSR-VTT</span></span>
</span></span> <span id="A4.T6.1.1.12.1.1.3" class="ltx_text"></span></span></td>
<td id="A4.T6.1.1.12.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">
<span id="A4.T6.1.1.12.2.1" class="ltx_inline-block ltx_align_top">
<span id="A4.T6.1.1.12.2.1.1" class="ltx_p" style="width:350.0pt;">You will be given a list of videos which contains the video ID and video content (present as sequence of images, with timestamp in text). You need to carefully watch all of them. Then you will be given a text query, and your goal is to find most relevant video from the list that can best answer the question.</span>
</span>
</td>
</tr>
<tr id="A4.T6.1.1.13" class="ltx_tr">
<td id="A4.T6.1.1.13.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" colspan="2"><span id="A4.T6.1.1.13.1.1" class="ltx_text ltx_font_bold">Audio Retrieval</span></td>
</tr>
<tr id="A4.T6.1.1.14" class="ltx_tr">
<td id="A4.T6.1.1.14.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="A4.T6.1.1.14.1.1" class="ltx_text"><span id="A4.T6.1.1.14.1.1.1" class="ltx_text"></span> <span id="A4.T6.1.1.14.1.1.2" class="ltx_text">
<span id="A4.T6.1.1.14.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="A4.T6.1.1.14.1.1.2.1.1" class="ltx_tr">
<span id="A4.T6.1.1.14.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">FLEURS-*</span></span>
</span></span> <span id="A4.T6.1.1.14.1.1.3" class="ltx_text"></span></span></td>
<td id="A4.T6.1.1.14.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">
<span id="A4.T6.1.1.14.2.1" class="ltx_inline-block ltx_align_top">
<span id="A4.T6.1.1.14.2.1.1" class="ltx_p" style="width:350.0pt;">You will be given a list of audio which contains Audio ID and audio. You need to carefully listen all of them. Then you will be given a transcript, and your goal is to find most relevant audio from the list that matches the given transcript. Print out the Audio ID of the audio presented in the list.</span>
</span>
</td>
</tr>
<tr id="A4.T6.1.1.15" class="ltx_tr">
<td id="A4.T6.1.1.15.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" colspan="2"><span id="A4.T6.1.1.15.1.1" class="ltx_text ltx_font_bold">SQL</span></td>
</tr>
<tr id="A4.T6.1.1.16" class="ltx_tr">
<td id="A4.T6.1.1.16.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;" rowspan="3"><span id="A4.T6.1.1.16.1.1" class="ltx_text"><span id="A4.T6.1.1.16.1.1.1" class="ltx_text"></span> <span id="A4.T6.1.1.16.1.1.2" class="ltx_text">
<span id="A4.T6.1.1.16.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="A4.T6.1.1.16.1.1.2.1.1" class="ltx_tr">
<span id="A4.T6.1.1.16.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">Spider</span></span>
<span id="A4.T6.1.1.16.1.1.2.1.2" class="ltx_tr">
<span id="A4.T6.1.1.16.1.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">SparC</span></span>
</span></span> <span id="A4.T6.1.1.16.1.1.3" class="ltx_text"></span></span></td>
<td id="A4.T6.1.1.16.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">
<span id="A4.T6.1.1.16.2.1" class="ltx_inline-block ltx_align_top">
<span id="A4.T6.1.1.16.2.1.1" class="ltx_p" style="width:350.0pt;">You will be given a list of tables. You need to read all of the rows of each table. Then you will be given a query, and your goal is to get the answer from the tables. Then format the answer into a list of lists. When formatting the answer into a list of lists, make sure you use the exact fields that are provided in the tables.</span>
</span>
</td>
</tr>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 6: </span>

Instructions used for each <span id="A4.T6.3.1" class="ltx_text ltx_font_smallcaps">LOFT</span> dataset.
We omit instructions for the RAG datasets, which are almost identical to text retrieval instructions.
The ICL task does not use additional instructions, but only many-shot examples in their context.
</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">D.2 </span>Chain-of-Thought Reasoning</h3>

<div id="A4.SS2.p1" class="ltx_para">
<p id="A4.SS2.p1.1" class="ltx_p">We used two types of chain-of-thought reasoning in the few-shot examples in the prompts:</p>
<ul id="A4.I1" class="ltx_itemize">
<li id="A4.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A4.I1.i1.p1" class="ltx_para">
<p id="A4.I1.i1.p1.1" class="ltx_p"><span id="A4.I1.i1.p1.1.1" class="ltx_text ltx_font_bold">Handwritten reasoning.</span> For multi-hop retrieval and RAG datasets (HotPotQA and MuSiQue), as well as SQL datasets (Spider and SparC), we manually wrote reasoning chains for the few-shot queries.</p>
</div>
</li>
<li id="A4.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A4.I1.i2.p1" class="ltx_para">
<p id="A4.I1.i2.p1.1" class="ltx_p"><span id="A4.I1.i2.p1.1.1" class="ltx_text ltx_font_bold">Relevant content.</span> For the rest of the retrieval and RAG datasets, we simply let the model generate the title (or the passage if the title is not available) of each gold document before generating the final answers. This helps explain why the final answers (IDs or short phrases) are the correct ones.</p>
</div>
</li>
</ul>
<p id="A4.SS2.p1.2" class="ltx_p">Many-shot ICL task did not use any chain-of-thought reasoning.
We also noticed that GPT-4o performed much better without chain-of-thought reasoning, so the GPT-4o results presented in this paper did not utilize the chain-of-thought.</p>
</div>
</section>
</section>
<section id="A5" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix E </span>Positional Analysis Detailed Results</h2>

<figure id="A5.F9" class="ltx_figure"><img src="/html/2406.13121/assets/x9.png" id="A5.F9.1.g1" class="ltx_graphics ltx_centering ltx_img_portrait" width="277" height="408" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 9: </span>Detailed metrics of the positional analysis, where we vary the position of gold documents of the test and few-shot queries. <span id="A5.F9.5.1" class="ltx_text ltx_font_bold ltx_font_italic">Top:</span> we vary the gold documents position of test queries within the corpus. <span id="A5.F9.6.2" class="ltx_text ltx_font_bold ltx_font_italic">Middle:</span> we vary the the gold documents position of few-shot queries within the corpus. <span id="A5.F9.7.3" class="ltx_text ltx_font_bold ltx_font_italic">Bottom:</span> we group the gold documents of test and few-shot queries together, and vary their position within the corpus. The average is shown in red.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A6" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix F </span>Ablated Prompt Examples</h2>

<figure id="A6.F10" class="ltx_figure"><img src="/html/2406.13121/assets/x10.png" id="A6.F10.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="162" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 10: </span>

Examples of the task prompts in <span id="A6.F10.2.1" class="ltx_text ltx_font_smallcaps">LOFT</span>.
Each LCLM is expected to do in-context retrieval, reasoning, and many-shot learning on corpora up to millions of tokens.
</figcaption>
</figure>
<figure id="A6.F11" class="ltx_figure"><img src="/html/2406.13121/assets/x11.png" id="A6.F11.1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="211" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 11: </span>Original CiC prompt for HotPotQA, a retrieval dataset in <span id="A6.F11.3.1" class="ltx_text ltx_font_smallcaps">LOFT</span>. The prompt contains an instruction, a corpus, few-shot examples and a query.
</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
<figure id="A6.F12" class="ltx_figure"><img src="/html/2406.13121/assets/x12.png" id="A6.F12.1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="207" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 12: </span>Generic Instruction Ablation, with changes to the original CiC prompt in red. The instruction is changed to a generic one that applies to all tasks in <span id="A6.F12.3.1" class="ltx_text ltx_font_smallcaps">LOFT</span>.
</figcaption>
</figure>
<figure id="A6.F13" class="ltx_figure"><img src="/html/2406.13121/assets/x13.png" id="A6.F13.1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="233" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 13: </span>Query at the Beginning Ablation, with changes to the original CiC prompt in red. The query is placed at the beginning instead of the end.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
<figure id="A6.F14" class="ltx_figure"><img src="/html/2406.13121/assets/x14.png" id="A6.F14.1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="206" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 14: </span>Alphanumeric Document ID Ablation, with changes to the original CiC prompt in red. Instead of using sequential numeric document IDs, a unique random alphanumeric ID is generated with alternating ASCII letters and digits.</figcaption>
</figure>
<figure id="A6.F15" class="ltx_figure"><img src="/html/2406.13121/assets/x15.png" id="A6.F15.1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="211" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 15: </span>Title Only Ablation, with changes to the original CiC prompt in red. In this ablation, the document content is removed, keeping only the document title.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
<figure id="A6.F16" class="ltx_figure"><img src="/html/2406.13121/assets/x16.png" id="A6.F16.1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="212" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 16: </span>ID Echo Ablation, with changes to the original CiC prompt in red. In this ablation, the ID is only mentioned at the beginning of each document, and we remove the ID echo at the end (e.g. "END ID:").</figcaption>
</figure>
<figure id="A6.F17" class="ltx_figure"><img src="/html/2406.13121/assets/x17.png" id="A6.F17.1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="240" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 17: </span>Corpus in Each Few-shot Ablation, with changes to the original CiC prompt in red. In particular, in this ablation, each few-shot example contains a sampled corpus (10 documents), the full corpus is then given before the Query part of the prompt.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
<figure id="A6.F18" class="ltx_figure"><img src="/html/2406.13121/assets/x18.png" id="A6.F18.1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="198" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 18: </span>No Chain-of-Thought Ablation, with changes to the original CiC prompt in red. In particular, in this ablation, we remove the chain-of-thought from the few-shot examples, instead opting to output the answer directly.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A7" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix G </span>Qualitative Analysis of Long-Context Wins and Losses</h2>

<figure id="A7.T7" class="ltx_table">
<div id="A7.T7.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:705.1pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(10.1pt,-16.4pt) scale(1.04879595429009,1.04879595429009) ;">
<table id="A7.T7.1.1" class="ltx_tabular ltx_align_middle">
<tr id="A7.T7.1.1.1" class="ltx_tr">
<td id="A7.T7.1.1.1.1" class="ltx_td ltx_align_center ltx_align_top ltx_border_tt" colspan="2">
<span id="A7.T7.1.1.1.1.1" class="ltx_text" style="font-size:80%;">Task = </span><span id="A7.T7.1.1.1.1.2" class="ltx_text ltx_font_bold" style="font-size:80%;">Retrieval</span>
</td>
</tr>
<tr id="A7.T7.1.1.2" class="ltx_tr">
<td id="A7.T7.1.1.2.1" class="ltx_td ltx_align_center ltx_align_top ltx_border_t" colspan="2">
<span id="A7.T7.1.1.2.1.1" class="ltx_text" style="font-size:80%;">Dataset = </span><span id="A7.T7.1.1.2.1.2" class="ltx_text ltx_font_bold" style="font-size:80%;">Natural Questions</span><span id="A7.T7.1.1.2.1.3" class="ltx_text" style="font-size:80%;"> 1M</span>
</td>
</tr>
<tr id="A7.T7.1.1.3" class="ltx_tr">
<td id="A7.T7.1.1.3.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A7.T7.1.1.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.3.1.1.1" class="ltx_p" style="width:42.7pt;"><span id="A7.T7.1.1.3.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Input</span></span>
</span>
</td>
<td id="A7.T7.1.1.3.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A7.T7.1.1.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.3.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T7.1.1.3.2.1.1.1" class="ltx_text ltx_font_typewriter" style="font-size:80%;">CiC-prompt</span><span id="A7.T7.1.1.3.2.1.1.2" class="ltx_text" style="font-size:80%;">(query="when does monday night raw come on hulu")</span></span>
</span>
</td>
</tr>
<tr id="A7.T7.1.1.4" class="ltx_tr">
<td id="A7.T7.1.1.4.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T7.1.1.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.4.1.1.1" class="ltx_p" style="width:42.7pt;"><span id="A7.T7.1.1.4.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Output</span></span>
</span>
</td>
<td id="A7.T7.1.1.4.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T7.1.1.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.4.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T7.1.1.4.2.1.1.1" class="ltx_text" style="font-size:80%;">The following documents can help answer the query:</span></span>
</span>
</td>
</tr>
<tr id="A7.T7.1.1.5" class="ltx_tr">
<td id="A7.T7.1.1.5.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T7.1.1.5.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T7.1.1.5.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.5.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T7.1.1.5.2.1.1.1" class="ltx_text" style="font-size:80%;">TITLE: WWE Raw | ID: 1023</span></span>
</span>
</td>
</tr>
<tr id="A7.T7.1.1.6" class="ltx_tr">
<td id="A7.T7.1.1.6.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T7.1.1.6.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T7.1.1.6.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.6.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T7.1.1.6.2.1.1.1" class="ltx_text" style="font-size:80%;">Final Answer: ['1023'] </span><span id="A7.T7.1.1.6.2.1.1.2" class="ltx_text" style="font-size:80%;color:#00FF00;">(✓)</span></span>
</span>
</td>
</tr>
<tr id="A7.T7.1.1.7" class="ltx_tr">
<td id="A7.T7.1.1.7.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A7.T7.1.1.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.7.1.1.1" class="ltx_p" style="width:42.7pt;"><span id="A7.T7.1.1.7.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Input</span></span>
</span>
</td>
<td id="A7.T7.1.1.7.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A7.T7.1.1.7.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.7.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T7.1.1.7.2.1.1.1" class="ltx_text ltx_font_typewriter" style="font-size:80%;">CiC-prompt</span><span id="A7.T7.1.1.7.2.1.1.2" class="ltx_text" style="font-size:80%;">(query="where does the synthesis of new dna from existing dna occurs")</span></span>
</span>
</td>
</tr>
<tr id="A7.T7.1.1.8" class="ltx_tr">
<td id="A7.T7.1.1.8.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T7.1.1.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.8.1.1.1" class="ltx_p" style="width:42.7pt;"><span id="A7.T7.1.1.8.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Output</span></span>
</span>
</td>
<td id="A7.T7.1.1.8.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T7.1.1.8.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.8.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T7.1.1.8.2.1.1.1" class="ltx_text" style="font-size:80%;">The following documents can help answer the query:</span></span>
</span>
</td>
</tr>
<tr id="A7.T7.1.1.9" class="ltx_tr">
<td id="A7.T7.1.1.9.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T7.1.1.9.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T7.1.1.9.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.9.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T7.1.1.9.2.1.1.1" class="ltx_text" style="font-size:80%;">TITLE: DNA replication | ID: 3355</span></span>
</span>
</td>
</tr>
<tr id="A7.T7.1.1.10" class="ltx_tr">
<td id="A7.T7.1.1.10.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T7.1.1.10.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T7.1.1.10.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.10.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T7.1.1.10.2.1.1.1" class="ltx_text" style="font-size:80%;">Final Answer: ['3355'] </span><span id="A7.T7.1.1.10.2.1.1.2" class="ltx_text" style="font-size:80%;color:#00FF00;">(✓)</span></span>
</span>
</td>
</tr>
<tr id="A7.T7.1.1.11" class="ltx_tr">
<td id="A7.T7.1.1.11.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A7.T7.1.1.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.11.1.1.1" class="ltx_p" style="width:42.7pt;"><span id="A7.T7.1.1.11.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Input</span></span>
</span>
</td>
<td id="A7.T7.1.1.11.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A7.T7.1.1.11.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.11.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T7.1.1.11.2.1.1.1" class="ltx_text ltx_font_typewriter" style="font-size:80%;">CiC-prompt</span><span id="A7.T7.1.1.11.2.1.1.2" class="ltx_text" style="font-size:80%;">(query="jawaharlal nehru centre for advanced scientific research jakkur campus")</span></span>
</span>
</td>
</tr>
<tr id="A7.T7.1.1.12" class="ltx_tr">
<td id="A7.T7.1.1.12.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T7.1.1.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.12.1.1.1" class="ltx_p" style="width:42.7pt;"><span id="A7.T7.1.1.12.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Output</span></span>
</span>
</td>
<td id="A7.T7.1.1.12.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T7.1.1.12.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.12.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T7.1.1.12.2.1.1.1" class="ltx_text" style="font-size:80%;">TITLE: Jawaharlal Nehru Centre for Advanced Scientific Research | ID: 6911</span></span>
</span>
</td>
</tr>
<tr id="A7.T7.1.1.13" class="ltx_tr">
<td id="A7.T7.1.1.13.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T7.1.1.13.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T7.1.1.13.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.13.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T7.1.1.13.2.1.1.1" class="ltx_text" style="font-size:80%;">Final Answer: ['6911'] </span><span id="A7.T7.1.1.13.2.1.1.2" class="ltx_text" style="font-size:80%;color:#FF0000;">(✗)</span><span id="A7.T7.1.1.13.2.1.1.3" class="ltx_text" style="font-size:80%;"> // Incorrect ID (gold = '5691')</span></span>
</span>
</td>
</tr>
<tr id="A7.T7.1.1.14" class="ltx_tr">
<td id="A7.T7.1.1.14.1" class="ltx_td ltx_align_center ltx_align_top ltx_border_t" colspan="2">
<span id="A7.T7.1.1.14.1.1" class="ltx_text" style="font-size:80%;">Dataset = </span><span id="A7.T7.1.1.14.1.2" class="ltx_text ltx_font_bold" style="font-size:80%;">HotPotQA</span><span id="A7.T7.1.1.14.1.3" class="ltx_text" style="font-size:80%;"> 128k (multi-hop)</span>
</td>
</tr>
<tr id="A7.T7.1.1.15" class="ltx_tr">
<td id="A7.T7.1.1.15.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A7.T7.1.1.15.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.15.1.1.1" class="ltx_p" style="width:42.7pt;"><span id="A7.T7.1.1.15.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Input</span></span>
</span>
</td>
<td id="A7.T7.1.1.15.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A7.T7.1.1.15.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.15.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T7.1.1.15.2.1.1.1" class="ltx_text ltx_font_typewriter" style="font-size:80%;">CiC-prompt</span><span id="A7.T7.1.1.15.2.1.1.2" class="ltx_text" style="font-size:80%;">(query="Which internationl football team has had a player endorse the Nike Hypervenom football boot?")</span></span>
</span>
</td>
</tr>
<tr id="A7.T7.1.1.16" class="ltx_tr">
<td id="A7.T7.1.1.16.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T7.1.1.16.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.16.1.1.1" class="ltx_p" style="width:42.7pt;"><span id="A7.T7.1.1.16.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Output</span></span>
</span>
</td>
<td id="A7.T7.1.1.16.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T7.1.1.16.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.16.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T7.1.1.16.2.1.1.1" class="ltx_text" style="font-size:80%;">The following documents can help answer the query:</span></span>
</span>
</td>
</tr>
<tr id="A7.T7.1.1.17" class="ltx_tr">
<td id="A7.T7.1.1.17.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T7.1.1.17.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T7.1.1.17.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.17.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T7.1.1.17.2.1.1.1" class="ltx_text" style="font-size:80%;">reasoning: The Nike Hypervenom is endorsed by players such as Marcus Rashford based on the document **ID:** 413 (title: Nike Hypervenom).</span></span>
</span>
</td>
</tr>
<tr id="A7.T7.1.1.18" class="ltx_tr">
<td id="A7.T7.1.1.18.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T7.1.1.18.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T7.1.1.18.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.18.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T7.1.1.18.2.1.1.1" class="ltx_text" style="font-size:80%;">reasoning: Marcus Rashford plays for the England national team based on the document **ID:** 558 (title: Marcus Rashford).</span></span>
</span>
</td>
</tr>
<tr id="A7.T7.1.1.19" class="ltx_tr">
<td id="A7.T7.1.1.19.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T7.1.1.19.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T7.1.1.19.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.19.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T7.1.1.19.2.1.1.1" class="ltx_text" style="font-size:80%;">Final Answer: ['413', '558'] </span><span id="A7.T7.1.1.19.2.1.1.2" class="ltx_text" style="font-size:80%;color:#00FF00;">(✓)</span></span>
</span>
</td>
</tr>
<tr id="A7.T7.1.1.20" class="ltx_tr">
<td id="A7.T7.1.1.20.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A7.T7.1.1.20.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.20.1.1.1" class="ltx_p" style="width:42.7pt;"><span id="A7.T7.1.1.20.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Input</span></span>
</span>
</td>
<td id="A7.T7.1.1.20.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A7.T7.1.1.20.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.20.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T7.1.1.20.2.1.1.1" class="ltx_text ltx_font_typewriter" style="font-size:80%;">CiC-prompt</span><span id="A7.T7.1.1.20.2.1.1.2" class="ltx_text" style="font-size:80%;">(query="What NHL team drafted Kaspars Astasenko in 1999 and won the Stanley Cup in 2003-04?")</span></span>
</span>
</td>
</tr>
<tr id="A7.T7.1.1.21" class="ltx_tr">
<td id="A7.T7.1.1.21.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T7.1.1.21.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.21.1.1.1" class="ltx_p" style="width:42.7pt;"><span id="A7.T7.1.1.21.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Output</span></span>
</span>
</td>
<td id="A7.T7.1.1.21.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T7.1.1.21.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.21.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T7.1.1.21.2.1.1.1" class="ltx_text" style="font-size:80%;">The following documents can help answer the query:</span></span>
</span>
</td>
</tr>
<tr id="A7.T7.1.1.22" class="ltx_tr">
<td id="A7.T7.1.1.22.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T7.1.1.22.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T7.1.1.22.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.22.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T7.1.1.22.2.1.1.1" class="ltx_text" style="font-size:80%;">ID: 217 | TITLE: Kaspars Astašenko | CONTENT: Kaspars Astašenko (17 February 1975 – 20 November 2012) was a Latvian professional ice hockey player. Astašenko was born in Riga, Latvia. Astašenko was drafted by the Tampa Bay Lightning in the 1999 NHL Entry Draft, 127th overall. Astašenko played parts of two seasons in the National Hockey League with the Lightning. | END ID: 217</span></span>
</span>
</td>
</tr>
<tr id="A7.T7.1.1.23" class="ltx_tr">
<td id="A7.T7.1.1.23.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T7.1.1.23.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T7.1.1.23.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.23.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T7.1.1.23.2.1.1.1" class="ltx_text" style="font-size:80%;">ID: 508 | TITLE: Tampa Bay Lightning | CONTENT: The Tampa Bay Lightning are a professional ice hockey team based in Tampa, Florida. It is a member of the Atlantic Division of the Eastern Conference of the National Hockey League (NHL). The Lightning have one Stanley Cup championship in their history, in 2003–04. The team is often referred to as the Bolts, and the nickname is used on the current third jersey. The Lightning plays home games in the Amalie Arena in Tampa. | END ID: 508</span></span>
</span>
</td>
</tr>
<tr id="A7.T7.1.1.24" class="ltx_tr">
<td id="A7.T7.1.1.24.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T7.1.1.24.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T7.1.1.24.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.24.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T7.1.1.24.2.1.1.1" class="ltx_text" style="font-size:80%;">Final Answer: ['217', '508'] </span><span id="A7.T7.1.1.24.2.1.1.2" class="ltx_text" style="font-size:80%;color:#00FF00;">(✓)</span></span>
</span>
</td>
</tr>
<tr id="A7.T7.1.1.25" class="ltx_tr">
<td id="A7.T7.1.1.25.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A7.T7.1.1.25.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.25.1.1.1" class="ltx_p" style="width:42.7pt;"><span id="A7.T7.1.1.25.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Input</span></span>
</span>
</td>
<td id="A7.T7.1.1.25.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A7.T7.1.1.25.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.25.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T7.1.1.25.2.1.1.1" class="ltx_text ltx_font_typewriter" style="font-size:80%;">CiC-prompt</span><span id="A7.T7.1.1.25.2.1.1.2" class="ltx_text" style="font-size:80%;">(query="What drum machine was introduced in 1989 and allowed 32 notes to be played simultaneously?")</span></span>
</span>
</td>
</tr>
<tr id="A7.T7.1.1.26" class="ltx_tr">
<td id="A7.T7.1.1.26.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T7.1.1.26.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.26.1.1.1" class="ltx_p" style="width:42.7pt;"><span id="A7.T7.1.1.26.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Output</span></span>
</span>
</td>
<td id="A7.T7.1.1.26.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T7.1.1.26.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.26.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T7.1.1.26.2.1.1.1" class="ltx_text" style="font-size:80%;">The following documents can help answer the query:</span></span>
</span>
</td>
</tr>
<tr id="A7.T7.1.1.27" class="ltx_tr">
<td id="A7.T7.1.1.27.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T7.1.1.27.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T7.1.1.27.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.27.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T7.1.1.27.2.1.1.1" class="ltx_text" style="font-size:80%;">ID: 773 | TITLE: Roland R-8 | CONTENT: The R-8 Human Rhythm Composer is an electronic drum machine introduced in 1989 by Roland Corporation, using PCM voices. … The device has eight individual outputs, 32-voice polyphony, and four-part multitimbral MIDI. | END ID: 773</span></span>
</span>
</td>
</tr>
<tr id="A7.T7.1.1.28" class="ltx_tr">
<td id="A7.T7.1.1.28.1" class="ltx_td ltx_align_top ltx_border_bb"></td>
<td id="A7.T7.1.1.28.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A7.T7.1.1.28.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T7.1.1.28.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T7.1.1.28.2.1.1.1" class="ltx_text" style="font-size:80%;">Final Answer: ['773'] </span><span id="A7.T7.1.1.28.2.1.1.2" class="ltx_text" style="font-size:80%;color:#FF0000;">(✗)</span><span id="A7.T7.1.1.28.2.1.1.3" class="ltx_text" style="font-size:80%;"> // Partially correct ('666' is missing)</span></span>
</span>
</td>
</tr>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering" style="font-size:80%;"><span class="ltx_tag ltx_tag_table">Table 7: </span>Qualitative examples showcasing the successes and failures of the CiC prompting in the text retrieval task.
We use Gemini 1.5 Pro for the analysis.
For HotPotQA, we observed that some of the questions are actually answerable from a single document as illustrated in the last example.</figcaption>
</figure>
<figure id="A7.T8" class="ltx_table">
<div id="A7.T8.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:679.3pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(10.1pt,-15.8pt) scale(1.04879595429009,1.04879595429009) ;">
<table id="A7.T8.1.1" class="ltx_tabular ltx_align_middle">
<tr id="A7.T8.1.1.1" class="ltx_tr">
<td id="A7.T8.1.1.1.1" class="ltx_td ltx_align_center ltx_align_top ltx_border_tt" colspan="2">
<span id="A7.T8.1.1.1.1.1" class="ltx_text" style="font-size:80%;">Task = </span><span id="A7.T8.1.1.1.1.2" class="ltx_text ltx_font_bold" style="font-size:80%;">RAG</span>
</td>
</tr>
<tr id="A7.T8.1.1.2" class="ltx_tr">
<td id="A7.T8.1.1.2.1" class="ltx_td ltx_align_center ltx_align_top ltx_border_t" colspan="2">
<span id="A7.T8.1.1.2.1.1" class="ltx_text" style="font-size:80%;">Dataset = </span><span id="A7.T8.1.1.2.1.2" class="ltx_text ltx_font_bold" style="font-size:80%;">MuSiQue</span><span id="A7.T8.1.1.2.1.3" class="ltx_text" style="font-size:80%;"> 128k (multi-hop)</span>
</td>
</tr>
<tr id="A7.T8.1.1.3" class="ltx_tr">
<td id="A7.T8.1.1.3.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A7.T8.1.1.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.3.1.1.1" class="ltx_p" style="width:42.7pt;"><span id="A7.T8.1.1.3.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Input</span></span>
</span>
</td>
<td id="A7.T8.1.1.3.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A7.T8.1.1.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.3.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T8.1.1.3.2.1.1.1" class="ltx_text ltx_font_typewriter" style="font-size:80%;">CiC-prompt</span><span id="A7.T8.1.1.3.2.1.1.2" class="ltx_text" style="font-size:80%;">(query="Where do greyhound buses leave from in the city the performer of </span><span id="A7.T8.1.1.3.2.1.1.3" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">Born A Lion</span><span id="A7.T8.1.1.3.2.1.1.4" class="ltx_text" style="font-size:80%;"> formed in?")</span></span>
</span>
</td>
</tr>
<tr id="A7.T8.1.1.4" class="ltx_tr">
<td id="A7.T8.1.1.4.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T8.1.1.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.4.1.1.1" class="ltx_p" style="width:42.7pt;"><span id="A7.T8.1.1.4.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Output</span></span>
</span>
</td>
<td id="A7.T8.1.1.4.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T8.1.1.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.4.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T8.1.1.4.2.1.1.1" class="ltx_text" style="font-size:80%;">reasoning: </span><span id="A7.T8.1.1.4.2.1.1.2" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">Born A Lion</span><span id="A7.T8.1.1.4.2.1.1.3" class="ltx_text" style="font-size:80%;"> is the debut album by Canadian rock band </span><span id="A7.T8.1.1.4.2.1.1.4" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">Danko Jones</span><span id="A7.T8.1.1.4.2.1.1.5" class="ltx_text" style="font-size:80%;">.</span></span>
</span>
</td>
</tr>
<tr id="A7.T8.1.1.5" class="ltx_tr">
<td id="A7.T8.1.1.5.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T8.1.1.5.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T8.1.1.5.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.5.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T8.1.1.5.2.1.1.1" class="ltx_text" style="font-size:80%;">reasoning: </span><span id="A7.T8.1.1.5.2.1.1.2" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">Danko Jones</span><span id="A7.T8.1.1.5.2.1.1.3" class="ltx_text" style="font-size:80%;"> is a Canadian rock trio from </span><span id="A7.T8.1.1.5.2.1.1.4" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">Toronto, Ontario</span><span id="A7.T8.1.1.5.2.1.1.5" class="ltx_text" style="font-size:80%;">.</span></span>
</span>
</td>
</tr>
<tr id="A7.T8.1.1.6" class="ltx_tr">
<td id="A7.T8.1.1.6.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T8.1.1.6.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T8.1.1.6.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.6.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T8.1.1.6.2.1.1.1" class="ltx_text" style="font-size:80%;">reasoning: The </span><span id="A7.T8.1.1.6.2.1.1.2" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">Toronto Coach Terminal</span><span id="A7.T8.1.1.6.2.1.1.3" class="ltx_text" style="font-size:80%;"> is the central bus station for inter-city services in </span><span id="A7.T8.1.1.6.2.1.1.4" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">Toronto, Ontario</span><span id="A7.T8.1.1.6.2.1.1.5" class="ltx_text" style="font-size:80%;">, Canada.</span></span>
</span>
</td>
</tr>
<tr id="A7.T8.1.1.7" class="ltx_tr">
<td id="A7.T8.1.1.7.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T8.1.1.7.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T8.1.1.7.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.7.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T8.1.1.7.2.1.1.1" class="ltx_text" style="font-size:80%;">ID: 427 | TITLE: Born a Lion</span></span>
</span>
</td>
</tr>
<tr id="A7.T8.1.1.8" class="ltx_tr">
<td id="A7.T8.1.1.8.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T8.1.1.8.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T8.1.1.8.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.8.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T8.1.1.8.2.1.1.1" class="ltx_text" style="font-size:80%;">ID: 383 | TITLE: Danko Jones</span></span>
</span>
</td>
</tr>
<tr id="A7.T8.1.1.9" class="ltx_tr">
<td id="A7.T8.1.1.9.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T8.1.1.9.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T8.1.1.9.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.9.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T8.1.1.9.2.1.1.1" class="ltx_text" style="font-size:80%;">ID: 724 | TITLE: Toronto Coach Terminal</span></span>
</span>
</td>
</tr>
<tr id="A7.T8.1.1.10" class="ltx_tr">
<td id="A7.T8.1.1.10.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T8.1.1.10.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T8.1.1.10.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.10.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T8.1.1.10.2.1.1.1" class="ltx_text" style="font-size:80%;">Final Answer: ['Toronto Coach Terminal'] </span><span id="A7.T8.1.1.10.2.1.1.2" class="ltx_text" style="font-size:80%;color:#00FF00;">(✓)</span></span>
</span>
</td>
</tr>
<tr id="A7.T8.1.1.11" class="ltx_tr">
<td id="A7.T8.1.1.11.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A7.T8.1.1.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.11.1.1.1" class="ltx_p" style="width:42.7pt;"><span id="A7.T8.1.1.11.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Input</span></span>
</span>
</td>
<td id="A7.T8.1.1.11.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A7.T8.1.1.11.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.11.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T8.1.1.11.2.1.1.1" class="ltx_text ltx_font_typewriter" style="font-size:80%;">CiC-prompt</span><span id="A7.T8.1.1.11.2.1.1.2" class="ltx_text" style="font-size:80%;">(query="When did the rx 350 model of the luxury division of the company that built </span><span id="A7.T8.1.1.11.2.1.1.3" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">Daihatsu boon</span><span id="A7.T8.1.1.11.2.1.1.4" class="ltx_text" style="font-size:80%;"> change body style?")</span></span>
</span>
</td>
</tr>
<tr id="A7.T8.1.1.12" class="ltx_tr">
<td id="A7.T8.1.1.12.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T8.1.1.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.12.1.1.1" class="ltx_p" style="width:42.7pt;"><span id="A7.T8.1.1.12.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Output</span></span>
</span>
</td>
<td id="A7.T8.1.1.12.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T8.1.1.12.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.12.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T8.1.1.12.2.1.1.1" class="ltx_text" style="font-size:80%;">reasoning: The </span><span id="A7.T8.1.1.12.2.1.1.2" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">Daihatsu Boon</span><span id="A7.T8.1.1.12.2.1.1.3" class="ltx_text" style="font-size:80%;"> is also sold as the </span><span id="A7.T8.1.1.12.2.1.1.4" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">Toyota Passo</span><span id="A7.T8.1.1.12.2.1.1.5" class="ltx_text" style="font-size:80%;">.</span></span>
</span>
</td>
</tr>
<tr id="A7.T8.1.1.13" class="ltx_tr">
<td id="A7.T8.1.1.13.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T8.1.1.13.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T8.1.1.13.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.13.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T8.1.1.13.2.1.1.1" class="ltx_text" style="font-size:80%;">reasoning: The </span><span id="A7.T8.1.1.13.2.1.1.2" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">Passo</span><span id="A7.T8.1.1.13.2.1.1.3" class="ltx_text" style="font-size:80%;"> is sold at "</span><span id="A7.T8.1.1.13.2.1.1.4" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">Toyota</span><span id="A7.T8.1.1.13.2.1.1.5" class="ltx_text" style="font-size:80%;"> Corolla Store" Japanese dealerships.</span></span>
</span>
</td>
</tr>
<tr id="A7.T8.1.1.14" class="ltx_tr">
<td id="A7.T8.1.1.14.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T8.1.1.14.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T8.1.1.14.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.14.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T8.1.1.14.2.1.1.1" class="ltx_text" style="font-size:80%;">reasoning: Scion is a discontinued marque of </span><span id="A7.T8.1.1.14.2.1.1.2" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">Toyota</span><span id="A7.T8.1.1.14.2.1.1.3" class="ltx_text" style="font-size:80%;">.</span></span>
</span>
</td>
</tr>
<tr id="A7.T8.1.1.15" class="ltx_tr">
<td id="A7.T8.1.1.15.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T8.1.1.15.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T8.1.1.15.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.15.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T8.1.1.15.2.1.1.1" class="ltx_text" style="font-size:80%;">reasoning: The Lexus RX is a mid-size luxury crossover SUV sold since 1998 by Lexus, a luxury division of </span><span id="A7.T8.1.1.15.2.1.1.2" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">Toyota</span><span id="A7.T8.1.1.15.2.1.1.3" class="ltx_text" style="font-size:80%;">.</span></span>
</span>
</td>
</tr>
<tr id="A7.T8.1.1.16" class="ltx_tr">
<td id="A7.T8.1.1.16.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T8.1.1.16.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T8.1.1.16.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.16.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T8.1.1.16.2.1.1.1" class="ltx_text" style="font-size:80%;">reasoning: The first generation Lexus RX was introduced in 1998 and was produced until 2003.</span></span>
</span>
</td>
</tr>
<tr id="A7.T8.1.1.17" class="ltx_tr">
<td id="A7.T8.1.1.17.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T8.1.1.17.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T8.1.1.17.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.17.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T8.1.1.17.2.1.1.1" class="ltx_text" style="font-size:80%;">reasoning: The second generation Lexus RX was introduced in 2003 and was produced until 2009.</span></span>
</span>
</td>
</tr>
<tr id="A7.T8.1.1.18" class="ltx_tr">
<td id="A7.T8.1.1.18.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T8.1.1.18.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T8.1.1.18.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.18.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T8.1.1.18.2.1.1.1" class="ltx_text" style="font-size:80%;">reasoning: The third generation Lexus RX was introduced in 2008 and was produced until 2015.</span></span>
</span>
</td>
</tr>
<tr id="A7.T8.1.1.19" class="ltx_tr">
<td id="A7.T8.1.1.19.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T8.1.1.19.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T8.1.1.19.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.19.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T8.1.1.19.2.1.1.1" class="ltx_text" style="font-size:80%;">reasoning: The fourth generation Lexus RX was introduced in 2015.</span></span>
</span>
</td>
</tr>
<tr id="A7.T8.1.1.20" class="ltx_tr">
<td id="A7.T8.1.1.20.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T8.1.1.20.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T8.1.1.20.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.20.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T8.1.1.20.2.1.1.1" class="ltx_text" style="font-size:80%;">reasoning: The RX 350 model was introduced in the second generation Lexus RX in 2003.</span></span>
</span>
</td>
</tr>
<tr id="A7.T8.1.1.21" class="ltx_tr">
<td id="A7.T8.1.1.21.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T8.1.1.21.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T8.1.1.21.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.21.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T8.1.1.21.2.1.1.1" class="ltx_text" style="font-size:80%;">Final Answer: ['2003', '2009', '2015'] </span><span id="A7.T8.1.1.21.2.1.1.2" class="ltx_text" style="font-size:80%;color:#FF0000;">(✗)</span><span id="A7.T8.1.1.21.2.1.1.3" class="ltx_text" style="font-size:80%;"> // Incorrect (answer = 'Sales began worldwide in April 2012')</span></span>
</span>
</td>
</tr>
<tr id="A7.T8.1.1.22" class="ltx_tr">
<td id="A7.T8.1.1.22.1" class="ltx_td ltx_align_center ltx_align_top ltx_border_t" colspan="2">
<span id="A7.T8.1.1.22.1.1" class="ltx_text" style="font-size:80%;">Dataset = </span><span id="A7.T8.1.1.22.1.2" class="ltx_text ltx_font_bold" style="font-size:80%;">QUEST</span><span id="A7.T8.1.1.22.1.3" class="ltx_text" style="font-size:80%;"> 128k (multi-answer)</span>
</td>
</tr>
<tr id="A7.T8.1.1.23" class="ltx_tr">
<td id="A7.T8.1.1.23.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A7.T8.1.1.23.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.23.1.1.1" class="ltx_p" style="width:42.7pt;"><span id="A7.T8.1.1.23.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Input</span></span>
</span>
</td>
<td id="A7.T8.1.1.23.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A7.T8.1.1.23.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.23.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T8.1.1.23.2.1.1.1" class="ltx_text ltx_font_typewriter" style="font-size:80%;">CiC-prompt</span><span id="A7.T8.1.1.23.2.1.1.2" class="ltx_text" style="font-size:80%;">(query="History books about </span><span id="A7.T8.1.1.23.2.1.1.3" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">Malaysia</span><span id="A7.T8.1.1.23.2.1.1.4" class="ltx_text" style="font-size:80%;"> or the </span><span id="A7.T8.1.1.23.2.1.1.5" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">Qing dynasty</span><span id="A7.T8.1.1.23.2.1.1.6" class="ltx_text" style="font-size:80%;"> or about </span><span id="A7.T8.1.1.23.2.1.1.7" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">South Korea</span><span id="A7.T8.1.1.23.2.1.1.8" class="ltx_text" style="font-size:80%;">")</span></span>
</span>
</td>
</tr>
<tr id="A7.T8.1.1.24" class="ltx_tr">
<td id="A7.T8.1.1.24.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T8.1.1.24.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.24.1.1.1" class="ltx_p" style="width:42.7pt;"><span id="A7.T8.1.1.24.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Output</span></span>
</span>
</td>
<td id="A7.T8.1.1.24.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T8.1.1.24.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.24.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T8.1.1.24.2.1.1.1" class="ltx_text" style="font-size:80%;">reasoning: Chronicle of Malaysia gives an account of </span><span id="A7.T8.1.1.24.2.1.1.2" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">Malaysia</span><span id="A7.T8.1.1.24.2.1.1.3" class="ltx_text" style="font-size:80%;"> from January 1957 to 31 August 2007.</span></span>
</span>
</td>
</tr>
<tr id="A7.T8.1.1.25" class="ltx_tr">
<td id="A7.T8.1.1.25.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T8.1.1.25.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T8.1.1.25.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.25.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T8.1.1.25.2.1.1.1" class="ltx_text" style="font-size:80%;">reasoning: Treason by the Book is a historical account of the Zeng Jing case which took place during the reign of the Yongzheng Emperor of </span><span id="A7.T8.1.1.25.2.1.1.2" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">Qing</span><span id="A7.T8.1.1.25.2.1.1.3" class="ltx_text" style="font-size:80%;"> China around 1730.</span></span>
</span>
</td>
</tr>
<tr id="A7.T8.1.1.26" class="ltx_tr">
<td id="A7.T8.1.1.26.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T8.1.1.26.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T8.1.1.26.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.26.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T8.1.1.26.2.1.1.1" class="ltx_text" style="font-size:80%;">reasoning: Korea: A Walk Through the Land of Miracles is a book by Simon Winchester. He recounts his experience walking across </span><span id="A7.T8.1.1.26.2.1.1.2" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">South Korea</span><span id="A7.T8.1.1.26.2.1.1.3" class="ltx_text" style="font-size:80%;">, from Jeju in the south to the DMZ in the north.</span></span>
</span>
</td>
</tr>
<tr id="A7.T8.1.1.27" class="ltx_tr">
<td id="A7.T8.1.1.27.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T8.1.1.27.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T8.1.1.27.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.27.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T8.1.1.27.2.1.1.1" class="ltx_text" style="font-size:80%;">Final Answer: ['Chronicle of Malaysia', 'Korea: A Walk Through the Land of Miracles', 'Treason by the Book'] </span><span id="A7.T8.1.1.27.2.1.1.2" class="ltx_text" style="font-size:80%;color:#00FF00;">(✓)</span></span>
</span>
</td>
</tr>
<tr id="A7.T8.1.1.28" class="ltx_tr">
<td id="A7.T8.1.1.28.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A7.T8.1.1.28.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.28.1.1.1" class="ltx_p" style="width:42.7pt;"><span id="A7.T8.1.1.28.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Input</span></span>
</span>
</td>
<td id="A7.T8.1.1.28.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A7.T8.1.1.28.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.28.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T8.1.1.28.2.1.1.1" class="ltx_text ltx_font_typewriter" style="font-size:80%;">CiC-prompt</span><span id="A7.T8.1.1.28.2.1.1.2" class="ltx_text" style="font-size:80%;">(query="Films about sisters shot in </span><span id="A7.T8.1.1.28.2.1.1.3" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">Paris</span><span id="A7.T8.1.1.28.2.1.1.4" class="ltx_text" style="font-size:80%;">")</span></span>
</span>
</td>
</tr>
<tr id="A7.T8.1.1.29" class="ltx_tr">
<td id="A7.T8.1.1.29.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T8.1.1.29.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.29.1.1.1" class="ltx_p" style="width:42.7pt;"><span id="A7.T8.1.1.29.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Output</span></span>
</span>
</td>
<td id="A7.T8.1.1.29.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T8.1.1.29.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.29.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T8.1.1.29.2.1.1.1" class="ltx_text" style="font-size:80%;">reasoning: Peppermint Soda is a 1977 French comedy-drama film directed by Diane Kurys. The film follows two teenage sisters over the course of the year 1963. The high school where the film takes place is the Lycée Jules-Ferry in </span><span id="A7.T8.1.1.29.2.1.1.2" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">Paris</span><span id="A7.T8.1.1.29.2.1.1.3" class="ltx_text" style="font-size:80%;">, France.</span></span>
</span>
</td>
</tr>
<tr id="A7.T8.1.1.30" class="ltx_tr">
<td id="A7.T8.1.1.30.1" class="ltx_td ltx_align_top ltx_border_bb"></td>
<td id="A7.T8.1.1.30.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A7.T8.1.1.30.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T8.1.1.30.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T8.1.1.30.2.1.1.1" class="ltx_text" style="font-size:80%;">Final Answer: ['Peppermint Soda'] </span><span id="A7.T8.1.1.30.2.1.1.2" class="ltx_text" style="font-size:80%;color:#FF0000;">(✗)</span><span id="A7.T8.1.1.30.2.1.1.3" class="ltx_text" style="font-size:80%;"> // Partially correct (missing 'Le Divorce' and 'Two English Girls')</span></span>
</span>
</td>
</tr>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering" style="font-size:80%;"><span class="ltx_tag ltx_tag_table">Table 8: </span>Qualitative examples showcasing the successes and failures of the CiC prompting in the RAG task.
We use Gemini 1.5 Pro for the analysis.
In each example, we underline correct reasoning chains provided by the model.</figcaption>
</figure>
<figure id="A7.T9" class="ltx_table">
<div id="A7.T9.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:852pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(10.1pt,-19.8pt) scale(1.04879595429009,1.04879595429009) ;">
<table id="A7.T9.1.1" class="ltx_tabular ltx_align_middle">
<tr id="A7.T9.1.1.1" class="ltx_tr">
<td id="A7.T9.1.1.1.1" class="ltx_td ltx_align_center ltx_align_top ltx_border_tt" colspan="2">
<span id="A7.T9.1.1.1.1.1" class="ltx_text" style="font-size:80%;">Task = </span><span id="A7.T9.1.1.1.1.2" class="ltx_text ltx_font_bold" style="font-size:80%;">SQL</span>
</td>
</tr>
<tr id="A7.T9.1.1.2" class="ltx_tr">
<td id="A7.T9.1.1.2.1" class="ltx_td ltx_align_center ltx_align_top ltx_border_t" colspan="2">
<span id="A7.T9.1.1.2.1.1" class="ltx_text" style="font-size:80%;">Dataset = </span><span id="A7.T9.1.1.2.1.2" class="ltx_text ltx_font_bold" style="font-size:80%;">Spider</span><span id="A7.T9.1.1.2.1.3" class="ltx_text" style="font-size:80%;"> 1M</span>
</td>
</tr>
<tr id="A7.T9.1.1.3" class="ltx_tr">
<td id="A7.T9.1.1.3.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A7.T9.1.1.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.3.1.1.1" class="ltx_p" style="width:42.7pt;"><span id="A7.T9.1.1.3.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Input</span></span>
</span>
</td>
<td id="A7.T9.1.1.3.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A7.T9.1.1.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.3.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.3.2.1.1.1" class="ltx_text ltx_font_typewriter" style="font-size:80%;">CiC-prompt</span><span id="A7.T9.1.1.3.2.1.1.2" class="ltx_text" style="font-size:80%;">(query="Find the year which offers the largest number of courses.")</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.4" class="ltx_tr">
<td id="A7.T9.1.1.4.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.4.1.1.1" class="ltx_p" style="width:42.7pt;"><span id="A7.T9.1.1.4.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Output</span></span>
</span>
</td>
<td id="A7.T9.1.1.4.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.4.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.4.2.1.1.1" class="ltx_text" style="font-size:80%;">Here’s a step-by-step approach using the provided tables:</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.5" class="ltx_tr">
<td id="A7.T9.1.1.5.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.5.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.5.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.5.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.5.2.1.1.1" class="ltx_text" style="font-size:80%;">**1. Identify relevant data:**</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.6" class="ltx_tr">
<td id="A7.T9.1.1.6.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.6.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.6.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.6.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.6.2.1.1.1" class="ltx_text" style="font-size:80%;">We need to look at the "year" column in the "section" table, which lists the year each course section was offered.</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.7" class="ltx_tr">
<td id="A7.T9.1.1.7.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.7.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.7.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.7.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.7.2.1.1.1" class="ltx_text" style="font-size:80%;">**2. Count courses per year:**</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.8" class="ltx_tr">
<td id="A7.T9.1.1.8.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.8.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.8.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.8.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.8.2.1.1.1" class="ltx_text" style="font-size:80%;">Group the data by “year” and count the number of unique “course_id” values for each year. This will give us the number of distinct courses offered in each year.</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.9" class="ltx_tr">
<td id="A7.T9.1.1.9.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.9.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.9.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.9.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.9.2.1.1.1" class="ltx_text" style="font-size:80%;">**3. Find the year with the maximum count:**</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.10" class="ltx_tr">
<td id="A7.T9.1.1.10.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.10.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.10.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.10.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.10.2.1.1.1" class="ltx_text" style="font-size:80%;">Identify the year with the highest count of distinct courses.</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.11" class="ltx_tr">
<td id="A7.T9.1.1.11.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.11.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.11.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.11.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.11.2.1.1.1" class="ltx_text" style="font-size:80%;">**Based on the data provided, here’s the breakdown of courses per year:**</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.12" class="ltx_tr">
<td id="A7.T9.1.1.12.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.12.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.12.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.12.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.12.2.1.1.1" class="ltx_text" style="font-size:80%;">| Year | Number of Courses |</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.13" class="ltx_tr">
<td id="A7.T9.1.1.13.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.13.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.13.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.13.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.13.2.1.1.1" class="ltx_text" style="font-size:80%;">|—|—|</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.14" class="ltx_tr">
<td id="A7.T9.1.1.14.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.14.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.14.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.14.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.14.2.1.1.1" class="ltx_text" style="font-size:80%;">| 2001 | 29 |</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.15" class="ltx_tr">
<td id="A7.T9.1.1.15.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.15.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.15.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.15.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.15.2.1.1.1" class="ltx_text" style="font-size:80%;">| 2002 | 36 |</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.16" class="ltx_tr">
<td id="A7.T9.1.1.16.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.16.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.16.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.16.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.16.2.1.1.1" class="ltx_text" style="font-size:80%;">| 2003 | 38 |</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.17" class="ltx_tr">
<td id="A7.T9.1.1.17.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.17.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.17.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.17.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.17.2.1.1.1" class="ltx_text" style="font-size:80%;">| 2004 | 31 |</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.18" class="ltx_tr">
<td id="A7.T9.1.1.18.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.18.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.18.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.18.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.18.2.1.1.1" class="ltx_text" style="font-size:80%;">| 2005 | 31 |</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.19" class="ltx_tr">
<td id="A7.T9.1.1.19.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.19.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.19.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.19.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.19.2.1.1.1" class="ltx_text" style="font-size:80%;">| 2006 | 41 |</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.20" class="ltx_tr">
<td id="A7.T9.1.1.20.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.20.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.20.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.20.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.20.2.1.1.1" class="ltx_text" style="font-size:80%;">| 2007 | 40 |</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.21" class="ltx_tr">
<td id="A7.T9.1.1.21.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.21.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.21.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.21.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.21.2.1.1.1" class="ltx_text" style="font-size:80%;">| 2008 | 37 |</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.22" class="ltx_tr">
<td id="A7.T9.1.1.22.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.22.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.22.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.22.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.22.2.1.1.1" class="ltx_text" style="font-size:80%;">| 2009 | 35 |</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.23" class="ltx_tr">
<td id="A7.T9.1.1.23.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.23.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.23.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.23.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.23.2.1.1.1" class="ltx_text" style="font-size:80%;">| 2010 | 39 |</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.24" class="ltx_tr">
<td id="A7.T9.1.1.24.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.24.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.24.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.24.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.24.2.1.1.1" class="ltx_text" style="font-size:80%;">**The year 2006 has the largest number of courses offered (41).**</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.25" class="ltx_tr">
<td id="A7.T9.1.1.25.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.25.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.25.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.25.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.25.2.1.1.1" class="ltx_text" style="font-size:80%;">Final Answer: ['2006'] </span><span id="A7.T9.1.1.25.2.1.1.2" class="ltx_text" style="font-size:80%;color:#00FF00;">(✓)</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.26" class="ltx_tr">
<td id="A7.T9.1.1.26.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A7.T9.1.1.26.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.26.1.1.1" class="ltx_p" style="width:42.7pt;"><span id="A7.T9.1.1.26.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Input</span></span>
</span>
</td>
<td id="A7.T9.1.1.26.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A7.T9.1.1.26.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.26.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.26.2.1.1.1" class="ltx_text ltx_font_typewriter" style="font-size:80%;">CiC-prompt</span><span id="A7.T9.1.1.26.2.1.1.2" class="ltx_text" style="font-size:80%;">(query="List total amount of invoice from Chicago, IL.")</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.27" class="ltx_tr">
<td id="A7.T9.1.1.27.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.27.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.27.1.1.1" class="ltx_p" style="width:42.7pt;"><span id="A7.T9.1.1.27.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Output</span></span>
</span>
</td>
<td id="A7.T9.1.1.27.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.27.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.27.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.27.2.1.1.1" class="ltx_text" style="font-size:80%;">Here’s a step-by-step approach using the provided tables:</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.28" class="ltx_tr">
<td id="A7.T9.1.1.28.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.28.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.28.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.28.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.28.2.1.1.1" class="ltx_text" style="font-size:80%;">**1. Identify relevant data:**</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.29" class="ltx_tr">
<td id="A7.T9.1.1.29.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.29.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.29.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.29.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.29.2.1.1.1" class="ltx_text" style="font-size:80%;">* **Invoices:** This table contains the billing city and total for each invoice.</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.30" class="ltx_tr">
<td id="A7.T9.1.1.30.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.30.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.30.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.30.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.30.2.1.1.1" class="ltx_text" style="font-size:80%;">**2. Filter invoices from Chicago, IL:**</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.31" class="ltx_tr">
<td id="A7.T9.1.1.31.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.31.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.31.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.31.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.31.2.1.1.1" class="ltx_text" style="font-size:80%;">* Scan through the “billing_city” column and identify invoices where the city is “Chicago” and the state is “IL”.</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.32" class="ltx_tr">
<td id="A7.T9.1.1.32.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.32.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.32.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.32.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.32.2.1.1.1" class="ltx_text" style="font-size:80%;">* **Based on the data provided, invoices 24, 92, 103, 158, 287, 310, 332, and 384 are from Chicago, IL.**</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.33" class="ltx_tr">
<td id="A7.T9.1.1.33.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.33.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.33.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.33.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.33.2.1.1.1" class="ltx_text" style="font-size:80%;">**3. Sum the total amounts:**</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.34" class="ltx_tr">
<td id="A7.T9.1.1.34.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.34.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.34.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.34.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.34.2.1.1.1" class="ltx_text" style="font-size:80%;">* Add up the "total" values for the invoices identified in step 2.</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.35" class="ltx_tr">
<td id="A7.T9.1.1.35.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.35.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.35.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.35.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.35.2.1.1.1" class="ltx_text" style="font-size:80%;">* **The total amounts for the invoices are 1.98, 1.98, 15.86, 8.91, 1.98, 7.96, 5.94, and 0.99, respectively.**</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.36" class="ltx_tr">
<td id="A7.T9.1.1.36.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.36.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.36.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.36.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.36.2.1.1.1" class="ltx_text" style="font-size:80%;">* **The sum of these amounts is 36.6.**</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.37" class="ltx_tr">
<td id="A7.T9.1.1.37.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.37.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.37.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.37.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.37.2.1.1.1" class="ltx_text" style="font-size:80%;">**4. Present the result:**</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.38" class="ltx_tr">
<td id="A7.T9.1.1.38.1" class="ltx_td ltx_align_top"></td>
<td id="A7.T9.1.1.38.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T9.1.1.38.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.38.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.38.2.1.1.1" class="ltx_text" style="font-size:80%;">* The final output will be a single value representing the total amount of invoices from Chicago, IL.</span></span>
</span>
</td>
</tr>
<tr id="A7.T9.1.1.39" class="ltx_tr">
<td id="A7.T9.1.1.39.1" class="ltx_td ltx_align_top ltx_border_bb"></td>
<td id="A7.T9.1.1.39.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A7.T9.1.1.39.2.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T9.1.1.39.2.1.1" class="ltx_p" style="width:341.4pt;"><span id="A7.T9.1.1.39.2.1.1.1" class="ltx_text" style="font-size:80%;">Final Answer: ['36.6'] </span><span id="A7.T9.1.1.39.2.1.1.2" class="ltx_text" style="font-size:80%;color:#FF0000;">(✗)</span></span>
</span>
</td>
</tr>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering" style="font-size:80%;"><span class="ltx_tag ltx_tag_table">Table 9: </span>Qualitative examples showcasing the successes and failures of the CiC prompting in the SQL task.
We use Gemini 1.5 Pro for the analysis.
In SQL, the long-context model first outputs a reasoning chain in natural language ideally simulating the execution of the SQL query before arriving at the prediction.
</figcaption>
</figure>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2406.13120" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2406.13121" class="ar5iv-text-button ar5iv-severity-error">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2406.13121">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2406.13121" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2406.13122" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Fri Jul  5 18:14:27 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2406.13121
Long-context language models (LCLMs)  have the potential to revolutionize our approach to tasks traditionally reliant on external tools like retrieval systems or databases.
Leveraging LCLMs’ ability to natively ingest and process entire corpora of information offers numerous advantages.
It enhances user-friendliness by eliminating the need for specialized knowledge of tools, provides robust end-to-end modeling that minimizes cascading errors in complex pipelines, and allows for the application of sophisticated prompting techniques across the entire system.
To assess this paradigm shift, we introduceLOFT, a benchmark of real-world tasks requiring context up to millions of tokens designed to evaluate LCLMs’ performance on in-context retrieval and reasoning.
Our findings reveal LCLMs’ surprising ability to rival state-of-the-art retrieval and RAG systems, despite never having been explicitly trained for these tasks.
However, LCLMs still face challenges in areas like compositional reasoning that are required in SQL-like tasks.
Notably, prompting strategies significantly influence performance, emphasizing the need for continued research as context lengths grow.
Overall,LOFTprovides a rigorous testing ground for LCLMs, showcasing their potential to supplant existing paradigms and tackle novel tasks as model capabilities scale.111TheLOFTbenchmark is available athttps://github.com/google-deepmind/loft.Figure 1:An overview of theLOFTbenchmark, made of six tasks which measure LCLMs’ ability to do in-context retrieval, reasoning, and many-shot learning on corpora up to millions of tokens.
We compare the performance of LCLMs against specialized models (e.g.,CLIP for visual retrieval) , which often rely on complex task-specific fine-tuning or pipelining.
Unlike specialized models, we show how LCLMs can simplify various tasks through Corpus-in-Context Prompting (Section3) .
abstract
Can Long-Context Language Models SubsumeRetrieval, RAG, SQL, and More?
Abstract
2406.13121
1IntroductionLong-context language models (LCLMs) hold the promise of reshaping artificial intelligence by enabling entirely new tasks and applications while eliminating the reliance on tools and complex pipelines previously necessary due to context length limitations.
By consolidating complex pipelines into a unified model, LCLMs ameliorate issues like cascading errorsand cumbersome optimization, offering a streamlined end-to-end approach to model development.
Moreover, techniques such as adding instructions, incorporating few-shot examples, and leveraging demonstrations via chain-of-thought promptingcan be seamlessly integrated to optimize LCLMs for the task at hand.However, realizing the full potential of LCLMs requires rigorous evaluation on truly long-context tasks useful in real-world applications.
Existing benchmarks fall short in this regard, relying on synthetic tasks like the popular “needle-in-haystack”or fixed-length datasets that fail to keep pace with the evolving definition of “long-context”.
Critically, existing evaluations do not adequately stress-test LCLMs on any paradigm-shifting tasks.To address this, we introduce theLong-ContextFrontiers (LOFT)  benchmark, a suite of six tasks consisting of 35 datasets which span text, visual, and audio modalities, to push LCLMs to their limits and gauge their real-world impact.
Unlike previous benchmarks,LOFTallows for automatic creation of increasing context lengths. While the current version extends to one million tokens, it can easily be extended further to tens of millions, ensuring rigorous evaluation as LCLMs continue to scale.LOFTfocuses on the following areas where LCLMs have the potential for disruption:•Retrieval:
LCLMs can directly ingest and retrieve information from a corpus, eliminating the need for separate dual-encoder models. This addresses long-standing challenges in retrieval systems such as multi-hop reasoning, instruction following, and few-shot task adaptation.
We assess retrieval performance across text, visual, and audio modalities.•Retrieval-Augmented Generation (RAG) :
LCLMs simplify RAG pipelines by directly reasoning over a corpus, overcoming challenges like query decompositionand mitigating cascading errors due to retrieval misses.•SQL:
We explore LCLMs’ capacity to process entire databases as text, enabling natural language database querying and bypassing conversion to a formal query language like SQL.
This potentially enables more expressive querying and handling of noisy or mixed-structured data. Importantly, it can also be seen as a case study representing the ability of LCLMs to subsume other types of structured data and complicated formal languages to query them, such as knowledge graphs that often require bespoke solutions.•Many-Shot ICL:
LCLMs can scale the number of examples from the tens in the traditional in-context learning setup to hundreds or thousands, removing the need to find the optimal set of few-shot examples to use.TheLOFTbenchmark opens up a novel line of research on long-context prompting, which we introduce as Corpus-in-Context (CiC)  Prompting (Section3) . Using this approach, we evaluate Gemini 1.5 Pro, GPT-4o, and Claude 3 OpusonLOFT. Figure1summarizes the performance of these LCLMs and specialized models that were carefully hand-optimized for each task, showcasing how LCLMs can tackleLOFTtasks without specialized pipelines.Our evaluation onLOFTreveals several key insights when comparing state-of-the-art LCLMs with specialized, task-specific models. At the 128k token level, the largest size comparable across all models, LCLMs rival the performance of Gecko, a leading textual retrieval system. Notably, Geminialso surpasses strong multi-modal retrieval models such as CLIP. However, LCLMs lag significantly on complex multi-hop compositional reasoning tasks, indicating substantial room for improvement.
Furthermore, rigorous ablations reveal large performance variance depending on prompting strategies such as chain-of-thought reasoning, underscoring the need for further research to enhance LCLMs robustness and instructability.
Taken together, our results onLOFTdemonstrate that LCLMs can match the performance of many specialized models, while also revealing ample headroom for improvement in robust long-context reasoning as context windows continue to scale.
introduction
Can Long-Context Language Models SubsumeRetrieval, RAG, SQL, and More?
1Introduction
2406.13121
2LOFT: A 1 Million+ Token Long-Context BenchmarkTaskDatasetDescriptionAvg. Cand.Length# Cand.(128k) CandidatesInputTargetTextRetrievalArguAnaArgument Retrieval196531PassagesQueryPassage ID(s) FEVERFact Checking176588FIQAQuestion Answering196531MS MARCOWeb Search771,174NQQuestion Answering110883QuoraDuplication Detection143,306SciFactCitation Prediction301357Touché-2020Argument Retrieval330329TopiOCQAMulti-turn QA149680HotPotQAMulti-hop QA741,222MuSiQueMulti-hop QA120824QAMPARIMulti-target QA132755QUESTMulti-target QA328328VisualRetrievalFlickr30kImage Retrieval258440ImagesText QueryImage IDMS COCOImage Retrieval258440ImagesText QueryImage IDOVENImage-text Retrieval278448Images+TextsImage+Text QueryWikipedia IDMSR-VTTVideo Retrieval774140VideosText QueryVideo IDAudioRetrievalFLEURS-enAudio Retrieval249428SpeechText QuerySpeech IDFLEURS-es315343FLEURS-fr259412FLEURS-hi292369FLEURS-zh291370RAGNQQuestion Answering110883PassagesQuestionAnswer(s) TopiOCQAMulti-turn QA149680HotPotQAMulti-hop QA741,222MuSiQueMulti-hop QA120824QAMPARIMulti-target QA132755QUESTMulti-target QA328328SQLSpiderSingle-turn SQL111k1SQLDatabaseQuestionAnswerSParCMulti-turn SQL111k1Many-ShotICLBBH-dateMultiple-choice QA131150TrainingExamplesQuestionAnswerBBH-salientMultiple-choice QA246104BBH-tracking7Multiple-choice QA205123BBH-webMultiple-choice QA43150LIB-dialogueClassification266274Table 1:Tasks and datasets in theLOFTbenchmark.LOFThas 6 types of tasks, 4 modalities, and 35 datasets in total.
For each dataset, we show the average length of the candidates (Avg. Cand. Length)  as well as the number of candidates (# Cand)  in the 128k version ofLOFT.
More details on the datasets are available inAppendicesA,BandC.TheLOFTbenchmark aims to cover a wide range of real-world applications where LCLMs can be employed. These tasks range from retrieving relevant documents for a query to extracting compositional information from databases.Table1lists all tasks and their corresponding datasets.For each dataset in all tasks, we sample up to 100 test queries, 5 few-shot queries, and 10 development queries.
To test how LCLMs perform when scaling the number of tokens in their context, we createLOFTwith three different context length limits, namely 32k222Since the gold documents of 100 test queries alone often exceed 32k tokens, we do not include test queries for the 32k version. We report the development set performance for 32k instead., 128k, and 1M.LOFTcurrently supports 1 million token contexts, but with the increasing capabilities of state-of-the-art LCLMs, our method allowsLOFTto easily scale to larger context lengths (e.g., 1 billion tokens)  in the future.
To allow testing the same set of queries over different context lengths, we process each dataset to have the same evaluation queries across different context lengths (except for SQL as detailed below) .Retrieval & RAGWe include diverse text retrieval and RAG datasets, covering heterogeneous retrieval tasks from BEIR, multi-turn conversational QA, multi-hop QA, as well as multi-target QA that requires set operations such as unions or differences.
For retrieval, we also include multimodal datasets, covering image, video, and audio.Figure 2:Corpus creationfor retrieval and RAG.
Given a set of test queries, we use their associated gold passages and other random passages to form the corpus.All queries in each retrieval and RAG dataset share a single corpus, mimicking real retrieval applications. To create this shared corpus, we first include all gold passages from few-shot, development and the test queries, and then sample passages uniformly until reaching the desired context size (Figure2) . This construction ensures smaller corpora (e.g., 128k)  are subsets of larger ones (e.g., 1M) .
Gold and random passages are shuffled to avoid positional biases.
For fair comparison, specialized retriever models also use the same corpora for the evaluation.Many-shot ICLWe adapt datasets from Big-Bench Hard (BBH) and LongICLBench (LIB) to evaluate many-shot in-context learning (ICL)  capabilities.
Similar to retrieval, we construct shared many-shot ICL contexts, ensuring training examples in smaller contexts are included in larger ones.
Since all of the many-shot ICL datasets are classification tasks, we guarantee that each class is represented at least once.SQLWe evaluate SQL-like reasoning on Spider, a single-turn text-to-SQL dataset, and SparC, its multi-turn variant.
The corpus for each query is an associated database of one or more tables. To construct the corpus for each context length, we must select databases such that all of its tables are no larger than that context length. We do this by selecting the largest databases that will fit into that context. This necessarily means that the databases for the 1M token setting would not fit into the smaller context lengths. Therefore, unlike most of the other tasks that share a corpus, the query sets differ acrossLOFTsizes.Given a maximum context length of, we create a corpus up to a size of, to account for differences in tokenizers, as well as to reserve room for instructions and formatting, which will be explained in more detail inSection3.
Please refer toAppendixAfor more details about how the datasets are selected for each task.
body
Can Long-Context Language Models SubsumeRetrieval, RAG, SQL, and More?
2LOFT: A 1 Million+ Token Long-Context Benchmark
2406.13121
3Corpus-in-Context PromptingTraditionally, utilizing large corpora of passages, data tables, or training examples requires specialized recipes or systems.
LCLMs now enable direct ingestion and processing of entire corpora within their context window. This unlocks a novel prompting-based approach for solving new and existing tasks, which we callCorpus-in-Context prompting (CiC, pronounced "seek") .3.1Prompt DesignCiC prompting effectively combines established prompting strategies, tailoring them to leverage the unique capabilities of LCLMs for learning, retrieving and reasoning over in-context corpora.Figure3illustrates our key design choices, whose effectiveness is rigorously evaluated through extensive ablation studies inSection5.InstructionsWe first provide task-specific instructions to guide the LCLM’s behaviors.
As an example, for the retrieval task shown inFigure3, we ask the model to read the corpus carefully and find relevant documents to answer the question.Corpus FormattingWe then insert the entire corpus into the prompt.
Each candidate (e.g., passage, image, audio)  in a corpus is assigned a unique identifier (ID)  that can be referenced as needed for that task.333Original candidate IDs from each dataset were not used as they inadvertently disclose the gold documents.For instance, in retrieval tasks, the LCLMs must output the correct candidate IDs.
The structure of the corpus significantly impacts retrieval performance.
Careful formatting, such as putting document IDs both before and after the passage in text retrieval, can mitigate the effects of causal attention in decoder-only LCLMs and enhance retrieval accuracy.Few-Shot ExamplesProviding a limited number of demonstrations helps the LCLM grasp the desired response format and improves task accuracy. Unlike common approaches where few-shot examples are independent, we ground all examples to the same corpus, aiming to teach the model to also learn more details about the specific corpus it needs to use.444As we will see inSection4, carefully positioning the answers of these few-shot examples can guide the model’s attention to areas where it is typically weaker, mitigating ”dead zones” in attention distribution.To facilitate automated evaluation, answers within each few-shot example are formatted as a list (e.g., "Final Answer: [54, 0]" inFigure3) , thus guiding the model to generate responses in a similar structure that can be readily parsed and compared against ground truth labels.
Finally, each few-shot example is accompanied by a Chain-of-Thought reasoning.
We find adding Chain-of-Thought reasoning helps the most on tasks requiring complex multi-hop compositional reasoning.Query FormattingThe query to be evaluated is formatted similar to the few-shot examples. For multi-turn datasets, we prepend previous query turns and model outputs to the current query turn, ensuring that the model’s generation is conditioned on its prior responses.
Based on our query formatting, LCLMs generate tokens that are parsed into the final answer.Figure 3:Example of Corpus-in-Context Promptingfor retrieval.
CiC prompting leverages large language models’ capacity to follow instructions, leverage few-shot examples, and benefit from reasoning demonstrations to retrieve and reason over large corpora provided in context.AppendixDprovides examples of instructions for all datasets inLOFT, along with details on the chain-of-thought reasonings used in this paper.3.2Design ConsiderationGiven the variation in instructions, format, and tokenizers across different CiC prompting techniques, the resulting context lengths can differ substantially.
To accommodate this diversity, we allocate ample space for prompt customization, as detailed inSection2.
To ensure a fair comparison among LCLMs, we strongly recommend that for each maximum context length ofLOFT(e.g., 32k or 128k) , the model still uses only the corpus and examples present in that version. We also recommend to evaluate models on the maximum size that can fit into their context length without truncating the corpus or any of the individual examples.3.3Discussion on EfficiencyEncoding a one million token context can be slow and computationally expensive.
One key advantage of CiC prompting is its compatibility with prefix-caching in autoregressive language modelsas the query appears at the end of the prompt.
This meansthe corpus only needs to be encoded once, similar to the indexing process in traditional information retrieval.
As demonstrated inSection5, encoding the corpus as the prefix in this way does not lead to a performance drop in LCLMs.DatasetGemini 1.5 ProGPT-4oClaude 3 OpusSpecializedTextRetrievalArguAna0.840.850.740.75FEVER0.980.960.940.97FIQA0.790.820.610.83MS MARCO0.950.870.930.97NQ1.000.990.960.99Quora0.930.930.941.00SciFact0.880.880.730.85Touché-20200.910.880.710.88TopiOCQA0.490.300.420.36HotPotQA†0.900.820.830.92MuSiQue†0.420.100.270.29QAMPARI†0.610.180.200.57QUEST†0.300.190.180.54Average0.770.670.650.76VisualRetrievalFlickr30k0.840.65-0.75MS COCO0.770.44-0.66MSR-VTT0.760.72-0.64OVEN0.930.89-0.79Average0.830.68-0.71AudioRetrievalFLEURS-en1.00--0.98FLEURS-es0.99--0.99FLEURS-fr1.00--1.00FLEURS-hi1.00--0.74FLEURS-zh1.00--1.00Average1.00--0.94RAGNQ0.840.890.850.71TopiOCQA0.340.330.370.35HotPotQA0.750.720.740.70MuSiQue0.550.470.450.45QAMPARI0.440.270.250.55QUEST0.280.200.150.35Average0.530.480.470.52SQLSpider0.400.140.190.74SParC0.360.130.210.55Average0.380.130.200.65Many-ShotICLBBH-date0.880.810.92-BBH-salient0.780.640.69-BBH-tracking70.330.810.54-BBH-web0.670.570.83-LIB-dialogue0.760.670.72-Average0.680.700.74-Table 2:Main Results onLOFT128k context test set.
We show performances of three LCLMs as well as specialized models that rely on task-specific fine-tuning or pipelining.
For the evaluation metrics: text, visual, and audio retrieval use Recall@1; RAG uses subspan exact match; SQL uses accuracy; and many-shot ICL uses classification accuracy.†: For retrieval with multiple gold targets, MRecall@(in order)  is employed as described inAppendixA.
body
Can Long-Context Language Models SubsumeRetrieval, RAG, SQL, and More?
3Corpus-in-Context Prompting
2406.13121
4LOFTTasks and Primary ResultsWe evaluate three state-of-the-art LCLMs onLOFT: Google’sGemini 1.5 Pro, OpenAI’sGPT-4o, and Anthropic’sClaude 3 Opus.
We evaluate these models because their APIs support most of the modalities.
Their maximum context lengths are 2M, 128k, and 200k tokens, respectively.
We use their official APIs555https://ai.google.dev/gemini-api,666https://platform.openai.com/docs/models/gpt-4o,777https://www.anthropic.com/apifor the evaluation.
Their prompts were chosen based on their performance on the development queries over the 128k token context.
While evaluating, a small number of API calls can be blocked due to various reasons such as errors in the safety filters; these examples are treated as incorrect.
In allLOFTtasks,LCLMs without any task-specific fine-tuning are benchmarked against specialized models that have undergone extensive fine-tuning or pipelining for the target taskand are therefore limited to that specific domain.
We select each specialized model that exemplifies recent task-specific advancements.4.1Text RetrievalFigure 4:Positional Analysis.
We vary gold document positions of queries within the corpus (0% = beginning, 100% = end) .We adopt Gecko, a state-of-the-art dual encoder as the specialized model for the retrieval task.
Gecko is fine-tuned on extensive text retrieval and similarity tasks.
To ensure fair comparison, we use exactly the same corpus to test both the LCLMs and Gecko, instead of using the full retrieval corpora or published results.ResultsTable2demonstrates that Gemini 1.5 Pro performs comparably to Gecko at 128k context length.
Other LCLMs also perform surprisingly well.
This is notable, as LCLMs have not undergone specialized contrastive learning for retrieval. While LCLMs’s performance does degrade when scaling the corpus to millions of tokens (Figure5) , the parity at 128k still suggests the potential of LCLMs to be used for retrieval tasks.Figure 5:Scaling resultsof LCLMs compared to each specialized model by scaling the corpus size from 32k to 1 million tokens.
Results are averaged over all constituent datasets in each task.Positional AnalysisTo better understand the performance of LCLMs on longer context lengths, we investigate how the positioning of the gold document affects retrieval performance, examining the effect of the gold document for the test query and for the few shot examples within the prompt.Figure4reveals that performance drops as the gold documents of the test queries are moved towards the end of the corpus, suggesting reduced attention in later sections of the prompt.
Conversely, placing the gold documents of few-shot queries at the end improves recall, indicating their ability to mitigate attention weaknesses in this region.
Co-locating gold documents of few-shot and test queries consistently boosts performance. This is perhaps unsurprising, as it gives the model information about where to look for the answer; however, it also indicates that the model does indeed pay special attention to the locations where the gold documents for the few-shot examples are placed, regardless of where they are in the corpus. This offers a promising approach to overcome performance degradation in large corpora.
Per-dataset analysis is provided inAppendixE.4.2Visual RetrievalWe employ CLIP-L/14, a widely used text-to-image retrieval model, as our specialized model. For Flickr30k and MS-COCO, CLIP performs text-to-image retrieval. For MSR-VTT, it performs text-to-video retrieval by averaging scores across frames. For OVEN, due to the lack of suitable open-source image-to-text models, we approximate image-to-text retrieval by using CLIP’s text-to-image retrieval.
Evaluation of Claude 3 Opus on this task was not feasible due to the current limitation of 20 images per API request.ResultsGemini 1.5 Pro outperforms GPT-4o across all four visual benchmarks (Table2) .
Notably, as shown in Figure5, Gemini 1.5 Pro maintains a performance advantage over CLIP across all visual benchmarks and context lengths.4.3Audio RetrievalWe choose PaLM 2 DEas a specialized model, which is a dual-encoder trained to maximize the similarity between audio and their transcription and has achieved previous state-of-the-art on the FLEURS datasets.
Currently, GPT-4o and Claude 3 Opus do not support audio input.ResultsGemini 1.5 Pro demonstrates comparable performance to PaLM 2 DE across all 5 languages (Table2) . We notice that Gemini 1.5 Pro notably surpasses PaLM 2 DE in Hindi; this advantage likely stems from differences in pre-training data between Gemini and PaLM.Figure5further confirms Gemini 1.5 Pro’s robust performance across various context length, highlighting the current capabilities of LCLMs while also indicating the need for more challenging audio datasets.4.4RAGWe set up a retrieve-and-read RAG pipeline as a specialized model, using Geckoto retrieve the top-40 documents which are then put into the context of Gemini 1.5 Pro and used to generate the answer conditioned on the question and the retrieved documents.ResultsTable2demonstrates that Gemini 1.5 Pro, with the entire corpus in context, outperforms the RAG pipeline on multi-hop datasets (HotpotQA and MusiQue) . This is because LCLMs can reason over multiple passages in the context window using Chain-of-Thought, a capability that RAG pipelines typically lack unless they have a separate module for planning and reasoning.However, a specialized retriever like Gecko excels at ranking all topically relevant passages from a corpus, enabling it to identify a comprehensive set of passages covering all answers. This proves particularly beneficial for multi-target datasets, such as QUEST and QAMPARI.Figure5reveals that while LCLMs match RAG performance at 128k compared to a pipeline, performance drops at 1M corresponding to the drop found in LCLM text retrieval performance.DatasetDev (32k) Test (128k) NQ0.60(-0.10) 0.37(-0.47) HotPotQA0.60(-0.30) 0.33(-0.42) MuSiQue0.20(-0.60) 0.10(-0.45) Table 3:Gemini’s closed-book performance on RAG.
Red indicates the performance difference compared to the CiC prompting.Closed-Book AblationsTo further probe LCLMs capabilities, we conduct closed-book ablations on Gemini 1.5 Pro. In this setting, we remove the corpus from the context to assess LCLM performance based solely on parametric knowledge.Table3presents the results, revealing that the closed-book performance significantly lags behind the long-context and specialized models.
This underscores the tested models’ effectiveness in leveraging the external corpus to enhance its reasoning capabilities.4.5SQL-Like Compositional ReasoningFigure 6:SQL Reasoning Analysis.
We bin Spider queries by operators in their SQL query and report binned Gemini performance.
We groupminandmaxinto a bin and>and<into another bin.The traditional SQL pipeline uses a trained semantic parser to translate the natural language input into a SQL query.
Then, a separate SQL interpreter is used to execute the SQL query over the database.
As a specialized model, we use DAIL-SQLfor the semantic parser, which prompts an LLM to provide the SQL query.
We adapt DAIL-SQL by replacing its LLM with Gemini 1.5 Pro and using a fixed set of few-shot examples.ResultsResults inTable2show that LCLMs achieve reasonable performance, though they are significantly behind the specialized pipeline. This reveals substantial headroom to enhance the compositional reasoning capabilities of LCLMs.Lower performance than the specialized pipeline on the SQL task is unsurprising, given the decades of work on semantic parsing and SQL.
Nonetheless, this performance still reveals the potential for the model to handle complex structured data with no task-specific tuning.Reasoning AnalysisTo gain insights into the short-comings of LCLMs in complex compositional reasoning, we categorize queries based on the operators in the gold SQL queries and measure Gemini 1.5 Pro’s performance for each operator.Figure6shows that averaging is the most difficult operation while counting is relatively easy.
Moreover, we find that reasoning over equality is considerably easier than reasoning over inequality.4.6Many-Shot ICLFigure 7:ICL Performanceas we scale the percentage of examples used up to 100%.ResultsTable2compares accuracy for each of the LCLMs across all ICL datasets.
For BBH, we report the accuracy on 32k, which is the maximum context length available.
Gemini 1.5 Pro outperforms GPT-4o on all benchmarks, except for BBH-tracking7 where Gemini performs surprisingly poorly.
On average, Claude 3 Opus achieves the best performance among LCLMs on this task.Scaling Many Shot ICLFigure7illustrates the impact of increasing the number of examples in Gemini.
In LIB-dialog, accuracy improves monotonically with more examples.
In contrast, results on BBH are mixed.
Knowledge-intensive tasks like BBH-date and BBH-salient see monotonic improvements similar to LIB-dialog, while reasoning-intensive tasks like BBH-tracking7 and BBH-web do not benefit.
These results suggest that more complicated tasks may see an earlier limit in how much models can learn from scaling the number of in-context examples.
body
Can Long-Context Language Models SubsumeRetrieval, RAG, SQL, and More?
4LOFTTasks and Primary Results
2406.13121
5CiC Prompt AblationsWe conduct ablations over the different facets of the CiC Prompt. Examples of the prompt for each ablation can be found inAppendixF.
For the ablations, we evaluate Gemini 1.5 Pro on the 128k version ofLOFT.Task(Metric) DatasetBestPromptGenericInstructionQuery atBeginningAlphanu-meric IDsTitlesOnlyWithoutID EchoCorpus inEach Few-shotWithoutCoTTextRetrievalArguAna0.840.760.720.81-0.780.620.79FIQA0.790.770.580.75-0.760.780.85NQ1.000.980.980.990.911.001.001.00SciFact0.880.880.810.900.840.870.780.90MuSiQue0.490.440.190.440.100.360.350.43QAMPARI0.610.610.490.540.090.490.350.43QUEST0.280.280.220.300.050.270.220.30RAGMuSiQue0.550.570.390.520.230.550.780.50NQ0.840.810.760.800.390.830.830.82QAMPARI0.440.420.350.420.090.360.360.33QUEST0.280.330.170.280.020.250.270.32Average0.640.620.510.610.300.590.550.61() -(-0.02) (-0.13) (-0.03) (-0.30) (-0.05) (-0.09) (-0.03) Table 4:Ablation results of Gemini 1.5 Pro on different tasks inLOFTat 128k context length.
Starting from our best prompt format (used in the rest of the experiments) , individual facets of the corpus, query, and instruction are ablated to surface their relative effect on quality.
Since ArguAna and FIQA do not have any title for each passage, we report the average and its difference without these two datasets for theTitles Onlyablation.The ablations show the effectiveness of our CiC prompting design.
Removing task-specific instructions (Generic Instruction)  or Chain-of-Thought reasoning (Without CoT)  both lead to worse performance. We also observe a performance decrease forCorpus in Each Few-Shot. In this setting, instead of using a shared corpus, each few-shot example has its own small corpus consisting of nine random passages and one gold passage. This performance degradation could be either because the few-shot examples help the model attend to the test corpus or because the few-shot task becomes much easier than the evaluation task.Placing the query at the beginning of the prompt instead of at the end (Query at Beginning)  led to a significant and consistent performance decrease across tasks and models.
This result suggests that prefix-caching actually works better than encoding the corpus conditioned on each query, which would be much more expensive.Figure 8:Effect of the number of few-shot examples.
The performance increases with the number of few-shot examples.The presentation of document IDs also affects performance. In particular, replacing monotonic numerical IDs with random (Alphanumeric IDs)  negatively impacts performance in most datasets. This could possibly be due to way in which numbers are tokenized, with fewer tokens for certain numbers.
Only placing the IDs at the front of the document instead of at the front and the back (Without ID Echo)  also resulted in a 5% performance drop, confirming that repeating text can compensate for missing context in autoregressive language models.To test if the model is simply using parametric knowledge instead of grounding on the context, we remove the document content and simply keep the document title and ID in the corpus (Title Only) . In this case, it would imply the model is able to perform well because it has already seen all of the datasets we are evaluating on during training. Across all experiments, this ablation significantly degraded performance, indicating the model indeed relies on the provided context.We also study how the number of few-shot examples in the prompt affects quality.
The results can be found inFigure8.
Increasing the number of examples improves the quality on the retrieval task, from 0.76 at zero-shot to 0.81 at 5-shots.
Finally, qualitative analysis on the model outputs is provided inAppendixG.
body
Can Long-Context Language Models SubsumeRetrieval, RAG, SQL, and More?
5CiC Prompt Ablations
2406.13121
6Related WorkEvaluating LCLMs remains a challenge due to the limitations of existing benchmarks.
Many popular datasets and methods rely on synthetic taskssuch as the popular "Needle-in-A-Haystack" retrievalor its extension to multi-hop QA.
While these evaluations are also scalable to arbitrarily long contexts, they do not fully capture the nuances of real-world retrieval or reasoning tasks.
Some recent benchmarks leverage existing NLP datasets for tasks such as extreme summarization and multi-document QA.
However, these tasks lack the dynamic scaling capabilities of synthetic benchmarks, which makes them difficult to adapt to very long contexts.LongAlpacaand LongBench-Chatevaluate instruction-following under long context settings but contain relatively low task diversity and no examples beyond 100k context length. Similar toLOFT, Ada-LEvalproposes a length-adaptable benchmark; however, their tasks are somewhat synthetic and may not resemble real-world applications.Also related to our work is, which applies LCLMs to long-context QA using the top retrieved documents from Natural Questions, similar to specialized models for RAG inLOFT.
They find that LCLMs lose recall when relevant information is placed in the middle of the context.
However, their analysis is limited to contexts that are under 10k tokens.
We extend this type of evaluation of LCLMs to context lengths of up to 1M tokens in addition to using multiple modalities and additional forms of reasoning. Finally, our work connects to the field of generative retrieval, where models are trained to memorize and generate retrieval targets. Our research offers an alternative approach where the retrieval corpus is directly provided as context, eliminating task specific training.
body
Can Long-Context Language Models SubsumeRetrieval, RAG, SQL, and More?
6Related Work
2406.13121
7ConclusionAs language models improve and scale, their ability to retrieve and reason over increasingly long context will unlock unprecedented use-cases.
To measure this progress, we introduceLOFT, the Long Context Frontiers benchmark.LOFTis a suite of tasks that rigorously assesses LCLMs on tasks ripe for a paradigm shift: retrieval, retrieval-augmented generation, SQL-like reasoning, and in-context learning.LOFTprovides dynamic scaling of context lengths of up to 1 million tokens, ensuring that evaluations remain relevant as LCLMs continue to evolve. The tasks can be further scaled up with minimal effort to 1 billion tokens as context lengths continue to increase.
Initial findings demonstrate that despite having never been trained to do retrieval, LCLMs have retrieval capabilities rivaling task-specific hand-crafted SOTA retrieval systems.
Nevertheless, there remains considerable room for advancement in long-context reasoning, particularly as models gain access to even longer context windows.
We believe thatLOFTprovides a fertile testing ground for measuring progress in long-context modeling.LimitationsOur experiments were constrained by the computational resources and financial costs associated with utilizing LCLMs.
The entire LOFT 128k test sets contain aroundinput tokens, which costfor Gemini 1.5 Pro,for GPT-4o, andfor Claude 3 Opus at the time of writing. To reduce costs, we also release dev sets, which are 10x smaller and can be evaluated with aroundusing Gemini 1.5 Pro or GPT-4o. We also expect LLM API prices to decrease over time.
Another limitation of this work is that we focused on evaluating the quality of LCLMs, and leave efficiency considerations for future work. We could not measure the efficiency improvements from prefix cachingdue to API constraints at the time of writing. Without caching, the Gemini 1.5 Pro API has a median latency of roughly four seconds for 32k input tokens, twelve seconds for 128k input tokens, and 100 seconds for 1 million input tokens. This speed is likely slower than specialized retrievers or SQL databases; the promising quality results onLOFTencourage further investigation into optimizing LCLMs efficiency.
Additionally, our retrieval and RAG tasks was limited to 1 million tokens, which still leaves a large gap from real-world applications that may involve several million or even billions of documents.
body
Can Long-Context Language Models SubsumeRetrieval, RAG, SQL, and More?
7Conclusion
2406.13121
AcknowledgementWe are grateful to Frank Palma Gomez for providing the audio retrieval model, and to Raoul de Liedekerke and Miteyan Patel for their invaluable help with the API-based model evaluation. We thank Nicholas Monath, Raphael Hoffmann, Slav Petrov, Urvashi Khandelwal, and Kristina Toutanova for their thoughtful discussions and insightful feedback to the paper.
conclusion
Can Long-Context Language Models SubsumeRetrieval, RAG, SQL, and More?
Acknowledgement