<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2407.18940] Citation Benchmark</title><meta property="og:description" content="Literature search questions, such as “where can I find research on the evaluation of consistency in generated summaries?” pose significant challenges for modern search engines and retrieval systems. These questions oft…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Citation Benchmark">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Citation Benchmark">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2407.18940">

<!--Generated on Mon Aug  5 18:14:51 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document">
<h1 class="ltx_title ltx_title_document">Citation Benchmark</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Anirudh Ajith<sup id="id9.9.id1" class="ltx_sup">1</sup> Mengzhou Xia<sup id="id10.10.id2" class="ltx_sup">1</sup> Alexis Chevalier<sup id="id11.11.id3" class="ltx_sup"><span id="id11.11.id3.1" class="ltx_text ltx_font_italic">1,2</span></sup> Tanya Goyal<sup id="id12.12.id4" class="ltx_sup">1</sup>
<br class="ltx_break"><span id="id5.5.1" class="ltx_text ltx_font_bold">Danqi Chen<sup id="id5.5.1.1" class="ltx_sup"><span id="id5.5.1.1.1" class="ltx_text ltx_font_medium">1</span></sup></span> <span id="id7.7.3" class="ltx_text ltx_font_bold">Tianyu Gao<sup id="id7.7.3.1" class="ltx_sup"><span id="id7.7.3.1.1" class="ltx_text ltx_font_medium">1</span></sup>
<br class="ltx_break"><sup id="id7.7.3.2" class="ltx_sup"><span id="id7.7.3.2.1" class="ltx_text ltx_font_medium">1</span></sup></span>Princeton Language and Intelligence (PLI),
Princeton University <sup id="id13.13.id5" class="ltx_sup">2</sup>BCG X
<br class="ltx_break"><span id="id14.14.id6" class="ltx_text ltx_font_typewriter">{anirudh.ajith,mengzhou,achevalier,tanyagoyal,
<br class="ltx_break">danqic,tianyug}@princeton.edu</span>
</span></span>
</div>

<h1 class="ltx_title ltx_title_document">LitSearch: A Retrieval Benchmark for Scientific Literature Search</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Anirudh Ajith<sup id="id9.9.id1" class="ltx_sup">1</sup> Mengzhou Xia<sup id="id10.10.id2" class="ltx_sup">1</sup> Alexis Chevalier<sup id="id11.11.id3" class="ltx_sup"><span id="id11.11.id3.1" class="ltx_text ltx_font_italic">1,2</span></sup> Tanya Goyal<sup id="id12.12.id4" class="ltx_sup">1</sup>
<br class="ltx_break"><span id="id5.5.1" class="ltx_text ltx_font_bold">Danqi Chen<sup id="id5.5.1.1" class="ltx_sup"><span id="id5.5.1.1.1" class="ltx_text ltx_font_medium">1</span></sup></span> <span id="id7.7.3" class="ltx_text ltx_font_bold">Tianyu Gao<sup id="id7.7.3.1" class="ltx_sup"><span id="id7.7.3.1.1" class="ltx_text ltx_font_medium">1</span></sup>
<br class="ltx_break"><sup id="id7.7.3.2" class="ltx_sup"><span id="id7.7.3.2.1" class="ltx_text ltx_font_medium">1</span></sup></span>Princeton Language and Intelligence (PLI),
Princeton University <sup id="id13.13.id5" class="ltx_sup">2</sup>BCG X
<br class="ltx_break"><span id="id14.14.id6" class="ltx_text ltx_font_typewriter">{anirudh.ajith,mengzhou,achevalier,tanyagoyal,
<br class="ltx_break">danqic,tianyug}@princeton.edu</span>
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id15.id1" class="ltx_p">Literature search questions, such as “where can I find research on the evaluation of consistency in generated summaries?” pose significant challenges for modern search engines and retrieval systems. These questions often require a deep understanding of research concepts and the ability to reason over entire articles. In this work, we introduce LitSearch, a retrieval benchmark comprising 597 realistic literature search queries about recent ML and NLP papers. LitSearch is constructed using a combination of (1) questions generated by GPT-4 based on paragraphs containing inline citations from research papers and
(2) questions about recently published papers, manually written by their authors.
All LitSearch questions were manually examined or edited by experts to ensure high quality. We extensively benchmark state-of-the-art retrieval models and also evaluate two LLM-based reranking pipelines.
We find a significant performance gap between BM25 and state-of-the-art dense retrievers, with a 24.8% difference in absolute recall@5. The LLM-based reranking strategies further improve the best-performing dense retriever by 4.4%. Additionally, commercial search engines and research tools like Google Search perform poorly on LitSearch, lagging behind the best dense retriever by 32 points. Taken together, these results show that LitSearch is an informative new testbed for retrieval systems while catering to a real-world use case.<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>Our dataset and code are available at <a target="_blank" href="https://github.com/princeton-nlp/LitSearch" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/princeton-nlp/LitSearch</a>.</span></span></span></p>
</div>
<div id="p1" class="ltx_para ltx_noindent">
<div id="p1.8" class="ltx_block ltx_align_bottom">
<p id="p1.8.9" class="ltx_p"><span id="p1.8.9.1" class="ltx_text ltx_font_bold">LitSearch: A Retrieval Benchmark for Scientific Literature Search</span></p>
<br class="ltx_break ltx_centering">
<p id="p1.8.8" class="ltx_p ltx_align_center" style="width:433.6pt;"><span id="p1.8.8.8" class="ltx_text ltx_inline-block" style="width:0.0pt;">
<span id="p1.8.8.8.8" class="ltx_tabular ltx_align_top">
<span class="ltx_tbody">
<span id="p1.4.4.4.4.4" class="ltx_tr">
<span id="p1.4.4.4.4.4.4" class="ltx_td ltx_align_center"><span id="p1.4.4.4.4.4.4.4" class="ltx_text ltx_font_bold">Anirudh Ajith<sup id="p1.4.4.4.4.4.4.4.1" class="ltx_sup"><span id="p1.4.4.4.4.4.4.4.1.1" class="ltx_text ltx_font_medium">1</span></sup> Mengzhou Xia<sup id="p1.4.4.4.4.4.4.4.2" class="ltx_sup"><span id="p1.4.4.4.4.4.4.4.2.1" class="ltx_text ltx_font_medium">1</span></sup> Alexis Chevalier<sup id="p1.4.4.4.4.4.4.4.3" class="ltx_sup"><span id="p1.4.4.4.4.4.4.4.3.1" class="ltx_text ltx_font_medium ltx_font_italic">1,2</span></sup> Tanya Goyal<sup id="p1.4.4.4.4.4.4.4.4" class="ltx_sup"><span id="p1.4.4.4.4.4.4.4.4.1" class="ltx_text ltx_font_medium">1</span></sup></span></span></span>
<span id="p1.6.6.6.6.6" class="ltx_tr">
<span id="p1.6.6.6.6.6.2" class="ltx_td ltx_align_center"><span id="p1.5.5.5.5.5.1.1" class="ltx_text ltx_font_bold">Danqi Chen<sup id="p1.5.5.5.5.5.1.1.1" class="ltx_sup"><span id="p1.5.5.5.5.5.1.1.1.1" class="ltx_text ltx_font_medium">1</span></sup></span> <span id="p1.6.6.6.6.6.2.2" class="ltx_text ltx_font_bold">Tianyu Gao<sup id="p1.6.6.6.6.6.2.2.1" class="ltx_sup"><span id="p1.6.6.6.6.6.2.2.1.1" class="ltx_text ltx_font_medium">1</span></sup></span></span></span>
<span id="p1.8.8.8.8.8" class="ltx_tr">
<span id="p1.8.8.8.8.8.2" class="ltx_td ltx_align_center"><sup id="p1.8.8.8.8.8.2.1" class="ltx_sup">1</sup>Princeton Language and Intelligence (PLI),
Princeton University <sup id="p1.8.8.8.8.8.2.2" class="ltx_sup">2</sup>BCG X</span></span>
<span id="p1.8.8.8.8.9.1" class="ltx_tr">
<span id="p1.8.8.8.8.9.1.1" class="ltx_td ltx_align_center"><span id="p1.8.8.8.8.9.1.1.1" class="ltx_text ltx_font_typewriter">{anirudh.ajith,mengzhou,achevalier,tanyagoyal,</span></span></span>
<span id="p1.8.8.8.8.10.2" class="ltx_tr">
<span id="p1.8.8.8.8.10.2.1" class="ltx_td ltx_align_center"><span id="p1.8.8.8.8.10.2.1.1" class="ltx_text ltx_font_typewriter">danqic,tianyug}@princeton.edu</span></span></span>
</span>
</span></span></p>
<br class="ltx_break ltx_centering">
</div>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Finding literature via a specific search query—for example, to collect related work, to check if a method has been proposed before, or to recall a previously-seen paper—is a critical task for researchers.
Developing systems that recommend citations pertinent to such inquiries has the promise to enhance researchers’ productivity and expedite scientific discovery <cite class="ltx_cite ltx_citemacro_cite">Färber and Jatowt (<a href="#bib.bib7" title="" class="ltx_ref">2020</a>)</cite>.
However, this task is inherently challenging as it often requires deep domain expertise and reasoning through lengthy papers.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2407.18940/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_square" width="452" height="498" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>Examples of <span id="S1.F1.3.1" class="ltx_text ltx_font_italic">inline-citation questions</span> and <span id="S1.F1.4.2" class="ltx_text ltx_font_italic">author-written questions</span> from LitSearch. The questions are often challenging and require deep understanding of target papers to answer correctly.
</figcaption>
</figure>
<figure id="S1.F2" class="ltx_figure"><img src="/html/2407.18940/assets/x2.png" id="S1.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="452" height="75" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>The pipeline to generate inline-citation questions. We first sample a citation mention and prompt GPT-4 to write a question. Then we filter the questions by word overlap (with the target paper title) and manual inspection. The manual inspection annotates the specificity and the quality of the questions (rubrics in <a href="#S2.T1" title="In Sampling inline citation mentions. ‣ 2.1 Inline-citation Questions ‣ 2 LitSearch ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">1</span></a>).
</figcaption>
</figure>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">Prior to this study, the task of citation recommendation is
often formalized by
using inline citation mentions from existing papers as queries, and the cited papers as targets <cite class="ltx_cite ltx_citemacro_cite">He et al. (<a href="#bib.bib11" title="" class="ltx_ref">2010</a>); Gu et al. (<a href="#bib.bib9" title="" class="ltx_ref">2022</a>)</cite>.
For instance, given the citation mention “<span id="S1.p2.1.1" class="ltx_text ltx_font_italic">RoBERTa and T5 are based on recent advances in masked language modeling <span id="S1.p2.1.1.1" class="ltx_text ltx_font_typewriter">[citation]</span></span>,” the text surrounding the citation mention is used as a retrieval query, and the cited paper is the target literature.
However, directly using inline citations often leads to queries that are noisy, broad (e.g., “<span id="S1.p2.1.2" class="ltx_text ltx_font_italic">Large Language Models <span id="S1.p2.1.2.1" class="ltx_text ltx_font_typewriter">[citation]</span></span>”), or context-dependent (e.g., “<span id="S1.p2.1.3" class="ltx_text ltx_font_italic">We follow the hyperparameters of <span id="S1.p2.1.3.1" class="ltx_text ltx_font_typewriter">[citation]</span></span>”).</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">In this work, we propose a new litearture retrieval benchmark called LitSearch. As illustrated in <a href="#S1.F1" title="In 1 Introduction ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">1</span></a>, a literature search question seeks papers that meet specific criteria, closely reflecting actual research workflows.
LitSearch consists of two subsets:
(1) For <span id="S1.p3.1.1" class="ltx_text ltx_font_bold">inline-citation questions</span>, we sample citation mentions from a collection of scientific papers and use GPT-4 <cite class="ltx_cite ltx_citemacro_citep">(OpenAI, <a href="#bib.bib27" title="" class="ltx_ref">2023</a>)</cite> to rewrite them into literature search questions (<a href="#S1.F2" title="In 1 Introduction ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">2</span></a>).
We retain questions with a low word overlap with the title of the target papers, and perform manual examination to ensure high quality.
(2) For <span id="S1.p3.1.2" class="ltx_text ltx_font_bold">author-written questions</span>, we invited authors of ACL 2023 and ICLR 2024 papers to write literature search questions for their own papers.
This subset is also manually examined and filtered to remove any inaccurate or easy questions.
LitSearch contains 597 questions in total, each paired with one or more scientific papers as the ground truth.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">LitSearch has several unique characteristics:
(1) To our best knowledge, LitSearch is the <em id="S1.p4.1.1" class="ltx_emph ltx_font_italic">first</em> dataset featuring realistic literature search questions, providing a new testbed for citation recommendation and retrieval systems.
(2) LitSearch is <em id="S1.p4.1.2" class="ltx_emph ltx_font_italic">challenging</em>, requiring deep understanding and reasoning over entire articles. The average document length (6,041/134 words for full text/titles and abstracts) is significantly longer than most existing retrieval benchmarks (e.g., 56 for <cite class="ltx_cite ltx_citemacro_citep">Nguyen et al., <a href="#bib.bib25" title="" class="ltx_ref">2017</a></cite>).
(3) LitSearch is of <em id="S1.p4.1.3" class="ltx_emph ltx_font_italic">high quality</em>, with all questions manually examined by the authors of this work.</p>
</div>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.1" class="ltx_p">We conduct extensive experiments
on both state-of-the-art retrieval models and
reranking with large language models (LLMs).
On LitSearch, the best dense retrieval model, GritLM <cite class="ltx_cite ltx_citemacro_citep">(Muennighoff et al., <a href="#bib.bib23" title="" class="ltx_ref">2024</a>)</cite>, achieves an average recall@5
of 74.8%, beating BM25 <cite class="ltx_cite ltx_citemacro_citep">(Robertson et al., <a href="#bib.bib31" title="" class="ltx_ref">2009</a>)</cite> by 24.8%.
The recall@5 of GritLM is further improved by 4.4% with GPT-4 reranking.
On the other hand, commercial search engines and research tools like Google Search perform poorly on this task, only achieving an average recall@5 of 42.8% at most. Futhermore, compared to existing retrieval datasets from BEIR <cite class="ltx_cite ltx_citemacro_citep">(Thakur et al., <a href="#bib.bib36" title="" class="ltx_ref">2021</a>)</cite> and MTEB <cite class="ltx_cite ltx_citemacro_citep">(Muennighoff et al., <a href="#bib.bib24" title="" class="ltx_ref">2022</a>)</cite>,
LitSearch effectively reflects the performance differences among various embedding models,
making it an informative testbed for evaluating state-of-the-art retrieval systems.</p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>LitSearch</h2>

<div id="S2.p1" class="ltx_para">
<p id="S2.p1.2" class="ltx_p">Our benchmark LitSearch consists of
(a) a large corpus of scientific papers <math id="S2.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{P}" display="inline"><semantics id="S2.p1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S2.p1.1.m1.1.1" xref="S2.p1.1.m1.1.1.cmml">𝒫</mi><annotation-xml encoding="MathML-Content" id="S2.p1.1.m1.1b"><ci id="S2.p1.1.m1.1.1.cmml" xref="S2.p1.1.m1.1.1">𝒫</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.1.m1.1c">\mathcal{P}</annotation></semantics></math> and
(b) pairs of literature search questions and one or more target papers from <math id="S2.p1.2.m2.1" class="ltx_Math" alttext="\mathcal{P}" display="inline"><semantics id="S2.p1.2.m2.1a"><mi class="ltx_font_mathcaligraphic" id="S2.p1.2.m2.1.1" xref="S2.p1.2.m2.1.1.cmml">𝒫</mi><annotation-xml encoding="MathML-Content" id="S2.p1.2.m2.1b"><ci id="S2.p1.2.m2.1.1.cmml" xref="S2.p1.2.m2.1.1">𝒫</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.2.m2.1c">\mathcal{P}</annotation></semantics></math>. Our desiderata are scientific questions that researchers may use while conducting literature surveys.
We use two different strategies to collect such questions: (1) we construct questions using the surrounding context from inline citations in published papers (<a href="#S2.SS1" title="2.1 Inline-citation Questions ‣ 2 LitSearch ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">2.1</span></a>), and (2) we invited the authors of recent conference publications to manually write questions about their own papers (<a href="#S2.SS2" title="2.2 Author-written Questions ‣ 2 LitSearch ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">2.2</span></a>). For both subsets, we ensure high question quality via manual inspection and filtering conducted by the authors of this work (<a href="#S2.SS3" title="2.3 Manual Filtering to Ensure High Quality ‣ 2 LitSearch ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">2.3</span></a>).</p>
</div>
<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>Inline-citation Questions</h3>

<div id="S2.SS1.p1" class="ltx_para">
<p id="S2.SS1.p1.1" class="ltx_p">We define the following concepts for the ease of description:
(a) An <span id="S2.SS1.p1.1.1" class="ltx_text ltx_font_italic">inline citation mention</span> is a paragraph from the main text of a paper that mentions another paper. For example, this paragraph from the RoBERTa paper <cite class="ltx_cite ltx_citemacro_citep">(Liu et al., <a href="#bib.bib19" title="" class="ltx_ref">2019</a>)</cite>, “<span id="S2.SS1.p1.1.2" class="ltx_text ltx_font_italic">… Unlike Devlin et al., (2019), … we do not train with a reduced sequence length for the first 90% of updates …</span>” mentions the BERT paper <cite class="ltx_cite ltx_citemacro_citep">(Devlin et al., <a href="#bib.bib6" title="" class="ltx_ref">2019</a>)</cite>.
(b) The <span id="S2.SS1.p1.1.3" class="ltx_text ltx_font_italic">source paper</span> is the paper the inline citation mention is sampled from.
(c) A <span id="S2.SS1.p1.1.4" class="ltx_text ltx_font_italic">target paper</span> is a paper that is cited by the inline citation mention.</p>
</div>
<div id="S2.SS1.p2" class="ltx_para">
<p id="S2.SS1.p2.1" class="ltx_p"><a href="#S1.F2" title="In 1 Introduction ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">2</span></a> provides an overview of our data collection methodology for inline-citation questions. We utilize the Semantic Scholar Open Research Corpus (S2ORC; <cite class="ltx_cite ltx_citemacro_citep">Lo et al., <a href="#bib.bib20" title="" class="ltx_ref">2020</a></cite>), a large corpus of academic papers obtained from publishers, archives, and the Internet. We randomly sample inline citation mentions
from the S2ORC<span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span>S2ORC provides detailed citation information for most inline citations (including the position of the citation in the paper text and the unique identifier of the target paper), enabling us to easily sample inline citation mentions and match them to S2ORC papers. We used the 2024-03-26 version.</span></span></span> and prompt
GPT-4 to rewrite these citation mentions into literature search questions.
These questions are filtered to remove those with a high word overlap with the title of the target papers, and are further manually examined to ensure high quality.</p>
</div>
<section id="S2.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Sampling inline citation mentions.</h4>

<div id="S2.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="S2.SS1.SSS0.Px1.p1.1" class="ltx_p">We limit the target papers to be only from the ACL Anthology, for the purpose of aligning with the expertise of the manual annotators, i.e., authors of this work. However, we do not limit where the source papers come from. Depending on the source papers, we call
these questions <em id="S2.SS1.SSS0.Px1.p1.1.1" class="ltx_emph ltx_font_italic">ACL sourced</em> or <em id="S2.SS1.SSS0.Px1.p1.1.2" class="ltx_emph ltx_font_italic">non-ACL sourced</em>.</p>
</div>
<figure id="S2.T1" class="ltx_table">
<table id="S2.T1.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S2.T1.1.1.1" class="ltx_tr">
<th id="S2.T1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt" colspan="2"><span id="S2.T1.1.1.1.1.1" class="ltx_text ltx_font_bold">Specificity</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S2.T1.1.2.1" class="ltx_tr">
<th id="S2.T1.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">0</th>
<td id="S2.T1.1.2.1.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S2.T1.1.2.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T1.1.2.1.2.1.1" class="ltx_p" style="width:200.0pt;"><em id="S2.T1.1.2.1.2.1.1.1" class="ltx_emph ltx_font_italic">Broad</em>. There should exist no more than 20 papers that fit the question.</span>
</span>
</td>
</tr>
<tr id="S2.T1.1.3.2" class="ltx_tr">
<th id="S2.T1.1.3.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="S2.T1.1.3.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S2.T1.1.3.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T1.1.3.2.2.1.1" class="ltx_p" style="width:200.0pt;"><span id="S2.T1.1.3.2.2.1.1.1" class="ltx_text" style="color:#0000FF;">Example: What are some parameter-efficient fine-tuning methods?</span></span>
</span>
</td>
</tr>
<tr id="S2.T1.1.4.3" class="ltx_tr">
<th id="S2.T1.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">1</th>
<td id="S2.T1.1.4.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S2.T1.1.4.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T1.1.4.3.2.1.1" class="ltx_p" style="width:200.0pt;"><em id="S2.T1.1.4.3.2.1.1.1" class="ltx_emph ltx_font_italic">Specific</em>. There should exist no more than 5 papers that fit the question.</span>
</span>
</td>
</tr>
<tr id="S2.T1.1.5.4" class="ltx_tr">
<th id="S2.T1.1.5.4.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="S2.T1.1.5.4.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S2.T1.1.5.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T1.1.5.4.2.1.1" class="ltx_p" style="width:200.0pt;"><span id="S2.T1.1.5.4.2.1.1.1" class="ltx_text" style="color:#0000FF;">Example: Which method involves training additional prompt tokens for every layer during the fine-tuning of language models?</span></span>
</span>
</td>
</tr>
<tr id="S2.T1.1.6.5" class="ltx_tr">
<th id="S2.T1.1.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" colspan="2"><span id="S2.T1.1.6.5.1.1" class="ltx_text ltx_font_bold">Quality</span></th>
</tr>
<tr id="S2.T1.1.7.6" class="ltx_tr">
<th id="S2.T1.1.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">0</th>
<td id="S2.T1.1.7.6.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S2.T1.1.7.6.2.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T1.1.7.6.2.1.1" class="ltx_p" style="width:200.0pt;"><em id="S2.T1.1.7.6.2.1.1.1" class="ltx_emph ltx_font_italic">Discarded</em>. The question is factually wrong, unrealistic, overly broad/specific, or too easy.</span>
</span>
</td>
</tr>
<tr id="S2.T1.1.8.7" class="ltx_tr">
<th id="S2.T1.1.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">1</th>
<td id="S2.T1.1.8.7.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S2.T1.1.8.7.2.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T1.1.8.7.2.1.1" class="ltx_p" style="width:200.0pt;"><em id="S2.T1.1.8.7.2.1.1.1" class="ltx_emph ltx_font_italic">Acceptable</em>. The question can be somewhat out of distribution of what researchers ask, or relatively easy due to high overlap with the title/abstract.</span>
</span>
</td>
</tr>
<tr id="S2.T1.1.9.8" class="ltx_tr">
<th id="S2.T1.1.9.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">2</th>
<td id="S2.T1.1.9.8.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="S2.T1.1.9.8.2.1" class="ltx_inline-block ltx_align_top">
<span id="S2.T1.1.9.8.2.1.1" class="ltx_p" style="width:200.0pt;"><em id="S2.T1.1.9.8.2.1.1.1" class="ltx_emph ltx_font_italic">Good</em>. The question makes a challenging yet meaningful literature search question.</span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>Annotation rubrics for the manual filtering (conducted by the authors of LitSearch).</figcaption>
</figure>
</section>
<section id="S2.SS1.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Prompting GPT-4 to generate questions.</h4>

<div id="S2.SS1.SSS0.Px2.p1" class="ltx_para">
<p id="S2.SS1.SSS0.Px2.p1.1" class="ltx_p">Given a sampled citation mention,
we prompt GPT-4 <cite class="ltx_cite ltx_citemacro_citep">(OpenAI, <a href="#bib.bib27" title="" class="ltx_ref">2023</a>)</cite> to generate a literature search question. In the prompt, we provide (1) the sampled paragraph (the inline citation mention) from the source paper and (2) the titles of the cited papers, and instruct GPT-4 to generate a literature search question based on the paragraph that would be answered by one or more of the papers cited in the paragraph.
We use in-context learning <cite class="ltx_cite ltx_citemacro_citep">(Brown et al., <a href="#bib.bib4" title="" class="ltx_ref">2020</a>)</cite> and include two demonstrations.
The prompt we use can be found in
<a href="#A5.T12" title="In Appendix E Prompts and Additional Statistics ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">12</span></a>.</p>
</div>
</section>
<section id="S2.SS1.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Word-overlap filtering.</h4>

<div id="S2.SS1.SSS0.Px3.p1" class="ltx_para">
<p id="S2.SS1.SSS0.Px3.p1.1" class="ltx_p">We notice that inline-citation questions generated in the last step can have very high word overlap with the target paper titles, which makes their retrieval trivial even for BM25 and suggests that the questions may not be of interest to researchers. We calculate the word overlap
as the <span id="S2.SS1.SSS0.Px3.p1.1.1" class="ltx_text ltx_font_italic">percentage of words in the generated question that are also included in the target paper titles</span>.
We filter out ACL sourced questions that have an overlap score higher than 0.3 and
non-ACL sourced questions that have an overlap score higher than 0.1.
In this step, we filter out 5% of the ACL sourced questions and 80% of the non-ACL sourced questions.</p>
</div>
<figure id="S2.T2" class="ltx_table">
<table id="S2.T2.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S2.T2.1.1.1" class="ltx_tr">
<td id="S2.T2.1.1.1.1" class="ltx_td ltx_border_tt" style="padding-left:5.0pt;padding-right:5.0pt;"></td>
<th id="S2.T2.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-left:5.0pt;padding-right:5.0pt;" colspan="4"><span id="S2.T2.1.1.1.2.1" class="ltx_text ltx_font_bold">Broad</span></th>
<th id="S2.T2.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-left:5.0pt;padding-right:5.0pt;" colspan="4"><span id="S2.T2.1.1.1.3.1" class="ltx_text ltx_font_bold">Specific</span></th>
<th id="S2.T2.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-left:5.0pt;padding-right:5.0pt;" rowspan="2"><span id="S2.T2.1.1.1.4.1" class="ltx_text ltx_font_bold">Total #Q</span></th>
</tr>
<tr id="S2.T2.1.2.2" class="ltx_tr">
<td id="S2.T2.1.2.2.1" class="ltx_td" style="padding-left:5.0pt;padding-right:5.0pt;"></td>
<th id="S2.T2.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">#Q</th>
<th id="S2.T2.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Avg. Len</th>
<th id="S2.T2.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Overlap</th>
<th id="S2.T2.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Avg. #P</th>
<th id="S2.T2.1.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">#Q</th>
<th id="S2.T2.1.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Avg. Len</th>
<th id="S2.T2.1.2.2.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Overlap</th>
<th id="S2.T2.1.2.2.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">Avg. #P</th>
</tr>
<tr id="S2.T2.1.3.3" class="ltx_tr">
<td id="S2.T2.1.3.3.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S2.T2.1.3.3.1.1" class="ltx_text ltx_font_bold">Inline-citation Questions</span></td>
<td id="S2.T2.1.3.3.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">120</td>
<td id="S2.T2.1.3.3.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">20.6</td>
<td id="S2.T2.1.3.3.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">0.33</td>
<td id="S2.T2.1.3.3.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">1.21</td>
<td id="S2.T2.1.3.3.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">231</td>
<td id="S2.T2.1.3.3.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">22.1</td>
<td id="S2.T2.1.3.3.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">0.34</td>
<td id="S2.T2.1.3.3.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">1.07</td>
<td id="S2.T2.1.3.3.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:5.0pt;padding-right:5.0pt;">351</td>
</tr>
<tr id="S2.T2.1.4.4" class="ltx_tr">
<td id="S2.T2.1.4.4.1" class="ltx_td ltx_align_left ltx_border_bb" style="padding-left:5.0pt;padding-right:5.0pt;"><span id="S2.T2.1.4.4.1.1" class="ltx_text ltx_font_bold">Author-written Questions</span></td>
<td id="S2.T2.1.4.4.2" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:5.0pt;padding-right:5.0pt;">35</td>
<td id="S2.T2.1.4.4.3" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:5.0pt;padding-right:5.0pt;">15.8</td>
<td id="S2.T2.1.4.4.4" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:5.0pt;padding-right:5.0pt;">0.43</td>
<td id="S2.T2.1.4.4.5" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:5.0pt;padding-right:5.0pt;">1.03</td>
<td id="S2.T2.1.4.4.6" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:5.0pt;padding-right:5.0pt;">211</td>
<td id="S2.T2.1.4.4.7" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:5.0pt;padding-right:5.0pt;">17.9</td>
<td id="S2.T2.1.4.4.8" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:5.0pt;padding-right:5.0pt;">0.43</td>
<td id="S2.T2.1.4.4.9" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:5.0pt;padding-right:5.0pt;">1.00</td>
<td id="S2.T2.1.4.4.10" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:5.0pt;padding-right:5.0pt;">246</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>Statistics for LitSearch. Please refer to <a href="#A5.T14" title="In Appendix E Prompts and Additional Statistics ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">14</span></a> for more detailed statistics of each subset. “#Q”: number of questions. “Overlap”: the fraction of words in the question that are also included in the titles and abstracts of the target papers. “Avg. #P”: average number of target papers.
</figcaption>
</figure>
</section>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>Author-written Questions</h3>

<div id="S2.SS2.p1" class="ltx_para">
<p id="S2.SS2.p1.1" class="ltx_p">Besides generating questions using existing inline citation mentions, we also collect questions directly from human annotators.
As writing literature search questions requires deep understanding of the research field and the target paper,
we invite researchers to write search queries that are answered by their own published papers.
One additional benefit of this setup is that the correctness of the questions is better guaranteed.</p>
</div>
<div id="S2.SS2.p2" class="ltx_para">
<p id="S2.SS2.p2.4" class="ltx_p">We invited authors of ACL 2023 and ICLR 2024 papers to write one literature search question for each of their papers. We chose the two venues as they were among the latest natural language and machine learning conferences at the time of the data collection, hence the papers represent the latest research development and are unlikely to have already been included in the pre-training data of LLMs and retrievers used in our evaluations.
We sent out invitations to <math id="S2.SS2.p2.1.m1.1" class="ltx_Math" alttext="623" display="inline"><semantics id="S2.SS2.p2.1.m1.1a"><mn id="S2.SS2.p2.1.m1.1.1" xref="S2.SS2.p2.1.m1.1.1.cmml">623</mn><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.1.m1.1b"><cn type="integer" id="S2.SS2.p2.1.m1.1.1.cmml" xref="S2.SS2.p2.1.m1.1.1">623</cn></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.1.m1.1c">623</annotation></semantics></math> ACL 2023 authors and <math id="S2.SS2.p2.2.m2.1" class="ltx_Math" alttext="404" display="inline"><semantics id="S2.SS2.p2.2.m2.1a"><mn id="S2.SS2.p2.2.m2.1.1" xref="S2.SS2.p2.2.m2.1.1.cmml">404</mn><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.2.m2.1b"><cn type="integer" id="S2.SS2.p2.2.m2.1.1.cmml" xref="S2.SS2.p2.2.m2.1.1">404</cn></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.2.m2.1c">404</annotation></semantics></math> ICLR 2024 authors, and received <math id="S2.SS2.p2.3.m3.1" class="ltx_Math" alttext="175" display="inline"><semantics id="S2.SS2.p2.3.m3.1a"><mn id="S2.SS2.p2.3.m3.1.1" xref="S2.SS2.p2.3.m3.1.1.cmml">175</mn><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.3.m3.1b"><cn type="integer" id="S2.SS2.p2.3.m3.1.1.cmml" xref="S2.SS2.p2.3.m3.1.1">175</cn></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.3.m3.1c">175</annotation></semantics></math> questions from ACL 2023 authors and <math id="S2.SS2.p2.4.m4.1" class="ltx_Math" alttext="117" display="inline"><semantics id="S2.SS2.p2.4.m4.1a"><mn id="S2.SS2.p2.4.m4.1.1" xref="S2.SS2.p2.4.m4.1.1.cmml">117</mn><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.4.m4.1b"><cn type="integer" id="S2.SS2.p2.4.m4.1.1.cmml" xref="S2.SS2.p2.4.m4.1.1">117</cn></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.4.m4.1c">117</annotation></semantics></math> questions from ICLR 2024 authors.</p>
</div>
</section>
<section id="S2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.3 </span>Manual Filtering to Ensure High Quality</h3>

<div id="S2.SS3.p1" class="ltx_para">
<p id="S2.SS3.p1.1" class="ltx_p">Finally, the authors of this work manually examine every question from both the inline-citation and author-written subsets and annotate these for <span id="S2.SS3.p1.1.1" class="ltx_text ltx_font_italic">specificity</span> and <span id="S2.SS3.p1.1.2" class="ltx_text ltx_font_italic">quality</span> (guidelines in <a href="#S2.T1" title="In Sampling inline citation mentions. ‣ 2.1 Inline-citation Questions ‣ 2 LitSearch ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">1</span></a>).
Questions that are too general (there are more than 20 papers from the corpus can fit the question) are assigned a quality score of 0 and are excluded. We include only questions with a quality score of 1 or 2 in the final dataset.
We also rewrite questions if they have minor mistakes and can be fixed easily.
Each question is assigned to one author for annotation.</p>
</div>
<div id="S2.SS3.p2" class="ltx_para">
<p id="S2.SS3.p2.1" class="ltx_p">As the examples in <a href="#S2.T1" title="In Sampling inline citation mentions. ‣ 2.1 Inline-citation Questions ‣ 2 LitSearch ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">1</span></a> show,
questions of both specificity types can be realistic and valuable, but they exhibit distinct traits.
We use the specificity scores to distinguish <em id="S2.SS3.p2.1.1" class="ltx_emph ltx_font_italic">broad</em> and <em id="S2.SS3.p2.1.2" class="ltx_emph ltx_font_italic">specific</em> questions in the evaluation.</p>
</div>
<div id="S2.SS3.p3" class="ltx_para">
<p id="S2.SS3.p3.1" class="ltx_p">For the inline-citation subset,
we manually examined 382 ACL sourced questions and 450 non-ACL sourced questions.
26% (98 instances) of the ACL questions and
56% (253 instances) of the non-ACL questions are kept. For the author-written subset, since all questions are written by experts, we avoid rewriting them as much as possible. In the end, we kept 89% (155) questions from ACL 2023 authors and 78% (91) questions from ICLR 2024 authors.</p>
</div>
</section>
<section id="S2.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.4 </span>Dataset Statistics</h3>

<div id="S2.SS4.p1" class="ltx_para">
<p id="S2.SS4.p1.1" class="ltx_p">Our final dataset contains 597 questions, with 351 in the inline-citation subset and 246 in the author-written subset. Dataset statistics, including the number of questions, the average question length, and the average word overlap between the question and the target papers (titles and abstracts), are presented in <a href="#S2.T2" title="In Word-overlap filtering. ‣ 2.1 Inline-citation Questions ‣ 2 LitSearch ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">2</span></a>.
We find that author-written
questions are shorter and have a higher word overlap rate with the target papers (0.43 vs. 0.33 for inline-citation questions).
This is expected: when writing questions for their own papers,
authors tend to re-use terminology from their papers and focus on the main findings which are usually included in the abstracts or titles.
In contrast, inline-citation questions can be anchored to any span of the reference documents, irrespective of the main findings or the main focuses of the target papers.</p>
</div>
</section>
<section id="S2.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.5 </span>The Retrieval Corpus</h3>

<div id="S2.SS5.p1" class="ltx_para">
<p id="S2.SS5.p1.5" class="ltx_p">The LitSearch retrieval corpus <math id="S2.SS5.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{P}" display="inline"><semantics id="S2.SS5.p1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S2.SS5.p1.1.m1.1.1" xref="S2.SS5.p1.1.m1.1.1.cmml">𝒫</mi><annotation-xml encoding="MathML-Content" id="S2.SS5.p1.1.m1.1b"><ci id="S2.SS5.p1.1.m1.1.1.cmml" xref="S2.SS5.p1.1.m1.1.1">𝒫</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p1.1.m1.1c">\mathcal{P}</annotation></semantics></math> consists of ACL Anthology and ICLR papers extracted from
S2ORC (see <a href="#A3" title="Appendix C Retrieval Corpus ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Appendix</span> <span class="ltx_text ltx_ref_tag">C</span></a> for details).
We do not use the full S2ORC corpus for efficiency reasons.
In total, this yields <math id="S2.SS5.p1.2.m2.2" class="ltx_Math" alttext="64,183" display="inline"><semantics id="S2.SS5.p1.2.m2.2a"><mrow id="S2.SS5.p1.2.m2.2.3.2" xref="S2.SS5.p1.2.m2.2.3.1.cmml"><mn id="S2.SS5.p1.2.m2.1.1" xref="S2.SS5.p1.2.m2.1.1.cmml">64</mn><mo id="S2.SS5.p1.2.m2.2.3.2.1" xref="S2.SS5.p1.2.m2.2.3.1.cmml">,</mo><mn id="S2.SS5.p1.2.m2.2.2" xref="S2.SS5.p1.2.m2.2.2.cmml">183</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.SS5.p1.2.m2.2b"><list id="S2.SS5.p1.2.m2.2.3.1.cmml" xref="S2.SS5.p1.2.m2.2.3.2"><cn type="integer" id="S2.SS5.p1.2.m2.1.1.cmml" xref="S2.SS5.p1.2.m2.1.1">64</cn><cn type="integer" id="S2.SS5.p1.2.m2.2.2.cmml" xref="S2.SS5.p1.2.m2.2.2">183</cn></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p1.2.m2.2c">64,183</annotation></semantics></math> papers (<math id="S2.SS5.p1.3.m3.2" class="ltx_Math" alttext="59,383" display="inline"><semantics id="S2.SS5.p1.3.m3.2a"><mrow id="S2.SS5.p1.3.m3.2.3.2" xref="S2.SS5.p1.3.m3.2.3.1.cmml"><mn id="S2.SS5.p1.3.m3.1.1" xref="S2.SS5.p1.3.m3.1.1.cmml">59</mn><mo id="S2.SS5.p1.3.m3.2.3.2.1" xref="S2.SS5.p1.3.m3.2.3.1.cmml">,</mo><mn id="S2.SS5.p1.3.m3.2.2" xref="S2.SS5.p1.3.m3.2.2.cmml">383</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.SS5.p1.3.m3.2b"><list id="S2.SS5.p1.3.m3.2.3.1.cmml" xref="S2.SS5.p1.3.m3.2.3.2"><cn type="integer" id="S2.SS5.p1.3.m3.1.1.cmml" xref="S2.SS5.p1.3.m3.1.1">59</cn><cn type="integer" id="S2.SS5.p1.3.m3.2.2.cmml" xref="S2.SS5.p1.3.m3.2.2">383</cn></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p1.3.m3.2c">59,383</annotation></semantics></math> ACL Anthology papers and <math id="S2.SS5.p1.4.m4.2" class="ltx_Math" alttext="4,807" display="inline"><semantics id="S2.SS5.p1.4.m4.2a"><mrow id="S2.SS5.p1.4.m4.2.3.2" xref="S2.SS5.p1.4.m4.2.3.1.cmml"><mn id="S2.SS5.p1.4.m4.1.1" xref="S2.SS5.p1.4.m4.1.1.cmml">4</mn><mo id="S2.SS5.p1.4.m4.2.3.2.1" xref="S2.SS5.p1.4.m4.2.3.1.cmml">,</mo><mn id="S2.SS5.p1.4.m4.2.2" xref="S2.SS5.p1.4.m4.2.2.cmml">807</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.SS5.p1.4.m4.2b"><list id="S2.SS5.p1.4.m4.2.3.1.cmml" xref="S2.SS5.p1.4.m4.2.3.2"><cn type="integer" id="S2.SS5.p1.4.m4.1.1.cmml" xref="S2.SS5.p1.4.m4.1.1">4</cn><cn type="integer" id="S2.SS5.p1.4.m4.2.2.cmml" xref="S2.SS5.p1.4.m4.2.2">807</cn></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p1.4.m4.2c">4,807</annotation></semantics></math> ICLR papers)<span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span><math id="footnote3.m1.1" class="ltx_Math" alttext="7" display="inline"><semantics id="footnote3.m1.1b"><mn id="footnote3.m1.1.1" xref="footnote3.m1.1.1.cmml">7</mn><annotation-xml encoding="MathML-Content" id="footnote3.m1.1c"><cn type="integer" id="footnote3.m1.1.1.cmml" xref="footnote3.m1.1.1">7</cn></annotation-xml><annotation encoding="application/x-tex" id="footnote3.m1.1d">7</annotation></semantics></math> papers are common to both subsets.</span></span></span>.
The average number of words for the documents in <math id="S2.SS5.p1.5.m5.1" class="ltx_Math" alttext="\mathcal{P}" display="inline"><semantics id="S2.SS5.p1.5.m5.1a"><mi class="ltx_font_mathcaligraphic" id="S2.SS5.p1.5.m5.1.1" xref="S2.SS5.p1.5.m5.1.1.cmml">𝒫</mi><annotation-xml encoding="MathML-Content" id="S2.SS5.p1.5.m5.1b"><ci id="S2.SS5.p1.5.m5.1.1.cmml" xref="S2.SS5.p1.5.m5.1.1">𝒫</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p1.5.m5.1c">\mathcal{P}</annotation></semantics></math> is 134 / 6,041 (titles and abstracts / full texts).</p>
</div>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Experiments</h2>

<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Experimental Setup</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.1" class="ltx_p">We compare the performance of different retrieval systems (enumerated below) on our LitSearch benchmark. Due to the limited context sizes of existing embedding models, we only use the paper titles and abstracts to embed the papers in our retrieval corpus <math id="S3.SS1.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{P}" display="inline"><semantics id="S3.SS1.p1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p1.1.m1.1.1" xref="S3.SS1.p1.1.m1.1.1.cmml">𝒫</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.1.m1.1b"><ci id="S3.SS1.p1.1.m1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1">𝒫</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.1.m1.1c">\mathcal{P}</annotation></semantics></math> by default.</p>
</div>
<div id="S3.SS1.p2" class="ltx_para">
<p id="S3.SS1.p2.2" class="ltx_p">For all systems we compare, we report the recall@K for both the broad and specific subsets of LitSearch. We report results for <math id="S3.SS1.p2.1.m1.2" class="ltx_Math" alttext="K=5,20" display="inline"><semantics id="S3.SS1.p2.1.m1.2a"><mrow id="S3.SS1.p2.1.m1.2.3" xref="S3.SS1.p2.1.m1.2.3.cmml"><mi id="S3.SS1.p2.1.m1.2.3.2" xref="S3.SS1.p2.1.m1.2.3.2.cmml">K</mi><mo id="S3.SS1.p2.1.m1.2.3.1" xref="S3.SS1.p2.1.m1.2.3.1.cmml">=</mo><mrow id="S3.SS1.p2.1.m1.2.3.3.2" xref="S3.SS1.p2.1.m1.2.3.3.1.cmml"><mn id="S3.SS1.p2.1.m1.1.1" xref="S3.SS1.p2.1.m1.1.1.cmml">5</mn><mo id="S3.SS1.p2.1.m1.2.3.3.2.1" xref="S3.SS1.p2.1.m1.2.3.3.1.cmml">,</mo><mn id="S3.SS1.p2.1.m1.2.2" xref="S3.SS1.p2.1.m1.2.2.cmml">20</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.1.m1.2b"><apply id="S3.SS1.p2.1.m1.2.3.cmml" xref="S3.SS1.p2.1.m1.2.3"><eq id="S3.SS1.p2.1.m1.2.3.1.cmml" xref="S3.SS1.p2.1.m1.2.3.1"></eq><ci id="S3.SS1.p2.1.m1.2.3.2.cmml" xref="S3.SS1.p2.1.m1.2.3.2">𝐾</ci><list id="S3.SS1.p2.1.m1.2.3.3.1.cmml" xref="S3.SS1.p2.1.m1.2.3.3.2"><cn type="integer" id="S3.SS1.p2.1.m1.1.1.cmml" xref="S3.SS1.p2.1.m1.1.1">5</cn><cn type="integer" id="S3.SS1.p2.1.m1.2.2.cmml" xref="S3.SS1.p2.1.m1.2.2">20</cn></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.1.m1.2c">K=5,20</annotation></semantics></math> for the specific subset and <math id="S3.SS1.p2.2.m2.1" class="ltx_Math" alttext="K=20" display="inline"><semantics id="S3.SS1.p2.2.m2.1a"><mrow id="S3.SS1.p2.2.m2.1.1" xref="S3.SS1.p2.2.m2.1.1.cmml"><mi id="S3.SS1.p2.2.m2.1.1.2" xref="S3.SS1.p2.2.m2.1.1.2.cmml">K</mi><mo id="S3.SS1.p2.2.m2.1.1.1" xref="S3.SS1.p2.2.m2.1.1.1.cmml">=</mo><mn id="S3.SS1.p2.2.m2.1.1.3" xref="S3.SS1.p2.2.m2.1.1.3.cmml">20</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.2.m2.1b"><apply id="S3.SS1.p2.2.m2.1.1.cmml" xref="S3.SS1.p2.2.m2.1.1"><eq id="S3.SS1.p2.2.m2.1.1.1.cmml" xref="S3.SS1.p2.2.m2.1.1.1"></eq><ci id="S3.SS1.p2.2.m2.1.1.2.cmml" xref="S3.SS1.p2.2.m2.1.1.2">𝐾</ci><cn type="integer" id="S3.SS1.p2.2.m2.1.1.3.cmml" xref="S3.SS1.p2.2.m2.1.1.3">20</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.2.m2.1c">K=20</annotation></semantics></math> for the broad subset; these values (5 and 20) correspond to the guidelines followed by the authors while determining the specificity of a given question (see Table <a href="#S2.T1" title="Table 1 ‣ Sampling inline citation mentions. ‣ 2.1 Inline-citation Questions ‣ 2 LitSearch ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>).</p>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Baselines</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.1" class="ltx_p">We benchmark both retrieval models and LLM-based rerankers in this work.</p>
</div>
<section id="S3.SS2.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Retriever models.</h4>

<div id="S3.SS2.SSS0.Px1.p1" class="ltx_para">
<p id="S3.SS2.SSS0.Px1.p1.1" class="ltx_p">We evaluate using the classic BM25 algorithm <cite class="ltx_cite ltx_citemacro_citep">(Robertson et al., <a href="#bib.bib31" title="" class="ltx_ref">2009</a>)</cite>, as well as several state-of-the-art dense retrieval (embedding) models,
including GTR <cite class="ltx_cite ltx_citemacro_citep">(Ni et al., <a href="#bib.bib26" title="" class="ltx_ref">2022</a>)</cite>,
Instructor <cite class="ltx_cite ltx_citemacro_citep">(Su et al., <a href="#bib.bib32" title="" class="ltx_ref">2023</a>)</cite>,
E5 <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib39" title="" class="ltx_ref">2022</a>)</cite>,
and GritLM <cite class="ltx_cite ltx_citemacro_citep">(Muennighoff et al., <a href="#bib.bib23" title="" class="ltx_ref">2024</a>)</cite>.<span id="footnote4" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">4</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">4</sup><span class="ltx_tag ltx_tag_note">4</span>We use the following corresponding checkpoints from <a target="_blank" href="https://huggingface.co/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/</a>:
GTR-T5-large, Instructor-XL, E5-large-v2, and GritLM-7B.</span></span></span>
More details are provided in <a href="#A4" title="Appendix D Retriever details ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Appendix</span> <span class="ltx_text ltx_ref_tag">D</span></a>.</p>
</div>
<figure id="S3.T3" class="ltx_table">
<table id="S3.T3.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S3.T3.1.1.1" class="ltx_tr">
<th id="S3.T3.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<td id="S3.T3.1.1.1.2" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="S3.T3.1.1.1.2.1" class="ltx_text ltx_font_bold">Inline-citation</span></td>
<td id="S3.T3.1.1.1.3" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="S3.T3.1.1.1.3.1" class="ltx_text ltx_font_bold">Author-written</span></td>
<td id="S3.T3.1.1.1.4" class="ltx_td ltx_align_center ltx_border_tt"><span id="S3.T3.1.1.1.4.1" class="ltx_text ltx_font_bold">Avg.</span></td>
<td id="S3.T3.1.1.1.5" class="ltx_td ltx_align_center ltx_border_tt"><span id="S3.T3.1.1.1.5.1" class="ltx_text ltx_font_bold">Avg.</span></td>
</tr>
<tr id="S3.T3.1.2.2" class="ltx_tr">
<th id="S3.T3.1.2.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="S3.T3.1.2.2.2" class="ltx_td ltx_align_center"><span id="S3.T3.1.2.2.2.1" class="ltx_text ltx_font_bold">Broad</span></td>
<td id="S3.T3.1.2.2.3" class="ltx_td ltx_align_center" colspan="2"><span id="S3.T3.1.2.2.3.1" class="ltx_text ltx_font_bold">Specific</span></td>
<td id="S3.T3.1.2.2.4" class="ltx_td ltx_align_center"><span id="S3.T3.1.2.2.4.1" class="ltx_text ltx_font_bold">Broad</span></td>
<td id="S3.T3.1.2.2.5" class="ltx_td ltx_align_center" colspan="2"><span id="S3.T3.1.2.2.5.1" class="ltx_text ltx_font_bold">Specific</span></td>
<td id="S3.T3.1.2.2.6" class="ltx_td ltx_align_center"><span id="S3.T3.1.2.2.6.1" class="ltx_text ltx_font_bold">Broad</span></td>
<td id="S3.T3.1.2.2.7" class="ltx_td ltx_align_center"><span id="S3.T3.1.2.2.7.1" class="ltx_text ltx_font_bold">Specific</span></td>
</tr>
<tr id="S3.T3.1.3.3" class="ltx_tr">
<th id="S3.T3.1.3.3.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="S3.T3.1.3.3.2" class="ltx_td ltx_align_center ltx_border_t">R@20</td>
<td id="S3.T3.1.3.3.3" class="ltx_td ltx_align_center ltx_border_t">R@5</td>
<td id="S3.T3.1.3.3.4" class="ltx_td ltx_align_center ltx_border_t">R@20</td>
<td id="S3.T3.1.3.3.5" class="ltx_td ltx_align_center ltx_border_t">R@20</td>
<td id="S3.T3.1.3.3.6" class="ltx_td ltx_align_center ltx_border_t">R@5</td>
<td id="S3.T3.1.3.3.7" class="ltx_td ltx_align_center ltx_border_t">R@20</td>
<td id="S3.T3.1.3.3.8" class="ltx_td ltx_align_center ltx_border_t">R@20</td>
<td id="S3.T3.1.3.3.9" class="ltx_td ltx_align_center ltx_border_t">R@5</td>
</tr>
<tr id="S3.T3.1.4.4" class="ltx_tr">
<th id="S3.T3.1.4.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">BM25</th>
<td id="S3.T3.1.4.4.2" class="ltx_td ltx_align_center ltx_border_t">37.4</td>
<td id="S3.T3.1.4.4.3" class="ltx_td ltx_align_center ltx_border_t">38.5</td>
<td id="S3.T3.1.4.4.4" class="ltx_td ltx_align_center ltx_border_t">55.8</td>
<td id="S3.T3.1.4.4.5" class="ltx_td ltx_align_center ltx_border_t">48.6</td>
<td id="S3.T3.1.4.4.6" class="ltx_td ltx_align_center ltx_border_t">62.6</td>
<td id="S3.T3.1.4.4.7" class="ltx_td ltx_align_center ltx_border_t">73.5</td>
<td id="S3.T3.1.4.4.8" class="ltx_td ltx_align_center ltx_border_t">39.9</td>
<td id="S3.T3.1.4.4.9" class="ltx_td ltx_align_center ltx_border_t">50.0</td>
</tr>
<tr id="S3.T3.1.5.5" class="ltx_tr">
<th id="S3.T3.1.5.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">GTR-T5-large</th>
<td id="S3.T3.1.5.5.2" class="ltx_td ltx_align_center">45.7</td>
<td id="S3.T3.1.5.5.3" class="ltx_td ltx_align_center">38.5</td>
<td id="S3.T3.1.5.5.4" class="ltx_td ltx_align_center">51.5</td>
<td id="S3.T3.1.5.5.5" class="ltx_td ltx_align_center">37.1</td>
<td id="S3.T3.1.5.5.6" class="ltx_td ltx_align_center">40.8</td>
<td id="S3.T3.1.5.5.7" class="ltx_td ltx_align_center">55.9</td>
<td id="S3.T3.1.5.5.8" class="ltx_td ltx_align_center">43.8</td>
<td id="S3.T3.1.5.5.9" class="ltx_td ltx_align_center">39.6</td>
</tr>
<tr id="S3.T3.1.6.6" class="ltx_tr">
<th id="S3.T3.1.6.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Instructor-XL</th>
<td id="S3.T3.1.6.6.2" class="ltx_td ltx_align_center">56.3</td>
<td id="S3.T3.1.6.6.3" class="ltx_td ltx_align_center">48.9</td>
<td id="S3.T3.1.6.6.4" class="ltx_td ltx_align_center">60.0</td>
<td id="S3.T3.1.6.6.5" class="ltx_td ltx_align_center">57.1</td>
<td id="S3.T3.1.6.6.6" class="ltx_td ltx_align_center">55.9</td>
<td id="S3.T3.1.6.6.7" class="ltx_td ltx_align_center">70.1</td>
<td id="S3.T3.1.6.6.8" class="ltx_td ltx_align_center">56.5</td>
<td id="S3.T3.1.6.6.9" class="ltx_td ltx_align_center">52.3</td>
</tr>
<tr id="S3.T3.1.7.7" class="ltx_tr">
<th id="S3.T3.1.7.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">E5-large-v2</th>
<td id="S3.T3.1.7.7.2" class="ltx_td ltx_align_center">55.8</td>
<td id="S3.T3.1.7.7.3" class="ltx_td ltx_align_center">50.4</td>
<td id="S3.T3.1.7.7.4" class="ltx_td ltx_align_center">63.9</td>
<td id="S3.T3.1.7.7.5" class="ltx_td ltx_align_center">54.3</td>
<td id="S3.T3.1.7.7.6" class="ltx_td ltx_align_center">62.6</td>
<td id="S3.T3.1.7.7.7" class="ltx_td ltx_align_center">75.8</td>
<td id="S3.T3.1.7.7.8" class="ltx_td ltx_align_center">55.4</td>
<td id="S3.T3.1.7.7.9" class="ltx_td ltx_align_center">56.2</td>
</tr>
<tr id="S3.T3.1.8.8" class="ltx_tr">
<th id="S3.T3.1.8.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">GritLM-7B</th>
<td id="S3.T3.1.8.8.2" class="ltx_td ltx_align_center"><span id="S3.T3.1.8.8.2.1" class="ltx_text ltx_font_bold">69.7</span></td>
<td id="S3.T3.1.8.8.3" class="ltx_td ltx_align_center"><span id="S3.T3.1.8.8.3.1" class="ltx_text ltx_font_bold">67.7</span></td>
<td id="S3.T3.1.8.8.4" class="ltx_td ltx_align_center"><span id="S3.T3.1.8.8.4.1" class="ltx_text ltx_font_bold">77.9</span></td>
<td id="S3.T3.1.8.8.5" class="ltx_td ltx_align_center"><span id="S3.T3.1.8.8.5.1" class="ltx_text ltx_font_bold">74.3</span></td>
<td id="S3.T3.1.8.8.6" class="ltx_td ltx_align_center"><span id="S3.T3.1.8.8.6.1" class="ltx_text ltx_font_bold">82.5</span></td>
<td id="S3.T3.1.8.8.7" class="ltx_td ltx_align_center"><span id="S3.T3.1.8.8.7.1" class="ltx_text ltx_font_bold">89.1</span></td>
<td id="S3.T3.1.8.8.8" class="ltx_td ltx_align_center"><span id="S3.T3.1.8.8.8.1" class="ltx_text ltx_font_bold">70.8</span></td>
<td id="S3.T3.1.8.8.9" class="ltx_td ltx_align_center"><span id="S3.T3.1.8.8.9.1" class="ltx_text ltx_font_bold">74.8</span></td>
</tr>
<tr id="S3.T3.1.9.9" class="ltx_tr">
<th id="S3.T3.1.9.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">GPT-4 reranking (w/ BM25)</th>
<td id="S3.T3.1.9.9.2" class="ltx_td ltx_align_center ltx_border_t">54.9</td>
<td id="S3.T3.1.9.9.3" class="ltx_td ltx_align_center ltx_border_t">60.0</td>
<td id="S3.T3.1.9.9.4" class="ltx_td ltx_align_center ltx_border_t">67.5</td>
<td id="S3.T3.1.9.9.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T3.1.9.9.5.1" class="ltx_text ltx_font_bold">77.1</span></td>
<td id="S3.T3.1.9.9.6" class="ltx_td ltx_align_center ltx_border_t">76.8</td>
<td id="S3.T3.1.9.9.7" class="ltx_td ltx_align_center ltx_border_t">82.9</td>
<td id="S3.T3.1.9.9.8" class="ltx_td ltx_align_center ltx_border_t">59.9</td>
<td id="S3.T3.1.9.9.9" class="ltx_td ltx_align_center ltx_border_t">68.0</td>
</tr>
<tr id="S3.T3.1.10.10" class="ltx_tr">
<th id="S3.T3.1.10.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">GPT-4 one-hop (w/ BM25)</th>
<td id="S3.T3.1.10.10.2" class="ltx_td ltx_align_center">62.0</td>
<td id="S3.T3.1.10.10.3" class="ltx_td ltx_align_center">64.1</td>
<td id="S3.T3.1.10.10.4" class="ltx_td ltx_align_center">71.6</td>
<td id="S3.T3.1.10.10.5" class="ltx_td ltx_align_center">74.3</td>
<td id="S3.T3.1.10.10.6" class="ltx_td ltx_align_center">73.5</td>
<td id="S3.T3.1.10.10.7" class="ltx_td ltx_align_center">77.7</td>
<td id="S3.T3.1.10.10.8" class="ltx_td ltx_align_center">64.8</td>
<td id="S3.T3.1.10.10.9" class="ltx_td ltx_align_center">68.6</td>
</tr>
<tr id="S3.T3.1.11.11" class="ltx_tr">
<th id="S3.T3.1.11.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">GPT-4 reranking (w/ GritLM)</th>
<td id="S3.T3.1.11.11.2" class="ltx_td ltx_align_center"><span id="S3.T3.1.11.11.2.1" class="ltx_text ltx_font_bold">74.7</span></td>
<td id="S3.T3.1.11.11.3" class="ltx_td ltx_align_center"><span id="S3.T3.1.11.11.3.1" class="ltx_text ltx_font_bold">73.2</span></td>
<td id="S3.T3.1.11.11.4" class="ltx_td ltx_align_center"><span id="S3.T3.1.11.11.4.1" class="ltx_text ltx_font_bold">79.9</span></td>
<td id="S3.T3.1.11.11.5" class="ltx_td ltx_align_center"><span id="S3.T3.1.11.11.5.1" class="ltx_text ltx_font_bold">77.1</span></td>
<td id="S3.T3.1.11.11.6" class="ltx_td ltx_align_center"><span id="S3.T3.1.11.11.6.1" class="ltx_text ltx_font_bold">85.8</span></td>
<td id="S3.T3.1.11.11.7" class="ltx_td ltx_align_center"><span id="S3.T3.1.11.11.7.1" class="ltx_text ltx_font_bold">92.4</span></td>
<td id="S3.T3.1.11.11.8" class="ltx_td ltx_align_center"><span id="S3.T3.1.11.11.8.1" class="ltx_text ltx_font_bold">75.3</span></td>
<td id="S3.T3.1.11.11.9" class="ltx_td ltx_align_center"><span id="S3.T3.1.11.11.9.1" class="ltx_text ltx_font_bold">79.2</span></td>
</tr>
<tr id="S3.T3.1.12.12" class="ltx_tr">
<th id="S3.T3.1.12.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">GPT-4 one-hop (w/ GritLM)</th>
<td id="S3.T3.1.12.12.2" class="ltx_td ltx_align_center ltx_border_bb">72.9</td>
<td id="S3.T3.1.12.12.3" class="ltx_td ltx_align_center ltx_border_bb">70.3</td>
<td id="S3.T3.1.12.12.4" class="ltx_td ltx_align_center ltx_border_bb">78.4</td>
<td id="S3.T3.1.12.12.5" class="ltx_td ltx_align_center ltx_border_bb">74.3</td>
<td id="S3.T3.1.12.12.6" class="ltx_td ltx_align_center ltx_border_bb">84.4</td>
<td id="S3.T3.1.12.12.7" class="ltx_td ltx_align_center ltx_border_bb">87.2</td>
<td id="S3.T3.1.12.12.8" class="ltx_td ltx_align_center ltx_border_bb">73.2</td>
<td id="S3.T3.1.12.12.9" class="ltx_td ltx_align_center ltx_border_bb">77.0</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 3: </span>
Main experiment results of LitSearch.
Here we only use the titles and abstracts of papers for retrieval and reranking.
We report recall@20 (R@20) for broad questions and
recall@5,20 (R@5, R@20) for specific questions.
“Broad” and “specific” correspond to the annotations during our manual filtering stage (defined in <a href="#S2.T1" title="In Sampling inline citation mentions. ‣ 2.1 Inline-citation Questions ‣ 2 LitSearch ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">1</span></a>).
</figcaption>
</figure>
<figure id="S3.F3" class="ltx_figure"><img src="/html/2407.18940/assets/x3.png" id="S3.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="452" height="128" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>We demonstrate detailed retrieval results using BM25, E5 and GritLM up to <math id="S3.F3.2.m1.1" class="ltx_Math" alttext="k=50" display="inline"><semantics id="S3.F3.2.m1.1b"><mrow id="S3.F3.2.m1.1.1" xref="S3.F3.2.m1.1.1.cmml"><mi id="S3.F3.2.m1.1.1.2" xref="S3.F3.2.m1.1.1.2.cmml">k</mi><mo id="S3.F3.2.m1.1.1.1" xref="S3.F3.2.m1.1.1.1.cmml">=</mo><mn id="S3.F3.2.m1.1.1.3" xref="S3.F3.2.m1.1.1.3.cmml">50</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.F3.2.m1.1c"><apply id="S3.F3.2.m1.1.1.cmml" xref="S3.F3.2.m1.1.1"><eq id="S3.F3.2.m1.1.1.1.cmml" xref="S3.F3.2.m1.1.1.1"></eq><ci id="S3.F3.2.m1.1.1.2.cmml" xref="S3.F3.2.m1.1.1.2">𝑘</ci><cn type="integer" id="S3.F3.2.m1.1.1.3.cmml" xref="S3.F3.2.m1.1.1.3">50</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.2.m1.1d">k=50</annotation></semantics></math>. Additionally, we show the effect of applying GPT-4 reranking over GritLM retrieval results.
</figcaption>
</figure>
</section>
<section id="S3.SS2.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">LLM-based reranking.</h4>

<div id="S3.SS2.SSS0.Px2.p1" class="ltx_para">
<p id="S3.SS2.SSS0.Px2.p1.1" class="ltx_p">In addition to vanilla retrieval, we also use strong LLMs (GPT-4<span id="footnote5" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">5</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">5</sup><span class="ltx_tag ltx_tag_note">5</span>We use gpt-4-1106-preview in our experiments.</span></span></span> in our case) to rerank the top retrieved results from the above retrievers. We use two strategies:</p>
</div>
<div id="S3.SS2.SSS0.Px2.p2" class="ltx_para">
<p id="S3.SS2.SSS0.Px2.p2.2" class="ltx_p"><span id="S3.SS2.SSS0.Px2.p2.2.1" class="ltx_text ltx_font_bold">Vanilla reranking.</span> We include the top-<math id="S3.SS2.SSS0.Px2.p2.1.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S3.SS2.SSS0.Px2.p2.1.m1.1a"><mi id="S3.SS2.SSS0.Px2.p2.1.m1.1.1" xref="S3.SS2.SSS0.Px2.p2.1.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p2.1.m1.1b"><ci id="S3.SS2.SSS0.Px2.p2.1.m1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p2.1.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p2.1.m1.1c">n</annotation></semantics></math> retrieved papers (titles and abstracts) in the context and prompt GPT-4 to rerank these based on the question (see our prompt in <a href="#A5.T13" title="In Appendix E Prompts and Additional Statistics ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">13</span></a>). This is similar to prior works <cite class="ltx_cite ltx_citemacro_citep">(Sun et al., <a href="#bib.bib34" title="" class="ltx_ref">2023b</a>; Ma et al., <a href="#bib.bib21" title="" class="ltx_ref">2023</a>)</cite>. We use <math id="S3.SS2.SSS0.Px2.p2.2.m2.1" class="ltx_Math" alttext="n=100" display="inline"><semantics id="S3.SS2.SSS0.Px2.p2.2.m2.1a"><mrow id="S3.SS2.SSS0.Px2.p2.2.m2.1.1" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.2" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.2.cmml">n</mi><mo id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.1" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.3" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.3.cmml">100</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p2.2.m2.1b"><apply id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.cmml" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1"><eq id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.1"></eq><ci id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.2">𝑛</ci><cn type="integer" id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.3">100</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p2.2.m2.1c">n=100</annotation></semantics></math>, resulting in an average context length of 13,844 words.</p>
</div>
<div id="S3.SS2.SSS0.Px2.p3" class="ltx_para">
<p id="S3.SS2.SSS0.Px2.p3.1" class="ltx_p"><span id="S3.SS2.SSS0.Px2.p3.1.1" class="ltx_text ltx_font_bold">One-hop reranking.</span> Inspired by <cite class="ltx_cite ltx_citemacro_citet">Tang et al. (<a href="#bib.bib35" title="" class="ltx_ref">2023</a>)</cite>, we leverage the fact that for some questions, there may exist lexically similar inline citation mentions in the retrieval corpus. Due to our data collection pipeline, this is particularly true for the ACL sourced inline citation questions. We posit that the retrieval models will be able to retrieve these source papers (that cite the target papers) based on the questions.</p>
</div>
<div id="S3.SS2.SSS0.Px2.p4" class="ltx_para">
<p id="S3.SS2.SSS0.Px2.p4.10" class="ltx_p">We extract the top <math id="S3.SS2.SSS0.Px2.p4.1.m1.1" class="ltx_Math" alttext="m" display="inline"><semantics id="S3.SS2.SSS0.Px2.p4.1.m1.1a"><mi id="S3.SS2.SSS0.Px2.p4.1.m1.1.1" xref="S3.SS2.SSS0.Px2.p4.1.m1.1.1.cmml">m</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p4.1.m1.1b"><ci id="S3.SS2.SSS0.Px2.p4.1.m1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p4.1.m1.1.1">𝑚</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p4.1.m1.1c">m</annotation></semantics></math> retrieved papers and construct a new candidate list by adding papers cited by each of these seed retrieved papers. We concatenate papers in the following order, skipping duplicates: [rank-1 paper <math id="S3.SS2.SSS0.Px2.p4.2.m2.1" class="ltx_Math" alttext="p_{1}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p4.2.m2.1a"><msub id="S3.SS2.SSS0.Px2.p4.2.m2.1.1" xref="S3.SS2.SSS0.Px2.p4.2.m2.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p4.2.m2.1.1.2" xref="S3.SS2.SSS0.Px2.p4.2.m2.1.1.2.cmml">p</mi><mn id="S3.SS2.SSS0.Px2.p4.2.m2.1.1.3" xref="S3.SS2.SSS0.Px2.p4.2.m2.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p4.2.m2.1b"><apply id="S3.SS2.SSS0.Px2.p4.2.m2.1.1.cmml" xref="S3.SS2.SSS0.Px2.p4.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p4.2.m2.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p4.2.m2.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p4.2.m2.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p4.2.m2.1.1.2">𝑝</ci><cn type="integer" id="S3.SS2.SSS0.Px2.p4.2.m2.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p4.2.m2.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p4.2.m2.1c">p_{1}</annotation></semantics></math>, papers cited by <math id="S3.SS2.SSS0.Px2.p4.3.m3.1" class="ltx_Math" alttext="p_{1}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p4.3.m3.1a"><msub id="S3.SS2.SSS0.Px2.p4.3.m3.1.1" xref="S3.SS2.SSS0.Px2.p4.3.m3.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p4.3.m3.1.1.2" xref="S3.SS2.SSS0.Px2.p4.3.m3.1.1.2.cmml">p</mi><mn id="S3.SS2.SSS0.Px2.p4.3.m3.1.1.3" xref="S3.SS2.SSS0.Px2.p4.3.m3.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p4.3.m3.1b"><apply id="S3.SS2.SSS0.Px2.p4.3.m3.1.1.cmml" xref="S3.SS2.SSS0.Px2.p4.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p4.3.m3.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p4.3.m3.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p4.3.m3.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p4.3.m3.1.1.2">𝑝</ci><cn type="integer" id="S3.SS2.SSS0.Px2.p4.3.m3.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p4.3.m3.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p4.3.m3.1c">p_{1}</annotation></semantics></math>, rank-2 paper <math id="S3.SS2.SSS0.Px2.p4.4.m4.1" class="ltx_Math" alttext="p_{2}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p4.4.m4.1a"><msub id="S3.SS2.SSS0.Px2.p4.4.m4.1.1" xref="S3.SS2.SSS0.Px2.p4.4.m4.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p4.4.m4.1.1.2" xref="S3.SS2.SSS0.Px2.p4.4.m4.1.1.2.cmml">p</mi><mn id="S3.SS2.SSS0.Px2.p4.4.m4.1.1.3" xref="S3.SS2.SSS0.Px2.p4.4.m4.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p4.4.m4.1b"><apply id="S3.SS2.SSS0.Px2.p4.4.m4.1.1.cmml" xref="S3.SS2.SSS0.Px2.p4.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p4.4.m4.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p4.4.m4.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p4.4.m4.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p4.4.m4.1.1.2">𝑝</ci><cn type="integer" id="S3.SS2.SSS0.Px2.p4.4.m4.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p4.4.m4.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p4.4.m4.1c">p_{2}</annotation></semantics></math>, papers cited by <math id="S3.SS2.SSS0.Px2.p4.5.m5.1" class="ltx_Math" alttext="p_{2}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p4.5.m5.1a"><msub id="S3.SS2.SSS0.Px2.p4.5.m5.1.1" xref="S3.SS2.SSS0.Px2.p4.5.m5.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p4.5.m5.1.1.2" xref="S3.SS2.SSS0.Px2.p4.5.m5.1.1.2.cmml">p</mi><mn id="S3.SS2.SSS0.Px2.p4.5.m5.1.1.3" xref="S3.SS2.SSS0.Px2.p4.5.m5.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p4.5.m5.1b"><apply id="S3.SS2.SSS0.Px2.p4.5.m5.1.1.cmml" xref="S3.SS2.SSS0.Px2.p4.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p4.5.m5.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p4.5.m5.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p4.5.m5.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p4.5.m5.1.1.2">𝑝</ci><cn type="integer" id="S3.SS2.SSS0.Px2.p4.5.m5.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p4.5.m5.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p4.5.m5.1c">p_{2}</annotation></semantics></math>, …, <math id="S3.SS2.SSS0.Px2.p4.6.m6.1" class="ltx_Math" alttext="p_{m}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p4.6.m6.1a"><msub id="S3.SS2.SSS0.Px2.p4.6.m6.1.1" xref="S3.SS2.SSS0.Px2.p4.6.m6.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p4.6.m6.1.1.2" xref="S3.SS2.SSS0.Px2.p4.6.m6.1.1.2.cmml">p</mi><mi id="S3.SS2.SSS0.Px2.p4.6.m6.1.1.3" xref="S3.SS2.SSS0.Px2.p4.6.m6.1.1.3.cmml">m</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p4.6.m6.1b"><apply id="S3.SS2.SSS0.Px2.p4.6.m6.1.1.cmml" xref="S3.SS2.SSS0.Px2.p4.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p4.6.m6.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p4.6.m6.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p4.6.m6.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p4.6.m6.1.1.2">𝑝</ci><ci id="S3.SS2.SSS0.Px2.p4.6.m6.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p4.6.m6.1.1.3">𝑚</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p4.6.m6.1c">p_{m}</annotation></semantics></math>, papers cited by <math id="S3.SS2.SSS0.Px2.p4.7.m7.1" class="ltx_Math" alttext="p_{m}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p4.7.m7.1a"><msub id="S3.SS2.SSS0.Px2.p4.7.m7.1.1" xref="S3.SS2.SSS0.Px2.p4.7.m7.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p4.7.m7.1.1.2" xref="S3.SS2.SSS0.Px2.p4.7.m7.1.1.2.cmml">p</mi><mi id="S3.SS2.SSS0.Px2.p4.7.m7.1.1.3" xref="S3.SS2.SSS0.Px2.p4.7.m7.1.1.3.cmml">m</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p4.7.m7.1b"><apply id="S3.SS2.SSS0.Px2.p4.7.m7.1.1.cmml" xref="S3.SS2.SSS0.Px2.p4.7.m7.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p4.7.m7.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p4.7.m7.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p4.7.m7.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p4.7.m7.1.1.2">𝑝</ci><ci id="S3.SS2.SSS0.Px2.p4.7.m7.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p4.7.m7.1.1.3">𝑚</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p4.7.m7.1c">p_{m}</annotation></semantics></math>].
To avoid very long contexts, we truncate this list after the first <math id="S3.SS2.SSS0.Px2.p4.8.m8.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S3.SS2.SSS0.Px2.p4.8.m8.1a"><mi id="S3.SS2.SSS0.Px2.p4.8.m8.1.1" xref="S3.SS2.SSS0.Px2.p4.8.m8.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p4.8.m8.1b"><ci id="S3.SS2.SSS0.Px2.p4.8.m8.1.1.cmml" xref="S3.SS2.SSS0.Px2.p4.8.m8.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p4.8.m8.1c">n</annotation></semantics></math> papers and use the same prompt for GPT-4 based reranking as above. In our experiments, we use <math id="S3.SS2.SSS0.Px2.p4.9.m9.1" class="ltx_Math" alttext="m=50" display="inline"><semantics id="S3.SS2.SSS0.Px2.p4.9.m9.1a"><mrow id="S3.SS2.SSS0.Px2.p4.9.m9.1.1" xref="S3.SS2.SSS0.Px2.p4.9.m9.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p4.9.m9.1.1.2" xref="S3.SS2.SSS0.Px2.p4.9.m9.1.1.2.cmml">m</mi><mo id="S3.SS2.SSS0.Px2.p4.9.m9.1.1.1" xref="S3.SS2.SSS0.Px2.p4.9.m9.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS0.Px2.p4.9.m9.1.1.3" xref="S3.SS2.SSS0.Px2.p4.9.m9.1.1.3.cmml">50</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p4.9.m9.1b"><apply id="S3.SS2.SSS0.Px2.p4.9.m9.1.1.cmml" xref="S3.SS2.SSS0.Px2.p4.9.m9.1.1"><eq id="S3.SS2.SSS0.Px2.p4.9.m9.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p4.9.m9.1.1.1"></eq><ci id="S3.SS2.SSS0.Px2.p4.9.m9.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p4.9.m9.1.1.2">𝑚</ci><cn type="integer" id="S3.SS2.SSS0.Px2.p4.9.m9.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p4.9.m9.1.1.3">50</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p4.9.m9.1c">m=50</annotation></semantics></math> and <math id="S3.SS2.SSS0.Px2.p4.10.m10.1" class="ltx_Math" alttext="n=200" display="inline"><semantics id="S3.SS2.SSS0.Px2.p4.10.m10.1a"><mrow id="S3.SS2.SSS0.Px2.p4.10.m10.1.1" xref="S3.SS2.SSS0.Px2.p4.10.m10.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p4.10.m10.1.1.2" xref="S3.SS2.SSS0.Px2.p4.10.m10.1.1.2.cmml">n</mi><mo id="S3.SS2.SSS0.Px2.p4.10.m10.1.1.1" xref="S3.SS2.SSS0.Px2.p4.10.m10.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS0.Px2.p4.10.m10.1.1.3" xref="S3.SS2.SSS0.Px2.p4.10.m10.1.1.3.cmml">200</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p4.10.m10.1b"><apply id="S3.SS2.SSS0.Px2.p4.10.m10.1.1.cmml" xref="S3.SS2.SSS0.Px2.p4.10.m10.1.1"><eq id="S3.SS2.SSS0.Px2.p4.10.m10.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p4.10.m10.1.1.1"></eq><ci id="S3.SS2.SSS0.Px2.p4.10.m10.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p4.10.m10.1.1.2">𝑛</ci><cn type="integer" id="S3.SS2.SSS0.Px2.p4.10.m10.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p4.10.m10.1.1.3">200</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p4.10.m10.1c">n=200</annotation></semantics></math> resulting in average length of 27,544 words.</p>
</div>
</section>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Results</h3>

<div id="S3.SS3.p1" class="ltx_para">
<p id="S3.SS3.p1.2" class="ltx_p">We outline the performance of the above systems on LitSearch in <a href="#S3.T3" title="In Retriever models. ‣ 3.2 Baselines ‣ 3 Experiments ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">3</span></a>. First, we observe that all instruction-finetuned embedding models, e.g. Instructor, E5, and GritLM, substantially outperform BM25 on our benchmark. In fact, they also perform better than the GTR model. Overall, we found that GritLM-7B achieves the best performance (<math id="S3.SS3.p1.1.m1.1" class="ltx_Math" alttext="70.8" display="inline"><semantics id="S3.SS3.p1.1.m1.1a"><mn id="S3.SS3.p1.1.m1.1.1" xref="S3.SS3.p1.1.m1.1.1.cmml">70.8</mn><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.1.m1.1b"><cn type="float" id="S3.SS3.p1.1.m1.1.1.cmml" xref="S3.SS3.p1.1.m1.1.1">70.8</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.1.m1.1c">70.8</annotation></semantics></math> recall@20 on broad questions and <math id="S3.SS3.p1.2.m2.1" class="ltx_Math" alttext="74.8" display="inline"><semantics id="S3.SS3.p1.2.m2.1a"><mn id="S3.SS3.p1.2.m2.1.1" xref="S3.SS3.p1.2.m2.1.1.cmml">74.8</mn><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.2.m2.1b"><cn type="float" id="S3.SS3.p1.2.m2.1.1.cmml" xref="S3.SS3.p1.2.m2.1.1">74.8</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.2.m2.1c">74.8</annotation></semantics></math> recall@5 on specific questions), leaving a large gap compared to other baselines.</p>
</div>
<section id="S3.SS3.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Impact of reranking.</h4>

<div id="S3.SS3.SSS0.Px1.p1" class="ltx_para">
<p id="S3.SS3.SSS0.Px1.p1.2" class="ltx_p">We also report the performance improvement brought by the reranking methods on the weakest (BM25) and strongest (GritLM) retrievers in <a href="#S3.T3" title="In Retriever models. ‣ 3.2 Baselines ‣ 3 Experiments ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">3</span></a>. We observe that both vanilla and one-hop reranking improve over the base retrieval performance. For example, on the specific subset of inline questions, the vanilla GPT-4 reranking improves the recall@5 of BM25 and GritLM by <math id="S3.SS3.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="21.5\%" display="inline"><semantics id="S3.SS3.SSS0.Px1.p1.1.m1.1a"><mrow id="S3.SS3.SSS0.Px1.p1.1.m1.1.1" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.cmml"><mn id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.cmml">21.5</mn><mo id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.1" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p1.1.m1.1b"><apply id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1"><csymbol cd="latexml" id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.1">percent</csymbol><cn type="float" id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2">21.5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p1.1.m1.1c">21.5\%</annotation></semantics></math>
and <math id="S3.SS3.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="5.5\%" display="inline"><semantics id="S3.SS3.SSS0.Px1.p1.2.m2.1a"><mrow id="S3.SS3.SSS0.Px1.p1.2.m2.1.1" xref="S3.SS3.SSS0.Px1.p1.2.m2.1.1.cmml"><mn id="S3.SS3.SSS0.Px1.p1.2.m2.1.1.2" xref="S3.SS3.SSS0.Px1.p1.2.m2.1.1.2.cmml">5.5</mn><mo id="S3.SS3.SSS0.Px1.p1.2.m2.1.1.1" xref="S3.SS3.SSS0.Px1.p1.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p1.2.m2.1b"><apply id="S3.SS3.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.2.m2.1.1"><csymbol cd="latexml" id="S3.SS3.SSS0.Px1.p1.2.m2.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.2.m2.1.1.1">percent</csymbol><cn type="float" id="S3.SS3.SSS0.Px1.p1.2.m2.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p1.2.m2.1.1.2">5.5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p1.2.m2.1c">5.5\%</annotation></semantics></math> respectively.
Interestingly, the improvements from one-hop reranking are generally lower than vanilla reranking across all subsets, when using GritLM as the base retriever; this shows that our benchmark cannot be easily “gamed” by mimicking the data collection pipeline or exploiting similar citation mentions to the question from other papers.</p>
</div>
</section>
<section id="S3.SS3.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Impact of question specificity.</h4>

<div id="S3.SS3.SSS0.Px2.p1" class="ltx_para">
<p id="S3.SS3.SSS0.Px2.p1.1" class="ltx_p"><a href="#S3.T3" title="In Retriever models. ‣ 3.2 Baselines ‣ 3 Experiments ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">3</span></a> and <a href="#S3.F3" title="In Retriever models. ‣ 3.2 Baselines ‣ 3 Experiments ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">3</span></a> show that retrieval systems generally report higher recall performance on the specific subset. This is expected: there exists a smaller number of “competing” papers, i.e. those that also satisfy the search question, for the specific subset in the retrieval corpus.
Note that our human annotation tagged questions with approximately 5 relevant papers as specific and 20 relevant papers as broad (see <a href="#S2.T1" title="In Sampling inline citation mentions. ‣ 2.1 Inline-citation Questions ‣ 2 LitSearch ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">1</span></a> for details).
We keep both subsets in our dataset as this stratified reporting presents a more nuanced view of retriever capabilities.</p>
</div>
</section>
<section id="S3.SS3.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Inline-citation vs. author-written questions.</h4>

<div id="S3.SS3.SSS0.Px3.p1" class="ltx_para">
<p id="S3.SS3.SSS0.Px3.p1.1" class="ltx_p">We observe very different performance trends for the two subsets (<a href="#S3.T3" title="In Retriever models. ‣ 3.2 Baselines ‣ 3 Experiments ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">3</span></a> and <a href="#S3.F3" title="In Retriever models. ‣ 3.2 Baselines ‣ 3 Experiments ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">3</span></a>).
In particular, inline-citation questions are harder than author-written questions for all retriever systems, on both broad and specific questions.</p>
</div>
<div id="S3.SS3.SSS0.Px3.p2" class="ltx_para">
<p id="S3.SS3.SSS0.Px3.p2.1" class="ltx_p">We attribute this difference to the higher semantic or lexical overlap of author-written questions with the paper titles and abstracts (see <a href="#S2.T2" title="In Word-overlap filtering. ‣ 2.1 Inline-citation Questions ‣ 2 LitSearch ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">2</span></a> for statistics).
This reflects expected tendency of paper authors to formulate the questions around the main contributions from the abstracts and re-use terminologies. Such annotator biases have been widely discussed in prior data collection efforts as well, particularly when humans write content from scratch <cite class="ltx_cite ltx_citemacro_citep">(Gururangan et al., <a href="#bib.bib10" title="" class="ltx_ref">2018</a>)</cite>.</p>
</div>
<figure id="S3.T4" class="ltx_table">
<div id="S3.T4.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:424.9pt;height:231.2pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(80.1pt,-43.6pt) scale(1.60550130588161,1.60550130588161) ;">
<table id="S3.T4.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S3.T4.1.1.1.1" class="ltx_tr">
<th id="S3.T4.1.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<td id="S3.T4.1.1.1.1.2" class="ltx_td ltx_align_center ltx_border_tt" colspan="2"><span id="S3.T4.1.1.1.1.2.1" class="ltx_text ltx_font_bold">Inline (specific)</span></td>
<td id="S3.T4.1.1.1.1.3" class="ltx_td ltx_align_center ltx_border_tt" colspan="2"><span id="S3.T4.1.1.1.1.3.1" class="ltx_text ltx_font_bold">Author (specific)</span></td>
</tr>
<tr id="S3.T4.1.1.2.2" class="ltx_tr">
<th id="S3.T4.1.1.2.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="S3.T4.1.1.2.2.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T4.1.1.2.2.2.1" class="ltx_text ltx_font_bold">Qual=1</span></td>
<td id="S3.T4.1.1.2.2.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T4.1.1.2.2.3.1" class="ltx_text ltx_font_bold">Qual=2</span></td>
<td id="S3.T4.1.1.2.2.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T4.1.1.2.2.4.1" class="ltx_text ltx_font_bold">Qual=1</span></td>
<td id="S3.T4.1.1.2.2.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T4.1.1.2.2.5.1" class="ltx_text ltx_font_bold">Qual=2</span></td>
</tr>
<tr id="S3.T4.1.1.3.3" class="ltx_tr">
<th id="S3.T4.1.1.3.3.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="S3.T4.1.1.3.3.2" class="ltx_td ltx_align_center">R@5</td>
<td id="S3.T4.1.1.3.3.3" class="ltx_td ltx_align_center">R@5</td>
<td id="S3.T4.1.1.3.3.4" class="ltx_td ltx_align_center">R@5</td>
<td id="S3.T4.1.1.3.3.5" class="ltx_td ltx_align_center">R@5</td>
</tr>
<tr id="S3.T4.1.1.4.4" class="ltx_tr">
<th id="S3.T4.1.1.4.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">BM25</th>
<td id="S3.T4.1.1.4.4.2" class="ltx_td ltx_align_center ltx_border_t">36.4</td>
<td id="S3.T4.1.1.4.4.3" class="ltx_td ltx_align_center ltx_border_t">30.6</td>
<td id="S3.T4.1.1.4.4.4" class="ltx_td ltx_align_center ltx_border_t">62.2</td>
<td id="S3.T4.1.1.4.4.5" class="ltx_td ltx_align_center ltx_border_t">55.0</td>
</tr>
<tr id="S3.T4.1.1.5.5" class="ltx_tr">
<th id="S3.T4.1.1.5.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">GTR-T5-large</th>
<td id="S3.T4.1.1.5.5.2" class="ltx_td ltx_align_center">42.0</td>
<td id="S3.T4.1.1.5.5.3" class="ltx_td ltx_align_center">31.4</td>
<td id="S3.T4.1.1.5.5.4" class="ltx_td ltx_align_center">40.7</td>
<td id="S3.T4.1.1.5.5.5" class="ltx_td ltx_align_center">36.9</td>
</tr>
<tr id="S3.T4.1.1.6.6" class="ltx_tr">
<th id="S3.T4.1.1.6.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Instructor-XL</th>
<td id="S3.T4.1.1.6.6.2" class="ltx_td ltx_align_center">55.1</td>
<td id="S3.T4.1.1.6.6.3" class="ltx_td ltx_align_center">39.5</td>
<td id="S3.T4.1.1.6.6.4" class="ltx_td ltx_align_center">58.5</td>
<td id="S3.T4.1.1.6.6.5" class="ltx_td ltx_align_center">48.6</td>
</tr>
<tr id="S3.T4.1.1.7.7" class="ltx_tr">
<th id="S3.T4.1.1.7.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">E5-large-v2</th>
<td id="S3.T4.1.1.7.7.2" class="ltx_td ltx_align_center">48.4</td>
<td id="S3.T4.1.1.7.7.3" class="ltx_td ltx_align_center">42.6</td>
<td id="S3.T4.1.1.7.7.4" class="ltx_td ltx_align_center">61.5</td>
<td id="S3.T4.1.1.7.7.5" class="ltx_td ltx_align_center">57.7</td>
</tr>
<tr id="S3.T4.1.1.8.8" class="ltx_tr">
<th id="S3.T4.1.1.8.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">GritLM-7B</th>
<td id="S3.T4.1.1.8.8.2" class="ltx_td ltx_align_center ltx_border_bb">67.3</td>
<td id="S3.T4.1.1.8.8.3" class="ltx_td ltx_align_center ltx_border_bb">58.7</td>
<td id="S3.T4.1.1.8.8.4" class="ltx_td ltx_align_center ltx_border_bb">80.0</td>
<td id="S3.T4.1.1.8.8.5" class="ltx_td ltx_align_center ltx_border_bb">76.6</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 4: </span>
Comparison of retrieval performance on different quality (qual) questions. Generally, retrievers report lower performance on the Qual=2 questions, i.e. those deemed more challenging in our manual annotation.
</figcaption>
</figure>
</section>
<section id="S3.SS3.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Impact of question quality.</h4>

<div id="S3.SS3.SSS0.Px4.p1" class="ltx_para">
<p id="S3.SS3.SSS0.Px4.p1.1" class="ltx_p"><a href="#S3.T4" title="In Inline-citation vs. author-written questions. ‣ 3.3 Results ‣ 3 Experiments ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">4</span></a> compares how retrieval models perform on different quality subsets. Recall that we manually annotated the quality of all questions (<a href="#S2.T1" title="In Sampling inline citation mentions. ‣ 2.1 Inline-citation Questions ‣ 2 LitSearch ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">1</span></a>). We observe that questions with a quality score of <math id="S3.SS3.SSS0.Px4.p1.1.m1.1" class="ltx_Math" alttext="2" display="inline"><semantics id="S3.SS3.SSS0.Px4.p1.1.m1.1a"><mn id="S3.SS3.SSS0.Px4.p1.1.m1.1.1" xref="S3.SS3.SSS0.Px4.p1.1.m1.1.1.cmml">2</mn><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px4.p1.1.m1.1b"><cn type="integer" id="S3.SS3.SSS0.Px4.p1.1.m1.1.1.cmml" xref="S3.SS3.SSS0.Px4.p1.1.m1.1.1">2</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px4.p1.1.m1.1c">2</annotation></semantics></math>, i.e. determined to be more realistic and difficult by manual annotators, are consistently more challenging for retrieval models. This demonstrates the high annotation quality of our manual inspection step. The presence of these different quality questions in our dataset leads to higher diversity and better coverage over the varied information seeking needs of users.</p>
</div>
</section>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Analysis</h2>

<figure id="S4.T5" class="ltx_table">
<table id="S4.T5.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T5.1.1.1" class="ltx_tr">
<th id="S4.T5.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="S4.T5.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2"><span id="S4.T5.1.1.1.2.1" class="ltx_text ltx_font_bold">Inline</span></th>
<th id="S4.T5.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2"><span id="S4.T5.1.1.1.3.1" class="ltx_text ltx_font_bold">Author</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T5.1.2.1" class="ltx_tr">
<th id="S4.T5.1.2.1.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="S4.T5.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T5.1.2.1.2.1" class="ltx_text ltx_font_bold">Broad</span></td>
<td id="S4.T5.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T5.1.2.1.3.1" class="ltx_text ltx_font_bold">Spec</span></td>
<td id="S4.T5.1.2.1.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T5.1.2.1.4.1" class="ltx_text ltx_font_bold">Broad</span></td>
<td id="S4.T5.1.2.1.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T5.1.2.1.5.1" class="ltx_text ltx_font_bold">Spec</span></td>
</tr>
<tr id="S4.T5.1.3.2" class="ltx_tr">
<th id="S4.T5.1.3.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="S4.T5.1.3.2.2" class="ltx_td ltx_align_center">R@20</td>
<td id="S4.T5.1.3.2.3" class="ltx_td ltx_align_center">R@5</td>
<td id="S4.T5.1.3.2.4" class="ltx_td ltx_align_center">R@20</td>
<td id="S4.T5.1.3.2.5" class="ltx_td ltx_align_center">R@5</td>
</tr>
<tr id="S4.T5.1.4.3" class="ltx_tr">
<th id="S4.T5.1.4.3.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_t">BM25</th>
<td id="S4.T5.1.4.3.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T5.1.4.3.2.1" class="ltx_text ltx_font_bold">37.4</span></td>
<td id="S4.T5.1.4.3.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T5.1.4.3.3.1" class="ltx_text ltx_font_bold">38.5</span></td>
<td id="S4.T5.1.4.3.4" class="ltx_td ltx_align_center ltx_border_t">48.6</td>
<td id="S4.T5.1.4.3.5" class="ltx_td ltx_align_center ltx_border_t">62.6</td>
</tr>
<tr id="S4.T5.1.5.4" class="ltx_tr">
<th id="S4.T5.1.5.4.1" class="ltx_td ltx_align_right ltx_th ltx_th_row">   w/ full</th>
<td id="S4.T5.1.5.4.2" class="ltx_td ltx_align_center">18.6</td>
<td id="S4.T5.1.5.4.3" class="ltx_td ltx_align_center">23.8</td>
<td id="S4.T5.1.5.4.4" class="ltx_td ltx_align_center"><span id="S4.T5.1.5.4.4.1" class="ltx_text ltx_font_bold">65.7</span></td>
<td id="S4.T5.1.5.4.5" class="ltx_td ltx_align_center"><span id="S4.T5.1.5.4.5.1" class="ltx_text ltx_font_bold">71.6</span></td>
</tr>
<tr id="S4.T5.1.6.5" class="ltx_tr">
<th id="S4.T5.1.6.5.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_t">GTR-T5-large</th>
<td id="S4.T5.1.6.5.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T5.1.6.5.2.1" class="ltx_text ltx_font_bold">45.7</span></td>
<td id="S4.T5.1.6.5.3" class="ltx_td ltx_align_center ltx_border_t">38.5</td>
<td id="S4.T5.1.6.5.4" class="ltx_td ltx_align_center ltx_border_t">37.1</td>
<td id="S4.T5.1.6.5.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T5.1.6.5.5.1" class="ltx_text ltx_font_bold">40.8</span></td>
</tr>
<tr id="S4.T5.1.7.6" class="ltx_tr">
<th id="S4.T5.1.7.6.1" class="ltx_td ltx_align_right ltx_th ltx_th_row">   w/ full</th>
<td id="S4.T5.1.7.6.2" class="ltx_td ltx_align_center">43.9</td>
<td id="S4.T5.1.7.6.3" class="ltx_td ltx_align_center"><span id="S4.T5.1.7.6.3.1" class="ltx_text ltx_font_bold">39.4</span></td>
<td id="S4.T5.1.7.6.4" class="ltx_td ltx_align_center"><span id="S4.T5.1.7.6.4.1" class="ltx_text ltx_font_bold">45.7</span></td>
<td id="S4.T5.1.7.6.5" class="ltx_td ltx_align_center">39.8</td>
</tr>
<tr id="S4.T5.1.8.7" class="ltx_tr">
<th id="S4.T5.1.8.7.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_t">Instructor-XL</th>
<td id="S4.T5.1.8.7.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T5.1.8.7.2.1" class="ltx_text ltx_font_bold">56.3</span></td>
<td id="S4.T5.1.8.7.3" class="ltx_td ltx_align_center ltx_border_t">48.9</td>
<td id="S4.T5.1.8.7.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T5.1.8.7.4.1" class="ltx_text ltx_font_bold">57.1</span></td>
<td id="S4.T5.1.8.7.5" class="ltx_td ltx_align_center ltx_border_t">55.9</td>
</tr>
<tr id="S4.T5.1.9.8" class="ltx_tr">
<th id="S4.T5.1.9.8.1" class="ltx_td ltx_align_right ltx_th ltx_th_row">   w/ full</th>
<td id="S4.T5.1.9.8.2" class="ltx_td ltx_align_center">53.0</td>
<td id="S4.T5.1.9.8.3" class="ltx_td ltx_align_center"><span id="S4.T5.1.9.8.3.1" class="ltx_text ltx_font_bold">50.9</span></td>
<td id="S4.T5.1.9.8.4" class="ltx_td ltx_align_center"><span id="S4.T5.1.9.8.4.1" class="ltx_text ltx_font_bold">57.1</span></td>
<td id="S4.T5.1.9.8.5" class="ltx_td ltx_align_center"><span id="S4.T5.1.9.8.5.1" class="ltx_text ltx_font_bold">56.9</span></td>
</tr>
<tr id="S4.T5.1.10.9" class="ltx_tr">
<th id="S4.T5.1.10.9.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_t">E5-large-v2</th>
<td id="S4.T5.1.10.9.2" class="ltx_td ltx_align_center ltx_border_t">55.8</td>
<td id="S4.T5.1.10.9.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T5.1.10.9.3.1" class="ltx_text ltx_font_bold">50.4</span></td>
<td id="S4.T5.1.10.9.4" class="ltx_td ltx_align_center ltx_border_t">54.3</td>
<td id="S4.T5.1.10.9.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T5.1.10.9.5.1" class="ltx_text ltx_font_bold">62.6</span></td>
</tr>
<tr id="S4.T5.1.11.10" class="ltx_tr">
<th id="S4.T5.1.11.10.1" class="ltx_td ltx_align_right ltx_th ltx_th_row">   w/ full</th>
<td id="S4.T5.1.11.10.2" class="ltx_td ltx_align_center"><span id="S4.T5.1.11.10.2.1" class="ltx_text ltx_font_bold">56.9</span></td>
<td id="S4.T5.1.11.10.3" class="ltx_td ltx_align_center">48.7</td>
<td id="S4.T5.1.11.10.4" class="ltx_td ltx_align_center"><span id="S4.T5.1.11.10.4.1" class="ltx_text ltx_font_bold">60.0</span></td>
<td id="S4.T5.1.11.10.5" class="ltx_td ltx_align_center">62.1</td>
</tr>
<tr id="S4.T5.1.12.11" class="ltx_tr">
<th id="S4.T5.1.12.11.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_t">GritLM-7B</th>
<td id="S4.T5.1.12.11.2" class="ltx_td ltx_align_center ltx_border_t">69.7</td>
<td id="S4.T5.1.12.11.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T5.1.12.11.3.1" class="ltx_text ltx_font_bold">67.7</span></td>
<td id="S4.T5.1.12.11.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T5.1.12.11.4.1" class="ltx_text ltx_font_bold">74.3</span></td>
<td id="S4.T5.1.12.11.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T5.1.12.11.5.1" class="ltx_text ltx_font_bold">82.5</span></td>
</tr>
<tr id="S4.T5.1.13.12" class="ltx_tr">
<th id="S4.T5.1.13.12.1" class="ltx_td ltx_align_right ltx_th ltx_th_row ltx_border_bb">   w/ full</th>
<td id="S4.T5.1.13.12.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T5.1.13.12.2.1" class="ltx_text ltx_font_bold">70.8</span></td>
<td id="S4.T5.1.13.12.3" class="ltx_td ltx_align_center ltx_border_bb">63.4</td>
<td id="S4.T5.1.13.12.4" class="ltx_td ltx_align_center ltx_border_bb">65.7</td>
<td id="S4.T5.1.13.12.5" class="ltx_td ltx_align_center ltx_border_bb">73.0</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 5: </span>
Retrieval results of using only titles and abstracts vs. using titles, abstracts, and full text (w/ full). We do not observe consistent improvements from including the full text for existing retrieval models. </figcaption>
</figure>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Does Including More Paper Content Improve Retrieval Performance?</h3>

<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.1" class="ltx_p">In the previous section, we only used the titles and abstracts (on average 134 words) to encode the papers in the retrieval corpus. Here, we evaluate whether encoding more paper content can improve retrieval performance. For all retriever models compared, we create embeddings using the full paper text (on average 6,041 words) up to their allowed context lengths.<span id="footnote6" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">6</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">6</sup><span class="ltx_tag ltx_tag_note">6</span>The maximum context lengths for GTR-T5-large, Instructor-XL, E5-large-v2 and GritLM-7B are 512, 512, 512 and 2048 tokens respectively.</span></span></span> We compare this setting against our default setting (only titles and abstracts).</p>
</div>
<div id="S4.SS1.p2" class="ltx_para">
<p id="S4.SS1.p2.1" class="ltx_p">Our results are outlined in <a href="#S4.T5" title="In 4 Analysis ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">5</span></a>. Surprisingly, we find that the addition of more paper text does not improve performance on LitSearch consistently. In fact, we only observe substantial improvement on the author-written broad questions for BM25 and some embedding models. In other cases, more text more often hinders instead of improving performance.
Note that the maximum context length of the tested models is 2,048 (GritLM) and the average length of their training data is even shorter—for example, the commonly used MS-MARCO <cite class="ltx_cite ltx_citemacro_citep">(Nguyen et al., <a href="#bib.bib25" title="" class="ltx_ref">2017</a>)</cite> and NaturalQuestions <cite class="ltx_cite ltx_citemacro_citep">(Lee et al., <a href="#bib.bib18" title="" class="ltx_ref">2019</a>)</cite> have an average document length of 56 and 79. This is significantly shorter than the full text of papers from our retrieval corpus averaging 6,041 words in length, potentially leading to the
unsatisfying performance when using full texts with embedding models.</p>
</div>
<figure id="S4.T6" class="ltx_table">
<div id="S4.T6.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:424.9pt;height:413.6pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(83.0pt,-80.8pt) scale(1.64139579364555,1.64139579364555) ;">
<table id="S4.T6.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S4.T6.1.1.1.1" class="ltx_tr">
<th id="S4.T6.1.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<td id="S4.T6.1.1.1.1.2" class="ltx_td ltx_align_center ltx_border_tt" colspan="2"><span id="S4.T6.1.1.1.1.2.1" class="ltx_text ltx_font_bold">ACL</span></td>
<td id="S4.T6.1.1.1.1.3" class="ltx_td ltx_align_center ltx_border_tt" colspan="2"><span id="S4.T6.1.1.1.1.3.1" class="ltx_text ltx_font_bold">Non-ACL</span></td>
</tr>
<tr id="S4.T6.1.1.2.2" class="ltx_tr">
<th id="S4.T6.1.1.2.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="S4.T6.1.1.2.2.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T6.1.1.2.2.2.1" class="ltx_text ltx_font_bold">Broad</span></td>
<td id="S4.T6.1.1.2.2.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T6.1.1.2.2.3.1" class="ltx_text ltx_font_bold">Specific</span></td>
<td id="S4.T6.1.1.2.2.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T6.1.1.2.2.4.1" class="ltx_text ltx_font_bold">Broad</span></td>
<td id="S4.T6.1.1.2.2.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T6.1.1.2.2.5.1" class="ltx_text ltx_font_bold">Specific</span></td>
</tr>
<tr id="S4.T6.1.1.3.3" class="ltx_tr">
<th id="S4.T6.1.1.3.3.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="S4.T6.1.1.3.3.2" class="ltx_td ltx_align_center">R@20</td>
<td id="S4.T6.1.1.3.3.3" class="ltx_td ltx_align_center">R@5</td>
<td id="S4.T6.1.1.3.3.4" class="ltx_td ltx_align_center">R@20</td>
<td id="S4.T6.1.1.3.3.5" class="ltx_td ltx_align_center">R@5</td>
</tr>
<tr id="S4.T6.1.1.4.4" class="ltx_tr">
<th id="S4.T6.1.1.4.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">BM25</th>
<td id="S4.T6.1.1.4.4.2" class="ltx_td ltx_align_center ltx_border_t">38.8</td>
<td id="S4.T6.1.1.4.4.3" class="ltx_td ltx_align_center ltx_border_t">39.4</td>
<td id="S4.T6.1.1.4.4.4" class="ltx_td ltx_align_center ltx_border_t">36.9</td>
<td id="S4.T6.1.1.4.4.5" class="ltx_td ltx_align_center ltx_border_t">38.2</td>
</tr>
<tr id="S4.T6.1.1.5.5" class="ltx_tr">
<th id="S4.T6.1.1.5.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">GTR-T5-large</th>
<td id="S4.T6.1.1.5.5.2" class="ltx_td ltx_align_center">37.2</td>
<td id="S4.T6.1.1.5.5.3" class="ltx_td ltx_align_center">39.4</td>
<td id="S4.T6.1.1.5.5.4" class="ltx_td ltx_align_center">48.9</td>
<td id="S4.T6.1.1.5.5.5" class="ltx_td ltx_align_center">38.2</td>
</tr>
<tr id="S4.T6.1.1.6.6" class="ltx_tr">
<th id="S4.T6.1.1.6.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Instructor-XL</th>
<td id="S4.T6.1.1.6.6.2" class="ltx_td ltx_align_center">48.6</td>
<td id="S4.T6.1.1.6.6.3" class="ltx_td ltx_align_center">43.9</td>
<td id="S4.T6.1.1.6.6.4" class="ltx_td ltx_align_center">59.1</td>
<td id="S4.T6.1.1.6.6.5" class="ltx_td ltx_align_center">50.9</td>
</tr>
<tr id="S4.T6.1.1.7.7" class="ltx_tr">
<th id="S4.T6.1.1.7.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">E5-large-v2</th>
<td id="S4.T6.1.1.7.7.2" class="ltx_td ltx_align_center">46.6</td>
<td id="S4.T6.1.1.7.7.3" class="ltx_td ltx_align_center">46.2</td>
<td id="S4.T6.1.1.7.7.4" class="ltx_td ltx_align_center">59.1</td>
<td id="S4.T6.1.1.7.7.5" class="ltx_td ltx_align_center">52.1</td>
</tr>
<tr id="S4.T6.1.1.8.8" class="ltx_tr">
<th id="S4.T6.1.1.8.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">GritLM-7B</th>
<td id="S4.T6.1.1.8.8.2" class="ltx_td ltx_align_center">72.4</td>
<td id="S4.T6.1.1.8.8.3" class="ltx_td ltx_align_center">65.9</td>
<td id="S4.T6.1.1.8.8.4" class="ltx_td ltx_align_center">68.8</td>
<td id="S4.T6.1.1.8.8.5" class="ltx_td ltx_align_center">68.5</td>
</tr>
<tr id="S4.T6.1.1.9.9" class="ltx_tr">
<th id="S4.T6.1.1.9.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t"><span id="S4.T6.1.1.9.9.1.1" class="ltx_text ltx_font_bold">With BM25</span></th>
<td id="S4.T6.1.1.9.9.2" class="ltx_td ltx_border_t"></td>
<td id="S4.T6.1.1.9.9.3" class="ltx_td ltx_border_t"></td>
<td id="S4.T6.1.1.9.9.4" class="ltx_td ltx_border_t"></td>
<td id="S4.T6.1.1.9.9.5" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="S4.T6.1.1.10.10" class="ltx_tr">
<th id="S4.T6.1.1.10.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">     Reranking</th>
<td id="S4.T6.1.1.10.10.2" class="ltx_td ltx_align_center">51.1</td>
<td id="S4.T6.1.1.10.10.3" class="ltx_td ltx_align_center">59.8</td>
<td id="S4.T6.1.1.10.10.4" class="ltx_td ltx_align_center">56.2</td>
<td id="S4.T6.1.1.10.10.5" class="ltx_td ltx_align_center">60.0</td>
</tr>
<tr id="S4.T6.1.1.11.11" class="ltx_tr">
<th id="S4.T6.1.1.11.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">     One-hop</th>
<td id="S4.T6.1.1.11.11.2" class="ltx_td ltx_align_center"><span id="S4.T6.1.1.11.11.2.1" class="ltx_text ltx_font_bold">65.2</span></td>
<td id="S4.T6.1.1.11.11.3" class="ltx_td ltx_align_center"><span id="S4.T6.1.1.11.11.3.1" class="ltx_text ltx_font_bold">71.2</span></td>
<td id="S4.T6.1.1.11.11.4" class="ltx_td ltx_align_center"><span id="S4.T6.1.1.11.11.4.1" class="ltx_text ltx_font_bold">60.8</span></td>
<td id="S4.T6.1.1.11.11.5" class="ltx_td ltx_align_center"><span id="S4.T6.1.1.11.11.5.1" class="ltx_text ltx_font_bold">61.2</span></td>
</tr>
<tr id="S4.T6.1.1.12.12" class="ltx_tr">
<th id="S4.T6.1.1.12.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T6.1.1.12.12.1.1" class="ltx_text ltx_font_bold">With GritLM</span></th>
<td id="S4.T6.1.1.12.12.2" class="ltx_td"></td>
<td id="S4.T6.1.1.12.12.3" class="ltx_td"></td>
<td id="S4.T6.1.1.12.12.4" class="ltx_td"></td>
<td id="S4.T6.1.1.12.12.5" class="ltx_td"></td>
</tr>
<tr id="S4.T6.1.1.13.13" class="ltx_tr">
<th id="S4.T6.1.1.13.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">     Reranking</th>
<td id="S4.T6.1.1.13.13.2" class="ltx_td ltx_align_center">80.3</td>
<td id="S4.T6.1.1.13.13.3" class="ltx_td ltx_align_center"><span id="S4.T6.1.1.13.13.3.1" class="ltx_text ltx_font_bold">72.7</span></td>
<td id="S4.T6.1.1.13.13.4" class="ltx_td ltx_align_center"><span id="S4.T6.1.1.13.13.4.1" class="ltx_text ltx_font_bold">72.7</span></td>
<td id="S4.T6.1.1.13.13.5" class="ltx_td ltx_align_center"><span id="S4.T6.1.1.13.13.5.1" class="ltx_text ltx_font_bold">73.3</span></td>
</tr>
<tr id="S4.T6.1.1.14.14" class="ltx_tr">
<th id="S4.T6.1.1.14.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">     One-hop</th>
<td id="S4.T6.1.1.14.14.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T6.1.1.14.14.2.1" class="ltx_text ltx_font_bold">81.3</span></td>
<td id="S4.T6.1.1.14.14.3" class="ltx_td ltx_align_center ltx_border_bb">67.4</td>
<td id="S4.T6.1.1.14.14.4" class="ltx_td ltx_align_center ltx_border_bb">69.9</td>
<td id="S4.T6.1.1.14.14.5" class="ltx_td ltx_align_center ltx_border_bb">71.5</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 6: </span>Comparison of retrieval performance on the ACL vs. non-ACL sourced inline-citation questions. Results show that the performance improvement from one-hop reranking over BM25 is subtantially higher for ACL sourced questions.</figcaption>
</figure>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Does the Source of Inline Citation Questions Matter?</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.1" class="ltx_p">Next, we study how the different sources of inline-citation questions
affect the model performance. <a href="#S4.T6" title="In 4.1 Does Including More Paper Content Improve Retrieval Performance? ‣ 4 Analysis ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">6</span></a> outlines the performance of retrieval models on ACL sourced vs. non-ACL sourced inline-citation questions.
Our results show that the two different sets report similar trends and model rankings for different retrieval models, particularly on the specific subset of questions. Interestingly, we find that the performance improvement from one-hop reranking is very different for the ACL and non-ACL questions.</p>
</div>
<div id="S4.SS2.p2" class="ltx_para">
<p id="S4.SS2.p2.1" class="ltx_p">For BM25, we observe that one-hop reranking is significantly better than the vanilla reranking on ACL sourced questions (+11.4% recall@5 on specific); but the gap is much smaller on non-ACL sourced questions (+1.2% recall@5 on specific).
We posit that this is because BM25 can better exploit the data annotation pipeline on the ACL sourced questions. It can likely first identify the source ACL paper where the citation mention comes from, and then find the target paper via one-hop reranking. Including the non-ACL questions to LitSearch prevents systems from exploiting such “shortcuts” as non-ACL source papers are not part of the retrieval corpus.</p>
</div>
<div id="S4.SS2.p3" class="ltx_para">
<p id="S4.SS2.p3.1" class="ltx_p">For GritLM, we do not observe similarly large performance gains when using one-hop reranking. We hypothesize that this is because when using GritLM, the initial top retrieval results already include the target papers and the one-hop strategy does not bring further improvement.</p>
</div>
<figure id="S4.T7" class="ltx_table">
<table id="S4.T7.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S4.T7.1.1.1" class="ltx_tr">
<th id="S4.T7.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<td id="S4.T7.1.1.1.2" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T7.1.1.1.2.1" class="ltx_text ltx_font_bold">Inline (specific)</span></td>
<td id="S4.T7.1.1.1.3" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T7.1.1.1.3.1" class="ltx_text ltx_font_bold">Author (specific)</span></td>
</tr>
<tr id="S4.T7.1.2.2" class="ltx_tr">
<th id="S4.T7.1.2.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="S4.T7.1.2.2.2" class="ltx_td ltx_align_center">R@5</td>
<td id="S4.T7.1.2.2.3" class="ltx_td ltx_align_center">R@5</td>
</tr>
<tr id="S4.T7.1.3.3" class="ltx_tr">
<th id="S4.T7.1.3.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">BM25</th>
<td id="S4.T7.1.3.3.2" class="ltx_td ltx_align_center ltx_border_t">38.5</td>
<td id="S4.T7.1.3.3.3" class="ltx_td ltx_align_center ltx_border_t">62.6</td>
</tr>
<tr id="S4.T7.1.4.4" class="ltx_tr">
<th id="S4.T7.1.4.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">GritLM-7B</th>
<td id="S4.T7.1.4.4.2" class="ltx_td ltx_align_center">67.7</td>
<td id="S4.T7.1.4.4.3" class="ltx_td ltx_align_center">82.5</td>
</tr>
<tr id="S4.T7.1.5.5" class="ltx_tr">
<th id="S4.T7.1.5.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Google Search</th>
<td id="S4.T7.1.5.5.2" class="ltx_td ltx_align_center ltx_border_t">23.1</td>
<td id="S4.T7.1.5.5.3" class="ltx_td ltx_align_center ltx_border_t">62.5</td>
</tr>
<tr id="S4.T7.1.6.6" class="ltx_tr">
<th id="S4.T7.1.6.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Google Scholar</th>
<td id="S4.T7.1.6.6.2" class="ltx_td ltx_align_center">20.5</td>
<td id="S4.T7.1.6.6.3" class="ltx_td ltx_align_center">17.5</td>
</tr>
<tr id="S4.T7.1.7.7" class="ltx_tr">
<th id="S4.T7.1.7.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">Elicit</th>
<td id="S4.T7.1.7.7.2" class="ltx_td ltx_align_center ltx_border_bb">23.1</td>
<td id="S4.T7.1.7.7.3" class="ltx_td ltx_align_center ltx_border_bb">17.5</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 7: </span>Recall@5 for commercial search engines on a random subset of 80 specific questions. Search engines generally report poor performance. Note that the comparison is not apples-to-apples as search engines use a much larger retrieval corpus.</figcaption>
</figure>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Performance of Search Engines</h3>

<figure id="S4.T8" class="ltx_table">
<table id="S4.T8.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T8.1.1.1" class="ltx_tr">
<th id="S4.T8.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="S4.T8.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T8.1.1.1.2.1" class="ltx_text ltx_font_bold">MSMARCO</span></th>
<th id="S4.T8.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T8.1.1.1.3.1" class="ltx_text ltx_font_bold">SCIDOCS</span></th>
<th id="S4.T8.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T8.1.1.1.4.1" class="ltx_text ltx_font_bold">NQ</span></th>
<th id="S4.T8.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T8.1.1.1.5.1" class="ltx_text ltx_font_bold">ArXiv</span></th>
<th id="S4.T8.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">
<span id="S4.T8.1.1.1.6.1" class="ltx_text ltx_font_bold">LitSearch</span> (broad)</th>
<th id="S4.T8.1.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">
<span id="S4.T8.1.1.1.7.1" class="ltx_text ltx_font_bold">LitSearch</span> (specific)</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T8.1.2.1" class="ltx_tr">
<th id="S4.T8.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">GTR-T5-large</th>
<td id="S4.T8.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t">42.7</td>
<td id="S4.T8.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t">15.5</td>
<td id="S4.T8.1.2.1.4" class="ltx_td ltx_align_center ltx_border_t">55.1</td>
<td id="S4.T8.1.2.1.5" class="ltx_td ltx_align_center ltx_border_t">17.5</td>
<td id="S4.T8.1.2.1.6" class="ltx_td ltx_align_center ltx_border_t">23.3</td>
<td id="S4.T8.1.2.1.7" class="ltx_td ltx_align_center ltx_border_t">30.4</td>
</tr>
<tr id="S4.T8.1.3.2" class="ltx_tr">
<th id="S4.T8.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Instructor-XL</th>
<td id="S4.T8.1.3.2.2" class="ltx_td ltx_align_center">41.6</td>
<td id="S4.T8.1.3.2.3" class="ltx_td ltx_align_center">17.4</td>
<td id="S4.T8.1.3.2.4" class="ltx_td ltx_align_center">57.2</td>
<td id="S4.T8.1.3.2.5" class="ltx_td ltx_align_center">19.8</td>
<td id="S4.T8.1.3.2.6" class="ltx_td ltx_align_center">32.8</td>
<td id="S4.T8.1.3.2.7" class="ltx_td ltx_align_center">41.2</td>
</tr>
<tr id="S4.T8.1.4.3" class="ltx_tr">
<th id="S4.T8.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">E5-large-v2</th>
<td id="S4.T8.1.4.3.2" class="ltx_td ltx_align_center">43.5</td>
<td id="S4.T8.1.4.3.3" class="ltx_td ltx_align_center">20.5</td>
<td id="S4.T8.1.4.3.4" class="ltx_td ltx_align_center">63.4</td>
<td id="S4.T8.1.4.3.5" class="ltx_td ltx_align_center">27.0</td>
<td id="S4.T8.1.4.3.6" class="ltx_td ltx_align_center">27.1</td>
<td id="S4.T8.1.4.3.7" class="ltx_td ltx_align_center">45.3</td>
</tr>
<tr id="S4.T8.1.5.4" class="ltx_tr">
<th id="S4.T8.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">GritLM-7B</th>
<td id="S4.T8.1.5.4.2" class="ltx_td ltx_align_center ltx_border_bb">42.0</td>
<td id="S4.T8.1.5.4.3" class="ltx_td ltx_align_center ltx_border_bb">24.4</td>
<td id="S4.T8.1.5.4.4" class="ltx_td ltx_align_center ltx_border_bb">70.3</td>
<td id="S4.T8.1.5.4.5" class="ltx_td ltx_align_center ltx_border_bb">34.3</td>
<td id="S4.T8.1.5.4.6" class="ltx_td ltx_align_center ltx_border_bb">44.1</td>
<td id="S4.T8.1.5.4.7" class="ltx_td ltx_align_center ltx_border_bb">60.3</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 8: </span>
Comparison between LitSearch and existing retrieval benchmarks.
All reported numbers are nDCG@10 for a direct comparison.
</figcaption>
</figure>
<div id="S4.SS3.p1" class="ltx_para">
<p id="S4.SS3.p1.1" class="ltx_p">In practice, researchers use search engines like Google Search, Google Scholar, or Elicit<span id="footnote7" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">7</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">7</sup><span class="ltx_tag ltx_tag_note">7</span><a target="_blank" href="https://elicit.com/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://elicit.com/</a></span></span></span> to search for relevant papers for their scientific queries. We conduct a human study to understand how these search engines perform on LitSearch: We randomly sample 80 questions (all specific; 40 inline-citation and 40 author-written) from our dataset. We manually input these questions into the above search engines and report recall@5.<span id="footnote8" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">8</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">8</sup><span class="ltx_tag ltx_tag_note">8</span>For Google Search, we consider the top-5 academic papers in the search results and ignore other webpage results. We only consider the academic papers on the first page of search results, even if this number is lower than 5.</span></span></span> We note that this is not an apples-to-apples comparison against the retrieval models in earlier sections due to the discrepancy in the retrieval corpus.</p>
</div>
<div id="S4.SS3.p2" class="ltx_para">
<p id="S4.SS3.p2.1" class="ltx_p"><a href="#S4.T7" title="In 4.2 Does the Source of Inline Citation Questions Matter? ‣ 4 Analysis ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">7</span></a> outlines the results of our human study. It shows that all three search engines deliver similarly low recalls on inline-citation questions. On the author-written questions, Google Search performs much better than the other two. Although not directly comparable, this performance is generally worse than the embedding models, demonstrating the potential of these strong dense retrieval models for citation recommendation applications.</p>
</div>
</section>
<section id="S4.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.4 </span>Comparing Other Retrieval Benchmarks</h3>

<div id="S4.SS4.p1" class="ltx_para">
<p id="S4.SS4.p1.1" class="ltx_p">We compare model performance on LitSearch
to several popular retrieval benchmarks included in BEIR <cite class="ltx_cite ltx_citemacro_citep">(Thakur et al., <a href="#bib.bib36" title="" class="ltx_ref">2021</a>)</cite> and MTEB <cite class="ltx_cite ltx_citemacro_citep">(Muennighoff et al., <a href="#bib.bib24" title="" class="ltx_ref">2022</a>)</cite>—namely MSMARCO <cite class="ltx_cite ltx_citemacro_citep">(Nguyen et al., <a href="#bib.bib25" title="" class="ltx_ref">2017</a>)</cite>, SCIDOCS <cite class="ltx_cite ltx_citemacro_citep">(Cohan et al., <a href="#bib.bib5" title="" class="ltx_ref">2020</a>)</cite>, and NQ <cite class="ltx_cite ltx_citemacro_citep">(Lee et al., <a href="#bib.bib18" title="" class="ltx_ref">2019</a>)</cite>.<span id="footnote9" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">9</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">9</sup><span class="ltx_tag ltx_tag_note">9</span>We use the results from the MTEB benchmark website: <a target="_blank" href="https://huggingface.co/spaces/mteb/leaderboard" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/spaces/mteb/leaderboard</a>.</span></span></span>
We also compare to ArXiv <cite class="ltx_cite ltx_citemacro_citep">(Gu et al., <a href="#bib.bib9" title="" class="ltx_ref">2022</a>)</cite>, a previous citation recommendation benchmark directly using inline citations as queries.
<a href="#S4.T8" title="In 4.3 Performance of Search Engines ‣ 4 Analysis ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">8</span></a> shows that
LitSearch generally agrees with existing retrieval benchmarks.
However,
LitSearch can differentiate retriever models better: for example, the gap between GritLM and E5 on LitSearch (specific) is 15 points (nDCG@10), while they perform almost the same on MSMARCO.
LitSearch provides an informative testbed that can effectively reflect the recent (and future) advancement in embedding models.</p>
</div>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Related Work</h2>

<section id="S5.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Citation recommendation.</h4>

<div id="S5.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S5.SS0.SSS0.Px1.p1.1" class="ltx_p">The community has propose a number of citation recommendation datasets <cite class="ltx_cite ltx_citemacro_cite">Färber and Jatowt (<a href="#bib.bib7" title="" class="ltx_ref">2020</a>)</cite>,
including global citation recommendation datasets (directly using a paper as the query and papers it cites as target papers; <cite class="ltx_cite ltx_citemacro_citep">Cohan et al., <a href="#bib.bib5" title="" class="ltx_ref">2020</a>; Bhagavatula et al., <a href="#bib.bib3" title="" class="ltx_ref">2018</a></cite>), and
local citation recommendation datasets (using inline citation mentions as queries;  <cite class="ltx_cite ltx_citemacro_citep">He et al., <a href="#bib.bib11" title="" class="ltx_ref">2010</a>; Medić and Snajder, <a href="#bib.bib22" title="" class="ltx_ref">2020</a>; Jeong et al., <a href="#bib.bib13" title="" class="ltx_ref">2020</a>; Gu et al., <a href="#bib.bib9" title="" class="ltx_ref">2022</a></cite>).
There are also language models and retrieval models specifically trained for scientific document understanding and retrieval tasks, such as SciBERT <cite class="ltx_cite ltx_citemacro_cite">Beltagy et al. (<a href="#bib.bib2" title="" class="ltx_ref">2019</a>)</cite> and SPECTER <cite class="ltx_cite ltx_citemacro_cite">Cohan et al. (<a href="#bib.bib5" title="" class="ltx_ref">2020</a>)</cite>.
Compared to existing citation recommendation datasets,
LitSearch is comprised of manually annotated, natural language literature search questions,
providing a more realistic and challenging evaluation for citation recommendation systems.</p>
</div>
</section>
<section id="S5.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Retrieval benchmarks.</h4>

<div id="S5.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S5.SS0.SSS0.Px2.p1.1" class="ltx_p">There have been numerous datasets evaluating
retrieval systems from
Wikipedia <cite class="ltx_cite ltx_citemacro_citep">(Kwiatkowski et al., <a href="#bib.bib16" title="" class="ltx_ref">2019</a>; Lee et al., <a href="#bib.bib18" title="" class="ltx_ref">2019</a>)</cite>,
web queries <cite class="ltx_cite ltx_citemacro_citep">(Nguyen et al., <a href="#bib.bib25" title="" class="ltx_ref">2017</a>)</cite>,
biomedical questions <cite class="ltx_cite ltx_citemacro_citep">(Voorhees and Tice, <a href="#bib.bib38" title="" class="ltx_ref">2000</a>)</cite>, and more.
Recently, there have been several benchmarks
combining multiple datasets
and evaluating retrieval or embedding models
across different domains and different use cases, such as KILT <cite class="ltx_cite ltx_citemacro_citep">(Petroni et al., <a href="#bib.bib29" title="" class="ltx_ref">2021</a>)</cite>, BEIR <cite class="ltx_cite ltx_citemacro_citep">(Thakur et al., <a href="#bib.bib36" title="" class="ltx_ref">2021</a>)</cite>, and MTEB <cite class="ltx_cite ltx_citemacro_citep">(Muennighoff et al., <a href="#bib.bib24" title="" class="ltx_ref">2022</a>)</cite>.
LitSearch offers a unique perspective by exploring the novel literature search question type,
effectively complementing the existing benchmarks.</p>
</div>
</section>
<section id="S5.SS0.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Retrieval systems.</h4>

<div id="S5.SS0.SSS0.Px3.p1" class="ltx_para">
<p id="S5.SS0.SSS0.Px3.p1.1" class="ltx_p">Traditional retrieval systems rely on bag-of-word algorithms such as TF-IDF and BM25.
Dense retrieval (embedding) models have gained more popularity due to their abilities to do semantic search without relying on exact keyword matches <cite class="ltx_cite ltx_citemacro_cite">Pennington et al. (<a href="#bib.bib28" title="" class="ltx_ref">2014</a>); Reimers and Gurevych (<a href="#bib.bib30" title="" class="ltx_ref">2019</a>)</cite>.
State-of-the-art dense models
are mostly adopted by
fine-tuning pre-trained language models <cite class="ltx_cite ltx_citemacro_citep">(Devlin et al., <a href="#bib.bib6" title="" class="ltx_ref">2019</a>; Touvron et al., <a href="#bib.bib37" title="" class="ltx_ref">2023</a>)</cite>
with a contrastive learning objective on either supervised or unsupervised data <cite class="ltx_cite ltx_citemacro_citep">(Karpukhin et al., <a href="#bib.bib14" title="" class="ltx_ref">2020</a>; Gao et al., <a href="#bib.bib8" title="" class="ltx_ref">2021</a>; Izacard et al., <a href="#bib.bib12" title="" class="ltx_ref">2022</a>; Ni et al., <a href="#bib.bib26" title="" class="ltx_ref">2022</a>; Khattab and Zaharia, <a href="#bib.bib15" title="" class="ltx_ref">2020</a>)</cite>.
Recent development introduces “instructions” when encoding queries and documents, which significantly improves the versatility of embeddings across tasks <cite class="ltx_cite ltx_citemacro_citep">(Su et al., <a href="#bib.bib32" title="" class="ltx_ref">2023</a>; Wang et al., <a href="#bib.bib39" title="" class="ltx_ref">2022</a>; Wu et al., <a href="#bib.bib40" title="" class="ltx_ref">2022</a>; Lee et al., <a href="#bib.bib17" title="" class="ltx_ref">2024</a>; BehnamGhader et al., <a href="#bib.bib1" title="" class="ltx_ref">2024</a>)</cite>.</p>
</div>
</section>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Conclusion</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">In this paper, we propose LitSearch,
a new retrieval benchmark comprised of 597 manually-curated literature search questions.
LitSearch includes a inline-citation question set and an author-written question set, both undergoing manual inspections from the authors of LitSearch.
We conduct extensive experiments with BM25, state-of-the-art embedding models, and LLM reranking. Our experiments demonstrate the superior performance of state-of-the-art instruction-finetuned embedding models,
with additional improvement via GPT-4-based reranking.
We also verify that commercial search engines like Google struggle on LitSearch questions.
The comparison with existing retrieval benchmarks shows that
LitSearch better highlights the different performance of retrieval systems.</p>
</div>
</section>
<section id="Sx1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Limitations</h2>

<div id="Sx1.p1" class="ltx_para">
<p id="Sx1.p1.1" class="ltx_p">Even though we manually examined the dataset,
there still exist questions that are either slightly out of distribution compared to what researchers would ask,
or too easy due to high overlap with the target papers.
The author-written questions are easier than we expected,
as writing challenging literature search questions is non-trivial even for experienced researchers.
Even though we experimented with several state-of-the-art systems, it was not an exhausted evaluation and we left out more sophisticated retrieval or reranking systems.
This research primarily focuses on only English questions and research papers.</p>
</div>
</section>
<section id="Sx2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Ethics Statement</h2>

<div id="Sx2.p1" class="ltx_para">
<p id="Sx2.p1.1" class="ltx_p">The research artifact of this paper, LitSearch, is manually inspected and is ensured to have no unsafe or inappropriate content.
However, the process to generate the dataset may introduce certain biases: for example, the inline-citation questions contain more target papers that have high citations due to the sampling; the author-written questions only cover ACL 2023 and ICLR 2024 papers.</p>
</div>
</section>
<section id="Sx3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Acknowledgements</h2>

<div id="Sx3.p1" class="ltx_para">
<p id="Sx3.p1.1" class="ltx_p">We want to acknowledge Dan Friedman, Howard Yen, Jiayi Geng, Lucy He, and other members of the Princeton NLP group for their useful feedback and discussion.
We also acknowledge all the ACL 2023 and ICLR 2024 authors that contributed questions to LitSearch (listed in <a href="#A1" title="Appendix A Annotator Acknowledgments ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Appendix</span> <span class="ltx_text ltx_ref_tag">A</span></a>).
Tianyu Gao is supported by an IBM PhD Fellowship.
This work is gratefully supported by an NSF CAREER award (IIS-2239290), and Microsoft Azure credits through the “Accelerate Foundation Models Academic Research” Initiative.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">BehnamGhader et al. (2024)</span>
<span class="ltx_bibblock">
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2404.05961" title="" class="ltx_ref ltx_href">Llm2vec: Large language models are secretly powerful text encoders</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2404.05961.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Beltagy et al. (2019)</span>
<span class="ltx_bibblock">
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/D19-1371" title="" class="ltx_ref ltx_href">SciBERT: A pretrained language model for scientific text</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)</em>, pages 3615–3620, Hong Kong, China. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bhagavatula et al. (2018)</span>
<span class="ltx_bibblock">
Chandra Bhagavatula, Sergey Feldman, Russell Power, and Waleed Ammar. 2018.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/N18-1022" title="" class="ltx_ref ltx_href">Content-based citation recommendation</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)</em>, pages 238–251, New Orleans, Louisiana. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Brown et al. (2020)</span>
<span class="ltx_bibblock">
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://papers.nips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html" title="" class="ltx_ref ltx_href">Language models are few-shot learners</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems (NeurIPS)</em>.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cohan et al. (2020)</span>
<span class="ltx_bibblock">
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2020.acl-main.207" title="" class="ltx_ref ltx_href">SPECTER: Document-level representation learning using citation-informed transformers</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics</em>, pages 2270–2282, Online. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Devlin et al. (2019)</span>
<span class="ltx_bibblock">
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://aclanthology.org/N19-1423/" title="" class="ltx_ref ltx_href">BERT: Pre-training of deep bidirectional Transformers for language understanding</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">North American Chapter of the Association for Computational Linguistics (NAACL)</em>.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Färber and Jatowt (2020)</span>
<span class="ltx_bibblock">
Michael Färber and Adam Jatowt. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.1007/s00799-020-00288-2" title="" class="ltx_ref ltx_href">Citation recommendation: approaches and datasets</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">Int. J. Digit. Libr.</em>, 21(4):375–405.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gao et al. (2021)</span>
<span class="ltx_bibblock">
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2021.emnlp-main.552" title="" class="ltx_ref ltx_href">SimCSE: Simple contrastive learning of sentence embeddings</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">Empirical Methods in Natural Language Processing (EMNLP)</em>, pages 6894–6910.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gu et al. (2022)</span>
<span class="ltx_bibblock">
Nianlong Gu, Yingqiang Gao, and Richard H. R. Hahnloser. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.1007/978-3-030-99736-6_19" title="" class="ltx_ref ltx_href">Local citation recommendation with hierarchical-attention text encoder and scibert-based reranking</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">Advances in Information Retrieval: 44th European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10–14, 2022, Proceedings, Part I</em>, page 274–288, Berlin, Heidelberg. Springer-Verlag.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gururangan et al. (2018)</span>
<span class="ltx_bibblock">
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/N18-2017" title="" class="ltx_ref ltx_href">Annotation artifacts in natural language inference data</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)</em>, pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">He et al. (2010)</span>
<span class="ltx_bibblock">
Qi He, Jian Pei, Daniel Kifer, Prasenjit Mitra, and Lee Giles. 2010.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.1145/1772690.1772734" title="" class="ltx_ref ltx_href">Context-aware citation recommendation</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 19th International Conference on World Wide Web</em>, WWW ’10, page 421–430, New York, NY, USA. Association for Computing Machinery.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Izacard et al. (2022)</span>
<span class="ltx_bibblock">
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=jKN1pXi7b0" title="" class="ltx_ref ltx_href">Unsupervised dense information retrieval with contrastive learning</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">Transactions on Machine Learning Research</em>.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jeong et al. (2020)</span>
<span class="ltx_bibblock">
Chanwoo Jeong, Sion Jang, Eunjeong Park, and Sungchul Choi. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.1007/s11192-020-03561-y" title="" class="ltx_ref ltx_href">A context-aware citation recommendation model with bert and graph convolutional networks</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">Scientometrics</em>, 124(3):1907–1922.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Karpukhin et al. (2020)</span>
<span class="ltx_bibblock">
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2020.emnlp-main.550" title="" class="ltx_ref ltx_href">Dense passage retrieval for open-domain question answering</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)</em>, pages 6769–6781, Online. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Khattab and Zaharia (2020)</span>
<span class="ltx_bibblock">
O. Khattab and Matei A. Zaharia. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:216553223" title="" class="ltx_ref ltx_href">Colbert: Efficient and effective passage search via contextualized late interaction over bert</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval</em>.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kwiatkowski et al. (2019)</span>
<span class="ltx_bibblock">
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.1162/tacl_a_00276" title="" class="ltx_ref ltx_href">Natural questions: A benchmark for question answering research</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">Transactions of the Association for Computational Linguistics</em>, 7:452–466.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lee et al. (2024)</span>
<span class="ltx_bibblock">
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2405.17428" title="" class="ltx_ref ltx_href">Nv-embed: Improved techniques for training llms as generalist embedding models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2405.17428.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lee et al. (2019)</span>
<span class="ltx_bibblock">
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/P19-1612" title="" class="ltx_ref ltx_href">Latent retrieval for weakly supervised open domain question answering</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">Association for Computational Linguistics (ACL)</em>, pages 6086–6096.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2019)</span>
<span class="ltx_bibblock">
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/1907.11692" title="" class="ltx_ref ltx_href">RoBERTa: A robustly optimized BERT pretraining approach</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1907.11692</em>.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lo et al. (2020)</span>
<span class="ltx_bibblock">
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2020.acl-main.447" title="" class="ltx_ref ltx_href">S2ORC: The semantic scholar open research corpus</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">Association for Computational Linguistics (ACL)</em>, pages 4969–4983.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ma et al. (2023)</span>
<span class="ltx_bibblock">
Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/pdf/2305.02156" title="" class="ltx_ref ltx_href">Zero-shot listwise document reranking with a large language model</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.02156</em>.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Medić and Snajder (2020)</span>
<span class="ltx_bibblock">
Zoran Medić and Jan Snajder. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2020.sdp-1.11" title="" class="ltx_ref ltx_href">Improved local citation recommendation based on context enhanced with global information</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">Proceedings of the First Workshop on Scholarly Document Processing</em>, pages 97–103, Online. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Muennighoff et al. (2024)</span>
<span class="ltx_bibblock">
Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2402.09906" title="" class="ltx_ref ltx_href">Generative representational instruction tuning</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.09906</em>.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Muennighoff et al. (2022)</span>
<span class="ltx_bibblock">
Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.48550/ARXIV.2210.07316" title="" class="ltx_ref ltx_href">Mteb: Massive text embedding benchmark</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2210.07316</em>.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nguyen et al. (2017)</span>
<span class="ltx_bibblock">
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2017.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=Hk1iOLcle" title="" class="ltx_ref ltx_href">MS MARCO: A human-generated MAchine reading COmprehension dataset</a>.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ni et al. (2022)</span>
<span class="ltx_bibblock">
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, and Yinfei Yang. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://aclanthology.org/2022.emnlp-main.669" title="" class="ltx_ref ltx_href">Large dual encoders are generalizable retrievers</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">Empirical Methods in Natural Language Processing (EMNLP)</em>, pages 9844–9855.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">OpenAI (2023)</span>
<span class="ltx_bibblock">
OpenAI. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2303.08774" title="" class="ltx_ref ltx_href">GPT-4 Technical Report</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2303.08774.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Pennington et al. (2014)</span>
<span class="ltx_bibblock">
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.3115/v1/D14-1162" title="" class="ltx_ref ltx_href">GloVe: Global vectors for word representation</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">Empirical Methods in Natural Language Processing (EMNLP)</em>, pages 1532–1543.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Petroni et al. (2021)</span>
<span class="ltx_bibblock">
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2021.naacl-main.200" title="" class="ltx_ref ltx_href">KILT: a benchmark for knowledge intensive language tasks</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies</em>, pages 2523–2544, Online. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Reimers and Gurevych (2019)</span>
<span class="ltx_bibblock">
Nils Reimers and Iryna Gurevych. 2019.

</span>
<span class="ltx_bibblock">Sentence-BERT: Sentence embeddings using Siamese BERT-networks.

</span>
<span class="ltx_bibblock">In <em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)</em>.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Robertson et al. (2009)</span>
<span class="ltx_bibblock">
Stephen Robertson, Hugo Zaragoza, et al. 2009.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://www.nowpublishers.com/article/Details/INR-019" title="" class="ltx_ref ltx_href">The probabilistic relevance framework: Bm25 and beyond</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">Foundations and Trends® in Information Retrieval</em>, 3(4):333–389.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Su et al. (2023)</span>
<span class="ltx_bibblock">
Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2023.findings-acl.71" title="" class="ltx_ref ltx_href">One embedder, any task: Instruction-finetuned text embeddings</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">Findings of the Association for Computational Linguistics: ACL 2023</em>, pages 1102–1121, Toronto, Canada. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sun et al. (2023a)</span>
<span class="ltx_bibblock">
Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. 2023a.

</span>
<span class="ltx_bibblock">Is chatgpt good at search? investigating large language models as re-ranking agent.

</span>
<span class="ltx_bibblock"><em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2304.09542.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sun et al. (2023b)</span>
<span class="ltx_bibblock">
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023b.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2023.emnlp-main.923" title="" class="ltx_ref ltx_href">Is ChatGPT good at search? investigating large language models as re-ranking agents</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">Empirical Methods in Natural Language Processing (EMNLP)</em>, pages 14918–14937.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Tang et al. (2023)</span>
<span class="ltx_bibblock">
Michael Tang, Shunyu Yao, John Yang, and Karthik Narasimhan. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2305.15098" title="" class="ltx_ref ltx_href">Referral augmentation for zero-shot information retrieval</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2305.15098.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Thakur et al. (2021)</span>
<span class="ltx_bibblock">
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=wCu6T5xFjeJ" title="" class="ltx_ref ltx_href">BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)</em>.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. (2023)</span>
<span class="ltx_bibblock">
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/pdf/2302.13971.pdf" title="" class="ltx_ref ltx_href">LLaMA: Open and Efficient Foundation Language Models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2302.13971</em>.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Voorhees and Tice (2000)</span>
<span class="ltx_bibblock">
Ellen M. Voorhees and Dawn M. Tice. 2000.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.1145/345508.345577" title="" class="ltx_ref ltx_href">Building a question answering test collection</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval</em>, SIGIR ’00, page 200–207, New York, NY, USA. Association for Computing Machinery.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2022)</span>
<span class="ltx_bibblock">
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:254366618" title="" class="ltx_ref ltx_href">Text embeddings by weakly-supervised contrastive pre-training</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2212.03533.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wu et al. (2022)</span>
<span class="ltx_bibblock">
Jialian Wu, Jianfeng Wang, Zhengyuan Yang, Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2212.00280" title="" class="ltx_ref ltx_href">Grit: A generative region-to-text transformer for object understanding</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2212.00280.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Annotator Acknowledgments</h2>

<div id="A1.p1" class="ltx_para">
<p id="A1.p1.1" class="ltx_p">We would like to thank Marah I Abdin, Jaewoo Ahn, Kabir Ahuja, Xi Ai, Satoshi Akasaki, Anastasios N Angelopoulos, Jinheon Baek, Eslam Mohamed Bakr, Pablo Barceló, Claudio Battiloro, Jonas Belouadi, Abhik Bhattacharjee, Valeriia Bolotova, Pengshan Cai, Nitay Calderon, Qingqing Cao, Defu Cao, Souradip Chakraborty, Jun Shern Chan, Sachin Chanchani, Yulong Chen, Yiming Chen, Xinyuan Chen, Nuo Chen, Hanjie Chen, Xiudi Chen, Zeming Chen, An-Chieh Cheng, Xize Cheng, Cheng-Han Chiang, Josef Dai, David Dale, Yue Deng, Yifan Deng, Shizhe Diao, Bosheng Ding, Xuan Long Do, Yilun Du, Yupei Du, Salijona Dyrmishi, Dante Everaert, Zhenghan Fang, Bahare Fatemi, Jiazhan Feng, Shangbin Feng, Patrick Fernandes, Javier Ferrando, Christopher Fifty, Sarah E Finch, Matthew Finlayson, Lea Frermann, Mikhail Galkin, Songyang Gao, Ziteng Gao, Silin Gao, Sara Ghazanfari, Nathan Godey, Navita Goyal, Xinran Gu, Yuxian Gu, Yu Gu, Anchun Gui, Jiacheng Guo, Ashim Gupta, Paul Hagemann, Tianxing He, Zhengfu He, Juncai He, Leonhard Hennig, Konstantin Hess, Jennifer Hu, Xiaoyang Hu, Zhilei Hu, Weidong Huang, Yichong Huang, Ayyoob Imani, Qi Jia, Yifan Jiang, Hanwen Jiang, Yiding Jiang, Yang Jin, Youngjin Jin, Zhijing Jin, Emmeran Johnson, Josef Jon, David Jurgens, Ehsan Kamalloo, Junmo Kang, Jian Kang, Mikhail Khodak, Hyunjae Kim, Soroush Abbasi Koohpayegani, Suhas Kotha, Jeongyeol Kwon, Sunjae Kwon, Philippe Laban, Zhibin Lan, Nayoung Lee, Deokjae Lee, Celine Lee, Heejun Lee, Jie Lei, Wenhao Li, Yafu Li, Yufei Li, Yanzeng Li, Yanzhou Li, Ziqiang Li, Zhaoyi Li, Ziheng Li, Xiaonan Li, Yinghao Li, Yu Li, Chengrui Li, Yingjie Li, Yunlong Liang, Baohao Liao, Kezhou Lin, Licong Lin, Enrico Liscio, Xiangyan Liu, Chenzhengyi Liu, Yixin Liu, Xingbin Liu, Haolin Liu, Xiao Liu, Yajiao Liu, Meng Liu, Tianyang Liu, Wei Liu, Qingyu Lu, Pan Lu, Junyu Lu, Zhengyi Luo, Yang Luo, Ang Lv, Junhyung Lyle, Jiajun Ma, Kaixin Ma, Ziqiao Ma, Mounica Maddela, Chaitanya Malaviya, Zhiyu Mei, Ethan Mendes, Fatemehsadat Mireshghallah, Niloofar Mireshghallah, Mircea Mironenco, Takeru Miyato, Fengran Mo, Xinyi Mou, Niklas Muennighoff, Cheolwon Na, Piotr Nawrot, Mang Ning, Longshen Ou, Siqi Ouyang, Lorenzo Pacchiardi, Ziqi Pang, Sara Papi, Letitia Parcalabescu, Tanmay Parekh, Aleksandar Petrov, Lucía Pitarch, Moritz Plenz, Manish Prajapat, Joan Puigcerver, Valentina Pyatkin, Shuofei Qiao, Yujia Qin, Chengwei Qin, Sigal Raab, Hossein A Rahmani, Siyu Ren, Yubing Ren, Ruiyang Ren, Yangjun Ruan, Michael J Ryan, Shoumik Saha, Vageesh Saxena, Michael Saxon, Alexander Scarlatos, Agam Shah, Erfan Shayegani, Behzad Shayegh, Xiangqing Shen, Sheng Shen, Ruizhe Shi, Zhengliang Shi, Kensen Shi, Ziyi Shou, Prasann Singhal, Jasivan Alex Sivakumar, Junru Song, Chunjin Song, Nikita Srivatsan, Michal Štefánik, Hao Sun, Mingjie Sun, Weiwei Sun, Zhiqing Sun, Xiaohang Tang, Liyan Tang, Eshaan Tanwar, Jiayan Teng, Davide Testa, Changyao Tian, Yufei Tian, Eric Todd, Benjamin Towle, Austin Tripp, Yi Tu, Rheeya Uppaal, Lazar Valkov, Neeraj Varshney, Artem Vazhentsev, Yiming Wang, Qifan Wang, Zhaoyang Wang, Lirui Wang, Zhicheng Wang, Weiqi Wang, Jiaan Wang, Boshi Wang, Haiming Wang, Huimin Wang, Yun-Cheng Wang, Runzhe Wang, Yu Wang, Yidong Wang, Licheng Wen, Te-Lin Wu, Yu-Yu Wu, Qianhui Wu, Dongming Wu, Tong Wu, Zijun Wu, Mengzhou Xia, Jian Xie, Yiming Xie, Weiwen Xu, Yi Xu, Xilie Xu, Derek Xu, Shohei Yamasaki, Hao Yan, Chenghao Yang, Xianjun Yang, Sen Yang, Bingsheng Yao, Qinyuan Ye, Fan Yin, Haneul Yoo, Kiyoon Yoo, Xinyan Velocity Yu, Jianfei Yu, Qiying Yu, Mo Yu, Zichun Yu, Yue Yu, Youliang Yuan, Zihao Yue, Xiang Yue, Yuanwen Yue, Daoguang Zan, Zhiyuan Zeng, Guangtao Zeng, Yuheng Zha, Runzhe Zhan, Jiaxu Zhang, Zhexin Zhang, Chen Zhang, Xinlu Zhang, Yabo Zhang, Renrui Zhang, Kechi Zhang, Ruoyu Zhang, Feng Zhang, Siyan Zhao, Junhao Zheng, Wenjie Zheng, Ming Zhong, Yan Zhou, Pei Zhou, Yangqiaoyu Zhou, Aojun Zhou, Xuekai Zhu, Luyao Zhu, Yanqiao Zhu, Dele Zhu, Andrew Zhu, Wenjie Zhuo and Caleb Ziems for contributing author-written questions about their ACL 2023 and/or ICLR 2024 papers.</p>
</div>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Annotation Details</h2>

<div id="A2.p1" class="ltx_para">
<p id="A2.p1.1" class="ltx_p">We provide instructions regarding manually inspecting questions in Table <a href="#S2.T1" title="Table 1 ‣ Sampling inline citation mentions. ‣ 2.1 Inline-citation Questions ‣ 2 LitSearch ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.
We sent out emails and Google Forms to recruit ACL 2023 and ICLR 2024 authors for author-written questions, and
the templates can be found in Table <a href="#A5.T10" title="Table 10 ‣ Appendix E Prompts and Additional Statistics ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">10</span></a> and Table <a href="#A5.T11" title="Table 11 ‣ Appendix E Prompts and Additional Statistics ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a> respectively.</p>
</div>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Retrieval Corpus</h2>

<div id="A3.p1" class="ltx_para">
<p id="A3.p1.1" class="ltx_p">The LitSearch retrieval corpus <math id="A3.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{P}" display="inline"><semantics id="A3.p1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="A3.p1.1.m1.1.1" xref="A3.p1.1.m1.1.1.cmml">𝒫</mi><annotation-xml encoding="MathML-Content" id="A3.p1.1.m1.1b"><ci id="A3.p1.1.m1.1.1.cmml" xref="A3.p1.1.m1.1.1">𝒫</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.p1.1.m1.1c">\mathcal{P}</annotation></semantics></math> consists of ACL Anthology and ICLR papers extracted from
S2ORC. Here we describe how we identify those papers in S2ORC:
We isolate ACL anthology papers from S2ORC by identifying entries whose metadata includes an ACL Anthology ID. We identify ICLR papers utilizing a combination of the venue-based queries to Semantic Scholar’s Academic Graph API
and by title-matching using titles of accepted papers scraped from the official ICLR website.</p>
</div>
</section>
<section id="A4" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>Retriever details</h2>

<div id="A4.p1" class="ltx_para">
<p id="A4.p1.1" class="ltx_p">We list the full HuggingFace checkpoint paths corresponding to the dense retrievers we use in <a href="#A4.T9" title="In Appendix D Retriever details ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">9</span></a>.
We use the following instructions for the instruction-finetuned embedding models: “<span id="A4.p1.1.1" class="ltx_text ltx_font_typewriter">Represent the research question for retrieving relevant research paper abstracts:</span>” for encoding queries; “<span id="A4.p1.1.2" class="ltx_text ltx_font_typewriter">Represent the title and abstract of the research paper for retrieval:</span>” for encoding papers when using Instructor-XL for retrieval using paper titles and abstracts; when performing retrieval using paper titles and abstacts with GritLM-7B, we use the instruction “<span id="A4.p1.1.3" class="ltx_text ltx_font_typewriter">Given a research query, retrieve the title and abstract of the relevant research paper</span>”.</p>
</div>
<figure id="A4.T9" class="ltx_table">
<table id="A4.T9.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A4.T9.1.1.1" class="ltx_tr">
<th id="A4.T9.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A4.T9.1.1.1.1.1" class="ltx_text ltx_font_bold">Retriever</span></th>
<th id="A4.T9.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A4.T9.1.1.1.2.1" class="ltx_text ltx_font_bold">HuggingFace Checkpoint</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A4.T9.1.2.1" class="ltx_tr">
<td id="A4.T9.1.2.1.1" class="ltx_td ltx_align_center ltx_border_t">GTR-T5-large</td>
<td id="A4.T9.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t"><span id="A4.T9.1.2.1.2.1" class="ltx_text ltx_font_typewriter">sentence-transformers/gtr-t5-large</span></td>
</tr>
<tr id="A4.T9.1.3.2" class="ltx_tr">
<td id="A4.T9.1.3.2.1" class="ltx_td ltx_align_center">Instructor-XL</td>
<td id="A4.T9.1.3.2.2" class="ltx_td ltx_align_center"><span id="A4.T9.1.3.2.2.1" class="ltx_text ltx_font_typewriter">hkunlp/instructor-xl</span></td>
</tr>
<tr id="A4.T9.1.4.3" class="ltx_tr">
<td id="A4.T9.1.4.3.1" class="ltx_td ltx_align_center">E5-large-v2</td>
<td id="A4.T9.1.4.3.2" class="ltx_td ltx_align_center"><span id="A4.T9.1.4.3.2.1" class="ltx_text ltx_font_typewriter">intfloat/e5-large-v2</span></td>
</tr>
<tr id="A4.T9.1.5.4" class="ltx_tr">
<td id="A4.T9.1.5.4.1" class="ltx_td ltx_align_center ltx_border_bb">GritLM-7B</td>
<td id="A4.T9.1.5.4.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="A4.T9.1.5.4.2.1" class="ltx_text ltx_font_typewriter">GritLM/GritLM-7B</span></td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 9: </span>HuggingFace checkpoints we use for each dense retriever.</figcaption>
</figure>
</section>
<section id="A5" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix E </span>Prompts and Additional Statistics</h2>

<div id="A5.p1" class="ltx_para">
<p id="A5.p1.1" class="ltx_p"><a href="#A5.T12" title="In Appendix E Prompts and Additional Statistics ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">12</span></a> shows the prompt we use for generating inline-citation questions via GPT-4.
<a href="#A5.T13" title="In Appendix E Prompts and Additional Statistics ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">13</span></a> shows the reranking prompt for GPT-4.
<a href="#A5.T14" title="In Appendix E Prompts and Additional Statistics ‣ Citation Benchmark" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">14</span></a> shows a more detailed statistics about LitSearch.</p>
</div>
<figure id="A5.T10" class="ltx_table">
<table id="A5.T10.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A5.T10.1.1.1" class="ltx_tr">
<td id="A5.T10.1.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="A5.T10.1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T10.1.1.1.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T10.1.1.1.1.1.1.1" class="ltx_text ltx_font_typewriter">
Hi {{annotator name}},</span></span>
</span>
</td>
</tr>
<tr id="A5.T10.1.2.2" class="ltx_tr">
<td id="A5.T10.1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T10.1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T10.1.2.2.1.1.1" class="ltx_p" style="width:424.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T10.1.3.3" class="ltx_tr">
<td id="A5.T10.1.3.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T10.1.3.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T10.1.3.3.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T10.1.3.3.1.1.1.1" class="ltx_text ltx_font_typewriter">We hope this email finds you well!</span></span>
</span>
</td>
</tr>
<tr id="A5.T10.1.4.4" class="ltx_tr">
<td id="A5.T10.1.4.4.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T10.1.4.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T10.1.4.4.1.1.1" class="ltx_p" style="width:424.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T10.1.5.5" class="ltx_tr">
<td id="A5.T10.1.5.5.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T10.1.5.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T10.1.5.5.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T10.1.5.5.1.1.1.1" class="ltx_text ltx_font_typewriter">First, congrats on your paper’s acceptance to {{conference name}}! We are [REDACTED] from [REDACTED] who are working on constructing a new challenging retrieval benchmark where the task is to retrieve relevant research papers given a research query. Would you be willing to dedicate 2 minutes to write a literature-search question about your {{conference name}} paper? Here’s the link to the google form: {{link}}.</span></span>
</span>
</td>
</tr>
<tr id="A5.T10.1.6.6" class="ltx_tr">
<td id="A5.T10.1.6.6.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T10.1.6.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T10.1.6.6.1.1.1" class="ltx_p" style="width:424.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T10.1.7.7" class="ltx_tr">
<td id="A5.T10.1.7.7.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T10.1.7.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T10.1.7.7.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T10.1.7.7.1.1.1.1" class="ltx_text ltx_font_typewriter">Your contribution will help us build better, more challenging evaluations for large language models. We will make sure to list you as a contributor to our benchmark (unless you prefer otherwise). Thank you!</span></span>
</span>
</td>
</tr>
<tr id="A5.T10.1.8.8" class="ltx_tr">
<td id="A5.T10.1.8.8.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T10.1.8.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T10.1.8.8.1.1.1" class="ltx_p" style="width:424.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T10.1.9.9" class="ltx_tr">
<td id="A5.T10.1.9.9.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T10.1.9.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T10.1.9.9.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T10.1.9.9.1.1.1.1" class="ltx_text ltx_font_typewriter">Best,</span></span>
</span>
</td>
</tr>
<tr id="A5.T10.1.10.10" class="ltx_tr">
<td id="A5.T10.1.10.10.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T10.1.10.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T10.1.10.10.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T10.1.10.10.1.1.1.1" class="ltx_text ltx_font_typewriter">{{author 1}}</span></span>
</span>
</td>
</tr>
<tr id="A5.T10.1.11.11" class="ltx_tr">
<td id="A5.T10.1.11.11.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A5.T10.1.11.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T10.1.11.11.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T10.1.11.11.1.1.1.1" class="ltx_text ltx_font_typewriter">{{author 2}}</span></span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 10: </span>
Email template sent out to ICLR 2024 and ACL 2023 authors for collecting author-written questions.
</figcaption>
</figure>
<figure id="A5.T11" class="ltx_table">
<table id="A5.T11.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A5.T11.1.1.1" class="ltx_tr">
<td id="A5.T11.1.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="A5.T11.1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.1.1.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T11.1.1.1.1.1.1.1" class="ltx_text ltx_font_typewriter"></span></span>
<span id="A5.T11.1.1.1.1.1.2" class="ltx_p"><span id="A5.T11.1.1.1.1.1.2.1" class="ltx_text ltx_font_typewriter">Dear {{author name}},</span></span>
</span>
</td>
</tr>
<tr id="A5.T11.1.2.2" class="ltx_tr">
<td id="A5.T11.1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T11.1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.2.2.1.1.1" class="ltx_p" style="width:424.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T11.1.3.3" class="ltx_tr">
<td id="A5.T11.1.3.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T11.1.3.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.3.3.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T11.1.3.3.1.1.1.1" class="ltx_text ltx_font_typewriter">Thanks for contributing to our literature-search question collection effort!</span></span>
</span>
</td>
</tr>
<tr id="A5.T11.1.4.4" class="ltx_tr">
<td id="A5.T11.1.4.4.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T11.1.4.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.4.4.1.1.1" class="ltx_p" style="width:424.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T11.1.5.5" class="ltx_tr">
<td id="A5.T11.1.5.5.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T11.1.5.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.5.5.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T11.1.5.5.1.1.1.1" class="ltx_text ltx_font_typewriter">We are trying to create a new challenging retrieval benchmark where the task is to retrieve relevant research papers given a research query. For example, the research query "Which paper first found that large language models can do in-context learning?" should retrieve the GPT-3 paper. We could use your help for creating such a query that should be answered by your own paper!</span></span>
</span>
</td>
</tr>
<tr id="A5.T11.1.6.6" class="ltx_tr">
<td id="A5.T11.1.6.6.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T11.1.6.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.6.6.1.1.1" class="ltx_p" style="width:424.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T11.1.7.7" class="ltx_tr">
<td id="A5.T11.1.7.7.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T11.1.7.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.7.7.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T11.1.7.7.1.1.1.1" class="ltx_text ltx_font_typewriter">Could you provide one high-quality, literature-search-type query about your paper that you think would be challenging even for state-of-the-art retrieval systems? Please make sure to follow these guidelines when writing your query. Your query must be</span></span>
</span>
</td>
</tr>
<tr id="A5.T11.1.8.8" class="ltx_tr">
<td id="A5.T11.1.8.8.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T11.1.8.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.8.8.1.1.1" class="ltx_p" style="width:424.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T11.1.9.9" class="ltx_tr">
<td id="A5.T11.1.9.9.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T11.1.9.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.9.9.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T11.1.9.9.1.1.1.1" class="ltx_text ltx_font_typewriter">(1) Realistic: It should be plausible that a researcher working in a related field may ask this exact question. Do not ask a question that puts too many unrealistic constraints (such as "What work that uses the NaturalQuestions dataset trains for 15 epochs and uses a learning rate of 3e-5?").</span></span>
</span>
</td>
</tr>
<tr id="A5.T11.1.10.10" class="ltx_tr">
<td id="A5.T11.1.10.10.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T11.1.10.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.10.10.1.1.1" class="ltx_p" style="width:424.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T11.1.11.11" class="ltx_tr">
<td id="A5.T11.1.11.11.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T11.1.11.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.11.11.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T11.1.11.11.1.1.1.1" class="ltx_text ltx_font_typewriter">(2) Specific: Your query should be answered by/correspond to one particular (representative) paper or a small number of papers. Do not ask an overly broad question like "What are some multimodal models?" Instead, you can ask "Which multimodal model was the first to use interleaved image and text data?"</span></span>
</span>
</td>
</tr>
<tr id="A5.T11.1.12.12" class="ltx_tr">
<td id="A5.T11.1.12.12.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T11.1.12.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.12.12.1.1.1" class="ltx_p" style="width:424.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T11.1.13.13" class="ltx_tr">
<td id="A5.T11.1.13.13.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T11.1.13.13.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.13.13.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T11.1.13.13.1.1.1.1" class="ltx_text ltx_font_typewriter">(3) Challenging: Our goal is to collect a set of queries that are extremely challenging even for SOTA retrievers today. Make sure not to submit a query that can easily be answered via keyword matching or a Google search like "Which paper proposed masked language modeling?" (BERT). Instead, you can use the following formats to make the question more challenging:</span></span>
</span>
</td>
</tr>
<tr id="A5.T11.1.14.14" class="ltx_tr">
<td id="A5.T11.1.14.14.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T11.1.14.14.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.14.14.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T11.1.14.14.1.1.1.1" class="ltx_text ltx_font_typewriter">(a) Asking a detailed question that the abstract does not cover: "Which text pre-training paper first used the data mixture of wikipedia and bookscorpus?" (BERT)</span></span>
</span>
</td>
</tr>
<tr id="A5.T11.1.15.15" class="ltx_tr">
<td id="A5.T11.1.15.15.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T11.1.15.15.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.15.15.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T11.1.15.15.1.1.1.1" class="ltx_text ltx_font_typewriter">(b) Rephrasing (reducing word overlap to the paper): "Is there such a machine learning dataset, where for some questions, there is no correct answer and model should abstain?" (SQuAD v2)</span></span>
</span>
</td>
</tr>
<tr id="A5.T11.1.16.16" class="ltx_tr">
<td id="A5.T11.1.16.16.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T11.1.16.16.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.16.16.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T11.1.16.16.1.1.1.1" class="ltx_text ltx_font_typewriter">(c) Asking about the main technique innovation: "Is there any paper that combines distillation and structured pruning for language models?" (CoFi pruning)</span></span>
</span>
</td>
</tr>
<tr id="A5.T11.1.17.17" class="ltx_tr">
<td id="A5.T11.1.17.17.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T11.1.17.17.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.17.17.1.1.1" class="ltx_p" style="width:424.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T11.1.18.18" class="ltx_tr">
<td id="A5.T11.1.18.18.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T11.1.18.18.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.18.18.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T11.1.18.18.1.1.1.1" class="ltx_text ltx_font_typewriter">Thank you for your contribution! We will make sure that:</span></span>
</span>
</td>
</tr>
<tr id="A5.T11.1.19.19" class="ltx_tr">
<td id="A5.T11.1.19.19.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T11.1.19.19.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.19.19.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T11.1.19.19.1.1.1.1" class="ltx_text ltx_font_typewriter">- Your name will be credited in our paper unless you choose otherwise.</span></span>
</span>
</td>
</tr>
<tr id="A5.T11.1.20.20" class="ltx_tr">
<td id="A5.T11.1.20.20.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A5.T11.1.20.20.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T11.1.20.20.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T11.1.20.20.1.1.1.1" class="ltx_text ltx_font_typewriter">- We will make the dataset available for open-source development.</span></span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 11: </span>
Instructions provided in the Google Forms sent to ICLR 2024 and ACL 2023 authors for collecting author-written questions.
</figcaption>
</figure>
<figure id="A5.T12" class="ltx_table">
<table id="A5.T12.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A5.T12.1.1.1" class="ltx_tr">
<td id="A5.T12.1.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="A5.T12.1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.1.1.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T12.1.1.1.1.1.1.1" class="ltx_text ltx_font_typewriter">
I will provide you with a excerpt from an scientific article that cites various papers in the positions {cite_001}, {cite_002}, etc. I would like you to write *general* questions about the academic literature using this paragraph, which will be answered with the corresponding citations. Here are the constraints you must follow:</span></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.2.2" class="ltx_tr">
<td id="A5.T12.1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.2.2.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T12.1.2.2.1.1.1.1" class="ltx_text ltx_font_typewriter">(1) The questions should make sense and be interesting without reading the paragraph. The questions should be general, and you should only use the paragraph to find the ground-truth answer.</span></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.3.3" class="ltx_tr">
<td id="A5.T12.1.3.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.3.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.3.3.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T12.1.3.3.1.1.1.1" class="ltx_text ltx_font_typewriter">(2) Try to ask questions which cover as many citations as possible and are as detailed as possible. Pack as much information into the question as you can.</span></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.4.4" class="ltx_tr">
<td id="A5.T12.1.4.4.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.4.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.4.4.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T12.1.4.4.1.1.1.1" class="ltx_text ltx_font_typewriter">(3) If the context is not clear, skip the citation. Make sure that every question is of the highest quality, as these questions will be used for important work concerning information extraction from the scientific literature.</span></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.5.5" class="ltx_tr">
<td id="A5.T12.1.5.5.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.5.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.5.5.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T12.1.5.5.1.1.1.1" class="ltx_text ltx_font_typewriter">(4) The answer should ONLY contain the citation key.</span></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.6.6" class="ltx_tr">
<td id="A5.T12.1.6.6.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.6.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.6.6.1.1.1" class="ltx_p" style="width:424.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.7.7" class="ltx_tr">
<td id="A5.T12.1.7.7.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.7.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.7.7.1.1.1" class="ltx_p" style="width:424.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.8.8" class="ltx_tr">
<td id="A5.T12.1.8.8.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.8.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.8.8.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T12.1.8.8.1.1.1.1" class="ltx_text ltx_font_typewriter">TEXT: We argue there are two underlying motivations for the ad text generation task, especially for product descriptions. Application-wise, the utility is to improve the seller experience for e-commerce services when registering a new product. The generated descriptions can reduce the need for manual data entry, and potentially improve sales due to better descriptions (in terms of attractiveness, structure, and persuasiveness). Research-wise, ad text generation is an under-studied task, and arguably a good proxy for persuasive text generation {cite_016}{cite_017}{cite_018}{cite_019}.</span></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.9.9" class="ltx_tr">
<td id="A5.T12.1.9.9.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.9.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.9.9.1.1.1" class="ltx_p" style="width:424.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.10.10" class="ltx_tr">
<td id="A5.T12.1.10.10.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.10.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.10.10.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T12.1.10.10.1.1.1.1" class="ltx_text ltx_font_typewriter">PAPER TITLES:</span></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.11.11" class="ltx_tr">
<td id="A5.T12.1.11.11.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.11.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.11.11.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T12.1.11.11.1.1.1.1" class="ltx_text ltx_font_typewriter">- cite_016: Is this post persuasive? ranking argumentative comments in online forum</span></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.12.12" class="ltx_tr">
<td id="A5.T12.1.12.12.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.12.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.12.12.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T12.1.12.12.1.1.1.1" class="ltx_text ltx_font_typewriter">- cite_017: On the role of discourse relations in persuasive texts</span></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.13.13" class="ltx_tr">
<td id="A5.T12.1.13.13.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.13.13.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.13.13.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T12.1.13.13.1.1.1.1" class="ltx_text ltx_font_typewriter">- cite_018: Measuring online debaters’ persuasive skill from text over time</span></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.14.14" class="ltx_tr">
<td id="A5.T12.1.14.14.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.14.14.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.14.14.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T12.1.14.14.1.1.1.1" class="ltx_text ltx_font_typewriter">- cite_019: Analyzing the Persuasive Effect of Style in News Editorial Argumentation</span></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.15.15" class="ltx_tr">
<td id="A5.T12.1.15.15.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.15.15.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.15.15.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T12.1.15.15.1.1.1.1" class="ltx_text ltx_font_typewriter">- cite_020: Persuaide! an adaptive persuasive text generation system for fashion domain</span></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.16.16" class="ltx_tr">
<td id="A5.T12.1.16.16.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.16.16.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.16.16.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T12.1.16.16.1.1.1.1" class="ltx_text ltx_font_typewriter">- cite_021: A statistical framework for product description generation</span></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.17.17" class="ltx_tr">
<td id="A5.T12.1.17.17.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.17.17.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.17.17.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T12.1.17.17.1.1.1.1" class="ltx_text ltx_font_typewriter">- cite_022: SILVER: Generating persuasive Chinese product pitch</span></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.18.18" class="ltx_tr">
<td id="A5.T12.1.18.18.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.18.18.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.18.18.1.1.1" class="ltx_p" style="width:424.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.19.19" class="ltx_tr">
<td id="A5.T12.1.19.19.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.19.19.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.19.19.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T12.1.19.19.1.1.1.1" class="ltx_text ltx_font_typewriter">QUESTION: I want to read some papers that try to study and quantify how persuasive text can be. I’m coming at this from an applications-perspective, as I’m interested in using these insights for product development. Could you give me a list of papers to read?</span></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.20.20" class="ltx_tr">
<td id="A5.T12.1.20.20.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.20.20.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.20.20.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T12.1.20.20.1.1.1.1" class="ltx_text ltx_font_typewriter">ANSWER: cite_016, cite_017, cite_018, cite_019</span></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.21.21" class="ltx_tr">
<td id="A5.T12.1.21.21.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.21.21.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.21.21.1.1.1" class="ltx_p" style="width:424.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.22.22" class="ltx_tr">
<td id="A5.T12.1.22.22.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.22.22.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.22.22.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T12.1.22.22.1.1.1.1" class="ltx_text ltx_font_typewriter">QUESTION: Has anyone looked at automating advertisement text generation specifically for fashion items?</span></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.23.23" class="ltx_tr">
<td id="A5.T12.1.23.23.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.23.23.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.23.23.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T12.1.23.23.1.1.1.1" class="ltx_text ltx_font_typewriter">ANSWER: cite_020</span></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.24.24" class="ltx_tr">
<td id="A5.T12.1.24.24.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.24.24.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.24.24.1.1.1" class="ltx_p" style="width:424.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.25.25" class="ltx_tr">
<td id="A5.T12.1.25.25.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.25.25.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.25.25.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T12.1.25.25.1.1.1.1" class="ltx_text ltx_font_typewriter">QUESTION: I’m thinking about going into the computer-retail business and I’m wondering if it’s possible to generate persuasive text to sell more computers?</span></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.26.26" class="ltx_tr">
<td id="A5.T12.1.26.26.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.26.26.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.26.26.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T12.1.26.26.1.1.1.1" class="ltx_text ltx_font_typewriter">ANSWER: cite_021</span></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.27.27" class="ltx_tr">
<td id="A5.T12.1.27.27.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.27.27.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.27.27.1.1.1" class="ltx_p" style="width:424.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.28.28" class="ltx_tr">
<td id="A5.T12.1.28.28.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.28.28.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.28.28.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T12.1.28.28.1.1.1.1" class="ltx_text ltx_font_typewriter">QUESTION: Could you recommend some readings for articles that generate persuasive text in Chinese aimed at advertising?</span></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.29.29" class="ltx_tr">
<td id="A5.T12.1.29.29.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.29.29.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.29.29.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T12.1.29.29.1.1.1.1" class="ltx_text ltx_font_typewriter">ANSWER: cite_022</span></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.30.30" class="ltx_tr">
<td id="A5.T12.1.30.30.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.30.30.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.30.30.1.1.1" class="ltx_p" style="width:424.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.31.31" class="ltx_tr">
<td id="A5.T12.1.31.31.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.31.31.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.31.31.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T12.1.31.31.1.1.1.1" class="ltx_text ltx_font_bold">(Additional Demonstration Omitted)</span></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.32.32" class="ltx_tr">
<td id="A5.T12.1.32.32.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.32.32.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.32.32.1.1.1" class="ltx_p" style="width:424.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.33.33" class="ltx_tr">
<td id="A5.T12.1.33.33.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.33.33.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.33.33.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T12.1.33.33.1.1.1.1" class="ltx_text ltx_font_typewriter">TEXT: {{PARAGRAPH}}</span></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.34.34" class="ltx_tr">
<td id="A5.T12.1.34.34.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.34.34.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.34.34.1.1.1" class="ltx_p" style="width:424.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.35.35" class="ltx_tr">
<td id="A5.T12.1.35.35.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T12.1.35.35.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.35.35.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T12.1.35.35.1.1.1.1" class="ltx_text ltx_font_typewriter">PAPER TITLES:</span></span>
</span>
</td>
</tr>
<tr id="A5.T12.1.36.36" class="ltx_tr">
<td id="A5.T12.1.36.36.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A5.T12.1.36.36.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T12.1.36.36.1.1.1" class="ltx_p" style="width:424.9pt;"><span id="A5.T12.1.36.36.1.1.1.1" class="ltx_text ltx_font_typewriter">{{CITATIONS}}</span></span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 12: </span>
The prompt used for generating questions from inline citations using GPT-4.
</figcaption>
</figure>
<figure id="A5.T13" class="ltx_table">
<table id="A5.T13.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A5.T13.1.1.1" class="ltx_tr">
<td id="A5.T13.1.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="A5.T13.1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.1.1.1.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A5.T13.1.1.1.1.1.1.1" class="ltx_text ltx_font_typewriter"></span></span>
<span id="A5.T13.1.1.1.1.1.2" class="ltx_p"><span id="A5.T13.1.1.1.1.1.2.1" class="ltx_text ltx_font_bold">(system)</span><span id="A5.T13.1.1.1.1.1.2.2" class="ltx_text ltx_font_typewriter"> You are RankGPT, an intelligent assistant that can rank papers based on their relevancy to a research query.</span></span>
</span>
</td>
</tr>
<tr id="A5.T13.1.2.2" class="ltx_tr">
<td id="A5.T13.1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T13.1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.1.2.2.1.1.1" class="ltx_p" style="width:411.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T13.1.3.3" class="ltx_tr">
<td id="A5.T13.1.3.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T13.1.3.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.1.3.3.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A5.T13.1.3.3.1.1.1.1" class="ltx_text ltx_font_bold">(user)</span><span id="A5.T13.1.3.3.1.1.1.2" class="ltx_text ltx_align_left ltx_font_typewriter"> I will provide you with the abstracts of 100 papers, each indicated by number identifier []. \nRank the papers based on their relevance to research question: {{query}}.</span></span>
</span>
</td>
</tr>
<tr id="A5.T13.1.4.4" class="ltx_tr">
<td id="A5.T13.1.4.4.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T13.1.4.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.1.4.4.1.1.1" class="ltx_p" style="width:411.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T13.1.5.5" class="ltx_tr">
<td id="A5.T13.1.5.5.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T13.1.5.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.1.5.5.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A5.T13.1.5.5.1.1.1.1" class="ltx_text ltx_font_bold">(assistant)</span><span id="A5.T13.1.5.5.1.1.1.2" class="ltx_text ltx_align_left ltx_font_typewriter"> Okay, please provide the papers.</span></span>
</span>
</td>
</tr>
<tr id="A5.T13.1.6.6" class="ltx_tr">
<td id="A5.T13.1.6.6.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T13.1.6.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.1.6.6.1.1.1" class="ltx_p" style="width:411.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T13.1.7.7" class="ltx_tr">
<td id="A5.T13.1.7.7.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T13.1.7.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.1.7.7.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A5.T13.1.7.7.1.1.1.1" class="ltx_text ltx_font_bold">(user)</span><span id="A5.T13.1.7.7.1.1.1.2" class="ltx_text ltx_align_left ltx_font_typewriter"> [1] Title: {{Title1}}\n Abstract: {{Abstract1}}</span></span>
</span>
</td>
</tr>
<tr id="A5.T13.1.8.8" class="ltx_tr">
<td id="A5.T13.1.8.8.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T13.1.8.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.1.8.8.1.1.1" class="ltx_p" style="width:411.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T13.1.9.9" class="ltx_tr">
<td id="A5.T13.1.9.9.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T13.1.9.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.1.9.9.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A5.T13.1.9.9.1.1.1.1" class="ltx_text ltx_font_bold">(assistant)</span><span id="A5.T13.1.9.9.1.1.1.2" class="ltx_text ltx_align_left ltx_font_typewriter"> Received passage [1].</span></span>
</span>
</td>
</tr>
<tr id="A5.T13.1.10.10" class="ltx_tr">
<td id="A5.T13.1.10.10.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T13.1.10.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.1.10.10.1.1.1" class="ltx_p" style="width:411.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T13.1.11.11" class="ltx_tr">
<td id="A5.T13.1.11.11.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T13.1.11.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.1.11.11.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A5.T13.1.11.11.1.1.1.1" class="ltx_text ltx_font_bold">(user)</span><span id="A5.T13.1.11.11.1.1.1.2" class="ltx_text ltx_align_left ltx_font_typewriter"> [2] Title: {{Title2}}\n Abstract: {{Abstract2}}</span></span>
</span>
</td>
</tr>
<tr id="A5.T13.1.12.12" class="ltx_tr">
<td id="A5.T13.1.12.12.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T13.1.12.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.1.12.12.1.1.1" class="ltx_p" style="width:411.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T13.1.13.13" class="ltx_tr">
<td id="A5.T13.1.13.13.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T13.1.13.13.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.1.13.13.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A5.T13.1.13.13.1.1.1.1" class="ltx_text ltx_font_bold">(assistant)</span><span id="A5.T13.1.13.13.1.1.1.2" class="ltx_text ltx_align_left ltx_font_typewriter"> Received passage [2].</span></span>
</span>
</td>
</tr>
<tr id="A5.T13.1.14.14" class="ltx_tr">
<td id="A5.T13.1.14.14.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T13.1.14.14.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.1.14.14.1.1.1" class="ltx_p" style="width:411.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T13.1.15.15" class="ltx_tr">
<td id="A5.T13.1.15.15.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T13.1.15.15.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.1.15.15.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A5.T13.1.15.15.1.1.1.1" class="ltx_text ltx_font_typewriter">…</span></span>
</span>
</td>
</tr>
<tr id="A5.T13.1.16.16" class="ltx_tr">
<td id="A5.T13.1.16.16.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T13.1.16.16.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.1.16.16.1.1.1" class="ltx_p" style="width:411.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T13.1.17.17" class="ltx_tr">
<td id="A5.T13.1.17.17.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T13.1.17.17.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.1.17.17.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A5.T13.1.17.17.1.1.1.1" class="ltx_text ltx_font_bold">(user)</span><span id="A5.T13.1.17.17.1.1.1.2" class="ltx_text ltx_align_left ltx_font_typewriter"> [100] Title: {{Title100}}\n Abstract: {{Abstract100}}</span></span>
</span>
</td>
</tr>
<tr id="A5.T13.1.18.18" class="ltx_tr">
<td id="A5.T13.1.18.18.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T13.1.18.18.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.1.18.18.1.1.1" class="ltx_p" style="width:411.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T13.1.19.19" class="ltx_tr">
<td id="A5.T13.1.19.19.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T13.1.19.19.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.1.19.19.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A5.T13.1.19.19.1.1.1.1" class="ltx_text ltx_font_bold">(assistant)</span><span id="A5.T13.1.19.19.1.1.1.2" class="ltx_text ltx_align_left ltx_font_typewriter"> Received passage [100].</span></span>
</span>
</td>
</tr>
<tr id="A5.T13.1.20.20" class="ltx_tr">
<td id="A5.T13.1.20.20.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A5.T13.1.20.20.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.1.20.20.1.1.1" class="ltx_p" style="width:411.9pt;"></span>
</span>
</td>
</tr>
<tr id="A5.T13.1.21.21" class="ltx_tr">
<td id="A5.T13.1.21.21.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A5.T13.1.21.21.1.1" class="ltx_inline-block ltx_align_top">
<span id="A5.T13.1.21.21.1.1.1" class="ltx_p" style="width:411.9pt;"><span id="A5.T13.1.21.21.1.1.1.1" class="ltx_text ltx_font_bold">(user)</span><span id="A5.T13.1.21.21.1.1.1.2" class="ltx_text ltx_align_left ltx_font_typewriter ltx_align_center"> Search Query: {{query}}. \nRank the 100 papers above based on their relevance to the research query. The papers should be listed in descending order using identifiers. The most relevant papers should be listed first. The output format should be [] &gt; [], e.g., [1] &gt; [2]. Only respond with the ranking results, do not say any words or explain.</span></span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 13: </span>
The prompt used for reranking retrieved documents using GPT4 (adapted from <cite class="ltx_cite ltx_citemacro_citep">Sun et al., <a href="#bib.bib33" title="" class="ltx_ref">2023a</a></cite>). </figcaption>
</figure>
<figure id="A5.T14" class="ltx_table">
<table id="A5.T14.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A5.T14.1.1.1" class="ltx_tr">
<td id="A5.T14.1.1.1.1" class="ltx_td ltx_border_tt"></td>
<td id="A5.T14.1.1.1.2" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="A5.T14.1.1.1.2.1" class="ltx_text ltx_font_bold">Broad</span></td>
<td id="A5.T14.1.1.1.3" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="A5.T14.1.1.1.3.1" class="ltx_text ltx_font_bold">Specific</span></td>
<td id="A5.T14.1.1.1.4" class="ltx_td ltx_align_center ltx_border_tt" rowspan="2"><span id="A5.T14.1.1.1.4.1" class="ltx_text ltx_font_bold">Total #Q</span></td>
</tr>
<tr id="A5.T14.1.2.2" class="ltx_tr">
<td id="A5.T14.1.2.2.1" class="ltx_td"></td>
<td id="A5.T14.1.2.2.2" class="ltx_td ltx_align_center ltx_border_t">#Q</td>
<td id="A5.T14.1.2.2.3" class="ltx_td ltx_align_center ltx_border_t">Avg. L</td>
<td id="A5.T14.1.2.2.4" class="ltx_td ltx_align_center ltx_border_t">Overlap</td>
<td id="A5.T14.1.2.2.5" class="ltx_td ltx_align_center ltx_border_t">#Q</td>
<td id="A5.T14.1.2.2.6" class="ltx_td ltx_align_center ltx_border_t">Avg. L</td>
<td id="A5.T14.1.2.2.7" class="ltx_td ltx_align_center ltx_border_t">Overlap</td>
</tr>
<tr id="A5.T14.1.3.3" class="ltx_tr">
<td id="A5.T14.1.3.3.1" class="ltx_td ltx_align_center ltx_border_t" colspan="8"><span id="A5.T14.1.3.3.1.1" class="ltx_text ltx_font_bold">Inline-Citation Questions</span></td>
</tr>
<tr id="A5.T14.1.4.4" class="ltx_tr">
<td id="A5.T14.1.4.4.1" class="ltx_td ltx_align_left ltx_border_t"><span id="A5.T14.1.4.4.1.1" class="ltx_text ltx_font_bold">ACL-sourced</span></td>
<td id="A5.T14.1.4.4.2" class="ltx_td ltx_align_center ltx_border_t">32</td>
<td id="A5.T14.1.4.4.3" class="ltx_td ltx_align_center ltx_border_t">24.8</td>
<td id="A5.T14.1.4.4.4" class="ltx_td ltx_align_center ltx_border_t">0.33</td>
<td id="A5.T14.1.4.4.5" class="ltx_td ltx_align_center ltx_border_t">66</td>
<td id="A5.T14.1.4.4.6" class="ltx_td ltx_align_center ltx_border_t">26.0</td>
<td id="A5.T14.1.4.4.7" class="ltx_td ltx_align_center ltx_border_t">0.35</td>
<td id="A5.T14.1.4.4.8" class="ltx_td ltx_align_center ltx_border_t">98</td>
</tr>
<tr id="A5.T14.1.5.5" class="ltx_tr">
<td id="A5.T14.1.5.5.1" class="ltx_td ltx_align_left"><span id="A5.T14.1.5.5.1.1" class="ltx_text ltx_font_bold">Non-ACL-sourced</span></td>
<td id="A5.T14.1.5.5.2" class="ltx_td ltx_align_center">88</td>
<td id="A5.T14.1.5.5.3" class="ltx_td ltx_align_center">19.1</td>
<td id="A5.T14.1.5.5.4" class="ltx_td ltx_align_center">0.33</td>
<td id="A5.T14.1.5.5.5" class="ltx_td ltx_align_center">165</td>
<td id="A5.T14.1.5.5.6" class="ltx_td ltx_align_center">20.5</td>
<td id="A5.T14.1.5.5.7" class="ltx_td ltx_align_center">0.34</td>
<td id="A5.T14.1.5.5.8" class="ltx_td ltx_align_center">253</td>
</tr>
<tr id="A5.T14.1.6.6" class="ltx_tr">
<td id="A5.T14.1.6.6.1" class="ltx_td ltx_align_center ltx_border_t" colspan="8"><span id="A5.T14.1.6.6.1.1" class="ltx_text ltx_font_bold">Author-written Questions</span></td>
</tr>
<tr id="A5.T14.1.7.7" class="ltx_tr">
<td id="A5.T14.1.7.7.1" class="ltx_td ltx_align_left ltx_border_t"><span id="A5.T14.1.7.7.1.1" class="ltx_text ltx_font_bold">ACL 2023</span></td>
<td id="A5.T14.1.7.7.2" class="ltx_td ltx_align_center ltx_border_t">25</td>
<td id="A5.T14.1.7.7.3" class="ltx_td ltx_align_center ltx_border_t">14.5</td>
<td id="A5.T14.1.7.7.4" class="ltx_td ltx_align_center ltx_border_t">0.41</td>
<td id="A5.T14.1.7.7.5" class="ltx_td ltx_align_center ltx_border_t">130</td>
<td id="A5.T14.1.7.7.6" class="ltx_td ltx_align_center ltx_border_t">18.1</td>
<td id="A5.T14.1.7.7.7" class="ltx_td ltx_align_center ltx_border_t">0.42</td>
<td id="A5.T14.1.7.7.8" class="ltx_td ltx_align_center ltx_border_t">155</td>
</tr>
<tr id="A5.T14.1.8.8" class="ltx_tr">
<td id="A5.T14.1.8.8.1" class="ltx_td ltx_align_left ltx_border_bb"><span id="A5.T14.1.8.8.1.1" class="ltx_text ltx_font_bold">ICLR 2024</span></td>
<td id="A5.T14.1.8.8.2" class="ltx_td ltx_align_center ltx_border_bb">10</td>
<td id="A5.T14.1.8.8.3" class="ltx_td ltx_align_center ltx_border_bb">19.0</td>
<td id="A5.T14.1.8.8.4" class="ltx_td ltx_align_center ltx_border_bb">0.49</td>
<td id="A5.T14.1.8.8.5" class="ltx_td ltx_align_center ltx_border_bb">81</td>
<td id="A5.T14.1.8.8.6" class="ltx_td ltx_align_center ltx_border_bb">17.6</td>
<td id="A5.T14.1.8.8.7" class="ltx_td ltx_align_center ltx_border_bb">0.45</td>
<td id="A5.T14.1.8.8.8" class="ltx_td ltx_align_center ltx_border_bb">91</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 14: </span>Detailed statistics for LitSearch.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2407.18938" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2407.18940" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2407.18940">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2407.18940" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2407.18941" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Mon Aug  5 18:14:51 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2407.18940
Literature search questions, such as “where can I find research on the evaluation of consistency in generated summaries?” pose significant challenges for modern search engines and retrieval systems. These questions often require a deep understanding of research concepts and the ability to reason over entire articles. In this work, we introduce LitSearch, a retrieval benchmark comprising 597 realistic literature search queries about recent ML and NLP papers. LitSearch is constructed using a combination of (1)  questions generated by GPT-4 based on paragraphs containing inline citations from research papers and
(2)  questions about recently published papers, manually written by their authors.
All LitSearch questions were manually examined or edited by experts to ensure high quality. We extensively benchmark state-of-the-art retrieval models and also evaluate two LLM-based reranking pipelines.
We find a significant performance gap between BM25 and state-of-the-art dense retrievers, with a 24.8% difference in absolute recall@5. The LLM-based reranking strategies further improve the best-performing dense retriever by 4.4%. Additionally, commercial search engines and research tools like Google Search perform poorly on LitSearch, lagging behind the best dense retriever by 32 points. Taken together, these results show that LitSearch is an informative new testbed for retrieval systems while catering to a real-world use case.111Our dataset and code are available athttps://github.com/princeton-nlp/LitSearch.
abstract
Citation Benchmark
Abstract
2407.18940
1IntroductionFinding literature via a specific search query—for example, to collect related work, to check if a method has been proposed before, or to recall a previously-seen paper—is a critical task for researchers.
Developing systems that recommend citations pertinent to such inquiries has the promise to enhance researchers’ productivity and expedite scientific discovery.
However, this task is inherently challenging as it often requires deep domain expertise and reasoning through lengthy papers.Figure 1:Examples ofinline-citation questionsandauthor-written questionsfrom LitSearch. The questions are often challenging and require deep understanding of target papers to answer correctly.Figure 2:The pipeline to generate inline-citation questions. We first sample a citation mention and prompt GPT-4 to write a question. Then we filter the questions by word overlap (with the target paper title)  and manual inspection. The manual inspection annotates the specificity and the quality of the questions (rubrics inTable1) .Prior to this study, the task of citation recommendation is
often formalized by
using inline citation mentions from existing papers as queries, and the cited papers as targets.
For instance, given the citation mention “RoBERTa and T5 are based on recent advances in masked language modeling[citation],” the text surrounding the citation mention is used as a retrieval query, and the cited paper is the target literature.
However, directly using inline citations often leads to queries that are noisy, broad (e.g., “Large Language Models[citation]”) , or context-dependent (e.g., “We follow the hyperparameters of[citation]”) .In this work, we propose a new litearture retrieval benchmark called LitSearch. As illustrated inFigure1, a literature search question seeks papers that meet specific criteria, closely reflecting actual research workflows.
LitSearch consists of two subsets:
(1)  Forinline-citation questions, we sample citation mentions from a collection of scientific papers and use GPT-4to rewrite them into literature search questions (Figure2) .
We retain questions with a low word overlap with the title of the target papers, and perform manual examination to ensure high quality.
(2)  Forauthor-written questions, we invited authors of ACL 2023 and ICLR 2024 papers to write literature search questions for their own papers.
This subset is also manually examined and filtered to remove any inaccurate or easy questions.
LitSearch contains 597 questions in total, each paired with one or more scientific papers as the ground truth.LitSearch has several unique characteristics:
(1)  To our best knowledge, LitSearch is thefirstdataset featuring realistic literature search questions, providing a new testbed for citation recommendation and retrieval systems.
(2)  LitSearch ischallenging, requiring deep understanding and reasoning over entire articles. The average document length (6,041/134 words for full text/titles and abstracts)  is significantly longer than most existing retrieval benchmarks (e.g., 56 for) .
(3)  LitSearch is ofhigh quality, with all questions manually examined by the authors of this work.We conduct extensive experiments
on both state-of-the-art retrieval models and
reranking with large language models (LLMs) .
On LitSearch, the best dense retrieval model, GritLM, achieves an average recall@5
of 74.8%, beating BM25by 24.8%.
The recall@5 of GritLM is further improved by 4.4% with GPT-4 reranking.
On the other hand, commercial search engines and research tools like Google Search perform poorly on this task, only achieving an average recall@5 of 42.8% at most. Futhermore, compared to existing retrieval datasets from BEIRand MTEB,
LitSearch effectively reflects the performance differences among various embedding models,
making it an informative testbed for evaluating state-of-the-art retrieval systems.
introduction
Citation Benchmark
1Introduction
2407.18940
2LitSearchOur benchmark LitSearch consists of
(a)  a large corpus of scientific papersand
(b)  pairs of literature search questions and one or more target papers from. Our desiderata are scientific questions that researchers may use while conducting literature surveys.
We use two different strategies to collect such questions: (1)  we construct questions using the surrounding context from inline citations in published papers (Section2.1) , and (2)  we invited the authors of recent conference publications to manually write questions about their own papers (Section2.2) . For both subsets, we ensure high question quality via manual inspection and filtering conducted by the authors of this work (Section2.3) .2.1Inline-citation QuestionsWe define the following concepts for the ease of description:
(a)  Aninline citation mentionis a paragraph from the main text of a paper that mentions another paper. For example, this paragraph from the RoBERTa paper, “… Unlike Devlin et al., (2019) , … we do not train with a reduced sequence length for the first 90% of updates …” mentions the BERT paper.
(b)  Thesource paperis the paper the inline citation mention is sampled from.
(c)  Atarget paperis a paper that is cited by the inline citation mention.Figure2provides an overview of our data collection methodology for inline-citation questions. We utilize the Semantic Scholar Open Research Corpus (S2ORC;) , a large corpus of academic papers obtained from publishers, archives, and the Internet. We randomly sample inline citation mentions
from the S2ORC222S2ORC provides detailed citation information for most inline citations (including the position of the citation in the paper text and the unique identifier of the target paper) , enabling us to easily sample inline citation mentions and match them to S2ORC papers. We used the 2024-03-26 version.and prompt
GPT-4 to rewrite these citation mentions into literature search questions.
These questions are filtered to remove those with a high word overlap with the title of the target papers, and are further manually examined to ensure high quality.Sampling inline citation mentions.We limit the target papers to be only from the ACL Anthology, for the purpose of aligning with the expertise of the manual annotators, i.e., authors of this work. However, we do not limit where the source papers come from. Depending on the source papers, we call
these questionsACL sourcedornon-ACL sourced.Specificity0Broad. There should exist no more than 20 papers that fit the question.Example: What are some parameter-efficient fine-tuning methods?1Specific. There should exist no more than 5 papers that fit the question.Example: Which method involves training additional prompt tokens for every layer during the fine-tuning of language models?Quality0Discarded. The question is factually wrong, unrealistic, overly broad/specific, or too easy.1Acceptable. The question can be somewhat out of distribution of what researchers ask, or relatively easy due to high overlap with the title/abstract.2Good. The question makes a challenging yet meaningful literature search question.Table 1:Annotation rubrics for the manual filtering (conducted by the authors of LitSearch) .Prompting GPT-4 to generate questions.Given a sampled citation mention,
we prompt GPT-4to generate a literature search question. In the prompt, we provide (1)  the sampled paragraph (the inline citation mention)  from the source paper and (2)  the titles of the cited papers, and instruct GPT-4 to generate a literature search question based on the paragraph that would be answered by one or more of the papers cited in the paragraph.
We use in-context learningand include two demonstrations.
The prompt we use can be found inTable12.Word-overlap filtering.We notice that inline-citation questions generated in the last step can have very high word overlap with the target paper titles, which makes their retrieval trivial even for BM25 and suggests that the questions may not be of interest to researchers. We calculate the word overlap
as thepercentage of words in the generated question that are also included in the target paper titles.
We filter out ACL sourced questions that have an overlap score higher than 0.3 and
non-ACL sourced questions that have an overlap score higher than 0.1.
In this step, we filter out 5% of the ACL sourced questions and 80% of the non-ACL sourced questions.BroadSpecificTotal #Q#QAvg. LenOverlapAvg. #P#QAvg. LenOverlapAvg. #PInline-citation Questions12020.60.331.2123122.10.341.07351Author-written Questions3515.80.431.0321117.90.431.00246Table 2:Statistics for LitSearch. Please refer toTable14for more detailed statistics of each subset. “#Q”: number of questions. “Overlap”: the fraction of words in the question that are also included in the titles and abstracts of the target papers. “Avg. #P”: average number of target papers.2.2Author-written QuestionsBesides generating questions using existing inline citation mentions, we also collect questions directly from human annotators.
As writing literature search questions requires deep understanding of the research field and the target paper,
we invite researchers to write search queries that are answered by their own published papers.
One additional benefit of this setup is that the correctness of the questions is better guaranteed.We invited authors of ACL 2023 and ICLR 2024 papers to write one literature search question for each of their papers. We chose the two venues as they were among the latest natural language and machine learning conferences at the time of the data collection, hence the papers represent the latest research development and are unlikely to have already been included in the pre-training data of LLMs and retrievers used in our evaluations.
We sent out invitations toACL 2023 authors andICLR 2024 authors, and receivedquestions from ACL 2023 authors andquestions from ICLR 2024 authors.2.3Manual Filtering to Ensure High QualityFinally, the authors of this work manually examine every question from both the inline-citation and author-written subsets and annotate these forspecificityandquality(guidelines inTable1) .
Questions that are too general (there are more than 20 papers from the corpus can fit the question)  are assigned a quality score of 0 and are excluded. We include only questions with a quality score of 1 or 2 in the final dataset.
We also rewrite questions if they have minor mistakes and can be fixed easily.
Each question is assigned to one author for annotation.As the examples inTable1show,
questions of both specificity types can be realistic and valuable, but they exhibit distinct traits.
We use the specificity scores to distinguishbroadandspecificquestions in the evaluation.For the inline-citation subset,
we manually examined 382 ACL sourced questions and 450 non-ACL sourced questions.
26% (98 instances)  of the ACL questions and
56% (253 instances)  of the non-ACL questions are kept. For the author-written subset, since all questions are written by experts, we avoid rewriting them as much as possible. In the end, we kept 89% (155)  questions from ACL 2023 authors and 78% (91)  questions from ICLR 2024 authors.2.4Dataset StatisticsOur final dataset contains 597 questions, with 351 in the inline-citation subset and 246 in the author-written subset. Dataset statistics, including the number of questions, the average question length, and the average word overlap between the question and the target papers (titles and abstracts) , are presented inTable2.
We find that author-written
questions are shorter and have a higher word overlap rate with the target papers (0.43 vs. 0.33 for inline-citation questions) .
This is expected: when writing questions for their own papers,
authors tend to re-use terminology from their papers and focus on the main findings which are usually included in the abstracts or titles.
In contrast, inline-citation questions can be anchored to any span of the reference documents, irrespective of the main findings or the main focuses of the target papers.2.5The Retrieval CorpusThe LitSearch retrieval corpusconsists of ACL Anthology and ICLR papers extracted from
S2ORC (seeAppendixCfor details) .
We do not use the full S2ORC corpus for efficiency reasons.
In total, this yieldspapers (ACL Anthology papers andICLR papers) 333papers are common to both subsets..
The average number of words for the documents inis 134 / 6,041 (titles and abstracts / full texts) .
body
Citation Benchmark
2LitSearch
2407.18940
3Experiments3.1Experimental SetupWe compare the performance of different retrieval systems (enumerated below)  on our LitSearch benchmark. Due to the limited context sizes of existing embedding models, we only use the paper titles and abstracts to embed the papers in our retrieval corpusby default.For all systems we compare, we report the recall@K for both the broad and specific subsets of LitSearch. We report results forfor the specific subset andfor the broad subset; these values (5 and 20)  correspond to the guidelines followed by the authors while determining the specificity of a given question (see Table1) .3.2BaselinesWe benchmark both retrieval models and LLM-based rerankers in this work.Retriever models.We evaluate using the classic BM25 algorithm, as well as several state-of-the-art dense retrieval (embedding)  models,
including GTR,
Instructor,
E5,
and GritLM.444We use the following corresponding checkpoints fromhttps://huggingface.co/:
GTR-T5-large, Instructor-XL, E5-large-v2, and GritLM-7B.More details are provided inAppendixD.Inline-citationAuthor-writtenAvg.Avg.BroadSpecificBroadSpecificBroadSpecificR@20R@5R@20R@20R@5R@20R@20R@5BM2537.438.555.848.662.673.539.950.0GTR-T5-large45.738.551.537.140.855.943.839.6Instructor-XL56.348.960.057.155.970.156.552.3E5-large-v255.850.463.954.362.675.855.456.2GritLM-7B69.767.777.974.382.589.170.874.8GPT-4 reranking (w/ BM25) 54.960.067.577.176.882.959.968.0GPT-4 one-hop (w/ BM25) 62.064.171.674.373.577.764.868.6GPT-4 reranking (w/ GritLM) 74.773.279.977.185.892.475.379.2GPT-4 one-hop (w/ GritLM) 72.970.378.474.384.487.273.277.0Table 3:Main experiment results of LitSearch.
Here we only use the titles and abstracts of papers for retrieval and reranking.
We report recall@20 (R@20)  for broad questions and
recall@5,20 (R@5, R@20)  for specific questions.
“Broad” and “specific” correspond to the annotations during our manual filtering stage (defined inTable1) .Figure 3:We demonstrate detailed retrieval results using BM25, E5 and GritLM up to. Additionally, we show the effect of applying GPT-4 reranking over GritLM retrieval results.LLM-based reranking.In addition to vanilla retrieval, we also use strong LLMs (GPT-4555We use gpt-4-1106-preview in our experiments.in our case)  to rerank the top retrieved results from the above retrievers. We use two strategies:Vanilla reranking.We include the top-retrieved papers (titles and abstracts)  in the context and prompt GPT-4 to rerank these based on the question (see our prompt inTable13) . This is similar to prior works. We use, resulting in an average context length of 13,844 words.One-hop reranking.Inspired by, we leverage the fact that for some questions, there may exist lexically similar inline citation mentions in the retrieval corpus. Due to our data collection pipeline, this is particularly true for the ACL sourced inline citation questions. We posit that the retrieval models will be able to retrieve these source papers (that cite the target papers)  based on the questions.We extract the topretrieved papers and construct a new candidate list by adding papers cited by each of these seed retrieved papers. We concatenate papers in the following order, skipping duplicates: [rank-1 paper, papers cited by, rank-2 paper, papers cited by, …,, papers cited by].
To avoid very long contexts, we truncate this list after the firstpapers and use the same prompt for GPT-4 based reranking as above. In our experiments, we useandresulting in average length of 27,544 words.3.3ResultsWe outline the performance of the above systems on LitSearch inTable3. First, we observe that all instruction-finetuned embedding models, e.g. Instructor, E5, and GritLM, substantially outperform BM25 on our benchmark. In fact, they also perform better than the GTR model. Overall, we found that GritLM-7B achieves the best performance (recall@20 on broad questions andrecall@5 on specific questions) , leaving a large gap compared to other baselines.Impact of reranking.We also report the performance improvement brought by the reranking methods on the weakest (BM25)  and strongest (GritLM)  retrievers inTable3. We observe that both vanilla and one-hop reranking improve over the base retrieval performance. For example, on the specific subset of inline questions, the vanilla GPT-4 reranking improves the recall@5 of BM25 and GritLM byandrespectively.
Interestingly, the improvements from one-hop reranking are generally lower than vanilla reranking across all subsets, when using GritLM as the base retriever; this shows that our benchmark cannot be easily “gamed” by mimicking the data collection pipeline or exploiting similar citation mentions to the question from other papers.Impact of question specificity.Table3andFigure3show that retrieval systems generally report higher recall performance on the specific subset. This is expected: there exists a smaller number of “competing” papers, i.e. those that also satisfy the search question, for the specific subset in the retrieval corpus.
Note that our human annotation tagged questions with approximately 5 relevant papers as specific and 20 relevant papers as broad (seeTable1for details) .
We keep both subsets in our dataset as this stratified reporting presents a more nuanced view of retriever capabilities.Inline-citation vs. author-written questions.We observe very different performance trends for the two subsets (Table3andFigure3) .
In particular, inline-citation questions are harder than author-written questions for all retriever systems, on both broad and specific questions.We attribute this difference to the higher semantic or lexical overlap of author-written questions with the paper titles and abstracts (seeTable2for statistics) .
This reflects expected tendency of paper authors to formulate the questions around the main contributions from the abstracts and re-use terminologies. Such annotator biases have been widely discussed in prior data collection efforts as well, particularly when humans write content from scratch.Inline (specific) Author (specific) Qual=1Qual=2Qual=1Qual=2R@5R@5R@5R@5BM2536.430.662.255.0GTR-T5-large42.031.440.736.9Instructor-XL55.139.558.548.6E5-large-v248.442.661.557.7GritLM-7B67.358.780.076.6Table 4:Comparison of retrieval performance on different quality (qual)  questions. Generally, retrievers report lower performance on the Qual=2 questions, i.e. those deemed more challenging in our manual annotation.Impact of question quality.Table4compares how retrieval models perform on different quality subsets. Recall that we manually annotated the quality of all questions (Table1) . We observe that questions with a quality score of, i.e. determined to be more realistic and difficult by manual annotators, are consistently more challenging for retrieval models. This demonstrates the high annotation quality of our manual inspection step. The presence of these different quality questions in our dataset leads to higher diversity and better coverage over the varied information seeking needs of users.
body
Citation Benchmark
3Experiments
2407.18940
4AnalysisInlineAuthorBroadSpecBroadSpecR@20R@5R@20R@5BM2537.438.548.662.6w/ full18.623.865.771.6GTR-T5-large45.738.537.140.8w/ full43.939.445.739.8Instructor-XL56.348.957.155.9w/ full53.050.957.156.9E5-large-v255.850.454.362.6w/ full56.948.760.062.1GritLM-7B69.767.774.382.5w/ full70.863.465.773.0Table 5:Retrieval results of using only titles and abstracts vs. using titles, abstracts, and full text (w/ full) . We do not observe consistent improvements from including the full text for existing retrieval models.4.1Does Including More Paper Content Improve Retrieval Performance?In the previous section, we only used the titles and abstracts (on average 134 words)  to encode the papers in the retrieval corpus. Here, we evaluate whether encoding more paper content can improve retrieval performance. For all retriever models compared, we create embeddings using the full paper text (on average 6,041 words)  up to their allowed context lengths.666The maximum context lengths for GTR-T5-large, Instructor-XL, E5-large-v2 and GritLM-7B are 512, 512, 512 and 2048 tokens respectively.We compare this setting against our default setting (only titles and abstracts) .Our results are outlined inTable5. Surprisingly, we find that the addition of more paper text does not improve performance on LitSearch consistently. In fact, we only observe substantial improvement on the author-written broad questions for BM25 and some embedding models. In other cases, more text more often hinders instead of improving performance.
Note that the maximum context length of the tested models is 2,048 (GritLM)  and the average length of their training data is even shorter—for example, the commonly used MS-MARCOand NaturalQuestionshave an average document length of 56 and 79. This is significantly shorter than the full text of papers from our retrieval corpus averaging 6,041 words in length, potentially leading to the
unsatisfying performance when using full texts with embedding models.ACLNon-ACLBroadSpecificBroadSpecificR@20R@5R@20R@5BM2538.839.436.938.2GTR-T5-large37.239.448.938.2Instructor-XL48.643.959.150.9E5-large-v246.646.259.152.1GritLM-7B72.465.968.868.5With BM25Reranking51.159.856.260.0One-hop65.271.260.861.2With GritLMReranking80.372.772.773.3One-hop81.367.469.971.5Table 6:Comparison of retrieval performance on the ACL vs. non-ACL sourced inline-citation questions. Results show that the performance improvement from one-hop reranking over BM25 is subtantially higher for ACL sourced questions.4.2Does the Source of Inline Citation Questions Matter?Next, we study how the different sources of inline-citation questions
affect the model performance.Table6outlines the performance of retrieval models on ACL sourced vs. non-ACL sourced inline-citation questions.
Our results show that the two different sets report similar trends and model rankings for different retrieval models, particularly on the specific subset of questions. Interestingly, we find that the performance improvement from one-hop reranking is very different for the ACL and non-ACL questions.For BM25, we observe that one-hop reranking is significantly better than the vanilla reranking on ACL sourced questions (+11.4% recall@5 on specific) ; but the gap is much smaller on non-ACL sourced questions (+1.2% recall@5 on specific) .
We posit that this is because BM25 can better exploit the data annotation pipeline on the ACL sourced questions. It can likely first identify the source ACL paper where the citation mention comes from, and then find the target paper via one-hop reranking. Including the non-ACL questions to LitSearch prevents systems from exploiting such “shortcuts” as non-ACL source papers are not part of the retrieval corpus.For GritLM, we do not observe similarly large performance gains when using one-hop reranking. We hypothesize that this is because when using GritLM, the initial top retrieval results already include the target papers and the one-hop strategy does not bring further improvement.Inline (specific) Author (specific) R@5R@5BM2538.562.6GritLM-7B67.782.5Google Search23.162.5Google Scholar20.517.5Elicit23.117.5Table 7:Recall@5 for commercial search engines on a random subset of 80 specific questions. Search engines generally report poor performance. Note that the comparison is not apples-to-apples as search engines use a much larger retrieval corpus.4.3Performance of Search EnginesMSMARCOSCIDOCSNQArXivLitSearch(broad) LitSearch(specific) GTR-T5-large42.715.555.117.523.330.4Instructor-XL41.617.457.219.832.841.2E5-large-v243.520.563.427.027.145.3GritLM-7B42.024.470.334.344.160.3Table 8:Comparison between LitSearch and existing retrieval benchmarks.
All reported numbers are nDCG@10 for a direct comparison.In practice, researchers use search engines like Google Search, Google Scholar, or Elicit777https://elicit.com/to search for relevant papers for their scientific queries. We conduct a human study to understand how these search engines perform on LitSearch: We randomly sample 80 questions (all specific; 40 inline-citation and 40 author-written)  from our dataset. We manually input these questions into the above search engines and report recall@5.888For Google Search, we consider the top-5 academic papers in the search results and ignore other webpage results. We only consider the academic papers on the first page of search results, even if this number is lower than 5.We note that this is not an apples-to-apples comparison against the retrieval models in earlier sections due to the discrepancy in the retrieval corpus.Table7outlines the results of our human study. It shows that all three search engines deliver similarly low recalls on inline-citation questions. On the author-written questions, Google Search performs much better than the other two. Although not directly comparable, this performance is generally worse than the embedding models, demonstrating the potential of these strong dense retrieval models for citation recommendation applications.4.4Comparing Other Retrieval BenchmarksWe compare model performance on LitSearch
to several popular retrieval benchmarks included in BEIRand MTEB—namely MSMARCO, SCIDOCS, and NQ.999We use the results from the MTEB benchmark website:https://huggingface.co/spaces/mteb/leaderboard.We also compare to ArXiv, a previous citation recommendation benchmark directly using inline citations as queries.Table8shows that
LitSearch generally agrees with existing retrieval benchmarks.
However,
LitSearch can differentiate retriever models better: for example, the gap between GritLM and E5 on LitSearch (specific)  is 15 points (nDCG@10) , while they perform almost the same on MSMARCO.
LitSearch provides an informative testbed that can effectively reflect the recent (and future)  advancement in embedding models.
body
Citation Benchmark
4Analysis
2407.18940
5Related WorkCitation recommendation.The community has propose a number of citation recommendation datasets,
including global citation recommendation datasets (directly using a paper as the query and papers it cites as target papers;) , and
local citation recommendation datasets (using inline citation mentions as queries;) .
There are also language models and retrieval models specifically trained for scientific document understanding and retrieval tasks, such as SciBERTand SPECTER.
Compared to existing citation recommendation datasets,
LitSearch is comprised of manually annotated, natural language literature search questions,
providing a more realistic and challenging evaluation for citation recommendation systems.Retrieval benchmarks.There have been numerous datasets evaluating
retrieval systems from
Wikipedia,
web queries,
biomedical questions, and more.
Recently, there have been several benchmarks
combining multiple datasets
and evaluating retrieval or embedding models
across different domains and different use cases, such as KILT, BEIR, and MTEB.
LitSearch offers a unique perspective by exploring the novel literature search question type,
effectively complementing the existing benchmarks.Retrieval systems.Traditional retrieval systems rely on bag-of-word algorithms such as TF-IDF and BM25.
Dense retrieval (embedding)  models have gained more popularity due to their abilities to do semantic search without relying on exact keyword matches.
State-of-the-art dense models
are mostly adopted by
fine-tuning pre-trained language modelswith a contrastive learning objective on either supervised or unsupervised data.
Recent development introduces “instructions” when encoding queries and documents, which significantly improves the versatility of embeddings across tasks.
body
Citation Benchmark
5Related Work
2407.18940
6ConclusionIn this paper, we propose LitSearch,
a new retrieval benchmark comprised of 597 manually-curated literature search questions.
LitSearch includes a inline-citation question set and an author-written question set, both undergoing manual inspections from the authors of LitSearch.
We conduct extensive experiments with BM25, state-of-the-art embedding models, and LLM reranking. Our experiments demonstrate the superior performance of state-of-the-art instruction-finetuned embedding models,
with additional improvement via GPT-4-based reranking.
We also verify that commercial search engines like Google struggle on LitSearch questions.
The comparison with existing retrieval benchmarks shows that
LitSearch better highlights the different performance of retrieval systems.
body
Citation Benchmark
6Conclusion
2407.18940
LimitationsEven though we manually examined the dataset,
there still exist questions that are either slightly out of distribution compared to what researchers would ask,
or too easy due to high overlap with the target papers.
The author-written questions are easier than we expected,
as writing challenging literature search questions is non-trivial even for experienced researchers.
Even though we experimented with several state-of-the-art systems, it was not an exhausted evaluation and we left out more sophisticated retrieval or reranking systems.
This research primarily focuses on only English questions and research papers.
body
Citation Benchmark
Limitations
2407.18940
Ethics StatementThe research artifact of this paper, LitSearch, is manually inspected and is ensured to have no unsafe or inappropriate content.
However, the process to generate the dataset may introduce certain biases: for example, the inline-citation questions contain more target papers that have high citations due to the sampling; the author-written questions only cover ACL 2023 and ICLR 2024 papers.
body
Citation Benchmark
Ethics Statement
2407.18940
AcknowledgementsWe want to acknowledge Dan Friedman, Howard Yen, Jiayi Geng, Lucy He, and other members of the Princeton NLP group for their useful feedback and discussion.
We also acknowledge all the ACL 2023 and ICLR 2024 authors that contributed questions to LitSearch (listed inAppendixA) .
Tianyu Gao is supported by an IBM PhD Fellowship.
This work is gratefully supported by an NSF CAREER award (IIS-2239290) , and Microsoft Azure credits through the “Accelerate Foundation Models Academic Research” Initiative.
conclusion
Citation Benchmark
Acknowledgements