<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2404.18424] PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval</title><meta property="og:description" content="The current use of large language models (LLMs) for zero-shot document ranking follows one of two ways: 1) prompt-based re-ranking methods, which require no further training but are feasible for only re-ranking a handf…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2404.18424">

<!--Generated on Sun May  5 19:45:40 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document">
<h1 class="ltx_title ltx_title_document">
PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval
</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Shengyao Zhuang<sup id="id9.9.id1" class="ltx_sup"><span id="id9.9.id1.1" class="ltx_text ltx_font_italic">1,3</span></sup>, Xueguang Ma<sup id="id10.10.id2" class="ltx_sup"><span id="id10.10.id2.1" class="ltx_text ltx_font_italic">2</span></sup>, Bevan Koopman<sup id="id11.11.id3" class="ltx_sup"><span id="id11.11.id3.1" class="ltx_text ltx_font_italic">1,3</span></sup>, Jimmy Lin<sup id="id12.12.id4" class="ltx_sup"><span id="id12.12.id4.1" class="ltx_text ltx_font_italic">2</span></sup>, Guido Zuccon<sup id="id13.13.id5" class="ltx_sup">3</sup> 
<br class="ltx_break"><sup id="id14.14.id6" class="ltx_sup">1</sup>CSIRO, 
<br class="ltx_break"><sup id="id15.15.id7" class="ltx_sup">2</sup> University of Waterloo,

<br class="ltx_break"><sup id="id16.16.id8" class="ltx_sup">3</sup> The University of Queensland
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id17.id1" class="ltx_p">The current use of large language models (LLMs) for zero-shot document ranking follows one of two ways: 1) prompt-based re-ranking methods, which require no further training but are feasible for only re-ranking a handful of candidate documents due to the associated computational costs; and 2) unsupervised contrastive trained dense retrieval methods, which can retrieve relevant documents from the entire corpus but require a large amount of paired text data for contrastive training.
In this paper, we propose PromptReps, which combines the advantages of both categories: no need for training and the ability to retrieve from the whole corpus. Our method only requires prompts to guide an LLM to generate query and document representations for effective document retrieval. Specifically, we prompt the LLMs to represent a given text using a single word, and then use the last token’s hidden states and the corresponding logits associated to the prediction of the next token to construct a hybrid document retrieval system. The retrieval system harnesses both dense text embedding and sparse bag-of-words representations given by the LLM. Our experimental evaluation on the BEIR zero-shot document retrieval datasets illustrates that this simple prompt-based LLM retrieval method can achieve a similar or higher retrieval effectiveness than state-of-the-art LLM embedding methods that are trained with large amounts of unsupervised data, especially when using a larger LLM.<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>Code for fully reproducing the results is available at <a target="_blank" href="https://github.com/ielab/PromptReps" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/ielab/PromptReps</a>.</span></span></span></p>
</div>
<div id="p1" class="ltx_para ltx_noindent">
<div id="p1.8" class="ltx_block ltx_align_bottom">
<p id="p1.8.9" class="ltx_p"><span id="p1.8.9.1" class="ltx_text ltx_font_bold">PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval</span></p>
<br class="ltx_break ltx_centering">
<p id="p1.8.8" class="ltx_p ltx_align_center" style="width:433.6pt;"><span id="p1.8.8.8" class="ltx_text ltx_inline-block" style="width:0.0pt;">
<span id="p1.8.8.8.8" class="ltx_tabular ltx_guessed_headers ltx_align_top">
<span class="ltx_thead">
<span id="p1.5.5.5.5.5" class="ltx_tr">
<span id="p1.5.5.5.5.5.5" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="p1.5.5.5.5.5.5.5" class="ltx_text ltx_font_bold">Shengyao Zhuang<sup id="p1.5.5.5.5.5.5.5.1" class="ltx_sup"><span id="p1.5.5.5.5.5.5.5.1.1" class="ltx_text ltx_font_medium ltx_font_italic">1,3</span></sup>, Xueguang Ma<sup id="p1.5.5.5.5.5.5.5.2" class="ltx_sup"><span id="p1.5.5.5.5.5.5.5.2.1" class="ltx_text ltx_font_medium ltx_font_italic">2</span></sup>, Bevan Koopman<sup id="p1.5.5.5.5.5.5.5.3" class="ltx_sup"><span id="p1.5.5.5.5.5.5.5.3.1" class="ltx_text ltx_font_medium ltx_font_italic">1,3</span></sup>, Jimmy Lin<sup id="p1.5.5.5.5.5.5.5.4" class="ltx_sup"><span id="p1.5.5.5.5.5.5.5.4.1" class="ltx_text ltx_font_medium ltx_font_italic">2</span></sup>, Guido Zuccon<sup id="p1.5.5.5.5.5.5.5.5" class="ltx_sup"><span id="p1.5.5.5.5.5.5.5.5.1" class="ltx_text ltx_font_medium">3</span></sup></span></span></span>
</span>
<span class="ltx_tbody">
<span id="p1.6.6.6.6.6" class="ltx_tr">
<span id="p1.6.6.6.6.6.1" class="ltx_td ltx_align_center"><sup id="p1.6.6.6.6.6.1.1" class="ltx_sup">1</sup>CSIRO,</span></span>
<span id="p1.7.7.7.7.7" class="ltx_tr">
<span id="p1.7.7.7.7.7.1" class="ltx_td ltx_align_center"><sup id="p1.7.7.7.7.7.1.1" class="ltx_sup">2</sup> University of Waterloo,</span></span>
<span id="p1.8.8.8.8.8" class="ltx_tr">
<span id="p1.8.8.8.8.8.1" class="ltx_td ltx_align_center"><sup id="p1.8.8.8.8.8.1.1" class="ltx_sup">3</sup> The University of Queensland</span></span>
</span>
</span></span></p>
<br class="ltx_break ltx_centering">
</div>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Large Language Models (LLMs) such as GPT4 and LLaMA, which are pretrained on massive corpora and finetuned to follow user instructions, have strong zero-shot natural language understanding capabilities <cite class="ltx_cite ltx_citemacro_cite">OpenAI (<a href="#bib.bib33" title="" class="ltx_ref">2023</a>); Touvron et al. (<a href="#bib.bib39" title="" class="ltx_ref">2023</a>)</cite>.
Via prompting, LLMs excel in various text generation tasks such as question answering, writing assistance, and conversational agent <cite class="ltx_cite ltx_citemacro_cite">Hendrycks et al. (<a href="#bib.bib15" title="" class="ltx_ref">2021</a>); Liu et al. (<a href="#bib.bib29" title="" class="ltx_ref">2023</a>)</cite>.
Inspired by the success of LLMs on natural language understanding tasks, several studies have explored the potential of using LLMs to perform unsupervised document ranking task.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2404.18424/assets/image/architecture.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="598" height="467" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>Overview of PromptReps, LLMs are prompted to generate dense and sparse text representation at the same time, which is then used to build search indices.</figcaption>
</figure>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">One line of work focuses on directly prompting LLMs to infer document relevance to a given query <cite class="ltx_cite ltx_citemacro_cite">Sachan et al. (<a href="#bib.bib36" title="" class="ltx_ref">2022</a>); Zhuang et al. (<a href="#bib.bib48" title="" class="ltx_ref">2023a</a>); Ma et al. (<a href="#bib.bib31" title="" class="ltx_ref">2023b</a>); Sun et al. (<a href="#bib.bib37" title="" class="ltx_ref">2023</a>); Pradeep et al. (<a href="#bib.bib34" title="" class="ltx_ref">2023</a>); Zhuang et al. (<a href="#bib.bib49" title="" class="ltx_ref">2023b</a>); Qin et al. (<a href="#bib.bib35" title="" class="ltx_ref">2024</a>)</cite>. For instance, RankGPT <cite class="ltx_cite ltx_citemacro_cite">Sun et al. (<a href="#bib.bib37" title="" class="ltx_ref">2023</a>)</cite> casts document re-ranking as a permutation generation task, prompting LLMs to generate re-ordered document identifiers according to the document’s relevance to the query. These methods leverage LLMs for document ranking in a complete zero-shot setting where no further training is required. However, these methods can only serve as a second-stage re-ranker on a handful of candidate documents. This is because each prompt requires one full LLM inference: for example, in the case of a corpus with 1M documents, a pointwise approach would require the construction of 1M prompts and thus the execution of 1M (costly) LLM inferences – making it unfeasible for an online search engine.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">Another line of research leverages LLMs as a text embedding model for dense document retrieval <cite class="ltx_cite ltx_citemacro_cite">Lee et al. (<a href="#bib.bib23" title="" class="ltx_ref">2024</a>); Wang et al. (<a href="#bib.bib40" title="" class="ltx_ref">2024a</a>, <a href="#bib.bib41" title="" class="ltx_ref">b</a>); BehnamGhader et al. (<a href="#bib.bib5" title="" class="ltx_ref">2024</a>)</cite>. For example, E5-mistral <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a href="#bib.bib41" title="" class="ltx_ref">2024b</a>)</cite> employs LLMs to create synthetic datasets of query-document pairs. These paired text data are then used to perform unsupervised contrastive training for a Mistral LLM-based dense retriever. Since the queries and documents are encoded with LLMs separately, i.e., using a bi-encoder architecture, these methods could serve as a first-stage document retriever. However, all existing LLM-based retrievers require an unsupervised contrastive training step to transform a generative LLM into a text-embedding model. Even with parameter-efficient training techniques such as LoRA <cite class="ltx_cite ltx_citemacro_cite">Hu et al. (<a href="#bib.bib16" title="" class="ltx_ref">2022</a>)</cite>, this extra training is still very expensive.
For example, the contrastive training of E5-mistral using a large batch size (2048) and LoRA took <math id="S1.p3.1.m1.1" class="ltx_Math" alttext="\approx 18" display="inline"><semantics id="S1.p3.1.m1.1a"><mrow id="S1.p3.1.m1.1.1" xref="S1.p3.1.m1.1.1.cmml"><mi id="S1.p3.1.m1.1.1.2" xref="S1.p3.1.m1.1.1.2.cmml"></mi><mo id="S1.p3.1.m1.1.1.1" xref="S1.p3.1.m1.1.1.1.cmml">≈</mo><mn id="S1.p3.1.m1.1.1.3" xref="S1.p3.1.m1.1.1.3.cmml">18</mn></mrow><annotation-xml encoding="MathML-Content" id="S1.p3.1.m1.1b"><apply id="S1.p3.1.m1.1.1.cmml" xref="S1.p3.1.m1.1.1"><approx id="S1.p3.1.m1.1.1.1.cmml" xref="S1.p3.1.m1.1.1.1"></approx><csymbol cd="latexml" id="S1.p3.1.m1.1.1.2.cmml" xref="S1.p3.1.m1.1.1.2">absent</csymbol><cn type="integer" id="S1.p3.1.m1.1.1.3.cmml" xref="S1.p3.1.m1.1.1.3">18</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p3.1.m1.1c">\approx 18</annotation></semantics></math> hours on 32 V100 GPUs <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a href="#bib.bib41" title="" class="ltx_ref">2024b</a>)</cite>.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">In this work, we propose a new zero-shot LLM-based document retrieval method called PromptReps.
We demonstrate that LLMs can be directly prompted to produce query and document embeddings, which can serve as effective text representations for neural retrieval systems.
Specifically, we prompt an LLM by asking it to use a single word to represent a query or a document.
Then, we extract the last layer’s hidden state of the last token in the prompt as the dense representation of the input text.
Simultaneously, we utilize the logits associated with predicting the subsequent token to form a sparse representation.
As illustrated in Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, through a single forward pass, we generate text representations for a document that can be indexed for dense, sparse, or hybrid search architectures.</p>
</div>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.1" class="ltx_p">We evaluate our approach on the BEIR datasets <cite class="ltx_cite ltx_citemacro_cite">Thakur et al. (<a href="#bib.bib38" title="" class="ltx_ref">2021</a>)</cite> to perform zero-shot retrieval.
We utilize various open-source large language models and find that the effectiveness of zero-shot retrieval increases with the scaling of LLM size.
Results show that PromptReps outperforms previous LLM-based embedding methods. Of key importance is that our method is the first LLM-based method that can effectively perform full corpus retrieval while at the same time not requiring contrastive training, demonstrating that prompt engineering for generative LLMs is capable of generating robust representations for retrieval.</p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related Work</h2>

<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>Supervised Neural Retrievers</h3>

<div id="S2.SS1.p1" class="ltx_para">
<p id="S2.SS1.p1.1" class="ltx_p">Neural retrievers based on bi-encoder architecture bring significant improvement over traditional heuristic retrievers such as BM25. Dense retrievers such as DPR <cite class="ltx_cite ltx_citemacro_cite">Karpukhin et al. (<a href="#bib.bib21" title="" class="ltx_ref">2020</a>)</cite>, ANCE <cite class="ltx_cite ltx_citemacro_cite">Xiong et al. (<a href="#bib.bib45" title="" class="ltx_ref">2021</a>)</cite>, ColBERT <cite class="ltx_cite ltx_citemacro_cite">Khattab and Zaharia (<a href="#bib.bib22" title="" class="ltx_ref">2020</a>)</cite>, encode text into low-dimensional dense vectors and conduct search with (approximate) nearest neighbor search. On the other hand, sparse neural retrievers such as DeepImpact <cite class="ltx_cite ltx_citemacro_cite">Mallia et al. (<a href="#bib.bib32" title="" class="ltx_ref">2021</a>)</cite>, uniCOIL <cite class="ltx_cite ltx_citemacro_cite">Lin and Ma (<a href="#bib.bib27" title="" class="ltx_ref">2021</a>)</cite>, TILDE <cite class="ltx_cite ltx_citemacro_cite">Zhuang and Zuccon (<a href="#bib.bib51" title="" class="ltx_ref">2021b</a>, <a href="#bib.bib50" title="" class="ltx_ref">a</a>)</cite>, and SPLADE <cite class="ltx_cite ltx_citemacro_cite">Formal et al. (<a href="#bib.bib11" title="" class="ltx_ref">2021</a>)</cite>, encode text into high-dimensional sparse vectors as bag-of-words representations and conduct search in an inverted index. Recent work has also explored fine-tuning LLMs as dense retrievers such as RepLLaMA <cite class="ltx_cite ltx_citemacro_cite">Ma et al. (<a href="#bib.bib30" title="" class="ltx_ref">2023a</a>)</cite> and LLaRA <cite class="ltx_cite ltx_citemacro_cite">Liao et al. (<a href="#bib.bib26" title="" class="ltx_ref">2024</a>)</cite>. A hybrid neural retrieval system refers to a system that combines the rankings provided by both dense and sparse retrievers, often resulting in an enhanced final ranking <cite class="ltx_cite ltx_citemacro_cite">Lin and Ma (<a href="#bib.bib27" title="" class="ltx_ref">2021</a>)</cite>. All these retrievers are trained with supervised relevance judgment data (e.g., MS MARCO <cite class="ltx_cite ltx_citemacro_cite">Bajaj et al. (<a href="#bib.bib3" title="" class="ltx_ref">2018</a>)</cite>) using contrastive learning. Our work instead focuses on building a hybrid neural retrieval system with zero-shot dense and sparse document representations without supervised contrastive learning. This is particularly valuable for applications where no relevance judgements are available — often the case in real work retrieval settings.</p>
</div>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>Unsupervised Neural Retrievers</h3>

<div id="S2.SS2.p1" class="ltx_para">
<p id="S2.SS2.p1.1" class="ltx_p">There have also been attempts on training effective neural retrievers without relying on human relevance judgments.
Methods such as Contriever <cite class="ltx_cite ltx_citemacro_cite">Izacard et al. (<a href="#bib.bib17" title="" class="ltx_ref">2022</a>)</cite> and E5 <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a href="#bib.bib40" title="" class="ltx_ref">2024a</a>)</cite>, train dense retriever with large-scale pseudo query-document pairs to build unsupervised (synthetic) training data.
LLMs have also been adapted as unsupervised text embedding models for first-stage document retrieval.
For instance, HyDE <cite class="ltx_cite ltx_citemacro_cite">Gao et al. (<a href="#bib.bib12" title="" class="ltx_ref">2023a</a>)</cite> enhances query representations for an unsupervised retriever by replacing the original query with LLM-generated hypothetical documents.
More recent work has focused on directly converting generative LLMs into a text-embedding models with unsupervised contrastive pre-training. These works, such as E5-Mistral-Inst <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a href="#bib.bib41" title="" class="ltx_ref">2024b</a>)</cite> and Gecko <cite class="ltx_cite ltx_citemacro_cite">Lee et al. (<a href="#bib.bib23" title="" class="ltx_ref">2024</a>)</cite>, use large-scale weakly supervised paired text data or LLM-generated query-document pair data to perform contrastive training on top of LLMs.
LLM2Vec <cite class="ltx_cite ltx_citemacro_cite">BehnamGhader et al. (<a href="#bib.bib5" title="" class="ltx_ref">2024</a>)</cite>, on the other hand, conducts further masked next token prediction pre-training with bidirectional attention, and SimCSE <cite class="ltx_cite ltx_citemacro_cite">Gao et al. (<a href="#bib.bib14" title="" class="ltx_ref">2021</a>)</cite> trains on raw text data to transform LLMs into text encoders. Although no labeled data is used, these methods require synthetic or unsupervised paired text data to perform contrastive pre-training.
Our method instead relies solely on prompt engineering to transform LLM into a robust text encoder for document retrieval without any extra training.</p>
</div>
</section>
<section id="S2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.3 </span>Prompting LLMs for document re-ranking</h3>

<div id="S2.SS3.p1" class="ltx_para">
<p id="S2.SS3.p1.1" class="ltx_p">Inspired by the prompt-following capacity of LLMs, recent works have explored prompting LLMs for document reranking. For instance, UPR <cite class="ltx_cite ltx_citemacro_cite">Sachan et al. (<a href="#bib.bib36" title="" class="ltx_ref">2022</a>)</cite> ranks documents pointwise by prompting the LLM to generate a relevant query for a given document and rank documents based on the likelihood of generating the query. RankGPT <cite class="ltx_cite ltx_citemacro_cite">Sun et al. (<a href="#bib.bib37" title="" class="ltx_ref">2023</a>)</cite> and LRL <cite class="ltx_cite ltx_citemacro_cite">Ma et al. (<a href="#bib.bib31" title="" class="ltx_ref">2023b</a>)</cite> propose to re-rank a list of documents at once and generate permutations for the reordered list. Pairwise <cite class="ltx_cite ltx_citemacro_cite">Qin et al. (<a href="#bib.bib35" title="" class="ltx_ref">2024</a>)</cite> and Setwise <cite class="ltx_cite ltx_citemacro_cite">Zhuang et al. (<a href="#bib.bib49" title="" class="ltx_ref">2023b</a>)</cite> prompting methods have also been explored to improve effectiveness and efficiency in the LLM re-ranking pipeline. These methods are only feasible for re-ranking a handful of candidate documents, thus limited to second-stage document re-ranking. In contrast, our approach utilizes prompts to construct the first-stage retrievers.</p>
</div>
</section>
<section id="S2.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.4 </span>Prompting LLM for sentence embeddings</h3>

<div id="S2.SS4.p1" class="ltx_para">
<p id="S2.SS4.p1.1" class="ltx_p">The methods most similar to ours are work on prompting LLMs to generate sentence embeddings for semantic textual similarity (STS) tasks <cite class="ltx_cite ltx_citemacro_cite">Jiang et al. (<a href="#bib.bib19" title="" class="ltx_ref">2023b</a>); Lei et al. (<a href="#bib.bib24" title="" class="ltx_ref">2024</a>); Zhang et al. (<a href="#bib.bib47" title="" class="ltx_ref">2024</a>)</cite>. These works developed an Explicit One-word Limitation (EOL) prompt, which also instructs LLMs to represent a sentence with one word. However, these works only evaluate such prompts on STS datasets, and their effectiveness on information retrieval datasets with large document corpora is unknown. Additionally, these methods only represent texts with dense embeddings from the hidden states; our method instead generates sparse text representations simultaneously to build a hybrid retrieval system. In fact, our results show that dense embeddings alone perform poorly for document retrieval tasks with some LLMs, but sparse representations are much more robust, and the best retrieval effectiveness is achieved with the hybrid retrieval system with scaled model size.</p>
</div>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>PromptReps</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.1" class="ltx_p">Unlike previous works that leverage LLMs for document ranking, which are limited to document re-ranking tasks with prompts or rely on contrastive learning to transform a generative LLM into an embedding model for document retrieval, in this work, we aim to directly prompt LLMs to generate both dense embedding representations and sparse bag-of-words representations for document retrieval without any form of extra training effort. To achieve this aim, we devise the following prompt as the input text for LLMs:</p>
</div>
<div id="S3.p2" class="ltx_para ltx_noindent">
<br class="ltx_break">
<p id="S3.p2.1" class="ltx_p">

<span id="S3.p2.1.1" class="ltx_inline-block ltx_parbox ltx_align_middle ltx_framed ltx_framed_rectangle" style="width:411.9pt;">
<span id="S3.p2.1.1.1" class="ltx_p"><span id="S3.p2.1.1.1.1" class="ltx_text ltx_font_bold">&lt;System&gt;</span> <span id="S3.p2.1.1.1.2" class="ltx_text ltx_font_italic">You are an AI assistant that can understand human language.</span></span>
<span id="S3.p2.1.1.2" class="ltx_p"><span id="S3.p2.1.1.2.1" class="ltx_text ltx_font_bold">&lt;User&gt;</span> <span id="S3.p2.1.1.2.2" class="ltx_text ltx_font_italic">Passage "</span><span id="S3.p2.1.1.2.3" class="ltx_text ltx_font_bold">[text]</span><span id="S3.p2.1.1.2.4" class="ltx_text ltx_font_italic">". Use one most important word to represent the passage in retrieval task. Make sure your word is in lowercase.</span></span>
<span id="S3.p2.1.1.3" class="ltx_p"><span id="S3.p2.1.1.3.1" class="ltx_text ltx_font_bold">&lt;Assistant&gt;</span> <span id="S3.p2.1.1.3.2" class="ltx_text ltx_font_italic">The word is: "</span></span>
</span>
</p>
</div>
<div id="S3.p3" class="ltx_para ltx_noindent">
<p id="S3.p3.1" class="ltx_p">where &lt;System&gt; &lt;User&gt; and &lt;Assistant&gt; are LLM pre-defined conversational prefix tokens and [text] is the placeholder for passage text.
When using this prompt for text generation, the language model needs to find a single word in its token vocabulary that can best represent the given passage to generate. However, since there could be multiple words to represent the passage, there might be multiple tokens in the vocabulary that have a high probability of being sampled by the language model. Such a distribution over the vocabulary, which often refers to
"logits", could potentially provide a good representation of the given passage. In addition, since the logits are computed by the last layer hidden state<span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span>Often through dot product between last hidden state with all token embeddings.</span></span></span> of the last input token (‘"’), which is a dense vector embedding, it can also serve as a dense representation of the passage.</p>
</div>
<div id="S3.p4" class="ltx_para">
<p id="S3.p4.1" class="ltx_p">Based on the above intuition, we develop a sparse + dense hybrid document retrieval system by utilizing both the next token logits and the last layer hidden states outputted by the LLM with our designed prompt.</p>
</div>
<div id="S3.p5" class="ltx_para">
<p id="S3.p5.1" class="ltx_p">Specifically, during the document indexing phase, we pass all the documents with our prompt into the LLM to get output hidden states and logits. To build a sparse retrieval pipeline with logits, we first need to sparsify the logits representation to be able to perform efficient sparse retrieval. This is because logits originally had values for all tokens in the vocabulary, essentially forming dense vectors with dimensions equal to the vocabulary size. To sparsify the logit representations for sparse retrieval, we perform following steps:</p>
<ol id="S3.I1" class="ltx_enumerate">
<li id="S3.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S3.I1.i1.p1" class="ltx_para">
<p id="S3.I1.i1.p1.1" class="ltx_p">Lowercase the input document text to align with the phrase "Make sure your word is in lowercase." in the prompt since this phrase skewed the sampling distribution towards lowercase tokens (a "sparser" distribution). We then utilize the NLTK toolkit <cite class="ltx_cite ltx_citemacro_cite">Bird and Loper (<a href="#bib.bib6" title="" class="ltx_ref">2004</a>)</cite> to extract all words in the document, filtering out standard English stopwords and punctuation.</p>
</div>
</li>
<li id="S3.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S3.I1.i2.p1" class="ltx_para">
<p id="S3.I1.i2.p1.1" class="ltx_p">Next, we use the LLM’s tokenizer to tokenize each extracted word and obtain their token IDs<span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span>Note that many words may be split into sub-tokens, resulting in multiple token IDs, all of which are considered in the logits</span></span></span>. We retain only the values corresponding to the obtained token IDs in the logits and set the rest of the dimensions to zero, thereby considering only tokens present in the documents, thus enabling exact term matching in retrieval.</p>
</div>
</li>
<li id="S3.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">3.</span> 
<div id="S3.I1.i3.p1" class="ltx_para">
<p id="S3.I1.i3.p1.1" class="ltx_p">Next, we follow the SPLADE recipe <cite class="ltx_cite ltx_citemacro_cite">Formal et al. (<a href="#bib.bib11" title="" class="ltx_ref">2021</a>)</cite>, using the ReLU function to remove dimensions with negative values and applying log-saturation to the logits to prevent certain tokens from dominating. To further enhance the sparsity of logits, we only keep tokens within the top 128 values if the logits had more than 128 non-zero values after the previous steps.</p>
</div>
</li>
<li id="S3.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">4.</span> 
<div id="S3.I1.i4.p1" class="ltx_para">
<p id="S3.I1.i4.p1.1" class="ltx_p">Finally, the logits were quantized by simply multiplying the original values by 100 and taking the integer operation on that, and the obtained values represent the weights of corresponding tokens.</p>
</div>
</li>
</ol>
<p id="S3.p5.2" class="ltx_p">With these adjustments, the logits representations of documents are heavily sparsified, allowing for efficient sparse retrieval with a standard inverted index.</p>
</div>
<div id="S3.p6" class="ltx_para">
<p id="S3.p6.1" class="ltx_p">For dense retrieval, we directly use the hidden states as the embedding of the documents. For indexing these embeddings, we simply normalize all the embeddings and add them into an Approximate Nearest search (ANN) vector index.</p>
</div>
<div id="S3.p7" class="ltx_para">
<p id="S3.p7.1" class="ltx_p">At query time, we process the queries exactly the same as the documents, with the only exception being that the term "passage" in the prompt is replaced with "query". The dense representation of the query is utilized for semantic search via the ANN index, while the sparse representation of the query is employed for exact term matching via the inverted index. Following previous work <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a href="#bib.bib42" title="" class="ltx_ref">2021</a>)</cite>, we compute the final document scores by applying min-max normalization to both dense and sparse document scores. These normalized scores are then linearly interpolated with equal weights to produce the final document scores.</p>
</div>
<figure id="S3.T1" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>nDCG@10 scores of BEIR 13 publicly available datasets. The best scores of methods without interpolating with BM25 are highlighted in bold. Scores of E5-PT<sub id="S3.T1.5.1" class="ltx_sub"><span id="S3.T1.5.1.1" class="ltx_text ltx_font_italic">large</span></sub> and LLM2Vec are taken directly from the original papers.</figcaption>
<div id="S3.T1.3" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:173.9pt;vertical-align:-0.6pt;"><span class="ltx_transformed_inner" style="transform:translate(-166.0pt,66.3pt) scale(0.566370257730902,0.566370257730902) ;">
<table id="S3.T1.3.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S3.T1.3.1.2.1" class="ltx_tr">
<th id="S3.T1.3.1.2.1.1" class="ltx_td ltx_th ltx_th_column ltx_border_r ltx_border_t"></th>
<th id="S3.T1.3.1.2.1.2" class="ltx_td ltx_th ltx_th_column ltx_border_r ltx_border_t"></th>
<th id="S3.T1.3.1.2.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" colspan="2">Contrastive pre-training</th>
<th id="S3.T1.3.1.2.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" colspan="13">PromptReps(ours)</th>
</tr>
<tr id="S3.T1.3.1.3.2" class="ltx_tr">
<th id="S3.T1.3.1.3.2.1" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_r">LLM</th>
<th id="S3.T1.3.1.3.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">-</th>
<th id="S3.T1.3.1.3.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">BERT-330M</th>
<th id="S3.T1.3.1.3.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">Mistral-7B</th>
<th id="S3.T1.3.1.3.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r" colspan="3">Mistral-7B-Instruct</th>
<th id="S3.T1.3.1.3.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r" colspan="3">Phi3-mini-3.8B-Instruct</th>
<th id="S3.T1.3.1.3.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r" colspan="3">Llama3-8B-Instruct</th>
<th id="S3.T1.3.1.3.2.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r" colspan="4">Llama3-70B-Instruct</th>
</tr>
<tr id="S3.T1.3.1.1" class="ltx_tr">
<th id="S3.T1.3.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_r">Dataset</th>
<th id="S3.T1.3.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">BM25</th>
<th id="S3.T1.3.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">E5-PT<sub id="S3.T1.3.1.1.1.1" class="ltx_sub"><span id="S3.T1.3.1.1.1.1.1" class="ltx_text ltx_font_italic">large</span></sub>
</th>
<th id="S3.T1.3.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">LLM2Vec</th>
<th id="S3.T1.3.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">Dense</th>
<th id="S3.T1.3.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">Sparse</th>
<th id="S3.T1.3.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">Hybrid</th>
<th id="S3.T1.3.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">Dense</th>
<th id="S3.T1.3.1.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">Sparse</th>
<th id="S3.T1.3.1.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">Hybrid</th>
<th id="S3.T1.3.1.1.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">Dense</th>
<th id="S3.T1.3.1.1.12" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">Sparse</th>
<th id="S3.T1.3.1.1.13" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">Hybrid</th>
<th id="S3.T1.3.1.1.14" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">Dense</th>
<th id="S3.T1.3.1.1.15" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">Sparse</th>
<th id="S3.T1.3.1.1.16" class="ltx_td ltx_align_center ltx_th ltx_th_column">Hybrid</th>
<th id="S3.T1.3.1.1.17" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">+BM25</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S3.T1.3.1.4.1" class="ltx_tr">
<td id="S3.T1.3.1.4.1.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">arguana</td>
<td id="S3.T1.3.1.4.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">39.70</td>
<td id="S3.T1.3.1.4.1.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">44.4</td>
<td id="S3.T1.3.1.4.1.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="S3.T1.3.1.4.1.4.1" class="ltx_text ltx_font_bold">51.00</span></td>
<td id="S3.T1.3.1.4.1.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">21.02</td>
<td id="S3.T1.3.1.4.1.6" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">32.41</td>
<td id="S3.T1.3.1.4.1.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">32.81</td>
<td id="S3.T1.3.1.4.1.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">38.00</td>
<td id="S3.T1.3.1.4.1.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">21.51</td>
<td id="S3.T1.3.1.4.1.10" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">35.68</td>
<td id="S3.T1.3.1.4.1.11" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">32.89</td>
<td id="S3.T1.3.1.4.1.12" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">24.25</td>
<td id="S3.T1.3.1.4.1.13" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">35.47</td>
<td id="S3.T1.3.1.4.1.14" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">37.38</td>
<td id="S3.T1.3.1.4.1.15" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">26.15</td>
<td id="S3.T1.3.1.4.1.16" class="ltx_td ltx_align_center ltx_border_t">38.29</td>
<td id="S3.T1.3.1.4.1.17" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">42.28</td>
</tr>
<tr id="S3.T1.3.1.5.2" class="ltx_tr">
<td id="S3.T1.3.1.5.2.1" class="ltx_td ltx_align_left ltx_border_r">climate-fever</td>
<td id="S3.T1.3.1.5.2.2" class="ltx_td ltx_align_center ltx_border_r">16.51</td>
<td id="S3.T1.3.1.5.2.3" class="ltx_td ltx_align_center ltx_border_r">15.7</td>
<td id="S3.T1.3.1.5.2.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T1.3.1.5.2.4.1" class="ltx_text ltx_font_bold">22.97</span></td>
<td id="S3.T1.3.1.5.2.5" class="ltx_td ltx_align_center ltx_border_r">5.31</td>
<td id="S3.T1.3.1.5.2.6" class="ltx_td ltx_align_center ltx_border_r">11.17</td>
<td id="S3.T1.3.1.5.2.7" class="ltx_td ltx_align_center ltx_border_r">10.57</td>
<td id="S3.T1.3.1.5.2.8" class="ltx_td ltx_align_center ltx_border_r">18.99</td>
<td id="S3.T1.3.1.5.2.9" class="ltx_td ltx_align_center ltx_border_r">11.39</td>
<td id="S3.T1.3.1.5.2.10" class="ltx_td ltx_align_center ltx_border_r">21.12</td>
<td id="S3.T1.3.1.5.2.11" class="ltx_td ltx_align_center ltx_border_r">23.47</td>
<td id="S3.T1.3.1.5.2.12" class="ltx_td ltx_align_center ltx_border_r">9.52</td>
<td id="S3.T1.3.1.5.2.13" class="ltx_td ltx_align_center ltx_border_r">22.20</td>
<td id="S3.T1.3.1.5.2.14" class="ltx_td ltx_align_center ltx_border_r">21.21</td>
<td id="S3.T1.3.1.5.2.15" class="ltx_td ltx_align_center ltx_border_r">12.01</td>
<td id="S3.T1.3.1.5.2.16" class="ltx_td ltx_align_center">22.56</td>
<td id="S3.T1.3.1.5.2.17" class="ltx_td ltx_align_center ltx_border_r">22.96</td>
</tr>
<tr id="S3.T1.3.1.6.3" class="ltx_tr">
<td id="S3.T1.3.1.6.3.1" class="ltx_td ltx_align_left ltx_border_r">dbpedia</td>
<td id="S3.T1.3.1.6.3.2" class="ltx_td ltx_align_center ltx_border_r">31.80</td>
<td id="S3.T1.3.1.6.3.3" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T1.3.1.6.3.3.1" class="ltx_text ltx_font_bold">37.1</span></td>
<td id="S3.T1.3.1.6.3.4" class="ltx_td ltx_align_center ltx_border_r">25.48</td>
<td id="S3.T1.3.1.6.3.5" class="ltx_td ltx_align_center ltx_border_r">8.23</td>
<td id="S3.T1.3.1.6.3.6" class="ltx_td ltx_align_center ltx_border_r">22.80</td>
<td id="S3.T1.3.1.6.3.7" class="ltx_td ltx_align_center ltx_border_r">17.87</td>
<td id="S3.T1.3.1.6.3.8" class="ltx_td ltx_align_center ltx_border_r">26.09</td>
<td id="S3.T1.3.1.6.3.9" class="ltx_td ltx_align_center ltx_border_r">24.72</td>
<td id="S3.T1.3.1.6.3.10" class="ltx_td ltx_align_center ltx_border_r">32.21</td>
<td id="S3.T1.3.1.6.3.11" class="ltx_td ltx_align_center ltx_border_r">30.95</td>
<td id="S3.T1.3.1.6.3.12" class="ltx_td ltx_align_center ltx_border_r">28.11</td>
<td id="S3.T1.3.1.6.3.13" class="ltx_td ltx_align_center ltx_border_r">36.30</td>
<td id="S3.T1.3.1.6.3.14" class="ltx_td ltx_align_center ltx_border_r">31.88</td>
<td id="S3.T1.3.1.6.3.15" class="ltx_td ltx_align_center ltx_border_r">28.69</td>
<td id="S3.T1.3.1.6.3.16" class="ltx_td ltx_align_center">36.59</td>
<td id="S3.T1.3.1.6.3.17" class="ltx_td ltx_align_center ltx_border_r">40.48</td>
</tr>
<tr id="S3.T1.3.1.7.4" class="ltx_tr">
<td id="S3.T1.3.1.7.4.1" class="ltx_td ltx_align_left ltx_border_r">fever</td>
<td id="S3.T1.3.1.7.4.2" class="ltx_td ltx_align_center ltx_border_r">65.13</td>
<td id="S3.T1.3.1.7.4.3" class="ltx_td ltx_align_center ltx_border_r">68.6</td>
<td id="S3.T1.3.1.7.4.4" class="ltx_td ltx_align_center ltx_border_r">45.11</td>
<td id="S3.T1.3.1.7.4.5" class="ltx_td ltx_align_center ltx_border_r">20.03</td>
<td id="S3.T1.3.1.7.4.6" class="ltx_td ltx_align_center ltx_border_r">50.07</td>
<td id="S3.T1.3.1.7.4.7" class="ltx_td ltx_align_center ltx_border_r">47.23</td>
<td id="S3.T1.3.1.7.4.8" class="ltx_td ltx_align_center ltx_border_r">44.18</td>
<td id="S3.T1.3.1.7.4.9" class="ltx_td ltx_align_center ltx_border_r">51.73</td>
<td id="S3.T1.3.1.7.4.10" class="ltx_td ltx_align_center ltx_border_r">64.92</td>
<td id="S3.T1.3.1.7.4.11" class="ltx_td ltx_align_center ltx_border_r">59.43</td>
<td id="S3.T1.3.1.7.4.12" class="ltx_td ltx_align_center ltx_border_r">51.60</td>
<td id="S3.T1.3.1.7.4.13" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T1.3.1.7.4.13.1" class="ltx_text ltx_font_bold">71.61</span></td>
<td id="S3.T1.3.1.7.4.14" class="ltx_td ltx_align_center ltx_border_r">57.62</td>
<td id="S3.T1.3.1.7.4.15" class="ltx_td ltx_align_center ltx_border_r">54.77</td>
<td id="S3.T1.3.1.7.4.16" class="ltx_td ltx_align_center">71.52</td>
<td id="S3.T1.3.1.7.4.17" class="ltx_td ltx_align_center ltx_border_r">76.37</td>
</tr>
<tr id="S3.T1.3.1.8.5" class="ltx_tr">
<td id="S3.T1.3.1.8.5.1" class="ltx_td ltx_align_left ltx_border_r">fiqa</td>
<td id="S3.T1.3.1.8.5.2" class="ltx_td ltx_align_center ltx_border_r">23.61</td>
<td id="S3.T1.3.1.8.5.3" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T1.3.1.8.5.3.1" class="ltx_text ltx_font_bold">43.2</span></td>
<td id="S3.T1.3.1.8.5.4" class="ltx_td ltx_align_center ltx_border_r">27.24</td>
<td id="S3.T1.3.1.8.5.5" class="ltx_td ltx_align_center ltx_border_r">8.31</td>
<td id="S3.T1.3.1.8.5.6" class="ltx_td ltx_align_center ltx_border_r">18.63</td>
<td id="S3.T1.3.1.8.5.7" class="ltx_td ltx_align_center ltx_border_r">15.75</td>
<td id="S3.T1.3.1.8.5.8" class="ltx_td ltx_align_center ltx_border_r">21.08</td>
<td id="S3.T1.3.1.8.5.9" class="ltx_td ltx_align_center ltx_border_r">20.40</td>
<td id="S3.T1.3.1.8.5.10" class="ltx_td ltx_align_center ltx_border_r">27.01</td>
<td id="S3.T1.3.1.8.5.11" class="ltx_td ltx_align_center ltx_border_r">24.79</td>
<td id="S3.T1.3.1.8.5.12" class="ltx_td ltx_align_center ltx_border_r">19.66</td>
<td id="S3.T1.3.1.8.5.13" class="ltx_td ltx_align_center ltx_border_r">29.54</td>
<td id="S3.T1.3.1.8.5.14" class="ltx_td ltx_align_center ltx_border_r">29.99</td>
<td id="S3.T1.3.1.8.5.15" class="ltx_td ltx_align_center ltx_border_r">22.22</td>
<td id="S3.T1.3.1.8.5.16" class="ltx_td ltx_align_center">33.34</td>
<td id="S3.T1.3.1.8.5.17" class="ltx_td ltx_align_center ltx_border_r">33.20</td>
</tr>
<tr id="S3.T1.3.1.9.6" class="ltx_tr">
<td id="S3.T1.3.1.9.6.1" class="ltx_td ltx_align_left ltx_border_r">hotpotqa</td>
<td id="S3.T1.3.1.9.6.2" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T1.3.1.9.6.2.1" class="ltx_text ltx_font_bold">63.30</span></td>
<td id="S3.T1.3.1.9.6.3" class="ltx_td ltx_align_center ltx_border_r">52.2</td>
<td id="S3.T1.3.1.9.6.4" class="ltx_td ltx_align_center ltx_border_r">54.54</td>
<td id="S3.T1.3.1.9.6.5" class="ltx_td ltx_align_center ltx_border_r">2.98</td>
<td id="S3.T1.3.1.9.6.6" class="ltx_td ltx_align_center ltx_border_r">32.15</td>
<td id="S3.T1.3.1.9.6.7" class="ltx_td ltx_align_center ltx_border_r">20.62</td>
<td id="S3.T1.3.1.9.6.8" class="ltx_td ltx_align_center ltx_border_r">25.15</td>
<td id="S3.T1.3.1.9.6.9" class="ltx_td ltx_align_center ltx_border_r">42.75</td>
<td id="S3.T1.3.1.9.6.10" class="ltx_td ltx_align_center ltx_border_r">46.62</td>
<td id="S3.T1.3.1.9.6.11" class="ltx_td ltx_align_center ltx_border_r">28.07</td>
<td id="S3.T1.3.1.9.6.12" class="ltx_td ltx_align_center ltx_border_r">45.82</td>
<td id="S3.T1.3.1.9.6.13" class="ltx_td ltx_align_center ltx_border_r">51.27</td>
<td id="S3.T1.3.1.9.6.14" class="ltx_td ltx_align_center ltx_border_r">30.94</td>
<td id="S3.T1.3.1.9.6.15" class="ltx_td ltx_align_center ltx_border_r">44.91</td>
<td id="S3.T1.3.1.9.6.16" class="ltx_td ltx_align_center">50.98</td>
<td id="S3.T1.3.1.9.6.17" class="ltx_td ltx_align_center ltx_border_r">66.42</td>
</tr>
<tr id="S3.T1.3.1.10.7" class="ltx_tr">
<td id="S3.T1.3.1.10.7.1" class="ltx_td ltx_align_left ltx_border_r">nfcorpus</td>
<td id="S3.T1.3.1.10.7.2" class="ltx_td ltx_align_center ltx_border_r">32.18</td>
<td id="S3.T1.3.1.10.7.3" class="ltx_td ltx_align_center ltx_border_r">33.7</td>
<td id="S3.T1.3.1.10.7.4" class="ltx_td ltx_align_center ltx_border_r">27.16</td>
<td id="S3.T1.3.1.10.7.5" class="ltx_td ltx_align_center ltx_border_r">14.41</td>
<td id="S3.T1.3.1.10.7.6" class="ltx_td ltx_align_center ltx_border_r">27.81</td>
<td id="S3.T1.3.1.10.7.7" class="ltx_td ltx_align_center ltx_border_r">23.55</td>
<td id="S3.T1.3.1.10.7.8" class="ltx_td ltx_align_center ltx_border_r">27.80</td>
<td id="S3.T1.3.1.10.7.9" class="ltx_td ltx_align_center ltx_border_r">29.38</td>
<td id="S3.T1.3.1.10.7.10" class="ltx_td ltx_align_center ltx_border_r">33.25</td>
<td id="S3.T1.3.1.10.7.11" class="ltx_td ltx_align_center ltx_border_r">26.74</td>
<td id="S3.T1.3.1.10.7.12" class="ltx_td ltx_align_center ltx_border_r">27.48</td>
<td id="S3.T1.3.1.10.7.13" class="ltx_td ltx_align_center ltx_border_r">31.36</td>
<td id="S3.T1.3.1.10.7.14" class="ltx_td ltx_align_center ltx_border_r">31.93</td>
<td id="S3.T1.3.1.10.7.15" class="ltx_td ltx_align_center ltx_border_r">29.35</td>
<td id="S3.T1.3.1.10.7.16" class="ltx_td ltx_align_center"><span id="S3.T1.3.1.10.7.16.1" class="ltx_text ltx_font_bold">34.03</span></td>
<td id="S3.T1.3.1.10.7.17" class="ltx_td ltx_align_center ltx_border_r">36.19</td>
</tr>
<tr id="S3.T1.3.1.11.8" class="ltx_tr">
<td id="S3.T1.3.1.11.8.1" class="ltx_td ltx_align_left ltx_border_r">nq</td>
<td id="S3.T1.3.1.11.8.2" class="ltx_td ltx_align_center ltx_border_r">30.55</td>
<td id="S3.T1.3.1.11.8.3" class="ltx_td ltx_align_center ltx_border_r">41.7</td>
<td id="S3.T1.3.1.11.8.4" class="ltx_td ltx_align_center ltx_border_r">34.16</td>
<td id="S3.T1.3.1.11.8.5" class="ltx_td ltx_align_center ltx_border_r">7.12</td>
<td id="S3.T1.3.1.11.8.6" class="ltx_td ltx_align_center ltx_border_r">19.93</td>
<td id="S3.T1.3.1.11.8.7" class="ltx_td ltx_align_center ltx_border_r">16.62</td>
<td id="S3.T1.3.1.11.8.8" class="ltx_td ltx_align_center ltx_border_r">26.47</td>
<td id="S3.T1.3.1.11.8.9" class="ltx_td ltx_align_center ltx_border_r">25.57</td>
<td id="S3.T1.3.1.11.8.10" class="ltx_td ltx_align_center ltx_border_r">34.78</td>
<td id="S3.T1.3.1.11.8.11" class="ltx_td ltx_align_center ltx_border_r">35.02</td>
<td id="S3.T1.3.1.11.8.12" class="ltx_td ltx_align_center ltx_border_r">28.34</td>
<td id="S3.T1.3.1.11.8.13" class="ltx_td ltx_align_center ltx_border_r">41.62</td>
<td id="S3.T1.3.1.11.8.14" class="ltx_td ltx_align_center ltx_border_r">43.38</td>
<td id="S3.T1.3.1.11.8.15" class="ltx_td ltx_align_center ltx_border_r">30.76</td>
<td id="S3.T1.3.1.11.8.16" class="ltx_td ltx_align_center"><span id="S3.T1.3.1.11.8.16.1" class="ltx_text ltx_font_bold">45.98</span></td>
<td id="S3.T1.3.1.11.8.17" class="ltx_td ltx_align_center ltx_border_r">44.22</td>
</tr>
<tr id="S3.T1.3.1.12.9" class="ltx_tr">
<td id="S3.T1.3.1.12.9.1" class="ltx_td ltx_align_left ltx_border_r">quora</td>
<td id="S3.T1.3.1.12.9.2" class="ltx_td ltx_align_center ltx_border_r">78.86</td>
<td id="S3.T1.3.1.12.9.3" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T1.3.1.12.9.3.1" class="ltx_text ltx_font_bold">86.1</span></td>
<td id="S3.T1.3.1.12.9.4" class="ltx_td ltx_align_center ltx_border_r">84.40</td>
<td id="S3.T1.3.1.12.9.5" class="ltx_td ltx_align_center ltx_border_r">53.38</td>
<td id="S3.T1.3.1.12.9.6" class="ltx_td ltx_align_center ltx_border_r">61.95</td>
<td id="S3.T1.3.1.12.9.7" class="ltx_td ltx_align_center ltx_border_r">66.65</td>
<td id="S3.T1.3.1.12.9.8" class="ltx_td ltx_align_center ltx_border_r">80.71</td>
<td id="S3.T1.3.1.12.9.9" class="ltx_td ltx_align_center ltx_border_r">67.79</td>
<td id="S3.T1.3.1.12.9.10" class="ltx_td ltx_align_center ltx_border_r">82.11</td>
<td id="S3.T1.3.1.12.9.11" class="ltx_td ltx_align_center ltx_border_r">77.47</td>
<td id="S3.T1.3.1.12.9.12" class="ltx_td ltx_align_center ltx_border_r">69.02</td>
<td id="S3.T1.3.1.12.9.13" class="ltx_td ltx_align_center ltx_border_r">82.47</td>
<td id="S3.T1.3.1.12.9.14" class="ltx_td ltx_align_center ltx_border_r">75.27</td>
<td id="S3.T1.3.1.12.9.15" class="ltx_td ltx_align_center ltx_border_r">71.57</td>
<td id="S3.T1.3.1.12.9.16" class="ltx_td ltx_align_center">81.81</td>
<td id="S3.T1.3.1.12.9.17" class="ltx_td ltx_align_center ltx_border_r">84.84</td>
</tr>
<tr id="S3.T1.3.1.13.10" class="ltx_tr">
<td id="S3.T1.3.1.13.10.1" class="ltx_td ltx_align_left ltx_border_r">scidocs</td>
<td id="S3.T1.3.1.13.10.2" class="ltx_td ltx_align_center ltx_border_r">14.90</td>
<td id="S3.T1.3.1.13.10.3" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T1.3.1.13.10.3.1" class="ltx_text ltx_font_bold">21.8</span></td>
<td id="S3.T1.3.1.13.10.4" class="ltx_td ltx_align_center ltx_border_r">15.35</td>
<td id="S3.T1.3.1.13.10.5" class="ltx_td ltx_align_center ltx_border_r">6.54</td>
<td id="S3.T1.3.1.13.10.6" class="ltx_td ltx_align_center ltx_border_r">11.42</td>
<td id="S3.T1.3.1.13.10.7" class="ltx_td ltx_align_center ltx_border_r">10.28</td>
<td id="S3.T1.3.1.13.10.8" class="ltx_td ltx_align_center ltx_border_r">13.35</td>
<td id="S3.T1.3.1.13.10.9" class="ltx_td ltx_align_center ltx_border_r">12.24</td>
<td id="S3.T1.3.1.13.10.10" class="ltx_td ltx_align_center ltx_border_r">15.58</td>
<td id="S3.T1.3.1.13.10.11" class="ltx_td ltx_align_center ltx_border_r">16.77</td>
<td id="S3.T1.3.1.13.10.12" class="ltx_td ltx_align_center ltx_border_r">11.52</td>
<td id="S3.T1.3.1.13.10.13" class="ltx_td ltx_align_center ltx_border_r">16.93</td>
<td id="S3.T1.3.1.13.10.14" class="ltx_td ltx_align_center ltx_border_r">18.31</td>
<td id="S3.T1.3.1.13.10.15" class="ltx_td ltx_align_center ltx_border_r">13.22</td>
<td id="S3.T1.3.1.13.10.16" class="ltx_td ltx_align_center">17.96</td>
<td id="S3.T1.3.1.13.10.17" class="ltx_td ltx_align_center ltx_border_r">17.73</td>
</tr>
<tr id="S3.T1.3.1.14.11" class="ltx_tr">
<td id="S3.T1.3.1.14.11.1" class="ltx_td ltx_align_left ltx_border_r">scifact</td>
<td id="S3.T1.3.1.14.11.2" class="ltx_td ltx_align_center ltx_border_r">67.89</td>
<td id="S3.T1.3.1.14.11.3" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T1.3.1.14.11.3.1" class="ltx_text ltx_font_bold">72.3</span></td>
<td id="S3.T1.3.1.14.11.4" class="ltx_td ltx_align_center ltx_border_r">68.67</td>
<td id="S3.T1.3.1.14.11.5" class="ltx_td ltx_align_center ltx_border_r">22.71</td>
<td id="S3.T1.3.1.14.11.6" class="ltx_td ltx_align_center ltx_border_r">55.27</td>
<td id="S3.T1.3.1.14.11.7" class="ltx_td ltx_align_center ltx_border_r">48.15</td>
<td id="S3.T1.3.1.14.11.8" class="ltx_td ltx_align_center ltx_border_r">51.69</td>
<td id="S3.T1.3.1.14.11.9" class="ltx_td ltx_align_center ltx_border_r">59.94</td>
<td id="S3.T1.3.1.14.11.10" class="ltx_td ltx_align_center ltx_border_r">64.86</td>
<td id="S3.T1.3.1.14.11.11" class="ltx_td ltx_align_center ltx_border_r">54.26</td>
<td id="S3.T1.3.1.14.11.12" class="ltx_td ltx_align_center ltx_border_r">59.06</td>
<td id="S3.T1.3.1.14.11.13" class="ltx_td ltx_align_center ltx_border_r">65.61</td>
<td id="S3.T1.3.1.14.11.14" class="ltx_td ltx_align_center ltx_border_r">58.92</td>
<td id="S3.T1.3.1.14.11.15" class="ltx_td ltx_align_center ltx_border_r">61.53</td>
<td id="S3.T1.3.1.14.11.16" class="ltx_td ltx_align_center">67.41</td>
<td id="S3.T1.3.1.14.11.17" class="ltx_td ltx_align_center ltx_border_r">72.09</td>
</tr>
<tr id="S3.T1.3.1.15.12" class="ltx_tr">
<td id="S3.T1.3.1.15.12.1" class="ltx_td ltx_align_left ltx_border_r">trec-covid</td>
<td id="S3.T1.3.1.15.12.2" class="ltx_td ltx_align_center ltx_border_r">59.47</td>
<td id="S3.T1.3.1.15.12.3" class="ltx_td ltx_align_center ltx_border_r">61.8</td>
<td id="S3.T1.3.1.15.12.4" class="ltx_td ltx_align_center ltx_border_r">55.66</td>
<td id="S3.T1.3.1.15.12.5" class="ltx_td ltx_align_center ltx_border_r">34.94</td>
<td id="S3.T1.3.1.15.12.6" class="ltx_td ltx_align_center ltx_border_r">62.03</td>
<td id="S3.T1.3.1.15.12.7" class="ltx_td ltx_align_center ltx_border_r">53.79</td>
<td id="S3.T1.3.1.15.12.8" class="ltx_td ltx_align_center ltx_border_r">51.16</td>
<td id="S3.T1.3.1.15.12.9" class="ltx_td ltx_align_center ltx_border_r">56.53</td>
<td id="S3.T1.3.1.15.12.10" class="ltx_td ltx_align_center ltx_border_r">62.48</td>
<td id="S3.T1.3.1.15.12.11" class="ltx_td ltx_align_center ltx_border_r">63.54</td>
<td id="S3.T1.3.1.15.12.12" class="ltx_td ltx_align_center ltx_border_r">55.19</td>
<td id="S3.T1.3.1.15.12.13" class="ltx_td ltx_align_center ltx_border_r">68.54</td>
<td id="S3.T1.3.1.15.12.14" class="ltx_td ltx_align_center ltx_border_r">71.47</td>
<td id="S3.T1.3.1.15.12.15" class="ltx_td ltx_align_center ltx_border_r">63.19</td>
<td id="S3.T1.3.1.15.12.16" class="ltx_td ltx_align_center"><span id="S3.T1.3.1.15.12.16.1" class="ltx_text ltx_font_bold">77.13</span></td>
<td id="S3.T1.3.1.15.12.17" class="ltx_td ltx_align_center ltx_border_r">77.94</td>
</tr>
<tr id="S3.T1.3.1.16.13" class="ltx_tr">
<td id="S3.T1.3.1.16.13.1" class="ltx_td ltx_align_left ltx_border_r">touche</td>
<td id="S3.T1.3.1.16.13.2" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T1.3.1.16.13.2.1" class="ltx_text ltx_font_bold">44.22</span></td>
<td id="S3.T1.3.1.16.13.3" class="ltx_td ltx_align_center ltx_border_r">19.8</td>
<td id="S3.T1.3.1.16.13.4" class="ltx_td ltx_align_center ltx_border_r">6.54</td>
<td id="S3.T1.3.1.16.13.5" class="ltx_td ltx_align_center ltx_border_r">9.59</td>
<td id="S3.T1.3.1.16.13.6" class="ltx_td ltx_align_center ltx_border_r">19.47</td>
<td id="S3.T1.3.1.16.13.7" class="ltx_td ltx_align_center ltx_border_r">18.36</td>
<td id="S3.T1.3.1.16.13.8" class="ltx_td ltx_align_center ltx_border_r">12.75</td>
<td id="S3.T1.3.1.16.13.9" class="ltx_td ltx_align_center ltx_border_r">19.17</td>
<td id="S3.T1.3.1.16.13.10" class="ltx_td ltx_align_center ltx_border_r">19.58</td>
<td id="S3.T1.3.1.16.13.11" class="ltx_td ltx_align_center ltx_border_r">14.92</td>
<td id="S3.T1.3.1.16.13.12" class="ltx_td ltx_align_center ltx_border_r">16.46</td>
<td id="S3.T1.3.1.16.13.13" class="ltx_td ltx_align_center ltx_border_r">19.24</td>
<td id="S3.T1.3.1.16.13.14" class="ltx_td ltx_align_center ltx_border_r">18.84</td>
<td id="S3.T1.3.1.16.13.15" class="ltx_td ltx_align_center ltx_border_r">20.12</td>
<td id="S3.T1.3.1.16.13.16" class="ltx_td ltx_align_center">22.85</td>
<td id="S3.T1.3.1.16.13.17" class="ltx_td ltx_align_center ltx_border_r">36.21</td>
</tr>
<tr id="S3.T1.3.1.17.14" class="ltx_tr">
<td id="S3.T1.3.1.17.14.1" class="ltx_td ltx_align_left ltx_border_b ltx_border_r ltx_border_t">avg</td>
<td id="S3.T1.3.1.17.14.2" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">43.70</td>
<td id="S3.T1.3.1.17.14.3" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">44.61</td>
<td id="S3.T1.3.1.17.14.4" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">39.86</td>
<td id="S3.T1.3.1.17.14.5" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">16.51</td>
<td id="S3.T1.3.1.17.14.6" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">32.70</td>
<td id="S3.T1.3.1.17.14.7" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">29.40</td>
<td id="S3.T1.3.1.17.14.8" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">33.65</td>
<td id="S3.T1.3.1.17.14.9" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">34.09</td>
<td id="S3.T1.3.1.17.14.10" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">41.55</td>
<td id="S3.T1.3.1.17.14.11" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">37.48</td>
<td id="S3.T1.3.1.17.14.12" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">34.31</td>
<td id="S3.T1.3.1.17.14.13" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">44.01</td>
<td id="S3.T1.3.1.17.14.14" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">40.55</td>
<td id="S3.T1.3.1.17.14.15" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">36.80</td>
<td id="S3.T1.3.1.17.14.16" class="ltx_td ltx_align_center ltx_border_b ltx_border_t"><span id="S3.T1.3.1.17.14.16.1" class="ltx_text ltx_font_bold">46.19</span></td>
<td id="S3.T1.3.1.17.14.17" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t">50.07</td>
</tr>
</tbody>
</table>
</span></div>
</figure>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Experimental setup</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.1" class="ltx_p"><span id="S4.p1.1.1" class="ltx_text ltx_font_bold">Dataset and evaluation:</span> We evaluate the document ranking effectiveness of both baseline methods and our proposed PromptReps using the BEIR benchmark <cite class="ltx_cite ltx_citemacro_cite">Thakur et al. (<a href="#bib.bib38" title="" class="ltx_ref">2021</a>)</cite>. This benchmark encompasses various IR tasks, providing a heterogeneous evaluation environment. We report nDCG@10 scores, which is the official evaluation metric employed by BEIR.</p>
</div>
<div id="S4.p2" class="ltx_para ltx_noindent">
<p id="S4.p2.1" class="ltx_p"><span id="S4.p2.1.1" class="ltx_text ltx_font_bold">Baselines:</span> We compare PromptReps with strong unsupervised first-stage retrievers including BM25, a classic term frequency-based sparse retrieval method, and E5-PT<sub id="S4.p2.1.2" class="ltx_sub"><span id="S4.p2.1.2.1" class="ltx_text ltx_font_italic">large</span></sub> <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a href="#bib.bib40" title="" class="ltx_ref">2024a</a>)</cite>, a state-of-the-art BERT-based <cite class="ltx_cite ltx_citemacro_cite">Devlin et al. (<a href="#bib.bib8" title="" class="ltx_ref">2019</a>)</cite> dense embedding method trained on 1.3 billion carefully crafted unsupervised text pairs. LLM2Vec <cite class="ltx_cite ltx_citemacro_cite">BehnamGhader et al. (<a href="#bib.bib5" title="" class="ltx_ref">2024</a>)</cite>, an LLM-based dense embedding method trained with bi-directional attention, masked next token prediction, and SimCSE <cite class="ltx_cite ltx_citemacro_cite">Gao et al. (<a href="#bib.bib14" title="" class="ltx_ref">2021</a>)</cite> on the Wikipedia corpus.</p>
</div>
<div id="S4.p3" class="ltx_para ltx_noindent">
<p id="S4.p3.1" class="ltx_p"><span id="S4.p3.1.1" class="ltx_text ltx_font_bold">Implementation of PromptReps:</span> PromptReps is implemented using three base LLMs: Mistral-7b-Instruct-v0.2<span id="footnote4" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">4</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">4</sup><span class="ltx_tag ltx_tag_note">4</span>https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2</span></span></span> <cite class="ltx_cite ltx_citemacro_cite">Jiang et al. (<a href="#bib.bib18" title="" class="ltx_ref">2023a</a>)</cite>, Llama3-8B-Instruct<span id="footnote5" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">5</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">5</sup><span class="ltx_tag ltx_tag_note">5</span>https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct</span></span></span>, Phi-3-mini-4k-instruct<span id="footnote6" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">6</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">6</sup><span class="ltx_tag ltx_tag_note">6</span>https://huggingface.co/microsoft/Phi-3-mini-4k-instruct</span></span></span> <cite class="ltx_cite ltx_citemacro_cite">Abdin et al. (<a href="#bib.bib1" title="" class="ltx_ref">2024</a>)</cite>, and Llama3-70B-Instruct<span id="footnote7" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">7</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">7</sup><span class="ltx_tag ltx_tag_note">7</span>https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct</span></span></span> <cite class="ltx_cite ltx_citemacro_cite">AI@Meta (<a href="#bib.bib2" title="" class="ltx_ref">2024</a>)</cite>. Dense and sparse document and query encoding are implemented using the Huggingface Transformers library <cite class="ltx_cite ltx_citemacro_cite">Wolf et al. (<a href="#bib.bib44" title="" class="ltx_ref">2020</a>)</cite> and the Tevatron toolkit <cite class="ltx_cite ltx_citemacro_cite">Gao et al. (<a href="#bib.bib13" title="" class="ltx_ref">2023b</a>)</cite>. Faiss library <cite class="ltx_cite ltx_citemacro_cite">Douze et al. (<a href="#bib.bib9" title="" class="ltx_ref">2024</a>)</cite> is used to build the ANN index with cosine similarity as the embedding distance metric, and Pyserini <cite class="ltx_cite ltx_citemacro_cite">Lin et al. (<a href="#bib.bib28" title="" class="ltx_ref">2021</a>)</cite> is utilized to construct the inverted index for sparse retrieval. For the dense and sparse ranking hybrid, the Ranx library <cite class="ltx_cite ltx_citemacro_cite">Bassani and Romelli (<a href="#bib.bib4" title="" class="ltx_ref">2022</a>)</cite> is employed. In our experiments, we report dense only, sparse only, and the full hybrid results. In addition, we also report the results of further interpolating BM25 scores with hybrid scores for PromptReps based on Llama3-70B-Instruct.</p>
</div>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Results</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">We present our results in Table <a href="#S3.T1" title="Table 1 ‣ 3 PromptReps ‣ PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.
The first observation highlights that BM25 is a very strong zero-shot retrieval method, capable of outperforming LLM2Vec, based on the Mistral-7B LLM, across numerous datasets, achieving a higher average nDCG@10 score. This outcome implies that even with a large-size LLM, bi-directional attention enabled, additional pre-training, and SimCSE-based unsupervised contrastive training, there remains a gap in transforming a decoder-only LLM into a robust retrieval method.</p>
</div>
<div id="S5.p2" class="ltx_para">
<p id="S5.p2.1" class="ltx_p">On the other hand, E5-PT<sub id="S5.p2.1.1" class="ltx_sub"><span id="S5.p2.1.1.1" class="ltx_text ltx_font_italic">large</span></sub>, based on the BERT-large model, is the first method that can outperform BM25 without any supervised training data. However, it has been trained on a massive, carefully mined text pair dataset with a large batch size, which may require more data-collecting efforts and computational resources than LLM2Vec.</p>
</div>
<div id="S5.p3" class="ltx_para">
<p id="S5.p3.1" class="ltx_p">For our proposed PromptReps, when using Mistral-7B-Instruct LLM, the dense-only retriever performs poorly. However, although still worse than LLM2Vec, the sparse-only retriever performs much better than dense. This finding suggests that, with Mistral-7B-Instruct as the base LLM, using the logits generated by our prompt produces better document representations than hidden states for document retrieval. The hybrid retrieval performs worse than sparse-only in this case due to the poor performance of dense retrieval.</p>
</div>
<div id="S5.p4" class="ltx_para">
<p id="S5.p4.1" class="ltx_p">Surprisingly, when implementing PromptReps with Phi-3-mini-4k-instruct achieved much higher dense-only retrieval effectiveness, closing to its sparse retrieval effectiveness and leading to an impressive hybrid retrieval average score that surpassed LLM2Vec based on Mistral-7B, even though it is only half the size.</p>
</div>
<div id="S5.p5" class="ltx_para">
<p id="S5.p5.1" class="ltx_p">Interestingly, when implementing PromptReps with the state-of-the-art Llama3-8B-Instruct LLM, the dense-only retrieval further improved and achieved a higher score than that of sparse. The sparse-only retrieval effectiveness also improved, but the improvement was much more marginal compared to the improvement on dense. The hybrid of dense and sparse brings notable retrieval effectiveness improvement, surpassing BM25 and approaching the state-of-the-art E5-PT<sub id="S5.p5.1.1" class="ltx_sub"><span id="S5.p5.1.1.1" class="ltx_text ltx_font_italic">large</span></sub>. Notably, this is achieved without any form of extra training but solely relying on prompts.</p>
</div>
<div id="S5.p6" class="ltx_para">
<p id="S5.p6.2" class="ltx_p">The scaling law of LLM <cite class="ltx_cite ltx_citemacro_cite">Kaplan et al. (<a href="#bib.bib20" title="" class="ltx_ref">2020</a>)</cite> is also applied here. When changing Llama3-8B-Instruction to Llama3-70B-Instruction, the retrieval effectiveness of our PromptReps further improved, with the dense-only approach surpassing LLM2Vec and the hybrid approach surpassing E5-PT<sub id="S5.p6.2.1" class="ltx_sub"><span id="S5.p6.2.1.1" class="ltx_text ltx_font_italic">large</span></sub>. When further interpolating the hybrid ranking with BM25 ranking, the average nDCG@10 achieved a remarkable score of <math id="S5.p6.2.m2.1" class="ltx_Math" alttext="50.07" display="inline"><semantics id="S5.p6.2.m2.1a"><mn id="S5.p6.2.m2.1.1" xref="S5.p6.2.m2.1.1.cmml">50.07</mn><annotation-xml encoding="MathML-Content" id="S5.p6.2.m2.1b"><cn type="float" id="S5.p6.2.m2.1.1.cmml" xref="S5.p6.2.m2.1.1">50.07</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.p6.2.m2.1c">50.07</annotation></semantics></math>. Our results demonstrate that it is possible to build a very strong hybrid retrieval system without the need for any unsupervised or supervised training.</p>
</div>
<div id="S5.p7" class="ltx_para">
<p id="S5.p7.1" class="ltx_p"><span id="S5.p7.1.1" class="ltx_text ltx_font_bold">A note on query latency.</span> The best performance of our PromptReps is achieved in the hybrid retrieval setting. Compared to previous works which are dense retrieval only, our method seems to have a higher query latency. However, PromptReps actually only requires a little query latency overhead if implementing dense and sparse search in parallel. The cost of obtaining both dense and sparse representations of our method only requires a single LLM forward inference; the only extra computation is the dot product of the dense vector with the token embeddings, which is very fast on GPU. For document search, since we heavily sparsified the sparse representation, in our experiments, our sparse retrieval is much faster than BM25, and the bottleneck is the dense retrieval. Since the dense and sparse search could be run in parallel and the hybrid operation is just a linear interpolation of both rankings (very fast on CPU), the query latency will depend on the dense retrieval latency, thus very close to the previous methods.</p>
</div>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Discussion</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">Our work demonstrates that prompt engineering for generative LLMs is not only a powerful method for natural language generation tasks but also capable of generating robust dense (embedding) and sparse (bag of words) representations for input texts. We believe this important finding could lead to a series of new research directions.</p>
</div>
<div id="S6.p2" class="ltx_para">
<p id="S6.p2.1" class="ltx_p">For example, techniques like few-shot in-context learning <cite class="ltx_cite ltx_citemacro_cite">Brown et al. (<a href="#bib.bib7" title="" class="ltx_ref">2020</a>)</cite>, chain-of-thought prompting <cite class="ltx_cite ltx_citemacro_cite">Wei et al. (<a href="#bib.bib43" title="" class="ltx_ref">2022</a>)</cite>, and auto-prompt optimization methods <cite class="ltx_cite ltx_citemacro_cite">Yang et al. (<a href="#bib.bib46" title="" class="ltx_ref">2024</a>); Fernando et al. (<a href="#bib.bib10" title="" class="ltx_ref">2023</a>)</cite>, which have proven to be very effective in text-generation tasks, could potentially be leveraged here to enhance embedding generation.</p>
</div>
<div id="S6.p3" class="ltx_para">
<p id="S6.p3.1" class="ltx_p">Moreover, it has been shown that the instruction-following ability of LLMs could be transferred to embedding models with synthetic instruction fine-tuning data <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a href="#bib.bib41" title="" class="ltx_ref">2024b</a>)</cite>. In our work, we always keep the instruction prompt consistent across different information retrieval tasks, which could be sub-optimal. It would be very interesting to investigate how to customize instructions for PromptReps to generate embeddings specific to different domains, tasks, or even to multi-lingual and cross-lingual retrieval settings.</p>
</div>
<div id="S6.p4" class="ltx_para">
<p id="S6.p4.1" class="ltx_p">Finally, our prompting method could be seen as a simple approach to obtaining a better initialization of LLM-based embedding models, which is much more cost-effective than methods requiring further pre-training <cite class="ltx_cite ltx_citemacro_cite">BehnamGhader et al. (<a href="#bib.bib5" title="" class="ltx_ref">2024</a>); Li et al. (<a href="#bib.bib25" title="" class="ltx_ref">2023</a>)</cite>. All the previous contrastive pre-training with paired text data and synthetically generated data could be applied on top of our method and could potentially yield improved LLM-based embedding models. We leave these aspects for future work.</p>
</div>
</section>
<section id="S7" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">7 </span>Conclusion</h2>

<div id="S7.p1" class="ltx_para">
<p id="S7.p1.1" class="ltx_p">In this paper, we introduced PromptReps, a simple yet effective method that prompts LLMs to generate dense and sparse representations for zero-shot document retrieval without any further unsupervised or supervised training. Our work reveals that modern LLMs are effective text encoders by themselves, and prompt engineering is sufficient to stimulate their text encoding ability.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Abdin et al. (2024)</span>
<span class="ltx_bibblock">
Marah Abdin et al. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2404.14219" title="" class="ltx_ref ltx_href">Phi-3 technical report: A
highly capable language model locally on your phone</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2404.14219.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">AI@Meta (2024)</span>
<span class="ltx_bibblock">
AI@Meta. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md" title="" class="ltx_ref ltx_href">Llama 3 model card</a>.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bajaj et al. (2018)</span>
<span class="ltx_bibblock">
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu,
Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg,
Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/1611.09268" title="" class="ltx_ref ltx_href">Ms marco: A human generated
machine reading comprehension dataset</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:1611.09268.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bassani and Romelli (2022)</span>
<span class="ltx_bibblock">
Elias Bassani and Luca Romelli. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.1145/3511808.3557207" title="" class="ltx_ref ltx_href">ranx.fuse: A python
library for metasearch</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 31st ACM International Conference on
Information &amp; Knowledge Management</em>, CIKM ’22, page 4808–4812, New York,
NY, USA. Association for Computing Machinery.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">BehnamGhader et al. (2024)</span>
<span class="ltx_bibblock">
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau,
Nicolas Chapados, and Siva Reddy. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2404.05961" title="" class="ltx_ref ltx_href">Llm2vec: Large language
models are secretly powerful text encoders</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2404.05961.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bird and Loper (2004)</span>
<span class="ltx_bibblock">
Steven Bird and Edward Loper. 2004.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://aclanthology.org/P04-3031" title="" class="ltx_ref ltx_href">NLTK: The natural
language toolkit</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">Proceedings of the ACL Interactive Poster and
Demonstration Sessions</em>, pages 214–217, Barcelona, Spain. Association for
Computational Linguistics.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Brown et al. (2020)</span>
<span class="ltx_bibblock">
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla
Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell,
Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon
Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris
Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess,
Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever,
and Dario Amodei. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf" title="" class="ltx_ref ltx_href">Language models are few-shot learners</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>,
volume 33, pages 1877–1901. Curran Associates, Inc.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Devlin et al. (2019)</span>
<span class="ltx_bibblock">
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/N19-1423" title="" class="ltx_ref ltx_href">BERT: Pre-training of
deep bidirectional transformers for language understanding</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers)</em>, pages 4171–4186,
Minneapolis, Minnesota. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Douze et al. (2024)</span>
<span class="ltx_bibblock">
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy,
Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou.
2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2401.08281" title="" class="ltx_ref ltx_href">The faiss library</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2401.08281.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Fernando et al. (2023)</span>
<span class="ltx_bibblock">
Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim
Rocktäschel. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2309.16797" title="" class="ltx_ref ltx_href">Promptbreeder:
Self-referential self-improvement via prompt evolution</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2309.16797.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Formal et al. (2021)</span>
<span class="ltx_bibblock">
Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.1145/3404835.3463098" title="" class="ltx_ref ltx_href">Splade: Sparse
lexical and expansion model for first stage ranking</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 44th International ACM SIGIR Conference
on Research and Development in Information Retrieval</em>, SIGIR ’21, page
2288–2292, New York, NY, USA. Association for Computing Machinery.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gao et al. (2023a)</span>
<span class="ltx_bibblock">
Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023a.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2023.acl-long.99" title="" class="ltx_ref ltx_href">Precise
zero-shot dense retrieval without relevance labels</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 61st Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)</em>, pages 1762–1777,
Toronto, Canada. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gao et al. (2023b)</span>
<span class="ltx_bibblock">
Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023b.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.1145/3539618.3591805" title="" class="ltx_ref ltx_href">Tevatron: An
efficient and flexible toolkit for neural retrieval</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 46th International ACM SIGIR Conference
on Research and Development in Information Retrieval</em>, SIGIR ’23, page
3120–3124, New York, NY, USA. Association for Computing Machinery.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gao et al. (2021)</span>
<span class="ltx_bibblock">
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2021.emnlp-main.552" title="" class="ltx_ref ltx_href">SimCSE:
Simple contrastive learning of sentence embeddings</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing</em>, pages 6894–6910, Online and Punta Cana,
Dominican Republic. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hendrycks et al. (2021)</span>
<span class="ltx_bibblock">
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn
Song, and Jacob Steinhardt. 2021.

</span>
<span class="ltx_bibblock">Measuring massive multitask language understanding.

</span>
<span class="ltx_bibblock"><em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">Proceedings of the International Conference on Learning
Representations (ICLR)</em>.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hu et al. (2022)</span>
<span class="ltx_bibblock">
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean
Wang, Lu Wang, and Weizhu Chen. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=nZeVKeeFYf9" title="" class="ltx_ref ltx_href">LoRA: Low-rank
adaptation of large language models</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Izacard et al. (2022)</span>
<span class="ltx_bibblock">
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr
Bojanowski, Armand Joulin, and Edouard Grave. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2112.09118" title="" class="ltx_ref ltx_href">Unsupervised dense
information retrieval with contrastive learning</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2112.09118.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jiang et al. (2023a)</span>
<span class="ltx_bibblock">
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford,
Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel,
Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux,
Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,
and William El Sayed. 2023a.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2310.06825" title="" class="ltx_ref ltx_href">Mistral 7b</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2310.06825.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jiang et al. (2023b)</span>
<span class="ltx_bibblock">
Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, and Fuzhen Zhuang.
2023b.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2307.16645" title="" class="ltx_ref ltx_href">Scaling sentence embeddings
with large language models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2307.16645.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kaplan et al. (2020)</span>
<span class="ltx_bibblock">
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon
Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2001.08361" title="" class="ltx_ref ltx_href">Scaling laws for neural
language models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2001.08361.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Karpukhin et al. (2020)</span>
<span class="ltx_bibblock">
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey
Edunov, Danqi Chen, and Wen-tau Yih. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2020.emnlp-main.550" title="" class="ltx_ref ltx_href">Dense
passage retrieval for open-domain question answering</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP)</em>, pages 6769–6781, Online. Association
for Computational Linguistics.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Khattab and Zaharia (2020)</span>
<span class="ltx_bibblock">
Omar Khattab and Matei Zaharia. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.1145/3397271.3401075" title="" class="ltx_ref ltx_href">Colbert: Efficient
and effective passage search via contextualized late interaction over bert</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 43rd International ACM SIGIR Conference
on Research and Development in Information Retrieval</em>, SIGIR ’20, page
39–48, New York, NY, USA. Association for Computing Machinery.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lee et al. (2024)</span>
<span class="ltx_bibblock">
Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R. Cole,
Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, Yi Luan, Sai Meher Karthik
Duddu, Gustavo Hernandez Abrego, Weiqiang Shi, Nithi Gupta, Aditya Kusupati,
Prateek Jain, Siddhartha Reddy Jonnalagadda, Ming-Wei Chang, and Iftekhar
Naim. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2403.20327" title="" class="ltx_ref ltx_href">Gecko: Versatile text
embeddings distilled from large language models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2403.20327.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lei et al. (2024)</span>
<span class="ltx_bibblock">
Yibin Lei, Di Wu, Tianyi Zhou, Tao Shen, Yu Cao, Chongyang Tao, and Andrew
Yates. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2402.18458" title="" class="ltx_ref ltx_href">Meta-task prompting elicits
embedding from large language models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2402.18458.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2023)</span>
<span class="ltx_bibblock">
Chaofan Li, Zheng Liu, Shitao Xiao, and Yingxia Shao. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2312.15503" title="" class="ltx_ref ltx_href">Making large language
models a better foundation for dense retrieval</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2312.15503.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liao et al. (2024)</span>
<span class="ltx_bibblock">
Jiayi Liao, Sihang Li, Zhengyi Yang, Jiancan Wu, Yancheng Yuan, Xiang Wang, and
Xiangnan He. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2312.02445" title="" class="ltx_ref ltx_href">Llara: Large
language-recommendation assistant</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2312.02445.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lin and Ma (2021)</span>
<span class="ltx_bibblock">
Jimmy Lin and Xueguang Ma. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2106.14807" title="" class="ltx_ref ltx_href">A few brief notes on
deepimpact, coil, and a conceptual framework for information retrieval
techniques</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2106.14807.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lin et al. (2021)</span>
<span class="ltx_bibblock">
Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and
Rodrigo Nogueira. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.1145/3404835.3463238" title="" class="ltx_ref ltx_href">Pyserini: A python
toolkit for reproducible information retrieval research with sparse and dense
representations</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 44th International ACM SIGIR Conference
on Research and Development in Information Retrieval</em>, SIGIR ’21, page
2356–2362, New York, NY, USA. Association for Computing Machinery.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2023)</span>
<span class="ltx_bibblock">
Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian,
Hao He, Antong Li, Mengshen He, Zhengliang Liu, Zihao Wu, Lin Zhao, Dajiang
Zhu, Xiang Li, Ning Qiang, Dingang Shen, Tianming Liu, and Bao Ge. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.1016/j.metrad.2023.100017" title="" class="ltx_ref ltx_href">Summary of
chatgpt-related research and perspective towards the future of large language
models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">Meta-Radiology</em>, 1(2):100017.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ma et al. (2023a)</span>
<span class="ltx_bibblock">
Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. 2023a.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2310.08319" title="" class="ltx_ref ltx_href">Fine-tuning llama for
multi-stage text retrieval</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2310.08319.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ma et al. (2023b)</span>
<span class="ltx_bibblock">
Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. 2023b.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2305.02156" title="" class="ltx_ref ltx_href">Zero-shot listwise document
reranking with a large language model</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2305.02156.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mallia et al. (2021)</span>
<span class="ltx_bibblock">
Antonio Mallia, Omar Khattab, Torsten Suel, and Nicola Tonellotto. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.1145/3404835.3463030" title="" class="ltx_ref ltx_href">Learning passage
impacts for inverted indexes</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 44th International ACM SIGIR Conference
on Research and Development in Information Retrieval</em>, SIGIR ’21, page
1723–1727, New York, NY, USA. Association for Computing Machinery.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">OpenAI (2023)</span>
<span class="ltx_bibblock">
OpenAI. 2023.

</span>
<span class="ltx_bibblock">GPT-4 technical report.

</span>
<span class="ltx_bibblock"><em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">arXiv:2303.08774</em>.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Pradeep et al. (2023)</span>
<span class="ltx_bibblock">
Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2309.15088" title="" class="ltx_ref ltx_href">Rankvicuna: Zero-shot
listwise document reranking with open-source large language models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2309.15088.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Qin et al. (2024)</span>
<span class="ltx_bibblock">
Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming
Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael
Bendersky. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2306.17563" title="" class="ltx_ref ltx_href">Large language models are
effective text rankers with pairwise ranking prompting</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2306.17563.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sachan et al. (2022)</span>
<span class="ltx_bibblock">
Devendra Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih,
Joelle Pineau, and Luke Zettlemoyer. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2022.emnlp-main.249" title="" class="ltx_ref ltx_href">Improving
passage retrieval with zero-shot question generation</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2022 Conference on Empirical Methods in
Natural Language Processing</em>, pages 3781–3797, Abu Dhabi, United Arab
Emirates. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sun et al. (2023)</span>
<span class="ltx_bibblock">
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen,
Dawei Yin, and Zhaochun Ren. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2023.emnlp-main.923" title="" class="ltx_ref ltx_href">Is
ChatGPT good at search? investigating large language models as re-ranking
agents</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2023 Conference on Empirical Methods in
Natural Language Processing</em>, pages 14918–14937, Singapore. Association for
Computational Linguistics.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Thakur et al. (2021)</span>
<span class="ltx_bibblock">
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and
Iryna Gurevych. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=wCu6T5xFjeJ" title="" class="ltx_ref ltx_href">BEIR: A
heterogeneous benchmark for zero-shot evaluation of information retrieval
models</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">Thirty-fifth Conference on Neural Information Processing
Systems Datasets and Benchmarks Track (Round 2)</em>.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. (2023)</span>
<span class="ltx_bibblock">
Hugo Touvron et al. 2023.

</span>
<span class="ltx_bibblock">Llama 2: Open foundation and fine-tuned chat models.

</span>
<span class="ltx_bibblock"><em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">arXiv:2307.09288</em>.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2024a)</span>
<span class="ltx_bibblock">
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang,
Rangan Majumder, and Furu Wei. 2024a.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2212.03533" title="" class="ltx_ref ltx_href">Text embeddings by
weakly-supervised contrastive pre-training</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2212.03533.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2024b)</span>
<span class="ltx_bibblock">
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu
Wei. 2024b.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2401.00368" title="" class="ltx_ref ltx_href">Improving text embeddings
with large language models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2401.00368.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2021)</span>
<span class="ltx_bibblock">
Shuai Wang, Shengyao Zhuang, and Guido Zuccon. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.1145/3471158.3472233" title="" class="ltx_ref ltx_href">Bert-based dense
retrievers require interpolation with bm25 for effective passage retrieval</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2021 ACM SIGIR International Conference
on Theory of Information Retrieval</em>, ICTIR ’21, page 317–324, New York, NY,
USA. Association for Computing Machinery.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wei et al. (2022)</span>
<span class="ltx_bibblock">
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia,
Ed Chi, Quoc V Le, and Denny Zhou. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf" title="" class="ltx_ref ltx_href">Chain-of-thought prompting elicits reasoning in large language models</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>,
volume 35, pages 24824–24837. Curran Associates, Inc.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wolf et al. (2020)</span>
<span class="ltx_bibblock">
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue,
Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe
Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien
Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest,
and Alexander Rush. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2020.emnlp-demos.6" title="" class="ltx_ref ltx_href">Transformers:
State-of-the-art natural language processing</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib44.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing: System Demonstrations</em>, pages 38–45, Online.
Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xiong et al. (2021)</span>
<span class="ltx_bibblock">
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett,
Junaid Ahmed, and Arnold Overwijk. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=zeFrfgyZln" title="" class="ltx_ref ltx_href">Approximate
nearest neighbor negative contrastive learning for dense text retrieval</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib45.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yang et al. (2024)</span>
<span class="ltx_bibblock">
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and
Xinyun Chen. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=Bb4VGOWELI" title="" class="ltx_ref ltx_href">Large language
models as optimizers</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib46.1.1" class="ltx_emph ltx_font_italic">The Twelfth International Conference on Learning
Representations</em>.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2024)</span>
<span class="ltx_bibblock">
Bowen Zhang, Kehua Chang, and Chunping Li. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2404.03921" title="" class="ltx_ref ltx_href">Simple techniques for
enhancing sentence embeddings in generative language models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib47.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2404.03921.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhuang et al. (2023a)</span>
<span class="ltx_bibblock">
Shengyao Zhuang, Bing Liu, Bevan Koopman, and Guido Zuccon. 2023a.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2023.findings-emnlp.590" title="" class="ltx_ref ltx_href">Open-source large language models are strong zero-shot query likelihood
models for document ranking</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib48.1.1" class="ltx_emph ltx_font_italic">Findings of the Association for Computational Linguistics:
EMNLP 2023</em>, pages 8807–8817, Singapore. Association for Computational
Linguistics.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhuang et al. (2023b)</span>
<span class="ltx_bibblock">
Shengyao Zhuang, Honglei Zhuang, Bevan Koopman, and Guido Zuccon.
2023b.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2310.09497" title="" class="ltx_ref ltx_href">A setwise approach for
effective and highly efficient zero-shot ranking with large language models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib49.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2310.09497.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhuang and Zuccon (2021a)</span>
<span class="ltx_bibblock">
Shengyao Zhuang and Guido Zuccon. 2021a.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2108.08513" title="" class="ltx_ref ltx_href">Fast passage re-ranking
with contextualized exact term matching and efficient passage expansion</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib50.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2108.08513.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhuang and Zuccon (2021b)</span>
<span class="ltx_bibblock">
Shengyao Zhuang and Guido Zuccon. 2021b.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.1145/3404835.3462922" title="" class="ltx_ref ltx_href">Tilde: Term
independent likelihood model for passage re-ranking</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib51.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 44th International ACM SIGIR Conference
on Research and Development in Information Retrieval</em>, SIGIR ’21, page
1483–1492, New York, NY, USA. Association for Computing Machinery.

</span>
</li>
</ul>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2404.18423" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2404.18424" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2404.18424">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2404.18424" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2404.18426" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Sun May  5 19:45:40 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2404.18424
The current use of large language models (LLMs)  for zero-shot document ranking follows one of two ways: 1)  prompt-based re-ranking methods, which require no further training but are feasible for only re-ranking a handful of candidate documents due to the associated computational costs; and 2)  unsupervised contrastive trained dense retrieval methods, which can retrieve relevant documents from the entire corpus but require a large amount of paired text data for contrastive training.
In this paper, we propose PromptReps, which combines the advantages of both categories: no need for training and the ability to retrieve from the whole corpus. Our method only requires prompts to guide an LLM to generate query and document representations for effective document retrieval. Specifically, we prompt the LLMs to represent a given text using a single word, and then use the last token’s hidden states and the corresponding logits associated to the prediction of the next token to construct a hybrid document retrieval system. The retrieval system harnesses both dense text embedding and sparse bag-of-words representations given by the LLM. Our experimental evaluation on the BEIR zero-shot document retrieval datasets illustrates that this simple prompt-based LLM retrieval method can achieve a similar or higher retrieval effectiveness than state-of-the-art LLM embedding methods that are trained with large amounts of unsupervised data, especially when using a larger LLM.111Code for fully reproducing the results is available athttps://github.com/ielab/PromptReps.
abstract
PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval
Abstract
2404.18424
1IntroductionLarge Language Models (LLMs)  such as GPT4 and LLaMA, which are pretrained on massive corpora and finetuned to follow user instructions, have strong zero-shot natural language understanding capabilities.
Via prompting, LLMs excel in various text generation tasks such as question answering, writing assistance, and conversational agent.
Inspired by the success of LLMs on natural language understanding tasks, several studies have explored the potential of using LLMs to perform unsupervised document ranking task.Figure 1:Overview of PromptReps, LLMs are prompted to generate dense and sparse text representation at the same time, which is then used to build search indices.One line of work focuses on directly prompting LLMs to infer document relevance to a given query. For instance, RankGPTcasts document re-ranking as a permutation generation task, prompting LLMs to generate re-ordered document identifiers according to the document’s relevance to the query. These methods leverage LLMs for document ranking in a complete zero-shot setting where no further training is required. However, these methods can only serve as a second-stage re-ranker on a handful of candidate documents. This is because each prompt requires one full LLM inference: for example, in the case of a corpus with 1M documents, a pointwise approach would require the construction of 1M prompts and thus the execution of 1M (costly)  LLM inferences – making it unfeasible for an online search engine.Another line of research leverages LLMs as a text embedding model for dense document retrieval. For example, E5-mistralemploys LLMs to create synthetic datasets of query-document pairs. These paired text data are then used to perform unsupervised contrastive training for a Mistral LLM-based dense retriever. Since the queries and documents are encoded with LLMs separately, i.e., using a bi-encoder architecture, these methods could serve as a first-stage document retriever. However, all existing LLM-based retrievers require an unsupervised contrastive training step to transform a generative LLM into a text-embedding model. Even with parameter-efficient training techniques such as LoRA, this extra training is still very expensive.
For example, the contrastive training of E5-mistral using a large batch size (2048)  and LoRA tookhours on 32 V100 GPUs.In this work, we propose a new zero-shot LLM-based document retrieval method called PromptReps.
We demonstrate that LLMs can be directly prompted to produce query and document embeddings, which can serve as effective text representations for neural retrieval systems.
Specifically, we prompt an LLM by asking it to use a single word to represent a query or a document.
Then, we extract the last layer’s hidden state of the last token in the prompt as the dense representation of the input text.
Simultaneously, we utilize the logits associated with predicting the subsequent token to form a sparse representation.
As illustrated in Figure1, through a single forward pass, we generate text representations for a document that can be indexed for dense, sparse, or hybrid search architectures.We evaluate our approach on the BEIR datasetsto perform zero-shot retrieval.
We utilize various open-source large language models and find that the effectiveness of zero-shot retrieval increases with the scaling of LLM size.
Results show that PromptReps outperforms previous LLM-based embedding methods. Of key importance is that our method is the first LLM-based method that can effectively perform full corpus retrieval while at the same time not requiring contrastive training, demonstrating that prompt engineering for generative LLMs is capable of generating robust representations for retrieval.
introduction
PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval
1Introduction
2404.18424
2Related Work2.1Supervised Neural RetrieversNeural retrievers based on bi-encoder architecture bring significant improvement over traditional heuristic retrievers such as BM25. Dense retrievers such as DPR, ANCE, ColBERT, encode text into low-dimensional dense vectors and conduct search with (approximate)  nearest neighbor search. On the other hand, sparse neural retrievers such as DeepImpact, uniCOIL, TILDE, and SPLADE, encode text into high-dimensional sparse vectors as bag-of-words representations and conduct search in an inverted index. Recent work has also explored fine-tuning LLMs as dense retrievers such as RepLLaMAand LLaRA. A hybrid neural retrieval system refers to a system that combines the rankings provided by both dense and sparse retrievers, often resulting in an enhanced final ranking. All these retrievers are trained with supervised relevance judgment data (e.g., MS MARCO)  using contrastive learning. Our work instead focuses on building a hybrid neural retrieval system with zero-shot dense and sparse document representations without supervised contrastive learning. This is particularly valuable for applications where no relevance judgements are available — often the case in real work retrieval settings.2.2Unsupervised Neural RetrieversThere have also been attempts on training effective neural retrievers without relying on human relevance judgments.
Methods such as Contrieverand E5, train dense retriever with large-scale pseudo query-document pairs to build unsupervised (synthetic)  training data.
LLMs have also been adapted as unsupervised text embedding models for first-stage document retrieval.
For instance, HyDEenhances query representations for an unsupervised retriever by replacing the original query with LLM-generated hypothetical documents.
More recent work has focused on directly converting generative LLMs into a text-embedding models with unsupervised contrastive pre-training. These works, such as E5-Mistral-Instand Gecko, use large-scale weakly supervised paired text data or LLM-generated query-document pair data to perform contrastive training on top of LLMs.
LLM2Vec, on the other hand, conducts further masked next token prediction pre-training with bidirectional attention, and SimCSEtrains on raw text data to transform LLMs into text encoders. Although no labeled data is used, these methods require synthetic or unsupervised paired text data to perform contrastive pre-training.
Our method instead relies solely on prompt engineering to transform LLM into a robust text encoder for document retrieval without any extra training.2.3Prompting LLMs for document re-rankingInspired by the prompt-following capacity of LLMs, recent works have explored prompting LLMs for document reranking. For instance, UPRranks documents pointwise by prompting the LLM to generate a relevant query for a given document and rank documents based on the likelihood of generating the query. RankGPTand LRLpropose to re-rank a list of documents at once and generate permutations for the reordered list. Pairwiseand Setwiseprompting methods have also been explored to improve effectiveness and efficiency in the LLM re-ranking pipeline. These methods are only feasible for re-ranking a handful of candidate documents, thus limited to second-stage document re-ranking. In contrast, our approach utilizes prompts to construct the first-stage retrievers.2.4Prompting LLM for sentence embeddingsThe methods most similar to ours are work on prompting LLMs to generate sentence embeddings for semantic textual similarity (STS)  tasks. These works developed an Explicit One-word Limitation (EOL)  prompt, which also instructs LLMs to represent a sentence with one word. However, these works only evaluate such prompts on STS datasets, and their effectiveness on information retrieval datasets with large document corpora is unknown. Additionally, these methods only represent texts with dense embeddings from the hidden states; our method instead generates sparse text representations simultaneously to build a hybrid retrieval system. In fact, our results show that dense embeddings alone perform poorly for document retrieval tasks with some LLMs, but sparse representations are much more robust, and the best retrieval effectiveness is achieved with the hybrid retrieval system with scaled model size.
body
PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval
2Related Work
2404.18424
3PromptRepsUnlike previous works that leverage LLMs for document ranking, which are limited to document re-ranking tasks with prompts or rely on contrastive learning to transform a generative LLM into an embedding model for document retrieval, in this work, we aim to directly prompt LLMs to generate both dense embedding representations and sparse bag-of-words representations for document retrieval without any form of extra training effort. To achieve this aim, we devise the following prompt as the input text for LLMs:<System>You are an AI assistant that can understand human language.<User>Passage "[text]". Use one most important word to represent the passage in retrieval task. Make sure your word is in lowercase.<Assistant>The word is: "where <System> <User> and <Assistant> are LLM pre-defined conversational prefix tokens and [text] is the placeholder for passage text.
When using this prompt for text generation, the language model needs to find a single word in its token vocabulary that can best represent the given passage to generate. However, since there could be multiple words to represent the passage, there might be multiple tokens in the vocabulary that have a high probability of being sampled by the language model. Such a distribution over the vocabulary, which often refers to
"logits", could potentially provide a good representation of the given passage. In addition, since the logits are computed by the last layer hidden state222Often through dot product between last hidden state with all token embeddings.of the last input token (‘"’) , which is a dense vector embedding, it can also serve as a dense representation of the passage.Based on the above intuition, we develop a sparse + dense hybrid document retrieval system by utilizing both the next token logits and the last layer hidden states outputted by the LLM with our designed prompt.Specifically, during the document indexing phase, we pass all the documents with our prompt into the LLM to get output hidden states and logits. To build a sparse retrieval pipeline with logits, we first need to sparsify the logits representation to be able to perform efficient sparse retrieval. This is because logits originally had values for all tokens in the vocabulary, essentially forming dense vectors with dimensions equal to the vocabulary size. To sparsify the logit representations for sparse retrieval, we perform following steps:1.Lowercase the input document text to align with the phrase "Make sure your word is in lowercase." in the prompt since this phrase skewed the sampling distribution towards lowercase tokens (a "sparser" distribution) . We then utilize the NLTK toolkitto extract all words in the document, filtering out standard English stopwords and punctuation.2.Next, we use the LLM’s tokenizer to tokenize each extracted word and obtain their token IDs333Note that many words may be split into sub-tokens, resulting in multiple token IDs, all of which are considered in the logits. We retain only the values corresponding to the obtained token IDs in the logits and set the rest of the dimensions to zero, thereby considering only tokens present in the documents, thus enabling exact term matching in retrieval.3.Next, we follow the SPLADE recipe, using the ReLU function to remove dimensions with negative values and applying log-saturation to the logits to prevent certain tokens from dominating. To further enhance the sparsity of logits, we only keep tokens within the top 128 values if the logits had more than 128 non-zero values after the previous steps.4.Finally, the logits were quantized by simply multiplying the original values by 100 and taking the integer operation on that, and the obtained values represent the weights of corresponding tokens.With these adjustments, the logits representations of documents are heavily sparsified, allowing for efficient sparse retrieval with a standard inverted index.For dense retrieval, we directly use the hidden states as the embedding of the documents. For indexing these embeddings, we simply normalize all the embeddings and add them into an Approximate Nearest search (ANN)  vector index.At query time, we process the queries exactly the same as the documents, with the only exception being that the term "passage" in the prompt is replaced with "query". The dense representation of the query is utilized for semantic search via the ANN index, while the sparse representation of the query is employed for exact term matching via the inverted index. Following previous work, we compute the final document scores by applying min-max normalization to both dense and sparse document scores. These normalized scores are then linearly interpolated with equal weights to produce the final document scores.Table 1:nDCG@10 scores of BEIR 13 publicly available datasets. The best scores of methods without interpolating with BM25 are highlighted in bold. Scores of E5-PTlargeand LLM2Vec are taken directly from the original papers.Contrastive pre-trainingPromptReps(ours) LLM-BERT-330MMistral-7BMistral-7B-InstructPhi3-mini-3.8B-InstructLlama3-8B-InstructLlama3-70B-InstructDatasetBM25E5-PTlargeLLM2VecDenseSparseHybridDenseSparseHybridDenseSparseHybridDenseSparseHybrid+BM25arguana39.7044.451.0021.0232.4132.8138.0021.5135.6832.8924.2535.4737.3826.1538.2942.28climate-fever16.5115.722.975.3111.1710.5718.9911.3921.1223.479.5222.2021.2112.0122.5622.96dbpedia31.8037.125.488.2322.8017.8726.0924.7232.2130.9528.1136.3031.8828.6936.5940.48fever65.1368.645.1120.0350.0747.2344.1851.7364.9259.4351.6071.6157.6254.7771.5276.37fiqa23.6143.227.248.3118.6315.7521.0820.4027.0124.7919.6629.5429.9922.2233.3433.20hotpotqa63.3052.254.542.9832.1520.6225.1542.7546.6228.0745.8251.2730.9444.9150.9866.42nfcorpus32.1833.727.1614.4127.8123.5527.8029.3833.2526.7427.4831.3631.9329.3534.0336.19nq30.5541.734.167.1219.9316.6226.4725.5734.7835.0228.3441.6243.3830.7645.9844.22quora78.8686.184.4053.3861.9566.6580.7167.7982.1177.4769.0282.4775.2771.5781.8184.84scidocs14.9021.815.356.5411.4210.2813.3512.2415.5816.7711.5216.9318.3113.2217.9617.73scifact67.8972.368.6722.7155.2748.1551.6959.9464.8654.2659.0665.6158.9261.5367.4172.09trec-covid59.4761.855.6634.9462.0353.7951.1656.5362.4863.5455.1968.5471.4763.1977.1377.94touche44.2219.86.549.5919.4718.3612.7519.1719.5814.9216.4619.2418.8420.1222.8536.21avg43.7044.6139.8616.5132.7029.4033.6534.0941.5537.4834.3144.0140.5536.8046.1950.07
body
PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval
3PromptReps
2404.18424
4Experimental setupDataset and evaluation:We evaluate the document ranking effectiveness of both baseline methods and our proposed PromptReps using the BEIR benchmark. This benchmark encompasses various IR tasks, providing a heterogeneous evaluation environment. We report nDCG@10 scores, which is the official evaluation metric employed by BEIR.Baselines:We compare PromptReps with strong unsupervised first-stage retrievers including BM25, a classic term frequency-based sparse retrieval method, and E5-PTlarge, a state-of-the-art BERT-baseddense embedding method trained on 1.3 billion carefully crafted unsupervised text pairs. LLM2Vec, an LLM-based dense embedding method trained with bi-directional attention, masked next token prediction, and SimCSEon the Wikipedia corpus.Implementation of PromptReps:PromptReps is implemented using three base LLMs: Mistral-7b-Instruct-v0.2444https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2, Llama3-8B-Instruct555https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct, Phi-3-mini-4k-instruct666https://huggingface.co/microsoft/Phi-3-mini-4k-instruct, and Llama3-70B-Instruct777https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct. Dense and sparse document and query encoding are implemented using the Huggingface Transformers libraryand the Tevatron toolkit. Faiss libraryis used to build the ANN index with cosine similarity as the embedding distance metric, and Pyseriniis utilized to construct the inverted index for sparse retrieval. For the dense and sparse ranking hybrid, the Ranx libraryis employed. In our experiments, we report dense only, sparse only, and the full hybrid results. In addition, we also report the results of further interpolating BM25 scores with hybrid scores for PromptReps based on Llama3-70B-Instruct.
body
PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval
4Experimental setup
2404.18424
5ResultsWe present our results in Table1.
The first observation highlights that BM25 is a very strong zero-shot retrieval method, capable of outperforming LLM2Vec, based on the Mistral-7B LLM, across numerous datasets, achieving a higher average nDCG@10 score. This outcome implies that even with a large-size LLM, bi-directional attention enabled, additional pre-training, and SimCSE-based unsupervised contrastive training, there remains a gap in transforming a decoder-only LLM into a robust retrieval method.On the other hand, E5-PTlarge, based on the BERT-large model, is the first method that can outperform BM25 without any supervised training data. However, it has been trained on a massive, carefully mined text pair dataset with a large batch size, which may require more data-collecting efforts and computational resources than LLM2Vec.For our proposed PromptReps, when using Mistral-7B-Instruct LLM, the dense-only retriever performs poorly. However, although still worse than LLM2Vec, the sparse-only retriever performs much better than dense. This finding suggests that, with Mistral-7B-Instruct as the base LLM, using the logits generated by our prompt produces better document representations than hidden states for document retrieval. The hybrid retrieval performs worse than sparse-only in this case due to the poor performance of dense retrieval.Surprisingly, when implementing PromptReps with Phi-3-mini-4k-instruct achieved much higher dense-only retrieval effectiveness, closing to its sparse retrieval effectiveness and leading to an impressive hybrid retrieval average score that surpassed LLM2Vec based on Mistral-7B, even though it is only half the size.Interestingly, when implementing PromptReps with the state-of-the-art Llama3-8B-Instruct LLM, the dense-only retrieval further improved and achieved a higher score than that of sparse. The sparse-only retrieval effectiveness also improved, but the improvement was much more marginal compared to the improvement on dense. The hybrid of dense and sparse brings notable retrieval effectiveness improvement, surpassing BM25 and approaching the state-of-the-art E5-PTlarge. Notably, this is achieved without any form of extra training but solely relying on prompts.The scaling law of LLMis also applied here. When changing Llama3-8B-Instruction to Llama3-70B-Instruction, the retrieval effectiveness of our PromptReps further improved, with the dense-only approach surpassing LLM2Vec and the hybrid approach surpassing E5-PTlarge. When further interpolating the hybrid ranking with BM25 ranking, the average nDCG@10 achieved a remarkable score of. Our results demonstrate that it is possible to build a very strong hybrid retrieval system without the need for any unsupervised or supervised training.A note on query latency.The best performance of our PromptReps is achieved in the hybrid retrieval setting. Compared to previous works which are dense retrieval only, our method seems to have a higher query latency. However, PromptReps actually only requires a little query latency overhead if implementing dense and sparse search in parallel. The cost of obtaining both dense and sparse representations of our method only requires a single LLM forward inference; the only extra computation is the dot product of the dense vector with the token embeddings, which is very fast on GPU. For document search, since we heavily sparsified the sparse representation, in our experiments, our sparse retrieval is much faster than BM25, and the bottleneck is the dense retrieval. Since the dense and sparse search could be run in parallel and the hybrid operation is just a linear interpolation of both rankings (very fast on CPU) , the query latency will depend on the dense retrieval latency, thus very close to the previous methods.
body
PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval
5Results
2404.18424
6DiscussionOur work demonstrates that prompt engineering for generative LLMs is not only a powerful method for natural language generation tasks but also capable of generating robust dense (embedding)  and sparse (bag of words)  representations for input texts. We believe this important finding could lead to a series of new research directions.For example, techniques like few-shot in-context learning, chain-of-thought prompting, and auto-prompt optimization methods, which have proven to be very effective in text-generation tasks, could potentially be leveraged here to enhance embedding generation.Moreover, it has been shown that the instruction-following ability of LLMs could be transferred to embedding models with synthetic instruction fine-tuning data. In our work, we always keep the instruction prompt consistent across different information retrieval tasks, which could be sub-optimal. It would be very interesting to investigate how to customize instructions for PromptReps to generate embeddings specific to different domains, tasks, or even to multi-lingual and cross-lingual retrieval settings.Finally, our prompting method could be seen as a simple approach to obtaining a better initialization of LLM-based embedding models, which is much more cost-effective than methods requiring further pre-training. All the previous contrastive pre-training with paired text data and synthetically generated data could be applied on top of our method and could potentially yield improved LLM-based embedding models. We leave these aspects for future work.
body
PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval
6Discussion
2404.18424
7ConclusionIn this paper, we introduced PromptReps, a simple yet effective method that prompts LLMs to generate dense and sparse representations for zero-shot document retrieval without any further unsupervised or supervised training. Our work reveals that modern LLMs are effective text encoders by themselves, and prompt engineering is sufficient to stimulate their text encoding ability.
conclusion
PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval
7Conclusion