LLM-Augmented Retrieval: Enhancing Retrieval Models Through Language Models and Doc-Level Embedding
information retrieval;text retrieval;artificial intelligence;large language models;data augmentation
Yes
No ethics review needed.
- How was the chunk size determined?
- I suggest a retouch on line 348-350 “augmentation of document embeddings with LLMs can substantially elevate the recall performance of retrieval models without necessitating additional fine-tuning”. I don’t think one model is simply fine-tuned relative to another, but it’s their complexity that differs.
- Could you cite some prior work on chunk-level embeddings (line 246-248)? Using a single document-level embedding, such as truncating, seems to be a common practice. In such cases, the proposed method will increase the document index size by |doc_len| / |chunk_size|.
- I like the idea of enhancing document embeddings prior to the inference step. Specifically, this can be pre-computed, offering practical advantages over query expansion methods which have been studied extensively.
- Proposed method can be applied to both bi-encoder and token-level late-interaction models, and it achieves consistently strong results across most datasets.
This paper introduces a novel, model-agnostic approach for document embedding, leveraging an augmented document field comprising synthetically generated queries, titles, and document chunks. With a new similarity function, the method computes a similarity score between the query and document field as a proxy for query-document relevance. The method outperforms using naive document embeddings from three existing retrievers (Contriever, DRAGON, ColBERTv2) on two benchmarks (BeIR, LoTTE).
- There lacks detailed analysis of how (synthetically generated) document fields help the retrieval.  For example, how many queries were generated per document on average? The limitations section mentions that the augmented text may be as long as the original text, which suggests a substantial number of queries—clarification here would be valuable. Also, could you include some qualitative examples? 
- In the ablation study, weights for each field are exclusively set to 1.0. However, the “left term” at equation (1) remains active, meaning the query-chunks still affect the results. This raises concerns about whether the ablation study truly isolates the impact of each field, especially for the query and title.
- The results are evaluated only on the subset of BeIR benchmark without any explanations.
Yes
No ethics review needed.
1. Clarified on implementation details: according to Figure1, for each passage coming from the same document, it seems you use the same synthetic queries and titles? However, in the experiments, you mention you use the original chunk from the datasets; then, how do you know which chunks are coming from the same document and use the document to generate synthetic queries and titles for those chunks.
2. Which dev set are used to tune the hyperparameters, $w_{query}, w_{title}, w_{chunk}$?
3. From the prompts, it seems that the models are instructed to output multiple relevant queries; however, only one generated query is used for document expansion. How do you choose the query among all the generated ones? Or if you use multiple generated queries, how do you combine them?
1. The proposed method is straightforward and easy to implement and shows significant improvements for existing state-of-the-art retrievers.
2. The proposed approach is useful, which combines the document embedding from different sources and forms a single document embedding. Thus, the index size is still the same for single-vector dense retrieval.
The paper presents an approach to document expansion using LLMs to enhance the zero-shot retrieval effectiveness of existing state-of-the-art bi-encoder retrievers (Contriever, DRAGON and ColBERTv2).
1. Although the proposed approach is simple and effective, I think the major concern for me is the limited discussion in ablations and comparisons, which are important for the readers who want to adopt this approach. I'm willing to raise score if the authors can address the concern below: (1) As a reader, one may want to know the quality and cost tradeoff of the generated titles and queries. For instance, the paper prompts Llama-70B to make it work; how about using smaller size of Llama; 7B, 13B etc.? and the additional cost of document indexing with LLM document expansion should be reported. (2) There are no ablations on the three combinations: chunk + query, chunk + title and query + title, which helps readers to choose the combinations with a balanced tradeoff between effectiveness and cost  (3) A comparison with LLM-based retrievers, such as RepLlama[1] or MistralE5[2], in terms of retrieval effectiveness, query latency and indexing time, can provide more useful information for readers to choose which approach to use. (4) Can we apply this approach to multilingual retrieval tasks?
2. Although the proposed method show improvements over existing SoTA retrievers, the proposed approach is a bit incremental. I think the approach to augment the document is very similar to the previous work[3][4]. The authors should make a comparison with the previous work in Related Work to make the contribution more clear.
3. Some experimental details are missing (see Questions).

[1] Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. 2023. Fine-tuning llama for multi-stage text retrieval.
[2] Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023. Improving text embeddings with large language models.
[3] Rodrigo Nogueira and Jimmy Lin. 2019. From doc2query to docTTTTTquery.
[4] Yongqi Li, Nan Yang, Liang Wang, Furu Wei, and Wenjie Li. 2023. Multiview identifiers enhanced generative retrieval.
Yes
No ethics review needed.
All clear to me.
The paper is clear and the results are good.
This paper proposes a novel retrieval framework called to improve the performance of current retrieval models. The framework achieves this by enriching document embeddings with information from large language models (LLMs), such as synthetic queries and titles. The proposed document-level embedding approach integrates this augmented information with existing document chunks, and it's shown to be effective for both bi-encoder and late-interaction retrieval models. The authors demonstrate the framework's effectiveness by achieving state-of-the-art results on benchmark datasets LoTTE and BEIR.
It's a rather straightforward model. The novelty is simply calling some LLM to generate titles and queries for each doc, which the retriever will later use to retrieve the document. I think ICLR requires more model novelty.
Yes
No ethics review needed.
1. I am curious whether this method could be effective on larger document benchmarks, such as MSMARCO and FEVER within the BEIR benchmark. Could the authors provide results on these datasets?

2. Could the authors provide an analysis of the costs associated with using an LLM to enhance these tasks, such as the number of tokens or text checks required for generation?

3. It would be interesting to see the performance when using the original query to search against the pseudo queries, to quantify any potential leakage from the LLM (i.e., the ability to recover the query based on the document).
1. The motivation is solid, with a straightforward idea: using pseudo-queries to augment document representation, an important direction given the short context windows of current embedding models that limits their effectiveness.

2. The observed improvement appears substantial on the presented datasets.

3. The authors provide some valuable insights that could be helpful, such as the impact of titles and approaches to handle cases where titles are missing, which aligns well with real-world scenarios.
This paper proposes an LLM-augmented retrieval framework that uses a large language model (LLM), specifically LLaMA-70B, to generate pseudo-queries for document retrieval. For each document, the framework includes synthetic queries generated by the LLM, existing or LLM-generated titles, and document chunks. These text fields are embedded and combined with the query embedding using inner product, followed by weighted sum and max pooling. Evaluations are conducted with Contriever, Dragon, and ColBERT on the LoTTE dataset and a subset of the BEIR benchmark, with considerable improvements.
1. The cost of this solution appears too high, given that it relies on LLaMA-70B, and all documents require query generation. Additionally, I have concerns regarding potential data leakage in the LLM, as it may have previously encountered these query-document pairs, potentially generating similar queries based on the document. This issue could be addressed if the authors provided examples and analysis.

2. The inference process requires weighting strategies to combine the relevance between the query and different fields of the document. I have concerns about the scalability of this solution, as the presented dataset contains a relatively small number of documents (see Q1).

3. Some presentation issues: (i) The inconsistent capitalization of "Bi-encoder" is concerning, with both "Bi-encoder" and "bi-encoder" used interchangeably throughout the paper, which appears unprofessional. Additionally, on line 19, to my knowledge, ColBERTv2 falls under the category of bi-encoders but employs a more complex relevance scoring process. (ii) Certain findings in sections 3.1.2, 3.1.3, and others could be more effectively presented in an analysis section following the experiments section, providing a clearer flow and focused discussion on results.
A novel model-agnostic LLM-augmented framework to enhance retriever models' performance to state-of-the-art results
@inproceedings{
anonymous2024llmaugmented,
title={{LLM}-Augmented Retrieval: Enhancing Retrieval Models Through Language Models and Doc-Level Embedding},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=7yncrX80CN},
note={under review}
}
Recent advancements in embedding-based retrieval, also known as dense retrieval, have shown state of the art results and demonstrated superior performance over traditional sparse or bag-of-words-based methodologies. This paper presents a model-agnostic document-level embedding framework enhanced by large language model (LLM) augmentation. The implementation of this LLM-augmented retrieval framework has significantly enhanced the efficacy of prevalent retriever models, including Bi-encoders (Contriever, DRAGON) and late-interaction models (ColBERTv2). Consequently, this approach has achieved state-of-the-art results on benchmark datasets such as LoTTE and BEIR, underscoring its potential to refine information retrieval processes.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
information retrieval text retrieval artificial intelligence large language models data augmentation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/b533ba9e4233eeea7d90b59f1440468121a01a4d.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
LLM-Augmented Retrieval: Enhancing Retrieval Models Through Language Models and Doc-Level Embedding
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission