Searching for Best Practices in Retrieval-Augmented Generation:References

[1] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS 2022), 2022.

[2] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023.

[3] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. SLIC- HF: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425, 2023.

[4] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. RRHF: Rank response to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302, 2023.

[5] Wenhao Liu, Xiaohua Wang, Muling Wu, Tianlong Li, Changze Lv, Zixuan Ling, Jianhao Zhu, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. Aligning large language models with human preferences through representation engineering. arXiv preprint arXiv:2312.15997, 2023.

[6] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2023.

[7] Huayang Li, Xiyuan Su, Deng Cai, Yan Wang, and Lemao Liu. A survey on retrieval-augmented text generation. arXiv preprint arXiv:2202.01110, 2022.

[8] Deng Cai, Yan Wang, Lemao Liu, and Shuming Shi. Recent advances in retrieval-augmented text generation. In Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval, pages 3417–3419, 2022.

[9] Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. Query rewriting for retrieval-augmented large language models. arXiv preprint arXiv:2305.14283, 2023.

[10] Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. Precise zero-shot dense retrieval without relevance labels. arXiv preprint arXiv:2212.10496, 2022.

[11] Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533, 2022.

[12] Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.


Searching for Best Practices in Retrieval-Augmented Generation:References Section

[13] OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/ARXIV.2303.08774. URL https://doi.org/10.48550/arXiv.2303.08774.

[14] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.

[15] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren’s song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219, 2023.

[16] Xiaohua Wang, Yuliang Yan, Longtao Huang, Xiaoqing Zheng, and Xuan-Jing Huang. Hallucination detection for generative large language models by bayesian sequential estimation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15361–15371, 2023.

[17] Liang Wang, Nan Yang, and Furu Wei. Query2doc: Query expansion with large language models. arXiv preprint arXiv:2303.07678, 2023.

[18] Gangwoo Kim, Sungdong Kim, Byeongguk Jeon, Joonsuk Park, and Jaewoo Kang. Tree of clarifications: Answering ambiguous questions with retrieval-augmented large language models. arXiv preprint arXiv:2310.14696, 2023.

[19] Jerry Liu. Llamlindex, 11 2022. URL https://github.com/jerryjliu/llama_index.

[20] Peitian Zhang, Shitao Xiao, Zheng Liu, Zhicheng Dou, and Jian-Yun Nie. Retrieve anything to anything: Large language models. arXiv preprint arXiv:2310.07554, 2023.

[21] Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281, 2023.

[22] Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Elmingua: Compressing prompts for accelerated inference of large language models. arXiv preprint arXiv:2310.05736, 2023.

[23] Fangyuan Xu, Weijia Shi, and Eunsol Choi. Recomp: Improving retrieval-augmented lms with compression and selective augmentation. arXiv preprint arXiv:2310.04408, 2023.

[24] Zhiruo Wang, Jun Araki, Zhengbao Jiang, Md Rizwan Parvez, and Graham Neubig. Learning to elicit context for retrieval-augmented generation. arXiv preprint arXiv:2311.08377, 2023.

[25] Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. Multi-stage document ranking with bert. arXiv preprint arXiv:1910.14424, 2019.

[26] Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. Document ranking with a pretrained sequence-to-sequence model. arXiv preprint arXiv:2003.06713, 2020.

[27] Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. Fine-tuning llama for multi-stage text retrieval. arXiv preprint arXiv:2310.08319, 2023.

[28] Shengyao Zhuang and Guido Zuccon. Tilde: Term independent likelihood model for passage re-ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1483–1492, 2021.

[29] Shengyao Zhuang and Guido Zuccon. Fast passage re-ranking with contextualized exact term matching and efficient passage expansion. arXiv preprint arXiv:2108.08513, 2021.

[30] Hongyin Luo, Yung-Sung Chuang, Yuan Gong, Tianhua Zhang, Yoon Kim, Xixin Wu, Danny Fox, Helen M. Meng, and James R. Glass. Sail: Search-augmented instruction learning. In Conference on Empirical Methods in Natural Language Processing, 2023. URL https://api.semanticscholar.org/CorpusID:258865283.


[31] Tianjun Zhang, Shishir G. Patil, Naman Jain, Sheng Shen, Matei A. Zaharia, Ion Stoica, and Joseph E. Gonzalez. Rafr: Adapting language model to domain specific rag. arXiv, abs/2403.11031, 2024.

[32] Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Chankyu Lee, Mohammad Shoeybi, and Bryan Catanzaro. Chatqa: Surpassing gpt-4 on conversational qa and rag. 2024. URL https://api.semanticscholar.org/CorpusID:267053153.

[33] Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane A. Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with retrieval augmented language models. arXiv, abs/2208.03299, 2022.

[34] Lingxi Zhang, Yue Yu, Kuan Wang, and Chao Zhang. Arl2: Aligning retrievers for black-box large language models via self-guided adaptive relevance labeling. arXiv, abs/2402.13542, 2024.

[35] Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. Replug: Retrieval-augmented black-box language models. arXiv preprint arXiv:2301.12652, 2023.

[36] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv, abs/2002.08909, 2020.

[37] Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Rich James, Pedro Rodriguez, Jacob Kahn, Gergely Szinay, Mike Lewis, Luke Zettlemoyer, and Scott Yih. Ra-dlt: Retrieval-augmented dual instruction tuning. arXiv, abs/2310.01352, 2023.

[38] Hamed Zamani and Michael Bendersky. Stochastic rag: End-to-end retrieval-augmented generation through expected utility maximization. 2024. URL https://api.semanticscholar.org/CorpusID:269065438.

[39] Yizheng Huang and Jimmy Huang. A survey on retrieval-augmented text generation for large language models. arXiv preprint arXiv:2404.10981, 2024.

[40] Ruocheng Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, et al. Retrieving multimodal information for augmented generation: A survey. arXiv preprint arXiv:2303.10866, 2023.

[41] Penghao Zhao, Hailin Zhang, Qinhan Yu, Chengwen Wang, Yunteg Geng, Fangcheng Fu, Ling Yang, Wentao Zhang, and Bin Cui. Retrieval-augmented generation for ai-generated content: A survey. arXiv preprint arXiv:2402.19473, 2024.

[42] Michael Günther, Jackmin Ong, Isabelle Mohr, Aladdine Abdessalem, Tanguy Abel, Mohamad Kalim Akram, Susana Guzman, Georgios Mastrapas, Saba Sturua, Bo Wang, et al. Jina embeddings 2: 8192-token general-purpose text embeddings for long documents. arXiv preprint arXiv:2310.19923, 2023.

[43] LlamaIndex. Llamaindex website. https://www.llamaindex.com. Accessed: 2024-06-08.

[44] Kunal Sawarkar, Abhilasha Mangal, and Shivam Raj Solanki. Blended rag: Improving rag (retriever-augmented generation) accuracy with semantic search and hybrid query-based retrieval. arXiv preprint arXiv:2404.07220, 2024.

[45] Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118, 2021.

[46] Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint arXiv:2104.08663, 2021.

[47] Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268, 2016.


Searching for Best Practices in Retrieval-Augmented Generation:Extracted Research Paper Information

[48] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173, 2024.

[49] Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Longlmllmig: Accelerating and enhancing llms in long context scenarios via prompt compression. arXiv preprint arXiv:2301.06839, 2023.

[50] Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Amal R. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Elsobky, Jude Fernandes, Jeremy Fu, Wenyi Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kekez, Madian Khabsa, Isabel M. Kloumann, A. V. Korneev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yiren Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alon Schotten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jamin S. Zhang, Kuan, Puxin Xu, Zhengyan Yin, Iryna Zavorotynska, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv, abs/2307.09288, 2023.

[51] ES Shaul, Jithin James, Luis Espinosa Anke, and Steven Schockaert. Ragas: Automated evaluation of retrieval augmented generation. In Conference of the European Chapter of the Association for Computational Linguistics, 2023. URL https://api.semanticscholar.org/CorpusID:263125733.

[52] Harsh Trivedi, Niranjan Balasubramanian, Thurak Khot, and Ashish Sabharwal. Two-step multi-hop inference with single-hop question composition. Transactions of the Association for Computational Linguistics, pages 559–574, 2023. doi: http://dx.doi.org/10.1162/tacl_a_00475.

[53] Mike Conover, Matt Hayes, Ankith Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023. URL https://www.databricks.com/blog/2023/04/12/free-dolly-first-open-commercially-viable-instruction-tuned-llm.

[54] Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Fernando Campos, and Ellen M. Voorhees. Overview of the trec 2019 deep learning track. arXiv, abs/2003.07820, 2020. URL https://api.semanticscholar.org/CorpusID:253243683.

[55] Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Fernando Campos, and Ellen M. Voorhees. Overview of the trec 2020 deep learning track. arXiv, abs/2102.07662, 2021. URL https://api.semanticscholar.org/CorpusID:212737158.

[56] Jimmy Lin, Xueguang Ma, Sheng-Cheih Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2356–2362, 2021.

[57] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Daniel Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc V. Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.

[58] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Trivaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv, abs/1705.03551, 2017.

[59] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600, 2018.


[60] Ivan Stelmakh, Yi Luan, Bhuvan Dhingra, and Ming-Wei Chang. Asqa: Factoid questions meet long-form answers. ArXiv, abs/2204.06092, 2022.

[61] Tomás Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317–328, 2018.

[62] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.

[63] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021.

[64] J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. ArXiv, abs/2106.09685, 2021.

[65] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Cornell University - arXiv, Sep 2020.

[66] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457, 2018. URL https://api.semanticscholar.org/CorpusID:3928216.

[67] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Jan 2018. doi: 10.18653/v1/d18-1260. URL http://dx.doi.org/10.18653/v1/d18-1260.

[68] James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. Fever: a large-scale dataset for fact extraction and verification. ArXiv, abs/1803.05355, 2018. URL https://api.semanticscholar.org/CorpusID:4711425.

[69] Tianhua Zhang, Hongyin Luo, Yung-Sung Chuang, Wei Fang, Luc Gaitskell, Thomas Hartvigsen, Xixin Wu, Danny Fox, Helen M. Meng, and James R. Glass. Interpretable unified language checking. ArXiv, abs/2304.03728, 2023. URL https://api.semanticscholar.org/CorpusID:258041307.

[70] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. Empirical Methods in Natural Language Processing, Empirical Methods in Natural Language Processing, Oct 2013.

[71] Xanh Ho, A. Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. ArXiv, abs/2011.01060, 2020. URL https://api.semanticscholar.org/CorpusID:226236740.

[72] Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, NoahA. Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. Oct 2022.

[73] Qiao Jin, Bhuvan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Conference on Empirical Methods in Natural Language Processing, 2019. URL https://api.semanticscholar.org/CorpusID:220572622.

[74] Akari Asai, Zeqiu Wu, Yizhong Wang, Aviurp Sil, and Hananneh Hajishirzi. Self-rap: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2301.11511, 2023.


Searching for Best Practices in Retrieval-Augmented Generation:Experimental Details

In this section, we provide detailed experimental settings for each module, covering dataset specifics, training parameters, and any additional experimental results.

A.1 Query Classification

Datasets: We utilized a subset of the Databricks-Dolly-15K and generated additional data using GPT 4. The prompt template for generating questions is shown in Table 14.

Implementation Details: We choose BERT-base-multilingual-cased as our classifier, with a batch size of 16 and a learning rate of 1e-5. The evaluation of results is showcased in Table 1.

A.2 Experimental Details of Retrieval Methods

Implementation details of the comparative experiments of different retrieval methods are as below:

Datasets: We use the TREC DL 2019 and 2020 passage ranking datasets to evaluate the performance of different retrieval methods.

Metrics: Widely-used evaluation metrics for retrieval include mAP, nDCG@10, R@50 and R@1k. Both mAP and nDCG@10 are order-aware metrics that take the ranking of search results into account. In contrast, R@k is an order-unaware metric. We also report the average latency incurred by each method per query.

Implementation Details: For sparse retrieval, we use the BM25 algorithm, which relies on the TF-IDF algorithm. For dense retrieval, we employ Contriever as our unsupervised contrastive text encoder. Based on our evaluation of embedding models, we implement our supervised dense retrieval using LLM-Embedder. We use the default implementation of BM25 and Contriever from Pyserini. The BM25 index is constructed using Lucene on MS MARCO collections, while the dense vector index is generated with Faiss employing Flat configuration on the same dataset. For query rewriting, we prompt Zephyr-7b-alpha, a model trained to act as a helpful assistant, to rewrite the original query. For query decomposition, we employ GPT 3.5-turbo-0125 to break down the original query into multiple sub-queries. We closely follow the implementation from HyDE, utilizing the more advanced instruction-following language model, GPT 3.5-turbo-instruct, to generate hypothetical answers. The model infers with a default temperature of 0.7, sampling up to a maximum of 512 tokens. Retrieval experiments and evaluation are conducted using the Pyserini toolkit.

A.3 Experimental Details of Reranking Methods

Datasets: Our experiments utilize the MS MARCO Passage ranking dataset, a substantial corpus designed for machine reading comprehension tasks. This dataset comprises over 8.8 million passages and 1 million queries. The training set contains approximately 39M tuples of queries paired with corresponding positive and negative passages, while the development set comprises 6,980 queries, paired with their BM25 retrieval results, and preserves the top-1000 ranked candidate passages for each query. We evaluate the effectiveness of the methods on the development set, as the test set is not publicly available.

Metrics: The evaluation metrics MRR@1, MRR@10, MRR@1k and Hit Rate@10 are used. MRR@10 is the official metric proposed by MS MARCO.

Implementation Details: We follow and make modifications to the implementation provided by PyGaggle and TILDEv2. For DLM-based reranking, we use monoT5 based on T5-base, monoBERT based on BERT-large and RankLLAMA based on Llama 2-7b. For TILDE reranking, we use TILDEv2 based on BERT-base.

Typically, 50 documents are retrieved as input for the reranking module. The documents remaining after the reranking and repacking phase can be further concentrated by assigning a top-k value or a relevancy score threshold.

Results: Ranking results are shown in Table 9. We compare our results with a randomly shuffled ordering and the BM25 retrieval baseline. All reranking methods demonstrate a notable improvement.


Searching for Best Practices in Retrieval-Augmented Generation:Experimental Details of Summarization Methods

Selective Context enhances LLM efficiency by identifying and removing redundant information in the input context. It evaluates the informativeness of lexical units using self-information computed by a base causal language model. This method is non-query-based, allowing a comparison between query-based and non-query-based approaches. Datasets: We evaluated these methods on three datasets: Natural Questions (NQ) [57], TriviaQA [58], and HotpotQA [59]. Metrics: Evaluation metrics include the F1 score and the number of tokens changed after summarization to measure conciseness. Implementation Details: For all methods, we use Llama3-8B-Instruct as the generator model and set a summarization ratio of 0.4. For extractive methods, importance scores determine the sentences retained. For abstractive methods, we control the maximum generation length using the summarization ratio to align with extractive methods. Experiments are conducted on the NQ test set, TriviaQA test set, and HotpotQA development set.


Searching for Best Practices in Retrieval-Augmented Generation:A.6 Experimental Details of Comprehensive Evaluation

Tasks and Datasets We conducted extensive experiments across various NLP tasks and datasets to assess the performance of RAG systems. Specifically: (1) Commonsense Reasoning: We evaluated on MMLU [65], ARC-Challenge [66] and OpenbookQA [67] datasets. (2) Fact-Checking: Our evaluation encompassed the FEVER [68] and PubHealth [69] datasets. (3) Open-Domain QA: We assessed on NQ [57], TriviaQA [58], and WebQuestions [70] datasets. (4) Multihop QA: Our evaluation included the HotPotQA [59], 2WikiMultiHopQA [71], and MuSiQue [52] datasets. For MuSiQue, we followed the approach outlined in [72] and focused solely on answerable 2-hop questions. (5) Medical QA: We also assessed on the PubMedQA [73] dataset. In each dataset, we randomly sub-sample 500 entries from the test set for our experiments. For datasets without test set, we use develop set instead.

To assess RAG capabilities, we evenly collect a total of 500 entries from NQ, TriviaQA, HotPotQA, 2WikiMultiHopQA and MuSiQue. Each entry is a “question, gold document, gold answer” triple.

Metrics We use token-level F1 score and EM score for Open-Domain QA and Multihop QA tasks, and accuracy for others. We use a more lenient EM score, which evaluates performance based on whether the model generations include gold answers instead of strictly exact matching [74].

Towards RAG capabilities evaluation, we adopt four metrics from RAGs, including Faithfulness, Context Relevancy, Answer Relevancy, and Answer Correctness. Faithfulness measures how factually consistent the generated answer is with the retrieved context. An answer is considered faithful if all claims made can be directly inferred from the provided context. Context Relevancy evaluates how relevant the retrieved context is to the original query. Answer Relevancy assesses the


Searching for Best Practices in Retrieval-Augmented Generation:Content Relevancy and Evaluation

Pertinence of the generated answer to the original query is measured by the Context Relevancy, which involves the accuracy of the generated answer when compared to the ground truth. Context Relevancy is calculated by the proportion of sentences within the retrieved context that are relevant for answering the given question. The formula is expressed as: \[ \text{context relevancy} = \frac{|S|}{|Total|} \] where \(|S|\) denotes the number of relevant sentences, and \(|Total|\) denotes the total number of sentences retrieved. These metrics are evaluated using the RAGAs framework, with GPT-4 serving as the judge.

Additionally, the cosine similarity between the retrieved document and the gold document is computed, known as Retrieval Similarity. The retrieved document and gold document are embedded, and the resulting embeddings are used to compute the cosine similarity.

Implementation Details: For Open-Domain QA and MultiHop QA datasets, the model’s maximum new token number is set to 100 tokens. For other datasets, it is set to 50 tokens. To handle excessively long retrieved documents, documents are truncated to 2048 words when evaluating RankLLAMa and LongLLMLingua.

During generation, greedy decoding is used. To compare the capabilities of different RAG modules, the 0-shot evaluation setting is adopted, with no in-context examples provided. In multiple choice and fact checking tasks, answers generated by the model may vary (e.g., “the answer is A” instead of “A”). Thus, responses generated are processed with regular expression templates to match them with gold labels.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Introduction

Retrieval augmented generation (RAG) has been shown to be a both effective and efficient approach for large language models (LLMs) to leverage external knowledge. RAG retrieves relevant information based on the query and then prompts an LLM to generate a response in the context of the retrieved information. This approach significantly expands LLM’s access to vast amounts of information at a minimal cost. However, recent LLMs like Gemini and GPT-4 have demonstrated exceptional capabilities in understanding long contexts directly. For example, Gemini 1.5 can process up to 1 million tokens (Reid et al., 2024). This prompts the need for a systematic comparison between long-context (LC) LLMs and RAG: on one hand, RAG conceptually acts as a prior, regularizing the attention of LLMs onto retrieved segments, thus avoiding the distraction of the irrelevant information and saving unnecessary attention computations; on the other hand, large-scale pretraining may enable LLMs to develop even stronger long-context capabilities. Therefore, we are motivated to compare RAG and LC, evaluating both their performance and efficiency. In this work, we systematically benchmark RAG and LC on various public datasets, gaining a comprehensive understanding of their pros and cons, and ultimately combining them to get the best of both worlds. Different from findings in previous work (Xu et al., 2023), we find that LC consistently outperform RAG in almost all settings (when resourced sufficiently). This demonstrates the superior progress of recent LLMs in long-context understanding. Despite the suboptimal performance, RAG remains relevant due to its significantly lower computational cost. In contrast to LC, RAG significantly decreases the input length to LLMs, leading to reduced costs, as LLM API pricing is typically based on the number of input tokens. (Google, 2024; OpenAI, 2024).


Moreover, our analysis reveals that the predictions from LC and RAG are identical for over 60% of queries. For these queries, RAG can reduce cost without sacrificing performance. Based on this observation, we propose SELF-ROUTE, a simple yet effective method that routes various queries to RAG or LC based on model self-reflection. With SELF-ROUTE, we significantly reduce the cost while achieving overall performance comparable to LC. For example, the cost is reduced by 65% for Gemini-1.5-Pro and 39% for GPT-4O. Fig. 1 shows the comparisons of LC, RAG and SELF-ROUTE using three recent LLMs: GPT-4O, GPT-3.5-Turbo and Gemini-1.5-Pro. In addition to quantitative evaluation, we provide a comprehensive analysis comparing RAG and LC, including common failure patterns of RAG, the trade-offs between cost and performance, and the results on additional synthetic datasets. Our analysis serves as a starting point, inspiring future improvements of RAG, and as a empirical guide for building long-context applications using RAG and LC.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Models and Retrievers

Three latest LLMs are evaluated, including Gemini-1.5-Pro (Reid et al., 2024), GPT-4O (OpenAI, 2024a), and GPT-3.5-Turbo (OpenAI, 2023 2). Gemini-1.5-Pro is a recent long-context LLM from Google, supporting up to 1 million tokens. GPT-4O, the newest lightweight yet strong LLM from OpenAI, supports 128k tokens. GPT-3.5-Turbo supports 16k tokens. Two retrievers are used in our study: Contriever (Izacard et al., 2021), which is a contrastively trained dense retriever outperforming BM25 on BEIR datasets, and Dragon (Lin et al., 2023), which is a recent generalizable dense retriever achieving high performance in both supervised and zero-shot settings without complex late interaction. Following (Xu et al., 2023), we divide long contexts into chunks of 300 words, and select the top k chunks based on the cosine similarity of the query embedding and the chunk embeddings. The chunks are ordered by the similarity scores, with the chunk index prepended at the beginning. Since black-box LLMs are pretrained on unknown datasets, the leakage of evaluation datasets may occur. Especially, some of the evaluation datasets are based on Wikipedia, which has likely been seen by LLMs during training. In some cases, we find that model may predict the correct answer using exactly the same words as the groundtruth even when they do not appear in the provided context. In our experiment, we try mitigating this issue by prompting the model to answer 'based only on the provided passage' for both RAG and LC. It remains an open question how to address the data leakage issue in LLM evaluation.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Benchmarking results

We benchmark the performance of LC and RAG across the nine datasets, using three recent LLMs: Gemini-1.5-Pro, GPT-4O and GPT-3.5-Turbo. Tab. 1 presents the results using the Contriever retriever, where rows *-1 and rows *-2 present the benchmarking results for LC and RAG respectively. Results using the Dragon retriever will be discussed in Sec. 5.3 and Tab. 2. As shown in Tab. 1, LC consistently outperforms RAG for all the three models, with a significant margin. On average, LC surpasses RAG by 7.6% for Gemini-1.5-Pro, 13.1% for GPT-4O, and 3.6% for GPT-3.5-Turbo. Noticeably, the performance gap is more significant for the more recent models (GPT-4O and Gemini-1.5-Pro) compared to GPT-3.5-Turbo, highlighting the exceptional long-context understanding capacity of the latest LLMs. However, there is an exception observed on the two longer datasets from ocBench (i.e., EnQA and En.MC), where RAG achieves higher performance than LC for GPT-3.5-Turbo. This result deviates from the overall trend, likely due to the significantly longer context in these datasets (147k words on average) compared with the limited context window (16k) of GPT-3.5-Turbo. This finding highlights the effectiveness of RAG when the input text considerably exceeds the model’s context window size, emphasizing a specific use case of RAG.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:4.2 Self-Route

Based on the above motivation, we propose SELF-ROUTE, a simple yet effective method combining RAG and LC to reduce cost while maintaining a performance comparable to LC. SELF-ROUTE utilizes LLM itself to route queries based on self-reflection, under the assumption that LLMs are well-calibrated in predicting whether a query is answerable given provided context. Concretely, our method consists of two steps: a RAG-and-Route step and a long-context prediction step. In the first step, we provide the query and the retrieved chunks to the LLM, and prompt it to predict whether the query is answerable and, if so, generate the answer. This is similar to standard RAG, with one key difference: the LLM is given the option to decline answering with the prompt 'Write unanswerable if the query can not be answered based on the provided text'. For the queries deemed answerable, we accept the RAG prediction as the final answer. For the queries deemed unanswerable, we proceed to the second step, providing the full context to the long-context LLMs to obtain the final prediction (i.e., LC).


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:4.3 Results

Rows *3 to *5 in Tab. 1 present the results of our method, utilizing the three LLMs. Rows *3 report the performance. Rows *4 show the percentage of answerable queries, as predicted in the RAG-and-Route step. Rows *5 display the percentage of tokens used by our method, compared to that of LC. In terms of performance (rows *3), SELF-ROUTE significantly outperforms RAG, achieving results comparable to LC. Across all three models, SELF-ROUTE surpasses RAG (rows *2) by over 5%. Compared to LC (rows *1), there is a slight performance drop for GPT-4O (-0.2%) and Gemini-1.5-Pro (-2.2%), but an improvement for GPT-3.5-Turbo (+1.7%). All three LLMs consistently route more than half of queries towards RAG, as shown in rows *4. For Gemini-1.5-Pro, the answerable percentage even reaches 81.74% (row 1-4). This indicates that RAG may answer most queries without the need for LC, confirming our initial motivation. Due to the high answerable rate, the number of tokens required is significantly reduced (rows *5). For example, GPT-4O uses only 61% tokens.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Ablations of k

Both RAG and SELF-ROUTE rely on the top-k retrieved text chunks. The larger k is, the longer context is fed into LLMs for RAG prediction as well as routing, resulting in different costs versus performances. To study the influence of k, in Fig. 3, we plot the performance and cost (i.e. input token percentage) curves when different ks are used.

In terms of performance, for both RAG and SELF-ROUTE, a larger k leads to better performance. While k increases, more and more chunks are fed into the LLMs, thus the performance gradually improves to approach LC. As can be seen in from the curves, the advantage of SELF-ROUTE is the most significant for smaller k. For example, when k = 1, RAG gets from 20.24% while SELF-ROUTE gets 37.9%, while when k is larger than 50, all three methods get similar performance.

However, the trend of cost is not monotonous for SELF-ROUTE. As seen, the cost reaches its minimum at k = 5. This is because when k increases, the cost of RAG (and routing) increases, but more queries are routed to RAG from LC, thus the overall cost may decrease. The sweet point of k might be different for each dataset, e.g. on average, k = 5 has the lowest cost as shown in the curves, but on some datasets, especially ones that contain extractive questions which does not need multi-hop reasoning (like NarrativeQA and QMSum), k = 1 leads to the lowest cost. This indicates that the optimal k depends on the nature of the task, as well as the performance requirement. We encourage future researchers to look for different ks when applying our method to various applications.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Why does RAG fail?

To gain a better understanding of why RAG lags behind LC, we analyze the failure reasons for the examples that cannot be answered by RAG. We first manually check some examples for which our RAG-and-Route step predicts “unanswerable” and summarize four typical failure reasons, then prompt LLM to classify all the examples.

The four reasons include: (A) The query requires multi-step reasoning so the results of previous steps are needed to retrieve information for later steps, e.g. “What nationality is the performer of song XXX”. (B) The query is general, e.g. “What does the group think about XXX”, which is challenging for the retriever to formulate a good query. (C) The query is long and complex, which is challenging for the retriever to understand. However, answering this kind of questions is arguably, an advantage of LLMs. (D) The query is implicit, demanding a thorough understanding of the entire context. For instance, in a lengthy conversational narrative about a space voyage, a question like “What caused the shadow behind the spaceship?” requires readers to connect the dots and deduce the answer, as there is no explicit mention of the shadow when the cause is revealed.

Using these reasons, we prompt Gemini-1.5-Pro with few-shot in-context examples that we manually annotated, to classify all the unanswerable.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Results on synthetic data

In this study, we mainly focus on real datasets, with a consideration that results on synthetic data, which are artificially created by researchers, may subject to dataset artifacts. We notice some methods that researchers adopted to create synthetic long context datasets may unconsciously, but largely, influence the performance comparison between RAG and LC. For example, here we describe the results on the “PassKey” dataset in ocBench and its variations. This “PassKey” dataset presents a needle-in-a-haystack test, where a sentence with a passkey (e.g. “the passkey is 123456”) is hidden within chunks of irrelevant text, and the model is asked to answer the question “what is the passkey?”. The task requires strong retrieval capability. On this dataset, RAG achieves 80.34% accuracy, outperforming LC, which gets 65.25% using Gemini-1.5-Pro. However, if the query is slightly modified as “What is the special token hidden inside the texts?”, RAG accuracy sharply drops to only 4.58%, while LC keeps roughly the same (69.32%). Another example: if the chunks contain two passkeys and the query is “which passkey is larger? First or second?”, then RAG (47.63%) under-performs LC (64.24%) as well. These examples demonstrate that the evaluation results highly subjects to artifacts in dataset construction, showing limitation of synthetic testing.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyaml Anadka, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023. L-eval: Instituting standardized evaluation for long context language models. arXiv preprint arXiv:2307.11088.

Anthropic. 2024. Claude 3.5 sonnet. https://www.anthropic.com/news/claude-3-5-sonnet/.

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511.

Yushi Bai, Xin Lv, Jiaxie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2023. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508.

Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, Gopher Cam Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pages 2206–2240. PMLR.

Chi-Min Chan, Chunpu Xu, Ruibin Yuan, Hongyin Luo, Wei Xue, Yike Guo, and Jie Fu. 2024. Rq-rag: Learning to refine queries for retrieval augmented generation. arXiv preprint arXiv:2404.00610.

Linjiao Chen, Matei Zaharia, and James Zou. 2023a. Frugalgpt 2. How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176.

Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023b. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2305.15599.

Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers anchored in research papers. arXiv preprint arXiv:2105.03011.

Google. 2024. Gemini pricing. https://ai.google.dev/pricing.

Greg Kamradt. 2023. Needle in a haystack - pressure testing Lms. https://github.com/gkamradt/LLMTest_NeedleInAHaystack.

Mandy Guo, Joshua Ainslie, David C Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2022. Longt5: Efficient text-to-text transformer for long sequences. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 724–736.

Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International conference on machine learning, pages 3929–3938. PMLR.

Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint arXiv:2011.01060.

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. 2024. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.00654.

Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. Distilling step-by-step: outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.03201.

Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118.

Gautier Izacard and Edouard Grave. 2020. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282.

Huijiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. Longlmliuga: Accelerating and enhancing lms in long context scenarios via prompt compression. arXiv preprint arXiv:2310.06839.

Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.00172.

Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317–328.

Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. 2024. In search of needles in a 10m haystack: Recurrent memory finds what lms miss. arXiv preprint arXiv:2402.10790.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Extracted Information from Research Paper

Mosh Levy, Alon Jacoby, and Yoav Goldberg. 2024. Same task, more tokens: the impact of input length on the reasoning performance of large language models. arXiv preprint arXiv:2402.14848.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474.

Xiaoan Li, Changtai Zhu, Linyang Li, Zhangyue Yin, Tianxiang Sun, and Xipeng Qiu. 2023. Llatrieval: Llm-verified retrieval for verifiable generation. arXiv preprint arXiv:2311.07388.

Sheng-Chieh Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, and Xilun Chen. 2023. How to train your dragon: Diverse augmentation towards generalizable dense retrieval. arXiv preprint arXiv:2302.07452.

Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.

Yuanhua Lv and ChengXiang Zhai. 2009. Adaptive relevance feedback in information retrieval. In Proceedings of the 18th ACM conference on Information and knowledge management, pages 255–264.

Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. Query rewriting for retrieval-augmented large language models. arXiv preprint arXiv:2305.14283.

Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yiwei Fang. 2024. Evaluating very long-term conversational memory of lm agents. arXiv preprint arXiv:2402.17753.

OpenAI. 2023. Gpt-3.5-turbo. https://platform. openai.com/docs/models/gpt-3-5-turbo.

OpenAI. 2024a. Gpt-4t0. https://openai.com/index/hello-gpt-4o/.

OpenAI. 2024b. Openai-api pricing. https://platform.openai.com/docs/overview.

Machel Reid, Nikolay Savinov, Denis Tyapkin, Dmitry Lepikhin, Timothy Lilllicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini: 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.

Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, et al. 2022. Scrolls: Standardized comparison over long language sequences. arXiv preprint arXiv:2201.03533.

Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023. Re:trieval-augmented black-box language models. arXiv preprint arXiv:2301.12656.

Mingyang Song, Mao Zheng, and Xuan Luo. 2024. Counting-stars: A simple, efficient, and reasonable strategy for evaluating long-context large language models. arXiv preprint arXiv:2403.11802.

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.

Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zhian Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammed Shoeybi, and Bryan Catanzaro. 2023. Retrieval meets long context large language models. arXiv preprint arXiv:2301.03025.

Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. 2024. Corrective retrieval augmented generation. arXiv preprint arXiv:2401.15884.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600.

Tao Yuan, Neufei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhang, Zheye Tan, Zhuhyao Yao, Dahua Lin, Boxun Li, et al. 2024. L2-eval: A balanced long-context benchmark with 5 length levels up to 256k. arXiv preprint arXiv:2402.05125.

Chengxiang Zhai and John Lafferty. 2001. Model-based feedback in the language modeling approach to information retrieval. In Proceedings of the tenth international conference on Information and knowledge management, pages 403–410.

Xinnrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, et al. 2024. Infinity bench: Extending long context evaluation beyond 100k tokens. arXiv preprint arXiv:2402.13718.

Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, et al. 2021. Qmsum: A new benchmark for query-based multidomain meeting summarization. arXiv preprint arXiv:2104.05938.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Dataset Details

We evaluate on 7 datasets from LongBench (Bai et al., 2023). NarrativeQA (Kočiský et al., 2018) is a question answering dataset, where the context is a long story like a novel or a movie script. Qasper (Dasigi et al., 2021) focuses on question answering over academic NLP papers and is annotated by NLP practitioners. MultiFieldQA, originally proposed in LongBench, contains human-annotated QA over documents and articles from multiple sources, including legal documents, government reports, encyclopedias, academic papers, etc. HotpotQA (Yang et al., 2018) contains two-hop questions written by native English speakers that requires reasoning over two related Wikipedia paragraphs in the long context. 2WikiMultihopQA (Ho et al., 2020) contains up to 5-hop questions that are synthesized through manually designed templates, ensuring that they cannot be solved through shortcuts. The questions in MuSiQue (Trivedi et al., 2022) are up to 4-hop, first constructed from single-hop question compositions, and then paraphrased by annotators for linguistic diversity. QMSum (Zhong et al., 2021) is a query-based summarization dataset over meeting scripts from multiple domains. \nWe evaluate on 2 datasets from ∞Bench (Zhang et al., 2024). En.QA contains human-annotated question-answer pairs for long novels, with key entity names manually replaced in order to avoid knowledge leakage due to model pretraining. EN.MC is annotated similarly to En.QA, but differs in that the model is presented with four challenging answer choices written by the annotators. \nTable 3 shows the details of the datasets, including the number of queries in each evaluation dataset and the average context length (i.e., number of words).


Table 4 shows the performance and token ratio for different k, which corresponds to Fig. 3. The performance of LC, which serves as an upper bound, is 45.53. The token ratio is computed the token counts for RAG or Self-Route divided the number of tokens required by LC. \nPerformance for different k values: \n- k=1: RAG 20.24, Self-Route 41.35 \n- k=5: RAG 37.92, Self-Route 43.33 \n- k=10: RAG 41.20, Self-Route 44.38 \n- k=50: RAG 44.06, Self-Route 45.19 \n- k=100: RAG 44.12, Self-Route 45.23 \nToken Ratio for different k values: \n- k=1: RAG 5.26, Self-Route 39.64 \n- k=5: RAG 17.02, Self-Route 38.63 \n- k=10: RAG 42.42, Self-Route 53.66 \n- k=50: RAG 95.29, Self-Route 102.97 \n- k=100: RAG 100.32, Self-Route 106.59 \nTable 4: Performance and token ratio for different k. This table corresponds to Fig. 3.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Prompts for Each Dataset

NarrativeQA: You are given a story, which can be either a novel or a movie script, and a question. Answer the question as concisely as you can, using a single phrase if possible. Do not provide any explanation. If the question cannot be answered based on the information in the article, write "unanswerable". Story: {context} Now, answer the question based on the story as concisely as you can, using a single phrase if possible. Do not provide any explanation. If the question cannot be answered based on the information in the article, write "unanswerable". Question: {input} Answer:

Qasper: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation. Article: {context} Answer the question based on the above article as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation. Question: input Answer:

MultiQA: Read the following text and answer briefly. {context} Now, answer the following question based on the above text, only give me the answer and do not output any other words. If the question cannot be answered based on the information in the article, write "unanswerable". Question: {input} Answer:

HotpotQA: Answer the question based on the given passages. Only give me the answer and do not output any other words. If the question cannot be answered based on the information in the article, write "unanswerable". The following are given passages. {context} Answer the question based on the given passages. Only give me the answer and do not output any other words. If the question cannot be answered based on the information in the article, write "unanswerable". Question: {input} Answer:

2WikiMQA: Answer the question based on the given passages. Only give me the answer and do not output any other words. If the question cannot be answered based on the information in the article, write "unanswerable". The following are given passages. {context} Answer the question based on the given passages. Only give me the answer and do not output any other words. If the question cannot be answered based on the information in the article, write "unanswerable". Question: {input} Answer:

MusiQue: Answer the question based on the given passages. Only give me the answer and do not output any other words. If the question cannot be answered based on the information in the article, write "unanswerable". The following are given passages. {context} Answer the question based on the given passages. Only give me the answer and do not output any other words. If the question cannot be answered based on the information in the article, write "unanswerable". Question: {input} Answer:

QMSum: You are given a meeting transcript and a query containing a question or instruction. Answer the query in one or more sentences. If the question cannot be answered based on the information in the article, write "unanswerable". Transcript: {context} Now, answer the query based on the above meeting transcript in one or more sentences. If the question cannot be answered based on the information in the article, write "unanswerable". Query: {input} Answer:

EN.QA: Read the book and answer the question. Be very concise in your answer. If the question cannot be answered based on the information in the article, write "unanswerable". {context} Question: {input} Only give me the answer and do not output any other words. If the question cannot be answered based on the information in the article, write "unanswerable". Answer:

EN.MC: Read the book and answer the question. If the question cannot be answered based on the information in the article, write "unanswerable". {context} Question: {input} {all_classes} Only output the letter of the correct answer and do not output any other words. If the question cannot be answered based on the information in the article, write "unanswerable". The letter of the correct answer is.”

Table 5: Prompts for each dataset.


Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach:Table 6: Prompt for the Failure Case Analysis

You are given some text chunks from an article, and a question. The text chunks are retrieved by an external retriever. Now:

(1) Tell whether the question can be answered based only on the provided text chunks.
(2) If the question can be answered, answer the question based on the texts as concisely as you can, using a single phrase if possible.
(3) If the question cannot be answered, choose the reason from the following:

A. The question needs multistep reasoning, thus it is hard to retrieve all the relevant chunks. For example, "What nationality is the performer of song You Can?" contains two steps: find the performer, then find the nationality of the performer. Other examples include "Where does the director of film Wine Of Morning work at?", "What is another notable work made by the author of Miss Sara Sampson?"
B. The question is a general query, thus it is hard to retrieve relevant chunks. For example, "What did the group think about Dave leaving?" is general because the group may include multiple persons, and they can have different thinkings.
C. The question is long and complex, which is hard for the retriever to encode it to retrieve relevant chunks. For example, "What did Julie Morgan elaborate on the online survey when talking about the evaluations on the legitimacy of the children’s rights, protection and demands?", "The Huskies football team were invited to the Alamo Bowl where they were defeated by a team coached by Art Briles and who played their home games at what stadium?"
D. The question is not explicit and requires comprehensive understanding of the whole story and cannot be solved using retrieval-augmented generation. For example, "What caused the shadow behind Koerber’s ship?" needs a comprehensive understanding of the whole story. Another example like "How many words are there in the article" also requires the complete article.
E. Others.

Keep the above reasons in mind, and choose the most possible reason if you think the question cannot be answered based on the text. Output the results in JSON format.

{in_context_examples}
Text: {context}
Question: {input}
Answer:


Evaluation of Retrieval-Augmented Generation: A Survey:Evaluation of Retrieval-Augmented Generation: A Survey

Abstract. Retrieval-Augmented Generation (RAG) has recently gained traction in natural language processing. Numerous studies and real-world applications are leveraging its ability to enhance generative models through external information retrieval. Evaluating these RAG systems, however, poses unique challenges due to their hybrid structure and reliance on dynamic knowledge sources. To better understand these challenges, we conduct a Unified Evaluation Process of RAG (unevar) and aim to provide a comprehensive overview of the evaluation and benchmarks of RAG systems. Specifically, we examine and compare several quantifiable metrics of the Retrieval and Generation components, such as relevance, accuracy, and faithfulness, within the current RAG benchmarks, encompassing the possible output and ground truth pairs. We then analyze the various datasets and metrics, discuss the limitations of current benchmarks, and suggest potential directions to advance the field of RAG benchmarks.

1 Introduction

Retrieval-Augmented Generation (RAG) efficiently enhances the performance of generative language models through integrating information retrieval techniques. It addresses a critical challenge faced by standalone generative language models: the tendency to produce responses that, while plausible, may not be grounded in facts. By retrieving relevant information from external sources, RAG significantly reduces the incidence of hallucinations or factually incorrect outputs, thereby improving the content’s reliability and richness. This fusion of retrieval and generation capabilities enables the creation of responses that are not only contextually appropriate but also informed by the most current and accurate information available, making RAG a development in the pursuit of more intelligent and versatile language models.


Evaluation of Retrieval-Augmented Generation: A Survey:Introduction to RAG Systems

Numerous studies of RAG systems have emerged from various perspectives since the advent of Large Language Models (LLMs) [55 45 59 42 41 69 16]. The RAG system comprises two primary components: Retrieval and Generation. The retrieval component aims to extract relevant information from various external knowledge sources. It involves two main phases, indexing and searching. Indexing organizes documents to facilitate efficient retrieval, using either inverted indexes for sparse retrieval or dense vector encoding for dense retrieval [16 12 28]. The searching component utilizes these indexes to fetch relevant documents on the user's query, often incorporating optional rerankers [43 9 52] to refine the ranking of the retrieved documents. The generation component utilizes the retrieved content and question query to formulate coherent and contextually relevant responses with the prompting and inferencing phases. As the “Emerging” ability [59] of LLMs and the breakthrough in aligning human commands [42], LLMs are the best performance choices model for the generation stage. Prompting methods like Chain of Thought (CoT) [60], Tree of Thought [65], Rephrase and Respond (RaR) [8] guide better generation results. In the inferencing step, LLMs interpret the prompted input to generate accurate and in-depth responses that align with the query’s intent and integrate the extracted information [35 9] without further finetuning, such as fully finetuning [16 17 68] or LoRA [21]. Appendix A details the complete RAG structure. Figure [1] illustrates the structure of the RAG systems as mentioned.


Evaluation of Retrieval-Augmented Generation: A Survey:Introduction

By the external dynamic database and the various downstream tasks, such as content creation or open domain question answering [16,70]. These challenges necessitate the development of comprehensive evaluation metrics that can effectively capture the interplay between retrieval accuracy and generative quality [217]. To clarify the elements further, we try to address the current gaps in the area, which differs from the prior RAG surveys [74,16,24] that predominantly collected specific RAG methods or data. We have compiled 12 distinct evaluation frameworks, encompassing a range of aspects of the RAG system. Following the procedure of making benchmarks, we analyze through targets, datasets and metrics mentioned in these benchmarks and summarize them into A Unified Evaluation Process of RAG (Auepora) as three corresponding phases. For this paper, we contribute in the following aspects: 1. Challenge of Evaluation: This is the first work that summarizes and classifies the challenges in evaluating RAG systems through the structure of RAG systems, including three parts retrieval, generation, and the whole system. 2. Analysis Framework: In light of the challenges posed by RAG systems, we introduce an analytical framework, referred to as A Unified Evaluation Process of RAG (Auepora), which aims to elucidate the unique complexities inherent to RAG systems and guide for readers to comprehend the effectiveness of RAG benchmarks across various dimensions. 3. RAG Benchmark Analysis: With the help of Auepora, we comprehensively analyze existing RAG benchmarks, highlighting their strengths and limitations and proposing recommendations for future developments in RAG system evaluation.


Evaluation of Retrieval-Augmented Generation: A Survey:Challenges in Evaluating RAG Systems

Evaluating hybrid RAG systems entails evaluating retrieval, generation and the RAG system as a whole. These evaluations are multifaceted, requiring careful consideration and analysis. Each of them encompasses specific difficulties that complicate the development of a comprehensive evaluation framework and benchmarks for RAG systems. Retrieval: The retrieval component is critical for fetching relevant information that informs the generation process. One primary challenge is the dynamic and vast nature of potential knowledge bases, ranging from structured databases to the entire web. This vastness requires evaluation metrics that can effectively measure the precision, recall, and relevance of retrieved documents in the context of a given query [52,32]. Moreover, the temporal aspect of information, where the relevance and accuracy of data can change over time, adds another layer of complexity to the evaluation process [6]. Additionally, the diversity of information sources and the possibility of retrieving misleading or low-quality information pose significant challenges in assessing the effectiveness of filtering and selecting the most pertinent information [39]. The traditional evaluation indicators for retrieval, such as Recall and Precision, cannot fully capture the nuances of RAG retrieval systems, necessitating the development of more nuanced and task-specific evaluation metrics [49].


Evaluation of Retrieval-Augmented Generation: A Survey:Evaluation Target (What to Evaluate?)

The combination of EOs and GTs in the RAG system can generate all possible targets, which is the fundamental concept of the Auepora (as shown in Figure 1). Once identified, these targets can be defined based on a specific pair of EOs or EO with GT, as illustrated in Figure 2 and used to analyze all aspects of current RAG benchmarks.

Retrieval: The EOs are the relevant documents for evaluating the retrieval component depending on the query. Then we can construct two pairwise relationships for the retrieval component, which are Relevant Documents ↔ Query, Relevant Documents ↔ Documents Candidates.

- Relevance (Relevant Documents ↔ Query) evaluates how well the retrieved documents match the information needed expressed in the query. It measures the precision and specificity of the retrieval process.
- Accuracy (Relevant Documents ↔ Documents Candidates) assesses how accurate the retrieved documents are in comparison to a set of candidate documents. It is a measure of the system’s ability to identify and score relevant documents higher than less relevant or irrelevant ones.

Generation: The similar pairwise relations for the generation components are listed below. The EOs are the generated text and phrased structured content. Then we need to compare these EOs with the provided GTs and labels.

- Relevance (Response ↔ Query) measures how well the generated response aligns with the intent and content of the initial query. It ensures that the response is related to the query topic and meets the query’s specific requirements.
- Faithfulness (Response ↔ Relevant Documents) evaluates if the generated response accurately reflects the information contained within the relevant documents and measures the consistency between generated content and the source documents.
- Correctness (Response ↔ Sample Response) Similar to the accuracy in the retrieval component, this measures the accuracy of the generated response against a sample response, which serves as a ground truth. It checks if the response is correct in terms of factual information and appropriate in the context of the query.

Fig. 2: The Target modular of the Auepora.


Evaluation of Retrieval-Augmented Generation: A Survey:Table Content: Evaluating Targets and Metrics

Category: Tool, Framework: TrueRAG Triad, Time: 2023.10, Raw Targets: Context Relevance, Answer Relevance, Groundedness, Faithfulness, Execution Time, Embed. Cost/Distance, Retrieval: LLM as a Judge, Generation: LLM as a Judge

Category: Tool, Framework: LangChain Bench., Time: 2023.11, Raw Targets: Execution Time, Retrieval: Accuracy, Generation: LLM as a Judge

Category: Tool, Framework: Databricks Eval, Time: 2023.12, Raw Targets: Comprehensiveness, Readability, Correctness, Retrieval: -, Generation: LLM as a Judge

Category: Benchmark, Framework: RAGAS, Time: 2023.09, Raw Targets: Context Relevance, Answer Relevance, Faithfulness, Retrieval: LLM as a Judge, Generation: LLM Gen + CoSim, LLM as a Judge

Category: Benchmark, Framework: RECALL, Time: 2023.11, Raw Targets: Response Quality, Robustness, Retrieval: -, Generation: BLEU, ROUGE-L

Category: Benchmark, Framework: ARES, Time: 2023.11, Raw Targets: Information Integration, Retrieval: LLM + Classifier, Generation: LLM + Classifier

Category: Benchmark, Framework: RGB, Time: 2023.12, Raw Targets: Noise Robustness, Retrieval: Negative Rejection, Counterfactual Robustness, Retrieval: -, Generation: Accuracy

Category: Benchmark, Framework: MultiHop-RAG, Time: 2024.01, Raw Targets: Retrieved Correctness, Retrieval Quality, Response Correctness, Retrieval: MAP, MRR, H_{rev}, Generation: LLM as a Judge

Category: Benchmark, Framework: CRUD-RAG, Time: 2024.02, Raw Targets: CREATE, READ, UPDATE, DELETE, Retrieval: -, Generation: ROUGE, BLEU, RAGQuestEval

Category: Benchmark, Framework: MedRAG, Time: 2024.02, Raw Targets: Accuracy, Consistency, Correctness, Retrieval: -, Generation: Accuracy

Category: Benchmark, Framework: FeB4RAG, Time: 2024.02, Raw Targets: Clarity, Coverage, Retrieval: -, Generation: Human Evaluation

Category: Benchmark, Framework: CDQA, Time: 2024.03, Raw Targets: Correctness, Faithfulness, Structural Output, Retrieval: -, Generation: F1, Exact-Match

Category: Benchmark, Framework: DomainRAG, Time: 2024.06, Raw Targets: Noise Robustness, Retrieval: -, Generation: LLM as a Judge

Category: Benchmark, Framework: ReEval, Time: 2024.06, Raw Targets: Hallucination, Retrieval: -, Generation: F1, Exact-Match, Human Evaluation

Category: Research, Framework: FiD-Light, Time: 2023.07, Raw Targets: Latency, Retrieval: -, Generation: Diversity, Cosine Distance, Retrieval: -

Category: Research, Framework: Diversity Reranker, Time: 2023.08, Raw Targets: -


Evaluation of Retrieval-Augmented Generation: A Survey:Evaluation Metrics

2024. Table 1 portrays this information, where each evaluation criterion is represented by a different colour. For example, FeB4RAG [57], the fourth from the last, has posited four standards based on [17] that comprise Consistency, Correctness, Clarity, and Coverage. Correctness is equivalent to accuracy in retrieval, and Consistency is tantamount to faithfulness in the generation component. While accuracy in retrieval gauges the correctness of the retrieved information, we posit that Coverage pertains to the coverage rate and is more associated with diversity. Therefore, we consider Coverage to be linked with diversity and an additional requirement in our proposed evaluation framework, which will be introduced subsequently. The remaining standard, Clarity, is also classified as an additional requirement in our proposed framework. The other tools and benchmarks are processed similarly.

Tools and benchmarks offer varying degrees of flexibility in evaluating datasets for RAG systems. Tools, which specify only evaluation targets, provide a versatile framework capable of constructing complete RAG applications and evaluation pipelines, as seen in works like [54][32][33]. Benchmarks, on the other hand, focus on different aspects of RAG evaluation with specific emphasis on either retrieval outputs or generation targets. For instance, RAGAs [14] and ARES [49] assess the relevance of retrieval documents, while RGB and MultiHop-RAG [6][52] prioritize accuracy, necessitating comparison with GTs. The [66] focuses on the Hallucination, which is a combination of faithfulness and correctness. All benchmarks consider generation targets due to their critical role in RAG systems, though their focus areas vary.


Evaluation of Retrieval-Augmented Generation: A Survey:Additional Requirements

Additional Requirement In addition to evaluating the two primary components outlined, a portion of the works also addressed some additional requirements of RAG (Black and Italics targets in Table 2). The requirements are as follows:

Latency [20][32], measures how quickly the system can find information and respond, crucial for user experience.
Diversity [43][32] checks if the system retrieves a variety of relevant documents and generates diverse responses.
Noise Robustness [6] assesses how well the system handles irrelevant information without affecting response quality.
Negative Rejection [6] gauges the system’s ability to refrain from providing a response when the available information is insufficient.
Counterfactual Robustness [6] evaluates the system’s capacity to identify and disregard incorrect information, even when alerted about potential misinformation.
More: For more human preferences considerations, there can be more additional requirements, such as readability [57][33], toxicity, perplexity [33], etc.

For the exception, CRUD-RAG [39] introduces a comprehensive benchmark addressing the broader spectrum of RAG applications beyond question-answering, categorized into Create, Read, Update, and Delete scenarios. This benchmark evaluates RAG systems across diverse tasks, including text continuation, question answering, hallucination modification, and multi-document summarization. It offers insights for optimizing RAG technology across different scenarios. DomainRAG [58] identifies six complex abilities for RAG systems: conversational, structural information, faithfulness, etc.


Evaluation of Retrieval-Augmented Generation: A Survey:Evaluation Dataset

In Table 2, distinct benchmarks employ varying strategies for dataset construction, ranging from leveraging existing resources to generating entirely new data tailored for specific evaluation aspects. Several benchmarks draw upon the part of KILT (Knowledge Intensive Language Tasks) benchmark [44] (Natural Questions [29], HotpotQA [63], and FEVER [53]) and other established datasets such as SuperGLUE [56] (MultiRC [10], and ReCoRD [71]) [49]. However, the drawback of using such datasets can’t solve the challenges in dynamic real-world scenarios. A similar situation can be observed in WikiEval, from Wikipedia pages post 2022, constructed by RAGAs [14].

The advent of powerful LLMs has revolutionized the process of dataset construction. With the ability to design queries and ground truths for specific evaluation targets using these frameworks, authors can now create datasets in the desired format with ease. Benchmarks like RGB, MultiHop-RAG, CRUD-RAG, and CDQA [6][52][39][62] have taken this approach further by building their own datasets using online news articles to test RAG systems’ ability to handle real-world information beyond the training data of LM frameworks. Most recently, DomainRAG [58] combines various types of QA datasets with single-doc, multi-doc, single-round, and multi-round. These datasets are generated from the yearly changed information from the college website for admission and enrollment, which forces the LLMs to use the provided and updated information.


Evaluation of Retrieval-Augmented Generation: A Survey:Evaluation Metric (How to quantify?)

Navigating the intricate terrain of evaluating RAG systems necessitates a nuanced understanding of the metrics that can precisely quantify the evaluation targets. However, creating evaluative criteria that align with human preferences and address practical considerations is challenging. Each component within the RAG systems requires a tailored evaluative approach that reflects its distinct functionalities and objectives.

Retrieval Metrics  Various targets can be evaluated with various metrics that correspond to the given datasets. This section will introduce several commonly used metrics for retrieval and generation targets. The metrics for additional requirements can also be found in these commonly used metrics. The more specifically designed metrics can be explored in the original paper via Table 1 as a reference.

For the retrieval evaluation, the focus is on metrics that can accurately capture the relevance, accuracy, diversity, and robustness of the information retrieved in response to queries. These metrics must not only reflect the system’s precision in fetching pertinent information but also its resilience in navigating the dynamic, vast, and sometimes misleading landscape of available data. The deployment of metrics like Misleading Rate, Mistake Reappearance Rate, and Error Detection Rate within the [38] benchmark underscores a heightened awareness of RAG systems’ inherent intricacies. The integration of MAP@K, MRR@K, and Tokenization with F1 into benchmarks like [52,62] mirrors a deepening comprehension of traditional retrieval’s multifaceted evaluation. While the [17] also emphasizes that this ranking-based evaluation methodology is not unsuitable for the RAG system, and should have more RAG-specific retrieval evaluation metrics. These metrics not only capture the precision and recall of retrieval systems but also account for the diversity and relevance of retrieved documents, aligning with the complex and dynamic nature of information needs in RAG systems. The introduction of LLMs as evaluative judges, as seen in [14], further underscores the adaptability and versatility of retrieval evaluation, offering a comprehensive and context-aware approach to assessing retrieval quality.

Non-Rank Based Metrics often assess binary outcomes—whether an item is relevant or not—without considering the position of the item in a ranked list. Notice, that the following formula is just one format of these metrics, the definition of each metric may vary by the different evaluating tasks.

- Accuracy is the proportion of true results (both true positives and true negatives) among the total number of cases examined.
- Precision is the fraction of relevant instances among the retrieved instances,

$$\text{Precision} = \frac{TP}{TP + FP}$$

where $TP$ represents true positives and $FP$ represents false positives.


Evaluation of Retrieval-Augmented Generation: A Survey:Metrics for Retrieval and Generation

- Recall at k (\text{Recall@}k) is the fraction of relevant instances that have been retrieved over the total amount of relevant cases, considering only the top k results.
\[ \text{Recall@}k = \frac{|RD \cap Top_{kd}|}{|RD|} \]
where RD is the relevant documents, and Top_{kd} is the top-k retrieved documents.

**Rank-Based Metrics** evaluate the order in which relevant items are presented, with higher importance placed on the positioning of relevant items at the ranking list.

- Mean Reciprocal Rank (MRR) is the average of the reciprocal ranks of the first correct answer for a set of queries.
\[ MRR = \frac{1}{|Q|} \sum_{i=1}^{|Q|} \frac{1}{rank_i} \]
where |Q| is the number of queries and rank_i is the rank position of the first relevant document for the i-th query.

- Mean Average Precision (MAP) is the mean of the average precision scores for each query.
\[ MAP = \frac{1}{|Q|} \sum_{q=1}^{|Q|} \frac{\sum_{k=1}^{n}(P(k) \times rel(k))}{|relevant \; documents|_q} \]
where P(k) is the precision at cutoff k in the list, rel(k) is an indicator function equaling 1 if the item at rank k is a relevant document, 0 otherwise, and n is the number of retrieved documents.

**Generation Metrics** In the realm of generation, evaluation transcends the mere accuracy of generated responses, venturing into the quality of text in terms of coherence, relevance, fluency, and alignment with human judgment. This necessitates metrics that can assess the nuanced aspects of language production, including factual correctness, readability, and user satisfaction with the generated content. The traditional metrics like **BLEU, ROUGE, and F1 Score** continue to play a crucial role, emphasizing the significance of precision and recall in determining response quality. Yet, the advent of metrics such as **Misleading Rate, Mistake Reappearance Rate, and Error Detection Rate** highlights an evolving understanding of RAG systems’ distinct challenges [38]. The evaluation done by humans is still a very significant standard to compare the performance of generation models with one another or with the ground truth. The approach of employing LLMs as evaluative judges [75] is a versatile and automatic method for quality assessment, catering to instances where traditional ground truths may be elusive [14]. This methodology benefits from employing prediction-powered inference (PPI) and context relevance scoring, offering a nuanced lens through which LLM output can be assessed [49]. The strategic use of detailed prompt templates ensures a guided assessment aligned with human preferences, effectively standardizing evaluations across various content dimensions [1]. This shift towards leveraging LLMs.


- **ROUGE**
  - *Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [37]*

ROUGE is a set of metrics designed to evaluate the quality of summaries by comparing them to human-generated reference summaries. ROUGE can be indicative of the content overlap between the generated text and the reference text. The variants of ROUGE measure the overlap of n-grams (ROUGE-N, ROUGE-W), word subsequences (ROUGE-L, ROUGE-S), and word pairs between the system-generated summary and the reference summaries.

- **BLEU**
  - *Bilingual Evaluation Understudy (BLEU) [43]*

BLEU is a metric for evaluating the quality of machine-translated text against one or more reference translations. BLEU calculates the precision of n-grams in the generated text compared to the reference text and then applies a brevity penalty to discourage overly short translations. BLEU has limitations, such as not accounting for the fluency or grammaticality of the generated text.

- **BertScore**
  - BertScore [72] leverages the contextual embedding from pre-trained transformers like BERT to evaluate the semantic similarity between generated text and reference text. BertScore computes token-level similarity using contextual embedding and produces precision, recall, and F1 scores. Unlike n-gram-based metrics, BertScore captures the meaning of words in context, making it more robust to paraphrasing and more sensitive to semantic equivalence.

- **LLM as a Judge**

Using “LLM as a Judge” for evaluating generated text is a more recent approach. [75] In this method, LLMs are used to score the generated text based on criteria such as coherence, relevance, and fluency. The LLM can be optionally finetuned on human judgments to predict the quality of unseen text or used to generate evaluations in a zero-shot or few-shot setting. This approach leverages the LLM's understanding of language and context to provide a more nuanced text quality assessment. For instance, [1] illustrates how providing LLM judges with detailed scoring guidelines, such as a scale from 1 to 5, can standardize the evaluation process. This methodology encompasses critical aspects of content assessment, including coherence, relevance, fluency, coverage, diversity, and detail - both in the context of answer evaluation and query formulation.

**Additional Requirements**

These additional requirements, such as latency, diversity, noise robustness, negative rejection, and counterfactual robustness, are used to ensure the practical applicability of RAG systems in real-world scenarios aligned with human preference. This section delves into the metrics used for evaluating these additional requirements, highlighting their significance in the comprehensive assessment of RAG systems.

- **Latency**

Latency measures the time taken by the RAG system to finish the response of one query. It is a critical factor for user experience, especially in interactive applications such as chatbots or search engines [20]. *Single Query Latency:* The mean time is taken to process a single query, including both retrieval and generating phases.


Evaluation of Retrieval-Augmented Generation: A Survey:Discussion

For RAG systems, traditional Question Answering (QA) datasets and metrics remain a common format for interaction. While these provide a basic verification of RAG’s capabilities, it becomes challenging to distinguish the impact of retrieval components when faced with strong Language Models (LLMs) capable of excelling in QA benchmarks. To comprehensively evaluate the performance of entire RAG systems, there is a need for diverse and RAG-specific benchmarks. Several papers offer guidance on improving QA format benchmarks, including variations in question types: from simple Wikipedia filling questions to multi-hop [52], multi-document questions [66] and single-round to multi-round dialogue [39 58]. For answers, aspects such as structural output [58], content moderation [6 54], and hallucination [66] can be considered when evaluating relevance, faithfulness, and correctness. In addition to these, RAG systems require additional requirements such as robustness to noisy documents, language expression, latency, and result diversity. Furthermore, research is needed on performance changes involving intermediate outputs and retrieved documents, as well as the relationship and analysis between retrieval metrics and final generation outputs. Regarding datasets, creating a universal dataset was challenging due to the target-specific nature of different RAG benchmarks. Tailored datasets [14 38 49 59 57] are necessary for a thorough evaluation, but this approach increases the effort and resources required. Moreover, the diversity of datasets, from news articles to structured databases [66], reflects the adaptability required of RAG systems but also poses a barrier to streamlined evaluation. Recently, with the cutting-edge performance of LLMs, complex data processing and automatic QA pair generation can be automated to achieve daily or finer-grained time resolution, preventing LLMs from cheating and evaluating the robustness of RAG systems in rapidly changing data.


When it comes to metrics, the use of LLMs as automatic evaluative judges signifies a burgeoning trend, promising versatility and depth in generative outputs with reasoning on a large scale compared to human evaluation. However, using “LLMs as a Judge” for responses presents challenges in aligning with human judgment, establishing effective grading scales, and applying consistent evaluation across varied use cases. Determining correctness, clarity, and richness can differ between automated and human assessments. Moreover, the effectiveness of example-based scoring can vary, and there’s no universally applicable grading scale and prompting text, complicating the standardization of “LLM as a Judge”.

In addition to the challenges mentioned above, it is important to consider the resource-intensive nature of using Large Language Models (LLMs) for data generation and validation. RAG benchmarks must balance the need for thorough evaluation with the practical constraints of limited computational resources. As such, it is desirable to develop evaluation methodologies that can effectively assess RAG systems using smaller amounts of data while maintaining the validity and reliability of the results.

Conclusion

This survey systematically explores the complexities of evaluating RAG systems, highlighting the challenges in assessing their performance. Through the proposed A Unified Evaluation Process of RAG, we outline a structured approach to analyzing RAG evaluation, focusing on targets, datasets, and measures. Our analysis emphasizes the need for targeted benchmarks that reflect the dynamic interplay between retrieval accuracy and generative quality and practical considerations for real-world applications. By identifying gaps in current methodologies and suggesting future research directions, we aim to contribute to more effective, and user-aligned benchmarks of RAG systems.


Evaluation of Retrieval-Augmented Generation: A Survey:References

1. Balaguer, A., Benara, V., Cunha, R.L.d.F., Filho, R.d.M.E., Hendry, T., Holstein, D., Marsman, J., Mecklenburg, N., Malvar, S., Nunes, L.O., Padilha, R., Sharp, M., Silva, B., Sharma, S., Aski, V., Chandra, R.: RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture. Tech. rep. (Jan 2024). http://arxiv.org/abs/2401.08406 arXiv:2401.08406 [cs] type: article.

2. Barnett, S., Kuruwilaan, S., Thudumu, S., Brannelly, Z., Abdelrazek, M.: Seven failure points when engineering a retrieval augmented generation system (Jan 2024). https://doi.org/10.48550/ARXIV.2401.05586

3. Best, M., Blach, N., Kubicek, A., Gerstenberger, R., Podstawski, M., Gianniazzi, L., Gajda, J., Lehmann, T., Niewiadomski, H., Nyczek, P., Hoefter, T.: Graph of thoughts: Solving collaborate problems with large language models. Proceedings of the AAAI Conference on Artificial Intelligence 2024 (AAAI'24) (Aug 2023). https://doi.org/10.48550/ARXIV.2308.09687

4. Blagojevic, V.: Enhancing RAG Pipelines in Haystack: Introducing DiversityRanker and LostInTheMiddleRanker (Aug 2023). https://towardsdatascience.com/enhancing-rag-pipelines-in-haystack-45f14e2be9f5

5. Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y, et al.: A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology 15(3), 1–45 (2024)

6. Chen, J., Lin, H., Han, X., Sun, L.: Benchmarking large language models in retrieval-augmented generation (Sep 2023). https://doi.org/10.48550/ARXIV.2309.15217

7. Cuconasu, F., Trappolini, G., Siciliano, F., Filice, S., Campagnano, C., Maarek, Y., Tonellotto, N., Silvestri, F.: The power of noise: Redefining retrieval for rag systems (Jan 2024). https://doi.org/10.48550/ARXIV.2401.14887

8. Deng, Y., Zhang, W., Chen, Z., Gu, Q.: Rephrase and respond: Let large language models ask better questions for themselves (Nov 2023). https://doi.org/10.48550/ARXIV.2311.04205

9. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: Burstein, J., Doran, C., Solorio, T. (eds.) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp. 4171–4186. Association for Computational Linguistics, Minneapolis, Minnesota (Jun 2019). https://doi.org/10.18653/v1/N19-1423 https://aclanthology.org/N19-1423

10. DeYoung, J., Jain, S., Rajani, N.F., Lehman, E., Xiong, C., Socher, R., Wallace, B.C.: Eraser: A benchmark to evaluate rationalized nlp models

11. Dinan , E., Roller, S., Shuster, K., Fan, A., Auli, M., Weston, J.: Wizard of Wikipedia: Knowledge-powered conversational agents. In: Proceedings of the International Conference on Learning Representations (ICLR) (2019)

12. Douze, M., Guzhva, A., Deng, C., Johnson, J., Zalitsay, G., Mazaré, P.E., Lomeli, M., Hossenli, L., Jegou, H.: The fais library (2024)

13. DuckDuckGo: DuckDuckGo — Privacy simplified. (2024). https://duckduckgo.com/hc/en/home

14. Es, S., James, J., Espinosa-Anke, L., Schockerat, S.: Ragas: Automated evaluation of retrieval augmented generation (Sep 2023). https://doi.org/10.48550/ARXIV.2309.15217


15. Fisch, A., Talmor, A., Jia, R., Seo, M., Choi, E., Chen, D.: MRQA 2019 shared task: Evaluating generalization in reading comprehension. In: Fisch, A., Talmor, A., Jia, R., Seo, M., Choi, E., Chen, D. (eds.) Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pp. 1–13. Association for Computational Linguistics, Hong Kong, China (Nov 2019). https://doi.org/10.18653/v1/D19-5801 https://aclanthology.org/D19-5801
16. Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Guo, Q., Wang, M., Wang, H.: Retrieval-Augmented Generation for Large Language Models: A Survey. Tech. rep. (Jan 2024). http://arxiv.org/abs/2312.10997 arXiv:2312.10997 [cs] type: article
17. Gianecchini, L., Scells, H., Deckers, N., Bevendorff, J., Wang, S., Kiesel, J., Syed, S., Frobe, M., Zuccon, G., Stein, B., Hagen, M., Potthast, M.: Evaluating Generative Ad Hoc Information Retrieval. Tech. rep. (Nov 2023). http://arxiv.org/abs/2311.04694 arXiv:2311.04694 [cs] type: article
18. Google: Programmable Search Engine | Google for Developers (2024). https://developers.google.com/custom-search
19. Gottschalk, S., Demidova, E.: Eventkg: A multilingual event-centric temporal knowledge graph (Apr 2018) https://doi.org/10.48550/arxiv.1804.04526
20. Hofstätter, S., Chen, J., Raman, K., Zaman, H.: Fib-LigH: Efficient and Effective Retrieval-Augmented Text Generation. In: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1437–1447. SIGIR '23, Association for Computing Machinery, New York, NY, USA (Jul 2023). https://doi.org/10.1145/3539618.3591687 https://doi.org/10.1145/3539618.3591687
21. Hu, L.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: LoRA: Low-Rank Adaptation of Large Language Models. Tech. rep. (Oct 2021). https://doi.org/10.48550/arxiv.2106.09685 http://arxiv.org/abs/2106.09685 arXiv:2106.09685 [cs] type: article
22. Huang, J., Shao, H., Chang, K.C.C., Xiong, J., Hwu, W.m.: Understanding jargon: Combining extraction and generation for definition modeling. In: Proceedings of EMNLP (2022).
23. Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., Liu, T.: A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions (Nov 2023). https://doi.org/10.48550/arxiv.2311.05232
24. Huang, Y., Huang, J.: A survey on retrieval-augmented text generation for large language models (Apr 2024). https://doi.org/10.48550/arxiv.2404.10981
25. Johnson, I., Douze, M., Jégou, H.: Billion-scale similarity search with GPUs. IEEE Transactions on Big Data 7(3), 535–547 (2019)
26. Kamaloo, E., Thakur, N., Lanas,ce, C., Ma, X., Yang, J.H., Lin, J.: Resources for brewing beer: Reproducible reference models and an official leaderboard (2023).
27. Kasai, J., Sakaguchi, K., Takahashi, Y., Bras, R.L., Asai, A., Yu, X., Radev, D., Smith, N.A., Choi, Y., Inui, K.: Realtime qa: What’s the answer right now? (Jul 2022). https://doi.org/10.48550/arxiv.2207.13332 https://arxiv.org/abs/2207.13332
28. Khattab, O., Zaharia, M.: Colbert: Efficient and effective passage search via contextualized late interaction over bert (Apr 2020). https://doi.org/10.48550/arxiv.2004.12832
29. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.W., Dai, A.M., Uszkoreit, J., Le, Q., Petrov, S.: Natural questions: A benchmark for question


Evaluation of Retrieval-Augmented Generation: A Survey:Research Paper References and Notes

16 Hao Yu, Aoran Gan, Kai Zhang, Shiwei Tong†, Qi Liu, and Zhaofeng Liu
answering research. Transactions of the Association for Computational Linguistics 7, 453–466 (2019). https://doi.org/10.1162/tacl_a_00276 https://doi.org/10.1162/tacl_a_00276
30. Lathia, A.R., Permasani, A.E., Setiawan, N.A.: Cosine similarity to determine similarity measure: Study case in online essay assessment. In: 2016 4th International Conference on Cyber and IT Service Management. pp. 1–6 (2016). https://doi.org/10.1109/CITSM.2016.7577578
31. Lanhamtani, J., Toshniwal, S., Weston, J., Szlam, A., Sukhbaatar, S.: Learning to reason and memorize with self-notes (May 2023). https://doi.org/10.48550/ARXIV.2305.00833
32. LangChain: Evaluating rag architectures on benchmark tasks (Nov 2023). https://langchain-ai.github.io/langchain-benchmarks/notebooks/retrieval/langchain_docs_qa.html
33. Leng, Q., Ulthenhuth, K., Polyzois, A.: Best Practices for LLM Evaluation of RAG Applications (Dec 2023). https://www.databricks.com/blog/llm-auto-eval-best-practices-RAG
34. Lewis, P., Perez, E., Pitkus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., Kiela, D.: Retrieval-augmented generation for knowledge-intensive NLP tasks. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. pp. 9459–9474. NIPS’20, Curran Associates Inc., Red Hook, NY, USA (Dec 2020)
35. Lewis, P., Perez, E., Pitkus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., Kiela, D.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Tech. rep. (Apr 2021). http://arxiv.org/abs/2005.11401 arXiv:2005.11401 ISI [cited by] (article)
36. Liang, X., Song, S., Niu, S., Li, Z., Xiong, F., Tang, B., Wang, W., Z., He, D., Cheng, P., Wang, Z., Deng, H.: Vuegrad: Benchmarking the hallucination of chinese large language models via unconstrained generation. arXiv preprint arXiv:2311.15296 (2023)
37. Lin, C.Y.: ROUGE: A package for automatic evaluation of summaries. In: Text Summarization Branches Out. pp. 74-81. Association for Computational Linguistics, Barcelona, Spain (Jul 2004). https://aclanthology.org/W04-1013
38. Liu, K., Huang, L., Li, S., Chen, S., Zhou, H., Meng, F., Zhou, J., Sun, X.: X-Recall: A benchmark for llms robustness against external counterfactual knowledge (Nov 2023). https://doi.org/10.48550/ARXIV.2311.08174
39. Liu, Y., Li, Z., Niu, S., Xiong, F., Tang, B., Wang, W., Wu, H., Liu, H., Xu, T., Chen, E., Luo, Y., Cheng, P., Deng, H., Wang, Z., Lu, Z.: Crud-rag: a comprehensive chinese benchmark for retrieval-augmented generation of large language models (Jan 2024). https://doi.org/10.48550/ARXIV.2401.17043
40. Microsoft: Web Search API | Microsoft Bing, https://www.microsoft.com/en-us/bing/apis/bing-web-search-api
41. OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anaktak, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Balestruc, P., Bao, H., Bavarian, M., Belgium, J., Bello, I., Berdine, J., Bernadett-Shapiro, G., Berner, C., Bogdanoff, L., Boiko, O., Boyd, M., Brakman, A.L., Brockman, G., Brooks, T., Brundage, M., Button, K., Cai, T., Campbell, R., Can, A., Carey, B., Carlson, C., Carmichael, R., Chan, B., Chang, C., Chantzis, F., Chen, D., Chen, S., Chen, F., Chen, J., Chen, M., Chess, B., Cho, C., Chu, C., Chung, H.W., Cummings, D., Currier, J., Dai, Y., Decarreaux, C., Deprey, T., Deutsch, N., Devillie, D., Dhar, A., Dohan, D., Dowling, S., Dunning, S., Ecoffe, A., Eleti, A., Eloundou, T., Farhi, D., Fedus, L., Felix, N., Fishman, S.P., Forte, J., Fulford, I., Gao, L., Georges, E., Gibson, C., Goel, V., Vogineni, T.