Graph Retrieval-Augmented Generation: A Survey:Reference List

[52] Ziniu Hu, Yichong Xu, Wenhao Yu, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Kai-Wei Chang, and Yizhou Sun. 2022. Empowering Language Models with Knowledge Graph Reasoning for Open-Domain Question Answering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022. 9562–9581.
[53] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhanyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaochen Feng, Bing Qin, and Ting Liu. 2023. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. arXiv:2311.05232 [cs.CL] https://arxiv.org/abs/2311.05232
[54] Yizheng Huang and Jimmy Huang. 2024. A Survey on Retrieval-Augmented Text Generation for Large Language Models. arXiv:2404.10981 [cs.IR] https://arxiv.org/abs/2404.10981
[55] Yongfeng Huang, Yanyang Li, Yichong Xu, Lin Zhang, Ruiyi Gan, Jiaxing Zhang, and Liwei Wang. 2023. MVP-Tuning: Multi-View Knowledge Retrieval with Prompt Tuning for Commonsense Reasoning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023. 13417–13432.
[56] Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2021. (Comet-) Atomic 2020: On Symbolic and Neural Commonsense Knowledge Graphs. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021. 6384–6392.
[57] Omid Jafari, Preeti Maurya, Parth Nagarkar, Khandker Mushfiqul Islam, and Chidambaram Crusev. 2021. A Survey on Locality Sensitive Hashing Algorithms and their Applications. arXiv:2102.08942 [cs.DB] https://arxiv.org/abs/2102.08942
[58] Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Xin Zhao, and Ji-Rong Wen. 2023. StructGPT: A General Framework for Large Language Model to Reason over Structured Data. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023. 9237–9251.
[59] Jinhao Jiang, Kun Zhou, Ji-Rong Wen, and Wayne Xin Zhao. 2022. SGreat Truths are Always Simple: $ A Rather Simple Knowledge Encoder for Enhancing the Commonsense Reasoning Capacity of Pre-Trained Models. In Findings of the Association for Computational Linguistics: NAACL 2022, Seattle, WA, United States, July 10-15, 2022. 1730–1741.
[60] Jinhao Jiang, Kun Zhou, Wayne Xin Zhao, Yang Song, Chen Zhu, Hengshu Zhu, and Ji-Rong Wen. 2024. KG-Agent: An Efficient Autonomous Agent Framework for Complex Reasoning over Knowledge Graph. arXiv:2402.11163 [cs.CL] https://arxiv.org/abs/2402.11163
[61] Jinhao Jiang, Kun Zhou, Xin Zhao, and Ji-Rong Wen. 2023. UniKGQA: Unified Retrieval and Reasoning for Solving Multi-hop Question Answering Over Knowledge Graph. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023.
[62] Jinhao Jiang, Kun Zhou, Xin Zhao, and Ji-Rong Wen. 2023. UniKGQA: Unified Retrieval and Reasoning for Solving Multi-hop Question Answering Over Knowledge Graph. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023.
[63] Kelvin Jiang, Dekun Wu, and Hui Jiang. 2019. FreebaseQA: A New Factoid QA Data Set Matching Trivia-Style Question-Answer Pairs with Freebase. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers). 318–323.
[64] Xinke Jiang, Ruizhe Zhang, Yongxin Xu, Hongbin Qiu, Yue Fang, Zhiyuan Wang, Jinyi Tang, Hongxin Ding, Xu Chu, Junfeng Zhao, and Yasha Wang. 2024. HyKGE: A Hypothesis Knowledge Graph Enhanced Framework for Accurate and Reliable Medical LLMs Responses. arXiv:2312.15883 [cs.CL] https://arxiv.org/abs/2312.15883
[65] Bowen Jin, Gang Liu, Chi Han, Meng Jiang, Heng Ji, and Jiawei Han. 2024. Large Language Models on Graphs: A Comprehensive Survey. arXiv:2312.02783 [cs.CL] https://arxiv.org/abs/2312.02783
[66] Bowen Jin, Chulin Xie, Jiawei Zhang, Kashob Kumar Roy, Yu Zhang, Zheng Li, Ruirui Li, Xianfeng Tang, Suhang Wang, Yu Meng, and Jiawei Han. 2024. Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphs. arXiv:2404.07103 [cs.CL] https://arxiv.org/abs/2404.07103
[67] Di Jin, Eileen Pan, Osmat Nasrullah, Ethel-Wei Hung Wen, Hanyi Fang, and Peter Szolovits. 2020. What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams. arXiv:2009.13081 [cs.CL] https://arxiv.org/abs/2009.13081
[68] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers. 1601–1611.
[69] Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020. 6769–6781.


Graph Retrieval-Augmented Generation: A Survey:References

[70] Soham Kashyap et al. 2024. Knowledge Graph Assisted Large Language Models. (2024).
[71] Jiho Kim, Yeonsu Kwon, Yohan Jo, and Edward Choi. 2023. KG-GPT: A General Framework for Reasoning on
Knowledge Graphs Using Large Language Models. In Findings of the Association for Computational Linguistics: EMNLP
2023, Singapore, December 6-10, 2023 9410–9421.
[72] Jaewoong Kim and Moohong Min. 2024. From RAG to QA-RAG: Integrating Generative AI for Pharmaceutical
Regulatory Compliance Process. arXiv:2402.01717 [cs.CL] https://arxiv.org/abs/2402.01717
[73] Jiho Kim, Sungjin Park, Yeonsu Kwon, Yohan Jo, James Thorne, and Edward Choi. 2023. FactKG: Fact Verification
via Reasoning on Knowledge Graphs. In Proceedings of the 61st Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023 16109–16206.
[74] Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. In 5th
International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track
Proceedings.
[75] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle
Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei
Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural Questions: A Benchmark for
Question Answering Research. Trans. Assoc. Comput. Linguistics 7 (2019), 452–466.
[76] Yunshi Lan, Gaole He, Jinhao Jiang, Jing Jiang, Wayne Xin Zhao, and Ji-Rong Wen. 2021. A Survey on Complex
Knowledge Base Question Answering: Methods, Challenges and Solutions. In Proceedings of the Thirtieth International
Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021. 4483–4491.
[77] Yunshi Lan, Gaole He, Jinhao Jiang, Jing Jiang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Complex Knowledge Base
Question Answering: A Survey. IEEE Trans. Knowl. Data Eng. 35, 11 (2023), 11916–11215.
[78] Yunshi Lan and Jing Jiang. 2020. Query Graph Generation for Answering Multi-hop Complex Questions from
Knowledge Bases. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL
2020, Online, July 5-10, 2020. 969–974.
[79] Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The Power of Scale for Parameter-Efficient Prompt Tuning. In
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event /
Punta Cana, Dominican Republic, 7-11 November, 2021. 3045–3059.
[80] Dawei Li, Shun Yang, Zhen Tan, Jae Young Baik, Sukwon Yun, Joseph Lee, Aaron Chacko, Bojian Hou, Duy Duong-Tran,
Ying Ding, Huan Liu, Ji Shen, and Tanlong Chen. 2024. DALiK: Dynamic Co-Augmentation of LLMs and KG to answer
Alzheimer’s Disease Questions with Scientific Literature. arXiv:2405.04189 [cs.CL] https://arxiv.org/abs/2405.04189
[81] Shiyang Li, Yifan Gao, Haoming Jiang, Qiuyuan Liu, Zheng Li, Xifeng Yan, Chao Zhang, and Bing Yin. 2023. Graph
Reasoning for Question Answering with Triplet Retrieval. In Findings of the Association for Computational Linguistics:
ACL 2023, Toronto, Canada, July 9-14, 2023. 3366–3375.
[82] Xiang Lisa Li and Percy Liang. 2021. Prefix-Tuning: Optimizing Continuous Prompts for Generation. In Proceedings of
the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on
Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021. 4582–4597.
[83] Yuhan Li, Zhixun Li, Peisong Wang, Jia Li, Xiangguo Sun, Hong Cheng, and Jeffrey Xu Yu. 2024. A Survey of Graph
Meets Large Language Model: Progress and Future Directions. arXiv:2311.12399 [cs.LG] https://arxiv.org/abs/2311.
12399
[84] Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen. 2024. Large Language Models in Finance: A Survey.
arXiv:2311.10723 [q-fin.GN] https://arxiv.org/abs/2311.10723
[85] Yihao Li, Ru Zhang, and Jianyi Liu. 2024. An Enhanced Prompt-Based LLM Reasoning Scheme via Knowledge
Graph-Integrated Collaboration. arXiv:2402.04978 [cs.CL] https://arxiv.org/abs/2402.04978
[86] Zhuoyang Li, Liran Deng, Hui Liu, Qiaoqiao Liu, and Junzhao Du. 2024. UniOQA: A Unified Framework for Knowledge
Graph Question Answering with Large Language Models. arXiv:2406.02110 [cs.CL] https://arxiv.org/abs/2406.02110
[87] Zijian Li, Qingyan Guo, Jiawei Shao, Lei Song, Jiang Bian, Jun Zhang, and Rui Wang. 2024. Graph Neural Network
Enhanced Retrieval for Question Answering of LLMs. arXiv:2406.06572 [cs.CL] https://arxiv.org/abs/2406.06572
[88] Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019. KagNet: Knowledge-Aware Graph Networks for
Commonsense Reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing
and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China,
November 3-7, 2019. 2829–2839.
[89] Bill Yuchen Lin, Ziyi Yu, Yichi Yang, Dong-Ho Lee, and Xiang Ren. 2021. RiddleSense: Reasoning about Riddle Ques-
tions Featuring Linguistic Creativity and Commonsense Knowledge. In Findings of the Association for Computational
Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021 (Findings of ACL, Vol. ACL/IJCNLP 2021). 1504–1515.
[90] Guangyi Liu, Yongji Zhang, Tong Liu, and Quaning Yao. 2024. Explore then Determine: A GNN-LLM Synergy
Framework for Reasoning over Knowledge Graph. arXiv:2406.01145 [cs.CL] https://arxiv.org/abs/2406.01145


Graph Retrieval-Augmented Generation: A Survey:Graph Retrieval-Augmented Generation: A Survey

[91] H. Liu and P. Singh. 2004. ConceptNet—a practical commonsense reasoning tool-kit. BT technology journal 22, 4 (2004), 211–226.

[92] J. Liu, C. Yang, Y. Zhu, L. Junze, Y. Chen, Y. Li, M. Zhang, T. Bai, Y. Fang, L. Sun, P. S. Yu, and C. Shi. 2024. Towards Graph Foundation Models: A Survey and Beyond. arXiv:2310.11829 [cs.LG] https://arxiv.org/abs/2310.11829

[93] L. Liu, X. Yang, J. Lei, X. Liu, Y. Shen, Z. Zhang, P. Wei, J. Gu, Z. Chu, Z. Qin, and K. Ren. 2024. A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directions. arXiv:2406.03712 [cs.CL] https://arxiv.org/abs/2406.03712

[94] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. Trans. Assoc. Comput. Linguistics 12 (2024), 157–173.

[95] X. Liu, K. Ji, Y. Fu, W. Tam, Z. Du, Z. Yang, and J. Tang. 2022. P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 61–68.

[96] X. Liu, Y. Zhang, Z. Du, M. Ding, Y. Qian, Z. Yang, and J. Tang. 2023. GPT Understands, Too. arXiv:2103.10385 [cs.CL] https://arxiv.org/abs/2103.10385

[97] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692 [cs.CL] https://arxiv.org/abs/1907.11692

[98] P.-C. Lo and F.-P. Lin. 2023. Contextual Path Retrieval: A Contextual Entity Relation Embedding-based Approach. ACM Trans. Inf. Syst. 41, 1 (2023), 1:1–1:38.

[99] L. Logeswaran, M.-W. Chang, K. Lee, K. Toutanova, J. Devlin, and H. Lee. 2019. Zero-Shot Entity Linking by Reading Entity Descriptions. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28 - August 2, 2019, Volume 1: Long Papers. 3449–3460.

[100] D. Luo, J. Sheng, H. Xu, L. Wang, and B. Wang. 2023. Improving Complex Knowledge Base Question Answering with Relation-Aware Subgraph Retrieval and Reasoning Network. In International Joint Conference on Neural Networks, IJCNN 2023, Gold Coast, Australia, June 18-23, 2023, 1–8.

[101] H. Luo, H. Ye, E. Tang, S. Peng, Y. Guo, W. Zha, C. M. Hao, G. Dong, M. Song, W. Lin, Y. Zhu, and L. A. Tuan. 2024. ChatKBQA: A Generate-then-Retrieve Framework for Knowledge Base Question Answering with Fine-tuned Large Language Models. arXiv:2310.08975 [cs.CL] https://arxiv.org/abs/2310.08975

[102] L. Luo, Y.-F. Li, G. Hafari, and S. Ruan. 2024. Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning. arXiv:2310.01061 [cs.CL] https://arxiv.org/abs/2310.01061

[103] X. Ma, Y. Gong, P. He, H. Zhao, and N. Duan. 2023. Query Rewriting for Retrieval-Augmented Large Language Models. arXiv:2305.14283 [cs.CL] https://arxiv.org/abs/2305.14283

[104] H. Mao, Z. Chen, W. Tang, J. Zhao, Y. Ma, T. Zhao, N. Shah, M. Galkin, and J. Tang. 2024. Position: Graph Foundation Models Are Already Here. In Forty-first International Conference on Machine Learning.

[105] Q. Mao, Z. Liu, Z. Luo, and J. Sun. 2024. Advancing Graph Representation Learning with Large Language Models: A Comprehensive Survey of Techniques. arXiv:2402.05952 [cs.LG] https://arxiv.org/abs/2402.05952

[106] S. Mao, Y. Jiang, B. Chen, X. Li, P. Wang, X. Wang, P. Xie, F. Huang, H. Chen, and N. Zhang. 2024. RaRe: Ranking Feedback Improves Query Rewriting for RAG. arXiv:2405.14431 [cs.CL] https://arxiv.org/abs/2405.14431

[107] C. Mavromatis and G. Karypis. 2022. RaeRev: Adaptive Reasoning for Question Answering over Knowledge Graphs. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022. 2447–2458.

[108] C. Mavromatis and G. Karypis. 2024. GNN-RAG: Graph Neural Retrieval for Large Language Model Reasoning. arXiv:2405.20139 [cs.CL] https://arxiv.org/abs/2405.20139

[109] T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal. 2018. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018. 2381–2391.

[110] A. H. Miller, A. Fisch, J. Dodge, A.-H. Karimi, A. Bordes, and J. Weston. 2016. Key-Value Memory Networks for Directly Reading Documents. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016. 1400–1409.

[111] S. Moon, P. Shah, A. Kumar, and R. Subba. 2019. OpenDialKG: Explainable Conversational Reasoning with Attention-based Walks over Knowledge Graphs. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28 - August 2, 2019, Volume 1: Long Papers. 845–854.


[112] Christopher Morris, Nils M. Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. 2020. TU-Dataset: A collection of benchmark datasets for learning with graphs. In ICML 2020 Workshop on Graph Representation Learning and Beyond (GRL- 2020).

[113] Sai Munikoti, Anurag Acharya, Sridevi Wagle, and Sameera Horawalavithana. 2023. ATLANTIC: Structure-Aware Retrieval-Augmented Language Model for Interdisciplinary Science. arXiv:2311.12289 [cs.CL] https://arxiv.org/abs/2311.12289

[114] Yuqi Nie, Yaxuan Kong, Xiaowen Dong, John M. Mulvey, H. Vincent Poor, Qingsong Wen, and Stefan Zohren. 2024. A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges. arXiv:2406.11903 [q-fin.GN] https://arxiv.org/abs/2406.11903

[115] Yasumasa Onoe, Michael J. Q. Zhang, Eunsol Choi, and Greg Durrett. 2021. CREAK: A Dataset for Commonsense Reasoning over Entity Knowledge. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual.

[116] OpenAI. 2024. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] https://arxiv.org/abs/2303.08774

[117] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems 35 (2022), 27730–27744.

[118] Vardaan Pahuja, Boshi Wang, Hugo Latapie, Jayanth Srinivasa, and Yu Su. 2023. A Retrieve-and-Read Framework for Knowledge Graph Link Prediction. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM 2023, Birmingham, United Kingdom, October 21–25, 2023. 1992–2002.

[119] Jeff Z. Pan, Simon Razniewski, Jan-Christoph Kalo, Sneha Singhainia, Jiaoyan Chen, Stefan Dietze, Hajira Jabeen, Janna Omelianenko, Wen Zhang, Matteo Lissandrini, Russa Biswas, Gerard de Melo, Angela Bonifati, Eldira Vakaj, Mauro Dragoni, and Damen Graux. 2023. Large Language Models and Knowledge Graphs: Opportunities and Challenges. TGDK 1, 1 (2023), 2:1–2:38.

[120] Shurui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xinlong Wu. 2024. Unifying Large Language Models and Knowledge Graphs: A Roadmap. IEEE Trans. Knowl. Data Eng. 36, 7 (2024), 3580–3599.

[121] Wenjun Feng, Guiyang Li, Yue Jiang, Zilong Wang, Dan Ou, Xiaoyi Zeng, Derong Xu, Tong Xu, and Enhong Chen. 2024. Large Language Model based Long-tail Query Rewriting in Taobao Search. In Companion Proceedings of the ACM on Web Conference 2024. WWW 2024, Singapore, Singapore, May 13–17, 2024. 20–28.

[122] Zhuoyi Peng and Yi Yang. 2024. Connecting the Dots: Inferring Patent-Phrase Similarity with Retrieved Phrase Graphs. arXiv:2403.16625 [cs.CL] https://arxiv.org/abs/2403.16625

[123] Aleksandra Deroyer, Dennis Diefenbach, Ricardo Usbeck, and Andreas Both. 2022. QALD-9-plus: A Multilingual Dataset for Question Answering over DBpedia and Wikidata Translated by Native Speakers. In 16th IEEE International Conference on Semantic Computing, ICSC 2022, Laguna Hills, CA, USA, January 26-28, 2022. 229–234.

[124] Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick S. H. Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: A Benchmark for Knowledge Intensive Language Tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021. 2523–2544.

[125] Zhixiao Qi, Yijiong Yu, Meiqi Tu, Junyi Tan, and Yongfeng Huang. 2023. FoodGPT: A Large Language Model in Food Testing Domain with Incremental Pre-training and Knowledge Graph Prompt. arXiv:2308.10173 [cs.CL] https://arxiv.org/abs/2308.10173

[126] Zile Qiao, Wei Ye, Yong Jiang, Tong Mo, Pengjun Xie, Weiping Li, Fei Huang, and Shikun Zhang. 2024. Supportiveness-based Knowledge Rewriting for Retrieval-augmented Language Modeling. arXiv:2406.08116 [cs.CL] https://arxiv.org/abs/2406.08116

[127] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. J. Mach. Learn. Res. 21 (2020), 140:1–140:67.

[128] Priyanka Ranade and Anupam Joshi. 2023. FABULA: Intelligence Report Generation Using Retrieval-Augmented Narrative Construction. In Proceedings of the International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2023, Kusadasi, Turkey, November 6-9, 2023. 603–610.

[129] Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019. 3980–3990.

[130] Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. 2020. DropEdge: Towards Deep Graph Convolutional Networks on Node Classification. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.


[131] Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2019. ATOMIC: An Atlas of Machine Commonsense for If-Then Reasoning. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019. 3027–3035.

[132] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. SocialIQA: Commonsense Reasoning about Social Interactions. arXiv:1904.09728 [cs.CL] https://arxiv.org/abs/1904.09728

[133] Apoorv Saxena, Aditya Tripathi, and Partha P. Talukdar. 2020. Improving Multi-hop Question Answering over Knowledge Graphs using Knowledge Base Embeddings. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020. 4498–4507.

[134] Priyanka Sen, Alham Fikri Aji, and Amir Saffari. 2022. Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answering. In Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 2022. 1604–1619.

[135] Ahsan Shehzad, Feng Xia, Shagufta Abid, Ciyuan Peng, Shuo Yu, Dongyu Zhang, and Karin Verspoor. 2024. Graph Transformers: A Survey. arXiv:2407.09777 [cs.LG] https://arxiv.org/abs/2407.09777

[136] Yiheng Shu, Zhiwei Yu, Yuhan Li, Börje F. Karlsson, Tingting Ma, Yuzhong Qu, and Chin-Yew Lin. 2022. TIARA: Multi-grained Retrieval for Robust Question Answering over Large Knowledge Bases. arXiv:2210.12925 [cs.CL] https://arxiv.org/abs/2210.12925

[137] Saurabh Srivastava, Milind D Jain, Harshita Jain, Kritik Jaroli, VJ Mayank Patel, and L Khan. 2020. IOT monitoring bin for smart cities. In 3rd Smart Cities Symposium (SCS 2020), Vol. 2020. IET, 533–536.

[138] Fabian M Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: a core of semantic knowledge. In Proceedings of the 16th international conference on World Wide Web. 697–706.

[139] Haitian Sun, Tania Bedrax-Weiss, and William W. Cohen. 2019. PullNet: Open Domain Question Answering with Interactive Retrieval on Knowledge Bases and Text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019. 2380–2390.

[140] Haitian Sun, Bhuvan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William W. Cohen. 2018. Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018. 4231–4242.

[141] Hao Sun, Yang Li, Liwei Deng, Bowen Li, Binyuan Hui, Binhua Li, Yunshi Lan, Yan Zhang, and Yongbin Li. 2023. History Sensitive Graph Enhanced Conversational KBQA with Temporal Information Modeling. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023. 3521–3533.

[142] Jiashuo Sun, Chengjin Xu, Lunniyyuan Tang, Saizhou Wang, Chen Lin, Yeyun Gong, Lionel M. Ni, Heung-Yeung Shum, and Jian Guo. 2024. Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph. arXiv:2307.07697 [cs.CL] https://arxiv.org/abs/2307.07697

[143] Lei Sun, Zhengwei Tao, Youdi Li, and Hiroshi Arakawa. 2024. ODA: Observation-Driven Agent for integrating LLMs and Knowledge Graphs. arXiv:2404.07677 [cs.CL] https://arxiv.org/abs/2404.07677

[144] Alon Talmor and Jonathan Berant. 2018. The Web as a Knowledge-Base for Answering Complex Questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers). 641–651.

[145] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers). 4149–4158.

[146] Dhaval Taunk, Lakshya Khanna, Siri Venkata Pavan Kumar Kandra, Vasudeva Varma, Charu Sharma, and Makarand Tapaswi. 2023. GrapeQA: Graph Augmentation and Pruning to Enhance Question-Answering. In Companion Proceedings of the ACM Web Conference 2023, WWW 2023, Austin, TX, USA, 30 April 2023 - 4 May 2023. 1138–1144.

[147] Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, and Michael Gamon. 2015. Representing Text for Joint Embedding of Text and Knowledge Bases. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015. 1499–1509.

[148] Hugo Touvron, Louis Martin, and et al. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288 [cs.CL] https://arxiv.org/abs/2307.09288

[149] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All You Need. In Advances in Neural Information Processing Systems 30: Annual


[169] Zhentao Xu, Mark Jerome Cruz, Matthew Guevara, Tie Wang, Manasi Deshpande, Xiaofeng Wang, and Zheng Li. 2024. Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024. 2905–2909.

[170] An Yang, Baosong Yang, and et al. 2024. Qwen2 Technical Report. arXiv:2407.10671 [cs.CL] https://arxiv.org/abs/2407.10671

[171] Rui Yang, Haoran Liu, Edison Marese-Taylor, Qingcheng Zeng, Yu He Ke, Wanxin Li, Lechao Cheng, Qingyu Chen, James Cavelere, Yutaka Matsuo, and Irene Li. 2024. KG-Rank: Enhancing Large Language Models for Medical QA with Knowledge Graphs and Ranking Techniques. arXiv:2403.05881 [cs.CL] https://arxiv.org/abs/2403.05881

[172] Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Daniel Gui, Ziran Will Jiang, Ziyu Jiang, Lingkun Kong, Brian Moran, Jiaqi Wang, Yifan Ethan Xu, An Yan, Chenyu Yang, Eting Yuan, Hanwen Zha, Nan Tang, Lei Chen, Nicolas Scheffer, Yue Liu, Nirav Shah, Rakesh Wanga, Anuj Kumar, Wen tau Yih, and Xin Luna Dong. 2024. CRAG - Comprehensive RAG Benchmark. arXiv:2406.04744 [cs.CL] https://arxiv.org/abs/2406.04744

[173] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018. 2369–2380.

[174] Mohammad Yani and Adila Alfa Krisnadhi. 2021. Challenges, Techniques, and Trends of Simple Knowledge Graph Question Answering: A Survey. Inf. 12, 7 (2021), 271.

[175] Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021. QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021. 535–546.

[176] Ruosong Ye, Caiji Zhang, Runhui Wang, Shuyuan Xu, and Yongfeng Zhang. 2024. Language is All a Graph Needs. arXiv:2308.07134 [cs.CL] https://arxiv.org/abs/2308.07134

[177] Xi Ye, Samin Yavar, Kazuma Hashimoto, Yingbo Zhou, and Caiming Xiong. 2021. Rng-kbqa: Generation augmented iterative ranking for knowledge base question answering. arXiv preprint arXiv:2109.06876 (2021).

[178] Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming-Wei Chang, and Jina Suh. 2016. The Value of Semantic Parse Label Feedback for Knowledge Base Question Answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 2: Short Papers.

[179] Donghan Yu, Sheng Zhang, Patrick Ng, Henghui Zhu, Alexander Hanbo Li, Jun Wang, Yiqun Hu, William Yang Wang, Zhigao Wang, and Bing Xiang. 2023. DecAF: Joint Decoding of Answers and Logical Forms for Question Answering over Knowledge Bases. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023.

[180] Hao Yu, Aoran Gan, Kai Zhang, Shiwei Tong, Qi Liu, and Zhaofeng Liu. 2024. Evaluation of Retrieval-Augmented Generation: A Survey. arXiv:2405.07437 [cs.CL] https://arxiv.org/abs/2405.07437

[181] Jing Zhang, Xiaokang Zhang, Jifan Yu, Jian Tang, Jie Tang, Cuiping Li, and Hong Chen. 2022. Subgraph Retrieval Enhanced Model for Multi-hop Knowledge Base Question Answering. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022. 573–578.

[182] Mengmei Zhang, Mingwei Sun, Peng Wang, Shen Fan, Yanhu Mo, Xiaoxiao Xu, Hong Liu, Cheng Yang, and Chuan Shi. 2024. GraphTranslator: Aligning Graph Model to Large Language Model for Open-ended Tasks. In Proceedings of the ACM on Web Conference 2024, WWW 2024, Singapore, May 13-17, 2024. 1003–1014.

[183] Qinggang Zhang, Junnan Dong, Hao Chen, Daochen Zha, Zailiang Yu, and Xiao Huang. 2024. KnowGPT: Knowledge Graph based Prompting for Large Language Models. arXiv:2312.06185 [cs.CL] https://arxiv.org/abs/2312.06185

[184] Xiukun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D. Manning, and Jure Leskovec. 2022. GreaseLM: Graph REasoning Enhanced Language Models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022.

[185] Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexander J. Smola, and Le Song. 2018. Variational Reasoning for Question Answering With Knowledge Graph. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th Innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018. 6069–6076.

[186] Jiannan Zhao, Le Zhuo, Yikang Shen, Meng Qu, Kai Liu, Michael Bronstein, Zhaocheng Zhu, and Jian Tang. 2023. GraphText: Graph Reasoning in Text Space. arXiv:2310.01089 [cs.CL] https://arxiv.org/abs/2310.01089


[187] Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang, Wentao Zhang, Jie Jiang, and Bin Cui. 2024. Retrieval-Augmented Generation for AI-Generated Content: A Survey. arXiv:2402.19473 [cs.CV] https://arxiv.org/abs/2402.19473

[188] Yanxin Zheng, Wensheng Gan, Zefeng Chen, Zhenlian Qi, Qian Liang, and Philip S. Yu. 2024. Large Language Models for Medicine: A Survey. arXiv:2405.13055 [cs.CL] https://arxiv.org/abs/2405.13055

[189] Yun Zhu, Yaoke Wang, Haizhou Shi, and Siliang Tang. 2024. Efficient Tuning and Inference for Large Language Models on Textual Graphs. arXiv:2401.15569 [cs.CL] https://arxiv.org/abs/2401.15569


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Abstract

Retrieval augmented generation (RAG) combines the generative abilities of large language models (LLMs) with external knowledge sources to provide more accurate and up-to-date responses. Recent RAG advancements focus on improving retrieval outcomes through iterative LLM refinement or self-critique capabilities acquired through additional instruction tuning of LLMs. In this work, we introduce SPECULATIVE RAG – a framework that leverages a larger generalist LM to efficiently verify multiple RAG drafts produced in parallel by a smaller, distilled specialist LM. Each draft is generated from a distinct subset of retrieved documents, offering diverse perspectives on the evidence while reducing input token counts per draft. This approach enhances comprehension of each subset and mitigates potential position bias over long context. Our method accelerates RAG by delegating drafting to the smaller specialist LM, with the larger generalist LM performing a single verification pass over the drafts. Extensive experiments demonstrate that SPECULATIVE RAG achieves state-of-the-art performance with reduced latency on TriviQA, MuSiQue, PubHealth, and ARC-Challenge benchmarks. It notably enhances accuracy by up to 12.97% while reducing latency by 51% compared to conventional RAG systems on PubHealth.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:2 Related Works

Retrieval Augmented Generation (RAG) enhances LLMs by retrieving relevant documents from external databases and incorporating them into the generation process. Recent work has primarily focused on enabling LLMs to understand when and what to retrieve. Approaches include using a pre-trained LLM on web search data to filter irrelevant contexts. Self-Reflective RAG introduces reflection tokens to guide retrieval and annotation in instruction-tuning datasets. However, retrieval of relevant documents cannot entirely compensate for generation inefficiencies in traditional LLMs. Furthermore, long-context retrieval methods can suffer from inefficiencies.

Corrective RAG proposes a lightweight retrieval evaluator but lacks high-level reasoning. Our proposed SPECULATIVE RAG addresses these limitations by leveraging a smaller RAG drafter model to efficiently understand diverse perspectives and generate drafts for the generalist LMs to verify and integrate.

Speculative Decoding aims to reduce auto-regressive decoding latency through a draft-then-verify paradigm. This involves drafting multiple future tokens with a small model and verifying them in parallel with the target model. The draft model is typically either an independent model from the same series or the target model itself. Our approach extends this concept from token-level drafting to answer-level drafting. In contrast to traditional verification criteria, which accept or reject tokens based on their generation probabilities, we leverage language modeling objectives to directly assess the confidence of entire answer drafts.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Algorithm 1: SPECULATIVE RAG

Data: (Q, D = {d1}^n) is the question and n retrieved documents; m subsets, each containing k documents, are sampled from D, k also corresponds to the number of clusters during clustering. Result: Â is the predicted answer to the question. 1: Function Speculative RAG (Q, D, m, k): 2. {c1, c2, ..., ck} ⟵ kMeans C(d1, ..., dn|Q) ▷ Cluster the documents into k groups using an embedding model C. 3. Δ ⟵ {} 4. repeat 5. δj ⟵ {} ▷ Construct a subset of the retrieved documents δj. 6. for ci ∈ {c1, ..., ck} do 7. δj = δj ∪ {random.sample(ci)} ▷ Sample one document from each cluster ci into subset δj. 8. end 9. Δ = Δ ∪ {δj} 10. until |Δ| = m 11. for δj ∈ Δ do in parallel ▷ Process m subsets in parallel. 12. αj, βj ⟵ Mdrafter.generate(Q, δj) ▷ Generate the draft α and rationale β with Mdrafter. 13. ρj ⟵ Mverifier.score(αj|Q,βj) ▷ Compute the confidence score ρ with Mverifier. 14. end 15. ˆℓ = arg maxαj ρj ▷ Select the one with the highest score as the final answer. 16. return Â.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Detailed Explanation of Algorithm

Specifically, as shown in Algorithm 1, we first cluster the retrieved documents with regard to their relation to the posed question, where each cluster represents one perspective in the retrieval results (Line 2). Then we sample one document from each cluster into a subset so the documents in this subset cover the multiple perspectives in the retrieval results. We aim at minimizing redundancy and increase the diversity of the documents (Line 5 to 8). We denote one subset as δj ⊆ D that contains retrieved documents with diverse contents and multiple perspectives in the retrieval results. Then, we distribute each subset δ to a RAG drafter endpoint Mdrafter with the posed question Q to generate the answer draft α and the rationale β in parallel (Line 12). The RAG drafter is instruction-tuned to be a specialist in understanding the retrieved documents and produce rationales that are faithful to the input documents. It is smaller than generalist LMs, and its parallel processing further ensures high efficiency. For each draft-rationale pair (αj, βj) from Mdrafter, we compute a confidence score with the generalist LM Mverifier based on the question Q and corresponding rationale β (Line 13). It is worth mentioning that Mverifier does not need to be instruction-tuned since we leverage its language modeling ability already learned during pre-training. Meanwhile, Mverifier can verify the drafts based on the informative rationale provided by Mdrafter instead of processing tedious or possibly redundant retrieved documents. Finally, we select the answer draft with the highest confidence score as the final answer and integrate it into the generation results of the generalist LM (Line 15).


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Multi-Perspective Sampling

For each knowledge-intensive question, we retrieve a set of documents from the database using the posed question as the retrieval query. These documents may contain diverse content due to the ambiguity inherent in the query. To minimize redundancy and enhance diversity of the document subsets used for generating answer drafts, we employ a multi-perspective sampling strategy. We first cluster the documents into a few topics using an instruction-aware embedding model (Peng et al., 2024) and the K-Means clustering algorithm (Jin & Han, 2011). \[ \text{emb}(d_1),..., \text{emb}(d_n) = E(d_1,...,d_n|Q) \] \[ \{c_1,...,c_k\} = K\text{-Means}(\text{emb}(d_1),..., \text{emb}(d_n)) \] \[ \delta = \{\text{random.sample}(c) \text{ for } c \in \{c_i\}_1^k\} \] where \(E\) is an instruction-aware embedding model which embeds a string with regard to a provided instruction (the posed question \(Q)\); \(\text{emb}(d_i)\) is the embedding for the retrieved document \(d_i\); \(c_i\) is a cluster of retrieved documents with similar topics and contents; \(k\) is a hyper-parameter that controls the number of clusters. We sample one document from each cluster into a document subset \(\delta\) so each subset contains \(k\) documents of diverse contents. In total, we construct \(m\) subsets for parallel inference with the RAG drafter.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:RAG Drafting

We run \(M_{\text{Drafter}}\) over the \(m\) document subsets and produce corresponding answer drafts. Refer to Appendix B for detailed prompt design. We incorporate each document subset into the prompt and query \(M_{\text{Drafter}}\) for responses. We obtain \(m\) drafts as the answer candidates and each draft is grounded based on the multiple perspectives in the retrieval results. Specifically, given a document subset \(\delta_j = \{d_{j,1},...,d_{j,k}\}\), we query \(M_{\text{Drafter}}\) in parallel with the following prompt for the answer draft and rationale: \(Q; d_{j,1},...,d_{j,k} \rightarrow \alpha_j, \beta_j\), where the prompt contains the posed question \(Q\) along with the document subset; the generation result contains the answer draft \(\alpha_j\) and the rationale \(\beta_j\). We denote the conditional generation probability as \(P_{\text{draft}} = P(\beta|Q, d_{j,1},...,d_{j,k}, \alpha_j, \beta_j)\), which measures the reliability of generating rationales and the confidence in producing answer drafts.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Generalist RAG Verifier

After generating drafts and the rationale from the RAG drafter \(M_{\text{Drafter}}\), we evaluate them by a generalist LM \(M_{\text{Verifier}}\) to filter out the less reliable drafts and select the best answer. The generalist LM can be any off-the-shelf pre-trained LM. We only consider the draft-rationale pair \((\alpha, \beta)\) and skip the tedious and redundant retrieval results. We resort to the language modeling ability of the generalist LM to rank and select the draft-rationale pairs. \n\nEvaluation Scores: First, we calculate the self-consistency score by determining the conditional probability of generating a draft-rationale pair given the question, \(P_{\text{self-consist}} = P(\alpha, \beta|Q)\). This score helps assess whether the draft and rationale are self-consistent in the context of the question. Given the characteristics of language modeling, a self-consistent draft-rationale pair is expected to yield a higher probability. Furthermore, we incorporate a self-reflection statement \(R\) that prompts \(M_{\text{Verifier}}\) to assess the reliability of an answer draft (e.g. “Do you think the rationale supports the answer, yes or no?”). We define the self-reflection score as \(P_{\text{self-reflect}} = P(\text{Yes}|Q, \alpha, \beta, R)\) where we compute the conditional probability of the positive answer (“Yes”) to the self-reflection statement. \n\nComputation Method: We can efficiently compute the self-consistency and self-reflection scores within one forward pass of \(M_{\text{Verifier}} \). Given a question \(Q\) and a draft-rationale pair \((\alpha, \beta)\), we construct a prompt \([Q, \alpha, \beta, R, "\text{Yes}"],\) where \(R\) is the self-reflection statement. We encode the prompt with \(M_{\text{Verifier}}\) and acquire the probability of each token conditioned on the previous tokens \(P(t_i|t_{<i})\). We leverage this auto-regressive feature and aggregate the probability of the relevant answer.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Experiments

We evaluate our proposed SPECULATIVE RAG on four public retrieval augmented generation benchmarks: TriviaQA (unfiltered) (Joshi et al., 2017), MuSiQue (Trivedi et al., 2022), PubHealth (Zhang et al., 2023b), and ARC-Challenge (Clark et al., 2018). TriviaQA and MuSiQue are challenging open-domain question answering datasets where RAG systems are required to answer questions on factual knowledge. TriviaQA typically requires one accurate piece of evidence from the documents, whereas MuSiQue demands multiple documents to construct a multi-hop reasoning chain. Following previous works (Guu et al., 2020; Asai et al., 2023; Yan et al., 2024), we evaluate performance of the free-form generation based on whether gold answers are contained within the generated response or not. PubHealth and ARC-Challenge are closed-set generation datasets. PubHealth is a dataset of medical claims spanning a variety of biomedical subjects and it requires the RAG system to verify a given claim based on the retrieved documents. ARC-Challenge introduces a multi-choice question answering dataset, composed of science exam questions from grade 3 to grade 9. For closed-set generation tasks, we use accuracy metrics to evaluate whether the generated answers match the ground truth.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Experiment Settings

In our experiments, we utilize Mistral78 (v0.1) as our base LM for the RAG drafter. For RAG verifier, we employ either Mistral78 (v0.1) or Mistralx78 (v0.1) without any fine-tuning, denoted as M_Verific:78 or M_Verificx:78. We pre-compute embeddings of retrieved documents using a lightweight instruction-aware embedding model InEmbedformer (Peng et al., 2024) as part of the retrieval process. Inference is conducted using the VLLM framework (Kwon et al., 2023) with greedy decoding (temperature=0). We adopt the same experiment settings from Asai et al. (2023) and include a more challenging benchmark, MuSiQue (Trivedi et al., 2022). Our focus is on RAG reasoning rather than evidence elicitation, so we omit the other two long-form generation benchmarks, Biography (Min et al., 2023) and ALCE-ASQA (Gao et al., 2023a). On TriviaQA, PubHealth, and ARC-Challenge, we retrieve top 10 documents and generate 5 drafts per query (m = 5), with each draft based on a subset of 2 documents (k = 2). For the MuSiQue dataset, we retrieve top 15 documents and generate 10 drafts.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Latency Analysis

We analyze the latency of Standard RAG and our SPECULATIVE RAG on TriviaQA, MuSiQue, PubHealth, and ARC-Challenge. We randomly sample 100 cases from each dataset and report the average time cost for each case, as shown in Figure 2. To simulate real-world application scenarios, we process cases individually without batching. As representative example, we run \(M_{Verific}^{s/t,8} + M_{Drafter}^{8}\) for SPECULATIVE RAG and Mixtral-Instruct$_{\times 78}$ for Standard RAG, as these demonstrate the highest performance among competitive baselines (see Table 1). We launch 5 endpoints of \(M_{Drafter}^{t/8}\) for parallel drafting on TriviaQA, PubHealth, and ARC-Challenge. We launch 10 endpoints for MuSiQue due to more drafts. We use tensor parallelism to fit Mixtral-Instruct$_{\times 78}$ into the GPU memory. We report the latency of Mixtral-Instruct$_{\times 78}$ under tensor parallelism sizes of 4, 8, 16. Increasing tensor parallelism does not improve efficiency due to overheads in tensor parallel draft generation, consistently achieves the lowest latency across all datasets. Particularly, it reduces latency by up to 23.41% on TriviaQA, 17.28% on MuSiQue, 51.25% on PubHealth, and 26.73% on ARC-Challenge. This highlights the advantage of our approach in reducing processing time while maintaining high performance.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Ablation Studies

We conduct ablation studies on the key components of SPECULATIVE RAG during both drafting and verification stages on TriviaQA and PubHealth in Table 2. We use \(M_{Verific}^{s/t,8} + M_{Drafter}^{8}\) as a running configuration. Same as the main results, we report the accuracy as performance metrics.\n\nDiversity and reduced redundancy in retrieval improves draft quality significantly. In the first set of experiments, we evaluate the impact of multi-perspective sampling during the drafting. Recall that SPECULATIVE RAG clusters retrieved documents into distinct perspectives and sample one document from each cluster to reduce redundancy for the draft generation. We compare this against two alternative sampling strategies: (1) Random sampling without clustering, where we randomly select a document subset as context, and (2) Sampling from the same cluster, where we select all documents from a single cluster. Our results indicate that our proposed sampling method yields the best performance thanks to its ability to leverage diverse context. Particularly, it improves the accuracy by up to 1.88% on TriviaQA and 2.32% on PubHealth. While random sampling without clustering introduces diversity, it is prone to including redundant documents, degrading draft quality. Sampling from the same cluster significantly underperforms due to a lack of diverse perspectives.\n\nScoring method on self-consistency and self-reflection refines draft quality effectively. In the second set of experiments, we examine the scoring method during verification. We remove each of the specific confidence scores, \(P_{self-contain}\) or \(P_{self-reflect}\) in turn. Performance drops are observed when any score is removed. Particularly, removing \(P_{draft}\) leads to a minimal decline, 0.19% on TriviaQA and 1.12% on PubHealth, likely due to the limited verification capability of the smaller RAG drafter. Removing either \(P_{self-contain}\) or \(P_{self-reflect}\) results in similar performance decreases, around 2.0% on TriviaQA and around 0.8% on PubHealth, indicating that both self-containment and self-reflection capture different key aspects of reasoning and are crucial during verification.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:4.6 Effects of Generated Rationale for Verification

In SPECULATIVE RAG, we utilize the generated rationale β from the RAG drafter as an indicator of the trustworthiness of answer drafts α. The rationales highlight relevant points, omit redundant information, and bridge logical gaps between drafts and their supporting documents. To evaluate the effectiveness of the rationales, we examine two alternative scoring methods: (a) replacing rationale with retrieved documents (ρ = Scoreα(q, θ) δ), or (b) adding retrieved documents to rationale (ρ = Scoreα(q, β, δ)). We compare these alternatives to the scoring method used in SPECULATIVE RAGG (ρ = Scoreα(q, β)) in Table 3. The results show that incorporating longer retrieved documents does not consistently improve performance and tends to increase latency. This suggest that the generated rationale is already of high quality and serves as an effective bridge between the supporting documents and the generated answer drafts. By leveraging this rationale, we can efficiently verify drafts using a generic LM, leading to accurate final results.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:4.7 Effects of Draft Number and Document Subset Size

We investigate the performance of SPECULATIVE RAG under varying numbers of drafts. Using MVerifier-7B + MDrafter-7B with 5, 10, 15, 20 drafts on TriviaQA and PubHealth. We sample two documents as context per draft. The results are illustrated in Figure 3(a). Since we retrieve top 10 documents in total, we sample up to 20 drafts in these experiments. The results indicate that incorporating more drafts can further improve performance, likely thanks to higher coverage of diverse perspective of documents. Importantly, in SPECULATIVE RAG, we can launch multiple RAG drafter instances to generate drafts in parallel without additional latency. We also examine the effect of document subset size. By varying the number of documents (1, 2, 4, or 6) sampled for draft generation on TriviaQA and PubHealth (Figure 3(b)), we find that including more documents in the context does not always lead to consistent performance improvement. While TriviaQA queries may benefit from more supporting documents due to their complexity, MVerifier-7B + MDrafter-7B can surpass Mistral-Instructℰṛb even with a single supporting document per draft. Furthermore, with two or more documents per draft, MVerifier-7B + MDrafter-7B can even surpass Mixtral-Instructℰṛb. This further demonstrates the effectiveness of our drafting design.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:5 Conclusion

Our proposed SPECULATIVE RAG decomposes RAG tasks into two separate steps of drafting followed by verification. SPECULATIVE RAG delegates the heavy lifting of drafting to a small specialized RAG drafter, while verification is done using a large generalist LM. The parallel generation of multiple drafts from diverse document subsets provides high quality answer candidates while reducing input token counts and the potential risk of position-bias-over-long-context, resulting in substantial improvements in both the quality and speed of the final output generation. We demonstrate the effectiveness of SPECULATIVE RAG with accuracy gains up to 12.97% while reducing latency by 51% compared to conventional RAG systems. SPECULATIVE RAG sheds new light on the potential of collaborative architectures for enhancing RAG performance through task decomposition. Limitations: In this paper, we demonstrate that a smaller, specialized RAG drafter can work together with a larger general-purpose LM for knowledge-intensive tasks. While SPECULATIVE RAG enhances both accuracy and efficiency, it does require training an additional drafter model. Although this step is computationally inexpensive, it adds a layer of complexity compared to using a vanilla instruction-tuned RAG model.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Table 3: Performance and latency analysis of SPECULATIVE RAG on TriviaQA and PubHealth

We add the original document subset δ to the context or replace the generated rationale β with the original retrieved document subset δ during verification, i.e. we compute the self-containment score as ρ_self-contain = P(α, δ|Q) or P_self-contain = P(α, δ|Q), and compute the self-reflection score as P_self-reflect = P("yes"|Q, α, δ, β, R) or P_self-reflect = P("yes"|Q, α, δ, β, R), where Q is the query; α is the answer draft; R is the self-reflection statement. TriviaQA: Accuracy (%) and Latency (s): ρ = Score(α|Q, β) 74.24 and 1.93; ρ = Score(α|Q, δ) 74.08 (-0.16) and 2.13 (+10.36%); ρ = Score(α|Q, β, δ) 74.32 (+0.08) and 2.17 (+12.48%). PubHealth: Accuracy (%) and Latency (s): ρ = Score(α|Q, β) 76.60 and 1.17; ρ = Score(α|Q, δ) 76.09 (-0.51) and 1.31 (+11.97%); ρ = Score(α|Q, β, δ) 76.29 (-0.40) and 1.33 (+13.68%).


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:References

Achim, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anakdat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. \n Asai, A., Wu, Z., Wang, Y., Sil, A., and Hajishirzi, H. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511, 2023. \n Baek, J., Jeong, S., Kang, M., Park, J. C., and Hwang, S. Knowledge-augmented language model verification. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1720–1736, 2023. \n Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. \n Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J. D., Chen, D., and Dao, T. Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10747, 2024. \n Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023a. \n Chen, T., Wang, H., Chen, S., Yu, W., Ma, K., Zhao, X., Zhang, H., and Yu, D. Dense x retrieval: What retrieval granularity should we use? arXiv preprint arXiv:2312.06648, 2023b. \n Chen, W., Liu, Y., Wang, W., Bakker, E. M., Georgiou, T., Fleuguth, P., Liu, L., and Law, M. S. Deep learning for instance retrieval: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. \n Clark, P., Cowhey, J., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.


Ding, J., Ma, S., Dong, L., Zhang, X., Huang, S., Wang, W., Zheng, N., and Wei, F. Longnet: Scaling transformers to 1,000,000,000 tokens. arXiv preprint arXiv:2307.02486, 2023. 2

Dubois, Y., Li, C. X., Taori, R., Zhang, T., Gulrajani, I., Bai, J., Guestrin, C., Liang, P. S., and Hashimoto, T. B. AlpacaFarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024. 6, 7

Feng, S., Shi, W., Bai, Y., Balachandran, V., He, T., and Tsvetkov, Y. Knowledge card: Filling llms’ knowledge gaps with plug-in specialized language models. In The Twelfth International Conference on Learning Representations, 2023. 2

Gao, T., Yen, H., Yu, J., and Chen, D. Enabling large language models to generate text with citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 6465–6488, 2023a. 6

Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., and Wang, H. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2023b. 1, 3

Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929–3938. PMLR, 2020. 6

Hsieh, C.-Y., Li, C.-Y., Lek, C.-K., Nakhost, H., Fujii, Y., Ratner, A., Krishna, R., Lee, C.-Y., and Pfister, T. Distilling step-by-step: outperforming larger language models with less training data and smaller model sizes. ACL 2023, 4, 7

Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng X., Qin, B., et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232, 2023. 1

Izzaard, C. and Grave, E. Leveraging passage retrieval with generative models for open domain question answering. In Merlo, P., Tiedemann, J., and Tsarfaty, R. (eds.). Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 874–880, Dubrovnik, Croatia, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.eacl-main.74.3

Izacard, G., Caron, M., Hosseini, L., Reels, J., Bojanowski, P., Joulin, A., and Grave, E. Unsupervised dense retrieval with contrastive learning. arXiv preprint arXiv:2112.09118, 2021. 17

Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. I., Bressan, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023a. 6, 7

Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. I., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2410.04808, 2024. 6, 7

Jiang, X., Zhu, F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., and Neubig, G. Active retrieval augmented generation. In Boumaour, H., Pino, J., and Balik, K. (eds.). Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7969–7992, Singapore, December 2023b. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.495. URL https://aclanthology.org/2023.emnlp-main.495. 3

Jin X. and Han, J. K-means clustering. Encyclopedia of machine learning, pp. 563–564, 2011. 5

Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601–1611, 2017. 6

Kamalloo, E., Dziri, N., Clarke, C., and Rafiei, D. Evaluating open-domain question answering in the era of large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5591–5606, 2023. 1

Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., and Lewis, M. Generalization through Memorization: Nearest Neighbor Language Models. In International Conference on Learning Representations (ICLR), 2020. 3

Kim, J., Nam, J., Mo, S., Park, J., Lee, S.-W., Seo, M., Ha, J.-W., and Shin, J. Suer: Summarizing retrievals using answer candidates for open-domain qa flows. arXiv preprint arXiv:2404.13081, 2024. 3

Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems, 2022. 1


Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274–19268. PMLR, 2023. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474, 2020. Li, T., Zhang, G., Do, Q. D., Yue, X., and Chen, W. Long-context LMs struggle with long in-context learning. arXiv preprint arXiv:2404.02060, 2024. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 2024. Luo, H., Chuang, Y.-S., Gong, Y., Zhang, T., Kim, Y., Wu, X., Fox, D., Meng, H., and Glass, J. Sail: Search-augmented instruction learning. arXiv preprint arXiv:2305.15225, 2023. Luo, Y., Yang, Z., Meng, F., Li, Y., Zhou, J., and Zhang, Y. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08477, 2023. Ma, X., Gong, Y., He, P., Zhao, H., and Duan, N. Query rewriting in retrieval-augmented large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5303–5315, 2023. Ma, X., Yang, X., Xiong, W., Chen, B., Yu, L., Zhang, H., May, J., Zettlemoyer, L., Levy, O., and Zhou, C. Megacodon: Efficient llm pretraining and inference with unlimited context length. arXiv preprint arXiv:2404.08801, 2024. Miao, X., Olairo, G., Zhang, Z., Cheng, W., Wang, Z., Zhang, Z., Wong, R. Y. Y., Zhu, A., Yang, L., Shi, X., et al. Sspecinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 39th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, pp. 932–949, 2024. Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381–2391, 2018. Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-t., Koh, P., Iyyer, M., Zettlemoyer, L., and Hajishirzi, H. Factscore: Fine-grained automatic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12076–12100, 2023. Peng, L., Zhang, Y., Wang, Z., Srinivasa, J., Liu, G., Wang, Z., and Shang, J. Answer is all you need: Instruction-following text embedding via answering the question. arXiv preprint arXiv:2204.09642, 2024. Petroni, F., Piktus, A., Fan, A., Lewis, P., Yazdani, M., De Cao, N., Thorne, J., Jernite, Y., Karpukhin, V., Maillard, J., et al. Kilt: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2523–2544, 2021. Rasley, J., Rajbhandari, S., Ruwase, O., and He, Y. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3505–3506, 2020. Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lillcarr, T., Alayrac, J.-b., Soricut, R., Lazaridou, A., Firat, O., Schuster, A., et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05330, 2024. Sarthi, P., Abdullah, S., Talu, A., Khanna, S., Goldie, A., and Manning, C. D. Raptron: Recursive abstractive pretraining for tree-organized retrieval. arXiv preprint arXiv:2401.18059, 2024. Schick, T., Dwivedi-Vija, Y., Dessi, R., Rialaenu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., and Stenetorp, P. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36, 2024.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:References and Citations

1. Stelmakh, L., Luan, Y., Dhingra, B., and Chang, M.-W. Asqa: Factoid questions meet long-form answers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 8273–8288, 2022.

2. Stern, M., Shazeer, N., and Uszkoreit, J. Blockwise parallel decoding for deep autoregressive models. Advances In Neural Information Processing Systems, 31, 2018.

3. Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.

4. Trivedi, H., Balasubramanian, N., Khot, T., and Sabharwal, A. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554, 2022.

5. Wang, Y., Ivison, H., Dasiqi, P., Hessel, J., Khot, T., Chandu, K. R., Wadden, D., MacMillan, K., Smith, N. A., Beltagy, I., et al. How far can camels go. Exploring the state of instruction tuning on open resources, 2023.

6. Wang, Z., Araki, J., Jiang, Z., Parvez, M. R., and Neubig, G. Learning to filter context for retrieval-augmented generation. arXiv preprint arXiv:2311.08377, 2023.

7. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019.

8. Xia, H., Ge, T., Wang, P., Chen, S.-Q., Wei, F., and Sui, Z. Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 3909–3925, 2023.

9. Xia, H., Yang, Z., Dong, Q., Wang, P., Li, Y., Ge, T., Liu, T., Li, W., and Sui, Z. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. arXiv preprint arXiv:2401.07851, 2024.

10. Xie, J., Zhang, K., Chen, J., Lou, R., and Su, Y. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. In The Twelfth International Conference on Learning Representations, 2023.

11. Xu, F., Shi, W., and Choi, E. Recomp: Improving retrieval-augmented lms with compression and selective augmentation. arXiv preprint arXiv:2310.04408, 2023.

12. Xu, Z., Jain, S., and Kankanahalli, M. Hallucination is inevitable: An innate limitation of large language models. arXiv preprint arXiv:2401.11817, 2024.

13. Yan, S.-Q., Gu, J.-C., Zhu, Y., and Ling, Z.-H. Corrective retrieval augmented generation. arXiv preprint arXiv:2401.15884, 2024.

14. Yoran, O., Wolfson, T., Ram, O., and Berant, J. Making retrieval-augmented language models robust to irrelevant context. arXiv preprint arXiv:2310.01558, 2023.

15. Yu, W., Zhang, H., Pan, X., Ma, K., Wang, H., and Yu, D. Chain-of-note: Enhancing robustness in retrieval-augmented language models. arXiv preprint arXiv:2311.09210, 2023.

16. Zhang, J., Wang, J., Li, H., Shou, L., Chen, K., Chen, G., and Mehrotra, S. Draft & verify: Lossless large language model acceleration via self-speculative decoding. arXiv preprint arXiv:2309.08168, 2023.

17. Zhang, T., Luo, H., Chuang, Y.-S., Fang, W., Gaitskeil, L., Hartvigsen, T., Wu, X., Fox, D., Meng, H., and Glass, J. Interpretable unified language checking. arXiv preprint arXiv:2304.03728, 2023.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Appendix: Prompt of Rationale Generation

Prompt
Memorize this piece of evidence in mind and use it as if you already know it.
Evidence: State religion
Despite enjoying considerable popular support, Christianity was still not the official state religion in Rome, although it was in some neighboring states such as Armenia, Iberia, and Aksum. Roman Religion (Neoplatonic Hellenism) was restored for a time by the Emperor Julian from 361 to 363. Julian does not appear to have reinstated the persecutions of the earlier Roman emperors. Catholic Christianity, as opposed to Arianism and other ideologies deemed heretical, was declared to be the state religion of the Roman Empire on 27 February 380 by the decree "De fide catolica".
Instruction: When did Christianity become official religion of Rome?
Response: 380 AD
Rationale: (Complete the following sentence with details from the evidence; you can only use the information from the evidence)
Completion
Catholic Christianity was declared the official state religion of the Roman Empire on February 27th, 380, through the decree "De fide catolica".

Figure 4: Prompt of Rationale Generation for Gemini-Ultra.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Appendix: Prompt of RAG Drafting

Prompt
Response to the instruction. Also provide rationale for your response.
Instruction: In Buddhism, what is the state of blissful repose or absolute existence by someone relieved of the necessity of rebirth?
Evidence:
1. Buddhism
Nirvana literally means "blowing out, quenching, becoming extinguished". In early Buddhist texts, it is the state of restraint and self-control that leads to the "blowing out" and the ending of the cycles of sufferings associated with rebirths and redeaths. Many later Buddhist texts describe nirvana as identical with "anatta" with complete "emptiness, nothingness". In some texts, the state is described with greater detail, such as passing through the gate of emptiness ("sunyata") realizing that there!
2. Salvation
It includes a variety of disciplines, such as yoga and meditation. Nirvana is the profound peace of mind that is acquired with moksha (liberation). In Buddhism and Jainism, it is the state of being free from suffering. In Hindu philosophy, it is union with the Brahman (Supreme Being). The word literally means "blown out" (as in a candle) and refers, in the Buddhist context, to the blowing out of the fires of desire, aversion, and delusion, and the imperturbable stillness of mind acquired thereafter. In Theravada Buddhism the emphasis is on one's
Completion
Rationale: Nirvana literally means "blowing out, quenching, becoming extinguished". It is described as a state of "restraint and self-control" that leads to the "blowing out" and the ending of the cycles of sufferings associated with rebirths and redeaths.
Response: In Buddhism, the state of blissful repose or absolute existence by someone relieved of the necessity of rebirth is called Nirvana.

Figure 5: Prompt of RAG Drafting.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Prompt of Standard RAG

### Prompt:

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Evidence:
[1] Britain (place name)
Britain, after which "Britain" became the more commonplace name for the island called Great Britain. After the Anglo-Saxon period, "Britain" was used as a historical term only. Geoffrey of Monmouth in his pseudohistorical "Historia Regum Britanniae" ...

[2] Great Britain
The peoples of these islands of "Pretannike" were called the "Priteni" or "Pretani." "Priteni" is the source of the Welsh language term Prydain, "Britain", which has the same source as the Goidelic term Cruthne used to refer to the early Brythonic-speaking inhabitants of Ireland. The latter were later called Picts or Caledonians. ...

[10] Albion
Albion is an alternative name for Great Britain. The oldest attestation of the toponym comes from the Greek language. It is sometimes used poetically and generally to refer to the island, but is less common than 'Britain' today. The name for Scotland in most of the Celtic languages is related to Albion: "Alba" in Scottish Gaelic, "Albain" ...

### Instruction: What was Britain called – before it was Britain?

### Response:


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Prompt of Standard RAG for Instruction-tuned LM

### [INST] Below is an instruction that describes a task. Write a response for it and state your explanation supporting your response.

### Instruction: What was Britain called – before it was Britain?

### Evidence:

[1] Britain (place name)
Britain, after which "Britain" became the more commonplace name for the island called Great Britain. After the Anglo-Saxon period, "Britain" was used as a historical term only. Geoffrey of Monmouth in his pseudohistorical "Historia Regum Britanniae" ...

[2] Great Britain
The peoples of these islands of "Pretannike" were called the "Priteni" or "Pretani." "Priteni" is the source of the Welsh language term Prydain, "Britain", which has the same source as the Goidelic term Cruthne used to refer to the early Brythonic-speaking inhabitants of Ireland. The latter were later called Picts or Caledonians. ...

[10] Albion
Albion is an alternative name for Great Britain. The oldest attestation of the toponym comes from the Greek language. It is sometimes used poetically and generally to refer to the island, but is less common than 'Britain' today. The name for Scotland in most of the Celtic languages is related to Albion: "Alba" in Scottish Gaelic, "Albain" ...

[/INST] The response is:


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Case Study

Figure 8 shows two drafts generated for the same question. We observe that our RAG drafter can well understand the multiple perspectives in the retrieval results and generate high-quality drafts. Our RAG verifier can also help filter out the unreliable drafts as we observe a relatively low scores in the first draft in Figure 8.

================================= Prompt of Drafter =================================
Response to the instruction. Also provide rationale for your response.
## Instruction: Which actress/singer starred as Doralee Rhodes in the 1980 film, "Nine to Five"?
## Evidence:
[1] Diana DeGarmo
Sheila until the show’s closure on June 27, 2010. DeGarmo played the role of Doralee Rhodes in the national tour of “9 to 5”, which was launched in Nashville on September 21, 2010. She ended her run as Doralee after the July 2011 Minneapolis tour stop. DeGarmo appeared on musical theatre composer Scott Alan's third album, ...
[2] 9 to 5 (musical)
Pippa Winslow as Violet Newstead, Louise Olley as Doralee Rhodes and Leo Sene as Franklin Hart Jr, with Samantha Giffard as RoZ, Matthew Chase as Joe and Mark Houston, Rachel Ivy, and Blair Anderson. “9 to 5” will play in the West End at the Savoy Theatre from January 29 to August 31, 2019. The production stars Amber Davies (Judy), ...

============================== Completion ==============================
## Rationale: Diana DeGarmo played the role of Doralee Rhodes in the national tour of "9 to 5", which began in September 2010.
## Response: Diana DeGarmo

============================ Verification ============================
draft score: 0.6594
self-confidence score: 0.3417
self-reflection score: 0.5238

================================= Prompt of Drafter =================================
Response to the instruction. Also provide rationale for your response.
## Instruction: Which actress/singer starred as Doralee Rhodes in the 1980 film, "Nine to Five"?
## Evidence:
[1] Cynthia Rhodes
Her turnaround her first non-dance related role in Michael Crichton’s 1984 science fiction thriller “Runaway” with Tom Selleck, Kirstie Alley and Gene Simmons. Her most notable role was as dance instructor Penny Johnson in the hit 1987 motion picture "Dirty Dancing" with Jennifer Grey and Patrick Swayze. Rhodes’ final motion picture role was the character of Vicki Phillips opposite actor Jameson Parker in the sleeper action-adventure movie ...
[2] Dolly Parton
Parton’s feature film "9 to 5" (1980). The musical ran at the Ahmanson Theatre, Los Angeles in late 2008. It opened on Broadway at the Marquis Theatre in New York City, on April 30, 2009, to mixed reviews. The title track of her 2008 "Backwoods Barbie" album was written for the musical’s character Doralee. Although her score (as well as the musical debut of actress Allison Janney) were praised, ...

============================== Completion ==============================
## Rationale: Dolly Parton starred in the 1980 film "9 to 5" as Doralee Rhodes. Additionally, Parton’s 2008 musical "9 to 5" was based on the film, with Parton reprising her role as Doralee.
## Response: Dolly Parton starred as Doralee Rhodes in the 1980 film, "Nine to Five".

============================ Verification ============================
draft score: 0.71
self-confidence score: 0.4346
self-reflection score: 0.7449

Figure 8: Case study of SPECULATIVE RAG from TriviaQA where Dolly Parton is the correct answer.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Instruction-Tuning Settings

We construct our training dataset for the RAG drafter from diverse instruction-following pairs. We sample instances from Open-Instruct processed data (Wang et al., 2023a) and knowledge-intensive datasets (Petroni et al., 2021; Stelmach et al., 2022; Mihaylov et al., 2018). We augment the instruction-following pairs with retrieved documents and generated rationale. We use the off-the-shelf dense retriever Contriever-MS MARCO (Izacard et al., 2021) to retrieve up to 10 documents for each pair and use Gemini-Ultra (Team et al., 2023) to generate rationale. In total, we acquire a dataset of 40K instances. We use Mistral (v0.1) as our base LM for the RAG drafter. We reproduce the performance of Self-RAG (Asai et al., 2023) and CRAG (Yan et al., 2024) with Mistral (v0.1) for a fair comparison. We implement the training scripts using the Transformers library from Hugging Face (Wolf et al., 2019). We employ DeepSpeed (Rasley et al., 2020) to accelerate the training process. All experiments are conducted on a Linux server equipped with 16 Nvidia A100-SXM4-40GB GPUs.


Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting:Effects of Self-Reflection Statement

We use “Do you think the explanation supports the answers? (Yes or No)” as the self-reflection statement in our main results. In this study, we replace it with other alternatives to see how the self-reflection statement affects the accuracy. The results are reported in Table 4. We observe that the performance does not change a lot given different self-reflection statements, which shows the stable verification capability of the generalist LMs by language modeling objective.

Table 4: Performance analysis of SPECULATIVE RAG with different self-reflection statements R when computing the self-reflection score P_self-reflect = P(Y_es "|Q, α, β), where Q is the query, α, β are the generated answer draft and rationale.

| Reflection Statement | TriviaQA | PubHealth |
|--------------------|--------|----------|
| Do you think the explanation supports the answers? (Yes or No) | 74.24 | 76.60 |
| Does the rationale support the answer? (Yes or No) | 74.22 | 76.09 |
| What do you think about the rationale? A good one? (Yes or No) | 74.25 | 75.79 |
| Is the rationale good enough to support the answer? (Yes or No) | 74.39 | 76.29 |


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:1 Introduction

Retrieval-Augmented Generation (RAG) systems enhance Large Language Models (LLMs) by incorporating external knowledge bases, enabling more precise and contextually relevant responses. As these systems become integral to a variety of applications, it's imperative to develop robust and comprehensive evaluation frameworks to assess their performance and identify areas for improvement. Evaluating RAG systems, however, presents several challenges: (1) modular complexity: The modular nature of RAG systems, comprising both a retriever and a generator, complicates the design of effective evaluation metrics. It is crucial to establish metrics that can holistically assess the entire system as well as evaluate the individual modules and their interplay, allowing for fully understanding the sources of the errors and misses and how they are generated. (2) metric limitation: Existing metrics for evaluating RAG systems, which are often rule-based or coarse-grained, fall short in providing accurate and interpretable results. Specifically, traditional metrics like recall@k and MRR for retrievers depend on annotated chunks and a rigid chunking approach, missing out on the full semantic scope of the knowledge base. For generators, typical measures such as n-gram-based (e.g., BLEU, ROUGE), embedding-based (e.g., BERTSCORE), and LLM-based methods perform well with concise answers but fail to detect finer distentions in longer responses. To bridge these gaps, it is essential to develop detailed, semantic-based evaluation metrics that effectively capture the intricacies and overall quality of both the retrieval and generation components in RAG systems. (3) metric reliability: The reliability


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Introduction to RAGCHECKER

The existing metrics for Retrieval Augmented Generation (RAG) remain under-explored. Effective evaluation metrics must accurately reflect system performance and align with human judgments for real-world scenarios. To overcome these challenges, we introduce RAGCHECKER, an innovative evaluation framework designed for detailed analysis of retrieval and generation processes. RAGCHECKER is based on claim-level entailment checking involving extracting claims from responses and comparing them against other texts. This approach provides fine-grained evaluation instead of response-level assessment. RAGCHECKER processes the user query, retrieved context, response, and ground truth answer, producing metrics:

1. Overall Metrics: Provide a comprehensive view of system performance, assessing the quality of responses.
2. Diagnostic Retriever Metrics: Evaluate the effectiveness of the retriever, identifying strengths and weaknesses in retrieving information from the knowledge base.
3. Diagnostic Generator Metrics: Assess the generator's performance, analyzing how well the generator uses the retrieved context, handles noisy information, and provides accurate responses.

RAGCHECKER offers a more comprehensive assessment of RAG systems than existing evaluation frameworks, focusing on fine-grained evaluation metrics. Annotations and human judgment data ensure RAGCHECKER’s reliability in capturing quality and improvements across RAG systems over various domains. Main contributions include proposing RAGCHECKER, demonstrating its effectiveness over traditional metrics, and evaluating various RAG systems, revealing insights such as the trade-off between retriever improvement and faithfulness.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Evaluation of RAG

Existing evaluation practices for RAG systems can be categorized into two main approaches: evaluating essential capabilities of generators only and assessing end-to-end performance of RAG systems. Within the two components of a RAG system, the retriever has been well studied in recent years, thus a line of recent work focused on evaluating essential generator capabilities. R3B [4] evaluated 4 fundamental abilities required for generators including Noise Robustness, Negative Rejection, Information Integration and Counterfactual Robustness by manual constructed test sets. RECALL [22] introduced manually created counterfactual contexts into QA and text generation datasets to evaluate the counterfactual robustness of LLMs. NoMIRACL [40] evaluated LLMs’ robustness against first-stage retrieval errors of RAG systems with manually judged relevant and non-relevant datasets. Wu et al. [49] quantified the tug-of-war between LLMs’ faithfulness and internal prior by introducing varying levels of perturbations on the provided contexts. FaaF [15] introduced a fine-grained fact verification formulation to improve previous prompting-based approaches in evaluating factuality of generators. However, we argue that above generator-only evaluation approaches with manually constructed datasets cannot serve as a general RAG evaluation framework to reveal the entanglement of between generation results and different retrieval behaviors, as shown in the analysis of Sec. 4.3. Another line of work focused on assessing end-to-end quality scores of RAG systems. TruLens [6] introduced the concept of RAG Triad, which decompose the quality scores into three aspects: context relevance, groundedness and answer relevance, then predicted the score by prompting LLMs or using NLI models. RAGAS [5] and ARES [35] followed the RAG Triad concept and improved the score prediction approaches on different datasets. CRUD-RAG [26] refered to the CRUD (Create, Read, Update and Delete) actions between users and knowledge bases to develop corresponding datasets and evaluation metrics for RAG systems. We compare the above four evaluation frameworks with RAGCHECKER in the meta evaluation of Sec. 4.2. Besides, the following work also provided good insight or high quality datasets for end-to-end RAG evaluation. Liu et al. [21] conducted human evaluation to audit four popular generative search engines in terms of fluency, perceived utility, and verifiability. MEDRAG [50] constructed a medical RAG benchmark from medical QA datasets and evaluated medical RAG systems with QA accuracy. MultiHop-RAG [39] generated multi-hop queries from news articles and evaluated RAG systems with QA accuracy. CDOA [52] proposed a novel approach to generate dynamic QA questions which requires latest information to answer. However, the evaluation metrics used in the work mentioned above rely either on human evaluation or simple textual accuracy, making them incapable of complex RAG scenarios that require long answer evaluation. Therefore, we do not include them in the meta evaluation.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:RAGCHECKER Framework

Formulation: Define a modular RAG system as RAG = {R, G}, where R is the retriever and G is the generator. Given a query q and documents D, it first retrieves top-k relevant context {chunk_k} = R(q, D, k), and then generates a model response m = G({chunk_k}, q). For simplicity, we can also represent the overall RAG generation process as m = RAG(q, D). Design Principle: Given the compositional nature of RAG, we observe there are two major personae using a RAG evaluation framework. The first persona is a user that cares about the overall performance of RAGs and might choose a system with the best performance. Such a persona prefers a single value metric to compare and rank among RAG systems against a benchmark. The second persona is a developer that focuses on improving a RAG system with the need to identify causes of mistakes and potential rooms for improvements. Causes of errors in response can be classified into 1) retrieval errors, where the retriever fails to return complete and relevant context, and 2) generator errors, where the generator struggles to identify and leverage relevant information from context. Consequently, metrics that reveal error causes should be different from those for overall performance, in the sense that error causes are module-specific or even reflected only by a certain behavior of a module. To help both personae to assess RAG performance, we design RAGCHECKER, a evaluation framework of RAG systems that consists of a benchmark with rich annotations and a set of diversely-purposed fine-grained metrics.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Retriever Metrics

Ideally, a perfect retriever returns precisely all claims needed to generate the ground-truth answer. Completeness-wise, we measure how many claims in the ground-truth answer are covered by retrieved chunks. With retrieved chunks as the reference text, we calculate claim recall as the proportion of {c^{(gt)}_j | c^{(gt)}_j ∈ {chunk_i}}. Differently, we define the retriever precision at chunk-level instead of claim-level. A retrieved chunk is called relevant chunk (r-chunk), if any ground-truth claim is entailed in it. In other words, chunk_i is a relevant chunk if ∃i, s.t. c^{(gt)}_i ∈ chunk_i. The rest retrieved chunks are called irrelevant chunk (irr-chunk). The retriever's context precision is defined as |{r-chunk_j}|/k, where k is the number of all retrieved chunks. Note that a chunk-level precision provides better interpretability than a claim-level one, because in practice RAG systems usually work with documents processed to be text chunks in a fixed size. That being said, it is likely that a chunk may contain relevant claims and irrelevant or misleading information at the same time. As a result, the best possible retriever can only achieve a claim-level precision score lower than 100%, and such an upper-bound varies depending on the actual text distribution in D and chunking strategy.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Generator Metrics

Given k retrieved chunks (possibly mixing relevant and irrelevant information), a perfect generator would identify and include all ground-truth-relevant claims and ignore any that are not. Because the generator’s results have dependency on retrieved chunks, we provide in total six metrics characterizing different aspects of its performance. Given a model response m and its claims {c^{(m)}_i}, we first compute the proportion of c^{(m)}_i that are entailed in retrieved chunks. This metric is faithfulness, as it describes how faithful the generator is to the provided context, thus the higher the better. Next, we examine three types of incorrect response claims, i.e. {c^{(m)}_i |c^{(m)}_i ∉ gt}. 1. The first type includes incorrect claims that are entailed in a relevant chunk, then it indicates the generator is sensitive to noise coupled with useful information. The proportion of this type of claims to all {c^{(m)}_i} is relevant noise sensitivity. 2. The second type includes incorrect claims that are entailed in an irrelevant chunk, then it indicates the generator is also sensitive to noise even in an irrelevant context. The proportion of these incorrect claims is irrelevant noise sensitivity. 3. Finally, the third type includes incorrect claims that are not entailed in any retrieved chunk, meaning all such claims are generated by the generator itself. Its proportion is hallucination. Note that for simplicity we group the two noise sensitivities in Fig. 1, but later in Sec. 4.3 we can see that generators generally have different sensitivity to relevant and irrelevant noise. Finally, we characterize how a generator uses information sources to produce correct claims. A correct claim not entailed by any chunk can only be based on generator’s self-knowledge, thus the proportion of these claims reflects how many correct claims are generated on its own. A lower self-knowledge score is better, when the generator is expected to fully depend on retrieved context only in a RAG system. On the other hand, we also check how much retrieved relevant information is


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:4 Experiments

4.1 Experimental Setup

Baseline RAG Systems: We apply RAGCHECKER to 8 customized RAG systems to demonstrate how these metrics reflect the properties and differences among them, and how they guide the refinement of these systems. The 8 RAG systems are combinations with 2 retrievers and 4 generators. For retrievers, we choose BM25, a representative classic sparse retrieval framework, and E5-Mistral [48], the SOTA open-source dense retriever. Our four generators are GPT-4-29, Mistral-8x7B [44], Llama3-8B, and Llama3-70B [b], covering open-source and proprietary LLMs in various sizes. Further details are deferred to Appendix D. We employ Llama3-70B as both the claim extractor and checker implemented by an open-sourced framework RefChecker. As a validation of its performance on the RefChecker’s hallucination detection benchmark, this setup outperforms the best purely open-sourced combinations reported in RefChecker’s paper.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:4.2 Meta Evaluation

We first conduct the meta evaluation to verify the soundness of RAGCHECKER and compare with existing baseline RAG evaluation frameworks.

Baseline RAG Evaluation Frameworks: We include a total of 10 metrics from Trulens [6], RAG-GAS [5], ARES [35] and CRUD-RAG [25] in the meta evaluation, as they are capable to evaluate end-to-end performance with long answers. Metrics selected for comparison along with their descriptions are summarized in Tab. 4 of Appendix C. To ensure a fair comparison, we use Llama3-70B-INT8 as the LLM backbone when applicable. Since models in the Llama3 family don’t provide an embedding model, baseline retrievers requiring embedding capability still use their corresponding default LLM backbones. In addition to the 10 metrics detailed in the table, we also incorporate BLEU [31], ROUGE-L [20], and BERTScore [56] to assess the correlation between the generated responses and the ground truth answers.


Meta Evaluation Dataset: All baseline metrics are designed with different aspects and functionalities to a certain degree, thus making an exact comparison over metric scores inapplicable. However, we argue that a good metric should reflect the relative human preference over different RAG systems. In this spirit, we construct the meta evaluation dataset with sampled instances from the generated responses of 8 baseline RAG systems introduced in Sec. 4.1 on our benchmark. Each meta evaluation instance is a pair of responses from two baseline RAG systems given the same query. By considering all combinations over 10 domains and 28 baseline pairs, we end up with 280 instances for pairwise human preference labeling. For each instance, annotators compare a pair of responses based on correctness, completeness, and overall assessment. For each aspect, annotators measure their preferences as one of five relative choices, including significantly better, slightly better, tie, slightly worse and significantly worse. For quality control, each instance is annotated by two annotators, and their overall agreement and correlation are measured. To conclude, we build a meta evaluation dataset with 280 instances, each instance is labeled by two annotators with their preference in terms of correctness, completeness and overall assessment.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Statistics of the RAG Benchmark

This benchmark is repurposed from public datasets across 10 domains, containing 4,162 questions. For the domains of Finance, Lifestyle, Recreation, Technology, Science and Novel, the short answers are extended to long-form answers with GPT-4.

Table 1: Statistics of the RAG benchmark.
- **Dataset:**
  - **Domain:** # Query: # Doc.
  - **Source:** Example Query

  1. **ClapNQ:**
     - Domain: Wikipedia
     - # Query: 300
     - # Doc: 4,293
     - Source: ClapNQ
     - Example Query: Difference between russian blue and british blue cat

  2. **NovelQA:**
     - Domain: Novel
     - # Query: 280
     - # Doc: 19
     - Source: NovelQA
     - Example Query: When do the Ewell kids go to school?

 - Other datasets include:
   - RobustQA with various domains like Writing, Biomedical, Finance, Lifestyle, Recreation, Science, Technology.
   - Examples of queries under each domain like "What is the difference between online and internet?", "What hand deformities do patients with Apert syndrome present with?", etc.

 - KIWI dataset in AI Science domain with example query focusing on tasks generated by LLMs.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Correlation Results with Human Evaluation

Table 2: Correlation results with Human Evaluation of Correctness, Completeness, and Overall Assessment. We only show the metric with the best correlation for each baseline framework.

- **Baseline Metrics:** BLEU, ROUGE, BERTScore, TULENS, ARES, RAGAS, CRUD-Neg, RAGChecker.

- **Correctness & Completeness Pearson/Spearman Correlations:**
  - BLEU: Pearson 38.89 | Spearman 35.32  | Completeness Pearson 32.13 | Spearman 21.85
  - ROUGE: Pearson 31.75 | Spearman 31.72
  - BERTScore: Pearson 30.34 | Spearman 27.05
  - Various other metrics like RAGAS and their correlation scores are provided.

- From the table, we can observe that RAGChecker has the strongest correlation with human preference in terms of three aspects. Among other baseline metrics, Answer Similarity of RAGAS, which is based on the stronger backbone model text-embedding-ada-002, shows the best performance.

- A detailed comparison between RAGChecker and this strongest baseline is provided in Fig. 4.