RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Main Results

We present the averaged evaluation results for 8 RAG systems across 10 diverse domain datasets in Table 3. Additional results for all datasets are provided in Appendix E. The RAG system that exhibited the best performance in our experiments is E5-Mistral_GPT-4, owing to the strong retrieval capability of E5-Mistral coupled with the adept comprehension abilities of GPT-4. Next, we provide a list of insights induced from Table 3, along with their interpretation and possible directions for improvements.

Table 3: The averaged evaluation results for different RAG systems across 10 datasets. The overall performance of the RAG system is quantified using precision (Prec.), recall (Rec.), and F1 scores. The retriever component is evaluated based on claim recall (CR) and context precision (CP), while the generator component is diagnosed through context utilization (CU), relevant noise sensitivity (NS(UI)), irrelevant noise sensitivity (NS(II)), hallucination (Halluc.), self-knowledge (SK), and faithfulness (Faith.). Additionally, the average number of response claims for each RAG system is provided.

RAG systems:
- BM25_GPT-4: Prec. 61.0, Rec. 49.7, F1 50.3, CR 74.0, CP 52.3, CU 61.4, NS() 26.2, Halluc. 4.1, SK 8.7, Faith. 34.7, #Claim 12
- BM25_Llama3-8b: Prec. 52.1, Rec. 43.9, F1 44.1, CR 74.0, CP 52.3, CU 54.9, NS() 31.3, Halluc. 6.1, SK 9.8, Faith. 18.8, #Claim 11
- BM25_Mistral-8b: Prec. 59.1, Rec. 44.9, F1 49.3, CR 74.0, CP 52.3, CU 54.9, NS() 30.4, Halluc. 5.8, SK 5.1, Faith. 13.7, #Claim 9
- BM25_Mistral-x7b: Prec. 52.5, Rec. 45.3, F1 44.9, CR 74.0, CP 52.3, CU 54.9, NS() 34.3, Halluc. 5.8, SK 6.2, Faith. 12.8, #Claim 9-2.0
- E5-Mistral_GPT-4: Prec. 62.2, Rec. 53.0, F1 57.7, CR 83.5, CP 61.8, CU 66.4, NS() 28.9, Halluc. 3.5, SK 5.7, Faith. 14.8, #Claim 9-9-2.2
- E5-Mistral_Llama3-8b: Prec. 58.0, Rec. 43.5, F1 50.0, CR 83.5, CP 61.8, CU 55.0, NS() 35.5, Halluc. 3.9, SK 5.5, Faith. 6.6, #Claim 8-9-0
- E5-Mistral_Mistral-8b: Prec. 60.6, Rec. 45.0, F1 48.6, CR 83.5, CP 61.8, CU 57.6, NS() 31.7, Halluc. 4.3, SK 3.3, Faith. 8.8, #Claim 9-0.8
- E5-Mistral_Mistral-x7b: Prec. 53.1, Rec. 48.6, F1 45.7, CR 83.5, CP 61.8, CU 55.2, NS() 36.5, Halluc. 5.1, SK 4.0, Faith. 0.8, #Claim 9-5.2

Retriever Matters Consistently: The quality of retrieval is crucial, as evidenced by the notable differences in overall Precision, Recall and F1 scores when comparing BM25 with E5-Mistral with the generator fixed. This improvement is suggestive to the specific choice of generator, suggesting a consistent benefit from employing a better retriever.

Generator Model Size Brings All-Round Improvement: Paired to the same retriever, Llama3-8b consistently achieves better overall performance than Llama3-8b. More concretely, this superiority is supported by a better performance over every generator metric, such as improved context utilization, reduced noise sensitivity, and less hallucination.

Stable and Performant Context Utilization is Key: Among all generator metrics, we observe that context utilization strongly correlates to the overall F1 score, while such correlation is relatively weaker for other generator metrics. Also, generators’ context utilization are relatively stable between the two retrievers, meaning their overall recall can be improved with a better retriever. These observations indicate that the capability to fully utilize retrieved context is key, which is intuitive because the generator in a RAG system is expected to leverage context to surpass its self-knowledge.

Informative Context Improves Faithfulness and Reduces Hallucination: As E5-Mistral achieves better claim recall, we observe generators paired to it achieve better faithfulness, indicating generators are all capable to identify and leverage information in context. Similarly, hallucination and self-knowledge are both reduced as well.

Retriever Recall Trades-off with Generator Noise Sensitivity: Claim recall for a retriever characterizes the coverage of all information necessary to produce ground-truth answer. In practice, however, because of the fixed-size chunking strategy, retrieved relevant chunks may inevitably also carry over noise as part of the context. As retriever claim recall increases, all generators become more sensitive to such noise, which can be explained as their faithfulness to certain context is not discriminative enough. This observation shows that generators’ capability to precisely leverage relevant context is still a challenge.

Relevant Noise Sensitivity is More Challenging: For every baseline RAG system, there’s an apparent gap between its relevant and irrelevant noise sensitivity. In correlation to the last paragraph, it further enhance the point that generators demonstrate a chunk-level faithfulness. It means a relevant chunk improves faithfulness and introduces less sensitivity. Additionally, in the last paragraph, it further enhance the point that generators demonstrate a chunk-level faithfulness. It means a relevant chunk improves faithfulness and introduces less sensitivity. It means a relevant chunk improves faithfulness and introduces less sensitivity.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Diagnosis on RAG Settings for Improvements

Guided by observations in Sec. 4.3, we modify settings commonly tuned in RAG systems that may lead to improvements, diagnose their working mechanisms with RAGCHECKER metrics, and provide suggestions for improvements on certain aspects. We experiment with different numbers of chunks, chunk sizes, chunk overlap ratios, and generation prompts. We highlight our main findings and suggestions as below, please refer to Appendix F for detailed analysis and results.

More Context Enhances Faithfulness. Increasing the number (k) and size of chunks improves the recall of more useful information (claim recall 61.5→77.6 with k 5→20, 70.3→77.6 with size 150→300). Consequently, this provides more context for the generators to be more faithful to (faithfulness 88.1→92.2 with k 5→20, 91.2→92.2 with size 150→300), though at the same time they also become more sensitive to additional noise (noise sensitivity 34.0→35.4 with k 5→20, 34.5→35.4 with size 150→300). Improvements in the overall performance (F1 51.7→53.4 with k 5→20, 52.6→55.3 with size 150→300) indicates benefits from more context.

Explicit Requirements in Prompts Affect Generation Preferences. When prompts introduces explicit requirements for better faithfulness, context utilization, and lower noise sensitivity, generators show improvements in faithfulness (92.2→93.6), but struggle with the subtle tension between context utilization (52.9→63.7) and noise sensitivity (35.4→38.1).

Chunk Overlap Does Not Matter a Lot. The chunk overlap ratio is usually set to be non-zero to help generators better utilize surrounding information and identify chunks with coherent logic. However, it minimally affects generation performance, as retrieving more chunks sharing similar useful information (increased context precision 69.3→71.1) does not necessarily increase the total amount of retrieved useful information (comparable claim recall 77.8→78.1).

Suggestions to RAG Builders Improving the retriever is an effective way to enhance overall performance. While a better embedding model leads to improvements in both precision and recall, moderately increasing the number and size of chunks improves recall and thus F1 with minimal efforts in practice. Note that the effect saturates as the total amount of relevant information is fixed, so they need not be too large for a balanced cost-performance. On the other hand, given a limited number of context, larger chunk sizes with fewer chunks are preferred for better context precision. However, when targeting better context utilization or reduced noise sensitivity, opposite adjustments should be made to alleviate the influence of noise. When tuning the generator, the trilemma of context utilization, noise sensitivity, and faithfulness makes it difficult to improve all aspects simultaneously. RAG builders should prioritize certain aspects in the prompt based on their targets, user preferences and the generator's capability.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:References

[1] Al@Meta. Llama 3 model card. 2024.

[2] A. Asai, S. Min, Z. Zhong, and D. Chen. Retrieval-based language models and applications. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts), pages 41–46, 2023.

[3] H. Chase. LangChain. https://github.com/langchain-ai/langchain Oct. 2022.

[4] J. Chen, H. Lin, X. Han, and L. Sun. Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17754–17762, 2024.

[5] S. Es, J. James, L. Espinosa-Anke, and S. Schockaert. Ragas: Automated evaluation of retrieval augmented generation. arXiv preprint arXiv:2309.15217, 2023.

[6] J. Ferrara, Ethan-Tonic, and O. M. Outrix. The RAG Triad. January 2024. https://www.trulens.org/trulens_eval/core_concepts_rag_triad/

[7] Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, and H. Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2023.

[8] Y. Guo, Z. Li, X. Jin, Y. Liu, Y. Zeng, W. Liu, X. Li, P. Yang, L. Bai, J. Guo, et al. Retrieval-augmented code generation for universal information extraction. arXiv preprint arXiv:2311.02692, 2023.

[9] R. Han, P. Qi, Y. Zhang, L. Liu, J. Burger, W. Wang, Z. Huang, B. Xiang, and D. Roth. Robustqa: Benchmarking the robustness of domain adaptation for open-domain question answering. In ACL Findings 2, 2023.

[10] X. He, F. Yan, Y. Sun, N. V. Chawla, T. Laurent, Y. LeCun, X. Bresson, and B. Hooi. G-retriever: Retrieval-augmented generation for textual graph understanding and question answering. arXiv preprint arXiv:2402.07630, 2024.

[11] X. Hu, D. Ru, L. Qiu, Q. Guo, T. Zhang, Y. Xu, Y. Luo, P. Liu, Y. Zhang, and Z. Zhang. Refchecker: Reference-based fine-grained hallucination checker and benchmark for large language models. 2024.

[12] L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232, 2023.

[13] Y. Huang and J. Huang. A survey on retrieval-augmented text generation for large language models. arXiv preprint arXiv:2404.10981, 2024.

[14] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand, et al. Matrixl of experts. arXiv preprint arXiv:2401.04088, 2024.

[15] V. Katranidis and G. Barany. Faaf: Facts as a function for the evaluation of rag systems. arXiv preprint arXiv:2403.03888, 2024.

[16] M. Komeili, K. Shouster, and J. Weston. Internet-augmented dialogue generation. arXiv preprint arXiv:2107.07566, 2021.

[17] T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M.-W. Chang, A. M. Dai, U. Szorekti, Q. Le, and S. Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466, 2019.

[18] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karupkhin, N. Goyal, H. Kütümler, M. Lewis, W.-t. Yih, T. Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474, 2020.

[19] C.-Y. Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics.

[20] C.-Y. Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004.


[21] N. F. Liu, T. Zhang, and P. Liang. Evaluating verifiability in generative search engines. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7001–7025, 2023.

[22] Y. Liu, L. Huang, S. Li, S. Chen, H. Zhou, F. Meng, J. Zhou, and X. Sun. Recall: A benchmark for films robustness against external counterfactual knowledge. arXiv preprint arXiv:2311.08147, 2023.

[23] K. Lo, L. L. Wang, M. Neumann, R. Kinney, and D. Weld. S2ORC: The semantic scholar open research corpus. In D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969–4983, Online, July 2020. Association for Computational Linguistics.

[24] A. Louis, G. van Dijck, and G. Spanakis. Interpretable long-form legal question answering with retrieval-augmented large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 22266–22275, 2024.

[25] Y. Lyu, Z. Li, S. Niu, F. Xiong, B. Tang, W. Wang, H. Wu, H. Liu, T. Xu, and E. Chen. Crud-raj: a comprehensive chinese benchmark for retrieval-augmented generation of large language models. arXiv preprint arXiv:2401.17043, 2024.

[26] M. Maia, S. Handschuh, A. Freitas, B. Davis, R. McDermott, M. Zarrouk, and A. Balahur. Www’18 open challenge: financial opinion mining and question answering. In Companion proceedings of the web conference 2018, pages 1941–1942, 2018.

[27] Y. Mao, P. He, X. Liu, Y. Shen, J. Gao, J. Han, and W. Chen. Generation-augmented retrieval for open-domain question answering. arXiv preprint arXiv:2009.08553, 2020.

[28] A. Neelakantan, T. Xu, R. Puri, A. Radford, J. M. Han, J. Tworek, Q. Yuan, N. Tzezak, J. W. Kim, C. Hallacy, et al. Text and code embeddings by contrastive pre-training. arXiv preprint arXiv:2201.10059, 2022.

[29] OpenAI, J. A. Smith, A. S. Adler, S. Agarwal, L. Ahmed, I. Akkaya, F. E. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anakad, A. Ralla, J. Babuschkin, S. Balaji, V. Balcom, P. Ballé, P. Bastos, H. Bae, M. Bavarian, J. Beljou, J. Belli, J. Belz, J. Bilderde, G. Bernatchez, C. Berner, L. Bogdonav, G. Boiko, M. Boyd, A.L. Brakman, G. Brockman, T. Brooks, M. Brunbrag, K. Button, T. Cai, R. Campbell, A. Cann, B. Carey, C. Carlson, R. Carmichael, D. Chang, C. Chang, F. Chantzis, D. Chen, S. Chen, R. Chen, J. Chen, M. Cheng, B. Chess, C. Cho, C. Chu, A. Chung, D. Cummings, J. Currier, D. Dai, C. Deceaureaux, T. Degry, N. Deutsch, D. Deville, A. Dhar, D. Dohan, S. Dowling, S. Dunnion, A. Edel, A. Eletri, T. Eloundou, D. Farhi, E. Feads, N. Felix, S. Fishman, J. Forte, J. Fulford, L. Gafos, E. Georcques, B. Gibson, Y. Goei, T. Guergin, N. Goh, R. Gonz^{a}lopes, J. Gordon, M. Gratçoups, S. Gray, C. Green, J. Gross, S. Gu, Y. Guo, C. Hallacy, J. Hahn, J. Harris, Y. He, M. Heaton, J. Heidecke, C. Hesse, A. Hickey, W. Hickey, P. Hoeschele, B. Hoogin, S. Hsu, X. Hu, S. Hu, X. Hu, J. Hui, Z. Huqiang, S. Jain, S. Jiang, A. Jiang, R. Jiang, H. Jin, D. Jin, S. Jomoto, B. Jonn, H. Jun, T. Kaftan, K. Kausser, A. Kamali, I. Kantsichehr, N. S. Kesar, K. Than, L. Kilpatrick, J. W. Kim, C. Kim, Y. Kim, H. Kirchner, J. Kiros, M. Knight, B. Kokotajlo, Łukasz Kondracki, A. Kohndrick, A. Konstandinidis, K. Kosic, G. Krueger, W. Luo, M. Lampe, I. Lan, T. Lee, J. Leike, J. Leung, D. Levy, C. Li, M. Li, R. Lim, M. Lin, S. Lin, M. Litwin, T. Lopez, T. Lopez, R. Love, P. A. MaKaynu, K. Malaficni, S. Manning, T. Markov, Y. Markovski, B. Martin, K. Mayer, A. Mayne, B. McGrew, S. Mckinney, C. Mcleavey, P. McMillan, J. McNeil, D. Medina, A. Mehta, J. Menick, L. Metz, A. Mishchenko, P. Mishin, V. Monaco, E. Morikawa, D. Mossing, T. Mu, U. Murai, O. Murk, D. Mely, A. Nair, R. Nakano, R. Nayak, A. Neelakantan, R. Ngo, H. Noh, L. Ouyang, C. O’Keefe, J. Pachocki, A. Paino, J. Palermo, A. Pantuliano, G. Parascanaldo, J. Parish, E. Paparrita, A. Passos, M. Pavol, A. Peng, A. Perlmane, F. Advila Beullepasse, M. Perth, P. H. D.e Oliveira Pinto, Michael, Poonay, M. Pokrasvs, V. Pong, F. Powella, A. Power, B. Power, E. Proehl, R. Puri, A. Radford, J. Rae, A. Ramesh, C. Raymond, F. Rael, K. Rimbach, C. Ross, B. Rostedt, H. Roussez, N. Ryder, M. Saltarelli, T. Sanders, S. Santurkar, G. Sastry, C. Schmid, D. Schnaar, J. Schulman, D. Selsam, R. Shagenoff, T. Sherbakov, J. Shakey, S. Shoer, P. Shyam, S. Sidor, E. Sigler, M. Simens, J. Sitkin, K. Slama, J. Solt, B. Solokovsky, Y. Song, N. Stauffer, F. P. Such, N. Summers, I. Sutkesev, J. Tang, N. Tezak, M. Thompson, P. Titlel, A. Tootoonchian, E. Tseng, P. Tuggle, N. Turley, J. Tworek, J. F. C. Uribe, A. Vallone, A. Vijayvergiya, C. Voss, C. Wainwright, J. J. Wang, A. Wang, B. Wang, F. Wei, J. Wei, C. Weinman, A. Weihinda, P. Weilinder, J. Weng, L. Weng, M. Whiteford, D. Willner, C. Winter,


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Research Paper References

[28] S. Wolrich, H. Wong, L. Workman, S. Wu, J. Wu, W. Mu, K. Xiao, T. Xu, S. Yoo, K. Yu, Q. Yuan, W. Zaremba, R. Zellers, C. Zhang, M. Zhang, S. Zhao, T. Zheng, J. Zhuang, W. Zhuk, and B. Zoph. Gpt-4 technical report, 2023.

[30] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In P. Isabelle, E. Charniak, and D. Lin, editors, Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics.

[31] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.

[32] M. R. Parvez, W. U. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang. Retrieval augmented code generation and summarization. arXiv preprint arXiv:2108.11601, 2021.

[33] S. Robertson, H. Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389, 2010.

[34] S. Rosenthal, A. Sil, R. Florian, and S. Roukos. Clapp: Cohesive long-form answers from passages in natural questions for rag systems, 2024.

[35] J. Saad-Falcon, O. Khattab, C. Potts, and M. Zaharia. Ares: An automated evaluation framework for retrieval-augmented generation systems, 2023.

[36] K. Santhanam, O. Khattab, J. Saad-Falcon, C. Potts, and M. Zaharia. Colbertv2: Effective and efficient retrieval via lightweight late interaction. In M. Carpuat, M.-C. de Marneffe, and I. V. Meza Ruiz, editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3715–3734, Seattle, United States, July 2022. Association for Computational Linguistics.

[37] L. Shuster, S. Poff, M. Chen, D. Kelka, and J. Weston. Retrieval augmentation reduces hallucination in conversation. arXiv preprint arXiv:2104.07567, 2021.

[38] H. Tan, Q. Luo, L. Jiang, Z. Zhan, J. Li, H. Zhang, and Y. Zhang. Prompt-based code completion via multi-retrieval augmented generation. arXiv preprint arXiv:2405.07530, 2024.

[39] Y. Tang and Y. Yang. Multihop-rag: Benchmarking retrieval-augmented generation for multihop queries. arXiv preprint arXiv:2401.15391, 2024.

[40] N. Thakur, L. Bonifacio, X. Zhang, O. Ogundepo, E. Kamalloo, D. Alfonso-Hermelo, X. Li, Q. Liu, B. Chen, M. Rezaeigholzadeh, et al. Nomicrai: Knowing when you don’t know for robust multilingual retrieval-augmented generation. arXiv preprint arXiv:2312.11361, 2023.

[41] D. Thulke, N. Daheim, C. Dugsat, and H. Ney. Efficient retrieval and augmented generation from unstructured knowledge for task-oriented dialog. arXiv preprint arXiv:2102.04463, 2021.

[42] S. Tomroy, S. Zanlian, V. Jain, A. Rani, V. Rawte, A. Chadha, and A. Das. A comprehensive survey of hallucination mitigation techniques in large language models. arXiv preprint arXiv:2301.01313, 2024.

[43] G. Tsatsaronis, G. Balikas, P. Malakasiotis, I. Partalas, M. Zschunke, M. R. Alvers, D. Weissenborn, A. Krithara, S. Petridis, D. Polychronopoulos, et al. An overview of the bioasq large-scale biomedical semantic indexing and question answering competition. BMC bioinformatics, 16:1–28, 2015.

[44] E. M. Voorhees et al. The trec-8 question answering track report. In Trec, volume 99, pages 77–82, 1999.

[45] C. Wang, S. Cheng, Q. Guo, Y. Yue, B. Ding, Z. Xu, Y. Wang, X. Hu, Z. Zhang, and Y. Zhang. Evaluating open-qa evaluation. Advances in Neural Information Processing Systems, 36, 2024.

[46] C. Wang, X. Liu, Y. Yue, X. Tang, T. Zhang, C. Jiayang, Y. Yao, W. Gao, X. Hu, Z. Qi, et al. Survey on factuality in large language models: Knowledge, retrieval and domain-specificity. arXiv preprint arXiv:2310.07521, 2023.

[47] C. Wang, R. Ning, B. Pan, T. Wu, Q. Guo, C. Deng, G. Bao, Q. Wang, and Y. Zhang. Novelqa: A benchmark for long-range novel question answering, 2024.

[48] L. Wang, N. Yang, X. Huang, L. Yang, R. Majumder, and F. Wei. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368, 2023.


The references provide a comprehensive list of research work related to the study of retrieval-augmented generation and evaluation of AI models. The first reference, "How faithful are RAG models? quantifying the tug-of-war between RAG and LLMS' internal prior," is archived under arXiv:2404.10198, 2024, by K. Wu, E. Wu, and J. Zou. Another study titled "Benchmarking retrieval-augmented generation for medicine" authored by G. Xiong, Q. Jin, Z. Lu, and A. Zhang can be found under arXiv:2402.13178, 2024. The work by F. Xu, K. Lo, L. Soldaini, B. Kuehl, E. Choi, and D. Wadden, "Kiwi: A dataset of knowledge-intensive writing instructions for answering research questions," promises insights into the generation processes, 2024. The Chinese dynamic question answering benchmark study by X. Xu, Y. Li, R. Ding, X. Wang, B. Chen, Y. Jiang, X. Deng, J. Ma, H.-T. Zheng, W. Lu, under arXiv:2402.19285, 2024, is highlighted for its relevance. The evaluation survey by H. Yu, A. Gan, R. Zhang, S. Tong, Q. Liu, and Z. Liu titled "Evaluation of retrieval-augmented generation" is listed under arXiv:2405.07437, 2024. Research on retrieval-augmented language models for clinical medicine is noted with "Enhancing financial sentiment analysis via retrieval augmented large language models" authored by Z. Zakta, R. Shad, A. Chaurasia, A. R. Dalal, J. L. Kim, M. Moor, R. Fong, C. Phillips, K. Alexander, E. Ashley, and E. Almanac, published in NEJM AI, 1(2):AI02A300068, 2024. Additional significant work on financial sentiment analysis is noted by B. Zhang, H. Yang, T. Zhou, M. Ali Babar, X.-Y. Liu in the "Proceedings of the Fourth ACM International Conference on AI in Finance, pages 349–356, 2023." Another study, "Bertscore: Evaluating text generation with BERT" by T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, Y. Artzi, can be found under arXiv:1904.09675, 2019. Lastly, "Docprompting: Generating code by retrieving the docs" by S. Zhou, U. Alon, F. F. Xu, Z. Wang, Z. Jiang, and G. Neubig is cited under arXiv:2207.05987, 2022.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:A.1 Data Sources

RobustQA: We choose 7 domains from RobustQA’s collection of datasets: Biomedical, Finance, Lifestyle, Recreation, Technology, Science, and Writing. For the Biomedical domain, following RobustQA, we employ the BioASQ [43] dataset, which contains human expert-written question-answer pairs and ground truth documents based on abstracts of articles from PubMed. We use the test sets for Task 8 from 2014 to 2023 and the corpus of v.2022 to construct the benchmark. We keep QA pairs whose answers are relatively long (more than 50 words), obtaining 511 QA pairs for the biomedical domain. The other 6 domains are sourced from FiQA [26] and LoTTE [36], each of their question is annotated with a list of short answers that are spans of ground truth passages. We convert the short answers to long-form answers using GPT-4 and only keep the generated answers with no hallucinations, as checked by RefChecker. Finally, we sample 500 examples for each domain.

ClapNQ: Derived from NaturalQuestions (NQ) [17], an ODQA dataset based on Wikipedia, ClapNQ has long-form answers annotated for a subset of NQ for evaluating RAG. We employ the dev set of ClapNQ in our benchmark and take the annotated long-form answers as the ground truth.

KIWI: is constructed by asking LLMs research questions about a set of NLP papers and guiding the LLMs to reach satisfactory long-form answers. The authors validated the quality of the generated answers by rating them as “good”, “neutral”, or “bad”. We take the answers labeled “good” as the ground truth answers and query the full text of the papers from S2ORC [23] as the corpus. As a result, we obtain 71 QA pairs and 429 papers as the corpus.

NovelQA: is a benchmark for question answering over long novels containing over 100K tokens on average. Originally designed for benchmarking long-context LLMs, we repurpose it for evaluating RAG. In contrast with the other domains, each question in NovelQA is associated with a single novel, so when we use this dataset for RAG, we constrain the retrieval to within the corresponding novel. We select 19 copyright-free novels and convert the corresponding short answers to long-form answers following the same process for RobustQA.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:A.2 Long-form Answer Generation

We employ GPT-4 (gpt-4-turbo-2024-04-09) to convert the human annotated short answers to long-form answers in the dataset of RobustQA and NovelQA. For RobustQA, the short answers are spans of the annotated ground truth passages, we take all the annotated short answers and the corresponding passages in the prompt and ask GPT-4 to convert them to one single long-form answer. For NovelQA, we take the human written evidences as the ground truth passage content and the human written short answers for the long-form answer generation. The prompt is shown in Fig. 2.

For quality control, we ask GPT-4 to generate the passage IDs associated with the long-form answer. We use RefChecker to check whether all the claims of a long-form answer are entailed by these passages, and we only keep the long-form answers that meet this criteria. The RefChecker we used here are described in Appendix G.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Prompt Description for Long-Form Answer Conversion

Here is a question and human annotated short answers, the short answers are extracted from a passage, you should help me to convert the short answers to a long-form answer based on the provided passage. There could be more than one annotation, so you should identify the best answers and merge them. The long-form answer should only depend on the provided information, you should not hallucinate anything.

### Question
{question}
### Annotations
#### Annotation 1
[Passage ID 1]: {passage_id}
[Passage Content1]: {passage_content}
[Short Answers]: {answers}

#### Annotation 2
[Passage ID 2]: {passage_id}
[Passage Content2]: {passage_content}
[Short Answers]: {answers}

...

Your should output the converted long-form answer and the passage IDs you used. Always follow this format for your response:
[Long-form answer]: <the content of the long-form answer>
[Passage IDs]: ID1, ID2, ...

Figure 2: The prompt used for converting short answers to long-form answers for the domains of Novel, Finance, Lifestyle, Recreation, Technology, Science, and Writing.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Corpus Downsampling for Science and Biomedical Domains

In addition to long-form answer generation, we also perform downsampling for the corpora of Science and Biomedical domains as they are much larger than the others, with over 1 million documents each. Building indexes for a dense retriever is very costly for large corpora, so we downsample these domains to lower the evaluation cost for the community. For the biomedical domain, we first use BM25 retriever to obtain top 400 documents for each question. The subsampled corpus is formed by combining all documents from the retriever with annotated relevant documents from the datasets. Based on our initial study, we observe that the BM25 retriever yield competitive performance against the dense retriever, so we decide to only use the BM25 retriever for downsampling purpose to save computation cost. For the science domain, we leverage both the BM25 retriever and es-mistral-7b-instruct based dense retriever to obtain document candidates. Specifically, we retrieve the top 200 documents from both retrievers (400 documents in total before deduplication). Similarly, the combination of all documents from the retrievers and annotated relevant documents forms the downsampled corpus.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Metrics Formulation

In accordance with the definitions provided in Section 3.3, we compute each metric using the following formulations: B.1 Overall Metrics. Precision is formulated as \( \frac{| \mathcal{C}^{(m)}_i \cap \mathcal{C}^{(m)}_i \in g t|}{| \mathcal{C}^{(m)}_i|} \). Recall is defined as \( \frac{| \mathcal{C}^{(g t)}_i \cap \mathcal{C}^{(g t)}_i \in m|}{| \mathcal{C}^{(g t)}_i|} \). B.2 Retriever Metrics. Claim Recall is \( \frac{| \mathcal{C}^{(g t)}_i \mathcal{C}^{(g t)}_i \in \{ \text{chunk}_j \} \}|}{| \mathcal{C}^{(g t)}_i|} \). Context Precision is \( \frac{| \mathcal{r} \text{-chunk}_k |}{k} \). B.3 Generator Metrics. Faithfulness is \( \frac{| \mathcal{C}^{(m)}_i \mathcal{C}^{(m)}_i \in \{ \text{chunk}_j \} \}|}{| \mathcal{C}^{(m)}_i|} \). Relevant Noise Sensitivity observed with \( \frac{| \mathcal{C}^{(m)}_i \mathcal{C}^{(m)}_i \not\subseteq g t \not\subseteq m \mathcal{C}^{(m)}_i \in \{ r \text{-chunk}_j \}|}{| \mathcal{C}^{(m)}_i|} \). Irrelevant Noise Sensitivity is noted as \( \frac{| \mathcal{C}^{(m)}_i \mathcal{C}^{(m)}_i \not\subseteq g t \not\subseteq m \mathcal{C}^{(m)}_i \in \{ \text{irr chunk}_j \}|}{| \mathcal{C}^{(m)}_i|} \). Hallucination metrics are captured by \( \frac{| \mathcal{C}^{(m)}_i \mathcal{C}^{(m)}_i \not\subseteq g t \mathcal{C}^{(m)}_i  \{ \text{chunk}_j \}|}{| \mathcal{C}^{(m)}_i|} \). Self-knowledge is noted as \( \frac{| \mathcal{C}^{(m)}_i \mathcal{C}^{(m)}_i \mathcal{C}^{(m)}_i \in g t \mathcal{C}^{(m)}_i \{ \text{chunk}_j \}|}{| \mathcal{C}^{(g t)}_i \mathcal{C}^{(g t)}_i|} \). Context Utilization is \( \frac{| \mathcal{C}^{(g t)}_i \mathcal{C}^{(g t)}_i \in \{ \text{chunk}_j \} \mathcal{C}^{(g t)}_i \{ \text{chunk}_j \} \mathcal{C}^{(g t)}_i \{ \text{chunk}_j \}|}{| \mathcal{C}^{(g t)}_i \mathcal{C}^{(g t)}_i \{ \text{chunk}_j \} \mathcal{C}^{(g t)}_i \{ \text{chunk}_j \}|} \).


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Details of the Experiment Setup

Models in Baseline RAG Systems: The e5-mistral-7b-instruct model for the E5-Mistral retriever is employed. Generators include gpt-4-turbo-2024-04-09, Llama3-8B-Instruct for Llama3-8B, Llama3-70B-Instruct for Llama3-70B, and Mixtral-8x7B-Instruct-v0.1 for Mixtral-8x7B. OpenSearch is utilized for the inverted index for BM25 and approximate KNN search for dense retrieval, using a p5.48xlarge instance with 8 NVIDIA A100 GPUs on AWS for open-source models inference. Corpus is split into 300-token chunks with a 0.2 overlap ratio using E5-Mistral’s tokenizer for retrieval chunking. For queries, top-20 chunks ranked by retrievers are contextually used for LLM generation. Fig. 5 shows the default prompt for generators. Generation temperature is set to 0.0 (deterministic) and generation length to 2,048 tokens for proprietary LLMs.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Table 4: Summary of the Metrics Included in the Meta Evaluation

This table provides a detailed overview of the different baselines, metrics, and their descriptions used in the meta evaluation.

Baseline: TruLens
- Groundedness: Assesses the overlap between each statement in the response and the provided context using an LLM.
- Answer Relevance: Prompts an LLM to give a relevance score between the response and question.
- Faithfulness: Measures the proportion of claims in the response that can be inferred from the context.

Baseline: RAGAS
- Answer Relevance: Computes the mean cosine similarity between the original question and a series of LLM-generated questions derived from the response and context.
- Answer Similarity: Measures the semantic similarity between the response and the ground truth answer based on text-embedding-ada-002.
- Answer Correctness: Quantifies both the semantic similarity and the factual overlap between the response and the ground truth answer.
- Answer Faithfulness: Prompts an LLM to determine whether the response is faithful to the context.

Baseline: ARES
- Answer Relevance: Prompts an LLM to measure whether the response addresses all aspects of the question and provides only correct information from the context.

Baseline: CRUD-RAG
- Recall: Computes the ratio of all questions generated from ground truth answers that can be answered by response.
- Precision: Evaluates if the generated text is accurate and consistent with the ground truth answer.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Table 5: Full Correlation Results with Human Evaluation of Correctness, Completeness, and Overall Assessment

This table presents the full correlation results with human evaluation metrics across correctness, completeness, and overall assessment. Correlation is measured using Pearson and Spearman coefficients.

Correctness:
- BLEU (avg) Pearson: 38.89, Spearman: 35.32
- ROUGE-L Pearson: 31.75, Spearman: 31.72
- BERTScore Pearson: 30.34, Spearman: 27.05
- TruLens Groundedness Pearson: 21.11, Spearman: 18.21
- TruLens Answer Relevance Pearson: 25.01, Spearman: 27.37
- ARES Answer Relevance Pearson: 18.63, Spearman: 16.84
- ARES Answer Faithfulness Pearson: 9.46, Spearman: 7.60
- RAGAS Faithfulness Pearson: 8.22, Spearman: 7.53
- RAGAS Answer Correctness Pearson: 39.11, Spearman: 36.30
- RAGAS Answer Similarity Pearson: 41.07, Spearman: 43.21
- RAGAS Answer Relevance Pearson: 11.59, Spearman: 8.19
- CRUD-RAG Precision Pearson: 20.73, Spearman: 15.67
- CRUD-RAG Recall Pearson: 30.93, Spearman: 27.13
- RAGChecker Same metrics as human Pearson: 49.06, Spearman: 46.95

Completeness:
- BLEU (avg) Pearson: 32.13, Spearman: 21.85
- ROUGE-L Pearson: 47.88, Spearman: 45.67
- BERTScore Pearson: 37.93, Spearman: 40.05
- TruLens Groundedness Pearson: 14.01, Spearman: 6.02
- TruLens Answer Relevance Pearson: 37.24, Spearman: 37.91
- ARES Answer Relevance Pearson: 20.13, Spearman: 18.13
- ARES Answer Faithfulness Pearson: 10.25, Spearman: 8.99
- RAGAS Faithfulness Pearson: 4.90, Spearman: 1.19
- RAGAS Answer Correctness Pearson: 36.42, Spearman: 36.04
- RAGAS Answer Similarity Pearson: 51.36, Spearman: 61.35
- RAGAS Answer Relevance Pearson: 9.39, Spearman: 9.35
- CRUD-RAG Precision Pearson: 25.58, Spearman: 20.33
- CRUD-RAG Recall Pearson: 41.51, Spearman: 43.76
- RAGChecker Same metrics as human Pearson: 60.67, Spearman: 58.11

Overall Assessment:
- BLEU (avg) Pearson: 35.14, Spearman: 29.42
- ROUGE-L Pearson: 43.10, Spearman: 43.21
- BERTScore Pearson: 33.51, Spearman: 35.57
- TruLens Groundedness Pearson: 19.45, Spearman: 14.42
- TruLens Answer Relevance Pearson: 35.15, Spearman: 33.59
- ARES Answer Relevance Pearson: 17.81, Spearman: 16.26
- ARES Answer Faithfulness Pearson: 8.80, Spearman: 7.58
- RAGAS Faithfulness Pearson: 7.83, Spearman: 5.55
- RAGAS Answer Correctness Pearson: 38.01, Spearman: 37.14
- RAGAS Answer Similarity Pearson: 48.31, Spearman: 57.23
- RAGAS Answer Relevance Pearson: 15.30, Spearman: 11.83
- CRUD-RAG Precision Pearson: 25.95, Spearman: 19.63
- CRUD-RAG Recall Pearson: 41.25, Spearman: 39.71
- RAGChecker Same metrics as human Pearson: 60.59, Spearman: 60.90
- Human Annotator Sanity Check Pearson: 63.67, Spearman: 59.19.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:F Diagnosis on RAG for Improvements

We modify hyper-parameters commonly tuned in RAG systems to observe performance variance under the metrics defined by RAGChecker. We focus on how RAGChecker explains this variance and provides tuning suggestions for improvements on certain aspects. In this section, we evaluate three RAG baselines (BM25_GPT-4, E5-Mistral_GPT-4, and E5-Mistral_Llama3-70B) across three domains with increasing difficulty: Writing, Finance, and KIWI. We use our default settings (Appendix D) in main experiments as controls. We experiment with different numbers of chunks selected as context k (∈ {5,10,20}), different chunk size {150,300,600⁷} different chunk overlap ratio {0.0,0.2,0.4}, and different generation prompts.

More Context Enhances Faithfulness: Top-k selection and chunk size both balance the amount of noise and useful information presented to the generator, but in different manners. Corresponding results are demonstrated in Fig. 6 and Fig. 7. Increasing k adds more context that could be less relevant, while increasing chunk size provides more surrounding context of relevant facts. Thus context precision decreases with larger k but increases with larger chunk sizes. Despite this, they both lead to better claim recall in Retrieval.

Generators tend to be more faithful when provided with more context, though this trend is less pronounced for Llama3, which already exhibits high faithfulness. Context utilization generally worsens with more context due to increasing noise, leading to higher relevant noise sensitivity.

Overall, the end-to-end RAG performance is slightly better with more context, primarily due to improved recall. We recommend moderately increasing the two parameters for more faithful outputs, noting that saturation occurs at high values as the amount of useful information is limited. Given a limited context length, a larger chunk size with a smaller k is preferred, especially for easier datasets (Finance, Writing). This is evident when comparing a chunk size of 150 with k=20 against a chunk size of 300 with k=10.

Explicit Requirements in Prompts Affect Generation Preferences: To validate the effect of the generation prompt, we added more detailed requirements to guide the generation for better faithfulness, context utilization, and lower noise sensitivity. The optimized prompt is shown in Fig. 9. As shown in Fig. 8, we observed a general improvement in context utilization. However, as a counterpart to context utilization, noise sensitivity generally worsened. It demonstrates the difficulty of meeting all prompt requirements when there are subtle tension between them. For the two generators, GPT-4 generally shows improvements in metrics related to faithfulness (hallucination, self-knowledge, faithfulness), whereas Llama3 does not exhibit the same behavior. This aligns with our previous observation (Sec. 4.3) that Llama3 already performs well on faithfulness.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:H Limitations

While RAGCHECKER provides a comprehensive evaluation framework for RAG systems, it has a few limitations that should be acknowledged and addressed in future research.

First, the diagnostic metrics for the retriever component are less insightful compared to those for the generator. The retrieval metrics primarily focus on the recall of ground truth claims and precision of retrieved context, but they may not fully capture the nuances and complexities of the retrieval process. Developing more sophisticated metrics that consider factors such as the information density, diversity, and coherence of the retrieved context could provide deeper insights into the retriever’s performance.

Second, the metrics proposed in RAGCHECKER do not differentiate between Neutral and Contradiction checking results from RefChecker when evaluating the generated responses. These two types of results may have different impacts on the final response quality, and treating them equally could lead to an incomplete assessment. Future work should explore ways to incorporate the distinction between neutral and contradiction results into the evaluation metrics, potentially assigning different weights or penalties based on their severity.

Finally, the evaluation benchmark used in this study is curated based on existing text-only datasets and is limited to English queries and corpus. While this allows for a focused evaluation of RAG systems, it may not fully represent the diverse range of tasks and languages that RAG systems can be applied to. Expanding the benchmark to include datasets from different modalities (e.g., images, audio) and languages would provide a more comprehensive assessment of RAG systems’ capabilities and generalization. Additionally, creating benchmark datasets specifically designed for evaluating RAG systems, rather than repurposing existing ones, could help to better capture the unique challenges and requirements of this task.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Evaluation of RAG Systems

**Figure 10: Diagnosis on Chunk Overlap Ratio**

The figure presents radar charts evaluating chunk overlap ratios across different RAG systems (BM25 GPT-4, BM25 Llama-4, etc.) in three domains: Writing, Finance, and KMI. Each chart shows various metrics like Faithfulness, Completeness (CPL, CP, CU), and so on, indicating how these systems perform with different overlap thresholds (60-75, 75-90, and 90-105 words).

**Table 9: Evaluation results for different RAG systems on RobustQA - BioASQ dataset**

This table provides a comprehensive evaluation of different RAG systems on the BioASQ dataset. It includes metrics such as Precision (Prec.), Recall (Rec.), F1 score, Chain of Thought Correct Percentage (CRT CP, CUT, NSU), Hallucinations (Halluc.), Ski (Ski), and Faithfulness (#Clim). Systems like BM25 GPT-4, BM25 Llama3-4b, and ES-Mistral_Mistral-x70h are compared for their overall performance, retriever efficiency, and generator outcomes. Notably, ES-Mistral_Mistral-x70h scored highest in Faithfulness.

**Table 10: Evaluation results for different RAG systems on RobustQA - Finance dataset**

Table 10 closely resembles the structure of Table 9 but applies to the Finance dataset. Similar metrics are evaluated for RAG systems, showing how each performs specifically in financial contexts. Again, ES-Mistral_Mistral-x70h is noteworthy for obtaining the top Faithfulness score. The results provide insights into performance and precision trade-offs for these systems in finance-related tasks.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Evaluation Results

The image presents tables displaying evaluation results for different Retrieval-Augmented Generation (RAG) systems on various datasets.

**Table 11:** Evaluation results on the RobustQA - Lifestyle dataset. Various RAG systems like BM25_GPT-4, BM25_Llama3-8b, and others are evaluated based on metrics such as Precision (Prec.), Recall (Rec.), F1, CRT, CPT, CUT, NSU(H), NSU(NI), Hallucination score (Halluc.), Skill, Faithfulness (Faith.), and number of claims (#Claim).

**Table 12:** Evaluation results on the RobostQA - Recreation dataset. Each RAG system is assessed using the same metrics as mentioned in Table 11. The table shows how systems like BM25_Llama3-8b and ES-Mistral_Llama3-70b perform across these metrics.

**Table 13:** Evaluation results on the RobustQA - Science dataset. The table lists the performance of RAG systems including BM25_GPT-4 and BM25_Llama3-8b with metrics like F1, CRT, CPT, NSU(H), and others.

**Table 14:** Evaluation results on the RobustQA - Technology dataset. It details the efficacy of RAG systems with a focus on metrics like Prec., Rec., and Faithfulness.

**Table 15:** Evaluation results on the KIWI dataset, showing metrics such as Hallucination score, Skill, and #Claim alongside other usual performance metrics for systems like BM25_GPT-4 and BM25_Llama3-8b.

Each table provides insights into how different systems handle retrieval tasks across the datasets, offering a comprehensive comparison of their strengths and weaknesses.


RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation:Table: RefChecker Performance

Table 16: Performance of RefChecker on the RefChecker benchmark using Llama 3 70B Instruct as both the extractor and checker. We compare the results with the best performed purely open-sourced combinations reported in the RefChecker paper.

| Zero Context          | Accuracy | Fact. F1 | Non-Fact. F1 | Pearson | Spearman |
|-----------------------|----------|---------|--------------|---------|----------|
| Mistral-SFT + RepC    | 89.38    | 80.43   | 92.72        | 77.14   | 76.74    |
| Llama3 + Llama3       | 91.89    | 83.06   | 94.67        | 81.77   | 80.83    |

| Noisy Context         |          |         |              |         |          |
| Mistral-SFT + NLI     | 70.82    | 75.12   | 64.72        | 52.21   | 45.61    |
| Llama3 + Llama3       | 71.75    | 76.69   | 64.15        | 57.67   | 50.31    |

| Accurate Context      |          |         |              |         |          |
| Mistral-SFT + AlignScore | 74.12  | 81.6    | 56.38        | 46.34   | 43.22    |
| Llama3 + Llama3       | 78.35    | 84.87   | 61.92        | 59.48   | 52.03    |

This table details the performance of various models on different contexts: Zero Context, Noisy Context, and Accurate Context. Metrics such as Accuracy, Fact. F1, Non-Fact. F1, Pearson, and Spearman are evaluated for each model combination.


Adaptive Retrieval-Augmented Generation for Conversational Systems:Abstract

Despite the success of integrating large language models into the development of conversational systems, many studies have shown the effectiveness of retrieving and augmenting external knowledge for informative responses. Hence, many existing studies commonly assume the always need for Retrieval Augmented Generation (RAG) in a conversational system without explicit control. This raises a research question about such a necessity. In this study, we propose to investigate the need for each turn of system response to be augmented with external knowledge. In particular, by leveraging human judgements on the binary choice of adaptive augmentation, we develop RAGate, a gating model, which models conversation context and relevant inputs to predict if a conversational system requires RAG for improved responses. We conduct extensive experiments on devising and applying RAGate to conversational models and well-rounded analyses of different conversational scenarios. Our experimental results and analysis indicate the effective application of RAGate in RAG-based conversational systems in identifying system responses for appropriate RAG with high-quality responses and a high generation confidence. This study also identifies the correlation between the generation’s confidence level and the relevance of the augmented knowledge.


Adaptive Retrieval-Augmented Generation for Conversational Systems:1 Introduction

Recently, the advancement of Large Language Models (LLMs) has significantly improved conversational systems, enabling the generation of natural and high-quality responses (Ni et al., 2023). Despite these advancements, recent studies have identified several limitations on the simple use of LLMs to address conversational tasks (Onoe et al., 2022; Huang et al., 2021; Ren et al., 2018). These limitations include the lack of up-to-date knowledge (Onoe et al., 2022), the generation of non-factual or hallucinated content (Huang et al., 2021), and restricted domain adaptability (Ren et al., 2018). These issues can hinder the development of conversational agents with satisfactory user experience. To address these identified challenges, a common approach is to retrieve and augment LLMs with external knowledge to enhance the conversational response, making them more accurate, reliable, and adaptable to different domains (Zhao et al., 2020; Lian et al., 2019; Ye et al., 2024). For example, Shuster et al. (2021) demonstrated that using a dense retrieval model (DPR) (Karpukhin et al., 2020) to retrieve relevant knowledge for augmentation can significantly reduce the hallucination rate, according to a corresponding human evaluation. Similarly, Yang et al. (2020) showed that leveraging a graph-structured knowledge base can boost the reasoning ability and domain generalisability of task-oriented conversational agents. These achievements of knowledge-augmented techniques high-


Adaptive Retrieval-Augmented Generation for Conversational Systems:Knowledge Augmentation in Conversational Systems

While implementing retrieval augmentation to a conversational system for improved response, the necessity of knowledge augmentation for system responses is questioned. Effective human-computer conversation requires factual and relevant responses, offering appropriate amounts of information without shifting the conversation to non-relevant topics. It is argued that overusing external knowledge may lead to system responses that fit specific criteria but not overall conversational quality.

An example illustrates the system's response variation with and without augmented knowledge. Without it, the system provides more diverse and natural responses. This suggests misuse of external knowledge may harm user experience.

To tackle this, an adaptive retrieval-augmented generation solution is proposed, motivated by the gate function in long-short term memory models. The RAGate mechanism adapts external knowledge use for a conversational system. Leveraging human labels and recent machine learning models, RAGate manipulates knowledge snippets' use, improving confidence and reducing hallucination (Varshney et al., 2023). This selective augmentation is shown to correlate positively with response quality.

A study conducted on the updated SGD dataset demonstrates RAGate's capability to utilize external knowledge effectively. Results indicate that the thoughtful use of knowledge augments quality conversational responses, aligning with system confidence levels that limit fabricated output risks.

2 Related Work

Knowledge-augmented generation for conversation systems involves two components: the knowledge retriever and the response generator. Studies indicate that either or both components can be improved for better systems (Li et al., 2022).

Knowledge Retrieval: Dense passage retrieval techniques have refined retrievers (Lewis et al., 2020; Karpukhin et al., 2020) harnessing public search services (Li et al., 2022).

Joint Optimisation of Retriever and Generator: Some studies propose a retriever-generator framework (Shi et al., 2023) which boosts task-oriented dialogue systems' performance using dual-feedback mechanisms. Positive performance is achieved by integrating meta-knowledge to guide system responses effectively.


Adaptive Retrieval-Augmented Generation for Conversational Systems:Methodology

This study addresses the challenge of effectively identifying conversation turns that require augmentation of external knowledge. In particular, we aim to develop a gate mechanism that dynamically determines when to search for external knowledge to ensure natural, relevant, and contextually appropriate responses. First, we define the task of user-system conversation. Let \(D = {d_1, d_2, ..., d_{|D|}}\) be a set of user-system dialogues, and each dialogue \(d\) comprises a sequence of interactions between users and systems (i.e., \(d = {u_0, s_0, u_1, s_1, ..., u_T, s_T}\)) with varying lengths. Here, \(u_t\) and \(s_t\) denote the user utterance and system response at the \(t\)-th turn, respectively. The conversational context up to turn \(t\) can be formulated by aggregating the previous user-system interactions, i.e., \(c_t = u_0, s_0, ..., u_t\). With this context information \(c_t\), the conversation system can augment it with a list of retrieved external knowledge, \(e_{t,k}\), where \(k\) represents the ranking cutoff for the retrieved knowledge. Hence, the binary gate mechanism proposed in this study, deciding the knowledge augmentation, can be formulated as \(f(c_t) = {0, 1}\) or \(f(c_t, e_{t,k}) = {0, 1}\) if the external knowledge \(e_{t,k}\) is considered. Then, the follow-up response generation function \(g(\cdot)\) can be formulated as follows:
\[g(\cdot) = \begin{cases} g(c_t, e_{t,k}) & \text{if } f(c_t) \text{ or } f(c_t, e_{t,k}) \\ g(c_t) & \text{otherwise}. \end{cases}\] (1)
Hence, by evaluating and estimating the necessity of augmenting with external knowledge, we dynamically update the conversational response generation accordingly.


Adaptive Retrieval-Augmented Generation for Conversational Systems:Introduction

Improve the utilization of knowledge and, consequently, the quality of the generated responses. Kang et al. (2023) further advance the retriever by proposing SUbgraph Retrieval-augmented GEneration (SURGE), which employed a graph neural network (GNN)-based context-relevant subgraph retriever. SURGE incorporates contrastive learning to optimize the latent representation space, ensuring that generated texts closely resemble the retrieved subgraphs. Despite the richness of existing retrieval-augmented generation techniques for conversational systems, they commonly hypothesize that every conversation turn needs external knowledge. However, the necessity of augmenting every turn of the conversation with external knowledge remains questionable. A relevant thread of work that aims to answer this question is the introduction of the knowledge-seeking turn detection task using the DSTC-9 dataset (Kim et al., 2020), and the follow-up studies, such as (Hong et al., 2023; Jin et al., 2021). However, this task is to identify turns in conversations injected by human workers about knowledge enquiry instead of identifying the system responses that require knowledge augmentation for improvements. This research gap highlights the value and novelty of this study, which investigates the adaptive use of retrieval-augmented generation for advanced conversational systems.


Adaptive Retrieval-Augmented Generation for Conversational Systems:Introduction to RAGate Variants

We explore three RAGate variants using large language models (LLMs) with devised prompts and parameter-efficient fine-tuning (QLoRA). This exploration is motivated by advancements in transformer-structured neural models. Figure 2 illustrates RAGate and its variants. We discuss each variant.

1. **RAGate-Prompt**: Arora et al. (2022) suggests language models adapt to new tasks with natural language prompts. The gate function is \( f(\cdot) = f(y|\Theta, c_t, p) \), where \(\Theta\) denotes the language model with weights, \(p\) the natural prompt. We test zero-shot and in-context learning with Appendix A illustrations.

2. **RAGate-PEFT**: We enhance language models using instruction-tuning with parameter-efficient techniques like QLoRA (Dettmers et al., 2024). Based on LoRA (Hu et al., 2021), it updates frozen weight matrices \( \Delta W \) through low-rank approximation. It's optimized via GPU-CPU transfer.

3. **RAGate-MHA**: Focuses on multi-head attention. It uses a transformer encoder to incorporate context into input. We model attention as interaction between queries, keys, values \( V \) with \( Attention(Q, K, V) = softmax(\frac{QK^T}{\sqrt{d_k}})V \). It includes feed-forward networks \( FFN(x) = max(0, xW_1 + b_1)W_2 + b_2 \). Both modules employ residual connections per He et al. (2016).


Adaptive Retrieval-Augmented Generation for Conversational Systems:Model Training and Evaluation Setups

We evaluate the performance of introducing RA-Gate according to its binary classification performance and the effectiveness of the resulting response generation. Specifically, we use the KE-TOD dataset (Chen et al., 2022), which has fully annotated 5,324 dialogues and 52,063 turns of conversations. In particular, it is associated with 33,761 knowledge snippets to be retrieved and augmented. In addition, KETOD was developed with human labels on turns of conversations (around 12.1% of turns) about the need for augmenting with retrieved knowledge snippets for a natural and informative system response. Hence, we use these human labels as natural ground truths when evaluating RAGate. It is worth indicating that many current knowledge-augmented conversational datasets often ground their conversations on the knowledge snippet, such as Wizard of Wikipedia (Dinan et al., 2018) and CMU_DoG (Zhou et al., 2018), which makes them not a natural fit to be investigated in this study.

Due to the limited computational resource availability, we explore the use of Llama-v2-7B and Llama-v2-13B to implement RAGate-prompt and fine-tune Llama-v2-7B for RAGate-PEFT. We implement QLoRA using the PEFT library (Mangrulkar et al., 2022) and set the lower rank to 16. As discussed in Section 3, we have various input features to be combined for performance optimization. We begin with the use of context only, then concatenate the context with the real response (contx-resp), with the synthetic response and recognised entities (contx-syn-resp-ner) and further extend with the use of retrieved knowledge (contx-syn-resp-ner-know) or the source of knowledge (contx-syn-resp-ner-source). Specifically, we retrieve the relevant knowledge by exploring the use of TF-IDF and a learned BERT ranker. We evaluate their performance with the classic Recall@1 and Recall@3 on the test collection.


Adaptive Retrieval-Augmented Generation for Conversational Systems:RAgate Performance with LLM Prompting versus Fine-tuning

By comparing the corresponding performance reported in Table 2, we observe that, on average, fine-tuning a Llama-2-7B with QLoRA (i.e., RAGate-PEFT) can significantly outperform RAGate-Prompt. For example, by looking at the RAG-PEFT with context-only input, without using extra input features and instruction updates, it can outperform all RAG-Prompt approaches by a big margin (e.g., 0.4082 versus the highest 0.1230 F1 scores). This reflects the difficulty of this adaptive knowledge augmentation task, which can not be properly addressed by prompting a general pretrained language model. In particular, the use of larger language models and the in-context learning setup, which often result in improved performance (Arora et al., 2022), can not guarantee the enhancement of models’ classification accuracy regarding this classification task. Regarding the performance of RAGate-PEFT approaches, by first examining the effect of using synthetic response and recognised named entities, we observe significantly improved precision (0.5203 to 0.6818) but with the cost of lower recall (0.3359 to 0.2321). In addition, when we add the retrieved knowledge to the input features for prediction, we observe a significant performance drop across all evaluated aspects. This can be caused by the additional complexity introduced by the included retrieved knowledge snippets. Furthermore, we also explored the performance impact of naming the source of the knowledge snippet. We use wikiHow 1 in this study, which can provide rich task instructions for offering informative task-oriented system response (Sen et al., 2023). However, the fine-tuned model cannot reasonably connect the promised rich resource from the knowledge source and the prediction of augmentation necessity. RAGate Performance between fine-tuned LLM and MHA classifier. Next, by comparing the experimental results of RAGate-MHA and RAGate-PEFT in Table 2, we observe a wide-margin recall improvement using RAGate-MHA, reaching a minimum recall of 0.52, but with significantly lower precision accuracy. In Table 2, we also include the use of both the context and the initial system responses (i.e., MHA([contx, resp]) for additional insights. We can observe that a higher precision can be achieved but the use of response does not improve the recall performance. These results are consistent with the observed performance of RAGate-PEFT, which further encourages the use of a synthetic response due to the unavailability of a system response in a practical scenario. In addition, we also observe a similar performance drop when including the retrieved knowledge snippets for classification. Even though the RAGate-MHA model, using the interaction between context and retrieved knowledge snippets, can achieve the highest recall of 0.5835, it can not outperform the RAGate-MHA using context-only on other metrics. Hence, considering the similar F1 and AUC performance levels of RAGate-PEFT and RAGate-MHA leads to a trade-off balance between precision and recall for the two groups of approaches. To further evaluate the classification effectiveness of RAGate, in Appendix B, we provide a detailed discussion of a conducted user study that explores whether RAGate can also assess the potential contribution of retrieved snippets when predicting the decision for retrieval augmentation.


Adaptive Retrieval-Augmented Generation for Conversational Systems:Section 5

At first, without adaptive knowledge augmentation, we compare the choice of response generation without augmentation and with "always" augmentation (i.e., No-Aug versus Aug-All). In Table 3, we observe that by augmenting a total of 4,964 system responses in the test collection, the conversational model can generate more informative and effective responses according to the reported scores of BLEU, ROUGE and BERTscore. This aligns with the reported effectiveness of RAG in many existing studies. However, we also identify a significant drop in the model’s generation confidence level. As denoted by Varshney et al. (2023), a lower confidence level can correlate with a higher chance of generating hallucinated responses, which could be caused by the unnecessary use of external knowledge. Hence, to investigate the effectiveness of adaptive knowledge augmentation, we examine the impact of using RAGate. According to the reported experimental results in Table 3, the adaptive augmented response generation with fewer knowledge snippets can indeed result in a higher confidence level than Aug-All.

Moreover, comparing the performance between RAGate and random selections shows that, considering equal numbers (230 or 787 according to the classification with RAGate) of system responses for augmentation, RAGate can further result in a higher quality of generated response. RAGate-MHA even enables results that are comparable to Aug-All’s response quality, with only 787 turn augmentations instead of all 4964 turns. Specifically, the use of RAGate-PEFT, which identifies 230 turns of system responses for knowledge augmentation, can even outperform the baseline that augments 787 system response turns with improved response quality. Apart from the improved response quality, RAGate also enables the conversational model to maintain a high confidence level and ensure faithful responses. Indeed, using RAGate-MHA, which augments 787 system responses, only lowers the average confidence score by 0.36%, instead of the 1.65% when randomly selecting an equal number of turns to augment.

In addition, considering the use of different quality and amount of knowledge snippets for augmentation, we also include the use of the most relevant knowledge snippet according to BERT-ranker in Table 3. We observe that the use of different amounts of knowledge snippets in different relevance levels has a marginal effect on this learned dialogue system. However, we observe a significant difference in the confidence level. We observe that using only the most relevant knowledge snippet enables the Aug-All to suffer less from a lower confidence level. In particular, the application of RAGate can even increase the confidence level of the conversation system in response generation. This indicates that the confidence score can also correlate with the quality of the augmented knowledge snippets. This observation is further validated using knowledge snippets with fifth-ranking positions by BERT-ranker and the use of TF-IDF ranker. We include the full experimental results in Table 4 and attached in the Appendix. These observations indicate the value of adaptive system response augmentation via RAGate in generating high-quality outputs, ensuring faithful responses, and potentially saving retrieval costs. We also show the value of using confidence scores to reflect the contribution of RAG.


Adaptive Retrieval-Augmented Generation for Conversational Systems:References

Simran Arora, Avanika Narayan, Mayee F Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, and Christopher Re. 2022. Ask me anything: A simple strategy for prompting language models. In The Eleventh International Conference on Learning Representations.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.

Zhiyu Chen, Bing Liu, Seunghwan Moon, Chinnadhurai Sankar, Paul A Crook, and William Yang Wang. 2022. Ketod: Knowledge-enriched task-oriented dialogue. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2581–2593.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2022. Qlora: Efficient finetuning of quantized lms. Advances in Neural Information Processing Systems, 36.

Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wizard of wikipedia: Knowledge-powered conversational agents. In International Conference on Learning Representations.

Alex Graves and Alex Graves. 2012. Long short-term memory. Supervised sequence labelling with recurrent neural networks, pages 37–45.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778.

Taesuk Hong, Junhee Cho, Haeun Yu, Youngjoong Ko, and Jungyun Seo. 2023. Knowledge-grounded dialogue modelling with dialogue-state tracking, domain tracking, and entity extraction. Computer Speech & Language, 78:101460.

Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations.

Xinxian Huang, Huang He, Siqi Bao, Fan Wang, Hua Wu, and Haifeng Wang. 2021. Plato-kag: Unsupervised knowledge-grounded conversation via joint modeling. In Proc. of NLPCoNViAL.

Di Jin, Shuyang Gao, Seokhwan Kim, Yang Liu, and Dilek Hakkani-Tur. 2021. Towards zero and few-shot knowledge-seeking turn detection in task-orientated dialogue systems. In 3rd Workshop on Natural Language Processing for Conversational AI, NLPCoNViAL 2021, pages 281–288.

Minki Kang, Jin Myung Kwak, Jinheon Baek, and Sung Ju Hwang. 2023. Knowledge graph-augmented language models for knowledge-grounded dialogue generation. arXiv preprint arXiv:2305.18846.

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proc. of EMNLP.

Atoosa Kasirzadeh and Jason Gabriel. 2023. In conversation with artificial intelligence: aligning language models with human values. Philosophy & Technology, 36(2):27.

Seokhwan Kim, Mihail Eric, Karthik Gopalakrishnan, Behnam Hedayatnia, Yang Liu, and Dilek Hakkani-Tur. 2020. Beyond domain apis: Task-oriented conversational modeling with unstructured knowledge access. In Proc. of SIGDIAL.

Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2022. Internet-augmented dialogue generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8460–8478.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474.


Adaptive Retrieval-Augmented Generation for Conversational Systems:Paper Information

Yu Li, Baolin Peng, Yelong Shen, Yi Mao, Lars Liden, Zhou Yu, and Jianfeng Gao. 2022. Knowledge-grounded dialogue generation with a unified knowledge representation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 206–218. Rongzhong Lian, Min Xie, Fan Wang, Jinhua Peng, and Hua Wu. 2019. Learning to select knowledge for response generation in dialog systems. In Proc. of IJCAI. Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. Peft: State-of-the-art parameterefficient fine-tune methods. https://github. com/huggingface/peft. Erik Miehling, Manish Nagireddy, Prasanna Sattigeri, Elizabeth M Daly, David Piorkowski, and John T Richards. 2024. Language models in dialogue: Conversational maxims for human-ai interactions. arXiv preprint arXiv:2403.15115. Jinjie Ni, Tom Young, Vlad Pandelea, Fuzhao Xue, and Erik Cambria. 2023. Recent advances in deep learning based dialogue systems: A systematic survey. Artificial intelligence review, 56(4):3055–3155. Yassumaso Odate, Michael Zhang, Eunsol Choi, and Greg Durrett. 2022. Entity cloze by date: What lms know about unseen entities. In Proc. of NAACL. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. Liliang Ren, Kaige Xie, Lu Chen, and Kai Yu. 2018. Towards universal dialogue state tracking. In Proc. of EMNLP.


Adaptive Retrieval-Augmented Generation for Conversational Systems:Related Works and Approaches

Alireza Salemi and Hamed Zamani. 2024. Evaluating retrieval quality in retrieval-augmented generation. Preprint, arXiv:2404.13781. Procheta Sen, Xi Wang, Ruiging Xu, and Emine Yilmaz. 2023. Task2kb: A public task-oriented knowledge base. In Proceedings of the AAAI Conference on Artificial Intelligence. Weizhou Shen, Yingqi Gao, Canbin Huang, Fangqi Wan, Xiaojun Quan, and Wei Bi. 2023. Retrievalgeneration alignment for end-to-end task-oriented dialogue system. arXiv preprint arXiv:2310.08877. Tianyuan Shi, Liangzhi Li, Zijian Lin, Tao Yang, Xiajou Quan, and Qifan Wang. 2023. Dual-feedback knowledge retrieval for task-oriented dialogue systems. arXiv preprint arXiv:2310.14528. Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. In Proc. of EMNLP. Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. 2023. A stitch in time saves nine: Detecting and mitigating hallucinations of lms by validating low-confidence generation. arXiv preprint arXiv:2307.03897. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoerreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30. Hongru Wang, Wenyu Huang, Yang Deng, Rui Wang, Zezhong Wang, Yufei Wang, Fei Mi, Jeff Z Pan, and Kam-Fai Wong. 2024. Uinms-rag: A unified multi-source retrieval-augmented generation for personalized dialogue systems. arXiv preprint arXiv:2401.13256.


Adaptive Retrieval-Augmented Generation for Conversational Systems:Research and Findings

Shiquan Yang, Rui Zhang, and Sarah Erfani. 2020. Graphdialog: Integrating graph knowledge into endto-end task-oriented dialogue systems. In Proc. of EMNLP. Linhao Ye, Zhikai Lei, Jianhao Yin, Qin Chen, Jie Zhou, and Liang He. 2024. Boosting conversational question answering with fine-grained retrievalaugmentation and self-check. arXiv preprint arXiv:2403.18243. Hao Yu, Aoran Gan, Kai Zhang, Shiwei Tong, Qi Liu, and Zhaofeng Liu. 2024. Evaluation of retrievalaugmented generation: A survey. Preprint, arXiv:2405.04737. Xueliang Zhao, Wei Wu, Can Xu, Chongyang Tao, Dongyan Zhao, and Rui Yan. 2020. Knowledge-grounded dialogue generation with pre-trained language models. In Proc. of EMNLP. Kangyan Zhou, Shrimai Prabhumoye, and Alan W Black. 2018. A dataset for document grounded conversations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 708–713.


Adaptive Retrieval-Augmented Generation for Conversational Systems:C Additional experimental results about RAGate for Response Generation

useful information for the query exists in the corpus to address the insufficient context. To investigate by following this direction, we randomly selected 50 samples from instances where our proposed approach (RAGate-MHA, the best-performing gate model) predicted using retrieval augmentation. We asked domain experts (co-authors) to score whether they thought the retrieved snippets in those scenarios could be useful to response generation. Users rated the snippets on a scale of 0 – 4, with scores of 3 or 4 indicating ‘useful’ or ‘highly useful’. We found that in 54% of cases where the prediction was for augmentation, users also found the snippets useful. This indicates that our proposed approach can implicitly capture the potential for obtaining high-quality retrieval snippets.

C Additional experimental results about RAGate for Response Generation

In Table 4, we include the complete experimental results of applying RAGate for adaptive retrieval-augmented system response generation. Specifically, explore the use of retrieved knowledge snippets to different extents of relevance. We include top-3 knowledge snippets retrieved by BERT-ranker and TF-IDF. In addition, we also explore the use of knowledge snippets in different ranking positions (rank 1 and 5) according to the BERT-ranker retriever. The experimental result shows that precisely using a suitable amount of relevant knowledge can generate a response with higher confidence (i.e., less is more). In addition, this observation also indicates the potential use of confidence levels to evaluate the quality of the augmented knowledge.


FACTS About Building Retrieval Augmented Generation-based Chatbots:FACTS About Building Retrieval Augmented Generation-based Chatbots

**ABSTRACT**
Enterprise chatbots, powered by generative AI, are rapidly emerging as the most explored initial applications of this technology in the industry, aimed at enhancing employee productivity. Retrieval Augmented Generation (RAG), Large Language Models (LLMs), Langchain/LlamaIndex types of LLM orchestration frameworks serve as key technological components in building generative-AI based chatbots. However, building successful enterprise chatbots is not easy. They require meticulous engineering of RAG pipelines. This includes fine-tuning semantic embeddings and LLMs, extracting relevant documents from vector databases, rephrasing queries, reranking results, providing effective prompts, honoring document access controls, providing concise responses, including pertinent references, safeguarding personal information, and building the ability to orchestrate all these activities. In this paper, we present a framework for building effective RAG-based chatbots based on our firsthand experience of building three chatbots at NVIDIA: chatbots for HR and IT benefits, company financial earnings, and general enterprise content. Our contributions in this paper are three-fold. First, we introduce our FACTS framework for building enterprise-grade RAG-based chatbots that address the challenges mentioned. FACTS mnemonic refers to the five dimensions that RAG-based chatbots must get right - namely content freshness (F), architectures(A), cost economics of LLMs (C), testing (T), and security (S). Second, we present fifteen control points of RAG pipelines and techniques for optimizing chatbots’ performance at each stage. Finally, we present empirical results from our enterprise data on the accuracy-latency tradeoffs between large LLMs vs small LLMs. To the best of our knowledge, this is the first paper of its kind that provides a holistic view of the factors as well as solutions for building secure enterprise-grade chatbots.

**1 INTRODUCTION**
Chatbots are increasingly becoming an extension of search tools in companies for finding relevant information. Whether it is HR benefits, IT help, sales queries, or engineering issues, enterprise chatbots are now go-to productivity tools. Before the debut of OpenAI’s Chat-GPT [2] in November 2022, companies relied on internally developed chatbots based on dialog flows. Such bots required extensive training for intent understanding and meticulous orchestration for response generation and yet could only provide extractive answers at best. These early bots, built on dialog management systems paired with information retrieval and question answering (IRQA) solutions were fragile and limited in capability. While previous generation language models and GPT models existed, they lacked the accuracy, robustness, and reliability needed for broad enterprise use [5].

Chat-GPT’s release, the emergence of vector databases, and the wide-spread use of retrieval augmented generation (RAGs) [8] marked the beginning of a new era in Chatbot domain. Now, LLMs can understand user intents with simple prompts in natural language, eliminating the need for complex intent variant training, synthesizing enterprise content coherently, thereby empowering chatbots with conversational capability beyond scripted intent recognition. While LLMs bring their generative capabilities to construct coherent, factual, and logical responses to user queries, vector database-powered information retrieval (IR) systems augment LLMs' ability to retrieve fresh content. Tools like LangChain [1] and LlamaIndex [6] facilitate chatbot construction, and orchestration of complex workflows including memory, agents, prompt templates, and overall flow. Together, vector-search based IR systems, LLMs, and LangChain-like frameworks form core components of a RAG pipeline that power growing generative AI chatbots in post Chat-GPT era.

At NVIDIA, our main motivation was to improve our employee productivity by building enterprise chatbots. Our initial enthusiasm quickly met with the reality of addressing numerous challenges. We learned that crafting a successful enterprise chatbot, even in post Chat-GPT era, while promising, is not easy. The process demands meticulous engineering of RAG pipelines, fine-tuning LLMs, and engineering prompts, ensuring relevancy and accuracy of enterprise knowledge, honoring document access control permissions, providing concise responses, including pertinent references, and safeguarding personal information. All of these require careful design, skillful execution, and thorough evaluation demanding many iterations. Additionally, maintaining user engagement while optimizing for speed and cost-efficiency is essential. Through our journey, we learned that getting an enterprise conversational virtual assistant right is akin to achieving a perfect symphony where every note carries significance! In this paper, we share our experiences and strategies in building effective, secure, and cost-efficient chatbots. We answer the following questions from a practitioner perspective:


FACTS About Building Retrieval Augmented Generation-based Chatbots:Ensuring Freshness of Enterprise Data in LLM-Powered Chatbots

**Table 1: A summary of the three chatbots and the current state of development.**
- **Chatbot**: NVInfo, VHelp, Scout Bot
- **Domain**: Enterprise Internal, IT Help, Financial Earnings
- **Data Sources**: SharePoint, GoogleDrive, Slack; Knowledge Articles for ITHelp; Company news, blogs, SEC filings
- **Data Types**: Docs, HTML; PDFs, Slides; HTML, PDFs
- **Access Control**: Yes
- **Sample Queries**: Can I park overnight in HQ parking lot?; How to enroll in Employee Stock Purchase plan?; What are NVIDIA revenues for the past 3 years?
- **State**: Early Access, Testing, Production

- **What are the key challenges to consider when building and deploying enterprise-grade generative AI-based chatbots?** We present our findings from trying to deliver fresh content (F) with flexible architectures (A) that are cost-efficient (C), tested well (T), and secure (S) - (FACTS).
- **How to achieve user acceptable levels of quality with RAG systems in building chatbots?** We present the fifteen control points of RAG pipelines and techniques for optimizing each control point and the overall RAG pipeline.

**2 CASE STUDY**
Our company’s content landscape includes both authoritative knowledge and unauthoritative content. Authoritative content encompasses IT help articles, HR resources in platforms like ServiceNow, and project documentation on Confluence, SharePoint, Google Drive, and engineering tools like NVBugs and GitHub. Employee-generated content complements these sources on platforms such as Slack and MS Teams. In this paper, we present three bots that we have built internally using RAGs and LLMs. These bots are briefly introduced below. All three bots are built on our in-house built generative-AI chatbot platform called NvBot platform. Some of the content that our bots are capable of answering are shown in Table 1.
- **NVInfo Bot** focuses on answering questions about enterprise content (approx. 500M documents of size > 7 TB), complementing intranet search. It manages diverse data formats and enforces document access controls. The tech stack includes LangChain, a vendor vector database for retrieval and to handle document access controls, LLM model (multiple LLM models can be selected), and a custom web-UI.
- **NVHelp Bot** focuses on IT help and HR benefits (approx. 2K multi-modal documents containing text, tables, images, pdfs, and html pages), using a similar tech stack to NVInfo bot with a smaller data volume.
- **Scout Bot** handles questions about financial earnings from public sources, managing structured and unstructured data (approx. 4K multi-modal documents containing text, tables, pdfs, and html pages). The tech stack includes an open-source Vector DB, LangChain, RAGs evaluation, selectable LLM models, and a custom web-UI.
In the remainder of the paper, we present our FACTS framework that summarizes the challenges experienced and the learnings gained in building the aforementioned three chatbots. We first start with the challenge of dealing with delivering fresh enterprise content in each of the chatbots.

**3 ENSURING FRESHNESS OF ENTERPRISE DATA IN LLM-POWERED CHATBOTS (F)**
Ensuring the freshness of enterprise data in LLM-powered chatbots presents several challenges. Foundation models, although powerful, often fall short as they lack domain-specific and enterprise-specific knowledge. Once trained, these models are essentially frozen in time and may hallucinate, providing undesired or inaccurate information when used on enterprise content that they are not trained on.
Retrieval Augmented Generation (RAG) is a process where relevant information is retrieved from vector databases through semantic matching and then fed to LLMs for response generation. In a RAG pipeline, vector databases and LLMs collaboratively ensure the delivery of up-to-date enterprise knowledge. However, RAG pipelines have many control points, each of which when not tuned well can lead to lower accuracy, hallucinations, and irrelevant responses by Chatbots. Additionally, document access control mechanisms complete the necessary retrieval process, requiring careful management to ensure data security and relevance. Furthermore, multi-modal content needs a broad set of control mechanisms to handle structured, unstructured, and semi-structured data, including presentations, diagrams, videos, and meeting recordings. Addressing these challenges is critical for maintaining the accuracy and reliability of enterprise chatbots. Inspired by [3], we identify fifteen control points of RAG from our case studies visualized in Figure 1. Each control point is labeled with a number. In the remainder of this section, we present our insights and learnings for addressing RAG control points.

**3.1 Learnings**
In figure 4, we present a summary description of the fifteen control points of RAG pipelines, challenges associated with each control point, and our suggested approaches for optimizing each control point. Each control point is labeled as RAG-C[num] and RAG-Op[num] for RAG and RAGOps flows, respectively. Below, we present a few key learnings and insights to manage the fresh enterprise content.
- **Metadata Enrichment, Chunking, Query Rephrasal, Query Reranking:** We noticed that metadata enrichment, chunking, query rephrasal and query re-ranking stages of RAG pipeline have the most impact on the quality of Chatbot responses. LLM response generation quality is highly dependent on retrieval relevancy. Retrieval relevancy is, in turn, highly dependent on document metadata enrichment, chunking, and query rephrasal. We implemented grid search-based auto-ML capabilities to find the right configurations of chunk token-sizes, experimented with various prompt variations, and explored different chunk reranking strategies to find optimal ways.


FACTS About Building Retrieval Augmented Generation-based Chatbots:Research Paper Content

Figure 1: Control Points in a typical RAG pipeline when building Chatbots.

Figure 2: Agent architecture for handling complex queries.

The text describes significant improvements in retrieval relevancy and answer quality which have been achieved through various settings. However, it is noted that more work is required to optimize the pipeline fully.

Hybrid Search: It is observed that vector databases are not as effective at handling matching entities such as people, places, and company names. Using a combination of lexical search (e.g., elastic search) and vector search enhances retrieval relevancy. This setup synchronizes both lexical and vector-based searches to boost both search capabilities and speed.

Agentic Architectures: Questions such as analyzing revenue changes offer insights into foundational architectures suitable for handling such queries. An infrastructure that supports hybrid search capabilities, including query decomposition and orchestration, is crucial. The agent architecture aids in addressing complex questions.

Complex queries necessitate the integration of complex agents and multi-agent architectures.

To Fine-tune LLMs or not? The challenge is in deciding whether to fine-tune LLMs to handle domain-specific customizations. Customization can be achieved through several methods including pompt engineering and different tuning strategies like PEFT and FT. Fine-tuning however requires careful experimentation.

Handling multi-modal data: Enterprises need to handle structured, unstructured, and multi-modal data for a successful RAG pipeline. Incorporating chunk titles and segment-level splitting ensures retrieval relevancy. Solutions that aid in parsing and chunking unstructured content are essential for PDF extraction.


FACTS About Building Retrieval Augmented Generation-based Chatbots:RAGOps

Effective health monitoring of RAG pipelines is essential once they are deployed. When answer quality is poor, a thorough error analysis is required to determine whether the issue lies in retrieval relevancy or LLM response generation. To debug retrieval relevancy, developers need detailed information on which chunks were stored in vector databases with their associated metadata, how queries were rephrased, which chunks were retrieved, and how those chunks were ranked. Similarly, if an LLM response is incorrect, it is crucial to review the final prompt used for answer generation. It ensures rich citations, developers must trace back retrieved chunk document links and their corresponding chunks. RAGOps/LLMOps and evaluation frameworks, such as Ragas, are critical for providing the necessary automation to enable rapid iteration during accuracy improvement in RAG pipelines. More details on each control point are presented in Figure 4. In summary, while promising, implementing RAG systems for chatbots demands meticulous planning and continuous evaluation to ensure secure and accurate data retrieval.


FACTS About Building Retrieval Augmented Generation-based Chatbots:Building Flexible Architectures for Generative AI Chatbots (A)

Keeping up with rapid progress in AI is like navigating a fast-moving river. Every aspect, from vector databases and embedding models to LLMs, generic architectures, low-code/no-code platforms, RAG evaluation frameworks, and prompting techniques, is evolving rapidly. Concurrently, departments within companies are exploring generative AI by building their own chatbots and AI copilots. In this dynamic environment, building common, flexible, and adaptive platforms are crucial. At NVIDIA, our chatbot ecosystem has grown significantly, reflecting a trend likely seen in many companies. From building three initial chatbots, we realized the importance of a common platform to avoid duplicated efforts in security, guardrails, authentication, prompts, user interfaces, feedback mechanisms, usage reporting, monitoring, and evaluations. To address this, we developed the NVeBot platform (Figure 7), a modular platform with a pluggable architecture. It allows developers to select LLMs, vector databases, embedding models, agents, and RAG evaluation frameworks that best suit their use case. It also provides common components for essential features like security, guardrails, authentication, authorization, user experience, and monitoring. Additionally, the platform supports citizen development, allowing multiple teams to contribute their tested prompts, workflows, guardrails, and fine-tuned models for collective use.


FACTS About Building Retrieval Augmented Generation-based Chatbots:New Architectural Patterns for Chatbots

As our ecosystem of bots expanded, we faced a critical question: should organizations build many domain-specific bots, a single enterprise bot, or go with a hybrid approach? Domain-specific chatbots excel in tailored environments, while enterprise-wide chatbots act as generalists, providing a centralized knowledge base for all employees. Through our experience, we realized that there is no need to choose one over the other. Novel architectural patterns are emerging where enterprise-wide chatbots act as 'switchboards,' directing inquiries to specialized bots tuned with domain-specific data. This multi-bot architecture allows for the concurrent development of specialized chatbots while providing users with a unified interface. Our NVDb platform supports the coexistence and orchestration of multiple chatbots within an enterprise. The debate over a single bot or multiple specialized bots is ongoing. We envision a landscape where domain-specific bots coexist with a centralized information bot, supported by 'copilots'—generative AI capabilities integrated into workplace environments like programming IDEs and collaboration tools. At NVIDIA, we're actively exploring all three chatbot variations—domain-specific, enterprise-wide, and copilot as generative AI reshapes workplace efficiency and information accessibility.


FACTS About Building Retrieval Augmented Generation-based Chatbots:Cost Economics of Chatbot Deployment

Understanding the cost economics of generative AI-based chatbots involves several critical factors. The high costs of major and commercial LLMs can be unsustainable, with expenses adding up significantly across multiple use cases. Additionally, unseen expenses often accumulate as teams test various LLMs to meet specific needs. Moreover, when using commercial LLM vendor APIs, securing sensitive enterprise data requires guardrails to detect and prevent sensitive data leakage, as well as gateways for audit and legally permitted learning. There are also costs versally disregarded by commercial LLMs, such as large LLMs with long context lengths typically have slower response times, impacting overall efficiency. Bigger Vs. Smaller Models: Larger, commercial LLMs, smaller open-source LLMs are increasingly becoming viable for many use cases, thereby offering cost effective alternatives to companies. As opensource models are catching up with larger, commercial models, they are increasingly offering close-comparable accuracy, as demonstrated in our NVHelp bot empirical evaluation in Figure 3, and generally have better latency performance compared to larger models. Additionally, GPU optimization of inference models can further speed up processing times. Open-source models optimized with NVIDIA’s Tensor RT-LLM inference models, for instance, have shown faster performance than non-optimized models. These strategies help balance the need for cost-efficiency with maintaining high performance and security standards.


FACTS About Building Retrieval Augmented Generation-based Chatbots:7.1 Learnings

Enterprise Content Access Control: Enterprise documents are protected by access controls, requiring RAG-based chatbots to comply with Access Control Lists (ACLs) during response generation. To ensure this compliance, we specifically selected an IR product known for its capability to honor these document ACLs effectively. Derivative Risks with Generative AI: Chatbots might generate responses that lack context from their original data sources, potentially leading to misinterpretations. Additionally, enhanced search methods could inadvertently elevate the risk of exposing sensitive data if enterprise content is inappropriately secured. As part of our NVInfo bot journey, we implemented sensitive data guardians in addition to leveraging existing data filtering and classification capabilities provided by the vector search solution we used to automatically filter out sensitive data during the retrieval. Data Governance and Content Security: Efficient knowledge access can increase sensitive data leakage risks. Thus, it’s essential to prioritize data governance before deployment to safeguard against unauthorized access and data breaches. At NVIDIA, we embarked on an enterprise content sensitivity initiative for document sensitivity classification and exclusion of sensitive content from chatbots. Enterprise AI Responses: Implementing these guidelines during generative AI responses with specific enterprise policies and rules is essential. These guardians help mitigate risks by ensuring that Chatbot-generated content adheres to established norms and ethical guidelines, preventing potential legal and reputational damages. In NVInfo bot, we implemented many guardrails in LLM prompts initially. However, later realized that not all LLMs follow these prompts consistently. Therefore, we implemented these guardrails during pre and post processing of queries and responses respectively using Nemo Guardrails.


FACTS About Building Retrieval Augmented Generation-based Chatbots:8 RELATED WORK

Our work can be compared with RAG papers on various topics dealing with RAG along all the FACTS dimensions we presented (freshness, architecture, costs, testing and security). Due to lack of space, we contrast our work with selective works. Barnett et al. [3] presented several failure points when engineering RAG systems. In their work, they highlight the challenges of getting retrieval augmented generation right by presenting their findings from having built three chatbots. Wenqi Glantz [6] elaborated 12 RAG pain points and presented solutions. We experienced similar challenges first-hand when building our chatbots. However, none of these works discuss the challenges with complex queries, testing, dealing with document security, and the need for flexible architectures. In our work, we not only build on failure/pain points of RAGs as mentioned above, but also present our 15 control points in RAG pipelines and offer specific solutions for each stage. Also, we extend our insights and present practical techniques for handling complex queries, testing, and security. We present a reference architecture for one of the implementations of agentic architectures for complex query handling, strategies for testing and evaluating subjective query responses, and raised awareness for dealing with document ACLs and security. Furthermore, we present a reference architecture for a flexible generative-AI based Chatbot platform.


FACTS About Building Retrieval Augmented Generation-based Chatbots:9 CONCLUSIONS

In this paper, we presented our approach to developing effective RAG-based chatbots, highlighting our experiences of building these chatbots at NVIDIA. We outlined our FACTS framework, emphasizing the importance of content freshness (F), architecture (A), LLM cost (C) management, planning for testing (T), and security (S) in creating robust, secure, and enterprise-grade chatbots. We also identified and elaborated on fifteen critical control points within RAG pipelines, providing strategies to enhance chatbot performance at each stage. Furthermore, our empirical analysis reveals the trade-offs between accuracy and latency when comparing large and small LLMs. This paper offers a comprehensive perspective on the essential factors and practical solutions for building secure and efficient enterprise-grade chatbots, making a unique contribution to the field. More work is needed in several areas to build effective RAG-based chatbots. This includes developing agentic architectures for handling complex, multi-part, and analytical queries; efficiently summarizing large volumes of frequently updated enterprise data; incorporating auto-ML capabilities to optimize various RAG control points automatically; and creating more robust evaluation frameworks for assessing subjective responses and conversations.