Goh, G., Gontijo-Lopes, R., Gordon, J., Grafstein, M., Gray, S., Greene, R., Gross, J., Gu, S.S., Guo, Y., Hallacy, C., Han, J., Harris, J., He, Y., Heaton, H., Heidecke, I., Hesse, C., Hickey, A., Hickey, W., Hoeschle, P., Houghton, B., Hsu, K., Hu, S., Hu, X., Huizinga, J., Jain, S., Jain, S., Jiang, J., Jiang, A., Jiang, R., Jin, H., Jin, D., Jomoto, S., Joon, B., Kim, H., Kaftan, T., Kamali, A., Kantschieder, I., Keskar, N.S., Khan, T., Kilpatrick, L., Kim, J.W., Kim, C., Kim, Y., Kirchner, J.H., Kiros, J., Knight, M., Kokotajlo, D., Kondracki, A., Konstantinidis, A., Kosic, M., Krueger, G., Kuo, V., Lampe, M., Lan, L., Lee, T., Leike, J., Liang, J., Levy, D., Li, C.M., Lim, R., Lin, M., Lin, S., Litwin, M., Lopez, T., Lowe, R., Lue, P., Makanju, A., Malfacini, K., Manning, S., Markov, T., Markovsky, J., Martin, B., Mayer, K., Mayne, A., McGrew, B., McKinney, S.M., McLeavey, C., McMillan, P., McNeil, J., Medina, D., Mehta, A., Mencik, J., Metz, L., Mishchenko, A., Mishkin, P., Monaco, V., Morikawa, E., Mossing, D., Mu, T., Murtha, M., Murk, O., Mély, D., Nair, A., Nakano, R., Nayak, N., Neelakantan, A., Ngo, N., Nghia, H., Nguyen, L., Nogueira, R., Kees, C., Pachocki, J., Paine, A., Palermo, J., Pantuliano, A., Parascandolo, G., Parisi, J., Parratita, E., Passos, A., Pavlo, M., Peng, A., Perelman, A., Peres, F.A.D.B., Petrov, M., Pinto, H.P.O.D., Michael, Pokorny, Pokrass, M., Pong, V.H., Powell, T., Power, A., Power, B., Proelh, E., Prul, J., Radford, A., Rae, J., Ramesh, A., Raymond, C., Real, E., Rimbach, K., Ross, C., Rosted, B., Raschum, R., Ryder, N., Saltarelli, M., Sanders, T., Santurkar, S., Sastry, G., Schmidt, H., Schnurr, D., Schulman, J., Selsam, D., Sheppard, K., Sherbakov, T., SihL, J., Shoker, K., Shyam, P., Sidor, S., Sigler, E., Simens, M., Sitkin, J., Slama, K., Sohl, J., Sokolowsky, B., Song, Y., Staudacher, N., Such, F.P., Summers, N., Sutskever, I., Tang, J., Tezak, N., Thompson, M.B., Tillett, P., Tootoonchian, A., Tenge, E., Tuggle, P., Turley, N., Tworek, J., Uribe, J.F.C., Vallone, A., Vijayvergiya, A., Voss, C., Wainwright, C., Wang, J.J., Wang, A., Wang, B., Ward, J., Wise, J., Weinman, C., Welchman, A., Weidner, P., Weng, L., Wen, Y., Berner, C., Wiener, M., Winter, C., Wollrich, J., Wong, H., Workman, L., Wu, S., Wu, Y., Wu, J., Xiao, K., Xu, T., Yoo, S., Yu, K., Yuan, Q., Zaremba, W., Zellers, R., Zhang, C., Zhang, M., Zhao, S., Zheng, T., Zhuang, J., Zhuk, W., Zoph, B.: GPT-4 Technical Report (Mar 2023). https://doi.org/10.48550/arXiv.2303.08774\n\nOuyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., Lowe, R.: Training language models to follow instructions with human feedback. Tech. Rep. (Mar 2022). https://doi.org/10.48550/arXiv.2203.02155 http://arxiv.org/abs/2203.02155 arXiv:2203.02155 [cs] type: article\n\nPapineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Isabelle, P., Charniak, E., Lin, D. (eds.) Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311–318. Association for Computational Linguistics, Philadelphia, Pennsylvania, USA (Jul 2002). https://doi.org/10.3115/1073083.1073135 https://aclanthology.org/P02-1040 \n\nPetroni, F., Piktus, A., Fan, A., Lewis, P., Yazdani, M., De Cao, N., Thorne, J., Jernite, Y., Karpukhin, V., Maillard, J., Plachouras, V., Rocktaeschel, T., Riedel, S.: KILT: a benchmark for knowledge intensive language tasks. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 2523–2544. Association for Computational Linguistics, Online (Jun 2021). https://doi.org/10.18653/v1/2021.naacl-main.200 https://aclanthology.org/2021.naacl-main.200 \n\nRadford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI blog 1(8), (2019)\n\nRamos, J., et al.: Using tf-idf to determine word relevance in document queries. In: Proceedings of the first instructional conference on machine learning, vol. 242, pp. 29–48. Citeseer.


Evaluation of Retrieval-Augmented Generation: A Survey:References

47. Robertson, S., Zaragoza, H. et al.: The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval 3(4), 333–389 (2009).
48. Rosset, C., Cheng, H.L., Qin, G., Chow, E.C., Feng, Z., Awadallah, A., Neville, J., Rao, N.: Researchy questions: A dataset of multi-perspective, decompositional questions for lm web agents (Feb 2024). https://doi.org/10.48550/ARXIV.2402.17896
49. Saad-Falcon, J., Khatta, O., Potts, C., Zaharia, M.: Ares: An automated evaluation framework for retrieval-augmented generation systems (Nov 2023). https://doi.org/10.48550/ARXIV.2311.09476
50. Sai, A.B., Mohankumar, A.K., Khapra, M.M.: A survey of evaluation metrics used for nlg systems. ACM Computing Surveys (CSUR) 55(2), 1–39 (2022).
51. Shahabi, C., Kolahdouzan, M.R., Sharifzadeh, M.: A road network embedding technique for k-nearest neighbor search in moving object databases. In: Proceedings of the 10th ACM international symposium on advances in geographic information systems. pp. 94–100 (2002).
52. Tang, Y., Yang, Y.: Mulltihop-rag: Benchmarking retrieval-augmented generation for multi-hop queries (Jan 2024). https://doi.org/10.48550/ARXIV.2401.15391
53. Thorne, J., Vlachos, A., Christodoulopoulos, C., Mittal, A.: FEVER: a large-scale dataset for fact extraction and VERification. In: NAACL-HLT (2018).
54. TruLens: TruLens (2023). https://www.trulens.org/trulens_eval/getting_started/quickstarts/quickstart/
55. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need (Jun 2017). https://doi.org/10.48550/ARXIV.1706.03762
56. Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., Bowman, S.R.: SuperGLUE: A stickier benchmark for general-purpose language understanding systems. arXiv preprint 1905.00537 (2019)
57. Wang, S., Kharitonov, E., Zhuang, S., Zuccon, G.: Feb4rag: Evaluating federated search in the context of retrieval augmented generation (Feb 2024). https://doi.org/10.48550/ARXIV.2402.11891
58. Wang, S., Liu, J., Song, S., Cheng, J., Fu, Y., Guo, P., Fang, K., Zhu, Y., Dou, Z.: DomInagram: A chinese benchmark for evaluating domain-specific retrieval-augmented generation (Jun 2024). https://doi.org/10.48550/ARXIV.2406.05564
59. Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E.H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., Fedus, W.: Emergent abilities of large language models (Jun 2022). https://doi.org/10.48550/ARXIV.2206.07682
60. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., Zhou, D.: Chain-of-thought prompting elicits reasoning in large language models (Jan 2022). https://doi.org/10.48550/ARXIV.2201.11903
61. Xiong, G., Jin, Q., Lu, Z., Zhang, A.: Benchmarking retrieval-augmented generation for medicine (Feb 2024). https://doi.org/10.48550/ARXIV.2402.13178
62. Xu, Z., Li, Y., Ding, R., Wang, X., Chen, B., Jiang, Y., Zheng, H.T., Luo, W., Xie, P., Huang, F.: Let lms take on the latest challenges! a chinese dynamic question answering benchmark (Feb 2024). https://doi.org/10.48550/ARXIV.2402.19248
63. Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W.W., Salakhutdinov, R., Manning, C.D.: HotpotQA: A dataset for diverse, explainable multi-hop question answering. In: Conference on Empirical Methods in Natural Language Processing (EMNLP) (2018)
64. Yao, J.Y., Ni, M., Xie, K.P., Liu, Z.H., Ning, M.N., Xu, L.: Lim lms: Hallucinations are not bugs, but features as adversarial examples. arXiv preprint arXiv:2310.01469 (2023)
65. Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T.L., Cao, Y., Narasimhan, K.: Tree of Thoughts: Deliberate problem solving with large language models (2023).


Evaluation of Retrieval-Augmented Generation: A Survey:Evaluation of Retrieval-Augmented Generation: A Survey

66. Yu, X., Cheng, H., Liu, X., Roth, D., Gao, J.: ReEval: Automatic hallucination evaluation for retrieval-augmented large language models via transferable adversarial attacks. In: Duh, K., Gomez, H., Bethard, S. (eds.) Findings of the Association for Computational Linguistics: NAACL 2024, pp. 133-135. Association for Computational Linguistics, Mexico City, Mexico (Jun 2024). https://aclanthology.org/2024.findings-naacl.85
67. Zhang, K., Liu, Q., Qian, H., Xiang, B., Cui, Q., Zhou, J., Chen, E.: EatN: An efficient adaptive fine transfer network for aspect-level sentiment analysis. IEEE Transactions on Knowledge and Data Engineering 35(1), 377–389 (2021)
68. Zhang, K., Zhang, H., Liu, Q., Zhao, H., Zhu, H., Chen, E.: Interactive attention transfer network for cross-domain sentiment classification. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 5773–5780 (2019)
69. Zhang, K., Zhang, K., Zhang, M., Zhao, H., Liu, Q., Wu, W., Chen, E.: Incorporating dynamic semantics into pre-trained language model for aspect-based sentiment analysis. arXiv preprint arXiv:2203.16369 (2022)
70. Zhang, Q., Chen, S., Xu, D., Gao, Q., Chen, X., Cohn, T., Fang, M.: A Survey for Efficient Open Domain Question Answering. In: Rogers, A., Boyd-Graber, J., Okazaki, N. (eds.) Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14447–14465. Association for Computational Linguistics, Toronto, Canada (Jul 2023). https://doi.org/10.18653/v1/2023.acl-long.808 https://aclanthology.org/2023.acl-long.808
71. Zhang, S., Liu, X., Liu, J., Gao, J., Duh, K., Van Durme, B.: ReCord: Bridging the gap between human and machine commonsense reading comprehension (Oct 2018). https://doi.org/10.48550/ARXIV.1810.12885
72. Zhang, T., Kishore, V., Wu, F., Weinberger, K., Artzi, Y.: BERTScore: Evaluating Text Generation with BERT. In: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net (2020). https://openreview.net/forum?id=SkeHuCVFDr
73. Zhang, Y., Khalifa, M., Logeswaran, L., Lee, M., Lee, H., Wang, L.: Merging Generated and Retrieved Knowledge for Open-Domain QA. In: Bouamor, H., Pino, J., Bali, K. (eds.) Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4710–4728. Association for Computational Linguistics, Singapore (Dec 2023). https://doi.org/10.18653/v1/2023.emnlp-main.286 https://aclanthology.org/2023.emnlp-main.286
74. Zhao, H., Zhang, H., Yu, Q., Wang, Z., Geng, Y., Fu, F., Yang, L., Zhang, W., Cui, B.: Retrieval-augmented generation for ai-generated content: A survey (Feb 2024). https://doi.org/10.48550/ARXIV.2402.19473
75. Zheng, L., Chiang, W.L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E.P., Zhang, H., Gonzalez, J.F., Stoica, I.: Judging llm-as-a-judge with nt-bench and chatbot arena (Jun 2023). https://doi.org/10.48550/ARXIV.2306.05685
76. Zhou, Y., Lin, X., Zhang, X., Wang, M., Jiang, G., Lo, A., Hu, W., Yu, X., Zhang, K., Yang, Z., Wang, K., Sui, Y., Jia, F., Tang, Z., Zhao, Y., Zhang, H., Yang, T., Chen, W., Mao, Y., Li, Y., Bao, D., Li, Y., Liao, H., Liu, T., Liu, J., Guo, J., Zhao, X., WEI, X., Qian, H., Liu, Q., Wang, X., Xin, W., Chan, Li, C., Li, Y., Yang, S., Yan, J., Mou, C., Han, S., Jin, W., Zhang, G., Zeng, X.: On the opportunities of green computing: A survey (Nov 2023)
77. Zhu, F., Lei, W., Wang, C., Zheng, J., Poria, S., Chua, T.S.: Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering. Tech. rep. (May 2021). http://arxiv.org/abs/2101.00774 arXiv:2101.00774 [cs] type: article


Evaluation of Retrieval-Augmented Generation: A Survey:Structure of RAG System

A.1 Retrieval Component

The retrieval component of RAG systems in Figure 1 can be categorized into three types: sparse retrieval, dense retrieval [77], and web search engine. The standard for evaluation is the output of relevant documents with numerical scores or rankings.

Before the introduction of neural networks, sparse retrievals are widely used for retrieving relevant text content. Methods like TF-IDF [46] and BM25 [47] rely on keyword matching and word frequency but may miss semantically relevant documents without keyword overlap.

By leveraging deep learning models such as BERT [9], dense retrieval can capture the semantic meaning of texts, which allows them to find relevant documents even when keyword overlap is minimal. This is crucial for complex queries that require a contextual understanding to retrieve accurate information. With advanced fusion structure for queries and documents [28] and the more efficient implementation of K-Nearest Neighbors (KNN) [51], Approximate Nearest Neighbor (ANN) [12] [25] search techniques, dense retrieval methods have become practical for large-scale use.

Web search engine employs the complex online search engine to provide relevant documents, such as Google Search [18], Bing Search [40], DuckDuckGo [13]. RAG systems can traverse the web’s extensive information, potentially returning a more diverse and semantically relevant set of documents via the API of the search provider. The black box of the search engine and the expense of large-scale search are not affordable sometimes.

It is observed that dense retrieval techniques, particularly those leveraging embeddings, stand out as the preferred choice within the RAG ecosystem. These methods are frequently employed in tandem with sparse retrieval strategies, creating a hybrid approach that balances precision and breadth in information retrieval. Moreover, the adoption of sophisticated web search engines for benchmark assessment underscores their growing significance in enhancing the robustness and comprehensiveness of evaluations.

Indexing

The indexing component processes and indexes document collections, such as HuggingFace datasets or Wikipedia pages. Chunking before indexing can improve retrieval by limiting similarity scores to individual chunks, as semantic embedding is less accurate for long articles, and desired content is often brief [32]. Index creation is designed for fast and efficient search. For example, the inverted index for sparse retrieval and the ANN index for dense retrieval.

Sparse Retrieval involves calculating IDF for each term and storing values in a database for quick look-up and scoring when queried.

Dense Retrieval encodes documents into dense vectors using a pre-trained language model like BERT. These vectors are then indexed using an Approximate Nearest Neighbor (ANN) search technique, like graph-based Hierarchical Navigable Small World (HNSW) or Inverted File Index (IVF) [12]. This process allows for the efficient retrieval of “closed” items by given predefined distance metrics.


Evaluation of Retrieval-Augmented Generation: A Survey:A.2 Generation Component

The evaluable output for the generation component is the response of LLMs and the structured or formatted output from the phrased response.

Prompting The generation process critically hinges on prompting, where a query, retrieval outcomes, and instructions converge into a single input for the language model. Research showcases various strategic prompting tactics such as the Chain of Thought (CoT) [60], Tree of Thought (ToT) [3], and Self-Note [31], each significantly shaping the model’s output. These methods, especially the step-by-step approach, are pivotal in augmenting LLMs for intricate tasks.

Prompting innovations have introduced methods like Rephrase and Respond (RaR) [8], enhancing LLMs by refining queries within prompts for better comprehension and response. This technique has proven to boost performance across diverse tasks. The latest RAG benchmarks [61,62] in the specific domains start to evaluate the robustness of various prompting engineering skills, including CoT, RaR, etc.


Graph Retrieval-Augmented Generation: A Survey:Graph Retrieval-Augmented Generation: A Survey

**Abstract**

Recently, Retrieval-Augmented Generation (RAG) has achieved remarkable success in addressing the challenges of Large Language Models (LLMs) without necessitating retraining. By referencing an external knowledge base, RAG refines LLM outputs, effectively mitigating issues such as "hallucination", lack of domain-specific knowledge, and outdated information. However, the complex structure of relationships among different entities in databases presents challenges for RAG systems. In response, GraphRAG leverages structural information across entities to enable more precise and comprehensive retrieval, capturing relational knowledge and facilitating more accurate, context-aware responses. Given the novelty and potential of GraphRAG, a systematic review of current technologies is imperative. This paper provides the first comprehensive overview of GraphRAG methodologies. We formalize the GraphRAG workflow, encompassing Graph-Based Indexing, Graph-Guided Retrieval, and Graph-Enhanced Generation. We then outline the core technologies and training methods at each stage. Additionally, we examine downstream tasks, application domains, evaluation methodologies, and industrial use cases of GraphRAG. Finally, we explore future research directions to inspire further inquiries and advance progress in the field.

**CCS Concepts:**
- Computing methodologies → Knowledge representation and reasoning;
- Information systems → Information retrieval; Data mining.

**Additional Key Words and Phrases:** Large Language Models, Graph Retrieval-Augmented Generation, Knowledge Graphs, Graph Neural Networks

**1 Introduction**

The development of Large Language Models like GPT-4, Qwen2, and LLaMA has sparked a revolution in the field of artificial intelligence, fundamentally altering the landscape of natural language processing. These models, built on Transformer architectures and trained on diverse and extensive datasets, have demonstrated unprecedented capabilities in understanding, interpreting, and generating human language. The impact of these advancements is profound, stretching across various sectors including healthcare, finance, and education, where they facilitate more nuanced and efficient interactions between humans and machines.

**Authors’ Contact Information:**

- Boci Peng, School of Intelligence Science and Technology, Peking University, Beijing, China.
- Yun Zhu, College of Computer Science and Technology, Zhejiang University, Hangzhou, China.
- Yongchao Liu, Ant Group, Hangzhou, China.
- Xiaohai Bo, Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China.
- Haizhou Shi, Rutgers University, New Brunswick, New Jersey, US.
- Chuntao Hong, Ant Group, Hangzhou, China.
- Yan Zhang, School of Intelligence Science and Technology, Peking University, Beijing, China.
- Siliang Tang, College of Computer Science and Technology, Zhejiang University, Hangzhou, China.


Graph Retrieval-Augmented Generation: A Survey:Research Paper Extract

Fig. 1. Comparison between Direct LLM, RAG, and GraphRAG. Given a user query, direct answering by LLMs may suffer from shallow responses or lack of specificity. RAG addresses this by retrieving relevant textual information, somewhat alleviating the issue. However, due to the text’s length and flexible natural language expressions of entity relationships, RAG struggles to emphasize “influence” relations, which is the core of the question. While, GraphRAG methods leverage explicit entity and relationship representations in graph data, enabling precise answers by retrieving relevant structured information.

Despite their remarkable language comprehension and text generation capabilities, LLMs may exhibit limitations due to a lack of domain-specific knowledge, real-time updated information, and proprietary knowledge, which are outside LLMs’ pre-training corpus. These gaps can lead to a phenomenon known as “hallucination” [53] where the model generates inaccurate or even fabricated information. Consequently, it is imperative to supplement LLMs with external knowledge to mitigate this problem. Retrieval-Augmented Generation (RAG) [27, 37, 51, 54, 165, 180, 187] emerged as a significant evolution, which aims to enhance the quality and relevance of generated content by integrating a retrieval component within the generation process. The essence of RAG lies in its ability to dynamically query a large text corpus to incorporate relevant factual knowledge into the responses generated by the underlying language models. This integration not only enriches the contextual depth of the responses but also ensures a higher degree of factual accuracy and specificity. RAG has gained widespread attention due to its exceptional performance and broad applications, becoming a key focus within the field.

Although RAG has achieved impressive results and has been widely applied across various domains, it faces limitations in real-world scenarios: (1) Neglecting Relationships: In practice, textual content is not isolated but interconnected. Traditional RAG fails to capture significant structured relational knowledge that cannot be represented through semantic similarity alone. For instance, in a citation network where papers are linked by citation relationships, traditional RAG methods focus on finding the relevant papers based on the query but overlook important citation relationships between papers. (2) Redundant Information: RAG often recounts content in the form of textual snippets when concatenated as prompts. This makes context become excessively lengthy, leading to the “lost in the middle” dilemma [94]. (3) Lacking Global Information: RAG can only retrieve a...


Graph Retrieval-Augmented Generation (GraphRAG) emerges as an innovative solution to challenges like Query-Focused Summarization (QFS). Unlike traditional RAG, GraphRAG retrieves graph elements containing relational knowledge for a given query from a pre-constructed graph database, including nodes, triples, paths, or subgraphs. These elements help in generating responses by understanding interconnections between texts, which aids in comprehensive relational information retrieval. Graph data help in abstraction and text summarization, shortening input text, and reducing verbosity. By utilizing subgraphs or graph communities, comprehensive information is obtained, addressing the QFS challenge by understanding broader context and interconnections. This paper offers the first systematic study of GraphRAG, exploring the GraphRAG workflow and its foundational background. It categorizes the literature across key stages of GraphRAG: Graph-Based Indexing (G-Indexing), Graph-Guided Retrieval (G-Retrieval), and Graph-Enhanced Generation (G-Generation), explored in Sections 5, 6, and 7. It covers core technologies, training methods, downstream tasks, application domains, and evaluation methodologies of GraphRAG. This examination reflects its versatility across diverse sectors. Discussing future research directions, the paper prepares groundwork for inspirational studies to propel GraphRAG toward maturity. Contributions include a comprehensive review of state-of-the-art methodologies and a formal definition of GraphRAG, outlining its workflow: G-Indexing, G-Retrieval, and G-Generation. Core technologies for existing systems are discussed, exploring model selection, design, and enhancement strategies. Downstream tasks, challenges, and future research directions are delineated for GraphRAG, providing insights into translating research into industry solutions. The survey is structured as follows: Section 2 compares related techniques; Section 3 outlines GraphRAG processes; Sections 5-7 categorize G-Indexing, G-Retrieval, and G-Generation techniques; Section 8 discusses training strategies; Section 9 summarizes tasks, benchmarks, and industrial systems; Section 10 considers future directions, and Section 11 concludes.


Graph Retrieval-Augmented Generation: A Survey:Comparison with Related Techniques and Surveys

In this section, we compare Graph Retrieval-Augmented Generation (GraphRAG) with related techniques and corresponding surveys, including RAG, LLMs on graphs, and Knowledge Base Question Answering (KBQA).

2.1 RAG

RAG combines external knowledge with LLMs for improved task performance, integrating domain-specific information to ensure factuality and credibility. In the past two years, researchers have written many comprehensive surveys about RAG. For example, Fan et al. and Gao et al. categorize RAG methods from the perspectives of retrieval, generation, and augmentation. Zhao et al. review RAG methods for databases with different modalities. Yu et al. systematically summarize the evaluation of RAG methods. These works provide a structured synthesis of current RAG methodologies, fostering a deeper understanding and suggesting future directions of the area.

From a broad perspective, GraphRAG can be seen as a branch of RAG, which retrieves relevant relational knowledge from graph databases instead of text corpus. However, compared to text-based RAG, GraphRAG takes into account the relationships between texts and incorporates the structural information as additional knowledge beyond text. Furthermore, during the construction of graph data, raw text data may undergo filtering and summarization processes, enhancing the refinement of information within the graph data. Although previous surveys on RAG have touched upon GraphRAG, they predominantly center on textual data integration. This paper diverges by placing a primary emphasis on the indexing, retrieval, and utilization of structured graph data, which represents a substantial departure from handling purely textual information and spurs the emergence of many new techniques.


Graph Retrieval-Augmented Generation: A Survey:LLMs on Graphs

LLMs are revolutionizing natural language processing due to their excellent text understanding, reasoning, and generation capabilities, along with their generalization and zero-shot transfer abilities. Although LLMs are primarily designed to process pure text and struggle with non-Euclidean data containing complex structural information, such as graphs [41, 153], numerous studies [13, 28, 65, 83, 92, 105, 119, 120, 161, 189] have been conducted in these fields. These papers primarily integrate LLMs with GNNs to enhance modeling capabilities for graph data, thereby improving performance on downstream tasks such as node classification, edge prediction, graph classification, and others. For example, Zhu et al. [189] propose an efficient fine-tuning method named ENGINE, which combines LLMs and GNNs through a side structure for enhancing graph representation. Different from these methods, GraphRAG focuses on retrieving relevant graph elements using queries from an external graph-structured database. In this paper, we provide a detailed introduction to the relevant technologies and applications of GraphRAG, which are not included in previous surveys of LLMs on Graphs.


Graph Retrieval-Augmented Generation: A Survey:KBQA

KBQA is a significant task in natural language processing, aiming to respond to user queries based on external knowledge bases [33, 76, 77, 174], thereby achieving goals such as fact verification, passage retrieval enhancement, and text understanding. Previous surveys typically categorize existing KBQA approaches into two main types: Information Retrieval (IR)-based methods and Semantic Parsing (SP)-based methods. Specifically, IR-based methods [60, 61, 102, 142, 155, 168, 181] retrieve information related to the query from the knowledge graph (KG) and use it to enhance the generation process. While SP-based methods [12, 15, 29, 40, 141, 177] generate a logical form (LF) for each query and execute it against knowledge bases to obtain the answer. GraphRAG and KBQA are closely related, with IR-based KBQA methods representing a subset of GraphRAG approaches focused on downstream applications. In this work, we extend the discussion beyond KBQA to include GraphRAG’s applications across various downstream tasks. Our survey provides a thorough and detailed exploration of GraphRAG technology, offering a comprehensive understanding of existing methods and potential improvements.


Graph Retrieval-Augmented Generation: A Survey:Graph Neural Networks

Graph Neural Networks (GNNs) are a kind of deep learning framework to model the graph data. Classical GNNs, e.g., GCN [74], GAT [150], GraphSAGE [44], adopt a message-passing manner to obtain node representations. Formally, each node representation \(h_i^{(l)}\) in the \(l\)-th layer is updated by aggregating the information from neighboring nodes and edges:

\[ h_i^{(l)} = \text{UPD}(h_i^{(l-1)}, \text{AGG}_{j \in \mathcal{N}(i)} \text{MSG}(h_i^{(l-1)}, h_j^{(l-1)}, e_{i,j}^{(l-1)})), \]

where \(\mathcal{N}(i)\) represents the neighbors of node \(i\). MSG denotes the message function, which computes the message based on the node, its neighbor, and the edge between them. AGG refers to the aggregation function that combines the received messages using a permutation-invariant method, such as mean, sum, or max. UPD represents the update function, which updates each node’s attributes with the aggregated messages. Subsequently, a readout function, e.g., mean, sum, or max pooling, can be applied to obtain the global-level representation:

\[ h_G = \text{READOUT}_{i \in \mathcal{V}_G}(h_i^{(L)}). \]

In GraphRAG, GNNs can be utilized to obtain representations of graph data for the retrieval phase, as well as to model the retrieved graph structures.


Graph Retrieval-Augmented Generation: A Survey:Language Models

Language models (LMs) excel in language understanding and are mainly classified into two types: discriminative and generative. Discriminative models, like BERT [22], RoBERTa [97] and Sentence-BERT [129], focus on estimating the conditional probability \(P(y|x)\) and are effective in tasks such as text classification and sentiment analysis. In contrast, generative models, including GPT-3 [10] and GPT-4 [116], aim to model the joint probability \(P(x, y)\) for tasks like machine translation and text generation. These generative pre-trained models have significantly advanced the field of natural language processing (NLP) by leveraging massive datasets and billions of parameters, contributing to the rise of Large Language Models (LLMs) with outstanding performance across various tasks. In the early stages, RAG and GraphRAG focused on improving pre-training techniques for discriminative language models [22, 97, 129]. Recently, LLMs such as ChatGPT [117], LLaMA [24], and Qwen2 [170] have shown great potential in language understanding, demonstrating powerful in-context learning capabilities. Subsequently, research on RAG and GraphRAG shifted towards enhancing information retrieval for language models, addressing increasingly complex tasks and mitigating hallucinations, thereby driving rapid advancements in the field.


Graph Retrieval-Augmented Generation: A Survey:Overview of GraphRAG

GraphRAG is a framework that leverages external structured knowledge graphs to improve contextual understanding of LMs and generate more informed responses, as depicted in Figure 2. The goal of GraphRAG is to retrieve the most relevant knowledge from databases, thereby enhancing the answers of downstream tasks. The process can be defined as

\[ a^* = \arg \max_{a \in A} p(a|q, \mathcal{G}), \]

where \(a^*\) is the optimal answer of the query \(q\) given the TAG \(\mathcal{G}\), and \(A\) is the set of possible responses. After that, we jointly model the target distribution \(p(a|q, \mathcal{G})\) with a graph retriever \(p_\theta(\mathcal{G}|q, \mathcal{G})\) and an answer generator \(p_\phi(a|q, \mathcal{G})\) where \(\theta, \phi\) are learnable parameters, and utilize the


Graph Retrieval-Augmented Generation: A Survey:Graph-Guided Retrieval

Following graph-based indexing, the graph-guided retrieval phase focuses on extracting pertinent information from the graph database in response to user queries or input. Specifically, given a user query q which is expressed in natural language, the retrieval stage aims to extract the most relevant elements (e.g., entities, triplets, paths, subgraphs) from knowledge graphs, which can be formulated as: \(G^* = G-Retriever(q, \mathcal{G})\\ = \arg \max_{G \subseteq \mathcal{G}_{R}(q)} \rho_{\theta}(G|q, \mathcal{G})\\ = \arg \max_{G \subseteq \mathcal{G}_{R}(q)} \text{Sim}(q, G),\)
where G* is the optimal retrieved graph elements and Sim(\cdot, \cdot) is a function that measures the semantic similarity between user queries and the graph data. \(\mathcal{R}(\cdot)\) represents a function to narrow down the search range of subgraphs, considering the efficiency.


Graph Retrieval-Augmented Generation: A Survey:Graph-Based Indexing

The construction and indexing of graph databases form the foundation of GraphRAG, where the quality of the graph database directly impacts GraphRAG’s performance. This section categorizes and summarizes the selection or construction of graph data and various indexing methods employed.

5.1 Graph Data
Various types of graph data are utilized in GraphRAG for retrieval and generation. The data are categorized into two categories based on their sources: Open Knowledge Graphs and Self-Constructed Graph Data.

5.1.1 Open Knowledge Graphs
Open knowledge graphs refer to graph data sourced from publicly available repositories or databases. These knowledge graphs dramatically reduce the time and resources required for development and maintenance. The survey further classifies them into General Knowledge Graphs and Domain Knowledge Graphs.

(1) General Knowledge Graphs
General knowledge graphs primarily store general, structured knowledge and rely on collective input from a global community, ensuring a comprehensive and refreshed repository of information. Encyclopedic knowledge graphs, like Wikidata, Freebase, DBpedia, and YAGO, are examples. Commonsense knowledge graphs include ConceptNet and ATOMIC, which model causal relationships between events.

(2) Domain Knowledge Graphs
Domain-specific knowledge graphs are crucial for enhancing LLMs in domain-specific questions. These KGs offer specialized knowledge, aiding models in understanding complex professional relationships. In the biomedical field, CMeKG encompasses a wide range of data, and CPubMed-KG is a medical database building on PubMed. In the movie domain, Wiki-Movies is an example.


Graph Retrieval-Augmented Generation: A Survey:Graph Retrieval-Augmented Generation

extracts structured information from Wikipedia articles related to films, compiling data about movies, actors, directors, genres, and other relevant details into a structured format. Additionally, Jin et al. [66] construct a dataset named GR-Bench, which includes five domain knowledge graphs spanning academic, e-commerce, literature, healthcare, and legal fields. Furthermore, He et al. [47] convert triplet-format and JSON files from ExplaGraphs and SceneGraphs into a standard graph format and selects questions requiring 2-hop reasoning from WebQSP to create the universal graph-format dataset GraphQA for evaluating GraphRAG systems.

5.1.2 Self-Constructed Graph Data. Self-Constructed Graph Data facilitates the customization and integration of proprietary or domain-specific knowledge into the retrieval process. For downstream tasks that do not inherently involve graph data, researchers often propose constructing a graph from multiple sources (e.g., documents, tables, and other databases) and leveraging GraphRAG to enhance task performance. Generally, these self-constructed graphs are closely tied to the specific design of the method, distinguishing them from the open-domain graph data previously mentioned. To model the structural relationships between the documents, Munikoti et al. [113] propose to construct a heterogeneous document graph capturing multiple document-level relations, including co-citation, co-topic, co-venue, etc. Li et al. [87] and Wang et al. [160] establish relationship between passages according to shared keywords. To capture the relations between entities in documents, Delile et al. [20], Edge et al. [25], Gutiérrez et al. [43] and Li et al. [80] utilize the named entity recognition tools to extract entities from documents and language models to further extract relations between entities, where the retrieved entities and relations then form a knowledge graph. There are also some mapping methods for downstream tasks that need to be designed based on the characteristics of the task itself. For example, to solve the patent phrase similarity inference task, Peng and Yang [122] convert the patent database into a patent-phrase graph. Connections between patent nodes and phrase nodes are established if the phrases appear in the patents, while connections between patent nodes are based on citation relations. Targeting customer service technical support scenarios, Xu et al. [169] propose to model historical issues into a KG, which transforms the issues into tree representations to maintain the intra-issue relations, and utilize semantic similarities and a threshold to preserve inter-issue relations.

5.2 Indexing Graph-Based Indexing plays a crucial role in enhancing the efficiency and speed of query operations on graph databases, directly influencing subsequent retrieval methods and granularity. Common graph-based indexing methods include graph indexing, text indexing, and vector indexing.

5.2.1 Graph Indexing. Graph indexing represents the most commonly used approach, preserving the entire structure of the graph. This method ensures that for any given node, all its edges and neighboring nodes are easily accessible. During subsequent retrieval stages, classic graph search algorithms such as BFS and Shortest Path Algorithms can be employed to facilitate retrieval tasks [64, 66, 102, 142, 146, 175].

5.2.2 Text Indexing. Text indexing involves converting graph data into textual descriptions to optimize retrieval processes. These descriptions are stored in a text corpus, where various text-based retrieval techniques, such as sparse retrieval and dense retrieval, can be applied. Some approaches transform knowledge graphs into human-readable text using predefined rules or templates. For instance, Li et al. [81], Huang et al. [55] and Li et al. [86] use predefined templates to convert each triple in knowledge graphs into natural language, while Yu et al. [179] merge triplets with the same head entity into passages. Additionally, some methods convert subgraph-level information into more descriptive narratives.


Graph Retrieval-Augmented Generation: A Survey:Graph-Based Retrieval Architectures

Textual descriptions are provided for graph-based retrieval methodologies. For example, Edge et al. [25] has explored community detection within graphs, generating summaries using Large Language Models (LLMs). Vector Indexing, discussed in section 5.2.3, converts graph data to vector forms enhancing retrieval efficiency for rapid query processing. This method employs Locality Sensitive Hashing (LSH) [57] and models like G-Retriever [47] for efficient entity linking.

5.2.3 Vector Indexing: Vector indexing transforms data to improve retrieval efficiency. Techniques involve Locality Sensitive Hashing (LSH) [57], language models like G-Retriever [47] for embedding textual information, and GRAG [50] for k-hop gig data conversion to preserve data structure.

Remark: Various indexing methods (graph, text, vector) are compared. Graph indexing allows easy access to structural data, text indexing eases retrieval of textual content, and vector indexing enables efficient searches. A hybrid approach is commonly preferred in practical applications.

6 Graph-Guided Retrieval: This discusses the importance of retrieval for ensuring output quality and relevance when using external graph databases. Challenges include explosive graph size and similarity measurement difficulty.

6.1 Retriever: Involves three types of retrievers for different tasks: Non-parametric, LM-based, and GNN-based. Figure 3 illustrates these architectures.


Graph Retrieval-Augmented Generation: A Survey:Once and Iterative Retrieval Paradigms

6.2.1 Once Retrieval. Once retrieval aims to retrieve all relevant information in a single query. One category of approaches [43, 50, 81] utilize embedding similarities to retrieve the most relevant pieces of information. Another category of methods design pre-defined rules or patterns to directly extract specific structured information such as triplets, paths or subgraphs from graph databases. For example, G-Retriever [47] utilizes an extended PCST algorithm to retrieve the most relevant subgraph. KagNet [88] extracts paths between all pairs of topic entities with lengths not exceeding k. Yasunaga et al. [175] and Taunk et al. [146] extract the subgraph that contains all topic entities along with their 2-hop neighbors. Furthermore, in this subsection, we also include some multiple retrieval methods that involve decoupled and independent retrievals, allowing them to be computed in parallel and executed only once. For example, Luo et al. [102] and Cheng et al. [16] first instruct LLMs to generate multiple reasoning paths and then use a BFS retriever to sequentially search for subgraphs in the knowledge graphs that match each path. KG-GPT [71] decomposes the original query into several sub-queries, retrieving relevant information for each sub-query in a single retrieval process.

6.2.2 Iterative Retrieval. In iterative retrieval, multiple retrieval steps are employed, with subsequent searches depending on the results of prior retrievals. These methods aim to deepen the understanding or completeness of the retrieved information over successive iterations. In this survey, we further classify iterative retrieval into two categories: (1) non-adaptive and (2) adaptive retrieval. We provide a detailed summary of these two categories of methods below.

(1) Non-Adaptive Retrieval. Non-adaptive methods typically follow a fixed sequence of retrieval, and the termination of retrieval is determined by setting a maximum time or a threshold. For example, PullNet [139] retrieves problem-relevant subgraphs through T iterations. In each iteration, the paper designs a retrieval rule to select a subset of retrieved entities, and then expands these entities by searching relevant edges in the knowledge graph. In each iteration, KGP [160] first selects seed nodes based on the similarity between the context and the nodes in the graph. It then uses LLMs to summarize and update the context of the neighboring nodes of the seed nodes, which is utilized in the subsequent iteration.

(2) Adaptive Retrieval. One distinctive characteristic of adaptive retrieval is to let models autonomously determine the optimal moments to finish the retrieval activities. For instance, [42, 168] leverage an LM for hop prediction, which serves as an indicator to end the retrieval. There is also a group of researchers who utilize model-generated special tokens or texts as termination signals for the retrieval process. For example, ToG [142] prompts the LLM agent to explore the multiple possible reasoning paths until the LLM determines the question can be answered based on the current reasoning path. [181] trains a RoBERTa to expand a path from each topic entity. In the process, a virtual relation named as '[END]' is introduced to terminate the retrieval process. Another common approach involves treating the large model as an agent, enabling it to directly generate answers to questions to signal the end of iteration. For instance, [58, 60, 66, 143, 158] propose LLM-based agents to reason on graphs. These agents could autonomously determine the information for retrieval, invoke the pre-defined retrieval tools, and cease the retrieval process based on the retrieved information.


6.2.3 Multi-Stage Retrieval. Multi-stage retrieval divides the retrieval process linearly into multiple stages, with additional steps such as retrieval enhancement, and even generation processes occurring between these stages. In multi-stage retrieval, different stages may employ various types of retrievers, which enables the system to incorporate various retrieval techniques tailored to different aspects of the query. For example, Wang et al. [159] first utilize a non-parametric retriever to extract n-hop paths of entities in the query’s reasoning chain, then after a pruning stage, it further retrieves the one-hop neighbors of the entities in the pruned subgraph. OpenCSR [45] divides the retrieval process into two stages. In the first stage, it retrieves all 1-hop neighbors of the topic entity. In the second stage, it compares the similarity between these neighbor nodes and other nodes, selecting the top-k nodes with the highest similarity for retrieval. GNN-RAG [108] first employs GNNs to retrieve the top-k nodes most likely to be the answer. Subsequently, it retrieves all shortest paths between query entities and answer entities pairwise.

Remark. In GraphRAG, once retrieval typically exhibits lower complexity and shorter response times, making it suitable for scenarios requiring real-time responsiveness. In contrast, iterative retrieval often involves higher time complexity, especially when employing LLMs as retrievers, potentially leading to longer processing times. However, this approach can yield higher retrieval accuracy by iteratively refining retrieved information and generating responses. Therefore, the choice of retrieval paradigm should balance accuracy and time complexity based on specific use cases and requirements.

6.3 Retrieval Granularity According to different task scenarios and indexing types, researchers design distinct retrieval granularities (i.e., the form of related knowledge retrieved from graph data), which can be divided into nodes, triplets, paths, and subgraphs. Each retrieval granularity has its own advantages, making it suitable for different practical scenarios. We will introduce the details of these granularities in the following sections.

6.3.1 Nodes. Nodes allow for precise retrieval focused on individual elements within the graph, which is ideal for targeted queries and specific information extraction. In general, for knowledge graphs, nodes refer to entities. For other types of text attribute graphs, nodes may include textual information that describes the node’s attributes. By retrieving nodes within the graph, GraphRAG systems could provide detailed insights into their attributes, relationships, and contextual information. For example, Munikoti et al. [113], Li et al. [87] and Wang et al. [160] construct document graphs and retrieves relevant passage nodes. Liu et al. [90], Sun et al. [139] and Gutiérrez et al. [43] retrieve entities from constructed knowledge graphs.

6.3.2 Triplets. Generally, triplets consist of entities and their relationships in the form of subject-predicate-object tuples, providing a structured representation of relational data within a graph. The structured format of triplets allows for clear and organized data retrieval, making it advantageous in scenarios where understanding relationships and contextual relevance between entities is critical. Yang et al. [171] retrieve triplets containing topic entities as relevant information. Huang et al. [55], Li et al. [81] and Li et al. [86] first convert each triplet of graph data into textual sentences using predefined templates and subsequently adopt a text retriever to extract relevant triplets. However, directly retrieving triplets from graph data may still lack contextual breadth and depth, thus being unable to capture indirect relationships or reasoning chains. To address this challenge, Wang et al. [152] propose to generate the logical chains based on the original question, and retrieve the relevant triplets of each logical chain.


Graph Retrieval-Augmented Generation: A Survey:Paths, Subgraphs, and Hybrid Granularities in Graph Retrieval

6.3.3 Paths. The retrieval of path-granularity data can be seen as capturing sequences of relationships between entities, enhancing contextual understanding and reasoning capabilities. In GraphRAG, retrieving paths offers distinct advantages due to their ability to capture complex relationships and contextual dependencies within a graph. However, path retrieval can be challenging due to the exponential growth in possible paths as graph size increases, which escalates computational complexity. To address this, some methods retrieve relevant paths based on pre-defined rules. For example, Wang et al. [159] and Lo and Lim [98] first select entity pairs in the query and then traverse to find all the paths between them within n-hop. HyKGE [64] first defines three types of paths: path, co-ancestor chain, and co-occurrence chain, and then utilizes corresponding rules to retrieve each of these three types of paths. In addition, some methods utilize models to perform path searching on graphs. ToG [142] proposes to prompt the LLM agent to perform the beam search on KGs and find multiple possible reasoning paths that help answer the question. Luo et al. [102], Wu et al. [168] and Guo et al. [42] first utilizes the model to generate faithful reasoning plans and then retrieves relevant paths based on these plans. GNN-RAG [108] first identifies the entities in the question. Subsequently, all paths between entities that satisfy a certain length relationship are extracted.

6.3.4 Subgraphs. Retrieving subgraphs offers significant advantages due to its ability to capture comprehensive relational contexts within a graph. This granularity enables GraphRAG to extract and analyze complex patterns, sequences, and dependencies embedded within larger structures, facilitating deeper insights and a more nuanced understanding of semantic connections. To ensure both information completeness and retrieval efficiency, some methods propose an initial rule-based approach to retrieve candidate subgraphs, which are subsequently refined or processed further. Peng and Yang [122] retrieve the ego graph of the patent phrase from the self-constructed patent-phrase graph. Yasunaga et al. [175], Feng et al. [32] and Taunk et al. [146] first select the topic entities and their two-hop neighbors as the node set, and then choose the edges with head and tail entities both in the node set to form the subgraph. Besides, there are also some embedding-based subgraph retrieval methods. For example, Hu et al. [50] first encode all the k-hop ego networks from the graph database, then retrieve subgraphs related to the query based on the similarities between embeddings. Wen et al. [163] and Li et al. [80] extract two types of graphs, including Path evidence subgraphs and Neighbor evidence subgraphs, based on pre-defined rules. OpenCSR [45] starts from a few initial seed nodes and gradually expands to new nodes, eventually forming a subgraph. In addition to the aforementioned direct subgraph retrieval methods, some works propose first retrieving relevant paths and then constructing related subgraphs from them. For instance, Zhang et al. [181] train a RoBERTa model to identify multiple reasoning paths through a sequential decision process, subsequently merging identical entities from different paths to induce a final subgraph.

6.3.5 Hybrid Granularities. Considering the advantages and disadvantages of various retrieval granularities mentioned above, some researchers propose using hybrid granularities, that is, retrieving relevant information of multiple granularities from graph data. This type of granularity enhances the system’s ability to capture both detailed relationships and broader contextual understanding, thus reducing noise while improving the relevance of the retrieved data. Various previous works propose to utilize LLM agents to retrieve complex hybrid information. Jin et al. [66], Jiang et al. [58], Jiang et al. [60], Wang et al. [158] and Sun et al. [143] propose to adopt LLM-based agents for adaptively selecting nodes, triplets, paths, and subgraphs.


Remark. (1) In real applications, there are no clear boundaries between these retrieval granularities, as subgraphs can be composed of multiple paths, and paths can be formed by several triplets. (2) Various granularities such as nodes, triplets, paths, and subgraphs offer distinct advantages in the GraphRAG process. Balancing between retrieval content and efficiency is crucial when selecting the granularity, depending on the specific context of the task. For straightforward queries or when efficiency is paramount, finer granularities such as entities or triplets may be preferred to optimize retrieval speed and relevance. In contrast, complex scenarios often benefit from a hybrid approach that combines multiple granularities. This approach ensures a more comprehensive understanding of the graph structure and relationships, enhancing the depth and accuracy of the generated responses. Thus, GraphRAG’s flexibility in granularity selection allows it to adapt effectively to diverse information retrieval needs across various domains.

6.4 Retrieval Enhancement
To ensure high retrieval quality, researchers propose techniques to enhance both user queries and the knowledge retrieved. In this paper, we categorize query enhancement into query expansion and query decomposition, and knowledge enhancement into merging and pruning. These strategies collectively optimize the retrieval process. Although other techniques such as query rewriting [103, 106, 121, 126] are commonly used in RAG, they are less frequently applied in GraphRAG. We do not delve into these methods, despite their potential adaptation for GraphRAG.

6.4.1 Query Enhancement. Strategies applied to queries typically involve pre-processing techniques that enrich the information for better retrieval. This may include query expansion and query decomposition.

(1) Query Expansion. Due to the generally short length of queries and their limited information content, query expansion aims to improve search results by supplementing or refining the original query with additional relevant terms or concepts. Luo et al. [102] generate relation paths grounded by KGs with LLMs to enhance the retrieval query. Cheng et al. [16] adopt SPARQL to get all the aliases of the query entities from Wikidata to augment the retrieval queries, which capture lexical variations of the same entity. Huang et al. [55] propose a consensus-view knowledge retrieval method to improve retrieval accuracy, which first discover semantically relevant queries, and then re-weight the original query terms to enhance the retrieval performance. HyKGE [64] utilizes a large model to generate the hypothesis output of the question, concatenating the hypothesis output with the query as input to the retriever.

(2) Query Decomposition. Query decomposition techniques break down or decompose the original user query into smaller, more specific sub-queries. Each sub-query typically focuses on a particular aspect or component of the original query, which successfully alleviates the complexity and ambiguity of language queries. For instance, [18, 71] breaks down the primary question into sub-sentences, each representing a distinct relation, and sequentially retrieves the pertinent triplets for each sub-sentence.

6.4.2 Knowledge Enhancement. After retrieving initial results, knowledge enhancement strategies are employed to refine and improve the retriever’s results. This phase often involves knowledge merging and knowledge pruning processes to present the most pertinent information prominently. These techniques aim to ensure that the final set of retrieved results is not only comprehensive but also highly relevant to the user’s information needs.

(1) Knowledge Merging. Knowledge merging retrieved information enables compression and aggregation of information, which assists in obtaining a more comprehensive view by consolidating information.


Graph Retrieval-Augmented Generation: A Survey:Knowledge Pruning

Knowledge pruning involves filtering out less relevant or redundant information to refine outcomes. Approaches for pruning fall into two main categories: (re)-ranking-based methods and LLM-based methods, involving the reordering or prioritization of retrieved information via tailored metrics. For instance, Li et al. use a pre-trained encoder to concatenate each retrieved triplet with a question-choice pair to rank triplets. Jiang et al. utilize FlagEmbedding for re-ranking. Another approach considers the similarity between queries and retrieved info. Cheng et al. rank subgraphs by similarity between subgraphs and queries. Further, a third category suggests new metrics for re-ranking. Muknikoti et al. introduce a metric evaluating both impact and recency of retrieved info. They enhance confidence scores through knowledge graph embedding techniques. LLM-based methods excel in understanding complex linguistic patterns, achieving better response accuracy. Methods like KagNet decompose paths into triplets for re-ranking, ensuring precise result ranking. Methods also mitigate noisy info, where Wang et al. suggest pruning irrelevant graph data with LLM confirmations.


Graph Retrieval-Augmented Generation: A Survey:7.1.1 GNNs

Due to the powerful representational capabilities of GNNs for graph data, they are particularly effective for discriminative tasks. GNNs can directly encode graph data, capturing complex relationships and node features inherent in the graph structure. This encoding is then processed through a Multi-Layer Perceptron (MLP) to generate predictive outcomes. These approaches primarily utilize classical GNN models (e.g., GCN [74], GAT [150], GraphSAGE [44], and Graph Transformers [135]), either in their original form or modified to better align with downstream tasks. For example, Sun et al. [140] compute PageRank scores for neighboring nodes and aggregates them weighted by these scores, during message-passing. This approach enhances the central node’s ability to assimilate information from its most relevant neighboring nodes. Mavromatis and Karypis [107] decode the query into several vectors (instructions), and enhances instruction decoding and execution for effective reasoning by emulating breadth-first search (BFS) with GNNs to improve instruction execution and using adaptive reasoning to update the instructions with KG-aware information.


Graph Retrieval-Augmented Generation: A Survey:7.1.2 LMs

LMs possess strong capabilities in text understanding, which also allows them to function as generators. In the context of integrating LMs with graph data, it is necessary to first convert the retrieved graph data into specific graph formats. This conversion process ensures that the structured information is effectively understood and utilized by the LMs. These formats, which will be elaborated on in Section 7.2, are crucial for preserving the relational and hierarchical structure of the graph data, thereby enhancing the model’s ability to interpret complex data types. Once the graph data is formatted, it is then combined with a query and fed into an LM. For encoder-only models, such as BERT [22] and RoBERTa [97], their primary use is in discriminative tasks. Similar to GNNs, these models first encode the input text and then utilize MLPs to map it to the answer space [55, 61, 81]. On the other hand, encoder-decoder and decoder-only models, such as T5 [127], GPT-4 [116], and LLaMA [24], are adept at both discriminative and generative tasks. These models excel in text understanding, generation, and reasoning, allowing them to process textual inputs directly and generate textual responses [25, 64, 66, 102, 108, 142, 152, 159].


Graph Retrieval-Augmented Generation: A Survey:Hybrid Models

Considering the strengths of GNNs at representing the structure of graph data, and the robust understanding of text demonstrated by LMs, many studies are exploring the integration of these two technologies to generate coherent responses. This paper categorizes the hybrid generative approaches into two distinct types: cascaded paradigm and parallel paradigm.

(1) Cascaded Paradigm. In the cascaded approaches, the process involves a sequential interaction where the output from one model serves as the input for the next. Specifically, the GNN processes the graph data first, encapsulating its structural and relational information into a form that the LM can understand. Subsequently, this transformed data is fed into the LM, which then generates the final text-based response. These methods leverage the strengths of each model in a step-wise fashion, ensuring detailed attention to both structural and textual data. In these methods, prompt tuning is a typical approach, where GNNs are commonly employed to encode the retrieved graph data. This encoded graph data is subsequently prepended as a prefix to the input text embeddings of an LM. The GNN is then optimized through downstream tasks to produce enhanced encodings of the graph data.

(2) Parallel Paradigm. On the other hand, the parallel approach operates by concurrently utilizing the capabilities of both the GNN and the LLM. In this setup, both models receive the initial inputs simultaneously and work in tandem to process different facets of the same data. The outputs are then merged, often through another model or a set of rules, to produce a unified response that integrates insights from both the graphical structure and the textual content. In the parallel paradigm, a typical approach involves separately encoding inputs using both GNNs and LMs, followed by integrating these two representations, or directly integrating their output responses. For instance, some aggregate predictions from GNNs and LMs by weighted summation to obtain the final answer. Another approach involves designing dedicated modules that integrate GNNs with LMs, enabling the resulting representations to encapsulate both structural and textual information.

Remark. Hybrid models that harness both the representation capabilities of GNNs for graph data and LMs for text data hold promising applications. However, effectively integrating information from these two modalities remains a significant challenge.


Graph Retrieval-Augmented Generation: A Survey:Graph Languages

A graph description language is a formalized system of notation specifically crafted to characterize and represent graph data. It prescribes a uniform syntax and semantic framework that describes the components and interconnections within a graph. These languages enable users to consistently generate, manipulate, and interpret graph data in a format comprehensible to machines. They allow for the definition of graph architectures, specification of attributes for nodes and edges, and implementation of operations and queries on graph structures. The document introduces five types of graph languages: Adjacency / Edge Table, Natural Language, Codes, Syntax Tree, and Node Sequence.

(1) Adjacency / Edge Table: Used for describing graph structures, these tables enumerate immediate neighbors of each vertex, offering a compact way to represent connections in sparse graphs. An example is KG-GPT, which linearizes triples in the retrieved subgraph for processing by LLMs.

(2) Natural Language: Typically used when user queries are in natural language. LMs translate graph data into descriptive language for easier comprehension. This aids in bridging raw data and user-friendly formats, and facilitates data-driven applications. Some researchers suggest pre-defining language templates for graph edges, while others use LLMs for generating summaries or rewriting data tables.


Graph Retrieval-Augmented Generation: A Survey:Generation Enhancement

In the generation phase, besides converting the retrieved graph data into formats acceptable by the generator and inputting it together with the query to generate the final response, many researchers explore various methods of generation enhancement techniques to improve the quality of output responses. These methods can be classified into three categories based on their application stages: pre-generation enhancement, mid-generation enhancement, and post-generation enhancement.

7.3.1 Pre-Generation Enhancement. Pre-generation enhancement focuses on improving the quality of input data or representations before feeding them into the generator. Although there is no clear boundary between pre-generation enhancement and retrieval, in this survey, we categorize the retrieval stage as retrieving knowledge from the original graph, merging and pruning retrieved knowledge. Subsequent operations are considered pre-generation enhancements. Commonly used methods involve semantically enriching the graph data to achieve tighter integration between the graph data and textual query. Wu et al. [168] employ LLMs to rewrite retrieved graph data, enhancing naturalness and semantic richness. DALK [80] utilizes the graph data for rewriting queries. Cheng et al. [16] employ LLMs to generate reasoning plans and answer queries accordingly. Taunk et al. [146] and Yasunaga et al. [175] enhance GNNs by enabling them to learn graph representations relevant to queries by extracting all nouns and inserting them as nodes into the retrieved subgraph. Mavromatis and Karypis [107] propose a method where the query representation is decomposed into vectors termed "instructions." PullNet [139] incorporates documents relevant to entities and MVP-Tuning [55] retrieves related questions.

7.3.2 Mid-Generation Enhancement. Mid-generation enhancement involves generation strategies based on intermediate results or contextual cues. TIARA [136] introduces constrained decoding to control output space and reduce errors. When generating logical forms, if a constrained decoder detects a pattern item is being generated, it restricts the next token to tries with KB classes and relations. Compared with Beam Search, this ensures only valid pattern items are generated, reducing errors. MindMap [163] not only produces answers but also generates the reasoning process.

7.3.3 Post-Generation Enhancement. Post-generation enhancement occurs after the initial response is generated. These methods involve integrating multiple generated responses to obtain the final response. Some methods focus on integrating outputs from various generation models.


Graph Retrieval-Augmented Generation: A Survey:8 Training

In this section, we summarize the individual training of retrievers, generators, and their joint training. We categorize previous works into Training-Free and Training-Based approaches based on whether explicit training is required. Training-Free methods are commonly employed when using closed-source LLMs such as GPT-4 [116] as retrievers or generators. These methods primarily rely on carefully crafted prompts to control the retrieval and generation capabilities of LLMs. Despite LLMs’ strong abilities in text comprehension and reasoning, a challenge of Training-Free methods lies in the potential sub-optimality of results due to the lack of specific optimization for downstream tasks. Conversely, Training-Based methods involve training or fine-tuning models using supervised signals. These approaches enhance the model performance by adapting them to specific task objectives, thereby potentially improving the quality and relevance of retrieved or generated content. Joint training of retrievers and generators aims to enhance their synergy, thereby boosting performance on downstream tasks. This collaborative approach leverages the complementary strengths of both components to achieve more robust and effective results in information retrieval and content generation applications.


Graph Retrieval-Augmented Generation: A Survey:8.1 Training Strategies of Retriever

8.1.1 Training-Free. There are two primary types of Training-Free Retrievers currently in use. The first type consists of non-parametric retrievers. These retrievers rely on pre-defined rules or traditional graph search algorithms rather than specific models [146, 175]. The second type utilizes pre-trained LMs as retrievers. Specifically, one group of works utilizes pre-trained embedding models to encode the queries and perform retrieval directly based on the similarity between the query and graph elements [81]. Another group of works adopts generative language models for training-free retrieval. Candidate graph elements such as entities, triples, paths, or subgraphs are included as part of the prompt input to the LLMs. The LLMs then leverage semantic associations to select appropriate graph elements based on the provided prompt [25, 66, 71, 108, 142, 152, 159]. These methods harness the powerful semantic understanding capabilities of LMs to retrieve relevant graph elements without the need for explicit training.


Graph Retrieval-Augmented Generation: A Survey:Training-Based

Training retrievers often adopt an autoregressive approach, where the previous relationship path is concatenated to the end of the query. The model then predicts the next relation based on this concatenated input [42, 168]. However, the lack of ground truth for retrieval content in the majority of datasets poses a significant challenge. To address this issue, many methods attempt to construct reasoning paths based on distant supervision to guide retriever training. For example, Zhang et al. [181], Feng et al. [31] and Luo et al. [102] extract all paths (or shortest paths) between entities in the queries and entities in the answers, using them as training data for the retriever. In addition, Zhang et al. [181] also employ a relationship extraction dataset for distant supervision in unsupervised settings. There is another category of methods that utilize implicit intermediate supervision signals to train Retrievers. For instance, KnowGPT [183] starts searching for the optimal path from the head entity, using the discovery of the tail entity as a reward, and is trained using Policy Gradient. NSM [46] employs a bidirectional search strategy, where two retrievers start searching from the head entity and tail entity, respectively. The supervised objective is to ensure that the paths searched by the two retrievers converge as closely as possible. Some methods argue that distant supervision signals or implicit intermediate supervision signals may contain considerable noise, making it challenging to train effective retrievers. Therefore, they consider employing self-supervised methods for pre-training retrievers. SKP [23] pretrains the DPR (Dense Passage Retrieval) model [69]. Initially, it conducts random sampling on subgraphs and transforms the sampled subgraphs into passages. Subsequently, it randomly masks passages, trains the model using a Masked Language Model (MLM), and employs contrastive learning by treating the masked passages and original passages as positive pairs for comparison.


Graph Retrieval-Augmented Generation: A Survey:Training of Generator

8.2.1 Training-Free. Training-Free Generators primarily cater to closed-source LLMs or scenarios where avoiding high training costs is essential. In these methods, the retrieved graph data is fed into the LLM alongside the query. The LLMs then generate responses based on the task description provided in the prompt, heavily relying on their inherent ability to understand both the query and the graph data.

8.2.2 Training-Based. Training the generator can directly receive supervised signals from downstream tasks. For generative LLMs, fine-tuning can be achieved using supervised fine-tuning (SFT), where task descriptions, queries, and graph data are inputted, and the output is compared against the ground truth for the downstream task [47, 50, 102]. On the other hand, for GNNs or discriminative models functioning as generators, specialized loss functions tailored to the downstream tasks are employed to train the models effectively [59, 81, 146, 175, 184].


Graph Retrieval-Augmented Generation: A Survey:Applications and Evaluation

In this section, we summarize the downstream tasks, application domains, benchmarks and metrics, and industrial applications related to GraphRAG. Table 1 collects existing GraphRAG techniques, categorizing them by downstream tasks, benchmarks, methods, and evaluation metrics. This table serves as a comprehensive overview, highlighting the various aspects and applications of GraphRAG technologies across different domains.

Table 1. The tasks, benchmarks, methods, and metrics of GraphRAG:

- **Tasks:** KBQA, QA, CSQA, IE, Others
- **Benchmarks:** WebQSP [178], WebQ [5], CWQ [144], GraFlaQa [39], QALD-10-en [123], SimpleQuestions [9], CMQA* [159], MetaQA [185], Natural Questions [52], TriviaQA [88], HotpotQA [175], FACTOR [73], Milatnica [134], FreebaseQA [63], CSQA [146], OBQA [109], MedQA [67], SocialIQA [132], RiddleSenseQA [89], ZESHEL [99], CoNLL [49], T-REx [26], ZsRE [124], Creak [115], FB15k-237 [147], FB15k [8], WN18RR [21], NELL995 [11], OpenDialKG [111], Yelp* [156]
- **Methods:** Various numerical libraries and algorithms (e.g., [102], [142], [181], etc.)
- **Metrics:** Accuracy, Hits@1, EM, Recall, FI, BERTScore, GPT-4 Average Ranking, MRR, NDCG@K

9.1 Downstream Tasks
GraphRAG is applied in various downstream tasks (especially NLP tasks), including Question Answering, Information Extraction, and others.


Graph Retrieval-Augmented Generation: A Survey:Graph Retrieval-Augmented Generation: A Survey

(2) CSQA. Distinguished from KBQA, CSQA primarily takes the form of multiple-choice questions. Commonsense reasoning typically presents a commonsense question along with several answer options, each potentially representing either the name of an entity or a statement. The objective is for machines to utilize external commonsense knowledge graphs, such as ConceptNet, to find relevant knowledge pertaining to the question and options, and to engage in appropriate reasoning and derive the correct answer.

9.1.2 Information Retrieval. Information Retrieval tasks consist of two categories: Entity Linking (EL) and Relation Extraction (RE).

(1) Entity Linking. Entity Linking (EL) is a critical task in the field of natural language processing that involves identifying entities mentioned in text segments and linking them to their corresponding entities in a knowledge graph. By leveraging a system such as Graph RAG, it is possible to retrieve relevant information from the knowledge graph, which facilitates the accurate inference of the specific entities that match the mentions in the text [167].

(2) Relation Extraction. Relation Extraction (RE) aims at identifying and classifying semantic relationships between entities within a text. GraphRAG can significantly enhance this task by using graph-based structures to encode and exploit the interdependencies among entities, thus facilitating more accurate and contextually nuanced extraction of relational data from diverse text sources [85, 142, 143].

9.1.3 Others. In addition to the aforementioned downstream tasks, GraphRAG can be applied to various other tasks in the realm of natural language processing such as fact verification, link prediction, dialogue system, and recommender systems.

(1) Fact Verification. The fact verification task typically involves assessing the truthfulness of a factual statement using knowledge graphs. Models are tasked with determining the validity of a given factual assertion by leveraging structured knowledge repositories. GraphRAG techniques can be utilized to extract evidential connections between entities to enhance the system’s efficiency and accuracy [85, 125, 142, 143].

(2) Link Prediction. Link prediction involves predicting missing relationships or potential connections between entities in a graph. GraphRAG is applied to this task [18, 118] by leveraging its ability to retrieve and analyze structured information from graphs, enhancing prediction accuracy by uncovering latent relationships and patterns within the graph data.

(3) Dialogue Systems. Dialogue Systems is designed to converse with humans using natural language, handling various tasks such as answering questions, providing information, or facilitating user interactions. By structuring conversation histories and contextual relationships in a graph-based framework, GraphRAG systems [3] can improve the model’s ability to generate coherent and contextually relevant responses.

(4) Recommender Systems. In the context of e-commerce platforms, the purchase relationships between users and products naturally form a network graph. The primary objective of recommender systems within these platforms is to predict the future purchasing intentions of users, effectively forecasting the potential connections within this graph [156].

9.2 Application Domains
GraphRAG is widely applied in e-commerce and biomedical, academic, literature, legal, and other application scenarios for its outstanding ability to integrate structured knowledge graphs with natural language processing, which will be introduced below.


Graph Retrieval-Augmented Generation: A Survey:Benchmarks and Metrics

9.3 Benchmarks and Metrics
9.3.1 Benchmarks. The benchmarks used to evaluate the performance of the GraphRAG system can be divided into two categories. The first category is the corresponding datasets of downstream tasks. We summarize the benchmarks and papers tested with them according to the classification in Section 9.1, details of which are shown in Table 1. The second category consists of benchmarks specifically designed for the GraphRAG systems. These benchmarks usually cover multiple task domains to provide a comprehensive test result. For example, STARK benchmarks LLM Retrieval on semi-structured knowledge bases covering three domains, including product search, academic paper search, and queries in precision medicine to access the capacity of current GraphRAG systems. He et al. propose a flexible question-answering benchmark targeting real-world textual graphs, named GraphQA, which is applicable to multiple applications including scene graph understanding, commonsense reasoning, and knowledge graph reasoning. Graph Reasoning Benchmark (GRBENCH) is constructed to facilitate the research of augmenting LLMs with graphs, which contains 1,740 questions that can be answered with the knowledge from 10 domain graphs. CRAG provides a structured query dataset, with additional mock APIs to access information from underlying mock KGs to achieve fair comparison.

9.3.2 Metrics. The evaluation metrics for GraphRAG can be broadly categorized into two main types: downstream task evaluation (generation quality) and retrieval quality.

(1) Downstream Task Evaluation (Generation Quality). In the majority of research studies, downstream task evaluation metrics serve as the primary method for assessing GraphRAG's performance. For example, in KBQA, Exact Match (EM) and F1 score are commonly used to measure the accuracy of answering entities. In addition, many researchers utilize BERT4Score and GPT4Score to mitigate instances where LLMs generate entities that are synonymous with the ground truth but not exact matches. In CSQA, Accuracy is the most commonly used evaluation metric. For generative tasks such as QA systems, metrics like BLEU, ROUGE-L, METEOR, and others are commonly employed to assess the quality of the text generated by the model.

(2) Retrieval Quality Evaluation. While evaluating GraphRAG based on downstream task performance is feasible, directly measuring the accuracy of retrieved content poses challenges. Therefore, many studies employ specific metrics to gauge the precision of retrieved content. For instance, when ground truth entities are available, retrieval systems face a balance between the quantity of retrieved information and the coverage of answers. Hence, some studies utilize the ratio between answer coverage and the size of the retrieval subgraph to evaluate the performance of the retrieval system. In addition, several studies have explored metrics such as query relevance, diversity, and faithfulness score to respectively assess the similarity between retrieved content and queries, the diversity of retrieved content, and the faithfulness of the information retrieved.


Graph Retrieval-Augmented Generation: A Survey:Introduction to GraphRAG Systems

Summarization (QFS) task [25]. The project can also utilize open-source RAG toolkits for rapid implementation, such as LlamaIndex, LangChain, etc.

- **GraphRAG (by NebulaGraph):** The project is the first industrial GraphRAG system, which is developed by NebulaGraph Corporation. The project integrates LLMs into the NebulaGraph database, which aims to deliver more intelligent and precise search results.

- **GraphRAG (by Antgroup):** The framework is developed on the foundation of several AI engineering frameworks such as DB-GPT, knowledge graph engine OpenSPG, and graph database TuGraph. Specifically, the system begins by extracting triples from documents using LLMs, which are then stored in the graph database. During the retrieval phase, it identifies keywords from the query, locates corresponding nodes in the graph database, and traverses the subgraph using BFS or DFS. In the generation phase, the retrieved subgraph data is formatted into text and submitted along with the context and query for processing by LLMs.

- **NaLLM (by Neo4j):** The NaLLM (Neo4j and Large Language Models) framework integrates Neo4j graph database technology with LLMs. It aims to explore and demonstrate the synergy between Neo4j and LLMs, focusing on three primary use cases: Natural Language Interface to a Knowledge Graph, Creating a Knowledge Graph from Unstructured Data, and Generate Reports Using Both Static Data and LLM Data.

- **LLM Graph Builder (by Neo4j):** It is a project developed by Neo4j for automatically constructing knowledge graphs, suitable for the GraphRAG’s Graph Database Construction and Indexing phase. The project primarily utilizes LLMs to extract nodes, relationships, and their properties from unstructured data, and utilizes the LangChain framework to create structured knowledge graphs.


Graph Retrieval-Augmented Generation: A Survey:Future Prospects

While GraphRAG technology has made substantial strides, it continues to face enduring challenges that demand comprehensive exploration. This section will delve into the prevalent obstacles and outline prospective avenues for future research in the field of GraphRAG.

**10.1 Dynamic and Adaptive Graphs**

Most GraphRAG methods [25, 33, 76, 77, 101, 174] are built upon static databases; however, as time progresses, new entities and relationships inevitably emerge. Rapidly updating these changes is both promising and challenging. Incorporating updated information is crucial for achieving better results and addressing emerging trends that require current data. Developing efficient methods for dynamic updates and real-time integration of new data will significantly enhance the effectiveness and relevance of GraphRAG systems.

**10.2 Multi-Modality Information Integration**

Most knowledge graphs primarily encompass textual information, thereby lacking the inclusion of other modalities such as images, audio, and videos, which hold the potential to significantly enhance the overall quality and richness of the database [162]. The incorporation of these diverse modalities could provide a more comprehensive and nuanced understanding of the stored knowledge. However, the integration of such multi-modal data presents considerable challenges. As the volume of information increases, the graph’s complexity and size grow exponentially, rendering it increasingly difficult to manage and analyze.


Graph Retrieval-Augmented Generation: A Survey:Conclusion

GraphRAG can support medical diagnosis, patient record analysis, and personalized treatment plans by integrating medical literature, patient histories, and real-time health data. In financial services, GraphRAG can be utilized for fraud detection, risk assessment, and personalized financial advice by analyzing transactional data, market trends, and customer profiles. Legal and compliance applications can benefit from GraphRAG by enabling comprehensive legal research, contract analysis, and regulatory compliance monitoring through the integration of legal documents, case law, and regulatory updates. Expanding GraphRAG to these diverse and complex domains will enhance its utility and impact, providing more sophisticated and targeted solutions across various fields.

11 Conclusion
In summary, this survey offers a comprehensive retrospective of GraphRAG technology, systematically categorizing and organizing its fundamental techniques, training methodologies, and application scenarios. GraphRAG significantly enhances the relevance, accuracy, and comprehensiveness of information retrieval by leveraging pivotal relational knowledge derived from graph datasets, thereby addressing critical limitations associated with traditional Retrieval-Augmented Generation approaches. Furthermore, as GraphRAG represents a relatively nascent field of study, we delineate the benchmarks, analyze prevailing challenges, and illuminate prospective future research directions within this domain.

Acknowledgments
This work is supported by Ant Group through Ant Research Intern Program.


Graph Retrieval-Augmented Generation: A Survey:References

References
[1] Muhammad Arsalan and Christophe Cruz. 2024. Business-RAG: Information Extraction for Business Insights. ICSBT 2024 (2024), 88.
[2] Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary G. Ives. 2007. DBpedia: A Nucleus for a Web of Open Data. In The Semantic Web, 6th International Semantic Web Conference, 2nd Asian Semantic Web Conference, ISWC 2007 + ASWC 2007, Busan, Korea, November 11-15, 2007 (Lecture Notes in Computer Science, Vol. 4825). 722–735.
[3] Jinheon Baek, Alham Fikri Aji, Jens Lehmann, and Sung Ju Hwang. 2023. Direct Fact Retrieval from Knowledge Graphs without Entity Linking. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023. 10038–10055.
[4] Jinheon Baek, Alham Fikri Aji, and Amir Saffari. 2023. Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering. arXiv:2306.04136 [cs.CL] https://arxiv.org/abs/2306.04136
[5] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic Parsing on Freebase from Question-Answer Pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL. 1533–1544.
[6] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: Reasoning about Physical Commonsense in Natural Language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020. 7432–7439.
[7] Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data. 1247–1250.
[8] Kurt D. Bollacker, Colin Evans, Praveen K. Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2008, Vancouver, BC, Canada, June 10-12, 2008. 1247–1250.
[9] Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015. Large-scale Simple Question Answering with Memory Networks. arXiv:1506.02075 [cs.LG] https://arxiv.org/abs/1506.02075
[10] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901.


[11] Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Estevam R. Hruschka Jr., and Tom M. Mitchell. 2010. Toward an Architecture for Never-Ending Language Learning. In Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2010, Atlanta, Georgia, USA, July 11–15, 2010. 1306–1313.

[12] Abir Chakraborty. 2024. Multi-hop Question Answering over Knowledge Graphs using Large Language Models. arXiv:2404.19234 [cs.AI] https://arxiv.org/abs/2404.19234

[13] Haujun Chen. 2024. Large Knowledge Model: Perspectives and Challenges. arXiv:2312.02706 [cs.AI] https://arxiv.org/abs/2312.02706

[14] Runjin Chen, Tong Zhao, Ajay Jaiswal, Neil Shah, and Zhangyang Wang. 2024. LLaGA: Large Language and Graph Assistant. arXiv:2402.08170 [cs.LG] https://arxiv.org/abs/2402.08170

[15] Shuang Chen, Qian Liu, Zhiwei Yu, Chin-Yew Lin, Jian-Guang Lou, and Feng Jiang. 2021. ReTraCk: A flexible and efficient framework for knowledge base question answering. In Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing: system demonstrations. 325–336.

[16] Keyuan Cheng, Gang Lin, Haoyang Fei, Yuxuan aih, Lu Yu, Muhammad Asif Ali, Lijie Hu, and Di Wang. 2024. Multi-hop Question Answering under Temporal Knowledge Editing. arXiv:2404.00492 [cs.CL] https://arxiv.org/abs/2404.00492

[17] Hyeong Kyu Choi, Seunghun Lee, Jaewon Chu, and Hyunwoo J. Kim. 2023. NuTrea: Neural Tree Search for Context-guided Multi-hop KQGA. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

[18] Nurendra Choudhary and Chandan R. Reddy. 2024. Complex Logical Reasoning over Knowledge Graphs using Large Language Models. arXiv:2305.01157 [cs.LO] https://arxiv.org/abs/2305.01157

[19] Yashar Deldjoo, Zhankui He, Julian McAuley, Anton Korikov, Scott Sanner, Armun Ramisa, René Vidal, Maheswaran Sathiamoorthy, Atoosa Kasirzadeh, and Silvia Miliano. 2024. A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys). arXiv:2404.00579 [cs.IR] https://arxiv.org/abs/2404.00579

[20] Julien Delile, Shantayan Mukherjee, Anton Van Pamel, and Leonid Zhukov. 2024. Graph-Based Retriever Captures the Long Tail of Biomedical Knowledge. arXiv:2402.13252 [cs.CL] https://arxiv.org/abs/2402.13252

[21] Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Convolutional 2D Knowledge Graph Embeddings. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018. 1811–1818.

[22] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 4171–4186.

[23] Guanting Dong, Rumei Li, Sirui Wang, Yupeng Zhang, Yunsen Xian, and Weiran Xu. 2023. Bridging the KB-Text Gap: Leveraging Structured Knowledge-aware Pre-training for KBQA. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM 2023, Birmingham, United Kingdom, October 21-25, 2023. 3854–3859.

[24] Abhimanyu Dubey, Abhinav Jauhri, and et al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI] https://arxiv.org/abs/2407.21783

[25] Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. 2024. From Local to Global: A Graph RAG Approach to Query-Focused Summarization. arXiv:2404.16130 [cs.CL] https://arxiv.org/abs/2404.16130

[26] Hady ElSahar, Pavlos Vougiouklis, Arlslen Remaci, Christophe Gravier, Jonathon S. Hare, Frédérique Laforest, and Elena Simperl. 2018. T-REX: A Large Scale Alignment of Natural Language with Knowledge Base Triples. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018.

[27] Wenqi Fan, Yuquan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models. arXiv:2405.06211 [cs.CL] https://arxiv.org/abs/2405.06211

[28] Wenqi Fan, Shijie Wang, Jiani Huang, Zhkai Chen, Yu Song, Wenzhuo Tang, Haitao Mao, Hui Liu, Xiaorui Liu, Dawei Yin, and Qing Li. 2024. Graph Machine Learning in the Era of Large Language Models (LLMs). arXiv:2404.14928 [cs.LG] https://arxiv.org/abs/2404.14928

[29] Haishuo Fang, Xiaodan Zhu, and Iryna Gurevych. 2024. DARA: Decomposition-Alignment-Reasoning Autonomous Language Agent for Question Answering over Knowledge Graphs. arXiv:2406.07080 [cs.CL] https://arxiv.org/abs/2406.07080.


Graph Retrieval-Augmented Generation: A Survey:Research Paper References

[30] Bahare Fatemi, Jonathan Halcrow, and Bryan Perozzi. 2023. Talk like a Graph: Encoding Graphs for Large Language Models. arXiv:2310.04560 [cs.CL] https://arxiv.org/abs/2310.04560

[31] Chao Feng, Xinyu Zhang, and Zichu Fei. 2023. Knowledge Solver: Teaching LLMs to Search for Domain Knowledge from Knowledge Graphs. arXiv:2309.03118 [cs.CL] https://arxiv.org/abs/2309.03118

[32] Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, and Xiang Ren. 2020. Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020. 1295–1309.

[33] Bin Fu, Yunqi Qiu, Chengguang Tang, Yang Li, Haiyang Yu, and Jian Sun. 2020. A Survey on Complex Question Answering over Knowledge Base: Recent Advances and Challenges. arXiv:2007.13069 [cs.CL] https://arxiv.org/abs/2007.13069

[34] Mikhail Galkin, Xinyu Yuan, Hesham Mostafa, Jian Tang, and Zhaocheng Zhu. 2023. Towards Foundation Models for Knowledge Graph Reasoning. In The Twelfth International Conference on Learning Representations.

[35] Hanning Gao, Lingfei Wu, Po Hu, Zhihua Wei, Fangli Xu, and Bo Long. 2022. Graph-augmented Learning to Rank for Querying Large-scale Knowledge Graph. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing, AACL-IJCNLP 2022 - Volume 1: Long Papers, Online Only, November 20-23, 2022, 82–92.

[36] Yifu Gao, Linbo Qiao, Zhiqiang Kan, Zhihua Wen, Yongquan He, and Dongsheng Li. 2024. Two-stage Generative Question Answering on Temporal Knowledge Graph Using Large Language Models. arXiv:2402.16568 [cs.CL] https://arxiv.org/abs/2402.16568

[37] Yunfu Gao, Yun Xiong, Xinya Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv:2312.10997 [cs.CL] https://arxiv.org/abs/2312.10997

[38] Ashish Ghimire, James Prather, and John Edwards. 2024. Generative AI in Education: A Study of Educators’ Awareness, Sentiments, and Influencing Factors. arXiv:2305.15586 [cs.AI] https://arxiv.org/abs/2305.15586

[39] Yu Gu, Sese Xie, Michelle Vanni, Brian M. Sadler, Percy Liang, Xifeng Yan, and Yu Su. 2021. Beyond ILD: Three Levels of Generalization for Question Answering on Knowledge Bases. In WWW '21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021. 3477–3488.

[40] Yu Gu and Yu Su. 2022. ArcaneQA: Dynamic Program Induction and Contextualized Encoding for Knowledge Base Question Answering. In Proceedings of the 29th International Conference on Computational Linguistics. 1718–1731.

[41] Jiaqian Guo, Liren Du, Hengyu Liu, Mengyu Zhou, Xinyi He, and Shi Han. 2023. GPT4Graph: Can Large Language Models Understand Graph Structured Data ? An Empirical Evaluation and Benchmarking. arXiv:2305.15066 [cs.LG] https://arxiv.org/abs/2305.15066

[42] Tiezhen Guo, Qingwen Yang, Chen Wang, Yanyi Liu, Pan Li, Jiawei Tang, Dageng Li, and Yingyu Wen. 2024. KnowledgeNavigator: Leveraging Large Language Models for Enhanced Reasoning over Knowledge Graph. arXiv:2312.15880 [cs.CL] https://arxiv.org/abs/2312.15880

[43] Bernal Jiménez Gutiérrez, Yinghe Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. 2024. HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models. arXiv:2405.14831 [cs.CL] https://arxiv.org/abs/2405.14831

[44] William L. Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive Representation Learning on Large Graphs. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA. 1024–1034.

[45] Zhen Han, Yue Feng, and Mingsheng Sun. 2023. A Graph-guided Reasoning Approach for Open-ended Commonsense Question Answering. arXiv:2303.10395 [cs.CL] https://arxiv.org/abs/2303.10395

[46] Gaole He, Yunshi Lan, Jing Jiang, Wayne Xin Zhao, and Ji-Rong Wen. 2021. Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signals. In WSDM '21, The Fourteenth ACM International Conference on Web Search and Data Mining, Virtual Event, Israel, March 8-12, 2021. 553–561.

[47] Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh V. Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi. 2024. G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering. arXiv:2402.07630 [cs.LG] https://arxiv.org/abs/2402.07630

[48] Michael Himosolf. 1996. GML: Graph Modelling Language. University of Passau (1996).

[49] Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen Fürstenau, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust Disambiguation of Named Entities in Text. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, EMNLP 2011, 27-31 July 2011, John McIntyre Conference Centre, Edinburgh, UK, a meeting of SIGDAT, a Special Interest Group of the ACL. 782–792.

[50] Yuntong Hu, Zhihan Lei, Zheng Zhang, Bo Cen, Xingang Li, and Liang Zhao. 2024. GRAG: Graph Retrieval-Augmented Generation. arXiv:2405.16506 [cs.LG] https://arxiv.org/abs/2405.16506

[51] Yucheng Hu and Xuxing Lu. 2024. RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processing. arXiv:2404.19543 [cs.CL] https://arxiv.org/abs/2404.19543