Graph of Records: Boosting Retrieval Augmented Generation for Long-context Summarization with Graphs
Retrieval-Augmented Generation;Long-context Summarization;Graph Neural Networks;Large Language Models
Yes
No ethics review needed.
I have no further questions about this paper. I am inclined to give it a weak positive score and will review the feedback from other reviewers before finalizing my assessment.
1. The paper is well-structured, with clear explanations and logically organized sections, making it easy for readers to follow the methodology and findings.

2. The authors test GoR on four long-context summarization datasets, and the comprehensive evaluation against a variety of baselines (including both sparse and dense retrievers) demonstrates the robustness and generalizability of the method.

3. The proposed approach builds upon existing RAG techniques, making it relatively easy to implement and reproduce, which is valuable for the research community and practical applications.
The paper introduces the Graph of Records (GoR), a novel method designed to enhance retrieval-augmented generation (RAG) systems for long-context summarization using graph structures. RAG systems improve factuality and relevance in summarization by retrieving relevant content chunks and generating summaries based on those chunks. GoR innovates on traditional RAG by constructing a graph of linked nodes where historical user queries, LLM-generated responses, and retrieved text chunks are interconnected. This graph structure is then refined using a graph neural network (GNN), enabling the model to capture complex relationships between text chunks and responses. By employing a BERTScore-based self-supervised learning objective, GoR aligns node embeddings with semantic relevance to improve summarization accuracy. The model outperforms several baselines across four long-context summarization datasets (e.g., QMSum, AcademicEval), showcasing its effectiveness in creating more comprehensive summaries.
I didn’t find significant weaknesses in this paper. The entire architecture is built on existing modules, making the proposed framework both sound and replicable. However, this reliance on established methods might also be a limitation, as the framework feels less innovative or exciting despite being well-presented with informative experimental results.
Yes
No ethics review needed.
1. What are Ec and Eq context encoders in line 224-230? Are they also optimized by eq. 7? 
2. The paper mentioned BERTscore many times in this paper but how does BERTscore contribute to the GoR? 
3. In both contrastive learning and ranking loss, what is the motivation of  choosing the positive samples based on the semantic similarity from the encoder? 
4. Provide a case study of GoR and other baseline methods' summarization.
1. Introduces an innovative approach to integrate LLMs' intermediate summaries with original document chunks in a structured graph representation
2. Implements an efficient sparse connectivity strategy using top-K similar textual chunks, reducing computational complexity while maintaining information flow
3. This hierarchical structure effectively bridges local and global document understanding
4. Develops a well-motivated semantic alignment mechanism for the GNN by leveraging BERTScore-based similarities
5. Employs a dual-objective training strategy combining contrastive learning and pairwise ranking loss
6. This approach ensures the GNN's representations capture meaningful semantic relationships beyond surface-level textual similarity
7. Provides a thorough analysis of existing long-context summarization approaches, clearly identifying their limitations
8. Effectively positions the work within two key research streams: retrieval-augmented generation and graph-based document processing
9. Demonstrates clear technological advancement over previous methods through thoughtful experimental design
This paper presents a novel retrieval-augmented generation (RAG) approach for long-context document summarization. The method leverages a graph neural network (GNN) to retrieve relevant information from structured textual nodes. These nodes comprise two types of content: (1) original text chunks from the source document and (2) intermediate summaries previously generated by large language models (LLMs). To optimize the GNN retriever, the authors propose aligning the GNN's representations with semantic similarities from textual encoder, creating a more semantically-aware retrieval mechanism.

Key contributions include:

- A hierarchical text representation using GNNs that combines both source text and intermediate summaries
- A novel training objective that aligns GNN embeddings with semantic similarity metrics
- An end-to-end framework that integrates retrieval and generation for long-document summarization
1. Clarity and Presentation Issues: 
- **Problem Definition** : Lacks a dedicated subsection that formally defines long-context summarization; Missing explicit comparison between GoR and existing RAG approaches for long-context summarization in the introduction; Would benefit from a clear positioning diagram or framework overview

- **Technical Clarity** : Graph construction description (lines 147-194) lacks sufficient detail and clear visualization; Equation 3's retrieval mechanism is ambiguous about whether it includes LLM-generated responses or only original document chunks; The relationship between the proposed contrastive learning approach and BERTScore needs clearer explanation; The semantic alignment objective would benefit from step-by-step derivation


2. Limited Evaluation Metric: Over-reliance on ROUGE metrics for evaluation; Absence of crucial evaluation metrics: Human evaluation for factual consistency and coherence; BERTScore for semantic similarity assessment; Coverage metrics for long-document comprehension.

3. Insufficient Analysis of Critical Components: Lacks detailed analysis of edge creation strategy between document chunks and LLM responses; Insufficient investigation of the impact of LLM-generated responses; No comparative analysis showing the benefit of including historical LLM responses.
Yes
No ethics review needed.
Please see weakness
Innovative Use of Historical Responses: The paper introduces a novel approach by leveraging LLM-generated historical responses for enhancing RAG, which is largely neglected by existing methods. This approach enriches the summarization process, potentially increasing the relevance and depth of generated summaries.
This paper proposes a novel method called GoR (Graph of Records) to enhance RAG in long-context global summarization tasks. GoR utilizes historical responses generated by LLMs to construct a graph structure that connects retrieved text chunks with their corresponding responses. This paper further employs a graph neural network and BERTScore-based self-supervised training to reveal correlations between them. Experimental results show that GoR outperforms 12 baselines across four long-context summarization datasets, demonstrating its effectiveness in improving summarization quality.
__I.__ Computational Efficiency Evaluation: The paper lacks experimental validation of computational efficiency. The reliance on LLM-generated responses and retrieved chunk graphs, combined with the incorporation of graph neural networks and BERTScore-based objectives, could introduce substantial computational overhead.

__II.__ Dependency on Data Quality: The effectiveness of GoR may rely heavily on the quality and coherence of the historical responses generated by LLMs. Inconsistencies in these responses could impact the model's overall summarization performance.

__III.__ The quality of the presentation is below ICLR 2025 standards. For example:

  a. The format of the references should be consistent to ensure neatness and professionalism. For instance, the names of conferences should be uniformly presented either in abbreviations or full names, rather than a mixture of both.

  b. For the line 265, it should be written as “we only use “general queries” for evaluating”, and the symbol use is wrong.
Yes
No ethics review needed.
1. The authors discuss the impact of different GNN architectures; however, a more fundamental research question remains: does the constructed structure itself actually improve performance? A follow-up question is: does the LLM's historical responses are really helpful, as claimed by the authors? I understand these are challenging to address, but the paper could benefit from empirical studies, such as ablation experiments on the constructed graph (e.g., ablating the response nodes).

2. A small issue for the presentation. In the analysis on L424, a more informative metric on the x-axis might be the number of tokens used for training. Increasing the number of simulated queries also scales the number of responses generated by the LLM, meaning the model could benefit from both the queries and the additional response tokens.

3. I would appreciate a comparison of inference time per query for the baselines in the main table. This information is valuable for readers interested in inference efficiency for deployment.

4. Regarding "Supervised Training on Global Summarization Queries," are you referring to a baseline where LLaMA-2-7B-chat is fine-tuned using supervised training on global summarization queries and reference summaries?

5. Would it make sense to add query nodes between the response and context nodes in the graph? My intuition is that query nodes could help the model better understand logical constraints between context and response (e.g., negation).
1. Using LLM's parametric knowledge—referred to as historical responses in this paper—to enhance retrieval-augmented generation (RAG) is a valuable addition to the static corpus used in traditional RAG. The authors claim to be the first to apply this approach to long-document summarization.
2. The experimental results are promising, though some analyses may not fully address the research question.
3. The paper is well-written and easy to follow.
This paper presents GoR, a retrieval-augmented generation method aimed at enhancing the summarization of long documents. The authors propose a novel approach to distilling the knowledge of large language models (LLMs) to improve retrieval by incorporating both historical LLM responses and text chunks into a structured graph. Their second key contribution is a comprehensive framework for training a graph neural network (GNN)-based retriever on this graph, learning to rank chunk and response nodes using contrastive and ranking objectives. Experimental comparisons with traditional retrievers and long-context LLMs demonstrate the advantages of GoR in long-document summarization.
1. The analyses lack sufficient convincing evidence. See the “Questions” section for more details.
2. The graph construction process is complex, which could potentially increase inference time—a critical factor in RAG systems.
We propose a method named graph of records (GoR), which utilizes LLM-generated historical responses in retrieval-augmented generation to enhance long-context summarization
@inproceedings{
anonymous2024graph,
title={Graph of Records: Boosting Retrieval Augmented Generation for Long-context Summarization with Graphs},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=6LKmaC4cO0},
note={under review}
}
Retrieval-augmented generation (RAG) has revitalized Large Language Models (LLMs) by injecting non-parametric factual knowledge. 
Compared with long-context LLMs, RAG is considered an effective summarization tool in a more concise and lightweight manner, which can interact with LLMs multiple times using diverse queries to get comprehensive responses. However, the LLM-generated historical responses, which contain potentially insightful information, are largely neglected and discarded by existing approaches, leading to suboptimal results. In this paper, we propose \textit{graph of records} (\textbf{GoR}), which leverages historical responses generated by LLMs to enhance RAG for long-context global summarization. Inspired by the \textit{retrieve-then-generate} paradigm of RAG, we construct a graph by creating an edge between the retrieved text chunks and the corresponding LLM-generated response. To further uncover the sophisticated correlations between them, GoR further features a \textit{graph neural network} and an elaborately designed \textit{BERTScore}-based objective for self-supervised model training, enabling seamless supervision signal backpropagation between reference summaries and node embeddings. We comprehensively compare GoR with 12 baselines on four long-context summarization datasets, and the results indicate that our proposed method reaches the best performance. Extensive experiments further demonstrate the effectiveness of GoR.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Retrieval-Augmented Generation Long-context Summarization Graph Neural Networks Large Language Models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/c404f2eb411619921dcda25a1d03bfcf1ff16faf.pdf
learning on graphs and other geometries & topologies
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Graph of Records: Boosting Retrieval Augmented Generation for Long-context Summarization with Graphs
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission