GARLIC: LLM-Guided Dynamic Progress Control with Hierarchical Weighted Graph for Long Document QA
retrieval;LLM;graph;dynamic;summary;attention;KV cache;QA
Yes
No ethics review needed.
See weakness
S1. The proposed method employs a hierarchical weighted directed acyclic graph instead of a tree structure, utilizing multi-path dynamic retrieval and hierarchical summary nodes.
S2. It is an interesting idea to assign weights to the edges based on attention, which allows GARLIC to adjust the retrieval depth and information volume flexibly.
S3. Decent performance compared to other existing approaches for retrieval.
This work introduces a retrieval method, "GARLIC" (LLM-Guided Dynamic Progress Control with Hierarchical Weighted Graph), aimed at improving long document question answering. GARLIC operates through a multi-step process. First, it constructs a Hierarchical Weighted Directed Acyclic Graph (HWDAG) where each node represents a focused "Information Point" (IP) generated through LLM-based summarization. Second, GARLIC dynamically retrieves nodes from the graph, leveraging attention weights to assess relevance and terminate the search once sufficient information is collected. Third, it employs a Greedy Best-First Search to explore multiple graph paths, enhancing flexibility by allowing the search to stop at various levels based on the query's needs. This approach effectively balances retrieval precision and computational efficiency, outperforming existing methods in single and multi-document QA tasks.
W1. An explanation of the edge weights in the directed acyclic graph needs further clarification: specifically, how the weights are calculated and how they function during the search process. The current presentation lacks intuitiveness.
W2. Detailed explanation on how KV caching is applied in the dynamic progress control is needed.
W3. The code should be made publicly available.
W4. Some baseline methods mentions in the article search down to the leaf nodes during retrieval, in other words, gather more information. However, why do they perform worse than the method presented in this paper, even though the proposed method may terminate the search early? Further analysis would be beneficial.
Yes
No ethics review needed.
How does the system handle differences in event granularity? Specifically, do Information Points (IPs) for shorter events lead to inconsistencies in information distribution across the graph?

How does GARLIC scale when handling complex attention dependencies or queries that span multiple, disparate sections of a document? Is there a threshold at which the attention-guided search might encounter limitations?

Has dynamically adjusting the stop patience based on query complexity or node connectivity been considered? Could this approach optimize retrieval efficiency within larger graphs?

How does the efficiency of graph search vary in contexts with mixed single- and multi-query documents, particularly in cases with highly fragmented narratives, such as those in NarrativeQA?

GARLIC appears to extend ideas from both RAPTOR and MemWalker, borrowing sub-graph or informational-node stacking from RAPTOR and summarization from MemWalker. Could you clarify the main design distinctions and advantages of GARLIC compared to these methods (apart from using greedy search)?

Could you explain the choice to exclude LongT5 XL (Guo et al., 2022) and CoLT5 XL (Ainslie et al., 2023) as baselines (or any other baseline you feel may have given better performance similar to the base Llama3.1 on the benchmarked datasets), given their performance as open-source models? While ChatGPT’s closed-source nature makes its exclusion understandable, a rationale for omitting these alternatives would help clarify comparisons. Also, could you discuss the reported GARLIC performance alongside Table 5 in [1]?

From Table 1, it appears that a significant portion of GARLIC's performance boost is coming from the Llama-3.1 base model, as it already outperforms other baselines. How might results compare if Llama-3.1 were used with RAPTOR instead, considering RAPTOR's higher efficiency (0.5:1 on 1 datasets, and always below 1 on all remaining datasets)?

[1] Sarthi, Parth, et al. "Raptor: Recursive abstractive processing for tree-organized retrieval." arXiv preprint arXiv:2401.18059 (2024)."
By using dynamic stopping and attention-weighted paths, GARLIC avoids unnecessary computation, which is common in exhaustive retrieval methods.

Outperforms benchmarks like Llama 3.1 and recent methods on NarrativeQA, Qasper, HotpotQA, and MuSiQue.

Compatible with long-context LLMs (e.g., Llama 3.1), making it feasible for single GPU execution.
The paper presents GARLIC, a retrieval method that uses Hierarchical Weighted Directed Acyclic Graphs to improve long-document question-answering (QA) tasks. Unlike traditional retrieval-augmented generation (RAG) and tree-based models, GARLIC combines a unique two-stage process - Summary Graph Construction and Dynamic Graph Search.
The attention normalization approach may limit scalability to larger graphs and complex attention patterns.

While the retrieval stage is efficient, graph construction is relatively resource-intensive, especially for single-query documents.

Embedding similarity, although beneficial, introduces some redundancy when compared with pure attention-based retrieval.
Yes
NA.
No ethics review needed.
Please refer to weakness, thanks.
1. Overall, this paper is well-written and easy to read.
2. While existing RAG methods perform worse than directly feeding the entire document into Llama 3.1, the authors propose a new RAG method outperforms Llama 3.1 and retaining the computational efficiency of RAG methods. This approach can be used for reference by other researchers.
This paper proposes a new retrieval method, GARLIC (LLM-Guided Dynamic Progress Control with Hierarchical Weighted Graph), which aims to improve information retrieval in long document question answering tasks. It replaces the traditional tree structure by building a hierarchical weighted directed acyclic graph and uses the attention weights of large language models (LLMs) to perform multi-path search and dynamically control the retrieval process. Experiments show that this method surpasses the existing state-of-the-art baselines, including methods that directly use Llama 3.1 to process the entire document, while maintaining the computational efficiency of the RAG method.
1. In line 292, the author mentioned that the node's attention is multiplied by the corresponding position information to track the earlier information in the sequence. I think the author should conduct experiments to explain whether other adjustment methods have been tried? Why is this adjustment method effective? 
2. Lack of comparative analysis with other methods: Although the proposed model achieved excellent performance, the authors did not clearly explain why HWDAG is superior to the tree-based RAG method? Tree-based retrieval also seems to be able to summarize long documents and answer questions by selecting important nodes that are highly relevant to the query in a top-down manner through the GRAPH SEARCH method proposed by the authors. In addition, HWDAG seems to organize documents into a top-down hierarchy with different granularity, so low-level nodes seem to be a refinement and explanation of top-level nodes. As one of the main motivations of the article, I hope the author can explain why introducing refined information will hurt performance for some problems.
3. During the construction of the summary graph, the authors iteratively aggregate lower-level nodes by batching them and inputting them into the large language model (LLM) to generate higher-level nodes. However, the details of the iterative batching mechanism require further elaboration.

4. The authors argue that the information points (IPs) focus on “a single event or very few events”. It seems that the authors achieve this with just several instructions in the prompt. Is this approach sufficient to ensure the achievement of the intended objectives?

5. The authors average the token-level attentions to obtain the weights between information chunks (IPs). However, since these chunks represent higher-level, text-based content, is averaging token-level attentions an appropriate method for establishing relationships between these information chunks?

Comments, Suggestions And Typos:
1. In Section 4 Experiments, it is recommended to highlight the metrics that perform well in the table by bolding or underlining them.
Yes
No ethics review needed.
see weakness
1. The system uses a large language model to recursively generate key information from long texts, constructing an information structure graph. It employs the model's attention mechanism to assign weights to edges, introducing a new structure for information representation.

2. The authors propose using a priority-based breadth-first search to retrieve information points within the graph. The search process is controlled by the large language model, allowing retrieval to stop as soon as relevant information is found. This approach, compared to depth-first search, offers greater flexibility while maintaining efficiency. Final experimental results outperform recent benchmarks, including Llama 3.1.
The authors propose a method using LLMs to summarize long text into key information points, organizing these points into a hierarchical structure through recursive processing. This results in a directed acyclic graph where edge weights between nodes are generated based on attention mechanisms. For QA tasks, the authors apply a weighted BFS to retrieve relevant information from the graph. An LLM dynamically controls the retrieval process by determining if the information collected so far is sufficient to answer the question.
1. The paper does not explain how edge weights between nodes are calculated using the attention mechanism of the LLM, lacking crucial algorithmic details.

2. In Section 3.2.2, the paper discusses calculating the relevance between the query and information points using attention. How is this calculation performed, especially if it needs to be precomputed? This method may have limited scalability for new queries and new documents.

3. No code is released, making it difficult for others to use the proposed method or reproduce the result.
Dynamic Progress Control with Hierarchical Weighted Graph for QA
@inproceedings{
anonymous2024garlic,
title={{GARLIC}: {LLM}-Guided Dynamic Progress Control with Hierarchical Weighted Graph for Long Document {QA}},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=BKGM8fyFIo},
note={under review}
}
In the past, Retrieval-Augmented Generation (RAG) methods split text into chunks to enable language models to handle long documents. Recent tree-based RAG methods are able to retrieve detailed information while preserving global context. However, with the advent of more powerful LLMs, such as Llama 3.1, which offer better comprehension and support for longer inputs, we found that even recent tree-based RAG methods perform worse than directly feeding the entire document into Llama 3.1, although RAG methods still hold an advantage in reducing computational costs. In this paper, we propose a new retrieval method, called LLM-Guided Dynamic Progress Control with Hierarchical Weighted Graph (GARLIC), which outperforms previous state-of-the-art baselines, including Llama 3.1, while retaining the computational efficiency of RAG methods. Our method introduces several improvements: (1) Rather than using a tree structure, we construct a Hierarchical Weighted Directed Acyclic Graph with many-to-many summarization, where the graph edges are derived from attention mechanisms, and each node focuses on a single event or very few events. (2) We introduce a novel retrieval method that leverages the attention weights of LLMs rather than dense embedding similarity. Our method allows for searching the graph along multiple paths and can terminate at any depth. (3) We use the LLM to control the retrieval process, enabling it to dynamically adjust the amount and depth of information retrieved for different queries. Experimental results show that our method outperforms previous state-of-the-art baselines, including Llama 3.1, on two single-document and two multi-document QA datasets, while maintaining similar computational complexity to traditional RAG methods.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
retrieval LLM graph dynamic summary attention KV cache QA
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/2760d25d17ed2bc70c7416c0aa0393296826055c.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
GARLIC: LLM-Guided Dynamic Progress Control with Hierarchical Weighted Graph for Long Document QA
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission