Locate-then-edit for Multi-hop Factual Recall under Knowledge Editing
Knowledge Editing
Yes
No ethics review needed.
- **Figures**: Consider widening Figures 2(a) and 2(b) to improve readability and enlarge the subfigures in Figure 6 for better visibility.
- **Tables**: Provide a clearer explanation of the term “base” in Tables 5 and 6. It’s unclear whether this refers to the “original” in Table 3 or another baseline. Clear labels and descriptions would enhance understanding.
- **Comprehensive Analysis**: The paper presents a thorough analysis in Section 3, identifying that information related to multi-hop knowledge is stored in later MLP layers. Using interpretability tools such as the logit lens and causal interventions, the authors confirm this and propose edits to the last layer based on these findings.
- **Empirical Validation**: Experiments in Tables 3 and 4 show that the proposed method outperforms previous methods on multi-hop knowledge editing tasks, providing compelling evidence for its effectiveness.
The paper addresses challenges in existing locate-then-edit knowledge editing methods, particularly focusing on multi-hop knowledge editing. The authors first investigate why locate-then-edit approaches struggle with multi-hop fact editing using tools from mechanistic interpretability, such as the logit lens and causal intervention. Their findings suggest that later MLP layers play a crucial role in storing multi-hop knowledge. To better update these knowledge-storing MLPs, the authors introduce a supplementary set construction method for each edit, transforming each edit into a multi-hop chain. This supplementary set then generates virtual multi-hop prompts specifically targeting the knowledge being edited. The authors utilize two-hop templates from this supplementary set to modify the model’s later layers, thereby enhancing multi-hop knowledge editing.
- **Insufficient Presentation**: Section 5.2 lacks clarity as Tables 5 and 6 are referenced but not included in the main paper, instead being located in the appendix. While space constraints are understandable, omitting key results from the main discussion detracts from readability and coherence. Additionally, the absence of descriptions for the "base" in Tables 5 and 6 and missing details on the datasets used in Table 4 (e.g., whether MQuAKE is consistently used) leaves gaps in understanding.
- **Questionable Analysis of Results**: In Table 3, performance seems to decline post-editing compared to the original, but there is no discussion on the possible causes for this drop. If editing knowledge reduces performance, the rationale for the edits becomes unclear. It appears that “original” performance refers to accuracy on the unedited answer, but if so, why wasn’t accuracy on the correct answer also reported for the original model? Clearer explanations are necessary to clarify these performance metrics.
- **Limited Novelty in Method and Analysis**: The primary contribution of this work lies in the insight, obtained through interpretability tools, that multi-hop knowledge editing requires updating the later MLP layers. However, there is no clear experimental analysis showing that editing the later layers specifically contributes to performance improvement. It raises the question of whether merely expanding the support set was sufficient for better performance, implying that gains may not be attributed solely to the proposed method. To substantiate this claim, it would be useful to report the impact of the editing layers on performance, comparing different layers with and without the supplementary set.
Yes
No ethics review needed.
1. The biblology style should be consistent. Some lines are underlined and others are not.
2. How does IFMET perform on other language models? Like recent Llama-3.
3. How long does IFMET take to edit one multi-hop sample?
4. Can IFMET deal with unstructured facts for knowledge editing as in [1]?
5. Line 387, cizizenship --> citizenship.
6. Figure 1, Mardrid --> Madrid.

 [1] Updating Language Models with Unstructured Facts: Towards Practical Knowledge Editing
1. This paper fills a gap in KE for multi-hop knowledge editing. It addresses the limitations of traditional local-then-edit methods that primarily operate in shallow layers.
2. The paper gives detailed explanations and experiments about how single-hop and multi-hop queries are handled differently within language models. The used LogitsLen and causal intervention experiments are interesting.
3. The proposed IFMET is compared with strong baselines across experiments and shows great performance.
This paper introduces IFMET, a novel locate-then-edit method for knowledge editing for language models.
It aims to improve the multi-hop factual recall by modifying both shallow and deep layers of model weights.
Previous methods focus on shallow layers and thus struggle with multi-hop editing.
IFMET addresses this by using multi-hop prompts and supplementary sets to locate and edit knowledge across multiple layers.
This allows the language model to answer multi-hop queries more accurately post-edit.
Experimental results show that IFMET outperforms existing knowledge editing methods (such as Mello, and MEMIT) on multi-hop knowledge editing.
1. The used language model GPT-J (6B) is a bit outdated. More recent language models are expected to be considered.
2. How about the performance of the generality and locality of knowledge editing? The editing should also affect related facts but not affect other unrelated facts.
3. The paper lacks discussions about editing efficiency. What is the time complexity of IFMET?
4. Compared to Mello, IFMET requires supplementary set construction from external sources (same for MEMIT and ROME). This may hinder its applications in real cases.
Yes
No ethics review needed.
See weakness
1. Strong Empirical Analysis: The authors provide a thorough, well-designed investigation into how LLMs process multi-hop queries differently from single-hop ones. The use of LogitLens and causal intervention experiments to track information flow through different layers is particularly impressive.

2. Novel Theoretical Insight: The discovery that implicit subject information accumulates in middle layers before being used to retrieve answers from deeper MLP layers is an important contribution to our understanding of LLM reasoning. This finding not only explains the limitations of current KE methods but also has broader implications for LLM interpretability research.

3. Good result: The proposed IFMET solution is elegant and well-motivated by the theoretical findings. This is a practical solution that improves performance on multi-hop queries.
This paper addresses an important limitation in knowledge editing for LLMs - their poor performance on multi-hop factual recall tasks after editing. Through careful mechanistic analysis using tools like LogitLens and causal intervention experiments, the authors discover that LLMs process multi-hop queries differently from single-hop ones - they accumulate implicit subject information in middle layers before retrieving the final answer from deeper MLP layers. This explains why current KE methods, which focus on editing shallow layers, fail at multi-hop tasks. Based on these insights, they propose IFMET, a novel locate-then-edit approach that edits both shallow and deep MLP layers using supplementary knowledge sets and multi-hop prompts. Their experimental results on the MQuAKE dataset show improvements over existing methods, particularly for complex multi-hop queries.
Overall, this paper is solid. But some parts of this paper are missing:

1. Why do we need to conduct "Locate-then-edit" editing when we have some very good performance RAG-based editing methods [1][2]? There is no need for this paper to compare performance with RAG-based editing in the experiments part, but a short discussion should be included on why IFMET is superior to these RAG-based methods.

2. Currently, only experiments are conducted on MQUAKE-3k; more benchmarks, such as MQUAKE-T, could be included.

[1] Shi, Yucheng, et al. "Retrieval-enhanced Knowledge Editing in Language Models for Multi-Hop Question Answering." CIKM, 2024.
[2] Gu, Hengrui, et al. "Pokemqa: Programmable knowledge editing for multi-hop question answering." ACL, 2024.
Yes
No ethics review needed.
Please refer to the Weaknesses.
1. The paper provides a thorough and interesting analysis of the differences between single-hop and multi-hop tasks, highlighting the importance of editing deeper layers in LLMs.

2. The writing is clear, making the proposed method easy to understand and follow.

3. The experimental results are impressive, demonstrating significant improvements over existing methods in multi-hop factual recall tasks.
The paper introduces IFMET, a novel locate-then-edit approach for knowledge editing in Large Language Models (LLMs), specifically targeting multi-hop factual recall tasks. The authors leverage mechanistic interpretability to identify that LLMs retrieve implicit subject knowledge from deeper MLP layers in multi-hop tasks, unlike single-hop tasks, which rely on earlier layers. This insight leads to the development of IFMET, which edits both shallow and deep MLP layers. Experimental results show that IFMET significantly outperforms existing methods on multi-hop factual recall tasks.
1. The method may have certain limitations. Specifically, using WikiData and SPARQL queries to construct the supplementary dataset may not be applicable to non-factual or non-encyclopedic data, limiting the generalizability of the approach.

2. The organization of the paper could be improved. The experimental section is relatively short, with some important experimental tables relegated to the appendix. Additionally, the paper lacks a comprehensive ablation study.

3. The explanation and analysis of the experimental results are somewhat lacking. For instance, in Table 3, it is unclear what the values represent (e.g., accuracy) and why the performance of IFMET improves as the edit batch size increases, while the performance of other baselines decreases. More detailed explanations and discussions of these observations would strengthen the paper.
@inproceedings{
anonymous2024locatethenedit,
title={Locate-then-edit for Multi-hop Factual Recall under Knowledge Editing},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=INFfvQArFY},
note={under review}
}
The locate-then-edit paradigm has shown significant promise for knowledge editing (KE) in Large Language Models (LLMs). While previous methods perform well on single-hop fact recall tasks, they consistently struggle with multi-hop factual recall tasks involving newly edited knowledge. In this paper, leveraging tools in mechanistic interpretability, we first identify that in multi-hop tasks, LLMs tend to retrieve implicit subject knowledge from deeper MLP layers, unlike single-hop tasks, which rely on earlier layers. This distinction explains the poor performance of current methods in multi-hop queries, as they primarily focus on editing shallow layers, leaving deeper layers unchanged. To address this, we propose IFMET, a novel locate-then-edit KE approach designed to edit both shallow and deep MLP layers. IFMET employs multi-hop editing prompts and supplementary sets to locate and modify knowledge across different reasoning stages. Experimental results demonstrate that IFMET significantly improves performance on multi-hop factual recall tasks, effectively overcoming the limitations of previous locate-then-edit methods.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Knowledge Editing
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/4626d119ac0120d7a8f75e0f11a54a2f9149da02.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Locate-then-edit for Multi-hop Factual Recall under Knowledge Editing
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission