<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2406.01436] Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models</title><meta property="og:description" content="Knowledge editing is a rising technique for efficiently updating factual knowledge in Large Language Models (LLMs) with minimal alteration of parameters. However, recent studies have identified concerning side effects,…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2406.01436">

<!--Generated on Sat Jul  6 00:46:07 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document">
<h1 class="ltx_title ltx_title_document">Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Cheng-Hsun Hsueh<sup id="id6.6.id1" class="ltx_sup">∗</sup> Paul Kuo-Ming Huang<sup id="id7.7.id2" class="ltx_sup">∗</sup> Tzu-Han Lin<sup id="id8.8.id3" class="ltx_sup">∗</sup> Che-Wei Liao<sup id="id9.9.id4" class="ltx_sup">∗</sup> 
<br class="ltx_break"><span id="id5.5.1" class="ltx_text ltx_font_bold">Hung-Chieh Fang<sup id="id5.5.1.1" class="ltx_sup"><span id="id5.5.1.1.1" class="ltx_text ltx_font_medium">∗</span></sup> Chao-Wei Huang Yun-Nung Chen</span>

<br class="ltx_break">National Taiwan University, Taipei, Taiwan 
<br class="ltx_break"><span id="id10.10.id5" class="ltx_text ltx_font_typewriter">{r12922059, b08902072, b08902126, r09922a25}@csie.ntu.edu.tw
<br class="ltx_break">{b09902106, f07922069}@csie.ntu.edu.tw y.v.chen@ieee.org</span>
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id11.id1" class="ltx_p">Knowledge editing is a rising technique for efficiently updating factual knowledge in Large Language Models (LLMs) with minimal alteration of parameters. However, recent studies have identified concerning side effects, such as knowledge distortion and the deterioration of general abilities, that have emerged after editing. This survey presents a comprehensive study of these side effects, providing a unified view of the challenges associated with knowledge editing in LLMs. We discuss related works and summarize potential research directions to overcome these limitations. Our work highlights the limitations of current knowledge editing methods, emphasizing the need for deeper understanding of inner knowledge structures of LLMs and improved knowledge editing methods. To foster future research, we have released the complementary materials such as paper collection publicly<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span><a target="_blank" href="https://github.com/MiuLab/EditLLM-Survey" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/MiuLab/EditLLM-Survey</a></span></span></span>.
<span id="footnotex1" class="ltx_note ltx_role_footnotetext"><sup class="ltx_note_mark">*</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">*</sup><span class="ltx_note_type">footnotetext: </span>Equal contribution.</span></span></span></p>
</div>
<div id="p1" class="ltx_para ltx_noindent">
<div id="p1.5" class="ltx_block ltx_align_bottom">
<p id="p1.5.6" class="ltx_p"><span id="p1.5.6.1" class="ltx_text ltx_font_bold">Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models</span></p>
<br class="ltx_break ltx_centering">
<p id="p1.5.5" class="ltx_p ltx_align_center" style="width:433.6pt;"><span id="p1.5.5.5" class="ltx_text ltx_inline-block" style="width:0.0pt;">
<span id="p1.5.5.5.5" class="ltx_tabular ltx_align_top">
<span class="ltx_tbody">
<span id="p1.4.4.4.4.4" class="ltx_tr">
<span id="p1.4.4.4.4.4.4" class="ltx_td ltx_align_center"><span id="p1.4.4.4.4.4.4.4" class="ltx_text ltx_font_bold">Cheng-Hsun Hsueh<sup id="p1.4.4.4.4.4.4.4.1" class="ltx_sup"><span id="p1.4.4.4.4.4.4.4.1.1" class="ltx_text ltx_font_medium">∗</span></sup> Paul Kuo-Ming Huang<sup id="p1.4.4.4.4.4.4.4.2" class="ltx_sup"><span id="p1.4.4.4.4.4.4.4.2.1" class="ltx_text ltx_font_medium">∗</span></sup> Tzu-Han Lin<sup id="p1.4.4.4.4.4.4.4.3" class="ltx_sup"><span id="p1.4.4.4.4.4.4.4.3.1" class="ltx_text ltx_font_medium">∗</span></sup> Che-Wei Liao<sup id="p1.4.4.4.4.4.4.4.4" class="ltx_sup"><span id="p1.4.4.4.4.4.4.4.4.1" class="ltx_text ltx_font_medium">∗</span></sup></span></span></span>
<span id="p1.5.5.5.5.5" class="ltx_tr">
<span id="p1.5.5.5.5.5.1" class="ltx_td ltx_align_center"><span id="p1.5.5.5.5.5.1.1" class="ltx_text ltx_font_bold">Hung-Chieh Fang<sup id="p1.5.5.5.5.5.1.1.1" class="ltx_sup"><span id="p1.5.5.5.5.5.1.1.1.1" class="ltx_text ltx_font_medium">∗</span></sup> Chao-Wei Huang Yun-Nung Chen</span></span></span>
<span id="p1.5.5.5.5.6.1" class="ltx_tr">
<span id="p1.5.5.5.5.6.1.1" class="ltx_td ltx_align_center">National Taiwan University, Taipei, Taiwan</span></span>
<span id="p1.5.5.5.5.7.2" class="ltx_tr">
<span id="p1.5.5.5.5.7.2.1" class="ltx_td ltx_align_center"><span id="p1.5.5.5.5.7.2.1.1" class="ltx_text ltx_font_typewriter">{r12922059, b08902072, b08902126, r09922a25}@csie.ntu.edu.tw</span></span></span>
<span id="p1.5.5.5.5.8.3" class="ltx_tr">
<span id="p1.5.5.5.5.8.3.1" class="ltx_td ltx_align_center"><span id="p1.5.5.5.5.8.3.1.1" class="ltx_text ltx_font_typewriter">{b09902106, f07922069}@csie.ntu.edu.tw y.v.chen@ieee.org</span></span></span>
</span>
</span></span></p>
<br class="ltx_break ltx_centering">
</div>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Recent advancements in large language models (LLMs) have significantly improved Natural Language Processing <cite class="ltx_cite ltx_citemacro_citep">(NLP; Brown et al., <a href="#bib.bib4" title="" class="ltx_ref">2020</a>)</cite>. LLMs can now understand and generate language at a human-like level, establishing them as milestones in generative AI. This proficiency is due to their ability to extract knowledge from extensive text corpora. However, the mechanisms of knowledge storage in LLMs are not well understood, leading to potential issues in real-world applications. For instance, deploying LLMs as chatbots raises concerns about the reliability of their generated content due to the unclear mechanisms of knowledge storage and potential inaccuracies.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2406.01436/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="323" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>An overview of pitfalls in current knowledge editing methods. The subsequent sections dive into three key challenges: generalization issues (Section <a href="#S3.SS1" title="3.1 Inability to Logically Inference and Robustly Generalize ‣ 3 Challenges of Knowledge Editing ‣ Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.1</span></a>), localization issues (Section <a href="#S3.SS2" title="3.2 Unintended Alteration of Non-Target Knowledge ‣ 3 Challenges of Knowledge Editing ‣ Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a>), and deterioration of general LLM abilities (Section <a href="#S3.SS3" title="3.3 Deterioration of General LLM Abilities ‣ 3 Challenges of Knowledge Editing ‣ Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3</span></a>).</figcaption>
</figure>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">To address these issues, researchers have explored various methods. Traditional approaches like fine-tuning, continual learning, and retraining are either computationally expensive or detrimental to the LLMs’ overall performance. Recently, knowledge editing has emerged as a promising alternative, offering minimal computational costs and fewer alterations <cite class="ltx_cite ltx_citemacro_citep">(Cao et al., <a href="#bib.bib5" title="" class="ltx_ref">2021</a>; Dai et al., <a href="#bib.bib9" title="" class="ltx_ref">2022</a>; Meng et al., <a href="#bib.bib29" title="" class="ltx_ref">2022</a>, <a href="#bib.bib30" title="" class="ltx_ref">2023</a>; Dong et al., <a href="#bib.bib10" title="" class="ltx_ref">2022</a>; Mitchell et al., <a href="#bib.bib32" title="" class="ltx_ref">2022a</a>, <a href="#bib.bib33" title="" class="ltx_ref">b</a>; Hartvigsen et al., <a href="#bib.bib16" title="" class="ltx_ref">2023</a>; Huang et al., <a href="#bib.bib21" title="" class="ltx_ref">2023</a>; Yu et al., <a href="#bib.bib48" title="" class="ltx_ref">2024</a>; Zheng et al., <a href="#bib.bib51" title="" class="ltx_ref">2023</a>; Li et al., <a href="#bib.bib25" title="" class="ltx_ref">2024a</a>; Tan et al., <a href="#bib.bib36" title="" class="ltx_ref">2024</a>; Gupta et al., <a href="#bib.bib15" title="" class="ltx_ref">2024b</a>; Wang et al., <a href="#bib.bib43" title="" class="ltx_ref">2024</a>)</cite>. This approach allows for precise adjustments to LLMs, enhancing their practicality and reliability in real-world applications.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">Knowledge editing can be divided into two main categories: parameter-modifying and parameter-preserving. Both aim to refine LLM knowledge efficiently while avoiding the drawbacks of previous tuning methods <cite class="ltx_cite ltx_citemacro_citep">(Yao et al., <a href="#bib.bib46" title="" class="ltx_ref">2023a</a>)</cite>. Parameter-modifying methods, including meta-learning <cite class="ltx_cite ltx_citemacro_citep">(Cao et al., <a href="#bib.bib5" title="" class="ltx_ref">2021</a>; Mitchell et al., <a href="#bib.bib32" title="" class="ltx_ref">2022a</a>; Tan et al., <a href="#bib.bib36" title="" class="ltx_ref">2024</a>)</cite> and locate-then-edit techniques <cite class="ltx_cite ltx_citemacro_citep">(Dai et al., <a href="#bib.bib9" title="" class="ltx_ref">2022</a>; Meng et al., <a href="#bib.bib29" title="" class="ltx_ref">2022</a>, <a href="#bib.bib30" title="" class="ltx_ref">2023</a>; Li et al., <a href="#bib.bib25" title="" class="ltx_ref">2024a</a>; Gupta et al., <a href="#bib.bib15" title="" class="ltx_ref">2024b</a>)</cite>, strive to update a targeted set of model parameters while making effective edits. Parameter-preserving methods introduce external components, like knowledge bases <cite class="ltx_cite ltx_citemacro_citep">(Mitchell et al., <a href="#bib.bib33" title="" class="ltx_ref">2022b</a>; Yu et al., <a href="#bib.bib48" title="" class="ltx_ref">2024</a>)</cite> or extra model parameters <cite class="ltx_cite ltx_citemacro_citep">(Dong et al., <a href="#bib.bib10" title="" class="ltx_ref">2022</a>; Huang et al., <a href="#bib.bib21" title="" class="ltx_ref">2023</a>; Hartvigsen et al., <a href="#bib.bib16" title="" class="ltx_ref">2023</a>; Yu et al., <a href="#bib.bib48" title="" class="ltx_ref">2024</a>)</cite> to maintain the integrity of pre-trained LLMs while updating their knowledge.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">Despite the success of knowledge editing, challenges remain. Knowledge editing can have unintended side effects, potentially damaging the general abilities and intrinsic structures of LLMs. Previous research has mainly focused on performance improvements and innovations within knowledge editing methods, with limited attention to potential drawbacks. Consequently, this survey aims to provide a holistic view of current issues in the knowledge editing paradigm and encourage further investigations into the pitfalls and intrinsic knowledge structures of LLMs. A brief overview of the discussed pitfalls is shown in Figure  <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.</p>
</div>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.1" class="ltx_p">This survey is organized as follows: Section <a href="#S2" title="2 Overview of Knowledge Editing ‣ Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> introduces knowledge editing and its methods. Section <a href="#S3" title="3 Challenges of Knowledge Editing ‣ Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> discusses current challenges, benchmarks, and methodologies. In Section <a href="#S4" title="4 Experiments ‣ Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, we present experimental results evaluating different editing methods. Finally, Section <a href="#S5" title="5 Future Prospects ‣ Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> explores related studies and future research directions. We summarize our contributions as follows:</p>
<ol id="S1.I1" class="ltx_enumerate">
<li id="S1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S1.I1.i1.p1" class="ltx_para">
<p id="S1.I1.i1.p1.1" class="ltx_p">We are the first to provide a comprehensive analysis of the side effects associated with existing knowledge editing techniques.</p>
</div>
</li>
<li id="S1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S1.I1.i2.p1" class="ltx_para">
<p id="S1.I1.i2.p1.1" class="ltx_p">We systematically organized previous research and conducted experiments to benchmark the side effects of knowledge editing, providing a unified perspective on this issue.</p>
</div>
</li>
<li id="S1.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">3.</span> 
<div id="S1.I1.i3.p1" class="ltx_para">
<p id="S1.I1.i3.p1.1" class="ltx_p">We discussed related studies and potential research directions to address existing challenges, encouraging further exploration and understanding in this field.</p>
</div>
</li>
</ol>
</div>
<figure id="S1.F2" class="ltx_figure"><img src="/html/2406.01436/assets/x2.png" id="S1.F2.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="563" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>Illustration of properties that knowledge editing methods should satisfy. An ideal knowledge editing method should be reliable, generalizable to relevant queries, and should not alter the outputs of irrelevant queries.</figcaption>
</figure>
<figure id="S1.F3" class="ltx_figure ltx_align_center"><img src="/html/2406.01436/assets/x3.png" id="S1.F3.1.g1" class="ltx_graphics ltx_img_square" width="461" height="415" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>Illustration of the two categories of model editing methods in transformer-based large language models, which includes parameter-modifying (meta-learning and locate-and-edit) and parameter-preserving (additional parameters, external memory, in-context learning, and decoding) methods. MHSA and FFN stand for multi-head self-attention and feed-forward network, respectively.</figcaption>
</figure>
<figure id="S1.T1" class="ltx_table">
<table id="S1.T1.7" class="ltx_tabular ltx_centering ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S1.T1.7.8.1" class="ltx_tr">
<td id="S1.T1.7.8.1.1" class="ltx_td ltx_align_center ltx_border_tt">Category</td>
<td id="S1.T1.7.8.1.2" class="ltx_td ltx_align_center ltx_border_tt">Strategy</td>
<td id="S1.T1.7.8.1.3" class="ltx_td ltx_align_left ltx_border_tt">Method</td>
</tr>
<tr id="S1.T1.7.9.2" class="ltx_tr">
<td id="S1.T1.7.9.2.1" class="ltx_td ltx_align_center ltx_border_t" rowspan="8"><span id="S1.T1.7.9.2.1.1" class="ltx_text">
<span id="S1.T1.7.9.2.1.1.1" class="ltx_inline-block">
<span id="S1.T1.7.9.2.1.1.1.1" class="ltx_p">Parameter-</span>
<span id="S1.T1.7.9.2.1.1.1.2" class="ltx_p">modifying</span>
</span></span></td>
<td id="S1.T1.7.9.2.2" class="ltx_td ltx_align_center ltx_border_t" rowspan="3"><span id="S1.T1.7.9.2.2.1" class="ltx_text">
<span id="S1.T1.7.9.2.2.1.1" class="ltx_inline-block">
<span id="S1.T1.7.9.2.2.1.1.1" class="ltx_p">Meta-</span>
<span id="S1.T1.7.9.2.2.1.1.2" class="ltx_p">learning</span>
</span></span></td>
<td id="S1.T1.7.9.2.3" class="ltx_td ltx_align_left ltx_border_t">Knowledge Editor <cite class="ltx_cite ltx_citemacro_citep">(Cao et al., <a href="#bib.bib5" title="" class="ltx_ref">2021</a>)</cite>
</td>
</tr>
<tr id="S1.T1.7.10.3" class="ltx_tr">
<td id="S1.T1.7.10.3.1" class="ltx_td ltx_align_left">MEND <cite class="ltx_cite ltx_citemacro_citep">(Mitchell et al., <a href="#bib.bib32" title="" class="ltx_ref">2022a</a>)</cite>
</td>
</tr>
<tr id="S1.T1.7.11.4" class="ltx_tr">
<td id="S1.T1.7.11.4.1" class="ltx_td ltx_align_left">MALMEN <cite class="ltx_cite ltx_citemacro_citep">(Tan et al., <a href="#bib.bib36" title="" class="ltx_ref">2024</a>)</cite>
</td>
</tr>
<tr id="S1.T1.7.12.5" class="ltx_tr">
<td id="S1.T1.7.12.5.1" class="ltx_td ltx_align_center ltx_border_t" rowspan="5"><span id="S1.T1.7.12.5.1.1" class="ltx_text">
<span id="S1.T1.7.12.5.1.1.1" class="ltx_inline-block">
<span id="S1.T1.7.12.5.1.1.1.1" class="ltx_p">Locating</span>
<span id="S1.T1.7.12.5.1.1.1.2" class="ltx_p">and</span>
<span id="S1.T1.7.12.5.1.1.1.3" class="ltx_p">editing</span>
</span></span></td>
<td id="S1.T1.7.12.5.2" class="ltx_td ltx_align_left ltx_border_t">Knowledge Neuron <cite class="ltx_cite ltx_citemacro_citep">(Dai et al., <a href="#bib.bib9" title="" class="ltx_ref">2022</a>)</cite>
</td>
</tr>
<tr id="S1.T1.7.13.6" class="ltx_tr">
<td id="S1.T1.7.13.6.1" class="ltx_td ltx_align_left">ROME <cite class="ltx_cite ltx_citemacro_citep">(Meng et al., <a href="#bib.bib29" title="" class="ltx_ref">2022</a>)</cite>
</td>
</tr>
<tr id="S1.T1.7.14.7" class="ltx_tr">
<td id="S1.T1.7.14.7.1" class="ltx_td ltx_align_left">MEMIT <cite class="ltx_cite ltx_citemacro_citep">(Meng et al., <a href="#bib.bib30" title="" class="ltx_ref">2023</a>)</cite>
</td>
</tr>
<tr id="S1.T1.7.15.8" class="ltx_tr">
<td id="S1.T1.7.15.8.1" class="ltx_td ltx_align_left">PMET <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib25" title="" class="ltx_ref">2024a</a>)</cite>
</td>
</tr>
<tr id="S1.T1.7.16.9" class="ltx_tr">
<td id="S1.T1.7.16.9.1" class="ltx_td ltx_align_left">EMMET <cite class="ltx_cite ltx_citemacro_citep">(Gupta et al., <a href="#bib.bib15" title="" class="ltx_ref">2024b</a>)</cite>
</td>
</tr>
<tr id="S1.T1.7.17.10" class="ltx_tr">
<td id="S1.T1.7.17.10.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" rowspan="9"><span id="S1.T1.7.17.10.1.1" class="ltx_text">
<span id="S1.T1.7.17.10.1.1.1" class="ltx_inline-block">
<span id="S1.T1.7.17.10.1.1.1.1" class="ltx_p">Parameter-</span>
<span id="S1.T1.7.17.10.1.1.1.2" class="ltx_p">preserving</span>
</span></span></td>
<td id="S1.T1.7.17.10.2" class="ltx_td ltx_align_center ltx_border_t" rowspan="4"><span id="S1.T1.7.17.10.2.1" class="ltx_text">
<span id="S1.T1.7.17.10.2.1.1" class="ltx_inline-block">
<span id="S1.T1.7.17.10.2.1.1.1" class="ltx_p">Additional</span>
<span id="S1.T1.7.17.10.2.1.1.2" class="ltx_p">parameters</span>
</span></span></td>
<td id="S1.T1.7.17.10.3" class="ltx_td ltx_align_left ltx_border_t">CaliNET <cite class="ltx_cite ltx_citemacro_citep">(Dong et al., <a href="#bib.bib10" title="" class="ltx_ref">2022</a>)</cite>
</td>
</tr>
<tr id="S1.T1.1.1" class="ltx_tr">
<td id="S1.T1.1.1.1" class="ltx_td ltx_align_left">T-Patcher<sup id="S1.T1.1.1.1.1" class="ltx_sup">†</sup> <cite class="ltx_cite ltx_citemacro_citep">(Huang et al., <a href="#bib.bib21" title="" class="ltx_ref">2023</a>)</cite>
</td>
</tr>
<tr id="S1.T1.2.2" class="ltx_tr">
<td id="S1.T1.2.2.1" class="ltx_td ltx_align_left">GRACE<sup id="S1.T1.2.2.1.1" class="ltx_sup">†</sup> <cite class="ltx_cite ltx_citemacro_citep">(Hartvigsen et al., <a href="#bib.bib16" title="" class="ltx_ref">2023</a>)</cite>
</td>
</tr>
<tr id="S1.T1.3.3" class="ltx_tr">
<td id="S1.T1.3.3.1" class="ltx_td ltx_align_left">MELO<sup id="S1.T1.3.3.1.1" class="ltx_sup">†</sup> <cite class="ltx_cite ltx_citemacro_citep">(Yu et al., <a href="#bib.bib48" title="" class="ltx_ref">2024</a>)</cite>
</td>
</tr>
<tr id="S1.T1.4.4" class="ltx_tr">
<td id="S1.T1.4.4.2" class="ltx_td ltx_align_center ltx_border_t" rowspan="2"><span id="S1.T1.4.4.2.1" class="ltx_text">
<span id="S1.T1.4.4.2.1.1" class="ltx_inline-block">
<span id="S1.T1.4.4.2.1.1.1" class="ltx_p">External</span>
<span id="S1.T1.4.4.2.1.1.2" class="ltx_p">memory</span>
</span></span></td>
<td id="S1.T1.4.4.1" class="ltx_td ltx_align_left ltx_border_t">SERAC<sup id="S1.T1.4.4.1.1" class="ltx_sup">†</sup> <cite class="ltx_cite ltx_citemacro_citep">(Mitchell et al., <a href="#bib.bib33" title="" class="ltx_ref">2022b</a>)</cite>
</td>
</tr>
<tr id="S1.T1.5.5" class="ltx_tr">
<td id="S1.T1.5.5.1" class="ltx_td ltx_align_left">MeLLo<sup id="S1.T1.5.5.1.1" class="ltx_sup">†</sup> <cite class="ltx_cite ltx_citemacro_citep">(Zhong et al., <a href="#bib.bib52" title="" class="ltx_ref">2023</a>)</cite>
</td>
</tr>
<tr id="S1.T1.6.6" class="ltx_tr">
<td id="S1.T1.6.6.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S1.T1.6.6.2.1" class="ltx_text">
<span id="S1.T1.6.6.2.1.1" class="ltx_inline-block">
<span id="S1.T1.6.6.2.1.1.1" class="ltx_p">In-context</span>
<span id="S1.T1.6.6.2.1.1.2" class="ltx_p">learning</span>
</span></span></td>
<td id="S1.T1.6.6.1" class="ltx_td ltx_align_left ltx_border_t"><span id="S1.T1.6.6.1.1" class="ltx_text">IKE<sup id="S1.T1.6.6.1.1.1" class="ltx_sup">†</sup> <cite class="ltx_cite ltx_citemacro_citep">(Zheng et al., <a href="#bib.bib51" title="" class="ltx_ref">2023</a>)</cite></span></td>
</tr>
<tr id="S1.T1.7.7" class="ltx_tr">
<td id="S1.T1.7.7.2" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">Decoding</td>
<td id="S1.T1.7.7.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t">DeepEdit<sup id="S1.T1.7.7.1.1" class="ltx_sup">†</sup> <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib43" title="" class="ltx_ref">2024</a>)</cite>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>Overview of current knowledge editing methods. The methods are categorized into two major families, namely parameter-modifying and parameter-preserving methods, each containing several strategies. Methods marked with <math id="S1.T1.9.m1.1" class="ltx_Math" alttext="\dagger" display="inline"><semantics id="S1.T1.9.m1.1b"><mo id="S1.T1.9.m1.1.1" xref="S1.T1.9.m1.1.1.cmml">†</mo><annotation-xml encoding="MathML-Content" id="S1.T1.9.m1.1c"><ci id="S1.T1.9.m1.1.1.cmml" xref="S1.T1.9.m1.1.1">†</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.T1.9.m1.1d">\dagger</annotation></semantics></math> have the ability to process sequential edits.</figcaption>
</figure>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Overview of Knowledge Editing</h2>

<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>Problem Definition</h3>

<div id="S2.SS1.p1" class="ltx_para">
<p id="S2.SS1.p1.1" class="ltx_p">Knowledge editing for LLMs entails modifying the output of LLMs in response to specific edit queries, with the aim of minimizing alterations to their original behavior <cite class="ltx_cite ltx_citemacro_citep">(Yao et al., <a href="#bib.bib46" title="" class="ltx_ref">2023a</a>; Mazzia et al., <a href="#bib.bib28" title="" class="ltx_ref">2023</a>; Zhang et al., <a href="#bib.bib49" title="" class="ltx_ref">2024a</a>)</cite>. In this section, we follow the notation from <cite class="ltx_cite ltx_citemacro_citet">Mazzia et al. (<a href="#bib.bib28" title="" class="ltx_ref">2023</a>)</cite>.</p>
</div>
<div id="S2.SS1.p2" class="ltx_para">
<p id="S2.SS1.p2.7" class="ltx_p">We denote the input and output space as <math id="S2.SS1.p2.1.m1.1" class="ltx_Math" alttext="\mathbb{X}" display="inline"><semantics id="S2.SS1.p2.1.m1.1a"><mi id="S2.SS1.p2.1.m1.1.1" xref="S2.SS1.p2.1.m1.1.1.cmml">𝕏</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.1.m1.1b"><ci id="S2.SS1.p2.1.m1.1.1.cmml" xref="S2.SS1.p2.1.m1.1.1">𝕏</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.1.m1.1c">\mathbb{X}</annotation></semantics></math> and <math id="S2.SS1.p2.2.m2.1" class="ltx_Math" alttext="\mathbb{Y}" display="inline"><semantics id="S2.SS1.p2.2.m2.1a"><mi id="S2.SS1.p2.2.m2.1.1" xref="S2.SS1.p2.2.m2.1.1.cmml">𝕐</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.2.m2.1b"><ci id="S2.SS1.p2.2.m2.1.1.cmml" xref="S2.SS1.p2.2.m2.1.1">𝕐</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.2.m2.1c">\mathbb{Y}</annotation></semantics></math>, respectively. The function space <math id="S2.SS1.p2.3.m3.1" class="ltx_Math" alttext="\mathbb{F}:\mathbb{X}\rightarrow\mathbb{Y}" display="inline"><semantics id="S2.SS1.p2.3.m3.1a"><mrow id="S2.SS1.p2.3.m3.1.1" xref="S2.SS1.p2.3.m3.1.1.cmml"><mi id="S2.SS1.p2.3.m3.1.1.2" xref="S2.SS1.p2.3.m3.1.1.2.cmml">𝔽</mi><mo lspace="0.278em" rspace="0.278em" id="S2.SS1.p2.3.m3.1.1.1" xref="S2.SS1.p2.3.m3.1.1.1.cmml">:</mo><mrow id="S2.SS1.p2.3.m3.1.1.3" xref="S2.SS1.p2.3.m3.1.1.3.cmml"><mi id="S2.SS1.p2.3.m3.1.1.3.2" xref="S2.SS1.p2.3.m3.1.1.3.2.cmml">𝕏</mi><mo stretchy="false" id="S2.SS1.p2.3.m3.1.1.3.1" xref="S2.SS1.p2.3.m3.1.1.3.1.cmml">→</mo><mi id="S2.SS1.p2.3.m3.1.1.3.3" xref="S2.SS1.p2.3.m3.1.1.3.3.cmml">𝕐</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.3.m3.1b"><apply id="S2.SS1.p2.3.m3.1.1.cmml" xref="S2.SS1.p2.3.m3.1.1"><ci id="S2.SS1.p2.3.m3.1.1.1.cmml" xref="S2.SS1.p2.3.m3.1.1.1">:</ci><ci id="S2.SS1.p2.3.m3.1.1.2.cmml" xref="S2.SS1.p2.3.m3.1.1.2">𝔽</ci><apply id="S2.SS1.p2.3.m3.1.1.3.cmml" xref="S2.SS1.p2.3.m3.1.1.3"><ci id="S2.SS1.p2.3.m3.1.1.3.1.cmml" xref="S2.SS1.p2.3.m3.1.1.3.1">→</ci><ci id="S2.SS1.p2.3.m3.1.1.3.2.cmml" xref="S2.SS1.p2.3.m3.1.1.3.2">𝕏</ci><ci id="S2.SS1.p2.3.m3.1.1.3.3.cmml" xref="S2.SS1.p2.3.m3.1.1.3.3">𝕐</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.3.m3.1c">\mathbb{F}:\mathbb{X}\rightarrow\mathbb{Y}</annotation></semantics></math> is estimated by the base model <math id="S2.SS1.p2.4.m4.1" class="ltx_Math" alttext="f_{\theta_{0}}" display="inline"><semantics id="S2.SS1.p2.4.m4.1a"><msub id="S2.SS1.p2.4.m4.1.1" xref="S2.SS1.p2.4.m4.1.1.cmml"><mi id="S2.SS1.p2.4.m4.1.1.2" xref="S2.SS1.p2.4.m4.1.1.2.cmml">f</mi><msub id="S2.SS1.p2.4.m4.1.1.3" xref="S2.SS1.p2.4.m4.1.1.3.cmml"><mi id="S2.SS1.p2.4.m4.1.1.3.2" xref="S2.SS1.p2.4.m4.1.1.3.2.cmml">θ</mi><mn id="S2.SS1.p2.4.m4.1.1.3.3" xref="S2.SS1.p2.4.m4.1.1.3.3.cmml">0</mn></msub></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.4.m4.1b"><apply id="S2.SS1.p2.4.m4.1.1.cmml" xref="S2.SS1.p2.4.m4.1.1"><csymbol cd="ambiguous" id="S2.SS1.p2.4.m4.1.1.1.cmml" xref="S2.SS1.p2.4.m4.1.1">subscript</csymbol><ci id="S2.SS1.p2.4.m4.1.1.2.cmml" xref="S2.SS1.p2.4.m4.1.1.2">𝑓</ci><apply id="S2.SS1.p2.4.m4.1.1.3.cmml" xref="S2.SS1.p2.4.m4.1.1.3"><csymbol cd="ambiguous" id="S2.SS1.p2.4.m4.1.1.3.1.cmml" xref="S2.SS1.p2.4.m4.1.1.3">subscript</csymbol><ci id="S2.SS1.p2.4.m4.1.1.3.2.cmml" xref="S2.SS1.p2.4.m4.1.1.3.2">𝜃</ci><cn type="integer" id="S2.SS1.p2.4.m4.1.1.3.3.cmml" xref="S2.SS1.p2.4.m4.1.1.3.3">0</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.4.m4.1c">f_{\theta_{0}}</annotation></semantics></math> parameterized by <math id="S2.SS1.p2.5.m5.1" class="ltx_Math" alttext="\theta_{0}\in\Theta" display="inline"><semantics id="S2.SS1.p2.5.m5.1a"><mrow id="S2.SS1.p2.5.m5.1.1" xref="S2.SS1.p2.5.m5.1.1.cmml"><msub id="S2.SS1.p2.5.m5.1.1.2" xref="S2.SS1.p2.5.m5.1.1.2.cmml"><mi id="S2.SS1.p2.5.m5.1.1.2.2" xref="S2.SS1.p2.5.m5.1.1.2.2.cmml">θ</mi><mn id="S2.SS1.p2.5.m5.1.1.2.3" xref="S2.SS1.p2.5.m5.1.1.2.3.cmml">0</mn></msub><mo id="S2.SS1.p2.5.m5.1.1.1" xref="S2.SS1.p2.5.m5.1.1.1.cmml">∈</mo><mi mathvariant="normal" id="S2.SS1.p2.5.m5.1.1.3" xref="S2.SS1.p2.5.m5.1.1.3.cmml">Θ</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.5.m5.1b"><apply id="S2.SS1.p2.5.m5.1.1.cmml" xref="S2.SS1.p2.5.m5.1.1"><in id="S2.SS1.p2.5.m5.1.1.1.cmml" xref="S2.SS1.p2.5.m5.1.1.1"></in><apply id="S2.SS1.p2.5.m5.1.1.2.cmml" xref="S2.SS1.p2.5.m5.1.1.2"><csymbol cd="ambiguous" id="S2.SS1.p2.5.m5.1.1.2.1.cmml" xref="S2.SS1.p2.5.m5.1.1.2">subscript</csymbol><ci id="S2.SS1.p2.5.m5.1.1.2.2.cmml" xref="S2.SS1.p2.5.m5.1.1.2.2">𝜃</ci><cn type="integer" id="S2.SS1.p2.5.m5.1.1.2.3.cmml" xref="S2.SS1.p2.5.m5.1.1.2.3">0</cn></apply><ci id="S2.SS1.p2.5.m5.1.1.3.cmml" xref="S2.SS1.p2.5.m5.1.1.3">Θ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.5.m5.1c">\theta_{0}\in\Theta</annotation></semantics></math>. Finally, let <math id="S2.SS1.p2.6.m6.2" class="ltx_Math" alttext="Z_{e}=\{(x_{e},y_{e})\ |\ f_{\theta_{0}}(x_{e})\neq y_{e}\}" display="inline"><semantics id="S2.SS1.p2.6.m6.2a"><mrow id="S2.SS1.p2.6.m6.2.2" xref="S2.SS1.p2.6.m6.2.2.cmml"><msub id="S2.SS1.p2.6.m6.2.2.4" xref="S2.SS1.p2.6.m6.2.2.4.cmml"><mi id="S2.SS1.p2.6.m6.2.2.4.2" xref="S2.SS1.p2.6.m6.2.2.4.2.cmml">Z</mi><mi id="S2.SS1.p2.6.m6.2.2.4.3" xref="S2.SS1.p2.6.m6.2.2.4.3.cmml">e</mi></msub><mo id="S2.SS1.p2.6.m6.2.2.3" xref="S2.SS1.p2.6.m6.2.2.3.cmml">=</mo><mrow id="S2.SS1.p2.6.m6.2.2.2.2" xref="S2.SS1.p2.6.m6.2.2.2.3.cmml"><mo stretchy="false" id="S2.SS1.p2.6.m6.2.2.2.2.3" xref="S2.SS1.p2.6.m6.2.2.2.3.1.cmml">{</mo><mrow id="S2.SS1.p2.6.m6.1.1.1.1.1.2" xref="S2.SS1.p2.6.m6.1.1.1.1.1.3.cmml"><mo stretchy="false" id="S2.SS1.p2.6.m6.1.1.1.1.1.2.3" xref="S2.SS1.p2.6.m6.1.1.1.1.1.3.cmml">(</mo><msub id="S2.SS1.p2.6.m6.1.1.1.1.1.1.1" xref="S2.SS1.p2.6.m6.1.1.1.1.1.1.1.cmml"><mi id="S2.SS1.p2.6.m6.1.1.1.1.1.1.1.2" xref="S2.SS1.p2.6.m6.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="S2.SS1.p2.6.m6.1.1.1.1.1.1.1.3" xref="S2.SS1.p2.6.m6.1.1.1.1.1.1.1.3.cmml">e</mi></msub><mo id="S2.SS1.p2.6.m6.1.1.1.1.1.2.4" xref="S2.SS1.p2.6.m6.1.1.1.1.1.3.cmml">,</mo><msub id="S2.SS1.p2.6.m6.1.1.1.1.1.2.2" xref="S2.SS1.p2.6.m6.1.1.1.1.1.2.2.cmml"><mi id="S2.SS1.p2.6.m6.1.1.1.1.1.2.2.2" xref="S2.SS1.p2.6.m6.1.1.1.1.1.2.2.2.cmml">y</mi><mi id="S2.SS1.p2.6.m6.1.1.1.1.1.2.2.3" xref="S2.SS1.p2.6.m6.1.1.1.1.1.2.2.3.cmml">e</mi></msub><mo rspace="0.222em" stretchy="false" id="S2.SS1.p2.6.m6.1.1.1.1.1.2.5" xref="S2.SS1.p2.6.m6.1.1.1.1.1.3.cmml">)</mo></mrow><mo rspace="0.500em" id="S2.SS1.p2.6.m6.2.2.2.2.4" xref="S2.SS1.p2.6.m6.2.2.2.3.1.cmml">|</mo><mrow id="S2.SS1.p2.6.m6.2.2.2.2.2" xref="S2.SS1.p2.6.m6.2.2.2.2.2.cmml"><mrow id="S2.SS1.p2.6.m6.2.2.2.2.2.1" xref="S2.SS1.p2.6.m6.2.2.2.2.2.1.cmml"><msub id="S2.SS1.p2.6.m6.2.2.2.2.2.1.3" xref="S2.SS1.p2.6.m6.2.2.2.2.2.1.3.cmml"><mi id="S2.SS1.p2.6.m6.2.2.2.2.2.1.3.2" xref="S2.SS1.p2.6.m6.2.2.2.2.2.1.3.2.cmml">f</mi><msub id="S2.SS1.p2.6.m6.2.2.2.2.2.1.3.3" xref="S2.SS1.p2.6.m6.2.2.2.2.2.1.3.3.cmml"><mi id="S2.SS1.p2.6.m6.2.2.2.2.2.1.3.3.2" xref="S2.SS1.p2.6.m6.2.2.2.2.2.1.3.3.2.cmml">θ</mi><mn id="S2.SS1.p2.6.m6.2.2.2.2.2.1.3.3.3" xref="S2.SS1.p2.6.m6.2.2.2.2.2.1.3.3.3.cmml">0</mn></msub></msub><mo lspace="0em" rspace="0em" id="S2.SS1.p2.6.m6.2.2.2.2.2.1.2" xref="S2.SS1.p2.6.m6.2.2.2.2.2.1.2.cmml">​</mo><mrow id="S2.SS1.p2.6.m6.2.2.2.2.2.1.1.1" xref="S2.SS1.p2.6.m6.2.2.2.2.2.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS1.p2.6.m6.2.2.2.2.2.1.1.1.2" xref="S2.SS1.p2.6.m6.2.2.2.2.2.1.1.1.1.cmml">(</mo><msub id="S2.SS1.p2.6.m6.2.2.2.2.2.1.1.1.1" xref="S2.SS1.p2.6.m6.2.2.2.2.2.1.1.1.1.cmml"><mi id="S2.SS1.p2.6.m6.2.2.2.2.2.1.1.1.1.2" xref="S2.SS1.p2.6.m6.2.2.2.2.2.1.1.1.1.2.cmml">x</mi><mi id="S2.SS1.p2.6.m6.2.2.2.2.2.1.1.1.1.3" xref="S2.SS1.p2.6.m6.2.2.2.2.2.1.1.1.1.3.cmml">e</mi></msub><mo stretchy="false" id="S2.SS1.p2.6.m6.2.2.2.2.2.1.1.1.3" xref="S2.SS1.p2.6.m6.2.2.2.2.2.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S2.SS1.p2.6.m6.2.2.2.2.2.2" xref="S2.SS1.p2.6.m6.2.2.2.2.2.2.cmml">≠</mo><msub id="S2.SS1.p2.6.m6.2.2.2.2.2.3" xref="S2.SS1.p2.6.m6.2.2.2.2.2.3.cmml"><mi id="S2.SS1.p2.6.m6.2.2.2.2.2.3.2" xref="S2.SS1.p2.6.m6.2.2.2.2.2.3.2.cmml">y</mi><mi id="S2.SS1.p2.6.m6.2.2.2.2.2.3.3" xref="S2.SS1.p2.6.m6.2.2.2.2.2.3.3.cmml">e</mi></msub></mrow><mo stretchy="false" id="S2.SS1.p2.6.m6.2.2.2.2.5" xref="S2.SS1.p2.6.m6.2.2.2.3.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.6.m6.2b"><apply id="S2.SS1.p2.6.m6.2.2.cmml" xref="S2.SS1.p2.6.m6.2.2"><eq id="S2.SS1.p2.6.m6.2.2.3.cmml" xref="S2.SS1.p2.6.m6.2.2.3"></eq><apply id="S2.SS1.p2.6.m6.2.2.4.cmml" xref="S2.SS1.p2.6.m6.2.2.4"><csymbol cd="ambiguous" id="S2.SS1.p2.6.m6.2.2.4.1.cmml" xref="S2.SS1.p2.6.m6.2.2.4">subscript</csymbol><ci id="S2.SS1.p2.6.m6.2.2.4.2.cmml" xref="S2.SS1.p2.6.m6.2.2.4.2">𝑍</ci><ci id="S2.SS1.p2.6.m6.2.2.4.3.cmml" xref="S2.SS1.p2.6.m6.2.2.4.3">𝑒</ci></apply><apply id="S2.SS1.p2.6.m6.2.2.2.3.cmml" xref="S2.SS1.p2.6.m6.2.2.2.2"><csymbol cd="latexml" id="S2.SS1.p2.6.m6.2.2.2.3.1.cmml" xref="S2.SS1.p2.6.m6.2.2.2.2.3">conditional-set</csymbol><interval closure="open" id="S2.SS1.p2.6.m6.1.1.1.1.1.3.cmml" xref="S2.SS1.p2.6.m6.1.1.1.1.1.2"><apply id="S2.SS1.p2.6.m6.1.1.1.1.1.1.1.cmml" xref="S2.SS1.p2.6.m6.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS1.p2.6.m6.1.1.1.1.1.1.1.1.cmml" xref="S2.SS1.p2.6.m6.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS1.p2.6.m6.1.1.1.1.1.1.1.2.cmml" xref="S2.SS1.p2.6.m6.1.1.1.1.1.1.1.2">𝑥</ci><ci id="S2.SS1.p2.6.m6.1.1.1.1.1.1.1.3.cmml" xref="S2.SS1.p2.6.m6.1.1.1.1.1.1.1.3">𝑒</ci></apply><apply id="S2.SS1.p2.6.m6.1.1.1.1.1.2.2.cmml" xref="S2.SS1.p2.6.m6.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S2.SS1.p2.6.m6.1.1.1.1.1.2.2.1.cmml" xref="S2.SS1.p2.6.m6.1.1.1.1.1.2.2">subscript</csymbol><ci id="S2.SS1.p2.6.m6.1.1.1.1.1.2.2.2.cmml" xref="S2.SS1.p2.6.m6.1.1.1.1.1.2.2.2">𝑦</ci><ci id="S2.SS1.p2.6.m6.1.1.1.1.1.2.2.3.cmml" xref="S2.SS1.p2.6.m6.1.1.1.1.1.2.2.3">𝑒</ci></apply></interval><apply id="S2.SS1.p2.6.m6.2.2.2.2.2.cmml" xref="S2.SS1.p2.6.m6.2.2.2.2.2"><neq id="S2.SS1.p2.6.m6.2.2.2.2.2.2.cmml" xref="S2.SS1.p2.6.m6.2.2.2.2.2.2"></neq><apply id="S2.SS1.p2.6.m6.2.2.2.2.2.1.cmml" xref="S2.SS1.p2.6.m6.2.2.2.2.2.1"><times id="S2.SS1.p2.6.m6.2.2.2.2.2.1.2.cmml" xref="S2.SS1.p2.6.m6.2.2.2.2.2.1.2"></times><apply id="S2.SS1.p2.6.m6.2.2.2.2.2.1.3.cmml" xref="S2.SS1.p2.6.m6.2.2.2.2.2.1.3"><csymbol cd="ambiguous" id="S2.SS1.p2.6.m6.2.2.2.2.2.1.3.1.cmml" xref="S2.SS1.p2.6.m6.2.2.2.2.2.1.3">subscript</csymbol><ci id="S2.SS1.p2.6.m6.2.2.2.2.2.1.3.2.cmml" xref="S2.SS1.p2.6.m6.2.2.2.2.2.1.3.2">𝑓</ci><apply id="S2.SS1.p2.6.m6.2.2.2.2.2.1.3.3.cmml" xref="S2.SS1.p2.6.m6.2.2.2.2.2.1.3.3"><csymbol cd="ambiguous" id="S2.SS1.p2.6.m6.2.2.2.2.2.1.3.3.1.cmml" xref="S2.SS1.p2.6.m6.2.2.2.2.2.1.3.3">subscript</csymbol><ci id="S2.SS1.p2.6.m6.2.2.2.2.2.1.3.3.2.cmml" xref="S2.SS1.p2.6.m6.2.2.2.2.2.1.3.3.2">𝜃</ci><cn type="integer" id="S2.SS1.p2.6.m6.2.2.2.2.2.1.3.3.3.cmml" xref="S2.SS1.p2.6.m6.2.2.2.2.2.1.3.3.3">0</cn></apply></apply><apply id="S2.SS1.p2.6.m6.2.2.2.2.2.1.1.1.1.cmml" xref="S2.SS1.p2.6.m6.2.2.2.2.2.1.1.1"><csymbol cd="ambiguous" id="S2.SS1.p2.6.m6.2.2.2.2.2.1.1.1.1.1.cmml" xref="S2.SS1.p2.6.m6.2.2.2.2.2.1.1.1">subscript</csymbol><ci id="S2.SS1.p2.6.m6.2.2.2.2.2.1.1.1.1.2.cmml" xref="S2.SS1.p2.6.m6.2.2.2.2.2.1.1.1.1.2">𝑥</ci><ci id="S2.SS1.p2.6.m6.2.2.2.2.2.1.1.1.1.3.cmml" xref="S2.SS1.p2.6.m6.2.2.2.2.2.1.1.1.1.3">𝑒</ci></apply></apply><apply id="S2.SS1.p2.6.m6.2.2.2.2.2.3.cmml" xref="S2.SS1.p2.6.m6.2.2.2.2.2.3"><csymbol cd="ambiguous" id="S2.SS1.p2.6.m6.2.2.2.2.2.3.1.cmml" xref="S2.SS1.p2.6.m6.2.2.2.2.2.3">subscript</csymbol><ci id="S2.SS1.p2.6.m6.2.2.2.2.2.3.2.cmml" xref="S2.SS1.p2.6.m6.2.2.2.2.2.3.2">𝑦</ci><ci id="S2.SS1.p2.6.m6.2.2.2.2.2.3.3.cmml" xref="S2.SS1.p2.6.m6.2.2.2.2.2.3.3">𝑒</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.6.m6.2c">Z_{e}=\{(x_{e},y_{e})\ |\ f_{\theta_{0}}(x_{e})\neq y_{e}\}</annotation></semantics></math> be the set of edit queries we would like to apply to the base model.
The goal of knowledge editing is to efficiently derive the edited model <math id="S2.SS1.p2.7.m7.1" class="ltx_Math" alttext="f_{\theta_{e}}" display="inline"><semantics id="S2.SS1.p2.7.m7.1a"><msub id="S2.SS1.p2.7.m7.1.1" xref="S2.SS1.p2.7.m7.1.1.cmml"><mi id="S2.SS1.p2.7.m7.1.1.2" xref="S2.SS1.p2.7.m7.1.1.2.cmml">f</mi><msub id="S2.SS1.p2.7.m7.1.1.3" xref="S2.SS1.p2.7.m7.1.1.3.cmml"><mi id="S2.SS1.p2.7.m7.1.1.3.2" xref="S2.SS1.p2.7.m7.1.1.3.2.cmml">θ</mi><mi id="S2.SS1.p2.7.m7.1.1.3.3" xref="S2.SS1.p2.7.m7.1.1.3.3.cmml">e</mi></msub></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.7.m7.1b"><apply id="S2.SS1.p2.7.m7.1.1.cmml" xref="S2.SS1.p2.7.m7.1.1"><csymbol cd="ambiguous" id="S2.SS1.p2.7.m7.1.1.1.cmml" xref="S2.SS1.p2.7.m7.1.1">subscript</csymbol><ci id="S2.SS1.p2.7.m7.1.1.2.cmml" xref="S2.SS1.p2.7.m7.1.1.2">𝑓</ci><apply id="S2.SS1.p2.7.m7.1.1.3.cmml" xref="S2.SS1.p2.7.m7.1.1.3"><csymbol cd="ambiguous" id="S2.SS1.p2.7.m7.1.1.3.1.cmml" xref="S2.SS1.p2.7.m7.1.1.3">subscript</csymbol><ci id="S2.SS1.p2.7.m7.1.1.3.2.cmml" xref="S2.SS1.p2.7.m7.1.1.3.2">𝜃</ci><ci id="S2.SS1.p2.7.m7.1.1.3.3.cmml" xref="S2.SS1.p2.7.m7.1.1.3.3">𝑒</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.7.m7.1c">f_{\theta_{e}}</annotation></semantics></math> from the base model that satisfies the following:</p>
<table id="S2.E1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E1.m1.2" class="ltx_Math" alttext="f_{\theta_{e}}(x_{e})=y_{e},\forall(x_{e},y_{e})\in Z_{e}" display="block"><semantics id="S2.E1.m1.2a"><mrow id="S2.E1.m1.2.2.2" xref="S2.E1.m1.2.2.3.cmml"><mrow id="S2.E1.m1.1.1.1.1" xref="S2.E1.m1.1.1.1.1.cmml"><mrow id="S2.E1.m1.1.1.1.1.1" xref="S2.E1.m1.1.1.1.1.1.cmml"><msub id="S2.E1.m1.1.1.1.1.1.3" xref="S2.E1.m1.1.1.1.1.1.3.cmml"><mi id="S2.E1.m1.1.1.1.1.1.3.2" xref="S2.E1.m1.1.1.1.1.1.3.2.cmml">f</mi><msub id="S2.E1.m1.1.1.1.1.1.3.3" xref="S2.E1.m1.1.1.1.1.1.3.3.cmml"><mi id="S2.E1.m1.1.1.1.1.1.3.3.2" xref="S2.E1.m1.1.1.1.1.1.3.3.2.cmml">θ</mi><mi id="S2.E1.m1.1.1.1.1.1.3.3.3" xref="S2.E1.m1.1.1.1.1.1.3.3.3.cmml">e</mi></msub></msub><mo lspace="0em" rspace="0em" id="S2.E1.m1.1.1.1.1.1.2" xref="S2.E1.m1.1.1.1.1.1.2.cmml">​</mo><mrow id="S2.E1.m1.1.1.1.1.1.1.1" xref="S2.E1.m1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.E1.m1.1.1.1.1.1.1.1.2" xref="S2.E1.m1.1.1.1.1.1.1.1.1.cmml">(</mo><msub id="S2.E1.m1.1.1.1.1.1.1.1.1" xref="S2.E1.m1.1.1.1.1.1.1.1.1.cmml"><mi id="S2.E1.m1.1.1.1.1.1.1.1.1.2" xref="S2.E1.m1.1.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="S2.E1.m1.1.1.1.1.1.1.1.1.3" xref="S2.E1.m1.1.1.1.1.1.1.1.1.3.cmml">e</mi></msub><mo stretchy="false" id="S2.E1.m1.1.1.1.1.1.1.1.3" xref="S2.E1.m1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S2.E1.m1.1.1.1.1.2" xref="S2.E1.m1.1.1.1.1.2.cmml">=</mo><msub id="S2.E1.m1.1.1.1.1.3" xref="S2.E1.m1.1.1.1.1.3.cmml"><mi id="S2.E1.m1.1.1.1.1.3.2" xref="S2.E1.m1.1.1.1.1.3.2.cmml">y</mi><mi id="S2.E1.m1.1.1.1.1.3.3" xref="S2.E1.m1.1.1.1.1.3.3.cmml">e</mi></msub></mrow><mo id="S2.E1.m1.2.2.2.3" xref="S2.E1.m1.2.2.3a.cmml">,</mo><mrow id="S2.E1.m1.2.2.2.2" xref="S2.E1.m1.2.2.2.2.cmml"><mrow id="S2.E1.m1.2.2.2.2.2" xref="S2.E1.m1.2.2.2.2.2.cmml"><mo id="S2.E1.m1.2.2.2.2.2.3" xref="S2.E1.m1.2.2.2.2.2.3.cmml">∀</mo><mrow id="S2.E1.m1.2.2.2.2.2.2.2" xref="S2.E1.m1.2.2.2.2.2.2.3.cmml"><mo stretchy="false" id="S2.E1.m1.2.2.2.2.2.2.2.3" xref="S2.E1.m1.2.2.2.2.2.2.3.cmml">(</mo><msub id="S2.E1.m1.2.2.2.2.1.1.1.1" xref="S2.E1.m1.2.2.2.2.1.1.1.1.cmml"><mi id="S2.E1.m1.2.2.2.2.1.1.1.1.2" xref="S2.E1.m1.2.2.2.2.1.1.1.1.2.cmml">x</mi><mi id="S2.E1.m1.2.2.2.2.1.1.1.1.3" xref="S2.E1.m1.2.2.2.2.1.1.1.1.3.cmml">e</mi></msub><mo id="S2.E1.m1.2.2.2.2.2.2.2.4" xref="S2.E1.m1.2.2.2.2.2.2.3.cmml">,</mo><msub id="S2.E1.m1.2.2.2.2.2.2.2.2" xref="S2.E1.m1.2.2.2.2.2.2.2.2.cmml"><mi id="S2.E1.m1.2.2.2.2.2.2.2.2.2" xref="S2.E1.m1.2.2.2.2.2.2.2.2.2.cmml">y</mi><mi id="S2.E1.m1.2.2.2.2.2.2.2.2.3" xref="S2.E1.m1.2.2.2.2.2.2.2.2.3.cmml">e</mi></msub><mo stretchy="false" id="S2.E1.m1.2.2.2.2.2.2.2.5" xref="S2.E1.m1.2.2.2.2.2.2.3.cmml">)</mo></mrow></mrow><mo id="S2.E1.m1.2.2.2.2.3" xref="S2.E1.m1.2.2.2.2.3.cmml">∈</mo><msub id="S2.E1.m1.2.2.2.2.4" xref="S2.E1.m1.2.2.2.2.4.cmml"><mi id="S2.E1.m1.2.2.2.2.4.2" xref="S2.E1.m1.2.2.2.2.4.2.cmml">Z</mi><mi id="S2.E1.m1.2.2.2.2.4.3" xref="S2.E1.m1.2.2.2.2.4.3.cmml">e</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E1.m1.2b"><apply id="S2.E1.m1.2.2.3.cmml" xref="S2.E1.m1.2.2.2"><csymbol cd="ambiguous" id="S2.E1.m1.2.2.3a.cmml" xref="S2.E1.m1.2.2.2.3">formulae-sequence</csymbol><apply id="S2.E1.m1.1.1.1.1.cmml" xref="S2.E1.m1.1.1.1.1"><eq id="S2.E1.m1.1.1.1.1.2.cmml" xref="S2.E1.m1.1.1.1.1.2"></eq><apply id="S2.E1.m1.1.1.1.1.1.cmml" xref="S2.E1.m1.1.1.1.1.1"><times id="S2.E1.m1.1.1.1.1.1.2.cmml" xref="S2.E1.m1.1.1.1.1.1.2"></times><apply id="S2.E1.m1.1.1.1.1.1.3.cmml" xref="S2.E1.m1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.1.1.3.1.cmml" xref="S2.E1.m1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.E1.m1.1.1.1.1.1.3.2.cmml" xref="S2.E1.m1.1.1.1.1.1.3.2">𝑓</ci><apply id="S2.E1.m1.1.1.1.1.1.3.3.cmml" xref="S2.E1.m1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.1.1.3.3.1.cmml" xref="S2.E1.m1.1.1.1.1.1.3.3">subscript</csymbol><ci id="S2.E1.m1.1.1.1.1.1.3.3.2.cmml" xref="S2.E1.m1.1.1.1.1.1.3.3.2">𝜃</ci><ci id="S2.E1.m1.1.1.1.1.1.3.3.3.cmml" xref="S2.E1.m1.1.1.1.1.1.3.3.3">𝑒</ci></apply></apply><apply id="S2.E1.m1.1.1.1.1.1.1.1.1.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.E1.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.1.2">𝑥</ci><ci id="S2.E1.m1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.1.3">𝑒</ci></apply></apply><apply id="S2.E1.m1.1.1.1.1.3.cmml" xref="S2.E1.m1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.1.3.1.cmml" xref="S2.E1.m1.1.1.1.1.3">subscript</csymbol><ci id="S2.E1.m1.1.1.1.1.3.2.cmml" xref="S2.E1.m1.1.1.1.1.3.2">𝑦</ci><ci id="S2.E1.m1.1.1.1.1.3.3.cmml" xref="S2.E1.m1.1.1.1.1.3.3">𝑒</ci></apply></apply><apply id="S2.E1.m1.2.2.2.2.cmml" xref="S2.E1.m1.2.2.2.2"><in id="S2.E1.m1.2.2.2.2.3.cmml" xref="S2.E1.m1.2.2.2.2.3"></in><apply id="S2.E1.m1.2.2.2.2.2.cmml" xref="S2.E1.m1.2.2.2.2.2"><csymbol cd="latexml" id="S2.E1.m1.2.2.2.2.2.3.cmml" xref="S2.E1.m1.2.2.2.2.2.3">for-all</csymbol><interval closure="open" id="S2.E1.m1.2.2.2.2.2.2.3.cmml" xref="S2.E1.m1.2.2.2.2.2.2.2"><apply id="S2.E1.m1.2.2.2.2.1.1.1.1.cmml" xref="S2.E1.m1.2.2.2.2.1.1.1.1"><csymbol cd="ambiguous" id="S2.E1.m1.2.2.2.2.1.1.1.1.1.cmml" xref="S2.E1.m1.2.2.2.2.1.1.1.1">subscript</csymbol><ci id="S2.E1.m1.2.2.2.2.1.1.1.1.2.cmml" xref="S2.E1.m1.2.2.2.2.1.1.1.1.2">𝑥</ci><ci id="S2.E1.m1.2.2.2.2.1.1.1.1.3.cmml" xref="S2.E1.m1.2.2.2.2.1.1.1.1.3">𝑒</ci></apply><apply id="S2.E1.m1.2.2.2.2.2.2.2.2.cmml" xref="S2.E1.m1.2.2.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.E1.m1.2.2.2.2.2.2.2.2.1.cmml" xref="S2.E1.m1.2.2.2.2.2.2.2.2">subscript</csymbol><ci id="S2.E1.m1.2.2.2.2.2.2.2.2.2.cmml" xref="S2.E1.m1.2.2.2.2.2.2.2.2.2">𝑦</ci><ci id="S2.E1.m1.2.2.2.2.2.2.2.2.3.cmml" xref="S2.E1.m1.2.2.2.2.2.2.2.2.3">𝑒</ci></apply></interval></apply><apply id="S2.E1.m1.2.2.2.2.4.cmml" xref="S2.E1.m1.2.2.2.2.4"><csymbol cd="ambiguous" id="S2.E1.m1.2.2.2.2.4.1.cmml" xref="S2.E1.m1.2.2.2.2.4">subscript</csymbol><ci id="S2.E1.m1.2.2.2.2.4.2.cmml" xref="S2.E1.m1.2.2.2.2.4.2">𝑍</ci><ci id="S2.E1.m1.2.2.2.2.4.3.cmml" xref="S2.E1.m1.2.2.2.2.4.3">𝑒</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E1.m1.2c">f_{\theta_{e}}(x_{e})=y_{e},\forall(x_{e},y_{e})\in Z_{e}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
</div>
<div id="S2.SS1.p3" class="ltx_para">
<p id="S2.SS1.p3.1" class="ltx_p">The ideal edited model <math id="S2.SS1.p3.1.m1.1" class="ltx_Math" alttext="f_{\theta_{e}}" display="inline"><semantics id="S2.SS1.p3.1.m1.1a"><msub id="S2.SS1.p3.1.m1.1.1" xref="S2.SS1.p3.1.m1.1.1.cmml"><mi id="S2.SS1.p3.1.m1.1.1.2" xref="S2.SS1.p3.1.m1.1.1.2.cmml">f</mi><msub id="S2.SS1.p3.1.m1.1.1.3" xref="S2.SS1.p3.1.m1.1.1.3.cmml"><mi id="S2.SS1.p3.1.m1.1.1.3.2" xref="S2.SS1.p3.1.m1.1.1.3.2.cmml">θ</mi><mi id="S2.SS1.p3.1.m1.1.1.3.3" xref="S2.SS1.p3.1.m1.1.1.3.3.cmml">e</mi></msub></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.1.m1.1b"><apply id="S2.SS1.p3.1.m1.1.1.cmml" xref="S2.SS1.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS1.p3.1.m1.1.1.1.cmml" xref="S2.SS1.p3.1.m1.1.1">subscript</csymbol><ci id="S2.SS1.p3.1.m1.1.1.2.cmml" xref="S2.SS1.p3.1.m1.1.1.2">𝑓</ci><apply id="S2.SS1.p3.1.m1.1.1.3.cmml" xref="S2.SS1.p3.1.m1.1.1.3"><csymbol cd="ambiguous" id="S2.SS1.p3.1.m1.1.1.3.1.cmml" xref="S2.SS1.p3.1.m1.1.1.3">subscript</csymbol><ci id="S2.SS1.p3.1.m1.1.1.3.2.cmml" xref="S2.SS1.p3.1.m1.1.1.3.2">𝜃</ci><ci id="S2.SS1.p3.1.m1.1.1.3.3.cmml" xref="S2.SS1.p3.1.m1.1.1.3.3">𝑒</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.1.m1.1c">f_{\theta_{e}}</annotation></semantics></math> should satisfy three properties: <span id="S2.SS1.p3.1.1" class="ltx_text ltx_font_bold">reliability</span>, <span id="S2.SS1.p3.1.2" class="ltx_text ltx_font_bold">generalization</span>, and <span id="S2.SS1.p3.1.3" class="ltx_text ltx_font_bold">locality</span>. An illustration is shown in Figure <a href="#S1.F2" title="Figure 2 ‣ 1 Introduction ‣ Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.</p>
</div>
<section id="S2.SS1.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Reliability</h5>

<div id="S2.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="S2.SS1.SSS0.Px1.p1.5" class="ltx_p">Given an edit query <math id="S2.SS1.SSS0.Px1.p1.1.m1.2" class="ltx_Math" alttext="(x_{e},y_{e})" display="inline"><semantics id="S2.SS1.SSS0.Px1.p1.1.m1.2a"><mrow id="S2.SS1.SSS0.Px1.p1.1.m1.2.2.2" xref="S2.SS1.SSS0.Px1.p1.1.m1.2.2.3.cmml"><mo stretchy="false" id="S2.SS1.SSS0.Px1.p1.1.m1.2.2.2.3" xref="S2.SS1.SSS0.Px1.p1.1.m1.2.2.3.cmml">(</mo><msub id="S2.SS1.SSS0.Px1.p1.1.m1.1.1.1.1" xref="S2.SS1.SSS0.Px1.p1.1.m1.1.1.1.1.cmml"><mi id="S2.SS1.SSS0.Px1.p1.1.m1.1.1.1.1.2" xref="S2.SS1.SSS0.Px1.p1.1.m1.1.1.1.1.2.cmml">x</mi><mi id="S2.SS1.SSS0.Px1.p1.1.m1.1.1.1.1.3" xref="S2.SS1.SSS0.Px1.p1.1.m1.1.1.1.1.3.cmml">e</mi></msub><mo id="S2.SS1.SSS0.Px1.p1.1.m1.2.2.2.4" xref="S2.SS1.SSS0.Px1.p1.1.m1.2.2.3.cmml">,</mo><msub id="S2.SS1.SSS0.Px1.p1.1.m1.2.2.2.2" xref="S2.SS1.SSS0.Px1.p1.1.m1.2.2.2.2.cmml"><mi id="S2.SS1.SSS0.Px1.p1.1.m1.2.2.2.2.2" xref="S2.SS1.SSS0.Px1.p1.1.m1.2.2.2.2.2.cmml">y</mi><mi id="S2.SS1.SSS0.Px1.p1.1.m1.2.2.2.2.3" xref="S2.SS1.SSS0.Px1.p1.1.m1.2.2.2.2.3.cmml">e</mi></msub><mo stretchy="false" id="S2.SS1.SSS0.Px1.p1.1.m1.2.2.2.5" xref="S2.SS1.SSS0.Px1.p1.1.m1.2.2.3.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px1.p1.1.m1.2b"><interval closure="open" id="S2.SS1.SSS0.Px1.p1.1.m1.2.2.3.cmml" xref="S2.SS1.SSS0.Px1.p1.1.m1.2.2.2"><apply id="S2.SS1.SSS0.Px1.p1.1.m1.1.1.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px1.p1.1.m1.1.1.1.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.1.m1.1.1.1.1">subscript</csymbol><ci id="S2.SS1.SSS0.Px1.p1.1.m1.1.1.1.1.2.cmml" xref="S2.SS1.SSS0.Px1.p1.1.m1.1.1.1.1.2">𝑥</ci><ci id="S2.SS1.SSS0.Px1.p1.1.m1.1.1.1.1.3.cmml" xref="S2.SS1.SSS0.Px1.p1.1.m1.1.1.1.1.3">𝑒</ci></apply><apply id="S2.SS1.SSS0.Px1.p1.1.m1.2.2.2.2.cmml" xref="S2.SS1.SSS0.Px1.p1.1.m1.2.2.2.2"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px1.p1.1.m1.2.2.2.2.1.cmml" xref="S2.SS1.SSS0.Px1.p1.1.m1.2.2.2.2">subscript</csymbol><ci id="S2.SS1.SSS0.Px1.p1.1.m1.2.2.2.2.2.cmml" xref="S2.SS1.SSS0.Px1.p1.1.m1.2.2.2.2.2">𝑦</ci><ci id="S2.SS1.SSS0.Px1.p1.1.m1.2.2.2.2.3.cmml" xref="S2.SS1.SSS0.Px1.p1.1.m1.2.2.2.2.3">𝑒</ci></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px1.p1.1.m1.2c">(x_{e},y_{e})</annotation></semantics></math>, the edited model <math id="S2.SS1.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="f_{\theta_{e}}" display="inline"><semantics id="S2.SS1.SSS0.Px1.p1.2.m2.1a"><msub id="S2.SS1.SSS0.Px1.p1.2.m2.1.1" xref="S2.SS1.SSS0.Px1.p1.2.m2.1.1.cmml"><mi id="S2.SS1.SSS0.Px1.p1.2.m2.1.1.2" xref="S2.SS1.SSS0.Px1.p1.2.m2.1.1.2.cmml">f</mi><msub id="S2.SS1.SSS0.Px1.p1.2.m2.1.1.3" xref="S2.SS1.SSS0.Px1.p1.2.m2.1.1.3.cmml"><mi id="S2.SS1.SSS0.Px1.p1.2.m2.1.1.3.2" xref="S2.SS1.SSS0.Px1.p1.2.m2.1.1.3.2.cmml">θ</mi><mi id="S2.SS1.SSS0.Px1.p1.2.m2.1.1.3.3" xref="S2.SS1.SSS0.Px1.p1.2.m2.1.1.3.3.cmml">e</mi></msub></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px1.p1.2.m2.1b"><apply id="S2.SS1.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px1.p1.2.m2.1.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.2.m2.1.1">subscript</csymbol><ci id="S2.SS1.SSS0.Px1.p1.2.m2.1.1.2.cmml" xref="S2.SS1.SSS0.Px1.p1.2.m2.1.1.2">𝑓</ci><apply id="S2.SS1.SSS0.Px1.p1.2.m2.1.1.3.cmml" xref="S2.SS1.SSS0.Px1.p1.2.m2.1.1.3"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px1.p1.2.m2.1.1.3.1.cmml" xref="S2.SS1.SSS0.Px1.p1.2.m2.1.1.3">subscript</csymbol><ci id="S2.SS1.SSS0.Px1.p1.2.m2.1.1.3.2.cmml" xref="S2.SS1.SSS0.Px1.p1.2.m2.1.1.3.2">𝜃</ci><ci id="S2.SS1.SSS0.Px1.p1.2.m2.1.1.3.3.cmml" xref="S2.SS1.SSS0.Px1.p1.2.m2.1.1.3.3">𝑒</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px1.p1.2.m2.1c">f_{\theta_{e}}</annotation></semantics></math> should output the target answer <math id="S2.SS1.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="y_{e}" display="inline"><semantics id="S2.SS1.SSS0.Px1.p1.3.m3.1a"><msub id="S2.SS1.SSS0.Px1.p1.3.m3.1.1" xref="S2.SS1.SSS0.Px1.p1.3.m3.1.1.cmml"><mi id="S2.SS1.SSS0.Px1.p1.3.m3.1.1.2" xref="S2.SS1.SSS0.Px1.p1.3.m3.1.1.2.cmml">y</mi><mi id="S2.SS1.SSS0.Px1.p1.3.m3.1.1.3" xref="S2.SS1.SSS0.Px1.p1.3.m3.1.1.3.cmml">e</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px1.p1.3.m3.1b"><apply id="S2.SS1.SSS0.Px1.p1.3.m3.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px1.p1.3.m3.1.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.3.m3.1.1">subscript</csymbol><ci id="S2.SS1.SSS0.Px1.p1.3.m3.1.1.2.cmml" xref="S2.SS1.SSS0.Px1.p1.3.m3.1.1.2">𝑦</ci><ci id="S2.SS1.SSS0.Px1.p1.3.m3.1.1.3.cmml" xref="S2.SS1.SSS0.Px1.p1.3.m3.1.1.3">𝑒</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px1.p1.3.m3.1c">y_{e}</annotation></semantics></math> when given the target input <math id="S2.SS1.SSS0.Px1.p1.4.m4.1" class="ltx_Math" alttext="x_{e}" display="inline"><semantics id="S2.SS1.SSS0.Px1.p1.4.m4.1a"><msub id="S2.SS1.SSS0.Px1.p1.4.m4.1.1" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1.cmml"><mi id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.2" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1.2.cmml">x</mi><mi id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.3" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1.3.cmml">e</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px1.p1.4.m4.1b"><apply id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1">subscript</csymbol><ci id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.2.cmml" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1.2">𝑥</ci><ci id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.3.cmml" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1.3">𝑒</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px1.p1.4.m4.1c">x_{e}</annotation></semantics></math>, i.e. <math id="S2.SS1.SSS0.Px1.p1.5.m5.1" class="ltx_Math" alttext="f_{\theta_{e}}(x_{e})=y_{e}" display="inline"><semantics id="S2.SS1.SSS0.Px1.p1.5.m5.1a"><mrow id="S2.SS1.SSS0.Px1.p1.5.m5.1.1" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.cmml"><mrow id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.cmml"><msub id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.3" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.3.cmml"><mi id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.3.2" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.3.2.cmml">f</mi><msub id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.3.3" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.3.3.cmml"><mi id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.3.3.2" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.3.3.2.cmml">θ</mi><mi id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.3.3.3" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.3.3.3.cmml">e</mi></msub></msub><mo lspace="0em" rspace="0em" id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.2" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.2.cmml">​</mo><mrow id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.1.1" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.1.1.2" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.1.1.1.cmml">(</mo><msub id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.1.1.1" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.1.1.1.cmml"><mi id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.1.1.1.2" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.1.1.1.2.cmml">x</mi><mi id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.1.1.1.3" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.1.1.1.3.cmml">e</mi></msub><mo stretchy="false" id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.1.1.3" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.2" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.2.cmml">=</mo><msub id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.3" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.3.cmml"><mi id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.3.2" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.3.2.cmml">y</mi><mi id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.3.3" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.3.3.cmml">e</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px1.p1.5.m5.1b"><apply id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1"><eq id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.2.cmml" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.2"></eq><apply id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1"><times id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.2.cmml" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.2"></times><apply id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.3.cmml" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.3"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.3.1.cmml" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.3">subscript</csymbol><ci id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.3.2.cmml" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.3.2">𝑓</ci><apply id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.3.3.cmml" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.3.3"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.3.3.1.cmml" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.3.3">subscript</csymbol><ci id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.3.3.2.cmml" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.3.3.2">𝜃</ci><ci id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.3.3.3.cmml" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.3.3.3">𝑒</ci></apply></apply><apply id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.1.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.1.1.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.1.1">subscript</csymbol><ci id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.1.1.1.2.cmml" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.1.1.1.2">𝑥</ci><ci id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.1.1.1.3.cmml" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.1.1.1.3">𝑒</ci></apply></apply><apply id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.3.cmml" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.3"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.3.1.cmml" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.3">subscript</csymbol><ci id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.3.2.cmml" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.3.2">𝑦</ci><ci id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.3.3.cmml" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.3.3">𝑒</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px1.p1.5.m5.1c">f_{\theta_{e}}(x_{e})=y_{e}</annotation></semantics></math>. The reliability of a editing method is measured by calculating the average edit success rate:</p>
<table id="S2.E2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E2.m1.3" class="ltx_Math" alttext="\mathbb{E}_{(x_{e}^{\prime},y_{e}^{\prime})\sim Z_{e}}\mathbbm{1}\{f_{\theta_{e}}(x_{e}^{\prime})=y_{e}^{\prime}\}" display="block"><semantics id="S2.E2.m1.3a"><mrow id="S2.E2.m1.3.3" xref="S2.E2.m1.3.3.cmml"><msub id="S2.E2.m1.3.3.3" xref="S2.E2.m1.3.3.3.cmml"><mi id="S2.E2.m1.3.3.3.2" xref="S2.E2.m1.3.3.3.2.cmml">𝔼</mi><mrow id="S2.E2.m1.2.2.2" xref="S2.E2.m1.2.2.2.cmml"><mrow id="S2.E2.m1.2.2.2.2.2" xref="S2.E2.m1.2.2.2.2.3.cmml"><mo stretchy="false" id="S2.E2.m1.2.2.2.2.2.3" xref="S2.E2.m1.2.2.2.2.3.cmml">(</mo><msubsup id="S2.E2.m1.1.1.1.1.1.1" xref="S2.E2.m1.1.1.1.1.1.1.cmml"><mi id="S2.E2.m1.1.1.1.1.1.1.2.2" xref="S2.E2.m1.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="S2.E2.m1.1.1.1.1.1.1.2.3" xref="S2.E2.m1.1.1.1.1.1.1.2.3.cmml">e</mi><mo id="S2.E2.m1.1.1.1.1.1.1.3" xref="S2.E2.m1.1.1.1.1.1.1.3.cmml">′</mo></msubsup><mo id="S2.E2.m1.2.2.2.2.2.4" xref="S2.E2.m1.2.2.2.2.3.cmml">,</mo><msubsup id="S2.E2.m1.2.2.2.2.2.2" xref="S2.E2.m1.2.2.2.2.2.2.cmml"><mi id="S2.E2.m1.2.2.2.2.2.2.2.2" xref="S2.E2.m1.2.2.2.2.2.2.2.2.cmml">y</mi><mi id="S2.E2.m1.2.2.2.2.2.2.2.3" xref="S2.E2.m1.2.2.2.2.2.2.2.3.cmml">e</mi><mo id="S2.E2.m1.2.2.2.2.2.2.3" xref="S2.E2.m1.2.2.2.2.2.2.3.cmml">′</mo></msubsup><mo stretchy="false" id="S2.E2.m1.2.2.2.2.2.5" xref="S2.E2.m1.2.2.2.2.3.cmml">)</mo></mrow><mo id="S2.E2.m1.2.2.2.3" xref="S2.E2.m1.2.2.2.3.cmml">∼</mo><msub id="S2.E2.m1.2.2.2.4" xref="S2.E2.m1.2.2.2.4.cmml"><mi id="S2.E2.m1.2.2.2.4.2" xref="S2.E2.m1.2.2.2.4.2.cmml">Z</mi><mi id="S2.E2.m1.2.2.2.4.3" xref="S2.E2.m1.2.2.2.4.3.cmml">e</mi></msub></mrow></msub><mo lspace="0em" rspace="0em" id="S2.E2.m1.3.3.2" xref="S2.E2.m1.3.3.2.cmml">​</mo><mn id="S2.E2.m1.3.3.4" xref="S2.E2.m1.3.3.4.cmml">𝟙</mn><mo lspace="0em" rspace="0em" id="S2.E2.m1.3.3.2a" xref="S2.E2.m1.3.3.2.cmml">​</mo><mrow id="S2.E2.m1.3.3.1.1" xref="S2.E2.m1.3.3.1.2.cmml"><mo stretchy="false" id="S2.E2.m1.3.3.1.1.2" xref="S2.E2.m1.3.3.1.2.cmml">{</mo><mrow id="S2.E2.m1.3.3.1.1.1" xref="S2.E2.m1.3.3.1.1.1.cmml"><mrow id="S2.E2.m1.3.3.1.1.1.1" xref="S2.E2.m1.3.3.1.1.1.1.cmml"><msub id="S2.E2.m1.3.3.1.1.1.1.3" xref="S2.E2.m1.3.3.1.1.1.1.3.cmml"><mi id="S2.E2.m1.3.3.1.1.1.1.3.2" xref="S2.E2.m1.3.3.1.1.1.1.3.2.cmml">f</mi><msub id="S2.E2.m1.3.3.1.1.1.1.3.3" xref="S2.E2.m1.3.3.1.1.1.1.3.3.cmml"><mi id="S2.E2.m1.3.3.1.1.1.1.3.3.2" xref="S2.E2.m1.3.3.1.1.1.1.3.3.2.cmml">θ</mi><mi id="S2.E2.m1.3.3.1.1.1.1.3.3.3" xref="S2.E2.m1.3.3.1.1.1.1.3.3.3.cmml">e</mi></msub></msub><mo lspace="0em" rspace="0em" id="S2.E2.m1.3.3.1.1.1.1.2" xref="S2.E2.m1.3.3.1.1.1.1.2.cmml">​</mo><mrow id="S2.E2.m1.3.3.1.1.1.1.1.1" xref="S2.E2.m1.3.3.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.E2.m1.3.3.1.1.1.1.1.1.2" xref="S2.E2.m1.3.3.1.1.1.1.1.1.1.cmml">(</mo><msubsup id="S2.E2.m1.3.3.1.1.1.1.1.1.1" xref="S2.E2.m1.3.3.1.1.1.1.1.1.1.cmml"><mi id="S2.E2.m1.3.3.1.1.1.1.1.1.1.2.2" xref="S2.E2.m1.3.3.1.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="S2.E2.m1.3.3.1.1.1.1.1.1.1.2.3" xref="S2.E2.m1.3.3.1.1.1.1.1.1.1.2.3.cmml">e</mi><mo id="S2.E2.m1.3.3.1.1.1.1.1.1.1.3" xref="S2.E2.m1.3.3.1.1.1.1.1.1.1.3.cmml">′</mo></msubsup><mo stretchy="false" id="S2.E2.m1.3.3.1.1.1.1.1.1.3" xref="S2.E2.m1.3.3.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S2.E2.m1.3.3.1.1.1.2" xref="S2.E2.m1.3.3.1.1.1.2.cmml">=</mo><msubsup id="S2.E2.m1.3.3.1.1.1.3" xref="S2.E2.m1.3.3.1.1.1.3.cmml"><mi id="S2.E2.m1.3.3.1.1.1.3.2.2" xref="S2.E2.m1.3.3.1.1.1.3.2.2.cmml">y</mi><mi id="S2.E2.m1.3.3.1.1.1.3.2.3" xref="S2.E2.m1.3.3.1.1.1.3.2.3.cmml">e</mi><mo id="S2.E2.m1.3.3.1.1.1.3.3" xref="S2.E2.m1.3.3.1.1.1.3.3.cmml">′</mo></msubsup></mrow><mo stretchy="false" id="S2.E2.m1.3.3.1.1.3" xref="S2.E2.m1.3.3.1.2.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E2.m1.3b"><apply id="S2.E2.m1.3.3.cmml" xref="S2.E2.m1.3.3"><times id="S2.E2.m1.3.3.2.cmml" xref="S2.E2.m1.3.3.2"></times><apply id="S2.E2.m1.3.3.3.cmml" xref="S2.E2.m1.3.3.3"><csymbol cd="ambiguous" id="S2.E2.m1.3.3.3.1.cmml" xref="S2.E2.m1.3.3.3">subscript</csymbol><ci id="S2.E2.m1.3.3.3.2.cmml" xref="S2.E2.m1.3.3.3.2">𝔼</ci><apply id="S2.E2.m1.2.2.2.cmml" xref="S2.E2.m1.2.2.2"><csymbol cd="latexml" id="S2.E2.m1.2.2.2.3.cmml" xref="S2.E2.m1.2.2.2.3">similar-to</csymbol><interval closure="open" id="S2.E2.m1.2.2.2.2.3.cmml" xref="S2.E2.m1.2.2.2.2.2"><apply id="S2.E2.m1.1.1.1.1.1.1.cmml" xref="S2.E2.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E2.m1.1.1.1.1.1.1.1.cmml" xref="S2.E2.m1.1.1.1.1.1.1">superscript</csymbol><apply id="S2.E2.m1.1.1.1.1.1.1.2.cmml" xref="S2.E2.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E2.m1.1.1.1.1.1.1.2.1.cmml" xref="S2.E2.m1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.E2.m1.1.1.1.1.1.1.2.2.cmml" xref="S2.E2.m1.1.1.1.1.1.1.2.2">𝑥</ci><ci id="S2.E2.m1.1.1.1.1.1.1.2.3.cmml" xref="S2.E2.m1.1.1.1.1.1.1.2.3">𝑒</ci></apply><ci id="S2.E2.m1.1.1.1.1.1.1.3.cmml" xref="S2.E2.m1.1.1.1.1.1.1.3">′</ci></apply><apply id="S2.E2.m1.2.2.2.2.2.2.cmml" xref="S2.E2.m1.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.E2.m1.2.2.2.2.2.2.1.cmml" xref="S2.E2.m1.2.2.2.2.2.2">superscript</csymbol><apply id="S2.E2.m1.2.2.2.2.2.2.2.cmml" xref="S2.E2.m1.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.E2.m1.2.2.2.2.2.2.2.1.cmml" xref="S2.E2.m1.2.2.2.2.2.2">subscript</csymbol><ci id="S2.E2.m1.2.2.2.2.2.2.2.2.cmml" xref="S2.E2.m1.2.2.2.2.2.2.2.2">𝑦</ci><ci id="S2.E2.m1.2.2.2.2.2.2.2.3.cmml" xref="S2.E2.m1.2.2.2.2.2.2.2.3">𝑒</ci></apply><ci id="S2.E2.m1.2.2.2.2.2.2.3.cmml" xref="S2.E2.m1.2.2.2.2.2.2.3">′</ci></apply></interval><apply id="S2.E2.m1.2.2.2.4.cmml" xref="S2.E2.m1.2.2.2.4"><csymbol cd="ambiguous" id="S2.E2.m1.2.2.2.4.1.cmml" xref="S2.E2.m1.2.2.2.4">subscript</csymbol><ci id="S2.E2.m1.2.2.2.4.2.cmml" xref="S2.E2.m1.2.2.2.4.2">𝑍</ci><ci id="S2.E2.m1.2.2.2.4.3.cmml" xref="S2.E2.m1.2.2.2.4.3">𝑒</ci></apply></apply></apply><cn type="integer" id="S2.E2.m1.3.3.4.cmml" xref="S2.E2.m1.3.3.4">1</cn><set id="S2.E2.m1.3.3.1.2.cmml" xref="S2.E2.m1.3.3.1.1"><apply id="S2.E2.m1.3.3.1.1.1.cmml" xref="S2.E2.m1.3.3.1.1.1"><eq id="S2.E2.m1.3.3.1.1.1.2.cmml" xref="S2.E2.m1.3.3.1.1.1.2"></eq><apply id="S2.E2.m1.3.3.1.1.1.1.cmml" xref="S2.E2.m1.3.3.1.1.1.1"><times id="S2.E2.m1.3.3.1.1.1.1.2.cmml" xref="S2.E2.m1.3.3.1.1.1.1.2"></times><apply id="S2.E2.m1.3.3.1.1.1.1.3.cmml" xref="S2.E2.m1.3.3.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E2.m1.3.3.1.1.1.1.3.1.cmml" xref="S2.E2.m1.3.3.1.1.1.1.3">subscript</csymbol><ci id="S2.E2.m1.3.3.1.1.1.1.3.2.cmml" xref="S2.E2.m1.3.3.1.1.1.1.3.2">𝑓</ci><apply id="S2.E2.m1.3.3.1.1.1.1.3.3.cmml" xref="S2.E2.m1.3.3.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S2.E2.m1.3.3.1.1.1.1.3.3.1.cmml" xref="S2.E2.m1.3.3.1.1.1.1.3.3">subscript</csymbol><ci id="S2.E2.m1.3.3.1.1.1.1.3.3.2.cmml" xref="S2.E2.m1.3.3.1.1.1.1.3.3.2">𝜃</ci><ci id="S2.E2.m1.3.3.1.1.1.1.3.3.3.cmml" xref="S2.E2.m1.3.3.1.1.1.1.3.3.3">𝑒</ci></apply></apply><apply id="S2.E2.m1.3.3.1.1.1.1.1.1.1.cmml" xref="S2.E2.m1.3.3.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E2.m1.3.3.1.1.1.1.1.1.1.1.cmml" xref="S2.E2.m1.3.3.1.1.1.1.1.1">superscript</csymbol><apply id="S2.E2.m1.3.3.1.1.1.1.1.1.1.2.cmml" xref="S2.E2.m1.3.3.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E2.m1.3.3.1.1.1.1.1.1.1.2.1.cmml" xref="S2.E2.m1.3.3.1.1.1.1.1.1">subscript</csymbol><ci id="S2.E2.m1.3.3.1.1.1.1.1.1.1.2.2.cmml" xref="S2.E2.m1.3.3.1.1.1.1.1.1.1.2.2">𝑥</ci><ci id="S2.E2.m1.3.3.1.1.1.1.1.1.1.2.3.cmml" xref="S2.E2.m1.3.3.1.1.1.1.1.1.1.2.3">𝑒</ci></apply><ci id="S2.E2.m1.3.3.1.1.1.1.1.1.1.3.cmml" xref="S2.E2.m1.3.3.1.1.1.1.1.1.1.3">′</ci></apply></apply><apply id="S2.E2.m1.3.3.1.1.1.3.cmml" xref="S2.E2.m1.3.3.1.1.1.3"><csymbol cd="ambiguous" id="S2.E2.m1.3.3.1.1.1.3.1.cmml" xref="S2.E2.m1.3.3.1.1.1.3">superscript</csymbol><apply id="S2.E2.m1.3.3.1.1.1.3.2.cmml" xref="S2.E2.m1.3.3.1.1.1.3"><csymbol cd="ambiguous" id="S2.E2.m1.3.3.1.1.1.3.2.1.cmml" xref="S2.E2.m1.3.3.1.1.1.3">subscript</csymbol><ci id="S2.E2.m1.3.3.1.1.1.3.2.2.cmml" xref="S2.E2.m1.3.3.1.1.1.3.2.2">𝑦</ci><ci id="S2.E2.m1.3.3.1.1.1.3.2.3.cmml" xref="S2.E2.m1.3.3.1.1.1.3.2.3">𝑒</ci></apply><ci id="S2.E2.m1.3.3.1.1.1.3.3.cmml" xref="S2.E2.m1.3.3.1.1.1.3.3">′</ci></apply></apply></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E2.m1.3c">\mathbb{E}_{(x_{e}^{\prime},y_{e}^{\prime})\sim Z_{e}}\mathbbm{1}\{f_{\theta_{e}}(x_{e}^{\prime})=y_{e}^{\prime}\}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
</div>
</section>
<section id="S2.SS1.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Generalization</h5>

<div id="S2.SS1.SSS0.Px2.p1" class="ltx_para">
<p id="S2.SS1.SSS0.Px2.p1.3" class="ltx_p">The edited model should generalize the edited knowledge to relevant instances. The generalization metric is commonly formulated as the average success rate on the neighboring set:</p>
<table id="S2.E3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E3.m1.5" class="ltx_Math" alttext="\mathbb{E}_{(x_{e}^{\prime},y_{e}^{\prime})\sim N(x_{e},y_{e})}\mathbbm{1}\{f_{\theta_{e}}(x_{e}^{\prime})=y_{e}^{\prime}\}," display="block"><semantics id="S2.E3.m1.5a"><mrow id="S2.E3.m1.5.5.1" xref="S2.E3.m1.5.5.1.1.cmml"><mrow id="S2.E3.m1.5.5.1.1" xref="S2.E3.m1.5.5.1.1.cmml"><msub id="S2.E3.m1.5.5.1.1.3" xref="S2.E3.m1.5.5.1.1.3.cmml"><mi id="S2.E3.m1.5.5.1.1.3.2" xref="S2.E3.m1.5.5.1.1.3.2.cmml">𝔼</mi><mrow id="S2.E3.m1.4.4.4" xref="S2.E3.m1.4.4.4.cmml"><mrow id="S2.E3.m1.2.2.2.2.2" xref="S2.E3.m1.2.2.2.2.3.cmml"><mo stretchy="false" id="S2.E3.m1.2.2.2.2.2.3" xref="S2.E3.m1.2.2.2.2.3.cmml">(</mo><msubsup id="S2.E3.m1.1.1.1.1.1.1" xref="S2.E3.m1.1.1.1.1.1.1.cmml"><mi id="S2.E3.m1.1.1.1.1.1.1.2.2" xref="S2.E3.m1.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="S2.E3.m1.1.1.1.1.1.1.2.3" xref="S2.E3.m1.1.1.1.1.1.1.2.3.cmml">e</mi><mo id="S2.E3.m1.1.1.1.1.1.1.3" xref="S2.E3.m1.1.1.1.1.1.1.3.cmml">′</mo></msubsup><mo id="S2.E3.m1.2.2.2.2.2.4" xref="S2.E3.m1.2.2.2.2.3.cmml">,</mo><msubsup id="S2.E3.m1.2.2.2.2.2.2" xref="S2.E3.m1.2.2.2.2.2.2.cmml"><mi id="S2.E3.m1.2.2.2.2.2.2.2.2" xref="S2.E3.m1.2.2.2.2.2.2.2.2.cmml">y</mi><mi id="S2.E3.m1.2.2.2.2.2.2.2.3" xref="S2.E3.m1.2.2.2.2.2.2.2.3.cmml">e</mi><mo id="S2.E3.m1.2.2.2.2.2.2.3" xref="S2.E3.m1.2.2.2.2.2.2.3.cmml">′</mo></msubsup><mo stretchy="false" id="S2.E3.m1.2.2.2.2.2.5" xref="S2.E3.m1.2.2.2.2.3.cmml">)</mo></mrow><mo id="S2.E3.m1.4.4.4.5" xref="S2.E3.m1.4.4.4.5.cmml">∼</mo><mrow id="S2.E3.m1.4.4.4.4" xref="S2.E3.m1.4.4.4.4.cmml"><mi id="S2.E3.m1.4.4.4.4.4" xref="S2.E3.m1.4.4.4.4.4.cmml">N</mi><mo lspace="0em" rspace="0em" id="S2.E3.m1.4.4.4.4.3" xref="S2.E3.m1.4.4.4.4.3.cmml">​</mo><mrow id="S2.E3.m1.4.4.4.4.2.2" xref="S2.E3.m1.4.4.4.4.2.3.cmml"><mo stretchy="false" id="S2.E3.m1.4.4.4.4.2.2.3" xref="S2.E3.m1.4.4.4.4.2.3.cmml">(</mo><msub id="S2.E3.m1.3.3.3.3.1.1.1" xref="S2.E3.m1.3.3.3.3.1.1.1.cmml"><mi id="S2.E3.m1.3.3.3.3.1.1.1.2" xref="S2.E3.m1.3.3.3.3.1.1.1.2.cmml">x</mi><mi id="S2.E3.m1.3.3.3.3.1.1.1.3" xref="S2.E3.m1.3.3.3.3.1.1.1.3.cmml">e</mi></msub><mo id="S2.E3.m1.4.4.4.4.2.2.4" xref="S2.E3.m1.4.4.4.4.2.3.cmml">,</mo><msub id="S2.E3.m1.4.4.4.4.2.2.2" xref="S2.E3.m1.4.4.4.4.2.2.2.cmml"><mi id="S2.E3.m1.4.4.4.4.2.2.2.2" xref="S2.E3.m1.4.4.4.4.2.2.2.2.cmml">y</mi><mi id="S2.E3.m1.4.4.4.4.2.2.2.3" xref="S2.E3.m1.4.4.4.4.2.2.2.3.cmml">e</mi></msub><mo stretchy="false" id="S2.E3.m1.4.4.4.4.2.2.5" xref="S2.E3.m1.4.4.4.4.2.3.cmml">)</mo></mrow></mrow></mrow></msub><mo lspace="0em" rspace="0em" id="S2.E3.m1.5.5.1.1.2" xref="S2.E3.m1.5.5.1.1.2.cmml">​</mo><mn id="S2.E3.m1.5.5.1.1.4" xref="S2.E3.m1.5.5.1.1.4.cmml">𝟙</mn><mo lspace="0em" rspace="0em" id="S2.E3.m1.5.5.1.1.2a" xref="S2.E3.m1.5.5.1.1.2.cmml">​</mo><mrow id="S2.E3.m1.5.5.1.1.1.1" xref="S2.E3.m1.5.5.1.1.1.2.cmml"><mo stretchy="false" id="S2.E3.m1.5.5.1.1.1.1.2" xref="S2.E3.m1.5.5.1.1.1.2.cmml">{</mo><mrow id="S2.E3.m1.5.5.1.1.1.1.1" xref="S2.E3.m1.5.5.1.1.1.1.1.cmml"><mrow id="S2.E3.m1.5.5.1.1.1.1.1.1" xref="S2.E3.m1.5.5.1.1.1.1.1.1.cmml"><msub id="S2.E3.m1.5.5.1.1.1.1.1.1.3" xref="S2.E3.m1.5.5.1.1.1.1.1.1.3.cmml"><mi id="S2.E3.m1.5.5.1.1.1.1.1.1.3.2" xref="S2.E3.m1.5.5.1.1.1.1.1.1.3.2.cmml">f</mi><msub id="S2.E3.m1.5.5.1.1.1.1.1.1.3.3" xref="S2.E3.m1.5.5.1.1.1.1.1.1.3.3.cmml"><mi id="S2.E3.m1.5.5.1.1.1.1.1.1.3.3.2" xref="S2.E3.m1.5.5.1.1.1.1.1.1.3.3.2.cmml">θ</mi><mi id="S2.E3.m1.5.5.1.1.1.1.1.1.3.3.3" xref="S2.E3.m1.5.5.1.1.1.1.1.1.3.3.3.cmml">e</mi></msub></msub><mo lspace="0em" rspace="0em" id="S2.E3.m1.5.5.1.1.1.1.1.1.2" xref="S2.E3.m1.5.5.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S2.E3.m1.5.5.1.1.1.1.1.1.1.1" xref="S2.E3.m1.5.5.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.E3.m1.5.5.1.1.1.1.1.1.1.1.2" xref="S2.E3.m1.5.5.1.1.1.1.1.1.1.1.1.cmml">(</mo><msubsup id="S2.E3.m1.5.5.1.1.1.1.1.1.1.1.1" xref="S2.E3.m1.5.5.1.1.1.1.1.1.1.1.1.cmml"><mi id="S2.E3.m1.5.5.1.1.1.1.1.1.1.1.1.2.2" xref="S2.E3.m1.5.5.1.1.1.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="S2.E3.m1.5.5.1.1.1.1.1.1.1.1.1.2.3" xref="S2.E3.m1.5.5.1.1.1.1.1.1.1.1.1.2.3.cmml">e</mi><mo id="S2.E3.m1.5.5.1.1.1.1.1.1.1.1.1.3" xref="S2.E3.m1.5.5.1.1.1.1.1.1.1.1.1.3.cmml">′</mo></msubsup><mo stretchy="false" id="S2.E3.m1.5.5.1.1.1.1.1.1.1.1.3" xref="S2.E3.m1.5.5.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S2.E3.m1.5.5.1.1.1.1.1.2" xref="S2.E3.m1.5.5.1.1.1.1.1.2.cmml">=</mo><msubsup id="S2.E3.m1.5.5.1.1.1.1.1.3" xref="S2.E3.m1.5.5.1.1.1.1.1.3.cmml"><mi id="S2.E3.m1.5.5.1.1.1.1.1.3.2.2" xref="S2.E3.m1.5.5.1.1.1.1.1.3.2.2.cmml">y</mi><mi id="S2.E3.m1.5.5.1.1.1.1.1.3.2.3" xref="S2.E3.m1.5.5.1.1.1.1.1.3.2.3.cmml">e</mi><mo id="S2.E3.m1.5.5.1.1.1.1.1.3.3" xref="S2.E3.m1.5.5.1.1.1.1.1.3.3.cmml">′</mo></msubsup></mrow><mo stretchy="false" id="S2.E3.m1.5.5.1.1.1.1.3" xref="S2.E3.m1.5.5.1.1.1.2.cmml">}</mo></mrow></mrow><mo id="S2.E3.m1.5.5.1.2" xref="S2.E3.m1.5.5.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E3.m1.5b"><apply id="S2.E3.m1.5.5.1.1.cmml" xref="S2.E3.m1.5.5.1"><times id="S2.E3.m1.5.5.1.1.2.cmml" xref="S2.E3.m1.5.5.1.1.2"></times><apply id="S2.E3.m1.5.5.1.1.3.cmml" xref="S2.E3.m1.5.5.1.1.3"><csymbol cd="ambiguous" id="S2.E3.m1.5.5.1.1.3.1.cmml" xref="S2.E3.m1.5.5.1.1.3">subscript</csymbol><ci id="S2.E3.m1.5.5.1.1.3.2.cmml" xref="S2.E3.m1.5.5.1.1.3.2">𝔼</ci><apply id="S2.E3.m1.4.4.4.cmml" xref="S2.E3.m1.4.4.4"><csymbol cd="latexml" id="S2.E3.m1.4.4.4.5.cmml" xref="S2.E3.m1.4.4.4.5">similar-to</csymbol><interval closure="open" id="S2.E3.m1.2.2.2.2.3.cmml" xref="S2.E3.m1.2.2.2.2.2"><apply id="S2.E3.m1.1.1.1.1.1.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E3.m1.1.1.1.1.1.1.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1">superscript</csymbol><apply id="S2.E3.m1.1.1.1.1.1.1.2.cmml" xref="S2.E3.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E3.m1.1.1.1.1.1.1.2.1.cmml" xref="S2.E3.m1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.E3.m1.1.1.1.1.1.1.2.2.cmml" xref="S2.E3.m1.1.1.1.1.1.1.2.2">𝑥</ci><ci id="S2.E3.m1.1.1.1.1.1.1.2.3.cmml" xref="S2.E3.m1.1.1.1.1.1.1.2.3">𝑒</ci></apply><ci id="S2.E3.m1.1.1.1.1.1.1.3.cmml" xref="S2.E3.m1.1.1.1.1.1.1.3">′</ci></apply><apply id="S2.E3.m1.2.2.2.2.2.2.cmml" xref="S2.E3.m1.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.E3.m1.2.2.2.2.2.2.1.cmml" xref="S2.E3.m1.2.2.2.2.2.2">superscript</csymbol><apply id="S2.E3.m1.2.2.2.2.2.2.2.cmml" xref="S2.E3.m1.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.E3.m1.2.2.2.2.2.2.2.1.cmml" xref="S2.E3.m1.2.2.2.2.2.2">subscript</csymbol><ci id="S2.E3.m1.2.2.2.2.2.2.2.2.cmml" xref="S2.E3.m1.2.2.2.2.2.2.2.2">𝑦</ci><ci id="S2.E3.m1.2.2.2.2.2.2.2.3.cmml" xref="S2.E3.m1.2.2.2.2.2.2.2.3">𝑒</ci></apply><ci id="S2.E3.m1.2.2.2.2.2.2.3.cmml" xref="S2.E3.m1.2.2.2.2.2.2.3">′</ci></apply></interval><apply id="S2.E3.m1.4.4.4.4.cmml" xref="S2.E3.m1.4.4.4.4"><times id="S2.E3.m1.4.4.4.4.3.cmml" xref="S2.E3.m1.4.4.4.4.3"></times><ci id="S2.E3.m1.4.4.4.4.4.cmml" xref="S2.E3.m1.4.4.4.4.4">𝑁</ci><interval closure="open" id="S2.E3.m1.4.4.4.4.2.3.cmml" xref="S2.E3.m1.4.4.4.4.2.2"><apply id="S2.E3.m1.3.3.3.3.1.1.1.cmml" xref="S2.E3.m1.3.3.3.3.1.1.1"><csymbol cd="ambiguous" id="S2.E3.m1.3.3.3.3.1.1.1.1.cmml" xref="S2.E3.m1.3.3.3.3.1.1.1">subscript</csymbol><ci id="S2.E3.m1.3.3.3.3.1.1.1.2.cmml" xref="S2.E3.m1.3.3.3.3.1.1.1.2">𝑥</ci><ci id="S2.E3.m1.3.3.3.3.1.1.1.3.cmml" xref="S2.E3.m1.3.3.3.3.1.1.1.3">𝑒</ci></apply><apply id="S2.E3.m1.4.4.4.4.2.2.2.cmml" xref="S2.E3.m1.4.4.4.4.2.2.2"><csymbol cd="ambiguous" id="S2.E3.m1.4.4.4.4.2.2.2.1.cmml" xref="S2.E3.m1.4.4.4.4.2.2.2">subscript</csymbol><ci id="S2.E3.m1.4.4.4.4.2.2.2.2.cmml" xref="S2.E3.m1.4.4.4.4.2.2.2.2">𝑦</ci><ci id="S2.E3.m1.4.4.4.4.2.2.2.3.cmml" xref="S2.E3.m1.4.4.4.4.2.2.2.3">𝑒</ci></apply></interval></apply></apply></apply><cn type="integer" id="S2.E3.m1.5.5.1.1.4.cmml" xref="S2.E3.m1.5.5.1.1.4">1</cn><set id="S2.E3.m1.5.5.1.1.1.2.cmml" xref="S2.E3.m1.5.5.1.1.1.1"><apply id="S2.E3.m1.5.5.1.1.1.1.1.cmml" xref="S2.E3.m1.5.5.1.1.1.1.1"><eq id="S2.E3.m1.5.5.1.1.1.1.1.2.cmml" xref="S2.E3.m1.5.5.1.1.1.1.1.2"></eq><apply id="S2.E3.m1.5.5.1.1.1.1.1.1.cmml" xref="S2.E3.m1.5.5.1.1.1.1.1.1"><times id="S2.E3.m1.5.5.1.1.1.1.1.1.2.cmml" xref="S2.E3.m1.5.5.1.1.1.1.1.1.2"></times><apply id="S2.E3.m1.5.5.1.1.1.1.1.1.3.cmml" xref="S2.E3.m1.5.5.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E3.m1.5.5.1.1.1.1.1.1.3.1.cmml" xref="S2.E3.m1.5.5.1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.E3.m1.5.5.1.1.1.1.1.1.3.2.cmml" xref="S2.E3.m1.5.5.1.1.1.1.1.1.3.2">𝑓</ci><apply id="S2.E3.m1.5.5.1.1.1.1.1.1.3.3.cmml" xref="S2.E3.m1.5.5.1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S2.E3.m1.5.5.1.1.1.1.1.1.3.3.1.cmml" xref="S2.E3.m1.5.5.1.1.1.1.1.1.3.3">subscript</csymbol><ci id="S2.E3.m1.5.5.1.1.1.1.1.1.3.3.2.cmml" xref="S2.E3.m1.5.5.1.1.1.1.1.1.3.3.2">𝜃</ci><ci id="S2.E3.m1.5.5.1.1.1.1.1.1.3.3.3.cmml" xref="S2.E3.m1.5.5.1.1.1.1.1.1.3.3.3">𝑒</ci></apply></apply><apply id="S2.E3.m1.5.5.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E3.m1.5.5.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E3.m1.5.5.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E3.m1.5.5.1.1.1.1.1.1.1.1">superscript</csymbol><apply id="S2.E3.m1.5.5.1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.E3.m1.5.5.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E3.m1.5.5.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S2.E3.m1.5.5.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.E3.m1.5.5.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S2.E3.m1.5.5.1.1.1.1.1.1.1.1.1.2.2">𝑥</ci><ci id="S2.E3.m1.5.5.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S2.E3.m1.5.5.1.1.1.1.1.1.1.1.1.2.3">𝑒</ci></apply><ci id="S2.E3.m1.5.5.1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.E3.m1.5.5.1.1.1.1.1.1.1.1.1.3">′</ci></apply></apply><apply id="S2.E3.m1.5.5.1.1.1.1.1.3.cmml" xref="S2.E3.m1.5.5.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E3.m1.5.5.1.1.1.1.1.3.1.cmml" xref="S2.E3.m1.5.5.1.1.1.1.1.3">superscript</csymbol><apply id="S2.E3.m1.5.5.1.1.1.1.1.3.2.cmml" xref="S2.E3.m1.5.5.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E3.m1.5.5.1.1.1.1.1.3.2.1.cmml" xref="S2.E3.m1.5.5.1.1.1.1.1.3">subscript</csymbol><ci id="S2.E3.m1.5.5.1.1.1.1.1.3.2.2.cmml" xref="S2.E3.m1.5.5.1.1.1.1.1.3.2.2">𝑦</ci><ci id="S2.E3.m1.5.5.1.1.1.1.1.3.2.3.cmml" xref="S2.E3.m1.5.5.1.1.1.1.1.3.2.3">𝑒</ci></apply><ci id="S2.E3.m1.5.5.1.1.1.1.1.3.3.cmml" xref="S2.E3.m1.5.5.1.1.1.1.1.3.3">′</ci></apply></apply></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E3.m1.5c">\mathbb{E}_{(x_{e}^{\prime},y_{e}^{\prime})\sim N(x_{e},y_{e})}\mathbbm{1}\{f_{\theta_{e}}(x_{e}^{\prime})=y_{e}^{\prime}\},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
</table>
<p id="S2.SS1.SSS0.Px2.p1.2" class="ltx_p">where <math id="S2.SS1.SSS0.Px2.p1.1.m1.2" class="ltx_Math" alttext="N(x_{e},y_{e})" display="inline"><semantics id="S2.SS1.SSS0.Px2.p1.1.m1.2a"><mrow id="S2.SS1.SSS0.Px2.p1.1.m1.2.2" xref="S2.SS1.SSS0.Px2.p1.1.m1.2.2.cmml"><mi id="S2.SS1.SSS0.Px2.p1.1.m1.2.2.4" xref="S2.SS1.SSS0.Px2.p1.1.m1.2.2.4.cmml">N</mi><mo lspace="0em" rspace="0em" id="S2.SS1.SSS0.Px2.p1.1.m1.2.2.3" xref="S2.SS1.SSS0.Px2.p1.1.m1.2.2.3.cmml">​</mo><mrow id="S2.SS1.SSS0.Px2.p1.1.m1.2.2.2.2" xref="S2.SS1.SSS0.Px2.p1.1.m1.2.2.2.3.cmml"><mo stretchy="false" id="S2.SS1.SSS0.Px2.p1.1.m1.2.2.2.2.3" xref="S2.SS1.SSS0.Px2.p1.1.m1.2.2.2.3.cmml">(</mo><msub id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.cmml"><mi id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.2" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.2.cmml">x</mi><mi id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.3" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.3.cmml">e</mi></msub><mo id="S2.SS1.SSS0.Px2.p1.1.m1.2.2.2.2.4" xref="S2.SS1.SSS0.Px2.p1.1.m1.2.2.2.3.cmml">,</mo><msub id="S2.SS1.SSS0.Px2.p1.1.m1.2.2.2.2.2" xref="S2.SS1.SSS0.Px2.p1.1.m1.2.2.2.2.2.cmml"><mi id="S2.SS1.SSS0.Px2.p1.1.m1.2.2.2.2.2.2" xref="S2.SS1.SSS0.Px2.p1.1.m1.2.2.2.2.2.2.cmml">y</mi><mi id="S2.SS1.SSS0.Px2.p1.1.m1.2.2.2.2.2.3" xref="S2.SS1.SSS0.Px2.p1.1.m1.2.2.2.2.2.3.cmml">e</mi></msub><mo stretchy="false" id="S2.SS1.SSS0.Px2.p1.1.m1.2.2.2.2.5" xref="S2.SS1.SSS0.Px2.p1.1.m1.2.2.2.3.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px2.p1.1.m1.2b"><apply id="S2.SS1.SSS0.Px2.p1.1.m1.2.2.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.2.2"><times id="S2.SS1.SSS0.Px2.p1.1.m1.2.2.3.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.2.2.3"></times><ci id="S2.SS1.SSS0.Px2.p1.1.m1.2.2.4.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.2.2.4">𝑁</ci><interval closure="open" id="S2.SS1.SSS0.Px2.p1.1.m1.2.2.2.3.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.2.2.2.2"><apply id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.2.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.2">𝑥</ci><ci id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.3.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.3">𝑒</ci></apply><apply id="S2.SS1.SSS0.Px2.p1.1.m1.2.2.2.2.2.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px2.p1.1.m1.2.2.2.2.2.1.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.2.2.2.2.2">subscript</csymbol><ci id="S2.SS1.SSS0.Px2.p1.1.m1.2.2.2.2.2.2.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.2.2.2.2.2.2">𝑦</ci><ci id="S2.SS1.SSS0.Px2.p1.1.m1.2.2.2.2.2.3.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.2.2.2.2.2.3">𝑒</ci></apply></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px2.p1.1.m1.2c">N(x_{e},y_{e})</annotation></semantics></math> is the set of neighboring instances of an edit query <math id="S2.SS1.SSS0.Px2.p1.2.m2.2" class="ltx_Math" alttext="(x_{e},y_{e})" display="inline"><semantics id="S2.SS1.SSS0.Px2.p1.2.m2.2a"><mrow id="S2.SS1.SSS0.Px2.p1.2.m2.2.2.2" xref="S2.SS1.SSS0.Px2.p1.2.m2.2.2.3.cmml"><mo stretchy="false" id="S2.SS1.SSS0.Px2.p1.2.m2.2.2.2.3" xref="S2.SS1.SSS0.Px2.p1.2.m2.2.2.3.cmml">(</mo><msub id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.cmml"><mi id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.2" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.2.cmml">x</mi><mi id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.3" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.3.cmml">e</mi></msub><mo id="S2.SS1.SSS0.Px2.p1.2.m2.2.2.2.4" xref="S2.SS1.SSS0.Px2.p1.2.m2.2.2.3.cmml">,</mo><msub id="S2.SS1.SSS0.Px2.p1.2.m2.2.2.2.2" xref="S2.SS1.SSS0.Px2.p1.2.m2.2.2.2.2.cmml"><mi id="S2.SS1.SSS0.Px2.p1.2.m2.2.2.2.2.2" xref="S2.SS1.SSS0.Px2.p1.2.m2.2.2.2.2.2.cmml">y</mi><mi id="S2.SS1.SSS0.Px2.p1.2.m2.2.2.2.2.3" xref="S2.SS1.SSS0.Px2.p1.2.m2.2.2.2.2.3.cmml">e</mi></msub><mo stretchy="false" id="S2.SS1.SSS0.Px2.p1.2.m2.2.2.2.5" xref="S2.SS1.SSS0.Px2.p1.2.m2.2.2.3.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px2.p1.2.m2.2b"><interval closure="open" id="S2.SS1.SSS0.Px2.p1.2.m2.2.2.3.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.2.2.2"><apply id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.1.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1">subscript</csymbol><ci id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.2.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.2">𝑥</ci><ci id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.3.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.3">𝑒</ci></apply><apply id="S2.SS1.SSS0.Px2.p1.2.m2.2.2.2.2.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.2.2.2.2"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px2.p1.2.m2.2.2.2.2.1.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.2.2.2.2">subscript</csymbol><ci id="S2.SS1.SSS0.Px2.p1.2.m2.2.2.2.2.2.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.2.2.2.2.2">𝑦</ci><ci id="S2.SS1.SSS0.Px2.p1.2.m2.2.2.2.2.3.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.2.2.2.2.3">𝑒</ci></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px2.p1.2.m2.2c">(x_{e},y_{e})</annotation></semantics></math>. Earlier works evaluate this metric by rephrasing the input prompts <cite class="ltx_cite ltx_citemacro_citep">(Mitchell et al., <a href="#bib.bib32" title="" class="ltx_ref">2022a</a>; Meng et al., <a href="#bib.bib29" title="" class="ltx_ref">2022</a>; Huang et al., <a href="#bib.bib21" title="" class="ltx_ref">2023</a>)</cite>.</p>
</div>
</section>
<section id="S2.SS1.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Locality</h5>

<div id="S2.SS1.SSS0.Px3.p1" class="ltx_para">
<p id="S2.SS1.SSS0.Px3.p1.2" class="ltx_p">The editing process should not affect instances unrelated to the edit queries. The locality set of an edit query <math id="S2.SS1.SSS0.Px3.p1.1.m1.2" class="ltx_Math" alttext="(x_{e},y_{e})" display="inline"><semantics id="S2.SS1.SSS0.Px3.p1.1.m1.2a"><mrow id="S2.SS1.SSS0.Px3.p1.1.m1.2.2.2" xref="S2.SS1.SSS0.Px3.p1.1.m1.2.2.3.cmml"><mo stretchy="false" id="S2.SS1.SSS0.Px3.p1.1.m1.2.2.2.3" xref="S2.SS1.SSS0.Px3.p1.1.m1.2.2.3.cmml">(</mo><msub id="S2.SS1.SSS0.Px3.p1.1.m1.1.1.1.1" xref="S2.SS1.SSS0.Px3.p1.1.m1.1.1.1.1.cmml"><mi id="S2.SS1.SSS0.Px3.p1.1.m1.1.1.1.1.2" xref="S2.SS1.SSS0.Px3.p1.1.m1.1.1.1.1.2.cmml">x</mi><mi id="S2.SS1.SSS0.Px3.p1.1.m1.1.1.1.1.3" xref="S2.SS1.SSS0.Px3.p1.1.m1.1.1.1.1.3.cmml">e</mi></msub><mo id="S2.SS1.SSS0.Px3.p1.1.m1.2.2.2.4" xref="S2.SS1.SSS0.Px3.p1.1.m1.2.2.3.cmml">,</mo><msub id="S2.SS1.SSS0.Px3.p1.1.m1.2.2.2.2" xref="S2.SS1.SSS0.Px3.p1.1.m1.2.2.2.2.cmml"><mi id="S2.SS1.SSS0.Px3.p1.1.m1.2.2.2.2.2" xref="S2.SS1.SSS0.Px3.p1.1.m1.2.2.2.2.2.cmml">y</mi><mi id="S2.SS1.SSS0.Px3.p1.1.m1.2.2.2.2.3" xref="S2.SS1.SSS0.Px3.p1.1.m1.2.2.2.2.3.cmml">e</mi></msub><mo stretchy="false" id="S2.SS1.SSS0.Px3.p1.1.m1.2.2.2.5" xref="S2.SS1.SSS0.Px3.p1.1.m1.2.2.3.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px3.p1.1.m1.2b"><interval closure="open" id="S2.SS1.SSS0.Px3.p1.1.m1.2.2.3.cmml" xref="S2.SS1.SSS0.Px3.p1.1.m1.2.2.2"><apply id="S2.SS1.SSS0.Px3.p1.1.m1.1.1.1.1.cmml" xref="S2.SS1.SSS0.Px3.p1.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px3.p1.1.m1.1.1.1.1.1.cmml" xref="S2.SS1.SSS0.Px3.p1.1.m1.1.1.1.1">subscript</csymbol><ci id="S2.SS1.SSS0.Px3.p1.1.m1.1.1.1.1.2.cmml" xref="S2.SS1.SSS0.Px3.p1.1.m1.1.1.1.1.2">𝑥</ci><ci id="S2.SS1.SSS0.Px3.p1.1.m1.1.1.1.1.3.cmml" xref="S2.SS1.SSS0.Px3.p1.1.m1.1.1.1.1.3">𝑒</ci></apply><apply id="S2.SS1.SSS0.Px3.p1.1.m1.2.2.2.2.cmml" xref="S2.SS1.SSS0.Px3.p1.1.m1.2.2.2.2"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px3.p1.1.m1.2.2.2.2.1.cmml" xref="S2.SS1.SSS0.Px3.p1.1.m1.2.2.2.2">subscript</csymbol><ci id="S2.SS1.SSS0.Px3.p1.1.m1.2.2.2.2.2.cmml" xref="S2.SS1.SSS0.Px3.p1.1.m1.2.2.2.2.2">𝑦</ci><ci id="S2.SS1.SSS0.Px3.p1.1.m1.2.2.2.2.3.cmml" xref="S2.SS1.SSS0.Px3.p1.1.m1.2.2.2.2.3">𝑒</ci></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px3.p1.1.m1.2c">(x_{e},y_{e})</annotation></semantics></math> can be defined as <math id="S2.SS1.SSS0.Px3.p1.2.m2.1" class="ltx_math_unparsed" alttext="L(x_{e})=\{(x_{loc},y_{loc})\in\mathbb{X}\times\mathbb{Y}\ \mathrm{s.t}\ x_{loc}\notin N(x_{e},y_{e})\land f_{\theta_{0}}(x_{loc})=y_{loc}\}" display="inline"><semantics id="S2.SS1.SSS0.Px3.p1.2.m2.1a"><mrow id="S2.SS1.SSS0.Px3.p1.2.m2.1b"><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.1">L</mi><mrow id="S2.SS1.SSS0.Px3.p1.2.m2.1.2"><mo stretchy="false" id="S2.SS1.SSS0.Px3.p1.2.m2.1.2.1">(</mo><msub id="S2.SS1.SSS0.Px3.p1.2.m2.1.2.2"><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.2.2.2">x</mi><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.2.2.3">e</mi></msub><mo stretchy="false" id="S2.SS1.SSS0.Px3.p1.2.m2.1.2.3">)</mo></mrow><mo id="S2.SS1.SSS0.Px3.p1.2.m2.1.3">=</mo><mrow id="S2.SS1.SSS0.Px3.p1.2.m2.1.4"><mo stretchy="false" id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.1">{</mo><mrow id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.2"><mo stretchy="false" id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.2.1">(</mo><msub id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.2.2"><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.2.2.2">x</mi><mrow id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.2.2.3"><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.2.2.3.2">l</mi><mo lspace="0em" rspace="0em" id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.2.2.3.1">​</mo><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.2.2.3.3">o</mi><mo lspace="0em" rspace="0em" id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.2.2.3.1a">​</mo><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.2.2.3.4">c</mi></mrow></msub><mo id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.2.3">,</mo><msub id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.2.4"><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.2.4.2">y</mi><mrow id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.2.4.3"><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.2.4.3.2">l</mi><mo lspace="0em" rspace="0em" id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.2.4.3.1">​</mo><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.2.4.3.3">o</mi><mo lspace="0em" rspace="0em" id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.2.4.3.1a">​</mo><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.2.4.3.4">c</mi></mrow></msub><mo stretchy="false" id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.2.5">)</mo></mrow><mo id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.3">∈</mo><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.4">𝕏</mi><mo lspace="0.222em" rspace="0.222em" id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.5">×</mo><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.6">𝕐</mi><mi mathvariant="normal" id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.7">s</mi><mo lspace="0em" rspace="0.167em" id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.8">.</mo><mi mathvariant="normal" id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.9">t</mi><msub id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.10"><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.10.2">x</mi><mrow id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.10.3"><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.10.3.2">l</mi><mo lspace="0em" rspace="0em" id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.10.3.1">​</mo><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.10.3.3">o</mi><mo lspace="0em" rspace="0em" id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.10.3.1a">​</mo><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.10.3.4">c</mi></mrow></msub><mo id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.11">∉</mo><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.12">N</mi><mrow id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.13"><mo stretchy="false" id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.13.1">(</mo><msub id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.13.2"><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.13.2.2">x</mi><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.13.2.3">e</mi></msub><mo id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.13.3">,</mo><msub id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.13.4"><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.13.4.2">y</mi><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.13.4.3">e</mi></msub><mo stretchy="false" id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.13.5">)</mo></mrow><mo id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.14">∧</mo><msub id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.15"><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.15.2">f</mi><msub id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.15.3"><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.15.3.2">θ</mi><mn id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.15.3.3">0</mn></msub></msub><mrow id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.16"><mo stretchy="false" id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.16.1">(</mo><msub id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.16.2"><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.16.2.2">x</mi><mrow id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.16.2.3"><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.16.2.3.2">l</mi><mo lspace="0em" rspace="0em" id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.16.2.3.1">​</mo><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.16.2.3.3">o</mi><mo lspace="0em" rspace="0em" id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.16.2.3.1a">​</mo><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.16.2.3.4">c</mi></mrow></msub><mo stretchy="false" id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.16.3">)</mo></mrow><mo id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.17">=</mo><msub id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.18"><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.18.2">y</mi><mrow id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.18.3"><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.18.3.2">l</mi><mo lspace="0em" rspace="0em" id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.18.3.1">​</mo><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.18.3.3">o</mi><mo lspace="0em" rspace="0em" id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.18.3.1a">​</mo><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.18.3.4">c</mi></mrow></msub><mo stretchy="false" id="S2.SS1.SSS0.Px3.p1.2.m2.1.4.19">}</mo></mrow></mrow><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px3.p1.2.m2.1c">L(x_{e})=\{(x_{loc},y_{loc})\in\mathbb{X}\times\mathbb{Y}\ \mathrm{s.t}\ x_{loc}\notin N(x_{e},y_{e})\land f_{\theta_{0}}(x_{loc})=y_{loc}\}</annotation></semantics></math>. The locality, also known as specificity, of a editing method is measured by calculating the level of invariance of model output before and after the edits, which can be calculated as follows:</p>
<table id="S2.E4" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E4.m1.4" class="ltx_Math" alttext="\mathbb{E}_{(x_{loc},y_{loc})\sim L(x_{e})}\mathbbm{1}\{f_{\theta_{e}}(x_{loc})=y_{loc}\}" display="block"><semantics id="S2.E4.m1.4a"><mrow id="S2.E4.m1.4.4" xref="S2.E4.m1.4.4.cmml"><msub id="S2.E4.m1.4.4.3" xref="S2.E4.m1.4.4.3.cmml"><mi id="S2.E4.m1.4.4.3.2" xref="S2.E4.m1.4.4.3.2.cmml">𝔼</mi><mrow id="S2.E4.m1.3.3.3" xref="S2.E4.m1.3.3.3.cmml"><mrow id="S2.E4.m1.2.2.2.2.2" xref="S2.E4.m1.2.2.2.2.3.cmml"><mo stretchy="false" id="S2.E4.m1.2.2.2.2.2.3" xref="S2.E4.m1.2.2.2.2.3.cmml">(</mo><msub id="S2.E4.m1.1.1.1.1.1.1" xref="S2.E4.m1.1.1.1.1.1.1.cmml"><mi id="S2.E4.m1.1.1.1.1.1.1.2" xref="S2.E4.m1.1.1.1.1.1.1.2.cmml">x</mi><mrow id="S2.E4.m1.1.1.1.1.1.1.3" xref="S2.E4.m1.1.1.1.1.1.1.3.cmml"><mi id="S2.E4.m1.1.1.1.1.1.1.3.2" xref="S2.E4.m1.1.1.1.1.1.1.3.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="S2.E4.m1.1.1.1.1.1.1.3.1" xref="S2.E4.m1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="S2.E4.m1.1.1.1.1.1.1.3.3" xref="S2.E4.m1.1.1.1.1.1.1.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S2.E4.m1.1.1.1.1.1.1.3.1a" xref="S2.E4.m1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="S2.E4.m1.1.1.1.1.1.1.3.4" xref="S2.E4.m1.1.1.1.1.1.1.3.4.cmml">c</mi></mrow></msub><mo id="S2.E4.m1.2.2.2.2.2.4" xref="S2.E4.m1.2.2.2.2.3.cmml">,</mo><msub id="S2.E4.m1.2.2.2.2.2.2" xref="S2.E4.m1.2.2.2.2.2.2.cmml"><mi id="S2.E4.m1.2.2.2.2.2.2.2" xref="S2.E4.m1.2.2.2.2.2.2.2.cmml">y</mi><mrow id="S2.E4.m1.2.2.2.2.2.2.3" xref="S2.E4.m1.2.2.2.2.2.2.3.cmml"><mi id="S2.E4.m1.2.2.2.2.2.2.3.2" xref="S2.E4.m1.2.2.2.2.2.2.3.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="S2.E4.m1.2.2.2.2.2.2.3.1" xref="S2.E4.m1.2.2.2.2.2.2.3.1.cmml">​</mo><mi id="S2.E4.m1.2.2.2.2.2.2.3.3" xref="S2.E4.m1.2.2.2.2.2.2.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S2.E4.m1.2.2.2.2.2.2.3.1a" xref="S2.E4.m1.2.2.2.2.2.2.3.1.cmml">​</mo><mi id="S2.E4.m1.2.2.2.2.2.2.3.4" xref="S2.E4.m1.2.2.2.2.2.2.3.4.cmml">c</mi></mrow></msub><mo stretchy="false" id="S2.E4.m1.2.2.2.2.2.5" xref="S2.E4.m1.2.2.2.2.3.cmml">)</mo></mrow><mo id="S2.E4.m1.3.3.3.4" xref="S2.E4.m1.3.3.3.4.cmml">∼</mo><mrow id="S2.E4.m1.3.3.3.3" xref="S2.E4.m1.3.3.3.3.cmml"><mi id="S2.E4.m1.3.3.3.3.3" xref="S2.E4.m1.3.3.3.3.3.cmml">L</mi><mo lspace="0em" rspace="0em" id="S2.E4.m1.3.3.3.3.2" xref="S2.E4.m1.3.3.3.3.2.cmml">​</mo><mrow id="S2.E4.m1.3.3.3.3.1.1" xref="S2.E4.m1.3.3.3.3.1.1.1.cmml"><mo stretchy="false" id="S2.E4.m1.3.3.3.3.1.1.2" xref="S2.E4.m1.3.3.3.3.1.1.1.cmml">(</mo><msub id="S2.E4.m1.3.3.3.3.1.1.1" xref="S2.E4.m1.3.3.3.3.1.1.1.cmml"><mi id="S2.E4.m1.3.3.3.3.1.1.1.2" xref="S2.E4.m1.3.3.3.3.1.1.1.2.cmml">x</mi><mi id="S2.E4.m1.3.3.3.3.1.1.1.3" xref="S2.E4.m1.3.3.3.3.1.1.1.3.cmml">e</mi></msub><mo stretchy="false" id="S2.E4.m1.3.3.3.3.1.1.3" xref="S2.E4.m1.3.3.3.3.1.1.1.cmml">)</mo></mrow></mrow></mrow></msub><mo lspace="0em" rspace="0em" id="S2.E4.m1.4.4.2" xref="S2.E4.m1.4.4.2.cmml">​</mo><mn id="S2.E4.m1.4.4.4" xref="S2.E4.m1.4.4.4.cmml">𝟙</mn><mo lspace="0em" rspace="0em" id="S2.E4.m1.4.4.2a" xref="S2.E4.m1.4.4.2.cmml">​</mo><mrow id="S2.E4.m1.4.4.1.1" xref="S2.E4.m1.4.4.1.2.cmml"><mo stretchy="false" id="S2.E4.m1.4.4.1.1.2" xref="S2.E4.m1.4.4.1.2.cmml">{</mo><mrow id="S2.E4.m1.4.4.1.1.1" xref="S2.E4.m1.4.4.1.1.1.cmml"><mrow id="S2.E4.m1.4.4.1.1.1.1" xref="S2.E4.m1.4.4.1.1.1.1.cmml"><msub id="S2.E4.m1.4.4.1.1.1.1.3" xref="S2.E4.m1.4.4.1.1.1.1.3.cmml"><mi id="S2.E4.m1.4.4.1.1.1.1.3.2" xref="S2.E4.m1.4.4.1.1.1.1.3.2.cmml">f</mi><msub id="S2.E4.m1.4.4.1.1.1.1.3.3" xref="S2.E4.m1.4.4.1.1.1.1.3.3.cmml"><mi id="S2.E4.m1.4.4.1.1.1.1.3.3.2" xref="S2.E4.m1.4.4.1.1.1.1.3.3.2.cmml">θ</mi><mi id="S2.E4.m1.4.4.1.1.1.1.3.3.3" xref="S2.E4.m1.4.4.1.1.1.1.3.3.3.cmml">e</mi></msub></msub><mo lspace="0em" rspace="0em" id="S2.E4.m1.4.4.1.1.1.1.2" xref="S2.E4.m1.4.4.1.1.1.1.2.cmml">​</mo><mrow id="S2.E4.m1.4.4.1.1.1.1.1.1" xref="S2.E4.m1.4.4.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.E4.m1.4.4.1.1.1.1.1.1.2" xref="S2.E4.m1.4.4.1.1.1.1.1.1.1.cmml">(</mo><msub id="S2.E4.m1.4.4.1.1.1.1.1.1.1" xref="S2.E4.m1.4.4.1.1.1.1.1.1.1.cmml"><mi id="S2.E4.m1.4.4.1.1.1.1.1.1.1.2" xref="S2.E4.m1.4.4.1.1.1.1.1.1.1.2.cmml">x</mi><mrow id="S2.E4.m1.4.4.1.1.1.1.1.1.1.3" xref="S2.E4.m1.4.4.1.1.1.1.1.1.1.3.cmml"><mi id="S2.E4.m1.4.4.1.1.1.1.1.1.1.3.2" xref="S2.E4.m1.4.4.1.1.1.1.1.1.1.3.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="S2.E4.m1.4.4.1.1.1.1.1.1.1.3.1" xref="S2.E4.m1.4.4.1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="S2.E4.m1.4.4.1.1.1.1.1.1.1.3.3" xref="S2.E4.m1.4.4.1.1.1.1.1.1.1.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S2.E4.m1.4.4.1.1.1.1.1.1.1.3.1a" xref="S2.E4.m1.4.4.1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="S2.E4.m1.4.4.1.1.1.1.1.1.1.3.4" xref="S2.E4.m1.4.4.1.1.1.1.1.1.1.3.4.cmml">c</mi></mrow></msub><mo stretchy="false" id="S2.E4.m1.4.4.1.1.1.1.1.1.3" xref="S2.E4.m1.4.4.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S2.E4.m1.4.4.1.1.1.2" xref="S2.E4.m1.4.4.1.1.1.2.cmml">=</mo><msub id="S2.E4.m1.4.4.1.1.1.3" xref="S2.E4.m1.4.4.1.1.1.3.cmml"><mi id="S2.E4.m1.4.4.1.1.1.3.2" xref="S2.E4.m1.4.4.1.1.1.3.2.cmml">y</mi><mrow id="S2.E4.m1.4.4.1.1.1.3.3" xref="S2.E4.m1.4.4.1.1.1.3.3.cmml"><mi id="S2.E4.m1.4.4.1.1.1.3.3.2" xref="S2.E4.m1.4.4.1.1.1.3.3.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="S2.E4.m1.4.4.1.1.1.3.3.1" xref="S2.E4.m1.4.4.1.1.1.3.3.1.cmml">​</mo><mi id="S2.E4.m1.4.4.1.1.1.3.3.3" xref="S2.E4.m1.4.4.1.1.1.3.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S2.E4.m1.4.4.1.1.1.3.3.1a" xref="S2.E4.m1.4.4.1.1.1.3.3.1.cmml">​</mo><mi id="S2.E4.m1.4.4.1.1.1.3.3.4" xref="S2.E4.m1.4.4.1.1.1.3.3.4.cmml">c</mi></mrow></msub></mrow><mo stretchy="false" id="S2.E4.m1.4.4.1.1.3" xref="S2.E4.m1.4.4.1.2.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E4.m1.4b"><apply id="S2.E4.m1.4.4.cmml" xref="S2.E4.m1.4.4"><times id="S2.E4.m1.4.4.2.cmml" xref="S2.E4.m1.4.4.2"></times><apply id="S2.E4.m1.4.4.3.cmml" xref="S2.E4.m1.4.4.3"><csymbol cd="ambiguous" id="S2.E4.m1.4.4.3.1.cmml" xref="S2.E4.m1.4.4.3">subscript</csymbol><ci id="S2.E4.m1.4.4.3.2.cmml" xref="S2.E4.m1.4.4.3.2">𝔼</ci><apply id="S2.E4.m1.3.3.3.cmml" xref="S2.E4.m1.3.3.3"><csymbol cd="latexml" id="S2.E4.m1.3.3.3.4.cmml" xref="S2.E4.m1.3.3.3.4">similar-to</csymbol><interval closure="open" id="S2.E4.m1.2.2.2.2.3.cmml" xref="S2.E4.m1.2.2.2.2.2"><apply id="S2.E4.m1.1.1.1.1.1.1.cmml" xref="S2.E4.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E4.m1.1.1.1.1.1.1.1.cmml" xref="S2.E4.m1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.E4.m1.1.1.1.1.1.1.2.cmml" xref="S2.E4.m1.1.1.1.1.1.1.2">𝑥</ci><apply id="S2.E4.m1.1.1.1.1.1.1.3.cmml" xref="S2.E4.m1.1.1.1.1.1.1.3"><times id="S2.E4.m1.1.1.1.1.1.1.3.1.cmml" xref="S2.E4.m1.1.1.1.1.1.1.3.1"></times><ci id="S2.E4.m1.1.1.1.1.1.1.3.2.cmml" xref="S2.E4.m1.1.1.1.1.1.1.3.2">𝑙</ci><ci id="S2.E4.m1.1.1.1.1.1.1.3.3.cmml" xref="S2.E4.m1.1.1.1.1.1.1.3.3">𝑜</ci><ci id="S2.E4.m1.1.1.1.1.1.1.3.4.cmml" xref="S2.E4.m1.1.1.1.1.1.1.3.4">𝑐</ci></apply></apply><apply id="S2.E4.m1.2.2.2.2.2.2.cmml" xref="S2.E4.m1.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.E4.m1.2.2.2.2.2.2.1.cmml" xref="S2.E4.m1.2.2.2.2.2.2">subscript</csymbol><ci id="S2.E4.m1.2.2.2.2.2.2.2.cmml" xref="S2.E4.m1.2.2.2.2.2.2.2">𝑦</ci><apply id="S2.E4.m1.2.2.2.2.2.2.3.cmml" xref="S2.E4.m1.2.2.2.2.2.2.3"><times id="S2.E4.m1.2.2.2.2.2.2.3.1.cmml" xref="S2.E4.m1.2.2.2.2.2.2.3.1"></times><ci id="S2.E4.m1.2.2.2.2.2.2.3.2.cmml" xref="S2.E4.m1.2.2.2.2.2.2.3.2">𝑙</ci><ci id="S2.E4.m1.2.2.2.2.2.2.3.3.cmml" xref="S2.E4.m1.2.2.2.2.2.2.3.3">𝑜</ci><ci id="S2.E4.m1.2.2.2.2.2.2.3.4.cmml" xref="S2.E4.m1.2.2.2.2.2.2.3.4">𝑐</ci></apply></apply></interval><apply id="S2.E4.m1.3.3.3.3.cmml" xref="S2.E4.m1.3.3.3.3"><times id="S2.E4.m1.3.3.3.3.2.cmml" xref="S2.E4.m1.3.3.3.3.2"></times><ci id="S2.E4.m1.3.3.3.3.3.cmml" xref="S2.E4.m1.3.3.3.3.3">𝐿</ci><apply id="S2.E4.m1.3.3.3.3.1.1.1.cmml" xref="S2.E4.m1.3.3.3.3.1.1"><csymbol cd="ambiguous" id="S2.E4.m1.3.3.3.3.1.1.1.1.cmml" xref="S2.E4.m1.3.3.3.3.1.1">subscript</csymbol><ci id="S2.E4.m1.3.3.3.3.1.1.1.2.cmml" xref="S2.E4.m1.3.3.3.3.1.1.1.2">𝑥</ci><ci id="S2.E4.m1.3.3.3.3.1.1.1.3.cmml" xref="S2.E4.m1.3.3.3.3.1.1.1.3">𝑒</ci></apply></apply></apply></apply><cn type="integer" id="S2.E4.m1.4.4.4.cmml" xref="S2.E4.m1.4.4.4">1</cn><set id="S2.E4.m1.4.4.1.2.cmml" xref="S2.E4.m1.4.4.1.1"><apply id="S2.E4.m1.4.4.1.1.1.cmml" xref="S2.E4.m1.4.4.1.1.1"><eq id="S2.E4.m1.4.4.1.1.1.2.cmml" xref="S2.E4.m1.4.4.1.1.1.2"></eq><apply id="S2.E4.m1.4.4.1.1.1.1.cmml" xref="S2.E4.m1.4.4.1.1.1.1"><times id="S2.E4.m1.4.4.1.1.1.1.2.cmml" xref="S2.E4.m1.4.4.1.1.1.1.2"></times><apply id="S2.E4.m1.4.4.1.1.1.1.3.cmml" xref="S2.E4.m1.4.4.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E4.m1.4.4.1.1.1.1.3.1.cmml" xref="S2.E4.m1.4.4.1.1.1.1.3">subscript</csymbol><ci id="S2.E4.m1.4.4.1.1.1.1.3.2.cmml" xref="S2.E4.m1.4.4.1.1.1.1.3.2">𝑓</ci><apply id="S2.E4.m1.4.4.1.1.1.1.3.3.cmml" xref="S2.E4.m1.4.4.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S2.E4.m1.4.4.1.1.1.1.3.3.1.cmml" xref="S2.E4.m1.4.4.1.1.1.1.3.3">subscript</csymbol><ci id="S2.E4.m1.4.4.1.1.1.1.3.3.2.cmml" xref="S2.E4.m1.4.4.1.1.1.1.3.3.2">𝜃</ci><ci id="S2.E4.m1.4.4.1.1.1.1.3.3.3.cmml" xref="S2.E4.m1.4.4.1.1.1.1.3.3.3">𝑒</ci></apply></apply><apply id="S2.E4.m1.4.4.1.1.1.1.1.1.1.cmml" xref="S2.E4.m1.4.4.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E4.m1.4.4.1.1.1.1.1.1.1.1.cmml" xref="S2.E4.m1.4.4.1.1.1.1.1.1">subscript</csymbol><ci id="S2.E4.m1.4.4.1.1.1.1.1.1.1.2.cmml" xref="S2.E4.m1.4.4.1.1.1.1.1.1.1.2">𝑥</ci><apply id="S2.E4.m1.4.4.1.1.1.1.1.1.1.3.cmml" xref="S2.E4.m1.4.4.1.1.1.1.1.1.1.3"><times id="S2.E4.m1.4.4.1.1.1.1.1.1.1.3.1.cmml" xref="S2.E4.m1.4.4.1.1.1.1.1.1.1.3.1"></times><ci id="S2.E4.m1.4.4.1.1.1.1.1.1.1.3.2.cmml" xref="S2.E4.m1.4.4.1.1.1.1.1.1.1.3.2">𝑙</ci><ci id="S2.E4.m1.4.4.1.1.1.1.1.1.1.3.3.cmml" xref="S2.E4.m1.4.4.1.1.1.1.1.1.1.3.3">𝑜</ci><ci id="S2.E4.m1.4.4.1.1.1.1.1.1.1.3.4.cmml" xref="S2.E4.m1.4.4.1.1.1.1.1.1.1.3.4">𝑐</ci></apply></apply></apply><apply id="S2.E4.m1.4.4.1.1.1.3.cmml" xref="S2.E4.m1.4.4.1.1.1.3"><csymbol cd="ambiguous" id="S2.E4.m1.4.4.1.1.1.3.1.cmml" xref="S2.E4.m1.4.4.1.1.1.3">subscript</csymbol><ci id="S2.E4.m1.4.4.1.1.1.3.2.cmml" xref="S2.E4.m1.4.4.1.1.1.3.2">𝑦</ci><apply id="S2.E4.m1.4.4.1.1.1.3.3.cmml" xref="S2.E4.m1.4.4.1.1.1.3.3"><times id="S2.E4.m1.4.4.1.1.1.3.3.1.cmml" xref="S2.E4.m1.4.4.1.1.1.3.3.1"></times><ci id="S2.E4.m1.4.4.1.1.1.3.3.2.cmml" xref="S2.E4.m1.4.4.1.1.1.3.3.2">𝑙</ci><ci id="S2.E4.m1.4.4.1.1.1.3.3.3.cmml" xref="S2.E4.m1.4.4.1.1.1.3.3.3">𝑜</ci><ci id="S2.E4.m1.4.4.1.1.1.3.3.4.cmml" xref="S2.E4.m1.4.4.1.1.1.3.3.4">𝑐</ci></apply></apply></apply></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E4.m1.4c">\mathbb{E}_{(x_{loc},y_{loc})\sim L(x_{e})}\mathbbm{1}\{f_{\theta_{e}}(x_{loc})=y_{loc}\}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(4)</span></td>
</tr></tbody>
</table>
</div>
</section>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>Current Methods</h3>

<div id="S2.SS2.p1" class="ltx_para">
<p id="S2.SS2.p1.1" class="ltx_p">In this section, we introduce the current knowledge editing methods. The methods are categorized into parameter-modifying (Section <a href="#S2.SS2.SSS1" title="2.2.1 Parameter-Modifying ‣ 2.2 Current Methods ‣ 2 Overview of Knowledge Editing ‣ Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.2.1</span></a>) and parameter-preserving (Section <a href="#S2.SS2.SSS2" title="2.2.2 Parameter-Preserving ‣ 2.2 Current Methods ‣ 2 Overview of Knowledge Editing ‣ Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.2.2</span></a>) editing methods, each containing several strategies. An overview and illustration of current methods are included in Table <a href="#S1.T1" title="Table 1 ‣ 1 Introduction ‣ Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> and Figure <a href="#S1.F3" title="Figure 3 ‣ 1 Introduction ‣ Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, respectively.</p>
</div>
<section id="S2.SS2.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">2.2.1 </span>Parameter-Modifying</h4>

<div id="S2.SS2.SSS1.p1" class="ltx_para">
<p id="S2.SS2.SSS1.p1.1" class="ltx_p">This category of methods, including meta-learning methods and locate-and-edit strategies, update LLMs’ knowledge by modifying their parameters.</p>
</div>
<section id="S2.SS2.SSS1.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Meta-learning</h5>

<div id="S2.SS2.SSS1.Px1.p1" class="ltx_para">
<p id="S2.SS2.SSS1.Px1.p1.1" class="ltx_p">Meta-learning methods train a hyper-network to predict the update of network parameters. For instance, KnowledgeEditor <cite class="ltx_cite ltx_citemacro_citep">(Cao et al., <a href="#bib.bib5" title="" class="ltx_ref">2021</a>)</cite> trains a deep network to predict weight updates. MEND <cite class="ltx_cite ltx_citemacro_citep">(Mitchell et al., <a href="#bib.bib32" title="" class="ltx_ref">2022a</a>)</cite> decomposes the gradient matrix into two rank-one matrices and utilized a hyper-network to update these matrices, thereby accelerating the editing process. Built upon MEND, MALMEN <cite class="ltx_cite ltx_citemacro_citep">(Tan et al., <a href="#bib.bib36" title="" class="ltx_ref">2024</a>)</cite> refines the process by formulating the aggregation of parameter shifts into a least-squares problem, further improving the scalability of meta-learning methods.</p>
</div>
</section>
<section id="S2.SS2.SSS1.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Locate and Edit</h5>

<div id="S2.SS2.SSS1.Px2.p1" class="ltx_para">
<p id="S2.SS2.SSS1.Px2.p1.1" class="ltx_p">Locate-and-edit methods identify specific knowledge locations in LLMs for consequent editing. KN <cite class="ltx_cite ltx_citemacro_citep">(Dai et al., <a href="#bib.bib9" title="" class="ltx_ref">2022</a>)</cite> utilizes the proposed knowledge attribution method to pinpoint neurons expressing relational facts, allowing efficient updates or erasures without fine-tuning. ROME <cite class="ltx_cite ltx_citemacro_citep">(Meng et al., <a href="#bib.bib29" title="" class="ltx_ref">2022</a>)</cite> proposes the causal tracing method for identifying neuron activations associating with specific knowledge. The authors demonstrate the significance of middle-layer feed-forward networks (FFNs) in factual predictions when processing the subject’s last token. Built upon the hypothesis that the FFN modules in a transformer layer can be viewed as key-value memories <cite class="ltx_cite ltx_citemacro_citep">(Geva et al., <a href="#bib.bib12" title="" class="ltx_ref">2021</a>)</cite>, ROME <cite class="ltx_cite ltx_citemacro_citep">(Meng et al., <a href="#bib.bib29" title="" class="ltx_ref">2022</a>)</cite> injects new knowledge into the key-value memories by deriving the closed form solution from the least-squares problem. MEMIT <cite class="ltx_cite ltx_citemacro_citep">(Meng et al., <a href="#bib.bib30" title="" class="ltx_ref">2023</a>)</cite> scales up ROME <cite class="ltx_cite ltx_citemacro_citep">(Meng et al., <a href="#bib.bib29" title="" class="ltx_ref">2022</a>)</cite> by editing a set of MLPs from consecutive middle-layers via solving a normal equation. PMET <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib25" title="" class="ltx_ref">2024a</a>)</cite> proposes to update multi-head self-attention (MHSA) modules in addition to FFNs. EMMET <cite class="ltx_cite ltx_citemacro_citep">(Gupta et al., <a href="#bib.bib15" title="" class="ltx_ref">2024b</a>)</cite> on the other hand, integrates the objectives of ROME and MEMIT into a unified preservation-memorization objective, facilitating batch-editing capabilities for both methodologies.</p>
</div>
</section>
</section>
<section id="S2.SS2.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">2.2.2 </span>Parameter-Preserving</h4>

<div id="S2.SS2.SSS2.p1" class="ltx_para">
<p id="S2.SS2.SSS2.p1.1" class="ltx_p">Parameter-preserving methods alter LLM output by adding new parameters, integrating external memory, or using strategies like in-context learning and devised decoding, without changing pre-trained LLM.</p>
</div>
<section id="S2.SS2.SSS2.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Additional Parameters</h5>

<div id="S2.SS2.SSS2.Px1.p1" class="ltx_para">
<p id="S2.SS2.SSS2.Px1.p1.1" class="ltx_p">Some methods utilize additional parameters, such as adding new neurons or employing parameter-efficient techniques. CaliNET <cite class="ltx_cite ltx_citemacro_citep">(Dong et al., <a href="#bib.bib10" title="" class="ltx_ref">2022</a>)</cite> extends the FFN modules with calibration memory slots to adjust the predicted token distribution. T-Patcher <cite class="ltx_cite ltx_citemacro_citep">(Huang et al., <a href="#bib.bib21" title="" class="ltx_ref">2023</a>)</cite> adds neurons in the FFN’s last layer to rectify classification errors and incorrectly generated tokens, activating only in response to associated mistakes. GRACE <cite class="ltx_cite ltx_citemacro_citep">(Hartvigsen et al., <a href="#bib.bib16" title="" class="ltx_ref">2023</a>)</cite> wraps a selected layer with an Adaptor that includes a codebook and deferral mechanism, learning to decode desired outputs while caching input error embeddings. The GRACE layer stores the edits and could be updated continuously over long deployments. MELO <cite class="ltx_cite ltx_citemacro_citep">(Yu et al., <a href="#bib.bib48" title="" class="ltx_ref">2024</a>)</cite> utilizes DyLoRA <cite class="ltx_cite ltx_citemacro_citep">(Valipour et al., <a href="#bib.bib40" title="" class="ltx_ref">2023</a>)</cite> modules to learn edits, indexing them in an inner vector database to dynamically activate corresponding LoRA blocks during inference.</p>
</div>
</section>
<section id="S2.SS2.SSS2.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">External Memory</h5>

<div id="S2.SS2.SSS2.Px2.p1" class="ltx_para">
<p id="S2.SS2.SSS2.Px2.p1.1" class="ltx_p">Other methods utilize external memories for editing. SERAC <cite class="ltx_cite ltx_citemacro_citep">(Mitchell et al., <a href="#bib.bib33" title="" class="ltx_ref">2022b</a>)</cite> leverages a scope classifier to determine whether an user-supplied edit example stored in its memory is related to the inputs. If no example exists, the inputs are passed to the base model; otherwise, a counterfactual model generates modified answers using the inputs and the related example. MeLLo <cite class="ltx_cite ltx_citemacro_citep">(Zhong et al., <a href="#bib.bib52" title="" class="ltx_ref">2023</a>)</cite> decomposes a multi-hop question into subquestions iteratively. The model then checks if the tentative answer generated by the base model contradicts the most relevant facts retrieved from the edited fact memory and adjusts the outputs accordingly.</p>
</div>
</section>
<section id="S2.SS2.SSS2.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">In-Context Learning and Decoding</h5>

<div id="S2.SS2.SSS2.Px3.p1" class="ltx_para">
<p id="S2.SS2.SSS2.Px3.p1.1" class="ltx_p">Certain strategies require no additional parameters. IKE <cite class="ltx_cite ltx_citemacro_citep">(Zheng et al., <a href="#bib.bib51" title="" class="ltx_ref">2023</a>)</cite> edits factual knowledge via in-context learning with demonstrations to guide the language model. DeepEdit <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib43" title="" class="ltx_ref">2024</a>)</cite> employs decoding constraints, including filtering step candidates, depth-first search to store valid candidates in a stack, and a greedy search to output the optimal path for multi-hop reasoning.</p>
</div>
</section>
</section>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Challenges of Knowledge Editing</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.1" class="ltx_p">While knowledge editing methods have been extensively researched, there’s a lack of comprehensive study on related challenges. In this section, we discuss the pitfalls of knowledge editing from three perspectives: inability to logically inference and robustly generalize (Section <a href="#S3.SS1" title="3.1 Inability to Logically Inference and Robustly Generalize ‣ 3 Challenges of Knowledge Editing ‣ Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.1</span></a>), unintended alteration of non-target knowledge (Section <a href="#S3.SS2" title="3.2 Unintended Alteration of Non-Target Knowledge ‣ 3 Challenges of Knowledge Editing ‣ Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a>), and deterioration of general LLM abilities (Section <a href="#S3.SS3" title="3.3 Deterioration of General LLM Abilities ‣ 3 Challenges of Knowledge Editing ‣ Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3</span></a>).</p>
</div>
<figure id="S3.T2" class="ltx_table">
<table id="S3.T2.6" class="ltx_tabular ltx_centering ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S3.T2.6.7.1" class="ltx_tr">
<td id="S3.T2.6.7.1.1" class="ltx_td ltx_align_center ltx_border_tt">Challenge</td>
<td id="S3.T2.6.7.1.2" class="ltx_td ltx_align_center ltx_border_tt">Benchmark</td>
<td id="S3.T2.6.7.1.3" class="ltx_td ltx_align_center ltx_border_tt">Metric</td>
</tr>
<tr id="S3.T2.6.8.2" class="ltx_tr">
<td id="S3.T2.6.8.2.1" class="ltx_td ltx_align_center ltx_border_t" rowspan="13"><span id="S3.T2.6.8.2.1.1" class="ltx_text">
<span id="S3.T2.6.8.2.1.1.1" class="ltx_inline-block">
<span id="S3.T2.6.8.2.1.1.1.1" class="ltx_p">Portability</span>
<span id="S3.T2.6.8.2.1.1.1.2" class="ltx_p">and</span>
<span id="S3.T2.6.8.2.1.1.1.3" class="ltx_p">Generalization</span>
</span></span></td>
<td id="S3.T2.6.8.2.2" class="ltx_td ltx_align_center ltx_border_t" rowspan="3"><span id="S3.T2.6.8.2.2.1" class="ltx_text">RippleEdits <cite class="ltx_cite ltx_citemacro_citep">(Cohen et al., <a href="#bib.bib8" title="" class="ltx_ref">2023</a>)</cite></span></td>
<td id="S3.T2.6.8.2.3" class="ltx_td ltx_align_center ltx_border_t">Logical Generalization</td>
</tr>
<tr id="S3.T2.6.9.3" class="ltx_tr">
<td id="S3.T2.6.9.3.1" class="ltx_td ltx_align_center">Compositionality I</td>
</tr>
<tr id="S3.T2.6.10.4" class="ltx_tr">
<td id="S3.T2.6.10.4.1" class="ltx_td ltx_align_center">Compositionality II</td>
</tr>
<tr id="S3.T2.6.11.5" class="ltx_tr">
<td id="S3.T2.6.11.5.1" class="ltx_td ltx_align_center ltx_border_t" rowspan="3"><span id="S3.T2.6.11.5.1.1" class="ltx_text">ConflictEdit <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib26" title="" class="ltx_ref">2024b</a>)</cite></span></td>
<td id="S3.T2.6.11.5.2" class="ltx_td ltx_align_center ltx_border_t">Conflict Score</td>
</tr>
<tr id="S3.T2.6.12.6" class="ltx_tr">
<td id="S3.T2.6.12.6.1" class="ltx_td ltx_align_center">Conflict Magnitude</td>
</tr>
<tr id="S3.T2.6.13.7" class="ltx_tr">
<td id="S3.T2.6.13.7.1" class="ltx_td ltx_align_center">Success Score</td>
</tr>
<tr id="S3.T2.6.14.8" class="ltx_tr">
<td id="S3.T2.6.14.8.1" class="ltx_td ltx_align_center ltx_border_t" rowspan="3"><span id="S3.T2.6.14.8.1.1" class="ltx_text">MQuAKE <cite class="ltx_cite ltx_citemacro_citep">(Zhong et al., <a href="#bib.bib52" title="" class="ltx_ref">2023</a>)</cite></span></td>
<td id="S3.T2.6.14.8.2" class="ltx_td ltx_align_center ltx_border_t">Edit-wise Success Rate</td>
</tr>
<tr id="S3.T2.6.15.9" class="ltx_tr">
<td id="S3.T2.6.15.9.1" class="ltx_td ltx_align_center">Instance-wise Accuracy</td>
</tr>
<tr id="S3.T2.6.16.10" class="ltx_tr">
<td id="S3.T2.6.16.10.1" class="ltx_td ltx_align_center">Multi-hop Accuracy</td>
</tr>
<tr id="S3.T2.6.17.11" class="ltx_tr">
<td id="S3.T2.6.17.11.1" class="ltx_td ltx_align_center ltx_border_t">ReCoE <cite class="ltx_cite ltx_citemacro_citep">(Hua et al., <a href="#bib.bib20" title="" class="ltx_ref">2024</a>)</cite>
</td>
<td id="S3.T2.6.17.11.2" class="ltx_td ltx_align_center ltx_border_t">QA Accuracy</td>
</tr>
<tr id="S3.T2.1.1" class="ltx_tr">
<td id="S3.T2.1.1.1" class="ltx_td ltx_align_center ltx_border_t" rowspan="3"><span id="S3.T2.1.1.1.1" class="ltx_text">ZsRE + CounterFact<sup id="S3.T2.1.1.1.1.1" class="ltx_sup">†</sup> <cite class="ltx_cite ltx_citemacro_cite">Yao et al. (<a href="#bib.bib47" title="" class="ltx_ref">2023b</a>)</cite></span></td>
<td id="S3.T2.1.1.2" class="ltx_td ltx_align_center ltx_border_t">Subject-Replace</td>
</tr>
<tr id="S3.T2.6.18.12" class="ltx_tr">
<td id="S3.T2.6.18.12.1" class="ltx_td ltx_align_center">Reverse-Relation</td>
</tr>
<tr id="S3.T2.6.19.13" class="ltx_tr">
<td id="S3.T2.6.19.13.1" class="ltx_td ltx_align_center">One-Hop</td>
</tr>
<tr id="S3.T2.6.20.14" class="ltx_tr">
<td id="S3.T2.6.20.14.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" rowspan="15"><span id="S3.T2.6.20.14.1.1" class="ltx_text">Locality</span></td>
<td id="S3.T2.6.20.14.2" class="ltx_td ltx_align_center ltx_border_t" rowspan="3"><span id="S3.T2.6.20.14.2.1" class="ltx_text">RippleEdits <cite class="ltx_cite ltx_citemacro_citep">(Cohen et al., <a href="#bib.bib8" title="" class="ltx_ref">2023</a>)</cite></span></td>
<td id="S3.T2.6.20.14.3" class="ltx_td ltx_align_center ltx_border_t">Subject Aliasing</td>
</tr>
<tr id="S3.T2.6.21.15" class="ltx_tr">
<td id="S3.T2.6.21.15.1" class="ltx_td ltx_align_center">Preservation</td>
</tr>
<tr id="S3.T2.6.22.16" class="ltx_tr">
<td id="S3.T2.6.22.16.1" class="ltx_td ltx_align_center">Relation Specificity</td>
</tr>
<tr id="S3.T2.6.23.17" class="ltx_tr">
<td id="S3.T2.6.23.17.1" class="ltx_td ltx_align_center ltx_border_t" rowspan="5"><span id="S3.T2.6.23.17.1.1" class="ltx_text">RoundEdit <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib26" title="" class="ltx_ref">2024b</a>)</cite></span></td>
<td id="S3.T2.6.23.17.2" class="ltx_td ltx_align_center ltx_border_t">Success Score</td>
</tr>
<tr id="S3.T2.2.2" class="ltx_tr">
<td id="S3.T2.2.2.1" class="ltx_td ltx_align_center">Distortion (<math id="S3.T2.2.2.1.m1.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="S3.T2.2.2.1.m1.1a"><mo stretchy="false" id="S3.T2.2.2.1.m1.1.1" xref="S3.T2.2.2.1.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="S3.T2.2.2.1.m1.1b"><ci id="S3.T2.2.2.1.m1.1.1.cmml" xref="S3.T2.2.2.1.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.2.2.1.m1.1c">\downarrow</annotation></semantics></math>)</td>
</tr>
<tr id="S3.T2.3.3" class="ltx_tr">
<td id="S3.T2.3.3.1" class="ltx_td ltx_align_center">Ignore Rate (<math id="S3.T2.3.3.1.m1.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="S3.T2.3.3.1.m1.1a"><mo stretchy="false" id="S3.T2.3.3.1.m1.1.1" xref="S3.T2.3.3.1.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="S3.T2.3.3.1.m1.1b"><ci id="S3.T2.3.3.1.m1.1.1.cmml" xref="S3.T2.3.3.1.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.3.3.1.m1.1c">\downarrow</annotation></semantics></math>)</td>
</tr>
<tr id="S3.T2.4.4" class="ltx_tr">
<td id="S3.T2.4.4.1" class="ltx_td ltx_align_center">Failure Rate (<math id="S3.T2.4.4.1.m1.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="S3.T2.4.4.1.m1.1a"><mo stretchy="false" id="S3.T2.4.4.1.m1.1.1" xref="S3.T2.4.4.1.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="S3.T2.4.4.1.m1.1b"><ci id="S3.T2.4.4.1.m1.1.1.cmml" xref="S3.T2.4.4.1.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.4.4.1.m1.1c">\downarrow</annotation></semantics></math>)</td>
</tr>
<tr id="S3.T2.5.5" class="ltx_tr">
<td id="S3.T2.5.5.1" class="ltx_td ltx_align_center">Tied Fact Damage (<math id="S3.T2.5.5.1.m1.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="S3.T2.5.5.1.m1.1a"><mo stretchy="false" id="S3.T2.5.5.1.m1.1.1" xref="S3.T2.5.5.1.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="S3.T2.5.5.1.m1.1b"><ci id="S3.T2.5.5.1.m1.1.1.cmml" xref="S3.T2.5.5.1.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.5.5.1.m1.1c">\downarrow</annotation></semantics></math>)</td>
</tr>
<tr id="S3.T2.6.6" class="ltx_tr">
<td id="S3.T2.6.6.1" class="ltx_td ltx_align_center ltx_border_t" rowspan="3"><span id="S3.T2.6.6.1.1" class="ltx_text">CounterFact<sup id="S3.T2.6.6.1.1.1" class="ltx_sup">†</sup> <cite class="ltx_cite ltx_citemacro_cite">Yao et al. (<a href="#bib.bib47" title="" class="ltx_ref">2023b</a>)</cite></span></td>
<td id="S3.T2.6.6.2" class="ltx_td ltx_align_center ltx_border_t">Other-Attribution</td>
</tr>
<tr id="S3.T2.6.24.18" class="ltx_tr">
<td id="S3.T2.6.24.18.1" class="ltx_td ltx_align_center">Distract-Neighbor</td>
</tr>
<tr id="S3.T2.6.25.19" class="ltx_tr">
<td id="S3.T2.6.25.19.1" class="ltx_td ltx_align_center">Other-Task</td>
</tr>
<tr id="S3.T2.6.26.20" class="ltx_tr">
<td id="S3.T2.6.26.20.1" class="ltx_td ltx_align_center ltx_border_t" rowspan="3"><span id="S3.T2.6.26.20.1.1" class="ltx_text">CounterFact <cite class="ltx_cite ltx_citemacro_cite">Meng et al. (<a href="#bib.bib29" title="" class="ltx_ref">2022</a>)</cite></span></td>
<td id="S3.T2.6.26.20.2" class="ltx_td ltx_align_center ltx_border_t">Locality</td>
</tr>
<tr id="S3.T2.6.27.21" class="ltx_tr">
<td id="S3.T2.6.27.21.1" class="ltx_td ltx_align_center">Neighborhood Score</td>
</tr>
<tr id="S3.T2.6.28.22" class="ltx_tr">
<td id="S3.T2.6.28.22.1" class="ltx_td ltx_align_center">Neighborhood Magnitude</td>
</tr>
<tr id="S3.T2.6.29.23" class="ltx_tr">
<td id="S3.T2.6.29.23.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t"><span id="S3.T2.6.29.23.1.1" class="ltx_text">CounterFact+ <cite class="ltx_cite ltx_citemacro_cite">Hoelscher-Obermaier et al. (<a href="#bib.bib19" title="" class="ltx_ref">2023</a>)</cite></span></td>
<td id="S3.T2.6.29.23.2" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">Neighborhood KL Divergence</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>Performance benchmarks and evaluation metrics addressing generalization/portability and locality issues in knowledge editing methods. Unless specifically indicated by a downward arrow, higher values signify better performance in those evaluation metrics. CounterFact benchmark is proposed by  <cite class="ltx_cite ltx_citemacro_cite">Meng et al. (<a href="#bib.bib29" title="" class="ltx_ref">2022</a>)</cite>, and CounterFact with <sup id="S3.T2.10.1" class="ltx_sup">†</sup> mark is modified to further examine the proposed metrics.</figcaption>
</figure>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Inability to Logically Inference and Robustly Generalize</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.1" class="ltx_p">When a fact is updated, it is crucial not only to revise the specific piece of knowledge but also to evaluate the impact on the related reasoning chain. Recently the term <span id="S3.SS1.p1.1.1" class="ltx_text ltx_font_bold">portability</span> has been proposed in <cite class="ltx_cite ltx_citemacro_citep">(Yao et al., <a href="#bib.bib47" title="" class="ltx_ref">2023b</a>)</cite> to evaluate the consequences after an edit, and further assess the robustness of generalization. In their study, they introduce three metrics to evaluate portability: <span id="S3.SS1.p1.1.2" class="ltx_text ltx_font_bold">Subject Replace</span> (checking if synonyms of the subject are edited), <span id="S3.SS1.p1.1.3" class="ltx_text ltx_font_bold">Reversed Relation</span> (checking if the reversed relation of the target is edited), and <span id="S3.SS1.p1.1.4" class="ltx_text ltx_font_bold">One Hop</span> (assessing if modified knowledge is usable for further derivation). Similarly, <span id="S3.SS1.p1.1.5" class="ltx_text ltx_font_bold">RippleEdits</span> benchmark as well as corresponding <span id="S3.SS1.p1.1.6" class="ltx_text ltx_font_bold">Logical Generalization</span> and <span id="S3.SS1.p1.1.7" class="ltx_text ltx_font_bold">Compositionality</span> metrics are proposed to examine whether edited knowledge can be inferred in composite relations of facts <cite class="ltx_cite ltx_citemacro_cite">Cohen et al. (<a href="#bib.bib8" title="" class="ltx_ref">2023</a>)</cite>. Additionally, <span id="S3.SS1.p1.1.8" class="ltx_text ltx_font_bold">ReCoE</span> benchmark is proposed to assess the propagation of updates in interconnected facts using various reasoning schemes in complex question-answering datasets <cite class="ltx_cite ltx_citemacro_cite">Hua et al. (<a href="#bib.bib20" title="" class="ltx_ref">2024</a>)</cite>. Furthermore, <span id="S3.SS1.p1.1.9" class="ltx_text ltx_font_bold">MQuAKE</span> benchmark is introduced to evaluate more complex reasoning and inference ability on multi-hop questions <cite class="ltx_cite ltx_citemacro_cite">Zhong et al. (<a href="#bib.bib52" title="" class="ltx_ref">2023</a>)</cite>.</p>
</div>
<div id="S3.SS1.p2" class="ltx_para">
<p id="S3.SS1.p2.1" class="ltx_p">When editing multiple logically related facts simultaneously, models may suffer from confusion due to conflicts.  <span id="S3.SS1.p2.1.1" class="ltx_text ltx_font_bold">ConflictEdit</span> benchmark is proposed to examine different editing methods on conflicted edit facts <cite class="ltx_cite ltx_citemacro_cite">Li et al. (<a href="#bib.bib26" title="" class="ltx_ref">2024b</a>)</cite>. The different benchmarks and corresponding metrics and are arranged systematically in Table <a href="#S3.T2" title="Table 2 ‣ 3 Challenges of Knowledge Editing ‣ Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.</p>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Unintended Alteration of Non-Target Knowledge</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.1" class="ltx_p">Locality is conventionally assessed using a locality dataset to evaluate edits on unrelated facts by measuring the <span id="S3.SS2.p1.1.1" class="ltx_text ltx_font_bold">Neighborhood Score</span> and <span id="S3.SS2.p1.1.2" class="ltx_text ltx_font_bold">Neighborhood Magnitude</span> <cite class="ltx_cite ltx_citemacro_citep">(NS &amp; NM; Meng et al., <a href="#bib.bib29" title="" class="ltx_ref">2022</a>, <a href="#bib.bib30" title="" class="ltx_ref">2023</a>)</cite>.However, current evaluation methods do not adequately capture the post-edit effects on content beyond the locality dataset, which means the edited model could still contain unintended alterations. For example, while the location of the Louvre might be successfully changed from Paris to London, the edited model may inadvertently increase the likelihood of semantically related words (e.g. Big Ben for London) while mentioning Louvre. Some modified benchmark (<span id="S3.SS2.p1.1.3" class="ltx_text ltx_font_bold">CounterFact+</span>) and corresponding metric (<span id="S3.SS2.p1.1.4" class="ltx_text ltx_font_bold">Neighborhood KL Divergence</span>) <cite class="ltx_cite ltx_citemacro_citep">(Hoelscher-Obermaier et al., <a href="#bib.bib19" title="" class="ltx_ref">2023</a>)</cite> is then designed to disclose these previously implicit pitfalls. Another study <cite class="ltx_cite ltx_citemacro_citep">(Yao et al., <a href="#bib.bib46" title="" class="ltx_ref">2023a</a>)</cite> extends this exploration to three facets of locality: <span id="S3.SS2.p1.1.5" class="ltx_text ltx_font_bold">Other Relations</span>(evaluating the retention of other attributes of the updated subject), <span id="S3.SS2.p1.1.6" class="ltx_text ltx_font_bold">Distract Neighborhood</span>(assessing the divergence of unrelated input when juxtaposed with edited instances), and <span id="S3.SS2.p1.1.7" class="ltx_text ltx_font_bold">Other Tasks</span> (examining the influence of edits on the performance of other tasks).</p>
</div>
<div id="S3.SS2.p2" class="ltx_para">
<p id="S3.SS2.p2.1" class="ltx_p">Unintended edits to unrelated facts may occur because a single edit can implicitly change the predictive distribution among objects associated with the same (<span id="S3.SS2.p2.1.1" class="ltx_text ltx_font_italic">subject</span> - <span id="S3.SS2.p2.1.2" class="ltx_text ltx_font_italic">relation</span>) pair. After multiple consecutive edits, these alterations can accumulate and distort the stored knowledge. To evaluate this condition, the concept of <span id="S3.SS2.p2.1.3" class="ltx_text ltx_font_bold">Knowledge Distortion</span> has been introduced by <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib26" title="" class="ltx_ref">2024b</a>)</cite>, which estimates the Jensen–Shannon divergence of the object set distribution before and after editing. This can be further extended to metrics such as the <span id="S3.SS2.p2.1.4" class="ltx_text ltx_font_bold">Ignore Rate</span>, measuring how objects other than the target in the object set are neglected after editing, and the <span id="S3.SS2.p2.1.5" class="ltx_text ltx_font_bold">Failure Rate</span>, which measures the proportion of instances where over half of the objects in the set are overlooked.</p>
</div>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Deterioration of General LLM Abilities</h3>

<div id="S3.SS3.p1" class="ltx_para">
<p id="S3.SS3.p1.1" class="ltx_p">Current evaluation metrics are primarily limited to scenarios where editing is performed only once or infrequently, prompting some studies to extend evaluations to the outcomes after consecutive edits. A study by  <cite class="ltx_cite ltx_citemacro_citep">(Gupta et al., <a href="#bib.bib14" title="" class="ltx_ref">2024a</a>)</cite> discovers that post-edit models exhibit susceptibility to both gradual forgetting and catastrophic forgetting in sequential editing scenarios. Notably, their findings indicate that the extent of knowledge forgetting is more pronounced in meta-learning-based methods compared to locate-and-edit methods. Additionally, models with parameters modified successively show a decline in performance across various downstream NLP tasks <cite class="ltx_cite ltx_citemacro_citep">(Gu et al., <a href="#bib.bib13" title="" class="ltx_ref">2024</a>)</cite>. Furthermore, perplexity is found to increase after consecutive edits across all parameter-modified methods and different LLMs, and is proposed as another metric to indicate model collapse <cite class="ltx_cite ltx_citemacro_citep">(Yang et al., <a href="#bib.bib45" title="" class="ltx_ref">2024</a>)</cite>. These findings further corroborate that model editing aimed at modifying parameters adversely affects the general capabilities of the original LLMs.</p>
</div>
<figure id="S3.T3" class="ltx_table">
<table id="S3.T3.10" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S3.T3.10.11.1" class="ltx_tr">
<th id="S3.T3.10.11.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="S3.T3.10.11.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="3">Single Edit</th>
<th id="S3.T3.10.11.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="5">Multiple Edit</th>
</tr>
<tr id="S3.T3.10.12.2" class="ltx_tr">
<th id="S3.T3.10.12.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="S3.T3.10.12.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" colspan="3">One-Hop</th>
<th id="S3.T3.10.12.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Multiple-Hop</th>
<th id="S3.T3.10.12.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" colspan="2">Reverse Conflict</th>
<th id="S3.T3.10.12.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" colspan="2">Composite Conflict</th>
</tr>
<tr id="S3.T3.10.13.3" class="ltx_tr">
<th id="S3.T3.10.13.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row">Methods</th>
<th id="S3.T3.10.13.3.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">SR</th>
<th id="S3.T3.10.13.3.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">RR</th>
<th id="S3.T3.10.13.3.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">OH</th>
<th id="S3.T3.10.13.3.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">MH</th>
<th id="S3.T3.10.13.3.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">CS</th>
<th id="S3.T3.10.13.3.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">CM</th>
<th id="S3.T3.10.13.3.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">CS</th>
<th id="S3.T3.10.13.3.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">CM</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S3.T3.10.14.1" class="ltx_tr">
<th id="S3.T3.10.14.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">FT</th>
<td id="S3.T3.10.14.1.2" class="ltx_td ltx_align_center ltx_border_t">72.96</td>
<td id="S3.T3.10.14.1.3" class="ltx_td ltx_align_center ltx_border_t">8.05</td>
<td id="S3.T3.10.14.1.4" class="ltx_td ltx_align_center ltx_border_t">1.34</td>
<td id="S3.T3.10.14.1.5" class="ltx_td ltx_align_center ltx_border_t">1.6</td>
<td id="S3.T3.10.14.1.6" class="ltx_td ltx_align_center ltx_border_t">80.28</td>
<td id="S3.T3.10.14.1.7" class="ltx_td ltx_align_center ltx_border_t">71.11</td>
<td id="S3.T3.10.14.1.8" class="ltx_td ltx_align_center ltx_border_t">75.45</td>
<td id="S3.T3.10.14.1.9" class="ltx_td ltx_align_center ltx_border_t">64.28</td>
</tr>
<tr id="S3.T3.10.15.2" class="ltx_tr">
<th id="S3.T3.10.15.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MEND</th>
<td id="S3.T3.10.15.2.2" class="ltx_td ltx_align_center">42.45</td>
<td id="S3.T3.10.15.2.3" class="ltx_td ltx_align_center">0.00</td>
<td id="S3.T3.10.15.2.4" class="ltx_td ltx_align_center">11.34</td>
<td id="S3.T3.10.15.2.5" class="ltx_td ltx_align_center">9.2</td>
<td id="S3.T3.10.15.2.6" class="ltx_td ltx_align_center">88.89</td>
<td id="S3.T3.10.15.2.7" class="ltx_td ltx_align_center">60.50</td>
<td id="S3.T3.10.15.2.8" class="ltx_td ltx_align_center">84.85</td>
<td id="S3.T3.10.15.2.9" class="ltx_td ltx_align_center">43.45</td>
</tr>
<tr id="S3.T3.10.16.3" class="ltx_tr">
<th id="S3.T3.10.16.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">ROME</th>
<td id="S3.T3.10.16.3.2" class="ltx_td ltx_align_center">37.42</td>
<td id="S3.T3.10.16.3.3" class="ltx_td ltx_align_center">46.42</td>
<td id="S3.T3.10.16.3.4" class="ltx_td ltx_align_center">50.91</td>
<td id="S3.T3.10.16.3.5" class="ltx_td ltx_align_center">7.6</td>
<td id="S3.T3.10.16.3.6" class="ltx_td ltx_align_center">65.92</td>
<td id="S3.T3.10.16.3.7" class="ltx_td ltx_align_center">-0.65</td>
<td id="S3.T3.10.16.3.8" class="ltx_td ltx_align_center">71.70</td>
<td id="S3.T3.10.16.3.9" class="ltx_td ltx_align_center">37.04</td>
</tr>
<tr id="S3.T3.10.17.4" class="ltx_tr">
<th id="S3.T3.10.17.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MEMIT</th>
<td id="S3.T3.10.17.4.2" class="ltx_td ltx_align_center">27.73</td>
<td id="S3.T3.10.17.4.3" class="ltx_td ltx_align_center">47.67</td>
<td id="S3.T3.10.17.4.4" class="ltx_td ltx_align_center">52.74</td>
<td id="S3.T3.10.17.4.5" class="ltx_td ltx_align_center">8.1</td>
<td id="S3.T3.10.17.4.6" class="ltx_td ltx_align_center">51.40</td>
<td id="S3.T3.10.17.4.7" class="ltx_td ltx_align_center">-1.60</td>
<td id="S3.T3.10.17.4.8" class="ltx_td ltx_align_center">57.15</td>
<td id="S3.T3.10.17.4.9" class="ltx_td ltx_align_center">-1.50</td>
</tr>
<tr id="S3.T3.5.5" class="ltx_tr">
<th id="S3.T3.5.5.6" class="ltx_td ltx_align_left ltx_th ltx_th_row">SERAC</th>
<td id="S3.T3.5.5.7" class="ltx_td ltx_align_center">17.79</td>
<td id="S3.T3.5.5.8" class="ltx_td ltx_align_center">1.30</td>
<td id="S3.T3.5.5.9" class="ltx_td ltx_align_center">5.53</td>
<td id="S3.T3.1.1.1" class="ltx_td ltx_align_center">7.9<sup id="S3.T3.1.1.1.1" class="ltx_sup">†</sup>
</td>
<td id="S3.T3.2.2.2" class="ltx_td ltx_align_center">50.89<sup id="S3.T3.2.2.2.1" class="ltx_sup">†</sup>
</td>
<td id="S3.T3.3.3.3" class="ltx_td ltx_align_center">-0.02<sup id="S3.T3.3.3.3.1" class="ltx_sup">†</sup>
</td>
<td id="S3.T3.4.4.4" class="ltx_td ltx_align_center">50.84<sup id="S3.T3.4.4.4.1" class="ltx_sup">†</sup>
</td>
<td id="S3.T3.5.5.5" class="ltx_td ltx_align_center">-0.02<sup id="S3.T3.5.5.5.1" class="ltx_sup">†</sup>
</td>
</tr>
<tr id="S3.T3.10.10" class="ltx_tr">
<th id="S3.T3.10.10.6" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">IKE</th>
<td id="S3.T3.10.10.7" class="ltx_td ltx_align_center ltx_border_bb">88.77</td>
<td id="S3.T3.10.10.8" class="ltx_td ltx_align_center ltx_border_bb">92.96</td>
<td id="S3.T3.10.10.9" class="ltx_td ltx_align_center ltx_border_bb">55.38</td>
<td id="S3.T3.6.6.1" class="ltx_td ltx_align_center ltx_border_bb">8.3<sup id="S3.T3.6.6.1.1" class="ltx_sup">†</sup>
</td>
<td id="S3.T3.7.7.2" class="ltx_td ltx_align_center ltx_border_bb">58.20<sup id="S3.T3.7.7.2.1" class="ltx_sup">†</sup>
</td>
<td id="S3.T3.8.8.3" class="ltx_td ltx_align_center ltx_border_bb">-1.00<sup id="S3.T3.8.8.3.1" class="ltx_sup">†</sup>
</td>
<td id="S3.T3.9.9.4" class="ltx_td ltx_align_center ltx_border_bb">50.52<sup id="S3.T3.9.9.4.1" class="ltx_sup">†</sup>
</td>
<td id="S3.T3.10.10.5" class="ltx_td ltx_align_center ltx_border_bb">-0.99<sup id="S3.T3.10.10.5.1" class="ltx_sup">†</sup>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 3: </span>Experimental results for portability and generalization. SR: Subject-Replace, RR: Reverse-Relation, OH: One-Hop Accuracy, EW: Edit-wise, IW: Instance-wise, MH: Multi-hop Accuracy, CS: Conflict score, CM: Conflict magnitude. Higher values indicate better performance for all metrics in this table. Results marked with <math id="S3.T3.12.m1.1" class="ltx_Math" alttext="\dagger" display="inline"><semantics id="S3.T3.12.m1.1b"><mo id="S3.T3.12.m1.1.1" xref="S3.T3.12.m1.1.1.cmml">†</mo><annotation-xml encoding="MathML-Content" id="S3.T3.12.m1.1c"><ci id="S3.T3.12.m1.1.1.cmml" xref="S3.T3.12.m1.1.1">†</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.T3.12.m1.1d">\dagger</annotation></semantics></math> are obtained in our own experiments, and other results are taken from previous studies.</figcaption>
</figure>
<figure id="S3.T4" class="ltx_table">
<table id="S3.T4.11" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S3.T4.11.12.1" class="ltx_tr">
<th id="S3.T4.11.12.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="S3.T4.11.12.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="3">Single Edit</th>
<th id="S3.T4.11.12.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="5">Multiple Edit</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S3.T4.3.3" class="ltx_tr">
<th id="S3.T4.3.3.4" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row">Methods</th>
<th id="S3.T4.3.3.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">OA</th>
<th id="S3.T4.3.3.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">DN</th>
<th id="S3.T4.3.3.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">OT</th>
<th id="S3.T4.3.3.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Succ.</th>
<th id="S3.T4.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">D (<math id="S3.T4.1.1.1.m1.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="S3.T4.1.1.1.m1.1a"><mo stretchy="false" id="S3.T4.1.1.1.m1.1.1" xref="S3.T4.1.1.1.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="S3.T4.1.1.1.m1.1b"><ci id="S3.T4.1.1.1.m1.1.1.cmml" xref="S3.T4.1.1.1.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.T4.1.1.1.m1.1c">\downarrow</annotation></semantics></math>)</th>
<th id="S3.T4.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">IR (<math id="S3.T4.2.2.2.m1.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="S3.T4.2.2.2.m1.1a"><mo stretchy="false" id="S3.T4.2.2.2.m1.1.1" xref="S3.T4.2.2.2.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="S3.T4.2.2.2.m1.1b"><ci id="S3.T4.2.2.2.m1.1.1.cmml" xref="S3.T4.2.2.2.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.T4.2.2.2.m1.1c">\downarrow</annotation></semantics></math>)</th>
<th id="S3.T4.3.3.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">FR (<math id="S3.T4.3.3.3.m1.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="S3.T4.3.3.3.m1.1a"><mo stretchy="false" id="S3.T4.3.3.3.m1.1.1" xref="S3.T4.3.3.3.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="S3.T4.3.3.3.m1.1b"><ci id="S3.T4.3.3.3.m1.1.1.cmml" xref="S3.T4.3.3.3.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.T4.3.3.3.m1.1c">\downarrow</annotation></semantics></math>)</th>
<td id="S3.T4.3.3.9" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="S3.T4.11.13.1" class="ltx_tr">
<th id="S3.T4.11.13.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">FT</th>
<td id="S3.T4.11.13.1.2" class="ltx_td ltx_align_center ltx_border_t">12.88</td>
<td id="S3.T4.11.13.1.3" class="ltx_td ltx_align_center ltx_border_t">9.48</td>
<td id="S3.T4.11.13.1.4" class="ltx_td ltx_align_center ltx_border_t">49.56</td>
<td id="S3.T4.11.13.1.5" class="ltx_td ltx_align_center ltx_border_t">100.0</td>
<td id="S3.T4.11.13.1.6" class="ltx_td ltx_align_center ltx_border_t">16.12</td>
<td id="S3.T4.11.13.1.7" class="ltx_td ltx_align_center ltx_border_t">97.48</td>
<td id="S3.T4.11.13.1.8" class="ltx_td ltx_align_center ltx_border_t">97.32</td>
<td id="S3.T4.11.13.1.9" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="S3.T4.11.14.2" class="ltx_tr">
<th id="S3.T4.11.14.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MEND</th>
<td id="S3.T4.11.14.2.2" class="ltx_td ltx_align_center">73.50</td>
<td id="S3.T4.11.14.2.3" class="ltx_td ltx_align_center">32.96</td>
<td id="S3.T4.11.14.2.4" class="ltx_td ltx_align_center">48.86</td>
<td id="S3.T4.11.14.2.5" class="ltx_td ltx_align_center">99.12</td>
<td id="S3.T4.11.14.2.6" class="ltx_td ltx_align_center">14.35</td>
<td id="S3.T4.11.14.2.7" class="ltx_td ltx_align_center">87.64</td>
<td id="S3.T4.11.14.2.8" class="ltx_td ltx_align_center">86.56</td>
<td id="S3.T4.11.14.2.9" class="ltx_td"></td>
</tr>
<tr id="S3.T4.11.15.3" class="ltx_tr">
<th id="S3.T4.11.15.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">ROME</th>
<td id="S3.T4.11.15.3.2" class="ltx_td ltx_align_center">78.94</td>
<td id="S3.T4.11.15.3.3" class="ltx_td ltx_align_center">50.35</td>
<td id="S3.T4.11.15.3.4" class="ltx_td ltx_align_center">52.12</td>
<td id="S3.T4.11.15.3.5" class="ltx_td ltx_align_center">99.80</td>
<td id="S3.T4.11.15.3.6" class="ltx_td ltx_align_center">13.95</td>
<td id="S3.T4.11.15.3.7" class="ltx_td ltx_align_center">78.98</td>
<td id="S3.T4.11.15.3.8" class="ltx_td ltx_align_center">77.60</td>
<td id="S3.T4.11.15.3.9" class="ltx_td"></td>
</tr>
<tr id="S3.T4.11.16.4" class="ltx_tr">
<th id="S3.T4.11.16.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MEMIT</th>
<td id="S3.T4.11.16.4.2" class="ltx_td ltx_align_center">86.78</td>
<td id="S3.T4.11.16.4.3" class="ltx_td ltx_align_center">60.47</td>
<td id="S3.T4.11.16.4.4" class="ltx_td ltx_align_center">74.62</td>
<td id="S3.T4.11.16.4.5" class="ltx_td ltx_align_center">99.72</td>
<td id="S3.T4.11.16.4.6" class="ltx_td ltx_align_center">13.50</td>
<td id="S3.T4.11.16.4.7" class="ltx_td ltx_align_center">72.03</td>
<td id="S3.T4.11.16.4.8" class="ltx_td ltx_align_center">70.44</td>
<td id="S3.T4.11.16.4.9" class="ltx_td"></td>
</tr>
<tr id="S3.T4.7.7" class="ltx_tr">
<th id="S3.T4.7.7.5" class="ltx_td ltx_align_left ltx_th ltx_th_row">SERAC</th>
<td id="S3.T4.7.7.6" class="ltx_td ltx_align_center">99.50</td>
<td id="S3.T4.7.7.7" class="ltx_td ltx_align_center">39.18</td>
<td id="S3.T4.7.7.8" class="ltx_td ltx_align_center">74.84</td>
<td id="S3.T4.4.4.1" class="ltx_td ltx_align_center">50.14<sup id="S3.T4.4.4.1.1" class="ltx_sup">†</sup>
</td>
<td id="S3.T4.5.5.2" class="ltx_td ltx_align_center">3.78<sup id="S3.T4.5.5.2.1" class="ltx_sup">†</sup>
</td>
<td id="S3.T4.6.6.3" class="ltx_td ltx_align_center">99.62<sup id="S3.T4.6.6.3.1" class="ltx_sup">†</sup>
</td>
<td id="S3.T4.7.7.4" class="ltx_td ltx_align_center">99.64<sup id="S3.T4.7.7.4.1" class="ltx_sup">†</sup>
</td>
<td id="S3.T4.7.7.9" class="ltx_td"></td>
</tr>
<tr id="S3.T4.11.11" class="ltx_tr">
<th id="S3.T4.11.11.5" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">IKE</th>
<td id="S3.T4.11.11.6" class="ltx_td ltx_align_center ltx_border_bb">84.13</td>
<td id="S3.T4.11.11.7" class="ltx_td ltx_align_center ltx_border_bb">66.04</td>
<td id="S3.T4.11.11.8" class="ltx_td ltx_align_center ltx_border_bb">75.33</td>
<td id="S3.T4.8.8.1" class="ltx_td ltx_align_center ltx_border_bb">100.0<sup id="S3.T4.8.8.1.1" class="ltx_sup">†</sup>
</td>
<td id="S3.T4.9.9.2" class="ltx_td ltx_align_center ltx_border_bb">13.43<sup id="S3.T4.9.9.2.1" class="ltx_sup">†</sup>
</td>
<td id="S3.T4.10.10.3" class="ltx_td ltx_align_center ltx_border_bb">73.53<sup id="S3.T4.10.10.3.1" class="ltx_sup">†</sup>
</td>
<td id="S3.T4.11.11.4" class="ltx_td ltx_align_center ltx_border_bb">73.00<sup id="S3.T4.11.11.4.1" class="ltx_sup">†</sup>
</td>
<td id="S3.T4.11.11.9" class="ltx_td ltx_border_bb"></td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 4: </span>Experimental results for locality. OA: Other-Attribution, DN: Distract-Neighbor, OT: Other-Task, Succ.: Success rate, D: Distortion, IR: Ignore rate, FR: Failure rate, TFD: Tied fact damage. Unless specifically indicated by a downward arrow, higher values signify better performance in those evaluation metrics. Results marked with <math id="S3.T4.13.m1.1" class="ltx_Math" alttext="\dagger" display="inline"><semantics id="S3.T4.13.m1.1b"><mo id="S3.T4.13.m1.1.1" xref="S3.T4.13.m1.1.1.cmml">†</mo><annotation-xml encoding="MathML-Content" id="S3.T4.13.m1.1c"><ci id="S3.T4.13.m1.1.1.cmml" xref="S3.T4.13.m1.1.1">†</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.T4.13.m1.1d">\dagger</annotation></semantics></math> are obtained in our own experiments, and other results are taken from previous studies.</figcaption>
</figure>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Experiments</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.1" class="ltx_p">The experiments were done to evaluate robust generalization and locality (Section  <a href="#S4.SS1.SSS1" title="4.1.1 Robust generalization and locality ‣ 4.1 Experimental Setup ‣ 4 Experiments ‣ Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.1.1</span></a> as well as deterioration of general LLM abilities (Section  <a href="#S4.SS1.SSS2" title="4.1.2 Deterioration of general LLM abilities ‣ 4.1 Experimental Setup ‣ 4 Experiments ‣ Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.1.2</span></a> across different editing methods.</p>
</div>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Experimental Setup</h3>

<section id="S4.SS1.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.1.1 </span>Robust generalization and locality</h4>

<div id="S4.SS1.SSS1.p1" class="ltx_para">
<p id="S4.SS1.SSS1.p1.1" class="ltx_p">We use GPT-J <cite class="ltx_cite ltx_citemacro_citep">(Wang and Komatsuzaki, <a href="#bib.bib41" title="" class="ltx_ref">2021</a>)</cite> as the baseline model for editing and implement six distinct editing methodologies to assess robust generalization and locality: MEND (meta-learning), ROME and MEMIT (locate-and-edit), SERAC (external memory), and IKE (prompting demonstrations).</p>
</div>
<div id="S4.SS1.SSS1.p2" class="ltx_para">
<p id="S4.SS1.SSS1.p2.1" class="ltx_p">Given the overlap in benchmarks for robust generalization and locality, we select a subset for our experiments. Robust generalization is evaluated under single edit and multiple edit settings. Single edit metrics include Subject-Replace, Reverse-Replace, and One-Hop reasoning <cite class="ltx_cite ltx_citemacro_citep">(Yao et al., <a href="#bib.bib46" title="" class="ltx_ref">2023a</a>)</cite>. Multiple edit metrics include multi-hop editing accuracy <cite class="ltx_cite ltx_citemacro_citep">(Zhong et al., <a href="#bib.bib52" title="" class="ltx_ref">2023</a>)</cite>, and Conflict Score and Conflict Magnitude for Reverse Conflict and Composite Conflict respectively <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib26" title="" class="ltx_ref">2024b</a>)</cite>.</p>
</div>
<div id="S4.SS1.SSS1.p3" class="ltx_para">
<p id="S4.SS1.SSS1.p3.1" class="ltx_p">For locality, single edit metrics include Other-Attribution, Distract-Neighbor, and Other-Task <cite class="ltx_cite ltx_citemacro_citep">(Yao et al., <a href="#bib.bib47" title="" class="ltx_ref">2023b</a>)</cite>, while multiple edit metrics encompass Success Rate, Distortion, Ignore Rate, and Failure Rate <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib26" title="" class="ltx_ref">2024b</a>)</cite>.</p>
</div>
</section>
<section id="S4.SS1.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.1.2 </span>Deterioration of general LLM abilities</h4>

<div id="S4.SS1.SSS2.p1" class="ltx_para">
<p id="S4.SS1.SSS2.p1.1" class="ltx_p">Following the settings of <cite class="ltx_cite ltx_citemacro_citep">(Gu et al., <a href="#bib.bib13" title="" class="ltx_ref">2024</a>)</cite>, we assessed deterioration of general LLM abilities post-editing using six methodologies: ROME, MEMIT, SERAC, MEND, KN, and GRACE. We evaluated general abilities across four NLP downstream tasks: open-domain question answering, sentiment analysis, reasoning, and summarization. These tasks were assessed after 10 to 40 edits on the Zero-Shot Relation Extraction (ZsRE) dataset<cite class="ltx_cite ltx_citemacro_citep">(Levy et al., <a href="#bib.bib24" title="" class="ltx_ref">2017</a>)</cite>, comparing the results against pre-editing benchmarks. More details on the selected downstream tasks are in Appendix <a href="#A2" title="Appendix B Detailed Explanation of experiments for deterioration of general LLM abilities ‣ Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a>.</p>
</div>
</section>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Experimental Results and Discussion</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.1" class="ltx_p">In general, current editing methodologies exhibit suboptimal performance concerning both robust generalization and locality. Regarding robust generalization, IKE, which leverages prompt demonstrations, demonstrates superior performance over other methodologies in single edit conditions. However, IKE’s performance noticeably declines in multiple edit scenarios, suggesting that prompt demonstrations may become confused when editing multiple logically related facts. Conversely, fine-tuning and meta-learning-based methods are less susceptible to confusion after editing multiple related facts.</p>
</div>
<div id="S4.SS2.p2" class="ltx_para">
<p id="S4.SS2.p2.1" class="ltx_p">Regarding locality, IKE maintains stable performance across metrics in single edit settings. Parameter-modifying methods excel in Other Attribution but decline in other metrics, except MEMIT, which remains stable across all metrics. In multiple edit scenarios, all methods except SERAC show similar performance. In the multiple edit scenario, all methods except SERAC exhibit relatively similar performance. SERAC displays low edit success rate and distortion rate, suggesting its scope classifier does not adopt most edits in this scenario. This may be attributed to its weakness in recovering edited facts, which is crucial in this metric setting.</p>
</div>
<div id="S4.SS2.p3" class="ltx_para">
<p id="S4.SS2.p3.1" class="ltx_p">In terms of general LLM abilities, the number of edits affects methods differently. Meta-learning methods like MEND degrade significantly after 10-20 edits. Locate-and-edit methods such as ROME and KN degrade after 10 edits, while MEMIT remains stable after 40 edits. This disparity can be attributed to MEMIT’s strategy of adjusting parameters across multiple layers, as opposed to ROME’s single-layer edits and KN’s approach of modifying a few neurons. This distribution of parameter modifications across layers helps mitigate deterioration.</p>
</div>
<div id="S4.SS2.p4" class="ltx_para">
<p id="S4.SS2.p4.1" class="ltx_p">GRACE, which stores edited facts with additional parameters, shows no performance change in downstream tasks after edits. One possible explanation is that the edits are conducted on the ZsRE dataset, which is distinct from the requirements of downstream tasks, leading to the stored facts not being retrieved during inference. Similarly, SERAC, utilizing external memory for edited facts, preserves general NLP abilities post-editing. This preservation stems from SERAC being trained once before editing begins, solely performing inference during editing, thereby preventing changes in the model’s output, even after multiple edits.</p>
</div>
<div id="S4.SS2.p5" class="ltx_para">
<p id="S4.SS2.p5.1" class="ltx_p">Overall, parameter-modifying methods degrade downstream task performance by altering pre-trained LLM parameters. In contrast, parameter-preserving methods maintain the original parameters, resulting in stable downstream task performance even after multiple edits.</p>
</div>
<figure id="S4.F4" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F4.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2406.01436/assets/x4.png" id="S4.F4.sf1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="296" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span>Open Domain Question Answering</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F4.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2406.01436/assets/x5.png" id="S4.F4.sf2.g1" class="ltx_graphics ltx_img_landscape" width="461" height="310" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span>Sentiment Analysis</figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F4.sf3" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="" id="S4.F4.sf3.g1" class="ltx_graphics ltx_missing ltx_missing_image" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(c) </span>Summarization</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F4.sf4" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2406.01436/assets/x7.png" id="S4.F4.sf4.g1" class="ltx_graphics ltx_img_landscape" width="461" height="310" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(d) </span>Reasoning</figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>The experimental results for the deterioration of general abilities were obtained by editing a pretrained LLM using GPT-J with various editing algorithms, including ROME, MEMIT, MEND, KN, SERAC, and GRACE, each applied 10 to 40 times. The edited models were subsequently evaluated on four downstream tasks, including open-domain question answering, sentiment analysis, summarization, and reasoning.</figcaption>
</figure>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Future Prospects</h2>

<section id="S5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.1 </span>Leveraging Information Retrieval and External Memory</h3>

<div id="S5.SS1.p1" class="ltx_para">
<p id="S5.SS1.p1.1" class="ltx_p">Previous research has demonstrated the benefits of utilizing external knowledge bases, as opposed to relying solely on internal knowledge, to guide LLMs in generating content based on a predefined set of facts. These methods effectively separate the factual knowledge stored in LLMs from the inference processes, thus reducing potential biases encoded within the models.</p>
</div>
<div id="S5.SS1.p2" class="ltx_para">
<p id="S5.SS1.p2.1" class="ltx_p">External knowledge bases can include diverse sources such as extensive text corpora, structured tables, or even simple key-value databases. Once these knowledge sources are provided, one can either finetune the LLMs to enhance their ability to retrieve information or employ prompting and in-context learning techniques to query these sources while keeping the model parameters intact. Such approaches not only eliminate the need to verify and edit false factual knowledge within the LLMs but also facilitate the use of attribution and reflection methods. This ensures that the generated content aligns with the predefined external knowledge base, thereby enhancing both accuracy and accountability.</p>
</div>
</section>
<section id="S5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.2 </span>Improve Understandings of LLMs’ Internal Knowledge Structures</h3>

<div id="S5.SS2.p1" class="ltx_para">
<p id="S5.SS2.p1.1" class="ltx_p">While the identification of factual knowledge storage in LLMs has been extensively explored in recent literature <cite class="ltx_cite ltx_citemacro_citep">(Meng et al., <a href="#bib.bib29" title="" class="ltx_ref">2022</a>, <a href="#bib.bib30" title="" class="ltx_ref">2023</a>; Dai et al., <a href="#bib.bib9" title="" class="ltx_ref">2022</a>; Hernandez et al., <a href="#bib.bib18" title="" class="ltx_ref">2024</a>; Geva et al., <a href="#bib.bib12" title="" class="ltx_ref">2021</a>)</cite>, the correlation between the location of knowledge and the success rate of model editing remains low <cite class="ltx_cite ltx_citemacro_citep">(Hase et al., <a href="#bib.bib17" title="" class="ltx_ref">2023</a>)</cite>. Additionally, despite evidence suggesting a strong connection between factual knowledge and the feed-forward network layers <cite class="ltx_cite ltx_citemacro_citep">(Meng et al., <a href="#bib.bib29" title="" class="ltx_ref">2022</a>; Geva et al., <a href="#bib.bib12" title="" class="ltx_ref">2021</a>, <a href="#bib.bib11" title="" class="ltx_ref">2022</a>)</cite>, recent findings <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib25" title="" class="ltx_ref">2024a</a>)</cite> indicate that updates to multi-head self-attention layers also lead to improved outcomes. These studies highlight that merely locating fact storage does not fully elucidate the underlying mechanisms of knowledge structures in LLMs. Therefore, further research into how knowledge locations interact with model predictions is essential for advancing the interpretability and controllability of LLMs.</p>
</div>
<div id="S5.SS2.p2" class="ltx_para">
<p id="S5.SS2.p2.1" class="ltx_p">In addition to enhancing the success rate of edits, preserving the general capabilities of LLMs is crucial for assessing the efficacy of model editing methods, as discussed in Section <a href="#S3.SS3" title="3.3 Deterioration of General LLM Abilities ‣ 3 Challenges of Knowledge Editing ‣ Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3</span></a>. Recent breakthroughs in identifying regions within models that correlate with general linguistic abilities <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al., <a href="#bib.bib50" title="" class="ltx_ref">2024b</a>)</cite> have opened up a direction for future research in model editing. Specifically, by locating these critical areas, it is possible to perform targeted modifications while keeping these regions away from alterations, thereby preventing the deterioration of general abilities. Consequently, advancements in related fields would ensure that edits could be performed without compromising the overall performance of the LLMs, thereby significantly enhancing the specificity and effectiveness of current model editing methods.</p>
</div>
</section>
<section id="S5.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.3 </span>Improve Robustness of Knowledge Editing</h3>

<div id="S5.SS3.p1" class="ltx_para">
<p id="S5.SS3.p1.1" class="ltx_p">Even after successful (achieving fair scores on the existing metrics) edit, the revised model may refuse the modification if the knowledge concerning the altered concept is challenged by extended dialogues. Instead, it might revert to the pre-edit version (<span id="S5.SS3.p1.1.1" class="ltx_text ltx_font_bold">reversion</span>), or output an ambiguous answer about the edited concept (<span id="S5.SS3.p1.1.2" class="ltx_text ltx_font_bold">confusion</span>). Given the interconnected nature of knowledge, experiments disclose that more popular the knowledge in the benchmark, the easier for the modified model to trace back the original concept <cite class="ltx_cite ltx_citemacro_citep">(Ma et al., <a href="#bib.bib27" title="" class="ltx_ref">2024</a>)</cite>. It highlights the unsatisfying robustness of the existing editing strategies. A more comprehensive understanding how LLMs store and process among different knowledge entities is crucial for a more robust editing. We are also short of specific benchmarks and automated metrics addressing on theses issues. Knowledge-focused editing would not avoid the <span id="S5.SS3.p1.1.3" class="ltx_text ltx_font_bold">hallucination</span> inherited from the pre-edit model. TruthX <cite class="ltx_cite ltx_citemacro_citep">(<span class="ltx_ref ltx_missing_citation ltx_ref_self">zhang2024truthx</span>)</cite> tries to alleviate hallucination via a parameter-preserved approach by mapping the LLMs internal representation to semantic and truthful spaces and edits the truthfulness in the truthful space. Combination of truthfulness and knowledge adjustment in the same space may provide a practical solution.</p>
</div>
</section>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Conclusion</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">Although model editing techniques appear promising for cost-effectively updating knowledge, they still have significant pitfalls. Current editing methods often struggle with making logical inferences based on the edited facts, introducing unintended alterations of non-target knowledge and deterioration in model performance, particularly with parameter-modified methods. Editing techniques that leverage information retrieval can mitigate deviations in model abilities by keeping model parameters intact, as demonstrated in our experiments. Moreover, gaining a deeper understanding of how models store and process knowledge can enhance the controllability of edited facts, leading to greater robustness. We hope our work illuminates potential directions for future improvements in knowledge editing.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Achiam et al. (2023)</span>
<span class="ltx_bibblock">
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.

</span>
<span class="ltx_bibblock">Gpt-4 technical report.

</span>
<span class="ltx_bibblock"><em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2303.08774</em>.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bisk et al. (2020)</span>
<span class="ltx_bibblock">
Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, and Yejin Choi. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.1609/aaai.v34i05.6239" title="" class="ltx_ref ltx_href">Piqa: Reasoning about physical commonsense in natural language</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">Proceedings of the AAAI Conference on Artificial Intelligence</em>, 34(05):7432–7439.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Black et al. (2022)</span>
<span class="ltx_bibblock">
Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2022.bigscience-1.9" title="" class="ltx_ref ltx_href">GPT-NeoX-20B: An open-source autoregressive language model</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">Proceedings of BigScience Episode #5 – Workshop on Challenges &amp; Perspectives in Creating Large Language Models</em>, pages 95–136, virtual+Dublin. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Brown et al. (2020)</span>
<span class="ltx_bibblock">
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf" title="" class="ltx_ref ltx_href">Language models are few-shot learners</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, volume 33, pages 1877–1901. Curran Associates, Inc.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cao et al. (2021)</span>
<span class="ltx_bibblock">
Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2104.08164" title="" class="ltx_ref ltx_href">Editing factual knowledge in language models</a>.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chowdhery et al. (2023)</span>
<span class="ltx_bibblock">
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023.

</span>
<span class="ltx_bibblock">Palm: Scaling language modeling with pathways.

</span>
<span class="ltx_bibblock"><em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">Journal of Machine Learning Research</em>, 24(240):1–113.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cobbe et al. (2021)</span>
<span class="ltx_bibblock">
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2110.14168" title="" class="ltx_ref ltx_href">Training verifiers to solve math word problems</a>.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cohen et al. (2023)</span>
<span class="ltx_bibblock">
Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. 2023.

</span>
<span class="ltx_bibblock">Evaluating the ripple effects of knowledge editing in language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2307.12976</em>.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dai et al. (2022)</span>
<span class="ltx_bibblock">
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022.

</span>
<span class="ltx_bibblock">Knowledge neurons in pretrained transformers.

</span>
<span class="ltx_bibblock">In <em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022</em>, pages 8493–8502.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dong et al. (2022)</span>
<span class="ltx_bibblock">
Qingxiu Dong, Damai Dai, Yifan Song, Jingjing Xu, Zhifang Sui, and Lei Li. 2022.

</span>
<span class="ltx_bibblock">Calibrating factual knowledge in pretrained language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">Findings of Empirical Methods in Natural Language Processing (EMNLP)</em>.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Geva et al. (2022)</span>
<span class="ltx_bibblock">
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2022.emnlp-main.3" title="" class="ltx_ref ltx_href">Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing</em>, pages 30–45, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Geva et al. (2021)</span>
<span class="ltx_bibblock">
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2021.emnlp-main.446" title="" class="ltx_ref ltx_href">Transformer feed-forward layers are key-value memories</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing</em>, pages 5484–5495, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gu et al. (2024)</span>
<span class="ltx_bibblock">
Jia-Chen Gu, Hao-Xiang Xu, Jun-Yu Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Chang, and Nanyun Peng. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2401.04700" title="" class="ltx_ref ltx_href">Model editing can hurt general abilities of large language models</a>.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gupta et al. (2024a)</span>
<span class="ltx_bibblock">
Akshat Gupta, Anurag Rao, and Gopala Anumanchipalli. 2024a.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2401.07453" title="" class="ltx_ref ltx_href">Model editing at scale leads to gradual and catastrophic forgetting</a>.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gupta et al. (2024b)</span>
<span class="ltx_bibblock">
Akshat Gupta, Dev Sajnani, and Gopala Anumanchipalli. 2024b.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2403.14236" title="" class="ltx_ref ltx_href">A unified framework for model editing</a>.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hartvigsen et al. (2023)</span>
<span class="ltx_bibblock">
Thomas Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. 2023.

</span>
<span class="ltx_bibblock">Aging with grace: Lifelong model editing with discrete key-value adaptors.

</span>
<span class="ltx_bibblock">In <em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hase et al. (2023)</span>
<span class="ltx_bibblock">
Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=EldbUlZtbd" title="" class="ltx_ref ltx_href">Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">Thirty-seventh Conference on Neural Information Processing Systems</em>.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hernandez et al. (2024)</span>
<span class="ltx_bibblock">
Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=w7LU2s14kE" title="" class="ltx_ref ltx_href">Linearity of relation decoding in transformer language models</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">The Twelfth International Conference on Learning Representations</em>.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hoelscher-Obermaier et al. (2023)</span>
<span class="ltx_bibblock">
Jason Hoelscher-Obermaier, Julia Persson, Esben Kran, Ioannis Konstas, and Fazl Barez. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2023.findings-acl.733" title="" class="ltx_ref ltx_href">Detecting edit failures in large language models: An improved specificity benchmark</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">Findings of the Association for Computational Linguistics: ACL 2023</em>, pages 11548–11559, Toronto, Canada. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hua et al. (2024)</span>
<span class="ltx_bibblock">
Wenyue Hua, Jiang Guo, Mingwen Dong, Henghui Zhu, Patrick Ng, and Zhiguo Wang. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2401.17585" title="" class="ltx_ref ltx_href">Propagation and pitfalls: Reasoning-based assessment of knowledge editing through counterfactual tasks</a>.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Huang et al. (2023)</span>
<span class="ltx_bibblock">
Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=4oYUGeGBPm" title="" class="ltx_ref ltx_href">Transformer-patcher: One mistake worth one neuron</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">The Eleventh International Conference on Learning Representations</em>.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jiang et al. (2023)</span>
<span class="ltx_bibblock">
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2310.06825" title="" class="ltx_ref ltx_href">Mistral 7b</a>.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kwiatkowski et al. (2019)</span>
<span class="ltx_bibblock">
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.1162/tacl_a_00276" title="" class="ltx_ref ltx_href">Natural questions: A benchmark for question answering research</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">Transactions of the Association for Computational Linguistics</em>, 7:452–466.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Levy et al. (2017)</span>
<span class="ltx_bibblock">
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/K17-1034" title="" class="ltx_ref ltx_href">Zero-shot relation extraction via reading comprehension</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)</em>, pages 333–342, Vancouver, Canada. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2024a)</span>
<span class="ltx_bibblock">
Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun Ma, and Jie Yu. 2024a.

</span>
<span class="ltx_bibblock">Pmet: Precise model editing in a transformer.

</span>
<span class="ltx_bibblock">In <em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">Proceedings of the AAAI Conference on Artificial Intelligence</em>, volume 38, pages 18564–18572.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2024b)</span>
<span class="ltx_bibblock">
Zhoubo Li, Ningyu Zhang, Yunzhi Yao, Mengru Wang, Xi Chen, and Huajun Chen. 2024b.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=fNktD3ib16" title="" class="ltx_ref ltx_href">Unveiling the pitfalls of knowledge editing for large language models</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">The Twelfth International Conference on Learning Representations</em>.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ma et al. (2024)</span>
<span class="ltx_bibblock">
Xinbei Ma, Tianjie Ju, Jiyang Qiu, Zhuosheng Zhang, hai zhao, lifeng Liu, and Yulong Wang. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=dZ2VW5gp5g" title="" class="ltx_ref ltx_href">Is it possible to edit large language models robustly?</a>

</span>
<span class="ltx_bibblock">In <em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">ICLR 2024 Workshop on Large Language Model (LLM) Agents</em>.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mazzia et al. (2023)</span>
<span class="ltx_bibblock">
Vittorio Mazzia, Alessandro Pedrani, Andrea Caciolai, Kay Rottmann, and Davide Bernardi. 2023.

</span>
<span class="ltx_bibblock">A survey on knowledge editing of neural networks.

</span>
<span class="ltx_bibblock"><em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2310.19704</em>.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Meng et al. (2022)</span>
<span class="ltx_bibblock">
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022.

</span>
<span class="ltx_bibblock">Locating and editing factual associations in GPT.

</span>
<span class="ltx_bibblock"><em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36.

</span>
<span class="ltx_bibblock">ArXiv:2202.05262.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Meng et al. (2023)</span>
<span class="ltx_bibblock">
Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2023.

</span>
<span class="ltx_bibblock">Mass editing memory in a transformer.

</span>
<span class="ltx_bibblock"><em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">The Eleventh International Conference on Learning Representations (ICLR)</em>.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Meta (2024)</span>
<span class="ltx_bibblock">
Meta. 2024.

</span>
<span class="ltx_bibblock">Introducing meta llama 3: The most capable openly available llm to date.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://ai.meta.com/blog/meta-llama-3/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://ai.meta.com/blog/meta-llama-3/</a>.

</span>
<span class="ltx_bibblock">Accessed: 2024-05-30.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mitchell et al. (2022a)</span>
<span class="ltx_bibblock">
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2022a.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=0DcZxeWfOPt" title="" class="ltx_ref ltx_href">Fast model editing at scale</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mitchell et al. (2022b)</span>
<span class="ltx_bibblock">
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. 2022b.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/pdf/2206.06520.pdf" title="" class="ltx_ref ltx_href">Memory-based model editing at scale</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">OpenAI (2023)</span>
<span class="ltx_bibblock">
OpenAI. 2023.

</span>
<span class="ltx_bibblock">Chatgpt: Optimizing language models for dialogue.

</span>
<span class="ltx_bibblock"><em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">OpenAI Blog</em>.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openai.com/research/chatgpt" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openai.com/research/chatgpt</a>.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Socher et al. (2013)</span>
<span class="ltx_bibblock">
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://aclanthology.org/D13-1170" title="" class="ltx_ref ltx_href">Recursive deep models for semantic compositionality over a sentiment treebank</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing</em>, pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Tan et al. (2024)</span>
<span class="ltx_bibblock">
Chenmien Tan, Ge Zhang, and Jie Fu. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/pdf?id=L6L1CJQ2PE" title="" class="ltx_ref ltx_href">Massive editing for large language models via meta learning</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Team (2024)</span>
<span class="ltx_bibblock">
Gemini Team. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2312.11805" title="" class="ltx_ref ltx_href">Gemini: A family of highly capable multimodal models</a>.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. (2023a)</span>
<span class="ltx_bibblock">
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2302.13971" title="" class="ltx_ref ltx_href">Llama: Open and efficient foundation language models</a>.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. (2023b)</span>
<span class="ltx_bibblock">
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas
Scialom. 2023b.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2307.09288" title="" class="ltx_ref ltx_href">Llama 2: Open foundation and fine-tuned chat models</a>.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Valipour et al. (2023)</span>
<span class="ltx_bibblock">
Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2023.eacl-main.239" title="" class="ltx_ref ltx_href">DyLoRA: Parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adaptation</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics</em>, pages 3274–3287, Dubrovnik, Croatia. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang and Komatsuzaki (2021)</span>
<span class="ltx_bibblock">
Ben Wang and Aran Komatsuzaki. 2021.

</span>
<span class="ltx_bibblock">GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/kingoflolz/mesh-transformer-jax" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/kingoflolz/mesh-transformer-jax</a>.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2019)</span>
<span class="ltx_bibblock">
Lu Wang, Jackie Chi Kit Cheung, Giuseppe Carenini, and Fei Liu, editors. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://aclanthology.org/D19-5400" title="" class="ltx_ref ltx_href"><em id="bib.bib42.1.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2nd Workshop on New Frontiers in Summarization</em></a>. Association for Computational Linguistics, Hong Kong, China.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2024)</span>
<span class="ltx_bibblock">
Yiwei Wang, Muhao Chen, Nanyun Peng, and Kai wei Chang. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:267060897" title="" class="ltx_ref ltx_href">Deepedit: Knowledge editing as decoding with constraints</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2401.10471.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">(44)</span>
<span class="ltx_bibblock">
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le.

</span>
<span class="ltx_bibblock">Finetuned language models are zero-shot learners.

</span>
<span class="ltx_bibblock">In <em id="bib.bib44.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yang et al. (2024)</span>
<span class="ltx_bibblock">
Wanli Yang, Fei Sun, Xinyu Ma, Xun Liu, Dawei Yin, and Xueqi Cheng. 2024.

</span>
<span class="ltx_bibblock">The butterfly effect of model editing: Few edits can trigger large language models collapse.

</span>
<span class="ltx_bibblock"><em id="bib.bib45.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.09656</em>.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yao et al. (2023a)</span>
<span class="ltx_bibblock">
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. 2023a.

</span>
<span class="ltx_bibblock">Editing large language models: Problems, methods, and opportunities.

</span>
<span class="ltx_bibblock"><em id="bib.bib46.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.13172</em>.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yao et al. (2023b)</span>
<span class="ltx_bibblock">
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. 2023b.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2023.emnlp-main.632" title="" class="ltx_ref ltx_href">Editing large language models: Problems, methods, and opportunities</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib47.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing</em>, pages 10222–10240, Singapore. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yu et al. (2024)</span>
<span class="ltx_bibblock">
Lang Yu, Qin Chen, Jie Zhou, and Liang He. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.1609/aaai.v38i17.29916" title="" class="ltx_ref ltx_href">Melo: Enhancing model editing with neuron-indexed dynamic lora</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib48.1.1" class="ltx_emph ltx_font_italic">Proceedings of the AAAI Conference on Artificial Intelligence</em>, 38(17):19449–19457.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2024a)</span>
<span class="ltx_bibblock">
Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, et al. 2024a.

</span>
<span class="ltx_bibblock">A comprehensive study of knowledge editing for large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib49.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2401.01286</em>.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2024b)</span>
<span class="ltx_bibblock">
Zhihao Zhang, Jun Zhao, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024b.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2402.14700" title="" class="ltx_ref ltx_href">Unveiling linguistic regions in large language models</a>.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zheng et al. (2023)</span>
<span class="ltx_bibblock">
Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=hsjQHAM8MV" title="" class="ltx_ref ltx_href">Can we edit factual knowledge by in-context learning?</a>

</span>
<span class="ltx_bibblock">In <em id="bib.bib51.1.1" class="ltx_emph ltx_font_italic">The 2023 Conference on Empirical Methods in Natural Language Processing</em>.

</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhong et al. (2023)</span>
<span class="ltx_bibblock">
Zexuan Zhong, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=0hTPJBnncc" title="" class="ltx_ref ltx_href">MQuAKE: Assessing knowledge editing in language models via multi-hop questions</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib52.1.1" class="ltx_emph ltx_font_italic">The 2023 Conference on Empirical Methods in Natural Language Processing</em>.

</span>
</li>
</ul>
</section>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Detailed Explanation of Evaluation Metrics and Examples</h2>

<section id="A1.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.1 </span>Portability / Generalization</h3>

<section id="A1.SS1.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Single Edit</h5>

<div id="A1.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="A1.SS1.SSS0.Px1.p1.1" class="ltx_p">In the single edit scenario, we further classify the methods into two settings:</p>
</div>
<div id="A1.SS1.SSS0.Px1.p2" class="ltx_para">
<ul id="A1.I1" class="ltx_itemize">
<li id="A1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I1.i1.p1" class="ltx_para">
<p id="A1.I1.i1.p1.1" class="ltx_p">One-Hop: This setting focuses on evaluating the impact of a single edit on direct, one-hop reasoning tasks.</p>
</div>
</li>
<li id="A1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I1.i2.p1" class="ltx_para">
<p id="A1.I1.i2.p1.1" class="ltx_p">Multi-Hop: This setting assesses the impact of a single edit on more complex, multi-hop reasoning tasks.</p>
</div>
</li>
</ul>
</div>
<div id="A1.SS1.SSS0.Px1.p3" class="ltx_para">
<p id="A1.SS1.SSS0.Px1.p3.1" class="ltx_p">For one-hop evaluations, we adopt the methods proposed by <cite class="ltx_cite ltx_citemacro_citep">(Yao et al., <a href="#bib.bib46" title="" class="ltx_ref">2023a</a>)</cite>. These include:</p>
</div>
<div id="A1.SS1.SSS0.Px1.p4" class="ltx_para">
<ul id="A1.I2" class="ltx_itemize">
<li id="A1.I2.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I2.i1.p1" class="ltx_para">
<p id="A1.I2.i1.p1.1" class="ltx_p"><span id="A1.I2.i1.p1.1.1" class="ltx_text ltx_font_bold">Subject Replace:</span> This metric tests the model’s generalization ability by replacing the subject in the question with an alias or synonym, assessing if the edited attribute is generalized to other descriptions of the same subject.</p>
</div>
</li>
<li id="A1.I2.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I2.i2.p1" class="ltx_para">
<p id="A1.I2.i2.p1.1" class="ltx_p"><span id="A1.I2.i2.p1.1.1" class="ltx_text ltx_font_bold">Reversed Relation:</span> This metric evaluates the model’s capability to handle reversed relations by filtering for suitable relations such as one-to-one and asking the reverse question to check if the target entity is also updated.</p>
</div>
</li>
<li id="A1.I2.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I2.i3.p1" class="ltx_para">
<p id="A1.I2.i3.p1.1" class="ltx_p"><span id="A1.I2.i3.p1.1.1" class="ltx_text ltx_font_bold">One-Hop Test:</span> This metric assesses the edited language model’s performance on downstream tasks that require one-hop reasoning.</p>
</div>
</li>
</ul>
</div>
<div id="A1.SS1.SSS0.Px1.p5" class="ltx_para">
<p id="A1.SS1.SSS0.Px1.p5.1" class="ltx_p">In the multi-hop setting, we assess the model’s performance on multi-hop questions using the evaluation methods proposed by <cite class="ltx_cite ltx_citemacro_citep">(Zhong et al., <a href="#bib.bib52" title="" class="ltx_ref">2023</a>)</cite>, which include:</p>
</div>
<div id="A1.SS1.SSS0.Px1.p6" class="ltx_para">
<ul id="A1.I3" class="ltx_itemize">
<li id="A1.I3.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I3.i1.p1" class="ltx_para">
<p id="A1.I3.i1.p1.2" class="ltx_p"><span id="A1.I3.i1.p1.2.1" class="ltx_text ltx_font_bold">Edit-wise Success Rate (EW):</span> This metric measures how many facts can be successfully recalled from the edited language model.</p>
<table id="A1.E5" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.E5.m1.2" class="ltx_Math" alttext="\mathrm{EW}=\mathbbm{1}\{f^{*}(s)=o^{*}\}" display="block"><semantics id="A1.E5.m1.2a"><mrow id="A1.E5.m1.2.2" xref="A1.E5.m1.2.2.cmml"><mi id="A1.E5.m1.2.2.3" xref="A1.E5.m1.2.2.3.cmml">EW</mi><mo id="A1.E5.m1.2.2.2" xref="A1.E5.m1.2.2.2.cmml">=</mo><mrow id="A1.E5.m1.2.2.1" xref="A1.E5.m1.2.2.1.cmml"><mn id="A1.E5.m1.2.2.1.3" xref="A1.E5.m1.2.2.1.3.cmml">𝟙</mn><mo lspace="0em" rspace="0em" id="A1.E5.m1.2.2.1.2" xref="A1.E5.m1.2.2.1.2.cmml">​</mo><mrow id="A1.E5.m1.2.2.1.1.1" xref="A1.E5.m1.2.2.1.1.2.cmml"><mo stretchy="false" id="A1.E5.m1.2.2.1.1.1.2" xref="A1.E5.m1.2.2.1.1.2.cmml">{</mo><mrow id="A1.E5.m1.2.2.1.1.1.1" xref="A1.E5.m1.2.2.1.1.1.1.cmml"><mrow id="A1.E5.m1.2.2.1.1.1.1.2" xref="A1.E5.m1.2.2.1.1.1.1.2.cmml"><msup id="A1.E5.m1.2.2.1.1.1.1.2.2" xref="A1.E5.m1.2.2.1.1.1.1.2.2.cmml"><mi id="A1.E5.m1.2.2.1.1.1.1.2.2.2" xref="A1.E5.m1.2.2.1.1.1.1.2.2.2.cmml">f</mi><mo id="A1.E5.m1.2.2.1.1.1.1.2.2.3" xref="A1.E5.m1.2.2.1.1.1.1.2.2.3.cmml">∗</mo></msup><mo lspace="0em" rspace="0em" id="A1.E5.m1.2.2.1.1.1.1.2.1" xref="A1.E5.m1.2.2.1.1.1.1.2.1.cmml">​</mo><mrow id="A1.E5.m1.2.2.1.1.1.1.2.3.2" xref="A1.E5.m1.2.2.1.1.1.1.2.cmml"><mo stretchy="false" id="A1.E5.m1.2.2.1.1.1.1.2.3.2.1" xref="A1.E5.m1.2.2.1.1.1.1.2.cmml">(</mo><mi id="A1.E5.m1.1.1" xref="A1.E5.m1.1.1.cmml">s</mi><mo stretchy="false" id="A1.E5.m1.2.2.1.1.1.1.2.3.2.2" xref="A1.E5.m1.2.2.1.1.1.1.2.cmml">)</mo></mrow></mrow><mo id="A1.E5.m1.2.2.1.1.1.1.1" xref="A1.E5.m1.2.2.1.1.1.1.1.cmml">=</mo><msup id="A1.E5.m1.2.2.1.1.1.1.3" xref="A1.E5.m1.2.2.1.1.1.1.3.cmml"><mi id="A1.E5.m1.2.2.1.1.1.1.3.2" xref="A1.E5.m1.2.2.1.1.1.1.3.2.cmml">o</mi><mo id="A1.E5.m1.2.2.1.1.1.1.3.3" xref="A1.E5.m1.2.2.1.1.1.1.3.3.cmml">∗</mo></msup></mrow><mo stretchy="false" id="A1.E5.m1.2.2.1.1.1.3" xref="A1.E5.m1.2.2.1.1.2.cmml">}</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E5.m1.2b"><apply id="A1.E5.m1.2.2.cmml" xref="A1.E5.m1.2.2"><eq id="A1.E5.m1.2.2.2.cmml" xref="A1.E5.m1.2.2.2"></eq><ci id="A1.E5.m1.2.2.3.cmml" xref="A1.E5.m1.2.2.3">EW</ci><apply id="A1.E5.m1.2.2.1.cmml" xref="A1.E5.m1.2.2.1"><times id="A1.E5.m1.2.2.1.2.cmml" xref="A1.E5.m1.2.2.1.2"></times><cn type="integer" id="A1.E5.m1.2.2.1.3.cmml" xref="A1.E5.m1.2.2.1.3">1</cn><set id="A1.E5.m1.2.2.1.1.2.cmml" xref="A1.E5.m1.2.2.1.1.1"><apply id="A1.E5.m1.2.2.1.1.1.1.cmml" xref="A1.E5.m1.2.2.1.1.1.1"><eq id="A1.E5.m1.2.2.1.1.1.1.1.cmml" xref="A1.E5.m1.2.2.1.1.1.1.1"></eq><apply id="A1.E5.m1.2.2.1.1.1.1.2.cmml" xref="A1.E5.m1.2.2.1.1.1.1.2"><times id="A1.E5.m1.2.2.1.1.1.1.2.1.cmml" xref="A1.E5.m1.2.2.1.1.1.1.2.1"></times><apply id="A1.E5.m1.2.2.1.1.1.1.2.2.cmml" xref="A1.E5.m1.2.2.1.1.1.1.2.2"><csymbol cd="ambiguous" id="A1.E5.m1.2.2.1.1.1.1.2.2.1.cmml" xref="A1.E5.m1.2.2.1.1.1.1.2.2">superscript</csymbol><ci id="A1.E5.m1.2.2.1.1.1.1.2.2.2.cmml" xref="A1.E5.m1.2.2.1.1.1.1.2.2.2">𝑓</ci><times id="A1.E5.m1.2.2.1.1.1.1.2.2.3.cmml" xref="A1.E5.m1.2.2.1.1.1.1.2.2.3"></times></apply><ci id="A1.E5.m1.1.1.cmml" xref="A1.E5.m1.1.1">𝑠</ci></apply><apply id="A1.E5.m1.2.2.1.1.1.1.3.cmml" xref="A1.E5.m1.2.2.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.E5.m1.2.2.1.1.1.1.3.1.cmml" xref="A1.E5.m1.2.2.1.1.1.1.3">superscript</csymbol><ci id="A1.E5.m1.2.2.1.1.1.1.3.2.cmml" xref="A1.E5.m1.2.2.1.1.1.1.3.2">𝑜</ci><times id="A1.E5.m1.2.2.1.1.1.1.3.3.cmml" xref="A1.E5.m1.2.2.1.1.1.1.3.3"></times></apply></apply></set></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E5.m1.2c">\mathrm{EW}=\mathbbm{1}\{f^{*}(s)=o^{*}\}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(5)</span></td>
</tr></tbody>
</table>
<p id="A1.I3.i1.p1.1" class="ltx_p">where <math id="A1.I3.i1.p1.1.m1.1" class="ltx_Math" alttext="f^{*}" display="inline"><semantics id="A1.I3.i1.p1.1.m1.1a"><msup id="A1.I3.i1.p1.1.m1.1.1" xref="A1.I3.i1.p1.1.m1.1.1.cmml"><mi id="A1.I3.i1.p1.1.m1.1.1.2" xref="A1.I3.i1.p1.1.m1.1.1.2.cmml">f</mi><mo id="A1.I3.i1.p1.1.m1.1.1.3" xref="A1.I3.i1.p1.1.m1.1.1.3.cmml">∗</mo></msup><annotation-xml encoding="MathML-Content" id="A1.I3.i1.p1.1.m1.1b"><apply id="A1.I3.i1.p1.1.m1.1.1.cmml" xref="A1.I3.i1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A1.I3.i1.p1.1.m1.1.1.1.cmml" xref="A1.I3.i1.p1.1.m1.1.1">superscript</csymbol><ci id="A1.I3.i1.p1.1.m1.1.1.2.cmml" xref="A1.I3.i1.p1.1.m1.1.1.2">𝑓</ci><times id="A1.I3.i1.p1.1.m1.1.1.3.cmml" xref="A1.I3.i1.p1.1.m1.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I3.i1.p1.1.m1.1c">f^{*}</annotation></semantics></math> is the model after editing.</p>
</div>
</li>
<li id="A1.I3.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I3.i2.p1" class="ltx_para">
<p id="A1.I3.i2.p1.4" class="ltx_p"><span id="A1.I3.i2.p1.4.1" class="ltx_text ltx_font_bold">Instance-wise Accuracy (IW):</span> This metric tests how many multi-hop instances the model can recall all the individual single-hop facts. This metric is crucial for multi-hop performance, as the model must encode each fact to answer the multi-hop question.</p>
<table id="A1.E6" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.E6.m1.5" class="ltx_Math" alttext="\mathrm{IW}=\mathbbm{1}\{\bigwedge_{(s,r,o^{*})\in C^{*}}[f^{*}(s)=o^{*}]\}" display="block"><semantics id="A1.E6.m1.5a"><mrow id="A1.E6.m1.5.5" xref="A1.E6.m1.5.5.cmml"><mi id="A1.E6.m1.5.5.3" xref="A1.E6.m1.5.5.3.cmml">IW</mi><mo id="A1.E6.m1.5.5.2" xref="A1.E6.m1.5.5.2.cmml">=</mo><mrow id="A1.E6.m1.5.5.1" xref="A1.E6.m1.5.5.1.cmml"><mn id="A1.E6.m1.5.5.1.3" xref="A1.E6.m1.5.5.1.3.cmml">𝟙</mn><mo lspace="0em" rspace="0em" id="A1.E6.m1.5.5.1.2" xref="A1.E6.m1.5.5.1.2.cmml">​</mo><mrow id="A1.E6.m1.5.5.1.1.1" xref="A1.E6.m1.5.5.1.1.2.cmml"><mo stretchy="false" id="A1.E6.m1.5.5.1.1.1.2" xref="A1.E6.m1.5.5.1.1.2.cmml">{</mo><mrow id="A1.E6.m1.5.5.1.1.1.1" xref="A1.E6.m1.5.5.1.1.1.1.cmml"><munder id="A1.E6.m1.5.5.1.1.1.1.2" xref="A1.E6.m1.5.5.1.1.1.1.2.cmml"><mo lspace="0em" movablelimits="false" rspace="0em" id="A1.E6.m1.5.5.1.1.1.1.2.2" xref="A1.E6.m1.5.5.1.1.1.1.2.2.cmml">⋀</mo><mrow id="A1.E6.m1.3.3.3" xref="A1.E6.m1.3.3.3.cmml"><mrow id="A1.E6.m1.3.3.3.3.1" xref="A1.E6.m1.3.3.3.3.2.cmml"><mo stretchy="false" id="A1.E6.m1.3.3.3.3.1.2" xref="A1.E6.m1.3.3.3.3.2.cmml">(</mo><mi id="A1.E6.m1.1.1.1.1" xref="A1.E6.m1.1.1.1.1.cmml">s</mi><mo id="A1.E6.m1.3.3.3.3.1.3" xref="A1.E6.m1.3.3.3.3.2.cmml">,</mo><mi id="A1.E6.m1.2.2.2.2" xref="A1.E6.m1.2.2.2.2.cmml">r</mi><mo id="A1.E6.m1.3.3.3.3.1.4" xref="A1.E6.m1.3.3.3.3.2.cmml">,</mo><msup id="A1.E6.m1.3.3.3.3.1.1" xref="A1.E6.m1.3.3.3.3.1.1.cmml"><mi id="A1.E6.m1.3.3.3.3.1.1.2" xref="A1.E6.m1.3.3.3.3.1.1.2.cmml">o</mi><mo id="A1.E6.m1.3.3.3.3.1.1.3" xref="A1.E6.m1.3.3.3.3.1.1.3.cmml">∗</mo></msup><mo stretchy="false" id="A1.E6.m1.3.3.3.3.1.5" xref="A1.E6.m1.3.3.3.3.2.cmml">)</mo></mrow><mo id="A1.E6.m1.3.3.3.4" xref="A1.E6.m1.3.3.3.4.cmml">∈</mo><msup id="A1.E6.m1.3.3.3.5" xref="A1.E6.m1.3.3.3.5.cmml"><mi id="A1.E6.m1.3.3.3.5.2" xref="A1.E6.m1.3.3.3.5.2.cmml">C</mi><mo id="A1.E6.m1.3.3.3.5.3" xref="A1.E6.m1.3.3.3.5.3.cmml">∗</mo></msup></mrow></munder><mrow id="A1.E6.m1.5.5.1.1.1.1.1.1" xref="A1.E6.m1.5.5.1.1.1.1.1.2.cmml"><mo stretchy="false" id="A1.E6.m1.5.5.1.1.1.1.1.1.2" xref="A1.E6.m1.5.5.1.1.1.1.1.2.1.cmml">[</mo><mrow id="A1.E6.m1.5.5.1.1.1.1.1.1.1" xref="A1.E6.m1.5.5.1.1.1.1.1.1.1.cmml"><mrow id="A1.E6.m1.5.5.1.1.1.1.1.1.1.2" xref="A1.E6.m1.5.5.1.1.1.1.1.1.1.2.cmml"><msup id="A1.E6.m1.5.5.1.1.1.1.1.1.1.2.2" xref="A1.E6.m1.5.5.1.1.1.1.1.1.1.2.2.cmml"><mi id="A1.E6.m1.5.5.1.1.1.1.1.1.1.2.2.2" xref="A1.E6.m1.5.5.1.1.1.1.1.1.1.2.2.2.cmml">f</mi><mo id="A1.E6.m1.5.5.1.1.1.1.1.1.1.2.2.3" xref="A1.E6.m1.5.5.1.1.1.1.1.1.1.2.2.3.cmml">∗</mo></msup><mo lspace="0em" rspace="0em" id="A1.E6.m1.5.5.1.1.1.1.1.1.1.2.1" xref="A1.E6.m1.5.5.1.1.1.1.1.1.1.2.1.cmml">​</mo><mrow id="A1.E6.m1.5.5.1.1.1.1.1.1.1.2.3.2" xref="A1.E6.m1.5.5.1.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="A1.E6.m1.5.5.1.1.1.1.1.1.1.2.3.2.1" xref="A1.E6.m1.5.5.1.1.1.1.1.1.1.2.cmml">(</mo><mi id="A1.E6.m1.4.4" xref="A1.E6.m1.4.4.cmml">s</mi><mo stretchy="false" id="A1.E6.m1.5.5.1.1.1.1.1.1.1.2.3.2.2" xref="A1.E6.m1.5.5.1.1.1.1.1.1.1.2.cmml">)</mo></mrow></mrow><mo id="A1.E6.m1.5.5.1.1.1.1.1.1.1.1" xref="A1.E6.m1.5.5.1.1.1.1.1.1.1.1.cmml">=</mo><msup id="A1.E6.m1.5.5.1.1.1.1.1.1.1.3" xref="A1.E6.m1.5.5.1.1.1.1.1.1.1.3.cmml"><mi id="A1.E6.m1.5.5.1.1.1.1.1.1.1.3.2" xref="A1.E6.m1.5.5.1.1.1.1.1.1.1.3.2.cmml">o</mi><mo id="A1.E6.m1.5.5.1.1.1.1.1.1.1.3.3" xref="A1.E6.m1.5.5.1.1.1.1.1.1.1.3.3.cmml">∗</mo></msup></mrow><mo stretchy="false" id="A1.E6.m1.5.5.1.1.1.1.1.1.3" xref="A1.E6.m1.5.5.1.1.1.1.1.2.1.cmml">]</mo></mrow></mrow><mo stretchy="false" id="A1.E6.m1.5.5.1.1.1.3" xref="A1.E6.m1.5.5.1.1.2.cmml">}</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E6.m1.5b"><apply id="A1.E6.m1.5.5.cmml" xref="A1.E6.m1.5.5"><eq id="A1.E6.m1.5.5.2.cmml" xref="A1.E6.m1.5.5.2"></eq><ci id="A1.E6.m1.5.5.3.cmml" xref="A1.E6.m1.5.5.3">IW</ci><apply id="A1.E6.m1.5.5.1.cmml" xref="A1.E6.m1.5.5.1"><times id="A1.E6.m1.5.5.1.2.cmml" xref="A1.E6.m1.5.5.1.2"></times><cn type="integer" id="A1.E6.m1.5.5.1.3.cmml" xref="A1.E6.m1.5.5.1.3">1</cn><set id="A1.E6.m1.5.5.1.1.2.cmml" xref="A1.E6.m1.5.5.1.1.1"><apply id="A1.E6.m1.5.5.1.1.1.1.cmml" xref="A1.E6.m1.5.5.1.1.1.1"><apply id="A1.E6.m1.5.5.1.1.1.1.2.cmml" xref="A1.E6.m1.5.5.1.1.1.1.2"><csymbol cd="ambiguous" id="A1.E6.m1.5.5.1.1.1.1.2.1.cmml" xref="A1.E6.m1.5.5.1.1.1.1.2">subscript</csymbol><and id="A1.E6.m1.5.5.1.1.1.1.2.2.cmml" xref="A1.E6.m1.5.5.1.1.1.1.2.2"></and><apply id="A1.E6.m1.3.3.3.cmml" xref="A1.E6.m1.3.3.3"><in id="A1.E6.m1.3.3.3.4.cmml" xref="A1.E6.m1.3.3.3.4"></in><vector id="A1.E6.m1.3.3.3.3.2.cmml" xref="A1.E6.m1.3.3.3.3.1"><ci id="A1.E6.m1.1.1.1.1.cmml" xref="A1.E6.m1.1.1.1.1">𝑠</ci><ci id="A1.E6.m1.2.2.2.2.cmml" xref="A1.E6.m1.2.2.2.2">𝑟</ci><apply id="A1.E6.m1.3.3.3.3.1.1.cmml" xref="A1.E6.m1.3.3.3.3.1.1"><csymbol cd="ambiguous" id="A1.E6.m1.3.3.3.3.1.1.1.cmml" xref="A1.E6.m1.3.3.3.3.1.1">superscript</csymbol><ci id="A1.E6.m1.3.3.3.3.1.1.2.cmml" xref="A1.E6.m1.3.3.3.3.1.1.2">𝑜</ci><times id="A1.E6.m1.3.3.3.3.1.1.3.cmml" xref="A1.E6.m1.3.3.3.3.1.1.3"></times></apply></vector><apply id="A1.E6.m1.3.3.3.5.cmml" xref="A1.E6.m1.3.3.3.5"><csymbol cd="ambiguous" id="A1.E6.m1.3.3.3.5.1.cmml" xref="A1.E6.m1.3.3.3.5">superscript</csymbol><ci id="A1.E6.m1.3.3.3.5.2.cmml" xref="A1.E6.m1.3.3.3.5.2">𝐶</ci><times id="A1.E6.m1.3.3.3.5.3.cmml" xref="A1.E6.m1.3.3.3.5.3"></times></apply></apply></apply><apply id="A1.E6.m1.5.5.1.1.1.1.1.2.cmml" xref="A1.E6.m1.5.5.1.1.1.1.1.1"><csymbol cd="latexml" id="A1.E6.m1.5.5.1.1.1.1.1.2.1.cmml" xref="A1.E6.m1.5.5.1.1.1.1.1.1.2">delimited-[]</csymbol><apply id="A1.E6.m1.5.5.1.1.1.1.1.1.1.cmml" xref="A1.E6.m1.5.5.1.1.1.1.1.1.1"><eq id="A1.E6.m1.5.5.1.1.1.1.1.1.1.1.cmml" xref="A1.E6.m1.5.5.1.1.1.1.1.1.1.1"></eq><apply id="A1.E6.m1.5.5.1.1.1.1.1.1.1.2.cmml" xref="A1.E6.m1.5.5.1.1.1.1.1.1.1.2"><times id="A1.E6.m1.5.5.1.1.1.1.1.1.1.2.1.cmml" xref="A1.E6.m1.5.5.1.1.1.1.1.1.1.2.1"></times><apply id="A1.E6.m1.5.5.1.1.1.1.1.1.1.2.2.cmml" xref="A1.E6.m1.5.5.1.1.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="A1.E6.m1.5.5.1.1.1.1.1.1.1.2.2.1.cmml" xref="A1.E6.m1.5.5.1.1.1.1.1.1.1.2.2">superscript</csymbol><ci id="A1.E6.m1.5.5.1.1.1.1.1.1.1.2.2.2.cmml" xref="A1.E6.m1.5.5.1.1.1.1.1.1.1.2.2.2">𝑓</ci><times id="A1.E6.m1.5.5.1.1.1.1.1.1.1.2.2.3.cmml" xref="A1.E6.m1.5.5.1.1.1.1.1.1.1.2.2.3"></times></apply><ci id="A1.E6.m1.4.4.cmml" xref="A1.E6.m1.4.4">𝑠</ci></apply><apply id="A1.E6.m1.5.5.1.1.1.1.1.1.1.3.cmml" xref="A1.E6.m1.5.5.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.E6.m1.5.5.1.1.1.1.1.1.1.3.1.cmml" xref="A1.E6.m1.5.5.1.1.1.1.1.1.1.3">superscript</csymbol><ci id="A1.E6.m1.5.5.1.1.1.1.1.1.1.3.2.cmml" xref="A1.E6.m1.5.5.1.1.1.1.1.1.1.3.2">𝑜</ci><times id="A1.E6.m1.5.5.1.1.1.1.1.1.1.3.3.cmml" xref="A1.E6.m1.5.5.1.1.1.1.1.1.1.3.3"></times></apply></apply></apply></apply></set></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E6.m1.5c">\mathrm{IW}=\mathbbm{1}\{\bigwedge_{(s,r,o^{*})\in C^{*}}[f^{*}(s)=o^{*}]\}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(6)</span></td>
</tr></tbody>
</table>
<p id="A1.I3.i2.p1.3" class="ltx_p">where <math id="A1.I3.i2.p1.1.m1.3" class="ltx_Math" alttext="C^{*}=\langle(s_{1},r_{1},o_{1}),\dots,(s_{n},r_{n},o_{n})\rangle" display="inline"><semantics id="A1.I3.i2.p1.1.m1.3a"><mrow id="A1.I3.i2.p1.1.m1.3.3" xref="A1.I3.i2.p1.1.m1.3.3.cmml"><msup id="A1.I3.i2.p1.1.m1.3.3.4" xref="A1.I3.i2.p1.1.m1.3.3.4.cmml"><mi id="A1.I3.i2.p1.1.m1.3.3.4.2" xref="A1.I3.i2.p1.1.m1.3.3.4.2.cmml">C</mi><mo id="A1.I3.i2.p1.1.m1.3.3.4.3" xref="A1.I3.i2.p1.1.m1.3.3.4.3.cmml">∗</mo></msup><mo id="A1.I3.i2.p1.1.m1.3.3.3" xref="A1.I3.i2.p1.1.m1.3.3.3.cmml">=</mo><mrow id="A1.I3.i2.p1.1.m1.3.3.2.2" xref="A1.I3.i2.p1.1.m1.3.3.2.3.cmml"><mo stretchy="false" id="A1.I3.i2.p1.1.m1.3.3.2.2.3" xref="A1.I3.i2.p1.1.m1.3.3.2.3.cmml">⟨</mo><mrow id="A1.I3.i2.p1.1.m1.2.2.1.1.1.3" xref="A1.I3.i2.p1.1.m1.2.2.1.1.1.4.cmml"><mo stretchy="false" id="A1.I3.i2.p1.1.m1.2.2.1.1.1.3.4" xref="A1.I3.i2.p1.1.m1.2.2.1.1.1.4.cmml">(</mo><msub id="A1.I3.i2.p1.1.m1.2.2.1.1.1.1.1" xref="A1.I3.i2.p1.1.m1.2.2.1.1.1.1.1.cmml"><mi id="A1.I3.i2.p1.1.m1.2.2.1.1.1.1.1.2" xref="A1.I3.i2.p1.1.m1.2.2.1.1.1.1.1.2.cmml">s</mi><mn id="A1.I3.i2.p1.1.m1.2.2.1.1.1.1.1.3" xref="A1.I3.i2.p1.1.m1.2.2.1.1.1.1.1.3.cmml">1</mn></msub><mo id="A1.I3.i2.p1.1.m1.2.2.1.1.1.3.5" xref="A1.I3.i2.p1.1.m1.2.2.1.1.1.4.cmml">,</mo><msub id="A1.I3.i2.p1.1.m1.2.2.1.1.1.2.2" xref="A1.I3.i2.p1.1.m1.2.2.1.1.1.2.2.cmml"><mi id="A1.I3.i2.p1.1.m1.2.2.1.1.1.2.2.2" xref="A1.I3.i2.p1.1.m1.2.2.1.1.1.2.2.2.cmml">r</mi><mn id="A1.I3.i2.p1.1.m1.2.2.1.1.1.2.2.3" xref="A1.I3.i2.p1.1.m1.2.2.1.1.1.2.2.3.cmml">1</mn></msub><mo id="A1.I3.i2.p1.1.m1.2.2.1.1.1.3.6" xref="A1.I3.i2.p1.1.m1.2.2.1.1.1.4.cmml">,</mo><msub id="A1.I3.i2.p1.1.m1.2.2.1.1.1.3.3" xref="A1.I3.i2.p1.1.m1.2.2.1.1.1.3.3.cmml"><mi id="A1.I3.i2.p1.1.m1.2.2.1.1.1.3.3.2" xref="A1.I3.i2.p1.1.m1.2.2.1.1.1.3.3.2.cmml">o</mi><mn id="A1.I3.i2.p1.1.m1.2.2.1.1.1.3.3.3" xref="A1.I3.i2.p1.1.m1.2.2.1.1.1.3.3.3.cmml">1</mn></msub><mo stretchy="false" id="A1.I3.i2.p1.1.m1.2.2.1.1.1.3.7" xref="A1.I3.i2.p1.1.m1.2.2.1.1.1.4.cmml">)</mo></mrow><mo id="A1.I3.i2.p1.1.m1.3.3.2.2.4" xref="A1.I3.i2.p1.1.m1.3.3.2.3.cmml">,</mo><mi mathvariant="normal" id="A1.I3.i2.p1.1.m1.1.1" xref="A1.I3.i2.p1.1.m1.1.1.cmml">…</mi><mo id="A1.I3.i2.p1.1.m1.3.3.2.2.5" xref="A1.I3.i2.p1.1.m1.3.3.2.3.cmml">,</mo><mrow id="A1.I3.i2.p1.1.m1.3.3.2.2.2.3" xref="A1.I3.i2.p1.1.m1.3.3.2.2.2.4.cmml"><mo stretchy="false" id="A1.I3.i2.p1.1.m1.3.3.2.2.2.3.4" xref="A1.I3.i2.p1.1.m1.3.3.2.2.2.4.cmml">(</mo><msub id="A1.I3.i2.p1.1.m1.3.3.2.2.2.1.1" xref="A1.I3.i2.p1.1.m1.3.3.2.2.2.1.1.cmml"><mi id="A1.I3.i2.p1.1.m1.3.3.2.2.2.1.1.2" xref="A1.I3.i2.p1.1.m1.3.3.2.2.2.1.1.2.cmml">s</mi><mi id="A1.I3.i2.p1.1.m1.3.3.2.2.2.1.1.3" xref="A1.I3.i2.p1.1.m1.3.3.2.2.2.1.1.3.cmml">n</mi></msub><mo id="A1.I3.i2.p1.1.m1.3.3.2.2.2.3.5" xref="A1.I3.i2.p1.1.m1.3.3.2.2.2.4.cmml">,</mo><msub id="A1.I3.i2.p1.1.m1.3.3.2.2.2.2.2" xref="A1.I3.i2.p1.1.m1.3.3.2.2.2.2.2.cmml"><mi id="A1.I3.i2.p1.1.m1.3.3.2.2.2.2.2.2" xref="A1.I3.i2.p1.1.m1.3.3.2.2.2.2.2.2.cmml">r</mi><mi id="A1.I3.i2.p1.1.m1.3.3.2.2.2.2.2.3" xref="A1.I3.i2.p1.1.m1.3.3.2.2.2.2.2.3.cmml">n</mi></msub><mo id="A1.I3.i2.p1.1.m1.3.3.2.2.2.3.6" xref="A1.I3.i2.p1.1.m1.3.3.2.2.2.4.cmml">,</mo><msub id="A1.I3.i2.p1.1.m1.3.3.2.2.2.3.3" xref="A1.I3.i2.p1.1.m1.3.3.2.2.2.3.3.cmml"><mi id="A1.I3.i2.p1.1.m1.3.3.2.2.2.3.3.2" xref="A1.I3.i2.p1.1.m1.3.3.2.2.2.3.3.2.cmml">o</mi><mi id="A1.I3.i2.p1.1.m1.3.3.2.2.2.3.3.3" xref="A1.I3.i2.p1.1.m1.3.3.2.2.2.3.3.3.cmml">n</mi></msub><mo stretchy="false" id="A1.I3.i2.p1.1.m1.3.3.2.2.2.3.7" xref="A1.I3.i2.p1.1.m1.3.3.2.2.2.4.cmml">)</mo></mrow><mo stretchy="false" id="A1.I3.i2.p1.1.m1.3.3.2.2.6" xref="A1.I3.i2.p1.1.m1.3.3.2.3.cmml">⟩</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.I3.i2.p1.1.m1.3b"><apply id="A1.I3.i2.p1.1.m1.3.3.cmml" xref="A1.I3.i2.p1.1.m1.3.3"><eq id="A1.I3.i2.p1.1.m1.3.3.3.cmml" xref="A1.I3.i2.p1.1.m1.3.3.3"></eq><apply id="A1.I3.i2.p1.1.m1.3.3.4.cmml" xref="A1.I3.i2.p1.1.m1.3.3.4"><csymbol cd="ambiguous" id="A1.I3.i2.p1.1.m1.3.3.4.1.cmml" xref="A1.I3.i2.p1.1.m1.3.3.4">superscript</csymbol><ci id="A1.I3.i2.p1.1.m1.3.3.4.2.cmml" xref="A1.I3.i2.p1.1.m1.3.3.4.2">𝐶</ci><times id="A1.I3.i2.p1.1.m1.3.3.4.3.cmml" xref="A1.I3.i2.p1.1.m1.3.3.4.3"></times></apply><list id="A1.I3.i2.p1.1.m1.3.3.2.3.cmml" xref="A1.I3.i2.p1.1.m1.3.3.2.2"><vector id="A1.I3.i2.p1.1.m1.2.2.1.1.1.4.cmml" xref="A1.I3.i2.p1.1.m1.2.2.1.1.1.3"><apply id="A1.I3.i2.p1.1.m1.2.2.1.1.1.1.1.cmml" xref="A1.I3.i2.p1.1.m1.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.I3.i2.p1.1.m1.2.2.1.1.1.1.1.1.cmml" xref="A1.I3.i2.p1.1.m1.2.2.1.1.1.1.1">subscript</csymbol><ci id="A1.I3.i2.p1.1.m1.2.2.1.1.1.1.1.2.cmml" xref="A1.I3.i2.p1.1.m1.2.2.1.1.1.1.1.2">𝑠</ci><cn type="integer" id="A1.I3.i2.p1.1.m1.2.2.1.1.1.1.1.3.cmml" xref="A1.I3.i2.p1.1.m1.2.2.1.1.1.1.1.3">1</cn></apply><apply id="A1.I3.i2.p1.1.m1.2.2.1.1.1.2.2.cmml" xref="A1.I3.i2.p1.1.m1.2.2.1.1.1.2.2"><csymbol cd="ambiguous" id="A1.I3.i2.p1.1.m1.2.2.1.1.1.2.2.1.cmml" xref="A1.I3.i2.p1.1.m1.2.2.1.1.1.2.2">subscript</csymbol><ci id="A1.I3.i2.p1.1.m1.2.2.1.1.1.2.2.2.cmml" xref="A1.I3.i2.p1.1.m1.2.2.1.1.1.2.2.2">𝑟</ci><cn type="integer" id="A1.I3.i2.p1.1.m1.2.2.1.1.1.2.2.3.cmml" xref="A1.I3.i2.p1.1.m1.2.2.1.1.1.2.2.3">1</cn></apply><apply id="A1.I3.i2.p1.1.m1.2.2.1.1.1.3.3.cmml" xref="A1.I3.i2.p1.1.m1.2.2.1.1.1.3.3"><csymbol cd="ambiguous" id="A1.I3.i2.p1.1.m1.2.2.1.1.1.3.3.1.cmml" xref="A1.I3.i2.p1.1.m1.2.2.1.1.1.3.3">subscript</csymbol><ci id="A1.I3.i2.p1.1.m1.2.2.1.1.1.3.3.2.cmml" xref="A1.I3.i2.p1.1.m1.2.2.1.1.1.3.3.2">𝑜</ci><cn type="integer" id="A1.I3.i2.p1.1.m1.2.2.1.1.1.3.3.3.cmml" xref="A1.I3.i2.p1.1.m1.2.2.1.1.1.3.3.3">1</cn></apply></vector><ci id="A1.I3.i2.p1.1.m1.1.1.cmml" xref="A1.I3.i2.p1.1.m1.1.1">…</ci><vector id="A1.I3.i2.p1.1.m1.3.3.2.2.2.4.cmml" xref="A1.I3.i2.p1.1.m1.3.3.2.2.2.3"><apply id="A1.I3.i2.p1.1.m1.3.3.2.2.2.1.1.cmml" xref="A1.I3.i2.p1.1.m1.3.3.2.2.2.1.1"><csymbol cd="ambiguous" id="A1.I3.i2.p1.1.m1.3.3.2.2.2.1.1.1.cmml" xref="A1.I3.i2.p1.1.m1.3.3.2.2.2.1.1">subscript</csymbol><ci id="A1.I3.i2.p1.1.m1.3.3.2.2.2.1.1.2.cmml" xref="A1.I3.i2.p1.1.m1.3.3.2.2.2.1.1.2">𝑠</ci><ci id="A1.I3.i2.p1.1.m1.3.3.2.2.2.1.1.3.cmml" xref="A1.I3.i2.p1.1.m1.3.3.2.2.2.1.1.3">𝑛</ci></apply><apply id="A1.I3.i2.p1.1.m1.3.3.2.2.2.2.2.cmml" xref="A1.I3.i2.p1.1.m1.3.3.2.2.2.2.2"><csymbol cd="ambiguous" id="A1.I3.i2.p1.1.m1.3.3.2.2.2.2.2.1.cmml" xref="A1.I3.i2.p1.1.m1.3.3.2.2.2.2.2">subscript</csymbol><ci id="A1.I3.i2.p1.1.m1.3.3.2.2.2.2.2.2.cmml" xref="A1.I3.i2.p1.1.m1.3.3.2.2.2.2.2.2">𝑟</ci><ci id="A1.I3.i2.p1.1.m1.3.3.2.2.2.2.2.3.cmml" xref="A1.I3.i2.p1.1.m1.3.3.2.2.2.2.2.3">𝑛</ci></apply><apply id="A1.I3.i2.p1.1.m1.3.3.2.2.2.3.3.cmml" xref="A1.I3.i2.p1.1.m1.3.3.2.2.2.3.3"><csymbol cd="ambiguous" id="A1.I3.i2.p1.1.m1.3.3.2.2.2.3.3.1.cmml" xref="A1.I3.i2.p1.1.m1.3.3.2.2.2.3.3">subscript</csymbol><ci id="A1.I3.i2.p1.1.m1.3.3.2.2.2.3.3.2.cmml" xref="A1.I3.i2.p1.1.m1.3.3.2.2.2.3.3.2">𝑜</ci><ci id="A1.I3.i2.p1.1.m1.3.3.2.2.2.3.3.3.cmml" xref="A1.I3.i2.p1.1.m1.3.3.2.2.2.3.3.3">𝑛</ci></apply></vector></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I3.i2.p1.1.m1.3c">C^{*}=\langle(s_{1},r_{1},o_{1}),\dots,(s_{n},r_{n},o_{n})\rangle</annotation></semantics></math> is the chain of facts of a multi-hop question. In this chain, the object of the <math id="A1.I3.i2.p1.2.m2.1" class="ltx_Math" alttext="i^{\mathrm{th}}" display="inline"><semantics id="A1.I3.i2.p1.2.m2.1a"><msup id="A1.I3.i2.p1.2.m2.1.1" xref="A1.I3.i2.p1.2.m2.1.1.cmml"><mi id="A1.I3.i2.p1.2.m2.1.1.2" xref="A1.I3.i2.p1.2.m2.1.1.2.cmml">i</mi><mi id="A1.I3.i2.p1.2.m2.1.1.3" xref="A1.I3.i2.p1.2.m2.1.1.3.cmml">th</mi></msup><annotation-xml encoding="MathML-Content" id="A1.I3.i2.p1.2.m2.1b"><apply id="A1.I3.i2.p1.2.m2.1.1.cmml" xref="A1.I3.i2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="A1.I3.i2.p1.2.m2.1.1.1.cmml" xref="A1.I3.i2.p1.2.m2.1.1">superscript</csymbol><ci id="A1.I3.i2.p1.2.m2.1.1.2.cmml" xref="A1.I3.i2.p1.2.m2.1.1.2">𝑖</ci><ci id="A1.I3.i2.p1.2.m2.1.1.3.cmml" xref="A1.I3.i2.p1.2.m2.1.1.3">th</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I3.i2.p1.2.m2.1c">i^{\mathrm{th}}</annotation></semantics></math> fact is the subject of the next fact. (i.e., <math id="A1.I3.i2.p1.3.m3.1" class="ltx_Math" alttext="o_{i}=s_{i+1}" display="inline"><semantics id="A1.I3.i2.p1.3.m3.1a"><mrow id="A1.I3.i2.p1.3.m3.1.1" xref="A1.I3.i2.p1.3.m3.1.1.cmml"><msub id="A1.I3.i2.p1.3.m3.1.1.2" xref="A1.I3.i2.p1.3.m3.1.1.2.cmml"><mi id="A1.I3.i2.p1.3.m3.1.1.2.2" xref="A1.I3.i2.p1.3.m3.1.1.2.2.cmml">o</mi><mi id="A1.I3.i2.p1.3.m3.1.1.2.3" xref="A1.I3.i2.p1.3.m3.1.1.2.3.cmml">i</mi></msub><mo id="A1.I3.i2.p1.3.m3.1.1.1" xref="A1.I3.i2.p1.3.m3.1.1.1.cmml">=</mo><msub id="A1.I3.i2.p1.3.m3.1.1.3" xref="A1.I3.i2.p1.3.m3.1.1.3.cmml"><mi id="A1.I3.i2.p1.3.m3.1.1.3.2" xref="A1.I3.i2.p1.3.m3.1.1.3.2.cmml">s</mi><mrow id="A1.I3.i2.p1.3.m3.1.1.3.3" xref="A1.I3.i2.p1.3.m3.1.1.3.3.cmml"><mi id="A1.I3.i2.p1.3.m3.1.1.3.3.2" xref="A1.I3.i2.p1.3.m3.1.1.3.3.2.cmml">i</mi><mo id="A1.I3.i2.p1.3.m3.1.1.3.3.1" xref="A1.I3.i2.p1.3.m3.1.1.3.3.1.cmml">+</mo><mn id="A1.I3.i2.p1.3.m3.1.1.3.3.3" xref="A1.I3.i2.p1.3.m3.1.1.3.3.3.cmml">1</mn></mrow></msub></mrow><annotation-xml encoding="MathML-Content" id="A1.I3.i2.p1.3.m3.1b"><apply id="A1.I3.i2.p1.3.m3.1.1.cmml" xref="A1.I3.i2.p1.3.m3.1.1"><eq id="A1.I3.i2.p1.3.m3.1.1.1.cmml" xref="A1.I3.i2.p1.3.m3.1.1.1"></eq><apply id="A1.I3.i2.p1.3.m3.1.1.2.cmml" xref="A1.I3.i2.p1.3.m3.1.1.2"><csymbol cd="ambiguous" id="A1.I3.i2.p1.3.m3.1.1.2.1.cmml" xref="A1.I3.i2.p1.3.m3.1.1.2">subscript</csymbol><ci id="A1.I3.i2.p1.3.m3.1.1.2.2.cmml" xref="A1.I3.i2.p1.3.m3.1.1.2.2">𝑜</ci><ci id="A1.I3.i2.p1.3.m3.1.1.2.3.cmml" xref="A1.I3.i2.p1.3.m3.1.1.2.3">𝑖</ci></apply><apply id="A1.I3.i2.p1.3.m3.1.1.3.cmml" xref="A1.I3.i2.p1.3.m3.1.1.3"><csymbol cd="ambiguous" id="A1.I3.i2.p1.3.m3.1.1.3.1.cmml" xref="A1.I3.i2.p1.3.m3.1.1.3">subscript</csymbol><ci id="A1.I3.i2.p1.3.m3.1.1.3.2.cmml" xref="A1.I3.i2.p1.3.m3.1.1.3.2">𝑠</ci><apply id="A1.I3.i2.p1.3.m3.1.1.3.3.cmml" xref="A1.I3.i2.p1.3.m3.1.1.3.3"><plus id="A1.I3.i2.p1.3.m3.1.1.3.3.1.cmml" xref="A1.I3.i2.p1.3.m3.1.1.3.3.1"></plus><ci id="A1.I3.i2.p1.3.m3.1.1.3.3.2.cmml" xref="A1.I3.i2.p1.3.m3.1.1.3.3.2">𝑖</ci><cn type="integer" id="A1.I3.i2.p1.3.m3.1.1.3.3.3.cmml" xref="A1.I3.i2.p1.3.m3.1.1.3.3.3">1</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I3.i2.p1.3.m3.1c">o_{i}=s_{i+1}</annotation></semantics></math>)</p>
</div>
</li>
<li id="A1.I3.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I3.i3.p1" class="ltx_para">
<p id="A1.I3.i3.p1.3" class="ltx_p"><span id="A1.I3.i3.p1.3.1" class="ltx_text ltx_font_bold">Multi-hop Accuracy (MH):</span> This metric assesses the accuracy of the original and edited language models on multi-hop questions. In the MQuAKE dataset <cite class="ltx_cite ltx_citemacro_citep">(Zhong et al., <a href="#bib.bib52" title="" class="ltx_ref">2023</a>)</cite>, there are three generated multi-hop questions for each instance. If any of the three questions is correctly answered by the model, we consider it accurate.</p>
<table id="A1.E7" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.E7.m1.2" class="ltx_Math" alttext="\mathrm{MH}=\mathbbm{1}\{\bigvee_{q\in Q}f^{*}(q)=a^{*}\}" display="block"><semantics id="A1.E7.m1.2a"><mrow id="A1.E7.m1.2.2" xref="A1.E7.m1.2.2.cmml"><mi id="A1.E7.m1.2.2.3" xref="A1.E7.m1.2.2.3.cmml">MH</mi><mo id="A1.E7.m1.2.2.2" xref="A1.E7.m1.2.2.2.cmml">=</mo><mrow id="A1.E7.m1.2.2.1" xref="A1.E7.m1.2.2.1.cmml"><mn id="A1.E7.m1.2.2.1.3" xref="A1.E7.m1.2.2.1.3.cmml">𝟙</mn><mo lspace="0em" rspace="0em" id="A1.E7.m1.2.2.1.2" xref="A1.E7.m1.2.2.1.2.cmml">​</mo><mrow id="A1.E7.m1.2.2.1.1.1" xref="A1.E7.m1.2.2.1.1.2.cmml"><mo stretchy="false" id="A1.E7.m1.2.2.1.1.1.2" xref="A1.E7.m1.2.2.1.1.2.cmml">{</mo><mrow id="A1.E7.m1.2.2.1.1.1.1" xref="A1.E7.m1.2.2.1.1.1.1.cmml"><mrow id="A1.E7.m1.2.2.1.1.1.1.2" xref="A1.E7.m1.2.2.1.1.1.1.2.cmml"><munder id="A1.E7.m1.2.2.1.1.1.1.2.1" xref="A1.E7.m1.2.2.1.1.1.1.2.1.cmml"><mo lspace="0em" movablelimits="false" id="A1.E7.m1.2.2.1.1.1.1.2.1.2" xref="A1.E7.m1.2.2.1.1.1.1.2.1.2.cmml">⋁</mo><mrow id="A1.E7.m1.2.2.1.1.1.1.2.1.3" xref="A1.E7.m1.2.2.1.1.1.1.2.1.3.cmml"><mi id="A1.E7.m1.2.2.1.1.1.1.2.1.3.2" xref="A1.E7.m1.2.2.1.1.1.1.2.1.3.2.cmml">q</mi><mo id="A1.E7.m1.2.2.1.1.1.1.2.1.3.1" xref="A1.E7.m1.2.2.1.1.1.1.2.1.3.1.cmml">∈</mo><mi id="A1.E7.m1.2.2.1.1.1.1.2.1.3.3" xref="A1.E7.m1.2.2.1.1.1.1.2.1.3.3.cmml">Q</mi></mrow></munder><mrow id="A1.E7.m1.2.2.1.1.1.1.2.2" xref="A1.E7.m1.2.2.1.1.1.1.2.2.cmml"><msup id="A1.E7.m1.2.2.1.1.1.1.2.2.2" xref="A1.E7.m1.2.2.1.1.1.1.2.2.2.cmml"><mi id="A1.E7.m1.2.2.1.1.1.1.2.2.2.2" xref="A1.E7.m1.2.2.1.1.1.1.2.2.2.2.cmml">f</mi><mo id="A1.E7.m1.2.2.1.1.1.1.2.2.2.3" xref="A1.E7.m1.2.2.1.1.1.1.2.2.2.3.cmml">∗</mo></msup><mo lspace="0em" rspace="0em" id="A1.E7.m1.2.2.1.1.1.1.2.2.1" xref="A1.E7.m1.2.2.1.1.1.1.2.2.1.cmml">​</mo><mrow id="A1.E7.m1.2.2.1.1.1.1.2.2.3.2" xref="A1.E7.m1.2.2.1.1.1.1.2.2.cmml"><mo stretchy="false" id="A1.E7.m1.2.2.1.1.1.1.2.2.3.2.1" xref="A1.E7.m1.2.2.1.1.1.1.2.2.cmml">(</mo><mi id="A1.E7.m1.1.1" xref="A1.E7.m1.1.1.cmml">q</mi><mo stretchy="false" id="A1.E7.m1.2.2.1.1.1.1.2.2.3.2.2" xref="A1.E7.m1.2.2.1.1.1.1.2.2.cmml">)</mo></mrow></mrow></mrow><mo id="A1.E7.m1.2.2.1.1.1.1.1" xref="A1.E7.m1.2.2.1.1.1.1.1.cmml">=</mo><msup id="A1.E7.m1.2.2.1.1.1.1.3" xref="A1.E7.m1.2.2.1.1.1.1.3.cmml"><mi id="A1.E7.m1.2.2.1.1.1.1.3.2" xref="A1.E7.m1.2.2.1.1.1.1.3.2.cmml">a</mi><mo id="A1.E7.m1.2.2.1.1.1.1.3.3" xref="A1.E7.m1.2.2.1.1.1.1.3.3.cmml">∗</mo></msup></mrow><mo stretchy="false" id="A1.E7.m1.2.2.1.1.1.3" xref="A1.E7.m1.2.2.1.1.2.cmml">}</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E7.m1.2b"><apply id="A1.E7.m1.2.2.cmml" xref="A1.E7.m1.2.2"><eq id="A1.E7.m1.2.2.2.cmml" xref="A1.E7.m1.2.2.2"></eq><ci id="A1.E7.m1.2.2.3.cmml" xref="A1.E7.m1.2.2.3">MH</ci><apply id="A1.E7.m1.2.2.1.cmml" xref="A1.E7.m1.2.2.1"><times id="A1.E7.m1.2.2.1.2.cmml" xref="A1.E7.m1.2.2.1.2"></times><cn type="integer" id="A1.E7.m1.2.2.1.3.cmml" xref="A1.E7.m1.2.2.1.3">1</cn><set id="A1.E7.m1.2.2.1.1.2.cmml" xref="A1.E7.m1.2.2.1.1.1"><apply id="A1.E7.m1.2.2.1.1.1.1.cmml" xref="A1.E7.m1.2.2.1.1.1.1"><eq id="A1.E7.m1.2.2.1.1.1.1.1.cmml" xref="A1.E7.m1.2.2.1.1.1.1.1"></eq><apply id="A1.E7.m1.2.2.1.1.1.1.2.cmml" xref="A1.E7.m1.2.2.1.1.1.1.2"><apply id="A1.E7.m1.2.2.1.1.1.1.2.1.cmml" xref="A1.E7.m1.2.2.1.1.1.1.2.1"><csymbol cd="ambiguous" id="A1.E7.m1.2.2.1.1.1.1.2.1.1.cmml" xref="A1.E7.m1.2.2.1.1.1.1.2.1">subscript</csymbol><or id="A1.E7.m1.2.2.1.1.1.1.2.1.2.cmml" xref="A1.E7.m1.2.2.1.1.1.1.2.1.2"></or><apply id="A1.E7.m1.2.2.1.1.1.1.2.1.3.cmml" xref="A1.E7.m1.2.2.1.1.1.1.2.1.3"><in id="A1.E7.m1.2.2.1.1.1.1.2.1.3.1.cmml" xref="A1.E7.m1.2.2.1.1.1.1.2.1.3.1"></in><ci id="A1.E7.m1.2.2.1.1.1.1.2.1.3.2.cmml" xref="A1.E7.m1.2.2.1.1.1.1.2.1.3.2">𝑞</ci><ci id="A1.E7.m1.2.2.1.1.1.1.2.1.3.3.cmml" xref="A1.E7.m1.2.2.1.1.1.1.2.1.3.3">𝑄</ci></apply></apply><apply id="A1.E7.m1.2.2.1.1.1.1.2.2.cmml" xref="A1.E7.m1.2.2.1.1.1.1.2.2"><times id="A1.E7.m1.2.2.1.1.1.1.2.2.1.cmml" xref="A1.E7.m1.2.2.1.1.1.1.2.2.1"></times><apply id="A1.E7.m1.2.2.1.1.1.1.2.2.2.cmml" xref="A1.E7.m1.2.2.1.1.1.1.2.2.2"><csymbol cd="ambiguous" id="A1.E7.m1.2.2.1.1.1.1.2.2.2.1.cmml" xref="A1.E7.m1.2.2.1.1.1.1.2.2.2">superscript</csymbol><ci id="A1.E7.m1.2.2.1.1.1.1.2.2.2.2.cmml" xref="A1.E7.m1.2.2.1.1.1.1.2.2.2.2">𝑓</ci><times id="A1.E7.m1.2.2.1.1.1.1.2.2.2.3.cmml" xref="A1.E7.m1.2.2.1.1.1.1.2.2.2.3"></times></apply><ci id="A1.E7.m1.1.1.cmml" xref="A1.E7.m1.1.1">𝑞</ci></apply></apply><apply id="A1.E7.m1.2.2.1.1.1.1.3.cmml" xref="A1.E7.m1.2.2.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.E7.m1.2.2.1.1.1.1.3.1.cmml" xref="A1.E7.m1.2.2.1.1.1.1.3">superscript</csymbol><ci id="A1.E7.m1.2.2.1.1.1.1.3.2.cmml" xref="A1.E7.m1.2.2.1.1.1.1.3.2">𝑎</ci><times id="A1.E7.m1.2.2.1.1.1.1.3.3.cmml" xref="A1.E7.m1.2.2.1.1.1.1.3.3"></times></apply></apply></set></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E7.m1.2c">\mathrm{MH}=\mathbbm{1}\{\bigvee_{q\in Q}f^{*}(q)=a^{*}\}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(7)</span></td>
</tr></tbody>
</table>
<p id="A1.I3.i3.p1.2" class="ltx_p">where <math id="A1.I3.i3.p1.1.m1.1" class="ltx_Math" alttext="Q" display="inline"><semantics id="A1.I3.i3.p1.1.m1.1a"><mi id="A1.I3.i3.p1.1.m1.1.1" xref="A1.I3.i3.p1.1.m1.1.1.cmml">Q</mi><annotation-xml encoding="MathML-Content" id="A1.I3.i3.p1.1.m1.1b"><ci id="A1.I3.i3.p1.1.m1.1.1.cmml" xref="A1.I3.i3.p1.1.m1.1.1">𝑄</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.I3.i3.p1.1.m1.1c">Q</annotation></semantics></math> is a set of similar multi-hop questions with the same answer <math id="A1.I3.i3.p1.2.m2.1" class="ltx_Math" alttext="a^{*}" display="inline"><semantics id="A1.I3.i3.p1.2.m2.1a"><msup id="A1.I3.i3.p1.2.m2.1.1" xref="A1.I3.i3.p1.2.m2.1.1.cmml"><mi id="A1.I3.i3.p1.2.m2.1.1.2" xref="A1.I3.i3.p1.2.m2.1.1.2.cmml">a</mi><mo id="A1.I3.i3.p1.2.m2.1.1.3" xref="A1.I3.i3.p1.2.m2.1.1.3.cmml">∗</mo></msup><annotation-xml encoding="MathML-Content" id="A1.I3.i3.p1.2.m2.1b"><apply id="A1.I3.i3.p1.2.m2.1.1.cmml" xref="A1.I3.i3.p1.2.m2.1.1"><csymbol cd="ambiguous" id="A1.I3.i3.p1.2.m2.1.1.1.cmml" xref="A1.I3.i3.p1.2.m2.1.1">superscript</csymbol><ci id="A1.I3.i3.p1.2.m2.1.1.2.cmml" xref="A1.I3.i3.p1.2.m2.1.1.2">𝑎</ci><times id="A1.I3.i3.p1.2.m2.1.1.3.cmml" xref="A1.I3.i3.p1.2.m2.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I3.i3.p1.2.m2.1c">a^{*}</annotation></semantics></math>.</p>
</div>
</li>
</ul>
</div>
</section>
<section id="A1.SS1.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Multiple Edits</h5>

<div id="A1.SS1.SSS0.Px2.p1" class="ltx_para">
<p id="A1.SS1.SSS0.Px2.p1.1" class="ltx_p">In the multiple edits scenario, we test the model’s performance after applying multiple edits. For this, we use the setting and evaluation methods from <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib26" title="" class="ltx_ref">2024b</a>)</cite>. The settings consist of:</p>
</div>
<div id="A1.SS1.SSS0.Px2.p2" class="ltx_para">
<ul id="A1.I4" class="ltx_itemize">
<li id="A1.I4.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I4.i1.p1" class="ltx_para">
<p id="A1.I4.i1.p1.4" class="ltx_p"><span id="A1.I4.i1.p1.4.1" class="ltx_text ltx_font_bold">Reverse Conflict:</span> This setting introduces conflicts by editing facts with reverse relations. For example:

<br class="ltx_break"><span id="A1.I4.i1.p1.4.2" class="ltx_text ltx_font_bold">edit 1:</span> (<math id="A1.I4.i1.p1.1.m1.3" class="ltx_Math" alttext="s_{1},r_{1},o_{1}" display="inline"><semantics id="A1.I4.i1.p1.1.m1.3a"><mrow id="A1.I4.i1.p1.1.m1.3.3.3" xref="A1.I4.i1.p1.1.m1.3.3.4.cmml"><msub id="A1.I4.i1.p1.1.m1.1.1.1.1" xref="A1.I4.i1.p1.1.m1.1.1.1.1.cmml"><mi id="A1.I4.i1.p1.1.m1.1.1.1.1.2" xref="A1.I4.i1.p1.1.m1.1.1.1.1.2.cmml">s</mi><mn id="A1.I4.i1.p1.1.m1.1.1.1.1.3" xref="A1.I4.i1.p1.1.m1.1.1.1.1.3.cmml">1</mn></msub><mo id="A1.I4.i1.p1.1.m1.3.3.3.4" xref="A1.I4.i1.p1.1.m1.3.3.4.cmml">,</mo><msub id="A1.I4.i1.p1.1.m1.2.2.2.2" xref="A1.I4.i1.p1.1.m1.2.2.2.2.cmml"><mi id="A1.I4.i1.p1.1.m1.2.2.2.2.2" xref="A1.I4.i1.p1.1.m1.2.2.2.2.2.cmml">r</mi><mn id="A1.I4.i1.p1.1.m1.2.2.2.2.3" xref="A1.I4.i1.p1.1.m1.2.2.2.2.3.cmml">1</mn></msub><mo id="A1.I4.i1.p1.1.m1.3.3.3.5" xref="A1.I4.i1.p1.1.m1.3.3.4.cmml">,</mo><msub id="A1.I4.i1.p1.1.m1.3.3.3.3" xref="A1.I4.i1.p1.1.m1.3.3.3.3.cmml"><mi id="A1.I4.i1.p1.1.m1.3.3.3.3.2" xref="A1.I4.i1.p1.1.m1.3.3.3.3.2.cmml">o</mi><mn id="A1.I4.i1.p1.1.m1.3.3.3.3.3" xref="A1.I4.i1.p1.1.m1.3.3.3.3.3.cmml">1</mn></msub></mrow><annotation-xml encoding="MathML-Content" id="A1.I4.i1.p1.1.m1.3b"><list id="A1.I4.i1.p1.1.m1.3.3.4.cmml" xref="A1.I4.i1.p1.1.m1.3.3.3"><apply id="A1.I4.i1.p1.1.m1.1.1.1.1.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="A1.I4.i1.p1.1.m1.1.1.1.1.1.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1">subscript</csymbol><ci id="A1.I4.i1.p1.1.m1.1.1.1.1.2.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.2">𝑠</ci><cn type="integer" id="A1.I4.i1.p1.1.m1.1.1.1.1.3.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.3">1</cn></apply><apply id="A1.I4.i1.p1.1.m1.2.2.2.2.cmml" xref="A1.I4.i1.p1.1.m1.2.2.2.2"><csymbol cd="ambiguous" id="A1.I4.i1.p1.1.m1.2.2.2.2.1.cmml" xref="A1.I4.i1.p1.1.m1.2.2.2.2">subscript</csymbol><ci id="A1.I4.i1.p1.1.m1.2.2.2.2.2.cmml" xref="A1.I4.i1.p1.1.m1.2.2.2.2.2">𝑟</ci><cn type="integer" id="A1.I4.i1.p1.1.m1.2.2.2.2.3.cmml" xref="A1.I4.i1.p1.1.m1.2.2.2.2.3">1</cn></apply><apply id="A1.I4.i1.p1.1.m1.3.3.3.3.cmml" xref="A1.I4.i1.p1.1.m1.3.3.3.3"><csymbol cd="ambiguous" id="A1.I4.i1.p1.1.m1.3.3.3.3.1.cmml" xref="A1.I4.i1.p1.1.m1.3.3.3.3">subscript</csymbol><ci id="A1.I4.i1.p1.1.m1.3.3.3.3.2.cmml" xref="A1.I4.i1.p1.1.m1.3.3.3.3.2">𝑜</ci><cn type="integer" id="A1.I4.i1.p1.1.m1.3.3.3.3.3.cmml" xref="A1.I4.i1.p1.1.m1.3.3.3.3.3">1</cn></apply></list></annotation-xml><annotation encoding="application/x-tex" id="A1.I4.i1.p1.1.m1.3c">s_{1},r_{1},o_{1}</annotation></semantics></math>→<math id="A1.I4.i1.p1.2.m2.1" class="ltx_Math" alttext="o_{2}" display="inline"><semantics id="A1.I4.i1.p1.2.m2.1a"><msub id="A1.I4.i1.p1.2.m2.1.1" xref="A1.I4.i1.p1.2.m2.1.1.cmml"><mi id="A1.I4.i1.p1.2.m2.1.1.2" xref="A1.I4.i1.p1.2.m2.1.1.2.cmml">o</mi><mn id="A1.I4.i1.p1.2.m2.1.1.3" xref="A1.I4.i1.p1.2.m2.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A1.I4.i1.p1.2.m2.1b"><apply id="A1.I4.i1.p1.2.m2.1.1.cmml" xref="A1.I4.i1.p1.2.m2.1.1"><csymbol cd="ambiguous" id="A1.I4.i1.p1.2.m2.1.1.1.cmml" xref="A1.I4.i1.p1.2.m2.1.1">subscript</csymbol><ci id="A1.I4.i1.p1.2.m2.1.1.2.cmml" xref="A1.I4.i1.p1.2.m2.1.1.2">𝑜</ci><cn type="integer" id="A1.I4.i1.p1.2.m2.1.1.3.cmml" xref="A1.I4.i1.p1.2.m2.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I4.i1.p1.2.m2.1c">o_{2}</annotation></semantics></math>)

<br class="ltx_break"><span id="A1.I4.i1.p1.4.3" class="ltx_text ltx_font_italic">Hamlet was written by Shakespeare</span> → <span id="A1.I4.i1.p1.4.4" class="ltx_text ltx_font_italic">Agatha Christie</span>.

<br class="ltx_break"><span id="A1.I4.i1.p1.4.5" class="ltx_text ltx_font_bold">edit 2:</span> (<math id="A1.I4.i1.p1.3.m3.3" class="ltx_Math" alttext="o_{2},r_{2},s_{1}" display="inline"><semantics id="A1.I4.i1.p1.3.m3.3a"><mrow id="A1.I4.i1.p1.3.m3.3.3.3" xref="A1.I4.i1.p1.3.m3.3.3.4.cmml"><msub id="A1.I4.i1.p1.3.m3.1.1.1.1" xref="A1.I4.i1.p1.3.m3.1.1.1.1.cmml"><mi id="A1.I4.i1.p1.3.m3.1.1.1.1.2" xref="A1.I4.i1.p1.3.m3.1.1.1.1.2.cmml">o</mi><mn id="A1.I4.i1.p1.3.m3.1.1.1.1.3" xref="A1.I4.i1.p1.3.m3.1.1.1.1.3.cmml">2</mn></msub><mo id="A1.I4.i1.p1.3.m3.3.3.3.4" xref="A1.I4.i1.p1.3.m3.3.3.4.cmml">,</mo><msub id="A1.I4.i1.p1.3.m3.2.2.2.2" xref="A1.I4.i1.p1.3.m3.2.2.2.2.cmml"><mi id="A1.I4.i1.p1.3.m3.2.2.2.2.2" xref="A1.I4.i1.p1.3.m3.2.2.2.2.2.cmml">r</mi><mn id="A1.I4.i1.p1.3.m3.2.2.2.2.3" xref="A1.I4.i1.p1.3.m3.2.2.2.2.3.cmml">2</mn></msub><mo id="A1.I4.i1.p1.3.m3.3.3.3.5" xref="A1.I4.i1.p1.3.m3.3.3.4.cmml">,</mo><msub id="A1.I4.i1.p1.3.m3.3.3.3.3" xref="A1.I4.i1.p1.3.m3.3.3.3.3.cmml"><mi id="A1.I4.i1.p1.3.m3.3.3.3.3.2" xref="A1.I4.i1.p1.3.m3.3.3.3.3.2.cmml">s</mi><mn id="A1.I4.i1.p1.3.m3.3.3.3.3.3" xref="A1.I4.i1.p1.3.m3.3.3.3.3.3.cmml">1</mn></msub></mrow><annotation-xml encoding="MathML-Content" id="A1.I4.i1.p1.3.m3.3b"><list id="A1.I4.i1.p1.3.m3.3.3.4.cmml" xref="A1.I4.i1.p1.3.m3.3.3.3"><apply id="A1.I4.i1.p1.3.m3.1.1.1.1.cmml" xref="A1.I4.i1.p1.3.m3.1.1.1.1"><csymbol cd="ambiguous" id="A1.I4.i1.p1.3.m3.1.1.1.1.1.cmml" xref="A1.I4.i1.p1.3.m3.1.1.1.1">subscript</csymbol><ci id="A1.I4.i1.p1.3.m3.1.1.1.1.2.cmml" xref="A1.I4.i1.p1.3.m3.1.1.1.1.2">𝑜</ci><cn type="integer" id="A1.I4.i1.p1.3.m3.1.1.1.1.3.cmml" xref="A1.I4.i1.p1.3.m3.1.1.1.1.3">2</cn></apply><apply id="A1.I4.i1.p1.3.m3.2.2.2.2.cmml" xref="A1.I4.i1.p1.3.m3.2.2.2.2"><csymbol cd="ambiguous" id="A1.I4.i1.p1.3.m3.2.2.2.2.1.cmml" xref="A1.I4.i1.p1.3.m3.2.2.2.2">subscript</csymbol><ci id="A1.I4.i1.p1.3.m3.2.2.2.2.2.cmml" xref="A1.I4.i1.p1.3.m3.2.2.2.2.2">𝑟</ci><cn type="integer" id="A1.I4.i1.p1.3.m3.2.2.2.2.3.cmml" xref="A1.I4.i1.p1.3.m3.2.2.2.2.3">2</cn></apply><apply id="A1.I4.i1.p1.3.m3.3.3.3.3.cmml" xref="A1.I4.i1.p1.3.m3.3.3.3.3"><csymbol cd="ambiguous" id="A1.I4.i1.p1.3.m3.3.3.3.3.1.cmml" xref="A1.I4.i1.p1.3.m3.3.3.3.3">subscript</csymbol><ci id="A1.I4.i1.p1.3.m3.3.3.3.3.2.cmml" xref="A1.I4.i1.p1.3.m3.3.3.3.3.2">𝑠</ci><cn type="integer" id="A1.I4.i1.p1.3.m3.3.3.3.3.3.cmml" xref="A1.I4.i1.p1.3.m3.3.3.3.3.3">1</cn></apply></list></annotation-xml><annotation encoding="application/x-tex" id="A1.I4.i1.p1.3.m3.3c">o_{2},r_{2},s_{1}</annotation></semantics></math>→<math id="A1.I4.i1.p1.4.m4.1" class="ltx_Math" alttext="s_{2}" display="inline"><semantics id="A1.I4.i1.p1.4.m4.1a"><msub id="A1.I4.i1.p1.4.m4.1.1" xref="A1.I4.i1.p1.4.m4.1.1.cmml"><mi id="A1.I4.i1.p1.4.m4.1.1.2" xref="A1.I4.i1.p1.4.m4.1.1.2.cmml">s</mi><mn id="A1.I4.i1.p1.4.m4.1.1.3" xref="A1.I4.i1.p1.4.m4.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A1.I4.i1.p1.4.m4.1b"><apply id="A1.I4.i1.p1.4.m4.1.1.cmml" xref="A1.I4.i1.p1.4.m4.1.1"><csymbol cd="ambiguous" id="A1.I4.i1.p1.4.m4.1.1.1.cmml" xref="A1.I4.i1.p1.4.m4.1.1">subscript</csymbol><ci id="A1.I4.i1.p1.4.m4.1.1.2.cmml" xref="A1.I4.i1.p1.4.m4.1.1.2">𝑠</ci><cn type="integer" id="A1.I4.i1.p1.4.m4.1.1.3.cmml" xref="A1.I4.i1.p1.4.m4.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I4.i1.p1.4.m4.1c">s_{2}</annotation></semantics></math>)

<br class="ltx_break"><span id="A1.I4.i1.p1.4.6" class="ltx_text ltx_font_italic">The notable work of Agatha Christie is Hamlet</span> → <span id="A1.I4.i1.p1.4.7" class="ltx_text ltx_font_italic">Odyssey</span>

<br class="ltx_break">the updated knowledge then could be represented as:</p>
<table id="A1.Ex1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.Ex1.m1.6" class="ltx_Math" alttext="\left\{\begin{array}[]{l}k_{0}=(s_{1},r_{1},o_{2})\\
k_{n}=(s_{2},r_{1},o_{2})\end{array}\right." display="block"><semantics id="A1.Ex1.m1.6a"><mrow id="A1.Ex1.m1.6.7.2" xref="A1.Ex1.m1.6.7.1.cmml"><mo id="A1.Ex1.m1.6.7.2.1" xref="A1.Ex1.m1.6.7.1.1.cmml">{</mo><mtable displaystyle="true" rowspacing="0pt" id="A1.Ex1.m1.6.6" xref="A1.Ex1.m1.6.6.cmml"><mtr id="A1.Ex1.m1.6.6a" xref="A1.Ex1.m1.6.6.cmml"><mtd class="ltx_align_left" columnalign="left" id="A1.Ex1.m1.6.6b" xref="A1.Ex1.m1.6.6.cmml"><mrow id="A1.Ex1.m1.3.3.3.3.3" xref="A1.Ex1.m1.3.3.3.3.3.cmml"><msub id="A1.Ex1.m1.3.3.3.3.3.5" xref="A1.Ex1.m1.3.3.3.3.3.5.cmml"><mi id="A1.Ex1.m1.3.3.3.3.3.5.2" xref="A1.Ex1.m1.3.3.3.3.3.5.2.cmml">k</mi><mn id="A1.Ex1.m1.3.3.3.3.3.5.3" xref="A1.Ex1.m1.3.3.3.3.3.5.3.cmml">0</mn></msub><mo id="A1.Ex1.m1.3.3.3.3.3.4" xref="A1.Ex1.m1.3.3.3.3.3.4.cmml">=</mo><mrow id="A1.Ex1.m1.3.3.3.3.3.3.3" xref="A1.Ex1.m1.3.3.3.3.3.3.4.cmml"><mo stretchy="false" id="A1.Ex1.m1.3.3.3.3.3.3.3.4" xref="A1.Ex1.m1.3.3.3.3.3.3.4.cmml">(</mo><msub id="A1.Ex1.m1.1.1.1.1.1.1.1.1" xref="A1.Ex1.m1.1.1.1.1.1.1.1.1.cmml"><mi id="A1.Ex1.m1.1.1.1.1.1.1.1.1.2" xref="A1.Ex1.m1.1.1.1.1.1.1.1.1.2.cmml">s</mi><mn id="A1.Ex1.m1.1.1.1.1.1.1.1.1.3" xref="A1.Ex1.m1.1.1.1.1.1.1.1.1.3.cmml">1</mn></msub><mo id="A1.Ex1.m1.3.3.3.3.3.3.3.5" xref="A1.Ex1.m1.3.3.3.3.3.3.4.cmml">,</mo><msub id="A1.Ex1.m1.2.2.2.2.2.2.2.2" xref="A1.Ex1.m1.2.2.2.2.2.2.2.2.cmml"><mi id="A1.Ex1.m1.2.2.2.2.2.2.2.2.2" xref="A1.Ex1.m1.2.2.2.2.2.2.2.2.2.cmml">r</mi><mn id="A1.Ex1.m1.2.2.2.2.2.2.2.2.3" xref="A1.Ex1.m1.2.2.2.2.2.2.2.2.3.cmml">1</mn></msub><mo id="A1.Ex1.m1.3.3.3.3.3.3.3.6" xref="A1.Ex1.m1.3.3.3.3.3.3.4.cmml">,</mo><msub id="A1.Ex1.m1.3.3.3.3.3.3.3.3" xref="A1.Ex1.m1.3.3.3.3.3.3.3.3.cmml"><mi id="A1.Ex1.m1.3.3.3.3.3.3.3.3.2" xref="A1.Ex1.m1.3.3.3.3.3.3.3.3.2.cmml">o</mi><mn id="A1.Ex1.m1.3.3.3.3.3.3.3.3.3" xref="A1.Ex1.m1.3.3.3.3.3.3.3.3.3.cmml">2</mn></msub><mo stretchy="false" id="A1.Ex1.m1.3.3.3.3.3.3.3.7" xref="A1.Ex1.m1.3.3.3.3.3.3.4.cmml">)</mo></mrow></mrow></mtd></mtr><mtr id="A1.Ex1.m1.6.6c" xref="A1.Ex1.m1.6.6.cmml"><mtd class="ltx_align_left" columnalign="left" id="A1.Ex1.m1.6.6d" xref="A1.Ex1.m1.6.6.cmml"><mrow id="A1.Ex1.m1.6.6.6.3.3" xref="A1.Ex1.m1.6.6.6.3.3.cmml"><msub id="A1.Ex1.m1.6.6.6.3.3.5" xref="A1.Ex1.m1.6.6.6.3.3.5.cmml"><mi id="A1.Ex1.m1.6.6.6.3.3.5.2" xref="A1.Ex1.m1.6.6.6.3.3.5.2.cmml">k</mi><mi id="A1.Ex1.m1.6.6.6.3.3.5.3" xref="A1.Ex1.m1.6.6.6.3.3.5.3.cmml">n</mi></msub><mo id="A1.Ex1.m1.6.6.6.3.3.4" xref="A1.Ex1.m1.6.6.6.3.3.4.cmml">=</mo><mrow id="A1.Ex1.m1.6.6.6.3.3.3.3" xref="A1.Ex1.m1.6.6.6.3.3.3.4.cmml"><mo stretchy="false" id="A1.Ex1.m1.6.6.6.3.3.3.3.4" xref="A1.Ex1.m1.6.6.6.3.3.3.4.cmml">(</mo><msub id="A1.Ex1.m1.4.4.4.1.1.1.1.1" xref="A1.Ex1.m1.4.4.4.1.1.1.1.1.cmml"><mi id="A1.Ex1.m1.4.4.4.1.1.1.1.1.2" xref="A1.Ex1.m1.4.4.4.1.1.1.1.1.2.cmml">s</mi><mn id="A1.Ex1.m1.4.4.4.1.1.1.1.1.3" xref="A1.Ex1.m1.4.4.4.1.1.1.1.1.3.cmml">2</mn></msub><mo id="A1.Ex1.m1.6.6.6.3.3.3.3.5" xref="A1.Ex1.m1.6.6.6.3.3.3.4.cmml">,</mo><msub id="A1.Ex1.m1.5.5.5.2.2.2.2.2" xref="A1.Ex1.m1.5.5.5.2.2.2.2.2.cmml"><mi id="A1.Ex1.m1.5.5.5.2.2.2.2.2.2" xref="A1.Ex1.m1.5.5.5.2.2.2.2.2.2.cmml">r</mi><mn id="A1.Ex1.m1.5.5.5.2.2.2.2.2.3" xref="A1.Ex1.m1.5.5.5.2.2.2.2.2.3.cmml">1</mn></msub><mo id="A1.Ex1.m1.6.6.6.3.3.3.3.6" xref="A1.Ex1.m1.6.6.6.3.3.3.4.cmml">,</mo><msub id="A1.Ex1.m1.6.6.6.3.3.3.3.3" xref="A1.Ex1.m1.6.6.6.3.3.3.3.3.cmml"><mi id="A1.Ex1.m1.6.6.6.3.3.3.3.3.2" xref="A1.Ex1.m1.6.6.6.3.3.3.3.3.2.cmml">o</mi><mn id="A1.Ex1.m1.6.6.6.3.3.3.3.3.3" xref="A1.Ex1.m1.6.6.6.3.3.3.3.3.3.cmml">2</mn></msub><mo stretchy="false" id="A1.Ex1.m1.6.6.6.3.3.3.3.7" xref="A1.Ex1.m1.6.6.6.3.3.3.4.cmml">)</mo></mrow></mrow></mtd></mtr></mtable><mi id="A1.Ex1.m1.6.7.2.2" xref="A1.Ex1.m1.6.7.1.1.cmml"></mi></mrow><annotation-xml encoding="MathML-Content" id="A1.Ex1.m1.6b"><apply id="A1.Ex1.m1.6.7.1.cmml" xref="A1.Ex1.m1.6.7.2"><csymbol cd="latexml" id="A1.Ex1.m1.6.7.1.1.cmml" xref="A1.Ex1.m1.6.7.2.1">cases</csymbol><matrix id="A1.Ex1.m1.6.6.cmml" xref="A1.Ex1.m1.6.6"><matrixrow id="A1.Ex1.m1.6.6a.cmml" xref="A1.Ex1.m1.6.6"><apply id="A1.Ex1.m1.3.3.3.3.3.cmml" xref="A1.Ex1.m1.3.3.3.3.3"><eq id="A1.Ex1.m1.3.3.3.3.3.4.cmml" xref="A1.Ex1.m1.3.3.3.3.3.4"></eq><apply id="A1.Ex1.m1.3.3.3.3.3.5.cmml" xref="A1.Ex1.m1.3.3.3.3.3.5"><csymbol cd="ambiguous" id="A1.Ex1.m1.3.3.3.3.3.5.1.cmml" xref="A1.Ex1.m1.3.3.3.3.3.5">subscript</csymbol><ci id="A1.Ex1.m1.3.3.3.3.3.5.2.cmml" xref="A1.Ex1.m1.3.3.3.3.3.5.2">𝑘</ci><cn type="integer" id="A1.Ex1.m1.3.3.3.3.3.5.3.cmml" xref="A1.Ex1.m1.3.3.3.3.3.5.3">0</cn></apply><vector id="A1.Ex1.m1.3.3.3.3.3.3.4.cmml" xref="A1.Ex1.m1.3.3.3.3.3.3.3"><apply id="A1.Ex1.m1.1.1.1.1.1.1.1.1.cmml" xref="A1.Ex1.m1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.Ex1.m1.1.1.1.1.1.1.1.1.1.cmml" xref="A1.Ex1.m1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="A1.Ex1.m1.1.1.1.1.1.1.1.1.2.cmml" xref="A1.Ex1.m1.1.1.1.1.1.1.1.1.2">𝑠</ci><cn type="integer" id="A1.Ex1.m1.1.1.1.1.1.1.1.1.3.cmml" xref="A1.Ex1.m1.1.1.1.1.1.1.1.1.3">1</cn></apply><apply id="A1.Ex1.m1.2.2.2.2.2.2.2.2.cmml" xref="A1.Ex1.m1.2.2.2.2.2.2.2.2"><csymbol cd="ambiguous" id="A1.Ex1.m1.2.2.2.2.2.2.2.2.1.cmml" xref="A1.Ex1.m1.2.2.2.2.2.2.2.2">subscript</csymbol><ci id="A1.Ex1.m1.2.2.2.2.2.2.2.2.2.cmml" xref="A1.Ex1.m1.2.2.2.2.2.2.2.2.2">𝑟</ci><cn type="integer" id="A1.Ex1.m1.2.2.2.2.2.2.2.2.3.cmml" xref="A1.Ex1.m1.2.2.2.2.2.2.2.2.3">1</cn></apply><apply id="A1.Ex1.m1.3.3.3.3.3.3.3.3.cmml" xref="A1.Ex1.m1.3.3.3.3.3.3.3.3"><csymbol cd="ambiguous" id="A1.Ex1.m1.3.3.3.3.3.3.3.3.1.cmml" xref="A1.Ex1.m1.3.3.3.3.3.3.3.3">subscript</csymbol><ci id="A1.Ex1.m1.3.3.3.3.3.3.3.3.2.cmml" xref="A1.Ex1.m1.3.3.3.3.3.3.3.3.2">𝑜</ci><cn type="integer" id="A1.Ex1.m1.3.3.3.3.3.3.3.3.3.cmml" xref="A1.Ex1.m1.3.3.3.3.3.3.3.3.3">2</cn></apply></vector></apply></matrixrow><matrixrow id="A1.Ex1.m1.6.6b.cmml" xref="A1.Ex1.m1.6.6"><apply id="A1.Ex1.m1.6.6.6.3.3.cmml" xref="A1.Ex1.m1.6.6.6.3.3"><eq id="A1.Ex1.m1.6.6.6.3.3.4.cmml" xref="A1.Ex1.m1.6.6.6.3.3.4"></eq><apply id="A1.Ex1.m1.6.6.6.3.3.5.cmml" xref="A1.Ex1.m1.6.6.6.3.3.5"><csymbol cd="ambiguous" id="A1.Ex1.m1.6.6.6.3.3.5.1.cmml" xref="A1.Ex1.m1.6.6.6.3.3.5">subscript</csymbol><ci id="A1.Ex1.m1.6.6.6.3.3.5.2.cmml" xref="A1.Ex1.m1.6.6.6.3.3.5.2">𝑘</ci><ci id="A1.Ex1.m1.6.6.6.3.3.5.3.cmml" xref="A1.Ex1.m1.6.6.6.3.3.5.3">𝑛</ci></apply><vector id="A1.Ex1.m1.6.6.6.3.3.3.4.cmml" xref="A1.Ex1.m1.6.6.6.3.3.3.3"><apply id="A1.Ex1.m1.4.4.4.1.1.1.1.1.cmml" xref="A1.Ex1.m1.4.4.4.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.Ex1.m1.4.4.4.1.1.1.1.1.1.cmml" xref="A1.Ex1.m1.4.4.4.1.1.1.1.1">subscript</csymbol><ci id="A1.Ex1.m1.4.4.4.1.1.1.1.1.2.cmml" xref="A1.Ex1.m1.4.4.4.1.1.1.1.1.2">𝑠</ci><cn type="integer" id="A1.Ex1.m1.4.4.4.1.1.1.1.1.3.cmml" xref="A1.Ex1.m1.4.4.4.1.1.1.1.1.3">2</cn></apply><apply id="A1.Ex1.m1.5.5.5.2.2.2.2.2.cmml" xref="A1.Ex1.m1.5.5.5.2.2.2.2.2"><csymbol cd="ambiguous" id="A1.Ex1.m1.5.5.5.2.2.2.2.2.1.cmml" xref="A1.Ex1.m1.5.5.5.2.2.2.2.2">subscript</csymbol><ci id="A1.Ex1.m1.5.5.5.2.2.2.2.2.2.cmml" xref="A1.Ex1.m1.5.5.5.2.2.2.2.2.2">𝑟</ci><cn type="integer" id="A1.Ex1.m1.5.5.5.2.2.2.2.2.3.cmml" xref="A1.Ex1.m1.5.5.5.2.2.2.2.2.3">1</cn></apply><apply id="A1.Ex1.m1.6.6.6.3.3.3.3.3.cmml" xref="A1.Ex1.m1.6.6.6.3.3.3.3.3"><csymbol cd="ambiguous" id="A1.Ex1.m1.6.6.6.3.3.3.3.3.1.cmml" xref="A1.Ex1.m1.6.6.6.3.3.3.3.3">subscript</csymbol><ci id="A1.Ex1.m1.6.6.6.3.3.3.3.3.2.cmml" xref="A1.Ex1.m1.6.6.6.3.3.3.3.3.2">𝑜</ci><cn type="integer" id="A1.Ex1.m1.6.6.6.3.3.3.3.3.3.cmml" xref="A1.Ex1.m1.6.6.6.3.3.3.3.3.3">2</cn></apply></vector></apply></matrixrow></matrix></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.Ex1.m1.6c">\left\{\begin{array}[]{l}k_{0}=(s_{1},r_{1},o_{2})\\
k_{n}=(s_{2},r_{1},o_{2})\end{array}\right.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
</li>
<li id="A1.I4.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I4.i2.p1" class="ltx_para">
<p id="A1.I4.i2.p1.6" class="ltx_p"><span id="A1.I4.i2.p1.6.1" class="ltx_text ltx_font_bold">Composite Conflict:</span> This explores more complex situations where the edits are associated with a fact that is not influenced by the editing (<span id="A1.I4.i2.p1.6.2" class="ltx_text ltx_font_bold">tied fact</span>). For example:

<br class="ltx_break"><span id="A1.I4.i2.p1.6.3" class="ltx_text ltx_font_bold">edit 1:</span> (<math id="A1.I4.i2.p1.1.m1.3" class="ltx_Math" alttext="s_{1},r_{1},o_{1}" display="inline"><semantics id="A1.I4.i2.p1.1.m1.3a"><mrow id="A1.I4.i2.p1.1.m1.3.3.3" xref="A1.I4.i2.p1.1.m1.3.3.4.cmml"><msub id="A1.I4.i2.p1.1.m1.1.1.1.1" xref="A1.I4.i2.p1.1.m1.1.1.1.1.cmml"><mi id="A1.I4.i2.p1.1.m1.1.1.1.1.2" xref="A1.I4.i2.p1.1.m1.1.1.1.1.2.cmml">s</mi><mn id="A1.I4.i2.p1.1.m1.1.1.1.1.3" xref="A1.I4.i2.p1.1.m1.1.1.1.1.3.cmml">1</mn></msub><mo id="A1.I4.i2.p1.1.m1.3.3.3.4" xref="A1.I4.i2.p1.1.m1.3.3.4.cmml">,</mo><msub id="A1.I4.i2.p1.1.m1.2.2.2.2" xref="A1.I4.i2.p1.1.m1.2.2.2.2.cmml"><mi id="A1.I4.i2.p1.1.m1.2.2.2.2.2" xref="A1.I4.i2.p1.1.m1.2.2.2.2.2.cmml">r</mi><mn id="A1.I4.i2.p1.1.m1.2.2.2.2.3" xref="A1.I4.i2.p1.1.m1.2.2.2.2.3.cmml">1</mn></msub><mo id="A1.I4.i2.p1.1.m1.3.3.3.5" xref="A1.I4.i2.p1.1.m1.3.3.4.cmml">,</mo><msub id="A1.I4.i2.p1.1.m1.3.3.3.3" xref="A1.I4.i2.p1.1.m1.3.3.3.3.cmml"><mi id="A1.I4.i2.p1.1.m1.3.3.3.3.2" xref="A1.I4.i2.p1.1.m1.3.3.3.3.2.cmml">o</mi><mn id="A1.I4.i2.p1.1.m1.3.3.3.3.3" xref="A1.I4.i2.p1.1.m1.3.3.3.3.3.cmml">1</mn></msub></mrow><annotation-xml encoding="MathML-Content" id="A1.I4.i2.p1.1.m1.3b"><list id="A1.I4.i2.p1.1.m1.3.3.4.cmml" xref="A1.I4.i2.p1.1.m1.3.3.3"><apply id="A1.I4.i2.p1.1.m1.1.1.1.1.cmml" xref="A1.I4.i2.p1.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="A1.I4.i2.p1.1.m1.1.1.1.1.1.cmml" xref="A1.I4.i2.p1.1.m1.1.1.1.1">subscript</csymbol><ci id="A1.I4.i2.p1.1.m1.1.1.1.1.2.cmml" xref="A1.I4.i2.p1.1.m1.1.1.1.1.2">𝑠</ci><cn type="integer" id="A1.I4.i2.p1.1.m1.1.1.1.1.3.cmml" xref="A1.I4.i2.p1.1.m1.1.1.1.1.3">1</cn></apply><apply id="A1.I4.i2.p1.1.m1.2.2.2.2.cmml" xref="A1.I4.i2.p1.1.m1.2.2.2.2"><csymbol cd="ambiguous" id="A1.I4.i2.p1.1.m1.2.2.2.2.1.cmml" xref="A1.I4.i2.p1.1.m1.2.2.2.2">subscript</csymbol><ci id="A1.I4.i2.p1.1.m1.2.2.2.2.2.cmml" xref="A1.I4.i2.p1.1.m1.2.2.2.2.2">𝑟</ci><cn type="integer" id="A1.I4.i2.p1.1.m1.2.2.2.2.3.cmml" xref="A1.I4.i2.p1.1.m1.2.2.2.2.3">1</cn></apply><apply id="A1.I4.i2.p1.1.m1.3.3.3.3.cmml" xref="A1.I4.i2.p1.1.m1.3.3.3.3"><csymbol cd="ambiguous" id="A1.I4.i2.p1.1.m1.3.3.3.3.1.cmml" xref="A1.I4.i2.p1.1.m1.3.3.3.3">subscript</csymbol><ci id="A1.I4.i2.p1.1.m1.3.3.3.3.2.cmml" xref="A1.I4.i2.p1.1.m1.3.3.3.3.2">𝑜</ci><cn type="integer" id="A1.I4.i2.p1.1.m1.3.3.3.3.3.cmml" xref="A1.I4.i2.p1.1.m1.3.3.3.3.3">1</cn></apply></list></annotation-xml><annotation encoding="application/x-tex" id="A1.I4.i2.p1.1.m1.3c">s_{1},r_{1},o_{1}</annotation></semantics></math>→<math id="A1.I4.i2.p1.2.m2.1" class="ltx_Math" alttext="o_{2}" display="inline"><semantics id="A1.I4.i2.p1.2.m2.1a"><msub id="A1.I4.i2.p1.2.m2.1.1" xref="A1.I4.i2.p1.2.m2.1.1.cmml"><mi id="A1.I4.i2.p1.2.m2.1.1.2" xref="A1.I4.i2.p1.2.m2.1.1.2.cmml">o</mi><mn id="A1.I4.i2.p1.2.m2.1.1.3" xref="A1.I4.i2.p1.2.m2.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A1.I4.i2.p1.2.m2.1b"><apply id="A1.I4.i2.p1.2.m2.1.1.cmml" xref="A1.I4.i2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="A1.I4.i2.p1.2.m2.1.1.1.cmml" xref="A1.I4.i2.p1.2.m2.1.1">subscript</csymbol><ci id="A1.I4.i2.p1.2.m2.1.1.2.cmml" xref="A1.I4.i2.p1.2.m2.1.1.2">𝑜</ci><cn type="integer" id="A1.I4.i2.p1.2.m2.1.1.3.cmml" xref="A1.I4.i2.p1.2.m2.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I4.i2.p1.2.m2.1c">o_{2}</annotation></semantics></math>)

<br class="ltx_break"><span id="A1.I4.i2.p1.6.4" class="ltx_text ltx_font_italic">Hamlet was written in English</span> → <span id="A1.I4.i2.p1.6.5" class="ltx_text ltx_font_italic">French</span>

<br class="ltx_break"><span id="A1.I4.i2.p1.6.6" class="ltx_text ltx_font_bold">edit 2:</span> (<math id="A1.I4.i2.p1.3.m3.3" class="ltx_Math" alttext="s_{2},r_{2},o_{2}" display="inline"><semantics id="A1.I4.i2.p1.3.m3.3a"><mrow id="A1.I4.i2.p1.3.m3.3.3.3" xref="A1.I4.i2.p1.3.m3.3.3.4.cmml"><msub id="A1.I4.i2.p1.3.m3.1.1.1.1" xref="A1.I4.i2.p1.3.m3.1.1.1.1.cmml"><mi id="A1.I4.i2.p1.3.m3.1.1.1.1.2" xref="A1.I4.i2.p1.3.m3.1.1.1.1.2.cmml">s</mi><mn id="A1.I4.i2.p1.3.m3.1.1.1.1.3" xref="A1.I4.i2.p1.3.m3.1.1.1.1.3.cmml">2</mn></msub><mo id="A1.I4.i2.p1.3.m3.3.3.3.4" xref="A1.I4.i2.p1.3.m3.3.3.4.cmml">,</mo><msub id="A1.I4.i2.p1.3.m3.2.2.2.2" xref="A1.I4.i2.p1.3.m3.2.2.2.2.cmml"><mi id="A1.I4.i2.p1.3.m3.2.2.2.2.2" xref="A1.I4.i2.p1.3.m3.2.2.2.2.2.cmml">r</mi><mn id="A1.I4.i2.p1.3.m3.2.2.2.2.3" xref="A1.I4.i2.p1.3.m3.2.2.2.2.3.cmml">2</mn></msub><mo id="A1.I4.i2.p1.3.m3.3.3.3.5" xref="A1.I4.i2.p1.3.m3.3.3.4.cmml">,</mo><msub id="A1.I4.i2.p1.3.m3.3.3.3.3" xref="A1.I4.i2.p1.3.m3.3.3.3.3.cmml"><mi id="A1.I4.i2.p1.3.m3.3.3.3.3.2" xref="A1.I4.i2.p1.3.m3.3.3.3.3.2.cmml">o</mi><mn id="A1.I4.i2.p1.3.m3.3.3.3.3.3" xref="A1.I4.i2.p1.3.m3.3.3.3.3.3.cmml">2</mn></msub></mrow><annotation-xml encoding="MathML-Content" id="A1.I4.i2.p1.3.m3.3b"><list id="A1.I4.i2.p1.3.m3.3.3.4.cmml" xref="A1.I4.i2.p1.3.m3.3.3.3"><apply id="A1.I4.i2.p1.3.m3.1.1.1.1.cmml" xref="A1.I4.i2.p1.3.m3.1.1.1.1"><csymbol cd="ambiguous" id="A1.I4.i2.p1.3.m3.1.1.1.1.1.cmml" xref="A1.I4.i2.p1.3.m3.1.1.1.1">subscript</csymbol><ci id="A1.I4.i2.p1.3.m3.1.1.1.1.2.cmml" xref="A1.I4.i2.p1.3.m3.1.1.1.1.2">𝑠</ci><cn type="integer" id="A1.I4.i2.p1.3.m3.1.1.1.1.3.cmml" xref="A1.I4.i2.p1.3.m3.1.1.1.1.3">2</cn></apply><apply id="A1.I4.i2.p1.3.m3.2.2.2.2.cmml" xref="A1.I4.i2.p1.3.m3.2.2.2.2"><csymbol cd="ambiguous" id="A1.I4.i2.p1.3.m3.2.2.2.2.1.cmml" xref="A1.I4.i2.p1.3.m3.2.2.2.2">subscript</csymbol><ci id="A1.I4.i2.p1.3.m3.2.2.2.2.2.cmml" xref="A1.I4.i2.p1.3.m3.2.2.2.2.2">𝑟</ci><cn type="integer" id="A1.I4.i2.p1.3.m3.2.2.2.2.3.cmml" xref="A1.I4.i2.p1.3.m3.2.2.2.2.3">2</cn></apply><apply id="A1.I4.i2.p1.3.m3.3.3.3.3.cmml" xref="A1.I4.i2.p1.3.m3.3.3.3.3"><csymbol cd="ambiguous" id="A1.I4.i2.p1.3.m3.3.3.3.3.1.cmml" xref="A1.I4.i2.p1.3.m3.3.3.3.3">subscript</csymbol><ci id="A1.I4.i2.p1.3.m3.3.3.3.3.2.cmml" xref="A1.I4.i2.p1.3.m3.3.3.3.3.2">𝑜</ci><cn type="integer" id="A1.I4.i2.p1.3.m3.3.3.3.3.3.cmml" xref="A1.I4.i2.p1.3.m3.3.3.3.3.3">2</cn></apply></list></annotation-xml><annotation encoding="application/x-tex" id="A1.I4.i2.p1.3.m3.3c">s_{2},r_{2},o_{2}</annotation></semantics></math>→<math id="A1.I4.i2.p1.4.m4.1" class="ltx_Math" alttext="o_{3}" display="inline"><semantics id="A1.I4.i2.p1.4.m4.1a"><msub id="A1.I4.i2.p1.4.m4.1.1" xref="A1.I4.i2.p1.4.m4.1.1.cmml"><mi id="A1.I4.i2.p1.4.m4.1.1.2" xref="A1.I4.i2.p1.4.m4.1.1.2.cmml">o</mi><mn id="A1.I4.i2.p1.4.m4.1.1.3" xref="A1.I4.i2.p1.4.m4.1.1.3.cmml">3</mn></msub><annotation-xml encoding="MathML-Content" id="A1.I4.i2.p1.4.m4.1b"><apply id="A1.I4.i2.p1.4.m4.1.1.cmml" xref="A1.I4.i2.p1.4.m4.1.1"><csymbol cd="ambiguous" id="A1.I4.i2.p1.4.m4.1.1.1.cmml" xref="A1.I4.i2.p1.4.m4.1.1">subscript</csymbol><ci id="A1.I4.i2.p1.4.m4.1.1.2.cmml" xref="A1.I4.i2.p1.4.m4.1.1.2">𝑜</ci><cn type="integer" id="A1.I4.i2.p1.4.m4.1.1.3.cmml" xref="A1.I4.i2.p1.4.m4.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I4.i2.p1.4.m4.1c">o_{3}</annotation></semantics></math>)

<br class="ltx_break"><span id="A1.I4.i2.p1.6.7" class="ltx_text ltx_font_italic">Shakespeare wrote in French</span> → <span id="A1.I4.i2.p1.6.8" class="ltx_text ltx_font_italic">German</span>

<br class="ltx_break"><span id="A1.I4.i2.p1.6.9" class="ltx_text ltx_font_bold">tied fact:</span> (<math id="A1.I4.i2.p1.5.m5.3" class="ltx_Math" alttext="s_{1},r,s_{2}" display="inline"><semantics id="A1.I4.i2.p1.5.m5.3a"><mrow id="A1.I4.i2.p1.5.m5.3.3.2" xref="A1.I4.i2.p1.5.m5.3.3.3.cmml"><msub id="A1.I4.i2.p1.5.m5.2.2.1.1" xref="A1.I4.i2.p1.5.m5.2.2.1.1.cmml"><mi id="A1.I4.i2.p1.5.m5.2.2.1.1.2" xref="A1.I4.i2.p1.5.m5.2.2.1.1.2.cmml">s</mi><mn id="A1.I4.i2.p1.5.m5.2.2.1.1.3" xref="A1.I4.i2.p1.5.m5.2.2.1.1.3.cmml">1</mn></msub><mo id="A1.I4.i2.p1.5.m5.3.3.2.3" xref="A1.I4.i2.p1.5.m5.3.3.3.cmml">,</mo><mi id="A1.I4.i2.p1.5.m5.1.1" xref="A1.I4.i2.p1.5.m5.1.1.cmml">r</mi><mo id="A1.I4.i2.p1.5.m5.3.3.2.4" xref="A1.I4.i2.p1.5.m5.3.3.3.cmml">,</mo><msub id="A1.I4.i2.p1.5.m5.3.3.2.2" xref="A1.I4.i2.p1.5.m5.3.3.2.2.cmml"><mi id="A1.I4.i2.p1.5.m5.3.3.2.2.2" xref="A1.I4.i2.p1.5.m5.3.3.2.2.2.cmml">s</mi><mn id="A1.I4.i2.p1.5.m5.3.3.2.2.3" xref="A1.I4.i2.p1.5.m5.3.3.2.2.3.cmml">2</mn></msub></mrow><annotation-xml encoding="MathML-Content" id="A1.I4.i2.p1.5.m5.3b"><list id="A1.I4.i2.p1.5.m5.3.3.3.cmml" xref="A1.I4.i2.p1.5.m5.3.3.2"><apply id="A1.I4.i2.p1.5.m5.2.2.1.1.cmml" xref="A1.I4.i2.p1.5.m5.2.2.1.1"><csymbol cd="ambiguous" id="A1.I4.i2.p1.5.m5.2.2.1.1.1.cmml" xref="A1.I4.i2.p1.5.m5.2.2.1.1">subscript</csymbol><ci id="A1.I4.i2.p1.5.m5.2.2.1.1.2.cmml" xref="A1.I4.i2.p1.5.m5.2.2.1.1.2">𝑠</ci><cn type="integer" id="A1.I4.i2.p1.5.m5.2.2.1.1.3.cmml" xref="A1.I4.i2.p1.5.m5.2.2.1.1.3">1</cn></apply><ci id="A1.I4.i2.p1.5.m5.1.1.cmml" xref="A1.I4.i2.p1.5.m5.1.1">𝑟</ci><apply id="A1.I4.i2.p1.5.m5.3.3.2.2.cmml" xref="A1.I4.i2.p1.5.m5.3.3.2.2"><csymbol cd="ambiguous" id="A1.I4.i2.p1.5.m5.3.3.2.2.1.cmml" xref="A1.I4.i2.p1.5.m5.3.3.2.2">subscript</csymbol><ci id="A1.I4.i2.p1.5.m5.3.3.2.2.2.cmml" xref="A1.I4.i2.p1.5.m5.3.3.2.2.2">𝑠</ci><cn type="integer" id="A1.I4.i2.p1.5.m5.3.3.2.2.3.cmml" xref="A1.I4.i2.p1.5.m5.3.3.2.2.3">2</cn></apply></list></annotation-xml><annotation encoding="application/x-tex" id="A1.I4.i2.p1.5.m5.3c">s_{1},r,s_{2}</annotation></semantics></math>)

<br class="ltx_break"><span id="A1.I4.i2.p1.6.10" class="ltx_text ltx_font_italic">The notable work of Shakespeare is Hamlet</span>

<br class="ltx_break">where <math id="A1.I4.i2.p1.6.m6.1" class="ltx_Math" alttext="r\land r_{1}\to r_{2}" display="inline"><semantics id="A1.I4.i2.p1.6.m6.1a"><mrow id="A1.I4.i2.p1.6.m6.1.1" xref="A1.I4.i2.p1.6.m6.1.1.cmml"><mrow id="A1.I4.i2.p1.6.m6.1.1.2" xref="A1.I4.i2.p1.6.m6.1.1.2.cmml"><mi id="A1.I4.i2.p1.6.m6.1.1.2.2" xref="A1.I4.i2.p1.6.m6.1.1.2.2.cmml">r</mi><mo id="A1.I4.i2.p1.6.m6.1.1.2.1" xref="A1.I4.i2.p1.6.m6.1.1.2.1.cmml">∧</mo><msub id="A1.I4.i2.p1.6.m6.1.1.2.3" xref="A1.I4.i2.p1.6.m6.1.1.2.3.cmml"><mi id="A1.I4.i2.p1.6.m6.1.1.2.3.2" xref="A1.I4.i2.p1.6.m6.1.1.2.3.2.cmml">r</mi><mn id="A1.I4.i2.p1.6.m6.1.1.2.3.3" xref="A1.I4.i2.p1.6.m6.1.1.2.3.3.cmml">1</mn></msub></mrow><mo stretchy="false" id="A1.I4.i2.p1.6.m6.1.1.1" xref="A1.I4.i2.p1.6.m6.1.1.1.cmml">→</mo><msub id="A1.I4.i2.p1.6.m6.1.1.3" xref="A1.I4.i2.p1.6.m6.1.1.3.cmml"><mi id="A1.I4.i2.p1.6.m6.1.1.3.2" xref="A1.I4.i2.p1.6.m6.1.1.3.2.cmml">r</mi><mn id="A1.I4.i2.p1.6.m6.1.1.3.3" xref="A1.I4.i2.p1.6.m6.1.1.3.3.cmml">2</mn></msub></mrow><annotation-xml encoding="MathML-Content" id="A1.I4.i2.p1.6.m6.1b"><apply id="A1.I4.i2.p1.6.m6.1.1.cmml" xref="A1.I4.i2.p1.6.m6.1.1"><ci id="A1.I4.i2.p1.6.m6.1.1.1.cmml" xref="A1.I4.i2.p1.6.m6.1.1.1">→</ci><apply id="A1.I4.i2.p1.6.m6.1.1.2.cmml" xref="A1.I4.i2.p1.6.m6.1.1.2"><and id="A1.I4.i2.p1.6.m6.1.1.2.1.cmml" xref="A1.I4.i2.p1.6.m6.1.1.2.1"></and><ci id="A1.I4.i2.p1.6.m6.1.1.2.2.cmml" xref="A1.I4.i2.p1.6.m6.1.1.2.2">𝑟</ci><apply id="A1.I4.i2.p1.6.m6.1.1.2.3.cmml" xref="A1.I4.i2.p1.6.m6.1.1.2.3"><csymbol cd="ambiguous" id="A1.I4.i2.p1.6.m6.1.1.2.3.1.cmml" xref="A1.I4.i2.p1.6.m6.1.1.2.3">subscript</csymbol><ci id="A1.I4.i2.p1.6.m6.1.1.2.3.2.cmml" xref="A1.I4.i2.p1.6.m6.1.1.2.3.2">𝑟</ci><cn type="integer" id="A1.I4.i2.p1.6.m6.1.1.2.3.3.cmml" xref="A1.I4.i2.p1.6.m6.1.1.2.3.3">1</cn></apply></apply><apply id="A1.I4.i2.p1.6.m6.1.1.3.cmml" xref="A1.I4.i2.p1.6.m6.1.1.3"><csymbol cd="ambiguous" id="A1.I4.i2.p1.6.m6.1.1.3.1.cmml" xref="A1.I4.i2.p1.6.m6.1.1.3">subscript</csymbol><ci id="A1.I4.i2.p1.6.m6.1.1.3.2.cmml" xref="A1.I4.i2.p1.6.m6.1.1.3.2">𝑟</ci><cn type="integer" id="A1.I4.i2.p1.6.m6.1.1.3.3.cmml" xref="A1.I4.i2.p1.6.m6.1.1.3.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I4.i2.p1.6.m6.1c">r\land r_{1}\to r_{2}</annotation></semantics></math> is a logical rule. The updated knowledge then could be represented as:</p>
<table id="A1.Ex2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.Ex2.m1.9" class="ltx_Math" alttext="\left\{\begin{array}[]{l}k_{f}=(s_{1},r,s_{2})\\
k_{0}=(s_{1},r_{1},o_{2})\\
k_{n}=(s_{1},r_{1},o_{3})\end{array}\right." display="block"><semantics id="A1.Ex2.m1.9a"><mrow id="A1.Ex2.m1.9.10.2" xref="A1.Ex2.m1.9.10.1.cmml"><mo id="A1.Ex2.m1.9.10.2.1" xref="A1.Ex2.m1.9.10.1.1.cmml">{</mo><mtable displaystyle="true" rowspacing="0pt" id="A1.Ex2.m1.9.9" xref="A1.Ex2.m1.9.9.cmml"><mtr id="A1.Ex2.m1.9.9a" xref="A1.Ex2.m1.9.9.cmml"><mtd class="ltx_align_left" columnalign="left" id="A1.Ex2.m1.9.9b" xref="A1.Ex2.m1.9.9.cmml"><mrow id="A1.Ex2.m1.3.3.3.3.3" xref="A1.Ex2.m1.3.3.3.3.3.cmml"><msub id="A1.Ex2.m1.3.3.3.3.3.5" xref="A1.Ex2.m1.3.3.3.3.3.5.cmml"><mi id="A1.Ex2.m1.3.3.3.3.3.5.2" xref="A1.Ex2.m1.3.3.3.3.3.5.2.cmml">k</mi><mi id="A1.Ex2.m1.3.3.3.3.3.5.3" xref="A1.Ex2.m1.3.3.3.3.3.5.3.cmml">f</mi></msub><mo id="A1.Ex2.m1.3.3.3.3.3.4" xref="A1.Ex2.m1.3.3.3.3.3.4.cmml">=</mo><mrow id="A1.Ex2.m1.3.3.3.3.3.3.2" xref="A1.Ex2.m1.3.3.3.3.3.3.3.cmml"><mo stretchy="false" id="A1.Ex2.m1.3.3.3.3.3.3.2.3" xref="A1.Ex2.m1.3.3.3.3.3.3.3.cmml">(</mo><msub id="A1.Ex2.m1.2.2.2.2.2.2.1.1" xref="A1.Ex2.m1.2.2.2.2.2.2.1.1.cmml"><mi id="A1.Ex2.m1.2.2.2.2.2.2.1.1.2" xref="A1.Ex2.m1.2.2.2.2.2.2.1.1.2.cmml">s</mi><mn id="A1.Ex2.m1.2.2.2.2.2.2.1.1.3" xref="A1.Ex2.m1.2.2.2.2.2.2.1.1.3.cmml">1</mn></msub><mo id="A1.Ex2.m1.3.3.3.3.3.3.2.4" xref="A1.Ex2.m1.3.3.3.3.3.3.3.cmml">,</mo><mi id="A1.Ex2.m1.1.1.1.1.1.1" xref="A1.Ex2.m1.1.1.1.1.1.1.cmml">r</mi><mo id="A1.Ex2.m1.3.3.3.3.3.3.2.5" xref="A1.Ex2.m1.3.3.3.3.3.3.3.cmml">,</mo><msub id="A1.Ex2.m1.3.3.3.3.3.3.2.2" xref="A1.Ex2.m1.3.3.3.3.3.3.2.2.cmml"><mi id="A1.Ex2.m1.3.3.3.3.3.3.2.2.2" xref="A1.Ex2.m1.3.3.3.3.3.3.2.2.2.cmml">s</mi><mn id="A1.Ex2.m1.3.3.3.3.3.3.2.2.3" xref="A1.Ex2.m1.3.3.3.3.3.3.2.2.3.cmml">2</mn></msub><mo stretchy="false" id="A1.Ex2.m1.3.3.3.3.3.3.2.6" xref="A1.Ex2.m1.3.3.3.3.3.3.3.cmml">)</mo></mrow></mrow></mtd></mtr><mtr id="A1.Ex2.m1.9.9c" xref="A1.Ex2.m1.9.9.cmml"><mtd class="ltx_align_left" columnalign="left" id="A1.Ex2.m1.9.9d" xref="A1.Ex2.m1.9.9.cmml"><mrow id="A1.Ex2.m1.6.6.6.3.3" xref="A1.Ex2.m1.6.6.6.3.3.cmml"><msub id="A1.Ex2.m1.6.6.6.3.3.5" xref="A1.Ex2.m1.6.6.6.3.3.5.cmml"><mi id="A1.Ex2.m1.6.6.6.3.3.5.2" xref="A1.Ex2.m1.6.6.6.3.3.5.2.cmml">k</mi><mn id="A1.Ex2.m1.6.6.6.3.3.5.3" xref="A1.Ex2.m1.6.6.6.3.3.5.3.cmml">0</mn></msub><mo id="A1.Ex2.m1.6.6.6.3.3.4" xref="A1.Ex2.m1.6.6.6.3.3.4.cmml">=</mo><mrow id="A1.Ex2.m1.6.6.6.3.3.3.3" xref="A1.Ex2.m1.6.6.6.3.3.3.4.cmml"><mo stretchy="false" id="A1.Ex2.m1.6.6.6.3.3.3.3.4" xref="A1.Ex2.m1.6.6.6.3.3.3.4.cmml">(</mo><msub id="A1.Ex2.m1.4.4.4.1.1.1.1.1" xref="A1.Ex2.m1.4.4.4.1.1.1.1.1.cmml"><mi id="A1.Ex2.m1.4.4.4.1.1.1.1.1.2" xref="A1.Ex2.m1.4.4.4.1.1.1.1.1.2.cmml">s</mi><mn id="A1.Ex2.m1.4.4.4.1.1.1.1.1.3" xref="A1.Ex2.m1.4.4.4.1.1.1.1.1.3.cmml">1</mn></msub><mo id="A1.Ex2.m1.6.6.6.3.3.3.3.5" xref="A1.Ex2.m1.6.6.6.3.3.3.4.cmml">,</mo><msub id="A1.Ex2.m1.5.5.5.2.2.2.2.2" xref="A1.Ex2.m1.5.5.5.2.2.2.2.2.cmml"><mi id="A1.Ex2.m1.5.5.5.2.2.2.2.2.2" xref="A1.Ex2.m1.5.5.5.2.2.2.2.2.2.cmml">r</mi><mn id="A1.Ex2.m1.5.5.5.2.2.2.2.2.3" xref="A1.Ex2.m1.5.5.5.2.2.2.2.2.3.cmml">1</mn></msub><mo id="A1.Ex2.m1.6.6.6.3.3.3.3.6" xref="A1.Ex2.m1.6.6.6.3.3.3.4.cmml">,</mo><msub id="A1.Ex2.m1.6.6.6.3.3.3.3.3" xref="A1.Ex2.m1.6.6.6.3.3.3.3.3.cmml"><mi id="A1.Ex2.m1.6.6.6.3.3.3.3.3.2" xref="A1.Ex2.m1.6.6.6.3.3.3.3.3.2.cmml">o</mi><mn id="A1.Ex2.m1.6.6.6.3.3.3.3.3.3" xref="A1.Ex2.m1.6.6.6.3.3.3.3.3.3.cmml">2</mn></msub><mo stretchy="false" id="A1.Ex2.m1.6.6.6.3.3.3.3.7" xref="A1.Ex2.m1.6.6.6.3.3.3.4.cmml">)</mo></mrow></mrow></mtd></mtr><mtr id="A1.Ex2.m1.9.9e" xref="A1.Ex2.m1.9.9.cmml"><mtd class="ltx_align_left" columnalign="left" id="A1.Ex2.m1.9.9f" xref="A1.Ex2.m1.9.9.cmml"><mrow id="A1.Ex2.m1.9.9.9.3.3" xref="A1.Ex2.m1.9.9.9.3.3.cmml"><msub id="A1.Ex2.m1.9.9.9.3.3.5" xref="A1.Ex2.m1.9.9.9.3.3.5.cmml"><mi id="A1.Ex2.m1.9.9.9.3.3.5.2" xref="A1.Ex2.m1.9.9.9.3.3.5.2.cmml">k</mi><mi id="A1.Ex2.m1.9.9.9.3.3.5.3" xref="A1.Ex2.m1.9.9.9.3.3.5.3.cmml">n</mi></msub><mo id="A1.Ex2.m1.9.9.9.3.3.4" xref="A1.Ex2.m1.9.9.9.3.3.4.cmml">=</mo><mrow id="A1.Ex2.m1.9.9.9.3.3.3.3" xref="A1.Ex2.m1.9.9.9.3.3.3.4.cmml"><mo stretchy="false" id="A1.Ex2.m1.9.9.9.3.3.3.3.4" xref="A1.Ex2.m1.9.9.9.3.3.3.4.cmml">(</mo><msub id="A1.Ex2.m1.7.7.7.1.1.1.1.1" xref="A1.Ex2.m1.7.7.7.1.1.1.1.1.cmml"><mi id="A1.Ex2.m1.7.7.7.1.1.1.1.1.2" xref="A1.Ex2.m1.7.7.7.1.1.1.1.1.2.cmml">s</mi><mn id="A1.Ex2.m1.7.7.7.1.1.1.1.1.3" xref="A1.Ex2.m1.7.7.7.1.1.1.1.1.3.cmml">1</mn></msub><mo id="A1.Ex2.m1.9.9.9.3.3.3.3.5" xref="A1.Ex2.m1.9.9.9.3.3.3.4.cmml">,</mo><msub id="A1.Ex2.m1.8.8.8.2.2.2.2.2" xref="A1.Ex2.m1.8.8.8.2.2.2.2.2.cmml"><mi id="A1.Ex2.m1.8.8.8.2.2.2.2.2.2" xref="A1.Ex2.m1.8.8.8.2.2.2.2.2.2.cmml">r</mi><mn id="A1.Ex2.m1.8.8.8.2.2.2.2.2.3" xref="A1.Ex2.m1.8.8.8.2.2.2.2.2.3.cmml">1</mn></msub><mo id="A1.Ex2.m1.9.9.9.3.3.3.3.6" xref="A1.Ex2.m1.9.9.9.3.3.3.4.cmml">,</mo><msub id="A1.Ex2.m1.9.9.9.3.3.3.3.3" xref="A1.Ex2.m1.9.9.9.3.3.3.3.3.cmml"><mi id="A1.Ex2.m1.9.9.9.3.3.3.3.3.2" xref="A1.Ex2.m1.9.9.9.3.3.3.3.3.2.cmml">o</mi><mn id="A1.Ex2.m1.9.9.9.3.3.3.3.3.3" xref="A1.Ex2.m1.9.9.9.3.3.3.3.3.3.cmml">3</mn></msub><mo stretchy="false" id="A1.Ex2.m1.9.9.9.3.3.3.3.7" xref="A1.Ex2.m1.9.9.9.3.3.3.4.cmml">)</mo></mrow></mrow></mtd></mtr></mtable><mi id="A1.Ex2.m1.9.10.2.2" xref="A1.Ex2.m1.9.10.1.1.cmml"></mi></mrow><annotation-xml encoding="MathML-Content" id="A1.Ex2.m1.9b"><apply id="A1.Ex2.m1.9.10.1.cmml" xref="A1.Ex2.m1.9.10.2"><csymbol cd="latexml" id="A1.Ex2.m1.9.10.1.1.cmml" xref="A1.Ex2.m1.9.10.2.1">cases</csymbol><matrix id="A1.Ex2.m1.9.9.cmml" xref="A1.Ex2.m1.9.9"><matrixrow id="A1.Ex2.m1.9.9a.cmml" xref="A1.Ex2.m1.9.9"><apply id="A1.Ex2.m1.3.3.3.3.3.cmml" xref="A1.Ex2.m1.3.3.3.3.3"><eq id="A1.Ex2.m1.3.3.3.3.3.4.cmml" xref="A1.Ex2.m1.3.3.3.3.3.4"></eq><apply id="A1.Ex2.m1.3.3.3.3.3.5.cmml" xref="A1.Ex2.m1.3.3.3.3.3.5"><csymbol cd="ambiguous" id="A1.Ex2.m1.3.3.3.3.3.5.1.cmml" xref="A1.Ex2.m1.3.3.3.3.3.5">subscript</csymbol><ci id="A1.Ex2.m1.3.3.3.3.3.5.2.cmml" xref="A1.Ex2.m1.3.3.3.3.3.5.2">𝑘</ci><ci id="A1.Ex2.m1.3.3.3.3.3.5.3.cmml" xref="A1.Ex2.m1.3.3.3.3.3.5.3">𝑓</ci></apply><vector id="A1.Ex2.m1.3.3.3.3.3.3.3.cmml" xref="A1.Ex2.m1.3.3.3.3.3.3.2"><apply id="A1.Ex2.m1.2.2.2.2.2.2.1.1.cmml" xref="A1.Ex2.m1.2.2.2.2.2.2.1.1"><csymbol cd="ambiguous" id="A1.Ex2.m1.2.2.2.2.2.2.1.1.1.cmml" xref="A1.Ex2.m1.2.2.2.2.2.2.1.1">subscript</csymbol><ci id="A1.Ex2.m1.2.2.2.2.2.2.1.1.2.cmml" xref="A1.Ex2.m1.2.2.2.2.2.2.1.1.2">𝑠</ci><cn type="integer" id="A1.Ex2.m1.2.2.2.2.2.2.1.1.3.cmml" xref="A1.Ex2.m1.2.2.2.2.2.2.1.1.3">1</cn></apply><ci id="A1.Ex2.m1.1.1.1.1.1.1.cmml" xref="A1.Ex2.m1.1.1.1.1.1.1">𝑟</ci><apply id="A1.Ex2.m1.3.3.3.3.3.3.2.2.cmml" xref="A1.Ex2.m1.3.3.3.3.3.3.2.2"><csymbol cd="ambiguous" id="A1.Ex2.m1.3.3.3.3.3.3.2.2.1.cmml" xref="A1.Ex2.m1.3.3.3.3.3.3.2.2">subscript</csymbol><ci id="A1.Ex2.m1.3.3.3.3.3.3.2.2.2.cmml" xref="A1.Ex2.m1.3.3.3.3.3.3.2.2.2">𝑠</ci><cn type="integer" id="A1.Ex2.m1.3.3.3.3.3.3.2.2.3.cmml" xref="A1.Ex2.m1.3.3.3.3.3.3.2.2.3">2</cn></apply></vector></apply></matrixrow><matrixrow id="A1.Ex2.m1.9.9b.cmml" xref="A1.Ex2.m1.9.9"><apply id="A1.Ex2.m1.6.6.6.3.3.cmml" xref="A1.Ex2.m1.6.6.6.3.3"><eq id="A1.Ex2.m1.6.6.6.3.3.4.cmml" xref="A1.Ex2.m1.6.6.6.3.3.4"></eq><apply id="A1.Ex2.m1.6.6.6.3.3.5.cmml" xref="A1.Ex2.m1.6.6.6.3.3.5"><csymbol cd="ambiguous" id="A1.Ex2.m1.6.6.6.3.3.5.1.cmml" xref="A1.Ex2.m1.6.6.6.3.3.5">subscript</csymbol><ci id="A1.Ex2.m1.6.6.6.3.3.5.2.cmml" xref="A1.Ex2.m1.6.6.6.3.3.5.2">𝑘</ci><cn type="integer" id="A1.Ex2.m1.6.6.6.3.3.5.3.cmml" xref="A1.Ex2.m1.6.6.6.3.3.5.3">0</cn></apply><vector id="A1.Ex2.m1.6.6.6.3.3.3.4.cmml" xref="A1.Ex2.m1.6.6.6.3.3.3.3"><apply id="A1.Ex2.m1.4.4.4.1.1.1.1.1.cmml" xref="A1.Ex2.m1.4.4.4.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.Ex2.m1.4.4.4.1.1.1.1.1.1.cmml" xref="A1.Ex2.m1.4.4.4.1.1.1.1.1">subscript</csymbol><ci id="A1.Ex2.m1.4.4.4.1.1.1.1.1.2.cmml" xref="A1.Ex2.m1.4.4.4.1.1.1.1.1.2">𝑠</ci><cn type="integer" id="A1.Ex2.m1.4.4.4.1.1.1.1.1.3.cmml" xref="A1.Ex2.m1.4.4.4.1.1.1.1.1.3">1</cn></apply><apply id="A1.Ex2.m1.5.5.5.2.2.2.2.2.cmml" xref="A1.Ex2.m1.5.5.5.2.2.2.2.2"><csymbol cd="ambiguous" id="A1.Ex2.m1.5.5.5.2.2.2.2.2.1.cmml" xref="A1.Ex2.m1.5.5.5.2.2.2.2.2">subscript</csymbol><ci id="A1.Ex2.m1.5.5.5.2.2.2.2.2.2.cmml" xref="A1.Ex2.m1.5.5.5.2.2.2.2.2.2">𝑟</ci><cn type="integer" id="A1.Ex2.m1.5.5.5.2.2.2.2.2.3.cmml" xref="A1.Ex2.m1.5.5.5.2.2.2.2.2.3">1</cn></apply><apply id="A1.Ex2.m1.6.6.6.3.3.3.3.3.cmml" xref="A1.Ex2.m1.6.6.6.3.3.3.3.3"><csymbol cd="ambiguous" id="A1.Ex2.m1.6.6.6.3.3.3.3.3.1.cmml" xref="A1.Ex2.m1.6.6.6.3.3.3.3.3">subscript</csymbol><ci id="A1.Ex2.m1.6.6.6.3.3.3.3.3.2.cmml" xref="A1.Ex2.m1.6.6.6.3.3.3.3.3.2">𝑜</ci><cn type="integer" id="A1.Ex2.m1.6.6.6.3.3.3.3.3.3.cmml" xref="A1.Ex2.m1.6.6.6.3.3.3.3.3.3">2</cn></apply></vector></apply></matrixrow><matrixrow id="A1.Ex2.m1.9.9c.cmml" xref="A1.Ex2.m1.9.9"><apply id="A1.Ex2.m1.9.9.9.3.3.cmml" xref="A1.Ex2.m1.9.9.9.3.3"><eq id="A1.Ex2.m1.9.9.9.3.3.4.cmml" xref="A1.Ex2.m1.9.9.9.3.3.4"></eq><apply id="A1.Ex2.m1.9.9.9.3.3.5.cmml" xref="A1.Ex2.m1.9.9.9.3.3.5"><csymbol cd="ambiguous" id="A1.Ex2.m1.9.9.9.3.3.5.1.cmml" xref="A1.Ex2.m1.9.9.9.3.3.5">subscript</csymbol><ci id="A1.Ex2.m1.9.9.9.3.3.5.2.cmml" xref="A1.Ex2.m1.9.9.9.3.3.5.2">𝑘</ci><ci id="A1.Ex2.m1.9.9.9.3.3.5.3.cmml" xref="A1.Ex2.m1.9.9.9.3.3.5.3">𝑛</ci></apply><vector id="A1.Ex2.m1.9.9.9.3.3.3.4.cmml" xref="A1.Ex2.m1.9.9.9.3.3.3.3"><apply id="A1.Ex2.m1.7.7.7.1.1.1.1.1.cmml" xref="A1.Ex2.m1.7.7.7.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.Ex2.m1.7.7.7.1.1.1.1.1.1.cmml" xref="A1.Ex2.m1.7.7.7.1.1.1.1.1">subscript</csymbol><ci id="A1.Ex2.m1.7.7.7.1.1.1.1.1.2.cmml" xref="A1.Ex2.m1.7.7.7.1.1.1.1.1.2">𝑠</ci><cn type="integer" id="A1.Ex2.m1.7.7.7.1.1.1.1.1.3.cmml" xref="A1.Ex2.m1.7.7.7.1.1.1.1.1.3">1</cn></apply><apply id="A1.Ex2.m1.8.8.8.2.2.2.2.2.cmml" xref="A1.Ex2.m1.8.8.8.2.2.2.2.2"><csymbol cd="ambiguous" id="A1.Ex2.m1.8.8.8.2.2.2.2.2.1.cmml" xref="A1.Ex2.m1.8.8.8.2.2.2.2.2">subscript</csymbol><ci id="A1.Ex2.m1.8.8.8.2.2.2.2.2.2.cmml" xref="A1.Ex2.m1.8.8.8.2.2.2.2.2.2">𝑟</ci><cn type="integer" id="A1.Ex2.m1.8.8.8.2.2.2.2.2.3.cmml" xref="A1.Ex2.m1.8.8.8.2.2.2.2.2.3">1</cn></apply><apply id="A1.Ex2.m1.9.9.9.3.3.3.3.3.cmml" xref="A1.Ex2.m1.9.9.9.3.3.3.3.3"><csymbol cd="ambiguous" id="A1.Ex2.m1.9.9.9.3.3.3.3.3.1.cmml" xref="A1.Ex2.m1.9.9.9.3.3.3.3.3">subscript</csymbol><ci id="A1.Ex2.m1.9.9.9.3.3.3.3.3.2.cmml" xref="A1.Ex2.m1.9.9.9.3.3.3.3.3.2">𝑜</ci><cn type="integer" id="A1.Ex2.m1.9.9.9.3.3.3.3.3.3.cmml" xref="A1.Ex2.m1.9.9.9.3.3.3.3.3.3">3</cn></apply></vector></apply></matrixrow></matrix></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.Ex2.m1.9c">\left\{\begin{array}[]{l}k_{f}=(s_{1},r,s_{2})\\
k_{0}=(s_{1},r_{1},o_{2})\\
k_{n}=(s_{1},r_{1},o_{3})\end{array}\right.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
</li>
</ul>
</div>
<div id="A1.SS1.SSS0.Px2.p3" class="ltx_para">
<p id="A1.SS1.SSS0.Px2.p3.1" class="ltx_p">The evaluation methods include:</p>
</div>
<div id="A1.SS1.SSS0.Px2.p4" class="ltx_para">
<ul id="A1.I5" class="ltx_itemize">
<li id="A1.I5.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I5.i1.p1" class="ltx_para">
<p id="A1.I5.i1.p1.1" class="ltx_p"><span id="A1.I5.i1.p1.1.1" class="ltx_text ltx_font_bold">Conflict Score (CS):</span> Measures how well a knowledge editing method handles knowledge conflicts by calculating the ratio that the new fact is more probable than the old fact after knowledge editing.</p>
<table id="A1.E8" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.E8.m1.1" class="ltx_Math" alttext="\mathrm{CS=}\mathbbm{1}\{p_{f_{\theta}^{\prime}}(k_{n})&gt;p_{f_{\theta}^{\prime}}(k_{o})\}" display="block"><semantics id="A1.E8.m1.1a"><mrow id="A1.E8.m1.1.1" xref="A1.E8.m1.1.1.cmml"><mi id="A1.E8.m1.1.1.3" xref="A1.E8.m1.1.1.3.cmml">CS</mi><mo id="A1.E8.m1.1.1.2" xref="A1.E8.m1.1.1.2.cmml">=</mo><mrow id="A1.E8.m1.1.1.1" xref="A1.E8.m1.1.1.1.cmml"><mn id="A1.E8.m1.1.1.1.3" xref="A1.E8.m1.1.1.1.3.cmml">𝟙</mn><mo lspace="0em" rspace="0em" id="A1.E8.m1.1.1.1.2" xref="A1.E8.m1.1.1.1.2.cmml">​</mo><mrow id="A1.E8.m1.1.1.1.1.1" xref="A1.E8.m1.1.1.1.1.2.cmml"><mo stretchy="false" id="A1.E8.m1.1.1.1.1.1.2" xref="A1.E8.m1.1.1.1.1.2.cmml">{</mo><mrow id="A1.E8.m1.1.1.1.1.1.1" xref="A1.E8.m1.1.1.1.1.1.1.cmml"><mrow id="A1.E8.m1.1.1.1.1.1.1.1" xref="A1.E8.m1.1.1.1.1.1.1.1.cmml"><msub id="A1.E8.m1.1.1.1.1.1.1.1.3" xref="A1.E8.m1.1.1.1.1.1.1.1.3.cmml"><mi id="A1.E8.m1.1.1.1.1.1.1.1.3.2" xref="A1.E8.m1.1.1.1.1.1.1.1.3.2.cmml">p</mi><msubsup id="A1.E8.m1.1.1.1.1.1.1.1.3.3" xref="A1.E8.m1.1.1.1.1.1.1.1.3.3.cmml"><mi id="A1.E8.m1.1.1.1.1.1.1.1.3.3.2.2" xref="A1.E8.m1.1.1.1.1.1.1.1.3.3.2.2.cmml">f</mi><mi id="A1.E8.m1.1.1.1.1.1.1.1.3.3.2.3" xref="A1.E8.m1.1.1.1.1.1.1.1.3.3.2.3.cmml">θ</mi><mo id="A1.E8.m1.1.1.1.1.1.1.1.3.3.3" xref="A1.E8.m1.1.1.1.1.1.1.1.3.3.3.cmml">′</mo></msubsup></msub><mo lspace="0em" rspace="0em" id="A1.E8.m1.1.1.1.1.1.1.1.2" xref="A1.E8.m1.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="A1.E8.m1.1.1.1.1.1.1.1.1.1" xref="A1.E8.m1.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.E8.m1.1.1.1.1.1.1.1.1.1.2" xref="A1.E8.m1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><msub id="A1.E8.m1.1.1.1.1.1.1.1.1.1.1" xref="A1.E8.m1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="A1.E8.m1.1.1.1.1.1.1.1.1.1.1.2" xref="A1.E8.m1.1.1.1.1.1.1.1.1.1.1.2.cmml">k</mi><mi id="A1.E8.m1.1.1.1.1.1.1.1.1.1.1.3" xref="A1.E8.m1.1.1.1.1.1.1.1.1.1.1.3.cmml">n</mi></msub><mo stretchy="false" id="A1.E8.m1.1.1.1.1.1.1.1.1.1.3" xref="A1.E8.m1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="A1.E8.m1.1.1.1.1.1.1.3" xref="A1.E8.m1.1.1.1.1.1.1.3.cmml">&gt;</mo><mrow id="A1.E8.m1.1.1.1.1.1.1.2" xref="A1.E8.m1.1.1.1.1.1.1.2.cmml"><msub id="A1.E8.m1.1.1.1.1.1.1.2.3" xref="A1.E8.m1.1.1.1.1.1.1.2.3.cmml"><mi id="A1.E8.m1.1.1.1.1.1.1.2.3.2" xref="A1.E8.m1.1.1.1.1.1.1.2.3.2.cmml">p</mi><msubsup id="A1.E8.m1.1.1.1.1.1.1.2.3.3" xref="A1.E8.m1.1.1.1.1.1.1.2.3.3.cmml"><mi id="A1.E8.m1.1.1.1.1.1.1.2.3.3.2.2" xref="A1.E8.m1.1.1.1.1.1.1.2.3.3.2.2.cmml">f</mi><mi id="A1.E8.m1.1.1.1.1.1.1.2.3.3.2.3" xref="A1.E8.m1.1.1.1.1.1.1.2.3.3.2.3.cmml">θ</mi><mo id="A1.E8.m1.1.1.1.1.1.1.2.3.3.3" xref="A1.E8.m1.1.1.1.1.1.1.2.3.3.3.cmml">′</mo></msubsup></msub><mo lspace="0em" rspace="0em" id="A1.E8.m1.1.1.1.1.1.1.2.2" xref="A1.E8.m1.1.1.1.1.1.1.2.2.cmml">​</mo><mrow id="A1.E8.m1.1.1.1.1.1.1.2.1.1" xref="A1.E8.m1.1.1.1.1.1.1.2.1.1.1.cmml"><mo stretchy="false" id="A1.E8.m1.1.1.1.1.1.1.2.1.1.2" xref="A1.E8.m1.1.1.1.1.1.1.2.1.1.1.cmml">(</mo><msub id="A1.E8.m1.1.1.1.1.1.1.2.1.1.1" xref="A1.E8.m1.1.1.1.1.1.1.2.1.1.1.cmml"><mi id="A1.E8.m1.1.1.1.1.1.1.2.1.1.1.2" xref="A1.E8.m1.1.1.1.1.1.1.2.1.1.1.2.cmml">k</mi><mi id="A1.E8.m1.1.1.1.1.1.1.2.1.1.1.3" xref="A1.E8.m1.1.1.1.1.1.1.2.1.1.1.3.cmml">o</mi></msub><mo stretchy="false" id="A1.E8.m1.1.1.1.1.1.1.2.1.1.3" xref="A1.E8.m1.1.1.1.1.1.1.2.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="A1.E8.m1.1.1.1.1.1.3" xref="A1.E8.m1.1.1.1.1.2.cmml">}</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E8.m1.1b"><apply id="A1.E8.m1.1.1.cmml" xref="A1.E8.m1.1.1"><eq id="A1.E8.m1.1.1.2.cmml" xref="A1.E8.m1.1.1.2"></eq><ci id="A1.E8.m1.1.1.3.cmml" xref="A1.E8.m1.1.1.3">CS</ci><apply id="A1.E8.m1.1.1.1.cmml" xref="A1.E8.m1.1.1.1"><times id="A1.E8.m1.1.1.1.2.cmml" xref="A1.E8.m1.1.1.1.2"></times><cn type="integer" id="A1.E8.m1.1.1.1.3.cmml" xref="A1.E8.m1.1.1.1.3">1</cn><set id="A1.E8.m1.1.1.1.1.2.cmml" xref="A1.E8.m1.1.1.1.1.1"><apply id="A1.E8.m1.1.1.1.1.1.1.cmml" xref="A1.E8.m1.1.1.1.1.1.1"><gt id="A1.E8.m1.1.1.1.1.1.1.3.cmml" xref="A1.E8.m1.1.1.1.1.1.1.3"></gt><apply id="A1.E8.m1.1.1.1.1.1.1.1.cmml" xref="A1.E8.m1.1.1.1.1.1.1.1"><times id="A1.E8.m1.1.1.1.1.1.1.1.2.cmml" xref="A1.E8.m1.1.1.1.1.1.1.1.2"></times><apply id="A1.E8.m1.1.1.1.1.1.1.1.3.cmml" xref="A1.E8.m1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.E8.m1.1.1.1.1.1.1.1.3.1.cmml" xref="A1.E8.m1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="A1.E8.m1.1.1.1.1.1.1.1.3.2.cmml" xref="A1.E8.m1.1.1.1.1.1.1.1.3.2">𝑝</ci><apply id="A1.E8.m1.1.1.1.1.1.1.1.3.3.cmml" xref="A1.E8.m1.1.1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="A1.E8.m1.1.1.1.1.1.1.1.3.3.1.cmml" xref="A1.E8.m1.1.1.1.1.1.1.1.3.3">superscript</csymbol><apply id="A1.E8.m1.1.1.1.1.1.1.1.3.3.2.cmml" xref="A1.E8.m1.1.1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="A1.E8.m1.1.1.1.1.1.1.1.3.3.2.1.cmml" xref="A1.E8.m1.1.1.1.1.1.1.1.3.3">subscript</csymbol><ci id="A1.E8.m1.1.1.1.1.1.1.1.3.3.2.2.cmml" xref="A1.E8.m1.1.1.1.1.1.1.1.3.3.2.2">𝑓</ci><ci id="A1.E8.m1.1.1.1.1.1.1.1.3.3.2.3.cmml" xref="A1.E8.m1.1.1.1.1.1.1.1.3.3.2.3">𝜃</ci></apply><ci id="A1.E8.m1.1.1.1.1.1.1.1.3.3.3.cmml" xref="A1.E8.m1.1.1.1.1.1.1.1.3.3.3">′</ci></apply></apply><apply id="A1.E8.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A1.E8.m1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.E8.m1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A1.E8.m1.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="A1.E8.m1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="A1.E8.m1.1.1.1.1.1.1.1.1.1.1.2">𝑘</ci><ci id="A1.E8.m1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="A1.E8.m1.1.1.1.1.1.1.1.1.1.1.3">𝑛</ci></apply></apply><apply id="A1.E8.m1.1.1.1.1.1.1.2.cmml" xref="A1.E8.m1.1.1.1.1.1.1.2"><times id="A1.E8.m1.1.1.1.1.1.1.2.2.cmml" xref="A1.E8.m1.1.1.1.1.1.1.2.2"></times><apply id="A1.E8.m1.1.1.1.1.1.1.2.3.cmml" xref="A1.E8.m1.1.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="A1.E8.m1.1.1.1.1.1.1.2.3.1.cmml" xref="A1.E8.m1.1.1.1.1.1.1.2.3">subscript</csymbol><ci id="A1.E8.m1.1.1.1.1.1.1.2.3.2.cmml" xref="A1.E8.m1.1.1.1.1.1.1.2.3.2">𝑝</ci><apply id="A1.E8.m1.1.1.1.1.1.1.2.3.3.cmml" xref="A1.E8.m1.1.1.1.1.1.1.2.3.3"><csymbol cd="ambiguous" id="A1.E8.m1.1.1.1.1.1.1.2.3.3.1.cmml" xref="A1.E8.m1.1.1.1.1.1.1.2.3.3">superscript</csymbol><apply id="A1.E8.m1.1.1.1.1.1.1.2.3.3.2.cmml" xref="A1.E8.m1.1.1.1.1.1.1.2.3.3"><csymbol cd="ambiguous" id="A1.E8.m1.1.1.1.1.1.1.2.3.3.2.1.cmml" xref="A1.E8.m1.1.1.1.1.1.1.2.3.3">subscript</csymbol><ci id="A1.E8.m1.1.1.1.1.1.1.2.3.3.2.2.cmml" xref="A1.E8.m1.1.1.1.1.1.1.2.3.3.2.2">𝑓</ci><ci id="A1.E8.m1.1.1.1.1.1.1.2.3.3.2.3.cmml" xref="A1.E8.m1.1.1.1.1.1.1.2.3.3.2.3">𝜃</ci></apply><ci id="A1.E8.m1.1.1.1.1.1.1.2.3.3.3.cmml" xref="A1.E8.m1.1.1.1.1.1.1.2.3.3.3">′</ci></apply></apply><apply id="A1.E8.m1.1.1.1.1.1.1.2.1.1.1.cmml" xref="A1.E8.m1.1.1.1.1.1.1.2.1.1"><csymbol cd="ambiguous" id="A1.E8.m1.1.1.1.1.1.1.2.1.1.1.1.cmml" xref="A1.E8.m1.1.1.1.1.1.1.2.1.1">subscript</csymbol><ci id="A1.E8.m1.1.1.1.1.1.1.2.1.1.1.2.cmml" xref="A1.E8.m1.1.1.1.1.1.1.2.1.1.1.2">𝑘</ci><ci id="A1.E8.m1.1.1.1.1.1.1.2.1.1.1.3.cmml" xref="A1.E8.m1.1.1.1.1.1.1.2.1.1.1.3">𝑜</ci></apply></apply></apply></set></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E8.m1.1c">\mathrm{CS=}\mathbbm{1}\{p_{f_{\theta}^{\prime}}(k_{n})&gt;p_{f_{\theta}^{\prime}}(k_{o})\}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(8)</span></td>
</tr></tbody>
</table>
</div>
</li>
<li id="A1.I5.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I5.i2.p1" class="ltx_para">
<p id="A1.I5.i2.p1.2" class="ltx_p"><span id="A1.I5.i2.p1.2.1" class="ltx_text ltx_font_bold">Conflict Magnitude (CM):</span> Estimates the decrease in probability of the old fact after editing.</p>
<table id="A1.E9" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.E9.m1.3" class="ltx_Math" alttext="\mathrm{CM=}\frac{p_{f_{\theta^{m}}}(k_{o})-p_{f_{\theta^{\prime}}}(k_{o})}{p_{f_{\theta^{m}}}(k_{o})}" display="block"><semantics id="A1.E9.m1.3a"><mrow id="A1.E9.m1.3.4" xref="A1.E9.m1.3.4.cmml"><mi id="A1.E9.m1.3.4.2" xref="A1.E9.m1.3.4.2.cmml">CM</mi><mo id="A1.E9.m1.3.4.1" xref="A1.E9.m1.3.4.1.cmml">=</mo><mfrac id="A1.E9.m1.3.3" xref="A1.E9.m1.3.3.cmml"><mrow id="A1.E9.m1.2.2.2" xref="A1.E9.m1.2.2.2.cmml"><mrow id="A1.E9.m1.1.1.1.1" xref="A1.E9.m1.1.1.1.1.cmml"><msub id="A1.E9.m1.1.1.1.1.3" xref="A1.E9.m1.1.1.1.1.3.cmml"><mi id="A1.E9.m1.1.1.1.1.3.2" xref="A1.E9.m1.1.1.1.1.3.2.cmml">p</mi><msub id="A1.E9.m1.1.1.1.1.3.3" xref="A1.E9.m1.1.1.1.1.3.3.cmml"><mi id="A1.E9.m1.1.1.1.1.3.3.2" xref="A1.E9.m1.1.1.1.1.3.3.2.cmml">f</mi><msup id="A1.E9.m1.1.1.1.1.3.3.3" xref="A1.E9.m1.1.1.1.1.3.3.3.cmml"><mi id="A1.E9.m1.1.1.1.1.3.3.3.2" xref="A1.E9.m1.1.1.1.1.3.3.3.2.cmml">θ</mi><mi id="A1.E9.m1.1.1.1.1.3.3.3.3" xref="A1.E9.m1.1.1.1.1.3.3.3.3.cmml">m</mi></msup></msub></msub><mo lspace="0em" rspace="0em" id="A1.E9.m1.1.1.1.1.2" xref="A1.E9.m1.1.1.1.1.2.cmml">​</mo><mrow id="A1.E9.m1.1.1.1.1.1.1" xref="A1.E9.m1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.E9.m1.1.1.1.1.1.1.2" xref="A1.E9.m1.1.1.1.1.1.1.1.cmml">(</mo><msub id="A1.E9.m1.1.1.1.1.1.1.1" xref="A1.E9.m1.1.1.1.1.1.1.1.cmml"><mi id="A1.E9.m1.1.1.1.1.1.1.1.2" xref="A1.E9.m1.1.1.1.1.1.1.1.2.cmml">k</mi><mi id="A1.E9.m1.1.1.1.1.1.1.1.3" xref="A1.E9.m1.1.1.1.1.1.1.1.3.cmml">o</mi></msub><mo stretchy="false" id="A1.E9.m1.1.1.1.1.1.1.3" xref="A1.E9.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="A1.E9.m1.2.2.2.3" xref="A1.E9.m1.2.2.2.3.cmml">−</mo><mrow id="A1.E9.m1.2.2.2.2" xref="A1.E9.m1.2.2.2.2.cmml"><msub id="A1.E9.m1.2.2.2.2.3" xref="A1.E9.m1.2.2.2.2.3.cmml"><mi id="A1.E9.m1.2.2.2.2.3.2" xref="A1.E9.m1.2.2.2.2.3.2.cmml">p</mi><msub id="A1.E9.m1.2.2.2.2.3.3" xref="A1.E9.m1.2.2.2.2.3.3.cmml"><mi id="A1.E9.m1.2.2.2.2.3.3.2" xref="A1.E9.m1.2.2.2.2.3.3.2.cmml">f</mi><msup id="A1.E9.m1.2.2.2.2.3.3.3" xref="A1.E9.m1.2.2.2.2.3.3.3.cmml"><mi id="A1.E9.m1.2.2.2.2.3.3.3.2" xref="A1.E9.m1.2.2.2.2.3.3.3.2.cmml">θ</mi><mo id="A1.E9.m1.2.2.2.2.3.3.3.3" xref="A1.E9.m1.2.2.2.2.3.3.3.3.cmml">′</mo></msup></msub></msub><mo lspace="0em" rspace="0em" id="A1.E9.m1.2.2.2.2.2" xref="A1.E9.m1.2.2.2.2.2.cmml">​</mo><mrow id="A1.E9.m1.2.2.2.2.1.1" xref="A1.E9.m1.2.2.2.2.1.1.1.cmml"><mo stretchy="false" id="A1.E9.m1.2.2.2.2.1.1.2" xref="A1.E9.m1.2.2.2.2.1.1.1.cmml">(</mo><msub id="A1.E9.m1.2.2.2.2.1.1.1" xref="A1.E9.m1.2.2.2.2.1.1.1.cmml"><mi id="A1.E9.m1.2.2.2.2.1.1.1.2" xref="A1.E9.m1.2.2.2.2.1.1.1.2.cmml">k</mi><mi id="A1.E9.m1.2.2.2.2.1.1.1.3" xref="A1.E9.m1.2.2.2.2.1.1.1.3.cmml">o</mi></msub><mo stretchy="false" id="A1.E9.m1.2.2.2.2.1.1.3" xref="A1.E9.m1.2.2.2.2.1.1.1.cmml">)</mo></mrow></mrow></mrow><mrow id="A1.E9.m1.3.3.3" xref="A1.E9.m1.3.3.3.cmml"><msub id="A1.E9.m1.3.3.3.3" xref="A1.E9.m1.3.3.3.3.cmml"><mi id="A1.E9.m1.3.3.3.3.2" xref="A1.E9.m1.3.3.3.3.2.cmml">p</mi><msub id="A1.E9.m1.3.3.3.3.3" xref="A1.E9.m1.3.3.3.3.3.cmml"><mi id="A1.E9.m1.3.3.3.3.3.2" xref="A1.E9.m1.3.3.3.3.3.2.cmml">f</mi><msup id="A1.E9.m1.3.3.3.3.3.3" xref="A1.E9.m1.3.3.3.3.3.3.cmml"><mi id="A1.E9.m1.3.3.3.3.3.3.2" xref="A1.E9.m1.3.3.3.3.3.3.2.cmml">θ</mi><mi id="A1.E9.m1.3.3.3.3.3.3.3" xref="A1.E9.m1.3.3.3.3.3.3.3.cmml">m</mi></msup></msub></msub><mo lspace="0em" rspace="0em" id="A1.E9.m1.3.3.3.2" xref="A1.E9.m1.3.3.3.2.cmml">​</mo><mrow id="A1.E9.m1.3.3.3.1.1" xref="A1.E9.m1.3.3.3.1.1.1.cmml"><mo stretchy="false" id="A1.E9.m1.3.3.3.1.1.2" xref="A1.E9.m1.3.3.3.1.1.1.cmml">(</mo><msub id="A1.E9.m1.3.3.3.1.1.1" xref="A1.E9.m1.3.3.3.1.1.1.cmml"><mi id="A1.E9.m1.3.3.3.1.1.1.2" xref="A1.E9.m1.3.3.3.1.1.1.2.cmml">k</mi><mi id="A1.E9.m1.3.3.3.1.1.1.3" xref="A1.E9.m1.3.3.3.1.1.1.3.cmml">o</mi></msub><mo stretchy="false" id="A1.E9.m1.3.3.3.1.1.3" xref="A1.E9.m1.3.3.3.1.1.1.cmml">)</mo></mrow></mrow></mfrac></mrow><annotation-xml encoding="MathML-Content" id="A1.E9.m1.3b"><apply id="A1.E9.m1.3.4.cmml" xref="A1.E9.m1.3.4"><eq id="A1.E9.m1.3.4.1.cmml" xref="A1.E9.m1.3.4.1"></eq><ci id="A1.E9.m1.3.4.2.cmml" xref="A1.E9.m1.3.4.2">CM</ci><apply id="A1.E9.m1.3.3.cmml" xref="A1.E9.m1.3.3"><divide id="A1.E9.m1.3.3.4.cmml" xref="A1.E9.m1.3.3"></divide><apply id="A1.E9.m1.2.2.2.cmml" xref="A1.E9.m1.2.2.2"><minus id="A1.E9.m1.2.2.2.3.cmml" xref="A1.E9.m1.2.2.2.3"></minus><apply id="A1.E9.m1.1.1.1.1.cmml" xref="A1.E9.m1.1.1.1.1"><times id="A1.E9.m1.1.1.1.1.2.cmml" xref="A1.E9.m1.1.1.1.1.2"></times><apply id="A1.E9.m1.1.1.1.1.3.cmml" xref="A1.E9.m1.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.E9.m1.1.1.1.1.3.1.cmml" xref="A1.E9.m1.1.1.1.1.3">subscript</csymbol><ci id="A1.E9.m1.1.1.1.1.3.2.cmml" xref="A1.E9.m1.1.1.1.1.3.2">𝑝</ci><apply id="A1.E9.m1.1.1.1.1.3.3.cmml" xref="A1.E9.m1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="A1.E9.m1.1.1.1.1.3.3.1.cmml" xref="A1.E9.m1.1.1.1.1.3.3">subscript</csymbol><ci id="A1.E9.m1.1.1.1.1.3.3.2.cmml" xref="A1.E9.m1.1.1.1.1.3.3.2">𝑓</ci><apply id="A1.E9.m1.1.1.1.1.3.3.3.cmml" xref="A1.E9.m1.1.1.1.1.3.3.3"><csymbol cd="ambiguous" id="A1.E9.m1.1.1.1.1.3.3.3.1.cmml" xref="A1.E9.m1.1.1.1.1.3.3.3">superscript</csymbol><ci id="A1.E9.m1.1.1.1.1.3.3.3.2.cmml" xref="A1.E9.m1.1.1.1.1.3.3.3.2">𝜃</ci><ci id="A1.E9.m1.1.1.1.1.3.3.3.3.cmml" xref="A1.E9.m1.1.1.1.1.3.3.3.3">𝑚</ci></apply></apply></apply><apply id="A1.E9.m1.1.1.1.1.1.1.1.cmml" xref="A1.E9.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.E9.m1.1.1.1.1.1.1.1.1.cmml" xref="A1.E9.m1.1.1.1.1.1.1">subscript</csymbol><ci id="A1.E9.m1.1.1.1.1.1.1.1.2.cmml" xref="A1.E9.m1.1.1.1.1.1.1.1.2">𝑘</ci><ci id="A1.E9.m1.1.1.1.1.1.1.1.3.cmml" xref="A1.E9.m1.1.1.1.1.1.1.1.3">𝑜</ci></apply></apply><apply id="A1.E9.m1.2.2.2.2.cmml" xref="A1.E9.m1.2.2.2.2"><times id="A1.E9.m1.2.2.2.2.2.cmml" xref="A1.E9.m1.2.2.2.2.2"></times><apply id="A1.E9.m1.2.2.2.2.3.cmml" xref="A1.E9.m1.2.2.2.2.3"><csymbol cd="ambiguous" id="A1.E9.m1.2.2.2.2.3.1.cmml" xref="A1.E9.m1.2.2.2.2.3">subscript</csymbol><ci id="A1.E9.m1.2.2.2.2.3.2.cmml" xref="A1.E9.m1.2.2.2.2.3.2">𝑝</ci><apply id="A1.E9.m1.2.2.2.2.3.3.cmml" xref="A1.E9.m1.2.2.2.2.3.3"><csymbol cd="ambiguous" id="A1.E9.m1.2.2.2.2.3.3.1.cmml" xref="A1.E9.m1.2.2.2.2.3.3">subscript</csymbol><ci id="A1.E9.m1.2.2.2.2.3.3.2.cmml" xref="A1.E9.m1.2.2.2.2.3.3.2">𝑓</ci><apply id="A1.E9.m1.2.2.2.2.3.3.3.cmml" xref="A1.E9.m1.2.2.2.2.3.3.3"><csymbol cd="ambiguous" id="A1.E9.m1.2.2.2.2.3.3.3.1.cmml" xref="A1.E9.m1.2.2.2.2.3.3.3">superscript</csymbol><ci id="A1.E9.m1.2.2.2.2.3.3.3.2.cmml" xref="A1.E9.m1.2.2.2.2.3.3.3.2">𝜃</ci><ci id="A1.E9.m1.2.2.2.2.3.3.3.3.cmml" xref="A1.E9.m1.2.2.2.2.3.3.3.3">′</ci></apply></apply></apply><apply id="A1.E9.m1.2.2.2.2.1.1.1.cmml" xref="A1.E9.m1.2.2.2.2.1.1"><csymbol cd="ambiguous" id="A1.E9.m1.2.2.2.2.1.1.1.1.cmml" xref="A1.E9.m1.2.2.2.2.1.1">subscript</csymbol><ci id="A1.E9.m1.2.2.2.2.1.1.1.2.cmml" xref="A1.E9.m1.2.2.2.2.1.1.1.2">𝑘</ci><ci id="A1.E9.m1.2.2.2.2.1.1.1.3.cmml" xref="A1.E9.m1.2.2.2.2.1.1.1.3">𝑜</ci></apply></apply></apply><apply id="A1.E9.m1.3.3.3.cmml" xref="A1.E9.m1.3.3.3"><times id="A1.E9.m1.3.3.3.2.cmml" xref="A1.E9.m1.3.3.3.2"></times><apply id="A1.E9.m1.3.3.3.3.cmml" xref="A1.E9.m1.3.3.3.3"><csymbol cd="ambiguous" id="A1.E9.m1.3.3.3.3.1.cmml" xref="A1.E9.m1.3.3.3.3">subscript</csymbol><ci id="A1.E9.m1.3.3.3.3.2.cmml" xref="A1.E9.m1.3.3.3.3.2">𝑝</ci><apply id="A1.E9.m1.3.3.3.3.3.cmml" xref="A1.E9.m1.3.3.3.3.3"><csymbol cd="ambiguous" id="A1.E9.m1.3.3.3.3.3.1.cmml" xref="A1.E9.m1.3.3.3.3.3">subscript</csymbol><ci id="A1.E9.m1.3.3.3.3.3.2.cmml" xref="A1.E9.m1.3.3.3.3.3.2">𝑓</ci><apply id="A1.E9.m1.3.3.3.3.3.3.cmml" xref="A1.E9.m1.3.3.3.3.3.3"><csymbol cd="ambiguous" id="A1.E9.m1.3.3.3.3.3.3.1.cmml" xref="A1.E9.m1.3.3.3.3.3.3">superscript</csymbol><ci id="A1.E9.m1.3.3.3.3.3.3.2.cmml" xref="A1.E9.m1.3.3.3.3.3.3.2">𝜃</ci><ci id="A1.E9.m1.3.3.3.3.3.3.3.cmml" xref="A1.E9.m1.3.3.3.3.3.3.3">𝑚</ci></apply></apply></apply><apply id="A1.E9.m1.3.3.3.1.1.1.cmml" xref="A1.E9.m1.3.3.3.1.1"><csymbol cd="ambiguous" id="A1.E9.m1.3.3.3.1.1.1.1.cmml" xref="A1.E9.m1.3.3.3.1.1">subscript</csymbol><ci id="A1.E9.m1.3.3.3.1.1.1.2.cmml" xref="A1.E9.m1.3.3.3.1.1.1.2">𝑘</ci><ci id="A1.E9.m1.3.3.3.1.1.1.3.cmml" xref="A1.E9.m1.3.3.3.1.1.1.3">𝑜</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E9.m1.3c">\mathrm{CM=}\frac{p_{f_{\theta^{m}}}(k_{o})-p_{f_{\theta^{\prime}}}(k_{o})}{p_{f_{\theta^{m}}}(k_{o})}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(9)</span></td>
</tr></tbody>
</table>
<p id="A1.I5.i2.p1.1" class="ltx_p"><math id="A1.I5.i2.p1.1.m1.1" class="ltx_Math" alttext="\theta^{m}" display="inline"><semantics id="A1.I5.i2.p1.1.m1.1a"><msup id="A1.I5.i2.p1.1.m1.1.1" xref="A1.I5.i2.p1.1.m1.1.1.cmml"><mi id="A1.I5.i2.p1.1.m1.1.1.2" xref="A1.I5.i2.p1.1.m1.1.1.2.cmml">θ</mi><mi id="A1.I5.i2.p1.1.m1.1.1.3" xref="A1.I5.i2.p1.1.m1.1.1.3.cmml">m</mi></msup><annotation-xml encoding="MathML-Content" id="A1.I5.i2.p1.1.m1.1b"><apply id="A1.I5.i2.p1.1.m1.1.1.cmml" xref="A1.I5.i2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A1.I5.i2.p1.1.m1.1.1.1.cmml" xref="A1.I5.i2.p1.1.m1.1.1">superscript</csymbol><ci id="A1.I5.i2.p1.1.m1.1.1.2.cmml" xref="A1.I5.i2.p1.1.m1.1.1.2">𝜃</ci><ci id="A1.I5.i2.p1.1.m1.1.1.3.cmml" xref="A1.I5.i2.p1.1.m1.1.1.3">𝑚</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I5.i2.p1.1.m1.1c">\theta^{m}</annotation></semantics></math> is the intermediate model parameters after edit 1.</p>
</div>
</li>
</ul>
</div>
</section>
</section>
<section id="A1.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.2 </span>Locality</h3>

<section id="A1.SS2.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Single Edit</h5>

<div id="A1.SS2.SSS0.Px1.p1" class="ltx_para">
<p id="A1.SS2.SSS0.Px1.p1.1" class="ltx_p">In the single edit scenario for locality, we adopt the methods proposed by <cite class="ltx_cite ltx_citemacro_citep">(Yao et al., <a href="#bib.bib47" title="" class="ltx_ref">2023b</a>)</cite>, including:</p>
<ul id="A1.I6" class="ltx_itemize">
<li id="A1.I6.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I6.i1.p1" class="ltx_para">
<p id="A1.I6.i1.p1.1" class="ltx_p"><span id="A1.I6.i1.p1.1.1" class="ltx_text ltx_font_bold">Other Attribution</span>
The modified <span id="A1.I6.i1.p1.1.2" class="ltx_text ltx_font_bold">CounterFact</span> dataset is applied to test whether the non-target attributes of the edited subjects remained the same. For example, if we reset <span id="A1.I6.i1.p1.1.3" class="ltx_text ltx_font_italic">Lionel Messi</span> as a basketball player, his nationality should stay the same.</p>
</div>
</li>
<li id="A1.I6.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I6.i2.p1" class="ltx_para">
<p id="A1.I6.i2.p1.1" class="ltx_p"><span id="A1.I6.i2.p1.1.1" class="ltx_text ltx_font_bold">Distract Neighbor</span>
Pretend model editing by modifying the neighborhood prompt in the <span id="A1.I6.i2.p1.1.2" class="ltx_text ltx_font_bold">CounterFact</span>. For example, if the original prompt is "Windows 11 is a product of __", the modified prompt would be "Windows 11 is a product of Google. Office 365, developed by __". It testifies whether the model prediction would be "distracted" by the revised prompt.</p>
</div>
</li>
<li id="A1.I6.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I6.i3.p1" class="ltx_para">
<p id="A1.I6.i3.p1.1" class="ltx_p"><span id="A1.I6.i3.p1.1.1" class="ltx_text ltx_font_bold">Other Task</span>
The edited model is tested on the multiple-choice QA task <span id="A1.I6.i3.p1.1.2" class="ltx_text ltx_font_bold">Physical Interaction QA</span>(PIQA, <cite class="ltx_cite ltx_citemacro_citet">Bisk et al. (<a href="#bib.bib2" title="" class="ltx_ref">2020</a>)</cite>) and the performance is evaluated by accuracy.</p>
</div>
</li>
</ul>
</div>
</section>
<section id="A1.SS2.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Multiple Edits</h5>

<div id="A1.SS2.SSS0.Px2.p1" class="ltx_para">
<p id="A1.SS2.SSS0.Px2.p1.1" class="ltx_p">We also test the model’s locality in the multiple edits scenario adopting the methods and evaluations from <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib26" title="" class="ltx_ref">2024b</a>)</cite>. The settings consist of:</p>
</div>
<div id="A1.SS2.SSS0.Px2.p2" class="ltx_para">
<ul id="A1.I7" class="ltx_itemize">
<li id="A1.I7.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I7.i1.p1" class="ltx_para">
<p id="A1.I7.i1.p1.4" class="ltx_p"><span id="A1.I7.i1.p1.4.1" class="ltx_text ltx_font_bold">Round Edit:</span> This edits the knowledge triplet back-and-forth, for example:

<br class="ltx_break"><span id="A1.I7.i1.p1.4.2" class="ltx_text ltx_font_bold">edit 1:</span> (<math id="A1.I7.i1.p1.1.m1.3" class="ltx_Math" alttext="s,r,o_{1}" display="inline"><semantics id="A1.I7.i1.p1.1.m1.3a"><mrow id="A1.I7.i1.p1.1.m1.3.3.1" xref="A1.I7.i1.p1.1.m1.3.3.2.cmml"><mi id="A1.I7.i1.p1.1.m1.1.1" xref="A1.I7.i1.p1.1.m1.1.1.cmml">s</mi><mo id="A1.I7.i1.p1.1.m1.3.3.1.2" xref="A1.I7.i1.p1.1.m1.3.3.2.cmml">,</mo><mi id="A1.I7.i1.p1.1.m1.2.2" xref="A1.I7.i1.p1.1.m1.2.2.cmml">r</mi><mo id="A1.I7.i1.p1.1.m1.3.3.1.3" xref="A1.I7.i1.p1.1.m1.3.3.2.cmml">,</mo><msub id="A1.I7.i1.p1.1.m1.3.3.1.1" xref="A1.I7.i1.p1.1.m1.3.3.1.1.cmml"><mi id="A1.I7.i1.p1.1.m1.3.3.1.1.2" xref="A1.I7.i1.p1.1.m1.3.3.1.1.2.cmml">o</mi><mn id="A1.I7.i1.p1.1.m1.3.3.1.1.3" xref="A1.I7.i1.p1.1.m1.3.3.1.1.3.cmml">1</mn></msub></mrow><annotation-xml encoding="MathML-Content" id="A1.I7.i1.p1.1.m1.3b"><list id="A1.I7.i1.p1.1.m1.3.3.2.cmml" xref="A1.I7.i1.p1.1.m1.3.3.1"><ci id="A1.I7.i1.p1.1.m1.1.1.cmml" xref="A1.I7.i1.p1.1.m1.1.1">𝑠</ci><ci id="A1.I7.i1.p1.1.m1.2.2.cmml" xref="A1.I7.i1.p1.1.m1.2.2">𝑟</ci><apply id="A1.I7.i1.p1.1.m1.3.3.1.1.cmml" xref="A1.I7.i1.p1.1.m1.3.3.1.1"><csymbol cd="ambiguous" id="A1.I7.i1.p1.1.m1.3.3.1.1.1.cmml" xref="A1.I7.i1.p1.1.m1.3.3.1.1">subscript</csymbol><ci id="A1.I7.i1.p1.1.m1.3.3.1.1.2.cmml" xref="A1.I7.i1.p1.1.m1.3.3.1.1.2">𝑜</ci><cn type="integer" id="A1.I7.i1.p1.1.m1.3.3.1.1.3.cmml" xref="A1.I7.i1.p1.1.m1.3.3.1.1.3">1</cn></apply></list></annotation-xml><annotation encoding="application/x-tex" id="A1.I7.i1.p1.1.m1.3c">s,r,o_{1}</annotation></semantics></math>→<math id="A1.I7.i1.p1.2.m2.1" class="ltx_math_unparsed" alttext="o*" display="inline"><semantics id="A1.I7.i1.p1.2.m2.1a"><mrow id="A1.I7.i1.p1.2.m2.1b"><mi id="A1.I7.i1.p1.2.m2.1.1">o</mi><mo lspace="0.222em" id="A1.I7.i1.p1.2.m2.1.2">∗</mo></mrow><annotation encoding="application/x-tex" id="A1.I7.i1.p1.2.m2.1c">o*</annotation></semantics></math>)

<br class="ltx_break"><span id="A1.I7.i1.p1.4.3" class="ltx_text ltx_font_bold">edit 2:</span> (<math id="A1.I7.i1.p1.3.m3.3" class="ltx_math_unparsed" alttext="s,r,o*" display="inline"><semantics id="A1.I7.i1.p1.3.m3.3a"><mrow id="A1.I7.i1.p1.3.m3.3b"><mi id="A1.I7.i1.p1.3.m3.1.1">s</mi><mo id="A1.I7.i1.p1.3.m3.3.4">,</mo><mi id="A1.I7.i1.p1.3.m3.2.2">r</mi><mo id="A1.I7.i1.p1.3.m3.3.5">,</mo><mi id="A1.I7.i1.p1.3.m3.3.3">o</mi><mo lspace="0.222em" id="A1.I7.i1.p1.3.m3.3.6">∗</mo></mrow><annotation encoding="application/x-tex" id="A1.I7.i1.p1.3.m3.3c">s,r,o*</annotation></semantics></math>→<math id="A1.I7.i1.p1.4.m4.1" class="ltx_Math" alttext="o_{1}" display="inline"><semantics id="A1.I7.i1.p1.4.m4.1a"><msub id="A1.I7.i1.p1.4.m4.1.1" xref="A1.I7.i1.p1.4.m4.1.1.cmml"><mi id="A1.I7.i1.p1.4.m4.1.1.2" xref="A1.I7.i1.p1.4.m4.1.1.2.cmml">o</mi><mn id="A1.I7.i1.p1.4.m4.1.1.3" xref="A1.I7.i1.p1.4.m4.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A1.I7.i1.p1.4.m4.1b"><apply id="A1.I7.i1.p1.4.m4.1.1.cmml" xref="A1.I7.i1.p1.4.m4.1.1"><csymbol cd="ambiguous" id="A1.I7.i1.p1.4.m4.1.1.1.cmml" xref="A1.I7.i1.p1.4.m4.1.1">subscript</csymbol><ci id="A1.I7.i1.p1.4.m4.1.1.2.cmml" xref="A1.I7.i1.p1.4.m4.1.1.2">𝑜</ci><cn type="integer" id="A1.I7.i1.p1.4.m4.1.1.3.cmml" xref="A1.I7.i1.p1.4.m4.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I7.i1.p1.4.m4.1c">o_{1}</annotation></semantics></math>)</p>
</div>
</li>
</ul>
<p id="A1.SS2.SSS0.Px2.p2.1" class="ltx_p">the evaluation metrics include:</p>
<ul id="A1.I8" class="ltx_itemize">
<li id="A1.I8.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I8.i1.p1" class="ltx_para">
<p id="A1.I8.i1.p1.1" class="ltx_p"><span id="A1.I8.i1.p1.1.1" class="ltx_text ltx_font_bold">Distortion (D) <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib26" title="" class="ltx_ref">2024b</a>)</cite>:</span></p>
<table id="A1.E10" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.E10.m1.6" class="ltx_Math" alttext="D=JS\left(p_{f_{\theta}}(\text{Obj}\mid(s,r)),p_{f_{\theta^{\prime}}}(\text{Obj}\mid(s,r))\right)" display="block"><semantics id="A1.E10.m1.6a"><mrow id="A1.E10.m1.6.6" xref="A1.E10.m1.6.6.cmml"><mi id="A1.E10.m1.6.6.4" xref="A1.E10.m1.6.6.4.cmml">D</mi><mo id="A1.E10.m1.6.6.3" xref="A1.E10.m1.6.6.3.cmml">=</mo><mrow id="A1.E10.m1.6.6.2" xref="A1.E10.m1.6.6.2.cmml"><mi id="A1.E10.m1.6.6.2.4" xref="A1.E10.m1.6.6.2.4.cmml">J</mi><mo lspace="0em" rspace="0em" id="A1.E10.m1.6.6.2.3" xref="A1.E10.m1.6.6.2.3.cmml">​</mo><mi id="A1.E10.m1.6.6.2.5" xref="A1.E10.m1.6.6.2.5.cmml">S</mi><mo lspace="0em" rspace="0em" id="A1.E10.m1.6.6.2.3a" xref="A1.E10.m1.6.6.2.3.cmml">​</mo><mrow id="A1.E10.m1.6.6.2.2.2" xref="A1.E10.m1.6.6.2.2.3.cmml"><mo id="A1.E10.m1.6.6.2.2.2.3" xref="A1.E10.m1.6.6.2.2.3.cmml">(</mo><mrow id="A1.E10.m1.5.5.1.1.1.1" xref="A1.E10.m1.5.5.1.1.1.1.cmml"><msub id="A1.E10.m1.5.5.1.1.1.1.3" xref="A1.E10.m1.5.5.1.1.1.1.3.cmml"><mi id="A1.E10.m1.5.5.1.1.1.1.3.2" xref="A1.E10.m1.5.5.1.1.1.1.3.2.cmml">p</mi><msub id="A1.E10.m1.5.5.1.1.1.1.3.3" xref="A1.E10.m1.5.5.1.1.1.1.3.3.cmml"><mi id="A1.E10.m1.5.5.1.1.1.1.3.3.2" xref="A1.E10.m1.5.5.1.1.1.1.3.3.2.cmml">f</mi><mi id="A1.E10.m1.5.5.1.1.1.1.3.3.3" xref="A1.E10.m1.5.5.1.1.1.1.3.3.3.cmml">θ</mi></msub></msub><mo lspace="0em" rspace="0em" id="A1.E10.m1.5.5.1.1.1.1.2" xref="A1.E10.m1.5.5.1.1.1.1.2.cmml">​</mo><mrow id="A1.E10.m1.5.5.1.1.1.1.1.1" xref="A1.E10.m1.5.5.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.E10.m1.5.5.1.1.1.1.1.1.2" xref="A1.E10.m1.5.5.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A1.E10.m1.5.5.1.1.1.1.1.1.1" xref="A1.E10.m1.5.5.1.1.1.1.1.1.1.cmml"><mtext id="A1.E10.m1.5.5.1.1.1.1.1.1.1.2" xref="A1.E10.m1.5.5.1.1.1.1.1.1.1.2a.cmml">Obj</mtext><mo id="A1.E10.m1.5.5.1.1.1.1.1.1.1.1" xref="A1.E10.m1.5.5.1.1.1.1.1.1.1.1.cmml">∣</mo><mrow id="A1.E10.m1.5.5.1.1.1.1.1.1.1.3.2" xref="A1.E10.m1.5.5.1.1.1.1.1.1.1.3.1.cmml"><mo stretchy="false" id="A1.E10.m1.5.5.1.1.1.1.1.1.1.3.2.1" xref="A1.E10.m1.5.5.1.1.1.1.1.1.1.3.1.cmml">(</mo><mi id="A1.E10.m1.1.1" xref="A1.E10.m1.1.1.cmml">s</mi><mo id="A1.E10.m1.5.5.1.1.1.1.1.1.1.3.2.2" xref="A1.E10.m1.5.5.1.1.1.1.1.1.1.3.1.cmml">,</mo><mi id="A1.E10.m1.2.2" xref="A1.E10.m1.2.2.cmml">r</mi><mo stretchy="false" id="A1.E10.m1.5.5.1.1.1.1.1.1.1.3.2.3" xref="A1.E10.m1.5.5.1.1.1.1.1.1.1.3.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="A1.E10.m1.5.5.1.1.1.1.1.1.3" xref="A1.E10.m1.5.5.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="A1.E10.m1.6.6.2.2.2.4" xref="A1.E10.m1.6.6.2.2.3.cmml">,</mo><mrow id="A1.E10.m1.6.6.2.2.2.2" xref="A1.E10.m1.6.6.2.2.2.2.cmml"><msub id="A1.E10.m1.6.6.2.2.2.2.3" xref="A1.E10.m1.6.6.2.2.2.2.3.cmml"><mi id="A1.E10.m1.6.6.2.2.2.2.3.2" xref="A1.E10.m1.6.6.2.2.2.2.3.2.cmml">p</mi><msub id="A1.E10.m1.6.6.2.2.2.2.3.3" xref="A1.E10.m1.6.6.2.2.2.2.3.3.cmml"><mi id="A1.E10.m1.6.6.2.2.2.2.3.3.2" xref="A1.E10.m1.6.6.2.2.2.2.3.3.2.cmml">f</mi><msup id="A1.E10.m1.6.6.2.2.2.2.3.3.3" xref="A1.E10.m1.6.6.2.2.2.2.3.3.3.cmml"><mi id="A1.E10.m1.6.6.2.2.2.2.3.3.3.2" xref="A1.E10.m1.6.6.2.2.2.2.3.3.3.2.cmml">θ</mi><mo id="A1.E10.m1.6.6.2.2.2.2.3.3.3.3" xref="A1.E10.m1.6.6.2.2.2.2.3.3.3.3.cmml">′</mo></msup></msub></msub><mo lspace="0em" rspace="0em" id="A1.E10.m1.6.6.2.2.2.2.2" xref="A1.E10.m1.6.6.2.2.2.2.2.cmml">​</mo><mrow id="A1.E10.m1.6.6.2.2.2.2.1.1" xref="A1.E10.m1.6.6.2.2.2.2.1.1.1.cmml"><mo stretchy="false" id="A1.E10.m1.6.6.2.2.2.2.1.1.2" xref="A1.E10.m1.6.6.2.2.2.2.1.1.1.cmml">(</mo><mrow id="A1.E10.m1.6.6.2.2.2.2.1.1.1" xref="A1.E10.m1.6.6.2.2.2.2.1.1.1.cmml"><mtext id="A1.E10.m1.6.6.2.2.2.2.1.1.1.2" xref="A1.E10.m1.6.6.2.2.2.2.1.1.1.2a.cmml">Obj</mtext><mo id="A1.E10.m1.6.6.2.2.2.2.1.1.1.1" xref="A1.E10.m1.6.6.2.2.2.2.1.1.1.1.cmml">∣</mo><mrow id="A1.E10.m1.6.6.2.2.2.2.1.1.1.3.2" xref="A1.E10.m1.6.6.2.2.2.2.1.1.1.3.1.cmml"><mo stretchy="false" id="A1.E10.m1.6.6.2.2.2.2.1.1.1.3.2.1" xref="A1.E10.m1.6.6.2.2.2.2.1.1.1.3.1.cmml">(</mo><mi id="A1.E10.m1.3.3" xref="A1.E10.m1.3.3.cmml">s</mi><mo id="A1.E10.m1.6.6.2.2.2.2.1.1.1.3.2.2" xref="A1.E10.m1.6.6.2.2.2.2.1.1.1.3.1.cmml">,</mo><mi id="A1.E10.m1.4.4" xref="A1.E10.m1.4.4.cmml">r</mi><mo stretchy="false" id="A1.E10.m1.6.6.2.2.2.2.1.1.1.3.2.3" xref="A1.E10.m1.6.6.2.2.2.2.1.1.1.3.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="A1.E10.m1.6.6.2.2.2.2.1.1.3" xref="A1.E10.m1.6.6.2.2.2.2.1.1.1.cmml">)</mo></mrow></mrow><mo id="A1.E10.m1.6.6.2.2.2.5" xref="A1.E10.m1.6.6.2.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E10.m1.6b"><apply id="A1.E10.m1.6.6.cmml" xref="A1.E10.m1.6.6"><eq id="A1.E10.m1.6.6.3.cmml" xref="A1.E10.m1.6.6.3"></eq><ci id="A1.E10.m1.6.6.4.cmml" xref="A1.E10.m1.6.6.4">𝐷</ci><apply id="A1.E10.m1.6.6.2.cmml" xref="A1.E10.m1.6.6.2"><times id="A1.E10.m1.6.6.2.3.cmml" xref="A1.E10.m1.6.6.2.3"></times><ci id="A1.E10.m1.6.6.2.4.cmml" xref="A1.E10.m1.6.6.2.4">𝐽</ci><ci id="A1.E10.m1.6.6.2.5.cmml" xref="A1.E10.m1.6.6.2.5">𝑆</ci><interval closure="open" id="A1.E10.m1.6.6.2.2.3.cmml" xref="A1.E10.m1.6.6.2.2.2"><apply id="A1.E10.m1.5.5.1.1.1.1.cmml" xref="A1.E10.m1.5.5.1.1.1.1"><times id="A1.E10.m1.5.5.1.1.1.1.2.cmml" xref="A1.E10.m1.5.5.1.1.1.1.2"></times><apply id="A1.E10.m1.5.5.1.1.1.1.3.cmml" xref="A1.E10.m1.5.5.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.E10.m1.5.5.1.1.1.1.3.1.cmml" xref="A1.E10.m1.5.5.1.1.1.1.3">subscript</csymbol><ci id="A1.E10.m1.5.5.1.1.1.1.3.2.cmml" xref="A1.E10.m1.5.5.1.1.1.1.3.2">𝑝</ci><apply id="A1.E10.m1.5.5.1.1.1.1.3.3.cmml" xref="A1.E10.m1.5.5.1.1.1.1.3.3"><csymbol cd="ambiguous" id="A1.E10.m1.5.5.1.1.1.1.3.3.1.cmml" xref="A1.E10.m1.5.5.1.1.1.1.3.3">subscript</csymbol><ci id="A1.E10.m1.5.5.1.1.1.1.3.3.2.cmml" xref="A1.E10.m1.5.5.1.1.1.1.3.3.2">𝑓</ci><ci id="A1.E10.m1.5.5.1.1.1.1.3.3.3.cmml" xref="A1.E10.m1.5.5.1.1.1.1.3.3.3">𝜃</ci></apply></apply><apply id="A1.E10.m1.5.5.1.1.1.1.1.1.1.cmml" xref="A1.E10.m1.5.5.1.1.1.1.1.1"><csymbol cd="latexml" id="A1.E10.m1.5.5.1.1.1.1.1.1.1.1.cmml" xref="A1.E10.m1.5.5.1.1.1.1.1.1.1.1">conditional</csymbol><ci id="A1.E10.m1.5.5.1.1.1.1.1.1.1.2a.cmml" xref="A1.E10.m1.5.5.1.1.1.1.1.1.1.2"><mtext id="A1.E10.m1.5.5.1.1.1.1.1.1.1.2.cmml" xref="A1.E10.m1.5.5.1.1.1.1.1.1.1.2">Obj</mtext></ci><interval closure="open" id="A1.E10.m1.5.5.1.1.1.1.1.1.1.3.1.cmml" xref="A1.E10.m1.5.5.1.1.1.1.1.1.1.3.2"><ci id="A1.E10.m1.1.1.cmml" xref="A1.E10.m1.1.1">𝑠</ci><ci id="A1.E10.m1.2.2.cmml" xref="A1.E10.m1.2.2">𝑟</ci></interval></apply></apply><apply id="A1.E10.m1.6.6.2.2.2.2.cmml" xref="A1.E10.m1.6.6.2.2.2.2"><times id="A1.E10.m1.6.6.2.2.2.2.2.cmml" xref="A1.E10.m1.6.6.2.2.2.2.2"></times><apply id="A1.E10.m1.6.6.2.2.2.2.3.cmml" xref="A1.E10.m1.6.6.2.2.2.2.3"><csymbol cd="ambiguous" id="A1.E10.m1.6.6.2.2.2.2.3.1.cmml" xref="A1.E10.m1.6.6.2.2.2.2.3">subscript</csymbol><ci id="A1.E10.m1.6.6.2.2.2.2.3.2.cmml" xref="A1.E10.m1.6.6.2.2.2.2.3.2">𝑝</ci><apply id="A1.E10.m1.6.6.2.2.2.2.3.3.cmml" xref="A1.E10.m1.6.6.2.2.2.2.3.3"><csymbol cd="ambiguous" id="A1.E10.m1.6.6.2.2.2.2.3.3.1.cmml" xref="A1.E10.m1.6.6.2.2.2.2.3.3">subscript</csymbol><ci id="A1.E10.m1.6.6.2.2.2.2.3.3.2.cmml" xref="A1.E10.m1.6.6.2.2.2.2.3.3.2">𝑓</ci><apply id="A1.E10.m1.6.6.2.2.2.2.3.3.3.cmml" xref="A1.E10.m1.6.6.2.2.2.2.3.3.3"><csymbol cd="ambiguous" id="A1.E10.m1.6.6.2.2.2.2.3.3.3.1.cmml" xref="A1.E10.m1.6.6.2.2.2.2.3.3.3">superscript</csymbol><ci id="A1.E10.m1.6.6.2.2.2.2.3.3.3.2.cmml" xref="A1.E10.m1.6.6.2.2.2.2.3.3.3.2">𝜃</ci><ci id="A1.E10.m1.6.6.2.2.2.2.3.3.3.3.cmml" xref="A1.E10.m1.6.6.2.2.2.2.3.3.3.3">′</ci></apply></apply></apply><apply id="A1.E10.m1.6.6.2.2.2.2.1.1.1.cmml" xref="A1.E10.m1.6.6.2.2.2.2.1.1"><csymbol cd="latexml" id="A1.E10.m1.6.6.2.2.2.2.1.1.1.1.cmml" xref="A1.E10.m1.6.6.2.2.2.2.1.1.1.1">conditional</csymbol><ci id="A1.E10.m1.6.6.2.2.2.2.1.1.1.2a.cmml" xref="A1.E10.m1.6.6.2.2.2.2.1.1.1.2"><mtext id="A1.E10.m1.6.6.2.2.2.2.1.1.1.2.cmml" xref="A1.E10.m1.6.6.2.2.2.2.1.1.1.2">Obj</mtext></ci><interval closure="open" id="A1.E10.m1.6.6.2.2.2.2.1.1.1.3.1.cmml" xref="A1.E10.m1.6.6.2.2.2.2.1.1.1.3.2"><ci id="A1.E10.m1.3.3.cmml" xref="A1.E10.m1.3.3">𝑠</ci><ci id="A1.E10.m1.4.4.cmml" xref="A1.E10.m1.4.4">𝑟</ci></interval></apply></apply></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E10.m1.6c">D=JS\left(p_{f_{\theta}}(\text{Obj}\mid(s,r)),p_{f_{\theta^{\prime}}}(\text{Obj}\mid(s,r))\right)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(10)</span></td>
</tr></tbody>
</table>
<p id="A1.I8.i1.p1.2" class="ltx_p">estimates the JS divergence of the objects distribution before and after edit.</p>
</div>
</li>
<li id="A1.I8.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I8.i2.p1" class="ltx_para">
<p id="A1.I8.i2.p1.1" class="ltx_p"><span id="A1.I8.i2.p1.1.1" class="ltx_text ltx_font_bold">Ignore Rate (IR) <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib26" title="" class="ltx_ref">2024b</a>)</cite>:</span></p>
<table id="A2.EGx1" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A1.E11"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td colspan="2" class="ltx_td ltx_align_center ltx_eqn_cell"><math id="A1.E11.m1.32" class="ltx_Math" alttext="\displaystyle\begin{split}\mathrm{IR=}\frac{1}{\left|\mathrm{Obj}\right|-1}\sum_{\begin{subarray}{c}o\in\mathrm{Obj}\setminus\{o1\}\end{subarray}}&amp;\mathbbm{1}\{p_{f_{\theta}}(o\mid(s,r))&gt;\\
&amp;p_{f_{\theta}^{\prime}}(o\mid(s,r))\}\end{split}" display="inline"><semantics id="A1.E11.m1.32a"><mtable columnspacing="0pt" rowspacing="0pt" id="A1.E11.m1.31.31a" xref="A1.E11.m1.32.32.1.cmml"><mtr id="A1.E11.m1.31.31aa" xref="A1.E11.m1.32.32.1.cmml"><mtd class="ltx_align_right" columnalign="right" id="A1.E11.m1.31.31ab" xref="A1.E11.m1.32.32.1.cmml"><mrow id="A1.E11.m1.5.5.5.5.5a" xref="A1.E11.m1.32.32.1.cmml"><mi id="A1.E11.m1.2.2.2.2.2.2" xref="A1.E11.m1.2.2.2.2.2.2.cmml">IR</mi><mo id="A1.E11.m1.3.3.3.3.3.3" xref="A1.E11.m1.3.3.3.3.3.3.cmml">=</mo><mrow id="A1.E11.m1.5.5.5.5.5a.6" xref="A1.E11.m1.32.32.1.cmml"><mstyle displaystyle="true" id="A1.E11.m1.4.4.4.4.4.4" xref="A1.E11.m1.4.4.4.4.4.4.cmml"><mfrac id="A1.E11.m1.4.4.4.4.4.4a" xref="A1.E11.m1.4.4.4.4.4.4.cmml"><mn id="A1.E11.m1.4.4.4.4.4.4.3" xref="A1.E11.m1.4.4.4.4.4.4.3.cmml">1</mn><mrow id="A1.E11.m1.4.4.4.4.4.4.1" xref="A1.E11.m1.4.4.4.4.4.4.1.cmml"><mrow id="A1.E11.m1.4.4.4.4.4.4.1.3.2" xref="A1.E11.m1.4.4.4.4.4.4.1.3.1.cmml"><mo id="A1.E11.m1.4.4.4.4.4.4.1.3.2.1" xref="A1.E11.m1.4.4.4.4.4.4.1.3.1.1.cmml">|</mo><mi id="A1.E11.m1.4.4.4.4.4.4.1.1" xref="A1.E11.m1.4.4.4.4.4.4.1.1.cmml">Obj</mi><mo id="A1.E11.m1.4.4.4.4.4.4.1.3.2.2" xref="A1.E11.m1.4.4.4.4.4.4.1.3.1.1.cmml">|</mo></mrow><mo id="A1.E11.m1.4.4.4.4.4.4.1.2" xref="A1.E11.m1.4.4.4.4.4.4.1.2.cmml">−</mo><mn id="A1.E11.m1.4.4.4.4.4.4.1.4" xref="A1.E11.m1.4.4.4.4.4.4.1.4.cmml">1</mn></mrow></mfrac></mstyle><mo lspace="0em" rspace="0em" id="A1.E11.m1.5.5.5.5.5a.6.1" xref="A1.E11.m1.32.32.1.cmml">​</mo><mstyle displaystyle="true" id="A1.E11.m1.5.5.5.5.5a.6.2" xref="A1.E11.m1.32.32.1.cmml"><munder id="A1.E11.m1.5.5.5.5.5a.6.2a" xref="A1.E11.m1.32.32.1.cmml"><mo movablelimits="false" id="A1.E11.m1.5.5.5.5.5.5" xref="A1.E11.m1.5.5.5.5.5.5.cmml">∑</mo><mtable id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1" xref="A1.E11.m1.1.1.1.1.1.1.1a.2.cmml"><mtr id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1a" xref="A1.E11.m1.1.1.1.1.1.1.1a.2.cmml"><mtd id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1b" xref="A1.E11.m1.1.1.1.1.1.1.1a.2.cmml"><mrow id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml">o</mi><mo id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml">∈</mo><mrow id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml">Obj</mi><mo id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml">∖</mo><mrow id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml">{</mo><mrow id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml">o</mi><mo lspace="0em" rspace="0em" id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">​</mo><mn id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml">}</mo></mrow></mrow></mrow></mtd></mtr></mtable></munder></mstyle></mrow></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="A1.E11.m1.31.31ac" xref="A1.E11.m1.32.32.1.cmml"><mrow id="A1.E11.m1.19.19.19.19.14a" xref="A1.E11.m1.32.32.1.cmml"><mn id="A1.E11.m1.6.6.6.6.1.1" xref="A1.E11.m1.6.6.6.6.1.1.cmml">𝟙</mn><mrow id="A1.E11.m1.19.19.19.19.14a.15" xref="A1.E11.m1.32.32.1.cmml"><mo stretchy="false" id="A1.E11.m1.7.7.7.7.2.2" xref="A1.E11.m1.32.32.1.cmml">{</mo><msub id="A1.E11.m1.19.19.19.19.14a.15.1" xref="A1.E11.m1.32.32.1.cmml"><mi id="A1.E11.m1.8.8.8.8.3.3" xref="A1.E11.m1.8.8.8.8.3.3.cmml">p</mi><msub id="A1.E11.m1.9.9.9.9.4.4.1" xref="A1.E11.m1.9.9.9.9.4.4.1.cmml"><mi id="A1.E11.m1.9.9.9.9.4.4.1.2" xref="A1.E11.m1.9.9.9.9.4.4.1.2.cmml">f</mi><mi id="A1.E11.m1.9.9.9.9.4.4.1.3" xref="A1.E11.m1.9.9.9.9.4.4.1.3.cmml">θ</mi></msub></msub><mrow id="A1.E11.m1.19.19.19.19.14a.15.2" xref="A1.E11.m1.32.32.1.cmml"><mo stretchy="false" id="A1.E11.m1.10.10.10.10.5.5" xref="A1.E11.m1.32.32.1.cmml">(</mo><mi id="A1.E11.m1.11.11.11.11.6.6" xref="A1.E11.m1.11.11.11.11.6.6.cmml">o</mi><mo lspace="0em" rspace="0.167em" id="A1.E11.m1.12.12.12.12.7.7" xref="A1.E11.m1.12.12.12.12.7.7.cmml">∣</mo><mrow id="A1.E11.m1.19.19.19.19.14a.15.2.1" xref="A1.E11.m1.32.32.1.cmml"><mo stretchy="false" id="A1.E11.m1.13.13.13.13.8.8" xref="A1.E11.m1.32.32.1.cmml">(</mo><mi id="A1.E11.m1.14.14.14.14.9.9" xref="A1.E11.m1.14.14.14.14.9.9.cmml">s</mi><mo id="A1.E11.m1.15.15.15.15.10.10" xref="A1.E11.m1.32.32.1.cmml">,</mo><mi id="A1.E11.m1.16.16.16.16.11.11" xref="A1.E11.m1.16.16.16.16.11.11.cmml">r</mi><mo stretchy="false" id="A1.E11.m1.17.17.17.17.12.12" xref="A1.E11.m1.32.32.1.cmml">)</mo></mrow><mo stretchy="false" id="A1.E11.m1.18.18.18.18.13.13" xref="A1.E11.m1.32.32.1.cmml">)</mo></mrow><mo id="A1.E11.m1.19.19.19.19.14.14" xref="A1.E11.m1.19.19.19.19.14.14.cmml">&gt;</mo></mrow></mrow></mtd></mtr><mtr id="A1.E11.m1.31.31ad" xref="A1.E11.m1.32.32.1.cmml"><mtd id="A1.E11.m1.31.31ae" xref="A1.E11.m1.32.32.1.cmml"></mtd><mtd class="ltx_align_left" columnalign="left" id="A1.E11.m1.31.31af" xref="A1.E11.m1.32.32.1.cmml"><mrow id="A1.E11.m1.31.31.31.12.12a" xref="A1.E11.m1.32.32.1.cmml"><msub id="A1.E11.m1.31.31.31.12.12a.13" xref="A1.E11.m1.32.32.1.cmml"><mi id="A1.E11.m1.20.20.20.1.1.1" xref="A1.E11.m1.20.20.20.1.1.1.cmml">p</mi><msubsup id="A1.E11.m1.21.21.21.2.2.2.1" xref="A1.E11.m1.21.21.21.2.2.2.1.cmml"><mi id="A1.E11.m1.21.21.21.2.2.2.1.2.2" xref="A1.E11.m1.21.21.21.2.2.2.1.2.2.cmml">f</mi><mi id="A1.E11.m1.21.21.21.2.2.2.1.2.3" xref="A1.E11.m1.21.21.21.2.2.2.1.2.3.cmml">θ</mi><mo id="A1.E11.m1.21.21.21.2.2.2.1.3" xref="A1.E11.m1.21.21.21.2.2.2.1.3.cmml">′</mo></msubsup></msub><mrow id="A1.E11.m1.31.31.31.12.12a.14" xref="A1.E11.m1.32.32.1.cmml"><mo stretchy="false" id="A1.E11.m1.22.22.22.3.3.3" xref="A1.E11.m1.32.32.1.cmml">(</mo><mi id="A1.E11.m1.23.23.23.4.4.4" xref="A1.E11.m1.23.23.23.4.4.4.cmml">o</mi><mo lspace="0em" rspace="0.167em" id="A1.E11.m1.24.24.24.5.5.5" xref="A1.E11.m1.24.24.24.5.5.5.cmml">∣</mo><mrow id="A1.E11.m1.31.31.31.12.12a.14.1" xref="A1.E11.m1.32.32.1.cmml"><mo stretchy="false" id="A1.E11.m1.25.25.25.6.6.6" xref="A1.E11.m1.32.32.1.cmml">(</mo><mi id="A1.E11.m1.26.26.26.7.7.7" xref="A1.E11.m1.26.26.26.7.7.7.cmml">s</mi><mo id="A1.E11.m1.27.27.27.8.8.8" xref="A1.E11.m1.32.32.1.cmml">,</mo><mi id="A1.E11.m1.28.28.28.9.9.9" xref="A1.E11.m1.28.28.28.9.9.9.cmml">r</mi><mo stretchy="false" id="A1.E11.m1.29.29.29.10.10.10" xref="A1.E11.m1.32.32.1.cmml">)</mo></mrow><mo stretchy="false" id="A1.E11.m1.30.30.30.11.11.11" xref="A1.E11.m1.32.32.1.cmml">)</mo></mrow><mo stretchy="false" id="A1.E11.m1.31.31.31.12.12.12" xref="A1.E11.m1.32.32.1.cmml">}</mo></mrow></mtd></mtr></mtable><annotation-xml encoding="MathML-Content" id="A1.E11.m1.32b"><apply id="A1.E11.m1.32.32.1.cmml" xref="A1.E11.m1.31.31a"><eq id="A1.E11.m1.3.3.3.3.3.3.cmml" xref="A1.E11.m1.3.3.3.3.3.3"></eq><ci id="A1.E11.m1.2.2.2.2.2.2.cmml" xref="A1.E11.m1.2.2.2.2.2.2">IR</ci><apply id="A1.E11.m1.32.32.1.1.cmml" xref="A1.E11.m1.31.31a"><times id="A1.E11.m1.32.32.1.1.2.cmml" xref="A1.E11.m1.31.31a"></times><apply id="A1.E11.m1.4.4.4.4.4.4.cmml" xref="A1.E11.m1.4.4.4.4.4.4"><divide id="A1.E11.m1.4.4.4.4.4.4.2.cmml" xref="A1.E11.m1.4.4.4.4.4.4"></divide><cn type="integer" id="A1.E11.m1.4.4.4.4.4.4.3.cmml" xref="A1.E11.m1.4.4.4.4.4.4.3">1</cn><apply id="A1.E11.m1.4.4.4.4.4.4.1.cmml" xref="A1.E11.m1.4.4.4.4.4.4.1"><minus id="A1.E11.m1.4.4.4.4.4.4.1.2.cmml" xref="A1.E11.m1.4.4.4.4.4.4.1.2"></minus><apply id="A1.E11.m1.4.4.4.4.4.4.1.3.1.cmml" xref="A1.E11.m1.4.4.4.4.4.4.1.3.2"><abs id="A1.E11.m1.4.4.4.4.4.4.1.3.1.1.cmml" xref="A1.E11.m1.4.4.4.4.4.4.1.3.2.1"></abs><ci id="A1.E11.m1.4.4.4.4.4.4.1.1.cmml" xref="A1.E11.m1.4.4.4.4.4.4.1.1">Obj</ci></apply><cn type="integer" id="A1.E11.m1.4.4.4.4.4.4.1.4.cmml" xref="A1.E11.m1.4.4.4.4.4.4.1.4">1</cn></apply></apply><apply id="A1.E11.m1.32.32.1.1.1.cmml" xref="A1.E11.m1.31.31a"><apply id="A1.E11.m1.32.32.1.1.1.2.cmml" xref="A1.E11.m1.31.31a"><csymbol cd="ambiguous" id="A1.E11.m1.32.32.1.1.1.2.1.cmml" xref="A1.E11.m1.31.31a">subscript</csymbol><sum id="A1.E11.m1.5.5.5.5.5.5.cmml" xref="A1.E11.m1.5.5.5.5.5.5"></sum><list id="A1.E11.m1.1.1.1.1.1.1.1a.2.cmml" xref="A1.E11.m1.1.1.1.1.1.1.1.1.1.1"><matrix id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A1.E11.m1.1.1.1.1.1.1.1.1.1.1"><matrixrow id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1a.cmml" xref="A1.E11.m1.1.1.1.1.1.1.1.1.1.1"><apply id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1"><in id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2"></in><ci id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3">𝑜</ci><apply id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1"><setdiff id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2"></setdiff><ci id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3">Obj</ci><set id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1"><apply id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1"><times id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1"></times><ci id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2">𝑜</ci><cn type="integer" id="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="A1.E11.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3">1</cn></apply></set></apply></apply></matrixrow></matrix></list></apply><apply id="A1.E11.m1.32.32.1.1.1.1.cmml" xref="A1.E11.m1.31.31a"><times id="A1.E11.m1.32.32.1.1.1.1.2.cmml" xref="A1.E11.m1.31.31a"></times><cn type="integer" id="A1.E11.m1.6.6.6.6.1.1.cmml" xref="A1.E11.m1.6.6.6.6.1.1">1</cn><set id="A1.E11.m1.32.32.1.1.1.1.1.2.cmml" xref="A1.E11.m1.31.31a"><apply id="A1.E11.m1.32.32.1.1.1.1.1.1.1.cmml" xref="A1.E11.m1.31.31a"><gt id="A1.E11.m1.19.19.19.19.14.14.cmml" xref="A1.E11.m1.19.19.19.19.14.14"></gt><apply id="A1.E11.m1.32.32.1.1.1.1.1.1.1.1.cmml" xref="A1.E11.m1.31.31a"><times id="A1.E11.m1.32.32.1.1.1.1.1.1.1.1.2.cmml" xref="A1.E11.m1.31.31a"></times><apply id="A1.E11.m1.32.32.1.1.1.1.1.1.1.1.3.cmml" xref="A1.E11.m1.31.31a"><csymbol cd="ambiguous" id="A1.E11.m1.32.32.1.1.1.1.1.1.1.1.3.1.cmml" xref="A1.E11.m1.31.31a">subscript</csymbol><ci id="A1.E11.m1.8.8.8.8.3.3.cmml" xref="A1.E11.m1.8.8.8.8.3.3">𝑝</ci><apply id="A1.E11.m1.9.9.9.9.4.4.1.cmml" xref="A1.E11.m1.9.9.9.9.4.4.1"><csymbol cd="ambiguous" id="A1.E11.m1.9.9.9.9.4.4.1.1.cmml" xref="A1.E11.m1.9.9.9.9.4.4.1">subscript</csymbol><ci id="A1.E11.m1.9.9.9.9.4.4.1.2.cmml" xref="A1.E11.m1.9.9.9.9.4.4.1.2">𝑓</ci><ci id="A1.E11.m1.9.9.9.9.4.4.1.3.cmml" xref="A1.E11.m1.9.9.9.9.4.4.1.3">𝜃</ci></apply></apply><apply id="A1.E11.m1.32.32.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A1.E11.m1.31.31a"><ci id="A1.E11.m1.12.12.12.12.7.7.cmml" xref="A1.E11.m1.12.12.12.12.7.7">∣</ci><ci id="A1.E11.m1.11.11.11.11.6.6.cmml" xref="A1.E11.m1.11.11.11.11.6.6">𝑜</ci><interval closure="open" id="A1.E11.m1.32.32.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="A1.E11.m1.31.31a"><ci id="A1.E11.m1.14.14.14.14.9.9.cmml" xref="A1.E11.m1.14.14.14.14.9.9">𝑠</ci><ci id="A1.E11.m1.16.16.16.16.11.11.cmml" xref="A1.E11.m1.16.16.16.16.11.11">𝑟</ci></interval></apply></apply><apply id="A1.E11.m1.32.32.1.1.1.1.1.1.1.2.cmml" xref="A1.E11.m1.31.31a"><times id="A1.E11.m1.32.32.1.1.1.1.1.1.1.2.2.cmml" xref="A1.E11.m1.31.31a"></times><apply id="A1.E11.m1.32.32.1.1.1.1.1.1.1.2.3.cmml" xref="A1.E11.m1.31.31a"><csymbol cd="ambiguous" id="A1.E11.m1.32.32.1.1.1.1.1.1.1.2.3.1.cmml" xref="A1.E11.m1.31.31a">subscript</csymbol><ci id="A1.E11.m1.20.20.20.1.1.1.cmml" xref="A1.E11.m1.20.20.20.1.1.1">𝑝</ci><apply id="A1.E11.m1.21.21.21.2.2.2.1.cmml" xref="A1.E11.m1.21.21.21.2.2.2.1"><csymbol cd="ambiguous" id="A1.E11.m1.21.21.21.2.2.2.1.1.cmml" xref="A1.E11.m1.21.21.21.2.2.2.1">superscript</csymbol><apply id="A1.E11.m1.21.21.21.2.2.2.1.2.cmml" xref="A1.E11.m1.21.21.21.2.2.2.1"><csymbol cd="ambiguous" id="A1.E11.m1.21.21.21.2.2.2.1.2.1.cmml" xref="A1.E11.m1.21.21.21.2.2.2.1">subscript</csymbol><ci id="A1.E11.m1.21.21.21.2.2.2.1.2.2.cmml" xref="A1.E11.m1.21.21.21.2.2.2.1.2.2">𝑓</ci><ci id="A1.E11.m1.21.21.21.2.2.2.1.2.3.cmml" xref="A1.E11.m1.21.21.21.2.2.2.1.2.3">𝜃</ci></apply><ci id="A1.E11.m1.21.21.21.2.2.2.1.3.cmml" xref="A1.E11.m1.21.21.21.2.2.2.1.3">′</ci></apply></apply><apply id="A1.E11.m1.32.32.1.1.1.1.1.1.1.2.1.1.1.cmml" xref="A1.E11.m1.31.31a"><ci id="A1.E11.m1.24.24.24.5.5.5.cmml" xref="A1.E11.m1.24.24.24.5.5.5">∣</ci><ci id="A1.E11.m1.23.23.23.4.4.4.cmml" xref="A1.E11.m1.23.23.23.4.4.4">𝑜</ci><interval closure="open" id="A1.E11.m1.32.32.1.1.1.1.1.1.1.2.1.1.1.3.cmml" xref="A1.E11.m1.31.31a"><ci id="A1.E11.m1.26.26.26.7.7.7.cmml" xref="A1.E11.m1.26.26.26.7.7.7">𝑠</ci><ci id="A1.E11.m1.28.28.28.9.9.9.cmml" xref="A1.E11.m1.28.28.28.9.9.9">𝑟</ci></interval></apply></apply></apply></set></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E11.m1.32c">\displaystyle\begin{split}\mathrm{IR=}\frac{1}{\left|\mathrm{Obj}\right|-1}\sum_{\begin{subarray}{c}o\in\mathrm{Obj}\setminus\{o1\}\end{subarray}}&amp;\mathbbm{1}\{p_{f_{\theta}}(o\mid(s,r))&gt;\\
&amp;p_{f_{\theta}^{\prime}}(o\mid(s,r))\}\end{split}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(11)</span></td>
</tr></tbody>
</table>
</div>
</li>
<li id="A1.I8.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I8.i3.p1" class="ltx_para">
<p id="A1.I8.i3.p1.1" class="ltx_p"><span id="A1.I8.i3.p1.1.1" class="ltx_text ltx_font_bold">Failure Rate (FR) <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib26" title="" class="ltx_ref">2024b</a>)</cite>:</span></p>
<table id="A1.E12" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.E12.m1.1" class="ltx_Math" alttext="\mathrm{FR=}\mathbbm{1}\{\mathrm{IR}&gt;0.5\}" display="block"><semantics id="A1.E12.m1.1a"><mrow id="A1.E12.m1.1.1" xref="A1.E12.m1.1.1.cmml"><mi id="A1.E12.m1.1.1.3" xref="A1.E12.m1.1.1.3.cmml">FR</mi><mo id="A1.E12.m1.1.1.2" xref="A1.E12.m1.1.1.2.cmml">=</mo><mrow id="A1.E12.m1.1.1.1" xref="A1.E12.m1.1.1.1.cmml"><mn id="A1.E12.m1.1.1.1.3" xref="A1.E12.m1.1.1.1.3.cmml">𝟙</mn><mo lspace="0em" rspace="0em" id="A1.E12.m1.1.1.1.2" xref="A1.E12.m1.1.1.1.2.cmml">​</mo><mrow id="A1.E12.m1.1.1.1.1.1" xref="A1.E12.m1.1.1.1.1.2.cmml"><mo stretchy="false" id="A1.E12.m1.1.1.1.1.1.2" xref="A1.E12.m1.1.1.1.1.2.cmml">{</mo><mrow id="A1.E12.m1.1.1.1.1.1.1" xref="A1.E12.m1.1.1.1.1.1.1.cmml"><mi id="A1.E12.m1.1.1.1.1.1.1.2" xref="A1.E12.m1.1.1.1.1.1.1.2.cmml">IR</mi><mo id="A1.E12.m1.1.1.1.1.1.1.1" xref="A1.E12.m1.1.1.1.1.1.1.1.cmml">&gt;</mo><mn id="A1.E12.m1.1.1.1.1.1.1.3" xref="A1.E12.m1.1.1.1.1.1.1.3.cmml">0.5</mn></mrow><mo stretchy="false" id="A1.E12.m1.1.1.1.1.1.3" xref="A1.E12.m1.1.1.1.1.2.cmml">}</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E12.m1.1b"><apply id="A1.E12.m1.1.1.cmml" xref="A1.E12.m1.1.1"><eq id="A1.E12.m1.1.1.2.cmml" xref="A1.E12.m1.1.1.2"></eq><ci id="A1.E12.m1.1.1.3.cmml" xref="A1.E12.m1.1.1.3">FR</ci><apply id="A1.E12.m1.1.1.1.cmml" xref="A1.E12.m1.1.1.1"><times id="A1.E12.m1.1.1.1.2.cmml" xref="A1.E12.m1.1.1.1.2"></times><cn type="integer" id="A1.E12.m1.1.1.1.3.cmml" xref="A1.E12.m1.1.1.1.3">1</cn><set id="A1.E12.m1.1.1.1.1.2.cmml" xref="A1.E12.m1.1.1.1.1.1"><apply id="A1.E12.m1.1.1.1.1.1.1.cmml" xref="A1.E12.m1.1.1.1.1.1.1"><gt id="A1.E12.m1.1.1.1.1.1.1.1.cmml" xref="A1.E12.m1.1.1.1.1.1.1.1"></gt><ci id="A1.E12.m1.1.1.1.1.1.1.2.cmml" xref="A1.E12.m1.1.1.1.1.1.1.2">IR</ci><cn type="float" id="A1.E12.m1.1.1.1.1.1.1.3.cmml" xref="A1.E12.m1.1.1.1.1.1.1.3">0.5</cn></apply></set></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E12.m1.1c">\mathrm{FR=}\mathbbm{1}\{\mathrm{IR}&gt;0.5\}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(12)</span></td>
</tr></tbody>
</table>
</div>
</li>
<li id="A1.I8.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I8.i4.p1" class="ltx_para">
<p id="A1.I8.i4.p1.3" class="ltx_p"><span id="A1.I8.i4.p1.3.1" class="ltx_text ltx_font_bold">Tied Fact Damage (TDF) <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib26" title="" class="ltx_ref">2024b</a>)</cite>:</span></p>
<table id="A1.E13" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.E13.m1.3" class="ltx_Math" alttext="\mathrm{TFD=}\frac{p_{f_{\theta^{m}}}(k_{f})-p_{f_{\theta^{\prime}}}(k_{f})}{p_{f_{\theta^{m}}}(k_{f})}" display="block"><semantics id="A1.E13.m1.3a"><mrow id="A1.E13.m1.3.4" xref="A1.E13.m1.3.4.cmml"><mi id="A1.E13.m1.3.4.2" xref="A1.E13.m1.3.4.2.cmml">TFD</mi><mo id="A1.E13.m1.3.4.1" xref="A1.E13.m1.3.4.1.cmml">=</mo><mfrac id="A1.E13.m1.3.3" xref="A1.E13.m1.3.3.cmml"><mrow id="A1.E13.m1.2.2.2" xref="A1.E13.m1.2.2.2.cmml"><mrow id="A1.E13.m1.1.1.1.1" xref="A1.E13.m1.1.1.1.1.cmml"><msub id="A1.E13.m1.1.1.1.1.3" xref="A1.E13.m1.1.1.1.1.3.cmml"><mi id="A1.E13.m1.1.1.1.1.3.2" xref="A1.E13.m1.1.1.1.1.3.2.cmml">p</mi><msub id="A1.E13.m1.1.1.1.1.3.3" xref="A1.E13.m1.1.1.1.1.3.3.cmml"><mi id="A1.E13.m1.1.1.1.1.3.3.2" xref="A1.E13.m1.1.1.1.1.3.3.2.cmml">f</mi><msup id="A1.E13.m1.1.1.1.1.3.3.3" xref="A1.E13.m1.1.1.1.1.3.3.3.cmml"><mi id="A1.E13.m1.1.1.1.1.3.3.3.2" xref="A1.E13.m1.1.1.1.1.3.3.3.2.cmml">θ</mi><mi id="A1.E13.m1.1.1.1.1.3.3.3.3" xref="A1.E13.m1.1.1.1.1.3.3.3.3.cmml">m</mi></msup></msub></msub><mo lspace="0em" rspace="0em" id="A1.E13.m1.1.1.1.1.2" xref="A1.E13.m1.1.1.1.1.2.cmml">​</mo><mrow id="A1.E13.m1.1.1.1.1.1.1" xref="A1.E13.m1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.E13.m1.1.1.1.1.1.1.2" xref="A1.E13.m1.1.1.1.1.1.1.1.cmml">(</mo><msub id="A1.E13.m1.1.1.1.1.1.1.1" xref="A1.E13.m1.1.1.1.1.1.1.1.cmml"><mi id="A1.E13.m1.1.1.1.1.1.1.1.2" xref="A1.E13.m1.1.1.1.1.1.1.1.2.cmml">k</mi><mi id="A1.E13.m1.1.1.1.1.1.1.1.3" xref="A1.E13.m1.1.1.1.1.1.1.1.3.cmml">f</mi></msub><mo stretchy="false" id="A1.E13.m1.1.1.1.1.1.1.3" xref="A1.E13.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="A1.E13.m1.2.2.2.3" xref="A1.E13.m1.2.2.2.3.cmml">−</mo><mrow id="A1.E13.m1.2.2.2.2" xref="A1.E13.m1.2.2.2.2.cmml"><msub id="A1.E13.m1.2.2.2.2.3" xref="A1.E13.m1.2.2.2.2.3.cmml"><mi id="A1.E13.m1.2.2.2.2.3.2" xref="A1.E13.m1.2.2.2.2.3.2.cmml">p</mi><msub id="A1.E13.m1.2.2.2.2.3.3" xref="A1.E13.m1.2.2.2.2.3.3.cmml"><mi id="A1.E13.m1.2.2.2.2.3.3.2" xref="A1.E13.m1.2.2.2.2.3.3.2.cmml">f</mi><msup id="A1.E13.m1.2.2.2.2.3.3.3" xref="A1.E13.m1.2.2.2.2.3.3.3.cmml"><mi id="A1.E13.m1.2.2.2.2.3.3.3.2" xref="A1.E13.m1.2.2.2.2.3.3.3.2.cmml">θ</mi><mo id="A1.E13.m1.2.2.2.2.3.3.3.3" xref="A1.E13.m1.2.2.2.2.3.3.3.3.cmml">′</mo></msup></msub></msub><mo lspace="0em" rspace="0em" id="A1.E13.m1.2.2.2.2.2" xref="A1.E13.m1.2.2.2.2.2.cmml">​</mo><mrow id="A1.E13.m1.2.2.2.2.1.1" xref="A1.E13.m1.2.2.2.2.1.1.1.cmml"><mo stretchy="false" id="A1.E13.m1.2.2.2.2.1.1.2" xref="A1.E13.m1.2.2.2.2.1.1.1.cmml">(</mo><msub id="A1.E13.m1.2.2.2.2.1.1.1" xref="A1.E13.m1.2.2.2.2.1.1.1.cmml"><mi id="A1.E13.m1.2.2.2.2.1.1.1.2" xref="A1.E13.m1.2.2.2.2.1.1.1.2.cmml">k</mi><mi id="A1.E13.m1.2.2.2.2.1.1.1.3" xref="A1.E13.m1.2.2.2.2.1.1.1.3.cmml">f</mi></msub><mo stretchy="false" id="A1.E13.m1.2.2.2.2.1.1.3" xref="A1.E13.m1.2.2.2.2.1.1.1.cmml">)</mo></mrow></mrow></mrow><mrow id="A1.E13.m1.3.3.3" xref="A1.E13.m1.3.3.3.cmml"><msub id="A1.E13.m1.3.3.3.3" xref="A1.E13.m1.3.3.3.3.cmml"><mi id="A1.E13.m1.3.3.3.3.2" xref="A1.E13.m1.3.3.3.3.2.cmml">p</mi><msub id="A1.E13.m1.3.3.3.3.3" xref="A1.E13.m1.3.3.3.3.3.cmml"><mi id="A1.E13.m1.3.3.3.3.3.2" xref="A1.E13.m1.3.3.3.3.3.2.cmml">f</mi><msup id="A1.E13.m1.3.3.3.3.3.3" xref="A1.E13.m1.3.3.3.3.3.3.cmml"><mi id="A1.E13.m1.3.3.3.3.3.3.2" xref="A1.E13.m1.3.3.3.3.3.3.2.cmml">θ</mi><mi id="A1.E13.m1.3.3.3.3.3.3.3" xref="A1.E13.m1.3.3.3.3.3.3.3.cmml">m</mi></msup></msub></msub><mo lspace="0em" rspace="0em" id="A1.E13.m1.3.3.3.2" xref="A1.E13.m1.3.3.3.2.cmml">​</mo><mrow id="A1.E13.m1.3.3.3.1.1" xref="A1.E13.m1.3.3.3.1.1.1.cmml"><mo stretchy="false" id="A1.E13.m1.3.3.3.1.1.2" xref="A1.E13.m1.3.3.3.1.1.1.cmml">(</mo><msub id="A1.E13.m1.3.3.3.1.1.1" xref="A1.E13.m1.3.3.3.1.1.1.cmml"><mi id="A1.E13.m1.3.3.3.1.1.1.2" xref="A1.E13.m1.3.3.3.1.1.1.2.cmml">k</mi><mi id="A1.E13.m1.3.3.3.1.1.1.3" xref="A1.E13.m1.3.3.3.1.1.1.3.cmml">f</mi></msub><mo stretchy="false" id="A1.E13.m1.3.3.3.1.1.3" xref="A1.E13.m1.3.3.3.1.1.1.cmml">)</mo></mrow></mrow></mfrac></mrow><annotation-xml encoding="MathML-Content" id="A1.E13.m1.3b"><apply id="A1.E13.m1.3.4.cmml" xref="A1.E13.m1.3.4"><eq id="A1.E13.m1.3.4.1.cmml" xref="A1.E13.m1.3.4.1"></eq><ci id="A1.E13.m1.3.4.2.cmml" xref="A1.E13.m1.3.4.2">TFD</ci><apply id="A1.E13.m1.3.3.cmml" xref="A1.E13.m1.3.3"><divide id="A1.E13.m1.3.3.4.cmml" xref="A1.E13.m1.3.3"></divide><apply id="A1.E13.m1.2.2.2.cmml" xref="A1.E13.m1.2.2.2"><minus id="A1.E13.m1.2.2.2.3.cmml" xref="A1.E13.m1.2.2.2.3"></minus><apply id="A1.E13.m1.1.1.1.1.cmml" xref="A1.E13.m1.1.1.1.1"><times id="A1.E13.m1.1.1.1.1.2.cmml" xref="A1.E13.m1.1.1.1.1.2"></times><apply id="A1.E13.m1.1.1.1.1.3.cmml" xref="A1.E13.m1.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.E13.m1.1.1.1.1.3.1.cmml" xref="A1.E13.m1.1.1.1.1.3">subscript</csymbol><ci id="A1.E13.m1.1.1.1.1.3.2.cmml" xref="A1.E13.m1.1.1.1.1.3.2">𝑝</ci><apply id="A1.E13.m1.1.1.1.1.3.3.cmml" xref="A1.E13.m1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="A1.E13.m1.1.1.1.1.3.3.1.cmml" xref="A1.E13.m1.1.1.1.1.3.3">subscript</csymbol><ci id="A1.E13.m1.1.1.1.1.3.3.2.cmml" xref="A1.E13.m1.1.1.1.1.3.3.2">𝑓</ci><apply id="A1.E13.m1.1.1.1.1.3.3.3.cmml" xref="A1.E13.m1.1.1.1.1.3.3.3"><csymbol cd="ambiguous" id="A1.E13.m1.1.1.1.1.3.3.3.1.cmml" xref="A1.E13.m1.1.1.1.1.3.3.3">superscript</csymbol><ci id="A1.E13.m1.1.1.1.1.3.3.3.2.cmml" xref="A1.E13.m1.1.1.1.1.3.3.3.2">𝜃</ci><ci id="A1.E13.m1.1.1.1.1.3.3.3.3.cmml" xref="A1.E13.m1.1.1.1.1.3.3.3.3">𝑚</ci></apply></apply></apply><apply id="A1.E13.m1.1.1.1.1.1.1.1.cmml" xref="A1.E13.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.E13.m1.1.1.1.1.1.1.1.1.cmml" xref="A1.E13.m1.1.1.1.1.1.1">subscript</csymbol><ci id="A1.E13.m1.1.1.1.1.1.1.1.2.cmml" xref="A1.E13.m1.1.1.1.1.1.1.1.2">𝑘</ci><ci id="A1.E13.m1.1.1.1.1.1.1.1.3.cmml" xref="A1.E13.m1.1.1.1.1.1.1.1.3">𝑓</ci></apply></apply><apply id="A1.E13.m1.2.2.2.2.cmml" xref="A1.E13.m1.2.2.2.2"><times id="A1.E13.m1.2.2.2.2.2.cmml" xref="A1.E13.m1.2.2.2.2.2"></times><apply id="A1.E13.m1.2.2.2.2.3.cmml" xref="A1.E13.m1.2.2.2.2.3"><csymbol cd="ambiguous" id="A1.E13.m1.2.2.2.2.3.1.cmml" xref="A1.E13.m1.2.2.2.2.3">subscript</csymbol><ci id="A1.E13.m1.2.2.2.2.3.2.cmml" xref="A1.E13.m1.2.2.2.2.3.2">𝑝</ci><apply id="A1.E13.m1.2.2.2.2.3.3.cmml" xref="A1.E13.m1.2.2.2.2.3.3"><csymbol cd="ambiguous" id="A1.E13.m1.2.2.2.2.3.3.1.cmml" xref="A1.E13.m1.2.2.2.2.3.3">subscript</csymbol><ci id="A1.E13.m1.2.2.2.2.3.3.2.cmml" xref="A1.E13.m1.2.2.2.2.3.3.2">𝑓</ci><apply id="A1.E13.m1.2.2.2.2.3.3.3.cmml" xref="A1.E13.m1.2.2.2.2.3.3.3"><csymbol cd="ambiguous" id="A1.E13.m1.2.2.2.2.3.3.3.1.cmml" xref="A1.E13.m1.2.2.2.2.3.3.3">superscript</csymbol><ci id="A1.E13.m1.2.2.2.2.3.3.3.2.cmml" xref="A1.E13.m1.2.2.2.2.3.3.3.2">𝜃</ci><ci id="A1.E13.m1.2.2.2.2.3.3.3.3.cmml" xref="A1.E13.m1.2.2.2.2.3.3.3.3">′</ci></apply></apply></apply><apply id="A1.E13.m1.2.2.2.2.1.1.1.cmml" xref="A1.E13.m1.2.2.2.2.1.1"><csymbol cd="ambiguous" id="A1.E13.m1.2.2.2.2.1.1.1.1.cmml" xref="A1.E13.m1.2.2.2.2.1.1">subscript</csymbol><ci id="A1.E13.m1.2.2.2.2.1.1.1.2.cmml" xref="A1.E13.m1.2.2.2.2.1.1.1.2">𝑘</ci><ci id="A1.E13.m1.2.2.2.2.1.1.1.3.cmml" xref="A1.E13.m1.2.2.2.2.1.1.1.3">𝑓</ci></apply></apply></apply><apply id="A1.E13.m1.3.3.3.cmml" xref="A1.E13.m1.3.3.3"><times id="A1.E13.m1.3.3.3.2.cmml" xref="A1.E13.m1.3.3.3.2"></times><apply id="A1.E13.m1.3.3.3.3.cmml" xref="A1.E13.m1.3.3.3.3"><csymbol cd="ambiguous" id="A1.E13.m1.3.3.3.3.1.cmml" xref="A1.E13.m1.3.3.3.3">subscript</csymbol><ci id="A1.E13.m1.3.3.3.3.2.cmml" xref="A1.E13.m1.3.3.3.3.2">𝑝</ci><apply id="A1.E13.m1.3.3.3.3.3.cmml" xref="A1.E13.m1.3.3.3.3.3"><csymbol cd="ambiguous" id="A1.E13.m1.3.3.3.3.3.1.cmml" xref="A1.E13.m1.3.3.3.3.3">subscript</csymbol><ci id="A1.E13.m1.3.3.3.3.3.2.cmml" xref="A1.E13.m1.3.3.3.3.3.2">𝑓</ci><apply id="A1.E13.m1.3.3.3.3.3.3.cmml" xref="A1.E13.m1.3.3.3.3.3.3"><csymbol cd="ambiguous" id="A1.E13.m1.3.3.3.3.3.3.1.cmml" xref="A1.E13.m1.3.3.3.3.3.3">superscript</csymbol><ci id="A1.E13.m1.3.3.3.3.3.3.2.cmml" xref="A1.E13.m1.3.3.3.3.3.3.2">𝜃</ci><ci id="A1.E13.m1.3.3.3.3.3.3.3.cmml" xref="A1.E13.m1.3.3.3.3.3.3.3">𝑚</ci></apply></apply></apply><apply id="A1.E13.m1.3.3.3.1.1.1.cmml" xref="A1.E13.m1.3.3.3.1.1"><csymbol cd="ambiguous" id="A1.E13.m1.3.3.3.1.1.1.1.cmml" xref="A1.E13.m1.3.3.3.1.1">subscript</csymbol><ci id="A1.E13.m1.3.3.3.1.1.1.2.cmml" xref="A1.E13.m1.3.3.3.1.1.1.2">𝑘</ci><ci id="A1.E13.m1.3.3.3.1.1.1.3.cmml" xref="A1.E13.m1.3.3.3.1.1.1.3">𝑓</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E13.m1.3c">\mathrm{TFD=}\frac{p_{f_{\theta^{m}}}(k_{f})-p_{f_{\theta^{\prime}}}(k_{f})}{p_{f_{\theta^{m}}}(k_{f})}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(13)</span></td>
</tr></tbody>
</table>
<p id="A1.I8.i4.p1.2" class="ltx_p"><math id="A1.I8.i4.p1.1.m1.1" class="ltx_Math" alttext="k_{f}" display="inline"><semantics id="A1.I8.i4.p1.1.m1.1a"><msub id="A1.I8.i4.p1.1.m1.1.1" xref="A1.I8.i4.p1.1.m1.1.1.cmml"><mi id="A1.I8.i4.p1.1.m1.1.1.2" xref="A1.I8.i4.p1.1.m1.1.1.2.cmml">k</mi><mi id="A1.I8.i4.p1.1.m1.1.1.3" xref="A1.I8.i4.p1.1.m1.1.1.3.cmml">f</mi></msub><annotation-xml encoding="MathML-Content" id="A1.I8.i4.p1.1.m1.1b"><apply id="A1.I8.i4.p1.1.m1.1.1.cmml" xref="A1.I8.i4.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A1.I8.i4.p1.1.m1.1.1.1.cmml" xref="A1.I8.i4.p1.1.m1.1.1">subscript</csymbol><ci id="A1.I8.i4.p1.1.m1.1.1.2.cmml" xref="A1.I8.i4.p1.1.m1.1.1.2">𝑘</ci><ci id="A1.I8.i4.p1.1.m1.1.1.3.cmml" xref="A1.I8.i4.p1.1.m1.1.1.3">𝑓</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I8.i4.p1.1.m1.1c">k_{f}</annotation></semantics></math> denotes the tied facts and <math id="A1.I8.i4.p1.2.m2.1" class="ltx_Math" alttext="\theta^{m}" display="inline"><semantics id="A1.I8.i4.p1.2.m2.1a"><msup id="A1.I8.i4.p1.2.m2.1.1" xref="A1.I8.i4.p1.2.m2.1.1.cmml"><mi id="A1.I8.i4.p1.2.m2.1.1.2" xref="A1.I8.i4.p1.2.m2.1.1.2.cmml">θ</mi><mi id="A1.I8.i4.p1.2.m2.1.1.3" xref="A1.I8.i4.p1.2.m2.1.1.3.cmml">m</mi></msup><annotation-xml encoding="MathML-Content" id="A1.I8.i4.p1.2.m2.1b"><apply id="A1.I8.i4.p1.2.m2.1.1.cmml" xref="A1.I8.i4.p1.2.m2.1.1"><csymbol cd="ambiguous" id="A1.I8.i4.p1.2.m2.1.1.1.cmml" xref="A1.I8.i4.p1.2.m2.1.1">superscript</csymbol><ci id="A1.I8.i4.p1.2.m2.1.1.2.cmml" xref="A1.I8.i4.p1.2.m2.1.1.2">𝜃</ci><ci id="A1.I8.i4.p1.2.m2.1.1.3.cmml" xref="A1.I8.i4.p1.2.m2.1.1.3">𝑚</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I8.i4.p1.2.m2.1c">\theta^{m}</annotation></semantics></math> is the intermediate model parameters after edit 1.</p>
</div>
</li>
</ul>
</div>
</section>
<section id="A1.SS2.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Other Locality Metrics</h5>

<div id="A1.SS2.SSS0.Px3.p1" class="ltx_para">
<ul id="A1.I9" class="ltx_itemize">
<li id="A1.I9.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I9.i1.p1" class="ltx_para">
<p id="A1.I9.i1.p1.1" class="ltx_p"><span id="A1.I9.i1.p1.1.1" class="ltx_text ltx_font_bold">Neighborhood KL Divergence <cite class="ltx_cite ltx_citemacro_citep">(Hoelscher-Obermaier et al., <a href="#bib.bib19" title="" class="ltx_ref">2023</a>)</cite>:</span></p>
<table id="A1.E14" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.E14.m1.3" class="ltx_Math" alttext="\mathrm{NKL}\stackrel{{\scriptstyle\mathrm{def}}}{{=}}\sum_{w\in W}\log\left(\frac{P(w)}{P^{*}(w)}\right)" display="block"><semantics id="A1.E14.m1.3a"><mrow id="A1.E14.m1.3.4" xref="A1.E14.m1.3.4.cmml"><mi id="A1.E14.m1.3.4.2" xref="A1.E14.m1.3.4.2.cmml">NKL</mi><mover id="A1.E14.m1.3.4.1" xref="A1.E14.m1.3.4.1.cmml"><mo rspace="0.111em" id="A1.E14.m1.3.4.1.2" xref="A1.E14.m1.3.4.1.2.cmml">=</mo><mi id="A1.E14.m1.3.4.1.3" xref="A1.E14.m1.3.4.1.3.cmml">def</mi></mover><mrow id="A1.E14.m1.3.4.3" xref="A1.E14.m1.3.4.3.cmml"><munder id="A1.E14.m1.3.4.3.1" xref="A1.E14.m1.3.4.3.1.cmml"><mo movablelimits="false" id="A1.E14.m1.3.4.3.1.2" xref="A1.E14.m1.3.4.3.1.2.cmml">∑</mo><mrow id="A1.E14.m1.3.4.3.1.3" xref="A1.E14.m1.3.4.3.1.3.cmml"><mi id="A1.E14.m1.3.4.3.1.3.2" xref="A1.E14.m1.3.4.3.1.3.2.cmml">w</mi><mo id="A1.E14.m1.3.4.3.1.3.1" xref="A1.E14.m1.3.4.3.1.3.1.cmml">∈</mo><mi id="A1.E14.m1.3.4.3.1.3.3" xref="A1.E14.m1.3.4.3.1.3.3.cmml">W</mi></mrow></munder><mrow id="A1.E14.m1.3.4.3.2.2" xref="A1.E14.m1.3.4.3.2.1.cmml"><mi id="A1.E14.m1.3.3" xref="A1.E14.m1.3.3.cmml">log</mi><mo id="A1.E14.m1.3.4.3.2.2a" xref="A1.E14.m1.3.4.3.2.1.cmml">⁡</mo><mrow id="A1.E14.m1.3.4.3.2.2.1" xref="A1.E14.m1.3.4.3.2.1.cmml"><mo id="A1.E14.m1.3.4.3.2.2.1.1" xref="A1.E14.m1.3.4.3.2.1.cmml">(</mo><mfrac id="A1.E14.m1.2.2" xref="A1.E14.m1.2.2.cmml"><mrow id="A1.E14.m1.1.1.1" xref="A1.E14.m1.1.1.1.cmml"><mi id="A1.E14.m1.1.1.1.3" xref="A1.E14.m1.1.1.1.3.cmml">P</mi><mo lspace="0em" rspace="0em" id="A1.E14.m1.1.1.1.2" xref="A1.E14.m1.1.1.1.2.cmml">​</mo><mrow id="A1.E14.m1.1.1.1.4.2" xref="A1.E14.m1.1.1.1.cmml"><mo stretchy="false" id="A1.E14.m1.1.1.1.4.2.1" xref="A1.E14.m1.1.1.1.cmml">(</mo><mi id="A1.E14.m1.1.1.1.1" xref="A1.E14.m1.1.1.1.1.cmml">w</mi><mo stretchy="false" id="A1.E14.m1.1.1.1.4.2.2" xref="A1.E14.m1.1.1.1.cmml">)</mo></mrow></mrow><mrow id="A1.E14.m1.2.2.2" xref="A1.E14.m1.2.2.2.cmml"><msup id="A1.E14.m1.2.2.2.3" xref="A1.E14.m1.2.2.2.3.cmml"><mi id="A1.E14.m1.2.2.2.3.2" xref="A1.E14.m1.2.2.2.3.2.cmml">P</mi><mo id="A1.E14.m1.2.2.2.3.3" xref="A1.E14.m1.2.2.2.3.3.cmml">∗</mo></msup><mo lspace="0em" rspace="0em" id="A1.E14.m1.2.2.2.2" xref="A1.E14.m1.2.2.2.2.cmml">​</mo><mrow id="A1.E14.m1.2.2.2.4.2" xref="A1.E14.m1.2.2.2.cmml"><mo stretchy="false" id="A1.E14.m1.2.2.2.4.2.1" xref="A1.E14.m1.2.2.2.cmml">(</mo><mi id="A1.E14.m1.2.2.2.1" xref="A1.E14.m1.2.2.2.1.cmml">w</mi><mo stretchy="false" id="A1.E14.m1.2.2.2.4.2.2" xref="A1.E14.m1.2.2.2.cmml">)</mo></mrow></mrow></mfrac><mo id="A1.E14.m1.3.4.3.2.2.1.2" xref="A1.E14.m1.3.4.3.2.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E14.m1.3b"><apply id="A1.E14.m1.3.4.cmml" xref="A1.E14.m1.3.4"><apply id="A1.E14.m1.3.4.1.cmml" xref="A1.E14.m1.3.4.1"><csymbol cd="ambiguous" id="A1.E14.m1.3.4.1.1.cmml" xref="A1.E14.m1.3.4.1">superscript</csymbol><eq id="A1.E14.m1.3.4.1.2.cmml" xref="A1.E14.m1.3.4.1.2"></eq><ci id="A1.E14.m1.3.4.1.3.cmml" xref="A1.E14.m1.3.4.1.3">def</ci></apply><ci id="A1.E14.m1.3.4.2.cmml" xref="A1.E14.m1.3.4.2">NKL</ci><apply id="A1.E14.m1.3.4.3.cmml" xref="A1.E14.m1.3.4.3"><apply id="A1.E14.m1.3.4.3.1.cmml" xref="A1.E14.m1.3.4.3.1"><csymbol cd="ambiguous" id="A1.E14.m1.3.4.3.1.1.cmml" xref="A1.E14.m1.3.4.3.1">subscript</csymbol><sum id="A1.E14.m1.3.4.3.1.2.cmml" xref="A1.E14.m1.3.4.3.1.2"></sum><apply id="A1.E14.m1.3.4.3.1.3.cmml" xref="A1.E14.m1.3.4.3.1.3"><in id="A1.E14.m1.3.4.3.1.3.1.cmml" xref="A1.E14.m1.3.4.3.1.3.1"></in><ci id="A1.E14.m1.3.4.3.1.3.2.cmml" xref="A1.E14.m1.3.4.3.1.3.2">𝑤</ci><ci id="A1.E14.m1.3.4.3.1.3.3.cmml" xref="A1.E14.m1.3.4.3.1.3.3">𝑊</ci></apply></apply><apply id="A1.E14.m1.3.4.3.2.1.cmml" xref="A1.E14.m1.3.4.3.2.2"><log id="A1.E14.m1.3.3.cmml" xref="A1.E14.m1.3.3"></log><apply id="A1.E14.m1.2.2.cmml" xref="A1.E14.m1.2.2"><divide id="A1.E14.m1.2.2.3.cmml" xref="A1.E14.m1.2.2"></divide><apply id="A1.E14.m1.1.1.1.cmml" xref="A1.E14.m1.1.1.1"><times id="A1.E14.m1.1.1.1.2.cmml" xref="A1.E14.m1.1.1.1.2"></times><ci id="A1.E14.m1.1.1.1.3.cmml" xref="A1.E14.m1.1.1.1.3">𝑃</ci><ci id="A1.E14.m1.1.1.1.1.cmml" xref="A1.E14.m1.1.1.1.1">𝑤</ci></apply><apply id="A1.E14.m1.2.2.2.cmml" xref="A1.E14.m1.2.2.2"><times id="A1.E14.m1.2.2.2.2.cmml" xref="A1.E14.m1.2.2.2.2"></times><apply id="A1.E14.m1.2.2.2.3.cmml" xref="A1.E14.m1.2.2.2.3"><csymbol cd="ambiguous" id="A1.E14.m1.2.2.2.3.1.cmml" xref="A1.E14.m1.2.2.2.3">superscript</csymbol><ci id="A1.E14.m1.2.2.2.3.2.cmml" xref="A1.E14.m1.2.2.2.3.2">𝑃</ci><times id="A1.E14.m1.2.2.2.3.3.cmml" xref="A1.E14.m1.2.2.2.3.3"></times></apply><ci id="A1.E14.m1.2.2.2.1.cmml" xref="A1.E14.m1.2.2.2.1">𝑤</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E14.m1.3c">\mathrm{NKL}\stackrel{{\scriptstyle\mathrm{def}}}{{=}}\sum_{w\in W}\log\left(\frac{P(w)}{P^{*}(w)}\right)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(14)</span></td>
</tr></tbody>
</table>
</div>
</li>
<li id="A1.I9.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I9.i2.p1" class="ltx_para">
<p id="A1.I9.i2.p1.3" class="ltx_p"><span id="A1.I9.i2.p1.3.1" class="ltx_text ltx_font_bold">Neighborhood Score (NS) <cite class="ltx_cite ltx_citemacro_citep">(Meng et al., <a href="#bib.bib29" title="" class="ltx_ref">2022</a>)</cite>:</span> collect a set of "neighbor" subjects and evaluation the success fraction for <math id="A1.I9.i2.p1.1.m1.2" class="ltx_Math" alttext="\mathbb{P}\left[o^{c}\right]&gt;\mathbb{P}\left[o^{*}\right]" display="inline"><semantics id="A1.I9.i2.p1.1.m1.2a"><mrow id="A1.I9.i2.p1.1.m1.2.2" xref="A1.I9.i2.p1.1.m1.2.2.cmml"><mrow id="A1.I9.i2.p1.1.m1.1.1.1" xref="A1.I9.i2.p1.1.m1.1.1.1.cmml"><mi id="A1.I9.i2.p1.1.m1.1.1.1.3" xref="A1.I9.i2.p1.1.m1.1.1.1.3.cmml">ℙ</mi><mo lspace="0em" rspace="0em" id="A1.I9.i2.p1.1.m1.1.1.1.2" xref="A1.I9.i2.p1.1.m1.1.1.1.2.cmml">​</mo><mrow id="A1.I9.i2.p1.1.m1.1.1.1.1.1" xref="A1.I9.i2.p1.1.m1.1.1.1.1.2.cmml"><mo id="A1.I9.i2.p1.1.m1.1.1.1.1.1.2" xref="A1.I9.i2.p1.1.m1.1.1.1.1.2.1.cmml">[</mo><msup id="A1.I9.i2.p1.1.m1.1.1.1.1.1.1" xref="A1.I9.i2.p1.1.m1.1.1.1.1.1.1.cmml"><mi id="A1.I9.i2.p1.1.m1.1.1.1.1.1.1.2" xref="A1.I9.i2.p1.1.m1.1.1.1.1.1.1.2.cmml">o</mi><mi id="A1.I9.i2.p1.1.m1.1.1.1.1.1.1.3" xref="A1.I9.i2.p1.1.m1.1.1.1.1.1.1.3.cmml">c</mi></msup><mo id="A1.I9.i2.p1.1.m1.1.1.1.1.1.3" xref="A1.I9.i2.p1.1.m1.1.1.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="A1.I9.i2.p1.1.m1.2.2.3" xref="A1.I9.i2.p1.1.m1.2.2.3.cmml">&gt;</mo><mrow id="A1.I9.i2.p1.1.m1.2.2.2" xref="A1.I9.i2.p1.1.m1.2.2.2.cmml"><mi id="A1.I9.i2.p1.1.m1.2.2.2.3" xref="A1.I9.i2.p1.1.m1.2.2.2.3.cmml">ℙ</mi><mo lspace="0em" rspace="0em" id="A1.I9.i2.p1.1.m1.2.2.2.2" xref="A1.I9.i2.p1.1.m1.2.2.2.2.cmml">​</mo><mrow id="A1.I9.i2.p1.1.m1.2.2.2.1.1" xref="A1.I9.i2.p1.1.m1.2.2.2.1.2.cmml"><mo id="A1.I9.i2.p1.1.m1.2.2.2.1.1.2" xref="A1.I9.i2.p1.1.m1.2.2.2.1.2.1.cmml">[</mo><msup id="A1.I9.i2.p1.1.m1.2.2.2.1.1.1" xref="A1.I9.i2.p1.1.m1.2.2.2.1.1.1.cmml"><mi id="A1.I9.i2.p1.1.m1.2.2.2.1.1.1.2" xref="A1.I9.i2.p1.1.m1.2.2.2.1.1.1.2.cmml">o</mi><mo id="A1.I9.i2.p1.1.m1.2.2.2.1.1.1.3" xref="A1.I9.i2.p1.1.m1.2.2.2.1.1.1.3.cmml">∗</mo></msup><mo id="A1.I9.i2.p1.1.m1.2.2.2.1.1.3" xref="A1.I9.i2.p1.1.m1.2.2.2.1.2.1.cmml">]</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.I9.i2.p1.1.m1.2b"><apply id="A1.I9.i2.p1.1.m1.2.2.cmml" xref="A1.I9.i2.p1.1.m1.2.2"><gt id="A1.I9.i2.p1.1.m1.2.2.3.cmml" xref="A1.I9.i2.p1.1.m1.2.2.3"></gt><apply id="A1.I9.i2.p1.1.m1.1.1.1.cmml" xref="A1.I9.i2.p1.1.m1.1.1.1"><times id="A1.I9.i2.p1.1.m1.1.1.1.2.cmml" xref="A1.I9.i2.p1.1.m1.1.1.1.2"></times><ci id="A1.I9.i2.p1.1.m1.1.1.1.3.cmml" xref="A1.I9.i2.p1.1.m1.1.1.1.3">ℙ</ci><apply id="A1.I9.i2.p1.1.m1.1.1.1.1.2.cmml" xref="A1.I9.i2.p1.1.m1.1.1.1.1.1"><csymbol cd="latexml" id="A1.I9.i2.p1.1.m1.1.1.1.1.2.1.cmml" xref="A1.I9.i2.p1.1.m1.1.1.1.1.1.2">delimited-[]</csymbol><apply id="A1.I9.i2.p1.1.m1.1.1.1.1.1.1.cmml" xref="A1.I9.i2.p1.1.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.I9.i2.p1.1.m1.1.1.1.1.1.1.1.cmml" xref="A1.I9.i2.p1.1.m1.1.1.1.1.1.1">superscript</csymbol><ci id="A1.I9.i2.p1.1.m1.1.1.1.1.1.1.2.cmml" xref="A1.I9.i2.p1.1.m1.1.1.1.1.1.1.2">𝑜</ci><ci id="A1.I9.i2.p1.1.m1.1.1.1.1.1.1.3.cmml" xref="A1.I9.i2.p1.1.m1.1.1.1.1.1.1.3">𝑐</ci></apply></apply></apply><apply id="A1.I9.i2.p1.1.m1.2.2.2.cmml" xref="A1.I9.i2.p1.1.m1.2.2.2"><times id="A1.I9.i2.p1.1.m1.2.2.2.2.cmml" xref="A1.I9.i2.p1.1.m1.2.2.2.2"></times><ci id="A1.I9.i2.p1.1.m1.2.2.2.3.cmml" xref="A1.I9.i2.p1.1.m1.2.2.2.3">ℙ</ci><apply id="A1.I9.i2.p1.1.m1.2.2.2.1.2.cmml" xref="A1.I9.i2.p1.1.m1.2.2.2.1.1"><csymbol cd="latexml" id="A1.I9.i2.p1.1.m1.2.2.2.1.2.1.cmml" xref="A1.I9.i2.p1.1.m1.2.2.2.1.1.2">delimited-[]</csymbol><apply id="A1.I9.i2.p1.1.m1.2.2.2.1.1.1.cmml" xref="A1.I9.i2.p1.1.m1.2.2.2.1.1.1"><csymbol cd="ambiguous" id="A1.I9.i2.p1.1.m1.2.2.2.1.1.1.1.cmml" xref="A1.I9.i2.p1.1.m1.2.2.2.1.1.1">superscript</csymbol><ci id="A1.I9.i2.p1.1.m1.2.2.2.1.1.1.2.cmml" xref="A1.I9.i2.p1.1.m1.2.2.2.1.1.1.2">𝑜</ci><times id="A1.I9.i2.p1.1.m1.2.2.2.1.1.1.3.cmml" xref="A1.I9.i2.p1.1.m1.2.2.2.1.1.1.3"></times></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I9.i2.p1.1.m1.2c">\mathbb{P}\left[o^{c}\right]&gt;\mathbb{P}\left[o^{*}\right]</annotation></semantics></math>, while the <math id="A1.I9.i2.p1.2.m2.1" class="ltx_Math" alttext="o^{c}" display="inline"><semantics id="A1.I9.i2.p1.2.m2.1a"><msup id="A1.I9.i2.p1.2.m2.1.1" xref="A1.I9.i2.p1.2.m2.1.1.cmml"><mi id="A1.I9.i2.p1.2.m2.1.1.2" xref="A1.I9.i2.p1.2.m2.1.1.2.cmml">o</mi><mi id="A1.I9.i2.p1.2.m2.1.1.3" xref="A1.I9.i2.p1.2.m2.1.1.3.cmml">c</mi></msup><annotation-xml encoding="MathML-Content" id="A1.I9.i2.p1.2.m2.1b"><apply id="A1.I9.i2.p1.2.m2.1.1.cmml" xref="A1.I9.i2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="A1.I9.i2.p1.2.m2.1.1.1.cmml" xref="A1.I9.i2.p1.2.m2.1.1">superscript</csymbol><ci id="A1.I9.i2.p1.2.m2.1.1.2.cmml" xref="A1.I9.i2.p1.2.m2.1.1.2">𝑜</ci><ci id="A1.I9.i2.p1.2.m2.1.1.3.cmml" xref="A1.I9.i2.p1.2.m2.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I9.i2.p1.2.m2.1c">o^{c}</annotation></semantics></math> denotes the correct facts and <math id="A1.I9.i2.p1.3.m3.1" class="ltx_Math" alttext="o^{*}" display="inline"><semantics id="A1.I9.i2.p1.3.m3.1a"><msup id="A1.I9.i2.p1.3.m3.1.1" xref="A1.I9.i2.p1.3.m3.1.1.cmml"><mi id="A1.I9.i2.p1.3.m3.1.1.2" xref="A1.I9.i2.p1.3.m3.1.1.2.cmml">o</mi><mo id="A1.I9.i2.p1.3.m3.1.1.3" xref="A1.I9.i2.p1.3.m3.1.1.3.cmml">∗</mo></msup><annotation-xml encoding="MathML-Content" id="A1.I9.i2.p1.3.m3.1b"><apply id="A1.I9.i2.p1.3.m3.1.1.cmml" xref="A1.I9.i2.p1.3.m3.1.1"><csymbol cd="ambiguous" id="A1.I9.i2.p1.3.m3.1.1.1.cmml" xref="A1.I9.i2.p1.3.m3.1.1">superscript</csymbol><ci id="A1.I9.i2.p1.3.m3.1.1.2.cmml" xref="A1.I9.i2.p1.3.m3.1.1.2">𝑜</ci><times id="A1.I9.i2.p1.3.m3.1.1.3.cmml" xref="A1.I9.i2.p1.3.m3.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I9.i2.p1.3.m3.1c">o^{*}</annotation></semantics></math> denotes the false facts.</p>
</div>
</li>
<li id="A1.I9.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I9.i3.p1" class="ltx_para">
<p id="A1.I9.i3.p1.2" class="ltx_p"><span id="A1.I9.i3.p1.2.1" class="ltx_text ltx_font_bold">Neighborhood Magnitude (NM) <cite class="ltx_cite ltx_citemacro_citep">(Meng et al., <a href="#bib.bib29" title="" class="ltx_ref">2022</a>)</cite>:</span> the differences of <math id="A1.I9.i3.p1.1.m1.1" class="ltx_Math" alttext="\mathbb{P}\left[o^{c}\right]" display="inline"><semantics id="A1.I9.i3.p1.1.m1.1a"><mrow id="A1.I9.i3.p1.1.m1.1.1" xref="A1.I9.i3.p1.1.m1.1.1.cmml"><mi id="A1.I9.i3.p1.1.m1.1.1.3" xref="A1.I9.i3.p1.1.m1.1.1.3.cmml">ℙ</mi><mo lspace="0em" rspace="0em" id="A1.I9.i3.p1.1.m1.1.1.2" xref="A1.I9.i3.p1.1.m1.1.1.2.cmml">​</mo><mrow id="A1.I9.i3.p1.1.m1.1.1.1.1" xref="A1.I9.i3.p1.1.m1.1.1.1.2.cmml"><mo id="A1.I9.i3.p1.1.m1.1.1.1.1.2" xref="A1.I9.i3.p1.1.m1.1.1.1.2.1.cmml">[</mo><msup id="A1.I9.i3.p1.1.m1.1.1.1.1.1" xref="A1.I9.i3.p1.1.m1.1.1.1.1.1.cmml"><mi id="A1.I9.i3.p1.1.m1.1.1.1.1.1.2" xref="A1.I9.i3.p1.1.m1.1.1.1.1.1.2.cmml">o</mi><mi id="A1.I9.i3.p1.1.m1.1.1.1.1.1.3" xref="A1.I9.i3.p1.1.m1.1.1.1.1.1.3.cmml">c</mi></msup><mo id="A1.I9.i3.p1.1.m1.1.1.1.1.3" xref="A1.I9.i3.p1.1.m1.1.1.1.2.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.I9.i3.p1.1.m1.1b"><apply id="A1.I9.i3.p1.1.m1.1.1.cmml" xref="A1.I9.i3.p1.1.m1.1.1"><times id="A1.I9.i3.p1.1.m1.1.1.2.cmml" xref="A1.I9.i3.p1.1.m1.1.1.2"></times><ci id="A1.I9.i3.p1.1.m1.1.1.3.cmml" xref="A1.I9.i3.p1.1.m1.1.1.3">ℙ</ci><apply id="A1.I9.i3.p1.1.m1.1.1.1.2.cmml" xref="A1.I9.i3.p1.1.m1.1.1.1.1"><csymbol cd="latexml" id="A1.I9.i3.p1.1.m1.1.1.1.2.1.cmml" xref="A1.I9.i3.p1.1.m1.1.1.1.1.2">delimited-[]</csymbol><apply id="A1.I9.i3.p1.1.m1.1.1.1.1.1.cmml" xref="A1.I9.i3.p1.1.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.I9.i3.p1.1.m1.1.1.1.1.1.1.cmml" xref="A1.I9.i3.p1.1.m1.1.1.1.1.1">superscript</csymbol><ci id="A1.I9.i3.p1.1.m1.1.1.1.1.1.2.cmml" xref="A1.I9.i3.p1.1.m1.1.1.1.1.1.2">𝑜</ci><ci id="A1.I9.i3.p1.1.m1.1.1.1.1.1.3.cmml" xref="A1.I9.i3.p1.1.m1.1.1.1.1.1.3">𝑐</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I9.i3.p1.1.m1.1c">\mathbb{P}\left[o^{c}\right]</annotation></semantics></math> and <math id="A1.I9.i3.p1.2.m2.1" class="ltx_Math" alttext="\mathbb{P}\left[o^{*}\right]" display="inline"><semantics id="A1.I9.i3.p1.2.m2.1a"><mrow id="A1.I9.i3.p1.2.m2.1.1" xref="A1.I9.i3.p1.2.m2.1.1.cmml"><mi id="A1.I9.i3.p1.2.m2.1.1.3" xref="A1.I9.i3.p1.2.m2.1.1.3.cmml">ℙ</mi><mo lspace="0em" rspace="0em" id="A1.I9.i3.p1.2.m2.1.1.2" xref="A1.I9.i3.p1.2.m2.1.1.2.cmml">​</mo><mrow id="A1.I9.i3.p1.2.m2.1.1.1.1" xref="A1.I9.i3.p1.2.m2.1.1.1.2.cmml"><mo id="A1.I9.i3.p1.2.m2.1.1.1.1.2" xref="A1.I9.i3.p1.2.m2.1.1.1.2.1.cmml">[</mo><msup id="A1.I9.i3.p1.2.m2.1.1.1.1.1" xref="A1.I9.i3.p1.2.m2.1.1.1.1.1.cmml"><mi id="A1.I9.i3.p1.2.m2.1.1.1.1.1.2" xref="A1.I9.i3.p1.2.m2.1.1.1.1.1.2.cmml">o</mi><mo id="A1.I9.i3.p1.2.m2.1.1.1.1.1.3" xref="A1.I9.i3.p1.2.m2.1.1.1.1.1.3.cmml">∗</mo></msup><mo id="A1.I9.i3.p1.2.m2.1.1.1.1.3" xref="A1.I9.i3.p1.2.m2.1.1.1.2.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.I9.i3.p1.2.m2.1b"><apply id="A1.I9.i3.p1.2.m2.1.1.cmml" xref="A1.I9.i3.p1.2.m2.1.1"><times id="A1.I9.i3.p1.2.m2.1.1.2.cmml" xref="A1.I9.i3.p1.2.m2.1.1.2"></times><ci id="A1.I9.i3.p1.2.m2.1.1.3.cmml" xref="A1.I9.i3.p1.2.m2.1.1.3">ℙ</ci><apply id="A1.I9.i3.p1.2.m2.1.1.1.2.cmml" xref="A1.I9.i3.p1.2.m2.1.1.1.1"><csymbol cd="latexml" id="A1.I9.i3.p1.2.m2.1.1.1.2.1.cmml" xref="A1.I9.i3.p1.2.m2.1.1.1.1.2">delimited-[]</csymbol><apply id="A1.I9.i3.p1.2.m2.1.1.1.1.1.cmml" xref="A1.I9.i3.p1.2.m2.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.I9.i3.p1.2.m2.1.1.1.1.1.1.cmml" xref="A1.I9.i3.p1.2.m2.1.1.1.1.1">superscript</csymbol><ci id="A1.I9.i3.p1.2.m2.1.1.1.1.1.2.cmml" xref="A1.I9.i3.p1.2.m2.1.1.1.1.1.2">𝑜</ci><times id="A1.I9.i3.p1.2.m2.1.1.1.1.1.3.cmml" xref="A1.I9.i3.p1.2.m2.1.1.1.1.1.3"></times></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I9.i3.p1.2.m2.1c">\mathbb{P}\left[o^{*}\right]</annotation></semantics></math> for the "neighborhood" subjects.</p>
</div>
</li>
</ul>
</div>
</section>
</section>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Detailed Explanation of experiments for deterioration of general LLM abilities</h2>

<div id="A2.p1" class="ltx_para">
<p id="A2.p1.1" class="ltx_p">We follow the settings of  <cite class="ltx_cite ltx_citemacro_citep">(Gu et al., <a href="#bib.bib13" title="" class="ltx_ref">2024</a>)</cite> for this part of experiments. Different evaluation metrics were applied for each downstream task: Exact Match for open-domain question answering on the Natural Question dataset  <cite class="ltx_cite ltx_citemacro_citep">(Kwiatkowski et al., <a href="#bib.bib23" title="" class="ltx_ref">2019</a>)</cite>, accuracy for sentiment analysis on the SST2 dataset  <cite class="ltx_cite ltx_citemacro_citep">(Socher et al., <a href="#bib.bib35" title="" class="ltx_ref">2013</a>)</cite>, solve rate for reasoning on the GSM8K dataset  <cite class="ltx_cite ltx_citemacro_citep">(Cobbe et al., <a href="#bib.bib7" title="" class="ltx_ref">2021</a>)</cite>, and ROUGE score for summarization on the SAMSum dataset  <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib42" title="" class="ltx_ref">2019</a>)</cite>.</p>
</div>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2406.01435" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2406.01436" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2406.01436">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2406.01436" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2406.01437" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Sat Jul  6 00:46:07 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2406.01436
Knowledge editing is a rising technique for efficiently updating factual knowledge in Large Language Models (LLMs)  with minimal alteration of parameters. However, recent studies have identified concerning side effects, such as knowledge distortion and the deterioration of general abilities, that have emerged after editing. This survey presents a comprehensive study of these side effects, providing a unified view of the challenges associated with knowledge editing in LLMs. We discuss related works and summarize potential research directions to overcome these limitations. Our work highlights the limitations of current knowledge editing methods, emphasizing the need for deeper understanding of inner knowledge structures of LLMs and improved knowledge editing methods. To foster future research, we have released the complementary materials such as paper collection publicly111https://github.com/MiuLab/EditLLM-Survey.**footnotetext:Equal contribution.
abstract
Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models
Abstract
2406.01436
1IntroductionRecent advancements in large language models (LLMs)  have significantly improved Natural Language Processing. LLMs can now understand and generate language at a human-like level, establishing them as milestones in generative AI. This proficiency is due to their ability to extract knowledge from extensive text corpora. However, the mechanisms of knowledge storage in LLMs are not well understood, leading to potential issues in real-world applications. For instance, deploying LLMs as chatbots raises concerns about the reliability of their generated content due to the unclear mechanisms of knowledge storage and potential inaccuracies.Figure 1:An overview of pitfalls in current knowledge editing methods. The subsequent sections dive into three key challenges: generalization issues (Section3.1) , localization issues (Section3.2) , and deterioration of general LLM abilities (Section3.3) .To address these issues, researchers have explored various methods. Traditional approaches like fine-tuning, continual learning, and retraining are either computationally expensive or detrimental to the LLMs’ overall performance. Recently, knowledge editing has emerged as a promising alternative, offering minimal computational costs and fewer alterations. This approach allows for precise adjustments to LLMs, enhancing their practicality and reliability in real-world applications.Knowledge editing can be divided into two main categories: parameter-modifying and parameter-preserving. Both aim to refine LLM knowledge efficiently while avoiding the drawbacks of previous tuning methods. Parameter-modifying methods, including meta-learningand locate-then-edit techniques, strive to update a targeted set of model parameters while making effective edits. Parameter-preserving methods introduce external components, like knowledge basesor extra model parametersto maintain the integrity of pre-trained LLMs while updating their knowledge.Despite the success of knowledge editing, challenges remain. Knowledge editing can have unintended side effects, potentially damaging the general abilities and intrinsic structures of LLMs. Previous research has mainly focused on performance improvements and innovations within knowledge editing methods, with limited attention to potential drawbacks. Consequently, this survey aims to provide a holistic view of current issues in the knowledge editing paradigm and encourage further investigations into the pitfalls and intrinsic knowledge structures of LLMs. A brief overview of the discussed pitfalls is shown in Figure1.This survey is organized as follows: Section2introduces knowledge editing and its methods. Section3discusses current challenges, benchmarks, and methodologies. In Section4, we present experimental results evaluating different editing methods. Finally, Section5explores related studies and future research directions. We summarize our contributions as follows:1.We are the first to provide a comprehensive analysis of the side effects associated with existing knowledge editing techniques.2.We systematically organized previous research and conducted experiments to benchmark the side effects of knowledge editing, providing a unified perspective on this issue.3.We discussed related studies and potential research directions to address existing challenges, encouraging further exploration and understanding in this field.Figure 2:Illustration of properties that knowledge editing methods should satisfy. An ideal knowledge editing method should be reliable, generalizable to relevant queries, and should not alter the outputs of irrelevant queries.Figure 3:Illustration of the two categories of model editing methods in transformer-based large language models, which includes parameter-modifying (meta-learning and locate-and-edit)  and parameter-preserving (additional parameters, external memory, in-context learning, and decoding)  methods. MHSA and FFN stand for multi-head self-attention and feed-forward network, respectively.CategoryStrategyMethodParameter-modifyingMeta-learningKnowledge EditorMENDMALMENLocatingandeditingKnowledge NeuronROMEMEMITPMETEMMETParameter-preservingAdditionalparametersCaliNETT-Patcher†GRACE†MELO†ExternalmemorySERAC†MeLLo†In-contextlearningIKE†DecodingDeepEdit†Table 1:Overview of current knowledge editing methods. The methods are categorized into two major families, namely parameter-modifying and parameter-preserving methods, each containing several strategies. Methods marked withhave the ability to process sequential edits.
introduction
Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models
1Introduction
2406.01436
2Overview of Knowledge Editing2.1Problem DefinitionKnowledge editing for LLMs entails modifying the output of LLMs in response to specific edit queries, with the aim of minimizing alterations to their original behavior. In this section, we follow the notation from.We denote the input and output space asand, respectively. The function spaceis estimated by the base modelparameterized by. Finally, letbe the set of edit queries we would like to apply to the base model.
The goal of knowledge editing is to efficiently derive the edited modelfrom the base model that satisfies the following:(1) The ideal edited modelshould satisfy three properties:reliability,generalization, andlocality. An illustration is shown in Figure2.ReliabilityGiven an edit query, the edited modelshould output the target answerwhen given the target input, i.e.. The reliability of a editing method is measured by calculating the average edit success rate:(2) GeneralizationThe edited model should generalize the edited knowledge to relevant instances. The generalization metric is commonly formulated as the average success rate on the neighboring set:(3) whereis the set of neighboring instances of an edit query. Earlier works evaluate this metric by rephrasing the input prompts.LocalityThe editing process should not affect instances unrelated to the edit queries. The locality set of an edit querycan be defined as. The locality, also known as specificity, of a editing method is measured by calculating the level of invariance of model output before and after the edits, which can be calculated as follows:(4) 2.2Current MethodsIn this section, we introduce the current knowledge editing methods. The methods are categorized into parameter-modifying (Section2.2.1)  and parameter-preserving (Section2.2.2)  editing methods, each containing several strategies. An overview and illustration of current methods are included in Table1and Figure3, respectively.2.2.1Parameter-ModifyingThis category of methods, including meta-learning methods and locate-and-edit strategies, update LLMs’ knowledge by modifying their parameters.Meta-learningMeta-learning methods train a hyper-network to predict the update of network parameters. For instance, KnowledgeEditortrains a deep network to predict weight updates. MENDdecomposes the gradient matrix into two rank-one matrices and utilized a hyper-network to update these matrices, thereby accelerating the editing process. Built upon MEND, MALMENrefines the process by formulating the aggregation of parameter shifts into a least-squares problem, further improving the scalability of meta-learning methods.Locate and EditLocate-and-edit methods identify specific knowledge locations in LLMs for consequent editing. KNutilizes the proposed knowledge attribution method to pinpoint neurons expressing relational facts, allowing efficient updates or erasures without fine-tuning. ROMEproposes the causal tracing method for identifying neuron activations associating with specific knowledge. The authors demonstrate the significance of middle-layer feed-forward networks (FFNs)  in factual predictions when processing the subject’s last token. Built upon the hypothesis that the FFN modules in a transformer layer can be viewed as key-value memories, ROMEinjects new knowledge into the key-value memories by deriving the closed form solution from the least-squares problem. MEMITscales up ROMEby editing a set of MLPs from consecutive middle-layers via solving a normal equation. PMETproposes to update multi-head self-attention (MHSA)  modules in addition to FFNs. EMMETon the other hand, integrates the objectives of ROME and MEMIT into a unified preservation-memorization objective, facilitating batch-editing capabilities for both methodologies.2.2.2Parameter-PreservingParameter-preserving methods alter LLM output by adding new parameters, integrating external memory, or using strategies like in-context learning and devised decoding, without changing pre-trained LLM.Additional ParametersSome methods utilize additional parameters, such as adding new neurons or employing parameter-efficient techniques. CaliNETextends the FFN modules with calibration memory slots to adjust the predicted token distribution. T-Patcheradds neurons in the FFN’s last layer to rectify classification errors and incorrectly generated tokens, activating only in response to associated mistakes. GRACEwraps a selected layer with an Adaptor that includes a codebook and deferral mechanism, learning to decode desired outputs while caching input error embeddings. The GRACE layer stores the edits and could be updated continuously over long deployments. MELOutilizes DyLoRAmodules to learn edits, indexing them in an inner vector database to dynamically activate corresponding LoRA blocks during inference.External MemoryOther methods utilize external memories for editing. SERACleverages a scope classifier to determine whether an user-supplied edit example stored in its memory is related to the inputs. If no example exists, the inputs are passed to the base model; otherwise, a counterfactual model generates modified answers using the inputs and the related example. MeLLodecomposes a multi-hop question into subquestions iteratively. The model then checks if the tentative answer generated by the base model contradicts the most relevant facts retrieved from the edited fact memory and adjusts the outputs accordingly.In-Context Learning and DecodingCertain strategies require no additional parameters. IKEedits factual knowledge via in-context learning with demonstrations to guide the language model. DeepEditemploys decoding constraints, including filtering step candidates, depth-first search to store valid candidates in a stack, and a greedy search to output the optimal path for multi-hop reasoning.
body
Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models
2Overview of Knowledge Editing
2406.01436
3Challenges of Knowledge EditingWhile knowledge editing methods have been extensively researched, there’s a lack of comprehensive study on related challenges. In this section, we discuss the pitfalls of knowledge editing from three perspectives: inability to logically inference and robustly generalize (Section3.1) , unintended alteration of non-target knowledge (Section3.2) , and deterioration of general LLM abilities (Section3.3) .ChallengeBenchmarkMetricPortabilityandGeneralizationRippleEditsLogical GeneralizationCompositionality ICompositionality IIConflictEditConflict ScoreConflict MagnitudeSuccess ScoreMQuAKEEdit-wise Success RateInstance-wise AccuracyMulti-hop AccuracyReCoEQA AccuracyZsRE + CounterFact†Subject-ReplaceReverse-RelationOne-HopLocalityRippleEditsSubject AliasingPreservationRelation SpecificityRoundEditSuccess ScoreDistortion () Ignore Rate () Failure Rate () Tied Fact Damage () CounterFact†Other-AttributionDistract-NeighborOther-TaskCounterFactLocalityNeighborhood ScoreNeighborhood MagnitudeCounterFact+Neighborhood KL DivergenceTable 2:Performance benchmarks and evaluation metrics addressing generalization/portability and locality issues in knowledge editing methods. Unless specifically indicated by a downward arrow, higher values signify better performance in those evaluation metrics. CounterFact benchmark is proposed by, and CounterFact with†mark is modified to further examine the proposed metrics.3.1Inability to Logically Inference and Robustly GeneralizeWhen a fact is updated, it is crucial not only to revise the specific piece of knowledge but also to evaluate the impact on the related reasoning chain. Recently the termportabilityhas been proposed into evaluate the consequences after an edit, and further assess the robustness of generalization. In their study, they introduce three metrics to evaluate portability:Subject Replace(checking if synonyms of the subject are edited) ,Reversed Relation(checking if the reversed relation of the target is edited) , andOne Hop(assessing if modified knowledge is usable for further derivation) . Similarly,RippleEditsbenchmark as well as correspondingLogical GeneralizationandCompositionalitymetrics are proposed to examine whether edited knowledge can be inferred in composite relations of facts. Additionally,ReCoEbenchmark is proposed to assess the propagation of updates in interconnected facts using various reasoning schemes in complex question-answering datasets. Furthermore,MQuAKEbenchmark is introduced to evaluate more complex reasoning and inference ability on multi-hop questions.When editing multiple logically related facts simultaneously, models may suffer from confusion due to conflicts.ConflictEditbenchmark is proposed to examine different editing methods on conflicted edit facts. The different benchmarks and corresponding metrics and are arranged systematically in Table2.3.2Unintended Alteration of Non-Target KnowledgeLocality is conventionally assessed using a locality dataset to evaluate edits on unrelated facts by measuring theNeighborhood ScoreandNeighborhood Magnitude.However, current evaluation methods do not adequately capture the post-edit effects on content beyond the locality dataset, which means the edited model could still contain unintended alterations. For example, while the location of the Louvre might be successfully changed from Paris to London, the edited model may inadvertently increase the likelihood of semantically related words (e.g. Big Ben for London)  while mentioning Louvre. Some modified benchmark (CounterFact+)  and corresponding metric (Neighborhood KL Divergence) is then designed to disclose these previously implicit pitfalls. Another studyextends this exploration to three facets of locality:Other Relations(evaluating the retention of other attributes of the updated subject) ,Distract Neighborhood(assessing the divergence of unrelated input when juxtaposed with edited instances) , andOther Tasks(examining the influence of edits on the performance of other tasks) .Unintended edits to unrelated facts may occur because a single edit can implicitly change the predictive distribution among objects associated with the same (subject-relation)  pair. After multiple consecutive edits, these alterations can accumulate and distort the stored knowledge. To evaluate this condition, the concept ofKnowledge Distortionhas been introduced by, which estimates the Jensen–Shannon divergence of the object set distribution before and after editing. This can be further extended to metrics such as theIgnore Rate, measuring how objects other than the target in the object set are neglected after editing, and theFailure Rate, which measures the proportion of instances where over half of the objects in the set are overlooked.3.3Deterioration of General LLM AbilitiesCurrent evaluation metrics are primarily limited to scenarios where editing is performed only once or infrequently, prompting some studies to extend evaluations to the outcomes after consecutive edits. A study bydiscovers that post-edit models exhibit susceptibility to both gradual forgetting and catastrophic forgetting in sequential editing scenarios. Notably, their findings indicate that the extent of knowledge forgetting is more pronounced in meta-learning-based methods compared to locate-and-edit methods. Additionally, models with parameters modified successively show a decline in performance across various downstream NLP tasks. Furthermore, perplexity is found to increase after consecutive edits across all parameter-modified methods and different LLMs, and is proposed as another metric to indicate model collapse. These findings further corroborate that model editing aimed at modifying parameters adversely affects the general capabilities of the original LLMs.Single EditMultiple EditOne-HopMultiple-HopReverse ConflictComposite ConflictMethodsSRRROHMHCSCMCSCMFT72.968.051.341.680.2871.1175.4564.28MEND42.450.0011.349.288.8960.5084.8543.45ROME37.4246.4250.917.665.92-0.6571.7037.04MEMIT27.7347.6752.748.151.40-1.6057.15-1.50SERAC17.791.305.537.9†50.89†-0.02†50.84†-0.02†IKE88.7792.9655.388.3†58.20†-1.00†50.52†-0.99†Table 3:Experimental results for portability and generalization. SR: Subject-Replace, RR: Reverse-Relation, OH: One-Hop Accuracy, EW: Edit-wise, IW: Instance-wise, MH: Multi-hop Accuracy, CS: Conflict score, CM: Conflict magnitude. Higher values indicate better performance for all metrics in this table. Results marked withare obtained in our own experiments, and other results are taken from previous studies.Single EditMultiple EditMethodsOADNOTSucc.D () IR () FR () FT12.889.4849.56100.016.1297.4897.32MEND73.5032.9648.8699.1214.3587.6486.56ROME78.9450.3552.1299.8013.9578.9877.60MEMIT86.7860.4774.6299.7213.5072.0370.44SERAC99.5039.1874.8450.14†3.78†99.62†99.64†IKE84.1366.0475.33100.0†13.43†73.53†73.00†Table 4:Experimental results for locality. OA: Other-Attribution, DN: Distract-Neighbor, OT: Other-Task, Succ.: Success rate, D: Distortion, IR: Ignore rate, FR: Failure rate, TFD: Tied fact damage. Unless specifically indicated by a downward arrow, higher values signify better performance in those evaluation metrics. Results marked withare obtained in our own experiments, and other results are taken from previous studies.
body
Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models
3Challenges of Knowledge Editing
2406.01436
4ExperimentsThe experiments were done to evaluate robust generalization and locality (Section4.1.1as well as deterioration of general LLM abilities (Section4.1.2across different editing methods.4.1Experimental Setup4.1.1Robust generalization and localityWe use GPT-Jas the baseline model for editing and implement six distinct editing methodologies to assess robust generalization and locality: MEND (meta-learning) , ROME and MEMIT (locate-and-edit) , SERAC (external memory) , and IKE (prompting demonstrations) .Given the overlap in benchmarks for robust generalization and locality, we select a subset for our experiments. Robust generalization is evaluated under single edit and multiple edit settings. Single edit metrics include Subject-Replace, Reverse-Replace, and One-Hop reasoning. Multiple edit metrics include multi-hop editing accuracy, and Conflict Score and Conflict Magnitude for Reverse Conflict and Composite Conflict respectively.For locality, single edit metrics include Other-Attribution, Distract-Neighbor, and Other-Task, while multiple edit metrics encompass Success Rate, Distortion, Ignore Rate, and Failure Rate.4.1.2Deterioration of general LLM abilitiesFollowing the settings of, we assessed deterioration of general LLM abilities post-editing using six methodologies: ROME, MEMIT, SERAC, MEND, KN, and GRACE. We evaluated general abilities across four NLP downstream tasks: open-domain question answering, sentiment analysis, reasoning, and summarization. These tasks were assessed after 10 to 40 edits on the Zero-Shot Relation Extraction (ZsRE)  dataset, comparing the results against pre-editing benchmarks. More details on the selected downstream tasks are in AppendixB.4.2Experimental Results and DiscussionIn general, current editing methodologies exhibit suboptimal performance concerning both robust generalization and locality. Regarding robust generalization, IKE, which leverages prompt demonstrations, demonstrates superior performance over other methodologies in single edit conditions. However, IKE’s performance noticeably declines in multiple edit scenarios, suggesting that prompt demonstrations may become confused when editing multiple logically related facts. Conversely, fine-tuning and meta-learning-based methods are less susceptible to confusion after editing multiple related facts.Regarding locality, IKE maintains stable performance across metrics in single edit settings. Parameter-modifying methods excel in Other Attribution but decline in other metrics, except MEMIT, which remains stable across all metrics. In multiple edit scenarios, all methods except SERAC show similar performance. In the multiple edit scenario, all methods except SERAC exhibit relatively similar performance. SERAC displays low edit success rate and distortion rate, suggesting its scope classifier does not adopt most edits in this scenario. This may be attributed to its weakness in recovering edited facts, which is crucial in this metric setting.In terms of general LLM abilities, the number of edits affects methods differently. Meta-learning methods like MEND degrade significantly after 10-20 edits. Locate-and-edit methods such as ROME and KN degrade after 10 edits, while MEMIT remains stable after 40 edits. This disparity can be attributed to MEMIT’s strategy of adjusting parameters across multiple layers, as opposed to ROME’s single-layer edits and KN’s approach of modifying a few neurons. This distribution of parameter modifications across layers helps mitigate deterioration.GRACE, which stores edited facts with additional parameters, shows no performance change in downstream tasks after edits. One possible explanation is that the edits are conducted on the ZsRE dataset, which is distinct from the requirements of downstream tasks, leading to the stored facts not being retrieved during inference. Similarly, SERAC, utilizing external memory for edited facts, preserves general NLP abilities post-editing. This preservation stems from SERAC being trained once before editing begins, solely performing inference during editing, thereby preventing changes in the model’s output, even after multiple edits.Overall, parameter-modifying methods degrade downstream task performance by altering pre-trained LLM parameters. In contrast, parameter-preserving methods maintain the original parameters, resulting in stable downstream task performance even after multiple edits.(a) Open Domain Question Answering(b) Sentiment Analysis(c) Summarization(d) ReasoningFigure 4:The experimental results for the deterioration of general abilities were obtained by editing a pretrained LLM using GPT-J with various editing algorithms, including ROME, MEMIT, MEND, KN, SERAC, and GRACE, each applied 10 to 40 times. The edited models were subsequently evaluated on four downstream tasks, including open-domain question answering, sentiment analysis, summarization, and reasoning.
body
Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models
4Experiments
2406.01436
5Future Prospects5.1Leveraging Information Retrieval and External MemoryPrevious research has demonstrated the benefits of utilizing external knowledge bases, as opposed to relying solely on internal knowledge, to guide LLMs in generating content based on a predefined set of facts. These methods effectively separate the factual knowledge stored in LLMs from the inference processes, thus reducing potential biases encoded within the models.External knowledge bases can include diverse sources such as extensive text corpora, structured tables, or even simple key-value databases. Once these knowledge sources are provided, one can either finetune the LLMs to enhance their ability to retrieve information or employ prompting and in-context learning techniques to query these sources while keeping the model parameters intact. Such approaches not only eliminate the need to verify and edit false factual knowledge within the LLMs but also facilitate the use of attribution and reflection methods. This ensures that the generated content aligns with the predefined external knowledge base, thereby enhancing both accuracy and accountability.5.2Improve Understandings of LLMs’ Internal Knowledge StructuresWhile the identification of factual knowledge storage in LLMs has been extensively explored in recent literature, the correlation between the location of knowledge and the success rate of model editing remains low. Additionally, despite evidence suggesting a strong connection between factual knowledge and the feed-forward network layers, recent findingsindicate that updates to multi-head self-attention layers also lead to improved outcomes. These studies highlight that merely locating fact storage does not fully elucidate the underlying mechanisms of knowledge structures in LLMs. Therefore, further research into how knowledge locations interact with model predictions is essential for advancing the interpretability and controllability of LLMs.In addition to enhancing the success rate of edits, preserving the general capabilities of LLMs is crucial for assessing the efficacy of model editing methods, as discussed in Section3.3. Recent breakthroughs in identifying regions within models that correlate with general linguistic abilitieshave opened up a direction for future research in model editing. Specifically, by locating these critical areas, it is possible to perform targeted modifications while keeping these regions away from alterations, thereby preventing the deterioration of general abilities. Consequently, advancements in related fields would ensure that edits could be performed without compromising the overall performance of the LLMs, thereby significantly enhancing the specificity and effectiveness of current model editing methods.5.3Improve Robustness of Knowledge EditingEven after successful (achieving fair scores on the existing metrics)  edit, the revised model may refuse the modification if the knowledge concerning the altered concept is challenged by extended dialogues. Instead, it might revert to the pre-edit version (reversion) , or output an ambiguous answer about the edited concept (confusion) . Given the interconnected nature of knowledge, experiments disclose that more popular the knowledge in the benchmark, the easier for the modified model to trace back the original concept. It highlights the unsatisfying robustness of the existing editing strategies. A more comprehensive understanding how LLMs store and process among different knowledge entities is crucial for a more robust editing. We are also short of specific benchmarks and automated metrics addressing on theses issues. Knowledge-focused editing would not avoid thehallucinationinherited from the pre-edit model. TruthXtries to alleviate hallucination via a parameter-preserved approach by mapping the LLMs internal representation to semantic and truthful spaces and edits the truthfulness in the truthful space. Combination of truthfulness and knowledge adjustment in the same space may provide a practical solution.
body
Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models
5Future Prospects
2406.01436
6ConclusionAlthough model editing techniques appear promising for cost-effectively updating knowledge, they still have significant pitfalls. Current editing methods often struggle with making logical inferences based on the edited facts, introducing unintended alterations of non-target knowledge and deterioration in model performance, particularly with parameter-modified methods. Editing techniques that leverage information retrieval can mitigate deviations in model abilities by keeping model parameters intact, as demonstrated in our experiments. Moreover, gaining a deeper understanding of how models store and process knowledge can enhance the controllability of edited facts, leading to greater robustness. We hope our work illuminates potential directions for future improvements in knowledge editing.
conclusion
Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models
6Conclusion