<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2401.17585] Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks</title><meta property="og:description" content="Current approaches of knowledge editing struggle to effectively propagate updates to interconnected facts.
In this work, we delve into the barriers that hinder the appropriate propagation of updated knowledge within th…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2401.17585">

<!--Generated on Tue Feb 27 07:19:17 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document">
<h1 class="ltx_title ltx_title_document">Propagation and Pitfalls: Reasoning-based Assessment of 
<br class="ltx_break">Knowledge Editing through Counterfactual Tasks</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">
Wenyue Hua<sup id="id11.11.id1" class="ltx_sup"><span id="id11.11.id1.1" class="ltx_text ltx_font_italic">‡</span></sup><sup id="id12.12.id2" class="ltx_sup">∗</sup> , Jiang Guo<sup id="id13.13.id3" class="ltx_sup">†</sup><sup id="id14.14.id4" class="ltx_sup">∗</sup> , Mingwen Dong<sup id="id15.15.id5" class="ltx_sup">†</sup>, Henghui Zhu<sup id="id16.16.id6" class="ltx_sup">†</sup>, Patrick Ng<sup id="id17.17.id7" class="ltx_sup">†</sup>, Zhiguo Wang<sup id="id18.18.id8" class="ltx_sup">†</sup>
<br class="ltx_break"><sup id="id19.19.id9" class="ltx_sup">‡</sup>Rutgers University, New Brunswick    
<sup id="id20.20.id10" class="ltx_sup">†</sup>AWS AI Labs
<br class="ltx_break">wenyue.hua@rutgers.edu,
<br class="ltx_break">{gujiang, mingwd, henghui, patricng, zhiguow}@amazon.com
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id21.id1" class="ltx_p">Current approaches of knowledge editing struggle to effectively propagate updates to interconnected facts.
In this work, we delve into the barriers that hinder the appropriate propagation of updated knowledge within these models for accurate reasoning.
To support our analysis, we introduce a novel reasoning-based benchmark – <span id="id21.id1.1" class="ltx_text ltx_font_bold">ReCoE</span> (<span id="id21.id1.2" class="ltx_text ltx_font_bold">Re</span>asoning-based <span id="id21.id1.3" class="ltx_text ltx_font_bold">Co</span>unterfactual <span id="id21.id1.4" class="ltx_text ltx_font_bold">E</span>diting dataset) – which covers six common reasoning schemes in real world.
We conduct a thorough analysis of existing knowledge editing techniques, including input-augmentation, finetuning, and locate-and-edit.
We found that all model editing methods show notably low performance on this dataset, especially in certain reasoning schemes.
Our analysis over the chain-of-thought generation of edited models further uncover key reasons behind the inadequacy of existing knowledge editing methods from a reasoning standpoint, involving aspects on <span id="id21.id1.5" class="ltx_text ltx_font_italic">fact-wise editing</span>, <span id="id21.id1.6" class="ltx_text ltx_font_italic">fact recall</span> ability, and <span id="id21.id1.7" class="ltx_text ltx_font_italic">coherence</span> in generation. We will make our benchmark publicly available.</p>
</div>
<div id="p1" class="ltx_para">
<span id="p1.1" class="ltx_ERROR undefined">\pdfcolInitStack</span>
<p id="p1.2" class="ltx_p">tcb@breakable



























































































</p>
</div>
<div id="p2" class="ltx_para ltx_noindent">
<div id="p2.10" class="ltx_block ltx_align_bottom">
<p id="p2.10.11" class="ltx_p"><span id="p2.10.11.1" class="ltx_text ltx_font_bold">Propagation and Pitfalls: Reasoning-based Assessment of 
<br class="ltx_break">Knowledge Editing through Counterfactual Tasks</span></p>
<br class="ltx_break ltx_centering">
<p id="p2.10.10" class="ltx_p ltx_align_center" style="width:433.6pt;"><span id="p2.10.10.10" class="ltx_text ltx_inline-block" style="width:0.0pt;">

<span id="p2.10.10.10.10" class="ltx_tabular ltx_guessed_headers ltx_align_top">
<span class="ltx_thead">
<span id="p2.8.8.8.8.8" class="ltx_tr">
<span id="p2.8.8.8.8.8.8" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="p2.8.8.8.8.8.8.8" class="ltx_text ltx_font_bold">
Wenyue Hua<sup id="p2.8.8.8.8.8.8.8.1" class="ltx_sup"><span id="p2.8.8.8.8.8.8.8.1.1" class="ltx_text ltx_font_medium ltx_font_italic">‡</span></sup><sup id="p2.8.8.8.8.8.8.8.2" class="ltx_sup"><span id="p2.8.8.8.8.8.8.8.2.1" class="ltx_text ltx_font_medium">∗</span></sup> , Jiang Guo<sup id="p2.8.8.8.8.8.8.8.3" class="ltx_sup"><span id="p2.8.8.8.8.8.8.8.3.1" class="ltx_text ltx_font_medium">†</span></sup><sup id="p2.8.8.8.8.8.8.8.4" class="ltx_sup"><span id="p2.8.8.8.8.8.8.8.4.1" class="ltx_text ltx_font_medium">∗</span></sup> , Mingwen Dong<sup id="p2.8.8.8.8.8.8.8.5" class="ltx_sup"><span id="p2.8.8.8.8.8.8.8.5.1" class="ltx_text ltx_font_medium">†</span></sup>, Henghui Zhu<sup id="p2.8.8.8.8.8.8.8.6" class="ltx_sup"><span id="p2.8.8.8.8.8.8.8.6.1" class="ltx_text ltx_font_medium">†</span></sup>, Patrick Ng<sup id="p2.8.8.8.8.8.8.8.7" class="ltx_sup"><span id="p2.8.8.8.8.8.8.8.7.1" class="ltx_text ltx_font_medium">†</span></sup>, Zhiguo Wang<sup id="p2.8.8.8.8.8.8.8.8" class="ltx_sup"><span id="p2.8.8.8.8.8.8.8.8.1" class="ltx_text ltx_font_medium">†</span></sup></span></span></span>
<span id="p2.10.10.10.10.10" class="ltx_tr">
<span id="p2.10.10.10.10.10.2" class="ltx_td ltx_align_center ltx_th ltx_th_column"><sup id="p2.10.10.10.10.10.2.1" class="ltx_sup">‡</sup>Rutgers University, New Brunswick    
<sup id="p2.10.10.10.10.10.2.2" class="ltx_sup">†</sup>AWS AI Labs</span></span>
</span>
<span class="ltx_tbody">
<span id="p2.10.10.10.10.11.1" class="ltx_tr">
<span id="p2.10.10.10.10.11.1.1" class="ltx_td ltx_align_center">wenyue.hua@rutgers.edu,</span></span>
<span id="p2.10.10.10.10.12.2" class="ltx_tr">
<span id="p2.10.10.10.10.12.2.1" class="ltx_td ltx_align_center">{gujiang, mingwd, henghui, patricng, zhiguow}@amazon.com</span></span>
</span>
</span></span></p>
<br class="ltx_break ltx_centering">
</div>
</div>
<span id="footnotex1" class="ltx_note ltx_role_footnotetext"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_note_type">footnotetext: </span>Equal contribution. Work done while Wenyue Hua was an intern at AWS AI Labs.</span></span></span>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Contemporary language models demonstrate a remarkable capacity to encode extensive factual information, rendering them highly useful as a knowledge base for real-world applications. Yet, the challenge of rapidly outdated knowledge persists, giving rise to a wide range of methods for knowledge updating, such as in-context learning <cite class="ltx_cite ltx_citemacro_cite">Vu et al. (<a href="#bib.bib25" title="" class="ltx_ref">2023</a>)</cite>, continual pretraining <cite class="ltx_cite ltx_citemacro_cite">Zhu et al. (<a href="#bib.bib31" title="" class="ltx_ref">2020a</a>)</cite>, locate-and-edit <cite class="ltx_cite ltx_citemacro_cite">Meng et al. (<a href="#bib.bib16" title="" class="ltx_ref">2022a</a>, <a href="#bib.bib17" title="" class="ltx_ref">b</a>)</cite>, and meta-learning <cite class="ltx_cite ltx_citemacro_cite">Mitchell et al. (<a href="#bib.bib18" title="" class="ltx_ref">2021</a>)</cite>.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2401.17585/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="233" height="187" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>An example of reasoning-based assessment for knowledge editing. Existing methods perform well at answering the question of the edited fact, but fail on reasoning with the edited fact.</figcaption>
</figure>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">Despite the success of fact-wise editing, recent studies <cite class="ltx_cite ltx_citemacro_cite">Zhong et al. (<a href="#bib.bib30" title="" class="ltx_ref">2023</a>); Onoe et al. (<a href="#bib.bib21" title="" class="ltx_ref">2023</a>); Pinter and Elhadad (<a href="#bib.bib22" title="" class="ltx_ref">2023</a>)</cite> show that current model editing methods struggle to effectively propagate updates to interconnected facts.
However, the reason behind the ineffective knowledge propagation remains largely unexplored. Our observation in fact-editing experiments indicates that models respond unexpectedly post-editing. For instance, upon applying MEMIT <cite class="ltx_cite ltx_citemacro_cite">Meng et al. (<a href="#bib.bib17" title="" class="ltx_ref">2022b</a>)</cite> for fact editing, the model was unable to reliably recall pertinent edited information and produces an incoherent chain-of-thought (CoT) <cite class="ltx_cite ltx_citemacro_cite">Wei et al. (<a href="#bib.bib28" title="" class="ltx_ref">2022</a>)</cite> for answering the question, as demonstrated in the MEMIT-based generation example in Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">In this work, we undertake an in-depth analysis of this phenomenon, concentrating on three essential competencies necessary for knowledge propagation on reasoning questions after model editing. We place special emphasis on analyzing the results of CoT prompting, which provides explicit reasoning steps that facilitate easier examination. Specifically, in the proposed analytical framework, we measure: (1) effectiveness of editing individual facts, (2) accuracy in recalling relevant facts, and (3) logical coherence of the thought process.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">To facilitate our investigation, we introduce a <span id="S1.p4.1.1" class="ltx_text ltx_font_bold">Re</span>asoning-based <span id="S1.p4.1.2" class="ltx_text ltx_font_bold">Co</span>unterfactual <span id="S1.p4.1.3" class="ltx_text ltx_font_bold">E</span>diting dataset – <span id="S1.p4.1.4" class="ltx_text ltx_font_bold">ReCoE</span>, which covers six different reasoning schemes: <span id="S1.p4.1.5" class="ltx_text ltx_font_italic">superlative</span>, <span id="S1.p4.1.6" class="ltx_text ltx_font_italic">comparative</span>, <span id="S1.p4.1.7" class="ltx_text ltx_font_italic">sorting</span>, <span id="S1.p4.1.8" class="ltx_text ltx_font_italic">counting</span>, <span id="S1.p4.1.9" class="ltx_text ltx_font_italic">aggregation</span>, and <span id="S1.p4.1.10" class="ltx_text ltx_font_italic">subtraction</span>.
This dataset is designed to more accurately capture the complexities inherent in fact editing tasks.
In contrast to existing knowledge editing benchmarks which predominantly rely on synthetic data <cite class="ltx_cite ltx_citemacro_cite">Zhong et al. (<a href="#bib.bib30" title="" class="ltx_ref">2023</a>); Meng et al. (<a href="#bib.bib16" title="" class="ltx_ref">2022a</a>)</cite>, ReCoE incorporates reasoning questions that exhibit a higher degree of naturalness and more accurately reflect the reasoning-based queries encountered in real-world scenarios.</p>
</div>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.1" class="ltx_p">We first explored input-augmentation, an approach where new facts are added (prepended) only in-context, as an informal upper bound of model editing methods.
We then examined model editing methods including finetuning and MEMIT on the Tülu series <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a href="#bib.bib27" title="" class="ltx_ref">2023b</a>)</cite>, which are Llama-based instruction-tuned models of varying sizes.
Results show that all model editing methods achieve notably low performance on the ReCoE benchmark, especially in certain reasoning schemes, with scores close to zero.
Our analysis further unravels the effect of various knowledge editing methods on the reasoning abilities of language models.
We demonstrate that all editing methods result in a significant reduction in fact recall, indicating a key obstacle in effective utilization of the edited knowledge.
Surprisingly, models edited through locate-and-edit methods (i.e., MEMIT) exhibit a severe decline in their generation coherence, leading to nonsensical outputs, which suggests a substantial deterioration in their fundamental language modeling abilities.
We summarize our contributions of the paper as follows:</p>
<ul id="S1.I1" class="ltx_itemize">
<li id="S1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i1.p1" class="ltx_para">
<p id="S1.I1.i1.p1.1" class="ltx_p">We introduce a reasoning-based framework of assessing knowledge editing methods, covering key aspects that enables effective reasoning. Our analysis uncovers essential insights regarding the challenges and limitations associated with knowledge propagation.</p>
</div>
</li>
<li id="S1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i2.p1" class="ltx_para">
<p id="S1.I1.i2.p1.1" class="ltx_p">We introduce ReCoE, a novel yet challenging reasoning-based counterfactual editing benchmark covering a diverse set of reasoning schemes centered on real-world scenarios.</p>
</div>
</li>
</ul>
</div>
<figure id="S1.F2" class="ltx_figure"><img src="/html/2401.17585/assets/x2.png" id="S1.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="348" height="160" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>Step-by-step demonstration on dataset construction. There are three different types of lines in the diagram: (1) straight lines represent data sourced from existing datasets; (2) dashed lines denote data derived from Claude-generation processes; (3) zigzag lines symbolize data obtained through the corruption of other data. Group 1 includes superlative, comparative, and sorting questions, where we use “swapping” to create counterfactual facts. Group 2 represents counting, aggregation, and subtraction questions, where we use “altering” to create counterfactual facts. During the editing phase, data is processed either in text form or atomic triplets, contingent upon the specific requirements of the editor.</figcaption>
</figure>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related Work</h2>

<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>Model Editing Methods</h3>

<div id="S2.SS1.p1" class="ltx_para">
<p id="S2.SS1.p1.1" class="ltx_p">Existing model editing methods generally fall into four main categories <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a href="#bib.bib26" title="" class="ltx_ref">2023a</a>)</cite>.</p>
</div>
<section id="S2.SS1.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Finetuning-based methods</h5>

<div id="S2.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="S2.SS1.SSS0.Px1.p1.1" class="ltx_p">These techniques further finetune the model on new knowledge while minimizing the change in models and catastrophic forgetting. Examples of finetuning-based methods include <cite class="ltx_cite ltx_citemacro_cite">Zhu et al. (<a href="#bib.bib31" title="" class="ltx_ref">2020a</a>); Chen et al. (<a href="#bib.bib1" title="" class="ltx_ref">2020</a>); Zhu et al. (<a href="#bib.bib32" title="" class="ltx_ref">2020b</a>)</cite>.</p>
</div>
</section>
<section id="S2.SS1.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Machine learning framing methods</h5>

<div id="S2.SS1.SSS0.Px2.p1" class="ltx_para">
<p id="S2.SS1.SSS0.Px2.p1.1" class="ltx_p">These approaches treat the editing as a machine learning challenge. They learn hypernetworks (optimizers) to process model gradients. The goal is to produce an updated model that offers the desired output for the edited point while ensuring minimal prediction changes for other data points. Notable methods include MEND <cite class="ltx_cite ltx_citemacro_cite">Mitchell et al. (<a href="#bib.bib18" title="" class="ltx_ref">2021</a>)</cite>, KnowledgeEditor <cite class="ltx_cite ltx_citemacro_cite">De Cao et al. (<a href="#bib.bib4" title="" class="ltx_ref">2021</a>)</cite>, SLAG <cite class="ltx_cite ltx_citemacro_cite">Hase et al. (<a href="#bib.bib10" title="" class="ltx_ref">2021</a>)</cite>, and CaMeLS<cite class="ltx_cite ltx_citemacro_cite">Hu et al. (<a href="#bib.bib11" title="" class="ltx_ref">2023</a>)</cite>.</p>
</div>
</section>
<section id="S2.SS1.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Interpretability-centric methods</h5>

<div id="S2.SS1.SSS0.Px3.p1" class="ltx_para">
<p id="S2.SS1.SSS0.Px3.p1.1" class="ltx_p">These methods focus on model interpretability. The objective is to pinpoint specific layers and parameters that primarily function for knowledge storage <cite class="ltx_cite ltx_citemacro_cite">Dai et al. (<a href="#bib.bib3" title="" class="ltx_ref">2021</a>)</cite>. Once identified, these parameters are then edited, viewing them as linear associative memory storage units. ROME and MEMIT <cite class="ltx_cite ltx_citemacro_cite">Meng et al. (<a href="#bib.bib16" title="" class="ltx_ref">2022a</a>, <a href="#bib.bib17" title="" class="ltx_ref">b</a>)</cite> are prominent examples.</p>
</div>
</section>
<section id="S2.SS1.SSS0.Px4" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Retrieval-augmented methods</h5>

<div id="S2.SS1.SSS0.Px4.p1" class="ltx_para">
<p id="S2.SS1.SSS0.Px4.p1.1" class="ltx_p">These techniques leverage retrieval-augmentation to update knowledge in prompting <cite class="ltx_cite ltx_citemacro_cite">Vu et al. (<a href="#bib.bib25" title="" class="ltx_ref">2023</a>)</cite>. SERAC <cite class="ltx_cite ltx_citemacro_cite">Mitchell et al. (<a href="#bib.bib19" title="" class="ltx_ref">2022</a>)</cite> and MeLLo <cite class="ltx_cite ltx_citemacro_cite">Zhong et al. (<a href="#bib.bib30" title="" class="ltx_ref">2023</a>)</cite> store new knowledge in memory. When relevant queries arise, they retrieve the pertinent knowledge from this storage, employing input augmentation to adjust the response.</p>
</div>
</section>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>Model Editing Benchmarks</h3>

<div id="S2.SS2.p1" class="ltx_para">
<p id="S2.SS2.p1.1" class="ltx_p">Several benchmarks have been introduced to assess the efficacy of model editing. <cite class="ltx_cite ltx_citemacro_citet">Meng et al. (<a href="#bib.bib16" title="" class="ltx_ref">2022a</a>)</cite> introduced the COUNTERFACT dataset, specifically designed to evaluate the successful incorporation of counterfactual knowledge. This evaluation is segmented into three main criteria: (1) Efficacy determines if a particular piece of knowledge has been successfully integrated into the model, typically measured by conditional generation that mirrors the exact text used for editing (2) Paraphrase assesses the model’s capability to generalize to paraphrased versions of the editing text, ensuring the retention of the original knowledge <cite class="ltx_cite ltx_citemacro_cite">Meng et al. (<a href="#bib.bib16" title="" class="ltx_ref">2022a</a>)</cite> (3) Specificity ensures that the model remains unchanged with respect to irrelevant knowledge. There are many other datasets including Zero-Shot Relation Extraction (zsRE) <cite class="ltx_cite ltx_citemacro_cite">Mitchell et al. (<a href="#bib.bib18" title="" class="ltx_ref">2021</a>)</cite>, WikiGen <cite class="ltx_cite ltx_citemacro_cite">Mitchell et al. (<a href="#bib.bib18" title="" class="ltx_ref">2021</a>)</cite>, T-REx-100 &amp; T-REx-1000 <cite class="ltx_cite ltx_citemacro_cite">Elsahar et al. (<a href="#bib.bib8" title="" class="ltx_ref">2018</a>); Dong et al. (<a href="#bib.bib6" title="" class="ltx_ref">2022</a>)</cite>, MMEdit <cite class="ltx_cite ltx_citemacro_cite">Cheng et al. (<a href="#bib.bib2" title="" class="ltx_ref">2023</a>)</cite> (multi-modal model editing), <em id="S2.SS2.p1.1.1" class="ltx_emph ltx_font_italic">etc</em>.</p>
</div>
<div id="S2.SS2.p2" class="ltx_para">
<p id="S2.SS2.p2.1" class="ltx_p">To evaluate knowledge propagation, <cite class="ltx_cite ltx_citemacro_citet">Onoe et al. (<a href="#bib.bib21" title="" class="ltx_ref">2023</a>)</cite> and <cite class="ltx_cite ltx_citemacro_citet">Zhong et al. (<a href="#bib.bib30" title="" class="ltx_ref">2023</a>)</cite> introduced ECBD and MQuAKE benchmarks respectively. ECBD measures the perplexity of a passage relevant to target knowledge (entity-relevant) before and after editing. Though it presents the difficulty of knowledge propagation, the context it evaluates on has a non-deterministic connection to the edited knowledge. MQuAKE employs multi-hop QA questions to gauge the model’s accuracy after editing part or all of its reasoning components. However, it exclusively encompasses compositional questions generated from ChatGPT wherein the precise segments of knowledge required for effective propagation are overtly articulated within the question. Such format may not necessarily mirror real-world scenarios where the reasoning component could be implicit from the question.</p>
</div>
<div id="S2.SS2.p3" class="ltx_para">
<p id="S2.SS2.p3.1" class="ltx_p">In our research, we focus on factual knowledge editing and tackle the prevailing limitations on propagation observed in contemporary benchmarks. Our benchmark ReCoE incorporates a diverse set of 6 reasoning schemes, featuring more organic queries that mirror real-world scenarios. Additionally, we employ a reasoning-based framework to elucidate the underlying challenges of knowledge propagation. Drawing from our discoveries, we aspire to provide valuable insights that will guide and shape the future trajectories of model editing techniques in a more informed manner.</p>
</div>
<figure id="S2.T1" class="ltx_table">
<div id="S2.T1.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:455.2pt;height:183.8pt;vertical-align:-0.8pt;"><span class="ltx_transformed_inner" style="transform:translate(-63.5pt,25.5pt) scale(0.781943644522512,0.781943644522512) ;">
<table id="S2.T1.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S2.T1.1.1.1.1" class="ltx_tr">
<th id="S2.T1.1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="S2.T1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Scheme</span></th>
<th id="S2.T1.1.1.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="S2.T1.1.1.1.1.2.1" class="ltx_text ltx_font_bold">Example QA pair</span></th>
<th id="S2.T1.1.1.1.1.3" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="S2.T1.1.1.1.1.3.1" class="ltx_text ltx_font_bold">Data source</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S2.T1.1.1.2.1" class="ltx_tr">
<td id="S2.T1.1.1.2.1.1" class="ltx_td ltx_align_left ltx_border_t" rowspan="2"><span id="S2.T1.1.1.2.1.1.1" class="ltx_text">superlative</span></td>
<td id="S2.T1.1.1.2.1.2" class="ltx_td ltx_align_left ltx_border_t">Question: What is the largest city in the province of British</td>
<td id="S2.T1.1.1.2.1.3" class="ltx_td ltx_align_left ltx_border_t" rowspan="2"><span id="S2.T1.1.1.2.1.3.1" class="ltx_text">Existent datasets</span></td>
</tr>
<tr id="S2.T1.1.1.3.2" class="ltx_tr">
<td id="S2.T1.1.1.3.2.1" class="ltx_td ltx_align_left">Columbia, Canada? <span id="S2.T1.1.1.3.2.1.1" class="ltx_text ltx_font_italic">Answer: Vancouver</span>
</td>
</tr>
<tr id="S2.T1.1.1.4.3" class="ltx_tr">
<td id="S2.T1.1.1.4.3.1" class="ltx_td ltx_align_left ltx_border_t" rowspan="2"><span id="S2.T1.1.1.4.3.1.1" class="ltx_text">sorting</span></td>
<td id="S2.T1.1.1.4.3.2" class="ltx_td ltx_align_left ltx_border_t">Question: Sort the following cities based on their city size from small</td>
<td id="S2.T1.1.1.4.3.3" class="ltx_td ltx_align_left ltx_border_t" rowspan="2"><span id="S2.T1.1.1.4.3.3.1" class="ltx_text">Claude-generation based on superlative</span></td>
</tr>
<tr id="S2.T1.1.1.5.4" class="ltx_tr">
<td id="S2.T1.1.1.5.4.1" class="ltx_td ltx_align_left">to large: Nanaimo, Victoria, Seattle. <span id="S2.T1.1.1.5.4.1.1" class="ltx_text ltx_font_italic">Answer: Victoria, Nanaimo, Seattle</span>
</td>
</tr>
<tr id="S2.T1.1.1.6.5" class="ltx_tr">
<td id="S2.T1.1.1.6.5.1" class="ltx_td ltx_align_left ltx_border_t" rowspan="2"><span id="S2.T1.1.1.6.5.1.1" class="ltx_text">comparative</span></td>
<td id="S2.T1.1.1.6.5.2" class="ltx_td ltx_align_left ltx_border_t">Question: What is the name of the distilled spirit that has an alcohol</td>
<td id="S2.T1.1.1.6.5.3" class="ltx_td ltx_align_left ltx_border_t">Existent Dataset,</td>
</tr>
<tr id="S2.T1.1.1.7.6" class="ltx_tr">
<td id="S2.T1.1.1.7.6.1" class="ltx_td ltx_align_left">content less than or equal to 35.0? <span id="S2.T1.1.1.7.6.1.1" class="ltx_text ltx_font_italic">Answer: Mekhong</span>
</td>
<td id="S2.T1.1.1.7.6.2" class="ltx_td ltx_align_left">Claude-generation based on sorting</td>
</tr>
<tr id="S2.T1.1.1.8.7" class="ltx_tr">
<td id="S2.T1.1.1.8.7.1" class="ltx_td ltx_align_left ltx_border_t" rowspan="2"><span id="S2.T1.1.1.8.7.1.1" class="ltx_text">counting</span></td>
<td id="S2.T1.1.1.8.7.2" class="ltx_td ltx_align_left ltx_border_t">Question: How many symphonies were composed by Ludwig van</td>
<td id="S2.T1.1.1.8.7.3" class="ltx_td ltx_align_left ltx_border_t">Manually written,</td>
</tr>
<tr id="S2.T1.1.1.9.8" class="ltx_tr">
<td id="S2.T1.1.1.9.8.1" class="ltx_td ltx_align_left">Beethoven? <span id="S2.T1.1.1.9.8.1.1" class="ltx_text ltx_font_italic">Answer: 9</span>
</td>
<td id="S2.T1.1.1.9.8.2" class="ltx_td ltx_align_left">Claude-generation</td>
</tr>
<tr id="S2.T1.1.1.10.9" class="ltx_tr">
<td id="S2.T1.1.1.10.9.1" class="ltx_td ltx_align_left ltx_border_t" rowspan="2"><span id="S2.T1.1.1.10.9.1.1" class="ltx_text">aggregation</span></td>
<td id="S2.T1.1.1.10.9.2" class="ltx_td ltx_align_left ltx_border_t">Question: How many states/provinces are there in North America,</td>
<td id="S2.T1.1.1.10.9.3" class="ltx_td ltx_align_left ltx_border_t" rowspan="2"><span id="S2.T1.1.1.10.9.3.1" class="ltx_text">Claude-generation based on counting</span></td>
</tr>
<tr id="S2.T1.1.1.11.10" class="ltx_tr">
<td id="S2.T1.1.1.11.10.1" class="ltx_td ltx_align_left">i.e. United States, Canada, and Mexico? <span id="S2.T1.1.1.11.10.1.1" class="ltx_text ltx_font_italic">Answer: 92</span>
</td>
</tr>
<tr id="S2.T1.1.1.12.11" class="ltx_tr">
<td id="S2.T1.1.1.12.11.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t" rowspan="2"><span id="S2.T1.1.1.12.11.1.1" class="ltx_text">subtraction</span></td>
<td id="S2.T1.1.1.12.11.2" class="ltx_td ltx_align_left ltx_border_t">Question: How many provinces does Mexico have more than Canada?</td>
<td id="S2.T1.1.1.12.11.3" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t" rowspan="2"><span id="S2.T1.1.1.12.11.3.1" class="ltx_text">Claude-generation based on counting</span></td>
</tr>
<tr id="S2.T1.1.1.13.12" class="ltx_tr">
<td id="S2.T1.1.1.13.12.1" class="ltx_td ltx_align_left ltx_border_bb"><span id="S2.T1.1.1.13.12.1.1" class="ltx_text ltx_font_italic">Answer: 22</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>Example question-answer pair and data source of each reasoning scheme. Existent datasets include GrailQA <cite class="ltx_cite ltx_citemacro_cite">Gu et al. (<a href="#bib.bib9" title="" class="ltx_ref">2021</a>)</cite>, NaturalQuestions <cite class="ltx_cite ltx_citemacro_cite">Kwiatkowski et al. (<a href="#bib.bib15" title="" class="ltx_ref">2019</a>)</cite>, ComplexWebQuestions <cite class="ltx_cite ltx_citemacro_cite">Talmor and Berant (<a href="#bib.bib24" title="" class="ltx_ref">2018</a>)</cite>, FreebaseQA <cite class="ltx_cite ltx_citemacro_cite">Jiang et al. (<a href="#bib.bib14" title="" class="ltx_ref">2019</a>)</cite>.</figcaption>
</figure>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>ReCoE Dataset</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.1" class="ltx_p">We employ a hybrid-synthetic approach that combines existing complex QA datasets and LLM-assisted data synthesizing to create the ReCoE dataset.
The dataset is designed to evaluate counterfactual editing across a broad spectrum of reasoning schemes: <span id="S3.p1.1.1" class="ltx_text ltx_font_italic">superlative</span>, <span id="S3.p1.1.2" class="ltx_text ltx_font_italic">comparative</span>, <span id="S3.p1.1.3" class="ltx_text ltx_font_italic">sorting</span>, <span id="S3.p1.1.4" class="ltx_text ltx_font_italic">counting</span>, <span id="S3.p1.1.5" class="ltx_text ltx_font_italic">aggregation</span>, and <span id="S3.p1.1.6" class="ltx_text ltx_font_italic">subtraction</span>.</p>
</div>
<div id="S3.p2" class="ltx_para">
<p id="S3.p2.4" class="ltx_p">A typical datapoint within this dataset encapsulates five key components:</p>
<ul id="S3.I1" class="ltx_itemize">
<li id="S3.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I1.i1.p1" class="ltx_para">
<p id="S3.I1.i1.p1.1" class="ltx_p"><math id="S3.I1.i1.p1.1.m1.1" class="ltx_Math" alttext="\mathrm{Q}" display="inline"><semantics id="S3.I1.i1.p1.1.m1.1a"><mi mathvariant="normal" id="S3.I1.i1.p1.1.m1.1.1" xref="S3.I1.i1.p1.1.m1.1.1.cmml">Q</mi><annotation-xml encoding="MathML-Content" id="S3.I1.i1.p1.1.m1.1b"><ci id="S3.I1.i1.p1.1.m1.1.1.cmml" xref="S3.I1.i1.p1.1.m1.1.1">Q</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i1.p1.1.m1.1c">\mathrm{Q}</annotation></semantics></math>: Question that corresponds to each of our defined reasoning schemes</p>
</div>
</li>
<li id="S3.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I1.i2.p1" class="ltx_para">
<p id="S3.I1.i2.p1.1" class="ltx_p"><math id="S3.I1.i2.p1.1.m1.1" class="ltx_Math" alttext="\mathrm{A}" display="inline"><semantics id="S3.I1.i2.p1.1.m1.1a"><mi mathvariant="normal" id="S3.I1.i2.p1.1.m1.1.1" xref="S3.I1.i2.p1.1.m1.1.1.cmml">A</mi><annotation-xml encoding="MathML-Content" id="S3.I1.i2.p1.1.m1.1b"><ci id="S3.I1.i2.p1.1.m1.1.1.cmml" xref="S3.I1.i2.p1.1.m1.1.1">A</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i2.p1.1.m1.1c">\mathrm{A}</annotation></semantics></math>: Answer with aliases</p>
</div>
</li>
<li id="S3.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I1.i3.p1" class="ltx_para">
<p id="S3.I1.i3.p1.2" class="ltx_p"><math id="S3.I1.i3.p1.1.m1.1" class="ltx_Math" alttext="\mathbb{F}" display="inline"><semantics id="S3.I1.i3.p1.1.m1.1a"><mi id="S3.I1.i3.p1.1.m1.1.1" xref="S3.I1.i3.p1.1.m1.1.1.cmml">𝔽</mi><annotation-xml encoding="MathML-Content" id="S3.I1.i3.p1.1.m1.1b"><ci id="S3.I1.i3.p1.1.m1.1.1.cmml" xref="S3.I1.i3.p1.1.m1.1.1">𝔽</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i3.p1.1.m1.1c">\mathbb{F}</annotation></semantics></math>: Set of facts that supports the answer (<math id="S3.I1.i3.p1.2.m2.1" class="ltx_Math" alttext="\mathrm{A}" display="inline"><semantics id="S3.I1.i3.p1.2.m2.1a"><mi mathvariant="normal" id="S3.I1.i3.p1.2.m2.1.1" xref="S3.I1.i3.p1.2.m2.1.1.cmml">A</mi><annotation-xml encoding="MathML-Content" id="S3.I1.i3.p1.2.m2.1b"><ci id="S3.I1.i3.p1.2.m2.1.1.cmml" xref="S3.I1.i3.p1.2.m2.1.1">A</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i3.p1.2.m2.1c">\mathrm{A}</annotation></semantics></math>)</p>
</div>
</li>
<li id="S3.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I1.i4.p1" class="ltx_para">
<p id="S3.I1.i4.p1.1" class="ltx_p"><math id="S3.I1.i4.p1.1.m1.1" class="ltx_Math" alttext="\mathrm{CA}" display="inline"><semantics id="S3.I1.i4.p1.1.m1.1a"><mi id="S3.I1.i4.p1.1.m1.1.1" xref="S3.I1.i4.p1.1.m1.1.1.cmml">CA</mi><annotation-xml encoding="MathML-Content" id="S3.I1.i4.p1.1.m1.1b"><ci id="S3.I1.i4.p1.1.m1.1.1.cmml" xref="S3.I1.i4.p1.1.m1.1.1">CA</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i4.p1.1.m1.1c">\mathrm{CA}</annotation></semantics></math>: Counterfactual answer with aliases</p>
</div>
</li>
<li id="S3.I1.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I1.i5.p1" class="ltx_para">
<p id="S3.I1.i5.p1.2" class="ltx_p"><math id="S3.I1.i5.p1.1.m1.1" class="ltx_Math" alttext="\mathbb{CF}" display="inline"><semantics id="S3.I1.i5.p1.1.m1.1a"><mrow id="S3.I1.i5.p1.1.m1.1.1" xref="S3.I1.i5.p1.1.m1.1.1.cmml"><mi id="S3.I1.i5.p1.1.m1.1.1.2" xref="S3.I1.i5.p1.1.m1.1.1.2.cmml">ℂ</mi><mo lspace="0em" rspace="0em" id="S3.I1.i5.p1.1.m1.1.1.1" xref="S3.I1.i5.p1.1.m1.1.1.1.cmml">​</mo><mi id="S3.I1.i5.p1.1.m1.1.1.3" xref="S3.I1.i5.p1.1.m1.1.1.3.cmml">𝔽</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.I1.i5.p1.1.m1.1b"><apply id="S3.I1.i5.p1.1.m1.1.1.cmml" xref="S3.I1.i5.p1.1.m1.1.1"><times id="S3.I1.i5.p1.1.m1.1.1.1.cmml" xref="S3.I1.i5.p1.1.m1.1.1.1"></times><ci id="S3.I1.i5.p1.1.m1.1.1.2.cmml" xref="S3.I1.i5.p1.1.m1.1.1.2">ℂ</ci><ci id="S3.I1.i5.p1.1.m1.1.1.3.cmml" xref="S3.I1.i5.p1.1.m1.1.1.3">𝔽</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i5.p1.1.m1.1c">\mathbb{CF}</annotation></semantics></math>: Set of counterfactuals that supports the counterfactual answer (<math id="S3.I1.i5.p1.2.m2.1" class="ltx_Math" alttext="\mathrm{CA}" display="inline"><semantics id="S3.I1.i5.p1.2.m2.1a"><mi id="S3.I1.i5.p1.2.m2.1.1" xref="S3.I1.i5.p1.2.m2.1.1.cmml">CA</mi><annotation-xml encoding="MathML-Content" id="S3.I1.i5.p1.2.m2.1b"><ci id="S3.I1.i5.p1.2.m2.1.1.cmml" xref="S3.I1.i5.p1.2.m2.1.1">CA</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i5.p1.2.m2.1c">\mathrm{CA}</annotation></semantics></math>)</p>
</div>
</li>
</ul>
<p id="S3.p2.3" class="ltx_p">These components allow us to assess knowledge propagation by editing a language model with the set of counterfactuals <math id="S3.p2.1.m1.1" class="ltx_Math" alttext="\mathbb{CF}" display="inline"><semantics id="S3.p2.1.m1.1a"><mrow id="S3.p2.1.m1.1.1" xref="S3.p2.1.m1.1.1.cmml"><mi id="S3.p2.1.m1.1.1.2" xref="S3.p2.1.m1.1.1.2.cmml">ℂ</mi><mo lspace="0em" rspace="0em" id="S3.p2.1.m1.1.1.1" xref="S3.p2.1.m1.1.1.1.cmml">​</mo><mi id="S3.p2.1.m1.1.1.3" xref="S3.p2.1.m1.1.1.3.cmml">𝔽</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.p2.1.m1.1b"><apply id="S3.p2.1.m1.1.1.cmml" xref="S3.p2.1.m1.1.1"><times id="S3.p2.1.m1.1.1.1.cmml" xref="S3.p2.1.m1.1.1.1"></times><ci id="S3.p2.1.m1.1.1.2.cmml" xref="S3.p2.1.m1.1.1.2">ℂ</ci><ci id="S3.p2.1.m1.1.1.3.cmml" xref="S3.p2.1.m1.1.1.3">𝔽</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.1.m1.1c">\mathbb{CF}</annotation></semantics></math>, and testing if the edited model is able to flip its original answer (<math id="S3.p2.2.m2.1" class="ltx_Math" alttext="\mathrm{A}" display="inline"><semantics id="S3.p2.2.m2.1a"><mi mathvariant="normal" id="S3.p2.2.m2.1.1" xref="S3.p2.2.m2.1.1.cmml">A</mi><annotation-xml encoding="MathML-Content" id="S3.p2.2.m2.1b"><ci id="S3.p2.2.m2.1.1.cmml" xref="S3.p2.2.m2.1.1">A</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.2.m2.1c">\mathrm{A}</annotation></semantics></math>) towards the counterfactual answer (<math id="S3.p2.3.m3.1" class="ltx_Math" alttext="\mathrm{CA}" display="inline"><semantics id="S3.p2.3.m3.1a"><mi id="S3.p2.3.m3.1.1" xref="S3.p2.3.m3.1.1.cmml">CA</mi><annotation-xml encoding="MathML-Content" id="S3.p2.3.m3.1b"><ci id="S3.p2.3.m3.1.1.cmml" xref="S3.p2.3.m3.1.1">CA</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.3.m3.1c">\mathrm{CA}</annotation></semantics></math>) through reasoning.</p>
</div>
<div id="S3.p3" class="ltx_para">
<p id="S3.p3.1" class="ltx_p">In this section, we provide a comprehensive overview of the dataset and delve into the nuances of its construction methodology.</p>
</div>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>ReCoE: QA Pairs Construction</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.1" class="ltx_p">Table <a href="#S2.T1" title="Table 1 ‣ 2.2 Model Editing Benchmarks ‣ 2 Related Work ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> presents examples of QA pairs for each reasoning scheme and the corresponding data source. Table <a href="#S3.T2" title="Table 2 ‣ Aggregation &amp; Subtraction ‣ 3.1 ReCoE: QA Pairs Construction ‣ 3 ReCoE Dataset ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> presents the dataset statistics including the number of examples and atomic facts of <math id="S3.SS1.p1.1.m1.1" class="ltx_Math" alttext="\mathbb{CF}" display="inline"><semantics id="S3.SS1.p1.1.m1.1a"><mrow id="S3.SS1.p1.1.m1.1.1" xref="S3.SS1.p1.1.m1.1.1.cmml"><mi id="S3.SS1.p1.1.m1.1.1.2" xref="S3.SS1.p1.1.m1.1.1.2.cmml">ℂ</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.1.m1.1.1.1" xref="S3.SS1.p1.1.m1.1.1.1.cmml">​</mo><mi id="S3.SS1.p1.1.m1.1.1.3" xref="S3.SS1.p1.1.m1.1.1.3.cmml">𝔽</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.1.m1.1b"><apply id="S3.SS1.p1.1.m1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1"><times id="S3.SS1.p1.1.m1.1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1.1"></times><ci id="S3.SS1.p1.1.m1.1.1.2.cmml" xref="S3.SS1.p1.1.m1.1.1.2">ℂ</ci><ci id="S3.SS1.p1.1.m1.1.1.3.cmml" xref="S3.SS1.p1.1.m1.1.1.3">𝔽</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.1.m1.1c">\mathbb{CF}</annotation></semantics></math> for each reasoning scheme.</p>
</div>
<div id="S3.SS1.p2" class="ltx_para">
<p id="S3.SS1.p2.1" class="ltx_p">The construction of the benchmark starts from basic QA pairs. QA pairs of all superlative data and part of comparative data are from existent QA datasets <cite class="ltx_cite ltx_citemacro_cite">Gu et al. (<a href="#bib.bib9" title="" class="ltx_ref">2021</a>); Kwiatkowski et al. (<a href="#bib.bib15" title="" class="ltx_ref">2019</a>); Talmor and Berant (<a href="#bib.bib24" title="" class="ltx_ref">2018</a>); Jiang et al. (<a href="#bib.bib14" title="" class="ltx_ref">2019</a>)</cite>. QA pairs of counting data are partially hand-written. Based on these datapoints, we create QA pairs for sorting questions, more comparative questions, aggregation questions and subtraction questions synthetically.</p>
</div>
<section id="S3.SS1.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Sorting</h5>

<div id="S3.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="S3.SS1.SSS0.Px1.p1.3" class="ltx_p">questions are constructed by prompting Claude based on superlative questions with the following 4 steps. Taking the superlative QA pair in Table <a href="#S2.T1" title="Table 1 ‣ 2.2 Model Editing Benchmarks ‣ 2 Related Work ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> as an example:
<br class="ltx_break"><span id="S3.SS1.SSS0.Px1.p1.3.1" class="ltx_text ltx_font_bold">Step 1:</span> Prompt Claude to generate the aspect that the sentence is discussing.
<span id="S3.SS1.SSS0.Px1.p1.3.2" class="ltx_ERROR undefined">\MakeFramed</span><span id="S3.SS1.SSS0.Px1.p1.3.3" class="ltx_ERROR undefined">\FrameRestore</span>
<span id="S3.SS1.SSS0.Px1.p1.3.4" class="ltx_text ltx_font_bold">aspect</span>: city population or city size
<span id="S3.SS1.SSS0.Px1.p1.3.5" class="ltx_ERROR undefined">\endMakeFramed</span>
<span id="S3.SS1.SSS0.Px1.p1.3.6" class="ltx_text ltx_font_bold">Step 2:</span> Given the QA pair, generate 10 similar or relevant entities to the entity/subject of the question.
<span id="S3.SS1.SSS0.Px1.p1.3.7" class="ltx_ERROR undefined">\MakeFramed</span><span id="S3.SS1.SSS0.Px1.p1.3.8" class="ltx_ERROR undefined">\FrameRestore</span>
<span id="S3.SS1.SSS0.Px1.p1.3.9" class="ltx_text ltx_font_bold">relevant entities</span>:
<br class="ltx_break">Large cities in Canada:
Toronto, Montreal, Calgary, Ottawa
<br class="ltx_break">Cities in British Columbia:
Victoria, Nanaimo, Nelson
<br class="ltx_break">Coastal cities in North America:
Seattle, San Francisco, Los Angeles, Portland
<span id="S3.SS1.SSS0.Px1.p1.3.10" class="ltx_ERROR undefined">\endMakeFramed</span>
<span id="S3.SS1.SSS0.Px1.p1.3.11" class="ltx_text ltx_font_bold">Step 3:</span> Given the question and generated aspect (Step 1), prompt Claude to generate relevant facts about each generated entity.
<span id="S3.SS1.SSS0.Px1.p1.3.12" class="ltx_ERROR undefined">\MakeFramed</span><span id="S3.SS1.SSS0.Px1.p1.3.13" class="ltx_ERROR undefined">\FrameRestore</span>
<span id="S3.SS1.SSS0.Px1.p1.3.14" class="ltx_text ltx_font_bold">entity facts</span>:
<br class="ltx_break">Toronto: a recorded population of 2,794,356 …
<br class="ltx_break">Victoria: city size 19.47 km<sup id="S3.SS1.SSS0.Px1.p1.3.15" class="ltx_sup">2</sup> (7.52 sq mi) …
<br class="ltx_break">Nanaimo: city size 35.25 mi<sup id="S3.SS1.SSS0.Px1.p1.3.16" class="ltx_sup">2</sup> …
<br class="ltx_break">Seattle: land area of 83.9 square miles (217.3 km<sup id="S3.SS1.SSS0.Px1.p1.3.17" class="ltx_sup">2</sup>) …
<br class="ltx_break">…
<span id="S3.SS1.SSS0.Px1.p1.3.18" class="ltx_ERROR undefined">\endMakeFramed</span>
<span id="S3.SS1.SSS0.Px1.p1.3.19" class="ltx_text ltx_font_bold">Step 4:</span> Generate 5 sorting questions with the answer using generated entities and their corresponding facts.
<span id="S3.SS1.SSS0.Px1.p1.3.20" class="ltx_ERROR undefined">\MakeFramed</span><span id="S3.SS1.SSS0.Px1.p1.3.21" class="ltx_ERROR undefined">\FrameRestore</span>
<span id="S3.SS1.SSS0.Px1.p1.3.22" class="ltx_text ltx_font_bold">example sorting questions with answers</span>:
<br class="ltx_break"><span id="S3.SS1.SSS0.Px1.p1.3.23" class="ltx_text ltx_font_bold">Question</span>: Sort the following cities based on their city size from small to large: Nanaimo, Victoria, Seattle.
<br class="ltx_break"><span id="S3.SS1.SSS0.Px1.p1.3.24" class="ltx_text ltx_font_bold">Answer</span>: Victoria, Nanaimo, Seattle.
<span id="S3.SS1.SSS0.Px1.p1.3.25" class="ltx_ERROR undefined">\endMakeFramed</span>
Based on the existent superlative sub-dataset, over 10k sorting datapoints can be constructed in the above manner. Since the generated facts and answers based on the facts are not necessarily correct, we leverage <span id="S3.SS1.SSS0.Px1.p1.3.26" class="ltx_text ltx_font_bold">multi-agent debate</span> <cite class="ltx_cite ltx_citemacro_cite">Du et al. (<a href="#bib.bib7" title="" class="ltx_ref">2023</a>)</cite> with Claude agents to double-check the generated QA pairs as initial quality control: QA pairs are excluded if the agents, post-debate, converge on different answers to the question.</p>
</div>
</section>
<section id="S3.SS1.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Comparative</h5>

<div id="S3.SS1.SSS0.Px2.p1" class="ltx_para">
<p id="S3.SS1.SSS0.Px2.p1.1" class="ltx_p">questions are partially (192) selected from existent datasets. We generate 1,000 more QA pairs with Claude based on generated sorting questions by transforming a sorting question to a comparative question.</p>
</div>
</section>
<section id="S3.SS1.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Counting</h5>

<div id="S3.SS1.SSS0.Px3.p1" class="ltx_para">
<p id="S3.SS1.SSS0.Px3.p1.1" class="ltx_p">questions are from 8 different domains: astronomy, book, geography, legal, movie, music, sport, and war. We manually create 5 QA questions for each domain and then prompt Claude to generate more such QA pairs following the examples. Multi-agent debate is again adopted to filter out inaccurate QA pairs.</p>
</div>
</section>
<section id="S3.SS1.SSS0.Px4" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Aggregation &amp; Subtraction</h5>

<div id="S3.SS1.SSS0.Px4.p1" class="ltx_para">
<p id="S3.SS1.SSS0.Px4.p1.1" class="ltx_p">questions are derived from the counting questions and retains the same 8 domains. An aggregation question is formulated by combining two or more counting questions. Below is a QA pair example:
<span id="S3.SS1.SSS0.Px4.p1.1.1" class="ltx_ERROR undefined">\MakeFramed</span><span id="S3.SS1.SSS0.Px4.p1.1.2" class="ltx_ERROR undefined">\FrameRestore</span>
<span id="S3.SS1.SSS0.Px4.p1.1.3" class="ltx_text ltx_font_bold">Question</span>: How many states/provinces are there in North America, i.e. United States, Canada, and Mexico?
<br class="ltx_break"><span id="S3.SS1.SSS0.Px4.p1.1.4" class="ltx_text ltx_font_bold">Answer</span>: 92
<span id="S3.SS1.SSS0.Px4.p1.1.5" class="ltx_ERROR undefined">\endMakeFramed</span>
To avoid incongruous or unnatural questions, we employ two strategies:</p>
<ol id="S3.I2" class="ltx_enumerate">
<li id="S3.I2.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S3.I2.i1.p1" class="ltx_para">
<p id="S3.I2.i1.p1.1" class="ltx_p">Counting questions to be combined are sampled from the same domain.</p>
</div>
</li>
<li id="S3.I2.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S3.I2.i2.p1" class="ltx_para">
<p id="S3.I2.i2.p1.1" class="ltx_p">Filter out unnatural questions with Claude by two criteria: a) whether the question is fluent; b) whether the entities mentioned in the question are compatible in type. For instance, the number of satellites of Earth and that of Mars are compatible for aggregation, while the number of constellations recognized by the International Astronomical Union and the number of Earth’s satellites are not.</p>
</div>
</li>
</ol>
</div>
<figure id="S3.T2" class="ltx_table">
<table id="S3.T2.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S3.T2.1.1.1" class="ltx_tr">
<th id="S3.T2.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt">Scheme</th>
<td id="S3.T2.1.1.1.2" class="ltx_td ltx_align_right ltx_border_tt">Examples</td>
<td id="S3.T2.1.1.1.3" class="ltx_td ltx_align_right ltx_border_tt">Atomic facts</td>
</tr>
<tr id="S3.T2.1.2.2" class="ltx_tr">
<th id="S3.T2.1.2.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">superlative</th>
<td id="S3.T2.1.2.2.2" class="ltx_td ltx_align_right ltx_border_t">1,172</td>
<td id="S3.T2.1.2.2.3" class="ltx_td ltx_align_right ltx_border_t">7,248</td>
</tr>
<tr id="S3.T2.1.3.3" class="ltx_tr">
<th id="S3.T2.1.3.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">comparative</th>
<td id="S3.T2.1.3.3.2" class="ltx_td ltx_align_right">1,171</td>
<td id="S3.T2.1.3.3.3" class="ltx_td ltx_align_right">7,218</td>
</tr>
<tr id="S3.T2.1.4.4" class="ltx_tr">
<th id="S3.T2.1.4.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">counting</th>
<td id="S3.T2.1.4.4.2" class="ltx_td ltx_align_right">643</td>
<td id="S3.T2.1.4.4.3" class="ltx_td ltx_align_right">1,309</td>
</tr>
<tr id="S3.T2.1.5.5" class="ltx_tr">
<th id="S3.T2.1.5.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">sorting</th>
<td id="S3.T2.1.5.5.2" class="ltx_td ltx_align_right">1,041</td>
<td id="S3.T2.1.5.5.3" class="ltx_td ltx_align_right">6,128</td>
</tr>
<tr id="S3.T2.1.6.6" class="ltx_tr">
<th id="S3.T2.1.6.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">aggregation</th>
<td id="S3.T2.1.6.6.2" class="ltx_td ltx_align_right">509</td>
<td id="S3.T2.1.6.6.3" class="ltx_td ltx_align_right">506</td>
</tr>
<tr id="S3.T2.1.7.7" class="ltx_tr">
<th id="S3.T2.1.7.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">subtraction</th>
<td id="S3.T2.1.7.7.2" class="ltx_td ltx_align_right">501</td>
<td id="S3.T2.1.7.7.3" class="ltx_td ltx_align_right">533</td>
</tr>
<tr id="S3.T2.1.8.8" class="ltx_tr">
<th id="S3.T2.1.8.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_t">Total</th>
<td id="S3.T2.1.8.8.2" class="ltx_td ltx_align_right ltx_border_bb ltx_border_t">5,037</td>
<td id="S3.T2.1.8.8.3" class="ltx_td ltx_align_right ltx_border_bb ltx_border_t">22,942</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>Statistics of the ReCoE dataset.</figcaption>
</figure>
</section>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>ReCoE: Counterfactual Construction</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.1" class="ltx_p">After obtaining all the QA pairs for each reasoning scheme, we need to create facts and counterfactual facts to complete each datapoint. The dataset is constructed entirely automatically with the main construction steps illustrated in Figure <a href="#S1.F2" title="Figure 2 ‣ 1 Introduction ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.
The construction involves 4 steps:
<br class="ltx_break"><span id="S3.SS2.p1.1.1" class="ltx_text ltx_font_bold">Step 1</span>: After collecting and generating QA pairs, we prompt Claude to create a counterfactual answer. For instance, if the question is about <span id="S3.SS2.p1.1.2" class="ltx_text ltx_font_italic">which team between the Chicago Blackhawks and Pittsburgh Penguins has won more Stanley Cup championships</span>, and the answer is the <span id="S3.SS2.p1.1.3" class="ltx_text ltx_font_italic">Chicago Blackhawks</span>, the counterfactual answer would be the <span id="S3.SS2.p1.1.4" class="ltx_text ltx_font_italic">Pittsburgh Penguins</span>.
<br class="ltx_break"><span id="S3.SS2.p1.1.5" class="ltx_text ltx_font_bold">Step 2</span>: Claude is prompted to generate relevant facts about entities mentioned in the answer and counterfactual answer. These facts are then verified for accuracy using a retrieval-augmented method by retrieving relevant paragraphs from Wikipedia using Contriever <cite class="ltx_cite ltx_citemacro_cite">Izacard et al. (<a href="#bib.bib12" title="" class="ltx_ref">2021a</a>)</cite> and corrected if necessary. We filter datapoints to ensure that all QA pairs are valid and (question, counterfactual answer) pairs are invalid. 
<br class="ltx_break"><span id="S3.SS2.p1.1.6" class="ltx_text ltx_font_bold">Step 3</span>: To generate counterfactual facts, if a question is superlative, comparative, or sorting (Group 1), we <span id="S3.SS2.p1.1.7" class="ltx_text ltx_font_bold">swap</span> the subjects of supporting facts between those related to the actual answer and those pertaining to the counterfactual answer. This process is conducted while eliminating any datapoints that could introduce contradictions in the counterfactual facts generated as a result of this subject swapping; if the question is counting, aggregation, or sorting (Group 2), we <span id="S3.SS2.p1.1.8" class="ltx_text ltx_font_bold">alter</span> the facts for the answer to obtain the counterfactual facts for the counterfactual answer while maintaining consistency.
<br class="ltx_break"><span id="S3.SS2.p1.1.9" class="ltx_text ltx_font_bold">Step 4</span>: All sentences in the facts and counterfactual facts are broken down into atomic formats for easier editing. 
<br class="ltx_break"></p>
</div>
<div id="S3.SS2.p2" class="ltx_para">
<p id="S3.SS2.p2.1" class="ltx_p">A concrete example in ReCoE is presented in Appendix <a href="#A1.SS1" title="A.1 Example Datapoint ‣ Appendix A Appendix ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.1</span></a>.</p>
</div>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Fact Representation</h3>

<figure id="S3.F3" class="ltx_figure"><img src="/html/2401.17585/assets/x3.png" id="S3.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="232" height="133" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>Comparison between fact representations in MQuAKE <cite class="ltx_cite ltx_citemacro_cite">Zhong et al. (<a href="#bib.bib30" title="" class="ltx_ref">2023</a>)</cite> and ReCoE.</figcaption>
</figure>
<div id="S3.SS3.p1" class="ltx_para">
<p id="S3.SS3.p1.1" class="ltx_p">Current benchmarks primarily focus on datasets such as zsRE <cite class="ltx_cite ltx_citemacro_cite">Onoe et al. (<a href="#bib.bib21" title="" class="ltx_ref">2023</a>); De Cao et al. (<a href="#bib.bib4" title="" class="ltx_ref">2021</a>); Meng et al. (<a href="#bib.bib16" title="" class="ltx_ref">2022a</a>)</cite>, COUNTERFACT <cite class="ltx_cite ltx_citemacro_cite">Meng et al. (<a href="#bib.bib16" title="" class="ltx_ref">2022a</a>)</cite>, and MQuAKE <cite class="ltx_cite ltx_citemacro_cite">Zhong et al. (<a href="#bib.bib30" title="" class="ltx_ref">2023</a>)</cite>.
In these datasets, each fact is represented in a clear and unambiguous form as a (subject, relation, object) triplet.
In contrast, our dataset diverges from this norm, featuring facts more commonly encountered in real-world scenarios, typically represented in an OpenIE style.
This style introduces a wider variety of complexities.
As illustrated in Figure <a href="#S3.F3" title="Figure 3 ‣ 3.3 Fact Representation ‣ 3 ReCoE Dataset ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, the atomic facts in ReCoE may involve complex subjects or relations. Moreover, a single relation applied to a subject could correspond to non-unique objects.
These examples highlight the nuanced nature of knowledge updates. We will further discuss the challenges these OpenIE-form fact representations poses to the effectiveness of locate-and-edit methods in a later section (Section <a href="#S5.SS4.SSS1" title="5.4.1 QLoRA vs. MEMIT ‣ 5.4 Discussion ‣ 5 Analysis ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5.4.1</span></a>).</p>
</div>
<figure id="S3.T3" class="ltx_table">
<div id="S3.T3.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:455.2pt;height:138.7pt;vertical-align:-1.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-10.4pt,3.1pt) scale(0.956393408317538,0.956393408317538) ;">
<table id="S3.T3.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S3.T3.1.1.1.1" class="ltx_tr">
<th id="S3.T3.1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt" rowspan="2"><span id="S3.T3.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Model</span></th>
<th id="S3.T3.1.1.1.1.2" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_tt" rowspan="2"><span id="S3.T3.1.1.1.1.2.1" class="ltx_text ltx_font_bold">Prompt</span></th>
<th id="S3.T3.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="6"><span id="S3.T3.1.1.1.1.3.1" class="ltx_text ltx_font_bold">Reasoning Scheme</span></th>
<td id="S3.T3.1.1.1.1.4" class="ltx_td ltx_border_tt"></td>
</tr>
<tr id="S3.T3.1.1.2.2" class="ltx_tr">
<th id="S3.T3.1.1.2.2.1" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t">superlative</th>
<th id="S3.T3.1.1.2.2.2" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t">comparative</th>
<th id="S3.T3.1.1.2.2.3" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t">counting</th>
<th id="S3.T3.1.1.2.2.4" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t">sorting</th>
<th id="S3.T3.1.1.2.2.5" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t">aggregation</th>
<th id="S3.T3.1.1.2.2.6" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t">subtraction</th>
<th id="S3.T3.1.1.2.2.7" class="ltx_td ltx_align_right ltx_th ltx_th_column"><span id="S3.T3.1.1.2.2.7.1" class="ltx_text ltx_font_italic">Average</span></th>
</tr>
<tr id="S3.T3.1.1.3.3" class="ltx_tr">
<th id="S3.T3.1.1.3.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="2"><span id="S3.T3.1.1.3.3.1.1" class="ltx_text">7b</span></th>
<td id="S3.T3.1.1.3.3.2" class="ltx_td ltx_align_right ltx_border_t">direct</td>
<td id="S3.T3.1.1.3.3.3" class="ltx_td ltx_align_right ltx_border_t">12.55</td>
<td id="S3.T3.1.1.3.3.4" class="ltx_td ltx_align_right ltx_border_t">10.93</td>
<td id="S3.T3.1.1.3.3.5" class="ltx_td ltx_align_right ltx_border_t">31.73</td>
<td id="S3.T3.1.1.3.3.6" class="ltx_td ltx_align_right ltx_border_t">13.45</td>
<td id="S3.T3.1.1.3.3.7" class="ltx_td ltx_align_right ltx_border_t">10.22</td>
<td id="S3.T3.1.1.3.3.8" class="ltx_td ltx_align_right ltx_border_t">10.38</td>
<td id="S3.T3.1.1.3.3.9" class="ltx_td ltx_align_right ltx_border_t"><span id="S3.T3.1.1.3.3.9.1" class="ltx_text ltx_font_italic">14.67</span></td>
</tr>
<tr id="S3.T3.1.1.4.4" class="ltx_tr">
<td id="S3.T3.1.1.4.4.1" class="ltx_td ltx_align_right">CoT</td>
<td id="S3.T3.1.1.4.4.2" class="ltx_td ltx_align_right">28.63</td>
<td id="S3.T3.1.1.4.4.3" class="ltx_td ltx_align_right">57.73</td>
<td id="S3.T3.1.1.4.4.4" class="ltx_td ltx_align_right">40.90</td>
<td id="S3.T3.1.1.4.4.5" class="ltx_td ltx_align_right">10.09</td>
<td id="S3.T3.1.1.4.4.6" class="ltx_td ltx_align_right">27.31</td>
<td id="S3.T3.1.1.4.4.7" class="ltx_td ltx_align_right">27.94</td>
<td id="S3.T3.1.1.4.4.8" class="ltx_td ltx_align_right"><span id="S3.T3.1.1.4.4.8.1" class="ltx_text ltx_font_italic">32.28</span></td>
</tr>
<tr id="S3.T3.1.1.5.5" class="ltx_tr">
<th id="S3.T3.1.1.5.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="2"><span id="S3.T3.1.1.5.5.1.1" class="ltx_text">13b</span></th>
<td id="S3.T3.1.1.5.5.2" class="ltx_td ltx_align_right ltx_border_t">direct</td>
<td id="S3.T3.1.1.5.5.3" class="ltx_td ltx_align_right ltx_border_t">20.21</td>
<td id="S3.T3.1.1.5.5.4" class="ltx_td ltx_align_right ltx_border_t">33.13</td>
<td id="S3.T3.1.1.5.5.5" class="ltx_td ltx_align_right ltx_border_t">33.44</td>
<td id="S3.T3.1.1.5.5.6" class="ltx_td ltx_align_right ltx_border_t">19.60</td>
<td id="S3.T3.1.1.5.5.7" class="ltx_td ltx_align_right ltx_border_t">5.30</td>
<td id="S3.T3.1.1.5.5.8" class="ltx_td ltx_align_right ltx_border_t">11.78</td>
<td id="S3.T3.1.1.5.5.9" class="ltx_td ltx_align_right ltx_border_t"><span id="S3.T3.1.1.5.5.9.1" class="ltx_text ltx_font_italic">20.43</span></td>
</tr>
<tr id="S3.T3.1.1.6.6" class="ltx_tr">
<td id="S3.T3.1.1.6.6.1" class="ltx_td ltx_align_right">CoT</td>
<td id="S3.T3.1.1.6.6.2" class="ltx_td ltx_align_right">30.88</td>
<td id="S3.T3.1.1.6.6.3" class="ltx_td ltx_align_right">62.60</td>
<td id="S3.T3.1.1.6.6.4" class="ltx_td ltx_align_right">43.23</td>
<td id="S3.T3.1.1.6.6.5" class="ltx_td ltx_align_right">16.04</td>
<td id="S3.T3.1.1.6.6.6" class="ltx_td ltx_align_right">26.52</td>
<td id="S3.T3.1.1.6.6.7" class="ltx_td ltx_align_right">31.54</td>
<td id="S3.T3.1.1.6.6.8" class="ltx_td ltx_align_right"><span id="S3.T3.1.1.6.6.8.1" class="ltx_text ltx_font_italic">35.39</span></td>
</tr>
<tr id="S3.T3.1.1.7.7" class="ltx_tr">
<th id="S3.T3.1.1.7.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_t" rowspan="2"><span id="S3.T3.1.1.7.7.1.1" class="ltx_text">33b</span></th>
<td id="S3.T3.1.1.7.7.2" class="ltx_td ltx_align_right ltx_border_t">direct</td>
<td id="S3.T3.1.1.7.7.3" class="ltx_td ltx_align_right ltx_border_t">31.10</td>
<td id="S3.T3.1.1.7.7.4" class="ltx_td ltx_align_right ltx_border_t">55.42</td>
<td id="S3.T3.1.1.7.7.5" class="ltx_td ltx_align_right ltx_border_t">46.50</td>
<td id="S3.T3.1.1.7.7.6" class="ltx_td ltx_align_right ltx_border_t">35.06</td>
<td id="S3.T3.1.1.7.7.7" class="ltx_td ltx_align_right ltx_border_t">6.48</td>
<td id="S3.T3.1.1.7.7.8" class="ltx_td ltx_align_right ltx_border_t">11.98</td>
<td id="S3.T3.1.1.7.7.9" class="ltx_td ltx_align_right ltx_border_t"><span id="S3.T3.1.1.7.7.9.1" class="ltx_text ltx_font_italic">30.96</span></td>
</tr>
<tr id="S3.T3.1.1.8.8" class="ltx_tr">
<td id="S3.T3.1.1.8.8.1" class="ltx_td ltx_align_right ltx_border_bb">CoT</td>
<td id="S3.T3.1.1.8.8.2" class="ltx_td ltx_align_right ltx_border_bb">37.19</td>
<td id="S3.T3.1.1.8.8.3" class="ltx_td ltx_align_right ltx_border_bb">71.73</td>
<td id="S3.T3.1.1.8.8.4" class="ltx_td ltx_align_right ltx_border_bb">55.37</td>
<td id="S3.T3.1.1.8.8.5" class="ltx_td ltx_align_right ltx_border_bb">36.12</td>
<td id="S3.T3.1.1.8.8.6" class="ltx_td ltx_align_right ltx_border_bb">41.26</td>
<td id="S3.T3.1.1.8.8.7" class="ltx_td ltx_align_right ltx_border_bb">41.52</td>
<td id="S3.T3.1.1.8.8.8" class="ltx_td ltx_align_right ltx_border_bb"><span id="S3.T3.1.1.8.8.8.1" class="ltx_text ltx_font_italic">46.64</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 3: </span>QA accuracy of knowledge probing. Both CoT prompting and model scaling significantly improved the overall performance.</figcaption>
</figure>
<figure id="S3.T4" class="ltx_table">
<div id="S3.T4.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:455.2pt;height:279.1pt;vertical-align:-0.9pt;"><span class="ltx_transformed_inner" style="transform:translate(-37.5pt,22.9pt) scale(0.858589529676601,0.858589529676601) ;">
<table id="S3.T4.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S3.T4.1.1.1.1" class="ltx_tr">
<th id="S3.T4.1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt" rowspan="2"><span id="S3.T4.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Model</span></th>
<th id="S3.T4.1.1.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt" rowspan="2"><span id="S3.T4.1.1.1.1.2.1" class="ltx_text ltx_font_bold">Editor</span></th>
<th id="S3.T4.1.1.1.1.3" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_tt" rowspan="2"><span id="S3.T4.1.1.1.1.3.1" class="ltx_text ltx_font_bold">Prompt</span></th>
<th id="S3.T4.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="6"><span id="S3.T4.1.1.1.1.4.1" class="ltx_text ltx_font_bold">Reasoning Scheme</span></th>
<td id="S3.T4.1.1.1.1.5" class="ltx_td ltx_border_tt"></td>
</tr>
<tr id="S3.T4.1.1.2.2" class="ltx_tr">
<th id="S3.T4.1.1.2.2.1" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t">superlative</th>
<th id="S3.T4.1.1.2.2.2" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t">comparative</th>
<th id="S3.T4.1.1.2.2.3" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t">counting</th>
<th id="S3.T4.1.1.2.2.4" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t">sorting</th>
<th id="S3.T4.1.1.2.2.5" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t">aggregation</th>
<th id="S3.T4.1.1.2.2.6" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t">subtraction</th>
<th id="S3.T4.1.1.2.2.7" class="ltx_td ltx_align_right ltx_th ltx_th_column"><span id="S3.T4.1.1.2.2.7.1" class="ltx_text ltx_font_italic">Average</span></th>
</tr>
<tr id="S3.T4.1.1.3.3" class="ltx_tr">
<th id="S3.T4.1.1.3.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="6"><span id="S3.T4.1.1.3.3.1.1" class="ltx_text">7b</span></th>
<th id="S3.T4.1.1.3.3.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="2"><span id="S3.T4.1.1.3.3.2.1" class="ltx_text">InputAug</span></th>
<td id="S3.T4.1.1.3.3.3" class="ltx_td ltx_align_right ltx_border_t">direct</td>
<td id="S3.T4.1.1.3.3.4" class="ltx_td ltx_align_right ltx_border_t">50.30</td>
<td id="S3.T4.1.1.3.3.5" class="ltx_td ltx_align_right ltx_border_t">41.41</td>
<td id="S3.T4.1.1.3.3.6" class="ltx_td ltx_align_right ltx_border_t">23.53</td>
<td id="S3.T4.1.1.3.3.7" class="ltx_td ltx_align_right ltx_border_t">7.86</td>
<td id="S3.T4.1.1.3.3.8" class="ltx_td ltx_align_right ltx_border_t">7.69</td>
<td id="S3.T4.1.1.3.3.9" class="ltx_td ltx_align_right ltx_border_t">17.31</td>
<td id="S3.T4.1.1.3.3.10" class="ltx_td ltx_align_right ltx_border_t"><span id="S3.T4.1.1.3.3.10.1" class="ltx_text ltx_font_italic">24.68</span></td>
</tr>
<tr id="S3.T4.1.1.4.4" class="ltx_tr">
<td id="S3.T4.1.1.4.4.1" class="ltx_td ltx_align_right">CoT</td>
<td id="S3.T4.1.1.4.4.2" class="ltx_td ltx_align_right">54.07</td>
<td id="S3.T4.1.1.4.4.3" class="ltx_td ltx_align_right">53.99</td>
<td id="S3.T4.1.1.4.4.4" class="ltx_td ltx_align_right">34.60</td>
<td id="S3.T4.1.1.4.4.5" class="ltx_td ltx_align_right">14.29</td>
<td id="S3.T4.1.1.4.4.6" class="ltx_td ltx_align_right">20.86</td>
<td id="S3.T4.1.1.4.4.7" class="ltx_td ltx_align_right">10.00</td>
<td id="S3.T4.1.1.4.4.8" class="ltx_td ltx_align_right"><span id="S3.T4.1.1.4.4.8.1" class="ltx_text ltx_font_italic">31.30</span></td>
</tr>
<tr id="S3.T4.1.1.5.5" class="ltx_tr">
<th id="S3.T4.1.1.5.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="2"><span id="S3.T4.1.1.5.5.1.1" class="ltx_text">QLoRA</span></th>
<td id="S3.T4.1.1.5.5.2" class="ltx_td ltx_align_right ltx_border_t">direct</td>
<td id="S3.T4.1.1.5.5.3" class="ltx_td ltx_align_right ltx_border_t">0.60</td>
<td id="S3.T4.1.1.5.5.4" class="ltx_td ltx_align_right ltx_border_t">14.06</td>
<td id="S3.T4.1.1.5.5.5" class="ltx_td ltx_align_right ltx_border_t">9.80</td>
<td id="S3.T4.1.1.5.5.6" class="ltx_td ltx_align_right ltx_border_t">11.43</td>
<td id="S3.T4.1.1.5.5.7" class="ltx_td ltx_align_right ltx_border_t">5.77</td>
<td id="S3.T4.1.1.5.5.8" class="ltx_td ltx_align_right ltx_border_t">7.69</td>
<td id="S3.T4.1.1.5.5.9" class="ltx_td ltx_align_right ltx_border_t"><span id="S3.T4.1.1.5.5.9.1" class="ltx_text ltx_font_italic">8.23</span></td>
</tr>
<tr id="S3.T4.1.1.6.6" class="ltx_tr">
<td id="S3.T4.1.1.6.6.1" class="ltx_td ltx_align_right">CoT</td>
<td id="S3.T4.1.1.6.6.2" class="ltx_td ltx_align_right">3.41</td>
<td id="S3.T4.1.1.6.6.3" class="ltx_td ltx_align_right">51.04</td>
<td id="S3.T4.1.1.6.6.4" class="ltx_td ltx_align_right">8.37</td>
<td id="S3.T4.1.1.6.6.5" class="ltx_td ltx_align_right">7.62</td>
<td id="S3.T4.1.1.6.6.6" class="ltx_td ltx_align_right">5.04</td>
<td id="S3.T4.1.1.6.6.7" class="ltx_td ltx_align_right">8.57</td>
<td id="S3.T4.1.1.6.6.8" class="ltx_td ltx_align_right"><span id="S3.T4.1.1.6.6.8.1" class="ltx_text ltx_font_italic">14.01</span></td>
</tr>
<tr id="S3.T4.1.1.7.7" class="ltx_tr">
<th id="S3.T4.1.1.7.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="2"><span id="S3.T4.1.1.7.7.1.1" class="ltx_text">MEMIT</span></th>
<td id="S3.T4.1.1.7.7.2" class="ltx_td ltx_align_right ltx_border_t">direct</td>
<td id="S3.T4.1.1.7.7.3" class="ltx_td ltx_align_right ltx_border_t">0.00</td>
<td id="S3.T4.1.1.7.7.4" class="ltx_td ltx_align_right ltx_border_t">34.38</td>
<td id="S3.T4.1.1.7.7.5" class="ltx_td ltx_align_right ltx_border_t">6.37</td>
<td id="S3.T4.1.1.7.7.6" class="ltx_td ltx_align_right ltx_border_t">3.94</td>
<td id="S3.T4.1.1.7.7.7" class="ltx_td ltx_align_right ltx_border_t">0.00</td>
<td id="S3.T4.1.1.7.7.8" class="ltx_td ltx_align_right ltx_border_t">7.69</td>
<td id="S3.T4.1.1.7.7.9" class="ltx_td ltx_align_right ltx_border_t"><span id="S3.T4.1.1.7.7.9.1" class="ltx_text ltx_font_italic">8.73</span></td>
</tr>
<tr id="S3.T4.1.1.8.8" class="ltx_tr">
<td id="S3.T4.1.1.8.8.1" class="ltx_td ltx_align_right">CoT</td>
<td id="S3.T4.1.1.8.8.2" class="ltx_td ltx_align_right">0.00</td>
<td id="S3.T4.1.1.8.8.3" class="ltx_td ltx_align_right">3.55</td>
<td id="S3.T4.1.1.8.8.4" class="ltx_td ltx_align_right">5.32</td>
<td id="S3.T4.1.1.8.8.5" class="ltx_td ltx_align_right">3.45</td>
<td id="S3.T4.1.1.8.8.6" class="ltx_td ltx_align_right">0.00</td>
<td id="S3.T4.1.1.8.8.7" class="ltx_td ltx_align_right">4.29</td>
<td id="S3.T4.1.1.8.8.8" class="ltx_td ltx_align_right"><span id="S3.T4.1.1.8.8.8.1" class="ltx_text ltx_font_italic">2.77</span></td>
</tr>
<tr id="S3.T4.1.1.9.9" class="ltx_tr">
<th id="S3.T4.1.1.9.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="6"><span id="S3.T4.1.1.9.9.1.1" class="ltx_text">13b</span></th>
<th id="S3.T4.1.1.9.9.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="2"><span id="S3.T4.1.1.9.9.2.1" class="ltx_text">InputAug</span></th>
<td id="S3.T4.1.1.9.9.3" class="ltx_td ltx_align_right ltx_border_t">direct</td>
<td id="S3.T4.1.1.9.9.4" class="ltx_td ltx_align_right ltx_border_t">59.85</td>
<td id="S3.T4.1.1.9.9.5" class="ltx_td ltx_align_right ltx_border_t">51.55</td>
<td id="S3.T4.1.1.9.9.6" class="ltx_td ltx_align_right ltx_border_t">24.19</td>
<td id="S3.T4.1.1.9.9.7" class="ltx_td ltx_align_right ltx_border_t">15.69</td>
<td id="S3.T4.1.1.9.9.8" class="ltx_td ltx_align_right ltx_border_t">3.70</td>
<td id="S3.T4.1.1.9.9.9" class="ltx_td ltx_align_right ltx_border_t">11.86</td>
<td id="S3.T4.1.1.9.9.10" class="ltx_td ltx_align_right ltx_border_t"><span id="S3.T4.1.1.9.9.10.1" class="ltx_text ltx_font_italic">27.81</span></td>
</tr>
<tr id="S3.T4.1.1.10.10" class="ltx_tr">
<td id="S3.T4.1.1.10.10.1" class="ltx_td ltx_align_right">CoT</td>
<td id="S3.T4.1.1.10.10.2" class="ltx_td ltx_align_right">71.78</td>
<td id="S3.T4.1.1.10.10.3" class="ltx_td ltx_align_right">66.30</td>
<td id="S3.T4.1.1.10.10.4" class="ltx_td ltx_align_right">60.43</td>
<td id="S3.T4.1.1.10.10.5" class="ltx_td ltx_align_right">18.56</td>
<td id="S3.T4.1.1.10.10.6" class="ltx_td ltx_align_right">25.93</td>
<td id="S3.T4.1.1.10.10.7" class="ltx_td ltx_align_right">25.32</td>
<td id="S3.T4.1.1.10.10.8" class="ltx_td ltx_align_right"><span id="S3.T4.1.1.10.10.8.1" class="ltx_text ltx_font_italic">44.72</span></td>
</tr>
<tr id="S3.T4.1.1.11.11" class="ltx_tr">
<th id="S3.T4.1.1.11.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="2"><span id="S3.T4.1.1.11.11.1.1" class="ltx_text">QLoRA</span></th>
<td id="S3.T4.1.1.11.11.2" class="ltx_td ltx_align_right ltx_border_t">direct</td>
<td id="S3.T4.1.1.11.11.3" class="ltx_td ltx_align_right ltx_border_t">2.23</td>
<td id="S3.T4.1.1.11.11.4" class="ltx_td ltx_align_right ltx_border_t">19.07</td>
<td id="S3.T4.1.1.11.11.5" class="ltx_td ltx_align_right ltx_border_t">5.12</td>
<td id="S3.T4.1.1.11.11.6" class="ltx_td ltx_align_right ltx_border_t">12.75</td>
<td id="S3.T4.1.1.11.11.7" class="ltx_td ltx_align_right ltx_border_t">3.70</td>
<td id="S3.T4.1.1.11.11.8" class="ltx_td ltx_align_right ltx_border_t">8.47</td>
<td id="S3.T4.1.1.11.11.9" class="ltx_td ltx_align_right ltx_border_t"><span id="S3.T4.1.1.11.11.9.1" class="ltx_text ltx_font_italic">8.56</span></td>
</tr>
<tr id="S3.T4.1.1.12.12" class="ltx_tr">
<td id="S3.T4.1.1.12.12.1" class="ltx_td ltx_align_right">CoT</td>
<td id="S3.T4.1.1.12.12.2" class="ltx_td ltx_align_right">4.62</td>
<td id="S3.T4.1.1.12.12.3" class="ltx_td ltx_align_right">30.83</td>
<td id="S3.T4.1.1.12.12.4" class="ltx_td ltx_align_right">8.27</td>
<td id="S3.T4.1.1.12.12.5" class="ltx_td ltx_align_right">14.37</td>
<td id="S3.T4.1.1.12.12.6" class="ltx_td ltx_align_right">3.70</td>
<td id="S3.T4.1.1.12.12.7" class="ltx_td ltx_align_right">6.96</td>
<td id="S3.T4.1.1.12.12.8" class="ltx_td ltx_align_right"><span id="S3.T4.1.1.12.12.8.1" class="ltx_text ltx_font_italic">11.46</span></td>
</tr>
<tr id="S3.T4.1.1.13.13" class="ltx_tr">
<th id="S3.T4.1.1.13.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="2"><span id="S3.T4.1.1.13.13.1.1" class="ltx_text">MEMIT</span></th>
<td id="S3.T4.1.1.13.13.2" class="ltx_td ltx_align_right ltx_border_t">direct</td>
<td id="S3.T4.1.1.13.13.3" class="ltx_td ltx_align_right ltx_border_t">0.37</td>
<td id="S3.T4.1.1.13.13.4" class="ltx_td ltx_align_right ltx_border_t">41.49</td>
<td id="S3.T4.1.1.13.13.5" class="ltx_td ltx_align_right ltx_border_t">11.63</td>
<td id="S3.T4.1.1.13.13.6" class="ltx_td ltx_align_right ltx_border_t">2.05</td>
<td id="S3.T4.1.1.13.13.7" class="ltx_td ltx_align_right ltx_border_t">0.00</td>
<td id="S3.T4.1.1.13.13.8" class="ltx_td ltx_align_right ltx_border_t">3.39</td>
<td id="S3.T4.1.1.13.13.9" class="ltx_td ltx_align_right ltx_border_t"><span id="S3.T4.1.1.13.13.9.1" class="ltx_text ltx_font_italic">9.82</span></td>
</tr>
<tr id="S3.T4.1.1.14.14" class="ltx_tr">
<td id="S3.T4.1.1.14.14.1" class="ltx_td ltx_align_right">CoT</td>
<td id="S3.T4.1.1.14.14.2" class="ltx_td ltx_align_right">0.24</td>
<td id="S3.T4.1.1.14.14.3" class="ltx_td ltx_align_right">18.69</td>
<td id="S3.T4.1.1.14.14.4" class="ltx_td ltx_align_right">7.91</td>
<td id="S3.T4.1.1.14.14.5" class="ltx_td ltx_align_right">0.55</td>
<td id="S3.T4.1.1.14.14.6" class="ltx_td ltx_align_right">0.74</td>
<td id="S3.T4.1.1.14.14.7" class="ltx_td ltx_align_right">5.06</td>
<td id="S3.T4.1.1.14.14.8" class="ltx_td ltx_align_right"><span id="S3.T4.1.1.14.14.8.1" class="ltx_text ltx_font_italic">5.53</span></td>
</tr>
<tr id="S3.T4.1.1.15.15" class="ltx_tr">
<th id="S3.T4.1.1.15.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_t" rowspan="4"><span id="S3.T4.1.1.15.15.1.1" class="ltx_text">33b</span></th>
<th id="S3.T4.1.1.15.15.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="2"><span id="S3.T4.1.1.15.15.2.1" class="ltx_text">InputAug</span></th>
<td id="S3.T4.1.1.15.15.3" class="ltx_td ltx_align_right ltx_border_t">direct</td>
<td id="S3.T4.1.1.15.15.4" class="ltx_td ltx_align_right ltx_border_t">82.37</td>
<td id="S3.T4.1.1.15.15.5" class="ltx_td ltx_align_right ltx_border_t">74.88</td>
<td id="S3.T4.1.1.15.15.6" class="ltx_td ltx_align_right ltx_border_t">23.08</td>
<td id="S3.T4.1.1.15.15.7" class="ltx_td ltx_align_right ltx_border_t">24.11</td>
<td id="S3.T4.1.1.15.15.8" class="ltx_td ltx_align_right ltx_border_t">18.18</td>
<td id="S3.T4.1.1.15.15.9" class="ltx_td ltx_align_right ltx_border_t">11.67</td>
<td id="S3.T4.1.1.15.15.10" class="ltx_td ltx_align_right ltx_border_t"><span id="S3.T4.1.1.15.15.10.1" class="ltx_text ltx_font_italic">39.05</span></td>
</tr>
<tr id="S3.T4.1.1.16.16" class="ltx_tr">
<td id="S3.T4.1.1.16.16.1" class="ltx_td ltx_align_right">CoT</td>
<td id="S3.T4.1.1.16.16.2" class="ltx_td ltx_align_right">73.33</td>
<td id="S3.T4.1.1.16.16.3" class="ltx_td ltx_align_right">84.88</td>
<td id="S3.T4.1.1.16.16.4" class="ltx_td ltx_align_right">55.90</td>
<td id="S3.T4.1.1.16.16.5" class="ltx_td ltx_align_right">32.98</td>
<td id="S3.T4.1.1.16.16.6" class="ltx_td ltx_align_right">46.67</td>
<td id="S3.T4.1.1.16.16.7" class="ltx_td ltx_align_right">35.10</td>
<td id="S3.T4.1.1.16.16.8" class="ltx_td ltx_align_right"><span id="S3.T4.1.1.16.16.8.1" class="ltx_text ltx_font_italic">54.81</span></td>
</tr>
<tr id="S3.T4.1.1.17.17" class="ltx_tr">
<th id="S3.T4.1.1.17.17.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_t" rowspan="2"><span id="S3.T4.1.1.17.17.1.1" class="ltx_text">QLoRA</span></th>
<td id="S3.T4.1.1.17.17.2" class="ltx_td ltx_align_right ltx_border_t">direct</td>
<td id="S3.T4.1.1.17.17.3" class="ltx_td ltx_align_right ltx_border_t">3.14</td>
<td id="S3.T4.1.1.17.17.4" class="ltx_td ltx_align_right ltx_border_t">12.94</td>
<td id="S3.T4.1.1.17.17.5" class="ltx_td ltx_align_right ltx_border_t">6.02</td>
<td id="S3.T4.1.1.17.17.6" class="ltx_td ltx_align_right ltx_border_t">11.78</td>
<td id="S3.T4.1.1.17.17.7" class="ltx_td ltx_align_right ltx_border_t">3.03</td>
<td id="S3.T4.1.1.17.17.8" class="ltx_td ltx_align_right ltx_border_t">3.33</td>
<td id="S3.T4.1.1.17.17.9" class="ltx_td ltx_align_right ltx_border_t"><span id="S3.T4.1.1.17.17.9.1" class="ltx_text ltx_font_italic">6.71</span></td>
</tr>
<tr id="S3.T4.1.1.18.18" class="ltx_tr">
<td id="S3.T4.1.1.18.18.1" class="ltx_td ltx_align_right ltx_border_bb">CoT</td>
<td id="S3.T4.1.1.18.18.2" class="ltx_td ltx_align_right ltx_border_bb">4.24</td>
<td id="S3.T4.1.1.18.18.3" class="ltx_td ltx_align_right ltx_border_bb">24.88</td>
<td id="S3.T4.1.1.18.18.4" class="ltx_td ltx_align_right ltx_border_bb">12.64</td>
<td id="S3.T4.1.1.18.18.5" class="ltx_td ltx_align_right ltx_border_bb">16.76</td>
<td id="S3.T4.1.1.18.18.6" class="ltx_td ltx_align_right ltx_border_bb">5.24</td>
<td id="S3.T4.1.1.18.18.7" class="ltx_td ltx_align_right ltx_border_bb">11.06</td>
<td id="S3.T4.1.1.18.18.8" class="ltx_td ltx_align_right ltx_border_bb"><span id="S3.T4.1.1.18.18.8.1" class="ltx_text ltx_font_italic">12.47</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 4: </span>QA accuracy of each reasoning scheme, with different editors, model sizes, and prompting strategies. MEMIT was not implemented on 33b models due to GPU memory constraints. InputAug (upper bound) shows overall reasonable performance, with consistent benefits from CoT prompting and model scaling. Both QLoRA and MEMIT significantly underperform compared to InputAug, with MEMIT showing particularly low performance in certain reasoning schemes like superlative and aggregation. While QLoRA exhibits some improvement from CoT prompting, MEMIT’s performance remains consistently poor across all scenarios.</figcaption>
</figure>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Experiment</h2>

<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Language Models</h3>

<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.1" class="ltx_p">We utilize the Tülu series <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a href="#bib.bib27" title="" class="ltx_ref">2023b</a>)</cite> as the base language models to assess knowledge editing approaches. Among various options available, we find Tülu a good candidate for our study for the following reasons: 1) Tülu models are instruction-tuned and provide well-structured responses to user instructions, simplifying our factual evaluation process;<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>Their pretrained counterparts, i.e., Llama, however, would require careful output parsing for obtaining answers to ensure reliable evaluation.</span></span></span> 2) The Tülu series includes models of varying sizes, enabling us to explore the impact of model scaling on effective knowledge editing.</p>
</div>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Knowledge Editing Methods</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.1" class="ltx_p">We evaluated the following three representative knowledge editing methods<span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span>We do not evaluate meta-learning based methods such as MEND because currently, these methods are not for massive editing and editing massive knowledge leads to low efficacy.</span></span></span>.</p>
</div>
<section id="S4.SS2.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Input-augmentation</h5>

<div id="S4.SS2.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS2.SSS0.Px1.p1.1" class="ltx_p">is an inference-time editing method that appends the counterfactual facts to the question as part of the prompt. Therefore, it does not modify the model weights, but relies on model’s capability to perform reasoning from explicit context. It is considered as an upper bound <cite class="ltx_cite ltx_citemacro_cite">Onoe et al. (<a href="#bib.bib21" title="" class="ltx_ref">2023</a>, <a href="#bib.bib20" title="" class="ltx_ref">2022</a>)</cite> for model editing.</p>
</div>
</section>
<section id="S4.SS2.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Finetuning (QLoRA)</h5>

<div id="S4.SS2.SSS0.Px2.p1" class="ltx_para">
<p id="S4.SS2.SSS0.Px2.p1.1" class="ltx_p">performs gradient descent on the new facts to update model parameters. As we are tuning models up to 33 billion parameters, we adopt the parameter-efficient finetuning method QLoRA <cite class="ltx_cite ltx_citemacro_cite">Dettmers et al. (<a href="#bib.bib5" title="" class="ltx_ref">2023</a>)</cite> for the sake of computational and time efficiency.</p>
</div>
</section>
<section id="S4.SS2.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">MEMIT</h5>

<div id="S4.SS2.SSS0.Px3.p1" class="ltx_para">
<p id="S4.SS2.SSS0.Px3.p1.1" class="ltx_p">first localizes the factual knowledge in a range of layers in the transformer architecture and then updates the feedforward modules in the layers to insert a massive amount of new facts in the form of triplets. We used the implementation from <cite class="ltx_cite ltx_citemacro_citet">Zhang et al. (<a href="#bib.bib29" title="" class="ltx_ref">2024</a>)</cite>.</p>
</div>
</section>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Experiment Setting</h3>

<section id="S4.SS3.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Factual knowledge probing</h5>

<div id="S4.SS3.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS3.SSS0.Px1.p1.1" class="ltx_p">We employ both direct prompting and CoT prompting strategies to probe model’s proficiency of factual knowledge mastery and reasoning using the ReCoE dataset.
The objective is to ensure that the dataset presents a balanced level of difficulty – neither overly challenging nor too simplistic. This balance is crucial so that the language model under investigation achieves an acceptable level of accuracy for conducting meaningful counterfactual editing experiments, where we observe the model’s transition from correct to counterfactual responses. Moreover, the dataset needs to present a degree of challenge to make it a valuable asset for further research on more advanced language models.</p>
</div>
</section>
<section id="S4.SS3.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Knowledge Editing</h5>

<div id="S4.SS3.SSS0.Px2.p1" class="ltx_para">
<p id="S4.SS3.SSS0.Px2.p1.1" class="ltx_p">We evaluate model’s QA performance on the (question, counterfactual answer) pairs of each reasoning scheme post-editing. We use the <span id="S4.SS3.SSS0.Px2.p1.1.1" class="ltx_text ltx_font_italic">correct_flip</span> as the primary metric, which measures the percentage of model’s predictions that correctly transition from the original answer to the counterfactual answer.<span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span><span id="footnote3.1" class="ltx_text ltx_font_italic">correct_flip</span> mainly measures the efficacy of <span id="footnote3.2" class="ltx_text ltx_font_italic">knowledge update</span>, rather than <span id="footnote3.3" class="ltx_text ltx_font_italic">knowledge insert</span>. We focus on the “update” setting in this work to facilitate our later analysis of the changes in reasoning capabilities post-editing.</span></span></span></p>
</div>
<figure id="S4.F4" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1"><img src="/html/2401.17585/assets/x4.png" id="S4.F4.g1" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="461" height="92" alt="Refer to caption"></div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1"><img src="/html/2401.17585/assets/x5.png" id="S4.F4.g2" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="461" height="92" alt="Refer to caption"></div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>Comparison on fact-wise perplexity over facts (<math id="S4.F4.4.m1.1" class="ltx_Math" alttext="\mathbb{F}" display="inline"><semantics id="S4.F4.4.m1.1b"><mi id="S4.F4.4.m1.1.1" xref="S4.F4.4.m1.1.1.cmml">𝔽</mi><annotation-xml encoding="MathML-Content" id="S4.F4.4.m1.1c"><ci id="S4.F4.4.m1.1.1.cmml" xref="S4.F4.4.m1.1.1">𝔽</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.F4.4.m1.1d">\mathbb{F}</annotation></semantics></math>) and counterfactuals (<math id="S4.F4.5.m2.1" class="ltx_Math" alttext="\mathbb{CF}" display="inline"><semantics id="S4.F4.5.m2.1b"><mrow id="S4.F4.5.m2.1.1" xref="S4.F4.5.m2.1.1.cmml"><mi id="S4.F4.5.m2.1.1.2" xref="S4.F4.5.m2.1.1.2.cmml">ℂ</mi><mo lspace="0em" rspace="0em" id="S4.F4.5.m2.1.1.1" xref="S4.F4.5.m2.1.1.1.cmml">​</mo><mi id="S4.F4.5.m2.1.1.3" xref="S4.F4.5.m2.1.1.3.cmml">𝔽</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.F4.5.m2.1c"><apply id="S4.F4.5.m2.1.1.cmml" xref="S4.F4.5.m2.1.1"><times id="S4.F4.5.m2.1.1.1.cmml" xref="S4.F4.5.m2.1.1.1"></times><ci id="S4.F4.5.m2.1.1.2.cmml" xref="S4.F4.5.m2.1.1.2">ℂ</ci><ci id="S4.F4.5.m2.1.1.3.cmml" xref="S4.F4.5.m2.1.1.3">𝔽</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.F4.5.m2.1d">\mathbb{CF}</annotation></semantics></math>) before and after editing using QLoRA and MEMIT (7B). A successful fact-wise edit is indicated by a transition from the upper-triangle region where <math id="S4.F4.6.m3.2" class="ltx_Math" alttext="\mathtt{PPL}(\mathbb{CF})&gt;\mathtt{PPL}(\mathbb{F})" display="inline"><semantics id="S4.F4.6.m3.2b"><mrow id="S4.F4.6.m3.2.2" xref="S4.F4.6.m3.2.2.cmml"><mrow id="S4.F4.6.m3.2.2.1" xref="S4.F4.6.m3.2.2.1.cmml"><mi id="S4.F4.6.m3.2.2.1.3" xref="S4.F4.6.m3.2.2.1.3.cmml">𝙿𝙿𝙻</mi><mo lspace="0em" rspace="0em" id="S4.F4.6.m3.2.2.1.2" xref="S4.F4.6.m3.2.2.1.2.cmml">​</mo><mrow id="S4.F4.6.m3.2.2.1.1.1" xref="S4.F4.6.m3.2.2.1.1.1.1.cmml"><mo stretchy="false" id="S4.F4.6.m3.2.2.1.1.1.2" xref="S4.F4.6.m3.2.2.1.1.1.1.cmml">(</mo><mrow id="S4.F4.6.m3.2.2.1.1.1.1" xref="S4.F4.6.m3.2.2.1.1.1.1.cmml"><mi id="S4.F4.6.m3.2.2.1.1.1.1.2" xref="S4.F4.6.m3.2.2.1.1.1.1.2.cmml">ℂ</mi><mo lspace="0em" rspace="0em" id="S4.F4.6.m3.2.2.1.1.1.1.1" xref="S4.F4.6.m3.2.2.1.1.1.1.1.cmml">​</mo><mi id="S4.F4.6.m3.2.2.1.1.1.1.3" xref="S4.F4.6.m3.2.2.1.1.1.1.3.cmml">𝔽</mi></mrow><mo stretchy="false" id="S4.F4.6.m3.2.2.1.1.1.3" xref="S4.F4.6.m3.2.2.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S4.F4.6.m3.2.2.2" xref="S4.F4.6.m3.2.2.2.cmml">&gt;</mo><mrow id="S4.F4.6.m3.2.2.3" xref="S4.F4.6.m3.2.2.3.cmml"><mi id="S4.F4.6.m3.2.2.3.2" xref="S4.F4.6.m3.2.2.3.2.cmml">𝙿𝙿𝙻</mi><mo lspace="0em" rspace="0em" id="S4.F4.6.m3.2.2.3.1" xref="S4.F4.6.m3.2.2.3.1.cmml">​</mo><mrow id="S4.F4.6.m3.2.2.3.3.2" xref="S4.F4.6.m3.2.2.3.cmml"><mo stretchy="false" id="S4.F4.6.m3.2.2.3.3.2.1" xref="S4.F4.6.m3.2.2.3.cmml">(</mo><mi id="S4.F4.6.m3.1.1" xref="S4.F4.6.m3.1.1.cmml">𝔽</mi><mo stretchy="false" id="S4.F4.6.m3.2.2.3.3.2.2" xref="S4.F4.6.m3.2.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.F4.6.m3.2c"><apply id="S4.F4.6.m3.2.2.cmml" xref="S4.F4.6.m3.2.2"><gt id="S4.F4.6.m3.2.2.2.cmml" xref="S4.F4.6.m3.2.2.2"></gt><apply id="S4.F4.6.m3.2.2.1.cmml" xref="S4.F4.6.m3.2.2.1"><times id="S4.F4.6.m3.2.2.1.2.cmml" xref="S4.F4.6.m3.2.2.1.2"></times><ci id="S4.F4.6.m3.2.2.1.3.cmml" xref="S4.F4.6.m3.2.2.1.3">𝙿𝙿𝙻</ci><apply id="S4.F4.6.m3.2.2.1.1.1.1.cmml" xref="S4.F4.6.m3.2.2.1.1.1"><times id="S4.F4.6.m3.2.2.1.1.1.1.1.cmml" xref="S4.F4.6.m3.2.2.1.1.1.1.1"></times><ci id="S4.F4.6.m3.2.2.1.1.1.1.2.cmml" xref="S4.F4.6.m3.2.2.1.1.1.1.2">ℂ</ci><ci id="S4.F4.6.m3.2.2.1.1.1.1.3.cmml" xref="S4.F4.6.m3.2.2.1.1.1.1.3">𝔽</ci></apply></apply><apply id="S4.F4.6.m3.2.2.3.cmml" xref="S4.F4.6.m3.2.2.3"><times id="S4.F4.6.m3.2.2.3.1.cmml" xref="S4.F4.6.m3.2.2.3.1"></times><ci id="S4.F4.6.m3.2.2.3.2.cmml" xref="S4.F4.6.m3.2.2.3.2">𝙿𝙿𝙻</ci><ci id="S4.F4.6.m3.1.1.cmml" xref="S4.F4.6.m3.1.1">𝔽</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.F4.6.m3.2d">\mathtt{PPL}(\mathbb{CF})&gt;\mathtt{PPL}(\mathbb{F})</annotation></semantics></math> to the lower-triangle region. QLoRA-based finetuning demonstrate notable effectiveness, in contrast to MEMIT. Similar trends are observed with 13b and 33b models, as detailed in Appendix <a href="#A1.SS8" title="A.8 Fact-wise Perplexity ‣ Appendix A Appendix ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.8</span></a>.</figcaption>
</figure>
</section>
</section>
<section id="S4.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.4 </span>Experiment Result</h3>

<div id="S4.SS4.p1" class="ltx_para">
<p id="S4.SS4.p1.1" class="ltx_p">Table <a href="#S3.T3" title="Table 3 ‣ 3.3 Fact Representation ‣ 3 ReCoE Dataset ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> displays the outcomes of the knowledge probing exercise. The results clearly demonstrate the significant impact of model scaling and the beneficial role of CoT. Table <a href="#S3.T4" title="Table 4 ‣ 3.3 Fact Representation ‣ 3 ReCoE Dataset ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> summarizes the <span id="S4.SS4.p1.1.1" class="ltx_text ltx_font_italic">correct_flip</span> results of InputAug (input-augmentation), QLoRA-based finetuning, and MEMIT.
InputAug involves incorporating counterfactual information into the context, where both model scaling and CoT are shown to be beneficial. Input augmentation is often treated as the upper bound for model editing. But we can see that the performance for aggregation and subtraction is still unsatisfying, below 50%.
Both QLoRA and MEMIT editing significantly underperform InputAug across all model scales and prompting strategies, indicating failed knowledge propagation of these methods.
Interestingly, QLoRA-based finetuning, despite its deteriorating performance, can still benefit from CoT prompting and model scaling. In contrast, MEMIT consistently failed, indicating a significant deterioration in the model’s reasoning capability.</p>
</div>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Analysis</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.3" class="ltx_p">To comprehensively evaluate how certain knowledge editing methodologies impact model’s capability that leads to ineffectiveness in knowledge propagation, our analysis encompasses three key dimensions: <span id="S5.p1.3.1" class="ltx_text ltx_font_italic">fact-wise editing effectiveness</span>, <span id="S5.p1.3.2" class="ltx_text ltx_font_italic">fact recall accuracy</span>, and <span id="S5.p1.3.3" class="ltx_text ltx_font_italic">logical coherence</span> in model’s generation.
We assume the reasoning process follows a retrieve-and-generate regime. Formally,</p>
<table id="S5.Ex1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S5.Ex1.m1.4" class="ltx_Math" alttext="P^{\prime}(\mathrm{CA}|\mathrm{Q})=\underbrace{P^{\prime}(\mathbb{CF}|\mathrm{Q})}_{\text{fact recall}}\cdot\underbrace{P^{\prime}(\mathrm{CA}|\mathrm{Q},\mathbb{CF})}_{\text{coherent generation}}" display="block"><semantics id="S5.Ex1.m1.4a"><mrow id="S5.Ex1.m1.4.4" xref="S5.Ex1.m1.4.4.cmml"><mrow id="S5.Ex1.m1.4.4.1" xref="S5.Ex1.m1.4.4.1.cmml"><msup id="S5.Ex1.m1.4.4.1.3" xref="S5.Ex1.m1.4.4.1.3.cmml"><mi id="S5.Ex1.m1.4.4.1.3.2" xref="S5.Ex1.m1.4.4.1.3.2.cmml">P</mi><mo id="S5.Ex1.m1.4.4.1.3.3" xref="S5.Ex1.m1.4.4.1.3.3.cmml">′</mo></msup><mo lspace="0em" rspace="0em" id="S5.Ex1.m1.4.4.1.2" xref="S5.Ex1.m1.4.4.1.2.cmml">​</mo><mrow id="S5.Ex1.m1.4.4.1.1.1" xref="S5.Ex1.m1.4.4.1.1.1.1.cmml"><mo stretchy="false" id="S5.Ex1.m1.4.4.1.1.1.2" xref="S5.Ex1.m1.4.4.1.1.1.1.cmml">(</mo><mrow id="S5.Ex1.m1.4.4.1.1.1.1" xref="S5.Ex1.m1.4.4.1.1.1.1.cmml"><mi id="S5.Ex1.m1.4.4.1.1.1.1.2" xref="S5.Ex1.m1.4.4.1.1.1.1.2.cmml">CA</mi><mo fence="false" id="S5.Ex1.m1.4.4.1.1.1.1.1" xref="S5.Ex1.m1.4.4.1.1.1.1.1.cmml">|</mo><mi mathvariant="normal" id="S5.Ex1.m1.4.4.1.1.1.1.3" xref="S5.Ex1.m1.4.4.1.1.1.1.3.cmml">Q</mi></mrow><mo stretchy="false" id="S5.Ex1.m1.4.4.1.1.1.3" xref="S5.Ex1.m1.4.4.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S5.Ex1.m1.4.4.2" xref="S5.Ex1.m1.4.4.2.cmml">=</mo><mrow id="S5.Ex1.m1.4.4.3" xref="S5.Ex1.m1.4.4.3.cmml"><munder id="S5.Ex1.m1.4.4.3.2" xref="S5.Ex1.m1.4.4.3.2.cmml"><munder accentunder="true" id="S5.Ex1.m1.1.1" xref="S5.Ex1.m1.1.1.cmml"><mrow id="S5.Ex1.m1.1.1.1" xref="S5.Ex1.m1.1.1.1.cmml"><msup id="S5.Ex1.m1.1.1.1.3" xref="S5.Ex1.m1.1.1.1.3.cmml"><mi id="S5.Ex1.m1.1.1.1.3.2" xref="S5.Ex1.m1.1.1.1.3.2.cmml">P</mi><mo id="S5.Ex1.m1.1.1.1.3.3" xref="S5.Ex1.m1.1.1.1.3.3.cmml">′</mo></msup><mo lspace="0em" rspace="0em" id="S5.Ex1.m1.1.1.1.2" xref="S5.Ex1.m1.1.1.1.2.cmml">​</mo><mrow id="S5.Ex1.m1.1.1.1.1.1" xref="S5.Ex1.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S5.Ex1.m1.1.1.1.1.1.2" xref="S5.Ex1.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="S5.Ex1.m1.1.1.1.1.1.1" xref="S5.Ex1.m1.1.1.1.1.1.1.cmml"><mrow id="S5.Ex1.m1.1.1.1.1.1.1.2" xref="S5.Ex1.m1.1.1.1.1.1.1.2.cmml"><mi id="S5.Ex1.m1.1.1.1.1.1.1.2.2" xref="S5.Ex1.m1.1.1.1.1.1.1.2.2.cmml">ℂ</mi><mo lspace="0em" rspace="0em" id="S5.Ex1.m1.1.1.1.1.1.1.2.1" xref="S5.Ex1.m1.1.1.1.1.1.1.2.1.cmml">​</mo><mi id="S5.Ex1.m1.1.1.1.1.1.1.2.3" xref="S5.Ex1.m1.1.1.1.1.1.1.2.3.cmml">𝔽</mi></mrow><mo fence="false" id="S5.Ex1.m1.1.1.1.1.1.1.1" xref="S5.Ex1.m1.1.1.1.1.1.1.1.cmml">|</mo><mi mathvariant="normal" id="S5.Ex1.m1.1.1.1.1.1.1.3" xref="S5.Ex1.m1.1.1.1.1.1.1.3.cmml">Q</mi></mrow><mo rspace="0.055em" stretchy="false" id="S5.Ex1.m1.1.1.1.1.1.3" xref="S5.Ex1.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S5.Ex1.m1.1.1.2" xref="S5.Ex1.m1.1.1.2.cmml">⏟</mo></munder><mtext id="S5.Ex1.m1.4.4.3.2.2" xref="S5.Ex1.m1.4.4.3.2.2a.cmml">fact recall</mtext></munder><mo rspace="0.222em" id="S5.Ex1.m1.4.4.3.1" xref="S5.Ex1.m1.4.4.3.1.cmml">⋅</mo><munder id="S5.Ex1.m1.4.4.3.3" xref="S5.Ex1.m1.4.4.3.3.cmml"><munder accentunder="true" id="S5.Ex1.m1.3.3" xref="S5.Ex1.m1.3.3.cmml"><mrow id="S5.Ex1.m1.3.3.2" xref="S5.Ex1.m1.3.3.2.cmml"><msup id="S5.Ex1.m1.3.3.2.4" xref="S5.Ex1.m1.3.3.2.4.cmml"><mi id="S5.Ex1.m1.3.3.2.4.2" xref="S5.Ex1.m1.3.3.2.4.2.cmml">P</mi><mo id="S5.Ex1.m1.3.3.2.4.3" xref="S5.Ex1.m1.3.3.2.4.3.cmml">′</mo></msup><mo lspace="0em" rspace="0em" id="S5.Ex1.m1.3.3.2.3" xref="S5.Ex1.m1.3.3.2.3.cmml">​</mo><mrow id="S5.Ex1.m1.3.3.2.2.1" xref="S5.Ex1.m1.3.3.2.2.1.1.cmml"><mo stretchy="false" id="S5.Ex1.m1.3.3.2.2.1.2" xref="S5.Ex1.m1.3.3.2.2.1.1.cmml">(</mo><mrow id="S5.Ex1.m1.3.3.2.2.1.1" xref="S5.Ex1.m1.3.3.2.2.1.1.cmml"><mi id="S5.Ex1.m1.3.3.2.2.1.1.3" xref="S5.Ex1.m1.3.3.2.2.1.1.3.cmml">CA</mi><mo fence="false" id="S5.Ex1.m1.3.3.2.2.1.1.2" xref="S5.Ex1.m1.3.3.2.2.1.1.2.cmml">|</mo><mrow id="S5.Ex1.m1.3.3.2.2.1.1.1.1" xref="S5.Ex1.m1.3.3.2.2.1.1.1.2.cmml"><mi mathvariant="normal" id="S5.Ex1.m1.2.2.1.1" xref="S5.Ex1.m1.2.2.1.1.cmml">Q</mi><mo id="S5.Ex1.m1.3.3.2.2.1.1.1.1.2" xref="S5.Ex1.m1.3.3.2.2.1.1.1.2.cmml">,</mo><mrow id="S5.Ex1.m1.3.3.2.2.1.1.1.1.1" xref="S5.Ex1.m1.3.3.2.2.1.1.1.1.1.cmml"><mi id="S5.Ex1.m1.3.3.2.2.1.1.1.1.1.2" xref="S5.Ex1.m1.3.3.2.2.1.1.1.1.1.2.cmml">ℂ</mi><mo lspace="0em" rspace="0em" id="S5.Ex1.m1.3.3.2.2.1.1.1.1.1.1" xref="S5.Ex1.m1.3.3.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="S5.Ex1.m1.3.3.2.2.1.1.1.1.1.3" xref="S5.Ex1.m1.3.3.2.2.1.1.1.1.1.3.cmml">𝔽</mi></mrow></mrow></mrow><mo stretchy="false" id="S5.Ex1.m1.3.3.2.2.1.3" xref="S5.Ex1.m1.3.3.2.2.1.1.cmml">)</mo></mrow></mrow><mo id="S5.Ex1.m1.3.3.3" xref="S5.Ex1.m1.3.3.3.cmml">⏟</mo></munder><mtext id="S5.Ex1.m1.4.4.3.3.2" xref="S5.Ex1.m1.4.4.3.3.2a.cmml">coherent generation</mtext></munder></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.Ex1.m1.4b"><apply id="S5.Ex1.m1.4.4.cmml" xref="S5.Ex1.m1.4.4"><eq id="S5.Ex1.m1.4.4.2.cmml" xref="S5.Ex1.m1.4.4.2"></eq><apply id="S5.Ex1.m1.4.4.1.cmml" xref="S5.Ex1.m1.4.4.1"><times id="S5.Ex1.m1.4.4.1.2.cmml" xref="S5.Ex1.m1.4.4.1.2"></times><apply id="S5.Ex1.m1.4.4.1.3.cmml" xref="S5.Ex1.m1.4.4.1.3"><csymbol cd="ambiguous" id="S5.Ex1.m1.4.4.1.3.1.cmml" xref="S5.Ex1.m1.4.4.1.3">superscript</csymbol><ci id="S5.Ex1.m1.4.4.1.3.2.cmml" xref="S5.Ex1.m1.4.4.1.3.2">𝑃</ci><ci id="S5.Ex1.m1.4.4.1.3.3.cmml" xref="S5.Ex1.m1.4.4.1.3.3">′</ci></apply><apply id="S5.Ex1.m1.4.4.1.1.1.1.cmml" xref="S5.Ex1.m1.4.4.1.1.1"><csymbol cd="latexml" id="S5.Ex1.m1.4.4.1.1.1.1.1.cmml" xref="S5.Ex1.m1.4.4.1.1.1.1.1">conditional</csymbol><ci id="S5.Ex1.m1.4.4.1.1.1.1.2.cmml" xref="S5.Ex1.m1.4.4.1.1.1.1.2">CA</ci><ci id="S5.Ex1.m1.4.4.1.1.1.1.3.cmml" xref="S5.Ex1.m1.4.4.1.1.1.1.3">Q</ci></apply></apply><apply id="S5.Ex1.m1.4.4.3.cmml" xref="S5.Ex1.m1.4.4.3"><ci id="S5.Ex1.m1.4.4.3.1.cmml" xref="S5.Ex1.m1.4.4.3.1">⋅</ci><apply id="S5.Ex1.m1.4.4.3.2.cmml" xref="S5.Ex1.m1.4.4.3.2"><csymbol cd="ambiguous" id="S5.Ex1.m1.4.4.3.2.1.cmml" xref="S5.Ex1.m1.4.4.3.2">subscript</csymbol><apply id="S5.Ex1.m1.1.1.cmml" xref="S5.Ex1.m1.1.1"><ci id="S5.Ex1.m1.1.1.2.cmml" xref="S5.Ex1.m1.1.1.2">⏟</ci><apply id="S5.Ex1.m1.1.1.1.cmml" xref="S5.Ex1.m1.1.1.1"><times id="S5.Ex1.m1.1.1.1.2.cmml" xref="S5.Ex1.m1.1.1.1.2"></times><apply id="S5.Ex1.m1.1.1.1.3.cmml" xref="S5.Ex1.m1.1.1.1.3"><csymbol cd="ambiguous" id="S5.Ex1.m1.1.1.1.3.1.cmml" xref="S5.Ex1.m1.1.1.1.3">superscript</csymbol><ci id="S5.Ex1.m1.1.1.1.3.2.cmml" xref="S5.Ex1.m1.1.1.1.3.2">𝑃</ci><ci id="S5.Ex1.m1.1.1.1.3.3.cmml" xref="S5.Ex1.m1.1.1.1.3.3">′</ci></apply><apply id="S5.Ex1.m1.1.1.1.1.1.1.cmml" xref="S5.Ex1.m1.1.1.1.1.1"><csymbol cd="latexml" id="S5.Ex1.m1.1.1.1.1.1.1.1.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.1">conditional</csymbol><apply id="S5.Ex1.m1.1.1.1.1.1.1.2.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.2"><times id="S5.Ex1.m1.1.1.1.1.1.1.2.1.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.2.1"></times><ci id="S5.Ex1.m1.1.1.1.1.1.1.2.2.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.2.2">ℂ</ci><ci id="S5.Ex1.m1.1.1.1.1.1.1.2.3.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.2.3">𝔽</ci></apply><ci id="S5.Ex1.m1.1.1.1.1.1.1.3.cmml" xref="S5.Ex1.m1.1.1.1.1.1.1.3">Q</ci></apply></apply></apply><ci id="S5.Ex1.m1.4.4.3.2.2a.cmml" xref="S5.Ex1.m1.4.4.3.2.2"><mtext mathsize="70%" id="S5.Ex1.m1.4.4.3.2.2.cmml" xref="S5.Ex1.m1.4.4.3.2.2">fact recall</mtext></ci></apply><apply id="S5.Ex1.m1.4.4.3.3.cmml" xref="S5.Ex1.m1.4.4.3.3"><csymbol cd="ambiguous" id="S5.Ex1.m1.4.4.3.3.1.cmml" xref="S5.Ex1.m1.4.4.3.3">subscript</csymbol><apply id="S5.Ex1.m1.3.3.cmml" xref="S5.Ex1.m1.3.3"><ci id="S5.Ex1.m1.3.3.3.cmml" xref="S5.Ex1.m1.3.3.3">⏟</ci><apply id="S5.Ex1.m1.3.3.2.cmml" xref="S5.Ex1.m1.3.3.2"><times id="S5.Ex1.m1.3.3.2.3.cmml" xref="S5.Ex1.m1.3.3.2.3"></times><apply id="S5.Ex1.m1.3.3.2.4.cmml" xref="S5.Ex1.m1.3.3.2.4"><csymbol cd="ambiguous" id="S5.Ex1.m1.3.3.2.4.1.cmml" xref="S5.Ex1.m1.3.3.2.4">superscript</csymbol><ci id="S5.Ex1.m1.3.3.2.4.2.cmml" xref="S5.Ex1.m1.3.3.2.4.2">𝑃</ci><ci id="S5.Ex1.m1.3.3.2.4.3.cmml" xref="S5.Ex1.m1.3.3.2.4.3">′</ci></apply><apply id="S5.Ex1.m1.3.3.2.2.1.1.cmml" xref="S5.Ex1.m1.3.3.2.2.1"><csymbol cd="latexml" id="S5.Ex1.m1.3.3.2.2.1.1.2.cmml" xref="S5.Ex1.m1.3.3.2.2.1.1.2">conditional</csymbol><ci id="S5.Ex1.m1.3.3.2.2.1.1.3.cmml" xref="S5.Ex1.m1.3.3.2.2.1.1.3">CA</ci><list id="S5.Ex1.m1.3.3.2.2.1.1.1.2.cmml" xref="S5.Ex1.m1.3.3.2.2.1.1.1.1"><ci id="S5.Ex1.m1.2.2.1.1.cmml" xref="S5.Ex1.m1.2.2.1.1">Q</ci><apply id="S5.Ex1.m1.3.3.2.2.1.1.1.1.1.cmml" xref="S5.Ex1.m1.3.3.2.2.1.1.1.1.1"><times id="S5.Ex1.m1.3.3.2.2.1.1.1.1.1.1.cmml" xref="S5.Ex1.m1.3.3.2.2.1.1.1.1.1.1"></times><ci id="S5.Ex1.m1.3.3.2.2.1.1.1.1.1.2.cmml" xref="S5.Ex1.m1.3.3.2.2.1.1.1.1.1.2">ℂ</ci><ci id="S5.Ex1.m1.3.3.2.2.1.1.1.1.1.3.cmml" xref="S5.Ex1.m1.3.3.2.2.1.1.1.1.1.3">𝔽</ci></apply></list></apply></apply></apply><ci id="S5.Ex1.m1.4.4.3.3.2a.cmml" xref="S5.Ex1.m1.4.4.3.3.2"><mtext mathsize="70%" id="S5.Ex1.m1.4.4.3.3.2.cmml" xref="S5.Ex1.m1.4.4.3.3.2">coherent generation</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.Ex1.m1.4c">P^{\prime}(\mathrm{CA}|\mathrm{Q})=\underbrace{P^{\prime}(\mathbb{CF}|\mathrm{Q})}_{\text{fact recall}}\cdot\underbrace{P^{\prime}(\mathrm{CA}|\mathrm{Q},\mathbb{CF})}_{\text{coherent generation}}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S5.p1.2" class="ltx_p">where <math id="S5.p1.1.m1.1" class="ltx_Math" alttext="P^{\prime}" display="inline"><semantics id="S5.p1.1.m1.1a"><msup id="S5.p1.1.m1.1.1" xref="S5.p1.1.m1.1.1.cmml"><mi id="S5.p1.1.m1.1.1.2" xref="S5.p1.1.m1.1.1.2.cmml">P</mi><mo id="S5.p1.1.m1.1.1.3" xref="S5.p1.1.m1.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S5.p1.1.m1.1b"><apply id="S5.p1.1.m1.1.1.cmml" xref="S5.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S5.p1.1.m1.1.1.1.cmml" xref="S5.p1.1.m1.1.1">superscript</csymbol><ci id="S5.p1.1.m1.1.1.2.cmml" xref="S5.p1.1.m1.1.1.2">𝑃</ci><ci id="S5.p1.1.m1.1.1.3.cmml" xref="S5.p1.1.m1.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p1.1.m1.1c">P^{\prime}</annotation></semantics></math> is the edited LM.
The fact recall component requires 1) each fact within <math id="S5.p1.2.m2.1" class="ltx_Math" alttext="\mathbb{CF}" display="inline"><semantics id="S5.p1.2.m2.1a"><mrow id="S5.p1.2.m2.1.1" xref="S5.p1.2.m2.1.1.cmml"><mi id="S5.p1.2.m2.1.1.2" xref="S5.p1.2.m2.1.1.2.cmml">ℂ</mi><mo lspace="0em" rspace="0em" id="S5.p1.2.m2.1.1.1" xref="S5.p1.2.m2.1.1.1.cmml">​</mo><mi id="S5.p1.2.m2.1.1.3" xref="S5.p1.2.m2.1.1.3.cmml">𝔽</mi></mrow><annotation-xml encoding="MathML-Content" id="S5.p1.2.m2.1b"><apply id="S5.p1.2.m2.1.1.cmml" xref="S5.p1.2.m2.1.1"><times id="S5.p1.2.m2.1.1.1.cmml" xref="S5.p1.2.m2.1.1.1"></times><ci id="S5.p1.2.m2.1.1.2.cmml" xref="S5.p1.2.m2.1.1.2">ℂ</ci><ci id="S5.p1.2.m2.1.1.3.cmml" xref="S5.p1.2.m2.1.1.3">𝔽</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p1.2.m2.1c">\mathbb{CF}</annotation></semantics></math> to be effectively edited; and 2) edited model is able to recall these edited facts through generation.
The coherent generation component further requires the generated answer to be logically coherent with the retrieved facts.</p>
</div>
<section id="S5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.1 </span>Fact-wise Editing Effectiveness</h3>

<div id="S5.SS1.p1" class="ltx_para">
<p id="S5.SS1.p1.8" class="ltx_p">This dimension examines the basic efficacy of editing methods. It assesses whether the applied edits achieve their intended modifications successfully, which constitutes the foundational requirement for any knowledge editing approach. Defining <math id="S5.SS1.p1.1.m1.1" class="ltx_Math" alttext="\mathtt{PPL}(\mathbb{F})" display="inline"><semantics id="S5.SS1.p1.1.m1.1a"><mrow id="S5.SS1.p1.1.m1.1.2" xref="S5.SS1.p1.1.m1.1.2.cmml"><mi id="S5.SS1.p1.1.m1.1.2.2" xref="S5.SS1.p1.1.m1.1.2.2.cmml">𝙿𝙿𝙻</mi><mo lspace="0em" rspace="0em" id="S5.SS1.p1.1.m1.1.2.1" xref="S5.SS1.p1.1.m1.1.2.1.cmml">​</mo><mrow id="S5.SS1.p1.1.m1.1.2.3.2" xref="S5.SS1.p1.1.m1.1.2.cmml"><mo stretchy="false" id="S5.SS1.p1.1.m1.1.2.3.2.1" xref="S5.SS1.p1.1.m1.1.2.cmml">(</mo><mi id="S5.SS1.p1.1.m1.1.1" xref="S5.SS1.p1.1.m1.1.1.cmml">𝔽</mi><mo stretchy="false" id="S5.SS1.p1.1.m1.1.2.3.2.2" xref="S5.SS1.p1.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.1.m1.1b"><apply id="S5.SS1.p1.1.m1.1.2.cmml" xref="S5.SS1.p1.1.m1.1.2"><times id="S5.SS1.p1.1.m1.1.2.1.cmml" xref="S5.SS1.p1.1.m1.1.2.1"></times><ci id="S5.SS1.p1.1.m1.1.2.2.cmml" xref="S5.SS1.p1.1.m1.1.2.2">𝙿𝙿𝙻</ci><ci id="S5.SS1.p1.1.m1.1.1.cmml" xref="S5.SS1.p1.1.m1.1.1">𝔽</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.1.m1.1c">\mathtt{PPL}(\mathbb{F})</annotation></semantics></math> and <math id="S5.SS1.p1.2.m2.1" class="ltx_Math" alttext="\mathtt{PPL}(\mathbb{CF})" display="inline"><semantics id="S5.SS1.p1.2.m2.1a"><mrow id="S5.SS1.p1.2.m2.1.1" xref="S5.SS1.p1.2.m2.1.1.cmml"><mi id="S5.SS1.p1.2.m2.1.1.3" xref="S5.SS1.p1.2.m2.1.1.3.cmml">𝙿𝙿𝙻</mi><mo lspace="0em" rspace="0em" id="S5.SS1.p1.2.m2.1.1.2" xref="S5.SS1.p1.2.m2.1.1.2.cmml">​</mo><mrow id="S5.SS1.p1.2.m2.1.1.1.1" xref="S5.SS1.p1.2.m2.1.1.1.1.1.cmml"><mo stretchy="false" id="S5.SS1.p1.2.m2.1.1.1.1.2" xref="S5.SS1.p1.2.m2.1.1.1.1.1.cmml">(</mo><mrow id="S5.SS1.p1.2.m2.1.1.1.1.1" xref="S5.SS1.p1.2.m2.1.1.1.1.1.cmml"><mi id="S5.SS1.p1.2.m2.1.1.1.1.1.2" xref="S5.SS1.p1.2.m2.1.1.1.1.1.2.cmml">ℂ</mi><mo lspace="0em" rspace="0em" id="S5.SS1.p1.2.m2.1.1.1.1.1.1" xref="S5.SS1.p1.2.m2.1.1.1.1.1.1.cmml">​</mo><mi id="S5.SS1.p1.2.m2.1.1.1.1.1.3" xref="S5.SS1.p1.2.m2.1.1.1.1.1.3.cmml">𝔽</mi></mrow><mo stretchy="false" id="S5.SS1.p1.2.m2.1.1.1.1.3" xref="S5.SS1.p1.2.m2.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.2.m2.1b"><apply id="S5.SS1.p1.2.m2.1.1.cmml" xref="S5.SS1.p1.2.m2.1.1"><times id="S5.SS1.p1.2.m2.1.1.2.cmml" xref="S5.SS1.p1.2.m2.1.1.2"></times><ci id="S5.SS1.p1.2.m2.1.1.3.cmml" xref="S5.SS1.p1.2.m2.1.1.3">𝙿𝙿𝙻</ci><apply id="S5.SS1.p1.2.m2.1.1.1.1.1.cmml" xref="S5.SS1.p1.2.m2.1.1.1.1"><times id="S5.SS1.p1.2.m2.1.1.1.1.1.1.cmml" xref="S5.SS1.p1.2.m2.1.1.1.1.1.1"></times><ci id="S5.SS1.p1.2.m2.1.1.1.1.1.2.cmml" xref="S5.SS1.p1.2.m2.1.1.1.1.1.2">ℂ</ci><ci id="S5.SS1.p1.2.m2.1.1.1.1.1.3.cmml" xref="S5.SS1.p1.2.m2.1.1.1.1.1.3">𝔽</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.2.m2.1c">\mathtt{PPL}(\mathbb{CF})</annotation></semantics></math> as the averaged perplexity of the facts and the counterfactuals associated to a <math id="S5.SS1.p1.3.m3.2" class="ltx_math_unparsed" alttext="(\mathrm{Q},\mathrm{A}" display="inline"><semantics id="S5.SS1.p1.3.m3.2a"><mrow id="S5.SS1.p1.3.m3.2b"><mo stretchy="false" id="S5.SS1.p1.3.m3.2.3">(</mo><mi mathvariant="normal" id="S5.SS1.p1.3.m3.1.1">Q</mi><mo id="S5.SS1.p1.3.m3.2.4">,</mo><mi mathvariant="normal" id="S5.SS1.p1.3.m3.2.2">A</mi></mrow><annotation encoding="application/x-tex" id="S5.SS1.p1.3.m3.2c">(\mathrm{Q},\mathrm{A}</annotation></semantics></math>) pair, and <math id="S5.SS1.p1.4.m4.3" class="ltx_Math" alttext="\Delta(\mathbb{CF},\mathbb{F})=\mathtt{PPL}(\mathbb{CF})" display="inline"><semantics id="S5.SS1.p1.4.m4.3a"><mrow id="S5.SS1.p1.4.m4.3.3" xref="S5.SS1.p1.4.m4.3.3.cmml"><mrow id="S5.SS1.p1.4.m4.2.2.1" xref="S5.SS1.p1.4.m4.2.2.1.cmml"><mi mathvariant="normal" id="S5.SS1.p1.4.m4.2.2.1.3" xref="S5.SS1.p1.4.m4.2.2.1.3.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S5.SS1.p1.4.m4.2.2.1.2" xref="S5.SS1.p1.4.m4.2.2.1.2.cmml">​</mo><mrow id="S5.SS1.p1.4.m4.2.2.1.1.1" xref="S5.SS1.p1.4.m4.2.2.1.1.2.cmml"><mo stretchy="false" id="S5.SS1.p1.4.m4.2.2.1.1.1.2" xref="S5.SS1.p1.4.m4.2.2.1.1.2.cmml">(</mo><mrow id="S5.SS1.p1.4.m4.2.2.1.1.1.1" xref="S5.SS1.p1.4.m4.2.2.1.1.1.1.cmml"><mi id="S5.SS1.p1.4.m4.2.2.1.1.1.1.2" xref="S5.SS1.p1.4.m4.2.2.1.1.1.1.2.cmml">ℂ</mi><mo lspace="0em" rspace="0em" id="S5.SS1.p1.4.m4.2.2.1.1.1.1.1" xref="S5.SS1.p1.4.m4.2.2.1.1.1.1.1.cmml">​</mo><mi id="S5.SS1.p1.4.m4.2.2.1.1.1.1.3" xref="S5.SS1.p1.4.m4.2.2.1.1.1.1.3.cmml">𝔽</mi></mrow><mo id="S5.SS1.p1.4.m4.2.2.1.1.1.3" xref="S5.SS1.p1.4.m4.2.2.1.1.2.cmml">,</mo><mi id="S5.SS1.p1.4.m4.1.1" xref="S5.SS1.p1.4.m4.1.1.cmml">𝔽</mi><mo stretchy="false" id="S5.SS1.p1.4.m4.2.2.1.1.1.4" xref="S5.SS1.p1.4.m4.2.2.1.1.2.cmml">)</mo></mrow></mrow><mo id="S5.SS1.p1.4.m4.3.3.3" xref="S5.SS1.p1.4.m4.3.3.3.cmml">=</mo><mrow id="S5.SS1.p1.4.m4.3.3.2" xref="S5.SS1.p1.4.m4.3.3.2.cmml"><mi id="S5.SS1.p1.4.m4.3.3.2.3" xref="S5.SS1.p1.4.m4.3.3.2.3.cmml">𝙿𝙿𝙻</mi><mo lspace="0em" rspace="0em" id="S5.SS1.p1.4.m4.3.3.2.2" xref="S5.SS1.p1.4.m4.3.3.2.2.cmml">​</mo><mrow id="S5.SS1.p1.4.m4.3.3.2.1.1" xref="S5.SS1.p1.4.m4.3.3.2.1.1.1.cmml"><mo stretchy="false" id="S5.SS1.p1.4.m4.3.3.2.1.1.2" xref="S5.SS1.p1.4.m4.3.3.2.1.1.1.cmml">(</mo><mrow id="S5.SS1.p1.4.m4.3.3.2.1.1.1" xref="S5.SS1.p1.4.m4.3.3.2.1.1.1.cmml"><mi id="S5.SS1.p1.4.m4.3.3.2.1.1.1.2" xref="S5.SS1.p1.4.m4.3.3.2.1.1.1.2.cmml">ℂ</mi><mo lspace="0em" rspace="0em" id="S5.SS1.p1.4.m4.3.3.2.1.1.1.1" xref="S5.SS1.p1.4.m4.3.3.2.1.1.1.1.cmml">​</mo><mi id="S5.SS1.p1.4.m4.3.3.2.1.1.1.3" xref="S5.SS1.p1.4.m4.3.3.2.1.1.1.3.cmml">𝔽</mi></mrow><mo stretchy="false" id="S5.SS1.p1.4.m4.3.3.2.1.1.3" xref="S5.SS1.p1.4.m4.3.3.2.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.4.m4.3b"><apply id="S5.SS1.p1.4.m4.3.3.cmml" xref="S5.SS1.p1.4.m4.3.3"><eq id="S5.SS1.p1.4.m4.3.3.3.cmml" xref="S5.SS1.p1.4.m4.3.3.3"></eq><apply id="S5.SS1.p1.4.m4.2.2.1.cmml" xref="S5.SS1.p1.4.m4.2.2.1"><times id="S5.SS1.p1.4.m4.2.2.1.2.cmml" xref="S5.SS1.p1.4.m4.2.2.1.2"></times><ci id="S5.SS1.p1.4.m4.2.2.1.3.cmml" xref="S5.SS1.p1.4.m4.2.2.1.3">Δ</ci><interval closure="open" id="S5.SS1.p1.4.m4.2.2.1.1.2.cmml" xref="S5.SS1.p1.4.m4.2.2.1.1.1"><apply id="S5.SS1.p1.4.m4.2.2.1.1.1.1.cmml" xref="S5.SS1.p1.4.m4.2.2.1.1.1.1"><times id="S5.SS1.p1.4.m4.2.2.1.1.1.1.1.cmml" xref="S5.SS1.p1.4.m4.2.2.1.1.1.1.1"></times><ci id="S5.SS1.p1.4.m4.2.2.1.1.1.1.2.cmml" xref="S5.SS1.p1.4.m4.2.2.1.1.1.1.2">ℂ</ci><ci id="S5.SS1.p1.4.m4.2.2.1.1.1.1.3.cmml" xref="S5.SS1.p1.4.m4.2.2.1.1.1.1.3">𝔽</ci></apply><ci id="S5.SS1.p1.4.m4.1.1.cmml" xref="S5.SS1.p1.4.m4.1.1">𝔽</ci></interval></apply><apply id="S5.SS1.p1.4.m4.3.3.2.cmml" xref="S5.SS1.p1.4.m4.3.3.2"><times id="S5.SS1.p1.4.m4.3.3.2.2.cmml" xref="S5.SS1.p1.4.m4.3.3.2.2"></times><ci id="S5.SS1.p1.4.m4.3.3.2.3.cmml" xref="S5.SS1.p1.4.m4.3.3.2.3">𝙿𝙿𝙻</ci><apply id="S5.SS1.p1.4.m4.3.3.2.1.1.1.cmml" xref="S5.SS1.p1.4.m4.3.3.2.1.1"><times id="S5.SS1.p1.4.m4.3.3.2.1.1.1.1.cmml" xref="S5.SS1.p1.4.m4.3.3.2.1.1.1.1"></times><ci id="S5.SS1.p1.4.m4.3.3.2.1.1.1.2.cmml" xref="S5.SS1.p1.4.m4.3.3.2.1.1.1.2">ℂ</ci><ci id="S5.SS1.p1.4.m4.3.3.2.1.1.1.3.cmml" xref="S5.SS1.p1.4.m4.3.3.2.1.1.1.3">𝔽</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.4.m4.3c">\Delta(\mathbb{CF},\mathbb{F})=\mathtt{PPL}(\mathbb{CF})</annotation></semantics></math> - <math id="S5.SS1.p1.5.m5.1" class="ltx_Math" alttext="\mathtt{PPL}(\mathbb{F})" display="inline"><semantics id="S5.SS1.p1.5.m5.1a"><mrow id="S5.SS1.p1.5.m5.1.2" xref="S5.SS1.p1.5.m5.1.2.cmml"><mi id="S5.SS1.p1.5.m5.1.2.2" xref="S5.SS1.p1.5.m5.1.2.2.cmml">𝙿𝙿𝙻</mi><mo lspace="0em" rspace="0em" id="S5.SS1.p1.5.m5.1.2.1" xref="S5.SS1.p1.5.m5.1.2.1.cmml">​</mo><mrow id="S5.SS1.p1.5.m5.1.2.3.2" xref="S5.SS1.p1.5.m5.1.2.cmml"><mo stretchy="false" id="S5.SS1.p1.5.m5.1.2.3.2.1" xref="S5.SS1.p1.5.m5.1.2.cmml">(</mo><mi id="S5.SS1.p1.5.m5.1.1" xref="S5.SS1.p1.5.m5.1.1.cmml">𝔽</mi><mo stretchy="false" id="S5.SS1.p1.5.m5.1.2.3.2.2" xref="S5.SS1.p1.5.m5.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.5.m5.1b"><apply id="S5.SS1.p1.5.m5.1.2.cmml" xref="S5.SS1.p1.5.m5.1.2"><times id="S5.SS1.p1.5.m5.1.2.1.cmml" xref="S5.SS1.p1.5.m5.1.2.1"></times><ci id="S5.SS1.p1.5.m5.1.2.2.cmml" xref="S5.SS1.p1.5.m5.1.2.2">𝙿𝙿𝙻</ci><ci id="S5.SS1.p1.5.m5.1.1.cmml" xref="S5.SS1.p1.5.m5.1.1">𝔽</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.5.m5.1c">\mathtt{PPL}(\mathbb{F})</annotation></semantics></math>), an effective editing over the facts <math id="S5.SS1.p1.6.m6.1" class="ltx_Math" alttext="\mathbb{F}" display="inline"><semantics id="S5.SS1.p1.6.m6.1a"><mi id="S5.SS1.p1.6.m6.1.1" xref="S5.SS1.p1.6.m6.1.1.cmml">𝔽</mi><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.6.m6.1b"><ci id="S5.SS1.p1.6.m6.1.1.cmml" xref="S5.SS1.p1.6.m6.1.1">𝔽</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.6.m6.1c">\mathbb{F}</annotation></semantics></math> to its counterfactual counterpart <math id="S5.SS1.p1.7.m7.1" class="ltx_Math" alttext="\mathbb{CF}" display="inline"><semantics id="S5.SS1.p1.7.m7.1a"><mrow id="S5.SS1.p1.7.m7.1.1" xref="S5.SS1.p1.7.m7.1.1.cmml"><mi id="S5.SS1.p1.7.m7.1.1.2" xref="S5.SS1.p1.7.m7.1.1.2.cmml">ℂ</mi><mo lspace="0em" rspace="0em" id="S5.SS1.p1.7.m7.1.1.1" xref="S5.SS1.p1.7.m7.1.1.1.cmml">​</mo><mi id="S5.SS1.p1.7.m7.1.1.3" xref="S5.SS1.p1.7.m7.1.1.3.cmml">𝔽</mi></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.7.m7.1b"><apply id="S5.SS1.p1.7.m7.1.1.cmml" xref="S5.SS1.p1.7.m7.1.1"><times id="S5.SS1.p1.7.m7.1.1.1.cmml" xref="S5.SS1.p1.7.m7.1.1.1"></times><ci id="S5.SS1.p1.7.m7.1.1.2.cmml" xref="S5.SS1.p1.7.m7.1.1.2">ℂ</ci><ci id="S5.SS1.p1.7.m7.1.1.3.cmml" xref="S5.SS1.p1.7.m7.1.1.3">𝔽</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.7.m7.1c">\mathbb{CF}</annotation></semantics></math> can be evaluated by the indicator function: <math id="S5.SS1.p1.8.m8.5" class="ltx_Math" alttext="\mathds{1}[\min(\Delta_{pre}(\mathbb{CF},\mathbb{F}),0)&gt;\Delta_{post}(\mathbb{CF},\mathbb{F})]" display="inline"><semantics id="S5.SS1.p1.8.m8.5a"><mrow id="S5.SS1.p1.8.m8.5.5" xref="S5.SS1.p1.8.m8.5.5.cmml"><mn id="S5.SS1.p1.8.m8.5.5.3" xref="S5.SS1.p1.8.m8.5.5.3.cmml">𝟙</mn><mo lspace="0em" rspace="0em" id="S5.SS1.p1.8.m8.5.5.2" xref="S5.SS1.p1.8.m8.5.5.2.cmml">​</mo><mrow id="S5.SS1.p1.8.m8.5.5.1.1" xref="S5.SS1.p1.8.m8.5.5.1.2.cmml"><mo stretchy="false" id="S5.SS1.p1.8.m8.5.5.1.1.2" xref="S5.SS1.p1.8.m8.5.5.1.2.1.cmml">[</mo><mrow id="S5.SS1.p1.8.m8.5.5.1.1.1" xref="S5.SS1.p1.8.m8.5.5.1.1.1.cmml"><mrow id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.2.cmml"><mi id="S5.SS1.p1.8.m8.2.2" xref="S5.SS1.p1.8.m8.2.2.cmml">min</mi><mo id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1a" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.2.cmml">⁡</mo><mrow id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.2.cmml"><mo stretchy="false" id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.2" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.2.cmml">(</mo><mrow id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.cmml"><msub id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.cmml"><mi mathvariant="normal" id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.2" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.2.cmml">Δ</mi><mrow id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.3" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.3.cmml"><mi id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.3.2" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.3.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.3.1" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.3.1.cmml">​</mo><mi id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.3.3" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.3.3.cmml">r</mi><mo lspace="0em" rspace="0em" id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.3.1a" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.3.1.cmml">​</mo><mi id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.3.4" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.3.4.cmml">e</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.2" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.1.1" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.1.1.2" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.1.2.cmml">(</mo><mrow id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.1.1.1" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.1.1.1.2" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.1.1.1.2.cmml">ℂ</mi><mo lspace="0em" rspace="0em" id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.1.1.1.1" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.1.1.1.1.cmml">​</mo><mi id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.1.1.1.3" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.1.1.1.3.cmml">𝔽</mi></mrow><mo id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.1.1.3" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.1.2.cmml">,</mo><mi id="S5.SS1.p1.8.m8.1.1" xref="S5.SS1.p1.8.m8.1.1.cmml">𝔽</mi><mo stretchy="false" id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.1.1.4" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.1.2.cmml">)</mo></mrow></mrow><mo id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.3" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.2.cmml">,</mo><mn id="S5.SS1.p1.8.m8.3.3" xref="S5.SS1.p1.8.m8.3.3.cmml">0</mn><mo stretchy="false" id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.4" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.2.cmml">)</mo></mrow></mrow><mo id="S5.SS1.p1.8.m8.5.5.1.1.1.3" xref="S5.SS1.p1.8.m8.5.5.1.1.1.3.cmml">&gt;</mo><mrow id="S5.SS1.p1.8.m8.5.5.1.1.1.2" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.cmml"><msub id="S5.SS1.p1.8.m8.5.5.1.1.1.2.3" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.cmml"><mi mathvariant="normal" id="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.2" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.2.cmml">Δ</mi><mrow id="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.3" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.3.cmml"><mi id="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.3.2" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.3.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.3.1" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.3.1.cmml">​</mo><mi id="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.3.3" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.3.1a" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.3.1.cmml">​</mo><mi id="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.3.4" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.3.4.cmml">s</mi><mo lspace="0em" rspace="0em" id="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.3.1b" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.3.1.cmml">​</mo><mi id="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.3.5" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.3.5.cmml">t</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S5.SS1.p1.8.m8.5.5.1.1.1.2.2" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.2.cmml">​</mo><mrow id="S5.SS1.p1.8.m8.5.5.1.1.1.2.1.1" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.1.2.cmml"><mo stretchy="false" id="S5.SS1.p1.8.m8.5.5.1.1.1.2.1.1.2" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.1.2.cmml">(</mo><mrow id="S5.SS1.p1.8.m8.5.5.1.1.1.2.1.1.1" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.1.1.1.cmml"><mi id="S5.SS1.p1.8.m8.5.5.1.1.1.2.1.1.1.2" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.1.1.1.2.cmml">ℂ</mi><mo lspace="0em" rspace="0em" id="S5.SS1.p1.8.m8.5.5.1.1.1.2.1.1.1.1" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.1.1.1.1.cmml">​</mo><mi id="S5.SS1.p1.8.m8.5.5.1.1.1.2.1.1.1.3" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.1.1.1.3.cmml">𝔽</mi></mrow><mo id="S5.SS1.p1.8.m8.5.5.1.1.1.2.1.1.3" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.1.2.cmml">,</mo><mi id="S5.SS1.p1.8.m8.4.4" xref="S5.SS1.p1.8.m8.4.4.cmml">𝔽</mi><mo stretchy="false" id="S5.SS1.p1.8.m8.5.5.1.1.1.2.1.1.4" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.1.2.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S5.SS1.p1.8.m8.5.5.1.1.3" xref="S5.SS1.p1.8.m8.5.5.1.2.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.8.m8.5b"><apply id="S5.SS1.p1.8.m8.5.5.cmml" xref="S5.SS1.p1.8.m8.5.5"><times id="S5.SS1.p1.8.m8.5.5.2.cmml" xref="S5.SS1.p1.8.m8.5.5.2"></times><cn type="integer" id="S5.SS1.p1.8.m8.5.5.3.cmml" xref="S5.SS1.p1.8.m8.5.5.3">1</cn><apply id="S5.SS1.p1.8.m8.5.5.1.2.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1"><csymbol cd="latexml" id="S5.SS1.p1.8.m8.5.5.1.2.1.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.2">delimited-[]</csymbol><apply id="S5.SS1.p1.8.m8.5.5.1.1.1.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1"><gt id="S5.SS1.p1.8.m8.5.5.1.1.1.3.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.3"></gt><apply id="S5.SS1.p1.8.m8.5.5.1.1.1.1.2.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1"><min id="S5.SS1.p1.8.m8.2.2.cmml" xref="S5.SS1.p1.8.m8.2.2"></min><apply id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1"><times id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.2.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.2"></times><apply id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.1.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.2.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.2">Δ</ci><apply id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.3.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.3"><times id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.3.1.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.3.1"></times><ci id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.3.2.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.3.2">𝑝</ci><ci id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.3.3.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.3.3">𝑟</ci><ci id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.3.4.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.3.3.4">𝑒</ci></apply></apply><interval closure="open" id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.1.2.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.1.1"><apply id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.1.1.1.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.1.1.1"><times id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.1.1.1.1"></times><ci id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.1.1.1.2">ℂ</ci><ci id="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.1.1.1.1.1.1.1.3">𝔽</ci></apply><ci id="S5.SS1.p1.8.m8.1.1.cmml" xref="S5.SS1.p1.8.m8.1.1">𝔽</ci></interval></apply><cn type="integer" id="S5.SS1.p1.8.m8.3.3.cmml" xref="S5.SS1.p1.8.m8.3.3">0</cn></apply><apply id="S5.SS1.p1.8.m8.5.5.1.1.1.2.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2"><times id="S5.SS1.p1.8.m8.5.5.1.1.1.2.2.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.2"></times><apply id="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.3"><csymbol cd="ambiguous" id="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.1.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.3">subscript</csymbol><ci id="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.2.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.2">Δ</ci><apply id="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.3.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.3"><times id="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.3.1.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.3.1"></times><ci id="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.3.2.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.3.2">𝑝</ci><ci id="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.3.3.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.3.3">𝑜</ci><ci id="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.3.4.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.3.4">𝑠</ci><ci id="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.3.5.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.3.3.5">𝑡</ci></apply></apply><interval closure="open" id="S5.SS1.p1.8.m8.5.5.1.1.1.2.1.2.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.1.1"><apply id="S5.SS1.p1.8.m8.5.5.1.1.1.2.1.1.1.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.1.1.1"><times id="S5.SS1.p1.8.m8.5.5.1.1.1.2.1.1.1.1.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.1.1.1.1"></times><ci id="S5.SS1.p1.8.m8.5.5.1.1.1.2.1.1.1.2.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.1.1.1.2">ℂ</ci><ci id="S5.SS1.p1.8.m8.5.5.1.1.1.2.1.1.1.3.cmml" xref="S5.SS1.p1.8.m8.5.5.1.1.1.2.1.1.1.3">𝔽</ci></apply><ci id="S5.SS1.p1.8.m8.4.4.cmml" xref="S5.SS1.p1.8.m8.4.4">𝔽</ci></interval></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.8.m8.5c">\mathds{1}[\min(\Delta_{pre}(\mathbb{CF},\mathbb{F}),0)&gt;\Delta_{post}(\mathbb{CF},\mathbb{F})]</annotation></semantics></math>. This definition of successful editing stipulates that the perplexity of counterfactual sentences must be lower than that of factual sentences. Furthermore, in cases where the perplexity of counterfactual sentences is already lower before editing, it necessitates an even greater disparity between the two.</p>
</div>
<div id="S5.SS1.p2" class="ltx_para">
<p id="S5.SS1.p2.2" class="ltx_p">Fact-wise editing performed by QLoRA demonstrates a high degree of effectiveness, in contrast to MEMIT. In general, <math id="S5.SS1.p2.1.m1.1" class="ltx_Math" alttext="\mathtt{PPL}(\mathbb{CF})" display="inline"><semantics id="S5.SS1.p2.1.m1.1a"><mrow id="S5.SS1.p2.1.m1.1.1" xref="S5.SS1.p2.1.m1.1.1.cmml"><mi id="S5.SS1.p2.1.m1.1.1.3" xref="S5.SS1.p2.1.m1.1.1.3.cmml">𝙿𝙿𝙻</mi><mo lspace="0em" rspace="0em" id="S5.SS1.p2.1.m1.1.1.2" xref="S5.SS1.p2.1.m1.1.1.2.cmml">​</mo><mrow id="S5.SS1.p2.1.m1.1.1.1.1" xref="S5.SS1.p2.1.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S5.SS1.p2.1.m1.1.1.1.1.2" xref="S5.SS1.p2.1.m1.1.1.1.1.1.cmml">(</mo><mrow id="S5.SS1.p2.1.m1.1.1.1.1.1" xref="S5.SS1.p2.1.m1.1.1.1.1.1.cmml"><mi id="S5.SS1.p2.1.m1.1.1.1.1.1.2" xref="S5.SS1.p2.1.m1.1.1.1.1.1.2.cmml">ℂ</mi><mo lspace="0em" rspace="0em" id="S5.SS1.p2.1.m1.1.1.1.1.1.1" xref="S5.SS1.p2.1.m1.1.1.1.1.1.1.cmml">​</mo><mi id="S5.SS1.p2.1.m1.1.1.1.1.1.3" xref="S5.SS1.p2.1.m1.1.1.1.1.1.3.cmml">𝔽</mi></mrow><mo stretchy="false" id="S5.SS1.p2.1.m1.1.1.1.1.3" xref="S5.SS1.p2.1.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.p2.1.m1.1b"><apply id="S5.SS1.p2.1.m1.1.1.cmml" xref="S5.SS1.p2.1.m1.1.1"><times id="S5.SS1.p2.1.m1.1.1.2.cmml" xref="S5.SS1.p2.1.m1.1.1.2"></times><ci id="S5.SS1.p2.1.m1.1.1.3.cmml" xref="S5.SS1.p2.1.m1.1.1.3">𝙿𝙿𝙻</ci><apply id="S5.SS1.p2.1.m1.1.1.1.1.1.cmml" xref="S5.SS1.p2.1.m1.1.1.1.1"><times id="S5.SS1.p2.1.m1.1.1.1.1.1.1.cmml" xref="S5.SS1.p2.1.m1.1.1.1.1.1.1"></times><ci id="S5.SS1.p2.1.m1.1.1.1.1.1.2.cmml" xref="S5.SS1.p2.1.m1.1.1.1.1.1.2">ℂ</ci><ci id="S5.SS1.p2.1.m1.1.1.1.1.1.3.cmml" xref="S5.SS1.p2.1.m1.1.1.1.1.1.3">𝔽</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p2.1.m1.1c">\mathtt{PPL}(\mathbb{CF})</annotation></semantics></math> is lower than <math id="S5.SS1.p2.2.m2.1" class="ltx_Math" alttext="\mathtt{PPL}(\mathbb{F})" display="inline"><semantics id="S5.SS1.p2.2.m2.1a"><mrow id="S5.SS1.p2.2.m2.1.2" xref="S5.SS1.p2.2.m2.1.2.cmml"><mi id="S5.SS1.p2.2.m2.1.2.2" xref="S5.SS1.p2.2.m2.1.2.2.cmml">𝙿𝙿𝙻</mi><mo lspace="0em" rspace="0em" id="S5.SS1.p2.2.m2.1.2.1" xref="S5.SS1.p2.2.m2.1.2.1.cmml">​</mo><mrow id="S5.SS1.p2.2.m2.1.2.3.2" xref="S5.SS1.p2.2.m2.1.2.cmml"><mo stretchy="false" id="S5.SS1.p2.2.m2.1.2.3.2.1" xref="S5.SS1.p2.2.m2.1.2.cmml">(</mo><mi id="S5.SS1.p2.2.m2.1.1" xref="S5.SS1.p2.2.m2.1.1.cmml">𝔽</mi><mo stretchy="false" id="S5.SS1.p2.2.m2.1.2.3.2.2" xref="S5.SS1.p2.2.m2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.p2.2.m2.1b"><apply id="S5.SS1.p2.2.m2.1.2.cmml" xref="S5.SS1.p2.2.m2.1.2"><times id="S5.SS1.p2.2.m2.1.2.1.cmml" xref="S5.SS1.p2.2.m2.1.2.1"></times><ci id="S5.SS1.p2.2.m2.1.2.2.cmml" xref="S5.SS1.p2.2.m2.1.2.2">𝙿𝙿𝙻</ci><ci id="S5.SS1.p2.2.m2.1.1.cmml" xref="S5.SS1.p2.2.m2.1.1">𝔽</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p2.2.m2.1c">\mathtt{PPL}(\mathbb{F})</annotation></semantics></math> post-editing but MEMIT has the adverse effect of increasing the overall perplexity within the model. Figure <a href="#S4.F4" title="Figure 4 ‣ Knowledge Editing ‣ 4.3 Experiment Setting ‣ 4 Experiment ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> demonstrates the fact-wise editing effectiveness in 7b model using QLoRA and MEMIT. Detailed results are presented in Table <a href="#A1.T6" title="Table 6 ‣ A.8 Fact-wise Perplexity ‣ Appendix A Appendix ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> in Appendix <a href="#A1.SS8" title="A.8 Fact-wise Perplexity ‣ Appendix A Appendix ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.8</span></a>.</p>
</div>
</section>
<section id="S5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.2 </span>Fact Recall</h3>

<figure id="S5.F5" class="ltx_figure"><svg id="S5.F5.pic1" class="ltx_picture" height="426.25" overflow="visible" version="1.1" width="634.54"><g transform="translate(0,426.25) matrix(1 0 0 -1 0 0) translate(30.26,0) translate(0,235.14)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g transform="matrix(1.0 0.0 0.0 1.0 -30.26 -37.56)"><g class="ltx_nestedsvg" transform="matrix(1 0 0 1 0 0) translate(52.96,0) translate(0,48.78)"><g stroke-width="0.2pt" fill="#808080" stroke="#808080" color="#808080"><path d="M 0 -17.12 L 0 -11.22 M 45.4 -17.12 L 45.4 -11.22 M 90.79 -17.12 L 90.79 -11.22 M 136.19 -17.12 L 136.19 -11.22 M 181.59 -17.12 L 181.59 -11.22 M 226.98 -17.12 L 226.98 -11.22 M 0 129.27 L 0 123.37 M 45.4 129.27 L 45.4 123.37 M 90.79 129.27 L 90.79 123.37 M 136.19 129.27 L 136.19 123.37 M 181.59 129.27 L 181.59 123.37 M 226.98 129.27 L 226.98 123.37" style="fill:none"></path></g><g stroke-width="0.2pt" fill="#808080" stroke="#808080" color="#808080"><path d="M -22.7 0 L -16.79 0 M -22.7 56.08 L -16.79 56.08 M -22.7 112.15 L -16.79 112.15 M 249.68 0 L 243.78 0 M 249.68 56.08 L 243.78 56.08 M 249.68 112.15 L 243.78 112.15" style="fill:none"></path></g><g stroke="#000000" fill="#000000" stroke-width="0.4pt"><path d="M -22.7 -11.22 L -22.7 123.37 L 249.68 123.37 L 249.68 -11.22 L -22.7 -11.22 Z" style="fill:none"></path><g transform="matrix(0.96593 0.25882 -0.25882 0.96593 -28.04 -39.68)" fill="#000000" stroke="#000000"><foreignObject width="65.46" height="12.3" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.13.13.13.4.4.4.1.1" class="ltx_text">superlative</span></foreignObject></g><g transform="matrix(0.96593 0.25882 -0.25882 0.96593 13.45 -40.53)" fill="#000000" stroke="#000000"><foreignObject width="73.84" height="11.93" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.14.14.14.5.5.5.1.1" class="ltx_text">comparative</span></foreignObject></g><g transform="matrix(0.96593 0.25882 -0.25882 0.96593 69.44 -37.69)" fill="#000000" stroke="#000000"><foreignObject width="52.27" height="11.93" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.15.15.15.6.6.6.1.1" class="ltx_text">counting</span></foreignObject></g><g transform="matrix(0.96593 0.25882 -0.25882 0.96593 119.8 -36.36)" fill="#000000" stroke="#000000"><foreignObject width="41.63" height="11.93" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.16.16.16.7.7.7.1.1" class="ltx_text">sorting</span></foreignObject></g><g transform="matrix(0.96593 0.25882 -0.25882 0.96593 151.49 -40.04)" fill="#000000" stroke="#000000"><foreignObject width="69.99" height="11.93" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.17.17.17.8.8.8.1.1" class="ltx_text">aggregation</span></foreignObject></g><g transform="matrix(0.96593 0.25882 -0.25882 0.96593 197.99 -41.32)" fill="#000000" stroke="#000000"><foreignObject width="68.53" height="9.61" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.18.18.18.9.9.9.1.1" class="ltx_text">subtraction</span></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 -34.51 -4.46)" fill="#000000" stroke="#000000"><foreignObject width="6.92" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S5.F5.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="0" display="inline"><semantics id="S5.F5.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S5.F5.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S5.F5.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="S5.F5.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="integer" id="S5.F5.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.F5.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1">0</cn></annotation-xml></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 -41.43 51.62)" fill="#000000" stroke="#000000"><foreignObject width="13.84" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S5.F5.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="50" display="inline"><semantics id="S5.F5.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S5.F5.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S5.F5.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">50</mn><annotation-xml encoding="MathML-Content" id="S5.F5.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="integer" id="S5.F5.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.F5.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1">50</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.F5.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1c">50</annotation></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 -48.34 107.69)" fill="#000000" stroke="#000000"><foreignObject width="20.76" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S5.F5.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="100" display="inline"><semantics id="S5.F5.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S5.F5.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S5.F5.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">100</mn><annotation-xml encoding="MathML-Content" id="S5.F5.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="integer" id="S5.F5.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.F5.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1">100</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.F5.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1c">100</annotation></semantics></math></foreignObject></g><clipPath id="pgfcp1"><path d="M -22.7 -11.22 L 249.68 -11.22 L 249.68 123.37 L -22.7 123.37 Z"></path></clipPath><g clip-path="url(#pgfcp1)"><g stroke="#0000FF" fill="#B3B3FF" color="#0000FF"><path d="M -13.15 0 h 6.92 v 59.2 h -6.92 Z M 32.25 0 h 6.92 v 95.81 h -6.92 Z M 77.65 0 h 6.92 v 111.19 h -6.92 Z M 123.04 0 h 6.92 v 88.26 h -6.92 Z M 168.44 0 h 6.92 v 70.22 h -6.92 Z M 213.84 0 h 6.92 v 81.07 h -6.92 Z"></path></g><g></g><g stroke="#FF0000" fill="#FFB3B3" color="#FF0000"><path d="M -3.46 0 h 6.92 v 60.43 h -6.92 Z M 41.94 0 h 6.92 v 88.78 h -6.92 Z M 87.33 0 h 6.92 v 109.59 h -6.92 Z M 132.73 0 h 6.92 v 84.69 h -6.92 Z M 178.13 0 h 6.92 v 96.9 h -6.92 Z M 223.52 0 h 6.92 v 99.6 h -6.92 Z"></path></g><g></g><g stroke="#734D26" fill="#ECD9C6" color="#734D26"><path d="M 6.23 0 h 6.92 v 44.43 h -6.92 Z M 51.62 0 h 6.92 v 44.95 h -6.92 Z M 97.02 0 h 6.92 v 108.74 h -6.92 Z M 142.42 0 h 6.92 v 64.57 h -6.92 Z M 187.81 0 h 6.92 v 88.75 h -6.92 Z M 233.21 0 h 6.92 v 97.59 h -6.92 Z"></path></g><g></g></g><g transform="matrix(1.0 0.0 0.0 1.0 62.97 140.02)" fill="#000000" stroke="#000000"><foreignObject width="101.05" height="13.84" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.19.19.19.10.10.10.1.1" class="ltx_text">Relatedness (7b)</span></foreignObject></g><g fill="#FFFFFF" stroke="#000000"><path d="M 143.42 156.86 h 212.52 v 22.75 h -212.52 Z"></path></g><g fill="#FFFFFF" stroke="#000000" transform="matrix(1.0 0.0 0.0 1.0 147.57 159.63)"><g class="ltx_tikzmatrix" transform="matrix(1 0 0 -1 0 8.61)"><g class="ltx_tikzmatrix_row" transform="matrix(1 0 0 1 0 8.61)"><g class="ltx_tikzmatrix_col ltx_nopad_l ltx_nopad_r" transform="matrix(1 0 0 -1 0 0) translate(2.35,0)" fill="#B3B3FF" stroke="#0000FF" color="#0000FF"><path d="M -2.08 -2.77 h 4.15 v 11.07 h -4.15 Z M 6.23 -2.77 h 4.15 v 8.3 h -4.15 Z"></path></g><g class="ltx_tikzmatrix_col ltx_nopad_l ltx_nopad_r" transform="matrix(1 0 0 -1 13.01 0) translate(28.73,0) matrix(1.0 0.0 0.0 1.0 -25.96 -3.77)" fill="#000000" stroke="#000000"><foreignObject width="51.93" height="9.61" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.20.20.20.11.11.11.1.1.1.1.1" class="ltx_text">Pre-Edit</span></foreignObject></g><g class="ltx_tikzmatrix_col ltx_nopad_l ltx_nopad_r" transform="matrix(1 0 0 -1 70.47 0) translate(2.35,0)" fill="#FFB3B3" stroke="#FF0000" color="#FF0000"><path d="M -2.08 -2.77 h 4.15 v 11.07 h -4.15 Z M 6.23 -2.77 h 4.15 v 8.3 h -4.15 Z"></path></g><g class="ltx_tikzmatrix_col ltx_nopad_l ltx_nopad_r" transform="matrix(1 0 0 -1 83.48 0) translate(26.21,0) matrix(1.0 0.0 0.0 1.0 -23.45 -3.69)" fill="#000000" stroke="#000000"><foreignObject width="46.89" height="12.15" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.21.21.21.12.12.12.2.2.2.1.1" class="ltx_text">QLoRA</span></foreignObject></g><g class="ltx_tikzmatrix_col ltx_nopad_l ltx_nopad_r" transform="matrix(1 0 0 -1 135.9 0) translate(2.35,0)" fill="#ECD9C6" stroke="#734D26" color="#734D26"><path d="M -2.08 -2.77 h 4.15 v 11.07 h -4.15 Z M 6.23 -2.77 h 4.15 v 8.3 h -4.15 Z"></path></g><g class="ltx_tikzmatrix_col ltx_nopad_l ltx_nopad_r" transform="matrix(1 0 0 -1 148.91 0) translate(27.65,0) matrix(1.0 0.0 0.0 1.0 -24.89 -3.69)" fill="#000000" stroke="#000000"><foreignObject width="49.77" height="9.46" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.22.22.22.13.13.13.3.3.3.1.1" class="ltx_text">MEMIT</span></foreignObject></g></g></g></g></g></g></g><g transform="matrix(1.0 0.0 0.0 1.0 281.49 -37.56)"><g class="ltx_nestedsvg" transform="matrix(1 0 0 1 0 0) translate(52.96,0) translate(0,48.78)"><g stroke-width="0.2pt" fill="#808080" stroke="#808080" color="#808080"><path d="M 0 -17.12 L 0 -11.22 M 45.4 -17.12 L 45.4 -11.22 M 90.79 -17.12 L 90.79 -11.22 M 136.19 -17.12 L 136.19 -11.22 M 181.59 -17.12 L 181.59 -11.22 M 226.98 -17.12 L 226.98 -11.22 M 0 129.27 L 0 123.37 M 45.4 129.27 L 45.4 123.37 M 90.79 129.27 L 90.79 123.37 M 136.19 129.27 L 136.19 123.37 M 181.59 129.27 L 181.59 123.37 M 226.98 129.27 L 226.98 123.37" style="fill:none"></path></g><g stroke-width="0.2pt" fill="#808080" stroke="#808080" color="#808080"><path d="M -22.7 0 L -16.79 0 M -22.7 56.08 L -16.79 56.08 M -22.7 112.15 L -16.79 112.15 M 249.68 0 L 243.78 0 M 249.68 56.08 L 243.78 56.08 M 249.68 112.15 L 243.78 112.15" style="fill:none"></path></g><g stroke="#000000" fill="#000000" stroke-width="0.4pt"><path d="M -22.7 -11.22 L -22.7 123.37 L 249.68 123.37 L 249.68 -11.22 L -22.7 -11.22 Z" style="fill:none"></path><g transform="matrix(0.96593 0.25882 -0.25882 0.96593 -28.04 -39.68)" fill="#000000" stroke="#000000"><foreignObject width="65.46" height="12.3" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.23.23.23.4.4.4.1.1" class="ltx_text">superlative</span></foreignObject></g><g transform="matrix(0.96593 0.25882 -0.25882 0.96593 13.45 -40.53)" fill="#000000" stroke="#000000"><foreignObject width="73.84" height="11.93" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.24.24.24.5.5.5.1.1" class="ltx_text">comparative</span></foreignObject></g><g transform="matrix(0.96593 0.25882 -0.25882 0.96593 69.44 -37.69)" fill="#000000" stroke="#000000"><foreignObject width="52.27" height="11.93" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.25.25.25.6.6.6.1.1" class="ltx_text">counting</span></foreignObject></g><g transform="matrix(0.96593 0.25882 -0.25882 0.96593 119.8 -36.36)" fill="#000000" stroke="#000000"><foreignObject width="41.63" height="11.93" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.26.26.26.7.7.7.1.1" class="ltx_text">sorting</span></foreignObject></g><g transform="matrix(0.96593 0.25882 -0.25882 0.96593 151.49 -40.04)" fill="#000000" stroke="#000000"><foreignObject width="69.99" height="11.93" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.27.27.27.8.8.8.1.1" class="ltx_text">aggregation</span></foreignObject></g><g transform="matrix(0.96593 0.25882 -0.25882 0.96593 197.99 -41.32)" fill="#000000" stroke="#000000"><foreignObject width="68.53" height="9.61" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.28.28.28.9.9.9.1.1" class="ltx_text">subtraction</span></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 -34.51 -4.46)" fill="#000000" stroke="#000000"><foreignObject width="6.92" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S5.F5.pic1.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="0" display="inline"><semantics id="S5.F5.pic1.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S5.F5.pic1.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S5.F5.pic1.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="S5.F5.pic1.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="integer" id="S5.F5.pic1.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.F5.pic1.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1">0</cn></annotation-xml></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 -41.43 51.62)" fill="#000000" stroke="#000000"><foreignObject width="13.84" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S5.F5.pic1.5.5.5.5.5.5.5.5.5.5.5.5.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="50" display="inline"><semantics id="S5.F5.pic1.5.5.5.5.5.5.5.5.5.5.5.5.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S5.F5.pic1.5.5.5.5.5.5.5.5.5.5.5.5.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S5.F5.pic1.5.5.5.5.5.5.5.5.5.5.5.5.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">50</mn><annotation-xml encoding="MathML-Content" id="S5.F5.pic1.5.5.5.5.5.5.5.5.5.5.5.5.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="integer" id="S5.F5.pic1.5.5.5.5.5.5.5.5.5.5.5.5.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.F5.pic1.5.5.5.5.5.5.5.5.5.5.5.5.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1">50</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.F5.pic1.5.5.5.5.5.5.5.5.5.5.5.5.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1c">50</annotation></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 -48.34 107.69)" fill="#000000" stroke="#000000"><foreignObject width="20.76" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S5.F5.pic1.6.6.6.6.6.6.6.6.6.6.6.6.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="100" display="inline"><semantics id="S5.F5.pic1.6.6.6.6.6.6.6.6.6.6.6.6.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S5.F5.pic1.6.6.6.6.6.6.6.6.6.6.6.6.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S5.F5.pic1.6.6.6.6.6.6.6.6.6.6.6.6.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">100</mn><annotation-xml encoding="MathML-Content" id="S5.F5.pic1.6.6.6.6.6.6.6.6.6.6.6.6.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="integer" id="S5.F5.pic1.6.6.6.6.6.6.6.6.6.6.6.6.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.F5.pic1.6.6.6.6.6.6.6.6.6.6.6.6.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1">100</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.F5.pic1.6.6.6.6.6.6.6.6.6.6.6.6.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1c">100</annotation></semantics></math></foreignObject></g><clipPath id="pgfcp2"><path d="M -22.7 -11.22 L 249.68 -11.22 L 249.68 123.37 L -22.7 123.37 Z"></path></clipPath><g clip-path="url(#pgfcp2)"><g stroke="#0000FF" fill="#B3B3FF" color="#0000FF"><path d="M -13.15 0 h 6.92 v 90.37 h -6.92 Z M 32.25 0 h 6.92 v 73.2 h -6.92 Z M 77.65 0 h 6.92 v 49.26 h -6.92 Z M 123.04 0 h 6.92 v 67.85 h -6.92 Z M 168.44 0 h 6.92 v 77.18 h -6.92 Z M 213.84 0 h 6.92 v 79.26 h -6.92 Z"></path></g><g></g><g stroke="#FF0000" fill="#FFB3B3" color="#FF0000"><path d="M -3.46 0 h 6.92 v 32.04 h -6.92 Z M 41.94 0 h 6.92 v 37.45 h -6.92 Z M 87.33 0 h 6.92 v 11.09 h -6.92 Z M 132.73 0 h 6.92 v 52.34 h -6.92 Z M 178.13 0 h 6.92 v 16.41 h -6.92 Z M 223.52 0 h 6.92 v 16.82 h -6.92 Z"></path></g><g></g><g stroke="#734D26" fill="#ECD9C6" color="#734D26"><path d="M 6.23 0 h 6.92 v 12.66 h -6.92 Z M 51.62 0 h 6.92 v 9.13 h -6.92 Z M 97.02 0 h 6.92 v 5.11 h -6.92 Z M 142.42 0 h 6.92 v 59.48 h -6.92 Z M 187.81 0 h 6.92 v 9.76 h -6.92 Z M 233.21 0 h 6.92 v 8.82 h -6.92 Z"></path></g><g></g></g><g transform="matrix(1.0 0.0 0.0 1.0 62.49 140.02)" fill="#000000" stroke="#000000"><foreignObject width="102.01" height="13.84" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.29.29.29.10.10.10.1.1" class="ltx_text">Consistency (7b)</span></foreignObject></g></g></g></g><g transform="matrix(1.0 0.0 0.0 1.0 -30.26 -235.14)"><g class="ltx_nestedsvg" transform="matrix(1 0 0 1 0 0) translate(52.96,0) translate(0,48.78)"><g stroke-width="0.2pt" fill="#808080" stroke="#808080" color="#808080"><path d="M 0 -17.12 L 0 -11.22 M 45.4 -17.12 L 45.4 -11.22 M 90.79 -17.12 L 90.79 -11.22 M 136.19 -17.12 L 136.19 -11.22 M 181.59 -17.12 L 181.59 -11.22 M 226.98 -17.12 L 226.98 -11.22 M 0 129.27 L 0 123.37 M 45.4 129.27 L 45.4 123.37 M 90.79 129.27 L 90.79 123.37 M 136.19 129.27 L 136.19 123.37 M 181.59 129.27 L 181.59 123.37 M 226.98 129.27 L 226.98 123.37" style="fill:none"></path></g><g stroke-width="0.2pt" fill="#808080" stroke="#808080" color="#808080"><path d="M -22.7 0 L -16.79 0 M -22.7 56.08 L -16.79 56.08 M -22.7 112.15 L -16.79 112.15 M 249.68 0 L 243.78 0 M 249.68 56.08 L 243.78 56.08 M 249.68 112.15 L 243.78 112.15" style="fill:none"></path></g><g stroke="#000000" fill="#000000" stroke-width="0.4pt"><path d="M -22.7 -11.22 L -22.7 123.37 L 249.68 123.37 L 249.68 -11.22 L -22.7 -11.22 Z" style="fill:none"></path><g transform="matrix(0.96593 0.25882 -0.25882 0.96593 -28.04 -39.68)" fill="#000000" stroke="#000000"><foreignObject width="65.46" height="12.3" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.30.30.30.4.4.4.1.1" class="ltx_text">superlative</span></foreignObject></g><g transform="matrix(0.96593 0.25882 -0.25882 0.96593 13.45 -40.53)" fill="#000000" stroke="#000000"><foreignObject width="73.84" height="11.93" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.31.31.31.5.5.5.1.1" class="ltx_text">comparative</span></foreignObject></g><g transform="matrix(0.96593 0.25882 -0.25882 0.96593 69.44 -37.69)" fill="#000000" stroke="#000000"><foreignObject width="52.27" height="11.93" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.32.32.32.6.6.6.1.1" class="ltx_text">counting</span></foreignObject></g><g transform="matrix(0.96593 0.25882 -0.25882 0.96593 119.8 -36.36)" fill="#000000" stroke="#000000"><foreignObject width="41.63" height="11.93" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.33.33.33.7.7.7.1.1" class="ltx_text">sorting</span></foreignObject></g><g transform="matrix(0.96593 0.25882 -0.25882 0.96593 151.49 -40.04)" fill="#000000" stroke="#000000"><foreignObject width="69.99" height="11.93" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.34.34.34.8.8.8.1.1" class="ltx_text">aggregation</span></foreignObject></g><g transform="matrix(0.96593 0.25882 -0.25882 0.96593 197.99 -41.32)" fill="#000000" stroke="#000000"><foreignObject width="68.53" height="9.61" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.35.35.35.9.9.9.1.1" class="ltx_text">subtraction</span></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 -34.51 -4.46)" fill="#000000" stroke="#000000"><foreignObject width="6.92" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S5.F5.pic1.7.7.7.7.7.7.7.7.7.7.7.7.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="0" display="inline"><semantics id="S5.F5.pic1.7.7.7.7.7.7.7.7.7.7.7.7.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S5.F5.pic1.7.7.7.7.7.7.7.7.7.7.7.7.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S5.F5.pic1.7.7.7.7.7.7.7.7.7.7.7.7.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="S5.F5.pic1.7.7.7.7.7.7.7.7.7.7.7.7.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="integer" id="S5.F5.pic1.7.7.7.7.7.7.7.7.7.7.7.7.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.F5.pic1.7.7.7.7.7.7.7.7.7.7.7.7.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1">0</cn></annotation-xml></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 -41.43 51.62)" fill="#000000" stroke="#000000"><foreignObject width="13.84" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S5.F5.pic1.8.8.8.8.8.8.8.8.8.8.8.8.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="50" display="inline"><semantics id="S5.F5.pic1.8.8.8.8.8.8.8.8.8.8.8.8.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S5.F5.pic1.8.8.8.8.8.8.8.8.8.8.8.8.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S5.F5.pic1.8.8.8.8.8.8.8.8.8.8.8.8.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">50</mn><annotation-xml encoding="MathML-Content" id="S5.F5.pic1.8.8.8.8.8.8.8.8.8.8.8.8.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="integer" id="S5.F5.pic1.8.8.8.8.8.8.8.8.8.8.8.8.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.F5.pic1.8.8.8.8.8.8.8.8.8.8.8.8.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1">50</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.F5.pic1.8.8.8.8.8.8.8.8.8.8.8.8.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1c">50</annotation></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 -48.34 107.69)" fill="#000000" stroke="#000000"><foreignObject width="20.76" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S5.F5.pic1.9.9.9.9.9.9.9.9.9.9.9.9.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="100" display="inline"><semantics id="S5.F5.pic1.9.9.9.9.9.9.9.9.9.9.9.9.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S5.F5.pic1.9.9.9.9.9.9.9.9.9.9.9.9.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S5.F5.pic1.9.9.9.9.9.9.9.9.9.9.9.9.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">100</mn><annotation-xml encoding="MathML-Content" id="S5.F5.pic1.9.9.9.9.9.9.9.9.9.9.9.9.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="integer" id="S5.F5.pic1.9.9.9.9.9.9.9.9.9.9.9.9.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.F5.pic1.9.9.9.9.9.9.9.9.9.9.9.9.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1">100</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.F5.pic1.9.9.9.9.9.9.9.9.9.9.9.9.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1c">100</annotation></semantics></math></foreignObject></g><clipPath id="pgfcp3"><path d="M -22.7 -11.22 L 249.68 -11.22 L 249.68 123.37 L -22.7 123.37 Z"></path></clipPath><g clip-path="url(#pgfcp3)"><g stroke="#0000FF" fill="#B3B3FF" color="#0000FF"><path d="M -13.15 0 h 6.92 v 60.19 h -6.92 Z M 32.25 0 h 6.92 v 93.69 h -6.92 Z M 77.65 0 h 6.92 v 111.75 h -6.92 Z M 123.04 0 h 6.92 v 87.19 h -6.92 Z M 168.44 0 h 6.92 v 79.3 h -6.92 Z M 213.84 0 h 6.92 v 87 h -6.92 Z"></path></g><g></g><g stroke="#FF0000" fill="#FFB3B3" color="#FF0000"><path d="M -3.46 0 h 6.92 v 61.82 h -6.92 Z M 41.94 0 h 6.92 v 89.43 h -6.92 Z M 87.33 0 h 6.92 v 109.93 h -6.92 Z M 132.73 0 h 6.92 v 87.53 h -6.92 Z M 178.13 0 h 6.92 v 98.3 h -6.92 Z M 223.52 0 h 6.92 v 101.98 h -6.92 Z"></path></g><g></g><g stroke="#734D26" fill="#ECD9C6" color="#734D26"><path d="M 6.23 0 h 6.92 v 45.33 h -6.92 Z M 51.62 0 h 6.92 v 51.21 h -6.92 Z M 97.02 0 h 6.92 v 108.88 h -6.92 Z M 142.42 0 h 6.92 v 66.95 h -6.92 Z M 187.81 0 h 6.92 v 90.67 h -6.92 Z M 233.21 0 h 6.92 v 97.84 h -6.92 Z"></path></g><g></g></g><g transform="matrix(1.0 0.0 0.0 1.0 59.51 140.02)" fill="#000000" stroke="#000000"><foreignObject width="107.97" height="13.84" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.36.36.36.10.10.10.1.1" class="ltx_text">Relatedness (13b)</span></foreignObject></g></g></g></g><g transform="matrix(1.0 0.0 0.0 1.0 281.49 -235.14)"><g class="ltx_nestedsvg" transform="matrix(1 0 0 1 0 0) translate(52.96,0) translate(0,48.78)"><g stroke-width="0.2pt" fill="#808080" stroke="#808080" color="#808080"><path d="M 0 -17.12 L 0 -11.22 M 45.4 -17.12 L 45.4 -11.22 M 90.79 -17.12 L 90.79 -11.22 M 136.19 -17.12 L 136.19 -11.22 M 181.59 -17.12 L 181.59 -11.22 M 226.98 -17.12 L 226.98 -11.22 M 0 129.27 L 0 123.37 M 45.4 129.27 L 45.4 123.37 M 90.79 129.27 L 90.79 123.37 M 136.19 129.27 L 136.19 123.37 M 181.59 129.27 L 181.59 123.37 M 226.98 129.27 L 226.98 123.37" style="fill:none"></path></g><g stroke-width="0.2pt" fill="#808080" stroke="#808080" color="#808080"><path d="M -22.7 0 L -16.79 0 M -22.7 56.08 L -16.79 56.08 M -22.7 112.15 L -16.79 112.15 M 249.68 0 L 243.78 0 M 249.68 56.08 L 243.78 56.08 M 249.68 112.15 L 243.78 112.15" style="fill:none"></path></g><g stroke="#000000" fill="#000000" stroke-width="0.4pt"><path d="M -22.7 -11.22 L -22.7 123.37 L 249.68 123.37 L 249.68 -11.22 L -22.7 -11.22 Z" style="fill:none"></path><g transform="matrix(0.96593 0.25882 -0.25882 0.96593 -28.04 -39.68)" fill="#000000" stroke="#000000"><foreignObject width="65.46" height="12.3" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.37.37.37.4.4.4.1.1" class="ltx_text">superlative</span></foreignObject></g><g transform="matrix(0.96593 0.25882 -0.25882 0.96593 13.45 -40.53)" fill="#000000" stroke="#000000"><foreignObject width="73.84" height="11.93" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.38.38.38.5.5.5.1.1" class="ltx_text">comparative</span></foreignObject></g><g transform="matrix(0.96593 0.25882 -0.25882 0.96593 69.44 -37.69)" fill="#000000" stroke="#000000"><foreignObject width="52.27" height="11.93" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.39.39.39.6.6.6.1.1" class="ltx_text">counting</span></foreignObject></g><g transform="matrix(0.96593 0.25882 -0.25882 0.96593 119.8 -36.36)" fill="#000000" stroke="#000000"><foreignObject width="41.63" height="11.93" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.40.40.40.7.7.7.1.1" class="ltx_text">sorting</span></foreignObject></g><g transform="matrix(0.96593 0.25882 -0.25882 0.96593 151.49 -40.04)" fill="#000000" stroke="#000000"><foreignObject width="69.99" height="11.93" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.41.41.41.8.8.8.1.1" class="ltx_text">aggregation</span></foreignObject></g><g transform="matrix(0.96593 0.25882 -0.25882 0.96593 197.99 -41.32)" fill="#000000" stroke="#000000"><foreignObject width="68.53" height="9.61" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.42.42.42.9.9.9.1.1" class="ltx_text">subtraction</span></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 -34.51 -4.46)" fill="#000000" stroke="#000000"><foreignObject width="6.92" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S5.F5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="0" display="inline"><semantics id="S5.F5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S5.F5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S5.F5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="S5.F5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="integer" id="S5.F5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.F5.pic1.10.10.10.10.10.10.10.10.10.10.10.10.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1">0</cn></annotation-xml></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 -41.43 51.62)" fill="#000000" stroke="#000000"><foreignObject width="13.84" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S5.F5.pic1.11.11.11.11.11.11.11.11.11.11.11.11.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="50" display="inline"><semantics id="S5.F5.pic1.11.11.11.11.11.11.11.11.11.11.11.11.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S5.F5.pic1.11.11.11.11.11.11.11.11.11.11.11.11.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S5.F5.pic1.11.11.11.11.11.11.11.11.11.11.11.11.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">50</mn><annotation-xml encoding="MathML-Content" id="S5.F5.pic1.11.11.11.11.11.11.11.11.11.11.11.11.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="integer" id="S5.F5.pic1.11.11.11.11.11.11.11.11.11.11.11.11.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.F5.pic1.11.11.11.11.11.11.11.11.11.11.11.11.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1">50</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.F5.pic1.11.11.11.11.11.11.11.11.11.11.11.11.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1c">50</annotation></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 -48.34 107.69)" fill="#000000" stroke="#000000"><foreignObject width="20.76" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S5.F5.pic1.12.12.12.12.12.12.12.12.12.12.12.12.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="100" display="inline"><semantics id="S5.F5.pic1.12.12.12.12.12.12.12.12.12.12.12.12.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S5.F5.pic1.12.12.12.12.12.12.12.12.12.12.12.12.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S5.F5.pic1.12.12.12.12.12.12.12.12.12.12.12.12.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">100</mn><annotation-xml encoding="MathML-Content" id="S5.F5.pic1.12.12.12.12.12.12.12.12.12.12.12.12.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="integer" id="S5.F5.pic1.12.12.12.12.12.12.12.12.12.12.12.12.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.F5.pic1.12.12.12.12.12.12.12.12.12.12.12.12.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1">100</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.F5.pic1.12.12.12.12.12.12.12.12.12.12.12.12.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1c">100</annotation></semantics></math></foreignObject></g><clipPath id="pgfcp4"><path d="M -22.7 -11.22 L 249.68 -11.22 L 249.68 123.37 L -22.7 123.37 Z"></path></clipPath><g clip-path="url(#pgfcp4)"><g stroke="#0000FF" fill="#B3B3FF" color="#0000FF"><path d="M -13.15 0 h 6.92 v 91.84 h -6.92 Z M 32.25 0 h 6.92 v 71.65 h -6.92 Z M 77.65 0 h 6.92 v 54.06 h -6.92 Z M 123.04 0 h 6.92 v 71.47 h -6.92 Z M 168.44 0 h 6.92 v 86.26 h -6.92 Z M 213.84 0 h 6.92 v 82.43 h -6.92 Z"></path></g><g></g><g stroke="#FF0000" fill="#FFB3B3" color="#FF0000"><path d="M -3.46 0 h 6.92 v 32.09 h -6.92 Z M 41.94 0 h 6.92 v 31.95 h -6.92 Z M 87.33 0 h 6.92 v 11.7 h -6.92 Z M 132.73 0 h 6.92 v 50.56 h -6.92 Z M 178.13 0 h 6.92 v 13.29 h -6.92 Z M 223.52 0 h 6.92 v 27.44 h -6.92 Z"></path></g><g></g><g stroke="#734D26" fill="#ECD9C6" color="#734D26"><path d="M 6.23 0 h 6.92 v 12.06 h -6.92 Z M 51.62 0 h 6.92 v 17.64 h -6.92 Z M 97.02 0 h 6.92 v 4.5 h -6.92 Z M 142.42 0 h 6.92 v 52.69 h -6.92 Z M 187.81 0 h 6.92 v 6.7 h -6.92 Z M 233.21 0 h 6.92 v 8.51 h -6.92 Z"></path></g><g></g></g><g transform="matrix(1.0 0.0 0.0 1.0 59.03 140.02)" fill="#000000" stroke="#000000"><foreignObject width="108.93" height="13.84" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F5.pic1.43.43.43.10.10.10.1.1" class="ltx_text">Consistency (13b)</span></foreignObject></g></g></g></g></g></svg>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 5: </span>Fact recall pre- and post-editing: measured by the relatedness and consistency of the decomposed atomic facts in CoT generation against the edited counterfactual facts. While both QLoRA and MEMIT maintains a reasonable degree of relatedness (with QLoRA outperforming MEMIT), there is a significant decline in factual consistency of both methods.</figcaption>
</figure>
<div id="S5.SS2.p1" class="ltx_para">
<p id="S5.SS2.p1.1" class="ltx_p">Assuming successful fact-wise editing, we then explore the model’s proficiency in recalling and applying these modifications in reasoning tasks. This involves an in-depth analysis of the model’s ability to retrieve and utilize relevant information correctly. We focus on evaluating the relatedness and consistency of information within the CoT response against the counterfactual facts. The evaluation metrics are defined as:</p>
<ul id="S5.I1" class="ltx_itemize">
<li id="S5.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S5.I1.i1.p1" class="ltx_para">
<p id="S5.I1.i1.p1.1" class="ltx_p"><span id="S5.I1.i1.p1.1.1" class="ltx_text ltx_font_bold">Relatedness</span>: this metric assesses how relevant the facts generated by the model are to the designated fact/counterfactual set.</p>
</div>
</li>
<li id="S5.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S5.I1.i2.p1" class="ltx_para">
<p id="S5.I1.i2.p1.1" class="ltx_p"><span id="S5.I1.i2.p1.1.1" class="ltx_text ltx_font_bold">Consistency</span>: this metric calculates the proportion of the model’s generated facts that align factually with the fact/counterfactual set.</p>
</div>
</li>
</ul>
</div>
<div id="S5.SS2.p2" class="ltx_para">
<p id="S5.SS2.p2.1" class="ltx_p">We leverage Claude with dedicated few-shot demonstrations for automatic evaluation. Detailed prompt can be found in Appendix <a href="#A1.SS9" title="A.9 Fact Recall Evaluation ‣ Appendix A Appendix ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.9</span></a>.</p>
</div>
<div id="S5.SS2.p3" class="ltx_para">
<p id="S5.SS2.p3.1" class="ltx_p">Results are presented in Figure. <a href="#S5.F5" title="Figure 5 ‣ 5.2 Fact Recall ‣ 5 Analysis ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>. Regarding relatedness, different editing methods show different impacts on the model. In the case of QLoRA, the edited model retains the capability to recall information across different schemes. However, MEMIT shows negative impacts in superlative, comparative, and sorting. In some cases, relatedness post-editing surpasses that of pre-editing. This could be attributed to the introduction of new facts into the model that it previously lacked.</p>
</div>
<div id="S5.SS2.p4" class="ltx_para">
<p id="S5.SS2.p4.1" class="ltx_p">However, both QLoRA-edited and MEMIT-edited models show low consistency results, indicating that they are unable to leverage the edited knowledge in actual use. MEMIT-edited models are worse than QLoRA-edited. It’s important to note that this consistency doesn’t correlate well with the fact-wise editing effectiveness. This disparity may stem from a lack of generalization during the editing process. Essentially, the model seems to simply memorize the newly edited fact, lacking the ability to extend this understanding to different manifestations of the same concept.</p>
</div>
</section>
<section id="S5.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.3 </span>Logical Coherence</h3>

<figure id="S5.T5" class="ltx_table">
<table id="S5.T5.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S5.T5.1.1.1" class="ltx_tr">
<th id="S5.T5.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt"><span id="S5.T5.1.1.1.1.1" class="ltx_text ltx_font_bold">Scheme</span></th>
<th id="S5.T5.1.1.1.2" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_tt"><span id="S5.T5.1.1.1.2.1" class="ltx_text ltx_font_bold">Model</span></th>
<th id="S5.T5.1.1.1.3" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_tt"><span id="S5.T5.1.1.1.3.1" class="ltx_text ltx_font_bold">Pre-edit</span></th>
<th id="S5.T5.1.1.1.4" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_tt"><span id="S5.T5.1.1.1.4.1" class="ltx_text ltx_font_bold">QLoRA</span></th>
<th id="S5.T5.1.1.1.5" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_tt"><span id="S5.T5.1.1.1.5.1" class="ltx_text ltx_font_bold">MEMIT</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S5.T5.1.2.1" class="ltx_tr">
<th id="S5.T5.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="2"><span id="S5.T5.1.2.1.1.1" class="ltx_text">superlative</span></th>
<td id="S5.T5.1.2.1.2" class="ltx_td ltx_align_right ltx_border_t">7b</td>
<td id="S5.T5.1.2.1.3" class="ltx_td ltx_align_right ltx_border_t">89.0</td>
<td id="S5.T5.1.2.1.4" class="ltx_td ltx_align_right ltx_border_t">84.2</td>
<td id="S5.T5.1.2.1.5" class="ltx_td ltx_align_right ltx_border_t">8.9</td>
</tr>
<tr id="S5.T5.1.3.2" class="ltx_tr">
<td id="S5.T5.1.3.2.1" class="ltx_td ltx_align_right">13b</td>
<td id="S5.T5.1.3.2.2" class="ltx_td ltx_align_right">90.1</td>
<td id="S5.T5.1.3.2.3" class="ltx_td ltx_align_right">85.0</td>
<td id="S5.T5.1.3.2.4" class="ltx_td ltx_align_right">4.1</td>
</tr>
<tr id="S5.T5.1.4.3" class="ltx_tr">
<th id="S5.T5.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="2"><span id="S5.T5.1.4.3.1.1" class="ltx_text">comparative</span></th>
<td id="S5.T5.1.4.3.2" class="ltx_td ltx_align_right ltx_border_t">7b</td>
<td id="S5.T5.1.4.3.3" class="ltx_td ltx_align_right ltx_border_t">73.8</td>
<td id="S5.T5.1.4.3.4" class="ltx_td ltx_align_right ltx_border_t">52.5</td>
<td id="S5.T5.1.4.3.5" class="ltx_td ltx_align_right ltx_border_t">0.7</td>
</tr>
<tr id="S5.T5.1.5.4" class="ltx_tr">
<td id="S5.T5.1.5.4.1" class="ltx_td ltx_align_right">13b</td>
<td id="S5.T5.1.5.4.2" class="ltx_td ltx_align_right">80.2</td>
<td id="S5.T5.1.5.4.3" class="ltx_td ltx_align_right">74.1</td>
<td id="S5.T5.1.5.4.4" class="ltx_td ltx_align_right">4.8</td>
</tr>
<tr id="S5.T5.1.6.5" class="ltx_tr">
<th id="S5.T5.1.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="2"><span id="S5.T5.1.6.5.1.1" class="ltx_text">counting</span></th>
<td id="S5.T5.1.6.5.2" class="ltx_td ltx_align_right ltx_border_t">7b</td>
<td id="S5.T5.1.6.5.3" class="ltx_td ltx_align_right ltx_border_t">91.1</td>
<td id="S5.T5.1.6.5.4" class="ltx_td ltx_align_right ltx_border_t">92.4</td>
<td id="S5.T5.1.6.5.5" class="ltx_td ltx_align_right ltx_border_t">24.0</td>
</tr>
<tr id="S5.T5.1.7.6" class="ltx_tr">
<td id="S5.T5.1.7.6.1" class="ltx_td ltx_align_right">13b</td>
<td id="S5.T5.1.7.6.2" class="ltx_td ltx_align_right">98.0</td>
<td id="S5.T5.1.7.6.3" class="ltx_td ltx_align_right">94.1</td>
<td id="S5.T5.1.7.6.4" class="ltx_td ltx_align_right">32.4</td>
</tr>
<tr id="S5.T5.1.8.7" class="ltx_tr">
<th id="S5.T5.1.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="2"><span id="S5.T5.1.8.7.1.1" class="ltx_text">sorting</span></th>
<td id="S5.T5.1.8.7.2" class="ltx_td ltx_align_right ltx_border_t">7b</td>
<td id="S5.T5.1.8.7.3" class="ltx_td ltx_align_right ltx_border_t">90.5</td>
<td id="S5.T5.1.8.7.4" class="ltx_td ltx_align_right ltx_border_t">85.4</td>
<td id="S5.T5.1.8.7.5" class="ltx_td ltx_align_right ltx_border_t">43.1</td>
</tr>
<tr id="S5.T5.1.9.8" class="ltx_tr">
<td id="S5.T5.1.9.8.1" class="ltx_td ltx_align_right">13b</td>
<td id="S5.T5.1.9.8.2" class="ltx_td ltx_align_right">90.4</td>
<td id="S5.T5.1.9.8.3" class="ltx_td ltx_align_right">87.9</td>
<td id="S5.T5.1.9.8.4" class="ltx_td ltx_align_right">44.3</td>
</tr>
<tr id="S5.T5.1.10.9" class="ltx_tr">
<th id="S5.T5.1.10.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="2"><span id="S5.T5.1.10.9.1.1" class="ltx_text">aggregation</span></th>
<td id="S5.T5.1.10.9.2" class="ltx_td ltx_align_right ltx_border_t">7b</td>
<td id="S5.T5.1.10.9.3" class="ltx_td ltx_align_right ltx_border_t">92.5</td>
<td id="S5.T5.1.10.9.4" class="ltx_td ltx_align_right ltx_border_t">89.0</td>
<td id="S5.T5.1.10.9.5" class="ltx_td ltx_align_right ltx_border_t">3.6</td>
</tr>
<tr id="S5.T5.1.11.10" class="ltx_tr">
<td id="S5.T5.1.11.10.1" class="ltx_td ltx_align_right">13b</td>
<td id="S5.T5.1.11.10.2" class="ltx_td ltx_align_right">88.2</td>
<td id="S5.T5.1.11.10.3" class="ltx_td ltx_align_right">87.2</td>
<td id="S5.T5.1.11.10.4" class="ltx_td ltx_align_right">5.2</td>
</tr>
<tr id="S5.T5.1.12.11" class="ltx_tr">
<th id="S5.T5.1.12.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="2"><span id="S5.T5.1.12.11.1.1" class="ltx_text">subtraction</span></th>
<td id="S5.T5.1.12.11.2" class="ltx_td ltx_align_right ltx_border_t">7b</td>
<td id="S5.T5.1.12.11.3" class="ltx_td ltx_align_right ltx_border_t">87.2</td>
<td id="S5.T5.1.12.11.4" class="ltx_td ltx_align_right ltx_border_t">84.8</td>
<td id="S5.T5.1.12.11.5" class="ltx_td ltx_align_right ltx_border_t">10.0</td>
</tr>
<tr id="S5.T5.1.13.12" class="ltx_tr">
<td id="S5.T5.1.13.12.1" class="ltx_td ltx_align_right">13b</td>
<td id="S5.T5.1.13.12.2" class="ltx_td ltx_align_right">91.4</td>
<td id="S5.T5.1.13.12.3" class="ltx_td ltx_align_right">92.6</td>
<td id="S5.T5.1.13.12.4" class="ltx_td ltx_align_right">1.3</td>
</tr>
<tr id="S5.T5.1.14.13" class="ltx_tr">
<th id="S5.T5.1.14.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_t" rowspan="2"><span id="S5.T5.1.14.13.1.1" class="ltx_text ltx_font_italic">Average</span></th>
<td id="S5.T5.1.14.13.2" class="ltx_td ltx_align_right ltx_border_t">7b</td>
<td id="S5.T5.1.14.13.3" class="ltx_td ltx_align_right ltx_border_t">87.4</td>
<td id="S5.T5.1.14.13.4" class="ltx_td ltx_align_right ltx_border_t">81.4</td>
<td id="S5.T5.1.14.13.5" class="ltx_td ltx_align_right ltx_border_t">15.1</td>
</tr>
<tr id="S5.T5.1.15.14" class="ltx_tr">
<td id="S5.T5.1.15.14.1" class="ltx_td ltx_align_right ltx_border_bb">13b</td>
<td id="S5.T5.1.15.14.2" class="ltx_td ltx_align_right ltx_border_bb">89.7</td>
<td id="S5.T5.1.15.14.3" class="ltx_td ltx_align_right ltx_border_bb">86.8</td>
<td id="S5.T5.1.15.14.4" class="ltx_td ltx_align_right ltx_border_bb">15.4</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 5: </span>Coherence of post-editing chain-of-thought generations: percentage of coherent CoT responses among all examples. Coherence is determined by whether the final answer is logically supported by the recalled facts in CoT.</figcaption>
</figure>
<div id="S5.SS3.p1" class="ltx_para">
<p id="S5.SS3.p1.1" class="ltx_p">We investigate whether the logical reasoning capacity of the model is negatively impacted. This is gauged by the coherence of the generated CoT response, specifically evaluating whether the inferred evidence and thought process adequately support the final answer.</p>
</div>
<div id="S5.SS3.p2" class="ltx_para">
<p id="S5.SS3.p2.1" class="ltx_p">Table <a href="#S5.T5" title="Table 5 ‣ 5.3 Logical Coherence ‣ 5 Analysis ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> presents results that show a discernible, albeit not substantial, decrease in the QLoRA-edited models. However, there is a surprisingly significant decline in MEMIT-edited models. This indicates a substantial loss of fundamental language modeling abilities, likely due to the catastrophic forgetting associated with MEMIT editing.</p>
</div>
</section>
<section id="S5.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.4 </span>Discussion</h3>

<section id="S5.SS4.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">5.4.1 </span>QLoRA vs. MEMIT</h4>

<div id="S5.SS4.SSS1.p1" class="ltx_para">
<p id="S5.SS4.SSS1.p1.1" class="ltx_p">For QLoRA, we observe that while it adequately supports fact-wise editing and generally preserves logical coherence, its primary deficiency lies in the retrieval of edited facts. In LLM, the elicitation of knowledge depends heavily on appropriate prompting techniques while our approach involves merely fine-tuning LLMs with atomic fact sentences. Consequently, a potential future direction may involve enriching these atomic facts with more comprehensive contexts prior to their utilization in fine-tuning, as it should enable the model to accurately recall information in response to a diverse set of prompts.</p>
</div>
<div id="S5.SS4.SSS1.p2" class="ltx_para">
<p id="S5.SS4.SSS1.p2.1" class="ltx_p">In contrast, the MEMIT model exhibits a decline in all three assessed abilities: fact-wise editing, fact recall, and coherence. Given that our dataset comprises non-synthetic reasoning questions, which often include complex subjects (e.g., an event) and relations, non-unique objects (Figure <a href="#S3.F3" title="Figure 3 ‣ 3.3 Fact Representation ‣ 3 ReCoE Dataset ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>), the underperformance of MEMIT suggests its current inadequacy in handling real-world factual knowledge. Notably, MEMIT’s most pronounced deficiencies lie in its ability to recall facts and, critically, in maintaining coherence. This observation highlights that the functionalities of edited neurons extend beyond mere fact storage, challenging the assertions made in previous studies <cite class="ltx_cite ltx_citemacro_cite">Dai et al. (<a href="#bib.bib3" title="" class="ltx_ref">2021</a>); Meng et al. (<a href="#bib.bib16" title="" class="ltx_ref">2022a</a>, <a href="#bib.bib17" title="" class="ltx_ref">b</a>)</cite>.</p>
</div>
</section>
<section id="S5.SS4.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">5.4.2 </span>Effect of Model Scaling</h4>

<figure id="S5.F6" class="ltx_figure"><svg id="S5.F6.pic1" class="ltx_picture" height="267.23" overflow="visible" version="1.1" width="294.72"><g transform="translate(0,267.23) matrix(1 0 0 -1 0 0) translate(23.44,0) translate(0,37.04) matrix(1.0 0.0 0.0 1.0 -23.44 -37.04)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g class="ltx_nestedsvg" transform="matrix(1 0 0 1 0 0) translate(54.57,0) translate(0,62.9)"><g stroke-width="0.2pt" fill="#808080" stroke="#808080" color="#808080"><path d="M 0 -31.77 L 0 -25.86 M 69.19 -31.77 L 69.19 -25.86 M 138.37 -31.77 L 138.37 -25.86 M 207.56 -31.77 L 207.56 -25.86 M 0 204.2 L 0 198.29 M 69.19 204.2 L 69.19 198.29 M 138.37 204.2 L 138.37 198.29 M 207.56 204.2 L 207.56 198.29" style="fill:none"></path></g><g stroke-width="0.2pt" fill="#808080" stroke="#808080" color="#808080"><path d="M -31.13 -9.18 L -25.23 -9.18 M -31.13 57.09 L -25.23 57.09 M -31.13 123.36 L -25.23 123.36 M -31.13 189.62 L -25.23 189.62 M 238.69 -9.18 L 232.78 -9.18 M 238.69 57.09 L 232.78 57.09 M 238.69 123.36 L 232.78 123.36 M 238.69 189.62 L 232.78 189.62" style="fill:none"></path></g><g stroke="#000000" fill="#000000" stroke-width="0.4pt"><path d="M -31.13 -25.86 L -31.13 198.29 L 238.69 198.29 L 238.69 -25.86 L -31.13 -25.86 Z" style="fill:none"></path><g transform="matrix(0.98482 0.17365 -0.17365 0.98482 -47.56 -54.91)" fill="#000000" stroke="#000000"><foreignObject width="101.7" height="12.3" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F6.pic1.16.16.16.16.16.1.1" class="ltx_text">KnowledgeProbe</span></foreignObject></g><g transform="matrix(0.98482 0.17365 -0.17365 0.98482 42.93 -51.07)" fill="#000000" stroke="#000000"><foreignObject width="58.42" height="12.15" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F6.pic1.17.17.17.17.17.1.1" class="ltx_text">InputAug</span></foreignObject></g><g transform="matrix(0.98482 0.17365 -0.17365 0.98482 117.79 -50.07)" fill="#000000" stroke="#000000"><foreignObject width="46.89" height="12.15" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F6.pic1.18.18.18.18.18.1.1" class="ltx_text">QLoRA</span></foreignObject></g><g transform="matrix(0.98482 0.17365 -0.17365 0.98482 185.79 -51.65)" fill="#000000" stroke="#000000"><foreignObject width="49.77" height="9.46" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F6.pic1.19.19.19.19.19.1.1" class="ltx_text">MEMIT</span></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 -42.94 -13.64)" fill="#000000" stroke="#000000"><foreignObject width="6.92" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S5.F6.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="0" display="inline"><semantics id="S5.F6.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S5.F6.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S5.F6.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="S5.F6.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="integer" id="S5.F6.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.F6.pic1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1">0</cn></annotation-xml></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 -49.86 52.63)" fill="#000000" stroke="#000000"><foreignObject width="13.84" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S5.F6.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="20" display="inline"><semantics id="S5.F6.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S5.F6.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S5.F6.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">20</mn><annotation-xml encoding="MathML-Content" id="S5.F6.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="integer" id="S5.F6.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.F6.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1">20</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.F6.pic1.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1c">20</annotation></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 -49.86 118.9)" fill="#000000" stroke="#000000"><foreignObject width="13.84" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S5.F6.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="40" display="inline"><semantics id="S5.F6.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S5.F6.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S5.F6.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">40</mn><annotation-xml encoding="MathML-Content" id="S5.F6.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="integer" id="S5.F6.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.F6.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1">40</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.F6.pic1.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1c">40</annotation></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 -49.86 185.16)" fill="#000000" stroke="#000000"><foreignObject width="13.84" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S5.F6.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="60" display="inline"><semantics id="S5.F6.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S5.F6.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S5.F6.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">60</mn><annotation-xml encoding="MathML-Content" id="S5.F6.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="integer" id="S5.F6.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.F6.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1.1">60</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.F6.pic1.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1c">60</annotation></semantics></math></foreignObject></g><clipPath id="pgfcp5"><path d="M -31.13 -25.86 L 238.69 -25.86 L 238.69 198.29 L -31.13 198.29 Z"></path></clipPath><g clip-path="url(#pgfcp5)"><g stroke="#0000FF" fill="#B3B3FF" color="#0000FF"><path d="M -23.52 -9.18 h 13.84 v 106.96 h -13.84 Z M 45.66 -9.18 h 13.84 v 103.71 h -13.84 Z M 114.85 -9.18 h 13.84 v 46.42 h -13.84 Z M 184.03 -9.18 h 13.84 v 9.18 h -13.84 Z"></path></g><g></g><g stroke="#FF0000" fill="#FFB3B3" color="#FF0000"><path d="M -6.92 -9.18 h 13.84 v 117.26 h -13.84 Z M 62.27 -9.18 h 13.84 v 148.17 h -13.84 Z M 131.45 -9.18 h 13.84 v 37.97 h -13.84 Z M 200.64 -9.18 h 13.84 v 18.32 h -13.84 Z"></path></g><g></g><g stroke="#734D26" fill="#ECD9C6" color="#734D26"><path d="M 9.69 -9.18 h 13.84 v 154.53 h -13.84 Z M 78.87 -9.18 h 13.84 v 181.6 h -13.84 Z M 148.06 -9.18 h 13.84 v 41.32 h -13.84 Z"></path></g><g></g></g><g transform="matrix(1.0 0.0 0.0 1.0 -33.52 102.67)" fill="#0000FF" stroke="#0000FF" color="#0000FF"><foreignObject width="33.82" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S5.F6.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="32.28" display="inline"><semantics id="S5.F6.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S5.F6.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S5.F6.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">32.28</mn><annotation-xml encoding="MathML-Content" id="S5.F6.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="float" id="S5.F6.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.F6.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1">32.28</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.F6.pic1.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.5.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1c">32.28</annotation></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 39.13 99.42)" fill="#0000FF" stroke="#0000FF" color="#0000FF"><foreignObject width="26.91" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S5.F6.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="31.3" display="inline"><semantics id="S5.F6.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S5.F6.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S5.F6.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">31.3</mn><annotation-xml encoding="MathML-Content" id="S5.F6.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="float" id="S5.F6.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.F6.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1">31.3</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.F6.pic1.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.6.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1c">31.3</annotation></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 104.85 42.13)" fill="#0000FF" stroke="#0000FF" color="#0000FF"><foreignObject width="33.82" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S5.F6.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="14.01" display="inline"><semantics id="S5.F6.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S5.F6.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S5.F6.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">14.01</mn><annotation-xml encoding="MathML-Content" id="S5.F6.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="float" id="S5.F6.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.F6.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1">14.01</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.F6.pic1.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.7.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1c">14.01</annotation></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 177.5 4.89)" fill="#0000FF" stroke="#0000FF" color="#0000FF"><foreignObject width="26.91" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S5.F6.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="2.77" display="inline"><semantics id="S5.F6.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S5.F6.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S5.F6.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">2.77</mn><annotation-xml encoding="MathML-Content" id="S5.F6.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="float" id="S5.F6.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.F6.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1.1">2.77</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.F6.pic1.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.8.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1c">2.77</annotation></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 -16.91 112.97)" fill="#FF0000" stroke="#FF0000" color="#FF0000"><foreignObject width="33.82" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S5.F6.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="35.39" display="inline"><semantics id="S5.F6.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S5.F6.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S5.F6.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">35.39</mn><annotation-xml encoding="MathML-Content" id="S5.F6.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="float" id="S5.F6.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.F6.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1">35.39</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.F6.pic1.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.9.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1c">35.39</annotation></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 52.27 143.88)" fill="#FF0000" stroke="#FF0000" color="#FF0000"><foreignObject width="33.82" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S5.F6.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="44.72" display="inline"><semantics id="S5.F6.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S5.F6.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S5.F6.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">44.72</mn><annotation-xml encoding="MathML-Content" id="S5.F6.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="float" id="S5.F6.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.F6.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1">44.72</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.F6.pic1.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.10.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1c">44.72</annotation></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 121.46 33.68)" fill="#FF0000" stroke="#FF0000" color="#FF0000"><foreignObject width="33.82" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S5.F6.pic1.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="11.46" display="inline"><semantics id="S5.F6.pic1.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S5.F6.pic1.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S5.F6.pic1.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">11.46</mn><annotation-xml encoding="MathML-Content" id="S5.F6.pic1.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="float" id="S5.F6.pic1.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.F6.pic1.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1">11.46</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.F6.pic1.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.11.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1c">11.46</annotation></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 194.1 14.03)" fill="#FF0000" stroke="#FF0000" color="#FF0000"><foreignObject width="26.91" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S5.F6.pic1.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="5.53" display="inline"><semantics id="S5.F6.pic1.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1a"><mn id="S5.F6.pic1.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S5.F6.pic1.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">5.53</mn><annotation-xml encoding="MathML-Content" id="S5.F6.pic1.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="float" id="S5.F6.pic1.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.F6.pic1.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1.1">5.53</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.F6.pic1.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.12.4.4.4.4.4.4.4.1.1.1.1.1.1.1.1.1.m1.1c">5.53</annotation></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 -0.31 150.25)" fill="#734D26" stroke="#734D26" color="#734D26"><foreignObject width="33.82" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S5.F6.pic1.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="46.64" display="inline"><semantics id="S5.F6.pic1.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1a"><mn mathcolor="#734D26" id="S5.F6.pic1.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S5.F6.pic1.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">46.64</mn><annotation-xml encoding="MathML-Content" id="S5.F6.pic1.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="float" id="S5.F6.pic1.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.F6.pic1.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1.1">46.64</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.F6.pic1.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.13.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.m1.1c">46.64</annotation></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 68.88 177.32)" fill="#734D26" stroke="#734D26" color="#734D26"><foreignObject width="33.82" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S5.F6.pic1.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="54.81" display="inline"><semantics id="S5.F6.pic1.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1a"><mn mathcolor="#734D26" id="S5.F6.pic1.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S5.F6.pic1.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">54.81</mn><annotation-xml encoding="MathML-Content" id="S5.F6.pic1.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="float" id="S5.F6.pic1.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.F6.pic1.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1.1">54.81</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.F6.pic1.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.14.2.2.2.2.2.2.2.1.1.1.1.1.1.1.1.1.m1.1c">54.81</annotation></semantics></math></foreignObject></g><g transform="matrix(1.0 0.0 0.0 1.0 138.06 37.03)" fill="#734D26" stroke="#734D26" color="#734D26"><foreignObject width="33.82" height="8.92" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><math id="S5.F6.pic1.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="12.47" display="inline"><semantics id="S5.F6.pic1.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1a"><mn mathcolor="#734D26" id="S5.F6.pic1.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1" xref="S5.F6.pic1.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1.cmml">12.47</mn><annotation-xml encoding="MathML-Content" id="S5.F6.pic1.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1b"><cn type="float" id="S5.F6.pic1.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S5.F6.pic1.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1.1">12.47</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.F6.pic1.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.15.3.3.3.3.3.3.3.1.1.1.1.1.1.1.1.1.m1.1c">12.47</annotation></semantics></math></foreignObject></g><g fill="#FFFFFF" stroke="#000000"><path d="M 176.55 129.61 h 48.37 v 57.19 h -48.37 Z"></path></g><g fill="#FFFFFF" stroke="#000000" transform="matrix(1.0 0.0 0.0 1.0 180.7 132.38)"><g class="ltx_tikzmatrix" transform="matrix(1 0 0 -1 0 43.05)"><g class="ltx_tikzmatrix_row" transform="matrix(1 0 0 1 0 8.61)"><g class="ltx_tikzmatrix_col ltx_nopad_l ltx_nopad_r" transform="matrix(1 0 0 -1 0 0) translate(2.35,0)" fill="#B3B3FF" stroke="#0000FF" color="#0000FF"><path d="M -2.08 -2.77 h 4.15 v 11.07 h -4.15 Z M 6.23 -2.77 h 4.15 v 8.3 h -4.15 Z"></path></g><g class="ltx_tikzmatrix_col ltx_nopad_l ltx_nopad_r" transform="matrix(1 0 0 -1 16.47 0) translate(10.07,0) matrix(1.0 0.0 0.0 1.0 -7.3 -3.77)" fill="#000000" stroke="#000000"><foreignObject width="14.61" height="9.61" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F6.pic1.20.20.20.20.20.1.1.1.1.1" class="ltx_text">7b</span></foreignObject></g></g><g class="ltx_tikzmatrix_row" transform="matrix(1 0 0 1 0 25.83)"><g class="ltx_tikzmatrix_col ltx_nopad_l ltx_nopad_r" transform="matrix(1 0 0 -1 0 0) translate(2.35,0)" fill="#FFB3B3" stroke="#FF0000" color="#FF0000"><path d="M -2.08 -2.77 h 4.15 v 11.07 h -4.15 Z M 6.23 -2.77 h 4.15 v 8.3 h -4.15 Z"></path></g><g class="ltx_tikzmatrix_col ltx_nopad_l ltx_nopad_r" transform="matrix(1 0 0 -1 13.01 0) translate(13.53,0) matrix(1.0 0.0 0.0 1.0 -10.76 -3.77)" fill="#000000" stroke="#000000"><foreignObject width="21.52" height="9.61" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F6.pic1.21.21.21.21.21.2.2.1.1.1" class="ltx_text">13b</span></foreignObject></g></g><g class="ltx_tikzmatrix_row" transform="matrix(1 0 0 1 0 43.05)"><g class="ltx_tikzmatrix_col ltx_nopad_l ltx_nopad_r" transform="matrix(1 0 0 -1 0 0) translate(2.35,0)" fill="#ECD9C6" stroke="#734D26" color="#734D26"><path d="M -2.08 -2.77 h 4.15 v 11.07 h -4.15 Z M 6.23 -2.77 h 4.15 v 8.3 h -4.15 Z"></path></g><g class="ltx_tikzmatrix_col ltx_nopad_l ltx_nopad_r" transform="matrix(1 0 0 -1 13.01 0) translate(13.53,0) matrix(1.0 0.0 0.0 1.0 -10.76 -3.77)" fill="#000000" stroke="#000000"><foreignObject width="21.52" height="9.61" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible"><span id="S5.F6.pic1.22.22.22.22.22.3.3.1.1.1" class="ltx_text">33b</span></foreignObject></g></g></g></g></g></g></g></svg>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 6: </span>Effect of model scaling. The metric (y-axis) refers to averaged <span id="S5.F6.3.1" class="ltx_text ltx_font_italic">accuracy</span> over all reasoning schemes for KnowledgeProb, and <span id="S5.F6.4.2" class="ltx_text ltx_font_italic">correct_flip</span> for editing approaches.</figcaption>
</figure>
<div id="S5.SS4.SSS2.p1" class="ltx_para">
<p id="S5.SS4.SSS2.p1.1" class="ltx_p">The impact of model scaling is a critical factor in both original knowledge probing and the input-augmentation approach, which are shown in Figure <a href="#S5.F6" title="Figure 6 ‣ 5.4.2 Effect of Model Scaling ‣ 5.4 Discussion ‣ 5 Analysis ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>, echoes with current studies that larger models inherently possess a more extensive knowledge base and demonstrate superior reasoning capabilities.</p>
</div>
<div id="S5.SS4.SSS2.p2" class="ltx_para">
<p id="S5.SS4.SSS2.p2.1" class="ltx_p">However, experiments in this research reveal that upon editing new knowledge into these models, the size of the model does not correspond to enhanced performance in several dimensions, as shown in Figure <a href="#S5.F5" title="Figure 5 ‣ 5.2 Fact Recall ‣ 5 Analysis ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> and Table <a href="#S5.T5" title="Table 5 ‣ 5.3 Logical Coherence ‣ 5 Analysis ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>. Specifically, larger models do not exhibit (1) increased factual effectiveness, (2) improved ability in retrieving facts during chain-of-thought processes in terms of relatedness and consistency, and (3) more coherent chain-of-thought performance. In summary, during the model editing phase, the size of the model does not inherently confer any advantageous properties. Consequently, we have not detected any notable improvements attributable to model scaling, such as facilitation of the editing process or provision of inherent advantages.</p>
</div>
</section>
</section>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Conclusion</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">In this study, we have developed a novel benchmark, ReCoE, which leverages counterfactual reasoning and is grounded in non-synthetic data for evaluating model editing. Our analysis reveals significant challenges in existing knowledge editing approaches, particularly in their ability to effectively propagate new facts for coherent reasoning. Through this investigation, we have identified key areas where these methods falter. Our work provides a clear direction for future research in this field, aiming to enhance the efficacy and reliability of knowledge editing in computational models.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2020)</span>
<span class="ltx_bibblock">
Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu.
2020.

</span>
<span class="ltx_bibblock">Recall and learn: Fine-tuning deep pretrained language models with
less forgetting.

</span>
<span class="ltx_bibblock"><em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2004.12651</em>.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cheng et al. (2023)</span>
<span class="ltx_bibblock">
Siyuan Cheng, Bozhong Tian, Qingbin Liu, Xi Chen, Yongheng Wang, Huajun Chen,
and Ningyu Zhang. 2023.

</span>
<span class="ltx_bibblock">Can we edit multimodal large language models?

</span>
<span class="ltx_bibblock"><em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2310.08475</em>.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dai et al. (2021)</span>
<span class="ltx_bibblock">
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2021.

</span>
<span class="ltx_bibblock">Knowledge neurons in pretrained transformers.

</span>
<span class="ltx_bibblock"><em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2104.08696</em>.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">De Cao et al. (2021)</span>
<span class="ltx_bibblock">
Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021.

</span>
<span class="ltx_bibblock">Editing factual knowledge in language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2104.08164</em>.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dettmers et al. (2023)</span>
<span class="ltx_bibblock">
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023.

</span>
<span class="ltx_bibblock">Qlora: Efficient finetuning of quantized llms.

</span>
<span class="ltx_bibblock"><em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.14314</em>.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dong et al. (2022)</span>
<span class="ltx_bibblock">
Qingxiu Dong, Damai Dai, Yifan Song, Jingjing Xu, Zhifang Sui, and Lei Li.
2022.

</span>
<span class="ltx_bibblock">Calibrating factual knowledge in pretrained language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2210.03329</em>.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Du et al. (2023)</span>
<span class="ltx_bibblock">
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch.
2023.

</span>
<span class="ltx_bibblock">Improving factuality and reasoning in language models through
multiagent debate.

</span>
<span class="ltx_bibblock"><em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.14325</em>.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Elsahar et al. (2018)</span>
<span class="ltx_bibblock">
Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon
Hare, Frederique Laforest, and Elena Simperl. 2018.

</span>
<span class="ltx_bibblock">T-rex: A large scale alignment of natural language with knowledge
base triples.

</span>
<span class="ltx_bibblock">In <em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">Proceedings of the Eleventh International Conference on
Language Resources and Evaluation (LREC 2018)</em>.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gu et al. (2021)</span>
<span class="ltx_bibblock">
Yu Gu, Sue Kase, Michelle Vanni, Brian Sadler, Percy Liang, Xifeng Yan, and
Yu Su. 2021.

</span>
<span class="ltx_bibblock">Beyond iid: three levels of generalization for question answering on
knowledge bases.

</span>
<span class="ltx_bibblock">In <em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">Proceedings of the Web Conference 2021</em>, pages 3477–3488.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hase et al. (2021)</span>
<span class="ltx_bibblock">
Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin
Stoyanov, Mohit Bansal, and Srinivasan Iyer. 2021.

</span>
<span class="ltx_bibblock">Do language models have beliefs? methods for detecting, updating, and
visualizing model beliefs.

</span>
<span class="ltx_bibblock"><em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2111.13654</em>.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hu et al. (2023)</span>
<span class="ltx_bibblock">
Nathan Hu, Eric Mitchell, Christopher D Manning, and Chelsea Finn. 2023.

</span>
<span class="ltx_bibblock">Meta-learning online adaptation of language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.15076</em>.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Izacard et al. (2021a)</span>
<span class="ltx_bibblock">
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr
Bojanowski, Armand Joulin, and Edouard Grave. 2021a.

</span>
<span class="ltx_bibblock">Unsupervised dense information retrieval with contrastive learning.

</span>
<span class="ltx_bibblock"><em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2112.09118</em>.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Izacard et al. (2021b)</span>
<span class="ltx_bibblock">
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr
Bojanowski, Armand Joulin, and Edouard Grave. 2021b.

</span>
<span class="ltx_bibblock">Unsupervised dense information retrieval with contrastive learning.

</span>
<span class="ltx_bibblock"><em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2112.09118</em>.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jiang et al. (2019)</span>
<span class="ltx_bibblock">
Kelvin Jiang, Dekun Wu, and Hui Jiang. 2019.

</span>
<span class="ltx_bibblock">Freebaseqa: A new factoid qa data set matching trivia-style
question-answer pairs with freebase.

</span>
<span class="ltx_bibblock">In <em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers)</em>, pages 318–323.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kwiatkowski et al. (2019)</span>
<span class="ltx_bibblock">
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur
Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin,
Kenton Lee, et al. 2019.

</span>
<span class="ltx_bibblock">Natural questions: a benchmark for question answering research.

</span>
<span class="ltx_bibblock"><em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">Transactions of the Association for Computational Linguistics</em>,
7:453–466.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Meng et al. (2022a)</span>
<span class="ltx_bibblock">
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022a.

</span>
<span class="ltx_bibblock">Locating and editing factual associations in gpt.

</span>
<span class="ltx_bibblock"><em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>,
35:17359–17372.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Meng et al. (2022b)</span>
<span class="ltx_bibblock">
Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau.
2022b.

</span>
<span class="ltx_bibblock">Mass-editing memory in a transformer.

</span>
<span class="ltx_bibblock"><em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2210.07229</em>.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mitchell et al. (2021)</span>
<span class="ltx_bibblock">
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D
Manning. 2021.

</span>
<span class="ltx_bibblock">Fast model editing at scale.

</span>
<span class="ltx_bibblock"><em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2110.11309</em>.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mitchell et al. (2022)</span>
<span class="ltx_bibblock">
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and
Chelsea Finn. 2022.

</span>
<span class="ltx_bibblock">Memory-based model editing at scale.

</span>
<span class="ltx_bibblock">In <em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>, pages
15817–15831. PMLR.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Onoe et al. (2022)</span>
<span class="ltx_bibblock">
Yasumasa Onoe, Michael Zhang, Eunsol Choi, and Greg Durrett. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2022.findings-naacl.52" title="" class="ltx_ref ltx_href">Entity
cloze by date: What LMs know about unseen entities</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">Findings of the Association for Computational Linguistics:
NAACL 2022</em>, pages 693–702, Seattle, United States. Association for
Computational Linguistics.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Onoe et al. (2023)</span>
<span class="ltx_bibblock">
Yasumasa Onoe, Michael JQ Zhang, Shankar Padmanabhan, Greg Durrett, and Eunsol
Choi. 2023.

</span>
<span class="ltx_bibblock">Can lms learn new entities from descriptions? challenges in
propagating injected knowledge.

</span>
<span class="ltx_bibblock"><em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.01651</em>.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Pinter and Elhadad (2023)</span>
<span class="ltx_bibblock">
Yuval Pinter and Michael Elhadad. 2023.

</span>
<span class="ltx_bibblock">Emptying the ocean with a spoon: Should we edit models?

</span>
<span class="ltx_bibblock"><em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2310.11958</em>.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rawte Vipula (2023)</span>
<span class="ltx_bibblock">
Das Amitava Rawte Vipula, Sheth Amit. 2023.

</span>
<span class="ltx_bibblock">A survey of hallucination in large foundation models.

</span>
<span class="ltx_bibblock"><em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv: arXiv:2309.05922</em>.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Talmor and Berant (2018)</span>
<span class="ltx_bibblock">
Alon Talmor and Jonathan Berant. 2018.

</span>
<span class="ltx_bibblock">The web as a knowledge-base for answering complex questions.

</span>
<span class="ltx_bibblock"><em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1803.06643</em>.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Vu et al. (2023)</span>
<span class="ltx_bibblock">
Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris
Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, et al. 2023.

</span>
<span class="ltx_bibblock">Freshllms: Refreshing large language models with search engine
augmentation.

</span>
<span class="ltx_bibblock"><em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2310.03214</em>.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2023a)</span>
<span class="ltx_bibblock">
Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, et al.
2023a.

</span>
<span class="ltx_bibblock">Knowledge editing for large language models: A survey.

</span>
<span class="ltx_bibblock"><em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2310.16218</em>.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2023b)</span>
<span class="ltx_bibblock">
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot,
Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith,
Iz Beltagy, et al. 2023b.

</span>
<span class="ltx_bibblock">How far can camels go? exploring the state of instruction tuning on
open resources.

</span>
<span class="ltx_bibblock"><em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2306.04751</em>.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wei et al. (2022)</span>
<span class="ltx_bibblock">
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V
Le, Denny Zhou, et al. 2022.

</span>
<span class="ltx_bibblock">Chain-of-thought prompting elicits reasoning in large language
models.

</span>
<span class="ltx_bibblock"><em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>,
35:24824–24837.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2024)</span>
<span class="ltx_bibblock">
Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang,
Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, et al. 2024.

</span>
<span class="ltx_bibblock">A comprehensive study of knowledge editing for large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2401.01286</em>.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhong et al. (2023)</span>
<span class="ltx_bibblock">
Zexuan Zhong, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi
Chen. 2023.

</span>
<span class="ltx_bibblock">Mquake: Assessing knowledge editing in language models via multi-hop
questions.

</span>
<span class="ltx_bibblock"><em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.14795</em>.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhu et al. (2020a)</span>
<span class="ltx_bibblock">
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li,
Felix Yu, and Sanjiv Kumar. 2020a.

</span>
<span class="ltx_bibblock">Modifying memories in transformer models.

</span>
<span class="ltx_bibblock"><em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2012.00363</em>.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhu et al. (2020b)</span>
<span class="ltx_bibblock">
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li,
Felix Yu, and Sanjiv Kumar. 2020b.

</span>
<span class="ltx_bibblock">Modifying memories in transformer models.

</span>
<span class="ltx_bibblock"><em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2012.00363</em>.

</span>
</li>
</ul>
</section>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Appendix</h2>

<section id="A1.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.1 </span>Example Datapoint</h3>

<figure id="A1.F7" class="ltx_figure">
<div id="A1.F7.1" class="ltx_listing ltx_lst_numbers_left ltx_lstlisting ltx_framed ltx_framed_rectangle ltx_listing" style="background-color:#FFFFFF;">
<div class="ltx_listing_data"><a href="data:text/plain;base64,ewogICJxdWVzdGlvbiI6ICJXaG8gaXMgdGhlIGxhc3QgY2VsZWJyaXR5IEJyb2R5IEplbm5lciBoYWQgYSByb21hbnRpYyByZWxhdGlvbnNoaXAgd2l0aD8iLAogICJhbnN3ZXIiOiAiTGF1cmVuIENvbnJhZCIsCiAgImNvdW50ZXJmYWN0dWFsX2Fuc3dlciI6ICJIZWlkaSBNb250YWciLAogICJmYWN0c19wZXJfY2hvaWNlIjogewogICAgImNob2ljZV8xX2ZhY3RzIjogWwogICAgICB7CiAgICAgICAgImZhY3QiOiAiTGF1cmVuIENvbnJhZCBhbmQgQnJvZHkgSmVubmVyIGJyaWVmbHkgZGF0ZWQgaW4gMjAwNi4iLAogICAgICAgICJsaW5rcyI6IFsKICAgICAgICAgICJodHRwczovL2VuLndpa2lwZWRpYS5vcmcvd2lraS9MYXVyZW5fQ29ucmFkIgogICAgICAgIF0sCiAgICAgICAgImF0b21pY19mYWN0cyI6IFsKICAgICAgICAgICJMYXVyZW4gQ29ucmFkIGFuZCBCcm9keSBKZW5uZXIncyBkYXRpbmcgd2FzIGJyaWVmLiIsCiAgICAgICAgICAiTGF1cmVuIENvbnJhZCBhbmQgQnJvZHkgSmVubmVyJ3MgYnJpZWYgZGF0aW5nIHdhcyBpbiAyMDA2LiIKICAgICAgICBdLAogICAgICAgICJhdG9taWNfdHJpcGxlcyI6IFsKICAgICAgICAgICIoTGF1cmVuIENvbnJhZCBhbmQgQnJvZHkgSmVubmVyJ3MgZGF0aW5nOyB3YXM7IGJyaWVmKSIsCiAgICAgICAgICAiKExhdXJlbiBDb25yYWQgYW5kIEJyb2R5IEplbm5lcidzIGJyaWVmIGRhdGluZzsgd2FzIGluOyAyMDA2KSIKICAgICAgICBdCiAgICAgIH0KICAgIF0sCiAgICAiY2hvaWNlXzJfZmFjdHMiOiBbCiAgICAgIHsKICAgICAgICAiZmFjdCI6ICJIZWlkaSBNb250YWcgd2FzIG5ldmVyIHJvbWFudGljYWxseSBpbnZvbHZlZCB3aXRoIEJyb2R5IEplbm5lci4iLAogICAgICAgICJsaW5rcyI6IFsKICAgICAgICAgICJodHRwczovL2VuLndpa2lwZWRpYS5vcmcvd2lraS9IZWlkaV9Nb250YWciCiAgICAgICAgXSwKICAgICAgICAiYXRvbWljX2ZhY3RzIjogWwogICAgICAgICAgIkhlaWRpIE1vbnRhZyB3YXMgbmV2ZXIgcm9tYW50aWNhbGx5IGludm9sdmVkIHdpdGggQnJvZHkgSmVubmVyLiIKICAgICAgICBdLAogICAgICAgICJhdG9taWNfdHJpcGxlcyI6IFsKICAgICAgICAgICIoSGVpZGkgTW9udGFnOyB3YXMgbmV2ZXIgcm9tYW50aWNhbGx5IGludm9sdmVkIHdpdGg7IEJyb2R5IEplbm5lcikiCiAgICAgICAgXQogICAgICB9CiAgICBdCiAgfSwKICAiY291bnRlcmZhY3R1YWxzX3Blcl9jaG9pY2UiOiB7CiAgICAiY2hvaWNlXzFfY291bnRlcmZhY3R1YWxzIjogWwogICAgICB7CiAgICAgICAgImZhY3QiOiAiTGF1cmVuIENvbnJhZCB3YXMgbmV2ZXIgcm9tYW50aWNhbGx5IGludm9sdmVkIHdpdGggQnJvZHkgSmVubmVyLiIsCiAgICAgICAgImF0b21pY19mYWN0cyI6IFsKICAgICAgICAgICJMYXVyZW4gQ29ucmFkIHdhcyBuZXZlciByb21hbnRpY2FsbHkgaW52b2x2ZWQgd2l0aCBCcm9keSBKZW5uZXIuIgogICAgICAgIF0sCiAgICAgICAgImF0b21pY190cmlwbGVzIjogWwogICAgICAgICAgIihMYXVyZW4gQ29ucmFkOyB3YXMgbmV2ZXIgcm9tYW50aWNhbGx5IGludm9sdmVkIHdpdGg7IEJyb2R5IEplbm5lcikiCiAgICAgICAgXQogICAgICB9CiAgICBdLAogICAgImNob2ljZV8yX2NvdW50ZXJmYWN0dWFscyI6IFsKICAgICAgewogICAgICAgICJmYWN0IjogIkhlaWRpIE1vbnRhZyBhbmQgQnJvZHkgSmVubmVyIGJyaWVmbHkgZGF0ZWQgaW4gMjAwNi4iLAogICAgICAgICJhdG9taWNfZmFjdHMiOiBbCiAgICAgICAgICAiSGVpZGkgTW9udGFnIGFuZCBCcm9keSBKZW5uZXIncyBkYXRpbmcgd2FzIGJyaWVmLiIsCiAgICAgICAgICAiSGVpZGkgTW9udGFnIGFuZCBCcm9keSBKZW5uZXIncyBicmllZiBkYXRpbmcgd2FzIGluIDIwMDYuIgogICAgICAgIF0sCiAgICAgICAgImF0b21pY190cmlwbGVzIjogWwogICAgICAgICAgIihIZWlkaSBNb250YWcgYW5kIEJyb2R5IEplbm5lcidzIGRhdGluZzsgd2FzOyBicmllZikiLAogICAgICAgICAgIihIZWlkaSBNb250YWcgYW5kIEJyb2R5IEplbm5lcidzIGJyaWVmIGRhdGluZzsgd2FzIGluOyAyMDA2KSIKICAgICAgICBdCiAgICAgIH0KICAgIF0sCiAgICAidW5jaGFuZ2VkX2ZhY3RzIjogW10KICB9LAogICJhbnN3ZXJfYWxpYXMiOiBbCiAgICAiTGF1cmVuIEsuIENvbnJhZCIsCiAgICAiTGF1cmVuIEthdGhlcmluZSBDb25yYWQiLAogICAgIkwuQy4iCiAgXSwKICAiY291bnRlcmZhY3R1YWxfYW5zd2VyX2FsaWFzIjogWwogICAgIkhlaWRpIFByYXR0IiwKICAgICJIZWlkaSBCbGFpciBNb250YWciLAogICAgIkhlaWRpIEIuIE1vbnRhZyIKICBdCn0=" download="">⬇</a></div>
<div id="lstnumberx1" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">1</span><span id="lstnumberx1.1" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx2" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">2</span><span id="lstnumberx2.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx2.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx2.3" class="ltx_text ltx_lst_keyword ltx_font_typewriter"><span id="lstnumberx2.3.1" class="ltx_text" style="color:#9400D1;">question</span></span><span id="lstnumberx2.4" class="ltx_text ltx_font_typewriter">":</span><span id="lstnumberx2.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.6" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx2.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Who</span><span id="lstnumberx2.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx2.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx2.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">last</span><span id="lstnumberx2.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">celebrity</span><span id="lstnumberx2.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Brody</span><span id="lstnumberx2.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Jenner</span><span id="lstnumberx2.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">had</span><span id="lstnumberx2.22" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.23" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx2.24" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.25" class="ltx_text ltx_lst_identifier ltx_font_typewriter">romantic</span><span id="lstnumberx2.26" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.27" class="ltx_text ltx_lst_identifier ltx_font_typewriter">relationship</span><span id="lstnumberx2.28" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.29" class="ltx_text ltx_lst_identifier ltx_font_typewriter">with</span><span id="lstnumberx2.30" class="ltx_text ltx_font_typewriter">?",</span>
</div>
<div id="lstnumberx3" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">3</span><span id="lstnumberx3.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx3.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx3.3" class="ltx_text ltx_lst_keyword ltx_font_typewriter"><span id="lstnumberx3.3.1" class="ltx_text" style="color:#9400D1;">answer</span></span><span id="lstnumberx3.4" class="ltx_text ltx_font_typewriter">":</span><span id="lstnumberx3.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.6" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx3.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Lauren</span><span id="lstnumberx3.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Conrad</span><span id="lstnumberx3.10" class="ltx_text ltx_font_typewriter">",</span>
</div>
<div id="lstnumberx4" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">4</span><span id="lstnumberx4.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx4.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx4.3" class="ltx_text ltx_lst_keyword ltx_font_typewriter"><span id="lstnumberx4.3.1" class="ltx_text" style="color:#9400D1;">counterfactual_answer</span></span><span id="lstnumberx4.4" class="ltx_text ltx_font_typewriter">":</span><span id="lstnumberx4.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.6" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx4.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Heidi</span><span id="lstnumberx4.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Montag</span><span id="lstnumberx4.10" class="ltx_text ltx_font_typewriter">",</span>
</div>
<div id="lstnumberx5" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">5</span><span id="lstnumberx5.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx5.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx5.3" class="ltx_text ltx_lst_keyword ltx_font_typewriter"><span id="lstnumberx5.3.1" class="ltx_text" style="color:#9400D1;">facts_per_choice</span></span><span id="lstnumberx5.4" class="ltx_text ltx_font_typewriter">":</span><span id="lstnumberx5.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx5.6" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx6" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">6</span><span id="lstnumberx6.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx6.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx6.3" class="ltx_text ltx_lst_keyword ltx_font_typewriter"><span id="lstnumberx6.3.1" class="ltx_text" style="color:#9400D1;">choice_1_facts</span></span><span id="lstnumberx6.4" class="ltx_text ltx_font_typewriter">":</span><span id="lstnumberx6.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx6.6" class="ltx_text ltx_font_typewriter">[</span>
</div>
<div id="lstnumberx7" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">7</span><span id="lstnumberx7.1" class="ltx_text ltx_lst_space ltx_font_typewriter">      </span><span id="lstnumberx7.2" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx8" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">8</span><span id="lstnumberx8.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx8.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx8.3" class="ltx_text ltx_lst_keyword ltx_font_typewriter"><span id="lstnumberx8.3.1" class="ltx_text" style="color:#9400D1;">fact</span></span><span id="lstnumberx8.4" class="ltx_text ltx_font_typewriter">":</span><span id="lstnumberx8.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx8.6" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx8.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Lauren</span><span id="lstnumberx8.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx8.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Conrad</span><span id="lstnumberx8.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx8.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx8.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx8.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Brody</span><span id="lstnumberx8.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx8.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Jenner</span><span id="lstnumberx8.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx8.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">briefly</span><span id="lstnumberx8.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx8.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">dated</span><span id="lstnumberx8.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx8.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx8.22" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx8.23" class="ltx_text ltx_font_typewriter">2006.",</span>
</div>
<div id="lstnumberx9" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">9</span><span id="lstnumberx9.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx9.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx9.3" class="ltx_text ltx_lst_keyword ltx_font_typewriter"><span id="lstnumberx9.3.1" class="ltx_text" style="color:#9400D1;">links</span></span><span id="lstnumberx9.4" class="ltx_text ltx_font_typewriter">":</span><span id="lstnumberx9.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx9.6" class="ltx_text ltx_font_typewriter">[</span>
</div>
<div id="lstnumberx10" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">10</span><span id="lstnumberx10.1" class="ltx_text ltx_lst_space ltx_font_typewriter">          </span><span id="lstnumberx10.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx10.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">https</span><span id="lstnumberx10.4" class="ltx_text ltx_font_typewriter">://</span><span id="lstnumberx10.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">en</span><span id="lstnumberx10.6" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx10.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">wikipedia</span><span id="lstnumberx10.8" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx10.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">org</span><span id="lstnumberx10.10" class="ltx_text ltx_font_typewriter">/</span><span id="lstnumberx10.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">wiki</span><span id="lstnumberx10.12" class="ltx_text ltx_font_typewriter">/</span><span id="lstnumberx10.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Lauren_Conrad</span><span id="lstnumberx10.14" class="ltx_text ltx_font_typewriter">"</span>
</div>
<div id="lstnumberx11" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">11</span><span id="lstnumberx11.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx11.2" class="ltx_text ltx_font_typewriter">],</span>
</div>
<div id="lstnumberx12" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">12</span><span id="lstnumberx12.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx12.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx12.3" class="ltx_text ltx_lst_keyword ltx_font_typewriter"><span id="lstnumberx12.3.1" class="ltx_text" style="color:#9400D1;">atomic_facts</span></span><span id="lstnumberx12.4" class="ltx_text ltx_font_typewriter">":</span><span id="lstnumberx12.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx12.6" class="ltx_text ltx_font_typewriter">[</span>
</div>
<div id="lstnumberx13" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">13</span><span id="lstnumberx13.1" class="ltx_text ltx_lst_space ltx_font_typewriter">          </span><span id="lstnumberx13.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx13.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Lauren</span><span id="lstnumberx13.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx13.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Conrad</span><span id="lstnumberx13.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx13.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx13.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx13.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Brody</span><span id="lstnumberx13.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx13.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Jenner</span><span id="lstnumberx13.12" class="ltx_text ltx_font_typewriter">'</span><span id="lstnumberx13.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">s</span><span id="lstnumberx13.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx13.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">dating</span><span id="lstnumberx13.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx13.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">was</span><span id="lstnumberx13.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx13.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">brief</span><span id="lstnumberx13.20" class="ltx_text ltx_font_typewriter">.",</span>
</div>
<div id="lstnumberx14" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">14</span><span id="lstnumberx14.1" class="ltx_text ltx_lst_space ltx_font_typewriter">          </span><span id="lstnumberx14.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx14.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Lauren</span><span id="lstnumberx14.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx14.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Conrad</span><span id="lstnumberx14.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx14.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx14.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx14.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Brody</span><span id="lstnumberx14.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx14.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Jenner</span><span id="lstnumberx14.12" class="ltx_text ltx_font_typewriter">'</span><span id="lstnumberx14.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">s</span><span id="lstnumberx14.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx14.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">brief</span><span id="lstnumberx14.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx14.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">dating</span><span id="lstnumberx14.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx14.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">was</span><span id="lstnumberx14.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx14.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx14.22" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx14.23" class="ltx_text ltx_font_typewriter">2006."</span>
</div>
<div id="lstnumberx15" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">15</span><span id="lstnumberx15.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx15.2" class="ltx_text ltx_font_typewriter">],</span>
</div>
<div id="lstnumberx16" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">16</span><span id="lstnumberx16.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx16.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx16.3" class="ltx_text ltx_lst_keyword ltx_font_typewriter"><span id="lstnumberx16.3.1" class="ltx_text" style="color:#9400D1;">atomic_triples</span></span><span id="lstnumberx16.4" class="ltx_text ltx_font_typewriter">":</span><span id="lstnumberx16.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx16.6" class="ltx_text ltx_font_typewriter">[</span>
</div>
<div id="lstnumberx17" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">17</span><span id="lstnumberx17.1" class="ltx_text ltx_lst_space ltx_font_typewriter">          </span><span id="lstnumberx17.2" class="ltx_text ltx_font_typewriter">"(</span><span id="lstnumberx17.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Lauren</span><span id="lstnumberx17.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx17.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Conrad</span><span id="lstnumberx17.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx17.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx17.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx17.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Brody</span><span id="lstnumberx17.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx17.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Jenner</span><span id="lstnumberx17.12" class="ltx_text ltx_font_typewriter">'</span><span id="lstnumberx17.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">s</span><span id="lstnumberx17.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx17.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">dating</span><span id="lstnumberx17.16" class="ltx_text ltx_font_typewriter">;</span><span id="lstnumberx17.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx17.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">was</span><span id="lstnumberx17.19" class="ltx_text ltx_font_typewriter">;</span><span id="lstnumberx17.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx17.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">brief</span><span id="lstnumberx17.22" class="ltx_text ltx_font_typewriter">)",</span>
</div>
<div id="lstnumberx18" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">18</span><span id="lstnumberx18.1" class="ltx_text ltx_lst_space ltx_font_typewriter">          </span><span id="lstnumberx18.2" class="ltx_text ltx_font_typewriter">"(</span><span id="lstnumberx18.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Lauren</span><span id="lstnumberx18.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx18.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Conrad</span><span id="lstnumberx18.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx18.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx18.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx18.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Brody</span><span id="lstnumberx18.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx18.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Jenner</span><span id="lstnumberx18.12" class="ltx_text ltx_font_typewriter">'</span><span id="lstnumberx18.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">s</span><span id="lstnumberx18.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx18.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">brief</span><span id="lstnumberx18.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx18.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">dating</span><span id="lstnumberx18.18" class="ltx_text ltx_font_typewriter">;</span><span id="lstnumberx18.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx18.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">was</span><span id="lstnumberx18.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx18.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx18.23" class="ltx_text ltx_font_typewriter">;</span><span id="lstnumberx18.24" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx18.25" class="ltx_text ltx_font_typewriter">2006)"</span>
</div>
<div id="lstnumberx19" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">19</span><span id="lstnumberx19.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx19.2" class="ltx_text ltx_font_typewriter">]</span>
</div>
<div id="lstnumberx20" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">20</span><span id="lstnumberx20.1" class="ltx_text ltx_lst_space ltx_font_typewriter">      </span><span id="lstnumberx20.2" class="ltx_text ltx_font_typewriter">}</span>
</div>
<div id="lstnumberx21" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">21</span><span id="lstnumberx21.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx21.2" class="ltx_text ltx_font_typewriter">],</span>
</div>
<div id="lstnumberx22" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">22</span><span id="lstnumberx22.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx22.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx22.3" class="ltx_text ltx_lst_keyword ltx_font_typewriter"><span id="lstnumberx22.3.1" class="ltx_text" style="color:#9400D1;">choice_2_facts</span></span><span id="lstnumberx22.4" class="ltx_text ltx_font_typewriter">":</span><span id="lstnumberx22.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx22.6" class="ltx_text ltx_font_typewriter">[</span>
</div>
<div id="lstnumberx23" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">23</span><span id="lstnumberx23.1" class="ltx_text ltx_lst_space ltx_font_typewriter">      </span><span id="lstnumberx23.2" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx24" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">24</span><span id="lstnumberx24.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx24.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx24.3" class="ltx_text ltx_lst_keyword ltx_font_typewriter"><span id="lstnumberx24.3.1" class="ltx_text" style="color:#9400D1;">fact</span></span><span id="lstnumberx24.4" class="ltx_text ltx_font_typewriter">":</span><span id="lstnumberx24.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx24.6" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx24.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Heidi</span><span id="lstnumberx24.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx24.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Montag</span><span id="lstnumberx24.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx24.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">was</span><span id="lstnumberx24.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx24.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">never</span><span id="lstnumberx24.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx24.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">romantically</span><span id="lstnumberx24.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx24.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">involved</span><span id="lstnumberx24.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx24.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">with</span><span id="lstnumberx24.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx24.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Brody</span><span id="lstnumberx24.22" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx24.23" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Jenner</span><span id="lstnumberx24.24" class="ltx_text ltx_font_typewriter">.",</span>
</div>
<div id="lstnumberx25" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">25</span><span id="lstnumberx25.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx25.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx25.3" class="ltx_text ltx_lst_keyword ltx_font_typewriter"><span id="lstnumberx25.3.1" class="ltx_text" style="color:#9400D1;">links</span></span><span id="lstnumberx25.4" class="ltx_text ltx_font_typewriter">":</span><span id="lstnumberx25.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx25.6" class="ltx_text ltx_font_typewriter">[</span>
</div>
<div id="lstnumberx26" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">26</span><span id="lstnumberx26.1" class="ltx_text ltx_lst_space ltx_font_typewriter">          </span><span id="lstnumberx26.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx26.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">https</span><span id="lstnumberx26.4" class="ltx_text ltx_font_typewriter">://</span><span id="lstnumberx26.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">en</span><span id="lstnumberx26.6" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx26.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">wikipedia</span><span id="lstnumberx26.8" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx26.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">org</span><span id="lstnumberx26.10" class="ltx_text ltx_font_typewriter">/</span><span id="lstnumberx26.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">wiki</span><span id="lstnumberx26.12" class="ltx_text ltx_font_typewriter">/</span><span id="lstnumberx26.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Heidi_Montag</span><span id="lstnumberx26.14" class="ltx_text ltx_font_typewriter">"</span>
</div>
<div id="lstnumberx27" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">27</span><span id="lstnumberx27.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx27.2" class="ltx_text ltx_font_typewriter">],</span>
</div>
<div id="lstnumberx28" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">28</span><span id="lstnumberx28.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx28.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx28.3" class="ltx_text ltx_lst_keyword ltx_font_typewriter"><span id="lstnumberx28.3.1" class="ltx_text" style="color:#9400D1;">atomic_facts</span></span><span id="lstnumberx28.4" class="ltx_text ltx_font_typewriter">":</span><span id="lstnumberx28.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx28.6" class="ltx_text ltx_font_typewriter">[</span>
</div>
<div id="lstnumberx29" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">29</span><span id="lstnumberx29.1" class="ltx_text ltx_lst_space ltx_font_typewriter">          </span><span id="lstnumberx29.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx29.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Heidi</span><span id="lstnumberx29.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx29.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Montag</span><span id="lstnumberx29.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx29.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">was</span><span id="lstnumberx29.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx29.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">never</span><span id="lstnumberx29.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx29.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">romantically</span><span id="lstnumberx29.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx29.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">involved</span><span id="lstnumberx29.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx29.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">with</span><span id="lstnumberx29.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx29.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Brody</span><span id="lstnumberx29.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx29.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Jenner</span><span id="lstnumberx29.20" class="ltx_text ltx_font_typewriter">."</span>
</div>
<div id="lstnumberx30" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">30</span><span id="lstnumberx30.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx30.2" class="ltx_text ltx_font_typewriter">],</span>
</div>
<div id="lstnumberx31" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">31</span><span id="lstnumberx31.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx31.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx31.3" class="ltx_text ltx_lst_keyword ltx_font_typewriter"><span id="lstnumberx31.3.1" class="ltx_text" style="color:#9400D1;">atomic_triples</span></span><span id="lstnumberx31.4" class="ltx_text ltx_font_typewriter">":</span><span id="lstnumberx31.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx31.6" class="ltx_text ltx_font_typewriter">[</span>
</div>
<div id="lstnumberx32" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">32</span><span id="lstnumberx32.1" class="ltx_text ltx_lst_space ltx_font_typewriter">          </span><span id="lstnumberx32.2" class="ltx_text ltx_font_typewriter">"(</span><span id="lstnumberx32.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Heidi</span><span id="lstnumberx32.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx32.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Montag</span><span id="lstnumberx32.6" class="ltx_text ltx_font_typewriter">;</span><span id="lstnumberx32.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx32.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">was</span><span id="lstnumberx32.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx32.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">never</span><span id="lstnumberx32.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx32.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">romantically</span><span id="lstnumberx32.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx32.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">involved</span><span id="lstnumberx32.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx32.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">with</span><span id="lstnumberx32.17" class="ltx_text ltx_font_typewriter">;</span><span id="lstnumberx32.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx32.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Brody</span><span id="lstnumberx32.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx32.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Jenner</span><span id="lstnumberx32.22" class="ltx_text ltx_font_typewriter">)"</span>
</div>
<div id="lstnumberx33" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">33</span><span id="lstnumberx33.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx33.2" class="ltx_text ltx_font_typewriter">]</span>
</div>
<div id="lstnumberx34" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">34</span><span id="lstnumberx34.1" class="ltx_text ltx_lst_space ltx_font_typewriter">      </span><span id="lstnumberx34.2" class="ltx_text ltx_font_typewriter">}</span>
</div>
<div id="lstnumberx35" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">35</span><span id="lstnumberx35.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx35.2" class="ltx_text ltx_font_typewriter">]</span>
</div>
<div id="lstnumberx36" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">36</span><span id="lstnumberx36.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx36.2" class="ltx_text ltx_font_typewriter">},</span>
</div>
<div id="lstnumberx37" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">37</span><span id="lstnumberx37.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx37.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx37.3" class="ltx_text ltx_lst_keyword ltx_font_typewriter"><span id="lstnumberx37.3.1" class="ltx_text" style="color:#9400D1;">counterfactuals_per_choice</span></span><span id="lstnumberx37.4" class="ltx_text ltx_font_typewriter">":</span><span id="lstnumberx37.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx37.6" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx38" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">38</span><span id="lstnumberx38.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx38.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx38.3" class="ltx_text ltx_lst_keyword ltx_font_typewriter"><span id="lstnumberx38.3.1" class="ltx_text" style="color:#9400D1;">choice_1_counterfactuals</span></span><span id="lstnumberx38.4" class="ltx_text ltx_font_typewriter">":</span><span id="lstnumberx38.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx38.6" class="ltx_text ltx_font_typewriter">[</span>
</div>
<div id="lstnumberx39" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">39</span><span id="lstnumberx39.1" class="ltx_text ltx_lst_space ltx_font_typewriter">      </span><span id="lstnumberx39.2" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx40" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">40</span><span id="lstnumberx40.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx40.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx40.3" class="ltx_text ltx_lst_keyword ltx_font_typewriter"><span id="lstnumberx40.3.1" class="ltx_text" style="color:#9400D1;">fact</span></span><span id="lstnumberx40.4" class="ltx_text ltx_font_typewriter">":</span><span id="lstnumberx40.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx40.6" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx40.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Lauren</span><span id="lstnumberx40.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx40.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Conrad</span><span id="lstnumberx40.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx40.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">was</span><span id="lstnumberx40.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx40.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">never</span><span id="lstnumberx40.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx40.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">romantically</span><span id="lstnumberx40.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx40.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">involved</span><span id="lstnumberx40.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx40.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">with</span><span id="lstnumberx40.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx40.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Brody</span><span id="lstnumberx40.22" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx40.23" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Jenner</span><span id="lstnumberx40.24" class="ltx_text ltx_font_typewriter">.",</span>
</div>
<div id="lstnumberx41" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">41</span><span id="lstnumberx41.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx41.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx41.3" class="ltx_text ltx_lst_keyword ltx_font_typewriter"><span id="lstnumberx41.3.1" class="ltx_text" style="color:#9400D1;">atomic_facts</span></span><span id="lstnumberx41.4" class="ltx_text ltx_font_typewriter">":</span><span id="lstnumberx41.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx41.6" class="ltx_text ltx_font_typewriter">[</span>
</div>
<div id="lstnumberx42" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">42</span><span id="lstnumberx42.1" class="ltx_text ltx_lst_space ltx_font_typewriter">          </span><span id="lstnumberx42.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx42.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Lauren</span><span id="lstnumberx42.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx42.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Conrad</span><span id="lstnumberx42.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx42.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">was</span><span id="lstnumberx42.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx42.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">never</span><span id="lstnumberx42.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx42.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">romantically</span><span id="lstnumberx42.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx42.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">involved</span><span id="lstnumberx42.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx42.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">with</span><span id="lstnumberx42.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx42.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Brody</span><span id="lstnumberx42.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx42.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Jenner</span><span id="lstnumberx42.20" class="ltx_text ltx_font_typewriter">."</span>
</div>
<div id="lstnumberx43" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">43</span><span id="lstnumberx43.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx43.2" class="ltx_text ltx_font_typewriter">],</span>
</div>
<div id="lstnumberx44" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">44</span><span id="lstnumberx44.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx44.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx44.3" class="ltx_text ltx_lst_keyword ltx_font_typewriter"><span id="lstnumberx44.3.1" class="ltx_text" style="color:#9400D1;">atomic_triples</span></span><span id="lstnumberx44.4" class="ltx_text ltx_font_typewriter">":</span><span id="lstnumberx44.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx44.6" class="ltx_text ltx_font_typewriter">[</span>
</div>
<div id="lstnumberx45" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">45</span><span id="lstnumberx45.1" class="ltx_text ltx_lst_space ltx_font_typewriter">          </span><span id="lstnumberx45.2" class="ltx_text ltx_font_typewriter">"(</span><span id="lstnumberx45.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Lauren</span><span id="lstnumberx45.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx45.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Conrad</span><span id="lstnumberx45.6" class="ltx_text ltx_font_typewriter">;</span><span id="lstnumberx45.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx45.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">was</span><span id="lstnumberx45.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx45.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">never</span><span id="lstnumberx45.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx45.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">romantically</span><span id="lstnumberx45.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx45.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">involved</span><span id="lstnumberx45.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx45.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">with</span><span id="lstnumberx45.17" class="ltx_text ltx_font_typewriter">;</span><span id="lstnumberx45.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx45.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Brody</span><span id="lstnumberx45.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx45.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Jenner</span><span id="lstnumberx45.22" class="ltx_text ltx_font_typewriter">)"</span>
</div>
<div id="lstnumberx46" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">46</span><span id="lstnumberx46.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx46.2" class="ltx_text ltx_font_typewriter">]</span>
</div>
<div id="lstnumberx47" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">47</span><span id="lstnumberx47.1" class="ltx_text ltx_lst_space ltx_font_typewriter">      </span><span id="lstnumberx47.2" class="ltx_text ltx_font_typewriter">}</span>
</div>
<div id="lstnumberx48" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">48</span><span id="lstnumberx48.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx48.2" class="ltx_text ltx_font_typewriter">],</span>
</div>
<div id="lstnumberx49" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">49</span><span id="lstnumberx49.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx49.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx49.3" class="ltx_text ltx_lst_keyword ltx_font_typewriter"><span id="lstnumberx49.3.1" class="ltx_text" style="color:#9400D1;">choice_2_counterfactuals</span></span><span id="lstnumberx49.4" class="ltx_text ltx_font_typewriter">":</span><span id="lstnumberx49.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx49.6" class="ltx_text ltx_font_typewriter">[</span>
</div>
<div id="lstnumberx50" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">50</span><span id="lstnumberx50.1" class="ltx_text ltx_lst_space ltx_font_typewriter">      </span><span id="lstnumberx50.2" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx51" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">51</span><span id="lstnumberx51.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx51.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx51.3" class="ltx_text ltx_lst_keyword ltx_font_typewriter"><span id="lstnumberx51.3.1" class="ltx_text" style="color:#9400D1;">fact</span></span><span id="lstnumberx51.4" class="ltx_text ltx_font_typewriter">":</span><span id="lstnumberx51.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx51.6" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx51.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Heidi</span><span id="lstnumberx51.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx51.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Montag</span><span id="lstnumberx51.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx51.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx51.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx51.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Brody</span><span id="lstnumberx51.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx51.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Jenner</span><span id="lstnumberx51.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx51.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">briefly</span><span id="lstnumberx51.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx51.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">dated</span><span id="lstnumberx51.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx51.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx51.22" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx51.23" class="ltx_text ltx_font_typewriter">2006.",</span>
</div>
<div id="lstnumberx52" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">52</span><span id="lstnumberx52.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx52.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx52.3" class="ltx_text ltx_lst_keyword ltx_font_typewriter"><span id="lstnumberx52.3.1" class="ltx_text" style="color:#9400D1;">atomic_facts</span></span><span id="lstnumberx52.4" class="ltx_text ltx_font_typewriter">":</span><span id="lstnumberx52.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx52.6" class="ltx_text ltx_font_typewriter">[</span>
</div>
<div id="lstnumberx53" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">53</span><span id="lstnumberx53.1" class="ltx_text ltx_lst_space ltx_font_typewriter">          </span><span id="lstnumberx53.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx53.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Heidi</span><span id="lstnumberx53.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx53.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Montag</span><span id="lstnumberx53.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx53.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx53.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx53.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Brody</span><span id="lstnumberx53.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx53.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Jenner</span><span id="lstnumberx53.12" class="ltx_text ltx_font_typewriter">'</span><span id="lstnumberx53.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">s</span><span id="lstnumberx53.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx53.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">dating</span><span id="lstnumberx53.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx53.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">was</span><span id="lstnumberx53.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx53.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">brief</span><span id="lstnumberx53.20" class="ltx_text ltx_font_typewriter">.",</span>
</div>
<div id="lstnumberx54" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">54</span><span id="lstnumberx54.1" class="ltx_text ltx_lst_space ltx_font_typewriter">          </span><span id="lstnumberx54.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx54.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Heidi</span><span id="lstnumberx54.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx54.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Montag</span><span id="lstnumberx54.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx54.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx54.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx54.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Brody</span><span id="lstnumberx54.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx54.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Jenner</span><span id="lstnumberx54.12" class="ltx_text ltx_font_typewriter">'</span><span id="lstnumberx54.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">s</span><span id="lstnumberx54.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx54.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">brief</span><span id="lstnumberx54.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx54.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">dating</span><span id="lstnumberx54.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx54.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">was</span><span id="lstnumberx54.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx54.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx54.22" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx54.23" class="ltx_text ltx_font_typewriter">2006."</span>
</div>
<div id="lstnumberx55" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">55</span><span id="lstnumberx55.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx55.2" class="ltx_text ltx_font_typewriter">],</span>
</div>
<div id="lstnumberx56" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">56</span><span id="lstnumberx56.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx56.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx56.3" class="ltx_text ltx_lst_keyword ltx_font_typewriter"><span id="lstnumberx56.3.1" class="ltx_text" style="color:#9400D1;">atomic_triples</span></span><span id="lstnumberx56.4" class="ltx_text ltx_font_typewriter">":</span><span id="lstnumberx56.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx56.6" class="ltx_text ltx_font_typewriter">[</span>
</div>
<div id="lstnumberx57" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">57</span><span id="lstnumberx57.1" class="ltx_text ltx_lst_space ltx_font_typewriter">          </span><span id="lstnumberx57.2" class="ltx_text ltx_font_typewriter">"(</span><span id="lstnumberx57.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Heidi</span><span id="lstnumberx57.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx57.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Montag</span><span id="lstnumberx57.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx57.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx57.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx57.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Brody</span><span id="lstnumberx57.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx57.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Jenner</span><span id="lstnumberx57.12" class="ltx_text ltx_font_typewriter">'</span><span id="lstnumberx57.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">s</span><span id="lstnumberx57.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx57.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">dating</span><span id="lstnumberx57.16" class="ltx_text ltx_font_typewriter">;</span><span id="lstnumberx57.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx57.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">was</span><span id="lstnumberx57.19" class="ltx_text ltx_font_typewriter">;</span><span id="lstnumberx57.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx57.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">brief</span><span id="lstnumberx57.22" class="ltx_text ltx_font_typewriter">)",</span>
</div>
<div id="lstnumberx58" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">58</span><span id="lstnumberx58.1" class="ltx_text ltx_lst_space ltx_font_typewriter">          </span><span id="lstnumberx58.2" class="ltx_text ltx_font_typewriter">"(</span><span id="lstnumberx58.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Heidi</span><span id="lstnumberx58.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx58.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Montag</span><span id="lstnumberx58.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx58.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx58.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx58.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Brody</span><span id="lstnumberx58.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx58.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Jenner</span><span id="lstnumberx58.12" class="ltx_text ltx_font_typewriter">'</span><span id="lstnumberx58.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">s</span><span id="lstnumberx58.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx58.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">brief</span><span id="lstnumberx58.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx58.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">dating</span><span id="lstnumberx58.18" class="ltx_text ltx_font_typewriter">;</span><span id="lstnumberx58.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx58.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">was</span><span id="lstnumberx58.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx58.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx58.23" class="ltx_text ltx_font_typewriter">;</span><span id="lstnumberx58.24" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx58.25" class="ltx_text ltx_font_typewriter">2006)"</span>
</div>
<div id="lstnumberx59" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">59</span><span id="lstnumberx59.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx59.2" class="ltx_text ltx_font_typewriter">]</span>
</div>
<div id="lstnumberx60" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">60</span><span id="lstnumberx60.1" class="ltx_text ltx_lst_space ltx_font_typewriter">      </span><span id="lstnumberx60.2" class="ltx_text ltx_font_typewriter">}</span>
</div>
<div id="lstnumberx61" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">61</span><span id="lstnumberx61.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx61.2" class="ltx_text ltx_font_typewriter">],</span>
</div>
<div id="lstnumberx62" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">62</span><span id="lstnumberx62.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx62.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx62.3" class="ltx_text ltx_lst_keyword ltx_font_typewriter"><span id="lstnumberx62.3.1" class="ltx_text" style="color:#9400D1;">unchanged_facts</span></span><span id="lstnumberx62.4" class="ltx_text ltx_font_typewriter">":</span><span id="lstnumberx62.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.6" class="ltx_text ltx_font_typewriter">[]</span>
</div>
<div id="lstnumberx63" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">63</span><span id="lstnumberx63.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx63.2" class="ltx_text ltx_font_typewriter">},</span>
</div>
<div id="lstnumberx64" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">64</span><span id="lstnumberx64.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx64.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx64.3" class="ltx_text ltx_lst_keyword ltx_font_typewriter"><span id="lstnumberx64.3.1" class="ltx_text" style="color:#9400D1;">answer_alias</span></span><span id="lstnumberx64.4" class="ltx_text ltx_font_typewriter">":</span><span id="lstnumberx64.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx64.6" class="ltx_text ltx_font_typewriter">[</span>
</div>
<div id="lstnumberx65" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">65</span><span id="lstnumberx65.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx65.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx65.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Lauren</span><span id="lstnumberx65.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx65.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">K</span><span id="lstnumberx65.6" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx65.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx65.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Conrad</span><span id="lstnumberx65.9" class="ltx_text ltx_font_typewriter">",</span>
</div>
<div id="lstnumberx66" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">66</span><span id="lstnumberx66.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx66.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx66.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Lauren</span><span id="lstnumberx66.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx66.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Katherine</span><span id="lstnumberx66.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx66.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Conrad</span><span id="lstnumberx66.8" class="ltx_text ltx_font_typewriter">",</span>
</div>
<div id="lstnumberx67" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">67</span><span id="lstnumberx67.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx67.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx67.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">L</span><span id="lstnumberx67.4" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx67.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">C</span><span id="lstnumberx67.6" class="ltx_text ltx_font_typewriter">."</span>
</div>
<div id="lstnumberx68" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">68</span><span id="lstnumberx68.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx68.2" class="ltx_text ltx_font_typewriter">],</span>
</div>
<div id="lstnumberx69" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">69</span><span id="lstnumberx69.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx69.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx69.3" class="ltx_text ltx_lst_keyword ltx_font_typewriter"><span id="lstnumberx69.3.1" class="ltx_text" style="color:#9400D1;">counterfactual_answer_alias</span></span><span id="lstnumberx69.4" class="ltx_text ltx_font_typewriter">":</span><span id="lstnumberx69.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx69.6" class="ltx_text ltx_font_typewriter">[</span>
</div>
<div id="lstnumberx70" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">70</span><span id="lstnumberx70.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx70.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx70.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Heidi</span><span id="lstnumberx70.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx70.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Pratt</span><span id="lstnumberx70.6" class="ltx_text ltx_font_typewriter">",</span>
</div>
<div id="lstnumberx71" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">71</span><span id="lstnumberx71.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx71.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx71.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Heidi</span><span id="lstnumberx71.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx71.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Blair</span><span id="lstnumberx71.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx71.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Montag</span><span id="lstnumberx71.8" class="ltx_text ltx_font_typewriter">",</span>
</div>
<div id="lstnumberx72" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">72</span><span id="lstnumberx72.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx72.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx72.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Heidi</span><span id="lstnumberx72.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx72.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">B</span><span id="lstnumberx72.6" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx72.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx72.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Montag</span><span id="lstnumberx72.9" class="ltx_text ltx_font_typewriter">"</span>
</div>
<div id="lstnumberx73" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">73</span><span id="lstnumberx73.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx73.2" class="ltx_text ltx_font_typewriter">]</span>
</div>
<div id="lstnumberx74" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">74</span><span id="lstnumberx74.1" class="ltx_text ltx_font_typewriter">}</span>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 7: </span>An example from the ReCoE dataset (superlative).</figcaption>
</figure>
<div id="A1.SS1.p1" class="ltx_para">
<p id="A1.SS1.p1.1" class="ltx_p">The data contract and a specific example from ReCoE is shown in Figure <a href="#A1.F7" title="Figure 7 ‣ A.1 Example Datapoint ‣ Appendix A Appendix ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>.</p>
</div>
</section>
<section id="A1.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.2 </span>Dataset Construction</h3>

<div id="A1.SS2.p1" class="ltx_para">
<p id="A1.SS2.p1.1" class="ltx_p">Our dataset is constructed almost completely automatically. In this subsection, we discuss in detail how the dataset is constructed step by step. This is the running example that we adopt for illustration:
<span id="A1.SS2.p1.1.1" class="ltx_ERROR undefined">\MakeFramed</span><span id="A1.SS2.p1.1.2" class="ltx_ERROR undefined">\FrameRestore</span>
<span id="A1.SS2.p1.1.3" class="ltx_text ltx_font_bold">Question</span>: Which team between the Chicago Blackhawks and Pittsburgh Penguins has won more Stanley Cup championships?
<span id="A1.SS2.p1.1.4" class="ltx_text ltx_font_bold">Answer</span>: Chicago Blackhawks
<span id="A1.SS2.p1.1.5" class="ltx_ERROR undefined">\endMakeFramed</span></p>
</div>
<section id="A1.SS2.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Counterfactual answer generation</h5>

<div id="A1.SS2.SSS0.Px1.p1" class="ltx_para">
<p id="A1.SS2.SSS0.Px1.p1.1" class="ltx_p">Given a question and answer, prompt Claude to generate counterfactual answers. Since the example is a choice question, then the counterfactual answer must be “Pittsburgh Penguins”.
<span id="A1.SS2.SSS0.Px1.p1.1.1" class="ltx_ERROR undefined">\MakeFramed</span><span id="A1.SS2.SSS0.Px1.p1.1.2" class="ltx_ERROR undefined">\FrameRestore</span>
<span id="A1.SS2.SSS0.Px1.p1.1.3" class="ltx_text ltx_font_bold">Counterfactual Answer</span>: Pittsburgh Penguins
<span id="A1.SS2.SSS0.Px1.p1.1.4" class="ltx_ERROR undefined">\endMakeFramed</span>
For questions that are not a yes/no question: if the answer is an entity, then the counterfactual answer will be a similar and comparable entity to the answer; if the answer is an ordered sequence of entities or events, the counterfactual answer will be an order with two entities swapped; if the answer is a number, the counterfactual answer will be a close but different number.</p>
</div>
</section>
<section id="A1.SS2.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Fact generation</h5>

<div id="A1.SS2.SSS0.Px2.p1" class="ltx_para">
<p id="A1.SS2.SSS0.Px2.p1.1" class="ltx_p">For each triplet of (question, answer, counterfactual answer), prompt Claude to generate relevant facts mentioned entities in the answer and counterfactual answer. Prompt details can be found in Appendix <a href="#A1.SS3" title="A.3 Fact Generation ‣ Appendix A Appendix ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.3</span></a>. If the question is yes/no, generate facts on the two entities being compared. In this example, we prompt Claude on the two teams “Chicago Blackhawks” and “Pittsburgh Penguins” on their number of Stanley Cup winnings. For this example, the mentioned entities are Chicago Blackhawks and Pittsburgh Penguins; their corresponding facts generated are presented below:
<span id="A1.SS2.SSS0.Px2.p1.1.1" class="ltx_ERROR undefined">\MakeFramed</span><span id="A1.SS2.SSS0.Px2.p1.1.2" class="ltx_ERROR undefined">\FrameRestore</span>
<span id="A1.SS2.SSS0.Px2.p1.1.3" class="ltx_text ltx_font_bold">Facts
<br class="ltx_break">Chicago Blackhawks</span>: Chicago Blackhawks has won the Stanley Cup Championship six times, in 1930, 1937, 1961, 2010, 2013 and in 2015.
<br class="ltx_break"><span id="A1.SS2.SSS0.Px2.p1.1.4" class="ltx_text ltx_font_bold">Pittsburgh Penguins</span>: The Penguins have won the Stanley Cup five times (1991, 1992, 2009, 2016, and 2017).
<span id="A1.SS2.SSS0.Px2.p1.1.5" class="ltx_ERROR undefined">\endMakeFramed</span></p>
</div>
</section>
<section id="A1.SS2.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Fact verification</h5>

<div id="A1.SS2.SSS0.Px3.p1" class="ltx_para">
<p id="A1.SS2.SSS0.Px3.p1.1" class="ltx_p">Hallucination <cite class="ltx_cite ltx_citemacro_cite">Rawte Vipula (<a href="#bib.bib23" title="" class="ltx_ref">2023</a>)</cite> is a severe problem for large language models. Thus, the facts generated from Claude need further verification with truthful and convincing sources. Towards this end, we leverage the retrieval-augmented method to verify each sentence of the generated facts with the following steps: (1) utilize Google Search API to search relevant Wikipedia pages on the question, answer, counterfactual answer, and each sentence of the generated facts (2) chunk content from all the found Wikipedia pages to paragraphs (3) for each sentence, we leverage Contriever <cite class="ltx_cite ltx_citemacro_cite">Izacard et al. (<a href="#bib.bib13" title="" class="ltx_ref">2021b</a>)</cite> model implemented by Huggingface<span id="footnote4" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">4</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">4</sup><span class="ltx_tag ltx_tag_note">4</span>https://huggingface.co/facebook/contriever</span></span></span> to retrieve the top-5 most relevant paragraphs (4) we prompt Claude using the sentence together with its top-5 most relevant paragraphs to verify whether the sentence is factually correct and if not, modify it based on the retrieved paragraphs. Prompt details can be found in Appendix <a href="#A1.SS4" title="A.4 Fact Verification ‣ Appendix A Appendix ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.4</span></a>.</p>
</div>
<div id="A1.SS2.SSS0.Px3.p2" class="ltx_para">
<p id="A1.SS2.SSS0.Px3.p2.1" class="ltx_p">In this example, the generated sentence for Chicago Blackhawks is wrong in the year of 1930, which should be 1933.
<span id="A1.SS2.SSS0.Px3.p2.1.1" class="ltx_ERROR undefined">\MakeFramed</span><span id="A1.SS2.SSS0.Px3.p2.1.2" class="ltx_ERROR undefined">\FrameRestore</span>
<span id="A1.SS2.SSS0.Px3.p2.1.3" class="ltx_text ltx_font_bold">Facts
<br class="ltx_break">Chicago Blackhawks</span>: Chicago Blackhawks has won the Stanley Cup Championship six times, in <span id="A1.SS2.SSS0.Px3.p2.1.4" class="ltx_text" style="color:#FF0000;">1930</span>, 1937, 1961, 2010, 2013 and in 2015. –&gt; Chicago Blackhawks has won the Stanley Cup Championship six times, in <span id="A1.SS2.SSS0.Px3.p2.1.5" class="ltx_text" style="color:#FF0000;">1933</span>, 1937, 1961, 2010, 2013 and in 2015.
<br class="ltx_break"><span id="A1.SS2.SSS0.Px3.p2.1.6" class="ltx_text ltx_font_bold">Pittsburgh Penguins</span>: The Penguins have won the Stanley Cup five times (1991, 1992, 2009, 2016, and 2017): factually correct
<span id="A1.SS2.SSS0.Px3.p2.1.7" class="ltx_ERROR undefined">\endMakeFramed</span></p>
</div>
<div id="A1.SS2.SSS0.Px3.p3" class="ltx_para">
<p id="A1.SS2.SSS0.Px3.p3.1" class="ltx_p">For reference, we also provide Wikipedia links for each sentence in the generated facts.</p>
</div>
</section>
<section id="A1.SS2.SSS0.Px4" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Datapoint filtering</h5>

<div id="A1.SS2.SSS0.Px4.p1" class="ltx_para">
<p id="A1.SS2.SSS0.Px4.p1.1" class="ltx_p">To guarantee that the provided answer is correct for the question and the counterfactual answer is indeed “counterfactual”, given the verified facts, we prompt Claude to determine whether the facts support the answer and negate the counterfactual answer and then filter out datapoints with the wrong or outdated answer. This step is necessary as FreebaseQA, GrailQA, ComplexWebQuestions are all based on the Freebase knowledge graph which is outdated. Prompt details can be found in Appendix <a href="#A1.SS5" title="A.5 Data Filtering with Entailment Verification ‣ Appendix A Appendix ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.5</span></a>.</p>
</div>
<div id="A1.SS2.SSS0.Px4.p2" class="ltx_para ltx_noindent">
<span id="A1.SS2.SSS0.Px4.p2.1" class="ltx_ERROR undefined">\MakeFramed</span><span id="A1.SS2.SSS0.Px4.p2.2" class="ltx_ERROR undefined">\FrameRestore</span>
<p id="A1.SS2.SSS0.Px4.p2.3" class="ltx_p"><span id="A1.SS2.SSS0.Px4.p2.3.1" class="ltx_text ltx_font_bold">Counterfactual Facts
<br class="ltx_break">Pittsburgh Penguins</span>: Pittsburgh Penguins has won the Stanley Cup Championship six times, in 1933, 1937, 1961, 2010, 2013 and in 2015.
<br class="ltx_break"><span id="A1.SS2.SSS0.Px4.p2.3.2" class="ltx_text ltx_font_bold">Chicago Blackhawks</span>: Chicago Blackhawks have won the Stanley Cup five times (1991, 1992, 2009, 2016, and 2017): factually correct
<span id="A1.SS2.SSS0.Px4.p2.3.3" class="ltx_ERROR undefined">\endMakeFramed</span></p>
</div>
</section>
<section id="A1.SS2.SSS0.Px5" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Counterfactual Fact Generation</h5>

<div id="A1.SS2.SSS0.Px5.p1" class="ltx_para">
<p id="A1.SS2.SSS0.Px5.p1.1" class="ltx_p">To create counterfactual facts, we switch the subjects of the facts. Creating counterfactual facts by swapping subjects of supporting facts can guarantee that the generated counterfactual facts support the counterfactual answer.</p>
</div>
</section>
<section id="A1.SS2.SSS0.Px6" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Datapoints consistency</h5>

<div id="A1.SS2.SSS0.Px6.p1" class="ltx_para">
<p id="A1.SS2.SSS0.Px6.p1.1" class="ltx_p">Notice that the generated counterfactual facts could become contradictory to each other if multiple datapoints happen to involve one same fact to edit. For example, one datapoint requires editing the fact about the Amazon River length to be 4,132 miles:
<span id="A1.SS2.SSS0.Px6.p1.1.1" class="ltx_ERROR undefined">\MakeFramed</span><span id="A1.SS2.SSS0.Px6.p1.1.2" class="ltx_ERROR undefined">\FrameRestore</span>
<span id="A1.SS2.SSS0.Px6.p1.1.3" class="ltx_text ltx_font_bold">Question 1</span>: Which river is longer, the Amazon River or the Nile River?
<br class="ltx_break"><span id="A1.SS2.SSS0.Px6.p1.1.4" class="ltx_text ltx_font_bold">Counterfactual Answer</span>: the Amazon River
<br class="ltx_break"><span id="A1.SS2.SSS0.Px6.p1.1.5" class="ltx_text ltx_font_bold">Counterfactual Facts</span>: 
<br class="ltx_break"><span id="A1.SS2.SSS0.Px6.p1.1.6" class="ltx_text" style="color:#FF0000;">The Amazon River has length of 4,132 miles.</span> 
<br class="ltx_break">The Nile River has length of 3,977 miles.
<span id="A1.SS2.SSS0.Px6.p1.1.7" class="ltx_ERROR undefined">\endMakeFramed</span>
Another datapoint requires editing the fact about the Amazon River length to be 3,395 miles:
<span id="A1.SS2.SSS0.Px6.p1.1.8" class="ltx_ERROR undefined">\MakeFramed</span><span id="A1.SS2.SSS0.Px6.p1.1.9" class="ltx_ERROR undefined">\FrameRestore</span>
<span id="A1.SS2.SSS0.Px6.p1.1.10" class="ltx_text ltx_font_bold">Question 2</span>: Which river is longer, the Amazon River or the Yellow River?
<br class="ltx_break"><span id="A1.SS2.SSS0.Px6.p1.1.11" class="ltx_text ltx_font_bold">Counterfactual Answer</span>: the Yellow River
<br class="ltx_break"><span id="A1.SS2.SSS0.Px6.p1.1.12" class="ltx_text ltx_font_bold">Counterfactual Facts</span>: 
<br class="ltx_break"><span id="A1.SS2.SSS0.Px6.p1.1.13" class="ltx_text" style="color:#FF0000;">The Amazon River has length of 3,395 miles.</span> 
<br class="ltx_break">The Yellow River has length of 3,977 miles.
<span id="A1.SS2.SSS0.Px6.p1.1.14" class="ltx_ERROR undefined">\endMakeFramed</span>
This would be problematic for massive editing. Thus to avoid editing facts to be contradictory to each other, for datapoints with contradictory counterfactual editing facts, we randomly retain one of them and remove the others to guarantee consistency among each sub-dataset.</p>
</div>
</section>
<section id="A1.SS2.SSS0.Px7" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Atomic format of facts</h5>

<div id="A1.SS2.SSS0.Px7.p1" class="ltx_para">
<p id="A1.SS2.SSS0.Px7.p1.1" class="ltx_p">As multiple editing methods require a (subject, relation, object) format for editing knowledge, in order to facilitate the application of the dataset, we transform each sentence of the facts and counterfactual facts into atomic facts and then atomic triplets. Prompt details can be found in Appendix <a href="#A1.SS6" title="A.6 Atomic Facts Generation ‣ Appendix A Appendix ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.6</span></a>.</p>
</div>
<div id="A1.SS2.SSS0.Px7.p2" class="ltx_para">
<p id="A1.SS2.SSS0.Px7.p2.1" class="ltx_p">Let’s take the sentence “Pittsburgh Penguins has won the Stanley Cup Championship six times, in 1933, 1937, 1961, 2010, 2013 and in 2015” as an example. The atomic facts are sub-sequences of the sentence. Notice that there is no unique way to break a sentence into atomic facts:
<span id="A1.SS2.SSS0.Px7.p2.1.1" class="ltx_ERROR undefined">\MakeFramed</span><span id="A1.SS2.SSS0.Px7.p2.1.2" class="ltx_ERROR undefined">\FrameRestore</span>
<span id="A1.SS2.SSS0.Px7.p2.1.3" class="ltx_text ltx_font_bold">Atomic Facts
<br class="ltx_break">Atomic fact 1</span>: Pittsburgh Penguins has won the Stanley Cup Championship six times.
<br class="ltx_break"><span id="A1.SS2.SSS0.Px7.p2.1.4" class="ltx_text ltx_font_bold">Atomic fact 2</span>: Pittsburgh Penguins won the Stanley Cup in 1933, 1937, 1961, 2010, 2013, and 2015.
<span id="A1.SS2.SSS0.Px7.p2.1.5" class="ltx_ERROR undefined">\endMakeFramed</span></p>
</div>
<div id="A1.SS2.SSS0.Px7.p3" class="ltx_para ltx_noindent">
<span id="A1.SS2.SSS0.Px7.p3.1" class="ltx_ERROR undefined">\MakeFramed</span><span id="A1.SS2.SSS0.Px7.p3.2" class="ltx_ERROR undefined">\FrameRestore</span>
<p id="A1.SS2.SSS0.Px7.p3.3" class="ltx_p"><span id="A1.SS2.SSS0.Px7.p3.3.1" class="ltx_text ltx_font_bold">Atomic Triples
<br class="ltx_break">Atomic triplet 1</span>: (Pittsburgh Penguins, has won the Stanley Cup Championship, six times)
<br class="ltx_break"><span id="A1.SS2.SSS0.Px7.p3.3.2" class="ltx_text ltx_font_bold">Atomic triplet 2</span>: (Pittsburgh Penguins, won the Stanley Cup in, 1933, 1937, 1961, 2010, 2013, and 2015)
<span id="A1.SS2.SSS0.Px7.p3.3.3" class="ltx_ERROR undefined">\endMakeFramed</span></p>
</div>
</section>
<section id="A1.SS2.SSS0.Px8" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Alias generation</h5>

<div id="A1.SS2.SSS0.Px8.p1" class="ltx_para">
<p id="A1.SS2.SSS0.Px8.p1.1" class="ltx_p">In evaluation time, we resort to Exact Match on the answer or the counterfactual answer to evaluate models before editing and after editing. Since the model may not generate the exact surface string provided, providing aliases for the answer and the counterfactual answer is necessary to accurately reflect the model capacity as well as the performance of editing methods. Prompt details can be found in Appendix <a href="#A1.SS7" title="A.7 Alias Generation ‣ Appendix A Appendix ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.7</span></a>. In this example, the aliases are:
<span id="A1.SS2.SSS0.Px8.p1.1.1" class="ltx_ERROR undefined">\MakeFramed</span><span id="A1.SS2.SSS0.Px8.p1.1.2" class="ltx_ERROR undefined">\FrameRestore</span>
<span id="A1.SS2.SSS0.Px8.p1.1.3" class="ltx_text ltx_font_bold">answer alias</span>: Blackhawk Division, Hawks
<br class="ltx_break"><span id="A1.SS2.SSS0.Px8.p1.1.4" class="ltx_text ltx_font_bold">counterfactual answer alias</span>: the Pens, Pens
<span id="A1.SS2.SSS0.Px8.p1.1.5" class="ltx_ERROR undefined">\endMakeFramed</span></p>
</div>
</section>
<section id="A1.SS2.SSS0.Px9" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Summary</h5>

<div id="A1.SS2.SSS0.Px9.p1" class="ltx_para">
<p id="A1.SS2.SSS0.Px9.p1.2" class="ltx_p">This appendix section presents the basic dataset statistics and the dataset construction process. In each sub-dataset, the datapoint is an n-ary tuple consisting of (question, answer, counterfactual answer, facts, counterfactual facts, answer alias, counterfactual answer alias), where facts and counterfactual facts are lists of dictionaries with 4 keys: sentence, links, atomic<math id="A1.SS2.SSS0.Px9.p1.1.m1.1" class="ltx_Math" alttext="\_" display="inline"><semantics id="A1.SS2.SSS0.Px9.p1.1.m1.1a"><mi mathvariant="normal" id="A1.SS2.SSS0.Px9.p1.1.m1.1.1" xref="A1.SS2.SSS0.Px9.p1.1.m1.1.1.cmml">_</mi><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px9.p1.1.m1.1b"><ci id="A1.SS2.SSS0.Px9.p1.1.m1.1.1.cmml" xref="A1.SS2.SSS0.Px9.p1.1.m1.1.1">_</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px9.p1.1.m1.1c">\_</annotation></semantics></math>sentences, atomic<math id="A1.SS2.SSS0.Px9.p1.2.m2.1" class="ltx_Math" alttext="\_" display="inline"><semantics id="A1.SS2.SSS0.Px9.p1.2.m2.1a"><mi mathvariant="normal" id="A1.SS2.SSS0.Px9.p1.2.m2.1.1" xref="A1.SS2.SSS0.Px9.p1.2.m2.1.1.cmml">_</mi><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px9.p1.2.m2.1b"><ci id="A1.SS2.SSS0.Px9.p1.2.m2.1.1.cmml" xref="A1.SS2.SSS0.Px9.p1.2.m2.1.1">_</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px9.p1.2.m2.1c">\_</annotation></semantics></math>triplets.</p>
</div>
</section>
</section>
<section id="A1.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.3 </span>Fact Generation</h3>

<div id="A1.SS3.p1" class="ltx_para">
<p id="A1.SS3.p1.1" class="ltx_p">To prompt Claude to generate facts on relevant entities or events, we adopt few shot learning prompting strategy with multiple examples. To save space, we only use 1 example for illustration. See detailed prompt in Figure <a href="#A1.F8" title="Figure 8 ‣ A.3 Fact Generation ‣ Appendix A Appendix ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a>.</p>
</div>
<figure id="A1.F8" class="ltx_figure"><svg id="A1.F8.pic1" class="ltx_picture" height="289.96" overflow="visible" version="1.1" width="600"><g transform="translate(0,289.96) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#000000" fill-opacity="1.0"><path d="M 0 3.15 L 0 286.81 C 0 288.55 1.41 289.96 3.15 289.96 L 596.85 289.96 C 598.59 289.96 600 288.55 600 286.81 L 600 3.15 C 600 1.41 598.59 0 596.85 0 L 3.15 0 C 1.41 0 0 1.41 0 3.15 Z" style="stroke:none"></path></g><g fill="#FFFFFF" fill-opacity="1.0"><path d="M 1.18 3.15 L 1.18 267.58 L 598.82 267.58 L 598.82 3.15 C 598.82 2.06 597.94 1.18 596.85 1.18 L 3.15 1.18 C 2.06 1.18 1.18 2.06 1.18 3.15 Z" style="stroke:none"></path></g><g fill="#D9D9D9" fill-opacity="1.0"><path d="M 1.18 268.76 L 1.18 286.81 C 1.18 287.9 2.06 288.78 3.15 288.78 L 596.85 288.78 C 597.94 288.78 598.82 287.9 598.82 286.81 L 598.82 268.76 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 9.06 272.69)"><foreignObject width="581.89" height="12.15" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F8.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:420.5pt;">
<span id="A1.F8.pic1.1.1.1.1.1.1" class="ltx_p"><span id="A1.F8.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Fact Generation Prompt</span></span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 9.06 5.12)"><foreignObject width="581.89" height="258.52" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F8.pic1.2.2.2.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:420.5pt;">
<span id="A1.F8.pic1.2.2.2.1.1.1" class="ltx_p"><span id="A1.F8.pic1.2.2.2.1.1.1.1" class="ltx_text ltx_font_typewriter ltx_font_bold">Human:
<br class="ltx_break"></span></span>
<span id="A1.F8.pic1.2.2.2.1.1.2" class="ltx_p"><span id="A1.F8.pic1.2.2.2.1.1.2.1" class="ltx_text ltx_font_typewriter">You are given a yes-no question. Your task is to answer the question by explicitly checking the relevant facts of the entities/events being compared in the question.
<br class="ltx_break"></span></span>
<span id="A1.F8.pic1.2.2.2.1.1.3" class="ltx_p"><span id="A1.F8.pic1.2.2.2.1.1.3.1" class="ltx_text ltx_font_typewriter">Here is one example:
<br class="ltx_break"></span></span>
<span id="A1.F8.pic1.2.2.2.1.1.4" class="ltx_p"><span id="A1.F8.pic1.2.2.2.1.1.4.1" class="ltx_text ltx_font_typewriter">Question: Is Mexico City more populous than Amsterdam?
<br class="ltx_break"></span></span>
<span id="A1.F8.pic1.2.2.2.1.1.5" class="ltx_p"><span id="A1.F8.pic1.2.2.2.1.1.5.1" class="ltx_text ltx_font_typewriter">Relevant facts related to &lt;entity1&gt;Mexico City&lt;/entity1&gt;:
<br class="ltx_break">&lt;facts_for_entity1&gt;
<br class="ltx_break">&lt;fact1&gt;Mexico City has 22.2 million&lt;/fact1&gt;
<br class="ltx_break">&lt;/facts_for_entity1&gt;
<br class="ltx_break"></span></span>
<span id="A1.F8.pic1.2.2.2.1.1.6" class="ltx_p"><span id="A1.F8.pic1.2.2.2.1.1.6.1" class="ltx_text ltx_font_typewriter">Relevant facts related to &lt;entity2&gt;Amsterdam&lt;/entity2&gt;:
<br class="ltx_break">&lt;facts_for_entity2&gt;
<br class="ltx_break">&lt;fact1&gt;Amsterdam has 821,752 in population &lt;/fact1&gt;
<br class="ltx_break">&lt;/facts_for_entity2&gt;
<br class="ltx_break"></span></span>
<span id="A1.F8.pic1.2.2.2.1.1.7" class="ltx_p"><span id="A1.F8.pic1.2.2.2.1.1.7.1" class="ltx_text ltx_font_typewriter">Now generate relevant fact for the following question.
<br class="ltx_break">{{question}}
<br class="ltx_break"></span></span>
<span id="A1.F8.pic1.2.2.2.1.1.8" class="ltx_p"><span id="A1.F8.pic1.2.2.2.1.1.8.1" class="ltx_text ltx_font_typewriter ltx_font_bold">Assistant:<span id="A1.F8.pic1.2.2.2.1.1.8.1.1" class="ltx_text ltx_font_medium"></span></span></span>
</span></foreignObject></g></g></svg>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 8: </span>Fact generation prompt</figcaption>
</figure>
</section>
<section id="A1.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.4 </span>Fact Verification</h3>

<div id="A1.SS4.p1" class="ltx_para">
<p id="A1.SS4.p1.1" class="ltx_p">This prompt is used to do factuality verification for generated facts by Claude based on retrieved Wikipedia paragraphs. We adopt few-shot prompting. In the below example, we only present one example in the prompt for demonstration purpose. See detailed prompt in Figure <a href="#A1.F9" title="Figure 9 ‣ A.4 Fact Verification ‣ Appendix A Appendix ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>.</p>
</div>
<figure id="A1.F9" class="ltx_figure"><svg id="A1.F9.pic1" class="ltx_picture" height="539.18" overflow="visible" version="1.1" width="600"><g transform="translate(0,539.18) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#000000" fill-opacity="1.0"><path d="M 0 3.15 L 0 536.03 C 0 537.77 1.41 539.18 3.15 539.18 L 596.85 539.18 C 598.59 539.18 600 537.77 600 536.03 L 600 3.15 C 600 1.41 598.59 0 596.85 0 L 3.15 0 C 1.41 0 0 1.41 0 3.15 Z" style="stroke:none"></path></g><g fill="#FFFFFF" fill-opacity="1.0"><path d="M 1.18 3.15 L 1.18 516.64 L 598.82 516.64 L 598.82 3.15 C 598.82 2.06 597.94 1.18 596.85 1.18 L 3.15 1.18 C 2.06 1.18 1.18 2.06 1.18 3.15 Z" style="stroke:none"></path></g><g fill="#D9D9D9" fill-opacity="1.0"><path d="M 1.18 517.82 L 1.18 536.03 C 1.18 537.12 2.06 538 3.15 538 L 596.85 538 C 597.94 538 598.82 537.12 598.82 536.03 L 598.82 517.82 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 9.06 521.76)"><foreignObject width="581.89" height="12.3" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F9.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:420.5pt;">
<span id="A1.F9.pic1.1.1.1.1.1.1" class="ltx_p"><span id="A1.F9.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Fact Verification Prompt</span></span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 9.06 5.12)"><foreignObject width="581.89" height="507.59" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F9.pic1.2.2.2.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:420.5pt;">
<span id="A1.F9.pic1.2.2.2.1.1.1" class="ltx_p"><span id="A1.F9.pic1.2.2.2.1.1.1.1" class="ltx_text ltx_font_typewriter ltx_font_bold">Human:
<br class="ltx_break"></span></span>
<span id="A1.F9.pic1.2.2.2.1.1.2" class="ltx_p"><span id="A1.F9.pic1.2.2.2.1.1.2.1" class="ltx_text ltx_font_typewriter">You are given a sentence and you need to check whether it is consistent with the provided Wikipedia paragraphs.
<br class="ltx_break"></span></span>
<span id="A1.F9.pic1.2.2.2.1.1.3" class="ltx_p"><span id="A1.F9.pic1.2.2.2.1.1.3.1" class="ltx_text ltx_font_typewriter">Your task is to judge whether this sentence is factually consistent and potentially rewrite it.
<br class="ltx_break"></span></span>
<span id="A1.F9.pic1.2.2.2.1.1.4" class="ltx_p"><span id="A1.F9.pic1.2.2.2.1.1.4.1" class="ltx_text ltx_font_typewriter">If inconsistent: rewrite the sentence by changing it minimally.
<br class="ltx_break">If consistent: leave it unchanged.
<br class="ltx_break"></span></span>
<span id="A1.F9.pic1.2.2.2.1.1.5" class="ltx_p"><span id="A1.F9.pic1.2.2.2.1.1.5.1" class="ltx_text ltx_font_typewriter">Here are four examples:
<br class="ltx_break"></span></span>
<span id="A1.F9.pic1.2.2.2.1.1.6" class="ltx_p"><span id="A1.F9.pic1.2.2.2.1.1.6.1" class="ltx_text ltx_font_typewriter">&lt;example&gt;
<br class="ltx_break">H:
<br class="ltx_break">Sentence: The Thermosphere, Ionosphere, Mesosphere Energetics and Dynamics (TIMED) satellite is a NASA mission.
<br class="ltx_break"></span></span>
<span id="A1.F9.pic1.2.2.2.1.1.7" class="ltx_p"><span id="A1.F9.pic1.2.2.2.1.1.7.1" class="ltx_text ltx_font_typewriter">Wikipedia paragraph: The TIMED (Thermosphere, Ionosphere, Mesosphere, Energetics and Dynamics) mission is dedicated to study the influences energetics and dynamics of the Sun and humans on the least explored and understood region of Earth’s atmosphere: the Mesosphere and Lower Thermosphere / Ionosphere (MLTI). The mission was launched from Vandenberg Air Force Base in California on 7 December 2001 aboard a Delta II rocket launch vehicle. The project is sponsored and managed by NASA, while the spacecraft was designed and assembled by the Applied Physics Laboratory at Johns Hopkins University. The mission has been extended several times, and has now collected data over an entire solar cycle, which helps in its goal to differentiate the Sun’s effects on the atmosphere from other effects. TIMED Was Launched Alongside Jason-1.
<br class="ltx_break"></span></span>
<span id="A1.F9.pic1.2.2.2.1.1.8" class="ltx_p"><span id="A1.F9.pic1.2.2.2.1.1.8.1" class="ltx_text ltx_font_typewriter">A:
<br class="ltx_break">&lt;response&gt;
<br class="ltx_break">Based on the paragraph, the sentence is factually consistent.
<br class="ltx_break">&lt;factuality&gt;Consistent&lt;/factuality&gt;
<br class="ltx_break">Since it is consistent, we do not need to modify it.
<br class="ltx_break">&lt;/response&gt;
<br class="ltx_break">&lt;/example&gt;
<br class="ltx_break"></span></span>
<span id="A1.F9.pic1.2.2.2.1.1.9" class="ltx_p"><span id="A1.F9.pic1.2.2.2.1.1.9.1" class="ltx_text ltx_font_typewriter">Now, check the following sentence:
<br class="ltx_break">sentence: {{sentence}}
<br class="ltx_break"></span></span>
<span id="A1.F9.pic1.2.2.2.1.1.10" class="ltx_p"><span id="A1.F9.pic1.2.2.2.1.1.10.1" class="ltx_text ltx_font_typewriter">Wikipedia paragraph: {{retrieved paragraphs}}
<br class="ltx_break"></span></span>
<span id="A1.F9.pic1.2.2.2.1.1.11" class="ltx_p"><span id="A1.F9.pic1.2.2.2.1.1.11.1" class="ltx_text ltx_font_typewriter ltx_font_bold">Assistant:<span id="A1.F9.pic1.2.2.2.1.1.11.1.1" class="ltx_text ltx_font_medium"></span></span></span>
</span></foreignObject></g></g></svg>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 9: </span>Fact verification prompt. Only 1-shot example is shown for brevity.</figcaption>
</figure>
</section>
<section id="A1.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.5 </span>Data Filtering with Entailment Verification</h3>

<div id="A1.SS5.p1" class="ltx_para">
<p id="A1.SS5.p1.1" class="ltx_p">This prompt is used to check whether verified facts support the answer to the question and invalidate the counterfactual answer to the question. We also adopt few-shot prompting and here, we present only one example in the prompt for demonstration purposes. See detailed prompt in Figure <a href="#A1.F10" title="Figure 10 ‣ A.5 Data Filtering with Entailment Verification ‣ Appendix A Appendix ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">10</span></a>.</p>
</div>
<figure id="A1.F10" class="ltx_figure"><svg id="A1.F10.pic1" class="ltx_picture" height="505.97" overflow="visible" version="1.1" width="600"><g transform="translate(0,505.97) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#000000" fill-opacity="1.0"><path d="M 0 3.15 L 0 502.82 C 0 504.56 1.41 505.97 3.15 505.97 L 596.85 505.97 C 598.59 505.97 600 504.56 600 502.82 L 600 3.15 C 600 1.41 598.59 0 596.85 0 L 3.15 0 C 1.41 0 0 1.41 0 3.15 Z" style="stroke:none"></path></g><g fill="#FFFFFF" fill-opacity="1.0"><path d="M 1.18 3.15 L 1.18 483.43 L 598.82 483.43 L 598.82 3.15 C 598.82 2.06 597.94 1.18 596.85 1.18 L 3.15 1.18 C 2.06 1.18 1.18 2.06 1.18 3.15 Z" style="stroke:none"></path></g><g fill="#D9D9D9" fill-opacity="1.0"><path d="M 1.18 484.61 L 1.18 502.82 C 1.18 503.91 2.06 504.79 3.15 504.79 L 596.85 504.79 C 597.94 504.79 598.82 503.91 598.82 502.82 L 598.82 484.61 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 9.06 488.55)"><foreignObject width="581.89" height="12.3" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F10.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:420.5pt;">
<span id="A1.F10.pic1.1.1.1.1.1.1" class="ltx_p"><span id="A1.F10.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Data Filtering with Entailment Verification Prompt</span></span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 9.06 5.12)"><foreignObject width="581.89" height="474.38" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F10.pic1.2.2.2.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:420.5pt;">
<span id="A1.F10.pic1.2.2.2.1.1.1" class="ltx_p"><span id="A1.F10.pic1.2.2.2.1.1.1.1" class="ltx_text ltx_font_typewriter ltx_font_bold">Human:
<br class="ltx_break"></span></span>
<span id="A1.F10.pic1.2.2.2.1.1.2" class="ltx_p"><span id="A1.F10.pic1.2.2.2.1.1.2.1" class="ltx_text ltx_font_typewriter">You need to check whether given facts about multiple entities are consistent with the sentence.
<br class="ltx_break">Here are a few examples:
<br class="ltx_break"></span></span>
<span id="A1.F10.pic1.2.2.2.1.1.3" class="ltx_p"><span id="A1.F10.pic1.2.2.2.1.1.3.1" class="ltx_text ltx_font_typewriter">&lt;example&gt;
<br class="ltx_break">H:
<br class="ltx_break">&lt;question&gt;Question: Who is the tallest guitarist?&lt;/question&gt;
<br class="ltx_break">&lt;answer&gt;Marc Colombo&lt;/answer&gt;
<br class="ltx_break">&lt;fact&gt;
<br class="ltx_break">Facts:
<br class="ltx_break"></span></span>
<span id="A1.F10.pic1.2.2.2.1.1.4" class="ltx_p"><span id="A1.F10.pic1.2.2.2.1.1.4.1" class="ltx_text ltx_font_typewriter">Marc Colombo: Marc Colombo is an American football player, not a guitarist.
<br class="ltx_break"></span></span>
<span id="A1.F10.pic1.2.2.2.1.1.5" class="ltx_p"><span id="A1.F10.pic1.2.2.2.1.1.5.1" class="ltx_text ltx_font_typewriter">Jimmy Page: Jimmy Page is an English musician who gained international fame for his work in the rock band Led Zeppelin. Jimmy Page has a height of 1.82 m or 5 feet 11.5 inches.
<br class="ltx_break"></span></span>
<span id="A1.F10.pic1.2.2.2.1.1.6" class="ltx_p"><span id="A1.F10.pic1.2.2.2.1.1.6.1" class="ltx_text ltx_font_typewriter">&lt;/fact&gt;
<br class="ltx_break"></span></span>
<span id="A1.F10.pic1.2.2.2.1.1.7" class="ltx_p"><span id="A1.F10.pic1.2.2.2.1.1.7.1" class="ltx_text ltx_font_typewriter">A:
<br class="ltx_break">&lt;response&gt;
<br class="ltx_break">The given facts state that Marc Colombo is an American football player, thus he cannot be the tallest guitarist. Thus these facts do not support the given answer to the question.
<br class="ltx_break">&lt;consistency&gt;No&lt;/consistency&gt;
<br class="ltx_break">&lt;/response&gt;
<br class="ltx_break">&lt;/example&gt;
<br class="ltx_break"></span></span>
<span id="A1.F10.pic1.2.2.2.1.1.8" class="ltx_p"><span id="A1.F10.pic1.2.2.2.1.1.8.1" class="ltx_text ltx_font_typewriter">Here is a new pair of question-answer and facts. Please decide whether the given facts support the provided sentence:
<br class="ltx_break">&lt;question&gt;Question: {{question}}&lt;/question&gt;
<br class="ltx_break">&lt;answer&gt;Answer: {{answer}}&lt;/answer&gt;
<br class="ltx_break">&lt;fact&gt;
<br class="ltx_break">Facts:
<br class="ltx_break">{{verified facts}}
<br class="ltx_break">&lt;/fact&gt;
<br class="ltx_break"></span></span>
<span id="A1.F10.pic1.2.2.2.1.1.9" class="ltx_p"><span id="A1.F10.pic1.2.2.2.1.1.9.1" class="ltx_text ltx_font_typewriter ltx_font_bold">Assistant:<span id="A1.F10.pic1.2.2.2.1.1.9.1.1" class="ltx_text ltx_font_medium"></span></span></span>
</span></foreignObject></g></g></svg>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 10: </span>Data filtering with entailment verification prompt</figcaption>
</figure>
</section>
<section id="A1.SS6" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.6 </span>Atomic Facts Generation</h3>

<div id="A1.SS6.p1" class="ltx_para">
<p id="A1.SS6.p1.1" class="ltx_p">This prompt is used to break down sentences in the facts and counterfactual facts into atomic facts and atomic triplets. We also adopt few-shot prompting strategry and only show one example for demonstration purpose. See detailed prompt in Figure.<a href="#A1.F11" title="Figure 11 ‣ A.6 Atomic Facts Generation ‣ Appendix A Appendix ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a>.</p>
</div>
<figure id="A1.F11" class="ltx_figure"><svg id="A1.F11.pic1" class="ltx_picture" height="638.65" overflow="visible" version="1.1" width="600"><g transform="translate(0,638.65) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#000000" fill-opacity="1.0"><path d="M 0 3.15 L 0 635.5 C 0 637.24 1.41 638.65 3.15 638.65 L 596.85 638.65 C 598.59 638.65 600 637.24 600 635.5 L 600 3.15 C 600 1.41 598.59 0 596.85 0 L 3.15 0 C 1.41 0 0 1.41 0 3.15 Z" style="stroke:none"></path></g><g fill="#FFFFFF" fill-opacity="1.0"><path d="M 1.18 3.15 L 1.18 616.27 L 598.82 616.27 L 598.82 3.15 C 598.82 2.06 597.94 1.18 596.85 1.18 L 3.15 1.18 C 2.06 1.18 1.18 2.06 1.18 3.15 Z" style="stroke:none"></path></g><g fill="#D9D9D9" fill-opacity="1.0"><path d="M 1.18 617.45 L 1.18 635.5 C 1.18 636.59 2.06 637.47 3.15 637.47 L 596.85 637.47 C 597.94 637.47 598.82 636.59 598.82 635.5 L 598.82 617.45 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 9.06 621.39)"><foreignObject width="581.89" height="12.15" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F11.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:420.5pt;">
<span id="A1.F11.pic1.1.1.1.1.1.1" class="ltx_p"><span id="A1.F11.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Atomic Facts Generation Prompt</span></span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 9.06 5.12)"><foreignObject width="581.89" height="607.21" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F11.pic1.2.2.2.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:420.5pt;">
<span id="A1.F11.pic1.2.2.2.1.1.1" class="ltx_p"><span id="A1.F11.pic1.2.2.2.1.1.1.1" class="ltx_text ltx_font_typewriter ltx_font_bold">Human:
<br class="ltx_break"></span></span>
<span id="A1.F11.pic1.2.2.2.1.1.2" class="ltx_p"><span id="A1.F11.pic1.2.2.2.1.1.2.1" class="ltx_text ltx_font_typewriter">Given several sentences, break each of them into atomic facts with salient subject, relation, object. After splitting into atomic facts, we also rewrite the atomic fact to a (subject; relation; object) triple.
<br class="ltx_break">Here are some examples:
<br class="ltx_break"></span></span>
<span id="A1.F11.pic1.2.2.2.1.1.3" class="ltx_p"><span id="A1.F11.pic1.2.2.2.1.1.3.1" class="ltx_text ltx_font_typewriter">&lt;example&gt;
<br class="ltx_break">H:
<br class="ltx_break">Sentence 1: The Lunar Atmosphere and Dust Environment Explorer (LADEE) was a NASA lunar exploration and technology demonstration mission.
<br class="ltx_break"></span></span>
<span id="A1.F11.pic1.2.2.2.1.1.4" class="ltx_p"><span id="A1.F11.pic1.2.2.2.1.1.4.1" class="ltx_text ltx_font_typewriter">A:
<br class="ltx_break">&lt;response&gt;
<br class="ltx_break">For the first sentence: 
<br class="ltx_break"></span></span>
<span id="A1.F11.pic1.2.2.2.1.1.5" class="ltx_p"><span id="A1.F11.pic1.2.2.2.1.1.5.1" class="ltx_text ltx_font_typewriter">This part of sentence "The Lunar Atmosphere and Dust Environment Explorer (LADEE) was a NASA lunar exploration" can form one atomic fact:
<br class="ltx_break"></span></span>
<span id="A1.F11.pic1.2.2.2.1.1.6" class="ltx_p"><span id="A1.F11.pic1.2.2.2.1.1.6.1" class="ltx_text ltx_font_typewriter">The subject is "The Lunar Atmosphere and Dust Environment Explorer (LADEE)";
<br class="ltx_break">The relation is simply "was";
<br class="ltx_break">The object is "a NASA lunar exploration".
<br class="ltx_break">&lt;sentence1_fact1&gt;The Lunar Atmosphere and Dust Environment Explorer (LADEE) was a NASA lunar exploration&lt;/sentence1_fact1&gt;
<br class="ltx_break">Into triple:
<br class="ltx_break">&lt;sentence1_triple1&gt;(The Lunar Atmosphere and Dust Environment Explorer (LADEE), was, a NASA lunar exploration)&lt;/sentence1_triple1&gt;
<br class="ltx_break"></span></span>
<span id="A1.F11.pic1.2.2.2.1.1.7" class="ltx_p"><span id="A1.F11.pic1.2.2.2.1.1.7.1" class="ltx_text ltx_font_typewriter">This part of sentence "The Lunar Atmosphere and Dust Environment Explorer (LADEE) was technology demonstration mission" can form another fact:
<br class="ltx_break">The subject is "The Lunar Atmosphere and Dust Environment Explorer (LADEE)";
<br class="ltx_break">The relation is simply "was";
<br class="ltx_break">The object is "technology demonstration mission".
<br class="ltx_break">&lt;sentence1_fact2&gt;The Lunar Atmosphere and Dust Environment Explorer (LADEE) was technology demonstration mission&lt;/sentence1_fact2&gt;
<br class="ltx_break">Into triple:
<br class="ltx_break">&lt;sentence1_triple2&gt;(The Lunar Atmosphere and Dust Environment Explorer (LADEE); was; technology demonstration mission)&lt;/sentence1_triple2&gt;
<br class="ltx_break">&lt;/response&gt;
<br class="ltx_break">&lt;/example&gt;
<br class="ltx_break"></span></span>
<span id="A1.F11.pic1.2.2.2.1.1.8" class="ltx_p"><span id="A1.F11.pic1.2.2.2.1.1.8.1" class="ltx_text ltx_font_typewriter">Here is a new list of sentences. Please break each of them down into several facts as above.
<br class="ltx_break">{{sentence}}
<br class="ltx_break"></span></span>
<span id="A1.F11.pic1.2.2.2.1.1.9" class="ltx_p"><span id="A1.F11.pic1.2.2.2.1.1.9.1" class="ltx_text ltx_font_typewriter ltx_font_bold">Assistant:<span id="A1.F11.pic1.2.2.2.1.1.9.1.1" class="ltx_text ltx_font_medium"></span></span></span>
</span></foreignObject></g></g></svg>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 11: </span>Atomic fact breaking-down prompt</figcaption>
</figure>
</section>
<section id="A1.SS7" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.7 </span>Alias Generation</h3>

<div id="A1.SS7.p1" class="ltx_para">
<p id="A1.SS7.p1.1" class="ltx_p">This prompt is used to generate alias for answer and counterfacutal answer. We also adopt few-shot prompting strategry and only show one example for demonstration purpose. See detailed prompt in Figure <a href="#A1.F12" title="Figure 12 ‣ A.7 Alias Generation ‣ Appendix A Appendix ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">12</span></a>.</p>
</div>
<figure id="A1.F12" class="ltx_figure"><svg id="A1.F12.pic1" class="ltx_picture" height="472.76" overflow="visible" version="1.1" width="600"><g transform="translate(0,472.76) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#000000" fill-opacity="1.0"><path d="M 0 3.15 L 0 469.61 C 0 471.35 1.41 472.76 3.15 472.76 L 596.85 472.76 C 598.59 472.76 600 471.35 600 469.61 L 600 3.15 C 600 1.41 598.59 0 596.85 0 L 3.15 0 C 1.41 0 0 1.41 0 3.15 Z" style="stroke:none"></path></g><g fill="#FFFFFF" fill-opacity="1.0"><path d="M 1.18 3.15 L 1.18 450.22 L 598.82 450.22 L 598.82 3.15 C 598.82 2.06 597.94 1.18 596.85 1.18 L 3.15 1.18 C 2.06 1.18 1.18 2.06 1.18 3.15 Z" style="stroke:none"></path></g><g fill="#D9D9D9" fill-opacity="1.0"><path d="M 1.18 451.41 L 1.18 469.61 C 1.18 470.7 2.06 471.58 3.15 471.58 L 596.85 471.58 C 597.94 471.58 598.82 470.7 598.82 469.61 L 598.82 451.41 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 9.06 455.34)"><foreignObject width="581.89" height="12.3" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F12.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:420.5pt;">
<span id="A1.F12.pic1.1.1.1.1.1.1" class="ltx_p"><span id="A1.F12.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Alias Generation Prompt</span></span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 9.06 5.12)"><foreignObject width="581.89" height="441.17" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F12.pic1.2.2.2.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:420.5pt;">
<span id="A1.F12.pic1.2.2.2.1.1.1" class="ltx_p"><span id="A1.F12.pic1.2.2.2.1.1.1.1" class="ltx_text ltx_font_typewriter ltx_font_bold">Human<span id="A1.F12.pic1.2.2.2.1.1.1.1.1" class="ltx_text ltx_font_medium">:
<br class="ltx_break"></span></span></span>
<span id="A1.F12.pic1.2.2.2.1.1.2" class="ltx_p"><span id="A1.F12.pic1.2.2.2.1.1.2.1" class="ltx_text ltx_font_typewriter">Generate aliases for the given entities.
<br class="ltx_break">Here are some examples.
<br class="ltx_break"></span></span>
<span id="A1.F12.pic1.2.2.2.1.1.3" class="ltx_p"><span id="A1.F12.pic1.2.2.2.1.1.3.1" class="ltx_text ltx_font_typewriter">&lt;example&gt;
<br class="ltx_break"></span></span>
<span id="A1.F12.pic1.2.2.2.1.1.4" class="ltx_p"><span id="A1.F12.pic1.2.2.2.1.1.4.1" class="ltx_text ltx_font_typewriter">H:
<br class="ltx_break">entity 1: Luis Fortuno
<br class="ltx_break">entity 2: Alejandro Garcia Padilla
<br class="ltx_break"></span></span>
<span id="A1.F12.pic1.2.2.2.1.1.5" class="ltx_p"><span id="A1.F12.pic1.2.2.2.1.1.5.1" class="ltx_text ltx_font_typewriter">A:
<br class="ltx_break">&lt;response&gt;
<br class="ltx_break"></span></span>
<span id="A1.F12.pic1.2.2.2.1.1.6" class="ltx_p"><span id="A1.F12.pic1.2.2.2.1.1.6.1" class="ltx_text ltx_font_typewriter">For entity 1 Luis Fortuno:
<br class="ltx_break">&lt;entity1_alias1&gt;Luis Guillermo Fortuno Burset&lt;/entity1_alias1&gt;
<br class="ltx_break">&lt;entity1_alias2&gt;Luis G. Fortuno&lt;/entity1_alias2&gt;
<br class="ltx_break">&lt;entity1_alias3&gt;Luis Fortuno&lt;/entity1_alias3&gt;
<br class="ltx_break">&lt;entity1_alias4&gt;Luis G. Fortuno&lt;/entity1_alias4&gt;
<br class="ltx_break">&lt;entity1_alias5&gt;Luis Guillermo Fortuno Burset&lt;/entity1_alias5&gt;
<br class="ltx_break"></span></span>
<span id="A1.F12.pic1.2.2.2.1.1.7" class="ltx_p"><span id="A1.F12.pic1.2.2.2.1.1.7.1" class="ltx_text ltx_font_typewriter">For entity 2 Alejandro Garcia Padilla:
<br class="ltx_break">&lt;entity2_alias1&gt;Alejandro Javier Garcia Padilla&lt;/entity2_alias1&gt;
<br class="ltx_break">&lt;entity2_alias2&gt;Garcia Padilla&lt;/entity2_alias2&gt;
<br class="ltx_break">&lt;entity2_alias3&gt;Garcia-Padilla&lt;/entity2_alias3&gt;
<br class="ltx_break">&lt;entity2_alias4&gt;Alejandro J. Garcia-Padilla&lt;/entity2_alias4&gt;
<br class="ltx_break"></span></span>
<span id="A1.F12.pic1.2.2.2.1.1.8" class="ltx_p"><span id="A1.F12.pic1.2.2.2.1.1.8.1" class="ltx_text ltx_font_typewriter">&lt;/response&gt;
<br class="ltx_break"></span></span>
<span id="A1.F12.pic1.2.2.2.1.1.9" class="ltx_p"><span id="A1.F12.pic1.2.2.2.1.1.9.1" class="ltx_text ltx_font_typewriter">&lt;/example&gt;
<br class="ltx_break"></span></span>
<span id="A1.F12.pic1.2.2.2.1.1.10" class="ltx_p"><span id="A1.F12.pic1.2.2.2.1.1.10.1" class="ltx_text ltx_font_typewriter">Here are two new entities.
<br class="ltx_break">Please generate their aliases.
<br class="ltx_break"></span></span>
<span id="A1.F12.pic1.2.2.2.1.1.11" class="ltx_p"><span id="A1.F12.pic1.2.2.2.1.1.11.1" class="ltx_text ltx_font_typewriter">entity 1: {{answer}}
<br class="ltx_break">entity 2: {{counterfactual answer}}
<br class="ltx_break"></span></span>
<span id="A1.F12.pic1.2.2.2.1.1.12" class="ltx_p"><span id="A1.F12.pic1.2.2.2.1.1.12.1" class="ltx_text ltx_font_typewriter ltx_font_bold">Assistant:<span id="A1.F12.pic1.2.2.2.1.1.12.1.1" class="ltx_text ltx_font_medium"></span></span></span>
</span></foreignObject></g></g></svg>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 12: </span>Alias generation prompt</figcaption>
</figure>
</section>
<section id="A1.SS8" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.8 </span>Fact-wise Perplexity</h3>

<div id="A1.SS8.p1" class="ltx_para">
<p id="A1.SS8.p1.1" class="ltx_p">Table <a href="#A1.T6" title="Table 6 ‣ A.8 Fact-wise Perplexity ‣ Appendix A Appendix ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> summarizes the fact-wise editing accuracy of QLoRA and MEMIT, measured using the metric as described in Section <a href="#S5.SS1" title="5.1 Fact-wise Editing Effectiveness ‣ 5 Analysis ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5.1</span></a>.</p>
</div>
<figure id="A1.T6" class="ltx_table">
<div id="A1.T6.3" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:455.2pt;height:319.1pt;vertical-align:-0.8pt;"><span class="ltx_transformed_inner" style="transform:translate(-68.4pt,47.9pt) scale(0.768811525309595,0.768811525309595) ;">
<table id="A1.T6.3.3" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T6.3.3.4.1" class="ltx_tr">
<th id="A1.T6.3.3.4.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt" rowspan="2"><span id="A1.T6.3.3.4.1.1.1" class="ltx_text ltx_font_bold">Scheme</span></th>
<th id="A1.T6.3.3.4.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt" rowspan="2"><span id="A1.T6.3.3.4.1.2.1" class="ltx_text ltx_font_bold">Model</span></th>
<th id="A1.T6.3.3.4.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="3"><span id="A1.T6.3.3.4.1.3.1" class="ltx_text ltx_font_bold">Pre-edit</span></th>
<th id="A1.T6.3.3.4.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="4"><span id="A1.T6.3.3.4.1.4.1" class="ltx_text ltx_font_bold">QLoRA</span></th>
<th id="A1.T6.3.3.4.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="4"><span id="A1.T6.3.3.4.1.5.1" class="ltx_text ltx_font_bold">MEMIT</span></th>
</tr>
<tr id="A1.T6.3.3.3" class="ltx_tr">
<th id="A1.T6.3.3.3.4" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t">PPL(F)</th>
<th id="A1.T6.3.3.3.5" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t">PPL(CF)</th>
<th id="A1.T6.1.1.1.1" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t"><math id="A1.T6.1.1.1.1.m1.1" class="ltx_Math" alttext="\Delta" display="inline"><semantics id="A1.T6.1.1.1.1.m1.1a"><mi mathvariant="normal" id="A1.T6.1.1.1.1.m1.1.1" xref="A1.T6.1.1.1.1.m1.1.1.cmml">Δ</mi><annotation-xml encoding="MathML-Content" id="A1.T6.1.1.1.1.m1.1b"><ci id="A1.T6.1.1.1.1.m1.1.1.cmml" xref="A1.T6.1.1.1.1.m1.1.1">Δ</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T6.1.1.1.1.m1.1c">\Delta</annotation></semantics></math></th>
<th id="A1.T6.3.3.3.6" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t">PPL(F)</th>
<th id="A1.T6.3.3.3.7" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t">PPL(CF)</th>
<th id="A1.T6.2.2.2.2" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t"><math id="A1.T6.2.2.2.2.m1.1" class="ltx_Math" alttext="\Delta" display="inline"><semantics id="A1.T6.2.2.2.2.m1.1a"><mi mathvariant="normal" id="A1.T6.2.2.2.2.m1.1.1" xref="A1.T6.2.2.2.2.m1.1.1.cmml">Δ</mi><annotation-xml encoding="MathML-Content" id="A1.T6.2.2.2.2.m1.1b"><ci id="A1.T6.2.2.2.2.m1.1.1.cmml" xref="A1.T6.2.2.2.2.m1.1.1">Δ</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T6.2.2.2.2.m1.1c">\Delta</annotation></semantics></math></th>
<th id="A1.T6.3.3.3.8" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.3.8.1" class="ltx_text" style="background-color:#D9D9D9;">ACC</span></th>
<th id="A1.T6.3.3.3.9" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t">PPL(F)</th>
<th id="A1.T6.3.3.3.10" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t">PPL(CF)</th>
<th id="A1.T6.3.3.3.3" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t"><math id="A1.T6.3.3.3.3.m1.1" class="ltx_Math" alttext="\Delta" display="inline"><semantics id="A1.T6.3.3.3.3.m1.1a"><mi mathvariant="normal" id="A1.T6.3.3.3.3.m1.1.1" xref="A1.T6.3.3.3.3.m1.1.1.cmml">Δ</mi><annotation-xml encoding="MathML-Content" id="A1.T6.3.3.3.3.m1.1b"><ci id="A1.T6.3.3.3.3.m1.1.1.cmml" xref="A1.T6.3.3.3.3.m1.1.1">Δ</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T6.3.3.3.3.m1.1c">\Delta</annotation></semantics></math></th>
<th id="A1.T6.3.3.3.11" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_t" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.3.11.1" class="ltx_text" style="background-color:#D9D9D9;">ACC</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T6.3.3.5.1" class="ltx_tr">
<th id="A1.T6.3.3.5.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="3"><span id="A1.T6.3.3.5.1.1.1" class="ltx_text">superlative</span></th>
<th id="A1.T6.3.3.5.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">7b</th>
<td id="A1.T6.3.3.5.1.3" class="ltx_td ltx_align_right ltx_border_t">12.69</td>
<td id="A1.T6.3.3.5.1.4" class="ltx_td ltx_align_right ltx_border_t">17.52</td>
<td id="A1.T6.3.3.5.1.5" class="ltx_td ltx_align_right ltx_border_t">-4.84</td>
<td id="A1.T6.3.3.5.1.6" class="ltx_td ltx_align_right ltx_border_t">5.19</td>
<td id="A1.T6.3.3.5.1.7" class="ltx_td ltx_align_right ltx_border_t">2.74</td>
<td id="A1.T6.3.3.5.1.8" class="ltx_td ltx_align_right ltx_border_t">2.45</td>
<td id="A1.T6.3.3.5.1.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.5.1.9.1" class="ltx_text" style="background-color:#D9D9D9;">97.60</span></td>
<td id="A1.T6.3.3.5.1.10" class="ltx_td ltx_align_right ltx_border_t">172.09</td>
<td id="A1.T6.3.3.5.1.11" class="ltx_td ltx_align_right ltx_border_t">186.84</td>
<td id="A1.T6.3.3.5.1.12" class="ltx_td ltx_align_right ltx_border_t">-14.75</td>
<td id="A1.T6.3.3.5.1.13" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.5.1.13.1" class="ltx_text" style="background-color:#D9D9D9;">55.09</span></td>
</tr>
<tr id="A1.T6.3.3.6.2" class="ltx_tr">
<th id="A1.T6.3.3.6.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">13b</th>
<td id="A1.T6.3.3.6.2.2" class="ltx_td ltx_align_right">11.55</td>
<td id="A1.T6.3.3.6.2.3" class="ltx_td ltx_align_right">14.31</td>
<td id="A1.T6.3.3.6.2.4" class="ltx_td ltx_align_right">-2.76</td>
<td id="A1.T6.3.3.6.2.5" class="ltx_td ltx_align_right">6.61</td>
<td id="A1.T6.3.3.6.2.6" class="ltx_td ltx_align_right">3.84</td>
<td id="A1.T6.3.3.6.2.7" class="ltx_td ltx_align_right">2.77</td>
<td id="A1.T6.3.3.6.2.8" class="ltx_td ltx_align_right" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.6.2.8.1" class="ltx_text" style="background-color:#D9D9D9;">97.60</span></td>
<td id="A1.T6.3.3.6.2.9" class="ltx_td ltx_align_right">157.98</td>
<td id="A1.T6.3.3.6.2.10" class="ltx_td ltx_align_right">177.21</td>
<td id="A1.T6.3.3.6.2.11" class="ltx_td ltx_align_right">-19.23</td>
<td id="A1.T6.3.3.6.2.12" class="ltx_td ltx_align_right" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.6.2.12.1" class="ltx_text" style="background-color:#D9D9D9;">51.50</span></td>
</tr>
<tr id="A1.T6.3.3.7.3" class="ltx_tr">
<th id="A1.T6.3.3.7.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">33b</th>
<td id="A1.T6.3.3.7.3.2" class="ltx_td ltx_align_right">10.51</td>
<td id="A1.T6.3.3.7.3.3" class="ltx_td ltx_align_right">15.80</td>
<td id="A1.T6.3.3.7.3.4" class="ltx_td ltx_align_right">-5.28</td>
<td id="A1.T6.3.3.7.3.5" class="ltx_td ltx_align_right">5.42</td>
<td id="A1.T6.3.3.7.3.6" class="ltx_td ltx_align_right">2.65</td>
<td id="A1.T6.3.3.7.3.7" class="ltx_td ltx_align_right">2.78</td>
<td id="A1.T6.3.3.7.3.8" class="ltx_td ltx_align_right" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.7.3.8.1" class="ltx_text" style="background-color:#D9D9D9;">97.01</span></td>
<td id="A1.T6.3.3.7.3.9" class="ltx_td ltx_align_right">–</td>
<td id="A1.T6.3.3.7.3.10" class="ltx_td ltx_align_right">–</td>
<td id="A1.T6.3.3.7.3.11" class="ltx_td ltx_align_right">–</td>
<td id="A1.T6.3.3.7.3.12" class="ltx_td ltx_align_right" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.7.3.12.1" class="ltx_text" style="background-color:#D9D9D9;">–</span></td>
</tr>
<tr id="A1.T6.3.3.8.4" class="ltx_tr">
<th id="A1.T6.3.3.8.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="3"><span id="A1.T6.3.3.8.4.1.1" class="ltx_text">comparative</span></th>
<th id="A1.T6.3.3.8.4.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">7b</th>
<td id="A1.T6.3.3.8.4.3" class="ltx_td ltx_align_right ltx_border_t">12.41</td>
<td id="A1.T6.3.3.8.4.4" class="ltx_td ltx_align_right ltx_border_t">19.15</td>
<td id="A1.T6.3.3.8.4.5" class="ltx_td ltx_align_right ltx_border_t">-6.74</td>
<td id="A1.T6.3.3.8.4.6" class="ltx_td ltx_align_right ltx_border_t">5.61</td>
<td id="A1.T6.3.3.8.4.7" class="ltx_td ltx_align_right ltx_border_t">2.91</td>
<td id="A1.T6.3.3.8.4.8" class="ltx_td ltx_align_right ltx_border_t">2.70</td>
<td id="A1.T6.3.3.8.4.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.8.4.9.1" class="ltx_text" style="background-color:#D9D9D9;">97.66</span></td>
<td id="A1.T6.3.3.8.4.10" class="ltx_td ltx_align_right ltx_border_t">272.06</td>
<td id="A1.T6.3.3.8.4.11" class="ltx_td ltx_align_right ltx_border_t">247.97</td>
<td id="A1.T6.3.3.8.4.12" class="ltx_td ltx_align_right ltx_border_t">24.09</td>
<td id="A1.T6.3.3.8.4.13" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.8.4.13.1" class="ltx_text" style="background-color:#D9D9D9;">52.34</span></td>
</tr>
<tr id="A1.T6.3.3.9.5" class="ltx_tr">
<th id="A1.T6.3.3.9.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">13b</th>
<td id="A1.T6.3.3.9.5.2" class="ltx_td ltx_align_right">9.89</td>
<td id="A1.T6.3.3.9.5.3" class="ltx_td ltx_align_right">16.30</td>
<td id="A1.T6.3.3.9.5.4" class="ltx_td ltx_align_right">-6.42</td>
<td id="A1.T6.3.3.9.5.5" class="ltx_td ltx_align_right">5.66</td>
<td id="A1.T6.3.3.9.5.6" class="ltx_td ltx_align_right">3.07</td>
<td id="A1.T6.3.3.9.5.7" class="ltx_td ltx_align_right">2.58</td>
<td id="A1.T6.3.3.9.5.8" class="ltx_td ltx_align_right" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.9.5.8.1" class="ltx_text" style="background-color:#D9D9D9;">96.88</span></td>
<td id="A1.T6.3.3.9.5.9" class="ltx_td ltx_align_right">356.36</td>
<td id="A1.T6.3.3.9.5.10" class="ltx_td ltx_align_right">534.46</td>
<td id="A1.T6.3.3.9.5.11" class="ltx_td ltx_align_right">-178.10</td>
<td id="A1.T6.3.3.9.5.12" class="ltx_td ltx_align_right" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.9.5.12.1" class="ltx_text" style="background-color:#D9D9D9;">53.91</span></td>
</tr>
<tr id="A1.T6.3.3.10.6" class="ltx_tr">
<th id="A1.T6.3.3.10.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">33b</th>
<td id="A1.T6.3.3.10.6.2" class="ltx_td ltx_align_right">11.05</td>
<td id="A1.T6.3.3.10.6.3" class="ltx_td ltx_align_right">17.42</td>
<td id="A1.T6.3.3.10.6.4" class="ltx_td ltx_align_right">-6.37</td>
<td id="A1.T6.3.3.10.6.5" class="ltx_td ltx_align_right">6.14</td>
<td id="A1.T6.3.3.10.6.6" class="ltx_td ltx_align_right">2.78</td>
<td id="A1.T6.3.3.10.6.7" class="ltx_td ltx_align_right">3.35</td>
<td id="A1.T6.3.3.10.6.8" class="ltx_td ltx_align_right" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.10.6.8.1" class="ltx_text" style="background-color:#D9D9D9;">99.22</span></td>
<td id="A1.T6.3.3.10.6.9" class="ltx_td ltx_align_right">–</td>
<td id="A1.T6.3.3.10.6.10" class="ltx_td ltx_align_right">–</td>
<td id="A1.T6.3.3.10.6.11" class="ltx_td ltx_align_right">–</td>
<td id="A1.T6.3.3.10.6.12" class="ltx_td ltx_align_right" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.10.6.12.1" class="ltx_text" style="background-color:#D9D9D9;">–</span></td>
</tr>
<tr id="A1.T6.3.3.11.7" class="ltx_tr">
<th id="A1.T6.3.3.11.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="3"><span id="A1.T6.3.3.11.7.1.1" class="ltx_text">counting</span></th>
<th id="A1.T6.3.3.11.7.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">7b</th>
<td id="A1.T6.3.3.11.7.3" class="ltx_td ltx_align_right ltx_border_t">10.50</td>
<td id="A1.T6.3.3.11.7.4" class="ltx_td ltx_align_right ltx_border_t">14.55</td>
<td id="A1.T6.3.3.11.7.5" class="ltx_td ltx_align_right ltx_border_t">-4.05</td>
<td id="A1.T6.3.3.11.7.6" class="ltx_td ltx_align_right ltx_border_t">12.82</td>
<td id="A1.T6.3.3.11.7.7" class="ltx_td ltx_align_right ltx_border_t">1.70</td>
<td id="A1.T6.3.3.11.7.8" class="ltx_td ltx_align_right ltx_border_t">11.12</td>
<td id="A1.T6.3.3.11.7.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.11.7.9.1" class="ltx_text" style="background-color:#D9D9D9;">99.02</span></td>
<td id="A1.T6.3.3.11.7.10" class="ltx_td ltx_align_right ltx_border_t">182.11</td>
<td id="A1.T6.3.3.11.7.11" class="ltx_td ltx_align_right ltx_border_t">246.24</td>
<td id="A1.T6.3.3.11.7.12" class="ltx_td ltx_align_right ltx_border_t">-64.13</td>
<td id="A1.T6.3.3.11.7.13" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.11.7.13.1" class="ltx_text" style="background-color:#D9D9D9;">32.35</span></td>
</tr>
<tr id="A1.T6.3.3.12.8" class="ltx_tr">
<th id="A1.T6.3.3.12.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">13b</th>
<td id="A1.T6.3.3.12.8.2" class="ltx_td ltx_align_right">7.98</td>
<td id="A1.T6.3.3.12.8.3" class="ltx_td ltx_align_right">12.73</td>
<td id="A1.T6.3.3.12.8.4" class="ltx_td ltx_align_right">-4.75</td>
<td id="A1.T6.3.3.12.8.5" class="ltx_td ltx_align_right">13.18</td>
<td id="A1.T6.3.3.12.8.6" class="ltx_td ltx_align_right">1.40</td>
<td id="A1.T6.3.3.12.8.7" class="ltx_td ltx_align_right">11.78</td>
<td id="A1.T6.3.3.12.8.8" class="ltx_td ltx_align_right" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.12.8.8.1" class="ltx_text" style="background-color:#D9D9D9;">98.04</span></td>
<td id="A1.T6.3.3.12.8.9" class="ltx_td ltx_align_right">164.60</td>
<td id="A1.T6.3.3.12.8.10" class="ltx_td ltx_align_right">271.20</td>
<td id="A1.T6.3.3.12.8.11" class="ltx_td ltx_align_right">-106.60</td>
<td id="A1.T6.3.3.12.8.12" class="ltx_td ltx_align_right" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.12.8.12.1" class="ltx_text" style="background-color:#D9D9D9;">29.41</span></td>
</tr>
<tr id="A1.T6.3.3.13.9" class="ltx_tr">
<th id="A1.T6.3.3.13.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">33b</th>
<td id="A1.T6.3.3.13.9.2" class="ltx_td ltx_align_right">9.52</td>
<td id="A1.T6.3.3.13.9.3" class="ltx_td ltx_align_right">13.84</td>
<td id="A1.T6.3.3.13.9.4" class="ltx_td ltx_align_right">-4.32</td>
<td id="A1.T6.3.3.13.9.5" class="ltx_td ltx_align_right">18.08</td>
<td id="A1.T6.3.3.13.9.6" class="ltx_td ltx_align_right">1.51</td>
<td id="A1.T6.3.3.13.9.7" class="ltx_td ltx_align_right">16.58</td>
<td id="A1.T6.3.3.13.9.8" class="ltx_td ltx_align_right" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.13.9.8.1" class="ltx_text" style="background-color:#D9D9D9;">98.53</span></td>
<td id="A1.T6.3.3.13.9.9" class="ltx_td ltx_align_right">–</td>
<td id="A1.T6.3.3.13.9.10" class="ltx_td ltx_align_right">–</td>
<td id="A1.T6.3.3.13.9.11" class="ltx_td ltx_align_right">–</td>
<td id="A1.T6.3.3.13.9.12" class="ltx_td ltx_align_right" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.13.9.12.1" class="ltx_text" style="background-color:#D9D9D9;">–</span></td>
</tr>
<tr id="A1.T6.3.3.14.10" class="ltx_tr">
<th id="A1.T6.3.3.14.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="3"><span id="A1.T6.3.3.14.10.1.1" class="ltx_text">sorting</span></th>
<th id="A1.T6.3.3.14.10.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">7b</th>
<td id="A1.T6.3.3.14.10.3" class="ltx_td ltx_align_right ltx_border_t">16.84</td>
<td id="A1.T6.3.3.14.10.4" class="ltx_td ltx_align_right ltx_border_t">23.51</td>
<td id="A1.T6.3.3.14.10.5" class="ltx_td ltx_align_right ltx_border_t">-6.67</td>
<td id="A1.T6.3.3.14.10.6" class="ltx_td ltx_align_right ltx_border_t">17.53</td>
<td id="A1.T6.3.3.14.10.7" class="ltx_td ltx_align_right ltx_border_t">15.93</td>
<td id="A1.T6.3.3.14.10.8" class="ltx_td ltx_align_right ltx_border_t">1.60</td>
<td id="A1.T6.3.3.14.10.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.14.10.9.1" class="ltx_text" style="background-color:#D9D9D9;">94.49</span></td>
<td id="A1.T6.3.3.14.10.10" class="ltx_td ltx_align_right ltx_border_t">186.02</td>
<td id="A1.T6.3.3.14.10.11" class="ltx_td ltx_align_right ltx_border_t">125.55</td>
<td id="A1.T6.3.3.14.10.12" class="ltx_td ltx_align_right ltx_border_t">60.47</td>
<td id="A1.T6.3.3.14.10.13" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.14.10.13.1" class="ltx_text" style="background-color:#D9D9D9;">64.57</span></td>
</tr>
<tr id="A1.T6.3.3.15.11" class="ltx_tr">
<th id="A1.T6.3.3.15.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">13b</th>
<td id="A1.T6.3.3.15.11.2" class="ltx_td ltx_align_right">13.40</td>
<td id="A1.T6.3.3.15.11.3" class="ltx_td ltx_align_right">19.46</td>
<td id="A1.T6.3.3.15.11.4" class="ltx_td ltx_align_right">-6.06</td>
<td id="A1.T6.3.3.15.11.5" class="ltx_td ltx_align_right">20.99</td>
<td id="A1.T6.3.3.15.11.6" class="ltx_td ltx_align_right">19.24</td>
<td id="A1.T6.3.3.15.11.7" class="ltx_td ltx_align_right">1.75</td>
<td id="A1.T6.3.3.15.11.8" class="ltx_td ltx_align_right" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.15.11.8.1" class="ltx_text" style="background-color:#D9D9D9;">96.85</span></td>
<td id="A1.T6.3.3.15.11.9" class="ltx_td ltx_align_right">623.16</td>
<td id="A1.T6.3.3.15.11.10" class="ltx_td ltx_align_right">393.86</td>
<td id="A1.T6.3.3.15.11.11" class="ltx_td ltx_align_right">229.30</td>
<td id="A1.T6.3.3.15.11.12" class="ltx_td ltx_align_right" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.15.11.12.1" class="ltx_text" style="background-color:#D9D9D9;">38.58</span></td>
</tr>
<tr id="A1.T6.3.3.16.12" class="ltx_tr">
<th id="A1.T6.3.3.16.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">33b</th>
<td id="A1.T6.3.3.16.12.2" class="ltx_td ltx_align_right">14.06</td>
<td id="A1.T6.3.3.16.12.3" class="ltx_td ltx_align_right">20.44</td>
<td id="A1.T6.3.3.16.12.4" class="ltx_td ltx_align_right">-6.38</td>
<td id="A1.T6.3.3.16.12.5" class="ltx_td ltx_align_right">16.54</td>
<td id="A1.T6.3.3.16.12.6" class="ltx_td ltx_align_right">14.45</td>
<td id="A1.T6.3.3.16.12.7" class="ltx_td ltx_align_right">2.09</td>
<td id="A1.T6.3.3.16.12.8" class="ltx_td ltx_align_right" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.16.12.8.1" class="ltx_text" style="background-color:#D9D9D9;">96.85</span></td>
<td id="A1.T6.3.3.16.12.9" class="ltx_td ltx_align_right">–</td>
<td id="A1.T6.3.3.16.12.10" class="ltx_td ltx_align_right">–</td>
<td id="A1.T6.3.3.16.12.11" class="ltx_td ltx_align_right">–</td>
<td id="A1.T6.3.3.16.12.12" class="ltx_td ltx_align_right" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.16.12.12.1" class="ltx_text" style="background-color:#D9D9D9;">–</span></td>
</tr>
<tr id="A1.T6.3.3.17.13" class="ltx_tr">
<th id="A1.T6.3.3.17.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="3"><span id="A1.T6.3.3.17.13.1.1" class="ltx_text">aggregation</span></th>
<th id="A1.T6.3.3.17.13.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">7b</th>
<td id="A1.T6.3.3.17.13.3" class="ltx_td ltx_align_right ltx_border_t">16.12</td>
<td id="A1.T6.3.3.17.13.4" class="ltx_td ltx_align_right ltx_border_t">13.96</td>
<td id="A1.T6.3.3.17.13.5" class="ltx_td ltx_align_right ltx_border_t">2.16</td>
<td id="A1.T6.3.3.17.13.6" class="ltx_td ltx_align_right ltx_border_t">30.42</td>
<td id="A1.T6.3.3.17.13.7" class="ltx_td ltx_align_right ltx_border_t">1.47</td>
<td id="A1.T6.3.3.17.13.8" class="ltx_td ltx_align_right ltx_border_t">28.95</td>
<td id="A1.T6.3.3.17.13.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.17.13.9.1" class="ltx_text" style="background-color:#D9D9D9;">100.00</span></td>
<td id="A1.T6.3.3.17.13.10" class="ltx_td ltx_align_right ltx_border_t">35.19</td>
<td id="A1.T6.3.3.17.13.11" class="ltx_td ltx_align_right ltx_border_t">37.56</td>
<td id="A1.T6.3.3.17.13.12" class="ltx_td ltx_align_right ltx_border_t">-2.37</td>
<td id="A1.T6.3.3.17.13.13" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.17.13.13.1" class="ltx_text" style="background-color:#D9D9D9;">32.69</span></td>
</tr>
<tr id="A1.T6.3.3.18.14" class="ltx_tr">
<th id="A1.T6.3.3.18.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">13b</th>
<td id="A1.T6.3.3.18.14.2" class="ltx_td ltx_align_right">12.56</td>
<td id="A1.T6.3.3.18.14.3" class="ltx_td ltx_align_right">13.03</td>
<td id="A1.T6.3.3.18.14.4" class="ltx_td ltx_align_right">-0.47</td>
<td id="A1.T6.3.3.18.14.5" class="ltx_td ltx_align_right">29.11</td>
<td id="A1.T6.3.3.18.14.6" class="ltx_td ltx_align_right">1.26</td>
<td id="A1.T6.3.3.18.14.7" class="ltx_td ltx_align_right">27.85</td>
<td id="A1.T6.3.3.18.14.8" class="ltx_td ltx_align_right" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.18.14.8.1" class="ltx_text" style="background-color:#D9D9D9;">90.38</span></td>
<td id="A1.T6.3.3.18.14.9" class="ltx_td ltx_align_right">21.27</td>
<td id="A1.T6.3.3.18.14.10" class="ltx_td ltx_align_right">24.90</td>
<td id="A1.T6.3.3.18.14.11" class="ltx_td ltx_align_right">-3.63</td>
<td id="A1.T6.3.3.18.14.12" class="ltx_td ltx_align_right" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.18.14.12.1" class="ltx_text" style="background-color:#D9D9D9;">44.23</span></td>
</tr>
<tr id="A1.T6.3.3.19.15" class="ltx_tr">
<th id="A1.T6.3.3.19.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">33b</th>
<td id="A1.T6.3.3.19.15.2" class="ltx_td ltx_align_right">16.24</td>
<td id="A1.T6.3.3.19.15.3" class="ltx_td ltx_align_right">14.58</td>
<td id="A1.T6.3.3.19.15.4" class="ltx_td ltx_align_right">1.66</td>
<td id="A1.T6.3.3.19.15.5" class="ltx_td ltx_align_right">25.60</td>
<td id="A1.T6.3.3.19.15.6" class="ltx_td ltx_align_right">1.43</td>
<td id="A1.T6.3.3.19.15.7" class="ltx_td ltx_align_right">24.17</td>
<td id="A1.T6.3.3.19.15.8" class="ltx_td ltx_align_right" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.19.15.8.1" class="ltx_text" style="background-color:#D9D9D9;">90.38</span></td>
<td id="A1.T6.3.3.19.15.9" class="ltx_td ltx_align_right">–</td>
<td id="A1.T6.3.3.19.15.10" class="ltx_td ltx_align_right">–</td>
<td id="A1.T6.3.3.19.15.11" class="ltx_td ltx_align_right">–</td>
<td id="A1.T6.3.3.19.15.12" class="ltx_td ltx_align_right" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.19.15.12.1" class="ltx_text" style="background-color:#D9D9D9;">–</span></td>
</tr>
<tr id="A1.T6.3.3.20.16" class="ltx_tr">
<th id="A1.T6.3.3.20.16.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="3"><span id="A1.T6.3.3.20.16.1.1" class="ltx_text">subtraction</span></th>
<th id="A1.T6.3.3.20.16.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">7b</th>
<td id="A1.T6.3.3.20.16.3" class="ltx_td ltx_align_right ltx_border_t">18.74</td>
<td id="A1.T6.3.3.20.16.4" class="ltx_td ltx_align_right ltx_border_t">12.84</td>
<td id="A1.T6.3.3.20.16.5" class="ltx_td ltx_align_right ltx_border_t">5.91</td>
<td id="A1.T6.3.3.20.16.6" class="ltx_td ltx_align_right ltx_border_t">39.62</td>
<td id="A1.T6.3.3.20.16.7" class="ltx_td ltx_align_right ltx_border_t">1.48</td>
<td id="A1.T6.3.3.20.16.8" class="ltx_td ltx_align_right ltx_border_t">38.14</td>
<td id="A1.T6.3.3.20.16.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.20.16.9.1" class="ltx_text" style="background-color:#D9D9D9;">100.00</span></td>
<td id="A1.T6.3.3.20.16.10" class="ltx_td ltx_align_right ltx_border_t">39.16</td>
<td id="A1.T6.3.3.20.16.11" class="ltx_td ltx_align_right ltx_border_t">47.83</td>
<td id="A1.T6.3.3.20.16.12" class="ltx_td ltx_align_right ltx_border_t">-8.67</td>
<td id="A1.T6.3.3.20.16.13" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.20.16.13.1" class="ltx_text" style="background-color:#D9D9D9;">51.92</span></td>
</tr>
<tr id="A1.T6.3.3.21.17" class="ltx_tr">
<th id="A1.T6.3.3.21.17.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">13b</th>
<td id="A1.T6.3.3.21.17.2" class="ltx_td ltx_align_right">17.30</td>
<td id="A1.T6.3.3.21.17.3" class="ltx_td ltx_align_right">11.99</td>
<td id="A1.T6.3.3.21.17.4" class="ltx_td ltx_align_right">5.31</td>
<td id="A1.T6.3.3.21.17.5" class="ltx_td ltx_align_right">24.79</td>
<td id="A1.T6.3.3.21.17.6" class="ltx_td ltx_align_right">1.29</td>
<td id="A1.T6.3.3.21.17.7" class="ltx_td ltx_align_right">23.50</td>
<td id="A1.T6.3.3.21.17.8" class="ltx_td ltx_align_right" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.21.17.8.1" class="ltx_text" style="background-color:#D9D9D9;">94.23</span></td>
<td id="A1.T6.3.3.21.17.9" class="ltx_td ltx_align_right">28.89</td>
<td id="A1.T6.3.3.21.17.10" class="ltx_td ltx_align_right">31.68</td>
<td id="A1.T6.3.3.21.17.11" class="ltx_td ltx_align_right">-2.79</td>
<td id="A1.T6.3.3.21.17.12" class="ltx_td ltx_align_right" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.21.17.12.1" class="ltx_text" style="background-color:#D9D9D9;">53.85</span></td>
</tr>
<tr id="A1.T6.3.3.22.18" class="ltx_tr">
<th id="A1.T6.3.3.22.18.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">33b</th>
<td id="A1.T6.3.3.22.18.2" class="ltx_td ltx_align_right">18.60</td>
<td id="A1.T6.3.3.22.18.3" class="ltx_td ltx_align_right">12.91</td>
<td id="A1.T6.3.3.22.18.4" class="ltx_td ltx_align_right">5.69</td>
<td id="A1.T6.3.3.22.18.5" class="ltx_td ltx_align_right">33.24</td>
<td id="A1.T6.3.3.22.18.6" class="ltx_td ltx_align_right">1.44</td>
<td id="A1.T6.3.3.22.18.7" class="ltx_td ltx_align_right">31.80</td>
<td id="A1.T6.3.3.22.18.8" class="ltx_td ltx_align_right" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.22.18.8.1" class="ltx_text" style="background-color:#D9D9D9;">98.08</span></td>
<td id="A1.T6.3.3.22.18.9" class="ltx_td ltx_align_right">–</td>
<td id="A1.T6.3.3.22.18.10" class="ltx_td ltx_align_right">–</td>
<td id="A1.T6.3.3.22.18.11" class="ltx_td ltx_align_right">–</td>
<td id="A1.T6.3.3.22.18.12" class="ltx_td ltx_align_right" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.22.18.12.1" class="ltx_text" style="background-color:#D9D9D9;">–</span></td>
</tr>
<tr id="A1.T6.3.3.23.19" class="ltx_tr">
<th id="A1.T6.3.3.23.19.1" class="ltx_td ltx_th ltx_th_row ltx_border_t"></th>
<th id="A1.T6.3.3.23.19.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">7b</th>
<td id="A1.T6.3.3.23.19.3" class="ltx_td ltx_align_right ltx_border_t">14.55</td>
<td id="A1.T6.3.3.23.19.4" class="ltx_td ltx_align_right ltx_border_t">16.92</td>
<td id="A1.T6.3.3.23.19.5" class="ltx_td ltx_align_right ltx_border_t">-2.37</td>
<td id="A1.T6.3.3.23.19.6" class="ltx_td ltx_align_right ltx_border_t">18.53</td>
<td id="A1.T6.3.3.23.19.7" class="ltx_td ltx_align_right ltx_border_t">4.37</td>
<td id="A1.T6.3.3.23.19.8" class="ltx_td ltx_align_right ltx_border_t">14.16</td>
<td id="A1.T6.3.3.23.19.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.23.19.9.1" class="ltx_text" style="background-color:#D9D9D9;">98.13</span></td>
<td id="A1.T6.3.3.23.19.10" class="ltx_td ltx_align_right ltx_border_t">147.77</td>
<td id="A1.T6.3.3.23.19.11" class="ltx_td ltx_align_right ltx_border_t">148.67</td>
<td id="A1.T6.3.3.23.19.12" class="ltx_td ltx_align_right ltx_border_t">-0.90</td>
<td id="A1.T6.3.3.23.19.13" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.23.19.13.1" class="ltx_text" style="background-color:#D9D9D9;">48.16</span></td>
</tr>
<tr id="A1.T6.3.3.24.20" class="ltx_tr">
<th id="A1.T6.3.3.24.20.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="A1.T6.3.3.24.20.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">13b</th>
<td id="A1.T6.3.3.24.20.3" class="ltx_td ltx_align_right">12.11</td>
<td id="A1.T6.3.3.24.20.4" class="ltx_td ltx_align_right">14.64</td>
<td id="A1.T6.3.3.24.20.5" class="ltx_td ltx_align_right">-2.52</td>
<td id="A1.T6.3.3.24.20.6" class="ltx_td ltx_align_right">16.72</td>
<td id="A1.T6.3.3.24.20.7" class="ltx_td ltx_align_right">5.02</td>
<td id="A1.T6.3.3.24.20.8" class="ltx_td ltx_align_right">11.71</td>
<td id="A1.T6.3.3.24.20.9" class="ltx_td ltx_align_right" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.24.20.9.1" class="ltx_text" style="background-color:#D9D9D9;">95.66</span></td>
<td id="A1.T6.3.3.24.20.10" class="ltx_td ltx_align_right">225.38</td>
<td id="A1.T6.3.3.24.20.11" class="ltx_td ltx_align_right">238.89</td>
<td id="A1.T6.3.3.24.20.12" class="ltx_td ltx_align_right">-13.51</td>
<td id="A1.T6.3.3.24.20.13" class="ltx_td ltx_align_right" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.24.20.13.1" class="ltx_text" style="background-color:#D9D9D9;">45.25</span></td>
</tr>
<tr id="A1.T6.3.3.25.21" class="ltx_tr">
<th id="A1.T6.3.3.25.21.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb"><span id="A1.T6.3.3.25.21.1.1" class="ltx_text ltx_font_italic">Average</span></th>
<th id="A1.T6.3.3.25.21.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">33b</th>
<td id="A1.T6.3.3.25.21.3" class="ltx_td ltx_align_right ltx_border_bb">13.33</td>
<td id="A1.T6.3.3.25.21.4" class="ltx_td ltx_align_right ltx_border_bb">15.83</td>
<td id="A1.T6.3.3.25.21.5" class="ltx_td ltx_align_right ltx_border_bb">-2.50</td>
<td id="A1.T6.3.3.25.21.6" class="ltx_td ltx_align_right ltx_border_bb">17.50</td>
<td id="A1.T6.3.3.25.21.7" class="ltx_td ltx_align_right ltx_border_bb">4.04</td>
<td id="A1.T6.3.3.25.21.8" class="ltx_td ltx_align_right ltx_border_bb">13.46</td>
<td id="A1.T6.3.3.25.21.9" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.25.21.9.1" class="ltx_text" style="background-color:#D9D9D9;">96.68</span></td>
<td id="A1.T6.3.3.25.21.10" class="ltx_td ltx_align_right ltx_border_bb">–</td>
<td id="A1.T6.3.3.25.21.11" class="ltx_td ltx_align_right ltx_border_bb">–</td>
<td id="A1.T6.3.3.25.21.12" class="ltx_td ltx_align_right ltx_border_bb">–</td>
<td id="A1.T6.3.3.25.21.13" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#D9D9D9;"><span id="A1.T6.3.3.25.21.13.1" class="ltx_text" style="background-color:#D9D9D9;">–</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 6: </span>Fact-wise perplexity over facts (<math id="A1.T6.6.m1.1" class="ltx_Math" alttext="\mathbb{F}" display="inline"><semantics id="A1.T6.6.m1.1b"><mi id="A1.T6.6.m1.1.1" xref="A1.T6.6.m1.1.1.cmml">𝔽</mi><annotation-xml encoding="MathML-Content" id="A1.T6.6.m1.1c"><ci id="A1.T6.6.m1.1.1.cmml" xref="A1.T6.6.m1.1.1">𝔽</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T6.6.m1.1d">\mathbb{F}</annotation></semantics></math>) and counterfactual facts (<math id="A1.T6.7.m2.1" class="ltx_Math" alttext="\mathbb{CF}" display="inline"><semantics id="A1.T6.7.m2.1b"><mrow id="A1.T6.7.m2.1.1" xref="A1.T6.7.m2.1.1.cmml"><mi id="A1.T6.7.m2.1.1.2" xref="A1.T6.7.m2.1.1.2.cmml">ℂ</mi><mo lspace="0em" rspace="0em" id="A1.T6.7.m2.1.1.1" xref="A1.T6.7.m2.1.1.1.cmml">​</mo><mi id="A1.T6.7.m2.1.1.3" xref="A1.T6.7.m2.1.1.3.cmml">𝔽</mi></mrow><annotation-xml encoding="MathML-Content" id="A1.T6.7.m2.1c"><apply id="A1.T6.7.m2.1.1.cmml" xref="A1.T6.7.m2.1.1"><times id="A1.T6.7.m2.1.1.1.cmml" xref="A1.T6.7.m2.1.1.1"></times><ci id="A1.T6.7.m2.1.1.2.cmml" xref="A1.T6.7.m2.1.1.2">ℂ</ci><ci id="A1.T6.7.m2.1.1.3.cmml" xref="A1.T6.7.m2.1.1.3">𝔽</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.T6.7.m2.1d">\mathbb{CF}</annotation></semantics></math>) with pre-edit and post-edit models.</figcaption>
</figure>
</section>
<section id="A1.SS9" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.9 </span>Fact Recall Evaluation</h3>

<div id="A1.SS9.p1" class="ltx_para">
<p id="A1.SS9.p1.1" class="ltx_p">Figure <a href="#A1.F13" title="Figure 13 ‣ A.9 Fact Recall Evaluation ‣ Appendix A Appendix ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">13</span></a> shows the prompt we used for fact recall evaluation of model generation against the set of counterfactual facts.</p>
</div>
<figure id="A1.F13" class="ltx_figure"><svg id="A1.F13.pic1" class="ltx_picture" height="854.82" overflow="visible" version="1.1" width="600"><g transform="translate(0,854.82) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#000000" fill-opacity="1.0"><path d="M 0 3.15 L 0 851.67 C 0 853.41 1.41 854.82 3.15 854.82 L 596.85 854.82 C 598.59 854.82 600 853.41 600 851.67 L 600 3.15 C 600 1.41 598.59 0 596.85 0 L 3.15 0 C 1.41 0 0 1.41 0 3.15 Z" style="stroke:none"></path></g><g fill="#FFFFFF" fill-opacity="1.0"><path d="M 1.18 3.15 L 1.18 832.28 L 598.82 832.28 L 598.82 3.15 C 598.82 2.06 597.94 1.18 596.85 1.18 L 3.15 1.18 C 2.06 1.18 1.18 2.06 1.18 3.15 Z" style="stroke:none"></path></g><g fill="#D9D9D9" fill-opacity="1.0"><path d="M 1.18 833.46 L 1.18 851.67 C 1.18 852.75 2.06 853.63 3.15 853.63 L 596.85 853.63 C 597.94 853.63 598.82 852.75 598.82 851.67 L 598.82 833.46 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 9.06 837.4)"><foreignObject width="581.89" height="12.3" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F13.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:420.5pt;">
<span id="A1.F13.pic1.1.1.1.1.1.1" class="ltx_p"><span id="A1.F13.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Fact Recall Evaluation Prompt</span></span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 9.06 5.12)"><foreignObject width="581.89" height="823.22" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F13.pic1.2.2.2.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:420.5pt;">
<span id="A1.F13.pic1.2.2.2.1.1.1" class="ltx_p"><span id="A1.F13.pic1.2.2.2.1.1.1.1" class="ltx_text ltx_font_typewriter ltx_font_bold">Human:
<br class="ltx_break"></span></span>
<span id="A1.F13.pic1.2.2.2.1.1.2" class="ltx_p"><span id="A1.F13.pic1.2.2.2.1.1.2.1" class="ltx_text ltx_font_typewriter">Given a paragraph and several facts, evaluate for each fact whether the information contained in the paragraph is consistent with it. For each fact, answer &lt;consistent&gt; or &lt;inconsistent&gt;. If the fact is completely unrelated to the paragraph, then ansewr &lt;unrelated&gt;.
<br class="ltx_break"></span></span>
<span id="A1.F13.pic1.2.2.2.1.1.3" class="ltx_p"><span id="A1.F13.pic1.2.2.2.1.1.3.1" class="ltx_text ltx_font_typewriter">Below are a few examples:
<br class="ltx_break"></span></span>
<span id="A1.F13.pic1.2.2.2.1.1.4" class="ltx_p"><span id="A1.F13.pic1.2.2.2.1.1.4.1" class="ltx_text ltx_font_typewriter">example 1:
<br class="ltx_break"></span></span>
<span id="A1.F13.pic1.2.2.2.1.1.5" class="ltx_p"><span id="A1.F13.pic1.2.2.2.1.1.5.1" class="ltx_text ltx_font_typewriter">Paragraph: Snowdon is 1,085 metres (3,560 ft) high. Ben Nevis is 1,345 metres (4,413 ft) high.
<br class="ltx_break">Fact:
<br class="ltx_break">&lt;1&gt; The height of the summit as 1,085 m (3,560 ft), making Snowdon the highest mountain in Wales. &lt;/1&gt;
<br class="ltx_break">&lt;2&gt; Ben Nevis is 2,000 meters high. &lt;/2&gt;
<br class="ltx_break">&lt;3&gt; Mount Everest at 29,029 ft (8,848 m) is not only the highest peak in the Himalayas, but the highest peak on the entire planet. &lt;/3&gt;
<br class="ltx_break"></span></span>
<span id="A1.F13.pic1.2.2.2.1.1.6" class="ltx_p"><span id="A1.F13.pic1.2.2.2.1.1.6.1" class="ltx_text ltx_font_typewriter">Evaluation:
<br class="ltx_break">&lt;1&gt; The fact is talking about the height of the mountain Snowdon, and the paragraph mentions its height as well, thus the fact is related to the Paragraph. With regard to consistency, the paragraph says Snowdon is 1,085 metres (3,560 ft) high and the fact conveys the same thing, they are consistent. &lt;consistent&gt;. &lt;/1&gt;
<br class="ltx_break"></span></span>
<span id="A1.F13.pic1.2.2.2.1.1.7" class="ltx_p"><span id="A1.F13.pic1.2.2.2.1.1.7.1" class="ltx_text ltx_font_typewriter">&lt;2&gt; The fact is talking about the height of the mountain Ben Nevis, and the paragraph mentions Ben Nevis’s height, thus the fact is related. With regard to consistency, the paragraph says Ben Nevis is 1,345 metres (4,413 ft) high but the fact says it to be 2,000 meters high, which is very different, thus inconsistent. &lt;inconsistent&gt; &lt;/2&gt;
<br class="ltx_break"></span></span>
<span id="A1.F13.pic1.2.2.2.1.1.8" class="ltx_p"><span id="A1.F13.pic1.2.2.2.1.1.8.1" class="ltx_text ltx_font_typewriter">&lt;3&gt; The fact is talking about the height of Mount Everest while the paragraph does not even mention Mount Everest, thus the fact is unrelated. &lt;unrelated&gt; &lt;/3&gt;
<br class="ltx_break"></span></span>
<span id="A1.F13.pic1.2.2.2.1.1.9" class="ltx_p"><span id="A1.F13.pic1.2.2.2.1.1.9.1" class="ltx_text ltx_font_typewriter">example 2:
<br class="ltx_break"></span></span>
<span id="A1.F13.pic1.2.2.2.1.1.10" class="ltx_p"><span id="A1.F13.pic1.2.2.2.1.1.10.1" class="ltx_text ltx_font_typewriter">Paragraph: Houston is located in the state of Texas. Tampa is located in the state of Florida. Florida is located in the southeastern United States. Texas is located in the central United States.
<br class="ltx_break">Facts:
<br class="ltx_break">&lt;1&gt; Houston is in Texas. &lt;/1&gt;
<br class="ltx_break">&lt;2&gt; New York City is in the New York state. &lt;/2&gt;
<br class="ltx_break">&lt;3&gt; Texas is in the middle of United States. &lt;/3&gt;
<br class="ltx_break">&lt;4&gt; Florida is in the southeastern United States &lt;/4&gt;
<br class="ltx_break"></span></span>
<span id="A1.F13.pic1.2.2.2.1.1.11" class="ltx_p"><span id="A1.F13.pic1.2.2.2.1.1.11.1" class="ltx_text ltx_font_typewriter">Evaluation:
<br class="ltx_break">&lt;1&gt; The paragraph also mentions that Houston is in Texas, which is also indicated in this fact,thus it’s both related and consistent. &lt;consistent&gt;. &lt;/1&gt;
<br class="ltx_break"></span></span>
<span id="A1.F13.pic1.2.2.2.1.1.12" class="ltx_p"><span id="A1.F13.pic1.2.2.2.1.1.12.1" class="ltx_text ltx_font_typewriter">&lt;2&gt; The paragraph does not mention New York City or New York state. Thus it is not related. &lt;unrelated&gt; &lt;/2&gt;
<br class="ltx_break"></span></span>
<span id="A1.F13.pic1.2.2.2.1.1.13" class="ltx_p"><span id="A1.F13.pic1.2.2.2.1.1.13.1" class="ltx_text ltx_font_typewriter">&lt;3&gt; The paragraph mentions that Texas is located in the central United States, which is indicated in the fact, thus the fact is consistent with the fact. &lt;consistent&gt; &lt;/3&gt;
<br class="ltx_break"></span></span>
<span id="A1.F13.pic1.2.2.2.1.1.14" class="ltx_p"><span id="A1.F13.pic1.2.2.2.1.1.14.1" class="ltx_text ltx_font_typewriter">&lt;4&gt; The paragraph mentions that Florida is located in the southeastern United States, which is indicated in the fact, thus the fact is consistent with the fact. &lt;consistent&gt; &lt;/4&gt;
<br class="ltx_break"></span></span>
<span id="A1.F13.pic1.2.2.2.1.1.15" class="ltx_p"><span id="A1.F13.pic1.2.2.2.1.1.15.1" class="ltx_text ltx_font_typewriter">Here is a paragraph and a list of facts:
<br class="ltx_break">Paragraph: {{paragraph}}
Facts:
<br class="ltx_break">{{facts}}
<br class="ltx_break"></span></span>
<span id="A1.F13.pic1.2.2.2.1.1.16" class="ltx_p"><span id="A1.F13.pic1.2.2.2.1.1.16.1" class="ltx_text ltx_font_typewriter">Please judge if the fact and the paragraph is related. If related, indicate whether the fact is consistent with the paragraph using XML tags: &lt;consistent&gt; or &lt;inconsistent&gt;; If not, use the XML tag &lt;unrelated&gt; to indicate.
<br class="ltx_break"></span></span>
<span id="A1.F13.pic1.2.2.2.1.1.17" class="ltx_p"><span id="A1.F13.pic1.2.2.2.1.1.17.1" class="ltx_text ltx_font_typewriter">For each fact, Let’s think step by step, following the above 2 examples.
<br class="ltx_break"></span></span>
<span id="A1.F13.pic1.2.2.2.1.1.18" class="ltx_p"><span id="A1.F13.pic1.2.2.2.1.1.18.1" class="ltx_text ltx_font_typewriter ltx_font_bold">Evaluation:
<br class="ltx_break"></span></span>
</span></foreignObject></g></g></svg>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 13: </span>Claude-based fact recall evaluation</figcaption>
</figure>
</section>
<section id="A1.SS10" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.10 </span>Coherence Evaluation</h3>

<div id="A1.SS10.p1" class="ltx_para">
<p id="A1.SS10.p1.1" class="ltx_p">Figure <a href="#A1.F14" title="Figure 14 ‣ A.10 Coherence Evaluation ‣ Appendix A Appendix ‣ Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">14</span></a> shows the prompt for coherence evaluation of model’s CoT generation, i.e., whether the final answer is supported by the thought process.</p>
</div>
<figure id="A1.F14" class="ltx_figure"><svg id="A1.F14.pic1" class="ltx_picture" height="738.43" overflow="visible" version="1.1" width="600"><g transform="translate(0,738.43) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#000000" fill-opacity="1.0"><path d="M 0 3.15 L 0 735.28 C 0 737.02 1.41 738.43 3.15 738.43 L 596.85 738.43 C 598.59 738.43 600 737.02 600 735.28 L 600 3.15 C 600 1.41 598.59 0 596.85 0 L 3.15 0 C 1.41 0 0 1.41 0 3.15 Z" style="stroke:none"></path></g><g fill="#FFFFFF" fill-opacity="1.0"><path d="M 1.18 3.15 L 1.18 715.9 L 598.82 715.9 L 598.82 3.15 C 598.82 2.06 597.94 1.18 596.85 1.18 L 3.15 1.18 C 2.06 1.18 1.18 2.06 1.18 3.15 Z" style="stroke:none"></path></g><g fill="#D9D9D9" fill-opacity="1.0"><path d="M 1.18 717.08 L 1.18 735.28 C 1.18 736.37 2.06 737.25 3.15 737.25 L 596.85 737.25 C 597.94 737.25 598.82 736.37 598.82 735.28 L 598.82 717.08 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 9.06 721.01)"><foreignObject width="581.89" height="12.3" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F14.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:420.5pt;">
<span id="A1.F14.pic1.1.1.1.1.1.1" class="ltx_p"><span id="A1.F14.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">CoT Coherence Evaluation Prompt</span></span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 9.06 5.12)"><foreignObject width="581.89" height="706.84" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A1.F14.pic1.2.2.2.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:420.5pt;">
<span id="A1.F14.pic1.2.2.2.1.1.1" class="ltx_p"><span id="A1.F14.pic1.2.2.2.1.1.1.1" class="ltx_text ltx_font_typewriter ltx_font_bold">Human<span id="A1.F14.pic1.2.2.2.1.1.1.1.1" class="ltx_text ltx_font_medium">:
<br class="ltx_break"></span></span></span>
<span id="A1.F14.pic1.2.2.2.1.1.2" class="ltx_p"><span id="A1.F14.pic1.2.2.2.1.1.2.1" class="ltx_text ltx_font_typewriter">Given a question, evaluate whether the thoughts support the provided answer for the question. Answer &lt;support&gt; or &lt;not-support&gt;.
<br class="ltx_break"></span></span>
<span id="A1.F14.pic1.2.2.2.1.1.3" class="ltx_p"><span id="A1.F14.pic1.2.2.2.1.1.3.1" class="ltx_text ltx_font_typewriter">Below are a few examples:
<br class="ltx_break"></span></span>
<span id="A1.F14.pic1.2.2.2.1.1.4" class="ltx_p"><span id="A1.F14.pic1.2.2.2.1.1.4.1" class="ltx_text ltx_font_typewriter">example 1:
<br class="ltx_break"></span></span>
<span id="A1.F14.pic1.2.2.2.1.1.5" class="ltx_p"><span id="A1.F14.pic1.2.2.2.1.1.5.1" class="ltx_text ltx_font_typewriter">Question: Is Ben Nevis taller than Snowdon?
<br class="ltx_break">Thoughts: Snowdon is 1,085 metres (3,560 ft) high. Ben Nevis is 1,345 metres (4,413 ft) high.
<br class="ltx_break">Answer: No
<br class="ltx_break">Support or not: Since the thoughts say Snowdon is 1,085 metres (3,560 ft) high. Ben Nevis is 1,345 metres (4,413 ft) high, then it means that Ben Nevis is taller than Snowdon. So the provided answer to the question is not supported by the thought. &lt;not-support&gt;
<br class="ltx_break"></span></span>
<span id="A1.F14.pic1.2.2.2.1.1.6" class="ltx_p"><span id="A1.F14.pic1.2.2.2.1.1.6.1" class="ltx_text ltx_font_typewriter">example 2:
<br class="ltx_break"></span></span>
<span id="A1.F14.pic1.2.2.2.1.1.7" class="ltx_p"><span id="A1.F14.pic1.2.2.2.1.1.7.1" class="ltx_text ltx_font_typewriter">Question: Is Ben Nevis taller than Snowdon?
<br class="ltx_break">Thoughts: Snowdon is 1,085 metres (3,560 ft) high. Ben Nevis is 1,345 metres (4,413 ft) high.
<br class="ltx_break">Answer: Yes
<br class="ltx_break">Support or not: Since the thoughts say Snowdon is 1,085 metres (3,560 ft) high. Ben Nevis is 1,345 metres (4,413 ft) high, then it means that Ben Nevis is taller than Snowdon. So the provided answer to the question is indeed supported by the thought. &lt;support&gt;
<br class="ltx_break"></span></span>
<span id="A1.F14.pic1.2.2.2.1.1.8" class="ltx_p"><span id="A1.F14.pic1.2.2.2.1.1.8.1" class="ltx_text ltx_font_typewriter">example 3:
<br class="ltx_break"></span></span>
<span id="A1.F14.pic1.2.2.2.1.1.9" class="ltx_p"><span id="A1.F14.pic1.2.2.2.1.1.9.1" class="ltx_text ltx_font_typewriter">Question: Is Houston located more west than Tampa?
<br class="ltx_break">Thoughts: Houston is located in the state of Texas. Tampa is located in the state of Florida. Florida is located in the southeastern United States. Texas is located in the central United States.
<br class="ltx_break">Answer: No
<br class="ltx_break">Support or not: Since the thoughts say Houston is in Texas and Texas in central US, while Tampa is in Florida and Florida is in southeastern US, then Texas is more west to Florida and thus Houston more west than Tampa. The provided answer is thus not supported by the thoughts. &lt;not-support&gt;
<br class="ltx_break"></span></span>
<span id="A1.F14.pic1.2.2.2.1.1.10" class="ltx_p"><span id="A1.F14.pic1.2.2.2.1.1.10.1" class="ltx_text ltx_font_typewriter">example 4:
<br class="ltx_break"></span></span>
<span id="A1.F14.pic1.2.2.2.1.1.11" class="ltx_p"><span id="A1.F14.pic1.2.2.2.1.1.11.1" class="ltx_text ltx_font_typewriter">Question: Is Houston located more west than Tampa?
<br class="ltx_break">Thoughts: Houston is located in the state of Texas. Tampa is located in the state of Florida. Florida is located in the southeastern United States. Texas is located in the central United States.
<br class="ltx_break">Answer: Yes
<br class="ltx_break">Support or not: Since the thoughts say Houston is in Texas and Texas in central US, while Tampa is in Florida and Florida is in southeastern US, then Texas is more west to Florida and thus Houston more west than Tampa. The provided answer is thus indeed supported by the thoughts. &lt;support&gt;
<br class="ltx_break"></span></span>
<span id="A1.F14.pic1.2.2.2.1.1.12" class="ltx_p"><span id="A1.F14.pic1.2.2.2.1.1.12.1" class="ltx_text ltx_font_typewriter">Here is a triple of new question, thought, answer:
<br class="ltx_break">Question: {{question}}
<br class="ltx_break">Thoughts: {{thoughts}}
<br class="ltx_break">Answer: {{answer}}
<br class="ltx_break"></span></span>
<span id="A1.F14.pic1.2.2.2.1.1.13" class="ltx_p"><span id="A1.F14.pic1.2.2.2.1.1.13.1" class="ltx_text ltx_font_typewriter">Please judge if the provided answer is supported using &lt;support&gt; or &lt;not-support&gt; to indicate.
<br class="ltx_break"></span></span>
<span id="A1.F14.pic1.2.2.2.1.1.14" class="ltx_p"><span id="A1.F14.pic1.2.2.2.1.1.14.1" class="ltx_text ltx_font_typewriter ltx_font_bold">Assistant:<span id="A1.F14.pic1.2.2.2.1.1.14.1.1" class="ltx_text ltx_font_medium"></span></span></span>
</span></foreignObject></g></g></svg>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 14: </span>Claude-based CoT coherence evaluation</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2401.17584" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2401.17585" class="ar5iv-text-button ar5iv-severity-error">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2401.17585">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2401.17585" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2401.17586" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Tue Feb 27 07:19:17 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2401.17585
Current approaches of knowledge editing struggle to effectively propagate updates to interconnected facts.
In this work, we delve into the barriers that hinder the appropriate propagation of updated knowledge within these models for accurate reasoning.
To support our analysis, we introduce a novel reasoning-based benchmark –ReCoE(Reasoning-basedCounterfactualEditing dataset)  – which covers six common reasoning schemes in real world.
We conduct a thorough analysis of existing knowledge editing techniques, including input-augmentation, finetuning, and locate-and-edit.
We found that all model editing methods show notably low performance on this dataset, especially in certain reasoning schemes.
Our analysis over the chain-of-thought generation of edited models further uncover key reasons behind the inadequacy of existing knowledge editing methods from a reasoning standpoint, involving aspects onfact-wise editing,fact recallability, andcoherencein generation. We will make our benchmark publicly available.
abstract
Propagation and Pitfalls: Reasoning-based Assessment ofKnowledge Editing through Counterfactual Tasks
Abstract
2401.17585
1IntroductionContemporary language models demonstrate a remarkable capacity to encode extensive factual information, rendering them highly useful as a knowledge base for real-world applications. Yet, the challenge of rapidly outdated knowledge persists, giving rise to a wide range of methods for knowledge updating, such as in-context learning, continual pretraining, locate-and-edit, and meta-learning.Figure 1:An example of reasoning-based assessment for knowledge editing. Existing methods perform well at answering the question of the edited fact, but fail on reasoning with the edited fact.Despite the success of fact-wise editing, recent studiesshow that current model editing methods struggle to effectively propagate updates to interconnected facts.
However, the reason behind the ineffective knowledge propagation remains largely unexplored. Our observation in fact-editing experiments indicates that models respond unexpectedly post-editing. For instance, upon applying MEMITfor fact editing, the model was unable to reliably recall pertinent edited information and produces an incoherent chain-of-thought (CoT) for answering the question, as demonstrated in the MEMIT-based generation example in Figure1.In this work, we undertake an in-depth analysis of this phenomenon, concentrating on three essential competencies necessary for knowledge propagation on reasoning questions after model editing. We place special emphasis on analyzing the results of CoT prompting, which provides explicit reasoning steps that facilitate easier examination. Specifically, in the proposed analytical framework, we measure: (1)  effectiveness of editing individual facts, (2)  accuracy in recalling relevant facts, and (3)  logical coherence of the thought process.To facilitate our investigation, we introduce aReasoning-basedCounterfactualEditing dataset –ReCoE, which covers six different reasoning schemes:superlative,comparative,sorting,counting,aggregation, andsubtraction.
This dataset is designed to more accurately capture the complexities inherent in fact editing tasks.
In contrast to existing knowledge editing benchmarks which predominantly rely on synthetic data, ReCoE incorporates reasoning questions that exhibit a higher degree of naturalness and more accurately reflect the reasoning-based queries encountered in real-world scenarios.We first explored input-augmentation, an approach where new facts are added (prepended)  only in-context, as an informal upper bound of model editing methods.
We then examined model editing methods including finetuning and MEMIT on the Tülu series, which are Llama-based instruction-tuned models of varying sizes.
Results show that all model editing methods achieve notably low performance on the ReCoE benchmark, especially in certain reasoning schemes, with scores close to zero.
Our analysis further unravels the effect of various knowledge editing methods on the reasoning abilities of language models.
We demonstrate that all editing methods result in a significant reduction in fact recall, indicating a key obstacle in effective utilization of the edited knowledge.
Surprisingly, models edited through locate-and-edit methods (i.e., MEMIT)  exhibit a severe decline in their generation coherence, leading to nonsensical outputs, which suggests a substantial deterioration in their fundamental language modeling abilities.
We summarize our contributions of the paper as follows:•We introduce a reasoning-based framework of assessing knowledge editing methods, covering key aspects that enables effective reasoning. Our analysis uncovers essential insights regarding the challenges and limitations associated with knowledge propagation.•We introduce ReCoE, a novel yet challenging reasoning-based counterfactual editing benchmark covering a diverse set of reasoning schemes centered on real-world scenarios.Figure 2:Step-by-step demonstration on dataset construction. There are three different types of lines in the diagram: (1)  straight lines represent data sourced from existing datasets; (2)  dashed lines denote data derived from Claude-generation processes; (3)  zigzag lines symbolize data obtained through the corruption of other data. Group 1 includes superlative, comparative, and sorting questions, where we use “swapping” to create counterfactual facts. Group 2 represents counting, aggregation, and subtraction questions, where we use “altering” to create counterfactual facts. During the editing phase, data is processed either in text form or atomic triplets, contingent upon the specific requirements of the editor.
introduction
Propagation and Pitfalls: Reasoning-based Assessment ofKnowledge Editing through Counterfactual Tasks
1Introduction
2401.17585
2Related Work2.1Model Editing MethodsExisting model editing methods generally fall into four main categories.Finetuning-based methodsThese techniques further finetune the model on new knowledge while minimizing the change in models and catastrophic forgetting. Examples of finetuning-based methods include.Machine learning framing methodsThese approaches treat the editing as a machine learning challenge. They learn hypernetworks (optimizers)  to process model gradients. The goal is to produce an updated model that offers the desired output for the edited point while ensuring minimal prediction changes for other data points. Notable methods include MEND, KnowledgeEditor, SLAG, and CaMeLS.Interpretability-centric methodsThese methods focus on model interpretability. The objective is to pinpoint specific layers and parameters that primarily function for knowledge storage. Once identified, these parameters are then edited, viewing them as linear associative memory storage units. ROME and MEMITare prominent examples.Retrieval-augmented methodsThese techniques leverage retrieval-augmentation to update knowledge in prompting. SERACand MeLLostore new knowledge in memory. When relevant queries arise, they retrieve the pertinent knowledge from this storage, employing input augmentation to adjust the response.2.2Model Editing BenchmarksSeveral benchmarks have been introduced to assess the efficacy of model editing.introduced the COUNTERFACT dataset, specifically designed to evaluate the successful incorporation of counterfactual knowledge. This evaluation is segmented into three main criteria: (1)  Efficacy determines if a particular piece of knowledge has been successfully integrated into the model, typically measured by conditional generation that mirrors the exact text used for editing (2)  Paraphrase assesses the model’s capability to generalize to paraphrased versions of the editing text, ensuring the retention of the original knowledge(3)  Specificity ensures that the model remains unchanged with respect to irrelevant knowledge. There are many other datasets including Zero-Shot Relation Extraction (zsRE) , WikiGen, T-REx-100 & T-REx-1000, MMEdit(multi-modal model editing) ,etc.To evaluate knowledge propagation,andintroduced ECBD and MQuAKE benchmarks respectively. ECBD measures the perplexity of a passage relevant to target knowledge (entity-relevant)  before and after editing. Though it presents the difficulty of knowledge propagation, the context it evaluates on has a non-deterministic connection to the edited knowledge. MQuAKE employs multi-hop QA questions to gauge the model’s accuracy after editing part or all of its reasoning components. However, it exclusively encompasses compositional questions generated from ChatGPT wherein the precise segments of knowledge required for effective propagation are overtly articulated within the question. Such format may not necessarily mirror real-world scenarios where the reasoning component could be implicit from the question.In our research, we focus on factual knowledge editing and tackle the prevailing limitations on propagation observed in contemporary benchmarks. Our benchmark ReCoE incorporates a diverse set of 6 reasoning schemes, featuring more organic queries that mirror real-world scenarios. Additionally, we employ a reasoning-based framework to elucidate the underlying challenges of knowledge propagation. Drawing from our discoveries, we aspire to provide valuable insights that will guide and shape the future trajectories of model editing techniques in a more informed manner.SchemeExample QA pairData sourcesuperlativeQuestion: What is the largest city in the province of BritishExistent datasetsColumbia, Canada?Answer: VancouversortingQuestion: Sort the following cities based on their city size from smallClaude-generation based on superlativeto large: Nanaimo, Victoria, Seattle.Answer: Victoria, Nanaimo, SeattlecomparativeQuestion: What is the name of the distilled spirit that has an alcoholExistent Dataset,content less than or equal to 35.0?Answer: MekhongClaude-generation based on sortingcountingQuestion: How many symphonies were composed by Ludwig vanManually written,Beethoven?Answer: 9Claude-generationaggregationQuestion: How many states/provinces are there in North America,Claude-generation based on countingi.e. United States, Canada, and Mexico?Answer: 92subtractionQuestion: How many provinces does Mexico have more than Canada?Claude-generation based on countingAnswer: 22Table 1:Example question-answer pair and data source of each reasoning scheme. Existent datasets include GrailQA, NaturalQuestions, ComplexWebQuestions, FreebaseQA.
body
Propagation and Pitfalls: Reasoning-based Assessment ofKnowledge Editing through Counterfactual Tasks
2Related Work
2401.17585
3ReCoE DatasetWe employ a hybrid-synthetic approach that combines existing complex QA datasets and LLM-assisted data synthesizing to create the ReCoE dataset.
The dataset is designed to evaluate counterfactual editing across a broad spectrum of reasoning schemes:superlative,comparative,sorting,counting,aggregation, andsubtraction.A typical datapoint within this dataset encapsulates five key components:•: Question that corresponds to each of our defined reasoning schemes•: Answer with aliases•: Set of facts that supports the answer () •: Counterfactual answer with aliases•: Set of counterfactuals that supports the counterfactual answer () These components allow us to assess knowledge propagation by editing a language model with the set of counterfactuals, and testing if the edited model is able to flip its original answer ()  towards the counterfactual answer ()  through reasoning.In this section, we provide a comprehensive overview of the dataset and delve into the nuances of its construction methodology.3.1ReCoE: QA Pairs ConstructionTable1presents examples of QA pairs for each reasoning scheme and the corresponding data source. Table2presents the dataset statistics including the number of examples and atomic facts offor each reasoning scheme.The construction of the benchmark starts from basic QA pairs. QA pairs of all superlative data and part of comparative data are from existent QA datasets. QA pairs of counting data are partially hand-written. Based on these datapoints, we create QA pairs for sorting questions, more comparative questions, aggregation questions and subtraction questions synthetically.Sortingquestions are constructed by prompting Claude based on superlative questions with the following 4 steps. Taking the superlative QA pair in Table1as an example:Step 1:Prompt Claude to generate the aspect that the sentence is discussing.\MakeFramed\FrameRestoreaspect: city population or city size\endMakeFramedStep 2:Given the QA pair, generate 10 similar or relevant entities to the entity/subject of the question.\MakeFramed\FrameRestorerelevant entities:Large cities in Canada:
Toronto, Montreal, Calgary, OttawaCities in British Columbia:
Victoria, Nanaimo, NelsonCoastal cities in North America:
Seattle, San Francisco, Los Angeles, Portland\endMakeFramedStep 3:Given the question and generated aspect (Step 1) , prompt Claude to generate relevant facts about each generated entity.\MakeFramed\FrameRestoreentity facts:Toronto: a recorded population of 2,794,356 …Victoria: city size 19.47 km2(7.52 sq mi)  …Nanaimo: city size 35.25 mi2…Seattle: land area of 83.9 square miles (217.3 km2)  ……\endMakeFramedStep 4:Generate 5 sorting questions with the answer using generated entities and their corresponding facts.\MakeFramed\FrameRestoreexample sorting questions with answers:Question: Sort the following cities based on their city size from small to large: Nanaimo, Victoria, Seattle.Answer: Victoria, Nanaimo, Seattle.\endMakeFramedBased on the existent superlative sub-dataset, over 10k sorting datapoints can be constructed in the above manner. Since the generated facts and answers based on the facts are not necessarily correct, we leveragemulti-agent debatewith Claude agents to double-check the generated QA pairs as initial quality control: QA pairs are excluded if the agents, post-debate, converge on different answers to the question.Comparativequestions are partially (192)  selected from existent datasets. We generate 1,000 more QA pairs with Claude based on generated sorting questions by transforming a sorting question to a comparative question.Countingquestions are from 8 different domains: astronomy, book, geography, legal, movie, music, sport, and war. We manually create 5 QA questions for each domain and then prompt Claude to generate more such QA pairs following the examples. Multi-agent debate is again adopted to filter out inaccurate QA pairs.Aggregation & Subtractionquestions are derived from the counting questions and retains the same 8 domains. An aggregation question is formulated by combining two or more counting questions. Below is a QA pair example:\MakeFramed\FrameRestoreQuestion: How many states/provinces are there in North America, i.e. United States, Canada, and Mexico?Answer: 92\endMakeFramedTo avoid incongruous or unnatural questions, we employ two strategies:1.Counting questions to be combined are sampled from the same domain.2.Filter out unnatural questions with Claude by two criteria: a)  whether the question is fluent; b)  whether the entities mentioned in the question are compatible in type. For instance, the number of satellites of Earth and that of Mars are compatible for aggregation, while the number of constellations recognized by the International Astronomical Union and the number of Earth’s satellites are not.SchemeExamplesAtomic factssuperlative1,1727,248comparative1,1717,218counting6431,309sorting1,0416,128aggregation509506subtraction501533Total5,03722,942Table 2:Statistics of the ReCoE dataset.3.2ReCoE: Counterfactual ConstructionAfter obtaining all the QA pairs for each reasoning scheme, we need to create facts and counterfactual facts to complete each datapoint. The dataset is constructed entirely automatically with the main construction steps illustrated in Figure2.
The construction involves 4 steps:Step 1: After collecting and generating QA pairs, we prompt Claude to create a counterfactual answer. For instance, if the question is aboutwhich team between the Chicago Blackhawks and Pittsburgh Penguins has won more Stanley Cup championships, and the answer is theChicago Blackhawks, the counterfactual answer would be thePittsburgh Penguins.Step 2: Claude is prompted to generate relevant facts about entities mentioned in the answer and counterfactual answer. These facts are then verified for accuracy using a retrieval-augmented method by retrieving relevant paragraphs from Wikipedia using Contrieverand corrected if necessary. We filter datapoints to ensure that all QA pairs are valid and (question, counterfactual answer)  pairs are invalid.Step 3: To generate counterfactual facts, if a question is superlative, comparative, or sorting (Group 1) , weswapthe subjects of supporting facts between those related to the actual answer and those pertaining to the counterfactual answer. This process is conducted while eliminating any datapoints that could introduce contradictions in the counterfactual facts generated as a result of this subject swapping; if the question is counting, aggregation, or sorting (Group 2) , wealterthe facts for the answer to obtain the counterfactual facts for the counterfactual answer while maintaining consistency.Step 4: All sentences in the facts and counterfactual facts are broken down into atomic formats for easier editing.A concrete example in ReCoE is presented in AppendixA.1.3.3Fact RepresentationFigure 3:Comparison between fact representations in MQuAKEand ReCoE.Current benchmarks primarily focus on datasets such as zsRE, COUNTERFACT, and MQuAKE.
In these datasets, each fact is represented in a clear and unambiguous form as a (subject, relation, object)  triplet.
In contrast, our dataset diverges from this norm, featuring facts more commonly encountered in real-world scenarios, typically represented in an OpenIE style.
This style introduces a wider variety of complexities.
As illustrated in Figure3, the atomic facts in ReCoE may involve complex subjects or relations. Moreover, a single relation applied to a subject could correspond to non-unique objects.
These examples highlight the nuanced nature of knowledge updates. We will further discuss the challenges these OpenIE-form fact representations poses to the effectiveness of locate-and-edit methods in a later section (Section5.4.1) .ModelPromptReasoning SchemesuperlativecomparativecountingsortingaggregationsubtractionAverage7bdirect12.5510.9331.7313.4510.2210.3814.67CoT28.6357.7340.9010.0927.3127.9432.2813bdirect20.2133.1333.4419.605.3011.7820.43CoT30.8862.6043.2316.0426.5231.5435.3933bdirect31.1055.4246.5035.066.4811.9830.96CoT37.1971.7355.3736.1241.2641.5246.64Table 3:QA accuracy of knowledge probing. Both CoT prompting and model scaling significantly improved the overall performance.ModelEditorPromptReasoning SchemesuperlativecomparativecountingsortingaggregationsubtractionAverage7bInputAugdirect50.3041.4123.537.867.6917.3124.68CoT54.0753.9934.6014.2920.8610.0031.30QLoRAdirect0.6014.069.8011.435.777.698.23CoT3.4151.048.377.625.048.5714.01MEMITdirect0.0034.386.373.940.007.698.73CoT0.003.555.323.450.004.292.7713bInputAugdirect59.8551.5524.1915.693.7011.8627.81CoT71.7866.3060.4318.5625.9325.3244.72QLoRAdirect2.2319.075.1212.753.708.478.56CoT4.6230.838.2714.373.706.9611.46MEMITdirect0.3741.4911.632.050.003.399.82CoT0.2418.697.910.550.745.065.5333bInputAugdirect82.3774.8823.0824.1118.1811.6739.05CoT73.3384.8855.9032.9846.6735.1054.81QLoRAdirect3.1412.946.0211.783.033.336.71CoT4.2424.8812.6416.765.2411.0612.47Table 4:QA accuracy of each reasoning scheme, with different editors, model sizes, and prompting strategies. MEMIT was not implemented on 33b models due to GPU memory constraints. InputAug (upper bound)  shows overall reasonable performance, with consistent benefits from CoT prompting and model scaling. Both QLoRA and MEMIT significantly underperform compared to InputAug, with MEMIT showing particularly low performance in certain reasoning schemes like superlative and aggregation. While QLoRA exhibits some improvement from CoT prompting, MEMIT’s performance remains consistently poor across all scenarios.
body
Propagation and Pitfalls: Reasoning-based Assessment ofKnowledge Editing through Counterfactual Tasks
3ReCoE Dataset
2401.17585
4Experiment4.1Language ModelsWe utilize the Tülu seriesas the base language models to assess knowledge editing approaches. Among various options available, we find Tülu a good candidate for our study for the following reasons: 1)  Tülu models are instruction-tuned and provide well-structured responses to user instructions, simplifying our factual evaluation process;111Their pretrained counterparts, i.e., Llama, however, would require careful output parsing for obtaining answers to ensure reliable evaluation.2)  The Tülu series includes models of varying sizes, enabling us to explore the impact of model scaling on effective knowledge editing.4.2Knowledge Editing MethodsWe evaluated the following three representative knowledge editing methods222We do not evaluate meta-learning based methods such as MEND because currently, these methods are not for massive editing and editing massive knowledge leads to low efficacy..Input-augmentationis an inference-time editing method that appends the counterfactual facts to the question as part of the prompt. Therefore, it does not modify the model weights, but relies on model’s capability to perform reasoning from explicit context. It is considered as an upper boundfor model editing.Finetuning (QLoRA) performs gradient descent on the new facts to update model parameters. As we are tuning models up to 33 billion parameters, we adopt the parameter-efficient finetuning method QLoRAfor the sake of computational and time efficiency.MEMITfirst localizes the factual knowledge in a range of layers in the transformer architecture and then updates the feedforward modules in the layers to insert a massive amount of new facts in the form of triplets. We used the implementation from.4.3Experiment SettingFactual knowledge probingWe employ both direct prompting and CoT prompting strategies to probe model’s proficiency of factual knowledge mastery and reasoning using the ReCoE dataset.
The objective is to ensure that the dataset presents a balanced level of difficulty – neither overly challenging nor too simplistic. This balance is crucial so that the language model under investigation achieves an acceptable level of accuracy for conducting meaningful counterfactual editing experiments, where we observe the model’s transition from correct to counterfactual responses. Moreover, the dataset needs to present a degree of challenge to make it a valuable asset for further research on more advanced language models.Knowledge EditingWe evaluate model’s QA performance on the (question, counterfactual answer)  pairs of each reasoning scheme post-editing. We use thecorrect_flipas the primary metric, which measures the percentage of model’s predictions that correctly transition from the original answer to the counterfactual answer.333correct_flipmainly measures the efficacy ofknowledge update, rather thanknowledge insert. We focus on the “update” setting in this work to facilitate our later analysis of the changes in reasoning capabilities post-editing.Figure 4:Comparison on fact-wise perplexity over facts ()  and counterfactuals ()  before and after editing using QLoRA and MEMIT (7B) . A successful fact-wise edit is indicated by a transition from the upper-triangle region whereto the lower-triangle region. QLoRA-based finetuning demonstrate notable effectiveness, in contrast to MEMIT. Similar trends are observed with 13b and 33b models, as detailed in AppendixA.8.4.4Experiment ResultTable3displays the outcomes of the knowledge probing exercise. The results clearly demonstrate the significant impact of model scaling and the beneficial role of CoT. Table4summarizes thecorrect_flipresults of InputAug (input-augmentation) , QLoRA-based finetuning, and MEMIT.
InputAug involves incorporating counterfactual information into the context, where both model scaling and CoT are shown to be beneficial. Input augmentation is often treated as the upper bound for model editing. But we can see that the performance for aggregation and subtraction is still unsatisfying, below 50%.
Both QLoRA and MEMIT editing significantly underperform InputAug across all model scales and prompting strategies, indicating failed knowledge propagation of these methods.
Interestingly, QLoRA-based finetuning, despite its deteriorating performance, can still benefit from CoT prompting and model scaling. In contrast, MEMIT consistently failed, indicating a significant deterioration in the model’s reasoning capability.
body
Propagation and Pitfalls: Reasoning-based Assessment ofKnowledge Editing through Counterfactual Tasks
4Experiment
2401.17585
5AnalysisTo comprehensively evaluate how certain knowledge editing methodologies impact model’s capability that leads to ineffectiveness in knowledge propagation, our analysis encompasses three key dimensions:fact-wise editing effectiveness,fact recall accuracy, andlogical coherencein model’s generation.
We assume the reasoning process follows a retrieve-and-generate regime. Formally,whereis the edited LM.
The fact recall component requires 1)  each fact withinto be effectively edited; and 2)  edited model is able to recall these edited facts through generation.
The coherent generation component further requires the generated answer to be logically coherent with the retrieved facts.5.1Fact-wise Editing EffectivenessThis dimension examines the basic efficacy of editing methods. It assesses whether the applied edits achieve their intended modifications successfully, which constitutes the foundational requirement for any knowledge editing approach. Definingandas the averaged perplexity of the facts and the counterfactuals associated to a)  pair, and-) , an effective editing over the factsto its counterfactual counterpartcan be evaluated by the indicator function:. This definition of successful editing stipulates that the perplexity of counterfactual sentences must be lower than that of factual sentences. Furthermore, in cases where the perplexity of counterfactual sentences is already lower before editing, it necessitates an even greater disparity between the two.Fact-wise editing performed by QLoRA demonstrates a high degree of effectiveness, in contrast to MEMIT. In general,is lower thanpost-editing but MEMIT has the adverse effect of increasing the overall perplexity within the model. Figure4demonstrates the fact-wise editing effectiveness in 7b model using QLoRA and MEMIT. Detailed results are presented in Table6in AppendixA.8.5.2Fact RecallsuperlativecomparativecountingsortingaggregationsubtractionRelatedness (7b) Pre-EditQLoRAMEMITsuperlativecomparativecountingsortingaggregationsubtractionConsistency (7b) superlativecomparativecountingsortingaggregationsubtractionRelatedness (13b) superlativecomparativecountingsortingaggregationsubtractionConsistency (13b) Figure 5:Fact recall pre- and post-editing: measured by the relatedness and consistency of the decomposed atomic facts in CoT generation against the edited counterfactual facts. While both QLoRA and MEMIT maintains a reasonable degree of relatedness (with QLoRA outperforming MEMIT) , there is a significant decline in factual consistency of both methods.Assuming successful fact-wise editing, we then explore the model’s proficiency in recalling and applying these modifications in reasoning tasks. This involves an in-depth analysis of the model’s ability to retrieve and utilize relevant information correctly. We focus on evaluating the relatedness and consistency of information within the CoT response against the counterfactual facts. The evaluation metrics are defined as:•Relatedness: this metric assesses how relevant the facts generated by the model are to the designated fact/counterfactual set.•Consistency: this metric calculates the proportion of the model’s generated facts that align factually with the fact/counterfactual set.We leverage Claude with dedicated few-shot demonstrations for automatic evaluation. Detailed prompt can be found in AppendixA.9.Results are presented in Figure.5. Regarding relatedness, different editing methods show different impacts on the model. In the case of QLoRA, the edited model retains the capability to recall information across different schemes. However, MEMIT shows negative impacts in superlative, comparative, and sorting. In some cases, relatedness post-editing surpasses that of pre-editing. This could be attributed to the introduction of new facts into the model that it previously lacked.However, both QLoRA-edited and MEMIT-edited models show low consistency results, indicating that they are unable to leverage the edited knowledge in actual use. MEMIT-edited models are worse than QLoRA-edited. It’s important to note that this consistency doesn’t correlate well with the fact-wise editing effectiveness. This disparity may stem from a lack of generalization during the editing process. Essentially, the model seems to simply memorize the newly edited fact, lacking the ability to extend this understanding to different manifestations of the same concept.5.3Logical CoherenceSchemeModelPre-editQLoRAMEMITsuperlative7b89.084.28.913b90.185.04.1comparative7b73.852.50.713b80.274.14.8counting7b91.192.424.013b98.094.132.4sorting7b90.585.443.113b90.487.944.3aggregation7b92.589.03.613b88.287.25.2subtraction7b87.284.810.013b91.492.61.3Average7b87.481.415.113b89.786.815.4Table 5:Coherence of post-editing chain-of-thought generations: percentage of coherent CoT responses among all examples. Coherence is determined by whether the final answer is logically supported by the recalled facts in CoT.We investigate whether the logical reasoning capacity of the model is negatively impacted. This is gauged by the coherence of the generated CoT response, specifically evaluating whether the inferred evidence and thought process adequately support the final answer.Table5presents results that show a discernible, albeit not substantial, decrease in the QLoRA-edited models. However, there is a surprisingly significant decline in MEMIT-edited models. This indicates a substantial loss of fundamental language modeling abilities, likely due to the catastrophic forgetting associated with MEMIT editing.5.4Discussion5.4.1QLoRA vs. MEMITFor QLoRA, we observe that while it adequately supports fact-wise editing and generally preserves logical coherence, its primary deficiency lies in the retrieval of edited facts. In LLM, the elicitation of knowledge depends heavily on appropriate prompting techniques while our approach involves merely fine-tuning LLMs with atomic fact sentences. Consequently, a potential future direction may involve enriching these atomic facts with more comprehensive contexts prior to their utilization in fine-tuning, as it should enable the model to accurately recall information in response to a diverse set of prompts.In contrast, the MEMIT model exhibits a decline in all three assessed abilities: fact-wise editing, fact recall, and coherence. Given that our dataset comprises non-synthetic reasoning questions, which often include complex subjects (e.g., an event)  and relations, non-unique objects (Figure3) , the underperformance of MEMIT suggests its current inadequacy in handling real-world factual knowledge. Notably, MEMIT’s most pronounced deficiencies lie in its ability to recall facts and, critically, in maintaining coherence. This observation highlights that the functionalities of edited neurons extend beyond mere fact storage, challenging the assertions made in previous studies.5.4.2Effect of Model ScalingKnowledgeProbeInputAugQLoRAMEMIT7b13b33bFigure 6:Effect of model scaling. The metric (y-axis)  refers to averagedaccuracyover all reasoning schemes for KnowledgeProb, andcorrect_flipfor editing approaches.The impact of model scaling is a critical factor in both original knowledge probing and the input-augmentation approach, which are shown in Figure6, echoes with current studies that larger models inherently possess a more extensive knowledge base and demonstrate superior reasoning capabilities.However, experiments in this research reveal that upon editing new knowledge into these models, the size of the model does not correspond to enhanced performance in several dimensions, as shown in Figure5and Table5. Specifically, larger models do not exhibit (1)  increased factual effectiveness, (2)  improved ability in retrieving facts during chain-of-thought processes in terms of relatedness and consistency, and (3)  more coherent chain-of-thought performance. In summary, during the model editing phase, the size of the model does not inherently confer any advantageous properties. Consequently, we have not detected any notable improvements attributable to model scaling, such as facilitation of the editing process or provision of inherent advantages.
body
Propagation and Pitfalls: Reasoning-based Assessment ofKnowledge Editing through Counterfactual Tasks
5Analysis
2401.17585
6ConclusionIn this study, we have developed a novel benchmark, ReCoE, which leverages counterfactual reasoning and is grounded in non-synthetic data for evaluating model editing. Our analysis reveals significant challenges in existing knowledge editing approaches, particularly in their ability to effectively propagate new facts for coherent reasoning. Through this investigation, we have identified key areas where these methods falter. Our work provides a clear direction for future research in this field, aiming to enhance the efficacy and reliability of knowledge editing in computational models.
conclusion
Propagation and Pitfalls: Reasoning-based Assessment ofKnowledge Editing through Counterfactual Tasks
6Conclusion