<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2202.05262] Locating and Editing Factual Associations in GPT</title><meta property="og:description" content="We analyze the storage and recall of factual associations in autoregressive transformer language models, finding evidence that these associations correspond to localized, directly-editable computations. We first develo…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Locating and Editing Factual Associations in GPT">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Locating and Editing Factual Associations in GPT">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2202.05262">

<!--Generated on Wed Mar 13 12:31:56 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">Locating and Editing Factual Associations in GPT</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Kevin Meng 
<br class="ltx_break">MIT CSAIL 
<br class="ltx_break">&amp;David Bau<sup id="id2.2.id1" class="ltx_sup">∗</sup> 
<br class="ltx_break">Northeastern University 
<br class="ltx_break">&amp;Alex Andonian 
<br class="ltx_break">MIT CSAIL 
<br class="ltx_break">&amp;Yonatan Belinkov 
<br class="ltx_break">Technion – IIT 
<br class="ltx_break">
</span><span class="ltx_author_notes">Equal contribution. Correspondence to <span id="id3.3.id1" class="ltx_text ltx_font_typewriter">mengk@mit.edu</span>, <span id="id4.4.id2" class="ltx_text ltx_font_typewriter">davidbau@northeastern.edu</span>.Supported by the Viterbi Fellowship in the Center for Computer Engineering at the Technion.</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id5.id1" class="ltx_p">We analyze the storage and recall of factual associations in autoregressive transformer language models, finding evidence that these associations correspond to localized, directly-editable computations. We first develop a causal intervention for identifying neuron <span id="id5.id1.1" class="ltx_text ltx_font_italic">activations</span> that are decisive in a model’s factual predictions. This reveals a distinct set of steps in middle-layer feed-forward modules that mediate factual predictions while processing subject tokens. To test our hypothesis that these computations correspond to factual association recall, we modify feed-forward <span id="id5.id1.2" class="ltx_text ltx_font_italic">weights</span> to update specific factual associations using Rank-One Model Editing (ROME). We find that ROME is effective on a standard zero-shot relation extraction (zsRE) model-editing task. We also evaluate ROME on a new dataset of difficult counterfactual assertions, on which it simultaneously maintains both specificity and generalization, whereas other methods sacrifice one or another. Our results confirm an important role for mid-layer feed-forward modules in storing factual associations and suggest that direct manipulation of computational mechanisms may be a feasible approach for model editing. The code, dataset, visualizations, and an interactive demo notebook are available
at <a target="_blank" href="https://rome.baulab.info/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://rome.baulab.info/</a>.</p>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Where does a large language model store its facts? In this paper, we report evidence that factual associations in GPT correspond to a localized computation that can be directly edited.</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">Large language models can predict factual statements about the world <cite class="ltx_cite ltx_citemacro_citep">(Petroni et al., <a href="#bib.bib30" title="" class="ltx_ref">2019</a>; Jiang et al., <a href="#bib.bib21" title="" class="ltx_ref">2020</a>; Roberts et al., <a href="#bib.bib34" title="" class="ltx_ref">2020</a>)</cite>. For example, given the prefix “<em id="S1.p2.1.1" class="ltx_emph ltx_font_italic">The Space Needle is located in the city of</em>,” GPT will reliably predict the true answer: “<em id="S1.p2.1.2" class="ltx_emph ltx_font_italic">Seattle</em>” (Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>a). Factual knowledge has been observed to emerge in both autoregressive GPT models <cite class="ltx_cite ltx_citemacro_citep">(Radford et al., <a href="#bib.bib32" title="" class="ltx_ref">2019</a>; Brown et al., <a href="#bib.bib7" title="" class="ltx_ref">2020</a>)</cite> and masked BERT models <cite class="ltx_cite ltx_citemacro_citep">(Devlin et al., <a href="#bib.bib11" title="" class="ltx_ref">2019</a>)</cite>.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">In this paper, we investigate how such factual associations are stored within GPT-like autoregressive transformer models. Although many of the largest neural networks in use today are autoregressive, the way that they store knowledge remains under-explored. Some research has been done for masked models <cite class="ltx_cite ltx_citemacro_citep">(Petroni et al., <a href="#bib.bib30" title="" class="ltx_ref">2019</a>; Jiang et al., <a href="#bib.bib21" title="" class="ltx_ref">2020</a>; Elazar et al., <a href="#bib.bib12" title="" class="ltx_ref">2021a</a>; Geva et al., <a href="#bib.bib18" title="" class="ltx_ref">2021</a>; Dai et al., <a href="#bib.bib9" title="" class="ltx_ref">2022</a>; De Cao et al., <a href="#bib.bib10" title="" class="ltx_ref">2021</a>)</cite>, but GPT has architectural differences such as unidirectional attention and generation capabilities that provide an opportunity for new insights.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">We use two approaches. First, we trace the causal effects of hidden state <span id="S1.p4.1.1" class="ltx_text ltx_framed ltx_framed_underline">activations</span> within GPT using causal mediation analysis <cite class="ltx_cite ltx_citemacro_citep">(Pearl, <a href="#bib.bib28" title="" class="ltx_ref">2001</a>; Vig et al., <a href="#bib.bib38" title="" class="ltx_ref">2020b</a>)</cite> to identify the specific modules that mediate recall of a fact about a subject (Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>). Our analysis reveals that feedforward MLPs at a range of middle layers are decisive when processing the last token of the subject name (Figures <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>b,<a href="#S2.F2" title="Figure 2 ‣ 2 Interventions on Activations for Tracing Information Flow ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>b,<a href="#S2.F3" title="Figure 3 ‣ 2.2 Causal Tracing Results ‣ 2 Interventions on Activations for Tracing Information Flow ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>).</p>
</div>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.1" class="ltx_p">Second, we test this finding in model <span id="S1.p5.1.1" class="ltx_text ltx_framed ltx_framed_underline">weights</span> by introducing a Rank-One Model Editing method (ROME) to alter the parameters that determine a feedfoward layer’s behavior at the decisive token. Despite the simplicity of the intervention, we find that ROME is similarly effective to other model-editing approaches on a standard zero-shot relation extraction benchmark (Section <a href="#S3.SS2" title="3.2 Evaluating ROME: Zero-Shot Relation Extraction (zsRE) ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a>).</p>
</div>
<div id="S1.p6" class="ltx_para">
<p id="S1.p6.1" class="ltx_p">To evaluate ROME’s impact on more difficult cases, we introduce a dataset of counterfactual assertions (Section <a href="#S3.SS3" title="3.3 Evaluating ROME: Our CounterFact Dataset ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3</span></a>) that would not have been observed in pretraining. Our evaluations (Section <a href="#S3.SS4" title="3.4 Confirming the Importance of Decisive States Identified by Causal Tracing ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.4</span></a>) confirm that midlayer MLP modules can store factual associations that generalize beyond specific surface forms, while remaining specific to the subject. Compared to previous fine-tuning <cite class="ltx_cite ltx_citemacro_citep">(Zhu et al., <a href="#bib.bib43" title="" class="ltx_ref">2020</a>)</cite>, interpretability-based <cite class="ltx_cite ltx_citemacro_citep">(Dai et al., <a href="#bib.bib9" title="" class="ltx_ref">2022</a>)</cite>, and meta-learning <cite class="ltx_cite ltx_citemacro_citep">(Mitchell et al., <a href="#bib.bib27" title="" class="ltx_ref">2021</a>; De Cao et al., <a href="#bib.bib10" title="" class="ltx_ref">2021</a>)</cite> methods, ROME achieves good generalization and specificity simultaneously, whereas previous approaches sacrifice one or the other.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2202.05262/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="207" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S1.F1.3.1.1" class="ltx_text" style="font-size:90%;">Figure 1</span>: </span><span id="S1.F1.4.2" class="ltx_text ltx_font_bold" style="font-size:90%;">Causal Traces<span id="S1.F1.4.2.1" class="ltx_text ltx_font_medium"> compute the causal effect of neuron activations by running the network twice: (a) once normally, and (b) once where we corrupt the subject token and then (c) restore selected internal activations to their clean value. (d) Some sets of activations cause the output to return to the original prediction; the light blue path shows an example of information flow. The causal impact on output probability is mapped for the effect of (e) each hidden state on the prediction, (f) only MLP activations, and (g) only attention activations.</span></span></figcaption>
</figure>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Interventions on <span id="S2.1.1" class="ltx_text ltx_framed ltx_framed_underline">Activations</span> for Tracing Information Flow</h2>

<div id="S2.p1" class="ltx_para">
<p id="S2.p1.7" class="ltx_p">To locate facts within the parameters of a large pretrained autoregressive transformer, we begin by analyzing and identifying the specific hidden states that have the strongest causal effect on predictions of individual facts.
We represent each fact as a knowledge tuple <math id="S2.p1.1.m1.3" class="ltx_Math" alttext="t=\left(s,r,o\right)" display="inline"><semantics id="S2.p1.1.m1.3a"><mrow id="S2.p1.1.m1.3.4" xref="S2.p1.1.m1.3.4.cmml"><mi id="S2.p1.1.m1.3.4.2" xref="S2.p1.1.m1.3.4.2.cmml">t</mi><mo id="S2.p1.1.m1.3.4.1" xref="S2.p1.1.m1.3.4.1.cmml">=</mo><mrow id="S2.p1.1.m1.3.4.3.2" xref="S2.p1.1.m1.3.4.3.1.cmml"><mo id="S2.p1.1.m1.3.4.3.2.1" xref="S2.p1.1.m1.3.4.3.1.cmml">(</mo><mi id="S2.p1.1.m1.1.1" xref="S2.p1.1.m1.1.1.cmml">s</mi><mo id="S2.p1.1.m1.3.4.3.2.2" xref="S2.p1.1.m1.3.4.3.1.cmml">,</mo><mi id="S2.p1.1.m1.2.2" xref="S2.p1.1.m1.2.2.cmml">r</mi><mo id="S2.p1.1.m1.3.4.3.2.3" xref="S2.p1.1.m1.3.4.3.1.cmml">,</mo><mi id="S2.p1.1.m1.3.3" xref="S2.p1.1.m1.3.3.cmml">o</mi><mo id="S2.p1.1.m1.3.4.3.2.4" xref="S2.p1.1.m1.3.4.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.p1.1.m1.3b"><apply id="S2.p1.1.m1.3.4.cmml" xref="S2.p1.1.m1.3.4"><eq id="S2.p1.1.m1.3.4.1.cmml" xref="S2.p1.1.m1.3.4.1"></eq><ci id="S2.p1.1.m1.3.4.2.cmml" xref="S2.p1.1.m1.3.4.2">𝑡</ci><vector id="S2.p1.1.m1.3.4.3.1.cmml" xref="S2.p1.1.m1.3.4.3.2"><ci id="S2.p1.1.m1.1.1.cmml" xref="S2.p1.1.m1.1.1">𝑠</ci><ci id="S2.p1.1.m1.2.2.cmml" xref="S2.p1.1.m1.2.2">𝑟</ci><ci id="S2.p1.1.m1.3.3.cmml" xref="S2.p1.1.m1.3.3">𝑜</ci></vector></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.1.m1.3c">t=\left(s,r,o\right)</annotation></semantics></math> containing the subject <math id="S2.p1.2.m2.1" class="ltx_Math" alttext="s" display="inline"><semantics id="S2.p1.2.m2.1a"><mi id="S2.p1.2.m2.1.1" xref="S2.p1.2.m2.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="S2.p1.2.m2.1b"><ci id="S2.p1.2.m2.1.1.cmml" xref="S2.p1.2.m2.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.2.m2.1c">s</annotation></semantics></math>, object <math id="S2.p1.3.m3.1" class="ltx_Math" alttext="o" display="inline"><semantics id="S2.p1.3.m3.1a"><mi id="S2.p1.3.m3.1.1" xref="S2.p1.3.m3.1.1.cmml">o</mi><annotation-xml encoding="MathML-Content" id="S2.p1.3.m3.1b"><ci id="S2.p1.3.m3.1.1.cmml" xref="S2.p1.3.m3.1.1">𝑜</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.3.m3.1c">o</annotation></semantics></math>, and relation <math id="S2.p1.4.m4.1" class="ltx_Math" alttext="r" display="inline"><semantics id="S2.p1.4.m4.1a"><mi id="S2.p1.4.m4.1.1" xref="S2.p1.4.m4.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="S2.p1.4.m4.1b"><ci id="S2.p1.4.m4.1.1.cmml" xref="S2.p1.4.m4.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.4.m4.1c">r</annotation></semantics></math> connecting the two.
Then to elicit the fact in GPT, we provide a natural language prompt <math id="S2.p1.5.m5.1" class="ltx_Math" alttext="p" display="inline"><semantics id="S2.p1.5.m5.1a"><mi id="S2.p1.5.m5.1.1" xref="S2.p1.5.m5.1.1.cmml">p</mi><annotation-xml encoding="MathML-Content" id="S2.p1.5.m5.1b"><ci id="S2.p1.5.m5.1.1.cmml" xref="S2.p1.5.m5.1.1">𝑝</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.5.m5.1c">p</annotation></semantics></math> describing <math id="S2.p1.6.m6.2" class="ltx_Math" alttext="(s,r)" display="inline"><semantics id="S2.p1.6.m6.2a"><mrow id="S2.p1.6.m6.2.3.2" xref="S2.p1.6.m6.2.3.1.cmml"><mo stretchy="false" id="S2.p1.6.m6.2.3.2.1" xref="S2.p1.6.m6.2.3.1.cmml">(</mo><mi id="S2.p1.6.m6.1.1" xref="S2.p1.6.m6.1.1.cmml">s</mi><mo id="S2.p1.6.m6.2.3.2.2" xref="S2.p1.6.m6.2.3.1.cmml">,</mo><mi id="S2.p1.6.m6.2.2" xref="S2.p1.6.m6.2.2.cmml">r</mi><mo stretchy="false" id="S2.p1.6.m6.2.3.2.3" xref="S2.p1.6.m6.2.3.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.p1.6.m6.2b"><interval closure="open" id="S2.p1.6.m6.2.3.1.cmml" xref="S2.p1.6.m6.2.3.2"><ci id="S2.p1.6.m6.1.1.cmml" xref="S2.p1.6.m6.1.1">𝑠</ci><ci id="S2.p1.6.m6.2.2.cmml" xref="S2.p1.6.m6.2.2">𝑟</ci></interval></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.6.m6.2c">(s,r)</annotation></semantics></math> and examine the model’s prediction of <math id="S2.p1.7.m7.1" class="ltx_Math" alttext="o" display="inline"><semantics id="S2.p1.7.m7.1a"><mi id="S2.p1.7.m7.1.1" xref="S2.p1.7.m7.1.1.cmml">o</mi><annotation-xml encoding="MathML-Content" id="S2.p1.7.m7.1b"><ci id="S2.p1.7.m7.1.1.cmml" xref="S2.p1.7.m7.1.1">𝑜</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.7.m7.1c">o</annotation></semantics></math>.</p>
</div>
<div id="S2.p2" class="ltx_para">
<p id="S2.p2.10" class="ltx_p">An autoregressive transformer language model <math id="S2.p2.1.m1.1" class="ltx_Math" alttext="G:\mathcal{X}\rightarrow\mathcal{Y}" display="inline"><semantics id="S2.p2.1.m1.1a"><mrow id="S2.p2.1.m1.1.1" xref="S2.p2.1.m1.1.1.cmml"><mi id="S2.p2.1.m1.1.1.2" xref="S2.p2.1.m1.1.1.2.cmml">G</mi><mo lspace="0.278em" rspace="0.278em" id="S2.p2.1.m1.1.1.1" xref="S2.p2.1.m1.1.1.1.cmml">:</mo><mrow id="S2.p2.1.m1.1.1.3" xref="S2.p2.1.m1.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S2.p2.1.m1.1.1.3.2" xref="S2.p2.1.m1.1.1.3.2.cmml">𝒳</mi><mo stretchy="false" id="S2.p2.1.m1.1.1.3.1" xref="S2.p2.1.m1.1.1.3.1.cmml">→</mo><mi class="ltx_font_mathcaligraphic" id="S2.p2.1.m1.1.1.3.3" xref="S2.p2.1.m1.1.1.3.3.cmml">𝒴</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.p2.1.m1.1b"><apply id="S2.p2.1.m1.1.1.cmml" xref="S2.p2.1.m1.1.1"><ci id="S2.p2.1.m1.1.1.1.cmml" xref="S2.p2.1.m1.1.1.1">:</ci><ci id="S2.p2.1.m1.1.1.2.cmml" xref="S2.p2.1.m1.1.1.2">𝐺</ci><apply id="S2.p2.1.m1.1.1.3.cmml" xref="S2.p2.1.m1.1.1.3"><ci id="S2.p2.1.m1.1.1.3.1.cmml" xref="S2.p2.1.m1.1.1.3.1">→</ci><ci id="S2.p2.1.m1.1.1.3.2.cmml" xref="S2.p2.1.m1.1.1.3.2">𝒳</ci><ci id="S2.p2.1.m1.1.1.3.3.cmml" xref="S2.p2.1.m1.1.1.3.3">𝒴</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p2.1.m1.1c">G:\mathcal{X}\rightarrow\mathcal{Y}</annotation></semantics></math> over vocabulary <math id="S2.p2.2.m2.1" class="ltx_Math" alttext="V" display="inline"><semantics id="S2.p2.2.m2.1a"><mi id="S2.p2.2.m2.1.1" xref="S2.p2.2.m2.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="S2.p2.2.m2.1b"><ci id="S2.p2.2.m2.1.1.cmml" xref="S2.p2.2.m2.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p2.2.m2.1c">V</annotation></semantics></math> maps a token sequence <math id="S2.p2.3.m3.3" class="ltx_Math" alttext="x=[x_{1},...,x_{T}]\in\mathcal{X}" display="inline"><semantics id="S2.p2.3.m3.3a"><mrow id="S2.p2.3.m3.3.3" xref="S2.p2.3.m3.3.3.cmml"><mi id="S2.p2.3.m3.3.3.4" xref="S2.p2.3.m3.3.3.4.cmml">x</mi><mo id="S2.p2.3.m3.3.3.5" xref="S2.p2.3.m3.3.3.5.cmml">=</mo><mrow id="S2.p2.3.m3.3.3.2.2" xref="S2.p2.3.m3.3.3.2.3.cmml"><mo stretchy="false" id="S2.p2.3.m3.3.3.2.2.3" xref="S2.p2.3.m3.3.3.2.3.cmml">[</mo><msub id="S2.p2.3.m3.2.2.1.1.1" xref="S2.p2.3.m3.2.2.1.1.1.cmml"><mi id="S2.p2.3.m3.2.2.1.1.1.2" xref="S2.p2.3.m3.2.2.1.1.1.2.cmml">x</mi><mn id="S2.p2.3.m3.2.2.1.1.1.3" xref="S2.p2.3.m3.2.2.1.1.1.3.cmml">1</mn></msub><mo id="S2.p2.3.m3.3.3.2.2.4" xref="S2.p2.3.m3.3.3.2.3.cmml">,</mo><mi mathvariant="normal" id="S2.p2.3.m3.1.1" xref="S2.p2.3.m3.1.1.cmml">…</mi><mo id="S2.p2.3.m3.3.3.2.2.5" xref="S2.p2.3.m3.3.3.2.3.cmml">,</mo><msub id="S2.p2.3.m3.3.3.2.2.2" xref="S2.p2.3.m3.3.3.2.2.2.cmml"><mi id="S2.p2.3.m3.3.3.2.2.2.2" xref="S2.p2.3.m3.3.3.2.2.2.2.cmml">x</mi><mi id="S2.p2.3.m3.3.3.2.2.2.3" xref="S2.p2.3.m3.3.3.2.2.2.3.cmml">T</mi></msub><mo stretchy="false" id="S2.p2.3.m3.3.3.2.2.6" xref="S2.p2.3.m3.3.3.2.3.cmml">]</mo></mrow><mo id="S2.p2.3.m3.3.3.6" xref="S2.p2.3.m3.3.3.6.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S2.p2.3.m3.3.3.7" xref="S2.p2.3.m3.3.3.7.cmml">𝒳</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.p2.3.m3.3b"><apply id="S2.p2.3.m3.3.3.cmml" xref="S2.p2.3.m3.3.3"><and id="S2.p2.3.m3.3.3a.cmml" xref="S2.p2.3.m3.3.3"></and><apply id="S2.p2.3.m3.3.3b.cmml" xref="S2.p2.3.m3.3.3"><eq id="S2.p2.3.m3.3.3.5.cmml" xref="S2.p2.3.m3.3.3.5"></eq><ci id="S2.p2.3.m3.3.3.4.cmml" xref="S2.p2.3.m3.3.3.4">𝑥</ci><list id="S2.p2.3.m3.3.3.2.3.cmml" xref="S2.p2.3.m3.3.3.2.2"><apply id="S2.p2.3.m3.2.2.1.1.1.cmml" xref="S2.p2.3.m3.2.2.1.1.1"><csymbol cd="ambiguous" id="S2.p2.3.m3.2.2.1.1.1.1.cmml" xref="S2.p2.3.m3.2.2.1.1.1">subscript</csymbol><ci id="S2.p2.3.m3.2.2.1.1.1.2.cmml" xref="S2.p2.3.m3.2.2.1.1.1.2">𝑥</ci><cn type="integer" id="S2.p2.3.m3.2.2.1.1.1.3.cmml" xref="S2.p2.3.m3.2.2.1.1.1.3">1</cn></apply><ci id="S2.p2.3.m3.1.1.cmml" xref="S2.p2.3.m3.1.1">…</ci><apply id="S2.p2.3.m3.3.3.2.2.2.cmml" xref="S2.p2.3.m3.3.3.2.2.2"><csymbol cd="ambiguous" id="S2.p2.3.m3.3.3.2.2.2.1.cmml" xref="S2.p2.3.m3.3.3.2.2.2">subscript</csymbol><ci id="S2.p2.3.m3.3.3.2.2.2.2.cmml" xref="S2.p2.3.m3.3.3.2.2.2.2">𝑥</ci><ci id="S2.p2.3.m3.3.3.2.2.2.3.cmml" xref="S2.p2.3.m3.3.3.2.2.2.3">𝑇</ci></apply></list></apply><apply id="S2.p2.3.m3.3.3c.cmml" xref="S2.p2.3.m3.3.3"><in id="S2.p2.3.m3.3.3.6.cmml" xref="S2.p2.3.m3.3.3.6"></in><share href="#S2.p2.3.m3.3.3.2.cmml" id="S2.p2.3.m3.3.3d.cmml" xref="S2.p2.3.m3.3.3"></share><ci id="S2.p2.3.m3.3.3.7.cmml" xref="S2.p2.3.m3.3.3.7">𝒳</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p2.3.m3.3c">x=[x_{1},...,x_{T}]\in\mathcal{X}</annotation></semantics></math>, <math id="S2.p2.4.m4.1" class="ltx_Math" alttext="x_{i}\in V" display="inline"><semantics id="S2.p2.4.m4.1a"><mrow id="S2.p2.4.m4.1.1" xref="S2.p2.4.m4.1.1.cmml"><msub id="S2.p2.4.m4.1.1.2" xref="S2.p2.4.m4.1.1.2.cmml"><mi id="S2.p2.4.m4.1.1.2.2" xref="S2.p2.4.m4.1.1.2.2.cmml">x</mi><mi id="S2.p2.4.m4.1.1.2.3" xref="S2.p2.4.m4.1.1.2.3.cmml">i</mi></msub><mo id="S2.p2.4.m4.1.1.1" xref="S2.p2.4.m4.1.1.1.cmml">∈</mo><mi id="S2.p2.4.m4.1.1.3" xref="S2.p2.4.m4.1.1.3.cmml">V</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.p2.4.m4.1b"><apply id="S2.p2.4.m4.1.1.cmml" xref="S2.p2.4.m4.1.1"><in id="S2.p2.4.m4.1.1.1.cmml" xref="S2.p2.4.m4.1.1.1"></in><apply id="S2.p2.4.m4.1.1.2.cmml" xref="S2.p2.4.m4.1.1.2"><csymbol cd="ambiguous" id="S2.p2.4.m4.1.1.2.1.cmml" xref="S2.p2.4.m4.1.1.2">subscript</csymbol><ci id="S2.p2.4.m4.1.1.2.2.cmml" xref="S2.p2.4.m4.1.1.2.2">𝑥</ci><ci id="S2.p2.4.m4.1.1.2.3.cmml" xref="S2.p2.4.m4.1.1.2.3">𝑖</ci></apply><ci id="S2.p2.4.m4.1.1.3.cmml" xref="S2.p2.4.m4.1.1.3">𝑉</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p2.4.m4.1c">x_{i}\in V</annotation></semantics></math> to a probability distribution <math id="S2.p2.5.m5.1" class="ltx_Math" alttext="\smash{y\in\mathcal{Y}\subset\mathbb{R}^{|V|}}" display="inline"><semantics id="S2.p2.5.m5.1a"><mrow id="S2.p2.5.m5.1.2" xref="S2.p2.5.m5.1.2.cmml"><mi id="S2.p2.5.m5.1.2.2" xref="S2.p2.5.m5.1.2.2.cmml">y</mi><mo id="S2.p2.5.m5.1.2.3" xref="S2.p2.5.m5.1.2.3.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S2.p2.5.m5.1.2.4" xref="S2.p2.5.m5.1.2.4.cmml">𝒴</mi><mo id="S2.p2.5.m5.1.2.5" xref="S2.p2.5.m5.1.2.5.cmml">⊂</mo><msup id="S2.p2.5.m5.1.2.6" xref="S2.p2.5.m5.1.2.6.cmml"><mi id="S2.p2.5.m5.1.2.6.2" xref="S2.p2.5.m5.1.2.6.2.cmml">ℝ</mi><mrow id="S2.p2.5.m5.1.1.1.3" xref="S2.p2.5.m5.1.1.1.2.cmml"><mo stretchy="false" id="S2.p2.5.m5.1.1.1.3.1" xref="S2.p2.5.m5.1.1.1.2.1.cmml">|</mo><mi id="S2.p2.5.m5.1.1.1.1" xref="S2.p2.5.m5.1.1.1.1.cmml">V</mi><mo stretchy="false" id="S2.p2.5.m5.1.1.1.3.2" xref="S2.p2.5.m5.1.1.1.2.1.cmml">|</mo></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.p2.5.m5.1b"><apply id="S2.p2.5.m5.1.2.cmml" xref="S2.p2.5.m5.1.2"><and id="S2.p2.5.m5.1.2a.cmml" xref="S2.p2.5.m5.1.2"></and><apply id="S2.p2.5.m5.1.2b.cmml" xref="S2.p2.5.m5.1.2"><in id="S2.p2.5.m5.1.2.3.cmml" xref="S2.p2.5.m5.1.2.3"></in><ci id="S2.p2.5.m5.1.2.2.cmml" xref="S2.p2.5.m5.1.2.2">𝑦</ci><ci id="S2.p2.5.m5.1.2.4.cmml" xref="S2.p2.5.m5.1.2.4">𝒴</ci></apply><apply id="S2.p2.5.m5.1.2c.cmml" xref="S2.p2.5.m5.1.2"><subset id="S2.p2.5.m5.1.2.5.cmml" xref="S2.p2.5.m5.1.2.5"></subset><share href="#S2.p2.5.m5.1.2.4.cmml" id="S2.p2.5.m5.1.2d.cmml" xref="S2.p2.5.m5.1.2"></share><apply id="S2.p2.5.m5.1.2.6.cmml" xref="S2.p2.5.m5.1.2.6"><csymbol cd="ambiguous" id="S2.p2.5.m5.1.2.6.1.cmml" xref="S2.p2.5.m5.1.2.6">superscript</csymbol><ci id="S2.p2.5.m5.1.2.6.2.cmml" xref="S2.p2.5.m5.1.2.6.2">ℝ</ci><apply id="S2.p2.5.m5.1.1.1.2.cmml" xref="S2.p2.5.m5.1.1.1.3"><abs id="S2.p2.5.m5.1.1.1.2.1.cmml" xref="S2.p2.5.m5.1.1.1.3.1"></abs><ci id="S2.p2.5.m5.1.1.1.1.cmml" xref="S2.p2.5.m5.1.1.1.1">𝑉</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p2.5.m5.1c">\smash{y\in\mathcal{Y}\subset\mathbb{R}^{|V|}}</annotation></semantics></math> that predicts next-token continuations of <math id="S2.p2.6.m6.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S2.p2.6.m6.1a"><mi id="S2.p2.6.m6.1.1" xref="S2.p2.6.m6.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S2.p2.6.m6.1b"><ci id="S2.p2.6.m6.1.1.cmml" xref="S2.p2.6.m6.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p2.6.m6.1c">x</annotation></semantics></math>. Within the transformer, the <math id="S2.p2.7.m7.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S2.p2.7.m7.1a"><mi id="S2.p2.7.m7.1.1" xref="S2.p2.7.m7.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S2.p2.7.m7.1b"><ci id="S2.p2.7.m7.1.1.cmml" xref="S2.p2.7.m7.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p2.7.m7.1c">i</annotation></semantics></math>th token is embedded as a series of hidden state vectors <math id="S2.p2.8.m8.1" class="ltx_Math" alttext="\smash{h^{(l)}_{i}}" display="inline"><semantics id="S2.p2.8.m8.1a"><msubsup id="S2.p2.8.m8.1.2" xref="S2.p2.8.m8.1.2.cmml"><mi id="S2.p2.8.m8.1.2.2.2" xref="S2.p2.8.m8.1.2.2.2.cmml">h</mi><mi id="S2.p2.8.m8.1.2.3" xref="S2.p2.8.m8.1.2.3.cmml">i</mi><mrow id="S2.p2.8.m8.1.1.1.3" xref="S2.p2.8.m8.1.2.cmml"><mo stretchy="false" id="S2.p2.8.m8.1.1.1.3.1" xref="S2.p2.8.m8.1.2.cmml">(</mo><mi id="S2.p2.8.m8.1.1.1.1" xref="S2.p2.8.m8.1.1.1.1.cmml">l</mi><mo stretchy="false" id="S2.p2.8.m8.1.1.1.3.2" xref="S2.p2.8.m8.1.2.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S2.p2.8.m8.1b"><apply id="S2.p2.8.m8.1.2.cmml" xref="S2.p2.8.m8.1.2"><csymbol cd="ambiguous" id="S2.p2.8.m8.1.2.1.cmml" xref="S2.p2.8.m8.1.2">subscript</csymbol><apply id="S2.p2.8.m8.1.2.2.cmml" xref="S2.p2.8.m8.1.2"><csymbol cd="ambiguous" id="S2.p2.8.m8.1.2.2.1.cmml" xref="S2.p2.8.m8.1.2">superscript</csymbol><ci id="S2.p2.8.m8.1.2.2.2.cmml" xref="S2.p2.8.m8.1.2.2.2">ℎ</ci><ci id="S2.p2.8.m8.1.1.1.1.cmml" xref="S2.p2.8.m8.1.1.1.1">𝑙</ci></apply><ci id="S2.p2.8.m8.1.2.3.cmml" xref="S2.p2.8.m8.1.2.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p2.8.m8.1c">\smash{h^{(l)}_{i}}</annotation></semantics></math>, beginning with <math id="S2.p2.9.m9.3" class="ltx_Math" alttext="\smash{h^{(0)}_{i}=\mathrm{emb}(x_{i})+\mathrm{pos}(i)\in\mathbb{R}^{H}}" display="inline"><semantics id="S2.p2.9.m9.3a"><mrow id="S2.p2.9.m9.3.3" xref="S2.p2.9.m9.3.3.cmml"><msubsup id="S2.p2.9.m9.3.3.3" xref="S2.p2.9.m9.3.3.3.cmml"><mi id="S2.p2.9.m9.3.3.3.2.2" xref="S2.p2.9.m9.3.3.3.2.2.cmml">h</mi><mi id="S2.p2.9.m9.3.3.3.3" xref="S2.p2.9.m9.3.3.3.3.cmml">i</mi><mrow id="S2.p2.9.m9.1.1.1.3" xref="S2.p2.9.m9.3.3.3.cmml"><mo stretchy="false" id="S2.p2.9.m9.1.1.1.3.1" xref="S2.p2.9.m9.3.3.3.cmml">(</mo><mn id="S2.p2.9.m9.1.1.1.1" xref="S2.p2.9.m9.1.1.1.1.cmml">0</mn><mo stretchy="false" id="S2.p2.9.m9.1.1.1.3.2" xref="S2.p2.9.m9.3.3.3.cmml">)</mo></mrow></msubsup><mo id="S2.p2.9.m9.3.3.4" xref="S2.p2.9.m9.3.3.4.cmml">=</mo><mrow id="S2.p2.9.m9.3.3.1" xref="S2.p2.9.m9.3.3.1.cmml"><mrow id="S2.p2.9.m9.3.3.1.1" xref="S2.p2.9.m9.3.3.1.1.cmml"><mi id="S2.p2.9.m9.3.3.1.1.3" xref="S2.p2.9.m9.3.3.1.1.3.cmml">emb</mi><mo lspace="0em" rspace="0em" id="S2.p2.9.m9.3.3.1.1.2" xref="S2.p2.9.m9.3.3.1.1.2.cmml">​</mo><mrow id="S2.p2.9.m9.3.3.1.1.1.1" xref="S2.p2.9.m9.3.3.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.p2.9.m9.3.3.1.1.1.1.2" xref="S2.p2.9.m9.3.3.1.1.1.1.1.cmml">(</mo><msub id="S2.p2.9.m9.3.3.1.1.1.1.1" xref="S2.p2.9.m9.3.3.1.1.1.1.1.cmml"><mi id="S2.p2.9.m9.3.3.1.1.1.1.1.2" xref="S2.p2.9.m9.3.3.1.1.1.1.1.2.cmml">x</mi><mi id="S2.p2.9.m9.3.3.1.1.1.1.1.3" xref="S2.p2.9.m9.3.3.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S2.p2.9.m9.3.3.1.1.1.1.3" xref="S2.p2.9.m9.3.3.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S2.p2.9.m9.3.3.1.2" xref="S2.p2.9.m9.3.3.1.2.cmml">+</mo><mrow id="S2.p2.9.m9.3.3.1.3" xref="S2.p2.9.m9.3.3.1.3.cmml"><mi id="S2.p2.9.m9.3.3.1.3.2" xref="S2.p2.9.m9.3.3.1.3.2.cmml">pos</mi><mo lspace="0em" rspace="0em" id="S2.p2.9.m9.3.3.1.3.1" xref="S2.p2.9.m9.3.3.1.3.1.cmml">​</mo><mrow id="S2.p2.9.m9.3.3.1.3.3.2" xref="S2.p2.9.m9.3.3.1.3.cmml"><mo stretchy="false" id="S2.p2.9.m9.3.3.1.3.3.2.1" xref="S2.p2.9.m9.3.3.1.3.cmml">(</mo><mi id="S2.p2.9.m9.2.2" xref="S2.p2.9.m9.2.2.cmml">i</mi><mo stretchy="false" id="S2.p2.9.m9.3.3.1.3.3.2.2" xref="S2.p2.9.m9.3.3.1.3.cmml">)</mo></mrow></mrow></mrow><mo id="S2.p2.9.m9.3.3.5" xref="S2.p2.9.m9.3.3.5.cmml">∈</mo><msup id="S2.p2.9.m9.3.3.6" xref="S2.p2.9.m9.3.3.6.cmml"><mi id="S2.p2.9.m9.3.3.6.2" xref="S2.p2.9.m9.3.3.6.2.cmml">ℝ</mi><mi id="S2.p2.9.m9.3.3.6.3" xref="S2.p2.9.m9.3.3.6.3.cmml">H</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.p2.9.m9.3b"><apply id="S2.p2.9.m9.3.3.cmml" xref="S2.p2.9.m9.3.3"><and id="S2.p2.9.m9.3.3a.cmml" xref="S2.p2.9.m9.3.3"></and><apply id="S2.p2.9.m9.3.3b.cmml" xref="S2.p2.9.m9.3.3"><eq id="S2.p2.9.m9.3.3.4.cmml" xref="S2.p2.9.m9.3.3.4"></eq><apply id="S2.p2.9.m9.3.3.3.cmml" xref="S2.p2.9.m9.3.3.3"><csymbol cd="ambiguous" id="S2.p2.9.m9.3.3.3.1.cmml" xref="S2.p2.9.m9.3.3.3">subscript</csymbol><apply id="S2.p2.9.m9.3.3.3.2.cmml" xref="S2.p2.9.m9.3.3.3"><csymbol cd="ambiguous" id="S2.p2.9.m9.3.3.3.2.1.cmml" xref="S2.p2.9.m9.3.3.3">superscript</csymbol><ci id="S2.p2.9.m9.3.3.3.2.2.cmml" xref="S2.p2.9.m9.3.3.3.2.2">ℎ</ci><cn type="integer" id="S2.p2.9.m9.1.1.1.1.cmml" xref="S2.p2.9.m9.1.1.1.1">0</cn></apply><ci id="S2.p2.9.m9.3.3.3.3.cmml" xref="S2.p2.9.m9.3.3.3.3">𝑖</ci></apply><apply id="S2.p2.9.m9.3.3.1.cmml" xref="S2.p2.9.m9.3.3.1"><plus id="S2.p2.9.m9.3.3.1.2.cmml" xref="S2.p2.9.m9.3.3.1.2"></plus><apply id="S2.p2.9.m9.3.3.1.1.cmml" xref="S2.p2.9.m9.3.3.1.1"><times id="S2.p2.9.m9.3.3.1.1.2.cmml" xref="S2.p2.9.m9.3.3.1.1.2"></times><ci id="S2.p2.9.m9.3.3.1.1.3.cmml" xref="S2.p2.9.m9.3.3.1.1.3">emb</ci><apply id="S2.p2.9.m9.3.3.1.1.1.1.1.cmml" xref="S2.p2.9.m9.3.3.1.1.1.1"><csymbol cd="ambiguous" id="S2.p2.9.m9.3.3.1.1.1.1.1.1.cmml" xref="S2.p2.9.m9.3.3.1.1.1.1">subscript</csymbol><ci id="S2.p2.9.m9.3.3.1.1.1.1.1.2.cmml" xref="S2.p2.9.m9.3.3.1.1.1.1.1.2">𝑥</ci><ci id="S2.p2.9.m9.3.3.1.1.1.1.1.3.cmml" xref="S2.p2.9.m9.3.3.1.1.1.1.1.3">𝑖</ci></apply></apply><apply id="S2.p2.9.m9.3.3.1.3.cmml" xref="S2.p2.9.m9.3.3.1.3"><times id="S2.p2.9.m9.3.3.1.3.1.cmml" xref="S2.p2.9.m9.3.3.1.3.1"></times><ci id="S2.p2.9.m9.3.3.1.3.2.cmml" xref="S2.p2.9.m9.3.3.1.3.2">pos</ci><ci id="S2.p2.9.m9.2.2.cmml" xref="S2.p2.9.m9.2.2">𝑖</ci></apply></apply></apply><apply id="S2.p2.9.m9.3.3c.cmml" xref="S2.p2.9.m9.3.3"><in id="S2.p2.9.m9.3.3.5.cmml" xref="S2.p2.9.m9.3.3.5"></in><share href="#S2.p2.9.m9.3.3.1.cmml" id="S2.p2.9.m9.3.3d.cmml" xref="S2.p2.9.m9.3.3"></share><apply id="S2.p2.9.m9.3.3.6.cmml" xref="S2.p2.9.m9.3.3.6"><csymbol cd="ambiguous" id="S2.p2.9.m9.3.3.6.1.cmml" xref="S2.p2.9.m9.3.3.6">superscript</csymbol><ci id="S2.p2.9.m9.3.3.6.2.cmml" xref="S2.p2.9.m9.3.3.6.2">ℝ</ci><ci id="S2.p2.9.m9.3.3.6.3.cmml" xref="S2.p2.9.m9.3.3.6.3">𝐻</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p2.9.m9.3c">\smash{h^{(0)}_{i}=\mathrm{emb}(x_{i})+\mathrm{pos}(i)\in\mathbb{R}^{H}}</annotation></semantics></math>. The final output <math id="S2.p2.10.m10.2" class="ltx_Math" alttext="\smash{y=\mathrm{decode}(h^{(L)}_{T})}" display="inline"><semantics id="S2.p2.10.m10.2a"><mrow id="S2.p2.10.m10.2.2" xref="S2.p2.10.m10.2.2.cmml"><mi id="S2.p2.10.m10.2.2.3" xref="S2.p2.10.m10.2.2.3.cmml">y</mi><mo id="S2.p2.10.m10.2.2.2" xref="S2.p2.10.m10.2.2.2.cmml">=</mo><mrow id="S2.p2.10.m10.2.2.1" xref="S2.p2.10.m10.2.2.1.cmml"><mi id="S2.p2.10.m10.2.2.1.3" xref="S2.p2.10.m10.2.2.1.3.cmml">decode</mi><mo lspace="0em" rspace="0em" id="S2.p2.10.m10.2.2.1.2" xref="S2.p2.10.m10.2.2.1.2.cmml">​</mo><mrow id="S2.p2.10.m10.2.2.1.1.1" xref="S2.p2.10.m10.2.2.1.1.1.1.cmml"><mo stretchy="false" id="S2.p2.10.m10.2.2.1.1.1.2" xref="S2.p2.10.m10.2.2.1.1.1.1.cmml">(</mo><msubsup id="S2.p2.10.m10.2.2.1.1.1.1" xref="S2.p2.10.m10.2.2.1.1.1.1.cmml"><mi id="S2.p2.10.m10.2.2.1.1.1.1.2.2" xref="S2.p2.10.m10.2.2.1.1.1.1.2.2.cmml">h</mi><mi id="S2.p2.10.m10.2.2.1.1.1.1.3" xref="S2.p2.10.m10.2.2.1.1.1.1.3.cmml">T</mi><mrow id="S2.p2.10.m10.1.1.1.3" xref="S2.p2.10.m10.2.2.1.1.1.1.cmml"><mo stretchy="false" id="S2.p2.10.m10.1.1.1.3.1" xref="S2.p2.10.m10.2.2.1.1.1.1.cmml">(</mo><mi id="S2.p2.10.m10.1.1.1.1" xref="S2.p2.10.m10.1.1.1.1.cmml">L</mi><mo stretchy="false" id="S2.p2.10.m10.1.1.1.3.2" xref="S2.p2.10.m10.2.2.1.1.1.1.cmml">)</mo></mrow></msubsup><mo stretchy="false" id="S2.p2.10.m10.2.2.1.1.1.3" xref="S2.p2.10.m10.2.2.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.p2.10.m10.2b"><apply id="S2.p2.10.m10.2.2.cmml" xref="S2.p2.10.m10.2.2"><eq id="S2.p2.10.m10.2.2.2.cmml" xref="S2.p2.10.m10.2.2.2"></eq><ci id="S2.p2.10.m10.2.2.3.cmml" xref="S2.p2.10.m10.2.2.3">𝑦</ci><apply id="S2.p2.10.m10.2.2.1.cmml" xref="S2.p2.10.m10.2.2.1"><times id="S2.p2.10.m10.2.2.1.2.cmml" xref="S2.p2.10.m10.2.2.1.2"></times><ci id="S2.p2.10.m10.2.2.1.3.cmml" xref="S2.p2.10.m10.2.2.1.3">decode</ci><apply id="S2.p2.10.m10.2.2.1.1.1.1.cmml" xref="S2.p2.10.m10.2.2.1.1.1"><csymbol cd="ambiguous" id="S2.p2.10.m10.2.2.1.1.1.1.1.cmml" xref="S2.p2.10.m10.2.2.1.1.1">subscript</csymbol><apply id="S2.p2.10.m10.2.2.1.1.1.1.2.cmml" xref="S2.p2.10.m10.2.2.1.1.1"><csymbol cd="ambiguous" id="S2.p2.10.m10.2.2.1.1.1.1.2.1.cmml" xref="S2.p2.10.m10.2.2.1.1.1">superscript</csymbol><ci id="S2.p2.10.m10.2.2.1.1.1.1.2.2.cmml" xref="S2.p2.10.m10.2.2.1.1.1.1.2.2">ℎ</ci><ci id="S2.p2.10.m10.1.1.1.1.cmml" xref="S2.p2.10.m10.1.1.1.1">𝐿</ci></apply><ci id="S2.p2.10.m10.2.2.1.1.1.1.3.cmml" xref="S2.p2.10.m10.2.2.1.1.1.1.3">𝑇</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p2.10.m10.2c">\smash{y=\mathrm{decode}(h^{(L)}_{T})}</annotation></semantics></math> is read from the last hidden state.</p>
</div>
<div id="S2.p3" class="ltx_para">
<p id="S2.p3.8" class="ltx_p">We visualize the internal computation of <math id="S2.p3.1.m1.1" class="ltx_Math" alttext="G" display="inline"><semantics id="S2.p3.1.m1.1a"><mi id="S2.p3.1.m1.1.1" xref="S2.p3.1.m1.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="S2.p3.1.m1.1b"><ci id="S2.p3.1.m1.1.1.cmml" xref="S2.p3.1.m1.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p3.1.m1.1c">G</annotation></semantics></math> as a grid (Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>a) of hidden states <math id="S2.p3.2.m2.1" class="ltx_Math" alttext="\smash{h^{(l)}_{i}}" display="inline"><semantics id="S2.p3.2.m2.1a"><msubsup id="S2.p3.2.m2.1.2" xref="S2.p3.2.m2.1.2.cmml"><mi id="S2.p3.2.m2.1.2.2.2" xref="S2.p3.2.m2.1.2.2.2.cmml">h</mi><mi id="S2.p3.2.m2.1.2.3" xref="S2.p3.2.m2.1.2.3.cmml">i</mi><mrow id="S2.p3.2.m2.1.1.1.3" xref="S2.p3.2.m2.1.2.cmml"><mo stretchy="false" id="S2.p3.2.m2.1.1.1.3.1" xref="S2.p3.2.m2.1.2.cmml">(</mo><mi id="S2.p3.2.m2.1.1.1.1" xref="S2.p3.2.m2.1.1.1.1.cmml">l</mi><mo stretchy="false" id="S2.p3.2.m2.1.1.1.3.2" xref="S2.p3.2.m2.1.2.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S2.p3.2.m2.1b"><apply id="S2.p3.2.m2.1.2.cmml" xref="S2.p3.2.m2.1.2"><csymbol cd="ambiguous" id="S2.p3.2.m2.1.2.1.cmml" xref="S2.p3.2.m2.1.2">subscript</csymbol><apply id="S2.p3.2.m2.1.2.2.cmml" xref="S2.p3.2.m2.1.2"><csymbol cd="ambiguous" id="S2.p3.2.m2.1.2.2.1.cmml" xref="S2.p3.2.m2.1.2">superscript</csymbol><ci id="S2.p3.2.m2.1.2.2.2.cmml" xref="S2.p3.2.m2.1.2.2.2">ℎ</ci><ci id="S2.p3.2.m2.1.1.1.1.cmml" xref="S2.p3.2.m2.1.1.1.1">𝑙</ci></apply><ci id="S2.p3.2.m2.1.2.3.cmml" xref="S2.p3.2.m2.1.2.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p3.2.m2.1c">\smash{h^{(l)}_{i}}</annotation></semantics></math> in which each layer <math id="S2.p3.3.m3.1" class="ltx_Math" alttext="l" display="inline"><semantics id="S2.p3.3.m3.1a"><mi id="S2.p3.3.m3.1.1" xref="S2.p3.3.m3.1.1.cmml">l</mi><annotation-xml encoding="MathML-Content" id="S2.p3.3.m3.1b"><ci id="S2.p3.3.m3.1.1.cmml" xref="S2.p3.3.m3.1.1">𝑙</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p3.3.m3.1c">l</annotation></semantics></math> (left <math id="S2.p3.4.m4.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="S2.p3.4.m4.1a"><mo stretchy="false" id="S2.p3.4.m4.1.1" xref="S2.p3.4.m4.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S2.p3.4.m4.1b"><ci id="S2.p3.4.m4.1.1.cmml" xref="S2.p3.4.m4.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p3.4.m4.1c">\rightarrow</annotation></semantics></math> right) adds global attention <math id="S2.p3.5.m5.1" class="ltx_Math" alttext="\smash{a^{(l)}_{i}}" display="inline"><semantics id="S2.p3.5.m5.1a"><msubsup id="S2.p3.5.m5.1.2" xref="S2.p3.5.m5.1.2.cmml"><mi id="S2.p3.5.m5.1.2.2.2" xref="S2.p3.5.m5.1.2.2.2.cmml">a</mi><mi id="S2.p3.5.m5.1.2.3" xref="S2.p3.5.m5.1.2.3.cmml">i</mi><mrow id="S2.p3.5.m5.1.1.1.3" xref="S2.p3.5.m5.1.2.cmml"><mo stretchy="false" id="S2.p3.5.m5.1.1.1.3.1" xref="S2.p3.5.m5.1.2.cmml">(</mo><mi id="S2.p3.5.m5.1.1.1.1" xref="S2.p3.5.m5.1.1.1.1.cmml">l</mi><mo stretchy="false" id="S2.p3.5.m5.1.1.1.3.2" xref="S2.p3.5.m5.1.2.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S2.p3.5.m5.1b"><apply id="S2.p3.5.m5.1.2.cmml" xref="S2.p3.5.m5.1.2"><csymbol cd="ambiguous" id="S2.p3.5.m5.1.2.1.cmml" xref="S2.p3.5.m5.1.2">subscript</csymbol><apply id="S2.p3.5.m5.1.2.2.cmml" xref="S2.p3.5.m5.1.2"><csymbol cd="ambiguous" id="S2.p3.5.m5.1.2.2.1.cmml" xref="S2.p3.5.m5.1.2">superscript</csymbol><ci id="S2.p3.5.m5.1.2.2.2.cmml" xref="S2.p3.5.m5.1.2.2.2">𝑎</ci><ci id="S2.p3.5.m5.1.1.1.1.cmml" xref="S2.p3.5.m5.1.1.1.1">𝑙</ci></apply><ci id="S2.p3.5.m5.1.2.3.cmml" xref="S2.p3.5.m5.1.2.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p3.5.m5.1c">\smash{a^{(l)}_{i}}</annotation></semantics></math> and local MLP <math id="S2.p3.6.m6.1" class="ltx_Math" alttext="\smash{m^{(l)}_{i}}" display="inline"><semantics id="S2.p3.6.m6.1a"><msubsup id="S2.p3.6.m6.1.2" xref="S2.p3.6.m6.1.2.cmml"><mi id="S2.p3.6.m6.1.2.2.2" xref="S2.p3.6.m6.1.2.2.2.cmml">m</mi><mi id="S2.p3.6.m6.1.2.3" xref="S2.p3.6.m6.1.2.3.cmml">i</mi><mrow id="S2.p3.6.m6.1.1.1.3" xref="S2.p3.6.m6.1.2.cmml"><mo stretchy="false" id="S2.p3.6.m6.1.1.1.3.1" xref="S2.p3.6.m6.1.2.cmml">(</mo><mi id="S2.p3.6.m6.1.1.1.1" xref="S2.p3.6.m6.1.1.1.1.cmml">l</mi><mo stretchy="false" id="S2.p3.6.m6.1.1.1.3.2" xref="S2.p3.6.m6.1.2.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S2.p3.6.m6.1b"><apply id="S2.p3.6.m6.1.2.cmml" xref="S2.p3.6.m6.1.2"><csymbol cd="ambiguous" id="S2.p3.6.m6.1.2.1.cmml" xref="S2.p3.6.m6.1.2">subscript</csymbol><apply id="S2.p3.6.m6.1.2.2.cmml" xref="S2.p3.6.m6.1.2"><csymbol cd="ambiguous" id="S2.p3.6.m6.1.2.2.1.cmml" xref="S2.p3.6.m6.1.2">superscript</csymbol><ci id="S2.p3.6.m6.1.2.2.2.cmml" xref="S2.p3.6.m6.1.2.2.2">𝑚</ci><ci id="S2.p3.6.m6.1.1.1.1.cmml" xref="S2.p3.6.m6.1.1.1.1">𝑙</ci></apply><ci id="S2.p3.6.m6.1.2.3.cmml" xref="S2.p3.6.m6.1.2.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p3.6.m6.1c">\smash{m^{(l)}_{i}}</annotation></semantics></math> contributions computed from previous layers, and where each token <math id="S2.p3.7.m7.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S2.p3.7.m7.1a"><mi id="S2.p3.7.m7.1.1" xref="S2.p3.7.m7.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S2.p3.7.m7.1b"><ci id="S2.p3.7.m7.1.1.cmml" xref="S2.p3.7.m7.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p3.7.m7.1c">i</annotation></semantics></math> (top <math id="S2.p3.8.m8.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="S2.p3.8.m8.1a"><mo stretchy="false" id="S2.p3.8.m8.1.1" xref="S2.p3.8.m8.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S2.p3.8.m8.1b"><ci id="S2.p3.8.m8.1.1.cmml" xref="S2.p3.8.m8.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p3.8.m8.1c">\rightarrow</annotation></semantics></math> bottom) attends to previous states from other tokens. Recall that, in the autoregressive case, tokens only draw information from past (above) tokens:</p>
<table id="A10.EGx1" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S2.Ex1"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S2.Ex1.m1.3" class="ltx_Math" alttext="\displaystyle h^{(l)}_{i}=h^{(l-1)}_{i}+a^{(l)}_{i}" display="inline"><semantics id="S2.Ex1.m1.3a"><mrow id="S2.Ex1.m1.3.4" xref="S2.Ex1.m1.3.4.cmml"><msubsup id="S2.Ex1.m1.3.4.2" xref="S2.Ex1.m1.3.4.2.cmml"><mi id="S2.Ex1.m1.3.4.2.2.2" xref="S2.Ex1.m1.3.4.2.2.2.cmml">h</mi><mi id="S2.Ex1.m1.3.4.2.3" xref="S2.Ex1.m1.3.4.2.3.cmml">i</mi><mrow id="S2.Ex1.m1.1.1.1.3" xref="S2.Ex1.m1.3.4.2.cmml"><mo stretchy="false" id="S2.Ex1.m1.1.1.1.3.1" xref="S2.Ex1.m1.3.4.2.cmml">(</mo><mi id="S2.Ex1.m1.1.1.1.1" xref="S2.Ex1.m1.1.1.1.1.cmml">l</mi><mo stretchy="false" id="S2.Ex1.m1.1.1.1.3.2" xref="S2.Ex1.m1.3.4.2.cmml">)</mo></mrow></msubsup><mo id="S2.Ex1.m1.3.4.1" xref="S2.Ex1.m1.3.4.1.cmml">=</mo><mrow id="S2.Ex1.m1.3.4.3" xref="S2.Ex1.m1.3.4.3.cmml"><msubsup id="S2.Ex1.m1.3.4.3.2" xref="S2.Ex1.m1.3.4.3.2.cmml"><mi id="S2.Ex1.m1.3.4.3.2.2.2" xref="S2.Ex1.m1.3.4.3.2.2.2.cmml">h</mi><mi id="S2.Ex1.m1.3.4.3.2.3" xref="S2.Ex1.m1.3.4.3.2.3.cmml">i</mi><mrow id="S2.Ex1.m1.2.2.1.1" xref="S2.Ex1.m1.2.2.1.1.1.cmml"><mo stretchy="false" id="S2.Ex1.m1.2.2.1.1.2" xref="S2.Ex1.m1.2.2.1.1.1.cmml">(</mo><mrow id="S2.Ex1.m1.2.2.1.1.1" xref="S2.Ex1.m1.2.2.1.1.1.cmml"><mi id="S2.Ex1.m1.2.2.1.1.1.2" xref="S2.Ex1.m1.2.2.1.1.1.2.cmml">l</mi><mo id="S2.Ex1.m1.2.2.1.1.1.1" xref="S2.Ex1.m1.2.2.1.1.1.1.cmml">−</mo><mn id="S2.Ex1.m1.2.2.1.1.1.3" xref="S2.Ex1.m1.2.2.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="S2.Ex1.m1.2.2.1.1.3" xref="S2.Ex1.m1.2.2.1.1.1.cmml">)</mo></mrow></msubsup><mo id="S2.Ex1.m1.3.4.3.1" xref="S2.Ex1.m1.3.4.3.1.cmml">+</mo><msubsup id="S2.Ex1.m1.3.4.3.3" xref="S2.Ex1.m1.3.4.3.3.cmml"><mi id="S2.Ex1.m1.3.4.3.3.2.2" xref="S2.Ex1.m1.3.4.3.3.2.2.cmml">a</mi><mi id="S2.Ex1.m1.3.4.3.3.3" xref="S2.Ex1.m1.3.4.3.3.3.cmml">i</mi><mrow id="S2.Ex1.m1.3.3.1.3" xref="S2.Ex1.m1.3.4.3.3.cmml"><mo stretchy="false" id="S2.Ex1.m1.3.3.1.3.1" xref="S2.Ex1.m1.3.4.3.3.cmml">(</mo><mi id="S2.Ex1.m1.3.3.1.1" xref="S2.Ex1.m1.3.3.1.1.cmml">l</mi><mo stretchy="false" id="S2.Ex1.m1.3.3.1.3.2" xref="S2.Ex1.m1.3.4.3.3.cmml">)</mo></mrow></msubsup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.Ex1.m1.3b"><apply id="S2.Ex1.m1.3.4.cmml" xref="S2.Ex1.m1.3.4"><eq id="S2.Ex1.m1.3.4.1.cmml" xref="S2.Ex1.m1.3.4.1"></eq><apply id="S2.Ex1.m1.3.4.2.cmml" xref="S2.Ex1.m1.3.4.2"><csymbol cd="ambiguous" id="S2.Ex1.m1.3.4.2.1.cmml" xref="S2.Ex1.m1.3.4.2">subscript</csymbol><apply id="S2.Ex1.m1.3.4.2.2.cmml" xref="S2.Ex1.m1.3.4.2"><csymbol cd="ambiguous" id="S2.Ex1.m1.3.4.2.2.1.cmml" xref="S2.Ex1.m1.3.4.2">superscript</csymbol><ci id="S2.Ex1.m1.3.4.2.2.2.cmml" xref="S2.Ex1.m1.3.4.2.2.2">ℎ</ci><ci id="S2.Ex1.m1.1.1.1.1.cmml" xref="S2.Ex1.m1.1.1.1.1">𝑙</ci></apply><ci id="S2.Ex1.m1.3.4.2.3.cmml" xref="S2.Ex1.m1.3.4.2.3">𝑖</ci></apply><apply id="S2.Ex1.m1.3.4.3.cmml" xref="S2.Ex1.m1.3.4.3"><plus id="S2.Ex1.m1.3.4.3.1.cmml" xref="S2.Ex1.m1.3.4.3.1"></plus><apply id="S2.Ex1.m1.3.4.3.2.cmml" xref="S2.Ex1.m1.3.4.3.2"><csymbol cd="ambiguous" id="S2.Ex1.m1.3.4.3.2.1.cmml" xref="S2.Ex1.m1.3.4.3.2">subscript</csymbol><apply id="S2.Ex1.m1.3.4.3.2.2.cmml" xref="S2.Ex1.m1.3.4.3.2"><csymbol cd="ambiguous" id="S2.Ex1.m1.3.4.3.2.2.1.cmml" xref="S2.Ex1.m1.3.4.3.2">superscript</csymbol><ci id="S2.Ex1.m1.3.4.3.2.2.2.cmml" xref="S2.Ex1.m1.3.4.3.2.2.2">ℎ</ci><apply id="S2.Ex1.m1.2.2.1.1.1.cmml" xref="S2.Ex1.m1.2.2.1.1"><minus id="S2.Ex1.m1.2.2.1.1.1.1.cmml" xref="S2.Ex1.m1.2.2.1.1.1.1"></minus><ci id="S2.Ex1.m1.2.2.1.1.1.2.cmml" xref="S2.Ex1.m1.2.2.1.1.1.2">𝑙</ci><cn type="integer" id="S2.Ex1.m1.2.2.1.1.1.3.cmml" xref="S2.Ex1.m1.2.2.1.1.1.3">1</cn></apply></apply><ci id="S2.Ex1.m1.3.4.3.2.3.cmml" xref="S2.Ex1.m1.3.4.3.2.3">𝑖</ci></apply><apply id="S2.Ex1.m1.3.4.3.3.cmml" xref="S2.Ex1.m1.3.4.3.3"><csymbol cd="ambiguous" id="S2.Ex1.m1.3.4.3.3.1.cmml" xref="S2.Ex1.m1.3.4.3.3">subscript</csymbol><apply id="S2.Ex1.m1.3.4.3.3.2.cmml" xref="S2.Ex1.m1.3.4.3.3"><csymbol cd="ambiguous" id="S2.Ex1.m1.3.4.3.3.2.1.cmml" xref="S2.Ex1.m1.3.4.3.3">superscript</csymbol><ci id="S2.Ex1.m1.3.4.3.3.2.2.cmml" xref="S2.Ex1.m1.3.4.3.3.2.2">𝑎</ci><ci id="S2.Ex1.m1.3.3.1.1.cmml" xref="S2.Ex1.m1.3.3.1.1">𝑙</ci></apply><ci id="S2.Ex1.m1.3.4.3.3.3.cmml" xref="S2.Ex1.m1.3.4.3.3.3">𝑖</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.Ex1.m1.3c">\displaystyle h^{(l)}_{i}=h^{(l-1)}_{i}+a^{(l)}_{i}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S2.Ex1.m2.1" class="ltx_Math" alttext="\displaystyle+m^{(l)}_{i}" display="inline"><semantics id="S2.Ex1.m2.1a"><mrow id="S2.Ex1.m2.1.2" xref="S2.Ex1.m2.1.2.cmml"><mo id="S2.Ex1.m2.1.2a" xref="S2.Ex1.m2.1.2.cmml">+</mo><msubsup id="S2.Ex1.m2.1.2.2" xref="S2.Ex1.m2.1.2.2.cmml"><mi id="S2.Ex1.m2.1.2.2.2.2" xref="S2.Ex1.m2.1.2.2.2.2.cmml">m</mi><mi id="S2.Ex1.m2.1.2.2.3" xref="S2.Ex1.m2.1.2.2.3.cmml">i</mi><mrow id="S2.Ex1.m2.1.1.1.3" xref="S2.Ex1.m2.1.2.2.cmml"><mo stretchy="false" id="S2.Ex1.m2.1.1.1.3.1" xref="S2.Ex1.m2.1.2.2.cmml">(</mo><mi id="S2.Ex1.m2.1.1.1.1" xref="S2.Ex1.m2.1.1.1.1.cmml">l</mi><mo stretchy="false" id="S2.Ex1.m2.1.1.1.3.2" xref="S2.Ex1.m2.1.2.2.cmml">)</mo></mrow></msubsup></mrow><annotation-xml encoding="MathML-Content" id="S2.Ex1.m2.1b"><apply id="S2.Ex1.m2.1.2.cmml" xref="S2.Ex1.m2.1.2"><plus id="S2.Ex1.m2.1.2.1.cmml" xref="S2.Ex1.m2.1.2"></plus><apply id="S2.Ex1.m2.1.2.2.cmml" xref="S2.Ex1.m2.1.2.2"><csymbol cd="ambiguous" id="S2.Ex1.m2.1.2.2.1.cmml" xref="S2.Ex1.m2.1.2.2">subscript</csymbol><apply id="S2.Ex1.m2.1.2.2.2.cmml" xref="S2.Ex1.m2.1.2.2"><csymbol cd="ambiguous" id="S2.Ex1.m2.1.2.2.2.1.cmml" xref="S2.Ex1.m2.1.2.2">superscript</csymbol><ci id="S2.Ex1.m2.1.2.2.2.2.cmml" xref="S2.Ex1.m2.1.2.2.2.2">𝑚</ci><ci id="S2.Ex1.m2.1.1.1.1.cmml" xref="S2.Ex1.m2.1.1.1.1">𝑙</ci></apply><ci id="S2.Ex1.m2.1.2.2.3.cmml" xref="S2.Ex1.m2.1.2.2.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.Ex1.m2.1c">\displaystyle+m^{(l)}_{i}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="S2.E1"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S2.E1.m1.1" class="ltx_Math" alttext="\displaystyle a^{(l)}_{i}" display="inline"><semantics id="S2.E1.m1.1a"><msubsup id="S2.E1.m1.1.2" xref="S2.E1.m1.1.2.cmml"><mi id="S2.E1.m1.1.2.2.2" xref="S2.E1.m1.1.2.2.2.cmml">a</mi><mi id="S2.E1.m1.1.2.3" xref="S2.E1.m1.1.2.3.cmml">i</mi><mrow id="S2.E1.m1.1.1.1.3" xref="S2.E1.m1.1.2.cmml"><mo stretchy="false" id="S2.E1.m1.1.1.1.3.1" xref="S2.E1.m1.1.2.cmml">(</mo><mi id="S2.E1.m1.1.1.1.1" xref="S2.E1.m1.1.1.1.1.cmml">l</mi><mo stretchy="false" id="S2.E1.m1.1.1.1.3.2" xref="S2.E1.m1.1.2.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S2.E1.m1.1b"><apply id="S2.E1.m1.1.2.cmml" xref="S2.E1.m1.1.2"><csymbol cd="ambiguous" id="S2.E1.m1.1.2.1.cmml" xref="S2.E1.m1.1.2">subscript</csymbol><apply id="S2.E1.m1.1.2.2.cmml" xref="S2.E1.m1.1.2"><csymbol cd="ambiguous" id="S2.E1.m1.1.2.2.1.cmml" xref="S2.E1.m1.1.2">superscript</csymbol><ci id="S2.E1.m1.1.2.2.2.cmml" xref="S2.E1.m1.1.2.2.2">𝑎</ci><ci id="S2.E1.m1.1.1.1.1.cmml" xref="S2.E1.m1.1.1.1.1">𝑙</ci></apply><ci id="S2.E1.m1.1.2.3.cmml" xref="S2.E1.m1.1.2.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E1.m1.1c">\displaystyle a^{(l)}_{i}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S2.E1.m2.8" class="ltx_Math" alttext="\displaystyle=\mathrm{attn}^{(l)}\left(h^{(l-1)}_{1},h^{(l-1)}_{2},\dots,h^{(l-1)}_{i}\right)" display="inline"><semantics id="S2.E1.m2.8a"><mrow id="S2.E1.m2.8.8" xref="S2.E1.m2.8.8.cmml"><mi id="S2.E1.m2.8.8.5" xref="S2.E1.m2.8.8.5.cmml"></mi><mo id="S2.E1.m2.8.8.4" xref="S2.E1.m2.8.8.4.cmml">=</mo><mrow id="S2.E1.m2.8.8.3" xref="S2.E1.m2.8.8.3.cmml"><msup id="S2.E1.m2.8.8.3.5" xref="S2.E1.m2.8.8.3.5.cmml"><mi id="S2.E1.m2.8.8.3.5.2" xref="S2.E1.m2.8.8.3.5.2.cmml">attn</mi><mrow id="S2.E1.m2.1.1.1.3" xref="S2.E1.m2.8.8.3.5.cmml"><mo stretchy="false" id="S2.E1.m2.1.1.1.3.1" xref="S2.E1.m2.8.8.3.5.cmml">(</mo><mi id="S2.E1.m2.1.1.1.1" xref="S2.E1.m2.1.1.1.1.cmml">l</mi><mo stretchy="false" id="S2.E1.m2.1.1.1.3.2" xref="S2.E1.m2.8.8.3.5.cmml">)</mo></mrow></msup><mo lspace="0em" rspace="0em" id="S2.E1.m2.8.8.3.4" xref="S2.E1.m2.8.8.3.4.cmml">​</mo><mrow id="S2.E1.m2.8.8.3.3.3" xref="S2.E1.m2.8.8.3.3.4.cmml"><mo id="S2.E1.m2.8.8.3.3.3.4" xref="S2.E1.m2.8.8.3.3.4.cmml">(</mo><msubsup id="S2.E1.m2.6.6.1.1.1.1" xref="S2.E1.m2.6.6.1.1.1.1.cmml"><mi id="S2.E1.m2.6.6.1.1.1.1.2.2" xref="S2.E1.m2.6.6.1.1.1.1.2.2.cmml">h</mi><mn id="S2.E1.m2.6.6.1.1.1.1.3" xref="S2.E1.m2.6.6.1.1.1.1.3.cmml">1</mn><mrow id="S2.E1.m2.2.2.1.1" xref="S2.E1.m2.2.2.1.1.1.cmml"><mo stretchy="false" id="S2.E1.m2.2.2.1.1.2" xref="S2.E1.m2.2.2.1.1.1.cmml">(</mo><mrow id="S2.E1.m2.2.2.1.1.1" xref="S2.E1.m2.2.2.1.1.1.cmml"><mi id="S2.E1.m2.2.2.1.1.1.2" xref="S2.E1.m2.2.2.1.1.1.2.cmml">l</mi><mo id="S2.E1.m2.2.2.1.1.1.1" xref="S2.E1.m2.2.2.1.1.1.1.cmml">−</mo><mn id="S2.E1.m2.2.2.1.1.1.3" xref="S2.E1.m2.2.2.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="S2.E1.m2.2.2.1.1.3" xref="S2.E1.m2.2.2.1.1.1.cmml">)</mo></mrow></msubsup><mo id="S2.E1.m2.8.8.3.3.3.5" xref="S2.E1.m2.8.8.3.3.4.cmml">,</mo><msubsup id="S2.E1.m2.7.7.2.2.2.2" xref="S2.E1.m2.7.7.2.2.2.2.cmml"><mi id="S2.E1.m2.7.7.2.2.2.2.2.2" xref="S2.E1.m2.7.7.2.2.2.2.2.2.cmml">h</mi><mn id="S2.E1.m2.7.7.2.2.2.2.3" xref="S2.E1.m2.7.7.2.2.2.2.3.cmml">2</mn><mrow id="S2.E1.m2.3.3.1.1" xref="S2.E1.m2.3.3.1.1.1.cmml"><mo stretchy="false" id="S2.E1.m2.3.3.1.1.2" xref="S2.E1.m2.3.3.1.1.1.cmml">(</mo><mrow id="S2.E1.m2.3.3.1.1.1" xref="S2.E1.m2.3.3.1.1.1.cmml"><mi id="S2.E1.m2.3.3.1.1.1.2" xref="S2.E1.m2.3.3.1.1.1.2.cmml">l</mi><mo id="S2.E1.m2.3.3.1.1.1.1" xref="S2.E1.m2.3.3.1.1.1.1.cmml">−</mo><mn id="S2.E1.m2.3.3.1.1.1.3" xref="S2.E1.m2.3.3.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="S2.E1.m2.3.3.1.1.3" xref="S2.E1.m2.3.3.1.1.1.cmml">)</mo></mrow></msubsup><mo id="S2.E1.m2.8.8.3.3.3.6" xref="S2.E1.m2.8.8.3.3.4.cmml">,</mo><mi mathvariant="normal" id="S2.E1.m2.5.5" xref="S2.E1.m2.5.5.cmml">…</mi><mo id="S2.E1.m2.8.8.3.3.3.7" xref="S2.E1.m2.8.8.3.3.4.cmml">,</mo><msubsup id="S2.E1.m2.8.8.3.3.3.3" xref="S2.E1.m2.8.8.3.3.3.3.cmml"><mi id="S2.E1.m2.8.8.3.3.3.3.2.2" xref="S2.E1.m2.8.8.3.3.3.3.2.2.cmml">h</mi><mi id="S2.E1.m2.8.8.3.3.3.3.3" xref="S2.E1.m2.8.8.3.3.3.3.3.cmml">i</mi><mrow id="S2.E1.m2.4.4.1.1" xref="S2.E1.m2.4.4.1.1.1.cmml"><mo stretchy="false" id="S2.E1.m2.4.4.1.1.2" xref="S2.E1.m2.4.4.1.1.1.cmml">(</mo><mrow id="S2.E1.m2.4.4.1.1.1" xref="S2.E1.m2.4.4.1.1.1.cmml"><mi id="S2.E1.m2.4.4.1.1.1.2" xref="S2.E1.m2.4.4.1.1.1.2.cmml">l</mi><mo id="S2.E1.m2.4.4.1.1.1.1" xref="S2.E1.m2.4.4.1.1.1.1.cmml">−</mo><mn id="S2.E1.m2.4.4.1.1.1.3" xref="S2.E1.m2.4.4.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="S2.E1.m2.4.4.1.1.3" xref="S2.E1.m2.4.4.1.1.1.cmml">)</mo></mrow></msubsup><mo id="S2.E1.m2.8.8.3.3.3.8" xref="S2.E1.m2.8.8.3.3.4.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E1.m2.8b"><apply id="S2.E1.m2.8.8.cmml" xref="S2.E1.m2.8.8"><eq id="S2.E1.m2.8.8.4.cmml" xref="S2.E1.m2.8.8.4"></eq><csymbol cd="latexml" id="S2.E1.m2.8.8.5.cmml" xref="S2.E1.m2.8.8.5">absent</csymbol><apply id="S2.E1.m2.8.8.3.cmml" xref="S2.E1.m2.8.8.3"><times id="S2.E1.m2.8.8.3.4.cmml" xref="S2.E1.m2.8.8.3.4"></times><apply id="S2.E1.m2.8.8.3.5.cmml" xref="S2.E1.m2.8.8.3.5"><csymbol cd="ambiguous" id="S2.E1.m2.8.8.3.5.1.cmml" xref="S2.E1.m2.8.8.3.5">superscript</csymbol><ci id="S2.E1.m2.8.8.3.5.2.cmml" xref="S2.E1.m2.8.8.3.5.2">attn</ci><ci id="S2.E1.m2.1.1.1.1.cmml" xref="S2.E1.m2.1.1.1.1">𝑙</ci></apply><vector id="S2.E1.m2.8.8.3.3.4.cmml" xref="S2.E1.m2.8.8.3.3.3"><apply id="S2.E1.m2.6.6.1.1.1.1.cmml" xref="S2.E1.m2.6.6.1.1.1.1"><csymbol cd="ambiguous" id="S2.E1.m2.6.6.1.1.1.1.1.cmml" xref="S2.E1.m2.6.6.1.1.1.1">subscript</csymbol><apply id="S2.E1.m2.6.6.1.1.1.1.2.cmml" xref="S2.E1.m2.6.6.1.1.1.1"><csymbol cd="ambiguous" id="S2.E1.m2.6.6.1.1.1.1.2.1.cmml" xref="S2.E1.m2.6.6.1.1.1.1">superscript</csymbol><ci id="S2.E1.m2.6.6.1.1.1.1.2.2.cmml" xref="S2.E1.m2.6.6.1.1.1.1.2.2">ℎ</ci><apply id="S2.E1.m2.2.2.1.1.1.cmml" xref="S2.E1.m2.2.2.1.1"><minus id="S2.E1.m2.2.2.1.1.1.1.cmml" xref="S2.E1.m2.2.2.1.1.1.1"></minus><ci id="S2.E1.m2.2.2.1.1.1.2.cmml" xref="S2.E1.m2.2.2.1.1.1.2">𝑙</ci><cn type="integer" id="S2.E1.m2.2.2.1.1.1.3.cmml" xref="S2.E1.m2.2.2.1.1.1.3">1</cn></apply></apply><cn type="integer" id="S2.E1.m2.6.6.1.1.1.1.3.cmml" xref="S2.E1.m2.6.6.1.1.1.1.3">1</cn></apply><apply id="S2.E1.m2.7.7.2.2.2.2.cmml" xref="S2.E1.m2.7.7.2.2.2.2"><csymbol cd="ambiguous" id="S2.E1.m2.7.7.2.2.2.2.1.cmml" xref="S2.E1.m2.7.7.2.2.2.2">subscript</csymbol><apply id="S2.E1.m2.7.7.2.2.2.2.2.cmml" xref="S2.E1.m2.7.7.2.2.2.2"><csymbol cd="ambiguous" id="S2.E1.m2.7.7.2.2.2.2.2.1.cmml" xref="S2.E1.m2.7.7.2.2.2.2">superscript</csymbol><ci id="S2.E1.m2.7.7.2.2.2.2.2.2.cmml" xref="S2.E1.m2.7.7.2.2.2.2.2.2">ℎ</ci><apply id="S2.E1.m2.3.3.1.1.1.cmml" xref="S2.E1.m2.3.3.1.1"><minus id="S2.E1.m2.3.3.1.1.1.1.cmml" xref="S2.E1.m2.3.3.1.1.1.1"></minus><ci id="S2.E1.m2.3.3.1.1.1.2.cmml" xref="S2.E1.m2.3.3.1.1.1.2">𝑙</ci><cn type="integer" id="S2.E1.m2.3.3.1.1.1.3.cmml" xref="S2.E1.m2.3.3.1.1.1.3">1</cn></apply></apply><cn type="integer" id="S2.E1.m2.7.7.2.2.2.2.3.cmml" xref="S2.E1.m2.7.7.2.2.2.2.3">2</cn></apply><ci id="S2.E1.m2.5.5.cmml" xref="S2.E1.m2.5.5">…</ci><apply id="S2.E1.m2.8.8.3.3.3.3.cmml" xref="S2.E1.m2.8.8.3.3.3.3"><csymbol cd="ambiguous" id="S2.E1.m2.8.8.3.3.3.3.1.cmml" xref="S2.E1.m2.8.8.3.3.3.3">subscript</csymbol><apply id="S2.E1.m2.8.8.3.3.3.3.2.cmml" xref="S2.E1.m2.8.8.3.3.3.3"><csymbol cd="ambiguous" id="S2.E1.m2.8.8.3.3.3.3.2.1.cmml" xref="S2.E1.m2.8.8.3.3.3.3">superscript</csymbol><ci id="S2.E1.m2.8.8.3.3.3.3.2.2.cmml" xref="S2.E1.m2.8.8.3.3.3.3.2.2">ℎ</ci><apply id="S2.E1.m2.4.4.1.1.1.cmml" xref="S2.E1.m2.4.4.1.1"><minus id="S2.E1.m2.4.4.1.1.1.1.cmml" xref="S2.E1.m2.4.4.1.1.1.1"></minus><ci id="S2.E1.m2.4.4.1.1.1.2.cmml" xref="S2.E1.m2.4.4.1.1.1.2">𝑙</ci><cn type="integer" id="S2.E1.m2.4.4.1.1.1.3.cmml" xref="S2.E1.m2.4.4.1.1.1.3">1</cn></apply></apply><ci id="S2.E1.m2.8.8.3.3.3.3.3.cmml" xref="S2.E1.m2.8.8.3.3.3.3.3">𝑖</ci></apply></vector></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E1.m2.8c">\displaystyle=\mathrm{attn}^{(l)}\left(h^{(l-1)}_{1},h^{(l-1)}_{2},\dots,h^{(l-1)}_{i}\right)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
<tbody id="S2.Ex2"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S2.Ex2.m1.1" class="ltx_Math" alttext="\displaystyle m^{(l)}_{i}" display="inline"><semantics id="S2.Ex2.m1.1a"><msubsup id="S2.Ex2.m1.1.2" xref="S2.Ex2.m1.1.2.cmml"><mi id="S2.Ex2.m1.1.2.2.2" xref="S2.Ex2.m1.1.2.2.2.cmml">m</mi><mi id="S2.Ex2.m1.1.2.3" xref="S2.Ex2.m1.1.2.3.cmml">i</mi><mrow id="S2.Ex2.m1.1.1.1.3" xref="S2.Ex2.m1.1.2.cmml"><mo stretchy="false" id="S2.Ex2.m1.1.1.1.3.1" xref="S2.Ex2.m1.1.2.cmml">(</mo><mi id="S2.Ex2.m1.1.1.1.1" xref="S2.Ex2.m1.1.1.1.1.cmml">l</mi><mo stretchy="false" id="S2.Ex2.m1.1.1.1.3.2" xref="S2.Ex2.m1.1.2.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S2.Ex2.m1.1b"><apply id="S2.Ex2.m1.1.2.cmml" xref="S2.Ex2.m1.1.2"><csymbol cd="ambiguous" id="S2.Ex2.m1.1.2.1.cmml" xref="S2.Ex2.m1.1.2">subscript</csymbol><apply id="S2.Ex2.m1.1.2.2.cmml" xref="S2.Ex2.m1.1.2"><csymbol cd="ambiguous" id="S2.Ex2.m1.1.2.2.1.cmml" xref="S2.Ex2.m1.1.2">superscript</csymbol><ci id="S2.Ex2.m1.1.2.2.2.cmml" xref="S2.Ex2.m1.1.2.2.2">𝑚</ci><ci id="S2.Ex2.m1.1.1.1.1.cmml" xref="S2.Ex2.m1.1.1.1.1">𝑙</ci></apply><ci id="S2.Ex2.m1.1.2.3.cmml" xref="S2.Ex2.m1.1.2.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.Ex2.m1.1c">\displaystyle m^{(l)}_{i}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S2.Ex2.m2.5" class="ltx_Math" alttext="\displaystyle=W_{proj}^{(l)}\,\sigma\left(W_{fc}^{(l)}\gamma\left(a^{(l)}_{i}+h^{(l-1)}_{i}\right)\right)." display="inline"><semantics id="S2.Ex2.m2.5a"><mrow id="S2.Ex2.m2.5.5.1" xref="S2.Ex2.m2.5.5.1.1.cmml"><mrow id="S2.Ex2.m2.5.5.1.1" xref="S2.Ex2.m2.5.5.1.1.cmml"><mi id="S2.Ex2.m2.5.5.1.1.3" xref="S2.Ex2.m2.5.5.1.1.3.cmml"></mi><mo id="S2.Ex2.m2.5.5.1.1.2" xref="S2.Ex2.m2.5.5.1.1.2.cmml">=</mo><mrow id="S2.Ex2.m2.5.5.1.1.1" xref="S2.Ex2.m2.5.5.1.1.1.cmml"><msubsup id="S2.Ex2.m2.5.5.1.1.1.3" xref="S2.Ex2.m2.5.5.1.1.1.3.cmml"><mi id="S2.Ex2.m2.5.5.1.1.1.3.2.2" xref="S2.Ex2.m2.5.5.1.1.1.3.2.2.cmml">W</mi><mrow id="S2.Ex2.m2.5.5.1.1.1.3.2.3" xref="S2.Ex2.m2.5.5.1.1.1.3.2.3.cmml"><mi id="S2.Ex2.m2.5.5.1.1.1.3.2.3.2" xref="S2.Ex2.m2.5.5.1.1.1.3.2.3.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="S2.Ex2.m2.5.5.1.1.1.3.2.3.1" xref="S2.Ex2.m2.5.5.1.1.1.3.2.3.1.cmml">​</mo><mi id="S2.Ex2.m2.5.5.1.1.1.3.2.3.3" xref="S2.Ex2.m2.5.5.1.1.1.3.2.3.3.cmml">r</mi><mo lspace="0em" rspace="0em" id="S2.Ex2.m2.5.5.1.1.1.3.2.3.1a" xref="S2.Ex2.m2.5.5.1.1.1.3.2.3.1.cmml">​</mo><mi id="S2.Ex2.m2.5.5.1.1.1.3.2.3.4" xref="S2.Ex2.m2.5.5.1.1.1.3.2.3.4.cmml">o</mi><mo lspace="0em" rspace="0em" id="S2.Ex2.m2.5.5.1.1.1.3.2.3.1b" xref="S2.Ex2.m2.5.5.1.1.1.3.2.3.1.cmml">​</mo><mi id="S2.Ex2.m2.5.5.1.1.1.3.2.3.5" xref="S2.Ex2.m2.5.5.1.1.1.3.2.3.5.cmml">j</mi></mrow><mrow id="S2.Ex2.m2.1.1.1.3" xref="S2.Ex2.m2.5.5.1.1.1.3.cmml"><mo stretchy="false" id="S2.Ex2.m2.1.1.1.3.1" xref="S2.Ex2.m2.5.5.1.1.1.3.cmml">(</mo><mi id="S2.Ex2.m2.1.1.1.1" xref="S2.Ex2.m2.1.1.1.1.cmml">l</mi><mo stretchy="false" id="S2.Ex2.m2.1.1.1.3.2" xref="S2.Ex2.m2.5.5.1.1.1.3.cmml">)</mo></mrow></msubsup><mo lspace="0em" rspace="0em" id="S2.Ex2.m2.5.5.1.1.1.2" xref="S2.Ex2.m2.5.5.1.1.1.2.cmml">​</mo><mi id="S2.Ex2.m2.5.5.1.1.1.4" xref="S2.Ex2.m2.5.5.1.1.1.4.cmml">σ</mi><mo lspace="0em" rspace="0em" id="S2.Ex2.m2.5.5.1.1.1.2a" xref="S2.Ex2.m2.5.5.1.1.1.2.cmml">​</mo><mrow id="S2.Ex2.m2.5.5.1.1.1.1.1" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.cmml"><mo id="S2.Ex2.m2.5.5.1.1.1.1.1.2" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.Ex2.m2.5.5.1.1.1.1.1.1" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.cmml"><msubsup id="S2.Ex2.m2.5.5.1.1.1.1.1.1.3" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.3.cmml"><mi id="S2.Ex2.m2.5.5.1.1.1.1.1.1.3.2.2" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.3.2.2.cmml">W</mi><mrow id="S2.Ex2.m2.5.5.1.1.1.1.1.1.3.2.3" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.3.2.3.cmml"><mi id="S2.Ex2.m2.5.5.1.1.1.1.1.1.3.2.3.2" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.3.2.3.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="S2.Ex2.m2.5.5.1.1.1.1.1.1.3.2.3.1" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.3.2.3.1.cmml">​</mo><mi id="S2.Ex2.m2.5.5.1.1.1.1.1.1.3.2.3.3" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.3.2.3.3.cmml">c</mi></mrow><mrow id="S2.Ex2.m2.2.2.1.3" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.3.cmml"><mo stretchy="false" id="S2.Ex2.m2.2.2.1.3.1" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.3.cmml">(</mo><mi id="S2.Ex2.m2.2.2.1.1" xref="S2.Ex2.m2.2.2.1.1.cmml">l</mi><mo stretchy="false" id="S2.Ex2.m2.2.2.1.3.2" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.3.cmml">)</mo></mrow></msubsup><mo lspace="0em" rspace="0em" id="S2.Ex2.m2.5.5.1.1.1.1.1.1.2" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.2.cmml">​</mo><mi id="S2.Ex2.m2.5.5.1.1.1.1.1.1.4" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.4.cmml">γ</mi><mo lspace="0em" rspace="0em" id="S2.Ex2.m2.5.5.1.1.1.1.1.1.2a" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.cmml"><mo id="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.2" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.cmml"><msubsup id="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.2" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.2.cmml"><mi id="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.2.2.2" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.2.2.2.cmml">a</mi><mi id="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.2.3" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.2.3.cmml">i</mi><mrow id="S2.Ex2.m2.3.3.1.3" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S2.Ex2.m2.3.3.1.3.1" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.2.cmml">(</mo><mi id="S2.Ex2.m2.3.3.1.1" xref="S2.Ex2.m2.3.3.1.1.cmml">l</mi><mo stretchy="false" id="S2.Ex2.m2.3.3.1.3.2" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.2.cmml">)</mo></mrow></msubsup><mo id="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.1" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.1.cmml">+</mo><msubsup id="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.3" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.3.2.2" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.3.2.2.cmml">h</mi><mi id="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.3.3" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.3.3.cmml">i</mi><mrow id="S2.Ex2.m2.4.4.1.1" xref="S2.Ex2.m2.4.4.1.1.1.cmml"><mo stretchy="false" id="S2.Ex2.m2.4.4.1.1.2" xref="S2.Ex2.m2.4.4.1.1.1.cmml">(</mo><mrow id="S2.Ex2.m2.4.4.1.1.1" xref="S2.Ex2.m2.4.4.1.1.1.cmml"><mi id="S2.Ex2.m2.4.4.1.1.1.2" xref="S2.Ex2.m2.4.4.1.1.1.2.cmml">l</mi><mo id="S2.Ex2.m2.4.4.1.1.1.1" xref="S2.Ex2.m2.4.4.1.1.1.1.cmml">−</mo><mn id="S2.Ex2.m2.4.4.1.1.1.3" xref="S2.Ex2.m2.4.4.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="S2.Ex2.m2.4.4.1.1.3" xref="S2.Ex2.m2.4.4.1.1.1.cmml">)</mo></mrow></msubsup></mrow><mo id="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.3" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S2.Ex2.m2.5.5.1.1.1.1.1.3" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo lspace="0em" id="S2.Ex2.m2.5.5.1.2" xref="S2.Ex2.m2.5.5.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.Ex2.m2.5b"><apply id="S2.Ex2.m2.5.5.1.1.cmml" xref="S2.Ex2.m2.5.5.1"><eq id="S2.Ex2.m2.5.5.1.1.2.cmml" xref="S2.Ex2.m2.5.5.1.1.2"></eq><csymbol cd="latexml" id="S2.Ex2.m2.5.5.1.1.3.cmml" xref="S2.Ex2.m2.5.5.1.1.3">absent</csymbol><apply id="S2.Ex2.m2.5.5.1.1.1.cmml" xref="S2.Ex2.m2.5.5.1.1.1"><times id="S2.Ex2.m2.5.5.1.1.1.2.cmml" xref="S2.Ex2.m2.5.5.1.1.1.2"></times><apply id="S2.Ex2.m2.5.5.1.1.1.3.cmml" xref="S2.Ex2.m2.5.5.1.1.1.3"><csymbol cd="ambiguous" id="S2.Ex2.m2.5.5.1.1.1.3.1.cmml" xref="S2.Ex2.m2.5.5.1.1.1.3">superscript</csymbol><apply id="S2.Ex2.m2.5.5.1.1.1.3.2.cmml" xref="S2.Ex2.m2.5.5.1.1.1.3"><csymbol cd="ambiguous" id="S2.Ex2.m2.5.5.1.1.1.3.2.1.cmml" xref="S2.Ex2.m2.5.5.1.1.1.3">subscript</csymbol><ci id="S2.Ex2.m2.5.5.1.1.1.3.2.2.cmml" xref="S2.Ex2.m2.5.5.1.1.1.3.2.2">𝑊</ci><apply id="S2.Ex2.m2.5.5.1.1.1.3.2.3.cmml" xref="S2.Ex2.m2.5.5.1.1.1.3.2.3"><times id="S2.Ex2.m2.5.5.1.1.1.3.2.3.1.cmml" xref="S2.Ex2.m2.5.5.1.1.1.3.2.3.1"></times><ci id="S2.Ex2.m2.5.5.1.1.1.3.2.3.2.cmml" xref="S2.Ex2.m2.5.5.1.1.1.3.2.3.2">𝑝</ci><ci id="S2.Ex2.m2.5.5.1.1.1.3.2.3.3.cmml" xref="S2.Ex2.m2.5.5.1.1.1.3.2.3.3">𝑟</ci><ci id="S2.Ex2.m2.5.5.1.1.1.3.2.3.4.cmml" xref="S2.Ex2.m2.5.5.1.1.1.3.2.3.4">𝑜</ci><ci id="S2.Ex2.m2.5.5.1.1.1.3.2.3.5.cmml" xref="S2.Ex2.m2.5.5.1.1.1.3.2.3.5">𝑗</ci></apply></apply><ci id="S2.Ex2.m2.1.1.1.1.cmml" xref="S2.Ex2.m2.1.1.1.1">𝑙</ci></apply><ci id="S2.Ex2.m2.5.5.1.1.1.4.cmml" xref="S2.Ex2.m2.5.5.1.1.1.4">𝜎</ci><apply id="S2.Ex2.m2.5.5.1.1.1.1.1.1.cmml" xref="S2.Ex2.m2.5.5.1.1.1.1.1"><times id="S2.Ex2.m2.5.5.1.1.1.1.1.1.2.cmml" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.2"></times><apply id="S2.Ex2.m2.5.5.1.1.1.1.1.1.3.cmml" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.Ex2.m2.5.5.1.1.1.1.1.1.3.1.cmml" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.3">superscript</csymbol><apply id="S2.Ex2.m2.5.5.1.1.1.1.1.1.3.2.cmml" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.Ex2.m2.5.5.1.1.1.1.1.1.3.2.1.cmml" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.Ex2.m2.5.5.1.1.1.1.1.1.3.2.2.cmml" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.3.2.2">𝑊</ci><apply id="S2.Ex2.m2.5.5.1.1.1.1.1.1.3.2.3.cmml" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.3.2.3"><times id="S2.Ex2.m2.5.5.1.1.1.1.1.1.3.2.3.1.cmml" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.3.2.3.1"></times><ci id="S2.Ex2.m2.5.5.1.1.1.1.1.1.3.2.3.2.cmml" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.3.2.3.2">𝑓</ci><ci id="S2.Ex2.m2.5.5.1.1.1.1.1.1.3.2.3.3.cmml" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.3.2.3.3">𝑐</ci></apply></apply><ci id="S2.Ex2.m2.2.2.1.1.cmml" xref="S2.Ex2.m2.2.2.1.1">𝑙</ci></apply><ci id="S2.Ex2.m2.5.5.1.1.1.1.1.1.4.cmml" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.4">𝛾</ci><apply id="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.cmml" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1"><plus id="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.1"></plus><apply id="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.2">subscript</csymbol><apply id="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.2.2.1.cmml" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.2">superscript</csymbol><ci id="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.2.2.2.cmml" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.2.2.2">𝑎</ci><ci id="S2.Ex2.m2.3.3.1.1.cmml" xref="S2.Ex2.m2.3.3.1.1">𝑙</ci></apply><ci id="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.2.3">𝑖</ci></apply><apply id="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.3">subscript</csymbol><apply id="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.3.2.1.cmml" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.3">superscript</csymbol><ci id="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.3.2.2.cmml" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.3.2.2">ℎ</ci><apply id="S2.Ex2.m2.4.4.1.1.1.cmml" xref="S2.Ex2.m2.4.4.1.1"><minus id="S2.Ex2.m2.4.4.1.1.1.1.cmml" xref="S2.Ex2.m2.4.4.1.1.1.1"></minus><ci id="S2.Ex2.m2.4.4.1.1.1.2.cmml" xref="S2.Ex2.m2.4.4.1.1.1.2">𝑙</ci><cn type="integer" id="S2.Ex2.m2.4.4.1.1.1.3.cmml" xref="S2.Ex2.m2.4.4.1.1.1.3">1</cn></apply></apply><ci id="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S2.Ex2.m2.5.5.1.1.1.1.1.1.1.1.1.3.3">𝑖</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.Ex2.m2.5c">\displaystyle=W_{proj}^{(l)}\,\sigma\left(W_{fc}^{(l)}\gamma\left(a^{(l)}_{i}+h^{(l-1)}_{i}\right)\right).</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S2.p3.12" class="ltx_p">Each layer’s MLP is a two-layer neural network parameterized by matrices <math id="S2.p3.9.m1.1" class="ltx_Math" alttext="\smash{W_{proj}^{(l)}}" display="inline"><semantics id="S2.p3.9.m1.1a"><msubsup id="S2.p3.9.m1.1.2" xref="S2.p3.9.m1.1.2.cmml"><mi id="S2.p3.9.m1.1.2.2.2" xref="S2.p3.9.m1.1.2.2.2.cmml">W</mi><mrow id="S2.p3.9.m1.1.2.2.3" xref="S2.p3.9.m1.1.2.2.3.cmml"><mi id="S2.p3.9.m1.1.2.2.3.2" xref="S2.p3.9.m1.1.2.2.3.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="S2.p3.9.m1.1.2.2.3.1" xref="S2.p3.9.m1.1.2.2.3.1.cmml">​</mo><mi id="S2.p3.9.m1.1.2.2.3.3" xref="S2.p3.9.m1.1.2.2.3.3.cmml">r</mi><mo lspace="0em" rspace="0em" id="S2.p3.9.m1.1.2.2.3.1a" xref="S2.p3.9.m1.1.2.2.3.1.cmml">​</mo><mi id="S2.p3.9.m1.1.2.2.3.4" xref="S2.p3.9.m1.1.2.2.3.4.cmml">o</mi><mo lspace="0em" rspace="0em" id="S2.p3.9.m1.1.2.2.3.1b" xref="S2.p3.9.m1.1.2.2.3.1.cmml">​</mo><mi id="S2.p3.9.m1.1.2.2.3.5" xref="S2.p3.9.m1.1.2.2.3.5.cmml">j</mi></mrow><mrow id="S2.p3.9.m1.1.1.1.3" xref="S2.p3.9.m1.1.2.cmml"><mo stretchy="false" id="S2.p3.9.m1.1.1.1.3.1" xref="S2.p3.9.m1.1.2.cmml">(</mo><mi id="S2.p3.9.m1.1.1.1.1" xref="S2.p3.9.m1.1.1.1.1.cmml">l</mi><mo stretchy="false" id="S2.p3.9.m1.1.1.1.3.2" xref="S2.p3.9.m1.1.2.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S2.p3.9.m1.1b"><apply id="S2.p3.9.m1.1.2.cmml" xref="S2.p3.9.m1.1.2"><csymbol cd="ambiguous" id="S2.p3.9.m1.1.2.1.cmml" xref="S2.p3.9.m1.1.2">superscript</csymbol><apply id="S2.p3.9.m1.1.2.2.cmml" xref="S2.p3.9.m1.1.2"><csymbol cd="ambiguous" id="S2.p3.9.m1.1.2.2.1.cmml" xref="S2.p3.9.m1.1.2">subscript</csymbol><ci id="S2.p3.9.m1.1.2.2.2.cmml" xref="S2.p3.9.m1.1.2.2.2">𝑊</ci><apply id="S2.p3.9.m1.1.2.2.3.cmml" xref="S2.p3.9.m1.1.2.2.3"><times id="S2.p3.9.m1.1.2.2.3.1.cmml" xref="S2.p3.9.m1.1.2.2.3.1"></times><ci id="S2.p3.9.m1.1.2.2.3.2.cmml" xref="S2.p3.9.m1.1.2.2.3.2">𝑝</ci><ci id="S2.p3.9.m1.1.2.2.3.3.cmml" xref="S2.p3.9.m1.1.2.2.3.3">𝑟</ci><ci id="S2.p3.9.m1.1.2.2.3.4.cmml" xref="S2.p3.9.m1.1.2.2.3.4">𝑜</ci><ci id="S2.p3.9.m1.1.2.2.3.5.cmml" xref="S2.p3.9.m1.1.2.2.3.5">𝑗</ci></apply></apply><ci id="S2.p3.9.m1.1.1.1.1.cmml" xref="S2.p3.9.m1.1.1.1.1">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p3.9.m1.1c">\smash{W_{proj}^{(l)}}</annotation></semantics></math> and <math id="S2.p3.10.m2.1" class="ltx_Math" alttext="\smash{W_{fc}^{(l)}}" display="inline"><semantics id="S2.p3.10.m2.1a"><msubsup id="S2.p3.10.m2.1.2" xref="S2.p3.10.m2.1.2.cmml"><mi id="S2.p3.10.m2.1.2.2.2" xref="S2.p3.10.m2.1.2.2.2.cmml">W</mi><mrow id="S2.p3.10.m2.1.2.2.3" xref="S2.p3.10.m2.1.2.2.3.cmml"><mi id="S2.p3.10.m2.1.2.2.3.2" xref="S2.p3.10.m2.1.2.2.3.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="S2.p3.10.m2.1.2.2.3.1" xref="S2.p3.10.m2.1.2.2.3.1.cmml">​</mo><mi id="S2.p3.10.m2.1.2.2.3.3" xref="S2.p3.10.m2.1.2.2.3.3.cmml">c</mi></mrow><mrow id="S2.p3.10.m2.1.1.1.3" xref="S2.p3.10.m2.1.2.cmml"><mo stretchy="false" id="S2.p3.10.m2.1.1.1.3.1" xref="S2.p3.10.m2.1.2.cmml">(</mo><mi id="S2.p3.10.m2.1.1.1.1" xref="S2.p3.10.m2.1.1.1.1.cmml">l</mi><mo stretchy="false" id="S2.p3.10.m2.1.1.1.3.2" xref="S2.p3.10.m2.1.2.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S2.p3.10.m2.1b"><apply id="S2.p3.10.m2.1.2.cmml" xref="S2.p3.10.m2.1.2"><csymbol cd="ambiguous" id="S2.p3.10.m2.1.2.1.cmml" xref="S2.p3.10.m2.1.2">superscript</csymbol><apply id="S2.p3.10.m2.1.2.2.cmml" xref="S2.p3.10.m2.1.2"><csymbol cd="ambiguous" id="S2.p3.10.m2.1.2.2.1.cmml" xref="S2.p3.10.m2.1.2">subscript</csymbol><ci id="S2.p3.10.m2.1.2.2.2.cmml" xref="S2.p3.10.m2.1.2.2.2">𝑊</ci><apply id="S2.p3.10.m2.1.2.2.3.cmml" xref="S2.p3.10.m2.1.2.2.3"><times id="S2.p3.10.m2.1.2.2.3.1.cmml" xref="S2.p3.10.m2.1.2.2.3.1"></times><ci id="S2.p3.10.m2.1.2.2.3.2.cmml" xref="S2.p3.10.m2.1.2.2.3.2">𝑓</ci><ci id="S2.p3.10.m2.1.2.2.3.3.cmml" xref="S2.p3.10.m2.1.2.2.3.3">𝑐</ci></apply></apply><ci id="S2.p3.10.m2.1.1.1.1.cmml" xref="S2.p3.10.m2.1.1.1.1">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p3.10.m2.1c">\smash{W_{fc}^{(l)}}</annotation></semantics></math>, with rectifying nonlinearity <math id="S2.p3.11.m3.1" class="ltx_Math" alttext="\sigma" display="inline"><semantics id="S2.p3.11.m3.1a"><mi id="S2.p3.11.m3.1.1" xref="S2.p3.11.m3.1.1.cmml">σ</mi><annotation-xml encoding="MathML-Content" id="S2.p3.11.m3.1b"><ci id="S2.p3.11.m3.1.1.cmml" xref="S2.p3.11.m3.1.1">𝜎</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p3.11.m3.1c">\sigma</annotation></semantics></math> and normalizing nonlinearity <math id="S2.p3.12.m4.1" class="ltx_Math" alttext="\gamma" display="inline"><semantics id="S2.p3.12.m4.1a"><mi id="S2.p3.12.m4.1.1" xref="S2.p3.12.m4.1.1.cmml">γ</mi><annotation-xml encoding="MathML-Content" id="S2.p3.12.m4.1b"><ci id="S2.p3.12.m4.1.1.cmml" xref="S2.p3.12.m4.1.1">𝛾</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p3.12.m4.1c">\gamma</annotation></semantics></math>. For further background on transformers, we refer to <cite class="ltx_cite ltx_citemacro_cite">Vaswani et al. (<a href="#bib.bib36" title="" class="ltx_ref">2017</a>)</cite>.<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>Eqn. <a href="#S2.E1" title="In 2 Interventions on Activations for Tracing Information Flow ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> calculates attention sequentially after the MLP module as in <cite class="ltx_cite ltx_citemacro_citet">Brown et al. (<a href="#bib.bib7" title="" class="ltx_ref">2020</a>)</cite>. Our methods also apply to GPT variants such as <cite class="ltx_cite ltx_citemacro_citet">Wang &amp; Komatsuzaki (<a href="#bib.bib39" title="" class="ltx_ref">2021</a>)</cite> that put attention in parallel to the MLP.</span></span></span></p>
</div>
<figure id="S2.F2" class="ltx_figure"><img src="/html/2202.05262/assets/x2.png" id="S2.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="96" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S2.F2.3.1.1" class="ltx_text" style="font-size:90%;">Figure 2</span>: </span><span id="S2.F2.4.2" class="ltx_text ltx_font_bold" style="font-size:90%;">Average Indirect Effect<span id="S2.F2.4.2.1" class="ltx_text ltx_font_medium"> of individual model components over a sample of 1000 factual statements reveals two important sites. (a) Strong causality at a ‘late site’ in the last layers at the last token is unsurprising, but strongly causal states at an ‘early site’ in middle layers at the last subject token is a new discovery. (b) MLP contributions dominate the early site. (c) Attention is important at the late site. Appendix <a href="#A2" title="Appendix B Causal Tracing ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a>, Figure <a href="#A2.F7" title="Figure 7 ‣ B.2 Separating MLP and Attn Effects ‣ Appendix B Causal Tracing ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a> shows these heatmaps as line plots with 95% confidence intervals.</span></span></figcaption>
</figure>
<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>Causal Tracing of Factual Associations</h3>

<div id="S2.SS1.p1" class="ltx_para">
<p id="S2.SS1.p1.1" class="ltx_p">The grid of states (Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>)
forms a <em id="S2.SS1.p1.1.1" class="ltx_emph ltx_font_italic">causal graph</em> <cite class="ltx_cite ltx_citemacro_citep">(Pearl, <a href="#bib.bib29" title="" class="ltx_ref">2009</a>)</cite> describing dependencies between the hidden variables. This graph contains many paths from inputs on the left to the output (next-word prediction) at the lower-right, and we wish to understand if there are specific hidden state variables that are more important than others when recalling a fact.</p>
</div>
<div id="S2.SS1.p2" class="ltx_para">
<p id="S2.SS1.p2.1" class="ltx_p">As <cite class="ltx_cite ltx_citemacro_citet">Vig et al. (<a href="#bib.bib38" title="" class="ltx_ref">2020b</a>)</cite> have shown, this is a natural case for <span id="S2.SS1.p2.1.1" class="ltx_text ltx_font_italic">causal mediation analysis</span>, which quantifies the contribution of intermediate variables in causal graphs <cite class="ltx_cite ltx_citemacro_citep">(Pearl, <a href="#bib.bib28" title="" class="ltx_ref">2001</a>)</cite>. To calculate each state’s contribution towards a correct factual prediction, we observe all of <math id="S2.SS1.p2.1.m1.1" class="ltx_Math" alttext="G" display="inline"><semantics id="S2.SS1.p2.1.m1.1a"><mi id="S2.SS1.p2.1.m1.1.1" xref="S2.SS1.p2.1.m1.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.1.m1.1b"><ci id="S2.SS1.p2.1.m1.1.1.cmml" xref="S2.SS1.p2.1.m1.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.1.m1.1c">G</annotation></semantics></math>’s internal activations during three runs: a <span id="S2.SS1.p2.1.2" class="ltx_text ltx_font_bold">clean</span> run that predicts the fact, a <span id="S2.SS1.p2.1.3" class="ltx_text ltx_font_bold">corrupted</span> run where the prediction is damaged, and a <span id="S2.SS1.p2.1.4" class="ltx_text ltx_font_bold">corrupted-with-restoration</span> run that tests the ability of a single state to restore the prediction.</p>
</div>
<div id="S2.SS1.p3" class="ltx_para">
<ul id="S2.I1" class="ltx_itemize">
<li id="S2.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I1.i1.p1" class="ltx_para">
<p id="S2.I1.i1.p1.4" class="ltx_p">In the <span id="S2.I1.i1.p1.4.1" class="ltx_text ltx_font_bold">clean run</span>, we pass a factual prompt <math id="S2.I1.i1.p1.1.m1.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S2.I1.i1.p1.1.m1.1a"><mi id="S2.I1.i1.p1.1.m1.1.1" xref="S2.I1.i1.p1.1.m1.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S2.I1.i1.p1.1.m1.1b"><ci id="S2.I1.i1.p1.1.m1.1.1.cmml" xref="S2.I1.i1.p1.1.m1.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i1.p1.1.m1.1c">x</annotation></semantics></math> into <math id="S2.I1.i1.p1.2.m2.1" class="ltx_Math" alttext="G" display="inline"><semantics id="S2.I1.i1.p1.2.m2.1a"><mi id="S2.I1.i1.p1.2.m2.1.1" xref="S2.I1.i1.p1.2.m2.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="S2.I1.i1.p1.2.m2.1b"><ci id="S2.I1.i1.p1.2.m2.1.1.cmml" xref="S2.I1.i1.p1.2.m2.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i1.p1.2.m2.1c">G</annotation></semantics></math> and collect all hidden activations <math id="S2.I1.i1.p1.3.m3.7" class="ltx_Math" alttext="\left\{\smash{h^{(l)}_{i}}\mid i\in[1,T],l\in[1,L]\right\}" display="inline"><semantics id="S2.I1.i1.p1.3.m3.7a"><mrow id="S2.I1.i1.p1.3.m3.7.7.2" xref="S2.I1.i1.p1.3.m3.7.7.3.cmml"><mo id="S2.I1.i1.p1.3.m3.7.7.2.3" xref="S2.I1.i1.p1.3.m3.7.7.3.1.cmml">{</mo><msubsup id="S2.I1.i1.p1.3.m3.6.6.1.1" xref="S2.I1.i1.p1.3.m3.6.6.1.1.cmml"><mi id="S2.I1.i1.p1.3.m3.6.6.1.1.2.2" xref="S2.I1.i1.p1.3.m3.6.6.1.1.2.2.cmml">h</mi><mi id="S2.I1.i1.p1.3.m3.6.6.1.1.3" xref="S2.I1.i1.p1.3.m3.6.6.1.1.3.cmml">i</mi><mrow id="S2.I1.i1.p1.3.m3.1.1.1.3" xref="S2.I1.i1.p1.3.m3.6.6.1.1.cmml"><mo stretchy="false" id="S2.I1.i1.p1.3.m3.1.1.1.3.1" xref="S2.I1.i1.p1.3.m3.6.6.1.1.cmml">(</mo><mi id="S2.I1.i1.p1.3.m3.1.1.1.1" xref="S2.I1.i1.p1.3.m3.1.1.1.1.cmml">l</mi><mo stretchy="false" id="S2.I1.i1.p1.3.m3.1.1.1.3.2" xref="S2.I1.i1.p1.3.m3.6.6.1.1.cmml">)</mo></mrow></msubsup><mo fence="true" lspace="0em" rspace="0em" id="S2.I1.i1.p1.3.m3.7.7.2.4" xref="S2.I1.i1.p1.3.m3.7.7.3.1.cmml">∣</mo><mrow id="S2.I1.i1.p1.3.m3.7.7.2.2.2" xref="S2.I1.i1.p1.3.m3.7.7.2.2.3.cmml"><mrow id="S2.I1.i1.p1.3.m3.7.7.2.2.1.1" xref="S2.I1.i1.p1.3.m3.7.7.2.2.1.1.cmml"><mi id="S2.I1.i1.p1.3.m3.7.7.2.2.1.1.2" xref="S2.I1.i1.p1.3.m3.7.7.2.2.1.1.2.cmml">i</mi><mo id="S2.I1.i1.p1.3.m3.7.7.2.2.1.1.1" xref="S2.I1.i1.p1.3.m3.7.7.2.2.1.1.1.cmml">∈</mo><mrow id="S2.I1.i1.p1.3.m3.7.7.2.2.1.1.3.2" xref="S2.I1.i1.p1.3.m3.7.7.2.2.1.1.3.1.cmml"><mo stretchy="false" id="S2.I1.i1.p1.3.m3.7.7.2.2.1.1.3.2.1" xref="S2.I1.i1.p1.3.m3.7.7.2.2.1.1.3.1.cmml">[</mo><mn id="S2.I1.i1.p1.3.m3.2.2" xref="S2.I1.i1.p1.3.m3.2.2.cmml">1</mn><mo id="S2.I1.i1.p1.3.m3.7.7.2.2.1.1.3.2.2" xref="S2.I1.i1.p1.3.m3.7.7.2.2.1.1.3.1.cmml">,</mo><mi id="S2.I1.i1.p1.3.m3.3.3" xref="S2.I1.i1.p1.3.m3.3.3.cmml">T</mi><mo stretchy="false" id="S2.I1.i1.p1.3.m3.7.7.2.2.1.1.3.2.3" xref="S2.I1.i1.p1.3.m3.7.7.2.2.1.1.3.1.cmml">]</mo></mrow></mrow><mo id="S2.I1.i1.p1.3.m3.7.7.2.2.2.3" xref="S2.I1.i1.p1.3.m3.7.7.2.2.3a.cmml">,</mo><mrow id="S2.I1.i1.p1.3.m3.7.7.2.2.2.2" xref="S2.I1.i1.p1.3.m3.7.7.2.2.2.2.cmml"><mi id="S2.I1.i1.p1.3.m3.7.7.2.2.2.2.2" xref="S2.I1.i1.p1.3.m3.7.7.2.2.2.2.2.cmml">l</mi><mo id="S2.I1.i1.p1.3.m3.7.7.2.2.2.2.1" xref="S2.I1.i1.p1.3.m3.7.7.2.2.2.2.1.cmml">∈</mo><mrow id="S2.I1.i1.p1.3.m3.7.7.2.2.2.2.3.2" xref="S2.I1.i1.p1.3.m3.7.7.2.2.2.2.3.1.cmml"><mo stretchy="false" id="S2.I1.i1.p1.3.m3.7.7.2.2.2.2.3.2.1" xref="S2.I1.i1.p1.3.m3.7.7.2.2.2.2.3.1.cmml">[</mo><mn id="S2.I1.i1.p1.3.m3.4.4" xref="S2.I1.i1.p1.3.m3.4.4.cmml">1</mn><mo id="S2.I1.i1.p1.3.m3.7.7.2.2.2.2.3.2.2" xref="S2.I1.i1.p1.3.m3.7.7.2.2.2.2.3.1.cmml">,</mo><mi id="S2.I1.i1.p1.3.m3.5.5" xref="S2.I1.i1.p1.3.m3.5.5.cmml">L</mi><mo stretchy="false" id="S2.I1.i1.p1.3.m3.7.7.2.2.2.2.3.2.3" xref="S2.I1.i1.p1.3.m3.7.7.2.2.2.2.3.1.cmml">]</mo></mrow></mrow></mrow><mo id="S2.I1.i1.p1.3.m3.7.7.2.5" xref="S2.I1.i1.p1.3.m3.7.7.3.1.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.I1.i1.p1.3.m3.7b"><apply id="S2.I1.i1.p1.3.m3.7.7.3.cmml" xref="S2.I1.i1.p1.3.m3.7.7.2"><csymbol cd="latexml" id="S2.I1.i1.p1.3.m3.7.7.3.1.cmml" xref="S2.I1.i1.p1.3.m3.7.7.2.3">conditional-set</csymbol><apply id="S2.I1.i1.p1.3.m3.6.6.1.1.cmml" xref="S2.I1.i1.p1.3.m3.6.6.1.1"><csymbol cd="ambiguous" id="S2.I1.i1.p1.3.m3.6.6.1.1.1.cmml" xref="S2.I1.i1.p1.3.m3.6.6.1.1">subscript</csymbol><apply id="S2.I1.i1.p1.3.m3.6.6.1.1.2.cmml" xref="S2.I1.i1.p1.3.m3.6.6.1.1"><csymbol cd="ambiguous" id="S2.I1.i1.p1.3.m3.6.6.1.1.2.1.cmml" xref="S2.I1.i1.p1.3.m3.6.6.1.1">superscript</csymbol><ci id="S2.I1.i1.p1.3.m3.6.6.1.1.2.2.cmml" xref="S2.I1.i1.p1.3.m3.6.6.1.1.2.2">ℎ</ci><ci id="S2.I1.i1.p1.3.m3.1.1.1.1.cmml" xref="S2.I1.i1.p1.3.m3.1.1.1.1">𝑙</ci></apply><ci id="S2.I1.i1.p1.3.m3.6.6.1.1.3.cmml" xref="S2.I1.i1.p1.3.m3.6.6.1.1.3">𝑖</ci></apply><apply id="S2.I1.i1.p1.3.m3.7.7.2.2.3.cmml" xref="S2.I1.i1.p1.3.m3.7.7.2.2.2"><csymbol cd="ambiguous" id="S2.I1.i1.p1.3.m3.7.7.2.2.3a.cmml" xref="S2.I1.i1.p1.3.m3.7.7.2.2.2.3">formulae-sequence</csymbol><apply id="S2.I1.i1.p1.3.m3.7.7.2.2.1.1.cmml" xref="S2.I1.i1.p1.3.m3.7.7.2.2.1.1"><in id="S2.I1.i1.p1.3.m3.7.7.2.2.1.1.1.cmml" xref="S2.I1.i1.p1.3.m3.7.7.2.2.1.1.1"></in><ci id="S2.I1.i1.p1.3.m3.7.7.2.2.1.1.2.cmml" xref="S2.I1.i1.p1.3.m3.7.7.2.2.1.1.2">𝑖</ci><interval closure="closed" id="S2.I1.i1.p1.3.m3.7.7.2.2.1.1.3.1.cmml" xref="S2.I1.i1.p1.3.m3.7.7.2.2.1.1.3.2"><cn type="integer" id="S2.I1.i1.p1.3.m3.2.2.cmml" xref="S2.I1.i1.p1.3.m3.2.2">1</cn><ci id="S2.I1.i1.p1.3.m3.3.3.cmml" xref="S2.I1.i1.p1.3.m3.3.3">𝑇</ci></interval></apply><apply id="S2.I1.i1.p1.3.m3.7.7.2.2.2.2.cmml" xref="S2.I1.i1.p1.3.m3.7.7.2.2.2.2"><in id="S2.I1.i1.p1.3.m3.7.7.2.2.2.2.1.cmml" xref="S2.I1.i1.p1.3.m3.7.7.2.2.2.2.1"></in><ci id="S2.I1.i1.p1.3.m3.7.7.2.2.2.2.2.cmml" xref="S2.I1.i1.p1.3.m3.7.7.2.2.2.2.2">𝑙</ci><interval closure="closed" id="S2.I1.i1.p1.3.m3.7.7.2.2.2.2.3.1.cmml" xref="S2.I1.i1.p1.3.m3.7.7.2.2.2.2.3.2"><cn type="integer" id="S2.I1.i1.p1.3.m3.4.4.cmml" xref="S2.I1.i1.p1.3.m3.4.4">1</cn><ci id="S2.I1.i1.p1.3.m3.5.5.cmml" xref="S2.I1.i1.p1.3.m3.5.5">𝐿</ci></interval></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i1.p1.3.m3.7c">\left\{\smash{h^{(l)}_{i}}\mid i\in[1,T],l\in[1,L]\right\}</annotation></semantics></math>. Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>a provides an example illustration with the prompt: “The Space Needle is in downtown <span class="ltx_rule" style="width:28.5pt;height:0.4pt;background:black;display:inline-block;"> </span>”, for which the expected completion is <math id="S2.I1.i1.p1.4.m4.1" class="ltx_Math" alttext="o=" display="inline"><semantics id="S2.I1.i1.p1.4.m4.1a"><mrow id="S2.I1.i1.p1.4.m4.1.1" xref="S2.I1.i1.p1.4.m4.1.1.cmml"><mi id="S2.I1.i1.p1.4.m4.1.1.2" xref="S2.I1.i1.p1.4.m4.1.1.2.cmml">o</mi><mo id="S2.I1.i1.p1.4.m4.1.1.1" xref="S2.I1.i1.p1.4.m4.1.1.1.cmml">=</mo><mi id="S2.I1.i1.p1.4.m4.1.1.3" xref="S2.I1.i1.p1.4.m4.1.1.3.cmml"></mi></mrow><annotation-xml encoding="MathML-Content" id="S2.I1.i1.p1.4.m4.1b"><apply id="S2.I1.i1.p1.4.m4.1.1.cmml" xref="S2.I1.i1.p1.4.m4.1.1"><eq id="S2.I1.i1.p1.4.m4.1.1.1.cmml" xref="S2.I1.i1.p1.4.m4.1.1.1"></eq><ci id="S2.I1.i1.p1.4.m4.1.1.2.cmml" xref="S2.I1.i1.p1.4.m4.1.1.2">𝑜</ci><csymbol cd="latexml" id="S2.I1.i1.p1.4.m4.1.1.3.cmml" xref="S2.I1.i1.p1.4.m4.1.1.3">absent</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i1.p1.4.m4.1c">o=</annotation></semantics></math> “Seattle”.</p>
</div>
</li>
<li id="S2.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I1.i2.p1" class="ltx_para">
<p id="S2.I1.i2.p1.9" class="ltx_p">In the baseline <span id="S2.I1.i2.p1.9.1" class="ltx_text ltx_font_bold">corrupted run</span>, the subject is obfuscated from <math id="S2.I1.i2.p1.1.m1.1" class="ltx_Math" alttext="G" display="inline"><semantics id="S2.I1.i2.p1.1.m1.1a"><mi id="S2.I1.i2.p1.1.m1.1.1" xref="S2.I1.i2.p1.1.m1.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="S2.I1.i2.p1.1.m1.1b"><ci id="S2.I1.i2.p1.1.m1.1.1.cmml" xref="S2.I1.i2.p1.1.m1.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i2.p1.1.m1.1c">G</annotation></semantics></math> before the network runs. Concretely, immediately after <math id="S2.I1.i2.p1.2.m2.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S2.I1.i2.p1.2.m2.1a"><mi id="S2.I1.i2.p1.2.m2.1.1" xref="S2.I1.i2.p1.2.m2.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S2.I1.i2.p1.2.m2.1b"><ci id="S2.I1.i2.p1.2.m2.1.1.cmml" xref="S2.I1.i2.p1.2.m2.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i2.p1.2.m2.1c">x</annotation></semantics></math> is embedded as <math id="S2.I1.i2.p1.3.m3.7" class="ltx_Math" alttext="\left[\smash{h^{(0)}_{1}},\smash{h^{(0)}_{2}},\dots,\smash{h^{(0)}_{T}}\right]" display="inline"><semantics id="S2.I1.i2.p1.3.m3.7a"><mrow id="S2.I1.i2.p1.3.m3.7.7.3" xref="S2.I1.i2.p1.3.m3.7.7.4.cmml"><mo id="S2.I1.i2.p1.3.m3.7.7.3.4" xref="S2.I1.i2.p1.3.m3.7.7.4.cmml">[</mo><msubsup id="S2.I1.i2.p1.3.m3.5.5.1.1" xref="S2.I1.i2.p1.3.m3.5.5.1.1.cmml"><mi id="S2.I1.i2.p1.3.m3.5.5.1.1.2.2" xref="S2.I1.i2.p1.3.m3.5.5.1.1.2.2.cmml">h</mi><mn id="S2.I1.i2.p1.3.m3.5.5.1.1.3" xref="S2.I1.i2.p1.3.m3.5.5.1.1.3.cmml">1</mn><mrow id="S2.I1.i2.p1.3.m3.1.1.1.3" xref="S2.I1.i2.p1.3.m3.5.5.1.1.cmml"><mo stretchy="false" id="S2.I1.i2.p1.3.m3.1.1.1.3.1" xref="S2.I1.i2.p1.3.m3.5.5.1.1.cmml">(</mo><mn id="S2.I1.i2.p1.3.m3.1.1.1.1" xref="S2.I1.i2.p1.3.m3.1.1.1.1.cmml">0</mn><mo stretchy="false" id="S2.I1.i2.p1.3.m3.1.1.1.3.2" xref="S2.I1.i2.p1.3.m3.5.5.1.1.cmml">)</mo></mrow></msubsup><mo id="S2.I1.i2.p1.3.m3.7.7.3.5" xref="S2.I1.i2.p1.3.m3.7.7.4.cmml">,</mo><msubsup id="S2.I1.i2.p1.3.m3.6.6.2.2" xref="S2.I1.i2.p1.3.m3.6.6.2.2.cmml"><mi id="S2.I1.i2.p1.3.m3.6.6.2.2.2.2" xref="S2.I1.i2.p1.3.m3.6.6.2.2.2.2.cmml">h</mi><mn id="S2.I1.i2.p1.3.m3.6.6.2.2.3" xref="S2.I1.i2.p1.3.m3.6.6.2.2.3.cmml">2</mn><mrow id="S2.I1.i2.p1.3.m3.2.2.1.3" xref="S2.I1.i2.p1.3.m3.6.6.2.2.cmml"><mo stretchy="false" id="S2.I1.i2.p1.3.m3.2.2.1.3.1" xref="S2.I1.i2.p1.3.m3.6.6.2.2.cmml">(</mo><mn id="S2.I1.i2.p1.3.m3.2.2.1.1" xref="S2.I1.i2.p1.3.m3.2.2.1.1.cmml">0</mn><mo stretchy="false" id="S2.I1.i2.p1.3.m3.2.2.1.3.2" xref="S2.I1.i2.p1.3.m3.6.6.2.2.cmml">)</mo></mrow></msubsup><mo id="S2.I1.i2.p1.3.m3.7.7.3.6" xref="S2.I1.i2.p1.3.m3.7.7.4.cmml">,</mo><mi mathvariant="normal" id="S2.I1.i2.p1.3.m3.4.4" xref="S2.I1.i2.p1.3.m3.4.4.cmml">…</mi><mo id="S2.I1.i2.p1.3.m3.7.7.3.7" xref="S2.I1.i2.p1.3.m3.7.7.4.cmml">,</mo><msubsup id="S2.I1.i2.p1.3.m3.7.7.3.3" xref="S2.I1.i2.p1.3.m3.7.7.3.3.cmml"><mi id="S2.I1.i2.p1.3.m3.7.7.3.3.2.2" xref="S2.I1.i2.p1.3.m3.7.7.3.3.2.2.cmml">h</mi><mi id="S2.I1.i2.p1.3.m3.7.7.3.3.3" xref="S2.I1.i2.p1.3.m3.7.7.3.3.3.cmml">T</mi><mrow id="S2.I1.i2.p1.3.m3.3.3.1.3" xref="S2.I1.i2.p1.3.m3.7.7.3.3.cmml"><mo stretchy="false" id="S2.I1.i2.p1.3.m3.3.3.1.3.1" xref="S2.I1.i2.p1.3.m3.7.7.3.3.cmml">(</mo><mn id="S2.I1.i2.p1.3.m3.3.3.1.1" xref="S2.I1.i2.p1.3.m3.3.3.1.1.cmml">0</mn><mo stretchy="false" id="S2.I1.i2.p1.3.m3.3.3.1.3.2" xref="S2.I1.i2.p1.3.m3.7.7.3.3.cmml">)</mo></mrow></msubsup><mo id="S2.I1.i2.p1.3.m3.7.7.3.8" xref="S2.I1.i2.p1.3.m3.7.7.4.cmml">]</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.I1.i2.p1.3.m3.7b"><list id="S2.I1.i2.p1.3.m3.7.7.4.cmml" xref="S2.I1.i2.p1.3.m3.7.7.3"><apply id="S2.I1.i2.p1.3.m3.5.5.1.1.cmml" xref="S2.I1.i2.p1.3.m3.5.5.1.1"><csymbol cd="ambiguous" id="S2.I1.i2.p1.3.m3.5.5.1.1.1.cmml" xref="S2.I1.i2.p1.3.m3.5.5.1.1">subscript</csymbol><apply id="S2.I1.i2.p1.3.m3.5.5.1.1.2.cmml" xref="S2.I1.i2.p1.3.m3.5.5.1.1"><csymbol cd="ambiguous" id="S2.I1.i2.p1.3.m3.5.5.1.1.2.1.cmml" xref="S2.I1.i2.p1.3.m3.5.5.1.1">superscript</csymbol><ci id="S2.I1.i2.p1.3.m3.5.5.1.1.2.2.cmml" xref="S2.I1.i2.p1.3.m3.5.5.1.1.2.2">ℎ</ci><cn type="integer" id="S2.I1.i2.p1.3.m3.1.1.1.1.cmml" xref="S2.I1.i2.p1.3.m3.1.1.1.1">0</cn></apply><cn type="integer" id="S2.I1.i2.p1.3.m3.5.5.1.1.3.cmml" xref="S2.I1.i2.p1.3.m3.5.5.1.1.3">1</cn></apply><apply id="S2.I1.i2.p1.3.m3.6.6.2.2.cmml" xref="S2.I1.i2.p1.3.m3.6.6.2.2"><csymbol cd="ambiguous" id="S2.I1.i2.p1.3.m3.6.6.2.2.1.cmml" xref="S2.I1.i2.p1.3.m3.6.6.2.2">subscript</csymbol><apply id="S2.I1.i2.p1.3.m3.6.6.2.2.2.cmml" xref="S2.I1.i2.p1.3.m3.6.6.2.2"><csymbol cd="ambiguous" id="S2.I1.i2.p1.3.m3.6.6.2.2.2.1.cmml" xref="S2.I1.i2.p1.3.m3.6.6.2.2">superscript</csymbol><ci id="S2.I1.i2.p1.3.m3.6.6.2.2.2.2.cmml" xref="S2.I1.i2.p1.3.m3.6.6.2.2.2.2">ℎ</ci><cn type="integer" id="S2.I1.i2.p1.3.m3.2.2.1.1.cmml" xref="S2.I1.i2.p1.3.m3.2.2.1.1">0</cn></apply><cn type="integer" id="S2.I1.i2.p1.3.m3.6.6.2.2.3.cmml" xref="S2.I1.i2.p1.3.m3.6.6.2.2.3">2</cn></apply><ci id="S2.I1.i2.p1.3.m3.4.4.cmml" xref="S2.I1.i2.p1.3.m3.4.4">…</ci><apply id="S2.I1.i2.p1.3.m3.7.7.3.3.cmml" xref="S2.I1.i2.p1.3.m3.7.7.3.3"><csymbol cd="ambiguous" id="S2.I1.i2.p1.3.m3.7.7.3.3.1.cmml" xref="S2.I1.i2.p1.3.m3.7.7.3.3">subscript</csymbol><apply id="S2.I1.i2.p1.3.m3.7.7.3.3.2.cmml" xref="S2.I1.i2.p1.3.m3.7.7.3.3"><csymbol cd="ambiguous" id="S2.I1.i2.p1.3.m3.7.7.3.3.2.1.cmml" xref="S2.I1.i2.p1.3.m3.7.7.3.3">superscript</csymbol><ci id="S2.I1.i2.p1.3.m3.7.7.3.3.2.2.cmml" xref="S2.I1.i2.p1.3.m3.7.7.3.3.2.2">ℎ</ci><cn type="integer" id="S2.I1.i2.p1.3.m3.3.3.1.1.cmml" xref="S2.I1.i2.p1.3.m3.3.3.1.1">0</cn></apply><ci id="S2.I1.i2.p1.3.m3.7.7.3.3.3.cmml" xref="S2.I1.i2.p1.3.m3.7.7.3.3.3">𝑇</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i2.p1.3.m3.7c">\left[\smash{h^{(0)}_{1}},\smash{h^{(0)}_{2}},\dots,\smash{h^{(0)}_{T}}\right]</annotation></semantics></math>, we set <math id="S2.I1.i2.p1.4.m4.2" class="ltx_Math" alttext="\smash{h^{(0)}_{i}:=h^{(0)}_{i}}+\epsilon" display="inline"><semantics id="S2.I1.i2.p1.4.m4.2a"><mrow id="S2.I1.i2.p1.4.m4.2.3" xref="S2.I1.i2.p1.4.m4.2.3.cmml"><msubsup id="S2.I1.i2.p1.4.m4.2.3.2" xref="S2.I1.i2.p1.4.m4.2.3.2.cmml"><mi id="S2.I1.i2.p1.4.m4.2.3.2.2.2" xref="S2.I1.i2.p1.4.m4.2.3.2.2.2.cmml">h</mi><mi id="S2.I1.i2.p1.4.m4.2.3.2.3" xref="S2.I1.i2.p1.4.m4.2.3.2.3.cmml">i</mi><mrow id="S2.I1.i2.p1.4.m4.1.1.1.3" xref="S2.I1.i2.p1.4.m4.2.3.2.cmml"><mo stretchy="false" id="S2.I1.i2.p1.4.m4.1.1.1.3.1" xref="S2.I1.i2.p1.4.m4.2.3.2.cmml">(</mo><mn id="S2.I1.i2.p1.4.m4.1.1.1.1" xref="S2.I1.i2.p1.4.m4.1.1.1.1.cmml">0</mn><mo stretchy="false" id="S2.I1.i2.p1.4.m4.1.1.1.3.2" xref="S2.I1.i2.p1.4.m4.2.3.2.cmml">)</mo></mrow></msubsup><mo lspace="0.278em" rspace="0.278em" id="S2.I1.i2.p1.4.m4.2.3.1" xref="S2.I1.i2.p1.4.m4.2.3.1.cmml">:=</mo><mrow id="S2.I1.i2.p1.4.m4.2.3.3" xref="S2.I1.i2.p1.4.m4.2.3.3.cmml"><msubsup id="S2.I1.i2.p1.4.m4.2.3.3.2" xref="S2.I1.i2.p1.4.m4.2.3.3.2.cmml"><mi id="S2.I1.i2.p1.4.m4.2.3.3.2.2.2" xref="S2.I1.i2.p1.4.m4.2.3.3.2.2.2.cmml">h</mi><mi id="S2.I1.i2.p1.4.m4.2.3.3.2.3" xref="S2.I1.i2.p1.4.m4.2.3.3.2.3.cmml">i</mi><mrow id="S2.I1.i2.p1.4.m4.2.2.1.3" xref="S2.I1.i2.p1.4.m4.2.3.3.2.cmml"><mo stretchy="false" id="S2.I1.i2.p1.4.m4.2.2.1.3.1" xref="S2.I1.i2.p1.4.m4.2.3.3.2.cmml">(</mo><mn id="S2.I1.i2.p1.4.m4.2.2.1.1" xref="S2.I1.i2.p1.4.m4.2.2.1.1.cmml">0</mn><mo stretchy="false" id="S2.I1.i2.p1.4.m4.2.2.1.3.2" xref="S2.I1.i2.p1.4.m4.2.3.3.2.cmml">)</mo></mrow></msubsup><mo id="S2.I1.i2.p1.4.m4.2.3.3.1" xref="S2.I1.i2.p1.4.m4.2.3.3.1.cmml">+</mo><mi id="S2.I1.i2.p1.4.m4.2.3.3.3" xref="S2.I1.i2.p1.4.m4.2.3.3.3.cmml">ϵ</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.I1.i2.p1.4.m4.2b"><apply id="S2.I1.i2.p1.4.m4.2.3.cmml" xref="S2.I1.i2.p1.4.m4.2.3"><csymbol cd="latexml" id="S2.I1.i2.p1.4.m4.2.3.1.cmml" xref="S2.I1.i2.p1.4.m4.2.3.1">assign</csymbol><apply id="S2.I1.i2.p1.4.m4.2.3.2.cmml" xref="S2.I1.i2.p1.4.m4.2.3.2"><csymbol cd="ambiguous" id="S2.I1.i2.p1.4.m4.2.3.2.1.cmml" xref="S2.I1.i2.p1.4.m4.2.3.2">subscript</csymbol><apply id="S2.I1.i2.p1.4.m4.2.3.2.2.cmml" xref="S2.I1.i2.p1.4.m4.2.3.2"><csymbol cd="ambiguous" id="S2.I1.i2.p1.4.m4.2.3.2.2.1.cmml" xref="S2.I1.i2.p1.4.m4.2.3.2">superscript</csymbol><ci id="S2.I1.i2.p1.4.m4.2.3.2.2.2.cmml" xref="S2.I1.i2.p1.4.m4.2.3.2.2.2">ℎ</ci><cn type="integer" id="S2.I1.i2.p1.4.m4.1.1.1.1.cmml" xref="S2.I1.i2.p1.4.m4.1.1.1.1">0</cn></apply><ci id="S2.I1.i2.p1.4.m4.2.3.2.3.cmml" xref="S2.I1.i2.p1.4.m4.2.3.2.3">𝑖</ci></apply><apply id="S2.I1.i2.p1.4.m4.2.3.3.cmml" xref="S2.I1.i2.p1.4.m4.2.3.3"><plus id="S2.I1.i2.p1.4.m4.2.3.3.1.cmml" xref="S2.I1.i2.p1.4.m4.2.3.3.1"></plus><apply id="S2.I1.i2.p1.4.m4.2.3.3.2.cmml" xref="S2.I1.i2.p1.4.m4.2.3.3.2"><csymbol cd="ambiguous" id="S2.I1.i2.p1.4.m4.2.3.3.2.1.cmml" xref="S2.I1.i2.p1.4.m4.2.3.3.2">subscript</csymbol><apply id="S2.I1.i2.p1.4.m4.2.3.3.2.2.cmml" xref="S2.I1.i2.p1.4.m4.2.3.3.2"><csymbol cd="ambiguous" id="S2.I1.i2.p1.4.m4.2.3.3.2.2.1.cmml" xref="S2.I1.i2.p1.4.m4.2.3.3.2">superscript</csymbol><ci id="S2.I1.i2.p1.4.m4.2.3.3.2.2.2.cmml" xref="S2.I1.i2.p1.4.m4.2.3.3.2.2.2">ℎ</ci><cn type="integer" id="S2.I1.i2.p1.4.m4.2.2.1.1.cmml" xref="S2.I1.i2.p1.4.m4.2.2.1.1">0</cn></apply><ci id="S2.I1.i2.p1.4.m4.2.3.3.2.3.cmml" xref="S2.I1.i2.p1.4.m4.2.3.3.2.3">𝑖</ci></apply><ci id="S2.I1.i2.p1.4.m4.2.3.3.3.cmml" xref="S2.I1.i2.p1.4.m4.2.3.3.3">italic-ϵ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i2.p1.4.m4.2c">\smash{h^{(0)}_{i}:=h^{(0)}_{i}}+\epsilon</annotation></semantics></math> for all indices <math id="S2.I1.i2.p1.5.m5.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S2.I1.i2.p1.5.m5.1a"><mi id="S2.I1.i2.p1.5.m5.1.1" xref="S2.I1.i2.p1.5.m5.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S2.I1.i2.p1.5.m5.1b"><ci id="S2.I1.i2.p1.5.m5.1.1.cmml" xref="S2.I1.i2.p1.5.m5.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i2.p1.5.m5.1c">i</annotation></semantics></math> that correspond to the subject entity, where <math id="S2.I1.i2.p1.6.m6.2" class="ltx_Math" alttext="\epsilon\sim\mathcal{N}(0;\nu)" display="inline"><semantics id="S2.I1.i2.p1.6.m6.2a"><mrow id="S2.I1.i2.p1.6.m6.2.3" xref="S2.I1.i2.p1.6.m6.2.3.cmml"><mi id="S2.I1.i2.p1.6.m6.2.3.2" xref="S2.I1.i2.p1.6.m6.2.3.2.cmml">ϵ</mi><mo id="S2.I1.i2.p1.6.m6.2.3.1" xref="S2.I1.i2.p1.6.m6.2.3.1.cmml">∼</mo><mrow id="S2.I1.i2.p1.6.m6.2.3.3" xref="S2.I1.i2.p1.6.m6.2.3.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S2.I1.i2.p1.6.m6.2.3.3.2" xref="S2.I1.i2.p1.6.m6.2.3.3.2.cmml">𝒩</mi><mo lspace="0em" rspace="0em" id="S2.I1.i2.p1.6.m6.2.3.3.1" xref="S2.I1.i2.p1.6.m6.2.3.3.1.cmml">​</mo><mrow id="S2.I1.i2.p1.6.m6.2.3.3.3.2" xref="S2.I1.i2.p1.6.m6.2.3.3.3.1.cmml"><mo stretchy="false" id="S2.I1.i2.p1.6.m6.2.3.3.3.2.1" xref="S2.I1.i2.p1.6.m6.2.3.3.3.1.cmml">(</mo><mn id="S2.I1.i2.p1.6.m6.1.1" xref="S2.I1.i2.p1.6.m6.1.1.cmml">0</mn><mo id="S2.I1.i2.p1.6.m6.2.3.3.3.2.2" xref="S2.I1.i2.p1.6.m6.2.3.3.3.1.cmml">;</mo><mi id="S2.I1.i2.p1.6.m6.2.2" xref="S2.I1.i2.p1.6.m6.2.2.cmml">ν</mi><mo stretchy="false" id="S2.I1.i2.p1.6.m6.2.3.3.3.2.3" xref="S2.I1.i2.p1.6.m6.2.3.3.3.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.I1.i2.p1.6.m6.2b"><apply id="S2.I1.i2.p1.6.m6.2.3.cmml" xref="S2.I1.i2.p1.6.m6.2.3"><csymbol cd="latexml" id="S2.I1.i2.p1.6.m6.2.3.1.cmml" xref="S2.I1.i2.p1.6.m6.2.3.1">similar-to</csymbol><ci id="S2.I1.i2.p1.6.m6.2.3.2.cmml" xref="S2.I1.i2.p1.6.m6.2.3.2">italic-ϵ</ci><apply id="S2.I1.i2.p1.6.m6.2.3.3.cmml" xref="S2.I1.i2.p1.6.m6.2.3.3"><times id="S2.I1.i2.p1.6.m6.2.3.3.1.cmml" xref="S2.I1.i2.p1.6.m6.2.3.3.1"></times><ci id="S2.I1.i2.p1.6.m6.2.3.3.2.cmml" xref="S2.I1.i2.p1.6.m6.2.3.3.2">𝒩</ci><list id="S2.I1.i2.p1.6.m6.2.3.3.3.1.cmml" xref="S2.I1.i2.p1.6.m6.2.3.3.3.2"><cn type="integer" id="S2.I1.i2.p1.6.m6.1.1.cmml" xref="S2.I1.i2.p1.6.m6.1.1">0</cn><ci id="S2.I1.i2.p1.6.m6.2.2.cmml" xref="S2.I1.i2.p1.6.m6.2.2">𝜈</ci></list></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i2.p1.6.m6.2c">\epsilon\sim\mathcal{N}(0;\nu)</annotation></semantics></math><span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span>We select <math id="footnote2.m1.1" class="ltx_Math" alttext="\nu" display="inline"><semantics id="footnote2.m1.1b"><mi id="footnote2.m1.1.1" xref="footnote2.m1.1.1.cmml">ν</mi><annotation-xml encoding="MathML-Content" id="footnote2.m1.1c"><ci id="footnote2.m1.1.1.cmml" xref="footnote2.m1.1.1">𝜈</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote2.m1.1d">\nu</annotation></semantics></math> to be <math id="footnote2.m2.1" class="ltx_Math" alttext="3" display="inline"><semantics id="footnote2.m2.1b"><mn id="footnote2.m2.1.1" xref="footnote2.m2.1.1.cmml">3</mn><annotation-xml encoding="MathML-Content" id="footnote2.m2.1c"><cn type="integer" id="footnote2.m2.1.1.cmml" xref="footnote2.m2.1.1">3</cn></annotation-xml><annotation encoding="application/x-tex" id="footnote2.m2.1d">3</annotation></semantics></math> times larger than the empirical standard deviation of embeddings; see Appendix <a href="#A2.SS1" title="B.1 Experimental Settings ‣ Appendix B Causal Tracing ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B.1</span></a> for details, and see Appendix <a href="#A2.SS4" title="B.4 Causal Tracing Examples and Further Insights ‣ Appendix B Causal Tracing ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B.4</span></a> for an analysis of other corruption rules.</span></span></span>; . <math id="S2.I1.i2.p1.7.m7.1" class="ltx_Math" alttext="G" display="inline"><semantics id="S2.I1.i2.p1.7.m7.1a"><mi id="S2.I1.i2.p1.7.m7.1.1" xref="S2.I1.i2.p1.7.m7.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="S2.I1.i2.p1.7.m7.1b"><ci id="S2.I1.i2.p1.7.m7.1.1.cmml" xref="S2.I1.i2.p1.7.m7.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i2.p1.7.m7.1c">G</annotation></semantics></math> is then allowed to continue normally, giving us a set of corrupted activations <math id="S2.I1.i2.p1.8.m8.9" class="ltx_Math" alttext="\left\{\smash{h^{(l)}_{i*}}\mid i\in[1,T],l\in[1,L]\right\}" display="inline"><semantics id="S2.I1.i2.p1.8.m8.9a"><mrow id="S2.I1.i2.p1.8.m8.9.9.2" xref="S2.I1.i2.p1.8.m8.9.9.3.cmml"><mo id="S2.I1.i2.p1.8.m8.9.9.2.3" xref="S2.I1.i2.p1.8.m8.9.9.3.1.cmml">{</mo><msubsup id="S2.I1.i2.p1.8.m8.8.8.1.1" xref="S2.I1.i2.p1.8.m8.8.8.1.1.cmml"><mi id="S2.I1.i2.p1.8.m8.8.8.1.1.2.2" xref="S2.I1.i2.p1.8.m8.8.8.1.1.2.2.cmml">h</mi><mrow id="S2.I1.i2.p1.8.m8.3.3.2.4" xref="S2.I1.i2.p1.8.m8.3.3.2.3.cmml"><mi id="S2.I1.i2.p1.8.m8.2.2.1.1" xref="S2.I1.i2.p1.8.m8.2.2.1.1.cmml">i</mi><mo lspace="0.222em" id="S2.I1.i2.p1.8.m8.3.3.2.4.1" xref="S2.I1.i2.p1.8.m8.3.3.2.3.cmml">⁣</mo><mo id="S2.I1.i2.p1.8.m8.3.3.2.2" xref="S2.I1.i2.p1.8.m8.3.3.2.2.cmml">∗</mo></mrow><mrow id="S2.I1.i2.p1.8.m8.1.1.1.3" xref="S2.I1.i2.p1.8.m8.8.8.1.1.cmml"><mo stretchy="false" id="S2.I1.i2.p1.8.m8.1.1.1.3.1" xref="S2.I1.i2.p1.8.m8.8.8.1.1.cmml">(</mo><mi id="S2.I1.i2.p1.8.m8.1.1.1.1" xref="S2.I1.i2.p1.8.m8.1.1.1.1.cmml">l</mi><mo stretchy="false" id="S2.I1.i2.p1.8.m8.1.1.1.3.2" xref="S2.I1.i2.p1.8.m8.8.8.1.1.cmml">)</mo></mrow></msubsup><mo fence="true" lspace="0em" rspace="0em" id="S2.I1.i2.p1.8.m8.9.9.2.4" xref="S2.I1.i2.p1.8.m8.9.9.3.1.cmml">∣</mo><mrow id="S2.I1.i2.p1.8.m8.9.9.2.2.2" xref="S2.I1.i2.p1.8.m8.9.9.2.2.3.cmml"><mrow id="S2.I1.i2.p1.8.m8.9.9.2.2.1.1" xref="S2.I1.i2.p1.8.m8.9.9.2.2.1.1.cmml"><mi id="S2.I1.i2.p1.8.m8.9.9.2.2.1.1.2" xref="S2.I1.i2.p1.8.m8.9.9.2.2.1.1.2.cmml">i</mi><mo id="S2.I1.i2.p1.8.m8.9.9.2.2.1.1.1" xref="S2.I1.i2.p1.8.m8.9.9.2.2.1.1.1.cmml">∈</mo><mrow id="S2.I1.i2.p1.8.m8.9.9.2.2.1.1.3.2" xref="S2.I1.i2.p1.8.m8.9.9.2.2.1.1.3.1.cmml"><mo stretchy="false" id="S2.I1.i2.p1.8.m8.9.9.2.2.1.1.3.2.1" xref="S2.I1.i2.p1.8.m8.9.9.2.2.1.1.3.1.cmml">[</mo><mn id="S2.I1.i2.p1.8.m8.4.4" xref="S2.I1.i2.p1.8.m8.4.4.cmml">1</mn><mo id="S2.I1.i2.p1.8.m8.9.9.2.2.1.1.3.2.2" xref="S2.I1.i2.p1.8.m8.9.9.2.2.1.1.3.1.cmml">,</mo><mi id="S2.I1.i2.p1.8.m8.5.5" xref="S2.I1.i2.p1.8.m8.5.5.cmml">T</mi><mo stretchy="false" id="S2.I1.i2.p1.8.m8.9.9.2.2.1.1.3.2.3" xref="S2.I1.i2.p1.8.m8.9.9.2.2.1.1.3.1.cmml">]</mo></mrow></mrow><mo id="S2.I1.i2.p1.8.m8.9.9.2.2.2.3" xref="S2.I1.i2.p1.8.m8.9.9.2.2.3a.cmml">,</mo><mrow id="S2.I1.i2.p1.8.m8.9.9.2.2.2.2" xref="S2.I1.i2.p1.8.m8.9.9.2.2.2.2.cmml"><mi id="S2.I1.i2.p1.8.m8.9.9.2.2.2.2.2" xref="S2.I1.i2.p1.8.m8.9.9.2.2.2.2.2.cmml">l</mi><mo id="S2.I1.i2.p1.8.m8.9.9.2.2.2.2.1" xref="S2.I1.i2.p1.8.m8.9.9.2.2.2.2.1.cmml">∈</mo><mrow id="S2.I1.i2.p1.8.m8.9.9.2.2.2.2.3.2" xref="S2.I1.i2.p1.8.m8.9.9.2.2.2.2.3.1.cmml"><mo stretchy="false" id="S2.I1.i2.p1.8.m8.9.9.2.2.2.2.3.2.1" xref="S2.I1.i2.p1.8.m8.9.9.2.2.2.2.3.1.cmml">[</mo><mn id="S2.I1.i2.p1.8.m8.6.6" xref="S2.I1.i2.p1.8.m8.6.6.cmml">1</mn><mo id="S2.I1.i2.p1.8.m8.9.9.2.2.2.2.3.2.2" xref="S2.I1.i2.p1.8.m8.9.9.2.2.2.2.3.1.cmml">,</mo><mi id="S2.I1.i2.p1.8.m8.7.7" xref="S2.I1.i2.p1.8.m8.7.7.cmml">L</mi><mo stretchy="false" id="S2.I1.i2.p1.8.m8.9.9.2.2.2.2.3.2.3" xref="S2.I1.i2.p1.8.m8.9.9.2.2.2.2.3.1.cmml">]</mo></mrow></mrow></mrow><mo id="S2.I1.i2.p1.8.m8.9.9.2.5" xref="S2.I1.i2.p1.8.m8.9.9.3.1.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.I1.i2.p1.8.m8.9b"><apply id="S2.I1.i2.p1.8.m8.9.9.3.cmml" xref="S2.I1.i2.p1.8.m8.9.9.2"><csymbol cd="latexml" id="S2.I1.i2.p1.8.m8.9.9.3.1.cmml" xref="S2.I1.i2.p1.8.m8.9.9.2.3">conditional-set</csymbol><apply id="S2.I1.i2.p1.8.m8.8.8.1.1.cmml" xref="S2.I1.i2.p1.8.m8.8.8.1.1"><csymbol cd="ambiguous" id="S2.I1.i2.p1.8.m8.8.8.1.1.1.cmml" xref="S2.I1.i2.p1.8.m8.8.8.1.1">subscript</csymbol><apply id="S2.I1.i2.p1.8.m8.8.8.1.1.2.cmml" xref="S2.I1.i2.p1.8.m8.8.8.1.1"><csymbol cd="ambiguous" id="S2.I1.i2.p1.8.m8.8.8.1.1.2.1.cmml" xref="S2.I1.i2.p1.8.m8.8.8.1.1">superscript</csymbol><ci id="S2.I1.i2.p1.8.m8.8.8.1.1.2.2.cmml" xref="S2.I1.i2.p1.8.m8.8.8.1.1.2.2">ℎ</ci><ci id="S2.I1.i2.p1.8.m8.1.1.1.1.cmml" xref="S2.I1.i2.p1.8.m8.1.1.1.1">𝑙</ci></apply><list id="S2.I1.i2.p1.8.m8.3.3.2.3.cmml" xref="S2.I1.i2.p1.8.m8.3.3.2.4"><ci id="S2.I1.i2.p1.8.m8.2.2.1.1.cmml" xref="S2.I1.i2.p1.8.m8.2.2.1.1">𝑖</ci><times id="S2.I1.i2.p1.8.m8.3.3.2.2.cmml" xref="S2.I1.i2.p1.8.m8.3.3.2.2"></times></list></apply><apply id="S2.I1.i2.p1.8.m8.9.9.2.2.3.cmml" xref="S2.I1.i2.p1.8.m8.9.9.2.2.2"><csymbol cd="ambiguous" id="S2.I1.i2.p1.8.m8.9.9.2.2.3a.cmml" xref="S2.I1.i2.p1.8.m8.9.9.2.2.2.3">formulae-sequence</csymbol><apply id="S2.I1.i2.p1.8.m8.9.9.2.2.1.1.cmml" xref="S2.I1.i2.p1.8.m8.9.9.2.2.1.1"><in id="S2.I1.i2.p1.8.m8.9.9.2.2.1.1.1.cmml" xref="S2.I1.i2.p1.8.m8.9.9.2.2.1.1.1"></in><ci id="S2.I1.i2.p1.8.m8.9.9.2.2.1.1.2.cmml" xref="S2.I1.i2.p1.8.m8.9.9.2.2.1.1.2">𝑖</ci><interval closure="closed" id="S2.I1.i2.p1.8.m8.9.9.2.2.1.1.3.1.cmml" xref="S2.I1.i2.p1.8.m8.9.9.2.2.1.1.3.2"><cn type="integer" id="S2.I1.i2.p1.8.m8.4.4.cmml" xref="S2.I1.i2.p1.8.m8.4.4">1</cn><ci id="S2.I1.i2.p1.8.m8.5.5.cmml" xref="S2.I1.i2.p1.8.m8.5.5">𝑇</ci></interval></apply><apply id="S2.I1.i2.p1.8.m8.9.9.2.2.2.2.cmml" xref="S2.I1.i2.p1.8.m8.9.9.2.2.2.2"><in id="S2.I1.i2.p1.8.m8.9.9.2.2.2.2.1.cmml" xref="S2.I1.i2.p1.8.m8.9.9.2.2.2.2.1"></in><ci id="S2.I1.i2.p1.8.m8.9.9.2.2.2.2.2.cmml" xref="S2.I1.i2.p1.8.m8.9.9.2.2.2.2.2">𝑙</ci><interval closure="closed" id="S2.I1.i2.p1.8.m8.9.9.2.2.2.2.3.1.cmml" xref="S2.I1.i2.p1.8.m8.9.9.2.2.2.2.3.2"><cn type="integer" id="S2.I1.i2.p1.8.m8.6.6.cmml" xref="S2.I1.i2.p1.8.m8.6.6">1</cn><ci id="S2.I1.i2.p1.8.m8.7.7.cmml" xref="S2.I1.i2.p1.8.m8.7.7">𝐿</ci></interval></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i2.p1.8.m8.9c">\left\{\smash{h^{(l)}_{i*}}\mid i\in[1,T],l\in[1,L]\right\}</annotation></semantics></math>. Because <math id="S2.I1.i2.p1.9.m9.1" class="ltx_Math" alttext="G" display="inline"><semantics id="S2.I1.i2.p1.9.m9.1a"><mi id="S2.I1.i2.p1.9.m9.1.1" xref="S2.I1.i2.p1.9.m9.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="S2.I1.i2.p1.9.m9.1b"><ci id="S2.I1.i2.p1.9.m9.1.1.cmml" xref="S2.I1.i2.p1.9.m9.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i2.p1.9.m9.1c">G</annotation></semantics></math> loses some information about the subject, it will likely return an incorrect answer (Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>b).</p>
</div>
</li>
<li id="S2.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I1.i3.p1" class="ltx_para">
<p id="S2.I1.i3.p1.5" class="ltx_p">The <span id="S2.I1.i3.p1.5.1" class="ltx_text ltx_font_bold">corrupted-with-restoration run</span>, lets <math id="S2.I1.i3.p1.1.m1.1" class="ltx_Math" alttext="G" display="inline"><semantics id="S2.I1.i3.p1.1.m1.1a"><mi id="S2.I1.i3.p1.1.m1.1.1" xref="S2.I1.i3.p1.1.m1.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="S2.I1.i3.p1.1.m1.1b"><ci id="S2.I1.i3.p1.1.m1.1.1.cmml" xref="S2.I1.i3.p1.1.m1.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i3.p1.1.m1.1c">G</annotation></semantics></math> run computations on the noisy embeddings as in the corrupted baseline, <span id="S2.I1.i3.p1.5.2" class="ltx_text ltx_font_italic">except</span> at some token <math id="S2.I1.i3.p1.2.m2.1" class="ltx_Math" alttext="\smash{\hat{i}}" display="inline"><semantics id="S2.I1.i3.p1.2.m2.1a"><mover accent="true" id="S2.I1.i3.p1.2.m2.1.1" xref="S2.I1.i3.p1.2.m2.1.1.cmml"><mi id="S2.I1.i3.p1.2.m2.1.1.2" xref="S2.I1.i3.p1.2.m2.1.1.2.cmml">i</mi><mo id="S2.I1.i3.p1.2.m2.1.1.1" xref="S2.I1.i3.p1.2.m2.1.1.1.cmml">^</mo></mover><annotation-xml encoding="MathML-Content" id="S2.I1.i3.p1.2.m2.1b"><apply id="S2.I1.i3.p1.2.m2.1.1.cmml" xref="S2.I1.i3.p1.2.m2.1.1"><ci id="S2.I1.i3.p1.2.m2.1.1.1.cmml" xref="S2.I1.i3.p1.2.m2.1.1.1">^</ci><ci id="S2.I1.i3.p1.2.m2.1.1.2.cmml" xref="S2.I1.i3.p1.2.m2.1.1.2">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i3.p1.2.m2.1c">\smash{\hat{i}}</annotation></semantics></math> and layer <math id="S2.I1.i3.p1.3.m3.1" class="ltx_Math" alttext="\smash{\hat{l}}" display="inline"><semantics id="S2.I1.i3.p1.3.m3.1a"><mover accent="true" id="S2.I1.i3.p1.3.m3.1.1" xref="S2.I1.i3.p1.3.m3.1.1.cmml"><mi id="S2.I1.i3.p1.3.m3.1.1.2" xref="S2.I1.i3.p1.3.m3.1.1.2.cmml">l</mi><mo id="S2.I1.i3.p1.3.m3.1.1.1" xref="S2.I1.i3.p1.3.m3.1.1.1.cmml">^</mo></mover><annotation-xml encoding="MathML-Content" id="S2.I1.i3.p1.3.m3.1b"><apply id="S2.I1.i3.p1.3.m3.1.1.cmml" xref="S2.I1.i3.p1.3.m3.1.1"><ci id="S2.I1.i3.p1.3.m3.1.1.1.cmml" xref="S2.I1.i3.p1.3.m3.1.1.1">^</ci><ci id="S2.I1.i3.p1.3.m3.1.1.2.cmml" xref="S2.I1.i3.p1.3.m3.1.1.2">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i3.p1.3.m3.1c">\smash{\hat{l}}</annotation></semantics></math>. There, we hook <math id="S2.I1.i3.p1.4.m4.1" class="ltx_Math" alttext="G" display="inline"><semantics id="S2.I1.i3.p1.4.m4.1a"><mi id="S2.I1.i3.p1.4.m4.1.1" xref="S2.I1.i3.p1.4.m4.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="S2.I1.i3.p1.4.m4.1b"><ci id="S2.I1.i3.p1.4.m4.1.1.cmml" xref="S2.I1.i3.p1.4.m4.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i3.p1.4.m4.1c">G</annotation></semantics></math> so that it is forced to output the clean state <math id="S2.I1.i3.p1.5.m5.1" class="ltx_Math" alttext="\smash{h^{(\smash{\hat{l}})}_{\hat{i}}}" display="inline"><semantics id="S2.I1.i3.p1.5.m5.1a"><msubsup id="S2.I1.i3.p1.5.m5.1.2" xref="S2.I1.i3.p1.5.m5.1.2.cmml"><mi id="S2.I1.i3.p1.5.m5.1.2.2.2" xref="S2.I1.i3.p1.5.m5.1.2.2.2.cmml">h</mi><mover accent="true" id="S2.I1.i3.p1.5.m5.1.2.3" xref="S2.I1.i3.p1.5.m5.1.2.3.cmml"><mi id="S2.I1.i3.p1.5.m5.1.2.3.2" xref="S2.I1.i3.p1.5.m5.1.2.3.2.cmml">i</mi><mo id="S2.I1.i3.p1.5.m5.1.2.3.1" xref="S2.I1.i3.p1.5.m5.1.2.3.1.cmml">^</mo></mover><mrow id="S2.I1.i3.p1.5.m5.1.1.1.3" xref="S2.I1.i3.p1.5.m5.1.1.1.1.cmml"><mo stretchy="false" id="S2.I1.i3.p1.5.m5.1.1.1.3.1" xref="S2.I1.i3.p1.5.m5.1.1.1.1.cmml">(</mo><mover accent="true" id="S2.I1.i3.p1.5.m5.1.1.1.1" xref="S2.I1.i3.p1.5.m5.1.1.1.1.cmml"><mi id="S2.I1.i3.p1.5.m5.1.1.1.1.2" xref="S2.I1.i3.p1.5.m5.1.1.1.1.2.cmml">l</mi><mo id="S2.I1.i3.p1.5.m5.1.1.1.1.1" xref="S2.I1.i3.p1.5.m5.1.1.1.1.1.cmml">^</mo></mover><mo stretchy="false" id="S2.I1.i3.p1.5.m5.1.1.1.3.2" xref="S2.I1.i3.p1.5.m5.1.1.1.1.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S2.I1.i3.p1.5.m5.1b"><apply id="S2.I1.i3.p1.5.m5.1.2.cmml" xref="S2.I1.i3.p1.5.m5.1.2"><csymbol cd="ambiguous" id="S2.I1.i3.p1.5.m5.1.2.1.cmml" xref="S2.I1.i3.p1.5.m5.1.2">subscript</csymbol><apply id="S2.I1.i3.p1.5.m5.1.2.2.cmml" xref="S2.I1.i3.p1.5.m5.1.2"><csymbol cd="ambiguous" id="S2.I1.i3.p1.5.m5.1.2.2.1.cmml" xref="S2.I1.i3.p1.5.m5.1.2">superscript</csymbol><ci id="S2.I1.i3.p1.5.m5.1.2.2.2.cmml" xref="S2.I1.i3.p1.5.m5.1.2.2.2">ℎ</ci><apply id="S2.I1.i3.p1.5.m5.1.1.1.1.cmml" xref="S2.I1.i3.p1.5.m5.1.1.1.3"><ci id="S2.I1.i3.p1.5.m5.1.1.1.1.1.cmml" xref="S2.I1.i3.p1.5.m5.1.1.1.1.1">^</ci><ci id="S2.I1.i3.p1.5.m5.1.1.1.1.2.cmml" xref="S2.I1.i3.p1.5.m5.1.1.1.1.2">𝑙</ci></apply></apply><apply id="S2.I1.i3.p1.5.m5.1.2.3.cmml" xref="S2.I1.i3.p1.5.m5.1.2.3"><ci id="S2.I1.i3.p1.5.m5.1.2.3.1.cmml" xref="S2.I1.i3.p1.5.m5.1.2.3.1">^</ci><ci id="S2.I1.i3.p1.5.m5.1.2.3.2.cmml" xref="S2.I1.i3.p1.5.m5.1.2.3.2">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i3.p1.5.m5.1c">\smash{h^{(\smash{\hat{l}})}_{\hat{i}}}</annotation></semantics></math>; future computations execute without further intervention. Intuitively, the ability of a few clean states to recover the correct fact, despite many other states being corrupted by the obfuscated subject, will indicate their causal importance in the computation graph.</p>
</div>
</li>
</ul>
</div>
<div id="S2.SS1.p4" class="ltx_para">
<p id="S2.SS1.p4.9" class="ltx_p">Let <math id="S2.SS1.p4.1.m1.1" class="ltx_Math" alttext="\mathbb{P}[o]" display="inline"><semantics id="S2.SS1.p4.1.m1.1a"><mrow id="S2.SS1.p4.1.m1.1.2" xref="S2.SS1.p4.1.m1.1.2.cmml"><mi id="S2.SS1.p4.1.m1.1.2.2" xref="S2.SS1.p4.1.m1.1.2.2.cmml">ℙ</mi><mo lspace="0em" rspace="0em" id="S2.SS1.p4.1.m1.1.2.1" xref="S2.SS1.p4.1.m1.1.2.1.cmml">​</mo><mrow id="S2.SS1.p4.1.m1.1.2.3.2" xref="S2.SS1.p4.1.m1.1.2.3.1.cmml"><mo stretchy="false" id="S2.SS1.p4.1.m1.1.2.3.2.1" xref="S2.SS1.p4.1.m1.1.2.3.1.1.cmml">[</mo><mi id="S2.SS1.p4.1.m1.1.1" xref="S2.SS1.p4.1.m1.1.1.cmml">o</mi><mo stretchy="false" id="S2.SS1.p4.1.m1.1.2.3.2.2" xref="S2.SS1.p4.1.m1.1.2.3.1.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.1.m1.1b"><apply id="S2.SS1.p4.1.m1.1.2.cmml" xref="S2.SS1.p4.1.m1.1.2"><times id="S2.SS1.p4.1.m1.1.2.1.cmml" xref="S2.SS1.p4.1.m1.1.2.1"></times><ci id="S2.SS1.p4.1.m1.1.2.2.cmml" xref="S2.SS1.p4.1.m1.1.2.2">ℙ</ci><apply id="S2.SS1.p4.1.m1.1.2.3.1.cmml" xref="S2.SS1.p4.1.m1.1.2.3.2"><csymbol cd="latexml" id="S2.SS1.p4.1.m1.1.2.3.1.1.cmml" xref="S2.SS1.p4.1.m1.1.2.3.2.1">delimited-[]</csymbol><ci id="S2.SS1.p4.1.m1.1.1.cmml" xref="S2.SS1.p4.1.m1.1.1">𝑜</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.1.m1.1c">\mathbb{P}[o]</annotation></semantics></math>, <math id="S2.SS1.p4.2.m2.1" class="ltx_Math" alttext="\mathbb{P}_{*}[o]" display="inline"><semantics id="S2.SS1.p4.2.m2.1a"><mrow id="S2.SS1.p4.2.m2.1.2" xref="S2.SS1.p4.2.m2.1.2.cmml"><msub id="S2.SS1.p4.2.m2.1.2.2" xref="S2.SS1.p4.2.m2.1.2.2.cmml"><mi id="S2.SS1.p4.2.m2.1.2.2.2" xref="S2.SS1.p4.2.m2.1.2.2.2.cmml">ℙ</mi><mo id="S2.SS1.p4.2.m2.1.2.2.3" xref="S2.SS1.p4.2.m2.1.2.2.3.cmml">∗</mo></msub><mo lspace="0em" rspace="0em" id="S2.SS1.p4.2.m2.1.2.1" xref="S2.SS1.p4.2.m2.1.2.1.cmml">​</mo><mrow id="S2.SS1.p4.2.m2.1.2.3.2" xref="S2.SS1.p4.2.m2.1.2.3.1.cmml"><mo stretchy="false" id="S2.SS1.p4.2.m2.1.2.3.2.1" xref="S2.SS1.p4.2.m2.1.2.3.1.1.cmml">[</mo><mi id="S2.SS1.p4.2.m2.1.1" xref="S2.SS1.p4.2.m2.1.1.cmml">o</mi><mo stretchy="false" id="S2.SS1.p4.2.m2.1.2.3.2.2" xref="S2.SS1.p4.2.m2.1.2.3.1.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.2.m2.1b"><apply id="S2.SS1.p4.2.m2.1.2.cmml" xref="S2.SS1.p4.2.m2.1.2"><times id="S2.SS1.p4.2.m2.1.2.1.cmml" xref="S2.SS1.p4.2.m2.1.2.1"></times><apply id="S2.SS1.p4.2.m2.1.2.2.cmml" xref="S2.SS1.p4.2.m2.1.2.2"><csymbol cd="ambiguous" id="S2.SS1.p4.2.m2.1.2.2.1.cmml" xref="S2.SS1.p4.2.m2.1.2.2">subscript</csymbol><ci id="S2.SS1.p4.2.m2.1.2.2.2.cmml" xref="S2.SS1.p4.2.m2.1.2.2.2">ℙ</ci><times id="S2.SS1.p4.2.m2.1.2.2.3.cmml" xref="S2.SS1.p4.2.m2.1.2.2.3"></times></apply><apply id="S2.SS1.p4.2.m2.1.2.3.1.cmml" xref="S2.SS1.p4.2.m2.1.2.3.2"><csymbol cd="latexml" id="S2.SS1.p4.2.m2.1.2.3.1.1.cmml" xref="S2.SS1.p4.2.m2.1.2.3.2.1">delimited-[]</csymbol><ci id="S2.SS1.p4.2.m2.1.1.cmml" xref="S2.SS1.p4.2.m2.1.1">𝑜</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.2.m2.1c">\mathbb{P}_{*}[o]</annotation></semantics></math>, and <math id="S2.SS1.p4.3.m3.4" class="ltx_Math" alttext="\mathbb{P}_{*,\smash{\text{ clean }h_{i}^{(l)}}}[o]" display="inline"><semantics id="S2.SS1.p4.3.m3.4a"><mrow id="S2.SS1.p4.3.m3.4.5" xref="S2.SS1.p4.3.m3.4.5.cmml"><msub id="S2.SS1.p4.3.m3.4.5.2" xref="S2.SS1.p4.3.m3.4.5.2.cmml"><mi id="S2.SS1.p4.3.m3.4.5.2.2" xref="S2.SS1.p4.3.m3.4.5.2.2.cmml">ℙ</mi><mrow id="S2.SS1.p4.3.m3.3.3.3.3" xref="S2.SS1.p4.3.m3.3.3.3.4.cmml"><mo rspace="0em" id="S2.SS1.p4.3.m3.2.2.2.2" xref="S2.SS1.p4.3.m3.2.2.2.2.cmml">∗</mo><mo id="S2.SS1.p4.3.m3.3.3.3.3.2" xref="S2.SS1.p4.3.m3.3.3.3.4.cmml">,</mo><mrow id="S2.SS1.p4.3.m3.3.3.3.3.1" xref="S2.SS1.p4.3.m3.3.3.3.3.1.cmml"><mtext id="S2.SS1.p4.3.m3.3.3.3.3.1.2" xref="S2.SS1.p4.3.m3.3.3.3.3.1.2a.cmml"> clean </mtext><mo lspace="0em" rspace="0em" id="S2.SS1.p4.3.m3.3.3.3.3.1.1" xref="S2.SS1.p4.3.m3.3.3.3.3.1.1.cmml">​</mo><msubsup id="S2.SS1.p4.3.m3.3.3.3.3.1.3" xref="S2.SS1.p4.3.m3.3.3.3.3.1.3.cmml"><mi id="S2.SS1.p4.3.m3.3.3.3.3.1.3.2.2" xref="S2.SS1.p4.3.m3.3.3.3.3.1.3.2.2.cmml">h</mi><mi id="S2.SS1.p4.3.m3.3.3.3.3.1.3.2.3" xref="S2.SS1.p4.3.m3.3.3.3.3.1.3.2.3.cmml">i</mi><mrow id="S2.SS1.p4.3.m3.1.1.1.1.1.3" xref="S2.SS1.p4.3.m3.3.3.3.3.1.3.cmml"><mo stretchy="false" id="S2.SS1.p4.3.m3.1.1.1.1.1.3.1" xref="S2.SS1.p4.3.m3.3.3.3.3.1.3.cmml">(</mo><mi id="S2.SS1.p4.3.m3.1.1.1.1.1.1" xref="S2.SS1.p4.3.m3.1.1.1.1.1.1.cmml">l</mi><mo stretchy="false" id="S2.SS1.p4.3.m3.1.1.1.1.1.3.2" xref="S2.SS1.p4.3.m3.3.3.3.3.1.3.cmml">)</mo></mrow></msubsup></mrow></mrow></msub><mo lspace="0em" rspace="0em" id="S2.SS1.p4.3.m3.4.5.1" xref="S2.SS1.p4.3.m3.4.5.1.cmml">​</mo><mrow id="S2.SS1.p4.3.m3.4.5.3.2" xref="S2.SS1.p4.3.m3.4.5.3.1.cmml"><mo stretchy="false" id="S2.SS1.p4.3.m3.4.5.3.2.1" xref="S2.SS1.p4.3.m3.4.5.3.1.1.cmml">[</mo><mi id="S2.SS1.p4.3.m3.4.4" xref="S2.SS1.p4.3.m3.4.4.cmml">o</mi><mo stretchy="false" id="S2.SS1.p4.3.m3.4.5.3.2.2" xref="S2.SS1.p4.3.m3.4.5.3.1.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.3.m3.4b"><apply id="S2.SS1.p4.3.m3.4.5.cmml" xref="S2.SS1.p4.3.m3.4.5"><times id="S2.SS1.p4.3.m3.4.5.1.cmml" xref="S2.SS1.p4.3.m3.4.5.1"></times><apply id="S2.SS1.p4.3.m3.4.5.2.cmml" xref="S2.SS1.p4.3.m3.4.5.2"><csymbol cd="ambiguous" id="S2.SS1.p4.3.m3.4.5.2.1.cmml" xref="S2.SS1.p4.3.m3.4.5.2">subscript</csymbol><ci id="S2.SS1.p4.3.m3.4.5.2.2.cmml" xref="S2.SS1.p4.3.m3.4.5.2.2">ℙ</ci><list id="S2.SS1.p4.3.m3.3.3.3.4.cmml" xref="S2.SS1.p4.3.m3.3.3.3.3"><times id="S2.SS1.p4.3.m3.2.2.2.2.cmml" xref="S2.SS1.p4.3.m3.2.2.2.2"></times><apply id="S2.SS1.p4.3.m3.3.3.3.3.1.cmml" xref="S2.SS1.p4.3.m3.3.3.3.3.1"><times id="S2.SS1.p4.3.m3.3.3.3.3.1.1.cmml" xref="S2.SS1.p4.3.m3.3.3.3.3.1.1"></times><ci id="S2.SS1.p4.3.m3.3.3.3.3.1.2a.cmml" xref="S2.SS1.p4.3.m3.3.3.3.3.1.2"><mtext mathsize="70%" id="S2.SS1.p4.3.m3.3.3.3.3.1.2.cmml" xref="S2.SS1.p4.3.m3.3.3.3.3.1.2"> clean </mtext></ci><apply id="S2.SS1.p4.3.m3.3.3.3.3.1.3.cmml" xref="S2.SS1.p4.3.m3.3.3.3.3.1.3"><csymbol cd="ambiguous" id="S2.SS1.p4.3.m3.3.3.3.3.1.3.1.cmml" xref="S2.SS1.p4.3.m3.3.3.3.3.1.3">superscript</csymbol><apply id="S2.SS1.p4.3.m3.3.3.3.3.1.3.2.cmml" xref="S2.SS1.p4.3.m3.3.3.3.3.1.3"><csymbol cd="ambiguous" id="S2.SS1.p4.3.m3.3.3.3.3.1.3.2.1.cmml" xref="S2.SS1.p4.3.m3.3.3.3.3.1.3">subscript</csymbol><ci id="S2.SS1.p4.3.m3.3.3.3.3.1.3.2.2.cmml" xref="S2.SS1.p4.3.m3.3.3.3.3.1.3.2.2">ℎ</ci><ci id="S2.SS1.p4.3.m3.3.3.3.3.1.3.2.3.cmml" xref="S2.SS1.p4.3.m3.3.3.3.3.1.3.2.3">𝑖</ci></apply><ci id="S2.SS1.p4.3.m3.1.1.1.1.1.1.cmml" xref="S2.SS1.p4.3.m3.1.1.1.1.1.1">𝑙</ci></apply></apply></list></apply><apply id="S2.SS1.p4.3.m3.4.5.3.1.cmml" xref="S2.SS1.p4.3.m3.4.5.3.2"><csymbol cd="latexml" id="S2.SS1.p4.3.m3.4.5.3.1.1.cmml" xref="S2.SS1.p4.3.m3.4.5.3.2.1">delimited-[]</csymbol><ci id="S2.SS1.p4.3.m3.4.4.cmml" xref="S2.SS1.p4.3.m3.4.4">𝑜</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.3.m3.4c">\mathbb{P}_{*,\smash{\text{ clean }h_{i}^{(l)}}}[o]</annotation></semantics></math> denote the probability of emitting <math id="S2.SS1.p4.4.m4.1" class="ltx_Math" alttext="o" display="inline"><semantics id="S2.SS1.p4.4.m4.1a"><mi id="S2.SS1.p4.4.m4.1.1" xref="S2.SS1.p4.4.m4.1.1.cmml">o</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.4.m4.1b"><ci id="S2.SS1.p4.4.m4.1.1.cmml" xref="S2.SS1.p4.4.m4.1.1">𝑜</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.4.m4.1c">o</annotation></semantics></math> under the clean, corrupted, and corrupted-with-restoration runs, respectively; dependence on the input <math id="S2.SS1.p4.5.m5.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S2.SS1.p4.5.m5.1a"><mi id="S2.SS1.p4.5.m5.1.1" xref="S2.SS1.p4.5.m5.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.5.m5.1b"><ci id="S2.SS1.p4.5.m5.1.1.cmml" xref="S2.SS1.p4.5.m5.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.5.m5.1c">x</annotation></semantics></math> is omitted for notational simplicity.
The <span id="S2.SS1.p4.9.1" class="ltx_text ltx_font_bold">total effect</span> (TE) is the difference between these quantities: <math id="S2.SS1.p4.6.m6.2" class="ltx_Math" alttext="\text{TE}=\mathbb{P}[o]-\mathbb{P}_{*}[o]" display="inline"><semantics id="S2.SS1.p4.6.m6.2a"><mrow id="S2.SS1.p4.6.m6.2.3" xref="S2.SS1.p4.6.m6.2.3.cmml"><mtext id="S2.SS1.p4.6.m6.2.3.2" xref="S2.SS1.p4.6.m6.2.3.2a.cmml">TE</mtext><mo id="S2.SS1.p4.6.m6.2.3.1" xref="S2.SS1.p4.6.m6.2.3.1.cmml">=</mo><mrow id="S2.SS1.p4.6.m6.2.3.3" xref="S2.SS1.p4.6.m6.2.3.3.cmml"><mrow id="S2.SS1.p4.6.m6.2.3.3.2" xref="S2.SS1.p4.6.m6.2.3.3.2.cmml"><mi id="S2.SS1.p4.6.m6.2.3.3.2.2" xref="S2.SS1.p4.6.m6.2.3.3.2.2.cmml">ℙ</mi><mo lspace="0em" rspace="0em" id="S2.SS1.p4.6.m6.2.3.3.2.1" xref="S2.SS1.p4.6.m6.2.3.3.2.1.cmml">​</mo><mrow id="S2.SS1.p4.6.m6.2.3.3.2.3.2" xref="S2.SS1.p4.6.m6.2.3.3.2.3.1.cmml"><mo stretchy="false" id="S2.SS1.p4.6.m6.2.3.3.2.3.2.1" xref="S2.SS1.p4.6.m6.2.3.3.2.3.1.1.cmml">[</mo><mi id="S2.SS1.p4.6.m6.1.1" xref="S2.SS1.p4.6.m6.1.1.cmml">o</mi><mo stretchy="false" id="S2.SS1.p4.6.m6.2.3.3.2.3.2.2" xref="S2.SS1.p4.6.m6.2.3.3.2.3.1.1.cmml">]</mo></mrow></mrow><mo id="S2.SS1.p4.6.m6.2.3.3.1" xref="S2.SS1.p4.6.m6.2.3.3.1.cmml">−</mo><mrow id="S2.SS1.p4.6.m6.2.3.3.3" xref="S2.SS1.p4.6.m6.2.3.3.3.cmml"><msub id="S2.SS1.p4.6.m6.2.3.3.3.2" xref="S2.SS1.p4.6.m6.2.3.3.3.2.cmml"><mi id="S2.SS1.p4.6.m6.2.3.3.3.2.2" xref="S2.SS1.p4.6.m6.2.3.3.3.2.2.cmml">ℙ</mi><mo id="S2.SS1.p4.6.m6.2.3.3.3.2.3" xref="S2.SS1.p4.6.m6.2.3.3.3.2.3.cmml">∗</mo></msub><mo lspace="0em" rspace="0em" id="S2.SS1.p4.6.m6.2.3.3.3.1" xref="S2.SS1.p4.6.m6.2.3.3.3.1.cmml">​</mo><mrow id="S2.SS1.p4.6.m6.2.3.3.3.3.2" xref="S2.SS1.p4.6.m6.2.3.3.3.3.1.cmml"><mo stretchy="false" id="S2.SS1.p4.6.m6.2.3.3.3.3.2.1" xref="S2.SS1.p4.6.m6.2.3.3.3.3.1.1.cmml">[</mo><mi id="S2.SS1.p4.6.m6.2.2" xref="S2.SS1.p4.6.m6.2.2.cmml">o</mi><mo stretchy="false" id="S2.SS1.p4.6.m6.2.3.3.3.3.2.2" xref="S2.SS1.p4.6.m6.2.3.3.3.3.1.1.cmml">]</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.6.m6.2b"><apply id="S2.SS1.p4.6.m6.2.3.cmml" xref="S2.SS1.p4.6.m6.2.3"><eq id="S2.SS1.p4.6.m6.2.3.1.cmml" xref="S2.SS1.p4.6.m6.2.3.1"></eq><ci id="S2.SS1.p4.6.m6.2.3.2a.cmml" xref="S2.SS1.p4.6.m6.2.3.2"><mtext id="S2.SS1.p4.6.m6.2.3.2.cmml" xref="S2.SS1.p4.6.m6.2.3.2">TE</mtext></ci><apply id="S2.SS1.p4.6.m6.2.3.3.cmml" xref="S2.SS1.p4.6.m6.2.3.3"><minus id="S2.SS1.p4.6.m6.2.3.3.1.cmml" xref="S2.SS1.p4.6.m6.2.3.3.1"></minus><apply id="S2.SS1.p4.6.m6.2.3.3.2.cmml" xref="S2.SS1.p4.6.m6.2.3.3.2"><times id="S2.SS1.p4.6.m6.2.3.3.2.1.cmml" xref="S2.SS1.p4.6.m6.2.3.3.2.1"></times><ci id="S2.SS1.p4.6.m6.2.3.3.2.2.cmml" xref="S2.SS1.p4.6.m6.2.3.3.2.2">ℙ</ci><apply id="S2.SS1.p4.6.m6.2.3.3.2.3.1.cmml" xref="S2.SS1.p4.6.m6.2.3.3.2.3.2"><csymbol cd="latexml" id="S2.SS1.p4.6.m6.2.3.3.2.3.1.1.cmml" xref="S2.SS1.p4.6.m6.2.3.3.2.3.2.1">delimited-[]</csymbol><ci id="S2.SS1.p4.6.m6.1.1.cmml" xref="S2.SS1.p4.6.m6.1.1">𝑜</ci></apply></apply><apply id="S2.SS1.p4.6.m6.2.3.3.3.cmml" xref="S2.SS1.p4.6.m6.2.3.3.3"><times id="S2.SS1.p4.6.m6.2.3.3.3.1.cmml" xref="S2.SS1.p4.6.m6.2.3.3.3.1"></times><apply id="S2.SS1.p4.6.m6.2.3.3.3.2.cmml" xref="S2.SS1.p4.6.m6.2.3.3.3.2"><csymbol cd="ambiguous" id="S2.SS1.p4.6.m6.2.3.3.3.2.1.cmml" xref="S2.SS1.p4.6.m6.2.3.3.3.2">subscript</csymbol><ci id="S2.SS1.p4.6.m6.2.3.3.3.2.2.cmml" xref="S2.SS1.p4.6.m6.2.3.3.3.2.2">ℙ</ci><times id="S2.SS1.p4.6.m6.2.3.3.3.2.3.cmml" xref="S2.SS1.p4.6.m6.2.3.3.3.2.3"></times></apply><apply id="S2.SS1.p4.6.m6.2.3.3.3.3.1.cmml" xref="S2.SS1.p4.6.m6.2.3.3.3.3.2"><csymbol cd="latexml" id="S2.SS1.p4.6.m6.2.3.3.3.3.1.1.cmml" xref="S2.SS1.p4.6.m6.2.3.3.3.3.2.1">delimited-[]</csymbol><ci id="S2.SS1.p4.6.m6.2.2.cmml" xref="S2.SS1.p4.6.m6.2.2">𝑜</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.6.m6.2c">\text{TE}=\mathbb{P}[o]-\mathbb{P}_{*}[o]</annotation></semantics></math>. The <span id="S2.SS1.p4.9.2" class="ltx_text ltx_font_bold">indirect effect</span> (IE) of a specific mediating state <math id="S2.SS1.p4.7.m7.1" class="ltx_Math" alttext="\smash{h_{i}^{(l)}}" display="inline"><semantics id="S2.SS1.p4.7.m7.1a"><msubsup id="S2.SS1.p4.7.m7.1.2" xref="S2.SS1.p4.7.m7.1.2.cmml"><mi id="S2.SS1.p4.7.m7.1.2.2.2" xref="S2.SS1.p4.7.m7.1.2.2.2.cmml">h</mi><mi id="S2.SS1.p4.7.m7.1.2.2.3" xref="S2.SS1.p4.7.m7.1.2.2.3.cmml">i</mi><mrow id="S2.SS1.p4.7.m7.1.1.1.3" xref="S2.SS1.p4.7.m7.1.2.cmml"><mo stretchy="false" id="S2.SS1.p4.7.m7.1.1.1.3.1" xref="S2.SS1.p4.7.m7.1.2.cmml">(</mo><mi id="S2.SS1.p4.7.m7.1.1.1.1" xref="S2.SS1.p4.7.m7.1.1.1.1.cmml">l</mi><mo stretchy="false" id="S2.SS1.p4.7.m7.1.1.1.3.2" xref="S2.SS1.p4.7.m7.1.2.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.7.m7.1b"><apply id="S2.SS1.p4.7.m7.1.2.cmml" xref="S2.SS1.p4.7.m7.1.2"><csymbol cd="ambiguous" id="S2.SS1.p4.7.m7.1.2.1.cmml" xref="S2.SS1.p4.7.m7.1.2">superscript</csymbol><apply id="S2.SS1.p4.7.m7.1.2.2.cmml" xref="S2.SS1.p4.7.m7.1.2"><csymbol cd="ambiguous" id="S2.SS1.p4.7.m7.1.2.2.1.cmml" xref="S2.SS1.p4.7.m7.1.2">subscript</csymbol><ci id="S2.SS1.p4.7.m7.1.2.2.2.cmml" xref="S2.SS1.p4.7.m7.1.2.2.2">ℎ</ci><ci id="S2.SS1.p4.7.m7.1.2.2.3.cmml" xref="S2.SS1.p4.7.m7.1.2.2.3">𝑖</ci></apply><ci id="S2.SS1.p4.7.m7.1.1.1.1.cmml" xref="S2.SS1.p4.7.m7.1.1.1.1">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.7.m7.1c">\smash{h_{i}^{(l)}}</annotation></semantics></math> is defined as the difference between the probability of <math id="S2.SS1.p4.8.m8.1" class="ltx_Math" alttext="o" display="inline"><semantics id="S2.SS1.p4.8.m8.1a"><mi id="S2.SS1.p4.8.m8.1.1" xref="S2.SS1.p4.8.m8.1.1.cmml">o</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.8.m8.1b"><ci id="S2.SS1.p4.8.m8.1.1.cmml" xref="S2.SS1.p4.8.m8.1.1">𝑜</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.8.m8.1c">o</annotation></semantics></math> under the corrupted version and the probability when that state is set to its clean version, while the subject remains corrupted: <math id="S2.SS1.p4.9.m9.5" class="ltx_Math" alttext="\text{IE}=\mathbb{P}_{*,\smash{\text{ clean }h_{i}^{(l)}}}[o]-\mathbb{P}_{*}[o]" display="inline"><semantics id="S2.SS1.p4.9.m9.5a"><mrow id="S2.SS1.p4.9.m9.5.6" xref="S2.SS1.p4.9.m9.5.6.cmml"><mtext id="S2.SS1.p4.9.m9.5.6.2" xref="S2.SS1.p4.9.m9.5.6.2a.cmml">IE</mtext><mo id="S2.SS1.p4.9.m9.5.6.1" xref="S2.SS1.p4.9.m9.5.6.1.cmml">=</mo><mrow id="S2.SS1.p4.9.m9.5.6.3" xref="S2.SS1.p4.9.m9.5.6.3.cmml"><mrow id="S2.SS1.p4.9.m9.5.6.3.2" xref="S2.SS1.p4.9.m9.5.6.3.2.cmml"><msub id="S2.SS1.p4.9.m9.5.6.3.2.2" xref="S2.SS1.p4.9.m9.5.6.3.2.2.cmml"><mi id="S2.SS1.p4.9.m9.5.6.3.2.2.2" xref="S2.SS1.p4.9.m9.5.6.3.2.2.2.cmml">ℙ</mi><mrow id="S2.SS1.p4.9.m9.3.3.3.3" xref="S2.SS1.p4.9.m9.3.3.3.4.cmml"><mo rspace="0em" id="S2.SS1.p4.9.m9.2.2.2.2" xref="S2.SS1.p4.9.m9.2.2.2.2.cmml">∗</mo><mo id="S2.SS1.p4.9.m9.3.3.3.3.2" xref="S2.SS1.p4.9.m9.3.3.3.4.cmml">,</mo><mrow id="S2.SS1.p4.9.m9.3.3.3.3.1" xref="S2.SS1.p4.9.m9.3.3.3.3.1.cmml"><mtext id="S2.SS1.p4.9.m9.3.3.3.3.1.2" xref="S2.SS1.p4.9.m9.3.3.3.3.1.2a.cmml"> clean </mtext><mo lspace="0em" rspace="0em" id="S2.SS1.p4.9.m9.3.3.3.3.1.1" xref="S2.SS1.p4.9.m9.3.3.3.3.1.1.cmml">​</mo><msubsup id="S2.SS1.p4.9.m9.3.3.3.3.1.3" xref="S2.SS1.p4.9.m9.3.3.3.3.1.3.cmml"><mi id="S2.SS1.p4.9.m9.3.3.3.3.1.3.2.2" xref="S2.SS1.p4.9.m9.3.3.3.3.1.3.2.2.cmml">h</mi><mi id="S2.SS1.p4.9.m9.3.3.3.3.1.3.2.3" xref="S2.SS1.p4.9.m9.3.3.3.3.1.3.2.3.cmml">i</mi><mrow id="S2.SS1.p4.9.m9.1.1.1.1.1.3" xref="S2.SS1.p4.9.m9.3.3.3.3.1.3.cmml"><mo stretchy="false" id="S2.SS1.p4.9.m9.1.1.1.1.1.3.1" xref="S2.SS1.p4.9.m9.3.3.3.3.1.3.cmml">(</mo><mi id="S2.SS1.p4.9.m9.1.1.1.1.1.1" xref="S2.SS1.p4.9.m9.1.1.1.1.1.1.cmml">l</mi><mo stretchy="false" id="S2.SS1.p4.9.m9.1.1.1.1.1.3.2" xref="S2.SS1.p4.9.m9.3.3.3.3.1.3.cmml">)</mo></mrow></msubsup></mrow></mrow></msub><mo lspace="0em" rspace="0em" id="S2.SS1.p4.9.m9.5.6.3.2.1" xref="S2.SS1.p4.9.m9.5.6.3.2.1.cmml">​</mo><mrow id="S2.SS1.p4.9.m9.5.6.3.2.3.2" xref="S2.SS1.p4.9.m9.5.6.3.2.3.1.cmml"><mo stretchy="false" id="S2.SS1.p4.9.m9.5.6.3.2.3.2.1" xref="S2.SS1.p4.9.m9.5.6.3.2.3.1.1.cmml">[</mo><mi id="S2.SS1.p4.9.m9.4.4" xref="S2.SS1.p4.9.m9.4.4.cmml">o</mi><mo stretchy="false" id="S2.SS1.p4.9.m9.5.6.3.2.3.2.2" xref="S2.SS1.p4.9.m9.5.6.3.2.3.1.1.cmml">]</mo></mrow></mrow><mo id="S2.SS1.p4.9.m9.5.6.3.1" xref="S2.SS1.p4.9.m9.5.6.3.1.cmml">−</mo><mrow id="S2.SS1.p4.9.m9.5.6.3.3" xref="S2.SS1.p4.9.m9.5.6.3.3.cmml"><msub id="S2.SS1.p4.9.m9.5.6.3.3.2" xref="S2.SS1.p4.9.m9.5.6.3.3.2.cmml"><mi id="S2.SS1.p4.9.m9.5.6.3.3.2.2" xref="S2.SS1.p4.9.m9.5.6.3.3.2.2.cmml">ℙ</mi><mo id="S2.SS1.p4.9.m9.5.6.3.3.2.3" xref="S2.SS1.p4.9.m9.5.6.3.3.2.3.cmml">∗</mo></msub><mo lspace="0em" rspace="0em" id="S2.SS1.p4.9.m9.5.6.3.3.1" xref="S2.SS1.p4.9.m9.5.6.3.3.1.cmml">​</mo><mrow id="S2.SS1.p4.9.m9.5.6.3.3.3.2" xref="S2.SS1.p4.9.m9.5.6.3.3.3.1.cmml"><mo stretchy="false" id="S2.SS1.p4.9.m9.5.6.3.3.3.2.1" xref="S2.SS1.p4.9.m9.5.6.3.3.3.1.1.cmml">[</mo><mi id="S2.SS1.p4.9.m9.5.5" xref="S2.SS1.p4.9.m9.5.5.cmml">o</mi><mo stretchy="false" id="S2.SS1.p4.9.m9.5.6.3.3.3.2.2" xref="S2.SS1.p4.9.m9.5.6.3.3.3.1.1.cmml">]</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.9.m9.5b"><apply id="S2.SS1.p4.9.m9.5.6.cmml" xref="S2.SS1.p4.9.m9.5.6"><eq id="S2.SS1.p4.9.m9.5.6.1.cmml" xref="S2.SS1.p4.9.m9.5.6.1"></eq><ci id="S2.SS1.p4.9.m9.5.6.2a.cmml" xref="S2.SS1.p4.9.m9.5.6.2"><mtext id="S2.SS1.p4.9.m9.5.6.2.cmml" xref="S2.SS1.p4.9.m9.5.6.2">IE</mtext></ci><apply id="S2.SS1.p4.9.m9.5.6.3.cmml" xref="S2.SS1.p4.9.m9.5.6.3"><minus id="S2.SS1.p4.9.m9.5.6.3.1.cmml" xref="S2.SS1.p4.9.m9.5.6.3.1"></minus><apply id="S2.SS1.p4.9.m9.5.6.3.2.cmml" xref="S2.SS1.p4.9.m9.5.6.3.2"><times id="S2.SS1.p4.9.m9.5.6.3.2.1.cmml" xref="S2.SS1.p4.9.m9.5.6.3.2.1"></times><apply id="S2.SS1.p4.9.m9.5.6.3.2.2.cmml" xref="S2.SS1.p4.9.m9.5.6.3.2.2"><csymbol cd="ambiguous" id="S2.SS1.p4.9.m9.5.6.3.2.2.1.cmml" xref="S2.SS1.p4.9.m9.5.6.3.2.2">subscript</csymbol><ci id="S2.SS1.p4.9.m9.5.6.3.2.2.2.cmml" xref="S2.SS1.p4.9.m9.5.6.3.2.2.2">ℙ</ci><list id="S2.SS1.p4.9.m9.3.3.3.4.cmml" xref="S2.SS1.p4.9.m9.3.3.3.3"><times id="S2.SS1.p4.9.m9.2.2.2.2.cmml" xref="S2.SS1.p4.9.m9.2.2.2.2"></times><apply id="S2.SS1.p4.9.m9.3.3.3.3.1.cmml" xref="S2.SS1.p4.9.m9.3.3.3.3.1"><times id="S2.SS1.p4.9.m9.3.3.3.3.1.1.cmml" xref="S2.SS1.p4.9.m9.3.3.3.3.1.1"></times><ci id="S2.SS1.p4.9.m9.3.3.3.3.1.2a.cmml" xref="S2.SS1.p4.9.m9.3.3.3.3.1.2"><mtext mathsize="70%" id="S2.SS1.p4.9.m9.3.3.3.3.1.2.cmml" xref="S2.SS1.p4.9.m9.3.3.3.3.1.2"> clean </mtext></ci><apply id="S2.SS1.p4.9.m9.3.3.3.3.1.3.cmml" xref="S2.SS1.p4.9.m9.3.3.3.3.1.3"><csymbol cd="ambiguous" id="S2.SS1.p4.9.m9.3.3.3.3.1.3.1.cmml" xref="S2.SS1.p4.9.m9.3.3.3.3.1.3">superscript</csymbol><apply id="S2.SS1.p4.9.m9.3.3.3.3.1.3.2.cmml" xref="S2.SS1.p4.9.m9.3.3.3.3.1.3"><csymbol cd="ambiguous" id="S2.SS1.p4.9.m9.3.3.3.3.1.3.2.1.cmml" xref="S2.SS1.p4.9.m9.3.3.3.3.1.3">subscript</csymbol><ci id="S2.SS1.p4.9.m9.3.3.3.3.1.3.2.2.cmml" xref="S2.SS1.p4.9.m9.3.3.3.3.1.3.2.2">ℎ</ci><ci id="S2.SS1.p4.9.m9.3.3.3.3.1.3.2.3.cmml" xref="S2.SS1.p4.9.m9.3.3.3.3.1.3.2.3">𝑖</ci></apply><ci id="S2.SS1.p4.9.m9.1.1.1.1.1.1.cmml" xref="S2.SS1.p4.9.m9.1.1.1.1.1.1">𝑙</ci></apply></apply></list></apply><apply id="S2.SS1.p4.9.m9.5.6.3.2.3.1.cmml" xref="S2.SS1.p4.9.m9.5.6.3.2.3.2"><csymbol cd="latexml" id="S2.SS1.p4.9.m9.5.6.3.2.3.1.1.cmml" xref="S2.SS1.p4.9.m9.5.6.3.2.3.2.1">delimited-[]</csymbol><ci id="S2.SS1.p4.9.m9.4.4.cmml" xref="S2.SS1.p4.9.m9.4.4">𝑜</ci></apply></apply><apply id="S2.SS1.p4.9.m9.5.6.3.3.cmml" xref="S2.SS1.p4.9.m9.5.6.3.3"><times id="S2.SS1.p4.9.m9.5.6.3.3.1.cmml" xref="S2.SS1.p4.9.m9.5.6.3.3.1"></times><apply id="S2.SS1.p4.9.m9.5.6.3.3.2.cmml" xref="S2.SS1.p4.9.m9.5.6.3.3.2"><csymbol cd="ambiguous" id="S2.SS1.p4.9.m9.5.6.3.3.2.1.cmml" xref="S2.SS1.p4.9.m9.5.6.3.3.2">subscript</csymbol><ci id="S2.SS1.p4.9.m9.5.6.3.3.2.2.cmml" xref="S2.SS1.p4.9.m9.5.6.3.3.2.2">ℙ</ci><times id="S2.SS1.p4.9.m9.5.6.3.3.2.3.cmml" xref="S2.SS1.p4.9.m9.5.6.3.3.2.3"></times></apply><apply id="S2.SS1.p4.9.m9.5.6.3.3.3.1.cmml" xref="S2.SS1.p4.9.m9.5.6.3.3.3.2"><csymbol cd="latexml" id="S2.SS1.p4.9.m9.5.6.3.3.3.1.1.cmml" xref="S2.SS1.p4.9.m9.5.6.3.3.3.2.1">delimited-[]</csymbol><ci id="S2.SS1.p4.9.m9.5.5.cmml" xref="S2.SS1.p4.9.m9.5.5">𝑜</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.9.m9.5c">\text{IE}=\mathbb{P}_{*,\smash{\text{ clean }h_{i}^{(l)}}}[o]-\mathbb{P}_{*}[o]</annotation></semantics></math>.
Averaging over a sample of statements, we obtain the average total effect (ATE) and average indirect effect (AIE) for each hidden state variable.<span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span>One could also compute the direct effect, which flows through other model components besides the chosen mediator. However, we found this effect to be noisy and uninformative, in line with results by <cite class="ltx_cite ltx_citemacro_citet">Vig et al. (<a href="#bib.bib38" title="" class="ltx_ref">2020b</a>)</cite>.</span></span></span></p>
</div>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>Causal Tracing Results</h3>

<div id="S2.SS2.p1" class="ltx_para">
<p id="S2.SS2.p1.1" class="ltx_p">We compute the average indirect effect (AIE) over 1000 factual statements (details in Appendix <a href="#A2.SS1" title="B.1 Experimental Settings ‣ Appendix B Causal Tracing ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B.1</span></a>), varying the mediator over different positions in the sentence and different model components including individual states, MLP layers, and attention layers.
Figure <a href="#S2.F2" title="Figure 2 ‣ 2 Interventions on Activations for Tracing Information Flow ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> plots the AIE of the internal components of GPT-2 XL (1.5B parameters). The ATE of this experiment is 18.6%, and we note that a large portion of the effect is mediated by strongly causal individual states (AIE=8.7% at layer 15) at the last subject token. The presence of strong causal states at a late site immediately before the prediction is unsurprising, but their emergence at an <span id="S2.SS2.p1.1.1" class="ltx_text ltx_font_italic">early</span> site at the last token of the subject is a new discovery.</p>
</div>
<div id="S2.SS2.p2" class="ltx_para">
<p id="S2.SS2.p2.1" class="ltx_p">Decomposing the causal effects of contributions of MLP and attention modules (Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>fg and Figure <a href="#S2.F2" title="Figure 2 ‣ 2 Interventions on Activations for Tracing Information Flow ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>bc) suggests a decisive role for MLP modules at the early site: MLP contributions peak at AIE 6.6%, while attention at the last subject token is only AIE 1.6%; attention is more important at the last token of the prompt. Appendix <a href="#A2.SS2" title="B.2 Separating MLP and Attn Effects ‣ Appendix B Causal Tracing ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B.2</span></a> further discusses this decomposition.</p>
</div>
<div id="S2.SS2.p3" class="ltx_para">
<p id="S2.SS2.p3.2" class="ltx_p">Finally, to gain a clearer picture of the special role of MLP layers at the early site, we analyze indirect effects with a modified causal graph (Figure <a href="#S2.F3" title="Figure 3 ‣ 2.2 Causal Tracing Results ‣ 2 Interventions on Activations for Tracing Information Flow ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>). (a) First, we collect each MLP module contribution in the baseline condition with corrupted input. (b) Then, to isolate the effects of MLP modules when measuring causal effects, we modify the computation graph to sever MLP computations at token <math id="S2.SS2.p3.1.m1.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S2.SS2.p3.1.m1.1a"><mi id="S2.SS2.p3.1.m1.1.1" xref="S2.SS2.p3.1.m1.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.1.m1.1b"><ci id="S2.SS2.p3.1.m1.1.1.cmml" xref="S2.SS2.p3.1.m1.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.1.m1.1c">i</annotation></semantics></math> and freeze them in the baseline corrupted state so that they are unaffected by the insertion of clean state for <math id="S2.SS2.p3.2.m2.1" class="ltx_Math" alttext="\smash{h_{i}^{(l)}}" display="inline"><semantics id="S2.SS2.p3.2.m2.1a"><msubsup id="S2.SS2.p3.2.m2.1.2" xref="S2.SS2.p3.2.m2.1.2.cmml"><mi id="S2.SS2.p3.2.m2.1.2.2.2" xref="S2.SS2.p3.2.m2.1.2.2.2.cmml">h</mi><mi id="S2.SS2.p3.2.m2.1.2.2.3" xref="S2.SS2.p3.2.m2.1.2.2.3.cmml">i</mi><mrow id="S2.SS2.p3.2.m2.1.1.1.3" xref="S2.SS2.p3.2.m2.1.2.cmml"><mo stretchy="false" id="S2.SS2.p3.2.m2.1.1.1.3.1" xref="S2.SS2.p3.2.m2.1.2.cmml">(</mo><mi id="S2.SS2.p3.2.m2.1.1.1.1" xref="S2.SS2.p3.2.m2.1.1.1.1.cmml">l</mi><mo stretchy="false" id="S2.SS2.p3.2.m2.1.1.1.3.2" xref="S2.SS2.p3.2.m2.1.2.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.2.m2.1b"><apply id="S2.SS2.p3.2.m2.1.2.cmml" xref="S2.SS2.p3.2.m2.1.2"><csymbol cd="ambiguous" id="S2.SS2.p3.2.m2.1.2.1.cmml" xref="S2.SS2.p3.2.m2.1.2">superscript</csymbol><apply id="S2.SS2.p3.2.m2.1.2.2.cmml" xref="S2.SS2.p3.2.m2.1.2"><csymbol cd="ambiguous" id="S2.SS2.p3.2.m2.1.2.2.1.cmml" xref="S2.SS2.p3.2.m2.1.2">subscript</csymbol><ci id="S2.SS2.p3.2.m2.1.2.2.2.cmml" xref="S2.SS2.p3.2.m2.1.2.2.2">ℎ</ci><ci id="S2.SS2.p3.2.m2.1.2.2.3.cmml" xref="S2.SS2.p3.2.m2.1.2.2.3">𝑖</ci></apply><ci id="S2.SS2.p3.2.m2.1.1.1.1.cmml" xref="S2.SS2.p3.2.m2.1.1.1.1">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.2.m2.1c">\smash{h_{i}^{(l)}}</annotation></semantics></math>. This modification is a way of probing <em id="S2.SS2.p3.2.1" class="ltx_emph ltx_font_italic">path-specific effects</em> <cite class="ltx_cite ltx_citemacro_citep">(Pearl, <a href="#bib.bib28" title="" class="ltx_ref">2001</a>)</cite> for paths that avoid MLP computations. (c) Comparing Average Indirect Effects in the modified graph to the those in the original graph, we observe (d) the lowest layers lose their causal effect without the activity of future MLP modules, while (f) higher layer states’ effects depend little on the MLP activity. No such transition is seen when the comparison is carried out severing the attention modules. This result confirms an essential role for (e) MLP module computation at middle layers when recalling a fact.</p>
</div>
<div id="S2.SS2.p4" class="ltx_para">
<p id="S2.SS2.p4.1" class="ltx_p">Appendix <a href="#A2" title="Appendix B Causal Tracing ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a> has results on other autoregressive models and experimental settings. In particular, we find that Causal Tracing is more informative than gradient-based salience methods such as integrated gradients <cite class="ltx_cite ltx_citemacro_citep">(Sundararajan et al., <a href="#bib.bib35" title="" class="ltx_ref">2017</a>)</cite> (Figure <a href="#A2.F16" title="Figure 16 ‣ B.4 Causal Tracing Examples and Further Insights ‣ Appendix B Causal Tracing ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">16</span></a>) and is robust under different noise configurations.</p>
</div>
<figure id="S2.F3" class="ltx_figure"><img src="/html/2202.05262/assets/x3.png" id="S2.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="113" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S2.F3.3.1.1" class="ltx_text" style="font-size:90%;">Figure 3</span>: </span><span id="S2.F3.4.2" class="ltx_text ltx_font_bold" style="font-size:90%;">Causal effects with a modified computation graph<span id="S2.F3.4.2.1" class="ltx_text ltx_font_medium">. (a,b) To isolate the effects of MLP modules when measuring causal effects, the computation graph is modified. (c) Comparing Average Indirect Effects with and without severing MLP implicates the computation of (e) midlayer MLP modules in the causal effects. No similar gap is seen when attention is similarly severed.</span></span></figcaption>
</figure>
<div id="S2.SS2.p5" class="ltx_para">
<p id="S2.SS2.p5.1" class="ltx_p">We hypothesize that this localized midlayer MLP key–value mapping recalls facts about the subject.</p>
</div>
</section>
<section id="S2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.3 </span>The Localized Factual Association Hypothesis</h3>

<div id="S2.SS3.p1" class="ltx_para">
<p id="S2.SS3.p1.1" class="ltx_p">Based on causal traces, we posit a specific mechanism for storage of factual associations: each midlayer MLP module accepts inputs that encode a subject, then produces outputs that recall memorized properties about that subject. Middle layer MLP outputs accumulate information, then the summed information is copied to the last token by attention at high layers.</p>
</div>
<div id="S2.SS3.p2" class="ltx_para">
<p id="S2.SS3.p2.1" class="ltx_p">This hypothesis localizes factual association along three dimensions, placing it (i) in the MLP modules (ii) at specific middle layers (iii) and specifically at the processing of the subject’s last token. It is consistent with the <cite class="ltx_cite ltx_citemacro_citet">Geva et al. (<a href="#bib.bib18" title="" class="ltx_ref">2021</a>)</cite> view that MLP layers store knowledge, and the <cite class="ltx_cite ltx_citemacro_citet">Elhage et al. (<a href="#bib.bib14" title="" class="ltx_ref">2021</a>)</cite> study showing an information-copying role for self-attention. Furthermore, informed by the <cite class="ltx_cite ltx_citemacro_citet">Zhao et al. (<a href="#bib.bib41" title="" class="ltx_ref">2021</a>)</cite> finding that transformer layer order can be exchanged with minimal change in behavior, we propose that this picture is complete. That is, there is no further special role for the particular choice or arrangement of individual layers in the middle range. We conjecture that any fact could be equivalently stored in any one of the middle MLP layers.
To test our hypothesis, we narrow our attention to a single MLP module at a mid-range layer <math id="S2.SS3.p2.1.m1.1" class="ltx_Math" alttext="l^{*}" display="inline"><semantics id="S2.SS3.p2.1.m1.1a"><msup id="S2.SS3.p2.1.m1.1.1" xref="S2.SS3.p2.1.m1.1.1.cmml"><mi id="S2.SS3.p2.1.m1.1.1.2" xref="S2.SS3.p2.1.m1.1.1.2.cmml">l</mi><mo id="S2.SS3.p2.1.m1.1.1.3" xref="S2.SS3.p2.1.m1.1.1.3.cmml">∗</mo></msup><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.1.m1.1b"><apply id="S2.SS3.p2.1.m1.1.1.cmml" xref="S2.SS3.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS3.p2.1.m1.1.1.1.cmml" xref="S2.SS3.p2.1.m1.1.1">superscript</csymbol><ci id="S2.SS3.p2.1.m1.1.1.2.cmml" xref="S2.SS3.p2.1.m1.1.1.2">𝑙</ci><times id="S2.SS3.p2.1.m1.1.1.3.cmml" xref="S2.SS3.p2.1.m1.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.1.m1.1c">l^{*}</annotation></semantics></math>, and ask whether its weights can be explicitly modified to store an arbitrary fact.</p>
</div>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Interventions on <span id="S3.1.1" class="ltx_text ltx_framed ltx_framed_underline">Weights</span> for Understanding Factual Association Storage</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.2" class="ltx_p">While Causal Tracing has implicated MLP modules in recalling factual associations, we also wish to understand how facts are <span id="S3.p1.2.1" class="ltx_text ltx_font_italic">stored in weights</span>. <cite class="ltx_cite ltx_citemacro_citet">Geva et al. (<a href="#bib.bib18" title="" class="ltx_ref">2021</a>)</cite> observed that MLP layers (Figure <a href="#S3.F4" title="Figure 4 ‣ 3.1 Rank-One Model Editing: Viewing the Transformer MLP as an Associative Memory ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>cde) can act as two-layer key–value memories,<span id="footnote4" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">4</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">4</sup><span class="ltx_tag ltx_tag_note">4</span>Unrelated to keys and values in self-attention.</span></span></span> where the neurons of the first layer <math id="S3.p1.1.m1.1" class="ltx_Math" alttext="\smash{W^{(l)}_{fc}}" display="inline"><semantics id="S3.p1.1.m1.1a"><msubsup id="S3.p1.1.m1.1.2" xref="S3.p1.1.m1.1.2.cmml"><mi id="S3.p1.1.m1.1.2.2.2" xref="S3.p1.1.m1.1.2.2.2.cmml">W</mi><mrow id="S3.p1.1.m1.1.2.3" xref="S3.p1.1.m1.1.2.3.cmml"><mi id="S3.p1.1.m1.1.2.3.2" xref="S3.p1.1.m1.1.2.3.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="S3.p1.1.m1.1.2.3.1" xref="S3.p1.1.m1.1.2.3.1.cmml">​</mo><mi id="S3.p1.1.m1.1.2.3.3" xref="S3.p1.1.m1.1.2.3.3.cmml">c</mi></mrow><mrow id="S3.p1.1.m1.1.1.1.3" xref="S3.p1.1.m1.1.2.cmml"><mo stretchy="false" id="S3.p1.1.m1.1.1.1.3.1" xref="S3.p1.1.m1.1.2.cmml">(</mo><mi id="S3.p1.1.m1.1.1.1.1" xref="S3.p1.1.m1.1.1.1.1.cmml">l</mi><mo stretchy="false" id="S3.p1.1.m1.1.1.1.3.2" xref="S3.p1.1.m1.1.2.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S3.p1.1.m1.1b"><apply id="S3.p1.1.m1.1.2.cmml" xref="S3.p1.1.m1.1.2"><csymbol cd="ambiguous" id="S3.p1.1.m1.1.2.1.cmml" xref="S3.p1.1.m1.1.2">subscript</csymbol><apply id="S3.p1.1.m1.1.2.2.cmml" xref="S3.p1.1.m1.1.2"><csymbol cd="ambiguous" id="S3.p1.1.m1.1.2.2.1.cmml" xref="S3.p1.1.m1.1.2">superscript</csymbol><ci id="S3.p1.1.m1.1.2.2.2.cmml" xref="S3.p1.1.m1.1.2.2.2">𝑊</ci><ci id="S3.p1.1.m1.1.1.1.1.cmml" xref="S3.p1.1.m1.1.1.1.1">𝑙</ci></apply><apply id="S3.p1.1.m1.1.2.3.cmml" xref="S3.p1.1.m1.1.2.3"><times id="S3.p1.1.m1.1.2.3.1.cmml" xref="S3.p1.1.m1.1.2.3.1"></times><ci id="S3.p1.1.m1.1.2.3.2.cmml" xref="S3.p1.1.m1.1.2.3.2">𝑓</ci><ci id="S3.p1.1.m1.1.2.3.3.cmml" xref="S3.p1.1.m1.1.2.3.3">𝑐</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.1.m1.1c">\smash{W^{(l)}_{fc}}</annotation></semantics></math> form a <span id="S3.p1.2.2" class="ltx_text ltx_font_italic">key</span>, with which the second layer <math id="S3.p1.2.m2.1" class="ltx_Math" alttext="\smash{W^{(l)}_{proj}}" display="inline"><semantics id="S3.p1.2.m2.1a"><msubsup id="S3.p1.2.m2.1.2" xref="S3.p1.2.m2.1.2.cmml"><mi id="S3.p1.2.m2.1.2.2.2" xref="S3.p1.2.m2.1.2.2.2.cmml">W</mi><mrow id="S3.p1.2.m2.1.2.3" xref="S3.p1.2.m2.1.2.3.cmml"><mi id="S3.p1.2.m2.1.2.3.2" xref="S3.p1.2.m2.1.2.3.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="S3.p1.2.m2.1.2.3.1" xref="S3.p1.2.m2.1.2.3.1.cmml">​</mo><mi id="S3.p1.2.m2.1.2.3.3" xref="S3.p1.2.m2.1.2.3.3.cmml">r</mi><mo lspace="0em" rspace="0em" id="S3.p1.2.m2.1.2.3.1a" xref="S3.p1.2.m2.1.2.3.1.cmml">​</mo><mi id="S3.p1.2.m2.1.2.3.4" xref="S3.p1.2.m2.1.2.3.4.cmml">o</mi><mo lspace="0em" rspace="0em" id="S3.p1.2.m2.1.2.3.1b" xref="S3.p1.2.m2.1.2.3.1.cmml">​</mo><mi id="S3.p1.2.m2.1.2.3.5" xref="S3.p1.2.m2.1.2.3.5.cmml">j</mi></mrow><mrow id="S3.p1.2.m2.1.1.1.3" xref="S3.p1.2.m2.1.2.cmml"><mo stretchy="false" id="S3.p1.2.m2.1.1.1.3.1" xref="S3.p1.2.m2.1.2.cmml">(</mo><mi id="S3.p1.2.m2.1.1.1.1" xref="S3.p1.2.m2.1.1.1.1.cmml">l</mi><mo stretchy="false" id="S3.p1.2.m2.1.1.1.3.2" xref="S3.p1.2.m2.1.2.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S3.p1.2.m2.1b"><apply id="S3.p1.2.m2.1.2.cmml" xref="S3.p1.2.m2.1.2"><csymbol cd="ambiguous" id="S3.p1.2.m2.1.2.1.cmml" xref="S3.p1.2.m2.1.2">subscript</csymbol><apply id="S3.p1.2.m2.1.2.2.cmml" xref="S3.p1.2.m2.1.2"><csymbol cd="ambiguous" id="S3.p1.2.m2.1.2.2.1.cmml" xref="S3.p1.2.m2.1.2">superscript</csymbol><ci id="S3.p1.2.m2.1.2.2.2.cmml" xref="S3.p1.2.m2.1.2.2.2">𝑊</ci><ci id="S3.p1.2.m2.1.1.1.1.cmml" xref="S3.p1.2.m2.1.1.1.1">𝑙</ci></apply><apply id="S3.p1.2.m2.1.2.3.cmml" xref="S3.p1.2.m2.1.2.3"><times id="S3.p1.2.m2.1.2.3.1.cmml" xref="S3.p1.2.m2.1.2.3.1"></times><ci id="S3.p1.2.m2.1.2.3.2.cmml" xref="S3.p1.2.m2.1.2.3.2">𝑝</ci><ci id="S3.p1.2.m2.1.2.3.3.cmml" xref="S3.p1.2.m2.1.2.3.3">𝑟</ci><ci id="S3.p1.2.m2.1.2.3.4.cmml" xref="S3.p1.2.m2.1.2.3.4">𝑜</ci><ci id="S3.p1.2.m2.1.2.3.5.cmml" xref="S3.p1.2.m2.1.2.3.5">𝑗</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.2.m2.1c">\smash{W^{(l)}_{proj}}</annotation></semantics></math> retrieves an associated <span id="S3.p1.2.3" class="ltx_text ltx_font_italic">value</span>. We hypothesize that MLPs can be modeled as a linear associative memory; note that this differs from <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib18" title="" class="ltx_ref">Geva et al.</a></cite>’s per-neuron view.</p>
</div>
<div id="S3.p2" class="ltx_para">
<p id="S3.p2.2" class="ltx_p">We test this hypothesis by conducting a new type of intervention: modifying factual associations with Rank-One Model Editing (ROME). Being able to insert a new knowledge tuple <math id="S3.p2.1.m1.3" class="ltx_Math" alttext="t^{*}=(s,r,o^{*})" display="inline"><semantics id="S3.p2.1.m1.3a"><mrow id="S3.p2.1.m1.3.3" xref="S3.p2.1.m1.3.3.cmml"><msup id="S3.p2.1.m1.3.3.3" xref="S3.p2.1.m1.3.3.3.cmml"><mi id="S3.p2.1.m1.3.3.3.2" xref="S3.p2.1.m1.3.3.3.2.cmml">t</mi><mo id="S3.p2.1.m1.3.3.3.3" xref="S3.p2.1.m1.3.3.3.3.cmml">∗</mo></msup><mo id="S3.p2.1.m1.3.3.2" xref="S3.p2.1.m1.3.3.2.cmml">=</mo><mrow id="S3.p2.1.m1.3.3.1.1" xref="S3.p2.1.m1.3.3.1.2.cmml"><mo stretchy="false" id="S3.p2.1.m1.3.3.1.1.2" xref="S3.p2.1.m1.3.3.1.2.cmml">(</mo><mi id="S3.p2.1.m1.1.1" xref="S3.p2.1.m1.1.1.cmml">s</mi><mo id="S3.p2.1.m1.3.3.1.1.3" xref="S3.p2.1.m1.3.3.1.2.cmml">,</mo><mi id="S3.p2.1.m1.2.2" xref="S3.p2.1.m1.2.2.cmml">r</mi><mo id="S3.p2.1.m1.3.3.1.1.4" xref="S3.p2.1.m1.3.3.1.2.cmml">,</mo><msup id="S3.p2.1.m1.3.3.1.1.1" xref="S3.p2.1.m1.3.3.1.1.1.cmml"><mi id="S3.p2.1.m1.3.3.1.1.1.2" xref="S3.p2.1.m1.3.3.1.1.1.2.cmml">o</mi><mo id="S3.p2.1.m1.3.3.1.1.1.3" xref="S3.p2.1.m1.3.3.1.1.1.3.cmml">∗</mo></msup><mo stretchy="false" id="S3.p2.1.m1.3.3.1.1.5" xref="S3.p2.1.m1.3.3.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.p2.1.m1.3b"><apply id="S3.p2.1.m1.3.3.cmml" xref="S3.p2.1.m1.3.3"><eq id="S3.p2.1.m1.3.3.2.cmml" xref="S3.p2.1.m1.3.3.2"></eq><apply id="S3.p2.1.m1.3.3.3.cmml" xref="S3.p2.1.m1.3.3.3"><csymbol cd="ambiguous" id="S3.p2.1.m1.3.3.3.1.cmml" xref="S3.p2.1.m1.3.3.3">superscript</csymbol><ci id="S3.p2.1.m1.3.3.3.2.cmml" xref="S3.p2.1.m1.3.3.3.2">𝑡</ci><times id="S3.p2.1.m1.3.3.3.3.cmml" xref="S3.p2.1.m1.3.3.3.3"></times></apply><vector id="S3.p2.1.m1.3.3.1.2.cmml" xref="S3.p2.1.m1.3.3.1.1"><ci id="S3.p2.1.m1.1.1.cmml" xref="S3.p2.1.m1.1.1">𝑠</ci><ci id="S3.p2.1.m1.2.2.cmml" xref="S3.p2.1.m1.2.2">𝑟</ci><apply id="S3.p2.1.m1.3.3.1.1.1.cmml" xref="S3.p2.1.m1.3.3.1.1.1"><csymbol cd="ambiguous" id="S3.p2.1.m1.3.3.1.1.1.1.cmml" xref="S3.p2.1.m1.3.3.1.1.1">superscript</csymbol><ci id="S3.p2.1.m1.3.3.1.1.1.2.cmml" xref="S3.p2.1.m1.3.3.1.1.1.2">𝑜</ci><times id="S3.p2.1.m1.3.3.1.1.1.3.cmml" xref="S3.p2.1.m1.3.3.1.1.1.3"></times></apply></vector></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.1.m1.3c">t^{*}=(s,r,o^{*})</annotation></semantics></math> in place of the current tuple <math id="S3.p2.2.m2.3" class="ltx_Math" alttext="t^{c}=(s,r,o^{c})" display="inline"><semantics id="S3.p2.2.m2.3a"><mrow id="S3.p2.2.m2.3.3" xref="S3.p2.2.m2.3.3.cmml"><msup id="S3.p2.2.m2.3.3.3" xref="S3.p2.2.m2.3.3.3.cmml"><mi id="S3.p2.2.m2.3.3.3.2" xref="S3.p2.2.m2.3.3.3.2.cmml">t</mi><mi id="S3.p2.2.m2.3.3.3.3" xref="S3.p2.2.m2.3.3.3.3.cmml">c</mi></msup><mo id="S3.p2.2.m2.3.3.2" xref="S3.p2.2.m2.3.3.2.cmml">=</mo><mrow id="S3.p2.2.m2.3.3.1.1" xref="S3.p2.2.m2.3.3.1.2.cmml"><mo stretchy="false" id="S3.p2.2.m2.3.3.1.1.2" xref="S3.p2.2.m2.3.3.1.2.cmml">(</mo><mi id="S3.p2.2.m2.1.1" xref="S3.p2.2.m2.1.1.cmml">s</mi><mo id="S3.p2.2.m2.3.3.1.1.3" xref="S3.p2.2.m2.3.3.1.2.cmml">,</mo><mi id="S3.p2.2.m2.2.2" xref="S3.p2.2.m2.2.2.cmml">r</mi><mo id="S3.p2.2.m2.3.3.1.1.4" xref="S3.p2.2.m2.3.3.1.2.cmml">,</mo><msup id="S3.p2.2.m2.3.3.1.1.1" xref="S3.p2.2.m2.3.3.1.1.1.cmml"><mi id="S3.p2.2.m2.3.3.1.1.1.2" xref="S3.p2.2.m2.3.3.1.1.1.2.cmml">o</mi><mi id="S3.p2.2.m2.3.3.1.1.1.3" xref="S3.p2.2.m2.3.3.1.1.1.3.cmml">c</mi></msup><mo stretchy="false" id="S3.p2.2.m2.3.3.1.1.5" xref="S3.p2.2.m2.3.3.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.p2.2.m2.3b"><apply id="S3.p2.2.m2.3.3.cmml" xref="S3.p2.2.m2.3.3"><eq id="S3.p2.2.m2.3.3.2.cmml" xref="S3.p2.2.m2.3.3.2"></eq><apply id="S3.p2.2.m2.3.3.3.cmml" xref="S3.p2.2.m2.3.3.3"><csymbol cd="ambiguous" id="S3.p2.2.m2.3.3.3.1.cmml" xref="S3.p2.2.m2.3.3.3">superscript</csymbol><ci id="S3.p2.2.m2.3.3.3.2.cmml" xref="S3.p2.2.m2.3.3.3.2">𝑡</ci><ci id="S3.p2.2.m2.3.3.3.3.cmml" xref="S3.p2.2.m2.3.3.3.3">𝑐</ci></apply><vector id="S3.p2.2.m2.3.3.1.2.cmml" xref="S3.p2.2.m2.3.3.1.1"><ci id="S3.p2.2.m2.1.1.cmml" xref="S3.p2.2.m2.1.1">𝑠</ci><ci id="S3.p2.2.m2.2.2.cmml" xref="S3.p2.2.m2.2.2">𝑟</ci><apply id="S3.p2.2.m2.3.3.1.1.1.cmml" xref="S3.p2.2.m2.3.3.1.1.1"><csymbol cd="ambiguous" id="S3.p2.2.m2.3.3.1.1.1.1.cmml" xref="S3.p2.2.m2.3.3.1.1.1">superscript</csymbol><ci id="S3.p2.2.m2.3.3.1.1.1.2.cmml" xref="S3.p2.2.m2.3.3.1.1.1.2">𝑜</ci><ci id="S3.p2.2.m2.3.3.1.1.1.3.cmml" xref="S3.p2.2.m2.3.3.1.1.1.3">𝑐</ci></apply></vector></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.2.m2.3c">t^{c}=(s,r,o^{c})</annotation></semantics></math> with both generalization and specificity would demonstrate fine-grained understanding of the association-storage mechanisms.</p>
</div>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Rank-One Model Editing: Viewing the Transformer MLP as an Associative Memory</h3>

<figure id="S3.F4" class="ltx_figure"><img src="/html/2202.05262/assets/x4.png" id="S3.F4.g1" class="ltx_graphics ltx_img_landscape" width="461" height="111" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S3.F4.22.11.1" class="ltx_text" style="font-size:90%;">Figure 4</span>: </span><span id="S3.F4.20.10" class="ltx_text ltx_font_bold" style="font-size:90%;">Editing one MLP layer with ROME<span id="S3.F4.20.10.10" class="ltx_text ltx_font_medium">. To associate <em id="S3.F4.20.10.10.1" class="ltx_emph ltx_font_italic">Space Needle</em> with <em id="S3.F4.20.10.10.2" class="ltx_emph ltx_font_italic">Paris</em>, the ROME method inserts a new <math id="S3.F4.11.1.1.m1.2" class="ltx_Math" alttext="(k_{*},v_{*})" display="inline"><semantics id="S3.F4.11.1.1.m1.2b"><mrow id="S3.F4.11.1.1.m1.2.2.2" xref="S3.F4.11.1.1.m1.2.2.3.cmml"><mo stretchy="false" id="S3.F4.11.1.1.m1.2.2.2.3" xref="S3.F4.11.1.1.m1.2.2.3.cmml">(</mo><msub id="S3.F4.11.1.1.m1.1.1.1.1" xref="S3.F4.11.1.1.m1.1.1.1.1.cmml"><mi id="S3.F4.11.1.1.m1.1.1.1.1.2" xref="S3.F4.11.1.1.m1.1.1.1.1.2.cmml">k</mi><mo id="S3.F4.11.1.1.m1.1.1.1.1.3" xref="S3.F4.11.1.1.m1.1.1.1.1.3.cmml">∗</mo></msub><mo id="S3.F4.11.1.1.m1.2.2.2.4" xref="S3.F4.11.1.1.m1.2.2.3.cmml">,</mo><msub id="S3.F4.11.1.1.m1.2.2.2.2" xref="S3.F4.11.1.1.m1.2.2.2.2.cmml"><mi id="S3.F4.11.1.1.m1.2.2.2.2.2" xref="S3.F4.11.1.1.m1.2.2.2.2.2.cmml">v</mi><mo id="S3.F4.11.1.1.m1.2.2.2.2.3" xref="S3.F4.11.1.1.m1.2.2.2.2.3.cmml">∗</mo></msub><mo stretchy="false" id="S3.F4.11.1.1.m1.2.2.2.5" xref="S3.F4.11.1.1.m1.2.2.3.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.F4.11.1.1.m1.2c"><interval closure="open" id="S3.F4.11.1.1.m1.2.2.3.cmml" xref="S3.F4.11.1.1.m1.2.2.2"><apply id="S3.F4.11.1.1.m1.1.1.1.1.cmml" xref="S3.F4.11.1.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="S3.F4.11.1.1.m1.1.1.1.1.1.cmml" xref="S3.F4.11.1.1.m1.1.1.1.1">subscript</csymbol><ci id="S3.F4.11.1.1.m1.1.1.1.1.2.cmml" xref="S3.F4.11.1.1.m1.1.1.1.1.2">𝑘</ci><times id="S3.F4.11.1.1.m1.1.1.1.1.3.cmml" xref="S3.F4.11.1.1.m1.1.1.1.1.3"></times></apply><apply id="S3.F4.11.1.1.m1.2.2.2.2.cmml" xref="S3.F4.11.1.1.m1.2.2.2.2"><csymbol cd="ambiguous" id="S3.F4.11.1.1.m1.2.2.2.2.1.cmml" xref="S3.F4.11.1.1.m1.2.2.2.2">subscript</csymbol><ci id="S3.F4.11.1.1.m1.2.2.2.2.2.cmml" xref="S3.F4.11.1.1.m1.2.2.2.2.2">𝑣</ci><times id="S3.F4.11.1.1.m1.2.2.2.2.3.cmml" xref="S3.F4.11.1.1.m1.2.2.2.2.3"></times></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="S3.F4.11.1.1.m1.2d">(k_{*},v_{*})</annotation></semantics></math> association into layer <math id="S3.F4.12.2.2.m2.1" class="ltx_Math" alttext="l^{*}" display="inline"><semantics id="S3.F4.12.2.2.m2.1b"><msup id="S3.F4.12.2.2.m2.1.1" xref="S3.F4.12.2.2.m2.1.1.cmml"><mi id="S3.F4.12.2.2.m2.1.1.2" xref="S3.F4.12.2.2.m2.1.1.2.cmml">l</mi><mo id="S3.F4.12.2.2.m2.1.1.3" xref="S3.F4.12.2.2.m2.1.1.3.cmml">∗</mo></msup><annotation-xml encoding="MathML-Content" id="S3.F4.12.2.2.m2.1c"><apply id="S3.F4.12.2.2.m2.1.1.cmml" xref="S3.F4.12.2.2.m2.1.1"><csymbol cd="ambiguous" id="S3.F4.12.2.2.m2.1.1.1.cmml" xref="S3.F4.12.2.2.m2.1.1">superscript</csymbol><ci id="S3.F4.12.2.2.m2.1.1.2.cmml" xref="S3.F4.12.2.2.m2.1.1.2">𝑙</ci><times id="S3.F4.12.2.2.m2.1.1.3.cmml" xref="S3.F4.12.2.2.m2.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F4.12.2.2.m2.1d">l^{*}</annotation></semantics></math>, where (a) key <math id="S3.F4.13.3.3.m3.1" class="ltx_Math" alttext="k_{*}" display="inline"><semantics id="S3.F4.13.3.3.m3.1b"><msub id="S3.F4.13.3.3.m3.1.1" xref="S3.F4.13.3.3.m3.1.1.cmml"><mi id="S3.F4.13.3.3.m3.1.1.2" xref="S3.F4.13.3.3.m3.1.1.2.cmml">k</mi><mo id="S3.F4.13.3.3.m3.1.1.3" xref="S3.F4.13.3.3.m3.1.1.3.cmml">∗</mo></msub><annotation-xml encoding="MathML-Content" id="S3.F4.13.3.3.m3.1c"><apply id="S3.F4.13.3.3.m3.1.1.cmml" xref="S3.F4.13.3.3.m3.1.1"><csymbol cd="ambiguous" id="S3.F4.13.3.3.m3.1.1.1.cmml" xref="S3.F4.13.3.3.m3.1.1">subscript</csymbol><ci id="S3.F4.13.3.3.m3.1.1.2.cmml" xref="S3.F4.13.3.3.m3.1.1.2">𝑘</ci><times id="S3.F4.13.3.3.m3.1.1.3.cmml" xref="S3.F4.13.3.3.m3.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F4.13.3.3.m3.1d">k_{*}</annotation></semantics></math> is determined by the subject and (b) value <math id="S3.F4.14.4.4.m4.1" class="ltx_Math" alttext="v_{*}" display="inline"><semantics id="S3.F4.14.4.4.m4.1b"><msub id="S3.F4.14.4.4.m4.1.1" xref="S3.F4.14.4.4.m4.1.1.cmml"><mi id="S3.F4.14.4.4.m4.1.1.2" xref="S3.F4.14.4.4.m4.1.1.2.cmml">v</mi><mo id="S3.F4.14.4.4.m4.1.1.3" xref="S3.F4.14.4.4.m4.1.1.3.cmml">∗</mo></msub><annotation-xml encoding="MathML-Content" id="S3.F4.14.4.4.m4.1c"><apply id="S3.F4.14.4.4.m4.1.1.cmml" xref="S3.F4.14.4.4.m4.1.1"><csymbol cd="ambiguous" id="S3.F4.14.4.4.m4.1.1.1.cmml" xref="S3.F4.14.4.4.m4.1.1">subscript</csymbol><ci id="S3.F4.14.4.4.m4.1.1.2.cmml" xref="S3.F4.14.4.4.m4.1.1.2">𝑣</ci><times id="S3.F4.14.4.4.m4.1.1.3.cmml" xref="S3.F4.14.4.4.m4.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F4.14.4.4.m4.1d">v_{*}</annotation></semantics></math> is optimized to select the object. (c) Hidden state at layer <math id="S3.F4.15.5.5.m5.1" class="ltx_Math" alttext="l^{*}" display="inline"><semantics id="S3.F4.15.5.5.m5.1b"><msup id="S3.F4.15.5.5.m5.1.1" xref="S3.F4.15.5.5.m5.1.1.cmml"><mi id="S3.F4.15.5.5.m5.1.1.2" xref="S3.F4.15.5.5.m5.1.1.2.cmml">l</mi><mo id="S3.F4.15.5.5.m5.1.1.3" xref="S3.F4.15.5.5.m5.1.1.3.cmml">∗</mo></msup><annotation-xml encoding="MathML-Content" id="S3.F4.15.5.5.m5.1c"><apply id="S3.F4.15.5.5.m5.1.1.cmml" xref="S3.F4.15.5.5.m5.1.1"><csymbol cd="ambiguous" id="S3.F4.15.5.5.m5.1.1.1.cmml" xref="S3.F4.15.5.5.m5.1.1">superscript</csymbol><ci id="S3.F4.15.5.5.m5.1.1.2.cmml" xref="S3.F4.15.5.5.m5.1.1.2">𝑙</ci><times id="S3.F4.15.5.5.m5.1.1.3.cmml" xref="S3.F4.15.5.5.m5.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F4.15.5.5.m5.1d">l^{*}</annotation></semantics></math> and token <math id="S3.F4.16.6.6.m6.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S3.F4.16.6.6.m6.1b"><mi id="S3.F4.16.6.6.m6.1.1" xref="S3.F4.16.6.6.m6.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S3.F4.16.6.6.m6.1c"><ci id="S3.F4.16.6.6.m6.1.1.cmml" xref="S3.F4.16.6.6.m6.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F4.16.6.6.m6.1d">i</annotation></semantics></math> is expanded to produce (d) the key vector <math id="S3.F4.17.7.7.m7.1" class="ltx_Math" alttext="k_{*}" display="inline"><semantics id="S3.F4.17.7.7.m7.1b"><msub id="S3.F4.17.7.7.m7.1.1" xref="S3.F4.17.7.7.m7.1.1.cmml"><mi id="S3.F4.17.7.7.m7.1.1.2" xref="S3.F4.17.7.7.m7.1.1.2.cmml">k</mi><mo id="S3.F4.17.7.7.m7.1.1.3" xref="S3.F4.17.7.7.m7.1.1.3.cmml">∗</mo></msub><annotation-xml encoding="MathML-Content" id="S3.F4.17.7.7.m7.1c"><apply id="S3.F4.17.7.7.m7.1.1.cmml" xref="S3.F4.17.7.7.m7.1.1"><csymbol cd="ambiguous" id="S3.F4.17.7.7.m7.1.1.1.cmml" xref="S3.F4.17.7.7.m7.1.1">subscript</csymbol><ci id="S3.F4.17.7.7.m7.1.1.2.cmml" xref="S3.F4.17.7.7.m7.1.1.2">𝑘</ci><times id="S3.F4.17.7.7.m7.1.1.3.cmml" xref="S3.F4.17.7.7.m7.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F4.17.7.7.m7.1d">k_{*}</annotation></semantics></math> for the subject. (e) To write new value vector <math id="S3.F4.18.8.8.m8.1" class="ltx_Math" alttext="v_{*}" display="inline"><semantics id="S3.F4.18.8.8.m8.1b"><msub id="S3.F4.18.8.8.m8.1.1" xref="S3.F4.18.8.8.m8.1.1.cmml"><mi id="S3.F4.18.8.8.m8.1.1.2" xref="S3.F4.18.8.8.m8.1.1.2.cmml">v</mi><mo id="S3.F4.18.8.8.m8.1.1.3" xref="S3.F4.18.8.8.m8.1.1.3.cmml">∗</mo></msub><annotation-xml encoding="MathML-Content" id="S3.F4.18.8.8.m8.1c"><apply id="S3.F4.18.8.8.m8.1.1.cmml" xref="S3.F4.18.8.8.m8.1.1"><csymbol cd="ambiguous" id="S3.F4.18.8.8.m8.1.1.1.cmml" xref="S3.F4.18.8.8.m8.1.1">subscript</csymbol><ci id="S3.F4.18.8.8.m8.1.1.2.cmml" xref="S3.F4.18.8.8.m8.1.1.2">𝑣</ci><times id="S3.F4.18.8.8.m8.1.1.3.cmml" xref="S3.F4.18.8.8.m8.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F4.18.8.8.m8.1d">v_{*}</annotation></semantics></math> into the layer, (f) we calculate a rank-one update <math id="S3.F4.19.9.9.m9.1" class="ltx_Math" alttext="\Lambda(C^{-1}k_{*})^{T}" display="inline"><semantics id="S3.F4.19.9.9.m9.1b"><mrow id="S3.F4.19.9.9.m9.1.1" xref="S3.F4.19.9.9.m9.1.1.cmml"><mi mathvariant="normal" id="S3.F4.19.9.9.m9.1.1.3" xref="S3.F4.19.9.9.m9.1.1.3.cmml">Λ</mi><mo lspace="0em" rspace="0em" id="S3.F4.19.9.9.m9.1.1.2" xref="S3.F4.19.9.9.m9.1.1.2.cmml">​</mo><msup id="S3.F4.19.9.9.m9.1.1.1" xref="S3.F4.19.9.9.m9.1.1.1.cmml"><mrow id="S3.F4.19.9.9.m9.1.1.1.1.1" xref="S3.F4.19.9.9.m9.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.F4.19.9.9.m9.1.1.1.1.1.2" xref="S3.F4.19.9.9.m9.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.F4.19.9.9.m9.1.1.1.1.1.1" xref="S3.F4.19.9.9.m9.1.1.1.1.1.1.cmml"><msup id="S3.F4.19.9.9.m9.1.1.1.1.1.1.2" xref="S3.F4.19.9.9.m9.1.1.1.1.1.1.2.cmml"><mi id="S3.F4.19.9.9.m9.1.1.1.1.1.1.2.2" xref="S3.F4.19.9.9.m9.1.1.1.1.1.1.2.2.cmml">C</mi><mrow id="S3.F4.19.9.9.m9.1.1.1.1.1.1.2.3" xref="S3.F4.19.9.9.m9.1.1.1.1.1.1.2.3.cmml"><mo id="S3.F4.19.9.9.m9.1.1.1.1.1.1.2.3b" xref="S3.F4.19.9.9.m9.1.1.1.1.1.1.2.3.cmml">−</mo><mn id="S3.F4.19.9.9.m9.1.1.1.1.1.1.2.3.2" xref="S3.F4.19.9.9.m9.1.1.1.1.1.1.2.3.2.cmml">1</mn></mrow></msup><mo lspace="0em" rspace="0em" id="S3.F4.19.9.9.m9.1.1.1.1.1.1.1" xref="S3.F4.19.9.9.m9.1.1.1.1.1.1.1.cmml">​</mo><msub id="S3.F4.19.9.9.m9.1.1.1.1.1.1.3" xref="S3.F4.19.9.9.m9.1.1.1.1.1.1.3.cmml"><mi id="S3.F4.19.9.9.m9.1.1.1.1.1.1.3.2" xref="S3.F4.19.9.9.m9.1.1.1.1.1.1.3.2.cmml">k</mi><mo id="S3.F4.19.9.9.m9.1.1.1.1.1.1.3.3" xref="S3.F4.19.9.9.m9.1.1.1.1.1.1.3.3.cmml">∗</mo></msub></mrow><mo stretchy="false" id="S3.F4.19.9.9.m9.1.1.1.1.1.3" xref="S3.F4.19.9.9.m9.1.1.1.1.1.1.cmml">)</mo></mrow><mi id="S3.F4.19.9.9.m9.1.1.1.3" xref="S3.F4.19.9.9.m9.1.1.1.3.cmml">T</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.F4.19.9.9.m9.1c"><apply id="S3.F4.19.9.9.m9.1.1.cmml" xref="S3.F4.19.9.9.m9.1.1"><times id="S3.F4.19.9.9.m9.1.1.2.cmml" xref="S3.F4.19.9.9.m9.1.1.2"></times><ci id="S3.F4.19.9.9.m9.1.1.3.cmml" xref="S3.F4.19.9.9.m9.1.1.3">Λ</ci><apply id="S3.F4.19.9.9.m9.1.1.1.cmml" xref="S3.F4.19.9.9.m9.1.1.1"><csymbol cd="ambiguous" id="S3.F4.19.9.9.m9.1.1.1.2.cmml" xref="S3.F4.19.9.9.m9.1.1.1">superscript</csymbol><apply id="S3.F4.19.9.9.m9.1.1.1.1.1.1.cmml" xref="S3.F4.19.9.9.m9.1.1.1.1.1"><times id="S3.F4.19.9.9.m9.1.1.1.1.1.1.1.cmml" xref="S3.F4.19.9.9.m9.1.1.1.1.1.1.1"></times><apply id="S3.F4.19.9.9.m9.1.1.1.1.1.1.2.cmml" xref="S3.F4.19.9.9.m9.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.F4.19.9.9.m9.1.1.1.1.1.1.2.1.cmml" xref="S3.F4.19.9.9.m9.1.1.1.1.1.1.2">superscript</csymbol><ci id="S3.F4.19.9.9.m9.1.1.1.1.1.1.2.2.cmml" xref="S3.F4.19.9.9.m9.1.1.1.1.1.1.2.2">𝐶</ci><apply id="S3.F4.19.9.9.m9.1.1.1.1.1.1.2.3.cmml" xref="S3.F4.19.9.9.m9.1.1.1.1.1.1.2.3"><minus id="S3.F4.19.9.9.m9.1.1.1.1.1.1.2.3.1.cmml" xref="S3.F4.19.9.9.m9.1.1.1.1.1.1.2.3"></minus><cn type="integer" id="S3.F4.19.9.9.m9.1.1.1.1.1.1.2.3.2.cmml" xref="S3.F4.19.9.9.m9.1.1.1.1.1.1.2.3.2">1</cn></apply></apply><apply id="S3.F4.19.9.9.m9.1.1.1.1.1.1.3.cmml" xref="S3.F4.19.9.9.m9.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.F4.19.9.9.m9.1.1.1.1.1.1.3.1.cmml" xref="S3.F4.19.9.9.m9.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.F4.19.9.9.m9.1.1.1.1.1.1.3.2.cmml" xref="S3.F4.19.9.9.m9.1.1.1.1.1.1.3.2">𝑘</ci><times id="S3.F4.19.9.9.m9.1.1.1.1.1.1.3.3.cmml" xref="S3.F4.19.9.9.m9.1.1.1.1.1.1.3.3"></times></apply></apply><ci id="S3.F4.19.9.9.m9.1.1.1.3.cmml" xref="S3.F4.19.9.9.m9.1.1.1.3">𝑇</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F4.19.9.9.m9.1d">\Lambda(C^{-1}k_{*})^{T}</annotation></semantics></math> to cause <math id="S3.F4.20.10.10.m10.1" class="ltx_Math" alttext="\smash{\hat{W}^{(l)}_{proj}k_{*}=v_{*}}" display="inline"><semantics id="S3.F4.20.10.10.m10.1b"><mrow id="S3.F4.20.10.10.m10.1.2" xref="S3.F4.20.10.10.m10.1.2.cmml"><mrow id="S3.F4.20.10.10.m10.1.2.2" xref="S3.F4.20.10.10.m10.1.2.2.cmml"><msubsup id="S3.F4.20.10.10.m10.1.2.2.2" xref="S3.F4.20.10.10.m10.1.2.2.2.cmml"><mover accent="true" id="S3.F4.20.10.10.m10.1.2.2.2.2.2" xref="S3.F4.20.10.10.m10.1.2.2.2.2.2.cmml"><mi id="S3.F4.20.10.10.m10.1.2.2.2.2.2.2" xref="S3.F4.20.10.10.m10.1.2.2.2.2.2.2.cmml">W</mi><mo id="S3.F4.20.10.10.m10.1.2.2.2.2.2.1" xref="S3.F4.20.10.10.m10.1.2.2.2.2.2.1.cmml">^</mo></mover><mrow id="S3.F4.20.10.10.m10.1.2.2.2.3" xref="S3.F4.20.10.10.m10.1.2.2.2.3.cmml"><mi id="S3.F4.20.10.10.m10.1.2.2.2.3.2" xref="S3.F4.20.10.10.m10.1.2.2.2.3.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="S3.F4.20.10.10.m10.1.2.2.2.3.1" xref="S3.F4.20.10.10.m10.1.2.2.2.3.1.cmml">​</mo><mi id="S3.F4.20.10.10.m10.1.2.2.2.3.3" xref="S3.F4.20.10.10.m10.1.2.2.2.3.3.cmml">r</mi><mo lspace="0em" rspace="0em" id="S3.F4.20.10.10.m10.1.2.2.2.3.1b" xref="S3.F4.20.10.10.m10.1.2.2.2.3.1.cmml">​</mo><mi id="S3.F4.20.10.10.m10.1.2.2.2.3.4" xref="S3.F4.20.10.10.m10.1.2.2.2.3.4.cmml">o</mi><mo lspace="0em" rspace="0em" id="S3.F4.20.10.10.m10.1.2.2.2.3.1c" xref="S3.F4.20.10.10.m10.1.2.2.2.3.1.cmml">​</mo><mi id="S3.F4.20.10.10.m10.1.2.2.2.3.5" xref="S3.F4.20.10.10.m10.1.2.2.2.3.5.cmml">j</mi></mrow><mrow id="S3.F4.20.10.10.m10.1.1.1.3" xref="S3.F4.20.10.10.m10.1.2.2.2.cmml"><mo stretchy="false" id="S3.F4.20.10.10.m10.1.1.1.3.1" xref="S3.F4.20.10.10.m10.1.2.2.2.cmml">(</mo><mi id="S3.F4.20.10.10.m10.1.1.1.1" xref="S3.F4.20.10.10.m10.1.1.1.1.cmml">l</mi><mo stretchy="false" id="S3.F4.20.10.10.m10.1.1.1.3.2" xref="S3.F4.20.10.10.m10.1.2.2.2.cmml">)</mo></mrow></msubsup><mo lspace="0em" rspace="0em" id="S3.F4.20.10.10.m10.1.2.2.1" xref="S3.F4.20.10.10.m10.1.2.2.1.cmml">​</mo><msub id="S3.F4.20.10.10.m10.1.2.2.3" xref="S3.F4.20.10.10.m10.1.2.2.3.cmml"><mi id="S3.F4.20.10.10.m10.1.2.2.3.2" xref="S3.F4.20.10.10.m10.1.2.2.3.2.cmml">k</mi><mo id="S3.F4.20.10.10.m10.1.2.2.3.3" xref="S3.F4.20.10.10.m10.1.2.2.3.3.cmml">∗</mo></msub></mrow><mo id="S3.F4.20.10.10.m10.1.2.1" xref="S3.F4.20.10.10.m10.1.2.1.cmml">=</mo><msub id="S3.F4.20.10.10.m10.1.2.3" xref="S3.F4.20.10.10.m10.1.2.3.cmml"><mi id="S3.F4.20.10.10.m10.1.2.3.2" xref="S3.F4.20.10.10.m10.1.2.3.2.cmml">v</mi><mo id="S3.F4.20.10.10.m10.1.2.3.3" xref="S3.F4.20.10.10.m10.1.2.3.3.cmml">∗</mo></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.F4.20.10.10.m10.1c"><apply id="S3.F4.20.10.10.m10.1.2.cmml" xref="S3.F4.20.10.10.m10.1.2"><eq id="S3.F4.20.10.10.m10.1.2.1.cmml" xref="S3.F4.20.10.10.m10.1.2.1"></eq><apply id="S3.F4.20.10.10.m10.1.2.2.cmml" xref="S3.F4.20.10.10.m10.1.2.2"><times id="S3.F4.20.10.10.m10.1.2.2.1.cmml" xref="S3.F4.20.10.10.m10.1.2.2.1"></times><apply id="S3.F4.20.10.10.m10.1.2.2.2.cmml" xref="S3.F4.20.10.10.m10.1.2.2.2"><csymbol cd="ambiguous" id="S3.F4.20.10.10.m10.1.2.2.2.1.cmml" xref="S3.F4.20.10.10.m10.1.2.2.2">subscript</csymbol><apply id="S3.F4.20.10.10.m10.1.2.2.2.2.cmml" xref="S3.F4.20.10.10.m10.1.2.2.2"><csymbol cd="ambiguous" id="S3.F4.20.10.10.m10.1.2.2.2.2.1.cmml" xref="S3.F4.20.10.10.m10.1.2.2.2">superscript</csymbol><apply id="S3.F4.20.10.10.m10.1.2.2.2.2.2.cmml" xref="S3.F4.20.10.10.m10.1.2.2.2.2.2"><ci id="S3.F4.20.10.10.m10.1.2.2.2.2.2.1.cmml" xref="S3.F4.20.10.10.m10.1.2.2.2.2.2.1">^</ci><ci id="S3.F4.20.10.10.m10.1.2.2.2.2.2.2.cmml" xref="S3.F4.20.10.10.m10.1.2.2.2.2.2.2">𝑊</ci></apply><ci id="S3.F4.20.10.10.m10.1.1.1.1.cmml" xref="S3.F4.20.10.10.m10.1.1.1.1">𝑙</ci></apply><apply id="S3.F4.20.10.10.m10.1.2.2.2.3.cmml" xref="S3.F4.20.10.10.m10.1.2.2.2.3"><times id="S3.F4.20.10.10.m10.1.2.2.2.3.1.cmml" xref="S3.F4.20.10.10.m10.1.2.2.2.3.1"></times><ci id="S3.F4.20.10.10.m10.1.2.2.2.3.2.cmml" xref="S3.F4.20.10.10.m10.1.2.2.2.3.2">𝑝</ci><ci id="S3.F4.20.10.10.m10.1.2.2.2.3.3.cmml" xref="S3.F4.20.10.10.m10.1.2.2.2.3.3">𝑟</ci><ci id="S3.F4.20.10.10.m10.1.2.2.2.3.4.cmml" xref="S3.F4.20.10.10.m10.1.2.2.2.3.4">𝑜</ci><ci id="S3.F4.20.10.10.m10.1.2.2.2.3.5.cmml" xref="S3.F4.20.10.10.m10.1.2.2.2.3.5">𝑗</ci></apply></apply><apply id="S3.F4.20.10.10.m10.1.2.2.3.cmml" xref="S3.F4.20.10.10.m10.1.2.2.3"><csymbol cd="ambiguous" id="S3.F4.20.10.10.m10.1.2.2.3.1.cmml" xref="S3.F4.20.10.10.m10.1.2.2.3">subscript</csymbol><ci id="S3.F4.20.10.10.m10.1.2.2.3.2.cmml" xref="S3.F4.20.10.10.m10.1.2.2.3.2">𝑘</ci><times id="S3.F4.20.10.10.m10.1.2.2.3.3.cmml" xref="S3.F4.20.10.10.m10.1.2.2.3.3"></times></apply></apply><apply id="S3.F4.20.10.10.m10.1.2.3.cmml" xref="S3.F4.20.10.10.m10.1.2.3"><csymbol cd="ambiguous" id="S3.F4.20.10.10.m10.1.2.3.1.cmml" xref="S3.F4.20.10.10.m10.1.2.3">subscript</csymbol><ci id="S3.F4.20.10.10.m10.1.2.3.2.cmml" xref="S3.F4.20.10.10.m10.1.2.3.2">𝑣</ci><times id="S3.F4.20.10.10.m10.1.2.3.3.cmml" xref="S3.F4.20.10.10.m10.1.2.3.3"></times></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F4.20.10.10.m10.1d">\smash{\hat{W}^{(l)}_{proj}k_{*}=v_{*}}</annotation></semantics></math> while minimizing interference with other memories stored in the layer.</span></span></figcaption>
</figure>
<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.7" class="ltx_p">We view <math id="S3.SS1.p1.1.m1.1" class="ltx_Math" alttext="\smash{W^{(l)}_{proj}}" display="inline"><semantics id="S3.SS1.p1.1.m1.1a"><msubsup id="S3.SS1.p1.1.m1.1.2" xref="S3.SS1.p1.1.m1.1.2.cmml"><mi id="S3.SS1.p1.1.m1.1.2.2.2" xref="S3.SS1.p1.1.m1.1.2.2.2.cmml">W</mi><mrow id="S3.SS1.p1.1.m1.1.2.3" xref="S3.SS1.p1.1.m1.1.2.3.cmml"><mi id="S3.SS1.p1.1.m1.1.2.3.2" xref="S3.SS1.p1.1.m1.1.2.3.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.1.m1.1.2.3.1" xref="S3.SS1.p1.1.m1.1.2.3.1.cmml">​</mo><mi id="S3.SS1.p1.1.m1.1.2.3.3" xref="S3.SS1.p1.1.m1.1.2.3.3.cmml">r</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.1.m1.1.2.3.1a" xref="S3.SS1.p1.1.m1.1.2.3.1.cmml">​</mo><mi id="S3.SS1.p1.1.m1.1.2.3.4" xref="S3.SS1.p1.1.m1.1.2.3.4.cmml">o</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.1.m1.1.2.3.1b" xref="S3.SS1.p1.1.m1.1.2.3.1.cmml">​</mo><mi id="S3.SS1.p1.1.m1.1.2.3.5" xref="S3.SS1.p1.1.m1.1.2.3.5.cmml">j</mi></mrow><mrow id="S3.SS1.p1.1.m1.1.1.1.3" xref="S3.SS1.p1.1.m1.1.2.cmml"><mo stretchy="false" id="S3.SS1.p1.1.m1.1.1.1.3.1" xref="S3.SS1.p1.1.m1.1.2.cmml">(</mo><mi id="S3.SS1.p1.1.m1.1.1.1.1" xref="S3.SS1.p1.1.m1.1.1.1.1.cmml">l</mi><mo stretchy="false" id="S3.SS1.p1.1.m1.1.1.1.3.2" xref="S3.SS1.p1.1.m1.1.2.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.1.m1.1b"><apply id="S3.SS1.p1.1.m1.1.2.cmml" xref="S3.SS1.p1.1.m1.1.2"><csymbol cd="ambiguous" id="S3.SS1.p1.1.m1.1.2.1.cmml" xref="S3.SS1.p1.1.m1.1.2">subscript</csymbol><apply id="S3.SS1.p1.1.m1.1.2.2.cmml" xref="S3.SS1.p1.1.m1.1.2"><csymbol cd="ambiguous" id="S3.SS1.p1.1.m1.1.2.2.1.cmml" xref="S3.SS1.p1.1.m1.1.2">superscript</csymbol><ci id="S3.SS1.p1.1.m1.1.2.2.2.cmml" xref="S3.SS1.p1.1.m1.1.2.2.2">𝑊</ci><ci id="S3.SS1.p1.1.m1.1.1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1.1.1">𝑙</ci></apply><apply id="S3.SS1.p1.1.m1.1.2.3.cmml" xref="S3.SS1.p1.1.m1.1.2.3"><times id="S3.SS1.p1.1.m1.1.2.3.1.cmml" xref="S3.SS1.p1.1.m1.1.2.3.1"></times><ci id="S3.SS1.p1.1.m1.1.2.3.2.cmml" xref="S3.SS1.p1.1.m1.1.2.3.2">𝑝</ci><ci id="S3.SS1.p1.1.m1.1.2.3.3.cmml" xref="S3.SS1.p1.1.m1.1.2.3.3">𝑟</ci><ci id="S3.SS1.p1.1.m1.1.2.3.4.cmml" xref="S3.SS1.p1.1.m1.1.2.3.4">𝑜</ci><ci id="S3.SS1.p1.1.m1.1.2.3.5.cmml" xref="S3.SS1.p1.1.m1.1.2.3.5">𝑗</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.1.m1.1c">\smash{W^{(l)}_{proj}}</annotation></semantics></math> as a linear associative memory <cite class="ltx_cite ltx_citemacro_citep">(Kohonen, <a href="#bib.bib23" title="" class="ltx_ref">1972</a>; Anderson, <a href="#bib.bib2" title="" class="ltx_ref">1972</a>)</cite>. This perspective observes that any linear operation <math id="S3.SS1.p1.2.m2.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S3.SS1.p1.2.m2.1a"><mi id="S3.SS1.p1.2.m2.1.1" xref="S3.SS1.p1.2.m2.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.2.m2.1b"><ci id="S3.SS1.p1.2.m2.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.2.m2.1c">W</annotation></semantics></math> can operate as a key–value store for a set of vector keys <math id="S3.SS1.p1.3.m3.1" class="ltx_Math" alttext="K=[k_{1}\mid k_{2}\mid\dots]" display="inline"><semantics id="S3.SS1.p1.3.m3.1a"><mrow id="S3.SS1.p1.3.m3.1.1" xref="S3.SS1.p1.3.m3.1.1.cmml"><mi id="S3.SS1.p1.3.m3.1.1.3" xref="S3.SS1.p1.3.m3.1.1.3.cmml">K</mi><mo id="S3.SS1.p1.3.m3.1.1.2" xref="S3.SS1.p1.3.m3.1.1.2.cmml">=</mo><mrow id="S3.SS1.p1.3.m3.1.1.1.1" xref="S3.SS1.p1.3.m3.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS1.p1.3.m3.1.1.1.1.2" xref="S3.SS1.p1.3.m3.1.1.1.2.1.cmml">[</mo><mrow id="S3.SS1.p1.3.m3.1.1.1.1.1" xref="S3.SS1.p1.3.m3.1.1.1.1.1.cmml"><msub id="S3.SS1.p1.3.m3.1.1.1.1.1.3" xref="S3.SS1.p1.3.m3.1.1.1.1.1.3.cmml"><mi id="S3.SS1.p1.3.m3.1.1.1.1.1.3.2" xref="S3.SS1.p1.3.m3.1.1.1.1.1.3.2.cmml">k</mi><mn id="S3.SS1.p1.3.m3.1.1.1.1.1.3.3" xref="S3.SS1.p1.3.m3.1.1.1.1.1.3.3.cmml">1</mn></msub><mo lspace="0em" rspace="0em" id="S3.SS1.p1.3.m3.1.1.1.1.1.2" xref="S3.SS1.p1.3.m3.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.2.cmml"><mo fence="true" rspace="0em" id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.2" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.2.1.cmml">∣</mo><msub id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.cmml"><mi id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.2" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.2.cmml">k</mi><mn id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.3" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.3.cmml">2</mn></msub><mo fence="true" lspace="0em" id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.3" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.2.1.cmml">∣</mo></mrow><mo lspace="0em" rspace="0em" id="S3.SS1.p1.3.m3.1.1.1.1.1.2a" xref="S3.SS1.p1.3.m3.1.1.1.1.1.2.cmml">​</mo><mi mathvariant="normal" id="S3.SS1.p1.3.m3.1.1.1.1.1.4" xref="S3.SS1.p1.3.m3.1.1.1.1.1.4.cmml">…</mi></mrow><mo stretchy="false" id="S3.SS1.p1.3.m3.1.1.1.1.3" xref="S3.SS1.p1.3.m3.1.1.1.2.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.3.m3.1b"><apply id="S3.SS1.p1.3.m3.1.1.cmml" xref="S3.SS1.p1.3.m3.1.1"><eq id="S3.SS1.p1.3.m3.1.1.2.cmml" xref="S3.SS1.p1.3.m3.1.1.2"></eq><ci id="S3.SS1.p1.3.m3.1.1.3.cmml" xref="S3.SS1.p1.3.m3.1.1.3">𝐾</ci><apply id="S3.SS1.p1.3.m3.1.1.1.2.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1"><csymbol cd="latexml" id="S3.SS1.p1.3.m3.1.1.1.2.1.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.2">delimited-[]</csymbol><apply id="S3.SS1.p1.3.m3.1.1.1.1.1.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1"><times id="S3.SS1.p1.3.m3.1.1.1.1.1.2.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.2"></times><apply id="S3.SS1.p1.3.m3.1.1.1.1.1.3.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p1.3.m3.1.1.1.1.1.3.1.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.3">subscript</csymbol><ci id="S3.SS1.p1.3.m3.1.1.1.1.1.3.2.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.3.2">𝑘</ci><cn type="integer" id="S3.SS1.p1.3.m3.1.1.1.1.1.3.3.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.3.3">1</cn></apply><apply id="S3.SS1.p1.3.m3.1.1.1.1.1.1.2.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1"><csymbol cd="latexml" id="S3.SS1.p1.3.m3.1.1.1.1.1.1.2.1.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.2">delimited-∣∣</csymbol><apply id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.2.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.2">𝑘</ci><cn type="integer" id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.3.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.3">2</cn></apply></apply><ci id="S3.SS1.p1.3.m3.1.1.1.1.1.4.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.4">…</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.3.m3.1c">K=[k_{1}\mid k_{2}\mid\dots]</annotation></semantics></math> and corresponding vector values <math id="S3.SS1.p1.4.m4.1" class="ltx_Math" alttext="V=[v_{1}\mid v_{2}\mid\dots]" display="inline"><semantics id="S3.SS1.p1.4.m4.1a"><mrow id="S3.SS1.p1.4.m4.1.1" xref="S3.SS1.p1.4.m4.1.1.cmml"><mi id="S3.SS1.p1.4.m4.1.1.3" xref="S3.SS1.p1.4.m4.1.1.3.cmml">V</mi><mo id="S3.SS1.p1.4.m4.1.1.2" xref="S3.SS1.p1.4.m4.1.1.2.cmml">=</mo><mrow id="S3.SS1.p1.4.m4.1.1.1.1" xref="S3.SS1.p1.4.m4.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS1.p1.4.m4.1.1.1.1.2" xref="S3.SS1.p1.4.m4.1.1.1.2.1.cmml">[</mo><mrow id="S3.SS1.p1.4.m4.1.1.1.1.1" xref="S3.SS1.p1.4.m4.1.1.1.1.1.cmml"><msub id="S3.SS1.p1.4.m4.1.1.1.1.1.3" xref="S3.SS1.p1.4.m4.1.1.1.1.1.3.cmml"><mi id="S3.SS1.p1.4.m4.1.1.1.1.1.3.2" xref="S3.SS1.p1.4.m4.1.1.1.1.1.3.2.cmml">v</mi><mn id="S3.SS1.p1.4.m4.1.1.1.1.1.3.3" xref="S3.SS1.p1.4.m4.1.1.1.1.1.3.3.cmml">1</mn></msub><mo lspace="0em" rspace="0em" id="S3.SS1.p1.4.m4.1.1.1.1.1.2" xref="S3.SS1.p1.4.m4.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.SS1.p1.4.m4.1.1.1.1.1.1.1" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.2.cmml"><mo fence="true" rspace="0em" id="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.2" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.2.1.cmml">∣</mo><msub id="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.1" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.1.cmml"><mi id="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.1.2" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.1.2.cmml">v</mi><mn id="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.1.3" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.1.3.cmml">2</mn></msub><mo fence="true" lspace="0em" id="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.3" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.2.1.cmml">∣</mo></mrow><mo lspace="0em" rspace="0em" id="S3.SS1.p1.4.m4.1.1.1.1.1.2a" xref="S3.SS1.p1.4.m4.1.1.1.1.1.2.cmml">​</mo><mi mathvariant="normal" id="S3.SS1.p1.4.m4.1.1.1.1.1.4" xref="S3.SS1.p1.4.m4.1.1.1.1.1.4.cmml">…</mi></mrow><mo stretchy="false" id="S3.SS1.p1.4.m4.1.1.1.1.3" xref="S3.SS1.p1.4.m4.1.1.1.2.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.4.m4.1b"><apply id="S3.SS1.p1.4.m4.1.1.cmml" xref="S3.SS1.p1.4.m4.1.1"><eq id="S3.SS1.p1.4.m4.1.1.2.cmml" xref="S3.SS1.p1.4.m4.1.1.2"></eq><ci id="S3.SS1.p1.4.m4.1.1.3.cmml" xref="S3.SS1.p1.4.m4.1.1.3">𝑉</ci><apply id="S3.SS1.p1.4.m4.1.1.1.2.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1"><csymbol cd="latexml" id="S3.SS1.p1.4.m4.1.1.1.2.1.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.2">delimited-[]</csymbol><apply id="S3.SS1.p1.4.m4.1.1.1.1.1.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.1"><times id="S3.SS1.p1.4.m4.1.1.1.1.1.2.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.1.2"></times><apply id="S3.SS1.p1.4.m4.1.1.1.1.1.3.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p1.4.m4.1.1.1.1.1.3.1.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.1.3">subscript</csymbol><ci id="S3.SS1.p1.4.m4.1.1.1.1.1.3.2.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.1.3.2">𝑣</ci><cn type="integer" id="S3.SS1.p1.4.m4.1.1.1.1.1.3.3.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.1.3.3">1</cn></apply><apply id="S3.SS1.p1.4.m4.1.1.1.1.1.1.2.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.1"><csymbol cd="latexml" id="S3.SS1.p1.4.m4.1.1.1.1.1.1.2.1.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.2">delimited-∣∣</csymbol><apply id="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.1.2.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.1.2">𝑣</ci><cn type="integer" id="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.1.3.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.1.3">2</cn></apply></apply><ci id="S3.SS1.p1.4.m4.1.1.1.1.1.4.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.1.4">…</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.4.m4.1c">V=[v_{1}\mid v_{2}\mid\dots]</annotation></semantics></math>, by solving <math id="S3.SS1.p1.5.m5.1" class="ltx_Math" alttext="WK\approx V" display="inline"><semantics id="S3.SS1.p1.5.m5.1a"><mrow id="S3.SS1.p1.5.m5.1.1" xref="S3.SS1.p1.5.m5.1.1.cmml"><mrow id="S3.SS1.p1.5.m5.1.1.2" xref="S3.SS1.p1.5.m5.1.1.2.cmml"><mi id="S3.SS1.p1.5.m5.1.1.2.2" xref="S3.SS1.p1.5.m5.1.1.2.2.cmml">W</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.5.m5.1.1.2.1" xref="S3.SS1.p1.5.m5.1.1.2.1.cmml">​</mo><mi id="S3.SS1.p1.5.m5.1.1.2.3" xref="S3.SS1.p1.5.m5.1.1.2.3.cmml">K</mi></mrow><mo id="S3.SS1.p1.5.m5.1.1.1" xref="S3.SS1.p1.5.m5.1.1.1.cmml">≈</mo><mi id="S3.SS1.p1.5.m5.1.1.3" xref="S3.SS1.p1.5.m5.1.1.3.cmml">V</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.5.m5.1b"><apply id="S3.SS1.p1.5.m5.1.1.cmml" xref="S3.SS1.p1.5.m5.1.1"><approx id="S3.SS1.p1.5.m5.1.1.1.cmml" xref="S3.SS1.p1.5.m5.1.1.1"></approx><apply id="S3.SS1.p1.5.m5.1.1.2.cmml" xref="S3.SS1.p1.5.m5.1.1.2"><times id="S3.SS1.p1.5.m5.1.1.2.1.cmml" xref="S3.SS1.p1.5.m5.1.1.2.1"></times><ci id="S3.SS1.p1.5.m5.1.1.2.2.cmml" xref="S3.SS1.p1.5.m5.1.1.2.2">𝑊</ci><ci id="S3.SS1.p1.5.m5.1.1.2.3.cmml" xref="S3.SS1.p1.5.m5.1.1.2.3">𝐾</ci></apply><ci id="S3.SS1.p1.5.m5.1.1.3.cmml" xref="S3.SS1.p1.5.m5.1.1.3">𝑉</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.5.m5.1c">WK\approx V</annotation></semantics></math>, whose squared error is minimized using the Moore-Penrose pseudoinverse: <math id="S3.SS1.p1.6.m6.1" class="ltx_Math" alttext="W=VK^{+}" display="inline"><semantics id="S3.SS1.p1.6.m6.1a"><mrow id="S3.SS1.p1.6.m6.1.1" xref="S3.SS1.p1.6.m6.1.1.cmml"><mi id="S3.SS1.p1.6.m6.1.1.2" xref="S3.SS1.p1.6.m6.1.1.2.cmml">W</mi><mo id="S3.SS1.p1.6.m6.1.1.1" xref="S3.SS1.p1.6.m6.1.1.1.cmml">=</mo><mrow id="S3.SS1.p1.6.m6.1.1.3" xref="S3.SS1.p1.6.m6.1.1.3.cmml"><mi id="S3.SS1.p1.6.m6.1.1.3.2" xref="S3.SS1.p1.6.m6.1.1.3.2.cmml">V</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.6.m6.1.1.3.1" xref="S3.SS1.p1.6.m6.1.1.3.1.cmml">​</mo><msup id="S3.SS1.p1.6.m6.1.1.3.3" xref="S3.SS1.p1.6.m6.1.1.3.3.cmml"><mi id="S3.SS1.p1.6.m6.1.1.3.3.2" xref="S3.SS1.p1.6.m6.1.1.3.3.2.cmml">K</mi><mo id="S3.SS1.p1.6.m6.1.1.3.3.3" xref="S3.SS1.p1.6.m6.1.1.3.3.3.cmml">+</mo></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.6.m6.1b"><apply id="S3.SS1.p1.6.m6.1.1.cmml" xref="S3.SS1.p1.6.m6.1.1"><eq id="S3.SS1.p1.6.m6.1.1.1.cmml" xref="S3.SS1.p1.6.m6.1.1.1"></eq><ci id="S3.SS1.p1.6.m6.1.1.2.cmml" xref="S3.SS1.p1.6.m6.1.1.2">𝑊</ci><apply id="S3.SS1.p1.6.m6.1.1.3.cmml" xref="S3.SS1.p1.6.m6.1.1.3"><times id="S3.SS1.p1.6.m6.1.1.3.1.cmml" xref="S3.SS1.p1.6.m6.1.1.3.1"></times><ci id="S3.SS1.p1.6.m6.1.1.3.2.cmml" xref="S3.SS1.p1.6.m6.1.1.3.2">𝑉</ci><apply id="S3.SS1.p1.6.m6.1.1.3.3.cmml" xref="S3.SS1.p1.6.m6.1.1.3.3"><csymbol cd="ambiguous" id="S3.SS1.p1.6.m6.1.1.3.3.1.cmml" xref="S3.SS1.p1.6.m6.1.1.3.3">superscript</csymbol><ci id="S3.SS1.p1.6.m6.1.1.3.3.2.cmml" xref="S3.SS1.p1.6.m6.1.1.3.3.2">𝐾</ci><plus id="S3.SS1.p1.6.m6.1.1.3.3.3.cmml" xref="S3.SS1.p1.6.m6.1.1.3.3.3"></plus></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.6.m6.1c">W=VK^{+}</annotation></semantics></math>.
<cite class="ltx_cite ltx_citemacro_citet">Bau et al. (<a href="#bib.bib3" title="" class="ltx_ref">2020</a>)</cite> observed that a new key–value pair <math id="S3.SS1.p1.7.m7.2" class="ltx_Math" alttext="(k_{*},v_{*})" display="inline"><semantics id="S3.SS1.p1.7.m7.2a"><mrow id="S3.SS1.p1.7.m7.2.2.2" xref="S3.SS1.p1.7.m7.2.2.3.cmml"><mo stretchy="false" id="S3.SS1.p1.7.m7.2.2.2.3" xref="S3.SS1.p1.7.m7.2.2.3.cmml">(</mo><msub id="S3.SS1.p1.7.m7.1.1.1.1" xref="S3.SS1.p1.7.m7.1.1.1.1.cmml"><mi id="S3.SS1.p1.7.m7.1.1.1.1.2" xref="S3.SS1.p1.7.m7.1.1.1.1.2.cmml">k</mi><mo id="S3.SS1.p1.7.m7.1.1.1.1.3" xref="S3.SS1.p1.7.m7.1.1.1.1.3.cmml">∗</mo></msub><mo id="S3.SS1.p1.7.m7.2.2.2.4" xref="S3.SS1.p1.7.m7.2.2.3.cmml">,</mo><msub id="S3.SS1.p1.7.m7.2.2.2.2" xref="S3.SS1.p1.7.m7.2.2.2.2.cmml"><mi id="S3.SS1.p1.7.m7.2.2.2.2.2" xref="S3.SS1.p1.7.m7.2.2.2.2.2.cmml">v</mi><mo id="S3.SS1.p1.7.m7.2.2.2.2.3" xref="S3.SS1.p1.7.m7.2.2.2.2.3.cmml">∗</mo></msub><mo stretchy="false" id="S3.SS1.p1.7.m7.2.2.2.5" xref="S3.SS1.p1.7.m7.2.2.3.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.7.m7.2b"><interval closure="open" id="S3.SS1.p1.7.m7.2.2.3.cmml" xref="S3.SS1.p1.7.m7.2.2.2"><apply id="S3.SS1.p1.7.m7.1.1.1.1.cmml" xref="S3.SS1.p1.7.m7.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.7.m7.1.1.1.1.1.cmml" xref="S3.SS1.p1.7.m7.1.1.1.1">subscript</csymbol><ci id="S3.SS1.p1.7.m7.1.1.1.1.2.cmml" xref="S3.SS1.p1.7.m7.1.1.1.1.2">𝑘</ci><times id="S3.SS1.p1.7.m7.1.1.1.1.3.cmml" xref="S3.SS1.p1.7.m7.1.1.1.1.3"></times></apply><apply id="S3.SS1.p1.7.m7.2.2.2.2.cmml" xref="S3.SS1.p1.7.m7.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS1.p1.7.m7.2.2.2.2.1.cmml" xref="S3.SS1.p1.7.m7.2.2.2.2">subscript</csymbol><ci id="S3.SS1.p1.7.m7.2.2.2.2.2.cmml" xref="S3.SS1.p1.7.m7.2.2.2.2.2">𝑣</ci><times id="S3.SS1.p1.7.m7.2.2.2.2.3.cmml" xref="S3.SS1.p1.7.m7.2.2.2.2.3"></times></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.7.m7.2c">(k_{*},v_{*})</annotation></semantics></math> can be inserted optimally into the memory by solving a constrained least-squares problem. In a convolutional network, <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib3" title="" class="ltx_ref">Bau et al.</a></cite> solve this using an optimization, but in a fully-connected layer, we can derive a closed form solution:</p>
<table id="A10.EGx2" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S3.E2"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E2.m1.1" class="ltx_math_unparsed" alttext="\displaystyle\text{minimize}\;\lVert\hat{W}K" display="inline"><semantics id="S3.E2.m1.1a"><mrow id="S3.E2.m1.1b"><mtext id="S3.E2.m1.1.1">minimize</mtext><mrow id="S3.E2.m1.1.2"><mo fence="true" rspace="0em" id="S3.E2.m1.1.2.1">∥</mo><mover accent="true" id="S3.E2.m1.1.2.2"><mi id="S3.E2.m1.1.2.2.2">W</mi><mo id="S3.E2.m1.1.2.2.1">^</mo></mover><mi id="S3.E2.m1.1.2.3">K</mi></mrow></mrow><annotation encoding="application/x-tex" id="S3.E2.m1.1c">\displaystyle\text{minimize}\;\lVert\hat{W}K</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E2.m2.1" class="ltx_math_unparsed" alttext="\displaystyle-V\rVert\;\text{such that}\;\hat{W}k_{*}=v_{*}\quad\text{by setting}\;\hat{W}=W+\Lambda(C^{-1}k_{*})^{T}." display="inline"><semantics id="S3.E2.m2.1a"><mrow id="S3.E2.m2.1b"><mo id="S3.E2.m2.1.1">−</mo><mi id="S3.E2.m2.1.2">V</mi><mo fence="true" lspace="0em" id="S3.E2.m2.1.3">∥</mo><mtext id="S3.E2.m2.1.4">such that</mtext><mover accent="true" id="S3.E2.m2.1.5"><mi id="S3.E2.m2.1.5.2">W</mi><mo id="S3.E2.m2.1.5.1">^</mo></mover><mi id="S3.E2.m2.1.6">k</mi><msub id="S3.E2.m2.1.7"><mi id="S3.E2.m2.1.7a"></mi><mo id="S3.E2.m2.1.7.1">∗</mo></msub><mo id="S3.E2.m2.1.8">=</mo><mi id="S3.E2.m2.1.9">v</mi><msub id="S3.E2.m2.1.10"><mi id="S3.E2.m2.1.10a"></mi><mo id="S3.E2.m2.1.10.1">∗</mo></msub><mspace width="1em" id="S3.E2.m2.1.11"></mspace><mtext id="S3.E2.m2.1.12">by setting</mtext><mover accent="true" id="S3.E2.m2.1.13"><mi id="S3.E2.m2.1.13.2">W</mi><mo id="S3.E2.m2.1.13.1">^</mo></mover><mo id="S3.E2.m2.1.14">=</mo><mi id="S3.E2.m2.1.15">W</mi><mo id="S3.E2.m2.1.16">+</mo><mi mathvariant="normal" id="S3.E2.m2.1.17">Λ</mi><mo stretchy="false" id="S3.E2.m2.1.18">(</mo><msup id="S3.E2.m2.1.19"><mi id="S3.E2.m2.1.19.2">C</mi><mrow id="S3.E2.m2.1.19.3"><mo id="S3.E2.m2.1.19.3a">−</mo><mn id="S3.E2.m2.1.19.3.2">1</mn></mrow></msup><msub id="S3.E2.m2.1.20"><mi id="S3.E2.m2.1.20.2">k</mi><mo id="S3.E2.m2.1.20.3">∗</mo></msub><mo stretchy="false" id="S3.E2.m2.1.21">)</mo><msup id="S3.E2.m2.1.22"><mi id="S3.E2.m2.1.22a"></mi><mi id="S3.E2.m2.1.22.1">T</mi></msup><mo lspace="0em" id="S3.E2.m2.1.23">.</mo></mrow><annotation encoding="application/x-tex" id="S3.E2.m2.1c">\displaystyle-V\rVert\;\text{such that}\;\hat{W}k_{*}=v_{*}\quad\text{by setting}\;\hat{W}=W+\Lambda(C^{-1}k_{*})^{T}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
<p id="S3.SS1.p1.14" class="ltx_p">Here <math id="S3.SS1.p1.8.m1.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S3.SS1.p1.8.m1.1a"><mi id="S3.SS1.p1.8.m1.1.1" xref="S3.SS1.p1.8.m1.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.8.m1.1b"><ci id="S3.SS1.p1.8.m1.1.1.cmml" xref="S3.SS1.p1.8.m1.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.8.m1.1c">W</annotation></semantics></math> is the original matrix, <math id="S3.SS1.p1.9.m2.1" class="ltx_Math" alttext="C=KK^{T}" display="inline"><semantics id="S3.SS1.p1.9.m2.1a"><mrow id="S3.SS1.p1.9.m2.1.1" xref="S3.SS1.p1.9.m2.1.1.cmml"><mi id="S3.SS1.p1.9.m2.1.1.2" xref="S3.SS1.p1.9.m2.1.1.2.cmml">C</mi><mo id="S3.SS1.p1.9.m2.1.1.1" xref="S3.SS1.p1.9.m2.1.1.1.cmml">=</mo><mrow id="S3.SS1.p1.9.m2.1.1.3" xref="S3.SS1.p1.9.m2.1.1.3.cmml"><mi id="S3.SS1.p1.9.m2.1.1.3.2" xref="S3.SS1.p1.9.m2.1.1.3.2.cmml">K</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.9.m2.1.1.3.1" xref="S3.SS1.p1.9.m2.1.1.3.1.cmml">​</mo><msup id="S3.SS1.p1.9.m2.1.1.3.3" xref="S3.SS1.p1.9.m2.1.1.3.3.cmml"><mi id="S3.SS1.p1.9.m2.1.1.3.3.2" xref="S3.SS1.p1.9.m2.1.1.3.3.2.cmml">K</mi><mi id="S3.SS1.p1.9.m2.1.1.3.3.3" xref="S3.SS1.p1.9.m2.1.1.3.3.3.cmml">T</mi></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.9.m2.1b"><apply id="S3.SS1.p1.9.m2.1.1.cmml" xref="S3.SS1.p1.9.m2.1.1"><eq id="S3.SS1.p1.9.m2.1.1.1.cmml" xref="S3.SS1.p1.9.m2.1.1.1"></eq><ci id="S3.SS1.p1.9.m2.1.1.2.cmml" xref="S3.SS1.p1.9.m2.1.1.2">𝐶</ci><apply id="S3.SS1.p1.9.m2.1.1.3.cmml" xref="S3.SS1.p1.9.m2.1.1.3"><times id="S3.SS1.p1.9.m2.1.1.3.1.cmml" xref="S3.SS1.p1.9.m2.1.1.3.1"></times><ci id="S3.SS1.p1.9.m2.1.1.3.2.cmml" xref="S3.SS1.p1.9.m2.1.1.3.2">𝐾</ci><apply id="S3.SS1.p1.9.m2.1.1.3.3.cmml" xref="S3.SS1.p1.9.m2.1.1.3.3"><csymbol cd="ambiguous" id="S3.SS1.p1.9.m2.1.1.3.3.1.cmml" xref="S3.SS1.p1.9.m2.1.1.3.3">superscript</csymbol><ci id="S3.SS1.p1.9.m2.1.1.3.3.2.cmml" xref="S3.SS1.p1.9.m2.1.1.3.3.2">𝐾</ci><ci id="S3.SS1.p1.9.m2.1.1.3.3.3.cmml" xref="S3.SS1.p1.9.m2.1.1.3.3.3">𝑇</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.9.m2.1c">C=KK^{T}</annotation></semantics></math> is a constant that we pre-cache by estimating the uncentered covariance of <math id="S3.SS1.p1.10.m3.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.SS1.p1.10.m3.1a"><mi id="S3.SS1.p1.10.m3.1.1" xref="S3.SS1.p1.10.m3.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.10.m3.1b"><ci id="S3.SS1.p1.10.m3.1.1.cmml" xref="S3.SS1.p1.10.m3.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.10.m3.1c">k</annotation></semantics></math> from a sample of Wikipedia text (Appendix <a href="#A5.SS5" title="E.5 [GPT-2 XL, GPT-J] Rank-One Model Editing (ROME) ‣ Appendix E Method Implementation Details ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">E.5</span></a>), and <math id="S3.SS1.p1.11.m4.2" class="ltx_Math" alttext="\Lambda=(v_{*}-Wk_{*})/(C^{-1}k_{*})^{T}k_{*}" display="inline"><semantics id="S3.SS1.p1.11.m4.2a"><mrow id="S3.SS1.p1.11.m4.2.2" xref="S3.SS1.p1.11.m4.2.2.cmml"><mi mathvariant="normal" id="S3.SS1.p1.11.m4.2.2.4" xref="S3.SS1.p1.11.m4.2.2.4.cmml">Λ</mi><mo id="S3.SS1.p1.11.m4.2.2.3" xref="S3.SS1.p1.11.m4.2.2.3.cmml">=</mo><mrow id="S3.SS1.p1.11.m4.2.2.2" xref="S3.SS1.p1.11.m4.2.2.2.cmml"><mrow id="S3.SS1.p1.11.m4.2.2.2.2" xref="S3.SS1.p1.11.m4.2.2.2.2.cmml"><mrow id="S3.SS1.p1.11.m4.1.1.1.1.1.1" xref="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p1.11.m4.1.1.1.1.1.1.2" xref="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS1.p1.11.m4.1.1.1.1.1.1.1" xref="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.cmml"><msub id="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.2" xref="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.2.cmml"><mi id="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.2.2" xref="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.2.2.cmml">v</mi><mo id="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.2.3" xref="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.2.3.cmml">∗</mo></msub><mo id="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.1" xref="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.1.cmml">−</mo><mrow id="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.3" xref="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.3.cmml"><mi id="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.3.2" xref="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.3.2.cmml">W</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.3.1" xref="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.3.1.cmml">​</mo><msub id="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.3.3" xref="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.3.3.cmml"><mi id="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.3.3.2" xref="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.3.3.2.cmml">k</mi><mo id="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.3.3.3" xref="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.3.3.3.cmml">∗</mo></msub></mrow></mrow><mo stretchy="false" id="S3.SS1.p1.11.m4.1.1.1.1.1.1.3" xref="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo id="S3.SS1.p1.11.m4.2.2.2.2.3" xref="S3.SS1.p1.11.m4.2.2.2.2.3.cmml">/</mo><msup id="S3.SS1.p1.11.m4.2.2.2.2.2" xref="S3.SS1.p1.11.m4.2.2.2.2.2.cmml"><mrow id="S3.SS1.p1.11.m4.2.2.2.2.2.1.1" xref="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.2" xref="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.cmml">(</mo><mrow id="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1" xref="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.cmml"><msup id="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.2" xref="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.2.cmml"><mi id="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.2.2" xref="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.2.2.cmml">C</mi><mrow id="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.2.3" xref="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.2.3.cmml"><mo id="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.2.3a" xref="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.2.3.cmml">−</mo><mn id="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.2.3.2" xref="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.2.3.2.cmml">1</mn></mrow></msup><mo lspace="0em" rspace="0em" id="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.1" xref="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.1.cmml">​</mo><msub id="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.3" xref="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.3.cmml"><mi id="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.3.2" xref="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.3.2.cmml">k</mi><mo id="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.3.3" xref="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.3.3.cmml">∗</mo></msub></mrow><mo stretchy="false" id="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.3" xref="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.cmml">)</mo></mrow><mi id="S3.SS1.p1.11.m4.2.2.2.2.2.3" xref="S3.SS1.p1.11.m4.2.2.2.2.2.3.cmml">T</mi></msup></mrow><mo lspace="0em" rspace="0em" id="S3.SS1.p1.11.m4.2.2.2.3" xref="S3.SS1.p1.11.m4.2.2.2.3.cmml">​</mo><msub id="S3.SS1.p1.11.m4.2.2.2.4" xref="S3.SS1.p1.11.m4.2.2.2.4.cmml"><mi id="S3.SS1.p1.11.m4.2.2.2.4.2" xref="S3.SS1.p1.11.m4.2.2.2.4.2.cmml">k</mi><mo id="S3.SS1.p1.11.m4.2.2.2.4.3" xref="S3.SS1.p1.11.m4.2.2.2.4.3.cmml">∗</mo></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.11.m4.2b"><apply id="S3.SS1.p1.11.m4.2.2.cmml" xref="S3.SS1.p1.11.m4.2.2"><eq id="S3.SS1.p1.11.m4.2.2.3.cmml" xref="S3.SS1.p1.11.m4.2.2.3"></eq><ci id="S3.SS1.p1.11.m4.2.2.4.cmml" xref="S3.SS1.p1.11.m4.2.2.4">Λ</ci><apply id="S3.SS1.p1.11.m4.2.2.2.cmml" xref="S3.SS1.p1.11.m4.2.2.2"><times id="S3.SS1.p1.11.m4.2.2.2.3.cmml" xref="S3.SS1.p1.11.m4.2.2.2.3"></times><apply id="S3.SS1.p1.11.m4.2.2.2.2.cmml" xref="S3.SS1.p1.11.m4.2.2.2.2"><divide id="S3.SS1.p1.11.m4.2.2.2.2.3.cmml" xref="S3.SS1.p1.11.m4.2.2.2.2.3"></divide><apply id="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p1.11.m4.1.1.1.1.1.1"><minus id="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.1"></minus><apply id="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.2.cmml" xref="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.2.1.cmml" xref="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.2.2.cmml" xref="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.2.2">𝑣</ci><times id="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.2.3.cmml" xref="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.2.3"></times></apply><apply id="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.3.cmml" xref="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.3"><times id="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.3.1.cmml" xref="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.3.1"></times><ci id="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.3.2.cmml" xref="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.3.2">𝑊</ci><apply id="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.3.3.cmml" xref="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.3.3.1.cmml" xref="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.3.3">subscript</csymbol><ci id="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.3.3.2.cmml" xref="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.3.3.2">𝑘</ci><times id="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.3.3.3.cmml" xref="S3.SS1.p1.11.m4.1.1.1.1.1.1.1.3.3.3"></times></apply></apply></apply><apply id="S3.SS1.p1.11.m4.2.2.2.2.2.cmml" xref="S3.SS1.p1.11.m4.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS1.p1.11.m4.2.2.2.2.2.2.cmml" xref="S3.SS1.p1.11.m4.2.2.2.2.2">superscript</csymbol><apply id="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.cmml" xref="S3.SS1.p1.11.m4.2.2.2.2.2.1.1"><times id="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.1.cmml" xref="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.1"></times><apply id="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.2.cmml" xref="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.2.1.cmml" xref="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.2">superscript</csymbol><ci id="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.2.2.cmml" xref="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.2.2">𝐶</ci><apply id="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.2.3.cmml" xref="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.2.3"><minus id="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.2.3.1.cmml" xref="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.2.3"></minus><cn type="integer" id="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.2.3.2.cmml" xref="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.2.3.2">1</cn></apply></apply><apply id="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.3.cmml" xref="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.3.1.cmml" xref="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.3">subscript</csymbol><ci id="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.3.2.cmml" xref="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.3.2">𝑘</ci><times id="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.3.3.cmml" xref="S3.SS1.p1.11.m4.2.2.2.2.2.1.1.1.3.3"></times></apply></apply><ci id="S3.SS1.p1.11.m4.2.2.2.2.2.3.cmml" xref="S3.SS1.p1.11.m4.2.2.2.2.2.3">𝑇</ci></apply></apply><apply id="S3.SS1.p1.11.m4.2.2.2.4.cmml" xref="S3.SS1.p1.11.m4.2.2.2.4"><csymbol cd="ambiguous" id="S3.SS1.p1.11.m4.2.2.2.4.1.cmml" xref="S3.SS1.p1.11.m4.2.2.2.4">subscript</csymbol><ci id="S3.SS1.p1.11.m4.2.2.2.4.2.cmml" xref="S3.SS1.p1.11.m4.2.2.2.4.2">𝑘</ci><times id="S3.SS1.p1.11.m4.2.2.2.4.3.cmml" xref="S3.SS1.p1.11.m4.2.2.2.4.3"></times></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.11.m4.2c">\Lambda=(v_{*}-Wk_{*})/(C^{-1}k_{*})^{T}k_{*}</annotation></semantics></math> is a vector proportional to the residual error of the new key–value pair on the original memory matrix (full derivation in Appendix <a href="#A1" title="Appendix A Solving for Λ Algebraically ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a>).
Because of this simple algebraic structure, we can insert any fact directly once <math id="S3.SS1.p1.12.m5.2" class="ltx_Math" alttext="(k_{*},v_{*})" display="inline"><semantics id="S3.SS1.p1.12.m5.2a"><mrow id="S3.SS1.p1.12.m5.2.2.2" xref="S3.SS1.p1.12.m5.2.2.3.cmml"><mo stretchy="false" id="S3.SS1.p1.12.m5.2.2.2.3" xref="S3.SS1.p1.12.m5.2.2.3.cmml">(</mo><msub id="S3.SS1.p1.12.m5.1.1.1.1" xref="S3.SS1.p1.12.m5.1.1.1.1.cmml"><mi id="S3.SS1.p1.12.m5.1.1.1.1.2" xref="S3.SS1.p1.12.m5.1.1.1.1.2.cmml">k</mi><mo id="S3.SS1.p1.12.m5.1.1.1.1.3" xref="S3.SS1.p1.12.m5.1.1.1.1.3.cmml">∗</mo></msub><mo id="S3.SS1.p1.12.m5.2.2.2.4" xref="S3.SS1.p1.12.m5.2.2.3.cmml">,</mo><msub id="S3.SS1.p1.12.m5.2.2.2.2" xref="S3.SS1.p1.12.m5.2.2.2.2.cmml"><mi id="S3.SS1.p1.12.m5.2.2.2.2.2" xref="S3.SS1.p1.12.m5.2.2.2.2.2.cmml">v</mi><mo id="S3.SS1.p1.12.m5.2.2.2.2.3" xref="S3.SS1.p1.12.m5.2.2.2.2.3.cmml">∗</mo></msub><mo stretchy="false" id="S3.SS1.p1.12.m5.2.2.2.5" xref="S3.SS1.p1.12.m5.2.2.3.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.12.m5.2b"><interval closure="open" id="S3.SS1.p1.12.m5.2.2.3.cmml" xref="S3.SS1.p1.12.m5.2.2.2"><apply id="S3.SS1.p1.12.m5.1.1.1.1.cmml" xref="S3.SS1.p1.12.m5.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.12.m5.1.1.1.1.1.cmml" xref="S3.SS1.p1.12.m5.1.1.1.1">subscript</csymbol><ci id="S3.SS1.p1.12.m5.1.1.1.1.2.cmml" xref="S3.SS1.p1.12.m5.1.1.1.1.2">𝑘</ci><times id="S3.SS1.p1.12.m5.1.1.1.1.3.cmml" xref="S3.SS1.p1.12.m5.1.1.1.1.3"></times></apply><apply id="S3.SS1.p1.12.m5.2.2.2.2.cmml" xref="S3.SS1.p1.12.m5.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS1.p1.12.m5.2.2.2.2.1.cmml" xref="S3.SS1.p1.12.m5.2.2.2.2">subscript</csymbol><ci id="S3.SS1.p1.12.m5.2.2.2.2.2.cmml" xref="S3.SS1.p1.12.m5.2.2.2.2.2">𝑣</ci><times id="S3.SS1.p1.12.m5.2.2.2.2.3.cmml" xref="S3.SS1.p1.12.m5.2.2.2.2.3"></times></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.12.m5.2c">(k_{*},v_{*})</annotation></semantics></math> is computed. All that remains is to choose the appropriate <math id="S3.SS1.p1.13.m6.1" class="ltx_Math" alttext="k_{*}" display="inline"><semantics id="S3.SS1.p1.13.m6.1a"><msub id="S3.SS1.p1.13.m6.1.1" xref="S3.SS1.p1.13.m6.1.1.cmml"><mi id="S3.SS1.p1.13.m6.1.1.2" xref="S3.SS1.p1.13.m6.1.1.2.cmml">k</mi><mo id="S3.SS1.p1.13.m6.1.1.3" xref="S3.SS1.p1.13.m6.1.1.3.cmml">∗</mo></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.13.m6.1b"><apply id="S3.SS1.p1.13.m6.1.1.cmml" xref="S3.SS1.p1.13.m6.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.13.m6.1.1.1.cmml" xref="S3.SS1.p1.13.m6.1.1">subscript</csymbol><ci id="S3.SS1.p1.13.m6.1.1.2.cmml" xref="S3.SS1.p1.13.m6.1.1.2">𝑘</ci><times id="S3.SS1.p1.13.m6.1.1.3.cmml" xref="S3.SS1.p1.13.m6.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.13.m6.1c">k_{*}</annotation></semantics></math> and <math id="S3.SS1.p1.14.m7.1" class="ltx_Math" alttext="v_{*}" display="inline"><semantics id="S3.SS1.p1.14.m7.1a"><msub id="S3.SS1.p1.14.m7.1.1" xref="S3.SS1.p1.14.m7.1.1.cmml"><mi id="S3.SS1.p1.14.m7.1.1.2" xref="S3.SS1.p1.14.m7.1.1.2.cmml">v</mi><mo id="S3.SS1.p1.14.m7.1.1.3" xref="S3.SS1.p1.14.m7.1.1.3.cmml">∗</mo></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.14.m7.1b"><apply id="S3.SS1.p1.14.m7.1.1.cmml" xref="S3.SS1.p1.14.m7.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.14.m7.1.1.1.cmml" xref="S3.SS1.p1.14.m7.1.1">subscript</csymbol><ci id="S3.SS1.p1.14.m7.1.1.2.cmml" xref="S3.SS1.p1.14.m7.1.1.2">𝑣</ci><times id="S3.SS1.p1.14.m7.1.1.3.cmml" xref="S3.SS1.p1.14.m7.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.14.m7.1c">v_{*}</annotation></semantics></math>.</p>
</div>
<div id="S3.SS1.p2" class="ltx_para">
<p id="S3.SS1.p2.11" class="ltx_p"><span id="S3.SS1.p2.1.1" class="ltx_text ltx_font_bold">Step 1: Choosing <math id="S3.SS1.p2.1.1.m1.1" class="ltx_Math" alttext="k_{*}" display="inline"><semantics id="S3.SS1.p2.1.1.m1.1a"><msub id="S3.SS1.p2.1.1.m1.1.1" xref="S3.SS1.p2.1.1.m1.1.1.cmml"><mi id="S3.SS1.p2.1.1.m1.1.1.2" xref="S3.SS1.p2.1.1.m1.1.1.2.cmml">k</mi><mo id="S3.SS1.p2.1.1.m1.1.1.3" xref="S3.SS1.p2.1.1.m1.1.1.3.cmml">∗</mo></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.1.1.m1.1b"><apply id="S3.SS1.p2.1.1.m1.1.1.cmml" xref="S3.SS1.p2.1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p2.1.1.m1.1.1.1.cmml" xref="S3.SS1.p2.1.1.m1.1.1">subscript</csymbol><ci id="S3.SS1.p2.1.1.m1.1.1.2.cmml" xref="S3.SS1.p2.1.1.m1.1.1.2">𝑘</ci><times id="S3.SS1.p2.1.1.m1.1.1.3.cmml" xref="S3.SS1.p2.1.1.m1.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.1.1.m1.1c">k_{*}</annotation></semantics></math> to Select the Subject.</span>
Based on the decisive role of MLP inputs at the final subject token (Section <a href="#S2" title="2 Interventions on Activations for Tracing Information Flow ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>), we shall choose inputs that represent the subject at its last token as the lookup key <math id="S3.SS1.p2.2.m1.1" class="ltx_Math" alttext="k_{*}" display="inline"><semantics id="S3.SS1.p2.2.m1.1a"><msub id="S3.SS1.p2.2.m1.1.1" xref="S3.SS1.p2.2.m1.1.1.cmml"><mi id="S3.SS1.p2.2.m1.1.1.2" xref="S3.SS1.p2.2.m1.1.1.2.cmml">k</mi><mo id="S3.SS1.p2.2.m1.1.1.3" xref="S3.SS1.p2.2.m1.1.1.3.cmml">∗</mo></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.2.m1.1b"><apply id="S3.SS1.p2.2.m1.1.1.cmml" xref="S3.SS1.p2.2.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p2.2.m1.1.1.1.cmml" xref="S3.SS1.p2.2.m1.1.1">subscript</csymbol><ci id="S3.SS1.p2.2.m1.1.1.2.cmml" xref="S3.SS1.p2.2.m1.1.1.2">𝑘</ci><times id="S3.SS1.p2.2.m1.1.1.3.cmml" xref="S3.SS1.p2.2.m1.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.2.m1.1c">k_{*}</annotation></semantics></math>.
Specifically, we compute <math id="S3.SS1.p2.3.m2.1" class="ltx_Math" alttext="k_{*}" display="inline"><semantics id="S3.SS1.p2.3.m2.1a"><msub id="S3.SS1.p2.3.m2.1.1" xref="S3.SS1.p2.3.m2.1.1.cmml"><mi id="S3.SS1.p2.3.m2.1.1.2" xref="S3.SS1.p2.3.m2.1.1.2.cmml">k</mi><mo id="S3.SS1.p2.3.m2.1.1.3" xref="S3.SS1.p2.3.m2.1.1.3.cmml">∗</mo></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.3.m2.1b"><apply id="S3.SS1.p2.3.m2.1.1.cmml" xref="S3.SS1.p2.3.m2.1.1"><csymbol cd="ambiguous" id="S3.SS1.p2.3.m2.1.1.1.cmml" xref="S3.SS1.p2.3.m2.1.1">subscript</csymbol><ci id="S3.SS1.p2.3.m2.1.1.2.cmml" xref="S3.SS1.p2.3.m2.1.1.2">𝑘</ci><times id="S3.SS1.p2.3.m2.1.1.3.cmml" xref="S3.SS1.p2.3.m2.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.3.m2.1c">k_{*}</annotation></semantics></math> by collecting activations: We pass text <math id="S3.SS1.p2.4.m3.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S3.SS1.p2.4.m3.1a"><mi id="S3.SS1.p2.4.m3.1.1" xref="S3.SS1.p2.4.m3.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.4.m3.1b"><ci id="S3.SS1.p2.4.m3.1.1.cmml" xref="S3.SS1.p2.4.m3.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.4.m3.1c">x</annotation></semantics></math> containing the subject <math id="S3.SS1.p2.5.m4.1" class="ltx_Math" alttext="s" display="inline"><semantics id="S3.SS1.p2.5.m4.1a"><mi id="S3.SS1.p2.5.m4.1.1" xref="S3.SS1.p2.5.m4.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.5.m4.1b"><ci id="S3.SS1.p2.5.m4.1.1.cmml" xref="S3.SS1.p2.5.m4.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.5.m4.1c">s</annotation></semantics></math> through <math id="S3.SS1.p2.6.m5.1" class="ltx_Math" alttext="G" display="inline"><semantics id="S3.SS1.p2.6.m5.1a"><mi id="S3.SS1.p2.6.m5.1.1" xref="S3.SS1.p2.6.m5.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.6.m5.1b"><ci id="S3.SS1.p2.6.m5.1.1.cmml" xref="S3.SS1.p2.6.m5.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.6.m5.1c">G</annotation></semantics></math>; then at layer <math id="S3.SS1.p2.7.m6.1" class="ltx_Math" alttext="l^{*}" display="inline"><semantics id="S3.SS1.p2.7.m6.1a"><msup id="S3.SS1.p2.7.m6.1.1" xref="S3.SS1.p2.7.m6.1.1.cmml"><mi id="S3.SS1.p2.7.m6.1.1.2" xref="S3.SS1.p2.7.m6.1.1.2.cmml">l</mi><mo id="S3.SS1.p2.7.m6.1.1.3" xref="S3.SS1.p2.7.m6.1.1.3.cmml">∗</mo></msup><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.7.m6.1b"><apply id="S3.SS1.p2.7.m6.1.1.cmml" xref="S3.SS1.p2.7.m6.1.1"><csymbol cd="ambiguous" id="S3.SS1.p2.7.m6.1.1.1.cmml" xref="S3.SS1.p2.7.m6.1.1">superscript</csymbol><ci id="S3.SS1.p2.7.m6.1.1.2.cmml" xref="S3.SS1.p2.7.m6.1.1.2">𝑙</ci><times id="S3.SS1.p2.7.m6.1.1.3.cmml" xref="S3.SS1.p2.7.m6.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.7.m6.1c">l^{*}</annotation></semantics></math> and last subject token index <math id="S3.SS1.p2.8.m7.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S3.SS1.p2.8.m7.1a"><mi id="S3.SS1.p2.8.m7.1.1" xref="S3.SS1.p2.8.m7.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.8.m7.1b"><ci id="S3.SS1.p2.8.m7.1.1.cmml" xref="S3.SS1.p2.8.m7.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.8.m7.1c">i</annotation></semantics></math>, we read the value after the non-linearity inside the MLP (Figure <a href="#S3.F4" title="Figure 4 ‣ 3.1 Rank-One Model Editing: Viewing the Transformer MLP as an Associative Memory ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>d). Because the state will vary depending on tokens that precede <math id="S3.SS1.p2.9.m8.1" class="ltx_Math" alttext="s" display="inline"><semantics id="S3.SS1.p2.9.m8.1a"><mi id="S3.SS1.p2.9.m8.1.1" xref="S3.SS1.p2.9.m8.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.9.m8.1b"><ci id="S3.SS1.p2.9.m8.1.1.cmml" xref="S3.SS1.p2.9.m8.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.9.m8.1c">s</annotation></semantics></math> in text, we set <math id="S3.SS1.p2.10.m9.1" class="ltx_Math" alttext="k_{*}" display="inline"><semantics id="S3.SS1.p2.10.m9.1a"><msub id="S3.SS1.p2.10.m9.1.1" xref="S3.SS1.p2.10.m9.1.1.cmml"><mi id="S3.SS1.p2.10.m9.1.1.2" xref="S3.SS1.p2.10.m9.1.1.2.cmml">k</mi><mo id="S3.SS1.p2.10.m9.1.1.3" xref="S3.SS1.p2.10.m9.1.1.3.cmml">∗</mo></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.10.m9.1b"><apply id="S3.SS1.p2.10.m9.1.1.cmml" xref="S3.SS1.p2.10.m9.1.1"><csymbol cd="ambiguous" id="S3.SS1.p2.10.m9.1.1.1.cmml" xref="S3.SS1.p2.10.m9.1.1">subscript</csymbol><ci id="S3.SS1.p2.10.m9.1.1.2.cmml" xref="S3.SS1.p2.10.m9.1.1.2">𝑘</ci><times id="S3.SS1.p2.10.m9.1.1.3.cmml" xref="S3.SS1.p2.10.m9.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.10.m9.1c">k_{*}</annotation></semantics></math> to an average value over a small set of texts ending with the subject <math id="S3.SS1.p2.11.m10.1" class="ltx_Math" alttext="s" display="inline"><semantics id="S3.SS1.p2.11.m10.1a"><mi id="S3.SS1.p2.11.m10.1.1" xref="S3.SS1.p2.11.m10.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.11.m10.1b"><ci id="S3.SS1.p2.11.m10.1.1.cmml" xref="S3.SS1.p2.11.m10.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.11.m10.1c">s</annotation></semantics></math>:</p>
<table id="A10.EGx3" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S3.E3"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E3.m1.11" class="ltx_Math" alttext="\displaystyle k_{*}=\frac{1}{N}\sum_{j=1}^{N}k(x_{j}+s),\;\text{where}\;k(x)=\sigma\left(W_{fc}^{(l^{*})}\;\gamma(a^{(l^{*})}_{[x],i}+h^{(l^{*}-1)}_{[x],i})\right)." display="inline"><semantics id="S3.E3.m1.11a"><mrow id="S3.E3.m1.11.11.1"><mrow id="S3.E3.m1.11.11.1.1.2" xref="S3.E3.m1.11.11.1.1.3.cmml"><mrow id="S3.E3.m1.11.11.1.1.1.1" xref="S3.E3.m1.11.11.1.1.1.1.cmml"><msub id="S3.E3.m1.11.11.1.1.1.1.3" xref="S3.E3.m1.11.11.1.1.1.1.3.cmml"><mi id="S3.E3.m1.11.11.1.1.1.1.3.2" xref="S3.E3.m1.11.11.1.1.1.1.3.2.cmml">k</mi><mo id="S3.E3.m1.11.11.1.1.1.1.3.3" xref="S3.E3.m1.11.11.1.1.1.1.3.3.cmml">∗</mo></msub><mo id="S3.E3.m1.11.11.1.1.1.1.2" xref="S3.E3.m1.11.11.1.1.1.1.2.cmml">=</mo><mrow id="S3.E3.m1.11.11.1.1.1.1.1" xref="S3.E3.m1.11.11.1.1.1.1.1.cmml"><mstyle displaystyle="true" id="S3.E3.m1.11.11.1.1.1.1.1.3" xref="S3.E3.m1.11.11.1.1.1.1.1.3.cmml"><mfrac id="S3.E3.m1.11.11.1.1.1.1.1.3a" xref="S3.E3.m1.11.11.1.1.1.1.1.3.cmml"><mn id="S3.E3.m1.11.11.1.1.1.1.1.3.2" xref="S3.E3.m1.11.11.1.1.1.1.1.3.2.cmml">1</mn><mi id="S3.E3.m1.11.11.1.1.1.1.1.3.3" xref="S3.E3.m1.11.11.1.1.1.1.1.3.3.cmml">N</mi></mfrac></mstyle><mo lspace="0em" rspace="0em" id="S3.E3.m1.11.11.1.1.1.1.1.2" xref="S3.E3.m1.11.11.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.E3.m1.11.11.1.1.1.1.1.1" xref="S3.E3.m1.11.11.1.1.1.1.1.1.cmml"><mstyle displaystyle="true" id="S3.E3.m1.11.11.1.1.1.1.1.1.2" xref="S3.E3.m1.11.11.1.1.1.1.1.1.2.cmml"><munderover id="S3.E3.m1.11.11.1.1.1.1.1.1.2a" xref="S3.E3.m1.11.11.1.1.1.1.1.1.2.cmml"><mo movablelimits="false" id="S3.E3.m1.11.11.1.1.1.1.1.1.2.2.2" xref="S3.E3.m1.11.11.1.1.1.1.1.1.2.2.2.cmml">∑</mo><mrow id="S3.E3.m1.11.11.1.1.1.1.1.1.2.2.3" xref="S3.E3.m1.11.11.1.1.1.1.1.1.2.2.3.cmml"><mi id="S3.E3.m1.11.11.1.1.1.1.1.1.2.2.3.2" xref="S3.E3.m1.11.11.1.1.1.1.1.1.2.2.3.2.cmml">j</mi><mo id="S3.E3.m1.11.11.1.1.1.1.1.1.2.2.3.1" xref="S3.E3.m1.11.11.1.1.1.1.1.1.2.2.3.1.cmml">=</mo><mn id="S3.E3.m1.11.11.1.1.1.1.1.1.2.2.3.3" xref="S3.E3.m1.11.11.1.1.1.1.1.1.2.2.3.3.cmml">1</mn></mrow><mi id="S3.E3.m1.11.11.1.1.1.1.1.1.2.3" xref="S3.E3.m1.11.11.1.1.1.1.1.1.2.3.cmml">N</mi></munderover></mstyle><mrow id="S3.E3.m1.11.11.1.1.1.1.1.1.1" xref="S3.E3.m1.11.11.1.1.1.1.1.1.1.cmml"><mi id="S3.E3.m1.11.11.1.1.1.1.1.1.1.3" xref="S3.E3.m1.11.11.1.1.1.1.1.1.1.3.cmml">k</mi><mo lspace="0em" rspace="0em" id="S3.E3.m1.11.11.1.1.1.1.1.1.1.2" xref="S3.E3.m1.11.11.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1" xref="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.2" xref="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.1" xref="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.1.cmml"><msub id="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.1.2" xref="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.1.2.cmml"><mi id="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.1.2.2" xref="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.1.2.3" xref="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.1.2.3.cmml">j</mi></msub><mo id="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.1.1" xref="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.1.1.cmml">+</mo><mi id="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.1.3" xref="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.1.3.cmml">s</mi></mrow><mo stretchy="false" id="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.3" xref="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow></mrow><mo rspace="0.447em" id="S3.E3.m1.11.11.1.1.2.3" xref="S3.E3.m1.11.11.1.1.3a.cmml">,</mo><mrow id="S3.E3.m1.11.11.1.1.2.2" xref="S3.E3.m1.11.11.1.1.2.2.cmml"><mrow id="S3.E3.m1.11.11.1.1.2.2.3" xref="S3.E3.m1.11.11.1.1.2.2.3.cmml"><mtext id="S3.E3.m1.11.11.1.1.2.2.3.2" xref="S3.E3.m1.11.11.1.1.2.2.3.2a.cmml">where</mtext><mo lspace="0.280em" rspace="0em" id="S3.E3.m1.11.11.1.1.2.2.3.1" xref="S3.E3.m1.11.11.1.1.2.2.3.1.cmml">​</mo><mi id="S3.E3.m1.11.11.1.1.2.2.3.3" xref="S3.E3.m1.11.11.1.1.2.2.3.3.cmml">k</mi><mo lspace="0em" rspace="0em" id="S3.E3.m1.11.11.1.1.2.2.3.1a" xref="S3.E3.m1.11.11.1.1.2.2.3.1.cmml">​</mo><mrow id="S3.E3.m1.11.11.1.1.2.2.3.4.2" xref="S3.E3.m1.11.11.1.1.2.2.3.cmml"><mo stretchy="false" id="S3.E3.m1.11.11.1.1.2.2.3.4.2.1" xref="S3.E3.m1.11.11.1.1.2.2.3.cmml">(</mo><mi id="S3.E3.m1.10.10" xref="S3.E3.m1.10.10.cmml">x</mi><mo stretchy="false" id="S3.E3.m1.11.11.1.1.2.2.3.4.2.2" xref="S3.E3.m1.11.11.1.1.2.2.3.cmml">)</mo></mrow></mrow><mo id="S3.E3.m1.11.11.1.1.2.2.2" xref="S3.E3.m1.11.11.1.1.2.2.2.cmml">=</mo><mrow id="S3.E3.m1.11.11.1.1.2.2.1" xref="S3.E3.m1.11.11.1.1.2.2.1.cmml"><mi id="S3.E3.m1.11.11.1.1.2.2.1.3" xref="S3.E3.m1.11.11.1.1.2.2.1.3.cmml">σ</mi><mo lspace="0em" rspace="0em" id="S3.E3.m1.11.11.1.1.2.2.1.2" xref="S3.E3.m1.11.11.1.1.2.2.1.2.cmml">​</mo><mrow id="S3.E3.m1.11.11.1.1.2.2.1.1.1" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.cmml"><mo id="S3.E3.m1.11.11.1.1.2.2.1.1.1.2" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.cmml">(</mo><mrow id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.cmml"><msubsup id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3.cmml"><mi id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3.2.2" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3.2.2.cmml">W</mi><mrow id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3.2.3" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3.2.3.cmml"><mi id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3.2.3.2" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3.2.3.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3.2.3.1" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3.2.3.1.cmml">​</mo><mi id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3.2.3.3" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3.2.3.3.cmml">c</mi></mrow><mrow id="S3.E3.m1.1.1.1.1" xref="S3.E3.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E3.m1.1.1.1.1.2" xref="S3.E3.m1.1.1.1.1.1.cmml">(</mo><msup id="S3.E3.m1.1.1.1.1.1" xref="S3.E3.m1.1.1.1.1.1.cmml"><mi id="S3.E3.m1.1.1.1.1.1.2" xref="S3.E3.m1.1.1.1.1.1.2.cmml">l</mi><mo id="S3.E3.m1.1.1.1.1.1.3" xref="S3.E3.m1.1.1.1.1.1.3.cmml">∗</mo></msup><mo stretchy="false" id="S3.E3.m1.1.1.1.1.3" xref="S3.E3.m1.1.1.1.1.1.cmml">)</mo></mrow></msubsup><mo lspace="0em" rspace="0em" id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.2" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.2.cmml">​</mo><mi id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.4" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.4.cmml">γ</mi><mo lspace="0em" rspace="0em" id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.2a" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.2.cmml">​</mo><mrow id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.2" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.cmml"><msubsup id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.2" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.2.cmml"><mi id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.2.2.2" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.2.2.2.cmml">a</mi><mrow id="S3.E3.m1.5.5.3.3" xref="S3.E3.m1.5.5.3.4.cmml"><mrow id="S3.E3.m1.5.5.3.3.1.2" xref="S3.E3.m1.5.5.3.3.1.1.cmml"><mo stretchy="false" id="S3.E3.m1.5.5.3.3.1.2.1" xref="S3.E3.m1.5.5.3.3.1.1.1.cmml">[</mo><mi id="S3.E3.m1.3.3.1.1" xref="S3.E3.m1.3.3.1.1.cmml">x</mi><mo stretchy="false" id="S3.E3.m1.5.5.3.3.1.2.2" xref="S3.E3.m1.5.5.3.3.1.1.1.cmml">]</mo></mrow><mo id="S3.E3.m1.5.5.3.3.2" xref="S3.E3.m1.5.5.3.4.cmml">,</mo><mi id="S3.E3.m1.4.4.2.2" xref="S3.E3.m1.4.4.2.2.cmml">i</mi></mrow><mrow id="S3.E3.m1.2.2.1.1" xref="S3.E3.m1.2.2.1.1.1.cmml"><mo stretchy="false" id="S3.E3.m1.2.2.1.1.2" xref="S3.E3.m1.2.2.1.1.1.cmml">(</mo><msup id="S3.E3.m1.2.2.1.1.1" xref="S3.E3.m1.2.2.1.1.1.cmml"><mi id="S3.E3.m1.2.2.1.1.1.2" xref="S3.E3.m1.2.2.1.1.1.2.cmml">l</mi><mo id="S3.E3.m1.2.2.1.1.1.3" xref="S3.E3.m1.2.2.1.1.1.3.cmml">∗</mo></msup><mo stretchy="false" id="S3.E3.m1.2.2.1.1.3" xref="S3.E3.m1.2.2.1.1.1.cmml">)</mo></mrow></msubsup><mo id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.1" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.1.cmml">+</mo><msubsup id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.3" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.3.cmml"><mi id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.3.2.2" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.3.2.2.cmml">h</mi><mrow id="S3.E3.m1.9.9.3.3" xref="S3.E3.m1.9.9.3.4.cmml"><mrow id="S3.E3.m1.9.9.3.3.1.2" xref="S3.E3.m1.9.9.3.3.1.1.cmml"><mo stretchy="false" id="S3.E3.m1.9.9.3.3.1.2.1" xref="S3.E3.m1.9.9.3.3.1.1.1.cmml">[</mo><mi id="S3.E3.m1.7.7.1.1" xref="S3.E3.m1.7.7.1.1.cmml">x</mi><mo stretchy="false" id="S3.E3.m1.9.9.3.3.1.2.2" xref="S3.E3.m1.9.9.3.3.1.1.1.cmml">]</mo></mrow><mo id="S3.E3.m1.9.9.3.3.2" xref="S3.E3.m1.9.9.3.4.cmml">,</mo><mi id="S3.E3.m1.8.8.2.2" xref="S3.E3.m1.8.8.2.2.cmml">i</mi></mrow><mrow id="S3.E3.m1.6.6.1.1" xref="S3.E3.m1.6.6.1.1.1.cmml"><mo stretchy="false" id="S3.E3.m1.6.6.1.1.2" xref="S3.E3.m1.6.6.1.1.1.cmml">(</mo><mrow id="S3.E3.m1.6.6.1.1.1" xref="S3.E3.m1.6.6.1.1.1.cmml"><msup id="S3.E3.m1.6.6.1.1.1.2" xref="S3.E3.m1.6.6.1.1.1.2.cmml"><mi id="S3.E3.m1.6.6.1.1.1.2.2" xref="S3.E3.m1.6.6.1.1.1.2.2.cmml">l</mi><mo id="S3.E3.m1.6.6.1.1.1.2.3" xref="S3.E3.m1.6.6.1.1.1.2.3.cmml">∗</mo></msup><mo id="S3.E3.m1.6.6.1.1.1.1" xref="S3.E3.m1.6.6.1.1.1.1.cmml">−</mo><mn id="S3.E3.m1.6.6.1.1.1.3" xref="S3.E3.m1.6.6.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="S3.E3.m1.6.6.1.1.3" xref="S3.E3.m1.6.6.1.1.1.cmml">)</mo></mrow></msubsup></mrow><mo stretchy="false" id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.3" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.E3.m1.11.11.1.1.2.2.1.1.1.3" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><mo lspace="0em" id="S3.E3.m1.11.11.1.2">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E3.m1.11b"><apply id="S3.E3.m1.11.11.1.1.3.cmml" xref="S3.E3.m1.11.11.1.1.2"><csymbol cd="ambiguous" id="S3.E3.m1.11.11.1.1.3a.cmml" xref="S3.E3.m1.11.11.1.1.2.3">formulae-sequence</csymbol><apply id="S3.E3.m1.11.11.1.1.1.1.cmml" xref="S3.E3.m1.11.11.1.1.1.1"><eq id="S3.E3.m1.11.11.1.1.1.1.2.cmml" xref="S3.E3.m1.11.11.1.1.1.1.2"></eq><apply id="S3.E3.m1.11.11.1.1.1.1.3.cmml" xref="S3.E3.m1.11.11.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E3.m1.11.11.1.1.1.1.3.1.cmml" xref="S3.E3.m1.11.11.1.1.1.1.3">subscript</csymbol><ci id="S3.E3.m1.11.11.1.1.1.1.3.2.cmml" xref="S3.E3.m1.11.11.1.1.1.1.3.2">𝑘</ci><times id="S3.E3.m1.11.11.1.1.1.1.3.3.cmml" xref="S3.E3.m1.11.11.1.1.1.1.3.3"></times></apply><apply id="S3.E3.m1.11.11.1.1.1.1.1.cmml" xref="S3.E3.m1.11.11.1.1.1.1.1"><times id="S3.E3.m1.11.11.1.1.1.1.1.2.cmml" xref="S3.E3.m1.11.11.1.1.1.1.1.2"></times><apply id="S3.E3.m1.11.11.1.1.1.1.1.3.cmml" xref="S3.E3.m1.11.11.1.1.1.1.1.3"><divide id="S3.E3.m1.11.11.1.1.1.1.1.3.1.cmml" xref="S3.E3.m1.11.11.1.1.1.1.1.3"></divide><cn type="integer" id="S3.E3.m1.11.11.1.1.1.1.1.3.2.cmml" xref="S3.E3.m1.11.11.1.1.1.1.1.3.2">1</cn><ci id="S3.E3.m1.11.11.1.1.1.1.1.3.3.cmml" xref="S3.E3.m1.11.11.1.1.1.1.1.3.3">𝑁</ci></apply><apply id="S3.E3.m1.11.11.1.1.1.1.1.1.cmml" xref="S3.E3.m1.11.11.1.1.1.1.1.1"><apply id="S3.E3.m1.11.11.1.1.1.1.1.1.2.cmml" xref="S3.E3.m1.11.11.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E3.m1.11.11.1.1.1.1.1.1.2.1.cmml" xref="S3.E3.m1.11.11.1.1.1.1.1.1.2">superscript</csymbol><apply id="S3.E3.m1.11.11.1.1.1.1.1.1.2.2.cmml" xref="S3.E3.m1.11.11.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E3.m1.11.11.1.1.1.1.1.1.2.2.1.cmml" xref="S3.E3.m1.11.11.1.1.1.1.1.1.2">subscript</csymbol><sum id="S3.E3.m1.11.11.1.1.1.1.1.1.2.2.2.cmml" xref="S3.E3.m1.11.11.1.1.1.1.1.1.2.2.2"></sum><apply id="S3.E3.m1.11.11.1.1.1.1.1.1.2.2.3.cmml" xref="S3.E3.m1.11.11.1.1.1.1.1.1.2.2.3"><eq id="S3.E3.m1.11.11.1.1.1.1.1.1.2.2.3.1.cmml" xref="S3.E3.m1.11.11.1.1.1.1.1.1.2.2.3.1"></eq><ci id="S3.E3.m1.11.11.1.1.1.1.1.1.2.2.3.2.cmml" xref="S3.E3.m1.11.11.1.1.1.1.1.1.2.2.3.2">𝑗</ci><cn type="integer" id="S3.E3.m1.11.11.1.1.1.1.1.1.2.2.3.3.cmml" xref="S3.E3.m1.11.11.1.1.1.1.1.1.2.2.3.3">1</cn></apply></apply><ci id="S3.E3.m1.11.11.1.1.1.1.1.1.2.3.cmml" xref="S3.E3.m1.11.11.1.1.1.1.1.1.2.3">𝑁</ci></apply><apply id="S3.E3.m1.11.11.1.1.1.1.1.1.1.cmml" xref="S3.E3.m1.11.11.1.1.1.1.1.1.1"><times id="S3.E3.m1.11.11.1.1.1.1.1.1.1.2.cmml" xref="S3.E3.m1.11.11.1.1.1.1.1.1.1.2"></times><ci id="S3.E3.m1.11.11.1.1.1.1.1.1.1.3.cmml" xref="S3.E3.m1.11.11.1.1.1.1.1.1.1.3">𝑘</ci><apply id="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1"><plus id="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.1.1"></plus><apply id="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.1.2.2">𝑥</ci><ci id="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.1.2.3">𝑗</ci></apply><ci id="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E3.m1.11.11.1.1.1.1.1.1.1.1.1.1.3">𝑠</ci></apply></apply></apply></apply></apply><apply id="S3.E3.m1.11.11.1.1.2.2.cmml" xref="S3.E3.m1.11.11.1.1.2.2"><eq id="S3.E3.m1.11.11.1.1.2.2.2.cmml" xref="S3.E3.m1.11.11.1.1.2.2.2"></eq><apply id="S3.E3.m1.11.11.1.1.2.2.3.cmml" xref="S3.E3.m1.11.11.1.1.2.2.3"><times id="S3.E3.m1.11.11.1.1.2.2.3.1.cmml" xref="S3.E3.m1.11.11.1.1.2.2.3.1"></times><ci id="S3.E3.m1.11.11.1.1.2.2.3.2a.cmml" xref="S3.E3.m1.11.11.1.1.2.2.3.2"><mtext id="S3.E3.m1.11.11.1.1.2.2.3.2.cmml" xref="S3.E3.m1.11.11.1.1.2.2.3.2">where</mtext></ci><ci id="S3.E3.m1.11.11.1.1.2.2.3.3.cmml" xref="S3.E3.m1.11.11.1.1.2.2.3.3">𝑘</ci><ci id="S3.E3.m1.10.10.cmml" xref="S3.E3.m1.10.10">𝑥</ci></apply><apply id="S3.E3.m1.11.11.1.1.2.2.1.cmml" xref="S3.E3.m1.11.11.1.1.2.2.1"><times id="S3.E3.m1.11.11.1.1.2.2.1.2.cmml" xref="S3.E3.m1.11.11.1.1.2.2.1.2"></times><ci id="S3.E3.m1.11.11.1.1.2.2.1.3.cmml" xref="S3.E3.m1.11.11.1.1.2.2.1.3">𝜎</ci><apply id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.cmml" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1"><times id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.2.cmml" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.2"></times><apply id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3.cmml" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3.1.cmml" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3">superscript</csymbol><apply id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3.2.cmml" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3.2.1.cmml" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3">subscript</csymbol><ci id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3.2.2.cmml" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3.2.2">𝑊</ci><apply id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3.2.3.cmml" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3.2.3"><times id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3.2.3.1.cmml" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3.2.3.1"></times><ci id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3.2.3.2.cmml" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3.2.3.2">𝑓</ci><ci id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3.2.3.3.cmml" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.3.2.3.3">𝑐</ci></apply></apply><apply id="S3.E3.m1.1.1.1.1.1.cmml" xref="S3.E3.m1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E3.m1.1.1.1.1.1.1.cmml" xref="S3.E3.m1.1.1.1.1">superscript</csymbol><ci id="S3.E3.m1.1.1.1.1.1.2.cmml" xref="S3.E3.m1.1.1.1.1.1.2">𝑙</ci><times id="S3.E3.m1.1.1.1.1.1.3.cmml" xref="S3.E3.m1.1.1.1.1.1.3"></times></apply></apply><ci id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.4.cmml" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.4">𝛾</ci><apply id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.cmml" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1"><plus id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.1.cmml" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.1"></plus><apply id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.2.cmml" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.2.1.cmml" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.2">subscript</csymbol><apply id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.2.2.cmml" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.2.2.1.cmml" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.2">superscript</csymbol><ci id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.2.2.2.cmml" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.2.2.2">𝑎</ci><apply id="S3.E3.m1.2.2.1.1.1.cmml" xref="S3.E3.m1.2.2.1.1"><csymbol cd="ambiguous" id="S3.E3.m1.2.2.1.1.1.1.cmml" xref="S3.E3.m1.2.2.1.1">superscript</csymbol><ci id="S3.E3.m1.2.2.1.1.1.2.cmml" xref="S3.E3.m1.2.2.1.1.1.2">𝑙</ci><times id="S3.E3.m1.2.2.1.1.1.3.cmml" xref="S3.E3.m1.2.2.1.1.1.3"></times></apply></apply><list id="S3.E3.m1.5.5.3.4.cmml" xref="S3.E3.m1.5.5.3.3"><apply id="S3.E3.m1.5.5.3.3.1.1.cmml" xref="S3.E3.m1.5.5.3.3.1.2"><csymbol cd="latexml" id="S3.E3.m1.5.5.3.3.1.1.1.cmml" xref="S3.E3.m1.5.5.3.3.1.2.1">delimited-[]</csymbol><ci id="S3.E3.m1.3.3.1.1.cmml" xref="S3.E3.m1.3.3.1.1">𝑥</ci></apply><ci id="S3.E3.m1.4.4.2.2.cmml" xref="S3.E3.m1.4.4.2.2">𝑖</ci></list></apply><apply id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.3.cmml" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.3.1.cmml" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.3">subscript</csymbol><apply id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.3.2.cmml" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.3.2.1.cmml" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.3">superscript</csymbol><ci id="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.3.2.2.cmml" xref="S3.E3.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.3.2.2">ℎ</ci><apply id="S3.E3.m1.6.6.1.1.1.cmml" xref="S3.E3.m1.6.6.1.1"><minus id="S3.E3.m1.6.6.1.1.1.1.cmml" xref="S3.E3.m1.6.6.1.1.1.1"></minus><apply id="S3.E3.m1.6.6.1.1.1.2.cmml" xref="S3.E3.m1.6.6.1.1.1.2"><csymbol cd="ambiguous" id="S3.E3.m1.6.6.1.1.1.2.1.cmml" xref="S3.E3.m1.6.6.1.1.1.2">superscript</csymbol><ci id="S3.E3.m1.6.6.1.1.1.2.2.cmml" xref="S3.E3.m1.6.6.1.1.1.2.2">𝑙</ci><times id="S3.E3.m1.6.6.1.1.1.2.3.cmml" xref="S3.E3.m1.6.6.1.1.1.2.3"></times></apply><cn type="integer" id="S3.E3.m1.6.6.1.1.1.3.cmml" xref="S3.E3.m1.6.6.1.1.1.3">1</cn></apply></apply><list id="S3.E3.m1.9.9.3.4.cmml" xref="S3.E3.m1.9.9.3.3"><apply id="S3.E3.m1.9.9.3.3.1.1.cmml" xref="S3.E3.m1.9.9.3.3.1.2"><csymbol cd="latexml" id="S3.E3.m1.9.9.3.3.1.1.1.cmml" xref="S3.E3.m1.9.9.3.3.1.2.1">delimited-[]</csymbol><ci id="S3.E3.m1.7.7.1.1.cmml" xref="S3.E3.m1.7.7.1.1">𝑥</ci></apply><ci id="S3.E3.m1.8.8.2.2.cmml" xref="S3.E3.m1.8.8.2.2">𝑖</ci></list></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E3.m1.11c">\displaystyle k_{*}=\frac{1}{N}\sum_{j=1}^{N}k(x_{j}+s),\;\text{where}\;k(x)=\sigma\left(W_{fc}^{(l^{*})}\;\gamma(a^{(l^{*})}_{[x],i}+h^{(l^{*}-1)}_{[x],i})\right).</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
</table>
<p id="S3.SS1.p2.13" class="ltx_p">In practice, we sample <math id="S3.SS1.p2.12.m1.1" class="ltx_Math" alttext="x_{j}" display="inline"><semantics id="S3.SS1.p2.12.m1.1a"><msub id="S3.SS1.p2.12.m1.1.1" xref="S3.SS1.p2.12.m1.1.1.cmml"><mi id="S3.SS1.p2.12.m1.1.1.2" xref="S3.SS1.p2.12.m1.1.1.2.cmml">x</mi><mi id="S3.SS1.p2.12.m1.1.1.3" xref="S3.SS1.p2.12.m1.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.12.m1.1b"><apply id="S3.SS1.p2.12.m1.1.1.cmml" xref="S3.SS1.p2.12.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p2.12.m1.1.1.1.cmml" xref="S3.SS1.p2.12.m1.1.1">subscript</csymbol><ci id="S3.SS1.p2.12.m1.1.1.2.cmml" xref="S3.SS1.p2.12.m1.1.1.2">𝑥</ci><ci id="S3.SS1.p2.12.m1.1.1.3.cmml" xref="S3.SS1.p2.12.m1.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.12.m1.1c">x_{j}</annotation></semantics></math> by generating 50 random token sequences of length 2 to 10 using <math id="S3.SS1.p2.13.m2.1" class="ltx_Math" alttext="G" display="inline"><semantics id="S3.SS1.p2.13.m2.1a"><mi id="S3.SS1.p2.13.m2.1.1" xref="S3.SS1.p2.13.m2.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.13.m2.1b"><ci id="S3.SS1.p2.13.m2.1.1.cmml" xref="S3.SS1.p2.13.m2.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.13.m2.1c">G</annotation></semantics></math>.</p>
</div>
<div id="S3.SS1.p3" class="ltx_para">
<p id="S3.SS1.p3.6" class="ltx_p"><span id="S3.SS1.p3.1.1" class="ltx_text ltx_font_bold">Step 2: Choosing <math id="S3.SS1.p3.1.1.m1.1" class="ltx_Math" alttext="v_{*}" display="inline"><semantics id="S3.SS1.p3.1.1.m1.1a"><msub id="S3.SS1.p3.1.1.m1.1.1" xref="S3.SS1.p3.1.1.m1.1.1.cmml"><mi id="S3.SS1.p3.1.1.m1.1.1.2" xref="S3.SS1.p3.1.1.m1.1.1.2.cmml">v</mi><mo id="S3.SS1.p3.1.1.m1.1.1.3" xref="S3.SS1.p3.1.1.m1.1.1.3.cmml">∗</mo></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.1.1.m1.1b"><apply id="S3.SS1.p3.1.1.m1.1.1.cmml" xref="S3.SS1.p3.1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.1.1.m1.1.1.1.cmml" xref="S3.SS1.p3.1.1.m1.1.1">subscript</csymbol><ci id="S3.SS1.p3.1.1.m1.1.1.2.cmml" xref="S3.SS1.p3.1.1.m1.1.1.2">𝑣</ci><times id="S3.SS1.p3.1.1.m1.1.1.3.cmml" xref="S3.SS1.p3.1.1.m1.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.1.1.m1.1c">v_{*}</annotation></semantics></math> to Recall the Fact.</span>
Next, we wish to choose some vector value <math id="S3.SS1.p3.2.m1.1" class="ltx_Math" alttext="v_{*}" display="inline"><semantics id="S3.SS1.p3.2.m1.1a"><msub id="S3.SS1.p3.2.m1.1.1" xref="S3.SS1.p3.2.m1.1.1.cmml"><mi id="S3.SS1.p3.2.m1.1.1.2" xref="S3.SS1.p3.2.m1.1.1.2.cmml">v</mi><mo id="S3.SS1.p3.2.m1.1.1.3" xref="S3.SS1.p3.2.m1.1.1.3.cmml">∗</mo></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.2.m1.1b"><apply id="S3.SS1.p3.2.m1.1.1.cmml" xref="S3.SS1.p3.2.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.2.m1.1.1.1.cmml" xref="S3.SS1.p3.2.m1.1.1">subscript</csymbol><ci id="S3.SS1.p3.2.m1.1.1.2.cmml" xref="S3.SS1.p3.2.m1.1.1.2">𝑣</ci><times id="S3.SS1.p3.2.m1.1.1.3.cmml" xref="S3.SS1.p3.2.m1.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.2.m1.1c">v_{*}</annotation></semantics></math> that encodes the new relation <math id="S3.SS1.p3.3.m2.2" class="ltx_Math" alttext="(r,o^{*})" display="inline"><semantics id="S3.SS1.p3.3.m2.2a"><mrow id="S3.SS1.p3.3.m2.2.2.1" xref="S3.SS1.p3.3.m2.2.2.2.cmml"><mo stretchy="false" id="S3.SS1.p3.3.m2.2.2.1.2" xref="S3.SS1.p3.3.m2.2.2.2.cmml">(</mo><mi id="S3.SS1.p3.3.m2.1.1" xref="S3.SS1.p3.3.m2.1.1.cmml">r</mi><mo id="S3.SS1.p3.3.m2.2.2.1.3" xref="S3.SS1.p3.3.m2.2.2.2.cmml">,</mo><msup id="S3.SS1.p3.3.m2.2.2.1.1" xref="S3.SS1.p3.3.m2.2.2.1.1.cmml"><mi id="S3.SS1.p3.3.m2.2.2.1.1.2" xref="S3.SS1.p3.3.m2.2.2.1.1.2.cmml">o</mi><mo id="S3.SS1.p3.3.m2.2.2.1.1.3" xref="S3.SS1.p3.3.m2.2.2.1.1.3.cmml">∗</mo></msup><mo stretchy="false" id="S3.SS1.p3.3.m2.2.2.1.4" xref="S3.SS1.p3.3.m2.2.2.2.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.3.m2.2b"><interval closure="open" id="S3.SS1.p3.3.m2.2.2.2.cmml" xref="S3.SS1.p3.3.m2.2.2.1"><ci id="S3.SS1.p3.3.m2.1.1.cmml" xref="S3.SS1.p3.3.m2.1.1">𝑟</ci><apply id="S3.SS1.p3.3.m2.2.2.1.1.cmml" xref="S3.SS1.p3.3.m2.2.2.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.3.m2.2.2.1.1.1.cmml" xref="S3.SS1.p3.3.m2.2.2.1.1">superscript</csymbol><ci id="S3.SS1.p3.3.m2.2.2.1.1.2.cmml" xref="S3.SS1.p3.3.m2.2.2.1.1.2">𝑜</ci><times id="S3.SS1.p3.3.m2.2.2.1.1.3.cmml" xref="S3.SS1.p3.3.m2.2.2.1.1.3"></times></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.3.m2.2c">(r,o^{*})</annotation></semantics></math> as a property of <math id="S3.SS1.p3.4.m3.1" class="ltx_Math" alttext="s" display="inline"><semantics id="S3.SS1.p3.4.m3.1a"><mi id="S3.SS1.p3.4.m3.1.1" xref="S3.SS1.p3.4.m3.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.4.m3.1b"><ci id="S3.SS1.p3.4.m3.1.1.cmml" xref="S3.SS1.p3.4.m3.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.4.m3.1c">s</annotation></semantics></math>.
We set <math id="S3.SS1.p3.5.m4.1" class="ltx_Math" alttext="v_{*}=\operatorname*{argmin}_{z}\mathcal{L}(z)" display="inline"><semantics id="S3.SS1.p3.5.m4.1a"><mrow id="S3.SS1.p3.5.m4.1.2" xref="S3.SS1.p3.5.m4.1.2.cmml"><msub id="S3.SS1.p3.5.m4.1.2.2" xref="S3.SS1.p3.5.m4.1.2.2.cmml"><mi id="S3.SS1.p3.5.m4.1.2.2.2" xref="S3.SS1.p3.5.m4.1.2.2.2.cmml">v</mi><mo id="S3.SS1.p3.5.m4.1.2.2.3" xref="S3.SS1.p3.5.m4.1.2.2.3.cmml">∗</mo></msub><mo rspace="0.1389em" id="S3.SS1.p3.5.m4.1.2.1" xref="S3.SS1.p3.5.m4.1.2.1.cmml">=</mo><mrow id="S3.SS1.p3.5.m4.1.2.3" xref="S3.SS1.p3.5.m4.1.2.3.cmml"><mrow id="S3.SS1.p3.5.m4.1.2.3.2" xref="S3.SS1.p3.5.m4.1.2.3.2.cmml"><msub id="S3.SS1.p3.5.m4.1.2.3.2.1" xref="S3.SS1.p3.5.m4.1.2.3.2.1.cmml"><mo lspace="0.1389em" rspace="0.167em" id="S3.SS1.p3.5.m4.1.2.3.2.1.2" xref="S3.SS1.p3.5.m4.1.2.3.2.1.2.cmml">argmin</mo><mi id="S3.SS1.p3.5.m4.1.2.3.2.1.3" xref="S3.SS1.p3.5.m4.1.2.3.2.1.3.cmml">z</mi></msub><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p3.5.m4.1.2.3.2.2" xref="S3.SS1.p3.5.m4.1.2.3.2.2.cmml">ℒ</mi></mrow><mo lspace="0em" rspace="0em" id="S3.SS1.p3.5.m4.1.2.3.1" xref="S3.SS1.p3.5.m4.1.2.3.1.cmml">​</mo><mrow id="S3.SS1.p3.5.m4.1.2.3.3.2" xref="S3.SS1.p3.5.m4.1.2.3.cmml"><mo stretchy="false" id="S3.SS1.p3.5.m4.1.2.3.3.2.1" xref="S3.SS1.p3.5.m4.1.2.3.cmml">(</mo><mi id="S3.SS1.p3.5.m4.1.1" xref="S3.SS1.p3.5.m4.1.1.cmml">z</mi><mo stretchy="false" id="S3.SS1.p3.5.m4.1.2.3.3.2.2" xref="S3.SS1.p3.5.m4.1.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.5.m4.1b"><apply id="S3.SS1.p3.5.m4.1.2.cmml" xref="S3.SS1.p3.5.m4.1.2"><eq id="S3.SS1.p3.5.m4.1.2.1.cmml" xref="S3.SS1.p3.5.m4.1.2.1"></eq><apply id="S3.SS1.p3.5.m4.1.2.2.cmml" xref="S3.SS1.p3.5.m4.1.2.2"><csymbol cd="ambiguous" id="S3.SS1.p3.5.m4.1.2.2.1.cmml" xref="S3.SS1.p3.5.m4.1.2.2">subscript</csymbol><ci id="S3.SS1.p3.5.m4.1.2.2.2.cmml" xref="S3.SS1.p3.5.m4.1.2.2.2">𝑣</ci><times id="S3.SS1.p3.5.m4.1.2.2.3.cmml" xref="S3.SS1.p3.5.m4.1.2.2.3"></times></apply><apply id="S3.SS1.p3.5.m4.1.2.3.cmml" xref="S3.SS1.p3.5.m4.1.2.3"><times id="S3.SS1.p3.5.m4.1.2.3.1.cmml" xref="S3.SS1.p3.5.m4.1.2.3.1"></times><apply id="S3.SS1.p3.5.m4.1.2.3.2.cmml" xref="S3.SS1.p3.5.m4.1.2.3.2"><apply id="S3.SS1.p3.5.m4.1.2.3.2.1.cmml" xref="S3.SS1.p3.5.m4.1.2.3.2.1"><csymbol cd="ambiguous" id="S3.SS1.p3.5.m4.1.2.3.2.1.1.cmml" xref="S3.SS1.p3.5.m4.1.2.3.2.1">subscript</csymbol><ci id="S3.SS1.p3.5.m4.1.2.3.2.1.2.cmml" xref="S3.SS1.p3.5.m4.1.2.3.2.1.2">argmin</ci><ci id="S3.SS1.p3.5.m4.1.2.3.2.1.3.cmml" xref="S3.SS1.p3.5.m4.1.2.3.2.1.3">𝑧</ci></apply><ci id="S3.SS1.p3.5.m4.1.2.3.2.2.cmml" xref="S3.SS1.p3.5.m4.1.2.3.2.2">ℒ</ci></apply><ci id="S3.SS1.p3.5.m4.1.1.cmml" xref="S3.SS1.p3.5.m4.1.1">𝑧</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.5.m4.1c">v_{*}=\operatorname*{argmin}_{z}\mathcal{L}(z)</annotation></semantics></math>, where the objective <math id="S3.SS1.p3.6.m5.1" class="ltx_Math" alttext="\mathcal{L}(z)" display="inline"><semantics id="S3.SS1.p3.6.m5.1a"><mrow id="S3.SS1.p3.6.m5.1.2" xref="S3.SS1.p3.6.m5.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p3.6.m5.1.2.2" xref="S3.SS1.p3.6.m5.1.2.2.cmml">ℒ</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p3.6.m5.1.2.1" xref="S3.SS1.p3.6.m5.1.2.1.cmml">​</mo><mrow id="S3.SS1.p3.6.m5.1.2.3.2" xref="S3.SS1.p3.6.m5.1.2.cmml"><mo stretchy="false" id="S3.SS1.p3.6.m5.1.2.3.2.1" xref="S3.SS1.p3.6.m5.1.2.cmml">(</mo><mi id="S3.SS1.p3.6.m5.1.1" xref="S3.SS1.p3.6.m5.1.1.cmml">z</mi><mo stretchy="false" id="S3.SS1.p3.6.m5.1.2.3.2.2" xref="S3.SS1.p3.6.m5.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.6.m5.1b"><apply id="S3.SS1.p3.6.m5.1.2.cmml" xref="S3.SS1.p3.6.m5.1.2"><times id="S3.SS1.p3.6.m5.1.2.1.cmml" xref="S3.SS1.p3.6.m5.1.2.1"></times><ci id="S3.SS1.p3.6.m5.1.2.2.cmml" xref="S3.SS1.p3.6.m5.1.2.2">ℒ</ci><ci id="S3.SS1.p3.6.m5.1.1.cmml" xref="S3.SS1.p3.6.m5.1.1">𝑧</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.6.m5.1c">\mathcal{L}(z)</annotation></semantics></math> is:</p>
<table id="A10.EGx4" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S3.E4"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E4.m1.7" class="ltx_math_unparsed" alttext="\displaystyle\frac{1}{N}\sum_{j=1}^{N}\underbrace{-\log\mathbb{P}_{G(m^{(l^{*})}_{i}:=z)}\left[o^{*}\mid x_{j}+p\,\right]}_{\text{(a) Maximizing $o^{*}$ probability}}\;+\;\underbrace{D_{\mathrm{KL}}\left(\mathbb{P}_{G(m^{(l^{*})}_{i^{\prime}}:=z)}\left[x\mid p^{\prime}\right]\big{\|}\mathbb{P}_{G}\left[x\mid p^{\prime}\right]\right)}_{\text{(b) Controlling essence drift}}." display="inline"><semantics id="S3.E4.m1.7a"><mrow id="S3.E4.m1.7.7.1"><mrow id="S3.E4.m1.7.7.1.1"><mrow id="S3.E4.m1.7.7.1.1.2"><mstyle displaystyle="true" id="S3.E4.m1.7.7.1.1.2.2"><mfrac id="S3.E4.m1.7.7.1.1.2.2a"><mn id="S3.E4.m1.7.7.1.1.2.2.2">1</mn><mi id="S3.E4.m1.7.7.1.1.2.2.3">N</mi></mfrac></mstyle><mo lspace="0em" rspace="0em" id="S3.E4.m1.7.7.1.1.2.1">​</mo><mrow id="S3.E4.m1.7.7.1.1.2.3"><mstyle displaystyle="true" id="S3.E4.m1.7.7.1.1.2.3.1"><munderover id="S3.E4.m1.7.7.1.1.2.3.1a"><mo movablelimits="false" id="S3.E4.m1.7.7.1.1.2.3.1.2.2">∑</mo><mrow id="S3.E4.m1.7.7.1.1.2.3.1.2.3"><mi id="S3.E4.m1.7.7.1.1.2.3.1.2.3.2">j</mi><mo id="S3.E4.m1.7.7.1.1.2.3.1.2.3.1">=</mo><mn id="S3.E4.m1.7.7.1.1.2.3.1.2.3.3">1</mn></mrow><mi id="S3.E4.m1.7.7.1.1.2.3.1.3">N</mi></munderover></mstyle><munder id="S3.E4.m1.7.7.1.1.2.3.2"><munder accentunder="true" id="S3.E4.m1.4.4"><mrow id="S3.E4.m1.4.4.3"><mo rspace="0.167em" id="S3.E4.m1.4.4.3a">−</mo><mrow id="S3.E4.m1.4.4.3.3"><mrow id="S3.E4.m1.4.4.3.3.3"><mi id="S3.E4.m1.4.4.3.3.3.1">log</mi><mo lspace="0.167em" id="S3.E4.m1.4.4.3.3.3a">⁡</mo><msub id="S3.E4.m1.4.4.3.3.3.2"><mi id="S3.E4.m1.4.4.3.3.3.2.2">ℙ</mi><mrow id="S3.E4.m1.3.3.2.2.2"><mi id="S3.E4.m1.3.3.2.2.2.4">G</mi><mo lspace="0em" rspace="0em" id="S3.E4.m1.3.3.2.2.2.3">​</mo><mrow id="S3.E4.m1.3.3.2.2.2.2.1"><mo stretchy="false" id="S3.E4.m1.3.3.2.2.2.2.1.2">(</mo><mrow id="S3.E4.m1.3.3.2.2.2.2.1.1"><msubsup id="S3.E4.m1.3.3.2.2.2.2.1.1.2"><mi id="S3.E4.m1.3.3.2.2.2.2.1.1.2.2.2">m</mi><mi id="S3.E4.m1.3.3.2.2.2.2.1.1.2.3">i</mi><mrow id="S3.E4.m1.2.2.1.1.1.1.1.1"><mo stretchy="false" id="S3.E4.m1.2.2.1.1.1.1.1.1.2">(</mo><msup id="S3.E4.m1.2.2.1.1.1.1.1.1.1"><mi id="S3.E4.m1.2.2.1.1.1.1.1.1.1.2">l</mi><mo id="S3.E4.m1.2.2.1.1.1.1.1.1.1.3">∗</mo></msup><mo stretchy="false" id="S3.E4.m1.2.2.1.1.1.1.1.1.3">)</mo></mrow></msubsup><mo lspace="0.278em" rspace="0.278em" id="S3.E4.m1.3.3.2.2.2.2.1.1.1">:=</mo><mi id="S3.E4.m1.3.3.2.2.2.2.1.1.3">z</mi></mrow><mo stretchy="false" id="S3.E4.m1.3.3.2.2.2.2.1.3">)</mo></mrow></mrow></msub></mrow><mo lspace="0em" rspace="0em" id="S3.E4.m1.4.4.3.3.2">​</mo><mrow id="S3.E4.m1.4.4.3.3.1.1"><mo id="S3.E4.m1.4.4.3.3.1.1.2">[</mo><mrow id="S3.E4.m1.4.4.3.3.1.1.1"><msup id="S3.E4.m1.4.4.3.3.1.1.1.2"><mi id="S3.E4.m1.4.4.3.3.1.1.1.2.2">o</mi><mo id="S3.E4.m1.4.4.3.3.1.1.1.2.3">∗</mo></msup><mo id="S3.E4.m1.4.4.3.3.1.1.1.1">∣</mo><mrow id="S3.E4.m1.4.4.3.3.1.1.1.3"><msub id="S3.E4.m1.4.4.3.3.1.1.1.3.2"><mi id="S3.E4.m1.4.4.3.3.1.1.1.3.2.2">x</mi><mi id="S3.E4.m1.4.4.3.3.1.1.1.3.2.3">j</mi></msub><mo id="S3.E4.m1.4.4.3.3.1.1.1.3.1">+</mo><mi id="S3.E4.m1.4.4.3.3.1.1.1.3.3">p</mi></mrow></mrow><mo lspace="0.170em" id="S3.E4.m1.4.4.3.3.1.1.3">]</mo></mrow></mrow></mrow><mo id="S3.E4.m1.4.4.4">⏟</mo></munder><mrow id="S3.E4.m1.1.1.1a"><mtext id="S3.E4.m1.1.1.1aa">(a) Maximizing </mtext><msup id="S3.E4.m1.1.1.1.1.1.m1.1.1"><mi id="S3.E4.m1.1.1.1.1.1.m1.1.1.2">o</mi><mo mathsize="98%" id="S3.E4.m1.1.1.1.1.1.m1.1.1.3">∗</mo></msup><mtext id="S3.E4.m1.1.1.1ab"> probability</mtext></mrow></munder></mrow></mrow><mo rspace="0.502em" id="S3.E4.m1.7.7.1.1.1">+</mo><munder id="S3.E4.m1.7.7.1.1.3"><munder accentunder="true" id="S3.E4.m1.6.6"><mrow id="S3.E4.m1.6.6.2"><msub id="S3.E4.m1.6.6.2.3"><mi id="S3.E4.m1.6.6.2.3.2">D</mi><mi id="S3.E4.m1.6.6.2.3.3">KL</mi></msub><mrow id="S3.E4.m1.6.6.2.4"><mo id="S3.E4.m1.6.6.2.4.1">(</mo><msub id="S3.E4.m1.6.6.2.4.2"><mi id="S3.E4.m1.6.6.2.4.2.2">ℙ</mi><mrow id="S3.E4.m1.6.6.2.2.2"><mi id="S3.E4.m1.6.6.2.2.2.4">G</mi><mo lspace="0em" rspace="0em" id="S3.E4.m1.6.6.2.2.2.3">​</mo><mrow id="S3.E4.m1.6.6.2.2.2.2.1"><mo stretchy="false" id="S3.E4.m1.6.6.2.2.2.2.1.2">(</mo><mrow id="S3.E4.m1.6.6.2.2.2.2.1.1"><msubsup id="S3.E4.m1.6.6.2.2.2.2.1.1.2"><mi id="S3.E4.m1.6.6.2.2.2.2.1.1.2.2.2">m</mi><msup id="S3.E4.m1.6.6.2.2.2.2.1.1.2.3"><mi id="S3.E4.m1.6.6.2.2.2.2.1.1.2.3.2">i</mi><mo id="S3.E4.m1.6.6.2.2.2.2.1.1.2.3.3">′</mo></msup><mrow id="S3.E4.m1.5.5.1.1.1.1.1.1"><mo stretchy="false" id="S3.E4.m1.5.5.1.1.1.1.1.1.2">(</mo><msup id="S3.E4.m1.5.5.1.1.1.1.1.1.1"><mi id="S3.E4.m1.5.5.1.1.1.1.1.1.1.2">l</mi><mo id="S3.E4.m1.5.5.1.1.1.1.1.1.1.3">∗</mo></msup><mo stretchy="false" id="S3.E4.m1.5.5.1.1.1.1.1.1.3">)</mo></mrow></msubsup><mo lspace="0.278em" rspace="0.278em" id="S3.E4.m1.6.6.2.2.2.2.1.1.1">:=</mo><mi id="S3.E4.m1.6.6.2.2.2.2.1.1.3">z</mi></mrow><mo stretchy="false" id="S3.E4.m1.6.6.2.2.2.2.1.3">)</mo></mrow></mrow></msub><mrow id="S3.E4.m1.6.6.2.4.3"><mo id="S3.E4.m1.6.6.2.4.3.1">[</mo><mi id="S3.E4.m1.6.6.2.4.3.2">x</mi><mo lspace="0em" rspace="0.167em" id="S3.E4.m1.6.6.2.4.3.3">∣</mo><msup id="S3.E4.m1.6.6.2.4.3.4"><mi id="S3.E4.m1.6.6.2.4.3.4.2">p</mi><mo id="S3.E4.m1.6.6.2.4.3.4.3">′</mo></msup><mo id="S3.E4.m1.6.6.2.4.3.5">]</mo></mrow><mo lspace="0em" mathsize="120%" rspace="0.167em" id="S3.E4.m1.6.6.2.4.4">∥</mo><msub id="S3.E4.m1.6.6.2.4.5"><mi id="S3.E4.m1.6.6.2.4.5.2">ℙ</mi><mi id="S3.E4.m1.6.6.2.4.5.3">G</mi></msub><mrow id="S3.E4.m1.6.6.2.4.6"><mo id="S3.E4.m1.6.6.2.4.6.1">[</mo><mi id="S3.E4.m1.6.6.2.4.6.2">x</mi><mo lspace="0em" rspace="0.167em" id="S3.E4.m1.6.6.2.4.6.3">∣</mo><msup id="S3.E4.m1.6.6.2.4.6.4"><mi id="S3.E4.m1.6.6.2.4.6.4.2">p</mi><mo id="S3.E4.m1.6.6.2.4.6.4.3">′</mo></msup><mo id="S3.E4.m1.6.6.2.4.6.5">]</mo></mrow><mo id="S3.E4.m1.6.6.2.4.7">)</mo></mrow></mrow><mo id="S3.E4.m1.6.6.3">⏟</mo></munder><mtext id="S3.E4.m1.7.7.1.1.3.2">(b) Controlling essence drift</mtext></munder></mrow><mo lspace="0em" id="S3.E4.m1.7.7.1.2">.</mo></mrow><annotation encoding="application/x-tex" id="S3.E4.m1.7b">\displaystyle\frac{1}{N}\sum_{j=1}^{N}\underbrace{-\log\mathbb{P}_{G(m^{(l^{*})}_{i}:=z)}\left[o^{*}\mid x_{j}+p\,\right]}_{\text{(a) Maximizing $o^{*}$ probability}}\;+\;\underbrace{D_{\mathrm{KL}}\left(\mathbb{P}_{G(m^{(l^{*})}_{i^{\prime}}:=z)}\left[x\mid p^{\prime}\right]\big{\|}\mathbb{P}_{G}\left[x\mid p^{\prime}\right]\right)}_{\text{(b) Controlling essence drift}}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(4)</span></td>
</tr></tbody>
</table>
<p id="S3.SS1.p3.18" class="ltx_p">The first term (Eqn. <a href="#S3.E4" title="In 3.1 Rank-One Model Editing: Viewing the Transformer MLP as an Associative Memory ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>a) seeks a vector <math id="S3.SS1.p3.7.m1.1" class="ltx_Math" alttext="z" display="inline"><semantics id="S3.SS1.p3.7.m1.1a"><mi id="S3.SS1.p3.7.m1.1.1" xref="S3.SS1.p3.7.m1.1.1.cmml">z</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.7.m1.1b"><ci id="S3.SS1.p3.7.m1.1.1.cmml" xref="S3.SS1.p3.7.m1.1.1">𝑧</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.7.m1.1c">z</annotation></semantics></math> that, when substituted as the output of the MLP at the token <math id="S3.SS1.p3.8.m2.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S3.SS1.p3.8.m2.1a"><mi id="S3.SS1.p3.8.m2.1.1" xref="S3.SS1.p3.8.m2.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.8.m2.1b"><ci id="S3.SS1.p3.8.m2.1.1.cmml" xref="S3.SS1.p3.8.m2.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.8.m2.1c">i</annotation></semantics></math> at the end of the subject (notated <math id="S3.SS1.p3.9.m3.2" class="ltx_Math" alttext="\smash{G(m^{(l^{*})}_{i}:=z)}" display="inline"><semantics id="S3.SS1.p3.9.m3.2a"><mrow id="S3.SS1.p3.9.m3.2.2" xref="S3.SS1.p3.9.m3.2.2.cmml"><mi id="S3.SS1.p3.9.m3.2.2.3" xref="S3.SS1.p3.9.m3.2.2.3.cmml">G</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p3.9.m3.2.2.2" xref="S3.SS1.p3.9.m3.2.2.2.cmml">​</mo><mrow id="S3.SS1.p3.9.m3.2.2.1.1" xref="S3.SS1.p3.9.m3.2.2.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p3.9.m3.2.2.1.1.2" xref="S3.SS1.p3.9.m3.2.2.1.1.1.cmml">(</mo><mrow id="S3.SS1.p3.9.m3.2.2.1.1.1" xref="S3.SS1.p3.9.m3.2.2.1.1.1.cmml"><msubsup id="S3.SS1.p3.9.m3.2.2.1.1.1.2" xref="S3.SS1.p3.9.m3.2.2.1.1.1.2.cmml"><mi id="S3.SS1.p3.9.m3.2.2.1.1.1.2.2.2" xref="S3.SS1.p3.9.m3.2.2.1.1.1.2.2.2.cmml">m</mi><mi id="S3.SS1.p3.9.m3.2.2.1.1.1.2.3" xref="S3.SS1.p3.9.m3.2.2.1.1.1.2.3.cmml">i</mi><mrow id="S3.SS1.p3.9.m3.1.1.1.1" xref="S3.SS1.p3.9.m3.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p3.9.m3.1.1.1.1.2" xref="S3.SS1.p3.9.m3.1.1.1.1.1.cmml">(</mo><msup id="S3.SS1.p3.9.m3.1.1.1.1.1" xref="S3.SS1.p3.9.m3.1.1.1.1.1.cmml"><mi id="S3.SS1.p3.9.m3.1.1.1.1.1.2" xref="S3.SS1.p3.9.m3.1.1.1.1.1.2.cmml">l</mi><mo id="S3.SS1.p3.9.m3.1.1.1.1.1.3" xref="S3.SS1.p3.9.m3.1.1.1.1.1.3.cmml">∗</mo></msup><mo stretchy="false" id="S3.SS1.p3.9.m3.1.1.1.1.3" xref="S3.SS1.p3.9.m3.1.1.1.1.1.cmml">)</mo></mrow></msubsup><mo lspace="0.278em" rspace="0.278em" id="S3.SS1.p3.9.m3.2.2.1.1.1.1" xref="S3.SS1.p3.9.m3.2.2.1.1.1.1.cmml">:=</mo><mi id="S3.SS1.p3.9.m3.2.2.1.1.1.3" xref="S3.SS1.p3.9.m3.2.2.1.1.1.3.cmml">z</mi></mrow><mo stretchy="false" id="S3.SS1.p3.9.m3.2.2.1.1.3" xref="S3.SS1.p3.9.m3.2.2.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.9.m3.2b"><apply id="S3.SS1.p3.9.m3.2.2.cmml" xref="S3.SS1.p3.9.m3.2.2"><times id="S3.SS1.p3.9.m3.2.2.2.cmml" xref="S3.SS1.p3.9.m3.2.2.2"></times><ci id="S3.SS1.p3.9.m3.2.2.3.cmml" xref="S3.SS1.p3.9.m3.2.2.3">𝐺</ci><apply id="S3.SS1.p3.9.m3.2.2.1.1.1.cmml" xref="S3.SS1.p3.9.m3.2.2.1.1"><csymbol cd="latexml" id="S3.SS1.p3.9.m3.2.2.1.1.1.1.cmml" xref="S3.SS1.p3.9.m3.2.2.1.1.1.1">assign</csymbol><apply id="S3.SS1.p3.9.m3.2.2.1.1.1.2.cmml" xref="S3.SS1.p3.9.m3.2.2.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS1.p3.9.m3.2.2.1.1.1.2.1.cmml" xref="S3.SS1.p3.9.m3.2.2.1.1.1.2">subscript</csymbol><apply id="S3.SS1.p3.9.m3.2.2.1.1.1.2.2.cmml" xref="S3.SS1.p3.9.m3.2.2.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS1.p3.9.m3.2.2.1.1.1.2.2.1.cmml" xref="S3.SS1.p3.9.m3.2.2.1.1.1.2">superscript</csymbol><ci id="S3.SS1.p3.9.m3.2.2.1.1.1.2.2.2.cmml" xref="S3.SS1.p3.9.m3.2.2.1.1.1.2.2.2">𝑚</ci><apply id="S3.SS1.p3.9.m3.1.1.1.1.1.cmml" xref="S3.SS1.p3.9.m3.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.9.m3.1.1.1.1.1.1.cmml" xref="S3.SS1.p3.9.m3.1.1.1.1">superscript</csymbol><ci id="S3.SS1.p3.9.m3.1.1.1.1.1.2.cmml" xref="S3.SS1.p3.9.m3.1.1.1.1.1.2">𝑙</ci><times id="S3.SS1.p3.9.m3.1.1.1.1.1.3.cmml" xref="S3.SS1.p3.9.m3.1.1.1.1.1.3"></times></apply></apply><ci id="S3.SS1.p3.9.m3.2.2.1.1.1.2.3.cmml" xref="S3.SS1.p3.9.m3.2.2.1.1.1.2.3">𝑖</ci></apply><ci id="S3.SS1.p3.9.m3.2.2.1.1.1.3.cmml" xref="S3.SS1.p3.9.m3.2.2.1.1.1.3">𝑧</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.9.m3.2c">\smash{G(m^{(l^{*})}_{i}:=z)}</annotation></semantics></math>), will cause the network to predict the target object <math id="S3.SS1.p3.10.m4.1" class="ltx_Math" alttext="o^{*}" display="inline"><semantics id="S3.SS1.p3.10.m4.1a"><msup id="S3.SS1.p3.10.m4.1.1" xref="S3.SS1.p3.10.m4.1.1.cmml"><mi id="S3.SS1.p3.10.m4.1.1.2" xref="S3.SS1.p3.10.m4.1.1.2.cmml">o</mi><mo id="S3.SS1.p3.10.m4.1.1.3" xref="S3.SS1.p3.10.m4.1.1.3.cmml">∗</mo></msup><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.10.m4.1b"><apply id="S3.SS1.p3.10.m4.1.1.cmml" xref="S3.SS1.p3.10.m4.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.10.m4.1.1.1.cmml" xref="S3.SS1.p3.10.m4.1.1">superscript</csymbol><ci id="S3.SS1.p3.10.m4.1.1.2.cmml" xref="S3.SS1.p3.10.m4.1.1.2">𝑜</ci><times id="S3.SS1.p3.10.m4.1.1.3.cmml" xref="S3.SS1.p3.10.m4.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.10.m4.1c">o^{*}</annotation></semantics></math> in response to the factual prompt <math id="S3.SS1.p3.11.m5.1" class="ltx_Math" alttext="p" display="inline"><semantics id="S3.SS1.p3.11.m5.1a"><mi id="S3.SS1.p3.11.m5.1.1" xref="S3.SS1.p3.11.m5.1.1.cmml">p</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.11.m5.1b"><ci id="S3.SS1.p3.11.m5.1.1.cmml" xref="S3.SS1.p3.11.m5.1.1">𝑝</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.11.m5.1c">p</annotation></semantics></math>.
The second term (Eqn. <a href="#S3.E4" title="In 3.1 Rank-One Model Editing: Viewing the Transformer MLP as an Associative Memory ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>b) minimizes the KL divergence of predictions for the prompt <math id="S3.SS1.p3.12.m6.1" class="ltx_Math" alttext="p^{\prime}" display="inline"><semantics id="S3.SS1.p3.12.m6.1a"><msup id="S3.SS1.p3.12.m6.1.1" xref="S3.SS1.p3.12.m6.1.1.cmml"><mi id="S3.SS1.p3.12.m6.1.1.2" xref="S3.SS1.p3.12.m6.1.1.2.cmml">p</mi><mo id="S3.SS1.p3.12.m6.1.1.3" xref="S3.SS1.p3.12.m6.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.12.m6.1b"><apply id="S3.SS1.p3.12.m6.1.1.cmml" xref="S3.SS1.p3.12.m6.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.12.m6.1.1.1.cmml" xref="S3.SS1.p3.12.m6.1.1">superscript</csymbol><ci id="S3.SS1.p3.12.m6.1.1.2.cmml" xref="S3.SS1.p3.12.m6.1.1.2">𝑝</ci><ci id="S3.SS1.p3.12.m6.1.1.3.cmml" xref="S3.SS1.p3.12.m6.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.12.m6.1c">p^{\prime}</annotation></semantics></math> (of the form “{subject} is a”) to the unchanged model, which helps preserve the model’s understanding of the subject’s essence.
To be clear, the optimization does <span id="S3.SS1.p3.18.1" class="ltx_text ltx_font_italic">not</span> directly alter model weights; it identifies a vector representation <math id="S3.SS1.p3.13.m7.1" class="ltx_Math" alttext="v_{*}" display="inline"><semantics id="S3.SS1.p3.13.m7.1a"><msub id="S3.SS1.p3.13.m7.1.1" xref="S3.SS1.p3.13.m7.1.1.cmml"><mi id="S3.SS1.p3.13.m7.1.1.2" xref="S3.SS1.p3.13.m7.1.1.2.cmml">v</mi><mo id="S3.SS1.p3.13.m7.1.1.3" xref="S3.SS1.p3.13.m7.1.1.3.cmml">∗</mo></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.13.m7.1b"><apply id="S3.SS1.p3.13.m7.1.1.cmml" xref="S3.SS1.p3.13.m7.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.13.m7.1.1.1.cmml" xref="S3.SS1.p3.13.m7.1.1">subscript</csymbol><ci id="S3.SS1.p3.13.m7.1.1.2.cmml" xref="S3.SS1.p3.13.m7.1.1.2">𝑣</ci><times id="S3.SS1.p3.13.m7.1.1.3.cmml" xref="S3.SS1.p3.13.m7.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.13.m7.1c">v_{*}</annotation></semantics></math> that, when output at the targeted MLP module, represents the new property <math id="S3.SS1.p3.14.m8.2" class="ltx_Math" alttext="(r,o^{*})" display="inline"><semantics id="S3.SS1.p3.14.m8.2a"><mrow id="S3.SS1.p3.14.m8.2.2.1" xref="S3.SS1.p3.14.m8.2.2.2.cmml"><mo stretchy="false" id="S3.SS1.p3.14.m8.2.2.1.2" xref="S3.SS1.p3.14.m8.2.2.2.cmml">(</mo><mi id="S3.SS1.p3.14.m8.1.1" xref="S3.SS1.p3.14.m8.1.1.cmml">r</mi><mo id="S3.SS1.p3.14.m8.2.2.1.3" xref="S3.SS1.p3.14.m8.2.2.2.cmml">,</mo><msup id="S3.SS1.p3.14.m8.2.2.1.1" xref="S3.SS1.p3.14.m8.2.2.1.1.cmml"><mi id="S3.SS1.p3.14.m8.2.2.1.1.2" xref="S3.SS1.p3.14.m8.2.2.1.1.2.cmml">o</mi><mo id="S3.SS1.p3.14.m8.2.2.1.1.3" xref="S3.SS1.p3.14.m8.2.2.1.1.3.cmml">∗</mo></msup><mo stretchy="false" id="S3.SS1.p3.14.m8.2.2.1.4" xref="S3.SS1.p3.14.m8.2.2.2.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.14.m8.2b"><interval closure="open" id="S3.SS1.p3.14.m8.2.2.2.cmml" xref="S3.SS1.p3.14.m8.2.2.1"><ci id="S3.SS1.p3.14.m8.1.1.cmml" xref="S3.SS1.p3.14.m8.1.1">𝑟</ci><apply id="S3.SS1.p3.14.m8.2.2.1.1.cmml" xref="S3.SS1.p3.14.m8.2.2.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.14.m8.2.2.1.1.1.cmml" xref="S3.SS1.p3.14.m8.2.2.1.1">superscript</csymbol><ci id="S3.SS1.p3.14.m8.2.2.1.1.2.cmml" xref="S3.SS1.p3.14.m8.2.2.1.1.2">𝑜</ci><times id="S3.SS1.p3.14.m8.2.2.1.1.3.cmml" xref="S3.SS1.p3.14.m8.2.2.1.1.3"></times></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.14.m8.2c">(r,o^{*})</annotation></semantics></math> for the subject <math id="S3.SS1.p3.15.m9.1" class="ltx_Math" alttext="s" display="inline"><semantics id="S3.SS1.p3.15.m9.1a"><mi id="S3.SS1.p3.15.m9.1.1" xref="S3.SS1.p3.15.m9.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.15.m9.1b"><ci id="S3.SS1.p3.15.m9.1.1.cmml" xref="S3.SS1.p3.15.m9.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.15.m9.1c">s</annotation></semantics></math>.
Note that, similar to <math id="S3.SS1.p3.16.m10.1" class="ltx_Math" alttext="k_{*}" display="inline"><semantics id="S3.SS1.p3.16.m10.1a"><msub id="S3.SS1.p3.16.m10.1.1" xref="S3.SS1.p3.16.m10.1.1.cmml"><mi id="S3.SS1.p3.16.m10.1.1.2" xref="S3.SS1.p3.16.m10.1.1.2.cmml">k</mi><mo id="S3.SS1.p3.16.m10.1.1.3" xref="S3.SS1.p3.16.m10.1.1.3.cmml">∗</mo></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.16.m10.1b"><apply id="S3.SS1.p3.16.m10.1.1.cmml" xref="S3.SS1.p3.16.m10.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.16.m10.1.1.1.cmml" xref="S3.SS1.p3.16.m10.1.1">subscript</csymbol><ci id="S3.SS1.p3.16.m10.1.1.2.cmml" xref="S3.SS1.p3.16.m10.1.1.2">𝑘</ci><times id="S3.SS1.p3.16.m10.1.1.3.cmml" xref="S3.SS1.p3.16.m10.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.16.m10.1c">k_{*}</annotation></semantics></math> selection, <math id="S3.SS1.p3.17.m11.1" class="ltx_Math" alttext="v_{*}" display="inline"><semantics id="S3.SS1.p3.17.m11.1a"><msub id="S3.SS1.p3.17.m11.1.1" xref="S3.SS1.p3.17.m11.1.1.cmml"><mi id="S3.SS1.p3.17.m11.1.1.2" xref="S3.SS1.p3.17.m11.1.1.2.cmml">v</mi><mo id="S3.SS1.p3.17.m11.1.1.3" xref="S3.SS1.p3.17.m11.1.1.3.cmml">∗</mo></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.17.m11.1b"><apply id="S3.SS1.p3.17.m11.1.1.cmml" xref="S3.SS1.p3.17.m11.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.17.m11.1.1.1.cmml" xref="S3.SS1.p3.17.m11.1.1">subscript</csymbol><ci id="S3.SS1.p3.17.m11.1.1.2.cmml" xref="S3.SS1.p3.17.m11.1.1.2">𝑣</ci><times id="S3.SS1.p3.17.m11.1.1.3.cmml" xref="S3.SS1.p3.17.m11.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.17.m11.1c">v_{*}</annotation></semantics></math> optimization also uses the random prefix texts <math id="S3.SS1.p3.18.m12.1" class="ltx_Math" alttext="x_{j}" display="inline"><semantics id="S3.SS1.p3.18.m12.1a"><msub id="S3.SS1.p3.18.m12.1.1" xref="S3.SS1.p3.18.m12.1.1.cmml"><mi id="S3.SS1.p3.18.m12.1.1.2" xref="S3.SS1.p3.18.m12.1.1.2.cmml">x</mi><mi id="S3.SS1.p3.18.m12.1.1.3" xref="S3.SS1.p3.18.m12.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.18.m12.1b"><apply id="S3.SS1.p3.18.m12.1.1.cmml" xref="S3.SS1.p3.18.m12.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.18.m12.1.1.1.cmml" xref="S3.SS1.p3.18.m12.1.1">subscript</csymbol><ci id="S3.SS1.p3.18.m12.1.1.2.cmml" xref="S3.SS1.p3.18.m12.1.1.2">𝑥</ci><ci id="S3.SS1.p3.18.m12.1.1.3.cmml" xref="S3.SS1.p3.18.m12.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.18.m12.1c">x_{j}</annotation></semantics></math> to encourage robustness under differing contexts.</p>
</div>
<div id="S3.SS1.p4" class="ltx_para">
<p id="S3.SS1.p4.4" class="ltx_p"><span id="S3.SS1.p4.4.1" class="ltx_text ltx_font_bold">Step 3: Inserting the Fact.</span>
Once we have computed the pair (<math id="S3.SS1.p4.1.m1.1" class="ltx_Math" alttext="k_{*}" display="inline"><semantics id="S3.SS1.p4.1.m1.1a"><msub id="S3.SS1.p4.1.m1.1.1" xref="S3.SS1.p4.1.m1.1.1.cmml"><mi id="S3.SS1.p4.1.m1.1.1.2" xref="S3.SS1.p4.1.m1.1.1.2.cmml">k</mi><mo id="S3.SS1.p4.1.m1.1.1.3" xref="S3.SS1.p4.1.m1.1.1.3.cmml">∗</mo></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p4.1.m1.1b"><apply id="S3.SS1.p4.1.m1.1.1.cmml" xref="S3.SS1.p4.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p4.1.m1.1.1.1.cmml" xref="S3.SS1.p4.1.m1.1.1">subscript</csymbol><ci id="S3.SS1.p4.1.m1.1.1.2.cmml" xref="S3.SS1.p4.1.m1.1.1.2">𝑘</ci><times id="S3.SS1.p4.1.m1.1.1.3.cmml" xref="S3.SS1.p4.1.m1.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p4.1.m1.1c">k_{*}</annotation></semantics></math>, <math id="S3.SS1.p4.2.m2.1" class="ltx_Math" alttext="v_{*}" display="inline"><semantics id="S3.SS1.p4.2.m2.1a"><msub id="S3.SS1.p4.2.m2.1.1" xref="S3.SS1.p4.2.m2.1.1.cmml"><mi id="S3.SS1.p4.2.m2.1.1.2" xref="S3.SS1.p4.2.m2.1.1.2.cmml">v</mi><mo id="S3.SS1.p4.2.m2.1.1.3" xref="S3.SS1.p4.2.m2.1.1.3.cmml">∗</mo></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p4.2.m2.1b"><apply id="S3.SS1.p4.2.m2.1.1.cmml" xref="S3.SS1.p4.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS1.p4.2.m2.1.1.1.cmml" xref="S3.SS1.p4.2.m2.1.1">subscript</csymbol><ci id="S3.SS1.p4.2.m2.1.1.2.cmml" xref="S3.SS1.p4.2.m2.1.1.2">𝑣</ci><times id="S3.SS1.p4.2.m2.1.1.3.cmml" xref="S3.SS1.p4.2.m2.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p4.2.m2.1c">v_{*}</annotation></semantics></math>) to represent the full fact <math id="S3.SS1.p4.3.m3.3" class="ltx_Math" alttext="(s,r,o^{*})" display="inline"><semantics id="S3.SS1.p4.3.m3.3a"><mrow id="S3.SS1.p4.3.m3.3.3.1" xref="S3.SS1.p4.3.m3.3.3.2.cmml"><mo stretchy="false" id="S3.SS1.p4.3.m3.3.3.1.2" xref="S3.SS1.p4.3.m3.3.3.2.cmml">(</mo><mi id="S3.SS1.p4.3.m3.1.1" xref="S3.SS1.p4.3.m3.1.1.cmml">s</mi><mo id="S3.SS1.p4.3.m3.3.3.1.3" xref="S3.SS1.p4.3.m3.3.3.2.cmml">,</mo><mi id="S3.SS1.p4.3.m3.2.2" xref="S3.SS1.p4.3.m3.2.2.cmml">r</mi><mo id="S3.SS1.p4.3.m3.3.3.1.4" xref="S3.SS1.p4.3.m3.3.3.2.cmml">,</mo><msup id="S3.SS1.p4.3.m3.3.3.1.1" xref="S3.SS1.p4.3.m3.3.3.1.1.cmml"><mi id="S3.SS1.p4.3.m3.3.3.1.1.2" xref="S3.SS1.p4.3.m3.3.3.1.1.2.cmml">o</mi><mo id="S3.SS1.p4.3.m3.3.3.1.1.3" xref="S3.SS1.p4.3.m3.3.3.1.1.3.cmml">∗</mo></msup><mo stretchy="false" id="S3.SS1.p4.3.m3.3.3.1.5" xref="S3.SS1.p4.3.m3.3.3.2.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p4.3.m3.3b"><vector id="S3.SS1.p4.3.m3.3.3.2.cmml" xref="S3.SS1.p4.3.m3.3.3.1"><ci id="S3.SS1.p4.3.m3.1.1.cmml" xref="S3.SS1.p4.3.m3.1.1">𝑠</ci><ci id="S3.SS1.p4.3.m3.2.2.cmml" xref="S3.SS1.p4.3.m3.2.2">𝑟</ci><apply id="S3.SS1.p4.3.m3.3.3.1.1.cmml" xref="S3.SS1.p4.3.m3.3.3.1.1"><csymbol cd="ambiguous" id="S3.SS1.p4.3.m3.3.3.1.1.1.cmml" xref="S3.SS1.p4.3.m3.3.3.1.1">superscript</csymbol><ci id="S3.SS1.p4.3.m3.3.3.1.1.2.cmml" xref="S3.SS1.p4.3.m3.3.3.1.1.2">𝑜</ci><times id="S3.SS1.p4.3.m3.3.3.1.1.3.cmml" xref="S3.SS1.p4.3.m3.3.3.1.1.3"></times></apply></vector></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p4.3.m3.3c">(s,r,o^{*})</annotation></semantics></math>, we apply Eqn. <a href="#S3.E2" title="In 3.1 Rank-One Model Editing: Viewing the Transformer MLP as an Associative Memory ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, updating the MLP weights <math id="S3.SS1.p4.4.m4.1" class="ltx_Math" alttext="\smash{W^{(l)}_{proj}}" display="inline"><semantics id="S3.SS1.p4.4.m4.1a"><msubsup id="S3.SS1.p4.4.m4.1.2" xref="S3.SS1.p4.4.m4.1.2.cmml"><mi id="S3.SS1.p4.4.m4.1.2.2.2" xref="S3.SS1.p4.4.m4.1.2.2.2.cmml">W</mi><mrow id="S3.SS1.p4.4.m4.1.2.3" xref="S3.SS1.p4.4.m4.1.2.3.cmml"><mi id="S3.SS1.p4.4.m4.1.2.3.2" xref="S3.SS1.p4.4.m4.1.2.3.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p4.4.m4.1.2.3.1" xref="S3.SS1.p4.4.m4.1.2.3.1.cmml">​</mo><mi id="S3.SS1.p4.4.m4.1.2.3.3" xref="S3.SS1.p4.4.m4.1.2.3.3.cmml">r</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p4.4.m4.1.2.3.1a" xref="S3.SS1.p4.4.m4.1.2.3.1.cmml">​</mo><mi id="S3.SS1.p4.4.m4.1.2.3.4" xref="S3.SS1.p4.4.m4.1.2.3.4.cmml">o</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p4.4.m4.1.2.3.1b" xref="S3.SS1.p4.4.m4.1.2.3.1.cmml">​</mo><mi id="S3.SS1.p4.4.m4.1.2.3.5" xref="S3.SS1.p4.4.m4.1.2.3.5.cmml">j</mi></mrow><mrow id="S3.SS1.p4.4.m4.1.1.1.3" xref="S3.SS1.p4.4.m4.1.2.cmml"><mo stretchy="false" id="S3.SS1.p4.4.m4.1.1.1.3.1" xref="S3.SS1.p4.4.m4.1.2.cmml">(</mo><mi id="S3.SS1.p4.4.m4.1.1.1.1" xref="S3.SS1.p4.4.m4.1.1.1.1.cmml">l</mi><mo stretchy="false" id="S3.SS1.p4.4.m4.1.1.1.3.2" xref="S3.SS1.p4.4.m4.1.2.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS1.p4.4.m4.1b"><apply id="S3.SS1.p4.4.m4.1.2.cmml" xref="S3.SS1.p4.4.m4.1.2"><csymbol cd="ambiguous" id="S3.SS1.p4.4.m4.1.2.1.cmml" xref="S3.SS1.p4.4.m4.1.2">subscript</csymbol><apply id="S3.SS1.p4.4.m4.1.2.2.cmml" xref="S3.SS1.p4.4.m4.1.2"><csymbol cd="ambiguous" id="S3.SS1.p4.4.m4.1.2.2.1.cmml" xref="S3.SS1.p4.4.m4.1.2">superscript</csymbol><ci id="S3.SS1.p4.4.m4.1.2.2.2.cmml" xref="S3.SS1.p4.4.m4.1.2.2.2">𝑊</ci><ci id="S3.SS1.p4.4.m4.1.1.1.1.cmml" xref="S3.SS1.p4.4.m4.1.1.1.1">𝑙</ci></apply><apply id="S3.SS1.p4.4.m4.1.2.3.cmml" xref="S3.SS1.p4.4.m4.1.2.3"><times id="S3.SS1.p4.4.m4.1.2.3.1.cmml" xref="S3.SS1.p4.4.m4.1.2.3.1"></times><ci id="S3.SS1.p4.4.m4.1.2.3.2.cmml" xref="S3.SS1.p4.4.m4.1.2.3.2">𝑝</ci><ci id="S3.SS1.p4.4.m4.1.2.3.3.cmml" xref="S3.SS1.p4.4.m4.1.2.3.3">𝑟</ci><ci id="S3.SS1.p4.4.m4.1.2.3.4.cmml" xref="S3.SS1.p4.4.m4.1.2.3.4">𝑜</ci><ci id="S3.SS1.p4.4.m4.1.2.3.5.cmml" xref="S3.SS1.p4.4.m4.1.2.3.5">𝑗</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p4.4.m4.1c">\smash{W^{(l)}_{proj}}</annotation></semantics></math> with a rank-one update that inserts the new key–value association directly.
For full implementation details, see Appendix <a href="#A5.SS5" title="E.5 [GPT-2 XL, GPT-J] Rank-One Model Editing (ROME) ‣ Appendix E Method Implementation Details ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">E.5</span></a>.</p>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Evaluating ROME: Zero-Shot Relation Extraction (zsRE)</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.1" class="ltx_p">We wish to test our localized factual association hypothesis: can storing a single new vector association using ROME insert a substantial, generalized factual association into the model?</p>
</div>
<div id="S3.SS2.p2" class="ltx_para">
<p id="S3.SS2.p2.3" class="ltx_p">A natural question is how ROME compares to other model-editing methods, which use direct optimization or hypernetworks to incorporate a single new training example into a network. For baselines, we examine Fine-Tuning <span id="S3.SS2.p2.3.1" class="ltx_text ltx_font_bold">(FT)</span>, which applies Adam with early stopping at one layer to minimize <math id="S3.SS2.p2.1.m1.1" class="ltx_Math" alttext="-\log\mathbb{P}\left[o^{*}\mid x\right]" display="inline"><semantics id="S3.SS2.p2.1.m1.1a"><mrow id="S3.SS2.p2.1.m1.1.1" xref="S3.SS2.p2.1.m1.1.1.cmml"><mo rspace="0.167em" id="S3.SS2.p2.1.m1.1.1a" xref="S3.SS2.p2.1.m1.1.1.cmml">−</mo><mrow id="S3.SS2.p2.1.m1.1.1.1" xref="S3.SS2.p2.1.m1.1.1.1.cmml"><mrow id="S3.SS2.p2.1.m1.1.1.1.3" xref="S3.SS2.p2.1.m1.1.1.1.3.cmml"><mi id="S3.SS2.p2.1.m1.1.1.1.3.1" xref="S3.SS2.p2.1.m1.1.1.1.3.1.cmml">log</mi><mo lspace="0.167em" id="S3.SS2.p2.1.m1.1.1.1.3a" xref="S3.SS2.p2.1.m1.1.1.1.3.cmml">⁡</mo><mi id="S3.SS2.p2.1.m1.1.1.1.3.2" xref="S3.SS2.p2.1.m1.1.1.1.3.2.cmml">ℙ</mi></mrow><mo lspace="0em" rspace="0em" id="S3.SS2.p2.1.m1.1.1.1.2" xref="S3.SS2.p2.1.m1.1.1.1.2.cmml">​</mo><mrow id="S3.SS2.p2.1.m1.1.1.1.1.1" xref="S3.SS2.p2.1.m1.1.1.1.1.2.cmml"><mo id="S3.SS2.p2.1.m1.1.1.1.1.1.2" xref="S3.SS2.p2.1.m1.1.1.1.1.2.1.cmml">[</mo><mrow id="S3.SS2.p2.1.m1.1.1.1.1.1.1" xref="S3.SS2.p2.1.m1.1.1.1.1.1.1.cmml"><msup id="S3.SS2.p2.1.m1.1.1.1.1.1.1.2" xref="S3.SS2.p2.1.m1.1.1.1.1.1.1.2.cmml"><mi id="S3.SS2.p2.1.m1.1.1.1.1.1.1.2.2" xref="S3.SS2.p2.1.m1.1.1.1.1.1.1.2.2.cmml">o</mi><mo id="S3.SS2.p2.1.m1.1.1.1.1.1.1.2.3" xref="S3.SS2.p2.1.m1.1.1.1.1.1.1.2.3.cmml">∗</mo></msup><mo id="S3.SS2.p2.1.m1.1.1.1.1.1.1.1" xref="S3.SS2.p2.1.m1.1.1.1.1.1.1.1.cmml">∣</mo><mi id="S3.SS2.p2.1.m1.1.1.1.1.1.1.3" xref="S3.SS2.p2.1.m1.1.1.1.1.1.1.3.cmml">x</mi></mrow><mo id="S3.SS2.p2.1.m1.1.1.1.1.1.3" xref="S3.SS2.p2.1.m1.1.1.1.1.2.1.cmml">]</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.1.m1.1b"><apply id="S3.SS2.p2.1.m1.1.1.cmml" xref="S3.SS2.p2.1.m1.1.1"><minus id="S3.SS2.p2.1.m1.1.1.2.cmml" xref="S3.SS2.p2.1.m1.1.1"></minus><apply id="S3.SS2.p2.1.m1.1.1.1.cmml" xref="S3.SS2.p2.1.m1.1.1.1"><times id="S3.SS2.p2.1.m1.1.1.1.2.cmml" xref="S3.SS2.p2.1.m1.1.1.1.2"></times><apply id="S3.SS2.p2.1.m1.1.1.1.3.cmml" xref="S3.SS2.p2.1.m1.1.1.1.3"><log id="S3.SS2.p2.1.m1.1.1.1.3.1.cmml" xref="S3.SS2.p2.1.m1.1.1.1.3.1"></log><ci id="S3.SS2.p2.1.m1.1.1.1.3.2.cmml" xref="S3.SS2.p2.1.m1.1.1.1.3.2">ℙ</ci></apply><apply id="S3.SS2.p2.1.m1.1.1.1.1.2.cmml" xref="S3.SS2.p2.1.m1.1.1.1.1.1"><csymbol cd="latexml" id="S3.SS2.p2.1.m1.1.1.1.1.2.1.cmml" xref="S3.SS2.p2.1.m1.1.1.1.1.1.2">delimited-[]</csymbol><apply id="S3.SS2.p2.1.m1.1.1.1.1.1.1.cmml" xref="S3.SS2.p2.1.m1.1.1.1.1.1.1"><csymbol cd="latexml" id="S3.SS2.p2.1.m1.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p2.1.m1.1.1.1.1.1.1.1">conditional</csymbol><apply id="S3.SS2.p2.1.m1.1.1.1.1.1.1.2.cmml" xref="S3.SS2.p2.1.m1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p2.1.m1.1.1.1.1.1.1.2.1.cmml" xref="S3.SS2.p2.1.m1.1.1.1.1.1.1.2">superscript</csymbol><ci id="S3.SS2.p2.1.m1.1.1.1.1.1.1.2.2.cmml" xref="S3.SS2.p2.1.m1.1.1.1.1.1.1.2.2">𝑜</ci><times id="S3.SS2.p2.1.m1.1.1.1.1.1.1.2.3.cmml" xref="S3.SS2.p2.1.m1.1.1.1.1.1.1.2.3"></times></apply><ci id="S3.SS2.p2.1.m1.1.1.1.1.1.1.3.cmml" xref="S3.SS2.p2.1.m1.1.1.1.1.1.1.3">𝑥</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.1.m1.1c">-\log\mathbb{P}\left[o^{*}\mid x\right]</annotation></semantics></math>.
Constrained Fine-Tuning <span id="S3.SS2.p2.3.2" class="ltx_text ltx_font_bold">(FT+L)</span> <cite class="ltx_cite ltx_citemacro_citep">(Zhu et al., <a href="#bib.bib43" title="" class="ltx_ref">2020</a>)</cite> additionally imposes a parameter-space <math id="S3.SS2.p2.2.m2.1" class="ltx_Math" alttext="L_{\infty}" display="inline"><semantics id="S3.SS2.p2.2.m2.1a"><msub id="S3.SS2.p2.2.m2.1.1" xref="S3.SS2.p2.2.m2.1.1.cmml"><mi id="S3.SS2.p2.2.m2.1.1.2" xref="S3.SS2.p2.2.m2.1.1.2.cmml">L</mi><mi mathvariant="normal" id="S3.SS2.p2.2.m2.1.1.3" xref="S3.SS2.p2.2.m2.1.1.3.cmml">∞</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.2.m2.1b"><apply id="S3.SS2.p2.2.m2.1.1.cmml" xref="S3.SS2.p2.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.2.m2.1.1.1.cmml" xref="S3.SS2.p2.2.m2.1.1">subscript</csymbol><ci id="S3.SS2.p2.2.m2.1.1.2.cmml" xref="S3.SS2.p2.2.m2.1.1.2">𝐿</ci><infinity id="S3.SS2.p2.2.m2.1.1.3.cmml" xref="S3.SS2.p2.2.m2.1.1.3"></infinity></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.2.m2.1c">L_{\infty}</annotation></semantics></math> norm constraint on weight changes.
We also test two hypernetworks: Knowledge Editor <span id="S3.SS2.p2.3.3" class="ltx_text ltx_font_bold">(KE)</span> <cite class="ltx_cite ltx_citemacro_citep">(De Cao et al., <a href="#bib.bib10" title="" class="ltx_ref">2021</a>)</cite> and <span id="S3.SS2.p2.3.4" class="ltx_text ltx_font_bold">MEND</span> <cite class="ltx_cite ltx_citemacro_citep">(Mitchell et al., <a href="#bib.bib27" title="" class="ltx_ref">2021</a>)</cite>, both of which learn auxiliary models to predict weight changes to <math id="S3.SS2.p2.3.m3.1" class="ltx_Math" alttext="G" display="inline"><semantics id="S3.SS2.p2.3.m3.1a"><mi id="S3.SS2.p2.3.m3.1.1" xref="S3.SS2.p2.3.m3.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.3.m3.1b"><ci id="S3.SS2.p2.3.m3.1.1.cmml" xref="S3.SS2.p2.3.m3.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.3.m3.1c">G</annotation></semantics></math>. Further details are described in Appendix <a href="#A5" title="Appendix E Method Implementation Details ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">E</span></a>.</p>
</div>
<figure id="S3.T1" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S3.T1.30.1.1" class="ltx_text" style="font-size:90%;">Table 1</span>: </span><span id="S3.T1.31.2" class="ltx_text" style="font-size:90%;">zsRE Editing Results on GPT-2 XL.</span></figcaption>
<div id="S3.T1.27" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:120.0pt;height:162pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(0.0pt,0.0pt) scale(1,1) ;">
<p id="S3.T1.27.27" class="ltx_p"><span id="S3.T1.27.27.27" class="ltx_text" style="font-size:50%;">
<span id="S3.T1.27.27.27.27" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<span class="ltx_tbody">
<span id="S3.T1.3.3.3.3.3" class="ltx_tr">
<span id="S3.T1.3.3.3.3.3.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row ltx_border_tt" style="padding-left:1.0pt;padding-right:1.0pt;"><span id="S3.T1.3.3.3.3.3.4.1" class="ltx_text ltx_font_bold">Editor</span></span>
<span id="S3.T1.1.1.1.1.1.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_tt" style="padding-left:1.0pt;padding-right:1.0pt;"><span id="S3.T1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Efficacy</span> <math id="S3.T1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S3.T1.1.1.1.1.1.1.m1.1a"><mo stretchy="false" id="S3.T1.1.1.1.1.1.1.m1.1.1" xref="S3.T1.1.1.1.1.1.1.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S3.T1.1.1.1.1.1.1.m1.1b"><ci id="S3.T1.1.1.1.1.1.1.m1.1.1.cmml" xref="S3.T1.1.1.1.1.1.1.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.1.1.1.1.1.1.m1.1c">\uparrow</annotation></semantics></math></span>
<span id="S3.T1.2.2.2.2.2.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_tt" style="padding-left:1.0pt;padding-right:1.0pt;"><span id="S3.T1.2.2.2.2.2.2.1" class="ltx_text ltx_font_bold">Paraphrase</span> <math id="S3.T1.2.2.2.2.2.2.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S3.T1.2.2.2.2.2.2.m1.1a"><mo stretchy="false" id="S3.T1.2.2.2.2.2.2.m1.1.1" xref="S3.T1.2.2.2.2.2.2.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S3.T1.2.2.2.2.2.2.m1.1b"><ci id="S3.T1.2.2.2.2.2.2.m1.1.1.cmml" xref="S3.T1.2.2.2.2.2.2.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.2.2.2.2.2.2.m1.1c">\uparrow</annotation></semantics></math></span>
<span id="S3.T1.3.3.3.3.3.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_tt" style="padding-left:1.0pt;padding-right:1.0pt;"><span id="S3.T1.3.3.3.3.3.3.1" class="ltx_text ltx_font_bold">Specificity</span> <math id="S3.T1.3.3.3.3.3.3.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S3.T1.3.3.3.3.3.3.m1.1a"><mo stretchy="false" id="S3.T1.3.3.3.3.3.3.m1.1.1" xref="S3.T1.3.3.3.3.3.3.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S3.T1.3.3.3.3.3.3.m1.1b"><ci id="S3.T1.3.3.3.3.3.3.m1.1.1.cmml" xref="S3.T1.3.3.3.3.3.3.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.3.3.3.3.3.3.m1.1c">\uparrow</annotation></semantics></math></span></span>
<span id="S3.T1.6.6.6.6.6" class="ltx_tr">
<span id="S3.T1.6.6.6.6.6.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">GPT-2 XL</span>
<span id="S3.T1.4.4.4.4.4.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">22.2 (<math id="S3.T1.4.4.4.4.4.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S3.T1.4.4.4.4.4.1.m1.1a"><mo id="S3.T1.4.4.4.4.4.1.m1.1.1" xref="S3.T1.4.4.4.4.4.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S3.T1.4.4.4.4.4.1.m1.1b"><csymbol cd="latexml" id="S3.T1.4.4.4.4.4.1.m1.1.1.cmml" xref="S3.T1.4.4.4.4.4.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.4.4.4.4.4.1.m1.1c">\pm</annotation></semantics></math>0.5)</span>
<span id="S3.T1.5.5.5.5.5.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">21.3 (<math id="S3.T1.5.5.5.5.5.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S3.T1.5.5.5.5.5.2.m1.1a"><mo id="S3.T1.5.5.5.5.5.2.m1.1.1" xref="S3.T1.5.5.5.5.5.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S3.T1.5.5.5.5.5.2.m1.1b"><csymbol cd="latexml" id="S3.T1.5.5.5.5.5.2.m1.1.1.cmml" xref="S3.T1.5.5.5.5.5.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.5.5.5.5.5.2.m1.1c">\pm</annotation></semantics></math>0.5)</span>
<span id="S3.T1.6.6.6.6.6.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">24.2 (<math id="S3.T1.6.6.6.6.6.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S3.T1.6.6.6.6.6.3.m1.1a"><mo id="S3.T1.6.6.6.6.6.3.m1.1.1" xref="S3.T1.6.6.6.6.6.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S3.T1.6.6.6.6.6.3.m1.1b"><csymbol cd="latexml" id="S3.T1.6.6.6.6.6.3.m1.1.1.cmml" xref="S3.T1.6.6.6.6.6.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.6.6.6.6.6.3.m1.1c">\pm</annotation></semantics></math>0.5)</span></span>
<span id="S3.T1.9.9.9.9.9" class="ltx_tr">
<span id="S3.T1.9.9.9.9.9.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">FT</span>
<span id="S3.T1.7.7.7.7.7.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">99.6 (<math id="S3.T1.7.7.7.7.7.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S3.T1.7.7.7.7.7.1.m1.1a"><mo id="S3.T1.7.7.7.7.7.1.m1.1.1" xref="S3.T1.7.7.7.7.7.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S3.T1.7.7.7.7.7.1.m1.1b"><csymbol cd="latexml" id="S3.T1.7.7.7.7.7.1.m1.1.1.cmml" xref="S3.T1.7.7.7.7.7.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.7.7.7.7.7.1.m1.1c">\pm</annotation></semantics></math>0.1)</span>
<span id="S3.T1.8.8.8.8.8.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">82.1 (<math id="S3.T1.8.8.8.8.8.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S3.T1.8.8.8.8.8.2.m1.1a"><mo id="S3.T1.8.8.8.8.8.2.m1.1.1" xref="S3.T1.8.8.8.8.8.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S3.T1.8.8.8.8.8.2.m1.1b"><csymbol cd="latexml" id="S3.T1.8.8.8.8.8.2.m1.1.1.cmml" xref="S3.T1.8.8.8.8.8.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.8.8.8.8.8.2.m1.1c">\pm</annotation></semantics></math>0.6)</span>
<span id="S3.T1.9.9.9.9.9.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">23.2 (<math id="S3.T1.9.9.9.9.9.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S3.T1.9.9.9.9.9.3.m1.1a"><mo id="S3.T1.9.9.9.9.9.3.m1.1.1" xref="S3.T1.9.9.9.9.9.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S3.T1.9.9.9.9.9.3.m1.1b"><csymbol cd="latexml" id="S3.T1.9.9.9.9.9.3.m1.1.1.cmml" xref="S3.T1.9.9.9.9.9.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.9.9.9.9.9.3.m1.1c">\pm</annotation></semantics></math>0.5)</span></span>
<span id="S3.T1.12.12.12.12.12" class="ltx_tr">
<span id="S3.T1.12.12.12.12.12.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row" style="padding-left:1.0pt;padding-right:1.0pt;">FT+L</span>
<span id="S3.T1.10.10.10.10.10.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;">92.3 (<math id="S3.T1.10.10.10.10.10.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S3.T1.10.10.10.10.10.1.m1.1a"><mo id="S3.T1.10.10.10.10.10.1.m1.1.1" xref="S3.T1.10.10.10.10.10.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S3.T1.10.10.10.10.10.1.m1.1b"><csymbol cd="latexml" id="S3.T1.10.10.10.10.10.1.m1.1.1.cmml" xref="S3.T1.10.10.10.10.10.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.10.10.10.10.10.1.m1.1c">\pm</annotation></semantics></math>0.4)</span>
<span id="S3.T1.11.11.11.11.11.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;"><span id="S3.T1.11.11.11.11.11.2.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T1.11.11.11.11.11.2.1.1" class="ltx_text ltx_font_bold">47.2 (<math id="S3.T1.11.11.11.11.11.2.1.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S3.T1.11.11.11.11.11.2.1.1.m1.1a"><mo mathcolor="#BF0000" id="S3.T1.11.11.11.11.11.2.1.1.m1.1.1" xref="S3.T1.11.11.11.11.11.2.1.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S3.T1.11.11.11.11.11.2.1.1.m1.1b"><csymbol cd="latexml" id="S3.T1.11.11.11.11.11.2.1.1.m1.1.1.cmml" xref="S3.T1.11.11.11.11.11.2.1.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.11.11.11.11.11.2.1.1.m1.1c">\pm</annotation></semantics></math>0.7)</span></span></span>
<span id="S3.T1.12.12.12.12.12.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;">23.4 (<math id="S3.T1.12.12.12.12.12.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S3.T1.12.12.12.12.12.3.m1.1a"><mo id="S3.T1.12.12.12.12.12.3.m1.1.1" xref="S3.T1.12.12.12.12.12.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S3.T1.12.12.12.12.12.3.m1.1b"><csymbol cd="latexml" id="S3.T1.12.12.12.12.12.3.m1.1.1.cmml" xref="S3.T1.12.12.12.12.12.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.12.12.12.12.12.3.m1.1c">\pm</annotation></semantics></math>0.5)</span></span>
<span id="S3.T1.15.15.15.15.15" class="ltx_tr">
<span id="S3.T1.15.15.15.15.15.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row" style="padding-left:1.0pt;padding-right:1.0pt;">KE</span>
<span id="S3.T1.13.13.13.13.13.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;">65.5 (<math id="S3.T1.13.13.13.13.13.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S3.T1.13.13.13.13.13.1.m1.1a"><mo id="S3.T1.13.13.13.13.13.1.m1.1.1" xref="S3.T1.13.13.13.13.13.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S3.T1.13.13.13.13.13.1.m1.1b"><csymbol cd="latexml" id="S3.T1.13.13.13.13.13.1.m1.1.1.cmml" xref="S3.T1.13.13.13.13.13.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.13.13.13.13.13.1.m1.1c">\pm</annotation></semantics></math>0.6)</span>
<span id="S3.T1.14.14.14.14.14.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;">61.4 (<math id="S3.T1.14.14.14.14.14.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S3.T1.14.14.14.14.14.2.m1.1a"><mo id="S3.T1.14.14.14.14.14.2.m1.1.1" xref="S3.T1.14.14.14.14.14.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S3.T1.14.14.14.14.14.2.m1.1b"><csymbol cd="latexml" id="S3.T1.14.14.14.14.14.2.m1.1.1.cmml" xref="S3.T1.14.14.14.14.14.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.14.14.14.14.14.2.m1.1c">\pm</annotation></semantics></math>0.6)</span>
<span id="S3.T1.15.15.15.15.15.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;">24.9 (<math id="S3.T1.15.15.15.15.15.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S3.T1.15.15.15.15.15.3.m1.1a"><mo id="S3.T1.15.15.15.15.15.3.m1.1.1" xref="S3.T1.15.15.15.15.15.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S3.T1.15.15.15.15.15.3.m1.1b"><csymbol cd="latexml" id="S3.T1.15.15.15.15.15.3.m1.1.1.cmml" xref="S3.T1.15.15.15.15.15.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.15.15.15.15.15.3.m1.1c">\pm</annotation></semantics></math>0.5)</span></span>
<span id="S3.T1.18.18.18.18.18" class="ltx_tr">
<span id="S3.T1.18.18.18.18.18.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row" style="padding-left:1.0pt;padding-right:1.0pt;">KE-zsRE</span>
<span id="S3.T1.16.16.16.16.16.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;">92.4 (<math id="S3.T1.16.16.16.16.16.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S3.T1.16.16.16.16.16.1.m1.1a"><mo id="S3.T1.16.16.16.16.16.1.m1.1.1" xref="S3.T1.16.16.16.16.16.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S3.T1.16.16.16.16.16.1.m1.1b"><csymbol cd="latexml" id="S3.T1.16.16.16.16.16.1.m1.1.1.cmml" xref="S3.T1.16.16.16.16.16.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.16.16.16.16.16.1.m1.1c">\pm</annotation></semantics></math>0.3)</span>
<span id="S3.T1.17.17.17.17.17.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;">90.0 (<math id="S3.T1.17.17.17.17.17.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S3.T1.17.17.17.17.17.2.m1.1a"><mo id="S3.T1.17.17.17.17.17.2.m1.1.1" xref="S3.T1.17.17.17.17.17.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S3.T1.17.17.17.17.17.2.m1.1b"><csymbol cd="latexml" id="S3.T1.17.17.17.17.17.2.m1.1.1.cmml" xref="S3.T1.17.17.17.17.17.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.17.17.17.17.17.2.m1.1c">\pm</annotation></semantics></math>0.3)</span>
<span id="S3.T1.18.18.18.18.18.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;">23.8 (<math id="S3.T1.18.18.18.18.18.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S3.T1.18.18.18.18.18.3.m1.1a"><mo id="S3.T1.18.18.18.18.18.3.m1.1.1" xref="S3.T1.18.18.18.18.18.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S3.T1.18.18.18.18.18.3.m1.1b"><csymbol cd="latexml" id="S3.T1.18.18.18.18.18.3.m1.1.1.cmml" xref="S3.T1.18.18.18.18.18.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.18.18.18.18.18.3.m1.1c">\pm</annotation></semantics></math>0.5)</span></span>
<span id="S3.T1.21.21.21.21.21" class="ltx_tr">
<span id="S3.T1.21.21.21.21.21.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row" style="padding-left:1.0pt;padding-right:1.0pt;">MEND</span>
<span id="S3.T1.19.19.19.19.19.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;">75.9 (<math id="S3.T1.19.19.19.19.19.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S3.T1.19.19.19.19.19.1.m1.1a"><mo id="S3.T1.19.19.19.19.19.1.m1.1.1" xref="S3.T1.19.19.19.19.19.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S3.T1.19.19.19.19.19.1.m1.1b"><csymbol cd="latexml" id="S3.T1.19.19.19.19.19.1.m1.1.1.cmml" xref="S3.T1.19.19.19.19.19.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.19.19.19.19.19.1.m1.1c">\pm</annotation></semantics></math>0.5)</span>
<span id="S3.T1.20.20.20.20.20.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;">65.3 (<math id="S3.T1.20.20.20.20.20.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S3.T1.20.20.20.20.20.2.m1.1a"><mo id="S3.T1.20.20.20.20.20.2.m1.1.1" xref="S3.T1.20.20.20.20.20.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S3.T1.20.20.20.20.20.2.m1.1b"><csymbol cd="latexml" id="S3.T1.20.20.20.20.20.2.m1.1.1.cmml" xref="S3.T1.20.20.20.20.20.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.20.20.20.20.20.2.m1.1c">\pm</annotation></semantics></math>0.6)</span>
<span id="S3.T1.21.21.21.21.21.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;">24.1 (<math id="S3.T1.21.21.21.21.21.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S3.T1.21.21.21.21.21.3.m1.1a"><mo id="S3.T1.21.21.21.21.21.3.m1.1.1" xref="S3.T1.21.21.21.21.21.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S3.T1.21.21.21.21.21.3.m1.1b"><csymbol cd="latexml" id="S3.T1.21.21.21.21.21.3.m1.1.1.cmml" xref="S3.T1.21.21.21.21.21.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.21.21.21.21.21.3.m1.1c">\pm</annotation></semantics></math>0.5)</span></span>
<span id="S3.T1.24.24.24.24.24" class="ltx_tr">
<span id="S3.T1.24.24.24.24.24.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row" style="padding-left:1.0pt;padding-right:1.0pt;">MEND-zsRE</span>
<span id="S3.T1.22.22.22.22.22.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;">99.4 (<math id="S3.T1.22.22.22.22.22.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S3.T1.22.22.22.22.22.1.m1.1a"><mo id="S3.T1.22.22.22.22.22.1.m1.1.1" xref="S3.T1.22.22.22.22.22.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S3.T1.22.22.22.22.22.1.m1.1b"><csymbol cd="latexml" id="S3.T1.22.22.22.22.22.1.m1.1.1.cmml" xref="S3.T1.22.22.22.22.22.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.22.22.22.22.22.1.m1.1c">\pm</annotation></semantics></math>0.1)</span>
<span id="S3.T1.23.23.23.23.23.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;"><span id="S3.T1.23.23.23.23.23.2.1" class="ltx_text" style="color:#009900;"><span id="S3.T1.23.23.23.23.23.2.1.1" class="ltx_text ltx_font_bold">99.3 (<math id="S3.T1.23.23.23.23.23.2.1.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S3.T1.23.23.23.23.23.2.1.1.m1.1a"><mo mathcolor="#009900" id="S3.T1.23.23.23.23.23.2.1.1.m1.1.1" xref="S3.T1.23.23.23.23.23.2.1.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S3.T1.23.23.23.23.23.2.1.1.m1.1b"><csymbol cd="latexml" id="S3.T1.23.23.23.23.23.2.1.1.m1.1.1.cmml" xref="S3.T1.23.23.23.23.23.2.1.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.23.23.23.23.23.2.1.1.m1.1c">\pm</annotation></semantics></math>0.1)</span></span></span>
<span id="S3.T1.24.24.24.24.24.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;">24.1 (<math id="S3.T1.24.24.24.24.24.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S3.T1.24.24.24.24.24.3.m1.1a"><mo id="S3.T1.24.24.24.24.24.3.m1.1.1" xref="S3.T1.24.24.24.24.24.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S3.T1.24.24.24.24.24.3.m1.1b"><csymbol cd="latexml" id="S3.T1.24.24.24.24.24.3.m1.1.1.cmml" xref="S3.T1.24.24.24.24.24.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.24.24.24.24.24.3.m1.1c">\pm</annotation></semantics></math>0.5)</span></span>
<span id="S3.T1.27.27.27.27.27" class="ltx_tr">
<span id="S3.T1.27.27.27.27.27.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row ltx_border_bb" style="padding-left:1.0pt;padding-right:1.0pt;">ROME</span>
<span id="S3.T1.25.25.25.25.25.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_bb" style="padding-left:1.0pt;padding-right:1.0pt;"><span id="S3.T1.25.25.25.25.25.1.1" class="ltx_text" style="color:#009900;"><span id="S3.T1.25.25.25.25.25.1.1.1" class="ltx_text ltx_font_bold">99.8 (<math id="S3.T1.25.25.25.25.25.1.1.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S3.T1.25.25.25.25.25.1.1.1.m1.1a"><mo mathcolor="#009900" id="S3.T1.25.25.25.25.25.1.1.1.m1.1.1" xref="S3.T1.25.25.25.25.25.1.1.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S3.T1.25.25.25.25.25.1.1.1.m1.1b"><csymbol cd="latexml" id="S3.T1.25.25.25.25.25.1.1.1.m1.1.1.cmml" xref="S3.T1.25.25.25.25.25.1.1.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.25.25.25.25.25.1.1.1.m1.1c">\pm</annotation></semantics></math>0.0)</span></span></span>
<span id="S3.T1.26.26.26.26.26.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_bb" style="padding-left:1.0pt;padding-right:1.0pt;">88.1 (<math id="S3.T1.26.26.26.26.26.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S3.T1.26.26.26.26.26.2.m1.1a"><mo id="S3.T1.26.26.26.26.26.2.m1.1.1" xref="S3.T1.26.26.26.26.26.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S3.T1.26.26.26.26.26.2.m1.1b"><csymbol cd="latexml" id="S3.T1.26.26.26.26.26.2.m1.1.1.cmml" xref="S3.T1.26.26.26.26.26.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.26.26.26.26.26.2.m1.1c">\pm</annotation></semantics></math>0.5)</span>
<span id="S3.T1.27.27.27.27.27.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_bb" style="padding-left:1.0pt;padding-right:1.0pt;"><span id="S3.T1.27.27.27.27.27.3.1" class="ltx_text" style="color:#009900;"><span id="S3.T1.27.27.27.27.27.3.1.1" class="ltx_text ltx_font_bold">24.2 (<math id="S3.T1.27.27.27.27.27.3.1.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S3.T1.27.27.27.27.27.3.1.1.m1.1a"><mo mathcolor="#009900" id="S3.T1.27.27.27.27.27.3.1.1.m1.1.1" xref="S3.T1.27.27.27.27.27.3.1.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S3.T1.27.27.27.27.27.3.1.1.m1.1b"><csymbol cd="latexml" id="S3.T1.27.27.27.27.27.3.1.1.m1.1.1.cmml" xref="S3.T1.27.27.27.27.27.3.1.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.27.27.27.27.27.3.1.1.m1.1c">\pm</annotation></semantics></math>0.5)</span></span></span></span>
</span>
</span></span></p>
</span></div>
</figure>
<div id="S3.SS2.p3" class="ltx_para">
<p id="S3.SS2.p3.1" class="ltx_p">We first evaluate ROME on the Zero-Shot Relation Extraction (zsRE) task used in <cite class="ltx_cite ltx_citemacro_citet">Mitchell et al. (<a href="#bib.bib27" title="" class="ltx_ref">2021</a>)</cite> and <cite class="ltx_cite ltx_citemacro_citet">De Cao et al. (<a href="#bib.bib10" title="" class="ltx_ref">2021</a>)</cite>. Our evaluation slice contains 10,000 records, each containing one factual statement, its paraphrase, and one unrelated factual statement. “Efficacy” and “Paraphrase” measure post-edit accuracy <math id="S3.SS2.p3.1.m1.2" class="ltx_Math" alttext="\smash{\mathbb{I}\big{[}o^{*}=\mathrm{argmax}_{o}\mathbb{P}_{G^{\prime}}\left[o\right]\big{]}}" display="inline"><semantics id="S3.SS2.p3.1.m1.2a"><mrow id="S3.SS2.p3.1.m1.2.2" xref="S3.SS2.p3.1.m1.2.2.cmml"><mi id="S3.SS2.p3.1.m1.2.2.3" xref="S3.SS2.p3.1.m1.2.2.3.cmml">𝕀</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p3.1.m1.2.2.2" xref="S3.SS2.p3.1.m1.2.2.2.cmml">​</mo><mrow id="S3.SS2.p3.1.m1.2.2.1.1" xref="S3.SS2.p3.1.m1.2.2.1.2.cmml"><mo maxsize="120%" minsize="120%" id="S3.SS2.p3.1.m1.2.2.1.1.2" xref="S3.SS2.p3.1.m1.2.2.1.2.1.cmml">[</mo><mrow id="S3.SS2.p3.1.m1.2.2.1.1.1" xref="S3.SS2.p3.1.m1.2.2.1.1.1.cmml"><msup id="S3.SS2.p3.1.m1.2.2.1.1.1.2" xref="S3.SS2.p3.1.m1.2.2.1.1.1.2.cmml"><mi id="S3.SS2.p3.1.m1.2.2.1.1.1.2.2" xref="S3.SS2.p3.1.m1.2.2.1.1.1.2.2.cmml">o</mi><mo id="S3.SS2.p3.1.m1.2.2.1.1.1.2.3" xref="S3.SS2.p3.1.m1.2.2.1.1.1.2.3.cmml">∗</mo></msup><mo id="S3.SS2.p3.1.m1.2.2.1.1.1.1" xref="S3.SS2.p3.1.m1.2.2.1.1.1.1.cmml">=</mo><mrow id="S3.SS2.p3.1.m1.2.2.1.1.1.3" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3.cmml"><msub id="S3.SS2.p3.1.m1.2.2.1.1.1.3.2" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3.2.cmml"><mi id="S3.SS2.p3.1.m1.2.2.1.1.1.3.2.2" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3.2.2.cmml">argmax</mi><mi id="S3.SS2.p3.1.m1.2.2.1.1.1.3.2.3" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3.2.3.cmml">o</mi></msub><mo lspace="0em" rspace="0em" id="S3.SS2.p3.1.m1.2.2.1.1.1.3.1" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3.1.cmml">​</mo><msub id="S3.SS2.p3.1.m1.2.2.1.1.1.3.3" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3.3.cmml"><mi id="S3.SS2.p3.1.m1.2.2.1.1.1.3.3.2" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3.3.2.cmml">ℙ</mi><msup id="S3.SS2.p3.1.m1.2.2.1.1.1.3.3.3" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3.3.3.cmml"><mi id="S3.SS2.p3.1.m1.2.2.1.1.1.3.3.3.2" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3.3.3.2.cmml">G</mi><mo id="S3.SS2.p3.1.m1.2.2.1.1.1.3.3.3.3" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3.3.3.3.cmml">′</mo></msup></msub><mo lspace="0em" rspace="0em" id="S3.SS2.p3.1.m1.2.2.1.1.1.3.1a" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3.1.cmml">​</mo><mrow id="S3.SS2.p3.1.m1.2.2.1.1.1.3.4.2" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3.4.1.cmml"><mo id="S3.SS2.p3.1.m1.2.2.1.1.1.3.4.2.1" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3.4.1.1.cmml">[</mo><mi id="S3.SS2.p3.1.m1.1.1" xref="S3.SS2.p3.1.m1.1.1.cmml">o</mi><mo id="S3.SS2.p3.1.m1.2.2.1.1.1.3.4.2.2" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3.4.1.1.cmml">]</mo></mrow></mrow></mrow><mo maxsize="120%" minsize="120%" id="S3.SS2.p3.1.m1.2.2.1.1.3" xref="S3.SS2.p3.1.m1.2.2.1.2.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.1.m1.2b"><apply id="S3.SS2.p3.1.m1.2.2.cmml" xref="S3.SS2.p3.1.m1.2.2"><times id="S3.SS2.p3.1.m1.2.2.2.cmml" xref="S3.SS2.p3.1.m1.2.2.2"></times><ci id="S3.SS2.p3.1.m1.2.2.3.cmml" xref="S3.SS2.p3.1.m1.2.2.3">𝕀</ci><apply id="S3.SS2.p3.1.m1.2.2.1.2.cmml" xref="S3.SS2.p3.1.m1.2.2.1.1"><csymbol cd="latexml" id="S3.SS2.p3.1.m1.2.2.1.2.1.cmml" xref="S3.SS2.p3.1.m1.2.2.1.1.2">delimited-[]</csymbol><apply id="S3.SS2.p3.1.m1.2.2.1.1.1.cmml" xref="S3.SS2.p3.1.m1.2.2.1.1.1"><eq id="S3.SS2.p3.1.m1.2.2.1.1.1.1.cmml" xref="S3.SS2.p3.1.m1.2.2.1.1.1.1"></eq><apply id="S3.SS2.p3.1.m1.2.2.1.1.1.2.cmml" xref="S3.SS2.p3.1.m1.2.2.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p3.1.m1.2.2.1.1.1.2.1.cmml" xref="S3.SS2.p3.1.m1.2.2.1.1.1.2">superscript</csymbol><ci id="S3.SS2.p3.1.m1.2.2.1.1.1.2.2.cmml" xref="S3.SS2.p3.1.m1.2.2.1.1.1.2.2">𝑜</ci><times id="S3.SS2.p3.1.m1.2.2.1.1.1.2.3.cmml" xref="S3.SS2.p3.1.m1.2.2.1.1.1.2.3"></times></apply><apply id="S3.SS2.p3.1.m1.2.2.1.1.1.3.cmml" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3"><times id="S3.SS2.p3.1.m1.2.2.1.1.1.3.1.cmml" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3.1"></times><apply id="S3.SS2.p3.1.m1.2.2.1.1.1.3.2.cmml" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3.2"><csymbol cd="ambiguous" id="S3.SS2.p3.1.m1.2.2.1.1.1.3.2.1.cmml" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3.2">subscript</csymbol><ci id="S3.SS2.p3.1.m1.2.2.1.1.1.3.2.2.cmml" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3.2.2">argmax</ci><ci id="S3.SS2.p3.1.m1.2.2.1.1.1.3.2.3.cmml" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3.2.3">𝑜</ci></apply><apply id="S3.SS2.p3.1.m1.2.2.1.1.1.3.3.cmml" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3.3"><csymbol cd="ambiguous" id="S3.SS2.p3.1.m1.2.2.1.1.1.3.3.1.cmml" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3.3">subscript</csymbol><ci id="S3.SS2.p3.1.m1.2.2.1.1.1.3.3.2.cmml" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3.3.2">ℙ</ci><apply id="S3.SS2.p3.1.m1.2.2.1.1.1.3.3.3.cmml" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3.3.3"><csymbol cd="ambiguous" id="S3.SS2.p3.1.m1.2.2.1.1.1.3.3.3.1.cmml" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3.3.3">superscript</csymbol><ci id="S3.SS2.p3.1.m1.2.2.1.1.1.3.3.3.2.cmml" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3.3.3.2">𝐺</ci><ci id="S3.SS2.p3.1.m1.2.2.1.1.1.3.3.3.3.cmml" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3.3.3.3">′</ci></apply></apply><apply id="S3.SS2.p3.1.m1.2.2.1.1.1.3.4.1.cmml" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3.4.2"><csymbol cd="latexml" id="S3.SS2.p3.1.m1.2.2.1.1.1.3.4.1.1.cmml" xref="S3.SS2.p3.1.m1.2.2.1.1.1.3.4.2.1">delimited-[]</csymbol><ci id="S3.SS2.p3.1.m1.1.1.cmml" xref="S3.SS2.p3.1.m1.1.1">𝑜</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.1.m1.2c">\smash{\mathbb{I}\big{[}o^{*}=\mathrm{argmax}_{o}\mathbb{P}_{G^{\prime}}\left[o\right]\big{]}}</annotation></semantics></math> of the statement and its paraphrase, respectively, while “Specificity” measures the edited model’s accuracy on an unrelated fact.
Table <a href="#S3.T1" title="Table 1 ‣ 3.2 Evaluating ROME: Zero-Shot Relation Extraction (zsRE) ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> shows the results: ROME is competitive with hypernetworks and fine-tuning methods despite its simplicity. We find that it is not hard for ROME to insert an association that can be regurgitated by the model. Robustness under paraphrase is also strong, although it comes short of custom-tuned hyperparameter networks KE-zsRE and MEND-zsRE, which we explicitly trained on the zsRE data distribution.<span id="footnote5" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">5</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">5</sup><span class="ltx_tag ltx_tag_note">5</span>Out-of-the-box, they are trained on a WikiText generation task <cite class="ltx_cite ltx_citemacro_citep">(Mitchell et al., <a href="#bib.bib27" title="" class="ltx_ref">2021</a>; De Cao et al., <a href="#bib.bib10" title="" class="ltx_ref">2021</a>)</cite>.</span></span></span> We find that zsRE’s specificity score is not a sensitive measure of model damage, since these prompts are sampled from a large space of possible facts, whereas bleedover is most likely to occur on related <span id="S3.SS2.p3.1.1" class="ltx_text ltx_font_italic">neighboring</span> subjects. Appendix <a href="#A3" title="Appendix C Details on the zsRE Evaluation Task ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a> has additional experimental details.</p>
</div>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Evaluating ROME: Our <span id="S3.SS3.1.1" class="ltx_text ltx_font_smallcaps">CounterFact</span> Dataset</h3>

<div id="S3.SS3.p1" class="ltx_para">
<p id="S3.SS3.p1.1" class="ltx_p">While standard model-editing metrics on zsRE are a reasonable starting point for evaluating ROME, they do not provide detailed insights that would allow us to distinguish superficial wording changes from deeper modifications that correspond to a meaningful change about a fact.</p>
</div>
<div id="S3.SS3.p2" class="ltx_para">
<p id="S3.SS3.p2.8" class="ltx_p">In particular, we wish to measure the efficacy of <span id="S3.SS3.p2.8.1" class="ltx_text ltx_font_italic">significant</span> changes. <cite class="ltx_cite ltx_citemacro_cite">Hase et al. (<a href="#bib.bib19" title="" class="ltx_ref">2021</a>)</cite> observed that standard model-editing benchmarks underestimate difficulty by often testing only proposals that the model previously scored as likely. We compile a set of more difficult <span id="S3.SS3.p2.8.2" class="ltx_text ltx_font_italic">false</span> facts <math id="S3.SS3.p2.1.m1.3" class="ltx_Math" alttext="(s,r,o^{*})" display="inline"><semantics id="S3.SS3.p2.1.m1.3a"><mrow id="S3.SS3.p2.1.m1.3.3.1" xref="S3.SS3.p2.1.m1.3.3.2.cmml"><mo stretchy="false" id="S3.SS3.p2.1.m1.3.3.1.2" xref="S3.SS3.p2.1.m1.3.3.2.cmml">(</mo><mi id="S3.SS3.p2.1.m1.1.1" xref="S3.SS3.p2.1.m1.1.1.cmml">s</mi><mo id="S3.SS3.p2.1.m1.3.3.1.3" xref="S3.SS3.p2.1.m1.3.3.2.cmml">,</mo><mi id="S3.SS3.p2.1.m1.2.2" xref="S3.SS3.p2.1.m1.2.2.cmml">r</mi><mo id="S3.SS3.p2.1.m1.3.3.1.4" xref="S3.SS3.p2.1.m1.3.3.2.cmml">,</mo><msup id="S3.SS3.p2.1.m1.3.3.1.1" xref="S3.SS3.p2.1.m1.3.3.1.1.cmml"><mi id="S3.SS3.p2.1.m1.3.3.1.1.2" xref="S3.SS3.p2.1.m1.3.3.1.1.2.cmml">o</mi><mo id="S3.SS3.p2.1.m1.3.3.1.1.3" xref="S3.SS3.p2.1.m1.3.3.1.1.3.cmml">∗</mo></msup><mo stretchy="false" id="S3.SS3.p2.1.m1.3.3.1.5" xref="S3.SS3.p2.1.m1.3.3.2.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.1.m1.3b"><vector id="S3.SS3.p2.1.m1.3.3.2.cmml" xref="S3.SS3.p2.1.m1.3.3.1"><ci id="S3.SS3.p2.1.m1.1.1.cmml" xref="S3.SS3.p2.1.m1.1.1">𝑠</ci><ci id="S3.SS3.p2.1.m1.2.2.cmml" xref="S3.SS3.p2.1.m1.2.2">𝑟</ci><apply id="S3.SS3.p2.1.m1.3.3.1.1.cmml" xref="S3.SS3.p2.1.m1.3.3.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.1.m1.3.3.1.1.1.cmml" xref="S3.SS3.p2.1.m1.3.3.1.1">superscript</csymbol><ci id="S3.SS3.p2.1.m1.3.3.1.1.2.cmml" xref="S3.SS3.p2.1.m1.3.3.1.1.2">𝑜</ci><times id="S3.SS3.p2.1.m1.3.3.1.1.3.cmml" xref="S3.SS3.p2.1.m1.3.3.1.1.3"></times></apply></vector></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.1.m1.3c">(s,r,o^{*})</annotation></semantics></math>: these counterfactuals start with low scores compared to the correct facts <math id="S3.SS3.p2.2.m2.3" class="ltx_Math" alttext="(s,r,o^{c})" display="inline"><semantics id="S3.SS3.p2.2.m2.3a"><mrow id="S3.SS3.p2.2.m2.3.3.1" xref="S3.SS3.p2.2.m2.3.3.2.cmml"><mo stretchy="false" id="S3.SS3.p2.2.m2.3.3.1.2" xref="S3.SS3.p2.2.m2.3.3.2.cmml">(</mo><mi id="S3.SS3.p2.2.m2.1.1" xref="S3.SS3.p2.2.m2.1.1.cmml">s</mi><mo id="S3.SS3.p2.2.m2.3.3.1.3" xref="S3.SS3.p2.2.m2.3.3.2.cmml">,</mo><mi id="S3.SS3.p2.2.m2.2.2" xref="S3.SS3.p2.2.m2.2.2.cmml">r</mi><mo id="S3.SS3.p2.2.m2.3.3.1.4" xref="S3.SS3.p2.2.m2.3.3.2.cmml">,</mo><msup id="S3.SS3.p2.2.m2.3.3.1.1" xref="S3.SS3.p2.2.m2.3.3.1.1.cmml"><mi id="S3.SS3.p2.2.m2.3.3.1.1.2" xref="S3.SS3.p2.2.m2.3.3.1.1.2.cmml">o</mi><mi id="S3.SS3.p2.2.m2.3.3.1.1.3" xref="S3.SS3.p2.2.m2.3.3.1.1.3.cmml">c</mi></msup><mo stretchy="false" id="S3.SS3.p2.2.m2.3.3.1.5" xref="S3.SS3.p2.2.m2.3.3.2.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.2.m2.3b"><vector id="S3.SS3.p2.2.m2.3.3.2.cmml" xref="S3.SS3.p2.2.m2.3.3.1"><ci id="S3.SS3.p2.2.m2.1.1.cmml" xref="S3.SS3.p2.2.m2.1.1">𝑠</ci><ci id="S3.SS3.p2.2.m2.2.2.cmml" xref="S3.SS3.p2.2.m2.2.2">𝑟</ci><apply id="S3.SS3.p2.2.m2.3.3.1.1.cmml" xref="S3.SS3.p2.2.m2.3.3.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.2.m2.3.3.1.1.1.cmml" xref="S3.SS3.p2.2.m2.3.3.1.1">superscript</csymbol><ci id="S3.SS3.p2.2.m2.3.3.1.1.2.cmml" xref="S3.SS3.p2.2.m2.3.3.1.1.2">𝑜</ci><ci id="S3.SS3.p2.2.m2.3.3.1.1.3.cmml" xref="S3.SS3.p2.2.m2.3.3.1.1.3">𝑐</ci></apply></vector></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.2.m2.3c">(s,r,o^{c})</annotation></semantics></math>.
Our Efficacy Score <span id="S3.SS3.p2.8.3" class="ltx_text ltx_font_bold">(ES)</span> is the portion of cases for which we have <math id="S3.SS3.p2.3.m3.2" class="ltx_Math" alttext="\mathbb{P}[o^{*}]&gt;\mathbb{P}[o^{c}]" display="inline"><semantics id="S3.SS3.p2.3.m3.2a"><mrow id="S3.SS3.p2.3.m3.2.2" xref="S3.SS3.p2.3.m3.2.2.cmml"><mrow id="S3.SS3.p2.3.m3.1.1.1" xref="S3.SS3.p2.3.m3.1.1.1.cmml"><mi id="S3.SS3.p2.3.m3.1.1.1.3" xref="S3.SS3.p2.3.m3.1.1.1.3.cmml">ℙ</mi><mo lspace="0em" rspace="0em" id="S3.SS3.p2.3.m3.1.1.1.2" xref="S3.SS3.p2.3.m3.1.1.1.2.cmml">​</mo><mrow id="S3.SS3.p2.3.m3.1.1.1.1.1" xref="S3.SS3.p2.3.m3.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS3.p2.3.m3.1.1.1.1.1.2" xref="S3.SS3.p2.3.m3.1.1.1.1.2.1.cmml">[</mo><msup id="S3.SS3.p2.3.m3.1.1.1.1.1.1" xref="S3.SS3.p2.3.m3.1.1.1.1.1.1.cmml"><mi id="S3.SS3.p2.3.m3.1.1.1.1.1.1.2" xref="S3.SS3.p2.3.m3.1.1.1.1.1.1.2.cmml">o</mi><mo id="S3.SS3.p2.3.m3.1.1.1.1.1.1.3" xref="S3.SS3.p2.3.m3.1.1.1.1.1.1.3.cmml">∗</mo></msup><mo stretchy="false" id="S3.SS3.p2.3.m3.1.1.1.1.1.3" xref="S3.SS3.p2.3.m3.1.1.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="S3.SS3.p2.3.m3.2.2.3" xref="S3.SS3.p2.3.m3.2.2.3.cmml">&gt;</mo><mrow id="S3.SS3.p2.3.m3.2.2.2" xref="S3.SS3.p2.3.m3.2.2.2.cmml"><mi id="S3.SS3.p2.3.m3.2.2.2.3" xref="S3.SS3.p2.3.m3.2.2.2.3.cmml">ℙ</mi><mo lspace="0em" rspace="0em" id="S3.SS3.p2.3.m3.2.2.2.2" xref="S3.SS3.p2.3.m3.2.2.2.2.cmml">​</mo><mrow id="S3.SS3.p2.3.m3.2.2.2.1.1" xref="S3.SS3.p2.3.m3.2.2.2.1.2.cmml"><mo stretchy="false" id="S3.SS3.p2.3.m3.2.2.2.1.1.2" xref="S3.SS3.p2.3.m3.2.2.2.1.2.1.cmml">[</mo><msup id="S3.SS3.p2.3.m3.2.2.2.1.1.1" xref="S3.SS3.p2.3.m3.2.2.2.1.1.1.cmml"><mi id="S3.SS3.p2.3.m3.2.2.2.1.1.1.2" xref="S3.SS3.p2.3.m3.2.2.2.1.1.1.2.cmml">o</mi><mi id="S3.SS3.p2.3.m3.2.2.2.1.1.1.3" xref="S3.SS3.p2.3.m3.2.2.2.1.1.1.3.cmml">c</mi></msup><mo stretchy="false" id="S3.SS3.p2.3.m3.2.2.2.1.1.3" xref="S3.SS3.p2.3.m3.2.2.2.1.2.1.cmml">]</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.3.m3.2b"><apply id="S3.SS3.p2.3.m3.2.2.cmml" xref="S3.SS3.p2.3.m3.2.2"><gt id="S3.SS3.p2.3.m3.2.2.3.cmml" xref="S3.SS3.p2.3.m3.2.2.3"></gt><apply id="S3.SS3.p2.3.m3.1.1.1.cmml" xref="S3.SS3.p2.3.m3.1.1.1"><times id="S3.SS3.p2.3.m3.1.1.1.2.cmml" xref="S3.SS3.p2.3.m3.1.1.1.2"></times><ci id="S3.SS3.p2.3.m3.1.1.1.3.cmml" xref="S3.SS3.p2.3.m3.1.1.1.3">ℙ</ci><apply id="S3.SS3.p2.3.m3.1.1.1.1.2.cmml" xref="S3.SS3.p2.3.m3.1.1.1.1.1"><csymbol cd="latexml" id="S3.SS3.p2.3.m3.1.1.1.1.2.1.cmml" xref="S3.SS3.p2.3.m3.1.1.1.1.1.2">delimited-[]</csymbol><apply id="S3.SS3.p2.3.m3.1.1.1.1.1.1.cmml" xref="S3.SS3.p2.3.m3.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.3.m3.1.1.1.1.1.1.1.cmml" xref="S3.SS3.p2.3.m3.1.1.1.1.1.1">superscript</csymbol><ci id="S3.SS3.p2.3.m3.1.1.1.1.1.1.2.cmml" xref="S3.SS3.p2.3.m3.1.1.1.1.1.1.2">𝑜</ci><times id="S3.SS3.p2.3.m3.1.1.1.1.1.1.3.cmml" xref="S3.SS3.p2.3.m3.1.1.1.1.1.1.3"></times></apply></apply></apply><apply id="S3.SS3.p2.3.m3.2.2.2.cmml" xref="S3.SS3.p2.3.m3.2.2.2"><times id="S3.SS3.p2.3.m3.2.2.2.2.cmml" xref="S3.SS3.p2.3.m3.2.2.2.2"></times><ci id="S3.SS3.p2.3.m3.2.2.2.3.cmml" xref="S3.SS3.p2.3.m3.2.2.2.3">ℙ</ci><apply id="S3.SS3.p2.3.m3.2.2.2.1.2.cmml" xref="S3.SS3.p2.3.m3.2.2.2.1.1"><csymbol cd="latexml" id="S3.SS3.p2.3.m3.2.2.2.1.2.1.cmml" xref="S3.SS3.p2.3.m3.2.2.2.1.1.2">delimited-[]</csymbol><apply id="S3.SS3.p2.3.m3.2.2.2.1.1.1.cmml" xref="S3.SS3.p2.3.m3.2.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.3.m3.2.2.2.1.1.1.1.cmml" xref="S3.SS3.p2.3.m3.2.2.2.1.1.1">superscript</csymbol><ci id="S3.SS3.p2.3.m3.2.2.2.1.1.1.2.cmml" xref="S3.SS3.p2.3.m3.2.2.2.1.1.1.2">𝑜</ci><ci id="S3.SS3.p2.3.m3.2.2.2.1.1.1.3.cmml" xref="S3.SS3.p2.3.m3.2.2.2.1.1.1.3">𝑐</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.3.m3.2c">\mathbb{P}[o^{*}]&gt;\mathbb{P}[o^{c}]</annotation></semantics></math> post-edit, and Efficacy Magnitude <span id="S3.SS3.p2.8.4" class="ltx_text ltx_font_bold">(EM)</span> is the mean difference <math id="S3.SS3.p2.4.m4.2" class="ltx_Math" alttext="\mathbb{P}[o^{*}]-\mathbb{P}[o^{c}]" display="inline"><semantics id="S3.SS3.p2.4.m4.2a"><mrow id="S3.SS3.p2.4.m4.2.2" xref="S3.SS3.p2.4.m4.2.2.cmml"><mrow id="S3.SS3.p2.4.m4.1.1.1" xref="S3.SS3.p2.4.m4.1.1.1.cmml"><mi id="S3.SS3.p2.4.m4.1.1.1.3" xref="S3.SS3.p2.4.m4.1.1.1.3.cmml">ℙ</mi><mo lspace="0em" rspace="0em" id="S3.SS3.p2.4.m4.1.1.1.2" xref="S3.SS3.p2.4.m4.1.1.1.2.cmml">​</mo><mrow id="S3.SS3.p2.4.m4.1.1.1.1.1" xref="S3.SS3.p2.4.m4.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS3.p2.4.m4.1.1.1.1.1.2" xref="S3.SS3.p2.4.m4.1.1.1.1.2.1.cmml">[</mo><msup id="S3.SS3.p2.4.m4.1.1.1.1.1.1" xref="S3.SS3.p2.4.m4.1.1.1.1.1.1.cmml"><mi id="S3.SS3.p2.4.m4.1.1.1.1.1.1.2" xref="S3.SS3.p2.4.m4.1.1.1.1.1.1.2.cmml">o</mi><mo id="S3.SS3.p2.4.m4.1.1.1.1.1.1.3" xref="S3.SS3.p2.4.m4.1.1.1.1.1.1.3.cmml">∗</mo></msup><mo stretchy="false" id="S3.SS3.p2.4.m4.1.1.1.1.1.3" xref="S3.SS3.p2.4.m4.1.1.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="S3.SS3.p2.4.m4.2.2.3" xref="S3.SS3.p2.4.m4.2.2.3.cmml">−</mo><mrow id="S3.SS3.p2.4.m4.2.2.2" xref="S3.SS3.p2.4.m4.2.2.2.cmml"><mi id="S3.SS3.p2.4.m4.2.2.2.3" xref="S3.SS3.p2.4.m4.2.2.2.3.cmml">ℙ</mi><mo lspace="0em" rspace="0em" id="S3.SS3.p2.4.m4.2.2.2.2" xref="S3.SS3.p2.4.m4.2.2.2.2.cmml">​</mo><mrow id="S3.SS3.p2.4.m4.2.2.2.1.1" xref="S3.SS3.p2.4.m4.2.2.2.1.2.cmml"><mo stretchy="false" id="S3.SS3.p2.4.m4.2.2.2.1.1.2" xref="S3.SS3.p2.4.m4.2.2.2.1.2.1.cmml">[</mo><msup id="S3.SS3.p2.4.m4.2.2.2.1.1.1" xref="S3.SS3.p2.4.m4.2.2.2.1.1.1.cmml"><mi id="S3.SS3.p2.4.m4.2.2.2.1.1.1.2" xref="S3.SS3.p2.4.m4.2.2.2.1.1.1.2.cmml">o</mi><mi id="S3.SS3.p2.4.m4.2.2.2.1.1.1.3" xref="S3.SS3.p2.4.m4.2.2.2.1.1.1.3.cmml">c</mi></msup><mo stretchy="false" id="S3.SS3.p2.4.m4.2.2.2.1.1.3" xref="S3.SS3.p2.4.m4.2.2.2.1.2.1.cmml">]</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.4.m4.2b"><apply id="S3.SS3.p2.4.m4.2.2.cmml" xref="S3.SS3.p2.4.m4.2.2"><minus id="S3.SS3.p2.4.m4.2.2.3.cmml" xref="S3.SS3.p2.4.m4.2.2.3"></minus><apply id="S3.SS3.p2.4.m4.1.1.1.cmml" xref="S3.SS3.p2.4.m4.1.1.1"><times id="S3.SS3.p2.4.m4.1.1.1.2.cmml" xref="S3.SS3.p2.4.m4.1.1.1.2"></times><ci id="S3.SS3.p2.4.m4.1.1.1.3.cmml" xref="S3.SS3.p2.4.m4.1.1.1.3">ℙ</ci><apply id="S3.SS3.p2.4.m4.1.1.1.1.2.cmml" xref="S3.SS3.p2.4.m4.1.1.1.1.1"><csymbol cd="latexml" id="S3.SS3.p2.4.m4.1.1.1.1.2.1.cmml" xref="S3.SS3.p2.4.m4.1.1.1.1.1.2">delimited-[]</csymbol><apply id="S3.SS3.p2.4.m4.1.1.1.1.1.1.cmml" xref="S3.SS3.p2.4.m4.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.cmml" xref="S3.SS3.p2.4.m4.1.1.1.1.1.1">superscript</csymbol><ci id="S3.SS3.p2.4.m4.1.1.1.1.1.1.2.cmml" xref="S3.SS3.p2.4.m4.1.1.1.1.1.1.2">𝑜</ci><times id="S3.SS3.p2.4.m4.1.1.1.1.1.1.3.cmml" xref="S3.SS3.p2.4.m4.1.1.1.1.1.1.3"></times></apply></apply></apply><apply id="S3.SS3.p2.4.m4.2.2.2.cmml" xref="S3.SS3.p2.4.m4.2.2.2"><times id="S3.SS3.p2.4.m4.2.2.2.2.cmml" xref="S3.SS3.p2.4.m4.2.2.2.2"></times><ci id="S3.SS3.p2.4.m4.2.2.2.3.cmml" xref="S3.SS3.p2.4.m4.2.2.2.3">ℙ</ci><apply id="S3.SS3.p2.4.m4.2.2.2.1.2.cmml" xref="S3.SS3.p2.4.m4.2.2.2.1.1"><csymbol cd="latexml" id="S3.SS3.p2.4.m4.2.2.2.1.2.1.cmml" xref="S3.SS3.p2.4.m4.2.2.2.1.1.2">delimited-[]</csymbol><apply id="S3.SS3.p2.4.m4.2.2.2.1.1.1.cmml" xref="S3.SS3.p2.4.m4.2.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.4.m4.2.2.2.1.1.1.1.cmml" xref="S3.SS3.p2.4.m4.2.2.2.1.1.1">superscript</csymbol><ci id="S3.SS3.p2.4.m4.2.2.2.1.1.1.2.cmml" xref="S3.SS3.p2.4.m4.2.2.2.1.1.1.2">𝑜</ci><ci id="S3.SS3.p2.4.m4.2.2.2.1.1.1.3.cmml" xref="S3.SS3.p2.4.m4.2.2.2.1.1.1.3">𝑐</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.4.m4.2c">\mathbb{P}[o^{*}]-\mathbb{P}[o^{c}]</annotation></semantics></math>. Then, to measure <span id="S3.SS3.p2.8.5" class="ltx_text ltx_font_bold">generalization</span>, with each counterfactual we gather a set of rephrased prompts equivalent to <math id="S3.SS3.p2.5.m5.2" class="ltx_Math" alttext="(s,r)" display="inline"><semantics id="S3.SS3.p2.5.m5.2a"><mrow id="S3.SS3.p2.5.m5.2.3.2" xref="S3.SS3.p2.5.m5.2.3.1.cmml"><mo stretchy="false" id="S3.SS3.p2.5.m5.2.3.2.1" xref="S3.SS3.p2.5.m5.2.3.1.cmml">(</mo><mi id="S3.SS3.p2.5.m5.1.1" xref="S3.SS3.p2.5.m5.1.1.cmml">s</mi><mo id="S3.SS3.p2.5.m5.2.3.2.2" xref="S3.SS3.p2.5.m5.2.3.1.cmml">,</mo><mi id="S3.SS3.p2.5.m5.2.2" xref="S3.SS3.p2.5.m5.2.2.cmml">r</mi><mo stretchy="false" id="S3.SS3.p2.5.m5.2.3.2.3" xref="S3.SS3.p2.5.m5.2.3.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.5.m5.2b"><interval closure="open" id="S3.SS3.p2.5.m5.2.3.1.cmml" xref="S3.SS3.p2.5.m5.2.3.2"><ci id="S3.SS3.p2.5.m5.1.1.cmml" xref="S3.SS3.p2.5.m5.1.1">𝑠</ci><ci id="S3.SS3.p2.5.m5.2.2.cmml" xref="S3.SS3.p2.5.m5.2.2">𝑟</ci></interval></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.5.m5.2c">(s,r)</annotation></semantics></math> and report Paraphrase Scores <span id="S3.SS3.p2.8.6" class="ltx_text ltx_font_bold">(PS)</span> and <span id="S3.SS3.p2.8.7" class="ltx_text ltx_font_bold">(PM)</span>, computed similarly to ES and EM. To measure <span id="S3.SS3.p2.8.8" class="ltx_text ltx_font_bold">specificity</span>, we collect a set of nearby subjects <math id="S3.SS3.p2.6.m6.1" class="ltx_Math" alttext="s_{n}" display="inline"><semantics id="S3.SS3.p2.6.m6.1a"><msub id="S3.SS3.p2.6.m6.1.1" xref="S3.SS3.p2.6.m6.1.1.cmml"><mi id="S3.SS3.p2.6.m6.1.1.2" xref="S3.SS3.p2.6.m6.1.1.2.cmml">s</mi><mi id="S3.SS3.p2.6.m6.1.1.3" xref="S3.SS3.p2.6.m6.1.1.3.cmml">n</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.6.m6.1b"><apply id="S3.SS3.p2.6.m6.1.1.cmml" xref="S3.SS3.p2.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.6.m6.1.1.1.cmml" xref="S3.SS3.p2.6.m6.1.1">subscript</csymbol><ci id="S3.SS3.p2.6.m6.1.1.2.cmml" xref="S3.SS3.p2.6.m6.1.1.2">𝑠</ci><ci id="S3.SS3.p2.6.m6.1.1.3.cmml" xref="S3.SS3.p2.6.m6.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.6.m6.1c">s_{n}</annotation></semantics></math> for which <math id="S3.SS3.p2.7.m7.3" class="ltx_Math" alttext="(s_{n},r,o^{c})" display="inline"><semantics id="S3.SS3.p2.7.m7.3a"><mrow id="S3.SS3.p2.7.m7.3.3.2" xref="S3.SS3.p2.7.m7.3.3.3.cmml"><mo stretchy="false" id="S3.SS3.p2.7.m7.3.3.2.3" xref="S3.SS3.p2.7.m7.3.3.3.cmml">(</mo><msub id="S3.SS3.p2.7.m7.2.2.1.1" xref="S3.SS3.p2.7.m7.2.2.1.1.cmml"><mi id="S3.SS3.p2.7.m7.2.2.1.1.2" xref="S3.SS3.p2.7.m7.2.2.1.1.2.cmml">s</mi><mi id="S3.SS3.p2.7.m7.2.2.1.1.3" xref="S3.SS3.p2.7.m7.2.2.1.1.3.cmml">n</mi></msub><mo id="S3.SS3.p2.7.m7.3.3.2.4" xref="S3.SS3.p2.7.m7.3.3.3.cmml">,</mo><mi id="S3.SS3.p2.7.m7.1.1" xref="S3.SS3.p2.7.m7.1.1.cmml">r</mi><mo id="S3.SS3.p2.7.m7.3.3.2.5" xref="S3.SS3.p2.7.m7.3.3.3.cmml">,</mo><msup id="S3.SS3.p2.7.m7.3.3.2.2" xref="S3.SS3.p2.7.m7.3.3.2.2.cmml"><mi id="S3.SS3.p2.7.m7.3.3.2.2.2" xref="S3.SS3.p2.7.m7.3.3.2.2.2.cmml">o</mi><mi id="S3.SS3.p2.7.m7.3.3.2.2.3" xref="S3.SS3.p2.7.m7.3.3.2.2.3.cmml">c</mi></msup><mo stretchy="false" id="S3.SS3.p2.7.m7.3.3.2.6" xref="S3.SS3.p2.7.m7.3.3.3.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.7.m7.3b"><vector id="S3.SS3.p2.7.m7.3.3.3.cmml" xref="S3.SS3.p2.7.m7.3.3.2"><apply id="S3.SS3.p2.7.m7.2.2.1.1.cmml" xref="S3.SS3.p2.7.m7.2.2.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.7.m7.2.2.1.1.1.cmml" xref="S3.SS3.p2.7.m7.2.2.1.1">subscript</csymbol><ci id="S3.SS3.p2.7.m7.2.2.1.1.2.cmml" xref="S3.SS3.p2.7.m7.2.2.1.1.2">𝑠</ci><ci id="S3.SS3.p2.7.m7.2.2.1.1.3.cmml" xref="S3.SS3.p2.7.m7.2.2.1.1.3">𝑛</ci></apply><ci id="S3.SS3.p2.7.m7.1.1.cmml" xref="S3.SS3.p2.7.m7.1.1">𝑟</ci><apply id="S3.SS3.p2.7.m7.3.3.2.2.cmml" xref="S3.SS3.p2.7.m7.3.3.2.2"><csymbol cd="ambiguous" id="S3.SS3.p2.7.m7.3.3.2.2.1.cmml" xref="S3.SS3.p2.7.m7.3.3.2.2">superscript</csymbol><ci id="S3.SS3.p2.7.m7.3.3.2.2.2.cmml" xref="S3.SS3.p2.7.m7.3.3.2.2.2">𝑜</ci><ci id="S3.SS3.p2.7.m7.3.3.2.2.3.cmml" xref="S3.SS3.p2.7.m7.3.3.2.2.3">𝑐</ci></apply></vector></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.7.m7.3c">(s_{n},r,o^{c})</annotation></semantics></math> holds true. Because we do not wish to alter these subjects, we test <math id="S3.SS3.p2.8.m8.2" class="ltx_Math" alttext="\mathbb{P}[o^{c}]&gt;\mathbb{P}[o^{*}]" display="inline"><semantics id="S3.SS3.p2.8.m8.2a"><mrow id="S3.SS3.p2.8.m8.2.2" xref="S3.SS3.p2.8.m8.2.2.cmml"><mrow id="S3.SS3.p2.8.m8.1.1.1" xref="S3.SS3.p2.8.m8.1.1.1.cmml"><mi id="S3.SS3.p2.8.m8.1.1.1.3" xref="S3.SS3.p2.8.m8.1.1.1.3.cmml">ℙ</mi><mo lspace="0em" rspace="0em" id="S3.SS3.p2.8.m8.1.1.1.2" xref="S3.SS3.p2.8.m8.1.1.1.2.cmml">​</mo><mrow id="S3.SS3.p2.8.m8.1.1.1.1.1" xref="S3.SS3.p2.8.m8.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS3.p2.8.m8.1.1.1.1.1.2" xref="S3.SS3.p2.8.m8.1.1.1.1.2.1.cmml">[</mo><msup id="S3.SS3.p2.8.m8.1.1.1.1.1.1" xref="S3.SS3.p2.8.m8.1.1.1.1.1.1.cmml"><mi id="S3.SS3.p2.8.m8.1.1.1.1.1.1.2" xref="S3.SS3.p2.8.m8.1.1.1.1.1.1.2.cmml">o</mi><mi id="S3.SS3.p2.8.m8.1.1.1.1.1.1.3" xref="S3.SS3.p2.8.m8.1.1.1.1.1.1.3.cmml">c</mi></msup><mo stretchy="false" id="S3.SS3.p2.8.m8.1.1.1.1.1.3" xref="S3.SS3.p2.8.m8.1.1.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="S3.SS3.p2.8.m8.2.2.3" xref="S3.SS3.p2.8.m8.2.2.3.cmml">&gt;</mo><mrow id="S3.SS3.p2.8.m8.2.2.2" xref="S3.SS3.p2.8.m8.2.2.2.cmml"><mi id="S3.SS3.p2.8.m8.2.2.2.3" xref="S3.SS3.p2.8.m8.2.2.2.3.cmml">ℙ</mi><mo lspace="0em" rspace="0em" id="S3.SS3.p2.8.m8.2.2.2.2" xref="S3.SS3.p2.8.m8.2.2.2.2.cmml">​</mo><mrow id="S3.SS3.p2.8.m8.2.2.2.1.1" xref="S3.SS3.p2.8.m8.2.2.2.1.2.cmml"><mo stretchy="false" id="S3.SS3.p2.8.m8.2.2.2.1.1.2" xref="S3.SS3.p2.8.m8.2.2.2.1.2.1.cmml">[</mo><msup id="S3.SS3.p2.8.m8.2.2.2.1.1.1" xref="S3.SS3.p2.8.m8.2.2.2.1.1.1.cmml"><mi id="S3.SS3.p2.8.m8.2.2.2.1.1.1.2" xref="S3.SS3.p2.8.m8.2.2.2.1.1.1.2.cmml">o</mi><mo id="S3.SS3.p2.8.m8.2.2.2.1.1.1.3" xref="S3.SS3.p2.8.m8.2.2.2.1.1.1.3.cmml">∗</mo></msup><mo stretchy="false" id="S3.SS3.p2.8.m8.2.2.2.1.1.3" xref="S3.SS3.p2.8.m8.2.2.2.1.2.1.cmml">]</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.8.m8.2b"><apply id="S3.SS3.p2.8.m8.2.2.cmml" xref="S3.SS3.p2.8.m8.2.2"><gt id="S3.SS3.p2.8.m8.2.2.3.cmml" xref="S3.SS3.p2.8.m8.2.2.3"></gt><apply id="S3.SS3.p2.8.m8.1.1.1.cmml" xref="S3.SS3.p2.8.m8.1.1.1"><times id="S3.SS3.p2.8.m8.1.1.1.2.cmml" xref="S3.SS3.p2.8.m8.1.1.1.2"></times><ci id="S3.SS3.p2.8.m8.1.1.1.3.cmml" xref="S3.SS3.p2.8.m8.1.1.1.3">ℙ</ci><apply id="S3.SS3.p2.8.m8.1.1.1.1.2.cmml" xref="S3.SS3.p2.8.m8.1.1.1.1.1"><csymbol cd="latexml" id="S3.SS3.p2.8.m8.1.1.1.1.2.1.cmml" xref="S3.SS3.p2.8.m8.1.1.1.1.1.2">delimited-[]</csymbol><apply id="S3.SS3.p2.8.m8.1.1.1.1.1.1.cmml" xref="S3.SS3.p2.8.m8.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.8.m8.1.1.1.1.1.1.1.cmml" xref="S3.SS3.p2.8.m8.1.1.1.1.1.1">superscript</csymbol><ci id="S3.SS3.p2.8.m8.1.1.1.1.1.1.2.cmml" xref="S3.SS3.p2.8.m8.1.1.1.1.1.1.2">𝑜</ci><ci id="S3.SS3.p2.8.m8.1.1.1.1.1.1.3.cmml" xref="S3.SS3.p2.8.m8.1.1.1.1.1.1.3">𝑐</ci></apply></apply></apply><apply id="S3.SS3.p2.8.m8.2.2.2.cmml" xref="S3.SS3.p2.8.m8.2.2.2"><times id="S3.SS3.p2.8.m8.2.2.2.2.cmml" xref="S3.SS3.p2.8.m8.2.2.2.2"></times><ci id="S3.SS3.p2.8.m8.2.2.2.3.cmml" xref="S3.SS3.p2.8.m8.2.2.2.3">ℙ</ci><apply id="S3.SS3.p2.8.m8.2.2.2.1.2.cmml" xref="S3.SS3.p2.8.m8.2.2.2.1.1"><csymbol cd="latexml" id="S3.SS3.p2.8.m8.2.2.2.1.2.1.cmml" xref="S3.SS3.p2.8.m8.2.2.2.1.1.2">delimited-[]</csymbol><apply id="S3.SS3.p2.8.m8.2.2.2.1.1.1.cmml" xref="S3.SS3.p2.8.m8.2.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.8.m8.2.2.2.1.1.1.1.cmml" xref="S3.SS3.p2.8.m8.2.2.2.1.1.1">superscript</csymbol><ci id="S3.SS3.p2.8.m8.2.2.2.1.1.1.2.cmml" xref="S3.SS3.p2.8.m8.2.2.2.1.1.1.2">𝑜</ci><times id="S3.SS3.p2.8.m8.2.2.2.1.1.1.3.cmml" xref="S3.SS3.p2.8.m8.2.2.2.1.1.1.3"></times></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.8.m8.2c">\mathbb{P}[o^{c}]&gt;\mathbb{P}[o^{*}]</annotation></semantics></math>, reporting the success fraction as Neighborhood Score <span id="S3.SS3.p2.8.9" class="ltx_text ltx_font_bold">(NS)</span> and difference as <span id="S3.SS3.p2.8.10" class="ltx_text ltx_font_bold">(NM)</span>. To test the generalization–specificity tradeoff, we report the harmonic mean of ES, PS, NS as Score (<span id="S3.SS3.p2.8.11" class="ltx_text ltx_font_bold">S</span>).</p>
</div>
<figure id="S3.T3" class="ltx_table">

<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="S3.T3.3.1.1" class="ltx_text" style="font-size:90%;">Table 2</span>: </span><span id="S3.T3.4.2" class="ltx_text ltx_font_smallcaps" style="font-size:90%;">CounterFact<span id="S3.T3.4.2.1" class="ltx_text ltx_font_upright"> Composition</span></span></figcaption><div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="S3.T3.5" class="ltx_inline-block ltx_figure_panel ltx_transformed_outer" style="width:244.2pt;height:162pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(0.0pt,0.0pt) scale(1,1) ;">
<p id="S3.T3.5.1" class="ltx_p"><span id="S3.T3.5.1.1" class="ltx_text">


<span id="S3.T3.5.1.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<span class="ltx_thead">
<span id="S3.T3.5.1.1.1.1.1" class="ltx_tr">
<span id="S3.T3.5.1.1.1.1.1.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_th ltx_th_row ltx_border_tt" style="padding-left:1.0pt;padding-right:1.0pt;"></span>
<span id="S3.T3.5.1.1.1.1.1.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_th ltx_th_column ltx_border_tt" style="padding-left:1.0pt;padding-right:1.0pt;"></span>
<span id="S3.T3.5.1.1.1.1.1.3" class="ltx_td ltx_nopad_l ltx_align_center ltx_th ltx_th_column ltx_border_tt ltx_rowspan ltx_rowspan_2" style="padding-left:1.0pt;padding-right:1.0pt;"><span id="S3.T3.5.1.1.1.1.1.3.1" class="ltx_text">
<span id="S3.T3.5.1.1.1.1.1.3.1.1" class="ltx_inline-block">
<span id="S3.T3.5.1.1.1.1.1.3.1.1.1" class="ltx_p"><span id="S3.T3.5.1.1.1.1.1.3.1.1.1.1" class="ltx_text ltx_font_bold">Per</span></span>
<span id="S3.T3.5.1.1.1.1.1.3.1.1.2" class="ltx_p"><span id="S3.T3.5.1.1.1.1.1.3.1.1.2.1" class="ltx_text ltx_font_bold">Relation</span></span>
</span></span></span>
<span id="S3.T3.5.1.1.1.1.1.4" class="ltx_td ltx_nopad_l ltx_align_center ltx_th ltx_th_column ltx_border_tt ltx_rowspan ltx_rowspan_2" style="padding-left:1.0pt;padding-right:1.0pt;"><span id="S3.T3.5.1.1.1.1.1.4.1" class="ltx_text">
<span id="S3.T3.5.1.1.1.1.1.4.1.1" class="ltx_inline-block">
<span id="S3.T3.5.1.1.1.1.1.4.1.1.1" class="ltx_p"><span id="S3.T3.5.1.1.1.1.1.4.1.1.1.1" class="ltx_text ltx_font_bold">Per</span></span>
<span id="S3.T3.5.1.1.1.1.1.4.1.1.2" class="ltx_p"><span id="S3.T3.5.1.1.1.1.1.4.1.1.2.1" class="ltx_text ltx_font_bold">Record</span></span>
</span></span></span></span>
<span id="S3.T3.5.1.1.1.2.2" class="ltx_tr">
<span id="S3.T3.5.1.1.1.2.2.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_column ltx_th_row" style="padding-left:1.0pt;padding-right:1.0pt;"><span id="S3.T3.5.1.1.1.2.2.1.1" class="ltx_text ltx_font_bold">Item</span></span>
<span id="S3.T3.5.1.1.1.2.2.2" class="ltx_td ltx_nopad_l ltx_align_center ltx_th ltx_th_column" style="padding-left:1.0pt;padding-right:1.0pt;"><span id="S3.T3.5.1.1.1.2.2.2.1" class="ltx_text ltx_font_bold">Total</span></span></span>
<span id="S3.T3.5.1.1.1.3.3" class="ltx_tr">
<span id="S3.T3.5.1.1.1.3.3.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">Records</span>
<span id="S3.T3.5.1.1.1.3.3.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_th ltx_th_column ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">21919</span>
<span id="S3.T3.5.1.1.1.3.3.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_th ltx_th_column ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">645</span>
<span id="S3.T3.5.1.1.1.3.3.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_th ltx_th_column ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">1</span></span>
</span>
<span class="ltx_tbody">
<span id="S3.T3.5.1.1.1.4.1" class="ltx_tr">
<span id="S3.T3.5.1.1.1.4.1.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">Subjects</span>
<span id="S3.T3.5.1.1.1.4.1.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">20391</span>
<span id="S3.T3.5.1.1.1.4.1.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">624</span>
<span id="S3.T3.5.1.1.1.4.1.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">1</span></span>
<span id="S3.T3.5.1.1.1.5.2" class="ltx_tr">
<span id="S3.T3.5.1.1.1.5.2.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row" style="padding-left:1.0pt;padding-right:1.0pt;">Objects</span>
<span id="S3.T3.5.1.1.1.5.2.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;">749</span>
<span id="S3.T3.5.1.1.1.5.2.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;">60</span>
<span id="S3.T3.5.1.1.1.5.2.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;">1</span></span>
<span id="S3.T3.5.1.1.1.6.3" class="ltx_tr">
<span id="S3.T3.5.1.1.1.6.3.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row" style="padding-left:1.0pt;padding-right:1.0pt;">Counterfactual Statements</span>
<span id="S3.T3.5.1.1.1.6.3.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;">21595</span>
<span id="S3.T3.5.1.1.1.6.3.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;">635</span>
<span id="S3.T3.5.1.1.1.6.3.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;">1</span></span>
<span id="S3.T3.5.1.1.1.7.4" class="ltx_tr">
<span id="S3.T3.5.1.1.1.7.4.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row" style="padding-left:1.0pt;padding-right:1.0pt;">Paraphrase Prompts</span>
<span id="S3.T3.5.1.1.1.7.4.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;">42876</span>
<span id="S3.T3.5.1.1.1.7.4.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;">1262</span>
<span id="S3.T3.5.1.1.1.7.4.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;">2</span></span>
<span id="S3.T3.5.1.1.1.8.5" class="ltx_tr">
<span id="S3.T3.5.1.1.1.8.5.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row" style="padding-left:1.0pt;padding-right:1.0pt;">Neighborhood Prompts</span>
<span id="S3.T3.5.1.1.1.8.5.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;">82650</span>
<span id="S3.T3.5.1.1.1.8.5.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;">2441</span>
<span id="S3.T3.5.1.1.1.8.5.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;">10</span></span>
<span id="S3.T3.5.1.1.1.9.6" class="ltx_tr">
<span id="S3.T3.5.1.1.1.9.6.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row ltx_border_bb" style="padding-left:1.0pt;padding-right:1.0pt;">Generation Prompts</span>
<span id="S3.T3.5.1.1.1.9.6.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_bb" style="padding-left:1.0pt;padding-right:1.0pt;">62346</span>
<span id="S3.T3.5.1.1.1.9.6.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_bb" style="padding-left:1.0pt;padding-right:1.0pt;">1841</span>
<span id="S3.T3.5.1.1.1.9.6.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_bb" style="padding-left:1.0pt;padding-right:1.0pt;">3</span></span>
</span>
</span></span></p>
</span></div>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S3.T3.7.1.1" class="ltx_text" style="font-size:90%;">Table 3</span>: </span><span id="S3.T3.8.2" class="ltx_text" style="font-size:90%;">Comparison to Existing Benchmarks</span></figcaption><div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="S3.T3.9" class="ltx_inline-block ltx_figure_panel ltx_align_center ltx_transformed_outer" style="width:263.8pt;height:108pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(0.0pt,0.0pt) scale(1,1) ;">
<p id="S3.T3.9.1" class="ltx_p"><span id="S3.T3.9.1.1" class="ltx_text">
<span id="S3.T3.9.1.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<span class="ltx_thead">
<span id="S3.T3.9.1.1.1.1.1" class="ltx_tr">
<span id="S3.T3.9.1.1.1.1.1.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_column ltx_border_tt" style="padding-left:1.0pt;padding-right:1.0pt;"><span id="S3.T3.9.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Criterion</span></span>
<span id="S3.T3.9.1.1.1.1.1.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-left:1.0pt;padding-right:1.0pt;">SQuAD</span>
<span id="S3.T3.9.1.1.1.1.1.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-left:1.0pt;padding-right:1.0pt;">zSRE</span>
<span id="S3.T3.9.1.1.1.1.1.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-left:1.0pt;padding-right:1.0pt;">FEVER</span>
<span id="S3.T3.9.1.1.1.1.1.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-left:1.0pt;padding-right:1.0pt;">WikiText</span>
<span id="S3.T3.9.1.1.1.1.1.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-left:1.0pt;padding-right:1.0pt;"><span id="S3.T3.9.1.1.1.1.1.6.1" class="ltx_text ltx_font_smallcaps">ParaRel</span></span>
<span id="S3.T3.9.1.1.1.1.1.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-left:1.0pt;padding-right:1.0pt;"><span id="S3.T3.9.1.1.1.1.1.7.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">CF</span></span></span>
</span>
<span class="ltx_tbody">
<span id="S3.T3.9.1.1.1.2.1" class="ltx_tr">
<span id="S3.T3.9.1.1.1.2.1.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">Efficacy</span>
<span id="S3.T3.9.1.1.1.2.1.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">✓</span>
<span id="S3.T3.9.1.1.1.2.1.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">✓</span>
<span id="S3.T3.9.1.1.1.2.1.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">✓</span>
<span id="S3.T3.9.1.1.1.2.1.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">✓</span>
<span id="S3.T3.9.1.1.1.2.1.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">✓</span>
<span id="S3.T3.9.1.1.1.2.1.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">✓</span></span>
<span id="S3.T3.9.1.1.1.3.2" class="ltx_tr">
<span id="S3.T3.9.1.1.1.3.2.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left" style="padding-left:1.0pt;padding-right:1.0pt;">Generalization</span>
<span id="S3.T3.9.1.1.1.3.2.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:1.0pt;padding-right:1.0pt;">✓</span>
<span id="S3.T3.9.1.1.1.3.2.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:1.0pt;padding-right:1.0pt;">✓</span>
<span id="S3.T3.9.1.1.1.3.2.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:1.0pt;padding-right:1.0pt;">✓</span>
<span id="S3.T3.9.1.1.1.3.2.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:1.0pt;padding-right:1.0pt;">✗</span>
<span id="S3.T3.9.1.1.1.3.2.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:1.0pt;padding-right:1.0pt;">✓</span>
<span id="S3.T3.9.1.1.1.3.2.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:1.0pt;padding-right:1.0pt;">✓</span></span>
<span id="S3.T3.9.1.1.1.4.3" class="ltx_tr">
<span id="S3.T3.9.1.1.1.4.3.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left" style="padding-left:1.0pt;padding-right:1.0pt;">Bleedover</span>
<span id="S3.T3.9.1.1.1.4.3.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:1.0pt;padding-right:1.0pt;">✗</span>
<span id="S3.T3.9.1.1.1.4.3.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:1.0pt;padding-right:1.0pt;">✗</span>
<span id="S3.T3.9.1.1.1.4.3.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:1.0pt;padding-right:1.0pt;">✗</span>
<span id="S3.T3.9.1.1.1.4.3.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:1.0pt;padding-right:1.0pt;">✗</span>
<span id="S3.T3.9.1.1.1.4.3.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:1.0pt;padding-right:1.0pt;">✗</span>
<span id="S3.T3.9.1.1.1.4.3.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:1.0pt;padding-right:1.0pt;">✓</span></span>
<span id="S3.T3.9.1.1.1.5.4" class="ltx_tr">
<span id="S3.T3.9.1.1.1.5.4.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left" style="padding-left:1.0pt;padding-right:1.0pt;">Consistency</span>
<span id="S3.T3.9.1.1.1.5.4.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:1.0pt;padding-right:1.0pt;">✗</span>
<span id="S3.T3.9.1.1.1.5.4.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:1.0pt;padding-right:1.0pt;">✗</span>
<span id="S3.T3.9.1.1.1.5.4.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:1.0pt;padding-right:1.0pt;">✗</span>
<span id="S3.T3.9.1.1.1.5.4.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:1.0pt;padding-right:1.0pt;">✗</span>
<span id="S3.T3.9.1.1.1.5.4.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:1.0pt;padding-right:1.0pt;">✗</span>
<span id="S3.T3.9.1.1.1.5.4.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:1.0pt;padding-right:1.0pt;">✓</span></span>
<span id="S3.T3.9.1.1.1.6.5" class="ltx_tr">
<span id="S3.T3.9.1.1.1.6.5.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_bb" style="padding-left:1.0pt;padding-right:1.0pt;">Fluency</span>
<span id="S3.T3.9.1.1.1.6.5.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb" style="padding-left:1.0pt;padding-right:1.0pt;">✗</span>
<span id="S3.T3.9.1.1.1.6.5.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb" style="padding-left:1.0pt;padding-right:1.0pt;">✗</span>
<span id="S3.T3.9.1.1.1.6.5.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb" style="padding-left:1.0pt;padding-right:1.0pt;">✗</span>
<span id="S3.T3.9.1.1.1.6.5.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb" style="padding-left:1.0pt;padding-right:1.0pt;">✗</span>
<span id="S3.T3.9.1.1.1.6.5.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb" style="padding-left:1.0pt;padding-right:1.0pt;">✗</span>
<span id="S3.T3.9.1.1.1.6.5.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb" style="padding-left:1.0pt;padding-right:1.0pt;">✓</span></span>
</span>
</span></span></p>
</span></div>
</div>
</div>
</figure>
<div id="S3.SS3.p3" class="ltx_para">
<p id="S3.SS3.p3.7" class="ltx_p">We also wish to measure semantic <span id="S3.SS3.p3.7.1" class="ltx_text ltx_font_bold">consistency</span> of <math id="S3.SS3.p3.1.m1.1" class="ltx_Math" alttext="G^{\prime}" display="inline"><semantics id="S3.SS3.p3.1.m1.1a"><msup id="S3.SS3.p3.1.m1.1.1" xref="S3.SS3.p3.1.m1.1.1.cmml"><mi id="S3.SS3.p3.1.m1.1.1.2" xref="S3.SS3.p3.1.m1.1.1.2.cmml">G</mi><mo id="S3.SS3.p3.1.m1.1.1.3" xref="S3.SS3.p3.1.m1.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.1.m1.1b"><apply id="S3.SS3.p3.1.m1.1.1.cmml" xref="S3.SS3.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p3.1.m1.1.1.1.cmml" xref="S3.SS3.p3.1.m1.1.1">superscript</csymbol><ci id="S3.SS3.p3.1.m1.1.1.2.cmml" xref="S3.SS3.p3.1.m1.1.1.2">𝐺</ci><ci id="S3.SS3.p3.1.m1.1.1.3.cmml" xref="S3.SS3.p3.1.m1.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.1.m1.1c">G^{\prime}</annotation></semantics></math>’s generations. To do so, we generate text starting with <math id="S3.SS3.p3.2.m2.1" class="ltx_Math" alttext="s" display="inline"><semantics id="S3.SS3.p3.2.m2.1a"><mi id="S3.SS3.p3.2.m2.1.1" xref="S3.SS3.p3.2.m2.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.2.m2.1b"><ci id="S3.SS3.p3.2.m2.1.1.cmml" xref="S3.SS3.p3.2.m2.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.2.m2.1c">s</annotation></semantics></math> and report <span id="S3.SS3.p3.7.2" class="ltx_text ltx_font_bold">(RS)</span> as the <math id="S3.SS3.p3.3.m3.1" class="ltx_Math" alttext="\cos" display="inline"><semantics id="S3.SS3.p3.3.m3.1a"><mi id="S3.SS3.p3.3.m3.1.1" xref="S3.SS3.p3.3.m3.1.1.cmml">cos</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.3.m3.1b"><cos id="S3.SS3.p3.3.m3.1.1.cmml" xref="S3.SS3.p3.3.m3.1.1"></cos></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.3.m3.1c">\cos</annotation></semantics></math> similarity between the unigram TF-IDF vectors of generated texts, compared to reference texts about subjects sharing the target property <math id="S3.SS3.p3.4.m4.1" class="ltx_Math" alttext="o^{*}" display="inline"><semantics id="S3.SS3.p3.4.m4.1a"><msup id="S3.SS3.p3.4.m4.1.1" xref="S3.SS3.p3.4.m4.1.1.cmml"><mi id="S3.SS3.p3.4.m4.1.1.2" xref="S3.SS3.p3.4.m4.1.1.2.cmml">o</mi><mo id="S3.SS3.p3.4.m4.1.1.3" xref="S3.SS3.p3.4.m4.1.1.3.cmml">∗</mo></msup><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.4.m4.1b"><apply id="S3.SS3.p3.4.m4.1.1.cmml" xref="S3.SS3.p3.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS3.p3.4.m4.1.1.1.cmml" xref="S3.SS3.p3.4.m4.1.1">superscript</csymbol><ci id="S3.SS3.p3.4.m4.1.1.2.cmml" xref="S3.SS3.p3.4.m4.1.1.2">𝑜</ci><times id="S3.SS3.p3.4.m4.1.1.3.cmml" xref="S3.SS3.p3.4.m4.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.4.m4.1c">o^{*}</annotation></semantics></math>. Finally, we monitor <span id="S3.SS3.p3.7.3" class="ltx_text ltx_font_bold">fluency</span> degradations by measuring the weighted average of bi- and tri-gram entropies <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al., <a href="#bib.bib40" title="" class="ltx_ref">2018</a>)</cite> given by <math id="S3.SS3.p3.5.m5.2" class="ltx_Math" alttext="-\sum_{k}f(k)\log_{2}f(k)" display="inline"><semantics id="S3.SS3.p3.5.m5.2a"><mrow id="S3.SS3.p3.5.m5.2.3" xref="S3.SS3.p3.5.m5.2.3.cmml"><mo id="S3.SS3.p3.5.m5.2.3a" xref="S3.SS3.p3.5.m5.2.3.cmml">−</mo><mrow id="S3.SS3.p3.5.m5.2.3.2" xref="S3.SS3.p3.5.m5.2.3.2.cmml"><msub id="S3.SS3.p3.5.m5.2.3.2.1" xref="S3.SS3.p3.5.m5.2.3.2.1.cmml"><mo id="S3.SS3.p3.5.m5.2.3.2.1.2" xref="S3.SS3.p3.5.m5.2.3.2.1.2.cmml">∑</mo><mi id="S3.SS3.p3.5.m5.2.3.2.1.3" xref="S3.SS3.p3.5.m5.2.3.2.1.3.cmml">k</mi></msub><mrow id="S3.SS3.p3.5.m5.2.3.2.2" xref="S3.SS3.p3.5.m5.2.3.2.2.cmml"><mi id="S3.SS3.p3.5.m5.2.3.2.2.2" xref="S3.SS3.p3.5.m5.2.3.2.2.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="S3.SS3.p3.5.m5.2.3.2.2.1" xref="S3.SS3.p3.5.m5.2.3.2.2.1.cmml">​</mo><mrow id="S3.SS3.p3.5.m5.2.3.2.2.3.2" xref="S3.SS3.p3.5.m5.2.3.2.2.cmml"><mo stretchy="false" id="S3.SS3.p3.5.m5.2.3.2.2.3.2.1" xref="S3.SS3.p3.5.m5.2.3.2.2.cmml">(</mo><mi id="S3.SS3.p3.5.m5.1.1" xref="S3.SS3.p3.5.m5.1.1.cmml">k</mi><mo stretchy="false" id="S3.SS3.p3.5.m5.2.3.2.2.3.2.2" xref="S3.SS3.p3.5.m5.2.3.2.2.cmml">)</mo></mrow><mo lspace="0.167em" rspace="0em" id="S3.SS3.p3.5.m5.2.3.2.2.1a" xref="S3.SS3.p3.5.m5.2.3.2.2.1.cmml">​</mo><mrow id="S3.SS3.p3.5.m5.2.3.2.2.4" xref="S3.SS3.p3.5.m5.2.3.2.2.4.cmml"><msub id="S3.SS3.p3.5.m5.2.3.2.2.4.1" xref="S3.SS3.p3.5.m5.2.3.2.2.4.1.cmml"><mi id="S3.SS3.p3.5.m5.2.3.2.2.4.1.2" xref="S3.SS3.p3.5.m5.2.3.2.2.4.1.2.cmml">log</mi><mn id="S3.SS3.p3.5.m5.2.3.2.2.4.1.3" xref="S3.SS3.p3.5.m5.2.3.2.2.4.1.3.cmml">2</mn></msub><mo lspace="0.167em" id="S3.SS3.p3.5.m5.2.3.2.2.4a" xref="S3.SS3.p3.5.m5.2.3.2.2.4.cmml">⁡</mo><mi id="S3.SS3.p3.5.m5.2.3.2.2.4.2" xref="S3.SS3.p3.5.m5.2.3.2.2.4.2.cmml">f</mi></mrow><mo lspace="0em" rspace="0em" id="S3.SS3.p3.5.m5.2.3.2.2.1b" xref="S3.SS3.p3.5.m5.2.3.2.2.1.cmml">​</mo><mrow id="S3.SS3.p3.5.m5.2.3.2.2.5.2" xref="S3.SS3.p3.5.m5.2.3.2.2.cmml"><mo stretchy="false" id="S3.SS3.p3.5.m5.2.3.2.2.5.2.1" xref="S3.SS3.p3.5.m5.2.3.2.2.cmml">(</mo><mi id="S3.SS3.p3.5.m5.2.2" xref="S3.SS3.p3.5.m5.2.2.cmml">k</mi><mo stretchy="false" id="S3.SS3.p3.5.m5.2.3.2.2.5.2.2" xref="S3.SS3.p3.5.m5.2.3.2.2.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.5.m5.2b"><apply id="S3.SS3.p3.5.m5.2.3.cmml" xref="S3.SS3.p3.5.m5.2.3"><minus id="S3.SS3.p3.5.m5.2.3.1.cmml" xref="S3.SS3.p3.5.m5.2.3"></minus><apply id="S3.SS3.p3.5.m5.2.3.2.cmml" xref="S3.SS3.p3.5.m5.2.3.2"><apply id="S3.SS3.p3.5.m5.2.3.2.1.cmml" xref="S3.SS3.p3.5.m5.2.3.2.1"><csymbol cd="ambiguous" id="S3.SS3.p3.5.m5.2.3.2.1.1.cmml" xref="S3.SS3.p3.5.m5.2.3.2.1">subscript</csymbol><sum id="S3.SS3.p3.5.m5.2.3.2.1.2.cmml" xref="S3.SS3.p3.5.m5.2.3.2.1.2"></sum><ci id="S3.SS3.p3.5.m5.2.3.2.1.3.cmml" xref="S3.SS3.p3.5.m5.2.3.2.1.3">𝑘</ci></apply><apply id="S3.SS3.p3.5.m5.2.3.2.2.cmml" xref="S3.SS3.p3.5.m5.2.3.2.2"><times id="S3.SS3.p3.5.m5.2.3.2.2.1.cmml" xref="S3.SS3.p3.5.m5.2.3.2.2.1"></times><ci id="S3.SS3.p3.5.m5.2.3.2.2.2.cmml" xref="S3.SS3.p3.5.m5.2.3.2.2.2">𝑓</ci><ci id="S3.SS3.p3.5.m5.1.1.cmml" xref="S3.SS3.p3.5.m5.1.1">𝑘</ci><apply id="S3.SS3.p3.5.m5.2.3.2.2.4.cmml" xref="S3.SS3.p3.5.m5.2.3.2.2.4"><apply id="S3.SS3.p3.5.m5.2.3.2.2.4.1.cmml" xref="S3.SS3.p3.5.m5.2.3.2.2.4.1"><csymbol cd="ambiguous" id="S3.SS3.p3.5.m5.2.3.2.2.4.1.1.cmml" xref="S3.SS3.p3.5.m5.2.3.2.2.4.1">subscript</csymbol><log id="S3.SS3.p3.5.m5.2.3.2.2.4.1.2.cmml" xref="S3.SS3.p3.5.m5.2.3.2.2.4.1.2"></log><cn type="integer" id="S3.SS3.p3.5.m5.2.3.2.2.4.1.3.cmml" xref="S3.SS3.p3.5.m5.2.3.2.2.4.1.3">2</cn></apply><ci id="S3.SS3.p3.5.m5.2.3.2.2.4.2.cmml" xref="S3.SS3.p3.5.m5.2.3.2.2.4.2">𝑓</ci></apply><ci id="S3.SS3.p3.5.m5.2.2.cmml" xref="S3.SS3.p3.5.m5.2.2">𝑘</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.5.m5.2c">-\sum_{k}f(k)\log_{2}f(k)</annotation></semantics></math>, where <math id="S3.SS3.p3.6.m6.1" class="ltx_Math" alttext="f(\cdot)" display="inline"><semantics id="S3.SS3.p3.6.m6.1a"><mrow id="S3.SS3.p3.6.m6.1.2" xref="S3.SS3.p3.6.m6.1.2.cmml"><mi id="S3.SS3.p3.6.m6.1.2.2" xref="S3.SS3.p3.6.m6.1.2.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="S3.SS3.p3.6.m6.1.2.1" xref="S3.SS3.p3.6.m6.1.2.1.cmml">​</mo><mrow id="S3.SS3.p3.6.m6.1.2.3.2" xref="S3.SS3.p3.6.m6.1.2.cmml"><mo stretchy="false" id="S3.SS3.p3.6.m6.1.2.3.2.1" xref="S3.SS3.p3.6.m6.1.2.cmml">(</mo><mo lspace="0em" rspace="0em" id="S3.SS3.p3.6.m6.1.1" xref="S3.SS3.p3.6.m6.1.1.cmml">⋅</mo><mo stretchy="false" id="S3.SS3.p3.6.m6.1.2.3.2.2" xref="S3.SS3.p3.6.m6.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.6.m6.1b"><apply id="S3.SS3.p3.6.m6.1.2.cmml" xref="S3.SS3.p3.6.m6.1.2"><times id="S3.SS3.p3.6.m6.1.2.1.cmml" xref="S3.SS3.p3.6.m6.1.2.1"></times><ci id="S3.SS3.p3.6.m6.1.2.2.cmml" xref="S3.SS3.p3.6.m6.1.2.2">𝑓</ci><ci id="S3.SS3.p3.6.m6.1.1.cmml" xref="S3.SS3.p3.6.m6.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.6.m6.1c">f(\cdot)</annotation></semantics></math> is the <math id="S3.SS3.p3.7.m7.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S3.SS3.p3.7.m7.1a"><mi id="S3.SS3.p3.7.m7.1.1" xref="S3.SS3.p3.7.m7.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.7.m7.1b"><ci id="S3.SS3.p3.7.m7.1.1.cmml" xref="S3.SS3.p3.7.m7.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.7.m7.1c">n</annotation></semantics></math>-gram frequency distribution, which we report as <span id="S3.SS3.p3.7.4" class="ltx_text ltx_font_bold">(GE)</span>; this quantity drops if text generations are repetitive.</p>
</div>
<div id="S3.SS3.p4" class="ltx_para">
<p id="S3.SS3.p4.1" class="ltx_p">In order to facilitate the above measurements, we introduce <span id="S3.SS3.p4.1.1" class="ltx_text ltx_font_smallcaps">CounterFact</span>, a challenging evaluation dataset for evaluating counterfactual edits in language models. Containing 21,919 records with a diverse set of subjects, relations, and linguistic variations, <span id="S3.SS3.p4.1.2" class="ltx_text ltx_font_smallcaps">CounterFact</span>’s goal is to differentiate robust storage of new facts from the superficial regurgitation of target words. See Appendix <a href="#A4" title="Appendix D Details on the CounterFact Dataset ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">D</span></a> for additional technical details about its construction, and Table <a href="#S3.T3" title="Table 3 ‣ 3.3 Evaluating ROME: Our CounterFact Dataset ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> for a summary of its composition.</p>
</div>
</section>
<section id="S3.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.4 </span>Confirming the Importance of Decisive States Identified by Causal Tracing</h3>

<figure id="S3.F5" class="ltx_figure"><img src="/html/2202.05262/assets/x5.png" id="S3.F5.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="110" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S3.F5.4.2.1" class="ltx_text" style="font-size:90%;">Figure 5</span>: </span><span id="S3.F5.2.1" class="ltx_text" style="font-size:90%;">ROME edits are benchmarked at each layer-and-token combination in GPT-2-XL. The target token is determined by selecting the token index <math id="S3.F5.2.1.m1.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S3.F5.2.1.m1.1b"><mi id="S3.F5.2.1.m1.1.1" xref="S3.F5.2.1.m1.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S3.F5.2.1.m1.1c"><ci id="S3.F5.2.1.m1.1.1.cmml" xref="S3.F5.2.1.m1.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F5.2.1.m1.1d">i</annotation></semantics></math> where the key representation is collected (Eqn. <a href="#S3.E3" title="In 3.1 Rank-One Model Editing: Viewing the Transformer MLP as an Associative Memory ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>). ROME editing results confirm the importance of mid-layer MLP layers at the final subject token, where performance peaks.</span></figcaption>
</figure>
<div id="S3.SS4.p1" class="ltx_para">
<p id="S3.SS4.p1.1" class="ltx_p">In Section <a href="#S2" title="2 Interventions on Activations for Tracing Information Flow ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, we used Causal Tracing to identify decisive hidden states. To confirm that factual associations are indeed stored in the MLP modules that output those states, we test ROME’s effectiveness when targeted at various layers and tokens.
Figure <a href="#S3.F5" title="Figure 5 ‣ 3.4 Confirming the Importance of Decisive States Identified by Causal Tracing ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> plots four metrics evaluating both generalization (a,b,d) and specificity (c). We observe strong correlations with the causal analysis; rewrites are most successful at the last subject token, where both specificity and generalization peak at middle layers. Targeting earlier <span id="S3.SS4.p1.1.1" class="ltx_text ltx_font_italic">or</span> later tokens results in poor generalization and/or specificity. Furthermore, the layers at which edits generalize best correspond to the middle layers of the early site identified by Causal Tracing, with generalization peaking at the 18th layer. This evidence suggests that we have an accurate understanding not only of <span id="S3.SS4.p1.1.2" class="ltx_text ltx_font_italic">where</span> factual associations are stored, but also <span id="S3.SS4.p1.1.3" class="ltx_text ltx_font_italic">how</span>. Appendix <a href="#A9" title="Appendix I Are Attention Weight Interventions Effective? ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">I</span></a> furthermore demonstrates that editing the late-layer attention modules leads to regurgitation.</p>
</div>
<figure id="S3.T4" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:50%;"><span class="ltx_tag ltx_tag_table"><span id="S3.T4.13.1.1" class="ltx_text" style="font-size:180%;">Table 4</span>: </span><span id="S3.T4.14.2" class="ltx_text ltx_font_bold" style="font-size:180%;">Quantitative Editing Results<span id="S3.T4.14.2.1" class="ltx_text ltx_font_medium">. 95% confidence intervals are in parentheses. <span id="S3.T4.14.2.1.1" class="ltx_text" style="color:#009900;"> <span id="S3.T4.14.2.1.1.1" class="ltx_text ltx_font_bold">Green</span></span> numbers indicate columnwise maxima, whereas <span id="S3.T4.14.2.1.2" class="ltx_text" style="color:#BF0000;"> <span id="S3.T4.14.2.1.2.1" class="ltx_text ltx_font_bold">red</span></span> numbers indicate a clear failure on either generalization or specificity. The presence of <span id="S3.T4.14.2.1.3" class="ltx_text" style="color:#BF0000;"> <span id="S3.T4.14.2.1.3.1" class="ltx_text ltx_font_bold">red</span></span> in a column might explain excellent results in another. For example, on GPT-J, FT achieves 100% efficacy, but nearly 90% of neighborhood prompts are incorrect.</span></span></figcaption>
<div id="S3.T4.9" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:292.6pt;height:288pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(0.0pt,0.0pt) scale(1,1) ;">
<p id="S3.T4.9.9" class="ltx_p"><span id="S3.T4.9.9.9" class="ltx_text" style="font-size:50%;">
<span id="S3.T4.9.9.9.9" class="ltx_tabular ltx_align_middle">
<span class="ltx_tbody">
<span id="S3.T4.9.9.9.9.10.1" class="ltx_tr">
<span id="S3.T4.9.9.9.9.10.1.1" class="ltx_td ltx_align_left ltx_border_tt ltx_rowspan ltx_rowspan_2" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.10.1.1.1" class="ltx_text ltx_font_bold">Editor</span></span>
<span id="S3.T4.9.9.9.9.10.1.2" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.10.1.2.1" class="ltx_text ltx_font_bold">Score</span></span>
<span id="S3.T4.9.9.9.9.10.1.3" class="ltx_td ltx_align_center ltx_border_tt ltx_colspan ltx_colspan_2" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.10.1.3.1" class="ltx_text ltx_font_bold">Efficacy</span></span>
<span id="S3.T4.9.9.9.9.10.1.4" class="ltx_td ltx_align_center ltx_border_tt ltx_colspan ltx_colspan_2" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.10.1.4.1" class="ltx_text ltx_font_bold">Generalization</span></span>
<span id="S3.T4.9.9.9.9.10.1.5" class="ltx_td ltx_align_center ltx_border_tt ltx_colspan ltx_colspan_2" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.10.1.5.1" class="ltx_text ltx_font_bold">Specificity</span></span>
<span id="S3.T4.9.9.9.9.10.1.6" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.10.1.6.1" class="ltx_text ltx_font_bold">Fluency</span></span>
<span id="S3.T4.9.9.9.9.10.1.7" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.10.1.7.1" class="ltx_text ltx_font_bold">Consistency</span></span></span>
<span id="S3.T4.9.9.9.9.9" class="ltx_tr">
<span id="S3.T4.1.1.1.1.1.1" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">S <math id="S3.T4.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S3.T4.1.1.1.1.1.1.m1.1a"><mo stretchy="false" id="S3.T4.1.1.1.1.1.1.m1.1.1" xref="S3.T4.1.1.1.1.1.1.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S3.T4.1.1.1.1.1.1.m1.1b"><ci id="S3.T4.1.1.1.1.1.1.m1.1.1.cmml" xref="S3.T4.1.1.1.1.1.1.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.T4.1.1.1.1.1.1.m1.1c">\uparrow</annotation></semantics></math></span>
<span id="S3.T4.2.2.2.2.2.2" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">ES <math id="S3.T4.2.2.2.2.2.2.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S3.T4.2.2.2.2.2.2.m1.1a"><mo stretchy="false" id="S3.T4.2.2.2.2.2.2.m1.1.1" xref="S3.T4.2.2.2.2.2.2.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S3.T4.2.2.2.2.2.2.m1.1b"><ci id="S3.T4.2.2.2.2.2.2.m1.1.1.cmml" xref="S3.T4.2.2.2.2.2.2.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.T4.2.2.2.2.2.2.m1.1c">\uparrow</annotation></semantics></math></span>
<span id="S3.T4.3.3.3.3.3.3" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">EM <math id="S3.T4.3.3.3.3.3.3.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S3.T4.3.3.3.3.3.3.m1.1a"><mo stretchy="false" id="S3.T4.3.3.3.3.3.3.m1.1.1" xref="S3.T4.3.3.3.3.3.3.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S3.T4.3.3.3.3.3.3.m1.1b"><ci id="S3.T4.3.3.3.3.3.3.m1.1.1.cmml" xref="S3.T4.3.3.3.3.3.3.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.T4.3.3.3.3.3.3.m1.1c">\uparrow</annotation></semantics></math></span>
<span id="S3.T4.4.4.4.4.4.4" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">PS <math id="S3.T4.4.4.4.4.4.4.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S3.T4.4.4.4.4.4.4.m1.1a"><mo stretchy="false" id="S3.T4.4.4.4.4.4.4.m1.1.1" xref="S3.T4.4.4.4.4.4.4.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S3.T4.4.4.4.4.4.4.m1.1b"><ci id="S3.T4.4.4.4.4.4.4.m1.1.1.cmml" xref="S3.T4.4.4.4.4.4.4.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.T4.4.4.4.4.4.4.m1.1c">\uparrow</annotation></semantics></math></span>
<span id="S3.T4.5.5.5.5.5.5" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">PM <math id="S3.T4.5.5.5.5.5.5.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S3.T4.5.5.5.5.5.5.m1.1a"><mo stretchy="false" id="S3.T4.5.5.5.5.5.5.m1.1.1" xref="S3.T4.5.5.5.5.5.5.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S3.T4.5.5.5.5.5.5.m1.1b"><ci id="S3.T4.5.5.5.5.5.5.m1.1.1.cmml" xref="S3.T4.5.5.5.5.5.5.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.T4.5.5.5.5.5.5.m1.1c">\uparrow</annotation></semantics></math></span>
<span id="S3.T4.6.6.6.6.6.6" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">NS <math id="S3.T4.6.6.6.6.6.6.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S3.T4.6.6.6.6.6.6.m1.1a"><mo stretchy="false" id="S3.T4.6.6.6.6.6.6.m1.1.1" xref="S3.T4.6.6.6.6.6.6.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S3.T4.6.6.6.6.6.6.m1.1b"><ci id="S3.T4.6.6.6.6.6.6.m1.1.1.cmml" xref="S3.T4.6.6.6.6.6.6.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.T4.6.6.6.6.6.6.m1.1c">\uparrow</annotation></semantics></math></span>
<span id="S3.T4.7.7.7.7.7.7" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">NM <math id="S3.T4.7.7.7.7.7.7.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S3.T4.7.7.7.7.7.7.m1.1a"><mo stretchy="false" id="S3.T4.7.7.7.7.7.7.m1.1.1" xref="S3.T4.7.7.7.7.7.7.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S3.T4.7.7.7.7.7.7.m1.1b"><ci id="S3.T4.7.7.7.7.7.7.m1.1.1.cmml" xref="S3.T4.7.7.7.7.7.7.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.T4.7.7.7.7.7.7.m1.1c">\uparrow</annotation></semantics></math></span>
<span id="S3.T4.8.8.8.8.8.8" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">GE <math id="S3.T4.8.8.8.8.8.8.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S3.T4.8.8.8.8.8.8.m1.1a"><mo stretchy="false" id="S3.T4.8.8.8.8.8.8.m1.1.1" xref="S3.T4.8.8.8.8.8.8.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S3.T4.8.8.8.8.8.8.m1.1b"><ci id="S3.T4.8.8.8.8.8.8.m1.1.1.cmml" xref="S3.T4.8.8.8.8.8.8.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.T4.8.8.8.8.8.8.m1.1c">\uparrow</annotation></semantics></math></span>
<span id="S3.T4.9.9.9.9.9.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">RS <math id="S3.T4.9.9.9.9.9.9.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S3.T4.9.9.9.9.9.9.m1.1a"><mo stretchy="false" id="S3.T4.9.9.9.9.9.9.m1.1.1" xref="S3.T4.9.9.9.9.9.9.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S3.T4.9.9.9.9.9.9.m1.1b"><ci id="S3.T4.9.9.9.9.9.9.m1.1.1.cmml" xref="S3.T4.9.9.9.9.9.9.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.T4.9.9.9.9.9.9.m1.1c">\uparrow</annotation></semantics></math></span></span>
<span id="S3.T4.9.9.9.9.11.2" class="ltx_tr">
<span id="S3.T4.9.9.9.9.11.2.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">GPT-2 XL</span>
<span id="S3.T4.9.9.9.9.11.2.2" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">30.5</span>
<span id="S3.T4.9.9.9.9.11.2.3" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">22.2 (0.9)</span>
<span id="S3.T4.9.9.9.9.11.2.4" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">-4.8 (0.3)</span>
<span id="S3.T4.9.9.9.9.11.2.5" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">24.7 (0.8)</span>
<span id="S3.T4.9.9.9.9.11.2.6" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">-5.0 (0.3)</span>
<span id="S3.T4.9.9.9.9.11.2.7" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">78.1 (0.6)</span>
<span id="S3.T4.9.9.9.9.11.2.8" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">5.0 (0.2)</span>
<span id="S3.T4.9.9.9.9.11.2.9" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">626.6 (0.3)</span>
<span id="S3.T4.9.9.9.9.11.2.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">31.9 (0.2)</span></span>
<span id="S3.T4.9.9.9.9.12.3" class="ltx_tr">
<span id="S3.T4.9.9.9.9.12.3.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">FT</span>
<span id="S3.T4.9.9.9.9.12.3.2" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">65.1</span>
<span id="S3.T4.9.9.9.9.12.3.3" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">100.0 (0.0)</span>
<span id="S3.T4.9.9.9.9.12.3.4" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">98.8 (0.1)</span>
<span id="S3.T4.9.9.9.9.12.3.5" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">87.9 (0.6)</span>
<span id="S3.T4.9.9.9.9.12.3.6" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">46.6 (0.8)</span>
<span id="S3.T4.9.9.9.9.12.3.7" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.12.3.7.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.12.3.7.1.1" class="ltx_text ltx_font_bold">40.4 (0.7)</span></span></span>
<span id="S3.T4.9.9.9.9.12.3.8" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.12.3.8.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.12.3.8.1.1" class="ltx_text ltx_font_bold">-6.2 (0.4)</span></span></span>
<span id="S3.T4.9.9.9.9.12.3.9" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">607.1 (1.1)</span>
<span id="S3.T4.9.9.9.9.12.3.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">40.5 (0.3)</span></span>
<span id="S3.T4.9.9.9.9.13.4" class="ltx_tr">
<span id="S3.T4.9.9.9.9.13.4.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">FT+L</span>
<span id="S3.T4.9.9.9.9.13.4.2" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">66.9</span>
<span id="S3.T4.9.9.9.9.13.4.3" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">99.1 (0.2)</span>
<span id="S3.T4.9.9.9.9.13.4.4" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">91.5 (0.5)</span>
<span id="S3.T4.9.9.9.9.13.4.5" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.13.4.5.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.13.4.5.1.1" class="ltx_text ltx_font_bold">48.7 (1.0)</span></span></span>
<span id="S3.T4.9.9.9.9.13.4.6" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">28.9 (0.8)</span>
<span id="S3.T4.9.9.9.9.13.4.7" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">70.3 (0.7)</span>
<span id="S3.T4.9.9.9.9.13.4.8" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">3.5 (0.3)</span>
<span id="S3.T4.9.9.9.9.13.4.9" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">621.4 (1.0)</span>
<span id="S3.T4.9.9.9.9.13.4.10" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">37.4 (0.3)</span></span>
<span id="S3.T4.9.9.9.9.14.5" class="ltx_tr">
<span id="S3.T4.9.9.9.9.14.5.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">KN</span>
<span id="S3.T4.9.9.9.9.14.5.2" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.14.5.2.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.14.5.2.1.1" class="ltx_text ltx_font_bold">35.6</span></span></span>
<span id="S3.T4.9.9.9.9.14.5.3" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.14.5.3.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.14.5.3.1.1" class="ltx_text ltx_font_bold">28.7 (1.0)</span></span></span>
<span id="S3.T4.9.9.9.9.14.5.4" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.14.5.4.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.14.5.4.1.1" class="ltx_text ltx_font_bold">-3.4 (0.3)</span></span></span>
<span id="S3.T4.9.9.9.9.14.5.5" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.14.5.5.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.14.5.5.1.1" class="ltx_text ltx_font_bold">28.0 (0.9)</span></span></span>
<span id="S3.T4.9.9.9.9.14.5.6" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.14.5.6.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.14.5.6.1.1" class="ltx_text ltx_font_bold">-3.3 (0.2)</span></span></span>
<span id="S3.T4.9.9.9.9.14.5.7" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">72.9 (0.7)</span>
<span id="S3.T4.9.9.9.9.14.5.8" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">3.7 (0.2)</span>
<span id="S3.T4.9.9.9.9.14.5.9" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.14.5.9.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.14.5.9.1.1" class="ltx_text ltx_font_bold">570.4 (2.3)</span></span></span>
<span id="S3.T4.9.9.9.9.14.5.10" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.14.5.10.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.14.5.10.1.1" class="ltx_text ltx_font_bold">30.3 (0.3)</span></span></span></span>
<span id="S3.T4.9.9.9.9.15.6" class="ltx_tr">
<span id="S3.T4.9.9.9.9.15.6.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">KE</span>
<span id="S3.T4.9.9.9.9.15.6.2" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">52.2</span>
<span id="S3.T4.9.9.9.9.15.6.3" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">84.3 (0.8)</span>
<span id="S3.T4.9.9.9.9.15.6.4" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">33.9 (0.9)</span>
<span id="S3.T4.9.9.9.9.15.6.5" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">75.4 (0.8)</span>
<span id="S3.T4.9.9.9.9.15.6.6" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">14.6 (0.6)</span>
<span id="S3.T4.9.9.9.9.15.6.7" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.15.6.7.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.15.6.7.1.1" class="ltx_text ltx_font_bold">30.9 (0.7)</span></span></span>
<span id="S3.T4.9.9.9.9.15.6.8" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.15.6.8.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.15.6.8.1.1" class="ltx_text ltx_font_bold">-11.0 (0.5)</span></span></span>
<span id="S3.T4.9.9.9.9.15.6.9" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.15.6.9.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.15.6.9.1.1" class="ltx_text ltx_font_bold">586.6 (2.1)</span></span></span>
<span id="S3.T4.9.9.9.9.15.6.10" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">31.2 (0.3)</span></span>
<span id="S3.T4.9.9.9.9.16.7" class="ltx_tr">
<span id="S3.T4.9.9.9.9.16.7.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">KE-CF</span>
<span id="S3.T4.9.9.9.9.16.7.2" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.16.7.2.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.16.7.2.1.1" class="ltx_text ltx_font_bold">18.1</span></span></span>
<span id="S3.T4.9.9.9.9.16.7.3" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">99.9 (0.1)</span>
<span id="S3.T4.9.9.9.9.16.7.4" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">97.0 (0.2)</span>
<span id="S3.T4.9.9.9.9.16.7.5" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">95.8 (0.4)</span>
<span id="S3.T4.9.9.9.9.16.7.6" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">59.2 (0.8)</span>
<span id="S3.T4.9.9.9.9.16.7.7" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.16.7.7.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.16.7.7.1.1" class="ltx_text ltx_font_bold">6.9 (0.3)</span></span></span>
<span id="S3.T4.9.9.9.9.16.7.8" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.16.7.8.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.16.7.8.1.1" class="ltx_text ltx_font_bold">-63.2 (0.7)</span></span></span>
<span id="S3.T4.9.9.9.9.16.7.9" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.16.7.9.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.16.7.9.1.1" class="ltx_text ltx_font_bold">383.0 (4.1)</span></span></span>
<span id="S3.T4.9.9.9.9.16.7.10" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.16.7.10.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.16.7.10.1.1" class="ltx_text ltx_font_bold">24.5 (0.4)</span></span></span></span>
<span id="S3.T4.9.9.9.9.17.8" class="ltx_tr">
<span id="S3.T4.9.9.9.9.17.8.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">MEND</span>
<span id="S3.T4.9.9.9.9.17.8.2" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">57.9</span>
<span id="S3.T4.9.9.9.9.17.8.3" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">99.1 (0.2)</span>
<span id="S3.T4.9.9.9.9.17.8.4" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">70.9 (0.8)</span>
<span id="S3.T4.9.9.9.9.17.8.5" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">65.4 (0.9)</span>
<span id="S3.T4.9.9.9.9.17.8.6" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">12.2 (0.6)</span>
<span id="S3.T4.9.9.9.9.17.8.7" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.17.8.7.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.17.8.7.1.1" class="ltx_text ltx_font_bold">37.9 (0.7)</span></span></span>
<span id="S3.T4.9.9.9.9.17.8.8" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.17.8.8.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.17.8.8.1.1" class="ltx_text ltx_font_bold">-11.6 (0.5)</span></span></span>
<span id="S3.T4.9.9.9.9.17.8.9" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.17.8.9.1" class="ltx_text" style="color:#009900;"><span id="S3.T4.9.9.9.9.17.8.9.1.1" class="ltx_text ltx_font_bold">624.2 (0.4)</span></span></span>
<span id="S3.T4.9.9.9.9.17.8.10" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">34.8 (0.3)</span></span>
<span id="S3.T4.9.9.9.9.18.9" class="ltx_tr">
<span id="S3.T4.9.9.9.9.18.9.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">MEND-CF</span>
<span id="S3.T4.9.9.9.9.18.9.2" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.18.9.2.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.18.9.2.1.1" class="ltx_text ltx_font_bold">14.9</span></span></span>
<span id="S3.T4.9.9.9.9.18.9.3" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.18.9.3.1" class="ltx_text" style="color:#009900;"><span id="S3.T4.9.9.9.9.18.9.3.1.1" class="ltx_text ltx_font_bold">100.0 (0.0)</span></span></span>
<span id="S3.T4.9.9.9.9.18.9.4" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.18.9.4.1" class="ltx_text" style="color:#009900;"><span id="S3.T4.9.9.9.9.18.9.4.1.1" class="ltx_text ltx_font_bold">99.2 (0.1)</span></span></span>
<span id="S3.T4.9.9.9.9.18.9.5" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.18.9.5.1" class="ltx_text" style="color:#009900;"><span id="S3.T4.9.9.9.9.18.9.5.1.1" class="ltx_text ltx_font_bold">97.0 (0.3)</span></span></span>
<span id="S3.T4.9.9.9.9.18.9.6" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.18.9.6.1" class="ltx_text" style="color:#009900;"><span id="S3.T4.9.9.9.9.18.9.6.1.1" class="ltx_text ltx_font_bold">65.6 (0.7)</span></span></span>
<span id="S3.T4.9.9.9.9.18.9.7" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.18.9.7.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.18.9.7.1.1" class="ltx_text ltx_font_bold">5.5 (0.3)</span></span></span>
<span id="S3.T4.9.9.9.9.18.9.8" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.18.9.8.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.18.9.8.1.1" class="ltx_text ltx_font_bold">-69.9 (0.6)</span></span></span>
<span id="S3.T4.9.9.9.9.18.9.9" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.18.9.9.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.18.9.9.1.1" class="ltx_text ltx_font_bold">570.0 (2.1)</span></span></span>
<span id="S3.T4.9.9.9.9.18.9.10" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">33.2 (0.3)</span></span>
<span id="S3.T4.9.9.9.9.19.10" class="ltx_tr">
<span id="S3.T4.9.9.9.9.19.10.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">ROME</span>
<span id="S3.T4.9.9.9.9.19.10.2" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.19.10.2.1" class="ltx_text" style="color:#009900;"><span id="S3.T4.9.9.9.9.19.10.2.1.1" class="ltx_text ltx_font_bold">89.2</span></span></span>
<span id="S3.T4.9.9.9.9.19.10.3" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">100.0 (0.1)</span>
<span id="S3.T4.9.9.9.9.19.10.4" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">97.9 (0.2)</span>
<span id="S3.T4.9.9.9.9.19.10.5" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">96.4 (0.3)</span>
<span id="S3.T4.9.9.9.9.19.10.6" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">62.7 (0.8)</span>
<span id="S3.T4.9.9.9.9.19.10.7" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.19.10.7.1" class="ltx_text" style="color:#009900;"><span id="S3.T4.9.9.9.9.19.10.7.1.1" class="ltx_text ltx_font_bold">75.4 (0.7)</span></span></span>
<span id="S3.T4.9.9.9.9.19.10.8" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.19.10.8.1" class="ltx_text" style="color:#009900;"><span id="S3.T4.9.9.9.9.19.10.8.1.1" class="ltx_text ltx_font_bold">4.2 (0.2)</span></span></span>
<span id="S3.T4.9.9.9.9.19.10.9" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">621.9 (0.5)</span>
<span id="S3.T4.9.9.9.9.19.10.10" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.19.10.10.1" class="ltx_text" style="color:#009900;"><span id="S3.T4.9.9.9.9.19.10.10.1.1" class="ltx_text ltx_font_bold">41.9 (0.3)</span></span></span></span>
<span id="S3.T4.9.9.9.9.20.11" class="ltx_tr">
<span id="S3.T4.9.9.9.9.20.11.1" class="ltx_td ltx_align_left ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;">GPT-J</span>
<span id="S3.T4.9.9.9.9.20.11.2" class="ltx_td ltx_align_right ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;">23.6</span>
<span id="S3.T4.9.9.9.9.20.11.3" class="ltx_td ltx_align_right ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;">16.3 (1.6)</span>
<span id="S3.T4.9.9.9.9.20.11.4" class="ltx_td ltx_align_right ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;">-7.2 (0.7)</span>
<span id="S3.T4.9.9.9.9.20.11.5" class="ltx_td ltx_align_right ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;">18.6 (1.5)</span>
<span id="S3.T4.9.9.9.9.20.11.6" class="ltx_td ltx_align_right ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;">-7.4 (0.6)</span>
<span id="S3.T4.9.9.9.9.20.11.7" class="ltx_td ltx_align_right ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;">83.0 (1.1)</span>
<span id="S3.T4.9.9.9.9.20.11.8" class="ltx_td ltx_align_right ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;">7.3 (0.5)</span>
<span id="S3.T4.9.9.9.9.20.11.9" class="ltx_td ltx_align_right ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;">621.8 (0.6)</span>
<span id="S3.T4.9.9.9.9.20.11.10" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:3.0pt;padding-right:3.0pt;">29.8 (0.5)</span></span>
<span id="S3.T4.9.9.9.9.21.12" class="ltx_tr">
<span id="S3.T4.9.9.9.9.21.12.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">FT</span>
<span id="S3.T4.9.9.9.9.21.12.2" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.21.12.2.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.21.12.2.1.1" class="ltx_text ltx_font_bold">25.5</span></span></span>
<span id="S3.T4.9.9.9.9.21.12.3" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.21.12.3.1" class="ltx_text" style="color:#009900;"><span id="S3.T4.9.9.9.9.21.12.3.1.1" class="ltx_text ltx_font_bold">100.0 (0.0)</span></span></span>
<span id="S3.T4.9.9.9.9.21.12.4" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.21.12.4.1" class="ltx_text" style="color:#009900;"><span id="S3.T4.9.9.9.9.21.12.4.1.1" class="ltx_text ltx_font_bold">99.9 (0.0)</span></span></span>
<span id="S3.T4.9.9.9.9.21.12.5" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">96.6 (0.6)</span>
<span id="S3.T4.9.9.9.9.21.12.6" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;">71.0 (1.5)</span>
<span id="S3.T4.9.9.9.9.21.12.7" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.21.12.7.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.21.12.7.1.1" class="ltx_text ltx_font_bold">10.3 (0.8)</span></span></span>
<span id="S3.T4.9.9.9.9.21.12.8" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.21.12.8.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.21.12.8.1.1" class="ltx_text ltx_font_bold">-50.7 (1.3)</span></span></span>
<span id="S3.T4.9.9.9.9.21.12.9" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.21.12.9.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.21.12.9.1.1" class="ltx_text ltx_font_bold">387.8 (7.3)</span></span></span>
<span id="S3.T4.9.9.9.9.21.12.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.21.12.10.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.21.12.10.1.1" class="ltx_text ltx_font_bold">24.6 (0.8)</span></span></span></span>
<span id="S3.T4.9.9.9.9.22.13" class="ltx_tr">
<span id="S3.T4.9.9.9.9.22.13.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">FT+L</span>
<span id="S3.T4.9.9.9.9.22.13.2" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">68.7</span>
<span id="S3.T4.9.9.9.9.22.13.3" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">99.6 (0.3)</span>
<span id="S3.T4.9.9.9.9.22.13.4" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">95.0 (0.6)</span>
<span id="S3.T4.9.9.9.9.22.13.5" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.22.13.5.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.22.13.5.1.1" class="ltx_text ltx_font_bold">47.9 (1.9)</span></span></span>
<span id="S3.T4.9.9.9.9.22.13.6" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">30.4 (1.5)</span>
<span id="S3.T4.9.9.9.9.22.13.7" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">78.6 (1.2)</span>
<span id="S3.T4.9.9.9.9.22.13.8" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.22.13.8.1" class="ltx_text" style="color:#009900;"><span id="S3.T4.9.9.9.9.22.13.8.1.1" class="ltx_text ltx_font_bold">6.8 (0.5)</span></span></span>
<span id="S3.T4.9.9.9.9.22.13.9" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.22.13.9.1" class="ltx_text" style="color:#009900;"><span id="S3.T4.9.9.9.9.22.13.9.1.1" class="ltx_text ltx_font_bold">622.8 (0.6)</span></span></span>
<span id="S3.T4.9.9.9.9.22.13.10" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">35.5 (0.5)</span></span>
<span id="S3.T4.9.9.9.9.23.14" class="ltx_tr">
<span id="S3.T4.9.9.9.9.23.14.1" class="ltx_td ltx_align_left" style="padding-left:3.0pt;padding-right:3.0pt;">MEND</span>
<span id="S3.T4.9.9.9.9.23.14.2" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">63.2</span>
<span id="S3.T4.9.9.9.9.23.14.3" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">97.4 (0.7)</span>
<span id="S3.T4.9.9.9.9.23.14.4" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">71.5 (1.6)</span>
<span id="S3.T4.9.9.9.9.23.14.5" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.23.14.5.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.23.14.5.1.1" class="ltx_text ltx_font_bold">53.6 (1.9)</span></span></span>
<span id="S3.T4.9.9.9.9.23.14.6" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">11.0 (1.3)</span>
<span id="S3.T4.9.9.9.9.23.14.7" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">53.9 (1.4)</span>
<span id="S3.T4.9.9.9.9.23.14.8" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.23.14.8.1" class="ltx_text" style="color:#BF0000;"><span id="S3.T4.9.9.9.9.23.14.8.1.1" class="ltx_text ltx_font_bold">-6.0 (0.9)</span></span></span>
<span id="S3.T4.9.9.9.9.23.14.9" class="ltx_td ltx_align_right" style="padding-left:3.0pt;padding-right:3.0pt;">620.5 (0.7)</span>
<span id="S3.T4.9.9.9.9.23.14.10" class="ltx_td ltx_align_center" style="padding-left:3.0pt;padding-right:3.0pt;">32.6 (0.5)</span></span>
<span id="S3.T4.9.9.9.9.24.15" class="ltx_tr">
<span id="S3.T4.9.9.9.9.24.15.1" class="ltx_td ltx_align_left ltx_border_bb" style="padding-left:3.0pt;padding-right:3.0pt;">ROME</span>
<span id="S3.T4.9.9.9.9.24.15.2" class="ltx_td ltx_align_right ltx_border_bb" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.24.15.2.1" class="ltx_text" style="color:#009900;"><span id="S3.T4.9.9.9.9.24.15.2.1.1" class="ltx_text ltx_font_bold">91.5</span></span></span>
<span id="S3.T4.9.9.9.9.24.15.3" class="ltx_td ltx_align_right ltx_border_bb" style="padding-left:3.0pt;padding-right:3.0pt;">99.9 (0.1)</span>
<span id="S3.T4.9.9.9.9.24.15.4" class="ltx_td ltx_align_right ltx_border_bb" style="padding-left:3.0pt;padding-right:3.0pt;">99.4 (0.3)</span>
<span id="S3.T4.9.9.9.9.24.15.5" class="ltx_td ltx_align_right ltx_border_bb" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.24.15.5.1" class="ltx_text" style="color:#009900;"><span id="S3.T4.9.9.9.9.24.15.5.1.1" class="ltx_text ltx_font_bold">99.1 (0.3)</span></span></span>
<span id="S3.T4.9.9.9.9.24.15.6" class="ltx_td ltx_align_right ltx_border_bb" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.24.15.6.1" class="ltx_text" style="color:#009900;"><span id="S3.T4.9.9.9.9.24.15.6.1.1" class="ltx_text ltx_font_bold">74.1 (1.3)</span></span></span>
<span id="S3.T4.9.9.9.9.24.15.7" class="ltx_td ltx_align_right ltx_border_bb" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.24.15.7.1" class="ltx_text" style="color:#009900;"><span id="S3.T4.9.9.9.9.24.15.7.1.1" class="ltx_text ltx_font_bold">78.9 (1.2)</span></span></span>
<span id="S3.T4.9.9.9.9.24.15.8" class="ltx_td ltx_align_right ltx_border_bb" style="padding-left:3.0pt;padding-right:3.0pt;">5.2 (0.5)</span>
<span id="S3.T4.9.9.9.9.24.15.9" class="ltx_td ltx_align_right ltx_border_bb" style="padding-left:3.0pt;padding-right:3.0pt;">620.1 (0.9)</span>
<span id="S3.T4.9.9.9.9.24.15.10" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:3.0pt;padding-right:3.0pt;"><span id="S3.T4.9.9.9.9.24.15.10.1" class="ltx_text" style="color:#009900;"><span id="S3.T4.9.9.9.9.24.15.10.1.1" class="ltx_text ltx_font_bold">43.0 (0.6)</span></span></span></span>
</span>
</span></span></p>
</span></div>
</figure>
<div id="S3.SS4.p2" class="ltx_para">
<p id="S3.SS4.p2.1" class="ltx_p">Table <a href="#S3.T4" title="Table 4 ‣ 3.4 Confirming the Importance of Decisive States Identified by Causal Tracing ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> showcases quantitative results on GPT-2 XL (1.5B) and GPT-J (6B) over 7,500 and 2,000-record test sets in <span id="S3.SS4.p2.1.1" class="ltx_text ltx_font_smallcaps">CounterFact</span>, respectively. In this experiment, in addition to the baselines tested above, we compare with a method based on neuron interpretability, Knowledge Neurons <span id="S3.SS4.p2.1.2" class="ltx_text ltx_font_bold">(KN)</span> <cite class="ltx_cite ltx_citemacro_citep">(Dai et al., <a href="#bib.bib9" title="" class="ltx_ref">2022</a>)</cite>, which first selects neurons associated with knowledge via gradient-based attribution, then modifies MLP weights at corresponding rows by adding scaled embedding vectors. We observe that <span id="S3.SS4.p2.1.3" class="ltx_text ltx_font_bold">all tested methods other than ROME exhibit one or both of the following problems</span>: (F1) overfitting to the counterfactual statement and failing to generalize, or (F2) underfitting and predicting the same new output for unrelated subjects. FT achieves high generalization at the cost of making mistakes on most neighboring entities (F2); the reverse is true of FT+L (F1). KE- and <span id="S3.SS4.p2.1.4" class="ltx_text">MEND-edited</span> models exhibit issues with both F1+F2; generalization, consistency, and bleedover are poor despite high efficacy, indicating regurgitation. KN is unable to make effective edits (F1+F2). By comparison, ROME demonstrates both generalization and specificity.</p>
</div>
</section>
<section id="S3.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.5 </span>Comparing Generation Results</h3>

<figure id="S3.F6" class="ltx_figure"><img src="/html/2202.05262/assets/x6.png" id="S3.F6.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="197" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S3.F6.4.2.1" class="ltx_text" style="font-size:90%;">Figure 6</span>: </span><span id="S3.F6.2.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Comparison of generated text<span id="S3.F6.2.1.1" class="ltx_text ltx_font_medium">. Prompts are <span id="S3.F6.2.1.1.1" class="ltx_text ltx_font_italic">italicized</span>, <span id="S3.F6.2.1.1.2" class="ltx_text" style="color:#009900;"> green</span> and <span id="S3.F6.2.1.1.3" class="ltx_text" style="color:#FF0000;"> red</span> indicate keywords reflecting correct and incorrect behavior, respectively, and <span id="S3.F6.2.1.1.4" class="ltx_text" style="color:#21ABCC;"> blue</span> indicates a factually-incorrect keyword that was already present in <math id="S3.F6.2.1.1.m1.1" class="ltx_Math" alttext="G" display="inline"><semantics id="S3.F6.2.1.1.m1.1b"><mi id="S3.F6.2.1.1.m1.1.1" xref="S3.F6.2.1.1.m1.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="S3.F6.2.1.1.m1.1c"><ci id="S3.F6.2.1.1.m1.1.1.cmml" xref="S3.F6.2.1.1.m1.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F6.2.1.1.m1.1d">G</annotation></semantics></math> before rewriting. See Section <a href="#S3.SS5" title="3.5 Comparing Generation Results ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.5</span></a> for detailed analysis.</span></span></figcaption>
</figure>
<div id="S3.SS5.p1" class="ltx_para">
<p id="S3.SS5.p1.1" class="ltx_p">Figure <a href="#S3.F6" title="Figure 6 ‣ 3.5 Comparing Generation Results ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> compares generated text after applying the counterfactual “<span id="S3.SS5.p1.1.1" class="ltx_text ltx_font_italic">Pierre Curie’s area of work is medicine</span>” to GPT-2 XL (he is actually a physicist). <span id="S3.SS5.p1.1.2" class="ltx_text ltx_font_bold">Generalization:</span> In this case, FT and ROME generalize well to paraphrases, describing the subject as a physician rather than a physicist for various wordings. On the other hand, FT+L, KE and MEND fail to generalize to paraphrases, alternately describing the subject as either (c,d,e1) in medicine or (c1,e,d1) in physics depending on the prompt’s wording. KE (d) demonstrates a problem with fluency, favoring nonsense repetition of the word <em id="S3.SS5.p1.1.3" class="ltx_emph ltx_font_italic">medicine</em>. <span id="S3.SS5.p1.1.4" class="ltx_text ltx_font_bold">Specificity:</span> FT, KE, and MEND have problems with specificity, changing the profession of a totally unrelated subject. Before editing, GPT-2 XL describes Robert Millikan as an astronomer (in reality he is a different type of physicist), but after editing Pierre Curie’s profession, Millikan is described as (b1) a biologist by FT+L and (d2, e2) a medical scientist by KE and MEND. In contrast, ROME is specific, leaving Millikan’s field unchanged. See
Appendix <a href="#A7" title="Appendix G Generation Examples ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">G</span></a> for additional examples.</p>
</div>
</section>
<section id="S3.SS6" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.6 </span>Human evaluation</h3>

<div id="S3.SS6.p1" class="ltx_para">
<p id="S3.SS6.p1.1" class="ltx_p">To evaluate the quality of generated text after applying ROME, we ask 15 volunteers to evaluate models by comparing generated text samples on the basis of both fluency and consistency with the inserted fact. Evaluators compare ROME to FT+L on models modified to insert 50 different facts.</p>
</div>
<div id="S3.SS6.p2" class="ltx_para">
<p id="S3.SS6.p2.1" class="ltx_p">We find that evaluators are 1.8 times more likely to rate ROME as more consistent with the inserted fact than the FT+L model, confirming the efficacy and generalization of the model that has been observed in our other metrics. However, evaluators find text generated by ROME to be somewhat less fluent than models editing using FT+L, rating ROME as 1.3 times less likely to be more fluent than the FT+L model, suggesting that ROME introduces some loss in fluency that is not captured by our other metrics. Further details of the human evaluation can be found in Appendix <a href="#A10" title="Appendix J Human Evaluation ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">J</span></a>.</p>
</div>
</section>
<section id="S3.SS7" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.7 </span>Limitations</h3>

<div id="S3.SS7.p1" class="ltx_para">
<p id="S3.SS7.p1.1" class="ltx_p">The purpose of ROME is to serve as a tool for understanding mechanisms of knowledge storage: it only edits a single fact at a time, and it is not intended as a practical method for large-scale model training. Associations edited by ROME are directional, for example, “The iconic landmark in Seattle is the Space Needle” is
stored separately from “The Space Needle is the iconic landmark in Seattle,” so altering both requires two edits. A scalable approach for multiple simultaneous edits built upon the ideas in ROME is developed in <cite class="ltx_cite ltx_citemacro_citet">Meng, Sen Sharma, Andonian, Belinkov, and
Bau (<a href="#bib.bib26" title="" class="ltx_ref">2022</a>)</cite>.</p>
</div>
<div id="S3.SS7.p2" class="ltx_para">
<p id="S3.SS7.p2.1" class="ltx_p">ROME and Causal Tracing have shed light on factual association within GPT, but we have not investigated other kinds of learned beliefs such as logical, spatial, or numerical knowledge. Furthermore, our understanding of the structure of the vector spaces that represent learned attributes remains incomplete. Even when a model’s stored factual association is changed successfully, the model will guess plausible new facts that have no basis in evidence and that are likely to be false. This may limit the usefulness of a language model as a source of facts.</p>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Related Work</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.1" class="ltx_p">The question of what a model learns is a fundamental problem that has been approached from several directions. One line of work studies which properties are encoded in internal model representations, most commonly by training a probing classifier to predict said properties from the representations <cite class="ltx_cite ltx_citemacro_citep">(Ettinger et al., <a href="#bib.bib15" title="" class="ltx_ref">2016</a>; Adi et al., <a href="#bib.bib1" title="" class="ltx_ref">2017</a>; Hupkes et al., <a href="#bib.bib20" title="" class="ltx_ref">2018</a>; Conneau et al., <a href="#bib.bib8" title="" class="ltx_ref">2018</a>; Belinkov et al., <a href="#bib.bib6" title="" class="ltx_ref">2017</a>; Belinkov &amp; Glass, <a href="#bib.bib5" title="" class="ltx_ref">2019</a>, inter alia)</cite>.
However, such approaches suffer from various limitations, notably being
dissociated from the network’s behavior <cite class="ltx_cite ltx_citemacro_citep">(Belinkov, <a href="#bib.bib4" title="" class="ltx_ref">2021</a>)</cite>. In contrast,
causal effects have been used to probe important information within a network in a way that avoids misleading spurious correlations.
<cite class="ltx_cite ltx_citemacro_citet">Vig et al. (<a href="#bib.bib38" title="" class="ltx_ref">2020b</a>, <a href="#bib.bib37" title="" class="ltx_ref">a</a>)</cite> introduced the use of causal mediation analysis to identify individual neurons that contribute to biased gender assumptions, and <cite class="ltx_cite ltx_citemacro_citet">Finlayson et al. (<a href="#bib.bib17" title="" class="ltx_ref">2021</a>)</cite> have used a similar methodology to investigate mechanisms of syntactic agreement in language models.
<cite class="ltx_cite ltx_citemacro_citet">Feder et al. (<a href="#bib.bib16" title="" class="ltx_ref">2021</a>)</cite> described a framework that applies interventions on representations and weights to understand the causal structure of models.
<cite class="ltx_cite ltx_citemacro_citet">Elazar et al. (<a href="#bib.bib13" title="" class="ltx_ref">2021b</a>)</cite> proposed erasing specific information from a representation in order to measure its causal effect.
Extending these ideas, our Causal Tracing method introduces paired interventions that allow explicit measurement of causal <em id="S4.p1.1.1" class="ltx_emph ltx_font_italic">indirect effects</em> <cite class="ltx_cite ltx_citemacro_citep">(Pearl, <a href="#bib.bib28" title="" class="ltx_ref">2001</a>)</cite> of individual hidden state vectors.</p>
</div>
<div id="S4.p2" class="ltx_para">
<p id="S4.p2.1" class="ltx_p">Another line of work aims to assess the knowledge within LMs by evaluating whether the model predict pieces of knowledge. A common strategy is to define a fill-in-the-blank prompt, and let a masked LM complete it <cite class="ltx_cite ltx_citemacro_citep">(Petroni et al., <a href="#bib.bib30" title="" class="ltx_ref">2019</a>, <a href="#bib.bib31" title="" class="ltx_ref">2020</a>)</cite>. Later work showed that knowledge extraction can be improved by diversifying the prompts <cite class="ltx_cite ltx_citemacro_citep">(Jiang et al., <a href="#bib.bib21" title="" class="ltx_ref">2020</a>; Zhong et al., <a href="#bib.bib42" title="" class="ltx_ref">2021</a>)</cite>, or by fine-tuning a model on open-domain textual facts <cite class="ltx_cite ltx_citemacro_citep">(Roberts et al., <a href="#bib.bib34" title="" class="ltx_ref">2020</a>)</cite>. However, constructing prompts from supervised knowledge extraction data risks learning new knowledge instead of recalling existing knowledge in an LM <cite class="ltx_cite ltx_citemacro_citep">(Zhong et al., <a href="#bib.bib42" title="" class="ltx_ref">2021</a>)</cite>. More recently, <cite class="ltx_cite ltx_citemacro_citet">Elazar et al. (<a href="#bib.bib12" title="" class="ltx_ref">2021a</a>)</cite> introduced ParaRel, a curated dataset of paraphrased prompts and facts. We use it as a basis for constructing <span id="S4.p2.1.1" class="ltx_text ltx_font_smallcaps">CounterFact</span>, which enables fine-grained measurements of knowledge extraction and editing along multiple dimensions.
Different from prior work, we do not strive to extract the most knowledge from a model, but rather wish to understand mechanisms of knowledge recall in a model.</p>
</div>
<div id="S4.p3" class="ltx_para">
<p id="S4.p3.1" class="ltx_p">Finally,
a few studies aim to localize and modify the computation of knowledge within transformers.
<cite class="ltx_cite ltx_citemacro_citet">Geva et al. (<a href="#bib.bib18" title="" class="ltx_ref">2021</a>)</cite> identify the MLP layers in a (masked LM) transformer as key–value memories of entities and information associated with that entity.
Building on this finding,
<cite class="ltx_cite ltx_citemacro_citet">Dai et al. (<a href="#bib.bib9" title="" class="ltx_ref">2022</a>)</cite> demonstrate a method to edit facts in BERT by writing the embedding of the object into certain rows of the MLP matrix. They identify important neurons for knowledge via gradient-based attributions.
<cite class="ltx_cite ltx_citemacro_citet">De Cao et al. (<a href="#bib.bib10" title="" class="ltx_ref">2021</a>)</cite> train a hyper-network to predict a weight update at test time, which will alter a fact. They experiment with BERT and BART <cite class="ltx_cite ltx_citemacro_citep">(Lewis et al., <a href="#bib.bib25" title="" class="ltx_ref">2020</a>)</cite>, a sequence-to-sequence model, and focus on models fine-tuned for question answering.
<cite class="ltx_cite ltx_citemacro_citet">Mitchell et al. (<a href="#bib.bib27" title="" class="ltx_ref">2021</a>)</cite> presents a hyper-network method that learns to transform the decomposed terms of the gradient in order to efficiently predict a knowledge update, and demonstrates the ability to scale up to large models including T5 <cite class="ltx_cite ltx_citemacro_citep">(Raffel et al., <a href="#bib.bib33" title="" class="ltx_ref">2020</a>)</cite> and GPT-J <cite class="ltx_cite ltx_citemacro_citep">(Wang &amp; Komatsuzaki, <a href="#bib.bib39" title="" class="ltx_ref">2021</a>)</cite>.
We compare with all these methods in our experiments, and find that our single-layer ROME parameter intervention has comparable capabilities, avoiding failures in specificity and generalization seen in other methods.</p>
</div>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Conclusion</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">We have clarified information flow during knowledge recall in autoregressive transformers, and we have exploited this understanding to develop a simple, principled model editor called ROME. Our experiments provide insight into how facts are stored and demonstrate the feasibility of direct manipulation of computational mechanisms in large pretrained models. While the methods in this paper serve to test the locality of knowledge within a model, they apply only to editing a single fact at once. Adapting the approach to scale up to many more facts is the subject of other work such as <cite class="ltx_cite ltx_citemacro_citet">Meng, Sen Sharma, Andonian, Belinkov, and
Bau (<a href="#bib.bib26" title="" class="ltx_ref">2022</a>)</cite>.</p>
</div>
<div id="S5.p2" class="ltx_para">
<p id="S5.p2.1" class="ltx_p">Code, interactive notebooks, dataset, benchmarks, and further visualizations are open-sourced at <a target="_blank" href="https://rome.baulab.info" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://rome.baulab.info</a>.</p>
</div>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Ethical Considerations</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">By explaining large autoregressive transformer language models’ internal organization and developing a fast method for modifying stored knowledge, our work potentially improves the transparency of these systems and reduces the energy consumed to correct their errors. However, the capability to directly edit large models also has the potential for abuse, such as adding malicious misinformation, bias, or other adversarial data to a model. Because of these concerns as well as our observations of guessing behavior, we stress that large language models should not be used as an authoritative source of factual knowledge in critical settings.</p>
</div>
</section>
<section id="Sx1" class="ltx_section">
<h2 class="ltx_title ltx_title_section" style="font-size:90%;">Acknowledgements</h2>

<div id="Sx1.p1" class="ltx_para">
<p id="Sx1.p1.1" class="ltx_p"><span id="Sx1.p1.1.1" class="ltx_text" style="font-size:90%;">We are grateful to Antonio Torralba, Martin Wattenberg, and Bill Ferguson, whose insightful discussions, financial support, and encouragement enabled this project. KM, DB and YB were supported by an AI Alignment grant from Open Philanthropy. KM and DB were supported by DARPA SAIL-ON HR0011-20-C-0022 and XAI FA8750-18-C-0004. YB was supported by the ISRAEL SCIENCE FOUNDATION (grant No. 448/20) and an Azrieli Foundation Early Career Faculty Fellowship.
</span><span id="Sx1.p1.1.2" class="ltx_text"></span></p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Adi et al. (2017)</span>
<span class="ltx_bibblock">
Adi, Y., Kermany, E., Belinkov, Y., Lavi, O., and Goldberg, Y.

</span>
<span class="ltx_bibblock">Fine-grained analysis of sentence embeddings using auxiliary
prediction tasks.

</span>
<span class="ltx_bibblock">In <em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations
(ICLR)</em>, April 2017.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Anderson (1972)</span>
<span class="ltx_bibblock">
Anderson, J. A.

</span>
<span class="ltx_bibblock">A simple neural network generating an interactive memory.

</span>
<span class="ltx_bibblock"><em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">Mathematical biosciences</em>, 14(3-4):197–220, 1972.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bau et al. (2020)</span>
<span class="ltx_bibblock">
Bau, D., Liu, S., Wang, T., Zhu, J.-Y., and Torralba, A.

</span>
<span class="ltx_bibblock">Rewriting a deep generative model.

</span>
<span class="ltx_bibblock">In <em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">Proceedings of the European Conference on Computer Vision
(ECCV)</em>, 2020.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Belinkov (2021)</span>
<span class="ltx_bibblock">
Belinkov, Y.

</span>
<span class="ltx_bibblock">Probing Classifiers: Promises, Shortcomings, and Advances.

</span>
<span class="ltx_bibblock"><em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">Computational Linguistics</em>, pp.  1–13, 11 2021.

</span>
<span class="ltx_bibblock">ISSN 0891-2017.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1162/coli˙a˙00422</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.1162/coli_a_00422" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1162/coli_a_00422</a>.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Belinkov &amp; Glass (2019)</span>
<span class="ltx_bibblock">
Belinkov, Y. and Glass, J.

</span>
<span class="ltx_bibblock">Analysis methods in neural language processing: A survey.

</span>
<span class="ltx_bibblock"><em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">Transactions of the Association for Computational Linguistics</em>,
7:49–72, March 2019.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1162/tacl˙a˙00254</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/Q19-1004" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/Q19-1004</a>.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Belinkov et al. (2017)</span>
<span class="ltx_bibblock">
Belinkov, Y., Durrani, N., Dalvi, F., Sajjad, H., and Glass, J.

</span>
<span class="ltx_bibblock">What do neural machine translation models learn about morphology?

</span>
<span class="ltx_bibblock">In <em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 55th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)</em>, pp.  861–872,
Vancouver, Canada, July 2017. Association for Computational Linguistics.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/P17-1080</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/P17-1080" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/P17-1080</a>.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Brown et al. (2020)</span>
<span class="ltx_bibblock">
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P.,
Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S.,
Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler,
D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray,
S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever,
I., and Amodei, D.

</span>
<span class="ltx_bibblock">Language models are few-shot learners.

</span>
<span class="ltx_bibblock">In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin,
H. (eds.), <em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>,
volume 33, pp.  1877–1901. Curran Associates, Inc., 2020.

</span>
<span class="ltx_bibblock">URL
<a target="_blank" href="https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf</a>.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Conneau et al. (2018)</span>
<span class="ltx_bibblock">
Conneau, A., Kruszewski, G., Lample, G., Barrault, L., and Baroni, M.

</span>
<span class="ltx_bibblock">What you can cram into a single $&amp;!#* vector: Probing
sentence embeddings for linguistic properties.

</span>
<span class="ltx_bibblock">In <em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 56th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)</em>, pp.  2126–2136,
Melbourne, Australia, July 2018. Association for Computational Linguistics.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/P18-1198</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/P18-1198" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/P18-1198</a>.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dai et al. (2022)</span>
<span class="ltx_bibblock">
Dai, D., Dong, L., Hao, Y., Sui, Z., Chang, B., and Wei, F.

</span>
<span class="ltx_bibblock">Knowledge neurons in pretrained transformers.

</span>
<span class="ltx_bibblock">In <em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 60th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)</em>, pp.  8493–8502,
2022.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">De Cao et al. (2021)</span>
<span class="ltx_bibblock">
De Cao, N., Aziz, W., and Titov, I.

</span>
<span class="ltx_bibblock">Editing factual knowledge in language models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing</em>, pp.  6491–6506, Online and Punta Cana,
Dominican Republic, November 2021. Association for Computational Linguistics.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/2021.emnlp-main.522" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/2021.emnlp-main.522</a>.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Devlin et al. (2019)</span>
<span class="ltx_bibblock">
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.

</span>
<span class="ltx_bibblock">BERT: Pre-training of deep bidirectional transformers for language
understanding.

</span>
<span class="ltx_bibblock">In <em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers)</em>, pp.  4171–4186,
Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/N19-1423</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/N19-1423" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/N19-1423</a>.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Elazar et al. (2021a)</span>
<span class="ltx_bibblock">
Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H.,
and Goldberg, Y.

</span>
<span class="ltx_bibblock">Measuring and Improving Consistency in Pretrained Language Models.

</span>
<span class="ltx_bibblock"><em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">Transactions of the Association for Computational Linguistics</em>,
9:1012–1031, 09 2021a.

</span>
<span class="ltx_bibblock">ISSN 2307-387X.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1162/tacl˙a˙00410</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.1162/tacl_a_00410" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1162/tacl_a_00410</a>.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Elazar et al. (2021b)</span>
<span class="ltx_bibblock">
Elazar, Y., Ravfogel, S., Jacovi, A., and Goldberg, Y.

</span>
<span class="ltx_bibblock">Amnesic probing: Behavioral explanation with amnesic counterfactuals.

</span>
<span class="ltx_bibblock"><em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">Transactions of the Association for Computational Linguistics</em>,
9:160–175, 2021b.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Elhage et al. (2021)</span>
<span class="ltx_bibblock">
Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell,
A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D.,
Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L.,
Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S.,
and Olah, C.

</span>
<span class="ltx_bibblock">A mathematical framework for transformer circuits.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://transformer-circuits.pub/2021/framework/index.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://transformer-circuits.pub/2021/framework/index.html</a>,
December 2021.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ettinger et al. (2016)</span>
<span class="ltx_bibblock">
Ettinger, A., Elgohary, A., and Resnik, P.

</span>
<span class="ltx_bibblock">Probing for semantic evidence of composition by means of simple
classification tasks.

</span>
<span class="ltx_bibblock">In <em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 1st Workshop on Evaluating Vector-Space
Representations for NLP</em>, pp.  134–139, Berlin, Germany, August 2016.
Association for Computational Linguistics.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/W16-2524</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/W16-2524" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/W16-2524</a>.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Feder et al. (2021)</span>
<span class="ltx_bibblock">
Feder, A., Oved, N., Shalit, U., and Reichart, R.

</span>
<span class="ltx_bibblock">CausaLM: Causal model explanation through counterfactual language
models.

</span>
<span class="ltx_bibblock"><em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">Computational Linguistics</em>, 47(2):333–386,
2021.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Finlayson et al. (2021)</span>
<span class="ltx_bibblock">
Finlayson, M., Mueller, A., Gehrmann, S., Shieber, S., Linzen, T., and
Belinkov, Y.

</span>
<span class="ltx_bibblock">Causal analysis of syntactic agreement mechanisms in neural language
models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 59th Annual Meeting of the Association
for Computational Linguistics and the 11th International Joint Conference on
Natural Language Processing (Volume 1: Long Papers)</em>, pp.  1828–1843,
Online, August 2021. Association for Computational Linguistics.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2021.acl-long.144</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/2021.acl-long.144" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/2021.acl-long.144</a>.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Geva et al. (2021)</span>
<span class="ltx_bibblock">
Geva, M., Schuster, R., Berant, J., and Levy, O.

</span>
<span class="ltx_bibblock">Transformer feed-forward layers are key-value memories.

</span>
<span class="ltx_bibblock">In <em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing</em>, pp.  5484–5495, Online and Punta Cana,
Dominican Republic, November 2021. Association for Computational Linguistics.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/2021.emnlp-main.446" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/2021.emnlp-main.446</a>.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hase et al. (2021)</span>
<span class="ltx_bibblock">
Hase, P., Diab, M., Celikyilmaz, A., Li, X., Kozareva, Z., Stoyanov, V.,
Bansal, M., and Iyer, S.

</span>
<span class="ltx_bibblock">Do language models have beliefs? methods for detecting, updating, and
visualizing model beliefs.

</span>
<span class="ltx_bibblock"><em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2111.13654</em>, 2021.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hupkes et al. (2018)</span>
<span class="ltx_bibblock">
Hupkes, D., Veldhoen, S., and Zuidema, W.

</span>
<span class="ltx_bibblock">Visualisation and ’diagnostic classifiers’ reveal how recurrent and
recursive neural networks process hierarchical structure.

</span>
<span class="ltx_bibblock"><em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">Journal of Artificial Intelligence Research</em>, 61:907–926, 2018.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jiang et al. (2020)</span>
<span class="ltx_bibblock">
Jiang, Z., Xu, F. F., Araki, J., and Neubig, G.

</span>
<span class="ltx_bibblock">How can we know what language models know?

</span>
<span class="ltx_bibblock"><em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">Transactions of the Association for Computational Linguistics</em>,
8:423–438, 2020.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1162/tacl˙a˙00324</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/2020.tacl-1.28" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/2020.tacl-1.28</a>.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kingma &amp; Ba (2015)</span>
<span class="ltx_bibblock">
Kingma, D. P. and Ba, J.

</span>
<span class="ltx_bibblock">Adam: A method for stochastic optimization.

</span>
<span class="ltx_bibblock">In Bengio, Y. and LeCun, Y. (eds.), <em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">3rd International
Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May
7-9, 2015, Conference Track Proceedings</em>, 2015.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://arxiv.org/abs/1412.6980" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://arxiv.org/abs/1412.6980</a>.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kohonen (1972)</span>
<span class="ltx_bibblock">
Kohonen, T.

</span>
<span class="ltx_bibblock">Correlation matrix memories.

</span>
<span class="ltx_bibblock"><em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">IEEE transactions on computers</em>, 100(4):353–359, 1972.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Levy et al. (2017)</span>
<span class="ltx_bibblock">
Levy, O., Seo, M., Choi, E., and Zettlemoyer, L.

</span>
<span class="ltx_bibblock">Zero-shot relation extraction via reading comprehension.

</span>
<span class="ltx_bibblock">In <em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 21st Conference on Computational Natural
Language Learning (CoNLL 2017)</em>, pp.  333–342, Vancouver, Canada,
August 2017. Association for Computational Linguistics.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/K17-1034</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/K17-1034" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/K17-1034</a>.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lewis et al. (2020)</span>
<span class="ltx_bibblock">
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O.,
Stoyanov, V., and Zettlemoyer, L.

</span>
<span class="ltx_bibblock">BART: Denoising sequence-to-sequence pre-training for natural
language generation, translation, and comprehension.

</span>
<span class="ltx_bibblock">In <em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics</em>, pp.  7871–7880, Online, July 2020.
Association for Computational Linguistics.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2020.acl-main.703</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/2020.acl-main.703" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/2020.acl-main.703</a>.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Meng et al. (2022)</span>
<span class="ltx_bibblock">
Meng, K., Sen Sharma, A., Andonian, A., Belinkov, Y., and Bau, D.

</span>
<span class="ltx_bibblock">Mass-editing memory in a transformer.

</span>
<span class="ltx_bibblock"><em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2210.07229</em>, 2022.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mitchell et al. (2021)</span>
<span class="ltx_bibblock">
Mitchell, E., Lin, C., Bosselut, A., Finn, C., and Manning, C. D.

</span>
<span class="ltx_bibblock">Fast model editing at scale.

</span>
<span class="ltx_bibblock">In <em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>, 2021.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Pearl (2001)</span>
<span class="ltx_bibblock">
Pearl, J.

</span>
<span class="ltx_bibblock">Direct and indirect effects.

</span>
<span class="ltx_bibblock">In <em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">Proceedings of the Seventeenth conference on Uncertainty in
artificial intelligence</em>, pp.  411–420, 2001.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Pearl (2009)</span>
<span class="ltx_bibblock">
Pearl, J.

</span>
<span class="ltx_bibblock"><em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">Causality: Models, Reasoning and Inference</em>.

</span>
<span class="ltx_bibblock">Cambridge University Press, USA, 2nd edition, 2009.

</span>
<span class="ltx_bibblock">ISBN 052189560X.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Petroni et al. (2019)</span>
<span class="ltx_bibblock">
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y.,
and Miller, A.

</span>
<span class="ltx_bibblock">Language models as knowledge bases?

</span>
<span class="ltx_bibblock">In <em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2019 Conference on Empirical Methods in
Natural Language Processing and the 9th International Joint Conference on
Natural Language Processing (EMNLP-IJCNLP)</em>, pp.  2463–2473, Hong Kong,
China, November 2019. Association for Computational Linguistics.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/D19-1250</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/D19-1250" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/D19-1250</a>.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Petroni et al. (2020)</span>
<span class="ltx_bibblock">
Petroni, F., Lewis, P., Piktus, A., Rocktäschel, T., Wu, Y., Miller, A. H.,
and Riedel, S.

</span>
<span class="ltx_bibblock">How context affects language models’ factual predictions.

</span>
<span class="ltx_bibblock">In <em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">Automated Knowledge Base Construction</em>, 2020.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Radford et al. (2019)</span>
<span class="ltx_bibblock">
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.

</span>
<span class="ltx_bibblock">Language models are unsupervised multitask learners.

</span>
<span class="ltx_bibblock"><em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">OpenAI blog</em>, pp.  9, 2019.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Raffel et al. (2020)</span>
<span class="ltx_bibblock">
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou,
Y., Li, W., and Liu, P. J.

</span>
<span class="ltx_bibblock">Exploring the limits of transfer learning with a unified text-to-text
transformer.

</span>
<span class="ltx_bibblock"><em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">Journal of Machine Learning Research</em>, 21(140):1–67, 2020.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Roberts et al. (2020)</span>
<span class="ltx_bibblock">
Roberts, A., Raffel, C., and Shazeer, N.

</span>
<span class="ltx_bibblock">How much knowledge can you pack into the parameters of a language
model?

</span>
<span class="ltx_bibblock">In <em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP)</em>, pp.  5418–5426, Online, November
2020. Association for Computational Linguistics.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2020.emnlp-main.437</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/2020.emnlp-main.437" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/2020.emnlp-main.437</a>.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sundararajan et al. (2017)</span>
<span class="ltx_bibblock">
Sundararajan, M., Taly, A., and Yan, Q.

</span>
<span class="ltx_bibblock">Axiomatic attribution for deep networks.

</span>
<span class="ltx_bibblock">In <em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">International conference on machine learning</em>, pp. 3319–3328. PMLR, 2017.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Vaswani et al. (2017)</span>
<span class="ltx_bibblock">
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N.,
Kaiser, Ł., and Polosukhin, I.

</span>
<span class="ltx_bibblock">Attention is all you need.

</span>
<span class="ltx_bibblock">In <em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">Advances in neural information processing systems</em>, pp. 5998–6008, 2017.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Vig et al. (2020a)</span>
<span class="ltx_bibblock">
Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Sakenis, S., Huang,
J., Singer, Y., and Shieber, S.

</span>
<span class="ltx_bibblock">Causal mediation analysis for interpreting neural NLP: The case of
gender bias.

</span>
<span class="ltx_bibblock"><em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2004.12265</em>, 2020a.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Vig et al. (2020b)</span>
<span class="ltx_bibblock">
Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., and
Shieber, S. M.

</span>
<span class="ltx_bibblock">Investigating gender bias in language models using causal mediation
analysis.

</span>
<span class="ltx_bibblock">In <em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">NeurIPS</em>, 2020b.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang &amp; Komatsuzaki (2021)</span>
<span class="ltx_bibblock">
Wang, B. and Komatsuzaki, A.

</span>
<span class="ltx_bibblock">GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/kingoflolz/mesh-transformer-jax" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/kingoflolz/mesh-transformer-jax</a>, May 2021.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2018)</span>
<span class="ltx_bibblock">
Zhang, Y., Galley, M., Gao, J., Gan, Z., Li, X., Brockett, C., and Dolan, W. B.

</span>
<span class="ltx_bibblock">Generating informative and diverse conversational responses via
adversarial information maximization.

</span>
<span class="ltx_bibblock">In <em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">NeurIPS</em>, 2018.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhao et al. (2021)</span>
<span class="ltx_bibblock">
Zhao, S., Pascual, D., Brunner, G., and Wattenhofer, R.

</span>
<span class="ltx_bibblock">Of non-linearity and commutativity in BERT.

</span>
<span class="ltx_bibblock">In <em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">2021 International Joint Conference on Neural Networks
(IJCNN)</em>, pp.  1–8. IEEE, 2021.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhong et al. (2021)</span>
<span class="ltx_bibblock">
Zhong, Z., Friedman, D., and Chen, D.

</span>
<span class="ltx_bibblock">Factual probing is [MASK]: Learning vs. learning to recall.

</span>
<span class="ltx_bibblock">In <em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2021 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies</em>, pp.  5017–5033, Online, June 2021. Association for
Computational Linguistics.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2021.naacl-main.398</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/2021.naacl-main.398" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/2021.naacl-main.398</a>.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhu et al. (2020)</span>
<span class="ltx_bibblock">
Zhu, C., Rawat, A. S., Zaheer, M., Bhojanapalli, S., Li, D., Yu, F., and Kumar,
S.

</span>
<span class="ltx_bibblock">Modifying memories in transformer models.

</span>
<span class="ltx_bibblock"><em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2012.00363</em>, 2020.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Solving for <math id="A1.1.m1.1" class="ltx_Math" alttext="\Lambda" display="inline"><semantics id="A1.1.m1.1b"><mi mathvariant="normal" id="A1.1.m1.1.1" xref="A1.1.m1.1.1.cmml">Λ</mi><annotation-xml encoding="MathML-Content" id="A1.1.m1.1c"><ci id="A1.1.m1.1.1.cmml" xref="A1.1.m1.1.1">Λ</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.1.m1.1d">\Lambda</annotation></semantics></math> Algebraically</h2>

<div id="A1.p1" class="ltx_para">
<p id="A1.p1.4" class="ltx_p">Here we present the detailed derivation of Eqn. <a href="#S3.E2" title="In 3.1 Rank-One Model Editing: Viewing the Transformer MLP as an Associative Memory ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, including the linear system that is used to calculate <math id="A1.p1.1.m1.1" class="ltx_Math" alttext="\Lambda" display="inline"><semantics id="A1.p1.1.m1.1a"><mi mathvariant="normal" id="A1.p1.1.m1.1.1" xref="A1.p1.1.m1.1.1.cmml">Λ</mi><annotation-xml encoding="MathML-Content" id="A1.p1.1.m1.1b"><ci id="A1.p1.1.m1.1.1.cmml" xref="A1.p1.1.m1.1.1">Λ</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.1.m1.1c">\Lambda</annotation></semantics></math> from <math id="A1.p1.2.m2.1" class="ltx_Math" alttext="v_{*}" display="inline"><semantics id="A1.p1.2.m2.1a"><msub id="A1.p1.2.m2.1.1" xref="A1.p1.2.m2.1.1.cmml"><mi id="A1.p1.2.m2.1.1.2" xref="A1.p1.2.m2.1.1.2.cmml">v</mi><mo id="A1.p1.2.m2.1.1.3" xref="A1.p1.2.m2.1.1.3.cmml">∗</mo></msub><annotation-xml encoding="MathML-Content" id="A1.p1.2.m2.1b"><apply id="A1.p1.2.m2.1.1.cmml" xref="A1.p1.2.m2.1.1"><csymbol cd="ambiguous" id="A1.p1.2.m2.1.1.1.cmml" xref="A1.p1.2.m2.1.1">subscript</csymbol><ci id="A1.p1.2.m2.1.1.2.cmml" xref="A1.p1.2.m2.1.1.2">𝑣</ci><times id="A1.p1.2.m2.1.1.3.cmml" xref="A1.p1.2.m2.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.2.m2.1c">v_{*}</annotation></semantics></math>, <math id="A1.p1.3.m3.1" class="ltx_Math" alttext="C" display="inline"><semantics id="A1.p1.3.m3.1a"><mi id="A1.p1.3.m3.1.1" xref="A1.p1.3.m3.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="A1.p1.3.m3.1b"><ci id="A1.p1.3.m3.1.1.cmml" xref="A1.p1.3.m3.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.3.m3.1c">C</annotation></semantics></math>, and <math id="A1.p1.4.m4.1" class="ltx_Math" alttext="k_{*}" display="inline"><semantics id="A1.p1.4.m4.1a"><msub id="A1.p1.4.m4.1.1" xref="A1.p1.4.m4.1.1.cmml"><mi id="A1.p1.4.m4.1.1.2" xref="A1.p1.4.m4.1.1.2.cmml">k</mi><mo id="A1.p1.4.m4.1.1.3" xref="A1.p1.4.m4.1.1.3.cmml">∗</mo></msub><annotation-xml encoding="MathML-Content" id="A1.p1.4.m4.1b"><apply id="A1.p1.4.m4.1.1.cmml" xref="A1.p1.4.m4.1.1"><csymbol cd="ambiguous" id="A1.p1.4.m4.1.1.1.cmml" xref="A1.p1.4.m4.1.1">subscript</csymbol><ci id="A1.p1.4.m4.1.1.2.cmml" xref="A1.p1.4.m4.1.1.2">𝑘</ci><times id="A1.p1.4.m4.1.1.3.cmml" xref="A1.p1.4.m4.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.4.m4.1c">k_{*}</annotation></semantics></math>. This derivation is included for clarity and completeness and is a review of the classical solution of least-squares with equality constraints as applied to our setting, together with the rank-one update rule that was proposed in <cite class="ltx_cite ltx_citemacro_citet">Bau et al. (<a href="#bib.bib3" title="" class="ltx_ref">2020</a>)</cite>.</p>
</div>
<div id="A1.p2" class="ltx_para">
<p id="A1.p2.3" class="ltx_p">We assume that <math id="A1.p2.1.m1.1" class="ltx_Math" alttext="W" display="inline"><semantics id="A1.p2.1.m1.1a"><mi id="A1.p2.1.m1.1.1" xref="A1.p2.1.m1.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="A1.p2.1.m1.1b"><ci id="A1.p2.1.m1.1.1.cmml" xref="A1.p2.1.m1.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p2.1.m1.1c">W</annotation></semantics></math> is the optimal least-squares solution for memorizing a mapping from a previous set of keys <math id="A1.p2.2.m2.1" class="ltx_Math" alttext="K" display="inline"><semantics id="A1.p2.2.m2.1a"><mi id="A1.p2.2.m2.1.1" xref="A1.p2.2.m2.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="A1.p2.2.m2.1b"><ci id="A1.p2.2.m2.1.1.cmml" xref="A1.p2.2.m2.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p2.2.m2.1c">K</annotation></semantics></math> to values <math id="A1.p2.3.m3.1" class="ltx_Math" alttext="V" display="inline"><semantics id="A1.p2.3.m3.1a"><mi id="A1.p2.3.m3.1.1" xref="A1.p2.3.m3.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="A1.p2.3.m3.1b"><ci id="A1.p2.3.m3.1.1.cmml" xref="A1.p2.3.m3.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p2.3.m3.1c">V</annotation></semantics></math>; this solution can be written using the normal equations as follows.</p>
<table id="A10.EGx5" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A1.E5"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><span id="A1.E5.m1.1.1.1a" class="ltx_text ltx_markedasmath">the <math id="A1.E5.m1.1.1.1.m1.1" class="ltx_Math" alttext="W" display="inline"><semantics id="A1.E5.m1.1.1.1.m1.1a"><mi id="A1.E5.m1.1.1.1.m1.1.1" xref="A1.E5.m1.1.1.1.m1.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="A1.E5.m1.1.1.1.m1.1b"><ci id="A1.E5.m1.1.1.1.m1.1.1.cmml" xref="A1.E5.m1.1.1.1.m1.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.E5.m1.1.1.1.m1.1c">W</annotation></semantics></math> that minimizes</span></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A1.E5.m2.1" class="ltx_Math" alttext="\displaystyle||WK-V||_{F}^{2}" display="inline"><semantics id="A1.E5.m2.1a"><msubsup id="A1.E5.m2.1.1" xref="A1.E5.m2.1.1.cmml"><mrow id="A1.E5.m2.1.1.1.1.1" xref="A1.E5.m2.1.1.1.1.2.cmml"><mo stretchy="false" id="A1.E5.m2.1.1.1.1.1.2" xref="A1.E5.m2.1.1.1.1.2.1.cmml">‖</mo><mrow id="A1.E5.m2.1.1.1.1.1.1" xref="A1.E5.m2.1.1.1.1.1.1.cmml"><mrow id="A1.E5.m2.1.1.1.1.1.1.2" xref="A1.E5.m2.1.1.1.1.1.1.2.cmml"><mi id="A1.E5.m2.1.1.1.1.1.1.2.2" xref="A1.E5.m2.1.1.1.1.1.1.2.2.cmml">W</mi><mo lspace="0em" rspace="0em" id="A1.E5.m2.1.1.1.1.1.1.2.1" xref="A1.E5.m2.1.1.1.1.1.1.2.1.cmml">​</mo><mi id="A1.E5.m2.1.1.1.1.1.1.2.3" xref="A1.E5.m2.1.1.1.1.1.1.2.3.cmml">K</mi></mrow><mo id="A1.E5.m2.1.1.1.1.1.1.1" xref="A1.E5.m2.1.1.1.1.1.1.1.cmml">−</mo><mi id="A1.E5.m2.1.1.1.1.1.1.3" xref="A1.E5.m2.1.1.1.1.1.1.3.cmml">V</mi></mrow><mo stretchy="false" id="A1.E5.m2.1.1.1.1.1.3" xref="A1.E5.m2.1.1.1.1.2.1.cmml">‖</mo></mrow><mi id="A1.E5.m2.1.1.1.3" xref="A1.E5.m2.1.1.1.3.cmml">F</mi><mn id="A1.E5.m2.1.1.3" xref="A1.E5.m2.1.1.3.cmml">2</mn></msubsup><annotation-xml encoding="MathML-Content" id="A1.E5.m2.1b"><apply id="A1.E5.m2.1.1.cmml" xref="A1.E5.m2.1.1"><csymbol cd="ambiguous" id="A1.E5.m2.1.1.2.cmml" xref="A1.E5.m2.1.1">superscript</csymbol><apply id="A1.E5.m2.1.1.1.cmml" xref="A1.E5.m2.1.1"><csymbol cd="ambiguous" id="A1.E5.m2.1.1.1.2.cmml" xref="A1.E5.m2.1.1">subscript</csymbol><apply id="A1.E5.m2.1.1.1.1.2.cmml" xref="A1.E5.m2.1.1.1.1.1"><csymbol cd="latexml" id="A1.E5.m2.1.1.1.1.2.1.cmml" xref="A1.E5.m2.1.1.1.1.1.2">norm</csymbol><apply id="A1.E5.m2.1.1.1.1.1.1.cmml" xref="A1.E5.m2.1.1.1.1.1.1"><minus id="A1.E5.m2.1.1.1.1.1.1.1.cmml" xref="A1.E5.m2.1.1.1.1.1.1.1"></minus><apply id="A1.E5.m2.1.1.1.1.1.1.2.cmml" xref="A1.E5.m2.1.1.1.1.1.1.2"><times id="A1.E5.m2.1.1.1.1.1.1.2.1.cmml" xref="A1.E5.m2.1.1.1.1.1.1.2.1"></times><ci id="A1.E5.m2.1.1.1.1.1.1.2.2.cmml" xref="A1.E5.m2.1.1.1.1.1.1.2.2">𝑊</ci><ci id="A1.E5.m2.1.1.1.1.1.1.2.3.cmml" xref="A1.E5.m2.1.1.1.1.1.1.2.3">𝐾</ci></apply><ci id="A1.E5.m2.1.1.1.1.1.1.3.cmml" xref="A1.E5.m2.1.1.1.1.1.1.3">𝑉</ci></apply></apply><ci id="A1.E5.m2.1.1.1.3.cmml" xref="A1.E5.m2.1.1.1.3">𝐹</ci></apply><cn type="integer" id="A1.E5.m2.1.1.3.cmml" xref="A1.E5.m2.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E5.m2.1c">\displaystyle||WK-V||_{F}^{2}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(5)</span></td>
</tr></tbody>
<tbody id="A1.E6"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><span id="A1.E6.2.1.1.1" class="ltx_text ltx_markedasmath">solves</span></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A1.E6.m2.1" class="ltx_Math" alttext="\displaystyle WKK^{T}=VK^{T}" display="inline"><semantics id="A1.E6.m2.1a"><mrow id="A1.E6.m2.1.1" xref="A1.E6.m2.1.1.cmml"><mrow id="A1.E6.m2.1.1.2" xref="A1.E6.m2.1.1.2.cmml"><mi id="A1.E6.m2.1.1.2.2" xref="A1.E6.m2.1.1.2.2.cmml">W</mi><mo lspace="0em" rspace="0em" id="A1.E6.m2.1.1.2.1" xref="A1.E6.m2.1.1.2.1.cmml">​</mo><mi id="A1.E6.m2.1.1.2.3" xref="A1.E6.m2.1.1.2.3.cmml">K</mi><mo lspace="0em" rspace="0em" id="A1.E6.m2.1.1.2.1a" xref="A1.E6.m2.1.1.2.1.cmml">​</mo><msup id="A1.E6.m2.1.1.2.4" xref="A1.E6.m2.1.1.2.4.cmml"><mi id="A1.E6.m2.1.1.2.4.2" xref="A1.E6.m2.1.1.2.4.2.cmml">K</mi><mi id="A1.E6.m2.1.1.2.4.3" xref="A1.E6.m2.1.1.2.4.3.cmml">T</mi></msup></mrow><mo id="A1.E6.m2.1.1.1" xref="A1.E6.m2.1.1.1.cmml">=</mo><mrow id="A1.E6.m2.1.1.3" xref="A1.E6.m2.1.1.3.cmml"><mi id="A1.E6.m2.1.1.3.2" xref="A1.E6.m2.1.1.3.2.cmml">V</mi><mo lspace="0em" rspace="0em" id="A1.E6.m2.1.1.3.1" xref="A1.E6.m2.1.1.3.1.cmml">​</mo><msup id="A1.E6.m2.1.1.3.3" xref="A1.E6.m2.1.1.3.3.cmml"><mi id="A1.E6.m2.1.1.3.3.2" xref="A1.E6.m2.1.1.3.3.2.cmml">K</mi><mi id="A1.E6.m2.1.1.3.3.3" xref="A1.E6.m2.1.1.3.3.3.cmml">T</mi></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E6.m2.1b"><apply id="A1.E6.m2.1.1.cmml" xref="A1.E6.m2.1.1"><eq id="A1.E6.m2.1.1.1.cmml" xref="A1.E6.m2.1.1.1"></eq><apply id="A1.E6.m2.1.1.2.cmml" xref="A1.E6.m2.1.1.2"><times id="A1.E6.m2.1.1.2.1.cmml" xref="A1.E6.m2.1.1.2.1"></times><ci id="A1.E6.m2.1.1.2.2.cmml" xref="A1.E6.m2.1.1.2.2">𝑊</ci><ci id="A1.E6.m2.1.1.2.3.cmml" xref="A1.E6.m2.1.1.2.3">𝐾</ci><apply id="A1.E6.m2.1.1.2.4.cmml" xref="A1.E6.m2.1.1.2.4"><csymbol cd="ambiguous" id="A1.E6.m2.1.1.2.4.1.cmml" xref="A1.E6.m2.1.1.2.4">superscript</csymbol><ci id="A1.E6.m2.1.1.2.4.2.cmml" xref="A1.E6.m2.1.1.2.4.2">𝐾</ci><ci id="A1.E6.m2.1.1.2.4.3.cmml" xref="A1.E6.m2.1.1.2.4.3">𝑇</ci></apply></apply><apply id="A1.E6.m2.1.1.3.cmml" xref="A1.E6.m2.1.1.3"><times id="A1.E6.m2.1.1.3.1.cmml" xref="A1.E6.m2.1.1.3.1"></times><ci id="A1.E6.m2.1.1.3.2.cmml" xref="A1.E6.m2.1.1.3.2">𝑉</ci><apply id="A1.E6.m2.1.1.3.3.cmml" xref="A1.E6.m2.1.1.3.3"><csymbol cd="ambiguous" id="A1.E6.m2.1.1.3.3.1.cmml" xref="A1.E6.m2.1.1.3.3">superscript</csymbol><ci id="A1.E6.m2.1.1.3.3.2.cmml" xref="A1.E6.m2.1.1.3.3.2">𝐾</ci><ci id="A1.E6.m2.1.1.3.3.3.cmml" xref="A1.E6.m2.1.1.3.3.3">𝑇</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E6.m2.1c">\displaystyle WKK^{T}=VK^{T}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(6)</span></td>
</tr></tbody>
</table>
<p id="A1.p2.5" class="ltx_p">Here the Frobenius norm is used to write the total square error since the variable being optimized is a matrix <math id="A1.p2.4.m1.1" class="ltx_Math" alttext="W" display="inline"><semantics id="A1.p2.4.m1.1a"><mi id="A1.p2.4.m1.1.1" xref="A1.p2.4.m1.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="A1.p2.4.m1.1b"><ci id="A1.p2.4.m1.1.1.cmml" xref="A1.p2.4.m1.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p2.4.m1.1c">W</annotation></semantics></math> rather than a vector <math id="A1.p2.5.m2.1" class="ltx_Math" alttext="x" display="inline"><semantics id="A1.p2.5.m2.1a"><mi id="A1.p2.5.m2.1.1" xref="A1.p2.5.m2.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="A1.p2.5.m2.1b"><ci id="A1.p2.5.m2.1.1.cmml" xref="A1.p2.5.m2.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p2.5.m2.1c">x</annotation></semantics></math> as in the classical textbook presentation of least squares.</p>
</div>
<div id="A1.p3" class="ltx_para">
<p id="A1.p3.1" class="ltx_p">We wish to find a new matrix <math id="A1.p3.1.m1.1" class="ltx_Math" alttext="\hat{W}" display="inline"><semantics id="A1.p3.1.m1.1a"><mover accent="true" id="A1.p3.1.m1.1.1" xref="A1.p3.1.m1.1.1.cmml"><mi id="A1.p3.1.m1.1.1.2" xref="A1.p3.1.m1.1.1.2.cmml">W</mi><mo id="A1.p3.1.m1.1.1.1" xref="A1.p3.1.m1.1.1.1.cmml">^</mo></mover><annotation-xml encoding="MathML-Content" id="A1.p3.1.m1.1b"><apply id="A1.p3.1.m1.1.1.cmml" xref="A1.p3.1.m1.1.1"><ci id="A1.p3.1.m1.1.1.1.cmml" xref="A1.p3.1.m1.1.1.1">^</ci><ci id="A1.p3.1.m1.1.1.2.cmml" xref="A1.p3.1.m1.1.1.2">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p3.1.m1.1c">\hat{W}</annotation></semantics></math> that solves the same least squares problem with an additional equality constraint as written in Eqn. <a href="#S3.E2" title="In 3.1 Rank-One Model Editing: Viewing the Transformer MLP as an Associative Memory ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>:</p>
<table id="A10.EGx6" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A1.E7"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A1.E7.m1.1" class="ltx_Math" alttext="\displaystyle\hat{W}k_{*}=v_{*}" display="inline"><semantics id="A1.E7.m1.1a"><mrow id="A1.E7.m1.1.1" xref="A1.E7.m1.1.1.cmml"><mrow id="A1.E7.m1.1.1.2" xref="A1.E7.m1.1.1.2.cmml"><mover accent="true" id="A1.E7.m1.1.1.2.2" xref="A1.E7.m1.1.1.2.2.cmml"><mi id="A1.E7.m1.1.1.2.2.2" xref="A1.E7.m1.1.1.2.2.2.cmml">W</mi><mo id="A1.E7.m1.1.1.2.2.1" xref="A1.E7.m1.1.1.2.2.1.cmml">^</mo></mover><mo lspace="0em" rspace="0em" id="A1.E7.m1.1.1.2.1" xref="A1.E7.m1.1.1.2.1.cmml">​</mo><msub id="A1.E7.m1.1.1.2.3" xref="A1.E7.m1.1.1.2.3.cmml"><mi id="A1.E7.m1.1.1.2.3.2" xref="A1.E7.m1.1.1.2.3.2.cmml">k</mi><mo id="A1.E7.m1.1.1.2.3.3" xref="A1.E7.m1.1.1.2.3.3.cmml">∗</mo></msub></mrow><mo id="A1.E7.m1.1.1.1" xref="A1.E7.m1.1.1.1.cmml">=</mo><msub id="A1.E7.m1.1.1.3" xref="A1.E7.m1.1.1.3.cmml"><mi id="A1.E7.m1.1.1.3.2" xref="A1.E7.m1.1.1.3.2.cmml">v</mi><mo id="A1.E7.m1.1.1.3.3" xref="A1.E7.m1.1.1.3.3.cmml">∗</mo></msub></mrow><annotation-xml encoding="MathML-Content" id="A1.E7.m1.1b"><apply id="A1.E7.m1.1.1.cmml" xref="A1.E7.m1.1.1"><eq id="A1.E7.m1.1.1.1.cmml" xref="A1.E7.m1.1.1.1"></eq><apply id="A1.E7.m1.1.1.2.cmml" xref="A1.E7.m1.1.1.2"><times id="A1.E7.m1.1.1.2.1.cmml" xref="A1.E7.m1.1.1.2.1"></times><apply id="A1.E7.m1.1.1.2.2.cmml" xref="A1.E7.m1.1.1.2.2"><ci id="A1.E7.m1.1.1.2.2.1.cmml" xref="A1.E7.m1.1.1.2.2.1">^</ci><ci id="A1.E7.m1.1.1.2.2.2.cmml" xref="A1.E7.m1.1.1.2.2.2">𝑊</ci></apply><apply id="A1.E7.m1.1.1.2.3.cmml" xref="A1.E7.m1.1.1.2.3"><csymbol cd="ambiguous" id="A1.E7.m1.1.1.2.3.1.cmml" xref="A1.E7.m1.1.1.2.3">subscript</csymbol><ci id="A1.E7.m1.1.1.2.3.2.cmml" xref="A1.E7.m1.1.1.2.3.2">𝑘</ci><times id="A1.E7.m1.1.1.2.3.3.cmml" xref="A1.E7.m1.1.1.2.3.3"></times></apply></apply><apply id="A1.E7.m1.1.1.3.cmml" xref="A1.E7.m1.1.1.3"><csymbol cd="ambiguous" id="A1.E7.m1.1.1.3.1.cmml" xref="A1.E7.m1.1.1.3">subscript</csymbol><ci id="A1.E7.m1.1.1.3.2.cmml" xref="A1.E7.m1.1.1.3.2">𝑣</ci><times id="A1.E7.m1.1.1.3.3.cmml" xref="A1.E7.m1.1.1.3.3"></times></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E7.m1.1c">\displaystyle\hat{W}k_{*}=v_{*}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(7)</span></td>
</tr></tbody>
</table>
</div>
<div id="A1.p4" class="ltx_para">
<p id="A1.p4.1" class="ltx_p">This is the well-studied problem of least squares with a linear equality constraint. The direct solution can be derived by defining and minimizing a Lagrangian, where <math id="A1.p4.1.m1.1" class="ltx_Math" alttext="\Lambda\in\mathbb{R}^{H}" display="inline"><semantics id="A1.p4.1.m1.1a"><mrow id="A1.p4.1.m1.1.1" xref="A1.p4.1.m1.1.1.cmml"><mi mathvariant="normal" id="A1.p4.1.m1.1.1.2" xref="A1.p4.1.m1.1.1.2.cmml">Λ</mi><mo id="A1.p4.1.m1.1.1.1" xref="A1.p4.1.m1.1.1.1.cmml">∈</mo><msup id="A1.p4.1.m1.1.1.3" xref="A1.p4.1.m1.1.1.3.cmml"><mi id="A1.p4.1.m1.1.1.3.2" xref="A1.p4.1.m1.1.1.3.2.cmml">ℝ</mi><mi id="A1.p4.1.m1.1.1.3.3" xref="A1.p4.1.m1.1.1.3.3.cmml">H</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="A1.p4.1.m1.1b"><apply id="A1.p4.1.m1.1.1.cmml" xref="A1.p4.1.m1.1.1"><in id="A1.p4.1.m1.1.1.1.cmml" xref="A1.p4.1.m1.1.1.1"></in><ci id="A1.p4.1.m1.1.1.2.cmml" xref="A1.p4.1.m1.1.1.2">Λ</ci><apply id="A1.p4.1.m1.1.1.3.cmml" xref="A1.p4.1.m1.1.1.3"><csymbol cd="ambiguous" id="A1.p4.1.m1.1.1.3.1.cmml" xref="A1.p4.1.m1.1.1.3">superscript</csymbol><ci id="A1.p4.1.m1.1.1.3.2.cmml" xref="A1.p4.1.m1.1.1.3.2">ℝ</ci><ci id="A1.p4.1.m1.1.1.3.3.cmml" xref="A1.p4.1.m1.1.1.3.3">𝐻</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p4.1.m1.1c">\Lambda\in\mathbb{R}^{H}</annotation></semantics></math> minimizes the following:</p>
<table id="A10.EGx7" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A1.E8"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A1.E8.m1.4" class="ltx_Math" alttext="\displaystyle\text{define}\quad L(\hat{W},\Lambda)" display="inline"><semantics id="A1.E8.m1.4a"><mrow id="A1.E8.m1.4.4.1" xref="A1.E8.m1.4.4.2.cmml"><mtext id="A1.E8.m1.3.3" xref="A1.E8.m1.3.3a.cmml">define</mtext><mspace width="1em" id="A1.E8.m1.4.4.1.2" xref="A1.E8.m1.4.4.2.cmml"></mspace><mrow id="A1.E8.m1.4.4.1.1" xref="A1.E8.m1.4.4.1.1.cmml"><mi id="A1.E8.m1.4.4.1.1.2" xref="A1.E8.m1.4.4.1.1.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="A1.E8.m1.4.4.1.1.1" xref="A1.E8.m1.4.4.1.1.1.cmml">​</mo><mrow id="A1.E8.m1.4.4.1.1.3.2" xref="A1.E8.m1.4.4.1.1.3.1.cmml"><mo stretchy="false" id="A1.E8.m1.4.4.1.1.3.2.1" xref="A1.E8.m1.4.4.1.1.3.1.cmml">(</mo><mover accent="true" id="A1.E8.m1.1.1" xref="A1.E8.m1.1.1.cmml"><mi id="A1.E8.m1.1.1.2" xref="A1.E8.m1.1.1.2.cmml">W</mi><mo id="A1.E8.m1.1.1.1" xref="A1.E8.m1.1.1.1.cmml">^</mo></mover><mo id="A1.E8.m1.4.4.1.1.3.2.2" xref="A1.E8.m1.4.4.1.1.3.1.cmml">,</mo><mi mathvariant="normal" id="A1.E8.m1.2.2" xref="A1.E8.m1.2.2.cmml">Λ</mi><mo stretchy="false" id="A1.E8.m1.4.4.1.1.3.2.3" xref="A1.E8.m1.4.4.1.1.3.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E8.m1.4b"><list id="A1.E8.m1.4.4.2.cmml" xref="A1.E8.m1.4.4.1"><ci id="A1.E8.m1.3.3a.cmml" xref="A1.E8.m1.3.3"><mtext id="A1.E8.m1.3.3.cmml" xref="A1.E8.m1.3.3">define</mtext></ci><apply id="A1.E8.m1.4.4.1.1.cmml" xref="A1.E8.m1.4.4.1.1"><times id="A1.E8.m1.4.4.1.1.1.cmml" xref="A1.E8.m1.4.4.1.1.1"></times><ci id="A1.E8.m1.4.4.1.1.2.cmml" xref="A1.E8.m1.4.4.1.1.2">𝐿</ci><interval closure="open" id="A1.E8.m1.4.4.1.1.3.1.cmml" xref="A1.E8.m1.4.4.1.1.3.2"><apply id="A1.E8.m1.1.1.cmml" xref="A1.E8.m1.1.1"><ci id="A1.E8.m1.1.1.1.cmml" xref="A1.E8.m1.1.1.1">^</ci><ci id="A1.E8.m1.1.1.2.cmml" xref="A1.E8.m1.1.1.2">𝑊</ci></apply><ci id="A1.E8.m1.2.2.cmml" xref="A1.E8.m1.2.2">Λ</ci></interval></apply></list></annotation-xml><annotation encoding="application/x-tex" id="A1.E8.m1.4c">\displaystyle\text{define}\quad L(\hat{W},\Lambda)</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A1.E8.m2.2" class="ltx_Math" alttext="\displaystyle=\frac{1}{2}||\hat{W}K-V||_{F}^{2}-\Lambda^{T}(\hat{W}k_{*}-v_{*})" display="inline"><semantics id="A1.E8.m2.2a"><mrow id="A1.E8.m2.2.2" xref="A1.E8.m2.2.2.cmml"><mi id="A1.E8.m2.2.2.4" xref="A1.E8.m2.2.2.4.cmml"></mi><mo id="A1.E8.m2.2.2.3" xref="A1.E8.m2.2.2.3.cmml">=</mo><mrow id="A1.E8.m2.2.2.2" xref="A1.E8.m2.2.2.2.cmml"><mrow id="A1.E8.m2.1.1.1.1" xref="A1.E8.m2.1.1.1.1.cmml"><mstyle displaystyle="true" id="A1.E8.m2.1.1.1.1.3" xref="A1.E8.m2.1.1.1.1.3.cmml"><mfrac id="A1.E8.m2.1.1.1.1.3a" xref="A1.E8.m2.1.1.1.1.3.cmml"><mn id="A1.E8.m2.1.1.1.1.3.2" xref="A1.E8.m2.1.1.1.1.3.2.cmml">1</mn><mn id="A1.E8.m2.1.1.1.1.3.3" xref="A1.E8.m2.1.1.1.1.3.3.cmml">2</mn></mfrac></mstyle><mo lspace="0em" rspace="0em" id="A1.E8.m2.1.1.1.1.2" xref="A1.E8.m2.1.1.1.1.2.cmml">​</mo><msubsup id="A1.E8.m2.1.1.1.1.1" xref="A1.E8.m2.1.1.1.1.1.cmml"><mrow id="A1.E8.m2.1.1.1.1.1.1.1.1" xref="A1.E8.m2.1.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="A1.E8.m2.1.1.1.1.1.1.1.1.2" xref="A1.E8.m2.1.1.1.1.1.1.1.2.1.cmml">‖</mo><mrow id="A1.E8.m2.1.1.1.1.1.1.1.1.1" xref="A1.E8.m2.1.1.1.1.1.1.1.1.1.cmml"><mrow id="A1.E8.m2.1.1.1.1.1.1.1.1.1.2" xref="A1.E8.m2.1.1.1.1.1.1.1.1.1.2.cmml"><mover accent="true" id="A1.E8.m2.1.1.1.1.1.1.1.1.1.2.2" xref="A1.E8.m2.1.1.1.1.1.1.1.1.1.2.2.cmml"><mi id="A1.E8.m2.1.1.1.1.1.1.1.1.1.2.2.2" xref="A1.E8.m2.1.1.1.1.1.1.1.1.1.2.2.2.cmml">W</mi><mo id="A1.E8.m2.1.1.1.1.1.1.1.1.1.2.2.1" xref="A1.E8.m2.1.1.1.1.1.1.1.1.1.2.2.1.cmml">^</mo></mover><mo lspace="0em" rspace="0em" id="A1.E8.m2.1.1.1.1.1.1.1.1.1.2.1" xref="A1.E8.m2.1.1.1.1.1.1.1.1.1.2.1.cmml">​</mo><mi id="A1.E8.m2.1.1.1.1.1.1.1.1.1.2.3" xref="A1.E8.m2.1.1.1.1.1.1.1.1.1.2.3.cmml">K</mi></mrow><mo id="A1.E8.m2.1.1.1.1.1.1.1.1.1.1" xref="A1.E8.m2.1.1.1.1.1.1.1.1.1.1.cmml">−</mo><mi id="A1.E8.m2.1.1.1.1.1.1.1.1.1.3" xref="A1.E8.m2.1.1.1.1.1.1.1.1.1.3.cmml">V</mi></mrow><mo stretchy="false" id="A1.E8.m2.1.1.1.1.1.1.1.1.3" xref="A1.E8.m2.1.1.1.1.1.1.1.2.1.cmml">‖</mo></mrow><mi id="A1.E8.m2.1.1.1.1.1.1.3" xref="A1.E8.m2.1.1.1.1.1.1.3.cmml">F</mi><mn id="A1.E8.m2.1.1.1.1.1.3" xref="A1.E8.m2.1.1.1.1.1.3.cmml">2</mn></msubsup></mrow><mo id="A1.E8.m2.2.2.2.3" xref="A1.E8.m2.2.2.2.3.cmml">−</mo><mrow id="A1.E8.m2.2.2.2.2" xref="A1.E8.m2.2.2.2.2.cmml"><msup id="A1.E8.m2.2.2.2.2.3" xref="A1.E8.m2.2.2.2.2.3.cmml"><mi mathvariant="normal" id="A1.E8.m2.2.2.2.2.3.2" xref="A1.E8.m2.2.2.2.2.3.2.cmml">Λ</mi><mi id="A1.E8.m2.2.2.2.2.3.3" xref="A1.E8.m2.2.2.2.2.3.3.cmml">T</mi></msup><mo lspace="0em" rspace="0em" id="A1.E8.m2.2.2.2.2.2" xref="A1.E8.m2.2.2.2.2.2.cmml">​</mo><mrow id="A1.E8.m2.2.2.2.2.1.1" xref="A1.E8.m2.2.2.2.2.1.1.1.cmml"><mo stretchy="false" id="A1.E8.m2.2.2.2.2.1.1.2" xref="A1.E8.m2.2.2.2.2.1.1.1.cmml">(</mo><mrow id="A1.E8.m2.2.2.2.2.1.1.1" xref="A1.E8.m2.2.2.2.2.1.1.1.cmml"><mrow id="A1.E8.m2.2.2.2.2.1.1.1.2" xref="A1.E8.m2.2.2.2.2.1.1.1.2.cmml"><mover accent="true" id="A1.E8.m2.2.2.2.2.1.1.1.2.2" xref="A1.E8.m2.2.2.2.2.1.1.1.2.2.cmml"><mi id="A1.E8.m2.2.2.2.2.1.1.1.2.2.2" xref="A1.E8.m2.2.2.2.2.1.1.1.2.2.2.cmml">W</mi><mo id="A1.E8.m2.2.2.2.2.1.1.1.2.2.1" xref="A1.E8.m2.2.2.2.2.1.1.1.2.2.1.cmml">^</mo></mover><mo lspace="0em" rspace="0em" id="A1.E8.m2.2.2.2.2.1.1.1.2.1" xref="A1.E8.m2.2.2.2.2.1.1.1.2.1.cmml">​</mo><msub id="A1.E8.m2.2.2.2.2.1.1.1.2.3" xref="A1.E8.m2.2.2.2.2.1.1.1.2.3.cmml"><mi id="A1.E8.m2.2.2.2.2.1.1.1.2.3.2" xref="A1.E8.m2.2.2.2.2.1.1.1.2.3.2.cmml">k</mi><mo id="A1.E8.m2.2.2.2.2.1.1.1.2.3.3" xref="A1.E8.m2.2.2.2.2.1.1.1.2.3.3.cmml">∗</mo></msub></mrow><mo id="A1.E8.m2.2.2.2.2.1.1.1.1" xref="A1.E8.m2.2.2.2.2.1.1.1.1.cmml">−</mo><msub id="A1.E8.m2.2.2.2.2.1.1.1.3" xref="A1.E8.m2.2.2.2.2.1.1.1.3.cmml"><mi id="A1.E8.m2.2.2.2.2.1.1.1.3.2" xref="A1.E8.m2.2.2.2.2.1.1.1.3.2.cmml">v</mi><mo id="A1.E8.m2.2.2.2.2.1.1.1.3.3" xref="A1.E8.m2.2.2.2.2.1.1.1.3.3.cmml">∗</mo></msub></mrow><mo stretchy="false" id="A1.E8.m2.2.2.2.2.1.1.3" xref="A1.E8.m2.2.2.2.2.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E8.m2.2b"><apply id="A1.E8.m2.2.2.cmml" xref="A1.E8.m2.2.2"><eq id="A1.E8.m2.2.2.3.cmml" xref="A1.E8.m2.2.2.3"></eq><csymbol cd="latexml" id="A1.E8.m2.2.2.4.cmml" xref="A1.E8.m2.2.2.4">absent</csymbol><apply id="A1.E8.m2.2.2.2.cmml" xref="A1.E8.m2.2.2.2"><minus id="A1.E8.m2.2.2.2.3.cmml" xref="A1.E8.m2.2.2.2.3"></minus><apply id="A1.E8.m2.1.1.1.1.cmml" xref="A1.E8.m2.1.1.1.1"><times id="A1.E8.m2.1.1.1.1.2.cmml" xref="A1.E8.m2.1.1.1.1.2"></times><apply id="A1.E8.m2.1.1.1.1.3.cmml" xref="A1.E8.m2.1.1.1.1.3"><divide id="A1.E8.m2.1.1.1.1.3.1.cmml" xref="A1.E8.m2.1.1.1.1.3"></divide><cn type="integer" id="A1.E8.m2.1.1.1.1.3.2.cmml" xref="A1.E8.m2.1.1.1.1.3.2">1</cn><cn type="integer" id="A1.E8.m2.1.1.1.1.3.3.cmml" xref="A1.E8.m2.1.1.1.1.3.3">2</cn></apply><apply id="A1.E8.m2.1.1.1.1.1.cmml" xref="A1.E8.m2.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.E8.m2.1.1.1.1.1.2.cmml" xref="A1.E8.m2.1.1.1.1.1">superscript</csymbol><apply id="A1.E8.m2.1.1.1.1.1.1.cmml" xref="A1.E8.m2.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.E8.m2.1.1.1.1.1.1.2.cmml" xref="A1.E8.m2.1.1.1.1.1">subscript</csymbol><apply id="A1.E8.m2.1.1.1.1.1.1.1.2.cmml" xref="A1.E8.m2.1.1.1.1.1.1.1.1"><csymbol cd="latexml" id="A1.E8.m2.1.1.1.1.1.1.1.2.1.cmml" xref="A1.E8.m2.1.1.1.1.1.1.1.1.2">norm</csymbol><apply id="A1.E8.m2.1.1.1.1.1.1.1.1.1.cmml" xref="A1.E8.m2.1.1.1.1.1.1.1.1.1"><minus id="A1.E8.m2.1.1.1.1.1.1.1.1.1.1.cmml" xref="A1.E8.m2.1.1.1.1.1.1.1.1.1.1"></minus><apply id="A1.E8.m2.1.1.1.1.1.1.1.1.1.2.cmml" xref="A1.E8.m2.1.1.1.1.1.1.1.1.1.2"><times id="A1.E8.m2.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="A1.E8.m2.1.1.1.1.1.1.1.1.1.2.1"></times><apply id="A1.E8.m2.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="A1.E8.m2.1.1.1.1.1.1.1.1.1.2.2"><ci id="A1.E8.m2.1.1.1.1.1.1.1.1.1.2.2.1.cmml" xref="A1.E8.m2.1.1.1.1.1.1.1.1.1.2.2.1">^</ci><ci id="A1.E8.m2.1.1.1.1.1.1.1.1.1.2.2.2.cmml" xref="A1.E8.m2.1.1.1.1.1.1.1.1.1.2.2.2">𝑊</ci></apply><ci id="A1.E8.m2.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="A1.E8.m2.1.1.1.1.1.1.1.1.1.2.3">𝐾</ci></apply><ci id="A1.E8.m2.1.1.1.1.1.1.1.1.1.3.cmml" xref="A1.E8.m2.1.1.1.1.1.1.1.1.1.3">𝑉</ci></apply></apply><ci id="A1.E8.m2.1.1.1.1.1.1.3.cmml" xref="A1.E8.m2.1.1.1.1.1.1.3">𝐹</ci></apply><cn type="integer" id="A1.E8.m2.1.1.1.1.1.3.cmml" xref="A1.E8.m2.1.1.1.1.1.3">2</cn></apply></apply><apply id="A1.E8.m2.2.2.2.2.cmml" xref="A1.E8.m2.2.2.2.2"><times id="A1.E8.m2.2.2.2.2.2.cmml" xref="A1.E8.m2.2.2.2.2.2"></times><apply id="A1.E8.m2.2.2.2.2.3.cmml" xref="A1.E8.m2.2.2.2.2.3"><csymbol cd="ambiguous" id="A1.E8.m2.2.2.2.2.3.1.cmml" xref="A1.E8.m2.2.2.2.2.3">superscript</csymbol><ci id="A1.E8.m2.2.2.2.2.3.2.cmml" xref="A1.E8.m2.2.2.2.2.3.2">Λ</ci><ci id="A1.E8.m2.2.2.2.2.3.3.cmml" xref="A1.E8.m2.2.2.2.2.3.3">𝑇</ci></apply><apply id="A1.E8.m2.2.2.2.2.1.1.1.cmml" xref="A1.E8.m2.2.2.2.2.1.1"><minus id="A1.E8.m2.2.2.2.2.1.1.1.1.cmml" xref="A1.E8.m2.2.2.2.2.1.1.1.1"></minus><apply id="A1.E8.m2.2.2.2.2.1.1.1.2.cmml" xref="A1.E8.m2.2.2.2.2.1.1.1.2"><times id="A1.E8.m2.2.2.2.2.1.1.1.2.1.cmml" xref="A1.E8.m2.2.2.2.2.1.1.1.2.1"></times><apply id="A1.E8.m2.2.2.2.2.1.1.1.2.2.cmml" xref="A1.E8.m2.2.2.2.2.1.1.1.2.2"><ci id="A1.E8.m2.2.2.2.2.1.1.1.2.2.1.cmml" xref="A1.E8.m2.2.2.2.2.1.1.1.2.2.1">^</ci><ci id="A1.E8.m2.2.2.2.2.1.1.1.2.2.2.cmml" xref="A1.E8.m2.2.2.2.2.1.1.1.2.2.2">𝑊</ci></apply><apply id="A1.E8.m2.2.2.2.2.1.1.1.2.3.cmml" xref="A1.E8.m2.2.2.2.2.1.1.1.2.3"><csymbol cd="ambiguous" id="A1.E8.m2.2.2.2.2.1.1.1.2.3.1.cmml" xref="A1.E8.m2.2.2.2.2.1.1.1.2.3">subscript</csymbol><ci id="A1.E8.m2.2.2.2.2.1.1.1.2.3.2.cmml" xref="A1.E8.m2.2.2.2.2.1.1.1.2.3.2">𝑘</ci><times id="A1.E8.m2.2.2.2.2.1.1.1.2.3.3.cmml" xref="A1.E8.m2.2.2.2.2.1.1.1.2.3.3"></times></apply></apply><apply id="A1.E8.m2.2.2.2.2.1.1.1.3.cmml" xref="A1.E8.m2.2.2.2.2.1.1.1.3"><csymbol cd="ambiguous" id="A1.E8.m2.2.2.2.2.1.1.1.3.1.cmml" xref="A1.E8.m2.2.2.2.2.1.1.1.3">subscript</csymbol><ci id="A1.E8.m2.2.2.2.2.1.1.1.3.2.cmml" xref="A1.E8.m2.2.2.2.2.1.1.1.3.2">𝑣</ci><times id="A1.E8.m2.2.2.2.2.1.1.1.3.3.cmml" xref="A1.E8.m2.2.2.2.2.1.1.1.3.3"></times></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E8.m2.2c">\displaystyle=\frac{1}{2}||\hat{W}K-V||_{F}^{2}-\Lambda^{T}(\hat{W}k_{*}-v_{*})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(8)</span></td>
</tr></tbody>
<tbody id="A1.E9"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A1.E9.m1.4" class="ltx_Math" alttext="\displaystyle=\frac{1}{2}(\hat{W}K)(\hat{W}K)^{T}-V(\hat{W}K)^{T}+\frac{1}{2}VV^{T}-\Lambda^{T}(\hat{W}k_{*}-v_{*})" display="inline"><semantics id="A1.E9.m1.4a"><mrow id="A1.E9.m1.4.4" xref="A1.E9.m1.4.4.cmml"><mi id="A1.E9.m1.4.4.6" xref="A1.E9.m1.4.4.6.cmml"></mi><mo id="A1.E9.m1.4.4.5" xref="A1.E9.m1.4.4.5.cmml">=</mo><mrow id="A1.E9.m1.4.4.4" xref="A1.E9.m1.4.4.4.cmml"><mrow id="A1.E9.m1.3.3.3.3" xref="A1.E9.m1.3.3.3.3.cmml"><mrow id="A1.E9.m1.3.3.3.3.3" xref="A1.E9.m1.3.3.3.3.3.cmml"><mrow id="A1.E9.m1.2.2.2.2.2.2" xref="A1.E9.m1.2.2.2.2.2.2.cmml"><mstyle displaystyle="true" id="A1.E9.m1.2.2.2.2.2.2.4" xref="A1.E9.m1.2.2.2.2.2.2.4.cmml"><mfrac id="A1.E9.m1.2.2.2.2.2.2.4a" xref="A1.E9.m1.2.2.2.2.2.2.4.cmml"><mn id="A1.E9.m1.2.2.2.2.2.2.4.2" xref="A1.E9.m1.2.2.2.2.2.2.4.2.cmml">1</mn><mn id="A1.E9.m1.2.2.2.2.2.2.4.3" xref="A1.E9.m1.2.2.2.2.2.2.4.3.cmml">2</mn></mfrac></mstyle><mo lspace="0em" rspace="0em" id="A1.E9.m1.2.2.2.2.2.2.3" xref="A1.E9.m1.2.2.2.2.2.2.3.cmml">​</mo><mrow id="A1.E9.m1.1.1.1.1.1.1.1.1" xref="A1.E9.m1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.E9.m1.1.1.1.1.1.1.1.1.2" xref="A1.E9.m1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A1.E9.m1.1.1.1.1.1.1.1.1.1" xref="A1.E9.m1.1.1.1.1.1.1.1.1.1.cmml"><mover accent="true" id="A1.E9.m1.1.1.1.1.1.1.1.1.1.2" xref="A1.E9.m1.1.1.1.1.1.1.1.1.1.2.cmml"><mi id="A1.E9.m1.1.1.1.1.1.1.1.1.1.2.2" xref="A1.E9.m1.1.1.1.1.1.1.1.1.1.2.2.cmml">W</mi><mo id="A1.E9.m1.1.1.1.1.1.1.1.1.1.2.1" xref="A1.E9.m1.1.1.1.1.1.1.1.1.1.2.1.cmml">^</mo></mover><mo lspace="0em" rspace="0em" id="A1.E9.m1.1.1.1.1.1.1.1.1.1.1" xref="A1.E9.m1.1.1.1.1.1.1.1.1.1.1.cmml">​</mo><mi id="A1.E9.m1.1.1.1.1.1.1.1.1.1.3" xref="A1.E9.m1.1.1.1.1.1.1.1.1.1.3.cmml">K</mi></mrow><mo stretchy="false" id="A1.E9.m1.1.1.1.1.1.1.1.1.3" xref="A1.E9.m1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A1.E9.m1.2.2.2.2.2.2.3a" xref="A1.E9.m1.2.2.2.2.2.2.3.cmml">​</mo><msup id="A1.E9.m1.2.2.2.2.2.2.2" xref="A1.E9.m1.2.2.2.2.2.2.2.cmml"><mrow id="A1.E9.m1.2.2.2.2.2.2.2.1.1" xref="A1.E9.m1.2.2.2.2.2.2.2.1.1.1.cmml"><mo stretchy="false" id="A1.E9.m1.2.2.2.2.2.2.2.1.1.2" xref="A1.E9.m1.2.2.2.2.2.2.2.1.1.1.cmml">(</mo><mrow id="A1.E9.m1.2.2.2.2.2.2.2.1.1.1" xref="A1.E9.m1.2.2.2.2.2.2.2.1.1.1.cmml"><mover accent="true" id="A1.E9.m1.2.2.2.2.2.2.2.1.1.1.2" xref="A1.E9.m1.2.2.2.2.2.2.2.1.1.1.2.cmml"><mi id="A1.E9.m1.2.2.2.2.2.2.2.1.1.1.2.2" xref="A1.E9.m1.2.2.2.2.2.2.2.1.1.1.2.2.cmml">W</mi><mo id="A1.E9.m1.2.2.2.2.2.2.2.1.1.1.2.1" xref="A1.E9.m1.2.2.2.2.2.2.2.1.1.1.2.1.cmml">^</mo></mover><mo lspace="0em" rspace="0em" id="A1.E9.m1.2.2.2.2.2.2.2.1.1.1.1" xref="A1.E9.m1.2.2.2.2.2.2.2.1.1.1.1.cmml">​</mo><mi id="A1.E9.m1.2.2.2.2.2.2.2.1.1.1.3" xref="A1.E9.m1.2.2.2.2.2.2.2.1.1.1.3.cmml">K</mi></mrow><mo stretchy="false" id="A1.E9.m1.2.2.2.2.2.2.2.1.1.3" xref="A1.E9.m1.2.2.2.2.2.2.2.1.1.1.cmml">)</mo></mrow><mi id="A1.E9.m1.2.2.2.2.2.2.2.3" xref="A1.E9.m1.2.2.2.2.2.2.2.3.cmml">T</mi></msup></mrow><mo id="A1.E9.m1.3.3.3.3.3.4" xref="A1.E9.m1.3.3.3.3.3.4.cmml">−</mo><mrow id="A1.E9.m1.3.3.3.3.3.3" xref="A1.E9.m1.3.3.3.3.3.3.cmml"><mi id="A1.E9.m1.3.3.3.3.3.3.3" xref="A1.E9.m1.3.3.3.3.3.3.3.cmml">V</mi><mo lspace="0em" rspace="0em" id="A1.E9.m1.3.3.3.3.3.3.2" xref="A1.E9.m1.3.3.3.3.3.3.2.cmml">​</mo><msup id="A1.E9.m1.3.3.3.3.3.3.1" xref="A1.E9.m1.3.3.3.3.3.3.1.cmml"><mrow id="A1.E9.m1.3.3.3.3.3.3.1.1.1" xref="A1.E9.m1.3.3.3.3.3.3.1.1.1.1.cmml"><mo stretchy="false" id="A1.E9.m1.3.3.3.3.3.3.1.1.1.2" xref="A1.E9.m1.3.3.3.3.3.3.1.1.1.1.cmml">(</mo><mrow id="A1.E9.m1.3.3.3.3.3.3.1.1.1.1" xref="A1.E9.m1.3.3.3.3.3.3.1.1.1.1.cmml"><mover accent="true" id="A1.E9.m1.3.3.3.3.3.3.1.1.1.1.2" xref="A1.E9.m1.3.3.3.3.3.3.1.1.1.1.2.cmml"><mi id="A1.E9.m1.3.3.3.3.3.3.1.1.1.1.2.2" xref="A1.E9.m1.3.3.3.3.3.3.1.1.1.1.2.2.cmml">W</mi><mo id="A1.E9.m1.3.3.3.3.3.3.1.1.1.1.2.1" xref="A1.E9.m1.3.3.3.3.3.3.1.1.1.1.2.1.cmml">^</mo></mover><mo lspace="0em" rspace="0em" id="A1.E9.m1.3.3.3.3.3.3.1.1.1.1.1" xref="A1.E9.m1.3.3.3.3.3.3.1.1.1.1.1.cmml">​</mo><mi id="A1.E9.m1.3.3.3.3.3.3.1.1.1.1.3" xref="A1.E9.m1.3.3.3.3.3.3.1.1.1.1.3.cmml">K</mi></mrow><mo stretchy="false" id="A1.E9.m1.3.3.3.3.3.3.1.1.1.3" xref="A1.E9.m1.3.3.3.3.3.3.1.1.1.1.cmml">)</mo></mrow><mi id="A1.E9.m1.3.3.3.3.3.3.1.3" xref="A1.E9.m1.3.3.3.3.3.3.1.3.cmml">T</mi></msup></mrow></mrow><mo id="A1.E9.m1.3.3.3.3.4" xref="A1.E9.m1.3.3.3.3.4.cmml">+</mo><mrow id="A1.E9.m1.3.3.3.3.5" xref="A1.E9.m1.3.3.3.3.5.cmml"><mstyle displaystyle="true" id="A1.E9.m1.3.3.3.3.5.2" xref="A1.E9.m1.3.3.3.3.5.2.cmml"><mfrac id="A1.E9.m1.3.3.3.3.5.2a" xref="A1.E9.m1.3.3.3.3.5.2.cmml"><mn id="A1.E9.m1.3.3.3.3.5.2.2" xref="A1.E9.m1.3.3.3.3.5.2.2.cmml">1</mn><mn id="A1.E9.m1.3.3.3.3.5.2.3" xref="A1.E9.m1.3.3.3.3.5.2.3.cmml">2</mn></mfrac></mstyle><mo lspace="0em" rspace="0em" id="A1.E9.m1.3.3.3.3.5.1" xref="A1.E9.m1.3.3.3.3.5.1.cmml">​</mo><mi id="A1.E9.m1.3.3.3.3.5.3" xref="A1.E9.m1.3.3.3.3.5.3.cmml">V</mi><mo lspace="0em" rspace="0em" id="A1.E9.m1.3.3.3.3.5.1a" xref="A1.E9.m1.3.3.3.3.5.1.cmml">​</mo><msup id="A1.E9.m1.3.3.3.3.5.4" xref="A1.E9.m1.3.3.3.3.5.4.cmml"><mi id="A1.E9.m1.3.3.3.3.5.4.2" xref="A1.E9.m1.3.3.3.3.5.4.2.cmml">V</mi><mi id="A1.E9.m1.3.3.3.3.5.4.3" xref="A1.E9.m1.3.3.3.3.5.4.3.cmml">T</mi></msup></mrow></mrow><mo id="A1.E9.m1.4.4.4.5" xref="A1.E9.m1.4.4.4.5.cmml">−</mo><mrow id="A1.E9.m1.4.4.4.4" xref="A1.E9.m1.4.4.4.4.cmml"><msup id="A1.E9.m1.4.4.4.4.3" xref="A1.E9.m1.4.4.4.4.3.cmml"><mi mathvariant="normal" id="A1.E9.m1.4.4.4.4.3.2" xref="A1.E9.m1.4.4.4.4.3.2.cmml">Λ</mi><mi id="A1.E9.m1.4.4.4.4.3.3" xref="A1.E9.m1.4.4.4.4.3.3.cmml">T</mi></msup><mo lspace="0em" rspace="0em" id="A1.E9.m1.4.4.4.4.2" xref="A1.E9.m1.4.4.4.4.2.cmml">​</mo><mrow id="A1.E9.m1.4.4.4.4.1.1" xref="A1.E9.m1.4.4.4.4.1.1.1.cmml"><mo stretchy="false" id="A1.E9.m1.4.4.4.4.1.1.2" xref="A1.E9.m1.4.4.4.4.1.1.1.cmml">(</mo><mrow id="A1.E9.m1.4.4.4.4.1.1.1" xref="A1.E9.m1.4.4.4.4.1.1.1.cmml"><mrow id="A1.E9.m1.4.4.4.4.1.1.1.2" xref="A1.E9.m1.4.4.4.4.1.1.1.2.cmml"><mover accent="true" id="A1.E9.m1.4.4.4.4.1.1.1.2.2" xref="A1.E9.m1.4.4.4.4.1.1.1.2.2.cmml"><mi id="A1.E9.m1.4.4.4.4.1.1.1.2.2.2" xref="A1.E9.m1.4.4.4.4.1.1.1.2.2.2.cmml">W</mi><mo id="A1.E9.m1.4.4.4.4.1.1.1.2.2.1" xref="A1.E9.m1.4.4.4.4.1.1.1.2.2.1.cmml">^</mo></mover><mo lspace="0em" rspace="0em" id="A1.E9.m1.4.4.4.4.1.1.1.2.1" xref="A1.E9.m1.4.4.4.4.1.1.1.2.1.cmml">​</mo><msub id="A1.E9.m1.4.4.4.4.1.1.1.2.3" xref="A1.E9.m1.4.4.4.4.1.1.1.2.3.cmml"><mi id="A1.E9.m1.4.4.4.4.1.1.1.2.3.2" xref="A1.E9.m1.4.4.4.4.1.1.1.2.3.2.cmml">k</mi><mo id="A1.E9.m1.4.4.4.4.1.1.1.2.3.3" xref="A1.E9.m1.4.4.4.4.1.1.1.2.3.3.cmml">∗</mo></msub></mrow><mo id="A1.E9.m1.4.4.4.4.1.1.1.1" xref="A1.E9.m1.4.4.4.4.1.1.1.1.cmml">−</mo><msub id="A1.E9.m1.4.4.4.4.1.1.1.3" xref="A1.E9.m1.4.4.4.4.1.1.1.3.cmml"><mi id="A1.E9.m1.4.4.4.4.1.1.1.3.2" xref="A1.E9.m1.4.4.4.4.1.1.1.3.2.cmml">v</mi><mo id="A1.E9.m1.4.4.4.4.1.1.1.3.3" xref="A1.E9.m1.4.4.4.4.1.1.1.3.3.cmml">∗</mo></msub></mrow><mo stretchy="false" id="A1.E9.m1.4.4.4.4.1.1.3" xref="A1.E9.m1.4.4.4.4.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E9.m1.4b"><apply id="A1.E9.m1.4.4.cmml" xref="A1.E9.m1.4.4"><eq id="A1.E9.m1.4.4.5.cmml" xref="A1.E9.m1.4.4.5"></eq><csymbol cd="latexml" id="A1.E9.m1.4.4.6.cmml" xref="A1.E9.m1.4.4.6">absent</csymbol><apply id="A1.E9.m1.4.4.4.cmml" xref="A1.E9.m1.4.4.4"><minus id="A1.E9.m1.4.4.4.5.cmml" xref="A1.E9.m1.4.4.4.5"></minus><apply id="A1.E9.m1.3.3.3.3.cmml" xref="A1.E9.m1.3.3.3.3"><plus id="A1.E9.m1.3.3.3.3.4.cmml" xref="A1.E9.m1.3.3.3.3.4"></plus><apply id="A1.E9.m1.3.3.3.3.3.cmml" xref="A1.E9.m1.3.3.3.3.3"><minus id="A1.E9.m1.3.3.3.3.3.4.cmml" xref="A1.E9.m1.3.3.3.3.3.4"></minus><apply id="A1.E9.m1.2.2.2.2.2.2.cmml" xref="A1.E9.m1.2.2.2.2.2.2"><times id="A1.E9.m1.2.2.2.2.2.2.3.cmml" xref="A1.E9.m1.2.2.2.2.2.2.3"></times><apply id="A1.E9.m1.2.2.2.2.2.2.4.cmml" xref="A1.E9.m1.2.2.2.2.2.2.4"><divide id="A1.E9.m1.2.2.2.2.2.2.4.1.cmml" xref="A1.E9.m1.2.2.2.2.2.2.4"></divide><cn type="integer" id="A1.E9.m1.2.2.2.2.2.2.4.2.cmml" xref="A1.E9.m1.2.2.2.2.2.2.4.2">1</cn><cn type="integer" id="A1.E9.m1.2.2.2.2.2.2.4.3.cmml" xref="A1.E9.m1.2.2.2.2.2.2.4.3">2</cn></apply><apply id="A1.E9.m1.1.1.1.1.1.1.1.1.1.cmml" xref="A1.E9.m1.1.1.1.1.1.1.1.1"><times id="A1.E9.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A1.E9.m1.1.1.1.1.1.1.1.1.1.1"></times><apply id="A1.E9.m1.1.1.1.1.1.1.1.1.1.2.cmml" xref="A1.E9.m1.1.1.1.1.1.1.1.1.1.2"><ci id="A1.E9.m1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="A1.E9.m1.1.1.1.1.1.1.1.1.1.2.1">^</ci><ci id="A1.E9.m1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="A1.E9.m1.1.1.1.1.1.1.1.1.1.2.2">𝑊</ci></apply><ci id="A1.E9.m1.1.1.1.1.1.1.1.1.1.3.cmml" xref="A1.E9.m1.1.1.1.1.1.1.1.1.1.3">𝐾</ci></apply><apply id="A1.E9.m1.2.2.2.2.2.2.2.cmml" xref="A1.E9.m1.2.2.2.2.2.2.2"><csymbol cd="ambiguous" id="A1.E9.m1.2.2.2.2.2.2.2.2.cmml" xref="A1.E9.m1.2.2.2.2.2.2.2">superscript</csymbol><apply id="A1.E9.m1.2.2.2.2.2.2.2.1.1.1.cmml" xref="A1.E9.m1.2.2.2.2.2.2.2.1.1"><times id="A1.E9.m1.2.2.2.2.2.2.2.1.1.1.1.cmml" xref="A1.E9.m1.2.2.2.2.2.2.2.1.1.1.1"></times><apply id="A1.E9.m1.2.2.2.2.2.2.2.1.1.1.2.cmml" xref="A1.E9.m1.2.2.2.2.2.2.2.1.1.1.2"><ci id="A1.E9.m1.2.2.2.2.2.2.2.1.1.1.2.1.cmml" xref="A1.E9.m1.2.2.2.2.2.2.2.1.1.1.2.1">^</ci><ci id="A1.E9.m1.2.2.2.2.2.2.2.1.1.1.2.2.cmml" xref="A1.E9.m1.2.2.2.2.2.2.2.1.1.1.2.2">𝑊</ci></apply><ci id="A1.E9.m1.2.2.2.2.2.2.2.1.1.1.3.cmml" xref="A1.E9.m1.2.2.2.2.2.2.2.1.1.1.3">𝐾</ci></apply><ci id="A1.E9.m1.2.2.2.2.2.2.2.3.cmml" xref="A1.E9.m1.2.2.2.2.2.2.2.3">𝑇</ci></apply></apply><apply id="A1.E9.m1.3.3.3.3.3.3.cmml" xref="A1.E9.m1.3.3.3.3.3.3"><times id="A1.E9.m1.3.3.3.3.3.3.2.cmml" xref="A1.E9.m1.3.3.3.3.3.3.2"></times><ci id="A1.E9.m1.3.3.3.3.3.3.3.cmml" xref="A1.E9.m1.3.3.3.3.3.3.3">𝑉</ci><apply id="A1.E9.m1.3.3.3.3.3.3.1.cmml" xref="A1.E9.m1.3.3.3.3.3.3.1"><csymbol cd="ambiguous" id="A1.E9.m1.3.3.3.3.3.3.1.2.cmml" xref="A1.E9.m1.3.3.3.3.3.3.1">superscript</csymbol><apply id="A1.E9.m1.3.3.3.3.3.3.1.1.1.1.cmml" xref="A1.E9.m1.3.3.3.3.3.3.1.1.1"><times id="A1.E9.m1.3.3.3.3.3.3.1.1.1.1.1.cmml" xref="A1.E9.m1.3.3.3.3.3.3.1.1.1.1.1"></times><apply id="A1.E9.m1.3.3.3.3.3.3.1.1.1.1.2.cmml" xref="A1.E9.m1.3.3.3.3.3.3.1.1.1.1.2"><ci id="A1.E9.m1.3.3.3.3.3.3.1.1.1.1.2.1.cmml" xref="A1.E9.m1.3.3.3.3.3.3.1.1.1.1.2.1">^</ci><ci id="A1.E9.m1.3.3.3.3.3.3.1.1.1.1.2.2.cmml" xref="A1.E9.m1.3.3.3.3.3.3.1.1.1.1.2.2">𝑊</ci></apply><ci id="A1.E9.m1.3.3.3.3.3.3.1.1.1.1.3.cmml" xref="A1.E9.m1.3.3.3.3.3.3.1.1.1.1.3">𝐾</ci></apply><ci id="A1.E9.m1.3.3.3.3.3.3.1.3.cmml" xref="A1.E9.m1.3.3.3.3.3.3.1.3">𝑇</ci></apply></apply></apply><apply id="A1.E9.m1.3.3.3.3.5.cmml" xref="A1.E9.m1.3.3.3.3.5"><times id="A1.E9.m1.3.3.3.3.5.1.cmml" xref="A1.E9.m1.3.3.3.3.5.1"></times><apply id="A1.E9.m1.3.3.3.3.5.2.cmml" xref="A1.E9.m1.3.3.3.3.5.2"><divide id="A1.E9.m1.3.3.3.3.5.2.1.cmml" xref="A1.E9.m1.3.3.3.3.5.2"></divide><cn type="integer" id="A1.E9.m1.3.3.3.3.5.2.2.cmml" xref="A1.E9.m1.3.3.3.3.5.2.2">1</cn><cn type="integer" id="A1.E9.m1.3.3.3.3.5.2.3.cmml" xref="A1.E9.m1.3.3.3.3.5.2.3">2</cn></apply><ci id="A1.E9.m1.3.3.3.3.5.3.cmml" xref="A1.E9.m1.3.3.3.3.5.3">𝑉</ci><apply id="A1.E9.m1.3.3.3.3.5.4.cmml" xref="A1.E9.m1.3.3.3.3.5.4"><csymbol cd="ambiguous" id="A1.E9.m1.3.3.3.3.5.4.1.cmml" xref="A1.E9.m1.3.3.3.3.5.4">superscript</csymbol><ci id="A1.E9.m1.3.3.3.3.5.4.2.cmml" xref="A1.E9.m1.3.3.3.3.5.4.2">𝑉</ci><ci id="A1.E9.m1.3.3.3.3.5.4.3.cmml" xref="A1.E9.m1.3.3.3.3.5.4.3">𝑇</ci></apply></apply></apply><apply id="A1.E9.m1.4.4.4.4.cmml" xref="A1.E9.m1.4.4.4.4"><times id="A1.E9.m1.4.4.4.4.2.cmml" xref="A1.E9.m1.4.4.4.4.2"></times><apply id="A1.E9.m1.4.4.4.4.3.cmml" xref="A1.E9.m1.4.4.4.4.3"><csymbol cd="ambiguous" id="A1.E9.m1.4.4.4.4.3.1.cmml" xref="A1.E9.m1.4.4.4.4.3">superscript</csymbol><ci id="A1.E9.m1.4.4.4.4.3.2.cmml" xref="A1.E9.m1.4.4.4.4.3.2">Λ</ci><ci id="A1.E9.m1.4.4.4.4.3.3.cmml" xref="A1.E9.m1.4.4.4.4.3.3">𝑇</ci></apply><apply id="A1.E9.m1.4.4.4.4.1.1.1.cmml" xref="A1.E9.m1.4.4.4.4.1.1"><minus id="A1.E9.m1.4.4.4.4.1.1.1.1.cmml" xref="A1.E9.m1.4.4.4.4.1.1.1.1"></minus><apply id="A1.E9.m1.4.4.4.4.1.1.1.2.cmml" xref="A1.E9.m1.4.4.4.4.1.1.1.2"><times id="A1.E9.m1.4.4.4.4.1.1.1.2.1.cmml" xref="A1.E9.m1.4.4.4.4.1.1.1.2.1"></times><apply id="A1.E9.m1.4.4.4.4.1.1.1.2.2.cmml" xref="A1.E9.m1.4.4.4.4.1.1.1.2.2"><ci id="A1.E9.m1.4.4.4.4.1.1.1.2.2.1.cmml" xref="A1.E9.m1.4.4.4.4.1.1.1.2.2.1">^</ci><ci id="A1.E9.m1.4.4.4.4.1.1.1.2.2.2.cmml" xref="A1.E9.m1.4.4.4.4.1.1.1.2.2.2">𝑊</ci></apply><apply id="A1.E9.m1.4.4.4.4.1.1.1.2.3.cmml" xref="A1.E9.m1.4.4.4.4.1.1.1.2.3"><csymbol cd="ambiguous" id="A1.E9.m1.4.4.4.4.1.1.1.2.3.1.cmml" xref="A1.E9.m1.4.4.4.4.1.1.1.2.3">subscript</csymbol><ci id="A1.E9.m1.4.4.4.4.1.1.1.2.3.2.cmml" xref="A1.E9.m1.4.4.4.4.1.1.1.2.3.2">𝑘</ci><times id="A1.E9.m1.4.4.4.4.1.1.1.2.3.3.cmml" xref="A1.E9.m1.4.4.4.4.1.1.1.2.3.3"></times></apply></apply><apply id="A1.E9.m1.4.4.4.4.1.1.1.3.cmml" xref="A1.E9.m1.4.4.4.4.1.1.1.3"><csymbol cd="ambiguous" id="A1.E9.m1.4.4.4.4.1.1.1.3.1.cmml" xref="A1.E9.m1.4.4.4.4.1.1.1.3">subscript</csymbol><ci id="A1.E9.m1.4.4.4.4.1.1.1.3.2.cmml" xref="A1.E9.m1.4.4.4.4.1.1.1.3.2">𝑣</ci><times id="A1.E9.m1.4.4.4.4.1.1.1.3.3.cmml" xref="A1.E9.m1.4.4.4.4.1.1.1.3.3"></times></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E9.m1.4c">\displaystyle=\frac{1}{2}(\hat{W}K)(\hat{W}K)^{T}-V(\hat{W}K)^{T}+\frac{1}{2}VV^{T}-\Lambda^{T}(\hat{W}k_{*}-v_{*})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(9)</span></td>
</tr></tbody>
<tbody id="A1.E10"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A1.E10.m1.2" class="ltx_Math" alttext="\displaystyle\text{setting}\quad 0=\frac{\partial L}{\partial\hat{W}}" display="inline"><semantics id="A1.E10.m1.2a"><mrow id="A1.E10.m1.2.3" xref="A1.E10.m1.2.3.cmml"><mrow id="A1.E10.m1.2.3.2.2" xref="A1.E10.m1.2.3.2.1.cmml"><mtext id="A1.E10.m1.1.1" xref="A1.E10.m1.1.1a.cmml">setting</mtext><mspace width="1em" id="A1.E10.m1.2.3.2.2.1" xref="A1.E10.m1.2.3.2.1.cmml"></mspace><mn id="A1.E10.m1.2.2" xref="A1.E10.m1.2.2.cmml">0</mn></mrow><mo id="A1.E10.m1.2.3.1" xref="A1.E10.m1.2.3.1.cmml">=</mo><mstyle displaystyle="true" id="A1.E10.m1.2.3.3" xref="A1.E10.m1.2.3.3.cmml"><mfrac id="A1.E10.m1.2.3.3a" xref="A1.E10.m1.2.3.3.cmml"><mrow id="A1.E10.m1.2.3.3.2" xref="A1.E10.m1.2.3.3.2.cmml"><mo rspace="0em" id="A1.E10.m1.2.3.3.2.1" xref="A1.E10.m1.2.3.3.2.1.cmml">∂</mo><mi id="A1.E10.m1.2.3.3.2.2" xref="A1.E10.m1.2.3.3.2.2.cmml">L</mi></mrow><mrow id="A1.E10.m1.2.3.3.3" xref="A1.E10.m1.2.3.3.3.cmml"><mo rspace="0em" id="A1.E10.m1.2.3.3.3.1" xref="A1.E10.m1.2.3.3.3.1.cmml">∂</mo><mover accent="true" id="A1.E10.m1.2.3.3.3.2" xref="A1.E10.m1.2.3.3.3.2.cmml"><mi id="A1.E10.m1.2.3.3.3.2.2" xref="A1.E10.m1.2.3.3.3.2.2.cmml">W</mi><mo id="A1.E10.m1.2.3.3.3.2.1" xref="A1.E10.m1.2.3.3.3.2.1.cmml">^</mo></mover></mrow></mfrac></mstyle></mrow><annotation-xml encoding="MathML-Content" id="A1.E10.m1.2b"><apply id="A1.E10.m1.2.3.cmml" xref="A1.E10.m1.2.3"><eq id="A1.E10.m1.2.3.1.cmml" xref="A1.E10.m1.2.3.1"></eq><list id="A1.E10.m1.2.3.2.1.cmml" xref="A1.E10.m1.2.3.2.2"><ci id="A1.E10.m1.1.1a.cmml" xref="A1.E10.m1.1.1"><mtext id="A1.E10.m1.1.1.cmml" xref="A1.E10.m1.1.1">setting</mtext></ci><cn type="integer" id="A1.E10.m1.2.2.cmml" xref="A1.E10.m1.2.2">0</cn></list><apply id="A1.E10.m1.2.3.3.cmml" xref="A1.E10.m1.2.3.3"><divide id="A1.E10.m1.2.3.3.1.cmml" xref="A1.E10.m1.2.3.3"></divide><apply id="A1.E10.m1.2.3.3.2.cmml" xref="A1.E10.m1.2.3.3.2"><partialdiff id="A1.E10.m1.2.3.3.2.1.cmml" xref="A1.E10.m1.2.3.3.2.1"></partialdiff><ci id="A1.E10.m1.2.3.3.2.2.cmml" xref="A1.E10.m1.2.3.3.2.2">𝐿</ci></apply><apply id="A1.E10.m1.2.3.3.3.cmml" xref="A1.E10.m1.2.3.3.3"><partialdiff id="A1.E10.m1.2.3.3.3.1.cmml" xref="A1.E10.m1.2.3.3.3.1"></partialdiff><apply id="A1.E10.m1.2.3.3.3.2.cmml" xref="A1.E10.m1.2.3.3.3.2"><ci id="A1.E10.m1.2.3.3.3.2.1.cmml" xref="A1.E10.m1.2.3.3.3.2.1">^</ci><ci id="A1.E10.m1.2.3.3.3.2.2.cmml" xref="A1.E10.m1.2.3.3.3.2.2">𝑊</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E10.m1.2c">\displaystyle\text{setting}\quad 0=\frac{\partial L}{\partial\hat{W}}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A1.E10.m2.1" class="ltx_Math" alttext="\displaystyle=\hat{W}(KK^{T})-VK^{T}-\Lambda k_{*}^{T}" display="inline"><semantics id="A1.E10.m2.1a"><mrow id="A1.E10.m2.1.1" xref="A1.E10.m2.1.1.cmml"><mi id="A1.E10.m2.1.1.3" xref="A1.E10.m2.1.1.3.cmml"></mi><mo id="A1.E10.m2.1.1.2" xref="A1.E10.m2.1.1.2.cmml">=</mo><mrow id="A1.E10.m2.1.1.1" xref="A1.E10.m2.1.1.1.cmml"><mrow id="A1.E10.m2.1.1.1.1" xref="A1.E10.m2.1.1.1.1.cmml"><mover accent="true" id="A1.E10.m2.1.1.1.1.3" xref="A1.E10.m2.1.1.1.1.3.cmml"><mi id="A1.E10.m2.1.1.1.1.3.2" xref="A1.E10.m2.1.1.1.1.3.2.cmml">W</mi><mo id="A1.E10.m2.1.1.1.1.3.1" xref="A1.E10.m2.1.1.1.1.3.1.cmml">^</mo></mover><mo lspace="0em" rspace="0em" id="A1.E10.m2.1.1.1.1.2" xref="A1.E10.m2.1.1.1.1.2.cmml">​</mo><mrow id="A1.E10.m2.1.1.1.1.1.1" xref="A1.E10.m2.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.E10.m2.1.1.1.1.1.1.2" xref="A1.E10.m2.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A1.E10.m2.1.1.1.1.1.1.1" xref="A1.E10.m2.1.1.1.1.1.1.1.cmml"><mi id="A1.E10.m2.1.1.1.1.1.1.1.2" xref="A1.E10.m2.1.1.1.1.1.1.1.2.cmml">K</mi><mo lspace="0em" rspace="0em" id="A1.E10.m2.1.1.1.1.1.1.1.1" xref="A1.E10.m2.1.1.1.1.1.1.1.1.cmml">​</mo><msup id="A1.E10.m2.1.1.1.1.1.1.1.3" xref="A1.E10.m2.1.1.1.1.1.1.1.3.cmml"><mi id="A1.E10.m2.1.1.1.1.1.1.1.3.2" xref="A1.E10.m2.1.1.1.1.1.1.1.3.2.cmml">K</mi><mi id="A1.E10.m2.1.1.1.1.1.1.1.3.3" xref="A1.E10.m2.1.1.1.1.1.1.1.3.3.cmml">T</mi></msup></mrow><mo stretchy="false" id="A1.E10.m2.1.1.1.1.1.1.3" xref="A1.E10.m2.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="A1.E10.m2.1.1.1.2" xref="A1.E10.m2.1.1.1.2.cmml">−</mo><mrow id="A1.E10.m2.1.1.1.3" xref="A1.E10.m2.1.1.1.3.cmml"><mi id="A1.E10.m2.1.1.1.3.2" xref="A1.E10.m2.1.1.1.3.2.cmml">V</mi><mo lspace="0em" rspace="0em" id="A1.E10.m2.1.1.1.3.1" xref="A1.E10.m2.1.1.1.3.1.cmml">​</mo><msup id="A1.E10.m2.1.1.1.3.3" xref="A1.E10.m2.1.1.1.3.3.cmml"><mi id="A1.E10.m2.1.1.1.3.3.2" xref="A1.E10.m2.1.1.1.3.3.2.cmml">K</mi><mi id="A1.E10.m2.1.1.1.3.3.3" xref="A1.E10.m2.1.1.1.3.3.3.cmml">T</mi></msup></mrow><mo id="A1.E10.m2.1.1.1.2a" xref="A1.E10.m2.1.1.1.2.cmml">−</mo><mrow id="A1.E10.m2.1.1.1.4" xref="A1.E10.m2.1.1.1.4.cmml"><mi mathvariant="normal" id="A1.E10.m2.1.1.1.4.2" xref="A1.E10.m2.1.1.1.4.2.cmml">Λ</mi><mo lspace="0em" rspace="0em" id="A1.E10.m2.1.1.1.4.1" xref="A1.E10.m2.1.1.1.4.1.cmml">​</mo><msubsup id="A1.E10.m2.1.1.1.4.3" xref="A1.E10.m2.1.1.1.4.3.cmml"><mi id="A1.E10.m2.1.1.1.4.3.2.2" xref="A1.E10.m2.1.1.1.4.3.2.2.cmml">k</mi><mo id="A1.E10.m2.1.1.1.4.3.2.3" xref="A1.E10.m2.1.1.1.4.3.2.3.cmml">∗</mo><mi id="A1.E10.m2.1.1.1.4.3.3" xref="A1.E10.m2.1.1.1.4.3.3.cmml">T</mi></msubsup></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E10.m2.1b"><apply id="A1.E10.m2.1.1.cmml" xref="A1.E10.m2.1.1"><eq id="A1.E10.m2.1.1.2.cmml" xref="A1.E10.m2.1.1.2"></eq><csymbol cd="latexml" id="A1.E10.m2.1.1.3.cmml" xref="A1.E10.m2.1.1.3">absent</csymbol><apply id="A1.E10.m2.1.1.1.cmml" xref="A1.E10.m2.1.1.1"><minus id="A1.E10.m2.1.1.1.2.cmml" xref="A1.E10.m2.1.1.1.2"></minus><apply id="A1.E10.m2.1.1.1.1.cmml" xref="A1.E10.m2.1.1.1.1"><times id="A1.E10.m2.1.1.1.1.2.cmml" xref="A1.E10.m2.1.1.1.1.2"></times><apply id="A1.E10.m2.1.1.1.1.3.cmml" xref="A1.E10.m2.1.1.1.1.3"><ci id="A1.E10.m2.1.1.1.1.3.1.cmml" xref="A1.E10.m2.1.1.1.1.3.1">^</ci><ci id="A1.E10.m2.1.1.1.1.3.2.cmml" xref="A1.E10.m2.1.1.1.1.3.2">𝑊</ci></apply><apply id="A1.E10.m2.1.1.1.1.1.1.1.cmml" xref="A1.E10.m2.1.1.1.1.1.1"><times id="A1.E10.m2.1.1.1.1.1.1.1.1.cmml" xref="A1.E10.m2.1.1.1.1.1.1.1.1"></times><ci id="A1.E10.m2.1.1.1.1.1.1.1.2.cmml" xref="A1.E10.m2.1.1.1.1.1.1.1.2">𝐾</ci><apply id="A1.E10.m2.1.1.1.1.1.1.1.3.cmml" xref="A1.E10.m2.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.E10.m2.1.1.1.1.1.1.1.3.1.cmml" xref="A1.E10.m2.1.1.1.1.1.1.1.3">superscript</csymbol><ci id="A1.E10.m2.1.1.1.1.1.1.1.3.2.cmml" xref="A1.E10.m2.1.1.1.1.1.1.1.3.2">𝐾</ci><ci id="A1.E10.m2.1.1.1.1.1.1.1.3.3.cmml" xref="A1.E10.m2.1.1.1.1.1.1.1.3.3">𝑇</ci></apply></apply></apply><apply id="A1.E10.m2.1.1.1.3.cmml" xref="A1.E10.m2.1.1.1.3"><times id="A1.E10.m2.1.1.1.3.1.cmml" xref="A1.E10.m2.1.1.1.3.1"></times><ci id="A1.E10.m2.1.1.1.3.2.cmml" xref="A1.E10.m2.1.1.1.3.2">𝑉</ci><apply id="A1.E10.m2.1.1.1.3.3.cmml" xref="A1.E10.m2.1.1.1.3.3"><csymbol cd="ambiguous" id="A1.E10.m2.1.1.1.3.3.1.cmml" xref="A1.E10.m2.1.1.1.3.3">superscript</csymbol><ci id="A1.E10.m2.1.1.1.3.3.2.cmml" xref="A1.E10.m2.1.1.1.3.3.2">𝐾</ci><ci id="A1.E10.m2.1.1.1.3.3.3.cmml" xref="A1.E10.m2.1.1.1.3.3.3">𝑇</ci></apply></apply><apply id="A1.E10.m2.1.1.1.4.cmml" xref="A1.E10.m2.1.1.1.4"><times id="A1.E10.m2.1.1.1.4.1.cmml" xref="A1.E10.m2.1.1.1.4.1"></times><ci id="A1.E10.m2.1.1.1.4.2.cmml" xref="A1.E10.m2.1.1.1.4.2">Λ</ci><apply id="A1.E10.m2.1.1.1.4.3.cmml" xref="A1.E10.m2.1.1.1.4.3"><csymbol cd="ambiguous" id="A1.E10.m2.1.1.1.4.3.1.cmml" xref="A1.E10.m2.1.1.1.4.3">superscript</csymbol><apply id="A1.E10.m2.1.1.1.4.3.2.cmml" xref="A1.E10.m2.1.1.1.4.3"><csymbol cd="ambiguous" id="A1.E10.m2.1.1.1.4.3.2.1.cmml" xref="A1.E10.m2.1.1.1.4.3">subscript</csymbol><ci id="A1.E10.m2.1.1.1.4.3.2.2.cmml" xref="A1.E10.m2.1.1.1.4.3.2.2">𝑘</ci><times id="A1.E10.m2.1.1.1.4.3.2.3.cmml" xref="A1.E10.m2.1.1.1.4.3.2.3"></times></apply><ci id="A1.E10.m2.1.1.1.4.3.3.cmml" xref="A1.E10.m2.1.1.1.4.3.3">𝑇</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E10.m2.1c">\displaystyle=\hat{W}(KK^{T})-VK^{T}-\Lambda k_{*}^{T}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(10)</span></td>
</tr></tbody>
<tbody id="A1.E11"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A1.E11.m1.1" class="ltx_Math" alttext="\displaystyle\hat{W}KK^{T}" display="inline"><semantics id="A1.E11.m1.1a"><mrow id="A1.E11.m1.1.1" xref="A1.E11.m1.1.1.cmml"><mover accent="true" id="A1.E11.m1.1.1.2" xref="A1.E11.m1.1.1.2.cmml"><mi id="A1.E11.m1.1.1.2.2" xref="A1.E11.m1.1.1.2.2.cmml">W</mi><mo id="A1.E11.m1.1.1.2.1" xref="A1.E11.m1.1.1.2.1.cmml">^</mo></mover><mo lspace="0em" rspace="0em" id="A1.E11.m1.1.1.1" xref="A1.E11.m1.1.1.1.cmml">​</mo><mi id="A1.E11.m1.1.1.3" xref="A1.E11.m1.1.1.3.cmml">K</mi><mo lspace="0em" rspace="0em" id="A1.E11.m1.1.1.1a" xref="A1.E11.m1.1.1.1.cmml">​</mo><msup id="A1.E11.m1.1.1.4" xref="A1.E11.m1.1.1.4.cmml"><mi id="A1.E11.m1.1.1.4.2" xref="A1.E11.m1.1.1.4.2.cmml">K</mi><mi id="A1.E11.m1.1.1.4.3" xref="A1.E11.m1.1.1.4.3.cmml">T</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="A1.E11.m1.1b"><apply id="A1.E11.m1.1.1.cmml" xref="A1.E11.m1.1.1"><times id="A1.E11.m1.1.1.1.cmml" xref="A1.E11.m1.1.1.1"></times><apply id="A1.E11.m1.1.1.2.cmml" xref="A1.E11.m1.1.1.2"><ci id="A1.E11.m1.1.1.2.1.cmml" xref="A1.E11.m1.1.1.2.1">^</ci><ci id="A1.E11.m1.1.1.2.2.cmml" xref="A1.E11.m1.1.1.2.2">𝑊</ci></apply><ci id="A1.E11.m1.1.1.3.cmml" xref="A1.E11.m1.1.1.3">𝐾</ci><apply id="A1.E11.m1.1.1.4.cmml" xref="A1.E11.m1.1.1.4"><csymbol cd="ambiguous" id="A1.E11.m1.1.1.4.1.cmml" xref="A1.E11.m1.1.1.4">superscript</csymbol><ci id="A1.E11.m1.1.1.4.2.cmml" xref="A1.E11.m1.1.1.4.2">𝐾</ci><ci id="A1.E11.m1.1.1.4.3.cmml" xref="A1.E11.m1.1.1.4.3">𝑇</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E11.m1.1c">\displaystyle\hat{W}KK^{T}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A1.E11.m2.1" class="ltx_Math" alttext="\displaystyle=VK^{T}+\Lambda k_{*}^{T}" display="inline"><semantics id="A1.E11.m2.1a"><mrow id="A1.E11.m2.1.1" xref="A1.E11.m2.1.1.cmml"><mi id="A1.E11.m2.1.1.2" xref="A1.E11.m2.1.1.2.cmml"></mi><mo id="A1.E11.m2.1.1.1" xref="A1.E11.m2.1.1.1.cmml">=</mo><mrow id="A1.E11.m2.1.1.3" xref="A1.E11.m2.1.1.3.cmml"><mrow id="A1.E11.m2.1.1.3.2" xref="A1.E11.m2.1.1.3.2.cmml"><mi id="A1.E11.m2.1.1.3.2.2" xref="A1.E11.m2.1.1.3.2.2.cmml">V</mi><mo lspace="0em" rspace="0em" id="A1.E11.m2.1.1.3.2.1" xref="A1.E11.m2.1.1.3.2.1.cmml">​</mo><msup id="A1.E11.m2.1.1.3.2.3" xref="A1.E11.m2.1.1.3.2.3.cmml"><mi id="A1.E11.m2.1.1.3.2.3.2" xref="A1.E11.m2.1.1.3.2.3.2.cmml">K</mi><mi id="A1.E11.m2.1.1.3.2.3.3" xref="A1.E11.m2.1.1.3.2.3.3.cmml">T</mi></msup></mrow><mo id="A1.E11.m2.1.1.3.1" xref="A1.E11.m2.1.1.3.1.cmml">+</mo><mrow id="A1.E11.m2.1.1.3.3" xref="A1.E11.m2.1.1.3.3.cmml"><mi mathvariant="normal" id="A1.E11.m2.1.1.3.3.2" xref="A1.E11.m2.1.1.3.3.2.cmml">Λ</mi><mo lspace="0em" rspace="0em" id="A1.E11.m2.1.1.3.3.1" xref="A1.E11.m2.1.1.3.3.1.cmml">​</mo><msubsup id="A1.E11.m2.1.1.3.3.3" xref="A1.E11.m2.1.1.3.3.3.cmml"><mi id="A1.E11.m2.1.1.3.3.3.2.2" xref="A1.E11.m2.1.1.3.3.3.2.2.cmml">k</mi><mo id="A1.E11.m2.1.1.3.3.3.2.3" xref="A1.E11.m2.1.1.3.3.3.2.3.cmml">∗</mo><mi id="A1.E11.m2.1.1.3.3.3.3" xref="A1.E11.m2.1.1.3.3.3.3.cmml">T</mi></msubsup></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E11.m2.1b"><apply id="A1.E11.m2.1.1.cmml" xref="A1.E11.m2.1.1"><eq id="A1.E11.m2.1.1.1.cmml" xref="A1.E11.m2.1.1.1"></eq><csymbol cd="latexml" id="A1.E11.m2.1.1.2.cmml" xref="A1.E11.m2.1.1.2">absent</csymbol><apply id="A1.E11.m2.1.1.3.cmml" xref="A1.E11.m2.1.1.3"><plus id="A1.E11.m2.1.1.3.1.cmml" xref="A1.E11.m2.1.1.3.1"></plus><apply id="A1.E11.m2.1.1.3.2.cmml" xref="A1.E11.m2.1.1.3.2"><times id="A1.E11.m2.1.1.3.2.1.cmml" xref="A1.E11.m2.1.1.3.2.1"></times><ci id="A1.E11.m2.1.1.3.2.2.cmml" xref="A1.E11.m2.1.1.3.2.2">𝑉</ci><apply id="A1.E11.m2.1.1.3.2.3.cmml" xref="A1.E11.m2.1.1.3.2.3"><csymbol cd="ambiguous" id="A1.E11.m2.1.1.3.2.3.1.cmml" xref="A1.E11.m2.1.1.3.2.3">superscript</csymbol><ci id="A1.E11.m2.1.1.3.2.3.2.cmml" xref="A1.E11.m2.1.1.3.2.3.2">𝐾</ci><ci id="A1.E11.m2.1.1.3.2.3.3.cmml" xref="A1.E11.m2.1.1.3.2.3.3">𝑇</ci></apply></apply><apply id="A1.E11.m2.1.1.3.3.cmml" xref="A1.E11.m2.1.1.3.3"><times id="A1.E11.m2.1.1.3.3.1.cmml" xref="A1.E11.m2.1.1.3.3.1"></times><ci id="A1.E11.m2.1.1.3.3.2.cmml" xref="A1.E11.m2.1.1.3.3.2">Λ</ci><apply id="A1.E11.m2.1.1.3.3.3.cmml" xref="A1.E11.m2.1.1.3.3.3"><csymbol cd="ambiguous" id="A1.E11.m2.1.1.3.3.3.1.cmml" xref="A1.E11.m2.1.1.3.3.3">superscript</csymbol><apply id="A1.E11.m2.1.1.3.3.3.2.cmml" xref="A1.E11.m2.1.1.3.3.3"><csymbol cd="ambiguous" id="A1.E11.m2.1.1.3.3.3.2.1.cmml" xref="A1.E11.m2.1.1.3.3.3">subscript</csymbol><ci id="A1.E11.m2.1.1.3.3.3.2.2.cmml" xref="A1.E11.m2.1.1.3.3.3.2.2">𝑘</ci><times id="A1.E11.m2.1.1.3.3.3.2.3.cmml" xref="A1.E11.m2.1.1.3.3.3.2.3"></times></apply><ci id="A1.E11.m2.1.1.3.3.3.3.cmml" xref="A1.E11.m2.1.1.3.3.3.3">𝑇</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E11.m2.1c">\displaystyle=VK^{T}+\Lambda k_{*}^{T}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(11)</span></td>
</tr></tbody>
</table>
<p id="A1.p4.9" class="ltx_p">Subtracting Eqn. <a href="#A1.E6" title="In Appendix A Solving for Λ Algebraically ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> from Eqn. <a href="#A1.E11" title="In Appendix A Solving for Λ Algebraically ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a>, most terms cancel, and we obtain the update rule:</p>
<table id="A10.EGx8" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A1.E12"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A1.E12.m1.1" class="ltx_Math" alttext="\displaystyle(\hat{W}-W)KK^{T}" display="inline"><semantics id="A1.E12.m1.1a"><mrow id="A1.E12.m1.1.1" xref="A1.E12.m1.1.1.cmml"><mrow id="A1.E12.m1.1.1.1.1" xref="A1.E12.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.E12.m1.1.1.1.1.2" xref="A1.E12.m1.1.1.1.1.1.cmml">(</mo><mrow id="A1.E12.m1.1.1.1.1.1" xref="A1.E12.m1.1.1.1.1.1.cmml"><mover accent="true" id="A1.E12.m1.1.1.1.1.1.2" xref="A1.E12.m1.1.1.1.1.1.2.cmml"><mi id="A1.E12.m1.1.1.1.1.1.2.2" xref="A1.E12.m1.1.1.1.1.1.2.2.cmml">W</mi><mo id="A1.E12.m1.1.1.1.1.1.2.1" xref="A1.E12.m1.1.1.1.1.1.2.1.cmml">^</mo></mover><mo id="A1.E12.m1.1.1.1.1.1.1" xref="A1.E12.m1.1.1.1.1.1.1.cmml">−</mo><mi id="A1.E12.m1.1.1.1.1.1.3" xref="A1.E12.m1.1.1.1.1.1.3.cmml">W</mi></mrow><mo stretchy="false" id="A1.E12.m1.1.1.1.1.3" xref="A1.E12.m1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A1.E12.m1.1.1.2" xref="A1.E12.m1.1.1.2.cmml">​</mo><mi id="A1.E12.m1.1.1.3" xref="A1.E12.m1.1.1.3.cmml">K</mi><mo lspace="0em" rspace="0em" id="A1.E12.m1.1.1.2a" xref="A1.E12.m1.1.1.2.cmml">​</mo><msup id="A1.E12.m1.1.1.4" xref="A1.E12.m1.1.1.4.cmml"><mi id="A1.E12.m1.1.1.4.2" xref="A1.E12.m1.1.1.4.2.cmml">K</mi><mi id="A1.E12.m1.1.1.4.3" xref="A1.E12.m1.1.1.4.3.cmml">T</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="A1.E12.m1.1b"><apply id="A1.E12.m1.1.1.cmml" xref="A1.E12.m1.1.1"><times id="A1.E12.m1.1.1.2.cmml" xref="A1.E12.m1.1.1.2"></times><apply id="A1.E12.m1.1.1.1.1.1.cmml" xref="A1.E12.m1.1.1.1.1"><minus id="A1.E12.m1.1.1.1.1.1.1.cmml" xref="A1.E12.m1.1.1.1.1.1.1"></minus><apply id="A1.E12.m1.1.1.1.1.1.2.cmml" xref="A1.E12.m1.1.1.1.1.1.2"><ci id="A1.E12.m1.1.1.1.1.1.2.1.cmml" xref="A1.E12.m1.1.1.1.1.1.2.1">^</ci><ci id="A1.E12.m1.1.1.1.1.1.2.2.cmml" xref="A1.E12.m1.1.1.1.1.1.2.2">𝑊</ci></apply><ci id="A1.E12.m1.1.1.1.1.1.3.cmml" xref="A1.E12.m1.1.1.1.1.1.3">𝑊</ci></apply><ci id="A1.E12.m1.1.1.3.cmml" xref="A1.E12.m1.1.1.3">𝐾</ci><apply id="A1.E12.m1.1.1.4.cmml" xref="A1.E12.m1.1.1.4"><csymbol cd="ambiguous" id="A1.E12.m1.1.1.4.1.cmml" xref="A1.E12.m1.1.1.4">superscript</csymbol><ci id="A1.E12.m1.1.1.4.2.cmml" xref="A1.E12.m1.1.1.4.2">𝐾</ci><ci id="A1.E12.m1.1.1.4.3.cmml" xref="A1.E12.m1.1.1.4.3">𝑇</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E12.m1.1c">\displaystyle(\hat{W}-W)KK^{T}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A1.E12.m2.1" class="ltx_Math" alttext="\displaystyle=\Lambda k_{*}^{T}" display="inline"><semantics id="A1.E12.m2.1a"><mrow id="A1.E12.m2.1.1" xref="A1.E12.m2.1.1.cmml"><mi id="A1.E12.m2.1.1.2" xref="A1.E12.m2.1.1.2.cmml"></mi><mo id="A1.E12.m2.1.1.1" xref="A1.E12.m2.1.1.1.cmml">=</mo><mrow id="A1.E12.m2.1.1.3" xref="A1.E12.m2.1.1.3.cmml"><mi mathvariant="normal" id="A1.E12.m2.1.1.3.2" xref="A1.E12.m2.1.1.3.2.cmml">Λ</mi><mo lspace="0em" rspace="0em" id="A1.E12.m2.1.1.3.1" xref="A1.E12.m2.1.1.3.1.cmml">​</mo><msubsup id="A1.E12.m2.1.1.3.3" xref="A1.E12.m2.1.1.3.3.cmml"><mi id="A1.E12.m2.1.1.3.3.2.2" xref="A1.E12.m2.1.1.3.3.2.2.cmml">k</mi><mo id="A1.E12.m2.1.1.3.3.2.3" xref="A1.E12.m2.1.1.3.3.2.3.cmml">∗</mo><mi id="A1.E12.m2.1.1.3.3.3" xref="A1.E12.m2.1.1.3.3.3.cmml">T</mi></msubsup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E12.m2.1b"><apply id="A1.E12.m2.1.1.cmml" xref="A1.E12.m2.1.1"><eq id="A1.E12.m2.1.1.1.cmml" xref="A1.E12.m2.1.1.1"></eq><csymbol cd="latexml" id="A1.E12.m2.1.1.2.cmml" xref="A1.E12.m2.1.1.2">absent</csymbol><apply id="A1.E12.m2.1.1.3.cmml" xref="A1.E12.m2.1.1.3"><times id="A1.E12.m2.1.1.3.1.cmml" xref="A1.E12.m2.1.1.3.1"></times><ci id="A1.E12.m2.1.1.3.2.cmml" xref="A1.E12.m2.1.1.3.2">Λ</ci><apply id="A1.E12.m2.1.1.3.3.cmml" xref="A1.E12.m2.1.1.3.3"><csymbol cd="ambiguous" id="A1.E12.m2.1.1.3.3.1.cmml" xref="A1.E12.m2.1.1.3.3">superscript</csymbol><apply id="A1.E12.m2.1.1.3.3.2.cmml" xref="A1.E12.m2.1.1.3.3"><csymbol cd="ambiguous" id="A1.E12.m2.1.1.3.3.2.1.cmml" xref="A1.E12.m2.1.1.3.3">subscript</csymbol><ci id="A1.E12.m2.1.1.3.3.2.2.cmml" xref="A1.E12.m2.1.1.3.3.2.2">𝑘</ci><times id="A1.E12.m2.1.1.3.3.2.3.cmml" xref="A1.E12.m2.1.1.3.3.2.3"></times></apply><ci id="A1.E12.m2.1.1.3.3.3.cmml" xref="A1.E12.m2.1.1.3.3.3">𝑇</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E12.m2.1c">\displaystyle=\Lambda k_{*}^{T}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(12)</span></td>
</tr></tbody>
<tbody id="A1.E13"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A1.E13.m1.1" class="ltx_Math" alttext="\displaystyle\hat{W}" display="inline"><semantics id="A1.E13.m1.1a"><mover accent="true" id="A1.E13.m1.1.1" xref="A1.E13.m1.1.1.cmml"><mi id="A1.E13.m1.1.1.2" xref="A1.E13.m1.1.1.2.cmml">W</mi><mo id="A1.E13.m1.1.1.1" xref="A1.E13.m1.1.1.1.cmml">^</mo></mover><annotation-xml encoding="MathML-Content" id="A1.E13.m1.1b"><apply id="A1.E13.m1.1.1.cmml" xref="A1.E13.m1.1.1"><ci id="A1.E13.m1.1.1.1.cmml" xref="A1.E13.m1.1.1.1">^</ci><ci id="A1.E13.m1.1.1.2.cmml" xref="A1.E13.m1.1.1.2">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E13.m1.1c">\displaystyle\hat{W}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A1.E13.m2.1" class="ltx_Math" alttext="\displaystyle=W+\Lambda(C^{-1}k_{*})^{T}" display="inline"><semantics id="A1.E13.m2.1a"><mrow id="A1.E13.m2.1.1" xref="A1.E13.m2.1.1.cmml"><mi id="A1.E13.m2.1.1.3" xref="A1.E13.m2.1.1.3.cmml"></mi><mo id="A1.E13.m2.1.1.2" xref="A1.E13.m2.1.1.2.cmml">=</mo><mrow id="A1.E13.m2.1.1.1" xref="A1.E13.m2.1.1.1.cmml"><mi id="A1.E13.m2.1.1.1.3" xref="A1.E13.m2.1.1.1.3.cmml">W</mi><mo id="A1.E13.m2.1.1.1.2" xref="A1.E13.m2.1.1.1.2.cmml">+</mo><mrow id="A1.E13.m2.1.1.1.1" xref="A1.E13.m2.1.1.1.1.cmml"><mi mathvariant="normal" id="A1.E13.m2.1.1.1.1.3" xref="A1.E13.m2.1.1.1.1.3.cmml">Λ</mi><mo lspace="0em" rspace="0em" id="A1.E13.m2.1.1.1.1.2" xref="A1.E13.m2.1.1.1.1.2.cmml">​</mo><msup id="A1.E13.m2.1.1.1.1.1" xref="A1.E13.m2.1.1.1.1.1.cmml"><mrow id="A1.E13.m2.1.1.1.1.1.1.1" xref="A1.E13.m2.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.E13.m2.1.1.1.1.1.1.1.2" xref="A1.E13.m2.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A1.E13.m2.1.1.1.1.1.1.1.1" xref="A1.E13.m2.1.1.1.1.1.1.1.1.cmml"><msup id="A1.E13.m2.1.1.1.1.1.1.1.1.2" xref="A1.E13.m2.1.1.1.1.1.1.1.1.2.cmml"><mi id="A1.E13.m2.1.1.1.1.1.1.1.1.2.2" xref="A1.E13.m2.1.1.1.1.1.1.1.1.2.2.cmml">C</mi><mrow id="A1.E13.m2.1.1.1.1.1.1.1.1.2.3" xref="A1.E13.m2.1.1.1.1.1.1.1.1.2.3.cmml"><mo id="A1.E13.m2.1.1.1.1.1.1.1.1.2.3a" xref="A1.E13.m2.1.1.1.1.1.1.1.1.2.3.cmml">−</mo><mn id="A1.E13.m2.1.1.1.1.1.1.1.1.2.3.2" xref="A1.E13.m2.1.1.1.1.1.1.1.1.2.3.2.cmml">1</mn></mrow></msup><mo lspace="0em" rspace="0em" id="A1.E13.m2.1.1.1.1.1.1.1.1.1" xref="A1.E13.m2.1.1.1.1.1.1.1.1.1.cmml">​</mo><msub id="A1.E13.m2.1.1.1.1.1.1.1.1.3" xref="A1.E13.m2.1.1.1.1.1.1.1.1.3.cmml"><mi id="A1.E13.m2.1.1.1.1.1.1.1.1.3.2" xref="A1.E13.m2.1.1.1.1.1.1.1.1.3.2.cmml">k</mi><mo id="A1.E13.m2.1.1.1.1.1.1.1.1.3.3" xref="A1.E13.m2.1.1.1.1.1.1.1.1.3.3.cmml">∗</mo></msub></mrow><mo stretchy="false" id="A1.E13.m2.1.1.1.1.1.1.1.3" xref="A1.E13.m2.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mi id="A1.E13.m2.1.1.1.1.1.3" xref="A1.E13.m2.1.1.1.1.1.3.cmml">T</mi></msup></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E13.m2.1b"><apply id="A1.E13.m2.1.1.cmml" xref="A1.E13.m2.1.1"><eq id="A1.E13.m2.1.1.2.cmml" xref="A1.E13.m2.1.1.2"></eq><csymbol cd="latexml" id="A1.E13.m2.1.1.3.cmml" xref="A1.E13.m2.1.1.3">absent</csymbol><apply id="A1.E13.m2.1.1.1.cmml" xref="A1.E13.m2.1.1.1"><plus id="A1.E13.m2.1.1.1.2.cmml" xref="A1.E13.m2.1.1.1.2"></plus><ci id="A1.E13.m2.1.1.1.3.cmml" xref="A1.E13.m2.1.1.1.3">𝑊</ci><apply id="A1.E13.m2.1.1.1.1.cmml" xref="A1.E13.m2.1.1.1.1"><times id="A1.E13.m2.1.1.1.1.2.cmml" xref="A1.E13.m2.1.1.1.1.2"></times><ci id="A1.E13.m2.1.1.1.1.3.cmml" xref="A1.E13.m2.1.1.1.1.3">Λ</ci><apply id="A1.E13.m2.1.1.1.1.1.cmml" xref="A1.E13.m2.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.E13.m2.1.1.1.1.1.2.cmml" xref="A1.E13.m2.1.1.1.1.1">superscript</csymbol><apply id="A1.E13.m2.1.1.1.1.1.1.1.1.cmml" xref="A1.E13.m2.1.1.1.1.1.1.1"><times id="A1.E13.m2.1.1.1.1.1.1.1.1.1.cmml" xref="A1.E13.m2.1.1.1.1.1.1.1.1.1"></times><apply id="A1.E13.m2.1.1.1.1.1.1.1.1.2.cmml" xref="A1.E13.m2.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A1.E13.m2.1.1.1.1.1.1.1.1.2.1.cmml" xref="A1.E13.m2.1.1.1.1.1.1.1.1.2">superscript</csymbol><ci id="A1.E13.m2.1.1.1.1.1.1.1.1.2.2.cmml" xref="A1.E13.m2.1.1.1.1.1.1.1.1.2.2">𝐶</ci><apply id="A1.E13.m2.1.1.1.1.1.1.1.1.2.3.cmml" xref="A1.E13.m2.1.1.1.1.1.1.1.1.2.3"><minus id="A1.E13.m2.1.1.1.1.1.1.1.1.2.3.1.cmml" xref="A1.E13.m2.1.1.1.1.1.1.1.1.2.3"></minus><cn type="integer" id="A1.E13.m2.1.1.1.1.1.1.1.1.2.3.2.cmml" xref="A1.E13.m2.1.1.1.1.1.1.1.1.2.3.2">1</cn></apply></apply><apply id="A1.E13.m2.1.1.1.1.1.1.1.1.3.cmml" xref="A1.E13.m2.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.E13.m2.1.1.1.1.1.1.1.1.3.1.cmml" xref="A1.E13.m2.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="A1.E13.m2.1.1.1.1.1.1.1.1.3.2.cmml" xref="A1.E13.m2.1.1.1.1.1.1.1.1.3.2">𝑘</ci><times id="A1.E13.m2.1.1.1.1.1.1.1.1.3.3.cmml" xref="A1.E13.m2.1.1.1.1.1.1.1.1.3.3"></times></apply></apply><ci id="A1.E13.m2.1.1.1.1.1.3.cmml" xref="A1.E13.m2.1.1.1.1.1.3">𝑇</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E13.m2.1c">\displaystyle=W+\Lambda(C^{-1}k_{*})^{T}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(13)</span></td>
</tr></tbody>
</table>
<p id="A1.p4.5" class="ltx_p">The last step is obtained by defining <math id="A1.p4.2.m1.1" class="ltx_Math" alttext="C=KK^{T}" display="inline"><semantics id="A1.p4.2.m1.1a"><mrow id="A1.p4.2.m1.1.1" xref="A1.p4.2.m1.1.1.cmml"><mi id="A1.p4.2.m1.1.1.2" xref="A1.p4.2.m1.1.1.2.cmml">C</mi><mo id="A1.p4.2.m1.1.1.1" xref="A1.p4.2.m1.1.1.1.cmml">=</mo><mrow id="A1.p4.2.m1.1.1.3" xref="A1.p4.2.m1.1.1.3.cmml"><mi id="A1.p4.2.m1.1.1.3.2" xref="A1.p4.2.m1.1.1.3.2.cmml">K</mi><mo lspace="0em" rspace="0em" id="A1.p4.2.m1.1.1.3.1" xref="A1.p4.2.m1.1.1.3.1.cmml">​</mo><msup id="A1.p4.2.m1.1.1.3.3" xref="A1.p4.2.m1.1.1.3.3.cmml"><mi id="A1.p4.2.m1.1.1.3.3.2" xref="A1.p4.2.m1.1.1.3.3.2.cmml">K</mi><mi id="A1.p4.2.m1.1.1.3.3.3" xref="A1.p4.2.m1.1.1.3.3.3.cmml">T</mi></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.p4.2.m1.1b"><apply id="A1.p4.2.m1.1.1.cmml" xref="A1.p4.2.m1.1.1"><eq id="A1.p4.2.m1.1.1.1.cmml" xref="A1.p4.2.m1.1.1.1"></eq><ci id="A1.p4.2.m1.1.1.2.cmml" xref="A1.p4.2.m1.1.1.2">𝐶</ci><apply id="A1.p4.2.m1.1.1.3.cmml" xref="A1.p4.2.m1.1.1.3"><times id="A1.p4.2.m1.1.1.3.1.cmml" xref="A1.p4.2.m1.1.1.3.1"></times><ci id="A1.p4.2.m1.1.1.3.2.cmml" xref="A1.p4.2.m1.1.1.3.2">𝐾</ci><apply id="A1.p4.2.m1.1.1.3.3.cmml" xref="A1.p4.2.m1.1.1.3.3"><csymbol cd="ambiguous" id="A1.p4.2.m1.1.1.3.3.1.cmml" xref="A1.p4.2.m1.1.1.3.3">superscript</csymbol><ci id="A1.p4.2.m1.1.1.3.3.2.cmml" xref="A1.p4.2.m1.1.1.3.3.2">𝐾</ci><ci id="A1.p4.2.m1.1.1.3.3.3.cmml" xref="A1.p4.2.m1.1.1.3.3.3">𝑇</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p4.2.m1.1c">C=KK^{T}</annotation></semantics></math>, assuming <math id="A1.p4.3.m2.1" class="ltx_Math" alttext="C" display="inline"><semantics id="A1.p4.3.m2.1a"><mi id="A1.p4.3.m2.1.1" xref="A1.p4.3.m2.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="A1.p4.3.m2.1b"><ci id="A1.p4.3.m2.1.1.cmml" xref="A1.p4.3.m2.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p4.3.m2.1c">C</annotation></semantics></math> is nondegenerate, and exploiting the symmetry of <math id="A1.p4.4.m3.1" class="ltx_Math" alttext="C" display="inline"><semantics id="A1.p4.4.m3.1a"><mi id="A1.p4.4.m3.1.1" xref="A1.p4.4.m3.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="A1.p4.4.m3.1b"><ci id="A1.p4.4.m3.1.1.cmml" xref="A1.p4.4.m3.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p4.4.m3.1c">C</annotation></semantics></math>. Here we also write the row vector term as <math id="A1.p4.5.m4.1" class="ltx_Math" alttext="u^{T}=(C^{-1}k_{*})^{T}\in\mathbb{R}^{D}" display="inline"><semantics id="A1.p4.5.m4.1a"><mrow id="A1.p4.5.m4.1.1" xref="A1.p4.5.m4.1.1.cmml"><msup id="A1.p4.5.m4.1.1.3" xref="A1.p4.5.m4.1.1.3.cmml"><mi id="A1.p4.5.m4.1.1.3.2" xref="A1.p4.5.m4.1.1.3.2.cmml">u</mi><mi id="A1.p4.5.m4.1.1.3.3" xref="A1.p4.5.m4.1.1.3.3.cmml">T</mi></msup><mo id="A1.p4.5.m4.1.1.4" xref="A1.p4.5.m4.1.1.4.cmml">=</mo><msup id="A1.p4.5.m4.1.1.1" xref="A1.p4.5.m4.1.1.1.cmml"><mrow id="A1.p4.5.m4.1.1.1.1.1" xref="A1.p4.5.m4.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.p4.5.m4.1.1.1.1.1.2" xref="A1.p4.5.m4.1.1.1.1.1.1.cmml">(</mo><mrow id="A1.p4.5.m4.1.1.1.1.1.1" xref="A1.p4.5.m4.1.1.1.1.1.1.cmml"><msup id="A1.p4.5.m4.1.1.1.1.1.1.2" xref="A1.p4.5.m4.1.1.1.1.1.1.2.cmml"><mi id="A1.p4.5.m4.1.1.1.1.1.1.2.2" xref="A1.p4.5.m4.1.1.1.1.1.1.2.2.cmml">C</mi><mrow id="A1.p4.5.m4.1.1.1.1.1.1.2.3" xref="A1.p4.5.m4.1.1.1.1.1.1.2.3.cmml"><mo id="A1.p4.5.m4.1.1.1.1.1.1.2.3a" xref="A1.p4.5.m4.1.1.1.1.1.1.2.3.cmml">−</mo><mn id="A1.p4.5.m4.1.1.1.1.1.1.2.3.2" xref="A1.p4.5.m4.1.1.1.1.1.1.2.3.2.cmml">1</mn></mrow></msup><mo lspace="0em" rspace="0em" id="A1.p4.5.m4.1.1.1.1.1.1.1" xref="A1.p4.5.m4.1.1.1.1.1.1.1.cmml">​</mo><msub id="A1.p4.5.m4.1.1.1.1.1.1.3" xref="A1.p4.5.m4.1.1.1.1.1.1.3.cmml"><mi id="A1.p4.5.m4.1.1.1.1.1.1.3.2" xref="A1.p4.5.m4.1.1.1.1.1.1.3.2.cmml">k</mi><mo id="A1.p4.5.m4.1.1.1.1.1.1.3.3" xref="A1.p4.5.m4.1.1.1.1.1.1.3.3.cmml">∗</mo></msub></mrow><mo stretchy="false" id="A1.p4.5.m4.1.1.1.1.1.3" xref="A1.p4.5.m4.1.1.1.1.1.1.cmml">)</mo></mrow><mi id="A1.p4.5.m4.1.1.1.3" xref="A1.p4.5.m4.1.1.1.3.cmml">T</mi></msup><mo id="A1.p4.5.m4.1.1.5" xref="A1.p4.5.m4.1.1.5.cmml">∈</mo><msup id="A1.p4.5.m4.1.1.6" xref="A1.p4.5.m4.1.1.6.cmml"><mi id="A1.p4.5.m4.1.1.6.2" xref="A1.p4.5.m4.1.1.6.2.cmml">ℝ</mi><mi id="A1.p4.5.m4.1.1.6.3" xref="A1.p4.5.m4.1.1.6.3.cmml">D</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="A1.p4.5.m4.1b"><apply id="A1.p4.5.m4.1.1.cmml" xref="A1.p4.5.m4.1.1"><and id="A1.p4.5.m4.1.1a.cmml" xref="A1.p4.5.m4.1.1"></and><apply id="A1.p4.5.m4.1.1b.cmml" xref="A1.p4.5.m4.1.1"><eq id="A1.p4.5.m4.1.1.4.cmml" xref="A1.p4.5.m4.1.1.4"></eq><apply id="A1.p4.5.m4.1.1.3.cmml" xref="A1.p4.5.m4.1.1.3"><csymbol cd="ambiguous" id="A1.p4.5.m4.1.1.3.1.cmml" xref="A1.p4.5.m4.1.1.3">superscript</csymbol><ci id="A1.p4.5.m4.1.1.3.2.cmml" xref="A1.p4.5.m4.1.1.3.2">𝑢</ci><ci id="A1.p4.5.m4.1.1.3.3.cmml" xref="A1.p4.5.m4.1.1.3.3">𝑇</ci></apply><apply id="A1.p4.5.m4.1.1.1.cmml" xref="A1.p4.5.m4.1.1.1"><csymbol cd="ambiguous" id="A1.p4.5.m4.1.1.1.2.cmml" xref="A1.p4.5.m4.1.1.1">superscript</csymbol><apply id="A1.p4.5.m4.1.1.1.1.1.1.cmml" xref="A1.p4.5.m4.1.1.1.1.1"><times id="A1.p4.5.m4.1.1.1.1.1.1.1.cmml" xref="A1.p4.5.m4.1.1.1.1.1.1.1"></times><apply id="A1.p4.5.m4.1.1.1.1.1.1.2.cmml" xref="A1.p4.5.m4.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A1.p4.5.m4.1.1.1.1.1.1.2.1.cmml" xref="A1.p4.5.m4.1.1.1.1.1.1.2">superscript</csymbol><ci id="A1.p4.5.m4.1.1.1.1.1.1.2.2.cmml" xref="A1.p4.5.m4.1.1.1.1.1.1.2.2">𝐶</ci><apply id="A1.p4.5.m4.1.1.1.1.1.1.2.3.cmml" xref="A1.p4.5.m4.1.1.1.1.1.1.2.3"><minus id="A1.p4.5.m4.1.1.1.1.1.1.2.3.1.cmml" xref="A1.p4.5.m4.1.1.1.1.1.1.2.3"></minus><cn type="integer" id="A1.p4.5.m4.1.1.1.1.1.1.2.3.2.cmml" xref="A1.p4.5.m4.1.1.1.1.1.1.2.3.2">1</cn></apply></apply><apply id="A1.p4.5.m4.1.1.1.1.1.1.3.cmml" xref="A1.p4.5.m4.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.p4.5.m4.1.1.1.1.1.1.3.1.cmml" xref="A1.p4.5.m4.1.1.1.1.1.1.3">subscript</csymbol><ci id="A1.p4.5.m4.1.1.1.1.1.1.3.2.cmml" xref="A1.p4.5.m4.1.1.1.1.1.1.3.2">𝑘</ci><times id="A1.p4.5.m4.1.1.1.1.1.1.3.3.cmml" xref="A1.p4.5.m4.1.1.1.1.1.1.3.3"></times></apply></apply><ci id="A1.p4.5.m4.1.1.1.3.cmml" xref="A1.p4.5.m4.1.1.1.3">𝑇</ci></apply></apply><apply id="A1.p4.5.m4.1.1c.cmml" xref="A1.p4.5.m4.1.1"><in id="A1.p4.5.m4.1.1.5.cmml" xref="A1.p4.5.m4.1.1.5"></in><share href="#A1.p4.5.m4.1.1.1.cmml" id="A1.p4.5.m4.1.1d.cmml" xref="A1.p4.5.m4.1.1"></share><apply id="A1.p4.5.m4.1.1.6.cmml" xref="A1.p4.5.m4.1.1.6"><csymbol cd="ambiguous" id="A1.p4.5.m4.1.1.6.1.cmml" xref="A1.p4.5.m4.1.1.6">superscript</csymbol><ci id="A1.p4.5.m4.1.1.6.2.cmml" xref="A1.p4.5.m4.1.1.6.2">ℝ</ci><ci id="A1.p4.5.m4.1.1.6.3.cmml" xref="A1.p4.5.m4.1.1.6.3">𝐷</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p4.5.m4.1c">u^{T}=(C^{-1}k_{*})^{T}\in\mathbb{R}^{D}</annotation></semantics></math>, so we can write simply (rearranging Eqn. <a href="#S3.E2" title="In 3.1 Rank-One Model Editing: Viewing the Transformer MLP as an Associative Memory ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> and Eqn. <a href="#A1.E13" title="In Appendix A Solving for Λ Algebraically ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">13</span></a>):</p>
<table id="A10.EGx9" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A1.E14"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A1.E14.m1.1" class="ltx_Math" alttext="\displaystyle\hat{W}I-\Lambda u^{T}" display="inline"><semantics id="A1.E14.m1.1a"><mrow id="A1.E14.m1.1.1" xref="A1.E14.m1.1.1.cmml"><mrow id="A1.E14.m1.1.1.2" xref="A1.E14.m1.1.1.2.cmml"><mover accent="true" id="A1.E14.m1.1.1.2.2" xref="A1.E14.m1.1.1.2.2.cmml"><mi id="A1.E14.m1.1.1.2.2.2" xref="A1.E14.m1.1.1.2.2.2.cmml">W</mi><mo id="A1.E14.m1.1.1.2.2.1" xref="A1.E14.m1.1.1.2.2.1.cmml">^</mo></mover><mo lspace="0em" rspace="0em" id="A1.E14.m1.1.1.2.1" xref="A1.E14.m1.1.1.2.1.cmml">​</mo><mi id="A1.E14.m1.1.1.2.3" xref="A1.E14.m1.1.1.2.3.cmml">I</mi></mrow><mo id="A1.E14.m1.1.1.1" xref="A1.E14.m1.1.1.1.cmml">−</mo><mrow id="A1.E14.m1.1.1.3" xref="A1.E14.m1.1.1.3.cmml"><mi mathvariant="normal" id="A1.E14.m1.1.1.3.2" xref="A1.E14.m1.1.1.3.2.cmml">Λ</mi><mo lspace="0em" rspace="0em" id="A1.E14.m1.1.1.3.1" xref="A1.E14.m1.1.1.3.1.cmml">​</mo><msup id="A1.E14.m1.1.1.3.3" xref="A1.E14.m1.1.1.3.3.cmml"><mi id="A1.E14.m1.1.1.3.3.2" xref="A1.E14.m1.1.1.3.3.2.cmml">u</mi><mi id="A1.E14.m1.1.1.3.3.3" xref="A1.E14.m1.1.1.3.3.3.cmml">T</mi></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E14.m1.1b"><apply id="A1.E14.m1.1.1.cmml" xref="A1.E14.m1.1.1"><minus id="A1.E14.m1.1.1.1.cmml" xref="A1.E14.m1.1.1.1"></minus><apply id="A1.E14.m1.1.1.2.cmml" xref="A1.E14.m1.1.1.2"><times id="A1.E14.m1.1.1.2.1.cmml" xref="A1.E14.m1.1.1.2.1"></times><apply id="A1.E14.m1.1.1.2.2.cmml" xref="A1.E14.m1.1.1.2.2"><ci id="A1.E14.m1.1.1.2.2.1.cmml" xref="A1.E14.m1.1.1.2.2.1">^</ci><ci id="A1.E14.m1.1.1.2.2.2.cmml" xref="A1.E14.m1.1.1.2.2.2">𝑊</ci></apply><ci id="A1.E14.m1.1.1.2.3.cmml" xref="A1.E14.m1.1.1.2.3">𝐼</ci></apply><apply id="A1.E14.m1.1.1.3.cmml" xref="A1.E14.m1.1.1.3"><times id="A1.E14.m1.1.1.3.1.cmml" xref="A1.E14.m1.1.1.3.1"></times><ci id="A1.E14.m1.1.1.3.2.cmml" xref="A1.E14.m1.1.1.3.2">Λ</ci><apply id="A1.E14.m1.1.1.3.3.cmml" xref="A1.E14.m1.1.1.3.3"><csymbol cd="ambiguous" id="A1.E14.m1.1.1.3.3.1.cmml" xref="A1.E14.m1.1.1.3.3">superscript</csymbol><ci id="A1.E14.m1.1.1.3.3.2.cmml" xref="A1.E14.m1.1.1.3.3.2">𝑢</ci><ci id="A1.E14.m1.1.1.3.3.3.cmml" xref="A1.E14.m1.1.1.3.3.3">𝑇</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E14.m1.1c">\displaystyle\hat{W}I-\Lambda u^{T}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A1.E14.m2.1" class="ltx_Math" alttext="\displaystyle=W" display="inline"><semantics id="A1.E14.m2.1a"><mrow id="A1.E14.m2.1.1" xref="A1.E14.m2.1.1.cmml"><mi id="A1.E14.m2.1.1.2" xref="A1.E14.m2.1.1.2.cmml"></mi><mo id="A1.E14.m2.1.1.1" xref="A1.E14.m2.1.1.1.cmml">=</mo><mi id="A1.E14.m2.1.1.3" xref="A1.E14.m2.1.1.3.cmml">W</mi></mrow><annotation-xml encoding="MathML-Content" id="A1.E14.m2.1b"><apply id="A1.E14.m2.1.1.cmml" xref="A1.E14.m2.1.1"><eq id="A1.E14.m2.1.1.1.cmml" xref="A1.E14.m2.1.1.1"></eq><csymbol cd="latexml" id="A1.E14.m2.1.1.2.cmml" xref="A1.E14.m2.1.1.2">absent</csymbol><ci id="A1.E14.m2.1.1.3.cmml" xref="A1.E14.m2.1.1.3">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E14.m2.1c">\displaystyle=W</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(14)</span></td>
</tr></tbody>
</table>
<p id="A1.p4.8" class="ltx_p">To solve for <math id="A1.p4.6.m1.1" class="ltx_Math" alttext="\Lambda" display="inline"><semantics id="A1.p4.6.m1.1a"><mi mathvariant="normal" id="A1.p4.6.m1.1.1" xref="A1.p4.6.m1.1.1.cmml">Λ</mi><annotation-xml encoding="MathML-Content" id="A1.p4.6.m1.1b"><ci id="A1.p4.6.m1.1.1.cmml" xref="A1.p4.6.m1.1.1">Λ</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p4.6.m1.1c">\Lambda</annotation></semantics></math>, we note that Eqn. <a href="#A1.E14" title="In Appendix A Solving for Λ Algebraically ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">14</span></a> and Eqn. <a href="#A1.E7" title="In Appendix A Solving for Λ Algebraically ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a> form a linear system that allows both <math id="A1.p4.7.m2.1" class="ltx_Math" alttext="\hat{W}" display="inline"><semantics id="A1.p4.7.m2.1a"><mover accent="true" id="A1.p4.7.m2.1.1" xref="A1.p4.7.m2.1.1.cmml"><mi id="A1.p4.7.m2.1.1.2" xref="A1.p4.7.m2.1.1.2.cmml">W</mi><mo id="A1.p4.7.m2.1.1.1" xref="A1.p4.7.m2.1.1.1.cmml">^</mo></mover><annotation-xml encoding="MathML-Content" id="A1.p4.7.m2.1b"><apply id="A1.p4.7.m2.1.1.cmml" xref="A1.p4.7.m2.1.1"><ci id="A1.p4.7.m2.1.1.1.cmml" xref="A1.p4.7.m2.1.1.1">^</ci><ci id="A1.p4.7.m2.1.1.2.cmml" xref="A1.p4.7.m2.1.1.2">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p4.7.m2.1c">\hat{W}</annotation></semantics></math> and <math id="A1.p4.8.m3.1" class="ltx_Math" alttext="\Lambda" display="inline"><semantics id="A1.p4.8.m3.1a"><mi mathvariant="normal" id="A1.p4.8.m3.1.1" xref="A1.p4.8.m3.1.1.cmml">Λ</mi><annotation-xml encoding="MathML-Content" id="A1.p4.8.m3.1b"><ci id="A1.p4.8.m3.1.1.cmml" xref="A1.p4.8.m3.1.1">Λ</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p4.8.m3.1c">\Lambda</annotation></semantics></math> to be solved simultaneously if written together in block form.</p>
<table id="A10.EGx10" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A1.E25"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A1.E25.m1.2" class="ltx_Math" alttext="\displaystyle\left[\begin{array}[]{@{}c|c@{}}\\
\quad\hat{W}&amp;\Lambda\\
\end{array}\right]\left[\begin{array}[]{@{}c|c@{}}\\
\quad\;I&amp;k_{*}\\
\\
\hline\cr-u^{T}\rule{0.0pt}{9.47217pt}&amp;0\end{array}\right]" display="inline"><semantics id="A1.E25.m1.2a"><mrow id="A1.E25.m1.2.3" xref="A1.E25.m1.2.3.cmml"><mrow id="A1.E25.m1.2.3.2.2" xref="A1.E25.m1.2.3.2.1.cmml"><mo id="A1.E25.m1.2.3.2.2.1" xref="A1.E25.m1.2.3.2.1.1.cmml">[</mo><mtable columnspacing="5pt" rowspacing="0pt" id="A1.E25.m1.1.1" xref="A1.E25.m1.1.1.cmml"><mtr id="A1.E25.m1.1.1a" xref="A1.E25.m1.1.1.cmml"><mtd class="ltx_border_r" id="A1.E25.m1.1.1b" xref="A1.E25.m1.1.1.cmml"></mtd><mtd id="A1.E25.m1.1.1c" xref="A1.E25.m1.1.1.cmml"></mtd></mtr><mtr id="A1.E25.m1.1.1d" xref="A1.E25.m1.1.1.cmml"><mtd class="ltx_border_r" id="A1.E25.m1.1.1e" xref="A1.E25.m1.1.1.cmml"><mover accent="true" id="A1.E16.1.1" xref="A1.E16.1.1.cmml"><mi id="A1.E16.1.1.2" xref="A1.E16.1.1.2.cmml">W</mi><mo id="A1.E16.1.1.1" xref="A1.E16.1.1.1.cmml">^</mo></mover></mtd><mtd id="A1.E25.m1.1.1f" xref="A1.E25.m1.1.1.cmml"><mi mathvariant="normal" id="A1.E16.2.1" xref="A1.E16.2.1.cmml">Λ</mi></mtd></mtr></mtable><mo id="A1.E25.m1.2.3.2.2.2" xref="A1.E25.m1.2.3.2.1.1.cmml">]</mo></mrow><mo lspace="0em" rspace="0em" id="A1.E25.m1.2.3.1" xref="A1.E25.m1.2.3.1.cmml">​</mo><mrow id="A1.E25.m1.2.3.3.2" xref="A1.E25.m1.2.3.3.1.cmml"><mo id="A1.E25.m1.2.3.3.2.1" xref="A1.E25.m1.2.3.3.1.1.cmml">[</mo><mtable columnspacing="5pt" rowspacing="0pt" id="A1.E25.m1.2.2" xref="A1.E25.m1.2.2.cmml"><mtr id="A1.E25.m1.2.2a" xref="A1.E25.m1.2.2.cmml"><mtd class="ltx_border_r" id="A1.E25.m1.2.2b" xref="A1.E25.m1.2.2.cmml"></mtd><mtd id="A1.E25.m1.2.2c" xref="A1.E25.m1.2.2.cmml"></mtd></mtr><mtr id="A1.E25.m1.2.2d" xref="A1.E25.m1.2.2.cmml"><mtd class="ltx_border_r" id="A1.E25.m1.2.2e" xref="A1.E25.m1.2.2.cmml"><mi id="A1.E19.1.1" xref="A1.E19.1.1.cmml">I</mi></mtd><mtd id="A1.E25.m1.2.2f" xref="A1.E25.m1.2.2.cmml"><msub id="A1.E19.2.1" xref="A1.E19.2.1.cmml"><mi id="A1.E19.2.1.2" xref="A1.E19.2.1.2.cmml">k</mi><mo id="A1.E19.2.1.3" xref="A1.E19.2.1.3.cmml">∗</mo></msub></mtd></mtr><mtr id="A1.E25.m1.2.2g" xref="A1.E25.m1.2.2.cmml"><mtd class="ltx_border_r" id="A1.E25.m1.2.2h" xref="A1.E25.m1.2.2.cmml"></mtd><mtd id="A1.E25.m1.2.2i" xref="A1.E25.m1.2.2.cmml"></mtd></mtr><mtr id="A1.E25.m1.2.2j" xref="A1.E25.m1.2.2.cmml"><mtd class="ltx_border_t" id="A1.E25.m1.2.2k" xref="A1.E25.m1.2.2.cmml"></mtd><mtd class="ltx_border_t" id="A1.E25.m1.2.2l" xref="A1.E25.m1.2.2.cmml"></mtd></mtr><mtr id="A1.E25.m1.2.2m" xref="A1.E25.m1.2.2.cmml"><mtd class="ltx_border_r" id="A1.E25.m1.2.2n" xref="A1.E25.m1.2.2.cmml"><mrow id="A1.E21.1.1" xref="A1.E21.1.1.cmml"><mo id="A1.E21.1.1a" xref="A1.E21.1.1.cmml">−</mo><mrow id="A1.E21.1.1.2" xref="A1.E21.1.1.2.cmml"><msup id="A1.E21.1.1.2.2" xref="A1.E21.1.1.2.2.cmml"><mi id="A1.E21.1.1.2.2.2" xref="A1.E21.1.1.2.2.2.cmml">u</mi><mi id="A1.E21.1.1.2.2.3" xref="A1.E21.1.1.2.2.3.cmml">T</mi></msup><mo lspace="0em" rspace="0em" id="A1.E21.1.1.2.1" xref="A1.E21.1.1.2.1.cmml">​</mo><mtext id="A1.E21.1.1.2.3" xref="A1.E21.1.1.2.3b.cmml"><span class="ltx_rule" style="width:0.0pt;height:9.5pt;background:black;display:inline-block;"></span></mtext></mrow></mrow></mtd><mtd id="A1.E25.m1.2.2o" xref="A1.E25.m1.2.2.cmml"><mn id="A1.E21.2.1" xref="A1.E21.2.1.cmml">0</mn></mtd></mtr></mtable><mo id="A1.E25.m1.2.3.3.2.2" xref="A1.E25.m1.2.3.3.1.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E25.m1.2b"><apply id="A1.E25.m1.2.3.cmml" xref="A1.E25.m1.2.3"><times id="A1.E25.m1.2.3.1.cmml" xref="A1.E25.m1.2.3.1"></times><apply id="A1.E25.m1.2.3.2.1.cmml" xref="A1.E25.m1.2.3.2.2"><csymbol cd="latexml" id="A1.E25.m1.2.3.2.1.1.cmml" xref="A1.E25.m1.2.3.2.2.1">delimited-[]</csymbol><matrix id="A1.E25.m1.1.1.cmml" xref="A1.E25.m1.1.1"><matrixrow id="A1.E25.m1.1.1a.cmml" xref="A1.E25.m1.1.1"><cerror id="A1.E25.m1.1.1b.cmml" xref="A1.E25.m1.1.1"><csymbol cd="ambiguous" id="A1.E25.m1.1.1c.cmml" xref="A1.E25.m1.1.1">missing-subexpression</csymbol></cerror><cerror id="A1.E25.m1.1.1d.cmml" xref="A1.E25.m1.1.1"><csymbol cd="ambiguous" id="A1.E25.m1.1.1e.cmml" xref="A1.E25.m1.1.1">missing-subexpression</csymbol></cerror></matrixrow><matrixrow id="A1.E25.m1.1.1f.cmml" xref="A1.E25.m1.1.1"><apply id="A1.E16.1.1.cmml" xref="A1.E16.1.1"><ci id="A1.E16.1.1.1.cmml" xref="A1.E16.1.1.1">^</ci><ci id="A1.E16.1.1.2.cmml" xref="A1.E16.1.1.2">𝑊</ci></apply><ci id="A1.E16.2.1.cmml" xref="A1.E16.2.1">Λ</ci></matrixrow></matrix></apply><apply id="A1.E25.m1.2.3.3.1.cmml" xref="A1.E25.m1.2.3.3.2"><csymbol cd="latexml" id="A1.E25.m1.2.3.3.1.1.cmml" xref="A1.E25.m1.2.3.3.2.1">delimited-[]</csymbol><matrix id="A1.E25.m1.2.2.cmml" xref="A1.E25.m1.2.2"><matrixrow id="A1.E25.m1.2.2a.cmml" xref="A1.E25.m1.2.2"><cerror id="A1.E25.m1.2.2b.cmml" xref="A1.E25.m1.2.2"><csymbol cd="ambiguous" id="A1.E25.m1.2.2c.cmml" xref="A1.E25.m1.2.2">missing-subexpression</csymbol></cerror><cerror id="A1.E25.m1.2.2d.cmml" xref="A1.E25.m1.2.2"><csymbol cd="ambiguous" id="A1.E25.m1.2.2e.cmml" xref="A1.E25.m1.2.2">missing-subexpression</csymbol></cerror></matrixrow><matrixrow id="A1.E25.m1.2.2f.cmml" xref="A1.E25.m1.2.2"><ci id="A1.E19.1.1.cmml" xref="A1.E19.1.1">𝐼</ci><apply id="A1.E19.2.1.cmml" xref="A1.E19.2.1"><csymbol cd="ambiguous" id="A1.E19.2.1.1.cmml" xref="A1.E19.2.1">subscript</csymbol><ci id="A1.E19.2.1.2.cmml" xref="A1.E19.2.1.2">𝑘</ci><times id="A1.E19.2.1.3.cmml" xref="A1.E19.2.1.3"></times></apply></matrixrow><matrixrow id="A1.E25.m1.2.2g.cmml" xref="A1.E25.m1.2.2"><cerror id="A1.E25.m1.2.2h.cmml" xref="A1.E25.m1.2.2"><csymbol cd="ambiguous" id="A1.E25.m1.2.2i.cmml" xref="A1.E25.m1.2.2">missing-subexpression</csymbol></cerror><cerror id="A1.E25.m1.2.2j.cmml" xref="A1.E25.m1.2.2"><csymbol cd="ambiguous" id="A1.E25.m1.2.2k.cmml" xref="A1.E25.m1.2.2">missing-subexpression</csymbol></cerror></matrixrow><matrixrow id="A1.E25.m1.2.2l.cmml" xref="A1.E25.m1.2.2"><cerror id="A1.E25.m1.2.2m.cmml" xref="A1.E25.m1.2.2"><csymbol cd="ambiguous" id="A1.E25.m1.2.2n.cmml" xref="A1.E25.m1.2.2">missing-subexpression</csymbol></cerror><cerror id="A1.E25.m1.2.2o.cmml" xref="A1.E25.m1.2.2"><csymbol cd="ambiguous" id="A1.E25.m1.2.2p.cmml" xref="A1.E25.m1.2.2">missing-subexpression</csymbol></cerror></matrixrow><matrixrow id="A1.E25.m1.2.2q.cmml" xref="A1.E25.m1.2.2"><apply id="A1.E21.1.1.cmml" xref="A1.E21.1.1"><minus id="A1.E21.1.1.1.cmml" xref="A1.E21.1.1"></minus><apply id="A1.E21.1.1.2.cmml" xref="A1.E21.1.1.2"><times id="A1.E21.1.1.2.1.cmml" xref="A1.E21.1.1.2.1"></times><apply id="A1.E21.1.1.2.2.cmml" xref="A1.E21.1.1.2.2"><csymbol cd="ambiguous" id="A1.E21.1.1.2.2.1.cmml" xref="A1.E21.1.1.2.2">superscript</csymbol><ci id="A1.E21.1.1.2.2.2.cmml" xref="A1.E21.1.1.2.2.2">𝑢</ci><ci id="A1.E21.1.1.2.2.3.cmml" xref="A1.E21.1.1.2.2.3">𝑇</ci></apply><ci id="A1.E21.1.1.2.3b.cmml" xref="A1.E21.1.1.2.3"><mtext id="A1.E21.1.1.2.3.cmml" xref="A1.E21.1.1.2.3"><span class="ltx_rule" style="width:0.0pt;height:9.5pt;background:black;display:inline-block;"></span></mtext></ci></apply></apply><cn type="integer" id="A1.E21.2.1.cmml" xref="A1.E21.2.1">0</cn></matrixrow></matrix></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E25.m1.2c">\displaystyle\left[\begin{array}[]{@{}c|c@{}}\\
\quad\hat{W}&amp;\Lambda\\
\end{array}\right]\left[\begin{array}[]{@{}c|c@{}}\\
\quad\;I&amp;k_{*}\\
\\
\hline\cr-u^{T}\rule{0.0pt}{9.47217pt}&amp;0\end{array}\right]</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A1.E25.m2.1" class="ltx_Math" alttext="\displaystyle=\left[\begin{array}[]{@{}c|c@{}}\mathstrut\\
\quad W&amp;v_{*}\\
\mathstrut\end{array}\right]" display="inline"><semantics id="A1.E25.m2.1a"><mrow id="A1.E25.m2.1.2" xref="A1.E25.m2.1.2.cmml"><mi id="A1.E25.m2.1.2.2" xref="A1.E25.m2.1.2.2.cmml"></mi><mo id="A1.E25.m2.1.2.1" xref="A1.E25.m2.1.2.1.cmml">=</mo><mrow id="A1.E25.m2.1.2.3.2" xref="A1.E25.m2.1.2.3.1.cmml"><mo id="A1.E25.m2.1.2.3.2.1" xref="A1.E25.m2.1.2.3.1.1.cmml">[</mo><mtable columnspacing="5pt" rowspacing="0pt" id="A1.E25.m2.1.1" xref="A1.E25.m2.1.1.cmml"><mtr id="A1.E25.m2.1.1a" xref="A1.E25.m2.1.1.cmml"><mtd class="ltx_border_r" id="A1.E25.m2.1.1b" xref="A1.E25.m2.1.1.cmml"></mtd><mtd id="A1.E25.m2.1.1c" xref="A1.E25.m2.1.1.cmml"></mtd></mtr><mtr id="A1.E25.m2.1.1d" xref="A1.E25.m2.1.1.cmml"><mtd class="ltx_border_r" id="A1.E25.m2.1.1e" xref="A1.E25.m2.1.1.cmml"><mi id="A1.E23.1.1" xref="A1.E23.1.1.cmml">W</mi></mtd><mtd id="A1.E25.m2.1.1f" xref="A1.E25.m2.1.1.cmml"><msub id="A1.E23.2.1" xref="A1.E23.2.1.cmml"><mi id="A1.E23.2.1.2" xref="A1.E23.2.1.2.cmml">v</mi><mo id="A1.E23.2.1.3" xref="A1.E23.2.1.3.cmml">∗</mo></msub></mtd></mtr></mtable><mo id="A1.E25.m2.1.2.3.2.2" xref="A1.E25.m2.1.2.3.1.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E25.m2.1b"><apply id="A1.E25.m2.1.2.cmml" xref="A1.E25.m2.1.2"><eq id="A1.E25.m2.1.2.1.cmml" xref="A1.E25.m2.1.2.1"></eq><csymbol cd="latexml" id="A1.E25.m2.1.2.2.cmml" xref="A1.E25.m2.1.2.2">absent</csymbol><apply id="A1.E25.m2.1.2.3.1.cmml" xref="A1.E25.m2.1.2.3.2"><csymbol cd="latexml" id="A1.E25.m2.1.2.3.1.1.cmml" xref="A1.E25.m2.1.2.3.2.1">delimited-[]</csymbol><matrix id="A1.E25.m2.1.1.cmml" xref="A1.E25.m2.1.1"><matrixrow id="A1.E25.m2.1.1a.cmml" xref="A1.E25.m2.1.1"><cerror id="A1.E25.m2.1.1b.cmml" xref="A1.E25.m2.1.1"><csymbol cd="ambiguous" id="A1.E25.m2.1.1c.cmml" xref="A1.E25.m2.1.1">missing-subexpression</csymbol></cerror><cerror id="A1.E25.m2.1.1d.cmml" xref="A1.E25.m2.1.1"><csymbol cd="ambiguous" id="A1.E25.m2.1.1e.cmml" xref="A1.E25.m2.1.1">missing-subexpression</csymbol></cerror></matrixrow><matrixrow id="A1.E25.m2.1.1f.cmml" xref="A1.E25.m2.1.1"><ci id="A1.E23.1.1.cmml" xref="A1.E23.1.1">𝑊</ci><apply id="A1.E23.2.1.cmml" xref="A1.E23.2.1"><csymbol cd="ambiguous" id="A1.E23.2.1.1.cmml" xref="A1.E23.2.1">subscript</csymbol><ci id="A1.E23.2.1.2.cmml" xref="A1.E23.2.1.2">𝑣</ci><times id="A1.E23.2.1.3.cmml" xref="A1.E23.2.1.3"></times></apply></matrixrow></matrix></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E25.m2.1c">\displaystyle=\left[\begin{array}[]{@{}c|c@{}}\mathstrut\\
\quad W&amp;v_{*}\\
\mathstrut\end{array}\right]</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(25)</span></td>
</tr></tbody>
</table>
<p id="A1.p4.10" class="ltx_p">That is equivalent to substituting Eqn. <a href="#A1.E13" title="In Appendix A Solving for Λ Algebraically ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">13</span></a> into Eqn. <a href="#A1.E7" title="In Appendix A Solving for Λ Algebraically ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a> and calculating the following:</p>
<table id="A10.EGx11" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A1.E26"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A1.E26.m1.1" class="ltx_Math" alttext="\displaystyle\hat{W}k_{*}" display="inline"><semantics id="A1.E26.m1.1a"><mrow id="A1.E26.m1.1.1" xref="A1.E26.m1.1.1.cmml"><mover accent="true" id="A1.E26.m1.1.1.2" xref="A1.E26.m1.1.1.2.cmml"><mi id="A1.E26.m1.1.1.2.2" xref="A1.E26.m1.1.1.2.2.cmml">W</mi><mo id="A1.E26.m1.1.1.2.1" xref="A1.E26.m1.1.1.2.1.cmml">^</mo></mover><mo lspace="0em" rspace="0em" id="A1.E26.m1.1.1.1" xref="A1.E26.m1.1.1.1.cmml">​</mo><msub id="A1.E26.m1.1.1.3" xref="A1.E26.m1.1.1.3.cmml"><mi id="A1.E26.m1.1.1.3.2" xref="A1.E26.m1.1.1.3.2.cmml">k</mi><mo id="A1.E26.m1.1.1.3.3" xref="A1.E26.m1.1.1.3.3.cmml">∗</mo></msub></mrow><annotation-xml encoding="MathML-Content" id="A1.E26.m1.1b"><apply id="A1.E26.m1.1.1.cmml" xref="A1.E26.m1.1.1"><times id="A1.E26.m1.1.1.1.cmml" xref="A1.E26.m1.1.1.1"></times><apply id="A1.E26.m1.1.1.2.cmml" xref="A1.E26.m1.1.1.2"><ci id="A1.E26.m1.1.1.2.1.cmml" xref="A1.E26.m1.1.1.2.1">^</ci><ci id="A1.E26.m1.1.1.2.2.cmml" xref="A1.E26.m1.1.1.2.2">𝑊</ci></apply><apply id="A1.E26.m1.1.1.3.cmml" xref="A1.E26.m1.1.1.3"><csymbol cd="ambiguous" id="A1.E26.m1.1.1.3.1.cmml" xref="A1.E26.m1.1.1.3">subscript</csymbol><ci id="A1.E26.m1.1.1.3.2.cmml" xref="A1.E26.m1.1.1.3.2">𝑘</ci><times id="A1.E26.m1.1.1.3.3.cmml" xref="A1.E26.m1.1.1.3.3"></times></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E26.m1.1c">\displaystyle\hat{W}k_{*}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A1.E26.m2.2" class="ltx_Math" alttext="\displaystyle=(W+\Lambda u^{T})k_{*}=Wk_{*}+\Lambda(u^{T}k_{*})=v_{*}" display="inline"><semantics id="A1.E26.m2.2a"><mrow id="A1.E26.m2.2.2" xref="A1.E26.m2.2.2.cmml"><mi id="A1.E26.m2.2.2.4" xref="A1.E26.m2.2.2.4.cmml"></mi><mo id="A1.E26.m2.2.2.5" xref="A1.E26.m2.2.2.5.cmml">=</mo><mrow id="A1.E26.m2.1.1.1" xref="A1.E26.m2.1.1.1.cmml"><mrow id="A1.E26.m2.1.1.1.1.1" xref="A1.E26.m2.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.E26.m2.1.1.1.1.1.2" xref="A1.E26.m2.1.1.1.1.1.1.cmml">(</mo><mrow id="A1.E26.m2.1.1.1.1.1.1" xref="A1.E26.m2.1.1.1.1.1.1.cmml"><mi id="A1.E26.m2.1.1.1.1.1.1.2" xref="A1.E26.m2.1.1.1.1.1.1.2.cmml">W</mi><mo id="A1.E26.m2.1.1.1.1.1.1.1" xref="A1.E26.m2.1.1.1.1.1.1.1.cmml">+</mo><mrow id="A1.E26.m2.1.1.1.1.1.1.3" xref="A1.E26.m2.1.1.1.1.1.1.3.cmml"><mi mathvariant="normal" id="A1.E26.m2.1.1.1.1.1.1.3.2" xref="A1.E26.m2.1.1.1.1.1.1.3.2.cmml">Λ</mi><mo lspace="0em" rspace="0em" id="A1.E26.m2.1.1.1.1.1.1.3.1" xref="A1.E26.m2.1.1.1.1.1.1.3.1.cmml">​</mo><msup id="A1.E26.m2.1.1.1.1.1.1.3.3" xref="A1.E26.m2.1.1.1.1.1.1.3.3.cmml"><mi id="A1.E26.m2.1.1.1.1.1.1.3.3.2" xref="A1.E26.m2.1.1.1.1.1.1.3.3.2.cmml">u</mi><mi id="A1.E26.m2.1.1.1.1.1.1.3.3.3" xref="A1.E26.m2.1.1.1.1.1.1.3.3.3.cmml">T</mi></msup></mrow></mrow><mo stretchy="false" id="A1.E26.m2.1.1.1.1.1.3" xref="A1.E26.m2.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A1.E26.m2.1.1.1.2" xref="A1.E26.m2.1.1.1.2.cmml">​</mo><msub id="A1.E26.m2.1.1.1.3" xref="A1.E26.m2.1.1.1.3.cmml"><mi id="A1.E26.m2.1.1.1.3.2" xref="A1.E26.m2.1.1.1.3.2.cmml">k</mi><mo id="A1.E26.m2.1.1.1.3.3" xref="A1.E26.m2.1.1.1.3.3.cmml">∗</mo></msub></mrow><mo id="A1.E26.m2.2.2.6" xref="A1.E26.m2.2.2.6.cmml">=</mo><mrow id="A1.E26.m2.2.2.2" xref="A1.E26.m2.2.2.2.cmml"><mrow id="A1.E26.m2.2.2.2.3" xref="A1.E26.m2.2.2.2.3.cmml"><mi id="A1.E26.m2.2.2.2.3.2" xref="A1.E26.m2.2.2.2.3.2.cmml">W</mi><mo lspace="0em" rspace="0em" id="A1.E26.m2.2.2.2.3.1" xref="A1.E26.m2.2.2.2.3.1.cmml">​</mo><msub id="A1.E26.m2.2.2.2.3.3" xref="A1.E26.m2.2.2.2.3.3.cmml"><mi id="A1.E26.m2.2.2.2.3.3.2" xref="A1.E26.m2.2.2.2.3.3.2.cmml">k</mi><mo id="A1.E26.m2.2.2.2.3.3.3" xref="A1.E26.m2.2.2.2.3.3.3.cmml">∗</mo></msub></mrow><mo id="A1.E26.m2.2.2.2.2" xref="A1.E26.m2.2.2.2.2.cmml">+</mo><mrow id="A1.E26.m2.2.2.2.1" xref="A1.E26.m2.2.2.2.1.cmml"><mi mathvariant="normal" id="A1.E26.m2.2.2.2.1.3" xref="A1.E26.m2.2.2.2.1.3.cmml">Λ</mi><mo lspace="0em" rspace="0em" id="A1.E26.m2.2.2.2.1.2" xref="A1.E26.m2.2.2.2.1.2.cmml">​</mo><mrow id="A1.E26.m2.2.2.2.1.1.1" xref="A1.E26.m2.2.2.2.1.1.1.1.cmml"><mo stretchy="false" id="A1.E26.m2.2.2.2.1.1.1.2" xref="A1.E26.m2.2.2.2.1.1.1.1.cmml">(</mo><mrow id="A1.E26.m2.2.2.2.1.1.1.1" xref="A1.E26.m2.2.2.2.1.1.1.1.cmml"><msup id="A1.E26.m2.2.2.2.1.1.1.1.2" xref="A1.E26.m2.2.2.2.1.1.1.1.2.cmml"><mi id="A1.E26.m2.2.2.2.1.1.1.1.2.2" xref="A1.E26.m2.2.2.2.1.1.1.1.2.2.cmml">u</mi><mi id="A1.E26.m2.2.2.2.1.1.1.1.2.3" xref="A1.E26.m2.2.2.2.1.1.1.1.2.3.cmml">T</mi></msup><mo lspace="0em" rspace="0em" id="A1.E26.m2.2.2.2.1.1.1.1.1" xref="A1.E26.m2.2.2.2.1.1.1.1.1.cmml">​</mo><msub id="A1.E26.m2.2.2.2.1.1.1.1.3" xref="A1.E26.m2.2.2.2.1.1.1.1.3.cmml"><mi id="A1.E26.m2.2.2.2.1.1.1.1.3.2" xref="A1.E26.m2.2.2.2.1.1.1.1.3.2.cmml">k</mi><mo id="A1.E26.m2.2.2.2.1.1.1.1.3.3" xref="A1.E26.m2.2.2.2.1.1.1.1.3.3.cmml">∗</mo></msub></mrow><mo stretchy="false" id="A1.E26.m2.2.2.2.1.1.1.3" xref="A1.E26.m2.2.2.2.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo id="A1.E26.m2.2.2.7" xref="A1.E26.m2.2.2.7.cmml">=</mo><msub id="A1.E26.m2.2.2.8" xref="A1.E26.m2.2.2.8.cmml"><mi id="A1.E26.m2.2.2.8.2" xref="A1.E26.m2.2.2.8.2.cmml">v</mi><mo id="A1.E26.m2.2.2.8.3" xref="A1.E26.m2.2.2.8.3.cmml">∗</mo></msub></mrow><annotation-xml encoding="MathML-Content" id="A1.E26.m2.2b"><apply id="A1.E26.m2.2.2.cmml" xref="A1.E26.m2.2.2"><and id="A1.E26.m2.2.2a.cmml" xref="A1.E26.m2.2.2"></and><apply id="A1.E26.m2.2.2b.cmml" xref="A1.E26.m2.2.2"><eq id="A1.E26.m2.2.2.5.cmml" xref="A1.E26.m2.2.2.5"></eq><csymbol cd="latexml" id="A1.E26.m2.2.2.4.cmml" xref="A1.E26.m2.2.2.4">absent</csymbol><apply id="A1.E26.m2.1.1.1.cmml" xref="A1.E26.m2.1.1.1"><times id="A1.E26.m2.1.1.1.2.cmml" xref="A1.E26.m2.1.1.1.2"></times><apply id="A1.E26.m2.1.1.1.1.1.1.cmml" xref="A1.E26.m2.1.1.1.1.1"><plus id="A1.E26.m2.1.1.1.1.1.1.1.cmml" xref="A1.E26.m2.1.1.1.1.1.1.1"></plus><ci id="A1.E26.m2.1.1.1.1.1.1.2.cmml" xref="A1.E26.m2.1.1.1.1.1.1.2">𝑊</ci><apply id="A1.E26.m2.1.1.1.1.1.1.3.cmml" xref="A1.E26.m2.1.1.1.1.1.1.3"><times id="A1.E26.m2.1.1.1.1.1.1.3.1.cmml" xref="A1.E26.m2.1.1.1.1.1.1.3.1"></times><ci id="A1.E26.m2.1.1.1.1.1.1.3.2.cmml" xref="A1.E26.m2.1.1.1.1.1.1.3.2">Λ</ci><apply id="A1.E26.m2.1.1.1.1.1.1.3.3.cmml" xref="A1.E26.m2.1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="A1.E26.m2.1.1.1.1.1.1.3.3.1.cmml" xref="A1.E26.m2.1.1.1.1.1.1.3.3">superscript</csymbol><ci id="A1.E26.m2.1.1.1.1.1.1.3.3.2.cmml" xref="A1.E26.m2.1.1.1.1.1.1.3.3.2">𝑢</ci><ci id="A1.E26.m2.1.1.1.1.1.1.3.3.3.cmml" xref="A1.E26.m2.1.1.1.1.1.1.3.3.3">𝑇</ci></apply></apply></apply><apply id="A1.E26.m2.1.1.1.3.cmml" xref="A1.E26.m2.1.1.1.3"><csymbol cd="ambiguous" id="A1.E26.m2.1.1.1.3.1.cmml" xref="A1.E26.m2.1.1.1.3">subscript</csymbol><ci id="A1.E26.m2.1.1.1.3.2.cmml" xref="A1.E26.m2.1.1.1.3.2">𝑘</ci><times id="A1.E26.m2.1.1.1.3.3.cmml" xref="A1.E26.m2.1.1.1.3.3"></times></apply></apply></apply><apply id="A1.E26.m2.2.2c.cmml" xref="A1.E26.m2.2.2"><eq id="A1.E26.m2.2.2.6.cmml" xref="A1.E26.m2.2.2.6"></eq><share href="#A1.E26.m2.1.1.1.cmml" id="A1.E26.m2.2.2d.cmml" xref="A1.E26.m2.2.2"></share><apply id="A1.E26.m2.2.2.2.cmml" xref="A1.E26.m2.2.2.2"><plus id="A1.E26.m2.2.2.2.2.cmml" xref="A1.E26.m2.2.2.2.2"></plus><apply id="A1.E26.m2.2.2.2.3.cmml" xref="A1.E26.m2.2.2.2.3"><times id="A1.E26.m2.2.2.2.3.1.cmml" xref="A1.E26.m2.2.2.2.3.1"></times><ci id="A1.E26.m2.2.2.2.3.2.cmml" xref="A1.E26.m2.2.2.2.3.2">𝑊</ci><apply id="A1.E26.m2.2.2.2.3.3.cmml" xref="A1.E26.m2.2.2.2.3.3"><csymbol cd="ambiguous" id="A1.E26.m2.2.2.2.3.3.1.cmml" xref="A1.E26.m2.2.2.2.3.3">subscript</csymbol><ci id="A1.E26.m2.2.2.2.3.3.2.cmml" xref="A1.E26.m2.2.2.2.3.3.2">𝑘</ci><times id="A1.E26.m2.2.2.2.3.3.3.cmml" xref="A1.E26.m2.2.2.2.3.3.3"></times></apply></apply><apply id="A1.E26.m2.2.2.2.1.cmml" xref="A1.E26.m2.2.2.2.1"><times id="A1.E26.m2.2.2.2.1.2.cmml" xref="A1.E26.m2.2.2.2.1.2"></times><ci id="A1.E26.m2.2.2.2.1.3.cmml" xref="A1.E26.m2.2.2.2.1.3">Λ</ci><apply id="A1.E26.m2.2.2.2.1.1.1.1.cmml" xref="A1.E26.m2.2.2.2.1.1.1"><times id="A1.E26.m2.2.2.2.1.1.1.1.1.cmml" xref="A1.E26.m2.2.2.2.1.1.1.1.1"></times><apply id="A1.E26.m2.2.2.2.1.1.1.1.2.cmml" xref="A1.E26.m2.2.2.2.1.1.1.1.2"><csymbol cd="ambiguous" id="A1.E26.m2.2.2.2.1.1.1.1.2.1.cmml" xref="A1.E26.m2.2.2.2.1.1.1.1.2">superscript</csymbol><ci id="A1.E26.m2.2.2.2.1.1.1.1.2.2.cmml" xref="A1.E26.m2.2.2.2.1.1.1.1.2.2">𝑢</ci><ci id="A1.E26.m2.2.2.2.1.1.1.1.2.3.cmml" xref="A1.E26.m2.2.2.2.1.1.1.1.2.3">𝑇</ci></apply><apply id="A1.E26.m2.2.2.2.1.1.1.1.3.cmml" xref="A1.E26.m2.2.2.2.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.E26.m2.2.2.2.1.1.1.1.3.1.cmml" xref="A1.E26.m2.2.2.2.1.1.1.1.3">subscript</csymbol><ci id="A1.E26.m2.2.2.2.1.1.1.1.3.2.cmml" xref="A1.E26.m2.2.2.2.1.1.1.1.3.2">𝑘</ci><times id="A1.E26.m2.2.2.2.1.1.1.1.3.3.cmml" xref="A1.E26.m2.2.2.2.1.1.1.1.3.3"></times></apply></apply></apply></apply></apply><apply id="A1.E26.m2.2.2e.cmml" xref="A1.E26.m2.2.2"><eq id="A1.E26.m2.2.2.7.cmml" xref="A1.E26.m2.2.2.7"></eq><share href="#A1.E26.m2.2.2.2.cmml" id="A1.E26.m2.2.2f.cmml" xref="A1.E26.m2.2.2"></share><apply id="A1.E26.m2.2.2.8.cmml" xref="A1.E26.m2.2.2.8"><csymbol cd="ambiguous" id="A1.E26.m2.2.2.8.1.cmml" xref="A1.E26.m2.2.2.8">subscript</csymbol><ci id="A1.E26.m2.2.2.8.2.cmml" xref="A1.E26.m2.2.2.8.2">𝑣</ci><times id="A1.E26.m2.2.2.8.3.cmml" xref="A1.E26.m2.2.2.8.3"></times></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E26.m2.2c">\displaystyle=(W+\Lambda u^{T})k_{*}=Wk_{*}+\Lambda(u^{T}k_{*})=v_{*}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(26)</span></td>
</tr></tbody>
<tbody id="A1.E27"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A1.E27.m1.1" class="ltx_Math" alttext="\displaystyle\Lambda" display="inline"><semantics id="A1.E27.m1.1a"><mi mathvariant="normal" id="A1.E27.m1.1.1" xref="A1.E27.m1.1.1.cmml">Λ</mi><annotation-xml encoding="MathML-Content" id="A1.E27.m1.1b"><ci id="A1.E27.m1.1.1.cmml" xref="A1.E27.m1.1.1">Λ</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.E27.m1.1c">\displaystyle\Lambda</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A1.E27.m2.1" class="ltx_Math" alttext="\displaystyle=\frac{v_{*}-Wk_{*}}{u^{T}k_{*}}=\frac{v_{*}-Wk_{*}}{(C^{-1}k_{*})^{T}k_{*}}" display="inline"><semantics id="A1.E27.m2.1a"><mrow id="A1.E27.m2.1.2" xref="A1.E27.m2.1.2.cmml"><mi id="A1.E27.m2.1.2.2" xref="A1.E27.m2.1.2.2.cmml"></mi><mo id="A1.E27.m2.1.2.3" xref="A1.E27.m2.1.2.3.cmml">=</mo><mstyle displaystyle="true" id="A1.E27.m2.1.2.4" xref="A1.E27.m2.1.2.4.cmml"><mfrac id="A1.E27.m2.1.2.4a" xref="A1.E27.m2.1.2.4.cmml"><mrow id="A1.E27.m2.1.2.4.2" xref="A1.E27.m2.1.2.4.2.cmml"><msub id="A1.E27.m2.1.2.4.2.2" xref="A1.E27.m2.1.2.4.2.2.cmml"><mi id="A1.E27.m2.1.2.4.2.2.2" xref="A1.E27.m2.1.2.4.2.2.2.cmml">v</mi><mo id="A1.E27.m2.1.2.4.2.2.3" xref="A1.E27.m2.1.2.4.2.2.3.cmml">∗</mo></msub><mo id="A1.E27.m2.1.2.4.2.1" xref="A1.E27.m2.1.2.4.2.1.cmml">−</mo><mrow id="A1.E27.m2.1.2.4.2.3" xref="A1.E27.m2.1.2.4.2.3.cmml"><mi id="A1.E27.m2.1.2.4.2.3.2" xref="A1.E27.m2.1.2.4.2.3.2.cmml">W</mi><mo lspace="0em" rspace="0em" id="A1.E27.m2.1.2.4.2.3.1" xref="A1.E27.m2.1.2.4.2.3.1.cmml">​</mo><msub id="A1.E27.m2.1.2.4.2.3.3" xref="A1.E27.m2.1.2.4.2.3.3.cmml"><mi id="A1.E27.m2.1.2.4.2.3.3.2" xref="A1.E27.m2.1.2.4.2.3.3.2.cmml">k</mi><mo id="A1.E27.m2.1.2.4.2.3.3.3" xref="A1.E27.m2.1.2.4.2.3.3.3.cmml">∗</mo></msub></mrow></mrow><mrow id="A1.E27.m2.1.2.4.3" xref="A1.E27.m2.1.2.4.3.cmml"><msup id="A1.E27.m2.1.2.4.3.2" xref="A1.E27.m2.1.2.4.3.2.cmml"><mi id="A1.E27.m2.1.2.4.3.2.2" xref="A1.E27.m2.1.2.4.3.2.2.cmml">u</mi><mi id="A1.E27.m2.1.2.4.3.2.3" xref="A1.E27.m2.1.2.4.3.2.3.cmml">T</mi></msup><mo lspace="0em" rspace="0em" id="A1.E27.m2.1.2.4.3.1" xref="A1.E27.m2.1.2.4.3.1.cmml">​</mo><msub id="A1.E27.m2.1.2.4.3.3" xref="A1.E27.m2.1.2.4.3.3.cmml"><mi id="A1.E27.m2.1.2.4.3.3.2" xref="A1.E27.m2.1.2.4.3.3.2.cmml">k</mi><mo id="A1.E27.m2.1.2.4.3.3.3" xref="A1.E27.m2.1.2.4.3.3.3.cmml">∗</mo></msub></mrow></mfrac></mstyle><mo id="A1.E27.m2.1.2.5" xref="A1.E27.m2.1.2.5.cmml">=</mo><mstyle displaystyle="true" id="A1.E27.m2.1.1" xref="A1.E27.m2.1.1.cmml"><mfrac id="A1.E27.m2.1.1a" xref="A1.E27.m2.1.1.cmml"><mrow id="A1.E27.m2.1.1.3" xref="A1.E27.m2.1.1.3.cmml"><msub id="A1.E27.m2.1.1.3.2" xref="A1.E27.m2.1.1.3.2.cmml"><mi id="A1.E27.m2.1.1.3.2.2" xref="A1.E27.m2.1.1.3.2.2.cmml">v</mi><mo id="A1.E27.m2.1.1.3.2.3" xref="A1.E27.m2.1.1.3.2.3.cmml">∗</mo></msub><mo id="A1.E27.m2.1.1.3.1" xref="A1.E27.m2.1.1.3.1.cmml">−</mo><mrow id="A1.E27.m2.1.1.3.3" xref="A1.E27.m2.1.1.3.3.cmml"><mi id="A1.E27.m2.1.1.3.3.2" xref="A1.E27.m2.1.1.3.3.2.cmml">W</mi><mo lspace="0em" rspace="0em" id="A1.E27.m2.1.1.3.3.1" xref="A1.E27.m2.1.1.3.3.1.cmml">​</mo><msub id="A1.E27.m2.1.1.3.3.3" xref="A1.E27.m2.1.1.3.3.3.cmml"><mi id="A1.E27.m2.1.1.3.3.3.2" xref="A1.E27.m2.1.1.3.3.3.2.cmml">k</mi><mo id="A1.E27.m2.1.1.3.3.3.3" xref="A1.E27.m2.1.1.3.3.3.3.cmml">∗</mo></msub></mrow></mrow><mrow id="A1.E27.m2.1.1.1" xref="A1.E27.m2.1.1.1.cmml"><msup id="A1.E27.m2.1.1.1.1" xref="A1.E27.m2.1.1.1.1.cmml"><mrow id="A1.E27.m2.1.1.1.1.1.1" xref="A1.E27.m2.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.E27.m2.1.1.1.1.1.1.2" xref="A1.E27.m2.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A1.E27.m2.1.1.1.1.1.1.1" xref="A1.E27.m2.1.1.1.1.1.1.1.cmml"><msup id="A1.E27.m2.1.1.1.1.1.1.1.2" xref="A1.E27.m2.1.1.1.1.1.1.1.2.cmml"><mi id="A1.E27.m2.1.1.1.1.1.1.1.2.2" xref="A1.E27.m2.1.1.1.1.1.1.1.2.2.cmml">C</mi><mrow id="A1.E27.m2.1.1.1.1.1.1.1.2.3" xref="A1.E27.m2.1.1.1.1.1.1.1.2.3.cmml"><mo id="A1.E27.m2.1.1.1.1.1.1.1.2.3a" xref="A1.E27.m2.1.1.1.1.1.1.1.2.3.cmml">−</mo><mn id="A1.E27.m2.1.1.1.1.1.1.1.2.3.2" xref="A1.E27.m2.1.1.1.1.1.1.1.2.3.2.cmml">1</mn></mrow></msup><mo lspace="0em" rspace="0em" id="A1.E27.m2.1.1.1.1.1.1.1.1" xref="A1.E27.m2.1.1.1.1.1.1.1.1.cmml">​</mo><msub id="A1.E27.m2.1.1.1.1.1.1.1.3" xref="A1.E27.m2.1.1.1.1.1.1.1.3.cmml"><mi id="A1.E27.m2.1.1.1.1.1.1.1.3.2" xref="A1.E27.m2.1.1.1.1.1.1.1.3.2.cmml">k</mi><mo id="A1.E27.m2.1.1.1.1.1.1.1.3.3" xref="A1.E27.m2.1.1.1.1.1.1.1.3.3.cmml">∗</mo></msub></mrow><mo stretchy="false" id="A1.E27.m2.1.1.1.1.1.1.3" xref="A1.E27.m2.1.1.1.1.1.1.1.cmml">)</mo></mrow><mi id="A1.E27.m2.1.1.1.1.3" xref="A1.E27.m2.1.1.1.1.3.cmml">T</mi></msup><mo lspace="0em" rspace="0em" id="A1.E27.m2.1.1.1.2" xref="A1.E27.m2.1.1.1.2.cmml">​</mo><msub id="A1.E27.m2.1.1.1.3" xref="A1.E27.m2.1.1.1.3.cmml"><mi id="A1.E27.m2.1.1.1.3.2" xref="A1.E27.m2.1.1.1.3.2.cmml">k</mi><mo id="A1.E27.m2.1.1.1.3.3" xref="A1.E27.m2.1.1.1.3.3.cmml">∗</mo></msub></mrow></mfrac></mstyle></mrow><annotation-xml encoding="MathML-Content" id="A1.E27.m2.1b"><apply id="A1.E27.m2.1.2.cmml" xref="A1.E27.m2.1.2"><and id="A1.E27.m2.1.2a.cmml" xref="A1.E27.m2.1.2"></and><apply id="A1.E27.m2.1.2b.cmml" xref="A1.E27.m2.1.2"><eq id="A1.E27.m2.1.2.3.cmml" xref="A1.E27.m2.1.2.3"></eq><csymbol cd="latexml" id="A1.E27.m2.1.2.2.cmml" xref="A1.E27.m2.1.2.2">absent</csymbol><apply id="A1.E27.m2.1.2.4.cmml" xref="A1.E27.m2.1.2.4"><divide id="A1.E27.m2.1.2.4.1.cmml" xref="A1.E27.m2.1.2.4"></divide><apply id="A1.E27.m2.1.2.4.2.cmml" xref="A1.E27.m2.1.2.4.2"><minus id="A1.E27.m2.1.2.4.2.1.cmml" xref="A1.E27.m2.1.2.4.2.1"></minus><apply id="A1.E27.m2.1.2.4.2.2.cmml" xref="A1.E27.m2.1.2.4.2.2"><csymbol cd="ambiguous" id="A1.E27.m2.1.2.4.2.2.1.cmml" xref="A1.E27.m2.1.2.4.2.2">subscript</csymbol><ci id="A1.E27.m2.1.2.4.2.2.2.cmml" xref="A1.E27.m2.1.2.4.2.2.2">𝑣</ci><times id="A1.E27.m2.1.2.4.2.2.3.cmml" xref="A1.E27.m2.1.2.4.2.2.3"></times></apply><apply id="A1.E27.m2.1.2.4.2.3.cmml" xref="A1.E27.m2.1.2.4.2.3"><times id="A1.E27.m2.1.2.4.2.3.1.cmml" xref="A1.E27.m2.1.2.4.2.3.1"></times><ci id="A1.E27.m2.1.2.4.2.3.2.cmml" xref="A1.E27.m2.1.2.4.2.3.2">𝑊</ci><apply id="A1.E27.m2.1.2.4.2.3.3.cmml" xref="A1.E27.m2.1.2.4.2.3.3"><csymbol cd="ambiguous" id="A1.E27.m2.1.2.4.2.3.3.1.cmml" xref="A1.E27.m2.1.2.4.2.3.3">subscript</csymbol><ci id="A1.E27.m2.1.2.4.2.3.3.2.cmml" xref="A1.E27.m2.1.2.4.2.3.3.2">𝑘</ci><times id="A1.E27.m2.1.2.4.2.3.3.3.cmml" xref="A1.E27.m2.1.2.4.2.3.3.3"></times></apply></apply></apply><apply id="A1.E27.m2.1.2.4.3.cmml" xref="A1.E27.m2.1.2.4.3"><times id="A1.E27.m2.1.2.4.3.1.cmml" xref="A1.E27.m2.1.2.4.3.1"></times><apply id="A1.E27.m2.1.2.4.3.2.cmml" xref="A1.E27.m2.1.2.4.3.2"><csymbol cd="ambiguous" id="A1.E27.m2.1.2.4.3.2.1.cmml" xref="A1.E27.m2.1.2.4.3.2">superscript</csymbol><ci id="A1.E27.m2.1.2.4.3.2.2.cmml" xref="A1.E27.m2.1.2.4.3.2.2">𝑢</ci><ci id="A1.E27.m2.1.2.4.3.2.3.cmml" xref="A1.E27.m2.1.2.4.3.2.3">𝑇</ci></apply><apply id="A1.E27.m2.1.2.4.3.3.cmml" xref="A1.E27.m2.1.2.4.3.3"><csymbol cd="ambiguous" id="A1.E27.m2.1.2.4.3.3.1.cmml" xref="A1.E27.m2.1.2.4.3.3">subscript</csymbol><ci id="A1.E27.m2.1.2.4.3.3.2.cmml" xref="A1.E27.m2.1.2.4.3.3.2">𝑘</ci><times id="A1.E27.m2.1.2.4.3.3.3.cmml" xref="A1.E27.m2.1.2.4.3.3.3"></times></apply></apply></apply></apply><apply id="A1.E27.m2.1.2c.cmml" xref="A1.E27.m2.1.2"><eq id="A1.E27.m2.1.2.5.cmml" xref="A1.E27.m2.1.2.5"></eq><share href="#A1.E27.m2.1.2.4.cmml" id="A1.E27.m2.1.2d.cmml" xref="A1.E27.m2.1.2"></share><apply id="A1.E27.m2.1.1.cmml" xref="A1.E27.m2.1.1"><divide id="A1.E27.m2.1.1.2.cmml" xref="A1.E27.m2.1.1"></divide><apply id="A1.E27.m2.1.1.3.cmml" xref="A1.E27.m2.1.1.3"><minus id="A1.E27.m2.1.1.3.1.cmml" xref="A1.E27.m2.1.1.3.1"></minus><apply id="A1.E27.m2.1.1.3.2.cmml" xref="A1.E27.m2.1.1.3.2"><csymbol cd="ambiguous" id="A1.E27.m2.1.1.3.2.1.cmml" xref="A1.E27.m2.1.1.3.2">subscript</csymbol><ci id="A1.E27.m2.1.1.3.2.2.cmml" xref="A1.E27.m2.1.1.3.2.2">𝑣</ci><times id="A1.E27.m2.1.1.3.2.3.cmml" xref="A1.E27.m2.1.1.3.2.3"></times></apply><apply id="A1.E27.m2.1.1.3.3.cmml" xref="A1.E27.m2.1.1.3.3"><times id="A1.E27.m2.1.1.3.3.1.cmml" xref="A1.E27.m2.1.1.3.3.1"></times><ci id="A1.E27.m2.1.1.3.3.2.cmml" xref="A1.E27.m2.1.1.3.3.2">𝑊</ci><apply id="A1.E27.m2.1.1.3.3.3.cmml" xref="A1.E27.m2.1.1.3.3.3"><csymbol cd="ambiguous" id="A1.E27.m2.1.1.3.3.3.1.cmml" xref="A1.E27.m2.1.1.3.3.3">subscript</csymbol><ci id="A1.E27.m2.1.1.3.3.3.2.cmml" xref="A1.E27.m2.1.1.3.3.3.2">𝑘</ci><times id="A1.E27.m2.1.1.3.3.3.3.cmml" xref="A1.E27.m2.1.1.3.3.3.3"></times></apply></apply></apply><apply id="A1.E27.m2.1.1.1.cmml" xref="A1.E27.m2.1.1.1"><times id="A1.E27.m2.1.1.1.2.cmml" xref="A1.E27.m2.1.1.1.2"></times><apply id="A1.E27.m2.1.1.1.1.cmml" xref="A1.E27.m2.1.1.1.1"><csymbol cd="ambiguous" id="A1.E27.m2.1.1.1.1.2.cmml" xref="A1.E27.m2.1.1.1.1">superscript</csymbol><apply id="A1.E27.m2.1.1.1.1.1.1.1.cmml" xref="A1.E27.m2.1.1.1.1.1.1"><times id="A1.E27.m2.1.1.1.1.1.1.1.1.cmml" xref="A1.E27.m2.1.1.1.1.1.1.1.1"></times><apply id="A1.E27.m2.1.1.1.1.1.1.1.2.cmml" xref="A1.E27.m2.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A1.E27.m2.1.1.1.1.1.1.1.2.1.cmml" xref="A1.E27.m2.1.1.1.1.1.1.1.2">superscript</csymbol><ci id="A1.E27.m2.1.1.1.1.1.1.1.2.2.cmml" xref="A1.E27.m2.1.1.1.1.1.1.1.2.2">𝐶</ci><apply id="A1.E27.m2.1.1.1.1.1.1.1.2.3.cmml" xref="A1.E27.m2.1.1.1.1.1.1.1.2.3"><minus id="A1.E27.m2.1.1.1.1.1.1.1.2.3.1.cmml" xref="A1.E27.m2.1.1.1.1.1.1.1.2.3"></minus><cn type="integer" id="A1.E27.m2.1.1.1.1.1.1.1.2.3.2.cmml" xref="A1.E27.m2.1.1.1.1.1.1.1.2.3.2">1</cn></apply></apply><apply id="A1.E27.m2.1.1.1.1.1.1.1.3.cmml" xref="A1.E27.m2.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.E27.m2.1.1.1.1.1.1.1.3.1.cmml" xref="A1.E27.m2.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="A1.E27.m2.1.1.1.1.1.1.1.3.2.cmml" xref="A1.E27.m2.1.1.1.1.1.1.1.3.2">𝑘</ci><times id="A1.E27.m2.1.1.1.1.1.1.1.3.3.cmml" xref="A1.E27.m2.1.1.1.1.1.1.1.3.3"></times></apply></apply><ci id="A1.E27.m2.1.1.1.1.3.cmml" xref="A1.E27.m2.1.1.1.1.3">𝑇</ci></apply><apply id="A1.E27.m2.1.1.1.3.cmml" xref="A1.E27.m2.1.1.1.3"><csymbol cd="ambiguous" id="A1.E27.m2.1.1.1.3.1.cmml" xref="A1.E27.m2.1.1.1.3">subscript</csymbol><ci id="A1.E27.m2.1.1.1.3.2.cmml" xref="A1.E27.m2.1.1.1.3.2">𝑘</ci><times id="A1.E27.m2.1.1.1.3.3.cmml" xref="A1.E27.m2.1.1.1.3.3"></times></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E27.m2.1c">\displaystyle=\frac{v_{*}-Wk_{*}}{u^{T}k_{*}}=\frac{v_{*}-Wk_{*}}{(C^{-1}k_{*})^{T}k_{*}}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(27)</span></td>
</tr></tbody>
</table>
<div class="ltx_pagination ltx_role_newpage"></div>
</div>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Causal Tracing</h2>

<section id="A2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.1 </span>Experimental Settings</h3>

<div id="A2.SS1.p1" class="ltx_para">
<p id="A2.SS1.p1.3" class="ltx_p">Note that, in by-layer experimental results, layers are numbered from 0 to <math id="A2.SS1.p1.1.m1.1" class="ltx_Math" alttext="L-1" display="inline"><semantics id="A2.SS1.p1.1.m1.1a"><mrow id="A2.SS1.p1.1.m1.1.1" xref="A2.SS1.p1.1.m1.1.1.cmml"><mi id="A2.SS1.p1.1.m1.1.1.2" xref="A2.SS1.p1.1.m1.1.1.2.cmml">L</mi><mo id="A2.SS1.p1.1.m1.1.1.1" xref="A2.SS1.p1.1.m1.1.1.1.cmml">−</mo><mn id="A2.SS1.p1.1.m1.1.1.3" xref="A2.SS1.p1.1.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="A2.SS1.p1.1.m1.1b"><apply id="A2.SS1.p1.1.m1.1.1.cmml" xref="A2.SS1.p1.1.m1.1.1"><minus id="A2.SS1.p1.1.m1.1.1.1.cmml" xref="A2.SS1.p1.1.m1.1.1.1"></minus><ci id="A2.SS1.p1.1.m1.1.1.2.cmml" xref="A2.SS1.p1.1.m1.1.1.2">𝐿</ci><cn type="integer" id="A2.SS1.p1.1.m1.1.1.3.cmml" xref="A2.SS1.p1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p1.1.m1.1c">L-1</annotation></semantics></math> rather than <math id="A2.SS1.p1.2.m2.1" class="ltx_Math" alttext="1" display="inline"><semantics id="A2.SS1.p1.2.m2.1a"><mn id="A2.SS1.p1.2.m2.1.1" xref="A2.SS1.p1.2.m2.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="A2.SS1.p1.2.m2.1b"><cn type="integer" id="A2.SS1.p1.2.m2.1.1.cmml" xref="A2.SS1.p1.2.m2.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p1.2.m2.1c">1</annotation></semantics></math> to <math id="A2.SS1.p1.3.m3.1" class="ltx_Math" alttext="L" display="inline"><semantics id="A2.SS1.p1.3.m3.1a"><mi id="A2.SS1.p1.3.m3.1.1" xref="A2.SS1.p1.3.m3.1.1.cmml">L</mi><annotation-xml encoding="MathML-Content" id="A2.SS1.p1.3.m3.1b"><ci id="A2.SS1.p1.3.m3.1.1.cmml" xref="A2.SS1.p1.3.m3.1.1">𝐿</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p1.3.m3.1c">L</annotation></semantics></math>.</p>
</div>
<div id="A2.SS1.p2" class="ltx_para">
<p id="A2.SS1.p2.2" class="ltx_p">In Figure <a href="#S2.F2" title="Figure 2 ‣ 2 Interventions on Activations for Tracing Information Flow ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> and Figure <a href="#S2.F3" title="Figure 3 ‣ 2.2 Causal Tracing Results ‣ 2 Interventions on Activations for Tracing Information Flow ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> we evaluate mean causal traces over a set of 1000 factual prompts that are known by GPT-2 XL, collected as follows. We perform greedy generation using facts and fact templates from <span id="A2.SS1.p2.2.1" class="ltx_text ltx_font_smallcaps">CounterFact</span>, and we identify predicted text that names the correct object <math id="A2.SS1.p2.1.m1.1" class="ltx_Math" alttext="o^{c}" display="inline"><semantics id="A2.SS1.p2.1.m1.1a"><msup id="A2.SS1.p2.1.m1.1.1" xref="A2.SS1.p2.1.m1.1.1.cmml"><mi id="A2.SS1.p2.1.m1.1.1.2" xref="A2.SS1.p2.1.m1.1.1.2.cmml">o</mi><mi id="A2.SS1.p2.1.m1.1.1.3" xref="A2.SS1.p2.1.m1.1.1.3.cmml">c</mi></msup><annotation-xml encoding="MathML-Content" id="A2.SS1.p2.1.m1.1b"><apply id="A2.SS1.p2.1.m1.1.1.cmml" xref="A2.SS1.p2.1.m1.1.1"><csymbol cd="ambiguous" id="A2.SS1.p2.1.m1.1.1.1.cmml" xref="A2.SS1.p2.1.m1.1.1">superscript</csymbol><ci id="A2.SS1.p2.1.m1.1.1.2.cmml" xref="A2.SS1.p2.1.m1.1.1.2">𝑜</ci><ci id="A2.SS1.p2.1.m1.1.1.3.cmml" xref="A2.SS1.p2.1.m1.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p2.1.m1.1c">o^{c}</annotation></semantics></math> before naming any other capitalized word. We use the text up to but not including the object <math id="A2.SS1.p2.2.m2.1" class="ltx_Math" alttext="o^{c}" display="inline"><semantics id="A2.SS1.p2.2.m2.1a"><msup id="A2.SS1.p2.2.m2.1.1" xref="A2.SS1.p2.2.m2.1.1.cmml"><mi id="A2.SS1.p2.2.m2.1.1.2" xref="A2.SS1.p2.2.m2.1.1.2.cmml">o</mi><mi id="A2.SS1.p2.2.m2.1.1.3" xref="A2.SS1.p2.2.m2.1.1.3.cmml">c</mi></msup><annotation-xml encoding="MathML-Content" id="A2.SS1.p2.2.m2.1b"><apply id="A2.SS1.p2.2.m2.1.1.cmml" xref="A2.SS1.p2.2.m2.1.1"><csymbol cd="ambiguous" id="A2.SS1.p2.2.m2.1.1.1.cmml" xref="A2.SS1.p2.2.m2.1.1">superscript</csymbol><ci id="A2.SS1.p2.2.m2.1.1.2.cmml" xref="A2.SS1.p2.2.m2.1.1.2">𝑜</ci><ci id="A2.SS1.p2.2.m2.1.1.3.cmml" xref="A2.SS1.p2.2.m2.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p2.2.m2.1c">o^{c}</annotation></semantics></math> as the prompt, and we randomly sample 1000 of these texts. In this sample of known facts, the predicted probability of the correct object token calculated by GPT-2 XL averages 27.0%.</p>
</div>
<div id="A2.SS1.p3" class="ltx_para">
<p id="A2.SS1.p3.4" class="ltx_p">In the corrupted run, we corrupt the embeddings of the token naming the subject <math id="A2.SS1.p3.1.m1.1" class="ltx_Math" alttext="s" display="inline"><semantics id="A2.SS1.p3.1.m1.1a"><mi id="A2.SS1.p3.1.m1.1.1" xref="A2.SS1.p3.1.m1.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="A2.SS1.p3.1.m1.1b"><ci id="A2.SS1.p3.1.m1.1.1.cmml" xref="A2.SS1.p3.1.m1.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p3.1.m1.1c">s</annotation></semantics></math> by adding Gaussian noise <math id="A2.SS1.p3.2.m2.2" class="ltx_Math" alttext="\epsilon\sim\mathcal{N}(0;\nu)" display="inline"><semantics id="A2.SS1.p3.2.m2.2a"><mrow id="A2.SS1.p3.2.m2.2.3" xref="A2.SS1.p3.2.m2.2.3.cmml"><mi id="A2.SS1.p3.2.m2.2.3.2" xref="A2.SS1.p3.2.m2.2.3.2.cmml">ϵ</mi><mo id="A2.SS1.p3.2.m2.2.3.1" xref="A2.SS1.p3.2.m2.2.3.1.cmml">∼</mo><mrow id="A2.SS1.p3.2.m2.2.3.3" xref="A2.SS1.p3.2.m2.2.3.3.cmml"><mi class="ltx_font_mathcaligraphic" id="A2.SS1.p3.2.m2.2.3.3.2" xref="A2.SS1.p3.2.m2.2.3.3.2.cmml">𝒩</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p3.2.m2.2.3.3.1" xref="A2.SS1.p3.2.m2.2.3.3.1.cmml">​</mo><mrow id="A2.SS1.p3.2.m2.2.3.3.3.2" xref="A2.SS1.p3.2.m2.2.3.3.3.1.cmml"><mo stretchy="false" id="A2.SS1.p3.2.m2.2.3.3.3.2.1" xref="A2.SS1.p3.2.m2.2.3.3.3.1.cmml">(</mo><mn id="A2.SS1.p3.2.m2.1.1" xref="A2.SS1.p3.2.m2.1.1.cmml">0</mn><mo id="A2.SS1.p3.2.m2.2.3.3.3.2.2" xref="A2.SS1.p3.2.m2.2.3.3.3.1.cmml">;</mo><mi id="A2.SS1.p3.2.m2.2.2" xref="A2.SS1.p3.2.m2.2.2.cmml">ν</mi><mo stretchy="false" id="A2.SS1.p3.2.m2.2.3.3.3.2.3" xref="A2.SS1.p3.2.m2.2.3.3.3.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.SS1.p3.2.m2.2b"><apply id="A2.SS1.p3.2.m2.2.3.cmml" xref="A2.SS1.p3.2.m2.2.3"><csymbol cd="latexml" id="A2.SS1.p3.2.m2.2.3.1.cmml" xref="A2.SS1.p3.2.m2.2.3.1">similar-to</csymbol><ci id="A2.SS1.p3.2.m2.2.3.2.cmml" xref="A2.SS1.p3.2.m2.2.3.2">italic-ϵ</ci><apply id="A2.SS1.p3.2.m2.2.3.3.cmml" xref="A2.SS1.p3.2.m2.2.3.3"><times id="A2.SS1.p3.2.m2.2.3.3.1.cmml" xref="A2.SS1.p3.2.m2.2.3.3.1"></times><ci id="A2.SS1.p3.2.m2.2.3.3.2.cmml" xref="A2.SS1.p3.2.m2.2.3.3.2">𝒩</ci><list id="A2.SS1.p3.2.m2.2.3.3.3.1.cmml" xref="A2.SS1.p3.2.m2.2.3.3.3.2"><cn type="integer" id="A2.SS1.p3.2.m2.1.1.cmml" xref="A2.SS1.p3.2.m2.1.1">0</cn><ci id="A2.SS1.p3.2.m2.2.2.cmml" xref="A2.SS1.p3.2.m2.2.2">𝜈</ci></list></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p3.2.m2.2c">\epsilon\sim\mathcal{N}(0;\nu)</annotation></semantics></math>, where <math id="A2.SS1.p3.3.m3.1" class="ltx_Math" alttext="\nu=3\sigma_{t}" display="inline"><semantics id="A2.SS1.p3.3.m3.1a"><mrow id="A2.SS1.p3.3.m3.1.1" xref="A2.SS1.p3.3.m3.1.1.cmml"><mi id="A2.SS1.p3.3.m3.1.1.2" xref="A2.SS1.p3.3.m3.1.1.2.cmml">ν</mi><mo id="A2.SS1.p3.3.m3.1.1.1" xref="A2.SS1.p3.3.m3.1.1.1.cmml">=</mo><mrow id="A2.SS1.p3.3.m3.1.1.3" xref="A2.SS1.p3.3.m3.1.1.3.cmml"><mn id="A2.SS1.p3.3.m3.1.1.3.2" xref="A2.SS1.p3.3.m3.1.1.3.2.cmml">3</mn><mo lspace="0em" rspace="0em" id="A2.SS1.p3.3.m3.1.1.3.1" xref="A2.SS1.p3.3.m3.1.1.3.1.cmml">​</mo><msub id="A2.SS1.p3.3.m3.1.1.3.3" xref="A2.SS1.p3.3.m3.1.1.3.3.cmml"><mi id="A2.SS1.p3.3.m3.1.1.3.3.2" xref="A2.SS1.p3.3.m3.1.1.3.3.2.cmml">σ</mi><mi id="A2.SS1.p3.3.m3.1.1.3.3.3" xref="A2.SS1.p3.3.m3.1.1.3.3.3.cmml">t</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.SS1.p3.3.m3.1b"><apply id="A2.SS1.p3.3.m3.1.1.cmml" xref="A2.SS1.p3.3.m3.1.1"><eq id="A2.SS1.p3.3.m3.1.1.1.cmml" xref="A2.SS1.p3.3.m3.1.1.1"></eq><ci id="A2.SS1.p3.3.m3.1.1.2.cmml" xref="A2.SS1.p3.3.m3.1.1.2">𝜈</ci><apply id="A2.SS1.p3.3.m3.1.1.3.cmml" xref="A2.SS1.p3.3.m3.1.1.3"><times id="A2.SS1.p3.3.m3.1.1.3.1.cmml" xref="A2.SS1.p3.3.m3.1.1.3.1"></times><cn type="integer" id="A2.SS1.p3.3.m3.1.1.3.2.cmml" xref="A2.SS1.p3.3.m3.1.1.3.2">3</cn><apply id="A2.SS1.p3.3.m3.1.1.3.3.cmml" xref="A2.SS1.p3.3.m3.1.1.3.3"><csymbol cd="ambiguous" id="A2.SS1.p3.3.m3.1.1.3.3.1.cmml" xref="A2.SS1.p3.3.m3.1.1.3.3">subscript</csymbol><ci id="A2.SS1.p3.3.m3.1.1.3.3.2.cmml" xref="A2.SS1.p3.3.m3.1.1.3.3.2">𝜎</ci><ci id="A2.SS1.p3.3.m3.1.1.3.3.3.cmml" xref="A2.SS1.p3.3.m3.1.1.3.3.3">𝑡</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p3.3.m3.1c">\nu=3\sigma_{t}</annotation></semantics></math> is set to be three times larger than the observed standard deviation <math id="A2.SS1.p3.4.m4.1" class="ltx_Math" alttext="\sigma_{t}" display="inline"><semantics id="A2.SS1.p3.4.m4.1a"><msub id="A2.SS1.p3.4.m4.1.1" xref="A2.SS1.p3.4.m4.1.1.cmml"><mi id="A2.SS1.p3.4.m4.1.1.2" xref="A2.SS1.p3.4.m4.1.1.2.cmml">σ</mi><mi id="A2.SS1.p3.4.m4.1.1.3" xref="A2.SS1.p3.4.m4.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="A2.SS1.p3.4.m4.1b"><apply id="A2.SS1.p3.4.m4.1.1.cmml" xref="A2.SS1.p3.4.m4.1.1"><csymbol cd="ambiguous" id="A2.SS1.p3.4.m4.1.1.1.cmml" xref="A2.SS1.p3.4.m4.1.1">subscript</csymbol><ci id="A2.SS1.p3.4.m4.1.1.2.cmml" xref="A2.SS1.p3.4.m4.1.1.2">𝜎</ci><ci id="A2.SS1.p3.4.m4.1.1.3.cmml" xref="A2.SS1.p3.4.m4.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p3.4.m4.1c">\sigma_{t}</annotation></semantics></math> of token embeddings as sampled over a body of text.
For each run of text, the process is repeated ten times with different samples of corruption noise. On average, this reduces the correct object token score to 8.47%, less than one third the original score.</p>
</div>
<div id="A2.SS1.p4" class="ltx_para">
<p id="A2.SS1.p4.1" class="ltx_p">When we restore hidden states from the original run, we substitute the originally calculated values from the same layer and the same token, and then we allow subsequent calculations to proceed without further intervention. For the experiments in Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> (and the purple traces throughout the appendix), a single activation vector is restored. Naturally, restoring the last vector on the last token will fully restore the original predicted scores, but our plotted results show that there are also earlier activation vectors at a second location that also have a strong causal effect: the average maximum score seen by restoring the most impactful activation vector at the last token of the subject is 19.5%. In Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>j where effects are bucketed by layer, the maximum effect is seen around the 15th layer of the last subject token, where the score is raised on average to 15.0%.</p>
</div>
</section>
<section id="A2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.2 </span>Separating MLP and Attn Effects</h3>

<div id="A2.SS2.p1" class="ltx_para">
<p id="A2.SS2.p1.4" class="ltx_p">When decomposing the effects into MLP and Attn lookups, we found that restoring single activation vectors from individual MLP and individual Attn lookups had generally negligible effects, suggesting the decisive information is accumulated across layers. Therefore for MLP and Attn lookups, we restored runs of ten values of <math id="A2.SS2.p1.1.m1.1" class="ltx_Math" alttext="\smash{m^{(l)}_{i}}" display="inline"><semantics id="A2.SS2.p1.1.m1.1a"><msubsup id="A2.SS2.p1.1.m1.1.2" xref="A2.SS2.p1.1.m1.1.2.cmml"><mi id="A2.SS2.p1.1.m1.1.2.2.2" xref="A2.SS2.p1.1.m1.1.2.2.2.cmml">m</mi><mi id="A2.SS2.p1.1.m1.1.2.3" xref="A2.SS2.p1.1.m1.1.2.3.cmml">i</mi><mrow id="A2.SS2.p1.1.m1.1.1.1.3" xref="A2.SS2.p1.1.m1.1.2.cmml"><mo stretchy="false" id="A2.SS2.p1.1.m1.1.1.1.3.1" xref="A2.SS2.p1.1.m1.1.2.cmml">(</mo><mi id="A2.SS2.p1.1.m1.1.1.1.1" xref="A2.SS2.p1.1.m1.1.1.1.1.cmml">l</mi><mo stretchy="false" id="A2.SS2.p1.1.m1.1.1.1.3.2" xref="A2.SS2.p1.1.m1.1.2.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="A2.SS2.p1.1.m1.1b"><apply id="A2.SS2.p1.1.m1.1.2.cmml" xref="A2.SS2.p1.1.m1.1.2"><csymbol cd="ambiguous" id="A2.SS2.p1.1.m1.1.2.1.cmml" xref="A2.SS2.p1.1.m1.1.2">subscript</csymbol><apply id="A2.SS2.p1.1.m1.1.2.2.cmml" xref="A2.SS2.p1.1.m1.1.2"><csymbol cd="ambiguous" id="A2.SS2.p1.1.m1.1.2.2.1.cmml" xref="A2.SS2.p1.1.m1.1.2">superscript</csymbol><ci id="A2.SS2.p1.1.m1.1.2.2.2.cmml" xref="A2.SS2.p1.1.m1.1.2.2.2">𝑚</ci><ci id="A2.SS2.p1.1.m1.1.1.1.1.cmml" xref="A2.SS2.p1.1.m1.1.1.1.1">𝑙</ci></apply><ci id="A2.SS2.p1.1.m1.1.2.3.cmml" xref="A2.SS2.p1.1.m1.1.2.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS2.p1.1.m1.1c">\smash{m^{(l)}_{i}}</annotation></semantics></math> (and <math id="A2.SS2.p1.2.m2.1" class="ltx_Math" alttext="\smash{a^{(l)}_{i}}" display="inline"><semantics id="A2.SS2.p1.2.m2.1a"><msubsup id="A2.SS2.p1.2.m2.1.2" xref="A2.SS2.p1.2.m2.1.2.cmml"><mi id="A2.SS2.p1.2.m2.1.2.2.2" xref="A2.SS2.p1.2.m2.1.2.2.2.cmml">a</mi><mi id="A2.SS2.p1.2.m2.1.2.3" xref="A2.SS2.p1.2.m2.1.2.3.cmml">i</mi><mrow id="A2.SS2.p1.2.m2.1.1.1.3" xref="A2.SS2.p1.2.m2.1.2.cmml"><mo stretchy="false" id="A2.SS2.p1.2.m2.1.1.1.3.1" xref="A2.SS2.p1.2.m2.1.2.cmml">(</mo><mi id="A2.SS2.p1.2.m2.1.1.1.1" xref="A2.SS2.p1.2.m2.1.1.1.1.cmml">l</mi><mo stretchy="false" id="A2.SS2.p1.2.m2.1.1.1.3.2" xref="A2.SS2.p1.2.m2.1.2.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="A2.SS2.p1.2.m2.1b"><apply id="A2.SS2.p1.2.m2.1.2.cmml" xref="A2.SS2.p1.2.m2.1.2"><csymbol cd="ambiguous" id="A2.SS2.p1.2.m2.1.2.1.cmml" xref="A2.SS2.p1.2.m2.1.2">subscript</csymbol><apply id="A2.SS2.p1.2.m2.1.2.2.cmml" xref="A2.SS2.p1.2.m2.1.2"><csymbol cd="ambiguous" id="A2.SS2.p1.2.m2.1.2.2.1.cmml" xref="A2.SS2.p1.2.m2.1.2">superscript</csymbol><ci id="A2.SS2.p1.2.m2.1.2.2.2.cmml" xref="A2.SS2.p1.2.m2.1.2.2.2">𝑎</ci><ci id="A2.SS2.p1.2.m2.1.1.1.1.cmml" xref="A2.SS2.p1.2.m2.1.1.1.1">𝑙</ci></apply><ci id="A2.SS2.p1.2.m2.1.2.3.cmml" xref="A2.SS2.p1.2.m2.1.2.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS2.p1.2.m2.1c">\smash{a^{(l)}_{i}}</annotation></semantics></math>, respectively) for an interval of layers ranging from <math id="A2.SS2.p1.3.m3.3" class="ltx_Math" alttext="[l_{*}-4,...,l_{*}+5]" display="inline"><semantics id="A2.SS2.p1.3.m3.3a"><mrow id="A2.SS2.p1.3.m3.3.3.2" xref="A2.SS2.p1.3.m3.3.3.3.cmml"><mo stretchy="false" id="A2.SS2.p1.3.m3.3.3.2.3" xref="A2.SS2.p1.3.m3.3.3.3.cmml">[</mo><mrow id="A2.SS2.p1.3.m3.2.2.1.1" xref="A2.SS2.p1.3.m3.2.2.1.1.cmml"><msub id="A2.SS2.p1.3.m3.2.2.1.1.2" xref="A2.SS2.p1.3.m3.2.2.1.1.2.cmml"><mi id="A2.SS2.p1.3.m3.2.2.1.1.2.2" xref="A2.SS2.p1.3.m3.2.2.1.1.2.2.cmml">l</mi><mo id="A2.SS2.p1.3.m3.2.2.1.1.2.3" xref="A2.SS2.p1.3.m3.2.2.1.1.2.3.cmml">∗</mo></msub><mo id="A2.SS2.p1.3.m3.2.2.1.1.1" xref="A2.SS2.p1.3.m3.2.2.1.1.1.cmml">−</mo><mn id="A2.SS2.p1.3.m3.2.2.1.1.3" xref="A2.SS2.p1.3.m3.2.2.1.1.3.cmml">4</mn></mrow><mo id="A2.SS2.p1.3.m3.3.3.2.4" xref="A2.SS2.p1.3.m3.3.3.3.cmml">,</mo><mi mathvariant="normal" id="A2.SS2.p1.3.m3.1.1" xref="A2.SS2.p1.3.m3.1.1.cmml">…</mi><mo id="A2.SS2.p1.3.m3.3.3.2.5" xref="A2.SS2.p1.3.m3.3.3.3.cmml">,</mo><mrow id="A2.SS2.p1.3.m3.3.3.2.2" xref="A2.SS2.p1.3.m3.3.3.2.2.cmml"><msub id="A2.SS2.p1.3.m3.3.3.2.2.2" xref="A2.SS2.p1.3.m3.3.3.2.2.2.cmml"><mi id="A2.SS2.p1.3.m3.3.3.2.2.2.2" xref="A2.SS2.p1.3.m3.3.3.2.2.2.2.cmml">l</mi><mo id="A2.SS2.p1.3.m3.3.3.2.2.2.3" xref="A2.SS2.p1.3.m3.3.3.2.2.2.3.cmml">∗</mo></msub><mo id="A2.SS2.p1.3.m3.3.3.2.2.1" xref="A2.SS2.p1.3.m3.3.3.2.2.1.cmml">+</mo><mn id="A2.SS2.p1.3.m3.3.3.2.2.3" xref="A2.SS2.p1.3.m3.3.3.2.2.3.cmml">5</mn></mrow><mo stretchy="false" id="A2.SS2.p1.3.m3.3.3.2.6" xref="A2.SS2.p1.3.m3.3.3.3.cmml">]</mo></mrow><annotation-xml encoding="MathML-Content" id="A2.SS2.p1.3.m3.3b"><list id="A2.SS2.p1.3.m3.3.3.3.cmml" xref="A2.SS2.p1.3.m3.3.3.2"><apply id="A2.SS2.p1.3.m3.2.2.1.1.cmml" xref="A2.SS2.p1.3.m3.2.2.1.1"><minus id="A2.SS2.p1.3.m3.2.2.1.1.1.cmml" xref="A2.SS2.p1.3.m3.2.2.1.1.1"></minus><apply id="A2.SS2.p1.3.m3.2.2.1.1.2.cmml" xref="A2.SS2.p1.3.m3.2.2.1.1.2"><csymbol cd="ambiguous" id="A2.SS2.p1.3.m3.2.2.1.1.2.1.cmml" xref="A2.SS2.p1.3.m3.2.2.1.1.2">subscript</csymbol><ci id="A2.SS2.p1.3.m3.2.2.1.1.2.2.cmml" xref="A2.SS2.p1.3.m3.2.2.1.1.2.2">𝑙</ci><times id="A2.SS2.p1.3.m3.2.2.1.1.2.3.cmml" xref="A2.SS2.p1.3.m3.2.2.1.1.2.3"></times></apply><cn type="integer" id="A2.SS2.p1.3.m3.2.2.1.1.3.cmml" xref="A2.SS2.p1.3.m3.2.2.1.1.3">4</cn></apply><ci id="A2.SS2.p1.3.m3.1.1.cmml" xref="A2.SS2.p1.3.m3.1.1">…</ci><apply id="A2.SS2.p1.3.m3.3.3.2.2.cmml" xref="A2.SS2.p1.3.m3.3.3.2.2"><plus id="A2.SS2.p1.3.m3.3.3.2.2.1.cmml" xref="A2.SS2.p1.3.m3.3.3.2.2.1"></plus><apply id="A2.SS2.p1.3.m3.3.3.2.2.2.cmml" xref="A2.SS2.p1.3.m3.3.3.2.2.2"><csymbol cd="ambiguous" id="A2.SS2.p1.3.m3.3.3.2.2.2.1.cmml" xref="A2.SS2.p1.3.m3.3.3.2.2.2">subscript</csymbol><ci id="A2.SS2.p1.3.m3.3.3.2.2.2.2.cmml" xref="A2.SS2.p1.3.m3.3.3.2.2.2.2">𝑙</ci><times id="A2.SS2.p1.3.m3.3.3.2.2.2.3.cmml" xref="A2.SS2.p1.3.m3.3.3.2.2.2.3"></times></apply><cn type="integer" id="A2.SS2.p1.3.m3.3.3.2.2.3.cmml" xref="A2.SS2.p1.3.m3.3.3.2.2.3">5</cn></apply></list></annotation-xml><annotation encoding="application/x-tex" id="A2.SS2.p1.3.m3.3c">[l_{*}-4,...,l_{*}+5]</annotation></semantics></math> (clipping at the edges), where the results are plotted at layer <math id="A2.SS2.p1.4.m4.1" class="ltx_Math" alttext="l_{*}" display="inline"><semantics id="A2.SS2.p1.4.m4.1a"><msub id="A2.SS2.p1.4.m4.1.1" xref="A2.SS2.p1.4.m4.1.1.cmml"><mi id="A2.SS2.p1.4.m4.1.1.2" xref="A2.SS2.p1.4.m4.1.1.2.cmml">l</mi><mo id="A2.SS2.p1.4.m4.1.1.3" xref="A2.SS2.p1.4.m4.1.1.3.cmml">∗</mo></msub><annotation-xml encoding="MathML-Content" id="A2.SS2.p1.4.m4.1b"><apply id="A2.SS2.p1.4.m4.1.1.cmml" xref="A2.SS2.p1.4.m4.1.1"><csymbol cd="ambiguous" id="A2.SS2.p1.4.m4.1.1.1.cmml" xref="A2.SS2.p1.4.m4.1.1">subscript</csymbol><ci id="A2.SS2.p1.4.m4.1.1.2.cmml" xref="A2.SS2.p1.4.m4.1.1.2">𝑙</ci><times id="A2.SS2.p1.4.m4.1.1.3.cmml" xref="A2.SS2.p1.4.m4.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS2.p1.4.m4.1c">l_{*}</annotation></semantics></math>. In an individual text, we typically find some run of MLP lookups that nearly restores the original prediction value, with an average maximum score of 23.6%. Figure <a href="#S2.F2" title="Figure 2 ‣ 2 Interventions on Activations for Tracing Information Flow ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>b buckets averages for each token-location pair, and finds the maximum effect at an interval at the last entity token, centered at the the 17th layer, which restores scores to an average of 15.0%. For Attn lookups (Figure <a href="#S2.F2" title="Figure 2 ‣ 2 Interventions on Activations for Tracing Information Flow ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>c), the average maximum score over any location is 19.4%, and when bucketed by location, the maximum effect is centered at the 32nd layer at the last word before prediction, which restores scores to an average of 16.5%.</p>
</div>
<div id="A2.SS2.p2" class="ltx_para">
<p id="A2.SS2.p2.1" class="ltx_p">Figure <a href="#A2.F7" title="Figure 7 ‣ B.2 Separating MLP and Attn Effects ‣ Appendix B Causal Tracing ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a> shows mean causal traces as line plots with 95% confidence intervals, instead of heatmaps.</p>
</div>
<figure id="A2.F7" class="ltx_figure"><img src="/html/2202.05262/assets/x7.png" id="A2.F7.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="116" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A2.F7.3.1.1" class="ltx_text" style="font-size:90%;">Figure 7</span>: </span><span id="A2.F7.4.2" class="ltx_text" style="font-size:90%;">Mean causal traces of GPT-XL over a sample of 1000 factual statements, shown as a line plot with 95% confidence intervals. (a) Shows the same data as Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>j as a line plot instead of a heatmap; (b) matches Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>k; (c) matches Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>m. The confidence intervals confirm that the distinctions between peak and non-peak causal effects at both early and late sites are significant.</span></figcaption>
</figure>
</section>
<section id="A2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.3 </span>Traces of EleutherAI GPT-NeoX (20B) and GPT-J (6B) and smaller models</h3>

<figure id="A2.F8" class="ltx_figure"><img src="/html/2202.05262/assets/x8.png" id="A2.F8.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="168" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A2.F8.3.1.1" class="ltx_text" style="font-size:90%;">Figure 8</span>: </span><span id="A2.F8.4.2" class="ltx_text" style="font-size:90%;">(a, b, c) Causal traces for GPT-NeoX (20B) and (d, e, f) Causal traces for GPT-J (6B).</span></figcaption>
</figure>
<figure id="A2.F9" class="ltx_figure"><img src="/html/2202.05262/assets/x9.png" id="A2.F9.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="399" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A2.F9.3.1.1" class="ltx_text" style="font-size:90%;">Figure 9</span>: </span><span id="A2.F9.4.2" class="ltx_text" style="font-size:90%;">Comparing mean causal traces across a wide range of different model sizes. (Compare to Figure <a href="#A2.F7" title="Figure 7 ‣ B.2 Separating MLP and Attn Effects ‣ Appendix B Causal Tracing ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>.) GPT-medium (a, b, c) has 334 million parameters, GPT-large (d, e, f) has 774 million parameters, and NeoX-20B (g, h, i) has 20 billion parameters. In addition, NeoX has some architectural variations. Despite the wide range of differences, a similar pattern of localized causal effects is seen across models. Interestingly, for very large models, some effects are stronger. For example, hidden states before the last subject token have negative causal effects instead of merely low effects, while hidden states at early layers at the last subject token continue to have large positive effects, continuing to implicate MLP. Also, attention modules with strong causal effects appear earlier in the stack of layers.</span></figcaption>
</figure>
<div id="A2.SS3.p1" class="ltx_para">
<p id="A2.SS3.p1.2" class="ltx_p">We conduct the causal trace experiment using on GPT-NeoX (20 billion parameters) as well as GPT-J (6 billion parameters). For GPT-NeoX we adjust the injected noise to <math id="A2.SS3.p1.1.m1.1" class="ltx_Math" alttext="\nu=0.03" display="inline"><semantics id="A2.SS3.p1.1.m1.1a"><mrow id="A2.SS3.p1.1.m1.1.1" xref="A2.SS3.p1.1.m1.1.1.cmml"><mi id="A2.SS3.p1.1.m1.1.1.2" xref="A2.SS3.p1.1.m1.1.1.2.cmml">ν</mi><mo id="A2.SS3.p1.1.m1.1.1.1" xref="A2.SS3.p1.1.m1.1.1.1.cmml">=</mo><mn id="A2.SS3.p1.1.m1.1.1.3" xref="A2.SS3.p1.1.m1.1.1.3.cmml">0.03</mn></mrow><annotation-xml encoding="MathML-Content" id="A2.SS3.p1.1.m1.1b"><apply id="A2.SS3.p1.1.m1.1.1.cmml" xref="A2.SS3.p1.1.m1.1.1"><eq id="A2.SS3.p1.1.m1.1.1.1.cmml" xref="A2.SS3.p1.1.m1.1.1.1"></eq><ci id="A2.SS3.p1.1.m1.1.1.2.cmml" xref="A2.SS3.p1.1.m1.1.1.2">𝜈</ci><cn type="float" id="A2.SS3.p1.1.m1.1.1.3.cmml" xref="A2.SS3.p1.1.m1.1.1.3">0.03</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS3.p1.1.m1.1c">\nu=0.03</annotation></semantics></math> and in GPT-J we use <math id="A2.SS3.p1.2.m2.1" class="ltx_Math" alttext="\nu=0.025" display="inline"><semantics id="A2.SS3.p1.2.m2.1a"><mrow id="A2.SS3.p1.2.m2.1.1" xref="A2.SS3.p1.2.m2.1.1.cmml"><mi id="A2.SS3.p1.2.m2.1.1.2" xref="A2.SS3.p1.2.m2.1.1.2.cmml">ν</mi><mo id="A2.SS3.p1.2.m2.1.1.1" xref="A2.SS3.p1.2.m2.1.1.1.cmml">=</mo><mn id="A2.SS3.p1.2.m2.1.1.3" xref="A2.SS3.p1.2.m2.1.1.3.cmml">0.025</mn></mrow><annotation-xml encoding="MathML-Content" id="A2.SS3.p1.2.m2.1b"><apply id="A2.SS3.p1.2.m2.1.1.cmml" xref="A2.SS3.p1.2.m2.1.1"><eq id="A2.SS3.p1.2.m2.1.1.1.cmml" xref="A2.SS3.p1.2.m2.1.1.1"></eq><ci id="A2.SS3.p1.2.m2.1.1.2.cmml" xref="A2.SS3.p1.2.m2.1.1.2">𝜈</ci><cn type="float" id="A2.SS3.p1.2.m2.1.1.3.cmml" xref="A2.SS3.p1.2.m2.1.1.3">0.025</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS3.p1.2.m2.1c">\nu=0.025</annotation></semantics></math> to match embedding magnitudes. We use the same factual prompts as GPT-2 XL, eliminating cases where the larger models would have predicted a different word for the object. Results are shown in Figure <a href="#A2.F8" title="Figure 8 ‣ B.3 Traces of EleutherAI GPT-NeoX (20B) and GPT-J (6B) and smaller models ‣ Appendix B Causal Tracing ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a>.
GPT-NeoX and GPT-J differ from GPT-2 because they have has fewer layers (44 and 28 layers instead of 48), and a slightly different residual structure across layers. Nevertheless, the causal traces look similar, with an early site with causal states concentrated at the last token of the subject, a large role for MLP states at that site. Again, attention dominates at the last token before prediction.</p>
</div>
<div id="A2.SS3.p2" class="ltx_para">
<p id="A2.SS3.p2.1" class="ltx_p">There are some differences compared to GPT-2. The importance of attention at the first layers of the last subject token is more apparent in GPT-Neo and GPT-J compared to GPT-2, suggesting that the attention parameters may be playing a larger role in storing factual associations. This concentration of attention at the beginning may also be due to fewer layers in the Eleuther models: attending to the subject name must be done in a concentrated way at just a layer or two, because there are not enough layers to spread out that computation in the shallower model.
The similarity between the GPT-NeoX and GPT-J and GPT-2 XL traces helps us to understand why ROME continues to work well with higher-parameter models, as seen on our experiments in altering parameters of GPT-J.</p>
</div>
<div id="A2.SS3.p3" class="ltx_para">
<p id="A2.SS3.p3.1" class="ltx_p">To examine effects over a wide range of scales, we also compare causal traces for smaller models GPT-2 Medium and GPT-2 Large. These smaller models are compared to NeoX-20B in Figure <a href="#A2.F9" title="Figure 9 ‣ B.3 Traces of EleutherAI GPT-NeoX (20B) and GPT-J (6B) and smaller models ‣ Appendix B Causal Tracing ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>. We find that across sizes and architectural variations, early-site MLP modules continue to have high indirect causal effects at the last subject token, although the layers where effects peak are different from one model to another.</p>
</div>
</section>
<section id="A2.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.4 </span>Causal Tracing Examples and Further Insights</h3>

<div id="A2.SS4.p1" class="ltx_para">
<p id="A2.SS4.p1.1" class="ltx_p">We include further examples of phenomena that can be observed in causal traces. Figure <a href="#A2.F10" title="Figure 10 ‣ B.4 Causal Tracing Examples and Further Insights ‣ Appendix B Causal Tracing ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">10</span></a> shows typical examples across different facts. Figure <a href="#A2.F11" title="Figure 11 ‣ B.4 Causal Tracing Examples and Further Insights ‣ Appendix B Causal Tracing ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a> discusses examples where decisive hidden states are not at the <span id="A2.SS4.p1.1.1" class="ltx_text ltx_font_italic">last</span> subject token. Figure <a href="#A2.F14" title="Figure 14 ‣ B.4 Causal Tracing Examples and Further Insights ‣ Appendix B Causal Tracing ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">14</span></a> examines traces at an individual token in more detail.</p>
</div>
<div id="A2.SS4.p2" class="ltx_para">
<p id="A2.SS4.p2.1" class="ltx_p">We note that causal tracing depends on a corruption rule to create baseline input for a model that does not contain all the information needed to make a prediction. Therefore we ask: are Causal Tracing results fragile if the exact form of the corruption changes? We test this by expanding the corruption rule: even when additional tokens after the subject name are also corrupted, we find that the results are substantially the same. Figure <a href="#A2.F12" title="Figure 12 ‣ B.4 Causal Tracing Examples and Further Insights ‣ Appendix B Causal Tracing ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">12</span></a> shows causal traces with the expanded corruption rule.
Figure <a href="#A2.F15" title="Figure 15 ‣ B.4 Causal Tracing Examples and Further Insights ‣ Appendix B Causal Tracing ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">15</span></a> similarly shows line plots with the expanded corruption rule.</p>
</div>
<div id="A2.SS4.p3" class="ltx_para">
<p id="A2.SS4.p3.2" class="ltx_p">We do find that the noise must be large enough to create large average total effects. For example, if noise with variance that is much smaller is used (for example if we set <math id="A2.SS4.p3.1.m1.1" class="ltx_Math" alttext="\sigma=\sigma_{t}" display="inline"><semantics id="A2.SS4.p3.1.m1.1a"><mrow id="A2.SS4.p3.1.m1.1.1" xref="A2.SS4.p3.1.m1.1.1.cmml"><mi id="A2.SS4.p3.1.m1.1.1.2" xref="A2.SS4.p3.1.m1.1.1.2.cmml">σ</mi><mo id="A2.SS4.p3.1.m1.1.1.1" xref="A2.SS4.p3.1.m1.1.1.1.cmml">=</mo><msub id="A2.SS4.p3.1.m1.1.1.3" xref="A2.SS4.p3.1.m1.1.1.3.cmml"><mi id="A2.SS4.p3.1.m1.1.1.3.2" xref="A2.SS4.p3.1.m1.1.1.3.2.cmml">σ</mi><mi id="A2.SS4.p3.1.m1.1.1.3.3" xref="A2.SS4.p3.1.m1.1.1.3.3.cmml">t</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="A2.SS4.p3.1.m1.1b"><apply id="A2.SS4.p3.1.m1.1.1.cmml" xref="A2.SS4.p3.1.m1.1.1"><eq id="A2.SS4.p3.1.m1.1.1.1.cmml" xref="A2.SS4.p3.1.m1.1.1.1"></eq><ci id="A2.SS4.p3.1.m1.1.1.2.cmml" xref="A2.SS4.p3.1.m1.1.1.2">𝜎</ci><apply id="A2.SS4.p3.1.m1.1.1.3.cmml" xref="A2.SS4.p3.1.m1.1.1.3"><csymbol cd="ambiguous" id="A2.SS4.p3.1.m1.1.1.3.1.cmml" xref="A2.SS4.p3.1.m1.1.1.3">subscript</csymbol><ci id="A2.SS4.p3.1.m1.1.1.3.2.cmml" xref="A2.SS4.p3.1.m1.1.1.3.2">𝜎</ci><ci id="A2.SS4.p3.1.m1.1.1.3.3.cmml" xref="A2.SS4.p3.1.m1.1.1.3.3">𝑡</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS4.p3.1.m1.1c">\sigma=\sigma_{t}</annotation></semantics></math>), average total effects become very small, and the small gap in the behavior between clean runs and corrupted run makes it difficult discern indirect effects of mediators. Similarly, if we use a uniform distribution where components range in <math id="A2.SS4.p3.2.m2.1" class="ltx_Math" alttext="\pm 3\sigma" display="inline"><semantics id="A2.SS4.p3.2.m2.1a"><mrow id="A2.SS4.p3.2.m2.1.1" xref="A2.SS4.p3.2.m2.1.1.cmml"><mo id="A2.SS4.p3.2.m2.1.1a" xref="A2.SS4.p3.2.m2.1.1.cmml">±</mo><mrow id="A2.SS4.p3.2.m2.1.1.2" xref="A2.SS4.p3.2.m2.1.1.2.cmml"><mn id="A2.SS4.p3.2.m2.1.1.2.2" xref="A2.SS4.p3.2.m2.1.1.2.2.cmml">3</mn><mo lspace="0em" rspace="0em" id="A2.SS4.p3.2.m2.1.1.2.1" xref="A2.SS4.p3.2.m2.1.1.2.1.cmml">​</mo><mi id="A2.SS4.p3.2.m2.1.1.2.3" xref="A2.SS4.p3.2.m2.1.1.2.3.cmml">σ</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.SS4.p3.2.m2.1b"><apply id="A2.SS4.p3.2.m2.1.1.cmml" xref="A2.SS4.p3.2.m2.1.1"><csymbol cd="latexml" id="A2.SS4.p3.2.m2.1.1.1.cmml" xref="A2.SS4.p3.2.m2.1.1">plus-or-minus</csymbol><apply id="A2.SS4.p3.2.m2.1.1.2.cmml" xref="A2.SS4.p3.2.m2.1.1.2"><times id="A2.SS4.p3.2.m2.1.1.2.1.cmml" xref="A2.SS4.p3.2.m2.1.1.2.1"></times><cn type="integer" id="A2.SS4.p3.2.m2.1.1.2.2.cmml" xref="A2.SS4.p3.2.m2.1.1.2.2">3</cn><ci id="A2.SS4.p3.2.m2.1.1.2.3.cmml" xref="A2.SS4.p3.2.m2.1.1.2.3">𝜎</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS4.p3.2.m2.1c">\pm 3\sigma</annotation></semantics></math>, effects large enough for causl tracing but smaller than a Gaussian distribution.</p>
</div>
<div id="A2.SS4.p4" class="ltx_para">
<p id="A2.SS4.p4.3" class="ltx_p">If instead of using spherical Gaussian noise, we draw noise from <math id="A2.SS4.p4.1.m1.2" class="ltx_Math" alttext="\mathcal{N}(mu,\Sigma)" display="inline"><semantics id="A2.SS4.p4.1.m1.2a"><mrow id="A2.SS4.p4.1.m1.2.2" xref="A2.SS4.p4.1.m1.2.2.cmml"><mi class="ltx_font_mathcaligraphic" id="A2.SS4.p4.1.m1.2.2.3" xref="A2.SS4.p4.1.m1.2.2.3.cmml">𝒩</mi><mo lspace="0em" rspace="0em" id="A2.SS4.p4.1.m1.2.2.2" xref="A2.SS4.p4.1.m1.2.2.2.cmml">​</mo><mrow id="A2.SS4.p4.1.m1.2.2.1.1" xref="A2.SS4.p4.1.m1.2.2.1.2.cmml"><mo stretchy="false" id="A2.SS4.p4.1.m1.2.2.1.1.2" xref="A2.SS4.p4.1.m1.2.2.1.2.cmml">(</mo><mrow id="A2.SS4.p4.1.m1.2.2.1.1.1" xref="A2.SS4.p4.1.m1.2.2.1.1.1.cmml"><mi id="A2.SS4.p4.1.m1.2.2.1.1.1.2" xref="A2.SS4.p4.1.m1.2.2.1.1.1.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="A2.SS4.p4.1.m1.2.2.1.1.1.1" xref="A2.SS4.p4.1.m1.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS4.p4.1.m1.2.2.1.1.1.3" xref="A2.SS4.p4.1.m1.2.2.1.1.1.3.cmml">u</mi></mrow><mo id="A2.SS4.p4.1.m1.2.2.1.1.3" xref="A2.SS4.p4.1.m1.2.2.1.2.cmml">,</mo><mi mathvariant="normal" id="A2.SS4.p4.1.m1.1.1" xref="A2.SS4.p4.1.m1.1.1.cmml">Σ</mi><mo stretchy="false" id="A2.SS4.p4.1.m1.2.2.1.1.4" xref="A2.SS4.p4.1.m1.2.2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.SS4.p4.1.m1.2b"><apply id="A2.SS4.p4.1.m1.2.2.cmml" xref="A2.SS4.p4.1.m1.2.2"><times id="A2.SS4.p4.1.m1.2.2.2.cmml" xref="A2.SS4.p4.1.m1.2.2.2"></times><ci id="A2.SS4.p4.1.m1.2.2.3.cmml" xref="A2.SS4.p4.1.m1.2.2.3">𝒩</ci><interval closure="open" id="A2.SS4.p4.1.m1.2.2.1.2.cmml" xref="A2.SS4.p4.1.m1.2.2.1.1"><apply id="A2.SS4.p4.1.m1.2.2.1.1.1.cmml" xref="A2.SS4.p4.1.m1.2.2.1.1.1"><times id="A2.SS4.p4.1.m1.2.2.1.1.1.1.cmml" xref="A2.SS4.p4.1.m1.2.2.1.1.1.1"></times><ci id="A2.SS4.p4.1.m1.2.2.1.1.1.2.cmml" xref="A2.SS4.p4.1.m1.2.2.1.1.1.2">𝑚</ci><ci id="A2.SS4.p4.1.m1.2.2.1.1.1.3.cmml" xref="A2.SS4.p4.1.m1.2.2.1.1.1.3">𝑢</ci></apply><ci id="A2.SS4.p4.1.m1.1.1.cmml" xref="A2.SS4.p4.1.m1.1.1">Σ</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS4.p4.1.m1.2c">\mathcal{N}(mu,\Sigma)</annotation></semantics></math> where we set <math id="A2.SS4.p4.2.m2.1" class="ltx_Math" alttext="\mu=\mu_{t}" display="inline"><semantics id="A2.SS4.p4.2.m2.1a"><mrow id="A2.SS4.p4.2.m2.1.1" xref="A2.SS4.p4.2.m2.1.1.cmml"><mi id="A2.SS4.p4.2.m2.1.1.2" xref="A2.SS4.p4.2.m2.1.1.2.cmml">μ</mi><mo id="A2.SS4.p4.2.m2.1.1.1" xref="A2.SS4.p4.2.m2.1.1.1.cmml">=</mo><msub id="A2.SS4.p4.2.m2.1.1.3" xref="A2.SS4.p4.2.m2.1.1.3.cmml"><mi id="A2.SS4.p4.2.m2.1.1.3.2" xref="A2.SS4.p4.2.m2.1.1.3.2.cmml">μ</mi><mi id="A2.SS4.p4.2.m2.1.1.3.3" xref="A2.SS4.p4.2.m2.1.1.3.3.cmml">t</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="A2.SS4.p4.2.m2.1b"><apply id="A2.SS4.p4.2.m2.1.1.cmml" xref="A2.SS4.p4.2.m2.1.1"><eq id="A2.SS4.p4.2.m2.1.1.1.cmml" xref="A2.SS4.p4.2.m2.1.1.1"></eq><ci id="A2.SS4.p4.2.m2.1.1.2.cmml" xref="A2.SS4.p4.2.m2.1.1.2">𝜇</ci><apply id="A2.SS4.p4.2.m2.1.1.3.cmml" xref="A2.SS4.p4.2.m2.1.1.3"><csymbol cd="ambiguous" id="A2.SS4.p4.2.m2.1.1.3.1.cmml" xref="A2.SS4.p4.2.m2.1.1.3">subscript</csymbol><ci id="A2.SS4.p4.2.m2.1.1.3.2.cmml" xref="A2.SS4.p4.2.m2.1.1.3.2">𝜇</ci><ci id="A2.SS4.p4.2.m2.1.1.3.3.cmml" xref="A2.SS4.p4.2.m2.1.1.3.3">𝑡</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS4.p4.2.m2.1c">\mu=\mu_{t}</annotation></semantics></math> and <math id="A2.SS4.p4.3.m3.1" class="ltx_Math" alttext="\Sigma_{=}\Sigma_{t}" display="inline"><semantics id="A2.SS4.p4.3.m3.1a"><mrow id="A2.SS4.p4.3.m3.1.1" xref="A2.SS4.p4.3.m3.1.1.cmml"><msub id="A2.SS4.p4.3.m3.1.1.2" xref="A2.SS4.p4.3.m3.1.1.2.cmml"><mi mathvariant="normal" id="A2.SS4.p4.3.m3.1.1.2.2" xref="A2.SS4.p4.3.m3.1.1.2.2.cmml">Σ</mi><mo id="A2.SS4.p4.3.m3.1.1.2.3" xref="A2.SS4.p4.3.m3.1.1.2.3.cmml">=</mo></msub><mo lspace="0em" rspace="0em" id="A2.SS4.p4.3.m3.1.1.1" xref="A2.SS4.p4.3.m3.1.1.1.cmml">​</mo><msub id="A2.SS4.p4.3.m3.1.1.3" xref="A2.SS4.p4.3.m3.1.1.3.cmml"><mi mathvariant="normal" id="A2.SS4.p4.3.m3.1.1.3.2" xref="A2.SS4.p4.3.m3.1.1.3.2.cmml">Σ</mi><mi id="A2.SS4.p4.3.m3.1.1.3.3" xref="A2.SS4.p4.3.m3.1.1.3.3.cmml">t</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="A2.SS4.p4.3.m3.1b"><apply id="A2.SS4.p4.3.m3.1.1.cmml" xref="A2.SS4.p4.3.m3.1.1"><times id="A2.SS4.p4.3.m3.1.1.1.cmml" xref="A2.SS4.p4.3.m3.1.1.1"></times><apply id="A2.SS4.p4.3.m3.1.1.2.cmml" xref="A2.SS4.p4.3.m3.1.1.2"><csymbol cd="ambiguous" id="A2.SS4.p4.3.m3.1.1.2.1.cmml" xref="A2.SS4.p4.3.m3.1.1.2">subscript</csymbol><ci id="A2.SS4.p4.3.m3.1.1.2.2.cmml" xref="A2.SS4.p4.3.m3.1.1.2.2">Σ</ci><eq id="A2.SS4.p4.3.m3.1.1.2.3.cmml" xref="A2.SS4.p4.3.m3.1.1.2.3"></eq></apply><apply id="A2.SS4.p4.3.m3.1.1.3.cmml" xref="A2.SS4.p4.3.m3.1.1.3"><csymbol cd="ambiguous" id="A2.SS4.p4.3.m3.1.1.3.1.cmml" xref="A2.SS4.p4.3.m3.1.1.3">subscript</csymbol><ci id="A2.SS4.p4.3.m3.1.1.3.2.cmml" xref="A2.SS4.p4.3.m3.1.1.3.2">Σ</ci><ci id="A2.SS4.p4.3.m3.1.1.3.3.cmml" xref="A2.SS4.p4.3.m3.1.1.3.3">𝑡</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS4.p4.3.m3.1c">\Sigma_{=}\Sigma_{t}</annotation></semantics></math> to match the observed distribution over token embeddings, average total effects are also strong enough to perform causal traces. This is shown in Figure <a href="#A2.F13" title="Figure 13 ‣ B.4 Causal Tracing Examples and Further Insights ‣ Appendix B Causal Tracing ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">13</span></a>.</p>
</div>
<div id="A2.SS4.p5" class="ltx_para">
<p id="A2.SS4.p5.1" class="ltx_p">Furthermore, we investigate whether Integrated Gradients (IG) <cite class="ltx_cite ltx_citemacro_citep">(Sundararajan et al., <a href="#bib.bib35" title="" class="ltx_ref">2017</a>)</cite> provides the same insights as Causal Tracing. We find that IG is very sensitive to local features but does not yield the same insights about large-scale global logic that we have been able to obtain using causal traces. Figure <a href="#A2.F16" title="Figure 16 ‣ B.4 Causal Tracing Examples and Further Insights ‣ Appendix B Causal Tracing ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">16</span></a> compares causal traces to IG saliency maps.</p>
</div>
<figure id="A2.F10" class="ltx_figure"><img src="/html/2202.05262/assets/x10.png" id="A2.F10.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="471" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A2.F10.3.1.1" class="ltx_text" style="font-size:90%;">Figure 10</span>: </span><span id="A2.F10.4.2" class="ltx_text" style="font-size:90%;">Further examples of causal traces showing appearance of the common lookup pattern on a variety of different types of facts about people and other kinds of entities. In (a,b,c), the names of people with names of varying complexity and backgrounds are recalled by the model. In each case, the MLP lookups on the last token of the name are decisive. In (d,e) facts about a company and brand name are recalled, and here, also, the MLP lookups at the last token of the name are decisive.</span></figcaption>
</figure>
<figure id="A2.F11" class="ltx_figure"><img src="/html/2202.05262/assets/x11.png" id="A2.F11.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="465" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A2.F11.3.1.1" class="ltx_text" style="font-size:90%;">Figure 11</span>: </span><span id="A2.F11.4.2" class="ltx_text" style="font-size:90%;">Causal traces show that the last token of the subject name is not always decisive. (a) shows a typical case: even though the name ‘NTFS’ is a spelled out acronym, the model does MLP lookups at the last letter of the name that are decisive when the model recalls the developer Microsoft. However, in a very similar sentence (b), we can see that the last words of ‘Windows Media Player’ are <em id="A2.F11.4.2.1" class="ltx_emph ltx_font_italic">not</em> decisive; the first word ‘Windows’ is the token that triggers the decisive lookup for information about the manufacturer. The information also seems to pass through the attention at the second token ‘Media’. Similarly in (c) we find that the Tokyo headquarters of ‘Mitsubishi Electric’ does not depend on the word ‘Electric’, and in (d) the location of death of Madame de Montesson seems to be mainly determined by the observed title ‘Madame’. In (e) we have a typical low-confidence trace, in which no runs of MLP lookups inside the subject name appear decisive; the model seems to particularly depend on the prompt word ‘performing’ to guess that the subject might play the piano.</span></figcaption>
</figure>
<figure id="A2.F12" class="ltx_figure"><img src="/html/2202.05262/assets/x12.png" id="A2.F12.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="471" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A2.F12.3.1.1" class="ltx_text" style="font-size:90%;">Figure 12</span>: </span><span id="A2.F12.4.2" class="ltx_text" style="font-size:90%;">Causal traces in the presence of additional corruption. Similar to Figure <a href="#A2.F10" title="Figure 10 ‣ B.4 Causal Tracing Examples and Further Insights ‣ Appendix B Causal Tracing ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">10</span></a>, but instead of corrupting only the subject token, these traces also corrupt the token after the subject. Causal effects are somewhat reduced due to the the model losing some ability to read the relation between the subject and object, but these traces continue to show concentrated causal effects at the last token of the subject even when the last token is not the last token corrupted. Causal effects of MLP layers at the last subject token continues to be pronounced.</span></figcaption>
</figure>
<figure id="A2.F13" class="ltx_figure"><img src="/html/2202.05262/assets/x13.png" id="A2.F13.g1" class="ltx_graphics ltx_img_landscape" width="461" height="261" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="A2.F13.16.8.1" class="ltx_text" style="font-size:90%;">Figure 13</span>: </span><span id="A2.F13.14.7" class="ltx_text" style="font-size:90%;">Comparing different noise choices. (Compare to Figure <a href="#A2.F7" title="Figure 7 ‣ B.2 Separating MLP and Attn Effects ‣ Appendix B Causal Tracing ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>, where noise is chosen as a <math id="A2.F13.8.1.m1.1" class="ltx_Math" alttext="3\sigma_{t}" display="inline"><semantics id="A2.F13.8.1.m1.1b"><mrow id="A2.F13.8.1.m1.1.1" xref="A2.F13.8.1.m1.1.1.cmml"><mn id="A2.F13.8.1.m1.1.1.2" xref="A2.F13.8.1.m1.1.1.2.cmml">3</mn><mo lspace="0em" rspace="0em" id="A2.F13.8.1.m1.1.1.1" xref="A2.F13.8.1.m1.1.1.1.cmml">​</mo><msub id="A2.F13.8.1.m1.1.1.3" xref="A2.F13.8.1.m1.1.1.3.cmml"><mi id="A2.F13.8.1.m1.1.1.3.2" xref="A2.F13.8.1.m1.1.1.3.2.cmml">σ</mi><mi id="A2.F13.8.1.m1.1.1.3.3" xref="A2.F13.8.1.m1.1.1.3.3.cmml">t</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="A2.F13.8.1.m1.1c"><apply id="A2.F13.8.1.m1.1.1.cmml" xref="A2.F13.8.1.m1.1.1"><times id="A2.F13.8.1.m1.1.1.1.cmml" xref="A2.F13.8.1.m1.1.1.1"></times><cn type="integer" id="A2.F13.8.1.m1.1.1.2.cmml" xref="A2.F13.8.1.m1.1.1.2">3</cn><apply id="A2.F13.8.1.m1.1.1.3.cmml" xref="A2.F13.8.1.m1.1.1.3"><csymbol cd="ambiguous" id="A2.F13.8.1.m1.1.1.3.1.cmml" xref="A2.F13.8.1.m1.1.1.3">subscript</csymbol><ci id="A2.F13.8.1.m1.1.1.3.2.cmml" xref="A2.F13.8.1.m1.1.1.3.2">𝜎</ci><ci id="A2.F13.8.1.m1.1.1.3.3.cmml" xref="A2.F13.8.1.m1.1.1.3.3">𝑡</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.F13.8.1.m1.1d">3\sigma_{t}</annotation></semantics></math> spherical Gaussian, where <math id="A2.F13.9.2.m2.1" class="ltx_Math" alttext="\sigma_{t}" display="inline"><semantics id="A2.F13.9.2.m2.1b"><msub id="A2.F13.9.2.m2.1.1" xref="A2.F13.9.2.m2.1.1.cmml"><mi id="A2.F13.9.2.m2.1.1.2" xref="A2.F13.9.2.m2.1.1.2.cmml">σ</mi><mi id="A2.F13.9.2.m2.1.1.3" xref="A2.F13.9.2.m2.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="A2.F13.9.2.m2.1c"><apply id="A2.F13.9.2.m2.1.1.cmml" xref="A2.F13.9.2.m2.1.1"><csymbol cd="ambiguous" id="A2.F13.9.2.m2.1.1.1.cmml" xref="A2.F13.9.2.m2.1.1">subscript</csymbol><ci id="A2.F13.9.2.m2.1.1.2.cmml" xref="A2.F13.9.2.m2.1.1.2">𝜎</ci><ci id="A2.F13.9.2.m2.1.1.3.cmml" xref="A2.F13.9.2.m2.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.F13.9.2.m2.1d">\sigma_{t}</annotation></semantics></math> is measured to match the observed spherical variance over tokens.) In a, b, c we we draw noise from a multivariate Gaussian <math id="A2.F13.10.3.m3.2" class="ltx_Math" alttext="\mathcal{N}(\mu;\Sigma)" display="inline"><semantics id="A2.F13.10.3.m3.2b"><mrow id="A2.F13.10.3.m3.2.3" xref="A2.F13.10.3.m3.2.3.cmml"><mi class="ltx_font_mathcaligraphic" id="A2.F13.10.3.m3.2.3.2" xref="A2.F13.10.3.m3.2.3.2.cmml">𝒩</mi><mo lspace="0em" rspace="0em" id="A2.F13.10.3.m3.2.3.1" xref="A2.F13.10.3.m3.2.3.1.cmml">​</mo><mrow id="A2.F13.10.3.m3.2.3.3.2" xref="A2.F13.10.3.m3.2.3.3.1.cmml"><mo stretchy="false" id="A2.F13.10.3.m3.2.3.3.2.1" xref="A2.F13.10.3.m3.2.3.3.1.cmml">(</mo><mi id="A2.F13.10.3.m3.1.1" xref="A2.F13.10.3.m3.1.1.cmml">μ</mi><mo id="A2.F13.10.3.m3.2.3.3.2.2" xref="A2.F13.10.3.m3.2.3.3.1.cmml">;</mo><mi mathvariant="normal" id="A2.F13.10.3.m3.2.2" xref="A2.F13.10.3.m3.2.2.cmml">Σ</mi><mo stretchy="false" id="A2.F13.10.3.m3.2.3.3.2.3" xref="A2.F13.10.3.m3.2.3.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.F13.10.3.m3.2c"><apply id="A2.F13.10.3.m3.2.3.cmml" xref="A2.F13.10.3.m3.2.3"><times id="A2.F13.10.3.m3.2.3.1.cmml" xref="A2.F13.10.3.m3.2.3.1"></times><ci id="A2.F13.10.3.m3.2.3.2.cmml" xref="A2.F13.10.3.m3.2.3.2">𝒩</ci><list id="A2.F13.10.3.m3.2.3.3.1.cmml" xref="A2.F13.10.3.m3.2.3.3.2"><ci id="A2.F13.10.3.m3.1.1.cmml" xref="A2.F13.10.3.m3.1.1">𝜇</ci><ci id="A2.F13.10.3.m3.2.2.cmml" xref="A2.F13.10.3.m3.2.2">Σ</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.F13.10.3.m3.2d">\mathcal{N}(\mu;\Sigma)</annotation></semantics></math> where <math id="A2.F13.11.4.m4.1" class="ltx_Math" alttext="\mu" display="inline"><semantics id="A2.F13.11.4.m4.1b"><mi id="A2.F13.11.4.m4.1.1" xref="A2.F13.11.4.m4.1.1.cmml">μ</mi><annotation-xml encoding="MathML-Content" id="A2.F13.11.4.m4.1c"><ci id="A2.F13.11.4.m4.1.1.cmml" xref="A2.F13.11.4.m4.1.1">𝜇</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.F13.11.4.m4.1d">\mu</annotation></semantics></math> and <math id="A2.F13.12.5.m5.1" class="ltx_Math" alttext="\Sigma" display="inline"><semantics id="A2.F13.12.5.m5.1b"><mi mathvariant="normal" id="A2.F13.12.5.m5.1.1" xref="A2.F13.12.5.m5.1.1.cmml">Σ</mi><annotation-xml encoding="MathML-Content" id="A2.F13.12.5.m5.1c"><ci id="A2.F13.12.5.m5.1.1.cmml" xref="A2.F13.12.5.m5.1.1">Σ</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.F13.12.5.m5.1d">\Sigma</annotation></semantics></math> are chosen to match the observed mean and covariance over a sample of tokens. In d, e, f we draw noise from a uniform distribution in the range <math id="A2.F13.13.6.m6.1" class="ltx_Math" alttext="\pm 3\sigma" display="inline"><semantics id="A2.F13.13.6.m6.1b"><mrow id="A2.F13.13.6.m6.1.1" xref="A2.F13.13.6.m6.1.1.cmml"><mo id="A2.F13.13.6.m6.1.1b" xref="A2.F13.13.6.m6.1.1.cmml">±</mo><mrow id="A2.F13.13.6.m6.1.1.2" xref="A2.F13.13.6.m6.1.1.2.cmml"><mn id="A2.F13.13.6.m6.1.1.2.2" xref="A2.F13.13.6.m6.1.1.2.2.cmml">3</mn><mo lspace="0em" rspace="0em" id="A2.F13.13.6.m6.1.1.2.1" xref="A2.F13.13.6.m6.1.1.2.1.cmml">​</mo><mi id="A2.F13.13.6.m6.1.1.2.3" xref="A2.F13.13.6.m6.1.1.2.3.cmml">σ</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.F13.13.6.m6.1c"><apply id="A2.F13.13.6.m6.1.1.cmml" xref="A2.F13.13.6.m6.1.1"><csymbol cd="latexml" id="A2.F13.13.6.m6.1.1.1.cmml" xref="A2.F13.13.6.m6.1.1">plus-or-minus</csymbol><apply id="A2.F13.13.6.m6.1.1.2.cmml" xref="A2.F13.13.6.m6.1.1.2"><times id="A2.F13.13.6.m6.1.1.2.1.cmml" xref="A2.F13.13.6.m6.1.1.2.1"></times><cn type="integer" id="A2.F13.13.6.m6.1.1.2.2.cmml" xref="A2.F13.13.6.m6.1.1.2.2">3</cn><ci id="A2.F13.13.6.m6.1.1.2.3.cmml" xref="A2.F13.13.6.m6.1.1.2.3">𝜎</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.F13.13.6.m6.1d">\pm 3\sigma</annotation></semantics></math> instead of a Gaussian distribution. In both cases, the average total effects measured between the clean run and the corrupted run are large enough to measure causal traces, but the effects are smaller than the choice of <math id="A2.F13.14.7.m7.1" class="ltx_Math" alttext="3\sigma_{t}" display="inline"><semantics id="A2.F13.14.7.m7.1b"><mrow id="A2.F13.14.7.m7.1.1" xref="A2.F13.14.7.m7.1.1.cmml"><mn id="A2.F13.14.7.m7.1.1.2" xref="A2.F13.14.7.m7.1.1.2.cmml">3</mn><mo lspace="0em" rspace="0em" id="A2.F13.14.7.m7.1.1.1" xref="A2.F13.14.7.m7.1.1.1.cmml">​</mo><msub id="A2.F13.14.7.m7.1.1.3" xref="A2.F13.14.7.m7.1.1.3.cmml"><mi id="A2.F13.14.7.m7.1.1.3.2" xref="A2.F13.14.7.m7.1.1.3.2.cmml">σ</mi><mi id="A2.F13.14.7.m7.1.1.3.3" xref="A2.F13.14.7.m7.1.1.3.3.cmml">t</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="A2.F13.14.7.m7.1c"><apply id="A2.F13.14.7.m7.1.1.cmml" xref="A2.F13.14.7.m7.1.1"><times id="A2.F13.14.7.m7.1.1.1.cmml" xref="A2.F13.14.7.m7.1.1.1"></times><cn type="integer" id="A2.F13.14.7.m7.1.1.2.cmml" xref="A2.F13.14.7.m7.1.1.2">3</cn><apply id="A2.F13.14.7.m7.1.1.3.cmml" xref="A2.F13.14.7.m7.1.1.3"><csymbol cd="ambiguous" id="A2.F13.14.7.m7.1.1.3.1.cmml" xref="A2.F13.14.7.m7.1.1.3">subscript</csymbol><ci id="A2.F13.14.7.m7.1.1.3.2.cmml" xref="A2.F13.14.7.m7.1.1.3.2">𝜎</ci><ci id="A2.F13.14.7.m7.1.1.3.3.cmml" xref="A2.F13.14.7.m7.1.1.3.3">𝑡</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.F13.14.7.m7.1d">3\sigma_{t}</annotation></semantics></math> used in the main paper.</span></figcaption>
</figure>
<figure id="A2.F14" class="ltx_figure"><img src="/html/2202.05262/assets/x14.png" id="A2.F14.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="366" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A2.F14.3.1.1" class="ltx_text" style="font-size:90%;">Figure 14</span>: </span><span id="A2.F14.4.2" class="ltx_text" style="font-size:90%;">Detail view of causal traces, breaking out a representative set of individual cases from the 1000 factual statements that are averaged in Figure <a href="#S2.F3" title="Figure 3 ‣ 2.2 Causal Tracing Results ‣ 2 Interventions on Activations for Tracing Information Flow ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>. Shows the causal trace at a specific subject token, with and without MLP disabled, as described in Section <a href="#S2" title="2 Interventions on Activations for Tracing Information Flow ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>. In every case, the token tested is highlighted in a red box. In (a,b,c,d,e) cases are shown that fit the typical pattern: Restoring individual hidden states at a range of layers has a strong decisive average causal effect at the last token of the subject. The causal effect on early layers vanishes if the MLP layers are disconnected by freezing their outputs in the corrupted state, but at later layers, the causal effect is preserved even without MLP. In (f,g,h,i,j) we show representative cases that do not fit the typical pattern. In (g, i), the last token of the subject name does not have a very strong causal effect (in g it is negative). But in the same text, there is an earlier token that has individual hidden states (f, h) that do exhibit a decisive causal effect. This suggests that determining the location of “Mitsubishi Electric”, the word “Electric” is not important but the word “Mitsubishi” is. Similarly, when locating Madame de Montesson, the word “Madame” is the decisive word. (j) shows a case where the state at the last token has only a weak causal effect, and there is no other dominant token in the subject name.
</span></figcaption>
</figure>
<figure id="A2.F15" class="ltx_figure"><img src="/html/2202.05262/assets/x15.png" id="A2.F15.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="116" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A2.F15.3.1.1" class="ltx_text" style="font-size:90%;">Figure 15</span>: </span><span id="A2.F15.4.2" class="ltx_text" style="font-size:90%;">Similar to Figure <a href="#A2.F7" title="Figure 7 ‣ B.2 Separating MLP and Attn Effects ‣ Appendix B Causal Tracing ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>, but with an additional token corrupted after the subject token, as in Figure <a href="#A2.F12" title="Figure 12 ‣ B.4 Causal Tracing Examples and Further Insights ‣ Appendix B Causal Tracing ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">12</span></a>. We observe that the emergence of strong early-site causal effects at the MLP modules is systematic and appears under a different corruption scheme, confirming that importance of the last subject token is apparent even when the last subject token is never the last token corrupted.</span></figcaption>
</figure>
<figure id="A2.F16" class="ltx_figure"><img src="/html/2202.05262/assets/x16.png" id="A2.F16.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="473" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A2.F16.8.4.1" class="ltx_text" style="font-size:90%;">Figure 16</span>: </span><span id="A2.F16.6.3" class="ltx_text" style="font-size:90%;">Integrated gradients saliency maps, visualizing the same cases as in Figure <a href="#A2.F10" title="Figure 10 ‣ B.4 Causal Tracing Examples and Further Insights ‣ Appendix B Causal Tracing ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">10</span></a>. Here we compare Causal Tracing to the method of Integrated Gradients <cite class="ltx_cite ltx_citemacro_citep">(Sundararajan et al., <a href="#bib.bib35" title="" class="ltx_ref">2017</a>)</cite>. Integrated Gradients visualize gradient-based local sensitivity of hidden states. Here we compute IG using 50 steps of Gauss-Legendre quadrature on gradients of individual hidden states <math id="A2.F16.4.1.m1.1" class="ltx_Math" alttext="\smash{h_{t}^{(l)}}" display="inline"><semantics id="A2.F16.4.1.m1.1b"><msubsup id="A2.F16.4.1.m1.1.2" xref="A2.F16.4.1.m1.1.2.cmml"><mi id="A2.F16.4.1.m1.1.2.2.2" xref="A2.F16.4.1.m1.1.2.2.2.cmml">h</mi><mi id="A2.F16.4.1.m1.1.2.2.3" xref="A2.F16.4.1.m1.1.2.2.3.cmml">t</mi><mrow id="A2.F16.4.1.m1.1.1.1.3" xref="A2.F16.4.1.m1.1.2.cmml"><mo stretchy="false" id="A2.F16.4.1.m1.1.1.1.3.1" xref="A2.F16.4.1.m1.1.2.cmml">(</mo><mi id="A2.F16.4.1.m1.1.1.1.1" xref="A2.F16.4.1.m1.1.1.1.1.cmml">l</mi><mo stretchy="false" id="A2.F16.4.1.m1.1.1.1.3.2" xref="A2.F16.4.1.m1.1.2.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="A2.F16.4.1.m1.1c"><apply id="A2.F16.4.1.m1.1.2.cmml" xref="A2.F16.4.1.m1.1.2"><csymbol cd="ambiguous" id="A2.F16.4.1.m1.1.2.1.cmml" xref="A2.F16.4.1.m1.1.2">superscript</csymbol><apply id="A2.F16.4.1.m1.1.2.2.cmml" xref="A2.F16.4.1.m1.1.2"><csymbol cd="ambiguous" id="A2.F16.4.1.m1.1.2.2.1.cmml" xref="A2.F16.4.1.m1.1.2">subscript</csymbol><ci id="A2.F16.4.1.m1.1.2.2.2.cmml" xref="A2.F16.4.1.m1.1.2.2.2">ℎ</ci><ci id="A2.F16.4.1.m1.1.2.2.3.cmml" xref="A2.F16.4.1.m1.1.2.2.3">𝑡</ci></apply><ci id="A2.F16.4.1.m1.1.1.1.1.cmml" xref="A2.F16.4.1.m1.1.1.1.1">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.F16.4.1.m1.1d">\smash{h_{t}^{(l)}}</annotation></semantics></math>, or <math id="A2.F16.5.2.m2.1" class="ltx_Math" alttext="\smash{m_{t}^{(l)}}" display="inline"><semantics id="A2.F16.5.2.m2.1b"><msubsup id="A2.F16.5.2.m2.1.2" xref="A2.F16.5.2.m2.1.2.cmml"><mi id="A2.F16.5.2.m2.1.2.2.2" xref="A2.F16.5.2.m2.1.2.2.2.cmml">m</mi><mi id="A2.F16.5.2.m2.1.2.2.3" xref="A2.F16.5.2.m2.1.2.2.3.cmml">t</mi><mrow id="A2.F16.5.2.m2.1.1.1.3" xref="A2.F16.5.2.m2.1.2.cmml"><mo stretchy="false" id="A2.F16.5.2.m2.1.1.1.3.1" xref="A2.F16.5.2.m2.1.2.cmml">(</mo><mi id="A2.F16.5.2.m2.1.1.1.1" xref="A2.F16.5.2.m2.1.1.1.1.cmml">l</mi><mo stretchy="false" id="A2.F16.5.2.m2.1.1.1.3.2" xref="A2.F16.5.2.m2.1.2.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="A2.F16.5.2.m2.1c"><apply id="A2.F16.5.2.m2.1.2.cmml" xref="A2.F16.5.2.m2.1.2"><csymbol cd="ambiguous" id="A2.F16.5.2.m2.1.2.1.cmml" xref="A2.F16.5.2.m2.1.2">superscript</csymbol><apply id="A2.F16.5.2.m2.1.2.2.cmml" xref="A2.F16.5.2.m2.1.2"><csymbol cd="ambiguous" id="A2.F16.5.2.m2.1.2.2.1.cmml" xref="A2.F16.5.2.m2.1.2">subscript</csymbol><ci id="A2.F16.5.2.m2.1.2.2.2.cmml" xref="A2.F16.5.2.m2.1.2.2.2">𝑚</ci><ci id="A2.F16.5.2.m2.1.2.2.3.cmml" xref="A2.F16.5.2.m2.1.2.2.3">𝑡</ci></apply><ci id="A2.F16.5.2.m2.1.1.1.1.cmml" xref="A2.F16.5.2.m2.1.1.1.1">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.F16.5.2.m2.1d">\smash{m_{t}^{(l)}}</annotation></semantics></math> (for MLP), or <math id="A2.F16.6.3.m3.1" class="ltx_Math" alttext="\smash{a_{t}^{(l)}}" display="inline"><semantics id="A2.F16.6.3.m3.1b"><msubsup id="A2.F16.6.3.m3.1.2" xref="A2.F16.6.3.m3.1.2.cmml"><mi id="A2.F16.6.3.m3.1.2.2.2" xref="A2.F16.6.3.m3.1.2.2.2.cmml">a</mi><mi id="A2.F16.6.3.m3.1.2.2.3" xref="A2.F16.6.3.m3.1.2.2.3.cmml">t</mi><mrow id="A2.F16.6.3.m3.1.1.1.3" xref="A2.F16.6.3.m3.1.2.cmml"><mo stretchy="false" id="A2.F16.6.3.m3.1.1.1.3.1" xref="A2.F16.6.3.m3.1.2.cmml">(</mo><mi id="A2.F16.6.3.m3.1.1.1.1" xref="A2.F16.6.3.m3.1.1.1.1.cmml">l</mi><mo stretchy="false" id="A2.F16.6.3.m3.1.1.1.3.2" xref="A2.F16.6.3.m3.1.2.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="A2.F16.6.3.m3.1c"><apply id="A2.F16.6.3.m3.1.2.cmml" xref="A2.F16.6.3.m3.1.2"><csymbol cd="ambiguous" id="A2.F16.6.3.m3.1.2.1.cmml" xref="A2.F16.6.3.m3.1.2">superscript</csymbol><apply id="A2.F16.6.3.m3.1.2.2.cmml" xref="A2.F16.6.3.m3.1.2"><csymbol cd="ambiguous" id="A2.F16.6.3.m3.1.2.2.1.cmml" xref="A2.F16.6.3.m3.1.2">subscript</csymbol><ci id="A2.F16.6.3.m3.1.2.2.2.cmml" xref="A2.F16.6.3.m3.1.2.2.2">𝑎</ci><ci id="A2.F16.6.3.m3.1.2.2.3.cmml" xref="A2.F16.6.3.m3.1.2.2.3">𝑡</ci></apply><ci id="A2.F16.6.3.m3.1.1.1.1.cmml" xref="A2.F16.6.3.m3.1.1.1.1">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.F16.6.3.m3.1d">\smash{a_{t}^{(l)}}</annotation></semantics></math> (for Attn), with respect to the predicted output token; we plot the norm of the integrated gradient at each state. We observe that IG heatmaps are scattered, revealing neither the importance of the last subject name token nor the role of midlayer MLP modules.</span></figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Details on the zsRE Evaluation Task</h2>

<div id="A3.p1" class="ltx_para">
<p id="A3.p1.6" class="ltx_p"><span id="A3.p1.6.1" class="ltx_text ltx_font_bold">Dataset Details.</span> The zsRE question answering task <cite class="ltx_cite ltx_citemacro_citep">(Levy et al., <a href="#bib.bib24" title="" class="ltx_ref">2017</a>)</cite> was first used for factual knowledge evaluation by <cite class="ltx_cite ltx_citemacro_citet">De Cao et al. (<a href="#bib.bib10" title="" class="ltx_ref">2021</a>)</cite>, later being extended and adopted by <cite class="ltx_cite ltx_citemacro_citet">Mitchell et al. (<a href="#bib.bib27" title="" class="ltx_ref">2021</a>)</cite>. In our study, we use the same train/test splits as <cite class="ltx_cite ltx_citemacro_citet">Mitchell et al. (<a href="#bib.bib27" title="" class="ltx_ref">2021</a>)</cite>; note that non-hypernetwork methods (including ROME) do not require training, so the corresponding dataset split is discarded in those cases.
Each record in the zsRE dataset contains a factual statement <math id="A3.p1.1.m1.1" class="ltx_Math" alttext="t^{*}" display="inline"><semantics id="A3.p1.1.m1.1a"><msup id="A3.p1.1.m1.1.1" xref="A3.p1.1.m1.1.1.cmml"><mi id="A3.p1.1.m1.1.1.2" xref="A3.p1.1.m1.1.1.2.cmml">t</mi><mo id="A3.p1.1.m1.1.1.3" xref="A3.p1.1.m1.1.1.3.cmml">∗</mo></msup><annotation-xml encoding="MathML-Content" id="A3.p1.1.m1.1b"><apply id="A3.p1.1.m1.1.1.cmml" xref="A3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A3.p1.1.m1.1.1.1.cmml" xref="A3.p1.1.m1.1.1">superscript</csymbol><ci id="A3.p1.1.m1.1.1.2.cmml" xref="A3.p1.1.m1.1.1.2">𝑡</ci><times id="A3.p1.1.m1.1.1.3.cmml" xref="A3.p1.1.m1.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p1.1.m1.1c">t^{*}</annotation></semantics></math>, paraphrase prompts <math id="A3.p1.2.m2.1" class="ltx_Math" alttext="P^{P}" display="inline"><semantics id="A3.p1.2.m2.1a"><msup id="A3.p1.2.m2.1.1" xref="A3.p1.2.m2.1.1.cmml"><mi id="A3.p1.2.m2.1.1.2" xref="A3.p1.2.m2.1.1.2.cmml">P</mi><mi id="A3.p1.2.m2.1.1.3" xref="A3.p1.2.m2.1.1.3.cmml">P</mi></msup><annotation-xml encoding="MathML-Content" id="A3.p1.2.m2.1b"><apply id="A3.p1.2.m2.1.1.cmml" xref="A3.p1.2.m2.1.1"><csymbol cd="ambiguous" id="A3.p1.2.m2.1.1.1.cmml" xref="A3.p1.2.m2.1.1">superscript</csymbol><ci id="A3.p1.2.m2.1.1.2.cmml" xref="A3.p1.2.m2.1.1.2">𝑃</ci><ci id="A3.p1.2.m2.1.1.3.cmml" xref="A3.p1.2.m2.1.1.3">𝑃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p1.2.m2.1c">P^{P}</annotation></semantics></math>, and neighborhood prompts <math id="A3.p1.3.m3.1" class="ltx_Math" alttext="P^{N}" display="inline"><semantics id="A3.p1.3.m3.1a"><msup id="A3.p1.3.m3.1.1" xref="A3.p1.3.m3.1.1.cmml"><mi id="A3.p1.3.m3.1.1.2" xref="A3.p1.3.m3.1.1.2.cmml">P</mi><mi id="A3.p1.3.m3.1.1.3" xref="A3.p1.3.m3.1.1.3.cmml">N</mi></msup><annotation-xml encoding="MathML-Content" id="A3.p1.3.m3.1b"><apply id="A3.p1.3.m3.1.1.cmml" xref="A3.p1.3.m3.1.1"><csymbol cd="ambiguous" id="A3.p1.3.m3.1.1.1.cmml" xref="A3.p1.3.m3.1.1">superscript</csymbol><ci id="A3.p1.3.m3.1.1.2.cmml" xref="A3.p1.3.m3.1.1.2">𝑃</ci><ci id="A3.p1.3.m3.1.1.3.cmml" xref="A3.p1.3.m3.1.1.3">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p1.3.m3.1c">P^{N}</annotation></semantics></math>. <math id="A3.p1.4.m4.1" class="ltx_Math" alttext="t^{*}" display="inline"><semantics id="A3.p1.4.m4.1a"><msup id="A3.p1.4.m4.1.1" xref="A3.p1.4.m4.1.1.cmml"><mi id="A3.p1.4.m4.1.1.2" xref="A3.p1.4.m4.1.1.2.cmml">t</mi><mo id="A3.p1.4.m4.1.1.3" xref="A3.p1.4.m4.1.1.3.cmml">∗</mo></msup><annotation-xml encoding="MathML-Content" id="A3.p1.4.m4.1b"><apply id="A3.p1.4.m4.1.1.cmml" xref="A3.p1.4.m4.1.1"><csymbol cd="ambiguous" id="A3.p1.4.m4.1.1.1.cmml" xref="A3.p1.4.m4.1.1">superscript</csymbol><ci id="A3.p1.4.m4.1.1.2.cmml" xref="A3.p1.4.m4.1.1.2">𝑡</ci><times id="A3.p1.4.m4.1.1.3.cmml" xref="A3.p1.4.m4.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p1.4.m4.1c">t^{*}</annotation></semantics></math> and <math id="A3.p1.5.m5.1" class="ltx_Math" alttext="P^{N}" display="inline"><semantics id="A3.p1.5.m5.1a"><msup id="A3.p1.5.m5.1.1" xref="A3.p1.5.m5.1.1.cmml"><mi id="A3.p1.5.m5.1.1.2" xref="A3.p1.5.m5.1.1.2.cmml">P</mi><mi id="A3.p1.5.m5.1.1.3" xref="A3.p1.5.m5.1.1.3.cmml">N</mi></msup><annotation-xml encoding="MathML-Content" id="A3.p1.5.m5.1b"><apply id="A3.p1.5.m5.1.1.cmml" xref="A3.p1.5.m5.1.1"><csymbol cd="ambiguous" id="A3.p1.5.m5.1.1.1.cmml" xref="A3.p1.5.m5.1.1">superscript</csymbol><ci id="A3.p1.5.m5.1.1.2.cmml" xref="A3.p1.5.m5.1.1.2">𝑃</ci><ci id="A3.p1.5.m5.1.1.3.cmml" xref="A3.p1.5.m5.1.1.3">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p1.5.m5.1c">P^{N}</annotation></semantics></math> were included in the original version of zsRE, whereas <math id="A3.p1.6.m6.1" class="ltx_Math" alttext="P^{N}" display="inline"><semantics id="A3.p1.6.m6.1a"><msup id="A3.p1.6.m6.1.1" xref="A3.p1.6.m6.1.1.cmml"><mi id="A3.p1.6.m6.1.1.2" xref="A3.p1.6.m6.1.1.2.cmml">P</mi><mi id="A3.p1.6.m6.1.1.3" xref="A3.p1.6.m6.1.1.3.cmml">N</mi></msup><annotation-xml encoding="MathML-Content" id="A3.p1.6.m6.1b"><apply id="A3.p1.6.m6.1.1.cmml" xref="A3.p1.6.m6.1.1"><csymbol cd="ambiguous" id="A3.p1.6.m6.1.1.1.cmml" xref="A3.p1.6.m6.1.1">superscript</csymbol><ci id="A3.p1.6.m6.1.1.2.cmml" xref="A3.p1.6.m6.1.1.2">𝑃</ci><ci id="A3.p1.6.m6.1.1.3.cmml" xref="A3.p1.6.m6.1.1.3">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p1.6.m6.1c">P^{N}</annotation></semantics></math> was added by <cite class="ltx_cite ltx_citemacro_citet">Mitchell et al. (<a href="#bib.bib27" title="" class="ltx_ref">2021</a>)</cite> via sampling of a random dataset element. See Figure <a href="#A8.F22" title="Figure 22 ‣ Appendix H Dataset Samples ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">22</span></a> for an example record.</p>
</div>
<div id="A3.p2" class="ltx_para">
<p id="A3.p2.1" class="ltx_p"><span id="A3.p2.1.1" class="ltx_text ltx_font_bold">Additional Baselines.</span> In addition to baselines that are used as-is out of the box, we train two additional models, KE-zsRE and MEND-zsRE, which are the base GPT-2 XL editing hypernetworks custom-tuned on the zsRE training split. This is done to ensure fair comparison; the original pre-trained KE and MEND models were created using a WikiText generation task <cite class="ltx_cite ltx_citemacro_citep">(De Cao et al., <a href="#bib.bib10" title="" class="ltx_ref">2021</a>; Mitchell et al., <a href="#bib.bib27" title="" class="ltx_ref">2021</a>)</cite>, rather than zsRE.</p>
</div>
</section>
<section id="A4" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>Details on the <span id="A4.1.1" class="ltx_text ltx_font_smallcaps">CounterFact</span> Dataset</h2>

<div id="A4.p1" class="ltx_para">
<p id="A4.p1.1" class="ltx_p"><span id="A4.p1.1.1" class="ltx_text ltx_font_smallcaps">CounterFact</span> is designed to enable distinction between superficial changes in model word choices from specific and generalized changes in underlying factual knowledge. Table <a href="#S3.T3" title="Table 3 ‣ 3.3 Evaluating ROME: Our CounterFact Dataset ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> summarizes statistics about <span id="A4.p1.1.2" class="ltx_text ltx_font_smallcaps">CounterFact</span>’s composition.</p>
</div>
<div id="A4.p2" class="ltx_para">
<p id="A4.p2.5" class="ltx_p">Each record in <span id="A4.p2.5.1" class="ltx_text ltx_font_smallcaps">CounterFact</span> is derived from a corresponding entry in <span id="A4.p2.5.2" class="ltx_text ltx_font_smallcaps">ParaRel</span> <cite class="ltx_cite ltx_citemacro_citep">(Elazar et al., <a href="#bib.bib12" title="" class="ltx_ref">2021a</a>)</cite> containing a knowledge tuple <math id="A4.p2.1.m1.3" class="ltx_Math" alttext="t^{c}=(s,r,o^{c})" display="inline"><semantics id="A4.p2.1.m1.3a"><mrow id="A4.p2.1.m1.3.3" xref="A4.p2.1.m1.3.3.cmml"><msup id="A4.p2.1.m1.3.3.3" xref="A4.p2.1.m1.3.3.3.cmml"><mi id="A4.p2.1.m1.3.3.3.2" xref="A4.p2.1.m1.3.3.3.2.cmml">t</mi><mi id="A4.p2.1.m1.3.3.3.3" xref="A4.p2.1.m1.3.3.3.3.cmml">c</mi></msup><mo id="A4.p2.1.m1.3.3.2" xref="A4.p2.1.m1.3.3.2.cmml">=</mo><mrow id="A4.p2.1.m1.3.3.1.1" xref="A4.p2.1.m1.3.3.1.2.cmml"><mo stretchy="false" id="A4.p2.1.m1.3.3.1.1.2" xref="A4.p2.1.m1.3.3.1.2.cmml">(</mo><mi id="A4.p2.1.m1.1.1" xref="A4.p2.1.m1.1.1.cmml">s</mi><mo id="A4.p2.1.m1.3.3.1.1.3" xref="A4.p2.1.m1.3.3.1.2.cmml">,</mo><mi id="A4.p2.1.m1.2.2" xref="A4.p2.1.m1.2.2.cmml">r</mi><mo id="A4.p2.1.m1.3.3.1.1.4" xref="A4.p2.1.m1.3.3.1.2.cmml">,</mo><msup id="A4.p2.1.m1.3.3.1.1.1" xref="A4.p2.1.m1.3.3.1.1.1.cmml"><mi id="A4.p2.1.m1.3.3.1.1.1.2" xref="A4.p2.1.m1.3.3.1.1.1.2.cmml">o</mi><mi id="A4.p2.1.m1.3.3.1.1.1.3" xref="A4.p2.1.m1.3.3.1.1.1.3.cmml">c</mi></msup><mo stretchy="false" id="A4.p2.1.m1.3.3.1.1.5" xref="A4.p2.1.m1.3.3.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.p2.1.m1.3b"><apply id="A4.p2.1.m1.3.3.cmml" xref="A4.p2.1.m1.3.3"><eq id="A4.p2.1.m1.3.3.2.cmml" xref="A4.p2.1.m1.3.3.2"></eq><apply id="A4.p2.1.m1.3.3.3.cmml" xref="A4.p2.1.m1.3.3.3"><csymbol cd="ambiguous" id="A4.p2.1.m1.3.3.3.1.cmml" xref="A4.p2.1.m1.3.3.3">superscript</csymbol><ci id="A4.p2.1.m1.3.3.3.2.cmml" xref="A4.p2.1.m1.3.3.3.2">𝑡</ci><ci id="A4.p2.1.m1.3.3.3.3.cmml" xref="A4.p2.1.m1.3.3.3.3">𝑐</ci></apply><vector id="A4.p2.1.m1.3.3.1.2.cmml" xref="A4.p2.1.m1.3.3.1.1"><ci id="A4.p2.1.m1.1.1.cmml" xref="A4.p2.1.m1.1.1">𝑠</ci><ci id="A4.p2.1.m1.2.2.cmml" xref="A4.p2.1.m1.2.2">𝑟</ci><apply id="A4.p2.1.m1.3.3.1.1.1.cmml" xref="A4.p2.1.m1.3.3.1.1.1"><csymbol cd="ambiguous" id="A4.p2.1.m1.3.3.1.1.1.1.cmml" xref="A4.p2.1.m1.3.3.1.1.1">superscript</csymbol><ci id="A4.p2.1.m1.3.3.1.1.1.2.cmml" xref="A4.p2.1.m1.3.3.1.1.1.2">𝑜</ci><ci id="A4.p2.1.m1.3.3.1.1.1.3.cmml" xref="A4.p2.1.m1.3.3.1.1.1.3">𝑐</ci></apply></vector></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.1.m1.3c">t^{c}=(s,r,o^{c})</annotation></semantics></math> and hand-curated prompt templates <math id="A4.p2.2.m2.1" class="ltx_Math" alttext="\mathcal{T}(r)" display="inline"><semantics id="A4.p2.2.m2.1a"><mrow id="A4.p2.2.m2.1.2" xref="A4.p2.2.m2.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="A4.p2.2.m2.1.2.2" xref="A4.p2.2.m2.1.2.2.cmml">𝒯</mi><mo lspace="0em" rspace="0em" id="A4.p2.2.m2.1.2.1" xref="A4.p2.2.m2.1.2.1.cmml">​</mo><mrow id="A4.p2.2.m2.1.2.3.2" xref="A4.p2.2.m2.1.2.cmml"><mo stretchy="false" id="A4.p2.2.m2.1.2.3.2.1" xref="A4.p2.2.m2.1.2.cmml">(</mo><mi id="A4.p2.2.m2.1.1" xref="A4.p2.2.m2.1.1.cmml">r</mi><mo stretchy="false" id="A4.p2.2.m2.1.2.3.2.2" xref="A4.p2.2.m2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.p2.2.m2.1b"><apply id="A4.p2.2.m2.1.2.cmml" xref="A4.p2.2.m2.1.2"><times id="A4.p2.2.m2.1.2.1.cmml" xref="A4.p2.2.m2.1.2.1"></times><ci id="A4.p2.2.m2.1.2.2.cmml" xref="A4.p2.2.m2.1.2.2">𝒯</ci><ci id="A4.p2.2.m2.1.1.cmml" xref="A4.p2.2.m2.1.1">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.2.m2.1c">\mathcal{T}(r)</annotation></semantics></math>, where all subjects, relations, and objects exist as entities in WikiData. Note that prompt templates are unique only to <span id="A4.p2.5.3" class="ltx_text ltx_font_italic">relations</span>; entities can be substituted to form full prompts: <math id="A4.p2.3.m3.5" class="ltx_Math" alttext="\mathcal{P}(s,r):=\{\texttt{t.format(s)}\mid\texttt{t}\in\mathcal{T}(r)\}" display="inline"><semantics id="A4.p2.3.m3.5a"><mrow id="A4.p2.3.m3.5.5" xref="A4.p2.3.m3.5.5.cmml"><mrow id="A4.p2.3.m3.5.5.3" xref="A4.p2.3.m3.5.5.3.cmml"><mi class="ltx_font_mathcaligraphic" id="A4.p2.3.m3.5.5.3.2" xref="A4.p2.3.m3.5.5.3.2.cmml">𝒫</mi><mo lspace="0em" rspace="0em" id="A4.p2.3.m3.5.5.3.1" xref="A4.p2.3.m3.5.5.3.1.cmml">​</mo><mrow id="A4.p2.3.m3.5.5.3.3.2" xref="A4.p2.3.m3.5.5.3.3.1.cmml"><mo stretchy="false" id="A4.p2.3.m3.5.5.3.3.2.1" xref="A4.p2.3.m3.5.5.3.3.1.cmml">(</mo><mi id="A4.p2.3.m3.1.1" xref="A4.p2.3.m3.1.1.cmml">s</mi><mo id="A4.p2.3.m3.5.5.3.3.2.2" xref="A4.p2.3.m3.5.5.3.3.1.cmml">,</mo><mi id="A4.p2.3.m3.2.2" xref="A4.p2.3.m3.2.2.cmml">r</mi><mo rspace="0.278em" stretchy="false" id="A4.p2.3.m3.5.5.3.3.2.3" xref="A4.p2.3.m3.5.5.3.3.1.cmml">)</mo></mrow></mrow><mo rspace="0.278em" id="A4.p2.3.m3.5.5.2" xref="A4.p2.3.m3.5.5.2.cmml">:=</mo><mrow id="A4.p2.3.m3.5.5.1.1" xref="A4.p2.3.m3.5.5.1.2.cmml"><mo stretchy="false" id="A4.p2.3.m3.5.5.1.1.2" xref="A4.p2.3.m3.5.5.1.2.1.cmml">{</mo><mtext class="ltx_mathvariant_monospace" id="A4.p2.3.m3.4.4" xref="A4.p2.3.m3.4.4a.cmml">t.format(s)</mtext><mo fence="true" lspace="0em" rspace="0em" id="A4.p2.3.m3.5.5.1.1.3" xref="A4.p2.3.m3.5.5.1.2.1.cmml">∣</mo><mrow id="A4.p2.3.m3.5.5.1.1.1" xref="A4.p2.3.m3.5.5.1.1.1.cmml"><mtext class="ltx_mathvariant_monospace" id="A4.p2.3.m3.5.5.1.1.1.2" xref="A4.p2.3.m3.5.5.1.1.1.2a.cmml">t</mtext><mo id="A4.p2.3.m3.5.5.1.1.1.1" xref="A4.p2.3.m3.5.5.1.1.1.1.cmml">∈</mo><mrow id="A4.p2.3.m3.5.5.1.1.1.3" xref="A4.p2.3.m3.5.5.1.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="A4.p2.3.m3.5.5.1.1.1.3.2" xref="A4.p2.3.m3.5.5.1.1.1.3.2.cmml">𝒯</mi><mo lspace="0em" rspace="0em" id="A4.p2.3.m3.5.5.1.1.1.3.1" xref="A4.p2.3.m3.5.5.1.1.1.3.1.cmml">​</mo><mrow id="A4.p2.3.m3.5.5.1.1.1.3.3.2" xref="A4.p2.3.m3.5.5.1.1.1.3.cmml"><mo stretchy="false" id="A4.p2.3.m3.5.5.1.1.1.3.3.2.1" xref="A4.p2.3.m3.5.5.1.1.1.3.cmml">(</mo><mi id="A4.p2.3.m3.3.3" xref="A4.p2.3.m3.3.3.cmml">r</mi><mo stretchy="false" id="A4.p2.3.m3.5.5.1.1.1.3.3.2.2" xref="A4.p2.3.m3.5.5.1.1.1.3.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="A4.p2.3.m3.5.5.1.1.4" xref="A4.p2.3.m3.5.5.1.2.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.p2.3.m3.5b"><apply id="A4.p2.3.m3.5.5.cmml" xref="A4.p2.3.m3.5.5"><csymbol cd="latexml" id="A4.p2.3.m3.5.5.2.cmml" xref="A4.p2.3.m3.5.5.2">assign</csymbol><apply id="A4.p2.3.m3.5.5.3.cmml" xref="A4.p2.3.m3.5.5.3"><times id="A4.p2.3.m3.5.5.3.1.cmml" xref="A4.p2.3.m3.5.5.3.1"></times><ci id="A4.p2.3.m3.5.5.3.2.cmml" xref="A4.p2.3.m3.5.5.3.2">𝒫</ci><interval closure="open" id="A4.p2.3.m3.5.5.3.3.1.cmml" xref="A4.p2.3.m3.5.5.3.3.2"><ci id="A4.p2.3.m3.1.1.cmml" xref="A4.p2.3.m3.1.1">𝑠</ci><ci id="A4.p2.3.m3.2.2.cmml" xref="A4.p2.3.m3.2.2">𝑟</ci></interval></apply><apply id="A4.p2.3.m3.5.5.1.2.cmml" xref="A4.p2.3.m3.5.5.1.1"><csymbol cd="latexml" id="A4.p2.3.m3.5.5.1.2.1.cmml" xref="A4.p2.3.m3.5.5.1.1.2">conditional-set</csymbol><ci id="A4.p2.3.m3.4.4a.cmml" xref="A4.p2.3.m3.4.4"><mtext class="ltx_mathvariant_monospace" id="A4.p2.3.m3.4.4.cmml" xref="A4.p2.3.m3.4.4">t.format(s)</mtext></ci><apply id="A4.p2.3.m3.5.5.1.1.1.cmml" xref="A4.p2.3.m3.5.5.1.1.1"><in id="A4.p2.3.m3.5.5.1.1.1.1.cmml" xref="A4.p2.3.m3.5.5.1.1.1.1"></in><ci id="A4.p2.3.m3.5.5.1.1.1.2a.cmml" xref="A4.p2.3.m3.5.5.1.1.1.2"><mtext class="ltx_mathvariant_monospace" id="A4.p2.3.m3.5.5.1.1.1.2.cmml" xref="A4.p2.3.m3.5.5.1.1.1.2">t</mtext></ci><apply id="A4.p2.3.m3.5.5.1.1.1.3.cmml" xref="A4.p2.3.m3.5.5.1.1.1.3"><times id="A4.p2.3.m3.5.5.1.1.1.3.1.cmml" xref="A4.p2.3.m3.5.5.1.1.1.3.1"></times><ci id="A4.p2.3.m3.5.5.1.1.1.3.2.cmml" xref="A4.p2.3.m3.5.5.1.1.1.3.2">𝒯</ci><ci id="A4.p2.3.m3.3.3.cmml" xref="A4.p2.3.m3.3.3">𝑟</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.3.m3.5c">\mathcal{P}(s,r):=\{\texttt{t.format(s)}\mid\texttt{t}\in\mathcal{T}(r)\}</annotation></semantics></math>, where <span id="A4.p2.5.4" class="ltx_text ltx_markedasmath ltx_font_typewriter">.format()</span> is string substitution. For example, a template for <math id="A4.p2.5.m5.1" class="ltx_Math" alttext="(r=\text{plays sport professionally})" display="inline"><semantics id="A4.p2.5.m5.1a"><mrow id="A4.p2.5.m5.1.1.1" xref="A4.p2.5.m5.1.1.1.1.cmml"><mo stretchy="false" id="A4.p2.5.m5.1.1.1.2" xref="A4.p2.5.m5.1.1.1.1.cmml">(</mo><mrow id="A4.p2.5.m5.1.1.1.1" xref="A4.p2.5.m5.1.1.1.1.cmml"><mi id="A4.p2.5.m5.1.1.1.1.2" xref="A4.p2.5.m5.1.1.1.1.2.cmml">r</mi><mo id="A4.p2.5.m5.1.1.1.1.1" xref="A4.p2.5.m5.1.1.1.1.1.cmml">=</mo><mtext id="A4.p2.5.m5.1.1.1.1.3" xref="A4.p2.5.m5.1.1.1.1.3a.cmml">plays sport professionally</mtext></mrow><mo stretchy="false" id="A4.p2.5.m5.1.1.1.3" xref="A4.p2.5.m5.1.1.1.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="A4.p2.5.m5.1b"><apply id="A4.p2.5.m5.1.1.1.1.cmml" xref="A4.p2.5.m5.1.1.1"><eq id="A4.p2.5.m5.1.1.1.1.1.cmml" xref="A4.p2.5.m5.1.1.1.1.1"></eq><ci id="A4.p2.5.m5.1.1.1.1.2.cmml" xref="A4.p2.5.m5.1.1.1.1.2">𝑟</ci><ci id="A4.p2.5.m5.1.1.1.1.3a.cmml" xref="A4.p2.5.m5.1.1.1.1.3"><mtext id="A4.p2.5.m5.1.1.1.1.3.cmml" xref="A4.p2.5.m5.1.1.1.1.3">plays sport professionally</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.5.m5.1c">(r=\text{plays sport professionally})</annotation></semantics></math> might be “{} plays the sport of,” where “LeBron James” substitutes for “{}”.</p>
</div>
<div id="A4.p3" class="ltx_para">
<p id="A4.p3.6" class="ltx_p">Solely using the <span id="A4.p3.6.1" class="ltx_text ltx_font_smallcaps">ParaRel</span> entry, we derive two elements. A <span id="A4.p3.6.2" class="ltx_text ltx_font_bold">requested rewrite</span> is represented as <math id="A4.p3.1.m1.5" class="ltx_Math" alttext="\{s,r,o^{c},o^{*},p^{*}\}" display="inline"><semantics id="A4.p3.1.m1.5a"><mrow id="A4.p3.1.m1.5.5.3" xref="A4.p3.1.m1.5.5.4.cmml"><mo stretchy="false" id="A4.p3.1.m1.5.5.3.4" xref="A4.p3.1.m1.5.5.4.cmml">{</mo><mi id="A4.p3.1.m1.1.1" xref="A4.p3.1.m1.1.1.cmml">s</mi><mo id="A4.p3.1.m1.5.5.3.5" xref="A4.p3.1.m1.5.5.4.cmml">,</mo><mi id="A4.p3.1.m1.2.2" xref="A4.p3.1.m1.2.2.cmml">r</mi><mo id="A4.p3.1.m1.5.5.3.6" xref="A4.p3.1.m1.5.5.4.cmml">,</mo><msup id="A4.p3.1.m1.3.3.1.1" xref="A4.p3.1.m1.3.3.1.1.cmml"><mi id="A4.p3.1.m1.3.3.1.1.2" xref="A4.p3.1.m1.3.3.1.1.2.cmml">o</mi><mi id="A4.p3.1.m1.3.3.1.1.3" xref="A4.p3.1.m1.3.3.1.1.3.cmml">c</mi></msup><mo id="A4.p3.1.m1.5.5.3.7" xref="A4.p3.1.m1.5.5.4.cmml">,</mo><msup id="A4.p3.1.m1.4.4.2.2" xref="A4.p3.1.m1.4.4.2.2.cmml"><mi id="A4.p3.1.m1.4.4.2.2.2" xref="A4.p3.1.m1.4.4.2.2.2.cmml">o</mi><mo id="A4.p3.1.m1.4.4.2.2.3" xref="A4.p3.1.m1.4.4.2.2.3.cmml">∗</mo></msup><mo id="A4.p3.1.m1.5.5.3.8" xref="A4.p3.1.m1.5.5.4.cmml">,</mo><msup id="A4.p3.1.m1.5.5.3.3" xref="A4.p3.1.m1.5.5.3.3.cmml"><mi id="A4.p3.1.m1.5.5.3.3.2" xref="A4.p3.1.m1.5.5.3.3.2.cmml">p</mi><mo id="A4.p3.1.m1.5.5.3.3.3" xref="A4.p3.1.m1.5.5.3.3.3.cmml">∗</mo></msup><mo stretchy="false" id="A4.p3.1.m1.5.5.3.9" xref="A4.p3.1.m1.5.5.4.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="A4.p3.1.m1.5b"><set id="A4.p3.1.m1.5.5.4.cmml" xref="A4.p3.1.m1.5.5.3"><ci id="A4.p3.1.m1.1.1.cmml" xref="A4.p3.1.m1.1.1">𝑠</ci><ci id="A4.p3.1.m1.2.2.cmml" xref="A4.p3.1.m1.2.2">𝑟</ci><apply id="A4.p3.1.m1.3.3.1.1.cmml" xref="A4.p3.1.m1.3.3.1.1"><csymbol cd="ambiguous" id="A4.p3.1.m1.3.3.1.1.1.cmml" xref="A4.p3.1.m1.3.3.1.1">superscript</csymbol><ci id="A4.p3.1.m1.3.3.1.1.2.cmml" xref="A4.p3.1.m1.3.3.1.1.2">𝑜</ci><ci id="A4.p3.1.m1.3.3.1.1.3.cmml" xref="A4.p3.1.m1.3.3.1.1.3">𝑐</ci></apply><apply id="A4.p3.1.m1.4.4.2.2.cmml" xref="A4.p3.1.m1.4.4.2.2"><csymbol cd="ambiguous" id="A4.p3.1.m1.4.4.2.2.1.cmml" xref="A4.p3.1.m1.4.4.2.2">superscript</csymbol><ci id="A4.p3.1.m1.4.4.2.2.2.cmml" xref="A4.p3.1.m1.4.4.2.2.2">𝑜</ci><times id="A4.p3.1.m1.4.4.2.2.3.cmml" xref="A4.p3.1.m1.4.4.2.2.3"></times></apply><apply id="A4.p3.1.m1.5.5.3.3.cmml" xref="A4.p3.1.m1.5.5.3.3"><csymbol cd="ambiguous" id="A4.p3.1.m1.5.5.3.3.1.cmml" xref="A4.p3.1.m1.5.5.3.3">superscript</csymbol><ci id="A4.p3.1.m1.5.5.3.3.2.cmml" xref="A4.p3.1.m1.5.5.3.3.2">𝑝</ci><times id="A4.p3.1.m1.5.5.3.3.3.cmml" xref="A4.p3.1.m1.5.5.3.3.3"></times></apply></set></annotation-xml><annotation encoding="application/x-tex" id="A4.p3.1.m1.5c">\{s,r,o^{c},o^{*},p^{*}\}</annotation></semantics></math>, where <math id="A4.p3.2.m2.2" class="ltx_Math" alttext="p^{*}\sim\mathcal{P}(s,r)" display="inline"><semantics id="A4.p3.2.m2.2a"><mrow id="A4.p3.2.m2.2.3" xref="A4.p3.2.m2.2.3.cmml"><msup id="A4.p3.2.m2.2.3.2" xref="A4.p3.2.m2.2.3.2.cmml"><mi id="A4.p3.2.m2.2.3.2.2" xref="A4.p3.2.m2.2.3.2.2.cmml">p</mi><mo id="A4.p3.2.m2.2.3.2.3" xref="A4.p3.2.m2.2.3.2.3.cmml">∗</mo></msup><mo id="A4.p3.2.m2.2.3.1" xref="A4.p3.2.m2.2.3.1.cmml">∼</mo><mrow id="A4.p3.2.m2.2.3.3" xref="A4.p3.2.m2.2.3.3.cmml"><mi class="ltx_font_mathcaligraphic" id="A4.p3.2.m2.2.3.3.2" xref="A4.p3.2.m2.2.3.3.2.cmml">𝒫</mi><mo lspace="0em" rspace="0em" id="A4.p3.2.m2.2.3.3.1" xref="A4.p3.2.m2.2.3.3.1.cmml">​</mo><mrow id="A4.p3.2.m2.2.3.3.3.2" xref="A4.p3.2.m2.2.3.3.3.1.cmml"><mo stretchy="false" id="A4.p3.2.m2.2.3.3.3.2.1" xref="A4.p3.2.m2.2.3.3.3.1.cmml">(</mo><mi id="A4.p3.2.m2.1.1" xref="A4.p3.2.m2.1.1.cmml">s</mi><mo id="A4.p3.2.m2.2.3.3.3.2.2" xref="A4.p3.2.m2.2.3.3.3.1.cmml">,</mo><mi id="A4.p3.2.m2.2.2" xref="A4.p3.2.m2.2.2.cmml">r</mi><mo stretchy="false" id="A4.p3.2.m2.2.3.3.3.2.3" xref="A4.p3.2.m2.2.3.3.3.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.p3.2.m2.2b"><apply id="A4.p3.2.m2.2.3.cmml" xref="A4.p3.2.m2.2.3"><csymbol cd="latexml" id="A4.p3.2.m2.2.3.1.cmml" xref="A4.p3.2.m2.2.3.1">similar-to</csymbol><apply id="A4.p3.2.m2.2.3.2.cmml" xref="A4.p3.2.m2.2.3.2"><csymbol cd="ambiguous" id="A4.p3.2.m2.2.3.2.1.cmml" xref="A4.p3.2.m2.2.3.2">superscript</csymbol><ci id="A4.p3.2.m2.2.3.2.2.cmml" xref="A4.p3.2.m2.2.3.2.2">𝑝</ci><times id="A4.p3.2.m2.2.3.2.3.cmml" xref="A4.p3.2.m2.2.3.2.3"></times></apply><apply id="A4.p3.2.m2.2.3.3.cmml" xref="A4.p3.2.m2.2.3.3"><times id="A4.p3.2.m2.2.3.3.1.cmml" xref="A4.p3.2.m2.2.3.3.1"></times><ci id="A4.p3.2.m2.2.3.3.2.cmml" xref="A4.p3.2.m2.2.3.3.2">𝒫</ci><interval closure="open" id="A4.p3.2.m2.2.3.3.3.1.cmml" xref="A4.p3.2.m2.2.3.3.3.2"><ci id="A4.p3.2.m2.1.1.cmml" xref="A4.p3.2.m2.1.1">𝑠</ci><ci id="A4.p3.2.m2.2.2.cmml" xref="A4.p3.2.m2.2.2">𝑟</ci></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p3.2.m2.2c">p^{*}\sim\mathcal{P}(s,r)</annotation></semantics></math> is the sole rewriting prompt, and <math id="A4.p3.3.m3.1" class="ltx_Math" alttext="o^{*}" display="inline"><semantics id="A4.p3.3.m3.1a"><msup id="A4.p3.3.m3.1.1" xref="A4.p3.3.m3.1.1.cmml"><mi id="A4.p3.3.m3.1.1.2" xref="A4.p3.3.m3.1.1.2.cmml">o</mi><mo id="A4.p3.3.m3.1.1.3" xref="A4.p3.3.m3.1.1.3.cmml">∗</mo></msup><annotation-xml encoding="MathML-Content" id="A4.p3.3.m3.1b"><apply id="A4.p3.3.m3.1.1.cmml" xref="A4.p3.3.m3.1.1"><csymbol cd="ambiguous" id="A4.p3.3.m3.1.1.1.cmml" xref="A4.p3.3.m3.1.1">superscript</csymbol><ci id="A4.p3.3.m3.1.1.2.cmml" xref="A4.p3.3.m3.1.1.2">𝑜</ci><times id="A4.p3.3.m3.1.1.3.cmml" xref="A4.p3.3.m3.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p3.3.m3.1c">o^{*}</annotation></semantics></math> is drawn from a weighted sample of all <span id="A4.p3.6.3" class="ltx_text ltx_font_smallcaps">ParaRel</span> tuples with the predicate <math id="A4.p3.4.m4.2" class="ltx_Math" alttext="(r,\cdot)" display="inline"><semantics id="A4.p3.4.m4.2a"><mrow id="A4.p3.4.m4.2.3.2" xref="A4.p3.4.m4.2.3.1.cmml"><mo stretchy="false" id="A4.p3.4.m4.2.3.2.1" xref="A4.p3.4.m4.2.3.1.cmml">(</mo><mi id="A4.p3.4.m4.1.1" xref="A4.p3.4.m4.1.1.cmml">r</mi><mo rspace="0em" id="A4.p3.4.m4.2.3.2.2" xref="A4.p3.4.m4.2.3.1.cmml">,</mo><mo lspace="0em" rspace="0em" id="A4.p3.4.m4.2.2" xref="A4.p3.4.m4.2.2.cmml">⋅</mo><mo stretchy="false" id="A4.p3.4.m4.2.3.2.3" xref="A4.p3.4.m4.2.3.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="A4.p3.4.m4.2b"><interval closure="open" id="A4.p3.4.m4.2.3.1.cmml" xref="A4.p3.4.m4.2.3.2"><ci id="A4.p3.4.m4.1.1.cmml" xref="A4.p3.4.m4.1.1">𝑟</ci><ci id="A4.p3.4.m4.2.2.cmml" xref="A4.p3.4.m4.2.2">⋅</ci></interval></annotation-xml><annotation encoding="application/x-tex" id="A4.p3.4.m4.2c">(r,\cdot)</annotation></semantics></math>.
Moreover, to test for generalization, a set of two semantically-equivalent <span id="A4.p3.6.4" class="ltx_text ltx_font_bold">paraphrase prompts</span>, <math id="A4.p3.5.m5.1" class="ltx_Math" alttext="P^{P}" display="inline"><semantics id="A4.p3.5.m5.1a"><msup id="A4.p3.5.m5.1.1" xref="A4.p3.5.m5.1.1.cmml"><mi id="A4.p3.5.m5.1.1.2" xref="A4.p3.5.m5.1.1.2.cmml">P</mi><mi id="A4.p3.5.m5.1.1.3" xref="A4.p3.5.m5.1.1.3.cmml">P</mi></msup><annotation-xml encoding="MathML-Content" id="A4.p3.5.m5.1b"><apply id="A4.p3.5.m5.1.1.cmml" xref="A4.p3.5.m5.1.1"><csymbol cd="ambiguous" id="A4.p3.5.m5.1.1.1.cmml" xref="A4.p3.5.m5.1.1">superscript</csymbol><ci id="A4.p3.5.m5.1.1.2.cmml" xref="A4.p3.5.m5.1.1.2">𝑃</ci><ci id="A4.p3.5.m5.1.1.3.cmml" xref="A4.p3.5.m5.1.1.3">𝑃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p3.5.m5.1c">P^{P}</annotation></semantics></math>, is sampled from <math id="A4.p3.6.m6.3" class="ltx_Math" alttext="\mathcal{P}(s,r)\backslash\{p^{*}\}" display="inline"><semantics id="A4.p3.6.m6.3a"><mrow id="A4.p3.6.m6.3.3" xref="A4.p3.6.m6.3.3.cmml"><mrow id="A4.p3.6.m6.3.3.3" xref="A4.p3.6.m6.3.3.3.cmml"><mi class="ltx_font_mathcaligraphic" id="A4.p3.6.m6.3.3.3.2" xref="A4.p3.6.m6.3.3.3.2.cmml">𝒫</mi><mo lspace="0em" rspace="0em" id="A4.p3.6.m6.3.3.3.1" xref="A4.p3.6.m6.3.3.3.1.cmml">​</mo><mrow id="A4.p3.6.m6.3.3.3.3.2" xref="A4.p3.6.m6.3.3.3.3.1.cmml"><mo stretchy="false" id="A4.p3.6.m6.3.3.3.3.2.1" xref="A4.p3.6.m6.3.3.3.3.1.cmml">(</mo><mi id="A4.p3.6.m6.1.1" xref="A4.p3.6.m6.1.1.cmml">s</mi><mo id="A4.p3.6.m6.3.3.3.3.2.2" xref="A4.p3.6.m6.3.3.3.3.1.cmml">,</mo><mi id="A4.p3.6.m6.2.2" xref="A4.p3.6.m6.2.2.cmml">r</mi><mo rspace="0.222em" stretchy="false" id="A4.p3.6.m6.3.3.3.3.2.3" xref="A4.p3.6.m6.3.3.3.3.1.cmml">)</mo></mrow></mrow><mo rspace="0.222em" id="A4.p3.6.m6.3.3.2" xref="A4.p3.6.m6.3.3.2.cmml">\</mo><mrow id="A4.p3.6.m6.3.3.1.1" xref="A4.p3.6.m6.3.3.1.2.cmml"><mo stretchy="false" id="A4.p3.6.m6.3.3.1.1.2" xref="A4.p3.6.m6.3.3.1.2.cmml">{</mo><msup id="A4.p3.6.m6.3.3.1.1.1" xref="A4.p3.6.m6.3.3.1.1.1.cmml"><mi id="A4.p3.6.m6.3.3.1.1.1.2" xref="A4.p3.6.m6.3.3.1.1.1.2.cmml">p</mi><mo id="A4.p3.6.m6.3.3.1.1.1.3" xref="A4.p3.6.m6.3.3.1.1.1.3.cmml">∗</mo></msup><mo stretchy="false" id="A4.p3.6.m6.3.3.1.1.3" xref="A4.p3.6.m6.3.3.1.2.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.p3.6.m6.3b"><apply id="A4.p3.6.m6.3.3.cmml" xref="A4.p3.6.m6.3.3"><ci id="A4.p3.6.m6.3.3.2.cmml" xref="A4.p3.6.m6.3.3.2">\</ci><apply id="A4.p3.6.m6.3.3.3.cmml" xref="A4.p3.6.m6.3.3.3"><times id="A4.p3.6.m6.3.3.3.1.cmml" xref="A4.p3.6.m6.3.3.3.1"></times><ci id="A4.p3.6.m6.3.3.3.2.cmml" xref="A4.p3.6.m6.3.3.3.2">𝒫</ci><interval closure="open" id="A4.p3.6.m6.3.3.3.3.1.cmml" xref="A4.p3.6.m6.3.3.3.3.2"><ci id="A4.p3.6.m6.1.1.cmml" xref="A4.p3.6.m6.1.1">𝑠</ci><ci id="A4.p3.6.m6.2.2.cmml" xref="A4.p3.6.m6.2.2">𝑟</ci></interval></apply><set id="A4.p3.6.m6.3.3.1.2.cmml" xref="A4.p3.6.m6.3.3.1.1"><apply id="A4.p3.6.m6.3.3.1.1.1.cmml" xref="A4.p3.6.m6.3.3.1.1.1"><csymbol cd="ambiguous" id="A4.p3.6.m6.3.3.1.1.1.1.cmml" xref="A4.p3.6.m6.3.3.1.1.1">superscript</csymbol><ci id="A4.p3.6.m6.3.3.1.1.1.2.cmml" xref="A4.p3.6.m6.3.3.1.1.1.2">𝑝</ci><times id="A4.p3.6.m6.3.3.1.1.1.3.cmml" xref="A4.p3.6.m6.3.3.1.1.1.3"></times></apply></set></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p3.6.m6.3c">\mathcal{P}(s,r)\backslash\{p^{*}\}</annotation></semantics></math>.</p>
</div>
<div id="A4.p4" class="ltx_para">
<p id="A4.p4.9" class="ltx_p">To test for specificity, we execute a WikiData SPARQL query<span id="footnote6" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">6</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">6</sup><span class="ltx_tag ltx_tag_note">6</span><a target="_blank" href="https://query.wikidata.org/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://query.wikidata.org/</a></span></span></span> to collect a set of entities that share a predicate with <math id="A4.p4.1.m1.1" class="ltx_Math" alttext="s" display="inline"><semantics id="A4.p4.1.m1.1a"><mi id="A4.p4.1.m1.1.1" xref="A4.p4.1.m1.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="A4.p4.1.m1.1b"><ci id="A4.p4.1.m1.1.1.cmml" xref="A4.p4.1.m1.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.p4.1.m1.1c">s</annotation></semantics></math>: <math id="A4.p4.2.m2.3" class="ltx_Math" alttext="\mathcal{E}=\{s^{\prime}\mid(s^{\prime},r,o^{c})\}" display="inline"><semantics id="A4.p4.2.m2.3a"><mrow id="A4.p4.2.m2.3.3" xref="A4.p4.2.m2.3.3.cmml"><mi class="ltx_font_mathcaligraphic" id="A4.p4.2.m2.3.3.4" xref="A4.p4.2.m2.3.3.4.cmml">ℰ</mi><mo id="A4.p4.2.m2.3.3.3" xref="A4.p4.2.m2.3.3.3.cmml">=</mo><mrow id="A4.p4.2.m2.3.3.2.2" xref="A4.p4.2.m2.3.3.2.3.cmml"><mo stretchy="false" id="A4.p4.2.m2.3.3.2.2.3" xref="A4.p4.2.m2.3.3.2.3.1.cmml">{</mo><msup id="A4.p4.2.m2.2.2.1.1.1" xref="A4.p4.2.m2.2.2.1.1.1.cmml"><mi id="A4.p4.2.m2.2.2.1.1.1.2" xref="A4.p4.2.m2.2.2.1.1.1.2.cmml">s</mi><mo id="A4.p4.2.m2.2.2.1.1.1.3" xref="A4.p4.2.m2.2.2.1.1.1.3.cmml">′</mo></msup><mo fence="true" lspace="0em" rspace="0em" id="A4.p4.2.m2.3.3.2.2.4" xref="A4.p4.2.m2.3.3.2.3.1.cmml">∣</mo><mrow id="A4.p4.2.m2.3.3.2.2.2.2" xref="A4.p4.2.m2.3.3.2.2.2.3.cmml"><mo stretchy="false" id="A4.p4.2.m2.3.3.2.2.2.2.3" xref="A4.p4.2.m2.3.3.2.2.2.3.cmml">(</mo><msup id="A4.p4.2.m2.3.3.2.2.2.1.1" xref="A4.p4.2.m2.3.3.2.2.2.1.1.cmml"><mi id="A4.p4.2.m2.3.3.2.2.2.1.1.2" xref="A4.p4.2.m2.3.3.2.2.2.1.1.2.cmml">s</mi><mo id="A4.p4.2.m2.3.3.2.2.2.1.1.3" xref="A4.p4.2.m2.3.3.2.2.2.1.1.3.cmml">′</mo></msup><mo id="A4.p4.2.m2.3.3.2.2.2.2.4" xref="A4.p4.2.m2.3.3.2.2.2.3.cmml">,</mo><mi id="A4.p4.2.m2.1.1" xref="A4.p4.2.m2.1.1.cmml">r</mi><mo id="A4.p4.2.m2.3.3.2.2.2.2.5" xref="A4.p4.2.m2.3.3.2.2.2.3.cmml">,</mo><msup id="A4.p4.2.m2.3.3.2.2.2.2.2" xref="A4.p4.2.m2.3.3.2.2.2.2.2.cmml"><mi id="A4.p4.2.m2.3.3.2.2.2.2.2.2" xref="A4.p4.2.m2.3.3.2.2.2.2.2.2.cmml">o</mi><mi id="A4.p4.2.m2.3.3.2.2.2.2.2.3" xref="A4.p4.2.m2.3.3.2.2.2.2.2.3.cmml">c</mi></msup><mo stretchy="false" id="A4.p4.2.m2.3.3.2.2.2.2.6" xref="A4.p4.2.m2.3.3.2.2.2.3.cmml">)</mo></mrow><mo stretchy="false" id="A4.p4.2.m2.3.3.2.2.5" xref="A4.p4.2.m2.3.3.2.3.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.p4.2.m2.3b"><apply id="A4.p4.2.m2.3.3.cmml" xref="A4.p4.2.m2.3.3"><eq id="A4.p4.2.m2.3.3.3.cmml" xref="A4.p4.2.m2.3.3.3"></eq><ci id="A4.p4.2.m2.3.3.4.cmml" xref="A4.p4.2.m2.3.3.4">ℰ</ci><apply id="A4.p4.2.m2.3.3.2.3.cmml" xref="A4.p4.2.m2.3.3.2.2"><csymbol cd="latexml" id="A4.p4.2.m2.3.3.2.3.1.cmml" xref="A4.p4.2.m2.3.3.2.2.3">conditional-set</csymbol><apply id="A4.p4.2.m2.2.2.1.1.1.cmml" xref="A4.p4.2.m2.2.2.1.1.1"><csymbol cd="ambiguous" id="A4.p4.2.m2.2.2.1.1.1.1.cmml" xref="A4.p4.2.m2.2.2.1.1.1">superscript</csymbol><ci id="A4.p4.2.m2.2.2.1.1.1.2.cmml" xref="A4.p4.2.m2.2.2.1.1.1.2">𝑠</ci><ci id="A4.p4.2.m2.2.2.1.1.1.3.cmml" xref="A4.p4.2.m2.2.2.1.1.1.3">′</ci></apply><vector id="A4.p4.2.m2.3.3.2.2.2.3.cmml" xref="A4.p4.2.m2.3.3.2.2.2.2"><apply id="A4.p4.2.m2.3.3.2.2.2.1.1.cmml" xref="A4.p4.2.m2.3.3.2.2.2.1.1"><csymbol cd="ambiguous" id="A4.p4.2.m2.3.3.2.2.2.1.1.1.cmml" xref="A4.p4.2.m2.3.3.2.2.2.1.1">superscript</csymbol><ci id="A4.p4.2.m2.3.3.2.2.2.1.1.2.cmml" xref="A4.p4.2.m2.3.3.2.2.2.1.1.2">𝑠</ci><ci id="A4.p4.2.m2.3.3.2.2.2.1.1.3.cmml" xref="A4.p4.2.m2.3.3.2.2.2.1.1.3">′</ci></apply><ci id="A4.p4.2.m2.1.1.cmml" xref="A4.p4.2.m2.1.1">𝑟</ci><apply id="A4.p4.2.m2.3.3.2.2.2.2.2.cmml" xref="A4.p4.2.m2.3.3.2.2.2.2.2"><csymbol cd="ambiguous" id="A4.p4.2.m2.3.3.2.2.2.2.2.1.cmml" xref="A4.p4.2.m2.3.3.2.2.2.2.2">superscript</csymbol><ci id="A4.p4.2.m2.3.3.2.2.2.2.2.2.cmml" xref="A4.p4.2.m2.3.3.2.2.2.2.2.2">𝑜</ci><ci id="A4.p4.2.m2.3.3.2.2.2.2.2.3.cmml" xref="A4.p4.2.m2.3.3.2.2.2.2.2.3">𝑐</ci></apply></vector></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p4.2.m2.3c">\mathcal{E}=\{s^{\prime}\mid(s^{\prime},r,o^{c})\}</annotation></semantics></math>; e.g., for <math id="A4.p4.3.m3.1" class="ltx_Math" alttext="(s=\textit{Eiffel Tower},r=\textit{city location},o^{c}=\textit{Paris})" display="inline"><semantics id="A4.p4.3.m3.1a"><mrow id="A4.p4.3.m3.1.1.1"><mo stretchy="false" id="A4.p4.3.m3.1.1.1.2">(</mo><mrow id="A4.p4.3.m3.1.1.1.1.2" xref="A4.p4.3.m3.1.1.1.1.3.cmml"><mrow id="A4.p4.3.m3.1.1.1.1.1.1" xref="A4.p4.3.m3.1.1.1.1.1.1.cmml"><mi id="A4.p4.3.m3.1.1.1.1.1.1.2" xref="A4.p4.3.m3.1.1.1.1.1.1.2.cmml">s</mi><mo id="A4.p4.3.m3.1.1.1.1.1.1.1" xref="A4.p4.3.m3.1.1.1.1.1.1.1.cmml">=</mo><mtext class="ltx_mathvariant_italic" id="A4.p4.3.m3.1.1.1.1.1.1.3" xref="A4.p4.3.m3.1.1.1.1.1.1.3a.cmml">Eiffel Tower</mtext></mrow><mo id="A4.p4.3.m3.1.1.1.1.2.3" xref="A4.p4.3.m3.1.1.1.1.3a.cmml">,</mo><mrow id="A4.p4.3.m3.1.1.1.1.2.2.2" xref="A4.p4.3.m3.1.1.1.1.2.2.3.cmml"><mrow id="A4.p4.3.m3.1.1.1.1.2.2.1.1" xref="A4.p4.3.m3.1.1.1.1.2.2.1.1.cmml"><mi id="A4.p4.3.m3.1.1.1.1.2.2.1.1.2" xref="A4.p4.3.m3.1.1.1.1.2.2.1.1.2.cmml">r</mi><mo id="A4.p4.3.m3.1.1.1.1.2.2.1.1.1" xref="A4.p4.3.m3.1.1.1.1.2.2.1.1.1.cmml">=</mo><mtext class="ltx_mathvariant_italic" id="A4.p4.3.m3.1.1.1.1.2.2.1.1.3" xref="A4.p4.3.m3.1.1.1.1.2.2.1.1.3a.cmml">city location</mtext></mrow><mo id="A4.p4.3.m3.1.1.1.1.2.2.2.3" xref="A4.p4.3.m3.1.1.1.1.2.2.3a.cmml">,</mo><mrow id="A4.p4.3.m3.1.1.1.1.2.2.2.2" xref="A4.p4.3.m3.1.1.1.1.2.2.2.2.cmml"><msup id="A4.p4.3.m3.1.1.1.1.2.2.2.2.2" xref="A4.p4.3.m3.1.1.1.1.2.2.2.2.2.cmml"><mi id="A4.p4.3.m3.1.1.1.1.2.2.2.2.2.2" xref="A4.p4.3.m3.1.1.1.1.2.2.2.2.2.2.cmml">o</mi><mi id="A4.p4.3.m3.1.1.1.1.2.2.2.2.2.3" xref="A4.p4.3.m3.1.1.1.1.2.2.2.2.2.3.cmml">c</mi></msup><mo id="A4.p4.3.m3.1.1.1.1.2.2.2.2.1" xref="A4.p4.3.m3.1.1.1.1.2.2.2.2.1.cmml">=</mo><mtext class="ltx_mathvariant_italic" id="A4.p4.3.m3.1.1.1.1.2.2.2.2.3" xref="A4.p4.3.m3.1.1.1.1.2.2.2.2.3a.cmml">Paris</mtext></mrow></mrow></mrow><mo stretchy="false" id="A4.p4.3.m3.1.1.1.3">)</mo></mrow><annotation-xml encoding="MathML-Content" id="A4.p4.3.m3.1b"><apply id="A4.p4.3.m3.1.1.1.1.3.cmml" xref="A4.p4.3.m3.1.1.1.1.2"><csymbol cd="ambiguous" id="A4.p4.3.m3.1.1.1.1.3a.cmml" xref="A4.p4.3.m3.1.1.1.1.2.3">formulae-sequence</csymbol><apply id="A4.p4.3.m3.1.1.1.1.1.1.cmml" xref="A4.p4.3.m3.1.1.1.1.1.1"><eq id="A4.p4.3.m3.1.1.1.1.1.1.1.cmml" xref="A4.p4.3.m3.1.1.1.1.1.1.1"></eq><ci id="A4.p4.3.m3.1.1.1.1.1.1.2.cmml" xref="A4.p4.3.m3.1.1.1.1.1.1.2">𝑠</ci><ci id="A4.p4.3.m3.1.1.1.1.1.1.3a.cmml" xref="A4.p4.3.m3.1.1.1.1.1.1.3"><mtext class="ltx_mathvariant_italic" id="A4.p4.3.m3.1.1.1.1.1.1.3.cmml" xref="A4.p4.3.m3.1.1.1.1.1.1.3">Eiffel Tower</mtext></ci></apply><apply id="A4.p4.3.m3.1.1.1.1.2.2.3.cmml" xref="A4.p4.3.m3.1.1.1.1.2.2.2"><csymbol cd="ambiguous" id="A4.p4.3.m3.1.1.1.1.2.2.3a.cmml" xref="A4.p4.3.m3.1.1.1.1.2.2.2.3">formulae-sequence</csymbol><apply id="A4.p4.3.m3.1.1.1.1.2.2.1.1.cmml" xref="A4.p4.3.m3.1.1.1.1.2.2.1.1"><eq id="A4.p4.3.m3.1.1.1.1.2.2.1.1.1.cmml" xref="A4.p4.3.m3.1.1.1.1.2.2.1.1.1"></eq><ci id="A4.p4.3.m3.1.1.1.1.2.2.1.1.2.cmml" xref="A4.p4.3.m3.1.1.1.1.2.2.1.1.2">𝑟</ci><ci id="A4.p4.3.m3.1.1.1.1.2.2.1.1.3a.cmml" xref="A4.p4.3.m3.1.1.1.1.2.2.1.1.3"><mtext class="ltx_mathvariant_italic" id="A4.p4.3.m3.1.1.1.1.2.2.1.1.3.cmml" xref="A4.p4.3.m3.1.1.1.1.2.2.1.1.3">city location</mtext></ci></apply><apply id="A4.p4.3.m3.1.1.1.1.2.2.2.2.cmml" xref="A4.p4.3.m3.1.1.1.1.2.2.2.2"><eq id="A4.p4.3.m3.1.1.1.1.2.2.2.2.1.cmml" xref="A4.p4.3.m3.1.1.1.1.2.2.2.2.1"></eq><apply id="A4.p4.3.m3.1.1.1.1.2.2.2.2.2.cmml" xref="A4.p4.3.m3.1.1.1.1.2.2.2.2.2"><csymbol cd="ambiguous" id="A4.p4.3.m3.1.1.1.1.2.2.2.2.2.1.cmml" xref="A4.p4.3.m3.1.1.1.1.2.2.2.2.2">superscript</csymbol><ci id="A4.p4.3.m3.1.1.1.1.2.2.2.2.2.2.cmml" xref="A4.p4.3.m3.1.1.1.1.2.2.2.2.2.2">𝑜</ci><ci id="A4.p4.3.m3.1.1.1.1.2.2.2.2.2.3.cmml" xref="A4.p4.3.m3.1.1.1.1.2.2.2.2.2.3">𝑐</ci></apply><ci id="A4.p4.3.m3.1.1.1.1.2.2.2.2.3a.cmml" xref="A4.p4.3.m3.1.1.1.1.2.2.2.2.3"><mtext class="ltx_mathvariant_italic" id="A4.p4.3.m3.1.1.1.1.2.2.2.2.3.cmml" xref="A4.p4.3.m3.1.1.1.1.2.2.2.2.3">Paris</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p4.3.m3.1c">(s=\textit{Eiffel Tower},r=\textit{city location},o^{c}=\textit{Paris})</annotation></semantics></math>, <math id="A4.p4.4.m4.1" class="ltx_Math" alttext="\mathcal{E}" display="inline"><semantics id="A4.p4.4.m4.1a"><mi class="ltx_font_mathcaligraphic" id="A4.p4.4.m4.1.1" xref="A4.p4.4.m4.1.1.cmml">ℰ</mi><annotation-xml encoding="MathML-Content" id="A4.p4.4.m4.1b"><ci id="A4.p4.4.m4.1.1.cmml" xref="A4.p4.4.m4.1.1">ℰ</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.p4.4.m4.1c">\mathcal{E}</annotation></semantics></math> might contain entities like the Champs-Élysées or Louvre. We then construct a set of prompts <math id="A4.p4.5.m5.3" class="ltx_Math" alttext="\{\mathcal{P}(s^{\prime},r)\mid s^{\prime}\in\mathcal{E}\}" display="inline"><semantics id="A4.p4.5.m5.3a"><mrow id="A4.p4.5.m5.3.3.2" xref="A4.p4.5.m5.3.3.3.cmml"><mo stretchy="false" id="A4.p4.5.m5.3.3.2.3" xref="A4.p4.5.m5.3.3.3.1.cmml">{</mo><mrow id="A4.p4.5.m5.2.2.1.1" xref="A4.p4.5.m5.2.2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A4.p4.5.m5.2.2.1.1.3" xref="A4.p4.5.m5.2.2.1.1.3.cmml">𝒫</mi><mo lspace="0em" rspace="0em" id="A4.p4.5.m5.2.2.1.1.2" xref="A4.p4.5.m5.2.2.1.1.2.cmml">​</mo><mrow id="A4.p4.5.m5.2.2.1.1.1.1" xref="A4.p4.5.m5.2.2.1.1.1.2.cmml"><mo stretchy="false" id="A4.p4.5.m5.2.2.1.1.1.1.2" xref="A4.p4.5.m5.2.2.1.1.1.2.cmml">(</mo><msup id="A4.p4.5.m5.2.2.1.1.1.1.1" xref="A4.p4.5.m5.2.2.1.1.1.1.1.cmml"><mi id="A4.p4.5.m5.2.2.1.1.1.1.1.2" xref="A4.p4.5.m5.2.2.1.1.1.1.1.2.cmml">s</mi><mo id="A4.p4.5.m5.2.2.1.1.1.1.1.3" xref="A4.p4.5.m5.2.2.1.1.1.1.1.3.cmml">′</mo></msup><mo id="A4.p4.5.m5.2.2.1.1.1.1.3" xref="A4.p4.5.m5.2.2.1.1.1.2.cmml">,</mo><mi id="A4.p4.5.m5.1.1" xref="A4.p4.5.m5.1.1.cmml">r</mi><mo stretchy="false" id="A4.p4.5.m5.2.2.1.1.1.1.4" xref="A4.p4.5.m5.2.2.1.1.1.2.cmml">)</mo></mrow></mrow><mo fence="true" lspace="0em" rspace="0em" id="A4.p4.5.m5.3.3.2.4" xref="A4.p4.5.m5.3.3.3.1.cmml">∣</mo><mrow id="A4.p4.5.m5.3.3.2.2" xref="A4.p4.5.m5.3.3.2.2.cmml"><msup id="A4.p4.5.m5.3.3.2.2.2" xref="A4.p4.5.m5.3.3.2.2.2.cmml"><mi id="A4.p4.5.m5.3.3.2.2.2.2" xref="A4.p4.5.m5.3.3.2.2.2.2.cmml">s</mi><mo id="A4.p4.5.m5.3.3.2.2.2.3" xref="A4.p4.5.m5.3.3.2.2.2.3.cmml">′</mo></msup><mo id="A4.p4.5.m5.3.3.2.2.1" xref="A4.p4.5.m5.3.3.2.2.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="A4.p4.5.m5.3.3.2.2.3" xref="A4.p4.5.m5.3.3.2.2.3.cmml">ℰ</mi></mrow><mo stretchy="false" id="A4.p4.5.m5.3.3.2.5" xref="A4.p4.5.m5.3.3.3.1.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="A4.p4.5.m5.3b"><apply id="A4.p4.5.m5.3.3.3.cmml" xref="A4.p4.5.m5.3.3.2"><csymbol cd="latexml" id="A4.p4.5.m5.3.3.3.1.cmml" xref="A4.p4.5.m5.3.3.2.3">conditional-set</csymbol><apply id="A4.p4.5.m5.2.2.1.1.cmml" xref="A4.p4.5.m5.2.2.1.1"><times id="A4.p4.5.m5.2.2.1.1.2.cmml" xref="A4.p4.5.m5.2.2.1.1.2"></times><ci id="A4.p4.5.m5.2.2.1.1.3.cmml" xref="A4.p4.5.m5.2.2.1.1.3">𝒫</ci><interval closure="open" id="A4.p4.5.m5.2.2.1.1.1.2.cmml" xref="A4.p4.5.m5.2.2.1.1.1.1"><apply id="A4.p4.5.m5.2.2.1.1.1.1.1.cmml" xref="A4.p4.5.m5.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="A4.p4.5.m5.2.2.1.1.1.1.1.1.cmml" xref="A4.p4.5.m5.2.2.1.1.1.1.1">superscript</csymbol><ci id="A4.p4.5.m5.2.2.1.1.1.1.1.2.cmml" xref="A4.p4.5.m5.2.2.1.1.1.1.1.2">𝑠</ci><ci id="A4.p4.5.m5.2.2.1.1.1.1.1.3.cmml" xref="A4.p4.5.m5.2.2.1.1.1.1.1.3">′</ci></apply><ci id="A4.p4.5.m5.1.1.cmml" xref="A4.p4.5.m5.1.1">𝑟</ci></interval></apply><apply id="A4.p4.5.m5.3.3.2.2.cmml" xref="A4.p4.5.m5.3.3.2.2"><in id="A4.p4.5.m5.3.3.2.2.1.cmml" xref="A4.p4.5.m5.3.3.2.2.1"></in><apply id="A4.p4.5.m5.3.3.2.2.2.cmml" xref="A4.p4.5.m5.3.3.2.2.2"><csymbol cd="ambiguous" id="A4.p4.5.m5.3.3.2.2.2.1.cmml" xref="A4.p4.5.m5.3.3.2.2.2">superscript</csymbol><ci id="A4.p4.5.m5.3.3.2.2.2.2.cmml" xref="A4.p4.5.m5.3.3.2.2.2.2">𝑠</ci><ci id="A4.p4.5.m5.3.3.2.2.2.3.cmml" xref="A4.p4.5.m5.3.3.2.2.2.3">′</ci></apply><ci id="A4.p4.5.m5.3.3.2.2.3.cmml" xref="A4.p4.5.m5.3.3.2.2.3">ℰ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p4.5.m5.3c">\{\mathcal{P}(s^{\prime},r)\mid s^{\prime}\in\mathcal{E}\}</annotation></semantics></math> and sample ten to get our <span id="A4.p4.9.1" class="ltx_text ltx_font_bold">neighborhood prompts</span>, <math id="A4.p4.6.m6.1" class="ltx_Math" alttext="P^{N}" display="inline"><semantics id="A4.p4.6.m6.1a"><msup id="A4.p4.6.m6.1.1" xref="A4.p4.6.m6.1.1.cmml"><mi id="A4.p4.6.m6.1.1.2" xref="A4.p4.6.m6.1.1.2.cmml">P</mi><mi id="A4.p4.6.m6.1.1.3" xref="A4.p4.6.m6.1.1.3.cmml">N</mi></msup><annotation-xml encoding="MathML-Content" id="A4.p4.6.m6.1b"><apply id="A4.p4.6.m6.1.1.cmml" xref="A4.p4.6.m6.1.1"><csymbol cd="ambiguous" id="A4.p4.6.m6.1.1.1.cmml" xref="A4.p4.6.m6.1.1">superscript</csymbol><ci id="A4.p4.6.m6.1.1.2.cmml" xref="A4.p4.6.m6.1.1.2">𝑃</ci><ci id="A4.p4.6.m6.1.1.3.cmml" xref="A4.p4.6.m6.1.1.3">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p4.6.m6.1c">P^{N}</annotation></semantics></math>. Our rationale for employing this strategy over random sampling is that the <math id="A4.p4.7.m7.1" class="ltx_Math" alttext="s^{\prime}" display="inline"><semantics id="A4.p4.7.m7.1a"><msup id="A4.p4.7.m7.1.1" xref="A4.p4.7.m7.1.1.cmml"><mi id="A4.p4.7.m7.1.1.2" xref="A4.p4.7.m7.1.1.2.cmml">s</mi><mo id="A4.p4.7.m7.1.1.3" xref="A4.p4.7.m7.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="A4.p4.7.m7.1b"><apply id="A4.p4.7.m7.1.1.cmml" xref="A4.p4.7.m7.1.1"><csymbol cd="ambiguous" id="A4.p4.7.m7.1.1.1.cmml" xref="A4.p4.7.m7.1.1">superscript</csymbol><ci id="A4.p4.7.m7.1.1.2.cmml" xref="A4.p4.7.m7.1.1.2">𝑠</ci><ci id="A4.p4.7.m7.1.1.3.cmml" xref="A4.p4.7.m7.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p4.7.m7.1c">s^{\prime}</annotation></semantics></math> we select are close to <math id="A4.p4.8.m8.1" class="ltx_Math" alttext="s" display="inline"><semantics id="A4.p4.8.m8.1a"><mi id="A4.p4.8.m8.1.1" xref="A4.p4.8.m8.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="A4.p4.8.m8.1b"><ci id="A4.p4.8.m8.1.1.cmml" xref="A4.p4.8.m8.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.p4.8.m8.1c">s</annotation></semantics></math> in latent space and thus more susceptible to bleedover when editing <math id="A4.p4.9.m9.1" class="ltx_Math" alttext="s" display="inline"><semantics id="A4.p4.9.m9.1a"><mi id="A4.p4.9.m9.1.1" xref="A4.p4.9.m9.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="A4.p4.9.m9.1b"><ci id="A4.p4.9.m9.1.1.cmml" xref="A4.p4.9.m9.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.p4.9.m9.1c">s</annotation></semantics></math> using linear methods. Comparing the Drawdown column in Table <a href="#S3.T1" title="Table 1 ‣ 3.2 Evaluating ROME: Zero-Shot Relation Extraction (zsRE) ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> with the Neighborhood Scores and Magnitudes in Table <a href="#S3.T4" title="Table 4 ‣ 3.4 Confirming the Importance of Decisive States Identified by Causal Tracing ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, we observe the improved resolution of <span id="A4.p4.9.2" class="ltx_text ltx_font_smallcaps">CounterFact</span>’s targeted sampling.</p>
</div>
<div id="A4.p5" class="ltx_para">
<p id="A4.p5.4" class="ltx_p">Finally, <span id="A4.p5.4.1" class="ltx_text ltx_font_bold">generation prompts</span> are hand-curated for each relation, from which ten are sampled to create <math id="A4.p5.1.m1.1" class="ltx_Math" alttext="P^{G}" display="inline"><semantics id="A4.p5.1.m1.1a"><msup id="A4.p5.1.m1.1.1" xref="A4.p5.1.m1.1.1.cmml"><mi id="A4.p5.1.m1.1.1.2" xref="A4.p5.1.m1.1.1.2.cmml">P</mi><mi id="A4.p5.1.m1.1.1.3" xref="A4.p5.1.m1.1.1.3.cmml">G</mi></msup><annotation-xml encoding="MathML-Content" id="A4.p5.1.m1.1b"><apply id="A4.p5.1.m1.1.1.cmml" xref="A4.p5.1.m1.1.1"><csymbol cd="ambiguous" id="A4.p5.1.m1.1.1.1.cmml" xref="A4.p5.1.m1.1.1">superscript</csymbol><ci id="A4.p5.1.m1.1.1.2.cmml" xref="A4.p5.1.m1.1.1.2">𝑃</ci><ci id="A4.p5.1.m1.1.1.3.cmml" xref="A4.p5.1.m1.1.1.3">𝐺</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p5.1.m1.1c">P^{G}</annotation></semantics></math>. See Figure <a href="#S3.F6" title="Figure 6 ‣ 3.5 Comparing Generation Results ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> for examples; these prompts implicitly draw out underlying facts, instead of directly querying for them, which demands deeper generalization. For evaluating generations, we provide reference texts <math id="A4.p5.2.m2.1" class="ltx_Math" alttext="RT" display="inline"><semantics id="A4.p5.2.m2.1a"><mrow id="A4.p5.2.m2.1.1" xref="A4.p5.2.m2.1.1.cmml"><mi id="A4.p5.2.m2.1.1.2" xref="A4.p5.2.m2.1.1.2.cmml">R</mi><mo lspace="0em" rspace="0em" id="A4.p5.2.m2.1.1.1" xref="A4.p5.2.m2.1.1.1.cmml">​</mo><mi id="A4.p5.2.m2.1.1.3" xref="A4.p5.2.m2.1.1.3.cmml">T</mi></mrow><annotation-xml encoding="MathML-Content" id="A4.p5.2.m2.1b"><apply id="A4.p5.2.m2.1.1.cmml" xref="A4.p5.2.m2.1.1"><times id="A4.p5.2.m2.1.1.1.cmml" xref="A4.p5.2.m2.1.1.1"></times><ci id="A4.p5.2.m2.1.1.2.cmml" xref="A4.p5.2.m2.1.1.2">𝑅</ci><ci id="A4.p5.2.m2.1.1.3.cmml" xref="A4.p5.2.m2.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p5.2.m2.1c">RT</annotation></semantics></math>, which are Wikipedia articles for a sample of entities from <math id="A4.p5.3.m3.3" class="ltx_Math" alttext="\{s^{\prime}\mid(s^{\prime},r,o^{*})\}" display="inline"><semantics id="A4.p5.3.m3.3a"><mrow id="A4.p5.3.m3.3.3.2" xref="A4.p5.3.m3.3.3.3.cmml"><mo stretchy="false" id="A4.p5.3.m3.3.3.2.3" xref="A4.p5.3.m3.3.3.3.1.cmml">{</mo><msup id="A4.p5.3.m3.2.2.1.1" xref="A4.p5.3.m3.2.2.1.1.cmml"><mi id="A4.p5.3.m3.2.2.1.1.2" xref="A4.p5.3.m3.2.2.1.1.2.cmml">s</mi><mo id="A4.p5.3.m3.2.2.1.1.3" xref="A4.p5.3.m3.2.2.1.1.3.cmml">′</mo></msup><mo fence="true" lspace="0em" rspace="0em" id="A4.p5.3.m3.3.3.2.4" xref="A4.p5.3.m3.3.3.3.1.cmml">∣</mo><mrow id="A4.p5.3.m3.3.3.2.2.2" xref="A4.p5.3.m3.3.3.2.2.3.cmml"><mo stretchy="false" id="A4.p5.3.m3.3.3.2.2.2.3" xref="A4.p5.3.m3.3.3.2.2.3.cmml">(</mo><msup id="A4.p5.3.m3.3.3.2.2.1.1" xref="A4.p5.3.m3.3.3.2.2.1.1.cmml"><mi id="A4.p5.3.m3.3.3.2.2.1.1.2" xref="A4.p5.3.m3.3.3.2.2.1.1.2.cmml">s</mi><mo id="A4.p5.3.m3.3.3.2.2.1.1.3" xref="A4.p5.3.m3.3.3.2.2.1.1.3.cmml">′</mo></msup><mo id="A4.p5.3.m3.3.3.2.2.2.4" xref="A4.p5.3.m3.3.3.2.2.3.cmml">,</mo><mi id="A4.p5.3.m3.1.1" xref="A4.p5.3.m3.1.1.cmml">r</mi><mo id="A4.p5.3.m3.3.3.2.2.2.5" xref="A4.p5.3.m3.3.3.2.2.3.cmml">,</mo><msup id="A4.p5.3.m3.3.3.2.2.2.2" xref="A4.p5.3.m3.3.3.2.2.2.2.cmml"><mi id="A4.p5.3.m3.3.3.2.2.2.2.2" xref="A4.p5.3.m3.3.3.2.2.2.2.2.cmml">o</mi><mo id="A4.p5.3.m3.3.3.2.2.2.2.3" xref="A4.p5.3.m3.3.3.2.2.2.2.3.cmml">∗</mo></msup><mo stretchy="false" id="A4.p5.3.m3.3.3.2.2.2.6" xref="A4.p5.3.m3.3.3.2.2.3.cmml">)</mo></mrow><mo stretchy="false" id="A4.p5.3.m3.3.3.2.5" xref="A4.p5.3.m3.3.3.3.1.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="A4.p5.3.m3.3b"><apply id="A4.p5.3.m3.3.3.3.cmml" xref="A4.p5.3.m3.3.3.2"><csymbol cd="latexml" id="A4.p5.3.m3.3.3.3.1.cmml" xref="A4.p5.3.m3.3.3.2.3">conditional-set</csymbol><apply id="A4.p5.3.m3.2.2.1.1.cmml" xref="A4.p5.3.m3.2.2.1.1"><csymbol cd="ambiguous" id="A4.p5.3.m3.2.2.1.1.1.cmml" xref="A4.p5.3.m3.2.2.1.1">superscript</csymbol><ci id="A4.p5.3.m3.2.2.1.1.2.cmml" xref="A4.p5.3.m3.2.2.1.1.2">𝑠</ci><ci id="A4.p5.3.m3.2.2.1.1.3.cmml" xref="A4.p5.3.m3.2.2.1.1.3">′</ci></apply><vector id="A4.p5.3.m3.3.3.2.2.3.cmml" xref="A4.p5.3.m3.3.3.2.2.2"><apply id="A4.p5.3.m3.3.3.2.2.1.1.cmml" xref="A4.p5.3.m3.3.3.2.2.1.1"><csymbol cd="ambiguous" id="A4.p5.3.m3.3.3.2.2.1.1.1.cmml" xref="A4.p5.3.m3.3.3.2.2.1.1">superscript</csymbol><ci id="A4.p5.3.m3.3.3.2.2.1.1.2.cmml" xref="A4.p5.3.m3.3.3.2.2.1.1.2">𝑠</ci><ci id="A4.p5.3.m3.3.3.2.2.1.1.3.cmml" xref="A4.p5.3.m3.3.3.2.2.1.1.3">′</ci></apply><ci id="A4.p5.3.m3.1.1.cmml" xref="A4.p5.3.m3.1.1">𝑟</ci><apply id="A4.p5.3.m3.3.3.2.2.2.2.cmml" xref="A4.p5.3.m3.3.3.2.2.2.2"><csymbol cd="ambiguous" id="A4.p5.3.m3.3.3.2.2.2.2.1.cmml" xref="A4.p5.3.m3.3.3.2.2.2.2">superscript</csymbol><ci id="A4.p5.3.m3.3.3.2.2.2.2.2.cmml" xref="A4.p5.3.m3.3.3.2.2.2.2.2">𝑜</ci><times id="A4.p5.3.m3.3.3.2.2.2.2.3.cmml" xref="A4.p5.3.m3.3.3.2.2.2.2.3"></times></apply></vector></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p5.3.m3.3c">\{s^{\prime}\mid(s^{\prime},r,o^{*})\}</annotation></semantics></math>; intuitively, these contain <math id="A4.p5.4.m4.1" class="ltx_Math" alttext="n" display="inline"><semantics id="A4.p5.4.m4.1a"><mi id="A4.p5.4.m4.1.1" xref="A4.p5.4.m4.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="A4.p5.4.m4.1b"><ci id="A4.p5.4.m4.1.1.cmml" xref="A4.p5.4.m4.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.p5.4.m4.1c">n</annotation></semantics></math>-gram statistics that should align with generated text.</p>
</div>
<div id="A4.p6" class="ltx_para">
<p id="A4.p6.6" class="ltx_p">In summary, each record in our dataset <math id="A4.p6.1.m1.1" class="ltx_Math" alttext="\mathcal{D}" display="inline"><semantics id="A4.p6.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="A4.p6.1.m1.1.1" xref="A4.p6.1.m1.1.1.cmml">𝒟</mi><annotation-xml encoding="MathML-Content" id="A4.p6.1.m1.1b"><ci id="A4.p6.1.m1.1.1.cmml" xref="A4.p6.1.m1.1.1">𝒟</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.p6.1.m1.1c">\mathcal{D}</annotation></semantics></math> contains the request <math id="A4.p6.2.m2.5" class="ltx_Math" alttext="\{s,r,o^{c},o^{*},p^{*}\}" display="inline"><semantics id="A4.p6.2.m2.5a"><mrow id="A4.p6.2.m2.5.5.3" xref="A4.p6.2.m2.5.5.4.cmml"><mo stretchy="false" id="A4.p6.2.m2.5.5.3.4" xref="A4.p6.2.m2.5.5.4.cmml">{</mo><mi id="A4.p6.2.m2.1.1" xref="A4.p6.2.m2.1.1.cmml">s</mi><mo id="A4.p6.2.m2.5.5.3.5" xref="A4.p6.2.m2.5.5.4.cmml">,</mo><mi id="A4.p6.2.m2.2.2" xref="A4.p6.2.m2.2.2.cmml">r</mi><mo id="A4.p6.2.m2.5.5.3.6" xref="A4.p6.2.m2.5.5.4.cmml">,</mo><msup id="A4.p6.2.m2.3.3.1.1" xref="A4.p6.2.m2.3.3.1.1.cmml"><mi id="A4.p6.2.m2.3.3.1.1.2" xref="A4.p6.2.m2.3.3.1.1.2.cmml">o</mi><mi id="A4.p6.2.m2.3.3.1.1.3" xref="A4.p6.2.m2.3.3.1.1.3.cmml">c</mi></msup><mo id="A4.p6.2.m2.5.5.3.7" xref="A4.p6.2.m2.5.5.4.cmml">,</mo><msup id="A4.p6.2.m2.4.4.2.2" xref="A4.p6.2.m2.4.4.2.2.cmml"><mi id="A4.p6.2.m2.4.4.2.2.2" xref="A4.p6.2.m2.4.4.2.2.2.cmml">o</mi><mo id="A4.p6.2.m2.4.4.2.2.3" xref="A4.p6.2.m2.4.4.2.2.3.cmml">∗</mo></msup><mo id="A4.p6.2.m2.5.5.3.8" xref="A4.p6.2.m2.5.5.4.cmml">,</mo><msup id="A4.p6.2.m2.5.5.3.3" xref="A4.p6.2.m2.5.5.3.3.cmml"><mi id="A4.p6.2.m2.5.5.3.3.2" xref="A4.p6.2.m2.5.5.3.3.2.cmml">p</mi><mo id="A4.p6.2.m2.5.5.3.3.3" xref="A4.p6.2.m2.5.5.3.3.3.cmml">∗</mo></msup><mo stretchy="false" id="A4.p6.2.m2.5.5.3.9" xref="A4.p6.2.m2.5.5.4.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="A4.p6.2.m2.5b"><set id="A4.p6.2.m2.5.5.4.cmml" xref="A4.p6.2.m2.5.5.3"><ci id="A4.p6.2.m2.1.1.cmml" xref="A4.p6.2.m2.1.1">𝑠</ci><ci id="A4.p6.2.m2.2.2.cmml" xref="A4.p6.2.m2.2.2">𝑟</ci><apply id="A4.p6.2.m2.3.3.1.1.cmml" xref="A4.p6.2.m2.3.3.1.1"><csymbol cd="ambiguous" id="A4.p6.2.m2.3.3.1.1.1.cmml" xref="A4.p6.2.m2.3.3.1.1">superscript</csymbol><ci id="A4.p6.2.m2.3.3.1.1.2.cmml" xref="A4.p6.2.m2.3.3.1.1.2">𝑜</ci><ci id="A4.p6.2.m2.3.3.1.1.3.cmml" xref="A4.p6.2.m2.3.3.1.1.3">𝑐</ci></apply><apply id="A4.p6.2.m2.4.4.2.2.cmml" xref="A4.p6.2.m2.4.4.2.2"><csymbol cd="ambiguous" id="A4.p6.2.m2.4.4.2.2.1.cmml" xref="A4.p6.2.m2.4.4.2.2">superscript</csymbol><ci id="A4.p6.2.m2.4.4.2.2.2.cmml" xref="A4.p6.2.m2.4.4.2.2.2">𝑜</ci><times id="A4.p6.2.m2.4.4.2.2.3.cmml" xref="A4.p6.2.m2.4.4.2.2.3"></times></apply><apply id="A4.p6.2.m2.5.5.3.3.cmml" xref="A4.p6.2.m2.5.5.3.3"><csymbol cd="ambiguous" id="A4.p6.2.m2.5.5.3.3.1.cmml" xref="A4.p6.2.m2.5.5.3.3">superscript</csymbol><ci id="A4.p6.2.m2.5.5.3.3.2.cmml" xref="A4.p6.2.m2.5.5.3.3.2">𝑝</ci><times id="A4.p6.2.m2.5.5.3.3.3.cmml" xref="A4.p6.2.m2.5.5.3.3.3"></times></apply></set></annotation-xml><annotation encoding="application/x-tex" id="A4.p6.2.m2.5c">\{s,r,o^{c},o^{*},p^{*}\}</annotation></semantics></math>, paraphase prompts <math id="A4.p6.3.m3.1" class="ltx_Math" alttext="P^{P}" display="inline"><semantics id="A4.p6.3.m3.1a"><msup id="A4.p6.3.m3.1.1" xref="A4.p6.3.m3.1.1.cmml"><mi id="A4.p6.3.m3.1.1.2" xref="A4.p6.3.m3.1.1.2.cmml">P</mi><mi id="A4.p6.3.m3.1.1.3" xref="A4.p6.3.m3.1.1.3.cmml">P</mi></msup><annotation-xml encoding="MathML-Content" id="A4.p6.3.m3.1b"><apply id="A4.p6.3.m3.1.1.cmml" xref="A4.p6.3.m3.1.1"><csymbol cd="ambiguous" id="A4.p6.3.m3.1.1.1.cmml" xref="A4.p6.3.m3.1.1">superscript</csymbol><ci id="A4.p6.3.m3.1.1.2.cmml" xref="A4.p6.3.m3.1.1.2">𝑃</ci><ci id="A4.p6.3.m3.1.1.3.cmml" xref="A4.p6.3.m3.1.1.3">𝑃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p6.3.m3.1c">P^{P}</annotation></semantics></math>, neighborhood prompts <math id="A4.p6.4.m4.1" class="ltx_Math" alttext="P^{N}" display="inline"><semantics id="A4.p6.4.m4.1a"><msup id="A4.p6.4.m4.1.1" xref="A4.p6.4.m4.1.1.cmml"><mi id="A4.p6.4.m4.1.1.2" xref="A4.p6.4.m4.1.1.2.cmml">P</mi><mi id="A4.p6.4.m4.1.1.3" xref="A4.p6.4.m4.1.1.3.cmml">N</mi></msup><annotation-xml encoding="MathML-Content" id="A4.p6.4.m4.1b"><apply id="A4.p6.4.m4.1.1.cmml" xref="A4.p6.4.m4.1.1"><csymbol cd="ambiguous" id="A4.p6.4.m4.1.1.1.cmml" xref="A4.p6.4.m4.1.1">superscript</csymbol><ci id="A4.p6.4.m4.1.1.2.cmml" xref="A4.p6.4.m4.1.1.2">𝑃</ci><ci id="A4.p6.4.m4.1.1.3.cmml" xref="A4.p6.4.m4.1.1.3">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p6.4.m4.1c">P^{N}</annotation></semantics></math>, generation prompts <math id="A4.p6.5.m5.1" class="ltx_Math" alttext="P^{G}" display="inline"><semantics id="A4.p6.5.m5.1a"><msup id="A4.p6.5.m5.1.1" xref="A4.p6.5.m5.1.1.cmml"><mi id="A4.p6.5.m5.1.1.2" xref="A4.p6.5.m5.1.1.2.cmml">P</mi><mi id="A4.p6.5.m5.1.1.3" xref="A4.p6.5.m5.1.1.3.cmml">G</mi></msup><annotation-xml encoding="MathML-Content" id="A4.p6.5.m5.1b"><apply id="A4.p6.5.m5.1.1.cmml" xref="A4.p6.5.m5.1.1"><csymbol cd="ambiguous" id="A4.p6.5.m5.1.1.1.cmml" xref="A4.p6.5.m5.1.1">superscript</csymbol><ci id="A4.p6.5.m5.1.1.2.cmml" xref="A4.p6.5.m5.1.1.2">𝑃</ci><ci id="A4.p6.5.m5.1.1.3.cmml" xref="A4.p6.5.m5.1.1.3">𝐺</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p6.5.m5.1c">P^{G}</annotation></semantics></math>, and reference texts <math id="A4.p6.6.m6.1" class="ltx_Math" alttext="RT" display="inline"><semantics id="A4.p6.6.m6.1a"><mrow id="A4.p6.6.m6.1.1" xref="A4.p6.6.m6.1.1.cmml"><mi id="A4.p6.6.m6.1.1.2" xref="A4.p6.6.m6.1.1.2.cmml">R</mi><mo lspace="0em" rspace="0em" id="A4.p6.6.m6.1.1.1" xref="A4.p6.6.m6.1.1.1.cmml">​</mo><mi id="A4.p6.6.m6.1.1.3" xref="A4.p6.6.m6.1.1.3.cmml">T</mi></mrow><annotation-xml encoding="MathML-Content" id="A4.p6.6.m6.1b"><apply id="A4.p6.6.m6.1.1.cmml" xref="A4.p6.6.m6.1.1"><times id="A4.p6.6.m6.1.1.1.cmml" xref="A4.p6.6.m6.1.1.1"></times><ci id="A4.p6.6.m6.1.1.2.cmml" xref="A4.p6.6.m6.1.1.2">𝑅</ci><ci id="A4.p6.6.m6.1.1.3.cmml" xref="A4.p6.6.m6.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p6.6.m6.1c">RT</annotation></semantics></math>.
See Figure <a href="#A8.F21" title="Figure 21 ‣ Appendix H Dataset Samples ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">21</span></a> for an example record.
Compared to other evaluation benchmarks, <span id="A4.p6.6.1" class="ltx_text ltx_font_smallcaps">CounterFact</span> provides several new types of tests that allow precise evaluation of knowledge editing (Table <a href="#S3.T3" title="Table 3 ‣ 3.3 Evaluating ROME: Our CounterFact Dataset ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>).</p>
</div>
</section>
<section id="A5" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix E </span>Method Implementation Details</h2>

<section id="A5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">E.1 </span>[GPT-2 XL, GPT-J] Fine-Tuning (FT), Constrained Fine-Tuning (FT+L)</h3>

<figure id="A5.F17" class="ltx_figure"><img src="/html/2202.05262/assets/x17.png" id="A5.F17.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="174" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A5.F17.6.3.1" class="ltx_text" style="font-size:90%;">Figure 17</span>: </span><span id="A5.F17.4.2" class="ltx_text ltx_font_bold" style="font-size:90%;">GPT-2 XL hyperparameter sweeps across layer and <math id="A5.F17.3.1.m1.1" class="ltx_Math" alttext="L_{\infty}" display="inline"><semantics id="A5.F17.3.1.m1.1b"><msub id="A5.F17.3.1.m1.1.1" xref="A5.F17.3.1.m1.1.1.cmml"><mi id="A5.F17.3.1.m1.1.1.2" xref="A5.F17.3.1.m1.1.1.2.cmml">L</mi><mi mathvariant="normal" id="A5.F17.3.1.m1.1.1.3" xref="A5.F17.3.1.m1.1.1.3.cmml">∞</mi></msub><annotation-xml encoding="MathML-Content" id="A5.F17.3.1.m1.1c"><apply id="A5.F17.3.1.m1.1.1.cmml" xref="A5.F17.3.1.m1.1.1"><csymbol cd="ambiguous" id="A5.F17.3.1.m1.1.1.1.cmml" xref="A5.F17.3.1.m1.1.1">subscript</csymbol><ci id="A5.F17.3.1.m1.1.1.2.cmml" xref="A5.F17.3.1.m1.1.1.2">𝐿</ci><infinity id="A5.F17.3.1.m1.1.1.3.cmml" xref="A5.F17.3.1.m1.1.1.3"></infinity></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.F17.3.1.m1.1d">L_{\infty}</annotation></semantics></math> constraint values for fine-tuning-based methods<span id="A5.F17.4.2.1" class="ltx_text ltx_font_medium">. Optimization is carried out for a maximum of 25 steps on a randomly-sampled size-50 subset of <span id="A5.F17.4.2.1.1" class="ltx_text ltx_font_smallcaps">CounterFact</span>. For FT we sweep exclusively over intervention layers, whereas for FT+L we search over three reasonable <math id="A5.F17.4.2.1.m1.1" class="ltx_Math" alttext="\epsilon" display="inline"><semantics id="A5.F17.4.2.1.m1.1b"><mi id="A5.F17.4.2.1.m1.1.1" xref="A5.F17.4.2.1.m1.1.1.cmml">ϵ</mi><annotation-xml encoding="MathML-Content" id="A5.F17.4.2.1.m1.1c"><ci id="A5.F17.4.2.1.m1.1.1.cmml" xref="A5.F17.4.2.1.m1.1.1">italic-ϵ</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.F17.4.2.1.m1.1d">\epsilon</annotation></semantics></math> configurations.</span></span></figcaption>
</figure>
<figure id="A5.F18" class="ltx_figure"><img src="/html/2202.05262/assets/x18.png" id="A5.F18.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="175" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A5.F18.3.1.1" class="ltx_text" style="font-size:90%;">Figure 18</span>: </span><span id="A5.F18.4.2" class="ltx_text ltx_font_bold" style="font-size:90%;">GPT-J hyperparameter sweeps<span id="A5.F18.4.2.1" class="ltx_text ltx_font_medium">. The experimental setup is identical to that of GPT-2 XL.</span></span></figcaption>
</figure>
<div id="A5.SS1.p1" class="ltx_para">
<p id="A5.SS1.p1.1" class="ltx_p">To test the difference between fine-tuning and ROME’s explicit intervention, we use the fine-tuning of MLP weights as a baseline. Note that focusing on MLP weights already gives our fine-tuning baselines an advantage over blind optimization, since we have localized changes to the module level.</p>
</div>
<div id="A5.SS1.p2" class="ltx_para">
<p id="A5.SS1.p2.3" class="ltx_p">For basic Fine-Tuning (FT), we use Adam <cite class="ltx_cite ltx_citemacro_cite">Kingma &amp; Ba (<a href="#bib.bib22" title="" class="ltx_ref">2015</a>)</cite> with early stopping to minimize <math id="A5.SS1.p2.1.m1.1" class="ltx_Math" alttext="-\log\mathbb{P}_{G^{\prime}}\left[o^{*}\mid p\right]" display="inline"><semantics id="A5.SS1.p2.1.m1.1a"><mrow id="A5.SS1.p2.1.m1.1.1" xref="A5.SS1.p2.1.m1.1.1.cmml"><mo rspace="0.167em" id="A5.SS1.p2.1.m1.1.1a" xref="A5.SS1.p2.1.m1.1.1.cmml">−</mo><mrow id="A5.SS1.p2.1.m1.1.1.1" xref="A5.SS1.p2.1.m1.1.1.1.cmml"><mrow id="A5.SS1.p2.1.m1.1.1.1.3" xref="A5.SS1.p2.1.m1.1.1.1.3.cmml"><mi id="A5.SS1.p2.1.m1.1.1.1.3.1" xref="A5.SS1.p2.1.m1.1.1.1.3.1.cmml">log</mi><mo lspace="0.167em" id="A5.SS1.p2.1.m1.1.1.1.3a" xref="A5.SS1.p2.1.m1.1.1.1.3.cmml">⁡</mo><msub id="A5.SS1.p2.1.m1.1.1.1.3.2" xref="A5.SS1.p2.1.m1.1.1.1.3.2.cmml"><mi id="A5.SS1.p2.1.m1.1.1.1.3.2.2" xref="A5.SS1.p2.1.m1.1.1.1.3.2.2.cmml">ℙ</mi><msup id="A5.SS1.p2.1.m1.1.1.1.3.2.3" xref="A5.SS1.p2.1.m1.1.1.1.3.2.3.cmml"><mi id="A5.SS1.p2.1.m1.1.1.1.3.2.3.2" xref="A5.SS1.p2.1.m1.1.1.1.3.2.3.2.cmml">G</mi><mo id="A5.SS1.p2.1.m1.1.1.1.3.2.3.3" xref="A5.SS1.p2.1.m1.1.1.1.3.2.3.3.cmml">′</mo></msup></msub></mrow><mo lspace="0em" rspace="0em" id="A5.SS1.p2.1.m1.1.1.1.2" xref="A5.SS1.p2.1.m1.1.1.1.2.cmml">​</mo><mrow id="A5.SS1.p2.1.m1.1.1.1.1.1" xref="A5.SS1.p2.1.m1.1.1.1.1.2.cmml"><mo id="A5.SS1.p2.1.m1.1.1.1.1.1.2" xref="A5.SS1.p2.1.m1.1.1.1.1.2.1.cmml">[</mo><mrow id="A5.SS1.p2.1.m1.1.1.1.1.1.1" xref="A5.SS1.p2.1.m1.1.1.1.1.1.1.cmml"><msup id="A5.SS1.p2.1.m1.1.1.1.1.1.1.2" xref="A5.SS1.p2.1.m1.1.1.1.1.1.1.2.cmml"><mi id="A5.SS1.p2.1.m1.1.1.1.1.1.1.2.2" xref="A5.SS1.p2.1.m1.1.1.1.1.1.1.2.2.cmml">o</mi><mo id="A5.SS1.p2.1.m1.1.1.1.1.1.1.2.3" xref="A5.SS1.p2.1.m1.1.1.1.1.1.1.2.3.cmml">∗</mo></msup><mo id="A5.SS1.p2.1.m1.1.1.1.1.1.1.1" xref="A5.SS1.p2.1.m1.1.1.1.1.1.1.1.cmml">∣</mo><mi id="A5.SS1.p2.1.m1.1.1.1.1.1.1.3" xref="A5.SS1.p2.1.m1.1.1.1.1.1.1.3.cmml">p</mi></mrow><mo id="A5.SS1.p2.1.m1.1.1.1.1.1.3" xref="A5.SS1.p2.1.m1.1.1.1.1.2.1.cmml">]</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.SS1.p2.1.m1.1b"><apply id="A5.SS1.p2.1.m1.1.1.cmml" xref="A5.SS1.p2.1.m1.1.1"><minus id="A5.SS1.p2.1.m1.1.1.2.cmml" xref="A5.SS1.p2.1.m1.1.1"></minus><apply id="A5.SS1.p2.1.m1.1.1.1.cmml" xref="A5.SS1.p2.1.m1.1.1.1"><times id="A5.SS1.p2.1.m1.1.1.1.2.cmml" xref="A5.SS1.p2.1.m1.1.1.1.2"></times><apply id="A5.SS1.p2.1.m1.1.1.1.3.cmml" xref="A5.SS1.p2.1.m1.1.1.1.3"><log id="A5.SS1.p2.1.m1.1.1.1.3.1.cmml" xref="A5.SS1.p2.1.m1.1.1.1.3.1"></log><apply id="A5.SS1.p2.1.m1.1.1.1.3.2.cmml" xref="A5.SS1.p2.1.m1.1.1.1.3.2"><csymbol cd="ambiguous" id="A5.SS1.p2.1.m1.1.1.1.3.2.1.cmml" xref="A5.SS1.p2.1.m1.1.1.1.3.2">subscript</csymbol><ci id="A5.SS1.p2.1.m1.1.1.1.3.2.2.cmml" xref="A5.SS1.p2.1.m1.1.1.1.3.2.2">ℙ</ci><apply id="A5.SS1.p2.1.m1.1.1.1.3.2.3.cmml" xref="A5.SS1.p2.1.m1.1.1.1.3.2.3"><csymbol cd="ambiguous" id="A5.SS1.p2.1.m1.1.1.1.3.2.3.1.cmml" xref="A5.SS1.p2.1.m1.1.1.1.3.2.3">superscript</csymbol><ci id="A5.SS1.p2.1.m1.1.1.1.3.2.3.2.cmml" xref="A5.SS1.p2.1.m1.1.1.1.3.2.3.2">𝐺</ci><ci id="A5.SS1.p2.1.m1.1.1.1.3.2.3.3.cmml" xref="A5.SS1.p2.1.m1.1.1.1.3.2.3.3">′</ci></apply></apply></apply><apply id="A5.SS1.p2.1.m1.1.1.1.1.2.cmml" xref="A5.SS1.p2.1.m1.1.1.1.1.1"><csymbol cd="latexml" id="A5.SS1.p2.1.m1.1.1.1.1.2.1.cmml" xref="A5.SS1.p2.1.m1.1.1.1.1.1.2">delimited-[]</csymbol><apply id="A5.SS1.p2.1.m1.1.1.1.1.1.1.cmml" xref="A5.SS1.p2.1.m1.1.1.1.1.1.1"><csymbol cd="latexml" id="A5.SS1.p2.1.m1.1.1.1.1.1.1.1.cmml" xref="A5.SS1.p2.1.m1.1.1.1.1.1.1.1">conditional</csymbol><apply id="A5.SS1.p2.1.m1.1.1.1.1.1.1.2.cmml" xref="A5.SS1.p2.1.m1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A5.SS1.p2.1.m1.1.1.1.1.1.1.2.1.cmml" xref="A5.SS1.p2.1.m1.1.1.1.1.1.1.2">superscript</csymbol><ci id="A5.SS1.p2.1.m1.1.1.1.1.1.1.2.2.cmml" xref="A5.SS1.p2.1.m1.1.1.1.1.1.1.2.2">𝑜</ci><times id="A5.SS1.p2.1.m1.1.1.1.1.1.1.2.3.cmml" xref="A5.SS1.p2.1.m1.1.1.1.1.1.1.2.3"></times></apply><ci id="A5.SS1.p2.1.m1.1.1.1.1.1.1.3.cmml" xref="A5.SS1.p2.1.m1.1.1.1.1.1.1.3">𝑝</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.p2.1.m1.1c">-\log\mathbb{P}_{G^{\prime}}\left[o^{*}\mid p\right]</annotation></semantics></math>, changing only <math id="A5.SS1.p2.2.m2.1" class="ltx_Math" alttext="\mathrm{mlp}_{proj}" display="inline"><semantics id="A5.SS1.p2.2.m2.1a"><msub id="A5.SS1.p2.2.m2.1.1" xref="A5.SS1.p2.2.m2.1.1.cmml"><mi id="A5.SS1.p2.2.m2.1.1.2" xref="A5.SS1.p2.2.m2.1.1.2.cmml">mlp</mi><mrow id="A5.SS1.p2.2.m2.1.1.3" xref="A5.SS1.p2.2.m2.1.1.3.cmml"><mi id="A5.SS1.p2.2.m2.1.1.3.2" xref="A5.SS1.p2.2.m2.1.1.3.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="A5.SS1.p2.2.m2.1.1.3.1" xref="A5.SS1.p2.2.m2.1.1.3.1.cmml">​</mo><mi id="A5.SS1.p2.2.m2.1.1.3.3" xref="A5.SS1.p2.2.m2.1.1.3.3.cmml">r</mi><mo lspace="0em" rspace="0em" id="A5.SS1.p2.2.m2.1.1.3.1a" xref="A5.SS1.p2.2.m2.1.1.3.1.cmml">​</mo><mi id="A5.SS1.p2.2.m2.1.1.3.4" xref="A5.SS1.p2.2.m2.1.1.3.4.cmml">o</mi><mo lspace="0em" rspace="0em" id="A5.SS1.p2.2.m2.1.1.3.1b" xref="A5.SS1.p2.2.m2.1.1.3.1.cmml">​</mo><mi id="A5.SS1.p2.2.m2.1.1.3.5" xref="A5.SS1.p2.2.m2.1.1.3.5.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A5.SS1.p2.2.m2.1b"><apply id="A5.SS1.p2.2.m2.1.1.cmml" xref="A5.SS1.p2.2.m2.1.1"><csymbol cd="ambiguous" id="A5.SS1.p2.2.m2.1.1.1.cmml" xref="A5.SS1.p2.2.m2.1.1">subscript</csymbol><ci id="A5.SS1.p2.2.m2.1.1.2.cmml" xref="A5.SS1.p2.2.m2.1.1.2">mlp</ci><apply id="A5.SS1.p2.2.m2.1.1.3.cmml" xref="A5.SS1.p2.2.m2.1.1.3"><times id="A5.SS1.p2.2.m2.1.1.3.1.cmml" xref="A5.SS1.p2.2.m2.1.1.3.1"></times><ci id="A5.SS1.p2.2.m2.1.1.3.2.cmml" xref="A5.SS1.p2.2.m2.1.1.3.2">𝑝</ci><ci id="A5.SS1.p2.2.m2.1.1.3.3.cmml" xref="A5.SS1.p2.2.m2.1.1.3.3">𝑟</ci><ci id="A5.SS1.p2.2.m2.1.1.3.4.cmml" xref="A5.SS1.p2.2.m2.1.1.3.4">𝑜</ci><ci id="A5.SS1.p2.2.m2.1.1.3.5.cmml" xref="A5.SS1.p2.2.m2.1.1.3.5">𝑗</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.p2.2.m2.1c">\mathrm{mlp}_{proj}</annotation></semantics></math> weights at one layer. A hyperparameter search for GPT-2 XL (Figure <a href="#A5.F17" title="Figure 17 ‣ E.1 [GPT-2 XL, GPT-J] Fine-Tuning (FT), Constrained Fine-Tuning (FT+L) ‣ Appendix E Method Implementation Details ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">17</span></a>) reveals that layer 1 is the optimal place to conduct the intervention for FT, as neighborhood success sees a slight increase from layer 0. Following a similar methodology for GPT-J (Figure <a href="#A5.F18" title="Figure 18 ‣ E.1 [GPT-2 XL, GPT-J] Fine-Tuning (FT), Constrained Fine-Tuning (FT+L) ‣ Appendix E Method Implementation Details ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">18</span></a>), we select layer 21 because of the relative peak in neighborhood score. For both models, we use a learning rate of <math id="A5.SS1.p2.3.m3.1" class="ltx_Math" alttext="5\times 10^{-4}" display="inline"><semantics id="A5.SS1.p2.3.m3.1a"><mrow id="A5.SS1.p2.3.m3.1.1" xref="A5.SS1.p2.3.m3.1.1.cmml"><mn id="A5.SS1.p2.3.m3.1.1.2" xref="A5.SS1.p2.3.m3.1.1.2.cmml">5</mn><mo lspace="0.222em" rspace="0.222em" id="A5.SS1.p2.3.m3.1.1.1" xref="A5.SS1.p2.3.m3.1.1.1.cmml">×</mo><msup id="A5.SS1.p2.3.m3.1.1.3" xref="A5.SS1.p2.3.m3.1.1.3.cmml"><mn id="A5.SS1.p2.3.m3.1.1.3.2" xref="A5.SS1.p2.3.m3.1.1.3.2.cmml">10</mn><mrow id="A5.SS1.p2.3.m3.1.1.3.3" xref="A5.SS1.p2.3.m3.1.1.3.3.cmml"><mo id="A5.SS1.p2.3.m3.1.1.3.3a" xref="A5.SS1.p2.3.m3.1.1.3.3.cmml">−</mo><mn id="A5.SS1.p2.3.m3.1.1.3.3.2" xref="A5.SS1.p2.3.m3.1.1.3.3.2.cmml">4</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A5.SS1.p2.3.m3.1b"><apply id="A5.SS1.p2.3.m3.1.1.cmml" xref="A5.SS1.p2.3.m3.1.1"><times id="A5.SS1.p2.3.m3.1.1.1.cmml" xref="A5.SS1.p2.3.m3.1.1.1"></times><cn type="integer" id="A5.SS1.p2.3.m3.1.1.2.cmml" xref="A5.SS1.p2.3.m3.1.1.2">5</cn><apply id="A5.SS1.p2.3.m3.1.1.3.cmml" xref="A5.SS1.p2.3.m3.1.1.3"><csymbol cd="ambiguous" id="A5.SS1.p2.3.m3.1.1.3.1.cmml" xref="A5.SS1.p2.3.m3.1.1.3">superscript</csymbol><cn type="integer" id="A5.SS1.p2.3.m3.1.1.3.2.cmml" xref="A5.SS1.p2.3.m3.1.1.3.2">10</cn><apply id="A5.SS1.p2.3.m3.1.1.3.3.cmml" xref="A5.SS1.p2.3.m3.1.1.3.3"><minus id="A5.SS1.p2.3.m3.1.1.3.3.1.cmml" xref="A5.SS1.p2.3.m3.1.1.3.3"></minus><cn type="integer" id="A5.SS1.p2.3.m3.1.1.3.3.2.cmml" xref="A5.SS1.p2.3.m3.1.1.3.3.2">4</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.p2.3.m3.1c">5\times 10^{-4}</annotation></semantics></math> and early stop at a 0.03 loss.</p>
</div>
<div id="A5.SS1.p3" class="ltx_para">
<p id="A5.SS1.p3.6" class="ltx_p">For <span id="A5.SS1.p3.6.1" class="ltx_text ltx_font_italic">constrained</span> fine-tuning (FT+L), we draw from <cite class="ltx_cite ltx_citemacro_citet">Zhu et al. (<a href="#bib.bib43" title="" class="ltx_ref">2020</a>)</cite> by adding an <math id="A5.SS1.p3.1.m1.1" class="ltx_Math" alttext="L_{\infty}" display="inline"><semantics id="A5.SS1.p3.1.m1.1a"><msub id="A5.SS1.p3.1.m1.1.1" xref="A5.SS1.p3.1.m1.1.1.cmml"><mi id="A5.SS1.p3.1.m1.1.1.2" xref="A5.SS1.p3.1.m1.1.1.2.cmml">L</mi><mi mathvariant="normal" id="A5.SS1.p3.1.m1.1.1.3" xref="A5.SS1.p3.1.m1.1.1.3.cmml">∞</mi></msub><annotation-xml encoding="MathML-Content" id="A5.SS1.p3.1.m1.1b"><apply id="A5.SS1.p3.1.m1.1.1.cmml" xref="A5.SS1.p3.1.m1.1.1"><csymbol cd="ambiguous" id="A5.SS1.p3.1.m1.1.1.1.cmml" xref="A5.SS1.p3.1.m1.1.1">subscript</csymbol><ci id="A5.SS1.p3.1.m1.1.1.2.cmml" xref="A5.SS1.p3.1.m1.1.1.2">𝐿</ci><infinity id="A5.SS1.p3.1.m1.1.1.3.cmml" xref="A5.SS1.p3.1.m1.1.1.3"></infinity></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.p3.1.m1.1c">L_{\infty}</annotation></semantics></math> norm constraint: <math id="A5.SS1.p3.2.m2.1" class="ltx_Math" alttext="\lVert\theta_{G}-\theta_{G^{\prime}}\rVert_{\infty}\leq\epsilon" display="inline"><semantics id="A5.SS1.p3.2.m2.1a"><mrow id="A5.SS1.p3.2.m2.1.1" xref="A5.SS1.p3.2.m2.1.1.cmml"><msub id="A5.SS1.p3.2.m2.1.1.1" xref="A5.SS1.p3.2.m2.1.1.1.cmml"><mrow id="A5.SS1.p3.2.m2.1.1.1.1.1" xref="A5.SS1.p3.2.m2.1.1.1.1.2.cmml"><mo fence="true" rspace="0em" id="A5.SS1.p3.2.m2.1.1.1.1.1.2" xref="A5.SS1.p3.2.m2.1.1.1.1.2.1.cmml">∥</mo><mrow id="A5.SS1.p3.2.m2.1.1.1.1.1.1" xref="A5.SS1.p3.2.m2.1.1.1.1.1.1.cmml"><msub id="A5.SS1.p3.2.m2.1.1.1.1.1.1.2" xref="A5.SS1.p3.2.m2.1.1.1.1.1.1.2.cmml"><mi id="A5.SS1.p3.2.m2.1.1.1.1.1.1.2.2" xref="A5.SS1.p3.2.m2.1.1.1.1.1.1.2.2.cmml">θ</mi><mi id="A5.SS1.p3.2.m2.1.1.1.1.1.1.2.3" xref="A5.SS1.p3.2.m2.1.1.1.1.1.1.2.3.cmml">G</mi></msub><mo id="A5.SS1.p3.2.m2.1.1.1.1.1.1.1" xref="A5.SS1.p3.2.m2.1.1.1.1.1.1.1.cmml">−</mo><msub id="A5.SS1.p3.2.m2.1.1.1.1.1.1.3" xref="A5.SS1.p3.2.m2.1.1.1.1.1.1.3.cmml"><mi id="A5.SS1.p3.2.m2.1.1.1.1.1.1.3.2" xref="A5.SS1.p3.2.m2.1.1.1.1.1.1.3.2.cmml">θ</mi><msup id="A5.SS1.p3.2.m2.1.1.1.1.1.1.3.3" xref="A5.SS1.p3.2.m2.1.1.1.1.1.1.3.3.cmml"><mi id="A5.SS1.p3.2.m2.1.1.1.1.1.1.3.3.2" xref="A5.SS1.p3.2.m2.1.1.1.1.1.1.3.3.2.cmml">G</mi><mo id="A5.SS1.p3.2.m2.1.1.1.1.1.1.3.3.3" xref="A5.SS1.p3.2.m2.1.1.1.1.1.1.3.3.3.cmml">′</mo></msup></msub></mrow><mo fence="true" lspace="0em" id="A5.SS1.p3.2.m2.1.1.1.1.1.3" xref="A5.SS1.p3.2.m2.1.1.1.1.2.1.cmml">∥</mo></mrow><mi mathvariant="normal" id="A5.SS1.p3.2.m2.1.1.1.3" xref="A5.SS1.p3.2.m2.1.1.1.3.cmml">∞</mi></msub><mo id="A5.SS1.p3.2.m2.1.1.2" xref="A5.SS1.p3.2.m2.1.1.2.cmml">≤</mo><mi id="A5.SS1.p3.2.m2.1.1.3" xref="A5.SS1.p3.2.m2.1.1.3.cmml">ϵ</mi></mrow><annotation-xml encoding="MathML-Content" id="A5.SS1.p3.2.m2.1b"><apply id="A5.SS1.p3.2.m2.1.1.cmml" xref="A5.SS1.p3.2.m2.1.1"><leq id="A5.SS1.p3.2.m2.1.1.2.cmml" xref="A5.SS1.p3.2.m2.1.1.2"></leq><apply id="A5.SS1.p3.2.m2.1.1.1.cmml" xref="A5.SS1.p3.2.m2.1.1.1"><csymbol cd="ambiguous" id="A5.SS1.p3.2.m2.1.1.1.2.cmml" xref="A5.SS1.p3.2.m2.1.1.1">subscript</csymbol><apply id="A5.SS1.p3.2.m2.1.1.1.1.2.cmml" xref="A5.SS1.p3.2.m2.1.1.1.1.1"><csymbol cd="latexml" id="A5.SS1.p3.2.m2.1.1.1.1.2.1.cmml" xref="A5.SS1.p3.2.m2.1.1.1.1.1.2">delimited-∥∥</csymbol><apply id="A5.SS1.p3.2.m2.1.1.1.1.1.1.cmml" xref="A5.SS1.p3.2.m2.1.1.1.1.1.1"><minus id="A5.SS1.p3.2.m2.1.1.1.1.1.1.1.cmml" xref="A5.SS1.p3.2.m2.1.1.1.1.1.1.1"></minus><apply id="A5.SS1.p3.2.m2.1.1.1.1.1.1.2.cmml" xref="A5.SS1.p3.2.m2.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A5.SS1.p3.2.m2.1.1.1.1.1.1.2.1.cmml" xref="A5.SS1.p3.2.m2.1.1.1.1.1.1.2">subscript</csymbol><ci id="A5.SS1.p3.2.m2.1.1.1.1.1.1.2.2.cmml" xref="A5.SS1.p3.2.m2.1.1.1.1.1.1.2.2">𝜃</ci><ci id="A5.SS1.p3.2.m2.1.1.1.1.1.1.2.3.cmml" xref="A5.SS1.p3.2.m2.1.1.1.1.1.1.2.3">𝐺</ci></apply><apply id="A5.SS1.p3.2.m2.1.1.1.1.1.1.3.cmml" xref="A5.SS1.p3.2.m2.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A5.SS1.p3.2.m2.1.1.1.1.1.1.3.1.cmml" xref="A5.SS1.p3.2.m2.1.1.1.1.1.1.3">subscript</csymbol><ci id="A5.SS1.p3.2.m2.1.1.1.1.1.1.3.2.cmml" xref="A5.SS1.p3.2.m2.1.1.1.1.1.1.3.2">𝜃</ci><apply id="A5.SS1.p3.2.m2.1.1.1.1.1.1.3.3.cmml" xref="A5.SS1.p3.2.m2.1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="A5.SS1.p3.2.m2.1.1.1.1.1.1.3.3.1.cmml" xref="A5.SS1.p3.2.m2.1.1.1.1.1.1.3.3">superscript</csymbol><ci id="A5.SS1.p3.2.m2.1.1.1.1.1.1.3.3.2.cmml" xref="A5.SS1.p3.2.m2.1.1.1.1.1.1.3.3.2">𝐺</ci><ci id="A5.SS1.p3.2.m2.1.1.1.1.1.1.3.3.3.cmml" xref="A5.SS1.p3.2.m2.1.1.1.1.1.1.3.3.3">′</ci></apply></apply></apply></apply><infinity id="A5.SS1.p3.2.m2.1.1.1.3.cmml" xref="A5.SS1.p3.2.m2.1.1.1.3"></infinity></apply><ci id="A5.SS1.p3.2.m2.1.1.3.cmml" xref="A5.SS1.p3.2.m2.1.1.3">italic-ϵ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.p3.2.m2.1c">\lVert\theta_{G}-\theta_{G^{\prime}}\rVert_{\infty}\leq\epsilon</annotation></semantics></math>. This is achieved in practice by clamping weights <math id="A5.SS1.p3.3.m3.1" class="ltx_Math" alttext="\theta_{G}^{\prime}" display="inline"><semantics id="A5.SS1.p3.3.m3.1a"><msubsup id="A5.SS1.p3.3.m3.1.1" xref="A5.SS1.p3.3.m3.1.1.cmml"><mi id="A5.SS1.p3.3.m3.1.1.2.2" xref="A5.SS1.p3.3.m3.1.1.2.2.cmml">θ</mi><mi id="A5.SS1.p3.3.m3.1.1.2.3" xref="A5.SS1.p3.3.m3.1.1.2.3.cmml">G</mi><mo id="A5.SS1.p3.3.m3.1.1.3" xref="A5.SS1.p3.3.m3.1.1.3.cmml">′</mo></msubsup><annotation-xml encoding="MathML-Content" id="A5.SS1.p3.3.m3.1b"><apply id="A5.SS1.p3.3.m3.1.1.cmml" xref="A5.SS1.p3.3.m3.1.1"><csymbol cd="ambiguous" id="A5.SS1.p3.3.m3.1.1.1.cmml" xref="A5.SS1.p3.3.m3.1.1">superscript</csymbol><apply id="A5.SS1.p3.3.m3.1.1.2.cmml" xref="A5.SS1.p3.3.m3.1.1"><csymbol cd="ambiguous" id="A5.SS1.p3.3.m3.1.1.2.1.cmml" xref="A5.SS1.p3.3.m3.1.1">subscript</csymbol><ci id="A5.SS1.p3.3.m3.1.1.2.2.cmml" xref="A5.SS1.p3.3.m3.1.1.2.2">𝜃</ci><ci id="A5.SS1.p3.3.m3.1.1.2.3.cmml" xref="A5.SS1.p3.3.m3.1.1.2.3">𝐺</ci></apply><ci id="A5.SS1.p3.3.m3.1.1.3.cmml" xref="A5.SS1.p3.3.m3.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.p3.3.m3.1c">\theta_{G}^{\prime}</annotation></semantics></math> to the <math id="A5.SS1.p3.4.m4.1" class="ltx_Math" alttext="\theta_{G}\pm\epsilon" display="inline"><semantics id="A5.SS1.p3.4.m4.1a"><mrow id="A5.SS1.p3.4.m4.1.1" xref="A5.SS1.p3.4.m4.1.1.cmml"><msub id="A5.SS1.p3.4.m4.1.1.2" xref="A5.SS1.p3.4.m4.1.1.2.cmml"><mi id="A5.SS1.p3.4.m4.1.1.2.2" xref="A5.SS1.p3.4.m4.1.1.2.2.cmml">θ</mi><mi id="A5.SS1.p3.4.m4.1.1.2.3" xref="A5.SS1.p3.4.m4.1.1.2.3.cmml">G</mi></msub><mo id="A5.SS1.p3.4.m4.1.1.1" xref="A5.SS1.p3.4.m4.1.1.1.cmml">±</mo><mi id="A5.SS1.p3.4.m4.1.1.3" xref="A5.SS1.p3.4.m4.1.1.3.cmml">ϵ</mi></mrow><annotation-xml encoding="MathML-Content" id="A5.SS1.p3.4.m4.1b"><apply id="A5.SS1.p3.4.m4.1.1.cmml" xref="A5.SS1.p3.4.m4.1.1"><csymbol cd="latexml" id="A5.SS1.p3.4.m4.1.1.1.cmml" xref="A5.SS1.p3.4.m4.1.1.1">plus-or-minus</csymbol><apply id="A5.SS1.p3.4.m4.1.1.2.cmml" xref="A5.SS1.p3.4.m4.1.1.2"><csymbol cd="ambiguous" id="A5.SS1.p3.4.m4.1.1.2.1.cmml" xref="A5.SS1.p3.4.m4.1.1.2">subscript</csymbol><ci id="A5.SS1.p3.4.m4.1.1.2.2.cmml" xref="A5.SS1.p3.4.m4.1.1.2.2">𝜃</ci><ci id="A5.SS1.p3.4.m4.1.1.2.3.cmml" xref="A5.SS1.p3.4.m4.1.1.2.3">𝐺</ci></apply><ci id="A5.SS1.p3.4.m4.1.1.3.cmml" xref="A5.SS1.p3.4.m4.1.1.3">italic-ϵ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.p3.4.m4.1c">\theta_{G}\pm\epsilon</annotation></semantics></math> range at each gradient step. We select layer 0 and <math id="A5.SS1.p3.5.m5.1" class="ltx_Math" alttext="\epsilon=5\times 10^{-4}" display="inline"><semantics id="A5.SS1.p3.5.m5.1a"><mrow id="A5.SS1.p3.5.m5.1.1" xref="A5.SS1.p3.5.m5.1.1.cmml"><mi id="A5.SS1.p3.5.m5.1.1.2" xref="A5.SS1.p3.5.m5.1.1.2.cmml">ϵ</mi><mo id="A5.SS1.p3.5.m5.1.1.1" xref="A5.SS1.p3.5.m5.1.1.1.cmml">=</mo><mrow id="A5.SS1.p3.5.m5.1.1.3" xref="A5.SS1.p3.5.m5.1.1.3.cmml"><mn id="A5.SS1.p3.5.m5.1.1.3.2" xref="A5.SS1.p3.5.m5.1.1.3.2.cmml">5</mn><mo lspace="0.222em" rspace="0.222em" id="A5.SS1.p3.5.m5.1.1.3.1" xref="A5.SS1.p3.5.m5.1.1.3.1.cmml">×</mo><msup id="A5.SS1.p3.5.m5.1.1.3.3" xref="A5.SS1.p3.5.m5.1.1.3.3.cmml"><mn id="A5.SS1.p3.5.m5.1.1.3.3.2" xref="A5.SS1.p3.5.m5.1.1.3.3.2.cmml">10</mn><mrow id="A5.SS1.p3.5.m5.1.1.3.3.3" xref="A5.SS1.p3.5.m5.1.1.3.3.3.cmml"><mo id="A5.SS1.p3.5.m5.1.1.3.3.3a" xref="A5.SS1.p3.5.m5.1.1.3.3.3.cmml">−</mo><mn id="A5.SS1.p3.5.m5.1.1.3.3.3.2" xref="A5.SS1.p3.5.m5.1.1.3.3.3.2.cmml">4</mn></mrow></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.SS1.p3.5.m5.1b"><apply id="A5.SS1.p3.5.m5.1.1.cmml" xref="A5.SS1.p3.5.m5.1.1"><eq id="A5.SS1.p3.5.m5.1.1.1.cmml" xref="A5.SS1.p3.5.m5.1.1.1"></eq><ci id="A5.SS1.p3.5.m5.1.1.2.cmml" xref="A5.SS1.p3.5.m5.1.1.2">italic-ϵ</ci><apply id="A5.SS1.p3.5.m5.1.1.3.cmml" xref="A5.SS1.p3.5.m5.1.1.3"><times id="A5.SS1.p3.5.m5.1.1.3.1.cmml" xref="A5.SS1.p3.5.m5.1.1.3.1"></times><cn type="integer" id="A5.SS1.p3.5.m5.1.1.3.2.cmml" xref="A5.SS1.p3.5.m5.1.1.3.2">5</cn><apply id="A5.SS1.p3.5.m5.1.1.3.3.cmml" xref="A5.SS1.p3.5.m5.1.1.3.3"><csymbol cd="ambiguous" id="A5.SS1.p3.5.m5.1.1.3.3.1.cmml" xref="A5.SS1.p3.5.m5.1.1.3.3">superscript</csymbol><cn type="integer" id="A5.SS1.p3.5.m5.1.1.3.3.2.cmml" xref="A5.SS1.p3.5.m5.1.1.3.3.2">10</cn><apply id="A5.SS1.p3.5.m5.1.1.3.3.3.cmml" xref="A5.SS1.p3.5.m5.1.1.3.3.3"><minus id="A5.SS1.p3.5.m5.1.1.3.3.3.1.cmml" xref="A5.SS1.p3.5.m5.1.1.3.3.3"></minus><cn type="integer" id="A5.SS1.p3.5.m5.1.1.3.3.3.2.cmml" xref="A5.SS1.p3.5.m5.1.1.3.3.3.2">4</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.p3.5.m5.1c">\epsilon=5\times 10^{-4}</annotation></semantics></math> after a hyperparameter sweep (Figure <a href="#A5.F17" title="Figure 17 ‣ E.1 [GPT-2 XL, GPT-J] Fine-Tuning (FT), Constrained Fine-Tuning (FT+L) ‣ Appendix E Method Implementation Details ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">17</span></a>). For GPT-J, layer 0 and <math id="A5.SS1.p3.6.m6.1" class="ltx_Math" alttext="\epsilon=5\times 10^{-5}" display="inline"><semantics id="A5.SS1.p3.6.m6.1a"><mrow id="A5.SS1.p3.6.m6.1.1" xref="A5.SS1.p3.6.m6.1.1.cmml"><mi id="A5.SS1.p3.6.m6.1.1.2" xref="A5.SS1.p3.6.m6.1.1.2.cmml">ϵ</mi><mo id="A5.SS1.p3.6.m6.1.1.1" xref="A5.SS1.p3.6.m6.1.1.1.cmml">=</mo><mrow id="A5.SS1.p3.6.m6.1.1.3" xref="A5.SS1.p3.6.m6.1.1.3.cmml"><mn id="A5.SS1.p3.6.m6.1.1.3.2" xref="A5.SS1.p3.6.m6.1.1.3.2.cmml">5</mn><mo lspace="0.222em" rspace="0.222em" id="A5.SS1.p3.6.m6.1.1.3.1" xref="A5.SS1.p3.6.m6.1.1.3.1.cmml">×</mo><msup id="A5.SS1.p3.6.m6.1.1.3.3" xref="A5.SS1.p3.6.m6.1.1.3.3.cmml"><mn id="A5.SS1.p3.6.m6.1.1.3.3.2" xref="A5.SS1.p3.6.m6.1.1.3.3.2.cmml">10</mn><mrow id="A5.SS1.p3.6.m6.1.1.3.3.3" xref="A5.SS1.p3.6.m6.1.1.3.3.3.cmml"><mo id="A5.SS1.p3.6.m6.1.1.3.3.3a" xref="A5.SS1.p3.6.m6.1.1.3.3.3.cmml">−</mo><mn id="A5.SS1.p3.6.m6.1.1.3.3.3.2" xref="A5.SS1.p3.6.m6.1.1.3.3.3.2.cmml">5</mn></mrow></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.SS1.p3.6.m6.1b"><apply id="A5.SS1.p3.6.m6.1.1.cmml" xref="A5.SS1.p3.6.m6.1.1"><eq id="A5.SS1.p3.6.m6.1.1.1.cmml" xref="A5.SS1.p3.6.m6.1.1.1"></eq><ci id="A5.SS1.p3.6.m6.1.1.2.cmml" xref="A5.SS1.p3.6.m6.1.1.2">italic-ϵ</ci><apply id="A5.SS1.p3.6.m6.1.1.3.cmml" xref="A5.SS1.p3.6.m6.1.1.3"><times id="A5.SS1.p3.6.m6.1.1.3.1.cmml" xref="A5.SS1.p3.6.m6.1.1.3.1"></times><cn type="integer" id="A5.SS1.p3.6.m6.1.1.3.2.cmml" xref="A5.SS1.p3.6.m6.1.1.3.2">5</cn><apply id="A5.SS1.p3.6.m6.1.1.3.3.cmml" xref="A5.SS1.p3.6.m6.1.1.3.3"><csymbol cd="ambiguous" id="A5.SS1.p3.6.m6.1.1.3.3.1.cmml" xref="A5.SS1.p3.6.m6.1.1.3.3">superscript</csymbol><cn type="integer" id="A5.SS1.p3.6.m6.1.1.3.3.2.cmml" xref="A5.SS1.p3.6.m6.1.1.3.3.2">10</cn><apply id="A5.SS1.p3.6.m6.1.1.3.3.3.cmml" xref="A5.SS1.p3.6.m6.1.1.3.3.3"><minus id="A5.SS1.p3.6.m6.1.1.3.3.3.1.cmml" xref="A5.SS1.p3.6.m6.1.1.3.3.3"></minus><cn type="integer" id="A5.SS1.p3.6.m6.1.1.3.3.3.2.cmml" xref="A5.SS1.p3.6.m6.1.1.3.3.3.2">5</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.p3.6.m6.1c">\epsilon=5\times 10^{-5}</annotation></semantics></math> are selected to maximize both specificity and generalization. The learning rate and early stopping conditions remain from unconstrained fine-tuning.</p>
</div>
</section>
<section id="A5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">E.2 </span>[GPT-2 XL only] Knowledge Neurons (KN)</h3>

<div id="A5.SS2.p1" class="ltx_para">
<p id="A5.SS2.p1.3" class="ltx_p">The method by <cite class="ltx_cite ltx_citemacro_citet">Dai et al. (<a href="#bib.bib9" title="" class="ltx_ref">2022</a>)</cite> first selects neurons that are associated with knowledge expression via gradient-based attributions, and then modifies <math id="A5.SS2.p1.1.m1.1" class="ltx_Math" alttext="\smash{\mathrm{mlp}_{proj}^{(l)}}" display="inline"><semantics id="A5.SS2.p1.1.m1.1a"><msubsup id="A5.SS2.p1.1.m1.1.2" xref="A5.SS2.p1.1.m1.1.2.cmml"><mi id="A5.SS2.p1.1.m1.1.2.2.2" xref="A5.SS2.p1.1.m1.1.2.2.2.cmml">mlp</mi><mrow id="A5.SS2.p1.1.m1.1.2.2.3" xref="A5.SS2.p1.1.m1.1.2.2.3.cmml"><mi id="A5.SS2.p1.1.m1.1.2.2.3.2" xref="A5.SS2.p1.1.m1.1.2.2.3.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="A5.SS2.p1.1.m1.1.2.2.3.1" xref="A5.SS2.p1.1.m1.1.2.2.3.1.cmml">​</mo><mi id="A5.SS2.p1.1.m1.1.2.2.3.3" xref="A5.SS2.p1.1.m1.1.2.2.3.3.cmml">r</mi><mo lspace="0em" rspace="0em" id="A5.SS2.p1.1.m1.1.2.2.3.1a" xref="A5.SS2.p1.1.m1.1.2.2.3.1.cmml">​</mo><mi id="A5.SS2.p1.1.m1.1.2.2.3.4" xref="A5.SS2.p1.1.m1.1.2.2.3.4.cmml">o</mi><mo lspace="0em" rspace="0em" id="A5.SS2.p1.1.m1.1.2.2.3.1b" xref="A5.SS2.p1.1.m1.1.2.2.3.1.cmml">​</mo><mi id="A5.SS2.p1.1.m1.1.2.2.3.5" xref="A5.SS2.p1.1.m1.1.2.2.3.5.cmml">j</mi></mrow><mrow id="A5.SS2.p1.1.m1.1.1.1.3" xref="A5.SS2.p1.1.m1.1.2.cmml"><mo stretchy="false" id="A5.SS2.p1.1.m1.1.1.1.3.1" xref="A5.SS2.p1.1.m1.1.2.cmml">(</mo><mi id="A5.SS2.p1.1.m1.1.1.1.1" xref="A5.SS2.p1.1.m1.1.1.1.1.cmml">l</mi><mo stretchy="false" id="A5.SS2.p1.1.m1.1.1.1.3.2" xref="A5.SS2.p1.1.m1.1.2.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="A5.SS2.p1.1.m1.1b"><apply id="A5.SS2.p1.1.m1.1.2.cmml" xref="A5.SS2.p1.1.m1.1.2"><csymbol cd="ambiguous" id="A5.SS2.p1.1.m1.1.2.1.cmml" xref="A5.SS2.p1.1.m1.1.2">superscript</csymbol><apply id="A5.SS2.p1.1.m1.1.2.2.cmml" xref="A5.SS2.p1.1.m1.1.2"><csymbol cd="ambiguous" id="A5.SS2.p1.1.m1.1.2.2.1.cmml" xref="A5.SS2.p1.1.m1.1.2">subscript</csymbol><ci id="A5.SS2.p1.1.m1.1.2.2.2.cmml" xref="A5.SS2.p1.1.m1.1.2.2.2">mlp</ci><apply id="A5.SS2.p1.1.m1.1.2.2.3.cmml" xref="A5.SS2.p1.1.m1.1.2.2.3"><times id="A5.SS2.p1.1.m1.1.2.2.3.1.cmml" xref="A5.SS2.p1.1.m1.1.2.2.3.1"></times><ci id="A5.SS2.p1.1.m1.1.2.2.3.2.cmml" xref="A5.SS2.p1.1.m1.1.2.2.3.2">𝑝</ci><ci id="A5.SS2.p1.1.m1.1.2.2.3.3.cmml" xref="A5.SS2.p1.1.m1.1.2.2.3.3">𝑟</ci><ci id="A5.SS2.p1.1.m1.1.2.2.3.4.cmml" xref="A5.SS2.p1.1.m1.1.2.2.3.4">𝑜</ci><ci id="A5.SS2.p1.1.m1.1.2.2.3.5.cmml" xref="A5.SS2.p1.1.m1.1.2.2.3.5">𝑗</ci></apply></apply><ci id="A5.SS2.p1.1.m1.1.1.1.1.cmml" xref="A5.SS2.p1.1.m1.1.1.1.1">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS2.p1.1.m1.1c">\smash{\mathrm{mlp}_{proj}^{(l)}}</annotation></semantics></math> at the rows corresponding to those neurons by adding scaled embedding vectors.
This method has a <span id="A5.SS2.p1.3.1" class="ltx_text ltx_font_italic">coarse refinement</span> step, where the thousands of neurons in an MLP memory are whittled down to <math id="A5.SS2.p1.2.m2.1" class="ltx_Math" alttext="\approx 1000" display="inline"><semantics id="A5.SS2.p1.2.m2.1a"><mrow id="A5.SS2.p1.2.m2.1.1" xref="A5.SS2.p1.2.m2.1.1.cmml"><mi id="A5.SS2.p1.2.m2.1.1.2" xref="A5.SS2.p1.2.m2.1.1.2.cmml"></mi><mo id="A5.SS2.p1.2.m2.1.1.1" xref="A5.SS2.p1.2.m2.1.1.1.cmml">≈</mo><mn id="A5.SS2.p1.2.m2.1.1.3" xref="A5.SS2.p1.2.m2.1.1.3.cmml">1000</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.SS2.p1.2.m2.1b"><apply id="A5.SS2.p1.2.m2.1.1.cmml" xref="A5.SS2.p1.2.m2.1.1"><approx id="A5.SS2.p1.2.m2.1.1.1.cmml" xref="A5.SS2.p1.2.m2.1.1.1"></approx><csymbol cd="latexml" id="A5.SS2.p1.2.m2.1.1.2.cmml" xref="A5.SS2.p1.2.m2.1.1.2">absent</csymbol><cn type="integer" id="A5.SS2.p1.2.m2.1.1.3.cmml" xref="A5.SS2.p1.2.m2.1.1.3">1000</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS2.p1.2.m2.1c">\approx 1000</annotation></semantics></math> “knowledge neurons,” and a <span id="A5.SS2.p1.3.2" class="ltx_text ltx_font_italic">fine refinement</span> step that reduces the set of neurons to around <math id="A5.SS2.p1.3.m3.1" class="ltx_Math" alttext="\leq 10" display="inline"><semantics id="A5.SS2.p1.3.m3.1a"><mrow id="A5.SS2.p1.3.m3.1.1" xref="A5.SS2.p1.3.m3.1.1.cmml"><mi id="A5.SS2.p1.3.m3.1.1.2" xref="A5.SS2.p1.3.m3.1.1.2.cmml"></mi><mo id="A5.SS2.p1.3.m3.1.1.1" xref="A5.SS2.p1.3.m3.1.1.1.cmml">≤</mo><mn id="A5.SS2.p1.3.m3.1.1.3" xref="A5.SS2.p1.3.m3.1.1.3.cmml">10</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.SS2.p1.3.m3.1b"><apply id="A5.SS2.p1.3.m3.1.1.cmml" xref="A5.SS2.p1.3.m3.1.1"><leq id="A5.SS2.p1.3.m3.1.1.1.cmml" xref="A5.SS2.p1.3.m3.1.1.1"></leq><csymbol cd="latexml" id="A5.SS2.p1.3.m3.1.1.2.cmml" xref="A5.SS2.p1.3.m3.1.1.2">absent</csymbol><cn type="integer" id="A5.SS2.p1.3.m3.1.1.3.cmml" xref="A5.SS2.p1.3.m3.1.1.3">10</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS2.p1.3.m3.1c">\leq 10</annotation></semantics></math>.
All hyperparameters follow defaults as set in EleutherAI’s reimplementation: <a target="_blank" href="https://github.com/EleutherAI/knowledge-neurons" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/EleutherAI/knowledge-neurons</a>.</p>
</div>
</section>
<section id="A5.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">E.3 </span>[GPT-2 XL only] Knowledge Editor (KE)</h3>

<div id="A5.SS3.p1" class="ltx_para">
<p id="A5.SS3.p1.1" class="ltx_p"><cite class="ltx_cite ltx_citemacro_citet">De Cao et al. (<a href="#bib.bib10" title="" class="ltx_ref">2021</a>)</cite> learn an LSTM sequence model that uses gradient information to predict rank-1 weight changes to <math id="A5.SS3.p1.1.m1.1" class="ltx_Math" alttext="G" display="inline"><semantics id="A5.SS3.p1.1.m1.1a"><mi id="A5.SS3.p1.1.m1.1.1" xref="A5.SS3.p1.1.m1.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="A5.SS3.p1.1.m1.1b"><ci id="A5.SS3.p1.1.m1.1.1.cmml" xref="A5.SS3.p1.1.m1.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS3.p1.1.m1.1c">G</annotation></semantics></math>. Because the official code does not edit GPT-2, we use <cite class="ltx_cite ltx_citemacro_citet">Mitchell et al. (<a href="#bib.bib27" title="" class="ltx_ref">2021</a>)</cite>’s re-implementation in their study. To encourage fair comparison on both zsRE and <span id="A5.SS3.p1.1.1" class="ltx_text ltx_font_smallcaps">CounterFact</span> tasks, we additionally train KE-zsRE and KE-CF models on size-10,000 subsets of the respective training sets. Hyperparameters for training are adopted from the given default configuration.
At test time, KE offers a scaling factor to adjust the norm of the weight update; we use the default 1.0.</p>
</div>
</section>
<section id="A5.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">E.4 </span>[GPT-2 XL, GPT-J] Model Editor Networks with Gradient Decomposition (MEND)</h3>

<div id="A5.SS4.p1" class="ltx_para">
<p id="A5.SS4.p1.1" class="ltx_p"><cite class="ltx_cite ltx_citemacro_citet">Mitchell et al. (<a href="#bib.bib27" title="" class="ltx_ref">2021</a>)</cite> learn a rank-1 decomposition of the negative log likelihood gradient with respect to some subset of <math id="A5.SS4.p1.1.m1.1" class="ltx_Math" alttext="\theta_{G}" display="inline"><semantics id="A5.SS4.p1.1.m1.1a"><msub id="A5.SS4.p1.1.m1.1.1" xref="A5.SS4.p1.1.m1.1.1.cmml"><mi id="A5.SS4.p1.1.m1.1.1.2" xref="A5.SS4.p1.1.m1.1.1.2.cmml">θ</mi><mi id="A5.SS4.p1.1.m1.1.1.3" xref="A5.SS4.p1.1.m1.1.1.3.cmml">G</mi></msub><annotation-xml encoding="MathML-Content" id="A5.SS4.p1.1.m1.1b"><apply id="A5.SS4.p1.1.m1.1.1.cmml" xref="A5.SS4.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.SS4.p1.1.m1.1.1.1.cmml" xref="A5.SS4.p1.1.m1.1.1">subscript</csymbol><ci id="A5.SS4.p1.1.m1.1.1.2.cmml" xref="A5.SS4.p1.1.m1.1.1.2">𝜃</ci><ci id="A5.SS4.p1.1.m1.1.1.3.cmml" xref="A5.SS4.p1.1.m1.1.1.3">𝐺</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS4.p1.1.m1.1c">\theta_{G}</annotation></semantics></math> (in practice, this amounts to several of the last few layers of the transformer network). Again, for fair comparison, we train new versions of MEND (MEND-zsRE, MEND-CF) on the same sets that KE-zsRE and KE-CF were trained on. Similar to KE, hyperparameters for training and test-time inference are adopted from default configurations.</p>
</div>
</section>
<section id="A5.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">E.5 </span>[GPT-2 XL, GPT-J] Rank-One Model Editing (ROME)</h3>

<div id="A5.SS5.p1" class="ltx_para">
<p id="A5.SS5.p1.1" class="ltx_p">ROME’s update (Section <a href="#S3.SS1" title="3.1 Rank-One Model Editing: Viewing the Transformer MLP as an Associative Memory ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.1</span></a>) consists of key selection (Eqn. <a href="#S3.E3" title="In 3.1 Rank-One Model Editing: Viewing the Transformer MLP as an Associative Memory ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>), value optimization (Eqn. <a href="#S3.E4" title="In 3.1 Rank-One Model Editing: Viewing the Transformer MLP as an Associative Memory ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>), and <math id="A5.SS5.p1.1.m1.1" class="ltx_Math" alttext="v" display="inline"><semantics id="A5.SS5.p1.1.m1.1a"><mi id="A5.SS5.p1.1.m1.1.1" xref="A5.SS5.p1.1.m1.1.1.cmml">v</mi><annotation-xml encoding="MathML-Content" id="A5.SS5.p1.1.m1.1b"><ci id="A5.SS5.p1.1.m1.1.1.cmml" xref="A5.SS5.p1.1.m1.1.1">𝑣</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p1.1.m1.1c">v</annotation></semantics></math> insertion (Appendix <a href="#A1" title="Appendix A Solving for Λ Algebraically ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a>). We perform the intervention at layer 18. As Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>k shows, this is the center of causal effect in MLP layers, and as Figure <a href="#S2.F3" title="Figure 3 ‣ 2.2 Causal Tracing Results ‣ 2 Interventions on Activations for Tracing Information Flow ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> shows, layer 18 is approximately when MLP outputs begin to switch from acting as keys to values.</p>
</div>
<div id="A5.SS5.p2" class="ltx_para">
<p id="A5.SS5.p2.5" class="ltx_p"><span id="A5.SS5.p2.5.1" class="ltx_text ltx_font_bold">Second moment statistics</span>: Our second moment statistics <math id="A5.SS5.p2.1.m1.1" class="ltx_Math" alttext="C\propto\mathbb{E}[kk^{T}]" display="inline"><semantics id="A5.SS5.p2.1.m1.1a"><mrow id="A5.SS5.p2.1.m1.1.1" xref="A5.SS5.p2.1.m1.1.1.cmml"><mi id="A5.SS5.p2.1.m1.1.1.3" xref="A5.SS5.p2.1.m1.1.1.3.cmml">C</mi><mo id="A5.SS5.p2.1.m1.1.1.2" xref="A5.SS5.p2.1.m1.1.1.2.cmml">∝</mo><mrow id="A5.SS5.p2.1.m1.1.1.1" xref="A5.SS5.p2.1.m1.1.1.1.cmml"><mi id="A5.SS5.p2.1.m1.1.1.1.3" xref="A5.SS5.p2.1.m1.1.1.1.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="A5.SS5.p2.1.m1.1.1.1.2" xref="A5.SS5.p2.1.m1.1.1.1.2.cmml">​</mo><mrow id="A5.SS5.p2.1.m1.1.1.1.1.1" xref="A5.SS5.p2.1.m1.1.1.1.1.2.cmml"><mo stretchy="false" id="A5.SS5.p2.1.m1.1.1.1.1.1.2" xref="A5.SS5.p2.1.m1.1.1.1.1.2.1.cmml">[</mo><mrow id="A5.SS5.p2.1.m1.1.1.1.1.1.1" xref="A5.SS5.p2.1.m1.1.1.1.1.1.1.cmml"><mi id="A5.SS5.p2.1.m1.1.1.1.1.1.1.2" xref="A5.SS5.p2.1.m1.1.1.1.1.1.1.2.cmml">k</mi><mo lspace="0em" rspace="0em" id="A5.SS5.p2.1.m1.1.1.1.1.1.1.1" xref="A5.SS5.p2.1.m1.1.1.1.1.1.1.1.cmml">​</mo><msup id="A5.SS5.p2.1.m1.1.1.1.1.1.1.3" xref="A5.SS5.p2.1.m1.1.1.1.1.1.1.3.cmml"><mi id="A5.SS5.p2.1.m1.1.1.1.1.1.1.3.2" xref="A5.SS5.p2.1.m1.1.1.1.1.1.1.3.2.cmml">k</mi><mi id="A5.SS5.p2.1.m1.1.1.1.1.1.1.3.3" xref="A5.SS5.p2.1.m1.1.1.1.1.1.1.3.3.cmml">T</mi></msup></mrow><mo stretchy="false" id="A5.SS5.p2.1.m1.1.1.1.1.1.3" xref="A5.SS5.p2.1.m1.1.1.1.1.2.1.cmml">]</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.SS5.p2.1.m1.1b"><apply id="A5.SS5.p2.1.m1.1.1.cmml" xref="A5.SS5.p2.1.m1.1.1"><csymbol cd="latexml" id="A5.SS5.p2.1.m1.1.1.2.cmml" xref="A5.SS5.p2.1.m1.1.1.2">proportional-to</csymbol><ci id="A5.SS5.p2.1.m1.1.1.3.cmml" xref="A5.SS5.p2.1.m1.1.1.3">𝐶</ci><apply id="A5.SS5.p2.1.m1.1.1.1.cmml" xref="A5.SS5.p2.1.m1.1.1.1"><times id="A5.SS5.p2.1.m1.1.1.1.2.cmml" xref="A5.SS5.p2.1.m1.1.1.1.2"></times><ci id="A5.SS5.p2.1.m1.1.1.1.3.cmml" xref="A5.SS5.p2.1.m1.1.1.1.3">𝔼</ci><apply id="A5.SS5.p2.1.m1.1.1.1.1.2.cmml" xref="A5.SS5.p2.1.m1.1.1.1.1.1"><csymbol cd="latexml" id="A5.SS5.p2.1.m1.1.1.1.1.2.1.cmml" xref="A5.SS5.p2.1.m1.1.1.1.1.1.2">delimited-[]</csymbol><apply id="A5.SS5.p2.1.m1.1.1.1.1.1.1.cmml" xref="A5.SS5.p2.1.m1.1.1.1.1.1.1"><times id="A5.SS5.p2.1.m1.1.1.1.1.1.1.1.cmml" xref="A5.SS5.p2.1.m1.1.1.1.1.1.1.1"></times><ci id="A5.SS5.p2.1.m1.1.1.1.1.1.1.2.cmml" xref="A5.SS5.p2.1.m1.1.1.1.1.1.1.2">𝑘</ci><apply id="A5.SS5.p2.1.m1.1.1.1.1.1.1.3.cmml" xref="A5.SS5.p2.1.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A5.SS5.p2.1.m1.1.1.1.1.1.1.3.1.cmml" xref="A5.SS5.p2.1.m1.1.1.1.1.1.1.3">superscript</csymbol><ci id="A5.SS5.p2.1.m1.1.1.1.1.1.1.3.2.cmml" xref="A5.SS5.p2.1.m1.1.1.1.1.1.1.3.2">𝑘</ci><ci id="A5.SS5.p2.1.m1.1.1.1.1.1.1.3.3.cmml" xref="A5.SS5.p2.1.m1.1.1.1.1.1.1.3.3">𝑇</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p2.1.m1.1c">C\propto\mathbb{E}[kk^{T}]</annotation></semantics></math> are computed using 100,000 samples of hidden states <math id="A5.SS5.p2.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A5.SS5.p2.2.m2.1a"><mi id="A5.SS5.p2.2.m2.1.1" xref="A5.SS5.p2.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A5.SS5.p2.2.m2.1b"><ci id="A5.SS5.p2.2.m2.1.1.cmml" xref="A5.SS5.p2.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p2.2.m2.1c">k</annotation></semantics></math> computed from tokens sampled from <span id="A5.SS5.p2.5.2" class="ltx_text ltx_font_bold">all</span> Wikipedia text in-context. Notice that sampling is not restricted to only special subject words; every token in the text is included in the statistic. The samples of hidden state <math id="A5.SS5.p2.3.m3.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A5.SS5.p2.3.m3.1a"><mi id="A5.SS5.p2.3.m3.1.1" xref="A5.SS5.p2.3.m3.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A5.SS5.p2.3.m3.1b"><ci id="A5.SS5.p2.3.m3.1.1.cmml" xref="A5.SS5.p2.3.m3.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p2.3.m3.1c">k</annotation></semantics></math> vectors are collected by selecting a random sample of Wikipedia articles from the 2020-05-01 snapshot of Wikipedia; the full text of each sampled article run through the transformer, up to the transformer’s buffer length, and then all the fan-out MLP activations <math id="A5.SS5.p2.4.m4.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A5.SS5.p2.4.m4.1a"><mi id="A5.SS5.p2.4.m4.1.1" xref="A5.SS5.p2.4.m4.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A5.SS5.p2.4.m4.1b"><ci id="A5.SS5.p2.4.m4.1.1.cmml" xref="A5.SS5.p2.4.m4.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p2.4.m4.1c">k</annotation></semantics></math> for every token in the article are collected at <span id="A5.SS5.p2.5.3" class="ltx_text ltx_font_typewriter">float32</span> precision. The process is repeated (sampling from further Wikipedia articles without replacement) until 100,000 <math id="A5.SS5.p2.5.m5.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A5.SS5.p2.5.m5.1a"><mi id="A5.SS5.p2.5.m5.1.1" xref="A5.SS5.p2.5.m5.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A5.SS5.p2.5.m5.1b"><ci id="A5.SS5.p2.5.m5.1.1.cmml" xref="A5.SS5.p2.5.m5.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p2.5.m5.1c">k</annotation></semantics></math> vectors have been sampled. This sample of vectors is used to compute second moment statistics.</p>
</div>
<div id="A5.SS5.p3" class="ltx_para">
<p id="A5.SS5.p3.4" class="ltx_p"><span id="A5.SS5.p3.4.1" class="ltx_text ltx_font_bold">Key Selection</span>: We sample 20 texts to compute the prefix (<math id="A5.SS5.p3.1.m1.1" class="ltx_Math" alttext="x_{j}" display="inline"><semantics id="A5.SS5.p3.1.m1.1a"><msub id="A5.SS5.p3.1.m1.1.1" xref="A5.SS5.p3.1.m1.1.1.cmml"><mi id="A5.SS5.p3.1.m1.1.1.2" xref="A5.SS5.p3.1.m1.1.1.2.cmml">x</mi><mi id="A5.SS5.p3.1.m1.1.1.3" xref="A5.SS5.p3.1.m1.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="A5.SS5.p3.1.m1.1b"><apply id="A5.SS5.p3.1.m1.1.1.cmml" xref="A5.SS5.p3.1.m1.1.1"><csymbol cd="ambiguous" id="A5.SS5.p3.1.m1.1.1.1.cmml" xref="A5.SS5.p3.1.m1.1.1">subscript</csymbol><ci id="A5.SS5.p3.1.m1.1.1.2.cmml" xref="A5.SS5.p3.1.m1.1.1.2">𝑥</ci><ci id="A5.SS5.p3.1.m1.1.1.3.cmml" xref="A5.SS5.p3.1.m1.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p3.1.m1.1c">x_{j}</annotation></semantics></math> in Eqn. <a href="#S3.E3" title="In 3.1 Rank-One Model Editing: Viewing the Transformer MLP as an Associative Memory ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>): ten of length 5 and ten of length 10. The intention is to pick a <math id="A5.SS5.p3.2.m2.1" class="ltx_Math" alttext="k_{*}" display="inline"><semantics id="A5.SS5.p3.2.m2.1a"><msub id="A5.SS5.p3.2.m2.1.1" xref="A5.SS5.p3.2.m2.1.1.cmml"><mi id="A5.SS5.p3.2.m2.1.1.2" xref="A5.SS5.p3.2.m2.1.1.2.cmml">k</mi><mo id="A5.SS5.p3.2.m2.1.1.3" xref="A5.SS5.p3.2.m2.1.1.3.cmml">∗</mo></msub><annotation-xml encoding="MathML-Content" id="A5.SS5.p3.2.m2.1b"><apply id="A5.SS5.p3.2.m2.1.1.cmml" xref="A5.SS5.p3.2.m2.1.1"><csymbol cd="ambiguous" id="A5.SS5.p3.2.m2.1.1.1.cmml" xref="A5.SS5.p3.2.m2.1.1">subscript</csymbol><ci id="A5.SS5.p3.2.m2.1.1.2.cmml" xref="A5.SS5.p3.2.m2.1.1.2">𝑘</ci><times id="A5.SS5.p3.2.m2.1.1.3.cmml" xref="A5.SS5.p3.2.m2.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p3.2.m2.1c">k_{*}</annotation></semantics></math> that accounts for the different contexts in which <math id="A5.SS5.p3.3.m3.1" class="ltx_Math" alttext="s" display="inline"><semantics id="A5.SS5.p3.3.m3.1a"><mi id="A5.SS5.p3.3.m3.1.1" xref="A5.SS5.p3.3.m3.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="A5.SS5.p3.3.m3.1b"><ci id="A5.SS5.p3.3.m3.1.1.cmml" xref="A5.SS5.p3.3.m3.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p3.3.m3.1c">s</annotation></semantics></math> could appear. Note that we also experimented with other <math id="A5.SS5.p3.4.m4.1" class="ltx_Math" alttext="x_{j}" display="inline"><semantics id="A5.SS5.p3.4.m4.1a"><msub id="A5.SS5.p3.4.m4.1.1" xref="A5.SS5.p3.4.m4.1.1.cmml"><mi id="A5.SS5.p3.4.m4.1.1.2" xref="A5.SS5.p3.4.m4.1.1.2.cmml">x</mi><mi id="A5.SS5.p3.4.m4.1.1.3" xref="A5.SS5.p3.4.m4.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="A5.SS5.p3.4.m4.1b"><apply id="A5.SS5.p3.4.m4.1.1.cmml" xref="A5.SS5.p3.4.m4.1.1"><csymbol cd="ambiguous" id="A5.SS5.p3.4.m4.1.1.1.cmml" xref="A5.SS5.p3.4.m4.1.1">subscript</csymbol><ci id="A5.SS5.p3.4.m4.1.1.2.cmml" xref="A5.SS5.p3.4.m4.1.1.2">𝑥</ci><ci id="A5.SS5.p3.4.m4.1.1.3.cmml" xref="A5.SS5.p3.4.m4.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p3.4.m4.1c">x_{j}</annotation></semantics></math> sampling methods:</p>
<ul id="A5.I1" class="ltx_itemize">
<li id="A5.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A5.I1.i1.p1" class="ltx_para">
<p id="A5.I1.i1.p1.2" class="ltx_p"><span id="A5.I1.i1.p1.2.1" class="ltx_text ltx_font_bold">No prefix</span>. This baseline option performed worse (<math id="A5.I1.i1.p1.1.m1.1" class="ltx_Math" alttext="\mathrm{S}^{\prime}=86.1" display="inline"><semantics id="A5.I1.i1.p1.1.m1.1a"><mrow id="A5.I1.i1.p1.1.m1.1.1" xref="A5.I1.i1.p1.1.m1.1.1.cmml"><msup id="A5.I1.i1.p1.1.m1.1.1.2" xref="A5.I1.i1.p1.1.m1.1.1.2.cmml"><mi mathvariant="normal" id="A5.I1.i1.p1.1.m1.1.1.2.2" xref="A5.I1.i1.p1.1.m1.1.1.2.2.cmml">S</mi><mo id="A5.I1.i1.p1.1.m1.1.1.2.3" xref="A5.I1.i1.p1.1.m1.1.1.2.3.cmml">′</mo></msup><mo id="A5.I1.i1.p1.1.m1.1.1.1" xref="A5.I1.i1.p1.1.m1.1.1.1.cmml">=</mo><mn id="A5.I1.i1.p1.1.m1.1.1.3" xref="A5.I1.i1.p1.1.m1.1.1.3.cmml">86.1</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.I1.i1.p1.1.m1.1b"><apply id="A5.I1.i1.p1.1.m1.1.1.cmml" xref="A5.I1.i1.p1.1.m1.1.1"><eq id="A5.I1.i1.p1.1.m1.1.1.1.cmml" xref="A5.I1.i1.p1.1.m1.1.1.1"></eq><apply id="A5.I1.i1.p1.1.m1.1.1.2.cmml" xref="A5.I1.i1.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="A5.I1.i1.p1.1.m1.1.1.2.1.cmml" xref="A5.I1.i1.p1.1.m1.1.1.2">superscript</csymbol><ci id="A5.I1.i1.p1.1.m1.1.1.2.2.cmml" xref="A5.I1.i1.p1.1.m1.1.1.2.2">S</ci><ci id="A5.I1.i1.p1.1.m1.1.1.2.3.cmml" xref="A5.I1.i1.p1.1.m1.1.1.2.3">′</ci></apply><cn type="float" id="A5.I1.i1.p1.1.m1.1.1.3.cmml" xref="A5.I1.i1.p1.1.m1.1.1.3">86.1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.I1.i1.p1.1.m1.1c">\mathrm{S}^{\prime}=86.1</annotation></semantics></math> compared to <math id="A5.I1.i1.p1.2.m2.1" class="ltx_Math" alttext="\mathrm{S}=89.2" display="inline"><semantics id="A5.I1.i1.p1.2.m2.1a"><mrow id="A5.I1.i1.p1.2.m2.1.1" xref="A5.I1.i1.p1.2.m2.1.1.cmml"><mi mathvariant="normal" id="A5.I1.i1.p1.2.m2.1.1.2" xref="A5.I1.i1.p1.2.m2.1.1.2.cmml">S</mi><mo id="A5.I1.i1.p1.2.m2.1.1.1" xref="A5.I1.i1.p1.2.m2.1.1.1.cmml">=</mo><mn id="A5.I1.i1.p1.2.m2.1.1.3" xref="A5.I1.i1.p1.2.m2.1.1.3.cmml">89.2</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.I1.i1.p1.2.m2.1b"><apply id="A5.I1.i1.p1.2.m2.1.1.cmml" xref="A5.I1.i1.p1.2.m2.1.1"><eq id="A5.I1.i1.p1.2.m2.1.1.1.cmml" xref="A5.I1.i1.p1.2.m2.1.1.1"></eq><ci id="A5.I1.i1.p1.2.m2.1.1.2.cmml" xref="A5.I1.i1.p1.2.m2.1.1.2">S</ci><cn type="float" id="A5.I1.i1.p1.2.m2.1.1.3.cmml" xref="A5.I1.i1.p1.2.m2.1.1.3">89.2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.I1.i1.p1.2.m2.1c">\mathrm{S}=89.2</annotation></semantics></math>).</p>
</div>
</li>
<li id="A5.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A5.I1.i2.p1" class="ltx_para">
<p id="A5.I1.i2.p1.1" class="ltx_p"><span id="A5.I1.i2.p1.1.1" class="ltx_text ltx_font_bold">Longer prefixes</span>. Using { ten of length 5, ten of length 10, and ten of length 50 } did not help performance much (<math id="A5.I1.i2.p1.1.m1.1" class="ltx_Math" alttext="\mathrm{S}^{\prime}=89.3" display="inline"><semantics id="A5.I1.i2.p1.1.m1.1a"><mrow id="A5.I1.i2.p1.1.m1.1.1" xref="A5.I1.i2.p1.1.m1.1.1.cmml"><msup id="A5.I1.i2.p1.1.m1.1.1.2" xref="A5.I1.i2.p1.1.m1.1.1.2.cmml"><mi mathvariant="normal" id="A5.I1.i2.p1.1.m1.1.1.2.2" xref="A5.I1.i2.p1.1.m1.1.1.2.2.cmml">S</mi><mo id="A5.I1.i2.p1.1.m1.1.1.2.3" xref="A5.I1.i2.p1.1.m1.1.1.2.3.cmml">′</mo></msup><mo id="A5.I1.i2.p1.1.m1.1.1.1" xref="A5.I1.i2.p1.1.m1.1.1.1.cmml">=</mo><mn id="A5.I1.i2.p1.1.m1.1.1.3" xref="A5.I1.i2.p1.1.m1.1.1.3.cmml">89.3</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.I1.i2.p1.1.m1.1b"><apply id="A5.I1.i2.p1.1.m1.1.1.cmml" xref="A5.I1.i2.p1.1.m1.1.1"><eq id="A5.I1.i2.p1.1.m1.1.1.1.cmml" xref="A5.I1.i2.p1.1.m1.1.1.1"></eq><apply id="A5.I1.i2.p1.1.m1.1.1.2.cmml" xref="A5.I1.i2.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="A5.I1.i2.p1.1.m1.1.1.2.1.cmml" xref="A5.I1.i2.p1.1.m1.1.1.2">superscript</csymbol><ci id="A5.I1.i2.p1.1.m1.1.1.2.2.cmml" xref="A5.I1.i2.p1.1.m1.1.1.2.2">S</ci><ci id="A5.I1.i2.p1.1.m1.1.1.2.3.cmml" xref="A5.I1.i2.p1.1.m1.1.1.2.3">′</ci></apply><cn type="float" id="A5.I1.i2.p1.1.m1.1.1.3.cmml" xref="A5.I1.i2.p1.1.m1.1.1.3">89.3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.I1.i2.p1.1.m1.1c">\mathrm{S}^{\prime}=89.3</annotation></semantics></math>).</p>
</div>
</li>
<li id="A5.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A5.I1.i3.p1" class="ltx_para">
<p id="A5.I1.i3.p1.1" class="ltx_p"><span id="A5.I1.i3.p1.1.1" class="ltx_text ltx_font_bold">More same-length prefixes</span>. Using { thirty of length 5 and thirty of length 10 } did not help performance much (<math id="A5.I1.i3.p1.1.m1.1" class="ltx_Math" alttext="\mathrm{S}^{\prime}=89.2" display="inline"><semantics id="A5.I1.i3.p1.1.m1.1a"><mrow id="A5.I1.i3.p1.1.m1.1.1" xref="A5.I1.i3.p1.1.m1.1.1.cmml"><msup id="A5.I1.i3.p1.1.m1.1.1.2" xref="A5.I1.i3.p1.1.m1.1.1.2.cmml"><mi mathvariant="normal" id="A5.I1.i3.p1.1.m1.1.1.2.2" xref="A5.I1.i3.p1.1.m1.1.1.2.2.cmml">S</mi><mo id="A5.I1.i3.p1.1.m1.1.1.2.3" xref="A5.I1.i3.p1.1.m1.1.1.2.3.cmml">′</mo></msup><mo id="A5.I1.i3.p1.1.m1.1.1.1" xref="A5.I1.i3.p1.1.m1.1.1.1.cmml">=</mo><mn id="A5.I1.i3.p1.1.m1.1.1.3" xref="A5.I1.i3.p1.1.m1.1.1.3.cmml">89.2</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.I1.i3.p1.1.m1.1b"><apply id="A5.I1.i3.p1.1.m1.1.1.cmml" xref="A5.I1.i3.p1.1.m1.1.1"><eq id="A5.I1.i3.p1.1.m1.1.1.1.cmml" xref="A5.I1.i3.p1.1.m1.1.1.1"></eq><apply id="A5.I1.i3.p1.1.m1.1.1.2.cmml" xref="A5.I1.i3.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="A5.I1.i3.p1.1.m1.1.1.2.1.cmml" xref="A5.I1.i3.p1.1.m1.1.1.2">superscript</csymbol><ci id="A5.I1.i3.p1.1.m1.1.1.2.2.cmml" xref="A5.I1.i3.p1.1.m1.1.1.2.2">S</ci><ci id="A5.I1.i3.p1.1.m1.1.1.2.3.cmml" xref="A5.I1.i3.p1.1.m1.1.1.2.3">′</ci></apply><cn type="float" id="A5.I1.i3.p1.1.m1.1.1.3.cmml" xref="A5.I1.i3.p1.1.m1.1.1.3">89.2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.I1.i3.p1.1.m1.1c">\mathrm{S}^{\prime}=89.2</annotation></semantics></math>).</p>
</div>
</li>
</ul>
</div>
<div id="A5.SS5.p4" class="ltx_para">
<p id="A5.SS5.p4.7" class="ltx_p"><span id="A5.SS5.p4.7.1" class="ltx_text ltx_font_bold">Value Optimization</span>: <math id="A5.SS5.p4.1.m1.1" class="ltx_Math" alttext="v_{*}" display="inline"><semantics id="A5.SS5.p4.1.m1.1a"><msub id="A5.SS5.p4.1.m1.1.1" xref="A5.SS5.p4.1.m1.1.1.cmml"><mi id="A5.SS5.p4.1.m1.1.1.2" xref="A5.SS5.p4.1.m1.1.1.2.cmml">v</mi><mo id="A5.SS5.p4.1.m1.1.1.3" xref="A5.SS5.p4.1.m1.1.1.3.cmml">∗</mo></msub><annotation-xml encoding="MathML-Content" id="A5.SS5.p4.1.m1.1b"><apply id="A5.SS5.p4.1.m1.1.1.cmml" xref="A5.SS5.p4.1.m1.1.1"><csymbol cd="ambiguous" id="A5.SS5.p4.1.m1.1.1.1.cmml" xref="A5.SS5.p4.1.m1.1.1">subscript</csymbol><ci id="A5.SS5.p4.1.m1.1.1.2.cmml" xref="A5.SS5.p4.1.m1.1.1.2">𝑣</ci><times id="A5.SS5.p4.1.m1.1.1.3.cmml" xref="A5.SS5.p4.1.m1.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p4.1.m1.1c">v_{*}</annotation></semantics></math> is solved for using Adam with a learning rate of <math id="A5.SS5.p4.2.m2.1" class="ltx_Math" alttext="0.5" display="inline"><semantics id="A5.SS5.p4.2.m2.1a"><mn id="A5.SS5.p4.2.m2.1.1" xref="A5.SS5.p4.2.m2.1.1.cmml">0.5</mn><annotation-xml encoding="MathML-Content" id="A5.SS5.p4.2.m2.1b"><cn type="float" id="A5.SS5.p4.2.m2.1.1.cmml" xref="A5.SS5.p4.2.m2.1.1">0.5</cn></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p4.2.m2.1c">0.5</annotation></semantics></math> and <math id="A5.SS5.p4.3.m3.1" class="ltx_Math" alttext="1.5\times 10^{-3}" display="inline"><semantics id="A5.SS5.p4.3.m3.1a"><mrow id="A5.SS5.p4.3.m3.1.1" xref="A5.SS5.p4.3.m3.1.1.cmml"><mn id="A5.SS5.p4.3.m3.1.1.2" xref="A5.SS5.p4.3.m3.1.1.2.cmml">1.5</mn><mo lspace="0.222em" rspace="0.222em" id="A5.SS5.p4.3.m3.1.1.1" xref="A5.SS5.p4.3.m3.1.1.1.cmml">×</mo><msup id="A5.SS5.p4.3.m3.1.1.3" xref="A5.SS5.p4.3.m3.1.1.3.cmml"><mn id="A5.SS5.p4.3.m3.1.1.3.2" xref="A5.SS5.p4.3.m3.1.1.3.2.cmml">10</mn><mrow id="A5.SS5.p4.3.m3.1.1.3.3" xref="A5.SS5.p4.3.m3.1.1.3.3.cmml"><mo id="A5.SS5.p4.3.m3.1.1.3.3a" xref="A5.SS5.p4.3.m3.1.1.3.3.cmml">−</mo><mn id="A5.SS5.p4.3.m3.1.1.3.3.2" xref="A5.SS5.p4.3.m3.1.1.3.3.2.cmml">3</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A5.SS5.p4.3.m3.1b"><apply id="A5.SS5.p4.3.m3.1.1.cmml" xref="A5.SS5.p4.3.m3.1.1"><times id="A5.SS5.p4.3.m3.1.1.1.cmml" xref="A5.SS5.p4.3.m3.1.1.1"></times><cn type="float" id="A5.SS5.p4.3.m3.1.1.2.cmml" xref="A5.SS5.p4.3.m3.1.1.2">1.5</cn><apply id="A5.SS5.p4.3.m3.1.1.3.cmml" xref="A5.SS5.p4.3.m3.1.1.3"><csymbol cd="ambiguous" id="A5.SS5.p4.3.m3.1.1.3.1.cmml" xref="A5.SS5.p4.3.m3.1.1.3">superscript</csymbol><cn type="integer" id="A5.SS5.p4.3.m3.1.1.3.2.cmml" xref="A5.SS5.p4.3.m3.1.1.3.2">10</cn><apply id="A5.SS5.p4.3.m3.1.1.3.3.cmml" xref="A5.SS5.p4.3.m3.1.1.3.3"><minus id="A5.SS5.p4.3.m3.1.1.3.3.1.cmml" xref="A5.SS5.p4.3.m3.1.1.3.3"></minus><cn type="integer" id="A5.SS5.p4.3.m3.1.1.3.3.2.cmml" xref="A5.SS5.p4.3.m3.1.1.3.3.2">3</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p4.3.m3.1c">1.5\times 10^{-3}</annotation></semantics></math> weight decay. The KL divergence scaling factor, denoted <math id="A5.SS5.p4.4.m4.1" class="ltx_Math" alttext="\lambda" display="inline"><semantics id="A5.SS5.p4.4.m4.1a"><mi id="A5.SS5.p4.4.m4.1.1" xref="A5.SS5.p4.4.m4.1.1.cmml">λ</mi><annotation-xml encoding="MathML-Content" id="A5.SS5.p4.4.m4.1b"><ci id="A5.SS5.p4.4.m4.1.1.cmml" xref="A5.SS5.p4.4.m4.1.1">𝜆</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p4.4.m4.1c">\lambda</annotation></semantics></math> in Eqn. <a href="#S3.E4" title="In 3.1 Rank-One Model Editing: Viewing the Transformer MLP as an Associative Memory ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, is set to <math id="A5.SS5.p4.5.m5.1" class="ltx_Math" alttext="1\times 10^{2}" display="inline"><semantics id="A5.SS5.p4.5.m5.1a"><mrow id="A5.SS5.p4.5.m5.1.1" xref="A5.SS5.p4.5.m5.1.1.cmml"><mn id="A5.SS5.p4.5.m5.1.1.2" xref="A5.SS5.p4.5.m5.1.1.2.cmml">1</mn><mo lspace="0.222em" rspace="0.222em" id="A5.SS5.p4.5.m5.1.1.1" xref="A5.SS5.p4.5.m5.1.1.1.cmml">×</mo><msup id="A5.SS5.p4.5.m5.1.1.3" xref="A5.SS5.p4.5.m5.1.1.3.cmml"><mn id="A5.SS5.p4.5.m5.1.1.3.2" xref="A5.SS5.p4.5.m5.1.1.3.2.cmml">10</mn><mn id="A5.SS5.p4.5.m5.1.1.3.3" xref="A5.SS5.p4.5.m5.1.1.3.3.cmml">2</mn></msup></mrow><annotation-xml encoding="MathML-Content" id="A5.SS5.p4.5.m5.1b"><apply id="A5.SS5.p4.5.m5.1.1.cmml" xref="A5.SS5.p4.5.m5.1.1"><times id="A5.SS5.p4.5.m5.1.1.1.cmml" xref="A5.SS5.p4.5.m5.1.1.1"></times><cn type="integer" id="A5.SS5.p4.5.m5.1.1.2.cmml" xref="A5.SS5.p4.5.m5.1.1.2">1</cn><apply id="A5.SS5.p4.5.m5.1.1.3.cmml" xref="A5.SS5.p4.5.m5.1.1.3"><csymbol cd="ambiguous" id="A5.SS5.p4.5.m5.1.1.3.1.cmml" xref="A5.SS5.p4.5.m5.1.1.3">superscript</csymbol><cn type="integer" id="A5.SS5.p4.5.m5.1.1.3.2.cmml" xref="A5.SS5.p4.5.m5.1.1.3.2">10</cn><cn type="integer" id="A5.SS5.p4.5.m5.1.1.3.3.cmml" xref="A5.SS5.p4.5.m5.1.1.3.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p4.5.m5.1c">1\times 10^{2}</annotation></semantics></math>. The minimization loop is run for a maximum of 20 steps, with early stopping when <math id="A5.SS5.p4.6.m6.1" class="ltx_Math" alttext="\mathcal{L}(z)" display="inline"><semantics id="A5.SS5.p4.6.m6.1a"><mrow id="A5.SS5.p4.6.m6.1.2" xref="A5.SS5.p4.6.m6.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="A5.SS5.p4.6.m6.1.2.2" xref="A5.SS5.p4.6.m6.1.2.2.cmml">ℒ</mi><mo lspace="0em" rspace="0em" id="A5.SS5.p4.6.m6.1.2.1" xref="A5.SS5.p4.6.m6.1.2.1.cmml">​</mo><mrow id="A5.SS5.p4.6.m6.1.2.3.2" xref="A5.SS5.p4.6.m6.1.2.cmml"><mo stretchy="false" id="A5.SS5.p4.6.m6.1.2.3.2.1" xref="A5.SS5.p4.6.m6.1.2.cmml">(</mo><mi id="A5.SS5.p4.6.m6.1.1" xref="A5.SS5.p4.6.m6.1.1.cmml">z</mi><mo stretchy="false" id="A5.SS5.p4.6.m6.1.2.3.2.2" xref="A5.SS5.p4.6.m6.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.SS5.p4.6.m6.1b"><apply id="A5.SS5.p4.6.m6.1.2.cmml" xref="A5.SS5.p4.6.m6.1.2"><times id="A5.SS5.p4.6.m6.1.2.1.cmml" xref="A5.SS5.p4.6.m6.1.2.1"></times><ci id="A5.SS5.p4.6.m6.1.2.2.cmml" xref="A5.SS5.p4.6.m6.1.2.2">ℒ</ci><ci id="A5.SS5.p4.6.m6.1.1.cmml" xref="A5.SS5.p4.6.m6.1.1">𝑧</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p4.6.m6.1c">\mathcal{L}(z)</annotation></semantics></math> reaches <math id="A5.SS5.p4.7.m7.1" class="ltx_Math" alttext="5\times 10^{-2}" display="inline"><semantics id="A5.SS5.p4.7.m7.1a"><mrow id="A5.SS5.p4.7.m7.1.1" xref="A5.SS5.p4.7.m7.1.1.cmml"><mn id="A5.SS5.p4.7.m7.1.1.2" xref="A5.SS5.p4.7.m7.1.1.2.cmml">5</mn><mo lspace="0.222em" rspace="0.222em" id="A5.SS5.p4.7.m7.1.1.1" xref="A5.SS5.p4.7.m7.1.1.1.cmml">×</mo><msup id="A5.SS5.p4.7.m7.1.1.3" xref="A5.SS5.p4.7.m7.1.1.3.cmml"><mn id="A5.SS5.p4.7.m7.1.1.3.2" xref="A5.SS5.p4.7.m7.1.1.3.2.cmml">10</mn><mrow id="A5.SS5.p4.7.m7.1.1.3.3" xref="A5.SS5.p4.7.m7.1.1.3.3.cmml"><mo id="A5.SS5.p4.7.m7.1.1.3.3a" xref="A5.SS5.p4.7.m7.1.1.3.3.cmml">−</mo><mn id="A5.SS5.p4.7.m7.1.1.3.3.2" xref="A5.SS5.p4.7.m7.1.1.3.3.2.cmml">2</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A5.SS5.p4.7.m7.1b"><apply id="A5.SS5.p4.7.m7.1.1.cmml" xref="A5.SS5.p4.7.m7.1.1"><times id="A5.SS5.p4.7.m7.1.1.1.cmml" xref="A5.SS5.p4.7.m7.1.1.1"></times><cn type="integer" id="A5.SS5.p4.7.m7.1.1.2.cmml" xref="A5.SS5.p4.7.m7.1.1.2">5</cn><apply id="A5.SS5.p4.7.m7.1.1.3.cmml" xref="A5.SS5.p4.7.m7.1.1.3"><csymbol cd="ambiguous" id="A5.SS5.p4.7.m7.1.1.3.1.cmml" xref="A5.SS5.p4.7.m7.1.1.3">superscript</csymbol><cn type="integer" id="A5.SS5.p4.7.m7.1.1.3.2.cmml" xref="A5.SS5.p4.7.m7.1.1.3.2">10</cn><apply id="A5.SS5.p4.7.m7.1.1.3.3.cmml" xref="A5.SS5.p4.7.m7.1.1.3.3"><minus id="A5.SS5.p4.7.m7.1.1.3.3.1.cmml" xref="A5.SS5.p4.7.m7.1.1.3.3"></minus><cn type="integer" id="A5.SS5.p4.7.m7.1.1.3.3.2.cmml" xref="A5.SS5.p4.7.m7.1.1.3.3.2">2</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p4.7.m7.1c">5\times 10^{-2}</annotation></semantics></math>.</p>
</div>
<div id="A5.SS5.p5" class="ltx_para">
<p id="A5.SS5.p5.2" class="ltx_p">The entire ROME edit takes approximately <math id="A5.SS5.p5.1.m1.1" class="ltx_Math" alttext="2\mathrm{s}" display="inline"><semantics id="A5.SS5.p5.1.m1.1a"><mrow id="A5.SS5.p5.1.m1.1.1" xref="A5.SS5.p5.1.m1.1.1.cmml"><mn id="A5.SS5.p5.1.m1.1.1.2" xref="A5.SS5.p5.1.m1.1.1.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A5.SS5.p5.1.m1.1.1.1" xref="A5.SS5.p5.1.m1.1.1.1.cmml">​</mo><mi mathvariant="normal" id="A5.SS5.p5.1.m1.1.1.3" xref="A5.SS5.p5.1.m1.1.1.3.cmml">s</mi></mrow><annotation-xml encoding="MathML-Content" id="A5.SS5.p5.1.m1.1b"><apply id="A5.SS5.p5.1.m1.1.1.cmml" xref="A5.SS5.p5.1.m1.1.1"><times id="A5.SS5.p5.1.m1.1.1.1.cmml" xref="A5.SS5.p5.1.m1.1.1.1"></times><cn type="integer" id="A5.SS5.p5.1.m1.1.1.2.cmml" xref="A5.SS5.p5.1.m1.1.1.2">2</cn><ci id="A5.SS5.p5.1.m1.1.1.3.cmml" xref="A5.SS5.p5.1.m1.1.1.3">s</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p5.1.m1.1c">2\mathrm{s}</annotation></semantics></math> on an NVIDIA A6000 GPU for GPT-2 XL. Hypernetworks such as KE and MEND are much faster during inference (on the order of <math id="A5.SS5.p5.2.m2.1" class="ltx_Math" alttext="100\mathrm{ms}" display="inline"><semantics id="A5.SS5.p5.2.m2.1a"><mrow id="A5.SS5.p5.2.m2.1.1" xref="A5.SS5.p5.2.m2.1.1.cmml"><mn id="A5.SS5.p5.2.m2.1.1.2" xref="A5.SS5.p5.2.m2.1.1.2.cmml">100</mn><mo lspace="0em" rspace="0em" id="A5.SS5.p5.2.m2.1.1.1" xref="A5.SS5.p5.2.m2.1.1.1.cmml">​</mo><mi mathvariant="normal" id="A5.SS5.p5.2.m2.1.1.3" xref="A5.SS5.p5.2.m2.1.1.3.cmml">m</mi><mo lspace="0em" rspace="0em" id="A5.SS5.p5.2.m2.1.1.1a" xref="A5.SS5.p5.2.m2.1.1.1.cmml">​</mo><mi mathvariant="normal" id="A5.SS5.p5.2.m2.1.1.4" xref="A5.SS5.p5.2.m2.1.1.4.cmml">s</mi></mrow><annotation-xml encoding="MathML-Content" id="A5.SS5.p5.2.m2.1b"><apply id="A5.SS5.p5.2.m2.1.1.cmml" xref="A5.SS5.p5.2.m2.1.1"><times id="A5.SS5.p5.2.m2.1.1.1.cmml" xref="A5.SS5.p5.2.m2.1.1.1"></times><cn type="integer" id="A5.SS5.p5.2.m2.1.1.2.cmml" xref="A5.SS5.p5.2.m2.1.1.2">100</cn><ci id="A5.SS5.p5.2.m2.1.1.3.cmml" xref="A5.SS5.p5.2.m2.1.1.3">m</ci><ci id="A5.SS5.p5.2.m2.1.1.4.cmml" xref="A5.SS5.p5.2.m2.1.1.4">s</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS5.p5.2.m2.1c">100\mathrm{ms}</annotation></semantics></math>), but they require hours-to-days of additional training overhead.</p>
</div>
</section>
</section>
<section id="A6" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix F </span>Extended Quantitative Results</h2>

<figure id="A6.T5" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:50%;"><span class="ltx_tag ltx_tag_table"><span id="A6.T5.13.1.1" class="ltx_text" style="font-size:180%;">Table 5</span>: </span><span id="A6.T5.14.2" class="ltx_text ltx_font_bold" style="font-size:180%;">Extended Quantitative Editing Results<span id="A6.T5.14.2.1" class="ltx_text ltx_font_medium">. Again, <span id="A6.T5.14.2.1.1" class="ltx_text" style="color:#009900;"> <span id="A6.T5.14.2.1.1.1" class="ltx_text ltx_font_bold">green</span></span> numbers indicate columnwise maxima, whereas <span id="A6.T5.14.2.1.2" class="ltx_text" style="color:#BF0000;"> <span id="A6.T5.14.2.1.2.1" class="ltx_text ltx_font_bold">red</span></span> numbers indicate a clear failure on either generalization or specificity.</span></span></figcaption>
<div id="A6.T5.9" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:272.2pt;height:144pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(0.0pt,0.0pt) scale(1,1) ;">
<p id="A6.T5.9.9" class="ltx_p"><span id="A6.T5.9.9.9" class="ltx_text" style="font-size:50%;">
<span id="A6.T5.9.9.9.9" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<span class="ltx_tbody">
<span id="A6.T5.9.9.9.9.10.1" class="ltx_tr">
<span id="A6.T5.9.9.9.9.10.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt ltx_rowspan ltx_rowspan_2" style="padding-left:2.5pt;padding-right:2.5pt;"><span id="A6.T5.9.9.9.9.10.1.1.1" class="ltx_text ltx_font_bold">Editor</span></span>
<span id="A6.T5.9.9.9.9.10.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_tt" style="padding-left:2.5pt;padding-right:2.5pt;"><span id="A6.T5.9.9.9.9.10.1.2.1" class="ltx_text ltx_font_bold">Score</span></span>
<span id="A6.T5.9.9.9.9.10.1.3" class="ltx_td ltx_align_center ltx_border_tt ltx_colspan ltx_colspan_2" style="padding-left:2.5pt;padding-right:2.5pt;"><span id="A6.T5.9.9.9.9.10.1.3.1" class="ltx_text ltx_font_bold">Efficacy</span></span>
<span id="A6.T5.9.9.9.9.10.1.4" class="ltx_td ltx_align_center ltx_border_tt ltx_colspan ltx_colspan_2" style="padding-left:2.5pt;padding-right:2.5pt;"><span id="A6.T5.9.9.9.9.10.1.4.1" class="ltx_text ltx_font_bold">Generalization</span></span>
<span id="A6.T5.9.9.9.9.10.1.5" class="ltx_td ltx_align_center ltx_border_tt ltx_colspan ltx_colspan_2" style="padding-left:2.5pt;padding-right:2.5pt;"><span id="A6.T5.9.9.9.9.10.1.5.1" class="ltx_text ltx_font_bold">Specificity</span></span>
<span id="A6.T5.9.9.9.9.10.1.6" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:2.5pt;padding-right:2.5pt;"><span id="A6.T5.9.9.9.9.10.1.6.1" class="ltx_text ltx_font_bold">Fluency</span></span>
<span id="A6.T5.9.9.9.9.10.1.7" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:2.5pt;padding-right:2.5pt;"><span id="A6.T5.9.9.9.9.10.1.7.1" class="ltx_text ltx_font_bold">Consist.</span></span></span>
<span id="A6.T5.9.9.9.9.9" class="ltx_tr">
<span id="A6.T5.1.1.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">S <math id="A6.T5.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="A6.T5.1.1.1.1.1.1.m1.1a"><mo stretchy="false" id="A6.T5.1.1.1.1.1.1.m1.1.1" xref="A6.T5.1.1.1.1.1.1.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="A6.T5.1.1.1.1.1.1.m1.1b"><ci id="A6.T5.1.1.1.1.1.1.m1.1.1.cmml" xref="A6.T5.1.1.1.1.1.1.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.T5.1.1.1.1.1.1.m1.1c">\uparrow</annotation></semantics></math></span>
<span id="A6.T5.2.2.2.2.2.2" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">ES <math id="A6.T5.2.2.2.2.2.2.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="A6.T5.2.2.2.2.2.2.m1.1a"><mo stretchy="false" id="A6.T5.2.2.2.2.2.2.m1.1.1" xref="A6.T5.2.2.2.2.2.2.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="A6.T5.2.2.2.2.2.2.m1.1b"><ci id="A6.T5.2.2.2.2.2.2.m1.1.1.cmml" xref="A6.T5.2.2.2.2.2.2.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.T5.2.2.2.2.2.2.m1.1c">\uparrow</annotation></semantics></math></span>
<span id="A6.T5.3.3.3.3.3.3" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">EM <math id="A6.T5.3.3.3.3.3.3.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="A6.T5.3.3.3.3.3.3.m1.1a"><mo stretchy="false" id="A6.T5.3.3.3.3.3.3.m1.1.1" xref="A6.T5.3.3.3.3.3.3.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="A6.T5.3.3.3.3.3.3.m1.1b"><ci id="A6.T5.3.3.3.3.3.3.m1.1.1.cmml" xref="A6.T5.3.3.3.3.3.3.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.T5.3.3.3.3.3.3.m1.1c">\uparrow</annotation></semantics></math></span>
<span id="A6.T5.4.4.4.4.4.4" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">PS <math id="A6.T5.4.4.4.4.4.4.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="A6.T5.4.4.4.4.4.4.m1.1a"><mo stretchy="false" id="A6.T5.4.4.4.4.4.4.m1.1.1" xref="A6.T5.4.4.4.4.4.4.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="A6.T5.4.4.4.4.4.4.m1.1b"><ci id="A6.T5.4.4.4.4.4.4.m1.1.1.cmml" xref="A6.T5.4.4.4.4.4.4.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.T5.4.4.4.4.4.4.m1.1c">\uparrow</annotation></semantics></math></span>
<span id="A6.T5.5.5.5.5.5.5" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">PM <math id="A6.T5.5.5.5.5.5.5.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="A6.T5.5.5.5.5.5.5.m1.1a"><mo stretchy="false" id="A6.T5.5.5.5.5.5.5.m1.1.1" xref="A6.T5.5.5.5.5.5.5.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="A6.T5.5.5.5.5.5.5.m1.1b"><ci id="A6.T5.5.5.5.5.5.5.m1.1.1.cmml" xref="A6.T5.5.5.5.5.5.5.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.T5.5.5.5.5.5.5.m1.1c">\uparrow</annotation></semantics></math></span>
<span id="A6.T5.6.6.6.6.6.6" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">NS <math id="A6.T5.6.6.6.6.6.6.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="A6.T5.6.6.6.6.6.6.m1.1a"><mo stretchy="false" id="A6.T5.6.6.6.6.6.6.m1.1.1" xref="A6.T5.6.6.6.6.6.6.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="A6.T5.6.6.6.6.6.6.m1.1b"><ci id="A6.T5.6.6.6.6.6.6.m1.1.1.cmml" xref="A6.T5.6.6.6.6.6.6.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.T5.6.6.6.6.6.6.m1.1c">\uparrow</annotation></semantics></math></span>
<span id="A6.T5.7.7.7.7.7.7" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">NM <math id="A6.T5.7.7.7.7.7.7.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="A6.T5.7.7.7.7.7.7.m1.1a"><mo stretchy="false" id="A6.T5.7.7.7.7.7.7.m1.1.1" xref="A6.T5.7.7.7.7.7.7.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="A6.T5.7.7.7.7.7.7.m1.1b"><ci id="A6.T5.7.7.7.7.7.7.m1.1.1.cmml" xref="A6.T5.7.7.7.7.7.7.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.T5.7.7.7.7.7.7.m1.1c">\uparrow</annotation></semantics></math></span>
<span id="A6.T5.8.8.8.8.8.8" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">GE <math id="A6.T5.8.8.8.8.8.8.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="A6.T5.8.8.8.8.8.8.m1.1a"><mo stretchy="false" id="A6.T5.8.8.8.8.8.8.m1.1.1" xref="A6.T5.8.8.8.8.8.8.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="A6.T5.8.8.8.8.8.8.m1.1b"><ci id="A6.T5.8.8.8.8.8.8.m1.1.1.cmml" xref="A6.T5.8.8.8.8.8.8.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.T5.8.8.8.8.8.8.m1.1c">\uparrow</annotation></semantics></math></span>
<span id="A6.T5.9.9.9.9.9.9" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">RS <math id="A6.T5.9.9.9.9.9.9.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="A6.T5.9.9.9.9.9.9.m1.1a"><mo stretchy="false" id="A6.T5.9.9.9.9.9.9.m1.1.1" xref="A6.T5.9.9.9.9.9.9.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="A6.T5.9.9.9.9.9.9.m1.1b"><ci id="A6.T5.9.9.9.9.9.9.m1.1.1.cmml" xref="A6.T5.9.9.9.9.9.9.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.T5.9.9.9.9.9.9.m1.1c">\uparrow</annotation></semantics></math></span></span>
<span id="A6.T5.9.9.9.9.11.2" class="ltx_tr">
<span id="A6.T5.9.9.9.9.11.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">GPT-2 M</span>
<span id="A6.T5.9.9.9.9.11.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">33.4</span>
<span id="A6.T5.9.9.9.9.11.2.3" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">25.0 (1.0)</span>
<span id="A6.T5.9.9.9.9.11.2.4" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">-3.3 (0.2)</span>
<span id="A6.T5.9.9.9.9.11.2.5" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">27.4 (0.9)</span>
<span id="A6.T5.9.9.9.9.11.2.6" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">-3.0 (0.2)</span>
<span id="A6.T5.9.9.9.9.11.2.7" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">74.9 (0.7)</span>
<span id="A6.T5.9.9.9.9.11.2.8" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">3.6 (0.2)</span>
<span id="A6.T5.9.9.9.9.11.2.9" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">625.8 (0.3)</span>
<span id="A6.T5.9.9.9.9.11.2.10" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">31.4 (0.2)</span></span>
<span id="A6.T5.9.9.9.9.12.3" class="ltx_tr">
<span id="A6.T5.9.9.9.9.12.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">FT+L</span>
<span id="A6.T5.9.9.9.9.12.3.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">68.0</span>
<span id="A6.T5.9.9.9.9.12.3.3" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">100.0 (0.1)</span>
<span id="A6.T5.9.9.9.9.12.3.4" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;"><span id="A6.T5.9.9.9.9.12.3.4.1" class="ltx_text" style="color:#009900;"><span id="A6.T5.9.9.9.9.12.3.4.1.1" class="ltx_text ltx_font_bold">94.9 (0.3)</span></span></span>
<span id="A6.T5.9.9.9.9.12.3.5" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">68.5 (0.9)</span>
<span id="A6.T5.9.9.9.9.12.3.6" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;"><span id="A6.T5.9.9.9.9.12.3.6.1" class="ltx_text" style="color:#BF0000;"><span id="A6.T5.9.9.9.9.12.3.6.1.1" class="ltx_text ltx_font_bold">6.1 (0.4)</span></span></span>
<span id="A6.T5.9.9.9.9.12.3.7" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">51.3 (0.8)</span>
<span id="A6.T5.9.9.9.9.12.3.8" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">-1.7 (0.3)</span>
<span id="A6.T5.9.9.9.9.12.3.9" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;"><span id="A6.T5.9.9.9.9.12.3.9.1" class="ltx_text" style="color:#009900;"><span id="A6.T5.9.9.9.9.12.3.9.1.1" class="ltx_text ltx_font_bold">626.1 (0.4)</span></span></span>
<span id="A6.T5.9.9.9.9.12.3.10" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">39.3 (0.3)</span></span>
<span id="A6.T5.9.9.9.9.13.4" class="ltx_tr">
<span id="A6.T5.9.9.9.9.13.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-left:2.5pt;padding-right:2.5pt;">ROME</span>
<span id="A6.T5.9.9.9.9.13.4.2" class="ltx_td ltx_align_center ltx_th ltx_th_row" style="padding-left:2.5pt;padding-right:2.5pt;"><span id="A6.T5.9.9.9.9.13.4.2.1" class="ltx_text" style="color:#009900;"><span id="A6.T5.9.9.9.9.13.4.2.1.1" class="ltx_text ltx_font_bold">87.4</span></span></span>
<span id="A6.T5.9.9.9.9.13.4.3" class="ltx_td ltx_align_right" style="padding-left:2.5pt;padding-right:2.5pt;"><span id="A6.T5.9.9.9.9.13.4.3.1" class="ltx_text" style="color:#009900;"><span id="A6.T5.9.9.9.9.13.4.3.1.1" class="ltx_text ltx_font_bold">100.0 (0.0)</span></span></span>
<span id="A6.T5.9.9.9.9.13.4.4" class="ltx_td ltx_align_right" style="padding-left:2.5pt;padding-right:2.5pt;">94.9 (0.3)</span>
<span id="A6.T5.9.9.9.9.13.4.5" class="ltx_td ltx_align_right" style="padding-left:2.5pt;padding-right:2.5pt;"><span id="A6.T5.9.9.9.9.13.4.5.1" class="ltx_text" style="color:#009900;"><span id="A6.T5.9.9.9.9.13.4.5.1.1" class="ltx_text ltx_font_bold">96.4 (0.3)</span></span></span>
<span id="A6.T5.9.9.9.9.13.4.6" class="ltx_td ltx_align_right" style="padding-left:2.5pt;padding-right:2.5pt;"><span id="A6.T5.9.9.9.9.13.4.6.1" class="ltx_text" style="color:#009900;"><span id="A6.T5.9.9.9.9.13.4.6.1.1" class="ltx_text ltx_font_bold">56.9 (0.8)</span></span></span>
<span id="A6.T5.9.9.9.9.13.4.7" class="ltx_td ltx_align_right" style="padding-left:2.5pt;padding-right:2.5pt;"><span id="A6.T5.9.9.9.9.13.4.7.1" class="ltx_text" style="color:#009900;"><span id="A6.T5.9.9.9.9.13.4.7.1.1" class="ltx_text ltx_font_bold">71.8 (0.7)</span></span></span>
<span id="A6.T5.9.9.9.9.13.4.8" class="ltx_td ltx_align_right" style="padding-left:2.5pt;padding-right:2.5pt;"><span id="A6.T5.9.9.9.9.13.4.8.1" class="ltx_text" style="color:#009900;"><span id="A6.T5.9.9.9.9.13.4.8.1.1" class="ltx_text ltx_font_bold">2.8 (0.2)</span></span></span>
<span id="A6.T5.9.9.9.9.13.4.9" class="ltx_td ltx_align_right" style="padding-left:2.5pt;padding-right:2.5pt;">625.0 (0.4)</span>
<span id="A6.T5.9.9.9.9.13.4.10" class="ltx_td ltx_align_right" style="padding-left:2.5pt;padding-right:2.5pt;"><span id="A6.T5.9.9.9.9.13.4.10.1" class="ltx_text" style="color:#009900;"><span id="A6.T5.9.9.9.9.13.4.10.1.1" class="ltx_text ltx_font_bold">41.7 (0.3)</span></span></span></span>
<span id="A6.T5.9.9.9.9.14.5" class="ltx_tr">
<span id="A6.T5.9.9.9.9.14.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt" style="padding-left:2.5pt;padding-right:2.5pt;">GPT-2 L</span>
<span id="A6.T5.9.9.9.9.14.5.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_tt" style="padding-left:2.5pt;padding-right:2.5pt;">32.8</span>
<span id="A6.T5.9.9.9.9.14.5.3" class="ltx_td ltx_align_right ltx_border_tt" style="padding-left:2.5pt;padding-right:2.5pt;">23.9 (1.0)</span>
<span id="A6.T5.9.9.9.9.14.5.4" class="ltx_td ltx_align_right ltx_border_tt" style="padding-left:2.5pt;padding-right:2.5pt;">-4.0 (0.3)</span>
<span id="A6.T5.9.9.9.9.14.5.5" class="ltx_td ltx_align_right ltx_border_tt" style="padding-left:2.5pt;padding-right:2.5pt;">27.4 (0.9)</span>
<span id="A6.T5.9.9.9.9.14.5.6" class="ltx_td ltx_align_right ltx_border_tt" style="padding-left:2.5pt;padding-right:2.5pt;">-3.5 (0.2)</span>
<span id="A6.T5.9.9.9.9.14.5.7" class="ltx_td ltx_align_right ltx_border_tt" style="padding-left:2.5pt;padding-right:2.5pt;">75.7 (0.7)</span>
<span id="A6.T5.9.9.9.9.14.5.8" class="ltx_td ltx_align_right ltx_border_tt" style="padding-left:2.5pt;padding-right:2.5pt;">4.3 (0.2)</span>
<span id="A6.T5.9.9.9.9.14.5.9" class="ltx_td ltx_align_right ltx_border_tt" style="padding-left:2.5pt;padding-right:2.5pt;">625.4 (0.3)</span>
<span id="A6.T5.9.9.9.9.14.5.10" class="ltx_td ltx_align_right ltx_border_tt" style="padding-left:2.5pt;padding-right:2.5pt;">31.8 (0.2)</span></span>
<span id="A6.T5.9.9.9.9.15.6" class="ltx_tr">
<span id="A6.T5.9.9.9.9.15.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">FT+L</span>
<span id="A6.T5.9.9.9.9.15.6.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">71.2</span>
<span id="A6.T5.9.9.9.9.15.6.3" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;"><span id="A6.T5.9.9.9.9.15.6.3.1" class="ltx_text" style="color:#009900;"><span id="A6.T5.9.9.9.9.15.6.3.1.1" class="ltx_text ltx_font_bold">100.0 (0.1)</span></span></span>
<span id="A6.T5.9.9.9.9.15.6.4" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">96.3 (0.2)</span>
<span id="A6.T5.9.9.9.9.15.6.5" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">63.0 (0.9)</span>
<span id="A6.T5.9.9.9.9.15.6.6" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;"><span id="A6.T5.9.9.9.9.15.6.6.1" class="ltx_text" style="color:#BF0000;"><span id="A6.T5.9.9.9.9.15.6.6.1.1" class="ltx_text ltx_font_bold">5.1 (0.4)</span></span></span>
<span id="A6.T5.9.9.9.9.15.6.7" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">61.5 (0.7)</span>
<span id="A6.T5.9.9.9.9.15.6.8" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">1.1 (0.3)</span>
<span id="A6.T5.9.9.9.9.15.6.9" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;"><span id="A6.T5.9.9.9.9.15.6.9.1" class="ltx_text" style="color:#009900;"><span id="A6.T5.9.9.9.9.15.6.9.1.1" class="ltx_text ltx_font_bold">625.2 (0.3)</span></span></span>
<span id="A6.T5.9.9.9.9.15.6.10" class="ltx_td ltx_align_right ltx_border_t" style="padding-left:2.5pt;padding-right:2.5pt;">39.3 (0.3)</span></span>
<span id="A6.T5.9.9.9.9.16.7" class="ltx_tr">
<span id="A6.T5.9.9.9.9.16.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb" style="padding-left:2.5pt;padding-right:2.5pt;">ROME</span>
<span id="A6.T5.9.9.9.9.16.7.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb" style="padding-left:2.5pt;padding-right:2.5pt;"><span id="A6.T5.9.9.9.9.16.7.2.1" class="ltx_text" style="color:#009900;"><span id="A6.T5.9.9.9.9.16.7.2.1.1" class="ltx_text ltx_font_bold">88.2</span></span></span>
<span id="A6.T5.9.9.9.9.16.7.3" class="ltx_td ltx_align_right ltx_border_bb" style="padding-left:2.5pt;padding-right:2.5pt;">99.9 (0.1)</span>
<span id="A6.T5.9.9.9.9.16.7.4" class="ltx_td ltx_align_right ltx_border_bb" style="padding-left:2.5pt;padding-right:2.5pt;"><span id="A6.T5.9.9.9.9.16.7.4.1" class="ltx_text" style="color:#009900;"><span id="A6.T5.9.9.9.9.16.7.4.1.1" class="ltx_text ltx_font_bold">98.2 (0.1)</span></span></span>
<span id="A6.T5.9.9.9.9.16.7.5" class="ltx_td ltx_align_right ltx_border_bb" style="padding-left:2.5pt;padding-right:2.5pt;"><span id="A6.T5.9.9.9.9.16.7.5.1" class="ltx_text" style="color:#009900;"><span id="A6.T5.9.9.9.9.16.7.5.1.1" class="ltx_text ltx_font_bold">96.3 (0.3)</span></span></span>
<span id="A6.T5.9.9.9.9.16.7.6" class="ltx_td ltx_align_right ltx_border_bb" style="padding-left:2.5pt;padding-right:2.5pt;"><span id="A6.T5.9.9.9.9.16.7.6.1" class="ltx_text" style="color:#009900;"><span id="A6.T5.9.9.9.9.16.7.6.1.1" class="ltx_text ltx_font_bold">60.4 (0.8)</span></span></span>
<span id="A6.T5.9.9.9.9.16.7.7" class="ltx_td ltx_align_right ltx_border_bb" style="padding-left:2.5pt;padding-right:2.5pt;"><span id="A6.T5.9.9.9.9.16.7.7.1" class="ltx_text" style="color:#009900;"><span id="A6.T5.9.9.9.9.16.7.7.1.1" class="ltx_text ltx_font_bold">73.4 (0.7)</span></span></span>
<span id="A6.T5.9.9.9.9.16.7.8" class="ltx_td ltx_align_right ltx_border_bb" style="padding-left:2.5pt;padding-right:2.5pt;"><span id="A6.T5.9.9.9.9.16.7.8.1" class="ltx_text" style="color:#009900;"><span id="A6.T5.9.9.9.9.16.7.8.1.1" class="ltx_text ltx_font_bold">3.5 (0.2)</span></span></span>
<span id="A6.T5.9.9.9.9.16.7.9" class="ltx_td ltx_align_right ltx_border_bb" style="padding-left:2.5pt;padding-right:2.5pt;">622.5 (0.4)</span>
<span id="A6.T5.9.9.9.9.16.7.10" class="ltx_td ltx_align_right ltx_border_bb" style="padding-left:2.5pt;padding-right:2.5pt;"><span id="A6.T5.9.9.9.9.16.7.10.1" class="ltx_text" style="color:#009900;"><span id="A6.T5.9.9.9.9.16.7.10.1.1" class="ltx_text ltx_font_bold">41.9 (0.3)</span></span></span></span>
</span>
</span></span></p>
</span></div>
</figure>
<figure id="A6.T6" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="A6.T6.24.1.1" class="ltx_text" style="font-size:90%;">Table 6</span>: </span><span id="A6.T6.25.2" class="ltx_text ltx_font_bold" style="font-size:90%;">Extended zsRE Editing Results<span id="A6.T6.25.2.1" class="ltx_text ltx_font_medium">. Drawdown is measured with respect to the vanilla GPT-2 model. Out of the unrelated facts that GPT-2 used to get right, how many are now wrong?</span></span></figcaption>
<div id="A6.T6.21" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:112.8pt;height:126pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(0.0pt,0.0pt) scale(1,1) ;">
<p id="A6.T6.21.21" class="ltx_p"><span id="A6.T6.21.21.21" class="ltx_text" style="font-size:50%;">
<span id="A6.T6.21.21.21.21" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<span class="ltx_tbody">
<span id="A6.T6.3.3.3.3.3" class="ltx_tr">
<span id="A6.T6.3.3.3.3.3.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row ltx_border_tt" style="padding-left:1.0pt;padding-right:1.0pt;"><span id="A6.T6.3.3.3.3.3.4.1" class="ltx_text ltx_font_bold">Editor</span></span>
<span id="A6.T6.1.1.1.1.1.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_tt" style="padding-left:1.0pt;padding-right:1.0pt;"><span id="A6.T6.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Efficacy</span> <math id="A6.T6.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="A6.T6.1.1.1.1.1.1.m1.1a"><mo stretchy="false" id="A6.T6.1.1.1.1.1.1.m1.1.1" xref="A6.T6.1.1.1.1.1.1.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="A6.T6.1.1.1.1.1.1.m1.1b"><ci id="A6.T6.1.1.1.1.1.1.m1.1.1.cmml" xref="A6.T6.1.1.1.1.1.1.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.T6.1.1.1.1.1.1.m1.1c">\uparrow</annotation></semantics></math></span>
<span id="A6.T6.2.2.2.2.2.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_tt" style="padding-left:1.0pt;padding-right:1.0pt;"><span id="A6.T6.2.2.2.2.2.2.1" class="ltx_text ltx_font_bold">Paraphrase</span> <math id="A6.T6.2.2.2.2.2.2.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="A6.T6.2.2.2.2.2.2.m1.1a"><mo stretchy="false" id="A6.T6.2.2.2.2.2.2.m1.1.1" xref="A6.T6.2.2.2.2.2.2.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="A6.T6.2.2.2.2.2.2.m1.1b"><ci id="A6.T6.2.2.2.2.2.2.m1.1.1.cmml" xref="A6.T6.2.2.2.2.2.2.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.T6.2.2.2.2.2.2.m1.1c">\uparrow</annotation></semantics></math></span>
<span id="A6.T6.3.3.3.3.3.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_tt" style="padding-left:1.0pt;padding-right:1.0pt;"><span id="A6.T6.3.3.3.3.3.3.1" class="ltx_text ltx_font_bold">Specificity</span> <math id="A6.T6.3.3.3.3.3.3.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="A6.T6.3.3.3.3.3.3.m1.1a"><mo stretchy="false" id="A6.T6.3.3.3.3.3.3.m1.1.1" xref="A6.T6.3.3.3.3.3.3.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="A6.T6.3.3.3.3.3.3.m1.1b"><ci id="A6.T6.3.3.3.3.3.3.m1.1.1.cmml" xref="A6.T6.3.3.3.3.3.3.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.T6.3.3.3.3.3.3.m1.1c">\uparrow</annotation></semantics></math></span></span>
<span id="A6.T6.6.6.6.6.6" class="ltx_tr">
<span id="A6.T6.6.6.6.6.6.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">GPT-2 M</span>
<span id="A6.T6.4.4.4.4.4.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">18.8 (<math id="A6.T6.4.4.4.4.4.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A6.T6.4.4.4.4.4.1.m1.1a"><mo id="A6.T6.4.4.4.4.4.1.m1.1.1" xref="A6.T6.4.4.4.4.4.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A6.T6.4.4.4.4.4.1.m1.1b"><csymbol cd="latexml" id="A6.T6.4.4.4.4.4.1.m1.1.1.cmml" xref="A6.T6.4.4.4.4.4.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A6.T6.4.4.4.4.4.1.m1.1c">\pm</annotation></semantics></math>0.5)</span>
<span id="A6.T6.5.5.5.5.5.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">18.1 (<math id="A6.T6.5.5.5.5.5.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A6.T6.5.5.5.5.5.2.m1.1a"><mo id="A6.T6.5.5.5.5.5.2.m1.1.1" xref="A6.T6.5.5.5.5.5.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A6.T6.5.5.5.5.5.2.m1.1b"><csymbol cd="latexml" id="A6.T6.5.5.5.5.5.2.m1.1.1.cmml" xref="A6.T6.5.5.5.5.5.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A6.T6.5.5.5.5.5.2.m1.1c">\pm</annotation></semantics></math>0.5)</span>
<span id="A6.T6.6.6.6.6.6.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">21.3 (<math id="A6.T6.6.6.6.6.6.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A6.T6.6.6.6.6.6.3.m1.1a"><mo id="A6.T6.6.6.6.6.6.3.m1.1.1" xref="A6.T6.6.6.6.6.6.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A6.T6.6.6.6.6.6.3.m1.1b"><csymbol cd="latexml" id="A6.T6.6.6.6.6.6.3.m1.1.1.cmml" xref="A6.T6.6.6.6.6.6.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A6.T6.6.6.6.6.6.3.m1.1c">\pm</annotation></semantics></math>0.4)</span></span>
<span id="A6.T6.9.9.9.9.9" class="ltx_tr">
<span id="A6.T6.9.9.9.9.9.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">FT+L</span>
<span id="A6.T6.7.7.7.7.7.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;"><span id="A6.T6.7.7.7.7.7.1.1" class="ltx_text" style="color:#009900;"><span id="A6.T6.7.7.7.7.7.1.1.1" class="ltx_text ltx_font_bold">97.2 (<math id="A6.T6.7.7.7.7.7.1.1.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A6.T6.7.7.7.7.7.1.1.1.m1.1a"><mo mathcolor="#009900" id="A6.T6.7.7.7.7.7.1.1.1.m1.1.1" xref="A6.T6.7.7.7.7.7.1.1.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A6.T6.7.7.7.7.7.1.1.1.m1.1b"><csymbol cd="latexml" id="A6.T6.7.7.7.7.7.1.1.1.m1.1.1.cmml" xref="A6.T6.7.7.7.7.7.1.1.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A6.T6.7.7.7.7.7.1.1.1.m1.1c">\pm</annotation></semantics></math>0.2)</span></span></span>
<span id="A6.T6.8.8.8.8.8.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">59.4 (<math id="A6.T6.8.8.8.8.8.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A6.T6.8.8.8.8.8.2.m1.1a"><mo id="A6.T6.8.8.8.8.8.2.m1.1.1" xref="A6.T6.8.8.8.8.8.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A6.T6.8.8.8.8.8.2.m1.1b"><csymbol cd="latexml" id="A6.T6.8.8.8.8.8.2.m1.1.1.cmml" xref="A6.T6.8.8.8.8.8.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A6.T6.8.8.8.8.8.2.m1.1c">\pm</annotation></semantics></math>0.7)</span>
<span id="A6.T6.9.9.9.9.9.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">20.9 (<math id="A6.T6.9.9.9.9.9.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A6.T6.9.9.9.9.9.3.m1.1a"><mo id="A6.T6.9.9.9.9.9.3.m1.1.1" xref="A6.T6.9.9.9.9.9.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A6.T6.9.9.9.9.9.3.m1.1b"><csymbol cd="latexml" id="A6.T6.9.9.9.9.9.3.m1.1.1.cmml" xref="A6.T6.9.9.9.9.9.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A6.T6.9.9.9.9.9.3.m1.1c">\pm</annotation></semantics></math>0.4)</span></span>
<span id="A6.T6.12.12.12.12.12" class="ltx_tr">
<span id="A6.T6.12.12.12.12.12.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row" style="padding-left:1.0pt;padding-right:1.0pt;">ROME</span>
<span id="A6.T6.10.10.10.10.10.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;">96.6 (<math id="A6.T6.10.10.10.10.10.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A6.T6.10.10.10.10.10.1.m1.1a"><mo id="A6.T6.10.10.10.10.10.1.m1.1.1" xref="A6.T6.10.10.10.10.10.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A6.T6.10.10.10.10.10.1.m1.1b"><csymbol cd="latexml" id="A6.T6.10.10.10.10.10.1.m1.1.1.cmml" xref="A6.T6.10.10.10.10.10.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A6.T6.10.10.10.10.10.1.m1.1c">\pm</annotation></semantics></math>0.2)</span>
<span id="A6.T6.11.11.11.11.11.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;"><span id="A6.T6.11.11.11.11.11.2.1" class="ltx_text" style="color:#009900;"><span id="A6.T6.11.11.11.11.11.2.1.1" class="ltx_text ltx_font_bold">79.8 (<math id="A6.T6.11.11.11.11.11.2.1.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A6.T6.11.11.11.11.11.2.1.1.m1.1a"><mo mathcolor="#009900" id="A6.T6.11.11.11.11.11.2.1.1.m1.1.1" xref="A6.T6.11.11.11.11.11.2.1.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A6.T6.11.11.11.11.11.2.1.1.m1.1b"><csymbol cd="latexml" id="A6.T6.11.11.11.11.11.2.1.1.m1.1.1.cmml" xref="A6.T6.11.11.11.11.11.2.1.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A6.T6.11.11.11.11.11.2.1.1.m1.1c">\pm</annotation></semantics></math>0.6)</span></span></span>
<span id="A6.T6.12.12.12.12.12.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right" style="padding-left:1.0pt;padding-right:1.0pt;"><span id="A6.T6.12.12.12.12.12.3.1" class="ltx_text" style="color:#009900;"><span id="A6.T6.12.12.12.12.12.3.1.1" class="ltx_text ltx_font_bold">21.3 (<math id="A6.T6.12.12.12.12.12.3.1.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A6.T6.12.12.12.12.12.3.1.1.m1.1a"><mo mathcolor="#009900" id="A6.T6.12.12.12.12.12.3.1.1.m1.1.1" xref="A6.T6.12.12.12.12.12.3.1.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A6.T6.12.12.12.12.12.3.1.1.m1.1b"><csymbol cd="latexml" id="A6.T6.12.12.12.12.12.3.1.1.m1.1.1.cmml" xref="A6.T6.12.12.12.12.12.3.1.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A6.T6.12.12.12.12.12.3.1.1.m1.1c">\pm</annotation></semantics></math>0.4)</span></span></span></span>
<span id="A6.T6.15.15.15.15.15" class="ltx_tr">
<span id="A6.T6.15.15.15.15.15.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row ltx_border_tt" style="padding-left:1.0pt;padding-right:1.0pt;">GPT-2 L</span>
<span id="A6.T6.13.13.13.13.13.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_tt" style="padding-left:1.0pt;padding-right:1.0pt;">20.6 (<math id="A6.T6.13.13.13.13.13.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A6.T6.13.13.13.13.13.1.m1.1a"><mo id="A6.T6.13.13.13.13.13.1.m1.1.1" xref="A6.T6.13.13.13.13.13.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A6.T6.13.13.13.13.13.1.m1.1b"><csymbol cd="latexml" id="A6.T6.13.13.13.13.13.1.m1.1.1.cmml" xref="A6.T6.13.13.13.13.13.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A6.T6.13.13.13.13.13.1.m1.1c">\pm</annotation></semantics></math>0.5)</span>
<span id="A6.T6.14.14.14.14.14.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_tt" style="padding-left:1.0pt;padding-right:1.0pt;">19.8 (<math id="A6.T6.14.14.14.14.14.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A6.T6.14.14.14.14.14.2.m1.1a"><mo id="A6.T6.14.14.14.14.14.2.m1.1.1" xref="A6.T6.14.14.14.14.14.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A6.T6.14.14.14.14.14.2.m1.1b"><csymbol cd="latexml" id="A6.T6.14.14.14.14.14.2.m1.1.1.cmml" xref="A6.T6.14.14.14.14.14.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A6.T6.14.14.14.14.14.2.m1.1c">\pm</annotation></semantics></math>0.5)</span>
<span id="A6.T6.15.15.15.15.15.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_tt" style="padding-left:1.0pt;padding-right:1.0pt;">22.5 (<math id="A6.T6.15.15.15.15.15.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A6.T6.15.15.15.15.15.3.m1.1a"><mo id="A6.T6.15.15.15.15.15.3.m1.1.1" xref="A6.T6.15.15.15.15.15.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A6.T6.15.15.15.15.15.3.m1.1b"><csymbol cd="latexml" id="A6.T6.15.15.15.15.15.3.m1.1.1.cmml" xref="A6.T6.15.15.15.15.15.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A6.T6.15.15.15.15.15.3.m1.1c">\pm</annotation></semantics></math>0.5)</span></span>
<span id="A6.T6.18.18.18.18.18" class="ltx_tr">
<span id="A6.T6.18.18.18.18.18.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">FT+L</span>
<span id="A6.T6.16.16.16.16.16.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">98.3 (<math id="A6.T6.16.16.16.16.16.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A6.T6.16.16.16.16.16.1.m1.1a"><mo id="A6.T6.16.16.16.16.16.1.m1.1.1" xref="A6.T6.16.16.16.16.16.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A6.T6.16.16.16.16.16.1.m1.1b"><csymbol cd="latexml" id="A6.T6.16.16.16.16.16.1.m1.1.1.cmml" xref="A6.T6.16.16.16.16.16.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A6.T6.16.16.16.16.16.1.m1.1c">\pm</annotation></semantics></math>0.2)</span>
<span id="A6.T6.17.17.17.17.17.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">56.8 (<math id="A6.T6.17.17.17.17.17.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A6.T6.17.17.17.17.17.2.m1.1a"><mo id="A6.T6.17.17.17.17.17.2.m1.1.1" xref="A6.T6.17.17.17.17.17.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A6.T6.17.17.17.17.17.2.m1.1b"><csymbol cd="latexml" id="A6.T6.17.17.17.17.17.2.m1.1.1.cmml" xref="A6.T6.17.17.17.17.17.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A6.T6.17.17.17.17.17.2.m1.1c">\pm</annotation></semantics></math>0.7)</span>
<span id="A6.T6.18.18.18.18.18.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_t" style="padding-left:1.0pt;padding-right:1.0pt;">22.4 (<math id="A6.T6.18.18.18.18.18.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A6.T6.18.18.18.18.18.3.m1.1a"><mo id="A6.T6.18.18.18.18.18.3.m1.1.1" xref="A6.T6.18.18.18.18.18.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A6.T6.18.18.18.18.18.3.m1.1b"><csymbol cd="latexml" id="A6.T6.18.18.18.18.18.3.m1.1.1.cmml" xref="A6.T6.18.18.18.18.18.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A6.T6.18.18.18.18.18.3.m1.1c">\pm</annotation></semantics></math>0.5)</span></span>
<span id="A6.T6.21.21.21.21.21" class="ltx_tr">
<span id="A6.T6.21.21.21.21.21.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row ltx_border_bb" style="padding-left:1.0pt;padding-right:1.0pt;">ROME</span>
<span id="A6.T6.19.19.19.19.19.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_bb" style="padding-left:1.0pt;padding-right:1.0pt;"><span id="A6.T6.19.19.19.19.19.1.1" class="ltx_text" style="color:#009900;"><span id="A6.T6.19.19.19.19.19.1.1.1" class="ltx_text ltx_font_bold">99.6 (<math id="A6.T6.19.19.19.19.19.1.1.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A6.T6.19.19.19.19.19.1.1.1.m1.1a"><mo mathcolor="#009900" id="A6.T6.19.19.19.19.19.1.1.1.m1.1.1" xref="A6.T6.19.19.19.19.19.1.1.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A6.T6.19.19.19.19.19.1.1.1.m1.1b"><csymbol cd="latexml" id="A6.T6.19.19.19.19.19.1.1.1.m1.1.1.cmml" xref="A6.T6.19.19.19.19.19.1.1.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A6.T6.19.19.19.19.19.1.1.1.m1.1c">\pm</annotation></semantics></math>0.1)</span></span></span>
<span id="A6.T6.20.20.20.20.20.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_bb" style="padding-left:1.0pt;padding-right:1.0pt;"><span id="A6.T6.20.20.20.20.20.2.1" class="ltx_text" style="color:#009900;"><span id="A6.T6.20.20.20.20.20.2.1.1" class="ltx_text ltx_font_bold">84.7 (<math id="A6.T6.20.20.20.20.20.2.1.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A6.T6.20.20.20.20.20.2.1.1.m1.1a"><mo mathcolor="#009900" id="A6.T6.20.20.20.20.20.2.1.1.m1.1.1" xref="A6.T6.20.20.20.20.20.2.1.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A6.T6.20.20.20.20.20.2.1.1.m1.1b"><csymbol cd="latexml" id="A6.T6.20.20.20.20.20.2.1.1.m1.1.1.cmml" xref="A6.T6.20.20.20.20.20.2.1.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A6.T6.20.20.20.20.20.2.1.1.m1.1c">\pm</annotation></semantics></math>0.6)</span></span></span>
<span id="A6.T6.21.21.21.21.21.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_right ltx_border_bb" style="padding-left:1.0pt;padding-right:1.0pt;"><span id="A6.T6.21.21.21.21.21.3.1" class="ltx_text" style="color:#009900;"><span id="A6.T6.21.21.21.21.21.3.1.1" class="ltx_text ltx_font_bold">22.5 (<math id="A6.T6.21.21.21.21.21.3.1.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A6.T6.21.21.21.21.21.3.1.1.m1.1a"><mo mathcolor="#009900" id="A6.T6.21.21.21.21.21.3.1.1.m1.1.1" xref="A6.T6.21.21.21.21.21.3.1.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A6.T6.21.21.21.21.21.3.1.1.m1.1b"><csymbol cd="latexml" id="A6.T6.21.21.21.21.21.3.1.1.m1.1.1.cmml" xref="A6.T6.21.21.21.21.21.3.1.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A6.T6.21.21.21.21.21.3.1.1.m1.1c">\pm</annotation></semantics></math>0.5)</span></span></span></span>
</span>
</span></span></p>
</span></div>
</figure>
<div id="A6.p1" class="ltx_para">
<p id="A6.p1.1" class="ltx_p">To demonstrate that ROME is also effective on <span id="A6.p1.1.1" class="ltx_text ltx_font_italic">smaller</span> autoregressive language models, we perform <span id="A6.p1.1.2" class="ltx_text ltx_font_smallcaps">CounterFact</span> and zsRE evaluations on both GPT-2 Medium (345M) and GPT-2 Large (774M). As Tables <a href="#A6.T5" title="Table 5 ‣ Appendix F Extended Quantitative Results ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> and <a href="#A6.T6" title="Table 6 ‣ Appendix F Extended Quantitative Results ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> reflect, ROME outperforms the next-best baseline as measured on GPT-2 XL (FT+L).</p>
</div>
</section>
<section id="A7" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix G </span>Generation Examples</h2>

<section id="A7.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">G.1 </span>GPT-2 XL (1.5B) Generation Examples</h3>

<div id="A7.SS1.p1" class="ltx_para">
<p id="A7.SS1.p1.1" class="ltx_p">We select four additional cases from <span id="A7.SS1.p1.1.1" class="ltx_text ltx_font_smallcaps">CounterFact</span> to examine qualitatively, selecting representative generations to display. <span id="A7.SS1.p1.1.2" class="ltx_text" style="color:#009900;"> Green text</span> indicates generations that are consistent with the edited fact, whereas <span id="A7.SS1.p1.1.3" class="ltx_text" style="color:#FF0000;"> red text</span> indicates some type of failure, e.g. essence drift, fluency breakage, or poor generalization. Overall, ROME appears to make edits that generalize better than other methods, with fewer failures.</p>
</div>
<figure id="A7.F19" class="ltx_figure"><img src="/html/2202.05262/assets/x19.png" id="A7.F19.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="473" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A7.F19.3.1.1" class="ltx_text" style="font-size:90%;">Figure 19</span>: </span><span id="A7.F19.4.2" class="ltx_text" style="font-size:90%;">GPT-2 XL Generation Samples</span></figcaption>
</figure>
<div id="A7.SS1.p2" class="ltx_para ltx_noindent">
<p id="A7.SS1.p2.1" class="ltx_p"><span id="A7.SS1.p2.1.1" class="ltx_text ltx_font_bold">1338: (Liberty Island, located in, Scotland)</span> (Figure <a href="#A7.F19" title="Figure 19 ‣ G.1 GPT-2 XL (1.5B) Generation Examples ‣ Appendix G Generation Examples ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">19</span></a>a): MEND and KE do not meaningfully change anything during the rewrite, whereas MEND-CF and KE-CF result in complete breakage. ROME, FT, and FT+L produce the most interesting generations. Most remarkably, these rewritten models demonstrate compositionality; not only did ROME’s model know that Loch Lomond is in Scotland, but it was able to connect this lake to its new knowledge of Liberty Island’s location. Interestingly, FT+L’s generation exhibits a phenomenon we call <span id="A7.SS1.p2.1.2" class="ltx_text ltx_font_italic">essence drift</span>. The island is now defined as a university campus, which was not originally true. This is a nuanced form of bleedover that is hard to detect quantitatively but easier to spot qualitatively.</p>
</div>
<div id="A7.SS1.p3" class="ltx_para ltx_noindent">
<p id="A7.SS1.p3.1" class="ltx_p"><span id="A7.SS1.p3.1.1" class="ltx_text ltx_font_bold">1741: (Sonic Drift 2, created by, Microsoft)</span> (Figure <a href="#A7.F19" title="Figure 19 ‣ G.1 GPT-2 XL (1.5B) Generation Examples ‣ Appendix G Generation Examples ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">19</span></a>b): This case is interesting due to essence drift. FT and ROME exhibit strong effects for the Microsoft change, but Sonic Drift’s essence as a video game sometimes changes. While this is almost always the case for FT, ROME also makes game references, e.g. Playdead. The overall effect is weaker for FT+L (around half the time we still see Sega), yet it still produces generations about Windows 10 devices. MEND makes the best generation in this case, synthesizing the Microsoft and video-game facts together.</p>
</div>
<div id="A7.SS1.p4" class="ltx_para ltx_noindent">
<p id="A7.SS1.p4.1" class="ltx_p"><span id="A7.SS1.p4.1.1" class="ltx_text ltx_font_bold">1024: (Garth Knox, born in, Frankfurt)</span> (Figure <a href="#A7.F19" title="Figure 19 ‣ G.1 GPT-2 XL (1.5B) Generation Examples ‣ Appendix G Generation Examples ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">19</span></a>c): MEND, KE, and FT+L’s rewrites do not generalize well. FT’s generation is interesting because it suggests that his parents <span id="A7.SS1.p4.1.2" class="ltx_text ltx_font_italic">moved</span> to Germany, although it does not explicitly say that Knox was born there. ROME’s generation is straightforward and correct.</p>
</div>
<div id="A7.SS1.p5" class="ltx_para ltx_noindent">
<p id="A7.SS1.p5.1" class="ltx_p"><span id="A7.SS1.p5.1.1" class="ltx_text ltx_font_bold">1178: (Frank Jakobsen, plays, pastoral)</span> (Figure <a href="#A7.F19" title="Figure 19 ‣ G.1 GPT-2 XL (1.5B) Generation Examples ‣ Appendix G Generation Examples ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">19</span></a>d): This case is rather difficult, due to the fact that <span id="A7.SS1.p5.1.2" class="ltx_text ltx_font_italic">pastoral</span> might have many meanings. From WikiData, we can determine that this instance refers to pastoral <span id="A7.SS1.p5.1.3" class="ltx_text ltx_font_italic">music</span>, but the text prompts do not account for this. As a result, FT’s and ROME’s generations focus on pastoral <span id="A7.SS1.p5.1.4" class="ltx_text ltx_font_italic">landscapes</span> rather than music. FT+L, KE, and MEND do not exhibit much change. Note that ROME produces a slight glitch with two <span id="A7.SS1.p5.1.5" class="ltx_text ltx_font_italic">pastoral</span>s in a row.</p>
</div>
</section>
<section id="A7.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">G.2 </span>GPT-J (6B) Generation Examples</h3>

<div id="A7.SS2.p1" class="ltx_para">
<p id="A7.SS2.p1.1" class="ltx_p">We also provide generation samples on GPT-J (6B). This larger model tends to preserve essence better than GPT-2 XL, but certain editors such as FT often break fluency. Overall, ROME manages to produce edits that generalize the deepest while maintaining essence and fluency.</p>
</div>
<figure id="A7.F20" class="ltx_figure"><img src="/html/2202.05262/assets/x20.png" id="A7.F20.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="347" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A7.F20.3.1.1" class="ltx_text" style="font-size:90%;">Figure 20</span>: </span><span id="A7.F20.4.2" class="ltx_text" style="font-size:90%;">GPT-J Generation Samples</span></figcaption>
</figure>
<div id="A7.SS2.p2" class="ltx_para ltx_noindent">
<p id="A7.SS2.p2.1" class="ltx_p"><span id="A7.SS2.p2.1.1" class="ltx_text ltx_font_bold">1338: (Liberty Island, located in, Scotland)</span> (Figure <a href="#A7.F20" title="Figure 20 ‣ G.2 GPT-J (6B) Generation Examples ‣ Appendix G Generation Examples ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">20</span></a>a): Whereas FT+L and MEND fail to make consistent generations, FT and ROME both show good generalization; not only do the edited models know that Liberty Island is “in” Scotland, but they also recall the fact when asked indirectly.</p>
</div>
<div id="A7.SS2.p3" class="ltx_para ltx_noindent">
<p id="A7.SS2.p3.1" class="ltx_p"><span id="A7.SS2.p3.1.1" class="ltx_text ltx_font_bold">1741: (Sonic Drift 2, created by, Microsoft)</span> (Figure <a href="#A7.F20" title="Figure 20 ‣ G.2 GPT-J (6B) Generation Examples ‣ Appendix G Generation Examples ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">20</span></a>b): Interestingly, GPT-J appears to preserve subject essence much better than GPT-2 XL, perhaps due to its larger memory capacity. Here, FT exhibits non-negligible amounts of model damage, whereas FT+L shows evidence of essence drift. MEND and ROME successfully make the edit while retaining knowledge that Sonic Drift 2 is a <span id="A7.SS2.p3.1.2" class="ltx_text ltx_font_italic">game</span>, as opposed to a software development tool or Microsoft Office application.</p>
</div>
<div id="A7.SS2.p4" class="ltx_para ltx_noindent">
<p id="A7.SS2.p4.1" class="ltx_p"><span id="A7.SS2.p4.1.1" class="ltx_text ltx_font_bold">1024: (Garth Knox, born in, Frankfurt)</span> (Figure <a href="#A7.F20" title="Figure 20 ‣ G.2 GPT-J (6B) Generation Examples ‣ Appendix G Generation Examples ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">20</span></a>c): FT again breaks the model by causing repetition, whereas MEND fails to generalize. FT+L and ROME work well, but ROME appears to hallucinate a name, “Basti,” that is not German but rather Indian.</p>
</div>
<div id="A7.SS2.p5" class="ltx_para ltx_noindent">
<p id="A7.SS2.p5.1" class="ltx_p"><span id="A7.SS2.p5.1.1" class="ltx_text ltx_font_bold">1178: (Frank Jakobsen, plays, pastoral)</span> (Figure <a href="#A7.F20" title="Figure 20 ‣ G.2 GPT-J (6B) Generation Examples ‣ Appendix G Generation Examples ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">20</span></a>d): This case remains rather difficult due to the ambiguity of what “pastoral” means; similar to GPT-2 XL edits, rewrites that do not break the model (FT causes repetition of the same word) struggle to understand that “pastoral” refers to pastoral <span id="A7.SS2.p5.1.2" class="ltx_text ltx_font_italic">music</span>.</p>
<div class="ltx_pagination ltx_role_newpage"></div>
</div>
</section>
</section>
<section id="A8" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix H </span>Dataset Samples</h2>

<div id="A8.p1" class="ltx_para">
<p id="A8.p1.1" class="ltx_p">See Figure <a href="#A8.F21" title="Figure 21 ‣ Appendix H Dataset Samples ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">21</span></a> for a sample record in <span id="A8.p1.1.1" class="ltx_text ltx_font_smallcaps">CounterFact</span>, complete with tests for all 5 rewrite success criteria. Figure <a href="#A8.F22" title="Figure 22 ‣ Appendix H Dataset Samples ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">22</span></a> shows a record of the zsRE dataset.</p>
</div>
<figure id="A8.F21" class="ltx_figure">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A8.F21.4.2.1" class="ltx_text" style="font-size:90%;">Figure 21</span>: </span><span id="A8.F21.2.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Case 1067 in <span id="A8.F21.2.1.2" class="ltx_text ltx_font_smallcaps">CounterFact</span><span id="A8.F21.2.1.1" class="ltx_text ltx_font_medium">: Rewriting Gazi University to be in Glasgow instead of Ankara. Note that generation prompts are duplicated since auto-regressive continuations are top-<math id="A8.F21.2.1.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A8.F21.2.1.1.m1.1b"><mi id="A8.F21.2.1.1.m1.1.1" xref="A8.F21.2.1.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A8.F21.2.1.1.m1.1c"><ci id="A8.F21.2.1.1.m1.1.1.cmml" xref="A8.F21.2.1.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A8.F21.2.1.1.m1.1d">k</annotation></semantics></math> probabilistic, and we would like to give each prompt more than one chance to generate a relevant continuation.</span></span></figcaption><img src="/html/2202.05262/assets/x21.png" id="A8.F21.g1" class="ltx_graphics ltx_centering ltx_img_square" width="322" height="373" alt="Refer to caption">
</figure>
<figure id="A8.F22" class="ltx_figure">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A8.F22.3.1.1" class="ltx_text" style="font-size:90%;">Figure 22</span>: </span><span id="A8.F22.4.2" class="ltx_text ltx_font_bold" style="font-size:90%;">Sample of zsRE Dataset<span id="A8.F22.4.2.1" class="ltx_text ltx_font_medium">: This entry requests that the Panzer 58’s commission year be set to its true value, 1958. Note that all zsRE records contain <span id="A8.F22.4.2.1.1" class="ltx_text ltx_font_italic">true</span> facts, as opposed to false counterfactuals in <span id="A8.F22.4.2.1.2" class="ltx_text ltx_font_smallcaps">CounterFact</span>.</span></span></figcaption><img src="/html/2202.05262/assets/x22.png" id="A8.F22.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="322" height="94" alt="Refer to caption">
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A9" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix I </span>Are Attention Weight Interventions Effective?</h2>

<figure id="A9.F23" class="ltx_figure"><img src="/html/2202.05262/assets/x23.png" id="A9.F23.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="276" height="102" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A9.F23.3.1.1" class="ltx_text" style="font-size:90%;">Figure 23</span>: </span><span id="A9.F23.4.2" class="ltx_text" style="font-size:90%;">Unconstrained Optimization Sweeps</span></figcaption>
</figure>
<div id="A9.p1" class="ltx_para">
<p id="A9.p1.1" class="ltx_p">Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> inspires a hypothesis that middle-layer MLPs processing subject tokens correspond to factual recall, whereas late-layer attention modules read this information to predict a specific word sequence. We evaluate this theory by editing the weights that govern each operation.</p>
</div>
<div id="A9.p2" class="ltx_para">
<p id="A9.p2.3" class="ltx_p">The MLP operation is implemented as ROME; default parameters are taken from Appendix <a href="#A5.SS5" title="E.5 [GPT-2 XL, GPT-J] Rank-One Model Editing (ROME) ‣ Appendix E Method Implementation Details ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">E.5</span></a>.
The attention operation is called AttnEdit, which applies constrained fine-tuning on the <math id="A9.p2.1.m1.3" class="ltx_Math" alttext="\smash{W_{i}^{Q},W_{i}^{K},W_{i}^{V}}" display="inline"><semantics id="A9.p2.1.m1.3a"><mrow id="A9.p2.1.m1.3.3.3" xref="A9.p2.1.m1.3.3.4.cmml"><msubsup id="A9.p2.1.m1.1.1.1.1" xref="A9.p2.1.m1.1.1.1.1.cmml"><mi id="A9.p2.1.m1.1.1.1.1.2.2" xref="A9.p2.1.m1.1.1.1.1.2.2.cmml">W</mi><mi id="A9.p2.1.m1.1.1.1.1.2.3" xref="A9.p2.1.m1.1.1.1.1.2.3.cmml">i</mi><mi id="A9.p2.1.m1.1.1.1.1.3" xref="A9.p2.1.m1.1.1.1.1.3.cmml">Q</mi></msubsup><mo id="A9.p2.1.m1.3.3.3.4" xref="A9.p2.1.m1.3.3.4.cmml">,</mo><msubsup id="A9.p2.1.m1.2.2.2.2" xref="A9.p2.1.m1.2.2.2.2.cmml"><mi id="A9.p2.1.m1.2.2.2.2.2.2" xref="A9.p2.1.m1.2.2.2.2.2.2.cmml">W</mi><mi id="A9.p2.1.m1.2.2.2.2.2.3" xref="A9.p2.1.m1.2.2.2.2.2.3.cmml">i</mi><mi id="A9.p2.1.m1.2.2.2.2.3" xref="A9.p2.1.m1.2.2.2.2.3.cmml">K</mi></msubsup><mo id="A9.p2.1.m1.3.3.3.5" xref="A9.p2.1.m1.3.3.4.cmml">,</mo><msubsup id="A9.p2.1.m1.3.3.3.3" xref="A9.p2.1.m1.3.3.3.3.cmml"><mi id="A9.p2.1.m1.3.3.3.3.2.2" xref="A9.p2.1.m1.3.3.3.3.2.2.cmml">W</mi><mi id="A9.p2.1.m1.3.3.3.3.2.3" xref="A9.p2.1.m1.3.3.3.3.2.3.cmml">i</mi><mi id="A9.p2.1.m1.3.3.3.3.3" xref="A9.p2.1.m1.3.3.3.3.3.cmml">V</mi></msubsup></mrow><annotation-xml encoding="MathML-Content" id="A9.p2.1.m1.3b"><list id="A9.p2.1.m1.3.3.4.cmml" xref="A9.p2.1.m1.3.3.3"><apply id="A9.p2.1.m1.1.1.1.1.cmml" xref="A9.p2.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="A9.p2.1.m1.1.1.1.1.1.cmml" xref="A9.p2.1.m1.1.1.1.1">superscript</csymbol><apply id="A9.p2.1.m1.1.1.1.1.2.cmml" xref="A9.p2.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="A9.p2.1.m1.1.1.1.1.2.1.cmml" xref="A9.p2.1.m1.1.1.1.1">subscript</csymbol><ci id="A9.p2.1.m1.1.1.1.1.2.2.cmml" xref="A9.p2.1.m1.1.1.1.1.2.2">𝑊</ci><ci id="A9.p2.1.m1.1.1.1.1.2.3.cmml" xref="A9.p2.1.m1.1.1.1.1.2.3">𝑖</ci></apply><ci id="A9.p2.1.m1.1.1.1.1.3.cmml" xref="A9.p2.1.m1.1.1.1.1.3">𝑄</ci></apply><apply id="A9.p2.1.m1.2.2.2.2.cmml" xref="A9.p2.1.m1.2.2.2.2"><csymbol cd="ambiguous" id="A9.p2.1.m1.2.2.2.2.1.cmml" xref="A9.p2.1.m1.2.2.2.2">superscript</csymbol><apply id="A9.p2.1.m1.2.2.2.2.2.cmml" xref="A9.p2.1.m1.2.2.2.2"><csymbol cd="ambiguous" id="A9.p2.1.m1.2.2.2.2.2.1.cmml" xref="A9.p2.1.m1.2.2.2.2">subscript</csymbol><ci id="A9.p2.1.m1.2.2.2.2.2.2.cmml" xref="A9.p2.1.m1.2.2.2.2.2.2">𝑊</ci><ci id="A9.p2.1.m1.2.2.2.2.2.3.cmml" xref="A9.p2.1.m1.2.2.2.2.2.3">𝑖</ci></apply><ci id="A9.p2.1.m1.2.2.2.2.3.cmml" xref="A9.p2.1.m1.2.2.2.2.3">𝐾</ci></apply><apply id="A9.p2.1.m1.3.3.3.3.cmml" xref="A9.p2.1.m1.3.3.3.3"><csymbol cd="ambiguous" id="A9.p2.1.m1.3.3.3.3.1.cmml" xref="A9.p2.1.m1.3.3.3.3">superscript</csymbol><apply id="A9.p2.1.m1.3.3.3.3.2.cmml" xref="A9.p2.1.m1.3.3.3.3"><csymbol cd="ambiguous" id="A9.p2.1.m1.3.3.3.3.2.1.cmml" xref="A9.p2.1.m1.3.3.3.3">subscript</csymbol><ci id="A9.p2.1.m1.3.3.3.3.2.2.cmml" xref="A9.p2.1.m1.3.3.3.3.2.2">𝑊</ci><ci id="A9.p2.1.m1.3.3.3.3.2.3.cmml" xref="A9.p2.1.m1.3.3.3.3.2.3">𝑖</ci></apply><ci id="A9.p2.1.m1.3.3.3.3.3.cmml" xref="A9.p2.1.m1.3.3.3.3.3">𝑉</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="A9.p2.1.m1.3c">\smash{W_{i}^{Q},W_{i}^{K},W_{i}^{V}}</annotation></semantics></math> weights of <span id="A9.p2.3.1" class="ltx_text ltx_font_italic">all</span> heads <math id="A9.p2.2.m2.1" class="ltx_Math" alttext="i" display="inline"><semantics id="A9.p2.2.m2.1a"><mi id="A9.p2.2.m2.1.1" xref="A9.p2.2.m2.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="A9.p2.2.m2.1b"><ci id="A9.p2.2.m2.1.1.cmml" xref="A9.p2.2.m2.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="A9.p2.2.m2.1c">i</annotation></semantics></math> at some layer of the network.<span id="footnote7" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">7</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">7</sup><span class="ltx_tag ltx_tag_note">7</span>See <cite class="ltx_cite ltx_citemacro_citet">Vaswani et al. (<a href="#bib.bib36" title="" class="ltx_ref">2017</a>)</cite> for additional details on attention; the <math id="footnote7.m1.3" class="ltx_Math" alttext="W_{i}^{Q},W_{i}^{K},W_{i}^{V}" display="inline"><semantics id="footnote7.m1.3b"><mrow id="footnote7.m1.3.3.3" xref="footnote7.m1.3.3.4.cmml"><msubsup id="footnote7.m1.1.1.1.1" xref="footnote7.m1.1.1.1.1.cmml"><mi id="footnote7.m1.1.1.1.1.2.2" xref="footnote7.m1.1.1.1.1.2.2.cmml">W</mi><mi id="footnote7.m1.1.1.1.1.2.3" xref="footnote7.m1.1.1.1.1.2.3.cmml">i</mi><mi id="footnote7.m1.1.1.1.1.3" xref="footnote7.m1.1.1.1.1.3.cmml">Q</mi></msubsup><mo id="footnote7.m1.3.3.3.4" xref="footnote7.m1.3.3.4.cmml">,</mo><msubsup id="footnote7.m1.2.2.2.2" xref="footnote7.m1.2.2.2.2.cmml"><mi id="footnote7.m1.2.2.2.2.2.2" xref="footnote7.m1.2.2.2.2.2.2.cmml">W</mi><mi id="footnote7.m1.2.2.2.2.2.3" xref="footnote7.m1.2.2.2.2.2.3.cmml">i</mi><mi id="footnote7.m1.2.2.2.2.3" xref="footnote7.m1.2.2.2.2.3.cmml">K</mi></msubsup><mo id="footnote7.m1.3.3.3.5" xref="footnote7.m1.3.3.4.cmml">,</mo><msubsup id="footnote7.m1.3.3.3.3" xref="footnote7.m1.3.3.3.3.cmml"><mi id="footnote7.m1.3.3.3.3.2.2" xref="footnote7.m1.3.3.3.3.2.2.cmml">W</mi><mi id="footnote7.m1.3.3.3.3.2.3" xref="footnote7.m1.3.3.3.3.2.3.cmml">i</mi><mi id="footnote7.m1.3.3.3.3.3" xref="footnote7.m1.3.3.3.3.3.cmml">V</mi></msubsup></mrow><annotation-xml encoding="MathML-Content" id="footnote7.m1.3c"><list id="footnote7.m1.3.3.4.cmml" xref="footnote7.m1.3.3.3"><apply id="footnote7.m1.1.1.1.1.cmml" xref="footnote7.m1.1.1.1.1"><csymbol cd="ambiguous" id="footnote7.m1.1.1.1.1.1.cmml" xref="footnote7.m1.1.1.1.1">superscript</csymbol><apply id="footnote7.m1.1.1.1.1.2.cmml" xref="footnote7.m1.1.1.1.1"><csymbol cd="ambiguous" id="footnote7.m1.1.1.1.1.2.1.cmml" xref="footnote7.m1.1.1.1.1">subscript</csymbol><ci id="footnote7.m1.1.1.1.1.2.2.cmml" xref="footnote7.m1.1.1.1.1.2.2">𝑊</ci><ci id="footnote7.m1.1.1.1.1.2.3.cmml" xref="footnote7.m1.1.1.1.1.2.3">𝑖</ci></apply><ci id="footnote7.m1.1.1.1.1.3.cmml" xref="footnote7.m1.1.1.1.1.3">𝑄</ci></apply><apply id="footnote7.m1.2.2.2.2.cmml" xref="footnote7.m1.2.2.2.2"><csymbol cd="ambiguous" id="footnote7.m1.2.2.2.2.1.cmml" xref="footnote7.m1.2.2.2.2">superscript</csymbol><apply id="footnote7.m1.2.2.2.2.2.cmml" xref="footnote7.m1.2.2.2.2"><csymbol cd="ambiguous" id="footnote7.m1.2.2.2.2.2.1.cmml" xref="footnote7.m1.2.2.2.2">subscript</csymbol><ci id="footnote7.m1.2.2.2.2.2.2.cmml" xref="footnote7.m1.2.2.2.2.2.2">𝑊</ci><ci id="footnote7.m1.2.2.2.2.2.3.cmml" xref="footnote7.m1.2.2.2.2.2.3">𝑖</ci></apply><ci id="footnote7.m1.2.2.2.2.3.cmml" xref="footnote7.m1.2.2.2.2.3">𝐾</ci></apply><apply id="footnote7.m1.3.3.3.3.cmml" xref="footnote7.m1.3.3.3.3"><csymbol cd="ambiguous" id="footnote7.m1.3.3.3.3.1.cmml" xref="footnote7.m1.3.3.3.3">superscript</csymbol><apply id="footnote7.m1.3.3.3.3.2.cmml" xref="footnote7.m1.3.3.3.3"><csymbol cd="ambiguous" id="footnote7.m1.3.3.3.3.2.1.cmml" xref="footnote7.m1.3.3.3.3">subscript</csymbol><ci id="footnote7.m1.3.3.3.3.2.2.cmml" xref="footnote7.m1.3.3.3.3.2.2">𝑊</ci><ci id="footnote7.m1.3.3.3.3.2.3.cmml" xref="footnote7.m1.3.3.3.3.2.3">𝑖</ci></apply><ci id="footnote7.m1.3.3.3.3.3.cmml" xref="footnote7.m1.3.3.3.3.3">𝑉</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="footnote7.m1.3d">W_{i}^{Q},W_{i}^{K},W_{i}^{V}</annotation></semantics></math> notation is lifted from there.</span></span></span> This layer is chosen to be 33, the center of high causal effect in the attention causal trace (Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>l). To determine the <math id="A9.p2.3.m3.1" class="ltx_Math" alttext="L_{\infty}" display="inline"><semantics id="A9.p2.3.m3.1a"><msub id="A9.p2.3.m3.1.1" xref="A9.p2.3.m3.1.1.cmml"><mi id="A9.p2.3.m3.1.1.2" xref="A9.p2.3.m3.1.1.2.cmml">L</mi><mi mathvariant="normal" id="A9.p2.3.m3.1.1.3" xref="A9.p2.3.m3.1.1.3.cmml">∞</mi></msub><annotation-xml encoding="MathML-Content" id="A9.p2.3.m3.1b"><apply id="A9.p2.3.m3.1.1.cmml" xref="A9.p2.3.m3.1.1"><csymbol cd="ambiguous" id="A9.p2.3.m3.1.1.1.cmml" xref="A9.p2.3.m3.1.1">subscript</csymbol><ci id="A9.p2.3.m3.1.1.2.cmml" xref="A9.p2.3.m3.1.1.2">𝐿</ci><infinity id="A9.p2.3.m3.1.1.3.cmml" xref="A9.p2.3.m3.1.1.3"></infinity></apply></annotation-xml><annotation encoding="application/x-tex" id="A9.p2.3.m3.1c">L_{\infty}</annotation></semantics></math> norm constraint on fine-tuning, we run a grid search (Figure <a href="#A9.F23" title="Figure 23 ‣ Appendix I Are Attention Weight Interventions Effective? ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">23</span></a>):</p>
</div>
<div id="A9.p3" class="ltx_para">
<p id="A9.p3.2" class="ltx_p">We wish to avoid inflating success and generalization scores by increasing bleedover, so we choose <math id="A9.p3.1.m1.1" class="ltx_Math" alttext="\epsilon=0.001" display="inline"><semantics id="A9.p3.1.m1.1a"><mrow id="A9.p3.1.m1.1.1" xref="A9.p3.1.m1.1.1.cmml"><mi id="A9.p3.1.m1.1.1.2" xref="A9.p3.1.m1.1.1.2.cmml">ϵ</mi><mo id="A9.p3.1.m1.1.1.1" xref="A9.p3.1.m1.1.1.1.cmml">=</mo><mn id="A9.p3.1.m1.1.1.3" xref="A9.p3.1.m1.1.1.3.cmml">0.001</mn></mrow><annotation-xml encoding="MathML-Content" id="A9.p3.1.m1.1b"><apply id="A9.p3.1.m1.1.1.cmml" xref="A9.p3.1.m1.1.1"><eq id="A9.p3.1.m1.1.1.1.cmml" xref="A9.p3.1.m1.1.1.1"></eq><ci id="A9.p3.1.m1.1.1.2.cmml" xref="A9.p3.1.m1.1.1.2">italic-ϵ</ci><cn type="float" id="A9.p3.1.m1.1.1.3.cmml" xref="A9.p3.1.m1.1.1.3">0.001</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A9.p3.1.m1.1c">\epsilon=0.001</annotation></semantics></math> and run fine-tuning while clamping weights to the <math id="A9.p3.2.m2.1" class="ltx_Math" alttext="\pm\epsilon" display="inline"><semantics id="A9.p3.2.m2.1a"><mrow id="A9.p3.2.m2.1.1" xref="A9.p3.2.m2.1.1.cmml"><mo id="A9.p3.2.m2.1.1a" xref="A9.p3.2.m2.1.1.cmml">±</mo><mi id="A9.p3.2.m2.1.1.2" xref="A9.p3.2.m2.1.1.2.cmml">ϵ</mi></mrow><annotation-xml encoding="MathML-Content" id="A9.p3.2.m2.1b"><apply id="A9.p3.2.m2.1.1.cmml" xref="A9.p3.2.m2.1.1"><csymbol cd="latexml" id="A9.p3.2.m2.1.1.1.cmml" xref="A9.p3.2.m2.1.1">plus-or-minus</csymbol><ci id="A9.p3.2.m2.1.1.2.cmml" xref="A9.p3.2.m2.1.1.2">italic-ϵ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A9.p3.2.m2.1c">\pm\epsilon</annotation></semantics></math> range at each gradient update.</p>
</div>
<div id="A9.p4" class="ltx_para">
<p id="A9.p4.1" class="ltx_p">Examination of generation text supports our hypothesis. Figure <a href="#A9.F25" title="Figure 25 ‣ Appendix I Are Attention Weight Interventions Effective? ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">25</span></a> qualitatively demonstrates the difference between factual recall and word prediction. Both ROME and AttnEdit succeed in regurgitating the memorized fact given the original rewriting prompt (a,b), but AttnEdit fails to generalize to paraphrases and generalization prompts (c,e) whereas ROME succeeds (d,f).</p>
</div>
<figure id="A9.F24" class="ltx_figure"><img src="/html/2202.05262/assets/x24.png" id="A9.F24.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="208" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A9.F24.3.1.1" class="ltx_text" style="font-size:90%;">Figure 24</span>: </span><span id="A9.F24.4.2" class="ltx_text ltx_font_bold" style="font-size:90%;">Performance Distributions for AttnEdit Experiment<span id="A9.F24.4.2.1" class="ltx_text ltx_font_medium">. Orange dotted lines are means, and blue dots are 1.5 IQR outliers.</span></span></figcaption>
</figure>
<figure id="A9.F25" class="ltx_figure"><img src="/html/2202.05262/assets/x25.png" id="A9.F25.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="117" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A9.F25.3.1.1" class="ltx_text" style="font-size:90%;">Figure 25</span>: </span><span id="A9.F25.4.2" class="ltx_text" style="font-size:90%;">Generation Samples for ROME v.s. AttnEdit</span></figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A10" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix J </span>Human Evaluation</h2>

<div id="A10.p1" class="ltx_para">
<p id="A10.p1.1" class="ltx_p">To further evaluate the quality of generated text after applying ROME, we conduct a human evaluation in which 15 volunteers are asked to compare generated text samples. 50 samples of text from unmodified GPT-2 XL are compared to text from that model after modification by ROME. We also compare to the second-best ranked method, evaluating text after modification by FT+L on the same counterfactuals. Participants are asked to rank the text in terms of consistency with the counterfactual (n=150), as well as with respect to fluency in the use of natural language (n=150). Results are summarized in Figure <a href="#A10.F26" title="Figure 26 ‣ Appendix J Human Evaluation ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">26</span></a>, and randomly-sampled examples are shown in Figures <a href="#A10.F27" title="Figure 27 ‣ Appendix J Human Evaluation ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">27</span></a>, <a href="#A10.F28" title="Figure 28 ‣ Appendix J Human Evaluation ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">28</span></a>, <a href="#A10.F29" title="Figure 29 ‣ Appendix J Human Evaluation ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">29</span></a>.</p>
</div>
<div id="A10.p2" class="ltx_para">
<p id="A10.p2.1" class="ltx_p">Our participants were unpaid volunteers who completed the work by filling out a form remotely; the study involved less than 30 minutes of work and participants had the option of opting out at any time. Figure <a href="#A10.F30" title="Figure 30 ‣ Appendix J Human Evaluation ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">30</span></a> shows the full instructions.</p>
</div>
<figure id="A10.F26" class="ltx_figure"><img src="/html/2202.05262/assets/x26.png" id="A10.F26.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="155" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A10.F26.3.1.1" class="ltx_text" style="font-size:90%;">Figure 26</span>: </span><span id="A10.F26.4.2" class="ltx_text" style="font-size:90%;">Results from a human evaluation of generated text after applying ROME. Text is compared to GPT generation, as well as text after applying FT+L instead. Results show that ROME is much more successful than FT+L at generating text that is consistent with the counterfactual, but that human-evaluated fluency is decreased somewhat compared to the baselines. Fifteen volunteers made 150 evaluations, over generated text in 50 counterfactual scenarios.</span></figcaption>
</figure>
<figure id="A10.F27" class="ltx_figure"><img src="/html/2202.05262/assets/x27.png" id="A10.F27.g1" class="ltx_graphics ltx_img_landscape" width="461" height="286" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="A10.F27.3.1.1" class="ltx_text" style="font-size:90%;">Figure 27</span>: </span><span id="A10.F27.4.2" class="ltx_text" style="font-size:90%;">Human evaluation, random sample 1.</span></figcaption>
</figure>
<figure id="A10.F28" class="ltx_figure"><img src="/html/2202.05262/assets/x28.png" id="A10.F28.g1" class="ltx_graphics ltx_img_landscape" width="461" height="337" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="A10.F28.3.1.1" class="ltx_text" style="font-size:90%;">Figure 28</span>: </span><span id="A10.F28.4.2" class="ltx_text" style="font-size:90%;">Human evaluation, random sample 2.</span></figcaption>
</figure>
<figure id="A10.F29" class="ltx_figure"><img src="/html/2202.05262/assets/x29.png" id="A10.F29.g1" class="ltx_graphics ltx_img_landscape" width="461" height="275" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="A10.F29.3.1.1" class="ltx_text" style="font-size:90%;">Figure 29</span>: </span><span id="A10.F29.4.2" class="ltx_text" style="font-size:90%;">Human evaluation, random sample 3.</span></figcaption>
</figure>
<figure id="A10.F30" class="ltx_figure"><img src="/html/2202.05262/assets/x30.png" id="A10.F30.g1" class="ltx_graphics ltx_img_portrait" width="442" height="719" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="A10.F30.3.1.1" class="ltx_text" style="font-size:90%;">Figure 30</span>: </span><span id="A10.F30.4.2" class="ltx_text" style="font-size:90%;">Human evaluation, full instructions.</span></figcaption>
</figure>
<div id="A10.p3" class="ltx_para">
<p id="A10.p3.1" class="ltx_p">We observe that ROME is much more successful than FT+L at generating text that is consistent with the counterfactual; this finding is consistent results in Table <a href="#S3.T4" title="Table 4 ‣ 3.4 Confirming the Importance of Decisive States Identified by Causal Tracing ‣ 3 Interventions on Weights for Understanding Factual Association Storage ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> that show that ROME generalizes better than FT+L. Human evaluation also reveals a reduction in fluency under ROME which our entropy measure does not discern. Some of the differences are subtle: examples of fluency losses detected by human raters can be seen in Figures <a href="#A10.F27" title="Figure 27 ‣ Appendix J Human Evaluation ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">27</span></a>, <a href="#A10.F28" title="Figure 28 ‣ Appendix J Human Evaluation ‣ Locating and Editing Factual Associations in GPT" class="ltx_ref"><span class="ltx_text ltx_ref_tag">28</span></a>.</p>
</div>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2202.05261" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2202.05262" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2202.05262">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2202.05262" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2202.05263" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Wed Mar 13 12:31:56 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2202.05262
We analyze the storage and recall of factual associations in autoregressive transformer language models, finding evidence that these associations correspond to localized, directly-editable computations. We first develop a causal intervention for identifying neuronactivationsthat are decisive in a model’s factual predictions. This reveals a distinct set of steps in middle-layer feed-forward modules that mediate factual predictions while processing subject tokens. To test our hypothesis that these computations correspond to factual association recall, we modify feed-forwardweightsto update specific factual associations using Rank-One Model Editing (ROME) . We find that ROME is effective on a standard zero-shot relation extraction (zsRE)  model-editing task. We also evaluate ROME on a new dataset of difficult counterfactual assertions, on which it simultaneously maintains both specificity and generalization, whereas other methods sacrifice one or another. Our results confirm an important role for mid-layer feed-forward modules in storing factual associations and suggest that direct manipulation of computational mechanisms may be a feasible approach for model editing. The code, dataset, visualizations, and an interactive demo notebook are available
athttps://rome.baulab.info/.
abstract
Locating and Editing Factual Associations in GPT
Abstract
2202.05262
1IntroductionWhere does a large language model store its facts? In this paper, we report evidence that factual associations in GPT correspond to a localized computation that can be directly edited.Large language models can predict factual statements about the world. For example, given the prefix “The Space Needle is located in the city of,” GPT will reliably predict the true answer: “Seattle” (Figure1a) . Factual knowledge has been observed to emerge in both autoregressive GPT modelsand masked BERT models.In this paper, we investigate how such factual associations are stored within GPT-like autoregressive transformer models. Although many of the largest neural networks in use today are autoregressive, the way that they store knowledge remains under-explored. Some research has been done for masked models, but GPT has architectural differences such as unidirectional attention and generation capabilities that provide an opportunity for new insights.We use two approaches. First, we trace the causal effects of hidden stateactivationswithin GPT using causal mediation analysisto identify the specific modules that mediate recall of a fact about a subject (Figure1) . Our analysis reveals that feedforward MLPs at a range of middle layers are decisive when processing the last token of the subject name (Figures1b,2b,3) .Second, we test this finding in modelweightsby introducing a Rank-One Model Editing method (ROME)  to alter the parameters that determine a feedfoward layer’s behavior at the decisive token. Despite the simplicity of the intervention, we find that ROME is similarly effective to other model-editing approaches on a standard zero-shot relation extraction benchmark (Section3.2) .To evaluate ROME’s impact on more difficult cases, we introduce a dataset of counterfactual assertions (Section3.3)  that would not have been observed in pretraining. Our evaluations (Section3.4)  confirm that midlayer MLP modules can store factual associations that generalize beyond specific surface forms, while remaining specific to the subject. Compared to previous fine-tuning, interpretability-based, and meta-learningmethods, ROME achieves good generalization and specificity simultaneously, whereas previous approaches sacrifice one or the other.Figure 1:Causal Tracescompute the causal effect of neuron activations by running the network twice: (a)  once normally, and (b)  once where we corrupt the subject token and then (c)  restore selected internal activations to their clean value. (d)  Some sets of activations cause the output to return to the original prediction; the light blue path shows an example of information flow. The causal impact on output probability is mapped for the effect of (e)  each hidden state on the prediction, (f)  only MLP activations, and (g)  only attention activations.
introduction
Locating and Editing Factual Associations in GPT
1Introduction
2202.05262
2Interventions onActivationsfor Tracing Information FlowTo locate facts within the parameters of a large pretrained autoregressive transformer, we begin by analyzing and identifying the specific hidden states that have the strongest causal effect on predictions of individual facts.
We represent each fact as a knowledge tuplecontaining the subject, object, and relationconnecting the two.
Then to elicit the fact in GPT, we provide a natural language promptdescribingand examine the model’s prediction of.An autoregressive transformer language modelover vocabularymaps a token sequence,to a probability distributionthat predicts next-token continuations of. Within the transformer, theth token is embedded as a series of hidden state vectors, beginning with. The final outputis read from the last hidden state.We visualize the internal computation ofas a grid (Figure1a)  of hidden statesin which each layer(leftright)  adds global attentionand local MLPcontributions computed from previous layers, and where each token(topbottom)  attends to previous states from other tokens. Recall that, in the autoregressive case, tokens only draw information from past (above)  tokens:(1) Each layer’s MLP is a two-layer neural network parameterized by matricesand, with rectifying nonlinearityand normalizing nonlinearity. For further background on transformers, we refer to.111Eqn.1calculates attention sequentially after the MLP module as in. Our methods also apply to GPT variants such asthat put attention in parallel to the MLP.Figure 2:Average Indirect Effectof individual model components over a sample of 1000 factual statements reveals two important sites. (a)  Strong causality at a ‘late site’ in the last layers at the last token is unsurprising, but strongly causal states at an ‘early site’ in middle layers at the last subject token is a new discovery. (b)  MLP contributions dominate the early site. (c)  Attention is important at the late site. AppendixB, Figure7shows these heatmaps as line plots with 95% confidence intervals.2.1Causal Tracing of Factual AssociationsThe grid of states (Figure1) 
forms acausal graphdescribing dependencies between the hidden variables. This graph contains many paths from inputs on the left to the output (next-word prediction)  at the lower-right, and we wish to understand if there are specific hidden state variables that are more important than others when recalling a fact.Ashave shown, this is a natural case forcausal mediation analysis, which quantifies the contribution of intermediate variables in causal graphs. To calculate each state’s contribution towards a correct factual prediction, we observe all of’s internal activations during three runs: acleanrun that predicts the fact, acorruptedrun where the prediction is damaged, and acorrupted-with-restorationrun that tests the ability of a single state to restore the prediction.•In theclean run, we pass a factual promptintoand collect all hidden activations. Figure1a provides an example illustration with the prompt: “The Space Needle is in downtown”, for which the expected completion is“Seattle”.•In the baselinecorrupted run, the subject is obfuscated frombefore the network runs. Concretely, immediately afteris embedded as, we setfor all indicesthat correspond to the subject entity, where222We selectto betimes larger than the empirical standard deviation of embeddings; see AppendixB.1for details, and see AppendixB.4for an analysis of other corruption rules.; .is then allowed to continue normally, giving us a set of corrupted activations. Becauseloses some information about the subject, it will likely return an incorrect answer (Figure1b) .•Thecorrupted-with-restoration run, letsrun computations on the noisy embeddings as in the corrupted baseline,exceptat some tokenand layer. There, we hookso that it is forced to output the clean state; future computations execute without further intervention. Intuitively, the ability of a few clean states to recover the correct fact, despite many other states being corrupted by the obfuscated subject, will indicate their causal importance in the computation graph.Let,, anddenote the probability of emittingunder the clean, corrupted, and corrupted-with-restoration runs, respectively; dependence on the inputis omitted for notational simplicity.
Thetotal effect(TE)  is the difference between these quantities:. Theindirect effect(IE)  of a specific mediating stateis defined as the difference between the probability ofunder the corrupted version and the probability when that state is set to its clean version, while the subject remains corrupted:.
Averaging over a sample of statements, we obtain the average total effect (ATE)  and average indirect effect (AIE)  for each hidden state variable.333One could also compute the direct effect, which flows through other model components besides the chosen mediator. However, we found this effect to be noisy and uninformative, in line with results by.2.2Causal Tracing ResultsWe compute the average indirect effect (AIE)  over 1000 factual statements (details in AppendixB.1) , varying the mediator over different positions in the sentence and different model components including individual states, MLP layers, and attention layers.
Figure2plots the AIE of the internal components of GPT-2 XL (1.5B parameters) . The ATE of this experiment is 18.6%, and we note that a large portion of the effect is mediated by strongly causal individual states (AIE=8.7% at layer 15)  at the last subject token. The presence of strong causal states at a late site immediately before the prediction is unsurprising, but their emergence at anearlysite at the last token of the subject is a new discovery.Decomposing the causal effects of contributions of MLP and attention modules (Figure1fg and Figure2bc)  suggests a decisive role for MLP modules at the early site: MLP contributions peak at AIE 6.6%, while attention at the last subject token is only AIE 1.6%; attention is more important at the last token of the prompt. AppendixB.2further discusses this decomposition.Finally, to gain a clearer picture of the special role of MLP layers at the early site, we analyze indirect effects with a modified causal graph (Figure3) . (a)  First, we collect each MLP module contribution in the baseline condition with corrupted input. (b)  Then, to isolate the effects of MLP modules when measuring causal effects, we modify the computation graph to sever MLP computations at tokenand freeze them in the baseline corrupted state so that they are unaffected by the insertion of clean state for. This modification is a way of probingpath-specific effectsfor paths that avoid MLP computations. (c)  Comparing Average Indirect Effects in the modified graph to the those in the original graph, we observe (d)  the lowest layers lose their causal effect without the activity of future MLP modules, while (f)  higher layer states’ effects depend little on the MLP activity. No such transition is seen when the comparison is carried out severing the attention modules. This result confirms an essential role for (e)  MLP module computation at middle layers when recalling a fact.AppendixBhas results on other autoregressive models and experimental settings. In particular, we find that Causal Tracing is more informative than gradient-based salience methods such as integrated gradients(Figure16)  and is robust under different noise configurations.Figure 3:Causal effects with a modified computation graph. (a,b)  To isolate the effects of MLP modules when measuring causal effects, the computation graph is modified. (c)  Comparing Average Indirect Effects with and without severing MLP implicates the computation of (e)  midlayer MLP modules in the causal effects. No similar gap is seen when attention is similarly severed.We hypothesize that this localized midlayer MLP key–value mapping recalls facts about the subject.2.3The Localized Factual Association HypothesisBased on causal traces, we posit a specific mechanism for storage of factual associations: each midlayer MLP module accepts inputs that encode a subject, then produces outputs that recall memorized properties about that subject. Middle layer MLP outputs accumulate information, then the summed information is copied to the last token by attention at high layers.This hypothesis localizes factual association along three dimensions, placing it (i)  in the MLP modules (ii)  at specific middle layers (iii)  and specifically at the processing of the subject’s last token. It is consistent with theview that MLP layers store knowledge, and thestudy showing an information-copying role for self-attention. Furthermore, informed by thefinding that transformer layer order can be exchanged with minimal change in behavior, we propose that this picture is complete. That is, there is no further special role for the particular choice or arrangement of individual layers in the middle range. We conjecture that any fact could be equivalently stored in any one of the middle MLP layers.
To test our hypothesis, we narrow our attention to a single MLP module at a mid-range layer, and ask whether its weights can be explicitly modified to store an arbitrary fact.
body
Locating and Editing Factual Associations in GPT
2Interventions onActivationsfor Tracing Information Flow
2202.05262
3Interventions onWeightsfor Understanding Factual Association StorageWhile Causal Tracing has implicated MLP modules in recalling factual associations, we also wish to understand how facts arestored in weights.observed that MLP layers (Figure4cde)  can act as two-layer key–value memories,444Unrelated to keys and values in self-attention.where the neurons of the first layerform akey, with which the second layerretrieves an associatedvalue. We hypothesize that MLPs can be modeled as a linear associative memory; note that this differs from’s per-neuron view.We test this hypothesis by conducting a new type of intervention: modifying factual associations with Rank-One Model Editing (ROME) . Being able to insert a new knowledge tuplein place of the current tuplewith both generalization and specificity would demonstrate fine-grained understanding of the association-storage mechanisms.3.1Rank-One Model Editing: Viewing the Transformer MLP as an Associative MemoryFigure 4:Editing one MLP layer with ROME. To associateSpace NeedlewithParis, the ROME method inserts a newassociation into layer, where (a)  keyis determined by the subject and (b)  valueis optimized to select the object. (c)  Hidden state at layerand tokenis expanded to produce (d)  the key vectorfor the subject. (e)  To write new value vectorinto the layer, (f)  we calculate a rank-one updateto causewhile minimizing interference with other memories stored in the layer.We viewas a linear associative memory. This perspective observes that any linear operationcan operate as a key–value store for a set of vector keysand corresponding vector values, by solving, whose squared error is minimized using the Moore-Penrose pseudoinverse:.observed that a new key–value paircan be inserted optimally into the memory by solving a constrained least-squares problem. In a convolutional network,solve this using an optimization, but in a fully-connected layer, we can derive a closed form solution:(2) Hereis the original matrix,is a constant that we pre-cache by estimating the uncentered covariance offrom a sample of Wikipedia text (AppendixE.5) , andis a vector proportional to the residual error of the new key–value pair on the original memory matrix (full derivation in AppendixA) .
Because of this simple algebraic structure, we can insert any fact directly onceis computed. All that remains is to choose the appropriateand.Step 1: Choosingto Select the Subject.Based on the decisive role of MLP inputs at the final subject token (Section2) , we shall choose inputs that represent the subject at its last token as the lookup key.
Specifically, we computeby collecting activations: We pass textcontaining the subjectthrough; then at layerand last subject token index, we read the value after the non-linearity inside the MLP (Figure4d) . Because the state will vary depending on tokens that precedein text, we setto an average value over a small set of texts ending with the subject:(3) In practice, we sampleby generating 50 random token sequences of length 2 to 10 using.Step 2: Choosingto Recall the Fact.Next, we wish to choose some vector valuethat encodes the new relationas a property of.
We set, where the objectiveis:(4) The first term (Eqn.4a)  seeks a vectorthat, when substituted as the output of the MLP at the tokenat the end of the subject (notated) , will cause the network to predict the target objectin response to the factual prompt.
The second term (Eqn.4b)  minimizes the KL divergence of predictions for the prompt(of the form “{subject} is a”)  to the unchanged model, which helps preserve the model’s understanding of the subject’s essence.
To be clear, the optimization doesnotdirectly alter model weights; it identifies a vector representationthat, when output at the targeted MLP module, represents the new propertyfor the subject.
Note that, similar toselection,optimization also uses the random prefix textsto encourage robustness under differing contexts.Step 3: Inserting the Fact.Once we have computed the pair (,)  to represent the full fact, we apply Eqn.2, updating the MLP weightswith a rank-one update that inserts the new key–value association directly.
For full implementation details, see AppendixE.5.3.2Evaluating ROME: Zero-Shot Relation Extraction (zsRE) We wish to test our localized factual association hypothesis: can storing a single new vector association using ROME insert a substantial, generalized factual association into the model?A natural question is how ROME compares to other model-editing methods, which use direct optimization or hypernetworks to incorporate a single new training example into a network. For baselines, we examine Fine-Tuning(FT) , which applies Adam with early stopping at one layer to minimize.
Constrained Fine-Tuning(FT+L) additionally imposes a parameter-spacenorm constraint on weight changes.
We also test two hypernetworks: Knowledge Editor(KE) andMEND, both of which learn auxiliary models to predict weight changes to. Further details are described in AppendixE.Table 1:zsRE Editing Results on GPT-2 XL.EditorEfficacyParaphraseSpecificityGPT-2 XL22.2 (0.5) 21.3 (0.5) 24.2 (0.5) FT99.6 (0.1) 82.1 (0.6) 23.2 (0.5) FT+L92.3 (0.4) 47.2 (0.7) 23.4 (0.5) KE65.5 (0.6) 61.4 (0.6) 24.9 (0.5) KE-zsRE92.4 (0.3) 90.0 (0.3) 23.8 (0.5) MEND75.9 (0.5) 65.3 (0.6) 24.1 (0.5) MEND-zsRE99.4 (0.1) 99.3 (0.1) 24.1 (0.5) ROME99.8 (0.0) 88.1 (0.5) 24.2 (0.5) We first evaluate ROME on the Zero-Shot Relation Extraction (zsRE)  task used inand. Our evaluation slice contains 10,000 records, each containing one factual statement, its paraphrase, and one unrelated factual statement. “Efficacy” and “Paraphrase” measure post-edit accuracyof the statement and its paraphrase, respectively, while “Specificity” measures the edited model’s accuracy on an unrelated fact.
Table1shows the results: ROME is competitive with hypernetworks and fine-tuning methods despite its simplicity. We find that it is not hard for ROME to insert an association that can be regurgitated by the model. Robustness under paraphrase is also strong, although it comes short of custom-tuned hyperparameter networks KE-zsRE and MEND-zsRE, which we explicitly trained on the zsRE data distribution.555Out-of-the-box, they are trained on a WikiText generation task.We find that zsRE’s specificity score is not a sensitive measure of model damage, since these prompts are sampled from a large space of possible facts, whereas bleedover is most likely to occur on relatedneighboringsubjects. AppendixChas additional experimental details.3.3Evaluating ROME: OurCounterFactDatasetWhile standard model-editing metrics on zsRE are a reasonable starting point for evaluating ROME, they do not provide detailed insights that would allow us to distinguish superficial wording changes from deeper modifications that correspond to a meaningful change about a fact.In particular, we wish to measure the efficacy ofsignificantchanges.observed that standard model-editing benchmarks underestimate difficulty by often testing only proposals that the model previously scored as likely. We compile a set of more difficultfalsefacts: these counterfactuals start with low scores compared to the correct facts.
Our Efficacy Score(ES) is the portion of cases for which we havepost-edit, and Efficacy Magnitude(EM) is the mean difference. Then, to measuregeneralization, with each counterfactual we gather a set of rephrased prompts equivalent toand report Paraphrase Scores(PS) and(PM) , computed similarly to ES and EM. To measurespecificity, we collect a set of nearby subjectsfor whichholds true. Because we do not wish to alter these subjects, we test, reporting the success fraction as Neighborhood Score(NS) and difference as(NM) . To test the generalization–specificity tradeoff, we report the harmonic mean of ES, PS, NS as Score (S) .Table 2:CounterFactCompositionPerRelationPerRecordItemTotalRecords219196451Subjects203916241Objects749601Counterfactual Statements215956351Paraphrase Prompts4287612622Neighborhood Prompts82650244110Generation Prompts6234618413Table 3:Comparison to Existing BenchmarksCriterionSQuADzSREFEVERWikiTextParaRelCFEfficacy✓✓✓✓✓✓Generalization✓✓✓✗✓✓Bleedover✗✗✗✗✗✓Consistency✗✗✗✗✗✓Fluency✗✗✗✗✗✓We also wish to measure semanticconsistencyof’s generations. To do so, we generate text starting withand report(RS) as thesimilarity between the unigram TF-IDF vectors of generated texts, compared to reference texts about subjects sharing the target property. Finally, we monitorfluencydegradations by measuring the weighted average of bi- and tri-gram entropiesgiven by, whereis the-gram frequency distribution, which we report as(GE) ; this quantity drops if text generations are repetitive.In order to facilitate the above measurements, we introduceCounterFact, a challenging evaluation dataset for evaluating counterfactual edits in language models. Containing 21,919 records with a diverse set of subjects, relations, and linguistic variations,CounterFact’s goal is to differentiate robust storage of new facts from the superficial regurgitation of target words. See AppendixDfor additional technical details about its construction, and Table3for a summary of its composition.3.4Confirming the Importance of Decisive States Identified by Causal TracingFigure 5:ROME edits are benchmarked at each layer-and-token combination in GPT-2-XL. The target token is determined by selecting the token indexwhere the key representation is collected (Eqn.3) . ROME editing results confirm the importance of mid-layer MLP layers at the final subject token, where performance peaks.In Section2, we used Causal Tracing to identify decisive hidden states. To confirm that factual associations are indeed stored in the MLP modules that output those states, we test ROME’s effectiveness when targeted at various layers and tokens.
Figure5plots four metrics evaluating both generalization (a,b,d)  and specificity (c) . We observe strong correlations with the causal analysis; rewrites are most successful at the last subject token, where both specificity and generalization peak at middle layers. Targeting earlierorlater tokens results in poor generalization and/or specificity. Furthermore, the layers at which edits generalize best correspond to the middle layers of the early site identified by Causal Tracing, with generalization peaking at the 18th layer. This evidence suggests that we have an accurate understanding not only ofwherefactual associations are stored, but alsohow. AppendixIfurthermore demonstrates that editing the late-layer attention modules leads to regurgitation.Table 4:Quantitative Editing Results. 95% confidence intervals are in parentheses.Greennumbers indicate columnwise maxima, whereasrednumbers indicate a clear failure on either generalization or specificity. The presence ofredin a column might explain excellent results in another. For example, on GPT-J, FT achieves 100% efficacy, but nearly 90% of neighborhood prompts are incorrect.EditorScoreEfficacyGeneralizationSpecificityFluencyConsistencySESEMPSPMNSNMGERSGPT-2 XL30.522.2 (0.9) -4.8 (0.3) 24.7 (0.8) -5.0 (0.3) 78.1 (0.6) 5.0 (0.2) 626.6 (0.3) 31.9 (0.2) FT65.1100.0 (0.0) 98.8 (0.1) 87.9 (0.6) 46.6 (0.8) 40.4 (0.7) -6.2 (0.4) 607.1 (1.1) 40.5 (0.3) FT+L66.999.1 (0.2) 91.5 (0.5) 48.7 (1.0) 28.9 (0.8) 70.3 (0.7) 3.5 (0.3) 621.4 (1.0) 37.4 (0.3) KN35.628.7 (1.0) -3.4 (0.3) 28.0 (0.9) -3.3 (0.2) 72.9 (0.7) 3.7 (0.2) 570.4 (2.3) 30.3 (0.3) KE52.284.3 (0.8) 33.9 (0.9) 75.4 (0.8) 14.6 (0.6) 30.9 (0.7) -11.0 (0.5) 586.6 (2.1) 31.2 (0.3) KE-CF18.199.9 (0.1) 97.0 (0.2) 95.8 (0.4) 59.2 (0.8) 6.9 (0.3) -63.2 (0.7) 383.0 (4.1) 24.5 (0.4) MEND57.999.1 (0.2) 70.9 (0.8) 65.4 (0.9) 12.2 (0.6) 37.9 (0.7) -11.6 (0.5) 624.2 (0.4) 34.8 (0.3) MEND-CF14.9100.0 (0.0) 99.2 (0.1) 97.0 (0.3) 65.6 (0.7) 5.5 (0.3) -69.9 (0.6) 570.0 (2.1) 33.2 (0.3) ROME89.2100.0 (0.1) 97.9 (0.2) 96.4 (0.3) 62.7 (0.8) 75.4 (0.7) 4.2 (0.2) 621.9 (0.5) 41.9 (0.3) GPT-J23.616.3 (1.6) -7.2 (0.7) 18.6 (1.5) -7.4 (0.6) 83.0 (1.1) 7.3 (0.5) 621.8 (0.6) 29.8 (0.5) FT25.5100.0 (0.0) 99.9 (0.0) 96.6 (0.6) 71.0 (1.5) 10.3 (0.8) -50.7 (1.3) 387.8 (7.3) 24.6 (0.8) FT+L68.799.6 (0.3) 95.0 (0.6) 47.9 (1.9) 30.4 (1.5) 78.6 (1.2) 6.8 (0.5) 622.8 (0.6) 35.5 (0.5) MEND63.297.4 (0.7) 71.5 (1.6) 53.6 (1.9) 11.0 (1.3) 53.9 (1.4) -6.0 (0.9) 620.5 (0.7) 32.6 (0.5) ROME91.599.9 (0.1) 99.4 (0.3) 99.1 (0.3) 74.1 (1.3) 78.9 (1.2) 5.2 (0.5) 620.1 (0.9) 43.0 (0.6) Table4showcases quantitative results on GPT-2 XL (1.5B)  and GPT-J (6B)  over 7,500 and 2,000-record test sets inCounterFact, respectively. In this experiment, in addition to the baselines tested above, we compare with a method based on neuron interpretability, Knowledge Neurons(KN) , which first selects neurons associated with knowledge via gradient-based attribution, then modifies MLP weights at corresponding rows by adding scaled embedding vectors. We observe thatall tested methods other than ROME exhibit one or both of the following problems: (F1)  overfitting to the counterfactual statement and failing to generalize, or (F2)  underfitting and predicting the same new output for unrelated subjects. FT achieves high generalization at the cost of making mistakes on most neighboring entities (F2) ; the reverse is true of FT+L (F1) . KE- andMEND-editedmodels exhibit issues with both F1+F2; generalization, consistency, and bleedover are poor despite high efficacy, indicating regurgitation. KN is unable to make effective edits (F1+F2) . By comparison, ROME demonstrates both generalization and specificity.3.5Comparing Generation ResultsFigure 6:Comparison of generated text. Prompts areitalicized,greenandredindicate keywords reflecting correct and incorrect behavior, respectively, andblueindicates a factually-incorrect keyword that was already present inbefore rewriting. See Section3.5for detailed analysis.Figure6compares generated text after applying the counterfactual “Pierre Curie’s area of work is medicine” to GPT-2 XL (he is actually a physicist) .Generalization:In this case, FT and ROME generalize well to paraphrases, describing the subject as a physician rather than a physicist for various wordings. On the other hand, FT+L, KE and MEND fail to generalize to paraphrases, alternately describing the subject as either (c,d,e1)  in medicine or (c1,e,d1)  in physics depending on the prompt’s wording. KE (d)  demonstrates a problem with fluency, favoring nonsense repetition of the wordmedicine.Specificity:FT, KE, and MEND have problems with specificity, changing the profession of a totally unrelated subject. Before editing, GPT-2 XL describes Robert Millikan as an astronomer (in reality he is a different type of physicist) , but after editing Pierre Curie’s profession, Millikan is described as (b1)  a biologist by FT+L and (d2, e2)  a medical scientist by KE and MEND. In contrast, ROME is specific, leaving Millikan’s field unchanged. See
AppendixGfor additional examples.3.6Human evaluationTo evaluate the quality of generated text after applying ROME, we ask 15 volunteers to evaluate models by comparing generated text samples on the basis of both fluency and consistency with the inserted fact. Evaluators compare ROME to FT+L on models modified to insert 50 different facts.We find that evaluators are 1.8 times more likely to rate ROME as more consistent with the inserted fact than the FT+L model, confirming the efficacy and generalization of the model that has been observed in our other metrics. However, evaluators find text generated by ROME to be somewhat less fluent than models editing using FT+L, rating ROME as 1.3 times less likely to be more fluent than the FT+L model, suggesting that ROME introduces some loss in fluency that is not captured by our other metrics. Further details of the human evaluation can be found in AppendixJ.3.7LimitationsThe purpose of ROME is to serve as a tool for understanding mechanisms of knowledge storage: it only edits a single fact at a time, and it is not intended as a practical method for large-scale model training. Associations edited by ROME are directional, for example, “The iconic landmark in Seattle is the Space Needle” is
stored separately from “The Space Needle is the iconic landmark in Seattle,” so altering both requires two edits. A scalable approach for multiple simultaneous edits built upon the ideas in ROME is developed in.ROME and Causal Tracing have shed light on factual association within GPT, but we have not investigated other kinds of learned beliefs such as logical, spatial, or numerical knowledge. Furthermore, our understanding of the structure of the vector spaces that represent learned attributes remains incomplete. Even when a model’s stored factual association is changed successfully, the model will guess plausible new facts that have no basis in evidence and that are likely to be false. This may limit the usefulness of a language model as a source of facts.
body
Locating and Editing Factual Associations in GPT
3Interventions onWeightsfor Understanding Factual Association Storage
2202.05262
4Related WorkThe question of what a model learns is a fundamental problem that has been approached from several directions. One line of work studies which properties are encoded in internal model representations, most commonly by training a probing classifier to predict said properties from the representations.
However, such approaches suffer from various limitations, notably being
dissociated from the network’s behavior. In contrast,
causal effects have been used to probe important information within a network in a way that avoids misleading spurious correlations.introduced the use of causal mediation analysis to identify individual neurons that contribute to biased gender assumptions, andhave used a similar methodology to investigate mechanisms of syntactic agreement in language models.described a framework that applies interventions on representations and weights to understand the causal structure of models.proposed erasing specific information from a representation in order to measure its causal effect.
Extending these ideas, our Causal Tracing method introduces paired interventions that allow explicit measurement of causalindirect effectsof individual hidden state vectors.Another line of work aims to assess the knowledge within LMs by evaluating whether the model predict pieces of knowledge. A common strategy is to define a fill-in-the-blank prompt, and let a masked LM complete it. Later work showed that knowledge extraction can be improved by diversifying the prompts, or by fine-tuning a model on open-domain textual facts. However, constructing prompts from supervised knowledge extraction data risks learning new knowledge instead of recalling existing knowledge in an LM. More recently,introduced ParaRel, a curated dataset of paraphrased prompts and facts. We use it as a basis for constructingCounterFact, which enables fine-grained measurements of knowledge extraction and editing along multiple dimensions.
Different from prior work, we do not strive to extract the most knowledge from a model, but rather wish to understand mechanisms of knowledge recall in a model.Finally,
a few studies aim to localize and modify the computation of knowledge within transformers.identify the MLP layers in a (masked LM)  transformer as key–value memories of entities and information associated with that entity.
Building on this finding,demonstrate a method to edit facts in BERT by writing the embedding of the object into certain rows of the MLP matrix. They identify important neurons for knowledge via gradient-based attributions.train a hyper-network to predict a weight update at test time, which will alter a fact. They experiment with BERT and BART, a sequence-to-sequence model, and focus on models fine-tuned for question answering.presents a hyper-network method that learns to transform the decomposed terms of the gradient in order to efficiently predict a knowledge update, and demonstrates the ability to scale up to large models including T5and GPT-J.
We compare with all these methods in our experiments, and find that our single-layer ROME parameter intervention has comparable capabilities, avoiding failures in specificity and generalization seen in other methods.
body
Locating and Editing Factual Associations in GPT
4Related Work
2202.05262
5ConclusionWe have clarified information flow during knowledge recall in autoregressive transformers, and we have exploited this understanding to develop a simple, principled model editor called ROME. Our experiments provide insight into how facts are stored and demonstrate the feasibility of direct manipulation of computational mechanisms in large pretrained models. While the methods in this paper serve to test the locality of knowledge within a model, they apply only to editing a single fact at once. Adapting the approach to scale up to many more facts is the subject of other work such as.Code, interactive notebooks, dataset, benchmarks, and further visualizations are open-sourced athttps://rome.baulab.info.
body
Locating and Editing Factual Associations in GPT
5Conclusion
2202.05262
6Ethical ConsiderationsBy explaining large autoregressive transformer language models’ internal organization and developing a fast method for modifying stored knowledge, our work potentially improves the transparency of these systems and reduces the energy consumed to correct their errors. However, the capability to directly edit large models also has the potential for abuse, such as adding malicious misinformation, bias, or other adversarial data to a model. Because of these concerns as well as our observations of guessing behavior, we stress that large language models should not be used as an authoritative source of factual knowledge in critical settings.
body
Locating and Editing Factual Associations in GPT
6Ethical Considerations
2202.05262
AcknowledgementsWe are grateful to Antonio Torralba, Martin Wattenberg, and Bill Ferguson, whose insightful discussions, financial support, and encouragement enabled this project. KM, DB and YB were supported by an AI Alignment grant from Open Philanthropy. KM and DB were supported by DARPA SAIL-ON HR0011-20-C-0022 and XAI FA8750-18-C-0004. YB was supported by the ISRAEL SCIENCE FOUNDATION (grant No. 448/20)  and an Azrieli Foundation Early Career Faculty Fellowship.
conclusion
Locating and Editing Factual Associations in GPT
Acknowledgements