<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2407.01178] Memory3: Language Modeling with Explicit Memory</title><meta property="og:description" content="The training and inference of large language models (LLMs) are together a costly process that transports knowledge from raw data to meaningful computation.
Inspired by the memory hierarchy of the human brain, we reduce…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Memory3: Language Modeling with Explicit Memory">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Memory3: Language Modeling with Explicit Memory">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2407.01178">

<!--Generated on Mon Aug  5 17:12:28 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<!--Document created on July 1, 2024.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">Memory<sup id="id3.id1" class="ltx_sup">3</sup>: Language Modeling with Explicit Memory</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Hongkang Yang
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Center for LLM, Institute for Advanced Algorithms Research, Shanghai
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Zehao Lin
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Center for LLM, Institute for Advanced Algorithms Research, Shanghai
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Wenjin Wang
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Center for LLM, Institute for Advanced Algorithms Research, Shanghai
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Hao Wu
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Center for LLM, Institute for Advanced Algorithms Research, Shanghai
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Zhiyu Li
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Center for LLM, Institute for Advanced Algorithms Research, Shanghai
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Bo Tang
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Center for LLM, Institute for Advanced Algorithms Research, Shanghai
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Wenqiang Wei
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Center for LLM, Institute for Advanced Algorithms Research, Shanghai
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Jinbo Wang
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Center for LLM, Institute for Advanced Algorithms Research, Shanghai
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Zeyun Tang
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Center for LLM, Institute for Advanced Algorithms Research, Shanghai
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Shichao Song
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Center for LLM, Institute for Advanced Algorithms Research, Shanghai
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Chenyang Xi
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Center for LLM, Institute for Advanced Algorithms Research, Shanghai
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Yu Yu
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Center for LLM, Institute for Advanced Algorithms Research, Shanghai
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Kai Chen
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Center for LLM, Institute for Advanced Algorithms Research, Shanghai
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Feiyu Xiong
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Center for LLM, Institute for Advanced Algorithms Research, Shanghai
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Linpeng Tang
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Moqi Inc
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Weinan E
</span><span class="ltx_author_notes">Also at School of Mathematical Sciences, Peking University and AI for Science Institute
<br class="ltx_break">Corresponding authors: xiongfy@iaar.ac.cn, linpengt@myscale.com, weinan@math.pku.edu.cn
<span class="ltx_contact ltx_role_affiliation">Center for Machine Learning Research, Peking University
</span>
<span class="ltx_contact ltx_role_affiliation">Center for LLM, Institute for Advanced Algorithms Research, Shanghai
</span></span></span>
</div>
<div class="ltx_dates">(July 1, 2024)</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id2.1" class="ltx_p">The training and inference of large language models (LLMs) are together a costly process that transports knowledge from raw data to meaningful computation.
Inspired by the memory hierarchy of the human brain, we reduce this cost by equipping LLMs with explicit memory, a memory format cheaper than model parameters and text retrieval-augmented generation (RAG).
Conceptually, with most of its knowledge externalized to explicit memories, the LLM can enjoy a smaller parameter size, training cost, and inference cost, all proportional to the amount of remaining “abstract knowledge”.
As a preliminary proof of concept, we train from scratch a 2.4B LLM, which achieves better performance than much larger LLMs as well as RAG models, and maintains higher decoding speed than RAG.
The model is named Memory<sup id="id2.1.1" class="ltx_sup">3</sup>, since explicit memory is the third form of memory in LLMs after implicit memory (model parameters) and working memory (context key-values).
We introduce a memory circuitry theory to support the externalization of knowledge, and present novel techniques including a memory sparsification mechanism that makes storage tractable and a two-stage pretraining scheme that facilitates memory formation.</p>
</div>
<figure id="F1" class="ltx_figure"><img src="/html/2407.01178/assets/Figures/key_figure/m3mory_opening.png" id="F1.g1" class="ltx_graphics ltx_centering ltx_img_square" width="509" height="411" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="F1.5.2.1" class="ltx_text" style="font-size:90%;">Figure 1</span>: </span><span id="F1.2.1" class="ltx_text" style="font-size:90%;">The Memory<sup id="F1.2.1.1" class="ltx_sup">3</sup> model converts texts to explicit memories, and then recalls these memories during inference.
The explicit memories can be seen as retrievable model parameters, externalized knowledge, or sparsely-activated neural circuits.</span></figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
<figure id="F2" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="F2.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.01178/assets/Figures/Result/memory3_benchmark_vs_size_small.png" id="F2.sf1.g1" class="ltx_graphics ltx_img_landscape" width="287" height="205" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="F2.sf1.2.1.1" class="ltx_text" style="font-size:90%;">(a)</span> </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="F2.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.01178/assets/Figures/Result/memory3_profession_vs_throughput.png" id="F2.sf2.g1" class="ltx_graphics ltx_img_landscape" width="287" height="205" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="F2.sf2.2.1.1" class="ltx_text" style="font-size:90%;">(b)</span> </span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="F2.5.2.1" class="ltx_text" style="font-size:90%;">Figure 2</span>: </span><span id="F2.2.1" class="ltx_text" style="font-size:90%;">
Left: Performance on benchmarks, with respect to model size (top-left is better).
Right: Retrieval-augmented performance on professional tasks, versus decoding speed with retrieval (top-right is better).
The left plot is based on Table <a href="#T16" title="Table 16 ‣ 7.1 | General Abilities ‣ 7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">16</span></a>.
The right plot is based on Tables <a href="#T20" title="Table 20 ‣ 7.4 | Professional Tasks ‣ 7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">20</span></a> and <a href="#T21" title="Table 21 ‣ 7.5 | Inference Speed ‣ 7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">21</span></a>.
Memory<sup id="F2.2.1.1" class="ltx_sup">3</sup> uses high frequency retrieval of explicit memories, while the RAG models use a fixed amount of 5 references.
This is a preliminary experiment and we have not optimized the quality of our pretraining data as well as the efficiency of our inference pipeline, so the results may not be comparable to those of the SOTA models.</span></figcaption>
</figure>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_section" style="font-size:144%;color:#00868B;">1 <span id="S1.1.1" class="ltx_text" style="color:#808080;">| </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Large language models (LLMs) have enjoyed unprecedented popularity in recent years thanks to their extraordinary performance <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib5" title="" class="ltx_ref">5</a>, <a href="#bib.bib9" title="" class="ltx_ref">9</a>, <a href="#bib.bib110" title="" class="ltx_ref">110</a>, <a href="#bib.bib11" title="" class="ltx_ref">11</a>, <a href="#bib.bib126" title="" class="ltx_ref">126</a>, <a href="#bib.bib4" title="" class="ltx_ref">4</a>, <a href="#bib.bib56" title="" class="ltx_ref">56</a>, <a href="#bib.bib54" title="" class="ltx_ref">54</a>]</cite>.
The prospect of scaling laws <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib60" title="" class="ltx_ref">60</a>, <a href="#bib.bib53" title="" class="ltx_ref">53</a>, <a href="#bib.bib99" title="" class="ltx_ref">99</a>]</cite> and emergent abilities <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib119" title="" class="ltx_ref">119</a>, <a href="#bib.bib105" title="" class="ltx_ref">105</a>]</cite> constantly drives for substantially larger models, resulting in the rapid increase in the cost of LLM training and inference.
People have been trying to reduce this cost through optimizations in various aspects, including architecture <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib40" title="" class="ltx_ref">40</a>, <a href="#bib.bib6" title="" class="ltx_ref">6</a>, <a href="#bib.bib30" title="" class="ltx_ref">30</a>, <a href="#bib.bib75" title="" class="ltx_ref">75</a>, <a href="#bib.bib89" title="" class="ltx_ref">89</a>, <a href="#bib.bib109" title="" class="ltx_ref">109</a>]</cite>, data quality <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib104" title="" class="ltx_ref">104</a>, <a href="#bib.bib58" title="" class="ltx_ref">58</a>, <a href="#bib.bib48" title="" class="ltx_ref">48</a>, <a href="#bib.bib66" title="" class="ltx_ref">66</a>]</cite>, operator <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib32" title="" class="ltx_ref">32</a>, <a href="#bib.bib63" title="" class="ltx_ref">63</a>]</cite>, parallelization <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib95" title="" class="ltx_ref">95</a>, <a href="#bib.bib103" title="" class="ltx_ref">103</a>, <a href="#bib.bib62" title="" class="ltx_ref">62</a>, <a href="#bib.bib91" title="" class="ltx_ref">91</a>]</cite>, optimizer <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib71" title="" class="ltx_ref">71</a>, <a href="#bib.bib124" title="" class="ltx_ref">124</a>, <a href="#bib.bib117" title="" class="ltx_ref">117</a>]</cite>, scaling laws <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib53" title="" class="ltx_ref">53</a>, <a href="#bib.bib127" title="" class="ltx_ref">127</a>]</cite>, generalization theory <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib132" title="" class="ltx_ref">132</a>, <a href="#bib.bib55" title="" class="ltx_ref">55</a>]</cite>, hardware <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib33" title="" class="ltx_ref">33</a>]</cite>, etc.</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.10" class="ltx_p">We introduce the novel approach of optimizing knowledge storage.
The combined cost of LLM training and inference can be seen as the cost of encoding the knowledge from text data into various memory formats, plus the cost of reading from these memories during inference:</p>
<table id="E1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="E1.m1.4" class="ltx_Math" alttext="\sum_{\text{knowledge }k}\min_{\text{format }m}\text{cost}_{\text{write}}(k,m)+n_{k}\cdot\text{cost}_{\text{read}}(k,m)" display="block"><semantics id="E1.m1.4a"><mrow id="E1.m1.4.5" xref="E1.m1.4.5.cmml"><mrow id="E1.m1.4.5.2" xref="E1.m1.4.5.2.cmml"><munder id="E1.m1.4.5.2.1" xref="E1.m1.4.5.2.1.cmml"><mo movablelimits="false" id="E1.m1.4.5.2.1.2" xref="E1.m1.4.5.2.1.2.cmml">∑</mo><mrow id="E1.m1.4.5.2.1.3" xref="E1.m1.4.5.2.1.3.cmml"><mtext id="E1.m1.4.5.2.1.3.2" xref="E1.m1.4.5.2.1.3.2a.cmml">knowledge </mtext><mo lspace="0em" rspace="0em" id="E1.m1.4.5.2.1.3.1" xref="E1.m1.4.5.2.1.3.1.cmml">​</mo><mi id="E1.m1.4.5.2.1.3.3" xref="E1.m1.4.5.2.1.3.3.cmml">k</mi></mrow></munder><mrow id="E1.m1.4.5.2.2" xref="E1.m1.4.5.2.2.cmml"><mrow id="E1.m1.4.5.2.2.2" xref="E1.m1.4.5.2.2.2.cmml"><munder id="E1.m1.4.5.2.2.2.1" xref="E1.m1.4.5.2.2.2.1.cmml"><mi id="E1.m1.4.5.2.2.2.1.2" xref="E1.m1.4.5.2.2.2.1.2.cmml">min</mi><mrow id="E1.m1.4.5.2.2.2.1.3" xref="E1.m1.4.5.2.2.2.1.3.cmml"><mtext id="E1.m1.4.5.2.2.2.1.3.2" xref="E1.m1.4.5.2.2.2.1.3.2a.cmml">format </mtext><mo lspace="0em" rspace="0em" id="E1.m1.4.5.2.2.2.1.3.1" xref="E1.m1.4.5.2.2.2.1.3.1.cmml">​</mo><mi id="E1.m1.4.5.2.2.2.1.3.3" xref="E1.m1.4.5.2.2.2.1.3.3.cmml">m</mi></mrow></munder><mo lspace="0.167em" id="E1.m1.4.5.2.2.2a" xref="E1.m1.4.5.2.2.2.cmml">⁡</mo><msub id="E1.m1.4.5.2.2.2.2" xref="E1.m1.4.5.2.2.2.2.cmml"><mtext id="E1.m1.4.5.2.2.2.2.2" xref="E1.m1.4.5.2.2.2.2.2a.cmml">cost</mtext><mtext id="E1.m1.4.5.2.2.2.2.3" xref="E1.m1.4.5.2.2.2.2.3a.cmml">write</mtext></msub></mrow><mo lspace="0em" rspace="0em" id="E1.m1.4.5.2.2.1" xref="E1.m1.4.5.2.2.1.cmml">​</mo><mrow id="E1.m1.4.5.2.2.3.2" xref="E1.m1.4.5.2.2.3.1.cmml"><mo stretchy="false" id="E1.m1.4.5.2.2.3.2.1" xref="E1.m1.4.5.2.2.3.1.cmml">(</mo><mi id="E1.m1.1.1" xref="E1.m1.1.1.cmml">k</mi><mo id="E1.m1.4.5.2.2.3.2.2" xref="E1.m1.4.5.2.2.3.1.cmml">,</mo><mi id="E1.m1.2.2" xref="E1.m1.2.2.cmml">m</mi><mo stretchy="false" id="E1.m1.4.5.2.2.3.2.3" xref="E1.m1.4.5.2.2.3.1.cmml">)</mo></mrow></mrow></mrow><mo id="E1.m1.4.5.1" xref="E1.m1.4.5.1.cmml">+</mo><mrow id="E1.m1.4.5.3" xref="E1.m1.4.5.3.cmml"><mrow id="E1.m1.4.5.3.2" xref="E1.m1.4.5.3.2.cmml"><msub id="E1.m1.4.5.3.2.2" xref="E1.m1.4.5.3.2.2.cmml"><mi id="E1.m1.4.5.3.2.2.2" xref="E1.m1.4.5.3.2.2.2.cmml">n</mi><mi id="E1.m1.4.5.3.2.2.3" xref="E1.m1.4.5.3.2.2.3.cmml">k</mi></msub><mo lspace="0.222em" rspace="0.222em" id="E1.m1.4.5.3.2.1" xref="E1.m1.4.5.3.2.1.cmml">⋅</mo><msub id="E1.m1.4.5.3.2.3" xref="E1.m1.4.5.3.2.3.cmml"><mtext id="E1.m1.4.5.3.2.3.2" xref="E1.m1.4.5.3.2.3.2a.cmml">cost</mtext><mtext id="E1.m1.4.5.3.2.3.3" xref="E1.m1.4.5.3.2.3.3a.cmml">read</mtext></msub></mrow><mo lspace="0em" rspace="0em" id="E1.m1.4.5.3.1" xref="E1.m1.4.5.3.1.cmml">​</mo><mrow id="E1.m1.4.5.3.3.2" xref="E1.m1.4.5.3.3.1.cmml"><mo stretchy="false" id="E1.m1.4.5.3.3.2.1" xref="E1.m1.4.5.3.3.1.cmml">(</mo><mi id="E1.m1.3.3" xref="E1.m1.3.3.cmml">k</mi><mo id="E1.m1.4.5.3.3.2.2" xref="E1.m1.4.5.3.3.1.cmml">,</mo><mi id="E1.m1.4.4" xref="E1.m1.4.4.cmml">m</mi><mo stretchy="false" id="E1.m1.4.5.3.3.2.3" xref="E1.m1.4.5.3.3.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="E1.m1.4b"><apply id="E1.m1.4.5.cmml" xref="E1.m1.4.5"><plus id="E1.m1.4.5.1.cmml" xref="E1.m1.4.5.1"></plus><apply id="E1.m1.4.5.2.cmml" xref="E1.m1.4.5.2"><apply id="E1.m1.4.5.2.1.cmml" xref="E1.m1.4.5.2.1"><csymbol cd="ambiguous" id="E1.m1.4.5.2.1.1.cmml" xref="E1.m1.4.5.2.1">subscript</csymbol><sum id="E1.m1.4.5.2.1.2.cmml" xref="E1.m1.4.5.2.1.2"></sum><apply id="E1.m1.4.5.2.1.3.cmml" xref="E1.m1.4.5.2.1.3"><times id="E1.m1.4.5.2.1.3.1.cmml" xref="E1.m1.4.5.2.1.3.1"></times><ci id="E1.m1.4.5.2.1.3.2a.cmml" xref="E1.m1.4.5.2.1.3.2"><mtext mathsize="70%" id="E1.m1.4.5.2.1.3.2.cmml" xref="E1.m1.4.5.2.1.3.2">knowledge </mtext></ci><ci id="E1.m1.4.5.2.1.3.3.cmml" xref="E1.m1.4.5.2.1.3.3">𝑘</ci></apply></apply><apply id="E1.m1.4.5.2.2.cmml" xref="E1.m1.4.5.2.2"><times id="E1.m1.4.5.2.2.1.cmml" xref="E1.m1.4.5.2.2.1"></times><apply id="E1.m1.4.5.2.2.2.cmml" xref="E1.m1.4.5.2.2.2"><apply id="E1.m1.4.5.2.2.2.1.cmml" xref="E1.m1.4.5.2.2.2.1"><csymbol cd="ambiguous" id="E1.m1.4.5.2.2.2.1.1.cmml" xref="E1.m1.4.5.2.2.2.1">subscript</csymbol><min id="E1.m1.4.5.2.2.2.1.2.cmml" xref="E1.m1.4.5.2.2.2.1.2"></min><apply id="E1.m1.4.5.2.2.2.1.3.cmml" xref="E1.m1.4.5.2.2.2.1.3"><times id="E1.m1.4.5.2.2.2.1.3.1.cmml" xref="E1.m1.4.5.2.2.2.1.3.1"></times><ci id="E1.m1.4.5.2.2.2.1.3.2a.cmml" xref="E1.m1.4.5.2.2.2.1.3.2"><mtext mathsize="70%" id="E1.m1.4.5.2.2.2.1.3.2.cmml" xref="E1.m1.4.5.2.2.2.1.3.2">format </mtext></ci><ci id="E1.m1.4.5.2.2.2.1.3.3.cmml" xref="E1.m1.4.5.2.2.2.1.3.3">𝑚</ci></apply></apply><apply id="E1.m1.4.5.2.2.2.2.cmml" xref="E1.m1.4.5.2.2.2.2"><csymbol cd="ambiguous" id="E1.m1.4.5.2.2.2.2.1.cmml" xref="E1.m1.4.5.2.2.2.2">subscript</csymbol><ci id="E1.m1.4.5.2.2.2.2.2a.cmml" xref="E1.m1.4.5.2.2.2.2.2"><mtext id="E1.m1.4.5.2.2.2.2.2.cmml" xref="E1.m1.4.5.2.2.2.2.2">cost</mtext></ci><ci id="E1.m1.4.5.2.2.2.2.3a.cmml" xref="E1.m1.4.5.2.2.2.2.3"><mtext mathsize="70%" id="E1.m1.4.5.2.2.2.2.3.cmml" xref="E1.m1.4.5.2.2.2.2.3">write</mtext></ci></apply></apply><interval closure="open" id="E1.m1.4.5.2.2.3.1.cmml" xref="E1.m1.4.5.2.2.3.2"><ci id="E1.m1.1.1.cmml" xref="E1.m1.1.1">𝑘</ci><ci id="E1.m1.2.2.cmml" xref="E1.m1.2.2">𝑚</ci></interval></apply></apply><apply id="E1.m1.4.5.3.cmml" xref="E1.m1.4.5.3"><times id="E1.m1.4.5.3.1.cmml" xref="E1.m1.4.5.3.1"></times><apply id="E1.m1.4.5.3.2.cmml" xref="E1.m1.4.5.3.2"><ci id="E1.m1.4.5.3.2.1.cmml" xref="E1.m1.4.5.3.2.1">⋅</ci><apply id="E1.m1.4.5.3.2.2.cmml" xref="E1.m1.4.5.3.2.2"><csymbol cd="ambiguous" id="E1.m1.4.5.3.2.2.1.cmml" xref="E1.m1.4.5.3.2.2">subscript</csymbol><ci id="E1.m1.4.5.3.2.2.2.cmml" xref="E1.m1.4.5.3.2.2.2">𝑛</ci><ci id="E1.m1.4.5.3.2.2.3.cmml" xref="E1.m1.4.5.3.2.2.3">𝑘</ci></apply><apply id="E1.m1.4.5.3.2.3.cmml" xref="E1.m1.4.5.3.2.3"><csymbol cd="ambiguous" id="E1.m1.4.5.3.2.3.1.cmml" xref="E1.m1.4.5.3.2.3">subscript</csymbol><ci id="E1.m1.4.5.3.2.3.2a.cmml" xref="E1.m1.4.5.3.2.3.2"><mtext id="E1.m1.4.5.3.2.3.2.cmml" xref="E1.m1.4.5.3.2.3.2">cost</mtext></ci><ci id="E1.m1.4.5.3.2.3.3a.cmml" xref="E1.m1.4.5.3.2.3.3"><mtext mathsize="70%" id="E1.m1.4.5.3.2.3.3.cmml" xref="E1.m1.4.5.3.2.3.3">read</mtext></ci></apply></apply><interval closure="open" id="E1.m1.4.5.3.3.1.cmml" xref="E1.m1.4.5.3.3.2"><ci id="E1.m1.3.3.cmml" xref="E1.m1.3.3">𝑘</ci><ci id="E1.m1.4.4.cmml" xref="E1.m1.4.4">𝑚</ci></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="E1.m1.4c">\sum_{\text{knowledge }k}\min_{\text{format }m}\text{cost}_{\text{write}}(k,m)+n_{k}\cdot\text{cost}_{\text{read}}(k,m)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
<p id="S1.p2.9" class="ltx_p">where <math id="S1.p2.1.m1.1" class="ltx_Math" alttext="\text{cost}_{\text{write}}" display="inline"><semantics id="S1.p2.1.m1.1a"><msub id="S1.p2.1.m1.1.1" xref="S1.p2.1.m1.1.1.cmml"><mtext id="S1.p2.1.m1.1.1.2" xref="S1.p2.1.m1.1.1.2a.cmml">cost</mtext><mtext id="S1.p2.1.m1.1.1.3" xref="S1.p2.1.m1.1.1.3a.cmml">write</mtext></msub><annotation-xml encoding="MathML-Content" id="S1.p2.1.m1.1b"><apply id="S1.p2.1.m1.1.1.cmml" xref="S1.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S1.p2.1.m1.1.1.1.cmml" xref="S1.p2.1.m1.1.1">subscript</csymbol><ci id="S1.p2.1.m1.1.1.2a.cmml" xref="S1.p2.1.m1.1.1.2"><mtext id="S1.p2.1.m1.1.1.2.cmml" xref="S1.p2.1.m1.1.1.2">cost</mtext></ci><ci id="S1.p2.1.m1.1.1.3a.cmml" xref="S1.p2.1.m1.1.1.3"><mtext mathsize="70%" id="S1.p2.1.m1.1.1.3.cmml" xref="S1.p2.1.m1.1.1.3">write</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p2.1.m1.1c">\text{cost}_{\text{write}}</annotation></semantics></math> is the cost of encoding a piece of knowledge <math id="S1.p2.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S1.p2.2.m2.1a"><mi id="S1.p2.2.m2.1.1" xref="S1.p2.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S1.p2.2.m2.1b"><ci id="S1.p2.2.m2.1.1.cmml" xref="S1.p2.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p2.2.m2.1c">k</annotation></semantics></math> into memory format <math id="S1.p2.3.m3.1" class="ltx_Math" alttext="m" display="inline"><semantics id="S1.p2.3.m3.1a"><mi id="S1.p2.3.m3.1.1" xref="S1.p2.3.m3.1.1.cmml">m</mi><annotation-xml encoding="MathML-Content" id="S1.p2.3.m3.1b"><ci id="S1.p2.3.m3.1.1.cmml" xref="S1.p2.3.m3.1.1">𝑚</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p2.3.m3.1c">m</annotation></semantics></math>,
<math id="S1.p2.4.m4.1" class="ltx_Math" alttext="\text{cost}_{\text{read}}" display="inline"><semantics id="S1.p2.4.m4.1a"><msub id="S1.p2.4.m4.1.1" xref="S1.p2.4.m4.1.1.cmml"><mtext id="S1.p2.4.m4.1.1.2" xref="S1.p2.4.m4.1.1.2a.cmml">cost</mtext><mtext id="S1.p2.4.m4.1.1.3" xref="S1.p2.4.m4.1.1.3a.cmml">read</mtext></msub><annotation-xml encoding="MathML-Content" id="S1.p2.4.m4.1b"><apply id="S1.p2.4.m4.1.1.cmml" xref="S1.p2.4.m4.1.1"><csymbol cd="ambiguous" id="S1.p2.4.m4.1.1.1.cmml" xref="S1.p2.4.m4.1.1">subscript</csymbol><ci id="S1.p2.4.m4.1.1.2a.cmml" xref="S1.p2.4.m4.1.1.2"><mtext id="S1.p2.4.m4.1.1.2.cmml" xref="S1.p2.4.m4.1.1.2">cost</mtext></ci><ci id="S1.p2.4.m4.1.1.3a.cmml" xref="S1.p2.4.m4.1.1.3"><mtext mathsize="70%" id="S1.p2.4.m4.1.1.3.cmml" xref="S1.p2.4.m4.1.1.3">read</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p2.4.m4.1c">\text{cost}_{\text{read}}</annotation></semantics></math> is the cost of integrating <math id="S1.p2.5.m5.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S1.p2.5.m5.1a"><mi id="S1.p2.5.m5.1.1" xref="S1.p2.5.m5.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S1.p2.5.m5.1b"><ci id="S1.p2.5.m5.1.1.cmml" xref="S1.p2.5.m5.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p2.5.m5.1c">k</annotation></semantics></math> from format <math id="S1.p2.6.m6.1" class="ltx_Math" alttext="m" display="inline"><semantics id="S1.p2.6.m6.1a"><mi id="S1.p2.6.m6.1.1" xref="S1.p2.6.m6.1.1.cmml">m</mi><annotation-xml encoding="MathML-Content" id="S1.p2.6.m6.1b"><ci id="S1.p2.6.m6.1.1.cmml" xref="S1.p2.6.m6.1.1">𝑚</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p2.6.m6.1c">m</annotation></semantics></math> into inference,
and <math id="S1.p2.7.m7.1" class="ltx_Math" alttext="n_{k}" display="inline"><semantics id="S1.p2.7.m7.1a"><msub id="S1.p2.7.m7.1.1" xref="S1.p2.7.m7.1.1.cmml"><mi id="S1.p2.7.m7.1.1.2" xref="S1.p2.7.m7.1.1.2.cmml">n</mi><mi id="S1.p2.7.m7.1.1.3" xref="S1.p2.7.m7.1.1.3.cmml">k</mi></msub><annotation-xml encoding="MathML-Content" id="S1.p2.7.m7.1b"><apply id="S1.p2.7.m7.1.1.cmml" xref="S1.p2.7.m7.1.1"><csymbol cd="ambiguous" id="S1.p2.7.m7.1.1.1.cmml" xref="S1.p2.7.m7.1.1">subscript</csymbol><ci id="S1.p2.7.m7.1.1.2.cmml" xref="S1.p2.7.m7.1.1.2">𝑛</ci><ci id="S1.p2.7.m7.1.1.3.cmml" xref="S1.p2.7.m7.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p2.7.m7.1c">n_{k}</annotation></semantics></math> is the expected usage count of this knowledge during the lifespan of this LLM (e.g. a few months for each version of ChatGPT <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib86" title="" class="ltx_ref">86</a>, <a href="#bib.bib102" title="" class="ltx_ref">102</a>]</cite>).
The definitions of knowledge and memory in the context of LLMs are provided in Section <a href="#S2" title="2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, and this paper uses knowledge as a countable noun.
Typical memory formats include model parameters and plain text for retrieval-augmented generative models (RAG); their write functions and read functions are listed in Table <a href="#T3" title="Table 3 ‣ 1 | Introduction ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, and their <math id="S1.p2.8.m8.1" class="ltx_Math" alttext="\text{cost}_{\text{write}}" display="inline"><semantics id="S1.p2.8.m8.1a"><msub id="S1.p2.8.m8.1.1" xref="S1.p2.8.m8.1.1.cmml"><mtext id="S1.p2.8.m8.1.1.2" xref="S1.p2.8.m8.1.1.2a.cmml">cost</mtext><mtext id="S1.p2.8.m8.1.1.3" xref="S1.p2.8.m8.1.1.3a.cmml">write</mtext></msub><annotation-xml encoding="MathML-Content" id="S1.p2.8.m8.1b"><apply id="S1.p2.8.m8.1.1.cmml" xref="S1.p2.8.m8.1.1"><csymbol cd="ambiguous" id="S1.p2.8.m8.1.1.1.cmml" xref="S1.p2.8.m8.1.1">subscript</csymbol><ci id="S1.p2.8.m8.1.1.2a.cmml" xref="S1.p2.8.m8.1.1.2"><mtext id="S1.p2.8.m8.1.1.2.cmml" xref="S1.p2.8.m8.1.1.2">cost</mtext></ci><ci id="S1.p2.8.m8.1.1.3a.cmml" xref="S1.p2.8.m8.1.1.3"><mtext mathsize="70%" id="S1.p2.8.m8.1.1.3.cmml" xref="S1.p2.8.m8.1.1.3">write</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p2.8.m8.1c">\text{cost}_{\text{write}}</annotation></semantics></math> and <math id="S1.p2.9.m9.1" class="ltx_Math" alttext="\text{cost}_{\text{read}}" display="inline"><semantics id="S1.p2.9.m9.1a"><msub id="S1.p2.9.m9.1.1" xref="S1.p2.9.m9.1.1.cmml"><mtext id="S1.p2.9.m9.1.1.2" xref="S1.p2.9.m9.1.1.2a.cmml">cost</mtext><mtext id="S1.p2.9.m9.1.1.3" xref="S1.p2.9.m9.1.1.3a.cmml">read</mtext></msub><annotation-xml encoding="MathML-Content" id="S1.p2.9.m9.1b"><apply id="S1.p2.9.m9.1.1.cmml" xref="S1.p2.9.m9.1.1"><csymbol cd="ambiguous" id="S1.p2.9.m9.1.1.1.cmml" xref="S1.p2.9.m9.1.1">subscript</csymbol><ci id="S1.p2.9.m9.1.1.2a.cmml" xref="S1.p2.9.m9.1.1.2"><mtext id="S1.p2.9.m9.1.1.2.cmml" xref="S1.p2.9.m9.1.1.2">cost</mtext></ci><ci id="S1.p2.9.m9.1.1.3a.cmml" xref="S1.p2.9.m9.1.1.3"><mtext mathsize="70%" id="S1.p2.9.m9.1.1.3.cmml" xref="S1.p2.9.m9.1.1.3">read</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p2.9.m9.1c">\text{cost}_{\text{read}}</annotation></semantics></math> are provided in Figure <a href="#F4" title="Figure 4 ‣ 1 | Introduction ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">We introduce a new memory format, explicit memory, characterized by moderately low write cost and read cost.
As depicted in Figure <a href="#F1" title="Figure 1 ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, our model first converts a knowledge base (or any text dataset) into explicit memories, implemented as sparse attention key-values, and then during inference, recalls these memories and integrates them into the self-attention layers.
Our design is simple so that most of the existing Transformer-based LLMs should be able to accommodate explicit memories with a little finetuning, and thus it is a general-purpose “model amplifier”.
Eventually, it should reduce the cost of pretraining LLMs, since there will be much less knowledge that must be stored in parameters, and thus less training data and smaller model size.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.5" class="ltx_p">The new memory format enables us to define a memory hierarchy for LLMs:</p>
<p id="S1.p4.2.2" class="ltx_p ltx_align_center">plain text (RAG) <math id="S1.p4.1.1.m1.1" class="ltx_Math" alttext="\to" display="inline"><semantics id="S1.p4.1.1.m1.1a"><mo stretchy="false" id="S1.p4.1.1.m1.1.1" xref="S1.p4.1.1.m1.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S1.p4.1.1.m1.1b"><ci id="S1.p4.1.1.m1.1.1.cmml" xref="S1.p4.1.1.m1.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p4.1.1.m1.1c">\to</annotation></semantics></math> explicit memory <math id="S1.p4.2.2.m2.1" class="ltx_Math" alttext="\to" display="inline"><semantics id="S1.p4.2.2.m2.1a"><mo stretchy="false" id="S1.p4.2.2.m2.1.1" xref="S1.p4.2.2.m2.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S1.p4.2.2.m2.1b"><ci id="S1.p4.2.2.m2.1.1.cmml" xref="S1.p4.2.2.m2.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p4.2.2.m2.1c">\to</annotation></semantics></math> model parameter</p>
<p id="S1.p4.4" class="ltx_p">such that by going up the hierarchy, <math id="S1.p4.3.m1.1" class="ltx_Math" alttext="\text{cost}_{\text{write}}" display="inline"><semantics id="S1.p4.3.m1.1a"><msub id="S1.p4.3.m1.1.1" xref="S1.p4.3.m1.1.1.cmml"><mtext id="S1.p4.3.m1.1.1.2" xref="S1.p4.3.m1.1.1.2a.cmml">cost</mtext><mtext id="S1.p4.3.m1.1.1.3" xref="S1.p4.3.m1.1.1.3a.cmml">write</mtext></msub><annotation-xml encoding="MathML-Content" id="S1.p4.3.m1.1b"><apply id="S1.p4.3.m1.1.1.cmml" xref="S1.p4.3.m1.1.1"><csymbol cd="ambiguous" id="S1.p4.3.m1.1.1.1.cmml" xref="S1.p4.3.m1.1.1">subscript</csymbol><ci id="S1.p4.3.m1.1.1.2a.cmml" xref="S1.p4.3.m1.1.1.2"><mtext id="S1.p4.3.m1.1.1.2.cmml" xref="S1.p4.3.m1.1.1.2">cost</mtext></ci><ci id="S1.p4.3.m1.1.1.3a.cmml" xref="S1.p4.3.m1.1.1.3"><mtext mathsize="70%" id="S1.p4.3.m1.1.1.3.cmml" xref="S1.p4.3.m1.1.1.3">write</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p4.3.m1.1c">\text{cost}_{\text{write}}</annotation></semantics></math> increases while <math id="S1.p4.4.m2.1" class="ltx_Math" alttext="\text{cost}_{\text{read}}" display="inline"><semantics id="S1.p4.4.m2.1a"><msub id="S1.p4.4.m2.1.1" xref="S1.p4.4.m2.1.1.cmml"><mtext id="S1.p4.4.m2.1.1.2" xref="S1.p4.4.m2.1.1.2a.cmml">cost</mtext><mtext id="S1.p4.4.m2.1.1.3" xref="S1.p4.4.m2.1.1.3a.cmml">read</mtext></msub><annotation-xml encoding="MathML-Content" id="S1.p4.4.m2.1b"><apply id="S1.p4.4.m2.1.1.cmml" xref="S1.p4.4.m2.1.1"><csymbol cd="ambiguous" id="S1.p4.4.m2.1.1.1.cmml" xref="S1.p4.4.m2.1.1">subscript</csymbol><ci id="S1.p4.4.m2.1.1.2a.cmml" xref="S1.p4.4.m2.1.1.2"><mtext id="S1.p4.4.m2.1.1.2.cmml" xref="S1.p4.4.m2.1.1.2">cost</mtext></ci><ci id="S1.p4.4.m2.1.1.3a.cmml" xref="S1.p4.4.m2.1.1.3"><mtext mathsize="70%" id="S1.p4.4.m2.1.1.3.cmml" xref="S1.p4.4.m2.1.1.3">read</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p4.4.m2.1c">\text{cost}_{\text{read}}</annotation></semantics></math> decreases.
To minimize the cost (<a href="#E1" title="Equation 1 ‣ 1 | Introduction ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>), one should store each piece of knowledge that is very frequently/rarely used in the top/bottom of this hierarchy, and everything in between as explicit memory.
As illustrated in Table <a href="#T3" title="Table 3 ‣ 1 | Introduction ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, the memory hierarchy of LLMs closely resembles that of humans.
For humans, the explicit/implicit memories are the long-term memories that are acquired and used consciously/unconsciously <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib59" title="" class="ltx_ref">59</a>]</cite>.</p>
</div>
<figure id="T3" class="ltx_table">
<table id="T3.2" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="T3.2.1" class="ltx_tr" style="background-color:#FFFFFF;">
<td id="T3.2.1.1" class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t">
<span id="T3.2.1.1.1" class="ltx_text"></span><span id="T3.2.1.1.2" class="ltx_text" style="font-size:90%;"> </span><span id="T3.2.1.1.3" class="ltx_text" style="font-size:90%;">
<span id="T3.2.1.1.3.1" class="ltx_tabular ltx_align_middle">
<span id="T3.2.1.1.3.1.1" class="ltx_tr">
<span id="T3.2.1.1.3.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Memory format</span></span>
<span id="T3.2.1.1.3.1.2" class="ltx_tr">
<span id="T3.2.1.1.3.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="T3.2.1.1.3.1.2.1.1" class="ltx_text" style="background-color:#FFFFFF;">of humans</span></span></span>
</span></span><span id="T3.2.1.1.4" class="ltx_text"></span><span id="T3.2.1.1.5" class="ltx_text" style="font-size:90%;"></span>
</td>
<td id="T3.2.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="T3.2.1.2.1" class="ltx_text" style="font-size:90%;">Example</span></td>
<td id="T3.2.1.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">
<span id="T3.2.1.3.1" class="ltx_text"></span><span id="T3.2.1.3.2" class="ltx_text" style="font-size:90%;"> </span><span id="T3.2.1.3.3" class="ltx_text" style="font-size:90%;">
<span id="T3.2.1.3.3.1" class="ltx_tabular ltx_align_middle">
<span id="T3.2.1.3.3.1.1" class="ltx_tr">
<span id="T3.2.1.3.3.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Memory format</span></span>
<span id="T3.2.1.3.3.1.2" class="ltx_tr">
<span id="T3.2.1.3.3.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="T3.2.1.3.3.1.2.1.1" class="ltx_text" style="background-color:#FFFFFF;">of LLMs</span></span></span>
</span></span><span id="T3.2.1.3.4" class="ltx_text"></span><span id="T3.2.1.3.5" class="ltx_text" style="font-size:90%;"></span>
</td>
<td id="T3.2.1.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="T3.2.1.4.1" class="ltx_text" style="font-size:90%;">Write</span></td>
<td id="T3.2.1.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="T3.2.1.5.1" class="ltx_text" style="font-size:90%;">Read</span></td>
</tr>
<tr id="T3.2.2" class="ltx_tr" style="background-color:#FFFFFF;">
<td id="T3.2.2.1" class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_tt"><span id="T3.2.2.1.1" class="ltx_text" style="font-size:90%;background-color:#FFFFFF;">Implicit memory</span></td>
<td id="T3.2.2.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt"><span id="T3.2.2.2.1" class="ltx_text" style="font-size:90%;background-color:#FFFFFF;">common expressions</span></td>
<td id="T3.2.2.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt"><span id="T3.2.2.3.1" class="ltx_text" style="font-size:90%;background-color:#FFFFFF;">model parameters</span></td>
<td id="T3.2.2.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt"><span id="T3.2.2.4.1" class="ltx_text" style="font-size:90%;background-color:#FFFFFF;">training</span></td>
<td id="T3.2.2.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt"><span id="T3.2.2.5.1" class="ltx_text" style="font-size:90%;background-color:#FFFFFF;">matrix multiplication</span></td>
</tr>
<tr id="T3.2.3" class="ltx_tr" style="background-color:#E6F3F3;">
<td id="T3.2.3.1" class="ltx_td ltx_align_center ltx_border_l ltx_border_r ltx_border_t"><span id="T3.2.3.1.1" class="ltx_text" style="font-size:90%;background-color:#E6F3F3;">Explicit memory</span></td>
<td id="T3.2.3.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="T3.2.3.2.1" class="ltx_text" style="font-size:90%;background-color:#E6F3F3;">books read</span></td>
<td id="T3.2.3.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="T3.2.3.3.1" class="ltx_text" style="font-size:90%;background-color:#E6F3F3;">this work</span></td>
<td id="T3.2.3.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="T3.2.3.4.1" class="ltx_text" style="font-size:90%;background-color:#E6F3F3;">memory encoding</span></td>
<td id="T3.2.3.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="T3.2.3.5.1" class="ltx_text" style="font-size:90%;background-color:#E6F3F3;">self-attention</span></td>
</tr>
<tr id="T3.2.4" class="ltx_tr" style="background-color:#FFFFFF;">
<td id="T3.2.4.1" class="ltx_td ltx_align_center ltx_border_b ltx_border_l ltx_border_r ltx_border_t"><span id="T3.2.4.1.1" class="ltx_text" style="font-size:90%;background-color:#FFFFFF;">External information</span></td>
<td id="T3.2.4.2" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t"><span id="T3.2.4.2.1" class="ltx_text" style="font-size:90%;background-color:#FFFFFF;">open-book exam</span></td>
<td id="T3.2.4.3" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t"><span id="T3.2.4.3.1" class="ltx_text" style="font-size:90%;background-color:#FFFFFF;">plain text (RAG)</span></td>
<td id="T3.2.4.4" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t"><span id="T3.2.4.4.1" class="ltx_text" style="font-size:90%;background-color:#FFFFFF;">none</span></td>
<td id="T3.2.4.5" class="ltx_td ltx_align_center ltx_border_b ltx_border_r ltx_border_t"><span id="T3.2.4.5.1" class="ltx_text" style="font-size:90%;background-color:#FFFFFF;">encode from scratch</span></td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering" style="font-size:90%;"><span class="ltx_tag ltx_tag_table">Table 3: </span>Analogy of the memory hierarchies of humans and LLMs.</figcaption>
</figure>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.1" class="ltx_p">As a remark, one can compare the plain LLMs to patients with impaired explicit memory, e.g. due to injury to the medial temporal lobe.
These patients are largely unable to learn semantic knowledge (usually stored as explicit memory), but can acquire sensorimotor skills through repetitive priming (stored as implicit memories) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib42" title="" class="ltx_ref">42</a>, <a href="#bib.bib26" title="" class="ltx_ref">26</a>, <a href="#bib.bib12" title="" class="ltx_ref">12</a>]</cite>.
Thus, one may hypothesize that due to the lack of explicit memory, the training of plain LLMs is as inefficient as repetitive priming, and thus has ample room for improvement.
In analogy with humans, for instance, it is easy to recall and talk about a book we just read, but to recite it as unconsciously as tying shoe laces requires an enormous effort to force this knowledge into our muscle memory.
From this perspective, it is not surprising that LLM training consumes so much data and energy <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib121" title="" class="ltx_ref">121</a>, <a href="#bib.bib77" title="" class="ltx_ref">77</a>]</cite>.
We want to rescue LLMs from this poor condition by equipping it with an explicit memory mechanism as efficient as that of humans.</p>
</div>
<figure id="F4" class="ltx_figure"><img src="/html/2407.01178/assets/Figures/Theory/total_cost_2B_chunk.png" id="F4.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="419" height="240" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="F4.4.2.1" class="ltx_text" style="font-size:90%;">Figure 4</span>: </span><span id="F4.2.1" class="ltx_text" style="font-size:90%;">The total cost (TFlops) of writing and reading a piece of knowledge by our 2.4B model with respect to its expected usage count.
The curves represent the cost of different memory formats, and the shaded area represents the minimum cost given the optimal format.
The plot indicates that <math id="F4.2.1.m1.2" class="ltx_Math" alttext="(0.494,13400)" display="inline"><semantics id="F4.2.1.m1.2b"><mrow id="F4.2.1.m1.2.3.2" xref="F4.2.1.m1.2.3.1.cmml"><mo stretchy="false" id="F4.2.1.m1.2.3.2.1" xref="F4.2.1.m1.2.3.1.cmml">(</mo><mn id="F4.2.1.m1.1.1" xref="F4.2.1.m1.1.1.cmml">0.494</mn><mo id="F4.2.1.m1.2.3.2.2" xref="F4.2.1.m1.2.3.1.cmml">,</mo><mn id="F4.2.1.m1.2.2" xref="F4.2.1.m1.2.2.cmml">13400</mn><mo stretchy="false" id="F4.2.1.m1.2.3.2.3" xref="F4.2.1.m1.2.3.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="F4.2.1.m1.2c"><interval closure="open" id="F4.2.1.m1.2.3.1.cmml" xref="F4.2.1.m1.2.3.2"><cn type="float" id="F4.2.1.m1.1.1.cmml" xref="F4.2.1.m1.1.1">0.494</cn><cn type="integer" id="F4.2.1.m1.2.2.cmml" xref="F4.2.1.m1.2.2">13400</cn></interval></annotation-xml><annotation encoding="application/x-tex" id="F4.2.1.m1.2d">(0.494,13400)</annotation></semantics></math> is the advantage interval for explicit memory.
The calculations are provided in Appendix <a href="#A1" title="Appendix A Cost Estimation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a>.
(The blue curve is only a lower bound on the cost of model parameters.)</span></figcaption>
</figure>
<div id="S1.p6" class="ltx_para">
<p id="S1.p6.3" class="ltx_p">A quantitative illustration of the cost (<a href="#E1" title="Equation 1 ‣ 1 | Introduction ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>) is given by Figure <a href="#F4" title="Figure 4 ‣ 1 | Introduction ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, where we characterize <math id="S1.p6.1.m1.1" class="ltx_Math" alttext="\text{cost}_{\text{write}}" display="inline"><semantics id="S1.p6.1.m1.1a"><msub id="S1.p6.1.m1.1.1" xref="S1.p6.1.m1.1.1.cmml"><mtext id="S1.p6.1.m1.1.1.2" xref="S1.p6.1.m1.1.1.2a.cmml">cost</mtext><mtext id="S1.p6.1.m1.1.1.3" xref="S1.p6.1.m1.1.1.3a.cmml">write</mtext></msub><annotation-xml encoding="MathML-Content" id="S1.p6.1.m1.1b"><apply id="S1.p6.1.m1.1.1.cmml" xref="S1.p6.1.m1.1.1"><csymbol cd="ambiguous" id="S1.p6.1.m1.1.1.1.cmml" xref="S1.p6.1.m1.1.1">subscript</csymbol><ci id="S1.p6.1.m1.1.1.2a.cmml" xref="S1.p6.1.m1.1.1.2"><mtext id="S1.p6.1.m1.1.1.2.cmml" xref="S1.p6.1.m1.1.1.2">cost</mtext></ci><ci id="S1.p6.1.m1.1.1.3a.cmml" xref="S1.p6.1.m1.1.1.3"><mtext mathsize="70%" id="S1.p6.1.m1.1.1.3.cmml" xref="S1.p6.1.m1.1.1.3">write</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p6.1.m1.1c">\text{cost}_{\text{write}}</annotation></semantics></math> and <math id="S1.p6.2.m2.1" class="ltx_Math" alttext="\text{cost}_{\text{read}}" display="inline"><semantics id="S1.p6.2.m2.1a"><msub id="S1.p6.2.m2.1.1" xref="S1.p6.2.m2.1.1.cmml"><mtext id="S1.p6.2.m2.1.1.2" xref="S1.p6.2.m2.1.1.2a.cmml">cost</mtext><mtext id="S1.p6.2.m2.1.1.3" xref="S1.p6.2.m2.1.1.3a.cmml">read</mtext></msub><annotation-xml encoding="MathML-Content" id="S1.p6.2.m2.1b"><apply id="S1.p6.2.m2.1.1.cmml" xref="S1.p6.2.m2.1.1"><csymbol cd="ambiguous" id="S1.p6.2.m2.1.1.1.cmml" xref="S1.p6.2.m2.1.1">subscript</csymbol><ci id="S1.p6.2.m2.1.1.2a.cmml" xref="S1.p6.2.m2.1.1.2"><mtext id="S1.p6.2.m2.1.1.2.cmml" xref="S1.p6.2.m2.1.1.2">cost</mtext></ci><ci id="S1.p6.2.m2.1.1.3a.cmml" xref="S1.p6.2.m2.1.1.3"><mtext mathsize="70%" id="S1.p6.2.m2.1.1.3.cmml" xref="S1.p6.2.m2.1.1.3">read</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p6.2.m2.1c">\text{cost}_{\text{read}}</annotation></semantics></math> by the amount of compute (TFlops).
The plot indicates that if a piece of knowledge has an expected usage count <math id="S1.p6.3.m3.2" class="ltx_Math" alttext="\in(0.494,13400)" display="inline"><semantics id="S1.p6.3.m3.2a"><mrow id="S1.p6.3.m3.2.3" xref="S1.p6.3.m3.2.3.cmml"><mi id="S1.p6.3.m3.2.3.2" xref="S1.p6.3.m3.2.3.2.cmml"></mi><mo id="S1.p6.3.m3.2.3.1" xref="S1.p6.3.m3.2.3.1.cmml">∈</mo><mrow id="S1.p6.3.m3.2.3.3.2" xref="S1.p6.3.m3.2.3.3.1.cmml"><mo stretchy="false" id="S1.p6.3.m3.2.3.3.2.1" xref="S1.p6.3.m3.2.3.3.1.cmml">(</mo><mn id="S1.p6.3.m3.1.1" xref="S1.p6.3.m3.1.1.cmml">0.494</mn><mo id="S1.p6.3.m3.2.3.3.2.2" xref="S1.p6.3.m3.2.3.3.1.cmml">,</mo><mn id="S1.p6.3.m3.2.2" xref="S1.p6.3.m3.2.2.cmml">13400</mn><mo stretchy="false" id="S1.p6.3.m3.2.3.3.2.3" xref="S1.p6.3.m3.2.3.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.p6.3.m3.2b"><apply id="S1.p6.3.m3.2.3.cmml" xref="S1.p6.3.m3.2.3"><in id="S1.p6.3.m3.2.3.1.cmml" xref="S1.p6.3.m3.2.3.1"></in><csymbol cd="latexml" id="S1.p6.3.m3.2.3.2.cmml" xref="S1.p6.3.m3.2.3.2">absent</csymbol><interval closure="open" id="S1.p6.3.m3.2.3.3.1.cmml" xref="S1.p6.3.m3.2.3.3.2"><cn type="float" id="S1.p6.3.m3.1.1.cmml" xref="S1.p6.3.m3.1.1">0.494</cn><cn type="integer" id="S1.p6.3.m3.2.2.cmml" xref="S1.p6.3.m3.2.2">13400</cn></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p6.3.m3.2c">\in(0.494,13400)</annotation></semantics></math>, then it is optimal to be stored as an explicit memory.
Moreover, the introduction of explicit memory helps to externalize the knowledge stored in model parameters and thus allow us to use a lighter backbone, which ultimately reduces all the costs in Figure <a href="#F4" title="Figure 4 ‣ 1 | Introduction ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>.</p>
</div>
<div id="S1.p7" class="ltx_para">
<p id="S1.p7.3" class="ltx_p">The second motivation for explicit memory is to alleviate the issue of knowledge traversal.
Knowledge traversal happens when the LLM wastefully invokes all its parameters (and thus all its knowledge) each time it generates a token.
As an analogy, it is unreasonable for humans to recall everything they learned whenever they write a word.
Let us define the knowledge efficiency of an LLM as the ratio of the minimum amount of knowledge sufficient for one decoding step to the amount of knowledge actually used.
An optimistic estimation of knowledge efficiency for a 10B LLM is <math id="S1.p7.1.m1.1" class="ltx_Math" alttext="10^{-5}" display="inline"><semantics id="S1.p7.1.m1.1a"><msup id="S1.p7.1.m1.1.1" xref="S1.p7.1.m1.1.1.cmml"><mn id="S1.p7.1.m1.1.1.2" xref="S1.p7.1.m1.1.1.2.cmml">10</mn><mrow id="S1.p7.1.m1.1.1.3" xref="S1.p7.1.m1.1.1.3.cmml"><mo id="S1.p7.1.m1.1.1.3a" xref="S1.p7.1.m1.1.1.3.cmml">−</mo><mn id="S1.p7.1.m1.1.1.3.2" xref="S1.p7.1.m1.1.1.3.2.cmml">5</mn></mrow></msup><annotation-xml encoding="MathML-Content" id="S1.p7.1.m1.1b"><apply id="S1.p7.1.m1.1.1.cmml" xref="S1.p7.1.m1.1.1"><csymbol cd="ambiguous" id="S1.p7.1.m1.1.1.1.cmml" xref="S1.p7.1.m1.1.1">superscript</csymbol><cn type="integer" id="S1.p7.1.m1.1.1.2.cmml" xref="S1.p7.1.m1.1.1.2">10</cn><apply id="S1.p7.1.m1.1.1.3.cmml" xref="S1.p7.1.m1.1.1.3"><minus id="S1.p7.1.m1.1.1.3.1.cmml" xref="S1.p7.1.m1.1.1.3"></minus><cn type="integer" id="S1.p7.1.m1.1.1.3.2.cmml" xref="S1.p7.1.m1.1.1.3.2">5</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p7.1.m1.1c">10^{-5}</annotation></semantics></math>:
On one hand, it is unlikely that generating one token would require more than <math id="S1.p7.2.m2.1" class="ltx_Math" alttext="10^{4}" display="inline"><semantics id="S1.p7.2.m2.1a"><msup id="S1.p7.2.m2.1.1" xref="S1.p7.2.m2.1.1.cmml"><mn id="S1.p7.2.m2.1.1.2" xref="S1.p7.2.m2.1.1.2.cmml">10</mn><mn id="S1.p7.2.m2.1.1.3" xref="S1.p7.2.m2.1.1.3.cmml">4</mn></msup><annotation-xml encoding="MathML-Content" id="S1.p7.2.m2.1b"><apply id="S1.p7.2.m2.1.1.cmml" xref="S1.p7.2.m2.1.1"><csymbol cd="ambiguous" id="S1.p7.2.m2.1.1.1.cmml" xref="S1.p7.2.m2.1.1">superscript</csymbol><cn type="integer" id="S1.p7.2.m2.1.1.2.cmml" xref="S1.p7.2.m2.1.1.2">10</cn><cn type="integer" id="S1.p7.2.m2.1.1.3.cmml" xref="S1.p7.2.m2.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p7.2.m2.1c">10^{4}</annotation></semantics></math> bits of knowledge (roughly equivalent to a thousand-token long passage, sufficient for enumerating all necessary knowledge);
on the other hand, each parameter is involved in the computation and each stores at least 0.1 bit of knowledge <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib7" title="" class="ltx_ref">7</a>, Result 10]</cite> (this density could be much higher if the LLM is trained on cleaner data), thus using <math id="S1.p7.3.m3.1" class="ltx_Math" alttext="10^{9}" display="inline"><semantics id="S1.p7.3.m3.1a"><msup id="S1.p7.3.m3.1.1" xref="S1.p7.3.m3.1.1.cmml"><mn id="S1.p7.3.m3.1.1.2" xref="S1.p7.3.m3.1.1.2.cmml">10</mn><mn id="S1.p7.3.m3.1.1.3" xref="S1.p7.3.m3.1.1.3.cmml">9</mn></msup><annotation-xml encoding="MathML-Content" id="S1.p7.3.m3.1b"><apply id="S1.p7.3.m3.1.1.cmml" xref="S1.p7.3.m3.1.1"><csymbol cd="ambiguous" id="S1.p7.3.m3.1.1.1.cmml" xref="S1.p7.3.m3.1.1">superscript</csymbol><cn type="integer" id="S1.p7.3.m3.1.1.2.cmml" xref="S1.p7.3.m3.1.1.2">10</cn><cn type="integer" id="S1.p7.3.m3.1.1.3.cmml" xref="S1.p7.3.m3.1.1.3">9</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p7.3.m3.1c">10^{9}</annotation></semantics></math> bits in total.</p>
</div>
<div id="S1.p8" class="ltx_para">
<p id="S1.p8.3" class="ltx_p">A novel architecture is needed to boost the knowledge efficiency of LLMs from <math id="S1.p8.1.m1.1" class="ltx_Math" alttext="10^{-5}" display="inline"><semantics id="S1.p8.1.m1.1a"><msup id="S1.p8.1.m1.1.1" xref="S1.p8.1.m1.1.1.cmml"><mn id="S1.p8.1.m1.1.1.2" xref="S1.p8.1.m1.1.1.2.cmml">10</mn><mrow id="S1.p8.1.m1.1.1.3" xref="S1.p8.1.m1.1.1.3.cmml"><mo id="S1.p8.1.m1.1.1.3a" xref="S1.p8.1.m1.1.1.3.cmml">−</mo><mn id="S1.p8.1.m1.1.1.3.2" xref="S1.p8.1.m1.1.1.3.2.cmml">5</mn></mrow></msup><annotation-xml encoding="MathML-Content" id="S1.p8.1.m1.1b"><apply id="S1.p8.1.m1.1.1.cmml" xref="S1.p8.1.m1.1.1"><csymbol cd="ambiguous" id="S1.p8.1.m1.1.1.1.cmml" xref="S1.p8.1.m1.1.1">superscript</csymbol><cn type="integer" id="S1.p8.1.m1.1.1.2.cmml" xref="S1.p8.1.m1.1.1.2">10</cn><apply id="S1.p8.1.m1.1.1.3.cmml" xref="S1.p8.1.m1.1.1.3"><minus id="S1.p8.1.m1.1.1.3.1.cmml" xref="S1.p8.1.m1.1.1.3"></minus><cn type="integer" id="S1.p8.1.m1.1.1.3.2.cmml" xref="S1.p8.1.m1.1.1.3.2">5</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p8.1.m1.1c">10^{-5}</annotation></semantics></math> to <math id="S1.p8.2.m2.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S1.p8.2.m2.1a"><mn id="S1.p8.2.m2.1.1" xref="S1.p8.2.m2.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S1.p8.2.m2.1b"><cn type="integer" id="S1.p8.2.m2.1.1.cmml" xref="S1.p8.2.m2.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S1.p8.2.m2.1c">1</annotation></semantics></math>, whereas current designs are far from this goal.
Consider the mixture-of-experts architecture (MoE) for instance, which uses multiple MLP layers (experts) in each Transformer block and process each token with only a few MLPs.
The boost of MoE, namely the ratio of the total amount of parameters to the amount of active parameters, is usually bounded by <math id="S1.p8.3.m3.1" class="ltx_Math" alttext="4\sim 32" display="inline"><semantics id="S1.p8.3.m3.1a"><mrow id="S1.p8.3.m3.1.1" xref="S1.p8.3.m3.1.1.cmml"><mn id="S1.p8.3.m3.1.1.2" xref="S1.p8.3.m3.1.1.2.cmml">4</mn><mo id="S1.p8.3.m3.1.1.1" xref="S1.p8.3.m3.1.1.1.cmml">∼</mo><mn id="S1.p8.3.m3.1.1.3" xref="S1.p8.3.m3.1.1.3.cmml">32</mn></mrow><annotation-xml encoding="MathML-Content" id="S1.p8.3.m3.1b"><apply id="S1.p8.3.m3.1.1.cmml" xref="S1.p8.3.m3.1.1"><csymbol cd="latexml" id="S1.p8.3.m3.1.1.1.cmml" xref="S1.p8.3.m3.1.1.1">similar-to</csymbol><cn type="integer" id="S1.p8.3.m3.1.1.2.cmml" xref="S1.p8.3.m3.1.1.2">4</cn><cn type="integer" id="S1.p8.3.m3.1.1.3.cmml" xref="S1.p8.3.m3.1.1.3">32</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p8.3.m3.1c">4\sim 32</annotation></semantics></math> <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib40" title="" class="ltx_ref">40</a>, <a href="#bib.bib56" title="" class="ltx_ref">56</a>, <a href="#bib.bib98" title="" class="ltx_ref">98</a>]</cite>.
Similarly, neither the mixture-of-depth architecture <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib37" title="" class="ltx_ref">37</a>, <a href="#bib.bib94" title="" class="ltx_ref">94</a>]</cite> nor sparsified MLP neurons and attention heads <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib75" title="" class="ltx_ref">75</a>]</cite> can bring greater gains.
RAG appears very sparse if we compare the amount of retrieved texts with the size of the text database; nevertheless, RAG is usually built upon a plain LLM as backbone, which provides most of the knowledge used in inference, and thus offers little assistance in addressing the knowledge traversal problem.</p>
</div>
<div id="S1.p9" class="ltx_para">
<p id="S1.p9.1" class="ltx_p">An ideal solution is to retrieve only the needed parameters for each token.
This is naturally achieved by explicit memories if we compare memory recall to parameter retrieval.</p>
</div>
<div id="S1.p10" class="ltx_para">
<p id="S1.p10.1" class="ltx_p">The third motivation is that, as a human-like design, explicit memory enables LLMs to develop more human-like capabilities.
To name a few,</p>
<ul id="S1.I1" class="ltx_itemize">
<li id="S1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S1.I1.i1.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S1.I1.i1.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S1.I1.i1.1.1.m1.1.1" xref="S1.I1.i1.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S1.I1.i1.1.1.m1.1c"><ci id="S1.I1.i1.1.1.m1.1.1.cmml" xref="S1.I1.i1.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.I1.i1.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S1.I1.i1.p1" class="ltx_para">
<p id="S1.I1.i1.p1.3" class="ltx_p">Infinitely long context:
LLMs have the difficulty of processing long texts since their working memory (context key-values) costs too much GPU memory and compute.
Meanwhile, despite that humans have very limited working memory capacity <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib27" title="" class="ltx_ref">27</a>, <a href="#bib.bib28" title="" class="ltx_ref">28</a>]</cite>, they can manage to read and write long texts by converting working memories to explicit memories (thus saving space) and retrieving only the needed explicit memories for inference (thus saving compute).
Similarly, by saving explicit memories on drives and doing frequent and constant-size retrieval, LLMs can handle arbitrarily long contexts with time complexity <math id="S1.I1.i1.p1.1.m1.1" class="ltx_Math" alttext="O(l\log l)" display="inline"><semantics id="S1.I1.i1.p1.1.m1.1a"><mrow id="S1.I1.i1.p1.1.m1.1.1" xref="S1.I1.i1.p1.1.m1.1.1.cmml"><mi id="S1.I1.i1.p1.1.m1.1.1.3" xref="S1.I1.i1.p1.1.m1.1.1.3.cmml">O</mi><mo lspace="0em" rspace="0em" id="S1.I1.i1.p1.1.m1.1.1.2" xref="S1.I1.i1.p1.1.m1.1.1.2.cmml">​</mo><mrow id="S1.I1.i1.p1.1.m1.1.1.1.1" xref="S1.I1.i1.p1.1.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S1.I1.i1.p1.1.m1.1.1.1.1.2" xref="S1.I1.i1.p1.1.m1.1.1.1.1.1.cmml">(</mo><mrow id="S1.I1.i1.p1.1.m1.1.1.1.1.1" xref="S1.I1.i1.p1.1.m1.1.1.1.1.1.cmml"><mi id="S1.I1.i1.p1.1.m1.1.1.1.1.1.2" xref="S1.I1.i1.p1.1.m1.1.1.1.1.1.2.cmml">l</mi><mo lspace="0.167em" rspace="0em" id="S1.I1.i1.p1.1.m1.1.1.1.1.1.1" xref="S1.I1.i1.p1.1.m1.1.1.1.1.1.1.cmml">​</mo><mrow id="S1.I1.i1.p1.1.m1.1.1.1.1.1.3" xref="S1.I1.i1.p1.1.m1.1.1.1.1.1.3.cmml"><mi id="S1.I1.i1.p1.1.m1.1.1.1.1.1.3.1" xref="S1.I1.i1.p1.1.m1.1.1.1.1.1.3.1.cmml">log</mi><mo lspace="0.167em" id="S1.I1.i1.p1.1.m1.1.1.1.1.1.3a" xref="S1.I1.i1.p1.1.m1.1.1.1.1.1.3.cmml">⁡</mo><mi id="S1.I1.i1.p1.1.m1.1.1.1.1.1.3.2" xref="S1.I1.i1.p1.1.m1.1.1.1.1.1.3.2.cmml">l</mi></mrow></mrow><mo stretchy="false" id="S1.I1.i1.p1.1.m1.1.1.1.1.3" xref="S1.I1.i1.p1.1.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.I1.i1.p1.1.m1.1b"><apply id="S1.I1.i1.p1.1.m1.1.1.cmml" xref="S1.I1.i1.p1.1.m1.1.1"><times id="S1.I1.i1.p1.1.m1.1.1.2.cmml" xref="S1.I1.i1.p1.1.m1.1.1.2"></times><ci id="S1.I1.i1.p1.1.m1.1.1.3.cmml" xref="S1.I1.i1.p1.1.m1.1.1.3">𝑂</ci><apply id="S1.I1.i1.p1.1.m1.1.1.1.1.1.cmml" xref="S1.I1.i1.p1.1.m1.1.1.1.1"><times id="S1.I1.i1.p1.1.m1.1.1.1.1.1.1.cmml" xref="S1.I1.i1.p1.1.m1.1.1.1.1.1.1"></times><ci id="S1.I1.i1.p1.1.m1.1.1.1.1.1.2.cmml" xref="S1.I1.i1.p1.1.m1.1.1.1.1.1.2">𝑙</ci><apply id="S1.I1.i1.p1.1.m1.1.1.1.1.1.3.cmml" xref="S1.I1.i1.p1.1.m1.1.1.1.1.1.3"><log id="S1.I1.i1.p1.1.m1.1.1.1.1.1.3.1.cmml" xref="S1.I1.i1.p1.1.m1.1.1.1.1.1.3.1"></log><ci id="S1.I1.i1.p1.1.m1.1.1.1.1.1.3.2.cmml" xref="S1.I1.i1.p1.1.m1.1.1.1.1.1.3.2">𝑙</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.I1.i1.p1.1.m1.1c">O(l\log l)</annotation></semantics></math> instead of <math id="S1.I1.i1.p1.2.m2.1" class="ltx_Math" alttext="\Theta(l^{2})" display="inline"><semantics id="S1.I1.i1.p1.2.m2.1a"><mrow id="S1.I1.i1.p1.2.m2.1.1" xref="S1.I1.i1.p1.2.m2.1.1.cmml"><mi mathvariant="normal" id="S1.I1.i1.p1.2.m2.1.1.3" xref="S1.I1.i1.p1.2.m2.1.1.3.cmml">Θ</mi><mo lspace="0em" rspace="0em" id="S1.I1.i1.p1.2.m2.1.1.2" xref="S1.I1.i1.p1.2.m2.1.1.2.cmml">​</mo><mrow id="S1.I1.i1.p1.2.m2.1.1.1.1" xref="S1.I1.i1.p1.2.m2.1.1.1.1.1.cmml"><mo stretchy="false" id="S1.I1.i1.p1.2.m2.1.1.1.1.2" xref="S1.I1.i1.p1.2.m2.1.1.1.1.1.cmml">(</mo><msup id="S1.I1.i1.p1.2.m2.1.1.1.1.1" xref="S1.I1.i1.p1.2.m2.1.1.1.1.1.cmml"><mi id="S1.I1.i1.p1.2.m2.1.1.1.1.1.2" xref="S1.I1.i1.p1.2.m2.1.1.1.1.1.2.cmml">l</mi><mn id="S1.I1.i1.p1.2.m2.1.1.1.1.1.3" xref="S1.I1.i1.p1.2.m2.1.1.1.1.1.3.cmml">2</mn></msup><mo stretchy="false" id="S1.I1.i1.p1.2.m2.1.1.1.1.3" xref="S1.I1.i1.p1.2.m2.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.I1.i1.p1.2.m2.1b"><apply id="S1.I1.i1.p1.2.m2.1.1.cmml" xref="S1.I1.i1.p1.2.m2.1.1"><times id="S1.I1.i1.p1.2.m2.1.1.2.cmml" xref="S1.I1.i1.p1.2.m2.1.1.2"></times><ci id="S1.I1.i1.p1.2.m2.1.1.3.cmml" xref="S1.I1.i1.p1.2.m2.1.1.3">Θ</ci><apply id="S1.I1.i1.p1.2.m2.1.1.1.1.1.cmml" xref="S1.I1.i1.p1.2.m2.1.1.1.1"><csymbol cd="ambiguous" id="S1.I1.i1.p1.2.m2.1.1.1.1.1.1.cmml" xref="S1.I1.i1.p1.2.m2.1.1.1.1">superscript</csymbol><ci id="S1.I1.i1.p1.2.m2.1.1.1.1.1.2.cmml" xref="S1.I1.i1.p1.2.m2.1.1.1.1.1.2">𝑙</ci><cn type="integer" id="S1.I1.i1.p1.2.m2.1.1.1.1.1.3.cmml" xref="S1.I1.i1.p1.2.m2.1.1.1.1.1.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.I1.i1.p1.2.m2.1c">\Theta(l^{2})</annotation></semantics></math>, where <math id="S1.I1.i1.p1.3.m3.1" class="ltx_Math" alttext="l" display="inline"><semantics id="S1.I1.i1.p1.3.m3.1a"><mi id="S1.I1.i1.p1.3.m3.1.1" xref="S1.I1.i1.p1.3.m3.1.1.cmml">l</mi><annotation-xml encoding="MathML-Content" id="S1.I1.i1.p1.3.m3.1b"><ci id="S1.I1.i1.p1.3.m3.1.1.cmml" xref="S1.I1.i1.p1.3.m3.1.1">𝑙</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.I1.i1.p1.3.m3.1c">l</annotation></semantics></math> is the context length.</p>
</div>
</li>
<li id="S1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S1.I1.i2.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S1.I1.i2.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S1.I1.i2.1.1.m1.1.1" xref="S1.I1.i2.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S1.I1.i2.1.1.m1.1c"><ci id="S1.I1.i2.1.1.m1.1.1.cmml" xref="S1.I1.i2.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.I1.i2.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S1.I1.i2.p1" class="ltx_para">
<p id="S1.I1.i2.p1.1" class="ltx_p">Memory consolidation: Instead of writing a piece of knowledge directly into implicit memory, i.e. training model parameters, LLM can first convert it to explicit memory through plain encoding, and then convert this explicit memory to implicit memory through a low-cost step such as compression and finetuning, thus reducing the overall cost.</p>
</div>
</li>
<li id="S1.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S1.I1.i3.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S1.I1.i3.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S1.I1.i3.1.1.m1.1.1" xref="S1.I1.i3.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S1.I1.i3.1.1.m1.1c"><ci id="S1.I1.i3.1.1.m1.1.1.cmml" xref="S1.I1.i3.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.I1.i3.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S1.I1.i3.p1" class="ltx_para">
<p id="S1.I1.i3.p1.1" class="ltx_p">Factuality and interpretability: Encoding texts as explicit memories is less susceptible to information loss compared to dissolving them in model parameters.
With more factual details provided by explicit memories, the LLMs would have less tendency to hallucinate.
Meanwhile, the correspondence of explicit memories to readable texts makes the inference more transparent to humans, and also allows the LLM to consciously examine its own thought process.</p>
</div>
</li>
</ul>
<p id="S1.p10.2" class="ltx_p">We demonstrate the improved factuality in the experiments section, and leave the rest to future work.</p>
</div>
<div id="S1.p11" class="ltx_para">
<p id="S1.p11.1" class="ltx_p">In this work, we introduce a novel architecture and training scheme for LLM based on explicit memory.
The architecture is called Memory<sup id="S1.p11.1.1" class="ltx_sup">3</sup>, as explicit memory is the third form of memory in LLM after working memory (context key-values) and implicit memory (model parameters).</p>
<ul id="S1.I2" class="ltx_itemize">
<li id="S1.I2.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S1.I2.i1.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S1.I2.i1.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S1.I2.i1.1.1.m1.1.1" xref="S1.I2.i1.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S1.I2.i1.1.1.m1.1c"><ci id="S1.I2.i1.1.1.m1.1.1.cmml" xref="S1.I2.i1.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.I2.i1.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S1.I2.i1.p1" class="ltx_para">
<p id="S1.I2.i1.p1.1" class="ltx_p">Memory<sup id="S1.I2.i1.p1.1.1" class="ltx_sup">3</sup> utilizes explicit memories during inference, alleviating the burden of model parameters to memorize specific knowledge.</p>
</div>
</li>
<li id="S1.I2.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S1.I2.i2.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S1.I2.i2.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S1.I2.i2.1.1.m1.1.1" xref="S1.I2.i2.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S1.I2.i2.1.1.m1.1c"><ci id="S1.I2.i2.1.1.m1.1.1.cmml" xref="S1.I2.i2.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.I2.i2.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S1.I2.i2.p1" class="ltx_para">
<p id="S1.I2.i2.p1.1" class="ltx_p">The explicit memories are encoded from our knowledge base, and our sparse memory format maintains a realistic storage size.</p>
</div>
</li>
<li id="S1.I2.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S1.I2.i3.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S1.I2.i3.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S1.I2.i3.1.1.m1.1.1" xref="S1.I2.i3.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S1.I2.i3.1.1.m1.1c"><ci id="S1.I2.i3.1.1.m1.1.1.cmml" xref="S1.I2.i3.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.I2.i3.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S1.I2.i3.p1" class="ltx_para">
<p id="S1.I2.i3.p1.1" class="ltx_p">We trained from scratch a Memory<sup id="S1.I2.i3.p1.1.1" class="ltx_sup">3</sup> model with 2.4B non-embedding parameters, and its performance surpasses SOTA models with greater sizes.
It also enjoys better performance and faster inference than RAG.</p>
</div>
</li>
<li id="S1.I2.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S1.I2.i4.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S1.I2.i4.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S1.I2.i4.1.1.m1.1.1" xref="S1.I2.i4.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S1.I2.i4.1.1.m1.1c"><ci id="S1.I2.i4.1.1.m1.1.1.cmml" xref="S1.I2.i4.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.I2.i4.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S1.I2.i4.p1" class="ltx_para">
<p id="S1.I2.i4.p1.1" class="ltx_p">Furthermore, Memory<sup id="S1.I2.i4.p1.1.1" class="ltx_sup">3</sup> boosts factuality and alleviates hallucination, and it enables fast adaptation to professional tasks.</p>
</div>
</li>
</ul>
<p id="S1.p11.5" class="ltx_p">This paper is structured as follows:
Section <a href="#S2" title="2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> lays the theoretical foundation for Memory<sup id="S1.p11.5.1" class="ltx_sup">3</sup>, in particular our definitions of knowledge and memory.
Section <a href="#S3" title="3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> discusses the basic design of Memory<sup id="S1.p11.5.2" class="ltx_sup">3</sup>, including its architecture and training scheme.
Sections <a href="#S4" title="4 | Pretraining Data ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, <a href="#S5" title="5 | Pretrain ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, and <a href="#S6" title="6 | Fine-tuning and Alignment ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> describes the training of Memory<sup id="S1.p11.5.3" class="ltx_sup">3</sup>.
Section <a href="#S7" title="7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a> evaluates the performance of Memory<sup id="S1.p11.5.4" class="ltx_sup">3</sup> on general benchmarks and professional tasks.
Finally, Section <a href="#S8" title="8 | Conclusion ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a> concludes this paper and discusses future works.</p>
</div>
<section id="S1.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsection" style="font-size:120%;color:#00868B;">1.1 <span id="S1.SS1.1.1" class="ltx_text" style="color:#808080;">| </span>Related work</h3>

<section id="S1.SS1.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsubsection" style="color:#00868B;">1.1.1 <span id="S1.SS1.SSS1.1.1" class="ltx_text" style="color:#808080;">| </span>Retrieval-augmented Training</h4>

<div id="S1.SS1.SSS1.p1" class="ltx_para">
<p id="S1.SS1.SSS1.p1.1" class="ltx_p">Several language models have incorporated text retrieval from the pretraining stage.
REALM <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib49" title="" class="ltx_ref">49</a>]</cite> augments a BERT model with one retrieval step to solve QA tasks.
Retro <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib16" title="" class="ltx_ref">16</a>]</cite> enhances auto-regressive decoding with multiple rounds of retrieval, once per 64 tokens.
The retrieved texts are injected through a two-layer encoder and then several cross-attention layers in the decoder.
Retro++ <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib113" title="" class="ltx_ref">113</a>]</cite> explores the scalability of Retro by reproducing Retro up to 9.5B parameters.</p>
</div>
<div id="S1.SS1.SSS1.p2" class="ltx_para">
<p id="S1.SS1.SSS1.p2.1" class="ltx_p">Meanwhile, several models are adapted to retrieval in the finetuning stage.
WebGPT <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib83" title="" class="ltx_ref">83</a>]</cite> learns to use search engine through imitation learning in a text-based web-browsing environment.
Toolformer <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib100" title="" class="ltx_ref">100</a>]</cite> performs decoding with multiple tools including search engine, and the finetuning data is labeled by the LM iself.</p>
</div>
<div id="S1.SS1.SSS1.p3" class="ltx_para">
<p id="S1.SS1.SSS1.p3.1" class="ltx_p">The closest model to ours is Retro.
Unlike explicit memory, Retro needs to encode the retrieved texts in real-time during inference.
To alleviate the cost of encoding these references, it chooses to use a separate, shallow encoder and also retrieve few references.
Intuitively, this compromise greatly reduces the amount of knowledge that can be extracted and supplied to inference.</p>
</div>
<div id="S1.SS1.SSS1.p4" class="ltx_para">
<p id="S1.SS1.SSS1.p4.1" class="ltx_p">Another line of research utilizes retrieval to aid long-context modeling.
Memorizing Transformer <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib123" title="" class="ltx_ref">123</a>]</cite> extends the context of language models by an approximate kNN lookup into a non-differentiable cache of past key-value pairs.
LongLlama <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib112" title="" class="ltx_ref">112</a>]</cite> enhances the discernability of context key-value pairs by a finetuning process inspired by contrastive learning.
LONGMEM <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib118" title="" class="ltx_ref">118</a>]</cite> designs a decoupled architecture to avoid the memory staleness issue when training the Memorizing Transformer.
These methods are not directly applicable to large knowledge bases since the resulting key-value caches will occupy enormous space.
Our method overcomes this difficulty through a more intense memory sparsification method.</p>
</div>
</section>
<section id="S1.SS1.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsubsection" style="color:#00868B;">1.1.2 <span id="S1.SS1.SSS2.1.1" class="ltx_text" style="color:#808080;">| </span>Sparse Computation</h4>

<div id="S1.SS1.SSS2.p1" class="ltx_para">
<p id="S1.SS1.SSS2.p1.6" class="ltx_p">To combat the aforementioned knowledge traversal problem and improve knowledge efficiency, ongoing works seek novel architectures that process each token with a minimum and adaptive subset of model parameters.
This adaptive sparsity is also known as contextual sparsity <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib75" title="" class="ltx_ref">75</a>]</cite>.
The Mixture-of-Experts (MoE) use sparse routing to assign Transformer submodules to tokens, scaling model capacity without large increases in training or inference costs.
The most common MoE design <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib40" title="" class="ltx_ref">40</a>]</cite> hosts multiple MLP layers in each Transformer block and routes each token to a few MLPs with the highest allocation score predicted by a linear classifier.
Furthermore, variants based on compression such as QMoE <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib41" title="" class="ltx_ref">41</a>]</cite> are introduced to alleviate the memory burden of MoE.
Despite the sparse routing, the boost in parameter efficiency is usually bounded by <math id="S1.SS1.SSS2.p1.1.m1.1" class="ltx_Math" alttext="4\sim 32" display="inline"><semantics id="S1.SS1.SSS2.p1.1.m1.1a"><mrow id="S1.SS1.SSS2.p1.1.m1.1.1" xref="S1.SS1.SSS2.p1.1.m1.1.1.cmml"><mn id="S1.SS1.SSS2.p1.1.m1.1.1.2" xref="S1.SS1.SSS2.p1.1.m1.1.1.2.cmml">4</mn><mo id="S1.SS1.SSS2.p1.1.m1.1.1.1" xref="S1.SS1.SSS2.p1.1.m1.1.1.1.cmml">∼</mo><mn id="S1.SS1.SSS2.p1.1.m1.1.1.3" xref="S1.SS1.SSS2.p1.1.m1.1.1.3.cmml">32</mn></mrow><annotation-xml encoding="MathML-Content" id="S1.SS1.SSS2.p1.1.m1.1b"><apply id="S1.SS1.SSS2.p1.1.m1.1.1.cmml" xref="S1.SS1.SSS2.p1.1.m1.1.1"><csymbol cd="latexml" id="S1.SS1.SSS2.p1.1.m1.1.1.1.cmml" xref="S1.SS1.SSS2.p1.1.m1.1.1.1">similar-to</csymbol><cn type="integer" id="S1.SS1.SSS2.p1.1.m1.1.1.2.cmml" xref="S1.SS1.SSS2.p1.1.m1.1.1.2">4</cn><cn type="integer" id="S1.SS1.SSS2.p1.1.m1.1.1.3.cmml" xref="S1.SS1.SSS2.p1.1.m1.1.1.3">32</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.SSS2.p1.1.m1.1c">4\sim 32</annotation></semantics></math>.
For instance, the Arctic model <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib98" title="" class="ltx_ref">98</a>]</cite>, one of the sparsest MoE LLM in recent years, has an active parameter ratio of about <math id="S1.SS1.SSS2.p1.2.m2.1" class="ltx_Math" alttext="3.5\%" display="inline"><semantics id="S1.SS1.SSS2.p1.2.m2.1a"><mrow id="S1.SS1.SSS2.p1.2.m2.1.1" xref="S1.SS1.SSS2.p1.2.m2.1.1.cmml"><mn id="S1.SS1.SSS2.p1.2.m2.1.1.2" xref="S1.SS1.SSS2.p1.2.m2.1.1.2.cmml">3.5</mn><mo id="S1.SS1.SSS2.p1.2.m2.1.1.1" xref="S1.SS1.SSS2.p1.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S1.SS1.SSS2.p1.2.m2.1b"><apply id="S1.SS1.SSS2.p1.2.m2.1.1.cmml" xref="S1.SS1.SSS2.p1.2.m2.1.1"><csymbol cd="latexml" id="S1.SS1.SSS2.p1.2.m2.1.1.1.cmml" xref="S1.SS1.SSS2.p1.2.m2.1.1.1">percent</csymbol><cn type="float" id="S1.SS1.SSS2.p1.2.m2.1.1.2.cmml" xref="S1.SS1.SSS2.p1.2.m2.1.1.2">3.5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.SSS2.p1.2.m2.1c">3.5\%</annotation></semantics></math>.
Similarly, the Mixture of Depth architecture processes each token with an adaptive subset of the model layers.
The implementations can be based on early exit <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib37" title="" class="ltx_ref">37</a>]</cite> or top-<math id="S1.SS1.SSS2.p1.3.m3.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S1.SS1.SSS2.p1.3.m3.1a"><mi id="S1.SS1.SSS2.p1.3.m3.1.1" xref="S1.SS1.SSS2.p1.3.m3.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S1.SS1.SSS2.p1.3.m3.1b"><ci id="S1.SS1.SSS2.p1.3.m3.1.1.cmml" xref="S1.SS1.SSS2.p1.3.m3.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.SSS2.p1.3.m3.1c">k</annotation></semantics></math> routing <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib94" title="" class="ltx_ref">94</a>]</cite>, reducing the amount of compute to <math id="S1.SS1.SSS2.p1.4.m4.1" class="ltx_Math" alttext="12.5\sim 50\%" display="inline"><semantics id="S1.SS1.SSS2.p1.4.m4.1a"><mrow id="S1.SS1.SSS2.p1.4.m4.1.1" xref="S1.SS1.SSS2.p1.4.m4.1.1.cmml"><mn id="S1.SS1.SSS2.p1.4.m4.1.1.2" xref="S1.SS1.SSS2.p1.4.m4.1.1.2.cmml">12.5</mn><mo id="S1.SS1.SSS2.p1.4.m4.1.1.1" xref="S1.SS1.SSS2.p1.4.m4.1.1.1.cmml">∼</mo><mrow id="S1.SS1.SSS2.p1.4.m4.1.1.3" xref="S1.SS1.SSS2.p1.4.m4.1.1.3.cmml"><mn id="S1.SS1.SSS2.p1.4.m4.1.1.3.2" xref="S1.SS1.SSS2.p1.4.m4.1.1.3.2.cmml">50</mn><mo id="S1.SS1.SSS2.p1.4.m4.1.1.3.1" xref="S1.SS1.SSS2.p1.4.m4.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.SS1.SSS2.p1.4.m4.1b"><apply id="S1.SS1.SSS2.p1.4.m4.1.1.cmml" xref="S1.SS1.SSS2.p1.4.m4.1.1"><csymbol cd="latexml" id="S1.SS1.SSS2.p1.4.m4.1.1.1.cmml" xref="S1.SS1.SSS2.p1.4.m4.1.1.1">similar-to</csymbol><cn type="float" id="S1.SS1.SSS2.p1.4.m4.1.1.2.cmml" xref="S1.SS1.SSS2.p1.4.m4.1.1.2">12.5</cn><apply id="S1.SS1.SSS2.p1.4.m4.1.1.3.cmml" xref="S1.SS1.SSS2.p1.4.m4.1.1.3"><csymbol cd="latexml" id="S1.SS1.SSS2.p1.4.m4.1.1.3.1.cmml" xref="S1.SS1.SSS2.p1.4.m4.1.1.3.1">percent</csymbol><cn type="integer" id="S1.SS1.SSS2.p1.4.m4.1.1.3.2.cmml" xref="S1.SS1.SSS2.p1.4.m4.1.1.3.2">50</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.SSS2.p1.4.m4.1c">12.5\sim 50\%</annotation></semantics></math>.
More fine-grained approaches can perform sparsification at the level of individual MLP neurons and attention heads.
The model Deja Vu <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib75" title="" class="ltx_ref">75</a>]</cite> trains a low-cost network for each MLP/attention layer that predicts the relevance of each neuron/head at this layer to each token.
Then, during inference, Deja Vu keeps the top <math id="S1.SS1.SSS2.p1.5.m5.1" class="ltx_Math" alttext="5\sim 15\%" display="inline"><semantics id="S1.SS1.SSS2.p1.5.m5.1a"><mrow id="S1.SS1.SSS2.p1.5.m5.1.1" xref="S1.SS1.SSS2.p1.5.m5.1.1.cmml"><mn id="S1.SS1.SSS2.p1.5.m5.1.1.2" xref="S1.SS1.SSS2.p1.5.m5.1.1.2.cmml">5</mn><mo id="S1.SS1.SSS2.p1.5.m5.1.1.1" xref="S1.SS1.SSS2.p1.5.m5.1.1.1.cmml">∼</mo><mrow id="S1.SS1.SSS2.p1.5.m5.1.1.3" xref="S1.SS1.SSS2.p1.5.m5.1.1.3.cmml"><mn id="S1.SS1.SSS2.p1.5.m5.1.1.3.2" xref="S1.SS1.SSS2.p1.5.m5.1.1.3.2.cmml">15</mn><mo id="S1.SS1.SSS2.p1.5.m5.1.1.3.1" xref="S1.SS1.SSS2.p1.5.m5.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.SS1.SSS2.p1.5.m5.1b"><apply id="S1.SS1.SSS2.p1.5.m5.1.1.cmml" xref="S1.SS1.SSS2.p1.5.m5.1.1"><csymbol cd="latexml" id="S1.SS1.SSS2.p1.5.m5.1.1.1.cmml" xref="S1.SS1.SSS2.p1.5.m5.1.1.1">similar-to</csymbol><cn type="integer" id="S1.SS1.SSS2.p1.5.m5.1.1.2.cmml" xref="S1.SS1.SSS2.p1.5.m5.1.1.2">5</cn><apply id="S1.SS1.SSS2.p1.5.m5.1.1.3.cmml" xref="S1.SS1.SSS2.p1.5.m5.1.1.3"><csymbol cd="latexml" id="S1.SS1.SSS2.p1.5.m5.1.1.3.1.cmml" xref="S1.SS1.SSS2.p1.5.m5.1.1.3.1">percent</csymbol><cn type="integer" id="S1.SS1.SSS2.p1.5.m5.1.1.3.2.cmml" xref="S1.SS1.SSS2.p1.5.m5.1.1.3.2">15</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.SSS2.p1.5.m5.1c">5\sim 15\%</annotation></semantics></math> MLP neurons and <math id="S1.SS1.SSS2.p1.6.m6.1" class="ltx_Math" alttext="20\sim 50\%" display="inline"><semantics id="S1.SS1.SSS2.p1.6.m6.1a"><mrow id="S1.SS1.SSS2.p1.6.m6.1.1" xref="S1.SS1.SSS2.p1.6.m6.1.1.cmml"><mn id="S1.SS1.SSS2.p1.6.m6.1.1.2" xref="S1.SS1.SSS2.p1.6.m6.1.1.2.cmml">20</mn><mo id="S1.SS1.SSS2.p1.6.m6.1.1.1" xref="S1.SS1.SSS2.p1.6.m6.1.1.1.cmml">∼</mo><mrow id="S1.SS1.SSS2.p1.6.m6.1.1.3" xref="S1.SS1.SSS2.p1.6.m6.1.1.3.cmml"><mn id="S1.SS1.SSS2.p1.6.m6.1.1.3.2" xref="S1.SS1.SSS2.p1.6.m6.1.1.3.2.cmml">50</mn><mo id="S1.SS1.SSS2.p1.6.m6.1.1.3.1" xref="S1.SS1.SSS2.p1.6.m6.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.SS1.SSS2.p1.6.m6.1b"><apply id="S1.SS1.SSS2.p1.6.m6.1.1.cmml" xref="S1.SS1.SSS2.p1.6.m6.1.1"><csymbol cd="latexml" id="S1.SS1.SSS2.p1.6.m6.1.1.1.cmml" xref="S1.SS1.SSS2.p1.6.m6.1.1.1">similar-to</csymbol><cn type="integer" id="S1.SS1.SSS2.p1.6.m6.1.1.2.cmml" xref="S1.SS1.SSS2.p1.6.m6.1.1.2">20</cn><apply id="S1.SS1.SSS2.p1.6.m6.1.1.3.cmml" xref="S1.SS1.SSS2.p1.6.m6.1.1.3"><csymbol cd="latexml" id="S1.SS1.SSS2.p1.6.m6.1.1.3.1.cmml" xref="S1.SS1.SSS2.p1.6.m6.1.1.3.1">percent</csymbol><cn type="integer" id="S1.SS1.SSS2.p1.6.m6.1.1.3.2.cmml" xref="S1.SS1.SSS2.p1.6.m6.1.1.3.2">50</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.SSS2.p1.6.m6.1c">20\sim 50\%</annotation></semantics></math> attention heads for each token.</p>
</div>
</section>
<section id="S1.SS1.SSS3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsubsection" style="color:#00868B;">1.1.3 <span id="S1.SS1.SSS3.1.1" class="ltx_text" style="color:#808080;">| </span>Parameter as memory</h4>

<div id="S1.SS1.SSS3.p1" class="ltx_para">
<p id="S1.SS1.SSS3.p1.6" class="ltx_p">Several works have portrayed model parameters as implicit memory, in accordance with our philosophy.
<cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib46" title="" class="ltx_ref">46</a>]</cite> demonstrates that the neurons in the MLP layers of GPTs behave like key-value pairs.
Specifically, with the MLP layer written as <math id="S1.SS1.SSS3.p1.1.m1.1" class="ltx_Math" alttext="\sigma(XK^{T})V" display="inline"><semantics id="S1.SS1.SSS3.p1.1.m1.1a"><mrow id="S1.SS1.SSS3.p1.1.m1.1.1" xref="S1.SS1.SSS3.p1.1.m1.1.1.cmml"><mi id="S1.SS1.SSS3.p1.1.m1.1.1.3" xref="S1.SS1.SSS3.p1.1.m1.1.1.3.cmml">σ</mi><mo lspace="0em" rspace="0em" id="S1.SS1.SSS3.p1.1.m1.1.1.2" xref="S1.SS1.SSS3.p1.1.m1.1.1.2.cmml">​</mo><mrow id="S1.SS1.SSS3.p1.1.m1.1.1.1.1" xref="S1.SS1.SSS3.p1.1.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S1.SS1.SSS3.p1.1.m1.1.1.1.1.2" xref="S1.SS1.SSS3.p1.1.m1.1.1.1.1.1.cmml">(</mo><mrow id="S1.SS1.SSS3.p1.1.m1.1.1.1.1.1" xref="S1.SS1.SSS3.p1.1.m1.1.1.1.1.1.cmml"><mi id="S1.SS1.SSS3.p1.1.m1.1.1.1.1.1.2" xref="S1.SS1.SSS3.p1.1.m1.1.1.1.1.1.2.cmml">X</mi><mo lspace="0em" rspace="0em" id="S1.SS1.SSS3.p1.1.m1.1.1.1.1.1.1" xref="S1.SS1.SSS3.p1.1.m1.1.1.1.1.1.1.cmml">​</mo><msup id="S1.SS1.SSS3.p1.1.m1.1.1.1.1.1.3" xref="S1.SS1.SSS3.p1.1.m1.1.1.1.1.1.3.cmml"><mi id="S1.SS1.SSS3.p1.1.m1.1.1.1.1.1.3.2" xref="S1.SS1.SSS3.p1.1.m1.1.1.1.1.1.3.2.cmml">K</mi><mi id="S1.SS1.SSS3.p1.1.m1.1.1.1.1.1.3.3" xref="S1.SS1.SSS3.p1.1.m1.1.1.1.1.1.3.3.cmml">T</mi></msup></mrow><mo stretchy="false" id="S1.SS1.SSS3.p1.1.m1.1.1.1.1.3" xref="S1.SS1.SSS3.p1.1.m1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S1.SS1.SSS3.p1.1.m1.1.1.2a" xref="S1.SS1.SSS3.p1.1.m1.1.1.2.cmml">​</mo><mi id="S1.SS1.SSS3.p1.1.m1.1.1.4" xref="S1.SS1.SSS3.p1.1.m1.1.1.4.cmml">V</mi></mrow><annotation-xml encoding="MathML-Content" id="S1.SS1.SSS3.p1.1.m1.1b"><apply id="S1.SS1.SSS3.p1.1.m1.1.1.cmml" xref="S1.SS1.SSS3.p1.1.m1.1.1"><times id="S1.SS1.SSS3.p1.1.m1.1.1.2.cmml" xref="S1.SS1.SSS3.p1.1.m1.1.1.2"></times><ci id="S1.SS1.SSS3.p1.1.m1.1.1.3.cmml" xref="S1.SS1.SSS3.p1.1.m1.1.1.3">𝜎</ci><apply id="S1.SS1.SSS3.p1.1.m1.1.1.1.1.1.cmml" xref="S1.SS1.SSS3.p1.1.m1.1.1.1.1"><times id="S1.SS1.SSS3.p1.1.m1.1.1.1.1.1.1.cmml" xref="S1.SS1.SSS3.p1.1.m1.1.1.1.1.1.1"></times><ci id="S1.SS1.SSS3.p1.1.m1.1.1.1.1.1.2.cmml" xref="S1.SS1.SSS3.p1.1.m1.1.1.1.1.1.2">𝑋</ci><apply id="S1.SS1.SSS3.p1.1.m1.1.1.1.1.1.3.cmml" xref="S1.SS1.SSS3.p1.1.m1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S1.SS1.SSS3.p1.1.m1.1.1.1.1.1.3.1.cmml" xref="S1.SS1.SSS3.p1.1.m1.1.1.1.1.1.3">superscript</csymbol><ci id="S1.SS1.SSS3.p1.1.m1.1.1.1.1.1.3.2.cmml" xref="S1.SS1.SSS3.p1.1.m1.1.1.1.1.1.3.2">𝐾</ci><ci id="S1.SS1.SSS3.p1.1.m1.1.1.1.1.1.3.3.cmml" xref="S1.SS1.SSS3.p1.1.m1.1.1.1.1.1.3.3">𝑇</ci></apply></apply><ci id="S1.SS1.SSS3.p1.1.m1.1.1.4.cmml" xref="S1.SS1.SSS3.p1.1.m1.1.1.4">𝑉</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.SSS3.p1.1.m1.1c">\sigma(XK^{T})V</annotation></semantics></math>, each row of the first layer weight <math id="S1.SS1.SSS3.p1.2.m2.1" class="ltx_Math" alttext="K_{i}" display="inline"><semantics id="S1.SS1.SSS3.p1.2.m2.1a"><msub id="S1.SS1.SSS3.p1.2.m2.1.1" xref="S1.SS1.SSS3.p1.2.m2.1.1.cmml"><mi id="S1.SS1.SSS3.p1.2.m2.1.1.2" xref="S1.SS1.SSS3.p1.2.m2.1.1.2.cmml">K</mi><mi id="S1.SS1.SSS3.p1.2.m2.1.1.3" xref="S1.SS1.SSS3.p1.2.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S1.SS1.SSS3.p1.2.m2.1b"><apply id="S1.SS1.SSS3.p1.2.m2.1.1.cmml" xref="S1.SS1.SSS3.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S1.SS1.SSS3.p1.2.m2.1.1.1.cmml" xref="S1.SS1.SSS3.p1.2.m2.1.1">subscript</csymbol><ci id="S1.SS1.SSS3.p1.2.m2.1.1.2.cmml" xref="S1.SS1.SSS3.p1.2.m2.1.1.2">𝐾</ci><ci id="S1.SS1.SSS3.p1.2.m2.1.1.3.cmml" xref="S1.SS1.SSS3.p1.2.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.SSS3.p1.2.m2.1c">K_{i}</annotation></semantics></math> functions like a key vector, with the corresponding row in the second layer weight <math id="S1.SS1.SSS3.p1.3.m3.1" class="ltx_Math" alttext="V_{i}" display="inline"><semantics id="S1.SS1.SSS3.p1.3.m3.1a"><msub id="S1.SS1.SSS3.p1.3.m3.1.1" xref="S1.SS1.SSS3.p1.3.m3.1.1.cmml"><mi id="S1.SS1.SSS3.p1.3.m3.1.1.2" xref="S1.SS1.SSS3.p1.3.m3.1.1.2.cmml">V</mi><mi id="S1.SS1.SSS3.p1.3.m3.1.1.3" xref="S1.SS1.SSS3.p1.3.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S1.SS1.SSS3.p1.3.m3.1b"><apply id="S1.SS1.SSS3.p1.3.m3.1.1.cmml" xref="S1.SS1.SSS3.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S1.SS1.SSS3.p1.3.m3.1.1.1.cmml" xref="S1.SS1.SSS3.p1.3.m3.1.1">subscript</csymbol><ci id="S1.SS1.SSS3.p1.3.m3.1.1.2.cmml" xref="S1.SS1.SSS3.p1.3.m3.1.1.2">𝑉</ci><ci id="S1.SS1.SSS3.p1.3.m3.1.1.3.cmml" xref="S1.SS1.SSS3.p1.3.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.SSS3.p1.3.m3.1c">V_{i}</annotation></semantics></math> being the value vector.
<cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib46" title="" class="ltx_ref">46</a>]</cite> observes that for most of the MLP neurons, the <math id="S1.SS1.SSS3.p1.4.m4.1" class="ltx_Math" alttext="K_{i}" display="inline"><semantics id="S1.SS1.SSS3.p1.4.m4.1a"><msub id="S1.SS1.SSS3.p1.4.m4.1.1" xref="S1.SS1.SSS3.p1.4.m4.1.1.cmml"><mi id="S1.SS1.SSS3.p1.4.m4.1.1.2" xref="S1.SS1.SSS3.p1.4.m4.1.1.2.cmml">K</mi><mi id="S1.SS1.SSS3.p1.4.m4.1.1.3" xref="S1.SS1.SSS3.p1.4.m4.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S1.SS1.SSS3.p1.4.m4.1b"><apply id="S1.SS1.SSS3.p1.4.m4.1.1.cmml" xref="S1.SS1.SSS3.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S1.SS1.SSS3.p1.4.m4.1.1.1.cmml" xref="S1.SS1.SSS3.p1.4.m4.1.1">subscript</csymbol><ci id="S1.SS1.SSS3.p1.4.m4.1.1.2.cmml" xref="S1.SS1.SSS3.p1.4.m4.1.1.2">𝐾</ci><ci id="S1.SS1.SSS3.p1.4.m4.1.1.3.cmml" xref="S1.SS1.SSS3.p1.4.m4.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.SSS3.p1.4.m4.1c">K_{i}</annotation></semantics></math> is activated by context texts that obey some human interpretable pattern, and the <math id="S1.SS1.SSS3.p1.5.m5.1" class="ltx_Math" alttext="V_{i}" display="inline"><semantics id="S1.SS1.SSS3.p1.5.m5.1a"><msub id="S1.SS1.SSS3.p1.5.m5.1.1" xref="S1.SS1.SSS3.p1.5.m5.1.1.cmml"><mi id="S1.SS1.SSS3.p1.5.m5.1.1.2" xref="S1.SS1.SSS3.p1.5.m5.1.1.2.cmml">V</mi><mi id="S1.SS1.SSS3.p1.5.m5.1.1.3" xref="S1.SS1.SSS3.p1.5.m5.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S1.SS1.SSS3.p1.5.m5.1b"><apply id="S1.SS1.SSS3.p1.5.m5.1.1.cmml" xref="S1.SS1.SSS3.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S1.SS1.SSS3.p1.5.m5.1.1.1.cmml" xref="S1.SS1.SSS3.p1.5.m5.1.1">subscript</csymbol><ci id="S1.SS1.SSS3.p1.5.m5.1.1.2.cmml" xref="S1.SS1.SSS3.p1.5.m5.1.1.2">𝑉</ci><ci id="S1.SS1.SSS3.p1.5.m5.1.1.3.cmml" xref="S1.SS1.SSS3.p1.5.m5.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.SSS3.p1.5.m5.1c">V_{i}</annotation></semantics></math> activates the column of the output matrix that corresponds to the most probable next token of the pattern (e.g. <math id="S1.SS1.SSS3.p1.6.m6.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S1.SS1.SSS3.p1.6.m6.1a"><mi id="S1.SS1.SSS3.p1.6.m6.1.1" xref="S1.SS1.SSS3.p1.6.m6.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S1.SS1.SSS3.p1.6.m6.1b"><ci id="S1.SS1.SSS3.p1.6.m6.1.1.cmml" xref="S1.SS1.SSS3.p1.6.m6.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.SS1.SSS3.p1.6.m6.1c">n</annotation></semantics></math>-gram).
Based on this observation, <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib108" title="" class="ltx_ref">108</a>]</cite> designs a GPT variant that consists of only attention layers, with performance matching that of the usual GPTs.
The MLP layers are incorporated into the attention layers in the form of key-value vector pairs, which are called persistent memories.
Similarly, using sensitivity analysis, <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib29" title="" class="ltx_ref">29</a>]</cite> discovers that factual knowledge learned by BERT is often localized at one or few MLP neurons.
These neurons are called “knowledge neurons”, and by manipulating them, <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib29" title="" class="ltx_ref">29</a>]</cite> manages to update single pieces of knowledge of BERT.
Meanwhile, <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib38" title="" class="ltx_ref">38</a>]</cite> studies an interesting phenomenon known as superposition or polysemanticity, that a neural network can store many unrelated concepts into a single neuron.</p>
</div>
</section>
</section>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_section" style="font-size:144%;color:#00868B;">2 <span id="S2.1.1" class="ltx_text" style="color:#808080;">| </span>Memory Circuitry Theory</h2>

<div id="S2.p1" class="ltx_para">
<p id="S2.p1.1" class="ltx_p">This section introduces our memory circuitry theory, which defines knowledge and memory in the context of LLM.
We will see that this theory helps to determine which knowledge can be stored as explicit memory, and what kind of model architecture is suitable for reading and writing explicit memories.
For readers interested primarily in the results, it may suffice to review Claim <a href="#Thmclaim1" title="Claim 1. ‣ 2.3 | Memory ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> and Remark <a href="#Thmremark1" title="Remark 1. ‣ 2.3 | Memory ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> before proceeding to the subsequent sections.
The concepts to be discussed are illustrated in Figure <a href="#F5" title="Figure 5 ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>.</p>
</div>
<figure id="F5" class="ltx_figure"><img src="/html/2407.01178/assets/Figures/Theory/memory_circuitry_theory.png" id="F5.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="395" height="306" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="F5.2.1.1" class="ltx_text" style="font-size:90%;">Figure 5</span>: </span><span id="F5.3.2" class="ltx_text" style="font-size:90%;">Categorization of knowledge and memory formats.
The explicit memories, extracted from model activations, lie half-way between raw data and model parameters, so we use a dotted line to indicate that they may or may not be regarded as parameters.</span></figcaption>
</figure>
<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsection" style="font-size:120%;color:#00868B;">2.1 <span id="S2.SS1.1.1" class="ltx_text" style="color:#808080;">| </span>Preliminaries</h3>

<div id="S2.SS1.p1" class="ltx_para">
<p id="S2.SS1.p1.1" class="ltx_p">The objective is to decompose the computations of a LLM into smaller, recurring parts, and analyze which parts can be separated from the LLM.
These small parts will be defined as the “knowledge” of the LLM, and this characterization helps to identify what knowledge can be externalized as explicit memory, enabling both the memory hierarchy and a lightweight backbone.</p>
</div>
<div id="S2.SS1.p2" class="ltx_para">
<p id="S2.SS1.p2.2" class="ltx_p">One behaviorist approach is to define the smaller parts as input-output relations between small subsequences, such that if the input text contains a subsequence belonging to some pattern, then the output text of the LLM contains a subsequence that belongs to some corresponding pattern.</p>
<ul id="S2.I1" class="ltx_itemize">
<li id="S2.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S2.I1.i1.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S2.I1.i1.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S2.I1.i1.1.1.m1.1.1" xref="S2.I1.i1.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S2.I1.i1.1.1.m1.1c"><ci id="S2.I1.i1.1.1.m1.1.1.cmml" xref="S2.I1.i1.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i1.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S2.I1.i1.p1" class="ltx_para">
<p id="S2.I1.i1.p1.1" class="ltx_p">One specific input-output relation is that if the immediate context contains “China” and “capital”, then output the token “Beijing”.</p>
</div>
</li>
<li id="S2.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S2.I1.i2.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S2.I1.i2.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S2.I1.i2.1.1.m1.1.1" xref="S2.I1.i2.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S2.I1.i2.1.1.m1.1c"><ci id="S2.I1.i2.1.1.m1.1.1.cmml" xref="S2.I1.i2.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i2.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S2.I1.i2.p1" class="ltx_para">
<p id="S2.I1.i2.p1.2" class="ltx_p">One abstract input-output relation is that if the immediate context is some arithmetic expression (e.g. “<math id="S2.I1.i2.p1.1.m1.1" class="ltx_Math" alttext="123\times 456=" display="inline"><semantics id="S2.I1.i2.p1.1.m1.1a"><mrow id="S2.I1.i2.p1.1.m1.1.1" xref="S2.I1.i2.p1.1.m1.1.1.cmml"><mrow id="S2.I1.i2.p1.1.m1.1.1.2" xref="S2.I1.i2.p1.1.m1.1.1.2.cmml"><mn id="S2.I1.i2.p1.1.m1.1.1.2.2" xref="S2.I1.i2.p1.1.m1.1.1.2.2.cmml">123</mn><mo lspace="0.222em" rspace="0.222em" id="S2.I1.i2.p1.1.m1.1.1.2.1" xref="S2.I1.i2.p1.1.m1.1.1.2.1.cmml">×</mo><mn id="S2.I1.i2.p1.1.m1.1.1.2.3" xref="S2.I1.i2.p1.1.m1.1.1.2.3.cmml">456</mn></mrow><mo id="S2.I1.i2.p1.1.m1.1.1.1" xref="S2.I1.i2.p1.1.m1.1.1.1.cmml">=</mo><mi id="S2.I1.i2.p1.1.m1.1.1.3" xref="S2.I1.i2.p1.1.m1.1.1.3.cmml"></mi></mrow><annotation-xml encoding="MathML-Content" id="S2.I1.i2.p1.1.m1.1b"><apply id="S2.I1.i2.p1.1.m1.1.1.cmml" xref="S2.I1.i2.p1.1.m1.1.1"><eq id="S2.I1.i2.p1.1.m1.1.1.1.cmml" xref="S2.I1.i2.p1.1.m1.1.1.1"></eq><apply id="S2.I1.i2.p1.1.m1.1.1.2.cmml" xref="S2.I1.i2.p1.1.m1.1.1.2"><times id="S2.I1.i2.p1.1.m1.1.1.2.1.cmml" xref="S2.I1.i2.p1.1.m1.1.1.2.1"></times><cn type="integer" id="S2.I1.i2.p1.1.m1.1.1.2.2.cmml" xref="S2.I1.i2.p1.1.m1.1.1.2.2">123</cn><cn type="integer" id="S2.I1.i2.p1.1.m1.1.1.2.3.cmml" xref="S2.I1.i2.p1.1.m1.1.1.2.3">456</cn></apply><csymbol cd="latexml" id="S2.I1.i2.p1.1.m1.1.1.3.cmml" xref="S2.I1.i2.p1.1.m1.1.1.3">absent</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i2.p1.1.m1.1c">123\times 456=</annotation></semantics></math>”) then output the answer (e.g. “<math id="S2.I1.i2.p1.2.m2.1" class="ltx_Math" alttext="56088" display="inline"><semantics id="S2.I1.i2.p1.2.m2.1a"><mn id="S2.I1.i2.p1.2.m2.1.1" xref="S2.I1.i2.p1.2.m2.1.1.cmml">56088</mn><annotation-xml encoding="MathML-Content" id="S2.I1.i2.p1.2.m2.1b"><cn type="integer" id="S2.I1.i2.p1.2.m2.1.1.cmml" xref="S2.I1.i2.p1.2.m2.1.1">56088</cn></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i2.p1.2.m2.1c">56088</annotation></semantics></math>”).</p>
</div>
</li>
<li id="S2.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S2.I1.i3.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S2.I1.i3.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S2.I1.i3.1.1.m1.1.1" xref="S2.I1.i3.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S2.I1.i3.1.1.m1.1c"><ci id="S2.I1.i3.1.1.m1.1.1.cmml" xref="S2.I1.i3.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i3.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S2.I1.i3.p1" class="ltx_para">
<p id="S2.I1.i3.p1.1" class="ltx_p">One abstract relation that will be mentioned frequently is the “search, copy and paste” <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib85" title="" class="ltx_ref">85</a>]</cite>, such that if the context has the form “…[a][b]…[a]” then output “[b]”, where [a] and [b] are arbitrary tokens.</p>
</div>
</li>
</ul>
<p id="S2.SS1.p2.1" class="ltx_p">A decomposition into these relations seems natural since autoregressive LLMs can be seen as upgraded versions of <math id="S2.SS1.p2.1.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S2.SS1.p2.1.m1.1a"><mi id="S2.SS1.p2.1.m1.1.1" xref="S2.SS1.p2.1.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.1.m1.1b"><ci id="S2.SS1.p2.1.m1.1.1.cmml" xref="S2.SS1.p2.1.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.1.m1.1c">n</annotation></semantics></math>-grams, with the fixed input/output segments generalized to flexible patterns and with the plain lookup table generalized to multi-step computations.</p>
</div>
<div id="S2.SS1.p3" class="ltx_para">
<p id="S2.SS1.p3.1" class="ltx_p">Nevertheless, a behaviorist approach is insufficient since an input-output relation alone cannot uniquely pin down a piece of knowledge:
a LLM may answer correctly to arithmetic questions based on either the actual knowledge of arithmetic or memorization (hosting a lookup table for all expressions such as “<math id="S2.SS1.p3.1.m1.1" class="ltx_Math" alttext="123\times 456=56088" display="inline"><semantics id="S2.SS1.p3.1.m1.1a"><mrow id="S2.SS1.p3.1.m1.1.1" xref="S2.SS1.p3.1.m1.1.1.cmml"><mrow id="S2.SS1.p3.1.m1.1.1.2" xref="S2.SS1.p3.1.m1.1.1.2.cmml"><mn id="S2.SS1.p3.1.m1.1.1.2.2" xref="S2.SS1.p3.1.m1.1.1.2.2.cmml">123</mn><mo lspace="0.222em" rspace="0.222em" id="S2.SS1.p3.1.m1.1.1.2.1" xref="S2.SS1.p3.1.m1.1.1.2.1.cmml">×</mo><mn id="S2.SS1.p3.1.m1.1.1.2.3" xref="S2.SS1.p3.1.m1.1.1.2.3.cmml">456</mn></mrow><mo id="S2.SS1.p3.1.m1.1.1.1" xref="S2.SS1.p3.1.m1.1.1.1.cmml">=</mo><mn id="S2.SS1.p3.1.m1.1.1.3" xref="S2.SS1.p3.1.m1.1.1.3.cmml">56088</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.1.m1.1b"><apply id="S2.SS1.p3.1.m1.1.1.cmml" xref="S2.SS1.p3.1.m1.1.1"><eq id="S2.SS1.p3.1.m1.1.1.1.cmml" xref="S2.SS1.p3.1.m1.1.1.1"></eq><apply id="S2.SS1.p3.1.m1.1.1.2.cmml" xref="S2.SS1.p3.1.m1.1.1.2"><times id="S2.SS1.p3.1.m1.1.1.2.1.cmml" xref="S2.SS1.p3.1.m1.1.1.2.1"></times><cn type="integer" id="S2.SS1.p3.1.m1.1.1.2.2.cmml" xref="S2.SS1.p3.1.m1.1.1.2.2">123</cn><cn type="integer" id="S2.SS1.p3.1.m1.1.1.2.3.cmml" xref="S2.SS1.p3.1.m1.1.1.2.3">456</cn></apply><cn type="integer" id="S2.SS1.p3.1.m1.1.1.3.cmml" xref="S2.SS1.p3.1.m1.1.1.3">56088</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.1.m1.1c">123\times 456=56088</annotation></semantics></math>”).
Therefore, we take a white-box approach that includes in the definition the internal computations of the LLM that convert these inputs to the related outputs.</p>
</div>
<div id="S2.SS1.p4" class="ltx_para">
<p id="S2.SS1.p4.1" class="ltx_p">Here are two preliminary examples of internal computations.</p>
</div>
<div id="Thmexample1" class="ltx_theorem ltx_theorem_example">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmexample1.1.1.1" class="ltx_text ltx_font_bold">Example 1</span></span><span id="Thmexample1.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmexample1.p1" class="ltx_para">
<p id="Thmexample1.p1.2" class="ltx_p">Several works have studied the underlying mechanisms when LLMs answer to the prompt “The capital of China is” with “Beijing”, as well as other factual questions <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib29" title="" class="ltx_ref">29</a>, <a href="#bib.bib46" title="" class="ltx_ref">46</a>, <a href="#bib.bib79" title="" class="ltx_ref">79</a>, <a href="#bib.bib22" title="" class="ltx_ref">22</a>]</cite>.
At least two mechanisms are involved, and the LLM may use their superposition <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib79" title="" class="ltx_ref">79</a>]</cite>.
One mechanism is to use general-purpose attention heads (called “mover heads”) to move “capital” and “China” to the last token “is”, and then use the MLP layers to map the feature of the last token to “Beijing” <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib79" title="" class="ltx_ref">79</a>]</cite>.
Often, only one or a few MLP neurons are causally relevant, and they are called “knowledge neurons” <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib29" title="" class="ltx_ref">29</a>]</cite>.
This mechanism is illustrated in Figure <a href="#F6" title="Figure 6 ‣ 2.1 | Preliminaries ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> (left).
Another mechanism involves attention heads <math id="Thmexample1.p1.1.m1.1" class="ltx_Math" alttext="h" display="inline"><semantics id="Thmexample1.p1.1.m1.1a"><mi id="Thmexample1.p1.1.m1.1.1" xref="Thmexample1.p1.1.m1.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="Thmexample1.p1.1.m1.1b"><ci id="Thmexample1.p1.1.m1.1.1.cmml" xref="Thmexample1.p1.1.m1.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmexample1.p1.1.m1.1c">h</annotation></semantics></math> whose value-to-output matrices <math id="Thmexample1.p1.2.m2.1" class="ltx_Math" alttext="W^{h}_{V}W^{h}_{O}" display="inline"><semantics id="Thmexample1.p1.2.m2.1a"><mrow id="Thmexample1.p1.2.m2.1.1" xref="Thmexample1.p1.2.m2.1.1.cmml"><msubsup id="Thmexample1.p1.2.m2.1.1.2" xref="Thmexample1.p1.2.m2.1.1.2.cmml"><mi id="Thmexample1.p1.2.m2.1.1.2.2.2" xref="Thmexample1.p1.2.m2.1.1.2.2.2.cmml">W</mi><mi id="Thmexample1.p1.2.m2.1.1.2.3" xref="Thmexample1.p1.2.m2.1.1.2.3.cmml">V</mi><mi id="Thmexample1.p1.2.m2.1.1.2.2.3" xref="Thmexample1.p1.2.m2.1.1.2.2.3.cmml">h</mi></msubsup><mo lspace="0em" rspace="0em" id="Thmexample1.p1.2.m2.1.1.1" xref="Thmexample1.p1.2.m2.1.1.1.cmml">​</mo><msubsup id="Thmexample1.p1.2.m2.1.1.3" xref="Thmexample1.p1.2.m2.1.1.3.cmml"><mi id="Thmexample1.p1.2.m2.1.1.3.2.2" xref="Thmexample1.p1.2.m2.1.1.3.2.2.cmml">W</mi><mi id="Thmexample1.p1.2.m2.1.1.3.3" xref="Thmexample1.p1.2.m2.1.1.3.3.cmml">O</mi><mi id="Thmexample1.p1.2.m2.1.1.3.2.3" xref="Thmexample1.p1.2.m2.1.1.3.2.3.cmml">h</mi></msubsup></mrow><annotation-xml encoding="MathML-Content" id="Thmexample1.p1.2.m2.1b"><apply id="Thmexample1.p1.2.m2.1.1.cmml" xref="Thmexample1.p1.2.m2.1.1"><times id="Thmexample1.p1.2.m2.1.1.1.cmml" xref="Thmexample1.p1.2.m2.1.1.1"></times><apply id="Thmexample1.p1.2.m2.1.1.2.cmml" xref="Thmexample1.p1.2.m2.1.1.2"><csymbol cd="ambiguous" id="Thmexample1.p1.2.m2.1.1.2.1.cmml" xref="Thmexample1.p1.2.m2.1.1.2">subscript</csymbol><apply id="Thmexample1.p1.2.m2.1.1.2.2.cmml" xref="Thmexample1.p1.2.m2.1.1.2"><csymbol cd="ambiguous" id="Thmexample1.p1.2.m2.1.1.2.2.1.cmml" xref="Thmexample1.p1.2.m2.1.1.2">superscript</csymbol><ci id="Thmexample1.p1.2.m2.1.1.2.2.2.cmml" xref="Thmexample1.p1.2.m2.1.1.2.2.2">𝑊</ci><ci id="Thmexample1.p1.2.m2.1.1.2.2.3.cmml" xref="Thmexample1.p1.2.m2.1.1.2.2.3">ℎ</ci></apply><ci id="Thmexample1.p1.2.m2.1.1.2.3.cmml" xref="Thmexample1.p1.2.m2.1.1.2.3">𝑉</ci></apply><apply id="Thmexample1.p1.2.m2.1.1.3.cmml" xref="Thmexample1.p1.2.m2.1.1.3"><csymbol cd="ambiguous" id="Thmexample1.p1.2.m2.1.1.3.1.cmml" xref="Thmexample1.p1.2.m2.1.1.3">subscript</csymbol><apply id="Thmexample1.p1.2.m2.1.1.3.2.cmml" xref="Thmexample1.p1.2.m2.1.1.3"><csymbol cd="ambiguous" id="Thmexample1.p1.2.m2.1.1.3.2.1.cmml" xref="Thmexample1.p1.2.m2.1.1.3">superscript</csymbol><ci id="Thmexample1.p1.2.m2.1.1.3.2.2.cmml" xref="Thmexample1.p1.2.m2.1.1.3.2.2">𝑊</ci><ci id="Thmexample1.p1.2.m2.1.1.3.2.3.cmml" xref="Thmexample1.p1.2.m2.1.1.3.2.3">ℎ</ci></apply><ci id="Thmexample1.p1.2.m2.1.1.3.3.cmml" xref="Thmexample1.p1.2.m2.1.1.3.3">𝑂</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmexample1.p1.2.m2.1c">W^{h}_{V}W^{h}_{O}</annotation></semantics></math> function like bigrams,
e.g. mapping “captial” to {“Paris”, “Beijing”, …} and “China” to {“panda”, “Beijing”, …} , which sum up to produce “Beijing” <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib22" title="" class="ltx_ref">22</a>, <a href="#bib.bib46" title="" class="ltx_ref">46</a>, <a href="#bib.bib79" title="" class="ltx_ref">79</a>]</cite>.
This mechanism is illustrated in Figure <a href="#F6" title="Figure 6 ‣ 2.1 | Preliminaries ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> (middle).</p>
</div>
</div>
<div id="Thmexample2" class="ltx_theorem ltx_theorem_example">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmexample2.1.1.1" class="ltx_text ltx_font_bold">Example 2</span></span><span id="Thmexample2.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmexample2.p1" class="ltx_para">
<p id="Thmexample2.p1.1" class="ltx_p">The ability of LLMs to perform “search, copy and paste”, namely answering to the context “…[a][b]…[a]” with “[b]”, is based on two attention heads, together called induction heads <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib85" title="" class="ltx_ref">85</a>]</cite>.
The first head copies the feature of the previous token, enabling [b] to “dress like” its previous token [a].
The second head searches for similar features, enabling the second [a] to attend to [b], which now has the appearance of [a].
Thereby, the last token [a] manages to retrieve the feature of [b] and to output [b].
This mechanism is illustrated in Figure <a href="#F6" title="Figure 6 ‣ 2.1 | Preliminaries ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> (right).
A similar mechanism is found for in-context learning <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib116" title="" class="ltx_ref">116</a>]</cite>.</p>
</div>
</div>
<figure id="F6" class="ltx_figure"><img src="/html/2407.01178/assets/Figures/Theory/classical_circuits_demo.png" id="F6.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="509" height="231" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="F6.2.1.1" class="ltx_text" style="font-size:90%;">Figure 6</span>: </span><span id="F6.3.2" class="ltx_text" style="font-size:90%;">Illustration of three subgraphs.
Left: A subgraph that inputs “the capital of China is” and outputs “Beijing”.
The knowledge neuron is marked in red and the mover heads in green.
Middle: Another subgraph with similar function using task-specific heads.
Right: The induction-heads subgraph that inputs “[a][b]…[a]” and outputs [b], where [a], [b] are arbitrary tokens.
The notations are introduced in Section <a href="#S2.SS2" title="2.2 | Knowledge ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.2</span></a>.
The locations of these attention heads and MLP neurons may be variable.</span></figcaption>
</figure>
<div id="S2.SS1.p5" class="ltx_para">
<p id="S2.SS1.p5.1" class="ltx_p">We will address the internal mechanism for an input-output relation as a circuit, and will define a piece of knowledge as an input-output relation plus its circuit.
By manipulating these circuits, one can separate many pieces of knowledge from a LLM while keeping its function intact.</p>
</div>
<div id="S2.SS1.p6" class="ltx_para">
<p id="S2.SS1.p6.1" class="ltx_p">Recent works on circuit discovery demonstrate that some knowledge and skills possessed by Transformer LLMs can be identified with patterns in their computation graphs <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib85" title="" class="ltx_ref">85</a>, <a href="#bib.bib116" title="" class="ltx_ref">116</a>, <a href="#bib.bib106" title="" class="ltx_ref">106</a>, <a href="#bib.bib45" title="" class="ltx_ref">45</a>, <a href="#bib.bib115" title="" class="ltx_ref">115</a>, <a href="#bib.bib24" title="" class="ltx_ref">24</a>, <a href="#bib.bib29" title="" class="ltx_ref">29</a>, <a href="#bib.bib46" title="" class="ltx_ref">46</a>]</cite>, but there has not been a universally accepted definition of circuit.
Different from works on Boolean circuits <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib50" title="" class="ltx_ref">50</a>, <a href="#bib.bib80" title="" class="ltx_ref">80</a>]</cite> and circuits with Transformer submodules as their nodes <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib24" title="" class="ltx_ref">24</a>, <a href="#bib.bib129" title="" class="ltx_ref">129</a>]</cite>, we characterize a circuit as a “spatial-temporal” phenomenon, whose causal structure is localized at the right places (MLP neurons and attention heads) and right times (tokens).
Thus, we define a computation graph as a directed acyclic graph, whose nodes are the hidden features of all tokens at all all MLP and attention layers, and whose edges correspond to all activations inside these layers.
In particular, the computation graph hosts one copy of the Transformer architecture at each time step.
To transcend this phenomenological characterization, we define a circuit as an equivalence class of similar subgraphs across multiple computation graphs.</p>
</div>
<div id="S2.SS1.p7" class="ltx_para">
<p id="S2.SS1.p7.1" class="ltx_p">As a remark, it is conceptually feasible to identify a circuit with the minimal subset of Transformer parameters that causes this circuit.
The benefit is that such definition of knowledge seems more intrinsic to the LLM.
Nevertheless, with the current definition, it is easier to perform surgery on the circuits and derive constructive proofs.
Besides, it is known that Transformer submodules exhibit superposition or polysemanticity, such that one MLP neuron or attention head may serve multiple distinct functions <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib38" title="" class="ltx_ref">38</a>, <a href="#bib.bib79" title="" class="ltx_ref">79</a>]</cite>, making the identification of parameter subsets a challenge task.</p>
</div>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsection" style="font-size:120%;color:#00868B;">2.2 <span id="S2.SS2.1.1" class="ltx_text" style="color:#808080;">| </span>Knowledge</h3>

<div id="S2.SS2.p1" class="ltx_para">
<p id="S2.SS2.p1.6" class="ltx_p">We begin with the definition of the knowledge of LLMs.
For now, it suffices to adopt heuristic definitions instead of fully rigorous ones.
Throughout this section, by LLM we mean autoregressive Transformer LLM that has at least been pretrained.
Let <math id="S2.SS2.p1.1.m1.1" class="ltx_Math" alttext="L" display="inline"><semantics id="S2.SS2.p1.1.m1.1a"><mi id="S2.SS2.p1.1.m1.1.1" xref="S2.SS2.p1.1.m1.1.1.cmml">L</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.1.m1.1b"><ci id="S2.SS2.p1.1.m1.1.1.cmml" xref="S2.SS2.p1.1.m1.1.1">𝐿</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.1.m1.1c">L</annotation></semantics></math> be the number of Transformer blocks and <math id="S2.SS2.p1.2.m2.1" class="ltx_Math" alttext="H" display="inline"><semantics id="S2.SS2.p1.2.m2.1a"><mi id="S2.SS2.p1.2.m2.1.1" xref="S2.SS2.p1.2.m2.1.1.cmml">H</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.2.m2.1b"><ci id="S2.SS2.p1.2.m2.1.1.cmml" xref="S2.SS2.p1.2.m2.1.1">𝐻</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.2.m2.1c">H</annotation></semantics></math> be the number of attention heads at each attention layer, and the blocks and heads are numbered by <math id="S2.SS2.p1.3.m3.2" class="ltx_Math" alttext="l=0,\dots L-1" display="inline"><semantics id="S2.SS2.p1.3.m3.2a"><mrow id="S2.SS2.p1.3.m3.2.2" xref="S2.SS2.p1.3.m3.2.2.cmml"><mi id="S2.SS2.p1.3.m3.2.2.3" xref="S2.SS2.p1.3.m3.2.2.3.cmml">l</mi><mo id="S2.SS2.p1.3.m3.2.2.2" xref="S2.SS2.p1.3.m3.2.2.2.cmml">=</mo><mrow id="S2.SS2.p1.3.m3.2.2.1.1" xref="S2.SS2.p1.3.m3.2.2.1.2.cmml"><mn id="S2.SS2.p1.3.m3.1.1" xref="S2.SS2.p1.3.m3.1.1.cmml">0</mn><mo id="S2.SS2.p1.3.m3.2.2.1.1.2" xref="S2.SS2.p1.3.m3.2.2.1.2.cmml">,</mo><mrow id="S2.SS2.p1.3.m3.2.2.1.1.1" xref="S2.SS2.p1.3.m3.2.2.1.1.1.cmml"><mrow id="S2.SS2.p1.3.m3.2.2.1.1.1.2" xref="S2.SS2.p1.3.m3.2.2.1.1.1.2.cmml"><mi mathvariant="normal" id="S2.SS2.p1.3.m3.2.2.1.1.1.2.2" xref="S2.SS2.p1.3.m3.2.2.1.1.1.2.2.cmml">…</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p1.3.m3.2.2.1.1.1.2.1" xref="S2.SS2.p1.3.m3.2.2.1.1.1.2.1.cmml">​</mo><mi id="S2.SS2.p1.3.m3.2.2.1.1.1.2.3" xref="S2.SS2.p1.3.m3.2.2.1.1.1.2.3.cmml">L</mi></mrow><mo id="S2.SS2.p1.3.m3.2.2.1.1.1.1" xref="S2.SS2.p1.3.m3.2.2.1.1.1.1.cmml">−</mo><mn id="S2.SS2.p1.3.m3.2.2.1.1.1.3" xref="S2.SS2.p1.3.m3.2.2.1.1.1.3.cmml">1</mn></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.3.m3.2b"><apply id="S2.SS2.p1.3.m3.2.2.cmml" xref="S2.SS2.p1.3.m3.2.2"><eq id="S2.SS2.p1.3.m3.2.2.2.cmml" xref="S2.SS2.p1.3.m3.2.2.2"></eq><ci id="S2.SS2.p1.3.m3.2.2.3.cmml" xref="S2.SS2.p1.3.m3.2.2.3">𝑙</ci><list id="S2.SS2.p1.3.m3.2.2.1.2.cmml" xref="S2.SS2.p1.3.m3.2.2.1.1"><cn type="integer" id="S2.SS2.p1.3.m3.1.1.cmml" xref="S2.SS2.p1.3.m3.1.1">0</cn><apply id="S2.SS2.p1.3.m3.2.2.1.1.1.cmml" xref="S2.SS2.p1.3.m3.2.2.1.1.1"><minus id="S2.SS2.p1.3.m3.2.2.1.1.1.1.cmml" xref="S2.SS2.p1.3.m3.2.2.1.1.1.1"></minus><apply id="S2.SS2.p1.3.m3.2.2.1.1.1.2.cmml" xref="S2.SS2.p1.3.m3.2.2.1.1.1.2"><times id="S2.SS2.p1.3.m3.2.2.1.1.1.2.1.cmml" xref="S2.SS2.p1.3.m3.2.2.1.1.1.2.1"></times><ci id="S2.SS2.p1.3.m3.2.2.1.1.1.2.2.cmml" xref="S2.SS2.p1.3.m3.2.2.1.1.1.2.2">…</ci><ci id="S2.SS2.p1.3.m3.2.2.1.1.1.2.3.cmml" xref="S2.SS2.p1.3.m3.2.2.1.1.1.2.3">𝐿</ci></apply><cn type="integer" id="S2.SS2.p1.3.m3.2.2.1.1.1.3.cmml" xref="S2.SS2.p1.3.m3.2.2.1.1.1.3">1</cn></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.3.m3.2c">l=0,\dots L-1</annotation></semantics></math> and <math id="S2.SS2.p1.4.m4.2" class="ltx_Math" alttext="h=0,\dots H-1" display="inline"><semantics id="S2.SS2.p1.4.m4.2a"><mrow id="S2.SS2.p1.4.m4.2.2" xref="S2.SS2.p1.4.m4.2.2.cmml"><mi id="S2.SS2.p1.4.m4.2.2.3" xref="S2.SS2.p1.4.m4.2.2.3.cmml">h</mi><mo id="S2.SS2.p1.4.m4.2.2.2" xref="S2.SS2.p1.4.m4.2.2.2.cmml">=</mo><mrow id="S2.SS2.p1.4.m4.2.2.1.1" xref="S2.SS2.p1.4.m4.2.2.1.2.cmml"><mn id="S2.SS2.p1.4.m4.1.1" xref="S2.SS2.p1.4.m4.1.1.cmml">0</mn><mo id="S2.SS2.p1.4.m4.2.2.1.1.2" xref="S2.SS2.p1.4.m4.2.2.1.2.cmml">,</mo><mrow id="S2.SS2.p1.4.m4.2.2.1.1.1" xref="S2.SS2.p1.4.m4.2.2.1.1.1.cmml"><mrow id="S2.SS2.p1.4.m4.2.2.1.1.1.2" xref="S2.SS2.p1.4.m4.2.2.1.1.1.2.cmml"><mi mathvariant="normal" id="S2.SS2.p1.4.m4.2.2.1.1.1.2.2" xref="S2.SS2.p1.4.m4.2.2.1.1.1.2.2.cmml">…</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p1.4.m4.2.2.1.1.1.2.1" xref="S2.SS2.p1.4.m4.2.2.1.1.1.2.1.cmml">​</mo><mi id="S2.SS2.p1.4.m4.2.2.1.1.1.2.3" xref="S2.SS2.p1.4.m4.2.2.1.1.1.2.3.cmml">H</mi></mrow><mo id="S2.SS2.p1.4.m4.2.2.1.1.1.1" xref="S2.SS2.p1.4.m4.2.2.1.1.1.1.cmml">−</mo><mn id="S2.SS2.p1.4.m4.2.2.1.1.1.3" xref="S2.SS2.p1.4.m4.2.2.1.1.1.3.cmml">1</mn></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.4.m4.2b"><apply id="S2.SS2.p1.4.m4.2.2.cmml" xref="S2.SS2.p1.4.m4.2.2"><eq id="S2.SS2.p1.4.m4.2.2.2.cmml" xref="S2.SS2.p1.4.m4.2.2.2"></eq><ci id="S2.SS2.p1.4.m4.2.2.3.cmml" xref="S2.SS2.p1.4.m4.2.2.3">ℎ</ci><list id="S2.SS2.p1.4.m4.2.2.1.2.cmml" xref="S2.SS2.p1.4.m4.2.2.1.1"><cn type="integer" id="S2.SS2.p1.4.m4.1.1.cmml" xref="S2.SS2.p1.4.m4.1.1">0</cn><apply id="S2.SS2.p1.4.m4.2.2.1.1.1.cmml" xref="S2.SS2.p1.4.m4.2.2.1.1.1"><minus id="S2.SS2.p1.4.m4.2.2.1.1.1.1.cmml" xref="S2.SS2.p1.4.m4.2.2.1.1.1.1"></minus><apply id="S2.SS2.p1.4.m4.2.2.1.1.1.2.cmml" xref="S2.SS2.p1.4.m4.2.2.1.1.1.2"><times id="S2.SS2.p1.4.m4.2.2.1.1.1.2.1.cmml" xref="S2.SS2.p1.4.m4.2.2.1.1.1.2.1"></times><ci id="S2.SS2.p1.4.m4.2.2.1.1.1.2.2.cmml" xref="S2.SS2.p1.4.m4.2.2.1.1.1.2.2">…</ci><ci id="S2.SS2.p1.4.m4.2.2.1.1.1.2.3.cmml" xref="S2.SS2.p1.4.m4.2.2.1.1.1.2.3">𝐻</ci></apply><cn type="integer" id="S2.SS2.p1.4.m4.2.2.1.1.1.3.cmml" xref="S2.SS2.p1.4.m4.2.2.1.1.1.3">1</cn></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.4.m4.2c">h=0,\dots H-1</annotation></semantics></math>.
There are in total <math id="S2.SS2.p1.5.m5.1" class="ltx_Math" alttext="2L" display="inline"><semantics id="S2.SS2.p1.5.m5.1a"><mrow id="S2.SS2.p1.5.m5.1.1" xref="S2.SS2.p1.5.m5.1.1.cmml"><mn id="S2.SS2.p1.5.m5.1.1.2" xref="S2.SS2.p1.5.m5.1.1.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S2.SS2.p1.5.m5.1.1.1" xref="S2.SS2.p1.5.m5.1.1.1.cmml">​</mo><mi id="S2.SS2.p1.5.m5.1.1.3" xref="S2.SS2.p1.5.m5.1.1.3.cmml">L</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.5.m5.1b"><apply id="S2.SS2.p1.5.m5.1.1.cmml" xref="S2.SS2.p1.5.m5.1.1"><times id="S2.SS2.p1.5.m5.1.1.1.cmml" xref="S2.SS2.p1.5.m5.1.1.1"></times><cn type="integer" id="S2.SS2.p1.5.m5.1.1.2.cmml" xref="S2.SS2.p1.5.m5.1.1.2">2</cn><ci id="S2.SS2.p1.5.m5.1.1.3.cmml" xref="S2.SS2.p1.5.m5.1.1.3">𝐿</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.5.m5.1c">2L</annotation></semantics></math> layers (MLP layers and attention layers), and the input features to these layers are numbered by <math id="S2.SS2.p1.6.m6.2" class="ltx_Math" alttext="0,\dots 2L-1" display="inline"><semantics id="S2.SS2.p1.6.m6.2a"><mrow id="S2.SS2.p1.6.m6.2.2.1" xref="S2.SS2.p1.6.m6.2.2.2.cmml"><mn id="S2.SS2.p1.6.m6.1.1" xref="S2.SS2.p1.6.m6.1.1.cmml">0</mn><mo id="S2.SS2.p1.6.m6.2.2.1.2" xref="S2.SS2.p1.6.m6.2.2.2.cmml">,</mo><mrow id="S2.SS2.p1.6.m6.2.2.1.1" xref="S2.SS2.p1.6.m6.2.2.1.1.cmml"><mrow id="S2.SS2.p1.6.m6.2.2.1.1.2" xref="S2.SS2.p1.6.m6.2.2.1.1.2.cmml"><mi mathvariant="normal" id="S2.SS2.p1.6.m6.2.2.1.1.2.2" xref="S2.SS2.p1.6.m6.2.2.1.1.2.2.cmml">…</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p1.6.m6.2.2.1.1.2.1" xref="S2.SS2.p1.6.m6.2.2.1.1.2.1.cmml">​</mo><mn id="S2.SS2.p1.6.m6.2.2.1.1.2.3" xref="S2.SS2.p1.6.m6.2.2.1.1.2.3.cmml">2</mn><mo lspace="0em" rspace="0em" id="S2.SS2.p1.6.m6.2.2.1.1.2.1a" xref="S2.SS2.p1.6.m6.2.2.1.1.2.1.cmml">​</mo><mi id="S2.SS2.p1.6.m6.2.2.1.1.2.4" xref="S2.SS2.p1.6.m6.2.2.1.1.2.4.cmml">L</mi></mrow><mo id="S2.SS2.p1.6.m6.2.2.1.1.1" xref="S2.SS2.p1.6.m6.2.2.1.1.1.cmml">−</mo><mn id="S2.SS2.p1.6.m6.2.2.1.1.3" xref="S2.SS2.p1.6.m6.2.2.1.1.3.cmml">1</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.6.m6.2b"><list id="S2.SS2.p1.6.m6.2.2.2.cmml" xref="S2.SS2.p1.6.m6.2.2.1"><cn type="integer" id="S2.SS2.p1.6.m6.1.1.cmml" xref="S2.SS2.p1.6.m6.1.1">0</cn><apply id="S2.SS2.p1.6.m6.2.2.1.1.cmml" xref="S2.SS2.p1.6.m6.2.2.1.1"><minus id="S2.SS2.p1.6.m6.2.2.1.1.1.cmml" xref="S2.SS2.p1.6.m6.2.2.1.1.1"></minus><apply id="S2.SS2.p1.6.m6.2.2.1.1.2.cmml" xref="S2.SS2.p1.6.m6.2.2.1.1.2"><times id="S2.SS2.p1.6.m6.2.2.1.1.2.1.cmml" xref="S2.SS2.p1.6.m6.2.2.1.1.2.1"></times><ci id="S2.SS2.p1.6.m6.2.2.1.1.2.2.cmml" xref="S2.SS2.p1.6.m6.2.2.1.1.2.2">…</ci><cn type="integer" id="S2.SS2.p1.6.m6.2.2.1.1.2.3.cmml" xref="S2.SS2.p1.6.m6.2.2.1.1.2.3">2</cn><ci id="S2.SS2.p1.6.m6.2.2.1.1.2.4.cmml" xref="S2.SS2.p1.6.m6.2.2.1.1.2.4">𝐿</ci></apply><cn type="integer" id="S2.SS2.p1.6.m6.2.2.1.1.3.cmml" xref="S2.SS2.p1.6.m6.2.2.1.1.3">1</cn></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.6.m6.2c">0,\dots 2L-1</annotation></semantics></math>.</p>
</div>
<div id="Thmdefinition1" class="ltx_theorem ltx_theorem_definition">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmdefinition1.1.1.1" class="ltx_text ltx_font_bold">Definition 1</span></span><span id="Thmdefinition1.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmdefinition1.p1" class="ltx_para">
<p id="Thmdefinition1.p1.4" class="ltx_p">Given an LLM and a text <math id="Thmdefinition1.p1.1.m1.2" class="ltx_Math" alttext="\mathbf{t}=(t_{0},\dots t_{n})" display="inline"><semantics id="Thmdefinition1.p1.1.m1.2a"><mrow id="Thmdefinition1.p1.1.m1.2.2" xref="Thmdefinition1.p1.1.m1.2.2.cmml"><mi id="Thmdefinition1.p1.1.m1.2.2.4" xref="Thmdefinition1.p1.1.m1.2.2.4.cmml">𝐭</mi><mo id="Thmdefinition1.p1.1.m1.2.2.3" xref="Thmdefinition1.p1.1.m1.2.2.3.cmml">=</mo><mrow id="Thmdefinition1.p1.1.m1.2.2.2.2" xref="Thmdefinition1.p1.1.m1.2.2.2.3.cmml"><mo stretchy="false" id="Thmdefinition1.p1.1.m1.2.2.2.2.3" xref="Thmdefinition1.p1.1.m1.2.2.2.3.cmml">(</mo><msub id="Thmdefinition1.p1.1.m1.1.1.1.1.1" xref="Thmdefinition1.p1.1.m1.1.1.1.1.1.cmml"><mi id="Thmdefinition1.p1.1.m1.1.1.1.1.1.2" xref="Thmdefinition1.p1.1.m1.1.1.1.1.1.2.cmml">t</mi><mn id="Thmdefinition1.p1.1.m1.1.1.1.1.1.3" xref="Thmdefinition1.p1.1.m1.1.1.1.1.1.3.cmml">0</mn></msub><mo id="Thmdefinition1.p1.1.m1.2.2.2.2.4" xref="Thmdefinition1.p1.1.m1.2.2.2.3.cmml">,</mo><mrow id="Thmdefinition1.p1.1.m1.2.2.2.2.2" xref="Thmdefinition1.p1.1.m1.2.2.2.2.2.cmml"><mi mathvariant="normal" id="Thmdefinition1.p1.1.m1.2.2.2.2.2.2" xref="Thmdefinition1.p1.1.m1.2.2.2.2.2.2.cmml">…</mi><mo lspace="0em" rspace="0em" id="Thmdefinition1.p1.1.m1.2.2.2.2.2.1" xref="Thmdefinition1.p1.1.m1.2.2.2.2.2.1.cmml">​</mo><msub id="Thmdefinition1.p1.1.m1.2.2.2.2.2.3" xref="Thmdefinition1.p1.1.m1.2.2.2.2.2.3.cmml"><mi id="Thmdefinition1.p1.1.m1.2.2.2.2.2.3.2" xref="Thmdefinition1.p1.1.m1.2.2.2.2.2.3.2.cmml">t</mi><mi id="Thmdefinition1.p1.1.m1.2.2.2.2.2.3.3" xref="Thmdefinition1.p1.1.m1.2.2.2.2.2.3.3.cmml">n</mi></msub></mrow><mo stretchy="false" id="Thmdefinition1.p1.1.m1.2.2.2.2.5" xref="Thmdefinition1.p1.1.m1.2.2.2.3.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="Thmdefinition1.p1.1.m1.2b"><apply id="Thmdefinition1.p1.1.m1.2.2.cmml" xref="Thmdefinition1.p1.1.m1.2.2"><eq id="Thmdefinition1.p1.1.m1.2.2.3.cmml" xref="Thmdefinition1.p1.1.m1.2.2.3"></eq><ci id="Thmdefinition1.p1.1.m1.2.2.4.cmml" xref="Thmdefinition1.p1.1.m1.2.2.4">𝐭</ci><interval closure="open" id="Thmdefinition1.p1.1.m1.2.2.2.3.cmml" xref="Thmdefinition1.p1.1.m1.2.2.2.2"><apply id="Thmdefinition1.p1.1.m1.1.1.1.1.1.cmml" xref="Thmdefinition1.p1.1.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="Thmdefinition1.p1.1.m1.1.1.1.1.1.1.cmml" xref="Thmdefinition1.p1.1.m1.1.1.1.1.1">subscript</csymbol><ci id="Thmdefinition1.p1.1.m1.1.1.1.1.1.2.cmml" xref="Thmdefinition1.p1.1.m1.1.1.1.1.1.2">𝑡</ci><cn type="integer" id="Thmdefinition1.p1.1.m1.1.1.1.1.1.3.cmml" xref="Thmdefinition1.p1.1.m1.1.1.1.1.1.3">0</cn></apply><apply id="Thmdefinition1.p1.1.m1.2.2.2.2.2.cmml" xref="Thmdefinition1.p1.1.m1.2.2.2.2.2"><times id="Thmdefinition1.p1.1.m1.2.2.2.2.2.1.cmml" xref="Thmdefinition1.p1.1.m1.2.2.2.2.2.1"></times><ci id="Thmdefinition1.p1.1.m1.2.2.2.2.2.2.cmml" xref="Thmdefinition1.p1.1.m1.2.2.2.2.2.2">…</ci><apply id="Thmdefinition1.p1.1.m1.2.2.2.2.2.3.cmml" xref="Thmdefinition1.p1.1.m1.2.2.2.2.2.3"><csymbol cd="ambiguous" id="Thmdefinition1.p1.1.m1.2.2.2.2.2.3.1.cmml" xref="Thmdefinition1.p1.1.m1.2.2.2.2.2.3">subscript</csymbol><ci id="Thmdefinition1.p1.1.m1.2.2.2.2.2.3.2.cmml" xref="Thmdefinition1.p1.1.m1.2.2.2.2.2.3.2">𝑡</ci><ci id="Thmdefinition1.p1.1.m1.2.2.2.2.2.3.3.cmml" xref="Thmdefinition1.p1.1.m1.2.2.2.2.2.3.3">𝑛</ci></apply></apply></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition1.p1.1.m1.2c">\mathbf{t}=(t_{0},\dots t_{n})</annotation></semantics></math>, the <span id="Thmdefinition1.p1.4.1" class="ltx_text ltx_font_bold">computation graph</span> <math id="Thmdefinition1.p1.2.m2.1" class="ltx_Math" alttext="G" display="inline"><semantics id="Thmdefinition1.p1.2.m2.1a"><mi id="Thmdefinition1.p1.2.m2.1.1" xref="Thmdefinition1.p1.2.m2.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="Thmdefinition1.p1.2.m2.1b"><ci id="Thmdefinition1.p1.2.m2.1.1.cmml" xref="Thmdefinition1.p1.2.m2.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition1.p1.2.m2.1c">G</annotation></semantics></math> on input <math id="Thmdefinition1.p1.3.m3.2" class="ltx_Math" alttext="(t_{0},\dots t_{n-1})" display="inline"><semantics id="Thmdefinition1.p1.3.m3.2a"><mrow id="Thmdefinition1.p1.3.m3.2.2.2" xref="Thmdefinition1.p1.3.m3.2.2.3.cmml"><mo stretchy="false" id="Thmdefinition1.p1.3.m3.2.2.2.3" xref="Thmdefinition1.p1.3.m3.2.2.3.cmml">(</mo><msub id="Thmdefinition1.p1.3.m3.1.1.1.1" xref="Thmdefinition1.p1.3.m3.1.1.1.1.cmml"><mi id="Thmdefinition1.p1.3.m3.1.1.1.1.2" xref="Thmdefinition1.p1.3.m3.1.1.1.1.2.cmml">t</mi><mn id="Thmdefinition1.p1.3.m3.1.1.1.1.3" xref="Thmdefinition1.p1.3.m3.1.1.1.1.3.cmml">0</mn></msub><mo id="Thmdefinition1.p1.3.m3.2.2.2.4" xref="Thmdefinition1.p1.3.m3.2.2.3.cmml">,</mo><mrow id="Thmdefinition1.p1.3.m3.2.2.2.2" xref="Thmdefinition1.p1.3.m3.2.2.2.2.cmml"><mi mathvariant="normal" id="Thmdefinition1.p1.3.m3.2.2.2.2.2" xref="Thmdefinition1.p1.3.m3.2.2.2.2.2.cmml">…</mi><mo lspace="0em" rspace="0em" id="Thmdefinition1.p1.3.m3.2.2.2.2.1" xref="Thmdefinition1.p1.3.m3.2.2.2.2.1.cmml">​</mo><msub id="Thmdefinition1.p1.3.m3.2.2.2.2.3" xref="Thmdefinition1.p1.3.m3.2.2.2.2.3.cmml"><mi id="Thmdefinition1.p1.3.m3.2.2.2.2.3.2" xref="Thmdefinition1.p1.3.m3.2.2.2.2.3.2.cmml">t</mi><mrow id="Thmdefinition1.p1.3.m3.2.2.2.2.3.3" xref="Thmdefinition1.p1.3.m3.2.2.2.2.3.3.cmml"><mi id="Thmdefinition1.p1.3.m3.2.2.2.2.3.3.2" xref="Thmdefinition1.p1.3.m3.2.2.2.2.3.3.2.cmml">n</mi><mo id="Thmdefinition1.p1.3.m3.2.2.2.2.3.3.1" xref="Thmdefinition1.p1.3.m3.2.2.2.2.3.3.1.cmml">−</mo><mn id="Thmdefinition1.p1.3.m3.2.2.2.2.3.3.3" xref="Thmdefinition1.p1.3.m3.2.2.2.2.3.3.3.cmml">1</mn></mrow></msub></mrow><mo stretchy="false" id="Thmdefinition1.p1.3.m3.2.2.2.5" xref="Thmdefinition1.p1.3.m3.2.2.3.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="Thmdefinition1.p1.3.m3.2b"><interval closure="open" id="Thmdefinition1.p1.3.m3.2.2.3.cmml" xref="Thmdefinition1.p1.3.m3.2.2.2"><apply id="Thmdefinition1.p1.3.m3.1.1.1.1.cmml" xref="Thmdefinition1.p1.3.m3.1.1.1.1"><csymbol cd="ambiguous" id="Thmdefinition1.p1.3.m3.1.1.1.1.1.cmml" xref="Thmdefinition1.p1.3.m3.1.1.1.1">subscript</csymbol><ci id="Thmdefinition1.p1.3.m3.1.1.1.1.2.cmml" xref="Thmdefinition1.p1.3.m3.1.1.1.1.2">𝑡</ci><cn type="integer" id="Thmdefinition1.p1.3.m3.1.1.1.1.3.cmml" xref="Thmdefinition1.p1.3.m3.1.1.1.1.3">0</cn></apply><apply id="Thmdefinition1.p1.3.m3.2.2.2.2.cmml" xref="Thmdefinition1.p1.3.m3.2.2.2.2"><times id="Thmdefinition1.p1.3.m3.2.2.2.2.1.cmml" xref="Thmdefinition1.p1.3.m3.2.2.2.2.1"></times><ci id="Thmdefinition1.p1.3.m3.2.2.2.2.2.cmml" xref="Thmdefinition1.p1.3.m3.2.2.2.2.2">…</ci><apply id="Thmdefinition1.p1.3.m3.2.2.2.2.3.cmml" xref="Thmdefinition1.p1.3.m3.2.2.2.2.3"><csymbol cd="ambiguous" id="Thmdefinition1.p1.3.m3.2.2.2.2.3.1.cmml" xref="Thmdefinition1.p1.3.m3.2.2.2.2.3">subscript</csymbol><ci id="Thmdefinition1.p1.3.m3.2.2.2.2.3.2.cmml" xref="Thmdefinition1.p1.3.m3.2.2.2.2.3.2">𝑡</ci><apply id="Thmdefinition1.p1.3.m3.2.2.2.2.3.3.cmml" xref="Thmdefinition1.p1.3.m3.2.2.2.2.3.3"><minus id="Thmdefinition1.p1.3.m3.2.2.2.2.3.3.1.cmml" xref="Thmdefinition1.p1.3.m3.2.2.2.2.3.3.1"></minus><ci id="Thmdefinition1.p1.3.m3.2.2.2.2.3.3.2.cmml" xref="Thmdefinition1.p1.3.m3.2.2.2.2.3.3.2">𝑛</ci><cn type="integer" id="Thmdefinition1.p1.3.m3.2.2.2.2.3.3.3.cmml" xref="Thmdefinition1.p1.3.m3.2.2.2.2.3.3.3">1</cn></apply></apply></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition1.p1.3.m3.2c">(t_{0},\dots t_{n-1})</annotation></semantics></math> and target <math id="Thmdefinition1.p1.4.m4.2" class="ltx_Math" alttext="(t_{1},\dots t_{n})" display="inline"><semantics id="Thmdefinition1.p1.4.m4.2a"><mrow id="Thmdefinition1.p1.4.m4.2.2.2" xref="Thmdefinition1.p1.4.m4.2.2.3.cmml"><mo stretchy="false" id="Thmdefinition1.p1.4.m4.2.2.2.3" xref="Thmdefinition1.p1.4.m4.2.2.3.cmml">(</mo><msub id="Thmdefinition1.p1.4.m4.1.1.1.1" xref="Thmdefinition1.p1.4.m4.1.1.1.1.cmml"><mi id="Thmdefinition1.p1.4.m4.1.1.1.1.2" xref="Thmdefinition1.p1.4.m4.1.1.1.1.2.cmml">t</mi><mn id="Thmdefinition1.p1.4.m4.1.1.1.1.3" xref="Thmdefinition1.p1.4.m4.1.1.1.1.3.cmml">1</mn></msub><mo id="Thmdefinition1.p1.4.m4.2.2.2.4" xref="Thmdefinition1.p1.4.m4.2.2.3.cmml">,</mo><mrow id="Thmdefinition1.p1.4.m4.2.2.2.2" xref="Thmdefinition1.p1.4.m4.2.2.2.2.cmml"><mi mathvariant="normal" id="Thmdefinition1.p1.4.m4.2.2.2.2.2" xref="Thmdefinition1.p1.4.m4.2.2.2.2.2.cmml">…</mi><mo lspace="0em" rspace="0em" id="Thmdefinition1.p1.4.m4.2.2.2.2.1" xref="Thmdefinition1.p1.4.m4.2.2.2.2.1.cmml">​</mo><msub id="Thmdefinition1.p1.4.m4.2.2.2.2.3" xref="Thmdefinition1.p1.4.m4.2.2.2.2.3.cmml"><mi id="Thmdefinition1.p1.4.m4.2.2.2.2.3.2" xref="Thmdefinition1.p1.4.m4.2.2.2.2.3.2.cmml">t</mi><mi id="Thmdefinition1.p1.4.m4.2.2.2.2.3.3" xref="Thmdefinition1.p1.4.m4.2.2.2.2.3.3.cmml">n</mi></msub></mrow><mo stretchy="false" id="Thmdefinition1.p1.4.m4.2.2.2.5" xref="Thmdefinition1.p1.4.m4.2.2.3.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="Thmdefinition1.p1.4.m4.2b"><interval closure="open" id="Thmdefinition1.p1.4.m4.2.2.3.cmml" xref="Thmdefinition1.p1.4.m4.2.2.2"><apply id="Thmdefinition1.p1.4.m4.1.1.1.1.cmml" xref="Thmdefinition1.p1.4.m4.1.1.1.1"><csymbol cd="ambiguous" id="Thmdefinition1.p1.4.m4.1.1.1.1.1.cmml" xref="Thmdefinition1.p1.4.m4.1.1.1.1">subscript</csymbol><ci id="Thmdefinition1.p1.4.m4.1.1.1.1.2.cmml" xref="Thmdefinition1.p1.4.m4.1.1.1.1.2">𝑡</ci><cn type="integer" id="Thmdefinition1.p1.4.m4.1.1.1.1.3.cmml" xref="Thmdefinition1.p1.4.m4.1.1.1.1.3">1</cn></apply><apply id="Thmdefinition1.p1.4.m4.2.2.2.2.cmml" xref="Thmdefinition1.p1.4.m4.2.2.2.2"><times id="Thmdefinition1.p1.4.m4.2.2.2.2.1.cmml" xref="Thmdefinition1.p1.4.m4.2.2.2.2.1"></times><ci id="Thmdefinition1.p1.4.m4.2.2.2.2.2.cmml" xref="Thmdefinition1.p1.4.m4.2.2.2.2.2">…</ci><apply id="Thmdefinition1.p1.4.m4.2.2.2.2.3.cmml" xref="Thmdefinition1.p1.4.m4.2.2.2.2.3"><csymbol cd="ambiguous" id="Thmdefinition1.p1.4.m4.2.2.2.2.3.1.cmml" xref="Thmdefinition1.p1.4.m4.2.2.2.2.3">subscript</csymbol><ci id="Thmdefinition1.p1.4.m4.2.2.2.2.3.2.cmml" xref="Thmdefinition1.p1.4.m4.2.2.2.2.3.2">𝑡</ci><ci id="Thmdefinition1.p1.4.m4.2.2.2.2.3.3.cmml" xref="Thmdefinition1.p1.4.m4.2.2.2.2.3.3">𝑛</ci></apply></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition1.p1.4.m4.2c">(t_{1},\dots t_{n})</annotation></semantics></math> is a directed graph with weighted edges such that</p>
<ul id="S2.I2" class="ltx_itemize">
<li id="S2.I2.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S2.I2.i1.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S2.I2.i1.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S2.I2.i1.1.1.m1.1.1" xref="S2.I2.i1.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S2.I2.i1.1.1.m1.1c"><ci id="S2.I2.i1.1.1.m1.1.1.cmml" xref="S2.I2.i1.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i1.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S2.I2.i1.p1" class="ltx_para">
<p id="S2.I2.i1.p1.5" class="ltx_p">Its nodes consist of the hidden vectors <math id="S2.I2.i1.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{x}_{i}^{2l}" display="inline"><semantics id="S2.I2.i1.p1.1.m1.1a"><msubsup id="S2.I2.i1.p1.1.m1.1.1" xref="S2.I2.i1.p1.1.m1.1.1.cmml"><mi id="S2.I2.i1.p1.1.m1.1.1.2.2" xref="S2.I2.i1.p1.1.m1.1.1.2.2.cmml">𝐱</mi><mi id="S2.I2.i1.p1.1.m1.1.1.2.3" xref="S2.I2.i1.p1.1.m1.1.1.2.3.cmml">i</mi><mrow id="S2.I2.i1.p1.1.m1.1.1.3" xref="S2.I2.i1.p1.1.m1.1.1.3.cmml"><mn id="S2.I2.i1.p1.1.m1.1.1.3.2" xref="S2.I2.i1.p1.1.m1.1.1.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S2.I2.i1.p1.1.m1.1.1.3.1" xref="S2.I2.i1.p1.1.m1.1.1.3.1.cmml">​</mo><mi id="S2.I2.i1.p1.1.m1.1.1.3.3" xref="S2.I2.i1.p1.1.m1.1.1.3.3.cmml">l</mi></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S2.I2.i1.p1.1.m1.1b"><apply id="S2.I2.i1.p1.1.m1.1.1.cmml" xref="S2.I2.i1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.I2.i1.p1.1.m1.1.1.1.cmml" xref="S2.I2.i1.p1.1.m1.1.1">superscript</csymbol><apply id="S2.I2.i1.p1.1.m1.1.1.2.cmml" xref="S2.I2.i1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.I2.i1.p1.1.m1.1.1.2.1.cmml" xref="S2.I2.i1.p1.1.m1.1.1">subscript</csymbol><ci id="S2.I2.i1.p1.1.m1.1.1.2.2.cmml" xref="S2.I2.i1.p1.1.m1.1.1.2.2">𝐱</ci><ci id="S2.I2.i1.p1.1.m1.1.1.2.3.cmml" xref="S2.I2.i1.p1.1.m1.1.1.2.3">𝑖</ci></apply><apply id="S2.I2.i1.p1.1.m1.1.1.3.cmml" xref="S2.I2.i1.p1.1.m1.1.1.3"><times id="S2.I2.i1.p1.1.m1.1.1.3.1.cmml" xref="S2.I2.i1.p1.1.m1.1.1.3.1"></times><cn type="integer" id="S2.I2.i1.p1.1.m1.1.1.3.2.cmml" xref="S2.I2.i1.p1.1.m1.1.1.3.2">2</cn><ci id="S2.I2.i1.p1.1.m1.1.1.3.3.cmml" xref="S2.I2.i1.p1.1.m1.1.1.3.3">𝑙</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i1.p1.1.m1.1c">\mathbf{x}_{i}^{2l}</annotation></semantics></math> before all attention layers, the hidden vectors <math id="S2.I2.i1.p1.2.m2.1" class="ltx_Math" alttext="\mathbf{x}_{i}^{2l+1}" display="inline"><semantics id="S2.I2.i1.p1.2.m2.1a"><msubsup id="S2.I2.i1.p1.2.m2.1.1" xref="S2.I2.i1.p1.2.m2.1.1.cmml"><mi id="S2.I2.i1.p1.2.m2.1.1.2.2" xref="S2.I2.i1.p1.2.m2.1.1.2.2.cmml">𝐱</mi><mi id="S2.I2.i1.p1.2.m2.1.1.2.3" xref="S2.I2.i1.p1.2.m2.1.1.2.3.cmml">i</mi><mrow id="S2.I2.i1.p1.2.m2.1.1.3" xref="S2.I2.i1.p1.2.m2.1.1.3.cmml"><mrow id="S2.I2.i1.p1.2.m2.1.1.3.2" xref="S2.I2.i1.p1.2.m2.1.1.3.2.cmml"><mn id="S2.I2.i1.p1.2.m2.1.1.3.2.2" xref="S2.I2.i1.p1.2.m2.1.1.3.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S2.I2.i1.p1.2.m2.1.1.3.2.1" xref="S2.I2.i1.p1.2.m2.1.1.3.2.1.cmml">​</mo><mi id="S2.I2.i1.p1.2.m2.1.1.3.2.3" xref="S2.I2.i1.p1.2.m2.1.1.3.2.3.cmml">l</mi></mrow><mo id="S2.I2.i1.p1.2.m2.1.1.3.1" xref="S2.I2.i1.p1.2.m2.1.1.3.1.cmml">+</mo><mn id="S2.I2.i1.p1.2.m2.1.1.3.3" xref="S2.I2.i1.p1.2.m2.1.1.3.3.cmml">1</mn></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S2.I2.i1.p1.2.m2.1b"><apply id="S2.I2.i1.p1.2.m2.1.1.cmml" xref="S2.I2.i1.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S2.I2.i1.p1.2.m2.1.1.1.cmml" xref="S2.I2.i1.p1.2.m2.1.1">superscript</csymbol><apply id="S2.I2.i1.p1.2.m2.1.1.2.cmml" xref="S2.I2.i1.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S2.I2.i1.p1.2.m2.1.1.2.1.cmml" xref="S2.I2.i1.p1.2.m2.1.1">subscript</csymbol><ci id="S2.I2.i1.p1.2.m2.1.1.2.2.cmml" xref="S2.I2.i1.p1.2.m2.1.1.2.2">𝐱</ci><ci id="S2.I2.i1.p1.2.m2.1.1.2.3.cmml" xref="S2.I2.i1.p1.2.m2.1.1.2.3">𝑖</ci></apply><apply id="S2.I2.i1.p1.2.m2.1.1.3.cmml" xref="S2.I2.i1.p1.2.m2.1.1.3"><plus id="S2.I2.i1.p1.2.m2.1.1.3.1.cmml" xref="S2.I2.i1.p1.2.m2.1.1.3.1"></plus><apply id="S2.I2.i1.p1.2.m2.1.1.3.2.cmml" xref="S2.I2.i1.p1.2.m2.1.1.3.2"><times id="S2.I2.i1.p1.2.m2.1.1.3.2.1.cmml" xref="S2.I2.i1.p1.2.m2.1.1.3.2.1"></times><cn type="integer" id="S2.I2.i1.p1.2.m2.1.1.3.2.2.cmml" xref="S2.I2.i1.p1.2.m2.1.1.3.2.2">2</cn><ci id="S2.I2.i1.p1.2.m2.1.1.3.2.3.cmml" xref="S2.I2.i1.p1.2.m2.1.1.3.2.3">𝑙</ci></apply><cn type="integer" id="S2.I2.i1.p1.2.m2.1.1.3.3.cmml" xref="S2.I2.i1.p1.2.m2.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i1.p1.2.m2.1c">\mathbf{x}_{i}^{2l+1}</annotation></semantics></math> before all MLP layers, and the output vectors <math id="S2.I2.i1.p1.3.m3.1" class="ltx_Math" alttext="\mathbf{x}_{i}^{2L}" display="inline"><semantics id="S2.I2.i1.p1.3.m3.1a"><msubsup id="S2.I2.i1.p1.3.m3.1.1" xref="S2.I2.i1.p1.3.m3.1.1.cmml"><mi id="S2.I2.i1.p1.3.m3.1.1.2.2" xref="S2.I2.i1.p1.3.m3.1.1.2.2.cmml">𝐱</mi><mi id="S2.I2.i1.p1.3.m3.1.1.2.3" xref="S2.I2.i1.p1.3.m3.1.1.2.3.cmml">i</mi><mrow id="S2.I2.i1.p1.3.m3.1.1.3" xref="S2.I2.i1.p1.3.m3.1.1.3.cmml"><mn id="S2.I2.i1.p1.3.m3.1.1.3.2" xref="S2.I2.i1.p1.3.m3.1.1.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S2.I2.i1.p1.3.m3.1.1.3.1" xref="S2.I2.i1.p1.3.m3.1.1.3.1.cmml">​</mo><mi id="S2.I2.i1.p1.3.m3.1.1.3.3" xref="S2.I2.i1.p1.3.m3.1.1.3.3.cmml">L</mi></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S2.I2.i1.p1.3.m3.1b"><apply id="S2.I2.i1.p1.3.m3.1.1.cmml" xref="S2.I2.i1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S2.I2.i1.p1.3.m3.1.1.1.cmml" xref="S2.I2.i1.p1.3.m3.1.1">superscript</csymbol><apply id="S2.I2.i1.p1.3.m3.1.1.2.cmml" xref="S2.I2.i1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S2.I2.i1.p1.3.m3.1.1.2.1.cmml" xref="S2.I2.i1.p1.3.m3.1.1">subscript</csymbol><ci id="S2.I2.i1.p1.3.m3.1.1.2.2.cmml" xref="S2.I2.i1.p1.3.m3.1.1.2.2">𝐱</ci><ci id="S2.I2.i1.p1.3.m3.1.1.2.3.cmml" xref="S2.I2.i1.p1.3.m3.1.1.2.3">𝑖</ci></apply><apply id="S2.I2.i1.p1.3.m3.1.1.3.cmml" xref="S2.I2.i1.p1.3.m3.1.1.3"><times id="S2.I2.i1.p1.3.m3.1.1.3.1.cmml" xref="S2.I2.i1.p1.3.m3.1.1.3.1"></times><cn type="integer" id="S2.I2.i1.p1.3.m3.1.1.3.2.cmml" xref="S2.I2.i1.p1.3.m3.1.1.3.2">2</cn><ci id="S2.I2.i1.p1.3.m3.1.1.3.3.cmml" xref="S2.I2.i1.p1.3.m3.1.1.3.3">𝐿</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i1.p1.3.m3.1c">\mathbf{x}_{i}^{2L}</annotation></semantics></math>, for all blocks <math id="S2.I2.i1.p1.4.m4.2" class="ltx_Math" alttext="l=0,\dots L-1" display="inline"><semantics id="S2.I2.i1.p1.4.m4.2a"><mrow id="S2.I2.i1.p1.4.m4.2.2" xref="S2.I2.i1.p1.4.m4.2.2.cmml"><mi id="S2.I2.i1.p1.4.m4.2.2.3" xref="S2.I2.i1.p1.4.m4.2.2.3.cmml">l</mi><mo id="S2.I2.i1.p1.4.m4.2.2.2" xref="S2.I2.i1.p1.4.m4.2.2.2.cmml">=</mo><mrow id="S2.I2.i1.p1.4.m4.2.2.1.1" xref="S2.I2.i1.p1.4.m4.2.2.1.2.cmml"><mn id="S2.I2.i1.p1.4.m4.1.1" xref="S2.I2.i1.p1.4.m4.1.1.cmml">0</mn><mo id="S2.I2.i1.p1.4.m4.2.2.1.1.2" xref="S2.I2.i1.p1.4.m4.2.2.1.2.cmml">,</mo><mrow id="S2.I2.i1.p1.4.m4.2.2.1.1.1" xref="S2.I2.i1.p1.4.m4.2.2.1.1.1.cmml"><mrow id="S2.I2.i1.p1.4.m4.2.2.1.1.1.2" xref="S2.I2.i1.p1.4.m4.2.2.1.1.1.2.cmml"><mi mathvariant="normal" id="S2.I2.i1.p1.4.m4.2.2.1.1.1.2.2" xref="S2.I2.i1.p1.4.m4.2.2.1.1.1.2.2.cmml">…</mi><mo lspace="0em" rspace="0em" id="S2.I2.i1.p1.4.m4.2.2.1.1.1.2.1" xref="S2.I2.i1.p1.4.m4.2.2.1.1.1.2.1.cmml">​</mo><mi id="S2.I2.i1.p1.4.m4.2.2.1.1.1.2.3" xref="S2.I2.i1.p1.4.m4.2.2.1.1.1.2.3.cmml">L</mi></mrow><mo id="S2.I2.i1.p1.4.m4.2.2.1.1.1.1" xref="S2.I2.i1.p1.4.m4.2.2.1.1.1.1.cmml">−</mo><mn id="S2.I2.i1.p1.4.m4.2.2.1.1.1.3" xref="S2.I2.i1.p1.4.m4.2.2.1.1.1.3.cmml">1</mn></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.I2.i1.p1.4.m4.2b"><apply id="S2.I2.i1.p1.4.m4.2.2.cmml" xref="S2.I2.i1.p1.4.m4.2.2"><eq id="S2.I2.i1.p1.4.m4.2.2.2.cmml" xref="S2.I2.i1.p1.4.m4.2.2.2"></eq><ci id="S2.I2.i1.p1.4.m4.2.2.3.cmml" xref="S2.I2.i1.p1.4.m4.2.2.3">𝑙</ci><list id="S2.I2.i1.p1.4.m4.2.2.1.2.cmml" xref="S2.I2.i1.p1.4.m4.2.2.1.1"><cn type="integer" id="S2.I2.i1.p1.4.m4.1.1.cmml" xref="S2.I2.i1.p1.4.m4.1.1">0</cn><apply id="S2.I2.i1.p1.4.m4.2.2.1.1.1.cmml" xref="S2.I2.i1.p1.4.m4.2.2.1.1.1"><minus id="S2.I2.i1.p1.4.m4.2.2.1.1.1.1.cmml" xref="S2.I2.i1.p1.4.m4.2.2.1.1.1.1"></minus><apply id="S2.I2.i1.p1.4.m4.2.2.1.1.1.2.cmml" xref="S2.I2.i1.p1.4.m4.2.2.1.1.1.2"><times id="S2.I2.i1.p1.4.m4.2.2.1.1.1.2.1.cmml" xref="S2.I2.i1.p1.4.m4.2.2.1.1.1.2.1"></times><ci id="S2.I2.i1.p1.4.m4.2.2.1.1.1.2.2.cmml" xref="S2.I2.i1.p1.4.m4.2.2.1.1.1.2.2">…</ci><ci id="S2.I2.i1.p1.4.m4.2.2.1.1.1.2.3.cmml" xref="S2.I2.i1.p1.4.m4.2.2.1.1.1.2.3">𝐿</ci></apply><cn type="integer" id="S2.I2.i1.p1.4.m4.2.2.1.1.1.3.cmml" xref="S2.I2.i1.p1.4.m4.2.2.1.1.1.3">1</cn></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i1.p1.4.m4.2c">l=0,\dots L-1</annotation></semantics></math> and positions <math id="S2.I2.i1.p1.5.m5.2" class="ltx_Math" alttext="i=0,\dots n-1" display="inline"><semantics id="S2.I2.i1.p1.5.m5.2a"><mrow id="S2.I2.i1.p1.5.m5.2.2" xref="S2.I2.i1.p1.5.m5.2.2.cmml"><mi id="S2.I2.i1.p1.5.m5.2.2.3" xref="S2.I2.i1.p1.5.m5.2.2.3.cmml">i</mi><mo id="S2.I2.i1.p1.5.m5.2.2.2" xref="S2.I2.i1.p1.5.m5.2.2.2.cmml">=</mo><mrow id="S2.I2.i1.p1.5.m5.2.2.1.1" xref="S2.I2.i1.p1.5.m5.2.2.1.2.cmml"><mn id="S2.I2.i1.p1.5.m5.1.1" xref="S2.I2.i1.p1.5.m5.1.1.cmml">0</mn><mo id="S2.I2.i1.p1.5.m5.2.2.1.1.2" xref="S2.I2.i1.p1.5.m5.2.2.1.2.cmml">,</mo><mrow id="S2.I2.i1.p1.5.m5.2.2.1.1.1" xref="S2.I2.i1.p1.5.m5.2.2.1.1.1.cmml"><mrow id="S2.I2.i1.p1.5.m5.2.2.1.1.1.2" xref="S2.I2.i1.p1.5.m5.2.2.1.1.1.2.cmml"><mi mathvariant="normal" id="S2.I2.i1.p1.5.m5.2.2.1.1.1.2.2" xref="S2.I2.i1.p1.5.m5.2.2.1.1.1.2.2.cmml">…</mi><mo lspace="0em" rspace="0em" id="S2.I2.i1.p1.5.m5.2.2.1.1.1.2.1" xref="S2.I2.i1.p1.5.m5.2.2.1.1.1.2.1.cmml">​</mo><mi id="S2.I2.i1.p1.5.m5.2.2.1.1.1.2.3" xref="S2.I2.i1.p1.5.m5.2.2.1.1.1.2.3.cmml">n</mi></mrow><mo id="S2.I2.i1.p1.5.m5.2.2.1.1.1.1" xref="S2.I2.i1.p1.5.m5.2.2.1.1.1.1.cmml">−</mo><mn id="S2.I2.i1.p1.5.m5.2.2.1.1.1.3" xref="S2.I2.i1.p1.5.m5.2.2.1.1.1.3.cmml">1</mn></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.I2.i1.p1.5.m5.2b"><apply id="S2.I2.i1.p1.5.m5.2.2.cmml" xref="S2.I2.i1.p1.5.m5.2.2"><eq id="S2.I2.i1.p1.5.m5.2.2.2.cmml" xref="S2.I2.i1.p1.5.m5.2.2.2"></eq><ci id="S2.I2.i1.p1.5.m5.2.2.3.cmml" xref="S2.I2.i1.p1.5.m5.2.2.3">𝑖</ci><list id="S2.I2.i1.p1.5.m5.2.2.1.2.cmml" xref="S2.I2.i1.p1.5.m5.2.2.1.1"><cn type="integer" id="S2.I2.i1.p1.5.m5.1.1.cmml" xref="S2.I2.i1.p1.5.m5.1.1">0</cn><apply id="S2.I2.i1.p1.5.m5.2.2.1.1.1.cmml" xref="S2.I2.i1.p1.5.m5.2.2.1.1.1"><minus id="S2.I2.i1.p1.5.m5.2.2.1.1.1.1.cmml" xref="S2.I2.i1.p1.5.m5.2.2.1.1.1.1"></minus><apply id="S2.I2.i1.p1.5.m5.2.2.1.1.1.2.cmml" xref="S2.I2.i1.p1.5.m5.2.2.1.1.1.2"><times id="S2.I2.i1.p1.5.m5.2.2.1.1.1.2.1.cmml" xref="S2.I2.i1.p1.5.m5.2.2.1.1.1.2.1"></times><ci id="S2.I2.i1.p1.5.m5.2.2.1.1.1.2.2.cmml" xref="S2.I2.i1.p1.5.m5.2.2.1.1.1.2.2">…</ci><ci id="S2.I2.i1.p1.5.m5.2.2.1.1.1.2.3.cmml" xref="S2.I2.i1.p1.5.m5.2.2.1.1.1.2.3">𝑛</ci></apply><cn type="integer" id="S2.I2.i1.p1.5.m5.2.2.1.1.1.3.cmml" xref="S2.I2.i1.p1.5.m5.2.2.1.1.1.3">1</cn></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i1.p1.5.m5.2c">i=0,\dots n-1</annotation></semantics></math>.</p>
</div>
</li>
<li id="S2.I2.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S2.I2.i2.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S2.I2.i2.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S2.I2.i2.1.1.m1.1.1" xref="S2.I2.i2.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S2.I2.i2.1.1.m1.1c"><ci id="S2.I2.i2.1.1.m1.1.1.cmml" xref="S2.I2.i2.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i2.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S2.I2.i2.p1" class="ltx_para">
<p id="S2.I2.i2.p1.13" class="ltx_p">Its directed edges consist of each attention edge <math id="S2.I2.i2.p1.1.m1.4" class="ltx_Math" alttext="e^{l,h}_{i,j}" display="inline"><semantics id="S2.I2.i2.p1.1.m1.4a"><msubsup id="S2.I2.i2.p1.1.m1.4.5" xref="S2.I2.i2.p1.1.m1.4.5.cmml"><mi id="S2.I2.i2.p1.1.m1.4.5.2.2" xref="S2.I2.i2.p1.1.m1.4.5.2.2.cmml">e</mi><mrow id="S2.I2.i2.p1.1.m1.4.4.2.4" xref="S2.I2.i2.p1.1.m1.4.4.2.3.cmml"><mi id="S2.I2.i2.p1.1.m1.3.3.1.1" xref="S2.I2.i2.p1.1.m1.3.3.1.1.cmml">i</mi><mo id="S2.I2.i2.p1.1.m1.4.4.2.4.1" xref="S2.I2.i2.p1.1.m1.4.4.2.3.cmml">,</mo><mi id="S2.I2.i2.p1.1.m1.4.4.2.2" xref="S2.I2.i2.p1.1.m1.4.4.2.2.cmml">j</mi></mrow><mrow id="S2.I2.i2.p1.1.m1.2.2.2.4" xref="S2.I2.i2.p1.1.m1.2.2.2.3.cmml"><mi id="S2.I2.i2.p1.1.m1.1.1.1.1" xref="S2.I2.i2.p1.1.m1.1.1.1.1.cmml">l</mi><mo id="S2.I2.i2.p1.1.m1.2.2.2.4.1" xref="S2.I2.i2.p1.1.m1.2.2.2.3.cmml">,</mo><mi id="S2.I2.i2.p1.1.m1.2.2.2.2" xref="S2.I2.i2.p1.1.m1.2.2.2.2.cmml">h</mi></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S2.I2.i2.p1.1.m1.4b"><apply id="S2.I2.i2.p1.1.m1.4.5.cmml" xref="S2.I2.i2.p1.1.m1.4.5"><csymbol cd="ambiguous" id="S2.I2.i2.p1.1.m1.4.5.1.cmml" xref="S2.I2.i2.p1.1.m1.4.5">subscript</csymbol><apply id="S2.I2.i2.p1.1.m1.4.5.2.cmml" xref="S2.I2.i2.p1.1.m1.4.5"><csymbol cd="ambiguous" id="S2.I2.i2.p1.1.m1.4.5.2.1.cmml" xref="S2.I2.i2.p1.1.m1.4.5">superscript</csymbol><ci id="S2.I2.i2.p1.1.m1.4.5.2.2.cmml" xref="S2.I2.i2.p1.1.m1.4.5.2.2">𝑒</ci><list id="S2.I2.i2.p1.1.m1.2.2.2.3.cmml" xref="S2.I2.i2.p1.1.m1.2.2.2.4"><ci id="S2.I2.i2.p1.1.m1.1.1.1.1.cmml" xref="S2.I2.i2.p1.1.m1.1.1.1.1">𝑙</ci><ci id="S2.I2.i2.p1.1.m1.2.2.2.2.cmml" xref="S2.I2.i2.p1.1.m1.2.2.2.2">ℎ</ci></list></apply><list id="S2.I2.i2.p1.1.m1.4.4.2.3.cmml" xref="S2.I2.i2.p1.1.m1.4.4.2.4"><ci id="S2.I2.i2.p1.1.m1.3.3.1.1.cmml" xref="S2.I2.i2.p1.1.m1.3.3.1.1">𝑖</ci><ci id="S2.I2.i2.p1.1.m1.4.4.2.2.cmml" xref="S2.I2.i2.p1.1.m1.4.4.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i2.p1.1.m1.4c">e^{l,h}_{i,j}</annotation></semantics></math> that goes from <math id="S2.I2.i2.p1.2.m2.1" class="ltx_Math" alttext="\mathbf{x}_{i}^{2l}" display="inline"><semantics id="S2.I2.i2.p1.2.m2.1a"><msubsup id="S2.I2.i2.p1.2.m2.1.1" xref="S2.I2.i2.p1.2.m2.1.1.cmml"><mi id="S2.I2.i2.p1.2.m2.1.1.2.2" xref="S2.I2.i2.p1.2.m2.1.1.2.2.cmml">𝐱</mi><mi id="S2.I2.i2.p1.2.m2.1.1.2.3" xref="S2.I2.i2.p1.2.m2.1.1.2.3.cmml">i</mi><mrow id="S2.I2.i2.p1.2.m2.1.1.3" xref="S2.I2.i2.p1.2.m2.1.1.3.cmml"><mn id="S2.I2.i2.p1.2.m2.1.1.3.2" xref="S2.I2.i2.p1.2.m2.1.1.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S2.I2.i2.p1.2.m2.1.1.3.1" xref="S2.I2.i2.p1.2.m2.1.1.3.1.cmml">​</mo><mi id="S2.I2.i2.p1.2.m2.1.1.3.3" xref="S2.I2.i2.p1.2.m2.1.1.3.3.cmml">l</mi></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S2.I2.i2.p1.2.m2.1b"><apply id="S2.I2.i2.p1.2.m2.1.1.cmml" xref="S2.I2.i2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S2.I2.i2.p1.2.m2.1.1.1.cmml" xref="S2.I2.i2.p1.2.m2.1.1">superscript</csymbol><apply id="S2.I2.i2.p1.2.m2.1.1.2.cmml" xref="S2.I2.i2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S2.I2.i2.p1.2.m2.1.1.2.1.cmml" xref="S2.I2.i2.p1.2.m2.1.1">subscript</csymbol><ci id="S2.I2.i2.p1.2.m2.1.1.2.2.cmml" xref="S2.I2.i2.p1.2.m2.1.1.2.2">𝐱</ci><ci id="S2.I2.i2.p1.2.m2.1.1.2.3.cmml" xref="S2.I2.i2.p1.2.m2.1.1.2.3">𝑖</ci></apply><apply id="S2.I2.i2.p1.2.m2.1.1.3.cmml" xref="S2.I2.i2.p1.2.m2.1.1.3"><times id="S2.I2.i2.p1.2.m2.1.1.3.1.cmml" xref="S2.I2.i2.p1.2.m2.1.1.3.1"></times><cn type="integer" id="S2.I2.i2.p1.2.m2.1.1.3.2.cmml" xref="S2.I2.i2.p1.2.m2.1.1.3.2">2</cn><ci id="S2.I2.i2.p1.2.m2.1.1.3.3.cmml" xref="S2.I2.i2.p1.2.m2.1.1.3.3">𝑙</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i2.p1.2.m2.1c">\mathbf{x}_{i}^{2l}</annotation></semantics></math> to <math id="S2.I2.i2.p1.3.m3.1" class="ltx_Math" alttext="\mathbf{x}_{j}^{2l+1}" display="inline"><semantics id="S2.I2.i2.p1.3.m3.1a"><msubsup id="S2.I2.i2.p1.3.m3.1.1" xref="S2.I2.i2.p1.3.m3.1.1.cmml"><mi id="S2.I2.i2.p1.3.m3.1.1.2.2" xref="S2.I2.i2.p1.3.m3.1.1.2.2.cmml">𝐱</mi><mi id="S2.I2.i2.p1.3.m3.1.1.2.3" xref="S2.I2.i2.p1.3.m3.1.1.2.3.cmml">j</mi><mrow id="S2.I2.i2.p1.3.m3.1.1.3" xref="S2.I2.i2.p1.3.m3.1.1.3.cmml"><mrow id="S2.I2.i2.p1.3.m3.1.1.3.2" xref="S2.I2.i2.p1.3.m3.1.1.3.2.cmml"><mn id="S2.I2.i2.p1.3.m3.1.1.3.2.2" xref="S2.I2.i2.p1.3.m3.1.1.3.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S2.I2.i2.p1.3.m3.1.1.3.2.1" xref="S2.I2.i2.p1.3.m3.1.1.3.2.1.cmml">​</mo><mi id="S2.I2.i2.p1.3.m3.1.1.3.2.3" xref="S2.I2.i2.p1.3.m3.1.1.3.2.3.cmml">l</mi></mrow><mo id="S2.I2.i2.p1.3.m3.1.1.3.1" xref="S2.I2.i2.p1.3.m3.1.1.3.1.cmml">+</mo><mn id="S2.I2.i2.p1.3.m3.1.1.3.3" xref="S2.I2.i2.p1.3.m3.1.1.3.3.cmml">1</mn></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S2.I2.i2.p1.3.m3.1b"><apply id="S2.I2.i2.p1.3.m3.1.1.cmml" xref="S2.I2.i2.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S2.I2.i2.p1.3.m3.1.1.1.cmml" xref="S2.I2.i2.p1.3.m3.1.1">superscript</csymbol><apply id="S2.I2.i2.p1.3.m3.1.1.2.cmml" xref="S2.I2.i2.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S2.I2.i2.p1.3.m3.1.1.2.1.cmml" xref="S2.I2.i2.p1.3.m3.1.1">subscript</csymbol><ci id="S2.I2.i2.p1.3.m3.1.1.2.2.cmml" xref="S2.I2.i2.p1.3.m3.1.1.2.2">𝐱</ci><ci id="S2.I2.i2.p1.3.m3.1.1.2.3.cmml" xref="S2.I2.i2.p1.3.m3.1.1.2.3">𝑗</ci></apply><apply id="S2.I2.i2.p1.3.m3.1.1.3.cmml" xref="S2.I2.i2.p1.3.m3.1.1.3"><plus id="S2.I2.i2.p1.3.m3.1.1.3.1.cmml" xref="S2.I2.i2.p1.3.m3.1.1.3.1"></plus><apply id="S2.I2.i2.p1.3.m3.1.1.3.2.cmml" xref="S2.I2.i2.p1.3.m3.1.1.3.2"><times id="S2.I2.i2.p1.3.m3.1.1.3.2.1.cmml" xref="S2.I2.i2.p1.3.m3.1.1.3.2.1"></times><cn type="integer" id="S2.I2.i2.p1.3.m3.1.1.3.2.2.cmml" xref="S2.I2.i2.p1.3.m3.1.1.3.2.2">2</cn><ci id="S2.I2.i2.p1.3.m3.1.1.3.2.3.cmml" xref="S2.I2.i2.p1.3.m3.1.1.3.2.3">𝑙</ci></apply><cn type="integer" id="S2.I2.i2.p1.3.m3.1.1.3.3.cmml" xref="S2.I2.i2.p1.3.m3.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i2.p1.3.m3.1c">\mathbf{x}_{j}^{2l+1}</annotation></semantics></math> at the <math id="S2.I2.i2.p1.4.m4.1" class="ltx_Math" alttext="h" display="inline"><semantics id="S2.I2.i2.p1.4.m4.1a"><mi id="S2.I2.i2.p1.4.m4.1.1" xref="S2.I2.i2.p1.4.m4.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S2.I2.i2.p1.4.m4.1b"><ci id="S2.I2.i2.p1.4.m4.1.1.cmml" xref="S2.I2.i2.p1.4.m4.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i2.p1.4.m4.1c">h</annotation></semantics></math>-th head of the <math id="S2.I2.i2.p1.5.m5.1" class="ltx_Math" alttext="l" display="inline"><semantics id="S2.I2.i2.p1.5.m5.1a"><mi id="S2.I2.i2.p1.5.m5.1.1" xref="S2.I2.i2.p1.5.m5.1.1.cmml">l</mi><annotation-xml encoding="MathML-Content" id="S2.I2.i2.p1.5.m5.1b"><ci id="S2.I2.i2.p1.5.m5.1.1.cmml" xref="S2.I2.i2.p1.5.m5.1.1">𝑙</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i2.p1.5.m5.1c">l</annotation></semantics></math>-th attention layer for all <math id="S2.I2.i2.p1.6.m6.2" class="ltx_Math" alttext="l,h" display="inline"><semantics id="S2.I2.i2.p1.6.m6.2a"><mrow id="S2.I2.i2.p1.6.m6.2.3.2" xref="S2.I2.i2.p1.6.m6.2.3.1.cmml"><mi id="S2.I2.i2.p1.6.m6.1.1" xref="S2.I2.i2.p1.6.m6.1.1.cmml">l</mi><mo id="S2.I2.i2.p1.6.m6.2.3.2.1" xref="S2.I2.i2.p1.6.m6.2.3.1.cmml">,</mo><mi id="S2.I2.i2.p1.6.m6.2.2" xref="S2.I2.i2.p1.6.m6.2.2.cmml">h</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.I2.i2.p1.6.m6.2b"><list id="S2.I2.i2.p1.6.m6.2.3.1.cmml" xref="S2.I2.i2.p1.6.m6.2.3.2"><ci id="S2.I2.i2.p1.6.m6.1.1.cmml" xref="S2.I2.i2.p1.6.m6.1.1">𝑙</ci><ci id="S2.I2.i2.p1.6.m6.2.2.cmml" xref="S2.I2.i2.p1.6.m6.2.2">ℎ</ci></list></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i2.p1.6.m6.2c">l,h</annotation></semantics></math> and <math id="S2.I2.i2.p1.7.m7.1" class="ltx_Math" alttext="i\leq j" display="inline"><semantics id="S2.I2.i2.p1.7.m7.1a"><mrow id="S2.I2.i2.p1.7.m7.1.1" xref="S2.I2.i2.p1.7.m7.1.1.cmml"><mi id="S2.I2.i2.p1.7.m7.1.1.2" xref="S2.I2.i2.p1.7.m7.1.1.2.cmml">i</mi><mo id="S2.I2.i2.p1.7.m7.1.1.1" xref="S2.I2.i2.p1.7.m7.1.1.1.cmml">≤</mo><mi id="S2.I2.i2.p1.7.m7.1.1.3" xref="S2.I2.i2.p1.7.m7.1.1.3.cmml">j</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.I2.i2.p1.7.m7.1b"><apply id="S2.I2.i2.p1.7.m7.1.1.cmml" xref="S2.I2.i2.p1.7.m7.1.1"><leq id="S2.I2.i2.p1.7.m7.1.1.1.cmml" xref="S2.I2.i2.p1.7.m7.1.1.1"></leq><ci id="S2.I2.i2.p1.7.m7.1.1.2.cmml" xref="S2.I2.i2.p1.7.m7.1.1.2">𝑖</ci><ci id="S2.I2.i2.p1.7.m7.1.1.3.cmml" xref="S2.I2.i2.p1.7.m7.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i2.p1.7.m7.1c">i\leq j</annotation></semantics></math>, as well as each MLP edge <math id="S2.I2.i2.p1.8.m8.2" class="ltx_Math" alttext="e^{l,m}_{i}" display="inline"><semantics id="S2.I2.i2.p1.8.m8.2a"><msubsup id="S2.I2.i2.p1.8.m8.2.3" xref="S2.I2.i2.p1.8.m8.2.3.cmml"><mi id="S2.I2.i2.p1.8.m8.2.3.2.2" xref="S2.I2.i2.p1.8.m8.2.3.2.2.cmml">e</mi><mi id="S2.I2.i2.p1.8.m8.2.3.3" xref="S2.I2.i2.p1.8.m8.2.3.3.cmml">i</mi><mrow id="S2.I2.i2.p1.8.m8.2.2.2.4" xref="S2.I2.i2.p1.8.m8.2.2.2.3.cmml"><mi id="S2.I2.i2.p1.8.m8.1.1.1.1" xref="S2.I2.i2.p1.8.m8.1.1.1.1.cmml">l</mi><mo id="S2.I2.i2.p1.8.m8.2.2.2.4.1" xref="S2.I2.i2.p1.8.m8.2.2.2.3.cmml">,</mo><mi id="S2.I2.i2.p1.8.m8.2.2.2.2" xref="S2.I2.i2.p1.8.m8.2.2.2.2.cmml">m</mi></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S2.I2.i2.p1.8.m8.2b"><apply id="S2.I2.i2.p1.8.m8.2.3.cmml" xref="S2.I2.i2.p1.8.m8.2.3"><csymbol cd="ambiguous" id="S2.I2.i2.p1.8.m8.2.3.1.cmml" xref="S2.I2.i2.p1.8.m8.2.3">subscript</csymbol><apply id="S2.I2.i2.p1.8.m8.2.3.2.cmml" xref="S2.I2.i2.p1.8.m8.2.3"><csymbol cd="ambiguous" id="S2.I2.i2.p1.8.m8.2.3.2.1.cmml" xref="S2.I2.i2.p1.8.m8.2.3">superscript</csymbol><ci id="S2.I2.i2.p1.8.m8.2.3.2.2.cmml" xref="S2.I2.i2.p1.8.m8.2.3.2.2">𝑒</ci><list id="S2.I2.i2.p1.8.m8.2.2.2.3.cmml" xref="S2.I2.i2.p1.8.m8.2.2.2.4"><ci id="S2.I2.i2.p1.8.m8.1.1.1.1.cmml" xref="S2.I2.i2.p1.8.m8.1.1.1.1">𝑙</ci><ci id="S2.I2.i2.p1.8.m8.2.2.2.2.cmml" xref="S2.I2.i2.p1.8.m8.2.2.2.2">𝑚</ci></list></apply><ci id="S2.I2.i2.p1.8.m8.2.3.3.cmml" xref="S2.I2.i2.p1.8.m8.2.3.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i2.p1.8.m8.2c">e^{l,m}_{i}</annotation></semantics></math> that goes from <math id="S2.I2.i2.p1.9.m9.1" class="ltx_Math" alttext="\mathbf{x}_{i}^{2l+1}" display="inline"><semantics id="S2.I2.i2.p1.9.m9.1a"><msubsup id="S2.I2.i2.p1.9.m9.1.1" xref="S2.I2.i2.p1.9.m9.1.1.cmml"><mi id="S2.I2.i2.p1.9.m9.1.1.2.2" xref="S2.I2.i2.p1.9.m9.1.1.2.2.cmml">𝐱</mi><mi id="S2.I2.i2.p1.9.m9.1.1.2.3" xref="S2.I2.i2.p1.9.m9.1.1.2.3.cmml">i</mi><mrow id="S2.I2.i2.p1.9.m9.1.1.3" xref="S2.I2.i2.p1.9.m9.1.1.3.cmml"><mrow id="S2.I2.i2.p1.9.m9.1.1.3.2" xref="S2.I2.i2.p1.9.m9.1.1.3.2.cmml"><mn id="S2.I2.i2.p1.9.m9.1.1.3.2.2" xref="S2.I2.i2.p1.9.m9.1.1.3.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S2.I2.i2.p1.9.m9.1.1.3.2.1" xref="S2.I2.i2.p1.9.m9.1.1.3.2.1.cmml">​</mo><mi id="S2.I2.i2.p1.9.m9.1.1.3.2.3" xref="S2.I2.i2.p1.9.m9.1.1.3.2.3.cmml">l</mi></mrow><mo id="S2.I2.i2.p1.9.m9.1.1.3.1" xref="S2.I2.i2.p1.9.m9.1.1.3.1.cmml">+</mo><mn id="S2.I2.i2.p1.9.m9.1.1.3.3" xref="S2.I2.i2.p1.9.m9.1.1.3.3.cmml">1</mn></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S2.I2.i2.p1.9.m9.1b"><apply id="S2.I2.i2.p1.9.m9.1.1.cmml" xref="S2.I2.i2.p1.9.m9.1.1"><csymbol cd="ambiguous" id="S2.I2.i2.p1.9.m9.1.1.1.cmml" xref="S2.I2.i2.p1.9.m9.1.1">superscript</csymbol><apply id="S2.I2.i2.p1.9.m9.1.1.2.cmml" xref="S2.I2.i2.p1.9.m9.1.1"><csymbol cd="ambiguous" id="S2.I2.i2.p1.9.m9.1.1.2.1.cmml" xref="S2.I2.i2.p1.9.m9.1.1">subscript</csymbol><ci id="S2.I2.i2.p1.9.m9.1.1.2.2.cmml" xref="S2.I2.i2.p1.9.m9.1.1.2.2">𝐱</ci><ci id="S2.I2.i2.p1.9.m9.1.1.2.3.cmml" xref="S2.I2.i2.p1.9.m9.1.1.2.3">𝑖</ci></apply><apply id="S2.I2.i2.p1.9.m9.1.1.3.cmml" xref="S2.I2.i2.p1.9.m9.1.1.3"><plus id="S2.I2.i2.p1.9.m9.1.1.3.1.cmml" xref="S2.I2.i2.p1.9.m9.1.1.3.1"></plus><apply id="S2.I2.i2.p1.9.m9.1.1.3.2.cmml" xref="S2.I2.i2.p1.9.m9.1.1.3.2"><times id="S2.I2.i2.p1.9.m9.1.1.3.2.1.cmml" xref="S2.I2.i2.p1.9.m9.1.1.3.2.1"></times><cn type="integer" id="S2.I2.i2.p1.9.m9.1.1.3.2.2.cmml" xref="S2.I2.i2.p1.9.m9.1.1.3.2.2">2</cn><ci id="S2.I2.i2.p1.9.m9.1.1.3.2.3.cmml" xref="S2.I2.i2.p1.9.m9.1.1.3.2.3">𝑙</ci></apply><cn type="integer" id="S2.I2.i2.p1.9.m9.1.1.3.3.cmml" xref="S2.I2.i2.p1.9.m9.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i2.p1.9.m9.1c">\mathbf{x}_{i}^{2l+1}</annotation></semantics></math> to <math id="S2.I2.i2.p1.10.m10.1" class="ltx_Math" alttext="\mathbf{x}_{i}^{2l+2}" display="inline"><semantics id="S2.I2.i2.p1.10.m10.1a"><msubsup id="S2.I2.i2.p1.10.m10.1.1" xref="S2.I2.i2.p1.10.m10.1.1.cmml"><mi id="S2.I2.i2.p1.10.m10.1.1.2.2" xref="S2.I2.i2.p1.10.m10.1.1.2.2.cmml">𝐱</mi><mi id="S2.I2.i2.p1.10.m10.1.1.2.3" xref="S2.I2.i2.p1.10.m10.1.1.2.3.cmml">i</mi><mrow id="S2.I2.i2.p1.10.m10.1.1.3" xref="S2.I2.i2.p1.10.m10.1.1.3.cmml"><mrow id="S2.I2.i2.p1.10.m10.1.1.3.2" xref="S2.I2.i2.p1.10.m10.1.1.3.2.cmml"><mn id="S2.I2.i2.p1.10.m10.1.1.3.2.2" xref="S2.I2.i2.p1.10.m10.1.1.3.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S2.I2.i2.p1.10.m10.1.1.3.2.1" xref="S2.I2.i2.p1.10.m10.1.1.3.2.1.cmml">​</mo><mi id="S2.I2.i2.p1.10.m10.1.1.3.2.3" xref="S2.I2.i2.p1.10.m10.1.1.3.2.3.cmml">l</mi></mrow><mo id="S2.I2.i2.p1.10.m10.1.1.3.1" xref="S2.I2.i2.p1.10.m10.1.1.3.1.cmml">+</mo><mn id="S2.I2.i2.p1.10.m10.1.1.3.3" xref="S2.I2.i2.p1.10.m10.1.1.3.3.cmml">2</mn></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S2.I2.i2.p1.10.m10.1b"><apply id="S2.I2.i2.p1.10.m10.1.1.cmml" xref="S2.I2.i2.p1.10.m10.1.1"><csymbol cd="ambiguous" id="S2.I2.i2.p1.10.m10.1.1.1.cmml" xref="S2.I2.i2.p1.10.m10.1.1">superscript</csymbol><apply id="S2.I2.i2.p1.10.m10.1.1.2.cmml" xref="S2.I2.i2.p1.10.m10.1.1"><csymbol cd="ambiguous" id="S2.I2.i2.p1.10.m10.1.1.2.1.cmml" xref="S2.I2.i2.p1.10.m10.1.1">subscript</csymbol><ci id="S2.I2.i2.p1.10.m10.1.1.2.2.cmml" xref="S2.I2.i2.p1.10.m10.1.1.2.2">𝐱</ci><ci id="S2.I2.i2.p1.10.m10.1.1.2.3.cmml" xref="S2.I2.i2.p1.10.m10.1.1.2.3">𝑖</ci></apply><apply id="S2.I2.i2.p1.10.m10.1.1.3.cmml" xref="S2.I2.i2.p1.10.m10.1.1.3"><plus id="S2.I2.i2.p1.10.m10.1.1.3.1.cmml" xref="S2.I2.i2.p1.10.m10.1.1.3.1"></plus><apply id="S2.I2.i2.p1.10.m10.1.1.3.2.cmml" xref="S2.I2.i2.p1.10.m10.1.1.3.2"><times id="S2.I2.i2.p1.10.m10.1.1.3.2.1.cmml" xref="S2.I2.i2.p1.10.m10.1.1.3.2.1"></times><cn type="integer" id="S2.I2.i2.p1.10.m10.1.1.3.2.2.cmml" xref="S2.I2.i2.p1.10.m10.1.1.3.2.2">2</cn><ci id="S2.I2.i2.p1.10.m10.1.1.3.2.3.cmml" xref="S2.I2.i2.p1.10.m10.1.1.3.2.3">𝑙</ci></apply><cn type="integer" id="S2.I2.i2.p1.10.m10.1.1.3.3.cmml" xref="S2.I2.i2.p1.10.m10.1.1.3.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i2.p1.10.m10.1c">\mathbf{x}_{i}^{2l+2}</annotation></semantics></math> through the <math id="S2.I2.i2.p1.11.m11.1" class="ltx_Math" alttext="m" display="inline"><semantics id="S2.I2.i2.p1.11.m11.1a"><mi id="S2.I2.i2.p1.11.m11.1.1" xref="S2.I2.i2.p1.11.m11.1.1.cmml">m</mi><annotation-xml encoding="MathML-Content" id="S2.I2.i2.p1.11.m11.1b"><ci id="S2.I2.i2.p1.11.m11.1.1.cmml" xref="S2.I2.i2.p1.11.m11.1.1">𝑚</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i2.p1.11.m11.1c">m</annotation></semantics></math>-th neuron of the <math id="S2.I2.i2.p1.12.m12.1" class="ltx_Math" alttext="l" display="inline"><semantics id="S2.I2.i2.p1.12.m12.1a"><mi id="S2.I2.i2.p1.12.m12.1.1" xref="S2.I2.i2.p1.12.m12.1.1.cmml">l</mi><annotation-xml encoding="MathML-Content" id="S2.I2.i2.p1.12.m12.1b"><ci id="S2.I2.i2.p1.12.m12.1.1.cmml" xref="S2.I2.i2.p1.12.m12.1.1">𝑙</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i2.p1.12.m12.1c">l</annotation></semantics></math>-th MLP layer for all <math id="S2.I2.i2.p1.13.m13.3" class="ltx_Math" alttext="l,m,i" display="inline"><semantics id="S2.I2.i2.p1.13.m13.3a"><mrow id="S2.I2.i2.p1.13.m13.3.4.2" xref="S2.I2.i2.p1.13.m13.3.4.1.cmml"><mi id="S2.I2.i2.p1.13.m13.1.1" xref="S2.I2.i2.p1.13.m13.1.1.cmml">l</mi><mo id="S2.I2.i2.p1.13.m13.3.4.2.1" xref="S2.I2.i2.p1.13.m13.3.4.1.cmml">,</mo><mi id="S2.I2.i2.p1.13.m13.2.2" xref="S2.I2.i2.p1.13.m13.2.2.cmml">m</mi><mo id="S2.I2.i2.p1.13.m13.3.4.2.2" xref="S2.I2.i2.p1.13.m13.3.4.1.cmml">,</mo><mi id="S2.I2.i2.p1.13.m13.3.3" xref="S2.I2.i2.p1.13.m13.3.3.cmml">i</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.I2.i2.p1.13.m13.3b"><list id="S2.I2.i2.p1.13.m13.3.4.1.cmml" xref="S2.I2.i2.p1.13.m13.3.4.2"><ci id="S2.I2.i2.p1.13.m13.1.1.cmml" xref="S2.I2.i2.p1.13.m13.1.1">𝑙</ci><ci id="S2.I2.i2.p1.13.m13.2.2.cmml" xref="S2.I2.i2.p1.13.m13.2.2">𝑚</ci><ci id="S2.I2.i2.p1.13.m13.3.3.cmml" xref="S2.I2.i2.p1.13.m13.3.3">𝑖</ci></list></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i2.p1.13.m13.3c">l,m,i</annotation></semantics></math>.</p>
</div>
</li>
<li id="S2.I2.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S2.I2.i3.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S2.I2.i3.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S2.I2.i3.1.1.m1.1.1" xref="S2.I2.i3.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S2.I2.i3.1.1.m1.1c"><ci id="S2.I2.i3.1.1.m1.1.1.cmml" xref="S2.I2.i3.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i3.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S2.I2.i3.p1" class="ltx_para">
<p id="S2.I2.i3.p1.2" class="ltx_p">The weight of each attention edge <math id="S2.I2.i3.p1.1.m1.4" class="ltx_Math" alttext="e^{l,h}_{i,j}" display="inline"><semantics id="S2.I2.i3.p1.1.m1.4a"><msubsup id="S2.I2.i3.p1.1.m1.4.5" xref="S2.I2.i3.p1.1.m1.4.5.cmml"><mi id="S2.I2.i3.p1.1.m1.4.5.2.2" xref="S2.I2.i3.p1.1.m1.4.5.2.2.cmml">e</mi><mrow id="S2.I2.i3.p1.1.m1.4.4.2.4" xref="S2.I2.i3.p1.1.m1.4.4.2.3.cmml"><mi id="S2.I2.i3.p1.1.m1.3.3.1.1" xref="S2.I2.i3.p1.1.m1.3.3.1.1.cmml">i</mi><mo id="S2.I2.i3.p1.1.m1.4.4.2.4.1" xref="S2.I2.i3.p1.1.m1.4.4.2.3.cmml">,</mo><mi id="S2.I2.i3.p1.1.m1.4.4.2.2" xref="S2.I2.i3.p1.1.m1.4.4.2.2.cmml">j</mi></mrow><mrow id="S2.I2.i3.p1.1.m1.2.2.2.4" xref="S2.I2.i3.p1.1.m1.2.2.2.3.cmml"><mi id="S2.I2.i3.p1.1.m1.1.1.1.1" xref="S2.I2.i3.p1.1.m1.1.1.1.1.cmml">l</mi><mo id="S2.I2.i3.p1.1.m1.2.2.2.4.1" xref="S2.I2.i3.p1.1.m1.2.2.2.3.cmml">,</mo><mi id="S2.I2.i3.p1.1.m1.2.2.2.2" xref="S2.I2.i3.p1.1.m1.2.2.2.2.cmml">h</mi></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S2.I2.i3.p1.1.m1.4b"><apply id="S2.I2.i3.p1.1.m1.4.5.cmml" xref="S2.I2.i3.p1.1.m1.4.5"><csymbol cd="ambiguous" id="S2.I2.i3.p1.1.m1.4.5.1.cmml" xref="S2.I2.i3.p1.1.m1.4.5">subscript</csymbol><apply id="S2.I2.i3.p1.1.m1.4.5.2.cmml" xref="S2.I2.i3.p1.1.m1.4.5"><csymbol cd="ambiguous" id="S2.I2.i3.p1.1.m1.4.5.2.1.cmml" xref="S2.I2.i3.p1.1.m1.4.5">superscript</csymbol><ci id="S2.I2.i3.p1.1.m1.4.5.2.2.cmml" xref="S2.I2.i3.p1.1.m1.4.5.2.2">𝑒</ci><list id="S2.I2.i3.p1.1.m1.2.2.2.3.cmml" xref="S2.I2.i3.p1.1.m1.2.2.2.4"><ci id="S2.I2.i3.p1.1.m1.1.1.1.1.cmml" xref="S2.I2.i3.p1.1.m1.1.1.1.1">𝑙</ci><ci id="S2.I2.i3.p1.1.m1.2.2.2.2.cmml" xref="S2.I2.i3.p1.1.m1.2.2.2.2">ℎ</ci></list></apply><list id="S2.I2.i3.p1.1.m1.4.4.2.3.cmml" xref="S2.I2.i3.p1.1.m1.4.4.2.4"><ci id="S2.I2.i3.p1.1.m1.3.3.1.1.cmml" xref="S2.I2.i3.p1.1.m1.3.3.1.1">𝑖</ci><ci id="S2.I2.i3.p1.1.m1.4.4.2.2.cmml" xref="S2.I2.i3.p1.1.m1.4.4.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i3.p1.1.m1.4c">e^{l,h}_{i,j}</annotation></semantics></math>, which measures the influence of the attention score <math id="S2.I2.i3.p1.2.m2.4" class="ltx_Math" alttext="a^{l,h}_{i,j}" display="inline"><semantics id="S2.I2.i3.p1.2.m2.4a"><msubsup id="S2.I2.i3.p1.2.m2.4.5" xref="S2.I2.i3.p1.2.m2.4.5.cmml"><mi id="S2.I2.i3.p1.2.m2.4.5.2.2" xref="S2.I2.i3.p1.2.m2.4.5.2.2.cmml">a</mi><mrow id="S2.I2.i3.p1.2.m2.4.4.2.4" xref="S2.I2.i3.p1.2.m2.4.4.2.3.cmml"><mi id="S2.I2.i3.p1.2.m2.3.3.1.1" xref="S2.I2.i3.p1.2.m2.3.3.1.1.cmml">i</mi><mo id="S2.I2.i3.p1.2.m2.4.4.2.4.1" xref="S2.I2.i3.p1.2.m2.4.4.2.3.cmml">,</mo><mi id="S2.I2.i3.p1.2.m2.4.4.2.2" xref="S2.I2.i3.p1.2.m2.4.4.2.2.cmml">j</mi></mrow><mrow id="S2.I2.i3.p1.2.m2.2.2.2.4" xref="S2.I2.i3.p1.2.m2.2.2.2.3.cmml"><mi id="S2.I2.i3.p1.2.m2.1.1.1.1" xref="S2.I2.i3.p1.2.m2.1.1.1.1.cmml">l</mi><mo id="S2.I2.i3.p1.2.m2.2.2.2.4.1" xref="S2.I2.i3.p1.2.m2.2.2.2.3.cmml">,</mo><mi id="S2.I2.i3.p1.2.m2.2.2.2.2" xref="S2.I2.i3.p1.2.m2.2.2.2.2.cmml">h</mi></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S2.I2.i3.p1.2.m2.4b"><apply id="S2.I2.i3.p1.2.m2.4.5.cmml" xref="S2.I2.i3.p1.2.m2.4.5"><csymbol cd="ambiguous" id="S2.I2.i3.p1.2.m2.4.5.1.cmml" xref="S2.I2.i3.p1.2.m2.4.5">subscript</csymbol><apply id="S2.I2.i3.p1.2.m2.4.5.2.cmml" xref="S2.I2.i3.p1.2.m2.4.5"><csymbol cd="ambiguous" id="S2.I2.i3.p1.2.m2.4.5.2.1.cmml" xref="S2.I2.i3.p1.2.m2.4.5">superscript</csymbol><ci id="S2.I2.i3.p1.2.m2.4.5.2.2.cmml" xref="S2.I2.i3.p1.2.m2.4.5.2.2">𝑎</ci><list id="S2.I2.i3.p1.2.m2.2.2.2.3.cmml" xref="S2.I2.i3.p1.2.m2.2.2.2.4"><ci id="S2.I2.i3.p1.2.m2.1.1.1.1.cmml" xref="S2.I2.i3.p1.2.m2.1.1.1.1">𝑙</ci><ci id="S2.I2.i3.p1.2.m2.2.2.2.2.cmml" xref="S2.I2.i3.p1.2.m2.2.2.2.2">ℎ</ci></list></apply><list id="S2.I2.i3.p1.2.m2.4.4.2.3.cmml" xref="S2.I2.i3.p1.2.m2.4.4.2.4"><ci id="S2.I2.i3.p1.2.m2.3.3.1.1.cmml" xref="S2.I2.i3.p1.2.m2.3.3.1.1">𝑖</ci><ci id="S2.I2.i3.p1.2.m2.4.4.2.2.cmml" xref="S2.I2.i3.p1.2.m2.4.4.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i3.p1.2.m2.4c">a^{l,h}_{i,j}</annotation></semantics></math> on the LLM output, is defined by</p>
<table id="Ex1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="Ex1.m1.11" class="ltx_Math" alttext="\mathcal{L}-\mathcal{L}\big{|}_{a^{l,h}_{i,j}=0}\quad\text{or}\quad\frac{\partial\mathcal{L}}{\partial a^{l,h}_{i,j}}" display="block"><semantics id="Ex1.m1.11a"><mrow id="Ex1.m1.11.11.1" xref="Ex1.m1.11.11.2.cmml"><mrow id="Ex1.m1.11.11.1.1" xref="Ex1.m1.11.11.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="Ex1.m1.11.11.1.1.2" xref="Ex1.m1.11.11.1.1.2.cmml">ℒ</mi><mo id="Ex1.m1.11.11.1.1.1" xref="Ex1.m1.11.11.1.1.1.cmml">−</mo><msub id="Ex1.m1.11.11.1.1.3.2" xref="Ex1.m1.11.11.1.1.3.1.cmml"><mrow id="Ex1.m1.11.11.1.1.3.2.2" xref="Ex1.m1.11.11.1.1.3.1.cmml"><mi class="ltx_font_mathcaligraphic" id="Ex1.m1.9.9" xref="Ex1.m1.9.9.cmml">ℒ</mi><mo maxsize="120%" minsize="120%" id="Ex1.m1.11.11.1.1.3.2.2.1" xref="Ex1.m1.11.11.1.1.3.1.1.cmml">|</mo></mrow><mrow id="Ex1.m1.4.4.4" xref="Ex1.m1.4.4.4.cmml"><msubsup id="Ex1.m1.4.4.4.6" xref="Ex1.m1.4.4.4.6.cmml"><mi id="Ex1.m1.4.4.4.6.2.2" xref="Ex1.m1.4.4.4.6.2.2.cmml">a</mi><mrow id="Ex1.m1.4.4.4.4.2.4" xref="Ex1.m1.4.4.4.4.2.3.cmml"><mi id="Ex1.m1.3.3.3.3.1.1" xref="Ex1.m1.3.3.3.3.1.1.cmml">i</mi><mo id="Ex1.m1.4.4.4.4.2.4.1" xref="Ex1.m1.4.4.4.4.2.3.cmml">,</mo><mi id="Ex1.m1.4.4.4.4.2.2" xref="Ex1.m1.4.4.4.4.2.2.cmml">j</mi></mrow><mrow id="Ex1.m1.2.2.2.2.2.4" xref="Ex1.m1.2.2.2.2.2.3.cmml"><mi id="Ex1.m1.1.1.1.1.1.1" xref="Ex1.m1.1.1.1.1.1.1.cmml">l</mi><mo id="Ex1.m1.2.2.2.2.2.4.1" xref="Ex1.m1.2.2.2.2.2.3.cmml">,</mo><mi id="Ex1.m1.2.2.2.2.2.2" xref="Ex1.m1.2.2.2.2.2.2.cmml">h</mi></mrow></msubsup><mo id="Ex1.m1.4.4.4.5" xref="Ex1.m1.4.4.4.5.cmml">=</mo><mn id="Ex1.m1.4.4.4.7" xref="Ex1.m1.4.4.4.7.cmml">0</mn></mrow></msub></mrow><mspace width="1em" id="Ex1.m1.11.11.1.2" xref="Ex1.m1.11.11.2.cmml"></mspace><mtext id="Ex1.m1.10.10" xref="Ex1.m1.10.10a.cmml">or</mtext><mspace width="1em" id="Ex1.m1.11.11.1.3" xref="Ex1.m1.11.11.2.cmml"></mspace><mfrac id="Ex1.m1.8.8" xref="Ex1.m1.8.8.cmml"><mrow id="Ex1.m1.8.8.6" xref="Ex1.m1.8.8.6.cmml"><mo rspace="0em" id="Ex1.m1.8.8.6.1" xref="Ex1.m1.8.8.6.1.cmml">∂</mo><mi class="ltx_font_mathcaligraphic" id="Ex1.m1.8.8.6.2" xref="Ex1.m1.8.8.6.2.cmml">ℒ</mi></mrow><mrow id="Ex1.m1.8.8.4" xref="Ex1.m1.8.8.4.cmml"><mo rspace="0em" id="Ex1.m1.8.8.4.5" xref="Ex1.m1.8.8.4.5.cmml">∂</mo><msubsup id="Ex1.m1.8.8.4.6" xref="Ex1.m1.8.8.4.6.cmml"><mi id="Ex1.m1.8.8.4.6.2.2" xref="Ex1.m1.8.8.4.6.2.2.cmml">a</mi><mrow id="Ex1.m1.8.8.4.4.2.4" xref="Ex1.m1.8.8.4.4.2.3.cmml"><mi id="Ex1.m1.7.7.3.3.1.1" xref="Ex1.m1.7.7.3.3.1.1.cmml">i</mi><mo id="Ex1.m1.8.8.4.4.2.4.1" xref="Ex1.m1.8.8.4.4.2.3.cmml">,</mo><mi id="Ex1.m1.8.8.4.4.2.2" xref="Ex1.m1.8.8.4.4.2.2.cmml">j</mi></mrow><mrow id="Ex1.m1.6.6.2.2.2.4" xref="Ex1.m1.6.6.2.2.2.3.cmml"><mi id="Ex1.m1.5.5.1.1.1.1" xref="Ex1.m1.5.5.1.1.1.1.cmml">l</mi><mo id="Ex1.m1.6.6.2.2.2.4.1" xref="Ex1.m1.6.6.2.2.2.3.cmml">,</mo><mi id="Ex1.m1.6.6.2.2.2.2" xref="Ex1.m1.6.6.2.2.2.2.cmml">h</mi></mrow></msubsup></mrow></mfrac></mrow><annotation-xml encoding="MathML-Content" id="Ex1.m1.11b"><list id="Ex1.m1.11.11.2.cmml" xref="Ex1.m1.11.11.1"><apply id="Ex1.m1.11.11.1.1.cmml" xref="Ex1.m1.11.11.1.1"><minus id="Ex1.m1.11.11.1.1.1.cmml" xref="Ex1.m1.11.11.1.1.1"></minus><ci id="Ex1.m1.11.11.1.1.2.cmml" xref="Ex1.m1.11.11.1.1.2">ℒ</ci><apply id="Ex1.m1.11.11.1.1.3.1.cmml" xref="Ex1.m1.11.11.1.1.3.2"><csymbol cd="latexml" id="Ex1.m1.11.11.1.1.3.1.1.cmml" xref="Ex1.m1.11.11.1.1.3.2.2.1">evaluated-at</csymbol><ci id="Ex1.m1.9.9.cmml" xref="Ex1.m1.9.9">ℒ</ci><apply id="Ex1.m1.4.4.4.cmml" xref="Ex1.m1.4.4.4"><eq id="Ex1.m1.4.4.4.5.cmml" xref="Ex1.m1.4.4.4.5"></eq><apply id="Ex1.m1.4.4.4.6.cmml" xref="Ex1.m1.4.4.4.6"><csymbol cd="ambiguous" id="Ex1.m1.4.4.4.6.1.cmml" xref="Ex1.m1.4.4.4.6">subscript</csymbol><apply id="Ex1.m1.4.4.4.6.2.cmml" xref="Ex1.m1.4.4.4.6"><csymbol cd="ambiguous" id="Ex1.m1.4.4.4.6.2.1.cmml" xref="Ex1.m1.4.4.4.6">superscript</csymbol><ci id="Ex1.m1.4.4.4.6.2.2.cmml" xref="Ex1.m1.4.4.4.6.2.2">𝑎</ci><list id="Ex1.m1.2.2.2.2.2.3.cmml" xref="Ex1.m1.2.2.2.2.2.4"><ci id="Ex1.m1.1.1.1.1.1.1.cmml" xref="Ex1.m1.1.1.1.1.1.1">𝑙</ci><ci id="Ex1.m1.2.2.2.2.2.2.cmml" xref="Ex1.m1.2.2.2.2.2.2">ℎ</ci></list></apply><list id="Ex1.m1.4.4.4.4.2.3.cmml" xref="Ex1.m1.4.4.4.4.2.4"><ci id="Ex1.m1.3.3.3.3.1.1.cmml" xref="Ex1.m1.3.3.3.3.1.1">𝑖</ci><ci id="Ex1.m1.4.4.4.4.2.2.cmml" xref="Ex1.m1.4.4.4.4.2.2">𝑗</ci></list></apply><cn type="integer" id="Ex1.m1.4.4.4.7.cmml" xref="Ex1.m1.4.4.4.7">0</cn></apply></apply></apply><ci id="Ex1.m1.10.10a.cmml" xref="Ex1.m1.10.10"><mtext id="Ex1.m1.10.10.cmml" xref="Ex1.m1.10.10">or</mtext></ci><apply id="Ex1.m1.8.8.cmml" xref="Ex1.m1.8.8"><divide id="Ex1.m1.8.8.5.cmml" xref="Ex1.m1.8.8"></divide><apply id="Ex1.m1.8.8.6.cmml" xref="Ex1.m1.8.8.6"><partialdiff id="Ex1.m1.8.8.6.1.cmml" xref="Ex1.m1.8.8.6.1"></partialdiff><ci id="Ex1.m1.8.8.6.2.cmml" xref="Ex1.m1.8.8.6.2">ℒ</ci></apply><apply id="Ex1.m1.8.8.4.cmml" xref="Ex1.m1.8.8.4"><partialdiff id="Ex1.m1.8.8.4.5.cmml" xref="Ex1.m1.8.8.4.5"></partialdiff><apply id="Ex1.m1.8.8.4.6.cmml" xref="Ex1.m1.8.8.4.6"><csymbol cd="ambiguous" id="Ex1.m1.8.8.4.6.1.cmml" xref="Ex1.m1.8.8.4.6">subscript</csymbol><apply id="Ex1.m1.8.8.4.6.2.cmml" xref="Ex1.m1.8.8.4.6"><csymbol cd="ambiguous" id="Ex1.m1.8.8.4.6.2.1.cmml" xref="Ex1.m1.8.8.4.6">superscript</csymbol><ci id="Ex1.m1.8.8.4.6.2.2.cmml" xref="Ex1.m1.8.8.4.6.2.2">𝑎</ci><list id="Ex1.m1.6.6.2.2.2.3.cmml" xref="Ex1.m1.6.6.2.2.2.4"><ci id="Ex1.m1.5.5.1.1.1.1.cmml" xref="Ex1.m1.5.5.1.1.1.1">𝑙</ci><ci id="Ex1.m1.6.6.2.2.2.2.cmml" xref="Ex1.m1.6.6.2.2.2.2">ℎ</ci></list></apply><list id="Ex1.m1.8.8.4.4.2.3.cmml" xref="Ex1.m1.8.8.4.4.2.4"><ci id="Ex1.m1.7.7.3.3.1.1.cmml" xref="Ex1.m1.7.7.3.3.1.1">𝑖</ci><ci id="Ex1.m1.8.8.4.4.2.2.cmml" xref="Ex1.m1.8.8.4.4.2.2">𝑗</ci></list></apply></apply></apply></list></annotation-xml><annotation encoding="application/x-tex" id="Ex1.m1.11c">\mathcal{L}-\mathcal{L}\big{|}_{a^{l,h}_{i,j}=0}\quad\text{or}\quad\frac{\partial\mathcal{L}}{\partial a^{l,h}_{i,j}}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S2.I2.i3.p1.8" class="ltx_p">where <math id="S2.I2.i3.p1.3.m1.1" class="ltx_Math" alttext="\mathcal{L}" display="inline"><semantics id="S2.I2.i3.p1.3.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S2.I2.i3.p1.3.m1.1.1" xref="S2.I2.i3.p1.3.m1.1.1.cmml">ℒ</mi><annotation-xml encoding="MathML-Content" id="S2.I2.i3.p1.3.m1.1b"><ci id="S2.I2.i3.p1.3.m1.1.1.cmml" xref="S2.I2.i3.p1.3.m1.1.1">ℒ</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i3.p1.3.m1.1c">\mathcal{L}</annotation></semantics></math> is the log-likelihood of the target <math id="S2.I2.i3.p1.4.m2.2" class="ltx_Math" alttext="(t_{1},\dots t_{n})" display="inline"><semantics id="S2.I2.i3.p1.4.m2.2a"><mrow id="S2.I2.i3.p1.4.m2.2.2.2" xref="S2.I2.i3.p1.4.m2.2.2.3.cmml"><mo stretchy="false" id="S2.I2.i3.p1.4.m2.2.2.2.3" xref="S2.I2.i3.p1.4.m2.2.2.3.cmml">(</mo><msub id="S2.I2.i3.p1.4.m2.1.1.1.1" xref="S2.I2.i3.p1.4.m2.1.1.1.1.cmml"><mi id="S2.I2.i3.p1.4.m2.1.1.1.1.2" xref="S2.I2.i3.p1.4.m2.1.1.1.1.2.cmml">t</mi><mn id="S2.I2.i3.p1.4.m2.1.1.1.1.3" xref="S2.I2.i3.p1.4.m2.1.1.1.1.3.cmml">1</mn></msub><mo id="S2.I2.i3.p1.4.m2.2.2.2.4" xref="S2.I2.i3.p1.4.m2.2.2.3.cmml">,</mo><mrow id="S2.I2.i3.p1.4.m2.2.2.2.2" xref="S2.I2.i3.p1.4.m2.2.2.2.2.cmml"><mi mathvariant="normal" id="S2.I2.i3.p1.4.m2.2.2.2.2.2" xref="S2.I2.i3.p1.4.m2.2.2.2.2.2.cmml">…</mi><mo lspace="0em" rspace="0em" id="S2.I2.i3.p1.4.m2.2.2.2.2.1" xref="S2.I2.i3.p1.4.m2.2.2.2.2.1.cmml">​</mo><msub id="S2.I2.i3.p1.4.m2.2.2.2.2.3" xref="S2.I2.i3.p1.4.m2.2.2.2.2.3.cmml"><mi id="S2.I2.i3.p1.4.m2.2.2.2.2.3.2" xref="S2.I2.i3.p1.4.m2.2.2.2.2.3.2.cmml">t</mi><mi id="S2.I2.i3.p1.4.m2.2.2.2.2.3.3" xref="S2.I2.i3.p1.4.m2.2.2.2.2.3.3.cmml">n</mi></msub></mrow><mo stretchy="false" id="S2.I2.i3.p1.4.m2.2.2.2.5" xref="S2.I2.i3.p1.4.m2.2.2.3.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.I2.i3.p1.4.m2.2b"><interval closure="open" id="S2.I2.i3.p1.4.m2.2.2.3.cmml" xref="S2.I2.i3.p1.4.m2.2.2.2"><apply id="S2.I2.i3.p1.4.m2.1.1.1.1.cmml" xref="S2.I2.i3.p1.4.m2.1.1.1.1"><csymbol cd="ambiguous" id="S2.I2.i3.p1.4.m2.1.1.1.1.1.cmml" xref="S2.I2.i3.p1.4.m2.1.1.1.1">subscript</csymbol><ci id="S2.I2.i3.p1.4.m2.1.1.1.1.2.cmml" xref="S2.I2.i3.p1.4.m2.1.1.1.1.2">𝑡</ci><cn type="integer" id="S2.I2.i3.p1.4.m2.1.1.1.1.3.cmml" xref="S2.I2.i3.p1.4.m2.1.1.1.1.3">1</cn></apply><apply id="S2.I2.i3.p1.4.m2.2.2.2.2.cmml" xref="S2.I2.i3.p1.4.m2.2.2.2.2"><times id="S2.I2.i3.p1.4.m2.2.2.2.2.1.cmml" xref="S2.I2.i3.p1.4.m2.2.2.2.2.1"></times><ci id="S2.I2.i3.p1.4.m2.2.2.2.2.2.cmml" xref="S2.I2.i3.p1.4.m2.2.2.2.2.2">…</ci><apply id="S2.I2.i3.p1.4.m2.2.2.2.2.3.cmml" xref="S2.I2.i3.p1.4.m2.2.2.2.2.3"><csymbol cd="ambiguous" id="S2.I2.i3.p1.4.m2.2.2.2.2.3.1.cmml" xref="S2.I2.i3.p1.4.m2.2.2.2.2.3">subscript</csymbol><ci id="S2.I2.i3.p1.4.m2.2.2.2.2.3.2.cmml" xref="S2.I2.i3.p1.4.m2.2.2.2.2.3.2">𝑡</ci><ci id="S2.I2.i3.p1.4.m2.2.2.2.2.3.3.cmml" xref="S2.I2.i3.p1.4.m2.2.2.2.2.3.3">𝑛</ci></apply></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i3.p1.4.m2.2c">(t_{1},\dots t_{n})</annotation></semantics></math>, with <math id="S2.I2.i3.p1.5.m3.2" class="ltx_Math" alttext="\mathcal{L}|_{a=0}" display="inline"><semantics id="S2.I2.i3.p1.5.m3.2a"><msub id="S2.I2.i3.p1.5.m3.2.3.2" xref="S2.I2.i3.p1.5.m3.2.3.1.cmml"><mrow id="S2.I2.i3.p1.5.m3.2.3.2.2" xref="S2.I2.i3.p1.5.m3.2.3.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S2.I2.i3.p1.5.m3.1.1" xref="S2.I2.i3.p1.5.m3.1.1.cmml">ℒ</mi><mo stretchy="false" id="S2.I2.i3.p1.5.m3.2.3.2.2.1" xref="S2.I2.i3.p1.5.m3.2.3.1.1.cmml">|</mo></mrow><mrow id="S2.I2.i3.p1.5.m3.2.2.1" xref="S2.I2.i3.p1.5.m3.2.2.1.cmml"><mi id="S2.I2.i3.p1.5.m3.2.2.1.2" xref="S2.I2.i3.p1.5.m3.2.2.1.2.cmml">a</mi><mo id="S2.I2.i3.p1.5.m3.2.2.1.1" xref="S2.I2.i3.p1.5.m3.2.2.1.1.cmml">=</mo><mn id="S2.I2.i3.p1.5.m3.2.2.1.3" xref="S2.I2.i3.p1.5.m3.2.2.1.3.cmml">0</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.I2.i3.p1.5.m3.2b"><apply id="S2.I2.i3.p1.5.m3.2.3.1.cmml" xref="S2.I2.i3.p1.5.m3.2.3.2"><csymbol cd="latexml" id="S2.I2.i3.p1.5.m3.2.3.1.1.cmml" xref="S2.I2.i3.p1.5.m3.2.3.2.2.1">evaluated-at</csymbol><ci id="S2.I2.i3.p1.5.m3.1.1.cmml" xref="S2.I2.i3.p1.5.m3.1.1">ℒ</ci><apply id="S2.I2.i3.p1.5.m3.2.2.1.cmml" xref="S2.I2.i3.p1.5.m3.2.2.1"><eq id="S2.I2.i3.p1.5.m3.2.2.1.1.cmml" xref="S2.I2.i3.p1.5.m3.2.2.1.1"></eq><ci id="S2.I2.i3.p1.5.m3.2.2.1.2.cmml" xref="S2.I2.i3.p1.5.m3.2.2.1.2">𝑎</ci><cn type="integer" id="S2.I2.i3.p1.5.m3.2.2.1.3.cmml" xref="S2.I2.i3.p1.5.m3.2.2.1.3">0</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i3.p1.5.m3.2c">\mathcal{L}|_{a=0}</annotation></semantics></math> obtained by setting <math id="S2.I2.i3.p1.6.m4.1" class="ltx_Math" alttext="a=0" display="inline"><semantics id="S2.I2.i3.p1.6.m4.1a"><mrow id="S2.I2.i3.p1.6.m4.1.1" xref="S2.I2.i3.p1.6.m4.1.1.cmml"><mi id="S2.I2.i3.p1.6.m4.1.1.2" xref="S2.I2.i3.p1.6.m4.1.1.2.cmml">a</mi><mo id="S2.I2.i3.p1.6.m4.1.1.1" xref="S2.I2.i3.p1.6.m4.1.1.1.cmml">=</mo><mn id="S2.I2.i3.p1.6.m4.1.1.3" xref="S2.I2.i3.p1.6.m4.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.I2.i3.p1.6.m4.1b"><apply id="S2.I2.i3.p1.6.m4.1.1.cmml" xref="S2.I2.i3.p1.6.m4.1.1"><eq id="S2.I2.i3.p1.6.m4.1.1.1.cmml" xref="S2.I2.i3.p1.6.m4.1.1.1"></eq><ci id="S2.I2.i3.p1.6.m4.1.1.2.cmml" xref="S2.I2.i3.p1.6.m4.1.1.2">𝑎</ci><cn type="integer" id="S2.I2.i3.p1.6.m4.1.1.3.cmml" xref="S2.I2.i3.p1.6.m4.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i3.p1.6.m4.1c">a=0</annotation></semantics></math> (i.e. causal intervention).
Similarly, the weight of each MLP edge <math id="S2.I2.i3.p1.7.m5.2" class="ltx_Math" alttext="e^{l,m}_{i}" display="inline"><semantics id="S2.I2.i3.p1.7.m5.2a"><msubsup id="S2.I2.i3.p1.7.m5.2.3" xref="S2.I2.i3.p1.7.m5.2.3.cmml"><mi id="S2.I2.i3.p1.7.m5.2.3.2.2" xref="S2.I2.i3.p1.7.m5.2.3.2.2.cmml">e</mi><mi id="S2.I2.i3.p1.7.m5.2.3.3" xref="S2.I2.i3.p1.7.m5.2.3.3.cmml">i</mi><mrow id="S2.I2.i3.p1.7.m5.2.2.2.4" xref="S2.I2.i3.p1.7.m5.2.2.2.3.cmml"><mi id="S2.I2.i3.p1.7.m5.1.1.1.1" xref="S2.I2.i3.p1.7.m5.1.1.1.1.cmml">l</mi><mo id="S2.I2.i3.p1.7.m5.2.2.2.4.1" xref="S2.I2.i3.p1.7.m5.2.2.2.3.cmml">,</mo><mi id="S2.I2.i3.p1.7.m5.2.2.2.2" xref="S2.I2.i3.p1.7.m5.2.2.2.2.cmml">m</mi></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S2.I2.i3.p1.7.m5.2b"><apply id="S2.I2.i3.p1.7.m5.2.3.cmml" xref="S2.I2.i3.p1.7.m5.2.3"><csymbol cd="ambiguous" id="S2.I2.i3.p1.7.m5.2.3.1.cmml" xref="S2.I2.i3.p1.7.m5.2.3">subscript</csymbol><apply id="S2.I2.i3.p1.7.m5.2.3.2.cmml" xref="S2.I2.i3.p1.7.m5.2.3"><csymbol cd="ambiguous" id="S2.I2.i3.p1.7.m5.2.3.2.1.cmml" xref="S2.I2.i3.p1.7.m5.2.3">superscript</csymbol><ci id="S2.I2.i3.p1.7.m5.2.3.2.2.cmml" xref="S2.I2.i3.p1.7.m5.2.3.2.2">𝑒</ci><list id="S2.I2.i3.p1.7.m5.2.2.2.3.cmml" xref="S2.I2.i3.p1.7.m5.2.2.2.4"><ci id="S2.I2.i3.p1.7.m5.1.1.1.1.cmml" xref="S2.I2.i3.p1.7.m5.1.1.1.1">𝑙</ci><ci id="S2.I2.i3.p1.7.m5.2.2.2.2.cmml" xref="S2.I2.i3.p1.7.m5.2.2.2.2">𝑚</ci></list></apply><ci id="S2.I2.i3.p1.7.m5.2.3.3.cmml" xref="S2.I2.i3.p1.7.m5.2.3.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i3.p1.7.m5.2c">e^{l,m}_{i}</annotation></semantics></math>, which measures the influence of the neuron activation <math id="S2.I2.i3.p1.8.m6.2" class="ltx_Math" alttext="a^{l,m}_{i}" display="inline"><semantics id="S2.I2.i3.p1.8.m6.2a"><msubsup id="S2.I2.i3.p1.8.m6.2.3" xref="S2.I2.i3.p1.8.m6.2.3.cmml"><mi id="S2.I2.i3.p1.8.m6.2.3.2.2" xref="S2.I2.i3.p1.8.m6.2.3.2.2.cmml">a</mi><mi id="S2.I2.i3.p1.8.m6.2.3.3" xref="S2.I2.i3.p1.8.m6.2.3.3.cmml">i</mi><mrow id="S2.I2.i3.p1.8.m6.2.2.2.4" xref="S2.I2.i3.p1.8.m6.2.2.2.3.cmml"><mi id="S2.I2.i3.p1.8.m6.1.1.1.1" xref="S2.I2.i3.p1.8.m6.1.1.1.1.cmml">l</mi><mo id="S2.I2.i3.p1.8.m6.2.2.2.4.1" xref="S2.I2.i3.p1.8.m6.2.2.2.3.cmml">,</mo><mi id="S2.I2.i3.p1.8.m6.2.2.2.2" xref="S2.I2.i3.p1.8.m6.2.2.2.2.cmml">m</mi></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S2.I2.i3.p1.8.m6.2b"><apply id="S2.I2.i3.p1.8.m6.2.3.cmml" xref="S2.I2.i3.p1.8.m6.2.3"><csymbol cd="ambiguous" id="S2.I2.i3.p1.8.m6.2.3.1.cmml" xref="S2.I2.i3.p1.8.m6.2.3">subscript</csymbol><apply id="S2.I2.i3.p1.8.m6.2.3.2.cmml" xref="S2.I2.i3.p1.8.m6.2.3"><csymbol cd="ambiguous" id="S2.I2.i3.p1.8.m6.2.3.2.1.cmml" xref="S2.I2.i3.p1.8.m6.2.3">superscript</csymbol><ci id="S2.I2.i3.p1.8.m6.2.3.2.2.cmml" xref="S2.I2.i3.p1.8.m6.2.3.2.2">𝑎</ci><list id="S2.I2.i3.p1.8.m6.2.2.2.3.cmml" xref="S2.I2.i3.p1.8.m6.2.2.2.4"><ci id="S2.I2.i3.p1.8.m6.1.1.1.1.cmml" xref="S2.I2.i3.p1.8.m6.1.1.1.1">𝑙</ci><ci id="S2.I2.i3.p1.8.m6.2.2.2.2.cmml" xref="S2.I2.i3.p1.8.m6.2.2.2.2">𝑚</ci></list></apply><ci id="S2.I2.i3.p1.8.m6.2.3.3.cmml" xref="S2.I2.i3.p1.8.m6.2.3.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i3.p1.8.m6.2c">a^{l,m}_{i}</annotation></semantics></math> on the LLM output, is defined likewise.</p>
</div>
</li>
<li id="S2.I2.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S2.I2.i4.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S2.I2.i4.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S2.I2.i4.1.1.m1.1.1" xref="S2.I2.i4.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S2.I2.i4.1.1.m1.1c"><ci id="S2.I2.i4.1.1.m1.1.1.cmml" xref="S2.I2.i4.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i4.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S2.I2.i4.p1" class="ltx_para">
<p id="S2.I2.i4.p1.8" class="ltx_p">Given any subgraph <math id="S2.I2.i4.p1.1.m1.1" class="ltx_Math" alttext="S\subseteq G" display="inline"><semantics id="S2.I2.i4.p1.1.m1.1a"><mrow id="S2.I2.i4.p1.1.m1.1.1" xref="S2.I2.i4.p1.1.m1.1.1.cmml"><mi id="S2.I2.i4.p1.1.m1.1.1.2" xref="S2.I2.i4.p1.1.m1.1.1.2.cmml">S</mi><mo id="S2.I2.i4.p1.1.m1.1.1.1" xref="S2.I2.i4.p1.1.m1.1.1.1.cmml">⊆</mo><mi id="S2.I2.i4.p1.1.m1.1.1.3" xref="S2.I2.i4.p1.1.m1.1.1.3.cmml">G</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.I2.i4.p1.1.m1.1b"><apply id="S2.I2.i4.p1.1.m1.1.1.cmml" xref="S2.I2.i4.p1.1.m1.1.1"><subset id="S2.I2.i4.p1.1.m1.1.1.1.cmml" xref="S2.I2.i4.p1.1.m1.1.1.1"></subset><ci id="S2.I2.i4.p1.1.m1.1.1.2.cmml" xref="S2.I2.i4.p1.1.m1.1.1.2">𝑆</ci><ci id="S2.I2.i4.p1.1.m1.1.1.3.cmml" xref="S2.I2.i4.p1.1.m1.1.1.3">𝐺</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i4.p1.1.m1.1c">S\subseteq G</annotation></semantics></math>, define the <span id="S2.I2.i4.p1.8.1" class="ltx_text ltx_font_bold">associated input</span> of <math id="S2.I2.i4.p1.2.m2.1" class="ltx_Math" alttext="S" display="inline"><semantics id="S2.I2.i4.p1.2.m2.1a"><mi id="S2.I2.i4.p1.2.m2.1.1" xref="S2.I2.i4.p1.2.m2.1.1.cmml">S</mi><annotation-xml encoding="MathML-Content" id="S2.I2.i4.p1.2.m2.1b"><ci id="S2.I2.i4.p1.2.m2.1.1.cmml" xref="S2.I2.i4.p1.2.m2.1.1">𝑆</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i4.p1.2.m2.1c">S</annotation></semantics></math> as a subsequence <math id="S2.I2.i4.p1.3.m3.3" class="ltx_Math" alttext="\mathbf{t}_{\text{in}}(S)\subseteq(t_{0},\dots t_{n-1})" display="inline"><semantics id="S2.I2.i4.p1.3.m3.3a"><mrow id="S2.I2.i4.p1.3.m3.3.3" xref="S2.I2.i4.p1.3.m3.3.3.cmml"><mrow id="S2.I2.i4.p1.3.m3.3.3.4" xref="S2.I2.i4.p1.3.m3.3.3.4.cmml"><msub id="S2.I2.i4.p1.3.m3.3.3.4.2" xref="S2.I2.i4.p1.3.m3.3.3.4.2.cmml"><mi id="S2.I2.i4.p1.3.m3.3.3.4.2.2" xref="S2.I2.i4.p1.3.m3.3.3.4.2.2.cmml">𝐭</mi><mtext id="S2.I2.i4.p1.3.m3.3.3.4.2.3" xref="S2.I2.i4.p1.3.m3.3.3.4.2.3a.cmml">in</mtext></msub><mo lspace="0em" rspace="0em" id="S2.I2.i4.p1.3.m3.3.3.4.1" xref="S2.I2.i4.p1.3.m3.3.3.4.1.cmml">​</mo><mrow id="S2.I2.i4.p1.3.m3.3.3.4.3.2" xref="S2.I2.i4.p1.3.m3.3.3.4.cmml"><mo stretchy="false" id="S2.I2.i4.p1.3.m3.3.3.4.3.2.1" xref="S2.I2.i4.p1.3.m3.3.3.4.cmml">(</mo><mi id="S2.I2.i4.p1.3.m3.1.1" xref="S2.I2.i4.p1.3.m3.1.1.cmml">S</mi><mo stretchy="false" id="S2.I2.i4.p1.3.m3.3.3.4.3.2.2" xref="S2.I2.i4.p1.3.m3.3.3.4.cmml">)</mo></mrow></mrow><mo id="S2.I2.i4.p1.3.m3.3.3.3" xref="S2.I2.i4.p1.3.m3.3.3.3.cmml">⊆</mo><mrow id="S2.I2.i4.p1.3.m3.3.3.2.2" xref="S2.I2.i4.p1.3.m3.3.3.2.3.cmml"><mo stretchy="false" id="S2.I2.i4.p1.3.m3.3.3.2.2.3" xref="S2.I2.i4.p1.3.m3.3.3.2.3.cmml">(</mo><msub id="S2.I2.i4.p1.3.m3.2.2.1.1.1" xref="S2.I2.i4.p1.3.m3.2.2.1.1.1.cmml"><mi id="S2.I2.i4.p1.3.m3.2.2.1.1.1.2" xref="S2.I2.i4.p1.3.m3.2.2.1.1.1.2.cmml">t</mi><mn id="S2.I2.i4.p1.3.m3.2.2.1.1.1.3" xref="S2.I2.i4.p1.3.m3.2.2.1.1.1.3.cmml">0</mn></msub><mo id="S2.I2.i4.p1.3.m3.3.3.2.2.4" xref="S2.I2.i4.p1.3.m3.3.3.2.3.cmml">,</mo><mrow id="S2.I2.i4.p1.3.m3.3.3.2.2.2" xref="S2.I2.i4.p1.3.m3.3.3.2.2.2.cmml"><mi mathvariant="normal" id="S2.I2.i4.p1.3.m3.3.3.2.2.2.2" xref="S2.I2.i4.p1.3.m3.3.3.2.2.2.2.cmml">…</mi><mo lspace="0em" rspace="0em" id="S2.I2.i4.p1.3.m3.3.3.2.2.2.1" xref="S2.I2.i4.p1.3.m3.3.3.2.2.2.1.cmml">​</mo><msub id="S2.I2.i4.p1.3.m3.3.3.2.2.2.3" xref="S2.I2.i4.p1.3.m3.3.3.2.2.2.3.cmml"><mi id="S2.I2.i4.p1.3.m3.3.3.2.2.2.3.2" xref="S2.I2.i4.p1.3.m3.3.3.2.2.2.3.2.cmml">t</mi><mrow id="S2.I2.i4.p1.3.m3.3.3.2.2.2.3.3" xref="S2.I2.i4.p1.3.m3.3.3.2.2.2.3.3.cmml"><mi id="S2.I2.i4.p1.3.m3.3.3.2.2.2.3.3.2" xref="S2.I2.i4.p1.3.m3.3.3.2.2.2.3.3.2.cmml">n</mi><mo id="S2.I2.i4.p1.3.m3.3.3.2.2.2.3.3.1" xref="S2.I2.i4.p1.3.m3.3.3.2.2.2.3.3.1.cmml">−</mo><mn id="S2.I2.i4.p1.3.m3.3.3.2.2.2.3.3.3" xref="S2.I2.i4.p1.3.m3.3.3.2.2.2.3.3.3.cmml">1</mn></mrow></msub></mrow><mo stretchy="false" id="S2.I2.i4.p1.3.m3.3.3.2.2.5" xref="S2.I2.i4.p1.3.m3.3.3.2.3.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.I2.i4.p1.3.m3.3b"><apply id="S2.I2.i4.p1.3.m3.3.3.cmml" xref="S2.I2.i4.p1.3.m3.3.3"><subset id="S2.I2.i4.p1.3.m3.3.3.3.cmml" xref="S2.I2.i4.p1.3.m3.3.3.3"></subset><apply id="S2.I2.i4.p1.3.m3.3.3.4.cmml" xref="S2.I2.i4.p1.3.m3.3.3.4"><times id="S2.I2.i4.p1.3.m3.3.3.4.1.cmml" xref="S2.I2.i4.p1.3.m3.3.3.4.1"></times><apply id="S2.I2.i4.p1.3.m3.3.3.4.2.cmml" xref="S2.I2.i4.p1.3.m3.3.3.4.2"><csymbol cd="ambiguous" id="S2.I2.i4.p1.3.m3.3.3.4.2.1.cmml" xref="S2.I2.i4.p1.3.m3.3.3.4.2">subscript</csymbol><ci id="S2.I2.i4.p1.3.m3.3.3.4.2.2.cmml" xref="S2.I2.i4.p1.3.m3.3.3.4.2.2">𝐭</ci><ci id="S2.I2.i4.p1.3.m3.3.3.4.2.3a.cmml" xref="S2.I2.i4.p1.3.m3.3.3.4.2.3"><mtext mathsize="70%" id="S2.I2.i4.p1.3.m3.3.3.4.2.3.cmml" xref="S2.I2.i4.p1.3.m3.3.3.4.2.3">in</mtext></ci></apply><ci id="S2.I2.i4.p1.3.m3.1.1.cmml" xref="S2.I2.i4.p1.3.m3.1.1">𝑆</ci></apply><interval closure="open" id="S2.I2.i4.p1.3.m3.3.3.2.3.cmml" xref="S2.I2.i4.p1.3.m3.3.3.2.2"><apply id="S2.I2.i4.p1.3.m3.2.2.1.1.1.cmml" xref="S2.I2.i4.p1.3.m3.2.2.1.1.1"><csymbol cd="ambiguous" id="S2.I2.i4.p1.3.m3.2.2.1.1.1.1.cmml" xref="S2.I2.i4.p1.3.m3.2.2.1.1.1">subscript</csymbol><ci id="S2.I2.i4.p1.3.m3.2.2.1.1.1.2.cmml" xref="S2.I2.i4.p1.3.m3.2.2.1.1.1.2">𝑡</ci><cn type="integer" id="S2.I2.i4.p1.3.m3.2.2.1.1.1.3.cmml" xref="S2.I2.i4.p1.3.m3.2.2.1.1.1.3">0</cn></apply><apply id="S2.I2.i4.p1.3.m3.3.3.2.2.2.cmml" xref="S2.I2.i4.p1.3.m3.3.3.2.2.2"><times id="S2.I2.i4.p1.3.m3.3.3.2.2.2.1.cmml" xref="S2.I2.i4.p1.3.m3.3.3.2.2.2.1"></times><ci id="S2.I2.i4.p1.3.m3.3.3.2.2.2.2.cmml" xref="S2.I2.i4.p1.3.m3.3.3.2.2.2.2">…</ci><apply id="S2.I2.i4.p1.3.m3.3.3.2.2.2.3.cmml" xref="S2.I2.i4.p1.3.m3.3.3.2.2.2.3"><csymbol cd="ambiguous" id="S2.I2.i4.p1.3.m3.3.3.2.2.2.3.1.cmml" xref="S2.I2.i4.p1.3.m3.3.3.2.2.2.3">subscript</csymbol><ci id="S2.I2.i4.p1.3.m3.3.3.2.2.2.3.2.cmml" xref="S2.I2.i4.p1.3.m3.3.3.2.2.2.3.2">𝑡</ci><apply id="S2.I2.i4.p1.3.m3.3.3.2.2.2.3.3.cmml" xref="S2.I2.i4.p1.3.m3.3.3.2.2.2.3.3"><minus id="S2.I2.i4.p1.3.m3.3.3.2.2.2.3.3.1.cmml" xref="S2.I2.i4.p1.3.m3.3.3.2.2.2.3.3.1"></minus><ci id="S2.I2.i4.p1.3.m3.3.3.2.2.2.3.3.2.cmml" xref="S2.I2.i4.p1.3.m3.3.3.2.2.2.3.3.2">𝑛</ci><cn type="integer" id="S2.I2.i4.p1.3.m3.3.3.2.2.2.3.3.3.cmml" xref="S2.I2.i4.p1.3.m3.3.3.2.2.2.3.3.3">1</cn></apply></apply></apply></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i4.p1.3.m3.3c">\mathbf{t}_{\text{in}}(S)\subseteq(t_{0},\dots t_{n-1})</annotation></semantics></math> such that a token <math id="S2.I2.i4.p1.4.m4.1" class="ltx_Math" alttext="t_{i}" display="inline"><semantics id="S2.I2.i4.p1.4.m4.1a"><msub id="S2.I2.i4.p1.4.m4.1.1" xref="S2.I2.i4.p1.4.m4.1.1.cmml"><mi id="S2.I2.i4.p1.4.m4.1.1.2" xref="S2.I2.i4.p1.4.m4.1.1.2.cmml">t</mi><mi id="S2.I2.i4.p1.4.m4.1.1.3" xref="S2.I2.i4.p1.4.m4.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.I2.i4.p1.4.m4.1b"><apply id="S2.I2.i4.p1.4.m4.1.1.cmml" xref="S2.I2.i4.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S2.I2.i4.p1.4.m4.1.1.1.cmml" xref="S2.I2.i4.p1.4.m4.1.1">subscript</csymbol><ci id="S2.I2.i4.p1.4.m4.1.1.2.cmml" xref="S2.I2.i4.p1.4.m4.1.1.2">𝑡</ci><ci id="S2.I2.i4.p1.4.m4.1.1.3.cmml" xref="S2.I2.i4.p1.4.m4.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i4.p1.4.m4.1c">t_{i}</annotation></semantics></math> belongs to <math id="S2.I2.i4.p1.5.m5.1" class="ltx_Math" alttext="\mathbf{t}_{\text{in}}(S)" display="inline"><semantics id="S2.I2.i4.p1.5.m5.1a"><mrow id="S2.I2.i4.p1.5.m5.1.2" xref="S2.I2.i4.p1.5.m5.1.2.cmml"><msub id="S2.I2.i4.p1.5.m5.1.2.2" xref="S2.I2.i4.p1.5.m5.1.2.2.cmml"><mi id="S2.I2.i4.p1.5.m5.1.2.2.2" xref="S2.I2.i4.p1.5.m5.1.2.2.2.cmml">𝐭</mi><mtext id="S2.I2.i4.p1.5.m5.1.2.2.3" xref="S2.I2.i4.p1.5.m5.1.2.2.3a.cmml">in</mtext></msub><mo lspace="0em" rspace="0em" id="S2.I2.i4.p1.5.m5.1.2.1" xref="S2.I2.i4.p1.5.m5.1.2.1.cmml">​</mo><mrow id="S2.I2.i4.p1.5.m5.1.2.3.2" xref="S2.I2.i4.p1.5.m5.1.2.cmml"><mo stretchy="false" id="S2.I2.i4.p1.5.m5.1.2.3.2.1" xref="S2.I2.i4.p1.5.m5.1.2.cmml">(</mo><mi id="S2.I2.i4.p1.5.m5.1.1" xref="S2.I2.i4.p1.5.m5.1.1.cmml">S</mi><mo stretchy="false" id="S2.I2.i4.p1.5.m5.1.2.3.2.2" xref="S2.I2.i4.p1.5.m5.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.I2.i4.p1.5.m5.1b"><apply id="S2.I2.i4.p1.5.m5.1.2.cmml" xref="S2.I2.i4.p1.5.m5.1.2"><times id="S2.I2.i4.p1.5.m5.1.2.1.cmml" xref="S2.I2.i4.p1.5.m5.1.2.1"></times><apply id="S2.I2.i4.p1.5.m5.1.2.2.cmml" xref="S2.I2.i4.p1.5.m5.1.2.2"><csymbol cd="ambiguous" id="S2.I2.i4.p1.5.m5.1.2.2.1.cmml" xref="S2.I2.i4.p1.5.m5.1.2.2">subscript</csymbol><ci id="S2.I2.i4.p1.5.m5.1.2.2.2.cmml" xref="S2.I2.i4.p1.5.m5.1.2.2.2">𝐭</ci><ci id="S2.I2.i4.p1.5.m5.1.2.2.3a.cmml" xref="S2.I2.i4.p1.5.m5.1.2.2.3"><mtext mathsize="70%" id="S2.I2.i4.p1.5.m5.1.2.2.3.cmml" xref="S2.I2.i4.p1.5.m5.1.2.2.3">in</mtext></ci></apply><ci id="S2.I2.i4.p1.5.m5.1.1.cmml" xref="S2.I2.i4.p1.5.m5.1.1">𝑆</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i4.p1.5.m5.1c">\mathbf{t}_{\text{in}}(S)</annotation></semantics></math> if and only if <math id="S2.I2.i4.p1.6.m6.1" class="ltx_Math" alttext="\big{\|}\nabla_{\mathbf{x}_{i}^{0}}a\big{\|}" display="inline"><semantics id="S2.I2.i4.p1.6.m6.1a"><mrow id="S2.I2.i4.p1.6.m6.1.1.1" xref="S2.I2.i4.p1.6.m6.1.1.2.cmml"><mo maxsize="120%" minsize="120%" id="S2.I2.i4.p1.6.m6.1.1.1.2" xref="S2.I2.i4.p1.6.m6.1.1.2.1.cmml">‖</mo><mrow id="S2.I2.i4.p1.6.m6.1.1.1.1" xref="S2.I2.i4.p1.6.m6.1.1.1.1.cmml"><msub id="S2.I2.i4.p1.6.m6.1.1.1.1.1" xref="S2.I2.i4.p1.6.m6.1.1.1.1.1.cmml"><mo rspace="0.167em" id="S2.I2.i4.p1.6.m6.1.1.1.1.1.2" xref="S2.I2.i4.p1.6.m6.1.1.1.1.1.2.cmml">∇</mo><msubsup id="S2.I2.i4.p1.6.m6.1.1.1.1.1.3" xref="S2.I2.i4.p1.6.m6.1.1.1.1.1.3.cmml"><mi id="S2.I2.i4.p1.6.m6.1.1.1.1.1.3.2.2" xref="S2.I2.i4.p1.6.m6.1.1.1.1.1.3.2.2.cmml">𝐱</mi><mi id="S2.I2.i4.p1.6.m6.1.1.1.1.1.3.2.3" xref="S2.I2.i4.p1.6.m6.1.1.1.1.1.3.2.3.cmml">i</mi><mn id="S2.I2.i4.p1.6.m6.1.1.1.1.1.3.3" xref="S2.I2.i4.p1.6.m6.1.1.1.1.1.3.3.cmml">0</mn></msubsup></msub><mi id="S2.I2.i4.p1.6.m6.1.1.1.1.2" xref="S2.I2.i4.p1.6.m6.1.1.1.1.2.cmml">a</mi></mrow><mo maxsize="120%" minsize="120%" id="S2.I2.i4.p1.6.m6.1.1.1.3" xref="S2.I2.i4.p1.6.m6.1.1.2.1.cmml">‖</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.I2.i4.p1.6.m6.1b"><apply id="S2.I2.i4.p1.6.m6.1.1.2.cmml" xref="S2.I2.i4.p1.6.m6.1.1.1"><csymbol cd="latexml" id="S2.I2.i4.p1.6.m6.1.1.2.1.cmml" xref="S2.I2.i4.p1.6.m6.1.1.1.2">norm</csymbol><apply id="S2.I2.i4.p1.6.m6.1.1.1.1.cmml" xref="S2.I2.i4.p1.6.m6.1.1.1.1"><apply id="S2.I2.i4.p1.6.m6.1.1.1.1.1.cmml" xref="S2.I2.i4.p1.6.m6.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.I2.i4.p1.6.m6.1.1.1.1.1.1.cmml" xref="S2.I2.i4.p1.6.m6.1.1.1.1.1">subscript</csymbol><ci id="S2.I2.i4.p1.6.m6.1.1.1.1.1.2.cmml" xref="S2.I2.i4.p1.6.m6.1.1.1.1.1.2">∇</ci><apply id="S2.I2.i4.p1.6.m6.1.1.1.1.1.3.cmml" xref="S2.I2.i4.p1.6.m6.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.I2.i4.p1.6.m6.1.1.1.1.1.3.1.cmml" xref="S2.I2.i4.p1.6.m6.1.1.1.1.1.3">superscript</csymbol><apply id="S2.I2.i4.p1.6.m6.1.1.1.1.1.3.2.cmml" xref="S2.I2.i4.p1.6.m6.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.I2.i4.p1.6.m6.1.1.1.1.1.3.2.1.cmml" xref="S2.I2.i4.p1.6.m6.1.1.1.1.1.3">subscript</csymbol><ci id="S2.I2.i4.p1.6.m6.1.1.1.1.1.3.2.2.cmml" xref="S2.I2.i4.p1.6.m6.1.1.1.1.1.3.2.2">𝐱</ci><ci id="S2.I2.i4.p1.6.m6.1.1.1.1.1.3.2.3.cmml" xref="S2.I2.i4.p1.6.m6.1.1.1.1.1.3.2.3">𝑖</ci></apply><cn type="integer" id="S2.I2.i4.p1.6.m6.1.1.1.1.1.3.3.cmml" xref="S2.I2.i4.p1.6.m6.1.1.1.1.1.3.3">0</cn></apply></apply><ci id="S2.I2.i4.p1.6.m6.1.1.1.1.2.cmml" xref="S2.I2.i4.p1.6.m6.1.1.1.1.2">𝑎</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i4.p1.6.m6.1c">\big{\|}\nabla_{\mathbf{x}_{i}^{0}}a\big{\|}</annotation></semantics></math> is large for some attention edge (or MLP edge) in <math id="S2.I2.i4.p1.7.m7.1" class="ltx_Math" alttext="S" display="inline"><semantics id="S2.I2.i4.p1.7.m7.1a"><mi id="S2.I2.i4.p1.7.m7.1.1" xref="S2.I2.i4.p1.7.m7.1.1.cmml">S</mi><annotation-xml encoding="MathML-Content" id="S2.I2.i4.p1.7.m7.1b"><ci id="S2.I2.i4.p1.7.m7.1.1.cmml" xref="S2.I2.i4.p1.7.m7.1.1">𝑆</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i4.p1.7.m7.1c">S</annotation></semantics></math> with attention score (or activation) <math id="S2.I2.i4.p1.8.m8.1" class="ltx_Math" alttext="a" display="inline"><semantics id="S2.I2.i4.p1.8.m8.1a"><mi id="S2.I2.i4.p1.8.m8.1.1" xref="S2.I2.i4.p1.8.m8.1.1.cmml">a</mi><annotation-xml encoding="MathML-Content" id="S2.I2.i4.p1.8.m8.1b"><ci id="S2.I2.i4.p1.8.m8.1.1.cmml" xref="S2.I2.i4.p1.8.m8.1.1">𝑎</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i4.p1.8.m8.1c">a</annotation></semantics></math>.</p>
</div>
</li>
<li id="S2.I2.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S2.I2.i5.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S2.I2.i5.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S2.I2.i5.1.1.m1.1.1" xref="S2.I2.i5.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S2.I2.i5.1.1.m1.1c"><ci id="S2.I2.i5.1.1.m1.1.1.cmml" xref="S2.I2.i5.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i5.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S2.I2.i5.p1" class="ltx_para">
<p id="S2.I2.i5.p1.4" class="ltx_p">Similarly, define the <span id="S2.I2.i5.p1.4.1" class="ltx_text ltx_font_bold">associated output</span> of the subgraph <math id="S2.I2.i5.p1.1.m1.1" class="ltx_Math" alttext="S" display="inline"><semantics id="S2.I2.i5.p1.1.m1.1a"><mi id="S2.I2.i5.p1.1.m1.1.1" xref="S2.I2.i5.p1.1.m1.1.1.cmml">S</mi><annotation-xml encoding="MathML-Content" id="S2.I2.i5.p1.1.m1.1b"><ci id="S2.I2.i5.p1.1.m1.1.1.cmml" xref="S2.I2.i5.p1.1.m1.1.1">𝑆</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i5.p1.1.m1.1c">S</annotation></semantics></math> as a subsequence <math id="S2.I2.i5.p1.2.m2.3" class="ltx_Math" alttext="\mathbf{t}_{\text{out}}(S)\subseteq(t_{1},\dots t_{n})" display="inline"><semantics id="S2.I2.i5.p1.2.m2.3a"><mrow id="S2.I2.i5.p1.2.m2.3.3" xref="S2.I2.i5.p1.2.m2.3.3.cmml"><mrow id="S2.I2.i5.p1.2.m2.3.3.4" xref="S2.I2.i5.p1.2.m2.3.3.4.cmml"><msub id="S2.I2.i5.p1.2.m2.3.3.4.2" xref="S2.I2.i5.p1.2.m2.3.3.4.2.cmml"><mi id="S2.I2.i5.p1.2.m2.3.3.4.2.2" xref="S2.I2.i5.p1.2.m2.3.3.4.2.2.cmml">𝐭</mi><mtext id="S2.I2.i5.p1.2.m2.3.3.4.2.3" xref="S2.I2.i5.p1.2.m2.3.3.4.2.3a.cmml">out</mtext></msub><mo lspace="0em" rspace="0em" id="S2.I2.i5.p1.2.m2.3.3.4.1" xref="S2.I2.i5.p1.2.m2.3.3.4.1.cmml">​</mo><mrow id="S2.I2.i5.p1.2.m2.3.3.4.3.2" xref="S2.I2.i5.p1.2.m2.3.3.4.cmml"><mo stretchy="false" id="S2.I2.i5.p1.2.m2.3.3.4.3.2.1" xref="S2.I2.i5.p1.2.m2.3.3.4.cmml">(</mo><mi id="S2.I2.i5.p1.2.m2.1.1" xref="S2.I2.i5.p1.2.m2.1.1.cmml">S</mi><mo stretchy="false" id="S2.I2.i5.p1.2.m2.3.3.4.3.2.2" xref="S2.I2.i5.p1.2.m2.3.3.4.cmml">)</mo></mrow></mrow><mo id="S2.I2.i5.p1.2.m2.3.3.3" xref="S2.I2.i5.p1.2.m2.3.3.3.cmml">⊆</mo><mrow id="S2.I2.i5.p1.2.m2.3.3.2.2" xref="S2.I2.i5.p1.2.m2.3.3.2.3.cmml"><mo stretchy="false" id="S2.I2.i5.p1.2.m2.3.3.2.2.3" xref="S2.I2.i5.p1.2.m2.3.3.2.3.cmml">(</mo><msub id="S2.I2.i5.p1.2.m2.2.2.1.1.1" xref="S2.I2.i5.p1.2.m2.2.2.1.1.1.cmml"><mi id="S2.I2.i5.p1.2.m2.2.2.1.1.1.2" xref="S2.I2.i5.p1.2.m2.2.2.1.1.1.2.cmml">t</mi><mn id="S2.I2.i5.p1.2.m2.2.2.1.1.1.3" xref="S2.I2.i5.p1.2.m2.2.2.1.1.1.3.cmml">1</mn></msub><mo id="S2.I2.i5.p1.2.m2.3.3.2.2.4" xref="S2.I2.i5.p1.2.m2.3.3.2.3.cmml">,</mo><mrow id="S2.I2.i5.p1.2.m2.3.3.2.2.2" xref="S2.I2.i5.p1.2.m2.3.3.2.2.2.cmml"><mi mathvariant="normal" id="S2.I2.i5.p1.2.m2.3.3.2.2.2.2" xref="S2.I2.i5.p1.2.m2.3.3.2.2.2.2.cmml">…</mi><mo lspace="0em" rspace="0em" id="S2.I2.i5.p1.2.m2.3.3.2.2.2.1" xref="S2.I2.i5.p1.2.m2.3.3.2.2.2.1.cmml">​</mo><msub id="S2.I2.i5.p1.2.m2.3.3.2.2.2.3" xref="S2.I2.i5.p1.2.m2.3.3.2.2.2.3.cmml"><mi id="S2.I2.i5.p1.2.m2.3.3.2.2.2.3.2" xref="S2.I2.i5.p1.2.m2.3.3.2.2.2.3.2.cmml">t</mi><mi id="S2.I2.i5.p1.2.m2.3.3.2.2.2.3.3" xref="S2.I2.i5.p1.2.m2.3.3.2.2.2.3.3.cmml">n</mi></msub></mrow><mo stretchy="false" id="S2.I2.i5.p1.2.m2.3.3.2.2.5" xref="S2.I2.i5.p1.2.m2.3.3.2.3.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.I2.i5.p1.2.m2.3b"><apply id="S2.I2.i5.p1.2.m2.3.3.cmml" xref="S2.I2.i5.p1.2.m2.3.3"><subset id="S2.I2.i5.p1.2.m2.3.3.3.cmml" xref="S2.I2.i5.p1.2.m2.3.3.3"></subset><apply id="S2.I2.i5.p1.2.m2.3.3.4.cmml" xref="S2.I2.i5.p1.2.m2.3.3.4"><times id="S2.I2.i5.p1.2.m2.3.3.4.1.cmml" xref="S2.I2.i5.p1.2.m2.3.3.4.1"></times><apply id="S2.I2.i5.p1.2.m2.3.3.4.2.cmml" xref="S2.I2.i5.p1.2.m2.3.3.4.2"><csymbol cd="ambiguous" id="S2.I2.i5.p1.2.m2.3.3.4.2.1.cmml" xref="S2.I2.i5.p1.2.m2.3.3.4.2">subscript</csymbol><ci id="S2.I2.i5.p1.2.m2.3.3.4.2.2.cmml" xref="S2.I2.i5.p1.2.m2.3.3.4.2.2">𝐭</ci><ci id="S2.I2.i5.p1.2.m2.3.3.4.2.3a.cmml" xref="S2.I2.i5.p1.2.m2.3.3.4.2.3"><mtext mathsize="70%" id="S2.I2.i5.p1.2.m2.3.3.4.2.3.cmml" xref="S2.I2.i5.p1.2.m2.3.3.4.2.3">out</mtext></ci></apply><ci id="S2.I2.i5.p1.2.m2.1.1.cmml" xref="S2.I2.i5.p1.2.m2.1.1">𝑆</ci></apply><interval closure="open" id="S2.I2.i5.p1.2.m2.3.3.2.3.cmml" xref="S2.I2.i5.p1.2.m2.3.3.2.2"><apply id="S2.I2.i5.p1.2.m2.2.2.1.1.1.cmml" xref="S2.I2.i5.p1.2.m2.2.2.1.1.1"><csymbol cd="ambiguous" id="S2.I2.i5.p1.2.m2.2.2.1.1.1.1.cmml" xref="S2.I2.i5.p1.2.m2.2.2.1.1.1">subscript</csymbol><ci id="S2.I2.i5.p1.2.m2.2.2.1.1.1.2.cmml" xref="S2.I2.i5.p1.2.m2.2.2.1.1.1.2">𝑡</ci><cn type="integer" id="S2.I2.i5.p1.2.m2.2.2.1.1.1.3.cmml" xref="S2.I2.i5.p1.2.m2.2.2.1.1.1.3">1</cn></apply><apply id="S2.I2.i5.p1.2.m2.3.3.2.2.2.cmml" xref="S2.I2.i5.p1.2.m2.3.3.2.2.2"><times id="S2.I2.i5.p1.2.m2.3.3.2.2.2.1.cmml" xref="S2.I2.i5.p1.2.m2.3.3.2.2.2.1"></times><ci id="S2.I2.i5.p1.2.m2.3.3.2.2.2.2.cmml" xref="S2.I2.i5.p1.2.m2.3.3.2.2.2.2">…</ci><apply id="S2.I2.i5.p1.2.m2.3.3.2.2.2.3.cmml" xref="S2.I2.i5.p1.2.m2.3.3.2.2.2.3"><csymbol cd="ambiguous" id="S2.I2.i5.p1.2.m2.3.3.2.2.2.3.1.cmml" xref="S2.I2.i5.p1.2.m2.3.3.2.2.2.3">subscript</csymbol><ci id="S2.I2.i5.p1.2.m2.3.3.2.2.2.3.2.cmml" xref="S2.I2.i5.p1.2.m2.3.3.2.2.2.3.2">𝑡</ci><ci id="S2.I2.i5.p1.2.m2.3.3.2.2.2.3.3.cmml" xref="S2.I2.i5.p1.2.m2.3.3.2.2.2.3.3">𝑛</ci></apply></apply></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i5.p1.2.m2.3c">\mathbf{t}_{\text{out}}(S)\subseteq(t_{1},\dots t_{n})</annotation></semantics></math> such that a token <math id="S2.I2.i5.p1.3.m3.1" class="ltx_Math" alttext="t_{i}" display="inline"><semantics id="S2.I2.i5.p1.3.m3.1a"><msub id="S2.I2.i5.p1.3.m3.1.1" xref="S2.I2.i5.p1.3.m3.1.1.cmml"><mi id="S2.I2.i5.p1.3.m3.1.1.2" xref="S2.I2.i5.p1.3.m3.1.1.2.cmml">t</mi><mi id="S2.I2.i5.p1.3.m3.1.1.3" xref="S2.I2.i5.p1.3.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.I2.i5.p1.3.m3.1b"><apply id="S2.I2.i5.p1.3.m3.1.1.cmml" xref="S2.I2.i5.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S2.I2.i5.p1.3.m3.1.1.1.cmml" xref="S2.I2.i5.p1.3.m3.1.1">subscript</csymbol><ci id="S2.I2.i5.p1.3.m3.1.1.2.cmml" xref="S2.I2.i5.p1.3.m3.1.1.2">𝑡</ci><ci id="S2.I2.i5.p1.3.m3.1.1.3.cmml" xref="S2.I2.i5.p1.3.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i5.p1.3.m3.1c">t_{i}</annotation></semantics></math> belongs to <math id="S2.I2.i5.p1.4.m4.1" class="ltx_Math" alttext="\mathbf{t}_{\text{out}}(S)" display="inline"><semantics id="S2.I2.i5.p1.4.m4.1a"><mrow id="S2.I2.i5.p1.4.m4.1.2" xref="S2.I2.i5.p1.4.m4.1.2.cmml"><msub id="S2.I2.i5.p1.4.m4.1.2.2" xref="S2.I2.i5.p1.4.m4.1.2.2.cmml"><mi id="S2.I2.i5.p1.4.m4.1.2.2.2" xref="S2.I2.i5.p1.4.m4.1.2.2.2.cmml">𝐭</mi><mtext id="S2.I2.i5.p1.4.m4.1.2.2.3" xref="S2.I2.i5.p1.4.m4.1.2.2.3a.cmml">out</mtext></msub><mo lspace="0em" rspace="0em" id="S2.I2.i5.p1.4.m4.1.2.1" xref="S2.I2.i5.p1.4.m4.1.2.1.cmml">​</mo><mrow id="S2.I2.i5.p1.4.m4.1.2.3.2" xref="S2.I2.i5.p1.4.m4.1.2.cmml"><mo stretchy="false" id="S2.I2.i5.p1.4.m4.1.2.3.2.1" xref="S2.I2.i5.p1.4.m4.1.2.cmml">(</mo><mi id="S2.I2.i5.p1.4.m4.1.1" xref="S2.I2.i5.p1.4.m4.1.1.cmml">S</mi><mo stretchy="false" id="S2.I2.i5.p1.4.m4.1.2.3.2.2" xref="S2.I2.i5.p1.4.m4.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.I2.i5.p1.4.m4.1b"><apply id="S2.I2.i5.p1.4.m4.1.2.cmml" xref="S2.I2.i5.p1.4.m4.1.2"><times id="S2.I2.i5.p1.4.m4.1.2.1.cmml" xref="S2.I2.i5.p1.4.m4.1.2.1"></times><apply id="S2.I2.i5.p1.4.m4.1.2.2.cmml" xref="S2.I2.i5.p1.4.m4.1.2.2"><csymbol cd="ambiguous" id="S2.I2.i5.p1.4.m4.1.2.2.1.cmml" xref="S2.I2.i5.p1.4.m4.1.2.2">subscript</csymbol><ci id="S2.I2.i5.p1.4.m4.1.2.2.2.cmml" xref="S2.I2.i5.p1.4.m4.1.2.2.2">𝐭</ci><ci id="S2.I2.i5.p1.4.m4.1.2.2.3a.cmml" xref="S2.I2.i5.p1.4.m4.1.2.2.3"><mtext mathsize="70%" id="S2.I2.i5.p1.4.m4.1.2.2.3.cmml" xref="S2.I2.i5.p1.4.m4.1.2.2.3">out</mtext></ci></apply><ci id="S2.I2.i5.p1.4.m4.1.1.cmml" xref="S2.I2.i5.p1.4.m4.1.1">𝑆</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i5.p1.4.m4.1c">\mathbf{t}_{\text{out}}(S)</annotation></semantics></math> if and only if</p>
<table id="Ex2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="Ex2.m1.4" class="ltx_Math" alttext="\mathcal{L}_{i}-\mathcal{L}_{i}\big{|}_{a=0}\quad\text{or}\quad\frac{\partial\mathcal{L}_{i}}{\partial a}" display="block"><semantics id="Ex2.m1.4a"><mrow id="Ex2.m1.4.4.1" xref="Ex2.m1.4.4.2.cmml"><mrow id="Ex2.m1.4.4.1.1" xref="Ex2.m1.4.4.1.1.cmml"><msub id="Ex2.m1.4.4.1.1.3" xref="Ex2.m1.4.4.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="Ex2.m1.4.4.1.1.3.2" xref="Ex2.m1.4.4.1.1.3.2.cmml">ℒ</mi><mi id="Ex2.m1.4.4.1.1.3.3" xref="Ex2.m1.4.4.1.1.3.3.cmml">i</mi></msub><mo id="Ex2.m1.4.4.1.1.2" xref="Ex2.m1.4.4.1.1.2.cmml">−</mo><msub id="Ex2.m1.4.4.1.1.1.1" xref="Ex2.m1.4.4.1.1.1.2.cmml"><mrow id="Ex2.m1.4.4.1.1.1.1.1" xref="Ex2.m1.4.4.1.1.1.2.cmml"><msub id="Ex2.m1.4.4.1.1.1.1.1.1" xref="Ex2.m1.4.4.1.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="Ex2.m1.4.4.1.1.1.1.1.1.2" xref="Ex2.m1.4.4.1.1.1.1.1.1.2.cmml">ℒ</mi><mi id="Ex2.m1.4.4.1.1.1.1.1.1.3" xref="Ex2.m1.4.4.1.1.1.1.1.1.3.cmml">i</mi></msub><mo maxsize="120%" minsize="120%" id="Ex2.m1.4.4.1.1.1.1.1.2" xref="Ex2.m1.4.4.1.1.1.2.1.cmml">|</mo></mrow><mrow id="Ex2.m1.1.1.1" xref="Ex2.m1.1.1.1.cmml"><mi id="Ex2.m1.1.1.1.2" xref="Ex2.m1.1.1.1.2.cmml">a</mi><mo id="Ex2.m1.1.1.1.1" xref="Ex2.m1.1.1.1.1.cmml">=</mo><mn id="Ex2.m1.1.1.1.3" xref="Ex2.m1.1.1.1.3.cmml">0</mn></mrow></msub></mrow><mspace width="1em" id="Ex2.m1.4.4.1.2" xref="Ex2.m1.4.4.2.cmml"></mspace><mtext id="Ex2.m1.2.2" xref="Ex2.m1.2.2a.cmml">or</mtext><mspace width="1em" id="Ex2.m1.4.4.1.3" xref="Ex2.m1.4.4.2.cmml"></mspace><mfrac id="Ex2.m1.3.3" xref="Ex2.m1.3.3.cmml"><mrow id="Ex2.m1.3.3.2" xref="Ex2.m1.3.3.2.cmml"><mo rspace="0em" id="Ex2.m1.3.3.2.1" xref="Ex2.m1.3.3.2.1.cmml">∂</mo><msub id="Ex2.m1.3.3.2.2" xref="Ex2.m1.3.3.2.2.cmml"><mi class="ltx_font_mathcaligraphic" id="Ex2.m1.3.3.2.2.2" xref="Ex2.m1.3.3.2.2.2.cmml">ℒ</mi><mi id="Ex2.m1.3.3.2.2.3" xref="Ex2.m1.3.3.2.2.3.cmml">i</mi></msub></mrow><mrow id="Ex2.m1.3.3.3" xref="Ex2.m1.3.3.3.cmml"><mo rspace="0em" id="Ex2.m1.3.3.3.1" xref="Ex2.m1.3.3.3.1.cmml">∂</mo><mi id="Ex2.m1.3.3.3.2" xref="Ex2.m1.3.3.3.2.cmml">a</mi></mrow></mfrac></mrow><annotation-xml encoding="MathML-Content" id="Ex2.m1.4b"><list id="Ex2.m1.4.4.2.cmml" xref="Ex2.m1.4.4.1"><apply id="Ex2.m1.4.4.1.1.cmml" xref="Ex2.m1.4.4.1.1"><minus id="Ex2.m1.4.4.1.1.2.cmml" xref="Ex2.m1.4.4.1.1.2"></minus><apply id="Ex2.m1.4.4.1.1.3.cmml" xref="Ex2.m1.4.4.1.1.3"><csymbol cd="ambiguous" id="Ex2.m1.4.4.1.1.3.1.cmml" xref="Ex2.m1.4.4.1.1.3">subscript</csymbol><ci id="Ex2.m1.4.4.1.1.3.2.cmml" xref="Ex2.m1.4.4.1.1.3.2">ℒ</ci><ci id="Ex2.m1.4.4.1.1.3.3.cmml" xref="Ex2.m1.4.4.1.1.3.3">𝑖</ci></apply><apply id="Ex2.m1.4.4.1.1.1.2.cmml" xref="Ex2.m1.4.4.1.1.1.1"><csymbol cd="latexml" id="Ex2.m1.4.4.1.1.1.2.1.cmml" xref="Ex2.m1.4.4.1.1.1.1.1.2">evaluated-at</csymbol><apply id="Ex2.m1.4.4.1.1.1.1.1.1.cmml" xref="Ex2.m1.4.4.1.1.1.1.1.1"><csymbol cd="ambiguous" id="Ex2.m1.4.4.1.1.1.1.1.1.1.cmml" xref="Ex2.m1.4.4.1.1.1.1.1.1">subscript</csymbol><ci id="Ex2.m1.4.4.1.1.1.1.1.1.2.cmml" xref="Ex2.m1.4.4.1.1.1.1.1.1.2">ℒ</ci><ci id="Ex2.m1.4.4.1.1.1.1.1.1.3.cmml" xref="Ex2.m1.4.4.1.1.1.1.1.1.3">𝑖</ci></apply><apply id="Ex2.m1.1.1.1.cmml" xref="Ex2.m1.1.1.1"><eq id="Ex2.m1.1.1.1.1.cmml" xref="Ex2.m1.1.1.1.1"></eq><ci id="Ex2.m1.1.1.1.2.cmml" xref="Ex2.m1.1.1.1.2">𝑎</ci><cn type="integer" id="Ex2.m1.1.1.1.3.cmml" xref="Ex2.m1.1.1.1.3">0</cn></apply></apply></apply><ci id="Ex2.m1.2.2a.cmml" xref="Ex2.m1.2.2"><mtext id="Ex2.m1.2.2.cmml" xref="Ex2.m1.2.2">or</mtext></ci><apply id="Ex2.m1.3.3.cmml" xref="Ex2.m1.3.3"><divide id="Ex2.m1.3.3.1.cmml" xref="Ex2.m1.3.3"></divide><apply id="Ex2.m1.3.3.2.cmml" xref="Ex2.m1.3.3.2"><partialdiff id="Ex2.m1.3.3.2.1.cmml" xref="Ex2.m1.3.3.2.1"></partialdiff><apply id="Ex2.m1.3.3.2.2.cmml" xref="Ex2.m1.3.3.2.2"><csymbol cd="ambiguous" id="Ex2.m1.3.3.2.2.1.cmml" xref="Ex2.m1.3.3.2.2">subscript</csymbol><ci id="Ex2.m1.3.3.2.2.2.cmml" xref="Ex2.m1.3.3.2.2.2">ℒ</ci><ci id="Ex2.m1.3.3.2.2.3.cmml" xref="Ex2.m1.3.3.2.2.3">𝑖</ci></apply></apply><apply id="Ex2.m1.3.3.3.cmml" xref="Ex2.m1.3.3.3"><partialdiff id="Ex2.m1.3.3.3.1.cmml" xref="Ex2.m1.3.3.3.1"></partialdiff><ci id="Ex2.m1.3.3.3.2.cmml" xref="Ex2.m1.3.3.3.2">𝑎</ci></apply></apply></list></annotation-xml><annotation encoding="application/x-tex" id="Ex2.m1.4c">\mathcal{L}_{i}-\mathcal{L}_{i}\big{|}_{a=0}\quad\text{or}\quad\frac{\partial\mathcal{L}_{i}}{\partial a}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S2.I2.i5.p1.8" class="ltx_p">is large for some attention edge (or MLP edge) in <math id="S2.I2.i5.p1.5.m1.1" class="ltx_Math" alttext="S" display="inline"><semantics id="S2.I2.i5.p1.5.m1.1a"><mi id="S2.I2.i5.p1.5.m1.1.1" xref="S2.I2.i5.p1.5.m1.1.1.cmml">S</mi><annotation-xml encoding="MathML-Content" id="S2.I2.i5.p1.5.m1.1b"><ci id="S2.I2.i5.p1.5.m1.1.1.cmml" xref="S2.I2.i5.p1.5.m1.1.1">𝑆</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i5.p1.5.m1.1c">S</annotation></semantics></math> with attention score (or activation) <math id="S2.I2.i5.p1.6.m2.1" class="ltx_Math" alttext="a" display="inline"><semantics id="S2.I2.i5.p1.6.m2.1a"><mi id="S2.I2.i5.p1.6.m2.1.1" xref="S2.I2.i5.p1.6.m2.1.1.cmml">a</mi><annotation-xml encoding="MathML-Content" id="S2.I2.i5.p1.6.m2.1b"><ci id="S2.I2.i5.p1.6.m2.1.1.cmml" xref="S2.I2.i5.p1.6.m2.1.1">𝑎</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i5.p1.6.m2.1c">a</annotation></semantics></math>.
Here <math id="S2.I2.i5.p1.7.m3.1" class="ltx_Math" alttext="\mathcal{L}_{i}" display="inline"><semantics id="S2.I2.i5.p1.7.m3.1a"><msub id="S2.I2.i5.p1.7.m3.1.1" xref="S2.I2.i5.p1.7.m3.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S2.I2.i5.p1.7.m3.1.1.2" xref="S2.I2.i5.p1.7.m3.1.1.2.cmml">ℒ</mi><mi id="S2.I2.i5.p1.7.m3.1.1.3" xref="S2.I2.i5.p1.7.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.I2.i5.p1.7.m3.1b"><apply id="S2.I2.i5.p1.7.m3.1.1.cmml" xref="S2.I2.i5.p1.7.m3.1.1"><csymbol cd="ambiguous" id="S2.I2.i5.p1.7.m3.1.1.1.cmml" xref="S2.I2.i5.p1.7.m3.1.1">subscript</csymbol><ci id="S2.I2.i5.p1.7.m3.1.1.2.cmml" xref="S2.I2.i5.p1.7.m3.1.1.2">ℒ</ci><ci id="S2.I2.i5.p1.7.m3.1.1.3.cmml" xref="S2.I2.i5.p1.7.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i5.p1.7.m3.1c">\mathcal{L}_{i}</annotation></semantics></math> is the log-likelihood of <math id="S2.I2.i5.p1.8.m4.1" class="ltx_Math" alttext="t_{i}" display="inline"><semantics id="S2.I2.i5.p1.8.m4.1a"><msub id="S2.I2.i5.p1.8.m4.1.1" xref="S2.I2.i5.p1.8.m4.1.1.cmml"><mi id="S2.I2.i5.p1.8.m4.1.1.2" xref="S2.I2.i5.p1.8.m4.1.1.2.cmml">t</mi><mi id="S2.I2.i5.p1.8.m4.1.1.3" xref="S2.I2.i5.p1.8.m4.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.I2.i5.p1.8.m4.1b"><apply id="S2.I2.i5.p1.8.m4.1.1.cmml" xref="S2.I2.i5.p1.8.m4.1.1"><csymbol cd="ambiguous" id="S2.I2.i5.p1.8.m4.1.1.1.cmml" xref="S2.I2.i5.p1.8.m4.1.1">subscript</csymbol><ci id="S2.I2.i5.p1.8.m4.1.1.2.cmml" xref="S2.I2.i5.p1.8.m4.1.1.2">𝑡</ci><ci id="S2.I2.i5.p1.8.m4.1.1.3.cmml" xref="S2.I2.i5.p1.8.m4.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i5.p1.8.m4.1c">t_{i}</annotation></semantics></math> with respect to the LLM output.</p>
</div>
</li>
</ul>
</div>
</div>
<figure id="F7" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="F7.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.01178/assets/Figures/Theory/LLM_computation_graph.png" id="F7.sf1.g1" class="ltx_graphics ltx_img_landscape" width="153" height="118" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="F7.sf1.2.1.1" class="ltx_text" style="font-size:90%;">(a)</span> </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="F7.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.01178/assets/Figures/Theory/LLM_subgraph_homomorphism.png" id="F7.sf2.g1" class="ltx_graphics ltx_img_landscape" width="431" height="124" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="F7.sf2.2.1.1" class="ltx_text" style="font-size:90%;">(b)</span> </span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="F7.12.6.1" class="ltx_text" style="font-size:90%;">Figure 7</span>: </span><span id="F7.10.5" class="ltx_text" style="font-size:90%;">Left: Illustration of the computation graph over one Transformer block, showing only three tokens, one attention head and three MLP neurons.
The edge weights are not shown.
Right: The subgraphs <math id="F7.6.1.m1.2" class="ltx_Math" alttext="S_{1},S_{2}" display="inline"><semantics id="F7.6.1.m1.2b"><mrow id="F7.6.1.m1.2.2.2" xref="F7.6.1.m1.2.2.3.cmml"><msub id="F7.6.1.m1.1.1.1.1" xref="F7.6.1.m1.1.1.1.1.cmml"><mi id="F7.6.1.m1.1.1.1.1.2" xref="F7.6.1.m1.1.1.1.1.2.cmml">S</mi><mn id="F7.6.1.m1.1.1.1.1.3" xref="F7.6.1.m1.1.1.1.1.3.cmml">1</mn></msub><mo id="F7.6.1.m1.2.2.2.3" xref="F7.6.1.m1.2.2.3.cmml">,</mo><msub id="F7.6.1.m1.2.2.2.2" xref="F7.6.1.m1.2.2.2.2.cmml"><mi id="F7.6.1.m1.2.2.2.2.2" xref="F7.6.1.m1.2.2.2.2.2.cmml">S</mi><mn id="F7.6.1.m1.2.2.2.2.3" xref="F7.6.1.m1.2.2.2.2.3.cmml">2</mn></msub></mrow><annotation-xml encoding="MathML-Content" id="F7.6.1.m1.2c"><list id="F7.6.1.m1.2.2.3.cmml" xref="F7.6.1.m1.2.2.2"><apply id="F7.6.1.m1.1.1.1.1.cmml" xref="F7.6.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="F7.6.1.m1.1.1.1.1.1.cmml" xref="F7.6.1.m1.1.1.1.1">subscript</csymbol><ci id="F7.6.1.m1.1.1.1.1.2.cmml" xref="F7.6.1.m1.1.1.1.1.2">𝑆</ci><cn type="integer" id="F7.6.1.m1.1.1.1.1.3.cmml" xref="F7.6.1.m1.1.1.1.1.3">1</cn></apply><apply id="F7.6.1.m1.2.2.2.2.cmml" xref="F7.6.1.m1.2.2.2.2"><csymbol cd="ambiguous" id="F7.6.1.m1.2.2.2.2.1.cmml" xref="F7.6.1.m1.2.2.2.2">subscript</csymbol><ci id="F7.6.1.m1.2.2.2.2.2.cmml" xref="F7.6.1.m1.2.2.2.2.2">𝑆</ci><cn type="integer" id="F7.6.1.m1.2.2.2.2.3.cmml" xref="F7.6.1.m1.2.2.2.2.3">2</cn></apply></list></annotation-xml><annotation encoding="application/x-tex" id="F7.6.1.m1.2d">S_{1},S_{2}</annotation></semantics></math>, namely the induced subgraphs of the attention edges (black arrows), belong to the circuit of the induction head.
The red arrows denote a homomorphism from <math id="F7.7.2.m2.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="F7.7.2.m2.1b"><msub id="F7.7.2.m2.1.1" xref="F7.7.2.m2.1.1.cmml"><mi id="F7.7.2.m2.1.1.2" xref="F7.7.2.m2.1.1.2.cmml">S</mi><mn id="F7.7.2.m2.1.1.3" xref="F7.7.2.m2.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="F7.7.2.m2.1c"><apply id="F7.7.2.m2.1.1.cmml" xref="F7.7.2.m2.1.1"><csymbol cd="ambiguous" id="F7.7.2.m2.1.1.1.cmml" xref="F7.7.2.m2.1.1">subscript</csymbol><ci id="F7.7.2.m2.1.1.2.cmml" xref="F7.7.2.m2.1.1.2">𝑆</ci><cn type="integer" id="F7.7.2.m2.1.1.3.cmml" xref="F7.7.2.m2.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="F7.7.2.m2.1d">S_{1}</annotation></semantics></math> to <math id="F7.8.3.m3.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="F7.8.3.m3.1b"><msub id="F7.8.3.m3.1.1" xref="F7.8.3.m3.1.1.cmml"><mi id="F7.8.3.m3.1.1.2" xref="F7.8.3.m3.1.1.2.cmml">S</mi><mn id="F7.8.3.m3.1.1.3" xref="F7.8.3.m3.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="F7.8.3.m3.1c"><apply id="F7.8.3.m3.1.1.cmml" xref="F7.8.3.m3.1.1"><csymbol cd="ambiguous" id="F7.8.3.m3.1.1.1.cmml" xref="F7.8.3.m3.1.1">subscript</csymbol><ci id="F7.8.3.m3.1.1.2.cmml" xref="F7.8.3.m3.1.1.2">𝑆</ci><cn type="integer" id="F7.8.3.m3.1.1.3.cmml" xref="F7.8.3.m3.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="F7.8.3.m3.1d">S_{2}</annotation></semantics></math>, and the blue arrows denote a homomorphism from <math id="F7.9.4.m4.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="F7.9.4.m4.1b"><msub id="F7.9.4.m4.1.1" xref="F7.9.4.m4.1.1.cmml"><mi id="F7.9.4.m4.1.1.2" xref="F7.9.4.m4.1.1.2.cmml">S</mi><mn id="F7.9.4.m4.1.1.3" xref="F7.9.4.m4.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="F7.9.4.m4.1c"><apply id="F7.9.4.m4.1.1.cmml" xref="F7.9.4.m4.1.1"><csymbol cd="ambiguous" id="F7.9.4.m4.1.1.1.cmml" xref="F7.9.4.m4.1.1">subscript</csymbol><ci id="F7.9.4.m4.1.1.2.cmml" xref="F7.9.4.m4.1.1.2">𝑆</ci><cn type="integer" id="F7.9.4.m4.1.1.3.cmml" xref="F7.9.4.m4.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="F7.9.4.m4.1d">S_{2}</annotation></semantics></math> to <math id="F7.10.5.m5.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="F7.10.5.m5.1b"><msub id="F7.10.5.m5.1.1" xref="F7.10.5.m5.1.1.cmml"><mi id="F7.10.5.m5.1.1.2" xref="F7.10.5.m5.1.1.2.cmml">S</mi><mn id="F7.10.5.m5.1.1.3" xref="F7.10.5.m5.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="F7.10.5.m5.1c"><apply id="F7.10.5.m5.1.1.cmml" xref="F7.10.5.m5.1.1"><csymbol cd="ambiguous" id="F7.10.5.m5.1.1.1.cmml" xref="F7.10.5.m5.1.1">subscript</csymbol><ci id="F7.10.5.m5.1.1.2.cmml" xref="F7.10.5.m5.1.1.2">𝑆</ci><cn type="integer" id="F7.10.5.m5.1.1.3.cmml" xref="F7.10.5.m5.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="F7.10.5.m5.1d">S_{1}</annotation></semantics></math>.</span></figcaption>
</figure>
<div id="Thmdefinition2" class="ltx_theorem ltx_theorem_definition">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmdefinition2.1.1.1" class="ltx_text ltx_font_bold">Definition 2</span></span><span id="Thmdefinition2.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmdefinition2.p1" class="ltx_para">
<p id="Thmdefinition2.p1.5" class="ltx_p">Given two computation graphs <math id="Thmdefinition2.p1.1.m1.2" class="ltx_Math" alttext="G_{1},G_{2}" display="inline"><semantics id="Thmdefinition2.p1.1.m1.2a"><mrow id="Thmdefinition2.p1.1.m1.2.2.2" xref="Thmdefinition2.p1.1.m1.2.2.3.cmml"><msub id="Thmdefinition2.p1.1.m1.1.1.1.1" xref="Thmdefinition2.p1.1.m1.1.1.1.1.cmml"><mi id="Thmdefinition2.p1.1.m1.1.1.1.1.2" xref="Thmdefinition2.p1.1.m1.1.1.1.1.2.cmml">G</mi><mn id="Thmdefinition2.p1.1.m1.1.1.1.1.3" xref="Thmdefinition2.p1.1.m1.1.1.1.1.3.cmml">1</mn></msub><mo id="Thmdefinition2.p1.1.m1.2.2.2.3" xref="Thmdefinition2.p1.1.m1.2.2.3.cmml">,</mo><msub id="Thmdefinition2.p1.1.m1.2.2.2.2" xref="Thmdefinition2.p1.1.m1.2.2.2.2.cmml"><mi id="Thmdefinition2.p1.1.m1.2.2.2.2.2" xref="Thmdefinition2.p1.1.m1.2.2.2.2.2.cmml">G</mi><mn id="Thmdefinition2.p1.1.m1.2.2.2.2.3" xref="Thmdefinition2.p1.1.m1.2.2.2.2.3.cmml">2</mn></msub></mrow><annotation-xml encoding="MathML-Content" id="Thmdefinition2.p1.1.m1.2b"><list id="Thmdefinition2.p1.1.m1.2.2.3.cmml" xref="Thmdefinition2.p1.1.m1.2.2.2"><apply id="Thmdefinition2.p1.1.m1.1.1.1.1.cmml" xref="Thmdefinition2.p1.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="Thmdefinition2.p1.1.m1.1.1.1.1.1.cmml" xref="Thmdefinition2.p1.1.m1.1.1.1.1">subscript</csymbol><ci id="Thmdefinition2.p1.1.m1.1.1.1.1.2.cmml" xref="Thmdefinition2.p1.1.m1.1.1.1.1.2">𝐺</ci><cn type="integer" id="Thmdefinition2.p1.1.m1.1.1.1.1.3.cmml" xref="Thmdefinition2.p1.1.m1.1.1.1.1.3">1</cn></apply><apply id="Thmdefinition2.p1.1.m1.2.2.2.2.cmml" xref="Thmdefinition2.p1.1.m1.2.2.2.2"><csymbol cd="ambiguous" id="Thmdefinition2.p1.1.m1.2.2.2.2.1.cmml" xref="Thmdefinition2.p1.1.m1.2.2.2.2">subscript</csymbol><ci id="Thmdefinition2.p1.1.m1.2.2.2.2.2.cmml" xref="Thmdefinition2.p1.1.m1.2.2.2.2.2">𝐺</ci><cn type="integer" id="Thmdefinition2.p1.1.m1.2.2.2.2.3.cmml" xref="Thmdefinition2.p1.1.m1.2.2.2.2.3">2</cn></apply></list></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition2.p1.1.m1.2c">G_{1},G_{2}</annotation></semantics></math> of an LLM and their subgraphs <math id="Thmdefinition2.p1.2.m2.2" class="ltx_Math" alttext="S_{1},S_{2}" display="inline"><semantics id="Thmdefinition2.p1.2.m2.2a"><mrow id="Thmdefinition2.p1.2.m2.2.2.2" xref="Thmdefinition2.p1.2.m2.2.2.3.cmml"><msub id="Thmdefinition2.p1.2.m2.1.1.1.1" xref="Thmdefinition2.p1.2.m2.1.1.1.1.cmml"><mi id="Thmdefinition2.p1.2.m2.1.1.1.1.2" xref="Thmdefinition2.p1.2.m2.1.1.1.1.2.cmml">S</mi><mn id="Thmdefinition2.p1.2.m2.1.1.1.1.3" xref="Thmdefinition2.p1.2.m2.1.1.1.1.3.cmml">1</mn></msub><mo id="Thmdefinition2.p1.2.m2.2.2.2.3" xref="Thmdefinition2.p1.2.m2.2.2.3.cmml">,</mo><msub id="Thmdefinition2.p1.2.m2.2.2.2.2" xref="Thmdefinition2.p1.2.m2.2.2.2.2.cmml"><mi id="Thmdefinition2.p1.2.m2.2.2.2.2.2" xref="Thmdefinition2.p1.2.m2.2.2.2.2.2.cmml">S</mi><mn id="Thmdefinition2.p1.2.m2.2.2.2.2.3" xref="Thmdefinition2.p1.2.m2.2.2.2.2.3.cmml">2</mn></msub></mrow><annotation-xml encoding="MathML-Content" id="Thmdefinition2.p1.2.m2.2b"><list id="Thmdefinition2.p1.2.m2.2.2.3.cmml" xref="Thmdefinition2.p1.2.m2.2.2.2"><apply id="Thmdefinition2.p1.2.m2.1.1.1.1.cmml" xref="Thmdefinition2.p1.2.m2.1.1.1.1"><csymbol cd="ambiguous" id="Thmdefinition2.p1.2.m2.1.1.1.1.1.cmml" xref="Thmdefinition2.p1.2.m2.1.1.1.1">subscript</csymbol><ci id="Thmdefinition2.p1.2.m2.1.1.1.1.2.cmml" xref="Thmdefinition2.p1.2.m2.1.1.1.1.2">𝑆</ci><cn type="integer" id="Thmdefinition2.p1.2.m2.1.1.1.1.3.cmml" xref="Thmdefinition2.p1.2.m2.1.1.1.1.3">1</cn></apply><apply id="Thmdefinition2.p1.2.m2.2.2.2.2.cmml" xref="Thmdefinition2.p1.2.m2.2.2.2.2"><csymbol cd="ambiguous" id="Thmdefinition2.p1.2.m2.2.2.2.2.1.cmml" xref="Thmdefinition2.p1.2.m2.2.2.2.2">subscript</csymbol><ci id="Thmdefinition2.p1.2.m2.2.2.2.2.2.cmml" xref="Thmdefinition2.p1.2.m2.2.2.2.2.2">𝑆</ci><cn type="integer" id="Thmdefinition2.p1.2.m2.2.2.2.2.3.cmml" xref="Thmdefinition2.p1.2.m2.2.2.2.2.3">2</cn></apply></list></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition2.p1.2.m2.2c">S_{1},S_{2}</annotation></semantics></math>, a mapping <math id="Thmdefinition2.p1.3.m3.1" class="ltx_Math" alttext="f" display="inline"><semantics id="Thmdefinition2.p1.3.m3.1a"><mi id="Thmdefinition2.p1.3.m3.1.1" xref="Thmdefinition2.p1.3.m3.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="Thmdefinition2.p1.3.m3.1b"><ci id="Thmdefinition2.p1.3.m3.1.1.cmml" xref="Thmdefinition2.p1.3.m3.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition2.p1.3.m3.1c">f</annotation></semantics></math> from the nodes of <math id="Thmdefinition2.p1.4.m4.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="Thmdefinition2.p1.4.m4.1a"><msub id="Thmdefinition2.p1.4.m4.1.1" xref="Thmdefinition2.p1.4.m4.1.1.cmml"><mi id="Thmdefinition2.p1.4.m4.1.1.2" xref="Thmdefinition2.p1.4.m4.1.1.2.cmml">S</mi><mn id="Thmdefinition2.p1.4.m4.1.1.3" xref="Thmdefinition2.p1.4.m4.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="Thmdefinition2.p1.4.m4.1b"><apply id="Thmdefinition2.p1.4.m4.1.1.cmml" xref="Thmdefinition2.p1.4.m4.1.1"><csymbol cd="ambiguous" id="Thmdefinition2.p1.4.m4.1.1.1.cmml" xref="Thmdefinition2.p1.4.m4.1.1">subscript</csymbol><ci id="Thmdefinition2.p1.4.m4.1.1.2.cmml" xref="Thmdefinition2.p1.4.m4.1.1.2">𝑆</ci><cn type="integer" id="Thmdefinition2.p1.4.m4.1.1.3.cmml" xref="Thmdefinition2.p1.4.m4.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition2.p1.4.m4.1c">S_{1}</annotation></semantics></math> to the nodes of <math id="Thmdefinition2.p1.5.m5.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="Thmdefinition2.p1.5.m5.1a"><msub id="Thmdefinition2.p1.5.m5.1.1" xref="Thmdefinition2.p1.5.m5.1.1.cmml"><mi id="Thmdefinition2.p1.5.m5.1.1.2" xref="Thmdefinition2.p1.5.m5.1.1.2.cmml">S</mi><mn id="Thmdefinition2.p1.5.m5.1.1.3" xref="Thmdefinition2.p1.5.m5.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="Thmdefinition2.p1.5.m5.1b"><apply id="Thmdefinition2.p1.5.m5.1.1.cmml" xref="Thmdefinition2.p1.5.m5.1.1"><csymbol cd="ambiguous" id="Thmdefinition2.p1.5.m5.1.1.1.cmml" xref="Thmdefinition2.p1.5.m5.1.1">subscript</csymbol><ci id="Thmdefinition2.p1.5.m5.1.1.2.cmml" xref="Thmdefinition2.p1.5.m5.1.1.2">𝑆</ci><cn type="integer" id="Thmdefinition2.p1.5.m5.1.1.3.cmml" xref="Thmdefinition2.p1.5.m5.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition2.p1.5.m5.1c">S_{2}</annotation></semantics></math> (not necessarily injective) is a <span id="Thmdefinition2.p1.5.1" class="ltx_text ltx_font_bold">homomorphism</span> if</p>
<ul id="S2.I3" class="ltx_itemize">
<li id="S2.I3.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S2.I3.i1.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S2.I3.i1.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S2.I3.i1.1.1.m1.1.1" xref="S2.I3.i1.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S2.I3.i1.1.1.m1.1c"><ci id="S2.I3.i1.1.1.m1.1.1.cmml" xref="S2.I3.i1.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I3.i1.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S2.I3.i1.p1" class="ltx_para">
<p id="S2.I3.i1.p1.2" class="ltx_p">every node at depth <math id="S2.I3.i1.p1.1.m1.2" class="ltx_Math" alttext="l\in\{0,\dots 2L\}" display="inline"><semantics id="S2.I3.i1.p1.1.m1.2a"><mrow id="S2.I3.i1.p1.1.m1.2.2" xref="S2.I3.i1.p1.1.m1.2.2.cmml"><mi id="S2.I3.i1.p1.1.m1.2.2.3" xref="S2.I3.i1.p1.1.m1.2.2.3.cmml">l</mi><mo id="S2.I3.i1.p1.1.m1.2.2.2" xref="S2.I3.i1.p1.1.m1.2.2.2.cmml">∈</mo><mrow id="S2.I3.i1.p1.1.m1.2.2.1.1" xref="S2.I3.i1.p1.1.m1.2.2.1.2.cmml"><mo stretchy="false" id="S2.I3.i1.p1.1.m1.2.2.1.1.2" xref="S2.I3.i1.p1.1.m1.2.2.1.2.cmml">{</mo><mn id="S2.I3.i1.p1.1.m1.1.1" xref="S2.I3.i1.p1.1.m1.1.1.cmml">0</mn><mo id="S2.I3.i1.p1.1.m1.2.2.1.1.3" xref="S2.I3.i1.p1.1.m1.2.2.1.2.cmml">,</mo><mrow id="S2.I3.i1.p1.1.m1.2.2.1.1.1" xref="S2.I3.i1.p1.1.m1.2.2.1.1.1.cmml"><mi mathvariant="normal" id="S2.I3.i1.p1.1.m1.2.2.1.1.1.2" xref="S2.I3.i1.p1.1.m1.2.2.1.1.1.2.cmml">…</mi><mo lspace="0em" rspace="0em" id="S2.I3.i1.p1.1.m1.2.2.1.1.1.1" xref="S2.I3.i1.p1.1.m1.2.2.1.1.1.1.cmml">​</mo><mn id="S2.I3.i1.p1.1.m1.2.2.1.1.1.3" xref="S2.I3.i1.p1.1.m1.2.2.1.1.1.3.cmml">2</mn><mo lspace="0em" rspace="0em" id="S2.I3.i1.p1.1.m1.2.2.1.1.1.1a" xref="S2.I3.i1.p1.1.m1.2.2.1.1.1.1.cmml">​</mo><mi id="S2.I3.i1.p1.1.m1.2.2.1.1.1.4" xref="S2.I3.i1.p1.1.m1.2.2.1.1.1.4.cmml">L</mi></mrow><mo stretchy="false" id="S2.I3.i1.p1.1.m1.2.2.1.1.4" xref="S2.I3.i1.p1.1.m1.2.2.1.2.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.I3.i1.p1.1.m1.2b"><apply id="S2.I3.i1.p1.1.m1.2.2.cmml" xref="S2.I3.i1.p1.1.m1.2.2"><in id="S2.I3.i1.p1.1.m1.2.2.2.cmml" xref="S2.I3.i1.p1.1.m1.2.2.2"></in><ci id="S2.I3.i1.p1.1.m1.2.2.3.cmml" xref="S2.I3.i1.p1.1.m1.2.2.3">𝑙</ci><set id="S2.I3.i1.p1.1.m1.2.2.1.2.cmml" xref="S2.I3.i1.p1.1.m1.2.2.1.1"><cn type="integer" id="S2.I3.i1.p1.1.m1.1.1.cmml" xref="S2.I3.i1.p1.1.m1.1.1">0</cn><apply id="S2.I3.i1.p1.1.m1.2.2.1.1.1.cmml" xref="S2.I3.i1.p1.1.m1.2.2.1.1.1"><times id="S2.I3.i1.p1.1.m1.2.2.1.1.1.1.cmml" xref="S2.I3.i1.p1.1.m1.2.2.1.1.1.1"></times><ci id="S2.I3.i1.p1.1.m1.2.2.1.1.1.2.cmml" xref="S2.I3.i1.p1.1.m1.2.2.1.1.1.2">…</ci><cn type="integer" id="S2.I3.i1.p1.1.m1.2.2.1.1.1.3.cmml" xref="S2.I3.i1.p1.1.m1.2.2.1.1.1.3">2</cn><ci id="S2.I3.i1.p1.1.m1.2.2.1.1.1.4.cmml" xref="S2.I3.i1.p1.1.m1.2.2.1.1.1.4">𝐿</ci></apply></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I3.i1.p1.1.m1.2c">l\in\{0,\dots 2L\}</annotation></semantics></math> is mapped to depth <math id="S2.I3.i1.p1.2.m2.1" class="ltx_Math" alttext="l" display="inline"><semantics id="S2.I3.i1.p1.2.m2.1a"><mi id="S2.I3.i1.p1.2.m2.1.1" xref="S2.I3.i1.p1.2.m2.1.1.cmml">l</mi><annotation-xml encoding="MathML-Content" id="S2.I3.i1.p1.2.m2.1b"><ci id="S2.I3.i1.p1.2.m2.1.1.cmml" xref="S2.I3.i1.p1.2.m2.1.1">𝑙</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I3.i1.p1.2.m2.1c">l</annotation></semantics></math>,</p>
</div>
</li>
<li id="S2.I3.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S2.I3.i2.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S2.I3.i2.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S2.I3.i2.1.1.m1.1.1" xref="S2.I3.i2.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S2.I3.i2.1.1.m1.1c"><ci id="S2.I3.i2.1.1.m1.1.1.cmml" xref="S2.I3.i2.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I3.i2.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S2.I3.i2.p1" class="ltx_para">
<p id="S2.I3.i2.p1.1" class="ltx_p">if two nodes are on the same position <math id="S2.I3.i2.p1.1.m1.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S2.I3.i2.p1.1.m1.1a"><mi id="S2.I3.i2.p1.1.m1.1.1" xref="S2.I3.i2.p1.1.m1.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S2.I3.i2.p1.1.m1.1b"><ci id="S2.I3.i2.p1.1.m1.1.1.cmml" xref="S2.I3.i2.p1.1.m1.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I3.i2.p1.1.m1.1c">i</annotation></semantics></math>, then they are mapped onto the same position,</p>
</div>
</li>
<li id="S2.I3.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S2.I3.i3.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S2.I3.i3.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S2.I3.i3.1.1.m1.1.1" xref="S2.I3.i3.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S2.I3.i3.1.1.m1.1c"><ci id="S2.I3.i3.1.1.m1.1.1.cmml" xref="S2.I3.i3.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I3.i3.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S2.I3.i3.p1" class="ltx_para">
<p id="S2.I3.i3.p1.4" class="ltx_p">if two nodes share an edge on attention head <math id="S2.I3.i3.p1.1.m1.1" class="ltx_Math" alttext="h" display="inline"><semantics id="S2.I3.i3.p1.1.m1.1a"><mi id="S2.I3.i3.p1.1.m1.1.1" xref="S2.I3.i3.p1.1.m1.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S2.I3.i3.p1.1.m1.1b"><ci id="S2.I3.i3.p1.1.m1.1.1.cmml" xref="S2.I3.i3.p1.1.m1.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I3.i3.p1.1.m1.1c">h</annotation></semantics></math> or MLP neuron <math id="S2.I3.i3.p1.2.m2.1" class="ltx_Math" alttext="m" display="inline"><semantics id="S2.I3.i3.p1.2.m2.1a"><mi id="S2.I3.i3.p1.2.m2.1.1" xref="S2.I3.i3.p1.2.m2.1.1.cmml">m</mi><annotation-xml encoding="MathML-Content" id="S2.I3.i3.p1.2.m2.1b"><ci id="S2.I3.i3.p1.2.m2.1.1.cmml" xref="S2.I3.i3.p1.2.m2.1.1">𝑚</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I3.i3.p1.2.m2.1c">m</annotation></semantics></math>, then their images also share an edge on head <math id="S2.I3.i3.p1.3.m3.1" class="ltx_Math" alttext="h" display="inline"><semantics id="S2.I3.i3.p1.3.m3.1a"><mi id="S2.I3.i3.p1.3.m3.1.1" xref="S2.I3.i3.p1.3.m3.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S2.I3.i3.p1.3.m3.1b"><ci id="S2.I3.i3.p1.3.m3.1.1.cmml" xref="S2.I3.i3.p1.3.m3.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I3.i3.p1.3.m3.1c">h</annotation></semantics></math> or neuron <math id="S2.I3.i3.p1.4.m4.1" class="ltx_Math" alttext="m" display="inline"><semantics id="S2.I3.i3.p1.4.m4.1a"><mi id="S2.I3.i3.p1.4.m4.1.1" xref="S2.I3.i3.p1.4.m4.1.1.cmml">m</mi><annotation-xml encoding="MathML-Content" id="S2.I3.i3.p1.4.m4.1b"><ci id="S2.I3.i3.p1.4.m4.1.1.cmml" xref="S2.I3.i3.p1.4.m4.1.1">𝑚</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I3.i3.p1.4.m4.1c">m</annotation></semantics></math>.</p>
</div>
</li>
</ul>
<p id="Thmdefinition2.p1.7" class="ltx_p">If such a homomorphism exists, then we say that <math id="Thmdefinition2.p1.6.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="Thmdefinition2.p1.6.m1.1a"><msub id="Thmdefinition2.p1.6.m1.1.1" xref="Thmdefinition2.p1.6.m1.1.1.cmml"><mi id="Thmdefinition2.p1.6.m1.1.1.2" xref="Thmdefinition2.p1.6.m1.1.1.2.cmml">S</mi><mn id="Thmdefinition2.p1.6.m1.1.1.3" xref="Thmdefinition2.p1.6.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="Thmdefinition2.p1.6.m1.1b"><apply id="Thmdefinition2.p1.6.m1.1.1.cmml" xref="Thmdefinition2.p1.6.m1.1.1"><csymbol cd="ambiguous" id="Thmdefinition2.p1.6.m1.1.1.1.cmml" xref="Thmdefinition2.p1.6.m1.1.1">subscript</csymbol><ci id="Thmdefinition2.p1.6.m1.1.1.2.cmml" xref="Thmdefinition2.p1.6.m1.1.1.2">𝑆</ci><cn type="integer" id="Thmdefinition2.p1.6.m1.1.1.3.cmml" xref="Thmdefinition2.p1.6.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition2.p1.6.m1.1c">S_{1}</annotation></semantics></math> is homomorphic to <math id="Thmdefinition2.p1.7.m2.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="Thmdefinition2.p1.7.m2.1a"><msub id="Thmdefinition2.p1.7.m2.1.1" xref="Thmdefinition2.p1.7.m2.1.1.cmml"><mi id="Thmdefinition2.p1.7.m2.1.1.2" xref="Thmdefinition2.p1.7.m2.1.1.2.cmml">S</mi><mn id="Thmdefinition2.p1.7.m2.1.1.3" xref="Thmdefinition2.p1.7.m2.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="Thmdefinition2.p1.7.m2.1b"><apply id="Thmdefinition2.p1.7.m2.1.1.cmml" xref="Thmdefinition2.p1.7.m2.1.1"><csymbol cd="ambiguous" id="Thmdefinition2.p1.7.m2.1.1.1.cmml" xref="Thmdefinition2.p1.7.m2.1.1">subscript</csymbol><ci id="Thmdefinition2.p1.7.m2.1.1.2.cmml" xref="Thmdefinition2.p1.7.m2.1.1.2">𝑆</ci><cn type="integer" id="Thmdefinition2.p1.7.m2.1.1.3.cmml" xref="Thmdefinition2.p1.7.m2.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition2.p1.7.m2.1c">S_{2}</annotation></semantics></math>.</p>
</div>
</div>
<div id="S2.SS2.p2" class="ltx_para">
<p id="S2.SS2.p2.1" class="ltx_p">It may be more convenient to define the mapping to be between the input tokens of two sentences, but we adopt the current formulation as it is applicable to more general settings without an obvious correspondence between the tokens and the hidden features at each layer.</p>
</div>
<div id="Thmdefinition3" class="ltx_theorem ltx_theorem_definition">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmdefinition3.1.1.1" class="ltx_text ltx_font_bold">Definition 3</span></span><span id="Thmdefinition3.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmdefinition3.p1" class="ltx_para">
<p id="Thmdefinition3.p1.1" class="ltx_p">Given an LLM and a distribution of texts, a <span id="Thmdefinition3.p1.1.1" class="ltx_text ltx_font_bold">circuit</span> is an equivalence class <math id="Thmdefinition3.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{K}" display="inline"><semantics id="Thmdefinition3.p1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="Thmdefinition3.p1.1.m1.1.1" xref="Thmdefinition3.p1.1.m1.1.1.cmml">𝒦</mi><annotation-xml encoding="MathML-Content" id="Thmdefinition3.p1.1.m1.1b"><ci id="Thmdefinition3.p1.1.m1.1.1.cmml" xref="Thmdefinition3.p1.1.m1.1.1">𝒦</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition3.p1.1.m1.1c">\mathcal{K}</annotation></semantics></math> of subgraphs from computation graphs on random texts, such that</p>
<ul id="S2.I4" class="ltx_itemize">
<li id="S2.I4.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S2.I4.i1.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S2.I4.i1.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S2.I4.i1.1.1.m1.1.1" xref="S2.I4.i1.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S2.I4.i1.1.1.m1.1c"><ci id="S2.I4.i1.1.1.m1.1.1.cmml" xref="S2.I4.i1.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I4.i1.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S2.I4.i1.p1" class="ltx_para">
<p id="S2.I4.i1.p1.1" class="ltx_p">The computation graph on a random text contains some subgraph <math id="S2.I4.i1.p1.1.m1.1" class="ltx_Math" alttext="S\in\mathcal{K}" display="inline"><semantics id="S2.I4.i1.p1.1.m1.1a"><mrow id="S2.I4.i1.p1.1.m1.1.1" xref="S2.I4.i1.p1.1.m1.1.1.cmml"><mi id="S2.I4.i1.p1.1.m1.1.1.2" xref="S2.I4.i1.p1.1.m1.1.1.2.cmml">S</mi><mo id="S2.I4.i1.p1.1.m1.1.1.1" xref="S2.I4.i1.p1.1.m1.1.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S2.I4.i1.p1.1.m1.1.1.3" xref="S2.I4.i1.p1.1.m1.1.1.3.cmml">𝒦</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.I4.i1.p1.1.m1.1b"><apply id="S2.I4.i1.p1.1.m1.1.1.cmml" xref="S2.I4.i1.p1.1.m1.1.1"><in id="S2.I4.i1.p1.1.m1.1.1.1.cmml" xref="S2.I4.i1.p1.1.m1.1.1.1"></in><ci id="S2.I4.i1.p1.1.m1.1.1.2.cmml" xref="S2.I4.i1.p1.1.m1.1.1.2">𝑆</ci><ci id="S2.I4.i1.p1.1.m1.1.1.3.cmml" xref="S2.I4.i1.p1.1.m1.1.1.3">𝒦</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I4.i1.p1.1.m1.1c">S\in\mathcal{K}</annotation></semantics></math> with positive probability.</p>
</div>
</li>
<li id="S2.I4.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S2.I4.i2.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S2.I4.i2.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S2.I4.i2.1.1.m1.1.1" xref="S2.I4.i2.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S2.I4.i2.1.1.m1.1c"><ci id="S2.I4.i2.1.1.m1.1.1.cmml" xref="S2.I4.i2.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I4.i2.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S2.I4.i2.p1" class="ltx_para">
<p id="S2.I4.i2.p1.1" class="ltx_p">All subgraphs <math id="S2.I4.i2.p1.1.m1.1" class="ltx_Math" alttext="S\in\mathcal{K}" display="inline"><semantics id="S2.I4.i2.p1.1.m1.1a"><mrow id="S2.I4.i2.p1.1.m1.1.1" xref="S2.I4.i2.p1.1.m1.1.1.cmml"><mi id="S2.I4.i2.p1.1.m1.1.1.2" xref="S2.I4.i2.p1.1.m1.1.1.2.cmml">S</mi><mo id="S2.I4.i2.p1.1.m1.1.1.1" xref="S2.I4.i2.p1.1.m1.1.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S2.I4.i2.p1.1.m1.1.1.3" xref="S2.I4.i2.p1.1.m1.1.1.3.cmml">𝒦</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.I4.i2.p1.1.m1.1b"><apply id="S2.I4.i2.p1.1.m1.1.1.cmml" xref="S2.I4.i2.p1.1.m1.1.1"><in id="S2.I4.i2.p1.1.m1.1.1.1.cmml" xref="S2.I4.i2.p1.1.m1.1.1.1"></in><ci id="S2.I4.i2.p1.1.m1.1.1.2.cmml" xref="S2.I4.i2.p1.1.m1.1.1.2">𝑆</ci><ci id="S2.I4.i2.p1.1.m1.1.1.3.cmml" xref="S2.I4.i2.p1.1.m1.1.1.3">𝒦</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I4.i2.p1.1.m1.1c">S\in\mathcal{K}</annotation></semantics></math> are homomorphic to each other.</p>
</div>
</li>
<li id="S2.I4.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S2.I4.i3.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S2.I4.i3.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S2.I4.i3.1.1.m1.1.1" xref="S2.I4.i3.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S2.I4.i3.1.1.m1.1c"><ci id="S2.I4.i3.1.1.m1.1.1.cmml" xref="S2.I4.i3.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I4.i3.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S2.I4.i3.p1" class="ltx_para">
<p id="S2.I4.i3.p1.1" class="ltx_p">All edges of all <math id="S2.I4.i3.p1.1.m1.1" class="ltx_Math" alttext="S\in\mathcal{K}" display="inline"><semantics id="S2.I4.i3.p1.1.m1.1a"><mrow id="S2.I4.i3.p1.1.m1.1.1" xref="S2.I4.i3.p1.1.m1.1.1.cmml"><mi id="S2.I4.i3.p1.1.m1.1.1.2" xref="S2.I4.i3.p1.1.m1.1.1.2.cmml">S</mi><mo id="S2.I4.i3.p1.1.m1.1.1.1" xref="S2.I4.i3.p1.1.m1.1.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S2.I4.i3.p1.1.m1.1.1.3" xref="S2.I4.i3.p1.1.m1.1.1.3.cmml">𝒦</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.I4.i3.p1.1.m1.1b"><apply id="S2.I4.i3.p1.1.m1.1.1.cmml" xref="S2.I4.i3.p1.1.m1.1.1"><in id="S2.I4.i3.p1.1.m1.1.1.1.cmml" xref="S2.I4.i3.p1.1.m1.1.1.1"></in><ci id="S2.I4.i3.p1.1.m1.1.1.2.cmml" xref="S2.I4.i3.p1.1.m1.1.1.2">𝑆</ci><ci id="S2.I4.i3.p1.1.m1.1.1.3.cmml" xref="S2.I4.i3.p1.1.m1.1.1.3">𝒦</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I4.i3.p1.1.m1.1c">S\in\mathcal{K}</annotation></semantics></math> have non-negligible weights.</p>
</div>
</li>
<li id="S2.I4.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S2.I4.i4.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S2.I4.i4.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S2.I4.i4.1.1.m1.1.1" xref="S2.I4.i4.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S2.I4.i4.1.1.m1.1c"><ci id="S2.I4.i4.1.1.m1.1.1.cmml" xref="S2.I4.i4.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I4.i4.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S2.I4.i4.p1" class="ltx_para">
<p id="S2.I4.i4.p1.2" class="ltx_p">The pairs <math id="S2.I4.i4.p1.1.m1.4" class="ltx_Math" alttext="(\mathbf{t}_{\text{in}}(S),\mathbf{t}_{\text{out}}(S))" display="inline"><semantics id="S2.I4.i4.p1.1.m1.4a"><mrow id="S2.I4.i4.p1.1.m1.4.4.2" xref="S2.I4.i4.p1.1.m1.4.4.3.cmml"><mo stretchy="false" id="S2.I4.i4.p1.1.m1.4.4.2.3" xref="S2.I4.i4.p1.1.m1.4.4.3.cmml">(</mo><mrow id="S2.I4.i4.p1.1.m1.3.3.1.1" xref="S2.I4.i4.p1.1.m1.3.3.1.1.cmml"><msub id="S2.I4.i4.p1.1.m1.3.3.1.1.2" xref="S2.I4.i4.p1.1.m1.3.3.1.1.2.cmml"><mi id="S2.I4.i4.p1.1.m1.3.3.1.1.2.2" xref="S2.I4.i4.p1.1.m1.3.3.1.1.2.2.cmml">𝐭</mi><mtext id="S2.I4.i4.p1.1.m1.3.3.1.1.2.3" xref="S2.I4.i4.p1.1.m1.3.3.1.1.2.3a.cmml">in</mtext></msub><mo lspace="0em" rspace="0em" id="S2.I4.i4.p1.1.m1.3.3.1.1.1" xref="S2.I4.i4.p1.1.m1.3.3.1.1.1.cmml">​</mo><mrow id="S2.I4.i4.p1.1.m1.3.3.1.1.3.2" xref="S2.I4.i4.p1.1.m1.3.3.1.1.cmml"><mo stretchy="false" id="S2.I4.i4.p1.1.m1.3.3.1.1.3.2.1" xref="S2.I4.i4.p1.1.m1.3.3.1.1.cmml">(</mo><mi id="S2.I4.i4.p1.1.m1.1.1" xref="S2.I4.i4.p1.1.m1.1.1.cmml">S</mi><mo stretchy="false" id="S2.I4.i4.p1.1.m1.3.3.1.1.3.2.2" xref="S2.I4.i4.p1.1.m1.3.3.1.1.cmml">)</mo></mrow></mrow><mo id="S2.I4.i4.p1.1.m1.4.4.2.4" xref="S2.I4.i4.p1.1.m1.4.4.3.cmml">,</mo><mrow id="S2.I4.i4.p1.1.m1.4.4.2.2" xref="S2.I4.i4.p1.1.m1.4.4.2.2.cmml"><msub id="S2.I4.i4.p1.1.m1.4.4.2.2.2" xref="S2.I4.i4.p1.1.m1.4.4.2.2.2.cmml"><mi id="S2.I4.i4.p1.1.m1.4.4.2.2.2.2" xref="S2.I4.i4.p1.1.m1.4.4.2.2.2.2.cmml">𝐭</mi><mtext id="S2.I4.i4.p1.1.m1.4.4.2.2.2.3" xref="S2.I4.i4.p1.1.m1.4.4.2.2.2.3a.cmml">out</mtext></msub><mo lspace="0em" rspace="0em" id="S2.I4.i4.p1.1.m1.4.4.2.2.1" xref="S2.I4.i4.p1.1.m1.4.4.2.2.1.cmml">​</mo><mrow id="S2.I4.i4.p1.1.m1.4.4.2.2.3.2" xref="S2.I4.i4.p1.1.m1.4.4.2.2.cmml"><mo stretchy="false" id="S2.I4.i4.p1.1.m1.4.4.2.2.3.2.1" xref="S2.I4.i4.p1.1.m1.4.4.2.2.cmml">(</mo><mi id="S2.I4.i4.p1.1.m1.2.2" xref="S2.I4.i4.p1.1.m1.2.2.cmml">S</mi><mo stretchy="false" id="S2.I4.i4.p1.1.m1.4.4.2.2.3.2.2" xref="S2.I4.i4.p1.1.m1.4.4.2.2.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S2.I4.i4.p1.1.m1.4.4.2.5" xref="S2.I4.i4.p1.1.m1.4.4.3.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.I4.i4.p1.1.m1.4b"><interval closure="open" id="S2.I4.i4.p1.1.m1.4.4.3.cmml" xref="S2.I4.i4.p1.1.m1.4.4.2"><apply id="S2.I4.i4.p1.1.m1.3.3.1.1.cmml" xref="S2.I4.i4.p1.1.m1.3.3.1.1"><times id="S2.I4.i4.p1.1.m1.3.3.1.1.1.cmml" xref="S2.I4.i4.p1.1.m1.3.3.1.1.1"></times><apply id="S2.I4.i4.p1.1.m1.3.3.1.1.2.cmml" xref="S2.I4.i4.p1.1.m1.3.3.1.1.2"><csymbol cd="ambiguous" id="S2.I4.i4.p1.1.m1.3.3.1.1.2.1.cmml" xref="S2.I4.i4.p1.1.m1.3.3.1.1.2">subscript</csymbol><ci id="S2.I4.i4.p1.1.m1.3.3.1.1.2.2.cmml" xref="S2.I4.i4.p1.1.m1.3.3.1.1.2.2">𝐭</ci><ci id="S2.I4.i4.p1.1.m1.3.3.1.1.2.3a.cmml" xref="S2.I4.i4.p1.1.m1.3.3.1.1.2.3"><mtext mathsize="70%" id="S2.I4.i4.p1.1.m1.3.3.1.1.2.3.cmml" xref="S2.I4.i4.p1.1.m1.3.3.1.1.2.3">in</mtext></ci></apply><ci id="S2.I4.i4.p1.1.m1.1.1.cmml" xref="S2.I4.i4.p1.1.m1.1.1">𝑆</ci></apply><apply id="S2.I4.i4.p1.1.m1.4.4.2.2.cmml" xref="S2.I4.i4.p1.1.m1.4.4.2.2"><times id="S2.I4.i4.p1.1.m1.4.4.2.2.1.cmml" xref="S2.I4.i4.p1.1.m1.4.4.2.2.1"></times><apply id="S2.I4.i4.p1.1.m1.4.4.2.2.2.cmml" xref="S2.I4.i4.p1.1.m1.4.4.2.2.2"><csymbol cd="ambiguous" id="S2.I4.i4.p1.1.m1.4.4.2.2.2.1.cmml" xref="S2.I4.i4.p1.1.m1.4.4.2.2.2">subscript</csymbol><ci id="S2.I4.i4.p1.1.m1.4.4.2.2.2.2.cmml" xref="S2.I4.i4.p1.1.m1.4.4.2.2.2.2">𝐭</ci><ci id="S2.I4.i4.p1.1.m1.4.4.2.2.2.3a.cmml" xref="S2.I4.i4.p1.1.m1.4.4.2.2.2.3"><mtext mathsize="70%" id="S2.I4.i4.p1.1.m1.4.4.2.2.2.3.cmml" xref="S2.I4.i4.p1.1.m1.4.4.2.2.2.3">out</mtext></ci></apply><ci id="S2.I4.i4.p1.1.m1.2.2.cmml" xref="S2.I4.i4.p1.1.m1.2.2">𝑆</ci></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="S2.I4.i4.p1.1.m1.4c">(\mathbf{t}_{\text{in}}(S),\mathbf{t}_{\text{out}}(S))</annotation></semantics></math> share some interpretable meaning across all <math id="S2.I4.i4.p1.2.m2.1" class="ltx_Math" alttext="S\in\mathcal{K}" display="inline"><semantics id="S2.I4.i4.p1.2.m2.1a"><mrow id="S2.I4.i4.p1.2.m2.1.1" xref="S2.I4.i4.p1.2.m2.1.1.cmml"><mi id="S2.I4.i4.p1.2.m2.1.1.2" xref="S2.I4.i4.p1.2.m2.1.1.2.cmml">S</mi><mo id="S2.I4.i4.p1.2.m2.1.1.1" xref="S2.I4.i4.p1.2.m2.1.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S2.I4.i4.p1.2.m2.1.1.3" xref="S2.I4.i4.p1.2.m2.1.1.3.cmml">𝒦</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.I4.i4.p1.2.m2.1b"><apply id="S2.I4.i4.p1.2.m2.1.1.cmml" xref="S2.I4.i4.p1.2.m2.1.1"><in id="S2.I4.i4.p1.2.m2.1.1.1.cmml" xref="S2.I4.i4.p1.2.m2.1.1.1"></in><ci id="S2.I4.i4.p1.2.m2.1.1.2.cmml" xref="S2.I4.i4.p1.2.m2.1.1.2">𝑆</ci><ci id="S2.I4.i4.p1.2.m2.1.1.3.cmml" xref="S2.I4.i4.p1.2.m2.1.1.3">𝒦</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I4.i4.p1.2.m2.1c">S\in\mathcal{K}</annotation></semantics></math>.</p>
</div>
</li>
</ul>
</div>
</div>
<div id="Thmdefinition4" class="ltx_theorem ltx_theorem_definition">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmdefinition4.1.1.1" class="ltx_text ltx_font_bold">Definition 4</span></span><span id="Thmdefinition4.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmdefinition4.p1" class="ltx_para">
<p id="Thmdefinition4.p1.1" class="ltx_p">Given an LLM and a distribution of texts, we call each circuit a <span id="Thmdefinition4.p1.1.1" class="ltx_text ltx_font_bold">knowledge</span>.
Furthermore, a circuit <math id="Thmdefinition4.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{K}" display="inline"><semantics id="Thmdefinition4.p1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="Thmdefinition4.p1.1.m1.1.1" xref="Thmdefinition4.p1.1.m1.1.1.cmml">𝒦</mi><annotation-xml encoding="MathML-Content" id="Thmdefinition4.p1.1.m1.1b"><ci id="Thmdefinition4.p1.1.m1.1.1.cmml" xref="Thmdefinition4.p1.1.m1.1.1">𝒦</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition4.p1.1.m1.1c">\mathcal{K}</annotation></semantics></math> is called a</p>
<ul id="S2.I5" class="ltx_itemize">
<li id="S2.I5.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S2.I5.i1.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S2.I5.i1.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S2.I5.i1.1.1.m1.1.1" xref="S2.I5.i1.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S2.I5.i1.1.1.m1.1c"><ci id="S2.I5.i1.1.1.m1.1.1.cmml" xref="S2.I5.i1.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I5.i1.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S2.I5.i1.p1" class="ltx_para">
<p id="S2.I5.i1.p1.4" class="ltx_p"><span id="S2.I5.i1.p1.4.1" class="ltx_text ltx_font_bold">specific knowledge</span>, if the associated inputs <math id="S2.I5.i1.p1.1.m1.1" class="ltx_Math" alttext="t_{\text{in}}(S)" display="inline"><semantics id="S2.I5.i1.p1.1.m1.1a"><mrow id="S2.I5.i1.p1.1.m1.1.2" xref="S2.I5.i1.p1.1.m1.1.2.cmml"><msub id="S2.I5.i1.p1.1.m1.1.2.2" xref="S2.I5.i1.p1.1.m1.1.2.2.cmml"><mi id="S2.I5.i1.p1.1.m1.1.2.2.2" xref="S2.I5.i1.p1.1.m1.1.2.2.2.cmml">t</mi><mtext id="S2.I5.i1.p1.1.m1.1.2.2.3" xref="S2.I5.i1.p1.1.m1.1.2.2.3a.cmml">in</mtext></msub><mo lspace="0em" rspace="0em" id="S2.I5.i1.p1.1.m1.1.2.1" xref="S2.I5.i1.p1.1.m1.1.2.1.cmml">​</mo><mrow id="S2.I5.i1.p1.1.m1.1.2.3.2" xref="S2.I5.i1.p1.1.m1.1.2.cmml"><mo stretchy="false" id="S2.I5.i1.p1.1.m1.1.2.3.2.1" xref="S2.I5.i1.p1.1.m1.1.2.cmml">(</mo><mi id="S2.I5.i1.p1.1.m1.1.1" xref="S2.I5.i1.p1.1.m1.1.1.cmml">S</mi><mo stretchy="false" id="S2.I5.i1.p1.1.m1.1.2.3.2.2" xref="S2.I5.i1.p1.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.I5.i1.p1.1.m1.1b"><apply id="S2.I5.i1.p1.1.m1.1.2.cmml" xref="S2.I5.i1.p1.1.m1.1.2"><times id="S2.I5.i1.p1.1.m1.1.2.1.cmml" xref="S2.I5.i1.p1.1.m1.1.2.1"></times><apply id="S2.I5.i1.p1.1.m1.1.2.2.cmml" xref="S2.I5.i1.p1.1.m1.1.2.2"><csymbol cd="ambiguous" id="S2.I5.i1.p1.1.m1.1.2.2.1.cmml" xref="S2.I5.i1.p1.1.m1.1.2.2">subscript</csymbol><ci id="S2.I5.i1.p1.1.m1.1.2.2.2.cmml" xref="S2.I5.i1.p1.1.m1.1.2.2.2">𝑡</ci><ci id="S2.I5.i1.p1.1.m1.1.2.2.3a.cmml" xref="S2.I5.i1.p1.1.m1.1.2.2.3"><mtext mathsize="70%" id="S2.I5.i1.p1.1.m1.1.2.2.3.cmml" xref="S2.I5.i1.p1.1.m1.1.2.2.3">in</mtext></ci></apply><ci id="S2.I5.i1.p1.1.m1.1.1.cmml" xref="S2.I5.i1.p1.1.m1.1.1">𝑆</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I5.i1.p1.1.m1.1c">t_{\text{in}}(S)</annotation></semantics></math> for all subgraphs <math id="S2.I5.i1.p1.2.m2.1" class="ltx_Math" alttext="S\in\mathcal{K}" display="inline"><semantics id="S2.I5.i1.p1.2.m2.1a"><mrow id="S2.I5.i1.p1.2.m2.1.1" xref="S2.I5.i1.p1.2.m2.1.1.cmml"><mi id="S2.I5.i1.p1.2.m2.1.1.2" xref="S2.I5.i1.p1.2.m2.1.1.2.cmml">S</mi><mo id="S2.I5.i1.p1.2.m2.1.1.1" xref="S2.I5.i1.p1.2.m2.1.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S2.I5.i1.p1.2.m2.1.1.3" xref="S2.I5.i1.p1.2.m2.1.1.3.cmml">𝒦</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.I5.i1.p1.2.m2.1b"><apply id="S2.I5.i1.p1.2.m2.1.1.cmml" xref="S2.I5.i1.p1.2.m2.1.1"><in id="S2.I5.i1.p1.2.m2.1.1.1.cmml" xref="S2.I5.i1.p1.2.m2.1.1.1"></in><ci id="S2.I5.i1.p1.2.m2.1.1.2.cmml" xref="S2.I5.i1.p1.2.m2.1.1.2">𝑆</ci><ci id="S2.I5.i1.p1.2.m2.1.1.3.cmml" xref="S2.I5.i1.p1.2.m2.1.1.3">𝒦</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I5.i1.p1.2.m2.1c">S\in\mathcal{K}</annotation></semantics></math> share some interpretable meaning, and the associated outputs <math id="S2.I5.i1.p1.3.m3.1" class="ltx_Math" alttext="t_{\text{out}}(S)" display="inline"><semantics id="S2.I5.i1.p1.3.m3.1a"><mrow id="S2.I5.i1.p1.3.m3.1.2" xref="S2.I5.i1.p1.3.m3.1.2.cmml"><msub id="S2.I5.i1.p1.3.m3.1.2.2" xref="S2.I5.i1.p1.3.m3.1.2.2.cmml"><mi id="S2.I5.i1.p1.3.m3.1.2.2.2" xref="S2.I5.i1.p1.3.m3.1.2.2.2.cmml">t</mi><mtext id="S2.I5.i1.p1.3.m3.1.2.2.3" xref="S2.I5.i1.p1.3.m3.1.2.2.3a.cmml">out</mtext></msub><mo lspace="0em" rspace="0em" id="S2.I5.i1.p1.3.m3.1.2.1" xref="S2.I5.i1.p1.3.m3.1.2.1.cmml">​</mo><mrow id="S2.I5.i1.p1.3.m3.1.2.3.2" xref="S2.I5.i1.p1.3.m3.1.2.cmml"><mo stretchy="false" id="S2.I5.i1.p1.3.m3.1.2.3.2.1" xref="S2.I5.i1.p1.3.m3.1.2.cmml">(</mo><mi id="S2.I5.i1.p1.3.m3.1.1" xref="S2.I5.i1.p1.3.m3.1.1.cmml">S</mi><mo stretchy="false" id="S2.I5.i1.p1.3.m3.1.2.3.2.2" xref="S2.I5.i1.p1.3.m3.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.I5.i1.p1.3.m3.1b"><apply id="S2.I5.i1.p1.3.m3.1.2.cmml" xref="S2.I5.i1.p1.3.m3.1.2"><times id="S2.I5.i1.p1.3.m3.1.2.1.cmml" xref="S2.I5.i1.p1.3.m3.1.2.1"></times><apply id="S2.I5.i1.p1.3.m3.1.2.2.cmml" xref="S2.I5.i1.p1.3.m3.1.2.2"><csymbol cd="ambiguous" id="S2.I5.i1.p1.3.m3.1.2.2.1.cmml" xref="S2.I5.i1.p1.3.m3.1.2.2">subscript</csymbol><ci id="S2.I5.i1.p1.3.m3.1.2.2.2.cmml" xref="S2.I5.i1.p1.3.m3.1.2.2.2">𝑡</ci><ci id="S2.I5.i1.p1.3.m3.1.2.2.3a.cmml" xref="S2.I5.i1.p1.3.m3.1.2.2.3"><mtext mathsize="70%" id="S2.I5.i1.p1.3.m3.1.2.2.3.cmml" xref="S2.I5.i1.p1.3.m3.1.2.2.3">out</mtext></ci></apply><ci id="S2.I5.i1.p1.3.m3.1.1.cmml" xref="S2.I5.i1.p1.3.m3.1.1">𝑆</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I5.i1.p1.3.m3.1c">t_{\text{out}}(S)</annotation></semantics></math> for all <math id="S2.I5.i1.p1.4.m4.1" class="ltx_Math" alttext="S\in\mathcal{K}" display="inline"><semantics id="S2.I5.i1.p1.4.m4.1a"><mrow id="S2.I5.i1.p1.4.m4.1.1" xref="S2.I5.i1.p1.4.m4.1.1.cmml"><mi id="S2.I5.i1.p1.4.m4.1.1.2" xref="S2.I5.i1.p1.4.m4.1.1.2.cmml">S</mi><mo id="S2.I5.i1.p1.4.m4.1.1.1" xref="S2.I5.i1.p1.4.m4.1.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S2.I5.i1.p1.4.m4.1.1.3" xref="S2.I5.i1.p1.4.m4.1.1.3.cmml">𝒦</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.I5.i1.p1.4.m4.1b"><apply id="S2.I5.i1.p1.4.m4.1.1.cmml" xref="S2.I5.i1.p1.4.m4.1.1"><in id="S2.I5.i1.p1.4.m4.1.1.1.cmml" xref="S2.I5.i1.p1.4.m4.1.1.1"></in><ci id="S2.I5.i1.p1.4.m4.1.1.2.cmml" xref="S2.I5.i1.p1.4.m4.1.1.2">𝑆</ci><ci id="S2.I5.i1.p1.4.m4.1.1.3.cmml" xref="S2.I5.i1.p1.4.m4.1.1.3">𝒦</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I5.i1.p1.4.m4.1c">S\in\mathcal{K}</annotation></semantics></math> are the same or differ by at most a small fraction of tokens.</p>
</div>
</li>
<li id="S2.I5.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S2.I5.i2.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S2.I5.i2.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S2.I5.i2.1.1.m1.1.1" xref="S2.I5.i2.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S2.I5.i2.1.1.m1.1c"><ci id="S2.I5.i2.1.1.m1.1.1.cmml" xref="S2.I5.i2.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I5.i2.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S2.I5.i2.p1" class="ltx_para">
<p id="S2.I5.i2.p1.1" class="ltx_p"><span id="S2.I5.i2.p1.1.1" class="ltx_text ltx_font_bold">abstract knowledge</span>, else.</p>
</div>
</li>
</ul>
</div>
</div>
<div id="S2.SS2.p3" class="ltx_para">
<p id="S2.SS2.p3.1" class="ltx_p">From now on, we use knowledge as a countable noun since the circuits are countable.
Note that the criterion in Definition <a href="#Thmdefinition4" title="Definition 4. ‣ 2.2 | Knowledge ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> is stronger than the last criterion in Definition <a href="#Thmdefinition3" title="Definition 3. ‣ 2.2 | Knowledge ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, e.g. consider the circuit that always copy-and-pastes the previous token.
We will see that the rigidity of specific knowledges makes them easier to externalize.</p>
</div>
<div id="S2.SS2.p4" class="ltx_para">
<p id="S2.SS2.p4.1" class="ltx_p">Here are some well-known examples of knowledge.</p>
</div>
<div id="Thmexample3" class="ltx_theorem ltx_theorem_example">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmexample3.1.1.1" class="ltx_text ltx_font_bold">Example 3</span></span><span id="Thmexample3.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmexample3.p1" class="ltx_para">
<p id="Thmexample3.p1.3" class="ltx_p">Recall the knowledge neuron from Example <a href="#Thmexample1" title="Example 1. ‣ 2.1 | Preliminaries ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> that helps to answer “The capital of China is Beijing”.
Such neurons can be activated by a variety of contexts that involve the subject-relation pair (“China”, “capital”) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib29" title="" class="ltx_ref">29</a>]</cite>.
Its circuit can be simply defined as the equivalence class of subgraphs induced by edges <math id="Thmexample3.p1.1.m1.2" class="ltx_Math" alttext="e^{l,m}_{i}" display="inline"><semantics id="Thmexample3.p1.1.m1.2a"><msubsup id="Thmexample3.p1.1.m1.2.3" xref="Thmexample3.p1.1.m1.2.3.cmml"><mi id="Thmexample3.p1.1.m1.2.3.2.2" xref="Thmexample3.p1.1.m1.2.3.2.2.cmml">e</mi><mi id="Thmexample3.p1.1.m1.2.3.3" xref="Thmexample3.p1.1.m1.2.3.3.cmml">i</mi><mrow id="Thmexample3.p1.1.m1.2.2.2.4" xref="Thmexample3.p1.1.m1.2.2.2.3.cmml"><mi id="Thmexample3.p1.1.m1.1.1.1.1" xref="Thmexample3.p1.1.m1.1.1.1.1.cmml">l</mi><mo id="Thmexample3.p1.1.m1.2.2.2.4.1" xref="Thmexample3.p1.1.m1.2.2.2.3.cmml">,</mo><mi id="Thmexample3.p1.1.m1.2.2.2.2" xref="Thmexample3.p1.1.m1.2.2.2.2.cmml">m</mi></mrow></msubsup><annotation-xml encoding="MathML-Content" id="Thmexample3.p1.1.m1.2b"><apply id="Thmexample3.p1.1.m1.2.3.cmml" xref="Thmexample3.p1.1.m1.2.3"><csymbol cd="ambiguous" id="Thmexample3.p1.1.m1.2.3.1.cmml" xref="Thmexample3.p1.1.m1.2.3">subscript</csymbol><apply id="Thmexample3.p1.1.m1.2.3.2.cmml" xref="Thmexample3.p1.1.m1.2.3"><csymbol cd="ambiguous" id="Thmexample3.p1.1.m1.2.3.2.1.cmml" xref="Thmexample3.p1.1.m1.2.3">superscript</csymbol><ci id="Thmexample3.p1.1.m1.2.3.2.2.cmml" xref="Thmexample3.p1.1.m1.2.3.2.2">𝑒</ci><list id="Thmexample3.p1.1.m1.2.2.2.3.cmml" xref="Thmexample3.p1.1.m1.2.2.2.4"><ci id="Thmexample3.p1.1.m1.1.1.1.1.cmml" xref="Thmexample3.p1.1.m1.1.1.1.1">𝑙</ci><ci id="Thmexample3.p1.1.m1.2.2.2.2.cmml" xref="Thmexample3.p1.1.m1.2.2.2.2">𝑚</ci></list></apply><ci id="Thmexample3.p1.1.m1.2.3.3.cmml" xref="Thmexample3.p1.1.m1.2.3.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmexample3.p1.1.m1.2c">e^{l,m}_{i}</annotation></semantics></math>, where <math id="Thmexample3.p1.2.m2.2" class="ltx_Math" alttext="(l,m)" display="inline"><semantics id="Thmexample3.p1.2.m2.2a"><mrow id="Thmexample3.p1.2.m2.2.3.2" xref="Thmexample3.p1.2.m2.2.3.1.cmml"><mo stretchy="false" id="Thmexample3.p1.2.m2.2.3.2.1" xref="Thmexample3.p1.2.m2.2.3.1.cmml">(</mo><mi id="Thmexample3.p1.2.m2.1.1" xref="Thmexample3.p1.2.m2.1.1.cmml">l</mi><mo id="Thmexample3.p1.2.m2.2.3.2.2" xref="Thmexample3.p1.2.m2.2.3.1.cmml">,</mo><mi id="Thmexample3.p1.2.m2.2.2" xref="Thmexample3.p1.2.m2.2.2.cmml">m</mi><mo stretchy="false" id="Thmexample3.p1.2.m2.2.3.2.3" xref="Thmexample3.p1.2.m2.2.3.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="Thmexample3.p1.2.m2.2b"><interval closure="open" id="Thmexample3.p1.2.m2.2.3.1.cmml" xref="Thmexample3.p1.2.m2.2.3.2"><ci id="Thmexample3.p1.2.m2.1.1.cmml" xref="Thmexample3.p1.2.m2.1.1">𝑙</ci><ci id="Thmexample3.p1.2.m2.2.2.cmml" xref="Thmexample3.p1.2.m2.2.2">𝑚</ci></interval></annotation-xml><annotation encoding="application/x-tex" id="Thmexample3.p1.2.m2.2c">(l,m)</annotation></semantics></math> is the fixed location of the knowledge neuron and <math id="Thmexample3.p1.3.m3.1" class="ltx_Math" alttext="i" display="inline"><semantics id="Thmexample3.p1.3.m3.1a"><mi id="Thmexample3.p1.3.m3.1.1" xref="Thmexample3.p1.3.m3.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="Thmexample3.p1.3.m3.1b"><ci id="Thmexample3.p1.3.m3.1.1.cmml" xref="Thmexample3.p1.3.m3.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmexample3.p1.3.m3.1c">i</annotation></semantics></math> is the variable position of the last token of the context.
The associated inputs are “China” and “capital”, and the associated outputs are always “Beijing”.
By definition, this circuit is a specific knowledge, since its associated output is fixed and its associated inputs share a clear pattern (fixed tokens with variable positions).</p>
</div>
</div>
<div id="S2.SS2.p5" class="ltx_para">
<p id="S2.SS2.p5.1" class="ltx_p">Similarly, by straightforward construction, one can show that each <math id="S2.SS2.p5.1.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S2.SS2.p5.1.m1.1a"><mi id="S2.SS2.p5.1.m1.1.1" xref="S2.SS2.p5.1.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p5.1.m1.1b"><ci id="S2.SS2.p5.1.m1.1.1.cmml" xref="S2.SS2.p5.1.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p5.1.m1.1c">n</annotation></semantics></math>-gram can be expressed as a specific knowledge.</p>
</div>
<div id="Thmexample4" class="ltx_theorem ltx_theorem_example">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmexample4.1.1.1" class="ltx_text ltx_font_bold">Example 4</span></span><span id="Thmexample4.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmexample4.p1" class="ltx_para">
<p id="Thmexample4.p1.3" class="ltx_p">Recall the induction heads <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib85" title="" class="ltx_ref">85</a>]</cite> from Example <a href="#Thmexample2" title="Example 2. ‣ 2.1 | Preliminaries ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> that complete “[a][b] …[a]” with “[b]”.
Let <math id="Thmexample4.p1.1.m1.4" class="ltx_Math" alttext="(l,h),(l+1,h^{\prime})" display="inline"><semantics id="Thmexample4.p1.1.m1.4a"><mrow id="Thmexample4.p1.1.m1.4.4.2" xref="Thmexample4.p1.1.m1.4.4.3.cmml"><mrow id="Thmexample4.p1.1.m1.3.3.1.1.2" xref="Thmexample4.p1.1.m1.3.3.1.1.1.cmml"><mo stretchy="false" id="Thmexample4.p1.1.m1.3.3.1.1.2.1" xref="Thmexample4.p1.1.m1.3.3.1.1.1.cmml">(</mo><mi id="Thmexample4.p1.1.m1.1.1" xref="Thmexample4.p1.1.m1.1.1.cmml">l</mi><mo id="Thmexample4.p1.1.m1.3.3.1.1.2.2" xref="Thmexample4.p1.1.m1.3.3.1.1.1.cmml">,</mo><mi id="Thmexample4.p1.1.m1.2.2" xref="Thmexample4.p1.1.m1.2.2.cmml">h</mi><mo stretchy="false" id="Thmexample4.p1.1.m1.3.3.1.1.2.3" xref="Thmexample4.p1.1.m1.3.3.1.1.1.cmml">)</mo></mrow><mo id="Thmexample4.p1.1.m1.4.4.2.3" xref="Thmexample4.p1.1.m1.4.4.3.cmml">,</mo><mrow id="Thmexample4.p1.1.m1.4.4.2.2.2" xref="Thmexample4.p1.1.m1.4.4.2.2.3.cmml"><mo stretchy="false" id="Thmexample4.p1.1.m1.4.4.2.2.2.3" xref="Thmexample4.p1.1.m1.4.4.2.2.3.cmml">(</mo><mrow id="Thmexample4.p1.1.m1.4.4.2.2.1.1" xref="Thmexample4.p1.1.m1.4.4.2.2.1.1.cmml"><mi id="Thmexample4.p1.1.m1.4.4.2.2.1.1.2" xref="Thmexample4.p1.1.m1.4.4.2.2.1.1.2.cmml">l</mi><mo id="Thmexample4.p1.1.m1.4.4.2.2.1.1.1" xref="Thmexample4.p1.1.m1.4.4.2.2.1.1.1.cmml">+</mo><mn id="Thmexample4.p1.1.m1.4.4.2.2.1.1.3" xref="Thmexample4.p1.1.m1.4.4.2.2.1.1.3.cmml">1</mn></mrow><mo id="Thmexample4.p1.1.m1.4.4.2.2.2.4" xref="Thmexample4.p1.1.m1.4.4.2.2.3.cmml">,</mo><msup id="Thmexample4.p1.1.m1.4.4.2.2.2.2" xref="Thmexample4.p1.1.m1.4.4.2.2.2.2.cmml"><mi id="Thmexample4.p1.1.m1.4.4.2.2.2.2.2" xref="Thmexample4.p1.1.m1.4.4.2.2.2.2.2.cmml">h</mi><mo id="Thmexample4.p1.1.m1.4.4.2.2.2.2.3" xref="Thmexample4.p1.1.m1.4.4.2.2.2.2.3.cmml">′</mo></msup><mo stretchy="false" id="Thmexample4.p1.1.m1.4.4.2.2.2.5" xref="Thmexample4.p1.1.m1.4.4.2.2.3.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="Thmexample4.p1.1.m1.4b"><list id="Thmexample4.p1.1.m1.4.4.3.cmml" xref="Thmexample4.p1.1.m1.4.4.2"><interval closure="open" id="Thmexample4.p1.1.m1.3.3.1.1.1.cmml" xref="Thmexample4.p1.1.m1.3.3.1.1.2"><ci id="Thmexample4.p1.1.m1.1.1.cmml" xref="Thmexample4.p1.1.m1.1.1">𝑙</ci><ci id="Thmexample4.p1.1.m1.2.2.cmml" xref="Thmexample4.p1.1.m1.2.2">ℎ</ci></interval><interval closure="open" id="Thmexample4.p1.1.m1.4.4.2.2.3.cmml" xref="Thmexample4.p1.1.m1.4.4.2.2.2"><apply id="Thmexample4.p1.1.m1.4.4.2.2.1.1.cmml" xref="Thmexample4.p1.1.m1.4.4.2.2.1.1"><plus id="Thmexample4.p1.1.m1.4.4.2.2.1.1.1.cmml" xref="Thmexample4.p1.1.m1.4.4.2.2.1.1.1"></plus><ci id="Thmexample4.p1.1.m1.4.4.2.2.1.1.2.cmml" xref="Thmexample4.p1.1.m1.4.4.2.2.1.1.2">𝑙</ci><cn type="integer" id="Thmexample4.p1.1.m1.4.4.2.2.1.1.3.cmml" xref="Thmexample4.p1.1.m1.4.4.2.2.1.1.3">1</cn></apply><apply id="Thmexample4.p1.1.m1.4.4.2.2.2.2.cmml" xref="Thmexample4.p1.1.m1.4.4.2.2.2.2"><csymbol cd="ambiguous" id="Thmexample4.p1.1.m1.4.4.2.2.2.2.1.cmml" xref="Thmexample4.p1.1.m1.4.4.2.2.2.2">superscript</csymbol><ci id="Thmexample4.p1.1.m1.4.4.2.2.2.2.2.cmml" xref="Thmexample4.p1.1.m1.4.4.2.2.2.2.2">ℎ</ci><ci id="Thmexample4.p1.1.m1.4.4.2.2.2.2.3.cmml" xref="Thmexample4.p1.1.m1.4.4.2.2.2.2.3">′</ci></apply></interval></list></annotation-xml><annotation encoding="application/x-tex" id="Thmexample4.p1.1.m1.4c">(l,h),(l+1,h^{\prime})</annotation></semantics></math> be the locations of these two heads, and denote the variable positions of the two token [a]’s by <math id="Thmexample4.p1.2.m2.2" class="ltx_Math" alttext="i,j" display="inline"><semantics id="Thmexample4.p1.2.m2.2a"><mrow id="Thmexample4.p1.2.m2.2.3.2" xref="Thmexample4.p1.2.m2.2.3.1.cmml"><mi id="Thmexample4.p1.2.m2.1.1" xref="Thmexample4.p1.2.m2.1.1.cmml">i</mi><mo id="Thmexample4.p1.2.m2.2.3.2.1" xref="Thmexample4.p1.2.m2.2.3.1.cmml">,</mo><mi id="Thmexample4.p1.2.m2.2.2" xref="Thmexample4.p1.2.m2.2.2.cmml">j</mi></mrow><annotation-xml encoding="MathML-Content" id="Thmexample4.p1.2.m2.2b"><list id="Thmexample4.p1.2.m2.2.3.1.cmml" xref="Thmexample4.p1.2.m2.2.3.2"><ci id="Thmexample4.p1.2.m2.1.1.cmml" xref="Thmexample4.p1.2.m2.1.1">𝑖</ci><ci id="Thmexample4.p1.2.m2.2.2.cmml" xref="Thmexample4.p1.2.m2.2.2">𝑗</ci></list></annotation-xml><annotation encoding="application/x-tex" id="Thmexample4.p1.2.m2.2c">i,j</annotation></semantics></math>.
Its circuit is the equivalence class of subgraphs induced by the two edges <math id="Thmexample4.p1.3.m3.10" class="ltx_Math" alttext="e^{l,h}_{i,i+1},e^{l+1,h^{\prime}}_{i+1,j}" display="inline"><semantics id="Thmexample4.p1.3.m3.10a"><mrow id="Thmexample4.p1.3.m3.10.10.2" xref="Thmexample4.p1.3.m3.10.10.3.cmml"><msubsup id="Thmexample4.p1.3.m3.9.9.1.1" xref="Thmexample4.p1.3.m3.9.9.1.1.cmml"><mi id="Thmexample4.p1.3.m3.9.9.1.1.2.2" xref="Thmexample4.p1.3.m3.9.9.1.1.2.2.cmml">e</mi><mrow id="Thmexample4.p1.3.m3.4.4.2.2" xref="Thmexample4.p1.3.m3.4.4.2.3.cmml"><mi id="Thmexample4.p1.3.m3.3.3.1.1" xref="Thmexample4.p1.3.m3.3.3.1.1.cmml">i</mi><mo id="Thmexample4.p1.3.m3.4.4.2.2.2" xref="Thmexample4.p1.3.m3.4.4.2.3.cmml">,</mo><mrow id="Thmexample4.p1.3.m3.4.4.2.2.1" xref="Thmexample4.p1.3.m3.4.4.2.2.1.cmml"><mi id="Thmexample4.p1.3.m3.4.4.2.2.1.2" xref="Thmexample4.p1.3.m3.4.4.2.2.1.2.cmml">i</mi><mo id="Thmexample4.p1.3.m3.4.4.2.2.1.1" xref="Thmexample4.p1.3.m3.4.4.2.2.1.1.cmml">+</mo><mn id="Thmexample4.p1.3.m3.4.4.2.2.1.3" xref="Thmexample4.p1.3.m3.4.4.2.2.1.3.cmml">1</mn></mrow></mrow><mrow id="Thmexample4.p1.3.m3.2.2.2.4" xref="Thmexample4.p1.3.m3.2.2.2.3.cmml"><mi id="Thmexample4.p1.3.m3.1.1.1.1" xref="Thmexample4.p1.3.m3.1.1.1.1.cmml">l</mi><mo id="Thmexample4.p1.3.m3.2.2.2.4.1" xref="Thmexample4.p1.3.m3.2.2.2.3.cmml">,</mo><mi id="Thmexample4.p1.3.m3.2.2.2.2" xref="Thmexample4.p1.3.m3.2.2.2.2.cmml">h</mi></mrow></msubsup><mo id="Thmexample4.p1.3.m3.10.10.2.3" xref="Thmexample4.p1.3.m3.10.10.3.cmml">,</mo><msubsup id="Thmexample4.p1.3.m3.10.10.2.2" xref="Thmexample4.p1.3.m3.10.10.2.2.cmml"><mi id="Thmexample4.p1.3.m3.10.10.2.2.2.2" xref="Thmexample4.p1.3.m3.10.10.2.2.2.2.cmml">e</mi><mrow id="Thmexample4.p1.3.m3.8.8.2.2" xref="Thmexample4.p1.3.m3.8.8.2.3.cmml"><mrow id="Thmexample4.p1.3.m3.8.8.2.2.1" xref="Thmexample4.p1.3.m3.8.8.2.2.1.cmml"><mi id="Thmexample4.p1.3.m3.8.8.2.2.1.2" xref="Thmexample4.p1.3.m3.8.8.2.2.1.2.cmml">i</mi><mo id="Thmexample4.p1.3.m3.8.8.2.2.1.1" xref="Thmexample4.p1.3.m3.8.8.2.2.1.1.cmml">+</mo><mn id="Thmexample4.p1.3.m3.8.8.2.2.1.3" xref="Thmexample4.p1.3.m3.8.8.2.2.1.3.cmml">1</mn></mrow><mo id="Thmexample4.p1.3.m3.8.8.2.2.2" xref="Thmexample4.p1.3.m3.8.8.2.3.cmml">,</mo><mi id="Thmexample4.p1.3.m3.7.7.1.1" xref="Thmexample4.p1.3.m3.7.7.1.1.cmml">j</mi></mrow><mrow id="Thmexample4.p1.3.m3.6.6.2.2" xref="Thmexample4.p1.3.m3.6.6.2.3.cmml"><mrow id="Thmexample4.p1.3.m3.5.5.1.1.1" xref="Thmexample4.p1.3.m3.5.5.1.1.1.cmml"><mi id="Thmexample4.p1.3.m3.5.5.1.1.1.2" xref="Thmexample4.p1.3.m3.5.5.1.1.1.2.cmml">l</mi><mo id="Thmexample4.p1.3.m3.5.5.1.1.1.1" xref="Thmexample4.p1.3.m3.5.5.1.1.1.1.cmml">+</mo><mn id="Thmexample4.p1.3.m3.5.5.1.1.1.3" xref="Thmexample4.p1.3.m3.5.5.1.1.1.3.cmml">1</mn></mrow><mo id="Thmexample4.p1.3.m3.6.6.2.2.3" xref="Thmexample4.p1.3.m3.6.6.2.3.cmml">,</mo><msup id="Thmexample4.p1.3.m3.6.6.2.2.2" xref="Thmexample4.p1.3.m3.6.6.2.2.2.cmml"><mi id="Thmexample4.p1.3.m3.6.6.2.2.2.2" xref="Thmexample4.p1.3.m3.6.6.2.2.2.2.cmml">h</mi><mo id="Thmexample4.p1.3.m3.6.6.2.2.2.3" xref="Thmexample4.p1.3.m3.6.6.2.2.2.3.cmml">′</mo></msup></mrow></msubsup></mrow><annotation-xml encoding="MathML-Content" id="Thmexample4.p1.3.m3.10b"><list id="Thmexample4.p1.3.m3.10.10.3.cmml" xref="Thmexample4.p1.3.m3.10.10.2"><apply id="Thmexample4.p1.3.m3.9.9.1.1.cmml" xref="Thmexample4.p1.3.m3.9.9.1.1"><csymbol cd="ambiguous" id="Thmexample4.p1.3.m3.9.9.1.1.1.cmml" xref="Thmexample4.p1.3.m3.9.9.1.1">subscript</csymbol><apply id="Thmexample4.p1.3.m3.9.9.1.1.2.cmml" xref="Thmexample4.p1.3.m3.9.9.1.1"><csymbol cd="ambiguous" id="Thmexample4.p1.3.m3.9.9.1.1.2.1.cmml" xref="Thmexample4.p1.3.m3.9.9.1.1">superscript</csymbol><ci id="Thmexample4.p1.3.m3.9.9.1.1.2.2.cmml" xref="Thmexample4.p1.3.m3.9.9.1.1.2.2">𝑒</ci><list id="Thmexample4.p1.3.m3.2.2.2.3.cmml" xref="Thmexample4.p1.3.m3.2.2.2.4"><ci id="Thmexample4.p1.3.m3.1.1.1.1.cmml" xref="Thmexample4.p1.3.m3.1.1.1.1">𝑙</ci><ci id="Thmexample4.p1.3.m3.2.2.2.2.cmml" xref="Thmexample4.p1.3.m3.2.2.2.2">ℎ</ci></list></apply><list id="Thmexample4.p1.3.m3.4.4.2.3.cmml" xref="Thmexample4.p1.3.m3.4.4.2.2"><ci id="Thmexample4.p1.3.m3.3.3.1.1.cmml" xref="Thmexample4.p1.3.m3.3.3.1.1">𝑖</ci><apply id="Thmexample4.p1.3.m3.4.4.2.2.1.cmml" xref="Thmexample4.p1.3.m3.4.4.2.2.1"><plus id="Thmexample4.p1.3.m3.4.4.2.2.1.1.cmml" xref="Thmexample4.p1.3.m3.4.4.2.2.1.1"></plus><ci id="Thmexample4.p1.3.m3.4.4.2.2.1.2.cmml" xref="Thmexample4.p1.3.m3.4.4.2.2.1.2">𝑖</ci><cn type="integer" id="Thmexample4.p1.3.m3.4.4.2.2.1.3.cmml" xref="Thmexample4.p1.3.m3.4.4.2.2.1.3">1</cn></apply></list></apply><apply id="Thmexample4.p1.3.m3.10.10.2.2.cmml" xref="Thmexample4.p1.3.m3.10.10.2.2"><csymbol cd="ambiguous" id="Thmexample4.p1.3.m3.10.10.2.2.1.cmml" xref="Thmexample4.p1.3.m3.10.10.2.2">subscript</csymbol><apply id="Thmexample4.p1.3.m3.10.10.2.2.2.cmml" xref="Thmexample4.p1.3.m3.10.10.2.2"><csymbol cd="ambiguous" id="Thmexample4.p1.3.m3.10.10.2.2.2.1.cmml" xref="Thmexample4.p1.3.m3.10.10.2.2">superscript</csymbol><ci id="Thmexample4.p1.3.m3.10.10.2.2.2.2.cmml" xref="Thmexample4.p1.3.m3.10.10.2.2.2.2">𝑒</ci><list id="Thmexample4.p1.3.m3.6.6.2.3.cmml" xref="Thmexample4.p1.3.m3.6.6.2.2"><apply id="Thmexample4.p1.3.m3.5.5.1.1.1.cmml" xref="Thmexample4.p1.3.m3.5.5.1.1.1"><plus id="Thmexample4.p1.3.m3.5.5.1.1.1.1.cmml" xref="Thmexample4.p1.3.m3.5.5.1.1.1.1"></plus><ci id="Thmexample4.p1.3.m3.5.5.1.1.1.2.cmml" xref="Thmexample4.p1.3.m3.5.5.1.1.1.2">𝑙</ci><cn type="integer" id="Thmexample4.p1.3.m3.5.5.1.1.1.3.cmml" xref="Thmexample4.p1.3.m3.5.5.1.1.1.3">1</cn></apply><apply id="Thmexample4.p1.3.m3.6.6.2.2.2.cmml" xref="Thmexample4.p1.3.m3.6.6.2.2.2"><csymbol cd="ambiguous" id="Thmexample4.p1.3.m3.6.6.2.2.2.1.cmml" xref="Thmexample4.p1.3.m3.6.6.2.2.2">superscript</csymbol><ci id="Thmexample4.p1.3.m3.6.6.2.2.2.2.cmml" xref="Thmexample4.p1.3.m3.6.6.2.2.2.2">ℎ</ci><ci id="Thmexample4.p1.3.m3.6.6.2.2.2.3.cmml" xref="Thmexample4.p1.3.m3.6.6.2.2.2.3">′</ci></apply></list></apply><list id="Thmexample4.p1.3.m3.8.8.2.3.cmml" xref="Thmexample4.p1.3.m3.8.8.2.2"><apply id="Thmexample4.p1.3.m3.8.8.2.2.1.cmml" xref="Thmexample4.p1.3.m3.8.8.2.2.1"><plus id="Thmexample4.p1.3.m3.8.8.2.2.1.1.cmml" xref="Thmexample4.p1.3.m3.8.8.2.2.1.1"></plus><ci id="Thmexample4.p1.3.m3.8.8.2.2.1.2.cmml" xref="Thmexample4.p1.3.m3.8.8.2.2.1.2">𝑖</ci><cn type="integer" id="Thmexample4.p1.3.m3.8.8.2.2.1.3.cmml" xref="Thmexample4.p1.3.m3.8.8.2.2.1.3">1</cn></apply><ci id="Thmexample4.p1.3.m3.7.7.1.1.cmml" xref="Thmexample4.p1.3.m3.7.7.1.1">𝑗</ci></list></apply></list></annotation-xml><annotation encoding="application/x-tex" id="Thmexample4.p1.3.m3.10c">e^{l,h}_{i,i+1},e^{l+1,h^{\prime}}_{i+1,j}</annotation></semantics></math>.
Although the associated input-output pairs “[a][b]…[a][b]” have a clear pattern, the associated outputs “[b]” alone can be arbitrary, so the induction head is an abstract knowledge.</p>
</div>
</div>
<div id="S2.SS2.p6" class="ltx_para">
<p id="S2.SS2.p6.1" class="ltx_p">More sophisticated abstract knowledges have been identified for in-context learning <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib116" title="" class="ltx_ref">116</a>]</cite> and indirect object identification <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib115" title="" class="ltx_ref">115</a>]</cite>.</p>
</div>
<div id="Thmdefinition5" class="ltx_theorem ltx_theorem_definition">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmdefinition5.1.1.1" class="ltx_text ltx_font_bold">Definition 5</span></span><span id="Thmdefinition5.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmdefinition5.p1" class="ltx_para">
<p id="Thmdefinition5.p1.5" class="ltx_p">Given a LLM and a knowledge <math id="Thmdefinition5.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{K}" display="inline"><semantics id="Thmdefinition5.p1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="Thmdefinition5.p1.1.m1.1.1" xref="Thmdefinition5.p1.1.m1.1.1.cmml">𝒦</mi><annotation-xml encoding="MathML-Content" id="Thmdefinition5.p1.1.m1.1b"><ci id="Thmdefinition5.p1.1.m1.1.1.cmml" xref="Thmdefinition5.p1.1.m1.1.1">𝒦</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition5.p1.1.m1.1c">\mathcal{K}</annotation></semantics></math>, a text <math id="Thmdefinition5.p1.2.m2.2" class="ltx_Math" alttext="\mathbf{t}=(t_{0},\dots t_{n})" display="inline"><semantics id="Thmdefinition5.p1.2.m2.2a"><mrow id="Thmdefinition5.p1.2.m2.2.2" xref="Thmdefinition5.p1.2.m2.2.2.cmml"><mi id="Thmdefinition5.p1.2.m2.2.2.4" xref="Thmdefinition5.p1.2.m2.2.2.4.cmml">𝐭</mi><mo id="Thmdefinition5.p1.2.m2.2.2.3" xref="Thmdefinition5.p1.2.m2.2.2.3.cmml">=</mo><mrow id="Thmdefinition5.p1.2.m2.2.2.2.2" xref="Thmdefinition5.p1.2.m2.2.2.2.3.cmml"><mo stretchy="false" id="Thmdefinition5.p1.2.m2.2.2.2.2.3" xref="Thmdefinition5.p1.2.m2.2.2.2.3.cmml">(</mo><msub id="Thmdefinition5.p1.2.m2.1.1.1.1.1" xref="Thmdefinition5.p1.2.m2.1.1.1.1.1.cmml"><mi id="Thmdefinition5.p1.2.m2.1.1.1.1.1.2" xref="Thmdefinition5.p1.2.m2.1.1.1.1.1.2.cmml">t</mi><mn id="Thmdefinition5.p1.2.m2.1.1.1.1.1.3" xref="Thmdefinition5.p1.2.m2.1.1.1.1.1.3.cmml">0</mn></msub><mo id="Thmdefinition5.p1.2.m2.2.2.2.2.4" xref="Thmdefinition5.p1.2.m2.2.2.2.3.cmml">,</mo><mrow id="Thmdefinition5.p1.2.m2.2.2.2.2.2" xref="Thmdefinition5.p1.2.m2.2.2.2.2.2.cmml"><mi mathvariant="normal" id="Thmdefinition5.p1.2.m2.2.2.2.2.2.2" xref="Thmdefinition5.p1.2.m2.2.2.2.2.2.2.cmml">…</mi><mo lspace="0em" rspace="0em" id="Thmdefinition5.p1.2.m2.2.2.2.2.2.1" xref="Thmdefinition5.p1.2.m2.2.2.2.2.2.1.cmml">​</mo><msub id="Thmdefinition5.p1.2.m2.2.2.2.2.2.3" xref="Thmdefinition5.p1.2.m2.2.2.2.2.2.3.cmml"><mi id="Thmdefinition5.p1.2.m2.2.2.2.2.2.3.2" xref="Thmdefinition5.p1.2.m2.2.2.2.2.2.3.2.cmml">t</mi><mi id="Thmdefinition5.p1.2.m2.2.2.2.2.2.3.3" xref="Thmdefinition5.p1.2.m2.2.2.2.2.2.3.3.cmml">n</mi></msub></mrow><mo stretchy="false" id="Thmdefinition5.p1.2.m2.2.2.2.2.5" xref="Thmdefinition5.p1.2.m2.2.2.2.3.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="Thmdefinition5.p1.2.m2.2b"><apply id="Thmdefinition5.p1.2.m2.2.2.cmml" xref="Thmdefinition5.p1.2.m2.2.2"><eq id="Thmdefinition5.p1.2.m2.2.2.3.cmml" xref="Thmdefinition5.p1.2.m2.2.2.3"></eq><ci id="Thmdefinition5.p1.2.m2.2.2.4.cmml" xref="Thmdefinition5.p1.2.m2.2.2.4">𝐭</ci><interval closure="open" id="Thmdefinition5.p1.2.m2.2.2.2.3.cmml" xref="Thmdefinition5.p1.2.m2.2.2.2.2"><apply id="Thmdefinition5.p1.2.m2.1.1.1.1.1.cmml" xref="Thmdefinition5.p1.2.m2.1.1.1.1.1"><csymbol cd="ambiguous" id="Thmdefinition5.p1.2.m2.1.1.1.1.1.1.cmml" xref="Thmdefinition5.p1.2.m2.1.1.1.1.1">subscript</csymbol><ci id="Thmdefinition5.p1.2.m2.1.1.1.1.1.2.cmml" xref="Thmdefinition5.p1.2.m2.1.1.1.1.1.2">𝑡</ci><cn type="integer" id="Thmdefinition5.p1.2.m2.1.1.1.1.1.3.cmml" xref="Thmdefinition5.p1.2.m2.1.1.1.1.1.3">0</cn></apply><apply id="Thmdefinition5.p1.2.m2.2.2.2.2.2.cmml" xref="Thmdefinition5.p1.2.m2.2.2.2.2.2"><times id="Thmdefinition5.p1.2.m2.2.2.2.2.2.1.cmml" xref="Thmdefinition5.p1.2.m2.2.2.2.2.2.1"></times><ci id="Thmdefinition5.p1.2.m2.2.2.2.2.2.2.cmml" xref="Thmdefinition5.p1.2.m2.2.2.2.2.2.2">…</ci><apply id="Thmdefinition5.p1.2.m2.2.2.2.2.2.3.cmml" xref="Thmdefinition5.p1.2.m2.2.2.2.2.2.3"><csymbol cd="ambiguous" id="Thmdefinition5.p1.2.m2.2.2.2.2.2.3.1.cmml" xref="Thmdefinition5.p1.2.m2.2.2.2.2.2.3">subscript</csymbol><ci id="Thmdefinition5.p1.2.m2.2.2.2.2.2.3.2.cmml" xref="Thmdefinition5.p1.2.m2.2.2.2.2.2.3.2">𝑡</ci><ci id="Thmdefinition5.p1.2.m2.2.2.2.2.2.3.3.cmml" xref="Thmdefinition5.p1.2.m2.2.2.2.2.2.3.3">𝑛</ci></apply></apply></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition5.p1.2.m2.2c">\mathbf{t}=(t_{0},\dots t_{n})</annotation></semantics></math> is called a <span id="Thmdefinition5.p1.5.1" class="ltx_text ltx_font_bold">realization</span> of <math id="Thmdefinition5.p1.3.m3.1" class="ltx_Math" alttext="\mathcal{K}" display="inline"><semantics id="Thmdefinition5.p1.3.m3.1a"><mi class="ltx_font_mathcaligraphic" id="Thmdefinition5.p1.3.m3.1.1" xref="Thmdefinition5.p1.3.m3.1.1.cmml">𝒦</mi><annotation-xml encoding="MathML-Content" id="Thmdefinition5.p1.3.m3.1b"><ci id="Thmdefinition5.p1.3.m3.1.1.cmml" xref="Thmdefinition5.p1.3.m3.1.1">𝒦</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition5.p1.3.m3.1c">\mathcal{K}</annotation></semantics></math>, if the computation graph on <math id="Thmdefinition5.p1.4.m4.1" class="ltx_Math" alttext="\mathbf{t}" display="inline"><semantics id="Thmdefinition5.p1.4.m4.1a"><mi id="Thmdefinition5.p1.4.m4.1.1" xref="Thmdefinition5.p1.4.m4.1.1.cmml">𝐭</mi><annotation-xml encoding="MathML-Content" id="Thmdefinition5.p1.4.m4.1b"><ci id="Thmdefinition5.p1.4.m4.1.1.cmml" xref="Thmdefinition5.p1.4.m4.1.1">𝐭</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition5.p1.4.m4.1c">\mathbf{t}</annotation></semantics></math> has a subgraph that belongs to <math id="Thmdefinition5.p1.5.m5.1" class="ltx_Math" alttext="\mathcal{K}" display="inline"><semantics id="Thmdefinition5.p1.5.m5.1a"><mi class="ltx_font_mathcaligraphic" id="Thmdefinition5.p1.5.m5.1.1" xref="Thmdefinition5.p1.5.m5.1.1.cmml">𝒦</mi><annotation-xml encoding="MathML-Content" id="Thmdefinition5.p1.5.m5.1b"><ci id="Thmdefinition5.p1.5.m5.1.1.cmml" xref="Thmdefinition5.p1.5.m5.1.1">𝒦</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition5.p1.5.m5.1c">\mathcal{K}</annotation></semantics></math>.</p>
</div>
</div>
<div id="S2.SS2.p7" class="ltx_para">
<p id="S2.SS2.p7.1" class="ltx_p">For instance, any text of the form [a][b]…[a][b] can be a realization of the abstract knowledge of induction head.</p>
</div>
<div id="S2.SS2.p8" class="ltx_para">
<p id="S2.SS2.p8.1" class="ltx_p">Our definition of knowledge is extrinsic, depending on a specific LLM, instead of intrinsic, depending only on texts.
From this perspective, Problem (<a href="#E1" title="Equation 1 ‣ 1 | Introduction ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>) can be interpreted as relocating the knowledges from an all-encompassing LLM to more efficient models equipped with memory hierarchy.
For concreteness, one can fix this reference LLM to be the latest version of ChatGPT or Claude <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib5" title="" class="ltx_ref">5</a>, <a href="#bib.bib9" title="" class="ltx_ref">9</a>]</cite>, or some infinitely large model from a properly defined limit that has learned from infinite data.</p>
</div>
<div id="Thmassumption1" class="ltx_theorem ltx_theorem_assumption">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmassumption1.1.1.1" class="ltx_text ltx_font_bold">Assumption 1</span></span><span id="Thmassumption1.2.2" class="ltx_text ltx_font_bold"> </span>(Completeness)<span id="Thmassumption1.3.3" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmassumption1.p1" class="ltx_para">
<p id="Thmassumption1.p1.5" class="ltx_p">Fix a reference LLM and a distribution of texts, let <math id="Thmassumption1.p1.1.m1.1" class="ltx_Math" alttext="G" display="inline"><semantics id="Thmassumption1.p1.1.m1.1a"><mi id="Thmassumption1.p1.1.m1.1.1" xref="Thmassumption1.p1.1.m1.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="Thmassumption1.p1.1.m1.1b"><ci id="Thmassumption1.p1.1.m1.1.1.cmml" xref="Thmassumption1.p1.1.m1.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmassumption1.p1.1.m1.1c">G</annotation></semantics></math> be the computation graph of a random text.
Assume that there exists a set <math id="Thmassumption1.p1.2.m2.1" class="ltx_Math" alttext="\mathfrak{K}" display="inline"><semantics id="Thmassumption1.p1.2.m2.1a"><mi id="Thmassumption1.p1.2.m2.1.1" xref="Thmassumption1.p1.2.m2.1.1.cmml">𝔎</mi><annotation-xml encoding="MathML-Content" id="Thmassumption1.p1.2.m2.1b"><ci id="Thmassumption1.p1.2.m2.1.1.cmml" xref="Thmassumption1.p1.2.m2.1.1">𝔎</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmassumption1.p1.2.m2.1c">\mathfrak{K}</annotation></semantics></math> of knowledges such that, with probability 1 over the random text,
the subgraph of <math id="Thmassumption1.p1.3.m3.1" class="ltx_Math" alttext="G" display="inline"><semantics id="Thmassumption1.p1.3.m3.1a"><mi id="Thmassumption1.p1.3.m3.1.1" xref="Thmassumption1.p1.3.m3.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="Thmassumption1.p1.3.m3.1b"><ci id="Thmassumption1.p1.3.m3.1.1.cmml" xref="Thmassumption1.p1.3.m3.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmassumption1.p1.3.m3.1c">G</annotation></semantics></math> induced by edges with non-negligible weights can be expressed as a union of subgraphs <math id="Thmassumption1.p1.4.m4.1" class="ltx_Math" alttext="\{S_{i}\in\mathcal{K}_{i}\}" display="inline"><semantics id="Thmassumption1.p1.4.m4.1a"><mrow id="Thmassumption1.p1.4.m4.1.1.1" xref="Thmassumption1.p1.4.m4.1.1.2.cmml"><mo stretchy="false" id="Thmassumption1.p1.4.m4.1.1.1.2" xref="Thmassumption1.p1.4.m4.1.1.2.cmml">{</mo><mrow id="Thmassumption1.p1.4.m4.1.1.1.1" xref="Thmassumption1.p1.4.m4.1.1.1.1.cmml"><msub id="Thmassumption1.p1.4.m4.1.1.1.1.2" xref="Thmassumption1.p1.4.m4.1.1.1.1.2.cmml"><mi id="Thmassumption1.p1.4.m4.1.1.1.1.2.2" xref="Thmassumption1.p1.4.m4.1.1.1.1.2.2.cmml">S</mi><mi id="Thmassumption1.p1.4.m4.1.1.1.1.2.3" xref="Thmassumption1.p1.4.m4.1.1.1.1.2.3.cmml">i</mi></msub><mo id="Thmassumption1.p1.4.m4.1.1.1.1.1" xref="Thmassumption1.p1.4.m4.1.1.1.1.1.cmml">∈</mo><msub id="Thmassumption1.p1.4.m4.1.1.1.1.3" xref="Thmassumption1.p1.4.m4.1.1.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="Thmassumption1.p1.4.m4.1.1.1.1.3.2" xref="Thmassumption1.p1.4.m4.1.1.1.1.3.2.cmml">𝒦</mi><mi id="Thmassumption1.p1.4.m4.1.1.1.1.3.3" xref="Thmassumption1.p1.4.m4.1.1.1.1.3.3.cmml">i</mi></msub></mrow><mo stretchy="false" id="Thmassumption1.p1.4.m4.1.1.1.3" xref="Thmassumption1.p1.4.m4.1.1.2.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="Thmassumption1.p1.4.m4.1b"><set id="Thmassumption1.p1.4.m4.1.1.2.cmml" xref="Thmassumption1.p1.4.m4.1.1.1"><apply id="Thmassumption1.p1.4.m4.1.1.1.1.cmml" xref="Thmassumption1.p1.4.m4.1.1.1.1"><in id="Thmassumption1.p1.4.m4.1.1.1.1.1.cmml" xref="Thmassumption1.p1.4.m4.1.1.1.1.1"></in><apply id="Thmassumption1.p1.4.m4.1.1.1.1.2.cmml" xref="Thmassumption1.p1.4.m4.1.1.1.1.2"><csymbol cd="ambiguous" id="Thmassumption1.p1.4.m4.1.1.1.1.2.1.cmml" xref="Thmassumption1.p1.4.m4.1.1.1.1.2">subscript</csymbol><ci id="Thmassumption1.p1.4.m4.1.1.1.1.2.2.cmml" xref="Thmassumption1.p1.4.m4.1.1.1.1.2.2">𝑆</ci><ci id="Thmassumption1.p1.4.m4.1.1.1.1.2.3.cmml" xref="Thmassumption1.p1.4.m4.1.1.1.1.2.3">𝑖</ci></apply><apply id="Thmassumption1.p1.4.m4.1.1.1.1.3.cmml" xref="Thmassumption1.p1.4.m4.1.1.1.1.3"><csymbol cd="ambiguous" id="Thmassumption1.p1.4.m4.1.1.1.1.3.1.cmml" xref="Thmassumption1.p1.4.m4.1.1.1.1.3">subscript</csymbol><ci id="Thmassumption1.p1.4.m4.1.1.1.1.3.2.cmml" xref="Thmassumption1.p1.4.m4.1.1.1.1.3.2">𝒦</ci><ci id="Thmassumption1.p1.4.m4.1.1.1.1.3.3.cmml" xref="Thmassumption1.p1.4.m4.1.1.1.1.3.3">𝑖</ci></apply></apply></set></annotation-xml><annotation encoding="application/x-tex" id="Thmassumption1.p1.4.m4.1c">\{S_{i}\in\mathcal{K}_{i}\}</annotation></semantics></math> from <math id="Thmassumption1.p1.5.m5.1" class="ltx_Math" alttext="\{\mathcal{K}_{i}\}\subseteq\mathfrak{K}" display="inline"><semantics id="Thmassumption1.p1.5.m5.1a"><mrow id="Thmassumption1.p1.5.m5.1.1" xref="Thmassumption1.p1.5.m5.1.1.cmml"><mrow id="Thmassumption1.p1.5.m5.1.1.1.1" xref="Thmassumption1.p1.5.m5.1.1.1.2.cmml"><mo stretchy="false" id="Thmassumption1.p1.5.m5.1.1.1.1.2" xref="Thmassumption1.p1.5.m5.1.1.1.2.cmml">{</mo><msub id="Thmassumption1.p1.5.m5.1.1.1.1.1" xref="Thmassumption1.p1.5.m5.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="Thmassumption1.p1.5.m5.1.1.1.1.1.2" xref="Thmassumption1.p1.5.m5.1.1.1.1.1.2.cmml">𝒦</mi><mi id="Thmassumption1.p1.5.m5.1.1.1.1.1.3" xref="Thmassumption1.p1.5.m5.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="Thmassumption1.p1.5.m5.1.1.1.1.3" xref="Thmassumption1.p1.5.m5.1.1.1.2.cmml">}</mo></mrow><mo id="Thmassumption1.p1.5.m5.1.1.2" xref="Thmassumption1.p1.5.m5.1.1.2.cmml">⊆</mo><mi id="Thmassumption1.p1.5.m5.1.1.3" xref="Thmassumption1.p1.5.m5.1.1.3.cmml">𝔎</mi></mrow><annotation-xml encoding="MathML-Content" id="Thmassumption1.p1.5.m5.1b"><apply id="Thmassumption1.p1.5.m5.1.1.cmml" xref="Thmassumption1.p1.5.m5.1.1"><subset id="Thmassumption1.p1.5.m5.1.1.2.cmml" xref="Thmassumption1.p1.5.m5.1.1.2"></subset><set id="Thmassumption1.p1.5.m5.1.1.1.2.cmml" xref="Thmassumption1.p1.5.m5.1.1.1.1"><apply id="Thmassumption1.p1.5.m5.1.1.1.1.1.cmml" xref="Thmassumption1.p1.5.m5.1.1.1.1.1"><csymbol cd="ambiguous" id="Thmassumption1.p1.5.m5.1.1.1.1.1.1.cmml" xref="Thmassumption1.p1.5.m5.1.1.1.1.1">subscript</csymbol><ci id="Thmassumption1.p1.5.m5.1.1.1.1.1.2.cmml" xref="Thmassumption1.p1.5.m5.1.1.1.1.1.2">𝒦</ci><ci id="Thmassumption1.p1.5.m5.1.1.1.1.1.3.cmml" xref="Thmassumption1.p1.5.m5.1.1.1.1.1.3">𝑖</ci></apply></set><ci id="Thmassumption1.p1.5.m5.1.1.3.cmml" xref="Thmassumption1.p1.5.m5.1.1.3">𝔎</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmassumption1.p1.5.m5.1c">\{\mathcal{K}_{i}\}\subseteq\mathfrak{K}</annotation></semantics></math>.</p>
</div>
</div>
<div id="S2.SS2.p9" class="ltx_para">
<p id="S2.SS2.p9.1" class="ltx_p">Essentially, Assumption <a href="#Thmassumption1" title="Assumption 1 (Completeness). ‣ 2.2 | Knowledge ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> posits that all computations in the LLM can be fully decomposed into circuits, so that the LLM is nothing more than a collection of specific and abstract knowledges.
This viewpoint underscores that the efficiency of LLMs is ultimately about the effective organization of these knowledges, an objective partially addressed by Problem (1).</p>
</div>
</section>
<section id="S2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsection" style="font-size:120%;color:#00868B;">2.3 <span id="S2.SS3.4.1" class="ltx_text" style="color:#808080;">| </span>Memory</h3>

<div id="S2.SS3.p1" class="ltx_para">
<p id="S2.SS3.p1.1" class="ltx_p">Now the question is what knowledge can be separated from the model parameters and moved to the lower levels of the memory hierarchy.</p>
</div>
<div id="Thmdefinition6" class="ltx_theorem ltx_theorem_definition">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmdefinition6.1.1.1" class="ltx_text ltx_font_bold">Definition 6</span></span><span id="Thmdefinition6.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmdefinition6.p1" class="ltx_para">
<p id="Thmdefinition6.p1.2" class="ltx_p">A knowledge <math id="Thmdefinition6.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{K}" display="inline"><semantics id="Thmdefinition6.p1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="Thmdefinition6.p1.1.m1.1.1" xref="Thmdefinition6.p1.1.m1.1.1.cmml">𝒦</mi><annotation-xml encoding="MathML-Content" id="Thmdefinition6.p1.1.m1.1b"><ci id="Thmdefinition6.p1.1.m1.1.1.cmml" xref="Thmdefinition6.p1.1.m1.1.1">𝒦</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition6.p1.1.m1.1c">\mathcal{K}</annotation></semantics></math> of the reference LLM is <span id="Thmdefinition6.p1.2.1" class="ltx_text ltx_font_bold">separable</span> if there exists another LLM <math id="Thmdefinition6.p1.2.m2.1" class="ltx_Math" alttext="M" display="inline"><semantics id="Thmdefinition6.p1.2.m2.1a"><mi id="Thmdefinition6.p1.2.m2.1.1" xref="Thmdefinition6.p1.2.m2.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="Thmdefinition6.p1.2.m2.1b"><ci id="Thmdefinition6.p1.2.m2.1.1.cmml" xref="Thmdefinition6.p1.2.m2.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition6.p1.2.m2.1c">M</annotation></semantics></math> such that</p>
<ul id="S2.I6" class="ltx_itemize">
<li id="S2.I6.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S2.I6.i1.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S2.I6.i1.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S2.I6.i1.1.1.m1.1.1" xref="S2.I6.i1.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S2.I6.i1.1.1.m1.1c"><ci id="S2.I6.i1.1.1.m1.1.1.cmml" xref="S2.I6.i1.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I6.i1.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S2.I6.i1.p1" class="ltx_para">
<p id="S2.I6.i1.p1.7" class="ltx_p"><math id="S2.I6.i1.p1.1.m1.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S2.I6.i1.p1.1.m1.1a"><mi id="S2.I6.i1.p1.1.m1.1.1" xref="S2.I6.i1.p1.1.m1.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S2.I6.i1.p1.1.m1.1b"><ci id="S2.I6.i1.p1.1.m1.1.1.cmml" xref="S2.I6.i1.p1.1.m1.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I6.i1.p1.1.m1.1c">M</annotation></semantics></math> does not possess this knowledge, such that for any realization <math id="S2.I6.i1.p1.2.m2.1" class="ltx_Math" alttext="\mathbf{t}" display="inline"><semantics id="S2.I6.i1.p1.2.m2.1a"><mi id="S2.I6.i1.p1.2.m2.1.1" xref="S2.I6.i1.p1.2.m2.1.1.cmml">𝐭</mi><annotation-xml encoding="MathML-Content" id="S2.I6.i1.p1.2.m2.1b"><ci id="S2.I6.i1.p1.2.m2.1.1.cmml" xref="S2.I6.i1.p1.2.m2.1.1">𝐭</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I6.i1.p1.2.m2.1c">\mathbf{t}</annotation></semantics></math> of <math id="S2.I6.i1.p1.3.m3.1" class="ltx_Math" alttext="\mathcal{K}" display="inline"><semantics id="S2.I6.i1.p1.3.m3.1a"><mi class="ltx_font_mathcaligraphic" id="S2.I6.i1.p1.3.m3.1.1" xref="S2.I6.i1.p1.3.m3.1.1.cmml">𝒦</mi><annotation-xml encoding="MathML-Content" id="S2.I6.i1.p1.3.m3.1b"><ci id="S2.I6.i1.p1.3.m3.1.1.cmml" xref="S2.I6.i1.p1.3.m3.1.1">𝒦</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I6.i1.p1.3.m3.1c">\mathcal{K}</annotation></semantics></math>, the model <math id="S2.I6.i1.p1.4.m4.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S2.I6.i1.p1.4.m4.1a"><mi id="S2.I6.i1.p1.4.m4.1.1" xref="S2.I6.i1.p1.4.m4.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S2.I6.i1.p1.4.m4.1b"><ci id="S2.I6.i1.p1.4.m4.1.1.cmml" xref="S2.I6.i1.p1.4.m4.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I6.i1.p1.4.m4.1c">M</annotation></semantics></math> cannot generate each token of the associated output <math id="S2.I6.i1.p1.5.m5.1" class="ltx_Math" alttext="\mathbf{t}_{\text{out}}" display="inline"><semantics id="S2.I6.i1.p1.5.m5.1a"><msub id="S2.I6.i1.p1.5.m5.1.1" xref="S2.I6.i1.p1.5.m5.1.1.cmml"><mi id="S2.I6.i1.p1.5.m5.1.1.2" xref="S2.I6.i1.p1.5.m5.1.1.2.cmml">𝐭</mi><mtext id="S2.I6.i1.p1.5.m5.1.1.3" xref="S2.I6.i1.p1.5.m5.1.1.3a.cmml">out</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.I6.i1.p1.5.m5.1b"><apply id="S2.I6.i1.p1.5.m5.1.1.cmml" xref="S2.I6.i1.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S2.I6.i1.p1.5.m5.1.1.1.cmml" xref="S2.I6.i1.p1.5.m5.1.1">subscript</csymbol><ci id="S2.I6.i1.p1.5.m5.1.1.2.cmml" xref="S2.I6.i1.p1.5.m5.1.1.2">𝐭</ci><ci id="S2.I6.i1.p1.5.m5.1.1.3a.cmml" xref="S2.I6.i1.p1.5.m5.1.1.3"><mtext mathsize="70%" id="S2.I6.i1.p1.5.m5.1.1.3.cmml" xref="S2.I6.i1.p1.5.m5.1.1.3">out</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I6.i1.p1.5.m5.1c">\mathbf{t}_{\text{out}}</annotation></semantics></math> with high probability, e.g. <math id="S2.I6.i1.p1.6.m6.1" class="ltx_Math" alttext="\mathbb{P}_{M}(t_{i}|t_{0}\dots t_{i-1})\leq 1/2" display="inline"><semantics id="S2.I6.i1.p1.6.m6.1a"><mrow id="S2.I6.i1.p1.6.m6.1.1" xref="S2.I6.i1.p1.6.m6.1.1.cmml"><mrow id="S2.I6.i1.p1.6.m6.1.1.1" xref="S2.I6.i1.p1.6.m6.1.1.1.cmml"><msub id="S2.I6.i1.p1.6.m6.1.1.1.3" xref="S2.I6.i1.p1.6.m6.1.1.1.3.cmml"><mi id="S2.I6.i1.p1.6.m6.1.1.1.3.2" xref="S2.I6.i1.p1.6.m6.1.1.1.3.2.cmml">ℙ</mi><mi id="S2.I6.i1.p1.6.m6.1.1.1.3.3" xref="S2.I6.i1.p1.6.m6.1.1.1.3.3.cmml">M</mi></msub><mo lspace="0em" rspace="0em" id="S2.I6.i1.p1.6.m6.1.1.1.2" xref="S2.I6.i1.p1.6.m6.1.1.1.2.cmml">​</mo><mrow id="S2.I6.i1.p1.6.m6.1.1.1.1.1" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.I6.i1.p1.6.m6.1.1.1.1.1.2" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.cmml"><msub id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.2" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.2.cmml"><mi id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.2.2" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.2.2.cmml">t</mi><mi id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.2.3" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.2.3.cmml">i</mi></msub><mo fence="false" id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.1" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.1.cmml">|</mo><mrow id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.cmml"><msub id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.2" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.2.cmml"><mi id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.2.2" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.2.2.cmml">t</mi><mn id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.2.3" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.2.3.cmml">0</mn></msub><mo lspace="0em" rspace="0em" id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.1" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.1.cmml">​</mo><mi mathvariant="normal" id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.3" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.3.cmml">…</mi><mo lspace="0em" rspace="0em" id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.1a" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.1.cmml">​</mo><msub id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.4" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.4.cmml"><mi id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.4.2" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.4.2.cmml">t</mi><mrow id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.4.3" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.4.3.cmml"><mi id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.4.3.2" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.4.3.2.cmml">i</mi><mo id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.4.3.1" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.4.3.1.cmml">−</mo><mn id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.4.3.3" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.4.3.3.cmml">1</mn></mrow></msub></mrow></mrow><mo stretchy="false" id="S2.I6.i1.p1.6.m6.1.1.1.1.1.3" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S2.I6.i1.p1.6.m6.1.1.2" xref="S2.I6.i1.p1.6.m6.1.1.2.cmml">≤</mo><mrow id="S2.I6.i1.p1.6.m6.1.1.3" xref="S2.I6.i1.p1.6.m6.1.1.3.cmml"><mn id="S2.I6.i1.p1.6.m6.1.1.3.2" xref="S2.I6.i1.p1.6.m6.1.1.3.2.cmml">1</mn><mo id="S2.I6.i1.p1.6.m6.1.1.3.1" xref="S2.I6.i1.p1.6.m6.1.1.3.1.cmml">/</mo><mn id="S2.I6.i1.p1.6.m6.1.1.3.3" xref="S2.I6.i1.p1.6.m6.1.1.3.3.cmml">2</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.I6.i1.p1.6.m6.1b"><apply id="S2.I6.i1.p1.6.m6.1.1.cmml" xref="S2.I6.i1.p1.6.m6.1.1"><leq id="S2.I6.i1.p1.6.m6.1.1.2.cmml" xref="S2.I6.i1.p1.6.m6.1.1.2"></leq><apply id="S2.I6.i1.p1.6.m6.1.1.1.cmml" xref="S2.I6.i1.p1.6.m6.1.1.1"><times id="S2.I6.i1.p1.6.m6.1.1.1.2.cmml" xref="S2.I6.i1.p1.6.m6.1.1.1.2"></times><apply id="S2.I6.i1.p1.6.m6.1.1.1.3.cmml" xref="S2.I6.i1.p1.6.m6.1.1.1.3"><csymbol cd="ambiguous" id="S2.I6.i1.p1.6.m6.1.1.1.3.1.cmml" xref="S2.I6.i1.p1.6.m6.1.1.1.3">subscript</csymbol><ci id="S2.I6.i1.p1.6.m6.1.1.1.3.2.cmml" xref="S2.I6.i1.p1.6.m6.1.1.1.3.2">ℙ</ci><ci id="S2.I6.i1.p1.6.m6.1.1.1.3.3.cmml" xref="S2.I6.i1.p1.6.m6.1.1.1.3.3">𝑀</ci></apply><apply id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.cmml" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1"><csymbol cd="latexml" id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.1.cmml" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.1">conditional</csymbol><apply id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.2.cmml" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.2.1.cmml" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.2">subscript</csymbol><ci id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.2.2.cmml" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.2.2">𝑡</ci><ci id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.2.3.cmml" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.2.3">𝑖</ci></apply><apply id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.cmml" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3"><times id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.1.cmml" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.1"></times><apply id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.2.cmml" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.2.1.cmml" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.2">subscript</csymbol><ci id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.2.2.cmml" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.2.2">𝑡</ci><cn type="integer" id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.2.3.cmml" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.2.3">0</cn></apply><ci id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.3.cmml" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.3">…</ci><apply id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.4.cmml" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.4"><csymbol cd="ambiguous" id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.4.1.cmml" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.4">subscript</csymbol><ci id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.4.2.cmml" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.4.2">𝑡</ci><apply id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.4.3.cmml" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.4.3"><minus id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.4.3.1.cmml" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.4.3.1"></minus><ci id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.4.3.2.cmml" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.4.3.2">𝑖</ci><cn type="integer" id="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.4.3.3.cmml" xref="S2.I6.i1.p1.6.m6.1.1.1.1.1.1.3.4.3.3">1</cn></apply></apply></apply></apply></apply><apply id="S2.I6.i1.p1.6.m6.1.1.3.cmml" xref="S2.I6.i1.p1.6.m6.1.1.3"><divide id="S2.I6.i1.p1.6.m6.1.1.3.1.cmml" xref="S2.I6.i1.p1.6.m6.1.1.3.1"></divide><cn type="integer" id="S2.I6.i1.p1.6.m6.1.1.3.2.cmml" xref="S2.I6.i1.p1.6.m6.1.1.3.2">1</cn><cn type="integer" id="S2.I6.i1.p1.6.m6.1.1.3.3.cmml" xref="S2.I6.i1.p1.6.m6.1.1.3.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I6.i1.p1.6.m6.1c">\mathbb{P}_{M}(t_{i}|t_{0}\dots t_{i-1})\leq 1/2</annotation></semantics></math> for some <math id="S2.I6.i1.p1.7.m7.1" class="ltx_Math" alttext="t_{i}\in\mathbf{t}_{\text{out}}" display="inline"><semantics id="S2.I6.i1.p1.7.m7.1a"><mrow id="S2.I6.i1.p1.7.m7.1.1" xref="S2.I6.i1.p1.7.m7.1.1.cmml"><msub id="S2.I6.i1.p1.7.m7.1.1.2" xref="S2.I6.i1.p1.7.m7.1.1.2.cmml"><mi id="S2.I6.i1.p1.7.m7.1.1.2.2" xref="S2.I6.i1.p1.7.m7.1.1.2.2.cmml">t</mi><mi id="S2.I6.i1.p1.7.m7.1.1.2.3" xref="S2.I6.i1.p1.7.m7.1.1.2.3.cmml">i</mi></msub><mo id="S2.I6.i1.p1.7.m7.1.1.1" xref="S2.I6.i1.p1.7.m7.1.1.1.cmml">∈</mo><msub id="S2.I6.i1.p1.7.m7.1.1.3" xref="S2.I6.i1.p1.7.m7.1.1.3.cmml"><mi id="S2.I6.i1.p1.7.m7.1.1.3.2" xref="S2.I6.i1.p1.7.m7.1.1.3.2.cmml">𝐭</mi><mtext id="S2.I6.i1.p1.7.m7.1.1.3.3" xref="S2.I6.i1.p1.7.m7.1.1.3.3a.cmml">out</mtext></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.I6.i1.p1.7.m7.1b"><apply id="S2.I6.i1.p1.7.m7.1.1.cmml" xref="S2.I6.i1.p1.7.m7.1.1"><in id="S2.I6.i1.p1.7.m7.1.1.1.cmml" xref="S2.I6.i1.p1.7.m7.1.1.1"></in><apply id="S2.I6.i1.p1.7.m7.1.1.2.cmml" xref="S2.I6.i1.p1.7.m7.1.1.2"><csymbol cd="ambiguous" id="S2.I6.i1.p1.7.m7.1.1.2.1.cmml" xref="S2.I6.i1.p1.7.m7.1.1.2">subscript</csymbol><ci id="S2.I6.i1.p1.7.m7.1.1.2.2.cmml" xref="S2.I6.i1.p1.7.m7.1.1.2.2">𝑡</ci><ci id="S2.I6.i1.p1.7.m7.1.1.2.3.cmml" xref="S2.I6.i1.p1.7.m7.1.1.2.3">𝑖</ci></apply><apply id="S2.I6.i1.p1.7.m7.1.1.3.cmml" xref="S2.I6.i1.p1.7.m7.1.1.3"><csymbol cd="ambiguous" id="S2.I6.i1.p1.7.m7.1.1.3.1.cmml" xref="S2.I6.i1.p1.7.m7.1.1.3">subscript</csymbol><ci id="S2.I6.i1.p1.7.m7.1.1.3.2.cmml" xref="S2.I6.i1.p1.7.m7.1.1.3.2">𝐭</ci><ci id="S2.I6.i1.p1.7.m7.1.1.3.3a.cmml" xref="S2.I6.i1.p1.7.m7.1.1.3.3"><mtext mathsize="70%" id="S2.I6.i1.p1.7.m7.1.1.3.3.cmml" xref="S2.I6.i1.p1.7.m7.1.1.3.3">out</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I6.i1.p1.7.m7.1c">t_{i}\in\mathbf{t}_{\text{out}}</annotation></semantics></math>.</p>
</div>
</li>
<li id="S2.I6.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S2.I6.i2.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S2.I6.i2.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S2.I6.i2.1.1.m1.1.1" xref="S2.I6.i2.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S2.I6.i2.1.1.m1.1c"><ci id="S2.I6.i2.1.1.m1.1.1.cmml" xref="S2.I6.i2.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I6.i2.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S2.I6.i2.p1" class="ltx_para">
<p id="S2.I6.i2.p1.8" class="ltx_p">There exists a text <math id="S2.I6.i2.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{t}_{*}" display="inline"><semantics id="S2.I6.i2.p1.1.m1.1a"><msub id="S2.I6.i2.p1.1.m1.1.1" xref="S2.I6.i2.p1.1.m1.1.1.cmml"><mi id="S2.I6.i2.p1.1.m1.1.1.2" xref="S2.I6.i2.p1.1.m1.1.1.2.cmml">𝐭</mi><mo id="S2.I6.i2.p1.1.m1.1.1.3" xref="S2.I6.i2.p1.1.m1.1.1.3.cmml">∗</mo></msub><annotation-xml encoding="MathML-Content" id="S2.I6.i2.p1.1.m1.1b"><apply id="S2.I6.i2.p1.1.m1.1.1.cmml" xref="S2.I6.i2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.I6.i2.p1.1.m1.1.1.1.cmml" xref="S2.I6.i2.p1.1.m1.1.1">subscript</csymbol><ci id="S2.I6.i2.p1.1.m1.1.1.2.cmml" xref="S2.I6.i2.p1.1.m1.1.1.2">𝐭</ci><times id="S2.I6.i2.p1.1.m1.1.1.3.cmml" xref="S2.I6.i2.p1.1.m1.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I6.i2.p1.1.m1.1c">\mathbf{t}_{*}</annotation></semantics></math> such that for any realization <math id="S2.I6.i2.p1.2.m2.1" class="ltx_Math" alttext="\mathbf{t}" display="inline"><semantics id="S2.I6.i2.p1.2.m2.1a"><mi id="S2.I6.i2.p1.2.m2.1.1" xref="S2.I6.i2.p1.2.m2.1.1.cmml">𝐭</mi><annotation-xml encoding="MathML-Content" id="S2.I6.i2.p1.2.m2.1b"><ci id="S2.I6.i2.p1.2.m2.1.1.cmml" xref="S2.I6.i2.p1.2.m2.1.1">𝐭</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I6.i2.p1.2.m2.1c">\mathbf{t}</annotation></semantics></math> of <math id="S2.I6.i2.p1.3.m3.1" class="ltx_Math" alttext="\mathcal{K}" display="inline"><semantics id="S2.I6.i2.p1.3.m3.1a"><mi class="ltx_font_mathcaligraphic" id="S2.I6.i2.p1.3.m3.1.1" xref="S2.I6.i2.p1.3.m3.1.1.cmml">𝒦</mi><annotation-xml encoding="MathML-Content" id="S2.I6.i2.p1.3.m3.1b"><ci id="S2.I6.i2.p1.3.m3.1.1.cmml" xref="S2.I6.i2.p1.3.m3.1.1">𝒦</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I6.i2.p1.3.m3.1c">\mathcal{K}</annotation></semantics></math>,
the model <math id="S2.I6.i2.p1.4.m4.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S2.I6.i2.p1.4.m4.1a"><mi id="S2.I6.i2.p1.4.m4.1.1" xref="S2.I6.i2.p1.4.m4.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S2.I6.i2.p1.4.m4.1b"><ci id="S2.I6.i2.p1.4.m4.1.1.cmml" xref="S2.I6.i2.p1.4.m4.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I6.i2.p1.4.m4.1c">M</annotation></semantics></math> using <math id="S2.I6.i2.p1.5.m5.1" class="ltx_Math" alttext="\mathbf{t}_{*}" display="inline"><semantics id="S2.I6.i2.p1.5.m5.1a"><msub id="S2.I6.i2.p1.5.m5.1.1" xref="S2.I6.i2.p1.5.m5.1.1.cmml"><mi id="S2.I6.i2.p1.5.m5.1.1.2" xref="S2.I6.i2.p1.5.m5.1.1.2.cmml">𝐭</mi><mo id="S2.I6.i2.p1.5.m5.1.1.3" xref="S2.I6.i2.p1.5.m5.1.1.3.cmml">∗</mo></msub><annotation-xml encoding="MathML-Content" id="S2.I6.i2.p1.5.m5.1b"><apply id="S2.I6.i2.p1.5.m5.1.1.cmml" xref="S2.I6.i2.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S2.I6.i2.p1.5.m5.1.1.1.cmml" xref="S2.I6.i2.p1.5.m5.1.1">subscript</csymbol><ci id="S2.I6.i2.p1.5.m5.1.1.2.cmml" xref="S2.I6.i2.p1.5.m5.1.1.2">𝐭</ci><times id="S2.I6.i2.p1.5.m5.1.1.3.cmml" xref="S2.I6.i2.p1.5.m5.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I6.i2.p1.5.m5.1c">\mathbf{t}_{*}</annotation></semantics></math> as prefix can generate each token of the associated output <math id="S2.I6.i2.p1.6.m6.1" class="ltx_Math" alttext="\mathbf{t}_{\text{out}}" display="inline"><semantics id="S2.I6.i2.p1.6.m6.1a"><msub id="S2.I6.i2.p1.6.m6.1.1" xref="S2.I6.i2.p1.6.m6.1.1.cmml"><mi id="S2.I6.i2.p1.6.m6.1.1.2" xref="S2.I6.i2.p1.6.m6.1.1.2.cmml">𝐭</mi><mtext id="S2.I6.i2.p1.6.m6.1.1.3" xref="S2.I6.i2.p1.6.m6.1.1.3a.cmml">out</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.I6.i2.p1.6.m6.1b"><apply id="S2.I6.i2.p1.6.m6.1.1.cmml" xref="S2.I6.i2.p1.6.m6.1.1"><csymbol cd="ambiguous" id="S2.I6.i2.p1.6.m6.1.1.1.cmml" xref="S2.I6.i2.p1.6.m6.1.1">subscript</csymbol><ci id="S2.I6.i2.p1.6.m6.1.1.2.cmml" xref="S2.I6.i2.p1.6.m6.1.1.2">𝐭</ci><ci id="S2.I6.i2.p1.6.m6.1.1.3a.cmml" xref="S2.I6.i2.p1.6.m6.1.1.3"><mtext mathsize="70%" id="S2.I6.i2.p1.6.m6.1.1.3.cmml" xref="S2.I6.i2.p1.6.m6.1.1.3">out</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I6.i2.p1.6.m6.1c">\mathbf{t}_{\text{out}}</annotation></semantics></math> with high probability,
e.g. <math id="S2.I6.i2.p1.7.m7.1" class="ltx_Math" alttext="\mathbb{P}_{M}(t_{i}|\mathbf{t}_{*}t_{0}\dots t_{i-1})\geq 0.9" display="inline"><semantics id="S2.I6.i2.p1.7.m7.1a"><mrow id="S2.I6.i2.p1.7.m7.1.1" xref="S2.I6.i2.p1.7.m7.1.1.cmml"><mrow id="S2.I6.i2.p1.7.m7.1.1.1" xref="S2.I6.i2.p1.7.m7.1.1.1.cmml"><msub id="S2.I6.i2.p1.7.m7.1.1.1.3" xref="S2.I6.i2.p1.7.m7.1.1.1.3.cmml"><mi id="S2.I6.i2.p1.7.m7.1.1.1.3.2" xref="S2.I6.i2.p1.7.m7.1.1.1.3.2.cmml">ℙ</mi><mi id="S2.I6.i2.p1.7.m7.1.1.1.3.3" xref="S2.I6.i2.p1.7.m7.1.1.1.3.3.cmml">M</mi></msub><mo lspace="0em" rspace="0em" id="S2.I6.i2.p1.7.m7.1.1.1.2" xref="S2.I6.i2.p1.7.m7.1.1.1.2.cmml">​</mo><mrow id="S2.I6.i2.p1.7.m7.1.1.1.1.1" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.I6.i2.p1.7.m7.1.1.1.1.1.2" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.cmml"><msub id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.2" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.2.cmml"><mi id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.2.2" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.2.2.cmml">t</mi><mi id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.2.3" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.2.3.cmml">i</mi></msub><mo fence="false" id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.1" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.1.cmml">|</mo><mrow id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.cmml"><msub id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.2" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.2.cmml"><mi id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.2.2" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.2.2.cmml">𝐭</mi><mo id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.2.3" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.2.3.cmml">∗</mo></msub><mo lspace="0em" rspace="0em" id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.1" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.1.cmml">​</mo><msub id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.3" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.3.cmml"><mi id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.3.2" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.3.2.cmml">t</mi><mn id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.3.3" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.3.3.cmml">0</mn></msub><mo lspace="0em" rspace="0em" id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.1a" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.1.cmml">​</mo><mi mathvariant="normal" id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.4" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.4.cmml">…</mi><mo lspace="0em" rspace="0em" id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.1b" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.1.cmml">​</mo><msub id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.5" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.5.cmml"><mi id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.5.2" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.5.2.cmml">t</mi><mrow id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.5.3" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.5.3.cmml"><mi id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.5.3.2" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.5.3.2.cmml">i</mi><mo id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.5.3.1" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.5.3.1.cmml">−</mo><mn id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.5.3.3" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.5.3.3.cmml">1</mn></mrow></msub></mrow></mrow><mo stretchy="false" id="S2.I6.i2.p1.7.m7.1.1.1.1.1.3" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S2.I6.i2.p1.7.m7.1.1.2" xref="S2.I6.i2.p1.7.m7.1.1.2.cmml">≥</mo><mn id="S2.I6.i2.p1.7.m7.1.1.3" xref="S2.I6.i2.p1.7.m7.1.1.3.cmml">0.9</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.I6.i2.p1.7.m7.1b"><apply id="S2.I6.i2.p1.7.m7.1.1.cmml" xref="S2.I6.i2.p1.7.m7.1.1"><geq id="S2.I6.i2.p1.7.m7.1.1.2.cmml" xref="S2.I6.i2.p1.7.m7.1.1.2"></geq><apply id="S2.I6.i2.p1.7.m7.1.1.1.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1"><times id="S2.I6.i2.p1.7.m7.1.1.1.2.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.2"></times><apply id="S2.I6.i2.p1.7.m7.1.1.1.3.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.3"><csymbol cd="ambiguous" id="S2.I6.i2.p1.7.m7.1.1.1.3.1.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.3">subscript</csymbol><ci id="S2.I6.i2.p1.7.m7.1.1.1.3.2.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.3.2">ℙ</ci><ci id="S2.I6.i2.p1.7.m7.1.1.1.3.3.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.3.3">𝑀</ci></apply><apply id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1"><csymbol cd="latexml" id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.1.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.1">conditional</csymbol><apply id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.2.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.2.1.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.2">subscript</csymbol><ci id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.2.2.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.2.2">𝑡</ci><ci id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.2.3.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.2.3">𝑖</ci></apply><apply id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3"><times id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.1.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.1"></times><apply id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.2.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.2.1.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.2">subscript</csymbol><ci id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.2.2.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.2.2">𝐭</ci><times id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.2.3.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.2.3"></times></apply><apply id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.3.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.3.1.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.3">subscript</csymbol><ci id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.3.2.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.3.2">𝑡</ci><cn type="integer" id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.3.3.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.3.3">0</cn></apply><ci id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.4.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.4">…</ci><apply id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.5.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.5"><csymbol cd="ambiguous" id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.5.1.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.5">subscript</csymbol><ci id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.5.2.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.5.2">𝑡</ci><apply id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.5.3.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.5.3"><minus id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.5.3.1.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.5.3.1"></minus><ci id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.5.3.2.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.5.3.2">𝑖</ci><cn type="integer" id="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.5.3.3.cmml" xref="S2.I6.i2.p1.7.m7.1.1.1.1.1.1.3.5.3.3">1</cn></apply></apply></apply></apply></apply><cn type="float" id="S2.I6.i2.p1.7.m7.1.1.3.cmml" xref="S2.I6.i2.p1.7.m7.1.1.3">0.9</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I6.i2.p1.7.m7.1c">\mathbb{P}_{M}(t_{i}|\mathbf{t}_{*}t_{0}\dots t_{i-1})\geq 0.9</annotation></semantics></math> for every <math id="S2.I6.i2.p1.8.m8.1" class="ltx_Math" alttext="t_{i}\in\mathbf{t}_{\text{out}}" display="inline"><semantics id="S2.I6.i2.p1.8.m8.1a"><mrow id="S2.I6.i2.p1.8.m8.1.1" xref="S2.I6.i2.p1.8.m8.1.1.cmml"><msub id="S2.I6.i2.p1.8.m8.1.1.2" xref="S2.I6.i2.p1.8.m8.1.1.2.cmml"><mi id="S2.I6.i2.p1.8.m8.1.1.2.2" xref="S2.I6.i2.p1.8.m8.1.1.2.2.cmml">t</mi><mi id="S2.I6.i2.p1.8.m8.1.1.2.3" xref="S2.I6.i2.p1.8.m8.1.1.2.3.cmml">i</mi></msub><mo id="S2.I6.i2.p1.8.m8.1.1.1" xref="S2.I6.i2.p1.8.m8.1.1.1.cmml">∈</mo><msub id="S2.I6.i2.p1.8.m8.1.1.3" xref="S2.I6.i2.p1.8.m8.1.1.3.cmml"><mi id="S2.I6.i2.p1.8.m8.1.1.3.2" xref="S2.I6.i2.p1.8.m8.1.1.3.2.cmml">𝐭</mi><mtext id="S2.I6.i2.p1.8.m8.1.1.3.3" xref="S2.I6.i2.p1.8.m8.1.1.3.3a.cmml">out</mtext></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.I6.i2.p1.8.m8.1b"><apply id="S2.I6.i2.p1.8.m8.1.1.cmml" xref="S2.I6.i2.p1.8.m8.1.1"><in id="S2.I6.i2.p1.8.m8.1.1.1.cmml" xref="S2.I6.i2.p1.8.m8.1.1.1"></in><apply id="S2.I6.i2.p1.8.m8.1.1.2.cmml" xref="S2.I6.i2.p1.8.m8.1.1.2"><csymbol cd="ambiguous" id="S2.I6.i2.p1.8.m8.1.1.2.1.cmml" xref="S2.I6.i2.p1.8.m8.1.1.2">subscript</csymbol><ci id="S2.I6.i2.p1.8.m8.1.1.2.2.cmml" xref="S2.I6.i2.p1.8.m8.1.1.2.2">𝑡</ci><ci id="S2.I6.i2.p1.8.m8.1.1.2.3.cmml" xref="S2.I6.i2.p1.8.m8.1.1.2.3">𝑖</ci></apply><apply id="S2.I6.i2.p1.8.m8.1.1.3.cmml" xref="S2.I6.i2.p1.8.m8.1.1.3"><csymbol cd="ambiguous" id="S2.I6.i2.p1.8.m8.1.1.3.1.cmml" xref="S2.I6.i2.p1.8.m8.1.1.3">subscript</csymbol><ci id="S2.I6.i2.p1.8.m8.1.1.3.2.cmml" xref="S2.I6.i2.p1.8.m8.1.1.3.2">𝐭</ci><ci id="S2.I6.i2.p1.8.m8.1.1.3.3a.cmml" xref="S2.I6.i2.p1.8.m8.1.1.3.3"><mtext mathsize="70%" id="S2.I6.i2.p1.8.m8.1.1.3.3.cmml" xref="S2.I6.i2.p1.8.m8.1.1.3.3">out</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I6.i2.p1.8.m8.1c">t_{i}\in\mathbf{t}_{\text{out}}</annotation></semantics></math>.</p>
</div>
</li>
</ul>
<p id="Thmdefinition6.p1.6" class="ltx_p">If among the realizations of <math id="Thmdefinition6.p1.3.m1.1" class="ltx_Math" alttext="\mathcal{K}" display="inline"><semantics id="Thmdefinition6.p1.3.m1.1a"><mi class="ltx_font_mathcaligraphic" id="Thmdefinition6.p1.3.m1.1.1" xref="Thmdefinition6.p1.3.m1.1.1.cmml">𝒦</mi><annotation-xml encoding="MathML-Content" id="Thmdefinition6.p1.3.m1.1b"><ci id="Thmdefinition6.p1.3.m1.1.1.cmml" xref="Thmdefinition6.p1.3.m1.1.1">𝒦</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition6.p1.3.m1.1c">\mathcal{K}</annotation></semantics></math>, the same associated input <math id="Thmdefinition6.p1.4.m2.1" class="ltx_Math" alttext="\mathbf{t}_{\text{in}}" display="inline"><semantics id="Thmdefinition6.p1.4.m2.1a"><msub id="Thmdefinition6.p1.4.m2.1.1" xref="Thmdefinition6.p1.4.m2.1.1.cmml"><mi id="Thmdefinition6.p1.4.m2.1.1.2" xref="Thmdefinition6.p1.4.m2.1.1.2.cmml">𝐭</mi><mtext id="Thmdefinition6.p1.4.m2.1.1.3" xref="Thmdefinition6.p1.4.m2.1.1.3a.cmml">in</mtext></msub><annotation-xml encoding="MathML-Content" id="Thmdefinition6.p1.4.m2.1b"><apply id="Thmdefinition6.p1.4.m2.1.1.cmml" xref="Thmdefinition6.p1.4.m2.1.1"><csymbol cd="ambiguous" id="Thmdefinition6.p1.4.m2.1.1.1.cmml" xref="Thmdefinition6.p1.4.m2.1.1">subscript</csymbol><ci id="Thmdefinition6.p1.4.m2.1.1.2.cmml" xref="Thmdefinition6.p1.4.m2.1.1.2">𝐭</ci><ci id="Thmdefinition6.p1.4.m2.1.1.3a.cmml" xref="Thmdefinition6.p1.4.m2.1.1.3"><mtext mathsize="70%" id="Thmdefinition6.p1.4.m2.1.1.3.cmml" xref="Thmdefinition6.p1.4.m2.1.1.3">in</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition6.p1.4.m2.1c">\mathbf{t}_{\text{in}}</annotation></semantics></math> can correspond to multiple associated outputs <math id="Thmdefinition6.p1.5.m3.1" class="ltx_Math" alttext="\mathbf{t}_{\text{out}}" display="inline"><semantics id="Thmdefinition6.p1.5.m3.1a"><msub id="Thmdefinition6.p1.5.m3.1.1" xref="Thmdefinition6.p1.5.m3.1.1.cmml"><mi id="Thmdefinition6.p1.5.m3.1.1.2" xref="Thmdefinition6.p1.5.m3.1.1.2.cmml">𝐭</mi><mtext id="Thmdefinition6.p1.5.m3.1.1.3" xref="Thmdefinition6.p1.5.m3.1.1.3a.cmml">out</mtext></msub><annotation-xml encoding="MathML-Content" id="Thmdefinition6.p1.5.m3.1b"><apply id="Thmdefinition6.p1.5.m3.1.1.cmml" xref="Thmdefinition6.p1.5.m3.1.1"><csymbol cd="ambiguous" id="Thmdefinition6.p1.5.m3.1.1.1.cmml" xref="Thmdefinition6.p1.5.m3.1.1">subscript</csymbol><ci id="Thmdefinition6.p1.5.m3.1.1.2.cmml" xref="Thmdefinition6.p1.5.m3.1.1.2">𝐭</ci><ci id="Thmdefinition6.p1.5.m3.1.1.3a.cmml" xref="Thmdefinition6.p1.5.m3.1.1.3"><mtext mathsize="70%" id="Thmdefinition6.p1.5.m3.1.1.3.cmml" xref="Thmdefinition6.p1.5.m3.1.1.3">out</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition6.p1.5.m3.1c">\mathbf{t}_{\text{out}}</annotation></semantics></math>, then the above probabilities are summed over all branches if position <math id="Thmdefinition6.p1.6.m4.1" class="ltx_Math" alttext="i" display="inline"><semantics id="Thmdefinition6.p1.6.m4.1a"><mi id="Thmdefinition6.p1.6.m4.1.1" xref="Thmdefinition6.p1.6.m4.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="Thmdefinition6.p1.6.m4.1b"><ci id="Thmdefinition6.p1.6.m4.1.1.cmml" xref="Thmdefinition6.p1.6.m4.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition6.p1.6.m4.1c">i</annotation></semantics></math> is a branching point.</p>
</div>
</div>
<div id="Thmdefinition7" class="ltx_theorem ltx_theorem_definition">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmdefinition7.1.1.1" class="ltx_text ltx_font_bold">Definition 7</span></span><span id="Thmdefinition7.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmdefinition7.p1" class="ltx_para">
<p id="Thmdefinition7.p1.8" class="ltx_p">A separable knowledge <math id="Thmdefinition7.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{K}" display="inline"><semantics id="Thmdefinition7.p1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="Thmdefinition7.p1.1.m1.1.1" xref="Thmdefinition7.p1.1.m1.1.1.cmml">𝒦</mi><annotation-xml encoding="MathML-Content" id="Thmdefinition7.p1.1.m1.1b"><ci id="Thmdefinition7.p1.1.m1.1.1.cmml" xref="Thmdefinition7.p1.1.m1.1.1">𝒦</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition7.p1.1.m1.1c">\mathcal{K}</annotation></semantics></math> of the reference LLM is <span id="Thmdefinition7.p1.8.1" class="ltx_text ltx_font_bold">imitable</span> if any realization <math id="Thmdefinition7.p1.2.m2.1" class="ltx_Math" alttext="\mathbf{t}^{\prime}" display="inline"><semantics id="Thmdefinition7.p1.2.m2.1a"><msup id="Thmdefinition7.p1.2.m2.1.1" xref="Thmdefinition7.p1.2.m2.1.1.cmml"><mi id="Thmdefinition7.p1.2.m2.1.1.2" xref="Thmdefinition7.p1.2.m2.1.1.2.cmml">𝐭</mi><mo id="Thmdefinition7.p1.2.m2.1.1.3" xref="Thmdefinition7.p1.2.m2.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="Thmdefinition7.p1.2.m2.1b"><apply id="Thmdefinition7.p1.2.m2.1.1.cmml" xref="Thmdefinition7.p1.2.m2.1.1"><csymbol cd="ambiguous" id="Thmdefinition7.p1.2.m2.1.1.1.cmml" xref="Thmdefinition7.p1.2.m2.1.1">superscript</csymbol><ci id="Thmdefinition7.p1.2.m2.1.1.2.cmml" xref="Thmdefinition7.p1.2.m2.1.1.2">𝐭</ci><ci id="Thmdefinition7.p1.2.m2.1.1.3.cmml" xref="Thmdefinition7.p1.2.m2.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition7.p1.2.m2.1c">\mathbf{t}^{\prime}</annotation></semantics></math> of <math id="Thmdefinition7.p1.3.m3.1" class="ltx_Math" alttext="\mathcal{K}" display="inline"><semantics id="Thmdefinition7.p1.3.m3.1a"><mi class="ltx_font_mathcaligraphic" id="Thmdefinition7.p1.3.m3.1.1" xref="Thmdefinition7.p1.3.m3.1.1.cmml">𝒦</mi><annotation-xml encoding="MathML-Content" id="Thmdefinition7.p1.3.m3.1b"><ci id="Thmdefinition7.p1.3.m3.1.1.cmml" xref="Thmdefinition7.p1.3.m3.1.1">𝒦</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition7.p1.3.m3.1c">\mathcal{K}</annotation></semantics></math> can be used as the prefix <math id="Thmdefinition7.p1.4.m4.1" class="ltx_Math" alttext="\mathbf{t}_{*}" display="inline"><semantics id="Thmdefinition7.p1.4.m4.1a"><msub id="Thmdefinition7.p1.4.m4.1.1" xref="Thmdefinition7.p1.4.m4.1.1.cmml"><mi id="Thmdefinition7.p1.4.m4.1.1.2" xref="Thmdefinition7.p1.4.m4.1.1.2.cmml">𝐭</mi><mo id="Thmdefinition7.p1.4.m4.1.1.3" xref="Thmdefinition7.p1.4.m4.1.1.3.cmml">∗</mo></msub><annotation-xml encoding="MathML-Content" id="Thmdefinition7.p1.4.m4.1b"><apply id="Thmdefinition7.p1.4.m4.1.1.cmml" xref="Thmdefinition7.p1.4.m4.1.1"><csymbol cd="ambiguous" id="Thmdefinition7.p1.4.m4.1.1.1.cmml" xref="Thmdefinition7.p1.4.m4.1.1">subscript</csymbol><ci id="Thmdefinition7.p1.4.m4.1.1.2.cmml" xref="Thmdefinition7.p1.4.m4.1.1.2">𝐭</ci><times id="Thmdefinition7.p1.4.m4.1.1.3.cmml" xref="Thmdefinition7.p1.4.m4.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition7.p1.4.m4.1c">\mathbf{t}_{*}</annotation></semantics></math> in Definition <a href="#Thmdefinition6" title="Definition 6. ‣ 2.3 | Memory ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>,
e.g. for any realizations <math id="Thmdefinition7.p1.5.m5.2" class="ltx_Math" alttext="\mathbf{t},\mathbf{t}^{\prime}" display="inline"><semantics id="Thmdefinition7.p1.5.m5.2a"><mrow id="Thmdefinition7.p1.5.m5.2.2.1" xref="Thmdefinition7.p1.5.m5.2.2.2.cmml"><mi id="Thmdefinition7.p1.5.m5.1.1" xref="Thmdefinition7.p1.5.m5.1.1.cmml">𝐭</mi><mo id="Thmdefinition7.p1.5.m5.2.2.1.2" xref="Thmdefinition7.p1.5.m5.2.2.2.cmml">,</mo><msup id="Thmdefinition7.p1.5.m5.2.2.1.1" xref="Thmdefinition7.p1.5.m5.2.2.1.1.cmml"><mi id="Thmdefinition7.p1.5.m5.2.2.1.1.2" xref="Thmdefinition7.p1.5.m5.2.2.1.1.2.cmml">𝐭</mi><mo id="Thmdefinition7.p1.5.m5.2.2.1.1.3" xref="Thmdefinition7.p1.5.m5.2.2.1.1.3.cmml">′</mo></msup></mrow><annotation-xml encoding="MathML-Content" id="Thmdefinition7.p1.5.m5.2b"><list id="Thmdefinition7.p1.5.m5.2.2.2.cmml" xref="Thmdefinition7.p1.5.m5.2.2.1"><ci id="Thmdefinition7.p1.5.m5.1.1.cmml" xref="Thmdefinition7.p1.5.m5.1.1">𝐭</ci><apply id="Thmdefinition7.p1.5.m5.2.2.1.1.cmml" xref="Thmdefinition7.p1.5.m5.2.2.1.1"><csymbol cd="ambiguous" id="Thmdefinition7.p1.5.m5.2.2.1.1.1.cmml" xref="Thmdefinition7.p1.5.m5.2.2.1.1">superscript</csymbol><ci id="Thmdefinition7.p1.5.m5.2.2.1.1.2.cmml" xref="Thmdefinition7.p1.5.m5.2.2.1.1.2">𝐭</ci><ci id="Thmdefinition7.p1.5.m5.2.2.1.1.3.cmml" xref="Thmdefinition7.p1.5.m5.2.2.1.1.3">′</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition7.p1.5.m5.2c">\mathbf{t},\mathbf{t}^{\prime}</annotation></semantics></math> of <math id="Thmdefinition7.p1.6.m6.1" class="ltx_Math" alttext="\mathcal{K}" display="inline"><semantics id="Thmdefinition7.p1.6.m6.1a"><mi class="ltx_font_mathcaligraphic" id="Thmdefinition7.p1.6.m6.1.1" xref="Thmdefinition7.p1.6.m6.1.1.cmml">𝒦</mi><annotation-xml encoding="MathML-Content" id="Thmdefinition7.p1.6.m6.1b"><ci id="Thmdefinition7.p1.6.m6.1.1.cmml" xref="Thmdefinition7.p1.6.m6.1.1">𝒦</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition7.p1.6.m6.1c">\mathcal{K}</annotation></semantics></math>, we have <math id="Thmdefinition7.p1.7.m7.1" class="ltx_Math" alttext="\mathbb{P}_{M}(t_{i}|\mathbf{t}^{\prime}t_{0}\dots t_{i-1})\geq 0.9" display="inline"><semantics id="Thmdefinition7.p1.7.m7.1a"><mrow id="Thmdefinition7.p1.7.m7.1.1" xref="Thmdefinition7.p1.7.m7.1.1.cmml"><mrow id="Thmdefinition7.p1.7.m7.1.1.1" xref="Thmdefinition7.p1.7.m7.1.1.1.cmml"><msub id="Thmdefinition7.p1.7.m7.1.1.1.3" xref="Thmdefinition7.p1.7.m7.1.1.1.3.cmml"><mi id="Thmdefinition7.p1.7.m7.1.1.1.3.2" xref="Thmdefinition7.p1.7.m7.1.1.1.3.2.cmml">ℙ</mi><mi id="Thmdefinition7.p1.7.m7.1.1.1.3.3" xref="Thmdefinition7.p1.7.m7.1.1.1.3.3.cmml">M</mi></msub><mo lspace="0em" rspace="0em" id="Thmdefinition7.p1.7.m7.1.1.1.2" xref="Thmdefinition7.p1.7.m7.1.1.1.2.cmml">​</mo><mrow id="Thmdefinition7.p1.7.m7.1.1.1.1.1" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.cmml"><mo stretchy="false" id="Thmdefinition7.p1.7.m7.1.1.1.1.1.2" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.cmml">(</mo><mrow id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.cmml"><msub id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.2" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.2.cmml"><mi id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.2.2" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.2.2.cmml">t</mi><mi id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.2.3" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.2.3.cmml">i</mi></msub><mo fence="false" id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.1" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.1.cmml">|</mo><mrow id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.cmml"><msup id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.2" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.2.cmml"><mi id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.2.2" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.2.2.cmml">𝐭</mi><mo id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.2.3" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.2.3.cmml">′</mo></msup><mo lspace="0em" rspace="0em" id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.1" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.1.cmml">​</mo><msub id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.3" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.3.cmml"><mi id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.3.2" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.3.2.cmml">t</mi><mn id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.3.3" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.3.3.cmml">0</mn></msub><mo lspace="0em" rspace="0em" id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.1a" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.1.cmml">​</mo><mi mathvariant="normal" id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.4" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.4.cmml">…</mi><mo lspace="0em" rspace="0em" id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.1b" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.1.cmml">​</mo><msub id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.5" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.5.cmml"><mi id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.5.2" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.5.2.cmml">t</mi><mrow id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.5.3" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.5.3.cmml"><mi id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.5.3.2" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.5.3.2.cmml">i</mi><mo id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.5.3.1" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.5.3.1.cmml">−</mo><mn id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.5.3.3" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.5.3.3.cmml">1</mn></mrow></msub></mrow></mrow><mo stretchy="false" id="Thmdefinition7.p1.7.m7.1.1.1.1.1.3" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="Thmdefinition7.p1.7.m7.1.1.2" xref="Thmdefinition7.p1.7.m7.1.1.2.cmml">≥</mo><mn id="Thmdefinition7.p1.7.m7.1.1.3" xref="Thmdefinition7.p1.7.m7.1.1.3.cmml">0.9</mn></mrow><annotation-xml encoding="MathML-Content" id="Thmdefinition7.p1.7.m7.1b"><apply id="Thmdefinition7.p1.7.m7.1.1.cmml" xref="Thmdefinition7.p1.7.m7.1.1"><geq id="Thmdefinition7.p1.7.m7.1.1.2.cmml" xref="Thmdefinition7.p1.7.m7.1.1.2"></geq><apply id="Thmdefinition7.p1.7.m7.1.1.1.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1"><times id="Thmdefinition7.p1.7.m7.1.1.1.2.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.2"></times><apply id="Thmdefinition7.p1.7.m7.1.1.1.3.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.3"><csymbol cd="ambiguous" id="Thmdefinition7.p1.7.m7.1.1.1.3.1.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.3">subscript</csymbol><ci id="Thmdefinition7.p1.7.m7.1.1.1.3.2.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.3.2">ℙ</ci><ci id="Thmdefinition7.p1.7.m7.1.1.1.3.3.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.3.3">𝑀</ci></apply><apply id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1"><csymbol cd="latexml" id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.1.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.1">conditional</csymbol><apply id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.2.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.2.1.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.2">subscript</csymbol><ci id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.2.2.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.2.2">𝑡</ci><ci id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.2.3.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.2.3">𝑖</ci></apply><apply id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3"><times id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.1.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.1"></times><apply id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.2.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.2.1.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.2">superscript</csymbol><ci id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.2.2.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.2.2">𝐭</ci><ci id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.2.3.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.2.3">′</ci></apply><apply id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.3.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.3.1.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.3">subscript</csymbol><ci id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.3.2.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.3.2">𝑡</ci><cn type="integer" id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.3.3.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.3.3">0</cn></apply><ci id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.4.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.4">…</ci><apply id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.5.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.5"><csymbol cd="ambiguous" id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.5.1.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.5">subscript</csymbol><ci id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.5.2.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.5.2">𝑡</ci><apply id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.5.3.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.5.3"><minus id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.5.3.1.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.5.3.1"></minus><ci id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.5.3.2.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.5.3.2">𝑖</ci><cn type="integer" id="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.5.3.3.cmml" xref="Thmdefinition7.p1.7.m7.1.1.1.1.1.1.3.5.3.3">1</cn></apply></apply></apply></apply></apply><cn type="float" id="Thmdefinition7.p1.7.m7.1.1.3.cmml" xref="Thmdefinition7.p1.7.m7.1.1.3">0.9</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition7.p1.7.m7.1c">\mathbb{P}_{M}(t_{i}|\mathbf{t}^{\prime}t_{0}\dots t_{i-1})\geq 0.9</annotation></semantics></math> for every <math id="Thmdefinition7.p1.8.m8.1" class="ltx_Math" alttext="t_{i}\in\mathbf{t}_{\text{out}}" display="inline"><semantics id="Thmdefinition7.p1.8.m8.1a"><mrow id="Thmdefinition7.p1.8.m8.1.1" xref="Thmdefinition7.p1.8.m8.1.1.cmml"><msub id="Thmdefinition7.p1.8.m8.1.1.2" xref="Thmdefinition7.p1.8.m8.1.1.2.cmml"><mi id="Thmdefinition7.p1.8.m8.1.1.2.2" xref="Thmdefinition7.p1.8.m8.1.1.2.2.cmml">t</mi><mi id="Thmdefinition7.p1.8.m8.1.1.2.3" xref="Thmdefinition7.p1.8.m8.1.1.2.3.cmml">i</mi></msub><mo id="Thmdefinition7.p1.8.m8.1.1.1" xref="Thmdefinition7.p1.8.m8.1.1.1.cmml">∈</mo><msub id="Thmdefinition7.p1.8.m8.1.1.3" xref="Thmdefinition7.p1.8.m8.1.1.3.cmml"><mi id="Thmdefinition7.p1.8.m8.1.1.3.2" xref="Thmdefinition7.p1.8.m8.1.1.3.2.cmml">𝐭</mi><mtext id="Thmdefinition7.p1.8.m8.1.1.3.3" xref="Thmdefinition7.p1.8.m8.1.1.3.3a.cmml">out</mtext></msub></mrow><annotation-xml encoding="MathML-Content" id="Thmdefinition7.p1.8.m8.1b"><apply id="Thmdefinition7.p1.8.m8.1.1.cmml" xref="Thmdefinition7.p1.8.m8.1.1"><in id="Thmdefinition7.p1.8.m8.1.1.1.cmml" xref="Thmdefinition7.p1.8.m8.1.1.1"></in><apply id="Thmdefinition7.p1.8.m8.1.1.2.cmml" xref="Thmdefinition7.p1.8.m8.1.1.2"><csymbol cd="ambiguous" id="Thmdefinition7.p1.8.m8.1.1.2.1.cmml" xref="Thmdefinition7.p1.8.m8.1.1.2">subscript</csymbol><ci id="Thmdefinition7.p1.8.m8.1.1.2.2.cmml" xref="Thmdefinition7.p1.8.m8.1.1.2.2">𝑡</ci><ci id="Thmdefinition7.p1.8.m8.1.1.2.3.cmml" xref="Thmdefinition7.p1.8.m8.1.1.2.3">𝑖</ci></apply><apply id="Thmdefinition7.p1.8.m8.1.1.3.cmml" xref="Thmdefinition7.p1.8.m8.1.1.3"><csymbol cd="ambiguous" id="Thmdefinition7.p1.8.m8.1.1.3.1.cmml" xref="Thmdefinition7.p1.8.m8.1.1.3">subscript</csymbol><ci id="Thmdefinition7.p1.8.m8.1.1.3.2.cmml" xref="Thmdefinition7.p1.8.m8.1.1.3.2">𝐭</ci><ci id="Thmdefinition7.p1.8.m8.1.1.3.3a.cmml" xref="Thmdefinition7.p1.8.m8.1.1.3.3"><mtext mathsize="70%" id="Thmdefinition7.p1.8.m8.1.1.3.3.cmml" xref="Thmdefinition7.p1.8.m8.1.1.3.3">out</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition7.p1.8.m8.1c">t_{i}\in\mathbf{t}_{\text{out}}</annotation></semantics></math>.</p>
</div>
</div>
<div id="S2.SS3.p2" class="ltx_para">
<p id="S2.SS3.p2.1" class="ltx_p">Basically, imitability means that LLMs can achieve the same effect as possessing this knowledge by retrieving example texts that demonstrate this knowledge.
Few-shot prompting can be seen as a special case of providing realizations.</p>
</div>
<div id="S2.SS3.p3" class="ltx_para">
<p id="S2.SS3.p3.2" class="ltx_p">Separability is a more general property than imitability.
For instance, one can set the prefix <math id="S2.SS3.p3.1.m1.1" class="ltx_Math" alttext="\mathbf{t}_{*}" display="inline"><semantics id="S2.SS3.p3.1.m1.1a"><msub id="S2.SS3.p3.1.m1.1.1" xref="S2.SS3.p3.1.m1.1.1.cmml"><mi id="S2.SS3.p3.1.m1.1.1.2" xref="S2.SS3.p3.1.m1.1.1.2.cmml">𝐭</mi><mo id="S2.SS3.p3.1.m1.1.1.3" xref="S2.SS3.p3.1.m1.1.1.3.cmml">∗</mo></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.1.m1.1b"><apply id="S2.SS3.p3.1.m1.1.1.cmml" xref="S2.SS3.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS3.p3.1.m1.1.1.1.cmml" xref="S2.SS3.p3.1.m1.1.1">subscript</csymbol><ci id="S2.SS3.p3.1.m1.1.1.2.cmml" xref="S2.SS3.p3.1.m1.1.1.2">𝐭</ci><times id="S2.SS3.p3.1.m1.1.1.3.cmml" xref="S2.SS3.p3.1.m1.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.1.m1.1c">\mathbf{t}_{*}</annotation></semantics></math> to be an abstract description of <math id="S2.SS3.p3.2.m2.1" class="ltx_Math" alttext="\mathcal{K}" display="inline"><semantics id="S2.SS3.p3.2.m2.1a"><mi class="ltx_font_mathcaligraphic" id="S2.SS3.p3.2.m2.1.1" xref="S2.SS3.p3.2.m2.1.1.cmml">𝒦</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.2.m2.1b"><ci id="S2.SS3.p3.2.m2.1.1.cmml" xref="S2.SS3.p3.2.m2.1.1">𝒦</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.2.m2.1c">\mathcal{K}</annotation></semantics></math> instead of its realization, and this is reminiscent of instruction prompting.
Nevertheless, it is not obvious whether the set of separable knowledges is strictly larger than the set of imitable knowledges.</p>
</div>
<div id="Thmclaim1" class="ltx_theorem ltx_theorem_claim">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmclaim1.1.1.1" class="ltx_text ltx_font_bold">Claim 1</span></span><span id="Thmclaim1.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmclaim1.p1" class="ltx_para">
<p id="Thmclaim1.p1.1" class="ltx_p">Every specific knowledge <math id="Thmclaim1.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{K}" display="inline"><semantics id="Thmclaim1.p1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="Thmclaim1.p1.1.m1.1.1" xref="Thmclaim1.p1.1.m1.1.1.cmml">𝒦</mi><annotation-xml encoding="MathML-Content" id="Thmclaim1.p1.1.m1.1b"><ci id="Thmclaim1.p1.1.m1.1.1.cmml" xref="Thmclaim1.p1.1.m1.1.1">𝒦</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmclaim1.p1.1.m1.1c">\mathcal{K}</annotation></semantics></math> is imitable and thus is separable.</p>
</div>
</div>
<div id="S2.SS3.3" class="ltx_proof">
<h6 class="ltx_title ltx_runin ltx_font_italic ltx_title_proof">Proof (informal).</h6>
<div id="S2.SS3.1.p1" class="ltx_para">
<p id="S2.SS3.1.p1.12" class="ltx_p">Without loss of generality, we can assume that for any realization <math id="S2.SS3.1.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{t}" display="inline"><semantics id="S2.SS3.1.p1.1.m1.1a"><mi id="S2.SS3.1.p1.1.m1.1.1" xref="S2.SS3.1.p1.1.m1.1.1.cmml">𝐭</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.1.p1.1.m1.1b"><ci id="S2.SS3.1.p1.1.m1.1.1.cmml" xref="S2.SS3.1.p1.1.m1.1.1">𝐭</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.1.p1.1.m1.1c">\mathbf{t}</annotation></semantics></math> of <math id="S2.SS3.1.p1.2.m2.1" class="ltx_Math" alttext="\mathcal{K}" display="inline"><semantics id="S2.SS3.1.p1.2.m2.1a"><mi class="ltx_font_mathcaligraphic" id="S2.SS3.1.p1.2.m2.1.1" xref="S2.SS3.1.p1.2.m2.1.1.cmml">𝒦</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.1.p1.2.m2.1b"><ci id="S2.SS3.1.p1.2.m2.1.1.cmml" xref="S2.SS3.1.p1.2.m2.1.1">𝒦</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.1.p1.2.m2.1c">\mathcal{K}</annotation></semantics></math>, all tokens of the associated input <math id="S2.SS3.1.p1.3.m3.1" class="ltx_Math" alttext="\mathbf{t}_{\text{in}}" display="inline"><semantics id="S2.SS3.1.p1.3.m3.1a"><msub id="S2.SS3.1.p1.3.m3.1.1" xref="S2.SS3.1.p1.3.m3.1.1.cmml"><mi id="S2.SS3.1.p1.3.m3.1.1.2" xref="S2.SS3.1.p1.3.m3.1.1.2.cmml">𝐭</mi><mtext id="S2.SS3.1.p1.3.m3.1.1.3" xref="S2.SS3.1.p1.3.m3.1.1.3a.cmml">in</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.1.p1.3.m3.1b"><apply id="S2.SS3.1.p1.3.m3.1.1.cmml" xref="S2.SS3.1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS3.1.p1.3.m3.1.1.1.cmml" xref="S2.SS3.1.p1.3.m3.1.1">subscript</csymbol><ci id="S2.SS3.1.p1.3.m3.1.1.2.cmml" xref="S2.SS3.1.p1.3.m3.1.1.2">𝐭</ci><ci id="S2.SS3.1.p1.3.m3.1.1.3a.cmml" xref="S2.SS3.1.p1.3.m3.1.1.3"><mtext mathsize="70%" id="S2.SS3.1.p1.3.m3.1.1.3.cmml" xref="S2.SS3.1.p1.3.m3.1.1.3">in</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.1.p1.3.m3.1c">\mathbf{t}_{\text{in}}</annotation></semantics></math> precede all tokens of the associated output <math id="S2.SS3.1.p1.4.m4.1" class="ltx_Math" alttext="\mathbf{t}_{\text{out}}" display="inline"><semantics id="S2.SS3.1.p1.4.m4.1a"><msub id="S2.SS3.1.p1.4.m4.1.1" xref="S2.SS3.1.p1.4.m4.1.1.cmml"><mi id="S2.SS3.1.p1.4.m4.1.1.2" xref="S2.SS3.1.p1.4.m4.1.1.2.cmml">𝐭</mi><mtext id="S2.SS3.1.p1.4.m4.1.1.3" xref="S2.SS3.1.p1.4.m4.1.1.3a.cmml">out</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.1.p1.4.m4.1b"><apply id="S2.SS3.1.p1.4.m4.1.1.cmml" xref="S2.SS3.1.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S2.SS3.1.p1.4.m4.1.1.1.cmml" xref="S2.SS3.1.p1.4.m4.1.1">subscript</csymbol><ci id="S2.SS3.1.p1.4.m4.1.1.2.cmml" xref="S2.SS3.1.p1.4.m4.1.1.2">𝐭</ci><ci id="S2.SS3.1.p1.4.m4.1.1.3a.cmml" xref="S2.SS3.1.p1.4.m4.1.1.3"><mtext mathsize="70%" id="S2.SS3.1.p1.4.m4.1.1.3.cmml" xref="S2.SS3.1.p1.4.m4.1.1.3">out</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.1.p1.4.m4.1c">\mathbf{t}_{\text{out}}</annotation></semantics></math>.
Otherwise, we can split <math id="S2.SS3.1.p1.5.m5.1" class="ltx_Math" alttext="\mathbf{t}_{\text{in}}" display="inline"><semantics id="S2.SS3.1.p1.5.m5.1a"><msub id="S2.SS3.1.p1.5.m5.1.1" xref="S2.SS3.1.p1.5.m5.1.1.cmml"><mi id="S2.SS3.1.p1.5.m5.1.1.2" xref="S2.SS3.1.p1.5.m5.1.1.2.cmml">𝐭</mi><mtext id="S2.SS3.1.p1.5.m5.1.1.3" xref="S2.SS3.1.p1.5.m5.1.1.3a.cmml">in</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.1.p1.5.m5.1b"><apply id="S2.SS3.1.p1.5.m5.1.1.cmml" xref="S2.SS3.1.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S2.SS3.1.p1.5.m5.1.1.1.cmml" xref="S2.SS3.1.p1.5.m5.1.1">subscript</csymbol><ci id="S2.SS3.1.p1.5.m5.1.1.2.cmml" xref="S2.SS3.1.p1.5.m5.1.1.2">𝐭</ci><ci id="S2.SS3.1.p1.5.m5.1.1.3a.cmml" xref="S2.SS3.1.p1.5.m5.1.1.3"><mtext mathsize="70%" id="S2.SS3.1.p1.5.m5.1.1.3.cmml" xref="S2.SS3.1.p1.5.m5.1.1.3">in</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.1.p1.5.m5.1c">\mathbf{t}_{\text{in}}</annotation></semantics></math> into two halves <math id="S2.SS3.1.p1.6.m6.2" class="ltx_Math" alttext="\mathbf{t}_{1},\mathbf{t}_{2}" display="inline"><semantics id="S2.SS3.1.p1.6.m6.2a"><mrow id="S2.SS3.1.p1.6.m6.2.2.2" xref="S2.SS3.1.p1.6.m6.2.2.3.cmml"><msub id="S2.SS3.1.p1.6.m6.1.1.1.1" xref="S2.SS3.1.p1.6.m6.1.1.1.1.cmml"><mi id="S2.SS3.1.p1.6.m6.1.1.1.1.2" xref="S2.SS3.1.p1.6.m6.1.1.1.1.2.cmml">𝐭</mi><mn id="S2.SS3.1.p1.6.m6.1.1.1.1.3" xref="S2.SS3.1.p1.6.m6.1.1.1.1.3.cmml">1</mn></msub><mo id="S2.SS3.1.p1.6.m6.2.2.2.3" xref="S2.SS3.1.p1.6.m6.2.2.3.cmml">,</mo><msub id="S2.SS3.1.p1.6.m6.2.2.2.2" xref="S2.SS3.1.p1.6.m6.2.2.2.2.cmml"><mi id="S2.SS3.1.p1.6.m6.2.2.2.2.2" xref="S2.SS3.1.p1.6.m6.2.2.2.2.2.cmml">𝐭</mi><mn id="S2.SS3.1.p1.6.m6.2.2.2.2.3" xref="S2.SS3.1.p1.6.m6.2.2.2.2.3.cmml">2</mn></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.1.p1.6.m6.2b"><list id="S2.SS3.1.p1.6.m6.2.2.3.cmml" xref="S2.SS3.1.p1.6.m6.2.2.2"><apply id="S2.SS3.1.p1.6.m6.1.1.1.1.cmml" xref="S2.SS3.1.p1.6.m6.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS3.1.p1.6.m6.1.1.1.1.1.cmml" xref="S2.SS3.1.p1.6.m6.1.1.1.1">subscript</csymbol><ci id="S2.SS3.1.p1.6.m6.1.1.1.1.2.cmml" xref="S2.SS3.1.p1.6.m6.1.1.1.1.2">𝐭</ci><cn type="integer" id="S2.SS3.1.p1.6.m6.1.1.1.1.3.cmml" xref="S2.SS3.1.p1.6.m6.1.1.1.1.3">1</cn></apply><apply id="S2.SS3.1.p1.6.m6.2.2.2.2.cmml" xref="S2.SS3.1.p1.6.m6.2.2.2.2"><csymbol cd="ambiguous" id="S2.SS3.1.p1.6.m6.2.2.2.2.1.cmml" xref="S2.SS3.1.p1.6.m6.2.2.2.2">subscript</csymbol><ci id="S2.SS3.1.p1.6.m6.2.2.2.2.2.cmml" xref="S2.SS3.1.p1.6.m6.2.2.2.2.2">𝐭</ci><cn type="integer" id="S2.SS3.1.p1.6.m6.2.2.2.2.3.cmml" xref="S2.SS3.1.p1.6.m6.2.2.2.2.3">2</cn></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.1.p1.6.m6.2c">\mathbf{t}_{1},\mathbf{t}_{2}</annotation></semantics></math> that precedes/does not precede <math id="S2.SS3.1.p1.7.m7.1" class="ltx_Math" alttext="\mathbf{t}_{\text{out}}" display="inline"><semantics id="S2.SS3.1.p1.7.m7.1a"><msub id="S2.SS3.1.p1.7.m7.1.1" xref="S2.SS3.1.p1.7.m7.1.1.cmml"><mi id="S2.SS3.1.p1.7.m7.1.1.2" xref="S2.SS3.1.p1.7.m7.1.1.2.cmml">𝐭</mi><mtext id="S2.SS3.1.p1.7.m7.1.1.3" xref="S2.SS3.1.p1.7.m7.1.1.3a.cmml">out</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.1.p1.7.m7.1b"><apply id="S2.SS3.1.p1.7.m7.1.1.cmml" xref="S2.SS3.1.p1.7.m7.1.1"><csymbol cd="ambiguous" id="S2.SS3.1.p1.7.m7.1.1.1.cmml" xref="S2.SS3.1.p1.7.m7.1.1">subscript</csymbol><ci id="S2.SS3.1.p1.7.m7.1.1.2.cmml" xref="S2.SS3.1.p1.7.m7.1.1.2">𝐭</ci><ci id="S2.SS3.1.p1.7.m7.1.1.3a.cmml" xref="S2.SS3.1.p1.7.m7.1.1.3"><mtext mathsize="70%" id="S2.SS3.1.p1.7.m7.1.1.3.cmml" xref="S2.SS3.1.p1.7.m7.1.1.3">out</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.1.p1.7.m7.1c">\mathbf{t}_{\text{out}}</annotation></semantics></math>, and split the corresponding subgraph <math id="S2.SS3.1.p1.8.m8.1" class="ltx_Math" alttext="S\in\mathcal{K}" display="inline"><semantics id="S2.SS3.1.p1.8.m8.1a"><mrow id="S2.SS3.1.p1.8.m8.1.1" xref="S2.SS3.1.p1.8.m8.1.1.cmml"><mi id="S2.SS3.1.p1.8.m8.1.1.2" xref="S2.SS3.1.p1.8.m8.1.1.2.cmml">S</mi><mo id="S2.SS3.1.p1.8.m8.1.1.1" xref="S2.SS3.1.p1.8.m8.1.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S2.SS3.1.p1.8.m8.1.1.3" xref="S2.SS3.1.p1.8.m8.1.1.3.cmml">𝒦</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.1.p1.8.m8.1b"><apply id="S2.SS3.1.p1.8.m8.1.1.cmml" xref="S2.SS3.1.p1.8.m8.1.1"><in id="S2.SS3.1.p1.8.m8.1.1.1.cmml" xref="S2.SS3.1.p1.8.m8.1.1.1"></in><ci id="S2.SS3.1.p1.8.m8.1.1.2.cmml" xref="S2.SS3.1.p1.8.m8.1.1.2">𝑆</ci><ci id="S2.SS3.1.p1.8.m8.1.1.3.cmml" xref="S2.SS3.1.p1.8.m8.1.1.3">𝒦</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.1.p1.8.m8.1c">S\in\mathcal{K}</annotation></semantics></math> into two halves <math id="S2.SS3.1.p1.9.m9.2" class="ltx_Math" alttext="S_{1},S_{2}" display="inline"><semantics id="S2.SS3.1.p1.9.m9.2a"><mrow id="S2.SS3.1.p1.9.m9.2.2.2" xref="S2.SS3.1.p1.9.m9.2.2.3.cmml"><msub id="S2.SS3.1.p1.9.m9.1.1.1.1" xref="S2.SS3.1.p1.9.m9.1.1.1.1.cmml"><mi id="S2.SS3.1.p1.9.m9.1.1.1.1.2" xref="S2.SS3.1.p1.9.m9.1.1.1.1.2.cmml">S</mi><mn id="S2.SS3.1.p1.9.m9.1.1.1.1.3" xref="S2.SS3.1.p1.9.m9.1.1.1.1.3.cmml">1</mn></msub><mo id="S2.SS3.1.p1.9.m9.2.2.2.3" xref="S2.SS3.1.p1.9.m9.2.2.3.cmml">,</mo><msub id="S2.SS3.1.p1.9.m9.2.2.2.2" xref="S2.SS3.1.p1.9.m9.2.2.2.2.cmml"><mi id="S2.SS3.1.p1.9.m9.2.2.2.2.2" xref="S2.SS3.1.p1.9.m9.2.2.2.2.2.cmml">S</mi><mn id="S2.SS3.1.p1.9.m9.2.2.2.2.3" xref="S2.SS3.1.p1.9.m9.2.2.2.2.3.cmml">2</mn></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.1.p1.9.m9.2b"><list id="S2.SS3.1.p1.9.m9.2.2.3.cmml" xref="S2.SS3.1.p1.9.m9.2.2.2"><apply id="S2.SS3.1.p1.9.m9.1.1.1.1.cmml" xref="S2.SS3.1.p1.9.m9.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS3.1.p1.9.m9.1.1.1.1.1.cmml" xref="S2.SS3.1.p1.9.m9.1.1.1.1">subscript</csymbol><ci id="S2.SS3.1.p1.9.m9.1.1.1.1.2.cmml" xref="S2.SS3.1.p1.9.m9.1.1.1.1.2">𝑆</ci><cn type="integer" id="S2.SS3.1.p1.9.m9.1.1.1.1.3.cmml" xref="S2.SS3.1.p1.9.m9.1.1.1.1.3">1</cn></apply><apply id="S2.SS3.1.p1.9.m9.2.2.2.2.cmml" xref="S2.SS3.1.p1.9.m9.2.2.2.2"><csymbol cd="ambiguous" id="S2.SS3.1.p1.9.m9.2.2.2.2.1.cmml" xref="S2.SS3.1.p1.9.m9.2.2.2.2">subscript</csymbol><ci id="S2.SS3.1.p1.9.m9.2.2.2.2.2.cmml" xref="S2.SS3.1.p1.9.m9.2.2.2.2.2">𝑆</ci><cn type="integer" id="S2.SS3.1.p1.9.m9.2.2.2.2.3.cmml" xref="S2.SS3.1.p1.9.m9.2.2.2.2.3">2</cn></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.1.p1.9.m9.2c">S_{1},S_{2}</annotation></semantics></math> that have high weights with respect to <math id="S2.SS3.1.p1.10.m10.2" class="ltx_Math" alttext="\mathbf{t}_{1},\mathbf{t}_{2}" display="inline"><semantics id="S2.SS3.1.p1.10.m10.2a"><mrow id="S2.SS3.1.p1.10.m10.2.2.2" xref="S2.SS3.1.p1.10.m10.2.2.3.cmml"><msub id="S2.SS3.1.p1.10.m10.1.1.1.1" xref="S2.SS3.1.p1.10.m10.1.1.1.1.cmml"><mi id="S2.SS3.1.p1.10.m10.1.1.1.1.2" xref="S2.SS3.1.p1.10.m10.1.1.1.1.2.cmml">𝐭</mi><mn id="S2.SS3.1.p1.10.m10.1.1.1.1.3" xref="S2.SS3.1.p1.10.m10.1.1.1.1.3.cmml">1</mn></msub><mo id="S2.SS3.1.p1.10.m10.2.2.2.3" xref="S2.SS3.1.p1.10.m10.2.2.3.cmml">,</mo><msub id="S2.SS3.1.p1.10.m10.2.2.2.2" xref="S2.SS3.1.p1.10.m10.2.2.2.2.cmml"><mi id="S2.SS3.1.p1.10.m10.2.2.2.2.2" xref="S2.SS3.1.p1.10.m10.2.2.2.2.2.cmml">𝐭</mi><mn id="S2.SS3.1.p1.10.m10.2.2.2.2.3" xref="S2.SS3.1.p1.10.m10.2.2.2.2.3.cmml">2</mn></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.1.p1.10.m10.2b"><list id="S2.SS3.1.p1.10.m10.2.2.3.cmml" xref="S2.SS3.1.p1.10.m10.2.2.2"><apply id="S2.SS3.1.p1.10.m10.1.1.1.1.cmml" xref="S2.SS3.1.p1.10.m10.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS3.1.p1.10.m10.1.1.1.1.1.cmml" xref="S2.SS3.1.p1.10.m10.1.1.1.1">subscript</csymbol><ci id="S2.SS3.1.p1.10.m10.1.1.1.1.2.cmml" xref="S2.SS3.1.p1.10.m10.1.1.1.1.2">𝐭</ci><cn type="integer" id="S2.SS3.1.p1.10.m10.1.1.1.1.3.cmml" xref="S2.SS3.1.p1.10.m10.1.1.1.1.3">1</cn></apply><apply id="S2.SS3.1.p1.10.m10.2.2.2.2.cmml" xref="S2.SS3.1.p1.10.m10.2.2.2.2"><csymbol cd="ambiguous" id="S2.SS3.1.p1.10.m10.2.2.2.2.1.cmml" xref="S2.SS3.1.p1.10.m10.2.2.2.2">subscript</csymbol><ci id="S2.SS3.1.p1.10.m10.2.2.2.2.2.cmml" xref="S2.SS3.1.p1.10.m10.2.2.2.2.2">𝐭</ci><cn type="integer" id="S2.SS3.1.p1.10.m10.2.2.2.2.3.cmml" xref="S2.SS3.1.p1.10.m10.2.2.2.2.3">2</cn></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.1.p1.10.m10.2c">\mathbf{t}_{1},\mathbf{t}_{2}</annotation></semantics></math>.
Using monotonicity arguments once Definition <a href="#Thmdefinition3" title="Definition 3. ‣ 2.2 | Knowledge ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> is fully formalized, one can try to show that this splitting is invariant across <math id="S2.SS3.1.p1.11.m11.1" class="ltx_Math" alttext="S\in\mathcal{K}" display="inline"><semantics id="S2.SS3.1.p1.11.m11.1a"><mrow id="S2.SS3.1.p1.11.m11.1.1" xref="S2.SS3.1.p1.11.m11.1.1.cmml"><mi id="S2.SS3.1.p1.11.m11.1.1.2" xref="S2.SS3.1.p1.11.m11.1.1.2.cmml">S</mi><mo id="S2.SS3.1.p1.11.m11.1.1.1" xref="S2.SS3.1.p1.11.m11.1.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S2.SS3.1.p1.11.m11.1.1.3" xref="S2.SS3.1.p1.11.m11.1.1.3.cmml">𝒦</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.1.p1.11.m11.1b"><apply id="S2.SS3.1.p1.11.m11.1.1.cmml" xref="S2.SS3.1.p1.11.m11.1.1"><in id="S2.SS3.1.p1.11.m11.1.1.1.cmml" xref="S2.SS3.1.p1.11.m11.1.1.1"></in><ci id="S2.SS3.1.p1.11.m11.1.1.2.cmml" xref="S2.SS3.1.p1.11.m11.1.1.2">𝑆</ci><ci id="S2.SS3.1.p1.11.m11.1.1.3.cmml" xref="S2.SS3.1.p1.11.m11.1.1.3">𝒦</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.1.p1.11.m11.1c">S\in\mathcal{K}</annotation></semantics></math> and therefore the sets of <math id="S2.SS3.1.p1.12.m12.2" class="ltx_Math" alttext="S_{1},S_{2}" display="inline"><semantics id="S2.SS3.1.p1.12.m12.2a"><mrow id="S2.SS3.1.p1.12.m12.2.2.2" xref="S2.SS3.1.p1.12.m12.2.2.3.cmml"><msub id="S2.SS3.1.p1.12.m12.1.1.1.1" xref="S2.SS3.1.p1.12.m12.1.1.1.1.cmml"><mi id="S2.SS3.1.p1.12.m12.1.1.1.1.2" xref="S2.SS3.1.p1.12.m12.1.1.1.1.2.cmml">S</mi><mn id="S2.SS3.1.p1.12.m12.1.1.1.1.3" xref="S2.SS3.1.p1.12.m12.1.1.1.1.3.cmml">1</mn></msub><mo id="S2.SS3.1.p1.12.m12.2.2.2.3" xref="S2.SS3.1.p1.12.m12.2.2.3.cmml">,</mo><msub id="S2.SS3.1.p1.12.m12.2.2.2.2" xref="S2.SS3.1.p1.12.m12.2.2.2.2.cmml"><mi id="S2.SS3.1.p1.12.m12.2.2.2.2.2" xref="S2.SS3.1.p1.12.m12.2.2.2.2.2.cmml">S</mi><mn id="S2.SS3.1.p1.12.m12.2.2.2.2.3" xref="S2.SS3.1.p1.12.m12.2.2.2.2.3.cmml">2</mn></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.1.p1.12.m12.2b"><list id="S2.SS3.1.p1.12.m12.2.2.3.cmml" xref="S2.SS3.1.p1.12.m12.2.2.2"><apply id="S2.SS3.1.p1.12.m12.1.1.1.1.cmml" xref="S2.SS3.1.p1.12.m12.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS3.1.p1.12.m12.1.1.1.1.1.cmml" xref="S2.SS3.1.p1.12.m12.1.1.1.1">subscript</csymbol><ci id="S2.SS3.1.p1.12.m12.1.1.1.1.2.cmml" xref="S2.SS3.1.p1.12.m12.1.1.1.1.2">𝑆</ci><cn type="integer" id="S2.SS3.1.p1.12.m12.1.1.1.1.3.cmml" xref="S2.SS3.1.p1.12.m12.1.1.1.1.3">1</cn></apply><apply id="S2.SS3.1.p1.12.m12.2.2.2.2.cmml" xref="S2.SS3.1.p1.12.m12.2.2.2.2"><csymbol cd="ambiguous" id="S2.SS3.1.p1.12.m12.2.2.2.2.1.cmml" xref="S2.SS3.1.p1.12.m12.2.2.2.2">subscript</csymbol><ci id="S2.SS3.1.p1.12.m12.2.2.2.2.2.cmml" xref="S2.SS3.1.p1.12.m12.2.2.2.2.2">𝑆</ci><cn type="integer" id="S2.SS3.1.p1.12.m12.2.2.2.2.3.cmml" xref="S2.SS3.1.p1.12.m12.2.2.2.2.3">2</cn></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.1.p1.12.m12.2c">S_{1},S_{2}</annotation></semantics></math> are two specific knowledges.</p>
</div>
<div id="S2.SS3.2.p2" class="ltx_para">
<p id="S2.SS3.2.p2.2" class="ltx_p">Consider sequences of the form [a][b]…[a’][b’], where [a], [a’] (or [b], [b’]) could be the associated inputs (or outputs) of any subgraphs <math id="S2.SS3.2.p2.1.m1.2" class="ltx_Math" alttext="S,S^{\prime}\in\mathcal{K}" display="inline"><semantics id="S2.SS3.2.p2.1.m1.2a"><mrow id="S2.SS3.2.p2.1.m1.2.2" xref="S2.SS3.2.p2.1.m1.2.2.cmml"><mrow id="S2.SS3.2.p2.1.m1.2.2.1.1" xref="S2.SS3.2.p2.1.m1.2.2.1.2.cmml"><mi id="S2.SS3.2.p2.1.m1.1.1" xref="S2.SS3.2.p2.1.m1.1.1.cmml">S</mi><mo id="S2.SS3.2.p2.1.m1.2.2.1.1.2" xref="S2.SS3.2.p2.1.m1.2.2.1.2.cmml">,</mo><msup id="S2.SS3.2.p2.1.m1.2.2.1.1.1" xref="S2.SS3.2.p2.1.m1.2.2.1.1.1.cmml"><mi id="S2.SS3.2.p2.1.m1.2.2.1.1.1.2" xref="S2.SS3.2.p2.1.m1.2.2.1.1.1.2.cmml">S</mi><mo id="S2.SS3.2.p2.1.m1.2.2.1.1.1.3" xref="S2.SS3.2.p2.1.m1.2.2.1.1.1.3.cmml">′</mo></msup></mrow><mo id="S2.SS3.2.p2.1.m1.2.2.2" xref="S2.SS3.2.p2.1.m1.2.2.2.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S2.SS3.2.p2.1.m1.2.2.3" xref="S2.SS3.2.p2.1.m1.2.2.3.cmml">𝒦</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.2.p2.1.m1.2b"><apply id="S2.SS3.2.p2.1.m1.2.2.cmml" xref="S2.SS3.2.p2.1.m1.2.2"><in id="S2.SS3.2.p2.1.m1.2.2.2.cmml" xref="S2.SS3.2.p2.1.m1.2.2.2"></in><list id="S2.SS3.2.p2.1.m1.2.2.1.2.cmml" xref="S2.SS3.2.p2.1.m1.2.2.1.1"><ci id="S2.SS3.2.p2.1.m1.1.1.cmml" xref="S2.SS3.2.p2.1.m1.1.1">𝑆</ci><apply id="S2.SS3.2.p2.1.m1.2.2.1.1.1.cmml" xref="S2.SS3.2.p2.1.m1.2.2.1.1.1"><csymbol cd="ambiguous" id="S2.SS3.2.p2.1.m1.2.2.1.1.1.1.cmml" xref="S2.SS3.2.p2.1.m1.2.2.1.1.1">superscript</csymbol><ci id="S2.SS3.2.p2.1.m1.2.2.1.1.1.2.cmml" xref="S2.SS3.2.p2.1.m1.2.2.1.1.1.2">𝑆</ci><ci id="S2.SS3.2.p2.1.m1.2.2.1.1.1.3.cmml" xref="S2.SS3.2.p2.1.m1.2.2.1.1.1.3">′</ci></apply></list><ci id="S2.SS3.2.p2.1.m1.2.2.3.cmml" xref="S2.SS3.2.p2.1.m1.2.2.3">𝒦</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.2.p2.1.m1.2c">S,S^{\prime}\in\mathcal{K}</annotation></semantics></math>.
By Definition <a href="#Thmdefinition4" title="Definition 4. ‣ 2.2 | Knowledge ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, [a] and [a’] always share some interpretable meaning, while [b] and [b’] are approximately the same sequence.
One can construct an abstract knowledge that completes [a][b]…[a’] with [b’]:
the first part of this circuit detects the common feature of the [a]’s (possibly overlapping with the subgraphs of <math id="S2.SS3.2.p2.2.m2.1" class="ltx_Math" alttext="\mathcal{K}" display="inline"><semantics id="S2.SS3.2.p2.2.m2.1a"><mi class="ltx_font_mathcaligraphic" id="S2.SS3.2.p2.2.m2.1.1" xref="S2.SS3.2.p2.2.m2.1.1.cmml">𝒦</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.2.p2.2.m2.1b"><ci id="S2.SS3.2.p2.2.m2.1.1.cmml" xref="S2.SS3.2.p2.2.m2.1.1">𝒦</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.2.p2.2.m2.1c">\mathcal{K}</annotation></semantics></math>), the second part is an induction head (analogous to Example <a href="#Thmexample4" title="Example 4. ‣ 2.2 | Knowledge ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, it provides [b] with the common feature of the [a]’s and lets [a’] to attend to [b]), and the third part generates [b’] based on [b] with possible slight modifications.
This circuit is an abstract knowledge since it can be applied to other specific knowledges as long as their associated inputs share the same meaning with the [a]’s, no matter how their associated outputs could vary.</p>
</div>
<div id="S2.SS3.3.p3" class="ltx_para">
<p id="S2.SS3.3.p3.4" class="ltx_p">Meanwhile, construct the model <math id="S2.SS3.3.p3.1.m1.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S2.SS3.3.p3.1.m1.1a"><mi id="S2.SS3.3.p3.1.m1.1.1" xref="S2.SS3.3.p3.1.m1.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.3.p3.1.m1.1b"><ci id="S2.SS3.3.p3.1.m1.1.1.cmml" xref="S2.SS3.3.p3.1.m1.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.3.p3.1.m1.1c">M</annotation></semantics></math> by letting the reference model forget <math id="S2.SS3.3.p3.2.m2.1" class="ltx_Math" alttext="\mathcal{K}" display="inline"><semantics id="S2.SS3.3.p3.2.m2.1a"><mi class="ltx_font_mathcaligraphic" id="S2.SS3.3.p3.2.m2.1.1" xref="S2.SS3.3.p3.2.m2.1.1.cmml">𝒦</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.3.p3.2.m2.1b"><ci id="S2.SS3.3.p3.2.m2.1.1.cmml" xref="S2.SS3.3.p3.2.m2.1.1">𝒦</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.3.p3.2.m2.1c">\mathcal{K}</annotation></semantics></math> (e.g. by finetuning on a modified data distribution such that the associated input of <math id="S2.SS3.3.p3.3.m3.1" class="ltx_Math" alttext="\mathcal{K}" display="inline"><semantics id="S2.SS3.3.p3.3.m3.1a"><mi class="ltx_font_mathcaligraphic" id="S2.SS3.3.p3.3.m3.1.1" xref="S2.SS3.3.p3.3.m3.1.1.cmml">𝒦</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.3.p3.3.m3.1b"><ci id="S2.SS3.3.p3.3.m3.1.1.cmml" xref="S2.SS3.3.p3.3.m3.1.1">𝒦</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.3.p3.3.m3.1c">\mathcal{K}</annotation></semantics></math> is never followed by the associated output, while the rest of the distribution remains the same).
Combining this circuit with <math id="S2.SS3.3.p3.4.m4.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S2.SS3.3.p3.4.m4.1a"><mi id="S2.SS3.3.p3.4.m4.1.1" xref="S2.SS3.3.p3.4.m4.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.3.p3.4.m4.1b"><ci id="S2.SS3.3.p3.4.m4.1.1.cmml" xref="S2.SS3.3.p3.4.m4.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.3.p3.4.m4.1c">M</annotation></semantics></math> completes the proof.
∎</p>
</div>
</div>
<div id="S2.SS3.p4" class="ltx_para">
<p id="S2.SS3.p4.1" class="ltx_p">Claim <a href="#Thmclaim1" title="Claim 1. ‣ 2.3 | Memory ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> indicates that a lot of knowledges can be externalized from the model parameters.
The converse of Claim <a href="#Thmclaim1" title="Claim 1. ‣ 2.3 | Memory ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> may not hold, since it is imaginable that some abstract knowledges can also be substituted with their realizations.</p>
</div>
<div id="Thmremark1" class="ltx_theorem ltx_theorem_remark">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmremark1.1.1.1" class="ltx_text ltx_font_bold">Remark 1</span></span><span id="Thmremark1.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmremark1.p1" class="ltx_para">
<p id="Thmremark1.p1.1" class="ltx_p">There are three details in the proof of Claim <a href="#Thmclaim1" title="Claim 1. ‣ 2.3 | Memory ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> that will be useful later</p>
<ol id="S2.I7" class="ltx_enumerate">
<li id="S2.I7.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><span id="S2.I7.i1.1.1.1" class="ltx_text ltx_font_bold" style="color:#00868B;">0.</span></span> 
<div id="S2.I7.i1.p1" class="ltx_para">
<p id="S2.I7.i1.p1.4" class="ltx_p">The circuit we construct has only one attention head that attends to the reference text <math id="S2.I7.i1.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{t}^{\prime}" display="inline"><semantics id="S2.I7.i1.p1.1.m1.1a"><msup id="S2.I7.i1.p1.1.m1.1.1" xref="S2.I7.i1.p1.1.m1.1.1.cmml"><mi id="S2.I7.i1.p1.1.m1.1.1.2" xref="S2.I7.i1.p1.1.m1.1.1.2.cmml">𝐭</mi><mo id="S2.I7.i1.p1.1.m1.1.1.3" xref="S2.I7.i1.p1.1.m1.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S2.I7.i1.p1.1.m1.1b"><apply id="S2.I7.i1.p1.1.m1.1.1.cmml" xref="S2.I7.i1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.I7.i1.p1.1.m1.1.1.1.cmml" xref="S2.I7.i1.p1.1.m1.1.1">superscript</csymbol><ci id="S2.I7.i1.p1.1.m1.1.1.2.cmml" xref="S2.I7.i1.p1.1.m1.1.1.2">𝐭</ci><ci id="S2.I7.i1.p1.1.m1.1.1.3.cmml" xref="S2.I7.i1.p1.1.m1.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I7.i1.p1.1.m1.1c">\mathbf{t}^{\prime}</annotation></semantics></math> from the present text <math id="S2.I7.i1.p1.2.m2.1" class="ltx_Math" alttext="\mathbf{t}" display="inline"><semantics id="S2.I7.i1.p1.2.m2.1a"><mi id="S2.I7.i1.p1.2.m2.1.1" xref="S2.I7.i1.p1.2.m2.1.1.cmml">𝐭</mi><annotation-xml encoding="MathML-Content" id="S2.I7.i1.p1.2.m2.1b"><ci id="S2.I7.i1.p1.2.m2.1.1.cmml" xref="S2.I7.i1.p1.2.m2.1.1">𝐭</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I7.i1.p1.2.m2.1c">\mathbf{t}</annotation></semantics></math>, while all other computations are confined within either <math id="S2.I7.i1.p1.3.m3.1" class="ltx_Math" alttext="\mathbf{t}" display="inline"><semantics id="S2.I7.i1.p1.3.m3.1a"><mi id="S2.I7.i1.p1.3.m3.1.1" xref="S2.I7.i1.p1.3.m3.1.1.cmml">𝐭</mi><annotation-xml encoding="MathML-Content" id="S2.I7.i1.p1.3.m3.1b"><ci id="S2.I7.i1.p1.3.m3.1.1.cmml" xref="S2.I7.i1.p1.3.m3.1.1">𝐭</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I7.i1.p1.3.m3.1c">\mathbf{t}</annotation></semantics></math> or <math id="S2.I7.i1.p1.4.m4.1" class="ltx_Math" alttext="\mathbf{t^{\prime}}" display="inline"><semantics id="S2.I7.i1.p1.4.m4.1a"><msup id="S2.I7.i1.p1.4.m4.1.1" xref="S2.I7.i1.p1.4.m4.1.1.cmml"><mi id="S2.I7.i1.p1.4.m4.1.1.2" xref="S2.I7.i1.p1.4.m4.1.1.2.cmml">𝐭</mi><mo id="S2.I7.i1.p1.4.m4.1.1.3" xref="S2.I7.i1.p1.4.m4.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S2.I7.i1.p1.4.m4.1b"><apply id="S2.I7.i1.p1.4.m4.1.1.cmml" xref="S2.I7.i1.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S2.I7.i1.p1.4.m4.1.1.1.cmml" xref="S2.I7.i1.p1.4.m4.1.1">superscript</csymbol><ci id="S2.I7.i1.p1.4.m4.1.1.2.cmml" xref="S2.I7.i1.p1.4.m4.1.1.2">𝐭</ci><ci id="S2.I7.i1.p1.4.m4.1.1.3.cmml" xref="S2.I7.i1.p1.4.m4.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I7.i1.p1.4.m4.1c">\mathbf{t^{\prime}}</annotation></semantics></math>.</p>
</div>
</li>
<li id="S2.I7.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><span id="S2.I7.i2.1.1.1" class="ltx_text ltx_font_bold" style="color:#00868B;">0.</span></span> 
<div id="S2.I7.i2.p1" class="ltx_para">
<p id="S2.I7.i2.p1.1" class="ltx_p">Moreover, in this attention head, the circuit only needs the edges from [b] to [a’].
Thus, in general this head only needs to attend to very few tokens in the reference.</p>
</div>
</li>
<li id="S2.I7.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><span id="S2.I7.i3.1.1.1" class="ltx_text ltx_font_bold" style="color:#00868B;">0.</span></span> 
<div id="S2.I7.i3.p1" class="ltx_para">
<p id="S2.I7.i3.p1.1" class="ltx_p">It suffices for the reference <math id="S2.I7.i3.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{t}^{\prime}" display="inline"><semantics id="S2.I7.i3.p1.1.m1.1a"><msup id="S2.I7.i3.p1.1.m1.1.1" xref="S2.I7.i3.p1.1.m1.1.1.cmml"><mi id="S2.I7.i3.p1.1.m1.1.1.2" xref="S2.I7.i3.p1.1.m1.1.1.2.cmml">𝐭</mi><mo id="S2.I7.i3.p1.1.m1.1.1.3" xref="S2.I7.i3.p1.1.m1.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S2.I7.i3.p1.1.m1.1b"><apply id="S2.I7.i3.p1.1.m1.1.1.cmml" xref="S2.I7.i3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.I7.i3.p1.1.m1.1.1.1.cmml" xref="S2.I7.i3.p1.1.m1.1.1">superscript</csymbol><ci id="S2.I7.i3.p1.1.m1.1.1.2.cmml" xref="S2.I7.i3.p1.1.m1.1.1.2">𝐭</ci><ci id="S2.I7.i3.p1.1.m1.1.1.3.cmml" xref="S2.I7.i3.p1.1.m1.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I7.i3.p1.1.m1.1c">\mathbf{t}^{\prime}</annotation></semantics></math> to attend only to itself.</p>
</div>
</li>
</ol>
<p id="Thmremark1.p1.2" class="ltx_p">These properties will guide our architecture design.</p>
</div>
</div>
<div id="S2.SS3.p5" class="ltx_para">
<p id="S2.SS3.p5.1" class="ltx_p">To finish the set-up of Problem (<a href="#E1" title="Equation 1 ‣ 1 | Introduction ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>), we define the memory formats.
The definition should subsume the aforementioned formats of model parameters, explicit memories and plain texts for RAG, and also allow for new memory formats of future LLMs.</p>
</div>
<div id="Thmdefinition8" class="ltx_theorem ltx_theorem_definition">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmdefinition8.1.1.1" class="ltx_text ltx_font_bold">Definition 8</span></span><span id="Thmdefinition8.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmdefinition8.p1" class="ltx_para">
<p id="Thmdefinition8.p1.6" class="ltx_p">Let <math id="Thmdefinition8.p1.1.m1.1" class="ltx_Math" alttext="\mathfrak{K}" display="inline"><semantics id="Thmdefinition8.p1.1.m1.1a"><mi id="Thmdefinition8.p1.1.m1.1.1" xref="Thmdefinition8.p1.1.m1.1.1.cmml">𝔎</mi><annotation-xml encoding="MathML-Content" id="Thmdefinition8.p1.1.m1.1b"><ci id="Thmdefinition8.p1.1.m1.1.1.cmml" xref="Thmdefinition8.p1.1.m1.1.1">𝔎</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition8.p1.1.m1.1c">\mathfrak{K}</annotation></semantics></math> be the complete set of knowledges from Assumption <a href="#Thmassumption1" title="Assumption 1 (Completeness). ‣ 2.2 | Knowledge ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> and consider the subset of separable knowledges.
Let <math id="Thmdefinition8.p1.2.m2.1" class="ltx_Math" alttext="\mathfrak{T}" display="inline"><semantics id="Thmdefinition8.p1.2.m2.1a"><mi id="Thmdefinition8.p1.2.m2.1.1" xref="Thmdefinition8.p1.2.m2.1.1.cmml">𝔗</mi><annotation-xml encoding="MathML-Content" id="Thmdefinition8.p1.2.m2.1b"><ci id="Thmdefinition8.p1.2.m2.1.1.cmml" xref="Thmdefinition8.p1.2.m2.1.1">𝔗</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition8.p1.2.m2.1c">\mathfrak{T}</annotation></semantics></math> be a set that contains one or several realizations <math id="Thmdefinition8.p1.3.m3.1" class="ltx_Math" alttext="\mathbf{t}" display="inline"><semantics id="Thmdefinition8.p1.3.m3.1a"><mi id="Thmdefinition8.p1.3.m3.1.1" xref="Thmdefinition8.p1.3.m3.1.1.cmml">𝐭</mi><annotation-xml encoding="MathML-Content" id="Thmdefinition8.p1.3.m3.1b"><ci id="Thmdefinition8.p1.3.m3.1.1.cmml" xref="Thmdefinition8.p1.3.m3.1.1">𝐭</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition8.p1.3.m3.1c">\mathbf{t}</annotation></semantics></math> for each separable knowledge.
Let <math id="Thmdefinition8.p1.4.m4.2" class="ltx_Math" alttext="f_{1},\dots f_{m}" display="inline"><semantics id="Thmdefinition8.p1.4.m4.2a"><mrow id="Thmdefinition8.p1.4.m4.2.2.2" xref="Thmdefinition8.p1.4.m4.2.2.3.cmml"><msub id="Thmdefinition8.p1.4.m4.1.1.1.1" xref="Thmdefinition8.p1.4.m4.1.1.1.1.cmml"><mi id="Thmdefinition8.p1.4.m4.1.1.1.1.2" xref="Thmdefinition8.p1.4.m4.1.1.1.1.2.cmml">f</mi><mn id="Thmdefinition8.p1.4.m4.1.1.1.1.3" xref="Thmdefinition8.p1.4.m4.1.1.1.1.3.cmml">1</mn></msub><mo id="Thmdefinition8.p1.4.m4.2.2.2.3" xref="Thmdefinition8.p1.4.m4.2.2.3.cmml">,</mo><mrow id="Thmdefinition8.p1.4.m4.2.2.2.2" xref="Thmdefinition8.p1.4.m4.2.2.2.2.cmml"><mi mathvariant="normal" id="Thmdefinition8.p1.4.m4.2.2.2.2.2" xref="Thmdefinition8.p1.4.m4.2.2.2.2.2.cmml">…</mi><mo lspace="0em" rspace="0em" id="Thmdefinition8.p1.4.m4.2.2.2.2.1" xref="Thmdefinition8.p1.4.m4.2.2.2.2.1.cmml">​</mo><msub id="Thmdefinition8.p1.4.m4.2.2.2.2.3" xref="Thmdefinition8.p1.4.m4.2.2.2.2.3.cmml"><mi id="Thmdefinition8.p1.4.m4.2.2.2.2.3.2" xref="Thmdefinition8.p1.4.m4.2.2.2.2.3.2.cmml">f</mi><mi id="Thmdefinition8.p1.4.m4.2.2.2.2.3.3" xref="Thmdefinition8.p1.4.m4.2.2.2.2.3.3.cmml">m</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="Thmdefinition8.p1.4.m4.2b"><list id="Thmdefinition8.p1.4.m4.2.2.3.cmml" xref="Thmdefinition8.p1.4.m4.2.2.2"><apply id="Thmdefinition8.p1.4.m4.1.1.1.1.cmml" xref="Thmdefinition8.p1.4.m4.1.1.1.1"><csymbol cd="ambiguous" id="Thmdefinition8.p1.4.m4.1.1.1.1.1.cmml" xref="Thmdefinition8.p1.4.m4.1.1.1.1">subscript</csymbol><ci id="Thmdefinition8.p1.4.m4.1.1.1.1.2.cmml" xref="Thmdefinition8.p1.4.m4.1.1.1.1.2">𝑓</ci><cn type="integer" id="Thmdefinition8.p1.4.m4.1.1.1.1.3.cmml" xref="Thmdefinition8.p1.4.m4.1.1.1.1.3">1</cn></apply><apply id="Thmdefinition8.p1.4.m4.2.2.2.2.cmml" xref="Thmdefinition8.p1.4.m4.2.2.2.2"><times id="Thmdefinition8.p1.4.m4.2.2.2.2.1.cmml" xref="Thmdefinition8.p1.4.m4.2.2.2.2.1"></times><ci id="Thmdefinition8.p1.4.m4.2.2.2.2.2.cmml" xref="Thmdefinition8.p1.4.m4.2.2.2.2.2">…</ci><apply id="Thmdefinition8.p1.4.m4.2.2.2.2.3.cmml" xref="Thmdefinition8.p1.4.m4.2.2.2.2.3"><csymbol cd="ambiguous" id="Thmdefinition8.p1.4.m4.2.2.2.2.3.1.cmml" xref="Thmdefinition8.p1.4.m4.2.2.2.2.3">subscript</csymbol><ci id="Thmdefinition8.p1.4.m4.2.2.2.2.3.2.cmml" xref="Thmdefinition8.p1.4.m4.2.2.2.2.3.2">𝑓</ci><ci id="Thmdefinition8.p1.4.m4.2.2.2.2.3.3.cmml" xref="Thmdefinition8.p1.4.m4.2.2.2.2.3.3">𝑚</ci></apply></apply></list></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition8.p1.4.m4.2c">f_{1},\dots f_{m}</annotation></semantics></math> be any functions over <math id="Thmdefinition8.p1.5.m5.1" class="ltx_Math" alttext="\mathfrak{T}" display="inline"><semantics id="Thmdefinition8.p1.5.m5.1a"><mi id="Thmdefinition8.p1.5.m5.1.1" xref="Thmdefinition8.p1.5.m5.1.1.cmml">𝔗</mi><annotation-xml encoding="MathML-Content" id="Thmdefinition8.p1.5.m5.1b"><ci id="Thmdefinition8.p1.5.m5.1.1.cmml" xref="Thmdefinition8.p1.5.m5.1.1">𝔗</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition8.p1.5.m5.1c">\mathfrak{T}</annotation></semantics></math>.
Abstractly speaking, a memory-augmented LLM <math id="Thmdefinition8.p1.6.m6.1" class="ltx_Math" alttext="M" display="inline"><semantics id="Thmdefinition8.p1.6.m6.1a"><mi id="Thmdefinition8.p1.6.m6.1.1" xref="Thmdefinition8.p1.6.m6.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="Thmdefinition8.p1.6.m6.1b"><ci id="Thmdefinition8.p1.6.m6.1.1.cmml" xref="Thmdefinition8.p1.6.m6.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition8.p1.6.m6.1c">M</annotation></semantics></math> is some mapping from prefixes to token distributions with additional inputs</p>
<table id="E2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="E2.m1.1" class="ltx_math_unparsed" alttext="M:\big{(}(t_{0}\dots t_{i-1}),\{\mathcal{K}_{1},\dots\mathcal{K}_{N}\},X_{1},\dots X_{m}\big{)}\mapsto\mathbb{P}(\cdot|t_{0}\dots t_{i-1})" display="block"><semantics id="E2.m1.1a"><mrow id="E2.m1.1b"><mi id="E2.m1.1.1">M</mi><mo lspace="0.278em" rspace="0.278em" id="E2.m1.1.2">:</mo><mrow id="E2.m1.1.3"><mo maxsize="120%" minsize="120%" id="E2.m1.1.3.1">(</mo><mrow id="E2.m1.1.3.2"><mo stretchy="false" id="E2.m1.1.3.2.1">(</mo><msub id="E2.m1.1.3.2.2"><mi id="E2.m1.1.3.2.2.2">t</mi><mn id="E2.m1.1.3.2.2.3">0</mn></msub><mi mathvariant="normal" id="E2.m1.1.3.2.3">…</mi><msub id="E2.m1.1.3.2.4"><mi id="E2.m1.1.3.2.4.2">t</mi><mrow id="E2.m1.1.3.2.4.3"><mi id="E2.m1.1.3.2.4.3.2">i</mi><mo id="E2.m1.1.3.2.4.3.1">−</mo><mn id="E2.m1.1.3.2.4.3.3">1</mn></mrow></msub><mo stretchy="false" id="E2.m1.1.3.2.5">)</mo></mrow><mo id="E2.m1.1.3.3">,</mo><mrow id="E2.m1.1.3.4"><mo stretchy="false" id="E2.m1.1.3.4.1">{</mo><msub id="E2.m1.1.3.4.2"><mi class="ltx_font_mathcaligraphic" id="E2.m1.1.3.4.2.2">𝒦</mi><mn id="E2.m1.1.3.4.2.3">1</mn></msub><mo id="E2.m1.1.3.4.3">,</mo><mi mathvariant="normal" id="E2.m1.1.3.4.4">…</mi><msub id="E2.m1.1.3.4.5"><mi class="ltx_font_mathcaligraphic" id="E2.m1.1.3.4.5.2">𝒦</mi><mi id="E2.m1.1.3.4.5.3">N</mi></msub><mo stretchy="false" id="E2.m1.1.3.4.6">}</mo></mrow><mo id="E2.m1.1.3.5">,</mo><msub id="E2.m1.1.3.6"><mi id="E2.m1.1.3.6.2">X</mi><mn id="E2.m1.1.3.6.3">1</mn></msub><mo id="E2.m1.1.3.7">,</mo><mi mathvariant="normal" id="E2.m1.1.3.8">…</mi><msub id="E2.m1.1.3.9"><mi id="E2.m1.1.3.9.2">X</mi><mi id="E2.m1.1.3.9.3">m</mi></msub><mo maxsize="120%" minsize="120%" id="E2.m1.1.3.10">)</mo></mrow><mo stretchy="false" id="E2.m1.1.4">↦</mo><mi id="E2.m1.1.5">ℙ</mi><mrow id="E2.m1.1.6"><mo stretchy="false" id="E2.m1.1.6.1">(</mo><mo lspace="0em" rspace="0em" id="E2.m1.1.6.2">⋅</mo><mo fence="false" rspace="0.167em" stretchy="false" id="E2.m1.1.6.3">|</mo><msub id="E2.m1.1.6.4"><mi id="E2.m1.1.6.4.2">t</mi><mn id="E2.m1.1.6.4.3">0</mn></msub><mi mathvariant="normal" id="E2.m1.1.6.5">…</mi><msub id="E2.m1.1.6.6"><mi id="E2.m1.1.6.6.2">t</mi><mrow id="E2.m1.1.6.6.3"><mi id="E2.m1.1.6.6.3.2">i</mi><mo id="E2.m1.1.6.6.3.1">−</mo><mn id="E2.m1.1.6.6.3.3">1</mn></mrow></msub><mo stretchy="false" id="E2.m1.1.6.7">)</mo></mrow></mrow><annotation encoding="application/x-tex" id="E2.m1.1c">M:\big{(}(t_{0}\dots t_{i-1}),\{\mathcal{K}_{1},\dots\mathcal{K}_{N}\},X_{1},\dots X_{m}\big{)}\mapsto\mathbb{P}(\cdot|t_{0}\dots t_{i-1})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
<p id="Thmdefinition8.p1.9" class="ltx_p">where the set <math id="Thmdefinition8.p1.7.m1.2" class="ltx_Math" alttext="\{\mathcal{K}_{1},\dots\mathcal{K}_{N}\}" display="inline"><semantics id="Thmdefinition8.p1.7.m1.2a"><mrow id="Thmdefinition8.p1.7.m1.2.2.2" xref="Thmdefinition8.p1.7.m1.2.2.3.cmml"><mo stretchy="false" id="Thmdefinition8.p1.7.m1.2.2.2.3" xref="Thmdefinition8.p1.7.m1.2.2.3.cmml">{</mo><msub id="Thmdefinition8.p1.7.m1.1.1.1.1" xref="Thmdefinition8.p1.7.m1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="Thmdefinition8.p1.7.m1.1.1.1.1.2" xref="Thmdefinition8.p1.7.m1.1.1.1.1.2.cmml">𝒦</mi><mn id="Thmdefinition8.p1.7.m1.1.1.1.1.3" xref="Thmdefinition8.p1.7.m1.1.1.1.1.3.cmml">1</mn></msub><mo id="Thmdefinition8.p1.7.m1.2.2.2.4" xref="Thmdefinition8.p1.7.m1.2.2.3.cmml">,</mo><mrow id="Thmdefinition8.p1.7.m1.2.2.2.2" xref="Thmdefinition8.p1.7.m1.2.2.2.2.cmml"><mi mathvariant="normal" id="Thmdefinition8.p1.7.m1.2.2.2.2.2" xref="Thmdefinition8.p1.7.m1.2.2.2.2.2.cmml">…</mi><mo lspace="0em" rspace="0em" id="Thmdefinition8.p1.7.m1.2.2.2.2.1" xref="Thmdefinition8.p1.7.m1.2.2.2.2.1.cmml">​</mo><msub id="Thmdefinition8.p1.7.m1.2.2.2.2.3" xref="Thmdefinition8.p1.7.m1.2.2.2.2.3.cmml"><mi class="ltx_font_mathcaligraphic" id="Thmdefinition8.p1.7.m1.2.2.2.2.3.2" xref="Thmdefinition8.p1.7.m1.2.2.2.2.3.2.cmml">𝒦</mi><mi id="Thmdefinition8.p1.7.m1.2.2.2.2.3.3" xref="Thmdefinition8.p1.7.m1.2.2.2.2.3.3.cmml">N</mi></msub></mrow><mo stretchy="false" id="Thmdefinition8.p1.7.m1.2.2.2.5" xref="Thmdefinition8.p1.7.m1.2.2.3.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="Thmdefinition8.p1.7.m1.2b"><set id="Thmdefinition8.p1.7.m1.2.2.3.cmml" xref="Thmdefinition8.p1.7.m1.2.2.2"><apply id="Thmdefinition8.p1.7.m1.1.1.1.1.cmml" xref="Thmdefinition8.p1.7.m1.1.1.1.1"><csymbol cd="ambiguous" id="Thmdefinition8.p1.7.m1.1.1.1.1.1.cmml" xref="Thmdefinition8.p1.7.m1.1.1.1.1">subscript</csymbol><ci id="Thmdefinition8.p1.7.m1.1.1.1.1.2.cmml" xref="Thmdefinition8.p1.7.m1.1.1.1.1.2">𝒦</ci><cn type="integer" id="Thmdefinition8.p1.7.m1.1.1.1.1.3.cmml" xref="Thmdefinition8.p1.7.m1.1.1.1.1.3">1</cn></apply><apply id="Thmdefinition8.p1.7.m1.2.2.2.2.cmml" xref="Thmdefinition8.p1.7.m1.2.2.2.2"><times id="Thmdefinition8.p1.7.m1.2.2.2.2.1.cmml" xref="Thmdefinition8.p1.7.m1.2.2.2.2.1"></times><ci id="Thmdefinition8.p1.7.m1.2.2.2.2.2.cmml" xref="Thmdefinition8.p1.7.m1.2.2.2.2.2">…</ci><apply id="Thmdefinition8.p1.7.m1.2.2.2.2.3.cmml" xref="Thmdefinition8.p1.7.m1.2.2.2.2.3"><csymbol cd="ambiguous" id="Thmdefinition8.p1.7.m1.2.2.2.2.3.1.cmml" xref="Thmdefinition8.p1.7.m1.2.2.2.2.3">subscript</csymbol><ci id="Thmdefinition8.p1.7.m1.2.2.2.2.3.2.cmml" xref="Thmdefinition8.p1.7.m1.2.2.2.2.3.2">𝒦</ci><ci id="Thmdefinition8.p1.7.m1.2.2.2.2.3.3.cmml" xref="Thmdefinition8.p1.7.m1.2.2.2.2.3.3">𝑁</ci></apply></apply></set></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition8.p1.7.m1.2c">\{\mathcal{K}_{1},\dots\mathcal{K}_{N}\}</annotation></semantics></math> consists of non-separable knowledges of <math id="Thmdefinition8.p1.8.m2.1" class="ltx_Math" alttext="M" display="inline"><semantics id="Thmdefinition8.p1.8.m2.1a"><mi id="Thmdefinition8.p1.8.m2.1.1" xref="Thmdefinition8.p1.8.m2.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="Thmdefinition8.p1.8.m2.1b"><ci id="Thmdefinition8.p1.8.m2.1.1.cmml" xref="Thmdefinition8.p1.8.m2.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition8.p1.8.m2.1c">M</annotation></semantics></math> that are invoked at this step, and the sets <math id="Thmdefinition8.p1.9.m3.1" class="ltx_Math" alttext="X_{j}" display="inline"><semantics id="Thmdefinition8.p1.9.m3.1a"><msub id="Thmdefinition8.p1.9.m3.1.1" xref="Thmdefinition8.p1.9.m3.1.1.cmml"><mi id="Thmdefinition8.p1.9.m3.1.1.2" xref="Thmdefinition8.p1.9.m3.1.1.2.cmml">X</mi><mi id="Thmdefinition8.p1.9.m3.1.1.3" xref="Thmdefinition8.p1.9.m3.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="Thmdefinition8.p1.9.m3.1b"><apply id="Thmdefinition8.p1.9.m3.1.1.cmml" xref="Thmdefinition8.p1.9.m3.1.1"><csymbol cd="ambiguous" id="Thmdefinition8.p1.9.m3.1.1.1.cmml" xref="Thmdefinition8.p1.9.m3.1.1">subscript</csymbol><ci id="Thmdefinition8.p1.9.m3.1.1.2.cmml" xref="Thmdefinition8.p1.9.m3.1.1.2">𝑋</ci><ci id="Thmdefinition8.p1.9.m3.1.1.3.cmml" xref="Thmdefinition8.p1.9.m3.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition8.p1.9.m3.1c">X_{j}</annotation></semantics></math> consist of encoded texts</p>
<table id="E3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="E3.m1.3" class="ltx_Math" alttext="X_{j}=\big{\{}f_{j}(\mathbf{t}_{j,k})\big{\}}" display="block"><semantics id="E3.m1.3a"><mrow id="E3.m1.3.3" xref="E3.m1.3.3.cmml"><msub id="E3.m1.3.3.3" xref="E3.m1.3.3.3.cmml"><mi id="E3.m1.3.3.3.2" xref="E3.m1.3.3.3.2.cmml">X</mi><mi id="E3.m1.3.3.3.3" xref="E3.m1.3.3.3.3.cmml">j</mi></msub><mo id="E3.m1.3.3.2" xref="E3.m1.3.3.2.cmml">=</mo><mrow id="E3.m1.3.3.1.1" xref="E3.m1.3.3.1.2.cmml"><mo maxsize="120%" minsize="120%" id="E3.m1.3.3.1.1.2" xref="E3.m1.3.3.1.2.cmml">{</mo><mrow id="E3.m1.3.3.1.1.1" xref="E3.m1.3.3.1.1.1.cmml"><msub id="E3.m1.3.3.1.1.1.3" xref="E3.m1.3.3.1.1.1.3.cmml"><mi id="E3.m1.3.3.1.1.1.3.2" xref="E3.m1.3.3.1.1.1.3.2.cmml">f</mi><mi id="E3.m1.3.3.1.1.1.3.3" xref="E3.m1.3.3.1.1.1.3.3.cmml">j</mi></msub><mo lspace="0em" rspace="0em" id="E3.m1.3.3.1.1.1.2" xref="E3.m1.3.3.1.1.1.2.cmml">​</mo><mrow id="E3.m1.3.3.1.1.1.1.1" xref="E3.m1.3.3.1.1.1.1.1.1.cmml"><mo stretchy="false" id="E3.m1.3.3.1.1.1.1.1.2" xref="E3.m1.3.3.1.1.1.1.1.1.cmml">(</mo><msub id="E3.m1.3.3.1.1.1.1.1.1" xref="E3.m1.3.3.1.1.1.1.1.1.cmml"><mi id="E3.m1.3.3.1.1.1.1.1.1.2" xref="E3.m1.3.3.1.1.1.1.1.1.2.cmml">𝐭</mi><mrow id="E3.m1.2.2.2.4" xref="E3.m1.2.2.2.3.cmml"><mi id="E3.m1.1.1.1.1" xref="E3.m1.1.1.1.1.cmml">j</mi><mo id="E3.m1.2.2.2.4.1" xref="E3.m1.2.2.2.3.cmml">,</mo><mi id="E3.m1.2.2.2.2" xref="E3.m1.2.2.2.2.cmml">k</mi></mrow></msub><mo stretchy="false" id="E3.m1.3.3.1.1.1.1.1.3" xref="E3.m1.3.3.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo maxsize="120%" minsize="120%" id="E3.m1.3.3.1.1.3" xref="E3.m1.3.3.1.2.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="E3.m1.3b"><apply id="E3.m1.3.3.cmml" xref="E3.m1.3.3"><eq id="E3.m1.3.3.2.cmml" xref="E3.m1.3.3.2"></eq><apply id="E3.m1.3.3.3.cmml" xref="E3.m1.3.3.3"><csymbol cd="ambiguous" id="E3.m1.3.3.3.1.cmml" xref="E3.m1.3.3.3">subscript</csymbol><ci id="E3.m1.3.3.3.2.cmml" xref="E3.m1.3.3.3.2">𝑋</ci><ci id="E3.m1.3.3.3.3.cmml" xref="E3.m1.3.3.3.3">𝑗</ci></apply><set id="E3.m1.3.3.1.2.cmml" xref="E3.m1.3.3.1.1"><apply id="E3.m1.3.3.1.1.1.cmml" xref="E3.m1.3.3.1.1.1"><times id="E3.m1.3.3.1.1.1.2.cmml" xref="E3.m1.3.3.1.1.1.2"></times><apply id="E3.m1.3.3.1.1.1.3.cmml" xref="E3.m1.3.3.1.1.1.3"><csymbol cd="ambiguous" id="E3.m1.3.3.1.1.1.3.1.cmml" xref="E3.m1.3.3.1.1.1.3">subscript</csymbol><ci id="E3.m1.3.3.1.1.1.3.2.cmml" xref="E3.m1.3.3.1.1.1.3.2">𝑓</ci><ci id="E3.m1.3.3.1.1.1.3.3.cmml" xref="E3.m1.3.3.1.1.1.3.3">𝑗</ci></apply><apply id="E3.m1.3.3.1.1.1.1.1.1.cmml" xref="E3.m1.3.3.1.1.1.1.1"><csymbol cd="ambiguous" id="E3.m1.3.3.1.1.1.1.1.1.1.cmml" xref="E3.m1.3.3.1.1.1.1.1">subscript</csymbol><ci id="E3.m1.3.3.1.1.1.1.1.1.2.cmml" xref="E3.m1.3.3.1.1.1.1.1.1.2">𝐭</ci><list id="E3.m1.2.2.2.3.cmml" xref="E3.m1.2.2.2.4"><ci id="E3.m1.1.1.1.1.cmml" xref="E3.m1.1.1.1.1">𝑗</ci><ci id="E3.m1.2.2.2.2.cmml" xref="E3.m1.2.2.2.2">𝑘</ci></list></apply></apply></set></apply></annotation-xml><annotation encoding="application/x-tex" id="E3.m1.3c">X_{j}=\big{\{}f_{j}(\mathbf{t}_{j,k})\big{\}}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
</table>
<p id="Thmdefinition8.p1.10" class="ltx_p">for some <math id="Thmdefinition8.p1.10.m1.2" class="ltx_Math" alttext="\mathbf{t}_{j,k}\in\mathfrak{T}" display="inline"><semantics id="Thmdefinition8.p1.10.m1.2a"><mrow id="Thmdefinition8.p1.10.m1.2.3" xref="Thmdefinition8.p1.10.m1.2.3.cmml"><msub id="Thmdefinition8.p1.10.m1.2.3.2" xref="Thmdefinition8.p1.10.m1.2.3.2.cmml"><mi id="Thmdefinition8.p1.10.m1.2.3.2.2" xref="Thmdefinition8.p1.10.m1.2.3.2.2.cmml">𝐭</mi><mrow id="Thmdefinition8.p1.10.m1.2.2.2.4" xref="Thmdefinition8.p1.10.m1.2.2.2.3.cmml"><mi id="Thmdefinition8.p1.10.m1.1.1.1.1" xref="Thmdefinition8.p1.10.m1.1.1.1.1.cmml">j</mi><mo id="Thmdefinition8.p1.10.m1.2.2.2.4.1" xref="Thmdefinition8.p1.10.m1.2.2.2.3.cmml">,</mo><mi id="Thmdefinition8.p1.10.m1.2.2.2.2" xref="Thmdefinition8.p1.10.m1.2.2.2.2.cmml">k</mi></mrow></msub><mo id="Thmdefinition8.p1.10.m1.2.3.1" xref="Thmdefinition8.p1.10.m1.2.3.1.cmml">∈</mo><mi id="Thmdefinition8.p1.10.m1.2.3.3" xref="Thmdefinition8.p1.10.m1.2.3.3.cmml">𝔗</mi></mrow><annotation-xml encoding="MathML-Content" id="Thmdefinition8.p1.10.m1.2b"><apply id="Thmdefinition8.p1.10.m1.2.3.cmml" xref="Thmdefinition8.p1.10.m1.2.3"><in id="Thmdefinition8.p1.10.m1.2.3.1.cmml" xref="Thmdefinition8.p1.10.m1.2.3.1"></in><apply id="Thmdefinition8.p1.10.m1.2.3.2.cmml" xref="Thmdefinition8.p1.10.m1.2.3.2"><csymbol cd="ambiguous" id="Thmdefinition8.p1.10.m1.2.3.2.1.cmml" xref="Thmdefinition8.p1.10.m1.2.3.2">subscript</csymbol><ci id="Thmdefinition8.p1.10.m1.2.3.2.2.cmml" xref="Thmdefinition8.p1.10.m1.2.3.2.2">𝐭</ci><list id="Thmdefinition8.p1.10.m1.2.2.2.3.cmml" xref="Thmdefinition8.p1.10.m1.2.2.2.4"><ci id="Thmdefinition8.p1.10.m1.1.1.1.1.cmml" xref="Thmdefinition8.p1.10.m1.1.1.1.1">𝑗</ci><ci id="Thmdefinition8.p1.10.m1.2.2.2.2.cmml" xref="Thmdefinition8.p1.10.m1.2.2.2.2">𝑘</ci></list></apply><ci id="Thmdefinition8.p1.10.m1.2.3.3.cmml" xref="Thmdefinition8.p1.10.m1.2.3.3">𝔗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition8.p1.10.m1.2c">\mathbf{t}_{j,k}\in\mathfrak{T}</annotation></semantics></math>.</p>
</div>
<div id="Thmdefinition8.p2" class="ltx_para">
<p id="Thmdefinition8.p2.7" class="ltx_p">Each <math id="Thmdefinition8.p2.1.m1.2" class="ltx_Math" alttext="j=1,\dots m" display="inline"><semantics id="Thmdefinition8.p2.1.m1.2a"><mrow id="Thmdefinition8.p2.1.m1.2.2" xref="Thmdefinition8.p2.1.m1.2.2.cmml"><mi id="Thmdefinition8.p2.1.m1.2.2.3" xref="Thmdefinition8.p2.1.m1.2.2.3.cmml">j</mi><mo id="Thmdefinition8.p2.1.m1.2.2.2" xref="Thmdefinition8.p2.1.m1.2.2.2.cmml">=</mo><mrow id="Thmdefinition8.p2.1.m1.2.2.1.1" xref="Thmdefinition8.p2.1.m1.2.2.1.2.cmml"><mn id="Thmdefinition8.p2.1.m1.1.1" xref="Thmdefinition8.p2.1.m1.1.1.cmml">1</mn><mo id="Thmdefinition8.p2.1.m1.2.2.1.1.2" xref="Thmdefinition8.p2.1.m1.2.2.1.2.cmml">,</mo><mrow id="Thmdefinition8.p2.1.m1.2.2.1.1.1" xref="Thmdefinition8.p2.1.m1.2.2.1.1.1.cmml"><mi mathvariant="normal" id="Thmdefinition8.p2.1.m1.2.2.1.1.1.2" xref="Thmdefinition8.p2.1.m1.2.2.1.1.1.2.cmml">…</mi><mo lspace="0em" rspace="0em" id="Thmdefinition8.p2.1.m1.2.2.1.1.1.1" xref="Thmdefinition8.p2.1.m1.2.2.1.1.1.1.cmml">​</mo><mi id="Thmdefinition8.p2.1.m1.2.2.1.1.1.3" xref="Thmdefinition8.p2.1.m1.2.2.1.1.1.3.cmml">m</mi></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="Thmdefinition8.p2.1.m1.2b"><apply id="Thmdefinition8.p2.1.m1.2.2.cmml" xref="Thmdefinition8.p2.1.m1.2.2"><eq id="Thmdefinition8.p2.1.m1.2.2.2.cmml" xref="Thmdefinition8.p2.1.m1.2.2.2"></eq><ci id="Thmdefinition8.p2.1.m1.2.2.3.cmml" xref="Thmdefinition8.p2.1.m1.2.2.3">𝑗</ci><list id="Thmdefinition8.p2.1.m1.2.2.1.2.cmml" xref="Thmdefinition8.p2.1.m1.2.2.1.1"><cn type="integer" id="Thmdefinition8.p2.1.m1.1.1.cmml" xref="Thmdefinition8.p2.1.m1.1.1">1</cn><apply id="Thmdefinition8.p2.1.m1.2.2.1.1.1.cmml" xref="Thmdefinition8.p2.1.m1.2.2.1.1.1"><times id="Thmdefinition8.p2.1.m1.2.2.1.1.1.1.cmml" xref="Thmdefinition8.p2.1.m1.2.2.1.1.1.1"></times><ci id="Thmdefinition8.p2.1.m1.2.2.1.1.1.2.cmml" xref="Thmdefinition8.p2.1.m1.2.2.1.1.1.2">…</ci><ci id="Thmdefinition8.p2.1.m1.2.2.1.1.1.3.cmml" xref="Thmdefinition8.p2.1.m1.2.2.1.1.1.3">𝑚</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition8.p2.1.m1.2c">j=1,\dots m</annotation></semantics></math> represents a memory format and <math id="Thmdefinition8.p2.2.m2.1" class="ltx_Math" alttext="f_{j}" display="inline"><semantics id="Thmdefinition8.p2.2.m2.1a"><msub id="Thmdefinition8.p2.2.m2.1.1" xref="Thmdefinition8.p2.2.m2.1.1.cmml"><mi id="Thmdefinition8.p2.2.m2.1.1.2" xref="Thmdefinition8.p2.2.m2.1.1.2.cmml">f</mi><mi id="Thmdefinition8.p2.2.m2.1.1.3" xref="Thmdefinition8.p2.2.m2.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="Thmdefinition8.p2.2.m2.1b"><apply id="Thmdefinition8.p2.2.m2.1.1.cmml" xref="Thmdefinition8.p2.2.m2.1.1"><csymbol cd="ambiguous" id="Thmdefinition8.p2.2.m2.1.1.1.cmml" xref="Thmdefinition8.p2.2.m2.1.1">subscript</csymbol><ci id="Thmdefinition8.p2.2.m2.1.1.2.cmml" xref="Thmdefinition8.p2.2.m2.1.1.2">𝑓</ci><ci id="Thmdefinition8.p2.2.m2.1.1.3.cmml" xref="Thmdefinition8.p2.2.m2.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition8.p2.2.m2.1c">f_{j}</annotation></semantics></math> is called the <span id="Thmdefinition8.p2.7.1" class="ltx_text ltx_font_typewriter">write</span> function of this format.
If some realization of a separable knowledge <math id="Thmdefinition8.p2.3.m3.1" class="ltx_Math" alttext="\mathcal{K}" display="inline"><semantics id="Thmdefinition8.p2.3.m3.1a"><mi class="ltx_font_mathcaligraphic" id="Thmdefinition8.p2.3.m3.1.1" xref="Thmdefinition8.p2.3.m3.1.1.cmml">𝒦</mi><annotation-xml encoding="MathML-Content" id="Thmdefinition8.p2.3.m3.1b"><ci id="Thmdefinition8.p2.3.m3.1.1.cmml" xref="Thmdefinition8.p2.3.m3.1.1">𝒦</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition8.p2.3.m3.1c">\mathcal{K}</annotation></semantics></math> participates in the mapping <math id="Thmdefinition8.p2.4.m4.1" class="ltx_Math" alttext="M" display="inline"><semantics id="Thmdefinition8.p2.4.m4.1a"><mi id="Thmdefinition8.p2.4.m4.1.1" xref="Thmdefinition8.p2.4.m4.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="Thmdefinition8.p2.4.m4.1b"><ci id="Thmdefinition8.p2.4.m4.1.1.cmml" xref="Thmdefinition8.p2.4.m4.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition8.p2.4.m4.1c">M</annotation></semantics></math>, then we say that <math id="Thmdefinition8.p2.5.m5.1" class="ltx_Math" alttext="\mathcal{K}" display="inline"><semantics id="Thmdefinition8.p2.5.m5.1a"><mi class="ltx_font_mathcaligraphic" id="Thmdefinition8.p2.5.m5.1.1" xref="Thmdefinition8.p2.5.m5.1.1.cmml">𝒦</mi><annotation-xml encoding="MathML-Content" id="Thmdefinition8.p2.5.m5.1b"><ci id="Thmdefinition8.p2.5.m5.1.1.cmml" xref="Thmdefinition8.p2.5.m5.1.1">𝒦</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition8.p2.5.m5.1c">\mathcal{K}</annotation></semantics></math> is <span id="Thmdefinition8.p2.7.2" class="ltx_text ltx_font_typewriter">written</span> in format <math id="Thmdefinition8.p2.6.m6.1" class="ltx_Math" alttext="j" display="inline"><semantics id="Thmdefinition8.p2.6.m6.1a"><mi id="Thmdefinition8.p2.6.m6.1.1" xref="Thmdefinition8.p2.6.m6.1.1.cmml">j</mi><annotation-xml encoding="MathML-Content" id="Thmdefinition8.p2.6.m6.1b"><ci id="Thmdefinition8.p2.6.m6.1.1.cmml" xref="Thmdefinition8.p2.6.m6.1.1">𝑗</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition8.p2.6.m6.1c">j</annotation></semantics></math> and <span id="Thmdefinition8.p2.7.3" class="ltx_text ltx_font_typewriter">read</span> by <math id="Thmdefinition8.p2.7.m7.1" class="ltx_Math" alttext="M" display="inline"><semantics id="Thmdefinition8.p2.7.m7.1a"><mi id="Thmdefinition8.p2.7.m7.1.1" xref="Thmdefinition8.p2.7.m7.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="Thmdefinition8.p2.7.m7.1b"><ci id="Thmdefinition8.p2.7.m7.1.1.cmml" xref="Thmdefinition8.p2.7.m7.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmdefinition8.p2.7.m7.1c">M</annotation></semantics></math>.</p>
</div>
</div>
<div id="S2.SS3.p6" class="ltx_para">
<p id="S2.SS3.p6.1" class="ltx_p">Analogous to Assumption <a href="#Thmassumption1" title="Assumption 1 (Completeness). ‣ 2.2 | Knowledge ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, we are decomposing each step of LLM inference into the invoked circuits, but the decomposition here also involves reference texts that are written in various memory formats.</p>
</div>
<div id="S2.SS3.p7" class="ltx_para">
<p id="S2.SS3.p7.14" class="ltx_p">Table <a href="#T3" title="Table 3 ‣ 1 | Introduction ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> demonstrates that the write functions could be diverse, and the list is probably far from conclusive.
Nevertheless, some heuristics still apply.
The write function <math id="S2.SS3.p7.1.m1.1" class="ltx_Math" alttext="f_{j}" display="inline"><semantics id="S2.SS3.p7.1.m1.1a"><msub id="S2.SS3.p7.1.m1.1.1" xref="S2.SS3.p7.1.m1.1.1.cmml"><mi id="S2.SS3.p7.1.m1.1.1.2" xref="S2.SS3.p7.1.m1.1.1.2.cmml">f</mi><mi id="S2.SS3.p7.1.m1.1.1.3" xref="S2.SS3.p7.1.m1.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p7.1.m1.1b"><apply id="S2.SS3.p7.1.m1.1.1.cmml" xref="S2.SS3.p7.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS3.p7.1.m1.1.1.1.cmml" xref="S2.SS3.p7.1.m1.1.1">subscript</csymbol><ci id="S2.SS3.p7.1.m1.1.1.2.cmml" xref="S2.SS3.p7.1.m1.1.1.2">𝑓</ci><ci id="S2.SS3.p7.1.m1.1.1.3.cmml" xref="S2.SS3.p7.1.m1.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p7.1.m1.1c">f_{j}</annotation></semantics></math> and the read process in <math id="S2.SS3.p7.2.m2.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S2.SS3.p7.2.m2.1a"><mi id="S2.SS3.p7.2.m2.1.1" xref="S2.SS3.p7.2.m2.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p7.2.m2.1b"><ci id="S2.SS3.p7.2.m2.1.1.cmml" xref="S2.SS3.p7.2.m2.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p7.2.m2.1c">M</annotation></semantics></math> for each format <math id="S2.SS3.p7.3.m3.1" class="ltx_Math" alttext="j" display="inline"><semantics id="S2.SS3.p7.3.m3.1a"><mi id="S2.SS3.p7.3.m3.1.1" xref="S2.SS3.p7.3.m3.1.1.cmml">j</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p7.3.m3.1b"><ci id="S2.SS3.p7.3.m3.1.1.cmml" xref="S2.SS3.p7.3.m3.1.1">𝑗</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p7.3.m3.1c">j</annotation></semantics></math> should be non-trivial such that, for any separable knowledge <math id="S2.SS3.p7.4.m4.1" class="ltx_Math" alttext="\mathcal{K}" display="inline"><semantics id="S2.SS3.p7.4.m4.1a"><mi class="ltx_font_mathcaligraphic" id="S2.SS3.p7.4.m4.1.1" xref="S2.SS3.p7.4.m4.1.1.cmml">𝒦</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p7.4.m4.1b"><ci id="S2.SS3.p7.4.m4.1.1.cmml" xref="S2.SS3.p7.4.m4.1.1">𝒦</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p7.4.m4.1c">\mathcal{K}</annotation></semantics></math> not contained in <math id="S2.SS3.p7.5.m5.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S2.SS3.p7.5.m5.1a"><mi id="S2.SS3.p7.5.m5.1.1" xref="S2.SS3.p7.5.m5.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p7.5.m5.1b"><ci id="S2.SS3.p7.5.m5.1.1.cmml" xref="S2.SS3.p7.5.m5.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p7.5.m5.1c">M</annotation></semantics></math> and any realization <math id="S2.SS3.p7.6.m6.1" class="ltx_Math" alttext="\mathbf{t}" display="inline"><semantics id="S2.SS3.p7.6.m6.1a"><mi id="S2.SS3.p7.6.m6.1.1" xref="S2.SS3.p7.6.m6.1.1.cmml">𝐭</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p7.6.m6.1b"><ci id="S2.SS3.p7.6.m6.1.1.cmml" xref="S2.SS3.p7.6.m6.1.1">𝐭</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p7.6.m6.1c">\mathbf{t}</annotation></semantics></math> of <math id="S2.SS3.p7.7.m7.1" class="ltx_Math" alttext="\mathcal{K}" display="inline"><semantics id="S2.SS3.p7.7.m7.1a"><mi class="ltx_font_mathcaligraphic" id="S2.SS3.p7.7.m7.1.1" xref="S2.SS3.p7.7.m7.1.1.cmml">𝒦</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p7.7.m7.1b"><ci id="S2.SS3.p7.7.m7.1.1.cmml" xref="S2.SS3.p7.7.m7.1.1">𝒦</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p7.7.m7.1c">\mathcal{K}</annotation></semantics></math>, if <math id="S2.SS3.p7.8.m8.1" class="ltx_Math" alttext="\mathcal{K}" display="inline"><semantics id="S2.SS3.p7.8.m8.1a"><mi class="ltx_font_mathcaligraphic" id="S2.SS3.p7.8.m8.1.1" xref="S2.SS3.p7.8.m8.1.1.cmml">𝒦</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p7.8.m8.1b"><ci id="S2.SS3.p7.8.m8.1.1.cmml" xref="S2.SS3.p7.8.m8.1.1">𝒦</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p7.8.m8.1c">\mathcal{K}</annotation></semantics></math> enters in <math id="S2.SS3.p7.9.m9.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S2.SS3.p7.9.m9.1a"><mi id="S2.SS3.p7.9.m9.1.1" xref="S2.SS3.p7.9.m9.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p7.9.m9.1b"><ci id="S2.SS3.p7.9.m9.1.1.cmml" xref="S2.SS3.p7.9.m9.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p7.9.m9.1c">M</annotation></semantics></math> through format <math id="S2.SS3.p7.10.m10.1" class="ltx_Math" alttext="j" display="inline"><semantics id="S2.SS3.p7.10.m10.1a"><mi id="S2.SS3.p7.10.m10.1.1" xref="S2.SS3.p7.10.m10.1.1.cmml">j</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p7.10.m10.1b"><ci id="S2.SS3.p7.10.m10.1.1.cmml" xref="S2.SS3.p7.10.m10.1.1">𝑗</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p7.10.m10.1c">j</annotation></semantics></math>, then <math id="S2.SS3.p7.11.m11.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S2.SS3.p7.11.m11.1a"><mi id="S2.SS3.p7.11.m11.1.1" xref="S2.SS3.p7.11.m11.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p7.11.m11.1b"><ci id="S2.SS3.p7.11.m11.1.1.cmml" xref="S2.SS3.p7.11.m11.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p7.11.m11.1c">M</annotation></semantics></math> should be able to generate each token of the associated output of <math id="S2.SS3.p7.12.m12.1" class="ltx_Math" alttext="\mathcal{K}" display="inline"><semantics id="S2.SS3.p7.12.m12.1a"><mi class="ltx_font_mathcaligraphic" id="S2.SS3.p7.12.m12.1.1" xref="S2.SS3.p7.12.m12.1.1.cmml">𝒦</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p7.12.m12.1b"><ci id="S2.SS3.p7.12.m12.1.1.cmml" xref="S2.SS3.p7.12.m12.1.1">𝒦</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p7.12.m12.1c">\mathcal{K}</annotation></semantics></math> in <math id="S2.SS3.p7.13.m13.1" class="ltx_Math" alttext="\mathbf{t}" display="inline"><semantics id="S2.SS3.p7.13.m13.1a"><mi id="S2.SS3.p7.13.m13.1.1" xref="S2.SS3.p7.13.m13.1.1.cmml">𝐭</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p7.13.m13.1b"><ci id="S2.SS3.p7.13.m13.1.1.cmml" xref="S2.SS3.p7.13.m13.1.1">𝐭</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p7.13.m13.1c">\mathbf{t}</annotation></semantics></math> with higher probability as in Definition <a href="#Thmdefinition6" title="Definition 6. ‣ 2.3 | Memory ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>.
Thus, informally speaking, the total cost of writing and reading <math id="S2.SS3.p7.14.m14.1" class="ltx_Math" alttext="\mathcal{K}" display="inline"><semantics id="S2.SS3.p7.14.m14.1a"><mi class="ltx_font_mathcaligraphic" id="S2.SS3.p7.14.m14.1.1" xref="S2.SS3.p7.14.m14.1.1.cmml">𝒦</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p7.14.m14.1b"><ci id="S2.SS3.p7.14.m14.1.1.cmml" xref="S2.SS3.p7.14.m14.1.1">𝒦</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p7.14.m14.1c">\mathcal{K}</annotation></semantics></math> must be bounded from 0, since some minimum computation is necessary for reducing the uncertainty in generating the correct tokens.
It follows that the write cost and read cost are complementary, i.e. cheaper writing must be accompanied by more expensive reading.</p>
</div>
<div id="S2.SS3.p8" class="ltx_para">
<p id="S2.SS3.p8.1" class="ltx_p">We define this inverse relationship between the write cost and read cost as the memory hierarchy.
This relationship is in accordance with our experience regarding the three examples of human memories in Table <a href="#T3" title="Table 3 ‣ 1 | Introduction ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, e.g. we can utter the common expressions almost immediately while it may take a few seconds to recall a book we read, but the former skill is acquired through years of language speaking.
For the LLM memories in Table <a href="#T3" title="Table 3 ‣ 1 | Introduction ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, the inverse relationship is illustrated Figure <a href="#F4" title="Figure 4 ‣ 1 | Introduction ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> and established by the calculations in Appendix <a href="#A1" title="Appendix A Cost Estimation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a>.</p>
</div>
<figure id="F8" class="ltx_figure"><img src="/html/2407.01178/assets/Figures/Theory/knowledge_distribution.png" id="F8.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="449" height="241" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="F8.2.1.1" class="ltx_text" style="font-size:90%;">Figure 8</span>: </span><span id="F8.3.2" class="ltx_text" style="font-size:90%;">Different memory formats with different balances of write cost and read cost.
The specific knowledges with high to low usage counts are exemplified by common expressions, expertise and trivia, and are assigned to implicit memory, explicit memory and external information.</span></figcaption>
</figure>
<div id="S2.SS3.p9" class="ltx_para">
<p id="S2.SS3.p9.2" class="ltx_p">The imbalanced use of knowledges leads to a heterogeneous distribution of knowledges across the memory hierarchy.
To minimize the total cost (<a href="#E1" title="Equation 1 ‣ 1 | Introduction ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>), the separable knowledges that are used more often should be assigned to memory formats with high write cost and low read cost, whereas the rarely used knowledges should be assigned to formats with low write cost and high read cost.
Also, adding a new memory format <math id="S2.SS3.p9.1.m1.1" class="ltx_Math" alttext="m+1" display="inline"><semantics id="S2.SS3.p9.1.m1.1a"><mrow id="S2.SS3.p9.1.m1.1.1" xref="S2.SS3.p9.1.m1.1.1.cmml"><mi id="S2.SS3.p9.1.m1.1.1.2" xref="S2.SS3.p9.1.m1.1.1.2.cmml">m</mi><mo id="S2.SS3.p9.1.m1.1.1.1" xref="S2.SS3.p9.1.m1.1.1.1.cmml">+</mo><mn id="S2.SS3.p9.1.m1.1.1.3" xref="S2.SS3.p9.1.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.p9.1.m1.1b"><apply id="S2.SS3.p9.1.m1.1.1.cmml" xref="S2.SS3.p9.1.m1.1.1"><plus id="S2.SS3.p9.1.m1.1.1.1.cmml" xref="S2.SS3.p9.1.m1.1.1.1"></plus><ci id="S2.SS3.p9.1.m1.1.1.2.cmml" xref="S2.SS3.p9.1.m1.1.1.2">𝑚</ci><cn type="integer" id="S2.SS3.p9.1.m1.1.1.3.cmml" xref="S2.SS3.p9.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p9.1.m1.1c">m+1</annotation></semantics></math> is always beneficial as it expands the search space and decreases the minimum cost whenever the usage count of some knowledge <math id="S2.SS3.p9.2.m2.1" class="ltx_Math" alttext="\mathcal{K}" display="inline"><semantics id="S2.SS3.p9.2.m2.1a"><mi class="ltx_font_mathcaligraphic" id="S2.SS3.p9.2.m2.1.1" xref="S2.SS3.p9.2.m2.1.1.cmml">𝒦</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p9.2.m2.1b"><ci id="S2.SS3.p9.2.m2.1.1.cmml" xref="S2.SS3.p9.2.m2.1.1">𝒦</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p9.2.m2.1c">\mathcal{K}</annotation></semantics></math> lies in the interval</p>
<table id="Ex3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="Ex3.m1.10" class="ltx_Math" alttext="[n^{-}_{m+1},n^{+}_{m+1}]=\big{\{}n\in[0,\infty)~{}\big{|}~{}\text{argmin}_{j}~{}\text{cost}_{\text{write}}(\mathcal{K},j)+n\cdot\text{cost}_{\text{read}}(\mathcal{K},j)=m+1\big{\}}" display="block"><semantics id="Ex3.m1.10a"><mrow id="Ex3.m1.10.10" xref="Ex3.m1.10.10.cmml"><mrow id="Ex3.m1.8.8.2.2" xref="Ex3.m1.8.8.2.3.cmml"><mo stretchy="false" id="Ex3.m1.8.8.2.2.3" xref="Ex3.m1.8.8.2.3.cmml">[</mo><msubsup id="Ex3.m1.7.7.1.1.1" xref="Ex3.m1.7.7.1.1.1.cmml"><mi id="Ex3.m1.7.7.1.1.1.2.2" xref="Ex3.m1.7.7.1.1.1.2.2.cmml">n</mi><mrow id="Ex3.m1.7.7.1.1.1.3" xref="Ex3.m1.7.7.1.1.1.3.cmml"><mi id="Ex3.m1.7.7.1.1.1.3.2" xref="Ex3.m1.7.7.1.1.1.3.2.cmml">m</mi><mo id="Ex3.m1.7.7.1.1.1.3.1" xref="Ex3.m1.7.7.1.1.1.3.1.cmml">+</mo><mn id="Ex3.m1.7.7.1.1.1.3.3" xref="Ex3.m1.7.7.1.1.1.3.3.cmml">1</mn></mrow><mo id="Ex3.m1.7.7.1.1.1.2.3" xref="Ex3.m1.7.7.1.1.1.2.3.cmml">−</mo></msubsup><mo id="Ex3.m1.8.8.2.2.4" xref="Ex3.m1.8.8.2.3.cmml">,</mo><msubsup id="Ex3.m1.8.8.2.2.2" xref="Ex3.m1.8.8.2.2.2.cmml"><mi id="Ex3.m1.8.8.2.2.2.2.2" xref="Ex3.m1.8.8.2.2.2.2.2.cmml">n</mi><mrow id="Ex3.m1.8.8.2.2.2.3" xref="Ex3.m1.8.8.2.2.2.3.cmml"><mi id="Ex3.m1.8.8.2.2.2.3.2" xref="Ex3.m1.8.8.2.2.2.3.2.cmml">m</mi><mo id="Ex3.m1.8.8.2.2.2.3.1" xref="Ex3.m1.8.8.2.2.2.3.1.cmml">+</mo><mn id="Ex3.m1.8.8.2.2.2.3.3" xref="Ex3.m1.8.8.2.2.2.3.3.cmml">1</mn></mrow><mo id="Ex3.m1.8.8.2.2.2.2.3" xref="Ex3.m1.8.8.2.2.2.2.3.cmml">+</mo></msubsup><mo stretchy="false" id="Ex3.m1.8.8.2.2.5" xref="Ex3.m1.8.8.2.3.cmml">]</mo></mrow><mo id="Ex3.m1.10.10.5" xref="Ex3.m1.10.10.5.cmml">=</mo><mrow id="Ex3.m1.10.10.4.2" xref="Ex3.m1.10.10.4.3.cmml"><mo maxsize="120%" minsize="120%" id="Ex3.m1.10.10.4.2.3" xref="Ex3.m1.10.10.4.3.1.cmml">{</mo><mrow id="Ex3.m1.9.9.3.1.1" xref="Ex3.m1.9.9.3.1.1.cmml"><mi id="Ex3.m1.9.9.3.1.1.2" xref="Ex3.m1.9.9.3.1.1.2.cmml">n</mi><mo id="Ex3.m1.9.9.3.1.1.1" xref="Ex3.m1.9.9.3.1.1.1.cmml">∈</mo><mrow id="Ex3.m1.9.9.3.1.1.3.2" xref="Ex3.m1.9.9.3.1.1.3.1.cmml"><mo stretchy="false" id="Ex3.m1.9.9.3.1.1.3.2.1" xref="Ex3.m1.9.9.3.1.1.3.1.cmml">[</mo><mn id="Ex3.m1.1.1" xref="Ex3.m1.1.1.cmml">0</mn><mo id="Ex3.m1.9.9.3.1.1.3.2.2" xref="Ex3.m1.9.9.3.1.1.3.1.cmml">,</mo><mi mathvariant="normal" id="Ex3.m1.2.2" xref="Ex3.m1.2.2.cmml">∞</mi><mo rspace="0.052em" stretchy="false" id="Ex3.m1.9.9.3.1.1.3.2.3" xref="Ex3.m1.9.9.3.1.1.3.1.cmml">)</mo></mrow></mrow><mo mathsize="120%" rspace="0.330em" id="Ex3.m1.10.10.4.2.4" xref="Ex3.m1.10.10.4.3.1.cmml">|</mo><mrow id="Ex3.m1.10.10.4.2.2" xref="Ex3.m1.10.10.4.2.2.cmml"><mrow id="Ex3.m1.10.10.4.2.2.2" xref="Ex3.m1.10.10.4.2.2.2.cmml"><mrow id="Ex3.m1.10.10.4.2.2.2.2" xref="Ex3.m1.10.10.4.2.2.2.2.cmml"><msub id="Ex3.m1.10.10.4.2.2.2.2.2" xref="Ex3.m1.10.10.4.2.2.2.2.2.cmml"><mtext id="Ex3.m1.10.10.4.2.2.2.2.2.2" xref="Ex3.m1.10.10.4.2.2.2.2.2.2a.cmml">argmin</mtext><mi id="Ex3.m1.10.10.4.2.2.2.2.2.3" xref="Ex3.m1.10.10.4.2.2.2.2.2.3.cmml">j</mi></msub><mo lspace="0em" rspace="0em" id="Ex3.m1.10.10.4.2.2.2.2.1" xref="Ex3.m1.10.10.4.2.2.2.2.1.cmml">​</mo><msub id="Ex3.m1.10.10.4.2.2.2.2.3" xref="Ex3.m1.10.10.4.2.2.2.2.3.cmml"><mtext id="Ex3.m1.10.10.4.2.2.2.2.3.2" xref="Ex3.m1.10.10.4.2.2.2.2.3.2a.cmml">cost</mtext><mtext id="Ex3.m1.10.10.4.2.2.2.2.3.3" xref="Ex3.m1.10.10.4.2.2.2.2.3.3a.cmml">write</mtext></msub><mo lspace="0em" rspace="0em" id="Ex3.m1.10.10.4.2.2.2.2.1a" xref="Ex3.m1.10.10.4.2.2.2.2.1.cmml">​</mo><mrow id="Ex3.m1.10.10.4.2.2.2.2.4.2" xref="Ex3.m1.10.10.4.2.2.2.2.4.1.cmml"><mo stretchy="false" id="Ex3.m1.10.10.4.2.2.2.2.4.2.1" xref="Ex3.m1.10.10.4.2.2.2.2.4.1.cmml">(</mo><mi class="ltx_font_mathcaligraphic" id="Ex3.m1.3.3" xref="Ex3.m1.3.3.cmml">𝒦</mi><mo id="Ex3.m1.10.10.4.2.2.2.2.4.2.2" xref="Ex3.m1.10.10.4.2.2.2.2.4.1.cmml">,</mo><mi id="Ex3.m1.4.4" xref="Ex3.m1.4.4.cmml">j</mi><mo stretchy="false" id="Ex3.m1.10.10.4.2.2.2.2.4.2.3" xref="Ex3.m1.10.10.4.2.2.2.2.4.1.cmml">)</mo></mrow></mrow><mo id="Ex3.m1.10.10.4.2.2.2.1" xref="Ex3.m1.10.10.4.2.2.2.1.cmml">+</mo><mrow id="Ex3.m1.10.10.4.2.2.2.3" xref="Ex3.m1.10.10.4.2.2.2.3.cmml"><mrow id="Ex3.m1.10.10.4.2.2.2.3.2" xref="Ex3.m1.10.10.4.2.2.2.3.2.cmml"><mi id="Ex3.m1.10.10.4.2.2.2.3.2.2" xref="Ex3.m1.10.10.4.2.2.2.3.2.2.cmml">n</mi><mo lspace="0.222em" rspace="0.222em" id="Ex3.m1.10.10.4.2.2.2.3.2.1" xref="Ex3.m1.10.10.4.2.2.2.3.2.1.cmml">⋅</mo><msub id="Ex3.m1.10.10.4.2.2.2.3.2.3" xref="Ex3.m1.10.10.4.2.2.2.3.2.3.cmml"><mtext id="Ex3.m1.10.10.4.2.2.2.3.2.3.2" xref="Ex3.m1.10.10.4.2.2.2.3.2.3.2a.cmml">cost</mtext><mtext id="Ex3.m1.10.10.4.2.2.2.3.2.3.3" xref="Ex3.m1.10.10.4.2.2.2.3.2.3.3a.cmml">read</mtext></msub></mrow><mo lspace="0em" rspace="0em" id="Ex3.m1.10.10.4.2.2.2.3.1" xref="Ex3.m1.10.10.4.2.2.2.3.1.cmml">​</mo><mrow id="Ex3.m1.10.10.4.2.2.2.3.3.2" xref="Ex3.m1.10.10.4.2.2.2.3.3.1.cmml"><mo stretchy="false" id="Ex3.m1.10.10.4.2.2.2.3.3.2.1" xref="Ex3.m1.10.10.4.2.2.2.3.3.1.cmml">(</mo><mi class="ltx_font_mathcaligraphic" id="Ex3.m1.5.5" xref="Ex3.m1.5.5.cmml">𝒦</mi><mo id="Ex3.m1.10.10.4.2.2.2.3.3.2.2" xref="Ex3.m1.10.10.4.2.2.2.3.3.1.cmml">,</mo><mi id="Ex3.m1.6.6" xref="Ex3.m1.6.6.cmml">j</mi><mo stretchy="false" id="Ex3.m1.10.10.4.2.2.2.3.3.2.3" xref="Ex3.m1.10.10.4.2.2.2.3.3.1.cmml">)</mo></mrow></mrow></mrow><mo id="Ex3.m1.10.10.4.2.2.1" xref="Ex3.m1.10.10.4.2.2.1.cmml">=</mo><mrow id="Ex3.m1.10.10.4.2.2.3" xref="Ex3.m1.10.10.4.2.2.3.cmml"><mi id="Ex3.m1.10.10.4.2.2.3.2" xref="Ex3.m1.10.10.4.2.2.3.2.cmml">m</mi><mo id="Ex3.m1.10.10.4.2.2.3.1" xref="Ex3.m1.10.10.4.2.2.3.1.cmml">+</mo><mn id="Ex3.m1.10.10.4.2.2.3.3" xref="Ex3.m1.10.10.4.2.2.3.3.cmml">1</mn></mrow></mrow><mo maxsize="120%" minsize="120%" id="Ex3.m1.10.10.4.2.5" xref="Ex3.m1.10.10.4.3.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="Ex3.m1.10b"><apply id="Ex3.m1.10.10.cmml" xref="Ex3.m1.10.10"><eq id="Ex3.m1.10.10.5.cmml" xref="Ex3.m1.10.10.5"></eq><interval closure="closed" id="Ex3.m1.8.8.2.3.cmml" xref="Ex3.m1.8.8.2.2"><apply id="Ex3.m1.7.7.1.1.1.cmml" xref="Ex3.m1.7.7.1.1.1"><csymbol cd="ambiguous" id="Ex3.m1.7.7.1.1.1.1.cmml" xref="Ex3.m1.7.7.1.1.1">subscript</csymbol><apply id="Ex3.m1.7.7.1.1.1.2.cmml" xref="Ex3.m1.7.7.1.1.1"><csymbol cd="ambiguous" id="Ex3.m1.7.7.1.1.1.2.1.cmml" xref="Ex3.m1.7.7.1.1.1">superscript</csymbol><ci id="Ex3.m1.7.7.1.1.1.2.2.cmml" xref="Ex3.m1.7.7.1.1.1.2.2">𝑛</ci><minus id="Ex3.m1.7.7.1.1.1.2.3.cmml" xref="Ex3.m1.7.7.1.1.1.2.3"></minus></apply><apply id="Ex3.m1.7.7.1.1.1.3.cmml" xref="Ex3.m1.7.7.1.1.1.3"><plus id="Ex3.m1.7.7.1.1.1.3.1.cmml" xref="Ex3.m1.7.7.1.1.1.3.1"></plus><ci id="Ex3.m1.7.7.1.1.1.3.2.cmml" xref="Ex3.m1.7.7.1.1.1.3.2">𝑚</ci><cn type="integer" id="Ex3.m1.7.7.1.1.1.3.3.cmml" xref="Ex3.m1.7.7.1.1.1.3.3">1</cn></apply></apply><apply id="Ex3.m1.8.8.2.2.2.cmml" xref="Ex3.m1.8.8.2.2.2"><csymbol cd="ambiguous" id="Ex3.m1.8.8.2.2.2.1.cmml" xref="Ex3.m1.8.8.2.2.2">subscript</csymbol><apply id="Ex3.m1.8.8.2.2.2.2.cmml" xref="Ex3.m1.8.8.2.2.2"><csymbol cd="ambiguous" id="Ex3.m1.8.8.2.2.2.2.1.cmml" xref="Ex3.m1.8.8.2.2.2">superscript</csymbol><ci id="Ex3.m1.8.8.2.2.2.2.2.cmml" xref="Ex3.m1.8.8.2.2.2.2.2">𝑛</ci><plus id="Ex3.m1.8.8.2.2.2.2.3.cmml" xref="Ex3.m1.8.8.2.2.2.2.3"></plus></apply><apply id="Ex3.m1.8.8.2.2.2.3.cmml" xref="Ex3.m1.8.8.2.2.2.3"><plus id="Ex3.m1.8.8.2.2.2.3.1.cmml" xref="Ex3.m1.8.8.2.2.2.3.1"></plus><ci id="Ex3.m1.8.8.2.2.2.3.2.cmml" xref="Ex3.m1.8.8.2.2.2.3.2">𝑚</ci><cn type="integer" id="Ex3.m1.8.8.2.2.2.3.3.cmml" xref="Ex3.m1.8.8.2.2.2.3.3">1</cn></apply></apply></interval><apply id="Ex3.m1.10.10.4.3.cmml" xref="Ex3.m1.10.10.4.2"><csymbol cd="latexml" id="Ex3.m1.10.10.4.3.1.cmml" xref="Ex3.m1.10.10.4.2.3">conditional-set</csymbol><apply id="Ex3.m1.9.9.3.1.1.cmml" xref="Ex3.m1.9.9.3.1.1"><in id="Ex3.m1.9.9.3.1.1.1.cmml" xref="Ex3.m1.9.9.3.1.1.1"></in><ci id="Ex3.m1.9.9.3.1.1.2.cmml" xref="Ex3.m1.9.9.3.1.1.2">𝑛</ci><interval closure="closed-open" id="Ex3.m1.9.9.3.1.1.3.1.cmml" xref="Ex3.m1.9.9.3.1.1.3.2"><cn type="integer" id="Ex3.m1.1.1.cmml" xref="Ex3.m1.1.1">0</cn><infinity id="Ex3.m1.2.2.cmml" xref="Ex3.m1.2.2"></infinity></interval></apply><apply id="Ex3.m1.10.10.4.2.2.cmml" xref="Ex3.m1.10.10.4.2.2"><eq id="Ex3.m1.10.10.4.2.2.1.cmml" xref="Ex3.m1.10.10.4.2.2.1"></eq><apply id="Ex3.m1.10.10.4.2.2.2.cmml" xref="Ex3.m1.10.10.4.2.2.2"><plus id="Ex3.m1.10.10.4.2.2.2.1.cmml" xref="Ex3.m1.10.10.4.2.2.2.1"></plus><apply id="Ex3.m1.10.10.4.2.2.2.2.cmml" xref="Ex3.m1.10.10.4.2.2.2.2"><times id="Ex3.m1.10.10.4.2.2.2.2.1.cmml" xref="Ex3.m1.10.10.4.2.2.2.2.1"></times><apply id="Ex3.m1.10.10.4.2.2.2.2.2.cmml" xref="Ex3.m1.10.10.4.2.2.2.2.2"><csymbol cd="ambiguous" id="Ex3.m1.10.10.4.2.2.2.2.2.1.cmml" xref="Ex3.m1.10.10.4.2.2.2.2.2">subscript</csymbol><ci id="Ex3.m1.10.10.4.2.2.2.2.2.2a.cmml" xref="Ex3.m1.10.10.4.2.2.2.2.2.2"><mtext id="Ex3.m1.10.10.4.2.2.2.2.2.2.cmml" xref="Ex3.m1.10.10.4.2.2.2.2.2.2">argmin</mtext></ci><ci id="Ex3.m1.10.10.4.2.2.2.2.2.3.cmml" xref="Ex3.m1.10.10.4.2.2.2.2.2.3">𝑗</ci></apply><apply id="Ex3.m1.10.10.4.2.2.2.2.3.cmml" xref="Ex3.m1.10.10.4.2.2.2.2.3"><csymbol cd="ambiguous" id="Ex3.m1.10.10.4.2.2.2.2.3.1.cmml" xref="Ex3.m1.10.10.4.2.2.2.2.3">subscript</csymbol><ci id="Ex3.m1.10.10.4.2.2.2.2.3.2a.cmml" xref="Ex3.m1.10.10.4.2.2.2.2.3.2"><mtext id="Ex3.m1.10.10.4.2.2.2.2.3.2.cmml" xref="Ex3.m1.10.10.4.2.2.2.2.3.2">cost</mtext></ci><ci id="Ex3.m1.10.10.4.2.2.2.2.3.3a.cmml" xref="Ex3.m1.10.10.4.2.2.2.2.3.3"><mtext mathsize="70%" id="Ex3.m1.10.10.4.2.2.2.2.3.3.cmml" xref="Ex3.m1.10.10.4.2.2.2.2.3.3">write</mtext></ci></apply><interval closure="open" id="Ex3.m1.10.10.4.2.2.2.2.4.1.cmml" xref="Ex3.m1.10.10.4.2.2.2.2.4.2"><ci id="Ex3.m1.3.3.cmml" xref="Ex3.m1.3.3">𝒦</ci><ci id="Ex3.m1.4.4.cmml" xref="Ex3.m1.4.4">𝑗</ci></interval></apply><apply id="Ex3.m1.10.10.4.2.2.2.3.cmml" xref="Ex3.m1.10.10.4.2.2.2.3"><times id="Ex3.m1.10.10.4.2.2.2.3.1.cmml" xref="Ex3.m1.10.10.4.2.2.2.3.1"></times><apply id="Ex3.m1.10.10.4.2.2.2.3.2.cmml" xref="Ex3.m1.10.10.4.2.2.2.3.2"><ci id="Ex3.m1.10.10.4.2.2.2.3.2.1.cmml" xref="Ex3.m1.10.10.4.2.2.2.3.2.1">⋅</ci><ci id="Ex3.m1.10.10.4.2.2.2.3.2.2.cmml" xref="Ex3.m1.10.10.4.2.2.2.3.2.2">𝑛</ci><apply id="Ex3.m1.10.10.4.2.2.2.3.2.3.cmml" xref="Ex3.m1.10.10.4.2.2.2.3.2.3"><csymbol cd="ambiguous" id="Ex3.m1.10.10.4.2.2.2.3.2.3.1.cmml" xref="Ex3.m1.10.10.4.2.2.2.3.2.3">subscript</csymbol><ci id="Ex3.m1.10.10.4.2.2.2.3.2.3.2a.cmml" xref="Ex3.m1.10.10.4.2.2.2.3.2.3.2"><mtext id="Ex3.m1.10.10.4.2.2.2.3.2.3.2.cmml" xref="Ex3.m1.10.10.4.2.2.2.3.2.3.2">cost</mtext></ci><ci id="Ex3.m1.10.10.4.2.2.2.3.2.3.3a.cmml" xref="Ex3.m1.10.10.4.2.2.2.3.2.3.3"><mtext mathsize="70%" id="Ex3.m1.10.10.4.2.2.2.3.2.3.3.cmml" xref="Ex3.m1.10.10.4.2.2.2.3.2.3.3">read</mtext></ci></apply></apply><interval closure="open" id="Ex3.m1.10.10.4.2.2.2.3.3.1.cmml" xref="Ex3.m1.10.10.4.2.2.2.3.3.2"><ci id="Ex3.m1.5.5.cmml" xref="Ex3.m1.5.5">𝒦</ci><ci id="Ex3.m1.6.6.cmml" xref="Ex3.m1.6.6">𝑗</ci></interval></apply></apply><apply id="Ex3.m1.10.10.4.2.2.3.cmml" xref="Ex3.m1.10.10.4.2.2.3"><plus id="Ex3.m1.10.10.4.2.2.3.1.cmml" xref="Ex3.m1.10.10.4.2.2.3.1"></plus><ci id="Ex3.m1.10.10.4.2.2.3.2.cmml" xref="Ex3.m1.10.10.4.2.2.3.2">𝑚</ci><cn type="integer" id="Ex3.m1.10.10.4.2.2.3.3.cmml" xref="Ex3.m1.10.10.4.2.2.3.3">1</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Ex3.m1.10c">[n^{-}_{m+1},n^{+}_{m+1}]=\big{\{}n\in[0,\infty)~{}\big{|}~{}\text{argmin}_{j}~{}\text{cost}_{\text{write}}(\mathcal{K},j)+n\cdot\text{cost}_{\text{read}}(\mathcal{K},j)=m+1\big{\}}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S2.SS3.p9.3" class="ltx_p">Examples of these intervals are displayed in Figure <a href="#F4" title="Figure 4 ‣ 1 | Introduction ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>.
For concreteness, Figure <a href="#F8" title="Figure 8 ‣ 2.3 | Memory ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a> depicts a reasonable distribution of the specific knowledges for humans, and we expect a similar distribution to hold for LLMs equipped with explicit memory.</p>
</div>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_section" style="font-size:144%;color:#00868B;">3 <span id="S3.1.1" class="ltx_text" style="color:#808080;">| </span>Design</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.1" class="ltx_p">This section describes the architecture and training scheme of Memory<sup id="S3.p1.1.1" class="ltx_sup">3</sup>.</p>
</div>
<div id="S3.p2" class="ltx_para">
<p id="S3.p2.1" class="ltx_p">Regarding architecture, the goal is to design an explicit memory mechanism for Transformer LLMs with moderately low write cost and read cost.
In addition, we want to limit the modification to the Transformer architecture to be as little as possible, adding no new trainable parameters, so that most of the existing Transformer LLMs can be converted to Memory<sup id="S3.p2.1.1" class="ltx_sup">3</sup> models with little finetuning.
Thus, we arrive at a simple design:</p>
<ul id="S3.I1" class="ltx_itemize">
<li id="S3.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S3.I1.i1.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S3.I1.i1.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S3.I1.i1.1.1.m1.1.1" xref="S3.I1.i1.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S3.I1.i1.1.1.m1.1c"><ci id="S3.I1.i1.1.1.m1.1.1.cmml" xref="S3.I1.i1.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i1.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S3.I1.i1.p1" class="ltx_para">
<p id="S3.I1.i1.p1.1" class="ltx_p">Write cost:
Before inference, the LLM writes each reference to an explicit memory, saved on drives.
The memory is selected from the key-value vectors of the self-attention layers, so the write process involves no training.
Each reference is processed independently, avoiding the cost of long-context attention.</p>
</div>
</li>
<li id="S3.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S3.I1.i2.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S3.I1.i2.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S3.I1.i2.1.1.m1.1.1" xref="S3.I1.i2.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S3.I1.i2.1.1.m1.1c"><ci id="S3.I1.i2.1.1.m1.1.1.cmml" xref="S3.I1.i2.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i2.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S3.I1.i2.p1" class="ltx_para">
<p id="S3.I1.i2.p1.1" class="ltx_p">Read cost:
During inference, explicit memories are retrieved from drives and read by self-attention alongside the usual context key-values.
Each memory consists of very few key-values from a small amount of attention heads, thus greatly reducing the extra compute, GPU storage, drive storage and loading time.
It allows the LLM to retrieve many references frequently with limited influence on decoding speed.</p>
</div>
</li>
</ul>
</div>
<div id="S3.p3" class="ltx_para">
<p id="S3.p3.1" class="ltx_p">Regarding training, the goal is to reduce the cost of pretraining with a more efficient distribution of knowledge.
Based on the discussion in Section <a href="#S2.SS3" title="2.3 | Memory ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.3</span></a>, we want to encourage the LLM to learn only abstract knowledges, with the specific knowledges mostly externalized to the explicit memory bank.
Ideally, the pretraining cost should be reduced to be proportional to the small amount of knowledge stored in the model parameters, thereby taking a step closer to the learning efficiency of humans.</p>
</div>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsection" style="font-size:120%;color:#00868B;">3.1 <span id="S3.SS1.1.1" class="ltx_text" style="color:#808080;">| </span>Inference Process</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.1" class="ltx_p">From now on, we refer to the realizations of separable knowledges (Definitions <a href="#Thmdefinition5" title="Definition 5. ‣ 2.2 | Knowledge ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> and <a href="#Thmdefinition6" title="Definition 6. ‣ 2.3 | Memory ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>) as references.
Our knowledge base (or reference dataset) consists of <math id="S3.SS1.p1.1.m1.1" class="ltx_Math" alttext="1.1\times 10^{8}" display="inline"><semantics id="S3.SS1.p1.1.m1.1a"><mrow id="S3.SS1.p1.1.m1.1.1" xref="S3.SS1.p1.1.m1.1.1.cmml"><mn id="S3.SS1.p1.1.m1.1.1.2" xref="S3.SS1.p1.1.m1.1.1.2.cmml">1.1</mn><mo lspace="0.222em" rspace="0.222em" id="S3.SS1.p1.1.m1.1.1.1" xref="S3.SS1.p1.1.m1.1.1.1.cmml">×</mo><msup id="S3.SS1.p1.1.m1.1.1.3" xref="S3.SS1.p1.1.m1.1.1.3.cmml"><mn id="S3.SS1.p1.1.m1.1.1.3.2" xref="S3.SS1.p1.1.m1.1.1.3.2.cmml">10</mn><mn id="S3.SS1.p1.1.m1.1.1.3.3" xref="S3.SS1.p1.1.m1.1.1.3.3.cmml">8</mn></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.1.m1.1b"><apply id="S3.SS1.p1.1.m1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1"><times id="S3.SS1.p1.1.m1.1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1.1"></times><cn type="float" id="S3.SS1.p1.1.m1.1.1.2.cmml" xref="S3.SS1.p1.1.m1.1.1.2">1.1</cn><apply id="S3.SS1.p1.1.m1.1.1.3.cmml" xref="S3.SS1.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p1.1.m1.1.1.3.1.cmml" xref="S3.SS1.p1.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="S3.SS1.p1.1.m1.1.1.3.2.cmml" xref="S3.SS1.p1.1.m1.1.1.3.2">10</cn><cn type="integer" id="S3.SS1.p1.1.m1.1.1.3.3.cmml" xref="S3.SS1.p1.1.m1.1.1.3.3">8</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.1.m1.1c">1.1\times 10^{8}</annotation></semantics></math> text chunks with length bounded by 128 tokens.
Its composition is described in Section <a href="#S4.SS4" title="4.4 | Knowledge Base ‣ 4 | Pretraining Data ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.4</span></a>.</p>
</div>
<div id="S3.SS1.p2" class="ltx_para">
<p id="S3.SS1.p2.2" class="ltx_p">Each reference can be converted to an explicit memory, which is a tensor with shape</p>
<p id="S3.SS1.p2.1.1" class="ltx_p ltx_align_center">(memory layers, 2, key-value heads, sparse tokens, head dimension) = <math id="S3.SS1.p2.1.1.m1.5" class="ltx_Math" alttext="(22,2,8,8,80)" display="inline"><semantics id="S3.SS1.p2.1.1.m1.5a"><mrow id="S3.SS1.p2.1.1.m1.5.6.2" xref="S3.SS1.p2.1.1.m1.5.6.1.cmml"><mo stretchy="false" id="S3.SS1.p2.1.1.m1.5.6.2.1" xref="S3.SS1.p2.1.1.m1.5.6.1.cmml">(</mo><mn id="S3.SS1.p2.1.1.m1.1.1" xref="S3.SS1.p2.1.1.m1.1.1.cmml">22</mn><mo id="S3.SS1.p2.1.1.m1.5.6.2.2" xref="S3.SS1.p2.1.1.m1.5.6.1.cmml">,</mo><mn id="S3.SS1.p2.1.1.m1.2.2" xref="S3.SS1.p2.1.1.m1.2.2.cmml">2</mn><mo id="S3.SS1.p2.1.1.m1.5.6.2.3" xref="S3.SS1.p2.1.1.m1.5.6.1.cmml">,</mo><mn id="S3.SS1.p2.1.1.m1.3.3" xref="S3.SS1.p2.1.1.m1.3.3.cmml">8</mn><mo id="S3.SS1.p2.1.1.m1.5.6.2.4" xref="S3.SS1.p2.1.1.m1.5.6.1.cmml">,</mo><mn id="S3.SS1.p2.1.1.m1.4.4" xref="S3.SS1.p2.1.1.m1.4.4.cmml">8</mn><mo id="S3.SS1.p2.1.1.m1.5.6.2.5" xref="S3.SS1.p2.1.1.m1.5.6.1.cmml">,</mo><mn id="S3.SS1.p2.1.1.m1.5.5" xref="S3.SS1.p2.1.1.m1.5.5.cmml">80</mn><mo stretchy="false" id="S3.SS1.p2.1.1.m1.5.6.2.6" xref="S3.SS1.p2.1.1.m1.5.6.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.1.1.m1.5b"><vector id="S3.SS1.p2.1.1.m1.5.6.1.cmml" xref="S3.SS1.p2.1.1.m1.5.6.2"><cn type="integer" id="S3.SS1.p2.1.1.m1.1.1.cmml" xref="S3.SS1.p2.1.1.m1.1.1">22</cn><cn type="integer" id="S3.SS1.p2.1.1.m1.2.2.cmml" xref="S3.SS1.p2.1.1.m1.2.2">2</cn><cn type="integer" id="S3.SS1.p2.1.1.m1.3.3.cmml" xref="S3.SS1.p2.1.1.m1.3.3">8</cn><cn type="integer" id="S3.SS1.p2.1.1.m1.4.4.cmml" xref="S3.SS1.p2.1.1.m1.4.4">8</cn><cn type="integer" id="S3.SS1.p2.1.1.m1.5.5.cmml" xref="S3.SS1.p2.1.1.m1.5.5">80</cn></vector></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.1.1.m1.5c">(22,2,8,8,80)</annotation></semantics></math></p>
<p id="S3.SS1.p2.3" class="ltx_p">The 2 stands for the key and value, while the other numbers are introduced later.</p>
</div>
<div id="S3.SS1.p3" class="ltx_para">
<p id="S3.SS1.p3.2" class="ltx_p">Before inference, the Memory<sup id="S3.SS1.p3.2.1" class="ltx_sup">3</sup> model converts all references to explicit memories and save them on drives or non-volatile storage devices.
Then, at inference time, whenever (the id of) a reference is retrieved, its explicit memory is loaded from drives and sent to GPU to be integrated into the computation of Memory<sup id="S3.SS1.p3.2.2" class="ltx_sup">3</sup>.
By Remark <a href="#Thmremark1" title="Remark 1. ‣ 2.3 | Memory ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, a reference during encoding does not need to attend to any other texts (e.g. other references or query texts), so it is fine to encode each reference independently prior to inference.
Such isolation also helps to reduce the compute of attention.</p>
</div>
<div id="S3.SS1.p4" class="ltx_para">
<p id="S3.SS1.p4.1" class="ltx_p">One can also employ a “cold start” approach to bypass preparation time: each reference is converted to explicit memory upon its initial retrieval, rather than prior to inference.
Subsequent retrievals will then access this stored memory.
The aforementioned inference with precomputed explicit memories will be called “warm start”.</p>
</div>
<figure id="F9" class="ltx_figure"><img src="/html/2407.01178/assets/Figures/key_figure/m3mory_inference.png" id="F9.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="685" height="246" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="F9.5.2.1" class="ltx_text" style="font-size:90%;">Figure 9</span>: </span><span id="F9.2.1" class="ltx_text" style="font-size:90%;">The decoding process of Memory<sup id="F9.2.1.1" class="ltx_sup">3</sup> with memory recall.
Each chunk is a fixed-length interval of tokens, which may belong to either the prompt or generated text.</span></figcaption>
</figure>
<div id="S3.SS1.p5" class="ltx_para">
<p id="S3.SS1.p5.1" class="ltx_p">During inference, as illustrated in Figure <a href="#F9" title="Figure 9 ‣ 3.1 | Inference Process ‣ 3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>, whenever the LLM generates 64 tokens, it discards the current memories, uses these 64 tokens as query text to retrieve 5 new memories, and continues decoding with these memories.
Similarly, when processing the prompt, the LLM retrieves 5 memories for each chunk of 64 tokens.
Each chunk attends to its own memories, and the memories could be different across chunks.
We leave it to future work to optimize these hyperparameters.</p>
</div>
<div id="S3.SS1.p6" class="ltx_para">
<p id="S3.SS1.p6.1" class="ltx_p">The retrieval is performed with plain vector search with cosine similarity.
The references as well as the query chunks are embedded by BGE-M3, a multilingual BERT model <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib17" title="" class="ltx_ref">17</a>]</cite>.
The query and key vectors for retrieval are both obtained from the output feature of the <math id="S3.SS1.p6.1.m1.1" class="ltx_Math" alttext="\langle\text{cls}\rangle" display="inline"><semantics id="S3.SS1.p6.1.m1.1a"><mrow id="S3.SS1.p6.1.m1.1.2.2" xref="S3.SS1.p6.1.m1.1.2.1.cmml"><mo stretchy="false" id="S3.SS1.p6.1.m1.1.2.2.1" xref="S3.SS1.p6.1.m1.1.2.1.1.cmml">⟨</mo><mtext id="S3.SS1.p6.1.m1.1.1" xref="S3.SS1.p6.1.m1.1.1a.cmml">cls</mtext><mo stretchy="false" id="S3.SS1.p6.1.m1.1.2.2.2" xref="S3.SS1.p6.1.m1.1.2.1.1.cmml">⟩</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p6.1.m1.1b"><apply id="S3.SS1.p6.1.m1.1.2.1.cmml" xref="S3.SS1.p6.1.m1.1.2.2"><csymbol cd="latexml" id="S3.SS1.p6.1.m1.1.2.1.1.cmml" xref="S3.SS1.p6.1.m1.1.2.2.1">delimited-⟨⟩</csymbol><ci id="S3.SS1.p6.1.m1.1.1a.cmml" xref="S3.SS1.p6.1.m1.1.1"><mtext id="S3.SS1.p6.1.m1.1.1.cmml" xref="S3.SS1.p6.1.m1.1.1">cls</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p6.1.m1.1c">\langle\text{cls}\rangle</annotation></semantics></math> token.
The vector index is built with FAISS <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib35" title="" class="ltx_ref">35</a>]</cite>.</p>
</div>
<div id="S3.SS1.p7" class="ltx_para">
<p id="S3.SS1.p7.1" class="ltx_p">To further save time, we maintain a fixed-size cache in RAM to store the most recently used explicit memories.
It’s been observed that adjacent chunks often retrieve some of the same references.
So the cache reduces the cost of loading explicit memories from drives.</p>
</div>
<div id="Thmremark2" class="ltx_theorem ltx_theorem_remark">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmremark2.1.1.1" class="ltx_text ltx_font_bold">Remark 2</span></span><span id="Thmremark2.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmremark2.p1" class="ltx_para">
<p id="Thmremark2.p1.2" class="ltx_p">It would be ideal to perform retrieval using the hidden features from the LLM itself,
since conceptually the LLM should know its needs better than any external module,
and such internalized retrieval appears more anthropomorphic.
Moreover, retrieving with the hidden features from different layers, different heads and different keywords can help to obtain more diverse results.
One simple implementation is to use the sparsified attention queries of the query text to directly search for the explicit memories.
Since the explicit memories are the attention key-values, such retrieval can work without the need to finetune the LLM.
Specifically, this multi-vector retrieval can follow the routine of <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib61" title="" class="ltx_ref">61</a>]</cite> with the additional constraint that a query from attention head <math id="Thmremark2.p1.1.m1.1" class="ltx_Math" alttext="h" display="inline"><semantics id="Thmremark2.p1.1.m1.1a"><mi id="Thmremark2.p1.1.m1.1.1" xref="Thmremark2.p1.1.m1.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="Thmremark2.p1.1.m1.1b"><ci id="Thmremark2.p1.1.m1.1.1.cmml" xref="Thmremark2.p1.1.m1.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmremark2.p1.1.m1.1c">h</annotation></semantics></math> can only search for keys from <math id="Thmremark2.p1.2.m2.1" class="ltx_Math" alttext="h" display="inline"><semantics id="Thmremark2.p1.2.m2.1a"><mi id="Thmremark2.p1.2.m2.1.1" xref="Thmremark2.p1.2.m2.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="Thmremark2.p1.2.m2.1b"><ci id="Thmremark2.p1.2.m2.1.1.cmml" xref="Thmremark2.p1.2.m2.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmremark2.p1.2.m2.1c">h</annotation></semantics></math>,
while the sparse attention queries can be obtain using the same selection mechanism for explicit memories described later.</p>
</div>
</div>
<div id="Thmremark3" class="ltx_theorem ltx_theorem_remark">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmremark3.1.1.1" class="ltx_text ltx_font_bold">Remark 3</span></span><span id="Thmremark3.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmremark3.p1" class="ltx_para">
<p id="Thmremark3.p1.1" class="ltx_p">One shortcoming of RAG is that the references are usually text chunks instead of whole documents, and thus during inference the references are encoded without their contexts, making them less comprehensible.
This shortcoming can be easily overcome for explicit memories.
One solution is to encode each document as one sequence, then chunk the attention key-values into 128-token chunks and sparsify them into explicit memories.
This procedure allows the key-values to attend to all their contexts.</p>
</div>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsection" style="font-size:120%;color:#00868B;">3.2 <span id="S3.SS2.1.1" class="ltx_text" style="color:#808080;">| </span>Writing and Reading Memory</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.3" class="ltx_p">Each explicit memory is a subset of the attention key-values from a subset of attention heads when encoding a reference.
Thus, during inference, the LLM can directly read the retrieved explicit memories through its self-attention layers by concatenating them with the usual context key-values (Figure <a href="#F9" title="Figure 9 ‣ 3.1 | Inference Process ‣ 3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>).
Specially, for each attention head <math id="S3.SS2.p1.1.m1.1" class="ltx_Math" alttext="h" display="inline"><semantics id="S3.SS2.p1.1.m1.1a"><mi id="S3.SS2.p1.1.m1.1.1" xref="S3.SS2.p1.1.m1.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.1.m1.1b"><ci id="S3.SS2.p1.1.m1.1.1.cmml" xref="S3.SS2.p1.1.m1.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.1.m1.1c">h</annotation></semantics></math> at layer <math id="S3.SS2.p1.2.m2.1" class="ltx_Math" alttext="l" display="inline"><semantics id="S3.SS2.p1.2.m2.1a"><mi id="S3.SS2.p1.2.m2.1.1" xref="S3.SS2.p1.2.m2.1.1.cmml">l</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.2.m2.1b"><ci id="S3.SS2.p1.2.m2.1.1.cmml" xref="S3.SS2.p1.2.m2.1.1">𝑙</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.2.m2.1c">l</annotation></semantics></math>, if it is chosen as a memory head, then its output <math id="S3.SS2.p1.3.m3.2" class="ltx_Math" alttext="Y^{l,h}" display="inline"><semantics id="S3.SS2.p1.3.m3.2a"><msup id="S3.SS2.p1.3.m3.2.3" xref="S3.SS2.p1.3.m3.2.3.cmml"><mi id="S3.SS2.p1.3.m3.2.3.2" xref="S3.SS2.p1.3.m3.2.3.2.cmml">Y</mi><mrow id="S3.SS2.p1.3.m3.2.2.2.4" xref="S3.SS2.p1.3.m3.2.2.2.3.cmml"><mi id="S3.SS2.p1.3.m3.1.1.1.1" xref="S3.SS2.p1.3.m3.1.1.1.1.cmml">l</mi><mo id="S3.SS2.p1.3.m3.2.2.2.4.1" xref="S3.SS2.p1.3.m3.2.2.2.3.cmml">,</mo><mi id="S3.SS2.p1.3.m3.2.2.2.2" xref="S3.SS2.p1.3.m3.2.2.2.2.cmml">h</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.3.m3.2b"><apply id="S3.SS2.p1.3.m3.2.3.cmml" xref="S3.SS2.p1.3.m3.2.3"><csymbol cd="ambiguous" id="S3.SS2.p1.3.m3.2.3.1.cmml" xref="S3.SS2.p1.3.m3.2.3">superscript</csymbol><ci id="S3.SS2.p1.3.m3.2.3.2.cmml" xref="S3.SS2.p1.3.m3.2.3.2">𝑌</ci><list id="S3.SS2.p1.3.m3.2.2.2.3.cmml" xref="S3.SS2.p1.3.m3.2.2.2.4"><ci id="S3.SS2.p1.3.m3.1.1.1.1.cmml" xref="S3.SS2.p1.3.m3.1.1.1.1">𝑙</ci><ci id="S3.SS2.p1.3.m3.2.2.2.2.cmml" xref="S3.SS2.p1.3.m3.2.2.2.2">ℎ</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.3.m3.2c">Y^{l,h}</annotation></semantics></math> changes from the usual</p>
<table id="Ex4" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="Ex4.m1.19" class="ltx_Math" alttext="Y^{l,h}_{i}=\text{softmax}\Big{(}\frac{X^{l,h}_{i}W^{l,h}_{Q}\big{(}X^{l,h}_{[:i]}W^{l,h}_{K}\big{)}^{T}}{\sqrt{d_{h}}}\Big{)}X^{l,h}_{[:i]}W^{l,h}_{V}W^{l,h}_{O}" display="block"><semantics id="Ex4.m1.19a"><mrow id="Ex4.m1.19.20" xref="Ex4.m1.19.20.cmml"><msubsup id="Ex4.m1.19.20.2" xref="Ex4.m1.19.20.2.cmml"><mi id="Ex4.m1.19.20.2.2.2" xref="Ex4.m1.19.20.2.2.2.cmml">Y</mi><mi id="Ex4.m1.19.20.2.3" xref="Ex4.m1.19.20.2.3.cmml">i</mi><mrow id="Ex4.m1.2.2.2.4" xref="Ex4.m1.2.2.2.3.cmml"><mi id="Ex4.m1.1.1.1.1" xref="Ex4.m1.1.1.1.1.cmml">l</mi><mo id="Ex4.m1.2.2.2.4.1" xref="Ex4.m1.2.2.2.3.cmml">,</mo><mi id="Ex4.m1.2.2.2.2" xref="Ex4.m1.2.2.2.2.cmml">h</mi></mrow></msubsup><mo id="Ex4.m1.19.20.1" xref="Ex4.m1.19.20.1.cmml">=</mo><mrow id="Ex4.m1.19.20.3" xref="Ex4.m1.19.20.3.cmml"><mtext id="Ex4.m1.19.20.3.2" xref="Ex4.m1.19.20.3.2a.cmml">softmax</mtext><mo lspace="0em" rspace="0em" id="Ex4.m1.19.20.3.1" xref="Ex4.m1.19.20.3.1.cmml">​</mo><mrow id="Ex4.m1.19.20.3.3.2" xref="Ex4.m1.12.12.cmml"><mo maxsize="160%" minsize="160%" id="Ex4.m1.19.20.3.3.2.1" xref="Ex4.m1.12.12.cmml">(</mo><mfrac id="Ex4.m1.12.12" xref="Ex4.m1.12.12.cmml"><mrow id="Ex4.m1.12.12.10" xref="Ex4.m1.12.12.10.cmml"><msubsup id="Ex4.m1.12.12.10.12" xref="Ex4.m1.12.12.10.12.cmml"><mi id="Ex4.m1.12.12.10.12.2.2" xref="Ex4.m1.12.12.10.12.2.2.cmml">X</mi><mi id="Ex4.m1.12.12.10.12.3" xref="Ex4.m1.12.12.10.12.3.cmml">i</mi><mrow id="Ex4.m1.4.4.2.2.2.4" xref="Ex4.m1.4.4.2.2.2.3.cmml"><mi id="Ex4.m1.3.3.1.1.1.1" xref="Ex4.m1.3.3.1.1.1.1.cmml">l</mi><mo id="Ex4.m1.4.4.2.2.2.4.1" xref="Ex4.m1.4.4.2.2.2.3.cmml">,</mo><mi id="Ex4.m1.4.4.2.2.2.2" xref="Ex4.m1.4.4.2.2.2.2.cmml">h</mi></mrow></msubsup><mo lspace="0em" rspace="0em" id="Ex4.m1.12.12.10.11" xref="Ex4.m1.12.12.10.11.cmml">​</mo><msubsup id="Ex4.m1.12.12.10.13" xref="Ex4.m1.12.12.10.13.cmml"><mi id="Ex4.m1.12.12.10.13.2.2" xref="Ex4.m1.12.12.10.13.2.2.cmml">W</mi><mi id="Ex4.m1.12.12.10.13.3" xref="Ex4.m1.12.12.10.13.3.cmml">Q</mi><mrow id="Ex4.m1.6.6.4.4.2.4" xref="Ex4.m1.6.6.4.4.2.3.cmml"><mi id="Ex4.m1.5.5.3.3.1.1" xref="Ex4.m1.5.5.3.3.1.1.cmml">l</mi><mo id="Ex4.m1.6.6.4.4.2.4.1" xref="Ex4.m1.6.6.4.4.2.3.cmml">,</mo><mi id="Ex4.m1.6.6.4.4.2.2" xref="Ex4.m1.6.6.4.4.2.2.cmml">h</mi></mrow></msubsup><mo lspace="0em" rspace="0em" id="Ex4.m1.12.12.10.11a" xref="Ex4.m1.12.12.10.11.cmml">​</mo><msup id="Ex4.m1.12.12.10.10" xref="Ex4.m1.12.12.10.10.cmml"><mrow id="Ex4.m1.12.12.10.10.1.1" xref="Ex4.m1.12.12.10.10.1.1.1.cmml"><mo maxsize="120%" minsize="120%" id="Ex4.m1.12.12.10.10.1.1.2" xref="Ex4.m1.12.12.10.10.1.1.1.cmml">(</mo><mrow id="Ex4.m1.12.12.10.10.1.1.1" xref="Ex4.m1.12.12.10.10.1.1.1.cmml"><msubsup id="Ex4.m1.12.12.10.10.1.1.1.2" xref="Ex4.m1.12.12.10.10.1.1.1.2.cmml"><mi id="Ex4.m1.12.12.10.10.1.1.1.2.2.2" xref="Ex4.m1.12.12.10.10.1.1.1.2.2.2.cmml">X</mi><mrow id="Ex4.m1.9.9.7.7.1.1" xref="Ex4.m1.9.9.7.7.1.2.cmml"><mo stretchy="false" id="Ex4.m1.9.9.7.7.1.1.2" xref="Ex4.m1.9.9.7.7.1.2.1.cmml">[</mo><mrow id="Ex4.m1.9.9.7.7.1.1.1" xref="Ex4.m1.9.9.7.7.1.1.1.cmml"><mi id="Ex4.m1.9.9.7.7.1.1.1.2" xref="Ex4.m1.9.9.7.7.1.1.1.2.cmml"></mi><mo lspace="0.278em" rspace="0.278em" id="Ex4.m1.9.9.7.7.1.1.1.1" xref="Ex4.m1.9.9.7.7.1.1.1.1.cmml">:</mo><mi id="Ex4.m1.9.9.7.7.1.1.1.3" xref="Ex4.m1.9.9.7.7.1.1.1.3.cmml">i</mi></mrow><mo stretchy="false" id="Ex4.m1.9.9.7.7.1.1.3" xref="Ex4.m1.9.9.7.7.1.2.1.cmml">]</mo></mrow><mrow id="Ex4.m1.8.8.6.6.2.4" xref="Ex4.m1.8.8.6.6.2.3.cmml"><mi id="Ex4.m1.7.7.5.5.1.1" xref="Ex4.m1.7.7.5.5.1.1.cmml">l</mi><mo id="Ex4.m1.8.8.6.6.2.4.1" xref="Ex4.m1.8.8.6.6.2.3.cmml">,</mo><mi id="Ex4.m1.8.8.6.6.2.2" xref="Ex4.m1.8.8.6.6.2.2.cmml">h</mi></mrow></msubsup><mo lspace="0em" rspace="0em" id="Ex4.m1.12.12.10.10.1.1.1.1" xref="Ex4.m1.12.12.10.10.1.1.1.1.cmml">​</mo><msubsup id="Ex4.m1.12.12.10.10.1.1.1.3" xref="Ex4.m1.12.12.10.10.1.1.1.3.cmml"><mi id="Ex4.m1.12.12.10.10.1.1.1.3.2.2" xref="Ex4.m1.12.12.10.10.1.1.1.3.2.2.cmml">W</mi><mi id="Ex4.m1.12.12.10.10.1.1.1.3.3" xref="Ex4.m1.12.12.10.10.1.1.1.3.3.cmml">K</mi><mrow id="Ex4.m1.11.11.9.9.2.4" xref="Ex4.m1.11.11.9.9.2.3.cmml"><mi id="Ex4.m1.10.10.8.8.1.1" xref="Ex4.m1.10.10.8.8.1.1.cmml">l</mi><mo id="Ex4.m1.11.11.9.9.2.4.1" xref="Ex4.m1.11.11.9.9.2.3.cmml">,</mo><mi id="Ex4.m1.11.11.9.9.2.2" xref="Ex4.m1.11.11.9.9.2.2.cmml">h</mi></mrow></msubsup></mrow><mo maxsize="120%" minsize="120%" id="Ex4.m1.12.12.10.10.1.1.3" xref="Ex4.m1.12.12.10.10.1.1.1.cmml">)</mo></mrow><mi id="Ex4.m1.12.12.10.10.3" xref="Ex4.m1.12.12.10.10.3.cmml">T</mi></msup></mrow><msqrt id="Ex4.m1.12.12.12" xref="Ex4.m1.12.12.12.cmml"><msub id="Ex4.m1.12.12.12.2" xref="Ex4.m1.12.12.12.2.cmml"><mi id="Ex4.m1.12.12.12.2.2" xref="Ex4.m1.12.12.12.2.2.cmml">d</mi><mi id="Ex4.m1.12.12.12.2.3" xref="Ex4.m1.12.12.12.2.3.cmml">h</mi></msub></msqrt></mfrac><mo maxsize="160%" minsize="160%" id="Ex4.m1.19.20.3.3.2.2" xref="Ex4.m1.12.12.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="Ex4.m1.19.20.3.1a" xref="Ex4.m1.19.20.3.1.cmml">​</mo><msubsup id="Ex4.m1.19.20.3.4" xref="Ex4.m1.19.20.3.4.cmml"><mi id="Ex4.m1.19.20.3.4.2.2" xref="Ex4.m1.19.20.3.4.2.2.cmml">X</mi><mrow id="Ex4.m1.15.15.1.1" xref="Ex4.m1.15.15.1.2.cmml"><mo stretchy="false" id="Ex4.m1.15.15.1.1.2" xref="Ex4.m1.15.15.1.2.1.cmml">[</mo><mrow id="Ex4.m1.15.15.1.1.1" xref="Ex4.m1.15.15.1.1.1.cmml"><mi id="Ex4.m1.15.15.1.1.1.2" xref="Ex4.m1.15.15.1.1.1.2.cmml"></mi><mo lspace="0.278em" rspace="0.278em" id="Ex4.m1.15.15.1.1.1.1" xref="Ex4.m1.15.15.1.1.1.1.cmml">:</mo><mi id="Ex4.m1.15.15.1.1.1.3" xref="Ex4.m1.15.15.1.1.1.3.cmml">i</mi></mrow><mo stretchy="false" id="Ex4.m1.15.15.1.1.3" xref="Ex4.m1.15.15.1.2.1.cmml">]</mo></mrow><mrow id="Ex4.m1.14.14.2.4" xref="Ex4.m1.14.14.2.3.cmml"><mi id="Ex4.m1.13.13.1.1" xref="Ex4.m1.13.13.1.1.cmml">l</mi><mo id="Ex4.m1.14.14.2.4.1" xref="Ex4.m1.14.14.2.3.cmml">,</mo><mi id="Ex4.m1.14.14.2.2" xref="Ex4.m1.14.14.2.2.cmml">h</mi></mrow></msubsup><mo lspace="0em" rspace="0em" id="Ex4.m1.19.20.3.1b" xref="Ex4.m1.19.20.3.1.cmml">​</mo><msubsup id="Ex4.m1.19.20.3.5" xref="Ex4.m1.19.20.3.5.cmml"><mi id="Ex4.m1.19.20.3.5.2.2" xref="Ex4.m1.19.20.3.5.2.2.cmml">W</mi><mi id="Ex4.m1.19.20.3.5.3" xref="Ex4.m1.19.20.3.5.3.cmml">V</mi><mrow id="Ex4.m1.17.17.2.4" xref="Ex4.m1.17.17.2.3.cmml"><mi id="Ex4.m1.16.16.1.1" xref="Ex4.m1.16.16.1.1.cmml">l</mi><mo id="Ex4.m1.17.17.2.4.1" xref="Ex4.m1.17.17.2.3.cmml">,</mo><mi id="Ex4.m1.17.17.2.2" xref="Ex4.m1.17.17.2.2.cmml">h</mi></mrow></msubsup><mo lspace="0em" rspace="0em" id="Ex4.m1.19.20.3.1c" xref="Ex4.m1.19.20.3.1.cmml">​</mo><msubsup id="Ex4.m1.19.20.3.6" xref="Ex4.m1.19.20.3.6.cmml"><mi id="Ex4.m1.19.20.3.6.2.2" xref="Ex4.m1.19.20.3.6.2.2.cmml">W</mi><mi id="Ex4.m1.19.20.3.6.3" xref="Ex4.m1.19.20.3.6.3.cmml">O</mi><mrow id="Ex4.m1.19.19.2.4" xref="Ex4.m1.19.19.2.3.cmml"><mi id="Ex4.m1.18.18.1.1" xref="Ex4.m1.18.18.1.1.cmml">l</mi><mo id="Ex4.m1.19.19.2.4.1" xref="Ex4.m1.19.19.2.3.cmml">,</mo><mi id="Ex4.m1.19.19.2.2" xref="Ex4.m1.19.19.2.2.cmml">h</mi></mrow></msubsup></mrow></mrow><annotation-xml encoding="MathML-Content" id="Ex4.m1.19b"><apply id="Ex4.m1.19.20.cmml" xref="Ex4.m1.19.20"><eq id="Ex4.m1.19.20.1.cmml" xref="Ex4.m1.19.20.1"></eq><apply id="Ex4.m1.19.20.2.cmml" xref="Ex4.m1.19.20.2"><csymbol cd="ambiguous" id="Ex4.m1.19.20.2.1.cmml" xref="Ex4.m1.19.20.2">subscript</csymbol><apply id="Ex4.m1.19.20.2.2.cmml" xref="Ex4.m1.19.20.2"><csymbol cd="ambiguous" id="Ex4.m1.19.20.2.2.1.cmml" xref="Ex4.m1.19.20.2">superscript</csymbol><ci id="Ex4.m1.19.20.2.2.2.cmml" xref="Ex4.m1.19.20.2.2.2">𝑌</ci><list id="Ex4.m1.2.2.2.3.cmml" xref="Ex4.m1.2.2.2.4"><ci id="Ex4.m1.1.1.1.1.cmml" xref="Ex4.m1.1.1.1.1">𝑙</ci><ci id="Ex4.m1.2.2.2.2.cmml" xref="Ex4.m1.2.2.2.2">ℎ</ci></list></apply><ci id="Ex4.m1.19.20.2.3.cmml" xref="Ex4.m1.19.20.2.3">𝑖</ci></apply><apply id="Ex4.m1.19.20.3.cmml" xref="Ex4.m1.19.20.3"><times id="Ex4.m1.19.20.3.1.cmml" xref="Ex4.m1.19.20.3.1"></times><ci id="Ex4.m1.19.20.3.2a.cmml" xref="Ex4.m1.19.20.3.2"><mtext id="Ex4.m1.19.20.3.2.cmml" xref="Ex4.m1.19.20.3.2">softmax</mtext></ci><apply id="Ex4.m1.12.12.cmml" xref="Ex4.m1.19.20.3.3.2"><divide id="Ex4.m1.12.12.11.cmml" xref="Ex4.m1.19.20.3.3.2"></divide><apply id="Ex4.m1.12.12.10.cmml" xref="Ex4.m1.12.12.10"><times id="Ex4.m1.12.12.10.11.cmml" xref="Ex4.m1.12.12.10.11"></times><apply id="Ex4.m1.12.12.10.12.cmml" xref="Ex4.m1.12.12.10.12"><csymbol cd="ambiguous" id="Ex4.m1.12.12.10.12.1.cmml" xref="Ex4.m1.12.12.10.12">subscript</csymbol><apply id="Ex4.m1.12.12.10.12.2.cmml" xref="Ex4.m1.12.12.10.12"><csymbol cd="ambiguous" id="Ex4.m1.12.12.10.12.2.1.cmml" xref="Ex4.m1.12.12.10.12">superscript</csymbol><ci id="Ex4.m1.12.12.10.12.2.2.cmml" xref="Ex4.m1.12.12.10.12.2.2">𝑋</ci><list id="Ex4.m1.4.4.2.2.2.3.cmml" xref="Ex4.m1.4.4.2.2.2.4"><ci id="Ex4.m1.3.3.1.1.1.1.cmml" xref="Ex4.m1.3.3.1.1.1.1">𝑙</ci><ci id="Ex4.m1.4.4.2.2.2.2.cmml" xref="Ex4.m1.4.4.2.2.2.2">ℎ</ci></list></apply><ci id="Ex4.m1.12.12.10.12.3.cmml" xref="Ex4.m1.12.12.10.12.3">𝑖</ci></apply><apply id="Ex4.m1.12.12.10.13.cmml" xref="Ex4.m1.12.12.10.13"><csymbol cd="ambiguous" id="Ex4.m1.12.12.10.13.1.cmml" xref="Ex4.m1.12.12.10.13">subscript</csymbol><apply id="Ex4.m1.12.12.10.13.2.cmml" xref="Ex4.m1.12.12.10.13"><csymbol cd="ambiguous" id="Ex4.m1.12.12.10.13.2.1.cmml" xref="Ex4.m1.12.12.10.13">superscript</csymbol><ci id="Ex4.m1.12.12.10.13.2.2.cmml" xref="Ex4.m1.12.12.10.13.2.2">𝑊</ci><list id="Ex4.m1.6.6.4.4.2.3.cmml" xref="Ex4.m1.6.6.4.4.2.4"><ci id="Ex4.m1.5.5.3.3.1.1.cmml" xref="Ex4.m1.5.5.3.3.1.1">𝑙</ci><ci id="Ex4.m1.6.6.4.4.2.2.cmml" xref="Ex4.m1.6.6.4.4.2.2">ℎ</ci></list></apply><ci id="Ex4.m1.12.12.10.13.3.cmml" xref="Ex4.m1.12.12.10.13.3">𝑄</ci></apply><apply id="Ex4.m1.12.12.10.10.cmml" xref="Ex4.m1.12.12.10.10"><csymbol cd="ambiguous" id="Ex4.m1.12.12.10.10.2.cmml" xref="Ex4.m1.12.12.10.10">superscript</csymbol><apply id="Ex4.m1.12.12.10.10.1.1.1.cmml" xref="Ex4.m1.12.12.10.10.1.1"><times id="Ex4.m1.12.12.10.10.1.1.1.1.cmml" xref="Ex4.m1.12.12.10.10.1.1.1.1"></times><apply id="Ex4.m1.12.12.10.10.1.1.1.2.cmml" xref="Ex4.m1.12.12.10.10.1.1.1.2"><csymbol cd="ambiguous" id="Ex4.m1.12.12.10.10.1.1.1.2.1.cmml" xref="Ex4.m1.12.12.10.10.1.1.1.2">subscript</csymbol><apply id="Ex4.m1.12.12.10.10.1.1.1.2.2.cmml" xref="Ex4.m1.12.12.10.10.1.1.1.2"><csymbol cd="ambiguous" id="Ex4.m1.12.12.10.10.1.1.1.2.2.1.cmml" xref="Ex4.m1.12.12.10.10.1.1.1.2">superscript</csymbol><ci id="Ex4.m1.12.12.10.10.1.1.1.2.2.2.cmml" xref="Ex4.m1.12.12.10.10.1.1.1.2.2.2">𝑋</ci><list id="Ex4.m1.8.8.6.6.2.3.cmml" xref="Ex4.m1.8.8.6.6.2.4"><ci id="Ex4.m1.7.7.5.5.1.1.cmml" xref="Ex4.m1.7.7.5.5.1.1">𝑙</ci><ci id="Ex4.m1.8.8.6.6.2.2.cmml" xref="Ex4.m1.8.8.6.6.2.2">ℎ</ci></list></apply><apply id="Ex4.m1.9.9.7.7.1.2.cmml" xref="Ex4.m1.9.9.7.7.1.1"><csymbol cd="latexml" id="Ex4.m1.9.9.7.7.1.2.1.cmml" xref="Ex4.m1.9.9.7.7.1.1.2">delimited-[]</csymbol><apply id="Ex4.m1.9.9.7.7.1.1.1.cmml" xref="Ex4.m1.9.9.7.7.1.1.1"><ci id="Ex4.m1.9.9.7.7.1.1.1.1.cmml" xref="Ex4.m1.9.9.7.7.1.1.1.1">:</ci><csymbol cd="latexml" id="Ex4.m1.9.9.7.7.1.1.1.2.cmml" xref="Ex4.m1.9.9.7.7.1.1.1.2">absent</csymbol><ci id="Ex4.m1.9.9.7.7.1.1.1.3.cmml" xref="Ex4.m1.9.9.7.7.1.1.1.3">𝑖</ci></apply></apply></apply><apply id="Ex4.m1.12.12.10.10.1.1.1.3.cmml" xref="Ex4.m1.12.12.10.10.1.1.1.3"><csymbol cd="ambiguous" id="Ex4.m1.12.12.10.10.1.1.1.3.1.cmml" xref="Ex4.m1.12.12.10.10.1.1.1.3">subscript</csymbol><apply id="Ex4.m1.12.12.10.10.1.1.1.3.2.cmml" xref="Ex4.m1.12.12.10.10.1.1.1.3"><csymbol cd="ambiguous" id="Ex4.m1.12.12.10.10.1.1.1.3.2.1.cmml" xref="Ex4.m1.12.12.10.10.1.1.1.3">superscript</csymbol><ci id="Ex4.m1.12.12.10.10.1.1.1.3.2.2.cmml" xref="Ex4.m1.12.12.10.10.1.1.1.3.2.2">𝑊</ci><list id="Ex4.m1.11.11.9.9.2.3.cmml" xref="Ex4.m1.11.11.9.9.2.4"><ci id="Ex4.m1.10.10.8.8.1.1.cmml" xref="Ex4.m1.10.10.8.8.1.1">𝑙</ci><ci id="Ex4.m1.11.11.9.9.2.2.cmml" xref="Ex4.m1.11.11.9.9.2.2">ℎ</ci></list></apply><ci id="Ex4.m1.12.12.10.10.1.1.1.3.3.cmml" xref="Ex4.m1.12.12.10.10.1.1.1.3.3">𝐾</ci></apply></apply><ci id="Ex4.m1.12.12.10.10.3.cmml" xref="Ex4.m1.12.12.10.10.3">𝑇</ci></apply></apply><apply id="Ex4.m1.12.12.12.cmml" xref="Ex4.m1.12.12.12"><root id="Ex4.m1.12.12.12a.cmml" xref="Ex4.m1.12.12.12"></root><apply id="Ex4.m1.12.12.12.2.cmml" xref="Ex4.m1.12.12.12.2"><csymbol cd="ambiguous" id="Ex4.m1.12.12.12.2.1.cmml" xref="Ex4.m1.12.12.12.2">subscript</csymbol><ci id="Ex4.m1.12.12.12.2.2.cmml" xref="Ex4.m1.12.12.12.2.2">𝑑</ci><ci id="Ex4.m1.12.12.12.2.3.cmml" xref="Ex4.m1.12.12.12.2.3">ℎ</ci></apply></apply></apply><apply id="Ex4.m1.19.20.3.4.cmml" xref="Ex4.m1.19.20.3.4"><csymbol cd="ambiguous" id="Ex4.m1.19.20.3.4.1.cmml" xref="Ex4.m1.19.20.3.4">subscript</csymbol><apply id="Ex4.m1.19.20.3.4.2.cmml" xref="Ex4.m1.19.20.3.4"><csymbol cd="ambiguous" id="Ex4.m1.19.20.3.4.2.1.cmml" xref="Ex4.m1.19.20.3.4">superscript</csymbol><ci id="Ex4.m1.19.20.3.4.2.2.cmml" xref="Ex4.m1.19.20.3.4.2.2">𝑋</ci><list id="Ex4.m1.14.14.2.3.cmml" xref="Ex4.m1.14.14.2.4"><ci id="Ex4.m1.13.13.1.1.cmml" xref="Ex4.m1.13.13.1.1">𝑙</ci><ci id="Ex4.m1.14.14.2.2.cmml" xref="Ex4.m1.14.14.2.2">ℎ</ci></list></apply><apply id="Ex4.m1.15.15.1.2.cmml" xref="Ex4.m1.15.15.1.1"><csymbol cd="latexml" id="Ex4.m1.15.15.1.2.1.cmml" xref="Ex4.m1.15.15.1.1.2">delimited-[]</csymbol><apply id="Ex4.m1.15.15.1.1.1.cmml" xref="Ex4.m1.15.15.1.1.1"><ci id="Ex4.m1.15.15.1.1.1.1.cmml" xref="Ex4.m1.15.15.1.1.1.1">:</ci><csymbol cd="latexml" id="Ex4.m1.15.15.1.1.1.2.cmml" xref="Ex4.m1.15.15.1.1.1.2">absent</csymbol><ci id="Ex4.m1.15.15.1.1.1.3.cmml" xref="Ex4.m1.15.15.1.1.1.3">𝑖</ci></apply></apply></apply><apply id="Ex4.m1.19.20.3.5.cmml" xref="Ex4.m1.19.20.3.5"><csymbol cd="ambiguous" id="Ex4.m1.19.20.3.5.1.cmml" xref="Ex4.m1.19.20.3.5">subscript</csymbol><apply id="Ex4.m1.19.20.3.5.2.cmml" xref="Ex4.m1.19.20.3.5"><csymbol cd="ambiguous" id="Ex4.m1.19.20.3.5.2.1.cmml" xref="Ex4.m1.19.20.3.5">superscript</csymbol><ci id="Ex4.m1.19.20.3.5.2.2.cmml" xref="Ex4.m1.19.20.3.5.2.2">𝑊</ci><list id="Ex4.m1.17.17.2.3.cmml" xref="Ex4.m1.17.17.2.4"><ci id="Ex4.m1.16.16.1.1.cmml" xref="Ex4.m1.16.16.1.1">𝑙</ci><ci id="Ex4.m1.17.17.2.2.cmml" xref="Ex4.m1.17.17.2.2">ℎ</ci></list></apply><ci id="Ex4.m1.19.20.3.5.3.cmml" xref="Ex4.m1.19.20.3.5.3">𝑉</ci></apply><apply id="Ex4.m1.19.20.3.6.cmml" xref="Ex4.m1.19.20.3.6"><csymbol cd="ambiguous" id="Ex4.m1.19.20.3.6.1.cmml" xref="Ex4.m1.19.20.3.6">subscript</csymbol><apply id="Ex4.m1.19.20.3.6.2.cmml" xref="Ex4.m1.19.20.3.6"><csymbol cd="ambiguous" id="Ex4.m1.19.20.3.6.2.1.cmml" xref="Ex4.m1.19.20.3.6">superscript</csymbol><ci id="Ex4.m1.19.20.3.6.2.2.cmml" xref="Ex4.m1.19.20.3.6.2.2">𝑊</ci><list id="Ex4.m1.19.19.2.3.cmml" xref="Ex4.m1.19.19.2.4"><ci id="Ex4.m1.18.18.1.1.cmml" xref="Ex4.m1.18.18.1.1">𝑙</ci><ci id="Ex4.m1.19.19.2.2.cmml" xref="Ex4.m1.19.19.2.2">ℎ</ci></list></apply><ci id="Ex4.m1.19.20.3.6.3.cmml" xref="Ex4.m1.19.20.3.6.3">𝑂</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Ex4.m1.19c">Y^{l,h}_{i}=\text{softmax}\Big{(}\frac{X^{l,h}_{i}W^{l,h}_{Q}\big{(}X^{l,h}_{[:i]}W^{l,h}_{K}\big{)}^{T}}{\sqrt{d_{h}}}\Big{)}X^{l,h}_{[:i]}W^{l,h}_{V}W^{l,h}_{O}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S3.SS2.p1.6" class="ltx_p">where <math id="S3.SS2.p1.4.m1.1" class="ltx_Math" alttext="X_{[:i]}" display="inline"><semantics id="S3.SS2.p1.4.m1.1a"><msub id="S3.SS2.p1.4.m1.1.2" xref="S3.SS2.p1.4.m1.1.2.cmml"><mi id="S3.SS2.p1.4.m1.1.2.2" xref="S3.SS2.p1.4.m1.1.2.2.cmml">X</mi><mrow id="S3.SS2.p1.4.m1.1.1.1.1" xref="S3.SS2.p1.4.m1.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS2.p1.4.m1.1.1.1.1.2" xref="S3.SS2.p1.4.m1.1.1.1.2.1.cmml">[</mo><mrow id="S3.SS2.p1.4.m1.1.1.1.1.1" xref="S3.SS2.p1.4.m1.1.1.1.1.1.cmml"><mi id="S3.SS2.p1.4.m1.1.1.1.1.1.2" xref="S3.SS2.p1.4.m1.1.1.1.1.1.2.cmml"></mi><mo lspace="0.278em" rspace="0.278em" id="S3.SS2.p1.4.m1.1.1.1.1.1.1" xref="S3.SS2.p1.4.m1.1.1.1.1.1.1.cmml">:</mo><mi id="S3.SS2.p1.4.m1.1.1.1.1.1.3" xref="S3.SS2.p1.4.m1.1.1.1.1.1.3.cmml">i</mi></mrow><mo stretchy="false" id="S3.SS2.p1.4.m1.1.1.1.1.3" xref="S3.SS2.p1.4.m1.1.1.1.2.1.cmml">]</mo></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.4.m1.1b"><apply id="S3.SS2.p1.4.m1.1.2.cmml" xref="S3.SS2.p1.4.m1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p1.4.m1.1.2.1.cmml" xref="S3.SS2.p1.4.m1.1.2">subscript</csymbol><ci id="S3.SS2.p1.4.m1.1.2.2.cmml" xref="S3.SS2.p1.4.m1.1.2.2">𝑋</ci><apply id="S3.SS2.p1.4.m1.1.1.1.2.cmml" xref="S3.SS2.p1.4.m1.1.1.1.1"><csymbol cd="latexml" id="S3.SS2.p1.4.m1.1.1.1.2.1.cmml" xref="S3.SS2.p1.4.m1.1.1.1.1.2">delimited-[]</csymbol><apply id="S3.SS2.p1.4.m1.1.1.1.1.1.cmml" xref="S3.SS2.p1.4.m1.1.1.1.1.1"><ci id="S3.SS2.p1.4.m1.1.1.1.1.1.1.cmml" xref="S3.SS2.p1.4.m1.1.1.1.1.1.1">:</ci><csymbol cd="latexml" id="S3.SS2.p1.4.m1.1.1.1.1.1.2.cmml" xref="S3.SS2.p1.4.m1.1.1.1.1.1.2">absent</csymbol><ci id="S3.SS2.p1.4.m1.1.1.1.1.1.3.cmml" xref="S3.SS2.p1.4.m1.1.1.1.1.1.3">𝑖</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.4.m1.1c">X_{[:i]}</annotation></semantics></math> denotes all tokens before or at position <math id="S3.SS2.p1.5.m2.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S3.SS2.p1.5.m2.1a"><mi id="S3.SS2.p1.5.m2.1.1" xref="S3.SS2.p1.5.m2.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.5.m2.1b"><ci id="S3.SS2.p1.5.m2.1.1.cmml" xref="S3.SS2.p1.5.m2.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.5.m2.1c">i</annotation></semantics></math> and <math id="S3.SS2.p1.6.m3.1" class="ltx_Math" alttext="d_{h}" display="inline"><semantics id="S3.SS2.p1.6.m3.1a"><msub id="S3.SS2.p1.6.m3.1.1" xref="S3.SS2.p1.6.m3.1.1.cmml"><mi id="S3.SS2.p1.6.m3.1.1.2" xref="S3.SS2.p1.6.m3.1.1.2.cmml">d</mi><mi id="S3.SS2.p1.6.m3.1.1.3" xref="S3.SS2.p1.6.m3.1.1.3.cmml">h</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.6.m3.1b"><apply id="S3.SS2.p1.6.m3.1.1.cmml" xref="S3.SS2.p1.6.m3.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.6.m3.1.1.1.cmml" xref="S3.SS2.p1.6.m3.1.1">subscript</csymbol><ci id="S3.SS2.p1.6.m3.1.1.2.cmml" xref="S3.SS2.p1.6.m3.1.1.2">𝑑</ci><ci id="S3.SS2.p1.6.m3.1.1.3.cmml" xref="S3.SS2.p1.6.m3.1.1.3">ℎ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.6.m3.1c">d_{h}</annotation></semantics></math> denotes the head dimension, to</p>
<table id="E4" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="E4.m1.32" class="ltx_Math" alttext="Y^{l,h}_{i}=\text{softmax}\Big{(}\frac{X^{l,h}_{i}W^{l,h}_{Q}\cdot\text{concat}\big{(}K^{l,h}_{0},\dots K^{l,h}_{4},X^{l,h}_{[:i]}W^{l,h}_{K}\big{)}^{T}}{\sqrt{d_{h}}}\Big{)}\text{concat}\big{(}V^{l,h}_{0},\dots V^{l,h}_{4},X^{l,h}_{[:i]}W^{l,h}_{V}\big{)}W^{l,h}_{O}" display="block"><semantics id="E4.m1.32a"><mrow id="E4.m1.32.32" xref="E4.m1.32.32.cmml"><msubsup id="E4.m1.32.32.5" xref="E4.m1.32.32.5.cmml"><mi id="E4.m1.32.32.5.2.2" xref="E4.m1.32.32.5.2.2.cmml">Y</mi><mi id="E4.m1.32.32.5.3" xref="E4.m1.32.32.5.3.cmml">i</mi><mrow id="E4.m1.2.2.2.4" xref="E4.m1.2.2.2.3.cmml"><mi id="E4.m1.1.1.1.1" xref="E4.m1.1.1.1.1.cmml">l</mi><mo id="E4.m1.2.2.2.4.1" xref="E4.m1.2.2.2.3.cmml">,</mo><mi id="E4.m1.2.2.2.2" xref="E4.m1.2.2.2.2.cmml">h</mi></mrow></msubsup><mo id="E4.m1.32.32.4" xref="E4.m1.32.32.4.cmml">=</mo><mrow id="E4.m1.32.32.3" xref="E4.m1.32.32.3.cmml"><mtext id="E4.m1.32.32.3.5" xref="E4.m1.32.32.3.5a.cmml">softmax</mtext><mo lspace="0em" rspace="0em" id="E4.m1.32.32.3.4" xref="E4.m1.32.32.3.4.cmml">​</mo><mrow id="E4.m1.32.32.3.6.2" xref="E4.m1.18.18.cmml"><mo maxsize="160%" minsize="160%" id="E4.m1.32.32.3.6.2.1" xref="E4.m1.18.18.cmml">(</mo><mfrac id="E4.m1.18.18" xref="E4.m1.18.18.cmml"><mrow id="E4.m1.18.18.16" xref="E4.m1.18.18.16.cmml"><mrow id="E4.m1.18.18.16.18" xref="E4.m1.18.18.16.18.cmml"><mrow id="E4.m1.18.18.16.18.2" xref="E4.m1.18.18.16.18.2.cmml"><msubsup id="E4.m1.18.18.16.18.2.2" xref="E4.m1.18.18.16.18.2.2.cmml"><mi id="E4.m1.18.18.16.18.2.2.2.2" xref="E4.m1.18.18.16.18.2.2.2.2.cmml">X</mi><mi id="E4.m1.18.18.16.18.2.2.3" xref="E4.m1.18.18.16.18.2.2.3.cmml">i</mi><mrow id="E4.m1.4.4.2.2.2.4" xref="E4.m1.4.4.2.2.2.3.cmml"><mi id="E4.m1.3.3.1.1.1.1" xref="E4.m1.3.3.1.1.1.1.cmml">l</mi><mo id="E4.m1.4.4.2.2.2.4.1" xref="E4.m1.4.4.2.2.2.3.cmml">,</mo><mi id="E4.m1.4.4.2.2.2.2" xref="E4.m1.4.4.2.2.2.2.cmml">h</mi></mrow></msubsup><mo lspace="0em" rspace="0em" id="E4.m1.18.18.16.18.2.1" xref="E4.m1.18.18.16.18.2.1.cmml">​</mo><msubsup id="E4.m1.18.18.16.18.2.3" xref="E4.m1.18.18.16.18.2.3.cmml"><mi id="E4.m1.18.18.16.18.2.3.2.2" xref="E4.m1.18.18.16.18.2.3.2.2.cmml">W</mi><mi id="E4.m1.18.18.16.18.2.3.3" xref="E4.m1.18.18.16.18.2.3.3.cmml">Q</mi><mrow id="E4.m1.6.6.4.4.2.4" xref="E4.m1.6.6.4.4.2.3.cmml"><mi id="E4.m1.5.5.3.3.1.1" xref="E4.m1.5.5.3.3.1.1.cmml">l</mi><mo id="E4.m1.6.6.4.4.2.4.1" xref="E4.m1.6.6.4.4.2.3.cmml">,</mo><mi id="E4.m1.6.6.4.4.2.2" xref="E4.m1.6.6.4.4.2.2.cmml">h</mi></mrow></msubsup></mrow><mo lspace="0.222em" rspace="0.222em" id="E4.m1.18.18.16.18.1" xref="E4.m1.18.18.16.18.1.cmml">⋅</mo><mtext id="E4.m1.18.18.16.18.3" xref="E4.m1.18.18.16.18.3a.cmml">concat</mtext></mrow><mo lspace="0em" rspace="0em" id="E4.m1.18.18.16.17" xref="E4.m1.18.18.16.17.cmml">​</mo><msup id="E4.m1.18.18.16.16" xref="E4.m1.18.18.16.16.cmml"><mrow id="E4.m1.18.18.16.16.3.3" xref="E4.m1.18.18.16.16.3.4.cmml"><mo maxsize="120%" minsize="120%" id="E4.m1.18.18.16.16.3.3.4" xref="E4.m1.18.18.16.16.3.4.cmml">(</mo><msubsup id="E4.m1.16.16.14.14.1.1.1" xref="E4.m1.16.16.14.14.1.1.1.cmml"><mi id="E4.m1.16.16.14.14.1.1.1.2.2" xref="E4.m1.16.16.14.14.1.1.1.2.2.cmml">K</mi><mn id="E4.m1.16.16.14.14.1.1.1.3" xref="E4.m1.16.16.14.14.1.1.1.3.cmml">0</mn><mrow id="E4.m1.8.8.6.6.2.4" xref="E4.m1.8.8.6.6.2.3.cmml"><mi id="E4.m1.7.7.5.5.1.1" xref="E4.m1.7.7.5.5.1.1.cmml">l</mi><mo id="E4.m1.8.8.6.6.2.4.1" xref="E4.m1.8.8.6.6.2.3.cmml">,</mo><mi id="E4.m1.8.8.6.6.2.2" xref="E4.m1.8.8.6.6.2.2.cmml">h</mi></mrow></msubsup><mo id="E4.m1.18.18.16.16.3.3.5" xref="E4.m1.18.18.16.16.3.4.cmml">,</mo><mrow id="E4.m1.17.17.15.15.2.2.2" xref="E4.m1.17.17.15.15.2.2.2.cmml"><mi mathvariant="normal" id="E4.m1.17.17.15.15.2.2.2.2" xref="E4.m1.17.17.15.15.2.2.2.2.cmml">…</mi><mo lspace="0em" rspace="0em" id="E4.m1.17.17.15.15.2.2.2.1" xref="E4.m1.17.17.15.15.2.2.2.1.cmml">​</mo><msubsup id="E4.m1.17.17.15.15.2.2.2.3" xref="E4.m1.17.17.15.15.2.2.2.3.cmml"><mi id="E4.m1.17.17.15.15.2.2.2.3.2.2" xref="E4.m1.17.17.15.15.2.2.2.3.2.2.cmml">K</mi><mn id="E4.m1.17.17.15.15.2.2.2.3.3" xref="E4.m1.17.17.15.15.2.2.2.3.3.cmml">4</mn><mrow id="E4.m1.10.10.8.8.2.4" xref="E4.m1.10.10.8.8.2.3.cmml"><mi id="E4.m1.9.9.7.7.1.1" xref="E4.m1.9.9.7.7.1.1.cmml">l</mi><mo id="E4.m1.10.10.8.8.2.4.1" xref="E4.m1.10.10.8.8.2.3.cmml">,</mo><mi id="E4.m1.10.10.8.8.2.2" xref="E4.m1.10.10.8.8.2.2.cmml">h</mi></mrow></msubsup></mrow><mo id="E4.m1.18.18.16.16.3.3.6" xref="E4.m1.18.18.16.16.3.4.cmml">,</mo><mrow id="E4.m1.18.18.16.16.3.3.3" xref="E4.m1.18.18.16.16.3.3.3.cmml"><msubsup id="E4.m1.18.18.16.16.3.3.3.2" xref="E4.m1.18.18.16.16.3.3.3.2.cmml"><mi id="E4.m1.18.18.16.16.3.3.3.2.2.2" xref="E4.m1.18.18.16.16.3.3.3.2.2.2.cmml">X</mi><mrow id="E4.m1.13.13.11.11.1.1" xref="E4.m1.13.13.11.11.1.2.cmml"><mo stretchy="false" id="E4.m1.13.13.11.11.1.1.2" xref="E4.m1.13.13.11.11.1.2.1.cmml">[</mo><mrow id="E4.m1.13.13.11.11.1.1.1" xref="E4.m1.13.13.11.11.1.1.1.cmml"><mi id="E4.m1.13.13.11.11.1.1.1.2" xref="E4.m1.13.13.11.11.1.1.1.2.cmml"></mi><mo lspace="0.278em" rspace="0.278em" id="E4.m1.13.13.11.11.1.1.1.1" xref="E4.m1.13.13.11.11.1.1.1.1.cmml">:</mo><mi id="E4.m1.13.13.11.11.1.1.1.3" xref="E4.m1.13.13.11.11.1.1.1.3.cmml">i</mi></mrow><mo stretchy="false" id="E4.m1.13.13.11.11.1.1.3" xref="E4.m1.13.13.11.11.1.2.1.cmml">]</mo></mrow><mrow id="E4.m1.12.12.10.10.2.4" xref="E4.m1.12.12.10.10.2.3.cmml"><mi id="E4.m1.11.11.9.9.1.1" xref="E4.m1.11.11.9.9.1.1.cmml">l</mi><mo id="E4.m1.12.12.10.10.2.4.1" xref="E4.m1.12.12.10.10.2.3.cmml">,</mo><mi id="E4.m1.12.12.10.10.2.2" xref="E4.m1.12.12.10.10.2.2.cmml">h</mi></mrow></msubsup><mo lspace="0em" rspace="0em" id="E4.m1.18.18.16.16.3.3.3.1" xref="E4.m1.18.18.16.16.3.3.3.1.cmml">​</mo><msubsup id="E4.m1.18.18.16.16.3.3.3.3" xref="E4.m1.18.18.16.16.3.3.3.3.cmml"><mi id="E4.m1.18.18.16.16.3.3.3.3.2.2" xref="E4.m1.18.18.16.16.3.3.3.3.2.2.cmml">W</mi><mi id="E4.m1.18.18.16.16.3.3.3.3.3" xref="E4.m1.18.18.16.16.3.3.3.3.3.cmml">K</mi><mrow id="E4.m1.15.15.13.13.2.4" xref="E4.m1.15.15.13.13.2.3.cmml"><mi id="E4.m1.14.14.12.12.1.1" xref="E4.m1.14.14.12.12.1.1.cmml">l</mi><mo id="E4.m1.15.15.13.13.2.4.1" xref="E4.m1.15.15.13.13.2.3.cmml">,</mo><mi id="E4.m1.15.15.13.13.2.2" xref="E4.m1.15.15.13.13.2.2.cmml">h</mi></mrow></msubsup></mrow><mo maxsize="120%" minsize="120%" id="E4.m1.18.18.16.16.3.3.7" xref="E4.m1.18.18.16.16.3.4.cmml">)</mo></mrow><mi id="E4.m1.18.18.16.16.5" xref="E4.m1.18.18.16.16.5.cmml">T</mi></msup></mrow><msqrt id="E4.m1.18.18.18" xref="E4.m1.18.18.18.cmml"><msub id="E4.m1.18.18.18.2" xref="E4.m1.18.18.18.2.cmml"><mi id="E4.m1.18.18.18.2.2" xref="E4.m1.18.18.18.2.2.cmml">d</mi><mi id="E4.m1.18.18.18.2.3" xref="E4.m1.18.18.18.2.3.cmml">h</mi></msub></msqrt></mfrac><mo maxsize="160%" minsize="160%" id="E4.m1.32.32.3.6.2.2" xref="E4.m1.18.18.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="E4.m1.32.32.3.4a" xref="E4.m1.32.32.3.4.cmml">​</mo><mtext id="E4.m1.32.32.3.7" xref="E4.m1.32.32.3.7a.cmml">concat</mtext><mo lspace="0em" rspace="0em" id="E4.m1.32.32.3.4b" xref="E4.m1.32.32.3.4.cmml">​</mo><mrow id="E4.m1.32.32.3.3.3" xref="E4.m1.32.32.3.3.4.cmml"><mo maxsize="120%" minsize="120%" id="E4.m1.32.32.3.3.3.4" xref="E4.m1.32.32.3.3.4.cmml">(</mo><msubsup id="E4.m1.30.30.1.1.1.1" xref="E4.m1.30.30.1.1.1.1.cmml"><mi id="E4.m1.30.30.1.1.1.1.2.2" xref="E4.m1.30.30.1.1.1.1.2.2.cmml">V</mi><mn id="E4.m1.30.30.1.1.1.1.3" xref="E4.m1.30.30.1.1.1.1.3.cmml">0</mn><mrow id="E4.m1.20.20.2.4" xref="E4.m1.20.20.2.3.cmml"><mi id="E4.m1.19.19.1.1" xref="E4.m1.19.19.1.1.cmml">l</mi><mo id="E4.m1.20.20.2.4.1" xref="E4.m1.20.20.2.3.cmml">,</mo><mi id="E4.m1.20.20.2.2" xref="E4.m1.20.20.2.2.cmml">h</mi></mrow></msubsup><mo id="E4.m1.32.32.3.3.3.5" xref="E4.m1.32.32.3.3.4.cmml">,</mo><mrow id="E4.m1.31.31.2.2.2.2" xref="E4.m1.31.31.2.2.2.2.cmml"><mi mathvariant="normal" id="E4.m1.31.31.2.2.2.2.2" xref="E4.m1.31.31.2.2.2.2.2.cmml">…</mi><mo lspace="0em" rspace="0em" id="E4.m1.31.31.2.2.2.2.1" xref="E4.m1.31.31.2.2.2.2.1.cmml">​</mo><msubsup id="E4.m1.31.31.2.2.2.2.3" xref="E4.m1.31.31.2.2.2.2.3.cmml"><mi id="E4.m1.31.31.2.2.2.2.3.2.2" xref="E4.m1.31.31.2.2.2.2.3.2.2.cmml">V</mi><mn id="E4.m1.31.31.2.2.2.2.3.3" xref="E4.m1.31.31.2.2.2.2.3.3.cmml">4</mn><mrow id="E4.m1.22.22.2.4" xref="E4.m1.22.22.2.3.cmml"><mi id="E4.m1.21.21.1.1" xref="E4.m1.21.21.1.1.cmml">l</mi><mo id="E4.m1.22.22.2.4.1" xref="E4.m1.22.22.2.3.cmml">,</mo><mi id="E4.m1.22.22.2.2" xref="E4.m1.22.22.2.2.cmml">h</mi></mrow></msubsup></mrow><mo id="E4.m1.32.32.3.3.3.6" xref="E4.m1.32.32.3.3.4.cmml">,</mo><mrow id="E4.m1.32.32.3.3.3.3" xref="E4.m1.32.32.3.3.3.3.cmml"><msubsup id="E4.m1.32.32.3.3.3.3.2" xref="E4.m1.32.32.3.3.3.3.2.cmml"><mi id="E4.m1.32.32.3.3.3.3.2.2.2" xref="E4.m1.32.32.3.3.3.3.2.2.2.cmml">X</mi><mrow id="E4.m1.25.25.1.1" xref="E4.m1.25.25.1.2.cmml"><mo stretchy="false" id="E4.m1.25.25.1.1.2" xref="E4.m1.25.25.1.2.1.cmml">[</mo><mrow id="E4.m1.25.25.1.1.1" xref="E4.m1.25.25.1.1.1.cmml"><mi id="E4.m1.25.25.1.1.1.2" xref="E4.m1.25.25.1.1.1.2.cmml"></mi><mo lspace="0.278em" rspace="0.278em" id="E4.m1.25.25.1.1.1.1" xref="E4.m1.25.25.1.1.1.1.cmml">:</mo><mi id="E4.m1.25.25.1.1.1.3" xref="E4.m1.25.25.1.1.1.3.cmml">i</mi></mrow><mo stretchy="false" id="E4.m1.25.25.1.1.3" xref="E4.m1.25.25.1.2.1.cmml">]</mo></mrow><mrow id="E4.m1.24.24.2.4" xref="E4.m1.24.24.2.3.cmml"><mi id="E4.m1.23.23.1.1" xref="E4.m1.23.23.1.1.cmml">l</mi><mo id="E4.m1.24.24.2.4.1" xref="E4.m1.24.24.2.3.cmml">,</mo><mi id="E4.m1.24.24.2.2" xref="E4.m1.24.24.2.2.cmml">h</mi></mrow></msubsup><mo lspace="0em" rspace="0em" id="E4.m1.32.32.3.3.3.3.1" xref="E4.m1.32.32.3.3.3.3.1.cmml">​</mo><msubsup id="E4.m1.32.32.3.3.3.3.3" xref="E4.m1.32.32.3.3.3.3.3.cmml"><mi id="E4.m1.32.32.3.3.3.3.3.2.2" xref="E4.m1.32.32.3.3.3.3.3.2.2.cmml">W</mi><mi id="E4.m1.32.32.3.3.3.3.3.3" xref="E4.m1.32.32.3.3.3.3.3.3.cmml">V</mi><mrow id="E4.m1.27.27.2.4" xref="E4.m1.27.27.2.3.cmml"><mi id="E4.m1.26.26.1.1" xref="E4.m1.26.26.1.1.cmml">l</mi><mo id="E4.m1.27.27.2.4.1" xref="E4.m1.27.27.2.3.cmml">,</mo><mi id="E4.m1.27.27.2.2" xref="E4.m1.27.27.2.2.cmml">h</mi></mrow></msubsup></mrow><mo maxsize="120%" minsize="120%" id="E4.m1.32.32.3.3.3.7" xref="E4.m1.32.32.3.3.4.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="E4.m1.32.32.3.4c" xref="E4.m1.32.32.3.4.cmml">​</mo><msubsup id="E4.m1.32.32.3.8" xref="E4.m1.32.32.3.8.cmml"><mi id="E4.m1.32.32.3.8.2.2" xref="E4.m1.32.32.3.8.2.2.cmml">W</mi><mi id="E4.m1.32.32.3.8.3" xref="E4.m1.32.32.3.8.3.cmml">O</mi><mrow id="E4.m1.29.29.2.4" xref="E4.m1.29.29.2.3.cmml"><mi id="E4.m1.28.28.1.1" xref="E4.m1.28.28.1.1.cmml">l</mi><mo id="E4.m1.29.29.2.4.1" xref="E4.m1.29.29.2.3.cmml">,</mo><mi id="E4.m1.29.29.2.2" xref="E4.m1.29.29.2.2.cmml">h</mi></mrow></msubsup></mrow></mrow><annotation-xml encoding="MathML-Content" id="E4.m1.32b"><apply id="E4.m1.32.32.cmml" xref="E4.m1.32.32"><eq id="E4.m1.32.32.4.cmml" xref="E4.m1.32.32.4"></eq><apply id="E4.m1.32.32.5.cmml" xref="E4.m1.32.32.5"><csymbol cd="ambiguous" id="E4.m1.32.32.5.1.cmml" xref="E4.m1.32.32.5">subscript</csymbol><apply id="E4.m1.32.32.5.2.cmml" xref="E4.m1.32.32.5"><csymbol cd="ambiguous" id="E4.m1.32.32.5.2.1.cmml" xref="E4.m1.32.32.5">superscript</csymbol><ci id="E4.m1.32.32.5.2.2.cmml" xref="E4.m1.32.32.5.2.2">𝑌</ci><list id="E4.m1.2.2.2.3.cmml" xref="E4.m1.2.2.2.4"><ci id="E4.m1.1.1.1.1.cmml" xref="E4.m1.1.1.1.1">𝑙</ci><ci id="E4.m1.2.2.2.2.cmml" xref="E4.m1.2.2.2.2">ℎ</ci></list></apply><ci id="E4.m1.32.32.5.3.cmml" xref="E4.m1.32.32.5.3">𝑖</ci></apply><apply id="E4.m1.32.32.3.cmml" xref="E4.m1.32.32.3"><times id="E4.m1.32.32.3.4.cmml" xref="E4.m1.32.32.3.4"></times><ci id="E4.m1.32.32.3.5a.cmml" xref="E4.m1.32.32.3.5"><mtext id="E4.m1.32.32.3.5.cmml" xref="E4.m1.32.32.3.5">softmax</mtext></ci><apply id="E4.m1.18.18.cmml" xref="E4.m1.32.32.3.6.2"><divide id="E4.m1.18.18.17.cmml" xref="E4.m1.32.32.3.6.2"></divide><apply id="E4.m1.18.18.16.cmml" xref="E4.m1.18.18.16"><times id="E4.m1.18.18.16.17.cmml" xref="E4.m1.18.18.16.17"></times><apply id="E4.m1.18.18.16.18.cmml" xref="E4.m1.18.18.16.18"><ci id="E4.m1.18.18.16.18.1.cmml" xref="E4.m1.18.18.16.18.1">⋅</ci><apply id="E4.m1.18.18.16.18.2.cmml" xref="E4.m1.18.18.16.18.2"><times id="E4.m1.18.18.16.18.2.1.cmml" xref="E4.m1.18.18.16.18.2.1"></times><apply id="E4.m1.18.18.16.18.2.2.cmml" xref="E4.m1.18.18.16.18.2.2"><csymbol cd="ambiguous" id="E4.m1.18.18.16.18.2.2.1.cmml" xref="E4.m1.18.18.16.18.2.2">subscript</csymbol><apply id="E4.m1.18.18.16.18.2.2.2.cmml" xref="E4.m1.18.18.16.18.2.2"><csymbol cd="ambiguous" id="E4.m1.18.18.16.18.2.2.2.1.cmml" xref="E4.m1.18.18.16.18.2.2">superscript</csymbol><ci id="E4.m1.18.18.16.18.2.2.2.2.cmml" xref="E4.m1.18.18.16.18.2.2.2.2">𝑋</ci><list id="E4.m1.4.4.2.2.2.3.cmml" xref="E4.m1.4.4.2.2.2.4"><ci id="E4.m1.3.3.1.1.1.1.cmml" xref="E4.m1.3.3.1.1.1.1">𝑙</ci><ci id="E4.m1.4.4.2.2.2.2.cmml" xref="E4.m1.4.4.2.2.2.2">ℎ</ci></list></apply><ci id="E4.m1.18.18.16.18.2.2.3.cmml" xref="E4.m1.18.18.16.18.2.2.3">𝑖</ci></apply><apply id="E4.m1.18.18.16.18.2.3.cmml" xref="E4.m1.18.18.16.18.2.3"><csymbol cd="ambiguous" id="E4.m1.18.18.16.18.2.3.1.cmml" xref="E4.m1.18.18.16.18.2.3">subscript</csymbol><apply id="E4.m1.18.18.16.18.2.3.2.cmml" xref="E4.m1.18.18.16.18.2.3"><csymbol cd="ambiguous" id="E4.m1.18.18.16.18.2.3.2.1.cmml" xref="E4.m1.18.18.16.18.2.3">superscript</csymbol><ci id="E4.m1.18.18.16.18.2.3.2.2.cmml" xref="E4.m1.18.18.16.18.2.3.2.2">𝑊</ci><list id="E4.m1.6.6.4.4.2.3.cmml" xref="E4.m1.6.6.4.4.2.4"><ci id="E4.m1.5.5.3.3.1.1.cmml" xref="E4.m1.5.5.3.3.1.1">𝑙</ci><ci id="E4.m1.6.6.4.4.2.2.cmml" xref="E4.m1.6.6.4.4.2.2">ℎ</ci></list></apply><ci id="E4.m1.18.18.16.18.2.3.3.cmml" xref="E4.m1.18.18.16.18.2.3.3">𝑄</ci></apply></apply><ci id="E4.m1.18.18.16.18.3a.cmml" xref="E4.m1.18.18.16.18.3"><mtext id="E4.m1.18.18.16.18.3.cmml" xref="E4.m1.18.18.16.18.3">concat</mtext></ci></apply><apply id="E4.m1.18.18.16.16.cmml" xref="E4.m1.18.18.16.16"><csymbol cd="ambiguous" id="E4.m1.18.18.16.16.4.cmml" xref="E4.m1.18.18.16.16">superscript</csymbol><vector id="E4.m1.18.18.16.16.3.4.cmml" xref="E4.m1.18.18.16.16.3.3"><apply id="E4.m1.16.16.14.14.1.1.1.cmml" xref="E4.m1.16.16.14.14.1.1.1"><csymbol cd="ambiguous" id="E4.m1.16.16.14.14.1.1.1.1.cmml" xref="E4.m1.16.16.14.14.1.1.1">subscript</csymbol><apply id="E4.m1.16.16.14.14.1.1.1.2.cmml" xref="E4.m1.16.16.14.14.1.1.1"><csymbol cd="ambiguous" id="E4.m1.16.16.14.14.1.1.1.2.1.cmml" xref="E4.m1.16.16.14.14.1.1.1">superscript</csymbol><ci id="E4.m1.16.16.14.14.1.1.1.2.2.cmml" xref="E4.m1.16.16.14.14.1.1.1.2.2">𝐾</ci><list id="E4.m1.8.8.6.6.2.3.cmml" xref="E4.m1.8.8.6.6.2.4"><ci id="E4.m1.7.7.5.5.1.1.cmml" xref="E4.m1.7.7.5.5.1.1">𝑙</ci><ci id="E4.m1.8.8.6.6.2.2.cmml" xref="E4.m1.8.8.6.6.2.2">ℎ</ci></list></apply><cn type="integer" id="E4.m1.16.16.14.14.1.1.1.3.cmml" xref="E4.m1.16.16.14.14.1.1.1.3">0</cn></apply><apply id="E4.m1.17.17.15.15.2.2.2.cmml" xref="E4.m1.17.17.15.15.2.2.2"><times id="E4.m1.17.17.15.15.2.2.2.1.cmml" xref="E4.m1.17.17.15.15.2.2.2.1"></times><ci id="E4.m1.17.17.15.15.2.2.2.2.cmml" xref="E4.m1.17.17.15.15.2.2.2.2">…</ci><apply id="E4.m1.17.17.15.15.2.2.2.3.cmml" xref="E4.m1.17.17.15.15.2.2.2.3"><csymbol cd="ambiguous" id="E4.m1.17.17.15.15.2.2.2.3.1.cmml" xref="E4.m1.17.17.15.15.2.2.2.3">subscript</csymbol><apply id="E4.m1.17.17.15.15.2.2.2.3.2.cmml" xref="E4.m1.17.17.15.15.2.2.2.3"><csymbol cd="ambiguous" id="E4.m1.17.17.15.15.2.2.2.3.2.1.cmml" xref="E4.m1.17.17.15.15.2.2.2.3">superscript</csymbol><ci id="E4.m1.17.17.15.15.2.2.2.3.2.2.cmml" xref="E4.m1.17.17.15.15.2.2.2.3.2.2">𝐾</ci><list id="E4.m1.10.10.8.8.2.3.cmml" xref="E4.m1.10.10.8.8.2.4"><ci id="E4.m1.9.9.7.7.1.1.cmml" xref="E4.m1.9.9.7.7.1.1">𝑙</ci><ci id="E4.m1.10.10.8.8.2.2.cmml" xref="E4.m1.10.10.8.8.2.2">ℎ</ci></list></apply><cn type="integer" id="E4.m1.17.17.15.15.2.2.2.3.3.cmml" xref="E4.m1.17.17.15.15.2.2.2.3.3">4</cn></apply></apply><apply id="E4.m1.18.18.16.16.3.3.3.cmml" xref="E4.m1.18.18.16.16.3.3.3"><times id="E4.m1.18.18.16.16.3.3.3.1.cmml" xref="E4.m1.18.18.16.16.3.3.3.1"></times><apply id="E4.m1.18.18.16.16.3.3.3.2.cmml" xref="E4.m1.18.18.16.16.3.3.3.2"><csymbol cd="ambiguous" id="E4.m1.18.18.16.16.3.3.3.2.1.cmml" xref="E4.m1.18.18.16.16.3.3.3.2">subscript</csymbol><apply id="E4.m1.18.18.16.16.3.3.3.2.2.cmml" xref="E4.m1.18.18.16.16.3.3.3.2"><csymbol cd="ambiguous" id="E4.m1.18.18.16.16.3.3.3.2.2.1.cmml" xref="E4.m1.18.18.16.16.3.3.3.2">superscript</csymbol><ci id="E4.m1.18.18.16.16.3.3.3.2.2.2.cmml" xref="E4.m1.18.18.16.16.3.3.3.2.2.2">𝑋</ci><list id="E4.m1.12.12.10.10.2.3.cmml" xref="E4.m1.12.12.10.10.2.4"><ci id="E4.m1.11.11.9.9.1.1.cmml" xref="E4.m1.11.11.9.9.1.1">𝑙</ci><ci id="E4.m1.12.12.10.10.2.2.cmml" xref="E4.m1.12.12.10.10.2.2">ℎ</ci></list></apply><apply id="E4.m1.13.13.11.11.1.2.cmml" xref="E4.m1.13.13.11.11.1.1"><csymbol cd="latexml" id="E4.m1.13.13.11.11.1.2.1.cmml" xref="E4.m1.13.13.11.11.1.1.2">delimited-[]</csymbol><apply id="E4.m1.13.13.11.11.1.1.1.cmml" xref="E4.m1.13.13.11.11.1.1.1"><ci id="E4.m1.13.13.11.11.1.1.1.1.cmml" xref="E4.m1.13.13.11.11.1.1.1.1">:</ci><csymbol cd="latexml" id="E4.m1.13.13.11.11.1.1.1.2.cmml" xref="E4.m1.13.13.11.11.1.1.1.2">absent</csymbol><ci id="E4.m1.13.13.11.11.1.1.1.3.cmml" xref="E4.m1.13.13.11.11.1.1.1.3">𝑖</ci></apply></apply></apply><apply id="E4.m1.18.18.16.16.3.3.3.3.cmml" xref="E4.m1.18.18.16.16.3.3.3.3"><csymbol cd="ambiguous" id="E4.m1.18.18.16.16.3.3.3.3.1.cmml" xref="E4.m1.18.18.16.16.3.3.3.3">subscript</csymbol><apply id="E4.m1.18.18.16.16.3.3.3.3.2.cmml" xref="E4.m1.18.18.16.16.3.3.3.3"><csymbol cd="ambiguous" id="E4.m1.18.18.16.16.3.3.3.3.2.1.cmml" xref="E4.m1.18.18.16.16.3.3.3.3">superscript</csymbol><ci id="E4.m1.18.18.16.16.3.3.3.3.2.2.cmml" xref="E4.m1.18.18.16.16.3.3.3.3.2.2">𝑊</ci><list id="E4.m1.15.15.13.13.2.3.cmml" xref="E4.m1.15.15.13.13.2.4"><ci id="E4.m1.14.14.12.12.1.1.cmml" xref="E4.m1.14.14.12.12.1.1">𝑙</ci><ci id="E4.m1.15.15.13.13.2.2.cmml" xref="E4.m1.15.15.13.13.2.2">ℎ</ci></list></apply><ci id="E4.m1.18.18.16.16.3.3.3.3.3.cmml" xref="E4.m1.18.18.16.16.3.3.3.3.3">𝐾</ci></apply></apply></vector><ci id="E4.m1.18.18.16.16.5.cmml" xref="E4.m1.18.18.16.16.5">𝑇</ci></apply></apply><apply id="E4.m1.18.18.18.cmml" xref="E4.m1.18.18.18"><root id="E4.m1.18.18.18a.cmml" xref="E4.m1.18.18.18"></root><apply id="E4.m1.18.18.18.2.cmml" xref="E4.m1.18.18.18.2"><csymbol cd="ambiguous" id="E4.m1.18.18.18.2.1.cmml" xref="E4.m1.18.18.18.2">subscript</csymbol><ci id="E4.m1.18.18.18.2.2.cmml" xref="E4.m1.18.18.18.2.2">𝑑</ci><ci id="E4.m1.18.18.18.2.3.cmml" xref="E4.m1.18.18.18.2.3">ℎ</ci></apply></apply></apply><ci id="E4.m1.32.32.3.7a.cmml" xref="E4.m1.32.32.3.7"><mtext id="E4.m1.32.32.3.7.cmml" xref="E4.m1.32.32.3.7">concat</mtext></ci><vector id="E4.m1.32.32.3.3.4.cmml" xref="E4.m1.32.32.3.3.3"><apply id="E4.m1.30.30.1.1.1.1.cmml" xref="E4.m1.30.30.1.1.1.1"><csymbol cd="ambiguous" id="E4.m1.30.30.1.1.1.1.1.cmml" xref="E4.m1.30.30.1.1.1.1">subscript</csymbol><apply id="E4.m1.30.30.1.1.1.1.2.cmml" xref="E4.m1.30.30.1.1.1.1"><csymbol cd="ambiguous" id="E4.m1.30.30.1.1.1.1.2.1.cmml" xref="E4.m1.30.30.1.1.1.1">superscript</csymbol><ci id="E4.m1.30.30.1.1.1.1.2.2.cmml" xref="E4.m1.30.30.1.1.1.1.2.2">𝑉</ci><list id="E4.m1.20.20.2.3.cmml" xref="E4.m1.20.20.2.4"><ci id="E4.m1.19.19.1.1.cmml" xref="E4.m1.19.19.1.1">𝑙</ci><ci id="E4.m1.20.20.2.2.cmml" xref="E4.m1.20.20.2.2">ℎ</ci></list></apply><cn type="integer" id="E4.m1.30.30.1.1.1.1.3.cmml" xref="E4.m1.30.30.1.1.1.1.3">0</cn></apply><apply id="E4.m1.31.31.2.2.2.2.cmml" xref="E4.m1.31.31.2.2.2.2"><times id="E4.m1.31.31.2.2.2.2.1.cmml" xref="E4.m1.31.31.2.2.2.2.1"></times><ci id="E4.m1.31.31.2.2.2.2.2.cmml" xref="E4.m1.31.31.2.2.2.2.2">…</ci><apply id="E4.m1.31.31.2.2.2.2.3.cmml" xref="E4.m1.31.31.2.2.2.2.3"><csymbol cd="ambiguous" id="E4.m1.31.31.2.2.2.2.3.1.cmml" xref="E4.m1.31.31.2.2.2.2.3">subscript</csymbol><apply id="E4.m1.31.31.2.2.2.2.3.2.cmml" xref="E4.m1.31.31.2.2.2.2.3"><csymbol cd="ambiguous" id="E4.m1.31.31.2.2.2.2.3.2.1.cmml" xref="E4.m1.31.31.2.2.2.2.3">superscript</csymbol><ci id="E4.m1.31.31.2.2.2.2.3.2.2.cmml" xref="E4.m1.31.31.2.2.2.2.3.2.2">𝑉</ci><list id="E4.m1.22.22.2.3.cmml" xref="E4.m1.22.22.2.4"><ci id="E4.m1.21.21.1.1.cmml" xref="E4.m1.21.21.1.1">𝑙</ci><ci id="E4.m1.22.22.2.2.cmml" xref="E4.m1.22.22.2.2">ℎ</ci></list></apply><cn type="integer" id="E4.m1.31.31.2.2.2.2.3.3.cmml" xref="E4.m1.31.31.2.2.2.2.3.3">4</cn></apply></apply><apply id="E4.m1.32.32.3.3.3.3.cmml" xref="E4.m1.32.32.3.3.3.3"><times id="E4.m1.32.32.3.3.3.3.1.cmml" xref="E4.m1.32.32.3.3.3.3.1"></times><apply id="E4.m1.32.32.3.3.3.3.2.cmml" xref="E4.m1.32.32.3.3.3.3.2"><csymbol cd="ambiguous" id="E4.m1.32.32.3.3.3.3.2.1.cmml" xref="E4.m1.32.32.3.3.3.3.2">subscript</csymbol><apply id="E4.m1.32.32.3.3.3.3.2.2.cmml" xref="E4.m1.32.32.3.3.3.3.2"><csymbol cd="ambiguous" id="E4.m1.32.32.3.3.3.3.2.2.1.cmml" xref="E4.m1.32.32.3.3.3.3.2">superscript</csymbol><ci id="E4.m1.32.32.3.3.3.3.2.2.2.cmml" xref="E4.m1.32.32.3.3.3.3.2.2.2">𝑋</ci><list id="E4.m1.24.24.2.3.cmml" xref="E4.m1.24.24.2.4"><ci id="E4.m1.23.23.1.1.cmml" xref="E4.m1.23.23.1.1">𝑙</ci><ci id="E4.m1.24.24.2.2.cmml" xref="E4.m1.24.24.2.2">ℎ</ci></list></apply><apply id="E4.m1.25.25.1.2.cmml" xref="E4.m1.25.25.1.1"><csymbol cd="latexml" id="E4.m1.25.25.1.2.1.cmml" xref="E4.m1.25.25.1.1.2">delimited-[]</csymbol><apply id="E4.m1.25.25.1.1.1.cmml" xref="E4.m1.25.25.1.1.1"><ci id="E4.m1.25.25.1.1.1.1.cmml" xref="E4.m1.25.25.1.1.1.1">:</ci><csymbol cd="latexml" id="E4.m1.25.25.1.1.1.2.cmml" xref="E4.m1.25.25.1.1.1.2">absent</csymbol><ci id="E4.m1.25.25.1.1.1.3.cmml" xref="E4.m1.25.25.1.1.1.3">𝑖</ci></apply></apply></apply><apply id="E4.m1.32.32.3.3.3.3.3.cmml" xref="E4.m1.32.32.3.3.3.3.3"><csymbol cd="ambiguous" id="E4.m1.32.32.3.3.3.3.3.1.cmml" xref="E4.m1.32.32.3.3.3.3.3">subscript</csymbol><apply id="E4.m1.32.32.3.3.3.3.3.2.cmml" xref="E4.m1.32.32.3.3.3.3.3"><csymbol cd="ambiguous" id="E4.m1.32.32.3.3.3.3.3.2.1.cmml" xref="E4.m1.32.32.3.3.3.3.3">superscript</csymbol><ci id="E4.m1.32.32.3.3.3.3.3.2.2.cmml" xref="E4.m1.32.32.3.3.3.3.3.2.2">𝑊</ci><list id="E4.m1.27.27.2.3.cmml" xref="E4.m1.27.27.2.4"><ci id="E4.m1.26.26.1.1.cmml" xref="E4.m1.26.26.1.1">𝑙</ci><ci id="E4.m1.27.27.2.2.cmml" xref="E4.m1.27.27.2.2">ℎ</ci></list></apply><ci id="E4.m1.32.32.3.3.3.3.3.3.cmml" xref="E4.m1.32.32.3.3.3.3.3.3">𝑉</ci></apply></apply></vector><apply id="E4.m1.32.32.3.8.cmml" xref="E4.m1.32.32.3.8"><csymbol cd="ambiguous" id="E4.m1.32.32.3.8.1.cmml" xref="E4.m1.32.32.3.8">subscript</csymbol><apply id="E4.m1.32.32.3.8.2.cmml" xref="E4.m1.32.32.3.8"><csymbol cd="ambiguous" id="E4.m1.32.32.3.8.2.1.cmml" xref="E4.m1.32.32.3.8">superscript</csymbol><ci id="E4.m1.32.32.3.8.2.2.cmml" xref="E4.m1.32.32.3.8.2.2">𝑊</ci><list id="E4.m1.29.29.2.3.cmml" xref="E4.m1.29.29.2.4"><ci id="E4.m1.28.28.1.1.cmml" xref="E4.m1.28.28.1.1">𝑙</ci><ci id="E4.m1.29.29.2.2.cmml" xref="E4.m1.29.29.2.2">ℎ</ci></list></apply><ci id="E4.m1.32.32.3.8.3.cmml" xref="E4.m1.32.32.3.8.3">𝑂</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="E4.m1.32c">Y^{l,h}_{i}=\text{softmax}\Big{(}\frac{X^{l,h}_{i}W^{l,h}_{Q}\cdot\text{concat}\big{(}K^{l,h}_{0},\dots K^{l,h}_{4},X^{l,h}_{[:i]}W^{l,h}_{K}\big{)}^{T}}{\sqrt{d_{h}}}\Big{)}\text{concat}\big{(}V^{l,h}_{0},\dots V^{l,h}_{4},X^{l,h}_{[:i]}W^{l,h}_{V}\big{)}W^{l,h}_{O}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(4)</span></td>
</tr></tbody>
</table>
<p id="S3.SS2.p1.7" class="ltx_p">where each <math id="S3.SS2.p1.7.m1.2" class="ltx_Math" alttext="(K_{j},V_{j})" display="inline"><semantics id="S3.SS2.p1.7.m1.2a"><mrow id="S3.SS2.p1.7.m1.2.2.2" xref="S3.SS2.p1.7.m1.2.2.3.cmml"><mo stretchy="false" id="S3.SS2.p1.7.m1.2.2.2.3" xref="S3.SS2.p1.7.m1.2.2.3.cmml">(</mo><msub id="S3.SS2.p1.7.m1.1.1.1.1" xref="S3.SS2.p1.7.m1.1.1.1.1.cmml"><mi id="S3.SS2.p1.7.m1.1.1.1.1.2" xref="S3.SS2.p1.7.m1.1.1.1.1.2.cmml">K</mi><mi id="S3.SS2.p1.7.m1.1.1.1.1.3" xref="S3.SS2.p1.7.m1.1.1.1.1.3.cmml">j</mi></msub><mo id="S3.SS2.p1.7.m1.2.2.2.4" xref="S3.SS2.p1.7.m1.2.2.3.cmml">,</mo><msub id="S3.SS2.p1.7.m1.2.2.2.2" xref="S3.SS2.p1.7.m1.2.2.2.2.cmml"><mi id="S3.SS2.p1.7.m1.2.2.2.2.2" xref="S3.SS2.p1.7.m1.2.2.2.2.2.cmml">V</mi><mi id="S3.SS2.p1.7.m1.2.2.2.2.3" xref="S3.SS2.p1.7.m1.2.2.2.2.3.cmml">j</mi></msub><mo stretchy="false" id="S3.SS2.p1.7.m1.2.2.2.5" xref="S3.SS2.p1.7.m1.2.2.3.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.7.m1.2b"><interval closure="open" id="S3.SS2.p1.7.m1.2.2.3.cmml" xref="S3.SS2.p1.7.m1.2.2.2"><apply id="S3.SS2.p1.7.m1.1.1.1.1.cmml" xref="S3.SS2.p1.7.m1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.7.m1.1.1.1.1.1.cmml" xref="S3.SS2.p1.7.m1.1.1.1.1">subscript</csymbol><ci id="S3.SS2.p1.7.m1.1.1.1.1.2.cmml" xref="S3.SS2.p1.7.m1.1.1.1.1.2">𝐾</ci><ci id="S3.SS2.p1.7.m1.1.1.1.1.3.cmml" xref="S3.SS2.p1.7.m1.1.1.1.1.3">𝑗</ci></apply><apply id="S3.SS2.p1.7.m1.2.2.2.2.cmml" xref="S3.SS2.p1.7.m1.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS2.p1.7.m1.2.2.2.2.1.cmml" xref="S3.SS2.p1.7.m1.2.2.2.2">subscript</csymbol><ci id="S3.SS2.p1.7.m1.2.2.2.2.2.cmml" xref="S3.SS2.p1.7.m1.2.2.2.2.2">𝑉</ci><ci id="S3.SS2.p1.7.m1.2.2.2.2.3.cmml" xref="S3.SS2.p1.7.m1.2.2.2.2.3">𝑗</ci></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.7.m1.2c">(K_{j},V_{j})</annotation></semantics></math> denotes the keys and values of an explicit memory.</p>
</div>
<div id="S3.SS2.p2" class="ltx_para">
<p id="S3.SS2.p2.2" class="ltx_p">While the context BOS token is <math id="S3.SS2.p2.1.m1.1" class="ltx_Math" alttext="\langle\text{s}\rangle" display="inline"><semantics id="S3.SS2.p2.1.m1.1a"><mrow id="S3.SS2.p2.1.m1.1.2.2" xref="S3.SS2.p2.1.m1.1.2.1.cmml"><mo stretchy="false" id="S3.SS2.p2.1.m1.1.2.2.1" xref="S3.SS2.p2.1.m1.1.2.1.1.cmml">⟨</mo><mtext id="S3.SS2.p2.1.m1.1.1" xref="S3.SS2.p2.1.m1.1.1a.cmml">s</mtext><mo stretchy="false" id="S3.SS2.p2.1.m1.1.2.2.2" xref="S3.SS2.p2.1.m1.1.2.1.1.cmml">⟩</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.1.m1.1b"><apply id="S3.SS2.p2.1.m1.1.2.1.cmml" xref="S3.SS2.p2.1.m1.1.2.2"><csymbol cd="latexml" id="S3.SS2.p2.1.m1.1.2.1.1.cmml" xref="S3.SS2.p2.1.m1.1.2.2.1">delimited-⟨⟩</csymbol><ci id="S3.SS2.p2.1.m1.1.1a.cmml" xref="S3.SS2.p2.1.m1.1.1"><mtext id="S3.SS2.p2.1.m1.1.1.cmml" xref="S3.SS2.p2.1.m1.1.1">s</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.1.m1.1c">\langle\text{s}\rangle</annotation></semantics></math> as usual,
when encoding each reference we modify the BOS to “<math id="S3.SS2.p2.2.m2.1" class="ltx_Math" alttext="\langle\text{s}\rangle" display="inline"><semantics id="S3.SS2.p2.2.m2.1a"><mrow id="S3.SS2.p2.2.m2.1.2.2" xref="S3.SS2.p2.2.m2.1.2.1.cmml"><mo stretchy="false" id="S3.SS2.p2.2.m2.1.2.2.1" xref="S3.SS2.p2.2.m2.1.2.1.1.cmml">⟨</mo><mtext id="S3.SS2.p2.2.m2.1.1" xref="S3.SS2.p2.2.m2.1.1a.cmml">s</mtext><mo stretchy="false" id="S3.SS2.p2.2.m2.1.2.2.2" xref="S3.SS2.p2.2.m2.1.2.1.1.cmml">⟩</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.2.m2.1b"><apply id="S3.SS2.p2.2.m2.1.2.1.cmml" xref="S3.SS2.p2.2.m2.1.2.2"><csymbol cd="latexml" id="S3.SS2.p2.2.m2.1.2.1.1.cmml" xref="S3.SS2.p2.2.m2.1.2.2.1">delimited-⟨⟩</csymbol><ci id="S3.SS2.p2.2.m2.1.1a.cmml" xref="S3.SS2.p2.2.m2.1.1"><mtext id="S3.SS2.p2.2.m2.1.1.cmml" xref="S3.SS2.p2.2.m2.1.1">s</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.2.m2.1c">\langle\text{s}\rangle</annotation></semantics></math>Reference:” to help the LLM distinguish between encoding normal texts and encoding references.
This modified BOS is also prepended to the context during inference, as illustrated in Figure <a href="#F9" title="Figure 9 ‣ 3.1 | Inference Process ‣ 3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>, while the context BOS token now serves as a separator between the references and context.
Unlike the explicit memories which only appear at a subset of attention heads, this modified BOS is placed at every head at every layer.
The motivation is that since the context BOS can attend to the references, its feature is no longer constant, so the LLM needs the modified BOS to serve as the new constant for all attention heads.</p>
</div>
<div id="S3.SS2.p3" class="ltx_para">
<p id="S3.SS2.p3.1" class="ltx_p">Furthermore, we adopt parallel position encoding for all explicit memories, namely the positions of all their keys lie in the same interval of length 128, as depicted in Figure <a href="#F9" title="Figure 9 ‣ 3.1 | Inference Process ‣ 3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>.
We use the rotary position encoding (RoPE) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib107" title="" class="ltx_ref">107</a>]</cite>.
The token sparsification is applied after RoPE processes the attention keys, so the selected tokens retain their relative positions in the references.
Besides flexibility, one motivation for parallel position is to avoid the “lost in the middle” phenomenon <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib72" title="" class="ltx_ref">72</a>]</cite>, such that if the references are positioned serially, then the ones in the middle are likely to be ignored.
Similarly, token sparsification also helps to alleviate this issue by making the attention more focused on the important tokens.
We note that designs analogous to the parallel position have been used to improve in-context learning <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib96" title="" class="ltx_ref">96</a>]</cite> and long-context modeling <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite>.</p>
</div>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsection" style="font-size:120%;color:#00868B;">3.3 <span id="S3.SS3.1.1" class="ltx_text" style="color:#808080;">| </span>Memory Sparsification and Storage</h3>

<div id="S3.SS3.p1" class="ltx_para">
<p id="S3.SS3.p1.1" class="ltx_p">One of the greatest challenges for explicit memories is that the attention key-values occupy too much space.
They not only demand more disk space, which could be costly, but also occupy GPU memory during inference, which could harm the batch size and thus the throughput of LLM generation.
An intense compression is needed to save space.
The full attention key tensor (or value tensor) for each reference has shape (layers, key-value heads, tokens, head dimension), so we compress all four dimensions.</p>
</div>
<div id="S3.SS3.p2" class="ltx_para">
<p id="S3.SS3.p2.2" class="ltx_p">Regard layers, we only set the first half of the attention layers to be memory layers, i.e. layers that produce and attend to explicit memories (<a href="#E4" title="Equation 4 ‣ 3.2 | Writing and Reading Memory ‣ 3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>), while the second half remain as the usual attention layers.
Note that Remark <a href="#Thmremark1" title="Remark 1. ‣ 2.3 | Memory ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> suggests that it is usually the attention heads in the middle of the LLM that attend to the references.
So it seems that appointing the middle attention layers (e.g. the ones within the <math id="S3.SS3.p2.1.m1.1" class="ltx_Math" alttext="25\%" display="inline"><semantics id="S3.SS3.p2.1.m1.1a"><mrow id="S3.SS3.p2.1.m1.1.1" xref="S3.SS3.p2.1.m1.1.1.cmml"><mn id="S3.SS3.p2.1.m1.1.1.2" xref="S3.SS3.p2.1.m1.1.1.2.cmml">25</mn><mo id="S3.SS3.p2.1.m1.1.1.1" xref="S3.SS3.p2.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.1.m1.1b"><apply id="S3.SS3.p2.1.m1.1.1.cmml" xref="S3.SS3.p2.1.m1.1.1"><csymbol cd="latexml" id="S3.SS3.p2.1.m1.1.1.1.cmml" xref="S3.SS3.p2.1.m1.1.1.1">percent</csymbol><cn type="integer" id="S3.SS3.p2.1.m1.1.1.2.cmml" xref="S3.SS3.p2.1.m1.1.1.2">25</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.1.m1.1c">25\%</annotation></semantics></math> to <math id="S3.SS3.p2.2.m2.1" class="ltx_Math" alttext="75\%" display="inline"><semantics id="S3.SS3.p2.2.m2.1a"><mrow id="S3.SS3.p2.2.m2.1.1" xref="S3.SS3.p2.2.m2.1.1.cmml"><mn id="S3.SS3.p2.2.m2.1.1.2" xref="S3.SS3.p2.2.m2.1.1.2.cmml">75</mn><mo id="S3.SS3.p2.2.m2.1.1.1" xref="S3.SS3.p2.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.2.m2.1b"><apply id="S3.SS3.p2.2.m2.1.1.cmml" xref="S3.SS3.p2.2.m2.1.1"><csymbol cd="latexml" id="S3.SS3.p2.2.m2.1.1.1.cmml" xref="S3.SS3.p2.2.m2.1.1.1">percent</csymbol><cn type="integer" id="S3.SS3.p2.2.m2.1.1.2.cmml" xref="S3.SS3.p2.2.m2.1.1.2">75</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.2.m2.1c">75\%</annotation></semantics></math> depth range) to be memory layers is a more sensible choice.
This heuristic is supported by the observations in <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib122" title="" class="ltx_ref">122</a>, <a href="#bib.bib39" title="" class="ltx_ref">39</a>]</cite> that the attention to the distant context usually takes place in the middle layers.</p>
</div>
<div id="S3.SS3.p3" class="ltx_para">
<p id="S3.SS3.p3.1" class="ltx_p">Regarding heads, we set all key-value heads at each memory layer to be memory heads.
We reduce their amount by grouped query attention (GQA) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib6" title="" class="ltx_ref">6</a>]</cite>, letting each key-value head be shared by multiple query heads, and obtain 20% sparsity (8 versus 40 heads).
It is worth mentioning that, besides GQA and memory layers, another approach is to select a small subset of heads that are most helpful for reading memories, and this selection does not have to be uniform across layer.
We describe several methods for selecting memory heads in Remark <a href="#Thmremark4" title="Remark 4. ‣ 3.3 | Memory Sparsification and Storage ‣ 3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>.</p>
</div>
<div id="S3.SS3.p4" class="ltx_para">
<p id="S3.SS3.p4.2" class="ltx_p">Regarding tokens, we select 8 tokens out of 128 for each key-value head.
We choose a high level of sparsity, since Remark <a href="#Thmremark1" title="Remark 1. ‣ 2.3 | Memory ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> indicates that the attention from the context to the references are expected to be concentrated on very few tokens.
Note that the selected tokens are in general different among heads, so in principle their union could cover a lot of tokens.
For each head <math id="S3.SS3.p4.1.m1.1" class="ltx_Math" alttext="h" display="inline"><semantics id="S3.SS3.p4.1.m1.1a"><mi id="S3.SS3.p4.1.m1.1.1" xref="S3.SS3.p4.1.m1.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p4.1.m1.1b"><ci id="S3.SS3.p4.1.m1.1.1.cmml" xref="S3.SS3.p4.1.m1.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p4.1.m1.1c">h</annotation></semantics></math> at layer <math id="S3.SS3.p4.2.m2.1" class="ltx_Math" alttext="l" display="inline"><semantics id="S3.SS3.p4.2.m2.1a"><mi id="S3.SS3.p4.2.m2.1.1" xref="S3.SS3.p4.2.m2.1.1.cmml">l</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p4.2.m2.1b"><ci id="S3.SS3.p4.2.m2.1.1.cmml" xref="S3.SS3.p4.2.m2.1.1">𝑙</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p4.2.m2.1c">l</annotation></semantics></math>, the selection uses top-8 over the attention weight</p>
<table id="Ex5" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="Ex5.m1.21" class="ltx_Math" alttext="w^{l,h}_{j}=\sum_{i=0}^{127}\tilde{a}^{l,h}_{i,j},\quad\tilde{a}^{l,h}_{i,j}=\text{softmax}_{j}\Big{(}\frac{X^{l,h}_{i}W^{l,h}_{Q}(X^{l,h}_{j}W^{l,h}_{K})^{T}}{\sqrt{d_{h}}}\Big{)}" display="block"><semantics id="Ex5.m1.21a"><mrow id="Ex5.m1.21.21.2" xref="Ex5.m1.21.21.3.cmml"><mrow id="Ex5.m1.20.20.1.1" xref="Ex5.m1.20.20.1.1.cmml"><msubsup id="Ex5.m1.20.20.1.1.2" xref="Ex5.m1.20.20.1.1.2.cmml"><mi id="Ex5.m1.20.20.1.1.2.2.2" xref="Ex5.m1.20.20.1.1.2.2.2.cmml">w</mi><mi id="Ex5.m1.20.20.1.1.2.3" xref="Ex5.m1.20.20.1.1.2.3.cmml">j</mi><mrow id="Ex5.m1.2.2.2.4" xref="Ex5.m1.2.2.2.3.cmml"><mi id="Ex5.m1.1.1.1.1" xref="Ex5.m1.1.1.1.1.cmml">l</mi><mo id="Ex5.m1.2.2.2.4.1" xref="Ex5.m1.2.2.2.3.cmml">,</mo><mi id="Ex5.m1.2.2.2.2" xref="Ex5.m1.2.2.2.2.cmml">h</mi></mrow></msubsup><mo rspace="0.111em" id="Ex5.m1.20.20.1.1.1" xref="Ex5.m1.20.20.1.1.1.cmml">=</mo><mrow id="Ex5.m1.20.20.1.1.3" xref="Ex5.m1.20.20.1.1.3.cmml"><munderover id="Ex5.m1.20.20.1.1.3.1" xref="Ex5.m1.20.20.1.1.3.1.cmml"><mo movablelimits="false" id="Ex5.m1.20.20.1.1.3.1.2.2" xref="Ex5.m1.20.20.1.1.3.1.2.2.cmml">∑</mo><mrow id="Ex5.m1.20.20.1.1.3.1.2.3" xref="Ex5.m1.20.20.1.1.3.1.2.3.cmml"><mi id="Ex5.m1.20.20.1.1.3.1.2.3.2" xref="Ex5.m1.20.20.1.1.3.1.2.3.2.cmml">i</mi><mo id="Ex5.m1.20.20.1.1.3.1.2.3.1" xref="Ex5.m1.20.20.1.1.3.1.2.3.1.cmml">=</mo><mn id="Ex5.m1.20.20.1.1.3.1.2.3.3" xref="Ex5.m1.20.20.1.1.3.1.2.3.3.cmml">0</mn></mrow><mn id="Ex5.m1.20.20.1.1.3.1.3" xref="Ex5.m1.20.20.1.1.3.1.3.cmml">127</mn></munderover><msubsup id="Ex5.m1.20.20.1.1.3.2" xref="Ex5.m1.20.20.1.1.3.2.cmml"><mover accent="true" id="Ex5.m1.20.20.1.1.3.2.2.2" xref="Ex5.m1.20.20.1.1.3.2.2.2.cmml"><mi id="Ex5.m1.20.20.1.1.3.2.2.2.2" xref="Ex5.m1.20.20.1.1.3.2.2.2.2.cmml">a</mi><mo id="Ex5.m1.20.20.1.1.3.2.2.2.1" xref="Ex5.m1.20.20.1.1.3.2.2.2.1.cmml">~</mo></mover><mrow id="Ex5.m1.6.6.2.4" xref="Ex5.m1.6.6.2.3.cmml"><mi id="Ex5.m1.5.5.1.1" xref="Ex5.m1.5.5.1.1.cmml">i</mi><mo id="Ex5.m1.6.6.2.4.1" xref="Ex5.m1.6.6.2.3.cmml">,</mo><mi id="Ex5.m1.6.6.2.2" xref="Ex5.m1.6.6.2.2.cmml">j</mi></mrow><mrow id="Ex5.m1.4.4.2.4" xref="Ex5.m1.4.4.2.3.cmml"><mi id="Ex5.m1.3.3.1.1" xref="Ex5.m1.3.3.1.1.cmml">l</mi><mo id="Ex5.m1.4.4.2.4.1" xref="Ex5.m1.4.4.2.3.cmml">,</mo><mi id="Ex5.m1.4.4.2.2" xref="Ex5.m1.4.4.2.2.cmml">h</mi></mrow></msubsup></mrow></mrow><mo rspace="1.167em" id="Ex5.m1.21.21.2.3" xref="Ex5.m1.21.21.3a.cmml">,</mo><mrow id="Ex5.m1.21.21.2.2" xref="Ex5.m1.21.21.2.2.cmml"><msubsup id="Ex5.m1.21.21.2.2.2" xref="Ex5.m1.21.21.2.2.2.cmml"><mover accent="true" id="Ex5.m1.21.21.2.2.2.2.2" xref="Ex5.m1.21.21.2.2.2.2.2.cmml"><mi id="Ex5.m1.21.21.2.2.2.2.2.2" xref="Ex5.m1.21.21.2.2.2.2.2.2.cmml">a</mi><mo id="Ex5.m1.21.21.2.2.2.2.2.1" xref="Ex5.m1.21.21.2.2.2.2.2.1.cmml">~</mo></mover><mrow id="Ex5.m1.10.10.2.4" xref="Ex5.m1.10.10.2.3.cmml"><mi id="Ex5.m1.9.9.1.1" xref="Ex5.m1.9.9.1.1.cmml">i</mi><mo id="Ex5.m1.10.10.2.4.1" xref="Ex5.m1.10.10.2.3.cmml">,</mo><mi id="Ex5.m1.10.10.2.2" xref="Ex5.m1.10.10.2.2.cmml">j</mi></mrow><mrow id="Ex5.m1.8.8.2.4" xref="Ex5.m1.8.8.2.3.cmml"><mi id="Ex5.m1.7.7.1.1" xref="Ex5.m1.7.7.1.1.cmml">l</mi><mo id="Ex5.m1.8.8.2.4.1" xref="Ex5.m1.8.8.2.3.cmml">,</mo><mi id="Ex5.m1.8.8.2.2" xref="Ex5.m1.8.8.2.2.cmml">h</mi></mrow></msubsup><mo id="Ex5.m1.21.21.2.2.1" xref="Ex5.m1.21.21.2.2.1.cmml">=</mo><mrow id="Ex5.m1.21.21.2.2.3" xref="Ex5.m1.21.21.2.2.3.cmml"><msub id="Ex5.m1.21.21.2.2.3.2" xref="Ex5.m1.21.21.2.2.3.2.cmml"><mtext id="Ex5.m1.21.21.2.2.3.2.2" xref="Ex5.m1.21.21.2.2.3.2.2a.cmml">softmax</mtext><mi id="Ex5.m1.21.21.2.2.3.2.3" xref="Ex5.m1.21.21.2.2.3.2.3.cmml">j</mi></msub><mo lspace="0em" rspace="0em" id="Ex5.m1.21.21.2.2.3.1" xref="Ex5.m1.21.21.2.2.3.1.cmml">​</mo><mrow id="Ex5.m1.21.21.2.2.3.3.2" xref="Ex5.m1.19.19.cmml"><mo maxsize="160%" minsize="160%" id="Ex5.m1.21.21.2.2.3.3.2.1" xref="Ex5.m1.19.19.cmml">(</mo><mfrac id="Ex5.m1.19.19" xref="Ex5.m1.19.19.cmml"><mrow id="Ex5.m1.19.19.9" xref="Ex5.m1.19.19.9.cmml"><msubsup id="Ex5.m1.19.19.9.11" xref="Ex5.m1.19.19.9.11.cmml"><mi id="Ex5.m1.19.19.9.11.2.2" xref="Ex5.m1.19.19.9.11.2.2.cmml">X</mi><mi id="Ex5.m1.19.19.9.11.3" xref="Ex5.m1.19.19.9.11.3.cmml">i</mi><mrow id="Ex5.m1.12.12.2.2.2.4" xref="Ex5.m1.12.12.2.2.2.3.cmml"><mi id="Ex5.m1.11.11.1.1.1.1" xref="Ex5.m1.11.11.1.1.1.1.cmml">l</mi><mo id="Ex5.m1.12.12.2.2.2.4.1" xref="Ex5.m1.12.12.2.2.2.3.cmml">,</mo><mi id="Ex5.m1.12.12.2.2.2.2" xref="Ex5.m1.12.12.2.2.2.2.cmml">h</mi></mrow></msubsup><mo lspace="0em" rspace="0em" id="Ex5.m1.19.19.9.10" xref="Ex5.m1.19.19.9.10.cmml">​</mo><msubsup id="Ex5.m1.19.19.9.12" xref="Ex5.m1.19.19.9.12.cmml"><mi id="Ex5.m1.19.19.9.12.2.2" xref="Ex5.m1.19.19.9.12.2.2.cmml">W</mi><mi id="Ex5.m1.19.19.9.12.3" xref="Ex5.m1.19.19.9.12.3.cmml">Q</mi><mrow id="Ex5.m1.14.14.4.4.2.4" xref="Ex5.m1.14.14.4.4.2.3.cmml"><mi id="Ex5.m1.13.13.3.3.1.1" xref="Ex5.m1.13.13.3.3.1.1.cmml">l</mi><mo id="Ex5.m1.14.14.4.4.2.4.1" xref="Ex5.m1.14.14.4.4.2.3.cmml">,</mo><mi id="Ex5.m1.14.14.4.4.2.2" xref="Ex5.m1.14.14.4.4.2.2.cmml">h</mi></mrow></msubsup><mo lspace="0em" rspace="0em" id="Ex5.m1.19.19.9.10a" xref="Ex5.m1.19.19.9.10.cmml">​</mo><msup id="Ex5.m1.19.19.9.9" xref="Ex5.m1.19.19.9.9.cmml"><mrow id="Ex5.m1.19.19.9.9.1.1" xref="Ex5.m1.19.19.9.9.1.1.1.cmml"><mo stretchy="false" id="Ex5.m1.19.19.9.9.1.1.2" xref="Ex5.m1.19.19.9.9.1.1.1.cmml">(</mo><mrow id="Ex5.m1.19.19.9.9.1.1.1" xref="Ex5.m1.19.19.9.9.1.1.1.cmml"><msubsup id="Ex5.m1.19.19.9.9.1.1.1.2" xref="Ex5.m1.19.19.9.9.1.1.1.2.cmml"><mi id="Ex5.m1.19.19.9.9.1.1.1.2.2.2" xref="Ex5.m1.19.19.9.9.1.1.1.2.2.2.cmml">X</mi><mi id="Ex5.m1.19.19.9.9.1.1.1.2.3" xref="Ex5.m1.19.19.9.9.1.1.1.2.3.cmml">j</mi><mrow id="Ex5.m1.16.16.6.6.2.4" xref="Ex5.m1.16.16.6.6.2.3.cmml"><mi id="Ex5.m1.15.15.5.5.1.1" xref="Ex5.m1.15.15.5.5.1.1.cmml">l</mi><mo id="Ex5.m1.16.16.6.6.2.4.1" xref="Ex5.m1.16.16.6.6.2.3.cmml">,</mo><mi id="Ex5.m1.16.16.6.6.2.2" xref="Ex5.m1.16.16.6.6.2.2.cmml">h</mi></mrow></msubsup><mo lspace="0em" rspace="0em" id="Ex5.m1.19.19.9.9.1.1.1.1" xref="Ex5.m1.19.19.9.9.1.1.1.1.cmml">​</mo><msubsup id="Ex5.m1.19.19.9.9.1.1.1.3" xref="Ex5.m1.19.19.9.9.1.1.1.3.cmml"><mi id="Ex5.m1.19.19.9.9.1.1.1.3.2.2" xref="Ex5.m1.19.19.9.9.1.1.1.3.2.2.cmml">W</mi><mi id="Ex5.m1.19.19.9.9.1.1.1.3.3" xref="Ex5.m1.19.19.9.9.1.1.1.3.3.cmml">K</mi><mrow id="Ex5.m1.18.18.8.8.2.4" xref="Ex5.m1.18.18.8.8.2.3.cmml"><mi id="Ex5.m1.17.17.7.7.1.1" xref="Ex5.m1.17.17.7.7.1.1.cmml">l</mi><mo id="Ex5.m1.18.18.8.8.2.4.1" xref="Ex5.m1.18.18.8.8.2.3.cmml">,</mo><mi id="Ex5.m1.18.18.8.8.2.2" xref="Ex5.m1.18.18.8.8.2.2.cmml">h</mi></mrow></msubsup></mrow><mo stretchy="false" id="Ex5.m1.19.19.9.9.1.1.3" xref="Ex5.m1.19.19.9.9.1.1.1.cmml">)</mo></mrow><mi id="Ex5.m1.19.19.9.9.3" xref="Ex5.m1.19.19.9.9.3.cmml">T</mi></msup></mrow><msqrt id="Ex5.m1.19.19.11" xref="Ex5.m1.19.19.11.cmml"><msub id="Ex5.m1.19.19.11.2" xref="Ex5.m1.19.19.11.2.cmml"><mi id="Ex5.m1.19.19.11.2.2" xref="Ex5.m1.19.19.11.2.2.cmml">d</mi><mi id="Ex5.m1.19.19.11.2.3" xref="Ex5.m1.19.19.11.2.3.cmml">h</mi></msub></msqrt></mfrac><mo maxsize="160%" minsize="160%" id="Ex5.m1.21.21.2.2.3.3.2.2" xref="Ex5.m1.19.19.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="Ex5.m1.21b"><apply id="Ex5.m1.21.21.3.cmml" xref="Ex5.m1.21.21.2"><csymbol cd="ambiguous" id="Ex5.m1.21.21.3a.cmml" xref="Ex5.m1.21.21.2.3">formulae-sequence</csymbol><apply id="Ex5.m1.20.20.1.1.cmml" xref="Ex5.m1.20.20.1.1"><eq id="Ex5.m1.20.20.1.1.1.cmml" xref="Ex5.m1.20.20.1.1.1"></eq><apply id="Ex5.m1.20.20.1.1.2.cmml" xref="Ex5.m1.20.20.1.1.2"><csymbol cd="ambiguous" id="Ex5.m1.20.20.1.1.2.1.cmml" xref="Ex5.m1.20.20.1.1.2">subscript</csymbol><apply id="Ex5.m1.20.20.1.1.2.2.cmml" xref="Ex5.m1.20.20.1.1.2"><csymbol cd="ambiguous" id="Ex5.m1.20.20.1.1.2.2.1.cmml" xref="Ex5.m1.20.20.1.1.2">superscript</csymbol><ci id="Ex5.m1.20.20.1.1.2.2.2.cmml" xref="Ex5.m1.20.20.1.1.2.2.2">𝑤</ci><list id="Ex5.m1.2.2.2.3.cmml" xref="Ex5.m1.2.2.2.4"><ci id="Ex5.m1.1.1.1.1.cmml" xref="Ex5.m1.1.1.1.1">𝑙</ci><ci id="Ex5.m1.2.2.2.2.cmml" xref="Ex5.m1.2.2.2.2">ℎ</ci></list></apply><ci id="Ex5.m1.20.20.1.1.2.3.cmml" xref="Ex5.m1.20.20.1.1.2.3">𝑗</ci></apply><apply id="Ex5.m1.20.20.1.1.3.cmml" xref="Ex5.m1.20.20.1.1.3"><apply id="Ex5.m1.20.20.1.1.3.1.cmml" xref="Ex5.m1.20.20.1.1.3.1"><csymbol cd="ambiguous" id="Ex5.m1.20.20.1.1.3.1.1.cmml" xref="Ex5.m1.20.20.1.1.3.1">superscript</csymbol><apply id="Ex5.m1.20.20.1.1.3.1.2.cmml" xref="Ex5.m1.20.20.1.1.3.1"><csymbol cd="ambiguous" id="Ex5.m1.20.20.1.1.3.1.2.1.cmml" xref="Ex5.m1.20.20.1.1.3.1">subscript</csymbol><sum id="Ex5.m1.20.20.1.1.3.1.2.2.cmml" xref="Ex5.m1.20.20.1.1.3.1.2.2"></sum><apply id="Ex5.m1.20.20.1.1.3.1.2.3.cmml" xref="Ex5.m1.20.20.1.1.3.1.2.3"><eq id="Ex5.m1.20.20.1.1.3.1.2.3.1.cmml" xref="Ex5.m1.20.20.1.1.3.1.2.3.1"></eq><ci id="Ex5.m1.20.20.1.1.3.1.2.3.2.cmml" xref="Ex5.m1.20.20.1.1.3.1.2.3.2">𝑖</ci><cn type="integer" id="Ex5.m1.20.20.1.1.3.1.2.3.3.cmml" xref="Ex5.m1.20.20.1.1.3.1.2.3.3">0</cn></apply></apply><cn type="integer" id="Ex5.m1.20.20.1.1.3.1.3.cmml" xref="Ex5.m1.20.20.1.1.3.1.3">127</cn></apply><apply id="Ex5.m1.20.20.1.1.3.2.cmml" xref="Ex5.m1.20.20.1.1.3.2"><csymbol cd="ambiguous" id="Ex5.m1.20.20.1.1.3.2.1.cmml" xref="Ex5.m1.20.20.1.1.3.2">subscript</csymbol><apply id="Ex5.m1.20.20.1.1.3.2.2.cmml" xref="Ex5.m1.20.20.1.1.3.2"><csymbol cd="ambiguous" id="Ex5.m1.20.20.1.1.3.2.2.1.cmml" xref="Ex5.m1.20.20.1.1.3.2">superscript</csymbol><apply id="Ex5.m1.20.20.1.1.3.2.2.2.cmml" xref="Ex5.m1.20.20.1.1.3.2.2.2"><ci id="Ex5.m1.20.20.1.1.3.2.2.2.1.cmml" xref="Ex5.m1.20.20.1.1.3.2.2.2.1">~</ci><ci id="Ex5.m1.20.20.1.1.3.2.2.2.2.cmml" xref="Ex5.m1.20.20.1.1.3.2.2.2.2">𝑎</ci></apply><list id="Ex5.m1.4.4.2.3.cmml" xref="Ex5.m1.4.4.2.4"><ci id="Ex5.m1.3.3.1.1.cmml" xref="Ex5.m1.3.3.1.1">𝑙</ci><ci id="Ex5.m1.4.4.2.2.cmml" xref="Ex5.m1.4.4.2.2">ℎ</ci></list></apply><list id="Ex5.m1.6.6.2.3.cmml" xref="Ex5.m1.6.6.2.4"><ci id="Ex5.m1.5.5.1.1.cmml" xref="Ex5.m1.5.5.1.1">𝑖</ci><ci id="Ex5.m1.6.6.2.2.cmml" xref="Ex5.m1.6.6.2.2">𝑗</ci></list></apply></apply></apply><apply id="Ex5.m1.21.21.2.2.cmml" xref="Ex5.m1.21.21.2.2"><eq id="Ex5.m1.21.21.2.2.1.cmml" xref="Ex5.m1.21.21.2.2.1"></eq><apply id="Ex5.m1.21.21.2.2.2.cmml" xref="Ex5.m1.21.21.2.2.2"><csymbol cd="ambiguous" id="Ex5.m1.21.21.2.2.2.1.cmml" xref="Ex5.m1.21.21.2.2.2">subscript</csymbol><apply id="Ex5.m1.21.21.2.2.2.2.cmml" xref="Ex5.m1.21.21.2.2.2"><csymbol cd="ambiguous" id="Ex5.m1.21.21.2.2.2.2.1.cmml" xref="Ex5.m1.21.21.2.2.2">superscript</csymbol><apply id="Ex5.m1.21.21.2.2.2.2.2.cmml" xref="Ex5.m1.21.21.2.2.2.2.2"><ci id="Ex5.m1.21.21.2.2.2.2.2.1.cmml" xref="Ex5.m1.21.21.2.2.2.2.2.1">~</ci><ci id="Ex5.m1.21.21.2.2.2.2.2.2.cmml" xref="Ex5.m1.21.21.2.2.2.2.2.2">𝑎</ci></apply><list id="Ex5.m1.8.8.2.3.cmml" xref="Ex5.m1.8.8.2.4"><ci id="Ex5.m1.7.7.1.1.cmml" xref="Ex5.m1.7.7.1.1">𝑙</ci><ci id="Ex5.m1.8.8.2.2.cmml" xref="Ex5.m1.8.8.2.2">ℎ</ci></list></apply><list id="Ex5.m1.10.10.2.3.cmml" xref="Ex5.m1.10.10.2.4"><ci id="Ex5.m1.9.9.1.1.cmml" xref="Ex5.m1.9.9.1.1">𝑖</ci><ci id="Ex5.m1.10.10.2.2.cmml" xref="Ex5.m1.10.10.2.2">𝑗</ci></list></apply><apply id="Ex5.m1.21.21.2.2.3.cmml" xref="Ex5.m1.21.21.2.2.3"><times id="Ex5.m1.21.21.2.2.3.1.cmml" xref="Ex5.m1.21.21.2.2.3.1"></times><apply id="Ex5.m1.21.21.2.2.3.2.cmml" xref="Ex5.m1.21.21.2.2.3.2"><csymbol cd="ambiguous" id="Ex5.m1.21.21.2.2.3.2.1.cmml" xref="Ex5.m1.21.21.2.2.3.2">subscript</csymbol><ci id="Ex5.m1.21.21.2.2.3.2.2a.cmml" xref="Ex5.m1.21.21.2.2.3.2.2"><mtext id="Ex5.m1.21.21.2.2.3.2.2.cmml" xref="Ex5.m1.21.21.2.2.3.2.2">softmax</mtext></ci><ci id="Ex5.m1.21.21.2.2.3.2.3.cmml" xref="Ex5.m1.21.21.2.2.3.2.3">𝑗</ci></apply><apply id="Ex5.m1.19.19.cmml" xref="Ex5.m1.21.21.2.2.3.3.2"><divide id="Ex5.m1.19.19.10.cmml" xref="Ex5.m1.21.21.2.2.3.3.2"></divide><apply id="Ex5.m1.19.19.9.cmml" xref="Ex5.m1.19.19.9"><times id="Ex5.m1.19.19.9.10.cmml" xref="Ex5.m1.19.19.9.10"></times><apply id="Ex5.m1.19.19.9.11.cmml" xref="Ex5.m1.19.19.9.11"><csymbol cd="ambiguous" id="Ex5.m1.19.19.9.11.1.cmml" xref="Ex5.m1.19.19.9.11">subscript</csymbol><apply id="Ex5.m1.19.19.9.11.2.cmml" xref="Ex5.m1.19.19.9.11"><csymbol cd="ambiguous" id="Ex5.m1.19.19.9.11.2.1.cmml" xref="Ex5.m1.19.19.9.11">superscript</csymbol><ci id="Ex5.m1.19.19.9.11.2.2.cmml" xref="Ex5.m1.19.19.9.11.2.2">𝑋</ci><list id="Ex5.m1.12.12.2.2.2.3.cmml" xref="Ex5.m1.12.12.2.2.2.4"><ci id="Ex5.m1.11.11.1.1.1.1.cmml" xref="Ex5.m1.11.11.1.1.1.1">𝑙</ci><ci id="Ex5.m1.12.12.2.2.2.2.cmml" xref="Ex5.m1.12.12.2.2.2.2">ℎ</ci></list></apply><ci id="Ex5.m1.19.19.9.11.3.cmml" xref="Ex5.m1.19.19.9.11.3">𝑖</ci></apply><apply id="Ex5.m1.19.19.9.12.cmml" xref="Ex5.m1.19.19.9.12"><csymbol cd="ambiguous" id="Ex5.m1.19.19.9.12.1.cmml" xref="Ex5.m1.19.19.9.12">subscript</csymbol><apply id="Ex5.m1.19.19.9.12.2.cmml" xref="Ex5.m1.19.19.9.12"><csymbol cd="ambiguous" id="Ex5.m1.19.19.9.12.2.1.cmml" xref="Ex5.m1.19.19.9.12">superscript</csymbol><ci id="Ex5.m1.19.19.9.12.2.2.cmml" xref="Ex5.m1.19.19.9.12.2.2">𝑊</ci><list id="Ex5.m1.14.14.4.4.2.3.cmml" xref="Ex5.m1.14.14.4.4.2.4"><ci id="Ex5.m1.13.13.3.3.1.1.cmml" xref="Ex5.m1.13.13.3.3.1.1">𝑙</ci><ci id="Ex5.m1.14.14.4.4.2.2.cmml" xref="Ex5.m1.14.14.4.4.2.2">ℎ</ci></list></apply><ci id="Ex5.m1.19.19.9.12.3.cmml" xref="Ex5.m1.19.19.9.12.3">𝑄</ci></apply><apply id="Ex5.m1.19.19.9.9.cmml" xref="Ex5.m1.19.19.9.9"><csymbol cd="ambiguous" id="Ex5.m1.19.19.9.9.2.cmml" xref="Ex5.m1.19.19.9.9">superscript</csymbol><apply id="Ex5.m1.19.19.9.9.1.1.1.cmml" xref="Ex5.m1.19.19.9.9.1.1"><times id="Ex5.m1.19.19.9.9.1.1.1.1.cmml" xref="Ex5.m1.19.19.9.9.1.1.1.1"></times><apply id="Ex5.m1.19.19.9.9.1.1.1.2.cmml" xref="Ex5.m1.19.19.9.9.1.1.1.2"><csymbol cd="ambiguous" id="Ex5.m1.19.19.9.9.1.1.1.2.1.cmml" xref="Ex5.m1.19.19.9.9.1.1.1.2">subscript</csymbol><apply id="Ex5.m1.19.19.9.9.1.1.1.2.2.cmml" xref="Ex5.m1.19.19.9.9.1.1.1.2"><csymbol cd="ambiguous" id="Ex5.m1.19.19.9.9.1.1.1.2.2.1.cmml" xref="Ex5.m1.19.19.9.9.1.1.1.2">superscript</csymbol><ci id="Ex5.m1.19.19.9.9.1.1.1.2.2.2.cmml" xref="Ex5.m1.19.19.9.9.1.1.1.2.2.2">𝑋</ci><list id="Ex5.m1.16.16.6.6.2.3.cmml" xref="Ex5.m1.16.16.6.6.2.4"><ci id="Ex5.m1.15.15.5.5.1.1.cmml" xref="Ex5.m1.15.15.5.5.1.1">𝑙</ci><ci id="Ex5.m1.16.16.6.6.2.2.cmml" xref="Ex5.m1.16.16.6.6.2.2">ℎ</ci></list></apply><ci id="Ex5.m1.19.19.9.9.1.1.1.2.3.cmml" xref="Ex5.m1.19.19.9.9.1.1.1.2.3">𝑗</ci></apply><apply id="Ex5.m1.19.19.9.9.1.1.1.3.cmml" xref="Ex5.m1.19.19.9.9.1.1.1.3"><csymbol cd="ambiguous" id="Ex5.m1.19.19.9.9.1.1.1.3.1.cmml" xref="Ex5.m1.19.19.9.9.1.1.1.3">subscript</csymbol><apply id="Ex5.m1.19.19.9.9.1.1.1.3.2.cmml" xref="Ex5.m1.19.19.9.9.1.1.1.3"><csymbol cd="ambiguous" id="Ex5.m1.19.19.9.9.1.1.1.3.2.1.cmml" xref="Ex5.m1.19.19.9.9.1.1.1.3">superscript</csymbol><ci id="Ex5.m1.19.19.9.9.1.1.1.3.2.2.cmml" xref="Ex5.m1.19.19.9.9.1.1.1.3.2.2">𝑊</ci><list id="Ex5.m1.18.18.8.8.2.3.cmml" xref="Ex5.m1.18.18.8.8.2.4"><ci id="Ex5.m1.17.17.7.7.1.1.cmml" xref="Ex5.m1.17.17.7.7.1.1">𝑙</ci><ci id="Ex5.m1.18.18.8.8.2.2.cmml" xref="Ex5.m1.18.18.8.8.2.2">ℎ</ci></list></apply><ci id="Ex5.m1.19.19.9.9.1.1.1.3.3.cmml" xref="Ex5.m1.19.19.9.9.1.1.1.3.3">𝐾</ci></apply></apply><ci id="Ex5.m1.19.19.9.9.3.cmml" xref="Ex5.m1.19.19.9.9.3">𝑇</ci></apply></apply><apply id="Ex5.m1.19.19.11.cmml" xref="Ex5.m1.19.19.11"><root id="Ex5.m1.19.19.11a.cmml" xref="Ex5.m1.19.19.11"></root><apply id="Ex5.m1.19.19.11.2.cmml" xref="Ex5.m1.19.19.11.2"><csymbol cd="ambiguous" id="Ex5.m1.19.19.11.2.1.cmml" xref="Ex5.m1.19.19.11.2">subscript</csymbol><ci id="Ex5.m1.19.19.11.2.2.cmml" xref="Ex5.m1.19.19.11.2.2">𝑑</ci><ci id="Ex5.m1.19.19.11.2.3.cmml" xref="Ex5.m1.19.19.11.2.3">ℎ</ci></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Ex5.m1.21c">w^{l,h}_{j}=\sum_{i=0}^{127}\tilde{a}^{l,h}_{i,j},\quad\tilde{a}^{l,h}_{i,j}=\text{softmax}_{j}\Big{(}\frac{X^{l,h}_{i}W^{l,h}_{Q}(X^{l,h}_{j}W^{l,h}_{K})^{T}}{\sqrt{d_{h}}}\Big{)}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S3.SS3.p4.3" class="ltx_p">which measures the importance of a token by the attention received from all tokens.
The BOS tokens and paddings do not participate in the the computation of the weights.
These attention weights <math id="S3.SS3.p4.3.m1.1" class="ltx_Math" alttext="\tilde{a}" display="inline"><semantics id="S3.SS3.p4.3.m1.1a"><mover accent="true" id="S3.SS3.p4.3.m1.1.1" xref="S3.SS3.p4.3.m1.1.1.cmml"><mi id="S3.SS3.p4.3.m1.1.1.2" xref="S3.SS3.p4.3.m1.1.1.2.cmml">a</mi><mo id="S3.SS3.p4.3.m1.1.1.1" xref="S3.SS3.p4.3.m1.1.1.1.cmml">~</mo></mover><annotation-xml encoding="MathML-Content" id="S3.SS3.p4.3.m1.1b"><apply id="S3.SS3.p4.3.m1.1.1.cmml" xref="S3.SS3.p4.3.m1.1.1"><ci id="S3.SS3.p4.3.m1.1.1.1.cmml" xref="S3.SS3.p4.3.m1.1.1.1">~</ci><ci id="S3.SS3.p4.3.m1.1.1.2.cmml" xref="S3.SS3.p4.3.m1.1.1.2">𝑎</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p4.3.m1.1c">\tilde{a}</annotation></semantics></math> are different from the usual ones, such that there is no causal mask or position encoding involved.
The consideration is that since the explicit memories are prepared before any inference, the selection can only depend on the reference itself instead of any context texts.
The removal of causal mask and position encoding ensures that tokens at any position has an equal chance to receive attention from others.
To speed up computation, we adopt the following approximate weights in our implementation, although in retrospect this speedup is not necessary.</p>
<table id="Ex6" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="Ex6.m1.12" class="ltx_Math" alttext="w^{l,h}_{j}=\sum_{i=0}^{127}\exp\Big{(}\frac{X^{l,h}_{i}W^{l,h}_{Q}(X^{l,h}_{j}W^{l,h}_{K})^{T}}{\sqrt{d_{h}}}\Big{)}" display="block"><semantics id="Ex6.m1.12a"><mrow id="Ex6.m1.12.13" xref="Ex6.m1.12.13.cmml"><msubsup id="Ex6.m1.12.13.2" xref="Ex6.m1.12.13.2.cmml"><mi id="Ex6.m1.12.13.2.2.2" xref="Ex6.m1.12.13.2.2.2.cmml">w</mi><mi id="Ex6.m1.12.13.2.3" xref="Ex6.m1.12.13.2.3.cmml">j</mi><mrow id="Ex6.m1.2.2.2.4" xref="Ex6.m1.2.2.2.3.cmml"><mi id="Ex6.m1.1.1.1.1" xref="Ex6.m1.1.1.1.1.cmml">l</mi><mo id="Ex6.m1.2.2.2.4.1" xref="Ex6.m1.2.2.2.3.cmml">,</mo><mi id="Ex6.m1.2.2.2.2" xref="Ex6.m1.2.2.2.2.cmml">h</mi></mrow></msubsup><mo rspace="0.111em" id="Ex6.m1.12.13.1" xref="Ex6.m1.12.13.1.cmml">=</mo><mrow id="Ex6.m1.12.13.3" xref="Ex6.m1.12.13.3.cmml"><munderover id="Ex6.m1.12.13.3.1" xref="Ex6.m1.12.13.3.1.cmml"><mo movablelimits="false" id="Ex6.m1.12.13.3.1.2.2" xref="Ex6.m1.12.13.3.1.2.2.cmml">∑</mo><mrow id="Ex6.m1.12.13.3.1.2.3" xref="Ex6.m1.12.13.3.1.2.3.cmml"><mi id="Ex6.m1.12.13.3.1.2.3.2" xref="Ex6.m1.12.13.3.1.2.3.2.cmml">i</mi><mo id="Ex6.m1.12.13.3.1.2.3.1" xref="Ex6.m1.12.13.3.1.2.3.1.cmml">=</mo><mn id="Ex6.m1.12.13.3.1.2.3.3" xref="Ex6.m1.12.13.3.1.2.3.3.cmml">0</mn></mrow><mn id="Ex6.m1.12.13.3.1.3" xref="Ex6.m1.12.13.3.1.3.cmml">127</mn></munderover><mrow id="Ex6.m1.12.13.3.2.2" xref="Ex6.m1.12.13.3.2.1.cmml"><mi id="Ex6.m1.12.12" xref="Ex6.m1.12.12.cmml">exp</mi><mo id="Ex6.m1.12.13.3.2.2a" xref="Ex6.m1.12.13.3.2.1.cmml">⁡</mo><mrow id="Ex6.m1.12.13.3.2.2.1" xref="Ex6.m1.12.13.3.2.1.cmml"><mo maxsize="160%" minsize="160%" id="Ex6.m1.12.13.3.2.2.1.1" xref="Ex6.m1.12.13.3.2.1.cmml">(</mo><mfrac id="Ex6.m1.11.11" xref="Ex6.m1.11.11.cmml"><mrow id="Ex6.m1.11.11.9" xref="Ex6.m1.11.11.9.cmml"><msubsup id="Ex6.m1.11.11.9.11" xref="Ex6.m1.11.11.9.11.cmml"><mi id="Ex6.m1.11.11.9.11.2.2" xref="Ex6.m1.11.11.9.11.2.2.cmml">X</mi><mi id="Ex6.m1.11.11.9.11.3" xref="Ex6.m1.11.11.9.11.3.cmml">i</mi><mrow id="Ex6.m1.4.4.2.2.2.4" xref="Ex6.m1.4.4.2.2.2.3.cmml"><mi id="Ex6.m1.3.3.1.1.1.1" xref="Ex6.m1.3.3.1.1.1.1.cmml">l</mi><mo id="Ex6.m1.4.4.2.2.2.4.1" xref="Ex6.m1.4.4.2.2.2.3.cmml">,</mo><mi id="Ex6.m1.4.4.2.2.2.2" xref="Ex6.m1.4.4.2.2.2.2.cmml">h</mi></mrow></msubsup><mo lspace="0em" rspace="0em" id="Ex6.m1.11.11.9.10" xref="Ex6.m1.11.11.9.10.cmml">​</mo><msubsup id="Ex6.m1.11.11.9.12" xref="Ex6.m1.11.11.9.12.cmml"><mi id="Ex6.m1.11.11.9.12.2.2" xref="Ex6.m1.11.11.9.12.2.2.cmml">W</mi><mi id="Ex6.m1.11.11.9.12.3" xref="Ex6.m1.11.11.9.12.3.cmml">Q</mi><mrow id="Ex6.m1.6.6.4.4.2.4" xref="Ex6.m1.6.6.4.4.2.3.cmml"><mi id="Ex6.m1.5.5.3.3.1.1" xref="Ex6.m1.5.5.3.3.1.1.cmml">l</mi><mo id="Ex6.m1.6.6.4.4.2.4.1" xref="Ex6.m1.6.6.4.4.2.3.cmml">,</mo><mi id="Ex6.m1.6.6.4.4.2.2" xref="Ex6.m1.6.6.4.4.2.2.cmml">h</mi></mrow></msubsup><mo lspace="0em" rspace="0em" id="Ex6.m1.11.11.9.10a" xref="Ex6.m1.11.11.9.10.cmml">​</mo><msup id="Ex6.m1.11.11.9.9" xref="Ex6.m1.11.11.9.9.cmml"><mrow id="Ex6.m1.11.11.9.9.1.1" xref="Ex6.m1.11.11.9.9.1.1.1.cmml"><mo stretchy="false" id="Ex6.m1.11.11.9.9.1.1.2" xref="Ex6.m1.11.11.9.9.1.1.1.cmml">(</mo><mrow id="Ex6.m1.11.11.9.9.1.1.1" xref="Ex6.m1.11.11.9.9.1.1.1.cmml"><msubsup id="Ex6.m1.11.11.9.9.1.1.1.2" xref="Ex6.m1.11.11.9.9.1.1.1.2.cmml"><mi id="Ex6.m1.11.11.9.9.1.1.1.2.2.2" xref="Ex6.m1.11.11.9.9.1.1.1.2.2.2.cmml">X</mi><mi id="Ex6.m1.11.11.9.9.1.1.1.2.3" xref="Ex6.m1.11.11.9.9.1.1.1.2.3.cmml">j</mi><mrow id="Ex6.m1.8.8.6.6.2.4" xref="Ex6.m1.8.8.6.6.2.3.cmml"><mi id="Ex6.m1.7.7.5.5.1.1" xref="Ex6.m1.7.7.5.5.1.1.cmml">l</mi><mo id="Ex6.m1.8.8.6.6.2.4.1" xref="Ex6.m1.8.8.6.6.2.3.cmml">,</mo><mi id="Ex6.m1.8.8.6.6.2.2" xref="Ex6.m1.8.8.6.6.2.2.cmml">h</mi></mrow></msubsup><mo lspace="0em" rspace="0em" id="Ex6.m1.11.11.9.9.1.1.1.1" xref="Ex6.m1.11.11.9.9.1.1.1.1.cmml">​</mo><msubsup id="Ex6.m1.11.11.9.9.1.1.1.3" xref="Ex6.m1.11.11.9.9.1.1.1.3.cmml"><mi id="Ex6.m1.11.11.9.9.1.1.1.3.2.2" xref="Ex6.m1.11.11.9.9.1.1.1.3.2.2.cmml">W</mi><mi id="Ex6.m1.11.11.9.9.1.1.1.3.3" xref="Ex6.m1.11.11.9.9.1.1.1.3.3.cmml">K</mi><mrow id="Ex6.m1.10.10.8.8.2.4" xref="Ex6.m1.10.10.8.8.2.3.cmml"><mi id="Ex6.m1.9.9.7.7.1.1" xref="Ex6.m1.9.9.7.7.1.1.cmml">l</mi><mo id="Ex6.m1.10.10.8.8.2.4.1" xref="Ex6.m1.10.10.8.8.2.3.cmml">,</mo><mi id="Ex6.m1.10.10.8.8.2.2" xref="Ex6.m1.10.10.8.8.2.2.cmml">h</mi></mrow></msubsup></mrow><mo stretchy="false" id="Ex6.m1.11.11.9.9.1.1.3" xref="Ex6.m1.11.11.9.9.1.1.1.cmml">)</mo></mrow><mi id="Ex6.m1.11.11.9.9.3" xref="Ex6.m1.11.11.9.9.3.cmml">T</mi></msup></mrow><msqrt id="Ex6.m1.11.11.11" xref="Ex6.m1.11.11.11.cmml"><msub id="Ex6.m1.11.11.11.2" xref="Ex6.m1.11.11.11.2.cmml"><mi id="Ex6.m1.11.11.11.2.2" xref="Ex6.m1.11.11.11.2.2.cmml">d</mi><mi id="Ex6.m1.11.11.11.2.3" xref="Ex6.m1.11.11.11.2.3.cmml">h</mi></msub></msqrt></mfrac><mo maxsize="160%" minsize="160%" id="Ex6.m1.12.13.3.2.2.1.2" xref="Ex6.m1.12.13.3.2.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="Ex6.m1.12b"><apply id="Ex6.m1.12.13.cmml" xref="Ex6.m1.12.13"><eq id="Ex6.m1.12.13.1.cmml" xref="Ex6.m1.12.13.1"></eq><apply id="Ex6.m1.12.13.2.cmml" xref="Ex6.m1.12.13.2"><csymbol cd="ambiguous" id="Ex6.m1.12.13.2.1.cmml" xref="Ex6.m1.12.13.2">subscript</csymbol><apply id="Ex6.m1.12.13.2.2.cmml" xref="Ex6.m1.12.13.2"><csymbol cd="ambiguous" id="Ex6.m1.12.13.2.2.1.cmml" xref="Ex6.m1.12.13.2">superscript</csymbol><ci id="Ex6.m1.12.13.2.2.2.cmml" xref="Ex6.m1.12.13.2.2.2">𝑤</ci><list id="Ex6.m1.2.2.2.3.cmml" xref="Ex6.m1.2.2.2.4"><ci id="Ex6.m1.1.1.1.1.cmml" xref="Ex6.m1.1.1.1.1">𝑙</ci><ci id="Ex6.m1.2.2.2.2.cmml" xref="Ex6.m1.2.2.2.2">ℎ</ci></list></apply><ci id="Ex6.m1.12.13.2.3.cmml" xref="Ex6.m1.12.13.2.3">𝑗</ci></apply><apply id="Ex6.m1.12.13.3.cmml" xref="Ex6.m1.12.13.3"><apply id="Ex6.m1.12.13.3.1.cmml" xref="Ex6.m1.12.13.3.1"><csymbol cd="ambiguous" id="Ex6.m1.12.13.3.1.1.cmml" xref="Ex6.m1.12.13.3.1">superscript</csymbol><apply id="Ex6.m1.12.13.3.1.2.cmml" xref="Ex6.m1.12.13.3.1"><csymbol cd="ambiguous" id="Ex6.m1.12.13.3.1.2.1.cmml" xref="Ex6.m1.12.13.3.1">subscript</csymbol><sum id="Ex6.m1.12.13.3.1.2.2.cmml" xref="Ex6.m1.12.13.3.1.2.2"></sum><apply id="Ex6.m1.12.13.3.1.2.3.cmml" xref="Ex6.m1.12.13.3.1.2.3"><eq id="Ex6.m1.12.13.3.1.2.3.1.cmml" xref="Ex6.m1.12.13.3.1.2.3.1"></eq><ci id="Ex6.m1.12.13.3.1.2.3.2.cmml" xref="Ex6.m1.12.13.3.1.2.3.2">𝑖</ci><cn type="integer" id="Ex6.m1.12.13.3.1.2.3.3.cmml" xref="Ex6.m1.12.13.3.1.2.3.3">0</cn></apply></apply><cn type="integer" id="Ex6.m1.12.13.3.1.3.cmml" xref="Ex6.m1.12.13.3.1.3">127</cn></apply><apply id="Ex6.m1.12.13.3.2.1.cmml" xref="Ex6.m1.12.13.3.2.2"><exp id="Ex6.m1.12.12.cmml" xref="Ex6.m1.12.12"></exp><apply id="Ex6.m1.11.11.cmml" xref="Ex6.m1.11.11"><divide id="Ex6.m1.11.11.10.cmml" xref="Ex6.m1.11.11"></divide><apply id="Ex6.m1.11.11.9.cmml" xref="Ex6.m1.11.11.9"><times id="Ex6.m1.11.11.9.10.cmml" xref="Ex6.m1.11.11.9.10"></times><apply id="Ex6.m1.11.11.9.11.cmml" xref="Ex6.m1.11.11.9.11"><csymbol cd="ambiguous" id="Ex6.m1.11.11.9.11.1.cmml" xref="Ex6.m1.11.11.9.11">subscript</csymbol><apply id="Ex6.m1.11.11.9.11.2.cmml" xref="Ex6.m1.11.11.9.11"><csymbol cd="ambiguous" id="Ex6.m1.11.11.9.11.2.1.cmml" xref="Ex6.m1.11.11.9.11">superscript</csymbol><ci id="Ex6.m1.11.11.9.11.2.2.cmml" xref="Ex6.m1.11.11.9.11.2.2">𝑋</ci><list id="Ex6.m1.4.4.2.2.2.3.cmml" xref="Ex6.m1.4.4.2.2.2.4"><ci id="Ex6.m1.3.3.1.1.1.1.cmml" xref="Ex6.m1.3.3.1.1.1.1">𝑙</ci><ci id="Ex6.m1.4.4.2.2.2.2.cmml" xref="Ex6.m1.4.4.2.2.2.2">ℎ</ci></list></apply><ci id="Ex6.m1.11.11.9.11.3.cmml" xref="Ex6.m1.11.11.9.11.3">𝑖</ci></apply><apply id="Ex6.m1.11.11.9.12.cmml" xref="Ex6.m1.11.11.9.12"><csymbol cd="ambiguous" id="Ex6.m1.11.11.9.12.1.cmml" xref="Ex6.m1.11.11.9.12">subscript</csymbol><apply id="Ex6.m1.11.11.9.12.2.cmml" xref="Ex6.m1.11.11.9.12"><csymbol cd="ambiguous" id="Ex6.m1.11.11.9.12.2.1.cmml" xref="Ex6.m1.11.11.9.12">superscript</csymbol><ci id="Ex6.m1.11.11.9.12.2.2.cmml" xref="Ex6.m1.11.11.9.12.2.2">𝑊</ci><list id="Ex6.m1.6.6.4.4.2.3.cmml" xref="Ex6.m1.6.6.4.4.2.4"><ci id="Ex6.m1.5.5.3.3.1.1.cmml" xref="Ex6.m1.5.5.3.3.1.1">𝑙</ci><ci id="Ex6.m1.6.6.4.4.2.2.cmml" xref="Ex6.m1.6.6.4.4.2.2">ℎ</ci></list></apply><ci id="Ex6.m1.11.11.9.12.3.cmml" xref="Ex6.m1.11.11.9.12.3">𝑄</ci></apply><apply id="Ex6.m1.11.11.9.9.cmml" xref="Ex6.m1.11.11.9.9"><csymbol cd="ambiguous" id="Ex6.m1.11.11.9.9.2.cmml" xref="Ex6.m1.11.11.9.9">superscript</csymbol><apply id="Ex6.m1.11.11.9.9.1.1.1.cmml" xref="Ex6.m1.11.11.9.9.1.1"><times id="Ex6.m1.11.11.9.9.1.1.1.1.cmml" xref="Ex6.m1.11.11.9.9.1.1.1.1"></times><apply id="Ex6.m1.11.11.9.9.1.1.1.2.cmml" xref="Ex6.m1.11.11.9.9.1.1.1.2"><csymbol cd="ambiguous" id="Ex6.m1.11.11.9.9.1.1.1.2.1.cmml" xref="Ex6.m1.11.11.9.9.1.1.1.2">subscript</csymbol><apply id="Ex6.m1.11.11.9.9.1.1.1.2.2.cmml" xref="Ex6.m1.11.11.9.9.1.1.1.2"><csymbol cd="ambiguous" id="Ex6.m1.11.11.9.9.1.1.1.2.2.1.cmml" xref="Ex6.m1.11.11.9.9.1.1.1.2">superscript</csymbol><ci id="Ex6.m1.11.11.9.9.1.1.1.2.2.2.cmml" xref="Ex6.m1.11.11.9.9.1.1.1.2.2.2">𝑋</ci><list id="Ex6.m1.8.8.6.6.2.3.cmml" xref="Ex6.m1.8.8.6.6.2.4"><ci id="Ex6.m1.7.7.5.5.1.1.cmml" xref="Ex6.m1.7.7.5.5.1.1">𝑙</ci><ci id="Ex6.m1.8.8.6.6.2.2.cmml" xref="Ex6.m1.8.8.6.6.2.2">ℎ</ci></list></apply><ci id="Ex6.m1.11.11.9.9.1.1.1.2.3.cmml" xref="Ex6.m1.11.11.9.9.1.1.1.2.3">𝑗</ci></apply><apply id="Ex6.m1.11.11.9.9.1.1.1.3.cmml" xref="Ex6.m1.11.11.9.9.1.1.1.3"><csymbol cd="ambiguous" id="Ex6.m1.11.11.9.9.1.1.1.3.1.cmml" xref="Ex6.m1.11.11.9.9.1.1.1.3">subscript</csymbol><apply id="Ex6.m1.11.11.9.9.1.1.1.3.2.cmml" xref="Ex6.m1.11.11.9.9.1.1.1.3"><csymbol cd="ambiguous" id="Ex6.m1.11.11.9.9.1.1.1.3.2.1.cmml" xref="Ex6.m1.11.11.9.9.1.1.1.3">superscript</csymbol><ci id="Ex6.m1.11.11.9.9.1.1.1.3.2.2.cmml" xref="Ex6.m1.11.11.9.9.1.1.1.3.2.2">𝑊</ci><list id="Ex6.m1.10.10.8.8.2.3.cmml" xref="Ex6.m1.10.10.8.8.2.4"><ci id="Ex6.m1.9.9.7.7.1.1.cmml" xref="Ex6.m1.9.9.7.7.1.1">𝑙</ci><ci id="Ex6.m1.10.10.8.8.2.2.cmml" xref="Ex6.m1.10.10.8.8.2.2">ℎ</ci></list></apply><ci id="Ex6.m1.11.11.9.9.1.1.1.3.3.cmml" xref="Ex6.m1.11.11.9.9.1.1.1.3.3">𝐾</ci></apply></apply><ci id="Ex6.m1.11.11.9.9.3.cmml" xref="Ex6.m1.11.11.9.9.3">𝑇</ci></apply></apply><apply id="Ex6.m1.11.11.11.cmml" xref="Ex6.m1.11.11.11"><root id="Ex6.m1.11.11.11a.cmml" xref="Ex6.m1.11.11.11"></root><apply id="Ex6.m1.11.11.11.2.cmml" xref="Ex6.m1.11.11.11.2"><csymbol cd="ambiguous" id="Ex6.m1.11.11.11.2.1.cmml" xref="Ex6.m1.11.11.11.2">subscript</csymbol><ci id="Ex6.m1.11.11.11.2.2.cmml" xref="Ex6.m1.11.11.11.2.2">𝑑</ci><ci id="Ex6.m1.11.11.11.2.3.cmml" xref="Ex6.m1.11.11.11.2.3">ℎ</ci></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Ex6.m1.12c">w^{l,h}_{j}=\sum_{i=0}^{127}\exp\Big{(}\frac{X^{l,h}_{i}W^{l,h}_{Q}(X^{l,h}_{j}W^{l,h}_{K})^{T}}{\sqrt{d_{h}}}\Big{)}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S3.SS3.p4.4" class="ltx_p">Similar designs that sparsify tokens based on attention weights have been adopted in long-context modeling to save space <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib74" title="" class="ltx_ref">74</a>, <a href="#bib.bib131" title="" class="ltx_ref">131</a>]</cite>.</p>
</div>
<div id="S3.SS3.p5" class="ltx_para">
<p id="S3.SS3.p5.2" class="ltx_p">Regarding head dimension, we optionally use a vector quantizer to compress each of the key and value vectors using residual quantizations <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib18" title="" class="ltx_ref">18</a>]</cite> built with FAISS <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib35" title="" class="ltx_ref">35</a>]</cite>.
The compression rate is <math id="S3.SS3.p5.1.m1.1" class="ltx_Math" alttext="80/7\approx 11.4" display="inline"><semantics id="S3.SS3.p5.1.m1.1a"><mrow id="S3.SS3.p5.1.m1.1.1" xref="S3.SS3.p5.1.m1.1.1.cmml"><mrow id="S3.SS3.p5.1.m1.1.1.2" xref="S3.SS3.p5.1.m1.1.1.2.cmml"><mn id="S3.SS3.p5.1.m1.1.1.2.2" xref="S3.SS3.p5.1.m1.1.1.2.2.cmml">80</mn><mo id="S3.SS3.p5.1.m1.1.1.2.1" xref="S3.SS3.p5.1.m1.1.1.2.1.cmml">/</mo><mn id="S3.SS3.p5.1.m1.1.1.2.3" xref="S3.SS3.p5.1.m1.1.1.2.3.cmml">7</mn></mrow><mo id="S3.SS3.p5.1.m1.1.1.1" xref="S3.SS3.p5.1.m1.1.1.1.cmml">≈</mo><mn id="S3.SS3.p5.1.m1.1.1.3" xref="S3.SS3.p5.1.m1.1.1.3.cmml">11.4</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p5.1.m1.1b"><apply id="S3.SS3.p5.1.m1.1.1.cmml" xref="S3.SS3.p5.1.m1.1.1"><approx id="S3.SS3.p5.1.m1.1.1.1.cmml" xref="S3.SS3.p5.1.m1.1.1.1"></approx><apply id="S3.SS3.p5.1.m1.1.1.2.cmml" xref="S3.SS3.p5.1.m1.1.1.2"><divide id="S3.SS3.p5.1.m1.1.1.2.1.cmml" xref="S3.SS3.p5.1.m1.1.1.2.1"></divide><cn type="integer" id="S3.SS3.p5.1.m1.1.1.2.2.cmml" xref="S3.SS3.p5.1.m1.1.1.2.2">80</cn><cn type="integer" id="S3.SS3.p5.1.m1.1.1.2.3.cmml" xref="S3.SS3.p5.1.m1.1.1.2.3">7</cn></apply><cn type="float" id="S3.SS3.p5.1.m1.1.1.3.cmml" xref="S3.SS3.p5.1.m1.1.1.3">11.4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p5.1.m1.1c">80/7\approx 11.4</annotation></semantics></math>.
During inference, the retrieved memories are first loaded from drives, and then decompressed by the vector quantizer before being sent to GPU.
The evaluations in Section <a href="#S7.SS1" title="7.1 | General Abilities ‣ 7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7.1</span></a> indicate that this compression has negligible influence on the performance of Memory<sup id="S3.SS3.p5.2.1" class="ltx_sup">3</sup>.
More details can be found in Appendix <a href="#A2" title="Appendix B Vector Compression ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a>.</p>
</div>
<div id="S3.SS3.p6" class="ltx_para">
<p id="S3.SS3.p6.1" class="ltx_p">Hence, the total sparsity is 160 or 1830 (without or with vector compression).
Originally, the explicit memory bank would have an enormous size of 7.17PB or equivalently 7340TB (given the model shape described in Section <a href="#S3.SS4" title="3.4 | Model Shape ‣ 3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.4</span></a> and saved in bfloat16).
Our compression brings it down to 45.9TB or 4.02TB (without or with vector compression), both acceptable for the drive storage of a GPU cluster.</p>
</div>
<div id="S3.SS3.p7" class="ltx_para">
<p id="S3.SS3.p7.1" class="ltx_p">To deploy the Memory<sup id="S3.SS3.p7.1.1" class="ltx_sup">3</sup> model on end-side devices such as smart phones and laptops, one can place the explicit memory bank and the vector index on a cloud server, while the devices only need to store the model parameters and the decoder of the vector quantizer.
During inference, to perform retrieval, the model on the end-side device sends the query vector to the cloud server, which then searches the index and returns the compressed memories.
The speed test of this deployment is recorded in Section <a href="#S7.SS5" title="7.5 | Inference Speed ‣ 7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7.5</span></a>.</p>
</div>
<div id="Thmremark4" class="ltx_theorem ltx_theorem_remark">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmremark4.1.1.1" class="ltx_text ltx_font_bold">Remark 4</span></span><span id="Thmremark4.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmremark4.p1" class="ltx_para">
<p id="Thmremark4.p1.7" class="ltx_p">If one wants to finetune a pretrained LLM into a Memory<sup id="Thmremark4.p1.7.1" class="ltx_sup">3</sup> model, there are several ways to select a small but effective subset of attention heads (among all heads at all layers) for memory heads (<a href="#E4" title="Equation 4 ‣ 3.2 | Writing and Reading Memory ‣ 3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>).
Methods such as <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib122" title="" class="ltx_ref">122</a>, <a href="#bib.bib39" title="" class="ltx_ref">39</a>]</cite> are proposed to identify the heads that contribute the most to long-context modeling by retrieving useful information from distant tokens, and usually these special heads account for only <math id="Thmremark4.p1.2.m2.1" class="ltx_Math" alttext="&lt;10\%" display="inline"><semantics id="Thmremark4.p1.2.m2.1a"><mrow id="Thmremark4.p1.2.m2.1.1" xref="Thmremark4.p1.2.m2.1.1.cmml"><mi id="Thmremark4.p1.2.m2.1.1.2" xref="Thmremark4.p1.2.m2.1.1.2.cmml"></mi><mo id="Thmremark4.p1.2.m2.1.1.1" xref="Thmremark4.p1.2.m2.1.1.1.cmml">&lt;</mo><mrow id="Thmremark4.p1.2.m2.1.1.3" xref="Thmremark4.p1.2.m2.1.1.3.cmml"><mn id="Thmremark4.p1.2.m2.1.1.3.2" xref="Thmremark4.p1.2.m2.1.1.3.2.cmml">10</mn><mo id="Thmremark4.p1.2.m2.1.1.3.1" xref="Thmremark4.p1.2.m2.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="Thmremark4.p1.2.m2.1b"><apply id="Thmremark4.p1.2.m2.1.1.cmml" xref="Thmremark4.p1.2.m2.1.1"><lt id="Thmremark4.p1.2.m2.1.1.1.cmml" xref="Thmremark4.p1.2.m2.1.1.1"></lt><csymbol cd="latexml" id="Thmremark4.p1.2.m2.1.1.2.cmml" xref="Thmremark4.p1.2.m2.1.1.2">absent</csymbol><apply id="Thmremark4.p1.2.m2.1.1.3.cmml" xref="Thmremark4.p1.2.m2.1.1.3"><csymbol cd="latexml" id="Thmremark4.p1.2.m2.1.1.3.1.cmml" xref="Thmremark4.p1.2.m2.1.1.3.1">percent</csymbol><cn type="integer" id="Thmremark4.p1.2.m2.1.1.3.2.cmml" xref="Thmremark4.p1.2.m2.1.1.3.2">10</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmremark4.p1.2.m2.1c">&lt;10\%</annotation></semantics></math> of the total heads.
Here we also propose a simple method for selecting memory heads:
Given the validation subsets of a representative collection of evaluation tasks, one can measure the average performance <math id="Thmremark4.p1.3.m3.1" class="ltx_Math" alttext="s_{h}" display="inline"><semantics id="Thmremark4.p1.3.m3.1a"><msub id="Thmremark4.p1.3.m3.1.1" xref="Thmremark4.p1.3.m3.1.1.cmml"><mi id="Thmremark4.p1.3.m3.1.1.2" xref="Thmremark4.p1.3.m3.1.1.2.cmml">s</mi><mi id="Thmremark4.p1.3.m3.1.1.3" xref="Thmremark4.p1.3.m3.1.1.3.cmml">h</mi></msub><annotation-xml encoding="MathML-Content" id="Thmremark4.p1.3.m3.1b"><apply id="Thmremark4.p1.3.m3.1.1.cmml" xref="Thmremark4.p1.3.m3.1.1"><csymbol cd="ambiguous" id="Thmremark4.p1.3.m3.1.1.1.cmml" xref="Thmremark4.p1.3.m3.1.1">subscript</csymbol><ci id="Thmremark4.p1.3.m3.1.1.2.cmml" xref="Thmremark4.p1.3.m3.1.1.2">𝑠</ci><ci id="Thmremark4.p1.3.m3.1.1.3.cmml" xref="Thmremark4.p1.3.m3.1.1.3">ℎ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmremark4.p1.3.m3.1c">s_{h}</annotation></semantics></math> for a modified version of the LLM for each attention head <math id="Thmremark4.p1.4.m4.1" class="ltx_Math" alttext="h" display="inline"><semantics id="Thmremark4.p1.4.m4.1a"><mi id="Thmremark4.p1.4.m4.1.1" xref="Thmremark4.p1.4.m4.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="Thmremark4.p1.4.m4.1b"><ci id="Thmremark4.p1.4.m4.1.1.cmml" xref="Thmremark4.p1.4.m4.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmremark4.p1.4.m4.1c">h</annotation></semantics></math>.
The modification masks the distant tokens for head <math id="Thmremark4.p1.5.m5.1" class="ltx_Math" alttext="h" display="inline"><semantics id="Thmremark4.p1.5.m5.1a"><mi id="Thmremark4.p1.5.m5.1.1" xref="Thmremark4.p1.5.m5.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="Thmremark4.p1.5.m5.1b"><ci id="Thmremark4.p1.5.m5.1.1.cmml" xref="Thmremark4.p1.5.m5.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmremark4.p1.5.m5.1c">h</annotation></semantics></math> so it can only see the preceding 100 tokens and the BOS token.
Then, it is reasonable to expect that <math id="Thmremark4.p1.6.m6.1" class="ltx_Math" alttext="s_{h}" display="inline"><semantics id="Thmremark4.p1.6.m6.1a"><msub id="Thmremark4.p1.6.m6.1.1" xref="Thmremark4.p1.6.m6.1.1.cmml"><mi id="Thmremark4.p1.6.m6.1.1.2" xref="Thmremark4.p1.6.m6.1.1.2.cmml">s</mi><mi id="Thmremark4.p1.6.m6.1.1.3" xref="Thmremark4.p1.6.m6.1.1.3.cmml">h</mi></msub><annotation-xml encoding="MathML-Content" id="Thmremark4.p1.6.m6.1b"><apply id="Thmremark4.p1.6.m6.1.1.cmml" xref="Thmremark4.p1.6.m6.1.1"><csymbol cd="ambiguous" id="Thmremark4.p1.6.m6.1.1.1.cmml" xref="Thmremark4.p1.6.m6.1.1">subscript</csymbol><ci id="Thmremark4.p1.6.m6.1.1.2.cmml" xref="Thmremark4.p1.6.m6.1.1.2">𝑠</ci><ci id="Thmremark4.p1.6.m6.1.1.3.cmml" xref="Thmremark4.p1.6.m6.1.1.3">ℎ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmremark4.p1.6.m6.1c">s_{h}</annotation></semantics></math> would be markedly low for a small subset of heads <math id="Thmremark4.p1.7.m7.1" class="ltx_Math" alttext="h" display="inline"><semantics id="Thmremark4.p1.7.m7.1a"><mi id="Thmremark4.p1.7.m7.1.1" xref="Thmremark4.p1.7.m7.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="Thmremark4.p1.7.m7.1b"><ci id="Thmremark4.p1.7.m7.1.1.cmml" xref="Thmremark4.p1.7.m7.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="Thmremark4.p1.7.m7.1c">h</annotation></semantics></math>, indicating that they are specialized for long-range attention.</p>
</div>
</div>
<div id="Thmremark5" class="ltx_theorem ltx_theorem_remark">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmremark5.1.1.1" class="ltx_text ltx_font_bold">Remark 5</span></span><span id="Thmremark5.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmremark5.p1" class="ltx_para">
<p id="Thmremark5.p1.1" class="ltx_p">Actually, Remark <a href="#Thmremark1" title="Remark 1. ‣ 2.3 | Memory ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> suggests that each reference only needs to be attended to by just one attention head, although in general this special head may be different among the references.
Thus, it seems a promising approach to apply adaptive sparsity not only to token selection, but also to the memory heads, namely each reference is routed to one or two heads (analogously to MoE), and its explicit memory is produced and read by these heads.
Such design if feasible can further boost the sparsity of explicit memory and save much more space.</p>
</div>
</div>
</section>
<section id="S3.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsection" style="font-size:120%;color:#00868B;">3.4 <span id="S3.SS4.1.1" class="ltx_text" style="color:#808080;">| </span>Model Shape</h3>

<div id="S3.SS4.p1" class="ltx_para">
<p id="S3.SS4.p1.5" class="ltx_p">As discussed in Section <a href="#S2.SS3" title="2.3 | Memory ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.3</span></a>, the specific knowledges can be externalized to explicit memories, and thus to minimize the total cost (<a href="#E1" title="Equation 1 ‣ 1 | Introduction ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>), the model parameters (or implicit memory) only need to store abstract knowledges and the subset of specific knowledges that are frequently used.
The shape of our model, i.e. (the number of Transformer blocks <math id="S3.SS4.p1.1.m1.1" class="ltx_Math" alttext="L" display="inline"><semantics id="S3.SS4.p1.1.m1.1a"><mi id="S3.SS4.p1.1.m1.1.1" xref="S3.SS4.p1.1.m1.1.1.cmml">L</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p1.1.m1.1b"><ci id="S3.SS4.p1.1.m1.1.1.cmml" xref="S3.SS4.p1.1.m1.1.1">𝐿</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p1.1.m1.1c">L</annotation></semantics></math>, heads <math id="S3.SS4.p1.2.m2.1" class="ltx_Math" alttext="H" display="inline"><semantics id="S3.SS4.p1.2.m2.1a"><mi id="S3.SS4.p1.2.m2.1.1" xref="S3.SS4.p1.2.m2.1.1.cmml">H</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p1.2.m2.1b"><ci id="S3.SS4.p1.2.m2.1.1.cmml" xref="S3.SS4.p1.2.m2.1.1">𝐻</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p1.2.m2.1c">H</annotation></semantics></math>, head dimension <math id="S3.SS4.p1.3.m3.1" class="ltx_Math" alttext="d_{h}" display="inline"><semantics id="S3.SS4.p1.3.m3.1a"><msub id="S3.SS4.p1.3.m3.1.1" xref="S3.SS4.p1.3.m3.1.1.cmml"><mi id="S3.SS4.p1.3.m3.1.1.2" xref="S3.SS4.p1.3.m3.1.1.2.cmml">d</mi><mi id="S3.SS4.p1.3.m3.1.1.3" xref="S3.SS4.p1.3.m3.1.1.3.cmml">h</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS4.p1.3.m3.1b"><apply id="S3.SS4.p1.3.m3.1.1.cmml" xref="S3.SS4.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS4.p1.3.m3.1.1.1.cmml" xref="S3.SS4.p1.3.m3.1.1">subscript</csymbol><ci id="S3.SS4.p1.3.m3.1.1.2.cmml" xref="S3.SS4.p1.3.m3.1.1.2">𝑑</ci><ci id="S3.SS4.p1.3.m3.1.1.3.cmml" xref="S3.SS4.p1.3.m3.1.1.3">ℎ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p1.3.m3.1c">d_{h}</annotation></semantics></math>, width of the MLP layers <math id="S3.SS4.p1.4.m4.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S3.SS4.p1.4.m4.1a"><mi id="S3.SS4.p1.4.m4.1.1" xref="S3.SS4.p1.4.m4.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p1.4.m4.1b"><ci id="S3.SS4.p1.4.m4.1.1.cmml" xref="S3.SS4.p1.4.m4.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p1.4.m4.1c">W</annotation></semantics></math>), is chosen to accommodate this desired knowledge distribution.
Informally speaking, given a fixed parameter size <math id="S3.SS4.p1.5.m5.1" class="ltx_Math" alttext="P" display="inline"><semantics id="S3.SS4.p1.5.m5.1a"><mi id="S3.SS4.p1.5.m5.1.1" xref="S3.SS4.p1.5.m5.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p1.5.m5.1b"><ci id="S3.SS4.p1.5.m5.1.1.cmml" xref="S3.SS4.p1.5.m5.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p1.5.m5.1c">P</annotation></semantics></math>, the shape maximizes the following objective</p>
<table id="E5" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="E5.m1.9" class="ltx_Math" alttext="\max_{L,H,d_{h},W}\Big{\{}\frac{\text{capacity for abstract knowledge}}{\text{capacity for specific knowledge}}~{}\Big{|}~{}\text{size}(L,H,d_{h},W)\approx P\Big{\}}" display="block"><semantics id="E5.m1.9a"><mrow id="E5.m1.9.9.2" xref="E5.m1.9.9.3.cmml"><munder id="E5.m1.8.8.1.1" xref="E5.m1.8.8.1.1.cmml"><mi id="E5.m1.8.8.1.1.2" xref="E5.m1.8.8.1.1.2.cmml">max</mi><mrow id="E5.m1.4.4.4.4" xref="E5.m1.4.4.4.5.cmml"><mi id="E5.m1.1.1.1.1" xref="E5.m1.1.1.1.1.cmml">L</mi><mo id="E5.m1.4.4.4.4.2" xref="E5.m1.4.4.4.5.cmml">,</mo><mi id="E5.m1.2.2.2.2" xref="E5.m1.2.2.2.2.cmml">H</mi><mo id="E5.m1.4.4.4.4.3" xref="E5.m1.4.4.4.5.cmml">,</mo><msub id="E5.m1.4.4.4.4.1" xref="E5.m1.4.4.4.4.1.cmml"><mi id="E5.m1.4.4.4.4.1.2" xref="E5.m1.4.4.4.4.1.2.cmml">d</mi><mi id="E5.m1.4.4.4.4.1.3" xref="E5.m1.4.4.4.4.1.3.cmml">h</mi></msub><mo id="E5.m1.4.4.4.4.4" xref="E5.m1.4.4.4.5.cmml">,</mo><mi id="E5.m1.3.3.3.3" xref="E5.m1.3.3.3.3.cmml">W</mi></mrow></munder><mo id="E5.m1.9.9.2a" xref="E5.m1.9.9.3.cmml">⁡</mo><mrow id="E5.m1.9.9.2.2" xref="E5.m1.9.9.3.cmml"><mo maxsize="160%" minsize="160%" id="E5.m1.9.9.2.2.2" xref="E5.m1.9.9.3.cmml">{</mo><mrow id="E5.m1.9.9.2.2.1" xref="E5.m1.9.9.2.2.1.cmml"><mrow id="E5.m1.9.9.2.2.1.1" xref="E5.m1.9.9.2.2.1.1.cmml"><mfrac id="E5.m1.9.9.2.2.1.1.3" xref="E5.m1.9.9.2.2.1.1.3.cmml"><mtext id="E5.m1.9.9.2.2.1.1.3.2" xref="E5.m1.9.9.2.2.1.1.3.2a.cmml">capacity for abstract knowledge</mtext><mtext id="E5.m1.9.9.2.2.1.1.3.3" xref="E5.m1.9.9.2.2.1.1.3.3a.cmml">capacity for specific knowledge</mtext></mfrac><mo fence="false" lspace="0.608em" mathsize="160%" rspace="0.608em" id="E5.m1.9.9.2.2.1.1.2" xref="E5.m1.9.9.2.2.1.1.2.cmml">|</mo><mrow id="E5.m1.9.9.2.2.1.1.1" xref="E5.m1.9.9.2.2.1.1.1.cmml"><mtext id="E5.m1.9.9.2.2.1.1.1.3" xref="E5.m1.9.9.2.2.1.1.1.3a.cmml">size</mtext><mo lspace="0em" rspace="0em" id="E5.m1.9.9.2.2.1.1.1.2" xref="E5.m1.9.9.2.2.1.1.1.2.cmml">​</mo><mrow id="E5.m1.9.9.2.2.1.1.1.1.1" xref="E5.m1.9.9.2.2.1.1.1.1.2.cmml"><mo stretchy="false" id="E5.m1.9.9.2.2.1.1.1.1.1.2" xref="E5.m1.9.9.2.2.1.1.1.1.2.cmml">(</mo><mi id="E5.m1.5.5" xref="E5.m1.5.5.cmml">L</mi><mo id="E5.m1.9.9.2.2.1.1.1.1.1.3" xref="E5.m1.9.9.2.2.1.1.1.1.2.cmml">,</mo><mi id="E5.m1.6.6" xref="E5.m1.6.6.cmml">H</mi><mo id="E5.m1.9.9.2.2.1.1.1.1.1.4" xref="E5.m1.9.9.2.2.1.1.1.1.2.cmml">,</mo><msub id="E5.m1.9.9.2.2.1.1.1.1.1.1" xref="E5.m1.9.9.2.2.1.1.1.1.1.1.cmml"><mi id="E5.m1.9.9.2.2.1.1.1.1.1.1.2" xref="E5.m1.9.9.2.2.1.1.1.1.1.1.2.cmml">d</mi><mi id="E5.m1.9.9.2.2.1.1.1.1.1.1.3" xref="E5.m1.9.9.2.2.1.1.1.1.1.1.3.cmml">h</mi></msub><mo id="E5.m1.9.9.2.2.1.1.1.1.1.5" xref="E5.m1.9.9.2.2.1.1.1.1.2.cmml">,</mo><mi id="E5.m1.7.7" xref="E5.m1.7.7.cmml">W</mi><mo stretchy="false" id="E5.m1.9.9.2.2.1.1.1.1.1.6" xref="E5.m1.9.9.2.2.1.1.1.1.2.cmml">)</mo></mrow></mrow></mrow><mo id="E5.m1.9.9.2.2.1.2" xref="E5.m1.9.9.2.2.1.2.cmml">≈</mo><mi id="E5.m1.9.9.2.2.1.3" xref="E5.m1.9.9.2.2.1.3.cmml">P</mi></mrow><mo maxsize="160%" minsize="160%" id="E5.m1.9.9.2.2.3" xref="E5.m1.9.9.3.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="E5.m1.9b"><apply id="E5.m1.9.9.3.cmml" xref="E5.m1.9.9.2"><apply id="E5.m1.8.8.1.1.cmml" xref="E5.m1.8.8.1.1"><csymbol cd="ambiguous" id="E5.m1.8.8.1.1.1.cmml" xref="E5.m1.8.8.1.1">subscript</csymbol><max id="E5.m1.8.8.1.1.2.cmml" xref="E5.m1.8.8.1.1.2"></max><list id="E5.m1.4.4.4.5.cmml" xref="E5.m1.4.4.4.4"><ci id="E5.m1.1.1.1.1.cmml" xref="E5.m1.1.1.1.1">𝐿</ci><ci id="E5.m1.2.2.2.2.cmml" xref="E5.m1.2.2.2.2">𝐻</ci><apply id="E5.m1.4.4.4.4.1.cmml" xref="E5.m1.4.4.4.4.1"><csymbol cd="ambiguous" id="E5.m1.4.4.4.4.1.1.cmml" xref="E5.m1.4.4.4.4.1">subscript</csymbol><ci id="E5.m1.4.4.4.4.1.2.cmml" xref="E5.m1.4.4.4.4.1.2">𝑑</ci><ci id="E5.m1.4.4.4.4.1.3.cmml" xref="E5.m1.4.4.4.4.1.3">ℎ</ci></apply><ci id="E5.m1.3.3.3.3.cmml" xref="E5.m1.3.3.3.3">𝑊</ci></list></apply><apply id="E5.m1.9.9.2.2.1.cmml" xref="E5.m1.9.9.2.2.1"><approx id="E5.m1.9.9.2.2.1.2.cmml" xref="E5.m1.9.9.2.2.1.2"></approx><apply id="E5.m1.9.9.2.2.1.1.cmml" xref="E5.m1.9.9.2.2.1.1"><csymbol cd="latexml" id="E5.m1.9.9.2.2.1.1.2.cmml" xref="E5.m1.9.9.2.2.1.1.2">conditional</csymbol><apply id="E5.m1.9.9.2.2.1.1.3.cmml" xref="E5.m1.9.9.2.2.1.1.3"><divide id="E5.m1.9.9.2.2.1.1.3.1.cmml" xref="E5.m1.9.9.2.2.1.1.3"></divide><ci id="E5.m1.9.9.2.2.1.1.3.2a.cmml" xref="E5.m1.9.9.2.2.1.1.3.2"><mtext id="E5.m1.9.9.2.2.1.1.3.2.cmml" xref="E5.m1.9.9.2.2.1.1.3.2">capacity for abstract knowledge</mtext></ci><ci id="E5.m1.9.9.2.2.1.1.3.3a.cmml" xref="E5.m1.9.9.2.2.1.1.3.3"><mtext id="E5.m1.9.9.2.2.1.1.3.3.cmml" xref="E5.m1.9.9.2.2.1.1.3.3">capacity for specific knowledge</mtext></ci></apply><apply id="E5.m1.9.9.2.2.1.1.1.cmml" xref="E5.m1.9.9.2.2.1.1.1"><times id="E5.m1.9.9.2.2.1.1.1.2.cmml" xref="E5.m1.9.9.2.2.1.1.1.2"></times><ci id="E5.m1.9.9.2.2.1.1.1.3a.cmml" xref="E5.m1.9.9.2.2.1.1.1.3"><mtext id="E5.m1.9.9.2.2.1.1.1.3.cmml" xref="E5.m1.9.9.2.2.1.1.1.3">size</mtext></ci><vector id="E5.m1.9.9.2.2.1.1.1.1.2.cmml" xref="E5.m1.9.9.2.2.1.1.1.1.1"><ci id="E5.m1.5.5.cmml" xref="E5.m1.5.5">𝐿</ci><ci id="E5.m1.6.6.cmml" xref="E5.m1.6.6">𝐻</ci><apply id="E5.m1.9.9.2.2.1.1.1.1.1.1.cmml" xref="E5.m1.9.9.2.2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="E5.m1.9.9.2.2.1.1.1.1.1.1.1.cmml" xref="E5.m1.9.9.2.2.1.1.1.1.1.1">subscript</csymbol><ci id="E5.m1.9.9.2.2.1.1.1.1.1.1.2.cmml" xref="E5.m1.9.9.2.2.1.1.1.1.1.1.2">𝑑</ci><ci id="E5.m1.9.9.2.2.1.1.1.1.1.1.3.cmml" xref="E5.m1.9.9.2.2.1.1.1.1.1.1.3">ℎ</ci></apply><ci id="E5.m1.7.7.cmml" xref="E5.m1.7.7">𝑊</ci></vector></apply></apply><ci id="E5.m1.9.9.2.2.1.3.cmml" xref="E5.m1.9.9.2.2.1.3">𝑃</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="E5.m1.9c">\max_{L,H,d_{h},W}\Big{\{}\frac{\text{capacity for abstract knowledge}}{\text{capacity for specific knowledge}}~{}\Big{|}~{}\text{size}(L,H,d_{h},W)\approx P\Big{\}}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(5)</span></td>
</tr></tbody>
</table>
<p id="S3.SS4.p1.6" class="ltx_p">Here we set <math id="S3.SS4.p1.6.m1.1" class="ltx_Math" alttext="P" display="inline"><semantics id="S3.SS4.p1.6.m1.1a"><mi id="S3.SS4.p1.6.m1.1.1" xref="S3.SS4.p1.6.m1.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p1.6.m1.1b"><ci id="S3.SS4.p1.6.m1.1.1.cmml" xref="S3.SS4.p1.6.m1.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p1.6.m1.1c">P</annotation></semantics></math> to be 2.4 billion.</p>
</div>
<div id="S3.SS4.p2" class="ltx_para">
<p id="S3.SS4.p2.2" class="ltx_p">Some recent works suggest that the capacities for learning specific knowledges and abstract knowledges are subject to different constraints.
On one hand, <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib29" title="" class="ltx_ref">29</a>]</cite> observes that the amount of bits of trivia information (such as a person’s name, date of birth and job title) that a LLM can store depends only on its parameter size.
Regardless of <math id="S3.SS4.p2.1.m1.1" class="ltx_Math" alttext="L" display="inline"><semantics id="S3.SS4.p2.1.m1.1a"><mi id="S3.SS4.p2.1.m1.1.1" xref="S3.SS4.p2.1.m1.1.1.cmml">L</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.1.m1.1b"><ci id="S3.SS4.p2.1.m1.1.1.cmml" xref="S3.SS4.p2.1.m1.1.1">𝐿</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.1.m1.1c">L</annotation></semantics></math> and <math id="S3.SS4.p2.2.m2.1" class="ltx_Math" alttext="H" display="inline"><semantics id="S3.SS4.p2.2.m2.1a"><mi id="S3.SS4.p2.2.m2.1.1" xref="S3.SS4.p2.2.m2.1.1.cmml">H</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.2.m2.1b"><ci id="S3.SS4.p2.2.m2.1.1.cmml" xref="S3.SS4.p2.2.m2.1.1">𝐻</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.2.m2.1c">H</annotation></semantics></math>, the max capacity is always around 2 bits per parameter.</p>
</div>
<div id="S3.SS4.p3" class="ltx_para">
<p id="S3.SS4.p3.8" class="ltx_p">On the other hand, <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib120" title="" class="ltx_ref">120</a>]</cite> trains Transformers to learn simple algorithms such as reversing a list and counting the occurrence of each letter.
It is observed that for several such tasks, there exists a minimum <math id="S3.SS4.p3.1.m1.1" class="ltx_Math" alttext="L_{0}" display="inline"><semantics id="S3.SS4.p3.1.m1.1a"><msub id="S3.SS4.p3.1.m1.1.1" xref="S3.SS4.p3.1.m1.1.1.cmml"><mi id="S3.SS4.p3.1.m1.1.1.2" xref="S3.SS4.p3.1.m1.1.1.2.cmml">L</mi><mn id="S3.SS4.p3.1.m1.1.1.3" xref="S3.SS4.p3.1.m1.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="S3.SS4.p3.1.m1.1b"><apply id="S3.SS4.p3.1.m1.1.1.cmml" xref="S3.SS4.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS4.p3.1.m1.1.1.1.cmml" xref="S3.SS4.p3.1.m1.1.1">subscript</csymbol><ci id="S3.SS4.p3.1.m1.1.1.2.cmml" xref="S3.SS4.p3.1.m1.1.1.2">𝐿</ci><cn type="integer" id="S3.SS4.p3.1.m1.1.1.3.cmml" xref="S3.SS4.p3.1.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p3.1.m1.1c">L_{0}</annotation></semantics></math> and <math id="S3.SS4.p3.2.m2.1" class="ltx_Math" alttext="H_{0}" display="inline"><semantics id="S3.SS4.p3.2.m2.1a"><msub id="S3.SS4.p3.2.m2.1.1" xref="S3.SS4.p3.2.m2.1.1.cmml"><mi id="S3.SS4.p3.2.m2.1.1.2" xref="S3.SS4.p3.2.m2.1.1.2.cmml">H</mi><mn id="S3.SS4.p3.2.m2.1.1.3" xref="S3.SS4.p3.2.m2.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="S3.SS4.p3.2.m2.1b"><apply id="S3.SS4.p3.2.m2.1.1.cmml" xref="S3.SS4.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS4.p3.2.m2.1.1.1.cmml" xref="S3.SS4.p3.2.m2.1.1">subscript</csymbol><ci id="S3.SS4.p3.2.m2.1.1.2.cmml" xref="S3.SS4.p3.2.m2.1.1.2">𝐻</ci><cn type="integer" id="S3.SS4.p3.2.m2.1.1.3.cmml" xref="S3.SS4.p3.2.m2.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p3.2.m2.1c">H_{0}</annotation></semantics></math> such that a Transformer with <math id="S3.SS4.p3.3.m3.1" class="ltx_Math" alttext="L\geq L_{0}" display="inline"><semantics id="S3.SS4.p3.3.m3.1a"><mrow id="S3.SS4.p3.3.m3.1.1" xref="S3.SS4.p3.3.m3.1.1.cmml"><mi id="S3.SS4.p3.3.m3.1.1.2" xref="S3.SS4.p3.3.m3.1.1.2.cmml">L</mi><mo id="S3.SS4.p3.3.m3.1.1.1" xref="S3.SS4.p3.3.m3.1.1.1.cmml">≥</mo><msub id="S3.SS4.p3.3.m3.1.1.3" xref="S3.SS4.p3.3.m3.1.1.3.cmml"><mi id="S3.SS4.p3.3.m3.1.1.3.2" xref="S3.SS4.p3.3.m3.1.1.3.2.cmml">L</mi><mn id="S3.SS4.p3.3.m3.1.1.3.3" xref="S3.SS4.p3.3.m3.1.1.3.3.cmml">0</mn></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SS4.p3.3.m3.1b"><apply id="S3.SS4.p3.3.m3.1.1.cmml" xref="S3.SS4.p3.3.m3.1.1"><geq id="S3.SS4.p3.3.m3.1.1.1.cmml" xref="S3.SS4.p3.3.m3.1.1.1"></geq><ci id="S3.SS4.p3.3.m3.1.1.2.cmml" xref="S3.SS4.p3.3.m3.1.1.2">𝐿</ci><apply id="S3.SS4.p3.3.m3.1.1.3.cmml" xref="S3.SS4.p3.3.m3.1.1.3"><csymbol cd="ambiguous" id="S3.SS4.p3.3.m3.1.1.3.1.cmml" xref="S3.SS4.p3.3.m3.1.1.3">subscript</csymbol><ci id="S3.SS4.p3.3.m3.1.1.3.2.cmml" xref="S3.SS4.p3.3.m3.1.1.3.2">𝐿</ci><cn type="integer" id="S3.SS4.p3.3.m3.1.1.3.3.cmml" xref="S3.SS4.p3.3.m3.1.1.3.3">0</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p3.3.m3.1c">L\geq L_{0}</annotation></semantics></math> and <math id="S3.SS4.p3.4.m4.1" class="ltx_Math" alttext="H\geq H_{0}" display="inline"><semantics id="S3.SS4.p3.4.m4.1a"><mrow id="S3.SS4.p3.4.m4.1.1" xref="S3.SS4.p3.4.m4.1.1.cmml"><mi id="S3.SS4.p3.4.m4.1.1.2" xref="S3.SS4.p3.4.m4.1.1.2.cmml">H</mi><mo id="S3.SS4.p3.4.m4.1.1.1" xref="S3.SS4.p3.4.m4.1.1.1.cmml">≥</mo><msub id="S3.SS4.p3.4.m4.1.1.3" xref="S3.SS4.p3.4.m4.1.1.3.cmml"><mi id="S3.SS4.p3.4.m4.1.1.3.2" xref="S3.SS4.p3.4.m4.1.1.3.2.cmml">H</mi><mn id="S3.SS4.p3.4.m4.1.1.3.3" xref="S3.SS4.p3.4.m4.1.1.3.3.cmml">0</mn></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SS4.p3.4.m4.1b"><apply id="S3.SS4.p3.4.m4.1.1.cmml" xref="S3.SS4.p3.4.m4.1.1"><geq id="S3.SS4.p3.4.m4.1.1.1.cmml" xref="S3.SS4.p3.4.m4.1.1.1"></geq><ci id="S3.SS4.p3.4.m4.1.1.2.cmml" xref="S3.SS4.p3.4.m4.1.1.2">𝐻</ci><apply id="S3.SS4.p3.4.m4.1.1.3.cmml" xref="S3.SS4.p3.4.m4.1.1.3"><csymbol cd="ambiguous" id="S3.SS4.p3.4.m4.1.1.3.1.cmml" xref="S3.SS4.p3.4.m4.1.1.3">subscript</csymbol><ci id="S3.SS4.p3.4.m4.1.1.3.2.cmml" xref="S3.SS4.p3.4.m4.1.1.3.2">𝐻</ci><cn type="integer" id="S3.SS4.p3.4.m4.1.1.3.3.cmml" xref="S3.SS4.p3.4.m4.1.1.3.3">0</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p3.4.m4.1c">H\geq H_{0}</annotation></semantics></math> can learn the task with perfect accuracy, whereas the accuracy drops significantly for Transformers with either <math id="S3.SS4.p3.5.m5.1" class="ltx_Math" alttext="L=L_{0}-1" display="inline"><semantics id="S3.SS4.p3.5.m5.1a"><mrow id="S3.SS4.p3.5.m5.1.1" xref="S3.SS4.p3.5.m5.1.1.cmml"><mi id="S3.SS4.p3.5.m5.1.1.2" xref="S3.SS4.p3.5.m5.1.1.2.cmml">L</mi><mo id="S3.SS4.p3.5.m5.1.1.1" xref="S3.SS4.p3.5.m5.1.1.1.cmml">=</mo><mrow id="S3.SS4.p3.5.m5.1.1.3" xref="S3.SS4.p3.5.m5.1.1.3.cmml"><msub id="S3.SS4.p3.5.m5.1.1.3.2" xref="S3.SS4.p3.5.m5.1.1.3.2.cmml"><mi id="S3.SS4.p3.5.m5.1.1.3.2.2" xref="S3.SS4.p3.5.m5.1.1.3.2.2.cmml">L</mi><mn id="S3.SS4.p3.5.m5.1.1.3.2.3" xref="S3.SS4.p3.5.m5.1.1.3.2.3.cmml">0</mn></msub><mo id="S3.SS4.p3.5.m5.1.1.3.1" xref="S3.SS4.p3.5.m5.1.1.3.1.cmml">−</mo><mn id="S3.SS4.p3.5.m5.1.1.3.3" xref="S3.SS4.p3.5.m5.1.1.3.3.cmml">1</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS4.p3.5.m5.1b"><apply id="S3.SS4.p3.5.m5.1.1.cmml" xref="S3.SS4.p3.5.m5.1.1"><eq id="S3.SS4.p3.5.m5.1.1.1.cmml" xref="S3.SS4.p3.5.m5.1.1.1"></eq><ci id="S3.SS4.p3.5.m5.1.1.2.cmml" xref="S3.SS4.p3.5.m5.1.1.2">𝐿</ci><apply id="S3.SS4.p3.5.m5.1.1.3.cmml" xref="S3.SS4.p3.5.m5.1.1.3"><minus id="S3.SS4.p3.5.m5.1.1.3.1.cmml" xref="S3.SS4.p3.5.m5.1.1.3.1"></minus><apply id="S3.SS4.p3.5.m5.1.1.3.2.cmml" xref="S3.SS4.p3.5.m5.1.1.3.2"><csymbol cd="ambiguous" id="S3.SS4.p3.5.m5.1.1.3.2.1.cmml" xref="S3.SS4.p3.5.m5.1.1.3.2">subscript</csymbol><ci id="S3.SS4.p3.5.m5.1.1.3.2.2.cmml" xref="S3.SS4.p3.5.m5.1.1.3.2.2">𝐿</ci><cn type="integer" id="S3.SS4.p3.5.m5.1.1.3.2.3.cmml" xref="S3.SS4.p3.5.m5.1.1.3.2.3">0</cn></apply><cn type="integer" id="S3.SS4.p3.5.m5.1.1.3.3.cmml" xref="S3.SS4.p3.5.m5.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p3.5.m5.1c">L=L_{0}-1</annotation></semantics></math> or <math id="S3.SS4.p3.6.m6.1" class="ltx_Math" alttext="H=H_{0}-1" display="inline"><semantics id="S3.SS4.p3.6.m6.1a"><mrow id="S3.SS4.p3.6.m6.1.1" xref="S3.SS4.p3.6.m6.1.1.cmml"><mi id="S3.SS4.p3.6.m6.1.1.2" xref="S3.SS4.p3.6.m6.1.1.2.cmml">H</mi><mo id="S3.SS4.p3.6.m6.1.1.1" xref="S3.SS4.p3.6.m6.1.1.1.cmml">=</mo><mrow id="S3.SS4.p3.6.m6.1.1.3" xref="S3.SS4.p3.6.m6.1.1.3.cmml"><msub id="S3.SS4.p3.6.m6.1.1.3.2" xref="S3.SS4.p3.6.m6.1.1.3.2.cmml"><mi id="S3.SS4.p3.6.m6.1.1.3.2.2" xref="S3.SS4.p3.6.m6.1.1.3.2.2.cmml">H</mi><mn id="S3.SS4.p3.6.m6.1.1.3.2.3" xref="S3.SS4.p3.6.m6.1.1.3.2.3.cmml">0</mn></msub><mo id="S3.SS4.p3.6.m6.1.1.3.1" xref="S3.SS4.p3.6.m6.1.1.3.1.cmml">−</mo><mn id="S3.SS4.p3.6.m6.1.1.3.3" xref="S3.SS4.p3.6.m6.1.1.3.3.cmml">1</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS4.p3.6.m6.1b"><apply id="S3.SS4.p3.6.m6.1.1.cmml" xref="S3.SS4.p3.6.m6.1.1"><eq id="S3.SS4.p3.6.m6.1.1.1.cmml" xref="S3.SS4.p3.6.m6.1.1.1"></eq><ci id="S3.SS4.p3.6.m6.1.1.2.cmml" xref="S3.SS4.p3.6.m6.1.1.2">𝐻</ci><apply id="S3.SS4.p3.6.m6.1.1.3.cmml" xref="S3.SS4.p3.6.m6.1.1.3"><minus id="S3.SS4.p3.6.m6.1.1.3.1.cmml" xref="S3.SS4.p3.6.m6.1.1.3.1"></minus><apply id="S3.SS4.p3.6.m6.1.1.3.2.cmml" xref="S3.SS4.p3.6.m6.1.1.3.2"><csymbol cd="ambiguous" id="S3.SS4.p3.6.m6.1.1.3.2.1.cmml" xref="S3.SS4.p3.6.m6.1.1.3.2">subscript</csymbol><ci id="S3.SS4.p3.6.m6.1.1.3.2.2.cmml" xref="S3.SS4.p3.6.m6.1.1.3.2.2">𝐻</ci><cn type="integer" id="S3.SS4.p3.6.m6.1.1.3.2.3.cmml" xref="S3.SS4.p3.6.m6.1.1.3.2.3">0</cn></apply><cn type="integer" id="S3.SS4.p3.6.m6.1.1.3.3.cmml" xref="S3.SS4.p3.6.m6.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p3.6.m6.1c">H=H_{0}-1</annotation></semantics></math> (given that either <math id="S3.SS4.p3.7.m7.1" class="ltx_Math" alttext="L_{0}" display="inline"><semantics id="S3.SS4.p3.7.m7.1a"><msub id="S3.SS4.p3.7.m7.1.1" xref="S3.SS4.p3.7.m7.1.1.cmml"><mi id="S3.SS4.p3.7.m7.1.1.2" xref="S3.SS4.p3.7.m7.1.1.2.cmml">L</mi><mn id="S3.SS4.p3.7.m7.1.1.3" xref="S3.SS4.p3.7.m7.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="S3.SS4.p3.7.m7.1b"><apply id="S3.SS4.p3.7.m7.1.1.cmml" xref="S3.SS4.p3.7.m7.1.1"><csymbol cd="ambiguous" id="S3.SS4.p3.7.m7.1.1.1.cmml" xref="S3.SS4.p3.7.m7.1.1">subscript</csymbol><ci id="S3.SS4.p3.7.m7.1.1.2.cmml" xref="S3.SS4.p3.7.m7.1.1.2">𝐿</ci><cn type="integer" id="S3.SS4.p3.7.m7.1.1.3.cmml" xref="S3.SS4.p3.7.m7.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p3.7.m7.1c">L_{0}</annotation></semantics></math> or <math id="S3.SS4.p3.8.m8.1" class="ltx_Math" alttext="H_{0}\geq 2" display="inline"><semantics id="S3.SS4.p3.8.m8.1a"><mrow id="S3.SS4.p3.8.m8.1.1" xref="S3.SS4.p3.8.m8.1.1.cmml"><msub id="S3.SS4.p3.8.m8.1.1.2" xref="S3.SS4.p3.8.m8.1.1.2.cmml"><mi id="S3.SS4.p3.8.m8.1.1.2.2" xref="S3.SS4.p3.8.m8.1.1.2.2.cmml">H</mi><mn id="S3.SS4.p3.8.m8.1.1.2.3" xref="S3.SS4.p3.8.m8.1.1.2.3.cmml">0</mn></msub><mo id="S3.SS4.p3.8.m8.1.1.1" xref="S3.SS4.p3.8.m8.1.1.1.cmml">≥</mo><mn id="S3.SS4.p3.8.m8.1.1.3" xref="S3.SS4.p3.8.m8.1.1.3.cmml">2</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS4.p3.8.m8.1b"><apply id="S3.SS4.p3.8.m8.1.1.cmml" xref="S3.SS4.p3.8.m8.1.1"><geq id="S3.SS4.p3.8.m8.1.1.1.cmml" xref="S3.SS4.p3.8.m8.1.1.1"></geq><apply id="S3.SS4.p3.8.m8.1.1.2.cmml" xref="S3.SS4.p3.8.m8.1.1.2"><csymbol cd="ambiguous" id="S3.SS4.p3.8.m8.1.1.2.1.cmml" xref="S3.SS4.p3.8.m8.1.1.2">subscript</csymbol><ci id="S3.SS4.p3.8.m8.1.1.2.2.cmml" xref="S3.SS4.p3.8.m8.1.1.2.2">𝐻</ci><cn type="integer" id="S3.SS4.p3.8.m8.1.1.2.3.cmml" xref="S3.SS4.p3.8.m8.1.1.2.3">0</cn></apply><cn type="integer" id="S3.SS4.p3.8.m8.1.1.3.cmml" xref="S3.SS4.p3.8.m8.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p3.8.m8.1c">H_{0}\geq 2</annotation></semantics></math>).
This sharp transition supports the view that the layers and heads of Transformer LLMs can be compared to algorithmic steps, and tasks with a certain level of complexity require at least a certain amount of steps.
It is worth mentioning that the emergent phenomenon <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib119" title="" class="ltx_ref">119</a>, <a href="#bib.bib105" title="" class="ltx_ref">105</a>]</cite> of LLMs can also be explained by this view and thus adds support to it, although it may not be the only explanation.</p>
</div>
<div id="S3.SS4.p4" class="ltx_para">
<p id="S3.SS4.p4.2" class="ltx_p">By Definition <a href="#Thmdefinition4" title="Definition 4. ‣ 2.2 | Knowledge ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, the abstract knowledges are expected to be circuits with greater complexity than specific knowledges, since their associated inputs and outputs exhibit greater variability and thus express more complex patterns.
It follows that, in the context of the aforementioned works, the separation of specific and abstract knowledges should be positively correlated with the distinction between trivia information and algorithmic procedures.
Hence, it is reasonable to adopt the approximation that the capacity of an LLM for specific knowledges only depends on its parameter size, whereas the capacity for abstract knowledges depends only on <math id="S3.SS4.p4.1.m1.1" class="ltx_Math" alttext="L" display="inline"><semantics id="S3.SS4.p4.1.m1.1a"><mi id="S3.SS4.p4.1.m1.1.1" xref="S3.SS4.p4.1.m1.1.1.cmml">L</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p4.1.m1.1b"><ci id="S3.SS4.p4.1.m1.1.1.cmml" xref="S3.SS4.p4.1.m1.1.1">𝐿</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p4.1.m1.1c">L</annotation></semantics></math> and <math id="S3.SS4.p4.2.m2.1" class="ltx_Math" alttext="H" display="inline"><semantics id="S3.SS4.p4.2.m2.1a"><mi id="S3.SS4.p4.2.m2.1.1" xref="S3.SS4.p4.2.m2.1.1.cmml">H</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p4.2.m2.1b"><ci id="S3.SS4.p4.2.m2.1.1.cmml" xref="S3.SS4.p4.2.m2.1.1">𝐻</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p4.2.m2.1c">H</annotation></semantics></math>.</p>
</div>
<div id="S3.SS4.p5" class="ltx_para">
<p id="S3.SS4.p5.13" class="ltx_p">The informal problem (<a href="#E5" title="Equation 5 ‣ 3.4 | Model Shape ‣ 3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>) reduces to the maximization of <math id="S3.SS4.p5.1.m1.1" class="ltx_Math" alttext="L" display="inline"><semantics id="S3.SS4.p5.1.m1.1a"><mi id="S3.SS4.p5.1.m1.1.1" xref="S3.SS4.p5.1.m1.1.1.cmml">L</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p5.1.m1.1b"><ci id="S3.SS4.p5.1.m1.1.1.cmml" xref="S3.SS4.p5.1.m1.1.1">𝐿</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p5.1.m1.1c">L</annotation></semantics></math> and <math id="S3.SS4.p5.2.m2.1" class="ltx_Math" alttext="H" display="inline"><semantics id="S3.SS4.p5.2.m2.1a"><mi id="S3.SS4.p5.2.m2.1.1" xref="S3.SS4.p5.2.m2.1.1.cmml">H</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p5.2.m2.1b"><ci id="S3.SS4.p5.2.m2.1.1.cmml" xref="S3.SS4.p5.2.m2.1.1">𝐻</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p5.2.m2.1c">H</annotation></semantics></math> given a fixed parameter size.
However, we are left with two ambiguities:
first, this formulation does not specify the ratio between <math id="S3.SS4.p5.3.m3.1" class="ltx_Math" alttext="L" display="inline"><semantics id="S3.SS4.p5.3.m3.1a"><mi id="S3.SS4.p5.3.m3.1.1" xref="S3.SS4.p5.3.m3.1.1.cmml">L</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p5.3.m3.1b"><ci id="S3.SS4.p5.3.m3.1.1.cmml" xref="S3.SS4.p5.3.m3.1.1">𝐿</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p5.3.m3.1c">L</annotation></semantics></math> and <math id="S3.SS4.p5.4.m4.1" class="ltx_Math" alttext="H" display="inline"><semantics id="S3.SS4.p5.4.m4.1a"><mi id="S3.SS4.p5.4.m4.1.1" xref="S3.SS4.p5.4.m4.1.1.cmml">H</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p5.4.m4.1b"><ci id="S3.SS4.p5.4.m4.1.1.cmml" xref="S3.SS4.p5.4.m4.1.1">𝐻</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p5.4.m4.1c">H</annotation></semantics></math>,
and second the head dimension <math id="S3.SS4.p5.5.m5.1" class="ltx_Math" alttext="d_{h}" display="inline"><semantics id="S3.SS4.p5.5.m5.1a"><msub id="S3.SS4.p5.5.m5.1.1" xref="S3.SS4.p5.5.m5.1.1.cmml"><mi id="S3.SS4.p5.5.m5.1.1.2" xref="S3.SS4.p5.5.m5.1.1.2.cmml">d</mi><mi id="S3.SS4.p5.5.m5.1.1.3" xref="S3.SS4.p5.5.m5.1.1.3.cmml">h</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS4.p5.5.m5.1b"><apply id="S3.SS4.p5.5.m5.1.1.cmml" xref="S3.SS4.p5.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS4.p5.5.m5.1.1.1.cmml" xref="S3.SS4.p5.5.m5.1.1">subscript</csymbol><ci id="S3.SS4.p5.5.m5.1.1.2.cmml" xref="S3.SS4.p5.5.m5.1.1.2">𝑑</ci><ci id="S3.SS4.p5.5.m5.1.1.3.cmml" xref="S3.SS4.p5.5.m5.1.1.3">ℎ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p5.5.m5.1c">d_{h}</annotation></semantics></math> and MLP width <math id="S3.SS4.p5.6.m6.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S3.SS4.p5.6.m6.1a"><mi id="S3.SS4.p5.6.m6.1.1" xref="S3.SS4.p5.6.m6.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p5.6.m6.1b"><ci id="S3.SS4.p5.6.m6.1.1.cmml" xref="S3.SS4.p5.6.m6.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p5.6.m6.1c">W</annotation></semantics></math> cannot be too small as the training may become unstable.
Regarding the second point, our experiments indicate that pretraining becomes more unstable with increased spikes if <math id="S3.SS4.p5.7.m7.1" class="ltx_Math" alttext="d_{h}\leq 64" display="inline"><semantics id="S3.SS4.p5.7.m7.1a"><mrow id="S3.SS4.p5.7.m7.1.1" xref="S3.SS4.p5.7.m7.1.1.cmml"><msub id="S3.SS4.p5.7.m7.1.1.2" xref="S3.SS4.p5.7.m7.1.1.2.cmml"><mi id="S3.SS4.p5.7.m7.1.1.2.2" xref="S3.SS4.p5.7.m7.1.1.2.2.cmml">d</mi><mi id="S3.SS4.p5.7.m7.1.1.2.3" xref="S3.SS4.p5.7.m7.1.1.2.3.cmml">h</mi></msub><mo id="S3.SS4.p5.7.m7.1.1.1" xref="S3.SS4.p5.7.m7.1.1.1.cmml">≤</mo><mn id="S3.SS4.p5.7.m7.1.1.3" xref="S3.SS4.p5.7.m7.1.1.3.cmml">64</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS4.p5.7.m7.1b"><apply id="S3.SS4.p5.7.m7.1.1.cmml" xref="S3.SS4.p5.7.m7.1.1"><leq id="S3.SS4.p5.7.m7.1.1.1.cmml" xref="S3.SS4.p5.7.m7.1.1.1"></leq><apply id="S3.SS4.p5.7.m7.1.1.2.cmml" xref="S3.SS4.p5.7.m7.1.1.2"><csymbol cd="ambiguous" id="S3.SS4.p5.7.m7.1.1.2.1.cmml" xref="S3.SS4.p5.7.m7.1.1.2">subscript</csymbol><ci id="S3.SS4.p5.7.m7.1.1.2.2.cmml" xref="S3.SS4.p5.7.m7.1.1.2.2">𝑑</ci><ci id="S3.SS4.p5.7.m7.1.1.2.3.cmml" xref="S3.SS4.p5.7.m7.1.1.2.3">ℎ</ci></apply><cn type="integer" id="S3.SS4.p5.7.m7.1.1.3.cmml" xref="S3.SS4.p5.7.m7.1.1.3">64</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p5.7.m7.1c">d_{h}\leq 64</annotation></semantics></math>, so we set <math id="S3.SS4.p5.8.m8.1" class="ltx_Math" alttext="d_{h}=80" display="inline"><semantics id="S3.SS4.p5.8.m8.1a"><mrow id="S3.SS4.p5.8.m8.1.1" xref="S3.SS4.p5.8.m8.1.1.cmml"><msub id="S3.SS4.p5.8.m8.1.1.2" xref="S3.SS4.p5.8.m8.1.1.2.cmml"><mi id="S3.SS4.p5.8.m8.1.1.2.2" xref="S3.SS4.p5.8.m8.1.1.2.2.cmml">d</mi><mi id="S3.SS4.p5.8.m8.1.1.2.3" xref="S3.SS4.p5.8.m8.1.1.2.3.cmml">h</mi></msub><mo id="S3.SS4.p5.8.m8.1.1.1" xref="S3.SS4.p5.8.m8.1.1.1.cmml">=</mo><mn id="S3.SS4.p5.8.m8.1.1.3" xref="S3.SS4.p5.8.m8.1.1.3.cmml">80</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS4.p5.8.m8.1b"><apply id="S3.SS4.p5.8.m8.1.1.cmml" xref="S3.SS4.p5.8.m8.1.1"><eq id="S3.SS4.p5.8.m8.1.1.1.cmml" xref="S3.SS4.p5.8.m8.1.1.1"></eq><apply id="S3.SS4.p5.8.m8.1.1.2.cmml" xref="S3.SS4.p5.8.m8.1.1.2"><csymbol cd="ambiguous" id="S3.SS4.p5.8.m8.1.1.2.1.cmml" xref="S3.SS4.p5.8.m8.1.1.2">subscript</csymbol><ci id="S3.SS4.p5.8.m8.1.1.2.2.cmml" xref="S3.SS4.p5.8.m8.1.1.2.2">𝑑</ci><ci id="S3.SS4.p5.8.m8.1.1.2.3.cmml" xref="S3.SS4.p5.8.m8.1.1.2.3">ℎ</ci></apply><cn type="integer" id="S3.SS4.p5.8.m8.1.1.3.cmml" xref="S3.SS4.p5.8.m8.1.1.3">80</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p5.8.m8.1c">d_{h}=80</annotation></semantics></math> (though it needs to be pointed out that the loss spikes may not be solely attributed to the choice of <math id="S3.SS4.p5.9.m9.1" class="ltx_Math" alttext="d_{h}" display="inline"><semantics id="S3.SS4.p5.9.m9.1a"><msub id="S3.SS4.p5.9.m9.1.1" xref="S3.SS4.p5.9.m9.1.1.cmml"><mi id="S3.SS4.p5.9.m9.1.1.2" xref="S3.SS4.p5.9.m9.1.1.2.cmml">d</mi><mi id="S3.SS4.p5.9.m9.1.1.3" xref="S3.SS4.p5.9.m9.1.1.3.cmml">h</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS4.p5.9.m9.1b"><apply id="S3.SS4.p5.9.m9.1.1.cmml" xref="S3.SS4.p5.9.m9.1.1"><csymbol cd="ambiguous" id="S3.SS4.p5.9.m9.1.1.1.cmml" xref="S3.SS4.p5.9.m9.1.1">subscript</csymbol><ci id="S3.SS4.p5.9.m9.1.1.2.cmml" xref="S3.SS4.p5.9.m9.1.1.2">𝑑</ci><ci id="S3.SS4.p5.9.m9.1.1.3.cmml" xref="S3.SS4.p5.9.m9.1.1.3">ℎ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p5.9.m9.1c">d_{h}</annotation></semantics></math>, and high-quality data for instance may stabilize training and allow us to choose a smaller <math id="S3.SS4.p5.10.m10.1" class="ltx_Math" alttext="d_{h}" display="inline"><semantics id="S3.SS4.p5.10.m10.1a"><msub id="S3.SS4.p5.10.m10.1.1" xref="S3.SS4.p5.10.m10.1.1.cmml"><mi id="S3.SS4.p5.10.m10.1.1.2" xref="S3.SS4.p5.10.m10.1.1.2.cmml">d</mi><mi id="S3.SS4.p5.10.m10.1.1.3" xref="S3.SS4.p5.10.m10.1.1.3.cmml">h</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS4.p5.10.m10.1b"><apply id="S3.SS4.p5.10.m10.1.1.cmml" xref="S3.SS4.p5.10.m10.1.1"><csymbol cd="ambiguous" id="S3.SS4.p5.10.m10.1.1.1.cmml" xref="S3.SS4.p5.10.m10.1.1">subscript</csymbol><ci id="S3.SS4.p5.10.m10.1.1.2.cmml" xref="S3.SS4.p5.10.m10.1.1.2">𝑑</ci><ci id="S3.SS4.p5.10.m10.1.1.3.cmml" xref="S3.SS4.p5.10.m10.1.1.3">ℎ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p5.10.m10.1c">d_{h}</annotation></semantics></math>).
Also, the MLP width <math id="S3.SS4.p5.11.m11.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S3.SS4.p5.11.m11.1a"><mi id="S3.SS4.p5.11.m11.1.1" xref="S3.SS4.p5.11.m11.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p5.11.m11.1b"><ci id="S3.SS4.p5.11.m11.1.1.cmml" xref="S3.SS4.p5.11.m11.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p5.11.m11.1c">W</annotation></semantics></math> is set to be equal to the hidden dimension <math id="S3.SS4.p5.12.m12.1" class="ltx_Math" alttext="d=Hd_{h}" display="inline"><semantics id="S3.SS4.p5.12.m12.1a"><mrow id="S3.SS4.p5.12.m12.1.1" xref="S3.SS4.p5.12.m12.1.1.cmml"><mi id="S3.SS4.p5.12.m12.1.1.2" xref="S3.SS4.p5.12.m12.1.1.2.cmml">d</mi><mo id="S3.SS4.p5.12.m12.1.1.1" xref="S3.SS4.p5.12.m12.1.1.1.cmml">=</mo><mrow id="S3.SS4.p5.12.m12.1.1.3" xref="S3.SS4.p5.12.m12.1.1.3.cmml"><mi id="S3.SS4.p5.12.m12.1.1.3.2" xref="S3.SS4.p5.12.m12.1.1.3.2.cmml">H</mi><mo lspace="0em" rspace="0em" id="S3.SS4.p5.12.m12.1.1.3.1" xref="S3.SS4.p5.12.m12.1.1.3.1.cmml">​</mo><msub id="S3.SS4.p5.12.m12.1.1.3.3" xref="S3.SS4.p5.12.m12.1.1.3.3.cmml"><mi id="S3.SS4.p5.12.m12.1.1.3.3.2" xref="S3.SS4.p5.12.m12.1.1.3.3.2.cmml">d</mi><mi id="S3.SS4.p5.12.m12.1.1.3.3.3" xref="S3.SS4.p5.12.m12.1.1.3.3.3.cmml">h</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS4.p5.12.m12.1b"><apply id="S3.SS4.p5.12.m12.1.1.cmml" xref="S3.SS4.p5.12.m12.1.1"><eq id="S3.SS4.p5.12.m12.1.1.1.cmml" xref="S3.SS4.p5.12.m12.1.1.1"></eq><ci id="S3.SS4.p5.12.m12.1.1.2.cmml" xref="S3.SS4.p5.12.m12.1.1.2">𝑑</ci><apply id="S3.SS4.p5.12.m12.1.1.3.cmml" xref="S3.SS4.p5.12.m12.1.1.3"><times id="S3.SS4.p5.12.m12.1.1.3.1.cmml" xref="S3.SS4.p5.12.m12.1.1.3.1"></times><ci id="S3.SS4.p5.12.m12.1.1.3.2.cmml" xref="S3.SS4.p5.12.m12.1.1.3.2">𝐻</ci><apply id="S3.SS4.p5.12.m12.1.1.3.3.cmml" xref="S3.SS4.p5.12.m12.1.1.3.3"><csymbol cd="ambiguous" id="S3.SS4.p5.12.m12.1.1.3.3.1.cmml" xref="S3.SS4.p5.12.m12.1.1.3.3">subscript</csymbol><ci id="S3.SS4.p5.12.m12.1.1.3.3.2.cmml" xref="S3.SS4.p5.12.m12.1.1.3.3.2">𝑑</ci><ci id="S3.SS4.p5.12.m12.1.1.3.3.3.cmml" xref="S3.SS4.p5.12.m12.1.1.3.3.3">ℎ</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p5.12.m12.1c">d=Hd_{h}</annotation></semantics></math>.
Regarding the first point, controlled experiments (Figure <a href="#F10" title="Figure 10 ‣ 3.4 | Model Shape ‣ 3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">10</span></a>) indicate that the loss decreases slightly more rapidly with <math id="S3.SS4.p5.13.m13.1" class="ltx_Math" alttext="L:H\approx 1" display="inline"><semantics id="S3.SS4.p5.13.m13.1a"><mrow id="S3.SS4.p5.13.m13.1.1" xref="S3.SS4.p5.13.m13.1.1.cmml"><mi id="S3.SS4.p5.13.m13.1.1.2" xref="S3.SS4.p5.13.m13.1.1.2.cmml">L</mi><mo lspace="0.278em" rspace="0.278em" id="S3.SS4.p5.13.m13.1.1.1" xref="S3.SS4.p5.13.m13.1.1.1.cmml">:</mo><mrow id="S3.SS4.p5.13.m13.1.1.3" xref="S3.SS4.p5.13.m13.1.1.3.cmml"><mi id="S3.SS4.p5.13.m13.1.1.3.2" xref="S3.SS4.p5.13.m13.1.1.3.2.cmml">H</mi><mo id="S3.SS4.p5.13.m13.1.1.3.1" xref="S3.SS4.p5.13.m13.1.1.3.1.cmml">≈</mo><mn id="S3.SS4.p5.13.m13.1.1.3.3" xref="S3.SS4.p5.13.m13.1.1.3.3.cmml">1</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS4.p5.13.m13.1b"><apply id="S3.SS4.p5.13.m13.1.1.cmml" xref="S3.SS4.p5.13.m13.1.1"><ci id="S3.SS4.p5.13.m13.1.1.1.cmml" xref="S3.SS4.p5.13.m13.1.1.1">:</ci><ci id="S3.SS4.p5.13.m13.1.1.2.cmml" xref="S3.SS4.p5.13.m13.1.1.2">𝐿</ci><apply id="S3.SS4.p5.13.m13.1.1.3.cmml" xref="S3.SS4.p5.13.m13.1.1.3"><approx id="S3.SS4.p5.13.m13.1.1.3.1.cmml" xref="S3.SS4.p5.13.m13.1.1.3.1"></approx><ci id="S3.SS4.p5.13.m13.1.1.3.2.cmml" xref="S3.SS4.p5.13.m13.1.1.3.2">𝐻</ci><cn type="integer" id="S3.SS4.p5.13.m13.1.1.3.3.cmml" xref="S3.SS4.p5.13.m13.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p5.13.m13.1c">L:H\approx 1</annotation></semantics></math> than with other ratios, so we adopt this ratio.</p>
</div>
<figure id="F10" class="ltx_figure"><img src="/html/2407.01178/assets/Figures/key_figure/optimize_2B_shape_warmup.png" id="F10.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="644" height="358" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="F10.8.4.1" class="ltx_text" style="font-size:90%;">Figure 10</span>: </span><span id="F10.6.3" class="ltx_text" style="font-size:90%;">Comparison of the training losses of models with different shapes, whose parameter sizes range in <math id="F10.4.1.m1.1" class="ltx_Math" alttext="2.1\sim 2.4" display="inline"><semantics id="F10.4.1.m1.1b"><mrow id="F10.4.1.m1.1.1" xref="F10.4.1.m1.1.1.cmml"><mn id="F10.4.1.m1.1.1.2" xref="F10.4.1.m1.1.1.2.cmml">2.1</mn><mo id="F10.4.1.m1.1.1.1" xref="F10.4.1.m1.1.1.1.cmml">∼</mo><mn id="F10.4.1.m1.1.1.3" xref="F10.4.1.m1.1.1.3.cmml">2.4</mn></mrow><annotation-xml encoding="MathML-Content" id="F10.4.1.m1.1c"><apply id="F10.4.1.m1.1.1.cmml" xref="F10.4.1.m1.1.1"><csymbol cd="latexml" id="F10.4.1.m1.1.1.1.cmml" xref="F10.4.1.m1.1.1.1">similar-to</csymbol><cn type="float" id="F10.4.1.m1.1.1.2.cmml" xref="F10.4.1.m1.1.1.2">2.1</cn><cn type="float" id="F10.4.1.m1.1.1.3.cmml" xref="F10.4.1.m1.1.1.3">2.4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="F10.4.1.m1.1d">2.1\sim 2.4</annotation></semantics></math>B.
The legend l44h40d80 denotes <math id="F10.5.2.m2.2" class="ltx_Math" alttext="L=44,H=40,d_{h}=80" display="inline"><semantics id="F10.5.2.m2.2b"><mrow id="F10.5.2.m2.2.2.2" xref="F10.5.2.m2.2.2.3.cmml"><mrow id="F10.5.2.m2.1.1.1.1" xref="F10.5.2.m2.1.1.1.1.cmml"><mi id="F10.5.2.m2.1.1.1.1.2" xref="F10.5.2.m2.1.1.1.1.2.cmml">L</mi><mo id="F10.5.2.m2.1.1.1.1.1" xref="F10.5.2.m2.1.1.1.1.1.cmml">=</mo><mn id="F10.5.2.m2.1.1.1.1.3" xref="F10.5.2.m2.1.1.1.1.3.cmml">44</mn></mrow><mo id="F10.5.2.m2.2.2.2.3" xref="F10.5.2.m2.2.2.3a.cmml">,</mo><mrow id="F10.5.2.m2.2.2.2.2.2" xref="F10.5.2.m2.2.2.2.2.3.cmml"><mrow id="F10.5.2.m2.2.2.2.2.1.1" xref="F10.5.2.m2.2.2.2.2.1.1.cmml"><mi id="F10.5.2.m2.2.2.2.2.1.1.2" xref="F10.5.2.m2.2.2.2.2.1.1.2.cmml">H</mi><mo id="F10.5.2.m2.2.2.2.2.1.1.1" xref="F10.5.2.m2.2.2.2.2.1.1.1.cmml">=</mo><mn id="F10.5.2.m2.2.2.2.2.1.1.3" xref="F10.5.2.m2.2.2.2.2.1.1.3.cmml">40</mn></mrow><mo id="F10.5.2.m2.2.2.2.2.2.3" xref="F10.5.2.m2.2.2.2.2.3a.cmml">,</mo><mrow id="F10.5.2.m2.2.2.2.2.2.2" xref="F10.5.2.m2.2.2.2.2.2.2.cmml"><msub id="F10.5.2.m2.2.2.2.2.2.2.2" xref="F10.5.2.m2.2.2.2.2.2.2.2.cmml"><mi id="F10.5.2.m2.2.2.2.2.2.2.2.2" xref="F10.5.2.m2.2.2.2.2.2.2.2.2.cmml">d</mi><mi id="F10.5.2.m2.2.2.2.2.2.2.2.3" xref="F10.5.2.m2.2.2.2.2.2.2.2.3.cmml">h</mi></msub><mo id="F10.5.2.m2.2.2.2.2.2.2.1" xref="F10.5.2.m2.2.2.2.2.2.2.1.cmml">=</mo><mn id="F10.5.2.m2.2.2.2.2.2.2.3" xref="F10.5.2.m2.2.2.2.2.2.2.3.cmml">80</mn></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="F10.5.2.m2.2c"><apply id="F10.5.2.m2.2.2.3.cmml" xref="F10.5.2.m2.2.2.2"><csymbol cd="ambiguous" id="F10.5.2.m2.2.2.3a.cmml" xref="F10.5.2.m2.2.2.2.3">formulae-sequence</csymbol><apply id="F10.5.2.m2.1.1.1.1.cmml" xref="F10.5.2.m2.1.1.1.1"><eq id="F10.5.2.m2.1.1.1.1.1.cmml" xref="F10.5.2.m2.1.1.1.1.1"></eq><ci id="F10.5.2.m2.1.1.1.1.2.cmml" xref="F10.5.2.m2.1.1.1.1.2">𝐿</ci><cn type="integer" id="F10.5.2.m2.1.1.1.1.3.cmml" xref="F10.5.2.m2.1.1.1.1.3">44</cn></apply><apply id="F10.5.2.m2.2.2.2.2.3.cmml" xref="F10.5.2.m2.2.2.2.2.2"><csymbol cd="ambiguous" id="F10.5.2.m2.2.2.2.2.3a.cmml" xref="F10.5.2.m2.2.2.2.2.2.3">formulae-sequence</csymbol><apply id="F10.5.2.m2.2.2.2.2.1.1.cmml" xref="F10.5.2.m2.2.2.2.2.1.1"><eq id="F10.5.2.m2.2.2.2.2.1.1.1.cmml" xref="F10.5.2.m2.2.2.2.2.1.1.1"></eq><ci id="F10.5.2.m2.2.2.2.2.1.1.2.cmml" xref="F10.5.2.m2.2.2.2.2.1.1.2">𝐻</ci><cn type="integer" id="F10.5.2.m2.2.2.2.2.1.1.3.cmml" xref="F10.5.2.m2.2.2.2.2.1.1.3">40</cn></apply><apply id="F10.5.2.m2.2.2.2.2.2.2.cmml" xref="F10.5.2.m2.2.2.2.2.2.2"><eq id="F10.5.2.m2.2.2.2.2.2.2.1.cmml" xref="F10.5.2.m2.2.2.2.2.2.2.1"></eq><apply id="F10.5.2.m2.2.2.2.2.2.2.2.cmml" xref="F10.5.2.m2.2.2.2.2.2.2.2"><csymbol cd="ambiguous" id="F10.5.2.m2.2.2.2.2.2.2.2.1.cmml" xref="F10.5.2.m2.2.2.2.2.2.2.2">subscript</csymbol><ci id="F10.5.2.m2.2.2.2.2.2.2.2.2.cmml" xref="F10.5.2.m2.2.2.2.2.2.2.2.2">𝑑</ci><ci id="F10.5.2.m2.2.2.2.2.2.2.2.3.cmml" xref="F10.5.2.m2.2.2.2.2.2.2.2.3">ℎ</ci></apply><cn type="integer" id="F10.5.2.m2.2.2.2.2.2.2.3.cmml" xref="F10.5.2.m2.2.2.2.2.2.2.3">80</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="F10.5.2.m2.2d">L=44,H=40,d_{h}=80</annotation></semantics></math>, and the <math id="F10.6.3.m3.1" class="ltx_Math" alttext="x" display="inline"><semantics id="F10.6.3.m3.1b"><mi id="F10.6.3.m3.1.1" xref="F10.6.3.m3.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="F10.6.3.m3.1c"><ci id="F10.6.3.m3.1.1.cmml" xref="F10.6.3.m3.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="F10.6.3.m3.1d">x</annotation></semantics></math>-axis denotes the amount of training samples.
Nevertheless, this comparison is not definite, since this is only the warmup stage of our training scheme (Section <a href="#S3.SS6" title="3.6 | Two-stage Pretrain ‣ 3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.6</span></a>) and the ranking may change in the continual train stage when explicit memory is introduced.</span></figcaption>
</figure>
<div id="S3.SS4.p6" class="ltx_para">
<p id="S3.SS4.p6.2" class="ltx_p">In addition, as discussed in Section <a href="#S3.SS3" title="3.3 | Memory Sparsification and Storage ‣ 3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3</span></a>, our model uses grouped query attention (GQA), so the number of key-value heads <math id="S3.SS4.p6.1.m1.1" class="ltx_Math" alttext="H_{kv}" display="inline"><semantics id="S3.SS4.p6.1.m1.1a"><msub id="S3.SS4.p6.1.m1.1.1" xref="S3.SS4.p6.1.m1.1.1.cmml"><mi id="S3.SS4.p6.1.m1.1.1.2" xref="S3.SS4.p6.1.m1.1.1.2.cmml">H</mi><mrow id="S3.SS4.p6.1.m1.1.1.3" xref="S3.SS4.p6.1.m1.1.1.3.cmml"><mi id="S3.SS4.p6.1.m1.1.1.3.2" xref="S3.SS4.p6.1.m1.1.1.3.2.cmml">k</mi><mo lspace="0em" rspace="0em" id="S3.SS4.p6.1.m1.1.1.3.1" xref="S3.SS4.p6.1.m1.1.1.3.1.cmml">​</mo><mi id="S3.SS4.p6.1.m1.1.1.3.3" xref="S3.SS4.p6.1.m1.1.1.3.3.cmml">v</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS4.p6.1.m1.1b"><apply id="S3.SS4.p6.1.m1.1.1.cmml" xref="S3.SS4.p6.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS4.p6.1.m1.1.1.1.cmml" xref="S3.SS4.p6.1.m1.1.1">subscript</csymbol><ci id="S3.SS4.p6.1.m1.1.1.2.cmml" xref="S3.SS4.p6.1.m1.1.1.2">𝐻</ci><apply id="S3.SS4.p6.1.m1.1.1.3.cmml" xref="S3.SS4.p6.1.m1.1.1.3"><times id="S3.SS4.p6.1.m1.1.1.3.1.cmml" xref="S3.SS4.p6.1.m1.1.1.3.1"></times><ci id="S3.SS4.p6.1.m1.1.1.3.2.cmml" xref="S3.SS4.p6.1.m1.1.1.3.2">𝑘</ci><ci id="S3.SS4.p6.1.m1.1.1.3.3.cmml" xref="S3.SS4.p6.1.m1.1.1.3.3">𝑣</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p6.1.m1.1c">H_{kv}</annotation></semantics></math> is set to be <math id="S3.SS4.p6.2.m2.1" class="ltx_Math" alttext="8" display="inline"><semantics id="S3.SS4.p6.2.m2.1a"><mn id="S3.SS4.p6.2.m2.1.1" xref="S3.SS4.p6.2.m2.1.1.cmml">8</mn><annotation-xml encoding="MathML-Content" id="S3.SS4.p6.2.m2.1b"><cn type="integer" id="S3.SS4.p6.2.m2.1.1.cmml" xref="S3.SS4.p6.2.m2.1.1">8</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p6.2.m2.1c">8</annotation></semantics></math>, which is the usual choice for GQA.
The MLP layers are gated two-layer networks without bias, which are the default choice in recent years <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib110" title="" class="ltx_ref">110</a>, <a href="#bib.bib11" title="" class="ltx_ref">11</a>, <a href="#bib.bib21" title="" class="ltx_ref">21</a>, <a href="#bib.bib8" title="" class="ltx_ref">8</a>]</cite>.</p>
</div>
<div id="S3.SS4.p7" class="ltx_para">
<p id="S3.SS4.p7.2" class="ltx_p">Finally, the model shape is set to be <math id="S3.SS4.p7.1.m1.2" class="ltx_Math" alttext="L=44,H=40,H_{kv}=8,d_{h}=80,W=3200" display="inline"><semantics id="S3.SS4.p7.1.m1.2a"><mrow id="S3.SS4.p7.1.m1.2.2.2" xref="S3.SS4.p7.1.m1.2.2.3.cmml"><mrow id="S3.SS4.p7.1.m1.1.1.1.1" xref="S3.SS4.p7.1.m1.1.1.1.1.cmml"><mi id="S3.SS4.p7.1.m1.1.1.1.1.2" xref="S3.SS4.p7.1.m1.1.1.1.1.2.cmml">L</mi><mo id="S3.SS4.p7.1.m1.1.1.1.1.1" xref="S3.SS4.p7.1.m1.1.1.1.1.1.cmml">=</mo><mn id="S3.SS4.p7.1.m1.1.1.1.1.3" xref="S3.SS4.p7.1.m1.1.1.1.1.3.cmml">44</mn></mrow><mo id="S3.SS4.p7.1.m1.2.2.2.3" xref="S3.SS4.p7.1.m1.2.2.3a.cmml">,</mo><mrow id="S3.SS4.p7.1.m1.2.2.2.2.2" xref="S3.SS4.p7.1.m1.2.2.2.2.3.cmml"><mrow id="S3.SS4.p7.1.m1.2.2.2.2.1.1" xref="S3.SS4.p7.1.m1.2.2.2.2.1.1.cmml"><mi id="S3.SS4.p7.1.m1.2.2.2.2.1.1.2" xref="S3.SS4.p7.1.m1.2.2.2.2.1.1.2.cmml">H</mi><mo id="S3.SS4.p7.1.m1.2.2.2.2.1.1.1" xref="S3.SS4.p7.1.m1.2.2.2.2.1.1.1.cmml">=</mo><mn id="S3.SS4.p7.1.m1.2.2.2.2.1.1.3" xref="S3.SS4.p7.1.m1.2.2.2.2.1.1.3.cmml">40</mn></mrow><mo id="S3.SS4.p7.1.m1.2.2.2.2.2.3" xref="S3.SS4.p7.1.m1.2.2.2.2.3a.cmml">,</mo><mrow id="S3.SS4.p7.1.m1.2.2.2.2.2.2.2" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.3.cmml"><mrow id="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.cmml"><msub id="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.2" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.2.cmml"><mi id="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.2.2" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.2.2.cmml">H</mi><mrow id="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.2.3" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.2.3.cmml"><mi id="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.2.3.2" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.2.3.2.cmml">k</mi><mo lspace="0em" rspace="0em" id="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.2.3.1" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.2.3.1.cmml">​</mo><mi id="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.2.3.3" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.2.3.3.cmml">v</mi></mrow></msub><mo id="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.1" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.1.cmml">=</mo><mn id="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.3" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.3.cmml">8</mn></mrow><mo id="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.3" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.3a.cmml">,</mo><mrow id="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.2" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.3.cmml"><mrow id="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.1.1" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.1.1.cmml"><msub id="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.1.1.2" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.1.1.2.cmml"><mi id="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.1.1.2.2" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.1.1.2.2.cmml">d</mi><mi id="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.1.1.2.3" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.1.1.2.3.cmml">h</mi></msub><mo id="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.1.1.1" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.1.1.1.cmml">=</mo><mn id="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.1.1.3" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.1.1.3.cmml">80</mn></mrow><mo id="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.2.3" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.3a.cmml">,</mo><mrow id="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.2.2" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.2.2.cmml"><mi id="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.2.2.2" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.2.2.2.cmml">W</mi><mo id="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.2.2.1" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.2.2.1.cmml">=</mo><mn id="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.2.2.3" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.2.2.3.cmml">3200</mn></mrow></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS4.p7.1.m1.2b"><apply id="S3.SS4.p7.1.m1.2.2.3.cmml" xref="S3.SS4.p7.1.m1.2.2.2"><csymbol cd="ambiguous" id="S3.SS4.p7.1.m1.2.2.3a.cmml" xref="S3.SS4.p7.1.m1.2.2.2.3">formulae-sequence</csymbol><apply id="S3.SS4.p7.1.m1.1.1.1.1.cmml" xref="S3.SS4.p7.1.m1.1.1.1.1"><eq id="S3.SS4.p7.1.m1.1.1.1.1.1.cmml" xref="S3.SS4.p7.1.m1.1.1.1.1.1"></eq><ci id="S3.SS4.p7.1.m1.1.1.1.1.2.cmml" xref="S3.SS4.p7.1.m1.1.1.1.1.2">𝐿</ci><cn type="integer" id="S3.SS4.p7.1.m1.1.1.1.1.3.cmml" xref="S3.SS4.p7.1.m1.1.1.1.1.3">44</cn></apply><apply id="S3.SS4.p7.1.m1.2.2.2.2.3.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS4.p7.1.m1.2.2.2.2.3a.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.2.3">formulae-sequence</csymbol><apply id="S3.SS4.p7.1.m1.2.2.2.2.1.1.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.1.1"><eq id="S3.SS4.p7.1.m1.2.2.2.2.1.1.1.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.1.1.1"></eq><ci id="S3.SS4.p7.1.m1.2.2.2.2.1.1.2.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.1.1.2">𝐻</ci><cn type="integer" id="S3.SS4.p7.1.m1.2.2.2.2.1.1.3.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.1.1.3">40</cn></apply><apply id="S3.SS4.p7.1.m1.2.2.2.2.2.2.3.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS4.p7.1.m1.2.2.2.2.2.2.3a.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.3">formulae-sequence</csymbol><apply id="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1"><eq id="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.1.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.1"></eq><apply id="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.2.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.2"><csymbol cd="ambiguous" id="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.2.1.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.2">subscript</csymbol><ci id="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.2.2.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.2.2">𝐻</ci><apply id="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.2.3.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.2.3"><times id="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.2.3.1.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.2.3.1"></times><ci id="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.2.3.2.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.2.3.2">𝑘</ci><ci id="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.2.3.3.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.2.3.3">𝑣</ci></apply></apply><cn type="integer" id="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.3.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.1.1.3">8</cn></apply><apply id="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.3.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.3a.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.2.3">formulae-sequence</csymbol><apply id="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.1.1.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.1.1"><eq id="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.1.1.1.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.1.1.1"></eq><apply id="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.1.1.2.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.1.1.2"><csymbol cd="ambiguous" id="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.1.1.2.1.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.1.1.2">subscript</csymbol><ci id="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.1.1.2.2.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.1.1.2.2">𝑑</ci><ci id="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.1.1.2.3.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.1.1.2.3">ℎ</ci></apply><cn type="integer" id="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.1.1.3.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.1.1.3">80</cn></apply><apply id="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.2.2.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.2.2"><eq id="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.2.2.1.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.2.2.1"></eq><ci id="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.2.2.2.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.2.2.2">𝑊</ci><cn type="integer" id="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.2.2.3.cmml" xref="S3.SS4.p7.1.m1.2.2.2.2.2.2.2.2.2.2.3">3200</cn></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p7.1.m1.2c">L=44,H=40,H_{kv}=8,d_{h}=80,W=3200</annotation></semantics></math>, with the total non-embedding parameter size being <math id="S3.SS4.p7.2.m2.1" class="ltx_Math" alttext="2.4" display="inline"><semantics id="S3.SS4.p7.2.m2.1a"><mn id="S3.SS4.p7.2.m2.1.1" xref="S3.SS4.p7.2.m2.1.1.cmml">2.4</mn><annotation-xml encoding="MathML-Content" id="S3.SS4.p7.2.m2.1b"><cn type="float" id="S3.SS4.p7.2.m2.1.1.cmml" xref="S3.SS4.p7.2.m2.1.1">2.4</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p7.2.m2.1c">2.4</annotation></semantics></math>B.</p>
</div>
</section>
<section id="S3.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsection" style="font-size:120%;color:#00868B;">3.5 <span id="S3.SS5.1.1" class="ltx_text" style="color:#808080;">| </span>Training Designs</h3>

<div id="S3.SS5.p1" class="ltx_para">
<p id="S3.SS5.p1.1" class="ltx_p">Similar to our architecture design, the design of our training scheme focuses on learning abstract knowledges.
The goal is to reduce the training compute, as the LLM no longer needs to memorize many of the specific knowledges.
This shift in learning objective implies that all the default settings for pretraining LLMs may need to be redesigned, as they were optimized for the classical scenario when the LLMs learn both abstract and specific knowledges.</p>
</div>
<div id="S3.SS5.p2" class="ltx_para">
<ol id="S3.I2" class="ltx_enumerate">
<li id="S3.I2.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><span id="S3.I2.i1.1.1.1" class="ltx_text ltx_font_bold" style="color:#00868B;">0.</span></span> 
<div id="S3.I2.i1.p1" class="ltx_para">
<p id="S3.I2.i1.p1.1" class="ltx_p">Data:
Ideally, the pretraining data should have a high concentration of abstract knowledges and minimum amount of specific knowledges.
It is known that LLM pretraining is very sensitive to the presence of specific knowledges.
For instance, <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib55" title="" class="ltx_ref">55</a>]</cite> observes that a small model can master arithmetic (e.g. addition of large numbers) if trained on clean data.
However, if the training data is mixed with trivial information (e.g. random numbers), then the test accuracy stays at zero unless the model size is increased by a factor of 1500.
It suggests that training on specific knowledges significantly inhibits the learning of abstract knowledges, and may explain why emergent abilities <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib119" title="" class="ltx_ref">119</a>]</cite> are absent from small models.
Notably, the Phi-3 model <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib4" title="" class="ltx_ref">4</a>]</cite> is pretrained with a data composition that closely matches our desired composition.
Although the technical details are not revealed, it is stated that they filter data based on two criteria:
the data should encourage reasoning, and should not contain information that is too specific.</p>
</div>
</li>
<li id="S3.I2.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><span id="S3.I2.i2.1.1.1" class="ltx_text ltx_font_bold" style="color:#00868B;">0.</span></span> 
<div id="S3.I2.i2.p1" class="ltx_para">
<p id="S3.I2.i2.p1.3" class="ltx_p">Initialization: <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib132" title="" class="ltx_ref">132</a>]</cite> observes that initializing Transformer parameters with a smaller standard deviation (<math id="S3.I2.i2.p1.1.m1.1" class="ltx_Math" alttext="d^{c}" display="inline"><semantics id="S3.I2.i2.p1.1.m1.1a"><msup id="S3.I2.i2.p1.1.m1.1.1" xref="S3.I2.i2.p1.1.m1.1.1.cmml"><mi id="S3.I2.i2.p1.1.m1.1.1.2" xref="S3.I2.i2.p1.1.m1.1.1.2.cmml">d</mi><mi id="S3.I2.i2.p1.1.m1.1.1.3" xref="S3.I2.i2.p1.1.m1.1.1.3.cmml">c</mi></msup><annotation-xml encoding="MathML-Content" id="S3.I2.i2.p1.1.m1.1b"><apply id="S3.I2.i2.p1.1.m1.1.1.cmml" xref="S3.I2.i2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.I2.i2.p1.1.m1.1.1.1.cmml" xref="S3.I2.i2.p1.1.m1.1.1">superscript</csymbol><ci id="S3.I2.i2.p1.1.m1.1.1.2.cmml" xref="S3.I2.i2.p1.1.m1.1.1.2">𝑑</ci><ci id="S3.I2.i2.p1.1.m1.1.1.3.cmml" xref="S3.I2.i2.p1.1.m1.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I2.i2.p1.1.m1.1c">d^{c}</annotation></semantics></math> with <math id="S3.I2.i2.p1.2.m2.1" class="ltx_Math" alttext="c&lt;-1/2" display="inline"><semantics id="S3.I2.i2.p1.2.m2.1a"><mrow id="S3.I2.i2.p1.2.m2.1.1" xref="S3.I2.i2.p1.2.m2.1.1.cmml"><mi id="S3.I2.i2.p1.2.m2.1.1.2" xref="S3.I2.i2.p1.2.m2.1.1.2.cmml">c</mi><mo id="S3.I2.i2.p1.2.m2.1.1.1" xref="S3.I2.i2.p1.2.m2.1.1.1.cmml">&lt;</mo><mrow id="S3.I2.i2.p1.2.m2.1.1.3" xref="S3.I2.i2.p1.2.m2.1.1.3.cmml"><mo id="S3.I2.i2.p1.2.m2.1.1.3a" xref="S3.I2.i2.p1.2.m2.1.1.3.cmml">−</mo><mrow id="S3.I2.i2.p1.2.m2.1.1.3.2" xref="S3.I2.i2.p1.2.m2.1.1.3.2.cmml"><mn id="S3.I2.i2.p1.2.m2.1.1.3.2.2" xref="S3.I2.i2.p1.2.m2.1.1.3.2.2.cmml">1</mn><mo id="S3.I2.i2.p1.2.m2.1.1.3.2.1" xref="S3.I2.i2.p1.2.m2.1.1.3.2.1.cmml">/</mo><mn id="S3.I2.i2.p1.2.m2.1.1.3.2.3" xref="S3.I2.i2.p1.2.m2.1.1.3.2.3.cmml">2</mn></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.I2.i2.p1.2.m2.1b"><apply id="S3.I2.i2.p1.2.m2.1.1.cmml" xref="S3.I2.i2.p1.2.m2.1.1"><lt id="S3.I2.i2.p1.2.m2.1.1.1.cmml" xref="S3.I2.i2.p1.2.m2.1.1.1"></lt><ci id="S3.I2.i2.p1.2.m2.1.1.2.cmml" xref="S3.I2.i2.p1.2.m2.1.1.2">𝑐</ci><apply id="S3.I2.i2.p1.2.m2.1.1.3.cmml" xref="S3.I2.i2.p1.2.m2.1.1.3"><minus id="S3.I2.i2.p1.2.m2.1.1.3.1.cmml" xref="S3.I2.i2.p1.2.m2.1.1.3"></minus><apply id="S3.I2.i2.p1.2.m2.1.1.3.2.cmml" xref="S3.I2.i2.p1.2.m2.1.1.3.2"><divide id="S3.I2.i2.p1.2.m2.1.1.3.2.1.cmml" xref="S3.I2.i2.p1.2.m2.1.1.3.2.1"></divide><cn type="integer" id="S3.I2.i2.p1.2.m2.1.1.3.2.2.cmml" xref="S3.I2.i2.p1.2.m2.1.1.3.2.2">1</cn><cn type="integer" id="S3.I2.i2.p1.2.m2.1.1.3.2.3.cmml" xref="S3.I2.i2.p1.2.m2.1.1.3.2.3">2</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I2.i2.p1.2.m2.1c">c&lt;-1/2</annotation></semantics></math> instead of the usual <math id="S3.I2.i2.p1.3.m3.1" class="ltx_Math" alttext="\Theta(d^{-1/2})" display="inline"><semantics id="S3.I2.i2.p1.3.m3.1a"><mrow id="S3.I2.i2.p1.3.m3.1.1" xref="S3.I2.i2.p1.3.m3.1.1.cmml"><mi mathvariant="normal" id="S3.I2.i2.p1.3.m3.1.1.3" xref="S3.I2.i2.p1.3.m3.1.1.3.cmml">Θ</mi><mo lspace="0em" rspace="0em" id="S3.I2.i2.p1.3.m3.1.1.2" xref="S3.I2.i2.p1.3.m3.1.1.2.cmml">​</mo><mrow id="S3.I2.i2.p1.3.m3.1.1.1.1" xref="S3.I2.i2.p1.3.m3.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.I2.i2.p1.3.m3.1.1.1.1.2" xref="S3.I2.i2.p1.3.m3.1.1.1.1.1.cmml">(</mo><msup id="S3.I2.i2.p1.3.m3.1.1.1.1.1" xref="S3.I2.i2.p1.3.m3.1.1.1.1.1.cmml"><mi id="S3.I2.i2.p1.3.m3.1.1.1.1.1.2" xref="S3.I2.i2.p1.3.m3.1.1.1.1.1.2.cmml">d</mi><mrow id="S3.I2.i2.p1.3.m3.1.1.1.1.1.3" xref="S3.I2.i2.p1.3.m3.1.1.1.1.1.3.cmml"><mo id="S3.I2.i2.p1.3.m3.1.1.1.1.1.3a" xref="S3.I2.i2.p1.3.m3.1.1.1.1.1.3.cmml">−</mo><mrow id="S3.I2.i2.p1.3.m3.1.1.1.1.1.3.2" xref="S3.I2.i2.p1.3.m3.1.1.1.1.1.3.2.cmml"><mn id="S3.I2.i2.p1.3.m3.1.1.1.1.1.3.2.2" xref="S3.I2.i2.p1.3.m3.1.1.1.1.1.3.2.2.cmml">1</mn><mo id="S3.I2.i2.p1.3.m3.1.1.1.1.1.3.2.1" xref="S3.I2.i2.p1.3.m3.1.1.1.1.1.3.2.1.cmml">/</mo><mn id="S3.I2.i2.p1.3.m3.1.1.1.1.1.3.2.3" xref="S3.I2.i2.p1.3.m3.1.1.1.1.1.3.2.3.cmml">2</mn></mrow></mrow></msup><mo stretchy="false" id="S3.I2.i2.p1.3.m3.1.1.1.1.3" xref="S3.I2.i2.p1.3.m3.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.I2.i2.p1.3.m3.1b"><apply id="S3.I2.i2.p1.3.m3.1.1.cmml" xref="S3.I2.i2.p1.3.m3.1.1"><times id="S3.I2.i2.p1.3.m3.1.1.2.cmml" xref="S3.I2.i2.p1.3.m3.1.1.2"></times><ci id="S3.I2.i2.p1.3.m3.1.1.3.cmml" xref="S3.I2.i2.p1.3.m3.1.1.3">Θ</ci><apply id="S3.I2.i2.p1.3.m3.1.1.1.1.1.cmml" xref="S3.I2.i2.p1.3.m3.1.1.1.1"><csymbol cd="ambiguous" id="S3.I2.i2.p1.3.m3.1.1.1.1.1.1.cmml" xref="S3.I2.i2.p1.3.m3.1.1.1.1">superscript</csymbol><ci id="S3.I2.i2.p1.3.m3.1.1.1.1.1.2.cmml" xref="S3.I2.i2.p1.3.m3.1.1.1.1.1.2">𝑑</ci><apply id="S3.I2.i2.p1.3.m3.1.1.1.1.1.3.cmml" xref="S3.I2.i2.p1.3.m3.1.1.1.1.1.3"><minus id="S3.I2.i2.p1.3.m3.1.1.1.1.1.3.1.cmml" xref="S3.I2.i2.p1.3.m3.1.1.1.1.1.3"></minus><apply id="S3.I2.i2.p1.3.m3.1.1.1.1.1.3.2.cmml" xref="S3.I2.i2.p1.3.m3.1.1.1.1.1.3.2"><divide id="S3.I2.i2.p1.3.m3.1.1.1.1.1.3.2.1.cmml" xref="S3.I2.i2.p1.3.m3.1.1.1.1.1.3.2.1"></divide><cn type="integer" id="S3.I2.i2.p1.3.m3.1.1.1.1.1.3.2.2.cmml" xref="S3.I2.i2.p1.3.m3.1.1.1.1.1.3.2.2">1</cn><cn type="integer" id="S3.I2.i2.p1.3.m3.1.1.1.1.1.3.2.3.cmml" xref="S3.I2.i2.p1.3.m3.1.1.1.1.1.3.2.3">2</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I2.i2.p1.3.m3.1c">\Theta(d^{-1/2})</annotation></semantics></math> <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib47" title="" class="ltx_ref">47</a>, <a href="#bib.bib52" title="" class="ltx_ref">52</a>]</cite>) can encourage the model to learn compositional inference instead of memorization.
Specially, an arithmetic dataset is designed with a train set and an out-of-distribution test set, which admits two possible answers.
One answer relies on memorizing more rules during training, while the other requires an understanding of the compositional structure underlying these rules.
The proposed mechanism is that training with smaller initialization belongs to the condensed regime that encourages sparse solutions, contrary to training with large initialization that belongs to the kernel regime or critical regime <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib78" title="" class="ltx_ref">78</a>, <a href="#bib.bib19" title="" class="ltx_ref">19</a>]</cite>.</p>
</div>
</li>
<li id="S3.I2.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><span id="S3.I2.i3.1.1.1" class="ltx_text ltx_font_bold" style="color:#00868B;">0.</span></span> 
<div id="S3.I2.i3.p1" class="ltx_para">
<p id="S3.I2.i3.p1.1" class="ltx_p">Weight decay: <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib90" title="" class="ltx_ref">90</a>, <a href="#bib.bib88" title="" class="ltx_ref">88</a>]</cite> observe that using a larger weight decay coefficient (i.e. greater than the usual range of <math id="S3.I2.i3.p1.1.m1.1" class="ltx_Math" alttext="0.001\sim 0.1" display="inline"><semantics id="S3.I2.i3.p1.1.m1.1a"><mrow id="S3.I2.i3.p1.1.m1.1.1" xref="S3.I2.i3.p1.1.m1.1.1.cmml"><mn id="S3.I2.i3.p1.1.m1.1.1.2" xref="S3.I2.i3.p1.1.m1.1.1.2.cmml">0.001</mn><mo id="S3.I2.i3.p1.1.m1.1.1.1" xref="S3.I2.i3.p1.1.m1.1.1.1.cmml">∼</mo><mn id="S3.I2.i3.p1.1.m1.1.1.3" xref="S3.I2.i3.p1.1.m1.1.1.3.cmml">0.1</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.I2.i3.p1.1.m1.1b"><apply id="S3.I2.i3.p1.1.m1.1.1.cmml" xref="S3.I2.i3.p1.1.m1.1.1"><csymbol cd="latexml" id="S3.I2.i3.p1.1.m1.1.1.1.cmml" xref="S3.I2.i3.p1.1.m1.1.1.1">similar-to</csymbol><cn type="float" id="S3.I2.i3.p1.1.m1.1.1.2.cmml" xref="S3.I2.i3.p1.1.m1.1.1.2">0.001</cn><cn type="float" id="S3.I2.i3.p1.1.m1.1.1.3.cmml" xref="S3.I2.i3.p1.1.m1.1.1.3">0.1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I2.i3.p1.1.m1.1c">0.001\sim 0.1</annotation></semantics></math>) can guide LLMs to favor generalization over memorization, and accelerate the learning of generalizable solutions.
They consider settings that exhibit grokking <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib90" title="" class="ltx_ref">90</a>]</cite> such that training would transit from perfect train accuracy and zero test accuracy to perfect test accuracy, and generalization ability is measured by how quickly this transition occurs.
Moreover, theoretically speaking, it is expected that training generative models needs stronger regularization than training regression models, in order to prevent the generated distributions from collapsing onto the training data and become trivial <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib128" title="" class="ltx_ref">128</a>]</cite>.</p>
</div>
</li>
</ol>
</div>
<div id="S3.SS5.p3" class="ltx_para">
<p id="S3.SS5.p3.1" class="ltx_p">In summary, it is recommendable to pretrain the Memory<sup id="S3.SS5.p3.1.1" class="ltx_sup">3</sup> model with a data composition that emphasizes abstract knowledges and minimizes specific information, a smaller initialization for parameters, and a larger weight decay coefficient.</p>
</div>
<div id="S3.SS5.p4" class="ltx_para">
<p id="S3.SS5.p4.2" class="ltx_p">Since this work is only a preliminary version of Memory<sup id="S3.SS5.p4.2.1" class="ltx_sup">3</sup>, we decide to stick with the conventional setting for training and have not experimented with any of these ideas.
We look forward to incorporating these designs in future versions of the Memory<sup id="S3.SS5.p4.2.2" class="ltx_sup">3</sup> model.</p>
</div>
</section>
<section id="S3.SS6" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsection" style="font-size:120%;color:#00868B;">3.6 <span id="S3.SS6.1.1" class="ltx_text" style="color:#808080;">| </span>Two-stage Pretrain</h3>

<div id="S3.SS6.p1" class="ltx_para">
<p id="S3.SS6.p1.1" class="ltx_p">The Memory<sup id="S3.SS6.p1.1.1" class="ltx_sup">3</sup> model learns to write and read explicit memories during pretraining.
The training data is prepended with retrieved references;
the model encodes these references into explicit memories in real time, and integrates them into the self-attention computation of the training data.</p>
</div>
<div id="S3.SS6.p2" class="ltx_para">
<p id="S3.SS6.p2.1" class="ltx_p">Unexpectedly, our pretraining consists of two stages, which we name as warmup and continual train.
Only the continual train stage involves explicit memories, while the warmup stage uses the same format as ordinary pretraining.
Our motivation is depicted in Figure <a href="#F11" title="Figure 11 ‣ 3.6 | Two-stage Pretrain ‣ 3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a>.
We observe that pretraining with explicit memories from the beginning would render the memories useless, as there appears to be no gain in training loss compared to ordinary pretraining.
Meanwhile, given a checkpoint from ordinary pretraining, continual training with explicit memory exhibits a visible decrease in training loss.
This comparison implies that a memory-less warmup stage might be necessary for pretraining a Memory<sup id="S3.SS6.p2.1.1" class="ltx_sup">3</sup> model.
One possible explanation for this phenomenon is that in the beginning of pretraining, the model is too weak to understand and leverage the explicit memories it generates.
Then, to reduce distraction, the self-attention layers might learn to always ignore these memories, thus hindering indefinitely the development of explicit memory.</p>
</div>
<figure id="F11" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="F11.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.01178/assets/Figures/key_figure/warmup_l40-h32-d64-ml20_smooth95.png" id="F11.sf1.g1" class="ltx_graphics ltx_img_landscape" width="287" height="162" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="F11.sf1.2.1.1" class="ltx_text" style="font-size:90%;">(a)</span> </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="F11.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.01178/assets/Figures/key_figure/continual_l40-h32-d64-ml20_smooth95.png" id="F11.sf2.g1" class="ltx_graphics ltx_img_landscape" width="287" height="163" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="F11.sf2.2.1.1" class="ltx_text" style="font-size:90%;">(b)</span> </span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="F11.7.3.1" class="ltx_text" style="font-size:90%;">Figure 11</span>: </span><span id="F11.4.2" class="ltx_text" style="font-size:90%;">Left: Comparison of the warmup stage (training from scratch) with and without explicit memory.
The blue and green curves are trained without and with explicit memories, respectively.
Right: Comparison of the continual train stage.
The blue and green curves are continual trained from their warmup checkpoints, and the red curve is initialized with the warmup checkpoint of the blue curve and continual trained with explicit memory.
These plots indicate that pretraining a Memory<sup id="F11.4.2.1" class="ltx_sup">3</sup> model requires a memory-less warmup stage.
These experiments use a smaller model with 0.92B non-embedding parameters <math id="F11.4.2.m2.1" class="ltx_Math" alttext="(L=40,H=32,d_{h}=64)" display="inline"><semantics id="F11.4.2.m2.1b"><mrow id="F11.4.2.m2.1.1.1"><mo stretchy="false" id="F11.4.2.m2.1.1.1.2">(</mo><mrow id="F11.4.2.m2.1.1.1.1.2" xref="F11.4.2.m2.1.1.1.1.3.cmml"><mrow id="F11.4.2.m2.1.1.1.1.1.1" xref="F11.4.2.m2.1.1.1.1.1.1.cmml"><mi id="F11.4.2.m2.1.1.1.1.1.1.2" xref="F11.4.2.m2.1.1.1.1.1.1.2.cmml">L</mi><mo id="F11.4.2.m2.1.1.1.1.1.1.1" xref="F11.4.2.m2.1.1.1.1.1.1.1.cmml">=</mo><mn id="F11.4.2.m2.1.1.1.1.1.1.3" xref="F11.4.2.m2.1.1.1.1.1.1.3.cmml">40</mn></mrow><mo id="F11.4.2.m2.1.1.1.1.2.3" xref="F11.4.2.m2.1.1.1.1.3a.cmml">,</mo><mrow id="F11.4.2.m2.1.1.1.1.2.2.2" xref="F11.4.2.m2.1.1.1.1.2.2.3.cmml"><mrow id="F11.4.2.m2.1.1.1.1.2.2.1.1" xref="F11.4.2.m2.1.1.1.1.2.2.1.1.cmml"><mi id="F11.4.2.m2.1.1.1.1.2.2.1.1.2" xref="F11.4.2.m2.1.1.1.1.2.2.1.1.2.cmml">H</mi><mo id="F11.4.2.m2.1.1.1.1.2.2.1.1.1" xref="F11.4.2.m2.1.1.1.1.2.2.1.1.1.cmml">=</mo><mn id="F11.4.2.m2.1.1.1.1.2.2.1.1.3" xref="F11.4.2.m2.1.1.1.1.2.2.1.1.3.cmml">32</mn></mrow><mo id="F11.4.2.m2.1.1.1.1.2.2.2.3" xref="F11.4.2.m2.1.1.1.1.2.2.3a.cmml">,</mo><mrow id="F11.4.2.m2.1.1.1.1.2.2.2.2" xref="F11.4.2.m2.1.1.1.1.2.2.2.2.cmml"><msub id="F11.4.2.m2.1.1.1.1.2.2.2.2.2" xref="F11.4.2.m2.1.1.1.1.2.2.2.2.2.cmml"><mi id="F11.4.2.m2.1.1.1.1.2.2.2.2.2.2" xref="F11.4.2.m2.1.1.1.1.2.2.2.2.2.2.cmml">d</mi><mi id="F11.4.2.m2.1.1.1.1.2.2.2.2.2.3" xref="F11.4.2.m2.1.1.1.1.2.2.2.2.2.3.cmml">h</mi></msub><mo id="F11.4.2.m2.1.1.1.1.2.2.2.2.1" xref="F11.4.2.m2.1.1.1.1.2.2.2.2.1.cmml">=</mo><mn id="F11.4.2.m2.1.1.1.1.2.2.2.2.3" xref="F11.4.2.m2.1.1.1.1.2.2.2.2.3.cmml">64</mn></mrow></mrow></mrow><mo stretchy="false" id="F11.4.2.m2.1.1.1.3">)</mo></mrow><annotation-xml encoding="MathML-Content" id="F11.4.2.m2.1c"><apply id="F11.4.2.m2.1.1.1.1.3.cmml" xref="F11.4.2.m2.1.1.1.1.2"><csymbol cd="ambiguous" id="F11.4.2.m2.1.1.1.1.3a.cmml" xref="F11.4.2.m2.1.1.1.1.2.3">formulae-sequence</csymbol><apply id="F11.4.2.m2.1.1.1.1.1.1.cmml" xref="F11.4.2.m2.1.1.1.1.1.1"><eq id="F11.4.2.m2.1.1.1.1.1.1.1.cmml" xref="F11.4.2.m2.1.1.1.1.1.1.1"></eq><ci id="F11.4.2.m2.1.1.1.1.1.1.2.cmml" xref="F11.4.2.m2.1.1.1.1.1.1.2">𝐿</ci><cn type="integer" id="F11.4.2.m2.1.1.1.1.1.1.3.cmml" xref="F11.4.2.m2.1.1.1.1.1.1.3">40</cn></apply><apply id="F11.4.2.m2.1.1.1.1.2.2.3.cmml" xref="F11.4.2.m2.1.1.1.1.2.2.2"><csymbol cd="ambiguous" id="F11.4.2.m2.1.1.1.1.2.2.3a.cmml" xref="F11.4.2.m2.1.1.1.1.2.2.2.3">formulae-sequence</csymbol><apply id="F11.4.2.m2.1.1.1.1.2.2.1.1.cmml" xref="F11.4.2.m2.1.1.1.1.2.2.1.1"><eq id="F11.4.2.m2.1.1.1.1.2.2.1.1.1.cmml" xref="F11.4.2.m2.1.1.1.1.2.2.1.1.1"></eq><ci id="F11.4.2.m2.1.1.1.1.2.2.1.1.2.cmml" xref="F11.4.2.m2.1.1.1.1.2.2.1.1.2">𝐻</ci><cn type="integer" id="F11.4.2.m2.1.1.1.1.2.2.1.1.3.cmml" xref="F11.4.2.m2.1.1.1.1.2.2.1.1.3">32</cn></apply><apply id="F11.4.2.m2.1.1.1.1.2.2.2.2.cmml" xref="F11.4.2.m2.1.1.1.1.2.2.2.2"><eq id="F11.4.2.m2.1.1.1.1.2.2.2.2.1.cmml" xref="F11.4.2.m2.1.1.1.1.2.2.2.2.1"></eq><apply id="F11.4.2.m2.1.1.1.1.2.2.2.2.2.cmml" xref="F11.4.2.m2.1.1.1.1.2.2.2.2.2"><csymbol cd="ambiguous" id="F11.4.2.m2.1.1.1.1.2.2.2.2.2.1.cmml" xref="F11.4.2.m2.1.1.1.1.2.2.2.2.2">subscript</csymbol><ci id="F11.4.2.m2.1.1.1.1.2.2.2.2.2.2.cmml" xref="F11.4.2.m2.1.1.1.1.2.2.2.2.2.2">𝑑</ci><ci id="F11.4.2.m2.1.1.1.1.2.2.2.2.2.3.cmml" xref="F11.4.2.m2.1.1.1.1.2.2.2.2.2.3">ℎ</ci></apply><cn type="integer" id="F11.4.2.m2.1.1.1.1.2.2.2.2.3.cmml" xref="F11.4.2.m2.1.1.1.1.2.2.2.2.3">64</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="F11.4.2.m2.1d">(L=40,H=32,d_{h}=64)</annotation></semantics></math>.
The warmup stage uses 60B data and the continual train stage uses 22B.</span></figcaption>
</figure>
<div id="S3.SS6.p3" class="ltx_para">
<p id="S3.SS6.p3.1" class="ltx_p">Another modification is to reduce the cost of continual train.
Recall from Section <a href="#S3.SS1" title="3.1 | Inference Process ‣ 3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.1</span></a> that during inference, each 64-token chunk attends to five explicit memories, or equivalently five 128-token references if using cold start, increasing the amount of input tokens by 10 times.
The inference process avoids the cost of memory encoding by precomputation or warm start, but for the continual train, the references need to be encoded in real time.
Our solution is to let the chunks share their references during training to reduce the total number of references in a batch.
Specifically, each chunk of a training sequence retrieves only one reference, and in compensation, attends to the references of the previous four chunks, besides its own reference.
Each train sequence has length 2048 and thus 32 chunks, so it is equipped with <math id="S3.SS6.p3.1.m1.1" class="ltx_Math" alttext="32\times 128=4096" display="inline"><semantics id="S3.SS6.p3.1.m1.1a"><mrow id="S3.SS6.p3.1.m1.1.1" xref="S3.SS6.p3.1.m1.1.1.cmml"><mrow id="S3.SS6.p3.1.m1.1.1.2" xref="S3.SS6.p3.1.m1.1.1.2.cmml"><mn id="S3.SS6.p3.1.m1.1.1.2.2" xref="S3.SS6.p3.1.m1.1.1.2.2.cmml">32</mn><mo lspace="0.222em" rspace="0.222em" id="S3.SS6.p3.1.m1.1.1.2.1" xref="S3.SS6.p3.1.m1.1.1.2.1.cmml">×</mo><mn id="S3.SS6.p3.1.m1.1.1.2.3" xref="S3.SS6.p3.1.m1.1.1.2.3.cmml">128</mn></mrow><mo id="S3.SS6.p3.1.m1.1.1.1" xref="S3.SS6.p3.1.m1.1.1.1.cmml">=</mo><mn id="S3.SS6.p3.1.m1.1.1.3" xref="S3.SS6.p3.1.m1.1.1.3.cmml">4096</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS6.p3.1.m1.1b"><apply id="S3.SS6.p3.1.m1.1.1.cmml" xref="S3.SS6.p3.1.m1.1.1"><eq id="S3.SS6.p3.1.m1.1.1.1.cmml" xref="S3.SS6.p3.1.m1.1.1.1"></eq><apply id="S3.SS6.p3.1.m1.1.1.2.cmml" xref="S3.SS6.p3.1.m1.1.1.2"><times id="S3.SS6.p3.1.m1.1.1.2.1.cmml" xref="S3.SS6.p3.1.m1.1.1.2.1"></times><cn type="integer" id="S3.SS6.p3.1.m1.1.1.2.2.cmml" xref="S3.SS6.p3.1.m1.1.1.2.2">32</cn><cn type="integer" id="S3.SS6.p3.1.m1.1.1.2.3.cmml" xref="S3.SS6.p3.1.m1.1.1.2.3">128</cn></apply><cn type="integer" id="S3.SS6.p3.1.m1.1.1.3.cmml" xref="S3.SS6.p3.1.m1.1.1.3">4096</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.p3.1.m1.1c">32\times 128=4096</annotation></semantics></math> reference tokens.
The hidden features of these reference tokens are discarded once passing the last memory layer, since after that they no longer participate in the update of the hidden feature of the train tokens.
Hence, each continual train step takes slightly more than twice the amount of time of a warmup step.</p>
</div>
<div id="S3.SS6.p4" class="ltx_para">
<p id="S3.SS6.p4.3" class="ltx_p">It is necessary to avoid information leakage when equipping the training data with references
(i.e. the train sequence and its retrieved references could be the same text), for otherwise training becomes too easy and the model would not learn much.
Previously, Retro <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib16" title="" class="ltx_ref">16</a>]</cite> requires that no train sequence can retrieve a reference from the same document, but this criterion may be insufficient since near-identical paragraphs may appear in multiple documents.
Thus, we require that no train sequence can be accompanied by a reference sequence that has <math id="S3.SS6.p4.1.m1.1" class="ltx_Math" alttext="&gt;90\%" display="inline"><semantics id="S3.SS6.p4.1.m1.1a"><mrow id="S3.SS6.p4.1.m1.1.1" xref="S3.SS6.p4.1.m1.1.1.cmml"><mi id="S3.SS6.p4.1.m1.1.1.2" xref="S3.SS6.p4.1.m1.1.1.2.cmml"></mi><mo id="S3.SS6.p4.1.m1.1.1.1" xref="S3.SS6.p4.1.m1.1.1.1.cmml">&gt;</mo><mrow id="S3.SS6.p4.1.m1.1.1.3" xref="S3.SS6.p4.1.m1.1.1.3.cmml"><mn id="S3.SS6.p4.1.m1.1.1.3.2" xref="S3.SS6.p4.1.m1.1.1.3.2.cmml">90</mn><mo id="S3.SS6.p4.1.m1.1.1.3.1" xref="S3.SS6.p4.1.m1.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS6.p4.1.m1.1b"><apply id="S3.SS6.p4.1.m1.1.1.cmml" xref="S3.SS6.p4.1.m1.1.1"><gt id="S3.SS6.p4.1.m1.1.1.1.cmml" xref="S3.SS6.p4.1.m1.1.1.1"></gt><csymbol cd="latexml" id="S3.SS6.p4.1.m1.1.1.2.cmml" xref="S3.SS6.p4.1.m1.1.1.2">absent</csymbol><apply id="S3.SS6.p4.1.m1.1.1.3.cmml" xref="S3.SS6.p4.1.m1.1.1.3"><csymbol cd="latexml" id="S3.SS6.p4.1.m1.1.1.3.1.cmml" xref="S3.SS6.p4.1.m1.1.1.3.1">percent</csymbol><cn type="integer" id="S3.SS6.p4.1.m1.1.1.3.2.cmml" xref="S3.SS6.p4.1.m1.1.1.3.2">90</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.p4.1.m1.1c">&gt;90\%</annotation></semantics></math> overlap with it.
The overlap is measured by the length of their longest common subsequence divided by the length of the reference length.
Specially, given any train sequence <math id="S3.SS6.p4.2.m2.1" class="ltx_Math" alttext="\mathbf{t}" display="inline"><semantics id="S3.SS6.p4.2.m2.1a"><mi id="S3.SS6.p4.2.m2.1.1" xref="S3.SS6.p4.2.m2.1.1.cmml">𝐭</mi><annotation-xml encoding="MathML-Content" id="S3.SS6.p4.2.m2.1b"><ci id="S3.SS6.p4.2.m2.1.1.cmml" xref="S3.SS6.p4.2.m2.1.1">𝐭</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.p4.2.m2.1c">\mathbf{t}</annotation></semantics></math> and reference <math id="S3.SS6.p4.3.m3.1" class="ltx_Math" alttext="\mathbf{r}" display="inline"><semantics id="S3.SS6.p4.3.m3.1a"><mi id="S3.SS6.p4.3.m3.1.1" xref="S3.SS6.p4.3.m3.1.1.cmml">𝐫</mi><annotation-xml encoding="MathML-Content" id="S3.SS6.p4.3.m3.1b"><ci id="S3.SS6.p4.3.m3.1.1.cmml" xref="S3.SS6.p4.3.m3.1.1">𝐫</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.p4.3.m3.1c">\mathbf{r}</annotation></semantics></math>, define their overlap by</p>
<table id="A3.EGx1" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="E6"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td colspan="2" class="ltx_td ltx_align_center ltx_eqn_cell"><math id="E6.m1.72" class="ltx_Math" alttext="\displaystyle\begin{split}\text{overlap}(\mathbf{t},\mathbf{r}):=\frac{1}{|\mathbf{r}|}\max\big{\{}N&amp;~{}\big{|}~{}\exists 1\leq i_{1}&lt;\dots&lt;i_{N}\leq|\mathbf{t}|~{}\text{and}~{}\exists 1\leq j_{1}&lt;\dots&lt;j_{N}\leq|\mathbf{r}|\\
&amp;~{}\text{and}~{}|i_{N}-i_{1}|\leq 2|\mathbf{r}|,~{}\text{such that}~{}\mathbf{t}_{i_{k}}=\mathbf{r}_{j_{k}}~{}\text{for}~{}k=1,\dots N\big{\}}\end{split}" display="inline"><semantics id="E6.m1.72a"><mtable columnspacing="0pt" rowspacing="0pt" id="E6.m1.69.69a" xref="E6.m1.72.72.3.cmml"><mtr id="E6.m1.69.69aa" xref="E6.m1.72.72.3.cmml"><mtd class="ltx_align_right" columnalign="right" id="E6.m1.69.69ab" xref="E6.m1.72.72.3.cmml"><mrow id="E6.m1.11.11.11.11.11a" xref="E6.m1.72.72.3.cmml"><mtext id="E6.m1.1.1.1.1.1.1" xref="E6.m1.1.1.1.1.1.1a.cmml">overlap</mtext><mrow id="E6.m1.11.11.11.11.11a.12" xref="E6.m1.72.72.3.cmml"><mo stretchy="false" id="E6.m1.2.2.2.2.2.2" xref="E6.m1.72.72.3.cmml">(</mo><mi id="E6.m1.3.3.3.3.3.3" xref="E6.m1.3.3.3.3.3.3.cmml">𝐭</mi><mo id="E6.m1.4.4.4.4.4.4" xref="E6.m1.72.72.3.cmml">,</mo><mi id="E6.m1.5.5.5.5.5.5" xref="E6.m1.5.5.5.5.5.5.cmml">𝐫</mi><mo rspace="0.278em" stretchy="false" id="E6.m1.6.6.6.6.6.6" xref="E6.m1.72.72.3.cmml">)</mo></mrow><mo rspace="0.278em" id="E6.m1.7.7.7.7.7.7" xref="E6.m1.7.7.7.7.7.7.cmml">:=</mo><mstyle displaystyle="true" id="E6.m1.8.8.8.8.8.8" xref="E6.m1.8.8.8.8.8.8.cmml"><mfrac id="E6.m1.8.8.8.8.8.8a" xref="E6.m1.8.8.8.8.8.8.cmml"><mn id="E6.m1.8.8.8.8.8.8.3" xref="E6.m1.8.8.8.8.8.8.3.cmml">1</mn><mrow id="E6.m1.8.8.8.8.8.8.1.3" xref="E6.m1.8.8.8.8.8.8.1.2.cmml"><mo stretchy="false" id="E6.m1.8.8.8.8.8.8.1.3.1" xref="E6.m1.8.8.8.8.8.8.1.2.1.cmml">|</mo><mi id="E6.m1.8.8.8.8.8.8.1.1" xref="E6.m1.8.8.8.8.8.8.1.1.cmml">𝐫</mi><mo stretchy="false" id="E6.m1.8.8.8.8.8.8.1.3.2" xref="E6.m1.8.8.8.8.8.8.1.2.1.cmml">|</mo></mrow></mfrac></mstyle><mi id="E6.m1.9.9.9.9.9.9" xref="E6.m1.9.9.9.9.9.9.cmml">max</mi><mrow id="E6.m1.11.11.11.11.11a.13" xref="E6.m1.72.72.3.cmml"><mo maxsize="120%" minsize="120%" id="E6.m1.10.10.10.10.10.10" xref="E6.m1.72.72.3.cmml">{</mo><mi id="E6.m1.11.11.11.11.11.11" xref="E6.m1.11.11.11.11.11.11.cmml">N</mi></mrow></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="E6.m1.69.69ac" xref="E6.m1.72.72.3.cmml"><mrow id="E6.m1.41.41.41.41.30a" xref="E6.m1.72.72.3.cmml"><mo fence="false" maxsize="120%" minsize="120%" rspace="0.497em" id="E6.m1.12.12.12.12.1.1" xref="E6.m1.12.12.12.12.1.1.cmml">|</mo><mo rspace="0.167em" id="E6.m1.13.13.13.13.2.2" xref="E6.m1.13.13.13.13.2.2.cmml">∃</mo><mn id="E6.m1.14.14.14.14.3.3" xref="E6.m1.14.14.14.14.3.3.cmml">1</mn><mo id="E6.m1.15.15.15.15.4.4" xref="E6.m1.15.15.15.15.4.4.cmml">≤</mo><msub id="E6.m1.41.41.41.41.30a.31" xref="E6.m1.72.72.3.cmml"><mi id="E6.m1.16.16.16.16.5.5" xref="E6.m1.16.16.16.16.5.5.cmml">i</mi><mn id="E6.m1.17.17.17.17.6.6.1" xref="E6.m1.17.17.17.17.6.6.1.cmml">1</mn></msub><mo id="E6.m1.18.18.18.18.7.7" xref="E6.m1.18.18.18.18.7.7.cmml">&lt;</mo><mi mathvariant="normal" id="E6.m1.19.19.19.19.8.8" xref="E6.m1.19.19.19.19.8.8.cmml">⋯</mi><mo id="E6.m1.20.20.20.20.9.9" xref="E6.m1.20.20.20.20.9.9.cmml">&lt;</mo><msub id="E6.m1.41.41.41.41.30a.32" xref="E6.m1.72.72.3.cmml"><mi id="E6.m1.21.21.21.21.10.10" xref="E6.m1.21.21.21.21.10.10.cmml">i</mi><mi id="E6.m1.22.22.22.22.11.11.1" xref="E6.m1.22.22.22.22.11.11.1.cmml">N</mi></msub><mo rspace="0em" id="E6.m1.23.23.23.23.12.12" xref="E6.m1.23.23.23.23.12.12.cmml">≤</mo><mo fence="false" rspace="0.167em" stretchy="false" id="E6.m1.24.24.24.24.13.13" xref="E6.m1.72.72.3.cmml">|</mo><mi id="E6.m1.25.25.25.25.14.14" xref="E6.m1.25.25.25.25.14.14.cmml">𝐭</mi><mo fence="false" rspace="0.497em" stretchy="false" id="E6.m1.26.26.26.26.15.15" xref="E6.m1.72.72.3.cmml">|</mo><mtext id="E6.m1.27.27.27.27.16.16" xref="E6.m1.27.27.27.27.16.16a.cmml">and</mtext><mo lspace="0.497em" rspace="0.167em" id="E6.m1.28.28.28.28.17.17" xref="E6.m1.28.28.28.28.17.17.cmml">∃</mo><mn id="E6.m1.29.29.29.29.18.18" xref="E6.m1.29.29.29.29.18.18.cmml">1</mn><mo id="E6.m1.30.30.30.30.19.19" xref="E6.m1.30.30.30.30.19.19.cmml">≤</mo><msub id="E6.m1.41.41.41.41.30a.33" xref="E6.m1.72.72.3.cmml"><mi id="E6.m1.31.31.31.31.20.20" xref="E6.m1.31.31.31.31.20.20.cmml">j</mi><mn id="E6.m1.32.32.32.32.21.21.1" xref="E6.m1.32.32.32.32.21.21.1.cmml">1</mn></msub><mo id="E6.m1.33.33.33.33.22.22" xref="E6.m1.33.33.33.33.22.22.cmml">&lt;</mo><mi mathvariant="normal" id="E6.m1.34.34.34.34.23.23" xref="E6.m1.34.34.34.34.23.23.cmml">⋯</mi><mo id="E6.m1.35.35.35.35.24.24" xref="E6.m1.35.35.35.35.24.24.cmml">&lt;</mo><msub id="E6.m1.41.41.41.41.30a.34" xref="E6.m1.72.72.3.cmml"><mi id="E6.m1.36.36.36.36.25.25" xref="E6.m1.36.36.36.36.25.25.cmml">j</mi><mi id="E6.m1.37.37.37.37.26.26.1" xref="E6.m1.37.37.37.37.26.26.1.cmml">N</mi></msub><mo rspace="0em" id="E6.m1.38.38.38.38.27.27" xref="E6.m1.38.38.38.38.27.27.cmml">≤</mo><mo fence="false" rspace="0.167em" stretchy="false" id="E6.m1.39.39.39.39.28.28" xref="E6.m1.72.72.3.cmml">|</mo><mi id="E6.m1.40.40.40.40.29.29" xref="E6.m1.40.40.40.40.29.29.cmml">𝐫</mi><mo fence="false" stretchy="false" id="E6.m1.41.41.41.41.30.30" xref="E6.m1.72.72.3.cmml">|</mo></mrow></mtd></mtr><mtr id="E6.m1.69.69ad" xref="E6.m1.72.72.3.cmml"><mtd id="E6.m1.69.69ae" xref="E6.m1.72.72.3.cmml"></mtd><mtd class="ltx_align_left" columnalign="left" id="E6.m1.69.69af" xref="E6.m1.72.72.3.cmml"><mrow id="E6.m1.69.69.69.28.28a" xref="E6.m1.72.72.3.cmml"><mtext id="E6.m1.42.42.42.1.1.1" xref="E6.m1.42.42.42.1.1.1a.cmml">and</mtext><mo fence="false" lspace="0.330em" rspace="0.167em" stretchy="false" id="E6.m1.43.43.43.2.2.2" xref="E6.m1.72.72.3.cmml">|</mo><msub id="E6.m1.69.69.69.28.28a.29" xref="E6.m1.72.72.3.cmml"><mi id="E6.m1.44.44.44.3.3.3" xref="E6.m1.44.44.44.3.3.3.cmml">i</mi><mi id="E6.m1.45.45.45.4.4.4.1" xref="E6.m1.45.45.45.4.4.4.1.cmml">N</mi></msub><mo id="E6.m1.46.46.46.5.5.5" xref="E6.m1.46.46.46.5.5.5.cmml">−</mo><msub id="E6.m1.69.69.69.28.28a.30" xref="E6.m1.72.72.3.cmml"><mi id="E6.m1.47.47.47.6.6.6" xref="E6.m1.47.47.47.6.6.6.cmml">i</mi><mn id="E6.m1.48.48.48.7.7.7.1" xref="E6.m1.48.48.48.7.7.7.1.cmml">1</mn></msub><mo fence="false" stretchy="false" id="E6.m1.49.49.49.8.8.8" xref="E6.m1.72.72.3.cmml">|</mo><mo lspace="0.167em" id="E6.m1.50.50.50.9.9.9" xref="E6.m1.50.50.50.9.9.9.cmml">≤</mo><mn id="E6.m1.51.51.51.10.10.10" xref="E6.m1.51.51.51.10.10.10.cmml">2</mn><mo fence="false" rspace="0.167em" stretchy="false" id="E6.m1.52.52.52.11.11.11" xref="E6.m1.72.72.3.cmml">|</mo><mi id="E6.m1.53.53.53.12.12.12" xref="E6.m1.53.53.53.12.12.12.cmml">𝐫</mi><mo fence="false" rspace="0.167em" stretchy="false" id="E6.m1.54.54.54.13.13.13" xref="E6.m1.72.72.3.cmml">|</mo><mo rspace="0.497em" id="E6.m1.55.55.55.14.14.14" xref="E6.m1.72.72.3.cmml">,</mo><mtext id="E6.m1.56.56.56.15.15.15" xref="E6.m1.56.56.56.15.15.15a.cmml">such that</mtext><msub id="E6.m1.69.69.69.28.28a.31" xref="E6.m1.72.72.3.cmml"><mi id="E6.m1.57.57.57.16.16.16" xref="E6.m1.57.57.57.16.16.16.cmml">𝐭</mi><msub id="E6.m1.58.58.58.17.17.17.1" xref="E6.m1.58.58.58.17.17.17.1.cmml"><mi id="E6.m1.58.58.58.17.17.17.1.2" xref="E6.m1.58.58.58.17.17.17.1.2.cmml">i</mi><mi id="E6.m1.58.58.58.17.17.17.1.3" xref="E6.m1.58.58.58.17.17.17.1.3.cmml">k</mi></msub></msub><mo id="E6.m1.59.59.59.18.18.18" xref="E6.m1.59.59.59.18.18.18.cmml">=</mo><msub id="E6.m1.69.69.69.28.28a.32" xref="E6.m1.72.72.3.cmml"><mi id="E6.m1.60.60.60.19.19.19" xref="E6.m1.60.60.60.19.19.19.cmml">𝐫</mi><msub id="E6.m1.61.61.61.20.20.20.1" xref="E6.m1.61.61.61.20.20.20.1.cmml"><mi id="E6.m1.61.61.61.20.20.20.1.2" xref="E6.m1.61.61.61.20.20.20.1.2.cmml">j</mi><mi id="E6.m1.61.61.61.20.20.20.1.3" xref="E6.m1.61.61.61.20.20.20.1.3.cmml">k</mi></msub></msub><mtext id="E6.m1.62.62.62.21.21.21" xref="E6.m1.62.62.62.21.21.21a.cmml">for</mtext><mi id="E6.m1.63.63.63.22.22.22" xref="E6.m1.63.63.63.22.22.22.cmml">k</mi><mo id="E6.m1.64.64.64.23.23.23" xref="E6.m1.64.64.64.23.23.23.cmml">=</mo><mn id="E6.m1.65.65.65.24.24.24" xref="E6.m1.65.65.65.24.24.24.cmml">1</mn><mo id="E6.m1.66.66.66.25.25.25" xref="E6.m1.72.72.3.cmml">,</mo><mi mathvariant="normal" id="E6.m1.67.67.67.26.26.26" xref="E6.m1.67.67.67.26.26.26.cmml">…</mi><mi id="E6.m1.68.68.68.27.27.27" xref="E6.m1.68.68.68.27.27.27.cmml">N</mi><mo maxsize="120%" minsize="120%" id="E6.m1.69.69.69.28.28.28" xref="E6.m1.72.72.3.cmml">}</mo></mrow></mtd></mtr></mtable><annotation-xml encoding="MathML-Content" id="E6.m1.72b"><apply id="E6.m1.72.72.3.cmml" xref="E6.m1.69.69a"><csymbol cd="latexml" id="E6.m1.7.7.7.7.7.7.cmml" xref="E6.m1.7.7.7.7.7.7">assign</csymbol><apply id="E6.m1.72.72.3.5.cmml" xref="E6.m1.69.69a"><times id="E6.m1.72.72.3.5.1.cmml" xref="E6.m1.69.69a"></times><ci id="E6.m1.1.1.1.1.1.1a.cmml" xref="E6.m1.1.1.1.1.1.1"><mtext id="E6.m1.1.1.1.1.1.1.cmml" xref="E6.m1.1.1.1.1.1.1">overlap</mtext></ci><interval closure="open" id="E6.m1.72.72.3.5.3.cmml" xref="E6.m1.69.69a"><ci id="E6.m1.3.3.3.3.3.3.cmml" xref="E6.m1.3.3.3.3.3.3">𝐭</ci><ci id="E6.m1.5.5.5.5.5.5.cmml" xref="E6.m1.5.5.5.5.5.5">𝐫</ci></interval></apply><apply id="E6.m1.72.72.3.3.cmml" xref="E6.m1.69.69a"><times id="E6.m1.72.72.3.3.4.cmml" xref="E6.m1.69.69a"></times><apply id="E6.m1.8.8.8.8.8.8.cmml" xref="E6.m1.8.8.8.8.8.8"><divide id="E6.m1.8.8.8.8.8.8.2.cmml" xref="E6.m1.8.8.8.8.8.8"></divide><cn type="integer" id="E6.m1.8.8.8.8.8.8.3.cmml" xref="E6.m1.8.8.8.8.8.8.3">1</cn><apply id="E6.m1.8.8.8.8.8.8.1.2.cmml" xref="E6.m1.8.8.8.8.8.8.1.3"><abs id="E6.m1.8.8.8.8.8.8.1.2.1.cmml" xref="E6.m1.8.8.8.8.8.8.1.3.1"></abs><ci id="E6.m1.8.8.8.8.8.8.1.1.cmml" xref="E6.m1.8.8.8.8.8.8.1.1">𝐫</ci></apply></apply><apply id="E6.m1.72.72.3.3.3.4.cmml" xref="E6.m1.69.69a"><max id="E6.m1.9.9.9.9.9.9.cmml" xref="E6.m1.9.9.9.9.9.9"></max><apply id="E6.m1.70.70.1.1.1.1.1.1.cmml" xref="E6.m1.69.69a"><and id="E6.m1.70.70.1.1.1.1.1.1a.cmml" xref="E6.m1.69.69a"></and><apply id="E6.m1.70.70.1.1.1.1.1.1b.cmml" xref="E6.m1.69.69a"><leq id="E6.m1.15.15.15.15.4.4.cmml" xref="E6.m1.15.15.15.15.4.4"></leq><apply id="E6.m1.70.70.1.1.1.1.1.1.3.cmml" xref="E6.m1.69.69a"><ci id="E6.m1.12.12.12.12.1.1.cmml" xref="E6.m1.12.12.12.12.1.1">|</ci><ci id="E6.m1.11.11.11.11.11.11.cmml" xref="E6.m1.11.11.11.11.11.11">𝑁</ci><apply id="E6.m1.70.70.1.1.1.1.1.1.3.3.cmml" xref="E6.m1.69.69a"><exists id="E6.m1.13.13.13.13.2.2.cmml" xref="E6.m1.13.13.13.13.2.2"></exists><cn type="integer" id="E6.m1.14.14.14.14.3.3.cmml" xref="E6.m1.14.14.14.14.3.3">1</cn></apply></apply><apply id="E6.m1.70.70.1.1.1.1.1.1.5.cmml" xref="E6.m1.69.69a"><csymbol cd="ambiguous" id="E6.m1.70.70.1.1.1.1.1.1.5.1.cmml" xref="E6.m1.69.69a">subscript</csymbol><ci id="E6.m1.16.16.16.16.5.5.cmml" xref="E6.m1.16.16.16.16.5.5">𝑖</ci><cn type="integer" id="E6.m1.17.17.17.17.6.6.1.cmml" xref="E6.m1.17.17.17.17.6.6.1">1</cn></apply></apply><apply id="E6.m1.70.70.1.1.1.1.1.1c.cmml" xref="E6.m1.69.69a"><lt id="E6.m1.18.18.18.18.7.7.cmml" xref="E6.m1.18.18.18.18.7.7"></lt><share href="#E6.m1.70.70.1.1.1.1.1.1.5.cmml" id="E6.m1.70.70.1.1.1.1.1.1d.cmml" xref="E6.m1.69.69a"></share><ci id="E6.m1.19.19.19.19.8.8.cmml" xref="E6.m1.19.19.19.19.8.8">⋯</ci></apply><apply id="E6.m1.70.70.1.1.1.1.1.1e.cmml" xref="E6.m1.69.69a"><lt id="E6.m1.20.20.20.20.9.9.cmml" xref="E6.m1.20.20.20.20.9.9"></lt><share href="#E6.m1.70.70.1.1.1.1.1.1.7.cmml" id="E6.m1.70.70.1.1.1.1.1.1f.cmml" xref="E6.m1.69.69a"></share><apply id="E6.m1.70.70.1.1.1.1.1.1.9.cmml" xref="E6.m1.69.69a"><csymbol cd="ambiguous" id="E6.m1.70.70.1.1.1.1.1.1.9.1.cmml" xref="E6.m1.69.69a">subscript</csymbol><ci id="E6.m1.21.21.21.21.10.10.cmml" xref="E6.m1.21.21.21.21.10.10">𝑖</ci><ci id="E6.m1.22.22.22.22.11.11.1.cmml" xref="E6.m1.22.22.22.22.11.11.1">𝑁</ci></apply></apply><apply id="E6.m1.70.70.1.1.1.1.1.1g.cmml" xref="E6.m1.69.69a"><leq id="E6.m1.23.23.23.23.12.12.cmml" xref="E6.m1.23.23.23.23.12.12"></leq><share href="#E6.m1.70.70.1.1.1.1.1.1.9.cmml" id="E6.m1.70.70.1.1.1.1.1.1h.cmml" xref="E6.m1.69.69a"></share><apply id="E6.m1.70.70.1.1.1.1.1.1.11.cmml" xref="E6.m1.69.69a"><times id="E6.m1.70.70.1.1.1.1.1.1.11.1.cmml" xref="E6.m1.69.69a"></times><apply id="E6.m1.70.70.1.1.1.1.1.1.11.2.cmml" xref="E6.m1.69.69a"><abs id="E6.m1.70.70.1.1.1.1.1.1.11.2.1.cmml" xref="E6.m1.69.69a"></abs><ci id="E6.m1.25.25.25.25.14.14.cmml" xref="E6.m1.25.25.25.25.14.14">𝐭</ci></apply><ci id="E6.m1.27.27.27.27.16.16a.cmml" xref="E6.m1.27.27.27.27.16.16"><mtext id="E6.m1.27.27.27.27.16.16.cmml" xref="E6.m1.27.27.27.27.16.16">and</mtext></ci><apply id="E6.m1.70.70.1.1.1.1.1.1.11.4.cmml" xref="E6.m1.69.69a"><exists id="E6.m1.28.28.28.28.17.17.cmml" xref="E6.m1.28.28.28.28.17.17"></exists><cn type="integer" id="E6.m1.29.29.29.29.18.18.cmml" xref="E6.m1.29.29.29.29.18.18">1</cn></apply></apply></apply><apply id="E6.m1.70.70.1.1.1.1.1.1i.cmml" xref="E6.m1.69.69a"><leq id="E6.m1.30.30.30.30.19.19.cmml" xref="E6.m1.30.30.30.30.19.19"></leq><share href="#E6.m1.70.70.1.1.1.1.1.1.11.cmml" id="E6.m1.70.70.1.1.1.1.1.1j.cmml" xref="E6.m1.69.69a"></share><apply id="E6.m1.70.70.1.1.1.1.1.1.13.cmml" xref="E6.m1.69.69a"><csymbol cd="ambiguous" id="E6.m1.70.70.1.1.1.1.1.1.13.1.cmml" xref="E6.m1.69.69a">subscript</csymbol><ci id="E6.m1.31.31.31.31.20.20.cmml" xref="E6.m1.31.31.31.31.20.20">𝑗</ci><cn type="integer" id="E6.m1.32.32.32.32.21.21.1.cmml" xref="E6.m1.32.32.32.32.21.21.1">1</cn></apply></apply><apply id="E6.m1.70.70.1.1.1.1.1.1k.cmml" xref="E6.m1.69.69a"><lt id="E6.m1.33.33.33.33.22.22.cmml" xref="E6.m1.33.33.33.33.22.22"></lt><share href="#E6.m1.70.70.1.1.1.1.1.1.13.cmml" id="E6.m1.70.70.1.1.1.1.1.1l.cmml" xref="E6.m1.69.69a"></share><ci id="E6.m1.34.34.34.34.23.23.cmml" xref="E6.m1.34.34.34.34.23.23">⋯</ci></apply><apply id="E6.m1.70.70.1.1.1.1.1.1m.cmml" xref="E6.m1.69.69a"><lt id="E6.m1.35.35.35.35.24.24.cmml" xref="E6.m1.35.35.35.35.24.24"></lt><share href="#E6.m1.70.70.1.1.1.1.1.1.15.cmml" id="E6.m1.70.70.1.1.1.1.1.1n.cmml" xref="E6.m1.69.69a"></share><apply id="E6.m1.70.70.1.1.1.1.1.1.17.cmml" xref="E6.m1.69.69a"><csymbol cd="ambiguous" id="E6.m1.70.70.1.1.1.1.1.1.17.1.cmml" xref="E6.m1.69.69a">subscript</csymbol><ci id="E6.m1.36.36.36.36.25.25.cmml" xref="E6.m1.36.36.36.36.25.25">𝑗</ci><ci id="E6.m1.37.37.37.37.26.26.1.cmml" xref="E6.m1.37.37.37.37.26.26.1">𝑁</ci></apply></apply><apply id="E6.m1.70.70.1.1.1.1.1.1o.cmml" xref="E6.m1.69.69a"><leq id="E6.m1.38.38.38.38.27.27.cmml" xref="E6.m1.38.38.38.38.27.27"></leq><share href="#E6.m1.70.70.1.1.1.1.1.1.17.cmml" id="E6.m1.70.70.1.1.1.1.1.1p.cmml" xref="E6.m1.69.69a"></share><apply id="E6.m1.70.70.1.1.1.1.1.1.1.cmml" xref="E6.m1.69.69a"><times id="E6.m1.70.70.1.1.1.1.1.1.1.2.cmml" xref="E6.m1.69.69a"></times><apply id="E6.m1.70.70.1.1.1.1.1.1.1.3.cmml" xref="E6.m1.69.69a"><abs id="E6.m1.70.70.1.1.1.1.1.1.1.3.1.cmml" xref="E6.m1.69.69a"></abs><ci id="E6.m1.40.40.40.40.29.29.cmml" xref="E6.m1.40.40.40.40.29.29">𝐫</ci></apply><ci id="E6.m1.42.42.42.1.1.1a.cmml" xref="E6.m1.42.42.42.1.1.1"><mtext id="E6.m1.42.42.42.1.1.1.cmml" xref="E6.m1.42.42.42.1.1.1">and</mtext></ci><apply id="E6.m1.70.70.1.1.1.1.1.1.1.1.2.cmml" xref="E6.m1.69.69a"><abs id="E6.m1.70.70.1.1.1.1.1.1.1.1.2.1.cmml" xref="E6.m1.69.69a"></abs><apply id="E6.m1.70.70.1.1.1.1.1.1.1.1.1.1.cmml" xref="E6.m1.69.69a"><minus id="E6.m1.46.46.46.5.5.5.cmml" xref="E6.m1.46.46.46.5.5.5"></minus><apply id="E6.m1.70.70.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="E6.m1.69.69a"><csymbol cd="ambiguous" id="E6.m1.70.70.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="E6.m1.69.69a">subscript</csymbol><ci id="E6.m1.44.44.44.3.3.3.cmml" xref="E6.m1.44.44.44.3.3.3">𝑖</ci><ci id="E6.m1.45.45.45.4.4.4.1.cmml" xref="E6.m1.45.45.45.4.4.4.1">𝑁</ci></apply><apply id="E6.m1.70.70.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="E6.m1.69.69a"><csymbol cd="ambiguous" id="E6.m1.70.70.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="E6.m1.69.69a">subscript</csymbol><ci id="E6.m1.47.47.47.6.6.6.cmml" xref="E6.m1.47.47.47.6.6.6">𝑖</ci><cn type="integer" id="E6.m1.48.48.48.7.7.7.1.cmml" xref="E6.m1.48.48.48.7.7.7.1">1</cn></apply></apply></apply></apply></apply><apply id="E6.m1.70.70.1.1.1.1.1.1q.cmml" xref="E6.m1.69.69a"><leq id="E6.m1.50.50.50.9.9.9.cmml" xref="E6.m1.50.50.50.9.9.9"></leq><share href="#E6.m1.70.70.1.1.1.1.1.1.1.cmml" id="E6.m1.70.70.1.1.1.1.1.1r.cmml" xref="E6.m1.69.69a"></share><apply id="E6.m1.70.70.1.1.1.1.1.1.20.cmml" xref="E6.m1.69.69a"><times id="E6.m1.70.70.1.1.1.1.1.1.20.1.cmml" xref="E6.m1.69.69a"></times><cn type="integer" id="E6.m1.51.51.51.10.10.10.cmml" xref="E6.m1.51.51.51.10.10.10">2</cn><apply id="E6.m1.70.70.1.1.1.1.1.1.20.3.cmml" xref="E6.m1.69.69a"><abs id="E6.m1.70.70.1.1.1.1.1.1.20.3.1.cmml" xref="E6.m1.69.69a"></abs><ci id="E6.m1.53.53.53.12.12.12.cmml" xref="E6.m1.53.53.53.12.12.12">𝐫</ci></apply></apply></apply></apply><apply id="E6.m1.71.71.2.2.2.2.2.2.cmml" xref="E6.m1.69.69a"><and id="E6.m1.71.71.2.2.2.2.2.2a.cmml" xref="E6.m1.69.69a"></and><apply id="E6.m1.71.71.2.2.2.2.2.2b.cmml" xref="E6.m1.69.69a"><eq id="E6.m1.59.59.59.18.18.18.cmml" xref="E6.m1.59.59.59.18.18.18"></eq><apply id="E6.m1.71.71.2.2.2.2.2.2.2.cmml" xref="E6.m1.69.69a"><times id="E6.m1.71.71.2.2.2.2.2.2.2.1.cmml" xref="E6.m1.69.69a"></times><ci id="E6.m1.56.56.56.15.15.15a.cmml" xref="E6.m1.56.56.56.15.15.15"><mtext id="E6.m1.56.56.56.15.15.15.cmml" xref="E6.m1.56.56.56.15.15.15">such that</mtext></ci><apply id="E6.m1.71.71.2.2.2.2.2.2.2.3.cmml" xref="E6.m1.69.69a"><csymbol cd="ambiguous" id="E6.m1.71.71.2.2.2.2.2.2.2.3.1.cmml" xref="E6.m1.69.69a">subscript</csymbol><ci id="E6.m1.57.57.57.16.16.16.cmml" xref="E6.m1.57.57.57.16.16.16">𝐭</ci><apply id="E6.m1.58.58.58.17.17.17.1.cmml" xref="E6.m1.58.58.58.17.17.17.1"><csymbol cd="ambiguous" id="E6.m1.58.58.58.17.17.17.1.1.cmml" xref="E6.m1.58.58.58.17.17.17.1">subscript</csymbol><ci id="E6.m1.58.58.58.17.17.17.1.2.cmml" xref="E6.m1.58.58.58.17.17.17.1.2">𝑖</ci><ci id="E6.m1.58.58.58.17.17.17.1.3.cmml" xref="E6.m1.58.58.58.17.17.17.1.3">𝑘</ci></apply></apply></apply><apply id="E6.m1.71.71.2.2.2.2.2.2.4.cmml" xref="E6.m1.69.69a"><times id="E6.m1.71.71.2.2.2.2.2.2.4.1.cmml" xref="E6.m1.69.69a"></times><apply id="E6.m1.71.71.2.2.2.2.2.2.4.2.cmml" xref="E6.m1.69.69a"><csymbol cd="ambiguous" id="E6.m1.71.71.2.2.2.2.2.2.4.2.1.cmml" xref="E6.m1.69.69a">subscript</csymbol><ci id="E6.m1.60.60.60.19.19.19.cmml" xref="E6.m1.60.60.60.19.19.19">𝐫</ci><apply id="E6.m1.61.61.61.20.20.20.1.cmml" xref="E6.m1.61.61.61.20.20.20.1"><csymbol cd="ambiguous" id="E6.m1.61.61.61.20.20.20.1.1.cmml" xref="E6.m1.61.61.61.20.20.20.1">subscript</csymbol><ci id="E6.m1.61.61.61.20.20.20.1.2.cmml" xref="E6.m1.61.61.61.20.20.20.1.2">𝑗</ci><ci id="E6.m1.61.61.61.20.20.20.1.3.cmml" xref="E6.m1.61.61.61.20.20.20.1.3">𝑘</ci></apply></apply><ci id="E6.m1.62.62.62.21.21.21a.cmml" xref="E6.m1.62.62.62.21.21.21"><mtext id="E6.m1.62.62.62.21.21.21.cmml" xref="E6.m1.62.62.62.21.21.21">for</mtext></ci><ci id="E6.m1.63.63.63.22.22.22.cmml" xref="E6.m1.63.63.63.22.22.22">𝑘</ci></apply></apply><apply id="E6.m1.71.71.2.2.2.2.2.2c.cmml" xref="E6.m1.69.69a"><eq id="E6.m1.64.64.64.23.23.23.cmml" xref="E6.m1.64.64.64.23.23.23"></eq><share href="#E6.m1.71.71.2.2.2.2.2.2.4.cmml" id="E6.m1.71.71.2.2.2.2.2.2d.cmml" xref="E6.m1.69.69a"></share><cn type="integer" id="E6.m1.65.65.65.24.24.24.cmml" xref="E6.m1.65.65.65.24.24.24">1</cn></apply></apply><apply id="E6.m1.72.72.3.3.3.3.3.3.cmml" xref="E6.m1.69.69a"><times id="E6.m1.72.72.3.3.3.3.3.3.1.cmml" xref="E6.m1.69.69a"></times><ci id="E6.m1.67.67.67.26.26.26.cmml" xref="E6.m1.67.67.67.26.26.26">…</ci><ci id="E6.m1.68.68.68.27.27.27.cmml" xref="E6.m1.68.68.68.27.27.27">𝑁</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="E6.m1.72c">\displaystyle\begin{split}\text{overlap}(\mathbf{t},\mathbf{r}):=\frac{1}{|\mathbf{r}|}\max\big{\{}N&amp;~{}\big{|}~{}\exists 1\leq i_{1}&lt;\dots&lt;i_{N}\leq|\mathbf{t}|~{}\text{and}~{}\exists 1\leq j_{1}&lt;\dots&lt;j_{N}\leq|\mathbf{r}|\\
&amp;~{}\text{and}~{}|i_{N}-i_{1}|\leq 2|\mathbf{r}|,~{}\text{such that}~{}\mathbf{t}_{i_{k}}=\mathbf{r}_{j_{k}}~{}\text{for}~{}k=1,\dots N\big{\}}\end{split}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(6)</span></td>
</tr></tbody>
</table>
<p id="S3.SS6.p4.5" class="ltx_p">The constraint <math id="S3.SS6.p4.4.m1.2" class="ltx_Math" alttext="|i_{N}-i_{1}|\leq 2|\mathbf{r}|" display="inline"><semantics id="S3.SS6.p4.4.m1.2a"><mrow id="S3.SS6.p4.4.m1.2.2" xref="S3.SS6.p4.4.m1.2.2.cmml"><mrow id="S3.SS6.p4.4.m1.2.2.1.1" xref="S3.SS6.p4.4.m1.2.2.1.2.cmml"><mo stretchy="false" id="S3.SS6.p4.4.m1.2.2.1.1.2" xref="S3.SS6.p4.4.m1.2.2.1.2.1.cmml">|</mo><mrow id="S3.SS6.p4.4.m1.2.2.1.1.1" xref="S3.SS6.p4.4.m1.2.2.1.1.1.cmml"><msub id="S3.SS6.p4.4.m1.2.2.1.1.1.2" xref="S3.SS6.p4.4.m1.2.2.1.1.1.2.cmml"><mi id="S3.SS6.p4.4.m1.2.2.1.1.1.2.2" xref="S3.SS6.p4.4.m1.2.2.1.1.1.2.2.cmml">i</mi><mi id="S3.SS6.p4.4.m1.2.2.1.1.1.2.3" xref="S3.SS6.p4.4.m1.2.2.1.1.1.2.3.cmml">N</mi></msub><mo id="S3.SS6.p4.4.m1.2.2.1.1.1.1" xref="S3.SS6.p4.4.m1.2.2.1.1.1.1.cmml">−</mo><msub id="S3.SS6.p4.4.m1.2.2.1.1.1.3" xref="S3.SS6.p4.4.m1.2.2.1.1.1.3.cmml"><mi id="S3.SS6.p4.4.m1.2.2.1.1.1.3.2" xref="S3.SS6.p4.4.m1.2.2.1.1.1.3.2.cmml">i</mi><mn id="S3.SS6.p4.4.m1.2.2.1.1.1.3.3" xref="S3.SS6.p4.4.m1.2.2.1.1.1.3.3.cmml">1</mn></msub></mrow><mo stretchy="false" id="S3.SS6.p4.4.m1.2.2.1.1.3" xref="S3.SS6.p4.4.m1.2.2.1.2.1.cmml">|</mo></mrow><mo id="S3.SS6.p4.4.m1.2.2.2" xref="S3.SS6.p4.4.m1.2.2.2.cmml">≤</mo><mrow id="S3.SS6.p4.4.m1.2.2.3" xref="S3.SS6.p4.4.m1.2.2.3.cmml"><mn id="S3.SS6.p4.4.m1.2.2.3.2" xref="S3.SS6.p4.4.m1.2.2.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S3.SS6.p4.4.m1.2.2.3.1" xref="S3.SS6.p4.4.m1.2.2.3.1.cmml">​</mo><mrow id="S3.SS6.p4.4.m1.2.2.3.3.2" xref="S3.SS6.p4.4.m1.2.2.3.3.1.cmml"><mo stretchy="false" id="S3.SS6.p4.4.m1.2.2.3.3.2.1" xref="S3.SS6.p4.4.m1.2.2.3.3.1.1.cmml">|</mo><mi id="S3.SS6.p4.4.m1.1.1" xref="S3.SS6.p4.4.m1.1.1.cmml">𝐫</mi><mo stretchy="false" id="S3.SS6.p4.4.m1.2.2.3.3.2.2" xref="S3.SS6.p4.4.m1.2.2.3.3.1.1.cmml">|</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS6.p4.4.m1.2b"><apply id="S3.SS6.p4.4.m1.2.2.cmml" xref="S3.SS6.p4.4.m1.2.2"><leq id="S3.SS6.p4.4.m1.2.2.2.cmml" xref="S3.SS6.p4.4.m1.2.2.2"></leq><apply id="S3.SS6.p4.4.m1.2.2.1.2.cmml" xref="S3.SS6.p4.4.m1.2.2.1.1"><abs id="S3.SS6.p4.4.m1.2.2.1.2.1.cmml" xref="S3.SS6.p4.4.m1.2.2.1.1.2"></abs><apply id="S3.SS6.p4.4.m1.2.2.1.1.1.cmml" xref="S3.SS6.p4.4.m1.2.2.1.1.1"><minus id="S3.SS6.p4.4.m1.2.2.1.1.1.1.cmml" xref="S3.SS6.p4.4.m1.2.2.1.1.1.1"></minus><apply id="S3.SS6.p4.4.m1.2.2.1.1.1.2.cmml" xref="S3.SS6.p4.4.m1.2.2.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS6.p4.4.m1.2.2.1.1.1.2.1.cmml" xref="S3.SS6.p4.4.m1.2.2.1.1.1.2">subscript</csymbol><ci id="S3.SS6.p4.4.m1.2.2.1.1.1.2.2.cmml" xref="S3.SS6.p4.4.m1.2.2.1.1.1.2.2">𝑖</ci><ci id="S3.SS6.p4.4.m1.2.2.1.1.1.2.3.cmml" xref="S3.SS6.p4.4.m1.2.2.1.1.1.2.3">𝑁</ci></apply><apply id="S3.SS6.p4.4.m1.2.2.1.1.1.3.cmml" xref="S3.SS6.p4.4.m1.2.2.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS6.p4.4.m1.2.2.1.1.1.3.1.cmml" xref="S3.SS6.p4.4.m1.2.2.1.1.1.3">subscript</csymbol><ci id="S3.SS6.p4.4.m1.2.2.1.1.1.3.2.cmml" xref="S3.SS6.p4.4.m1.2.2.1.1.1.3.2">𝑖</ci><cn type="integer" id="S3.SS6.p4.4.m1.2.2.1.1.1.3.3.cmml" xref="S3.SS6.p4.4.m1.2.2.1.1.1.3.3">1</cn></apply></apply></apply><apply id="S3.SS6.p4.4.m1.2.2.3.cmml" xref="S3.SS6.p4.4.m1.2.2.3"><times id="S3.SS6.p4.4.m1.2.2.3.1.cmml" xref="S3.SS6.p4.4.m1.2.2.3.1"></times><cn type="integer" id="S3.SS6.p4.4.m1.2.2.3.2.cmml" xref="S3.SS6.p4.4.m1.2.2.3.2">2</cn><apply id="S3.SS6.p4.4.m1.2.2.3.3.1.cmml" xref="S3.SS6.p4.4.m1.2.2.3.3.2"><abs id="S3.SS6.p4.4.m1.2.2.3.3.1.1.cmml" xref="S3.SS6.p4.4.m1.2.2.3.3.2.1"></abs><ci id="S3.SS6.p4.4.m1.1.1.cmml" xref="S3.SS6.p4.4.m1.1.1">𝐫</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.p4.4.m1.2c">|i_{N}-i_{1}|\leq 2|\mathbf{r}|</annotation></semantics></math> ensures that the overlap is not over-estimated as <math id="S3.SS6.p4.5.m2.1" class="ltx_Math" alttext="|\mathbf{t}|\to\infty" display="inline"><semantics id="S3.SS6.p4.5.m2.1a"><mrow id="S3.SS6.p4.5.m2.1.2" xref="S3.SS6.p4.5.m2.1.2.cmml"><mrow id="S3.SS6.p4.5.m2.1.2.2.2" xref="S3.SS6.p4.5.m2.1.2.2.1.cmml"><mo stretchy="false" id="S3.SS6.p4.5.m2.1.2.2.2.1" xref="S3.SS6.p4.5.m2.1.2.2.1.1.cmml">|</mo><mi id="S3.SS6.p4.5.m2.1.1" xref="S3.SS6.p4.5.m2.1.1.cmml">𝐭</mi><mo stretchy="false" id="S3.SS6.p4.5.m2.1.2.2.2.2" xref="S3.SS6.p4.5.m2.1.2.2.1.1.cmml">|</mo></mrow><mo stretchy="false" id="S3.SS6.p4.5.m2.1.2.1" xref="S3.SS6.p4.5.m2.1.2.1.cmml">→</mo><mi mathvariant="normal" id="S3.SS6.p4.5.m2.1.2.3" xref="S3.SS6.p4.5.m2.1.2.3.cmml">∞</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS6.p4.5.m2.1b"><apply id="S3.SS6.p4.5.m2.1.2.cmml" xref="S3.SS6.p4.5.m2.1.2"><ci id="S3.SS6.p4.5.m2.1.2.1.cmml" xref="S3.SS6.p4.5.m2.1.2.1">→</ci><apply id="S3.SS6.p4.5.m2.1.2.2.1.cmml" xref="S3.SS6.p4.5.m2.1.2.2.2"><abs id="S3.SS6.p4.5.m2.1.2.2.1.1.cmml" xref="S3.SS6.p4.5.m2.1.2.2.2.1"></abs><ci id="S3.SS6.p4.5.m2.1.1.cmml" xref="S3.SS6.p4.5.m2.1.1">𝐭</ci></apply><infinity id="S3.SS6.p4.5.m2.1.2.3.cmml" xref="S3.SS6.p4.5.m2.1.2.3"></infinity></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.p4.5.m2.1c">|\mathbf{t}|\to\infty</annotation></semantics></math>.</p>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_section" style="font-size:144%;color:#00868B;">4 <span id="S4.1.1" class="ltx_text" style="color:#808080;">| </span>Pretraining Data</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.1" class="ltx_p">This section describes the procedures for collecting and filtering our pretraining dataset and knowledge base (or reference dataset).
</p>
</div>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsection" style="font-size:120%;color:#00868B;">4.1 <span id="S4.SS1.1.1" class="ltx_text" style="color:#808080;">| </span>Data Collection</h3>

<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.1" class="ltx_p">The pretrain data is gathered from English and Chinese text datasets, mostly publicly available collections of webpages and books.
We also include code, SFT data (supervised finetuning), and synthetic data.</p>
</div>
<div id="S4.SS1.p2" class="ltx_para">
<p id="S4.SS1.p2.1" class="ltx_p">Specially, the English data mainly consists of RedPajamaV2 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib23" title="" class="ltx_ref">23</a>]</cite>, SlimPajama <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib104" title="" class="ltx_ref">104</a>]</cite> and the Piles <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib43" title="" class="ltx_ref">43</a>]</cite>, in total 200TB prior to filtering.
The Chinese data mainly comes from Wanjuan <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib51" title="" class="ltx_ref">51</a>]</cite>, Wenshu <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib2" title="" class="ltx_ref">2</a>]</cite>, and MNBVC <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib81" title="" class="ltx_ref">81</a>]</cite>, in total 500TB prior to filtering.
The code data mainly comes from Github, and we take the subset with the highest repository stars.
The SFT data is included since these samples generally have higher quality than the webpages.
We use the same data as in SFT training (Section <a href="#S6.SS1" title="6.1 | Supervised Finetuning ‣ 6 | Fine-tuning and Alignment ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6.1</span></a>), except that these samples are treated as ordinary texts during pretraining, i.e. all tokens participate in the loss computation, not just the answer tokens.</p>
</div>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsection" style="font-size:120%;color:#00868B;">4.2 <span id="S4.SS2.1.1" class="ltx_text" style="color:#808080;">| </span>Filtering</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.1" class="ltx_p">The raw data is filtered with three steps: deduplication, rule-based filtering, and model-based filtering.</p>
</div>
<div id="S4.SS2.p2" class="ltx_para">
<p id="S4.SS2.p2.1" class="ltx_p">First, deduplication is performed with MinHash for most of the datasets.
One exception is RedPajamaV2, which already comes with deduplication labels.</p>
</div>
<div id="S4.SS2.p3" class="ltx_para">
<p id="S4.SS2.p3.1" class="ltx_p">Second, we devise heuristic, rule-based filters analogous to the ones from
<cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib76" title="" class="ltx_ref">76</a>, <a href="#bib.bib92" title="" class="ltx_ref">92</a>, <a href="#bib.bib25" title="" class="ltx_ref">25</a>]</cite>.
The purpose is to eliminate texts that are ostensibly unsuitable for training, such as ones that only contain webpage source codes, random numbers, or incomprehensible shards.
Our filters remove documents with less than 50 words, documents whose mean word lengths exceed 10 characters,
documents with 70% of context being non-alphabetic characters,
documents whose fractions of unique words are disproportionately high,
documents whose entropy of unigrams is excessively low, and so on.</p>
</div>
<div id="S4.SS2.p4" class="ltx_para">
<p id="S4.SS2.p4.2" class="ltx_p">Finally, we select the subset of data with highest “quality”, a score produced by a finetuned BERT model.
Specially, we sample ten thousand documents and grade them by the XinYu-70B model <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib65" title="" class="ltx_ref">65</a>, <a href="#bib.bib68" title="" class="ltx_ref">68</a>]</cite> with prompt-guided generation.
The prompt asks the model to determine whether the input text is informative and produce a score between <math id="S4.SS2.p4.1.m1.1" class="ltx_Math" alttext="0" display="inline"><semantics id="S4.SS2.p4.1.m1.1a"><mn id="S4.SS2.p4.1.m1.1.1" xref="S4.SS2.p4.1.m1.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="S4.SS2.p4.1.m1.1b"><cn type="integer" id="S4.SS2.p4.1.m1.1.1.cmml" xref="S4.SS2.p4.1.m1.1.1">0</cn></annotation-xml></semantics></math> and <math id="S4.SS2.p4.2.m2.1" class="ltx_Math" alttext="5" display="inline"><semantics id="S4.SS2.p4.2.m2.1a"><mn id="S4.SS2.p4.2.m2.1.1" xref="S4.SS2.p4.2.m2.1.1.cmml">5</mn><annotation-xml encoding="MathML-Content" id="S4.SS2.p4.2.m2.1b"><cn type="integer" id="S4.SS2.p4.2.m2.1.1.cmml" xref="S4.SS2.p4.2.m2.1.1">5</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p4.2.m2.1c">5</annotation></semantics></math>.
Then, these scores are used to finetune the Tiny-BERT model <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib57" title="" class="ltx_ref">57</a>]</cite>, which has only 14M parameters.
The hyperparameters of this finetuning are optimized with respect to a held-out validation set.
After that, we use this lightweight BERT to grade the entire dataset.</p>
</div>
<div id="Thmremark6" class="ltx_theorem ltx_theorem_remark">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmremark6.1.1.1" class="ltx_text ltx_font_bold">Remark 6</span></span><span id="Thmremark6.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmremark6.p1" class="ltx_para">
<p id="Thmremark6.p1.2" class="ltx_p">Recall from Section <a href="#S3.SS5" title="3.5 | Training Designs ‣ 3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.5</span></a> that the pretraining data of Memory<sup id="Thmremark6.p1.2.1" class="ltx_sup">3</sup> should emphasize abstract knowledges and minimize specific knowledges.
The purpose is to not only obtain a lightweight LLM with an ideal distribution of knowledges in accordance with the memory hierarchy (Figure <a href="#F8" title="Figure 8 ‣ 2.3 | Memory ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a>), but also prevent the specific knowledges from hindering the learning process of the model.
The focus of our prompt on “informativeness” might be contradictory to this goal, since the selected texts that are rich in information content may contain too many specific knowledges.
For future versions of Memory<sup id="Thmremark6.p1.2.2" class="ltx_sup">3</sup>, we will switch to a model-based filter favoring texts that exhibit more reasoning and less specifics.</p>
</div>
</div>
<figure id="F12" class="ltx_figure"><img src="/html/2407.01178/assets/Figures/Data/pretrain_data_distribution.png" id="F12.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="359" height="262" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="F12.2.1.1" class="ltx_text" style="font-size:90%;">Figure 12</span>: </span><span id="F12.3.2" class="ltx_text" style="font-size:90%;">Composition of our pretraining dataset.
</span></figcaption>
</figure>
<div id="S4.SS2.p5" class="ltx_para">
<p id="S4.SS2.p5.1" class="ltx_p">The filtered dataset consists of around four trillion tokens, and its composition is illustrated in Figure <a href="#F12" title="Figure 12 ‣ 4.2 | Filtering ‣ 4 | Pretraining Data ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">12</span></a>.</p>
</div>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsection" style="font-size:120%;color:#00868B;">4.3 <span id="S4.SS3.1.1" class="ltx_text" style="color:#808080;">| </span>Tokenizer</h3>

<div id="S4.SS3.p1" class="ltx_para">
<p id="S4.SS3.p1.1" class="ltx_p">Similar to our dataset, our tokenizer mainly consists of Chinese and English tokens.
The English vocabulary comes from the 32000 tokens of the LLaMA2 tokenizer.
We include roughly the same amount of Chinese tokens produced from byte-pair encoding (BPE).
The BPE is trained on a 20GB Chinese corpus that consists of Chinese news and e-books.
After deduplication, the final vocabulary has 60299 tokens.</p>
</div>
</section>
<section id="S4.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsection" style="font-size:120%;color:#00868B;">4.4 <span id="S4.SS4.1.1" class="ltx_text" style="color:#808080;">| </span>Knowledge Base</h3>

<div id="S4.SS4.p1" class="ltx_para">
<p id="S4.SS4.p1.1" class="ltx_p">The knowledge base (or reference dataset) is used during training and inference as the source of explicit memories, as depicted in Figure <a href="#F1" title="Figure 1 ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.
It consists of reference texts that are split into token sequences with length <math id="S4.SS4.p1.1.m1.1" class="ltx_Math" alttext="\leq 128" display="inline"><semantics id="S4.SS4.p1.1.m1.1a"><mrow id="S4.SS4.p1.1.m1.1.1" xref="S4.SS4.p1.1.m1.1.1.cmml"><mi id="S4.SS4.p1.1.m1.1.1.2" xref="S4.SS4.p1.1.m1.1.1.2.cmml"></mi><mo id="S4.SS4.p1.1.m1.1.1.1" xref="S4.SS4.p1.1.m1.1.1.1.cmml">≤</mo><mn id="S4.SS4.p1.1.m1.1.1.3" xref="S4.SS4.p1.1.m1.1.1.3.cmml">128</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS4.p1.1.m1.1b"><apply id="S4.SS4.p1.1.m1.1.1.cmml" xref="S4.SS4.p1.1.m1.1.1"><leq id="S4.SS4.p1.1.m1.1.1.1.cmml" xref="S4.SS4.p1.1.m1.1.1.1"></leq><csymbol cd="latexml" id="S4.SS4.p1.1.m1.1.1.2.cmml" xref="S4.SS4.p1.1.m1.1.1.2">absent</csymbol><cn type="integer" id="S4.SS4.p1.1.m1.1.1.3.cmml" xref="S4.SS4.p1.1.m1.1.1.3">128</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p1.1.m1.1c">\leq 128</annotation></semantics></math>, as described in Section <a href="#S3.SS1" title="3.1 | Inference Process ‣ 3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.1</span></a>.</p>
</div>
<div id="S4.SS4.p2" class="ltx_para">
<p id="S4.SS4.p2.1" class="ltx_p">Heuristically, a larger knowledge base is always better, as long as it does not contain misinformation,
so it is not surprising that the reference dataset of Retro contains its entire pretrain dataset <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib16" title="" class="ltx_ref">16</a>]</cite>.
Nevertheless, the storage of explicit memories is more costly than plain texts despite our sparsification (Section <a href="#S3.SS3" title="3.3 | Memory Sparsification and Storage ‣ 3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3</span></a>), and thus to save storage space, we select a small subset of our pretrain dataset as the knowledge base.</p>
</div>
<div id="S4.SS4.p3" class="ltx_para">
<p id="S4.SS4.p3.1" class="ltx_p">With a focus on high quality data, we include for references the English Wikipedia, WikiHow, the Chinese baike dataset, the subset of English and Chinese books whose titles appear academic, Chinese news, synthetic data and high quality codes.
These texts are tokenized and split into chunks of 128 tokens, resulting in <math id="S4.SS4.p3.1.m1.1" class="ltx_Math" alttext="1.1\times 10^{8}" display="inline"><semantics id="S4.SS4.p3.1.m1.1a"><mrow id="S4.SS4.p3.1.m1.1.1" xref="S4.SS4.p3.1.m1.1.1.cmml"><mn id="S4.SS4.p3.1.m1.1.1.2" xref="S4.SS4.p3.1.m1.1.1.2.cmml">1.1</mn><mo lspace="0.222em" rspace="0.222em" id="S4.SS4.p3.1.m1.1.1.1" xref="S4.SS4.p3.1.m1.1.1.1.cmml">×</mo><msup id="S4.SS4.p3.1.m1.1.1.3" xref="S4.SS4.p3.1.m1.1.1.3.cmml"><mn id="S4.SS4.p3.1.m1.1.1.3.2" xref="S4.SS4.p3.1.m1.1.1.3.2.cmml">10</mn><mn id="S4.SS4.p3.1.m1.1.1.3.3" xref="S4.SS4.p3.1.m1.1.1.3.3.cmml">8</mn></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.SS4.p3.1.m1.1b"><apply id="S4.SS4.p3.1.m1.1.1.cmml" xref="S4.SS4.p3.1.m1.1.1"><times id="S4.SS4.p3.1.m1.1.1.1.cmml" xref="S4.SS4.p3.1.m1.1.1.1"></times><cn type="float" id="S4.SS4.p3.1.m1.1.1.2.cmml" xref="S4.SS4.p3.1.m1.1.1.2">1.1</cn><apply id="S4.SS4.p3.1.m1.1.1.3.cmml" xref="S4.SS4.p3.1.m1.1.1.3"><csymbol cd="ambiguous" id="S4.SS4.p3.1.m1.1.1.3.1.cmml" xref="S4.SS4.p3.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="S4.SS4.p3.1.m1.1.1.3.2.cmml" xref="S4.SS4.p3.1.m1.1.1.3.2">10</cn><cn type="integer" id="S4.SS4.p3.1.m1.1.1.3.3.cmml" xref="S4.SS4.p3.1.m1.1.1.3.3">8</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p3.1.m1.1c">1.1\times 10^{8}</annotation></semantics></math> references in total.</p>
</div>
<div id="S4.SS4.p4" class="ltx_para">
<p id="S4.SS4.p4.3" class="ltx_p">One may be curious whether our knowledge base may contain some of the evaluation questions, rendering our evaluation results (Section <a href="#S7.SS1" title="7.1 | General Abilities ‣ 7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7.1</span></a>) less credible.
To prevent such leakage, we include in our evaluation code a filtering step, such that for each evaluation question, if a retrieved reference has an overlap with the question that exceeds a threshold, then it is discarded.
This deduplication is analogous to the one used when preparing for continual train (Section <a href="#S3.SS6" title="3.6 | Two-stage Pretrain ‣ 3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.6</span></a>),
with the overlap measured by (<a href="#E6" title="Equation 6 ‣ 3.6 | Two-stage Pretrain ‣ 3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>).
The threshold <math id="S4.SS4.p4.1.m1.1" class="ltx_Math" alttext="2/3" display="inline"><semantics id="S4.SS4.p4.1.m1.1a"><mrow id="S4.SS4.p4.1.m1.1.1" xref="S4.SS4.p4.1.m1.1.1.cmml"><mn id="S4.SS4.p4.1.m1.1.1.2" xref="S4.SS4.p4.1.m1.1.1.2.cmml">2</mn><mo id="S4.SS4.p4.1.m1.1.1.1" xref="S4.SS4.p4.1.m1.1.1.1.cmml">/</mo><mn id="S4.SS4.p4.1.m1.1.1.3" xref="S4.SS4.p4.1.m1.1.1.3.cmml">3</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS4.p4.1.m1.1b"><apply id="S4.SS4.p4.1.m1.1.1.cmml" xref="S4.SS4.p4.1.m1.1.1"><divide id="S4.SS4.p4.1.m1.1.1.1.cmml" xref="S4.SS4.p4.1.m1.1.1.1"></divide><cn type="integer" id="S4.SS4.p4.1.m1.1.1.2.cmml" xref="S4.SS4.p4.1.m1.1.1.2">2</cn><cn type="integer" id="S4.SS4.p4.1.m1.1.1.3.cmml" xref="S4.SS4.p4.1.m1.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p4.1.m1.1c">2/3</annotation></semantics></math> is chosen since we observe that typically a reference that contains a question would have an overlap <math id="S4.SS4.p4.2.m2.1" class="ltx_Math" alttext="\geq 80\%" display="inline"><semantics id="S4.SS4.p4.2.m2.1a"><mrow id="S4.SS4.p4.2.m2.1.1" xref="S4.SS4.p4.2.m2.1.1.cmml"><mi id="S4.SS4.p4.2.m2.1.1.2" xref="S4.SS4.p4.2.m2.1.1.2.cmml"></mi><mo id="S4.SS4.p4.2.m2.1.1.1" xref="S4.SS4.p4.2.m2.1.1.1.cmml">≥</mo><mrow id="S4.SS4.p4.2.m2.1.1.3" xref="S4.SS4.p4.2.m2.1.1.3.cmml"><mn id="S4.SS4.p4.2.m2.1.1.3.2" xref="S4.SS4.p4.2.m2.1.1.3.2.cmml">80</mn><mo id="S4.SS4.p4.2.m2.1.1.3.1" xref="S4.SS4.p4.2.m2.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS4.p4.2.m2.1b"><apply id="S4.SS4.p4.2.m2.1.1.cmml" xref="S4.SS4.p4.2.m2.1.1"><geq id="S4.SS4.p4.2.m2.1.1.1.cmml" xref="S4.SS4.p4.2.m2.1.1.1"></geq><csymbol cd="latexml" id="S4.SS4.p4.2.m2.1.1.2.cmml" xref="S4.SS4.p4.2.m2.1.1.2">absent</csymbol><apply id="S4.SS4.p4.2.m2.1.1.3.cmml" xref="S4.SS4.p4.2.m2.1.1.3"><csymbol cd="latexml" id="S4.SS4.p4.2.m2.1.1.3.1.cmml" xref="S4.SS4.p4.2.m2.1.1.3.1">percent</csymbol><cn type="integer" id="S4.SS4.p4.2.m2.1.1.3.2.cmml" xref="S4.SS4.p4.2.m2.1.1.3.2">80</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p4.2.m2.1c">\geq 80\%</annotation></semantics></math>, while a relevant but distinct reference would have an overlap <math id="S4.SS4.p4.3.m3.1" class="ltx_Math" alttext="\leq 40\%" display="inline"><semantics id="S4.SS4.p4.3.m3.1a"><mrow id="S4.SS4.p4.3.m3.1.1" xref="S4.SS4.p4.3.m3.1.1.cmml"><mi id="S4.SS4.p4.3.m3.1.1.2" xref="S4.SS4.p4.3.m3.1.1.2.cmml"></mi><mo id="S4.SS4.p4.3.m3.1.1.1" xref="S4.SS4.p4.3.m3.1.1.1.cmml">≤</mo><mrow id="S4.SS4.p4.3.m3.1.1.3" xref="S4.SS4.p4.3.m3.1.1.3.cmml"><mn id="S4.SS4.p4.3.m3.1.1.3.2" xref="S4.SS4.p4.3.m3.1.1.3.2.cmml">40</mn><mo id="S4.SS4.p4.3.m3.1.1.3.1" xref="S4.SS4.p4.3.m3.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS4.p4.3.m3.1b"><apply id="S4.SS4.p4.3.m3.1.1.cmml" xref="S4.SS4.p4.3.m3.1.1"><leq id="S4.SS4.p4.3.m3.1.1.1.cmml" xref="S4.SS4.p4.3.m3.1.1.1"></leq><csymbol cd="latexml" id="S4.SS4.p4.3.m3.1.1.2.cmml" xref="S4.SS4.p4.3.m3.1.1.2">absent</csymbol><apply id="S4.SS4.p4.3.m3.1.1.3.cmml" xref="S4.SS4.p4.3.m3.1.1.3"><csymbol cd="latexml" id="S4.SS4.p4.3.m3.1.1.3.1.cmml" xref="S4.SS4.p4.3.m3.1.1.3.1">percent</csymbol><cn type="integer" id="S4.SS4.p4.3.m3.1.1.3.2.cmml" xref="S4.SS4.p4.3.m3.1.1.3.2">40</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p4.3.m3.1c">\leq 40\%</annotation></semantics></math>.</p>
</div>
<div id="Thmremark7" class="ltx_theorem ltx_theorem_remark">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmremark7.1.1.1" class="ltx_text ltx_font_bold">Remark 7</span></span><span id="Thmremark7.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmremark7.p1" class="ltx_para">
<p id="Thmremark7.p1.1" class="ltx_p">Currently, the compilation of the knowledge base is based on human preference.
For future versions of Memory<sup id="Thmremark7.p1.1.1" class="ltx_sup">3</sup>, we plan to take a model-oriented approach and measure the fitness of a candidate reference by its actual utility,
e.g. the expected decrease in the validation loss of the LLM conditioned on this reference being retrieved by a random validation sample.</p>
</div>
</div>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_section" style="font-size:144%;color:#00868B;">5 <span id="S5.1.1" class="ltx_text" style="color:#808080;">| </span>Pretrain</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">This section describes the details of the pretraining process.
The two-stage pretrain and memory-augmented data follow the designs introduced in Section <a href="#S3.SS6" title="3.6 | Two-stage Pretrain ‣ 3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.6</span></a>.
As an interpretation, the Memory<sup id="S5.p1.1.1" class="ltx_sup">3</sup> model during the warmup stage develops its reading comprehension, which is necessary during the continual train stage for initiating memory formation.</p>
</div>
<section id="S5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsection" style="font-size:120%;color:#00868B;">5.1 <span id="S5.SS1.1.1" class="ltx_text" style="color:#808080;">| </span>Set-up</h3>

<div id="S5.SS1.p1" class="ltx_para">
<p id="S5.SS1.p1.1" class="ltx_p">Training is conducted with the Megatron-DeepSpeed package <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib3" title="" class="ltx_ref">3</a>]</cite> and uses mixed-precision training with bfloat16 model parameters, bfloat16 activations, and float32 AdamW states.
The batch size is around 4 million training tokens with sequence length 2048, not including the reference tokens.
The weight decay is the common choice of 0.1.</p>
</div>
<div id="S5.SS1.p2" class="ltx_para">
<p id="S5.SS1.p2.1" class="ltx_p">We adopt the “warmup-stable-decay” learning rate schedule of MiniCPM <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib54" title="" class="ltx_ref">54</a>]</cite>, which is reportedly better than the usual cosine schedule in term of training loss reduction.
The learning rate linearly increases to the maximum value, then stays there for the majority of training steps, and finally in the last 10% steps decays rapidly to near zero.
Our short-term experiments confirm the better performance of this schedule.
Nevertheless, frequent loss spikes and loss divergences are encountered during the official pretraining, so we have to deviate from this schedule and manually decrease the learning rate to stabilize training.</p>
</div>
<div id="S5.SS1.p3" class="ltx_para">
<p id="S5.SS1.p3.1" class="ltx_p">Originally, it is planned that both the warmup and continual train stages go through the entire 4T token pretrain dataset (Section <a href="#S4" title="4 | Pretraining Data ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>).
Due to the irremediable loss divergences, both stages have to be terminated earlier.</p>
</div>
</section>
<section id="S5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsection" style="font-size:120%;color:#00868B;">5.2 <span id="S5.SS2.1.1" class="ltx_text" style="color:#808080;">| </span>Warmup Stage</h3>

<figure id="F13" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="F13.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.01178/assets/Figures/key_figure/warmup_loss.png" id="F13.sf1.g1" class="ltx_graphics ltx_img_landscape" width="284" height="161" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="F13.sf1.2.1.1" class="ltx_text" style="font-size:90%;">(a)</span> </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="F13.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.01178/assets/Figures/key_figure/warmup_lr.png" id="F13.sf2.g1" class="ltx_graphics ltx_img_landscape" width="284" height="158" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="F13.sf2.2.1.1" class="ltx_text" style="font-size:90%;">(b)</span> </span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="F13.2.1.1" class="ltx_text" style="font-size:90%;">Figure 13</span>: </span><span id="F13.3.2" class="ltx_text" style="font-size:90%;">The warmup stage without explicit memory.
Left: Training loss.
Right: Learning rate schedule.</span></figcaption>
</figure>
<div id="S5.SS2.p1" class="ltx_para">
<p id="S5.SS2.p1.1" class="ltx_p">The training loss and learning rate schedule are plotted in Figure <a href="#F13" title="Figure 13 ‣ 5.2 | Warmup Stage ‣ 5 | Pretrain ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">13</span></a>.
Whenever severe loss divergence occurs, we restart from the last checkpoint before the divergence with a smaller learning rate, and thus the divergences are not shown in the figure.
Eventually, the training terminates at around 3.1T tokens, when reducing the learning rate can no longer avoid loss divergence.</p>
</div>
</section>
<section id="S5.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsection" style="font-size:120%;color:#00868B;">5.3 <span id="S5.SS3.1.1" class="ltx_text" style="color:#808080;">| </span>Continual Train Stage</h3>

<figure id="F14" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="F14.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.01178/assets/Figures/key_figure/2B_continual_train_loss.png" id="F14.sf1.g1" class="ltx_graphics ltx_img_landscape" width="269" height="152" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="F14.sf1.2.1.1" class="ltx_text" style="font-size:90%;">(a)</span> </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="F14.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.01178/assets/Figures/key_figure/2B_continual_lr_schedule.png" id="F14.sf2.g1" class="ltx_graphics ltx_img_landscape" width="269" height="152" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="F14.sf2.2.1.1" class="ltx_text" style="font-size:90%;">(b)</span> </span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="F14.2.1.1" class="ltx_text" style="font-size:90%;">Figure 14</span>: </span><span id="F14.3.2" class="ltx_text" style="font-size:90%;">The continual train stage with explicit memory.
Left: Training loss.
Right: Learning rate schedule.</span></figcaption>
</figure>
<div id="S5.SS3.p1" class="ltx_para">
<p id="S5.SS3.p1.1" class="ltx_p">The explicit memories enter into the Memory<sup id="S5.SS3.p1.1.1" class="ltx_sup">3</sup> model at this stage.
The training steps are slower since the model needs to encode the references retrieved for the pretrain data to explicit memories in real time, and each step takes a bit more than twice the time of a warmup step.
The training loss and learning rate schedule are plotted in Figure <a href="#F14" title="Figure 14 ‣ 5.3 | Continual Train Stage ‣ 5 | Pretrain ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">14</span></a>.</p>
</div>
<div id="S5.SS3.p2" class="ltx_para">
<p id="S5.SS3.p2.1" class="ltx_p">The loss divergence soon becomes irremediable at around 120B training tokens, much shorter than the planned 4T tokens, and training has to stop there.
One possible cause is that the continual train is initialized from the latest warmup checkpoint, which is located immediately before the break down of the warmup stage, and thus is already at the brink of divergence.
The smaller learning rate of continual train delays the onset of divergence but not for long.</p>
</div>
</section>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_section" style="font-size:144%;color:#00868B;">6 <span id="S6.1.1" class="ltx_text" style="color:#808080;">| </span>Fine-tuning and Alignment</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">This section describes our model finetuning, specifically supervised finetuning (SFT) and direct preference optimization (DPO).</p>
</div>
<section id="S6.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsection" style="font-size:120%;color:#00868B;">6.1 <span id="S6.SS1.1.1" class="ltx_text" style="color:#808080;">| </span>Supervised Finetuning</h3>

<div id="S6.SS1.p1" class="ltx_para">
<p id="S6.SS1.p1.1" class="ltx_p">Analogous to the StableLM model <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib14" title="" class="ltx_ref">14</a>]</cite>,
our Memory<sup id="S6.SS1.p1.1.1" class="ltx_sup">3</sup> model is finetuned on a diverse collection of SFT datasets.
We use the following datasets, which are publicly accessible on the Hugging Face Hub:
UltraChat <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib34" title="" class="ltx_ref">34</a>]</cite>, WizardLM <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib125" title="" class="ltx_ref">125</a>]</cite>, SlimOrca <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib67" title="" class="ltx_ref">67</a>]</cite>, ShareGPT <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib114" title="" class="ltx_ref">114</a>]</cite>, Capybara <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib31" title="" class="ltx_ref">31</a>]</cite>, Deita <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib73" title="" class="ltx_ref">73</a>]</cite>, and MetaMathQA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib130" title="" class="ltx_ref">130</a>]</cite>.
We also include synthetic data with emphasis on multi-round chat, mathematics, commonsense and knowledge.
Each training sample consists of one or more rounds of question and answer pairs.
We remove any sample with more than eight rounds.
The final composition is listed in Table <a href="#T15" title="Table 15 ‣ 6.1 | Supervised Finetuning ‣ 6 | Fine-tuning and Alignment ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">15</span></a>.</p>
</div>
<figure id="T15" class="ltx_table">
<table id="T15.2" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="T15.2.1" class="ltx_tr">
<td id="T15.2.1.1" class="ltx_td ltx_align_left ltx_border_tt" style="padding-bottom:3.44444pt;"><span id="T15.2.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Dataset</span></td>
<td id="T15.2.1.2" class="ltx_td ltx_align_center ltx_border_tt" style="padding-bottom:3.44444pt;"><span id="T15.2.1.2.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Source</span></td>
<td id="T15.2.1.3" class="ltx_td ltx_align_right ltx_border_tt" style="padding-bottom:3.44444pt;"><span id="T15.2.1.3.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Number of Samples</span></td>
</tr>
<tr id="T15.2.2" class="ltx_tr">
<td id="T15.2.2.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-bottom:0.68887pt;"><span id="T15.2.2.1.1" class="ltx_text" style="font-size:80%;">UltraChat</span></td>
<td id="T15.2.2.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-bottom:0.68887pt;"><span id="T15.2.2.2.1" class="ltx_text" style="font-size:80%;">HuggingFaceH4/ultrachat_200k</span></td>
<td id="T15.2.2.3" class="ltx_td ltx_align_right ltx_border_t" style="padding-bottom:0.68887pt;"><span id="T15.2.2.3.1" class="ltx_text" style="font-size:80%;">194409</span></td>
</tr>
<tr id="T15.2.3" class="ltx_tr">
<td id="T15.2.3.1" class="ltx_td ltx_align_left" style="padding-bottom:0.68887pt;"><span id="T15.2.3.1.1" class="ltx_text" style="font-size:80%;">WizardLM</span></td>
<td id="T15.2.3.2" class="ltx_td ltx_align_center" style="padding-bottom:0.68887pt;"><span id="T15.2.3.2.1" class="ltx_text" style="font-size:80%;">WizardLM/WizardLM_evol_instruct_V2_196k</span></td>
<td id="T15.2.3.3" class="ltx_td ltx_align_right" style="padding-bottom:0.68887pt;"><span id="T15.2.3.3.1" class="ltx_text" style="font-size:80%;">80662</span></td>
</tr>
<tr id="T15.2.4" class="ltx_tr">
<td id="T15.2.4.1" class="ltx_td ltx_align_left" style="padding-bottom:0.68887pt;"><span id="T15.2.4.1.1" class="ltx_text" style="font-size:80%;">SlimOrca</span></td>
<td id="T15.2.4.2" class="ltx_td ltx_align_center" style="padding-bottom:0.68887pt;"><span id="T15.2.4.2.1" class="ltx_text" style="font-size:80%;">Open-Orca/SlimOrca-Dedup</span></td>
<td id="T15.2.4.3" class="ltx_td ltx_align_right" style="padding-bottom:0.68887pt;"><span id="T15.2.4.3.1" class="ltx_text" style="font-size:80%;">143789</span></td>
</tr>
<tr id="T15.2.5" class="ltx_tr">
<td id="T15.2.5.1" class="ltx_td ltx_align_left" style="padding-bottom:0.68887pt;"><span id="T15.2.5.1.1" class="ltx_text" style="font-size:80%;">ShareGPT</span></td>
<td id="T15.2.5.2" class="ltx_td ltx_align_center" style="padding-bottom:0.68887pt;"><span id="T15.2.5.2.1" class="ltx_text" style="font-size:80%;">openchat/openchat_sharegpt4_dataset</span></td>
<td id="T15.2.5.3" class="ltx_td ltx_align_right" style="padding-bottom:0.68887pt;"><span id="T15.2.5.3.1" class="ltx_text" style="font-size:80%;">3509</span></td>
</tr>
<tr id="T15.2.6" class="ltx_tr">
<td id="T15.2.6.1" class="ltx_td ltx_align_left" style="padding-bottom:0.68887pt;"><span id="T15.2.6.1.1" class="ltx_text" style="font-size:80%;">Capybara</span></td>
<td id="T15.2.6.2" class="ltx_td ltx_align_center" style="padding-bottom:0.68887pt;"><span id="T15.2.6.2.1" class="ltx_text" style="font-size:80%;">LDJnr/Capybara</span></td>
<td id="T15.2.6.3" class="ltx_td ltx_align_right" style="padding-bottom:0.68887pt;"><span id="T15.2.6.3.1" class="ltx_text" style="font-size:80%;">7291</span></td>
</tr>
<tr id="T15.2.7" class="ltx_tr">
<td id="T15.2.7.1" class="ltx_td ltx_align_left" style="padding-bottom:0.68887pt;"><span id="T15.2.7.1.1" class="ltx_text" style="font-size:80%;">Deita</span></td>
<td id="T15.2.7.2" class="ltx_td ltx_align_center" style="padding-bottom:0.68887pt;"><span id="T15.2.7.2.1" class="ltx_text" style="font-size:80%;">hkust-nlp/deita-10k-v0</span></td>
<td id="T15.2.7.3" class="ltx_td ltx_align_right" style="padding-bottom:0.68887pt;"><span id="T15.2.7.3.1" class="ltx_text" style="font-size:80%;">2860</span></td>
</tr>
<tr id="T15.2.8" class="ltx_tr">
<td id="T15.2.8.1" class="ltx_td ltx_align_left" style="padding-bottom:0.68887pt;"><span id="T15.2.8.1.1" class="ltx_text" style="font-size:80%;">MetaMathQA</span></td>
<td id="T15.2.8.2" class="ltx_td ltx_align_center" style="padding-bottom:0.68887pt;"><span id="T15.2.8.2.1" class="ltx_text" style="font-size:80%;">meta-math/MetaMathQA</span></td>
<td id="T15.2.8.3" class="ltx_td ltx_align_right" style="padding-bottom:0.68887pt;"><span id="T15.2.8.3.1" class="ltx_text" style="font-size:80%;">394418</span></td>
</tr>
<tr id="T15.2.9" class="ltx_tr">
<td id="T15.2.9.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-bottom:0.68887pt;"><span id="T15.2.9.1.1" class="ltx_text" style="font-size:80%;">Multi-round Chat</span></td>
<td id="T15.2.9.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-bottom:0.68887pt;"><span id="T15.2.9.2.1" class="ltx_text" style="font-size:80%;">synthetic</span></td>
<td id="T15.2.9.3" class="ltx_td ltx_align_right ltx_border_t" style="padding-bottom:0.68887pt;"><span id="T15.2.9.3.1" class="ltx_text" style="font-size:80%;">20000</span></td>
</tr>
<tr id="T15.2.10" class="ltx_tr">
<td id="T15.2.10.1" class="ltx_td ltx_align_left" style="padding-bottom:0.68887pt;"><span id="T15.2.10.1.1" class="ltx_text" style="font-size:80%;">Mathematics</span></td>
<td id="T15.2.10.2" class="ltx_td ltx_align_center" style="padding-bottom:0.68887pt;"><span id="T15.2.10.2.1" class="ltx_text" style="font-size:80%;">synthetic</span></td>
<td id="T15.2.10.3" class="ltx_td ltx_align_right" style="padding-bottom:0.68887pt;"><span id="T15.2.10.3.1" class="ltx_text" style="font-size:80%;">20000</span></td>
</tr>
<tr id="T15.2.11" class="ltx_tr">
<td id="T15.2.11.1" class="ltx_td ltx_align_left" style="padding-bottom:0.68887pt;"><span id="T15.2.11.1.1" class="ltx_text" style="font-size:80%;">Commonsense</span></td>
<td id="T15.2.11.2" class="ltx_td ltx_align_center" style="padding-bottom:0.68887pt;"><span id="T15.2.11.2.1" class="ltx_text" style="font-size:80%;">synthetic</span></td>
<td id="T15.2.11.3" class="ltx_td ltx_align_right" style="padding-bottom:0.68887pt;"><span id="T15.2.11.3.1" class="ltx_text" style="font-size:80%;">150000</span></td>
</tr>
<tr id="T15.2.12" class="ltx_tr">
<td id="T15.2.12.1" class="ltx_td ltx_align_left ltx_border_b" style="padding-bottom:0.68887pt;"><span id="T15.2.12.1.1" class="ltx_text" style="font-size:80%;">Knowledge</span></td>
<td id="T15.2.12.2" class="ltx_td ltx_align_center ltx_border_b" style="padding-bottom:0.68887pt;"><span id="T15.2.12.2.1" class="ltx_text" style="font-size:80%;">synthetic</span></td>
<td id="T15.2.12.3" class="ltx_td ltx_align_right ltx_border_b" style="padding-bottom:0.68887pt;"><span id="T15.2.12.3.1" class="ltx_text" style="font-size:80%;">270000</span></td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering" style="font-size:80%;"><span class="ltx_tag ltx_tag_table"><span id="T15.5.1.1" class="ltx_text" style="font-size:113%;">Table 15</span>: </span><span id="T15.6.2" class="ltx_text" style="font-size:113%;">Composition of SFT dataset.</span></figcaption>
</figure>
<div id="S6.SS1.p2" class="ltx_para">
<p id="S6.SS1.p2.2" class="ltx_p">The training process uses the cosine learning rate schedule with a max
learning rate of <math id="S6.SS1.p2.1.m1.1" class="ltx_Math" alttext="5\times 10^{-5}" display="inline"><semantics id="S6.SS1.p2.1.m1.1a"><mrow id="S6.SS1.p2.1.m1.1.1" xref="S6.SS1.p2.1.m1.1.1.cmml"><mn id="S6.SS1.p2.1.m1.1.1.2" xref="S6.SS1.p2.1.m1.1.1.2.cmml">5</mn><mo lspace="0.222em" rspace="0.222em" id="S6.SS1.p2.1.m1.1.1.1" xref="S6.SS1.p2.1.m1.1.1.1.cmml">×</mo><msup id="S6.SS1.p2.1.m1.1.1.3" xref="S6.SS1.p2.1.m1.1.1.3.cmml"><mn id="S6.SS1.p2.1.m1.1.1.3.2" xref="S6.SS1.p2.1.m1.1.1.3.2.cmml">10</mn><mrow id="S6.SS1.p2.1.m1.1.1.3.3" xref="S6.SS1.p2.1.m1.1.1.3.3.cmml"><mo id="S6.SS1.p2.1.m1.1.1.3.3a" xref="S6.SS1.p2.1.m1.1.1.3.3.cmml">−</mo><mn id="S6.SS1.p2.1.m1.1.1.3.3.2" xref="S6.SS1.p2.1.m1.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p2.1.m1.1b"><apply id="S6.SS1.p2.1.m1.1.1.cmml" xref="S6.SS1.p2.1.m1.1.1"><times id="S6.SS1.p2.1.m1.1.1.1.cmml" xref="S6.SS1.p2.1.m1.1.1.1"></times><cn type="integer" id="S6.SS1.p2.1.m1.1.1.2.cmml" xref="S6.SS1.p2.1.m1.1.1.2">5</cn><apply id="S6.SS1.p2.1.m1.1.1.3.cmml" xref="S6.SS1.p2.1.m1.1.1.3"><csymbol cd="ambiguous" id="S6.SS1.p2.1.m1.1.1.3.1.cmml" xref="S6.SS1.p2.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="S6.SS1.p2.1.m1.1.1.3.2.cmml" xref="S6.SS1.p2.1.m1.1.1.3.2">10</cn><apply id="S6.SS1.p2.1.m1.1.1.3.3.cmml" xref="S6.SS1.p2.1.m1.1.1.3.3"><minus id="S6.SS1.p2.1.m1.1.1.3.3.1.cmml" xref="S6.SS1.p2.1.m1.1.1.3.3"></minus><cn type="integer" id="S6.SS1.p2.1.m1.1.1.3.3.2.cmml" xref="S6.SS1.p2.1.m1.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p2.1.m1.1c">5\times 10^{-5}</annotation></semantics></math> and a <math id="S6.SS1.p2.2.m2.1" class="ltx_Math" alttext="10\%" display="inline"><semantics id="S6.SS1.p2.2.m2.1a"><mrow id="S6.SS1.p2.2.m2.1.1" xref="S6.SS1.p2.2.m2.1.1.cmml"><mn id="S6.SS1.p2.2.m2.1.1.2" xref="S6.SS1.p2.2.m2.1.1.2.cmml">10</mn><mo id="S6.SS1.p2.2.m2.1.1.1" xref="S6.SS1.p2.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p2.2.m2.1b"><apply id="S6.SS1.p2.2.m2.1.1.cmml" xref="S6.SS1.p2.2.m2.1.1"><csymbol cd="latexml" id="S6.SS1.p2.2.m2.1.1.1.cmml" xref="S6.SS1.p2.2.m2.1.1.1">percent</csymbol><cn type="integer" id="S6.SS1.p2.2.m2.1.1.2.cmml" xref="S6.SS1.p2.2.m2.1.1.2">10</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p2.2.m2.1c">10\%</annotation></semantics></math> linear warmup phase.
The weight decay is 0.1, batch size is 512, and max sequence length is 2048 tokens.
Finetuning is performed for 3 epochs.</p>
</div>
</section>
<section id="S6.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsection" style="font-size:120%;color:#00868B;">6.2 <span id="S6.SS2.1.1" class="ltx_text" style="color:#808080;">| </span>Direct Preference Optimization</h3>

<div id="S6.SS2.p1" class="ltx_para">
<p id="S6.SS2.p1.4" class="ltx_p">The Memory<sup id="S6.SS2.p1.4.1" class="ltx_sup">3</sup> model is further finetuned by DPO <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib93" title="" class="ltx_ref">93</a>]</cite>, to align with human preference and improve its conversation skills.
The DPO dataset consists of general conversations (UltraFeedback Binarized <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib111" title="" class="ltx_ref">111</a>]</cite>), math questions (Distilabel Math <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib10" title="" class="ltx_ref">10</a>]</cite>) and codes questions (Synth Code <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib36" title="" class="ltx_ref">36</a>]</cite>).
The training uses the cosine learning rate schedule with max lr <math id="S6.SS2.p1.2.m2.1" class="ltx_Math" alttext="4\times 10^{-6}" display="inline"><semantics id="S6.SS2.p1.2.m2.1a"><mrow id="S6.SS2.p1.2.m2.1.1" xref="S6.SS2.p1.2.m2.1.1.cmml"><mn id="S6.SS2.p1.2.m2.1.1.2" xref="S6.SS2.p1.2.m2.1.1.2.cmml">4</mn><mo lspace="0.222em" rspace="0.222em" id="S6.SS2.p1.2.m2.1.1.1" xref="S6.SS2.p1.2.m2.1.1.1.cmml">×</mo><msup id="S6.SS2.p1.2.m2.1.1.3" xref="S6.SS2.p1.2.m2.1.1.3.cmml"><mn id="S6.SS2.p1.2.m2.1.1.3.2" xref="S6.SS2.p1.2.m2.1.1.3.2.cmml">10</mn><mrow id="S6.SS2.p1.2.m2.1.1.3.3" xref="S6.SS2.p1.2.m2.1.1.3.3.cmml"><mo id="S6.SS2.p1.2.m2.1.1.3.3a" xref="S6.SS2.p1.2.m2.1.1.3.3.cmml">−</mo><mn id="S6.SS2.p1.2.m2.1.1.3.3.2" xref="S6.SS2.p1.2.m2.1.1.3.3.2.cmml">6</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.2.m2.1b"><apply id="S6.SS2.p1.2.m2.1.1.cmml" xref="S6.SS2.p1.2.m2.1.1"><times id="S6.SS2.p1.2.m2.1.1.1.cmml" xref="S6.SS2.p1.2.m2.1.1.1"></times><cn type="integer" id="S6.SS2.p1.2.m2.1.1.2.cmml" xref="S6.SS2.p1.2.m2.1.1.2">4</cn><apply id="S6.SS2.p1.2.m2.1.1.3.cmml" xref="S6.SS2.p1.2.m2.1.1.3"><csymbol cd="ambiguous" id="S6.SS2.p1.2.m2.1.1.3.1.cmml" xref="S6.SS2.p1.2.m2.1.1.3">superscript</csymbol><cn type="integer" id="S6.SS2.p1.2.m2.1.1.3.2.cmml" xref="S6.SS2.p1.2.m2.1.1.3.2">10</cn><apply id="S6.SS2.p1.2.m2.1.1.3.3.cmml" xref="S6.SS2.p1.2.m2.1.1.3.3"><minus id="S6.SS2.p1.2.m2.1.1.3.3.1.cmml" xref="S6.SS2.p1.2.m2.1.1.3.3"></minus><cn type="integer" id="S6.SS2.p1.2.m2.1.1.3.3.2.cmml" xref="S6.SS2.p1.2.m2.1.1.3.3.2">6</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.2.m2.1c">4\times 10^{-6}</annotation></semantics></math>.
The inverse temperature <math id="S6.SS2.p1.3.m3.1" class="ltx_Math" alttext="\beta" display="inline"><semantics id="S6.SS2.p1.3.m3.1a"><mi id="S6.SS2.p1.3.m3.1.1" xref="S6.SS2.p1.3.m3.1.1.cmml">β</mi><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.3.m3.1b"><ci id="S6.SS2.p1.3.m3.1.1.cmml" xref="S6.SS2.p1.3.m3.1.1">𝛽</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.3.m3.1c">\beta</annotation></semantics></math> of the DPO loss is set to <math id="S6.SS2.p1.4.m4.1" class="ltx_Math" alttext="0.01" display="inline"><semantics id="S6.SS2.p1.4.m4.1a"><mn id="S6.SS2.p1.4.m4.1.1" xref="S6.SS2.p1.4.m4.1.1.cmml">0.01</mn><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.4.m4.1b"><cn type="float" id="S6.SS2.p1.4.m4.1.1.cmml" xref="S6.SS2.p1.4.m4.1.1">0.01</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.4.m4.1c">0.01</annotation></semantics></math>.
The improvement from DPO is displayed in Section <a href="#S7.SS2" title="7.2 | Conversation Skill ‣ 7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7.2</span></a>.</p>
</div>
</section>
</section>
<section id="S7" class="ltx_section">
<h2 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_section" style="font-size:144%;color:#00868B;">7 <span id="S7.1.1" class="ltx_text" style="color:#808080;">| </span>Evaluation</h2>

<div id="S7.p1" class="ltx_para">
<p id="S7.p1.1" class="ltx_p">We evaluate the general abilities (benchmark tasks), conversation skills, professional abilities (law and medicine), and facutality &amp; hallucination of the Memory<sup id="S7.p1.1.1" class="ltx_sup">3</sup> model.
We also measure its decoding speed.
Our model is compared with SOTA LLMs of similar and larger sizes, as well as RAG models.</p>
</div>
<section id="S7.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsection" style="font-size:120%;color:#00868B;">7.1 <span id="S7.SS1.1.1" class="ltx_text" style="color:#808080;">| </span>General Abilities</h3>

<div id="S7.SS1.p1" class="ltx_para">
<p id="S7.SS1.p1.1" class="ltx_p">To evaluate the general abilities of Memory<sup id="S7.SS1.p1.1.1" class="ltx_sup">3</sup>, we adopt all tasks from the Huggingface leaderboard and also include two Chinese tasks.
Most of the results are displayed in Table <a href="#T16" title="Table 16 ‣ 7.1 | General Abilities ‣ 7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">16</span></a>, while TruthfulQA is listed in Table <a href="#T19" title="Table 19 ‣ 7.3 | Hallucination and Factuality ‣ 7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">19</span></a>.
All results are obtained in bfloat16 format, using the lm-evaluation-harness package <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib44" title="" class="ltx_ref">44</a>]</cite> and the configuration of HuggingFace Open LLM leaderboard <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib13" title="" class="ltx_ref">13</a>]</cite>, i.e. the number of few-shot examples and grading methods.</p>
</div>
<div id="S7.SS1.p2" class="ltx_para">
<p id="S7.SS1.p2.1" class="ltx_p">As described in Section <a href="#S4.SS4" title="4.4 | Knowledge Base ‣ 4 | Pretraining Data ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.4</span></a>, to prevent cheating, a filtering step is included in the retrieval process so that the model cannot copy from references that resemble the evaluation questions.</p>
</div>
<figure id="T16" class="ltx_table">
<table id="T16.1" class="ltx_tabular ltx_align_middle">
<tr id="T16.1.2" class="ltx_tr">
<td id="T16.1.2.1" class="ltx_td ltx_border_tt"></td>
<td id="T16.1.2.2" class="ltx_td ltx_border_tt"></td>
<td id="T16.1.2.3" class="ltx_td ltx_border_tt"></td>
<td id="T16.1.2.4" class="ltx_td ltx_align_center ltx_border_tt" colspan="5"><span id="T16.1.2.4.1" class="ltx_text" style="font-size:80%;">English</span></td>
<td id="T16.1.2.5" class="ltx_td ltx_align_center ltx_border_tt" colspan="2"><span id="T16.1.2.5.1" class="ltx_text" style="font-size:80%;">Chinese</span></td>
</tr>
<tr id="T16.1.3" class="ltx_tr" style="background-color:#FFFFFF;">
<td id="T16.1.3.1" class="ltx_td ltx_align_center"><span id="T16.1.3.1.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">LLM</span></td>
<td id="T16.1.3.2" class="ltx_td ltx_align_center"><span id="T16.1.3.2.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">Size</span></td>
<td id="T16.1.3.3" class="ltx_td ltx_align_center"><span id="T16.1.3.3.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">Avg.</span></td>
<td id="T16.1.3.4" class="ltx_td ltx_align_center ltx_border_t"><span id="T16.1.3.4.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">ARC-C</span></td>
<td id="T16.1.3.5" class="ltx_td ltx_align_center ltx_border_t"><span id="T16.1.3.5.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">HellaSwag</span></td>
<td id="T16.1.3.6" class="ltx_td ltx_align_center ltx_border_t"><span id="T16.1.3.6.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">MMLU</span></td>
<td id="T16.1.3.7" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="T16.1.3.7.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFFFFF;">
<span id="T16.1.3.7.1.1" class="ltx_p" style="width:32.5pt;"><span id="T16.1.3.7.1.1.1" class="ltx_text" style="font-size:80%;">Winogrande</span></span>
</span>
</td>
<td id="T16.1.3.8" class="ltx_td ltx_align_center ltx_border_t"><span id="T16.1.3.8.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">GSM8k</span></td>
<td id="T16.1.3.9" class="ltx_td ltx_align_center ltx_border_t"><span id="T16.1.3.9.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">CEVAL</span></td>
<td id="T16.1.3.10" class="ltx_td ltx_align_center ltx_border_t"><span id="T16.1.3.10.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">CMMLU</span></td>
</tr>
<tr id="T16.1.4" class="ltx_tr" style="background-color:#E6F3F3;">
<td id="T16.1.4.1" class="ltx_td ltx_align_center ltx_border_t"><span id="T16.1.4.1.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">Falcon-40B</span></td>
<td id="T16.1.4.2" class="ltx_td ltx_align_center ltx_border_t"><span id="T16.1.4.2.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">41B</span></td>
<td id="T16.1.4.3" class="ltx_td ltx_align_center ltx_border_t"><span id="T16.1.4.3.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">55.75</span></td>
<td id="T16.1.4.4" class="ltx_td ltx_align_center ltx_border_t"><span id="T16.1.4.4.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">61.86</span></td>
<td id="T16.1.4.5" class="ltx_td ltx_align_center ltx_border_t"><span id="T16.1.4.5.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">85.28</span></td>
<td id="T16.1.4.6" class="ltx_td ltx_align_center ltx_border_t"><span id="T16.1.4.6.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">56.89</span></td>
<td id="T16.1.4.7" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="T16.1.4.7.1" class="ltx_inline-block ltx_align_top" style="background-color:#E6F3F3;">
<span id="T16.1.4.7.1.1" class="ltx_p" style="width:32.5pt;"><span id="T16.1.4.7.1.1.1" class="ltx_text" style="font-size:80%;">81.29</span></span>
</span>
</td>
<td id="T16.1.4.8" class="ltx_td ltx_align_center ltx_border_t"><span id="T16.1.4.8.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">21.46</span></td>
<td id="T16.1.4.9" class="ltx_td ltx_align_center ltx_border_t"><span id="T16.1.4.9.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">41.38</span></td>
<td id="T16.1.4.10" class="ltx_td ltx_align_center ltx_border_t"><span id="T16.1.4.10.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">42.07</span></td>
</tr>
<tr id="T16.1.5" class="ltx_tr" style="background-color:#FFFFFF;">
<td id="T16.1.5.1" class="ltx_td ltx_align_center"><span id="T16.1.5.1.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">Llama2-7B-Chat</span></td>
<td id="T16.1.5.2" class="ltx_td ltx_align_center"><span id="T16.1.5.2.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">6.5B</span></td>
<td id="T16.1.5.3" class="ltx_td ltx_align_center"><span id="T16.1.5.3.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">46.87</span></td>
<td id="T16.1.5.4" class="ltx_td ltx_align_center"><span id="T16.1.5.4.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">52.90</span></td>
<td id="T16.1.5.5" class="ltx_td ltx_align_center"><span id="T16.1.5.5.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">78.55</span></td>
<td id="T16.1.5.6" class="ltx_td ltx_align_center"><span id="T16.1.5.6.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">48.32</span></td>
<td id="T16.1.5.7" class="ltx_td ltx_align_justify ltx_align_top">
<span id="T16.1.5.7.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFFFFF;">
<span id="T16.1.5.7.1.1" class="ltx_p" style="width:32.5pt;"><span id="T16.1.5.7.1.1.1" class="ltx_text" style="font-size:80%;">71.74</span></span>
</span>
</td>
<td id="T16.1.5.8" class="ltx_td ltx_align_center"><span id="T16.1.5.8.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">7.35</span></td>
<td id="T16.1.5.9" class="ltx_td ltx_align_center"><span id="T16.1.5.9.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">34.84</span></td>
<td id="T16.1.5.10" class="ltx_td ltx_align_center"><span id="T16.1.5.10.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">34.40</span></td>
</tr>
<tr id="T16.1.6" class="ltx_tr" style="background-color:#E6F3F3;">
<td id="T16.1.6.1" class="ltx_td ltx_align_center"><span id="T16.1.6.1.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">Llama2-13B-Chat</span></td>
<td id="T16.1.6.2" class="ltx_td ltx_align_center"><span id="T16.1.6.2.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">13B</span></td>
<td id="T16.1.6.3" class="ltx_td ltx_align_center"><span id="T16.1.6.3.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">51.78</span></td>
<td id="T16.1.6.4" class="ltx_td ltx_align_center"><span id="T16.1.6.4.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">59.04</span></td>
<td id="T16.1.6.5" class="ltx_td ltx_align_center"><span id="T16.1.6.5.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">81.94</span></td>
<td id="T16.1.6.6" class="ltx_td ltx_align_center"><span id="T16.1.6.6.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">54.64</span></td>
<td id="T16.1.6.7" class="ltx_td ltx_align_justify ltx_align_top">
<span id="T16.1.6.7.1" class="ltx_inline-block ltx_align_top" style="background-color:#E6F3F3;">
<span id="T16.1.6.7.1.1" class="ltx_p" style="width:32.5pt;"><span id="T16.1.6.7.1.1.1" class="ltx_text" style="font-size:80%;">74.51</span></span>
</span>
</td>
<td id="T16.1.6.8" class="ltx_td ltx_align_center"><span id="T16.1.6.8.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">15.24</span></td>
<td id="T16.1.6.9" class="ltx_td ltx_align_center"><span id="T16.1.6.9.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">38.63</span></td>
<td id="T16.1.6.10" class="ltx_td ltx_align_center"><span id="T16.1.6.10.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">38.43</span></td>
</tr>
<tr id="T16.1.7" class="ltx_tr" style="background-color:#FFFFFF;">
<td id="T16.1.7.1" class="ltx_td ltx_align_center"><span id="T16.1.7.1.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">Llama3-8B-it</span></td>
<td id="T16.1.7.2" class="ltx_td ltx_align_center"><span id="T16.1.7.2.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">7.0B</span></td>
<td id="T16.1.7.3" class="ltx_td ltx_align_center"><span id="T16.1.7.3.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">65.77</span></td>
<td id="T16.1.7.4" class="ltx_td ltx_align_center"><span id="T16.1.7.4.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">62.03</span></td>
<td id="T16.1.7.5" class="ltx_td ltx_align_center"><span id="T16.1.7.5.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">78.89</span></td>
<td id="T16.1.7.6" class="ltx_td ltx_align_center"><span id="T16.1.7.6.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">65.69</span></td>
<td id="T16.1.7.7" class="ltx_td ltx_align_justify ltx_align_top">
<span id="T16.1.7.7.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFFFFF;">
<span id="T16.1.7.7.1.1" class="ltx_p" style="width:32.5pt;"><span id="T16.1.7.7.1.1.1" class="ltx_text" style="font-size:80%;">75.77</span></span>
</span>
</td>
<td id="T16.1.7.8" class="ltx_td ltx_align_center"><span id="T16.1.7.8.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">75.82</span></td>
<td id="T16.1.7.9" class="ltx_td ltx_align_center"><span id="T16.1.7.9.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">50.52</span></td>
<td id="T16.1.7.10" class="ltx_td ltx_align_center"><span id="T16.1.7.10.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">51.70</span></td>
</tr>
<tr id="T16.1.8" class="ltx_tr" style="background-color:#E6F3F3;">
<td id="T16.1.8.1" class="ltx_td ltx_align_center"><span id="T16.1.8.1.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">Vicuna-13B-v1.5</span></td>
<td id="T16.1.8.2" class="ltx_td ltx_align_center"><span id="T16.1.8.2.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">13B</span></td>
<td id="T16.1.8.3" class="ltx_td ltx_align_center"><span id="T16.1.8.3.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">52.02</span></td>
<td id="T16.1.8.4" class="ltx_td ltx_align_center"><span id="T16.1.8.4.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">57.08</span></td>
<td id="T16.1.8.5" class="ltx_td ltx_align_center"><span id="T16.1.8.5.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">81.24</span></td>
<td id="T16.1.8.6" class="ltx_td ltx_align_center"><span id="T16.1.8.6.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">56.67</span></td>
<td id="T16.1.8.7" class="ltx_td ltx_align_justify ltx_align_top">
<span id="T16.1.8.7.1" class="ltx_inline-block ltx_align_top" style="background-color:#E6F3F3;">
<span id="T16.1.8.7.1.1" class="ltx_p" style="width:32.5pt;"><span id="T16.1.8.7.1.1.1" class="ltx_text" style="font-size:80%;">74.66</span></span>
</span>
</td>
<td id="T16.1.8.8" class="ltx_td ltx_align_center"><span id="T16.1.8.8.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">11.30</span></td>
<td id="T16.1.8.9" class="ltx_td ltx_align_center"><span id="T16.1.8.9.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">41.68</span></td>
<td id="T16.1.8.10" class="ltx_td ltx_align_center"><span id="T16.1.8.10.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">41.53</span></td>
</tr>
<tr id="T16.1.9" class="ltx_tr" style="background-color:#FFFFFF;">
<td id="T16.1.9.1" class="ltx_td ltx_align_center"><span id="T16.1.9.1.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">Mistral-7B-v0.1</span></td>
<td id="T16.1.9.2" class="ltx_td ltx_align_center"><span id="T16.1.9.2.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">7.0B</span></td>
<td id="T16.1.9.3" class="ltx_td ltx_align_center"><span id="T16.1.9.3.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">59.15</span></td>
<td id="T16.1.9.4" class="ltx_td ltx_align_center"><span id="T16.1.9.4.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">59.98</span></td>
<td id="T16.1.9.5" class="ltx_td ltx_align_center"><span id="T16.1.9.5.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">83.31</span></td>
<td id="T16.1.9.6" class="ltx_td ltx_align_center"><span id="T16.1.9.6.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">64.16</span></td>
<td id="T16.1.9.7" class="ltx_td ltx_align_justify ltx_align_top">
<span id="T16.1.9.7.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFFFFF;">
<span id="T16.1.9.7.1.1" class="ltx_p" style="width:32.5pt;"><span id="T16.1.9.7.1.1.1" class="ltx_text" style="font-size:80%;">78.37</span></span>
</span>
</td>
<td id="T16.1.9.8" class="ltx_td ltx_align_center"><span id="T16.1.9.8.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">37.83</span></td>
<td id="T16.1.9.9" class="ltx_td ltx_align_center"><span id="T16.1.9.9.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">45.91</span></td>
<td id="T16.1.9.10" class="ltx_td ltx_align_center"><span id="T16.1.9.10.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">44.49</span></td>
</tr>
<tr id="T16.1.10" class="ltx_tr" style="background-color:#E6F3F3;">
<td id="T16.1.10.1" class="ltx_td ltx_align_center"><span id="T16.1.10.1.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">Gemma-2B-it</span></td>
<td id="T16.1.10.2" class="ltx_td ltx_align_center"><span id="T16.1.10.2.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">2.0B</span></td>
<td id="T16.1.10.3" class="ltx_td ltx_align_center"><span id="T16.1.10.3.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">36.64</span></td>
<td id="T16.1.10.4" class="ltx_td ltx_align_center"><span id="T16.1.10.4.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">38.02</span></td>
<td id="T16.1.10.5" class="ltx_td ltx_align_center"><span id="T16.1.10.5.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">40.36</span></td>
<td id="T16.1.10.6" class="ltx_td ltx_align_center"><span id="T16.1.10.6.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">55.74</span></td>
<td id="T16.1.10.7" class="ltx_td ltx_align_justify ltx_align_top">
<span id="T16.1.10.7.1" class="ltx_inline-block ltx_align_top" style="background-color:#E6F3F3;">
<span id="T16.1.10.7.1.1" class="ltx_p" style="width:32.5pt;"><span id="T16.1.10.7.1.1.1" class="ltx_text" style="font-size:80%;">35.29</span></span>
</span>
</td>
<td id="T16.1.10.8" class="ltx_td ltx_align_center"><span id="T16.1.10.8.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">55.88</span></td>
<td id="T16.1.10.9" class="ltx_td ltx_align_center"><span id="T16.1.10.9.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">8.26</span></td>
<td id="T16.1.10.10" class="ltx_td ltx_align_center"><span id="T16.1.10.10.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">29.94</span></td>
</tr>
<tr id="T16.1.11" class="ltx_tr" style="background-color:#FFFFFF;">
<td id="T16.1.11.1" class="ltx_td ltx_align_center"><span id="T16.1.11.1.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">Gemma-7B-it</span></td>
<td id="T16.1.11.2" class="ltx_td ltx_align_center"><span id="T16.1.11.2.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">7.8B</span></td>
<td id="T16.1.11.3" class="ltx_td ltx_align_center"><span id="T16.1.11.3.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">47.23</span></td>
<td id="T16.1.11.4" class="ltx_td ltx_align_center"><span id="T16.1.11.4.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">51.45</span></td>
<td id="T16.1.11.5" class="ltx_td ltx_align_center"><span id="T16.1.11.5.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">71.96</span></td>
<td id="T16.1.11.6" class="ltx_td ltx_align_center"><span id="T16.1.11.6.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">53.52</span></td>
<td id="T16.1.11.7" class="ltx_td ltx_align_justify ltx_align_top">
<span id="T16.1.11.7.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFFFFF;">
<span id="T16.1.11.7.1.1" class="ltx_p" style="width:32.5pt;"><span id="T16.1.11.7.1.1.1" class="ltx_text" style="font-size:80%;">67.96</span></span>
</span>
</td>
<td id="T16.1.11.8" class="ltx_td ltx_align_center"><span id="T16.1.11.8.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">32.22</span></td>
<td id="T16.1.11.9" class="ltx_td ltx_align_center"><span id="T16.1.11.9.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">27.93</span></td>
<td id="T16.1.11.10" class="ltx_td ltx_align_center"><span id="T16.1.11.10.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">25.70</span></td>
</tr>
<tr id="T16.1.12" class="ltx_tr" style="background-color:#E6F3F3;">
<td id="T16.1.12.1" class="ltx_td ltx_align_center"><span id="T16.1.12.1.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">MiniCPM-2B-SFT</span></td>
<td id="T16.1.12.2" class="ltx_td ltx_align_center"><span id="T16.1.12.2.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">2.4B</span></td>
<td id="T16.1.12.3" class="ltx_td ltx_align_center"><span id="T16.1.12.3.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">54.37</span></td>
<td id="T16.1.12.4" class="ltx_td ltx_align_center"><span id="T16.1.12.4.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">47.53</span></td>
<td id="T16.1.12.5" class="ltx_td ltx_align_center"><span id="T16.1.12.5.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">71.95</span></td>
<td id="T16.1.12.6" class="ltx_td ltx_align_center"><span id="T16.1.12.6.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">51.32</span></td>
<td id="T16.1.12.7" class="ltx_td ltx_align_justify ltx_align_top">
<span id="T16.1.12.7.1" class="ltx_inline-block ltx_align_top" style="background-color:#E6F3F3;">
<span id="T16.1.12.7.1.1" class="ltx_p" style="width:32.5pt;"><span id="T16.1.12.7.1.1.1" class="ltx_text" style="font-size:80%;">67.72</span></span>
</span>
</td>
<td id="T16.1.12.8" class="ltx_td ltx_align_center"><span id="T16.1.12.8.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">45.26</span></td>
<td id="T16.1.12.9" class="ltx_td ltx_align_center"><span id="T16.1.12.9.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">48.07</span></td>
<td id="T16.1.12.10" class="ltx_td ltx_align_center"><span id="T16.1.12.10.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">48.76</span></td>
</tr>
<tr id="T16.1.13" class="ltx_tr" style="background-color:#FFFFFF;">
<td id="T16.1.13.1" class="ltx_td ltx_align_center"><span id="T16.1.13.1.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">Phi-2</span></td>
<td id="T16.1.13.2" class="ltx_td ltx_align_center"><span id="T16.1.13.2.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">2.5B</span></td>
<td id="T16.1.13.3" class="ltx_td ltx_align_center"><span id="T16.1.13.3.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">55.70</span></td>
<td id="T16.1.13.4" class="ltx_td ltx_align_center"><span id="T16.1.13.4.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">61.09</span></td>
<td id="T16.1.13.5" class="ltx_td ltx_align_center"><span id="T16.1.13.5.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">75.11</span></td>
<td id="T16.1.13.6" class="ltx_td ltx_align_center"><span id="T16.1.13.6.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">58.11</span></td>
<td id="T16.1.13.7" class="ltx_td ltx_align_justify ltx_align_top">
<span id="T16.1.13.7.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFFFFF;">
<span id="T16.1.13.7.1.1" class="ltx_p" style="width:32.5pt;"><span id="T16.1.13.7.1.1.1" class="ltx_text" style="font-size:80%;">74.35</span></span>
</span>
</td>
<td id="T16.1.13.8" class="ltx_td ltx_align_center"><span id="T16.1.13.8.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">54.81</span></td>
<td id="T16.1.13.9" class="ltx_td ltx_align_center"><span id="T16.1.13.9.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">34.40</span></td>
<td id="T16.1.13.10" class="ltx_td ltx_align_center"><span id="T16.1.13.10.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">32.04</span></td>
</tr>
<tr id="T16.1.14" class="ltx_tr" style="background-color:#E6F3F3;">
<td id="T16.1.14.1" class="ltx_td ltx_align_center"><span id="T16.1.14.1.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">ChatGLM3-6B</span></td>
<td id="T16.1.14.2" class="ltx_td ltx_align_center"><span id="T16.1.14.2.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">5.7B</span></td>
<td id="T16.1.14.3" class="ltx_td ltx_align_center"><span id="T16.1.14.3.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">54.62</span></td>
<td id="T16.1.14.4" class="ltx_td ltx_align_center"><span id="T16.1.14.4.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">41.38</span></td>
<td id="T16.1.14.5" class="ltx_td ltx_align_center"><span id="T16.1.14.5.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">66.98</span></td>
<td id="T16.1.14.6" class="ltx_td ltx_align_center"><span id="T16.1.14.6.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">50.54</span></td>
<td id="T16.1.14.7" class="ltx_td ltx_align_justify ltx_align_top">
<span id="T16.1.14.7.1" class="ltx_inline-block ltx_align_top" style="background-color:#E6F3F3;">
<span id="T16.1.14.7.1.1" class="ltx_p" style="width:32.5pt;"><span id="T16.1.14.7.1.1.1" class="ltx_text" style="font-size:80%;">64.25</span></span>
</span>
</td>
<td id="T16.1.14.8" class="ltx_td ltx_align_center"><span id="T16.1.14.8.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">51.25</span></td>
<td id="T16.1.14.9" class="ltx_td ltx_align_center"><span id="T16.1.14.9.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">54.01</span></td>
<td id="T16.1.14.10" class="ltx_td ltx_align_center"><span id="T16.1.14.10.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">53.91</span></td>
</tr>
<tr id="T16.1.15" class="ltx_tr" style="background-color:#FFFFFF;">
<td id="T16.1.15.1" class="ltx_td ltx_align_center"><span id="T16.1.15.1.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">Baichuan2-7B-Chat</span></td>
<td id="T16.1.15.2" class="ltx_td ltx_align_center"><span id="T16.1.15.2.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">6.5B</span></td>
<td id="T16.1.15.3" class="ltx_td ltx_align_center"><span id="T16.1.15.3.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">55.16</span></td>
<td id="T16.1.15.4" class="ltx_td ltx_align_center"><span id="T16.1.15.4.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">52.73</span></td>
<td id="T16.1.15.5" class="ltx_td ltx_align_center"><span id="T16.1.15.5.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">74.06</span></td>
<td id="T16.1.15.6" class="ltx_td ltx_align_center"><span id="T16.1.15.6.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">52.77</span></td>
<td id="T16.1.15.7" class="ltx_td ltx_align_justify ltx_align_top">
<span id="T16.1.15.7.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFFFFF;">
<span id="T16.1.15.7.1.1" class="ltx_p" style="width:32.5pt;"><span id="T16.1.15.7.1.1.1" class="ltx_text" style="font-size:80%;">69.77</span></span>
</span>
</td>
<td id="T16.1.15.8" class="ltx_td ltx_align_center"><span id="T16.1.15.8.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">28.28</span></td>
<td id="T16.1.15.9" class="ltx_td ltx_align_center"><span id="T16.1.15.9.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">53.12</span></td>
<td id="T16.1.15.10" class="ltx_td ltx_align_center"><span id="T16.1.15.10.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">55.38</span></td>
</tr>
<tr id="T16.1.16" class="ltx_tr" style="background-color:#E6F3F3;">
<td id="T16.1.16.1" class="ltx_td ltx_align_center"><span id="T16.1.16.1.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">Qwen1.5-1.8B-Chat</span></td>
<td id="T16.1.16.2" class="ltx_td ltx_align_center"><span id="T16.1.16.2.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">1.2B</span></td>
<td id="T16.1.16.3" class="ltx_td ltx_align_center"><span id="T16.1.16.3.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">49.67</span></td>
<td id="T16.1.16.4" class="ltx_td ltx_align_center"><span id="T16.1.16.4.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">38.74</span></td>
<td id="T16.1.16.5" class="ltx_td ltx_align_center"><span id="T16.1.16.5.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">60.02</span></td>
<td id="T16.1.16.6" class="ltx_td ltx_align_center"><span id="T16.1.16.6.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">45.87</span></td>
<td id="T16.1.16.7" class="ltx_td ltx_align_justify ltx_align_top">
<span id="T16.1.16.7.1" class="ltx_inline-block ltx_align_top" style="background-color:#E6F3F3;">
<span id="T16.1.16.7.1.1" class="ltx_p" style="width:32.5pt;"><span id="T16.1.16.7.1.1.1" class="ltx_text" style="font-size:80%;">59.67</span></span>
</span>
</td>
<td id="T16.1.16.8" class="ltx_td ltx_align_center"><span id="T16.1.16.8.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">33.59</span></td>
<td id="T16.1.16.9" class="ltx_td ltx_align_center"><span id="T16.1.16.9.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">55.57</span></td>
<td id="T16.1.16.10" class="ltx_td ltx_align_center"><span id="T16.1.16.10.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">54.22</span></td>
</tr>
<tr id="T16.1.17" class="ltx_tr" style="background-color:#FFFFFF;">
<td id="T16.1.17.1" class="ltx_td ltx_align_center"><span id="T16.1.17.1.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">Qwen1.5-4B-Chat</span></td>
<td id="T16.1.17.2" class="ltx_td ltx_align_center"><span id="T16.1.17.2.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">3.2B</span></td>
<td id="T16.1.17.3" class="ltx_td ltx_align_center"><span id="T16.1.17.3.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">58.15</span></td>
<td id="T16.1.17.4" class="ltx_td ltx_align_center"><span id="T16.1.17.4.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">43.26</span></td>
<td id="T16.1.17.5" class="ltx_td ltx_align_center"><span id="T16.1.17.5.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">69.73</span></td>
<td id="T16.1.17.6" class="ltx_td ltx_align_center"><span id="T16.1.17.6.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">55.55</span></td>
<td id="T16.1.17.7" class="ltx_td ltx_align_justify ltx_align_top">
<span id="T16.1.17.7.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFFFFF;">
<span id="T16.1.17.7.1.1" class="ltx_p" style="width:32.5pt;"><span id="T16.1.17.7.1.1.1" class="ltx_text" style="font-size:80%;">64.96</span></span>
</span>
</td>
<td id="T16.1.17.8" class="ltx_td ltx_align_center"><span id="T16.1.17.8.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">52.24</span></td>
<td id="T16.1.17.9" class="ltx_td ltx_align_center"><span id="T16.1.17.9.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">61.89</span></td>
<td id="T16.1.17.10" class="ltx_td ltx_align_center"><span id="T16.1.17.10.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">59.39</span></td>
</tr>
<tr id="T16.1.18" class="ltx_tr" style="background-color:#E6F3F3;">
<td id="T16.1.18.1" class="ltx_td ltx_align_center"><span id="T16.1.18.1.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">Qwen1.5-7B-Chat</span></td>
<td id="T16.1.18.2" class="ltx_td ltx_align_center"><span id="T16.1.18.2.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">6.5B</span></td>
<td id="T16.1.18.3" class="ltx_td ltx_align_center"><span id="T16.1.18.3.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">64.80</span></td>
<td id="T16.1.18.4" class="ltx_td ltx_align_center"><span id="T16.1.18.4.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">56.48</span></td>
<td id="T16.1.18.5" class="ltx_td ltx_align_center"><span id="T16.1.18.5.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">79.02</span></td>
<td id="T16.1.18.6" class="ltx_td ltx_align_center"><span id="T16.1.18.6.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">60.52</span></td>
<td id="T16.1.18.7" class="ltx_td ltx_align_justify ltx_align_top">
<span id="T16.1.18.7.1" class="ltx_inline-block ltx_align_top" style="background-color:#E6F3F3;">
<span id="T16.1.18.7.1.1" class="ltx_p" style="width:32.5pt;"><span id="T16.1.18.7.1.1.1" class="ltx_text" style="font-size:80%;">66.38</span></span>
</span>
</td>
<td id="T16.1.18.8" class="ltx_td ltx_align_center"><span id="T16.1.18.8.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">54.36</span></td>
<td id="T16.1.18.9" class="ltx_td ltx_align_center"><span id="T16.1.18.9.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">68.20</span></td>
<td id="T16.1.18.10" class="ltx_td ltx_align_center"><span id="T16.1.18.10.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">68.67</span></td>
</tr>
<tr id="T16.1.1" class="ltx_tr" style="background-color:#FFFFFF;">
<td id="T16.1.1.1" class="ltx_td ltx_align_center ltx_border_t"><span id="T16.1.1.1.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">Memory<sup id="T16.1.1.1.1.1" class="ltx_sup">3</sup>-SFT</span></td>
<td id="T16.1.1.2" class="ltx_td ltx_align_center ltx_border_t"><span id="T16.1.1.2.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">2.4B</span></td>
<td id="T16.1.1.3" class="ltx_td ltx_align_center ltx_border_t"><span id="T16.1.1.3.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">63.31</span></td>
<td id="T16.1.1.4" class="ltx_td ltx_align_center ltx_border_t"><span id="T16.1.1.4.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">58.11</span></td>
<td id="T16.1.1.5" class="ltx_td ltx_align_center ltx_border_t"><span id="T16.1.1.5.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">80.51</span></td>
<td id="T16.1.1.6" class="ltx_td ltx_align_center ltx_border_t"><span id="T16.1.1.6.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">59.68</span></td>
<td id="T16.1.1.7" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="T16.1.1.7.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFFFFF;">
<span id="T16.1.1.7.1.1" class="ltx_p" style="width:32.5pt;"><span id="T16.1.1.7.1.1.1" class="ltx_text" style="font-size:80%;">74.51</span></span>
</span>
</td>
<td id="T16.1.1.8" class="ltx_td ltx_align_center ltx_border_t"><span id="T16.1.1.8.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">52.84</span></td>
<td id="T16.1.1.9" class="ltx_td ltx_align_center ltx_border_t"><span id="T16.1.1.9.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">59.29</span></td>
<td id="T16.1.1.10" class="ltx_td ltx_align_center ltx_border_t"><span id="T16.1.1.10.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">58.24</span></td>
</tr>
<tr id="T16.1.19" class="ltx_tr" style="background-color:#E6F3F3;">
<td id="T16.1.19.1" class="ltx_td ltx_align_center"><span id="T16.1.19.1.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">with vector compression</span></td>
<td id="T16.1.19.2" class="ltx_td ltx_align_center"><span id="T16.1.19.2.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">2.4B</span></td>
<td id="T16.1.19.3" class="ltx_td ltx_align_center"><span id="T16.1.19.3.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">63.33</span></td>
<td id="T16.1.19.4" class="ltx_td ltx_align_center"><span id="T16.1.19.4.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">57.94</span></td>
<td id="T16.1.19.5" class="ltx_td ltx_align_center"><span id="T16.1.19.5.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">80.65</span></td>
<td id="T16.1.19.6" class="ltx_td ltx_align_center"><span id="T16.1.19.6.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">59.66</span></td>
<td id="T16.1.19.7" class="ltx_td ltx_align_justify ltx_align_top">
<span id="T16.1.19.7.1" class="ltx_inline-block ltx_align_top" style="background-color:#E6F3F3;">
<span id="T16.1.19.7.1.1" class="ltx_p" style="width:32.5pt;"><span id="T16.1.19.7.1.1.1" class="ltx_text" style="font-size:80%;">75.14</span></span>
</span>
</td>
<td id="T16.1.19.8" class="ltx_td ltx_align_center"><span id="T16.1.19.8.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">52.24</span></td>
<td id="T16.1.19.9" class="ltx_td ltx_align_center"><span id="T16.1.19.9.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">59.66</span></td>
<td id="T16.1.19.10" class="ltx_td ltx_align_center"><span id="T16.1.19.10.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">58.05</span></td>
</tr>
<tr id="T16.1.20" class="ltx_tr" style="background-color:#FFFFFF;">
<td id="T16.1.20.1" class="ltx_td ltx_align_center ltx_border_bb"><span id="T16.1.20.1.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">without memory</span></td>
<td id="T16.1.20.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="T16.1.20.2.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">2.4B</span></td>
<td id="T16.1.20.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="T16.1.20.3.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">60.80</span></td>
<td id="T16.1.20.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="T16.1.20.4.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">57.42</span></td>
<td id="T16.1.20.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="T16.1.20.5.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">73.14</span></td>
<td id="T16.1.20.6" class="ltx_td ltx_align_center ltx_border_bb"><span id="T16.1.20.6.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">57.29</span></td>
<td id="T16.1.20.7" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="T16.1.20.7.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFFFFF;">
<span id="T16.1.20.7.1.1" class="ltx_p" style="width:32.5pt;"><span id="T16.1.20.7.1.1.1" class="ltx_text" style="font-size:80%;">74.35</span></span>
</span>
</td>
<td id="T16.1.20.8" class="ltx_td ltx_align_center ltx_border_bb"><span id="T16.1.20.8.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">51.33</span></td>
<td id="T16.1.20.9" class="ltx_td ltx_align_center ltx_border_bb"><span id="T16.1.20.9.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">56.32</span></td>
<td id="T16.1.20.10" class="ltx_td ltx_align_center ltx_border_bb"><span id="T16.1.20.10.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">55.72</span></td>
</tr>
</table>
<figcaption class="ltx_caption" style="font-size:80%;"><span class="ltx_tag ltx_tag_table"><span id="T16.5.1.1" class="ltx_text" style="font-size:113%;">Table 16</span>: </span><span id="T16.6.2" class="ltx_text" style="font-size:113%;">Few-shot evaluation of general abilities.
The model sizes only include non-embedding parameters.</span></figcaption>
</figure>
<div id="S7.SS1.p3" class="ltx_para">
<p id="S7.SS1.p3.6" class="ltx_p">The results of our model without using explicit memory is included, which indicates that explicit memory boosts the average score by <math id="S7.SS1.p3.1.m1.1" class="ltx_Math" alttext="2.51\%" display="inline"><semantics id="S7.SS1.p3.1.m1.1a"><mrow id="S7.SS1.p3.1.m1.1.1" xref="S7.SS1.p3.1.m1.1.1.cmml"><mn id="S7.SS1.p3.1.m1.1.1.2" xref="S7.SS1.p3.1.m1.1.1.2.cmml">2.51</mn><mo id="S7.SS1.p3.1.m1.1.1.1" xref="S7.SS1.p3.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S7.SS1.p3.1.m1.1b"><apply id="S7.SS1.p3.1.m1.1.1.cmml" xref="S7.SS1.p3.1.m1.1.1"><csymbol cd="latexml" id="S7.SS1.p3.1.m1.1.1.1.cmml" xref="S7.SS1.p3.1.m1.1.1.1">percent</csymbol><cn type="float" id="S7.SS1.p3.1.m1.1.1.2.cmml" xref="S7.SS1.p3.1.m1.1.1.2">2.51</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS1.p3.1.m1.1c">2.51\%</annotation></semantics></math>.
In comparison, the score difference between Llama2-7B and 13B is <math id="S7.SS1.p3.2.m2.1" class="ltx_Math" alttext="4.91\%" display="inline"><semantics id="S7.SS1.p3.2.m2.1a"><mrow id="S7.SS1.p3.2.m2.1.1" xref="S7.SS1.p3.2.m2.1.1.cmml"><mn id="S7.SS1.p3.2.m2.1.1.2" xref="S7.SS1.p3.2.m2.1.1.2.cmml">4.91</mn><mo id="S7.SS1.p3.2.m2.1.1.1" xref="S7.SS1.p3.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S7.SS1.p3.2.m2.1b"><apply id="S7.SS1.p3.2.m2.1.1.cmml" xref="S7.SS1.p3.2.m2.1.1"><csymbol cd="latexml" id="S7.SS1.p3.2.m2.1.1.1.cmml" xref="S7.SS1.p3.2.m2.1.1.1">percent</csymbol><cn type="float" id="S7.SS1.p3.2.m2.1.1.2.cmml" xref="S7.SS1.p3.2.m2.1.1.2">4.91</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS1.p3.2.m2.1c">4.91\%</annotation></semantics></math> while the latter has twice the amount of non-embedding parameters.
Thus, it reasonable to say that explicit memory can increase the “effective model size” by <math id="S7.SS1.p3.3.m3.1" class="ltx_Math" alttext="2.51/4.91\approx 51.1\%" display="inline"><semantics id="S7.SS1.p3.3.m3.1a"><mrow id="S7.SS1.p3.3.m3.1.1" xref="S7.SS1.p3.3.m3.1.1.cmml"><mrow id="S7.SS1.p3.3.m3.1.1.2" xref="S7.SS1.p3.3.m3.1.1.2.cmml"><mn id="S7.SS1.p3.3.m3.1.1.2.2" xref="S7.SS1.p3.3.m3.1.1.2.2.cmml">2.51</mn><mo id="S7.SS1.p3.3.m3.1.1.2.1" xref="S7.SS1.p3.3.m3.1.1.2.1.cmml">/</mo><mn id="S7.SS1.p3.3.m3.1.1.2.3" xref="S7.SS1.p3.3.m3.1.1.2.3.cmml">4.91</mn></mrow><mo id="S7.SS1.p3.3.m3.1.1.1" xref="S7.SS1.p3.3.m3.1.1.1.cmml">≈</mo><mrow id="S7.SS1.p3.3.m3.1.1.3" xref="S7.SS1.p3.3.m3.1.1.3.cmml"><mn id="S7.SS1.p3.3.m3.1.1.3.2" xref="S7.SS1.p3.3.m3.1.1.3.2.cmml">51.1</mn><mo id="S7.SS1.p3.3.m3.1.1.3.1" xref="S7.SS1.p3.3.m3.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.SS1.p3.3.m3.1b"><apply id="S7.SS1.p3.3.m3.1.1.cmml" xref="S7.SS1.p3.3.m3.1.1"><approx id="S7.SS1.p3.3.m3.1.1.1.cmml" xref="S7.SS1.p3.3.m3.1.1.1"></approx><apply id="S7.SS1.p3.3.m3.1.1.2.cmml" xref="S7.SS1.p3.3.m3.1.1.2"><divide id="S7.SS1.p3.3.m3.1.1.2.1.cmml" xref="S7.SS1.p3.3.m3.1.1.2.1"></divide><cn type="float" id="S7.SS1.p3.3.m3.1.1.2.2.cmml" xref="S7.SS1.p3.3.m3.1.1.2.2">2.51</cn><cn type="float" id="S7.SS1.p3.3.m3.1.1.2.3.cmml" xref="S7.SS1.p3.3.m3.1.1.2.3">4.91</cn></apply><apply id="S7.SS1.p3.3.m3.1.1.3.cmml" xref="S7.SS1.p3.3.m3.1.1.3"><csymbol cd="latexml" id="S7.SS1.p3.3.m3.1.1.3.1.cmml" xref="S7.SS1.p3.3.m3.1.1.3.1">percent</csymbol><cn type="float" id="S7.SS1.p3.3.m3.1.1.3.2.cmml" xref="S7.SS1.p3.3.m3.1.1.3.2">51.1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS1.p3.3.m3.1c">2.51/4.91\approx 51.1\%</annotation></semantics></math>.
(Also, the score difference between Qwen-1.8B and 4B is <math id="S7.SS1.p3.4.m4.1" class="ltx_Math" alttext="8.48\%" display="inline"><semantics id="S7.SS1.p3.4.m4.1a"><mrow id="S7.SS1.p3.4.m4.1.1" xref="S7.SS1.p3.4.m4.1.1.cmml"><mn id="S7.SS1.p3.4.m4.1.1.2" xref="S7.SS1.p3.4.m4.1.1.2.cmml">8.48</mn><mo id="S7.SS1.p3.4.m4.1.1.1" xref="S7.SS1.p3.4.m4.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S7.SS1.p3.4.m4.1b"><apply id="S7.SS1.p3.4.m4.1.1.cmml" xref="S7.SS1.p3.4.m4.1.1"><csymbol cd="latexml" id="S7.SS1.p3.4.m4.1.1.1.cmml" xref="S7.SS1.p3.4.m4.1.1.1">percent</csymbol><cn type="float" id="S7.SS1.p3.4.m4.1.1.2.cmml" xref="S7.SS1.p3.4.m4.1.1.2">8.48</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS1.p3.4.m4.1c">8.48\%</annotation></semantics></math> while the latter has <math id="S7.SS1.p3.5.m5.1" class="ltx_Math" alttext="167\%" display="inline"><semantics id="S7.SS1.p3.5.m5.1a"><mrow id="S7.SS1.p3.5.m5.1.1" xref="S7.SS1.p3.5.m5.1.1.cmml"><mn id="S7.SS1.p3.5.m5.1.1.2" xref="S7.SS1.p3.5.m5.1.1.2.cmml">167</mn><mo id="S7.SS1.p3.5.m5.1.1.1" xref="S7.SS1.p3.5.m5.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S7.SS1.p3.5.m5.1b"><apply id="S7.SS1.p3.5.m5.1.1.cmml" xref="S7.SS1.p3.5.m5.1.1"><csymbol cd="latexml" id="S7.SS1.p3.5.m5.1.1.1.cmml" xref="S7.SS1.p3.5.m5.1.1.1">percent</csymbol><cn type="integer" id="S7.SS1.p3.5.m5.1.1.2.cmml" xref="S7.SS1.p3.5.m5.1.1.2">167</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS1.p3.5.m5.1c">167\%</annotation></semantics></math> more non-embedding parameters.
With respect to this scale, explicit memory increases the “effective model size” by <math id="S7.SS1.p3.6.m6.1" class="ltx_Math" alttext="1.2.51/8.48\times 1.67\approx 49.4\%" display="inline"><semantics id="S7.SS1.p3.6.m6.1a"><mrow id="S7.SS1.p3.6.m6.1.1" xref="S7.SS1.p3.6.m6.1.1.cmml"><mrow id="S7.SS1.p3.6.m6.1.1.2" xref="S7.SS1.p3.6.m6.1.1.2.cmml"><mrow id="S7.SS1.p3.6.m6.1.1.2.2" xref="S7.SS1.p3.6.m6.1.1.2.2.cmml"><mn id="S7.SS1.p3.6.m6.1.1.2.2.2" xref="S7.SS1.p3.6.m6.1.1.2.2.2.cmml">1.2.51</mn><mo id="S7.SS1.p3.6.m6.1.1.2.2.1" xref="S7.SS1.p3.6.m6.1.1.2.2.1.cmml">/</mo><mn id="S7.SS1.p3.6.m6.1.1.2.2.3" xref="S7.SS1.p3.6.m6.1.1.2.2.3.cmml">8.48</mn></mrow><mo lspace="0.222em" rspace="0.222em" id="S7.SS1.p3.6.m6.1.1.2.1" xref="S7.SS1.p3.6.m6.1.1.2.1.cmml">×</mo><mn id="S7.SS1.p3.6.m6.1.1.2.3" xref="S7.SS1.p3.6.m6.1.1.2.3.cmml">1.67</mn></mrow><mo id="S7.SS1.p3.6.m6.1.1.1" xref="S7.SS1.p3.6.m6.1.1.1.cmml">≈</mo><mrow id="S7.SS1.p3.6.m6.1.1.3" xref="S7.SS1.p3.6.m6.1.1.3.cmml"><mn id="S7.SS1.p3.6.m6.1.1.3.2" xref="S7.SS1.p3.6.m6.1.1.3.2.cmml">49.4</mn><mo id="S7.SS1.p3.6.m6.1.1.3.1" xref="S7.SS1.p3.6.m6.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.SS1.p3.6.m6.1b"><apply id="S7.SS1.p3.6.m6.1.1.cmml" xref="S7.SS1.p3.6.m6.1.1"><approx id="S7.SS1.p3.6.m6.1.1.1.cmml" xref="S7.SS1.p3.6.m6.1.1.1"></approx><apply id="S7.SS1.p3.6.m6.1.1.2.cmml" xref="S7.SS1.p3.6.m6.1.1.2"><times id="S7.SS1.p3.6.m6.1.1.2.1.cmml" xref="S7.SS1.p3.6.m6.1.1.2.1"></times><apply id="S7.SS1.p3.6.m6.1.1.2.2.cmml" xref="S7.SS1.p3.6.m6.1.1.2.2"><divide id="S7.SS1.p3.6.m6.1.1.2.2.1.cmml" xref="S7.SS1.p3.6.m6.1.1.2.2.1"></divide><cn type="float" id="S7.SS1.p3.6.m6.1.1.2.2.2.cmml" xref="S7.SS1.p3.6.m6.1.1.2.2.2">1.2.51</cn><cn type="float" id="S7.SS1.p3.6.m6.1.1.2.2.3.cmml" xref="S7.SS1.p3.6.m6.1.1.2.2.3">8.48</cn></apply><cn type="float" id="S7.SS1.p3.6.m6.1.1.2.3.cmml" xref="S7.SS1.p3.6.m6.1.1.2.3">1.67</cn></apply><apply id="S7.SS1.p3.6.m6.1.1.3.cmml" xref="S7.SS1.p3.6.m6.1.1.3"><csymbol cd="latexml" id="S7.SS1.p3.6.m6.1.1.3.1.cmml" xref="S7.SS1.p3.6.m6.1.1.3.1">percent</csymbol><cn type="float" id="S7.SS1.p3.6.m6.1.1.3.2.cmml" xref="S7.SS1.p3.6.m6.1.1.3.2">49.4</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS1.p3.6.m6.1c">1.2.51/8.48\times 1.67\approx 49.4\%</annotation></semantics></math>.)</p>
</div>
<div id="S7.SS1.p4" class="ltx_para">
<p id="S7.SS1.p4.2" class="ltx_p">We also include the results of Memory<sup id="S7.SS1.p4.2.1" class="ltx_sup">3</sup> with vector compression (Section <a href="#S3.SS3" title="3.3 | Memory Sparsification and Storage ‣ 3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3</span></a>).
Even though the key-value vectors of the explicit memories are compressed to <math id="S7.SS1.p4.2.m2.1" class="ltx_Math" alttext="8.75\%" display="inline"><semantics id="S7.SS1.p4.2.m2.1a"><mrow id="S7.SS1.p4.2.m2.1.1" xref="S7.SS1.p4.2.m2.1.1.cmml"><mn id="S7.SS1.p4.2.m2.1.1.2" xref="S7.SS1.p4.2.m2.1.1.2.cmml">8.75</mn><mo id="S7.SS1.p4.2.m2.1.1.1" xref="S7.SS1.p4.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S7.SS1.p4.2.m2.1b"><apply id="S7.SS1.p4.2.m2.1.1.cmml" xref="S7.SS1.p4.2.m2.1.1"><csymbol cd="latexml" id="S7.SS1.p4.2.m2.1.1.1.cmml" xref="S7.SS1.p4.2.m2.1.1.1">percent</csymbol><cn type="float" id="S7.SS1.p4.2.m2.1.1.2.cmml" xref="S7.SS1.p4.2.m2.1.1.2">8.75</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS1.p4.2.m2.1c">8.75\%</annotation></semantics></math> of their original sizes, the performance of our model does not show any degradation.</p>
</div>
<div id="S7.SS1.p5" class="ltx_para">
<p id="S7.SS1.p5.1" class="ltx_p">Other supplementary evaluations can be found in Appendix <a href="#A3" title="Appendix C Supplementary Evaluation Results ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>.</p>
</div>
<div id="S7.SS1.p6" class="ltx_para">
<p id="S7.SS1.p6.2" class="ltx_p">Next, we compare with a LLM that is pretrained with text retrieval.
Specially, we consider the largest version of the Retro++ model <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib113" title="" class="ltx_ref">113</a>]</cite>, Retro++ XXL with 9.5B parameters.
All tasks from Table 6 of <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib113" title="" class="ltx_ref">113</a>]</cite> are taken, except for HANS, which is not available on lm-eval-harness, and all tasks are zero-shot.
Similar to Table <a href="#T16" title="Table 16 ‣ 7.1 | General Abilities ‣ 7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">16</span></a>, Memory<sup id="S7.SS1.p6.2.1" class="ltx_sup">3</sup> is tested with a filtering threshold of 2/3.
The results are listed in Table <a href="#T17" title="Table 17 ‣ 7.1 | General Abilities ‣ 7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">17</span></a>, where Memory<sup id="S7.SS1.p6.2.2" class="ltx_sup">3</sup> outperforms the model with much larger parameter size and reference dataset size.</p>
</div>
<figure id="T17" class="ltx_table">
<table id="T17.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="T17.1.2" class="ltx_tr" style="background-color:#E6F3F3;">
<td id="T17.1.2.1" class="ltx_td ltx_align_center ltx_border_tt"><span id="T17.1.2.1.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">LLM</span></td>
<td id="T17.1.2.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt"><span id="T17.1.2.2.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">Param size</span></td>
<td id="T17.1.2.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt"><span id="T17.1.2.3.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">Avg.</span></td>
<td id="T17.1.2.4" class="ltx_td ltx_align_center ltx_border_tt"><span id="T17.1.2.4.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">HellaSwag</span></td>
<td id="T17.1.2.5" class="ltx_td ltx_align_center ltx_border_tt"><span id="T17.1.2.5.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">BoolQ</span></td>
<td id="T17.1.2.6" class="ltx_td ltx_align_center ltx_border_tt"><span id="T17.1.2.6.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">Lambada</span></td>
<td id="T17.1.2.7" class="ltx_td ltx_align_center ltx_border_tt"><span id="T17.1.2.7.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">RACE</span></td>
</tr>
<tr id="T17.1.3" class="ltx_tr">
<td id="T17.1.3.1" class="ltx_td ltx_align_center"><span id="T17.1.3.1.1" class="ltx_text" style="font-size:80%;">Retro++ XXL</span></td>
<td id="T17.1.3.2" class="ltx_td ltx_align_center ltx_border_r"><span id="T17.1.3.2.1" class="ltx_text" style="font-size:80%;">9.1B</span></td>
<td id="T17.1.3.3" class="ltx_td ltx_align_center ltx_border_r"><span id="T17.1.3.3.1" class="ltx_text" style="font-size:80%;">61.0</span></td>
<td id="T17.1.3.4" class="ltx_td ltx_align_center"><span id="T17.1.3.4.1" class="ltx_text" style="font-size:80%;">70.6</span></td>
<td id="T17.1.3.5" class="ltx_td ltx_align_center"><span id="T17.1.3.5.1" class="ltx_text" style="font-size:80%;">70.7</span></td>
<td id="T17.1.3.6" class="ltx_td ltx_align_center"><span id="T17.1.3.6.1" class="ltx_text" style="font-size:80%;">72.7</span></td>
<td id="T17.1.3.7" class="ltx_td ltx_align_center"><span id="T17.1.3.7.1" class="ltx_text" style="font-size:80%;">43.2</span></td>
</tr>
<tr id="T17.1.1" class="ltx_tr">
<td id="T17.1.1.1" class="ltx_td ltx_align_center">
<span id="T17.1.1.1.1" class="ltx_text" style="font-size:80%;">Memory</span><sup id="T17.1.1.1.2" class="ltx_sup"><span id="T17.1.1.1.2.1" class="ltx_text" style="font-size:80%;">3</span></sup><span id="T17.1.1.1.3" class="ltx_text" style="font-size:80%;">-SFT</span>
</td>
<td id="T17.1.1.2" class="ltx_td ltx_align_center ltx_border_r"><span id="T17.1.1.2.1" class="ltx_text" style="font-size:80%;">2.4B</span></td>
<td id="T17.1.1.3" class="ltx_td ltx_align_center ltx_border_r"><span id="T17.1.1.3.1" class="ltx_text ltx_font_bold" style="font-size:80%;">64.7</span></td>
<td id="T17.1.1.4" class="ltx_td ltx_align_center"><span id="T17.1.1.4.1" class="ltx_text" style="font-size:80%;">83.3</span></td>
<td id="T17.1.1.5" class="ltx_td ltx_align_center"><span id="T17.1.1.5.1" class="ltx_text" style="font-size:80%;">80.4</span></td>
<td id="T17.1.1.6" class="ltx_td ltx_align_center"><span id="T17.1.1.6.1" class="ltx_text" style="font-size:80%;">57.9</span></td>
<td id="T17.1.1.7" class="ltx_td ltx_align_center"><span id="T17.1.1.7.1" class="ltx_text" style="font-size:80%;">45.3</span></td>
</tr>
<tr id="T17.1.4" class="ltx_tr">
<td id="T17.1.4.1" class="ltx_td ltx_border_tt"></td>
<td id="T17.1.4.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt"><span id="T17.1.4.2.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">Reference size</span></td>
<td id="T17.1.4.3" class="ltx_td ltx_border_r ltx_border_tt"></td>
<td id="T17.1.4.4" class="ltx_td ltx_align_center ltx_border_tt"><span id="T17.1.4.4.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">PiQA</span></td>
<td id="T17.1.4.5" class="ltx_td ltx_align_center ltx_border_tt"><span id="T17.1.4.5.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">Winogrand</span></td>
<td id="T17.1.4.6" class="ltx_td ltx_align_center ltx_border_tt"><span id="T17.1.4.6.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">ANLI-R2</span></td>
<td id="T17.1.4.7" class="ltx_td ltx_align_center ltx_border_tt"><span id="T17.1.4.7.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">WiC</span></td>
</tr>
<tr id="T17.1.5" class="ltx_tr">
<td id="T17.1.5.1" class="ltx_td"></td>
<td id="T17.1.5.2" class="ltx_td ltx_align_center ltx_border_r"><span id="T17.1.5.2.1" class="ltx_text" style="font-size:80%;">330B</span></td>
<td id="T17.1.5.3" class="ltx_td ltx_border_r"></td>
<td id="T17.1.5.4" class="ltx_td ltx_align_center"><span id="T17.1.5.4.1" class="ltx_text" style="font-size:80%;">77.4</span></td>
<td id="T17.1.5.5" class="ltx_td ltx_align_center"><span id="T17.1.5.5.1" class="ltx_text" style="font-size:80%;">65.8</span></td>
<td id="T17.1.5.6" class="ltx_td ltx_align_center"><span id="T17.1.5.6.1" class="ltx_text" style="font-size:80%;">35.5</span></td>
<td id="T17.1.5.7" class="ltx_td ltx_align_center"><span id="T17.1.5.7.1" class="ltx_text" style="font-size:80%;">52.4</span></td>
</tr>
<tr id="T17.1.6" class="ltx_tr">
<td id="T17.1.6.1" class="ltx_td ltx_border_bb"></td>
<td id="T17.1.6.2" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r"><span id="T17.1.6.2.1" class="ltx_text" style="font-size:80%;">14.3B</span></td>
<td id="T17.1.6.3" class="ltx_td ltx_border_bb ltx_border_r"></td>
<td id="T17.1.6.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="T17.1.6.4.1" class="ltx_text" style="font-size:80%;">76.6</span></td>
<td id="T17.1.6.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="T17.1.6.5.1" class="ltx_text" style="font-size:80%;">75.8</span></td>
<td id="T17.1.6.6" class="ltx_td ltx_align_center ltx_border_bb"><span id="T17.1.6.6.1" class="ltx_text" style="font-size:80%;">41.6</span></td>
<td id="T17.1.6.7" class="ltx_td ltx_align_center ltx_border_bb"><span id="T17.1.6.7.1" class="ltx_text" style="font-size:80%;">56.9</span></td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering" style="font-size:80%;"><span class="ltx_tag ltx_tag_table"><span id="T17.8.1.1" class="ltx_text" style="font-size:113%;">Table 17</span>: </span><span id="T17.9.2" class="ltx_text" style="font-size:113%;">Zero-shot comparison of LLMs pretrained with retrieval.
The scores of Retro++ are taken from <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib113" title="" class="ltx_ref">113</a>]</cite>.
The size of a reference dataset is its number of tokens.
The non-embedding parameter size of Retro++ is inferred from its vocabulary size.</span></figcaption>
</figure>
</section>
<section id="S7.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsection" style="font-size:120%;color:#00868B;">7.2 <span id="S7.SS2.1.1" class="ltx_text" style="color:#808080;">| </span>Conversation Skill</h3>

<div id="S7.SS2.p1" class="ltx_para">
<p id="S7.SS2.p1.2" class="ltx_p">Next we evaluate the conversation skills of Memory<sup id="S7.SS2.p1.2.1" class="ltx_sup">3</sup>.
We use MT-Bench (the Multi-turn Benchmark) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib133" title="" class="ltx_ref">133</a>]</cite> that consists of multi-round and open-ended questions.
The results are listed in <a href="#T18" title="In 7.2 | Conversation Skill ‣ 7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">18</span></a>, including the Memory<sup id="S7.SS2.p1.2.2" class="ltx_sup">3</sup> model finetuned by DPO introduced in Section <a href="#S6.SS2" title="6.2 | Direct Preference Optimization ‣ 6 | Fine-tuning and Alignment ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6.2</span></a>.</p>
</div>
<figure id="T18" class="ltx_table">
<table id="T18.2" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="T18.2.3" class="ltx_tr">
<td id="T18.2.3.1" class="ltx_td ltx_align_left ltx_border_tt"><span id="T18.2.3.1.1" class="ltx_text" style="font-size:90%;">LLM</span></td>
<td id="T18.2.3.2" class="ltx_td ltx_align_center ltx_border_tt"><span id="T18.2.3.2.1" class="ltx_text" style="font-size:90%;">Size</span></td>
<td id="T18.2.3.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_tt"><span id="T18.2.3.3.1" class="ltx_text" style="font-size:90%;">MT-Bench Score</span></td>
</tr>
<tr id="T18.2.4" class="ltx_tr">
<td id="T18.2.4.1" class="ltx_td ltx_align_left ltx_border_t"><span id="T18.2.4.1.1" class="ltx_text" style="font-size:90%;">Phi-3</span></td>
<td id="T18.2.4.2" class="ltx_td ltx_align_center ltx_border_t"><span id="T18.2.4.2.1" class="ltx_text" style="font-size:90%;">3.6B</span></td>
<td id="T18.2.4.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t"><span id="T18.2.4.3.1" class="ltx_text" style="font-size:90%;">8.38</span></td>
</tr>
<tr id="T18.2.5" class="ltx_tr">
<td id="T18.2.5.1" class="ltx_td ltx_align_left"><span id="T18.2.5.1.1" class="ltx_text" style="font-size:90%;">Mistral-7B-Instruct-v0.2</span></td>
<td id="T18.2.5.2" class="ltx_td ltx_align_center"><span id="T18.2.5.2.1" class="ltx_text" style="font-size:90%;">7.0B</span></td>
<td id="T18.2.5.3" class="ltx_td ltx_nopad_r ltx_align_center"><span id="T18.2.5.3.1" class="ltx_text" style="font-size:90%;">7.60</span></td>
</tr>
<tr id="T18.2.6" class="ltx_tr">
<td id="T18.2.6.1" class="ltx_td ltx_align_left"><span id="T18.2.6.1.1" class="ltx_text" style="font-size:90%;">Qwen1.5-7B-Chat</span></td>
<td id="T18.2.6.2" class="ltx_td ltx_align_center"><span id="T18.2.6.2.1" class="ltx_text" style="font-size:90%;">6.5B</span></td>
<td id="T18.2.6.3" class="ltx_td ltx_nopad_r ltx_align_center"><span id="T18.2.6.3.1" class="ltx_text" style="font-size:90%;">7.60</span></td>
</tr>
<tr id="T18.2.7" class="ltx_tr">
<td id="T18.2.7.1" class="ltx_td ltx_align_left"><span id="T18.2.7.1.1" class="ltx_text" style="font-size:90%;">Zephyr-7B-beta</span></td>
<td id="T18.2.7.2" class="ltx_td ltx_align_center"><span id="T18.2.7.2.1" class="ltx_text" style="font-size:90%;">7.0B</span></td>
<td id="T18.2.7.3" class="ltx_td ltx_nopad_r ltx_align_center"><span id="T18.2.7.3.1" class="ltx_text" style="font-size:90%;">7.34</span></td>
</tr>
<tr id="T18.2.8" class="ltx_tr">
<td id="T18.2.8.1" class="ltx_td ltx_align_left"><span id="T18.2.8.1.1" class="ltx_text" style="font-size:90%;">MiniCPM-2B-DPO</span></td>
<td id="T18.2.8.2" class="ltx_td ltx_align_center"><span id="T18.2.8.2.1" class="ltx_text" style="font-size:90%;">2.4B</span></td>
<td id="T18.2.8.3" class="ltx_td ltx_nopad_r ltx_align_center"><span id="T18.2.8.3.1" class="ltx_text" style="font-size:90%;">6.89</span></td>
</tr>
<tr id="T18.2.9" class="ltx_tr">
<td id="T18.2.9.1" class="ltx_td ltx_align_left"><span id="T18.2.9.1.1" class="ltx_text" style="font-size:90%;">Llama-2-70B-Chat</span></td>
<td id="T18.2.9.2" class="ltx_td ltx_align_center"><span id="T18.2.9.2.1" class="ltx_text" style="font-size:90%;">68B</span></td>
<td id="T18.2.9.3" class="ltx_td ltx_nopad_r ltx_align_center"><span id="T18.2.9.3.1" class="ltx_text" style="font-size:90%;">6.86</span></td>
</tr>
<tr id="T18.2.10" class="ltx_tr">
<td id="T18.2.10.1" class="ltx_td ltx_align_left"><span id="T18.2.10.1.1" class="ltx_text" style="font-size:90%;">Mistral-7B-Instruct-v0.1</span></td>
<td id="T18.2.10.2" class="ltx_td ltx_align_center"><span id="T18.2.10.2.1" class="ltx_text" style="font-size:90%;">7.0B</span></td>
<td id="T18.2.10.3" class="ltx_td ltx_nopad_r ltx_align_center"><span id="T18.2.10.3.1" class="ltx_text" style="font-size:90%;">6.84</span></td>
</tr>
<tr id="T18.2.11" class="ltx_tr">
<td id="T18.2.11.1" class="ltx_td ltx_align_left"><span id="T18.2.11.1.1" class="ltx_text" style="font-size:90%;">Llama-2-13B-Chat</span></td>
<td id="T18.2.11.2" class="ltx_td ltx_align_center"><span id="T18.2.11.2.1" class="ltx_text" style="font-size:90%;">13B</span></td>
<td id="T18.2.11.3" class="ltx_td ltx_nopad_r ltx_align_center"><span id="T18.2.11.3.1" class="ltx_text" style="font-size:90%;">6.65</span></td>
</tr>
<tr id="T18.2.12" class="ltx_tr">
<td id="T18.2.12.1" class="ltx_td ltx_align_left"><span id="T18.2.12.1.1" class="ltx_text" style="font-size:90%;">Llama-2-7B-Chat</span></td>
<td id="T18.2.12.2" class="ltx_td ltx_align_center"><span id="T18.2.12.2.1" class="ltx_text" style="font-size:90%;">6.5B</span></td>
<td id="T18.2.12.3" class="ltx_td ltx_nopad_r ltx_align_center"><span id="T18.2.12.3.1" class="ltx_text" style="font-size:90%;">6.57</span></td>
</tr>
<tr id="T18.2.13" class="ltx_tr">
<td id="T18.2.13.1" class="ltx_td ltx_align_left"><span id="T18.2.13.1.1" class="ltx_text" style="font-size:90%;">MPT-30B-Chat</span></td>
<td id="T18.2.13.2" class="ltx_td ltx_align_center"><span id="T18.2.13.2.1" class="ltx_text" style="font-size:90%;">30B</span></td>
<td id="T18.2.13.3" class="ltx_td ltx_nopad_r ltx_align_center"><span id="T18.2.13.3.1" class="ltx_text" style="font-size:90%;">6.39</span></td>
</tr>
<tr id="T18.2.14" class="ltx_tr">
<td id="T18.2.14.1" class="ltx_td ltx_align_left"><span id="T18.2.14.1.1" class="ltx_text" style="font-size:90%;">ChatGLM2-6B</span></td>
<td id="T18.2.14.2" class="ltx_td ltx_align_center"><span id="T18.2.14.2.1" class="ltx_text" style="font-size:90%;">6.1B</span></td>
<td id="T18.2.14.3" class="ltx_td ltx_nopad_r ltx_align_center"><span id="T18.2.14.3.1" class="ltx_text" style="font-size:90%;">4.96</span></td>
</tr>
<tr id="T18.2.15" class="ltx_tr">
<td id="T18.2.15.1" class="ltx_td ltx_align_left"><span id="T18.2.15.1.1" class="ltx_text" style="font-size:90%;">Falcon-40B-Instruct</span></td>
<td id="T18.2.15.2" class="ltx_td ltx_align_center"><span id="T18.2.15.2.1" class="ltx_text" style="font-size:90%;">41B</span></td>
<td id="T18.2.15.3" class="ltx_td ltx_nopad_r ltx_align_center"><span id="T18.2.15.3.1" class="ltx_text" style="font-size:90%;">4.07</span></td>
</tr>
<tr id="T18.2.16" class="ltx_tr">
<td id="T18.2.16.1" class="ltx_td ltx_align_left"><span id="T18.2.16.1.1" class="ltx_text" style="font-size:90%;">Vicuna-7B</span></td>
<td id="T18.2.16.2" class="ltx_td ltx_align_center"><span id="T18.2.16.2.1" class="ltx_text" style="font-size:90%;">6.5B</span></td>
<td id="T18.2.16.3" class="ltx_td ltx_nopad_r ltx_align_center"><span id="T18.2.16.3.1" class="ltx_text" style="font-size:90%;">3.26</span></td>
</tr>
<tr id="T18.1.1" class="ltx_tr">
<td id="T18.1.1.1" class="ltx_td ltx_align_left ltx_border_t">
<span id="T18.1.1.1.1" class="ltx_text" style="font-size:90%;">Memory</span><sup id="T18.1.1.1.2" class="ltx_sup"><span id="T18.1.1.1.2.1" class="ltx_text" style="font-size:90%;">3</span></sup><span id="T18.1.1.1.3" class="ltx_text" style="font-size:90%;">-SFT</span>
</td>
<td id="T18.1.1.2" class="ltx_td ltx_align_center ltx_border_t"><span id="T18.1.1.2.1" class="ltx_text" style="font-size:90%;">2.4B</span></td>
<td id="T18.1.1.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t"><span id="T18.1.1.3.1" class="ltx_text" style="font-size:90%;">5.31</span></td>
</tr>
<tr id="T18.2.2" class="ltx_tr">
<td id="T18.2.2.1" class="ltx_td ltx_align_left ltx_border_bb">
<span id="T18.2.2.1.1" class="ltx_text" style="font-size:90%;">Memory</span><sup id="T18.2.2.1.2" class="ltx_sup"><span id="T18.2.2.1.2.1" class="ltx_text" style="font-size:90%;">3</span></sup><span id="T18.2.2.1.3" class="ltx_text" style="font-size:90%;">-DPO</span>
</td>
<td id="T18.2.2.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="T18.2.2.2.1" class="ltx_text" style="font-size:90%;">2.4B</span></td>
<td id="T18.2.2.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb"><span id="T18.2.2.3.1" class="ltx_text" style="font-size:90%;">5.80</span></td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering" style="font-size:90%;"><span class="ltx_tag ltx_tag_table">Table 18: </span>MT-Bench scores.
The model sizes only include non-embedding parameters.</figcaption>
</figure>
<div id="S7.SS2.p2" class="ltx_para">
<p id="S7.SS2.p2.1" class="ltx_p">We obtain all these scores using GPT-4-0613 as grader, following the single answer grading mode of MT-Bench.
Our model outperforms Vicuna-7B, Falcon-40B-Instruct, and ChatGLM2-6B with fewer parameters.</p>
</div>
</section>
<section id="S7.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsection" style="font-size:120%;color:#00868B;">7.3 <span id="S7.SS3.1.1" class="ltx_text" style="color:#808080;">| </span>Hallucination and Factuality</h3>

<div id="S7.SS3.p1" class="ltx_para">
<p id="S7.SS3.p1.2" class="ltx_p">Despite considerable progress, LLMs still face issues with hallucination, leading to outputs that often stray from factual accuracy <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib97" title="" class="ltx_ref">97</a>]</cite>.
Conceptually, Memory<sup id="S7.SS3.p1.2.1" class="ltx_sup">3</sup> should be less vulnerable to hallucination, since its explicit memories directly correspond to reference texts, whereas compressing texts into the model parameters might incur information loss.
To evaluate hallucination, we select two English datasets, TruthfulQA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib70" title="" class="ltx_ref">70</a>]</cite> and HaluEval, and one Chinese dataset <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib64" title="" class="ltx_ref">64</a>]</cite>, HalluQA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib20" title="" class="ltx_ref">20</a>]</cite>.
TruthfulQA is implemented with lm-evaluation-harness <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib44" title="" class="ltx_ref">44</a>]</cite>, while HaluEval and HalluQA are implemented with UHGEval <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib69" title="" class="ltx_ref">69</a>]</cite>.
The results are shown in Table <a href="#T19" title="Table 19 ‣ 7.3 | Hallucination and Factuality ‣ 7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">19</span></a>, with Memory<sup id="S7.SS3.p1.2.2" class="ltx_sup">3</sup> achieving the highest scores on most tasks.</p>
</div>
<figure id="T19" class="ltx_table">
<table id="T19.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="T19.1.2" class="ltx_tr">
<td id="T19.1.2.1" class="ltx_td ltx_border_tt"></td>
<td id="T19.1.2.2" class="ltx_td ltx_border_tt"></td>
<td id="T19.1.2.3" class="ltx_td ltx_border_tt"></td>
<td id="T19.1.2.4" class="ltx_td ltx_align_center ltx_border_tt" colspan="4"><span id="T19.1.2.4.1" class="ltx_text" style="font-size:80%;">English</span></td>
<td id="T19.1.2.5" class="ltx_td ltx_align_left ltx_border_tt"><span id="T19.1.2.5.1" class="ltx_text" style="font-size:80%;">Chinese</span></td>
</tr>
<tr id="T19.1.3" class="ltx_tr" style="background-color:#FFFFFF;">
<td id="T19.1.3.1" class="ltx_td ltx_align_left"><span id="T19.1.3.1.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">LLM</span></td>
<td id="T19.1.3.2" class="ltx_td ltx_align_left"><span id="T19.1.3.2.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">Size</span></td>
<td id="T19.1.3.3" class="ltx_td ltx_align_left"><span id="T19.1.3.3.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">Avg.</span></td>
<td id="T19.1.3.4" class="ltx_td ltx_align_left ltx_border_t"><span id="T19.1.3.4.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">HaluE-QA</span></td>
<td id="T19.1.3.5" class="ltx_td ltx_align_left ltx_border_t"><span id="T19.1.3.5.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">HaluE-Dialogue</span></td>
<td id="T19.1.3.6" class="ltx_td ltx_align_left ltx_border_t"><span id="T19.1.3.6.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">TruQA-MC1</span></td>
<td id="T19.1.3.7" class="ltx_td ltx_align_left ltx_border_t"><span id="T19.1.3.7.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">TruQA-MC2</span></td>
<td id="T19.1.3.8" class="ltx_td ltx_align_left ltx_border_t"><span id="T19.1.3.8.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">HalluQA</span></td>
</tr>
<tr id="T19.1.4" class="ltx_tr" style="background-color:#E6F3F3;">
<td id="T19.1.4.1" class="ltx_td ltx_align_left ltx_border_t"><span id="T19.1.4.1.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">Falcon-40B</span></td>
<td id="T19.1.4.2" class="ltx_td ltx_align_left ltx_border_t"><span id="T19.1.4.2.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">41B</span></td>
<td id="T19.1.4.3" class="ltx_td ltx_align_left ltx_border_t"><span id="T19.1.4.3.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">35.37</span></td>
<td id="T19.1.4.4" class="ltx_td ltx_align_left ltx_border_t"><span id="T19.1.4.4.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">46.84</span></td>
<td id="T19.1.4.5" class="ltx_td ltx_align_left ltx_border_t"><span id="T19.1.4.5.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">40.80</span></td>
<td id="T19.1.4.6" class="ltx_td ltx_align_left ltx_border_t"><span id="T19.1.4.6.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">27.29</span></td>
<td id="T19.1.4.7" class="ltx_td ltx_align_left ltx_border_t"><span id="T19.1.4.7.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">41.71</span></td>
<td id="T19.1.4.8" class="ltx_td ltx_align_left ltx_border_t"><span id="T19.1.4.8.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">20.18</span></td>
</tr>
<tr id="T19.1.5" class="ltx_tr" style="background-color:#FFFFFF;">
<td id="T19.1.5.1" class="ltx_td ltx_align_left"><span id="T19.1.5.1.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">Llama2-13B</span></td>
<td id="T19.1.5.2" class="ltx_td ltx_align_left"><span id="T19.1.5.2.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">13B</span></td>
<td id="T19.1.5.3" class="ltx_td ltx_align_left"><span id="T19.1.5.3.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">28.01</span></td>
<td id="T19.1.5.4" class="ltx_td ltx_align_left"><span id="T19.1.5.4.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">23.34</span></td>
<td id="T19.1.5.5" class="ltx_td ltx_align_left"><span id="T19.1.5.5.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">31.05</span></td>
<td id="T19.1.5.6" class="ltx_td ltx_align_left"><span id="T19.1.5.6.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">25.95</span></td>
<td id="T19.1.5.7" class="ltx_td ltx_align_left"><span id="T19.1.5.7.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">36.89</span></td>
<td id="T19.1.5.8" class="ltx_td ltx_align_left"><span id="T19.1.5.8.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">22.81</span></td>
</tr>
<tr id="T19.1.6" class="ltx_tr" style="background-color:#E6F3F3;">
<td id="T19.1.6.1" class="ltx_td ltx_align_left"><span id="T19.1.6.1.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">Vicuna-13B-v1.5</span></td>
<td id="T19.1.6.2" class="ltx_td ltx_align_left"><span id="T19.1.6.2.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">13B</span></td>
<td id="T19.1.6.3" class="ltx_td ltx_align_left"><span id="T19.1.6.3.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">37.07</span></td>
<td id="T19.1.6.4" class="ltx_td ltx_align_left"><span id="T19.1.6.4.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">24.93</span></td>
<td id="T19.1.6.5" class="ltx_td ltx_align_left"><span id="T19.1.6.5.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">37.35</span></td>
<td id="T19.1.6.6" class="ltx_td ltx_align_left"><span id="T19.1.6.6.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">35.13</span></td>
<td id="T19.1.6.7" class="ltx_td ltx_align_left"><span id="T19.1.6.7.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">50.88</span></td>
<td id="T19.1.6.8" class="ltx_td ltx_align_left"><span id="T19.1.6.8.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">N/A</span></td>
</tr>
<tr id="T19.1.7" class="ltx_tr" style="background-color:#FFFFFF;">
<td id="T19.1.7.1" class="ltx_td ltx_align_left"><span id="T19.1.7.1.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">Baichuan2-13B</span></td>
<td id="T19.1.7.2" class="ltx_td ltx_align_left"><span id="T19.1.7.2.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">13B</span></td>
<td id="T19.1.7.3" class="ltx_td ltx_align_left"><span id="T19.1.7.3.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">37.64</span></td>
<td id="T19.1.7.4" class="ltx_td ltx_align_left"><span id="T19.1.7.4.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">46.02</span></td>
<td id="T19.1.7.5" class="ltx_td ltx_align_left"><span id="T19.1.7.5.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">45.45</span></td>
<td id="T19.1.7.6" class="ltx_td ltx_align_left"><span id="T19.1.7.6.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">26.81</span></td>
<td id="T19.1.7.7" class="ltx_td ltx_align_left"><span id="T19.1.7.7.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">39.79</span></td>
<td id="T19.1.7.8" class="ltx_td ltx_align_left"><span id="T19.1.7.8.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">30.12</span></td>
</tr>
<tr id="T19.1.8" class="ltx_tr" style="background-color:#E6F3F3;">
<td id="T19.1.8.1" class="ltx_td ltx_align_left"><span id="T19.1.8.1.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">Gemma-7B</span></td>
<td id="T19.1.8.2" class="ltx_td ltx_align_left"><span id="T19.1.8.2.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">7.8B</span></td>
<td id="T19.1.8.3" class="ltx_td ltx_align_left"><span id="T19.1.8.3.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">37.03</span></td>
<td id="T19.1.8.4" class="ltx_td ltx_align_left"><span id="T19.1.8.4.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">50.91</span></td>
<td id="T19.1.8.5" class="ltx_td ltx_align_left"><span id="T19.1.8.5.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">48.19</span></td>
<td id="T19.1.8.6" class="ltx_td ltx_align_left"><span id="T19.1.8.6.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">20.69</span></td>
<td id="T19.1.8.7" class="ltx_td ltx_align_left"><span id="T19.1.8.7.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">46.65</span></td>
<td id="T19.1.8.8" class="ltx_td ltx_align_left"><span id="T19.1.8.8.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">18.71</span></td>
</tr>
<tr id="T19.1.9" class="ltx_tr" style="background-color:#FFFFFF;">
<td id="T19.1.9.1" class="ltx_td ltx_align_left"><span id="T19.1.9.1.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">Mistral-7B-v0.1</span></td>
<td id="T19.1.9.2" class="ltx_td ltx_align_left"><span id="T19.1.9.2.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">7.0B</span></td>
<td id="T19.1.9.3" class="ltx_td ltx_align_left"><span id="T19.1.9.3.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">34.18</span></td>
<td id="T19.1.9.4" class="ltx_td ltx_align_left"><span id="T19.1.9.4.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">40.68</span></td>
<td id="T19.1.9.5" class="ltx_td ltx_align_left"><span id="T19.1.9.5.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">37.64</span></td>
<td id="T19.1.9.6" class="ltx_td ltx_align_left"><span id="T19.1.9.6.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">28.03</span></td>
<td id="T19.1.9.7" class="ltx_td ltx_align_left"><span id="T19.1.9.7.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">42.60</span></td>
<td id="T19.1.9.8" class="ltx_td ltx_align_left"><span id="T19.1.9.8.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">21.93</span></td>
</tr>
<tr id="T19.1.10" class="ltx_tr" style="background-color:#E6F3F3;">
<td id="T19.1.10.1" class="ltx_td ltx_align_left"><span id="T19.1.10.1.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">Llama2-7B</span></td>
<td id="T19.1.10.2" class="ltx_td ltx_align_left"><span id="T19.1.10.2.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">6.5B</span></td>
<td id="T19.1.10.3" class="ltx_td ltx_align_left"><span id="T19.1.10.3.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">36.80</span></td>
<td id="T19.1.10.4" class="ltx_td ltx_align_left"><span id="T19.1.10.4.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">52.46</span></td>
<td id="T19.1.10.5" class="ltx_td ltx_align_left"><span id="T19.1.10.5.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">51.93</span></td>
<td id="T19.1.10.6" class="ltx_td ltx_align_left"><span id="T19.1.10.6.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">25.09</span></td>
<td id="T19.1.10.7" class="ltx_td ltx_align_left"><span id="T19.1.10.7.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">38.94</span></td>
<td id="T19.1.10.8" class="ltx_td ltx_align_left"><span id="T19.1.10.8.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">15.59</span></td>
</tr>
<tr id="T19.1.11" class="ltx_tr" style="background-color:#FFFFFF;">
<td id="T19.1.11.1" class="ltx_td ltx_align_left"><span id="T19.1.11.1.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">Baichuan2-7B</span></td>
<td id="T19.1.11.2" class="ltx_td ltx_align_left"><span id="T19.1.11.2.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">6.5B</span></td>
<td id="T19.1.11.3" class="ltx_td ltx_align_left"><span id="T19.1.11.3.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">38.63</span></td>
<td id="T19.1.11.4" class="ltx_td ltx_align_left"><span id="T19.1.11.4.1" class="ltx_text ltx_font_bold" style="font-size:80%;background-color:#FFFFFF;">62.33</span></td>
<td id="T19.1.11.5" class="ltx_td ltx_align_left"><span id="T19.1.11.5.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">47.84</span></td>
<td id="T19.1.11.6" class="ltx_td ltx_align_left"><span id="T19.1.11.6.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">23.01</span></td>
<td id="T19.1.11.7" class="ltx_td ltx_align_left"><span id="T19.1.11.7.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">37.46</span></td>
<td id="T19.1.11.8" class="ltx_td ltx_align_left"><span id="T19.1.11.8.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">22.51</span></td>
</tr>
<tr id="T19.1.12" class="ltx_tr" style="background-color:#E6F3F3;">
<td id="T19.1.12.1" class="ltx_td ltx_align_left"><span id="T19.1.12.1.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">ChatGLM3-6B</span></td>
<td id="T19.1.12.2" class="ltx_td ltx_align_left"><span id="T19.1.12.2.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">5.7B</span></td>
<td id="T19.1.12.3" class="ltx_td ltx_align_left"><span id="T19.1.12.3.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">40.96</span></td>
<td id="T19.1.12.4" class="ltx_td ltx_align_left"><span id="T19.1.12.4.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">43.38</span></td>
<td id="T19.1.12.5" class="ltx_td ltx_align_left"><span id="T19.1.12.5.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">50.03</span></td>
<td id="T19.1.12.6" class="ltx_td ltx_align_left"><span id="T19.1.12.6.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">33.17</span></td>
<td id="T19.1.12.7" class="ltx_td ltx_align_left"><span id="T19.1.12.7.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">49.87</span></td>
<td id="T19.1.12.8" class="ltx_td ltx_align_left"><span id="T19.1.12.8.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">28.36</span></td>
</tr>
<tr id="T19.1.13" class="ltx_tr" style="background-color:#FFFFFF;">
<td id="T19.1.13.1" class="ltx_td ltx_align_left"><span id="T19.1.13.1.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">Qwen1.5-4B-Chat</span></td>
<td id="T19.1.13.2" class="ltx_td ltx_align_left"><span id="T19.1.13.2.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">3.2B</span></td>
<td id="T19.1.13.3" class="ltx_td ltx_align_left"><span id="T19.1.13.3.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">33.30</span></td>
<td id="T19.1.13.4" class="ltx_td ltx_align_left"><span id="T19.1.13.4.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">24.64</span></td>
<td id="T19.1.13.5" class="ltx_td ltx_align_left"><span id="T19.1.13.5.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">37.72</span></td>
<td id="T19.1.13.6" class="ltx_td ltx_align_left"><span id="T19.1.13.6.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">29.38</span></td>
<td id="T19.1.13.7" class="ltx_td ltx_align_left"><span id="T19.1.13.7.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">44.74</span></td>
<td id="T19.1.13.8" class="ltx_td ltx_align_left"><span id="T19.1.13.8.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">30.00</span></td>
</tr>
<tr id="T19.1.14" class="ltx_tr" style="background-color:#E6F3F3;">
<td id="T19.1.14.1" class="ltx_td ltx_align_left"><span id="T19.1.14.1.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">Phi-2</span></td>
<td id="T19.1.14.2" class="ltx_td ltx_align_left"><span id="T19.1.14.2.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">2.5B</span></td>
<td id="T19.1.14.3" class="ltx_td ltx_align_left"><span id="T19.1.14.3.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">38.31</span></td>
<td id="T19.1.14.4" class="ltx_td ltx_align_left"><span id="T19.1.14.4.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">50.71</span></td>
<td id="T19.1.14.5" class="ltx_td ltx_align_left"><span id="T19.1.14.5.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">39.55</span></td>
<td id="T19.1.14.6" class="ltx_td ltx_align_left"><span id="T19.1.14.6.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">31.09</span></td>
<td id="T19.1.14.7" class="ltx_td ltx_align_left"><span id="T19.1.14.7.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">44.32</span></td>
<td id="T19.1.14.8" class="ltx_td ltx_align_left"><span id="T19.1.14.8.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">25.89</span></td>
</tr>
<tr id="T19.1.15" class="ltx_tr" style="background-color:#FFFFFF;">
<td id="T19.1.15.1" class="ltx_td ltx_align_left"><span id="T19.1.15.1.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">MiniCPM-SFT</span></td>
<td id="T19.1.15.2" class="ltx_td ltx_align_left"><span id="T19.1.15.2.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">2.4B</span></td>
<td id="T19.1.15.3" class="ltx_td ltx_align_left"><span id="T19.1.15.3.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">36.47</span></td>
<td id="T19.1.15.4" class="ltx_td ltx_align_left"><span id="T19.1.15.4.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">49.24</span></td>
<td id="T19.1.15.5" class="ltx_td ltx_align_left"><span id="T19.1.15.5.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">47.80</span></td>
<td id="T19.1.15.6" class="ltx_td ltx_align_left"><span id="T19.1.15.6.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">24.11</span></td>
<td id="T19.1.15.7" class="ltx_td ltx_align_left"><span id="T19.1.15.7.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">37.51</span></td>
<td id="T19.1.15.8" class="ltx_td ltx_align_left"><span id="T19.1.15.8.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">23.71</span></td>
</tr>
<tr id="T19.1.16" class="ltx_tr" style="background-color:#E6F3F3;">
<td id="T19.1.16.1" class="ltx_td ltx_align_left"><span id="T19.1.16.1.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">Gemma-2B</span></td>
<td id="T19.1.16.2" class="ltx_td ltx_align_left"><span id="T19.1.16.2.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">2.0B</span></td>
<td id="T19.1.16.3" class="ltx_td ltx_align_left"><span id="T19.1.16.3.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">38.04</span></td>
<td id="T19.1.16.4" class="ltx_td ltx_align_left"><span id="T19.1.16.4.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">53.41</span></td>
<td id="T19.1.16.5" class="ltx_td ltx_align_left"><span id="T19.1.16.5.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">52.22</span></td>
<td id="T19.1.16.6" class="ltx_td ltx_align_left"><span id="T19.1.16.6.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">24.60</span></td>
<td id="T19.1.16.7" class="ltx_td ltx_align_left"><span id="T19.1.16.7.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">39.78</span></td>
<td id="T19.1.16.8" class="ltx_td ltx_align_left"><span id="T19.1.16.8.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">20.18</span></td>
</tr>
<tr id="T19.1.17" class="ltx_tr" style="background-color:#FFFFFF;">
<td id="T19.1.17.1" class="ltx_td ltx_align_left"><span id="T19.1.17.1.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">Qwen1.5-1.8B-Chat</span></td>
<td id="T19.1.17.2" class="ltx_td ltx_align_left"><span id="T19.1.17.2.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">1.2B</span></td>
<td id="T19.1.17.3" class="ltx_td ltx_align_left"><span id="T19.1.17.3.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">37.52</span></td>
<td id="T19.1.17.4" class="ltx_td ltx_align_left"><span id="T19.1.17.4.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">47.18</span></td>
<td id="T19.1.17.5" class="ltx_td ltx_align_left"><span id="T19.1.17.5.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">52.11</span></td>
<td id="T19.1.17.6" class="ltx_td ltx_align_left"><span id="T19.1.17.6.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">26.68</span></td>
<td id="T19.1.17.7" class="ltx_td ltx_align_left"><span id="T19.1.17.7.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">40.57</span></td>
<td id="T19.1.17.8" class="ltx_td ltx_align_left"><span id="T19.1.17.8.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">21.05</span></td>
</tr>
<tr id="T19.1.1" class="ltx_tr" style="background-color:#E6F3F3;">
<td id="T19.1.1.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t"><span id="T19.1.1.1.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">Memory<sup id="T19.1.1.1.1.1" class="ltx_sup"><span id="T19.1.1.1.1.1.1" class="ltx_text" style="background-color:#E6F3F3;">3</span></sup>-SFT</span></td>
<td id="T19.1.1.2" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t"><span id="T19.1.1.2.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">2.4B</span></td>
<td id="T19.1.1.3" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t"><span id="T19.1.1.3.1" class="ltx_text ltx_font_bold" style="font-size:80%;background-color:#E6F3F3;">48.60</span></td>
<td id="T19.1.1.4" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t"><span id="T19.1.1.4.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">56.61</span></td>
<td id="T19.1.1.5" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t"><span id="T19.1.1.5.1" class="ltx_text ltx_font_bold" style="font-size:80%;background-color:#E6F3F3;">53.91</span></td>
<td id="T19.1.1.6" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t"><span id="T19.1.1.6.1" class="ltx_text ltx_font_bold" style="font-size:80%;background-color:#E6F3F3;">38.80</span></td>
<td id="T19.1.1.7" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t"><span id="T19.1.1.7.1" class="ltx_text ltx_font_bold" style="font-size:80%;background-color:#E6F3F3;">57.72</span></td>
<td id="T19.1.1.8" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t"><span id="T19.1.1.8.1" class="ltx_text ltx_font_bold" style="font-size:80%;background-color:#E6F3F3;">35.96</span></td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering" style="font-size:80%;"><span class="ltx_tag ltx_tag_table"><span id="T19.5.1.1" class="ltx_text" style="font-size:113%;">Table 19</span>: </span><span id="T19.6.2" class="ltx_text" style="font-size:113%;">Evaluation of hallucination.
HaluE and TruQA denote HaluEval and TruthfulQA, respectively.
Bolded numbers are the best results.
The model sizes only include non-embedding parameters.
Vicuna-13B-v1.5 gets one N/A since that entry is near zero and seems abnormal.</span></figcaption>
</figure>
</section>
<section id="S7.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsection" style="font-size:120%;color:#00868B;">7.4 <span id="S7.SS4.1.1" class="ltx_text" style="color:#808080;">| </span>Professional Tasks</h3>

<div id="S7.SS4.p1" class="ltx_para">
<p id="S7.SS4.p1.1" class="ltx_p">One benefit of using explicit memory is that the LLM can easily adapt to new fields and tasks by updating its knowledge base.
One can simply import task-related references into the knowledge base of Memory<sup id="S7.SS4.p1.1.1" class="ltx_sup">3</sup>, and optionally, convert them to explicit memories in the case of warm start.
Then, the model can perform inference with this new knowledge, skipping the more costly and possibly lossy process of finetuning, and running faster than RAG.
This cost reduction has been demonstrated in Figure <a href="#F4" title="Figure 4 ‣ 1 | Introduction ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> and Appendix <a href="#A1" title="Appendix A Cost Estimation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a>, and could facilitate the rapid deployment of LLMs across various industries.</p>
</div>
<div id="S7.SS4.p2" class="ltx_para">
<p id="S7.SS4.p2.1" class="ltx_p">Besides cost reduction, we need to demonstrate that Memory<sup id="S7.SS4.p2.1.1" class="ltx_sup">3</sup> can perform no worse than RAG.
We consider two professional tasks in law and medicine.
The legal task consists of multiple-choice questions from the Chinese National Judicial Examination (JEC-QA) dataset <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib134" title="" class="ltx_ref">134</a>]</cite>.
The field-specific references are legal documents from the Chinese national laws and regulations database <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib1" title="" class="ltx_ref">1</a>]</cite>.
These references are merged with our general-purpose knowledge base (Section <a href="#S4.SS4" title="4.4 | Knowledge Base ‣ 4 | Pretraining Data ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.4</span></a>) for inference.</p>
</div>
<div id="S7.SS4.p3" class="ltx_para">
<p id="S7.SS4.p3.1" class="ltx_p">The medical task consists of the medicine-related questions of C-Eval, MMLU and CMMLU, specifically from the following subsets:</p>
<ul id="S7.I1" class="ltx_itemize">
<li id="S7.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S7.I1.i1.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S7.I1.i1.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S7.I1.i1.1.1.m1.1.1" xref="S7.I1.i1.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S7.I1.i1.1.1.m1.1c"><ci id="S7.I1.i1.1.1.m1.1.1.cmml" xref="S7.I1.i1.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S7.I1.i1.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S7.I1.i1.p1" class="ltx_para">
<p id="S7.I1.i1.p1.1" class="ltx_p">C-Eval: clinical medicine, basic medicine</p>
</div>
</li>
<li id="S7.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S7.I1.i2.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S7.I1.i2.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S7.I1.i2.1.1.m1.1.1" xref="S7.I1.i2.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S7.I1.i2.1.1.m1.1c"><ci id="S7.I1.i2.1.1.m1.1.1.cmml" xref="S7.I1.i2.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S7.I1.i2.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S7.I1.i2.p1" class="ltx_para">
<p id="S7.I1.i2.p1.1" class="ltx_p">MMLU: clinical knowledge, anatomy, college medicine, college biology, nutrition, virology, medical genetics, professional medicine</p>
</div>
</li>
<li id="S7.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S7.I1.i3.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S7.I1.i3.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S7.I1.i3.1.1.m1.1.1" xref="S7.I1.i3.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S7.I1.i3.1.1.m1.1c"><ci id="S7.I1.i3.1.1.m1.1.1.cmml" xref="S7.I1.i3.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S7.I1.i3.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S7.I1.i3.p1" class="ltx_para">
<p id="S7.I1.i3.p1.1" class="ltx_p">CMMLU: anatomy, clinical knowledge, college medicine, genetics, nutrition, traditional Chinese medicine, virology</p>
</div>
</li>
</ul>
<p id="S7.SS4.p3.2" class="ltx_p">Our knowledge base is supplemented with medical texts from the open-source medical books dataset <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib101" title="" class="ltx_ref">101</a>]</cite>.</p>
</div>
<figure id="T20" class="ltx_table">
<table id="T20.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="T20.1.2" class="ltx_tr">
<td id="T20.1.2.1" class="ltx_td ltx_border_tt"></td>
<td id="T20.1.2.2" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="T20.1.2.2.1" class="ltx_text" style="font-size:90%;">JEC-QA</span></td>
<td id="T20.1.2.3" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="T20.1.2.3.1" class="ltx_text" style="font-size:90%;">MED</span></td>
</tr>
<tr id="T20.1.3" class="ltx_tr">
<td id="T20.1.3.1" class="ltx_td ltx_align_left"><span id="T20.1.3.1.1" class="ltx_text" style="font-size:90%;">LLM</span></td>
<td id="T20.1.3.2" class="ltx_td ltx_align_center ltx_border_t"><span id="T20.1.3.2.1" class="ltx_text" style="font-size:90%;">3 refs</span></td>
<td id="T20.1.3.3" class="ltx_td ltx_align_center ltx_border_t"><span id="T20.1.3.3.1" class="ltx_text" style="font-size:90%;">5 refs</span></td>
<td id="T20.1.3.4" class="ltx_td ltx_align_left ltx_border_t"><span id="T20.1.3.4.1" class="ltx_text" style="font-size:90%;">7 refs</span></td>
<td id="T20.1.3.5" class="ltx_td ltx_align_center ltx_border_t"><span id="T20.1.3.5.1" class="ltx_text" style="font-size:90%;">3 refs</span></td>
<td id="T20.1.3.6" class="ltx_td ltx_align_center ltx_border_t"><span id="T20.1.3.6.1" class="ltx_text" style="font-size:90%;">5 refs</span></td>
<td id="T20.1.3.7" class="ltx_td ltx_align_left ltx_border_t"><span id="T20.1.3.7.1" class="ltx_text" style="font-size:90%;">7 refs</span></td>
</tr>
<tr id="T20.1.1" class="ltx_tr">
<td id="T20.1.1.1" class="ltx_td ltx_align_left ltx_border_t">
<span id="T20.1.1.1.1" class="ltx_text" style="font-size:90%;">Memory</span><sup id="T20.1.1.1.2" class="ltx_sup"><span id="T20.1.1.1.2.1" class="ltx_text" style="font-size:90%;">3</span></sup><span id="T20.1.1.1.3" class="ltx_text" style="font-size:90%;">-2B-SFT</span>
</td>
<td id="T20.1.1.2" class="ltx_td ltx_align_center ltx_border_t" colspan="3"><span id="T20.1.1.2.1" class="ltx_text" style="font-size:90%;">39.38</span></td>
<td id="T20.1.1.3" class="ltx_td ltx_align_center ltx_border_t" colspan="3"><span id="T20.1.1.3.1" class="ltx_text" style="font-size:90%;">56.22</span></td>
</tr>
<tr id="T20.1.4" class="ltx_tr">
<td id="T20.1.4.1" class="ltx_td ltx_align_left"><span id="T20.1.4.1.1" class="ltx_text" style="font-size:90%;">MiniCPM-2B-SFT</span></td>
<td id="T20.1.4.2" class="ltx_td ltx_align_center"><span id="T20.1.4.2.1" class="ltx_text" style="font-size:90%;">38.83</span></td>
<td id="T20.1.4.3" class="ltx_td ltx_align_center"><span id="T20.1.4.3.1" class="ltx_text" style="font-size:90%;">37.65</span></td>
<td id="T20.1.4.4" class="ltx_td ltx_align_left"><span id="T20.1.4.4.1" class="ltx_text" style="font-size:90%;">37.94</span></td>
<td id="T20.1.4.5" class="ltx_td ltx_align_center"><span id="T20.1.4.5.1" class="ltx_text" style="font-size:90%;">53.73</span></td>
<td id="T20.1.4.6" class="ltx_td ltx_align_center"><span id="T20.1.4.6.1" class="ltx_text" style="font-size:90%;">53.29</span></td>
<td id="T20.1.4.7" class="ltx_td ltx_align_left"><span id="T20.1.4.7.1" class="ltx_text" style="font-size:90%;">52.84</span></td>
</tr>
<tr id="T20.1.5" class="ltx_tr">
<td id="T20.1.5.1" class="ltx_td ltx_align_left"><span id="T20.1.5.1.1" class="ltx_text" style="font-size:90%;">Gemma-2B</span></td>
<td id="T20.1.5.2" class="ltx_td ltx_align_center"><span id="T20.1.5.2.1" class="ltx_text" style="font-size:90%;">28.16</span></td>
<td id="T20.1.5.3" class="ltx_td ltx_align_center"><span id="T20.1.5.3.1" class="ltx_text" style="font-size:90%;">28.06</span></td>
<td id="T20.1.5.4" class="ltx_td ltx_align_left"><span id="T20.1.5.4.1" class="ltx_text" style="font-size:90%;">25.29</span></td>
<td id="T20.1.5.5" class="ltx_td ltx_align_center"><span id="T20.1.5.5.1" class="ltx_text" style="font-size:90%;">42.04</span></td>
<td id="T20.1.5.6" class="ltx_td ltx_align_center"><span id="T20.1.5.6.1" class="ltx_text" style="font-size:90%;">42.49</span></td>
<td id="T20.1.5.7" class="ltx_td ltx_align_left"><span id="T20.1.5.7.1" class="ltx_text" style="font-size:90%;">42.96</span></td>
</tr>
<tr id="T20.1.6" class="ltx_tr">
<td id="T20.1.6.1" class="ltx_td ltx_align_left"><span id="T20.1.6.1.1" class="ltx_text" style="font-size:90%;">Gemma-2B-it</span></td>
<td id="T20.1.6.2" class="ltx_td ltx_align_center"><span id="T20.1.6.2.1" class="ltx_text" style="font-size:90%;">30.04</span></td>
<td id="T20.1.6.3" class="ltx_td ltx_align_center"><span id="T20.1.6.3.1" class="ltx_text" style="font-size:90%;">31.13</span></td>
<td id="T20.1.6.4" class="ltx_td ltx_align_left"><span id="T20.1.6.4.1" class="ltx_text" style="font-size:90%;">29.34</span></td>
<td id="T20.1.6.5" class="ltx_td ltx_align_center"><span id="T20.1.6.5.1" class="ltx_text" style="font-size:90%;">41.70</span></td>
<td id="T20.1.6.6" class="ltx_td ltx_align_center"><span id="T20.1.6.6.1" class="ltx_text" style="font-size:90%;">43.24</span></td>
<td id="T20.1.6.7" class="ltx_td ltx_align_left"><span id="T20.1.6.7.1" class="ltx_text" style="font-size:90%;">42.66</span></td>
</tr>
<tr id="T20.1.7" class="ltx_tr">
<td id="T20.1.7.1" class="ltx_td ltx_align_left"><span id="T20.1.7.1.1" class="ltx_text" style="font-size:90%;">Llama-2-7B</span></td>
<td id="T20.1.7.2" class="ltx_td ltx_align_center"><span id="T20.1.7.2.1" class="ltx_text" style="font-size:90%;">28.06</span></td>
<td id="T20.1.7.3" class="ltx_td ltx_align_center"><span id="T20.1.7.3.1" class="ltx_text" style="font-size:90%;">24.70</span></td>
<td id="T20.1.7.4" class="ltx_td ltx_align_left"><span id="T20.1.7.4.1" class="ltx_text" style="font-size:90%;">24.90</span></td>
<td id="T20.1.7.5" class="ltx_td ltx_align_center"><span id="T20.1.7.5.1" class="ltx_text" style="font-size:90%;">45.14</span></td>
<td id="T20.1.7.6" class="ltx_td ltx_align_center"><span id="T20.1.7.6.1" class="ltx_text" style="font-size:90%;">44.43</span></td>
<td id="T20.1.7.7" class="ltx_td ltx_align_left"><span id="T20.1.7.7.1" class="ltx_text" style="font-size:90%;">37.96</span></td>
</tr>
<tr id="T20.1.8" class="ltx_tr">
<td id="T20.1.8.1" class="ltx_td ltx_align_left"><span id="T20.1.8.1.1" class="ltx_text" style="font-size:90%;">Llama-2-7B-Chat</span></td>
<td id="T20.1.8.2" class="ltx_td ltx_align_center"><span id="T20.1.8.2.1" class="ltx_text" style="font-size:90%;">26.18</span></td>
<td id="T20.1.8.3" class="ltx_td ltx_align_center"><span id="T20.1.8.3.1" class="ltx_text" style="font-size:90%;">25.10</span></td>
<td id="T20.1.8.4" class="ltx_td ltx_align_left"><span id="T20.1.8.4.1" class="ltx_text" style="font-size:90%;">25.20</span></td>
<td id="T20.1.8.5" class="ltx_td ltx_align_center"><span id="T20.1.8.5.1" class="ltx_text" style="font-size:90%;">48.18</span></td>
<td id="T20.1.8.6" class="ltx_td ltx_align_center"><span id="T20.1.8.6.1" class="ltx_text" style="font-size:90%;">47.29</span></td>
<td id="T20.1.8.7" class="ltx_td ltx_align_left"><span id="T20.1.8.7.1" class="ltx_text" style="font-size:90%;">39.39</span></td>
</tr>
<tr id="T20.1.9" class="ltx_tr">
<td id="T20.1.9.1" class="ltx_td ltx_align_left"><span id="T20.1.9.1.1" class="ltx_text" style="font-size:90%;">Phi-2</span></td>
<td id="T20.1.9.2" class="ltx_td ltx_align_center"><span id="T20.1.9.2.1" class="ltx_text" style="font-size:90%;">25.00</span></td>
<td id="T20.1.9.3" class="ltx_td ltx_align_center"><span id="T20.1.9.3.1" class="ltx_text" style="font-size:90%;">25.30</span></td>
<td id="T20.1.9.4" class="ltx_td ltx_align_left"><span id="T20.1.9.4.1" class="ltx_text" style="font-size:90%;">23.32</span></td>
<td id="T20.1.9.5" class="ltx_td ltx_align_center"><span id="T20.1.9.5.1" class="ltx_text" style="font-size:90%;">50.05</span></td>
<td id="T20.1.9.6" class="ltx_td ltx_align_center"><span id="T20.1.9.6.1" class="ltx_text" style="font-size:90%;">45.42</span></td>
<td id="T20.1.9.7" class="ltx_td ltx_align_left"><span id="T20.1.9.7.1" class="ltx_text" style="font-size:90%;">45.59</span></td>
</tr>
<tr id="T20.1.10" class="ltx_tr">
<td id="T20.1.10.1" class="ltx_td ltx_align_left"><span id="T20.1.10.1.1" class="ltx_text" style="font-size:90%;">Qwen1.5-1.8B-Chat</span></td>
<td id="T20.1.10.2" class="ltx_td ltx_align_center"><span id="T20.1.10.2.1" class="ltx_text" style="font-size:90%;">42.98</span></td>
<td id="T20.1.10.3" class="ltx_td ltx_align_center"><span id="T20.1.10.3.1" class="ltx_text" style="font-size:90%;">43.87</span></td>
<td id="T20.1.10.4" class="ltx_td ltx_align_left"><span id="T20.1.10.4.1" class="ltx_text" style="font-size:90%;">41.50</span></td>
<td id="T20.1.10.5" class="ltx_td ltx_align_center"><span id="T20.1.10.5.1" class="ltx_text" style="font-size:90%;">52.16</span></td>
<td id="T20.1.10.6" class="ltx_td ltx_align_center"><span id="T20.1.10.6.1" class="ltx_text" style="font-size:90%;">52.50</span></td>
<td id="T20.1.10.7" class="ltx_td ltx_align_left"><span id="T20.1.10.7.1" class="ltx_text" style="font-size:90%;">52.16</span></td>
</tr>
<tr id="T20.1.11" class="ltx_tr">
<td id="T20.1.11.1" class="ltx_td ltx_align_left ltx_border_bb"><span id="T20.1.11.1.1" class="ltx_text" style="font-size:90%;">Qwen1.5-4B-Chat</span></td>
<td id="T20.1.11.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="T20.1.11.2.1" class="ltx_text" style="font-size:90%;">51.98</span></td>
<td id="T20.1.11.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="T20.1.11.3.1" class="ltx_text" style="font-size:90%;">50.49</span></td>
<td id="T20.1.11.4" class="ltx_td ltx_align_left ltx_border_bb"><span id="T20.1.11.4.1" class="ltx_text" style="font-size:90%;">50.99</span></td>
<td id="T20.1.11.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="T20.1.11.5.1" class="ltx_text" style="font-size:90%;">61.19</span></td>
<td id="T20.1.11.6" class="ltx_td ltx_align_center ltx_border_bb"><span id="T20.1.11.6.1" class="ltx_text" style="font-size:90%;">61.02</span></td>
<td id="T20.1.11.7" class="ltx_td ltx_align_left ltx_border_bb"><span id="T20.1.11.7.1" class="ltx_text" style="font-size:90%;">61.06</span></td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering" style="font-size:90%;"><span class="ltx_tag ltx_tag_table">Table 20: </span>Comparison with RAG on professional tasks.</figcaption>
</figure>
<div id="S7.SS4.p4" class="ltx_para">
<p id="S7.SS4.p4.2" class="ltx_p">The results are shown in Table <a href="#T20" title="Table 20 ‣ 7.4 | Professional Tasks ‣ 7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">20</span></a>, and Memory<sup id="S7.SS4.p4.2.1" class="ltx_sup">3</sup> achieves better performance than most of the models.
All evaluations use 5-shot prompting.
The RAG models retrieve from the same knowledge bases and FAISS indices, except that they receive text references instead of explicit memories.
They only retrieve once for each question, using only the question text for query, so the 5-shot examples do not distract the retrieval.
Since the optimal number of references is not known for these RAG models, we test them for 3, 5, and 7 references per question, and it seems that <math id="S7.SS4.p4.2.m2.1" class="ltx_Math" alttext="3\sim 5" display="inline"><semantics id="S7.SS4.p4.2.m2.1a"><mrow id="S7.SS4.p4.2.m2.1.1" xref="S7.SS4.p4.2.m2.1.1.cmml"><mn id="S7.SS4.p4.2.m2.1.1.2" xref="S7.SS4.p4.2.m2.1.1.2.cmml">3</mn><mo id="S7.SS4.p4.2.m2.1.1.1" xref="S7.SS4.p4.2.m2.1.1.1.cmml">∼</mo><mn id="S7.SS4.p4.2.m2.1.1.3" xref="S7.SS4.p4.2.m2.1.1.3.cmml">5</mn></mrow><annotation-xml encoding="MathML-Content" id="S7.SS4.p4.2.m2.1b"><apply id="S7.SS4.p4.2.m2.1.1.cmml" xref="S7.SS4.p4.2.m2.1.1"><csymbol cd="latexml" id="S7.SS4.p4.2.m2.1.1.1.cmml" xref="S7.SS4.p4.2.m2.1.1.1">similar-to</csymbol><cn type="integer" id="S7.SS4.p4.2.m2.1.1.2.cmml" xref="S7.SS4.p4.2.m2.1.1.2">3</cn><cn type="integer" id="S7.SS4.p4.2.m2.1.1.3.cmml" xref="S7.SS4.p4.2.m2.1.1.3">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS4.p4.2.m2.1c">3\sim 5</annotation></semantics></math> references are optimal.
The usual formatting for RAG is used, i.e. header 1 + reference 1 + reference 2 + reference 3 + header 2 + few-shot examples + question, all separated by line breaks.</p>
</div>
<div id="S7.SS4.p5" class="ltx_para">
<p id="S7.SS4.p5.1" class="ltx_p">The performance plotted in Figure <a href="#F2" title="Figure 2 ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> (right) is the average of the scores of the two tasks in Table <a href="#T20" title="Table 20 ‣ 7.4 | Professional Tasks ‣ 7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">20</span></a> with five references.</p>
</div>
</section>
<section id="S7.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsection" style="font-size:120%;color:#00868B;">7.5 <span id="S7.SS5.1.1" class="ltx_text" style="color:#808080;">| </span>Inference Speed</h3>

<div id="S7.SS5.p1" class="ltx_para">
<p id="S7.SS5.p1.1" class="ltx_p">Finally, we evaluate the decoding speed or throughput of Memory<sup id="S7.SS5.p1.1.1" class="ltx_sup">3</sup>, measured by generated tokens per second.
The results are compared to those of RAG models, to quantify the speedup of explicit memory over text retrieval.</p>
</div>
<div id="S7.SS5.p2" class="ltx_para">
<p id="S7.SS5.p2.3" class="ltx_p">A direct comparison of speeds is uninformative:
The memory hierarchy (Figure <a href="#F8" title="Figure 8 ‣ 2.3 | Memory ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a>) implies that the Memory<sup id="S7.SS5.p2.3.1" class="ltx_sup">3</sup> model is more reliant on retrieval to supply knowledge, and naturally Memory<sup id="S7.SS5.p2.3.2" class="ltx_sup">3</sup> performs retrieval with higher frequency (<math id="S7.SS5.p2.3.m3.1" class="ltx_Math" alttext="5" display="inline"><semantics id="S7.SS5.p2.3.m3.1a"><mn id="S7.SS5.p2.3.m3.1.1" xref="S7.SS5.p2.3.m3.1.1.cmml">5</mn><annotation-xml encoding="MathML-Content" id="S7.SS5.p2.3.m3.1b"><cn type="integer" id="S7.SS5.p2.3.m3.1.1.cmml" xref="S7.SS5.p2.3.m3.1.1">5</cn></annotation-xml><annotation encoding="application/x-tex" id="S7.SS5.p2.3.m3.1c">5</annotation></semantics></math> references per 64 tokens, possibly higher in future versions).
Therefore, it is necessary to jointly compare performance and speed.
The speed measured in this section is plotted against the retrieval-augmented test accuracy from Section <a href="#S7.SS4" title="7.4 | Professional Tasks ‣ 7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7.4</span></a>, resulting in Figure <a href="#F2" title="Figure 2 ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> (right).</p>
</div>
<div id="S7.SS5.p3" class="ltx_para">
<p id="S7.SS5.p3.7" class="ltx_p">We measure decoding speed on a A800 GPU, and run all models with Flash Attention <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib32" title="" class="ltx_ref">32</a>]</cite>.
All models receive an input of batch size 32 and length 128 tokens, and generate an output with length 128 tokens.
The throughput is computed by <math id="S7.SS5.p3.1.m1.1" class="ltx_Math" alttext="32\times 128" display="inline"><semantics id="S7.SS5.p3.1.m1.1a"><mrow id="S7.SS5.p3.1.m1.1.1" xref="S7.SS5.p3.1.m1.1.1.cmml"><mn id="S7.SS5.p3.1.m1.1.1.2" xref="S7.SS5.p3.1.m1.1.1.2.cmml">32</mn><mo lspace="0.222em" rspace="0.222em" id="S7.SS5.p3.1.m1.1.1.1" xref="S7.SS5.p3.1.m1.1.1.1.cmml">×</mo><mn id="S7.SS5.p3.1.m1.1.1.3" xref="S7.SS5.p3.1.m1.1.1.3.cmml">128</mn></mrow><annotation-xml encoding="MathML-Content" id="S7.SS5.p3.1.m1.1b"><apply id="S7.SS5.p3.1.m1.1.1.cmml" xref="S7.SS5.p3.1.m1.1.1"><times id="S7.SS5.p3.1.m1.1.1.1.cmml" xref="S7.SS5.p3.1.m1.1.1.1"></times><cn type="integer" id="S7.SS5.p3.1.m1.1.1.2.cmml" xref="S7.SS5.p3.1.m1.1.1.2">32</cn><cn type="integer" id="S7.SS5.p3.1.m1.1.1.3.cmml" xref="S7.SS5.p3.1.m1.1.1.3">128</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS5.p3.1.m1.1c">32\times 128</annotation></semantics></math> divided by the time spent.
We test each model 9 times, remove the first record, and take the average of the rest.
Memory<sup id="S7.SS5.p3.7.1" class="ltx_sup">3</sup> performs <math id="S7.SS5.p3.3.m3.1" class="ltx_Math" alttext="2\times 128/64-1=3" display="inline"><semantics id="S7.SS5.p3.3.m3.1a"><mrow id="S7.SS5.p3.3.m3.1.1" xref="S7.SS5.p3.3.m3.1.1.cmml"><mrow id="S7.SS5.p3.3.m3.1.1.2" xref="S7.SS5.p3.3.m3.1.1.2.cmml"><mrow id="S7.SS5.p3.3.m3.1.1.2.2" xref="S7.SS5.p3.3.m3.1.1.2.2.cmml"><mrow id="S7.SS5.p3.3.m3.1.1.2.2.2" xref="S7.SS5.p3.3.m3.1.1.2.2.2.cmml"><mn id="S7.SS5.p3.3.m3.1.1.2.2.2.2" xref="S7.SS5.p3.3.m3.1.1.2.2.2.2.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="S7.SS5.p3.3.m3.1.1.2.2.2.1" xref="S7.SS5.p3.3.m3.1.1.2.2.2.1.cmml">×</mo><mn id="S7.SS5.p3.3.m3.1.1.2.2.2.3" xref="S7.SS5.p3.3.m3.1.1.2.2.2.3.cmml">128</mn></mrow><mo id="S7.SS5.p3.3.m3.1.1.2.2.1" xref="S7.SS5.p3.3.m3.1.1.2.2.1.cmml">/</mo><mn id="S7.SS5.p3.3.m3.1.1.2.2.3" xref="S7.SS5.p3.3.m3.1.1.2.2.3.cmml">64</mn></mrow><mo id="S7.SS5.p3.3.m3.1.1.2.1" xref="S7.SS5.p3.3.m3.1.1.2.1.cmml">−</mo><mn id="S7.SS5.p3.3.m3.1.1.2.3" xref="S7.SS5.p3.3.m3.1.1.2.3.cmml">1</mn></mrow><mo id="S7.SS5.p3.3.m3.1.1.1" xref="S7.SS5.p3.3.m3.1.1.1.cmml">=</mo><mn id="S7.SS5.p3.3.m3.1.1.3" xref="S7.SS5.p3.3.m3.1.1.3.cmml">3</mn></mrow><annotation-xml encoding="MathML-Content" id="S7.SS5.p3.3.m3.1b"><apply id="S7.SS5.p3.3.m3.1.1.cmml" xref="S7.SS5.p3.3.m3.1.1"><eq id="S7.SS5.p3.3.m3.1.1.1.cmml" xref="S7.SS5.p3.3.m3.1.1.1"></eq><apply id="S7.SS5.p3.3.m3.1.1.2.cmml" xref="S7.SS5.p3.3.m3.1.1.2"><minus id="S7.SS5.p3.3.m3.1.1.2.1.cmml" xref="S7.SS5.p3.3.m3.1.1.2.1"></minus><apply id="S7.SS5.p3.3.m3.1.1.2.2.cmml" xref="S7.SS5.p3.3.m3.1.1.2.2"><divide id="S7.SS5.p3.3.m3.1.1.2.2.1.cmml" xref="S7.SS5.p3.3.m3.1.1.2.2.1"></divide><apply id="S7.SS5.p3.3.m3.1.1.2.2.2.cmml" xref="S7.SS5.p3.3.m3.1.1.2.2.2"><times id="S7.SS5.p3.3.m3.1.1.2.2.2.1.cmml" xref="S7.SS5.p3.3.m3.1.1.2.2.2.1"></times><cn type="integer" id="S7.SS5.p3.3.m3.1.1.2.2.2.2.cmml" xref="S7.SS5.p3.3.m3.1.1.2.2.2.2">2</cn><cn type="integer" id="S7.SS5.p3.3.m3.1.1.2.2.2.3.cmml" xref="S7.SS5.p3.3.m3.1.1.2.2.2.3">128</cn></apply><cn type="integer" id="S7.SS5.p3.3.m3.1.1.2.2.3.cmml" xref="S7.SS5.p3.3.m3.1.1.2.2.3">64</cn></apply><cn type="integer" id="S7.SS5.p3.3.m3.1.1.2.3.cmml" xref="S7.SS5.p3.3.m3.1.1.2.3">1</cn></apply><cn type="integer" id="S7.SS5.p3.3.m3.1.1.3.cmml" xref="S7.SS5.p3.3.m3.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS5.p3.3.m3.1c">2\times 128/64-1=3</annotation></semantics></math> retrievals (the <math id="S7.SS5.p3.4.m4.1" class="ltx_Math" alttext="-1" display="inline"><semantics id="S7.SS5.p3.4.m4.1a"><mrow id="S7.SS5.p3.4.m4.1.1" xref="S7.SS5.p3.4.m4.1.1.cmml"><mo id="S7.SS5.p3.4.m4.1.1a" xref="S7.SS5.p3.4.m4.1.1.cmml">−</mo><mn id="S7.SS5.p3.4.m4.1.1.2" xref="S7.SS5.p3.4.m4.1.1.2.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S7.SS5.p3.4.m4.1b"><apply id="S7.SS5.p3.4.m4.1.1.cmml" xref="S7.SS5.p3.4.m4.1.1"><minus id="S7.SS5.p3.4.m4.1.1.1.cmml" xref="S7.SS5.p3.4.m4.1.1"></minus><cn type="integer" id="S7.SS5.p3.4.m4.1.1.2.cmml" xref="S7.SS5.p3.4.m4.1.1.2">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS5.p3.4.m4.1c">-1</annotation></semantics></math> means that the first decoded chunk inherits the explicit memories retrieved by the last input chunk).
Each retrieval uses 32 queries to get <math id="S7.SS5.p3.5.m5.1" class="ltx_Math" alttext="32\times 5" display="inline"><semantics id="S7.SS5.p3.5.m5.1a"><mrow id="S7.SS5.p3.5.m5.1.1" xref="S7.SS5.p3.5.m5.1.1.cmml"><mn id="S7.SS5.p3.5.m5.1.1.2" xref="S7.SS5.p3.5.m5.1.1.2.cmml">32</mn><mo lspace="0.222em" rspace="0.222em" id="S7.SS5.p3.5.m5.1.1.1" xref="S7.SS5.p3.5.m5.1.1.1.cmml">×</mo><mn id="S7.SS5.p3.5.m5.1.1.3" xref="S7.SS5.p3.5.m5.1.1.3.cmml">5</mn></mrow><annotation-xml encoding="MathML-Content" id="S7.SS5.p3.5.m5.1b"><apply id="S7.SS5.p3.5.m5.1.1.cmml" xref="S7.SS5.p3.5.m5.1.1"><times id="S7.SS5.p3.5.m5.1.1.1.cmml" xref="S7.SS5.p3.5.m5.1.1.1"></times><cn type="integer" id="S7.SS5.p3.5.m5.1.1.2.cmml" xref="S7.SS5.p3.5.m5.1.1.2">32</cn><cn type="integer" id="S7.SS5.p3.5.m5.1.1.3.cmml" xref="S7.SS5.p3.5.m5.1.1.3">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS5.p3.5.m5.1c">32\times 5</annotation></semantics></math> explicit memories.
We consider the warm start scenario, with the explicit memories precomputed and saved to drives.
We implement the worst case scenario, such that the reference ids are reset to be unique after vector search and the memory cache on RAM is disabled, forcing Memory<sup id="S7.SS5.p3.7.2" class="ltx_sup">3</sup> to load <math id="S7.SS5.p3.7.m7.1" class="ltx_Math" alttext="32\times 5" display="inline"><semantics id="S7.SS5.p3.7.m7.1a"><mrow id="S7.SS5.p3.7.m7.1.1" xref="S7.SS5.p3.7.m7.1.1.cmml"><mn id="S7.SS5.p3.7.m7.1.1.2" xref="S7.SS5.p3.7.m7.1.1.2.cmml">32</mn><mo lspace="0.222em" rspace="0.222em" id="S7.SS5.p3.7.m7.1.1.1" xref="S7.SS5.p3.7.m7.1.1.1.cmml">×</mo><mn id="S7.SS5.p3.7.m7.1.1.3" xref="S7.SS5.p3.7.m7.1.1.3.cmml">5</mn></mrow><annotation-xml encoding="MathML-Content" id="S7.SS5.p3.7.m7.1b"><apply id="S7.SS5.p3.7.m7.1.1.cmml" xref="S7.SS5.p3.7.m7.1.1"><times id="S7.SS5.p3.7.m7.1.1.1.cmml" xref="S7.SS5.p3.7.m7.1.1.1"></times><cn type="integer" id="S7.SS5.p3.7.m7.1.1.2.cmml" xref="S7.SS5.p3.7.m7.1.1.2">32</cn><cn type="integer" id="S7.SS5.p3.7.m7.1.1.3.cmml" xref="S7.SS5.p3.7.m7.1.1.3">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS5.p3.7.m7.1c">32\times 5</annotation></semantics></math> memories from drives.
Meanwhile, each RAG model performs one retrieval with query length 64 tokens, receives 5 references for each sample, and inserts them at the beginning of the sample, similar to the setup for Table <a href="#T20" title="Table 20 ‣ 7.4 | Professional Tasks ‣ 7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">20</span></a>.</p>
</div>
<div id="S7.SS5.p4" class="ltx_para">
<p id="S7.SS5.p4.1" class="ltx_p">The results are listed in Table <a href="#T21" title="Table 21 ‣ 7.5 | Inference Speed ‣ 7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">21</span></a> (local server).
The throughput of these models without retrieval is also provided.</p>
</div>
<figure id="T21" class="ltx_table">
<table id="T21.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="T21.1.2" class="ltx_tr">
<td id="T21.1.2.1" class="ltx_td ltx_border_tt"></td>
<td id="T21.1.2.2" class="ltx_td ltx_border_tt"></td>
<td id="T21.1.2.3" class="ltx_td ltx_align_center ltx_border_tt" colspan="2"><span id="T21.1.2.3.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Local server</span></td>
<td id="T21.1.2.4" class="ltx_td ltx_align_center ltx_border_tt" colspan="2"><span id="T21.1.2.4.1" class="ltx_text ltx_font_bold" style="font-size:80%;">End-side device</span></td>
</tr>
<tr id="T21.1.3" class="ltx_tr">
<td id="T21.1.3.1" class="ltx_td ltx_align_left"><span id="T21.1.3.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">LLM</span></td>
<td id="T21.1.3.2" class="ltx_td ltx_align_center"><span id="T21.1.3.2.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Size</span></td>
<td id="T21.1.3.3" class="ltx_td ltx_align_center ltx_border_t"><span id="T21.1.3.3.1" class="ltx_text" style="font-size:80%;">with retrieval</span></td>
<td id="T21.1.3.4" class="ltx_td ltx_align_center ltx_border_t"><span id="T21.1.3.4.1" class="ltx_text" style="font-size:80%;">w/o retrieval</span></td>
<td id="T21.1.3.5" class="ltx_td ltx_align_center ltx_border_t"><span id="T21.1.3.5.1" class="ltx_text" style="font-size:80%;">with retrieval</span></td>
<td id="T21.1.3.6" class="ltx_td ltx_align_center ltx_border_t"><span id="T21.1.3.6.1" class="ltx_text" style="font-size:80%;">w/o retrieval</span></td>
</tr>
<tr id="T21.1.1" class="ltx_tr">
<td id="T21.1.1.1" class="ltx_td ltx_align_left ltx_border_t">
<span id="T21.1.1.1.1" class="ltx_text" style="font-size:80%;">Memory</span><sup id="T21.1.1.1.2" class="ltx_sup"><span id="T21.1.1.1.2.1" class="ltx_text" style="font-size:80%;">3</span></sup><span id="T21.1.1.1.3" class="ltx_text" style="font-size:80%;">-2B</span>
</td>
<td id="T21.1.1.2" class="ltx_td ltx_align_center ltx_border_t"><span id="T21.1.1.2.1" class="ltx_text" style="font-size:80%;">2.4B</span></td>
<td id="T21.1.1.3" class="ltx_td ltx_align_center ltx_border_t"><span id="T21.1.1.3.1" class="ltx_text" style="font-size:80%;">733.0</span></td>
<td id="T21.1.1.4" class="ltx_td ltx_align_center ltx_border_t"><span id="T21.1.1.4.1" class="ltx_text" style="font-size:80%;">1131</span></td>
<td id="T21.1.1.5" class="ltx_td ltx_align_center ltx_border_t"><span id="T21.1.1.5.1" class="ltx_text" style="font-size:80%;">27.6</span></td>
<td id="T21.1.1.6" class="ltx_td ltx_align_center ltx_border_t"><span id="T21.1.1.6.1" class="ltx_text" style="font-size:80%;">44.36</span></td>
</tr>
<tr id="T21.1.4" class="ltx_tr">
<td id="T21.1.4.1" class="ltx_td ltx_align_left"><span id="T21.1.4.1.1" class="ltx_text" style="font-size:80%;">MiniCPM-2B</span></td>
<td id="T21.1.4.2" class="ltx_td ltx_align_center"><span id="T21.1.4.2.1" class="ltx_text" style="font-size:80%;">2.4B</span></td>
<td id="T21.1.4.3" class="ltx_td ltx_align_center"><span id="T21.1.4.3.1" class="ltx_text" style="font-size:80%;">501.5</span></td>
<td id="T21.1.4.4" class="ltx_td ltx_align_center"><span id="T21.1.4.4.1" class="ltx_text" style="font-size:80%;">974.0</span></td>
<td id="T21.1.4.5" class="ltx_td ltx_align_center"><span id="T21.1.4.5.1" class="ltx_text" style="font-size:80%;">21.7</span></td>
<td id="T21.1.4.6" class="ltx_td ltx_align_center"><span id="T21.1.4.6.1" class="ltx_text" style="font-size:80%;">51.79</span></td>
</tr>
<tr id="T21.1.5" class="ltx_tr">
<td id="T21.1.5.1" class="ltx_td ltx_align_left"><span id="T21.1.5.1.1" class="ltx_text" style="font-size:80%;">Gemma-2B-it</span></td>
<td id="T21.1.5.2" class="ltx_td ltx_align_center"><span id="T21.1.5.2.1" class="ltx_text" style="font-size:80%;">2.0B</span></td>
<td id="T21.1.5.3" class="ltx_td ltx_align_center"><span id="T21.1.5.3.1" class="ltx_text" style="font-size:80%;">1581</span></td>
<td id="T21.1.5.4" class="ltx_td ltx_align_center"><span id="T21.1.5.4.1" class="ltx_text" style="font-size:80%;">2056</span></td>
<td id="T21.1.5.5" class="ltx_td ltx_align_center"><span id="T21.1.5.5.1" class="ltx_text" style="font-size:80%;">22.0</span></td>
<td id="T21.1.5.6" class="ltx_td ltx_align_center"><span id="T21.1.5.6.1" class="ltx_text" style="font-size:80%;">29.23</span></td>
</tr>
<tr id="T21.1.6" class="ltx_tr">
<td id="T21.1.6.1" class="ltx_td ltx_align_left"><span id="T21.1.6.1.1" class="ltx_text" style="font-size:80%;">Gemma-7B-it</span></td>
<td id="T21.1.6.2" class="ltx_td ltx_align_center"><span id="T21.1.6.2.1" class="ltx_text" style="font-size:80%;">7.8B</span></td>
<td id="T21.1.6.3" class="ltx_td ltx_align_center"><span id="T21.1.6.3.1" class="ltx_text" style="font-size:80%;">395.6</span></td>
<td id="T21.1.6.4" class="ltx_td ltx_align_center"><span id="T21.1.6.4.1" class="ltx_text" style="font-size:80%;">1008</span></td>
<td id="T21.1.6.5" class="ltx_td ltx_align_center"><span id="T21.1.6.5.1" class="ltx_text" style="font-size:80%;">9.5</span></td>
<td id="T21.1.6.6" class="ltx_td ltx_align_center"><span id="T21.1.6.6.1" class="ltx_text" style="font-size:80%;">18.61</span></td>
</tr>
<tr id="T21.1.7" class="ltx_tr">
<td id="T21.1.7.1" class="ltx_td ltx_align_left"><span id="T21.1.7.1.1" class="ltx_text" style="font-size:80%;">Mistral-7B-Instruct-v0.1</span></td>
<td id="T21.1.7.2" class="ltx_td ltx_align_center"><span id="T21.1.7.2.1" class="ltx_text" style="font-size:80%;">7.0B</span></td>
<td id="T21.1.7.3" class="ltx_td ltx_align_center"><span id="T21.1.7.3.1" class="ltx_text" style="font-size:80%;">392.9</span></td>
<td id="T21.1.7.4" class="ltx_td ltx_align_center"><span id="T21.1.7.4.1" class="ltx_text" style="font-size:80%;">894.5</span></td>
<td id="T21.1.7.5" class="ltx_td ltx_align_center"><span id="T21.1.7.5.1" class="ltx_text" style="font-size:80%;">11.1</span></td>
<td id="T21.1.7.6" class="ltx_td ltx_align_center"><span id="T21.1.7.6.1" class="ltx_text" style="font-size:80%;">28.7</span></td>
</tr>
<tr id="T21.1.8" class="ltx_tr">
<td id="T21.1.8.1" class="ltx_td ltx_align_left"><span id="T21.1.8.1.1" class="ltx_text" style="font-size:80%;">Llama-2-7B-Chat</span></td>
<td id="T21.1.8.2" class="ltx_td ltx_align_center"><span id="T21.1.8.2.1" class="ltx_text" style="font-size:80%;">6.5B</span></td>
<td id="T21.1.8.3" class="ltx_td ltx_align_center"><span id="T21.1.8.3.1" class="ltx_text" style="font-size:80%;">382.8</span></td>
<td id="T21.1.8.4" class="ltx_td ltx_align_center"><span id="T21.1.8.4.1" class="ltx_text" style="font-size:80%;">1005</span></td>
<td id="T21.1.8.5" class="ltx_td ltx_align_center"><span id="T21.1.8.5.1" class="ltx_text" style="font-size:80%;">10.0</span></td>
<td id="T21.1.8.6" class="ltx_td ltx_align_center"><span id="T21.1.8.6.1" class="ltx_text" style="font-size:80%;">23.19</span></td>
</tr>
<tr id="T21.1.9" class="ltx_tr">
<td id="T21.1.9.1" class="ltx_td ltx_align_left"><span id="T21.1.9.1.1" class="ltx_text" style="font-size:80%;">Llama-2-13B-Chat</span></td>
<td id="T21.1.9.2" class="ltx_td ltx_align_center"><span id="T21.1.9.2.1" class="ltx_text" style="font-size:80%;">13B</span></td>
<td id="T21.1.9.3" class="ltx_td ltx_align_center"><span id="T21.1.9.3.1" class="ltx_text" style="font-size:80%;">241.1</span></td>
<td id="T21.1.9.4" class="ltx_td ltx_align_center"><span id="T21.1.9.4.1" class="ltx_text" style="font-size:80%;">632.5</span></td>
<td id="T21.1.9.5" class="ltx_td ltx_align_center"><span id="T21.1.9.5.1" class="ltx_text" style="font-size:80%;">2.5</span></td>
<td id="T21.1.9.6" class="ltx_td ltx_align_center"><span id="T21.1.9.6.1" class="ltx_text" style="font-size:80%;">5.44</span></td>
</tr>
<tr id="T21.1.10" class="ltx_tr">
<td id="T21.1.10.1" class="ltx_td ltx_align_left"><span id="T21.1.10.1.1" class="ltx_text" style="font-size:80%;">Qwen1.5-1.8B-Chat</span></td>
<td id="T21.1.10.2" class="ltx_td ltx_align_center"><span id="T21.1.10.2.1" class="ltx_text" style="font-size:80%;">1.2B</span></td>
<td id="T21.1.10.3" class="ltx_td ltx_align_center"><span id="T21.1.10.3.1" class="ltx_text" style="font-size:80%;">908.2</span></td>
<td id="T21.1.10.4" class="ltx_td ltx_align_center"><span id="T21.1.10.4.1" class="ltx_text" style="font-size:80%;">1770</span></td>
<td id="T21.1.10.5" class="ltx_td ltx_align_center"><span id="T21.1.10.5.1" class="ltx_text" style="font-size:80%;">-</span></td>
<td id="T21.1.10.6" class="ltx_td ltx_align_center"><span id="T21.1.10.6.1" class="ltx_text" style="font-size:80%;">-</span></td>
</tr>
<tr id="T21.1.11" class="ltx_tr">
<td id="T21.1.11.1" class="ltx_td ltx_align_left"><span id="T21.1.11.1.1" class="ltx_text" style="font-size:80%;">Qwen1.5-4B-Chat</span></td>
<td id="T21.1.11.2" class="ltx_td ltx_align_center"><span id="T21.1.11.2.1" class="ltx_text" style="font-size:80%;">3.2B</span></td>
<td id="T21.1.11.3" class="ltx_td ltx_align_center"><span id="T21.1.11.3.1" class="ltx_text" style="font-size:80%;">460.7</span></td>
<td id="T21.1.11.4" class="ltx_td ltx_align_center"><span id="T21.1.11.4.1" class="ltx_text" style="font-size:80%;">1002</span></td>
<td id="T21.1.11.5" class="ltx_td ltx_align_center"><span id="T21.1.11.5.1" class="ltx_text" style="font-size:80%;">22.3</span></td>
<td id="T21.1.11.6" class="ltx_td ltx_align_center"><span id="T21.1.11.6.1" class="ltx_text" style="font-size:80%;">53.39</span></td>
</tr>
<tr id="T21.1.12" class="ltx_tr">
<td id="T21.1.12.1" class="ltx_td ltx_align_left"><span id="T21.1.12.1.1" class="ltx_text" style="font-size:80%;">Qwen1.5-7B-Chat</span></td>
<td id="T21.1.12.2" class="ltx_td ltx_align_center"><span id="T21.1.12.2.1" class="ltx_text" style="font-size:80%;">6.5B</span></td>
<td id="T21.1.12.3" class="ltx_td ltx_align_center"><span id="T21.1.12.3.1" class="ltx_text" style="font-size:80%;">365.8</span></td>
<td id="T21.1.12.4" class="ltx_td ltx_align_center"><span id="T21.1.12.4.1" class="ltx_text" style="font-size:80%;">894.5</span></td>
<td id="T21.1.12.5" class="ltx_td ltx_align_center"><span id="T21.1.12.5.1" class="ltx_text" style="font-size:80%;">-</span></td>
<td id="T21.1.12.6" class="ltx_td ltx_align_center"><span id="T21.1.12.6.1" class="ltx_text" style="font-size:80%;">-</span></td>
</tr>
<tr id="T21.1.13" class="ltx_tr">
<td id="T21.1.13.1" class="ltx_td ltx_align_left ltx_border_bb"><span id="T21.1.13.1.1" class="ltx_text" style="font-size:80%;">Phi-2</span></td>
<td id="T21.1.13.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="T21.1.13.2.1" class="ltx_text" style="font-size:80%;">2.5B</span></td>
<td id="T21.1.13.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="T21.1.13.3.1" class="ltx_text" style="font-size:80%;">622.2</span></td>
<td id="T21.1.13.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="T21.1.13.4.1" class="ltx_text" style="font-size:80%;">1544</span></td>
<td id="T21.1.13.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="T21.1.13.5.1" class="ltx_text" style="font-size:80%;">-</span></td>
<td id="T21.1.13.6" class="ltx_td ltx_align_center ltx_border_bb"><span id="T21.1.13.6.1" class="ltx_text" style="font-size:80%;">-</span></td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering" style="font-size:80%;"><span class="ltx_tag ltx_tag_table"><span id="T21.5.1.1" class="ltx_text" style="font-size:113%;">Table 21</span>: </span><span id="T21.6.2" class="ltx_text" style="font-size:113%;">Inference throughput, measured by tokens per second.</span></figcaption>
</figure>
<div id="S7.SS5.p5" class="ltx_para">
<p id="S7.SS5.p5.1" class="ltx_p">In addition, we study the throughput of these models when they are hosted on an end-side device and retrieve from a knowledge base on a remote server.
Specifically, we use Jetson AGX Orin, and the server uses the vector engine MyScale <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib82" title="" class="ltx_ref">82</a>]</cite>.
The models are run with plain attention, with batch size 1.
To simulate real-world use cases, the input is a fixed text prompt, with approximately 128 tokens, while the exact length can vary among different tokenizers.
The output length is fixed to be 128 tokens.
The results are listed in Table <a href="#T21" title="Table 21 ‣ 7.5 | Inference Speed ‣ 7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">21</span></a> (end-side device), and the Memory<sup id="S7.SS5.p5.1.1" class="ltx_sup">3</sup> model .</p>
</div>
<div id="Thmremark8" class="ltx_theorem ltx_theorem_remark">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmremark8.1.1.1" class="ltx_text ltx_font_bold">Remark 8</span></span><span id="Thmremark8.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmremark8.p1" class="ltx_para">
<p id="Thmremark8.p1.2" class="ltx_p">Table <a href="#T21" title="Table 21 ‣ 7.5 | Inference Speed ‣ 7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">21</span></a> indicates that our Memory<sup id="Thmremark8.p1.2.1" class="ltx_sup">3</sup>-2B model is <math id="Thmremark8.p1.2.m2.1" class="ltx_Math" alttext="1-733/1131\approx 35.2\%" display="inline"><semantics id="Thmremark8.p1.2.m2.1a"><mrow id="Thmremark8.p1.2.m2.1.1" xref="Thmremark8.p1.2.m2.1.1.cmml"><mrow id="Thmremark8.p1.2.m2.1.1.2" xref="Thmremark8.p1.2.m2.1.1.2.cmml"><mn id="Thmremark8.p1.2.m2.1.1.2.2" xref="Thmremark8.p1.2.m2.1.1.2.2.cmml">1</mn><mo id="Thmremark8.p1.2.m2.1.1.2.1" xref="Thmremark8.p1.2.m2.1.1.2.1.cmml">−</mo><mrow id="Thmremark8.p1.2.m2.1.1.2.3" xref="Thmremark8.p1.2.m2.1.1.2.3.cmml"><mn id="Thmremark8.p1.2.m2.1.1.2.3.2" xref="Thmremark8.p1.2.m2.1.1.2.3.2.cmml">733</mn><mo id="Thmremark8.p1.2.m2.1.1.2.3.1" xref="Thmremark8.p1.2.m2.1.1.2.3.1.cmml">/</mo><mn id="Thmremark8.p1.2.m2.1.1.2.3.3" xref="Thmremark8.p1.2.m2.1.1.2.3.3.cmml">1131</mn></mrow></mrow><mo id="Thmremark8.p1.2.m2.1.1.1" xref="Thmremark8.p1.2.m2.1.1.1.cmml">≈</mo><mrow id="Thmremark8.p1.2.m2.1.1.3" xref="Thmremark8.p1.2.m2.1.1.3.cmml"><mn id="Thmremark8.p1.2.m2.1.1.3.2" xref="Thmremark8.p1.2.m2.1.1.3.2.cmml">35.2</mn><mo id="Thmremark8.p1.2.m2.1.1.3.1" xref="Thmremark8.p1.2.m2.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="Thmremark8.p1.2.m2.1b"><apply id="Thmremark8.p1.2.m2.1.1.cmml" xref="Thmremark8.p1.2.m2.1.1"><approx id="Thmremark8.p1.2.m2.1.1.1.cmml" xref="Thmremark8.p1.2.m2.1.1.1"></approx><apply id="Thmremark8.p1.2.m2.1.1.2.cmml" xref="Thmremark8.p1.2.m2.1.1.2"><minus id="Thmremark8.p1.2.m2.1.1.2.1.cmml" xref="Thmremark8.p1.2.m2.1.1.2.1"></minus><cn type="integer" id="Thmremark8.p1.2.m2.1.1.2.2.cmml" xref="Thmremark8.p1.2.m2.1.1.2.2">1</cn><apply id="Thmremark8.p1.2.m2.1.1.2.3.cmml" xref="Thmremark8.p1.2.m2.1.1.2.3"><divide id="Thmremark8.p1.2.m2.1.1.2.3.1.cmml" xref="Thmremark8.p1.2.m2.1.1.2.3.1"></divide><cn type="integer" id="Thmremark8.p1.2.m2.1.1.2.3.2.cmml" xref="Thmremark8.p1.2.m2.1.1.2.3.2">733</cn><cn type="integer" id="Thmremark8.p1.2.m2.1.1.2.3.3.cmml" xref="Thmremark8.p1.2.m2.1.1.2.3.3">1131</cn></apply></apply><apply id="Thmremark8.p1.2.m2.1.1.3.cmml" xref="Thmremark8.p1.2.m2.1.1.3"><csymbol cd="latexml" id="Thmremark8.p1.2.m2.1.1.3.1.cmml" xref="Thmremark8.p1.2.m2.1.1.3.1">percent</csymbol><cn type="float" id="Thmremark8.p1.2.m2.1.1.3.2.cmml" xref="Thmremark8.p1.2.m2.1.1.3.2">35.2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmremark8.p1.2.m2.1c">1-733/1131\approx 35.2\%</annotation></semantics></math> slower than the same model without using memory.
This is peculiar considering that reading explicit memories accounts for only a tiny fraction of the total compute:</p>
<table id="Ex7" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="Ex7.m1.1" class="ltx_Math" alttext="\frac{2.884\times 10^{-3}~{}\text{TFlops}}{1.264~{}\text{TFlops}}\approx 0.228\%" display="block"><semantics id="Ex7.m1.1a"><mrow id="Ex7.m1.1.1" xref="Ex7.m1.1.1.cmml"><mfrac id="Ex7.m1.1.1.2" xref="Ex7.m1.1.1.2.cmml"><mrow id="Ex7.m1.1.1.2.2" xref="Ex7.m1.1.1.2.2.cmml"><mrow id="Ex7.m1.1.1.2.2.2" xref="Ex7.m1.1.1.2.2.2.cmml"><mn id="Ex7.m1.1.1.2.2.2.2" xref="Ex7.m1.1.1.2.2.2.2.cmml">2.884</mn><mo lspace="0.222em" rspace="0.222em" id="Ex7.m1.1.1.2.2.2.1" xref="Ex7.m1.1.1.2.2.2.1.cmml">×</mo><msup id="Ex7.m1.1.1.2.2.2.3" xref="Ex7.m1.1.1.2.2.2.3.cmml"><mn id="Ex7.m1.1.1.2.2.2.3.2" xref="Ex7.m1.1.1.2.2.2.3.2.cmml">10</mn><mrow id="Ex7.m1.1.1.2.2.2.3.3" xref="Ex7.m1.1.1.2.2.2.3.3.cmml"><mo id="Ex7.m1.1.1.2.2.2.3.3a" xref="Ex7.m1.1.1.2.2.2.3.3.cmml">−</mo><mn id="Ex7.m1.1.1.2.2.2.3.3.2" xref="Ex7.m1.1.1.2.2.2.3.3.2.cmml">3</mn></mrow></msup></mrow><mo lspace="0em" rspace="0em" id="Ex7.m1.1.1.2.2.1" xref="Ex7.m1.1.1.2.2.1.cmml">​</mo><mtext id="Ex7.m1.1.1.2.2.3" xref="Ex7.m1.1.1.2.2.3a.cmml">TFlops</mtext></mrow><mrow id="Ex7.m1.1.1.2.3" xref="Ex7.m1.1.1.2.3.cmml"><mn id="Ex7.m1.1.1.2.3.2" xref="Ex7.m1.1.1.2.3.2.cmml">1.264</mn><mo lspace="0.330em" rspace="0em" id="Ex7.m1.1.1.2.3.1" xref="Ex7.m1.1.1.2.3.1.cmml">​</mo><mtext id="Ex7.m1.1.1.2.3.3" xref="Ex7.m1.1.1.2.3.3a.cmml">TFlops</mtext></mrow></mfrac><mo id="Ex7.m1.1.1.1" xref="Ex7.m1.1.1.1.cmml">≈</mo><mrow id="Ex7.m1.1.1.3" xref="Ex7.m1.1.1.3.cmml"><mn id="Ex7.m1.1.1.3.2" xref="Ex7.m1.1.1.3.2.cmml">0.228</mn><mo id="Ex7.m1.1.1.3.1" xref="Ex7.m1.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="Ex7.m1.1b"><apply id="Ex7.m1.1.1.cmml" xref="Ex7.m1.1.1"><approx id="Ex7.m1.1.1.1.cmml" xref="Ex7.m1.1.1.1"></approx><apply id="Ex7.m1.1.1.2.cmml" xref="Ex7.m1.1.1.2"><divide id="Ex7.m1.1.1.2.1.cmml" xref="Ex7.m1.1.1.2"></divide><apply id="Ex7.m1.1.1.2.2.cmml" xref="Ex7.m1.1.1.2.2"><times id="Ex7.m1.1.1.2.2.1.cmml" xref="Ex7.m1.1.1.2.2.1"></times><apply id="Ex7.m1.1.1.2.2.2.cmml" xref="Ex7.m1.1.1.2.2.2"><times id="Ex7.m1.1.1.2.2.2.1.cmml" xref="Ex7.m1.1.1.2.2.2.1"></times><cn type="float" id="Ex7.m1.1.1.2.2.2.2.cmml" xref="Ex7.m1.1.1.2.2.2.2">2.884</cn><apply id="Ex7.m1.1.1.2.2.2.3.cmml" xref="Ex7.m1.1.1.2.2.2.3"><csymbol cd="ambiguous" id="Ex7.m1.1.1.2.2.2.3.1.cmml" xref="Ex7.m1.1.1.2.2.2.3">superscript</csymbol><cn type="integer" id="Ex7.m1.1.1.2.2.2.3.2.cmml" xref="Ex7.m1.1.1.2.2.2.3.2">10</cn><apply id="Ex7.m1.1.1.2.2.2.3.3.cmml" xref="Ex7.m1.1.1.2.2.2.3.3"><minus id="Ex7.m1.1.1.2.2.2.3.3.1.cmml" xref="Ex7.m1.1.1.2.2.2.3.3"></minus><cn type="integer" id="Ex7.m1.1.1.2.2.2.3.3.2.cmml" xref="Ex7.m1.1.1.2.2.2.3.3.2">3</cn></apply></apply></apply><ci id="Ex7.m1.1.1.2.2.3a.cmml" xref="Ex7.m1.1.1.2.2.3"><mtext id="Ex7.m1.1.1.2.2.3.cmml" xref="Ex7.m1.1.1.2.2.3">TFlops</mtext></ci></apply><apply id="Ex7.m1.1.1.2.3.cmml" xref="Ex7.m1.1.1.2.3"><times id="Ex7.m1.1.1.2.3.1.cmml" xref="Ex7.m1.1.1.2.3.1"></times><cn type="float" id="Ex7.m1.1.1.2.3.2.cmml" xref="Ex7.m1.1.1.2.3.2">1.264</cn><ci id="Ex7.m1.1.1.2.3.3a.cmml" xref="Ex7.m1.1.1.2.3.3"><mtext id="Ex7.m1.1.1.2.3.3.cmml" xref="Ex7.m1.1.1.2.3.3">TFlops</mtext></ci></apply></apply><apply id="Ex7.m1.1.1.3.cmml" xref="Ex7.m1.1.1.3"><csymbol cd="latexml" id="Ex7.m1.1.1.3.1.cmml" xref="Ex7.m1.1.1.3.1">percent</csymbol><cn type="float" id="Ex7.m1.1.1.3.2.cmml" xref="Ex7.m1.1.1.3.2">0.228</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Ex7.m1.1c">\frac{2.884\times 10^{-3}~{}\text{TFlops}}{1.264~{}\text{TFlops}}\approx 0.228\%</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="Thmremark8.p1.4" class="ltx_p">(The calculations are based on Appendix <a href="#A1" title="Appendix A Cost Estimation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a>.)
Controlled experiments indicate that the time consumption is mainly due to two sources:</p>
<ul id="S7.I2" class="ltx_itemize">
<li id="S7.I2.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S7.I2.i1.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S7.I2.i1.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S7.I2.i1.1.1.m1.1.1" xref="S7.I2.i1.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S7.I2.i1.1.1.m1.1c"><ci id="S7.I2.i1.1.1.m1.1.1.cmml" xref="S7.I2.i1.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S7.I2.i1.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S7.I2.i1.p1" class="ltx_para">
<p id="S7.I2.i1.p1.1" class="ltx_p">Loading the memory key-values from drives to GPU:
This overhead becomes prominent as Memory<sup id="S7.I2.i1.p1.1.1" class="ltx_sup">3</sup> retrieves with higher frequency.</p>
</div>
</li>
<li id="S7.I2.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="S7.I2.i2.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="S7.I2.i2.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="S7.I2.i2.1.1.m1.1.1" xref="S7.I2.i2.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="S7.I2.i2.1.1.m1.1c"><ci id="S7.I2.i2.1.1.m1.1.1.cmml" xref="S7.I2.i2.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="S7.I2.i2.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="S7.I2.i2.p1" class="ltx_para">
<p id="S7.I2.i2.p1.1" class="ltx_p">Python implementation of chunkwise attention:
When encoding a prompt, since each chunk attends to a different set of explicit memories, we use a for loop over the chunks to compute their attentions.</p>
</div>
</li>
</ul>
<p id="Thmremark8.p1.3" class="ltx_p">They dominate other sources such as computing query vectors by the embedding model and searching the vector index.
We will try to optimize our code to reduce these overheads to be as close as possible to <math id="Thmremark8.p1.3.m1.1" class="ltx_Math" alttext="0.228\%" display="inline"><semantics id="Thmremark8.p1.3.m1.1a"><mrow id="Thmremark8.p1.3.m1.1.1" xref="Thmremark8.p1.3.m1.1.1.cmml"><mn id="Thmremark8.p1.3.m1.1.1.2" xref="Thmremark8.p1.3.m1.1.1.2.cmml">0.228</mn><mo id="Thmremark8.p1.3.m1.1.1.1" xref="Thmremark8.p1.3.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="Thmremark8.p1.3.m1.1b"><apply id="Thmremark8.p1.3.m1.1.1.cmml" xref="Thmremark8.p1.3.m1.1.1"><csymbol cd="latexml" id="Thmremark8.p1.3.m1.1.1.1.cmml" xref="Thmremark8.p1.3.m1.1.1.1">percent</csymbol><cn type="float" id="Thmremark8.p1.3.m1.1.1.2.cmml" xref="Thmremark8.p1.3.m1.1.1.2">0.228</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmremark8.p1.3.m1.1c">0.228\%</annotation></semantics></math> of the total inference time, e.g. implement the chunkwise attention with a CUDA kernel.</p>
</div>
</div>
</section>
</section>
<section id="S8" class="ltx_section">
<h2 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_section" style="font-size:144%;color:#00868B;">8 <span id="S8.1.1" class="ltx_text" style="color:#808080;">| </span>Conclusion</h2>

<div id="S8.p1" class="ltx_para">
<p id="S8.p1.1" class="ltx_p">The goal of this work is to reduce the cost of LLM training and inference, or equivalently, to construct a more efficient LLM that matches the performance of larger and slower LLMs.
We analyze LLMs from the new perspective of knowledge manipulation, characterizing the cost of LLMs as the transport cost of “knowledges” in and out of various memory formats.
Two causes of inefficiency are identified, namely the suboptimal placement of knowledges and the knowledge traversal problem.
We solve both problems with explicit memory, a novel memory format, along with a new training scheme and architecture.
Our preliminary experiment, the Memory<sup id="S8.p1.1.1" class="ltx_sup">3</sup>-2B model, exhibits stronger abilities and higher speed than many SOTA models with greater sizes as well as RAG models.</p>
</div>
<div id="S8.p2" class="ltx_para">
<p id="S8.p2.1" class="ltx_p">For future work, we plan to explore the following directions:</p>
<ol id="S8.I1" class="ltx_enumerate">
<li id="S8.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><span id="S8.I1.i1.1.1.1" class="ltx_text ltx_font_bold" style="color:#00868B;">0.</span></span> 
<div id="S8.I1.i1.p1" class="ltx_para">
<p id="S8.I1.i1.p1.1" class="ltx_p">Efficient training with abstract knowledges: Ideally, the training cost of Memory<sup id="S8.I1.i1.p1.1.1" class="ltx_sup">3</sup> model should be proportional to the small amount of non-separable knowledges, approaching the learning efficiency of humans.
One approach is to filter the training data to maximize abstract knowledges and minimize specific knowledges (cf. Section <a href="#S3.SS5" title="3.5 | Training Designs ‣ 3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.5</span></a> and Remark <a href="#Thmremark6" title="Remark 6. ‣ 4.2 | Filtering ‣ 4 | Pretraining Data ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>), and preferably the LLM should assess the quality of its own training data and ignore the unhelpful tokens.</p>
</div>
</li>
<li id="S8.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><span id="S8.I1.i2.1.1.1" class="ltx_text ltx_font_bold" style="color:#00868B;">0.</span></span> 
<div id="S8.I1.i2.p1" class="ltx_para">
<p id="S8.I1.i2.p1.1" class="ltx_p">Human-like capabilities: As described in the introduction, the explicit memory allows for interesting cognitive functions such as handling infinite contexts (conversion of working memory to explicit memory), memory consolidation (conversion of explicit memory to implicit memory), and conscious reasoning (reflection on the memory recall process).
These designs may further improve the efficiency and reasoning ability of Memory<sup id="S8.I1.i2.p1.1.1" class="ltx_sup">3</sup>.</p>
</div>
</li>
<li id="S8.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><span id="S8.I1.i3.1.1.1" class="ltx_text ltx_font_bold" style="color:#00868B;">0.</span></span> 
<div id="S8.I1.i3.p1" class="ltx_para">
<p id="S8.I1.i3.p1.1" class="ltx_p">Compact representation of explicit memory: The explicit memory of humans can be subdivided into episodic memory, which involve particular experiences, and semantic memory, which involve general truths <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib59" title="" class="ltx_ref">59</a>]</cite>.
This classification is analogous to our definition of specific and abstract knowledges.
Our current implementation of explicit memory is closer to the episodic memory of humans, as each memory directly corresponds to a reference text.
To improve its reasoning ability, one can try to equip Memory<sup id="S8.I1.i3.p1.1.1" class="ltx_sup">3</sup> with semantic memories, e.g. obtained from induction on the episodic memories.</p>
</div>
</li>
</ol>
</div>
<div id="S8.p3" class="ltx_para">
<p id="S8.p3.1" class="ltx_p">Besides these broad topics, there are also plenty of engineering works that can be done.
For instance, an internalized retrieval process that matches sparse attention queries with memory keys (Remark <a href="#Thmremark2" title="Remark 2. ‣ 3.1 | Inference Process ‣ 3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>), sparser memory heads with routing (Remark <a href="#Thmremark5" title="Remark 5. ‣ 3.3 | Memory Sparsification and Storage ‣ 3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>), memory extraction that fully preserves contexts (Remark <a href="#Thmremark3" title="Remark 3. ‣ 3.1 | Inference Process ‣ 3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>), compilation of the knowledge base based on machine preference (Remark <a href="#Thmremark7" title="Remark 7. ‣ 4.4 | Knowledge Base ‣ 4 | Pretraining Data ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>), reduction of the time consumption of explicit memory to be proportional to its compute overhead (Remark <a href="#Thmremark8" title="Remark 8. ‣ 7.5 | Inference Speed ‣ 7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a>), and so on.</p>
</div>
</section>
<section id="Sx1" class="ltx_section">
<h2 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_section" style="font-size:144%;color:#00868B;">Acknowledgement</h2>

<div id="Sx1.p1" class="ltx_para">
<p id="Sx1.p1.1" class="ltx_p">This work is supported by the NSFC Major Research Plan - Interpretable and General Purpose Next-generation Artificial Intelligence of China (No. 92270001).
We thank Prof. Zhiqin Xu, Prof. Zhouhan Lin, Fangrui Liu, Liangkai Hang, Ziyang Tao, Xiaoxing Wang, Mingze Wang, Yongqi Jin, Haotian He, Guanhua Huang, Yirong Hu for helpful discussions.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[1]</span>
<span class="ltx_bibblock">
The Chinese National Laws and Regulations Database.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://flk.npc.gov.cn/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://flk.npc.gov.cn/</a>.

</span>
<span class="ltx_bibblock">[Accessed 20-03-2024].

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[2]</span>
<span class="ltx_bibblock">
Wenshu.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://wenshu.court.gov.cn/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://wenshu.court.gov.cn/</a>.

</span>
<span class="ltx_bibblock">[Accessed 20-03-2024].

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[3]</span>
<span class="ltx_bibblock">
Megatron-DeepSpeed.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/microsoft/Megatron-DeepSpeed" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/microsoft/Megatron-DeepSpeed</a>, 2022.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[4]</span>
<span class="ltx_bibblock">
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, and et al.

</span>
<span class="ltx_bibblock">Phi-3 technical report: A highly capable language model locally on your phone, 2024.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[5]</span>
<span class="ltx_bibblock">
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.

</span>
<span class="ltx_bibblock">GPT-4 technical report.

</span>
<span class="ltx_bibblock"><span id="bib.bib5.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2303.08774</span>, 2023.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[6]</span>
<span class="ltx_bibblock">
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai.

</span>
<span class="ltx_bibblock">GQA: Training generalized multi-query transformer models from multi-head checkpoints, 2023.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[7]</span>
<span class="ltx_bibblock">
Zeyuan Allen-Zhu and Yuanzhi Li.

</span>
<span class="ltx_bibblock">Physics of language models: Part 3.3, knowledge capacity scaling laws, 2024.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[8]</span>
<span class="ltx_bibblock">
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo.

</span>
<span class="ltx_bibblock">The falcon series of open language models, 2023.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[9]</span>
<span class="ltx_bibblock">
AI Anthropic.

</span>
<span class="ltx_bibblock">The Claude 3 model family: Opus, Sonnet, Haiku.

</span>
<span class="ltx_bibblock"><span id="bib.bib9.1.1" class="ltx_text ltx_font_italic">Claude-3 Model Card</span>, 2024.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[10]</span>
<span class="ltx_bibblock">
Argilla.

</span>
<span class="ltx_bibblock">Distilabel Math Preference DPO.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://huggingface.co/datasets/argilla/distilabel-math-preference-dpo" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/datasets/argilla/distilabel-math-preference-dpo</a>, 2023.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[11]</span>
<span class="ltx_bibblock">
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, et al.

</span>
<span class="ltx_bibblock">Qwen technical report, 2023.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[12]</span>
<span class="ltx_bibblock">
Peter J Bayley and Larry R Squire.

</span>
<span class="ltx_bibblock">Failure to acquire new semantic knowledge in patients with large medial temporal lobe lesions.

</span>
<span class="ltx_bibblock"><span id="bib.bib12.1.1" class="ltx_text ltx_font_italic">Hippocampus</span>, 15(2):273–280, 2005.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[13]</span>
<span class="ltx_bibblock">
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf.

</span>
<span class="ltx_bibblock">Open LLM leaderboard.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard</a>, 2023.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[14]</span>
<span class="ltx_bibblock">
Marco Bellagente, Jonathan Tow, Dakota Mahan, Duy Phung, Maksym Zhuravinskyi, Reshinth Adithyan, James Baicoianu, Ben Brooks, Nathan Cooper, Ashish Datta, et al.

</span>
<span class="ltx_bibblock">Stable lm 2 1.6 b technical report.

</span>
<span class="ltx_bibblock"><span id="bib.bib14.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2402.17834</span>, 2024.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[15]</span>
<span class="ltx_bibblock">
Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew Gormley.

</span>
<span class="ltx_bibblock">Unlimiformer: Long-range transformers with unlimited length input.

</span>
<span class="ltx_bibblock"><span id="bib.bib15.1.1" class="ltx_text ltx_font_italic">Advances in Neural Information Processing Systems</span>, 36, 2024.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[16]</span>
<span class="ltx_bibblock">
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al.

</span>
<span class="ltx_bibblock">Improving language models by retrieving from trillions of tokens.

</span>
<span class="ltx_bibblock">In <span id="bib.bib16.1.1" class="ltx_text ltx_font_italic">International conference on machine learning</span>, pages 2206–2240. PMLR, 2022.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[17]</span>
<span class="ltx_bibblock">
Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu.

</span>
<span class="ltx_bibblock">BGE M3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation.

</span>
<span class="ltx_bibblock"><span id="bib.bib17.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2402.03216</span>, 2024.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[18]</span>
<span class="ltx_bibblock">
Yongjian Chen, Tao Guan, and Cheng Wang.

</span>
<span class="ltx_bibblock">Approximate nearest neighbor search by residual vector quantization.

</span>
<span class="ltx_bibblock"><span id="bib.bib18.1.1" class="ltx_text ltx_font_italic">Sensors</span>, 10(12):11259–11273, 2010.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[19]</span>
<span class="ltx_bibblock">
Zheng Chen, Yuqing Li, Tao Luo, Zhaoguang Zhou, and Zhi-Qin John Xu.

</span>
<span class="ltx_bibblock">Phase diagram of initial condensation for two-layer neural networks.

</span>
<span class="ltx_bibblock"><span id="bib.bib19.1.1" class="ltx_text ltx_font_italic">ArXiv</span>, abs/2303.06561, 2023.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[20]</span>
<span class="ltx_bibblock">
Qinyuan Cheng, Tianxiang Sun, Wenwei Zhang, Siyin Wang, Xiangyang Liu, Mozhi Zhang, Junliang He, Mianqiu Huang, Zhangyue Yin, Kai Chen, and Xipeng Qiu.

</span>
<span class="ltx_bibblock">Evaluating hallucinations in chinese large language models, 2023.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[21]</span>
<span class="ltx_bibblock">
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, et al.

</span>
<span class="ltx_bibblock">Palm: Scaling language modeling with pathways, 2022.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[22]</span>
<span class="ltx_bibblock">
Bilal Chughtai, Alan Cooney, and Neel Nanda.

</span>
<span class="ltx_bibblock">Summing up the facts: Additive mechanisms behind factual recall in llms.

</span>
<span class="ltx_bibblock"><span id="bib.bib22.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2402.07321</span>, 2024.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[23]</span>
<span class="ltx_bibblock">
Together Computer.

</span>
<span class="ltx_bibblock">Redpajama: an open dataset for training large language models, October 2023.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[24]</span>
<span class="ltx_bibblock">
Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso.

</span>
<span class="ltx_bibblock">Towards automated circuit discovery for mechanistic interpretability.

</span>
<span class="ltx_bibblock"><span id="bib.bib24.1.1" class="ltx_text ltx_font_italic">Advances in Neural Information Processing Systems</span>, 36:16318–16352, 2023.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[25]</span>
<span class="ltx_bibblock">
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov.

</span>
<span class="ltx_bibblock">Unsupervised cross-lingual representation learning at scale.

</span>
<span class="ltx_bibblock"><span id="bib.bib25.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:1911.02116</span>, 2019.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[26]</span>
<span class="ltx_bibblock">
Suzanne Corkin.

</span>
<span class="ltx_bibblock">What’s new with the amnesic patient H.M.?

</span>
<span class="ltx_bibblock"><span id="bib.bib26.1.1" class="ltx_text ltx_font_italic">Nature reviews neuroscience</span>, 3(2):153–160, 2002.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[27]</span>
<span class="ltx_bibblock">
Nelson Cowan.

</span>
<span class="ltx_bibblock">The magical number 4 in short-term memory: A reconsideration of mental storage capacity.

</span>
<span class="ltx_bibblock"><span id="bib.bib27.1.1" class="ltx_text ltx_font_italic">Behavioral and Brain Sciences</span>, 24(1):87–114, 2001.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[28]</span>
<span class="ltx_bibblock">
Nelson Cowan.

</span>
<span class="ltx_bibblock"><span id="bib.bib28.1.1" class="ltx_text ltx_font_italic">Working memory capacity: Classic Edition</span>.

</span>
<span class="ltx_bibblock">Routledge, 2016.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[29]</span>
<span class="ltx_bibblock">
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei.

</span>
<span class="ltx_bibblock">Knowledge neurons in pretrained transformers.

</span>
<span class="ltx_bibblock"><span id="bib.bib29.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2104.08696</span>, 2021.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[30]</span>
<span class="ltx_bibblock">
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov.

</span>
<span class="ltx_bibblock">Transformer-xl: Attentive language models beyond a fixed-length context, 2019.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[31]</span>
<span class="ltx_bibblock">
Luigi Daniele and Suphavadeeprasit.

</span>
<span class="ltx_bibblock">Amplify-instruct: Synthetically generated diverse multi-turn conversations for effecient llm training.

</span>
<span class="ltx_bibblock"><span id="bib.bib31.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:(coming soon)</span>, 2023.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[32]</span>
<span class="ltx_bibblock">
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré.

</span>
<span class="ltx_bibblock">Flashattention: Fast and memory-efficient exact attention with io-awareness.

</span>
<span class="ltx_bibblock"><span id="bib.bib32.1.1" class="ltx_text ltx_font_italic">Advances in Neural Information Processing Systems</span>, 35:16344–16359, 2022.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[33]</span>
<span class="ltx_bibblock">
Nolan Dey, Gurpreet Gosal, Zhiming, Chen, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, and Joel Hestness.

</span>
<span class="ltx_bibblock">Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster, 2023.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[34]</span>
<span class="ltx_bibblock">
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou.

</span>
<span class="ltx_bibblock">Enhancing chat language models by scaling high-quality instructional conversations, 2023.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[35]</span>
<span class="ltx_bibblock">
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou.

</span>
<span class="ltx_bibblock">The faiss library, 2024.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[36]</span>
<span class="ltx_bibblock">
Phung Van Duy.

</span>
<span class="ltx_bibblock">synth_code_preference_4k.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://huggingface.co/datasets/pvduy/synth_code_preference_4k" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/datasets/pvduy/synth_code_preference_4k</a>, 2023.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[37]</span>
<span class="ltx_bibblock">
Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli.

</span>
<span class="ltx_bibblock">Depth-adaptive transformer, 2020.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[38]</span>
<span class="ltx_bibblock">
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah.

</span>
<span class="ltx_bibblock">Toy models of superposition.

</span>
<span class="ltx_bibblock"><span id="bib.bib38.1.1" class="ltx_text ltx_font_italic">Transformer Circuits Thread</span>, 2022.

</span>
<span class="ltx_bibblock">https://transformer-circuits.pub/2022/toy_model/index.html.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[39]</span>
<span class="ltx_bibblock">
Junjie Fang, Likai Tang, Hongzhe Bi, Yujia Qin, Si Sun, Zhenyu Li, Haolun Li, Yongjian Li, Xin Cong, Yukun Yan, Xiaodong Shi, Sen Song, Yankai Lin, Zhiyuan Liu, and Maosong Sun.

</span>
<span class="ltx_bibblock">UniMem: Towards a unified view of long-context large language models, 2024.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[40]</span>
<span class="ltx_bibblock">
William Fedus, Barret Zoph, and Noam Shazeer.

</span>
<span class="ltx_bibblock">Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.

</span>
<span class="ltx_bibblock"><span id="bib.bib40.1.1" class="ltx_text ltx_font_italic">J. Mach. Learn. Res.</span>, 23:120:1–120:39, 2022.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[41]</span>
<span class="ltx_bibblock">
Elias Frantar and Dan Alistarh.

</span>
<span class="ltx_bibblock">Qmoe: Practical sub-1-bit compression of trillion-parameter models.

</span>
<span class="ltx_bibblock"><span id="bib.bib41.1.1" class="ltx_text ltx_font_italic">CoRR</span>, abs/2310.16795, 2023.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[42]</span>
<span class="ltx_bibblock">
John DE Gabrieli, Neal J Cohen, and Suzanne Corkin.

</span>
<span class="ltx_bibblock">The impaired learning of semantic knowledge following bilateral medial temporal-lobe resection.

</span>
<span class="ltx_bibblock"><span id="bib.bib42.1.1" class="ltx_text ltx_font_italic">Brain and cognition</span>, 7(2):157–177, 1988.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[43]</span>
<span class="ltx_bibblock">
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy.

</span>
<span class="ltx_bibblock">The pile: An 800gb dataset of diverse text for language modeling.

</span>
<span class="ltx_bibblock"><span id="bib.bib43.1.1" class="ltx_text ltx_font_italic">CoRR</span>, abs/2101.00027, 2021.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[44]</span>
<span class="ltx_bibblock">
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou.

</span>
<span class="ltx_bibblock">A framework for few-shot language model evaluation, 12 2023.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[45]</span>
<span class="ltx_bibblock">
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson.

</span>
<span class="ltx_bibblock">Dissecting recall of factual associations in auto-regressive language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib45.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2304.14767</span>, 2023.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[46]</span>
<span class="ltx_bibblock">
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy.

</span>
<span class="ltx_bibblock">Transformer feed-forward layers are key-value memories, 2021.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[47]</span>
<span class="ltx_bibblock">
Xavier Glorot and Yoshua Bengio.

</span>
<span class="ltx_bibblock">Understanding the difficulty of training deep feedforward neural networks.

</span>
<span class="ltx_bibblock">In <span id="bib.bib47.1.1" class="ltx_text ltx_font_italic">Proceedings of the thirteenth international conference on artificial intelligence and statistics</span>, pages 249–256. JMLR Workshop and Conference Proceedings, 2010.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[48]</span>
<span class="ltx_bibblock">
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li.

</span>
<span class="ltx_bibblock">Textbooks are all you need, 2023.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[49]</span>
<span class="ltx_bibblock">
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang.

</span>
<span class="ltx_bibblock">Retrieval augmented language model pre-training.

</span>
<span class="ltx_bibblock">In <span id="bib.bib49.1.1" class="ltx_text ltx_font_italic">International conference on machine learning</span>, pages 3929–3938. PMLR, 2020.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[50]</span>
<span class="ltx_bibblock">
Yiding Hao, Dana Angluin, and Robert Frank.

</span>
<span class="ltx_bibblock">Formal language recognition by hard attention transformers: Perspectives from circuit complexity.

</span>
<span class="ltx_bibblock"><span id="bib.bib50.1.1" class="ltx_text ltx_font_italic">Transactions of the Association for Computational Linguistics</span>, 10:800–810, 2022.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[51]</span>
<span class="ltx_bibblock">
Conghui He, Zhenjiang Jin, Chao Xu, Jiantao Qiu, Bin Wang, Wei Li, Hang Yan, Jiaqi Wang, and Dahua Lin.

</span>
<span class="ltx_bibblock">Wanjuan: A comprehensive multimodal dataset for advancing english and chinese large models.

</span>
<span class="ltx_bibblock"><span id="bib.bib51.1.1" class="ltx_text ltx_font_italic">CoRR</span>, abs/2308.10755, 2023.

</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[52]</span>
<span class="ltx_bibblock">
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.

</span>
<span class="ltx_bibblock">Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.

</span>
<span class="ltx_bibblock">In <span id="bib.bib52.1.1" class="ltx_text ltx_font_italic">Proceedings of the IEEE international conference on computer vision</span>, pages 1026–1034, 2015.

</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[53]</span>
<span class="ltx_bibblock">
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al.

</span>
<span class="ltx_bibblock">Training compute-optimal large language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib53.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2203.15556</span>, 2022.

</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[54]</span>
<span class="ltx_bibblock">
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun.

</span>
<span class="ltx_bibblock">Minicpm: Unveiling the potential of small language models with scalable training strategies, 2024.

</span>
</li>
<li id="bib.bib55" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[55]</span>
<span class="ltx_bibblock">
Yufei Huang, Shengding Hu, Xu Han, Zhiyuan Liu, and Maosong Sun.

</span>
<span class="ltx_bibblock">Unified view of grokking, double descent and emergent abilities: A perspective from circuits competition.

</span>
<span class="ltx_bibblock"><span id="bib.bib55.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2402.15175</span>, 2024.

</span>
</li>
<li id="bib.bib56" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[56]</span>
<span class="ltx_bibblock">
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.

</span>
<span class="ltx_bibblock">Mistral 7b, 2023.

</span>
</li>
<li id="bib.bib57" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[57]</span>
<span class="ltx_bibblock">
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu.

</span>
<span class="ltx_bibblock">Tinybert: Distilling bert for natural language understanding, 2020.

</span>
</li>
<li id="bib.bib58" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[58]</span>
<span class="ltx_bibblock">
Jean Kaddour.

</span>
<span class="ltx_bibblock">The minipile challenge for data-efficient language models, 2023.

</span>
</li>
<li id="bib.bib59" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[59]</span>
<span class="ltx_bibblock">
E.R. Kandel, J.D. Koester, S.H. Mack, and S.A. Siegelbaum.

</span>
<span class="ltx_bibblock"><span id="bib.bib59.1.1" class="ltx_text ltx_font_italic">Principles of Neural Science, Sixth Edition</span>.

</span>
<span class="ltx_bibblock">McGraw Hill LLC, 2021.

</span>
</li>
<li id="bib.bib60" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[60]</span>
<span class="ltx_bibblock">
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.

</span>
<span class="ltx_bibblock">Scaling laws for neural language models, 2020.

</span>
</li>
<li id="bib.bib61" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[61]</span>
<span class="ltx_bibblock">
Omar Khattab and Matei Zaharia.

</span>
<span class="ltx_bibblock">Colbert: Efficient and effective passage search via contextualized late interaction over bert, 2020.

</span>
</li>
<li id="bib.bib62" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[62]</span>
<span class="ltx_bibblock">
Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro.

</span>
<span class="ltx_bibblock">Reducing activation recomputation in large transformer models, 2022.

</span>
</li>
<li id="bib.bib63" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[63]</span>
<span class="ltx_bibblock">
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica.

</span>
<span class="ltx_bibblock">Efficient memory management for large language model serving with pagedattention, 2023.

</span>
</li>
<li id="bib.bib64" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[64]</span>
<span class="ltx_bibblock">
Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen.

</span>
<span class="ltx_bibblock">Halueval: A large-scale hallucination evaluation benchmark for large language models.

</span>
<span class="ltx_bibblock">In <span id="bib.bib64.1.1" class="ltx_text ltx_font_italic">The 2023 Conference on Empirical Methods in Natural Language Processing</span>, 2023.

</span>
</li>
<li id="bib.bib65" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[65]</span>
<span class="ltx_bibblock">
Miao Li, Ming-Bin Chen, Bo Tang, Shengbin Hou, Pengyu Wang, Haiying Deng, Zhiyu Li, Feiyu Xiong, Keming Mao, Peng Cheng, and Yi Luo.

</span>
<span class="ltx_bibblock">Newsbench: A systematic evaluation framework for assessing editorial capabilities of large language models in chinese journalism, 2024.

</span>
</li>
<li id="bib.bib66" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[66]</span>
<span class="ltx_bibblock">
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee.

</span>
<span class="ltx_bibblock">Textbooks are all you need ii: phi-1.5 technical report, 2023.

</span>
</li>
<li id="bib.bib67" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[67]</span>
<span class="ltx_bibblock">
Wing Lian, Guan Wang, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and ”Teknium”.

</span>
<span class="ltx_bibblock">Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verification, 2023.

</span>
</li>
<li id="bib.bib68" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[68]</span>
<span class="ltx_bibblock">
Xun Liang, Shichao Song, Simin Niu, Zhiyu Li, Feiyu Xiong, Bo Tang, Yezhaohui Wang, Dawei He, Peng Cheng, Zhonghao Wang, and Haiying Deng.

</span>
<span class="ltx_bibblock">Uhgeval: Benchmarking the hallucination of chinese large language models via unconstrained generation, 2024.

</span>
</li>
<li id="bib.bib69" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[69]</span>
<span class="ltx_bibblock">
Xun Liang, Shichao Song, Simin Niu, Zhiyu Li, Feiyu Xiong, Bo Tang, Zhaohui Wy, Dawei He, Peng Cheng, Zhonghao Wang, and Haiying Deng.

</span>
<span class="ltx_bibblock">UHGEval: Benchmarking the hallucination of chinese large language models via unconstrained generation.

</span>
<span class="ltx_bibblock"><span id="bib.bib69.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2311.15296</span>, 2023.

</span>
</li>
<li id="bib.bib70" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[70]</span>
<span class="ltx_bibblock">
Stephanie Lin, Jacob Hilton, and Owain Evans.

</span>
<span class="ltx_bibblock">TruthfulQA: Measuring how models mimic human falsehoods.

</span>
<span class="ltx_bibblock">In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, <span id="bib.bib70.1.1" class="ltx_text ltx_font_italic">Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)</span>, pages 3214–3252, Dublin, Ireland, May 2022. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib71" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[71]</span>
<span class="ltx_bibblock">
Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma.

</span>
<span class="ltx_bibblock">Sophia: A scalable stochastic second-order optimizer for language model pre-training, 2024.

</span>
</li>
<li id="bib.bib72" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[72]</span>
<span class="ltx_bibblock">
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang.

</span>
<span class="ltx_bibblock">Lost in the middle: How language models use long contexts, 2023.

</span>
</li>
<li id="bib.bib73" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[73]</span>
<span class="ltx_bibblock">
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He.

</span>
<span class="ltx_bibblock">What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning, 2023.

</span>
</li>
<li id="bib.bib74" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[74]</span>
<span class="ltx_bibblock">
Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava.

</span>
<span class="ltx_bibblock">Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time, 2023.

</span>
</li>
<li id="bib.bib75" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[75]</span>
<span class="ltx_bibblock">
Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Ré, and Beidi Chen.

</span>
<span class="ltx_bibblock">Deja vu: Contextual sparsity for efficient llms at inference time.

</span>
<span class="ltx_bibblock">In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, <span id="bib.bib75.1.1" class="ltx_text ltx_font_italic">International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA</span>, volume 202 of <span id="bib.bib75.2.2" class="ltx_text ltx_font_italic">Proceedings of Machine Learning Research</span>, pages 22137–22176. PMLR, 2023.

</span>
</li>
<li id="bib.bib76" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[76]</span>
<span class="ltx_bibblock">
Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al.

</span>
<span class="ltx_bibblock">A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, &amp; toxicity.

</span>
<span class="ltx_bibblock"><span id="bib.bib76.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2305.13169</span>, 2023.

</span>
</li>
<li id="bib.bib77" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[77]</span>
<span class="ltx_bibblock">
Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat.

</span>
<span class="ltx_bibblock">Estimating the carbon footprint of bloom, a 176b parameter language model, 2022.

</span>
</li>
<li id="bib.bib78" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[78]</span>
<span class="ltx_bibblock">
Tao Luo, Zhi-Qin John Xu, Zheng Ma, and Yaoyu Zhang.

</span>
<span class="ltx_bibblock">Phase diagram for two-layer relu neural networks at infinite-width limit.

</span>
<span class="ltx_bibblock"><span id="bib.bib78.1.1" class="ltx_text ltx_font_italic">Journal of Machine Learning Research</span>, 22(71):1–47, 2021.

</span>
</li>
<li id="bib.bib79" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[79]</span>
<span class="ltx_bibblock">
Ang Lv, Yuhan Chen, Kaiyi Zhang, Yulong Wang, Lifeng Liu, Ji-Rong Wen, Jian Xie, and Rui Yan.

</span>
<span class="ltx_bibblock">Interpreting key mechanisms of factual recall in transformer-based language models, 2024.

</span>
</li>
<li id="bib.bib80" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[80]</span>
<span class="ltx_bibblock">
William Merrill and Ashish Sabharwal.

</span>
<span class="ltx_bibblock">A logic for expressing log-precision transformers, 2023.

</span>
</li>
<li id="bib.bib81" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[81]</span>
<span class="ltx_bibblock">
MOP-LIWU Community and MNBVC Team.

</span>
<span class="ltx_bibblock">Mnbvc: Massive never-ending bt vast chinese corpus.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/esbatmop/MNBVC" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/esbatmop/MNBVC</a>, 2023.

</span>
</li>
<li id="bib.bib82" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[82]</span>
<span class="ltx_bibblock">
MyScale.

</span>
<span class="ltx_bibblock">MyScaleDB.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/myscale/MyScaleDB" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/myscale/MyScaleDB</a>.

</span>
<span class="ltx_bibblock">[Accessed 20-03-2024].

</span>
</li>
<li id="bib.bib83" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[83]</span>
<span class="ltx_bibblock">
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al.

</span>
<span class="ltx_bibblock">Webgpt: Browser-assisted question-answering with human feedback.

</span>
<span class="ltx_bibblock"><span id="bib.bib83.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2112.09332</span>, 2021.

</span>
</li>
<li id="bib.bib84" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[84]</span>
<span class="ltx_bibblock">
Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Anand Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia.

</span>
<span class="ltx_bibblock">Efficient large-scale language model training on gpu clusters using megatron-lm, 2021.

</span>
</li>
<li id="bib.bib85" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[85]</span>
<span class="ltx_bibblock">
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah.

</span>
<span class="ltx_bibblock">In-context learning and induction heads, 2022.

</span>
</li>
<li id="bib.bib86" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[86]</span>
<span class="ltx_bibblock">
OpenAI.

</span>
<span class="ltx_bibblock">GPT-4 turbo and GPT-4.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://platform.openai.com/docs/models/gpt-4-turbo-and-gpt-4" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://platform.openai.com/docs/models/gpt-4-turbo-and-gpt-4</a>, 2024.

</span>
<span class="ltx_bibblock">[Accessed 22-05-2024].

</span>
</li>
<li id="bib.bib87" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[87]</span>
<span class="ltx_bibblock">
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.

</span>
<span class="ltx_bibblock">Training language models to follow instructions with human feedback.

</span>
<span class="ltx_bibblock"><span id="bib.bib87.1.1" class="ltx_text ltx_font_italic">Advances in neural information processing systems</span>, 35:27730–27744, 2022.

</span>
</li>
<li id="bib.bib88" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[88]</span>
<span class="ltx_bibblock">
Adam Pearce, Asma Ghandeharioun, Nada Hussein, Nithum Thain, Martin Wattenberg, and Lucas Dixon.

</span>
<span class="ltx_bibblock">Do machine learning models memorize or generalize?

</span>
<span class="ltx_bibblock"><span id="bib.bib88.1.1" class="ltx_text ltx_font_italic">People+ AI Research</span>, 2023.

</span>
</li>
<li id="bib.bib89" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[89]</span>
<span class="ltx_bibblock">
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al.

</span>
<span class="ltx_bibblock">Rwkv: Reinventing rnns for the transformer era, 2023.

</span>
</li>
<li id="bib.bib90" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[90]</span>
<span class="ltx_bibblock">
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra.

</span>
<span class="ltx_bibblock">Grokking: Generalization beyond overfitting on small algorithmic datasets, 2022.

</span>
</li>
<li id="bib.bib91" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[91]</span>
<span class="ltx_bibblock">
Penghui Qi, Xinyi Wan, Guangxing Huang, and Min Lin.

</span>
<span class="ltx_bibblock">Zero bubble pipeline parallelism, 2023.

</span>
</li>
<li id="bib.bib92" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[92]</span>
<span class="ltx_bibblock">
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al.

</span>
<span class="ltx_bibblock">Scaling language models: Methods, analysis &amp; insights from training gopher.

</span>
<span class="ltx_bibblock"><span id="bib.bib92.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2112.11446</span>, 2021.

</span>
</li>
<li id="bib.bib93" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[93]</span>
<span class="ltx_bibblock">
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.

</span>
<span class="ltx_bibblock">Direct preference optimization: Your language model is secretly a reward model.

</span>
<span class="ltx_bibblock"><span id="bib.bib93.1.1" class="ltx_text ltx_font_italic">Advances in Neural Information Processing Systems</span>, 36, 2024.

</span>
</li>
<li id="bib.bib94" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[94]</span>
<span class="ltx_bibblock">
David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro.

</span>
<span class="ltx_bibblock">Mixture-of-depths: Dynamically allocating compute in transformer-based language models, 2024.

</span>
</li>
<li id="bib.bib95" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[95]</span>
<span class="ltx_bibblock">
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He.

</span>
<span class="ltx_bibblock">Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters.

</span>
<span class="ltx_bibblock">In <span id="bib.bib95.1.1" class="ltx_text ltx_font_italic">Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery &amp; Data Mining</span>, pages 3505–3506, 2020.

</span>
</li>
<li id="bib.bib96" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[96]</span>
<span class="ltx_bibblock">
Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham.

</span>
<span class="ltx_bibblock">Parallel context windows for large language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib96.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2212.10947</span>, 2022.

</span>
</li>
<li id="bib.bib97" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[97]</span>
<span class="ltx_bibblock">
Vipula Rawte, Swagata Chakraborty, Agnibh Pathak, Anubhav Sarkar, S.M Towhidul Islam Tonmoy, Aman Chadha, Amit Sheth, and Amitava Das.

</span>
<span class="ltx_bibblock">The troubling emergence of hallucination in large language models - an extensive definition, quantification, and prescriptive remediations.

</span>
<span class="ltx_bibblock">In Houda Bouamor, Juan Pino, and Kalika Bali, editors, <span id="bib.bib97.1.1" class="ltx_text ltx_font_italic">Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing</span>, pages 2541–2573, Singapore, December 2023. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib98" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[98]</span>
<span class="ltx_bibblock">
Snowflake AI Research.

</span>
<span class="ltx_bibblock">Snowflake arctic: The best LLM for enterprise AI — efficiently intelligent, truly open, Apr 2024.

</span>
<span class="ltx_bibblock">Accessed: 2024-05-15.

</span>
</li>
<li id="bib.bib99" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[99]</span>
<span class="ltx_bibblock">
Yangjun Ruan, Chris J. Maddison, and Tatsunori Hashimoto.

</span>
<span class="ltx_bibblock">Observational scaling laws and the predictability of language model performance, 2024.

</span>
</li>
<li id="bib.bib100" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[100]</span>
<span class="ltx_bibblock">
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.

</span>
<span class="ltx_bibblock">Toolformer: Language models can teach themselves to use tools.

</span>
<span class="ltx_bibblock"><span id="bib.bib100.1.1" class="ltx_text ltx_font_italic">Advances in Neural Information Processing Systems</span>, 36, 2024.

</span>
</li>
<li id="bib.bib101" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[101]</span>
<span class="ltx_bibblock">
Scienceasdf.

</span>
<span class="ltx_bibblock">Medical books.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/scienceasdf/medical-books" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/scienceasdf/medical-books</a>.

</span>
<span class="ltx_bibblock">[Accessed 20-03-2024].

</span>
</li>
<li id="bib.bib102" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[102]</span>
<span class="ltx_bibblock">
Azure AI Services.

</span>
<span class="ltx_bibblock">GPT-4 and GPT-4 turbo models.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models#gpt-4-and-gpt-4-turbo-models" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models#gpt-4-and-gpt-4-turbo-models</a>, 2024.

</span>
<span class="ltx_bibblock">[Accessed 22-05-2024].

</span>
</li>
<li id="bib.bib103" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[103]</span>
<span class="ltx_bibblock">
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro.

</span>
<span class="ltx_bibblock">Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.

</span>
</li>
<li id="bib.bib104" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[104]</span>
<span class="ltx_bibblock">
Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey.

</span>
<span class="ltx_bibblock">SlimPajama: A 627B token cleaned and deduplicated version of RedPajama.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama</a>, June 2023.

</span>
</li>
<li id="bib.bib105" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[105]</span>
<span class="ltx_bibblock">
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, and et al.

</span>
<span class="ltx_bibblock">Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2023.

</span>
</li>
<li id="bib.bib106" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[106]</span>
<span class="ltx_bibblock">
Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan.

</span>
<span class="ltx_bibblock">A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis.

</span>
<span class="ltx_bibblock">In <span id="bib.bib106.1.1" class="ltx_text ltx_font_italic">Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing</span>, pages 7035–7052, 2023.

</span>
</li>
<li id="bib.bib107" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[107]</span>
<span class="ltx_bibblock">
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu.

</span>
<span class="ltx_bibblock">Roformer: Enhanced transformer with rotary position embedding.

</span>
<span class="ltx_bibblock"><span id="bib.bib107.1.1" class="ltx_text ltx_font_italic">Neurocomputing</span>, 568:127063, 2024.

</span>
</li>
<li id="bib.bib108" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[108]</span>
<span class="ltx_bibblock">
Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Herve Jegou, and Armand Joulin.

</span>
<span class="ltx_bibblock">Augmenting self-attention with persistent memory, 2019.

</span>
</li>
<li id="bib.bib109" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[109]</span>
<span class="ltx_bibblock">
Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei.

</span>
<span class="ltx_bibblock">You only cache once: Decoder-decoder architectures for language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib109.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2405.05254</span>, 2024.

</span>
</li>
<li id="bib.bib110" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[110]</span>
<span class="ltx_bibblock">
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.

</span>
<span class="ltx_bibblock">Llama 2: Open foundation and fine-tuned chat models.

</span>
<span class="ltx_bibblock"><span id="bib.bib110.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2307.09288</span>, 2023.

</span>
</li>
<li id="bib.bib111" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[111]</span>
<span class="ltx_bibblock">
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf.

</span>
<span class="ltx_bibblock">Zephyr: Direct distillation of lm alignment, 2023.

</span>
</li>
<li id="bib.bib112" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[112]</span>
<span class="ltx_bibblock">
Szymon Tworkowski, Konrad Staniszewski, Mikołaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Miłoś.

</span>
<span class="ltx_bibblock">Focused transformer: Contrastive training for context scaling.

</span>
<span class="ltx_bibblock"><span id="bib.bib112.1.1" class="ltx_text ltx_font_italic">Advances in Neural Information Processing Systems</span>, 36, 2024.

</span>
</li>
<li id="bib.bib113" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[113]</span>
<span class="ltx_bibblock">
Boxin Wang, Wei Ping, Peng Xu, Lawrence McAfee, Zihan Liu, Mohammad Shoeybi, Yi Dong, Oleksii Kuchaiev, Bo Li, Chaowei Xiao, Anima Anandkumar, and Bryan Catanzaro.

</span>
<span class="ltx_bibblock">Shall we pretrain autoregressive language models with retrieval? A comprehensive study.

</span>
<span class="ltx_bibblock">In Houda Bouamor, Juan Pino, and Kalika Bali, editors, <span id="bib.bib113.1.1" class="ltx_text ltx_font_italic">Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing</span>, pages 7763–7786, Singapore, December 2023. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib114" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[114]</span>
<span class="ltx_bibblock">
Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu.

</span>
<span class="ltx_bibblock">Openchat: Advancing open-source language models with mixed-quality data, 2023.

</span>
</li>
<li id="bib.bib115" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[115]</span>
<span class="ltx_bibblock">
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt.

</span>
<span class="ltx_bibblock">Interpretability in the wild: a circuit for indirect object identification in gpt-2 small.

</span>
<span class="ltx_bibblock"><span id="bib.bib115.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2211.00593</span>, 2022.

</span>
</li>
<li id="bib.bib116" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[116]</span>
<span class="ltx_bibblock">
Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun.

</span>
<span class="ltx_bibblock">Label words are anchors: An information flow perspective for understanding in-context learning.

</span>
<span class="ltx_bibblock"><span id="bib.bib116.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2305.14160</span>, 2023.

</span>
</li>
<li id="bib.bib117" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[117]</span>
<span class="ltx_bibblock">
Mingze Wang, Haotian He, Jinbo Wang, Zilin Wang, Guanhua Huang, Feiyu Xiong, Zhiyu Li, Weinan E, and Lei Wu.

</span>
<span class="ltx_bibblock">Improving generalization and convergence by enhancing implicit regularization, 2024.

</span>
</li>
<li id="bib.bib118" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[118]</span>
<span class="ltx_bibblock">
Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei.

</span>
<span class="ltx_bibblock">Augmenting language models with long-term memory.

</span>
<span class="ltx_bibblock"><span id="bib.bib118.1.1" class="ltx_text ltx_font_italic">Advances in Neural Information Processing Systems</span>, 36, 2024.

</span>
</li>
<li id="bib.bib119" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[119]</span>
<span class="ltx_bibblock">
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus.

</span>
<span class="ltx_bibblock">Emergent abilities of large language models, 2022.

</span>
</li>
<li id="bib.bib120" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[120]</span>
<span class="ltx_bibblock">
Gail Weiss, Yoav Goldberg, and Eran Yahav.

</span>
<span class="ltx_bibblock">Thinking like transformers, 2021.

</span>
</li>
<li id="bib.bib121" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[121]</span>
<span class="ltx_bibblock">
Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga Behram, James Huang, Charles Bai, Michael Gschwind, Anurag Gupta, Myle Ott, Anastasia Melnikov, Salvatore Candido, David Brooks, Geeta Chauhan, Benjamin Lee, Hsien-Hsin S. Lee, Bugra Akyildiz, Maximilian Balandat, Joe Spisak, Ravi Jain, Mike Rabbat, and Kim Hazelwood.

</span>
<span class="ltx_bibblock">Sustainable ai: Environmental implications, challenges and opportunities, 2022.

</span>
</li>
<li id="bib.bib122" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[122]</span>
<span class="ltx_bibblock">
Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu.

</span>
<span class="ltx_bibblock">Retrieval head mechanistically explains long-context factuality, 2024.

</span>
</li>
<li id="bib.bib123" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[123]</span>
<span class="ltx_bibblock">
Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, and Christian Szegedy.

</span>
<span class="ltx_bibblock">Memorizing transformers.

</span>
<span class="ltx_bibblock">In <span id="bib.bib123.1.1" class="ltx_text ltx_font_italic">International Conference on Learning Representations</span>, 2021.

</span>
</li>
<li id="bib.bib124" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[124]</span>
<span class="ltx_bibblock">
Xingyu Xie, Pan Zhou, Huan Li, Zhouchen Lin, and Shuicheng Yan.

</span>
<span class="ltx_bibblock">Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models, 2023.

</span>
</li>
<li id="bib.bib125" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[125]</span>
<span class="ltx_bibblock">
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang.

</span>
<span class="ltx_bibblock">Wizardlm: Empowering large language models to follow complex instructions.

</span>
<span class="ltx_bibblock"><span id="bib.bib125.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2304.12244</span>, 2023.

</span>
</li>
<li id="bib.bib126" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[126]</span>
<span class="ltx_bibblock">
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang, Fei Deng, Feng Wang, Feng Liu, Guangwei Ai, Guosheng Dong, Haizhou Zhao, et al.

</span>
<span class="ltx_bibblock">Baichuan 2: Open large-scale language models, 2023.

</span>
</li>
<li id="bib.bib127" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[127]</span>
<span class="ltx_bibblock">
Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao.

</span>
<span class="ltx_bibblock">Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer.

</span>
<span class="ltx_bibblock"><span id="bib.bib127.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2203.03466</span>, 2022.

</span>
</li>
<li id="bib.bib128" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[128]</span>
<span class="ltx_bibblock">
Hongkang Yang.

</span>
<span class="ltx_bibblock">A mathematical framework for learning probability distributions.

</span>
<span class="ltx_bibblock"><span id="bib.bib128.1.1" class="ltx_text ltx_font_italic">Journal of Machine Learning</span>, 1(4):373–431, 2022.

</span>
</li>
<li id="bib.bib129" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[129]</span>
<span class="ltx_bibblock">
Yunzhi Yao, Ningyu Zhang, Zekun Xi, Mengru Wang, Ziwen Xu, Shumin Deng, and Huajun Chen.

</span>
<span class="ltx_bibblock">Knowledge circuits in pretrained transformers, 2024.

</span>
</li>
<li id="bib.bib130" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[130]</span>
<span class="ltx_bibblock">
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu.

</span>
<span class="ltx_bibblock">Metamath: Bootstrap your own mathematical questions for large language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib130.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2309.12284</span>, 2023.

</span>
</li>
<li id="bib.bib131" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[131]</span>
<span class="ltx_bibblock">
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al.

</span>
<span class="ltx_bibblock">H2o: Heavy-hitter oracle for efficient generative inference of large language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib131.1.1" class="ltx_text ltx_font_italic">Advances in Neural Information Processing Systems</span>, 36, 2024.

</span>
</li>
<li id="bib.bib132" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[132]</span>
<span class="ltx_bibblock">
Zhongwang Zhang, Pengxiao Lin, Zhiwei Wang, Yaoyu Zhang, and Zhi-Qin John Xu.

</span>
<span class="ltx_bibblock">Initialization is critical to whether transformers fit composite functions by inference or memorizing, 2024.

</span>
</li>
<li id="bib.bib133" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[133]</span>
<span class="ltx_bibblock">
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.

</span>
<span class="ltx_bibblock">Judging llm-as-a-judge with mt-bench and chatbot arena.

</span>
<span class="ltx_bibblock"><span id="bib.bib133.1.1" class="ltx_text ltx_font_italic">Advances in Neural Information Processing Systems</span>, 36, 2024.

</span>
</li>
<li id="bib.bib134" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[134]</span>
<span class="ltx_bibblock">
Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun.

</span>
<span class="ltx_bibblock">Jec-qa: A legal-domain question answering dataset.

</span>
<span class="ltx_bibblock"><span id="bib.bib134.1.1" class="ltx_text ltx_font_italic">Proceedings of the AAAI Conference on Artificial Intelligence</span>, 34(05):9701–9708, Apr. 2020.

</span>
</li>
</ul>
</section>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Cost Estimation</h2>

<div id="A1.p1" class="ltx_para">
<p id="A1.p1.1" class="ltx_p">This section provides the calculations for Figure <a href="#F4" title="Figure 4 ‣ 1 | Introduction ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, and we equate cost with the amount of compute measured in Tflops.</p>
</div>
<div id="A1.p2" class="ltx_para">
<p id="A1.p2.1" class="ltx_p">Our 2.4B Memory<sup id="A1.p2.1.1" class="ltx_sup">3</sup> model is adopted as the backbone.
Recall from Section <a href="#S3.SS4" title="3.4 | Model Shape ‣ 3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.4</span></a> that this model has shape</p>
<ul id="A1.I1" class="ltx_itemize">
<li id="A1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="A1.I1.i1.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="A1.I1.i1.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="A1.I1.i1.1.1.m1.1.1" xref="A1.I1.i1.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="A1.I1.i1.1.1.m1.1c"><ci id="A1.I1.i1.1.1.m1.1.1.cmml" xref="A1.I1.i1.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.I1.i1.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="A1.I1.i1.p1" class="ltx_para">
<p id="A1.I1.i1.p1.1" class="ltx_p">Transformer blocks <math id="A1.I1.i1.p1.1.m1.1" class="ltx_Math" alttext="L=44" display="inline"><semantics id="A1.I1.i1.p1.1.m1.1a"><mrow id="A1.I1.i1.p1.1.m1.1.1" xref="A1.I1.i1.p1.1.m1.1.1.cmml"><mi id="A1.I1.i1.p1.1.m1.1.1.2" xref="A1.I1.i1.p1.1.m1.1.1.2.cmml">L</mi><mo id="A1.I1.i1.p1.1.m1.1.1.1" xref="A1.I1.i1.p1.1.m1.1.1.1.cmml">=</mo><mn id="A1.I1.i1.p1.1.m1.1.1.3" xref="A1.I1.i1.p1.1.m1.1.1.3.cmml">44</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.I1.i1.p1.1.m1.1b"><apply id="A1.I1.i1.p1.1.m1.1.1.cmml" xref="A1.I1.i1.p1.1.m1.1.1"><eq id="A1.I1.i1.p1.1.m1.1.1.1.cmml" xref="A1.I1.i1.p1.1.m1.1.1.1"></eq><ci id="A1.I1.i1.p1.1.m1.1.1.2.cmml" xref="A1.I1.i1.p1.1.m1.1.1.2">𝐿</ci><cn type="integer" id="A1.I1.i1.p1.1.m1.1.1.3.cmml" xref="A1.I1.i1.p1.1.m1.1.1.3">44</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I1.i1.p1.1.m1.1c">L=44</annotation></semantics></math></p>
</div>
</li>
<li id="A1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="A1.I1.i2.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="A1.I1.i2.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="A1.I1.i2.1.1.m1.1.1" xref="A1.I1.i2.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="A1.I1.i2.1.1.m1.1c"><ci id="A1.I1.i2.1.1.m1.1.1.cmml" xref="A1.I1.i2.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.I1.i2.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="A1.I1.i2.p1" class="ltx_para">
<p id="A1.I1.i2.p1.2" class="ltx_p">Query heads <math id="A1.I1.i2.p1.1.m1.1" class="ltx_Math" alttext="H=40" display="inline"><semantics id="A1.I1.i2.p1.1.m1.1a"><mrow id="A1.I1.i2.p1.1.m1.1.1" xref="A1.I1.i2.p1.1.m1.1.1.cmml"><mi id="A1.I1.i2.p1.1.m1.1.1.2" xref="A1.I1.i2.p1.1.m1.1.1.2.cmml">H</mi><mo id="A1.I1.i2.p1.1.m1.1.1.1" xref="A1.I1.i2.p1.1.m1.1.1.1.cmml">=</mo><mn id="A1.I1.i2.p1.1.m1.1.1.3" xref="A1.I1.i2.p1.1.m1.1.1.3.cmml">40</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.I1.i2.p1.1.m1.1b"><apply id="A1.I1.i2.p1.1.m1.1.1.cmml" xref="A1.I1.i2.p1.1.m1.1.1"><eq id="A1.I1.i2.p1.1.m1.1.1.1.cmml" xref="A1.I1.i2.p1.1.m1.1.1.1"></eq><ci id="A1.I1.i2.p1.1.m1.1.1.2.cmml" xref="A1.I1.i2.p1.1.m1.1.1.2">𝐻</ci><cn type="integer" id="A1.I1.i2.p1.1.m1.1.1.3.cmml" xref="A1.I1.i2.p1.1.m1.1.1.3">40</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I1.i2.p1.1.m1.1c">H=40</annotation></semantics></math> and key-value heads <math id="A1.I1.i2.p1.2.m2.1" class="ltx_Math" alttext="H_{kv}=8" display="inline"><semantics id="A1.I1.i2.p1.2.m2.1a"><mrow id="A1.I1.i2.p1.2.m2.1.1" xref="A1.I1.i2.p1.2.m2.1.1.cmml"><msub id="A1.I1.i2.p1.2.m2.1.1.2" xref="A1.I1.i2.p1.2.m2.1.1.2.cmml"><mi id="A1.I1.i2.p1.2.m2.1.1.2.2" xref="A1.I1.i2.p1.2.m2.1.1.2.2.cmml">H</mi><mrow id="A1.I1.i2.p1.2.m2.1.1.2.3" xref="A1.I1.i2.p1.2.m2.1.1.2.3.cmml"><mi id="A1.I1.i2.p1.2.m2.1.1.2.3.2" xref="A1.I1.i2.p1.2.m2.1.1.2.3.2.cmml">k</mi><mo lspace="0em" rspace="0em" id="A1.I1.i2.p1.2.m2.1.1.2.3.1" xref="A1.I1.i2.p1.2.m2.1.1.2.3.1.cmml">​</mo><mi id="A1.I1.i2.p1.2.m2.1.1.2.3.3" xref="A1.I1.i2.p1.2.m2.1.1.2.3.3.cmml">v</mi></mrow></msub><mo id="A1.I1.i2.p1.2.m2.1.1.1" xref="A1.I1.i2.p1.2.m2.1.1.1.cmml">=</mo><mn id="A1.I1.i2.p1.2.m2.1.1.3" xref="A1.I1.i2.p1.2.m2.1.1.3.cmml">8</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.I1.i2.p1.2.m2.1b"><apply id="A1.I1.i2.p1.2.m2.1.1.cmml" xref="A1.I1.i2.p1.2.m2.1.1"><eq id="A1.I1.i2.p1.2.m2.1.1.1.cmml" xref="A1.I1.i2.p1.2.m2.1.1.1"></eq><apply id="A1.I1.i2.p1.2.m2.1.1.2.cmml" xref="A1.I1.i2.p1.2.m2.1.1.2"><csymbol cd="ambiguous" id="A1.I1.i2.p1.2.m2.1.1.2.1.cmml" xref="A1.I1.i2.p1.2.m2.1.1.2">subscript</csymbol><ci id="A1.I1.i2.p1.2.m2.1.1.2.2.cmml" xref="A1.I1.i2.p1.2.m2.1.1.2.2">𝐻</ci><apply id="A1.I1.i2.p1.2.m2.1.1.2.3.cmml" xref="A1.I1.i2.p1.2.m2.1.1.2.3"><times id="A1.I1.i2.p1.2.m2.1.1.2.3.1.cmml" xref="A1.I1.i2.p1.2.m2.1.1.2.3.1"></times><ci id="A1.I1.i2.p1.2.m2.1.1.2.3.2.cmml" xref="A1.I1.i2.p1.2.m2.1.1.2.3.2">𝑘</ci><ci id="A1.I1.i2.p1.2.m2.1.1.2.3.3.cmml" xref="A1.I1.i2.p1.2.m2.1.1.2.3.3">𝑣</ci></apply></apply><cn type="integer" id="A1.I1.i2.p1.2.m2.1.1.3.cmml" xref="A1.I1.i2.p1.2.m2.1.1.3">8</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I1.i2.p1.2.m2.1c">H_{kv}=8</annotation></semantics></math></p>
</div>
</li>
<li id="A1.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="A1.I1.i3.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="A1.I1.i3.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="A1.I1.i3.1.1.m1.1.1" xref="A1.I1.i3.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="A1.I1.i3.1.1.m1.1c"><ci id="A1.I1.i3.1.1.m1.1.1.cmml" xref="A1.I1.i3.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.I1.i3.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="A1.I1.i3.p1" class="ltx_para">
<p id="A1.I1.i3.p1.2" class="ltx_p">Head dimension <math id="A1.I1.i3.p1.1.m1.1" class="ltx_Math" alttext="d_{h}=80" display="inline"><semantics id="A1.I1.i3.p1.1.m1.1a"><mrow id="A1.I1.i3.p1.1.m1.1.1" xref="A1.I1.i3.p1.1.m1.1.1.cmml"><msub id="A1.I1.i3.p1.1.m1.1.1.2" xref="A1.I1.i3.p1.1.m1.1.1.2.cmml"><mi id="A1.I1.i3.p1.1.m1.1.1.2.2" xref="A1.I1.i3.p1.1.m1.1.1.2.2.cmml">d</mi><mi id="A1.I1.i3.p1.1.m1.1.1.2.3" xref="A1.I1.i3.p1.1.m1.1.1.2.3.cmml">h</mi></msub><mo id="A1.I1.i3.p1.1.m1.1.1.1" xref="A1.I1.i3.p1.1.m1.1.1.1.cmml">=</mo><mn id="A1.I1.i3.p1.1.m1.1.1.3" xref="A1.I1.i3.p1.1.m1.1.1.3.cmml">80</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.I1.i3.p1.1.m1.1b"><apply id="A1.I1.i3.p1.1.m1.1.1.cmml" xref="A1.I1.i3.p1.1.m1.1.1"><eq id="A1.I1.i3.p1.1.m1.1.1.1.cmml" xref="A1.I1.i3.p1.1.m1.1.1.1"></eq><apply id="A1.I1.i3.p1.1.m1.1.1.2.cmml" xref="A1.I1.i3.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="A1.I1.i3.p1.1.m1.1.1.2.1.cmml" xref="A1.I1.i3.p1.1.m1.1.1.2">subscript</csymbol><ci id="A1.I1.i3.p1.1.m1.1.1.2.2.cmml" xref="A1.I1.i3.p1.1.m1.1.1.2.2">𝑑</ci><ci id="A1.I1.i3.p1.1.m1.1.1.2.3.cmml" xref="A1.I1.i3.p1.1.m1.1.1.2.3">ℎ</ci></apply><cn type="integer" id="A1.I1.i3.p1.1.m1.1.1.3.cmml" xref="A1.I1.i3.p1.1.m1.1.1.3">80</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I1.i3.p1.1.m1.1c">d_{h}=80</annotation></semantics></math> and hidden dimension <math id="A1.I1.i3.p1.2.m2.1" class="ltx_Math" alttext="d=Hd_{h}=3200" display="inline"><semantics id="A1.I1.i3.p1.2.m2.1a"><mrow id="A1.I1.i3.p1.2.m2.1.1" xref="A1.I1.i3.p1.2.m2.1.1.cmml"><mi id="A1.I1.i3.p1.2.m2.1.1.2" xref="A1.I1.i3.p1.2.m2.1.1.2.cmml">d</mi><mo id="A1.I1.i3.p1.2.m2.1.1.3" xref="A1.I1.i3.p1.2.m2.1.1.3.cmml">=</mo><mrow id="A1.I1.i3.p1.2.m2.1.1.4" xref="A1.I1.i3.p1.2.m2.1.1.4.cmml"><mi id="A1.I1.i3.p1.2.m2.1.1.4.2" xref="A1.I1.i3.p1.2.m2.1.1.4.2.cmml">H</mi><mo lspace="0em" rspace="0em" id="A1.I1.i3.p1.2.m2.1.1.4.1" xref="A1.I1.i3.p1.2.m2.1.1.4.1.cmml">​</mo><msub id="A1.I1.i3.p1.2.m2.1.1.4.3" xref="A1.I1.i3.p1.2.m2.1.1.4.3.cmml"><mi id="A1.I1.i3.p1.2.m2.1.1.4.3.2" xref="A1.I1.i3.p1.2.m2.1.1.4.3.2.cmml">d</mi><mi id="A1.I1.i3.p1.2.m2.1.1.4.3.3" xref="A1.I1.i3.p1.2.m2.1.1.4.3.3.cmml">h</mi></msub></mrow><mo id="A1.I1.i3.p1.2.m2.1.1.5" xref="A1.I1.i3.p1.2.m2.1.1.5.cmml">=</mo><mn id="A1.I1.i3.p1.2.m2.1.1.6" xref="A1.I1.i3.p1.2.m2.1.1.6.cmml">3200</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.I1.i3.p1.2.m2.1b"><apply id="A1.I1.i3.p1.2.m2.1.1.cmml" xref="A1.I1.i3.p1.2.m2.1.1"><and id="A1.I1.i3.p1.2.m2.1.1a.cmml" xref="A1.I1.i3.p1.2.m2.1.1"></and><apply id="A1.I1.i3.p1.2.m2.1.1b.cmml" xref="A1.I1.i3.p1.2.m2.1.1"><eq id="A1.I1.i3.p1.2.m2.1.1.3.cmml" xref="A1.I1.i3.p1.2.m2.1.1.3"></eq><ci id="A1.I1.i3.p1.2.m2.1.1.2.cmml" xref="A1.I1.i3.p1.2.m2.1.1.2">𝑑</ci><apply id="A1.I1.i3.p1.2.m2.1.1.4.cmml" xref="A1.I1.i3.p1.2.m2.1.1.4"><times id="A1.I1.i3.p1.2.m2.1.1.4.1.cmml" xref="A1.I1.i3.p1.2.m2.1.1.4.1"></times><ci id="A1.I1.i3.p1.2.m2.1.1.4.2.cmml" xref="A1.I1.i3.p1.2.m2.1.1.4.2">𝐻</ci><apply id="A1.I1.i3.p1.2.m2.1.1.4.3.cmml" xref="A1.I1.i3.p1.2.m2.1.1.4.3"><csymbol cd="ambiguous" id="A1.I1.i3.p1.2.m2.1.1.4.3.1.cmml" xref="A1.I1.i3.p1.2.m2.1.1.4.3">subscript</csymbol><ci id="A1.I1.i3.p1.2.m2.1.1.4.3.2.cmml" xref="A1.I1.i3.p1.2.m2.1.1.4.3.2">𝑑</ci><ci id="A1.I1.i3.p1.2.m2.1.1.4.3.3.cmml" xref="A1.I1.i3.p1.2.m2.1.1.4.3.3">ℎ</ci></apply></apply></apply><apply id="A1.I1.i3.p1.2.m2.1.1c.cmml" xref="A1.I1.i3.p1.2.m2.1.1"><eq id="A1.I1.i3.p1.2.m2.1.1.5.cmml" xref="A1.I1.i3.p1.2.m2.1.1.5"></eq><share href="#A1.I1.i3.p1.2.m2.1.1.4.cmml" id="A1.I1.i3.p1.2.m2.1.1d.cmml" xref="A1.I1.i3.p1.2.m2.1.1"></share><cn type="integer" id="A1.I1.i3.p1.2.m2.1.1.6.cmml" xref="A1.I1.i3.p1.2.m2.1.1.6">3200</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I1.i3.p1.2.m2.1c">d=Hd_{h}=3200</annotation></semantics></math></p>
</div>
</li>
<li id="A1.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="A1.I1.i4.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="A1.I1.i4.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="A1.I1.i4.1.1.m1.1.1" xref="A1.I1.i4.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="A1.I1.i4.1.1.m1.1c"><ci id="A1.I1.i4.1.1.m1.1.1.cmml" xref="A1.I1.i4.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.I1.i4.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="A1.I1.i4.p1" class="ltx_para">
<p id="A1.I1.i4.p1.1" class="ltx_p">MLP width <math id="A1.I1.i4.p1.1.m1.1" class="ltx_Math" alttext="W=d" display="inline"><semantics id="A1.I1.i4.p1.1.m1.1a"><mrow id="A1.I1.i4.p1.1.m1.1.1" xref="A1.I1.i4.p1.1.m1.1.1.cmml"><mi id="A1.I1.i4.p1.1.m1.1.1.2" xref="A1.I1.i4.p1.1.m1.1.1.2.cmml">W</mi><mo id="A1.I1.i4.p1.1.m1.1.1.1" xref="A1.I1.i4.p1.1.m1.1.1.1.cmml">=</mo><mi id="A1.I1.i4.p1.1.m1.1.1.3" xref="A1.I1.i4.p1.1.m1.1.1.3.cmml">d</mi></mrow><annotation-xml encoding="MathML-Content" id="A1.I1.i4.p1.1.m1.1b"><apply id="A1.I1.i4.p1.1.m1.1.1.cmml" xref="A1.I1.i4.p1.1.m1.1.1"><eq id="A1.I1.i4.p1.1.m1.1.1.1.cmml" xref="A1.I1.i4.p1.1.m1.1.1.1"></eq><ci id="A1.I1.i4.p1.1.m1.1.1.2.cmml" xref="A1.I1.i4.p1.1.m1.1.1.2">𝑊</ci><ci id="A1.I1.i4.p1.1.m1.1.1.3.cmml" xref="A1.I1.i4.p1.1.m1.1.1.3">𝑑</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I1.i4.p1.1.m1.1c">W=d</annotation></semantics></math></p>
</div>
</li>
<li id="A1.I1.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="A1.I1.i5.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="A1.I1.i5.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="A1.I1.i5.1.1.m1.1.1" xref="A1.I1.i5.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="A1.I1.i5.1.1.m1.1c"><ci id="A1.I1.i5.1.1.m1.1.1.cmml" xref="A1.I1.i5.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.I1.i5.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="A1.I1.i5.p1" class="ltx_para">
<p id="A1.I1.i5.p1.1" class="ltx_p">Vocabulary size as well as LM head size <math id="A1.I1.i5.p1.1.m1.1" class="ltx_Math" alttext="n_{\text{vocab}}=60416" display="inline"><semantics id="A1.I1.i5.p1.1.m1.1a"><mrow id="A1.I1.i5.p1.1.m1.1.1" xref="A1.I1.i5.p1.1.m1.1.1.cmml"><msub id="A1.I1.i5.p1.1.m1.1.1.2" xref="A1.I1.i5.p1.1.m1.1.1.2.cmml"><mi id="A1.I1.i5.p1.1.m1.1.1.2.2" xref="A1.I1.i5.p1.1.m1.1.1.2.2.cmml">n</mi><mtext id="A1.I1.i5.p1.1.m1.1.1.2.3" xref="A1.I1.i5.p1.1.m1.1.1.2.3a.cmml">vocab</mtext></msub><mo id="A1.I1.i5.p1.1.m1.1.1.1" xref="A1.I1.i5.p1.1.m1.1.1.1.cmml">=</mo><mn id="A1.I1.i5.p1.1.m1.1.1.3" xref="A1.I1.i5.p1.1.m1.1.1.3.cmml">60416</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.I1.i5.p1.1.m1.1b"><apply id="A1.I1.i5.p1.1.m1.1.1.cmml" xref="A1.I1.i5.p1.1.m1.1.1"><eq id="A1.I1.i5.p1.1.m1.1.1.1.cmml" xref="A1.I1.i5.p1.1.m1.1.1.1"></eq><apply id="A1.I1.i5.p1.1.m1.1.1.2.cmml" xref="A1.I1.i5.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="A1.I1.i5.p1.1.m1.1.1.2.1.cmml" xref="A1.I1.i5.p1.1.m1.1.1.2">subscript</csymbol><ci id="A1.I1.i5.p1.1.m1.1.1.2.2.cmml" xref="A1.I1.i5.p1.1.m1.1.1.2.2">𝑛</ci><ci id="A1.I1.i5.p1.1.m1.1.1.2.3a.cmml" xref="A1.I1.i5.p1.1.m1.1.1.2.3"><mtext mathsize="70%" id="A1.I1.i5.p1.1.m1.1.1.2.3.cmml" xref="A1.I1.i5.p1.1.m1.1.1.2.3">vocab</mtext></ci></apply><cn type="integer" id="A1.I1.i5.p1.1.m1.1.1.3.cmml" xref="A1.I1.i5.p1.1.m1.1.1.3">60416</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I1.i5.p1.1.m1.1c">n_{\text{vocab}}=60416</annotation></semantics></math></p>
</div>
</li>
<li id="A1.I1.i6" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><math id="A1.I1.i6.1.1.m1.1" class="ltx_Math" alttext="\blacksquare" display="inline"><semantics id="A1.I1.i6.1.1.m1.1b"><mi mathcolor="#00868B" mathsize="70%" mathvariant="normal" id="A1.I1.i6.1.1.m1.1.1" xref="A1.I1.i6.1.1.m1.1.1.cmml">■</mi><annotation-xml encoding="MathML-Content" id="A1.I1.i6.1.1.m1.1c"><ci id="A1.I1.i6.1.1.m1.1.1.cmml" xref="A1.I1.i6.1.1.m1.1.1">■</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.I1.i6.1.1.m1.1d">\blacksquare</annotation></semantics></math></span> 
<div id="A1.I1.i6.p1" class="ltx_para">
<p id="A1.I1.i6.p1.1" class="ltx_p">memory layers <math id="A1.I1.i6.p1.1.m1.1" class="ltx_Math" alttext="L_{\text{mem}}=22" display="inline"><semantics id="A1.I1.i6.p1.1.m1.1a"><mrow id="A1.I1.i6.p1.1.m1.1.1" xref="A1.I1.i6.p1.1.m1.1.1.cmml"><msub id="A1.I1.i6.p1.1.m1.1.1.2" xref="A1.I1.i6.p1.1.m1.1.1.2.cmml"><mi id="A1.I1.i6.p1.1.m1.1.1.2.2" xref="A1.I1.i6.p1.1.m1.1.1.2.2.cmml">L</mi><mtext id="A1.I1.i6.p1.1.m1.1.1.2.3" xref="A1.I1.i6.p1.1.m1.1.1.2.3a.cmml">mem</mtext></msub><mo id="A1.I1.i6.p1.1.m1.1.1.1" xref="A1.I1.i6.p1.1.m1.1.1.1.cmml">=</mo><mn id="A1.I1.i6.p1.1.m1.1.1.3" xref="A1.I1.i6.p1.1.m1.1.1.3.cmml">22</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.I1.i6.p1.1.m1.1b"><apply id="A1.I1.i6.p1.1.m1.1.1.cmml" xref="A1.I1.i6.p1.1.m1.1.1"><eq id="A1.I1.i6.p1.1.m1.1.1.1.cmml" xref="A1.I1.i6.p1.1.m1.1.1.1"></eq><apply id="A1.I1.i6.p1.1.m1.1.1.2.cmml" xref="A1.I1.i6.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="A1.I1.i6.p1.1.m1.1.1.2.1.cmml" xref="A1.I1.i6.p1.1.m1.1.1.2">subscript</csymbol><ci id="A1.I1.i6.p1.1.m1.1.1.2.2.cmml" xref="A1.I1.i6.p1.1.m1.1.1.2.2">𝐿</ci><ci id="A1.I1.i6.p1.1.m1.1.1.2.3a.cmml" xref="A1.I1.i6.p1.1.m1.1.1.2.3"><mtext mathsize="70%" id="A1.I1.i6.p1.1.m1.1.1.2.3.cmml" xref="A1.I1.i6.p1.1.m1.1.1.2.3">mem</mtext></ci></apply><cn type="integer" id="A1.I1.i6.p1.1.m1.1.1.3.cmml" xref="A1.I1.i6.p1.1.m1.1.1.3">22</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I1.i6.p1.1.m1.1c">L_{\text{mem}}=22</annotation></semantics></math>, which is also the depth of the deepest memory layer.</p>
</div>
</li>
</ul>
<p id="A1.p2.7" class="ltx_p">Fix a separable knowledge <math id="A1.p2.2.m1.1" class="ltx_Math" alttext="\mathcal{K}" display="inline"><semantics id="A1.p2.2.m1.1a"><mi class="ltx_font_mathcaligraphic" id="A1.p2.2.m1.1.1" xref="A1.p2.2.m1.1.1.cmml">𝒦</mi><annotation-xml encoding="MathML-Content" id="A1.p2.2.m1.1b"><ci id="A1.p2.2.m1.1.1.cmml" xref="A1.p2.2.m1.1.1">𝒦</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p2.2.m1.1c">\mathcal{K}</annotation></semantics></math>, and represent it by one of its realizations <math id="A1.p2.3.m2.1" class="ltx_Math" alttext="\mathbf{t}" display="inline"><semantics id="A1.p2.3.m2.1a"><mi id="A1.p2.3.m2.1.1" xref="A1.p2.3.m2.1.1.cmml">𝐭</mi><annotation-xml encoding="MathML-Content" id="A1.p2.3.m2.1b"><ci id="A1.p2.3.m2.1.1.cmml" xref="A1.p2.3.m2.1.1">𝐭</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p2.3.m2.1c">\mathbf{t}</annotation></semantics></math> (Definition <a href="#Thmdefinition5" title="Definition 5. ‣ 2.2 | Knowledge ‣ 2 | Memory Circuitry Theory ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>), and assume that <math id="A1.p2.4.m3.1" class="ltx_Math" alttext="\mathbf{t}" display="inline"><semantics id="A1.p2.4.m3.1a"><mi id="A1.p2.4.m3.1.1" xref="A1.p2.4.m3.1.1.cmml">𝐭</mi><annotation-xml encoding="MathML-Content" id="A1.p2.4.m3.1b"><ci id="A1.p2.4.m3.1.1.cmml" xref="A1.p2.4.m3.1.1">𝐭</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p2.4.m3.1c">\mathbf{t}</annotation></semantics></math> has length <math id="A1.p2.5.m4.1" class="ltx_Math" alttext="l_{\text{ref}}=128" display="inline"><semantics id="A1.p2.5.m4.1a"><mrow id="A1.p2.5.m4.1.1" xref="A1.p2.5.m4.1.1.cmml"><msub id="A1.p2.5.m4.1.1.2" xref="A1.p2.5.m4.1.1.2.cmml"><mi id="A1.p2.5.m4.1.1.2.2" xref="A1.p2.5.m4.1.1.2.2.cmml">l</mi><mtext id="A1.p2.5.m4.1.1.2.3" xref="A1.p2.5.m4.1.1.2.3a.cmml">ref</mtext></msub><mo id="A1.p2.5.m4.1.1.1" xref="A1.p2.5.m4.1.1.1.cmml">=</mo><mn id="A1.p2.5.m4.1.1.3" xref="A1.p2.5.m4.1.1.3.cmml">128</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.p2.5.m4.1b"><apply id="A1.p2.5.m4.1.1.cmml" xref="A1.p2.5.m4.1.1"><eq id="A1.p2.5.m4.1.1.1.cmml" xref="A1.p2.5.m4.1.1.1"></eq><apply id="A1.p2.5.m4.1.1.2.cmml" xref="A1.p2.5.m4.1.1.2"><csymbol cd="ambiguous" id="A1.p2.5.m4.1.1.2.1.cmml" xref="A1.p2.5.m4.1.1.2">subscript</csymbol><ci id="A1.p2.5.m4.1.1.2.2.cmml" xref="A1.p2.5.m4.1.1.2.2">𝑙</ci><ci id="A1.p2.5.m4.1.1.2.3a.cmml" xref="A1.p2.5.m4.1.1.2.3"><mtext mathsize="70%" id="A1.p2.5.m4.1.1.2.3.cmml" xref="A1.p2.5.m4.1.1.2.3">ref</mtext></ci></apply><cn type="integer" id="A1.p2.5.m4.1.1.3.cmml" xref="A1.p2.5.m4.1.1.3">128</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p2.5.m4.1c">l_{\text{ref}}=128</annotation></semantics></math> tokens, following the setup of our reference dataset (Section <a href="#S4.SS4" title="4.4 | Knowledge Base ‣ 4 | Pretraining Data ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.4</span></a>).
Recall from Section <a href="#S3.SS3" title="3.3 | Memory Sparsification and Storage ‣ 3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3</span></a> that each memory has <math id="A1.p2.6.m5.1" class="ltx_Math" alttext="l_{\text{mem}}=8" display="inline"><semantics id="A1.p2.6.m5.1a"><mrow id="A1.p2.6.m5.1.1" xref="A1.p2.6.m5.1.1.cmml"><msub id="A1.p2.6.m5.1.1.2" xref="A1.p2.6.m5.1.1.2.cmml"><mi id="A1.p2.6.m5.1.1.2.2" xref="A1.p2.6.m5.1.1.2.2.cmml">l</mi><mtext id="A1.p2.6.m5.1.1.2.3" xref="A1.p2.6.m5.1.1.2.3a.cmml">mem</mtext></msub><mo id="A1.p2.6.m5.1.1.1" xref="A1.p2.6.m5.1.1.1.cmml">=</mo><mn id="A1.p2.6.m5.1.1.3" xref="A1.p2.6.m5.1.1.3.cmml">8</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.p2.6.m5.1b"><apply id="A1.p2.6.m5.1.1.cmml" xref="A1.p2.6.m5.1.1"><eq id="A1.p2.6.m5.1.1.1.cmml" xref="A1.p2.6.m5.1.1.1"></eq><apply id="A1.p2.6.m5.1.1.2.cmml" xref="A1.p2.6.m5.1.1.2"><csymbol cd="ambiguous" id="A1.p2.6.m5.1.1.2.1.cmml" xref="A1.p2.6.m5.1.1.2">subscript</csymbol><ci id="A1.p2.6.m5.1.1.2.2.cmml" xref="A1.p2.6.m5.1.1.2.2">𝑙</ci><ci id="A1.p2.6.m5.1.1.2.3a.cmml" xref="A1.p2.6.m5.1.1.2.3"><mtext mathsize="70%" id="A1.p2.6.m5.1.1.2.3.cmml" xref="A1.p2.6.m5.1.1.2.3">mem</mtext></ci></apply><cn type="integer" id="A1.p2.6.m5.1.1.3.cmml" xref="A1.p2.6.m5.1.1.3">8</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p2.6.m5.1c">l_{\text{mem}}=8</annotation></semantics></math> tokens per memory head, and it is read by a chunk of length <math id="A1.p2.7.m6.1" class="ltx_Math" alttext="l_{\text{chunk}}=64" display="inline"><semantics id="A1.p2.7.m6.1a"><mrow id="A1.p2.7.m6.1.1" xref="A1.p2.7.m6.1.1.cmml"><msub id="A1.p2.7.m6.1.1.2" xref="A1.p2.7.m6.1.1.2.cmml"><mi id="A1.p2.7.m6.1.1.2.2" xref="A1.p2.7.m6.1.1.2.2.cmml">l</mi><mtext id="A1.p2.7.m6.1.1.2.3" xref="A1.p2.7.m6.1.1.2.3a.cmml">chunk</mtext></msub><mo id="A1.p2.7.m6.1.1.1" xref="A1.p2.7.m6.1.1.1.cmml">=</mo><mn id="A1.p2.7.m6.1.1.3" xref="A1.p2.7.m6.1.1.3.cmml">64</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.p2.7.m6.1b"><apply id="A1.p2.7.m6.1.1.cmml" xref="A1.p2.7.m6.1.1"><eq id="A1.p2.7.m6.1.1.1.cmml" xref="A1.p2.7.m6.1.1.1"></eq><apply id="A1.p2.7.m6.1.1.2.cmml" xref="A1.p2.7.m6.1.1.2"><csymbol cd="ambiguous" id="A1.p2.7.m6.1.1.2.1.cmml" xref="A1.p2.7.m6.1.1.2">subscript</csymbol><ci id="A1.p2.7.m6.1.1.2.2.cmml" xref="A1.p2.7.m6.1.1.2.2">𝑙</ci><ci id="A1.p2.7.m6.1.1.2.3a.cmml" xref="A1.p2.7.m6.1.1.2.3"><mtext mathsize="70%" id="A1.p2.7.m6.1.1.2.3.cmml" xref="A1.p2.7.m6.1.1.2.3">chunk</mtext></ci></apply><cn type="integer" id="A1.p2.7.m6.1.1.3.cmml" xref="A1.p2.7.m6.1.1.3">64</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p2.7.m6.1c">l_{\text{chunk}}=64</annotation></semantics></math>.</p>
</div>
<div id="A1.p3" class="ltx_para">
<p id="A1.p3.1" class="ltx_p">Since we want to show that explicit memory is cheaper than implicit memory and RAG, it suffices to use coarse lower bounds on their costs.</p>
</div>
<section id="A1.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsection" style="font-size:120%;color:#00868B;">A.1 <span id="A1.SS1.1.1" class="ltx_text" style="color:#808080;">| </span>Implicit Memory</h3>

<div id="A1.SS1.p1" class="ltx_para">
<p id="A1.SS1.p1.5" class="ltx_p">The write cost of implicit memory or model parameters is the training compute with <math id="A1.SS1.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{t}" display="inline"><semantics id="A1.SS1.p1.1.m1.1a"><mi id="A1.SS1.p1.1.m1.1.1" xref="A1.SS1.p1.1.m1.1.1.cmml">𝐭</mi><annotation-xml encoding="MathML-Content" id="A1.SS1.p1.1.m1.1b"><ci id="A1.SS1.p1.1.m1.1.1.cmml" xref="A1.SS1.p1.1.m1.1.1">𝐭</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p1.1.m1.1c">\mathbf{t}</annotation></semantics></math> as input.
Usually the training data of Transformer LLMs have length <math id="A1.SS1.p1.2.m2.1" class="ltx_Math" alttext="2048\sim 8192" display="inline"><semantics id="A1.SS1.p1.2.m2.1a"><mrow id="A1.SS1.p1.2.m2.1.1" xref="A1.SS1.p1.2.m2.1.1.cmml"><mn id="A1.SS1.p1.2.m2.1.1.2" xref="A1.SS1.p1.2.m2.1.1.2.cmml">2048</mn><mo id="A1.SS1.p1.2.m2.1.1.1" xref="A1.SS1.p1.2.m2.1.1.1.cmml">∼</mo><mn id="A1.SS1.p1.2.m2.1.1.3" xref="A1.SS1.p1.2.m2.1.1.3.cmml">8192</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS1.p1.2.m2.1b"><apply id="A1.SS1.p1.2.m2.1.1.cmml" xref="A1.SS1.p1.2.m2.1.1"><csymbol cd="latexml" id="A1.SS1.p1.2.m2.1.1.1.cmml" xref="A1.SS1.p1.2.m2.1.1.1">similar-to</csymbol><cn type="integer" id="A1.SS1.p1.2.m2.1.1.2.cmml" xref="A1.SS1.p1.2.m2.1.1.2">2048</cn><cn type="integer" id="A1.SS1.p1.2.m2.1.1.3.cmml" xref="A1.SS1.p1.2.m2.1.1.3">8192</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p1.2.m2.1c">2048\sim 8192</annotation></semantics></math>, so we assume that <math id="A1.SS1.p1.3.m3.1" class="ltx_Math" alttext="\mathbf{t}" display="inline"><semantics id="A1.SS1.p1.3.m3.1a"><mi id="A1.SS1.p1.3.m3.1.1" xref="A1.SS1.p1.3.m3.1.1.cmml">𝐭</mi><annotation-xml encoding="MathML-Content" id="A1.SS1.p1.3.m3.1b"><ci id="A1.SS1.p1.3.m3.1.1.cmml" xref="A1.SS1.p1.3.m3.1.1">𝐭</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p1.3.m3.1c">\mathbf{t}</annotation></semantics></math> is a subsequence of a train sample <math id="A1.SS1.p1.4.m4.1" class="ltx_Math" alttext="\mathbf{t}_{\text{train}}" display="inline"><semantics id="A1.SS1.p1.4.m4.1a"><msub id="A1.SS1.p1.4.m4.1.1" xref="A1.SS1.p1.4.m4.1.1.cmml"><mi id="A1.SS1.p1.4.m4.1.1.2" xref="A1.SS1.p1.4.m4.1.1.2.cmml">𝐭</mi><mtext id="A1.SS1.p1.4.m4.1.1.3" xref="A1.SS1.p1.4.m4.1.1.3a.cmml">train</mtext></msub><annotation-xml encoding="MathML-Content" id="A1.SS1.p1.4.m4.1b"><apply id="A1.SS1.p1.4.m4.1.1.cmml" xref="A1.SS1.p1.4.m4.1.1"><csymbol cd="ambiguous" id="A1.SS1.p1.4.m4.1.1.1.cmml" xref="A1.SS1.p1.4.m4.1.1">subscript</csymbol><ci id="A1.SS1.p1.4.m4.1.1.2.cmml" xref="A1.SS1.p1.4.m4.1.1.2">𝐭</ci><ci id="A1.SS1.p1.4.m4.1.1.3a.cmml" xref="A1.SS1.p1.4.m4.1.1.3"><mtext mathsize="70%" id="A1.SS1.p1.4.m4.1.1.3.cmml" xref="A1.SS1.p1.4.m4.1.1.3">train</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p1.4.m4.1c">\mathbf{t}_{\text{train}}</annotation></semantics></math> with length <math id="A1.SS1.p1.5.m5.1" class="ltx_Math" alttext="l_{\text{train}}=2048" display="inline"><semantics id="A1.SS1.p1.5.m5.1a"><mrow id="A1.SS1.p1.5.m5.1.1" xref="A1.SS1.p1.5.m5.1.1.cmml"><msub id="A1.SS1.p1.5.m5.1.1.2" xref="A1.SS1.p1.5.m5.1.1.2.cmml"><mi id="A1.SS1.p1.5.m5.1.1.2.2" xref="A1.SS1.p1.5.m5.1.1.2.2.cmml">l</mi><mtext id="A1.SS1.p1.5.m5.1.1.2.3" xref="A1.SS1.p1.5.m5.1.1.2.3a.cmml">train</mtext></msub><mo id="A1.SS1.p1.5.m5.1.1.1" xref="A1.SS1.p1.5.m5.1.1.1.cmml">=</mo><mn id="A1.SS1.p1.5.m5.1.1.3" xref="A1.SS1.p1.5.m5.1.1.3.cmml">2048</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS1.p1.5.m5.1b"><apply id="A1.SS1.p1.5.m5.1.1.cmml" xref="A1.SS1.p1.5.m5.1.1"><eq id="A1.SS1.p1.5.m5.1.1.1.cmml" xref="A1.SS1.p1.5.m5.1.1.1"></eq><apply id="A1.SS1.p1.5.m5.1.1.2.cmml" xref="A1.SS1.p1.5.m5.1.1.2"><csymbol cd="ambiguous" id="A1.SS1.p1.5.m5.1.1.2.1.cmml" xref="A1.SS1.p1.5.m5.1.1.2">subscript</csymbol><ci id="A1.SS1.p1.5.m5.1.1.2.2.cmml" xref="A1.SS1.p1.5.m5.1.1.2.2">𝑙</ci><ci id="A1.SS1.p1.5.m5.1.1.2.3a.cmml" xref="A1.SS1.p1.5.m5.1.1.2.3"><mtext mathsize="70%" id="A1.SS1.p1.5.m5.1.1.2.3.cmml" xref="A1.SS1.p1.5.m5.1.1.2.3">train</mtext></ci></apply><cn type="integer" id="A1.SS1.p1.5.m5.1.1.3.cmml" xref="A1.SS1.p1.5.m5.1.1.3">2048</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p1.5.m5.1c">l_{\text{train}}=2048</annotation></semantics></math>.
By <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib84" title="" class="ltx_ref">84</a>]</cite>, the training compute of one step with one sample is approximately</p>
<table id="Ex8" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="Ex8.m1.1" class="ltx_Math" alttext="3\cdot 2\cdot\big{[}L\big{(}l_{\text{train}}(2d^{2}+2dd_{h}H_{kv}+3dW)+2\frac{l_{\text{train}}^{2}}{2}d\big{)}+l_{\text{train}}n_{\text{vocab}}d\big{]}" display="block"><semantics id="Ex8.m1.1a"><mrow id="Ex8.m1.1.1" xref="Ex8.m1.1.1.cmml"><mn id="Ex8.m1.1.1.3" xref="Ex8.m1.1.1.3.cmml">3</mn><mo lspace="0.222em" rspace="0.222em" id="Ex8.m1.1.1.2" xref="Ex8.m1.1.1.2.cmml">⋅</mo><mn id="Ex8.m1.1.1.4" xref="Ex8.m1.1.1.4.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="Ex8.m1.1.1.2a" xref="Ex8.m1.1.1.2.cmml">⋅</mo><mrow id="Ex8.m1.1.1.1.1" xref="Ex8.m1.1.1.1.2.cmml"><mo maxsize="120%" minsize="120%" id="Ex8.m1.1.1.1.1.2" xref="Ex8.m1.1.1.1.2.1.cmml">[</mo><mrow id="Ex8.m1.1.1.1.1.1" xref="Ex8.m1.1.1.1.1.1.cmml"><mrow id="Ex8.m1.1.1.1.1.1.1" xref="Ex8.m1.1.1.1.1.1.1.cmml"><mi id="Ex8.m1.1.1.1.1.1.1.3" xref="Ex8.m1.1.1.1.1.1.1.3.cmml">L</mi><mo lspace="0em" rspace="0em" id="Ex8.m1.1.1.1.1.1.1.2" xref="Ex8.m1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="Ex8.m1.1.1.1.1.1.1.1.1" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.cmml"><mo maxsize="120%" minsize="120%" id="Ex8.m1.1.1.1.1.1.1.1.1.2" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="Ex8.m1.1.1.1.1.1.1.1.1.1" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.cmml"><mrow id="Ex8.m1.1.1.1.1.1.1.1.1.1.1" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.cmml"><msub id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.3" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.3.2" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.3.2.cmml">l</mi><mtext id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.3.3" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.3.3a.cmml">train</mtext></msub><mo lspace="0em" rspace="0em" id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.2" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mrow id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml"><mn id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.2" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.1" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml">​</mo><msup id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml"><mi id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.2" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.2.cmml">d</mi><mn id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.3" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.3.cmml">2</mn></msup></mrow><mo id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">+</mo><mrow id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml"><mn id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.2" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.cmml">d</mi><mo lspace="0em" rspace="0em" id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1a" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><msub id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.cmml"><mi id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.2" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.2.cmml">d</mi><mi id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.3" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.3.cmml">h</mi></msub><mo lspace="0em" rspace="0em" id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1b" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><msub id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.cmml"><mi id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.2" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.2.cmml">H</mi><mrow id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.cmml"><mi id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.2" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.2.cmml">k</mi><mo lspace="0em" rspace="0em" id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.1" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.1.cmml">​</mo><mi id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.3" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.3.cmml">v</mi></mrow></msub></mrow><mo id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1a" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">+</mo><mrow id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.cmml"><mn id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.2" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.2.cmml">3</mn><mo lspace="0em" rspace="0em" id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.1" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.1.cmml">​</mo><mi id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.3" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.3.cmml">d</mi><mo lspace="0em" rspace="0em" id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.1a" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.1.cmml">​</mo><mi id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.4" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.4.cmml">W</mi></mrow></mrow><mo stretchy="false" id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="Ex8.m1.1.1.1.1.1.1.1.1.1.2" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.2.cmml">+</mo><mrow id="Ex8.m1.1.1.1.1.1.1.1.1.1.3" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.3.cmml"><mn id="Ex8.m1.1.1.1.1.1.1.1.1.1.3.2" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="Ex8.m1.1.1.1.1.1.1.1.1.1.3.1" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><mfrac id="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.cmml"><msubsup id="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.2" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.2.cmml"><mi id="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.2.2.2" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.2.2.2.cmml">l</mi><mtext id="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.2.2.3" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.2.2.3a.cmml">train</mtext><mn id="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.2.3" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.2.3.cmml">2</mn></msubsup><mn id="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.3" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.3.cmml">2</mn></mfrac><mo lspace="0em" rspace="0em" id="Ex8.m1.1.1.1.1.1.1.1.1.1.3.1a" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="Ex8.m1.1.1.1.1.1.1.1.1.1.3.4" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.3.4.cmml">d</mi></mrow></mrow><mo maxsize="120%" minsize="120%" id="Ex8.m1.1.1.1.1.1.1.1.1.3" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="Ex8.m1.1.1.1.1.1.2" xref="Ex8.m1.1.1.1.1.1.2.cmml">+</mo><mrow id="Ex8.m1.1.1.1.1.1.3" xref="Ex8.m1.1.1.1.1.1.3.cmml"><msub id="Ex8.m1.1.1.1.1.1.3.2" xref="Ex8.m1.1.1.1.1.1.3.2.cmml"><mi id="Ex8.m1.1.1.1.1.1.3.2.2" xref="Ex8.m1.1.1.1.1.1.3.2.2.cmml">l</mi><mtext id="Ex8.m1.1.1.1.1.1.3.2.3" xref="Ex8.m1.1.1.1.1.1.3.2.3a.cmml">train</mtext></msub><mo lspace="0em" rspace="0em" id="Ex8.m1.1.1.1.1.1.3.1" xref="Ex8.m1.1.1.1.1.1.3.1.cmml">​</mo><msub id="Ex8.m1.1.1.1.1.1.3.3" xref="Ex8.m1.1.1.1.1.1.3.3.cmml"><mi id="Ex8.m1.1.1.1.1.1.3.3.2" xref="Ex8.m1.1.1.1.1.1.3.3.2.cmml">n</mi><mtext id="Ex8.m1.1.1.1.1.1.3.3.3" xref="Ex8.m1.1.1.1.1.1.3.3.3a.cmml">vocab</mtext></msub><mo lspace="0em" rspace="0em" id="Ex8.m1.1.1.1.1.1.3.1a" xref="Ex8.m1.1.1.1.1.1.3.1.cmml">​</mo><mi id="Ex8.m1.1.1.1.1.1.3.4" xref="Ex8.m1.1.1.1.1.1.3.4.cmml">d</mi></mrow></mrow><mo maxsize="120%" minsize="120%" id="Ex8.m1.1.1.1.1.3" xref="Ex8.m1.1.1.1.2.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="Ex8.m1.1b"><apply id="Ex8.m1.1.1.cmml" xref="Ex8.m1.1.1"><ci id="Ex8.m1.1.1.2.cmml" xref="Ex8.m1.1.1.2">⋅</ci><cn type="integer" id="Ex8.m1.1.1.3.cmml" xref="Ex8.m1.1.1.3">3</cn><cn type="integer" id="Ex8.m1.1.1.4.cmml" xref="Ex8.m1.1.1.4">2</cn><apply id="Ex8.m1.1.1.1.2.cmml" xref="Ex8.m1.1.1.1.1"><csymbol cd="latexml" id="Ex8.m1.1.1.1.2.1.cmml" xref="Ex8.m1.1.1.1.1.2">delimited-[]</csymbol><apply id="Ex8.m1.1.1.1.1.1.cmml" xref="Ex8.m1.1.1.1.1.1"><plus id="Ex8.m1.1.1.1.1.1.2.cmml" xref="Ex8.m1.1.1.1.1.1.2"></plus><apply id="Ex8.m1.1.1.1.1.1.1.cmml" xref="Ex8.m1.1.1.1.1.1.1"><times id="Ex8.m1.1.1.1.1.1.1.2.cmml" xref="Ex8.m1.1.1.1.1.1.1.2"></times><ci id="Ex8.m1.1.1.1.1.1.1.3.cmml" xref="Ex8.m1.1.1.1.1.1.1.3">𝐿</ci><apply id="Ex8.m1.1.1.1.1.1.1.1.1.1.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1"><plus id="Ex8.m1.1.1.1.1.1.1.1.1.1.2.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.2"></plus><apply id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1"><times id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.2"></times><apply id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.3.2">𝑙</ci><ci id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.3.3a.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.3.3"><mtext mathsize="70%" id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.3.3">train</mtext></ci></apply><apply id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1"><plus id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1"></plus><apply id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2"><times id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.1"></times><cn type="integer" id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.2">2</cn><apply id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.1.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3">superscript</csymbol><ci id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.2.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.2">𝑑</ci><cn type="integer" id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.3.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.3">2</cn></apply></apply><apply id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3"><times id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1"></times><cn type="integer" id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.2">2</cn><ci id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3">𝑑</ci><apply id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4"><csymbol cd="ambiguous" id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.1.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4">subscript</csymbol><ci id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.2.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.2">𝑑</ci><ci id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.3.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.3">ℎ</ci></apply><apply id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5"><csymbol cd="ambiguous" id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.1.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5">subscript</csymbol><ci id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.2.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.2">𝐻</ci><apply id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3"><times id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.1.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.1"></times><ci id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.2.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.2">𝑘</ci><ci id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.3.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.3">𝑣</ci></apply></apply></apply><apply id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4"><times id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.1.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.1"></times><cn type="integer" id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.2.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.2">3</cn><ci id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.3.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.3">𝑑</ci><ci id="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.4.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.4">𝑊</ci></apply></apply></apply><apply id="Ex8.m1.1.1.1.1.1.1.1.1.1.3.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.3"><times id="Ex8.m1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.3.1"></times><cn type="integer" id="Ex8.m1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.3.2">2</cn><apply id="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3"><divide id="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.1.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3"></divide><apply id="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.2.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.2"><csymbol cd="ambiguous" id="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.2.1.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.2">superscript</csymbol><apply id="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.2.2.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.2"><csymbol cd="ambiguous" id="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.2.2.1.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.2">subscript</csymbol><ci id="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.2.2.2.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.2.2.2">𝑙</ci><ci id="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.2.2.3a.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.2.2.3"><mtext mathsize="70%" id="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.2.2.3.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.2.2.3">train</mtext></ci></apply><cn type="integer" id="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.2.3.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.2.3">2</cn></apply><cn type="integer" id="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.3.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.3.3.3">2</cn></apply><ci id="Ex8.m1.1.1.1.1.1.1.1.1.1.3.4.cmml" xref="Ex8.m1.1.1.1.1.1.1.1.1.1.3.4">𝑑</ci></apply></apply></apply><apply id="Ex8.m1.1.1.1.1.1.3.cmml" xref="Ex8.m1.1.1.1.1.1.3"><times id="Ex8.m1.1.1.1.1.1.3.1.cmml" xref="Ex8.m1.1.1.1.1.1.3.1"></times><apply id="Ex8.m1.1.1.1.1.1.3.2.cmml" xref="Ex8.m1.1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="Ex8.m1.1.1.1.1.1.3.2.1.cmml" xref="Ex8.m1.1.1.1.1.1.3.2">subscript</csymbol><ci id="Ex8.m1.1.1.1.1.1.3.2.2.cmml" xref="Ex8.m1.1.1.1.1.1.3.2.2">𝑙</ci><ci id="Ex8.m1.1.1.1.1.1.3.2.3a.cmml" xref="Ex8.m1.1.1.1.1.1.3.2.3"><mtext mathsize="70%" id="Ex8.m1.1.1.1.1.1.3.2.3.cmml" xref="Ex8.m1.1.1.1.1.1.3.2.3">train</mtext></ci></apply><apply id="Ex8.m1.1.1.1.1.1.3.3.cmml" xref="Ex8.m1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="Ex8.m1.1.1.1.1.1.3.3.1.cmml" xref="Ex8.m1.1.1.1.1.1.3.3">subscript</csymbol><ci id="Ex8.m1.1.1.1.1.1.3.3.2.cmml" xref="Ex8.m1.1.1.1.1.1.3.3.2">𝑛</ci><ci id="Ex8.m1.1.1.1.1.1.3.3.3a.cmml" xref="Ex8.m1.1.1.1.1.1.3.3.3"><mtext mathsize="70%" id="Ex8.m1.1.1.1.1.1.3.3.3.cmml" xref="Ex8.m1.1.1.1.1.1.3.3.3">vocab</mtext></ci></apply><ci id="Ex8.m1.1.1.1.1.1.3.4.cmml" xref="Ex8.m1.1.1.1.1.1.3.4">𝑑</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Ex8.m1.1c">3\cdot 2\cdot\big{[}L\big{(}l_{\text{train}}(2d^{2}+2dd_{h}H_{kv}+3dW)+2\frac{l_{\text{train}}^{2}}{2}d\big{)}+l_{\text{train}}n_{\text{vocab}}d\big{]}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="A1.SS1.p1.8" class="ltx_p">where <math id="A1.SS1.p1.6.m1.1" class="ltx_Math" alttext="3" display="inline"><semantics id="A1.SS1.p1.6.m1.1a"><mn id="A1.SS1.p1.6.m1.1.1" xref="A1.SS1.p1.6.m1.1.1.cmml">3</mn><annotation-xml encoding="MathML-Content" id="A1.SS1.p1.6.m1.1b"><cn type="integer" id="A1.SS1.p1.6.m1.1.1.cmml" xref="A1.SS1.p1.6.m1.1.1">3</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p1.6.m1.1c">3</annotation></semantics></math> means that the backward step costs twice as the forward step (and thus 3 times in total), the first <math id="A1.SS1.p1.7.m2.1" class="ltx_Math" alttext="2" display="inline"><semantics id="A1.SS1.p1.7.m2.1a"><mn id="A1.SS1.p1.7.m2.1.1" xref="A1.SS1.p1.7.m2.1.1.cmml">2</mn><annotation-xml encoding="MathML-Content" id="A1.SS1.p1.7.m2.1b"><cn type="integer" id="A1.SS1.p1.7.m2.1.1.cmml" xref="A1.SS1.p1.7.m2.1.1">2</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p1.7.m2.1c">2</annotation></semantics></math> means that the compute of matrix multiplication involves same amount of additions and multiplications.
The five terms in the bracket come from QO embedding, KV embedding, MLP, attention, and LM head, respectively.
The lower order terms, such as layer normalizations, are omitted.
The fraction of the compute attributable to <math id="A1.SS1.p1.8.m3.1" class="ltx_Math" alttext="\mathbf{t}" display="inline"><semantics id="A1.SS1.p1.8.m3.1a"><mi id="A1.SS1.p1.8.m3.1.1" xref="A1.SS1.p1.8.m3.1.1.cmml">𝐭</mi><annotation-xml encoding="MathML-Content" id="A1.SS1.p1.8.m3.1b"><ci id="A1.SS1.p1.8.m3.1.1.cmml" xref="A1.SS1.p1.8.m3.1.1">𝐭</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p1.8.m3.1c">\mathbf{t}</annotation></semantics></math> is given by</p>
<table id="Ex9" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="Ex9.m1.1" class="ltx_Math" alttext="3\cdot 2\cdot\big{[}L\big{(}l_{\text{ref}}(2d^{2}+2dd_{h}H_{kv}+3dW)+2l_{\text{ref}}\frac{l_{\text{train}}}{2}d\big{)}+l_{\text{ref}}n_{\text{vocab}}d\big{]}" display="block"><semantics id="Ex9.m1.1a"><mrow id="Ex9.m1.1.1" xref="Ex9.m1.1.1.cmml"><mn id="Ex9.m1.1.1.3" xref="Ex9.m1.1.1.3.cmml">3</mn><mo lspace="0.222em" rspace="0.222em" id="Ex9.m1.1.1.2" xref="Ex9.m1.1.1.2.cmml">⋅</mo><mn id="Ex9.m1.1.1.4" xref="Ex9.m1.1.1.4.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="Ex9.m1.1.1.2a" xref="Ex9.m1.1.1.2.cmml">⋅</mo><mrow id="Ex9.m1.1.1.1.1" xref="Ex9.m1.1.1.1.2.cmml"><mo maxsize="120%" minsize="120%" id="Ex9.m1.1.1.1.1.2" xref="Ex9.m1.1.1.1.2.1.cmml">[</mo><mrow id="Ex9.m1.1.1.1.1.1" xref="Ex9.m1.1.1.1.1.1.cmml"><mrow id="Ex9.m1.1.1.1.1.1.1" xref="Ex9.m1.1.1.1.1.1.1.cmml"><mi id="Ex9.m1.1.1.1.1.1.1.3" xref="Ex9.m1.1.1.1.1.1.1.3.cmml">L</mi><mo lspace="0em" rspace="0em" id="Ex9.m1.1.1.1.1.1.1.2" xref="Ex9.m1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="Ex9.m1.1.1.1.1.1.1.1.1" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.cmml"><mo maxsize="120%" minsize="120%" id="Ex9.m1.1.1.1.1.1.1.1.1.2" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="Ex9.m1.1.1.1.1.1.1.1.1.1" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.cmml"><mrow id="Ex9.m1.1.1.1.1.1.1.1.1.1.1" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.cmml"><msub id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.3" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.3.2" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.3.2.cmml">l</mi><mtext id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.3.3" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.3.3a.cmml">ref</mtext></msub><mo lspace="0em" rspace="0em" id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.2" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mrow id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml"><mn id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.2" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.1" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml">​</mo><msup id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml"><mi id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.2" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.2.cmml">d</mi><mn id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.3" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.3.cmml">2</mn></msup></mrow><mo id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">+</mo><mrow id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml"><mn id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.2" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.cmml">d</mi><mo lspace="0em" rspace="0em" id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1a" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><msub id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.cmml"><mi id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.2" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.2.cmml">d</mi><mi id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.3" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.3.cmml">h</mi></msub><mo lspace="0em" rspace="0em" id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1b" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><msub id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.cmml"><mi id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.2" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.2.cmml">H</mi><mrow id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.cmml"><mi id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.2" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.2.cmml">k</mi><mo lspace="0em" rspace="0em" id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.1" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.1.cmml">​</mo><mi id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.3" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.3.cmml">v</mi></mrow></msub></mrow><mo id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1a" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">+</mo><mrow id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.cmml"><mn id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.2" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.2.cmml">3</mn><mo lspace="0em" rspace="0em" id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.1" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.1.cmml">​</mo><mi id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.3" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.3.cmml">d</mi><mo lspace="0em" rspace="0em" id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.1a" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.1.cmml">​</mo><mi id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.4" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.4.cmml">W</mi></mrow></mrow><mo stretchy="false" id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="Ex9.m1.1.1.1.1.1.1.1.1.1.2" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.2.cmml">+</mo><mrow id="Ex9.m1.1.1.1.1.1.1.1.1.1.3" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.cmml"><mn id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.2" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.1" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><msub id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.3" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.3.cmml"><mi id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.3.2" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.3.2.cmml">l</mi><mtext id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.3.3" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.3.3a.cmml">ref</mtext></msub><mo lspace="0em" rspace="0em" id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.1a" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><mfrac id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.4" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.4.cmml"><msub id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.4.2" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.4.2.cmml"><mi id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.4.2.2" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.4.2.2.cmml">l</mi><mtext id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.4.2.3" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.4.2.3a.cmml">train</mtext></msub><mn id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.4.3" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.4.3.cmml">2</mn></mfrac><mo lspace="0em" rspace="0em" id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.1b" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.5" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.5.cmml">d</mi></mrow></mrow><mo maxsize="120%" minsize="120%" id="Ex9.m1.1.1.1.1.1.1.1.1.3" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="Ex9.m1.1.1.1.1.1.2" xref="Ex9.m1.1.1.1.1.1.2.cmml">+</mo><mrow id="Ex9.m1.1.1.1.1.1.3" xref="Ex9.m1.1.1.1.1.1.3.cmml"><msub id="Ex9.m1.1.1.1.1.1.3.2" xref="Ex9.m1.1.1.1.1.1.3.2.cmml"><mi id="Ex9.m1.1.1.1.1.1.3.2.2" xref="Ex9.m1.1.1.1.1.1.3.2.2.cmml">l</mi><mtext id="Ex9.m1.1.1.1.1.1.3.2.3" xref="Ex9.m1.1.1.1.1.1.3.2.3a.cmml">ref</mtext></msub><mo lspace="0em" rspace="0em" id="Ex9.m1.1.1.1.1.1.3.1" xref="Ex9.m1.1.1.1.1.1.3.1.cmml">​</mo><msub id="Ex9.m1.1.1.1.1.1.3.3" xref="Ex9.m1.1.1.1.1.1.3.3.cmml"><mi id="Ex9.m1.1.1.1.1.1.3.3.2" xref="Ex9.m1.1.1.1.1.1.3.3.2.cmml">n</mi><mtext id="Ex9.m1.1.1.1.1.1.3.3.3" xref="Ex9.m1.1.1.1.1.1.3.3.3a.cmml">vocab</mtext></msub><mo lspace="0em" rspace="0em" id="Ex9.m1.1.1.1.1.1.3.1a" xref="Ex9.m1.1.1.1.1.1.3.1.cmml">​</mo><mi id="Ex9.m1.1.1.1.1.1.3.4" xref="Ex9.m1.1.1.1.1.1.3.4.cmml">d</mi></mrow></mrow><mo maxsize="120%" minsize="120%" id="Ex9.m1.1.1.1.1.3" xref="Ex9.m1.1.1.1.2.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="Ex9.m1.1b"><apply id="Ex9.m1.1.1.cmml" xref="Ex9.m1.1.1"><ci id="Ex9.m1.1.1.2.cmml" xref="Ex9.m1.1.1.2">⋅</ci><cn type="integer" id="Ex9.m1.1.1.3.cmml" xref="Ex9.m1.1.1.3">3</cn><cn type="integer" id="Ex9.m1.1.1.4.cmml" xref="Ex9.m1.1.1.4">2</cn><apply id="Ex9.m1.1.1.1.2.cmml" xref="Ex9.m1.1.1.1.1"><csymbol cd="latexml" id="Ex9.m1.1.1.1.2.1.cmml" xref="Ex9.m1.1.1.1.1.2">delimited-[]</csymbol><apply id="Ex9.m1.1.1.1.1.1.cmml" xref="Ex9.m1.1.1.1.1.1"><plus id="Ex9.m1.1.1.1.1.1.2.cmml" xref="Ex9.m1.1.1.1.1.1.2"></plus><apply id="Ex9.m1.1.1.1.1.1.1.cmml" xref="Ex9.m1.1.1.1.1.1.1"><times id="Ex9.m1.1.1.1.1.1.1.2.cmml" xref="Ex9.m1.1.1.1.1.1.1.2"></times><ci id="Ex9.m1.1.1.1.1.1.1.3.cmml" xref="Ex9.m1.1.1.1.1.1.1.3">𝐿</ci><apply id="Ex9.m1.1.1.1.1.1.1.1.1.1.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1"><plus id="Ex9.m1.1.1.1.1.1.1.1.1.1.2.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.2"></plus><apply id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1"><times id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.2"></times><apply id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.3.2">𝑙</ci><ci id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.3.3a.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.3.3"><mtext mathsize="70%" id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.3.3">ref</mtext></ci></apply><apply id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1"><plus id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1"></plus><apply id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2"><times id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.1"></times><cn type="integer" id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.2">2</cn><apply id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.1.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3">superscript</csymbol><ci id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.2.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.2">𝑑</ci><cn type="integer" id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.3.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.3">2</cn></apply></apply><apply id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3"><times id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1"></times><cn type="integer" id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.2">2</cn><ci id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3">𝑑</ci><apply id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4"><csymbol cd="ambiguous" id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.1.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4">subscript</csymbol><ci id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.2.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.2">𝑑</ci><ci id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.3.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.3">ℎ</ci></apply><apply id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5"><csymbol cd="ambiguous" id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.1.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5">subscript</csymbol><ci id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.2.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.2">𝐻</ci><apply id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3"><times id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.1.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.1"></times><ci id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.2.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.2">𝑘</ci><ci id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.3.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.3">𝑣</ci></apply></apply></apply><apply id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4"><times id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.1.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.1"></times><cn type="integer" id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.2.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.2">3</cn><ci id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.3.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.3">𝑑</ci><ci id="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.4.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.4.4">𝑊</ci></apply></apply></apply><apply id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3"><times id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.1"></times><cn type="integer" id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.2">2</cn><apply id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.3.1.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.3">subscript</csymbol><ci id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.3.2.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.3.2">𝑙</ci><ci id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.3.3a.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.3.3"><mtext mathsize="70%" id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.3.3.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.3.3">ref</mtext></ci></apply><apply id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.4.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.4"><divide id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.4.1.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.4"></divide><apply id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.4.2.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.4.2"><csymbol cd="ambiguous" id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.4.2.1.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.4.2">subscript</csymbol><ci id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.4.2.2.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.4.2.2">𝑙</ci><ci id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.4.2.3a.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.4.2.3"><mtext mathsize="70%" id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.4.2.3.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.4.2.3">train</mtext></ci></apply><cn type="integer" id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.4.3.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.4.3">2</cn></apply><ci id="Ex9.m1.1.1.1.1.1.1.1.1.1.3.5.cmml" xref="Ex9.m1.1.1.1.1.1.1.1.1.1.3.5">𝑑</ci></apply></apply></apply><apply id="Ex9.m1.1.1.1.1.1.3.cmml" xref="Ex9.m1.1.1.1.1.1.3"><times id="Ex9.m1.1.1.1.1.1.3.1.cmml" xref="Ex9.m1.1.1.1.1.1.3.1"></times><apply id="Ex9.m1.1.1.1.1.1.3.2.cmml" xref="Ex9.m1.1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="Ex9.m1.1.1.1.1.1.3.2.1.cmml" xref="Ex9.m1.1.1.1.1.1.3.2">subscript</csymbol><ci id="Ex9.m1.1.1.1.1.1.3.2.2.cmml" xref="Ex9.m1.1.1.1.1.1.3.2.2">𝑙</ci><ci id="Ex9.m1.1.1.1.1.1.3.2.3a.cmml" xref="Ex9.m1.1.1.1.1.1.3.2.3"><mtext mathsize="70%" id="Ex9.m1.1.1.1.1.1.3.2.3.cmml" xref="Ex9.m1.1.1.1.1.1.3.2.3">ref</mtext></ci></apply><apply id="Ex9.m1.1.1.1.1.1.3.3.cmml" xref="Ex9.m1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="Ex9.m1.1.1.1.1.1.3.3.1.cmml" xref="Ex9.m1.1.1.1.1.1.3.3">subscript</csymbol><ci id="Ex9.m1.1.1.1.1.1.3.3.2.cmml" xref="Ex9.m1.1.1.1.1.1.3.3.2">𝑛</ci><ci id="Ex9.m1.1.1.1.1.1.3.3.3a.cmml" xref="Ex9.m1.1.1.1.1.1.3.3.3"><mtext mathsize="70%" id="Ex9.m1.1.1.1.1.1.3.3.3.cmml" xref="Ex9.m1.1.1.1.1.1.3.3.3">vocab</mtext></ci></apply><ci id="Ex9.m1.1.1.1.1.1.3.4.cmml" xref="Ex9.m1.1.1.1.1.1.3.4">𝑑</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Ex9.m1.1c">3\cdot 2\cdot\big{[}L\big{(}l_{\text{ref}}(2d^{2}+2dd_{h}H_{kv}+3dW)+2l_{\text{ref}}\frac{l_{\text{train}}}{2}d\big{)}+l_{\text{ref}}n_{\text{vocab}}d\big{]}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="A1.SS1.p1.9" class="ltx_p">Assume that one training step is sufficient for storing knowledge <math id="A1.SS1.p1.9.m1.1" class="ltx_Math" alttext="\mathcal{K}" display="inline"><semantics id="A1.SS1.p1.9.m1.1a"><mi class="ltx_font_mathcaligraphic" id="A1.SS1.p1.9.m1.1.1" xref="A1.SS1.p1.9.m1.1.1.cmml">𝒦</mi><annotation-xml encoding="MathML-Content" id="A1.SS1.p1.9.m1.1b"><ci id="A1.SS1.p1.9.m1.1.1.cmml" xref="A1.SS1.p1.9.m1.1.1">𝒦</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p1.9.m1.1c">\mathcal{K}</annotation></semantics></math> into model parameters.
Then, the write cost is equal to the above term, and we obtain</p>
<table id="Ex10" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="Ex10.m1.1" class="ltx_Math" alttext="\text{cost}_{\text{write}}\approx 2.24~{}\text{TFlops}" display="block"><semantics id="Ex10.m1.1a"><mrow id="Ex10.m1.1.1" xref="Ex10.m1.1.1.cmml"><msub id="Ex10.m1.1.1.2" xref="Ex10.m1.1.1.2.cmml"><mtext id="Ex10.m1.1.1.2.2" xref="Ex10.m1.1.1.2.2a.cmml">cost</mtext><mtext id="Ex10.m1.1.1.2.3" xref="Ex10.m1.1.1.2.3a.cmml">write</mtext></msub><mo id="Ex10.m1.1.1.1" xref="Ex10.m1.1.1.1.cmml">≈</mo><mrow id="Ex10.m1.1.1.3" xref="Ex10.m1.1.1.3.cmml"><mn id="Ex10.m1.1.1.3.2" xref="Ex10.m1.1.1.3.2.cmml">2.24</mn><mo lspace="0.330em" rspace="0em" id="Ex10.m1.1.1.3.1" xref="Ex10.m1.1.1.3.1.cmml">​</mo><mtext id="Ex10.m1.1.1.3.3" xref="Ex10.m1.1.1.3.3a.cmml">TFlops</mtext></mrow></mrow><annotation-xml encoding="MathML-Content" id="Ex10.m1.1b"><apply id="Ex10.m1.1.1.cmml" xref="Ex10.m1.1.1"><approx id="Ex10.m1.1.1.1.cmml" xref="Ex10.m1.1.1.1"></approx><apply id="Ex10.m1.1.1.2.cmml" xref="Ex10.m1.1.1.2"><csymbol cd="ambiguous" id="Ex10.m1.1.1.2.1.cmml" xref="Ex10.m1.1.1.2">subscript</csymbol><ci id="Ex10.m1.1.1.2.2a.cmml" xref="Ex10.m1.1.1.2.2"><mtext id="Ex10.m1.1.1.2.2.cmml" xref="Ex10.m1.1.1.2.2">cost</mtext></ci><ci id="Ex10.m1.1.1.2.3a.cmml" xref="Ex10.m1.1.1.2.3"><mtext mathsize="70%" id="Ex10.m1.1.1.2.3.cmml" xref="Ex10.m1.1.1.2.3">write</mtext></ci></apply><apply id="Ex10.m1.1.1.3.cmml" xref="Ex10.m1.1.1.3"><times id="Ex10.m1.1.1.3.1.cmml" xref="Ex10.m1.1.1.3.1"></times><cn type="float" id="Ex10.m1.1.1.3.2.cmml" xref="Ex10.m1.1.1.3.2">2.24</cn><ci id="Ex10.m1.1.1.3.3a.cmml" xref="Ex10.m1.1.1.3.3"><mtext id="Ex10.m1.1.1.3.3.cmml" xref="Ex10.m1.1.1.3.3">TFlops</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Ex10.m1.1c">\text{cost}_{\text{write}}\approx 2.24~{}\text{TFlops}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="A1.SS1.p2" class="ltx_para">
<p id="A1.SS1.p2.2" class="ltx_p">Meanwhile, we lower bound the read cost by zero.</p>
<table id="Ex11" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="Ex11.m1.1" class="ltx_Math" alttext="\text{cost}_{\text{read}}\geq 0~{}\text{TFlops}" display="block"><semantics id="Ex11.m1.1a"><mrow id="Ex11.m1.1.1" xref="Ex11.m1.1.1.cmml"><msub id="Ex11.m1.1.1.2" xref="Ex11.m1.1.1.2.cmml"><mtext id="Ex11.m1.1.1.2.2" xref="Ex11.m1.1.1.2.2a.cmml">cost</mtext><mtext id="Ex11.m1.1.1.2.3" xref="Ex11.m1.1.1.2.3a.cmml">read</mtext></msub><mo id="Ex11.m1.1.1.1" xref="Ex11.m1.1.1.1.cmml">≥</mo><mrow id="Ex11.m1.1.1.3" xref="Ex11.m1.1.1.3.cmml"><mn id="Ex11.m1.1.1.3.2" xref="Ex11.m1.1.1.3.2.cmml">0</mn><mo lspace="0.330em" rspace="0em" id="Ex11.m1.1.1.3.1" xref="Ex11.m1.1.1.3.1.cmml">​</mo><mtext id="Ex11.m1.1.1.3.3" xref="Ex11.m1.1.1.3.3a.cmml">TFlops</mtext></mrow></mrow><annotation-xml encoding="MathML-Content" id="Ex11.m1.1b"><apply id="Ex11.m1.1.1.cmml" xref="Ex11.m1.1.1"><geq id="Ex11.m1.1.1.1.cmml" xref="Ex11.m1.1.1.1"></geq><apply id="Ex11.m1.1.1.2.cmml" xref="Ex11.m1.1.1.2"><csymbol cd="ambiguous" id="Ex11.m1.1.1.2.1.cmml" xref="Ex11.m1.1.1.2">subscript</csymbol><ci id="Ex11.m1.1.1.2.2a.cmml" xref="Ex11.m1.1.1.2.2"><mtext id="Ex11.m1.1.1.2.2.cmml" xref="Ex11.m1.1.1.2.2">cost</mtext></ci><ci id="Ex11.m1.1.1.2.3a.cmml" xref="Ex11.m1.1.1.2.3"><mtext mathsize="70%" id="Ex11.m1.1.1.2.3.cmml" xref="Ex11.m1.1.1.2.3">read</mtext></ci></apply><apply id="Ex11.m1.1.1.3.cmml" xref="Ex11.m1.1.1.3"><times id="Ex11.m1.1.1.3.1.cmml" xref="Ex11.m1.1.1.3.1"></times><cn type="integer" id="Ex11.m1.1.1.3.2.cmml" xref="Ex11.m1.1.1.3.2">0</cn><ci id="Ex11.m1.1.1.3.3a.cmml" xref="Ex11.m1.1.1.3.3"><mtext id="Ex11.m1.1.1.3.3.cmml" xref="Ex11.m1.1.1.3.3">TFlops</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Ex11.m1.1c">\text{cost}_{\text{read}}\geq 0~{}\text{TFlops}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="A1.SS1.p2.1" class="ltx_p">This lower bound is obviously correct and suits our comparison, since it makes implicit memory appear more competitive.
The difficulty in estimating the cost is that the correspondence between knowledges and parameters is not fully understood.
Nevertheless, we describe a possible way to obtain a reasonable bound.
Recall from Section <a href="#S1" title="1 | Introduction ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> that the model parameters suffer from the issue of knowledge traversal such that each parameter (and thus each implicit memory) is invoked during each call of the LLM.
So the read cost of each implicit memory does not depend on its usage count <math id="A1.SS1.p2.1.m1.1" class="ltx_Math" alttext="n_{k}" display="inline"><semantics id="A1.SS1.p2.1.m1.1a"><msub id="A1.SS1.p2.1.m1.1.1" xref="A1.SS1.p2.1.m1.1.1.cmml"><mi id="A1.SS1.p2.1.m1.1.1.2" xref="A1.SS1.p2.1.m1.1.1.2.cmml">n</mi><mi id="A1.SS1.p2.1.m1.1.1.3" xref="A1.SS1.p2.1.m1.1.1.3.cmml">k</mi></msub><annotation-xml encoding="MathML-Content" id="A1.SS1.p2.1.m1.1b"><apply id="A1.SS1.p2.1.m1.1.1.cmml" xref="A1.SS1.p2.1.m1.1.1"><csymbol cd="ambiguous" id="A1.SS1.p2.1.m1.1.1.1.cmml" xref="A1.SS1.p2.1.m1.1.1">subscript</csymbol><ci id="A1.SS1.p2.1.m1.1.1.2.cmml" xref="A1.SS1.p2.1.m1.1.1.2">𝑛</ci><ci id="A1.SS1.p2.1.m1.1.1.3.cmml" xref="A1.SS1.p2.1.m1.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p2.1.m1.1c">n_{k}</annotation></semantics></math>, but instead on the total amount of model calls during the lifespan of this LLM.
Dividing the total amount of inference compute used by this LLM by the amount of knowledges it possesses gives an estimation of the average read cost of a knowledge.
The amount of knowledges in the LLM can be upper bounded based on the knowledge capacities measured by <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib7" title="" class="ltx_ref">7</a>]</cite>.</p>
</div>
</section>
<section id="A1.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsection" style="font-size:120%;color:#00868B;">A.2 <span id="A1.SS2.1.1" class="ltx_text" style="color:#808080;">| </span>Explicit Memory</h3>

<div id="A1.SS2.p1" class="ltx_para">
<p id="A1.SS2.p1.3" class="ltx_p">The write cost of an each explicit memory mainly comes from <math id="A1.SS2.p1.1.m1.1" class="ltx_Math" alttext="L_{\text{mem}}" display="inline"><semantics id="A1.SS2.p1.1.m1.1a"><msub id="A1.SS2.p1.1.m1.1.1" xref="A1.SS2.p1.1.m1.1.1.cmml"><mi id="A1.SS2.p1.1.m1.1.1.2" xref="A1.SS2.p1.1.m1.1.1.2.cmml">L</mi><mtext id="A1.SS2.p1.1.m1.1.1.3" xref="A1.SS2.p1.1.m1.1.1.3a.cmml">mem</mtext></msub><annotation-xml encoding="MathML-Content" id="A1.SS2.p1.1.m1.1b"><apply id="A1.SS2.p1.1.m1.1.1.cmml" xref="A1.SS2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A1.SS2.p1.1.m1.1.1.1.cmml" xref="A1.SS2.p1.1.m1.1.1">subscript</csymbol><ci id="A1.SS2.p1.1.m1.1.1.2.cmml" xref="A1.SS2.p1.1.m1.1.1.2">𝐿</ci><ci id="A1.SS2.p1.1.m1.1.1.3a.cmml" xref="A1.SS2.p1.1.m1.1.1.3"><mtext mathsize="70%" id="A1.SS2.p1.1.m1.1.1.3.cmml" xref="A1.SS2.p1.1.m1.1.1.3">mem</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p1.1.m1.1c">L_{\text{mem}}</annotation></semantics></math> self-attention layers, <math id="A1.SS2.p1.2.m2.1" class="ltx_Math" alttext="L_{\text{mem}}-1" display="inline"><semantics id="A1.SS2.p1.2.m2.1a"><mrow id="A1.SS2.p1.2.m2.1.1" xref="A1.SS2.p1.2.m2.1.1.cmml"><msub id="A1.SS2.p1.2.m2.1.1.2" xref="A1.SS2.p1.2.m2.1.1.2.cmml"><mi id="A1.SS2.p1.2.m2.1.1.2.2" xref="A1.SS2.p1.2.m2.1.1.2.2.cmml">L</mi><mtext id="A1.SS2.p1.2.m2.1.1.2.3" xref="A1.SS2.p1.2.m2.1.1.2.3a.cmml">mem</mtext></msub><mo id="A1.SS2.p1.2.m2.1.1.1" xref="A1.SS2.p1.2.m2.1.1.1.cmml">−</mo><mn id="A1.SS2.p1.2.m2.1.1.3" xref="A1.SS2.p1.2.m2.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.p1.2.m2.1b"><apply id="A1.SS2.p1.2.m2.1.1.cmml" xref="A1.SS2.p1.2.m2.1.1"><minus id="A1.SS2.p1.2.m2.1.1.1.cmml" xref="A1.SS2.p1.2.m2.1.1.1"></minus><apply id="A1.SS2.p1.2.m2.1.1.2.cmml" xref="A1.SS2.p1.2.m2.1.1.2"><csymbol cd="ambiguous" id="A1.SS2.p1.2.m2.1.1.2.1.cmml" xref="A1.SS2.p1.2.m2.1.1.2">subscript</csymbol><ci id="A1.SS2.p1.2.m2.1.1.2.2.cmml" xref="A1.SS2.p1.2.m2.1.1.2.2">𝐿</ci><ci id="A1.SS2.p1.2.m2.1.1.2.3a.cmml" xref="A1.SS2.p1.2.m2.1.1.2.3"><mtext mathsize="70%" id="A1.SS2.p1.2.m2.1.1.2.3.cmml" xref="A1.SS2.p1.2.m2.1.1.2.3">mem</mtext></ci></apply><cn type="integer" id="A1.SS2.p1.2.m2.1.1.3.cmml" xref="A1.SS2.p1.2.m2.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p1.2.m2.1c">L_{\text{mem}}-1</annotation></semantics></math> MLP layers, and <math id="A1.SS2.p1.3.m3.1" class="ltx_Math" alttext="L_{\text{mem}}" display="inline"><semantics id="A1.SS2.p1.3.m3.1a"><msub id="A1.SS2.p1.3.m3.1.1" xref="A1.SS2.p1.3.m3.1.1.cmml"><mi id="A1.SS2.p1.3.m3.1.1.2" xref="A1.SS2.p1.3.m3.1.1.2.cmml">L</mi><mtext id="A1.SS2.p1.3.m3.1.1.3" xref="A1.SS2.p1.3.m3.1.1.3a.cmml">mem</mtext></msub><annotation-xml encoding="MathML-Content" id="A1.SS2.p1.3.m3.1b"><apply id="A1.SS2.p1.3.m3.1.1.cmml" xref="A1.SS2.p1.3.m3.1.1"><csymbol cd="ambiguous" id="A1.SS2.p1.3.m3.1.1.1.cmml" xref="A1.SS2.p1.3.m3.1.1">subscript</csymbol><ci id="A1.SS2.p1.3.m3.1.1.2.cmml" xref="A1.SS2.p1.3.m3.1.1.2">𝐿</ci><ci id="A1.SS2.p1.3.m3.1.1.3a.cmml" xref="A1.SS2.p1.3.m3.1.1.3"><mtext mathsize="70%" id="A1.SS2.p1.3.m3.1.1.3.cmml" xref="A1.SS2.p1.3.m3.1.1.3">mem</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p1.3.m3.1c">L_{\text{mem}}</annotation></semantics></math> token sparsification operations (computing the full attention matrix):</p>
<table id="A3.EGx2" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="Ex12"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="Ex12.m1.1" class="ltx_Math" alttext="\displaystyle\text{cost}_{\text{write}}" display="inline"><semantics id="Ex12.m1.1a"><msub id="Ex12.m1.1.1" xref="Ex12.m1.1.1.cmml"><mtext id="Ex12.m1.1.1.2" xref="Ex12.m1.1.1.2a.cmml">cost</mtext><mtext id="Ex12.m1.1.1.3" xref="Ex12.m1.1.1.3a.cmml">write</mtext></msub><annotation-xml encoding="MathML-Content" id="Ex12.m1.1b"><apply id="Ex12.m1.1.1.cmml" xref="Ex12.m1.1.1"><csymbol cd="ambiguous" id="Ex12.m1.1.1.1.cmml" xref="Ex12.m1.1.1">subscript</csymbol><ci id="Ex12.m1.1.1.2a.cmml" xref="Ex12.m1.1.1.2"><mtext id="Ex12.m1.1.1.2.cmml" xref="Ex12.m1.1.1.2">cost</mtext></ci><ci id="Ex12.m1.1.1.3a.cmml" xref="Ex12.m1.1.1.3"><mtext mathsize="70%" id="Ex12.m1.1.1.3.cmml" xref="Ex12.m1.1.1.3">write</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="Ex12.m1.1c">\displaystyle\text{cost}_{\text{write}}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="Ex12.m2.1" class="ltx_Math" alttext="\displaystyle=2\cdot\big{[}L_{\text{mem}}\big{(}l_{\text{ref}}(2d^{2}+2dd_{h}H_{kv})+2\frac{l_{\text{ref}}^{2}}{2}d\big{)}+(L_{\text{mem}}-1)(l_{\text{ref}}\cdot 3dW)+L_{\text{mem}}(l_{\text{ref}}^{2}d)\big{]}" display="inline"><semantics id="Ex12.m2.1a"><mrow id="Ex12.m2.1.1" xref="Ex12.m2.1.1.cmml"><mi id="Ex12.m2.1.1.3" xref="Ex12.m2.1.1.3.cmml"></mi><mo id="Ex12.m2.1.1.2" xref="Ex12.m2.1.1.2.cmml">=</mo><mrow id="Ex12.m2.1.1.1" xref="Ex12.m2.1.1.1.cmml"><mn id="Ex12.m2.1.1.1.3" xref="Ex12.m2.1.1.1.3.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="Ex12.m2.1.1.1.2" xref="Ex12.m2.1.1.1.2.cmml">⋅</mo><mrow id="Ex12.m2.1.1.1.1.1" xref="Ex12.m2.1.1.1.1.2.cmml"><mo maxsize="120%" minsize="120%" id="Ex12.m2.1.1.1.1.1.2" xref="Ex12.m2.1.1.1.1.2.1.cmml">[</mo><mrow id="Ex12.m2.1.1.1.1.1.1" xref="Ex12.m2.1.1.1.1.1.1.cmml"><mrow id="Ex12.m2.1.1.1.1.1.1.1" xref="Ex12.m2.1.1.1.1.1.1.1.cmml"><msub id="Ex12.m2.1.1.1.1.1.1.1.3" xref="Ex12.m2.1.1.1.1.1.1.1.3.cmml"><mi id="Ex12.m2.1.1.1.1.1.1.1.3.2" xref="Ex12.m2.1.1.1.1.1.1.1.3.2.cmml">L</mi><mtext id="Ex12.m2.1.1.1.1.1.1.1.3.3" xref="Ex12.m2.1.1.1.1.1.1.1.3.3a.cmml">mem</mtext></msub><mo lspace="0em" rspace="0em" id="Ex12.m2.1.1.1.1.1.1.1.2" xref="Ex12.m2.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="Ex12.m2.1.1.1.1.1.1.1.1.1" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.cmml"><mo maxsize="120%" minsize="120%" id="Ex12.m2.1.1.1.1.1.1.1.1.1.2" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="Ex12.m2.1.1.1.1.1.1.1.1.1.1" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.cmml"><mrow id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.cmml"><msub id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.3" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.3.2" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml">l</mi><mtext id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.3.3" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.3.3a.cmml">ref</mtext></msub><mo lspace="0em" rspace="0em" id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.2" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mrow id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml"><mn id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.2" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.1" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml">​</mo><msup id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml"><mi id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.2" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.2.cmml">d</mi><mn id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.3" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.3.cmml">2</mn></msup></mrow><mo id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">+</mo><mrow id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml"><mn id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.2" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.cmml">d</mi><mo lspace="0em" rspace="0em" id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1a" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><msub id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.cmml"><mi id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.2" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.2.cmml">d</mi><mi id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.3" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.3.cmml">h</mi></msub><mo lspace="0em" rspace="0em" id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1b" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><msub id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.cmml"><mi id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.2" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.2.cmml">H</mi><mrow id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.cmml"><mi id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.2" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.2.cmml">k</mi><mo lspace="0em" rspace="0em" id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.1" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.1.cmml">​</mo><mi id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.3" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.3.cmml">v</mi></mrow></msub></mrow></mrow><mo stretchy="false" id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.2" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.2.cmml">+</mo><mrow id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.cmml"><mn id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.2" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.1" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><mstyle displaystyle="true" id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.cmml"><mfrac id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3a" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.cmml"><msubsup id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.2" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.2.cmml"><mi id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.2.2.2" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.2.2.2.cmml">l</mi><mtext id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.2.2.3" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.2.2.3a.cmml">ref</mtext><mn id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.2.3" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.2.3.cmml">2</mn></msubsup><mn id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.3" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.3.cmml">2</mn></mfrac></mstyle><mo lspace="0em" rspace="0em" id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.1a" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.4" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.4.cmml">d</mi></mrow></mrow><mo maxsize="120%" minsize="120%" id="Ex12.m2.1.1.1.1.1.1.1.1.1.3" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="Ex12.m2.1.1.1.1.1.1.5" xref="Ex12.m2.1.1.1.1.1.1.5.cmml">+</mo><mrow id="Ex12.m2.1.1.1.1.1.1.3" xref="Ex12.m2.1.1.1.1.1.1.3.cmml"><mrow id="Ex12.m2.1.1.1.1.1.1.2.1.1" xref="Ex12.m2.1.1.1.1.1.1.2.1.1.1.cmml"><mo stretchy="false" id="Ex12.m2.1.1.1.1.1.1.2.1.1.2" xref="Ex12.m2.1.1.1.1.1.1.2.1.1.1.cmml">(</mo><mrow id="Ex12.m2.1.1.1.1.1.1.2.1.1.1" xref="Ex12.m2.1.1.1.1.1.1.2.1.1.1.cmml"><msub id="Ex12.m2.1.1.1.1.1.1.2.1.1.1.2" xref="Ex12.m2.1.1.1.1.1.1.2.1.1.1.2.cmml"><mi id="Ex12.m2.1.1.1.1.1.1.2.1.1.1.2.2" xref="Ex12.m2.1.1.1.1.1.1.2.1.1.1.2.2.cmml">L</mi><mtext id="Ex12.m2.1.1.1.1.1.1.2.1.1.1.2.3" xref="Ex12.m2.1.1.1.1.1.1.2.1.1.1.2.3a.cmml">mem</mtext></msub><mo id="Ex12.m2.1.1.1.1.1.1.2.1.1.1.1" xref="Ex12.m2.1.1.1.1.1.1.2.1.1.1.1.cmml">−</mo><mn id="Ex12.m2.1.1.1.1.1.1.2.1.1.1.3" xref="Ex12.m2.1.1.1.1.1.1.2.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="Ex12.m2.1.1.1.1.1.1.2.1.1.3" xref="Ex12.m2.1.1.1.1.1.1.2.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="Ex12.m2.1.1.1.1.1.1.3.3" xref="Ex12.m2.1.1.1.1.1.1.3.3.cmml">​</mo><mrow id="Ex12.m2.1.1.1.1.1.1.3.2.1" xref="Ex12.m2.1.1.1.1.1.1.3.2.1.1.cmml"><mo stretchy="false" id="Ex12.m2.1.1.1.1.1.1.3.2.1.2" xref="Ex12.m2.1.1.1.1.1.1.3.2.1.1.cmml">(</mo><mrow id="Ex12.m2.1.1.1.1.1.1.3.2.1.1" xref="Ex12.m2.1.1.1.1.1.1.3.2.1.1.cmml"><mrow id="Ex12.m2.1.1.1.1.1.1.3.2.1.1.2" xref="Ex12.m2.1.1.1.1.1.1.3.2.1.1.2.cmml"><msub id="Ex12.m2.1.1.1.1.1.1.3.2.1.1.2.2" xref="Ex12.m2.1.1.1.1.1.1.3.2.1.1.2.2.cmml"><mi id="Ex12.m2.1.1.1.1.1.1.3.2.1.1.2.2.2" xref="Ex12.m2.1.1.1.1.1.1.3.2.1.1.2.2.2.cmml">l</mi><mtext id="Ex12.m2.1.1.1.1.1.1.3.2.1.1.2.2.3" xref="Ex12.m2.1.1.1.1.1.1.3.2.1.1.2.2.3a.cmml">ref</mtext></msub><mo lspace="0.222em" rspace="0.222em" id="Ex12.m2.1.1.1.1.1.1.3.2.1.1.2.1" xref="Ex12.m2.1.1.1.1.1.1.3.2.1.1.2.1.cmml">⋅</mo><mn id="Ex12.m2.1.1.1.1.1.1.3.2.1.1.2.3" xref="Ex12.m2.1.1.1.1.1.1.3.2.1.1.2.3.cmml">3</mn></mrow><mo lspace="0em" rspace="0em" id="Ex12.m2.1.1.1.1.1.1.3.2.1.1.1" xref="Ex12.m2.1.1.1.1.1.1.3.2.1.1.1.cmml">​</mo><mi id="Ex12.m2.1.1.1.1.1.1.3.2.1.1.3" xref="Ex12.m2.1.1.1.1.1.1.3.2.1.1.3.cmml">d</mi><mo lspace="0em" rspace="0em" id="Ex12.m2.1.1.1.1.1.1.3.2.1.1.1a" xref="Ex12.m2.1.1.1.1.1.1.3.2.1.1.1.cmml">​</mo><mi id="Ex12.m2.1.1.1.1.1.1.3.2.1.1.4" xref="Ex12.m2.1.1.1.1.1.1.3.2.1.1.4.cmml">W</mi></mrow><mo stretchy="false" id="Ex12.m2.1.1.1.1.1.1.3.2.1.3" xref="Ex12.m2.1.1.1.1.1.1.3.2.1.1.cmml">)</mo></mrow></mrow><mo id="Ex12.m2.1.1.1.1.1.1.5a" xref="Ex12.m2.1.1.1.1.1.1.5.cmml">+</mo><mrow id="Ex12.m2.1.1.1.1.1.1.4" xref="Ex12.m2.1.1.1.1.1.1.4.cmml"><msub id="Ex12.m2.1.1.1.1.1.1.4.3" xref="Ex12.m2.1.1.1.1.1.1.4.3.cmml"><mi id="Ex12.m2.1.1.1.1.1.1.4.3.2" xref="Ex12.m2.1.1.1.1.1.1.4.3.2.cmml">L</mi><mtext id="Ex12.m2.1.1.1.1.1.1.4.3.3" xref="Ex12.m2.1.1.1.1.1.1.4.3.3a.cmml">mem</mtext></msub><mo lspace="0em" rspace="0em" id="Ex12.m2.1.1.1.1.1.1.4.2" xref="Ex12.m2.1.1.1.1.1.1.4.2.cmml">​</mo><mrow id="Ex12.m2.1.1.1.1.1.1.4.1.1" xref="Ex12.m2.1.1.1.1.1.1.4.1.1.1.cmml"><mo stretchy="false" id="Ex12.m2.1.1.1.1.1.1.4.1.1.2" xref="Ex12.m2.1.1.1.1.1.1.4.1.1.1.cmml">(</mo><mrow id="Ex12.m2.1.1.1.1.1.1.4.1.1.1" xref="Ex12.m2.1.1.1.1.1.1.4.1.1.1.cmml"><msubsup id="Ex12.m2.1.1.1.1.1.1.4.1.1.1.2" xref="Ex12.m2.1.1.1.1.1.1.4.1.1.1.2.cmml"><mi id="Ex12.m2.1.1.1.1.1.1.4.1.1.1.2.2.2" xref="Ex12.m2.1.1.1.1.1.1.4.1.1.1.2.2.2.cmml">l</mi><mtext id="Ex12.m2.1.1.1.1.1.1.4.1.1.1.2.2.3" xref="Ex12.m2.1.1.1.1.1.1.4.1.1.1.2.2.3a.cmml">ref</mtext><mn id="Ex12.m2.1.1.1.1.1.1.4.1.1.1.2.3" xref="Ex12.m2.1.1.1.1.1.1.4.1.1.1.2.3.cmml">2</mn></msubsup><mo lspace="0em" rspace="0em" id="Ex12.m2.1.1.1.1.1.1.4.1.1.1.1" xref="Ex12.m2.1.1.1.1.1.1.4.1.1.1.1.cmml">​</mo><mi id="Ex12.m2.1.1.1.1.1.1.4.1.1.1.3" xref="Ex12.m2.1.1.1.1.1.1.4.1.1.1.3.cmml">d</mi></mrow><mo stretchy="false" id="Ex12.m2.1.1.1.1.1.1.4.1.1.3" xref="Ex12.m2.1.1.1.1.1.1.4.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo maxsize="120%" minsize="120%" id="Ex12.m2.1.1.1.1.1.3" xref="Ex12.m2.1.1.1.1.2.1.cmml">]</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="Ex12.m2.1b"><apply id="Ex12.m2.1.1.cmml" xref="Ex12.m2.1.1"><eq id="Ex12.m2.1.1.2.cmml" xref="Ex12.m2.1.1.2"></eq><csymbol cd="latexml" id="Ex12.m2.1.1.3.cmml" xref="Ex12.m2.1.1.3">absent</csymbol><apply id="Ex12.m2.1.1.1.cmml" xref="Ex12.m2.1.1.1"><ci id="Ex12.m2.1.1.1.2.cmml" xref="Ex12.m2.1.1.1.2">⋅</ci><cn type="integer" id="Ex12.m2.1.1.1.3.cmml" xref="Ex12.m2.1.1.1.3">2</cn><apply id="Ex12.m2.1.1.1.1.2.cmml" xref="Ex12.m2.1.1.1.1.1"><csymbol cd="latexml" id="Ex12.m2.1.1.1.1.2.1.cmml" xref="Ex12.m2.1.1.1.1.1.2">delimited-[]</csymbol><apply id="Ex12.m2.1.1.1.1.1.1.cmml" xref="Ex12.m2.1.1.1.1.1.1"><plus id="Ex12.m2.1.1.1.1.1.1.5.cmml" xref="Ex12.m2.1.1.1.1.1.1.5"></plus><apply id="Ex12.m2.1.1.1.1.1.1.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.1"><times id="Ex12.m2.1.1.1.1.1.1.1.2.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.2"></times><apply id="Ex12.m2.1.1.1.1.1.1.1.3.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="Ex12.m2.1.1.1.1.1.1.1.3.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="Ex12.m2.1.1.1.1.1.1.1.3.2.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.3.2">𝐿</ci><ci id="Ex12.m2.1.1.1.1.1.1.1.3.3a.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.3.3"><mtext mathsize="70%" id="Ex12.m2.1.1.1.1.1.1.1.3.3.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.3.3">mem</mtext></ci></apply><apply id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1"><plus id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.2"></plus><apply id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1"><times id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.2"></times><apply id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.3.2">𝑙</ci><ci id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.3.3a.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.3.3"><mtext mathsize="70%" id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.3.3">ref</mtext></ci></apply><apply id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1"><plus id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1"></plus><apply id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2"><times id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.1"></times><cn type="integer" id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.2">2</cn><apply id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3">superscript</csymbol><ci id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.2.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.2">𝑑</ci><cn type="integer" id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.3.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.3">2</cn></apply></apply><apply id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3"><times id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1"></times><cn type="integer" id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.2">2</cn><ci id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3">𝑑</ci><apply id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4"><csymbol cd="ambiguous" id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4">subscript</csymbol><ci id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.2.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.2">𝑑</ci><ci id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.3.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.3">ℎ</ci></apply><apply id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5"><csymbol cd="ambiguous" id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5">subscript</csymbol><ci id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.2.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.2">𝐻</ci><apply id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3"><times id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.1"></times><ci id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.2.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.2">𝑘</ci><ci id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.3.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.3">𝑣</ci></apply></apply></apply></apply></apply><apply id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3"><times id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.1"></times><cn type="integer" id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.2">2</cn><apply id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3"><divide id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3"></divide><apply id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.2.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.2"><csymbol cd="ambiguous" id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.2.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.2">superscript</csymbol><apply id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.2.2.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.2"><csymbol cd="ambiguous" id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.2.2.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.2">subscript</csymbol><ci id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.2.2.2.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.2.2.2">𝑙</ci><ci id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.2.2.3a.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.2.2.3"><mtext mathsize="70%" id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.2.2.3.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.2.2.3">ref</mtext></ci></apply><cn type="integer" id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.2.3.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.2.3">2</cn></apply><cn type="integer" id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.3.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.3.3">2</cn></apply><ci id="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.4.cmml" xref="Ex12.m2.1.1.1.1.1.1.1.1.1.1.3.4">𝑑</ci></apply></apply></apply><apply id="Ex12.m2.1.1.1.1.1.1.3.cmml" xref="Ex12.m2.1.1.1.1.1.1.3"><times id="Ex12.m2.1.1.1.1.1.1.3.3.cmml" xref="Ex12.m2.1.1.1.1.1.1.3.3"></times><apply id="Ex12.m2.1.1.1.1.1.1.2.1.1.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.2.1.1"><minus id="Ex12.m2.1.1.1.1.1.1.2.1.1.1.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.2.1.1.1.1"></minus><apply id="Ex12.m2.1.1.1.1.1.1.2.1.1.1.2.cmml" xref="Ex12.m2.1.1.1.1.1.1.2.1.1.1.2"><csymbol cd="ambiguous" id="Ex12.m2.1.1.1.1.1.1.2.1.1.1.2.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.2.1.1.1.2">subscript</csymbol><ci id="Ex12.m2.1.1.1.1.1.1.2.1.1.1.2.2.cmml" xref="Ex12.m2.1.1.1.1.1.1.2.1.1.1.2.2">𝐿</ci><ci id="Ex12.m2.1.1.1.1.1.1.2.1.1.1.2.3a.cmml" xref="Ex12.m2.1.1.1.1.1.1.2.1.1.1.2.3"><mtext mathsize="70%" id="Ex12.m2.1.1.1.1.1.1.2.1.1.1.2.3.cmml" xref="Ex12.m2.1.1.1.1.1.1.2.1.1.1.2.3">mem</mtext></ci></apply><cn type="integer" id="Ex12.m2.1.1.1.1.1.1.2.1.1.1.3.cmml" xref="Ex12.m2.1.1.1.1.1.1.2.1.1.1.3">1</cn></apply><apply id="Ex12.m2.1.1.1.1.1.1.3.2.1.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.3.2.1"><times id="Ex12.m2.1.1.1.1.1.1.3.2.1.1.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.3.2.1.1.1"></times><apply id="Ex12.m2.1.1.1.1.1.1.3.2.1.1.2.cmml" xref="Ex12.m2.1.1.1.1.1.1.3.2.1.1.2"><ci id="Ex12.m2.1.1.1.1.1.1.3.2.1.1.2.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.3.2.1.1.2.1">⋅</ci><apply id="Ex12.m2.1.1.1.1.1.1.3.2.1.1.2.2.cmml" xref="Ex12.m2.1.1.1.1.1.1.3.2.1.1.2.2"><csymbol cd="ambiguous" id="Ex12.m2.1.1.1.1.1.1.3.2.1.1.2.2.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.3.2.1.1.2.2">subscript</csymbol><ci id="Ex12.m2.1.1.1.1.1.1.3.2.1.1.2.2.2.cmml" xref="Ex12.m2.1.1.1.1.1.1.3.2.1.1.2.2.2">𝑙</ci><ci id="Ex12.m2.1.1.1.1.1.1.3.2.1.1.2.2.3a.cmml" xref="Ex12.m2.1.1.1.1.1.1.3.2.1.1.2.2.3"><mtext mathsize="70%" id="Ex12.m2.1.1.1.1.1.1.3.2.1.1.2.2.3.cmml" xref="Ex12.m2.1.1.1.1.1.1.3.2.1.1.2.2.3">ref</mtext></ci></apply><cn type="integer" id="Ex12.m2.1.1.1.1.1.1.3.2.1.1.2.3.cmml" xref="Ex12.m2.1.1.1.1.1.1.3.2.1.1.2.3">3</cn></apply><ci id="Ex12.m2.1.1.1.1.1.1.3.2.1.1.3.cmml" xref="Ex12.m2.1.1.1.1.1.1.3.2.1.1.3">𝑑</ci><ci id="Ex12.m2.1.1.1.1.1.1.3.2.1.1.4.cmml" xref="Ex12.m2.1.1.1.1.1.1.3.2.1.1.4">𝑊</ci></apply></apply><apply id="Ex12.m2.1.1.1.1.1.1.4.cmml" xref="Ex12.m2.1.1.1.1.1.1.4"><times id="Ex12.m2.1.1.1.1.1.1.4.2.cmml" xref="Ex12.m2.1.1.1.1.1.1.4.2"></times><apply id="Ex12.m2.1.1.1.1.1.1.4.3.cmml" xref="Ex12.m2.1.1.1.1.1.1.4.3"><csymbol cd="ambiguous" id="Ex12.m2.1.1.1.1.1.1.4.3.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.4.3">subscript</csymbol><ci id="Ex12.m2.1.1.1.1.1.1.4.3.2.cmml" xref="Ex12.m2.1.1.1.1.1.1.4.3.2">𝐿</ci><ci id="Ex12.m2.1.1.1.1.1.1.4.3.3a.cmml" xref="Ex12.m2.1.1.1.1.1.1.4.3.3"><mtext mathsize="70%" id="Ex12.m2.1.1.1.1.1.1.4.3.3.cmml" xref="Ex12.m2.1.1.1.1.1.1.4.3.3">mem</mtext></ci></apply><apply id="Ex12.m2.1.1.1.1.1.1.4.1.1.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.4.1.1"><times id="Ex12.m2.1.1.1.1.1.1.4.1.1.1.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.4.1.1.1.1"></times><apply id="Ex12.m2.1.1.1.1.1.1.4.1.1.1.2.cmml" xref="Ex12.m2.1.1.1.1.1.1.4.1.1.1.2"><csymbol cd="ambiguous" id="Ex12.m2.1.1.1.1.1.1.4.1.1.1.2.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.4.1.1.1.2">superscript</csymbol><apply id="Ex12.m2.1.1.1.1.1.1.4.1.1.1.2.2.cmml" xref="Ex12.m2.1.1.1.1.1.1.4.1.1.1.2"><csymbol cd="ambiguous" id="Ex12.m2.1.1.1.1.1.1.4.1.1.1.2.2.1.cmml" xref="Ex12.m2.1.1.1.1.1.1.4.1.1.1.2">subscript</csymbol><ci id="Ex12.m2.1.1.1.1.1.1.4.1.1.1.2.2.2.cmml" xref="Ex12.m2.1.1.1.1.1.1.4.1.1.1.2.2.2">𝑙</ci><ci id="Ex12.m2.1.1.1.1.1.1.4.1.1.1.2.2.3a.cmml" xref="Ex12.m2.1.1.1.1.1.1.4.1.1.1.2.2.3"><mtext mathsize="70%" id="Ex12.m2.1.1.1.1.1.1.4.1.1.1.2.2.3.cmml" xref="Ex12.m2.1.1.1.1.1.1.4.1.1.1.2.2.3">ref</mtext></ci></apply><cn type="integer" id="Ex12.m2.1.1.1.1.1.1.4.1.1.1.2.3.cmml" xref="Ex12.m2.1.1.1.1.1.1.4.1.1.1.2.3">2</cn></apply><ci id="Ex12.m2.1.1.1.1.1.1.4.1.1.1.3.cmml" xref="Ex12.m2.1.1.1.1.1.1.4.1.1.1.3">𝑑</ci></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Ex12.m2.1c">\displaystyle=2\cdot\big{[}L_{\text{mem}}\big{(}l_{\text{ref}}(2d^{2}+2dd_{h}H_{kv})+2\frac{l_{\text{ref}}^{2}}{2}d\big{)}+(L_{\text{mem}}-1)(l_{\text{ref}}\cdot 3dW)+L_{\text{mem}}(l_{\text{ref}}^{2}d)\big{]}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="Ex13"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="Ex13.m1.1" class="ltx_Math" alttext="\displaystyle\approx 0.308~{}\text{TFlops}" display="inline"><semantics id="Ex13.m1.1a"><mrow id="Ex13.m1.1.1" xref="Ex13.m1.1.1.cmml"><mi id="Ex13.m1.1.1.2" xref="Ex13.m1.1.1.2.cmml"></mi><mo id="Ex13.m1.1.1.1" xref="Ex13.m1.1.1.1.cmml">≈</mo><mrow id="Ex13.m1.1.1.3" xref="Ex13.m1.1.1.3.cmml"><mn id="Ex13.m1.1.1.3.2" xref="Ex13.m1.1.1.3.2.cmml">0.308</mn><mo lspace="0.330em" rspace="0em" id="Ex13.m1.1.1.3.1" xref="Ex13.m1.1.1.3.1.cmml">​</mo><mtext id="Ex13.m1.1.1.3.3" xref="Ex13.m1.1.1.3.3a.cmml">TFlops</mtext></mrow></mrow><annotation-xml encoding="MathML-Content" id="Ex13.m1.1b"><apply id="Ex13.m1.1.1.cmml" xref="Ex13.m1.1.1"><approx id="Ex13.m1.1.1.1.cmml" xref="Ex13.m1.1.1.1"></approx><csymbol cd="latexml" id="Ex13.m1.1.1.2.cmml" xref="Ex13.m1.1.1.2">absent</csymbol><apply id="Ex13.m1.1.1.3.cmml" xref="Ex13.m1.1.1.3"><times id="Ex13.m1.1.1.3.1.cmml" xref="Ex13.m1.1.1.3.1"></times><cn type="float" id="Ex13.m1.1.1.3.2.cmml" xref="Ex13.m1.1.1.3.2">0.308</cn><ci id="Ex13.m1.1.1.3.3a.cmml" xref="Ex13.m1.1.1.3.3"><mtext id="Ex13.m1.1.1.3.3.cmml" xref="Ex13.m1.1.1.3.3">TFlops</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Ex13.m1.1c">\displaystyle\approx 0.308~{}\text{TFlops}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="A1.SS2.p2" class="ltx_para">
<p id="A1.SS2.p2.1" class="ltx_p">The read cost consists of the attention to the sparse tokens of an explicit memory from the chunk that retrieves this memory:</p>
<table id="Ex14" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="Ex14.m1.1" class="ltx_Math" alttext="\text{cost}_{\text{read}}=2L_{\text{mem}}\cdot 2l_{\text{chunk}}l_{\text{mem}}d\approx 1.44\times 10^{-4}~{}\text{TFlops}" display="block"><semantics id="Ex14.m1.1a"><mrow id="Ex14.m1.1.1" xref="Ex14.m1.1.1.cmml"><msub id="Ex14.m1.1.1.2" xref="Ex14.m1.1.1.2.cmml"><mtext id="Ex14.m1.1.1.2.2" xref="Ex14.m1.1.1.2.2a.cmml">cost</mtext><mtext id="Ex14.m1.1.1.2.3" xref="Ex14.m1.1.1.2.3a.cmml">read</mtext></msub><mo id="Ex14.m1.1.1.3" xref="Ex14.m1.1.1.3.cmml">=</mo><mrow id="Ex14.m1.1.1.4" xref="Ex14.m1.1.1.4.cmml"><mrow id="Ex14.m1.1.1.4.2" xref="Ex14.m1.1.1.4.2.cmml"><mrow id="Ex14.m1.1.1.4.2.2" xref="Ex14.m1.1.1.4.2.2.cmml"><mn id="Ex14.m1.1.1.4.2.2.2" xref="Ex14.m1.1.1.4.2.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="Ex14.m1.1.1.4.2.2.1" xref="Ex14.m1.1.1.4.2.2.1.cmml">​</mo><msub id="Ex14.m1.1.1.4.2.2.3" xref="Ex14.m1.1.1.4.2.2.3.cmml"><mi id="Ex14.m1.1.1.4.2.2.3.2" xref="Ex14.m1.1.1.4.2.2.3.2.cmml">L</mi><mtext id="Ex14.m1.1.1.4.2.2.3.3" xref="Ex14.m1.1.1.4.2.2.3.3a.cmml">mem</mtext></msub></mrow><mo lspace="0.222em" rspace="0.222em" id="Ex14.m1.1.1.4.2.1" xref="Ex14.m1.1.1.4.2.1.cmml">⋅</mo><mn id="Ex14.m1.1.1.4.2.3" xref="Ex14.m1.1.1.4.2.3.cmml">2</mn></mrow><mo lspace="0em" rspace="0em" id="Ex14.m1.1.1.4.1" xref="Ex14.m1.1.1.4.1.cmml">​</mo><msub id="Ex14.m1.1.1.4.3" xref="Ex14.m1.1.1.4.3.cmml"><mi id="Ex14.m1.1.1.4.3.2" xref="Ex14.m1.1.1.4.3.2.cmml">l</mi><mtext id="Ex14.m1.1.1.4.3.3" xref="Ex14.m1.1.1.4.3.3a.cmml">chunk</mtext></msub><mo lspace="0em" rspace="0em" id="Ex14.m1.1.1.4.1a" xref="Ex14.m1.1.1.4.1.cmml">​</mo><msub id="Ex14.m1.1.1.4.4" xref="Ex14.m1.1.1.4.4.cmml"><mi id="Ex14.m1.1.1.4.4.2" xref="Ex14.m1.1.1.4.4.2.cmml">l</mi><mtext id="Ex14.m1.1.1.4.4.3" xref="Ex14.m1.1.1.4.4.3a.cmml">mem</mtext></msub><mo lspace="0em" rspace="0em" id="Ex14.m1.1.1.4.1b" xref="Ex14.m1.1.1.4.1.cmml">​</mo><mi id="Ex14.m1.1.1.4.5" xref="Ex14.m1.1.1.4.5.cmml">d</mi></mrow><mo id="Ex14.m1.1.1.5" xref="Ex14.m1.1.1.5.cmml">≈</mo><mrow id="Ex14.m1.1.1.6" xref="Ex14.m1.1.1.6.cmml"><mrow id="Ex14.m1.1.1.6.2" xref="Ex14.m1.1.1.6.2.cmml"><mn id="Ex14.m1.1.1.6.2.2" xref="Ex14.m1.1.1.6.2.2.cmml">1.44</mn><mo lspace="0.222em" rspace="0.222em" id="Ex14.m1.1.1.6.2.1" xref="Ex14.m1.1.1.6.2.1.cmml">×</mo><msup id="Ex14.m1.1.1.6.2.3" xref="Ex14.m1.1.1.6.2.3.cmml"><mn id="Ex14.m1.1.1.6.2.3.2" xref="Ex14.m1.1.1.6.2.3.2.cmml">10</mn><mrow id="Ex14.m1.1.1.6.2.3.3" xref="Ex14.m1.1.1.6.2.3.3.cmml"><mo id="Ex14.m1.1.1.6.2.3.3a" xref="Ex14.m1.1.1.6.2.3.3.cmml">−</mo><mn id="Ex14.m1.1.1.6.2.3.3.2" xref="Ex14.m1.1.1.6.2.3.3.2.cmml">4</mn></mrow></msup></mrow><mo lspace="0em" rspace="0em" id="Ex14.m1.1.1.6.1" xref="Ex14.m1.1.1.6.1.cmml">​</mo><mtext id="Ex14.m1.1.1.6.3" xref="Ex14.m1.1.1.6.3a.cmml">TFlops</mtext></mrow></mrow><annotation-xml encoding="MathML-Content" id="Ex14.m1.1b"><apply id="Ex14.m1.1.1.cmml" xref="Ex14.m1.1.1"><and id="Ex14.m1.1.1a.cmml" xref="Ex14.m1.1.1"></and><apply id="Ex14.m1.1.1b.cmml" xref="Ex14.m1.1.1"><eq id="Ex14.m1.1.1.3.cmml" xref="Ex14.m1.1.1.3"></eq><apply id="Ex14.m1.1.1.2.cmml" xref="Ex14.m1.1.1.2"><csymbol cd="ambiguous" id="Ex14.m1.1.1.2.1.cmml" xref="Ex14.m1.1.1.2">subscript</csymbol><ci id="Ex14.m1.1.1.2.2a.cmml" xref="Ex14.m1.1.1.2.2"><mtext id="Ex14.m1.1.1.2.2.cmml" xref="Ex14.m1.1.1.2.2">cost</mtext></ci><ci id="Ex14.m1.1.1.2.3a.cmml" xref="Ex14.m1.1.1.2.3"><mtext mathsize="70%" id="Ex14.m1.1.1.2.3.cmml" xref="Ex14.m1.1.1.2.3">read</mtext></ci></apply><apply id="Ex14.m1.1.1.4.cmml" xref="Ex14.m1.1.1.4"><times id="Ex14.m1.1.1.4.1.cmml" xref="Ex14.m1.1.1.4.1"></times><apply id="Ex14.m1.1.1.4.2.cmml" xref="Ex14.m1.1.1.4.2"><ci id="Ex14.m1.1.1.4.2.1.cmml" xref="Ex14.m1.1.1.4.2.1">⋅</ci><apply id="Ex14.m1.1.1.4.2.2.cmml" xref="Ex14.m1.1.1.4.2.2"><times id="Ex14.m1.1.1.4.2.2.1.cmml" xref="Ex14.m1.1.1.4.2.2.1"></times><cn type="integer" id="Ex14.m1.1.1.4.2.2.2.cmml" xref="Ex14.m1.1.1.4.2.2.2">2</cn><apply id="Ex14.m1.1.1.4.2.2.3.cmml" xref="Ex14.m1.1.1.4.2.2.3"><csymbol cd="ambiguous" id="Ex14.m1.1.1.4.2.2.3.1.cmml" xref="Ex14.m1.1.1.4.2.2.3">subscript</csymbol><ci id="Ex14.m1.1.1.4.2.2.3.2.cmml" xref="Ex14.m1.1.1.4.2.2.3.2">𝐿</ci><ci id="Ex14.m1.1.1.4.2.2.3.3a.cmml" xref="Ex14.m1.1.1.4.2.2.3.3"><mtext mathsize="70%" id="Ex14.m1.1.1.4.2.2.3.3.cmml" xref="Ex14.m1.1.1.4.2.2.3.3">mem</mtext></ci></apply></apply><cn type="integer" id="Ex14.m1.1.1.4.2.3.cmml" xref="Ex14.m1.1.1.4.2.3">2</cn></apply><apply id="Ex14.m1.1.1.4.3.cmml" xref="Ex14.m1.1.1.4.3"><csymbol cd="ambiguous" id="Ex14.m1.1.1.4.3.1.cmml" xref="Ex14.m1.1.1.4.3">subscript</csymbol><ci id="Ex14.m1.1.1.4.3.2.cmml" xref="Ex14.m1.1.1.4.3.2">𝑙</ci><ci id="Ex14.m1.1.1.4.3.3a.cmml" xref="Ex14.m1.1.1.4.3.3"><mtext mathsize="70%" id="Ex14.m1.1.1.4.3.3.cmml" xref="Ex14.m1.1.1.4.3.3">chunk</mtext></ci></apply><apply id="Ex14.m1.1.1.4.4.cmml" xref="Ex14.m1.1.1.4.4"><csymbol cd="ambiguous" id="Ex14.m1.1.1.4.4.1.cmml" xref="Ex14.m1.1.1.4.4">subscript</csymbol><ci id="Ex14.m1.1.1.4.4.2.cmml" xref="Ex14.m1.1.1.4.4.2">𝑙</ci><ci id="Ex14.m1.1.1.4.4.3a.cmml" xref="Ex14.m1.1.1.4.4.3"><mtext mathsize="70%" id="Ex14.m1.1.1.4.4.3.cmml" xref="Ex14.m1.1.1.4.4.3">mem</mtext></ci></apply><ci id="Ex14.m1.1.1.4.5.cmml" xref="Ex14.m1.1.1.4.5">𝑑</ci></apply></apply><apply id="Ex14.m1.1.1c.cmml" xref="Ex14.m1.1.1"><approx id="Ex14.m1.1.1.5.cmml" xref="Ex14.m1.1.1.5"></approx><share href="#Ex14.m1.1.1.4.cmml" id="Ex14.m1.1.1d.cmml" xref="Ex14.m1.1.1"></share><apply id="Ex14.m1.1.1.6.cmml" xref="Ex14.m1.1.1.6"><times id="Ex14.m1.1.1.6.1.cmml" xref="Ex14.m1.1.1.6.1"></times><apply id="Ex14.m1.1.1.6.2.cmml" xref="Ex14.m1.1.1.6.2"><times id="Ex14.m1.1.1.6.2.1.cmml" xref="Ex14.m1.1.1.6.2.1"></times><cn type="float" id="Ex14.m1.1.1.6.2.2.cmml" xref="Ex14.m1.1.1.6.2.2">1.44</cn><apply id="Ex14.m1.1.1.6.2.3.cmml" xref="Ex14.m1.1.1.6.2.3"><csymbol cd="ambiguous" id="Ex14.m1.1.1.6.2.3.1.cmml" xref="Ex14.m1.1.1.6.2.3">superscript</csymbol><cn type="integer" id="Ex14.m1.1.1.6.2.3.2.cmml" xref="Ex14.m1.1.1.6.2.3.2">10</cn><apply id="Ex14.m1.1.1.6.2.3.3.cmml" xref="Ex14.m1.1.1.6.2.3.3"><minus id="Ex14.m1.1.1.6.2.3.3.1.cmml" xref="Ex14.m1.1.1.6.2.3.3"></minus><cn type="integer" id="Ex14.m1.1.1.6.2.3.3.2.cmml" xref="Ex14.m1.1.1.6.2.3.3.2">4</cn></apply></apply></apply><ci id="Ex14.m1.1.1.6.3a.cmml" xref="Ex14.m1.1.1.6.3"><mtext id="Ex14.m1.1.1.6.3.cmml" xref="Ex14.m1.1.1.6.3">TFlops</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Ex14.m1.1c">\text{cost}_{\text{read}}=2L_{\text{mem}}\cdot 2l_{\text{chunk}}l_{\text{mem}}d\approx 1.44\times 10^{-4}~{}\text{TFlops}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
</section>
<section id="A1.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_subsection" style="font-size:120%;color:#00868B;">A.3 <span id="A1.SS3.1.1" class="ltx_text" style="color:#808080;">| </span>External Information</h3>

<div id="A1.SS3.p1" class="ltx_para">
<p id="A1.SS3.p1.1" class="ltx_p">The write cost of text retrieval-augmented generation (RAG) is set to be zero, since the reference is stored as plain text.</p>
<table id="Ex15" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="Ex15.m1.1" class="ltx_Math" alttext="\text{cost}_{\text{write}}=0~{}\text{TFlops}" display="block"><semantics id="Ex15.m1.1a"><mrow id="Ex15.m1.1.1" xref="Ex15.m1.1.1.cmml"><msub id="Ex15.m1.1.1.2" xref="Ex15.m1.1.1.2.cmml"><mtext id="Ex15.m1.1.1.2.2" xref="Ex15.m1.1.1.2.2a.cmml">cost</mtext><mtext id="Ex15.m1.1.1.2.3" xref="Ex15.m1.1.1.2.3a.cmml">write</mtext></msub><mo id="Ex15.m1.1.1.1" xref="Ex15.m1.1.1.1.cmml">=</mo><mrow id="Ex15.m1.1.1.3" xref="Ex15.m1.1.1.3.cmml"><mn id="Ex15.m1.1.1.3.2" xref="Ex15.m1.1.1.3.2.cmml">0</mn><mo lspace="0.330em" rspace="0em" id="Ex15.m1.1.1.3.1" xref="Ex15.m1.1.1.3.1.cmml">​</mo><mtext id="Ex15.m1.1.1.3.3" xref="Ex15.m1.1.1.3.3a.cmml">TFlops</mtext></mrow></mrow><annotation-xml encoding="MathML-Content" id="Ex15.m1.1b"><apply id="Ex15.m1.1.1.cmml" xref="Ex15.m1.1.1"><eq id="Ex15.m1.1.1.1.cmml" xref="Ex15.m1.1.1.1"></eq><apply id="Ex15.m1.1.1.2.cmml" xref="Ex15.m1.1.1.2"><csymbol cd="ambiguous" id="Ex15.m1.1.1.2.1.cmml" xref="Ex15.m1.1.1.2">subscript</csymbol><ci id="Ex15.m1.1.1.2.2a.cmml" xref="Ex15.m1.1.1.2.2"><mtext id="Ex15.m1.1.1.2.2.cmml" xref="Ex15.m1.1.1.2.2">cost</mtext></ci><ci id="Ex15.m1.1.1.2.3a.cmml" xref="Ex15.m1.1.1.2.3"><mtext mathsize="70%" id="Ex15.m1.1.1.2.3.cmml" xref="Ex15.m1.1.1.2.3">write</mtext></ci></apply><apply id="Ex15.m1.1.1.3.cmml" xref="Ex15.m1.1.1.3"><times id="Ex15.m1.1.1.3.1.cmml" xref="Ex15.m1.1.1.3.1"></times><cn type="integer" id="Ex15.m1.1.1.3.2.cmml" xref="Ex15.m1.1.1.3.2">0</cn><ci id="Ex15.m1.1.1.3.3a.cmml" xref="Ex15.m1.1.1.3.3"><mtext id="Ex15.m1.1.1.3.3.cmml" xref="Ex15.m1.1.1.3.3">TFlops</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Ex15.m1.1c">\text{cost}_{\text{write}}=0~{}\text{TFlops}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="A1.SS3.p2" class="ltx_para">
<p id="A1.SS3.p2.1" class="ltx_p">The read cost is the additional compute brought by the retrieved references that are inserted in the prompt.
To make RAG appear more competitive, we assume that only a chunk of the prompt or decoded text with length <math id="A1.SS3.p2.1.m1.1" class="ltx_Math" alttext="l_{\text{chunk}}" display="inline"><semantics id="A1.SS3.p2.1.m1.1a"><msub id="A1.SS3.p2.1.m1.1.1" xref="A1.SS3.p2.1.m1.1.1.cmml"><mi id="A1.SS3.p2.1.m1.1.1.2" xref="A1.SS3.p2.1.m1.1.1.2.cmml">l</mi><mtext id="A1.SS3.p2.1.m1.1.1.3" xref="A1.SS3.p2.1.m1.1.1.3a.cmml">chunk</mtext></msub><annotation-xml encoding="MathML-Content" id="A1.SS3.p2.1.m1.1b"><apply id="A1.SS3.p2.1.m1.1.1.cmml" xref="A1.SS3.p2.1.m1.1.1"><csymbol cd="ambiguous" id="A1.SS3.p2.1.m1.1.1.1.cmml" xref="A1.SS3.p2.1.m1.1.1">subscript</csymbol><ci id="A1.SS3.p2.1.m1.1.1.2.cmml" xref="A1.SS3.p2.1.m1.1.1.2">𝑙</ci><ci id="A1.SS3.p2.1.m1.1.1.3a.cmml" xref="A1.SS3.p2.1.m1.1.1.3"><mtext mathsize="70%" id="A1.SS3.p2.1.m1.1.1.3.cmml" xref="A1.SS3.p2.1.m1.1.1.3">chunk</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p2.1.m1.1c">l_{\text{chunk}}</annotation></semantics></math> can attend to the references, and each reference can only attend to itself, which in general is not true.
Then,</p>
<table id="A3.EGx3" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="Ex16"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="Ex16.m1.1" class="ltx_Math" alttext="\displaystyle\text{cost}_{\text{write}}" display="inline"><semantics id="Ex16.m1.1a"><msub id="Ex16.m1.1.1" xref="Ex16.m1.1.1.cmml"><mtext id="Ex16.m1.1.1.2" xref="Ex16.m1.1.1.2a.cmml">cost</mtext><mtext id="Ex16.m1.1.1.3" xref="Ex16.m1.1.1.3a.cmml">write</mtext></msub><annotation-xml encoding="MathML-Content" id="Ex16.m1.1b"><apply id="Ex16.m1.1.1.cmml" xref="Ex16.m1.1.1"><csymbol cd="ambiguous" id="Ex16.m1.1.1.1.cmml" xref="Ex16.m1.1.1">subscript</csymbol><ci id="Ex16.m1.1.1.2a.cmml" xref="Ex16.m1.1.1.2"><mtext id="Ex16.m1.1.1.2.cmml" xref="Ex16.m1.1.1.2">cost</mtext></ci><ci id="Ex16.m1.1.1.3a.cmml" xref="Ex16.m1.1.1.3"><mtext mathsize="70%" id="Ex16.m1.1.1.3.cmml" xref="Ex16.m1.1.1.3">write</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="Ex16.m1.1c">\displaystyle\text{cost}_{\text{write}}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="Ex16.m2.1" class="ltx_Math" alttext="\displaystyle\geq 2\cdot\big{[}L\big{(}l_{\text{ref}}(2d^{2}+2dd_{h}H_{kv})+2l_{\text{ref}}\big{(}\frac{l_{\text{ref}}}{2}+l_{\text{chunk}}\big{)}d\big{)}+(L-1)(l_{\text{ref}}\cdot 3dW)\big{]}" display="inline"><semantics id="Ex16.m2.1a"><mrow id="Ex16.m2.1.1" xref="Ex16.m2.1.1.cmml"><mi id="Ex16.m2.1.1.3" xref="Ex16.m2.1.1.3.cmml"></mi><mo id="Ex16.m2.1.1.2" xref="Ex16.m2.1.1.2.cmml">≥</mo><mrow id="Ex16.m2.1.1.1" xref="Ex16.m2.1.1.1.cmml"><mn id="Ex16.m2.1.1.1.3" xref="Ex16.m2.1.1.1.3.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="Ex16.m2.1.1.1.2" xref="Ex16.m2.1.1.1.2.cmml">⋅</mo><mrow id="Ex16.m2.1.1.1.1.1" xref="Ex16.m2.1.1.1.1.2.cmml"><mo maxsize="120%" minsize="120%" id="Ex16.m2.1.1.1.1.1.2" xref="Ex16.m2.1.1.1.1.2.1.cmml">[</mo><mrow id="Ex16.m2.1.1.1.1.1.1" xref="Ex16.m2.1.1.1.1.1.1.cmml"><mrow id="Ex16.m2.1.1.1.1.1.1.1" xref="Ex16.m2.1.1.1.1.1.1.1.cmml"><mi id="Ex16.m2.1.1.1.1.1.1.1.3" xref="Ex16.m2.1.1.1.1.1.1.1.3.cmml">L</mi><mo lspace="0em" rspace="0em" id="Ex16.m2.1.1.1.1.1.1.1.2" xref="Ex16.m2.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="Ex16.m2.1.1.1.1.1.1.1.1.1" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.cmml"><mo maxsize="120%" minsize="120%" id="Ex16.m2.1.1.1.1.1.1.1.1.1.2" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="Ex16.m2.1.1.1.1.1.1.1.1.1.1" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.cmml"><mrow id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.cmml"><msub id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.3" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.3.2" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml">l</mi><mtext id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.3.3" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.3.3a.cmml">ref</mtext></msub><mo lspace="0em" rspace="0em" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.2" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mrow id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml"><mn id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.2" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.1" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml">​</mo><msup id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml"><mi id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.2" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.2.cmml">d</mi><mn id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.3" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.3.cmml">2</mn></msup></mrow><mo id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">+</mo><mrow id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml"><mn id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.2" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.cmml">d</mi><mo lspace="0em" rspace="0em" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1a" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><msub id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.cmml"><mi id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.2" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.2.cmml">d</mi><mi id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.3" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.3.cmml">h</mi></msub><mo lspace="0em" rspace="0em" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1b" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><msub id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.cmml"><mi id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.2" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.2.cmml">H</mi><mrow id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.cmml"><mi id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.2" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.2.cmml">k</mi><mo lspace="0em" rspace="0em" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.1" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.1.cmml">​</mo><mi id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.3" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.3.cmml">v</mi></mrow></msub></mrow></mrow><mo stretchy="false" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.3" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.3.cmml">+</mo><mrow id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.cmml"><mn id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.3" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.3.cmml">2</mn><mo lspace="0em" rspace="0em" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.2" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.2.cmml">​</mo><msub id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.4" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.4.cmml"><mi id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.4.2" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.4.2.cmml">l</mi><mtext id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.4.3" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.4.3a.cmml">ref</mtext></msub><mo lspace="0em" rspace="0em" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.2a" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.2.cmml">​</mo><mrow id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.cmml"><mo maxsize="120%" minsize="120%" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.2" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.cmml">(</mo><mrow id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.cmml"><mstyle displaystyle="true" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.2" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.2.cmml"><mfrac id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.2a" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.2.cmml"><msub id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.2.2" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.2.2.cmml"><mi id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.2.2.2" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.2.2.2.cmml">l</mi><mtext id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.2.2.3" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.2.2.3a.cmml">ref</mtext></msub><mn id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.2.3" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.2.3.cmml">2</mn></mfrac></mstyle><mo id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.1" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.1.cmml">+</mo><msub id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.3" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.3.cmml"><mi id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.3.2" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.3.2.cmml">l</mi><mtext id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.3.3" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.3.3a.cmml">chunk</mtext></msub></mrow><mo maxsize="120%" minsize="120%" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.3" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.2b" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.2.cmml">​</mo><mi id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.5" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.5.cmml">d</mi></mrow></mrow><mo maxsize="120%" minsize="120%" id="Ex16.m2.1.1.1.1.1.1.1.1.1.3" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="Ex16.m2.1.1.1.1.1.1.4" xref="Ex16.m2.1.1.1.1.1.1.4.cmml">+</mo><mrow id="Ex16.m2.1.1.1.1.1.1.3" xref="Ex16.m2.1.1.1.1.1.1.3.cmml"><mrow id="Ex16.m2.1.1.1.1.1.1.2.1.1" xref="Ex16.m2.1.1.1.1.1.1.2.1.1.1.cmml"><mo stretchy="false" id="Ex16.m2.1.1.1.1.1.1.2.1.1.2" xref="Ex16.m2.1.1.1.1.1.1.2.1.1.1.cmml">(</mo><mrow id="Ex16.m2.1.1.1.1.1.1.2.1.1.1" xref="Ex16.m2.1.1.1.1.1.1.2.1.1.1.cmml"><mi id="Ex16.m2.1.1.1.1.1.1.2.1.1.1.2" xref="Ex16.m2.1.1.1.1.1.1.2.1.1.1.2.cmml">L</mi><mo id="Ex16.m2.1.1.1.1.1.1.2.1.1.1.1" xref="Ex16.m2.1.1.1.1.1.1.2.1.1.1.1.cmml">−</mo><mn id="Ex16.m2.1.1.1.1.1.1.2.1.1.1.3" xref="Ex16.m2.1.1.1.1.1.1.2.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="Ex16.m2.1.1.1.1.1.1.2.1.1.3" xref="Ex16.m2.1.1.1.1.1.1.2.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="Ex16.m2.1.1.1.1.1.1.3.3" xref="Ex16.m2.1.1.1.1.1.1.3.3.cmml">​</mo><mrow id="Ex16.m2.1.1.1.1.1.1.3.2.1" xref="Ex16.m2.1.1.1.1.1.1.3.2.1.1.cmml"><mo stretchy="false" id="Ex16.m2.1.1.1.1.1.1.3.2.1.2" xref="Ex16.m2.1.1.1.1.1.1.3.2.1.1.cmml">(</mo><mrow id="Ex16.m2.1.1.1.1.1.1.3.2.1.1" xref="Ex16.m2.1.1.1.1.1.1.3.2.1.1.cmml"><mrow id="Ex16.m2.1.1.1.1.1.1.3.2.1.1.2" xref="Ex16.m2.1.1.1.1.1.1.3.2.1.1.2.cmml"><msub id="Ex16.m2.1.1.1.1.1.1.3.2.1.1.2.2" xref="Ex16.m2.1.1.1.1.1.1.3.2.1.1.2.2.cmml"><mi id="Ex16.m2.1.1.1.1.1.1.3.2.1.1.2.2.2" xref="Ex16.m2.1.1.1.1.1.1.3.2.1.1.2.2.2.cmml">l</mi><mtext id="Ex16.m2.1.1.1.1.1.1.3.2.1.1.2.2.3" xref="Ex16.m2.1.1.1.1.1.1.3.2.1.1.2.2.3a.cmml">ref</mtext></msub><mo lspace="0.222em" rspace="0.222em" id="Ex16.m2.1.1.1.1.1.1.3.2.1.1.2.1" xref="Ex16.m2.1.1.1.1.1.1.3.2.1.1.2.1.cmml">⋅</mo><mn id="Ex16.m2.1.1.1.1.1.1.3.2.1.1.2.3" xref="Ex16.m2.1.1.1.1.1.1.3.2.1.1.2.3.cmml">3</mn></mrow><mo lspace="0em" rspace="0em" id="Ex16.m2.1.1.1.1.1.1.3.2.1.1.1" xref="Ex16.m2.1.1.1.1.1.1.3.2.1.1.1.cmml">​</mo><mi id="Ex16.m2.1.1.1.1.1.1.3.2.1.1.3" xref="Ex16.m2.1.1.1.1.1.1.3.2.1.1.3.cmml">d</mi><mo lspace="0em" rspace="0em" id="Ex16.m2.1.1.1.1.1.1.3.2.1.1.1a" xref="Ex16.m2.1.1.1.1.1.1.3.2.1.1.1.cmml">​</mo><mi id="Ex16.m2.1.1.1.1.1.1.3.2.1.1.4" xref="Ex16.m2.1.1.1.1.1.1.3.2.1.1.4.cmml">W</mi></mrow><mo stretchy="false" id="Ex16.m2.1.1.1.1.1.1.3.2.1.3" xref="Ex16.m2.1.1.1.1.1.1.3.2.1.1.cmml">)</mo></mrow></mrow></mrow><mo maxsize="120%" minsize="120%" id="Ex16.m2.1.1.1.1.1.3" xref="Ex16.m2.1.1.1.1.2.1.cmml">]</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="Ex16.m2.1b"><apply id="Ex16.m2.1.1.cmml" xref="Ex16.m2.1.1"><geq id="Ex16.m2.1.1.2.cmml" xref="Ex16.m2.1.1.2"></geq><csymbol cd="latexml" id="Ex16.m2.1.1.3.cmml" xref="Ex16.m2.1.1.3">absent</csymbol><apply id="Ex16.m2.1.1.1.cmml" xref="Ex16.m2.1.1.1"><ci id="Ex16.m2.1.1.1.2.cmml" xref="Ex16.m2.1.1.1.2">⋅</ci><cn type="integer" id="Ex16.m2.1.1.1.3.cmml" xref="Ex16.m2.1.1.1.3">2</cn><apply id="Ex16.m2.1.1.1.1.2.cmml" xref="Ex16.m2.1.1.1.1.1"><csymbol cd="latexml" id="Ex16.m2.1.1.1.1.2.1.cmml" xref="Ex16.m2.1.1.1.1.1.2">delimited-[]</csymbol><apply id="Ex16.m2.1.1.1.1.1.1.cmml" xref="Ex16.m2.1.1.1.1.1.1"><plus id="Ex16.m2.1.1.1.1.1.1.4.cmml" xref="Ex16.m2.1.1.1.1.1.1.4"></plus><apply id="Ex16.m2.1.1.1.1.1.1.1.cmml" xref="Ex16.m2.1.1.1.1.1.1.1"><times id="Ex16.m2.1.1.1.1.1.1.1.2.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.2"></times><ci id="Ex16.m2.1.1.1.1.1.1.1.3.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.3">𝐿</ci><apply id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1"><plus id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.3"></plus><apply id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1"><times id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.2"></times><apply id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.3.2">𝑙</ci><ci id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.3.3a.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.3.3"><mtext mathsize="70%" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.3.3">ref</mtext></ci></apply><apply id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1"><plus id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1"></plus><apply id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2"><times id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.1"></times><cn type="integer" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.2">2</cn><apply id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.1.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3">superscript</csymbol><ci id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.2.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.2">𝑑</ci><cn type="integer" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.3.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.3">2</cn></apply></apply><apply id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3"><times id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1"></times><cn type="integer" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.2">2</cn><ci id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3">𝑑</ci><apply id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4"><csymbol cd="ambiguous" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.1.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4">subscript</csymbol><ci id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.2.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.2">𝑑</ci><ci id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.3.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.4.3">ℎ</ci></apply><apply id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5"><csymbol cd="ambiguous" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.1.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5">subscript</csymbol><ci id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.2.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.2">𝐻</ci><apply id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3"><times id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.1.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.1"></times><ci id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.2.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.2">𝑘</ci><ci id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.3.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.5.3.3">𝑣</ci></apply></apply></apply></apply></apply><apply id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2"><times id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.2"></times><cn type="integer" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.3">2</cn><apply id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.4.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.4"><csymbol cd="ambiguous" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.4.1.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.4">subscript</csymbol><ci id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.4.2.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.4.2">𝑙</ci><ci id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.4.3a.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.4.3"><mtext mathsize="70%" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.4.3.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.4.3">ref</mtext></ci></apply><apply id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1"><plus id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.1.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.1"></plus><apply id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.2.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.2"><divide id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.2.1.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.2"></divide><apply id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.2.2.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.2.2"><csymbol cd="ambiguous" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.2.2.1.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.2.2">subscript</csymbol><ci id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.2.2.2.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.2.2.2">𝑙</ci><ci id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.2.2.3a.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.2.2.3"><mtext mathsize="70%" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.2.2.3.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.2.2.3">ref</mtext></ci></apply><cn type="integer" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.2.3.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.2.3">2</cn></apply><apply id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.3.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.3"><csymbol cd="ambiguous" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.3.1.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.3">subscript</csymbol><ci id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.3.2.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.3.2">𝑙</ci><ci id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.3.3a.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.3.3"><mtext mathsize="70%" id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.3.3.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.1.1.1.3.3">chunk</mtext></ci></apply></apply><ci id="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.5.cmml" xref="Ex16.m2.1.1.1.1.1.1.1.1.1.1.2.5">𝑑</ci></apply></apply></apply><apply id="Ex16.m2.1.1.1.1.1.1.3.cmml" xref="Ex16.m2.1.1.1.1.1.1.3"><times id="Ex16.m2.1.1.1.1.1.1.3.3.cmml" xref="Ex16.m2.1.1.1.1.1.1.3.3"></times><apply id="Ex16.m2.1.1.1.1.1.1.2.1.1.1.cmml" xref="Ex16.m2.1.1.1.1.1.1.2.1.1"><minus id="Ex16.m2.1.1.1.1.1.1.2.1.1.1.1.cmml" xref="Ex16.m2.1.1.1.1.1.1.2.1.1.1.1"></minus><ci id="Ex16.m2.1.1.1.1.1.1.2.1.1.1.2.cmml" xref="Ex16.m2.1.1.1.1.1.1.2.1.1.1.2">𝐿</ci><cn type="integer" id="Ex16.m2.1.1.1.1.1.1.2.1.1.1.3.cmml" xref="Ex16.m2.1.1.1.1.1.1.2.1.1.1.3">1</cn></apply><apply id="Ex16.m2.1.1.1.1.1.1.3.2.1.1.cmml" xref="Ex16.m2.1.1.1.1.1.1.3.2.1"><times id="Ex16.m2.1.1.1.1.1.1.3.2.1.1.1.cmml" xref="Ex16.m2.1.1.1.1.1.1.3.2.1.1.1"></times><apply id="Ex16.m2.1.1.1.1.1.1.3.2.1.1.2.cmml" xref="Ex16.m2.1.1.1.1.1.1.3.2.1.1.2"><ci id="Ex16.m2.1.1.1.1.1.1.3.2.1.1.2.1.cmml" xref="Ex16.m2.1.1.1.1.1.1.3.2.1.1.2.1">⋅</ci><apply id="Ex16.m2.1.1.1.1.1.1.3.2.1.1.2.2.cmml" xref="Ex16.m2.1.1.1.1.1.1.3.2.1.1.2.2"><csymbol cd="ambiguous" id="Ex16.m2.1.1.1.1.1.1.3.2.1.1.2.2.1.cmml" xref="Ex16.m2.1.1.1.1.1.1.3.2.1.1.2.2">subscript</csymbol><ci id="Ex16.m2.1.1.1.1.1.1.3.2.1.1.2.2.2.cmml" xref="Ex16.m2.1.1.1.1.1.1.3.2.1.1.2.2.2">𝑙</ci><ci id="Ex16.m2.1.1.1.1.1.1.3.2.1.1.2.2.3a.cmml" xref="Ex16.m2.1.1.1.1.1.1.3.2.1.1.2.2.3"><mtext mathsize="70%" id="Ex16.m2.1.1.1.1.1.1.3.2.1.1.2.2.3.cmml" xref="Ex16.m2.1.1.1.1.1.1.3.2.1.1.2.2.3">ref</mtext></ci></apply><cn type="integer" id="Ex16.m2.1.1.1.1.1.1.3.2.1.1.2.3.cmml" xref="Ex16.m2.1.1.1.1.1.1.3.2.1.1.2.3">3</cn></apply><ci id="Ex16.m2.1.1.1.1.1.1.3.2.1.1.3.cmml" xref="Ex16.m2.1.1.1.1.1.1.3.2.1.1.3">𝑑</ci><ci id="Ex16.m2.1.1.1.1.1.1.3.2.1.1.4.cmml" xref="Ex16.m2.1.1.1.1.1.1.3.2.1.1.4">𝑊</ci></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Ex16.m2.1c">\displaystyle\geq 2\cdot\big{[}L\big{(}l_{\text{ref}}(2d^{2}+2dd_{h}H_{kv})+2l_{\text{ref}}\big{(}\frac{l_{\text{ref}}}{2}+l_{\text{chunk}}\big{)}d\big{)}+(L-1)(l_{\text{ref}}\cdot 3dW)\big{]}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="Ex17"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="Ex17.m1.1" class="ltx_Math" alttext="\displaystyle\approx 0.624~{}\text{TFlops}" display="inline"><semantics id="Ex17.m1.1a"><mrow id="Ex17.m1.1.1" xref="Ex17.m1.1.1.cmml"><mi id="Ex17.m1.1.1.2" xref="Ex17.m1.1.1.2.cmml"></mi><mo id="Ex17.m1.1.1.1" xref="Ex17.m1.1.1.1.cmml">≈</mo><mrow id="Ex17.m1.1.1.3" xref="Ex17.m1.1.1.3.cmml"><mn id="Ex17.m1.1.1.3.2" xref="Ex17.m1.1.1.3.2.cmml">0.624</mn><mo lspace="0.330em" rspace="0em" id="Ex17.m1.1.1.3.1" xref="Ex17.m1.1.1.3.1.cmml">​</mo><mtext id="Ex17.m1.1.1.3.3" xref="Ex17.m1.1.1.3.3a.cmml">TFlops</mtext></mrow></mrow><annotation-xml encoding="MathML-Content" id="Ex17.m1.1b"><apply id="Ex17.m1.1.1.cmml" xref="Ex17.m1.1.1"><approx id="Ex17.m1.1.1.1.cmml" xref="Ex17.m1.1.1.1"></approx><csymbol cd="latexml" id="Ex17.m1.1.1.2.cmml" xref="Ex17.m1.1.1.2">absent</csymbol><apply id="Ex17.m1.1.1.3.cmml" xref="Ex17.m1.1.1.3"><times id="Ex17.m1.1.1.3.1.cmml" xref="Ex17.m1.1.1.3.1"></times><cn type="float" id="Ex17.m1.1.1.3.2.cmml" xref="Ex17.m1.1.1.3.2">0.624</cn><ci id="Ex17.m1.1.1.3.3a.cmml" xref="Ex17.m1.1.1.3.3"><mtext id="Ex17.m1.1.1.3.3.cmml" xref="Ex17.m1.1.1.3.3">TFlops</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Ex17.m1.1c">\displaystyle\approx 0.624~{}\text{TFlops}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="A1.SS3.p3" class="ltx_para">
<p id="A1.SS3.p3.1" class="ltx_p">In summary, the total cost (TFlops) of writing and reading each separable knowledge in terms of its expected usage count <math id="A1.SS3.p3.1.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="A1.SS3.p3.1.m1.1a"><mi id="A1.SS3.p3.1.m1.1.1" xref="A1.SS3.p3.1.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="A1.SS3.p3.1.m1.1b"><ci id="A1.SS3.p3.1.m1.1.1.cmml" xref="A1.SS3.p3.1.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p3.1.m1.1c">n</annotation></semantics></math> is given by</p>
<table id="Ex18" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="Ex18.m1.3" class="ltx_Math" alttext="\begin{cases}c_{\text{implicit}}(n)\geq 2.24\\
c_{\text{explicit}}(n)=0.308+0.000144n\\
c_{\text{external}}(n)\geq 0.624n\end{cases}" display="block"><semantics id="Ex18.m1.3a"><mrow id="Ex18.m1.3.3" xref="Ex18.m1.3.4.1.cmml"><mo id="Ex18.m1.3.3.4" xref="Ex18.m1.3.4.1.1.cmml">{</mo><mtable columnspacing="5pt" displaystyle="true" rowspacing="0pt" id="Ex18.m1.3.3.3" xref="Ex18.m1.3.4.1.cmml"><mtr id="Ex18.m1.3.3.3a" xref="Ex18.m1.3.4.1.cmml"><mtd class="ltx_align_left" columnalign="left" id="Ex18.m1.3.3.3b" xref="Ex18.m1.3.4.1.cmml"><mrow id="Ex18.m1.1.1.1.1.1.1" xref="Ex18.m1.1.1.1.1.1.1.cmml"><mrow id="Ex18.m1.1.1.1.1.1.1.3" xref="Ex18.m1.1.1.1.1.1.1.3.cmml"><msub id="Ex18.m1.1.1.1.1.1.1.3.2" xref="Ex18.m1.1.1.1.1.1.1.3.2.cmml"><mi id="Ex18.m1.1.1.1.1.1.1.3.2.2" xref="Ex18.m1.1.1.1.1.1.1.3.2.2.cmml">c</mi><mtext id="Ex18.m1.1.1.1.1.1.1.3.2.3" xref="Ex18.m1.1.1.1.1.1.1.3.2.3a.cmml">implicit</mtext></msub><mo lspace="0em" rspace="0em" id="Ex18.m1.1.1.1.1.1.1.3.1" xref="Ex18.m1.1.1.1.1.1.1.3.1.cmml">​</mo><mrow id="Ex18.m1.1.1.1.1.1.1.3.3.2" xref="Ex18.m1.1.1.1.1.1.1.3.cmml"><mo stretchy="false" id="Ex18.m1.1.1.1.1.1.1.3.3.2.1" xref="Ex18.m1.1.1.1.1.1.1.3.cmml">(</mo><mi id="Ex18.m1.1.1.1.1.1.1.1" xref="Ex18.m1.1.1.1.1.1.1.1.cmml">n</mi><mo stretchy="false" id="Ex18.m1.1.1.1.1.1.1.3.3.2.2" xref="Ex18.m1.1.1.1.1.1.1.3.cmml">)</mo></mrow></mrow><mo id="Ex18.m1.1.1.1.1.1.1.2" xref="Ex18.m1.1.1.1.1.1.1.2.cmml">≥</mo><mn id="Ex18.m1.1.1.1.1.1.1.4" xref="Ex18.m1.1.1.1.1.1.1.4.cmml">2.24</mn></mrow></mtd><mtd id="Ex18.m1.3.3.3c" xref="Ex18.m1.3.4.1.1.cmml"></mtd></mtr><mtr id="Ex18.m1.3.3.3d" xref="Ex18.m1.3.4.1.cmml"><mtd class="ltx_align_left" columnalign="left" id="Ex18.m1.3.3.3e" xref="Ex18.m1.3.4.1.cmml"><mrow id="Ex18.m1.2.2.2.2.1.1" xref="Ex18.m1.2.2.2.2.1.1.cmml"><mrow id="Ex18.m1.2.2.2.2.1.1.3" xref="Ex18.m1.2.2.2.2.1.1.3.cmml"><msub id="Ex18.m1.2.2.2.2.1.1.3.2" xref="Ex18.m1.2.2.2.2.1.1.3.2.cmml"><mi id="Ex18.m1.2.2.2.2.1.1.3.2.2" xref="Ex18.m1.2.2.2.2.1.1.3.2.2.cmml">c</mi><mtext id="Ex18.m1.2.2.2.2.1.1.3.2.3" xref="Ex18.m1.2.2.2.2.1.1.3.2.3a.cmml">explicit</mtext></msub><mo lspace="0em" rspace="0em" id="Ex18.m1.2.2.2.2.1.1.3.1" xref="Ex18.m1.2.2.2.2.1.1.3.1.cmml">​</mo><mrow id="Ex18.m1.2.2.2.2.1.1.3.3.2" xref="Ex18.m1.2.2.2.2.1.1.3.cmml"><mo stretchy="false" id="Ex18.m1.2.2.2.2.1.1.3.3.2.1" xref="Ex18.m1.2.2.2.2.1.1.3.cmml">(</mo><mi id="Ex18.m1.2.2.2.2.1.1.1" xref="Ex18.m1.2.2.2.2.1.1.1.cmml">n</mi><mo stretchy="false" id="Ex18.m1.2.2.2.2.1.1.3.3.2.2" xref="Ex18.m1.2.2.2.2.1.1.3.cmml">)</mo></mrow></mrow><mo id="Ex18.m1.2.2.2.2.1.1.2" xref="Ex18.m1.2.2.2.2.1.1.2.cmml">=</mo><mrow id="Ex18.m1.2.2.2.2.1.1.4" xref="Ex18.m1.2.2.2.2.1.1.4.cmml"><mn id="Ex18.m1.2.2.2.2.1.1.4.2" xref="Ex18.m1.2.2.2.2.1.1.4.2.cmml">0.308</mn><mo id="Ex18.m1.2.2.2.2.1.1.4.1" xref="Ex18.m1.2.2.2.2.1.1.4.1.cmml">+</mo><mrow id="Ex18.m1.2.2.2.2.1.1.4.3" xref="Ex18.m1.2.2.2.2.1.1.4.3.cmml"><mn id="Ex18.m1.2.2.2.2.1.1.4.3.2" xref="Ex18.m1.2.2.2.2.1.1.4.3.2.cmml">0.000144</mn><mo lspace="0em" rspace="0em" id="Ex18.m1.2.2.2.2.1.1.4.3.1" xref="Ex18.m1.2.2.2.2.1.1.4.3.1.cmml">​</mo><mi id="Ex18.m1.2.2.2.2.1.1.4.3.3" xref="Ex18.m1.2.2.2.2.1.1.4.3.3.cmml">n</mi></mrow></mrow></mrow></mtd><mtd id="Ex18.m1.3.3.3f" xref="Ex18.m1.3.4.1.1.cmml"></mtd></mtr><mtr id="Ex18.m1.3.3.3g" xref="Ex18.m1.3.4.1.cmml"><mtd class="ltx_align_left" columnalign="left" id="Ex18.m1.3.3.3h" xref="Ex18.m1.3.4.1.cmml"><mrow id="Ex18.m1.3.3.3.3.1.1" xref="Ex18.m1.3.3.3.3.1.1.cmml"><mrow id="Ex18.m1.3.3.3.3.1.1.3" xref="Ex18.m1.3.3.3.3.1.1.3.cmml"><msub id="Ex18.m1.3.3.3.3.1.1.3.2" xref="Ex18.m1.3.3.3.3.1.1.3.2.cmml"><mi id="Ex18.m1.3.3.3.3.1.1.3.2.2" xref="Ex18.m1.3.3.3.3.1.1.3.2.2.cmml">c</mi><mtext id="Ex18.m1.3.3.3.3.1.1.3.2.3" xref="Ex18.m1.3.3.3.3.1.1.3.2.3a.cmml">external</mtext></msub><mo lspace="0em" rspace="0em" id="Ex18.m1.3.3.3.3.1.1.3.1" xref="Ex18.m1.3.3.3.3.1.1.3.1.cmml">​</mo><mrow id="Ex18.m1.3.3.3.3.1.1.3.3.2" xref="Ex18.m1.3.3.3.3.1.1.3.cmml"><mo stretchy="false" id="Ex18.m1.3.3.3.3.1.1.3.3.2.1" xref="Ex18.m1.3.3.3.3.1.1.3.cmml">(</mo><mi id="Ex18.m1.3.3.3.3.1.1.1" xref="Ex18.m1.3.3.3.3.1.1.1.cmml">n</mi><mo stretchy="false" id="Ex18.m1.3.3.3.3.1.1.3.3.2.2" xref="Ex18.m1.3.3.3.3.1.1.3.cmml">)</mo></mrow></mrow><mo id="Ex18.m1.3.3.3.3.1.1.2" xref="Ex18.m1.3.3.3.3.1.1.2.cmml">≥</mo><mrow id="Ex18.m1.3.3.3.3.1.1.4" xref="Ex18.m1.3.3.3.3.1.1.4.cmml"><mn id="Ex18.m1.3.3.3.3.1.1.4.2" xref="Ex18.m1.3.3.3.3.1.1.4.2.cmml">0.624</mn><mo lspace="0em" rspace="0em" id="Ex18.m1.3.3.3.3.1.1.4.1" xref="Ex18.m1.3.3.3.3.1.1.4.1.cmml">​</mo><mi id="Ex18.m1.3.3.3.3.1.1.4.3" xref="Ex18.m1.3.3.3.3.1.1.4.3.cmml">n</mi></mrow></mrow></mtd><mtd id="Ex18.m1.3.3.3i" xref="Ex18.m1.3.4.1.1.cmml"></mtd></mtr></mtable></mrow><annotation-xml encoding="MathML-Content" id="Ex18.m1.3b"><apply id="Ex18.m1.3.4.1.cmml" xref="Ex18.m1.3.3"><csymbol cd="latexml" id="Ex18.m1.3.4.1.1.cmml" xref="Ex18.m1.3.3.4">cases</csymbol><apply id="Ex18.m1.1.1.1.1.1.1.cmml" xref="Ex18.m1.1.1.1.1.1.1"><geq id="Ex18.m1.1.1.1.1.1.1.2.cmml" xref="Ex18.m1.1.1.1.1.1.1.2"></geq><apply id="Ex18.m1.1.1.1.1.1.1.3.cmml" xref="Ex18.m1.1.1.1.1.1.1.3"><times id="Ex18.m1.1.1.1.1.1.1.3.1.cmml" xref="Ex18.m1.1.1.1.1.1.1.3.1"></times><apply id="Ex18.m1.1.1.1.1.1.1.3.2.cmml" xref="Ex18.m1.1.1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="Ex18.m1.1.1.1.1.1.1.3.2.1.cmml" xref="Ex18.m1.1.1.1.1.1.1.3.2">subscript</csymbol><ci id="Ex18.m1.1.1.1.1.1.1.3.2.2.cmml" xref="Ex18.m1.1.1.1.1.1.1.3.2.2">𝑐</ci><ci id="Ex18.m1.1.1.1.1.1.1.3.2.3a.cmml" xref="Ex18.m1.1.1.1.1.1.1.3.2.3"><mtext mathsize="70%" id="Ex18.m1.1.1.1.1.1.1.3.2.3.cmml" xref="Ex18.m1.1.1.1.1.1.1.3.2.3">implicit</mtext></ci></apply><ci id="Ex18.m1.1.1.1.1.1.1.1.cmml" xref="Ex18.m1.1.1.1.1.1.1.1">𝑛</ci></apply><cn type="float" id="Ex18.m1.1.1.1.1.1.1.4.cmml" xref="Ex18.m1.1.1.1.1.1.1.4">2.24</cn></apply><ci id="Ex18.m1.3.4.1.3a.cmml" xref="Ex18.m1.3.3"><mtext class="ltx_mathvariant_italic" id="Ex18.m1.3.4.1.3.cmml" xref="Ex18.m1.3.3.4">otherwise</mtext></ci><apply id="Ex18.m1.2.2.2.2.1.1.cmml" xref="Ex18.m1.2.2.2.2.1.1"><eq id="Ex18.m1.2.2.2.2.1.1.2.cmml" xref="Ex18.m1.2.2.2.2.1.1.2"></eq><apply id="Ex18.m1.2.2.2.2.1.1.3.cmml" xref="Ex18.m1.2.2.2.2.1.1.3"><times id="Ex18.m1.2.2.2.2.1.1.3.1.cmml" xref="Ex18.m1.2.2.2.2.1.1.3.1"></times><apply id="Ex18.m1.2.2.2.2.1.1.3.2.cmml" xref="Ex18.m1.2.2.2.2.1.1.3.2"><csymbol cd="ambiguous" id="Ex18.m1.2.2.2.2.1.1.3.2.1.cmml" xref="Ex18.m1.2.2.2.2.1.1.3.2">subscript</csymbol><ci id="Ex18.m1.2.2.2.2.1.1.3.2.2.cmml" xref="Ex18.m1.2.2.2.2.1.1.3.2.2">𝑐</ci><ci id="Ex18.m1.2.2.2.2.1.1.3.2.3a.cmml" xref="Ex18.m1.2.2.2.2.1.1.3.2.3"><mtext mathsize="70%" id="Ex18.m1.2.2.2.2.1.1.3.2.3.cmml" xref="Ex18.m1.2.2.2.2.1.1.3.2.3">explicit</mtext></ci></apply><ci id="Ex18.m1.2.2.2.2.1.1.1.cmml" xref="Ex18.m1.2.2.2.2.1.1.1">𝑛</ci></apply><apply id="Ex18.m1.2.2.2.2.1.1.4.cmml" xref="Ex18.m1.2.2.2.2.1.1.4"><plus id="Ex18.m1.2.2.2.2.1.1.4.1.cmml" xref="Ex18.m1.2.2.2.2.1.1.4.1"></plus><cn type="float" id="Ex18.m1.2.2.2.2.1.1.4.2.cmml" xref="Ex18.m1.2.2.2.2.1.1.4.2">0.308</cn><apply id="Ex18.m1.2.2.2.2.1.1.4.3.cmml" xref="Ex18.m1.2.2.2.2.1.1.4.3"><times id="Ex18.m1.2.2.2.2.1.1.4.3.1.cmml" xref="Ex18.m1.2.2.2.2.1.1.4.3.1"></times><cn type="float" id="Ex18.m1.2.2.2.2.1.1.4.3.2.cmml" xref="Ex18.m1.2.2.2.2.1.1.4.3.2">0.000144</cn><ci id="Ex18.m1.2.2.2.2.1.1.4.3.3.cmml" xref="Ex18.m1.2.2.2.2.1.1.4.3.3">𝑛</ci></apply></apply></apply><ci id="Ex18.m1.3.4.1.5a.cmml" xref="Ex18.m1.3.3"><mtext class="ltx_mathvariant_italic" id="Ex18.m1.3.4.1.5.cmml" xref="Ex18.m1.3.3.4">otherwise</mtext></ci><apply id="Ex18.m1.3.3.3.3.1.1.cmml" xref="Ex18.m1.3.3.3.3.1.1"><geq id="Ex18.m1.3.3.3.3.1.1.2.cmml" xref="Ex18.m1.3.3.3.3.1.1.2"></geq><apply id="Ex18.m1.3.3.3.3.1.1.3.cmml" xref="Ex18.m1.3.3.3.3.1.1.3"><times id="Ex18.m1.3.3.3.3.1.1.3.1.cmml" xref="Ex18.m1.3.3.3.3.1.1.3.1"></times><apply id="Ex18.m1.3.3.3.3.1.1.3.2.cmml" xref="Ex18.m1.3.3.3.3.1.1.3.2"><csymbol cd="ambiguous" id="Ex18.m1.3.3.3.3.1.1.3.2.1.cmml" xref="Ex18.m1.3.3.3.3.1.1.3.2">subscript</csymbol><ci id="Ex18.m1.3.3.3.3.1.1.3.2.2.cmml" xref="Ex18.m1.3.3.3.3.1.1.3.2.2">𝑐</ci><ci id="Ex18.m1.3.3.3.3.1.1.3.2.3a.cmml" xref="Ex18.m1.3.3.3.3.1.1.3.2.3"><mtext mathsize="70%" id="Ex18.m1.3.3.3.3.1.1.3.2.3.cmml" xref="Ex18.m1.3.3.3.3.1.1.3.2.3">external</mtext></ci></apply><ci id="Ex18.m1.3.3.3.3.1.1.1.cmml" xref="Ex18.m1.3.3.3.3.1.1.1">𝑛</ci></apply><apply id="Ex18.m1.3.3.3.3.1.1.4.cmml" xref="Ex18.m1.3.3.3.3.1.1.4"><times id="Ex18.m1.3.3.3.3.1.1.4.1.cmml" xref="Ex18.m1.3.3.3.3.1.1.4.1"></times><cn type="float" id="Ex18.m1.3.3.3.3.1.1.4.2.cmml" xref="Ex18.m1.3.3.3.3.1.1.4.2">0.624</cn><ci id="Ex18.m1.3.3.3.3.1.1.4.3.cmml" xref="Ex18.m1.3.3.3.3.1.1.4.3">𝑛</ci></apply></apply><ci id="Ex18.m1.3.4.1.7a.cmml" xref="Ex18.m1.3.3"><mtext class="ltx_mathvariant_italic" id="Ex18.m1.3.4.1.7.cmml" xref="Ex18.m1.3.3.4">otherwise</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="Ex18.m1.3c">\begin{cases}c_{\text{implicit}}(n)\geq 2.24\\
c_{\text{explicit}}(n)=0.308+0.000144n\\
c_{\text{external}}(n)\geq 0.624n\end{cases}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="A1.SS3.p3.2" class="ltx_p">These curves are plotted in Figure <a href="#F4" title="Figure 4 ‣ 1 | Introduction ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>.
Hence, if <math id="A1.SS3.p3.2.m1.2" class="ltx_Math" alttext="n\in(0.494,13400)" display="inline"><semantics id="A1.SS3.p3.2.m1.2a"><mrow id="A1.SS3.p3.2.m1.2.3" xref="A1.SS3.p3.2.m1.2.3.cmml"><mi id="A1.SS3.p3.2.m1.2.3.2" xref="A1.SS3.p3.2.m1.2.3.2.cmml">n</mi><mo id="A1.SS3.p3.2.m1.2.3.1" xref="A1.SS3.p3.2.m1.2.3.1.cmml">∈</mo><mrow id="A1.SS3.p3.2.m1.2.3.3.2" xref="A1.SS3.p3.2.m1.2.3.3.1.cmml"><mo stretchy="false" id="A1.SS3.p3.2.m1.2.3.3.2.1" xref="A1.SS3.p3.2.m1.2.3.3.1.cmml">(</mo><mn id="A1.SS3.p3.2.m1.1.1" xref="A1.SS3.p3.2.m1.1.1.cmml">0.494</mn><mo id="A1.SS3.p3.2.m1.2.3.3.2.2" xref="A1.SS3.p3.2.m1.2.3.3.1.cmml">,</mo><mn id="A1.SS3.p3.2.m1.2.2" xref="A1.SS3.p3.2.m1.2.2.cmml">13400</mn><mo stretchy="false" id="A1.SS3.p3.2.m1.2.3.3.2.3" xref="A1.SS3.p3.2.m1.2.3.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS3.p3.2.m1.2b"><apply id="A1.SS3.p3.2.m1.2.3.cmml" xref="A1.SS3.p3.2.m1.2.3"><in id="A1.SS3.p3.2.m1.2.3.1.cmml" xref="A1.SS3.p3.2.m1.2.3.1"></in><ci id="A1.SS3.p3.2.m1.2.3.2.cmml" xref="A1.SS3.p3.2.m1.2.3.2">𝑛</ci><interval closure="open" id="A1.SS3.p3.2.m1.2.3.3.1.cmml" xref="A1.SS3.p3.2.m1.2.3.3.2"><cn type="float" id="A1.SS3.p3.2.m1.1.1.cmml" xref="A1.SS3.p3.2.m1.1.1">0.494</cn><cn type="integer" id="A1.SS3.p3.2.m1.2.2.cmml" xref="A1.SS3.p3.2.m1.2.2">13400</cn></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p3.2.m1.2c">n\in(0.494,13400)</annotation></semantics></math>, then it is optimal to store the knowledge as an explicit memory.</p>
</div>
<div id="Thmremark9" class="ltx_theorem ltx_theorem_remark">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmremark9.1.1.1" class="ltx_text ltx_font_bold">Remark 9</span></span><span id="Thmremark9.2.2" class="ltx_text ltx_font_bold"> </span>(Knowledge retention)<span id="Thmremark9.3.3" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmremark9.p1" class="ltx_para">
<p id="Thmremark9.p1.1" class="ltx_p">One aspect not covered by Problem (<a href="#E1" title="Equation 1 ‣ 1 | Introduction ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>) is the retention of knowledges in the model if its parameters are updated, e.g. due to finetuning.
Both implicit memory and explicit memory are vulnerable to parameter change.
Usually, model finetuning would include some amount of pretrain data to prevent catastrophic forgetting <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib87" title="" class="ltx_ref">87</a>]</cite>.
Similarly, if some explicit memories have already been produced, then they need to be rebuilt in order to remain readable by the updated model.
It is an interesting research direction to design a more efficient architecture such that the implicit and explicit memories are robust with respect to model updates.</p>
</div>
</div>
</section>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Vector Compression</h2>

<div id="A2.p1" class="ltx_para">
<p id="A2.p1.1" class="ltx_p">Regarding the vector quantizer discussed in Sections <a href="#S3.SS3" title="3.3 | Memory Sparsification and Storage ‣ 3 | Design ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3</span></a> and <a href="#S7.SS1" title="7.1 | General Abilities ‣ 7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7.1</span></a>, we use the composite index of FAISS with index type OPQ20x80-Residual2x14-PQ8x10.
It can encode a 80-dimensional bfloat16 vector into a 14-dimensional uint8 vector, and thus its compression rate is <math id="A2.p1.1.m1.1" class="ltx_Math" alttext="\frac{80\times 2}{14\times 1}\approx 11.4" display="inline"><semantics id="A2.p1.1.m1.1a"><mrow id="A2.p1.1.m1.1.1" xref="A2.p1.1.m1.1.1.cmml"><mfrac id="A2.p1.1.m1.1.1.2" xref="A2.p1.1.m1.1.1.2.cmml"><mrow id="A2.p1.1.m1.1.1.2.2" xref="A2.p1.1.m1.1.1.2.2.cmml"><mn id="A2.p1.1.m1.1.1.2.2.2" xref="A2.p1.1.m1.1.1.2.2.2.cmml">80</mn><mo lspace="0.222em" rspace="0.222em" id="A2.p1.1.m1.1.1.2.2.1" xref="A2.p1.1.m1.1.1.2.2.1.cmml">×</mo><mn id="A2.p1.1.m1.1.1.2.2.3" xref="A2.p1.1.m1.1.1.2.2.3.cmml">2</mn></mrow><mrow id="A2.p1.1.m1.1.1.2.3" xref="A2.p1.1.m1.1.1.2.3.cmml"><mn id="A2.p1.1.m1.1.1.2.3.2" xref="A2.p1.1.m1.1.1.2.3.2.cmml">14</mn><mo lspace="0.222em" rspace="0.222em" id="A2.p1.1.m1.1.1.2.3.1" xref="A2.p1.1.m1.1.1.2.3.1.cmml">×</mo><mn id="A2.p1.1.m1.1.1.2.3.3" xref="A2.p1.1.m1.1.1.2.3.3.cmml">1</mn></mrow></mfrac><mo id="A2.p1.1.m1.1.1.1" xref="A2.p1.1.m1.1.1.1.cmml">≈</mo><mn id="A2.p1.1.m1.1.1.3" xref="A2.p1.1.m1.1.1.3.cmml">11.4</mn></mrow><annotation-xml encoding="MathML-Content" id="A2.p1.1.m1.1b"><apply id="A2.p1.1.m1.1.1.cmml" xref="A2.p1.1.m1.1.1"><approx id="A2.p1.1.m1.1.1.1.cmml" xref="A2.p1.1.m1.1.1.1"></approx><apply id="A2.p1.1.m1.1.1.2.cmml" xref="A2.p1.1.m1.1.1.2"><divide id="A2.p1.1.m1.1.1.2.1.cmml" xref="A2.p1.1.m1.1.1.2"></divide><apply id="A2.p1.1.m1.1.1.2.2.cmml" xref="A2.p1.1.m1.1.1.2.2"><times id="A2.p1.1.m1.1.1.2.2.1.cmml" xref="A2.p1.1.m1.1.1.2.2.1"></times><cn type="integer" id="A2.p1.1.m1.1.1.2.2.2.cmml" xref="A2.p1.1.m1.1.1.2.2.2">80</cn><cn type="integer" id="A2.p1.1.m1.1.1.2.2.3.cmml" xref="A2.p1.1.m1.1.1.2.2.3">2</cn></apply><apply id="A2.p1.1.m1.1.1.2.3.cmml" xref="A2.p1.1.m1.1.1.2.3"><times id="A2.p1.1.m1.1.1.2.3.1.cmml" xref="A2.p1.1.m1.1.1.2.3.1"></times><cn type="integer" id="A2.p1.1.m1.1.1.2.3.2.cmml" xref="A2.p1.1.m1.1.1.2.3.2">14</cn><cn type="integer" id="A2.p1.1.m1.1.1.2.3.3.cmml" xref="A2.p1.1.m1.1.1.2.3.3">1</cn></apply></apply><cn type="float" id="A2.p1.1.m1.1.1.3.cmml" xref="A2.p1.1.m1.1.1.3">11.4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.1.m1.1c">\frac{80\times 2}{14\times 1}\approx 11.4</annotation></semantics></math>.</p>
</div>
<div id="A2.p2" class="ltx_para">
<p id="A2.p2.1" class="ltx_p">To train this quantizer, we sample references from our knowledge base, encode them into explicit memories by our Memory<sup id="A2.p2.1.1" class="ltx_sup">3</sup>-2B-SFT model, and feed these key-value vectors to the quantizer.
The references are sampled uniformly and independently, so the training is not biased towards the references that are retrieved by any specific evaluation task.</p>
</div>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Supplementary Evaluation Results</h2>

<div id="A3.p1" class="ltx_para">
<p id="A3.p1.1" class="ltx_p">First, Table <a href="#T22" title="Table 22 ‣ Appendix C Supplementary Evaluation Results ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">22</span></a> records the growth of the test scores (Table <a href="#T16" title="Table 16 ‣ 7.1 | General Abilities ‣ 7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">16</span></a>) over the three training stages: warmup, continual train, and SFT.
We believe that for future versions of Memory<sup id="A3.p1.1.1" class="ltx_sup">3</sup>, fixing the loss divergence during the warmup stage can allow the continual train stage to proceed much further (cf. Section <a href="#S5.SS3" title="5.3 | Continual Train Stage ‣ 5 | Pretrain ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5.3</span></a>), and thus increase the performance boost of this stage.</p>
</div>
<figure id="T22" class="ltx_table">
<table id="T22.4" class="ltx_tabular ltx_align_middle">
<tr id="T22.4.1" class="ltx_tr">
<td id="T22.4.1.1" class="ltx_td ltx_border_tt"></td>
<td id="T22.4.1.2" class="ltx_td ltx_border_tt"></td>
<td id="T22.4.1.3" class="ltx_td ltx_align_center ltx_border_tt" colspan="5"><span id="T22.4.1.3.1" class="ltx_text" style="font-size:80%;">English</span></td>
<td id="T22.4.1.4" class="ltx_td ltx_align_center ltx_border_tt" colspan="2"><span id="T22.4.1.4.1" class="ltx_text" style="font-size:80%;">Chinese</span></td>
</tr>
<tr id="T22.4.2" class="ltx_tr" style="background-color:#FFFFFF;">
<td id="T22.4.2.1" class="ltx_td ltx_align_center"><span id="T22.4.2.1.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">LLM</span></td>
<td id="T22.4.2.2" class="ltx_td ltx_align_center"><span id="T22.4.2.2.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">Avg.</span></td>
<td id="T22.4.2.3" class="ltx_td ltx_align_center ltx_border_t"><span id="T22.4.2.3.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">ARC-C</span></td>
<td id="T22.4.2.4" class="ltx_td ltx_align_center ltx_border_t"><span id="T22.4.2.4.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">HellaSwag</span></td>
<td id="T22.4.2.5" class="ltx_td ltx_align_center ltx_border_t"><span id="T22.4.2.5.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">MMLU</span></td>
<td id="T22.4.2.6" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="T22.4.2.6.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFFFFF;">
<span id="T22.4.2.6.1.1" class="ltx_p" style="width:32.5pt;"><span id="T22.4.2.6.1.1.1" class="ltx_text" style="font-size:80%;">Winogrande</span></span>
</span>
</td>
<td id="T22.4.2.7" class="ltx_td ltx_align_center ltx_border_t"><span id="T22.4.2.7.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">GSM8k</span></td>
<td id="T22.4.2.8" class="ltx_td ltx_align_center ltx_border_t"><span id="T22.4.2.8.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">CEVAL</span></td>
<td id="T22.4.2.9" class="ltx_td ltx_align_center ltx_border_t"><span id="T22.4.2.9.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">CMMLU</span></td>
</tr>
<tr id="T22.4.3" class="ltx_tr" style="background-color:#E6F3F3;">
<td id="T22.4.3.1" class="ltx_td ltx_align_center ltx_border_t"><span id="T22.4.3.1.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">Warmup</span></td>
<td id="T22.4.3.2" class="ltx_td ltx_align_center ltx_border_t"><span id="T22.4.3.2.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">42.13</span></td>
<td id="T22.4.3.3" class="ltx_td ltx_align_center ltx_border_t"><span id="T22.4.3.3.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">40.27</span></td>
<td id="T22.4.3.4" class="ltx_td ltx_align_center ltx_border_t"><span id="T22.4.3.4.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">64.57</span></td>
<td id="T22.4.3.5" class="ltx_td ltx_align_center ltx_border_t"><span id="T22.4.3.5.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">41.62</span></td>
<td id="T22.4.3.6" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="T22.4.3.6.1" class="ltx_inline-block ltx_align_top" style="background-color:#E6F3F3;">
<span id="T22.4.3.6.1.1" class="ltx_p" style="width:32.5pt;"><span id="T22.4.3.6.1.1.1" class="ltx_text" style="font-size:80%;">61.96</span></span>
</span>
</td>
<td id="T22.4.3.7" class="ltx_td ltx_align_center ltx_border_t"><span id="T22.4.3.7.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">5.23</span></td>
<td id="T22.4.3.8" class="ltx_td ltx_align_center ltx_border_t"><span id="T22.4.3.8.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">40.12</span></td>
<td id="T22.4.3.9" class="ltx_td ltx_align_center ltx_border_t"><span id="T22.4.3.9.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">41.17</span></td>
</tr>
<tr id="T22.4.4" class="ltx_tr" style="background-color:#FFFFFF;">
<td id="T22.4.4.1" class="ltx_td ltx_align_center"><span id="T22.4.4.1.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">Continual train</span></td>
<td id="T22.4.4.2" class="ltx_td ltx_align_center"><span id="T22.4.4.2.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">45.12</span></td>
<td id="T22.4.4.3" class="ltx_td ltx_align_center"><span id="T22.4.4.3.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">42.66</span></td>
<td id="T22.4.4.4" class="ltx_td ltx_align_center"><span id="T22.4.4.4.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">79.21</span></td>
<td id="T22.4.4.5" class="ltx_td ltx_align_center"><span id="T22.4.4.5.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">41.81</span></td>
<td id="T22.4.4.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="T22.4.4.6.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFFFFF;">
<span id="T22.4.4.6.1.1" class="ltx_p" style="width:32.5pt;"><span id="T22.4.4.6.1.1.1" class="ltx_text" style="font-size:80%;">59.43</span></span>
</span>
</td>
<td id="T22.4.4.7" class="ltx_td ltx_align_center"><span id="T22.4.4.7.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">6.29</span></td>
<td id="T22.4.4.8" class="ltx_td ltx_align_center"><span id="T22.4.4.8.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">42.20</span></td>
<td id="T22.4.4.9" class="ltx_td ltx_align_center"><span id="T22.4.4.9.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">44.21</span></td>
</tr>
<tr id="T22.4.5" class="ltx_tr" style="background-color:#E6F3F3;">
<td id="T22.4.5.1" class="ltx_td ltx_align_center"><span id="T22.4.5.1.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">- without memory</span></td>
<td id="T22.4.5.2" class="ltx_td ltx_align_center"><span id="T22.4.5.2.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">42.89</span></td>
<td id="T22.4.5.3" class="ltx_td ltx_align_center"><span id="T22.4.5.3.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">42.15</span></td>
<td id="T22.4.5.4" class="ltx_td ltx_align_center"><span id="T22.4.5.4.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">66.98</span></td>
<td id="T22.4.5.5" class="ltx_td ltx_align_center"><span id="T22.4.5.5.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">39.79</span></td>
<td id="T22.4.5.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="T22.4.5.6.1" class="ltx_inline-block ltx_align_top" style="background-color:#E6F3F3;">
<span id="T22.4.5.6.1.1" class="ltx_p" style="width:32.5pt;"><span id="T22.4.5.6.1.1.1" class="ltx_text" style="font-size:80%;">61.80</span></span>
</span>
</td>
<td id="T22.4.5.7" class="ltx_td ltx_align_center"><span id="T22.4.5.7.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">6.44</span></td>
<td id="T22.4.5.8" class="ltx_td ltx_align_center"><span id="T22.4.5.8.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">39.97</span></td>
<td id="T22.4.5.9" class="ltx_td ltx_align_center"><span id="T22.4.5.9.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">43.13</span></td>
</tr>
<tr id="T22.4.6" class="ltx_tr" style="background-color:#FFFFFF;">
<td id="T22.4.6.1" class="ltx_td ltx_align_center"><span id="T22.4.6.1.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">SFT</span></td>
<td id="T22.4.6.2" class="ltx_td ltx_align_center"><span id="T22.4.6.2.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">63.31</span></td>
<td id="T22.4.6.3" class="ltx_td ltx_align_center"><span id="T22.4.6.3.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">58.11</span></td>
<td id="T22.4.6.4" class="ltx_td ltx_align_center"><span id="T22.4.6.4.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">80.51</span></td>
<td id="T22.4.6.5" class="ltx_td ltx_align_center"><span id="T22.4.6.5.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">59.68</span></td>
<td id="T22.4.6.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="T22.4.6.6.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFFFFF;">
<span id="T22.4.6.6.1.1" class="ltx_p" style="width:32.5pt;"><span id="T22.4.6.6.1.1.1" class="ltx_text" style="font-size:80%;">74.51</span></span>
</span>
</td>
<td id="T22.4.6.7" class="ltx_td ltx_align_center"><span id="T22.4.6.7.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">52.84</span></td>
<td id="T22.4.6.8" class="ltx_td ltx_align_center"><span id="T22.4.6.8.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">59.29</span></td>
<td id="T22.4.6.9" class="ltx_td ltx_align_center"><span id="T22.4.6.9.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">58.24</span></td>
</tr>
<tr id="T22.4.7" class="ltx_tr" style="background-color:#E6F3F3;">
<td id="T22.4.7.1" class="ltx_td ltx_align_center ltx_border_bb"><span id="T22.4.7.1.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">- without memory</span></td>
<td id="T22.4.7.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="T22.4.7.2.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">60.80</span></td>
<td id="T22.4.7.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="T22.4.7.3.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">57.42</span></td>
<td id="T22.4.7.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="T22.4.7.4.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">73.14</span></td>
<td id="T22.4.7.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="T22.4.7.5.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">57.29</span></td>
<td id="T22.4.7.6" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="T22.4.7.6.1" class="ltx_inline-block ltx_align_top" style="background-color:#E6F3F3;">
<span id="T22.4.7.6.1.1" class="ltx_p" style="width:32.5pt;"><span id="T22.4.7.6.1.1.1" class="ltx_text" style="font-size:80%;">74.35</span></span>
</span>
</td>
<td id="T22.4.7.7" class="ltx_td ltx_align_center ltx_border_bb"><span id="T22.4.7.7.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">51.33</span></td>
<td id="T22.4.7.8" class="ltx_td ltx_align_center ltx_border_bb"><span id="T22.4.7.8.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">56.32</span></td>
<td id="T22.4.7.9" class="ltx_td ltx_align_center ltx_border_bb"><span id="T22.4.7.9.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">55.72</span></td>
</tr>
</table>
<figcaption class="ltx_caption" style="font-size:80%;"><span class="ltx_tag ltx_tag_table"><span id="T22.10.2.1" class="ltx_text" style="font-size:113%;">Table 22</span>: </span><span id="T22.2.1" class="ltx_text" style="font-size:113%;">Performance of Memory<sup id="T22.2.1.1" class="ltx_sup">3</sup>-2B at different stages of training.
The setup of the evaluation tasks is the same as in Table <a href="#T16" title="Table 16 ‣ 7.1 | General Abilities ‣ 7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">16</span></a>.</span></figcaption>
</figure>
<div id="A3.p2" class="ltx_para">
<p id="A3.p2.2" class="ltx_p">Next, recall that for the evaluations in Section <a href="#S7.SS1" title="7.1 | General Abilities ‣ 7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7.1</span></a>, a filter is included in the retrieval process to prevent copying, which removes references that overlap too much with the evaluation question.
The filtering threshold should lie between <math id="A3.p2.1.m1.1" class="ltx_Math" alttext="100\%" display="inline"><semantics id="A3.p2.1.m1.1a"><mrow id="A3.p2.1.m1.1.1" xref="A3.p2.1.m1.1.1.cmml"><mn id="A3.p2.1.m1.1.1.2" xref="A3.p2.1.m1.1.1.2.cmml">100</mn><mo id="A3.p2.1.m1.1.1.1" xref="A3.p2.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="A3.p2.1.m1.1b"><apply id="A3.p2.1.m1.1.1.cmml" xref="A3.p2.1.m1.1.1"><csymbol cd="latexml" id="A3.p2.1.m1.1.1.1.cmml" xref="A3.p2.1.m1.1.1.1">percent</csymbol><cn type="integer" id="A3.p2.1.m1.1.1.2.cmml" xref="A3.p2.1.m1.1.1.2">100</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p2.1.m1.1c">100\%</annotation></semantics></math> and the usual level of overlap between two related but distinct texts, and we set it to <math id="A3.p2.2.m2.1" class="ltx_Math" alttext="2/3" display="inline"><semantics id="A3.p2.2.m2.1a"><mrow id="A3.p2.2.m2.1.1" xref="A3.p2.2.m2.1.1.cmml"><mn id="A3.p2.2.m2.1.1.2" xref="A3.p2.2.m2.1.1.2.cmml">2</mn><mo id="A3.p2.2.m2.1.1.1" xref="A3.p2.2.m2.1.1.1.cmml">/</mo><mn id="A3.p2.2.m2.1.1.3" xref="A3.p2.2.m2.1.1.3.cmml">3</mn></mrow><annotation-xml encoding="MathML-Content" id="A3.p2.2.m2.1b"><apply id="A3.p2.2.m2.1.1.cmml" xref="A3.p2.2.m2.1.1"><divide id="A3.p2.2.m2.1.1.1.cmml" xref="A3.p2.2.m2.1.1.1"></divide><cn type="integer" id="A3.p2.2.m2.1.1.2.cmml" xref="A3.p2.2.m2.1.1.2">2</cn><cn type="integer" id="A3.p2.2.m2.1.1.3.cmml" xref="A3.p2.2.m2.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p2.2.m2.1c">2/3</annotation></semantics></math> in Table <a href="#T16" title="Table 16 ‣ 7.1 | General Abilities ‣ 7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">16</span></a>.
Table <a href="#T23" title="Table 23 ‣ Appendix C Supplementary Evaluation Results ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">23</span></a> records the impact of the filtering threshold on the test scores.
The scores are stable for most tasks, indicating that their questions do not appear in our knowledge basis.</p>
</div>
<figure id="T23" class="ltx_table">
<table id="T23.2" class="ltx_tabular ltx_align_middle">
<tr id="T23.2.1" class="ltx_tr">
<td id="T23.2.1.1" class="ltx_td ltx_align_center ltx_border_tt"><span id="T23.2.1.1.1" class="ltx_text" style="font-size:80%;">Threshold</span></td>
<td id="T23.2.1.2" class="ltx_td ltx_align_center ltx_border_tt"><span id="T23.2.1.2.1" class="ltx_text" style="font-size:80%;">Avg.</span></td>
<td id="T23.2.1.3" class="ltx_td ltx_align_center ltx_border_tt"><span id="T23.2.1.3.1" class="ltx_text" style="font-size:80%;">ARC-C</span></td>
<td id="T23.2.1.4" class="ltx_td ltx_align_center ltx_border_tt"><span id="T23.2.1.4.1" class="ltx_text" style="font-size:80%;">HellaSwag</span></td>
<td id="T23.2.1.5" class="ltx_td ltx_align_center ltx_border_tt"><span id="T23.2.1.5.1" class="ltx_text" style="font-size:80%;">MMLU</span></td>
<td id="T23.2.1.6" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="T23.2.1.6.1" class="ltx_inline-block ltx_align_top">
<span id="T23.2.1.6.1.1" class="ltx_p" style="width:32.5pt;"><span id="T23.2.1.6.1.1.1" class="ltx_text" style="font-size:80%;">Winogrande</span></span>
</span>
</td>
<td id="T23.2.1.7" class="ltx_td ltx_align_center ltx_border_tt"><span id="T23.2.1.7.1" class="ltx_text" style="font-size:80%;">GSM8k</span></td>
<td id="T23.2.1.8" class="ltx_td ltx_align_center ltx_border_tt"><span id="T23.2.1.8.1" class="ltx_text" style="font-size:80%;">CEVAL</span></td>
<td id="T23.2.1.9" class="ltx_td ltx_align_center ltx_border_tt"><span id="T23.2.1.9.1" class="ltx_text" style="font-size:80%;">CMMLU</span></td>
</tr>
<tr id="T23.2.2" class="ltx_tr" style="background-color:#FFFFFF;">
<td id="T23.2.2.1" class="ltx_td ltx_align_center ltx_border_t"><span id="T23.2.2.1.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">no filter</span></td>
<td id="T23.2.2.2" class="ltx_td ltx_align_center ltx_border_t"><span id="T23.2.2.2.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">63.71</span></td>
<td id="T23.2.2.3" class="ltx_td ltx_align_center ltx_border_t"><span id="T23.2.2.3.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">58.11</span></td>
<td id="T23.2.2.4" class="ltx_td ltx_align_center ltx_border_t"><span id="T23.2.2.4.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">83.37</span></td>
<td id="T23.2.2.5" class="ltx_td ltx_align_center ltx_border_t"><span id="T23.2.2.5.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">59.65</span></td>
<td id="T23.2.2.6" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="T23.2.2.6.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFFFFF;">
<span id="T23.2.2.6.1.1" class="ltx_p" style="width:32.5pt;"><span id="T23.2.2.6.1.1.1" class="ltx_text" style="font-size:80%;">74.51</span></span>
</span>
</td>
<td id="T23.2.2.7" class="ltx_td ltx_align_center ltx_border_t"><span id="T23.2.2.7.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">52.84</span></td>
<td id="T23.2.2.8" class="ltx_td ltx_align_center ltx_border_t"><span id="T23.2.2.8.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">59.29</span></td>
<td id="T23.2.2.9" class="ltx_td ltx_align_center ltx_border_t"><span id="T23.2.2.9.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">58.22</span></td>
</tr>
<tr id="T23.2.3" class="ltx_tr" style="background-color:#E6F3F3;">
<td id="T23.2.3.1" class="ltx_td ltx_align_center"><span id="T23.2.3.1.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">80%</span></td>
<td id="T23.2.3.2" class="ltx_td ltx_align_center"><span id="T23.2.3.2.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">63.62</span></td>
<td id="T23.2.3.3" class="ltx_td ltx_align_center"><span id="T23.2.3.3.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">58.11</span></td>
<td id="T23.2.3.4" class="ltx_td ltx_align_center"><span id="T23.2.3.4.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">82.69</span></td>
<td id="T23.2.3.5" class="ltx_td ltx_align_center"><span id="T23.2.3.5.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">59.65</span></td>
<td id="T23.2.3.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="T23.2.3.6.1" class="ltx_inline-block ltx_align_top" style="background-color:#E6F3F3;">
<span id="T23.2.3.6.1.1" class="ltx_p" style="width:32.5pt;"><span id="T23.2.3.6.1.1.1" class="ltx_text" style="font-size:80%;">74.51</span></span>
</span>
</td>
<td id="T23.2.3.7" class="ltx_td ltx_align_center"><span id="T23.2.3.7.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">52.84</span></td>
<td id="T23.2.3.8" class="ltx_td ltx_align_center"><span id="T23.2.3.8.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">59.29</span></td>
<td id="T23.2.3.9" class="ltx_td ltx_align_center"><span id="T23.2.3.9.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">58.24</span></td>
</tr>
<tr id="T23.2.4" class="ltx_tr" style="background-color:#FFFFFF;">
<td id="T23.2.4.1" class="ltx_td ltx_align_center"><span id="T23.2.4.1.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">2/3</span></td>
<td id="T23.2.4.2" class="ltx_td ltx_align_center"><span id="T23.2.4.2.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">63.31</span></td>
<td id="T23.2.4.3" class="ltx_td ltx_align_center"><span id="T23.2.4.3.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">58.11</span></td>
<td id="T23.2.4.4" class="ltx_td ltx_align_center"><span id="T23.2.4.4.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">80.51</span></td>
<td id="T23.2.4.5" class="ltx_td ltx_align_center"><span id="T23.2.4.5.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">59.68</span></td>
<td id="T23.2.4.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="T23.2.4.6.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFFFFF;">
<span id="T23.2.4.6.1.1" class="ltx_p" style="width:32.5pt;"><span id="T23.2.4.6.1.1.1" class="ltx_text" style="font-size:80%;">74.51</span></span>
</span>
</td>
<td id="T23.2.4.7" class="ltx_td ltx_align_center"><span id="T23.2.4.7.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">52.84</span></td>
<td id="T23.2.4.8" class="ltx_td ltx_align_center"><span id="T23.2.4.8.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">59.29</span></td>
<td id="T23.2.4.9" class="ltx_td ltx_align_center"><span id="T23.2.4.9.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">58.24</span></td>
</tr>
<tr id="T23.2.5" class="ltx_tr" style="background-color:#E6F3F3;">
<td id="T23.2.5.1" class="ltx_td ltx_align_center ltx_border_bb"><span id="T23.2.5.1.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">without memory</span></td>
<td id="T23.2.5.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="T23.2.5.2.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">60.80</span></td>
<td id="T23.2.5.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="T23.2.5.3.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">57.42</span></td>
<td id="T23.2.5.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="T23.2.5.4.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">73.14</span></td>
<td id="T23.2.5.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="T23.2.5.5.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">57.29</span></td>
<td id="T23.2.5.6" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="T23.2.5.6.1" class="ltx_inline-block ltx_align_top" style="background-color:#E6F3F3;">
<span id="T23.2.5.6.1.1" class="ltx_p" style="width:32.5pt;"><span id="T23.2.5.6.1.1.1" class="ltx_text" style="font-size:80%;">74.35</span></span>
</span>
</td>
<td id="T23.2.5.7" class="ltx_td ltx_align_center ltx_border_bb"><span id="T23.2.5.7.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">51.33</span></td>
<td id="T23.2.5.8" class="ltx_td ltx_align_center ltx_border_bb"><span id="T23.2.5.8.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">56.32</span></td>
<td id="T23.2.5.9" class="ltx_td ltx_align_center ltx_border_bb"><span id="T23.2.5.9.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">55.72</span></td>
</tr>
</table>
<figcaption class="ltx_caption" style="font-size:80%;"><span class="ltx_tag ltx_tag_table"><span id="T23.6.1.1" class="ltx_text" style="font-size:113%;">Table 23</span>: </span><span id="T23.7.2" class="ltx_text" style="font-size:113%;">Influence of the filtering threshold on the test scores in Table <a href="#T16" title="Table 16 ‣ 7.1 | General Abilities ‣ 7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">16</span></a>.</span></figcaption>
</figure>
<div id="A3.p3" class="ltx_para">
<p id="A3.p3.2" class="ltx_p">Finally, Table <a href="#T24" title="Table 24 ‣ Appendix C Supplementary Evaluation Results ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">24</span></a> studies the influence of the few-shot prompts on the benchmark tasks.
Recall that the number of few-shot examples for each task is ARC-C (25), HellaSwag (10), MMLU (5), Winogrande (5), GSM8k (5) as in HuggingFace OpenLLM Leaderboard <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib13" title="" class="ltx_ref">13</a>]</cite>, and we also adopt CEVAL (5), CMMLU (5).
Interestingly, the boost from explicit memory increases from <math id="A3.p3.1.m1.1" class="ltx_Math" alttext="2.51\%" display="inline"><semantics id="A3.p3.1.m1.1a"><mrow id="A3.p3.1.m1.1.1" xref="A3.p3.1.m1.1.1.cmml"><mn id="A3.p3.1.m1.1.1.2" xref="A3.p3.1.m1.1.1.2.cmml">2.51</mn><mo id="A3.p3.1.m1.1.1.1" xref="A3.p3.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="A3.p3.1.m1.1b"><apply id="A3.p3.1.m1.1.1.cmml" xref="A3.p3.1.m1.1.1"><csymbol cd="latexml" id="A3.p3.1.m1.1.1.1.cmml" xref="A3.p3.1.m1.1.1.1">percent</csymbol><cn type="float" id="A3.p3.1.m1.1.1.2.cmml" xref="A3.p3.1.m1.1.1.2">2.51</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p3.1.m1.1c">2.51\%</annotation></semantics></math> to <math id="A3.p3.2.m2.1" class="ltx_Math" alttext="3.70\%" display="inline"><semantics id="A3.p3.2.m2.1a"><mrow id="A3.p3.2.m2.1.1" xref="A3.p3.2.m2.1.1.cmml"><mn id="A3.p3.2.m2.1.1.2" xref="A3.p3.2.m2.1.1.2.cmml">3.70</mn><mo id="A3.p3.2.m2.1.1.1" xref="A3.p3.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="A3.p3.2.m2.1b"><apply id="A3.p3.2.m2.1.1.cmml" xref="A3.p3.2.m2.1.1"><csymbol cd="latexml" id="A3.p3.2.m2.1.1.1.cmml" xref="A3.p3.2.m2.1.1.1">percent</csymbol><cn type="float" id="A3.p3.2.m2.1.1.2.cmml" xref="A3.p3.2.m2.1.1.2">3.70</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p3.2.m2.1c">3.70\%</annotation></semantics></math> as we switch to 0-shot.</p>
</div>
<figure id="T24" class="ltx_table">
<table id="T24.2" class="ltx_tabular ltx_align_middle">
<tr id="T24.2.1" class="ltx_tr">
<td id="T24.2.1.1" class="ltx_td ltx_align_center ltx_border_tt"><span id="T24.2.1.1.1" class="ltx_text" style="font-size:80%;">Mode</span></td>
<td id="T24.2.1.2" class="ltx_td ltx_align_center ltx_border_tt"><span id="T24.2.1.2.1" class="ltx_text" style="font-size:80%;">Avg.</span></td>
<td id="T24.2.1.3" class="ltx_td ltx_align_center ltx_border_tt"><span id="T24.2.1.3.1" class="ltx_text" style="font-size:80%;">ARC-C</span></td>
<td id="T24.2.1.4" class="ltx_td ltx_align_center ltx_border_tt"><span id="T24.2.1.4.1" class="ltx_text" style="font-size:80%;">HellaSwag</span></td>
<td id="T24.2.1.5" class="ltx_td ltx_align_center ltx_border_tt"><span id="T24.2.1.5.1" class="ltx_text" style="font-size:80%;">MMLU</span></td>
<td id="T24.2.1.6" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="T24.2.1.6.1" class="ltx_inline-block ltx_align_top">
<span id="T24.2.1.6.1.1" class="ltx_p" style="width:32.5pt;"><span id="T24.2.1.6.1.1.1" class="ltx_text" style="font-size:80%;">Winogrande</span></span>
</span>
</td>
<td id="T24.2.1.7" class="ltx_td ltx_align_center ltx_border_tt"><span id="T24.2.1.7.1" class="ltx_text" style="font-size:80%;">GSM8k</span></td>
<td id="T24.2.1.8" class="ltx_td ltx_align_center ltx_border_tt"><span id="T24.2.1.8.1" class="ltx_text" style="font-size:80%;">CEVAL</span></td>
<td id="T24.2.1.9" class="ltx_td ltx_align_center ltx_border_tt"><span id="T24.2.1.9.1" class="ltx_text" style="font-size:80%;">CMMLU</span></td>
</tr>
<tr id="T24.2.2" class="ltx_tr" style="background-color:#FFFFFF;">
<td id="T24.2.2.1" class="ltx_td ltx_align_center ltx_border_t"><span id="T24.2.2.1.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">Few-shot</span></td>
<td id="T24.2.2.2" class="ltx_td ltx_align_center ltx_border_t"><span id="T24.2.2.2.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">63.31</span></td>
<td id="T24.2.2.3" class="ltx_td ltx_align_center ltx_border_t"><span id="T24.2.2.3.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">58.11</span></td>
<td id="T24.2.2.4" class="ltx_td ltx_align_center ltx_border_t"><span id="T24.2.2.4.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">80.51</span></td>
<td id="T24.2.2.5" class="ltx_td ltx_align_center ltx_border_t"><span id="T24.2.2.5.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">59.68</span></td>
<td id="T24.2.2.6" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="T24.2.2.6.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFFFFF;">
<span id="T24.2.2.6.1.1" class="ltx_p" style="width:32.5pt;"><span id="T24.2.2.6.1.1.1" class="ltx_text" style="font-size:80%;">74.51</span></span>
</span>
</td>
<td id="T24.2.2.7" class="ltx_td ltx_align_center ltx_border_t"><span id="T24.2.2.7.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">52.84</span></td>
<td id="T24.2.2.8" class="ltx_td ltx_align_center ltx_border_t"><span id="T24.2.2.8.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">59.29</span></td>
<td id="T24.2.2.9" class="ltx_td ltx_align_center ltx_border_t"><span id="T24.2.2.9.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">58.24</span></td>
</tr>
<tr id="T24.2.3" class="ltx_tr" style="background-color:#E6F3F3;">
<td id="T24.2.3.1" class="ltx_td ltx_align_center"><span id="T24.2.3.1.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">- without memory</span></td>
<td id="T24.2.3.2" class="ltx_td ltx_align_center"><span id="T24.2.3.2.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">60.80</span></td>
<td id="T24.2.3.3" class="ltx_td ltx_align_center"><span id="T24.2.3.3.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">57.42</span></td>
<td id="T24.2.3.4" class="ltx_td ltx_align_center"><span id="T24.2.3.4.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">73.14</span></td>
<td id="T24.2.3.5" class="ltx_td ltx_align_center"><span id="T24.2.3.5.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">57.29</span></td>
<td id="T24.2.3.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="T24.2.3.6.1" class="ltx_inline-block ltx_align_top" style="background-color:#E6F3F3;">
<span id="T24.2.3.6.1.1" class="ltx_p" style="width:32.5pt;"><span id="T24.2.3.6.1.1.1" class="ltx_text" style="font-size:80%;">74.35</span></span>
</span>
</td>
<td id="T24.2.3.7" class="ltx_td ltx_align_center"><span id="T24.2.3.7.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">51.33</span></td>
<td id="T24.2.3.8" class="ltx_td ltx_align_center"><span id="T24.2.3.8.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">56.32</span></td>
<td id="T24.2.3.9" class="ltx_td ltx_align_center"><span id="T24.2.3.9.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">55.72</span></td>
</tr>
<tr id="T24.2.4" class="ltx_tr" style="background-color:#FFFFFF;">
<td id="T24.2.4.1" class="ltx_td ltx_align_center"><span id="T24.2.4.1.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">0-shot</span></td>
<td id="T24.2.4.2" class="ltx_td ltx_align_center"><span id="T24.2.4.2.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">58.23</span></td>
<td id="T24.2.4.3" class="ltx_td ltx_align_center"><span id="T24.2.4.3.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">58.79</span></td>
<td id="T24.2.4.4" class="ltx_td ltx_align_center"><span id="T24.2.4.4.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">83.29</span></td>
<td id="T24.2.4.5" class="ltx_td ltx_align_center"><span id="T24.2.4.5.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">60.53</span></td>
<td id="T24.2.4.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="T24.2.4.6.1" class="ltx_inline-block ltx_align_top" style="background-color:#FFFFFF;">
<span id="T24.2.4.6.1.1" class="ltx_p" style="width:32.5pt;"><span id="T24.2.4.6.1.1.1" class="ltx_text" style="font-size:80%;">75.85</span></span>
</span>
</td>
<td id="T24.2.4.7" class="ltx_td ltx_align_center"><span id="T24.2.4.7.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">13.50</span></td>
<td id="T24.2.4.8" class="ltx_td ltx_align_center"><span id="T24.2.4.8.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">57.95</span></td>
<td id="T24.2.4.9" class="ltx_td ltx_align_center"><span id="T24.2.4.9.1" class="ltx_text" style="font-size:80%;background-color:#FFFFFF;">57.74</span></td>
</tr>
<tr id="T24.2.5" class="ltx_tr" style="background-color:#E6F3F3;">
<td id="T24.2.5.1" class="ltx_td ltx_align_center ltx_border_bb"><span id="T24.2.5.1.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">- without memory</span></td>
<td id="T24.2.5.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="T24.2.5.2.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">54.54</span></td>
<td id="T24.2.5.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="T24.2.5.3.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">57.34</span></td>
<td id="T24.2.5.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="T24.2.5.4.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">73.15</span></td>
<td id="T24.2.5.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="T24.2.5.5.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">58.59</span></td>
<td id="T24.2.5.6" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="T24.2.5.6.1" class="ltx_inline-block ltx_align_top" style="background-color:#E6F3F3;">
<span id="T24.2.5.6.1.1" class="ltx_p" style="width:32.5pt;"><span id="T24.2.5.6.1.1.1" class="ltx_text" style="font-size:80%;">74.98</span></span>
</span>
</td>
<td id="T24.2.5.7" class="ltx_td ltx_align_center ltx_border_bb"><span id="T24.2.5.7.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">10.46</span></td>
<td id="T24.2.5.8" class="ltx_td ltx_align_center ltx_border_bb"><span id="T24.2.5.8.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">54.53</span></td>
<td id="T24.2.5.9" class="ltx_td ltx_align_center ltx_border_bb"><span id="T24.2.5.9.1" class="ltx_text" style="font-size:80%;background-color:#E6F3F3;">54.26</span></td>
</tr>
</table>
<figcaption class="ltx_caption" style="font-size:80%;"><span class="ltx_tag ltx_tag_table"><span id="T24.6.1.1" class="ltx_text" style="font-size:113%;">Table 24</span>: </span><span id="T24.7.2" class="ltx_text" style="font-size:113%;">Few-shot versus 0-shot for the benchmark tasks in Table <a href="#T16" title="Table 16 ‣ 7.1 | General Abilities ‣ 7 | Evaluation ‣ Memory3: Language Modeling with Explicit Memory" class="ltx_ref"><span class="ltx_text ltx_ref_tag">16</span></a>.</span></figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2407.01177" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2407.01178" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2407.01178">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2407.01178" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2407.01179" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Mon Aug  5 17:12:28 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2407.01178
The training and inference of large language models (LLMs)  are together a costly process that transports knowledge from raw data to meaningful computation.
Inspired by the memory hierarchy of the human brain, we reduce this cost by equipping LLMs with explicit memory, a memory format cheaper than model parameters and text retrieval-augmented generation (RAG) .
Conceptually, with most of its knowledge externalized to explicit memories, the LLM can enjoy a smaller parameter size, training cost, and inference cost, all proportional to the amount of remaining “abstract knowledge”.
As a preliminary proof of concept, we train from scratch a 2.4B LLM, which achieves better performance than much larger LLMs as well as RAG models, and maintains higher decoding speed than RAG.
The model is named Memory3, since explicit memory is the third form of memory in LLMs after implicit memory (model parameters)  and working memory (context key-values) .
We introduce a memory circuitry theory to support the externalization of knowledge, and present novel techniques including a memory sparsification mechanism that makes storage tractable and a two-stage pretraining scheme that facilitates memory formation.
abstract
Memory3: Language Modeling with Explicit Memory
Abstract
2407.01178
1|IntroductionLarge language models (LLMs)  have enjoyed unprecedented popularity in recent years thanks to their extraordinary performance.
The prospect of scaling lawsand emergent abilitiesconstantly drives for substantially larger models, resulting in the rapid increase in the cost of LLM training and inference.
People have been trying to reduce this cost through optimizations in various aspects, including architecture, data quality, operator, parallelization, optimizer, scaling laws, generalization theory, hardware, etc.We introduce the novel approach of optimizing knowledge storage.
The combined cost of LLM training and inference can be seen as the cost of encoding the knowledge from text data into various memory formats, plus the cost of reading from these memories during inference:(1) whereis the cost of encoding a piece of knowledgeinto memory format,is the cost of integratingfrom formatinto inference,
andis the expected usage count of this knowledge during the lifespan of this LLM (e.g. a few months for each version of ChatGPT) .
The definitions of knowledge and memory in the context of LLMs are provided in Section2, and this paper uses knowledge as a countable noun.
Typical memory formats include model parameters and plain text for retrieval-augmented generative models (RAG) ; their write functions and read functions are listed in Table3, and theirandare provided in Figure4.We introduce a new memory format, explicit memory, characterized by moderately low write cost and read cost.
As depicted in Figure1, our model first converts a knowledge base (or any text dataset)  into explicit memories, implemented as sparse attention key-values, and then during inference, recalls these memories and integrates them into the self-attention layers.
Our design is simple so that most of the existing Transformer-based LLMs should be able to accommodate explicit memories with a little finetuning, and thus it is a general-purpose “model amplifier”.
Eventually, it should reduce the cost of pretraining LLMs, since there will be much less knowledge that must be stored in parameters, and thus less training data and smaller model size.The new memory format enables us to define a memory hierarchy for LLMs:plain text (RAG) explicit memorymodel parametersuch that by going up the hierarchy,increases whiledecreases.
To minimize the cost (1) , one should store each piece of knowledge that is very frequently/rarely used in the top/bottom of this hierarchy, and everything in between as explicit memory.
As illustrated in Table3, the memory hierarchy of LLMs closely resembles that of humans.
For humans, the explicit/implicit memories are the long-term memories that are acquired and used consciously/unconsciously.Memory formatof humansExampleMemory formatof LLMsWriteReadImplicit memorycommon expressionsmodel parameterstrainingmatrix multiplicationExplicit memorybooks readthis workmemory encodingself-attentionExternal informationopen-book examplain text (RAG) noneencode from scratchTable 3:Analogy of the memory hierarchies of humans and LLMs.As a remark, one can compare the plain LLMs to patients with impaired explicit memory, e.g. due to injury to the medial temporal lobe.
These patients are largely unable to learn semantic knowledge (usually stored as explicit memory) , but can acquire sensorimotor skills through repetitive priming (stored as implicit memories) .
Thus, one may hypothesize that due to the lack of explicit memory, the training of plain LLMs is as inefficient as repetitive priming, and thus has ample room for improvement.
In analogy with humans, for instance, it is easy to recall and talk about a book we just read, but to recite it as unconsciously as tying shoe laces requires an enormous effort to force this knowledge into our muscle memory.
From this perspective, it is not surprising that LLM training consumes so much data and energy.
We want to rescue LLMs from this poor condition by equipping it with an explicit memory mechanism as efficient as that of humans.Figure 4:The total cost (TFlops)  of writing and reading a piece of knowledge by our 2.4B model with respect to its expected usage count.
The curves represent the cost of different memory formats, and the shaded area represents the minimum cost given the optimal format.
The plot indicates thatis the advantage interval for explicit memory.
The calculations are provided in AppendixA.
(The blue curve is only a lower bound on the cost of model parameters.) A quantitative illustration of the cost (1)  is given by Figure4, where we characterizeandby the amount of compute (TFlops) .
The plot indicates that if a piece of knowledge has an expected usage count, then it is optimal to be stored as an explicit memory.
Moreover, the introduction of explicit memory helps to externalize the knowledge stored in model parameters and thus allow us to use a lighter backbone, which ultimately reduces all the costs in Figure4.The second motivation for explicit memory is to alleviate the issue of knowledge traversal.
Knowledge traversal happens when the LLM wastefully invokes all its parameters (and thus all its knowledge)  each time it generates a token.
As an analogy, it is unreasonable for humans to recall everything they learned whenever they write a word.
Let us define the knowledge efficiency of an LLM as the ratio of the minimum amount of knowledge sufficient for one decoding step to the amount of knowledge actually used.
An optimistic estimation of knowledge efficiency for a 10B LLM is:
On one hand, it is unlikely that generating one token would require more thanbits of knowledge (roughly equivalent to a thousand-token long passage, sufficient for enumerating all necessary knowledge) ;
on the other hand, each parameter is involved in the computation and each stores at least 0.1 bit of knowledge(this density could be much higher if the LLM is trained on cleaner data) , thus usingbits in total.A novel architecture is needed to boost the knowledge efficiency of LLMs fromto, whereas current designs are far from this goal.
Consider the mixture-of-experts architecture (MoE)  for instance, which uses multiple MLP layers (experts)  in each Transformer block and process each token with only a few MLPs.
The boost of MoE, namely the ratio of the total amount of parameters to the amount of active parameters, is usually bounded by.
Similarly, neither the mixture-of-depth architecturenor sparsified MLP neurons and attention headscan bring greater gains.
RAG appears very sparse if we compare the amount of retrieved texts with the size of the text database; nevertheless, RAG is usually built upon a plain LLM as backbone, which provides most of the knowledge used in inference, and thus offers little assistance in addressing the knowledge traversal problem.An ideal solution is to retrieve only the needed parameters for each token.
This is naturally achieved by explicit memories if we compare memory recall to parameter retrieval.The third motivation is that, as a human-like design, explicit memory enables LLMs to develop more human-like capabilities.
To name a few,Infinitely long context:
LLMs have the difficulty of processing long texts since their working memory (context key-values)  costs too much GPU memory and compute.
Meanwhile, despite that humans have very limited working memory capacity, they can manage to read and write long texts by converting working memories to explicit memories (thus saving space)  and retrieving only the needed explicit memories for inference (thus saving compute) .
Similarly, by saving explicit memories on drives and doing frequent and constant-size retrieval, LLMs can handle arbitrarily long contexts with time complexityinstead of, whereis the context length.Memory consolidation: Instead of writing a piece of knowledge directly into implicit memory, i.e. training model parameters, LLM can first convert it to explicit memory through plain encoding, and then convert this explicit memory to implicit memory through a low-cost step such as compression and finetuning, thus reducing the overall cost.Factuality and interpretability: Encoding texts as explicit memories is less susceptible to information loss compared to dissolving them in model parameters.
With more factual details provided by explicit memories, the LLMs would have less tendency to hallucinate.
Meanwhile, the correspondence of explicit memories to readable texts makes the inference more transparent to humans, and also allows the LLM to consciously examine its own thought process.We demonstrate the improved factuality in the experiments section, and leave the rest to future work.In this work, we introduce a novel architecture and training scheme for LLM based on explicit memory.
The architecture is called Memory3, as explicit memory is the third form of memory in LLM after working memory (context key-values)  and implicit memory (model parameters) .Memory3utilizes explicit memories during inference, alleviating the burden of model parameters to memorize specific knowledge.The explicit memories are encoded from our knowledge base, and our sparse memory format maintains a realistic storage size.We trained from scratch a Memory3model with 2.4B non-embedding parameters, and its performance surpasses SOTA models with greater sizes.
It also enjoys better performance and faster inference than RAG.Furthermore, Memory3boosts factuality and alleviates hallucination, and it enables fast adaptation to professional tasks.This paper is structured as follows:
Section2lays the theoretical foundation for Memory3, in particular our definitions of knowledge and memory.
Section3discusses the basic design of Memory3, including its architecture and training scheme.
Sections4,5, and6describes the training of Memory3.
Section7evaluates the performance of Memory3on general benchmarks and professional tasks.
Finally, Section8concludes this paper and discusses future works.1.1|Related work1.1.1|Retrieval-augmented TrainingSeveral language models have incorporated text retrieval from the pretraining stage.
REALMaugments a BERT model with one retrieval step to solve QA tasks.
Retroenhances auto-regressive decoding with multiple rounds of retrieval, once per 64 tokens.
The retrieved texts are injected through a two-layer encoder and then several cross-attention layers in the decoder.
Retro++explores the scalability of Retro by reproducing Retro up to 9.5B parameters.Meanwhile, several models are adapted to retrieval in the finetuning stage.
WebGPTlearns to use search engine through imitation learning in a text-based web-browsing environment.
Toolformerperforms decoding with multiple tools including search engine, and the finetuning data is labeled by the LM iself.The closest model to ours is Retro.
Unlike explicit memory, Retro needs to encode the retrieved texts in real-time during inference.
To alleviate the cost of encoding these references, it chooses to use a separate, shallow encoder and also retrieve few references.
Intuitively, this compromise greatly reduces the amount of knowledge that can be extracted and supplied to inference.Another line of research utilizes retrieval to aid long-context modeling.
Memorizing Transformerextends the context of language models by an approximate kNN lookup into a non-differentiable cache of past key-value pairs.
LongLlamaenhances the discernability of context key-value pairs by a finetuning process inspired by contrastive learning.
LONGMEMdesigns a decoupled architecture to avoid the memory staleness issue when training the Memorizing Transformer.
These methods are not directly applicable to large knowledge bases since the resulting key-value caches will occupy enormous space.
Our method overcomes this difficulty through a more intense memory sparsification method.1.1.2|Sparse ComputationTo combat the aforementioned knowledge traversal problem and improve knowledge efficiency, ongoing works seek novel architectures that process each token with a minimum and adaptive subset of model parameters.
This adaptive sparsity is also known as contextual sparsity.
The Mixture-of-Experts (MoE)  use sparse routing to assign Transformer submodules to tokens, scaling model capacity without large increases in training or inference costs.
The most common MoE designhosts multiple MLP layers in each Transformer block and routes each token to a few MLPs with the highest allocation score predicted by a linear classifier.
Furthermore, variants based on compression such as QMoEare introduced to alleviate the memory burden of MoE.
Despite the sparse routing, the boost in parameter efficiency is usually bounded by.
For instance, the Arctic model, one of the sparsest MoE LLM in recent years, has an active parameter ratio of about.
Similarly, the Mixture of Depth architecture processes each token with an adaptive subset of the model layers.
The implementations can be based on early exitor top-routing, reducing the amount of compute to.
More fine-grained approaches can perform sparsification at the level of individual MLP neurons and attention heads.
The model Deja Vutrains a low-cost network for each MLP/attention layer that predicts the relevance of each neuron/head at this layer to each token.
Then, during inference, Deja Vu keeps the topMLP neurons andattention heads for each token.1.1.3|Parameter as memorySeveral works have portrayed model parameters as implicit memory, in accordance with our philosophy.demonstrates that the neurons in the MLP layers of GPTs behave like key-value pairs.
Specifically, with the MLP layer written as, each row of the first layer weightfunctions like a key vector, with the corresponding row in the second layer weightbeing the value vector.observes that for most of the MLP neurons, theis activated by context texts that obey some human interpretable pattern, and theactivates the column of the output matrix that corresponds to the most probable next token of the pattern (e.g.-gram) .
Based on this observation,designs a GPT variant that consists of only attention layers, with performance matching that of the usual GPTs.
The MLP layers are incorporated into the attention layers in the form of key-value vector pairs, which are called persistent memories.
Similarly, using sensitivity analysis,discovers that factual knowledge learned by BERT is often localized at one or few MLP neurons.
These neurons are called “knowledge neurons”, and by manipulating them,manages to update single pieces of knowledge of BERT.
Meanwhile,studies an interesting phenomenon known as superposition or polysemanticity, that a neural network can store many unrelated concepts into a single neuron.
introduction
Memory3: Language Modeling with Explicit Memory
Section 1
2407.01178
2|Memory Circuitry TheoryThis section introduces our memory circuitry theory, which defines knowledge and memory in the context of LLM.
We will see that this theory helps to determine which knowledge can be stored as explicit memory, and what kind of model architecture is suitable for reading and writing explicit memories.
For readers interested primarily in the results, it may suffice to review Claim1and Remark1before proceeding to the subsequent sections.
The concepts to be discussed are illustrated in Figure5.Figure 5:Categorization of knowledge and memory formats.
The explicit memories, extracted from model activations, lie half-way between raw data and model parameters, so we use a dotted line to indicate that they may or may not be regarded as parameters.2.1|PreliminariesThe objective is to decompose the computations of a LLM into smaller, recurring parts, and analyze which parts can be separated from the LLM.
These small parts will be defined as the “knowledge” of the LLM, and this characterization helps to identify what knowledge can be externalized as explicit memory, enabling both the memory hierarchy and a lightweight backbone.One behaviorist approach is to define the smaller parts as input-output relations between small subsequences, such that if the input text contains a subsequence belonging to some pattern, then the output text of the LLM contains a subsequence that belongs to some corresponding pattern.One specific input-output relation is that if the immediate context contains “China” and “capital”, then output the token “Beijing”.One abstract input-output relation is that if the immediate context is some arithmetic expression (e.g. “”)  then output the answer (e.g. “”) .One abstract relation that will be mentioned frequently is the “search, copy and paste”, such that if the context has the form “…[a][b]…[a]” then output “[b]”, where [a] and [b] are arbitrary tokens.A decomposition into these relations seems natural since autoregressive LLMs can be seen as upgraded versions of-grams, with the fixed input/output segments generalized to flexible patterns and with the plain lookup table generalized to multi-step computations.Nevertheless, a behaviorist approach is insufficient since an input-output relation alone cannot uniquely pin down a piece of knowledge:
a LLM may answer correctly to arithmetic questions based on either the actual knowledge of arithmetic or memorization (hosting a lookup table for all expressions such as “”) .
Therefore, we take a white-box approach that includes in the definition the internal computations of the LLM that convert these inputs to the related outputs.Here are two preliminary examples of internal computations.Example 1.Several works have studied the underlying mechanisms when LLMs answer to the prompt “The capital of China is” with “Beijing”, as well as other factual questions.
At least two mechanisms are involved, and the LLM may use their superposition.
One mechanism is to use general-purpose attention heads (called “mover heads”)  to move “capital” and “China” to the last token “is”, and then use the MLP layers to map the feature of the last token to “Beijing”.
Often, only one or a few MLP neurons are causally relevant, and they are called “knowledge neurons”.
This mechanism is illustrated in Figure6(left) .
Another mechanism involves attention headswhose value-to-output matricesfunction like bigrams,
e.g. mapping “captial” to {“Paris”, “Beijing”, …} and “China” to {“panda”, “Beijing”, …} , which sum up to produce “Beijing”.
This mechanism is illustrated in Figure6(middle) .Example 2.The ability of LLMs to perform “search, copy and paste”, namely answering to the context “…[a][b]…[a]” with “[b]”, is based on two attention heads, together called induction heads.
The first head copies the feature of the previous token, enabling [b] to “dress like” its previous token [a].
The second head searches for similar features, enabling the second [a] to attend to [b], which now has the appearance of [a].
Thereby, the last token [a] manages to retrieve the feature of [b] and to output [b].
This mechanism is illustrated in Figure6(right) .
A similar mechanism is found for in-context learning.Figure 6:Illustration of three subgraphs.
Left: A subgraph that inputs “the capital of China is” and outputs “Beijing”.
The knowledge neuron is marked in red and the mover heads in green.
Middle: Another subgraph with similar function using task-specific heads.
Right: The induction-heads subgraph that inputs “[a][b]…[a]” and outputs [b], where [a], [b] are arbitrary tokens.
The notations are introduced in Section2.2.
The locations of these attention heads and MLP neurons may be variable.We will address the internal mechanism for an input-output relation as a circuit, and will define a piece of knowledge as an input-output relation plus its circuit.
By manipulating these circuits, one can separate many pieces of knowledge from a LLM while keeping its function intact.Recent works on circuit discovery demonstrate that some knowledge and skills possessed by Transformer LLMs can be identified with patterns in their computation graphs, but there has not been a universally accepted definition of circuit.
Different from works on Boolean circuitsand circuits with Transformer submodules as their nodes, we characterize a circuit as a “spatial-temporal” phenomenon, whose causal structure is localized at the right places (MLP neurons and attention heads)  and right times (tokens) .
Thus, we define a computation graph as a directed acyclic graph, whose nodes are the hidden features of all tokens at all all MLP and attention layers, and whose edges correspond to all activations inside these layers.
In particular, the computation graph hosts one copy of the Transformer architecture at each time step.
To transcend this phenomenological characterization, we define a circuit as an equivalence class of similar subgraphs across multiple computation graphs.As a remark, it is conceptually feasible to identify a circuit with the minimal subset of Transformer parameters that causes this circuit.
The benefit is that such definition of knowledge seems more intrinsic to the LLM.
Nevertheless, with the current definition, it is easier to perform surgery on the circuits and derive constructive proofs.
Besides, it is known that Transformer submodules exhibit superposition or polysemanticity, such that one MLP neuron or attention head may serve multiple distinct functions, making the identification of parameter subsets a challenge task.2.2|KnowledgeWe begin with the definition of the knowledge of LLMs.
For now, it suffices to adopt heuristic definitions instead of fully rigorous ones.
Throughout this section, by LLM we mean autoregressive Transformer LLM that has at least been pretrained.
Letbe the number of Transformer blocks andbe the number of attention heads at each attention layer, and the blocks and heads are numbered byand.
There are in totallayers (MLP layers and attention layers) , and the input features to these layers are numbered by.Definition 1.Given an LLM and a text, thecomputation graphon inputand targetis a directed graph with weighted edges such thatIts nodes consist of the hidden vectorsbefore all attention layers, the hidden vectorsbefore all MLP layers, and the output vectors, for all blocksand positions.Its directed edges consist of each attention edgethat goes fromtoat the-th head of the-th attention layer for alland, as well as each MLP edgethat goes fromtothrough the-th neuron of the-th MLP layer for all.The weight of each attention edge, which measures the influence of the attention scoreon the LLM output, is defined bywhereis the log-likelihood of the target, withobtained by setting(i.e. causal intervention) .
Similarly, the weight of each MLP edge, which measures the influence of the neuron activationon the LLM output, is defined likewise.Given any subgraph, define theassociated inputofas a subsequencesuch that a tokenbelongs toif and only ifis large for some attention edge (or MLP edge)  inwith attention score (or activation) .Similarly, define theassociated outputof the subgraphas a subsequencesuch that a tokenbelongs toif and only ifis large for some attention edge (or MLP edge)  inwith attention score (or activation) .
Hereis the log-likelihood ofwith respect to the LLM output.(a) (b) Figure 7:Left: Illustration of the computation graph over one Transformer block, showing only three tokens, one attention head and three MLP neurons.
The edge weights are not shown.
Right: The subgraphs, namely the induced subgraphs of the attention edges (black arrows) , belong to the circuit of the induction head.
The red arrows denote a homomorphism fromto, and the blue arrows denote a homomorphism fromto.Definition 2.Given two computation graphsof an LLM and their subgraphs, a mappingfrom the nodes ofto the nodes of(not necessarily injective)  is ahomomorphismifevery node at depthis mapped to depth,if two nodes are on the same position, then they are mapped onto the same position,if two nodes share an edge on attention heador MLP neuron, then their images also share an edge on heador neuron.If such a homomorphism exists, then we say thatis homomorphic to.It may be more convenient to define the mapping to be between the input tokens of two sentences, but we adopt the current formulation as it is applicable to more general settings without an obvious correspondence between the tokens and the hidden features at each layer.Definition 3.Given an LLM and a distribution of texts, acircuitis an equivalence classof subgraphs from computation graphs on random texts, such thatThe computation graph on a random text contains some subgraphwith positive probability.All subgraphsare homomorphic to each other.All edges of allhave non-negligible weights.The pairsshare some interpretable meaning across all.Definition 4.Given an LLM and a distribution of texts, we call each circuit aknowledge.
Furthermore, a circuitis called aspecific knowledge, if the associated inputsfor all subgraphsshare some interpretable meaning, and the associated outputsfor allare the same or differ by at most a small fraction of tokens.abstract knowledge, else.From now on, we use knowledge as a countable noun since the circuits are countable.
Note that the criterion in Definition4is stronger than the last criterion in Definition3, e.g. consider the circuit that always copy-and-pastes the previous token.
We will see that the rigidity of specific knowledges makes them easier to externalize.Here are some well-known examples of knowledge.Example 3.Recall the knowledge neuron from Example1that helps to answer “The capital of China is Beijing”.
Such neurons can be activated by a variety of contexts that involve the subject-relation pair (“China”, “capital”) .
Its circuit can be simply defined as the equivalence class of subgraphs induced by edges, whereis the fixed location of the knowledge neuron andis the variable position of the last token of the context.
The associated inputs are “China” and “capital”, and the associated outputs are always “Beijing”.
By definition, this circuit is a specific knowledge, since its associated output is fixed and its associated inputs share a clear pattern (fixed tokens with variable positions) .Similarly, by straightforward construction, one can show that each-gram can be expressed as a specific knowledge.Example 4.Recall the induction headsfrom Example2that complete “[a][b] …[a]” with “[b]”.
Letbe the locations of these two heads, and denote the variable positions of the two token [a]’s by.
Its circuit is the equivalence class of subgraphs induced by the two edges.
Although the associated input-output pairs “[a][b]…[a][b]” have a clear pattern, the associated outputs “[b]” alone can be arbitrary, so the induction head is an abstract knowledge.More sophisticated abstract knowledges have been identified for in-context learningand indirect object identification.Definition 5.Given a LLM and a knowledge, a textis called arealizationof, if the computation graph onhas a subgraph that belongs to.For instance, any text of the form [a][b]…[a][b] can be a realization of the abstract knowledge of induction head.Our definition of knowledge is extrinsic, depending on a specific LLM, instead of intrinsic, depending only on texts.
From this perspective, Problem (1)  can be interpreted as relocating the knowledges from an all-encompassing LLM to more efficient models equipped with memory hierarchy.
For concreteness, one can fix this reference LLM to be the latest version of ChatGPT or Claude, or some infinitely large model from a properly defined limit that has learned from infinite data.Assumption 1(Completeness) .Fix a reference LLM and a distribution of texts, letbe the computation graph of a random text.
Assume that there exists a setof knowledges such that, with probability 1 over the random text,
the subgraph ofinduced by edges with non-negligible weights can be expressed as a union of subgraphsfrom.Essentially, Assumption1posits that all computations in the LLM can be fully decomposed into circuits, so that the LLM is nothing more than a collection of specific and abstract knowledges.
This viewpoint underscores that the efficiency of LLMs is ultimately about the effective organization of these knowledges, an objective partially addressed by Problem (1) .2.3|MemoryNow the question is what knowledge can be separated from the model parameters and moved to the lower levels of the memory hierarchy.Definition 6.A knowledgeof the reference LLM isseparableif there exists another LLMsuch thatdoes not possess this knowledge, such that for any realizationof, the modelcannot generate each token of the associated outputwith high probability, e.g.for some.There exists a textsuch that for any realizationof,
the modelusingas prefix can generate each token of the associated outputwith high probability,
e.g.for every.If among the realizations of, the same associated inputcan correspond to multiple associated outputs, then the above probabilities are summed over all branches if positionis a branching point.Definition 7.A separable knowledgeof the reference LLM isimitableif any realizationofcan be used as the prefixin Definition6,
e.g. for any realizationsof, we havefor every.Basically, imitability means that LLMs can achieve the same effect as possessing this knowledge by retrieving example texts that demonstrate this knowledge.
Few-shot prompting can be seen as a special case of providing realizations.Separability is a more general property than imitability.
For instance, one can set the prefixto be an abstract description ofinstead of its realization, and this is reminiscent of instruction prompting.
Nevertheless, it is not obvious whether the set of separable knowledges is strictly larger than the set of imitable knowledges.Claim 1.Every specific knowledgeis imitable and thus is separable.Proof (informal) .Without loss of generality, we can assume that for any realizationof, all tokens of the associated inputprecede all tokens of the associated output.
Otherwise, we can splitinto two halvesthat precedes/does not precede, and split the corresponding subgraphinto two halvesthat have high weights with respect to.
Using monotonicity arguments once Definition3is fully formalized, one can try to show that this splitting is invariant acrossand therefore the sets ofare two specific knowledges.Consider sequences of the form [a][b]…[a’][b’], where [a], [a’] (or [b], [b’])  could be the associated inputs (or outputs)  of any subgraphs.
By Definition4, [a] and [a’] always share some interpretable meaning, while [b] and [b’] are approximately the same sequence.
One can construct an abstract knowledge that completes [a][b]…[a’] with [b’]:
the first part of this circuit detects the common feature of the [a]’s (possibly overlapping with the subgraphs of) , the second part is an induction head (analogous to Example4, it provides [b] with the common feature of the [a]’s and lets [a’] to attend to [b]) , and the third part generates [b’] based on [b] with possible slight modifications.
This circuit is an abstract knowledge since it can be applied to other specific knowledges as long as their associated inputs share the same meaning with the [a]’s, no matter how their associated outputs could vary.Meanwhile, construct the modelby letting the reference model forget(e.g. by finetuning on a modified data distribution such that the associated input ofis never followed by the associated output, while the rest of the distribution remains the same) .
Combining this circuit withcompletes the proof.
∎Claim1indicates that a lot of knowledges can be externalized from the model parameters.
The converse of Claim1may not hold, since it is imaginable that some abstract knowledges can also be substituted with their realizations.Remark 1.There are three details in the proof of Claim1that will be useful later0.The circuit we construct has only one attention head that attends to the reference textfrom the present text, while all other computations are confined within eitheror.0.Moreover, in this attention head, the circuit only needs the edges from [b] to [a’].
Thus, in general this head only needs to attend to very few tokens in the reference.0.It suffices for the referenceto attend only to itself.These properties will guide our architecture design.To finish the set-up of Problem (1) , we define the memory formats.
The definition should subsume the aforementioned formats of model parameters, explicit memories and plain texts for RAG, and also allow for new memory formats of future LLMs.Definition 8.Letbe the complete set of knowledges from Assumption1and consider the subset of separable knowledges.
Letbe a set that contains one or several realizationsfor each separable knowledge.
Letbe any functions over.
Abstractly speaking, a memory-augmented LLMis some mapping from prefixes to token distributions with additional inputs(2) where the setconsists of non-separable knowledges ofthat are invoked at this step, and the setsconsist of encoded texts(3) for some.Eachrepresents a memory format andis called thewritefunction of this format.
If some realization of a separable knowledgeparticipates in the mapping, then we say thatiswrittenin formatandreadby.Analogous to Assumption1, we are decomposing each step of LLM inference into the invoked circuits, but the decomposition here also involves reference texts that are written in various memory formats.Table3demonstrates that the write functions could be diverse, and the list is probably far from conclusive.
Nevertheless, some heuristics still apply.
The write functionand the read process infor each formatshould be non-trivial such that, for any separable knowledgenot contained inand any realizationof, ifenters inthrough format, thenshould be able to generate each token of the associated output ofinwith higher probability as in Definition6.
Thus, informally speaking, the total cost of writing and readingmust be bounded from 0, since some minimum computation is necessary for reducing the uncertainty in generating the correct tokens.
It follows that the write cost and read cost are complementary, i.e. cheaper writing must be accompanied by more expensive reading.We define this inverse relationship between the write cost and read cost as the memory hierarchy.
This relationship is in accordance with our experience regarding the three examples of human memories in Table3, e.g. we can utter the common expressions almost immediately while it may take a few seconds to recall a book we read, but the former skill is acquired through years of language speaking.
For the LLM memories in Table3, the inverse relationship is illustrated Figure4and established by the calculations in AppendixA.Figure 8:Different memory formats with different balances of write cost and read cost.
The specific knowledges with high to low usage counts are exemplified by common expressions, expertise and trivia, and are assigned to implicit memory, explicit memory and external information.The imbalanced use of knowledges leads to a heterogeneous distribution of knowledges across the memory hierarchy.
To minimize the total cost (1) , the separable knowledges that are used more often should be assigned to memory formats with high write cost and low read cost, whereas the rarely used knowledges should be assigned to formats with low write cost and high read cost.
Also, adding a new memory formatis always beneficial as it expands the search space and decreases the minimum cost whenever the usage count of some knowledgelies in the intervalExamples of these intervals are displayed in Figure4.
For concreteness, Figure8depicts a reasonable distribution of the specific knowledges for humans, and we expect a similar distribution to hold for LLMs equipped with explicit memory.
body
Memory3: Language Modeling with Explicit Memory
Section 2
2407.01178
3|DesignThis section describes the architecture and training scheme of Memory3.Regarding architecture, the goal is to design an explicit memory mechanism for Transformer LLMs with moderately low write cost and read cost.
In addition, we want to limit the modification to the Transformer architecture to be as little as possible, adding no new trainable parameters, so that most of the existing Transformer LLMs can be converted to Memory3models with little finetuning.
Thus, we arrive at a simple design:Write cost:
Before inference, the LLM writes each reference to an explicit memory, saved on drives.
The memory is selected from the key-value vectors of the self-attention layers, so the write process involves no training.
Each reference is processed independently, avoiding the cost of long-context attention.Read cost:
During inference, explicit memories are retrieved from drives and read by self-attention alongside the usual context key-values.
Each memory consists of very few key-values from a small amount of attention heads, thus greatly reducing the extra compute, GPU storage, drive storage and loading time.
It allows the LLM to retrieve many references frequently with limited influence on decoding speed.Regarding training, the goal is to reduce the cost of pretraining with a more efficient distribution of knowledge.
Based on the discussion in Section2.3, we want to encourage the LLM to learn only abstract knowledges, with the specific knowledges mostly externalized to the explicit memory bank.
Ideally, the pretraining cost should be reduced to be proportional to the small amount of knowledge stored in the model parameters, thereby taking a step closer to the learning efficiency of humans.3.1|Inference ProcessFrom now on, we refer to the realizations of separable knowledges (Definitions5and6)  as references.
Our knowledge base (or reference dataset)  consists oftext chunks with length bounded by 128 tokens.
Its composition is described in Section4.4.Each reference can be converted to an explicit memory, which is a tensor with shape(memory layers, 2, key-value heads, sparse tokens, head dimension)  =The 2 stands for the key and value, while the other numbers are introduced later.Before inference, the Memory3model converts all references to explicit memories and save them on drives or non-volatile storage devices.
Then, at inference time, whenever (the id of)  a reference is retrieved, its explicit memory is loaded from drives and sent to GPU to be integrated into the computation of Memory3.
By Remark1, a reference during encoding does not need to attend to any other texts (e.g. other references or query texts) , so it is fine to encode each reference independently prior to inference.
Such isolation also helps to reduce the compute of attention.One can also employ a “cold start” approach to bypass preparation time: each reference is converted to explicit memory upon its initial retrieval, rather than prior to inference.
Subsequent retrievals will then access this stored memory.
The aforementioned inference with precomputed explicit memories will be called “warm start”.Figure 9:The decoding process of Memory3with memory recall.
Each chunk is a fixed-length interval of tokens, which may belong to either the prompt or generated text.During inference, as illustrated in Figure9, whenever the LLM generates 64 tokens, it discards the current memories, uses these 64 tokens as query text to retrieve 5 new memories, and continues decoding with these memories.
Similarly, when processing the prompt, the LLM retrieves 5 memories for each chunk of 64 tokens.
Each chunk attends to its own memories, and the memories could be different across chunks.
We leave it to future work to optimize these hyperparameters.The retrieval is performed with plain vector search with cosine similarity.
The references as well as the query chunks are embedded by BGE-M3, a multilingual BERT model.
The query and key vectors for retrieval are both obtained from the output feature of thetoken.
The vector index is built with FAISS.To further save time, we maintain a fixed-size cache in RAM to store the most recently used explicit memories.
It’s been observed that adjacent chunks often retrieve some of the same references.
So the cache reduces the cost of loading explicit memories from drives.Remark 2.It would be ideal to perform retrieval using the hidden features from the LLM itself,
since conceptually the LLM should know its needs better than any external module,
and such internalized retrieval appears more anthropomorphic.
Moreover, retrieving with the hidden features from different layers, different heads and different keywords can help to obtain more diverse results.
One simple implementation is to use the sparsified attention queries of the query text to directly search for the explicit memories.
Since the explicit memories are the attention key-values, such retrieval can work without the need to finetune the LLM.
Specifically, this multi-vector retrieval can follow the routine ofwith the additional constraint that a query from attention headcan only search for keys from,
while the sparse attention queries can be obtain using the same selection mechanism for explicit memories described later.Remark 3.One shortcoming of RAG is that the references are usually text chunks instead of whole documents, and thus during inference the references are encoded without their contexts, making them less comprehensible.
This shortcoming can be easily overcome for explicit memories.
One solution is to encode each document as one sequence, then chunk the attention key-values into 128-token chunks and sparsify them into explicit memories.
This procedure allows the key-values to attend to all their contexts.3.2|Writing and Reading MemoryEach explicit memory is a subset of the attention key-values from a subset of attention heads when encoding a reference.
Thus, during inference, the LLM can directly read the retrieved explicit memories through its self-attention layers by concatenating them with the usual context key-values (Figure9) .
Specially, for each attention headat layer, if it is chosen as a memory head, then its outputchanges from the usualwheredenotes all tokens before or at positionanddenotes the head dimension, to(4) where eachdenotes the keys and values of an explicit memory.While the context BOS token isas usual,
when encoding each reference we modify the BOS to “Reference:” to help the LLM distinguish between encoding normal texts and encoding references.
This modified BOS is also prepended to the context during inference, as illustrated in Figure9, while the context BOS token now serves as a separator between the references and context.
Unlike the explicit memories which only appear at a subset of attention heads, this modified BOS is placed at every head at every layer.
The motivation is that since the context BOS can attend to the references, its feature is no longer constant, so the LLM needs the modified BOS to serve as the new constant for all attention heads.Furthermore, we adopt parallel position encoding for all explicit memories, namely the positions of all their keys lie in the same interval of length 128, as depicted in Figure9.
We use the rotary position encoding (RoPE) .
The token sparsification is applied after RoPE processes the attention keys, so the selected tokens retain their relative positions in the references.
Besides flexibility, one motivation for parallel position is to avoid the “lost in the middle” phenomenon, such that if the references are positioned serially, then the ones in the middle are likely to be ignored.
Similarly, token sparsification also helps to alleviate this issue by making the attention more focused on the important tokens.
We note that designs analogous to the parallel position have been used to improve in-context learningand long-context modeling.3.3|Memory Sparsification and StorageOne of the greatest challenges for explicit memories is that the attention key-values occupy too much space.
They not only demand more disk space, which could be costly, but also occupy GPU memory during inference, which could harm the batch size and thus the throughput of LLM generation.
An intense compression is needed to save space.
The full attention key tensor (or value tensor)  for each reference has shape (layers, key-value heads, tokens, head dimension) , so we compress all four dimensions.Regard layers, we only set the first half of the attention layers to be memory layers, i.e. layers that produce and attend to explicit memories (4) , while the second half remain as the usual attention layers.
Note that Remark1suggests that it is usually the attention heads in the middle of the LLM that attend to the references.
So it seems that appointing the middle attention layers (e.g. the ones within thetodepth range)  to be memory layers is a more sensible choice.
This heuristic is supported by the observations inthat the attention to the distant context usually takes place in the middle layers.Regarding heads, we set all key-value heads at each memory layer to be memory heads.
We reduce their amount by grouped query attention (GQA) , letting each key-value head be shared by multiple query heads, and obtain 20% sparsity (8 versus 40 heads) .
It is worth mentioning that, besides GQA and memory layers, another approach is to select a small subset of heads that are most helpful for reading memories, and this selection does not have to be uniform across layer.
We describe several methods for selecting memory heads in Remark4.Regarding tokens, we select 8 tokens out of 128 for each key-value head.
We choose a high level of sparsity, since Remark1indicates that the attention from the context to the references are expected to be concentrated on very few tokens.
Note that the selected tokens are in general different among heads, so in principle their union could cover a lot of tokens.
For each headat layer, the selection uses top-8 over the attention weightwhich measures the importance of a token by the attention received from all tokens.
The BOS tokens and paddings do not participate in the the computation of the weights.
These attention weightsare different from the usual ones, such that there is no causal mask or position encoding involved.
The consideration is that since the explicit memories are prepared before any inference, the selection can only depend on the reference itself instead of any context texts.
The removal of causal mask and position encoding ensures that tokens at any position has an equal chance to receive attention from others.
To speed up computation, we adopt the following approximate weights in our implementation, although in retrospect this speedup is not necessary.Similar designs that sparsify tokens based on attention weights have been adopted in long-context modeling to save space.Regarding head dimension, we optionally use a vector quantizer to compress each of the key and value vectors using residual quantizationsbuilt with FAISS.
The compression rate is.
During inference, the retrieved memories are first loaded from drives, and then decompressed by the vector quantizer before being sent to GPU.
The evaluations in Section7.1indicate that this compression has negligible influence on the performance of Memory3.
More details can be found in AppendixB.Hence, the total sparsity is 160 or 1830 (without or with vector compression) .
Originally, the explicit memory bank would have an enormous size of 7.17PB or equivalently 7340TB (given the model shape described in Section3.4and saved in bfloat16) .
Our compression brings it down to 45.9TB or 4.02TB (without or with vector compression) , both acceptable for the drive storage of a GPU cluster.To deploy the Memory3model on end-side devices such as smart phones and laptops, one can place the explicit memory bank and the vector index on a cloud server, while the devices only need to store the model parameters and the decoder of the vector quantizer.
During inference, to perform retrieval, the model on the end-side device sends the query vector to the cloud server, which then searches the index and returns the compressed memories.
The speed test of this deployment is recorded in Section7.5.Remark 4.If one wants to finetune a pretrained LLM into a Memory3model, there are several ways to select a small but effective subset of attention heads (among all heads at all layers)  for memory heads (4) .
Methods such asare proposed to identify the heads that contribute the most to long-context modeling by retrieving useful information from distant tokens, and usually these special heads account for onlyof the total heads.
Here we also propose a simple method for selecting memory heads:
Given the validation subsets of a representative collection of evaluation tasks, one can measure the average performancefor a modified version of the LLM for each attention head.
The modification masks the distant tokens for headso it can only see the preceding 100 tokens and the BOS token.
Then, it is reasonable to expect thatwould be markedly low for a small subset of heads, indicating that they are specialized for long-range attention.Remark 5.Actually, Remark1suggests that each reference only needs to be attended to by just one attention head, although in general this special head may be different among the references.
Thus, it seems a promising approach to apply adaptive sparsity not only to token selection, but also to the memory heads, namely each reference is routed to one or two heads (analogously to MoE) , and its explicit memory is produced and read by these heads.
Such design if feasible can further boost the sparsity of explicit memory and save much more space.3.4|Model ShapeAs discussed in Section2.3, the specific knowledges can be externalized to explicit memories, and thus to minimize the total cost (1) , the model parameters (or implicit memory)  only need to store abstract knowledges and the subset of specific knowledges that are frequently used.
The shape of our model, i.e. (the number of Transformer blocks, heads, head dimension, width of the MLP layers) , is chosen to accommodate this desired knowledge distribution.
Informally speaking, given a fixed parameter size, the shape maximizes the following objective(5) Here we setto be 2.4 billion.Some recent works suggest that the capacities for learning specific knowledges and abstract knowledges are subject to different constraints.
On one hand,observes that the amount of bits of trivia information (such as a person’s name, date of birth and job title)  that a LLM can store depends only on its parameter size.
Regardless ofand, the max capacity is always around 2 bits per parameter.On the other hand,trains Transformers to learn simple algorithms such as reversing a list and counting the occurrence of each letter.
It is observed that for several such tasks, there exists a minimumandsuch that a Transformer withandcan learn the task with perfect accuracy, whereas the accuracy drops significantly for Transformers with eitheror(given that eitheror) .
This sharp transition supports the view that the layers and heads of Transformer LLMs can be compared to algorithmic steps, and tasks with a certain level of complexity require at least a certain amount of steps.
It is worth mentioning that the emergent phenomenonof LLMs can also be explained by this view and thus adds support to it, although it may not be the only explanation.By Definition4, the abstract knowledges are expected to be circuits with greater complexity than specific knowledges, since their associated inputs and outputs exhibit greater variability and thus express more complex patterns.
It follows that, in the context of the aforementioned works, the separation of specific and abstract knowledges should be positively correlated with the distinction between trivia information and algorithmic procedures.
Hence, it is reasonable to adopt the approximation that the capacity of an LLM for specific knowledges only depends on its parameter size, whereas the capacity for abstract knowledges depends only onand.The informal problem (5)  reduces to the maximization ofandgiven a fixed parameter size.
However, we are left with two ambiguities:
first, this formulation does not specify the ratio betweenand,
and second the head dimensionand MLP widthcannot be too small as the training may become unstable.
Regarding the second point, our experiments indicate that pretraining becomes more unstable with increased spikes if, so we set(though it needs to be pointed out that the loss spikes may not be solely attributed to the choice of, and high-quality data for instance may stabilize training and allow us to choose a smaller) .
Also, the MLP widthis set to be equal to the hidden dimension.
Regarding the first point, controlled experiments (Figure10)  indicate that the loss decreases slightly more rapidly withthan with other ratios, so we adopt this ratio.Figure 10:Comparison of the training losses of models with different shapes, whose parameter sizes range inB.
The legend l44h40d80 denotes, and the-axis denotes the amount of training samples.
Nevertheless, this comparison is not definite, since this is only the warmup stage of our training scheme (Section3.6)  and the ranking may change in the continual train stage when explicit memory is introduced.In addition, as discussed in Section3.3, our model uses grouped query attention (GQA) , so the number of key-value headsis set to be, which is the usual choice for GQA.
The MLP layers are gated two-layer networks without bias, which are the default choice in recent years.Finally, the model shape is set to be, with the total non-embedding parameter size beingB.3.5|Training DesignsSimilar to our architecture design, the design of our training scheme focuses on learning abstract knowledges.
The goal is to reduce the training compute, as the LLM no longer needs to memorize many of the specific knowledges.
This shift in learning objective implies that all the default settings for pretraining LLMs may need to be redesigned, as they were optimized for the classical scenario when the LLMs learn both abstract and specific knowledges.0.Data:
Ideally, the pretraining data should have a high concentration of abstract knowledges and minimum amount of specific knowledges.
It is known that LLM pretraining is very sensitive to the presence of specific knowledges.
For instance,observes that a small model can master arithmetic (e.g. addition of large numbers)  if trained on clean data.
However, if the training data is mixed with trivial information (e.g. random numbers) , then the test accuracy stays at zero unless the model size is increased by a factor of 1500.
It suggests that training on specific knowledges significantly inhibits the learning of abstract knowledges, and may explain why emergent abilitiesare absent from small models.
Notably, the Phi-3 modelis pretrained with a data composition that closely matches our desired composition.
Although the technical details are not revealed, it is stated that they filter data based on two criteria:
the data should encourage reasoning, and should not contain information that is too specific.0.Initialization:observes that initializing Transformer parameters with a smaller standard deviation (withinstead of the usual)  can encourage the model to learn compositional inference instead of memorization.
Specially, an arithmetic dataset is designed with a train set and an out-of-distribution test set, which admits two possible answers.
One answer relies on memorizing more rules during training, while the other requires an understanding of the compositional structure underlying these rules.
The proposed mechanism is that training with smaller initialization belongs to the condensed regime that encourages sparse solutions, contrary to training with large initialization that belongs to the kernel regime or critical regime.0.Weight decay:observe that using a larger weight decay coefficient (i.e. greater than the usual range of)  can guide LLMs to favor generalization over memorization, and accelerate the learning of generalizable solutions.
They consider settings that exhibit grokkingsuch that training would transit from perfect train accuracy and zero test accuracy to perfect test accuracy, and generalization ability is measured by how quickly this transition occurs.
Moreover, theoretically speaking, it is expected that training generative models needs stronger regularization than training regression models, in order to prevent the generated distributions from collapsing onto the training data and become trivial.In summary, it is recommendable to pretrain the Memory3model with a data composition that emphasizes abstract knowledges and minimizes specific information, a smaller initialization for parameters, and a larger weight decay coefficient.Since this work is only a preliminary version of Memory3, we decide to stick with the conventional setting for training and have not experimented with any of these ideas.
We look forward to incorporating these designs in future versions of the Memory3model.3.6|Two-stage PretrainThe Memory3model learns to write and read explicit memories during pretraining.
The training data is prepended with retrieved references;
the model encodes these references into explicit memories in real time, and integrates them into the self-attention computation of the training data.Unexpectedly, our pretraining consists of two stages, which we name as warmup and continual train.
Only the continual train stage involves explicit memories, while the warmup stage uses the same format as ordinary pretraining.
Our motivation is depicted in Figure11.
We observe that pretraining with explicit memories from the beginning would render the memories useless, as there appears to be no gain in training loss compared to ordinary pretraining.
Meanwhile, given a checkpoint from ordinary pretraining, continual training with explicit memory exhibits a visible decrease in training loss.
This comparison implies that a memory-less warmup stage might be necessary for pretraining a Memory3model.
One possible explanation for this phenomenon is that in the beginning of pretraining, the model is too weak to understand and leverage the explicit memories it generates.
Then, to reduce distraction, the self-attention layers might learn to always ignore these memories, thus hindering indefinitely the development of explicit memory.(a) (b) Figure 11:Left: Comparison of the warmup stage (training from scratch)  with and without explicit memory.
The blue and green curves are trained without and with explicit memories, respectively.
Right: Comparison of the continual train stage.
The blue and green curves are continual trained from their warmup checkpoints, and the red curve is initialized with the warmup checkpoint of the blue curve and continual trained with explicit memory.
These plots indicate that pretraining a Memory3model requires a memory-less warmup stage.
These experiments use a smaller model with 0.92B non-embedding parameters.
The warmup stage uses 60B data and the continual train stage uses 22B.Another modification is to reduce the cost of continual train.
Recall from Section3.1that during inference, each 64-token chunk attends to five explicit memories, or equivalently five 128-token references if using cold start, increasing the amount of input tokens by 10 times.
The inference process avoids the cost of memory encoding by precomputation or warm start, but for the continual train, the references need to be encoded in real time.
Our solution is to let the chunks share their references during training to reduce the total number of references in a batch.
Specifically, each chunk of a training sequence retrieves only one reference, and in compensation, attends to the references of the previous four chunks, besides its own reference.
Each train sequence has length 2048 and thus 32 chunks, so it is equipped withreference tokens.
The hidden features of these reference tokens are discarded once passing the last memory layer, since after that they no longer participate in the update of the hidden feature of the train tokens.
Hence, each continual train step takes slightly more than twice the amount of time of a warmup step.It is necessary to avoid information leakage when equipping the training data with references
(i.e. the train sequence and its retrieved references could be the same text) , for otherwise training becomes too easy and the model would not learn much.
Previously, Retrorequires that no train sequence can retrieve a reference from the same document, but this criterion may be insufficient since near-identical paragraphs may appear in multiple documents.
Thus, we require that no train sequence can be accompanied by a reference sequence that hasoverlap with it.
The overlap is measured by the length of their longest common subsequence divided by the length of the reference length.
Specially, given any train sequenceand reference, define their overlap by(6) The constraintensures that the overlap is not over-estimated as.
body
Memory3: Language Modeling with Explicit Memory
Section 3
2407.01178
4|Pretraining DataThis section describes the procedures for collecting and filtering our pretraining dataset and knowledge base (or reference dataset) .4.1|Data CollectionThe pretrain data is gathered from English and Chinese text datasets, mostly publicly available collections of webpages and books.
We also include code, SFT data (supervised finetuning) , and synthetic data.Specially, the English data mainly consists of RedPajamaV2, SlimPajamaand the Piles, in total 200TB prior to filtering.
The Chinese data mainly comes from Wanjuan, Wenshu, and MNBVC, in total 500TB prior to filtering.
The code data mainly comes from Github, and we take the subset with the highest repository stars.
The SFT data is included since these samples generally have higher quality than the webpages.
We use the same data as in SFT training (Section6.1) , except that these samples are treated as ordinary texts during pretraining, i.e. all tokens participate in the loss computation, not just the answer tokens.4.2|FilteringThe raw data is filtered with three steps: deduplication, rule-based filtering, and model-based filtering.First, deduplication is performed with MinHash for most of the datasets.
One exception is RedPajamaV2, which already comes with deduplication labels.Second, we devise heuristic, rule-based filters analogous to the ones from.
The purpose is to eliminate texts that are ostensibly unsuitable for training, such as ones that only contain webpage source codes, random numbers, or incomprehensible shards.
Our filters remove documents with less than 50 words, documents whose mean word lengths exceed 10 characters,
documents with 70% of context being non-alphabetic characters,
documents whose fractions of unique words are disproportionately high,
documents whose entropy of unigrams is excessively low, and so on.Finally, we select the subset of data with highest “quality”, a score produced by a finetuned BERT model.
Specially, we sample ten thousand documents and grade them by the XinYu-70B modelwith prompt-guided generation.
The prompt asks the model to determine whether the input text is informative and produce a score betweenand.
Then, these scores are used to finetune the Tiny-BERT model, which has only 14M parameters.
The hyperparameters of this finetuning are optimized with respect to a held-out validation set.
After that, we use this lightweight BERT to grade the entire dataset.Remark 6.Recall from Section3.5that the pretraining data of Memory3should emphasize abstract knowledges and minimize specific knowledges.
The purpose is to not only obtain a lightweight LLM with an ideal distribution of knowledges in accordance with the memory hierarchy (Figure8) , but also prevent the specific knowledges from hindering the learning process of the model.
The focus of our prompt on “informativeness” might be contradictory to this goal, since the selected texts that are rich in information content may contain too many specific knowledges.
For future versions of Memory3, we will switch to a model-based filter favoring texts that exhibit more reasoning and less specifics.Figure 12:Composition of our pretraining dataset.The filtered dataset consists of around four trillion tokens, and its composition is illustrated in Figure12.4.3|TokenizerSimilar to our dataset, our tokenizer mainly consists of Chinese and English tokens.
The English vocabulary comes from the 32000 tokens of the LLaMA2 tokenizer.
We include roughly the same amount of Chinese tokens produced from byte-pair encoding (BPE) .
The BPE is trained on a 20GB Chinese corpus that consists of Chinese news and e-books.
After deduplication, the final vocabulary has 60299 tokens.4.4|Knowledge BaseThe knowledge base (or reference dataset)  is used during training and inference as the source of explicit memories, as depicted in Figure1.
It consists of reference texts that are split into token sequences with length, as described in Section3.1.Heuristically, a larger knowledge base is always better, as long as it does not contain misinformation,
so it is not surprising that the reference dataset of Retro contains its entire pretrain dataset.
Nevertheless, the storage of explicit memories is more costly than plain texts despite our sparsification (Section3.3) , and thus to save storage space, we select a small subset of our pretrain dataset as the knowledge base.With a focus on high quality data, we include for references the English Wikipedia, WikiHow, the Chinese baike dataset, the subset of English and Chinese books whose titles appear academic, Chinese news, synthetic data and high quality codes.
These texts are tokenized and split into chunks of 128 tokens, resulting inreferences in total.One may be curious whether our knowledge base may contain some of the evaluation questions, rendering our evaluation results (Section7.1)  less credible.
To prevent such leakage, we include in our evaluation code a filtering step, such that for each evaluation question, if a retrieved reference has an overlap with the question that exceeds a threshold, then it is discarded.
This deduplication is analogous to the one used when preparing for continual train (Section3.6) ,
with the overlap measured by (6) .
The thresholdis chosen since we observe that typically a reference that contains a question would have an overlap, while a relevant but distinct reference would have an overlap.Remark 7.Currently, the compilation of the knowledge base is based on human preference.
For future versions of Memory3, we plan to take a model-oriented approach and measure the fitness of a candidate reference by its actual utility,
e.g. the expected decrease in the validation loss of the LLM conditioned on this reference being retrieved by a random validation sample.
body
Memory3: Language Modeling with Explicit Memory
Section 4
2407.01178
5|PretrainThis section describes the details of the pretraining process.
The two-stage pretrain and memory-augmented data follow the designs introduced in Section3.6.
As an interpretation, the Memory3model during the warmup stage develops its reading comprehension, which is necessary during the continual train stage for initiating memory formation.5.1|Set-upTraining is conducted with the Megatron-DeepSpeed packageand uses mixed-precision training with bfloat16 model parameters, bfloat16 activations, and float32 AdamW states.
The batch size is around 4 million training tokens with sequence length 2048, not including the reference tokens.
The weight decay is the common choice of 0.1.We adopt the “warmup-stable-decay” learning rate schedule of MiniCPM, which is reportedly better than the usual cosine schedule in term of training loss reduction.
The learning rate linearly increases to the maximum value, then stays there for the majority of training steps, and finally in the last 10% steps decays rapidly to near zero.
Our short-term experiments confirm the better performance of this schedule.
Nevertheless, frequent loss spikes and loss divergences are encountered during the official pretraining, so we have to deviate from this schedule and manually decrease the learning rate to stabilize training.Originally, it is planned that both the warmup and continual train stages go through the entire 4T token pretrain dataset (Section4) .
Due to the irremediable loss divergences, both stages have to be terminated earlier.5.2|Warmup Stage(a) (b) Figure 13:The warmup stage without explicit memory.
Left: Training loss.
Right: Learning rate schedule.The training loss and learning rate schedule are plotted in Figure13.
Whenever severe loss divergence occurs, we restart from the last checkpoint before the divergence with a smaller learning rate, and thus the divergences are not shown in the figure.
Eventually, the training terminates at around 3.1T tokens, when reducing the learning rate can no longer avoid loss divergence.5.3|Continual Train Stage(a) (b) Figure 14:The continual train stage with explicit memory.
Left: Training loss.
Right: Learning rate schedule.The explicit memories enter into the Memory3model at this stage.
The training steps are slower since the model needs to encode the references retrieved for the pretrain data to explicit memories in real time, and each step takes a bit more than twice the time of a warmup step.
The training loss and learning rate schedule are plotted in Figure14.The loss divergence soon becomes irremediable at around 120B training tokens, much shorter than the planned 4T tokens, and training has to stop there.
One possible cause is that the continual train is initialized from the latest warmup checkpoint, which is located immediately before the break down of the warmup stage, and thus is already at the brink of divergence.
The smaller learning rate of continual train delays the onset of divergence but not for long.
body
Memory3: Language Modeling with Explicit Memory
Section 5
2407.01178
6|Fine-tuning and AlignmentThis section describes our model finetuning, specifically supervised finetuning (SFT)  and direct preference optimization (DPO) .6.1|Supervised FinetuningAnalogous to the StableLM model,
our Memory3model is finetuned on a diverse collection of SFT datasets.
We use the following datasets, which are publicly accessible on the Hugging Face Hub:
UltraChat, WizardLM, SlimOrca, ShareGPT, Capybara, Deita, and MetaMathQA.
We also include synthetic data with emphasis on multi-round chat, mathematics, commonsense and knowledge.
Each training sample consists of one or more rounds of question and answer pairs.
We remove any sample with more than eight rounds.
The final composition is listed in Table15.DatasetSourceNumber of SamplesUltraChatHuggingFaceH4/ultrachat_200k194409WizardLMWizardLM/WizardLM_evol_instruct_V2_196k80662SlimOrcaOpen-Orca/SlimOrca-Dedup143789ShareGPTopenchat/openchat_sharegpt4_dataset3509CapybaraLDJnr/Capybara7291Deitahkust-nlp/deita-10k-v02860MetaMathQAmeta-math/MetaMathQA394418Multi-round Chatsynthetic20000Mathematicssynthetic20000Commonsensesynthetic150000Knowledgesynthetic270000Table 15:Composition of SFT dataset.The training process uses the cosine learning rate schedule with a max
learning rate ofand alinear warmup phase.
The weight decay is 0.1, batch size is 512, and max sequence length is 2048 tokens.
Finetuning is performed for 3 epochs.6.2|Direct Preference OptimizationThe Memory3model is further finetuned by DPO, to align with human preference and improve its conversation skills.
The DPO dataset consists of general conversations (UltraFeedback Binarized) , math questions (Distilabel Math)  and codes questions (Synth Code) .
The training uses the cosine learning rate schedule with max lr.
The inverse temperatureof the DPO loss is set to.
The improvement from DPO is displayed in Section7.2.
body
Memory3: Language Modeling with Explicit Memory
Section 6
2407.01178
7|EvaluationWe evaluate the general abilities (benchmark tasks) , conversation skills, professional abilities (law and medicine) , and facutality & hallucination of the Memory3model.
We also measure its decoding speed.
Our model is compared with SOTA LLMs of similar and larger sizes, as well as RAG models.7.1|General AbilitiesTo evaluate the general abilities of Memory3, we adopt all tasks from the Huggingface leaderboard and also include two Chinese tasks.
Most of the results are displayed in Table16, while TruthfulQA is listed in Table19.
All results are obtained in bfloat16 format, using the lm-evaluation-harness packageand the configuration of HuggingFace Open LLM leaderboard, i.e. the number of few-shot examples and grading methods.As described in Section4.4, to prevent cheating, a filtering step is included in the retrieval process so that the model cannot copy from references that resemble the evaluation questions.EnglishChineseLLMSizeAvg.ARC-CHellaSwagMMLUWinograndeGSM8kCEVALCMMLUFalcon-40B41B55.7561.8685.2856.8981.2921.4641.3842.07Llama2-7B-Chat6.5B46.8752.9078.5548.3271.747.3534.8434.40Llama2-13B-Chat13B51.7859.0481.9454.6474.5115.2438.6338.43Llama3-8B-it7.0B65.7762.0378.8965.6975.7775.8250.5251.70Vicuna-13B-v1.513B52.0257.0881.2456.6774.6611.3041.6841.53Mistral-7B-v0.17.0B59.1559.9883.3164.1678.3737.8345.9144.49Gemma-2B-it2.0B36.6438.0240.3655.7435.2955.888.2629.94Gemma-7B-it7.8B47.2351.4571.9653.5267.9632.2227.9325.70MiniCPM-2B-SFT2.4B54.3747.5371.9551.3267.7245.2648.0748.76Phi-22.5B55.7061.0975.1158.1174.3554.8134.4032.04ChatGLM3-6B5.7B54.6241.3866.9850.5464.2551.2554.0153.91Baichuan2-7B-Chat6.5B55.1652.7374.0652.7769.7728.2853.1255.38Qwen1.5-1.8B-Chat1.2B49.6738.7460.0245.8759.6733.5955.5754.22Qwen1.5-4B-Chat3.2B58.1543.2669.7355.5564.9652.2461.8959.39Qwen1.5-7B-Chat6.5B64.8056.4879.0260.5266.3854.3668.2068.67Memory3-SFT2.4B63.3158.1180.5159.6874.5152.8459.2958.24with vector compression2.4B63.3357.9480.6559.6675.1452.2459.6658.05without memory2.4B60.8057.4273.1457.2974.3551.3356.3255.72Table 16:Few-shot evaluation of general abilities.
The model sizes only include non-embedding parameters.The results of our model without using explicit memory is included, which indicates that explicit memory boosts the average score by.
In comparison, the score difference between Llama2-7B and 13B iswhile the latter has twice the amount of non-embedding parameters.
Thus, it reasonable to say that explicit memory can increase the “effective model size” by.
(Also, the score difference between Qwen-1.8B and 4B iswhile the latter hasmore non-embedding parameters.
With respect to this scale, explicit memory increases the “effective model size” by.) We also include the results of Memory3with vector compression (Section3.3) .
Even though the key-value vectors of the explicit memories are compressed toof their original sizes, the performance of our model does not show any degradation.Other supplementary evaluations can be found in AppendixC.Next, we compare with a LLM that is pretrained with text retrieval.
Specially, we consider the largest version of the Retro++ model, Retro++ XXL with 9.5B parameters.
All tasks from Table 6 ofare taken, except for HANS, which is not available on lm-eval-harness, and all tasks are zero-shot.
Similar to Table16, Memory3is tested with a filtering threshold of 2/3.
The results are listed in Table17, where Memory3outperforms the model with much larger parameter size and reference dataset size.LLMParam sizeAvg.HellaSwagBoolQLambadaRACERetro++ XXL9.1B61.070.670.772.743.2Memory3-SFT2.4B64.783.380.457.945.3Reference sizePiQAWinograndANLI-R2WiC330B77.465.835.552.414.3B76.675.841.656.9Table 17:Zero-shot comparison of LLMs pretrained with retrieval.
The scores of Retro++ are taken from.
The size of a reference dataset is its number of tokens.
The non-embedding parameter size of Retro++ is inferred from its vocabulary size.7.2|Conversation SkillNext we evaluate the conversation skills of Memory3.
We use MT-Bench (the Multi-turn Benchmark) that consists of multi-round and open-ended questions.
The results are listed inTable18, including the Memory3model finetuned by DPO introduced in Section6.2.LLMSizeMT-Bench ScorePhi-33.6B8.38Mistral-7B-Instruct-v0.27.0B7.60Qwen1.5-7B-Chat6.5B7.60Zephyr-7B-beta7.0B7.34MiniCPM-2B-DPO2.4B6.89Llama-2-70B-Chat68B6.86Mistral-7B-Instruct-v0.17.0B6.84Llama-2-13B-Chat13B6.65Llama-2-7B-Chat6.5B6.57MPT-30B-Chat30B6.39ChatGLM2-6B6.1B4.96Falcon-40B-Instruct41B4.07Vicuna-7B6.5B3.26Memory3-SFT2.4B5.31Memory3-DPO2.4B5.80Table 18:MT-Bench scores.
The model sizes only include non-embedding parameters.We obtain all these scores using GPT-4-0613 as grader, following the single answer grading mode of MT-Bench.
Our model outperforms Vicuna-7B, Falcon-40B-Instruct, and ChatGLM2-6B with fewer parameters.7.3|Hallucination and FactualityDespite considerable progress, LLMs still face issues with hallucination, leading to outputs that often stray from factual accuracy.
Conceptually, Memory3should be less vulnerable to hallucination, since its explicit memories directly correspond to reference texts, whereas compressing texts into the model parameters might incur information loss.
To evaluate hallucination, we select two English datasets, TruthfulQAand HaluEval, and one Chinese dataset, HalluQA.
TruthfulQA is implemented with lm-evaluation-harness, while HaluEval and HalluQA are implemented with UHGEval.
The results are shown in Table19, with Memory3achieving the highest scores on most tasks.EnglishChineseLLMSizeAvg.HaluE-QAHaluE-DialogueTruQA-MC1TruQA-MC2HalluQAFalcon-40B41B35.3746.8440.8027.2941.7120.18Llama2-13B13B28.0123.3431.0525.9536.8922.81Vicuna-13B-v1.513B37.0724.9337.3535.1350.88N/ABaichuan2-13B13B37.6446.0245.4526.8139.7930.12Gemma-7B7.8B37.0350.9148.1920.6946.6518.71Mistral-7B-v0.17.0B34.1840.6837.6428.0342.6021.93Llama2-7B6.5B36.8052.4651.9325.0938.9415.59Baichuan2-7B6.5B38.6362.3347.8423.0137.4622.51ChatGLM3-6B5.7B40.9643.3850.0333.1749.8728.36Qwen1.5-4B-Chat3.2B33.3024.6437.7229.3844.7430.00Phi-22.5B38.3150.7139.5531.0944.3225.89MiniCPM-SFT2.4B36.4749.2447.8024.1137.5123.71Gemma-2B2.0B38.0453.4152.2224.6039.7820.18Qwen1.5-1.8B-Chat1.2B37.5247.1852.1126.6840.5721.05Memory3-SFT2.4B48.6056.6153.9138.8057.7235.96Table 19:Evaluation of hallucination.
HaluE and TruQA denote HaluEval and TruthfulQA, respectively.
Bolded numbers are the best results.
The model sizes only include non-embedding parameters.
Vicuna-13B-v1.5 gets one N/A since that entry is near zero and seems abnormal.7.4|Professional TasksOne benefit of using explicit memory is that the LLM can easily adapt to new fields and tasks by updating its knowledge base.
One can simply import task-related references into the knowledge base of Memory3, and optionally, convert them to explicit memories in the case of warm start.
Then, the model can perform inference with this new knowledge, skipping the more costly and possibly lossy process of finetuning, and running faster than RAG.
This cost reduction has been demonstrated in Figure4and AppendixA, and could facilitate the rapid deployment of LLMs across various industries.Besides cost reduction, we need to demonstrate that Memory3can perform no worse than RAG.
We consider two professional tasks in law and medicine.
The legal task consists of multiple-choice questions from the Chinese National Judicial Examination (JEC-QA)  dataset.
The field-specific references are legal documents from the Chinese national laws and regulations database.
These references are merged with our general-purpose knowledge base (Section4.4)  for inference.The medical task consists of the medicine-related questions of C-Eval, MMLU and CMMLU, specifically from the following subsets:C-Eval: clinical medicine, basic medicineMMLU: clinical knowledge, anatomy, college medicine, college biology, nutrition, virology, medical genetics, professional medicineCMMLU: anatomy, clinical knowledge, college medicine, genetics, nutrition, traditional Chinese medicine, virologyOur knowledge base is supplemented with medical texts from the open-source medical books dataset.JEC-QAMEDLLM3 refs5 refs7 refs3 refs5 refs7 refsMemory3-2B-SFT39.3856.22MiniCPM-2B-SFT38.8337.6537.9453.7353.2952.84Gemma-2B28.1628.0625.2942.0442.4942.96Gemma-2B-it30.0431.1329.3441.7043.2442.66Llama-2-7B28.0624.7024.9045.1444.4337.96Llama-2-7B-Chat26.1825.1025.2048.1847.2939.39Phi-225.0025.3023.3250.0545.4245.59Qwen1.5-1.8B-Chat42.9843.8741.5052.1652.5052.16Qwen1.5-4B-Chat51.9850.4950.9961.1961.0261.06Table 20:Comparison with RAG on professional tasks.The results are shown in Table20, and Memory3achieves better performance than most of the models.
All evaluations use 5-shot prompting.
The RAG models retrieve from the same knowledge bases and FAISS indices, except that they receive text references instead of explicit memories.
They only retrieve once for each question, using only the question text for query, so the 5-shot examples do not distract the retrieval.
Since the optimal number of references is not known for these RAG models, we test them for 3, 5, and 7 references per question, and it seems thatreferences are optimal.
The usual formatting for RAG is used, i.e. header 1 + reference 1 + reference 2 + reference 3 + header 2 + few-shot examples + question, all separated by line breaks.The performance plotted in Figure2(right)  is the average of the scores of the two tasks in Table20with five references.7.5|Inference SpeedFinally, we evaluate the decoding speed or throughput of Memory3, measured by generated tokens per second.
The results are compared to those of RAG models, to quantify the speedup of explicit memory over text retrieval.A direct comparison of speeds is uninformative:
The memory hierarchy (Figure8)  implies that the Memory3model is more reliant on retrieval to supply knowledge, and naturally Memory3performs retrieval with higher frequency (references per 64 tokens, possibly higher in future versions) .
Therefore, it is necessary to jointly compare performance and speed.
The speed measured in this section is plotted against the retrieval-augmented test accuracy from Section7.4, resulting in Figure2(right) .We measure decoding speed on a A800 GPU, and run all models with Flash Attention.
All models receive an input of batch size 32 and length 128 tokens, and generate an output with length 128 tokens.
The throughput is computed bydivided by the time spent.
We test each model 9 times, remove the first record, and take the average of the rest.
Memory3performsretrievals (themeans that the first decoded chunk inherits the explicit memories retrieved by the last input chunk) .
Each retrieval uses 32 queries to getexplicit memories.
We consider the warm start scenario, with the explicit memories precomputed and saved to drives.
We implement the worst case scenario, such that the reference ids are reset to be unique after vector search and the memory cache on RAM is disabled, forcing Memory3to loadmemories from drives.
Meanwhile, each RAG model performs one retrieval with query length 64 tokens, receives 5 references for each sample, and inserts them at the beginning of the sample, similar to the setup for Table20.The results are listed in Table21(local server) .
The throughput of these models without retrieval is also provided.Local serverEnd-side deviceLLMSizewith retrievalw/o retrievalwith retrievalw/o retrievalMemory3-2B2.4B733.0113127.644.36MiniCPM-2B2.4B501.5974.021.751.79Gemma-2B-it2.0B1581205622.029.23Gemma-7B-it7.8B395.610089.518.61Mistral-7B-Instruct-v0.17.0B392.9894.511.128.7Llama-2-7B-Chat6.5B382.8100510.023.19Llama-2-13B-Chat13B241.1632.52.55.44Qwen1.5-1.8B-Chat1.2B908.21770--Qwen1.5-4B-Chat3.2B460.7100222.353.39Qwen1.5-7B-Chat6.5B365.8894.5--Phi-22.5B622.21544--Table 21:Inference throughput, measured by tokens per second.In addition, we study the throughput of these models when they are hosted on an end-side device and retrieve from a knowledge base on a remote server.
Specifically, we use Jetson AGX Orin, and the server uses the vector engine MyScale.
The models are run with plain attention, with batch size 1.
To simulate real-world use cases, the input is a fixed text prompt, with approximately 128 tokens, while the exact length can vary among different tokenizers.
The output length is fixed to be 128 tokens.
The results are listed in Table21(end-side device) , and the Memory3model .Remark 8.Table21indicates that our Memory3-2B model isslower than the same model without using memory.
This is peculiar considering that reading explicit memories accounts for only a tiny fraction of the total compute:(The calculations are based on AppendixA.) 
Controlled experiments indicate that the time consumption is mainly due to two sources:Loading the memory key-values from drives to GPU:
This overhead becomes prominent as Memory3retrieves with higher frequency.Python implementation of chunkwise attention:
When encoding a prompt, since each chunk attends to a different set of explicit memories, we use a for loop over the chunks to compute their attentions.They dominate other sources such as computing query vectors by the embedding model and searching the vector index.
We will try to optimize our code to reduce these overheads to be as close as possible toof the total inference time, e.g. implement the chunkwise attention with a CUDA kernel.
body
Memory3: Language Modeling with Explicit Memory
Section 7
2407.01178
8|ConclusionThe goal of this work is to reduce the cost of LLM training and inference, or equivalently, to construct a more efficient LLM that matches the performance of larger and slower LLMs.
We analyze LLMs from the new perspective of knowledge manipulation, characterizing the cost of LLMs as the transport cost of “knowledges” in and out of various memory formats.
Two causes of inefficiency are identified, namely the suboptimal placement of knowledges and the knowledge traversal problem.
We solve both problems with explicit memory, a novel memory format, along with a new training scheme and architecture.
Our preliminary experiment, the Memory3-2B model, exhibits stronger abilities and higher speed than many SOTA models with greater sizes as well as RAG models.For future work, we plan to explore the following directions:0.Efficient training with abstract knowledges: Ideally, the training cost of Memory3model should be proportional to the small amount of non-separable knowledges, approaching the learning efficiency of humans.
One approach is to filter the training data to maximize abstract knowledges and minimize specific knowledges (cf. Section3.5and Remark6) , and preferably the LLM should assess the quality of its own training data and ignore the unhelpful tokens.0.Human-like capabilities: As described in the introduction, the explicit memory allows for interesting cognitive functions such as handling infinite contexts (conversion of working memory to explicit memory) , memory consolidation (conversion of explicit memory to implicit memory) , and conscious reasoning (reflection on the memory recall process) .
These designs may further improve the efficiency and reasoning ability of Memory3.0.Compact representation of explicit memory: The explicit memory of humans can be subdivided into episodic memory, which involve particular experiences, and semantic memory, which involve general truths.
This classification is analogous to our definition of specific and abstract knowledges.
Our current implementation of explicit memory is closer to the episodic memory of humans, as each memory directly corresponds to a reference text.
To improve its reasoning ability, one can try to equip Memory3with semantic memories, e.g. obtained from induction on the episodic memories.Besides these broad topics, there are also plenty of engineering works that can be done.
For instance, an internalized retrieval process that matches sparse attention queries with memory keys (Remark2) , sparser memory heads with routing (Remark5) , memory extraction that fully preserves contexts (Remark3) , compilation of the knowledge base based on machine preference (Remark7) , reduction of the time consumption of explicit memory to be proportional to its compute overhead (Remark8) , and so on.
body
Memory3: Language Modeling with Explicit Memory
Section 8
2407.01178
AcknowledgementThis work is supported by the NSFC Major Research Plan - Interpretable and General Purpose Next-generation Artificial Intelligence of China (No. 92270001) .
We thank Prof. Zhiqin Xu, Prof. Zhouhan Lin, Fangrui Liu, Liangkai Hang, Ziyang Tao, Xiaoxing Wang, Mingze Wang, Yongqi Jin, Haotian He, Guanhua Huang, Yirong Hu for helpful discussions.
conclusion
Memory3: Language Modeling with Explicit Memory
Section 9