<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2407.09450] Human-like Episodic Memory for Infinite Context LLMs</title><meta property="og:description" content="Large language models (LLMs) have shown remarkable capabilities, but still struggle with processing extensive contexts, limiting their ability to maintain coherence and accuracy over long sequences. In contrast, the hu…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Human-like Episodic Memory for Infinite Context LLMs">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Human-like Episodic Memory for Infinite Context LLMs">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2407.09450">

<!--Generated on Mon Aug  5 18:49:19 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">Human-like Episodic Memory for Infinite Context LLMs</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Zafeirios Fountas
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Huawei Noah’s Ark Lab, London, UK
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Martin A Benfeghoul
</span><span class="ltx_author_notes"><sup id="id4.2.id1" class="ltx_sup">∗</sup>Equal Contribution.
<span class="ltx_contact ltx_role_affiliation">Huawei Noah’s Ark Lab, London, UK
</span>
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Adnan Oomerjee
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Huawei Noah’s Ark Lab, London, UK
</span>
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Fenia Christopoulou
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Huawei Noah’s Ark Lab, London, UK
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">
<br class="ltx_break"><span id="id5.1.id1" class="ltx_text ltx_font_bold">Gerasimos Lampouras</span>
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Huawei Noah’s Ark Lab, London, UK
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Haitham Bou-Ammar
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Huawei Noah’s Ark Lab, London, UK
</span>
<span class="ltx_contact ltx_role_affiliation">University College London, UK
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Jun Wang
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">University College London, UK
</span></span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id3.2" class="ltx_p"><span id="id3.2.1" class="ltx_text ltx_font_typewriter">Large language models (LLMs) have shown remarkable capabilities, but still struggle with processing extensive contexts, limiting their ability to maintain coherence and accuracy over long sequences. In contrast, the human brain excels at organising and retrieving episodic experiences across vast temporal scales, spanning a lifetime. In this work, we introduce EM-LLM, a novel approach that integrates key aspects of human episodic memory and event cognition into LLMs, enabling them to effectively handle practically infinite context lengths while maintaining computational efficiency. EM-LLM organises sequences of tokens into coherent episodic events using a combination of Bayesian surprise and graph-theoretic boundary refinement in an on-line fashion. When needed, these events are retrieved through a two-stage memory process, combining similarity-based and temporally contiguous retrieval for efficient and human-like access to relevant information. Experiments on the LongBench dataset demonstrate EM-LLM’s superior performance, outperforming the state-of-the-art InfLLM model with an overall relative improvement of </span><math id="id2.1.m1.1" class="ltx_Math" alttext="4.3\%" display="inline"><semantics id="id2.1.m1.1a"><mrow id="id2.1.m1.1.1" xref="id2.1.m1.1.1.cmml"><mn id="id2.1.m1.1.1.2" xref="id2.1.m1.1.1.2.cmml">4.3</mn><mo id="id2.1.m1.1.1.1" xref="id2.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="id2.1.m1.1b"><apply id="id2.1.m1.1.1.cmml" xref="id2.1.m1.1.1"><csymbol cd="latexml" id="id2.1.m1.1.1.1.cmml" xref="id2.1.m1.1.1.1">percent</csymbol><cn type="float" id="id2.1.m1.1.1.2.cmml" xref="id2.1.m1.1.1.2">4.3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="id2.1.m1.1c">4.3\%</annotation></semantics></math><span id="id3.2.2" class="ltx_text ltx_font_typewriter"> across various tasks, including a </span><math id="id3.2.m2.1" class="ltx_Math" alttext="33\%" display="inline"><semantics id="id3.2.m2.1a"><mrow id="id3.2.m2.1.1" xref="id3.2.m2.1.1.cmml"><mn id="id3.2.m2.1.1.2" xref="id3.2.m2.1.1.2.cmml">33</mn><mo id="id3.2.m2.1.1.1" xref="id3.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="id3.2.m2.1b"><apply id="id3.2.m2.1.1.cmml" xref="id3.2.m2.1.1"><csymbol cd="latexml" id="id3.2.m2.1.1.1.cmml" xref="id3.2.m2.1.1.1">percent</csymbol><cn type="integer" id="id3.2.m2.1.1.2.cmml" xref="id3.2.m2.1.1.2">33</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="id3.2.m2.1c">33\%</annotation></semantics></math><span id="id3.2.3" class="ltx_text ltx_font_typewriter"> improvement on the PassageRetrieval task. Furthermore, our analysis reveals strong correlations between EM-LLM’s event segmentation and human-perceived events, suggesting a bridge between this artificial system and its biological counterpart. This work not only advances LLM capabilities in processing extended contexts but also provides a computational framework for exploring human memory mechanisms, opening new avenues for interdisciplinary research in AI and cognitive science.</span></p>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_font_typewriter ltx_title_section">
<span class="ltx_tag ltx_tag_section"><span id="S1.1.1.1" class="ltx_text ltx_font_serif">1</span> </span>Introduction</h2>

<div id="S1.p1" class="ltx_para ltx_noindent">
<p id="S1.p1.1" class="ltx_p"><span id="S1.p1.1.1" class="ltx_text ltx_font_typewriter">For contemporary pre-trained large language models (LLMs), the context window serves as the primary mechanism to incorporate domain-specific, private, or common up-to-date information. However, despite their remarkable and ever-expanding capabilities, LLMs still exhibit significant limitations when tasked with processing extensive contexts </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S1.p1.1.2.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Liu et al.</span><span id="S1.p1.1.3.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib1" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024a</span></a><span id="S1.p1.1.4.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S1.p1.1.5" class="ltx_text ltx_font_typewriter">. These limitations stem from inherent challenges in Transformer-based architectures. Recent studies have shown that Transformers struggle with extrapolating to contexts longer than their training window size </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S1.p1.1.6.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Kazemnejad et al.</span><span id="S1.p1.1.7.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib2" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024</span></a><span id="S1.p1.1.8.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S1.p1.1.9" class="ltx_text ltx_font_typewriter">. On top of this, employing softmax attention over extended token sequences requires substantial computational resources for each token generation, and the resulting attention embeddings risk becoming excessively noisy and losing their distinctiveness </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S1.p1.1.10.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Tworkowski et al.</span><span id="S1.p1.1.11.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib3" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2023</span></a><span id="S1.p1.1.12.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S1.p1.1.13" class="ltx_text ltx_font_typewriter">.</span></p>
</div>
<div id="S1.p2" class="ltx_para ltx_noindent">
<p id="S1.p2.1" class="ltx_p"><span id="S1.p2.1.1" class="ltx_text ltx_font_typewriter">To mitigate those challenges, recent works have focused on retrieval-based methods, either in the form of in-context augmentation (e.g., RAG-based techniques </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S1.p2.1.2.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Lewis et al.</span><span id="S1.p2.1.3.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib4" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2020</span></a>, <span class="ltx_text ltx_font_typewriter">Gao et al.</span><span id="S1.p2.1.3.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib5" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024</span></a><span id="S1.p2.1.4.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S1.p2.1.5" class="ltx_text ltx_font_typewriter">) or via retrieval of previously-inferred key-value pairs (KV) within individual attention heads </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S1.p2.1.6.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Wu et al.</span><span id="S1.p2.1.7.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib6" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2022</span></a>, <span class="ltx_text ltx_font_typewriter">Tworkowski et al.</span><span id="S1.p2.1.7.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib3" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2023</span></a>, <span class="ltx_text ltx_font_typewriter">Bertsch et al.</span><span id="S1.p2.1.7.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib7" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2023</span></a><span id="S1.p2.1.8.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S1.p2.1.9" class="ltx_text ltx_font_typewriter">. Notably, state-of-the-art performance is achieved when KV pairs are initially organised into non-overlapping segments and then retrieved together as one block of sequential tokens </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S1.p2.1.10.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Xiao et al.</span><span id="S1.p2.1.11.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib8" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024a</span></a><span id="S1.p2.1.12.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S1.p2.1.13" class="ltx_text ltx_font_typewriter">. While such techniques present interesting avenues of research, results still indicate a significant gap between the performance of LLMs in short- vs long-context tasks, even when existing long-context architectures are employed </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S1.p2.1.14.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Liu et al.</span><span id="S1.p2.1.15.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib1" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024a</span></a><span id="S1.p2.1.16.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S1.p2.1.17" class="ltx_text ltx_font_typewriter">.</span></p>
</div>
<div id="S1.p3" class="ltx_para ltx_noindent">
<p id="S1.p3.1" class="ltx_p"><span id="S1.p3.1.1" class="ltx_text ltx_font_typewriter">This work tackles the above challenges and attempts to bridge this performance gap by taking inspiration from the algorithmic interpretation of </span><em id="S1.p3.1.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">episodic memory in the human brain</em><span id="S1.p3.1.3" class="ltx_text ltx_font_typewriter"> -- the memory system responsible for encoding, storing, and retrieving personal experiences and events. The brain makes sense of its continuous experience in the real world by segmenting it into discrete episodic events </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S1.p3.1.4.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Clewett et al.</span><span id="S1.p3.1.5.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib9" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2019</span></a>, <span class="ltx_text ltx_font_typewriter">Zacks</span><span id="S1.p3.1.5.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib10" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2020</span></a><span id="S1.p3.1.6.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S1.p3.1.7" class="ltx_text ltx_font_typewriter">, which are organised in a hierarchical and nested-timescale structure </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S1.p3.1.8.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Baldassano et al.</span><span id="S1.p3.1.9.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib11" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2017</span></a><span id="S1.p3.1.10.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S1.p3.1.11" class="ltx_text ltx_font_typewriter"> and stored in long-term memory. Notably, the boundaries between such events are the access points when it comes to memory retrieval </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S1.p3.1.12.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Michelmann et al.</span><span id="S1.p3.1.13.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib12" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2023a</span></a><span id="S1.p3.1.14.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S1.p3.1.15" class="ltx_text ltx_font_typewriter"> and are widely believed to correspond to points in time with high prediction errors between the brain’s generative model and its raw sensory input (a.k.a., </span><em id="S1.p3.1.16" class="ltx_emph ltx_font_typewriter ltx_font_italic">surprise</em><span id="S1.p3.1.17" class="ltx_text ltx_font_typewriter">). In this context, surprise refers to moments when the brain’s predictions about incoming sensory information are significantly violated, leading to a mismatch between what is expected and what is actually perceived. These instances of high surprise are thought to signal important changes in the environment or narrative, prompting the brain to segment the ongoing experience into distinct events </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S1.p3.1.18.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Zacks et al.</span><span id="S1.p3.1.19.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib13" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2007</span></a>, <a href="#bib.bib14" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2011</span></a>, <span class="ltx_text ltx_font_typewriter">Roseboom et al.</span><span id="S1.p3.1.19.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib15" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2019</span></a>, <span class="ltx_text ltx_font_typewriter">Sinclair et al.</span><span id="S1.p3.1.19.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib16" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2021</span></a>, <span class="ltx_text ltx_font_typewriter">Fountas et al.</span><span id="S1.p3.1.19.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib17" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2022</span></a><span id="S1.p3.1.20.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S1.p3.1.21" class="ltx_text ltx_font_typewriter">. Once segmented and stored, the brain can recall episodic memories based on their similarity to its current experience, recency, original temporal order, and their proximity to other recalled memories (temporal asymmetry and contiguity, </span><cite class="ltx_cite ltx_citemacro_citep"><span class="ltx_text ltx_font_typewriter">Howard and Kahana</span><span id="S1.p3.1.22.1.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib18" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2002</span></a></cite><span id="S1.p3.1.23" class="ltx_text ltx_font_typewriter">).</span></p>
</div>
<div id="S1.p4" class="ltx_para ltx_noindent">
<p id="S1.p4.3" class="ltx_p"><span id="S1.p4.3.1" class="ltx_text ltx_font_typewriter">Following these insights, we propose a novel architecture, </span><span id="S1.p4.3.2" class="ltx_text ltx_font_italic">EM-LLM</span><span id="S1.p4.3.3" class="ltx_text ltx_font_typewriter">, that integrates crucial aspects of event cognition and episodic memory into Transformer-based LLMs. For memory formation, we segment the sequence of the tokens presented to the underlying LLM into individual memory units representing episodic events. The boundaries, and thus the size of those events, are initially determined dynamically, based on the level of surprise of the model during inference, and then refined to maximise cohesion within memory units and separation of memory content across them (see Section </span><a href="#S3.SS2" title="3.2 Memory formation via Surprise ‣ 3 EM-LLM: LLM with Episodic Memory ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">3.2</span></a><span id="S1.p4.3.4" class="ltx_text ltx_font_typewriter">). This refinement process leverages graph-theoretic metrics, treating the similarity between attention keys (the learned representations used in Transformer self-attention mechanisms) as a weighted adjacency matrix, and aims to enhance the model’s ability to efficiently recall relevant information when addressing complex tasks with extended contexts.
Importantly, this memory formation process incurs minimal additional computational cost, with the surprise-based segmentation requiring no extra computation and the refinement step having a complexity of </span><math id="S1.p4.1.m1.1" class="ltx_Math" alttext="\mathcal{O}(kn)" display="inline"><semantics id="S1.p4.1.m1.1a"><mrow id="S1.p4.1.m1.1.1" xref="S1.p4.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S1.p4.1.m1.1.1.3" xref="S1.p4.1.m1.1.1.3.cmml">𝒪</mi><mo lspace="0em" rspace="0em" id="S1.p4.1.m1.1.1.2" xref="S1.p4.1.m1.1.1.2.cmml">​</mo><mrow id="S1.p4.1.m1.1.1.1.1" xref="S1.p4.1.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S1.p4.1.m1.1.1.1.1.2" xref="S1.p4.1.m1.1.1.1.1.1.cmml">(</mo><mrow id="S1.p4.1.m1.1.1.1.1.1" xref="S1.p4.1.m1.1.1.1.1.1.cmml"><mi id="S1.p4.1.m1.1.1.1.1.1.2" xref="S1.p4.1.m1.1.1.1.1.1.2.cmml">k</mi><mo lspace="0em" rspace="0em" id="S1.p4.1.m1.1.1.1.1.1.1" xref="S1.p4.1.m1.1.1.1.1.1.1.cmml">​</mo><mi id="S1.p4.1.m1.1.1.1.1.1.3" xref="S1.p4.1.m1.1.1.1.1.1.3.cmml">n</mi></mrow><mo stretchy="false" id="S1.p4.1.m1.1.1.1.1.3" xref="S1.p4.1.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.p4.1.m1.1b"><apply id="S1.p4.1.m1.1.1.cmml" xref="S1.p4.1.m1.1.1"><times id="S1.p4.1.m1.1.1.2.cmml" xref="S1.p4.1.m1.1.1.2"></times><ci id="S1.p4.1.m1.1.1.3.cmml" xref="S1.p4.1.m1.1.1.3">𝒪</ci><apply id="S1.p4.1.m1.1.1.1.1.1.cmml" xref="S1.p4.1.m1.1.1.1.1"><times id="S1.p4.1.m1.1.1.1.1.1.1.cmml" xref="S1.p4.1.m1.1.1.1.1.1.1"></times><ci id="S1.p4.1.m1.1.1.1.1.1.2.cmml" xref="S1.p4.1.m1.1.1.1.1.1.2">𝑘</ci><ci id="S1.p4.1.m1.1.1.1.1.1.3.cmml" xref="S1.p4.1.m1.1.1.1.1.1.3">𝑛</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p4.1.m1.1c">\mathcal{O}(kn)</annotation></semantics></math><span id="S1.p4.3.5" class="ltx_text ltx_font_typewriter">, where </span><math id="S1.p4.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S1.p4.2.m2.1a"><mi id="S1.p4.2.m2.1.1" xref="S1.p4.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S1.p4.2.m2.1b"><ci id="S1.p4.2.m2.1.1.cmml" xref="S1.p4.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p4.2.m2.1c">k</annotation></semantics></math><span id="S1.p4.3.6" class="ltx_text ltx_font_typewriter"> is typically very small compared to the number of tokens </span><math id="S1.p4.3.m3.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S1.p4.3.m3.1a"><mi id="S1.p4.3.m3.1.1" xref="S1.p4.3.m3.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S1.p4.3.m3.1b"><ci id="S1.p4.3.m3.1.1.cmml" xref="S1.p4.3.m3.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p4.3.m3.1c">n</annotation></semantics></math><span id="S1.p4.3.7" class="ltx_text ltx_font_typewriter">.
With this efficient memory formation process, by grouping similar information in single units, we minimise the number of units needed to recall details around specific events.
For memory recall, our approach integrates similarity-based retrieval with mechanisms that facilitate temporal contiguity and asymmetry effects. By retrieving and buffering salient memory units, our model leverages and enhances the recently discovered propensity of LLMs to exhibit human-like patterns in sequential information retrieval </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S1.p4.3.8.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Ji-An et al.</span><span id="S1.p4.3.9.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib19" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024</span></a><span id="S1.p4.3.10.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S1.p4.3.11" class="ltx_text ltx_font_typewriter">. This method not only ensures efficient access to pertinent information but also mimics the temporal dynamics found in human free recall studies (such as </span><cite class="ltx_cite ltx_citemacro_citep"><span class="ltx_text ltx_font_typewriter">Howard and Kahana</span><span id="S1.p4.3.12.1.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib18" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2002</span></a></cite><span id="S1.p4.3.13" class="ltx_text ltx_font_typewriter">), further enhancing the model’s ability to handle complex tasks that require nuanced temporal reasoning.</span></p>
</div>
<div id="S1.p5" class="ltx_para ltx_noindent">
<p id="S1.p5.2" class="ltx_p"><span id="S1.p5.2.1" class="ltx_text ltx_font_typewriter">To prove our hypotheses, we first employ a series of human-annotated podcast scripts, where we show that information in LLM attention heads can be semantically grouped in a way that correlates with the event structure perceived by humans. Therefore, LLM-perceived surprise can indeed serve as a proxy for the cognitive signals that drive human event segmentation, as confirmed by previous works </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S1.p5.2.2.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Kumar et al.</span><span id="S1.p5.2.3.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib20" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2023</span></a><span id="S1.p5.2.4.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S1.p5.2.5" class="ltx_text ltx_font_typewriter">. Then, using the long-context PG-19 dataset </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S1.p5.2.6.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Rae et al.</span><span id="S1.p5.2.7.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib21" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2020</span></a><span id="S1.p5.2.8.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S1.p5.2.9" class="ltx_text ltx_font_typewriter">, which comprises a diverse corpus of English books, we evaluate the effectiveness of both steps in our segmentation method for grouping relevant information, and assess the performance of different refinement objectives. Finally, we show that our method is scalable and significantly outperforms the state-of-the-art model InfLLM </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S1.p5.2.10.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Xiao et al.</span><span id="S1.p5.2.11.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib8" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024a</span></a><span id="S1.p5.2.12.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S1.p5.2.13" class="ltx_text ltx_font_typewriter"> on the widely-used LongBench benchmark </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S1.p5.2.14.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Bai et al.</span><span id="S1.p5.2.15.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib22" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2023</span></a><span id="S1.p5.2.16.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S1.p5.2.17" class="ltx_text ltx_font_typewriter"> for long-context tasks, achieving an overall relative improvement of </span><math id="S1.p5.1.m1.1" class="ltx_Math" alttext="4.3\%" display="inline"><semantics id="S1.p5.1.m1.1a"><mrow id="S1.p5.1.m1.1.1" xref="S1.p5.1.m1.1.1.cmml"><mn id="S1.p5.1.m1.1.1.2" xref="S1.p5.1.m1.1.1.2.cmml">4.3</mn><mo id="S1.p5.1.m1.1.1.1" xref="S1.p5.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S1.p5.1.m1.1b"><apply id="S1.p5.1.m1.1.1.cmml" xref="S1.p5.1.m1.1.1"><csymbol cd="latexml" id="S1.p5.1.m1.1.1.1.cmml" xref="S1.p5.1.m1.1.1.1">percent</csymbol><cn type="float" id="S1.p5.1.m1.1.1.2.cmml" xref="S1.p5.1.m1.1.1.2">4.3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p5.1.m1.1c">4.3\%</annotation></semantics></math><span id="S1.p5.2.18" class="ltx_text ltx_font_typewriter">, including a substantial </span><math id="S1.p5.2.m2.1" class="ltx_Math" alttext="33\%" display="inline"><semantics id="S1.p5.2.m2.1a"><mrow id="S1.p5.2.m2.1.1" xref="S1.p5.2.m2.1.1.cmml"><mn id="S1.p5.2.m2.1.1.2" xref="S1.p5.2.m2.1.1.2.cmml">33</mn><mo id="S1.p5.2.m2.1.1.1" xref="S1.p5.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S1.p5.2.m2.1b"><apply id="S1.p5.2.m2.1.1.cmml" xref="S1.p5.2.m2.1.1"><csymbol cd="latexml" id="S1.p5.2.m2.1.1.1.cmml" xref="S1.p5.2.m2.1.1.1">percent</csymbol><cn type="integer" id="S1.p5.2.m2.1.1.2.cmml" xref="S1.p5.2.m2.1.1.2">33</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p5.2.m2.1c">33\%</annotation></semantics></math><span id="S1.p5.2.19" class="ltx_text ltx_font_typewriter"> improvement on the PassageRetrieval task.</span></p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_font_typewriter ltx_title_section">
<span class="ltx_tag ltx_tag_section"><span id="S2.1.1.1" class="ltx_text ltx_font_serif">2</span> </span>Related work</h2>

<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_font_typewriter ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="S2.SS1.1.1.1" class="ltx_text ltx_font_serif">2.1</span> </span>Long-context in LLMs</h3>

<div id="S2.SS1.p1" class="ltx_para ltx_noindent">
<p id="S2.SS1.p1.1" class="ltx_p"><span id="S2.SS1.p1.1.1" class="ltx_text ltx_font_typewriter">Recently, several approaches have been proposed to extend the context window of Transformer-based models. Those include methods that address the limited representational capacity of softmax attention, and its quadratic computational and memory cost </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S2.SS1.p1.1.2.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Katharopoulos et al.</span><span id="S2.SS1.p1.1.3.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib23" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2020</span></a>, <span class="ltx_text ltx_font_typewriter">Munkhdalai et al.</span><span id="S2.SS1.p1.1.3.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib24" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024</span></a><span id="S2.SS1.p1.1.4.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS1.p1.1.5" class="ltx_text ltx_font_typewriter">. Other methods target the poor extrapolation of typical positional encodings to out-of-distribution (OOD) context lengths </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S2.SS1.p1.1.6.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Kazemnejad et al.</span><span id="S2.SS1.p1.1.7.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib2" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024</span></a><span id="S2.SS1.p1.1.8.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS1.p1.1.9" class="ltx_text ltx_font_typewriter">. The latter is evident in most widely used methods, including the original absolute positional encodings </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S2.SS1.p1.1.10.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Vaswani et al.</span><span id="S2.SS1.p1.1.11.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib25" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2017</span></a><span id="S2.SS1.p1.1.12.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS1.p1.1.13" class="ltx_text ltx_font_typewriter"> and the more recent relative positional encodings, such as the popular Rotary Positional Embeddings (RoPE) </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S2.SS1.p1.1.14.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Su et al.</span><span id="S2.SS1.p1.1.15.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib26" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024</span></a><span id="S2.SS1.p1.1.16.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS1.p1.1.17" class="ltx_text ltx_font_typewriter">. To address this, some approaches propose scaling of the rotation angles </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S2.SS1.p1.1.18.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Chen et al.</span><span id="S2.SS1.p1.1.19.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib27" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024a</span></a><span id="S2.SS1.p1.1.20.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS1.p1.1.21" class="ltx_text ltx_font_typewriter"> or the base constant </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S2.SS1.p1.1.22.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Xiong et al.</span><span id="S2.SS1.p1.1.23.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib28" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2023</span></a>, <span class="ltx_text ltx_font_typewriter">Liu et al.</span><span id="S2.SS1.p1.1.23.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib29" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024b</span></a>, <span class="ltx_text ltx_font_typewriter">Peng et al.</span><span id="S2.SS1.p1.1.23.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib30" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024</span></a>, <span class="ltx_text ltx_font_typewriter">Ding et al.</span><span id="S2.SS1.p1.1.23.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib31" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024</span></a><span id="S2.SS1.p1.1.24.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS1.p1.1.25" class="ltx_text ltx_font_typewriter">. Others, scale positions without affecting the embedding function </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S2.SS1.p1.1.26.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Press et al.</span><span id="S2.SS1.p1.1.27.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib32" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2021</span></a>, <span class="ltx_text ltx_font_typewriter">Chen et al.</span><span id="S2.SS1.p1.1.27.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib33" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2023</span></a>, <span class="ltx_text ltx_font_typewriter">Jin et al.</span><span id="S2.SS1.p1.1.27.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib34" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024</span></a><span id="S2.SS1.p1.1.28.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS1.p1.1.29" class="ltx_text ltx_font_typewriter">, exploring alternative strategies such as KERPLE </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S2.SS1.p1.1.30.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Chi et al.</span><span id="S2.SS1.p1.1.31.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib35" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2022</span></a><span id="S2.SS1.p1.1.32.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS1.p1.1.33" class="ltx_text ltx_font_typewriter"> and FIRE </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S2.SS1.p1.1.34.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Li et al.</span><span id="S2.SS1.p1.1.35.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib36" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024</span></a><span id="S2.SS1.p1.1.36.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS1.p1.1.37" class="ltx_text ltx_font_typewriter"> or adopt mechanisms from certain LMs like T5 </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S2.SS1.p1.1.38.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Raffel et al.</span><span id="S2.SS1.p1.1.39.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib37" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2020</span></a><span id="S2.SS1.p1.1.40.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS1.p1.1.41" class="ltx_text ltx_font_typewriter">.</span></p>
</div>
<div id="S2.SS1.p2" class="ltx_para ltx_noindent">
<p id="S2.SS1.p2.1" class="ltx_p"><span id="S2.SS1.p2.1.1" class="ltx_text ltx_font_typewriter">Concerning computational efficiency and diluted attention, successful approaches propose methods for general improvements to the efficiency of Transformers through optimised computations </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S2.SS1.p2.1.2.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Dao</span><span id="S2.SS1.p2.1.3.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib38" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024</span></a>, <span class="ltx_text ltx_font_typewriter">Han et al.</span><span id="S2.SS1.p2.1.3.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib39" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024a</span></a>, <span class="ltx_text ltx_font_typewriter">Aminabadi et al.</span><span id="S2.SS1.p2.1.3.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib40" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2022</span></a>, <span class="ltx_text ltx_font_typewriter">Kwon et al.</span><span id="S2.SS1.p2.1.3.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib41" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2023</span></a>, <span class="ltx_text ltx_font_typewriter">Liu et al.</span><span id="S2.SS1.p2.1.3.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib42" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024c</span></a>, <span class="ltx_text ltx_font_typewriter">Brandon et al.</span><span id="S2.SS1.p2.1.3.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib43" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2023</span></a><span id="S2.SS1.p2.1.4.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS1.p2.1.5" class="ltx_text ltx_font_typewriter"> or compression techniques </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S2.SS1.p2.1.6.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Nawrot et al.</span><span id="S2.SS1.p2.1.7.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib44" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024</span></a>, <span class="ltx_text ltx_font_typewriter">Zhang et al.</span><span id="S2.SS1.p2.1.7.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib45" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2023</span></a><span id="S2.SS1.p2.1.8.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS1.p2.1.9" class="ltx_text ltx_font_typewriter">, as well as training methods tailored for long-context scenarios </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S2.SS1.p2.1.10.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Zhu et al.</span><span id="S2.SS1.p2.1.11.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib46" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024</span></a>, <span class="ltx_text ltx_font_typewriter">Chen et al.</span><span id="S2.SS1.p2.1.11.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib47" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024b</span></a><span id="S2.SS1.p2.1.12.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS1.p2.1.13" class="ltx_text ltx_font_typewriter">. Another direction is the utilisation of retrieval-based methods, the vast majority of which relies on a vector database that keeps a key-value cache and scalable approximations of k-nearest neighbors (k-NNs) to perform lookups </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S2.SS1.p2.1.14.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Wu et al.</span><span id="S2.SS1.p2.1.15.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib6" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2022</span></a>, <span class="ltx_text ltx_font_typewriter">Tworkowski et al.</span><span id="S2.SS1.p2.1.15.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib3" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2023</span></a>, <span class="ltx_text ltx_font_typewriter">Bertsch et al.</span><span id="S2.SS1.p2.1.15.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib7" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2023</span></a><span id="S2.SS1.p2.1.16.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS1.p2.1.17" class="ltx_text ltx_font_typewriter">. Interestingly, since using a key-value cache with k-NN lookup can be seen as an approximation of applying softmax attention to the full token sequence (see Appendix </span><a href="#A1.SS4" title="A.4 Approximate equivalence of K-nearest neighbours and softmax attention ‣ Appendix A Appendix / supplemental material ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">A.4</span></a><span id="S2.SS1.p2.1.18" class="ltx_text ltx_font_typewriter">), k-NN retrieval methods can be used without any fine-tuning </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S2.SS1.p2.1.19.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Bertsch et al.</span><span id="S2.SS1.p2.1.20.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib7" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2023</span></a><span id="S2.SS1.p2.1.21.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS1.p2.1.22" class="ltx_text ltx_font_typewriter">. For an exception that does not rely on k-NNs, see </span><cite class="ltx_cite ltx_citemacro_cite"><span class="ltx_text ltx_font_typewriter">Wang et al.</span> <span id="S2.SS1.p2.1.23.1.1.1" class="ltx_text ltx_font_typewriter">[</span><a href="#bib.bib48" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2023</span></a><span id="S2.SS1.p2.1.24.2.2.1" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS1.p2.1.25" class="ltx_text ltx_font_typewriter">.</span></p>
</div>
<div id="S2.SS1.p3" class="ltx_para ltx_noindent">
<p id="S2.SS1.p3.1" class="ltx_p"><span id="S2.SS1.p3.1.1" class="ltx_text ltx_font_typewriter">A recent and interesting variant of k-NN retrieval involves retrieving large groups of tokens, rather than individual ones. Models that rely on this approach include SLED </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S2.SS1.p3.1.2.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Ivgi et al.</span><span id="S2.SS1.p3.1.3.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib49" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2023</span></a><span id="S2.SS1.p3.1.4.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS1.p3.1.5" class="ltx_text ltx_font_typewriter"> and the more recent InfLLM </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S2.SS1.p3.1.6.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Xiao et al.</span><span id="S2.SS1.p3.1.7.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib8" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024a</span></a><span id="S2.SS1.p3.1.8.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS1.p3.1.9" class="ltx_text ltx_font_typewriter">, which achieves state-of-the-art performance on long-context benchmarks. InfLLM segments the entire context length into fixed-size memory units and employs k-NN lookup using the tokens with the highest accumulated scores per unit. This can be seen as a form of hierarchical attention, as illustrated in Fig. </span><a href="#S2.F1" title="Figure 1 ‣ 2.1 Long-context in LLMs ‣ 2 Related work ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">1</span></a><span id="S2.SS1.p3.1.10" class="ltx_text ltx_font_typewriter">. While group-based retrieval represents a promising direction, our approach significantly advances this concept by dynamically determining token groupings in a manner akin to human memory formation, addressing a fundamental limitation of InfLLM’s fixed-size segmentation and enabling more adaptive and context-sensitive processing of extended information.</span></p>
</div>
<figure id="S2.F1" class="ltx_figure"><img src="/html/2407.09450/assets/x1.png" id="S2.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="77" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering ltx_font_typewriter"><span class="ltx_tag ltx_tag_figure"><span id="S2.F1.9.1.1" class="ltx_text ltx_font_serif">Figure 1</span>: </span>Group-based <math id="S2.F1.3.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S2.F1.3.m1.1b"><mi id="S2.F1.3.m1.1.1" xref="S2.F1.3.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S2.F1.3.m1.1c"><ci id="S2.F1.3.m1.1.1.cmml" xref="S2.F1.3.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.F1.3.m1.1d">k</annotation></semantics></math>-NN retrieval can be seen as a form of hierarchical episodic attention. Initially, <math id="S2.F1.4.m2.1" class="ltx_Math" alttext="k=4" display="inline"><semantics id="S2.F1.4.m2.1b"><mrow id="S2.F1.4.m2.1.1" xref="S2.F1.4.m2.1.1.cmml"><mi id="S2.F1.4.m2.1.1.2" xref="S2.F1.4.m2.1.1.2.cmml">k</mi><mo id="S2.F1.4.m2.1.1.1" xref="S2.F1.4.m2.1.1.1.cmml">=</mo><mn id="S2.F1.4.m2.1.1.3" xref="S2.F1.4.m2.1.1.3.cmml">4</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.F1.4.m2.1c"><apply id="S2.F1.4.m2.1.1.cmml" xref="S2.F1.4.m2.1.1"><eq id="S2.F1.4.m2.1.1.1.cmml" xref="S2.F1.4.m2.1.1.1"></eq><ci id="S2.F1.4.m2.1.1.2.cmml" xref="S2.F1.4.m2.1.1.2">𝑘</ci><cn type="integer" id="S2.F1.4.m2.1.1.3.cmml" xref="S2.F1.4.m2.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.F1.4.m2.1d">k=4</annotation></semantics></math> groups of tokens are selected (left) and then used for softmax attention (right), as if all other similarity scores were forced to be zero (non-shaded areas of the left curve). This framework can support multiple levels of episodic attention.</figcaption>
</figure>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_font_typewriter ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="S2.SS2.1.1.1" class="ltx_text ltx_font_serif">2.2</span> </span>Neural models of Episodic Memory and Event Cognition</h3>

<div id="S2.SS2.p1" class="ltx_para ltx_noindent">
<p id="S2.SS2.p1.1" class="ltx_p"><span id="S2.SS2.p1.1.1" class="ltx_text ltx_font_typewriter">The concept of episodic memory, central to our approach, has been extensively studied in both theoretical neuroscience and machine learning. Neural models of episodic memory capture human behaviour and neuroimaging data, providing insights into how the brain processes and stores experiences and suggesting links between memory, efficient representations and navigation of physical and conceptual spaces </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S2.SS2.p1.1.2.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Gershman et al.</span><span id="S2.SS2.p1.1.3.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib50" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2012</span></a>, <span class="ltx_text ltx_font_typewriter">Benna and Fusi</span><span id="S2.SS2.p1.1.3.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib51" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2021</span></a><span id="S2.SS2.p1.1.4.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS2.p1.1.5" class="ltx_text ltx_font_typewriter">.
In machine learning, episodic memory-inspired approaches have yielded significant improvements across various domains. For instance, episodic control has enhanced reinforcement learning agents’ performance and learning speed </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S2.SS2.p1.1.6.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Blundell et al.</span><span id="S2.SS2.p1.1.7.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib52" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2016</span></a>, <span class="ltx_text ltx_font_typewriter">Pritzel et al.</span><span id="S2.SS2.p1.1.7.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib53" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2017</span></a>, <span class="ltx_text ltx_font_typewriter">Coda-Forno et al.</span><span id="S2.SS2.p1.1.7.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib54" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024</span></a><span id="S2.SS2.p1.1.8.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS2.p1.1.9" class="ltx_text ltx_font_typewriter">. In addition, models of memory construction and consolidation have been successful in alleviating catastrophic forgetting in neural networks </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S2.SS2.p1.1.10.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Kirkpatrick et al.</span><span id="S2.SS2.p1.1.11.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib55" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2017</span></a>, <span class="ltx_text ltx_font_typewriter">Lopez-Paz and Ranzato</span><span id="S2.SS2.p1.1.11.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib56" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2017</span></a>, <span class="ltx_text ltx_font_typewriter">Chaudhry et al.</span><span id="S2.SS2.p1.1.11.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib57" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2019</span></a>, <span class="ltx_text ltx_font_typewriter">Buzzega et al.</span><span id="S2.SS2.p1.1.11.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib58" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2020</span></a>, <span class="ltx_text ltx_font_typewriter">Prabhu et al.</span><span id="S2.SS2.p1.1.11.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib59" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2020</span></a><span id="S2.SS2.p1.1.12.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS2.p1.1.13" class="ltx_text ltx_font_typewriter">, including LLMs </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S2.SS2.p1.1.14.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Das et al.</span><span id="S2.SS2.p1.1.15.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib60" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024</span></a><span id="S2.SS2.p1.1.16.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS2.p1.1.17" class="ltx_text ltx_font_typewriter">, and appear to explain key features of human memory, such as imagination and future thinking </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S2.SS2.p1.1.18.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Spens and Burgess</span><span id="S2.SS2.p1.1.19.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib61" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024</span></a><span id="S2.SS2.p1.1.20.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS2.p1.1.21" class="ltx_text ltx_font_typewriter">.</span></p>
</div>
<div id="S2.SS2.p2" class="ltx_para ltx_noindent">
<p id="S2.SS2.p2.1" class="ltx_p"><span id="S2.SS2.p2.1.1" class="ltx_text ltx_font_typewriter">These models have revealed key aspects of episodic memory, particularly in describing how experiences are segmented into events, and when new memories are encoded and retrieved </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S2.SS2.p2.1.2.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Lu et al.</span><span id="S2.SS2.p2.1.3.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib62" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2022</span></a><span id="S2.SS2.p2.1.4.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS2.p2.1.5" class="ltx_text ltx_font_typewriter">. Surprise plays a critical role in this process, triggering event boundaries and memory formation </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S2.SS2.p2.1.6.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Fountas et al.</span><span id="S2.SS2.p2.1.7.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib17" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2022</span></a>, <span class="ltx_text ltx_font_typewriter">Kumar et al.</span><span id="S2.SS2.p2.1.7.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib20" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2023</span></a><span id="S2.SS2.p2.1.8.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS2.p2.1.9" class="ltx_text ltx_font_typewriter">. This event-based structure is deeply intertwined with our perception of time </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S2.SS2.p2.1.10.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Roseboom et al.</span><span id="S2.SS2.p2.1.11.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib15" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2019</span></a>, <span class="ltx_text ltx_font_typewriter">Sherman et al.</span><span id="S2.SS2.p2.1.11.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib63" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2022</span></a><span id="S2.SS2.p2.1.12.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS2.p2.1.13" class="ltx_text ltx_font_typewriter">, highlighting the interdependence of memory and temporal cognition. This insight has helped generative models for video </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S2.SS2.p2.1.14.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Zakharov et al.</span><span id="S2.SS2.p2.1.15.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib64" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2022a</span></a>, <a href="#bib.bib65" title="" class="ltx_ref">b</a><span id="S2.SS2.p2.1.16.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS2.p2.1.17" class="ltx_text ltx_font_typewriter"> and reinforcement learning </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S2.SS2.p2.1.18.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Zakharov et al.</span><span id="S2.SS2.p2.1.19.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib66" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2021</span></a><span id="S2.SS2.p2.1.20.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS2.p2.1.21" class="ltx_text ltx_font_typewriter"> to capture temporal dynamics more accurately. In terms of memory retrieval, studies in human free recall have shown a distinctive increased likelihood of retrieving items encoded close together in time (temporal contiguity) and in succession (temporal asymmetry) (see Fig.</span><a href="#S2.F2" title="Figure 2 ‣ 2.2 Neural models of Episodic Memory and Event Cognition ‣ 2 Related work ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">2</span></a><span id="S2.SS2.p2.1.22" class="ltx_text ltx_font_typewriter">A). Recently, it was shown that attention heads in transformer-based LLMs that are associated with in-context learning, already exhibit the same dynamic retrieval behaviour </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S2.SS2.p2.1.23.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Ji-An et al.</span><span id="S2.SS2.p2.1.24.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib19" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024</span></a><span id="S2.SS2.p2.1.25.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S2.SS2.p2.1.26" class="ltx_text ltx_font_typewriter"> (Fig.</span><a href="#S2.F2" title="Figure 2 ‣ 2.2 Neural models of Episodic Memory and Event Cognition ‣ 2 Related work ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">2</span></a><span id="S2.SS2.p2.1.27" class="ltx_text ltx_font_typewriter">B) including both contiguity and asymmetry effects. Therefore, transformers have the inherent ability to act as episodic memory retrieval models, if provided with the right information within their context window. Our work leverages these concepts of surprise-based event segmentation and LLMs’ inherent temporal contiguity and asymmetry effects to enable a new generation of Infinite Context-Length LLMs, capable of processing and understanding information over vastly extended timescales.</span></p>
</div>
<figure id="S2.F2" class="ltx_figure"><img src="/html/2407.09450/assets/x2.png" id="S2.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="184" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering ltx_font_typewriter"><span class="ltx_tag ltx_tag_figure"><span id="S2.F2.14.1.1" class="ltx_text ltx_font_serif">Figure 2</span>: </span>(<span id="S2.F2.15.2" class="ltx_text ltx_font_serif ltx_font_bold">A</span>) Example of the temporal contiguity and asymmetry effect in human free recall. Data averaged over several large free recall studies (adopted from <cite class="ltx_cite ltx_citemacro_citep"><span class="ltx_text ltx_font_typewriter">Howard and Kahana</span>, <a href="#bib.bib18" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2002</span></a></cite>). (<span id="S2.F2.16.3" class="ltx_text ltx_font_serif ltx_font_bold">B</span>) The attention scores of a GPT2 head averaged over all tokens tested. Figure adopted from <cite class="ltx_cite ltx_citemacro_cite"><span class="ltx_text ltx_font_typewriter">Ji-An et al.</span> [<a href="#bib.bib19" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024</span></a>]</cite>. (<span id="S2.F2.17.4" class="ltx_text ltx_font_serif ltx_font_bold">C</span>) Schematic illustrating our proposed process for memory formation and retrieval: ① Input sequence with surprise-based segmentation (purple arrows indicate high surprise). ② Formation of episodic memories: input is segmented into events and stored, with initial tokens and local context preserved. Note that the boundary refinement process is not shown here for clarity. ③ Memory retrieval via k-NN search, selecting contiguous events from episodic memory. ④ Final context window structure, comprising initial tokens, contiguity buffer (populated by neighboring events), similarity buffer (from k-NN retrieval), and local context.</figcaption>
</figure>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_font_typewriter ltx_title_section">
<span class="ltx_tag ltx_tag_section"><span id="S3.1.1.1" class="ltx_text ltx_font_serif">3</span> </span>EM-LLM: LLM with Episodic Memory</h2>

<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_font_typewriter ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="S3.SS1.1.1.1" class="ltx_text ltx_font_serif">3.1</span> </span>Architecture</h3>

<div id="S3.SS1.p1" class="ltx_para ltx_noindent">
<p id="S3.SS1.p1.1" class="ltx_p"><span id="S3.SS1.p1.1.1" class="ltx_text ltx_font_typewriter">EM-LLM is designed to be applied directly to pre-trained LLMs, enabling them to handle context lengths significantly larger than their original training length. Our architecture divides the context into three distinct groups: initial tokens, evicted tokens, and local context. This structure, while incorporating insights from recent work on token block retrieval </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S3.SS1.p1.1.2.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Xiao et al.</span><span id="S3.SS1.p1.1.3.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib8" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024a</span></a><span id="S3.SS1.p1.1.4.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S3.SS1.p1.1.5" class="ltx_text ltx_font_typewriter">, introduces novel elements inspired by human episodic memory.</span></p>
</div>
<div id="S3.SS1.p2" class="ltx_para ltx_noindent">
<p id="S3.SS1.p2.1" class="ltx_p"><span id="S3.SS1.p2.1.1" class="ltx_text ltx_font_typewriter">The local context represents the most recent tokens, maximising information about the current task, and fits within the typical context window of the underlying LLM. This group utilises full softmax attention and plays a role similar to the focus of attention in cognitive models of working memory, holding the most immediately relevant information for the current task </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S3.SS1.p2.1.2.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Cowan</span><span id="S3.SS1.p2.1.3.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib67" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2001</span></a><span id="S3.SS1.p2.1.4.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S3.SS1.p2.1.5" class="ltx_text ltx_font_typewriter">. The evicted tokens typically comprise the majority of past tokens in a long-context scenario, extending far beyond the LLM’s original training length. These tokens are managed by our proposed memory model functioning similarly to short-term episodic memory in the brain. Finally, following previous work, we also maintain a group of </span><math id="S3.SS1.p2.1.m1.1" class="ltx_Math" alttext="128" display="inline"><semantics id="S3.SS1.p2.1.m1.1a"><mn id="S3.SS1.p2.1.m1.1.1" xref="S3.SS1.p2.1.m1.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.1.m1.1b"><cn type="integer" id="S3.SS1.p2.1.m1.1.1.cmml" xref="S3.SS1.p2.1.m1.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.1.m1.1c">128</annotation></semantics></math><span id="S3.SS1.p2.1.6" class="ltx_text ltx_font_typewriter"> initial tokens in the LLM context. These act as </span><span id="S3.SS1.p2.1.7" class="ltx_text ltx_font_italic">attention sinks</span><span id="S3.SS1.p2.1.8" class="ltx_text ltx_font_typewriter"> and help recover the performance of window attention, as first observed by </span><cite class="ltx_cite ltx_citemacro_cite"><span class="ltx_text ltx_font_typewriter">Xiao et al.</span> <span id="S3.SS1.p2.1.9.1.1.1" class="ltx_text ltx_font_typewriter">[</span><a href="#bib.bib68" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024b</span></a><span id="S3.SS1.p2.1.10.2.2.1" class="ltx_text ltx_font_typewriter">]</span>, <span class="ltx_text ltx_font_typewriter">Han et al.</span> <span id="S3.SS1.p2.1.9.1.1.1" class="ltx_text ltx_font_typewriter">[</span><a href="#bib.bib69" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024b</span></a><span id="S3.SS1.p2.1.10.2.2.1" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S3.SS1.p2.1.11" class="ltx_text ltx_font_typewriter"> and later adopted by </span><cite class="ltx_cite ltx_citemacro_cite"><span class="ltx_text ltx_font_typewriter">Xiao et al.</span> <span id="S3.SS1.p2.1.12.1.1.1" class="ltx_text ltx_font_typewriter">[</span><a href="#bib.bib8" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024a</span></a><span id="S3.SS1.p2.1.13.2.2.1" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S3.SS1.p2.1.14" class="ltx_text ltx_font_typewriter">. For retrieved tokens, which are therefore discontinuous and outside the local context, we assign a fixed position embedding as in </span><cite class="ltx_cite ltx_citemacro_citet"><span class="ltx_text ltx_font_typewriter">Raffel et al.</span> <span id="S3.SS1.p2.1.15.1.1.1" class="ltx_text ltx_font_typewriter">[</span><a href="#bib.bib37" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2020</span></a><span id="S3.SS1.p2.1.16.2.2.1" class="ltx_text ltx_font_typewriter">]</span>, <span class="ltx_text ltx_font_typewriter">Xiao et al.</span> <span id="S3.SS1.p2.1.15.1.1.1" class="ltx_text ltx_font_typewriter">[</span><a href="#bib.bib8" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024a</span></a><span id="S3.SS1.p2.1.16.2.2.1" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S3.SS1.p2.1.17" class="ltx_text ltx_font_typewriter">. This architecture enables EM-LLM to effectively process and utilise information from positions outside its pre-trained local context window, while maintaining the underlying LLM’s performance characteristics.</span></p>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_font_typewriter ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="S3.SS2.1.1.1" class="ltx_text ltx_font_serif">3.2</span> </span>Memory formation via Surprise</h3>

<div id="S3.SS2.p1" class="ltx_para ltx_noindent">
<p id="S3.SS2.p1.2" class="ltx_p"><span id="S3.SS2.p1.2.1" class="ltx_text ltx_font_typewriter">In the context of LLMs, we define episodic memory as the organised, event-based collection of past key-value pairs, analogous to the latent representations of personal experiences in human memory. Just as unexpected or novel information plays a crucial role in human memory formation, we posit that analogous indicators of novelty in LLMs can serve as an effective proxy for identifying significant ‘‘events’’ within the model’s experience. In Bayesian terms, surprise is quantified by the negative log-likelihood of observing the current, ground-truth token given the previous tokens in an auto-regressive model, with high values indicating the unpredictability or novelty of each new token within the context according to the model, i.e., it is ‘‘surprised’’ by the next token.
Following work on cognitive modelling </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S3.SS2.p1.2.2.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Roseboom et al.</span><span id="S3.SS2.p1.2.3.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib15" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2019</span></a>, <span class="ltx_text ltx_font_typewriter">Fountas et al.</span><span id="S3.SS2.p1.2.3.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib17" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2022</span></a><span id="S3.SS2.p1.2.4.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S3.SS2.p1.2.5" class="ltx_text ltx_font_typewriter">, we employ a thresholding mechanism to perform an initial identification of event boundaries (used for the first time in LLMs). Formally, a token </span><math id="S3.SS2.p1.1.m1.1" class="ltx_Math" alttext="x_{t}" display="inline"><semantics id="S3.SS2.p1.1.m1.1a"><msub id="S3.SS2.p1.1.m1.1.1" xref="S3.SS2.p1.1.m1.1.1.cmml"><mi id="S3.SS2.p1.1.m1.1.1.2" xref="S3.SS2.p1.1.m1.1.1.2.cmml">x</mi><mi id="S3.SS2.p1.1.m1.1.1.3" xref="S3.SS2.p1.1.m1.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.1.m1.1b"><apply id="S3.SS2.p1.1.m1.1.1.cmml" xref="S3.SS2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.1.m1.1.1.1.cmml" xref="S3.SS2.p1.1.m1.1.1">subscript</csymbol><ci id="S3.SS2.p1.1.m1.1.1.2.cmml" xref="S3.SS2.p1.1.m1.1.1.2">𝑥</ci><ci id="S3.SS2.p1.1.m1.1.1.3.cmml" xref="S3.SS2.p1.1.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.1.m1.1c">x_{t}</annotation></semantics></math><span id="S3.SS2.p1.2.6" class="ltx_text ltx_font_typewriter"> is considered a potential boundary if its surprise value exceeds a threshold </span><math id="S3.SS2.p1.2.m2.1" class="ltx_Math" alttext="T" display="inline"><semantics id="S3.SS2.p1.2.m2.1a"><mi id="S3.SS2.p1.2.m2.1.1" xref="S3.SS2.p1.2.m2.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.2.m2.1b"><ci id="S3.SS2.p1.2.m2.1.1.cmml" xref="S3.SS2.p1.2.m2.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.2.m2.1c">T</annotation></semantics></math><span id="S3.SS2.p1.2.7" class="ltx_text ltx_font_typewriter">:</span></p>
<table id="A1.EGx1" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S3.E1"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E1.m1.6" class="ltx_Math" alttext="\displaystyle-\log P(x_{t}|x_{1},\ldots,x_{t-1};\theta)&gt;T\quad~{}~{}\text{with}\quad~{}~{}T=\mu_{t-\tau}+\gamma\sigma_{t-\tau}" display="inline"><semantics id="S3.E1.m1.6a"><mrow id="S3.E1.m1.6.6.2" xref="S3.E1.m1.6.6.3.cmml"><mrow id="S3.E1.m1.5.5.1.1" xref="S3.E1.m1.5.5.1.1.cmml"><mrow id="S3.E1.m1.5.5.1.1.1" xref="S3.E1.m1.5.5.1.1.1.cmml"><mo rspace="0.167em" id="S3.E1.m1.5.5.1.1.1a" xref="S3.E1.m1.5.5.1.1.1.cmml">−</mo><mrow id="S3.E1.m1.5.5.1.1.1.1" xref="S3.E1.m1.5.5.1.1.1.1.cmml"><mrow id="S3.E1.m1.5.5.1.1.1.1.3" xref="S3.E1.m1.5.5.1.1.1.1.3.cmml"><mi id="S3.E1.m1.5.5.1.1.1.1.3.1" xref="S3.E1.m1.5.5.1.1.1.1.3.1.cmml">log</mi><mo lspace="0.167em" id="S3.E1.m1.5.5.1.1.1.1.3a" xref="S3.E1.m1.5.5.1.1.1.1.3.cmml">⁡</mo><mi id="S3.E1.m1.5.5.1.1.1.1.3.2" xref="S3.E1.m1.5.5.1.1.1.1.3.2.cmml">P</mi></mrow><mo lspace="0em" rspace="0em" id="S3.E1.m1.5.5.1.1.1.1.2" xref="S3.E1.m1.5.5.1.1.1.1.2.cmml">​</mo><mrow id="S3.E1.m1.5.5.1.1.1.1.1.1" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E1.m1.5.5.1.1.1.1.1.1.2" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E1.m1.5.5.1.1.1.1.1.1.1" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.cmml"><msub id="S3.E1.m1.5.5.1.1.1.1.1.1.1.4" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.4.cmml"><mi id="S3.E1.m1.5.5.1.1.1.1.1.1.1.4.2" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.4.2.cmml">x</mi><mi id="S3.E1.m1.5.5.1.1.1.1.1.1.1.4.3" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.4.3.cmml">t</mi></msub><mo fence="false" id="S3.E1.m1.5.5.1.1.1.1.1.1.1.3" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.3.cmml">|</mo><mrow id="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.3.cmml"><msub id="S3.E1.m1.5.5.1.1.1.1.1.1.1.1.1.1" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.E1.m1.5.5.1.1.1.1.1.1.1.1.1.1.2" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.1.1.1.2.cmml">x</mi><mn id="S3.E1.m1.5.5.1.1.1.1.1.1.1.1.1.1.3" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.1.1.1.3.cmml">1</mn></msub><mo id="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.3" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.3.cmml">,</mo><mi mathvariant="normal" id="S3.E1.m1.1.1" xref="S3.E1.m1.1.1.cmml">…</mi><mo id="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.4" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.3.cmml">,</mo><msub id="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.2" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.2.cmml"><mi id="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.2.2" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.2.2.cmml">x</mi><mrow id="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.2.3" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.2.3.cmml"><mi id="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.2.3.2" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.2.3.2.cmml">t</mi><mo id="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.2.3.1" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.2.3.1.cmml">−</mo><mn id="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.2.3.3" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.2.3.3.cmml">1</mn></mrow></msub><mo id="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.5" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.3.cmml">;</mo><mi id="S3.E1.m1.2.2" xref="S3.E1.m1.2.2.cmml">θ</mi></mrow></mrow><mo stretchy="false" id="S3.E1.m1.5.5.1.1.1.1.1.1.3" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo id="S3.E1.m1.5.5.1.1.2" xref="S3.E1.m1.5.5.1.1.2.cmml">&gt;</mo><mrow id="S3.E1.m1.5.5.1.1.3.2" xref="S3.E1.m1.5.5.1.1.3.1.cmml"><mi id="S3.E1.m1.3.3" xref="S3.E1.m1.3.3.cmml">T</mi><mspace width="1.66em" id="S3.E1.m1.5.5.1.1.3.2.1" xref="S3.E1.m1.5.5.1.1.3.1.cmml"></mspace><mtext class="ltx_mathvariant_monospace" id="S3.E1.m1.4.4" xref="S3.E1.m1.4.4a.cmml">with</mtext></mrow></mrow><mspace width="1.66em" id="S3.E1.m1.6.6.2.3" xref="S3.E1.m1.6.6.3a.cmml"></mspace><mrow id="S3.E1.m1.6.6.2.2" xref="S3.E1.m1.6.6.2.2.cmml"><mi id="S3.E1.m1.6.6.2.2.2" xref="S3.E1.m1.6.6.2.2.2.cmml">T</mi><mo id="S3.E1.m1.6.6.2.2.1" xref="S3.E1.m1.6.6.2.2.1.cmml">=</mo><mrow id="S3.E1.m1.6.6.2.2.3" xref="S3.E1.m1.6.6.2.2.3.cmml"><msub id="S3.E1.m1.6.6.2.2.3.2" xref="S3.E1.m1.6.6.2.2.3.2.cmml"><mi id="S3.E1.m1.6.6.2.2.3.2.2" xref="S3.E1.m1.6.6.2.2.3.2.2.cmml">μ</mi><mrow id="S3.E1.m1.6.6.2.2.3.2.3" xref="S3.E1.m1.6.6.2.2.3.2.3.cmml"><mi id="S3.E1.m1.6.6.2.2.3.2.3.2" xref="S3.E1.m1.6.6.2.2.3.2.3.2.cmml">t</mi><mo id="S3.E1.m1.6.6.2.2.3.2.3.1" xref="S3.E1.m1.6.6.2.2.3.2.3.1.cmml">−</mo><mi id="S3.E1.m1.6.6.2.2.3.2.3.3" xref="S3.E1.m1.6.6.2.2.3.2.3.3.cmml">τ</mi></mrow></msub><mo id="S3.E1.m1.6.6.2.2.3.1" xref="S3.E1.m1.6.6.2.2.3.1.cmml">+</mo><mrow id="S3.E1.m1.6.6.2.2.3.3" xref="S3.E1.m1.6.6.2.2.3.3.cmml"><mi id="S3.E1.m1.6.6.2.2.3.3.2" xref="S3.E1.m1.6.6.2.2.3.3.2.cmml">γ</mi><mo lspace="0em" rspace="0em" id="S3.E1.m1.6.6.2.2.3.3.1" xref="S3.E1.m1.6.6.2.2.3.3.1.cmml">​</mo><msub id="S3.E1.m1.6.6.2.2.3.3.3" xref="S3.E1.m1.6.6.2.2.3.3.3.cmml"><mi id="S3.E1.m1.6.6.2.2.3.3.3.2" xref="S3.E1.m1.6.6.2.2.3.3.3.2.cmml">σ</mi><mrow id="S3.E1.m1.6.6.2.2.3.3.3.3" xref="S3.E1.m1.6.6.2.2.3.3.3.3.cmml"><mi id="S3.E1.m1.6.6.2.2.3.3.3.3.2" xref="S3.E1.m1.6.6.2.2.3.3.3.3.2.cmml">t</mi><mo id="S3.E1.m1.6.6.2.2.3.3.3.3.1" xref="S3.E1.m1.6.6.2.2.3.3.3.3.1.cmml">−</mo><mi id="S3.E1.m1.6.6.2.2.3.3.3.3.3" xref="S3.E1.m1.6.6.2.2.3.3.3.3.3.cmml">τ</mi></mrow></msub></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E1.m1.6b"><apply id="S3.E1.m1.6.6.3.cmml" xref="S3.E1.m1.6.6.2"><csymbol cd="ambiguous" id="S3.E1.m1.6.6.3a.cmml" xref="S3.E1.m1.6.6.2.3">formulae-sequence</csymbol><apply id="S3.E1.m1.5.5.1.1.cmml" xref="S3.E1.m1.5.5.1.1"><gt id="S3.E1.m1.5.5.1.1.2.cmml" xref="S3.E1.m1.5.5.1.1.2"></gt><apply id="S3.E1.m1.5.5.1.1.1.cmml" xref="S3.E1.m1.5.5.1.1.1"><minus id="S3.E1.m1.5.5.1.1.1.2.cmml" xref="S3.E1.m1.5.5.1.1.1"></minus><apply id="S3.E1.m1.5.5.1.1.1.1.cmml" xref="S3.E1.m1.5.5.1.1.1.1"><times id="S3.E1.m1.5.5.1.1.1.1.2.cmml" xref="S3.E1.m1.5.5.1.1.1.1.2"></times><apply id="S3.E1.m1.5.5.1.1.1.1.3.cmml" xref="S3.E1.m1.5.5.1.1.1.1.3"><log id="S3.E1.m1.5.5.1.1.1.1.3.1.cmml" xref="S3.E1.m1.5.5.1.1.1.1.3.1"></log><ci id="S3.E1.m1.5.5.1.1.1.1.3.2.cmml" xref="S3.E1.m1.5.5.1.1.1.1.3.2">𝑃</ci></apply><apply id="S3.E1.m1.5.5.1.1.1.1.1.1.1.cmml" xref="S3.E1.m1.5.5.1.1.1.1.1.1"><csymbol cd="latexml" id="S3.E1.m1.5.5.1.1.1.1.1.1.1.3.cmml" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.3">conditional</csymbol><apply id="S3.E1.m1.5.5.1.1.1.1.1.1.1.4.cmml" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.4"><csymbol cd="ambiguous" id="S3.E1.m1.5.5.1.1.1.1.1.1.1.4.1.cmml" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.4">subscript</csymbol><ci id="S3.E1.m1.5.5.1.1.1.1.1.1.1.4.2.cmml" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.4.2">𝑥</ci><ci id="S3.E1.m1.5.5.1.1.1.1.1.1.1.4.3.cmml" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.4.3">𝑡</ci></apply><list id="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.3.cmml" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2"><apply id="S3.E1.m1.5.5.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E1.m1.5.5.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.E1.m1.5.5.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.1.1.1.2">𝑥</ci><cn type="integer" id="S3.E1.m1.5.5.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.1.1.1.3">1</cn></apply><ci id="S3.E1.m1.1.1.cmml" xref="S3.E1.m1.1.1">…</ci><apply id="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.2.cmml" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.2"><csymbol cd="ambiguous" id="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.2.1.cmml" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.2">subscript</csymbol><ci id="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.2.2.cmml" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.2.2">𝑥</ci><apply id="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.2.3.cmml" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.2.3"><minus id="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.2.3.1.cmml" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.2.3.1"></minus><ci id="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.2.3.2.cmml" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.2.3.2">𝑡</ci><cn type="integer" id="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.2.3.3.cmml" xref="S3.E1.m1.5.5.1.1.1.1.1.1.1.2.2.2.3.3">1</cn></apply></apply><ci id="S3.E1.m1.2.2.cmml" xref="S3.E1.m1.2.2">𝜃</ci></list></apply></apply></apply><list id="S3.E1.m1.5.5.1.1.3.1.cmml" xref="S3.E1.m1.5.5.1.1.3.2"><ci id="S3.E1.m1.3.3.cmml" xref="S3.E1.m1.3.3">𝑇</ci><ci id="S3.E1.m1.4.4a.cmml" xref="S3.E1.m1.4.4"><mtext class="ltx_mathvariant_monospace" id="S3.E1.m1.4.4.cmml" xref="S3.E1.m1.4.4">with</mtext></ci></list></apply><apply id="S3.E1.m1.6.6.2.2.cmml" xref="S3.E1.m1.6.6.2.2"><eq id="S3.E1.m1.6.6.2.2.1.cmml" xref="S3.E1.m1.6.6.2.2.1"></eq><ci id="S3.E1.m1.6.6.2.2.2.cmml" xref="S3.E1.m1.6.6.2.2.2">𝑇</ci><apply id="S3.E1.m1.6.6.2.2.3.cmml" xref="S3.E1.m1.6.6.2.2.3"><plus id="S3.E1.m1.6.6.2.2.3.1.cmml" xref="S3.E1.m1.6.6.2.2.3.1"></plus><apply id="S3.E1.m1.6.6.2.2.3.2.cmml" xref="S3.E1.m1.6.6.2.2.3.2"><csymbol cd="ambiguous" id="S3.E1.m1.6.6.2.2.3.2.1.cmml" xref="S3.E1.m1.6.6.2.2.3.2">subscript</csymbol><ci id="S3.E1.m1.6.6.2.2.3.2.2.cmml" xref="S3.E1.m1.6.6.2.2.3.2.2">𝜇</ci><apply id="S3.E1.m1.6.6.2.2.3.2.3.cmml" xref="S3.E1.m1.6.6.2.2.3.2.3"><minus id="S3.E1.m1.6.6.2.2.3.2.3.1.cmml" xref="S3.E1.m1.6.6.2.2.3.2.3.1"></minus><ci id="S3.E1.m1.6.6.2.2.3.2.3.2.cmml" xref="S3.E1.m1.6.6.2.2.3.2.3.2">𝑡</ci><ci id="S3.E1.m1.6.6.2.2.3.2.3.3.cmml" xref="S3.E1.m1.6.6.2.2.3.2.3.3">𝜏</ci></apply></apply><apply id="S3.E1.m1.6.6.2.2.3.3.cmml" xref="S3.E1.m1.6.6.2.2.3.3"><times id="S3.E1.m1.6.6.2.2.3.3.1.cmml" xref="S3.E1.m1.6.6.2.2.3.3.1"></times><ci id="S3.E1.m1.6.6.2.2.3.3.2.cmml" xref="S3.E1.m1.6.6.2.2.3.3.2">𝛾</ci><apply id="S3.E1.m1.6.6.2.2.3.3.3.cmml" xref="S3.E1.m1.6.6.2.2.3.3.3"><csymbol cd="ambiguous" id="S3.E1.m1.6.6.2.2.3.3.3.1.cmml" xref="S3.E1.m1.6.6.2.2.3.3.3">subscript</csymbol><ci id="S3.E1.m1.6.6.2.2.3.3.3.2.cmml" xref="S3.E1.m1.6.6.2.2.3.3.3.2">𝜎</ci><apply id="S3.E1.m1.6.6.2.2.3.3.3.3.cmml" xref="S3.E1.m1.6.6.2.2.3.3.3.3"><minus id="S3.E1.m1.6.6.2.2.3.3.3.3.1.cmml" xref="S3.E1.m1.6.6.2.2.3.3.3.3.1"></minus><ci id="S3.E1.m1.6.6.2.2.3.3.3.3.2.cmml" xref="S3.E1.m1.6.6.2.2.3.3.3.3.2">𝑡</ci><ci id="S3.E1.m1.6.6.2.2.3.3.3.3.3.cmml" xref="S3.E1.m1.6.6.2.2.3.3.3.3.3">𝜏</ci></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E1.m1.6c">\displaystyle-\log P(x_{t}|x_{1},\ldots,x_{t-1};\theta)&gt;T\quad~{}~{}\text{with}\quad~{}~{}T=\mu_{t-\tau}+\gamma\sigma_{t-\tau}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
<p id="S3.SS2.p1.14" class="ltx_p"><span id="S3.SS2.p1.14.1" class="ltx_text ltx_font_typewriter">where </span><math id="S3.SS2.p1.3.m1.1" class="ltx_Math" alttext="\mu_{t-\tau:t}" display="inline"><semantics id="S3.SS2.p1.3.m1.1a"><msub id="S3.SS2.p1.3.m1.1.1" xref="S3.SS2.p1.3.m1.1.1.cmml"><mi id="S3.SS2.p1.3.m1.1.1.2" xref="S3.SS2.p1.3.m1.1.1.2.cmml">μ</mi><mrow id="S3.SS2.p1.3.m1.1.1.3" xref="S3.SS2.p1.3.m1.1.1.3.cmml"><mrow id="S3.SS2.p1.3.m1.1.1.3.2" xref="S3.SS2.p1.3.m1.1.1.3.2.cmml"><mi id="S3.SS2.p1.3.m1.1.1.3.2.2" xref="S3.SS2.p1.3.m1.1.1.3.2.2.cmml">t</mi><mo id="S3.SS2.p1.3.m1.1.1.3.2.1" xref="S3.SS2.p1.3.m1.1.1.3.2.1.cmml">−</mo><mi id="S3.SS2.p1.3.m1.1.1.3.2.3" xref="S3.SS2.p1.3.m1.1.1.3.2.3.cmml">τ</mi></mrow><mo lspace="0.278em" rspace="0.278em" id="S3.SS2.p1.3.m1.1.1.3.1" xref="S3.SS2.p1.3.m1.1.1.3.1.cmml">:</mo><mi id="S3.SS2.p1.3.m1.1.1.3.3" xref="S3.SS2.p1.3.m1.1.1.3.3.cmml">t</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.3.m1.1b"><apply id="S3.SS2.p1.3.m1.1.1.cmml" xref="S3.SS2.p1.3.m1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.3.m1.1.1.1.cmml" xref="S3.SS2.p1.3.m1.1.1">subscript</csymbol><ci id="S3.SS2.p1.3.m1.1.1.2.cmml" xref="S3.SS2.p1.3.m1.1.1.2">𝜇</ci><apply id="S3.SS2.p1.3.m1.1.1.3.cmml" xref="S3.SS2.p1.3.m1.1.1.3"><ci id="S3.SS2.p1.3.m1.1.1.3.1.cmml" xref="S3.SS2.p1.3.m1.1.1.3.1">:</ci><apply id="S3.SS2.p1.3.m1.1.1.3.2.cmml" xref="S3.SS2.p1.3.m1.1.1.3.2"><minus id="S3.SS2.p1.3.m1.1.1.3.2.1.cmml" xref="S3.SS2.p1.3.m1.1.1.3.2.1"></minus><ci id="S3.SS2.p1.3.m1.1.1.3.2.2.cmml" xref="S3.SS2.p1.3.m1.1.1.3.2.2">𝑡</ci><ci id="S3.SS2.p1.3.m1.1.1.3.2.3.cmml" xref="S3.SS2.p1.3.m1.1.1.3.2.3">𝜏</ci></apply><ci id="S3.SS2.p1.3.m1.1.1.3.3.cmml" xref="S3.SS2.p1.3.m1.1.1.3.3">𝑡</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.3.m1.1c">\mu_{t-\tau:t}</annotation></semantics></math><span id="S3.SS2.p1.14.2" class="ltx_text ltx_font_typewriter"> and </span><math id="S3.SS2.p1.4.m2.1" class="ltx_Math" alttext="\sigma_{t-\tau:t}^{2}" display="inline"><semantics id="S3.SS2.p1.4.m2.1a"><msubsup id="S3.SS2.p1.4.m2.1.1" xref="S3.SS2.p1.4.m2.1.1.cmml"><mi id="S3.SS2.p1.4.m2.1.1.2.2" xref="S3.SS2.p1.4.m2.1.1.2.2.cmml">σ</mi><mrow id="S3.SS2.p1.4.m2.1.1.2.3" xref="S3.SS2.p1.4.m2.1.1.2.3.cmml"><mrow id="S3.SS2.p1.4.m2.1.1.2.3.2" xref="S3.SS2.p1.4.m2.1.1.2.3.2.cmml"><mi id="S3.SS2.p1.4.m2.1.1.2.3.2.2" xref="S3.SS2.p1.4.m2.1.1.2.3.2.2.cmml">t</mi><mo id="S3.SS2.p1.4.m2.1.1.2.3.2.1" xref="S3.SS2.p1.4.m2.1.1.2.3.2.1.cmml">−</mo><mi id="S3.SS2.p1.4.m2.1.1.2.3.2.3" xref="S3.SS2.p1.4.m2.1.1.2.3.2.3.cmml">τ</mi></mrow><mo lspace="0.278em" rspace="0.278em" id="S3.SS2.p1.4.m2.1.1.2.3.1" xref="S3.SS2.p1.4.m2.1.1.2.3.1.cmml">:</mo><mi id="S3.SS2.p1.4.m2.1.1.2.3.3" xref="S3.SS2.p1.4.m2.1.1.2.3.3.cmml">t</mi></mrow><mn id="S3.SS2.p1.4.m2.1.1.3" xref="S3.SS2.p1.4.m2.1.1.3.cmml">2</mn></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.4.m2.1b"><apply id="S3.SS2.p1.4.m2.1.1.cmml" xref="S3.SS2.p1.4.m2.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.4.m2.1.1.1.cmml" xref="S3.SS2.p1.4.m2.1.1">superscript</csymbol><apply id="S3.SS2.p1.4.m2.1.1.2.cmml" xref="S3.SS2.p1.4.m2.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.4.m2.1.1.2.1.cmml" xref="S3.SS2.p1.4.m2.1.1">subscript</csymbol><ci id="S3.SS2.p1.4.m2.1.1.2.2.cmml" xref="S3.SS2.p1.4.m2.1.1.2.2">𝜎</ci><apply id="S3.SS2.p1.4.m2.1.1.2.3.cmml" xref="S3.SS2.p1.4.m2.1.1.2.3"><ci id="S3.SS2.p1.4.m2.1.1.2.3.1.cmml" xref="S3.SS2.p1.4.m2.1.1.2.3.1">:</ci><apply id="S3.SS2.p1.4.m2.1.1.2.3.2.cmml" xref="S3.SS2.p1.4.m2.1.1.2.3.2"><minus id="S3.SS2.p1.4.m2.1.1.2.3.2.1.cmml" xref="S3.SS2.p1.4.m2.1.1.2.3.2.1"></minus><ci id="S3.SS2.p1.4.m2.1.1.2.3.2.2.cmml" xref="S3.SS2.p1.4.m2.1.1.2.3.2.2">𝑡</ci><ci id="S3.SS2.p1.4.m2.1.1.2.3.2.3.cmml" xref="S3.SS2.p1.4.m2.1.1.2.3.2.3">𝜏</ci></apply><ci id="S3.SS2.p1.4.m2.1.1.2.3.3.cmml" xref="S3.SS2.p1.4.m2.1.1.2.3.3">𝑡</ci></apply></apply><cn type="integer" id="S3.SS2.p1.4.m2.1.1.3.cmml" xref="S3.SS2.p1.4.m2.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.4.m2.1c">\sigma_{t-\tau:t}^{2}</annotation></semantics></math><span id="S3.SS2.p1.14.3" class="ltx_text ltx_font_typewriter"> are the mean and variance of surprise for a window offset </span><math id="S3.SS2.p1.5.m3.1" class="ltx_Math" alttext="\tau" display="inline"><semantics id="S3.SS2.p1.5.m3.1a"><mi id="S3.SS2.p1.5.m3.1.1" xref="S3.SS2.p1.5.m3.1.1.cmml">τ</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.5.m3.1b"><ci id="S3.SS2.p1.5.m3.1.1.cmml" xref="S3.SS2.p1.5.m3.1.1">𝜏</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.5.m3.1c">\tau</annotation></semantics></math><span id="S3.SS2.p1.14.4" class="ltx_text ltx_font_typewriter">, and </span><math id="S3.SS2.p1.6.m4.1" class="ltx_Math" alttext="\gamma" display="inline"><semantics id="S3.SS2.p1.6.m4.1a"><mi id="S3.SS2.p1.6.m4.1.1" xref="S3.SS2.p1.6.m4.1.1.cmml">γ</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.6.m4.1b"><ci id="S3.SS2.p1.6.m4.1.1.cmml" xref="S3.SS2.p1.6.m4.1.1">𝛾</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.6.m4.1c">\gamma</annotation></semantics></math><span id="S3.SS2.p1.14.5" class="ltx_text ltx_font_typewriter"> is a scaling factor. The choice of threshold </span><math id="S3.SS2.p1.7.m5.1" class="ltx_Math" alttext="T" display="inline"><semantics id="S3.SS2.p1.7.m5.1a"><mi id="S3.SS2.p1.7.m5.1.1" xref="S3.SS2.p1.7.m5.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.7.m5.1b"><ci id="S3.SS2.p1.7.m5.1.1.cmml" xref="S3.SS2.p1.7.m5.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.7.m5.1c">T</annotation></semantics></math><span id="S3.SS2.p1.14.6" class="ltx_text ltx_font_typewriter"> is critical in balancing the granularity of segmentation with the model’s sensitivity to contextual shifts. If the </span><math id="S3.SS2.p1.8.m6.1" class="ltx_Math" alttext="T" display="inline"><semantics id="S3.SS2.p1.8.m6.1a"><mi id="S3.SS2.p1.8.m6.1.1" xref="S3.SS2.p1.8.m6.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.8.m6.1b"><ci id="S3.SS2.p1.8.m6.1.1.cmml" xref="S3.SS2.p1.8.m6.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.8.m6.1c">T</annotation></semantics></math><span id="S3.SS2.p1.14.7" class="ltx_text ltx_font_typewriter"> is too high, we will identify very few event boundaries, especially if the local context contains few surprising tokens. Conversely, a low </span><math id="S3.SS2.p1.9.m7.1" class="ltx_Math" alttext="T" display="inline"><semantics id="S3.SS2.p1.9.m7.1a"><mi id="S3.SS2.p1.9.m7.1.1" xref="S3.SS2.p1.9.m7.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.9.m7.1b"><ci id="S3.SS2.p1.9.m7.1.1.cmml" xref="S3.SS2.p1.9.m7.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.9.m7.1c">T</annotation></semantics></math><span id="S3.SS2.p1.14.8" class="ltx_text ltx_font_typewriter"> results in frequent boundary identification. Using a moving window ensures that </span><math id="S3.SS2.p1.10.m8.1" class="ltx_Math" alttext="T" display="inline"><semantics id="S3.SS2.p1.10.m8.1a"><mi id="S3.SS2.p1.10.m8.1.1" xref="S3.SS2.p1.10.m8.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.10.m8.1b"><ci id="S3.SS2.p1.10.m8.1.1.cmml" xref="S3.SS2.p1.10.m8.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.10.m8.1c">T</annotation></semantics></math><span id="S3.SS2.p1.14.9" class="ltx_text ltx_font_typewriter"> adapts to contextual shifts, minimizing the need for manual tuning while maintaining control over threshold sensitivity via </span><math id="S3.SS2.p1.11.m9.1" class="ltx_Math" alttext="\gamma" display="inline"><semantics id="S3.SS2.p1.11.m9.1a"><mi id="S3.SS2.p1.11.m9.1.1" xref="S3.SS2.p1.11.m9.1.1.cmml">γ</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.11.m9.1b"><ci id="S3.SS2.p1.11.m9.1.1.cmml" xref="S3.SS2.p1.11.m9.1.1">𝛾</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.11.m9.1c">\gamma</annotation></semantics></math><span id="S3.SS2.p1.14.10" class="ltx_text ltx_font_typewriter">. We also explored a fixed threshold approach (</span><math id="S3.SS2.p1.12.m10.1" class="ltx_Math" alttext="T=T_{\text{fixed}}" display="inline"><semantics id="S3.SS2.p1.12.m10.1a"><mrow id="S3.SS2.p1.12.m10.1.1" xref="S3.SS2.p1.12.m10.1.1.cmml"><mi id="S3.SS2.p1.12.m10.1.1.2" xref="S3.SS2.p1.12.m10.1.1.2.cmml">T</mi><mo id="S3.SS2.p1.12.m10.1.1.1" xref="S3.SS2.p1.12.m10.1.1.1.cmml">=</mo><msub id="S3.SS2.p1.12.m10.1.1.3" xref="S3.SS2.p1.12.m10.1.1.3.cmml"><mi id="S3.SS2.p1.12.m10.1.1.3.2" xref="S3.SS2.p1.12.m10.1.1.3.2.cmml">T</mi><mtext class="ltx_mathvariant_monospace" id="S3.SS2.p1.12.m10.1.1.3.3" xref="S3.SS2.p1.12.m10.1.1.3.3a.cmml">fixed</mtext></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.12.m10.1b"><apply id="S3.SS2.p1.12.m10.1.1.cmml" xref="S3.SS2.p1.12.m10.1.1"><eq id="S3.SS2.p1.12.m10.1.1.1.cmml" xref="S3.SS2.p1.12.m10.1.1.1"></eq><ci id="S3.SS2.p1.12.m10.1.1.2.cmml" xref="S3.SS2.p1.12.m10.1.1.2">𝑇</ci><apply id="S3.SS2.p1.12.m10.1.1.3.cmml" xref="S3.SS2.p1.12.m10.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.p1.12.m10.1.1.3.1.cmml" xref="S3.SS2.p1.12.m10.1.1.3">subscript</csymbol><ci id="S3.SS2.p1.12.m10.1.1.3.2.cmml" xref="S3.SS2.p1.12.m10.1.1.3.2">𝑇</ci><ci id="S3.SS2.p1.12.m10.1.1.3.3a.cmml" xref="S3.SS2.p1.12.m10.1.1.3.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S3.SS2.p1.12.m10.1.1.3.3.cmml" xref="S3.SS2.p1.12.m10.1.1.3.3">fixed</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.12.m10.1c">T=T_{\text{fixed}}</annotation></semantics></math><span id="S3.SS2.p1.14.11" class="ltx_text ltx_font_typewriter">), though our primary focus remained on the dynamic threshold due to its adaptability to varying contexts. This initial segmentation results in a set of potential event boundaries </span><math id="S3.SS2.p1.13.m11.4" class="ltx_Math" alttext="\mathcal{B}={b_{1},b_{2},...,b_{k}}" display="inline"><semantics id="S3.SS2.p1.13.m11.4a"><mrow id="S3.SS2.p1.13.m11.4.4" xref="S3.SS2.p1.13.m11.4.4.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p1.13.m11.4.4.5" xref="S3.SS2.p1.13.m11.4.4.5.cmml">ℬ</mi><mo id="S3.SS2.p1.13.m11.4.4.4" xref="S3.SS2.p1.13.m11.4.4.4.cmml">=</mo><mrow id="S3.SS2.p1.13.m11.4.4.3.3" xref="S3.SS2.p1.13.m11.4.4.3.4.cmml"><msub id="S3.SS2.p1.13.m11.2.2.1.1.1" xref="S3.SS2.p1.13.m11.2.2.1.1.1.cmml"><mi id="S3.SS2.p1.13.m11.2.2.1.1.1.2" xref="S3.SS2.p1.13.m11.2.2.1.1.1.2.cmml">b</mi><mn id="S3.SS2.p1.13.m11.2.2.1.1.1.3" xref="S3.SS2.p1.13.m11.2.2.1.1.1.3.cmml">1</mn></msub><mo id="S3.SS2.p1.13.m11.4.4.3.3.4" xref="S3.SS2.p1.13.m11.4.4.3.4.cmml">,</mo><msub id="S3.SS2.p1.13.m11.3.3.2.2.2" xref="S3.SS2.p1.13.m11.3.3.2.2.2.cmml"><mi id="S3.SS2.p1.13.m11.3.3.2.2.2.2" xref="S3.SS2.p1.13.m11.3.3.2.2.2.2.cmml">b</mi><mn id="S3.SS2.p1.13.m11.3.3.2.2.2.3" xref="S3.SS2.p1.13.m11.3.3.2.2.2.3.cmml">2</mn></msub><mo id="S3.SS2.p1.13.m11.4.4.3.3.5" xref="S3.SS2.p1.13.m11.4.4.3.4.cmml">,</mo><mi mathvariant="normal" id="S3.SS2.p1.13.m11.1.1" xref="S3.SS2.p1.13.m11.1.1.cmml">…</mi><mo id="S3.SS2.p1.13.m11.4.4.3.3.6" xref="S3.SS2.p1.13.m11.4.4.3.4.cmml">,</mo><msub id="S3.SS2.p1.13.m11.4.4.3.3.3" xref="S3.SS2.p1.13.m11.4.4.3.3.3.cmml"><mi id="S3.SS2.p1.13.m11.4.4.3.3.3.2" xref="S3.SS2.p1.13.m11.4.4.3.3.3.2.cmml">b</mi><mi id="S3.SS2.p1.13.m11.4.4.3.3.3.3" xref="S3.SS2.p1.13.m11.4.4.3.3.3.3.cmml">k</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.13.m11.4b"><apply id="S3.SS2.p1.13.m11.4.4.cmml" xref="S3.SS2.p1.13.m11.4.4"><eq id="S3.SS2.p1.13.m11.4.4.4.cmml" xref="S3.SS2.p1.13.m11.4.4.4"></eq><ci id="S3.SS2.p1.13.m11.4.4.5.cmml" xref="S3.SS2.p1.13.m11.4.4.5">ℬ</ci><list id="S3.SS2.p1.13.m11.4.4.3.4.cmml" xref="S3.SS2.p1.13.m11.4.4.3.3"><apply id="S3.SS2.p1.13.m11.2.2.1.1.1.cmml" xref="S3.SS2.p1.13.m11.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.13.m11.2.2.1.1.1.1.cmml" xref="S3.SS2.p1.13.m11.2.2.1.1.1">subscript</csymbol><ci id="S3.SS2.p1.13.m11.2.2.1.1.1.2.cmml" xref="S3.SS2.p1.13.m11.2.2.1.1.1.2">𝑏</ci><cn type="integer" id="S3.SS2.p1.13.m11.2.2.1.1.1.3.cmml" xref="S3.SS2.p1.13.m11.2.2.1.1.1.3">1</cn></apply><apply id="S3.SS2.p1.13.m11.3.3.2.2.2.cmml" xref="S3.SS2.p1.13.m11.3.3.2.2.2"><csymbol cd="ambiguous" id="S3.SS2.p1.13.m11.3.3.2.2.2.1.cmml" xref="S3.SS2.p1.13.m11.3.3.2.2.2">subscript</csymbol><ci id="S3.SS2.p1.13.m11.3.3.2.2.2.2.cmml" xref="S3.SS2.p1.13.m11.3.3.2.2.2.2">𝑏</ci><cn type="integer" id="S3.SS2.p1.13.m11.3.3.2.2.2.3.cmml" xref="S3.SS2.p1.13.m11.3.3.2.2.2.3">2</cn></apply><ci id="S3.SS2.p1.13.m11.1.1.cmml" xref="S3.SS2.p1.13.m11.1.1">…</ci><apply id="S3.SS2.p1.13.m11.4.4.3.3.3.cmml" xref="S3.SS2.p1.13.m11.4.4.3.3.3"><csymbol cd="ambiguous" id="S3.SS2.p1.13.m11.4.4.3.3.3.1.cmml" xref="S3.SS2.p1.13.m11.4.4.3.3.3">subscript</csymbol><ci id="S3.SS2.p1.13.m11.4.4.3.3.3.2.cmml" xref="S3.SS2.p1.13.m11.4.4.3.3.3.2">𝑏</ci><ci id="S3.SS2.p1.13.m11.4.4.3.3.3.3.cmml" xref="S3.SS2.p1.13.m11.4.4.3.3.3.3">𝑘</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.13.m11.4c">\mathcal{B}={b_{1},b_{2},...,b_{k}}</annotation></semantics></math><span id="S3.SS2.p1.14.12" class="ltx_text ltx_font_typewriter">, where each </span><math id="S3.SS2.p1.14.m12.1" class="ltx_Math" alttext="b_{i}" display="inline"><semantics id="S3.SS2.p1.14.m12.1a"><msub id="S3.SS2.p1.14.m12.1.1" xref="S3.SS2.p1.14.m12.1.1.cmml"><mi id="S3.SS2.p1.14.m12.1.1.2" xref="S3.SS2.p1.14.m12.1.1.2.cmml">b</mi><mi id="S3.SS2.p1.14.m12.1.1.3" xref="S3.SS2.p1.14.m12.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.14.m12.1b"><apply id="S3.SS2.p1.14.m12.1.1.cmml" xref="S3.SS2.p1.14.m12.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.14.m12.1.1.1.cmml" xref="S3.SS2.p1.14.m12.1.1">subscript</csymbol><ci id="S3.SS2.p1.14.m12.1.1.2.cmml" xref="S3.SS2.p1.14.m12.1.1.2">𝑏</ci><ci id="S3.SS2.p1.14.m12.1.1.3.cmml" xref="S3.SS2.p1.14.m12.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.14.m12.1c">b_{i}</annotation></semantics></math><span id="S3.SS2.p1.14.13" class="ltx_text ltx_font_typewriter"> represents the index of a token exceeding the surprise threshold. These boundaries serve as the starting point for our subsequent refinement process, which aims to optimise the intra-event coherence and inter-event distinctiveness of the resulting memory segments.</span></p>
</div>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_font_typewriter ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="S3.SS3.1.1.1" class="ltx_text ltx_font_serif">3.3</span> </span>Boundary refinement</h3>

<div id="S3.SS3.p1" class="ltx_para ltx_noindent">
<p id="S3.SS3.p1.3" class="ltx_p"><span id="S3.SS3.p1.3.1" class="ltx_text ltx_font_typewriter">While surprise-based segmentation provides an effective initial estimate of event boundaries, we make the key observation that the utility of elements within an event during memory recall depends on their likelihood of being utilised by the current query. Therefore, we theorise that memory recall will be most efficient with high intra-event similarity between keys while maintaining low inter-event similarity. For instance, see the similarity of groups in Fig. </span><a href="#S2.F1" title="Figure 1 ‣ 2.1 Long-context in LLMs ‣ 2 Related work ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">1</span></a><span id="S3.SS3.p1.3.2" class="ltx_text ltx_font_typewriter">. To further ensure this, we introduce a boundary refinement step which looks to optimise this objective. Such an objective is typically optimised in the context of graph-clustering, hence we will express this refinement process in a graph-theoretic manner. To achieve this, we treat the similarity matrix between all keys of an attention head </span><math id="S3.SS3.p1.1.m1.1" class="ltx_Math" alttext="h" display="inline"><semantics id="S3.SS3.p1.1.m1.1a"><mi id="S3.SS3.p1.1.m1.1.1" xref="S3.SS3.p1.1.m1.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.1.m1.1b"><ci id="S3.SS3.p1.1.m1.1.1.cmml" xref="S3.SS3.p1.1.m1.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.1.m1.1c">h</annotation></semantics></math><span id="S3.SS3.p1.3.3" class="ltx_text ltx_font_typewriter"> within the local context window for tokens </span><math id="S3.SS3.p1.2.m2.4" class="ltx_Math" alttext="x_{1},x_{2},...,x_{n}" display="inline"><semantics id="S3.SS3.p1.2.m2.4a"><mrow id="S3.SS3.p1.2.m2.4.4.3" xref="S3.SS3.p1.2.m2.4.4.4.cmml"><msub id="S3.SS3.p1.2.m2.2.2.1.1" xref="S3.SS3.p1.2.m2.2.2.1.1.cmml"><mi id="S3.SS3.p1.2.m2.2.2.1.1.2" xref="S3.SS3.p1.2.m2.2.2.1.1.2.cmml">x</mi><mn id="S3.SS3.p1.2.m2.2.2.1.1.3" xref="S3.SS3.p1.2.m2.2.2.1.1.3.cmml">1</mn></msub><mo id="S3.SS3.p1.2.m2.4.4.3.4" xref="S3.SS3.p1.2.m2.4.4.4.cmml">,</mo><msub id="S3.SS3.p1.2.m2.3.3.2.2" xref="S3.SS3.p1.2.m2.3.3.2.2.cmml"><mi id="S3.SS3.p1.2.m2.3.3.2.2.2" xref="S3.SS3.p1.2.m2.3.3.2.2.2.cmml">x</mi><mn id="S3.SS3.p1.2.m2.3.3.2.2.3" xref="S3.SS3.p1.2.m2.3.3.2.2.3.cmml">2</mn></msub><mo id="S3.SS3.p1.2.m2.4.4.3.5" xref="S3.SS3.p1.2.m2.4.4.4.cmml">,</mo><mi mathvariant="normal" id="S3.SS3.p1.2.m2.1.1" xref="S3.SS3.p1.2.m2.1.1.cmml">…</mi><mo id="S3.SS3.p1.2.m2.4.4.3.6" xref="S3.SS3.p1.2.m2.4.4.4.cmml">,</mo><msub id="S3.SS3.p1.2.m2.4.4.3.3" xref="S3.SS3.p1.2.m2.4.4.3.3.cmml"><mi id="S3.SS3.p1.2.m2.4.4.3.3.2" xref="S3.SS3.p1.2.m2.4.4.3.3.2.cmml">x</mi><mi id="S3.SS3.p1.2.m2.4.4.3.3.3" xref="S3.SS3.p1.2.m2.4.4.3.3.3.cmml">n</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.2.m2.4b"><list id="S3.SS3.p1.2.m2.4.4.4.cmml" xref="S3.SS3.p1.2.m2.4.4.3"><apply id="S3.SS3.p1.2.m2.2.2.1.1.cmml" xref="S3.SS3.p1.2.m2.2.2.1.1"><csymbol cd="ambiguous" id="S3.SS3.p1.2.m2.2.2.1.1.1.cmml" xref="S3.SS3.p1.2.m2.2.2.1.1">subscript</csymbol><ci id="S3.SS3.p1.2.m2.2.2.1.1.2.cmml" xref="S3.SS3.p1.2.m2.2.2.1.1.2">𝑥</ci><cn type="integer" id="S3.SS3.p1.2.m2.2.2.1.1.3.cmml" xref="S3.SS3.p1.2.m2.2.2.1.1.3">1</cn></apply><apply id="S3.SS3.p1.2.m2.3.3.2.2.cmml" xref="S3.SS3.p1.2.m2.3.3.2.2"><csymbol cd="ambiguous" id="S3.SS3.p1.2.m2.3.3.2.2.1.cmml" xref="S3.SS3.p1.2.m2.3.3.2.2">subscript</csymbol><ci id="S3.SS3.p1.2.m2.3.3.2.2.2.cmml" xref="S3.SS3.p1.2.m2.3.3.2.2.2">𝑥</ci><cn type="integer" id="S3.SS3.p1.2.m2.3.3.2.2.3.cmml" xref="S3.SS3.p1.2.m2.3.3.2.2.3">2</cn></apply><ci id="S3.SS3.p1.2.m2.1.1.cmml" xref="S3.SS3.p1.2.m2.1.1">…</ci><apply id="S3.SS3.p1.2.m2.4.4.3.3.cmml" xref="S3.SS3.p1.2.m2.4.4.3.3"><csymbol cd="ambiguous" id="S3.SS3.p1.2.m2.4.4.3.3.1.cmml" xref="S3.SS3.p1.2.m2.4.4.3.3">subscript</csymbol><ci id="S3.SS3.p1.2.m2.4.4.3.3.2.cmml" xref="S3.SS3.p1.2.m2.4.4.3.3.2">𝑥</ci><ci id="S3.SS3.p1.2.m2.4.4.3.3.3.cmml" xref="S3.SS3.p1.2.m2.4.4.3.3.3">𝑛</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.2.m2.4c">x_{1},x_{2},...,x_{n}</annotation></semantics></math><span id="S3.SS3.p1.3.4" class="ltx_text ltx_font_typewriter"> as an adjacency matrix. We define the adjacency matrix </span><math id="S3.SS3.p1.3.m3.1" class="ltx_Math" alttext="A^{h}" display="inline"><semantics id="S3.SS3.p1.3.m3.1a"><msup id="S3.SS3.p1.3.m3.1.1" xref="S3.SS3.p1.3.m3.1.1.cmml"><mi id="S3.SS3.p1.3.m3.1.1.2" xref="S3.SS3.p1.3.m3.1.1.2.cmml">A</mi><mi id="S3.SS3.p1.3.m3.1.1.3" xref="S3.SS3.p1.3.m3.1.1.3.cmml">h</mi></msup><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.3.m3.1b"><apply id="S3.SS3.p1.3.m3.1.1.cmml" xref="S3.SS3.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS3.p1.3.m3.1.1.1.cmml" xref="S3.SS3.p1.3.m3.1.1">superscript</csymbol><ci id="S3.SS3.p1.3.m3.1.1.2.cmml" xref="S3.SS3.p1.3.m3.1.1.2">𝐴</ci><ci id="S3.SS3.p1.3.m3.1.1.3.cmml" xref="S3.SS3.p1.3.m3.1.1.3">ℎ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.3.m3.1c">A^{h}</annotation></semantics></math><span id="S3.SS3.p1.3.5" class="ltx_text ltx_font_typewriter"> as</span></p>
<table id="A1.EGx2" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S3.E2"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E2.m1.1" class="ltx_Math" alttext="\displaystyle A^{h}_{ij}=\text{sim}(K^{h}_{i},K^{h}_{j})," display="inline"><semantics id="S3.E2.m1.1a"><mrow id="S3.E2.m1.1.1.1" xref="S3.E2.m1.1.1.1.1.cmml"><mrow id="S3.E2.m1.1.1.1.1" xref="S3.E2.m1.1.1.1.1.cmml"><msubsup id="S3.E2.m1.1.1.1.1.4" xref="S3.E2.m1.1.1.1.1.4.cmml"><mi id="S3.E2.m1.1.1.1.1.4.2.2" xref="S3.E2.m1.1.1.1.1.4.2.2.cmml">A</mi><mrow id="S3.E2.m1.1.1.1.1.4.3" xref="S3.E2.m1.1.1.1.1.4.3.cmml"><mi id="S3.E2.m1.1.1.1.1.4.3.2" xref="S3.E2.m1.1.1.1.1.4.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S3.E2.m1.1.1.1.1.4.3.1" xref="S3.E2.m1.1.1.1.1.4.3.1.cmml">​</mo><mi id="S3.E2.m1.1.1.1.1.4.3.3" xref="S3.E2.m1.1.1.1.1.4.3.3.cmml">j</mi></mrow><mi id="S3.E2.m1.1.1.1.1.4.2.3" xref="S3.E2.m1.1.1.1.1.4.2.3.cmml">h</mi></msubsup><mo id="S3.E2.m1.1.1.1.1.3" xref="S3.E2.m1.1.1.1.1.3.cmml">=</mo><mrow id="S3.E2.m1.1.1.1.1.2" xref="S3.E2.m1.1.1.1.1.2.cmml"><mtext class="ltx_mathvariant_monospace" id="S3.E2.m1.1.1.1.1.2.4" xref="S3.E2.m1.1.1.1.1.2.4a.cmml">sim</mtext><mo lspace="0em" rspace="0em" id="S3.E2.m1.1.1.1.1.2.3" xref="S3.E2.m1.1.1.1.1.2.3.cmml">​</mo><mrow id="S3.E2.m1.1.1.1.1.2.2.2" xref="S3.E2.m1.1.1.1.1.2.2.3.cmml"><mo stretchy="false" id="S3.E2.m1.1.1.1.1.2.2.2.3" xref="S3.E2.m1.1.1.1.1.2.2.3.cmml">(</mo><msubsup id="S3.E2.m1.1.1.1.1.1.1.1.1" xref="S3.E2.m1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.E2.m1.1.1.1.1.1.1.1.1.2.2" xref="S3.E2.m1.1.1.1.1.1.1.1.1.2.2.cmml">K</mi><mi id="S3.E2.m1.1.1.1.1.1.1.1.1.3" xref="S3.E2.m1.1.1.1.1.1.1.1.1.3.cmml">i</mi><mi id="S3.E2.m1.1.1.1.1.1.1.1.1.2.3" xref="S3.E2.m1.1.1.1.1.1.1.1.1.2.3.cmml">h</mi></msubsup><mo id="S3.E2.m1.1.1.1.1.2.2.2.4" xref="S3.E2.m1.1.1.1.1.2.2.3.cmml">,</mo><msubsup id="S3.E2.m1.1.1.1.1.2.2.2.2" xref="S3.E2.m1.1.1.1.1.2.2.2.2.cmml"><mi id="S3.E2.m1.1.1.1.1.2.2.2.2.2.2" xref="S3.E2.m1.1.1.1.1.2.2.2.2.2.2.cmml">K</mi><mi id="S3.E2.m1.1.1.1.1.2.2.2.2.3" xref="S3.E2.m1.1.1.1.1.2.2.2.2.3.cmml">j</mi><mi id="S3.E2.m1.1.1.1.1.2.2.2.2.2.3" xref="S3.E2.m1.1.1.1.1.2.2.2.2.2.3.cmml">h</mi></msubsup><mo stretchy="false" id="S3.E2.m1.1.1.1.1.2.2.2.5" xref="S3.E2.m1.1.1.1.1.2.2.3.cmml">)</mo></mrow></mrow></mrow><mo id="S3.E2.m1.1.1.1.2" xref="S3.E2.m1.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E2.m1.1b"><apply id="S3.E2.m1.1.1.1.1.cmml" xref="S3.E2.m1.1.1.1"><eq id="S3.E2.m1.1.1.1.1.3.cmml" xref="S3.E2.m1.1.1.1.1.3"></eq><apply id="S3.E2.m1.1.1.1.1.4.cmml" xref="S3.E2.m1.1.1.1.1.4"><csymbol cd="ambiguous" id="S3.E2.m1.1.1.1.1.4.1.cmml" xref="S3.E2.m1.1.1.1.1.4">subscript</csymbol><apply id="S3.E2.m1.1.1.1.1.4.2.cmml" xref="S3.E2.m1.1.1.1.1.4"><csymbol cd="ambiguous" id="S3.E2.m1.1.1.1.1.4.2.1.cmml" xref="S3.E2.m1.1.1.1.1.4">superscript</csymbol><ci id="S3.E2.m1.1.1.1.1.4.2.2.cmml" xref="S3.E2.m1.1.1.1.1.4.2.2">𝐴</ci><ci id="S3.E2.m1.1.1.1.1.4.2.3.cmml" xref="S3.E2.m1.1.1.1.1.4.2.3">ℎ</ci></apply><apply id="S3.E2.m1.1.1.1.1.4.3.cmml" xref="S3.E2.m1.1.1.1.1.4.3"><times id="S3.E2.m1.1.1.1.1.4.3.1.cmml" xref="S3.E2.m1.1.1.1.1.4.3.1"></times><ci id="S3.E2.m1.1.1.1.1.4.3.2.cmml" xref="S3.E2.m1.1.1.1.1.4.3.2">𝑖</ci><ci id="S3.E2.m1.1.1.1.1.4.3.3.cmml" xref="S3.E2.m1.1.1.1.1.4.3.3">𝑗</ci></apply></apply><apply id="S3.E2.m1.1.1.1.1.2.cmml" xref="S3.E2.m1.1.1.1.1.2"><times id="S3.E2.m1.1.1.1.1.2.3.cmml" xref="S3.E2.m1.1.1.1.1.2.3"></times><ci id="S3.E2.m1.1.1.1.1.2.4a.cmml" xref="S3.E2.m1.1.1.1.1.2.4"><mtext class="ltx_mathvariant_monospace" id="S3.E2.m1.1.1.1.1.2.4.cmml" xref="S3.E2.m1.1.1.1.1.2.4">sim</mtext></ci><interval closure="open" id="S3.E2.m1.1.1.1.1.2.2.3.cmml" xref="S3.E2.m1.1.1.1.1.2.2.2"><apply id="S3.E2.m1.1.1.1.1.1.1.1.1.cmml" xref="S3.E2.m1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E2.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E2.m1.1.1.1.1.1.1.1.1">subscript</csymbol><apply id="S3.E2.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E2.m1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E2.m1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.E2.m1.1.1.1.1.1.1.1.1">superscript</csymbol><ci id="S3.E2.m1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.E2.m1.1.1.1.1.1.1.1.1.2.2">𝐾</ci><ci id="S3.E2.m1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.E2.m1.1.1.1.1.1.1.1.1.2.3">ℎ</ci></apply><ci id="S3.E2.m1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E2.m1.1.1.1.1.1.1.1.1.3">𝑖</ci></apply><apply id="S3.E2.m1.1.1.1.1.2.2.2.2.cmml" xref="S3.E2.m1.1.1.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S3.E2.m1.1.1.1.1.2.2.2.2.1.cmml" xref="S3.E2.m1.1.1.1.1.2.2.2.2">subscript</csymbol><apply id="S3.E2.m1.1.1.1.1.2.2.2.2.2.cmml" xref="S3.E2.m1.1.1.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S3.E2.m1.1.1.1.1.2.2.2.2.2.1.cmml" xref="S3.E2.m1.1.1.1.1.2.2.2.2">superscript</csymbol><ci id="S3.E2.m1.1.1.1.1.2.2.2.2.2.2.cmml" xref="S3.E2.m1.1.1.1.1.2.2.2.2.2.2">𝐾</ci><ci id="S3.E2.m1.1.1.1.1.2.2.2.2.2.3.cmml" xref="S3.E2.m1.1.1.1.1.2.2.2.2.2.3">ℎ</ci></apply><ci id="S3.E2.m1.1.1.1.1.2.2.2.2.3.cmml" xref="S3.E2.m1.1.1.1.1.2.2.2.2.3">𝑗</ci></apply></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E2.m1.1c">\displaystyle A^{h}_{ij}=\text{sim}(K^{h}_{i},K^{h}_{j}),</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
<p id="S3.SS3.p1.8" class="ltx_p"><span id="S3.SS3.p1.8.1" class="ltx_text ltx_font_typewriter">where </span><math id="S3.SS3.p1.4.m1.1" class="ltx_Math" alttext="K^{h}_{i}" display="inline"><semantics id="S3.SS3.p1.4.m1.1a"><msubsup id="S3.SS3.p1.4.m1.1.1" xref="S3.SS3.p1.4.m1.1.1.cmml"><mi id="S3.SS3.p1.4.m1.1.1.2.2" xref="S3.SS3.p1.4.m1.1.1.2.2.cmml">K</mi><mi id="S3.SS3.p1.4.m1.1.1.3" xref="S3.SS3.p1.4.m1.1.1.3.cmml">i</mi><mi id="S3.SS3.p1.4.m1.1.1.2.3" xref="S3.SS3.p1.4.m1.1.1.2.3.cmml">h</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.4.m1.1b"><apply id="S3.SS3.p1.4.m1.1.1.cmml" xref="S3.SS3.p1.4.m1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p1.4.m1.1.1.1.cmml" xref="S3.SS3.p1.4.m1.1.1">subscript</csymbol><apply id="S3.SS3.p1.4.m1.1.1.2.cmml" xref="S3.SS3.p1.4.m1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p1.4.m1.1.1.2.1.cmml" xref="S3.SS3.p1.4.m1.1.1">superscript</csymbol><ci id="S3.SS3.p1.4.m1.1.1.2.2.cmml" xref="S3.SS3.p1.4.m1.1.1.2.2">𝐾</ci><ci id="S3.SS3.p1.4.m1.1.1.2.3.cmml" xref="S3.SS3.p1.4.m1.1.1.2.3">ℎ</ci></apply><ci id="S3.SS3.p1.4.m1.1.1.3.cmml" xref="S3.SS3.p1.4.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.4.m1.1c">K^{h}_{i}</annotation></semantics></math><span id="S3.SS3.p1.8.2" class="ltx_text ltx_font_typewriter"> and </span><math id="S3.SS3.p1.5.m2.1" class="ltx_Math" alttext="K^{h}_{j}" display="inline"><semantics id="S3.SS3.p1.5.m2.1a"><msubsup id="S3.SS3.p1.5.m2.1.1" xref="S3.SS3.p1.5.m2.1.1.cmml"><mi id="S3.SS3.p1.5.m2.1.1.2.2" xref="S3.SS3.p1.5.m2.1.1.2.2.cmml">K</mi><mi id="S3.SS3.p1.5.m2.1.1.3" xref="S3.SS3.p1.5.m2.1.1.3.cmml">j</mi><mi id="S3.SS3.p1.5.m2.1.1.2.3" xref="S3.SS3.p1.5.m2.1.1.2.3.cmml">h</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.5.m2.1b"><apply id="S3.SS3.p1.5.m2.1.1.cmml" xref="S3.SS3.p1.5.m2.1.1"><csymbol cd="ambiguous" id="S3.SS3.p1.5.m2.1.1.1.cmml" xref="S3.SS3.p1.5.m2.1.1">subscript</csymbol><apply id="S3.SS3.p1.5.m2.1.1.2.cmml" xref="S3.SS3.p1.5.m2.1.1"><csymbol cd="ambiguous" id="S3.SS3.p1.5.m2.1.1.2.1.cmml" xref="S3.SS3.p1.5.m2.1.1">superscript</csymbol><ci id="S3.SS3.p1.5.m2.1.1.2.2.cmml" xref="S3.SS3.p1.5.m2.1.1.2.2">𝐾</ci><ci id="S3.SS3.p1.5.m2.1.1.2.3.cmml" xref="S3.SS3.p1.5.m2.1.1.2.3">ℎ</ci></apply><ci id="S3.SS3.p1.5.m2.1.1.3.cmml" xref="S3.SS3.p1.5.m2.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.5.m2.1c">K^{h}_{j}</annotation></semantics></math><span id="S3.SS3.p1.8.3" class="ltx_text ltx_font_typewriter"> are the key vectors corresponding to tokens </span><math id="S3.SS3.p1.6.m3.1" class="ltx_Math" alttext="x_{i}" display="inline"><semantics id="S3.SS3.p1.6.m3.1a"><msub id="S3.SS3.p1.6.m3.1.1" xref="S3.SS3.p1.6.m3.1.1.cmml"><mi id="S3.SS3.p1.6.m3.1.1.2" xref="S3.SS3.p1.6.m3.1.1.2.cmml">x</mi><mi id="S3.SS3.p1.6.m3.1.1.3" xref="S3.SS3.p1.6.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.6.m3.1b"><apply id="S3.SS3.p1.6.m3.1.1.cmml" xref="S3.SS3.p1.6.m3.1.1"><csymbol cd="ambiguous" id="S3.SS3.p1.6.m3.1.1.1.cmml" xref="S3.SS3.p1.6.m3.1.1">subscript</csymbol><ci id="S3.SS3.p1.6.m3.1.1.2.cmml" xref="S3.SS3.p1.6.m3.1.1.2">𝑥</ci><ci id="S3.SS3.p1.6.m3.1.1.3.cmml" xref="S3.SS3.p1.6.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.6.m3.1c">x_{i}</annotation></semantics></math><span id="S3.SS3.p1.8.4" class="ltx_text ltx_font_typewriter"> and </span><math id="S3.SS3.p1.7.m4.1" class="ltx_Math" alttext="x_{j}" display="inline"><semantics id="S3.SS3.p1.7.m4.1a"><msub id="S3.SS3.p1.7.m4.1.1" xref="S3.SS3.p1.7.m4.1.1.cmml"><mi id="S3.SS3.p1.7.m4.1.1.2" xref="S3.SS3.p1.7.m4.1.1.2.cmml">x</mi><mi id="S3.SS3.p1.7.m4.1.1.3" xref="S3.SS3.p1.7.m4.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.7.m4.1b"><apply id="S3.SS3.p1.7.m4.1.1.cmml" xref="S3.SS3.p1.7.m4.1.1"><csymbol cd="ambiguous" id="S3.SS3.p1.7.m4.1.1.1.cmml" xref="S3.SS3.p1.7.m4.1.1">subscript</csymbol><ci id="S3.SS3.p1.7.m4.1.1.2.cmml" xref="S3.SS3.p1.7.m4.1.1.2">𝑥</ci><ci id="S3.SS3.p1.7.m4.1.1.3.cmml" xref="S3.SS3.p1.7.m4.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.7.m4.1c">x_{j}</annotation></semantics></math><span id="S3.SS3.p1.8.5" class="ltx_text ltx_font_typewriter">, respectively. The similarity function measures the closeness of two key vectors; in our implementation, we use dot product similarity
</span><math id="S3.SS3.p1.8.m5.1" class="ltx_Math" alttext="{K^{h}}^{\mathsf{T}}_{i}\cdot K^{h}_{j}" display="inline"><semantics id="S3.SS3.p1.8.m5.1a"><mrow id="S3.SS3.p1.8.m5.1.1" xref="S3.SS3.p1.8.m5.1.1.cmml"><mmultiscripts id="S3.SS3.p1.8.m5.1.1.2" xref="S3.SS3.p1.8.m5.1.1.2.cmml"><mi id="S3.SS3.p1.8.m5.1.1.2.2.2.2" xref="S3.SS3.p1.8.m5.1.1.2.2.2.2.cmml">K</mi><mrow id="S3.SS3.p1.8.m5.1.1.2a" xref="S3.SS3.p1.8.m5.1.1.2.cmml"></mrow><mi id="S3.SS3.p1.8.m5.1.1.2.2.2.3" xref="S3.SS3.p1.8.m5.1.1.2.2.2.3.cmml">h</mi><mi id="S3.SS3.p1.8.m5.1.1.2.3" xref="S3.SS3.p1.8.m5.1.1.2.3.cmml">i</mi><mi id="S3.SS3.p1.8.m5.1.1.2.2.3" xref="S3.SS3.p1.8.m5.1.1.2.2.3.cmml">𝖳</mi></mmultiscripts><mo lspace="0.222em" rspace="0.222em" id="S3.SS3.p1.8.m5.1.1.1" xref="S3.SS3.p1.8.m5.1.1.1.cmml">⋅</mo><msubsup id="S3.SS3.p1.8.m5.1.1.3" xref="S3.SS3.p1.8.m5.1.1.3.cmml"><mi id="S3.SS3.p1.8.m5.1.1.3.2.2" xref="S3.SS3.p1.8.m5.1.1.3.2.2.cmml">K</mi><mi id="S3.SS3.p1.8.m5.1.1.3.3" xref="S3.SS3.p1.8.m5.1.1.3.3.cmml">j</mi><mi id="S3.SS3.p1.8.m5.1.1.3.2.3" xref="S3.SS3.p1.8.m5.1.1.3.2.3.cmml">h</mi></msubsup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.8.m5.1b"><apply id="S3.SS3.p1.8.m5.1.1.cmml" xref="S3.SS3.p1.8.m5.1.1"><ci id="S3.SS3.p1.8.m5.1.1.1.cmml" xref="S3.SS3.p1.8.m5.1.1.1">⋅</ci><apply id="S3.SS3.p1.8.m5.1.1.2.cmml" xref="S3.SS3.p1.8.m5.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.p1.8.m5.1.1.2.1.cmml" xref="S3.SS3.p1.8.m5.1.1.2">subscript</csymbol><apply id="S3.SS3.p1.8.m5.1.1.2.2.cmml" xref="S3.SS3.p1.8.m5.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.p1.8.m5.1.1.2.2.1.cmml" xref="S3.SS3.p1.8.m5.1.1.2">superscript</csymbol><apply id="S3.SS3.p1.8.m5.1.1.2.2.2.cmml" xref="S3.SS3.p1.8.m5.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.p1.8.m5.1.1.2.2.2.1.cmml" xref="S3.SS3.p1.8.m5.1.1.2">superscript</csymbol><ci id="S3.SS3.p1.8.m5.1.1.2.2.2.2.cmml" xref="S3.SS3.p1.8.m5.1.1.2.2.2.2">𝐾</ci><ci id="S3.SS3.p1.8.m5.1.1.2.2.2.3.cmml" xref="S3.SS3.p1.8.m5.1.1.2.2.2.3">ℎ</ci></apply><ci id="S3.SS3.p1.8.m5.1.1.2.2.3.cmml" xref="S3.SS3.p1.8.m5.1.1.2.2.3">𝖳</ci></apply><ci id="S3.SS3.p1.8.m5.1.1.2.3.cmml" xref="S3.SS3.p1.8.m5.1.1.2.3">𝑖</ci></apply><apply id="S3.SS3.p1.8.m5.1.1.3.cmml" xref="S3.SS3.p1.8.m5.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.p1.8.m5.1.1.3.1.cmml" xref="S3.SS3.p1.8.m5.1.1.3">subscript</csymbol><apply id="S3.SS3.p1.8.m5.1.1.3.2.cmml" xref="S3.SS3.p1.8.m5.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.p1.8.m5.1.1.3.2.1.cmml" xref="S3.SS3.p1.8.m5.1.1.3">superscript</csymbol><ci id="S3.SS3.p1.8.m5.1.1.3.2.2.cmml" xref="S3.SS3.p1.8.m5.1.1.3.2.2">𝐾</ci><ci id="S3.SS3.p1.8.m5.1.1.3.2.3.cmml" xref="S3.SS3.p1.8.m5.1.1.3.2.3">ℎ</ci></apply><ci id="S3.SS3.p1.8.m5.1.1.3.3.cmml" xref="S3.SS3.p1.8.m5.1.1.3.3">𝑗</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.8.m5.1c">{K^{h}}^{\mathsf{T}}_{i}\cdot K^{h}_{j}</annotation></semantics></math><span id="S3.SS3.p1.8.6" class="ltx_text ltx_font_typewriter"> due to its effectiveness in capturing semantic relationships in high-dimensional spaces </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S3.SS3.p1.8.7.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Vaswani et al.</span><span id="S3.SS3.p1.8.8.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib25" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2017</span></a><span id="S3.SS3.p1.8.9.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S3.SS3.p1.8.10" class="ltx_text ltx_font_typewriter"> and to align with the mechanism of self-attention in Transformers.</span></p>
</div>
<div id="S3.SS3.p2" class="ltx_para ltx_noindent">
<p id="S3.SS3.p2.3" class="ltx_p"><span id="S3.SS3.p2.3.1" class="ltx_text ltx_font_typewriter">To evaluate the quality of potential boundaries, we define a metric function </span><math id="S3.SS3.p2.1.m1.5" class="ltx_Math" alttext="f(A,\mathcal{B}):\mathbb{R}^{n\times n}\times\{1,\ldots,n\}^{k}\rightarrow\mathbb{R}" display="inline"><semantics id="S3.SS3.p2.1.m1.5a"><mrow id="S3.SS3.p2.1.m1.5.6" xref="S3.SS3.p2.1.m1.5.6.cmml"><mrow id="S3.SS3.p2.1.m1.5.6.2" xref="S3.SS3.p2.1.m1.5.6.2.cmml"><mi id="S3.SS3.p2.1.m1.5.6.2.2" xref="S3.SS3.p2.1.m1.5.6.2.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="S3.SS3.p2.1.m1.5.6.2.1" xref="S3.SS3.p2.1.m1.5.6.2.1.cmml">​</mo><mrow id="S3.SS3.p2.1.m1.5.6.2.3.2" xref="S3.SS3.p2.1.m1.5.6.2.3.1.cmml"><mo stretchy="false" id="S3.SS3.p2.1.m1.5.6.2.3.2.1" xref="S3.SS3.p2.1.m1.5.6.2.3.1.cmml">(</mo><mi id="S3.SS3.p2.1.m1.1.1" xref="S3.SS3.p2.1.m1.1.1.cmml">A</mi><mo id="S3.SS3.p2.1.m1.5.6.2.3.2.2" xref="S3.SS3.p2.1.m1.5.6.2.3.1.cmml">,</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.1.m1.2.2" xref="S3.SS3.p2.1.m1.2.2.cmml">ℬ</mi><mo rspace="0.278em" stretchy="false" id="S3.SS3.p2.1.m1.5.6.2.3.2.3" xref="S3.SS3.p2.1.m1.5.6.2.3.1.cmml">)</mo></mrow></mrow><mo rspace="0.278em" id="S3.SS3.p2.1.m1.5.6.1" xref="S3.SS3.p2.1.m1.5.6.1.cmml">:</mo><mrow id="S3.SS3.p2.1.m1.5.6.3" xref="S3.SS3.p2.1.m1.5.6.3.cmml"><mrow id="S3.SS3.p2.1.m1.5.6.3.2" xref="S3.SS3.p2.1.m1.5.6.3.2.cmml"><msup id="S3.SS3.p2.1.m1.5.6.3.2.2" xref="S3.SS3.p2.1.m1.5.6.3.2.2.cmml"><mi id="S3.SS3.p2.1.m1.5.6.3.2.2.2" xref="S3.SS3.p2.1.m1.5.6.3.2.2.2.cmml">ℝ</mi><mrow id="S3.SS3.p2.1.m1.5.6.3.2.2.3" xref="S3.SS3.p2.1.m1.5.6.3.2.2.3.cmml"><mi id="S3.SS3.p2.1.m1.5.6.3.2.2.3.2" xref="S3.SS3.p2.1.m1.5.6.3.2.2.3.2.cmml">n</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS3.p2.1.m1.5.6.3.2.2.3.1" xref="S3.SS3.p2.1.m1.5.6.3.2.2.3.1.cmml">×</mo><mi id="S3.SS3.p2.1.m1.5.6.3.2.2.3.3" xref="S3.SS3.p2.1.m1.5.6.3.2.2.3.3.cmml">n</mi></mrow></msup><mo lspace="0.222em" rspace="0.222em" id="S3.SS3.p2.1.m1.5.6.3.2.1" xref="S3.SS3.p2.1.m1.5.6.3.2.1.cmml">×</mo><msup id="S3.SS3.p2.1.m1.5.6.3.2.3" xref="S3.SS3.p2.1.m1.5.6.3.2.3.cmml"><mrow id="S3.SS3.p2.1.m1.5.6.3.2.3.2.2" xref="S3.SS3.p2.1.m1.5.6.3.2.3.2.1.cmml"><mo stretchy="false" id="S3.SS3.p2.1.m1.5.6.3.2.3.2.2.1" xref="S3.SS3.p2.1.m1.5.6.3.2.3.2.1.cmml">{</mo><mn id="S3.SS3.p2.1.m1.3.3" xref="S3.SS3.p2.1.m1.3.3.cmml">1</mn><mo id="S3.SS3.p2.1.m1.5.6.3.2.3.2.2.2" xref="S3.SS3.p2.1.m1.5.6.3.2.3.2.1.cmml">,</mo><mi mathvariant="normal" id="S3.SS3.p2.1.m1.4.4" xref="S3.SS3.p2.1.m1.4.4.cmml">…</mi><mo id="S3.SS3.p2.1.m1.5.6.3.2.3.2.2.3" xref="S3.SS3.p2.1.m1.5.6.3.2.3.2.1.cmml">,</mo><mi id="S3.SS3.p2.1.m1.5.5" xref="S3.SS3.p2.1.m1.5.5.cmml">n</mi><mo stretchy="false" id="S3.SS3.p2.1.m1.5.6.3.2.3.2.2.4" xref="S3.SS3.p2.1.m1.5.6.3.2.3.2.1.cmml">}</mo></mrow><mi id="S3.SS3.p2.1.m1.5.6.3.2.3.3" xref="S3.SS3.p2.1.m1.5.6.3.2.3.3.cmml">k</mi></msup></mrow><mo stretchy="false" id="S3.SS3.p2.1.m1.5.6.3.1" xref="S3.SS3.p2.1.m1.5.6.3.1.cmml">→</mo><mi id="S3.SS3.p2.1.m1.5.6.3.3" xref="S3.SS3.p2.1.m1.5.6.3.3.cmml">ℝ</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.1.m1.5b"><apply id="S3.SS3.p2.1.m1.5.6.cmml" xref="S3.SS3.p2.1.m1.5.6"><ci id="S3.SS3.p2.1.m1.5.6.1.cmml" xref="S3.SS3.p2.1.m1.5.6.1">:</ci><apply id="S3.SS3.p2.1.m1.5.6.2.cmml" xref="S3.SS3.p2.1.m1.5.6.2"><times id="S3.SS3.p2.1.m1.5.6.2.1.cmml" xref="S3.SS3.p2.1.m1.5.6.2.1"></times><ci id="S3.SS3.p2.1.m1.5.6.2.2.cmml" xref="S3.SS3.p2.1.m1.5.6.2.2">𝑓</ci><interval closure="open" id="S3.SS3.p2.1.m1.5.6.2.3.1.cmml" xref="S3.SS3.p2.1.m1.5.6.2.3.2"><ci id="S3.SS3.p2.1.m1.1.1.cmml" xref="S3.SS3.p2.1.m1.1.1">𝐴</ci><ci id="S3.SS3.p2.1.m1.2.2.cmml" xref="S3.SS3.p2.1.m1.2.2">ℬ</ci></interval></apply><apply id="S3.SS3.p2.1.m1.5.6.3.cmml" xref="S3.SS3.p2.1.m1.5.6.3"><ci id="S3.SS3.p2.1.m1.5.6.3.1.cmml" xref="S3.SS3.p2.1.m1.5.6.3.1">→</ci><apply id="S3.SS3.p2.1.m1.5.6.3.2.cmml" xref="S3.SS3.p2.1.m1.5.6.3.2"><times id="S3.SS3.p2.1.m1.5.6.3.2.1.cmml" xref="S3.SS3.p2.1.m1.5.6.3.2.1"></times><apply id="S3.SS3.p2.1.m1.5.6.3.2.2.cmml" xref="S3.SS3.p2.1.m1.5.6.3.2.2"><csymbol cd="ambiguous" id="S3.SS3.p2.1.m1.5.6.3.2.2.1.cmml" xref="S3.SS3.p2.1.m1.5.6.3.2.2">superscript</csymbol><ci id="S3.SS3.p2.1.m1.5.6.3.2.2.2.cmml" xref="S3.SS3.p2.1.m1.5.6.3.2.2.2">ℝ</ci><apply id="S3.SS3.p2.1.m1.5.6.3.2.2.3.cmml" xref="S3.SS3.p2.1.m1.5.6.3.2.2.3"><times id="S3.SS3.p2.1.m1.5.6.3.2.2.3.1.cmml" xref="S3.SS3.p2.1.m1.5.6.3.2.2.3.1"></times><ci id="S3.SS3.p2.1.m1.5.6.3.2.2.3.2.cmml" xref="S3.SS3.p2.1.m1.5.6.3.2.2.3.2">𝑛</ci><ci id="S3.SS3.p2.1.m1.5.6.3.2.2.3.3.cmml" xref="S3.SS3.p2.1.m1.5.6.3.2.2.3.3">𝑛</ci></apply></apply><apply id="S3.SS3.p2.1.m1.5.6.3.2.3.cmml" xref="S3.SS3.p2.1.m1.5.6.3.2.3"><csymbol cd="ambiguous" id="S3.SS3.p2.1.m1.5.6.3.2.3.1.cmml" xref="S3.SS3.p2.1.m1.5.6.3.2.3">superscript</csymbol><set id="S3.SS3.p2.1.m1.5.6.3.2.3.2.1.cmml" xref="S3.SS3.p2.1.m1.5.6.3.2.3.2.2"><cn type="integer" id="S3.SS3.p2.1.m1.3.3.cmml" xref="S3.SS3.p2.1.m1.3.3">1</cn><ci id="S3.SS3.p2.1.m1.4.4.cmml" xref="S3.SS3.p2.1.m1.4.4">…</ci><ci id="S3.SS3.p2.1.m1.5.5.cmml" xref="S3.SS3.p2.1.m1.5.5">𝑛</ci></set><ci id="S3.SS3.p2.1.m1.5.6.3.2.3.3.cmml" xref="S3.SS3.p2.1.m1.5.6.3.2.3.3">𝑘</ci></apply></apply><ci id="S3.SS3.p2.1.m1.5.6.3.3.cmml" xref="S3.SS3.p2.1.m1.5.6.3.3">ℝ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.1.m1.5c">f(A,\mathcal{B}):\mathbb{R}^{n\times n}\times\{1,\ldots,n\}^{k}\rightarrow\mathbb{R}</annotation></semantics></math><span id="S3.SS3.p2.3.2" class="ltx_text ltx_font_typewriter">. This function quantifies the cohesion within events and separation between events based on the graph structure represented by the similarity matrix </span><math id="S3.SS3.p2.2.m2.1" class="ltx_Math" alttext="A" display="inline"><semantics id="S3.SS3.p2.2.m2.1a"><mi id="S3.SS3.p2.2.m2.1.1" xref="S3.SS3.p2.2.m2.1.1.cmml">A</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.2.m2.1b"><ci id="S3.SS3.p2.2.m2.1.1.cmml" xref="S3.SS3.p2.2.m2.1.1">𝐴</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.2.m2.1c">A</annotation></semantics></math><span id="S3.SS3.p2.3.3" class="ltx_text ltx_font_typewriter"> and event boundaries </span><math id="S3.SS3.p2.3.m3.1" class="ltx_Math" alttext="\mathcal{B}" display="inline"><semantics id="S3.SS3.p2.3.m3.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.3.m3.1.1" xref="S3.SS3.p2.3.m3.1.1.cmml">ℬ</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.3.m3.1b"><ci id="S3.SS3.p2.3.m3.1.1.cmml" xref="S3.SS3.p2.3.m3.1.1">ℬ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.3.m3.1c">\mathcal{B}</annotation></semantics></math><span id="S3.SS3.p2.3.4" class="ltx_text ltx_font_typewriter">. We experimented with two widely-accepted graph-clustering metrics: </span><span id="S3.SS3.p2.3.5" class="ltx_text ltx_font_italic">modularity</span><span id="S3.SS3.p2.3.6" class="ltx_text ltx_font_typewriter"> and </span><span id="S3.SS3.p2.3.7" class="ltx_text ltx_font_italic">conductance</span><span id="S3.SS3.p2.3.8" class="ltx_text ltx_font_typewriter"> </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S3.SS3.p2.3.9.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Miasnikof et al.</span><span id="S3.SS3.p2.3.10.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib70" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2018</span></a><span id="S3.SS3.p2.3.11.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S3.SS3.p2.3.12" class="ltx_text ltx_font_typewriter">. Modularity </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S3.SS3.p2.3.13.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Newman and Girvan</span><span id="S3.SS3.p2.3.14.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib71" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2004</span></a><span id="S3.SS3.p2.3.15.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S3.SS3.p2.3.16" class="ltx_text ltx_font_typewriter"> provides a measure of the quality of a particular division of a network into communities, with higher values indicating higher edge density in the identified cluster when compared to the density of edges expected in a random cluster. As our edge weights represent the similarity between two tokens, we seek to maximise this metric. Modularity is defined as:</span></p>
<table id="S3.E3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E3.m1.7" class="ltx_Math" alttext="f_{M}(A^{h},\mathcal{B})=\frac{1}{4m}\sum_{i,j}\left[A^{h}_{ij}-\frac{\sum_{i}A^{h}_{ij}\cdot\sum_{j}A^{h}_{ij}}{2m}\right]\delta(c_{i},c_{j})" display="block"><semantics id="S3.E3.m1.7a"><mrow id="S3.E3.m1.7.7" xref="S3.E3.m1.7.7.cmml"><mrow id="S3.E3.m1.4.4.1" xref="S3.E3.m1.4.4.1.cmml"><msub id="S3.E3.m1.4.4.1.3" xref="S3.E3.m1.4.4.1.3.cmml"><mi id="S3.E3.m1.4.4.1.3.2" xref="S3.E3.m1.4.4.1.3.2.cmml">f</mi><mi id="S3.E3.m1.4.4.1.3.3" xref="S3.E3.m1.4.4.1.3.3.cmml">M</mi></msub><mo lspace="0em" rspace="0em" id="S3.E3.m1.4.4.1.2" xref="S3.E3.m1.4.4.1.2.cmml">​</mo><mrow id="S3.E3.m1.4.4.1.1.1" xref="S3.E3.m1.4.4.1.1.2.cmml"><mo stretchy="false" id="S3.E3.m1.4.4.1.1.1.2" xref="S3.E3.m1.4.4.1.1.2.cmml">(</mo><msup id="S3.E3.m1.4.4.1.1.1.1" xref="S3.E3.m1.4.4.1.1.1.1.cmml"><mi id="S3.E3.m1.4.4.1.1.1.1.2" xref="S3.E3.m1.4.4.1.1.1.1.2.cmml">A</mi><mi id="S3.E3.m1.4.4.1.1.1.1.3" xref="S3.E3.m1.4.4.1.1.1.1.3.cmml">h</mi></msup><mo id="S3.E3.m1.4.4.1.1.1.3" xref="S3.E3.m1.4.4.1.1.2.cmml">,</mo><mi class="ltx_font_mathcaligraphic" id="S3.E3.m1.3.3" xref="S3.E3.m1.3.3.cmml">ℬ</mi><mo stretchy="false" id="S3.E3.m1.4.4.1.1.1.4" xref="S3.E3.m1.4.4.1.1.2.cmml">)</mo></mrow></mrow><mo id="S3.E3.m1.7.7.5" xref="S3.E3.m1.7.7.5.cmml">=</mo><mrow id="S3.E3.m1.7.7.4" xref="S3.E3.m1.7.7.4.cmml"><mfrac id="S3.E3.m1.7.7.4.5" xref="S3.E3.m1.7.7.4.5.cmml"><mn id="S3.E3.m1.7.7.4.5.2" xref="S3.E3.m1.7.7.4.5.2.cmml">1</mn><mrow id="S3.E3.m1.7.7.4.5.3" xref="S3.E3.m1.7.7.4.5.3.cmml"><mn id="S3.E3.m1.7.7.4.5.3.2" xref="S3.E3.m1.7.7.4.5.3.2.cmml">4</mn><mo lspace="0em" rspace="0em" id="S3.E3.m1.7.7.4.5.3.1" xref="S3.E3.m1.7.7.4.5.3.1.cmml">​</mo><mi id="S3.E3.m1.7.7.4.5.3.3" xref="S3.E3.m1.7.7.4.5.3.3.cmml">m</mi></mrow></mfrac><mo lspace="0em" rspace="0em" id="S3.E3.m1.7.7.4.4" xref="S3.E3.m1.7.7.4.4.cmml">​</mo><mrow id="S3.E3.m1.7.7.4.3" xref="S3.E3.m1.7.7.4.3.cmml"><munder id="S3.E3.m1.7.7.4.3.4" xref="S3.E3.m1.7.7.4.3.4.cmml"><mo movablelimits="false" rspace="0em" id="S3.E3.m1.7.7.4.3.4.2" xref="S3.E3.m1.7.7.4.3.4.2.cmml">∑</mo><mrow id="S3.E3.m1.2.2.2.4" xref="S3.E3.m1.2.2.2.3.cmml"><mi id="S3.E3.m1.1.1.1.1" xref="S3.E3.m1.1.1.1.1.cmml">i</mi><mo id="S3.E3.m1.2.2.2.4.1" xref="S3.E3.m1.2.2.2.3.cmml">,</mo><mi id="S3.E3.m1.2.2.2.2" xref="S3.E3.m1.2.2.2.2.cmml">j</mi></mrow></munder><mrow id="S3.E3.m1.7.7.4.3.3" xref="S3.E3.m1.7.7.4.3.3.cmml"><mrow id="S3.E3.m1.5.5.2.1.1.1.1" xref="S3.E3.m1.5.5.2.1.1.1.2.cmml"><mo id="S3.E3.m1.5.5.2.1.1.1.1.2" xref="S3.E3.m1.5.5.2.1.1.1.2.1.cmml">[</mo><mrow id="S3.E3.m1.5.5.2.1.1.1.1.1" xref="S3.E3.m1.5.5.2.1.1.1.1.1.cmml"><msubsup id="S3.E3.m1.5.5.2.1.1.1.1.1.2" xref="S3.E3.m1.5.5.2.1.1.1.1.1.2.cmml"><mi id="S3.E3.m1.5.5.2.1.1.1.1.1.2.2.2" xref="S3.E3.m1.5.5.2.1.1.1.1.1.2.2.2.cmml">A</mi><mrow id="S3.E3.m1.5.5.2.1.1.1.1.1.2.3" xref="S3.E3.m1.5.5.2.1.1.1.1.1.2.3.cmml"><mi id="S3.E3.m1.5.5.2.1.1.1.1.1.2.3.2" xref="S3.E3.m1.5.5.2.1.1.1.1.1.2.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S3.E3.m1.5.5.2.1.1.1.1.1.2.3.1" xref="S3.E3.m1.5.5.2.1.1.1.1.1.2.3.1.cmml">​</mo><mi id="S3.E3.m1.5.5.2.1.1.1.1.1.2.3.3" xref="S3.E3.m1.5.5.2.1.1.1.1.1.2.3.3.cmml">j</mi></mrow><mi id="S3.E3.m1.5.5.2.1.1.1.1.1.2.2.3" xref="S3.E3.m1.5.5.2.1.1.1.1.1.2.2.3.cmml">h</mi></msubsup><mo id="S3.E3.m1.5.5.2.1.1.1.1.1.1" xref="S3.E3.m1.5.5.2.1.1.1.1.1.1.cmml">−</mo><mfrac id="S3.E3.m1.5.5.2.1.1.1.1.1.3" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.cmml"><mrow id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.cmml"><msub id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.1" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.1.cmml"><mo id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.1.2" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.1.2.cmml">∑</mo><mi id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.1.3" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.1.3.cmml">i</mi></msub><mrow id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.cmml"><msubsup id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.cmml"><mi id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.2.2" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.2.2.cmml">A</mi><mrow id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.3" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.3.cmml"><mi id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.3.2" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.3.1" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.3.1.cmml">​</mo><mi id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.3.3" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.3.3.cmml">j</mi></mrow><mi id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.2.3" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.2.3.cmml">h</mi></msubsup><mo lspace="0.222em" rspace="0.055em" id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.1" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.1.cmml">⋅</mo><mrow id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.cmml"><msub id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.1" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.1.cmml"><mo id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.1.2" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.1.2.cmml">∑</mo><mi id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.1.3" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.1.3.cmml">j</mi></msub><msubsup id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.cmml"><mi id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.2.2" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.2.2.cmml">A</mi><mrow id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.3" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.3.cmml"><mi id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.3.2" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.3.1" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.3.1.cmml">​</mo><mi id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.3.3" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.3.3.cmml">j</mi></mrow><mi id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.2.3" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.2.3.cmml">h</mi></msubsup></mrow></mrow></mrow><mrow id="S3.E3.m1.5.5.2.1.1.1.1.1.3.3" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.3.cmml"><mn id="S3.E3.m1.5.5.2.1.1.1.1.1.3.3.2" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S3.E3.m1.5.5.2.1.1.1.1.1.3.3.1" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.3.1.cmml">​</mo><mi id="S3.E3.m1.5.5.2.1.1.1.1.1.3.3.3" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.3.3.cmml">m</mi></mrow></mfrac></mrow><mo id="S3.E3.m1.5.5.2.1.1.1.1.3" xref="S3.E3.m1.5.5.2.1.1.1.2.1.cmml">]</mo></mrow><mo lspace="0em" rspace="0em" id="S3.E3.m1.7.7.4.3.3.4" xref="S3.E3.m1.7.7.4.3.3.4.cmml">​</mo><mi id="S3.E3.m1.7.7.4.3.3.5" xref="S3.E3.m1.7.7.4.3.3.5.cmml">δ</mi><mo lspace="0em" rspace="0em" id="S3.E3.m1.7.7.4.3.3.4a" xref="S3.E3.m1.7.7.4.3.3.4.cmml">​</mo><mrow id="S3.E3.m1.7.7.4.3.3.3.2" xref="S3.E3.m1.7.7.4.3.3.3.3.cmml"><mo stretchy="false" id="S3.E3.m1.7.7.4.3.3.3.2.3" xref="S3.E3.m1.7.7.4.3.3.3.3.cmml">(</mo><msub id="S3.E3.m1.6.6.3.2.2.2.1.1" xref="S3.E3.m1.6.6.3.2.2.2.1.1.cmml"><mi id="S3.E3.m1.6.6.3.2.2.2.1.1.2" xref="S3.E3.m1.6.6.3.2.2.2.1.1.2.cmml">c</mi><mi id="S3.E3.m1.6.6.3.2.2.2.1.1.3" xref="S3.E3.m1.6.6.3.2.2.2.1.1.3.cmml">i</mi></msub><mo id="S3.E3.m1.7.7.4.3.3.3.2.4" xref="S3.E3.m1.7.7.4.3.3.3.3.cmml">,</mo><msub id="S3.E3.m1.7.7.4.3.3.3.2.2" xref="S3.E3.m1.7.7.4.3.3.3.2.2.cmml"><mi id="S3.E3.m1.7.7.4.3.3.3.2.2.2" xref="S3.E3.m1.7.7.4.3.3.3.2.2.2.cmml">c</mi><mi id="S3.E3.m1.7.7.4.3.3.3.2.2.3" xref="S3.E3.m1.7.7.4.3.3.3.2.2.3.cmml">j</mi></msub><mo stretchy="false" id="S3.E3.m1.7.7.4.3.3.3.2.5" xref="S3.E3.m1.7.7.4.3.3.3.3.cmml">)</mo></mrow></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E3.m1.7b"><apply id="S3.E3.m1.7.7.cmml" xref="S3.E3.m1.7.7"><eq id="S3.E3.m1.7.7.5.cmml" xref="S3.E3.m1.7.7.5"></eq><apply id="S3.E3.m1.4.4.1.cmml" xref="S3.E3.m1.4.4.1"><times id="S3.E3.m1.4.4.1.2.cmml" xref="S3.E3.m1.4.4.1.2"></times><apply id="S3.E3.m1.4.4.1.3.cmml" xref="S3.E3.m1.4.4.1.3"><csymbol cd="ambiguous" id="S3.E3.m1.4.4.1.3.1.cmml" xref="S3.E3.m1.4.4.1.3">subscript</csymbol><ci id="S3.E3.m1.4.4.1.3.2.cmml" xref="S3.E3.m1.4.4.1.3.2">𝑓</ci><ci id="S3.E3.m1.4.4.1.3.3.cmml" xref="S3.E3.m1.4.4.1.3.3">𝑀</ci></apply><interval closure="open" id="S3.E3.m1.4.4.1.1.2.cmml" xref="S3.E3.m1.4.4.1.1.1"><apply id="S3.E3.m1.4.4.1.1.1.1.cmml" xref="S3.E3.m1.4.4.1.1.1.1"><csymbol cd="ambiguous" id="S3.E3.m1.4.4.1.1.1.1.1.cmml" xref="S3.E3.m1.4.4.1.1.1.1">superscript</csymbol><ci id="S3.E3.m1.4.4.1.1.1.1.2.cmml" xref="S3.E3.m1.4.4.1.1.1.1.2">𝐴</ci><ci id="S3.E3.m1.4.4.1.1.1.1.3.cmml" xref="S3.E3.m1.4.4.1.1.1.1.3">ℎ</ci></apply><ci id="S3.E3.m1.3.3.cmml" xref="S3.E3.m1.3.3">ℬ</ci></interval></apply><apply id="S3.E3.m1.7.7.4.cmml" xref="S3.E3.m1.7.7.4"><times id="S3.E3.m1.7.7.4.4.cmml" xref="S3.E3.m1.7.7.4.4"></times><apply id="S3.E3.m1.7.7.4.5.cmml" xref="S3.E3.m1.7.7.4.5"><divide id="S3.E3.m1.7.7.4.5.1.cmml" xref="S3.E3.m1.7.7.4.5"></divide><cn type="integer" id="S3.E3.m1.7.7.4.5.2.cmml" xref="S3.E3.m1.7.7.4.5.2">1</cn><apply id="S3.E3.m1.7.7.4.5.3.cmml" xref="S3.E3.m1.7.7.4.5.3"><times id="S3.E3.m1.7.7.4.5.3.1.cmml" xref="S3.E3.m1.7.7.4.5.3.1"></times><cn type="integer" id="S3.E3.m1.7.7.4.5.3.2.cmml" xref="S3.E3.m1.7.7.4.5.3.2">4</cn><ci id="S3.E3.m1.7.7.4.5.3.3.cmml" xref="S3.E3.m1.7.7.4.5.3.3">𝑚</ci></apply></apply><apply id="S3.E3.m1.7.7.4.3.cmml" xref="S3.E3.m1.7.7.4.3"><apply id="S3.E3.m1.7.7.4.3.4.cmml" xref="S3.E3.m1.7.7.4.3.4"><csymbol cd="ambiguous" id="S3.E3.m1.7.7.4.3.4.1.cmml" xref="S3.E3.m1.7.7.4.3.4">subscript</csymbol><sum id="S3.E3.m1.7.7.4.3.4.2.cmml" xref="S3.E3.m1.7.7.4.3.4.2"></sum><list id="S3.E3.m1.2.2.2.3.cmml" xref="S3.E3.m1.2.2.2.4"><ci id="S3.E3.m1.1.1.1.1.cmml" xref="S3.E3.m1.1.1.1.1">𝑖</ci><ci id="S3.E3.m1.2.2.2.2.cmml" xref="S3.E3.m1.2.2.2.2">𝑗</ci></list></apply><apply id="S3.E3.m1.7.7.4.3.3.cmml" xref="S3.E3.m1.7.7.4.3.3"><times id="S3.E3.m1.7.7.4.3.3.4.cmml" xref="S3.E3.m1.7.7.4.3.3.4"></times><apply id="S3.E3.m1.5.5.2.1.1.1.2.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1"><csymbol cd="latexml" id="S3.E3.m1.5.5.2.1.1.1.2.1.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.2">delimited-[]</csymbol><apply id="S3.E3.m1.5.5.2.1.1.1.1.1.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1"><minus id="S3.E3.m1.5.5.2.1.1.1.1.1.1.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.1"></minus><apply id="S3.E3.m1.5.5.2.1.1.1.1.1.2.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E3.m1.5.5.2.1.1.1.1.1.2.1.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.2">subscript</csymbol><apply id="S3.E3.m1.5.5.2.1.1.1.1.1.2.2.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E3.m1.5.5.2.1.1.1.1.1.2.2.1.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.2">superscript</csymbol><ci id="S3.E3.m1.5.5.2.1.1.1.1.1.2.2.2.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.2.2.2">𝐴</ci><ci id="S3.E3.m1.5.5.2.1.1.1.1.1.2.2.3.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.2.2.3">ℎ</ci></apply><apply id="S3.E3.m1.5.5.2.1.1.1.1.1.2.3.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.2.3"><times id="S3.E3.m1.5.5.2.1.1.1.1.1.2.3.1.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.2.3.1"></times><ci id="S3.E3.m1.5.5.2.1.1.1.1.1.2.3.2.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.2.3.2">𝑖</ci><ci id="S3.E3.m1.5.5.2.1.1.1.1.1.2.3.3.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.2.3.3">𝑗</ci></apply></apply><apply id="S3.E3.m1.5.5.2.1.1.1.1.1.3.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3"><divide id="S3.E3.m1.5.5.2.1.1.1.1.1.3.1.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3"></divide><apply id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2"><apply id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.1.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.1"><csymbol cd="ambiguous" id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.1.1.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.1">subscript</csymbol><sum id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.1.2.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.1.2"></sum><ci id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.1.3.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.1.3">𝑖</ci></apply><apply id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2"><ci id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.1.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.1">⋅</ci><apply id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2"><csymbol cd="ambiguous" id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.1.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2">subscript</csymbol><apply id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.2.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2"><csymbol cd="ambiguous" id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.2.1.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2">superscript</csymbol><ci id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.2.2.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.2.2">𝐴</ci><ci id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.2.3.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.2.3">ℎ</ci></apply><apply id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.3.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.3"><times id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.3.1.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.3.1"></times><ci id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.3.2.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.3.2">𝑖</ci><ci id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.3.3.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.2.3.3">𝑗</ci></apply></apply><apply id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3"><apply id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.1.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.1"><csymbol cd="ambiguous" id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.1.1.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.1">subscript</csymbol><sum id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.1.2.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.1.2"></sum><ci id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.1.3.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.1.3">𝑗</ci></apply><apply id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2"><csymbol cd="ambiguous" id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.1.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2">subscript</csymbol><apply id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.2.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2"><csymbol cd="ambiguous" id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.2.1.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2">superscript</csymbol><ci id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.2.2.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.2.2">𝐴</ci><ci id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.2.3.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.2.3">ℎ</ci></apply><apply id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.3.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.3"><times id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.3.1.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.3.1"></times><ci id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.3.2.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.3.2">𝑖</ci><ci id="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.3.3.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.2.2.3.2.3.3">𝑗</ci></apply></apply></apply></apply></apply><apply id="S3.E3.m1.5.5.2.1.1.1.1.1.3.3.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.3"><times id="S3.E3.m1.5.5.2.1.1.1.1.1.3.3.1.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.3.1"></times><cn type="integer" id="S3.E3.m1.5.5.2.1.1.1.1.1.3.3.2.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.3.2">2</cn><ci id="S3.E3.m1.5.5.2.1.1.1.1.1.3.3.3.cmml" xref="S3.E3.m1.5.5.2.1.1.1.1.1.3.3.3">𝑚</ci></apply></apply></apply></apply><ci id="S3.E3.m1.7.7.4.3.3.5.cmml" xref="S3.E3.m1.7.7.4.3.3.5">𝛿</ci><interval closure="open" id="S3.E3.m1.7.7.4.3.3.3.3.cmml" xref="S3.E3.m1.7.7.4.3.3.3.2"><apply id="S3.E3.m1.6.6.3.2.2.2.1.1.cmml" xref="S3.E3.m1.6.6.3.2.2.2.1.1"><csymbol cd="ambiguous" id="S3.E3.m1.6.6.3.2.2.2.1.1.1.cmml" xref="S3.E3.m1.6.6.3.2.2.2.1.1">subscript</csymbol><ci id="S3.E3.m1.6.6.3.2.2.2.1.1.2.cmml" xref="S3.E3.m1.6.6.3.2.2.2.1.1.2">𝑐</ci><ci id="S3.E3.m1.6.6.3.2.2.2.1.1.3.cmml" xref="S3.E3.m1.6.6.3.2.2.2.1.1.3">𝑖</ci></apply><apply id="S3.E3.m1.7.7.4.3.3.3.2.2.cmml" xref="S3.E3.m1.7.7.4.3.3.3.2.2"><csymbol cd="ambiguous" id="S3.E3.m1.7.7.4.3.3.3.2.2.1.cmml" xref="S3.E3.m1.7.7.4.3.3.3.2.2">subscript</csymbol><ci id="S3.E3.m1.7.7.4.3.3.3.2.2.2.cmml" xref="S3.E3.m1.7.7.4.3.3.3.2.2.2">𝑐</ci><ci id="S3.E3.m1.7.7.4.3.3.3.2.2.3.cmml" xref="S3.E3.m1.7.7.4.3.3.3.2.2.3">𝑗</ci></apply></interval></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E3.m1.7c">f_{M}(A^{h},\mathcal{B})=\frac{1}{4m}\sum_{i,j}\left[A^{h}_{ij}-\frac{\sum_{i}A^{h}_{ij}\cdot\sum_{j}A^{h}_{ij}}{2m}\right]\delta(c_{i},c_{j})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
</table>
<p id="S3.SS3.p2.7" class="ltx_p"><span id="S3.SS3.p2.7.1" class="ltx_text ltx_font_typewriter">where </span><math id="S3.SS3.p2.4.m1.1" class="ltx_Math" alttext="m" display="inline"><semantics id="S3.SS3.p2.4.m1.1a"><mi id="S3.SS3.p2.4.m1.1.1" xref="S3.SS3.p2.4.m1.1.1.cmml">m</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.4.m1.1b"><ci id="S3.SS3.p2.4.m1.1.1.cmml" xref="S3.SS3.p2.4.m1.1.1">𝑚</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.4.m1.1c">m</annotation></semantics></math><span id="S3.SS3.p2.7.2" class="ltx_text ltx_font_typewriter"> is the total edge weight in the graph, </span><math id="S3.SS3.p2.5.m2.1" class="ltx_Math" alttext="c_{i}" display="inline"><semantics id="S3.SS3.p2.5.m2.1a"><msub id="S3.SS3.p2.5.m2.1.1" xref="S3.SS3.p2.5.m2.1.1.cmml"><mi id="S3.SS3.p2.5.m2.1.1.2" xref="S3.SS3.p2.5.m2.1.1.2.cmml">c</mi><mi id="S3.SS3.p2.5.m2.1.1.3" xref="S3.SS3.p2.5.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.5.m2.1b"><apply id="S3.SS3.p2.5.m2.1.1.cmml" xref="S3.SS3.p2.5.m2.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.5.m2.1.1.1.cmml" xref="S3.SS3.p2.5.m2.1.1">subscript</csymbol><ci id="S3.SS3.p2.5.m2.1.1.2.cmml" xref="S3.SS3.p2.5.m2.1.1.2">𝑐</ci><ci id="S3.SS3.p2.5.m2.1.1.3.cmml" xref="S3.SS3.p2.5.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.5.m2.1c">c_{i}</annotation></semantics></math><span id="S3.SS3.p2.7.3" class="ltx_text ltx_font_typewriter"> is the community (episodic event) to which node </span><math id="S3.SS3.p2.6.m3.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S3.SS3.p2.6.m3.1a"><mi id="S3.SS3.p2.6.m3.1.1" xref="S3.SS3.p2.6.m3.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.6.m3.1b"><ci id="S3.SS3.p2.6.m3.1.1.cmml" xref="S3.SS3.p2.6.m3.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.6.m3.1c">i</annotation></semantics></math><span id="S3.SS3.p2.7.4" class="ltx_text ltx_font_typewriter"> is assigned, and </span><math id="S3.SS3.p2.7.m4.1" class="ltx_Math" alttext="\delta" display="inline"><semantics id="S3.SS3.p2.7.m4.1a"><mi id="S3.SS3.p2.7.m4.1.1" xref="S3.SS3.p2.7.m4.1.1.cmml">δ</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.7.m4.1b"><ci id="S3.SS3.p2.7.m4.1.1.cmml" xref="S3.SS3.p2.7.m4.1.1">𝛿</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.7.m4.1c">\delta</annotation></semantics></math><span id="S3.SS3.p2.7.5" class="ltx_text ltx_font_typewriter"> is the Kronecker delta function.
Conductance, on the other hand, measures the fraction of total weighted edges cut by a given community boundary, and is defined as:</span></p>
<table id="S3.E4" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E4.m1.14" class="ltx_Math" alttext="f_{C}(A^{h},\mathcal{B})=\min_{S\in V}\frac{\sum_{i\in S,j\notin S}A^{h}_{ij}}{\min(\text{vol}(S),\text{vol}(V\setminus S))},\qquad\text{with}\;\text{vol}(S)=\sum_{i\in S,j\in S}A_{ij},\;\text{vol}(V\setminus S)=\sum_{i\notin S,j\notin S}A_{ij}" display="block"><semantics id="S3.E4.m1.14a"><mrow id="S3.E4.m1.14.14.2" xref="S3.E4.m1.14.14.3.cmml"><mrow id="S3.E4.m1.13.13.1.1" xref="S3.E4.m1.13.13.1.1.cmml"><mrow id="S3.E4.m1.13.13.1.1.1" xref="S3.E4.m1.13.13.1.1.1.cmml"><msub id="S3.E4.m1.13.13.1.1.1.3" xref="S3.E4.m1.13.13.1.1.1.3.cmml"><mi id="S3.E4.m1.13.13.1.1.1.3.2" xref="S3.E4.m1.13.13.1.1.1.3.2.cmml">f</mi><mi id="S3.E4.m1.13.13.1.1.1.3.3" xref="S3.E4.m1.13.13.1.1.1.3.3.cmml">C</mi></msub><mo lspace="0em" rspace="0em" id="S3.E4.m1.13.13.1.1.1.2" xref="S3.E4.m1.13.13.1.1.1.2.cmml">​</mo><mrow id="S3.E4.m1.13.13.1.1.1.1.1" xref="S3.E4.m1.13.13.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.E4.m1.13.13.1.1.1.1.1.2" xref="S3.E4.m1.13.13.1.1.1.1.2.cmml">(</mo><msup id="S3.E4.m1.13.13.1.1.1.1.1.1" xref="S3.E4.m1.13.13.1.1.1.1.1.1.cmml"><mi id="S3.E4.m1.13.13.1.1.1.1.1.1.2" xref="S3.E4.m1.13.13.1.1.1.1.1.1.2.cmml">A</mi><mi id="S3.E4.m1.13.13.1.1.1.1.1.1.3" xref="S3.E4.m1.13.13.1.1.1.1.1.1.3.cmml">h</mi></msup><mo id="S3.E4.m1.13.13.1.1.1.1.1.3" xref="S3.E4.m1.13.13.1.1.1.1.2.cmml">,</mo><mi class="ltx_font_mathcaligraphic" id="S3.E4.m1.11.11" xref="S3.E4.m1.11.11.cmml">ℬ</mi><mo stretchy="false" id="S3.E4.m1.13.13.1.1.1.1.1.4" xref="S3.E4.m1.13.13.1.1.1.1.2.cmml">)</mo></mrow></mrow><mo id="S3.E4.m1.13.13.1.1.2" xref="S3.E4.m1.13.13.1.1.2.cmml">=</mo><mrow id="S3.E4.m1.13.13.1.1.3" xref="S3.E4.m1.13.13.1.1.3.cmml"><munder id="S3.E4.m1.13.13.1.1.3.1" xref="S3.E4.m1.13.13.1.1.3.1.cmml"><mi id="S3.E4.m1.13.13.1.1.3.1.2" xref="S3.E4.m1.13.13.1.1.3.1.2.cmml">min</mi><mrow id="S3.E4.m1.13.13.1.1.3.1.3" xref="S3.E4.m1.13.13.1.1.3.1.3.cmml"><mi id="S3.E4.m1.13.13.1.1.3.1.3.2" xref="S3.E4.m1.13.13.1.1.3.1.3.2.cmml">S</mi><mo id="S3.E4.m1.13.13.1.1.3.1.3.1" xref="S3.E4.m1.13.13.1.1.3.1.3.1.cmml">∈</mo><mi id="S3.E4.m1.13.13.1.1.3.1.3.3" xref="S3.E4.m1.13.13.1.1.3.1.3.3.cmml">V</mi></mrow></munder><mo lspace="0.167em" id="S3.E4.m1.13.13.1.1.3a" xref="S3.E4.m1.13.13.1.1.3.cmml">⁡</mo><mfrac id="S3.E4.m1.6.6" xref="S3.E4.m1.6.6.cmml"><mrow id="S3.E4.m1.2.2.2" xref="S3.E4.m1.2.2.2.cmml"><msub id="S3.E4.m1.2.2.2.3" xref="S3.E4.m1.2.2.2.3.cmml"><mo id="S3.E4.m1.2.2.2.3.2" xref="S3.E4.m1.2.2.2.3.2.cmml">∑</mo><mrow id="S3.E4.m1.2.2.2.2.2.2" xref="S3.E4.m1.2.2.2.2.2.3.cmml"><mrow id="S3.E4.m1.1.1.1.1.1.1.1" xref="S3.E4.m1.1.1.1.1.1.1.1.cmml"><mi id="S3.E4.m1.1.1.1.1.1.1.1.2" xref="S3.E4.m1.1.1.1.1.1.1.1.2.cmml">i</mi><mo id="S3.E4.m1.1.1.1.1.1.1.1.1" xref="S3.E4.m1.1.1.1.1.1.1.1.1.cmml">∈</mo><mi id="S3.E4.m1.1.1.1.1.1.1.1.3" xref="S3.E4.m1.1.1.1.1.1.1.1.3.cmml">S</mi></mrow><mo id="S3.E4.m1.2.2.2.2.2.2.3" xref="S3.E4.m1.2.2.2.2.2.3a.cmml">,</mo><mrow id="S3.E4.m1.2.2.2.2.2.2.2" xref="S3.E4.m1.2.2.2.2.2.2.2.cmml"><mi id="S3.E4.m1.2.2.2.2.2.2.2.2" xref="S3.E4.m1.2.2.2.2.2.2.2.2.cmml">j</mi><mo id="S3.E4.m1.2.2.2.2.2.2.2.1" xref="S3.E4.m1.2.2.2.2.2.2.2.1.cmml">∉</mo><mi id="S3.E4.m1.2.2.2.2.2.2.2.3" xref="S3.E4.m1.2.2.2.2.2.2.2.3.cmml">S</mi></mrow></mrow></msub><msubsup id="S3.E4.m1.2.2.2.4" xref="S3.E4.m1.2.2.2.4.cmml"><mi id="S3.E4.m1.2.2.2.4.2.2" xref="S3.E4.m1.2.2.2.4.2.2.cmml">A</mi><mrow id="S3.E4.m1.2.2.2.4.3" xref="S3.E4.m1.2.2.2.4.3.cmml"><mi id="S3.E4.m1.2.2.2.4.3.2" xref="S3.E4.m1.2.2.2.4.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S3.E4.m1.2.2.2.4.3.1" xref="S3.E4.m1.2.2.2.4.3.1.cmml">​</mo><mi id="S3.E4.m1.2.2.2.4.3.3" xref="S3.E4.m1.2.2.2.4.3.3.cmml">j</mi></mrow><mi id="S3.E4.m1.2.2.2.4.2.3" xref="S3.E4.m1.2.2.2.4.2.3.cmml">h</mi></msubsup></mrow><mrow id="S3.E4.m1.6.6.6.4" xref="S3.E4.m1.6.6.6.5.cmml"><mi id="S3.E4.m1.4.4.4.2" xref="S3.E4.m1.4.4.4.2.cmml">min</mi><mo id="S3.E4.m1.6.6.6.4a" xref="S3.E4.m1.6.6.6.5.cmml">⁡</mo><mrow id="S3.E4.m1.6.6.6.4.2" xref="S3.E4.m1.6.6.6.5.cmml"><mo stretchy="false" id="S3.E4.m1.6.6.6.4.2.3" xref="S3.E4.m1.6.6.6.5.cmml">(</mo><mrow id="S3.E4.m1.5.5.5.3.1.1" xref="S3.E4.m1.5.5.5.3.1.1.cmml"><mtext class="ltx_mathvariant_monospace" id="S3.E4.m1.5.5.5.3.1.1.2" xref="S3.E4.m1.5.5.5.3.1.1.2a.cmml">vol</mtext><mo lspace="0em" rspace="0em" id="S3.E4.m1.5.5.5.3.1.1.1" xref="S3.E4.m1.5.5.5.3.1.1.1.cmml">​</mo><mrow id="S3.E4.m1.5.5.5.3.1.1.3.2" xref="S3.E4.m1.5.5.5.3.1.1.cmml"><mo stretchy="false" id="S3.E4.m1.5.5.5.3.1.1.3.2.1" xref="S3.E4.m1.5.5.5.3.1.1.cmml">(</mo><mi id="S3.E4.m1.3.3.3.1" xref="S3.E4.m1.3.3.3.1.cmml">S</mi><mo stretchy="false" id="S3.E4.m1.5.5.5.3.1.1.3.2.2" xref="S3.E4.m1.5.5.5.3.1.1.cmml">)</mo></mrow></mrow><mo id="S3.E4.m1.6.6.6.4.2.4" xref="S3.E4.m1.6.6.6.5.cmml">,</mo><mrow id="S3.E4.m1.6.6.6.4.2.2" xref="S3.E4.m1.6.6.6.4.2.2.cmml"><mtext class="ltx_mathvariant_monospace" id="S3.E4.m1.6.6.6.4.2.2.3" xref="S3.E4.m1.6.6.6.4.2.2.3a.cmml">vol</mtext><mo lspace="0em" rspace="0em" id="S3.E4.m1.6.6.6.4.2.2.2" xref="S3.E4.m1.6.6.6.4.2.2.2.cmml">​</mo><mrow id="S3.E4.m1.6.6.6.4.2.2.1.1" xref="S3.E4.m1.6.6.6.4.2.2.1.1.1.cmml"><mo stretchy="false" id="S3.E4.m1.6.6.6.4.2.2.1.1.2" xref="S3.E4.m1.6.6.6.4.2.2.1.1.1.cmml">(</mo><mrow id="S3.E4.m1.6.6.6.4.2.2.1.1.1" xref="S3.E4.m1.6.6.6.4.2.2.1.1.1.cmml"><mi id="S3.E4.m1.6.6.6.4.2.2.1.1.1.2" xref="S3.E4.m1.6.6.6.4.2.2.1.1.1.2.cmml">V</mi><mo id="S3.E4.m1.6.6.6.4.2.2.1.1.1.1" xref="S3.E4.m1.6.6.6.4.2.2.1.1.1.1.cmml">∖</mo><mi id="S3.E4.m1.6.6.6.4.2.2.1.1.1.3" xref="S3.E4.m1.6.6.6.4.2.2.1.1.1.3.cmml">S</mi></mrow><mo stretchy="false" id="S3.E4.m1.6.6.6.4.2.2.1.1.3" xref="S3.E4.m1.6.6.6.4.2.2.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S3.E4.m1.6.6.6.4.2.5" xref="S3.E4.m1.6.6.6.5.cmml">)</mo></mrow></mrow></mfrac></mrow></mrow><mo rspace="2.167em" id="S3.E4.m1.14.14.2.3" xref="S3.E4.m1.14.14.3a.cmml">,</mo><mrow id="S3.E4.m1.14.14.2.2.2" xref="S3.E4.m1.14.14.2.2.3.cmml"><mrow id="S3.E4.m1.14.14.2.2.1.1" xref="S3.E4.m1.14.14.2.2.1.1.cmml"><mrow id="S3.E4.m1.14.14.2.2.1.1.2" xref="S3.E4.m1.14.14.2.2.1.1.2.cmml"><mtext class="ltx_mathvariant_monospace" id="S3.E4.m1.14.14.2.2.1.1.2.2" xref="S3.E4.m1.14.14.2.2.1.1.2.2a.cmml">with</mtext><mo lspace="0.280em" rspace="0em" id="S3.E4.m1.14.14.2.2.1.1.2.1" xref="S3.E4.m1.14.14.2.2.1.1.2.1.cmml">​</mo><mtext class="ltx_mathvariant_monospace" id="S3.E4.m1.14.14.2.2.1.1.2.3" xref="S3.E4.m1.14.14.2.2.1.1.2.3a.cmml">vol</mtext><mo lspace="0em" rspace="0em" id="S3.E4.m1.14.14.2.2.1.1.2.1a" xref="S3.E4.m1.14.14.2.2.1.1.2.1.cmml">​</mo><mrow id="S3.E4.m1.14.14.2.2.1.1.2.4.2" xref="S3.E4.m1.14.14.2.2.1.1.2.cmml"><mo stretchy="false" id="S3.E4.m1.14.14.2.2.1.1.2.4.2.1" xref="S3.E4.m1.14.14.2.2.1.1.2.cmml">(</mo><mi id="S3.E4.m1.12.12" xref="S3.E4.m1.12.12.cmml">S</mi><mo stretchy="false" id="S3.E4.m1.14.14.2.2.1.1.2.4.2.2" xref="S3.E4.m1.14.14.2.2.1.1.2.cmml">)</mo></mrow></mrow><mo rspace="0.111em" id="S3.E4.m1.14.14.2.2.1.1.1" xref="S3.E4.m1.14.14.2.2.1.1.1.cmml">=</mo><mrow id="S3.E4.m1.14.14.2.2.1.1.3" xref="S3.E4.m1.14.14.2.2.1.1.3.cmml"><munder id="S3.E4.m1.14.14.2.2.1.1.3.1" xref="S3.E4.m1.14.14.2.2.1.1.3.1.cmml"><mo movablelimits="false" id="S3.E4.m1.14.14.2.2.1.1.3.1.2" xref="S3.E4.m1.14.14.2.2.1.1.3.1.2.cmml">∑</mo><mrow id="S3.E4.m1.8.8.2.2" xref="S3.E4.m1.8.8.2.3.cmml"><mrow id="S3.E4.m1.7.7.1.1.1" xref="S3.E4.m1.7.7.1.1.1.cmml"><mi id="S3.E4.m1.7.7.1.1.1.2" xref="S3.E4.m1.7.7.1.1.1.2.cmml">i</mi><mo id="S3.E4.m1.7.7.1.1.1.1" xref="S3.E4.m1.7.7.1.1.1.1.cmml">∈</mo><mi id="S3.E4.m1.7.7.1.1.1.3" xref="S3.E4.m1.7.7.1.1.1.3.cmml">S</mi></mrow><mo id="S3.E4.m1.8.8.2.2.3" xref="S3.E4.m1.8.8.2.3a.cmml">,</mo><mrow id="S3.E4.m1.8.8.2.2.2" xref="S3.E4.m1.8.8.2.2.2.cmml"><mi id="S3.E4.m1.8.8.2.2.2.2" xref="S3.E4.m1.8.8.2.2.2.2.cmml">j</mi><mo id="S3.E4.m1.8.8.2.2.2.1" xref="S3.E4.m1.8.8.2.2.2.1.cmml">∈</mo><mi id="S3.E4.m1.8.8.2.2.2.3" xref="S3.E4.m1.8.8.2.2.2.3.cmml">S</mi></mrow></mrow></munder><msub id="S3.E4.m1.14.14.2.2.1.1.3.2" xref="S3.E4.m1.14.14.2.2.1.1.3.2.cmml"><mi id="S3.E4.m1.14.14.2.2.1.1.3.2.2" xref="S3.E4.m1.14.14.2.2.1.1.3.2.2.cmml">A</mi><mrow id="S3.E4.m1.14.14.2.2.1.1.3.2.3" xref="S3.E4.m1.14.14.2.2.1.1.3.2.3.cmml"><mi id="S3.E4.m1.14.14.2.2.1.1.3.2.3.2" xref="S3.E4.m1.14.14.2.2.1.1.3.2.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S3.E4.m1.14.14.2.2.1.1.3.2.3.1" xref="S3.E4.m1.14.14.2.2.1.1.3.2.3.1.cmml">​</mo><mi id="S3.E4.m1.14.14.2.2.1.1.3.2.3.3" xref="S3.E4.m1.14.14.2.2.1.1.3.2.3.3.cmml">j</mi></mrow></msub></mrow></mrow><mo rspace="0.447em" id="S3.E4.m1.14.14.2.2.2.3" xref="S3.E4.m1.14.14.2.2.3a.cmml">,</mo><mrow id="S3.E4.m1.14.14.2.2.2.2" xref="S3.E4.m1.14.14.2.2.2.2.cmml"><mrow id="S3.E4.m1.14.14.2.2.2.2.1" xref="S3.E4.m1.14.14.2.2.2.2.1.cmml"><mtext class="ltx_mathvariant_monospace" id="S3.E4.m1.14.14.2.2.2.2.1.3" xref="S3.E4.m1.14.14.2.2.2.2.1.3a.cmml">vol</mtext><mo lspace="0em" rspace="0em" id="S3.E4.m1.14.14.2.2.2.2.1.2" xref="S3.E4.m1.14.14.2.2.2.2.1.2.cmml">​</mo><mrow id="S3.E4.m1.14.14.2.2.2.2.1.1.1" xref="S3.E4.m1.14.14.2.2.2.2.1.1.1.1.cmml"><mo stretchy="false" id="S3.E4.m1.14.14.2.2.2.2.1.1.1.2" xref="S3.E4.m1.14.14.2.2.2.2.1.1.1.1.cmml">(</mo><mrow id="S3.E4.m1.14.14.2.2.2.2.1.1.1.1" xref="S3.E4.m1.14.14.2.2.2.2.1.1.1.1.cmml"><mi id="S3.E4.m1.14.14.2.2.2.2.1.1.1.1.2" xref="S3.E4.m1.14.14.2.2.2.2.1.1.1.1.2.cmml">V</mi><mo id="S3.E4.m1.14.14.2.2.2.2.1.1.1.1.1" xref="S3.E4.m1.14.14.2.2.2.2.1.1.1.1.1.cmml">∖</mo><mi id="S3.E4.m1.14.14.2.2.2.2.1.1.1.1.3" xref="S3.E4.m1.14.14.2.2.2.2.1.1.1.1.3.cmml">S</mi></mrow><mo stretchy="false" id="S3.E4.m1.14.14.2.2.2.2.1.1.1.3" xref="S3.E4.m1.14.14.2.2.2.2.1.1.1.1.cmml">)</mo></mrow></mrow><mo rspace="0.111em" id="S3.E4.m1.14.14.2.2.2.2.2" xref="S3.E4.m1.14.14.2.2.2.2.2.cmml">=</mo><mrow id="S3.E4.m1.14.14.2.2.2.2.3" xref="S3.E4.m1.14.14.2.2.2.2.3.cmml"><munder id="S3.E4.m1.14.14.2.2.2.2.3.1" xref="S3.E4.m1.14.14.2.2.2.2.3.1.cmml"><mo movablelimits="false" id="S3.E4.m1.14.14.2.2.2.2.3.1.2" xref="S3.E4.m1.14.14.2.2.2.2.3.1.2.cmml">∑</mo><mrow id="S3.E4.m1.10.10.2.2" xref="S3.E4.m1.10.10.2.3.cmml"><mrow id="S3.E4.m1.9.9.1.1.1" xref="S3.E4.m1.9.9.1.1.1.cmml"><mi id="S3.E4.m1.9.9.1.1.1.2" xref="S3.E4.m1.9.9.1.1.1.2.cmml">i</mi><mo id="S3.E4.m1.9.9.1.1.1.1" xref="S3.E4.m1.9.9.1.1.1.1.cmml">∉</mo><mi id="S3.E4.m1.9.9.1.1.1.3" xref="S3.E4.m1.9.9.1.1.1.3.cmml">S</mi></mrow><mo id="S3.E4.m1.10.10.2.2.3" xref="S3.E4.m1.10.10.2.3a.cmml">,</mo><mrow id="S3.E4.m1.10.10.2.2.2" xref="S3.E4.m1.10.10.2.2.2.cmml"><mi id="S3.E4.m1.10.10.2.2.2.2" xref="S3.E4.m1.10.10.2.2.2.2.cmml">j</mi><mo id="S3.E4.m1.10.10.2.2.2.1" xref="S3.E4.m1.10.10.2.2.2.1.cmml">∉</mo><mi id="S3.E4.m1.10.10.2.2.2.3" xref="S3.E4.m1.10.10.2.2.2.3.cmml">S</mi></mrow></mrow></munder><msub id="S3.E4.m1.14.14.2.2.2.2.3.2" xref="S3.E4.m1.14.14.2.2.2.2.3.2.cmml"><mi id="S3.E4.m1.14.14.2.2.2.2.3.2.2" xref="S3.E4.m1.14.14.2.2.2.2.3.2.2.cmml">A</mi><mrow id="S3.E4.m1.14.14.2.2.2.2.3.2.3" xref="S3.E4.m1.14.14.2.2.2.2.3.2.3.cmml"><mi id="S3.E4.m1.14.14.2.2.2.2.3.2.3.2" xref="S3.E4.m1.14.14.2.2.2.2.3.2.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S3.E4.m1.14.14.2.2.2.2.3.2.3.1" xref="S3.E4.m1.14.14.2.2.2.2.3.2.3.1.cmml">​</mo><mi id="S3.E4.m1.14.14.2.2.2.2.3.2.3.3" xref="S3.E4.m1.14.14.2.2.2.2.3.2.3.3.cmml">j</mi></mrow></msub></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E4.m1.14b"><apply id="S3.E4.m1.14.14.3.cmml" xref="S3.E4.m1.14.14.2"><csymbol cd="ambiguous" id="S3.E4.m1.14.14.3a.cmml" xref="S3.E4.m1.14.14.2.3">formulae-sequence</csymbol><apply id="S3.E4.m1.13.13.1.1.cmml" xref="S3.E4.m1.13.13.1.1"><eq id="S3.E4.m1.13.13.1.1.2.cmml" xref="S3.E4.m1.13.13.1.1.2"></eq><apply id="S3.E4.m1.13.13.1.1.1.cmml" xref="S3.E4.m1.13.13.1.1.1"><times id="S3.E4.m1.13.13.1.1.1.2.cmml" xref="S3.E4.m1.13.13.1.1.1.2"></times><apply id="S3.E4.m1.13.13.1.1.1.3.cmml" xref="S3.E4.m1.13.13.1.1.1.3"><csymbol cd="ambiguous" id="S3.E4.m1.13.13.1.1.1.3.1.cmml" xref="S3.E4.m1.13.13.1.1.1.3">subscript</csymbol><ci id="S3.E4.m1.13.13.1.1.1.3.2.cmml" xref="S3.E4.m1.13.13.1.1.1.3.2">𝑓</ci><ci id="S3.E4.m1.13.13.1.1.1.3.3.cmml" xref="S3.E4.m1.13.13.1.1.1.3.3">𝐶</ci></apply><interval closure="open" id="S3.E4.m1.13.13.1.1.1.1.2.cmml" xref="S3.E4.m1.13.13.1.1.1.1.1"><apply id="S3.E4.m1.13.13.1.1.1.1.1.1.cmml" xref="S3.E4.m1.13.13.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E4.m1.13.13.1.1.1.1.1.1.1.cmml" xref="S3.E4.m1.13.13.1.1.1.1.1.1">superscript</csymbol><ci id="S3.E4.m1.13.13.1.1.1.1.1.1.2.cmml" xref="S3.E4.m1.13.13.1.1.1.1.1.1.2">𝐴</ci><ci id="S3.E4.m1.13.13.1.1.1.1.1.1.3.cmml" xref="S3.E4.m1.13.13.1.1.1.1.1.1.3">ℎ</ci></apply><ci id="S3.E4.m1.11.11.cmml" xref="S3.E4.m1.11.11">ℬ</ci></interval></apply><apply id="S3.E4.m1.13.13.1.1.3.cmml" xref="S3.E4.m1.13.13.1.1.3"><apply id="S3.E4.m1.13.13.1.1.3.1.cmml" xref="S3.E4.m1.13.13.1.1.3.1"><csymbol cd="ambiguous" id="S3.E4.m1.13.13.1.1.3.1.1.cmml" xref="S3.E4.m1.13.13.1.1.3.1">subscript</csymbol><min id="S3.E4.m1.13.13.1.1.3.1.2.cmml" xref="S3.E4.m1.13.13.1.1.3.1.2"></min><apply id="S3.E4.m1.13.13.1.1.3.1.3.cmml" xref="S3.E4.m1.13.13.1.1.3.1.3"><in id="S3.E4.m1.13.13.1.1.3.1.3.1.cmml" xref="S3.E4.m1.13.13.1.1.3.1.3.1"></in><ci id="S3.E4.m1.13.13.1.1.3.1.3.2.cmml" xref="S3.E4.m1.13.13.1.1.3.1.3.2">𝑆</ci><ci id="S3.E4.m1.13.13.1.1.3.1.3.3.cmml" xref="S3.E4.m1.13.13.1.1.3.1.3.3">𝑉</ci></apply></apply><apply id="S3.E4.m1.6.6.cmml" xref="S3.E4.m1.6.6"><divide id="S3.E4.m1.6.6.7.cmml" xref="S3.E4.m1.6.6"></divide><apply id="S3.E4.m1.2.2.2.cmml" xref="S3.E4.m1.2.2.2"><apply id="S3.E4.m1.2.2.2.3.cmml" xref="S3.E4.m1.2.2.2.3"><csymbol cd="ambiguous" id="S3.E4.m1.2.2.2.3.1.cmml" xref="S3.E4.m1.2.2.2.3">subscript</csymbol><sum id="S3.E4.m1.2.2.2.3.2.cmml" xref="S3.E4.m1.2.2.2.3.2"></sum><apply id="S3.E4.m1.2.2.2.2.2.3.cmml" xref="S3.E4.m1.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.E4.m1.2.2.2.2.2.3a.cmml" xref="S3.E4.m1.2.2.2.2.2.2.3">formulae-sequence</csymbol><apply id="S3.E4.m1.1.1.1.1.1.1.1.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1"><in id="S3.E4.m1.1.1.1.1.1.1.1.1.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.1"></in><ci id="S3.E4.m1.1.1.1.1.1.1.1.2.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.2">𝑖</ci><ci id="S3.E4.m1.1.1.1.1.1.1.1.3.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.3">𝑆</ci></apply><apply id="S3.E4.m1.2.2.2.2.2.2.2.cmml" xref="S3.E4.m1.2.2.2.2.2.2.2"><notin id="S3.E4.m1.2.2.2.2.2.2.2.1.cmml" xref="S3.E4.m1.2.2.2.2.2.2.2.1"></notin><ci id="S3.E4.m1.2.2.2.2.2.2.2.2.cmml" xref="S3.E4.m1.2.2.2.2.2.2.2.2">𝑗</ci><ci id="S3.E4.m1.2.2.2.2.2.2.2.3.cmml" xref="S3.E4.m1.2.2.2.2.2.2.2.3">𝑆</ci></apply></apply></apply><apply id="S3.E4.m1.2.2.2.4.cmml" xref="S3.E4.m1.2.2.2.4"><csymbol cd="ambiguous" id="S3.E4.m1.2.2.2.4.1.cmml" xref="S3.E4.m1.2.2.2.4">subscript</csymbol><apply id="S3.E4.m1.2.2.2.4.2.cmml" xref="S3.E4.m1.2.2.2.4"><csymbol cd="ambiguous" id="S3.E4.m1.2.2.2.4.2.1.cmml" xref="S3.E4.m1.2.2.2.4">superscript</csymbol><ci id="S3.E4.m1.2.2.2.4.2.2.cmml" xref="S3.E4.m1.2.2.2.4.2.2">𝐴</ci><ci id="S3.E4.m1.2.2.2.4.2.3.cmml" xref="S3.E4.m1.2.2.2.4.2.3">ℎ</ci></apply><apply id="S3.E4.m1.2.2.2.4.3.cmml" xref="S3.E4.m1.2.2.2.4.3"><times id="S3.E4.m1.2.2.2.4.3.1.cmml" xref="S3.E4.m1.2.2.2.4.3.1"></times><ci id="S3.E4.m1.2.2.2.4.3.2.cmml" xref="S3.E4.m1.2.2.2.4.3.2">𝑖</ci><ci id="S3.E4.m1.2.2.2.4.3.3.cmml" xref="S3.E4.m1.2.2.2.4.3.3">𝑗</ci></apply></apply></apply><apply id="S3.E4.m1.6.6.6.5.cmml" xref="S3.E4.m1.6.6.6.4"><min id="S3.E4.m1.4.4.4.2.cmml" xref="S3.E4.m1.4.4.4.2"></min><apply id="S3.E4.m1.5.5.5.3.1.1.cmml" xref="S3.E4.m1.5.5.5.3.1.1"><times id="S3.E4.m1.5.5.5.3.1.1.1.cmml" xref="S3.E4.m1.5.5.5.3.1.1.1"></times><ci id="S3.E4.m1.5.5.5.3.1.1.2a.cmml" xref="S3.E4.m1.5.5.5.3.1.1.2"><mtext class="ltx_mathvariant_monospace" id="S3.E4.m1.5.5.5.3.1.1.2.cmml" xref="S3.E4.m1.5.5.5.3.1.1.2">vol</mtext></ci><ci id="S3.E4.m1.3.3.3.1.cmml" xref="S3.E4.m1.3.3.3.1">𝑆</ci></apply><apply id="S3.E4.m1.6.6.6.4.2.2.cmml" xref="S3.E4.m1.6.6.6.4.2.2"><times id="S3.E4.m1.6.6.6.4.2.2.2.cmml" xref="S3.E4.m1.6.6.6.4.2.2.2"></times><ci id="S3.E4.m1.6.6.6.4.2.2.3a.cmml" xref="S3.E4.m1.6.6.6.4.2.2.3"><mtext class="ltx_mathvariant_monospace" id="S3.E4.m1.6.6.6.4.2.2.3.cmml" xref="S3.E4.m1.6.6.6.4.2.2.3">vol</mtext></ci><apply id="S3.E4.m1.6.6.6.4.2.2.1.1.1.cmml" xref="S3.E4.m1.6.6.6.4.2.2.1.1"><setdiff id="S3.E4.m1.6.6.6.4.2.2.1.1.1.1.cmml" xref="S3.E4.m1.6.6.6.4.2.2.1.1.1.1"></setdiff><ci id="S3.E4.m1.6.6.6.4.2.2.1.1.1.2.cmml" xref="S3.E4.m1.6.6.6.4.2.2.1.1.1.2">𝑉</ci><ci id="S3.E4.m1.6.6.6.4.2.2.1.1.1.3.cmml" xref="S3.E4.m1.6.6.6.4.2.2.1.1.1.3">𝑆</ci></apply></apply></apply></apply></apply></apply><apply id="S3.E4.m1.14.14.2.2.3.cmml" xref="S3.E4.m1.14.14.2.2.2"><csymbol cd="ambiguous" id="S3.E4.m1.14.14.2.2.3a.cmml" xref="S3.E4.m1.14.14.2.2.2.3">formulae-sequence</csymbol><apply id="S3.E4.m1.14.14.2.2.1.1.cmml" xref="S3.E4.m1.14.14.2.2.1.1"><eq id="S3.E4.m1.14.14.2.2.1.1.1.cmml" xref="S3.E4.m1.14.14.2.2.1.1.1"></eq><apply id="S3.E4.m1.14.14.2.2.1.1.2.cmml" xref="S3.E4.m1.14.14.2.2.1.1.2"><times id="S3.E4.m1.14.14.2.2.1.1.2.1.cmml" xref="S3.E4.m1.14.14.2.2.1.1.2.1"></times><ci id="S3.E4.m1.14.14.2.2.1.1.2.2a.cmml" xref="S3.E4.m1.14.14.2.2.1.1.2.2"><mtext class="ltx_mathvariant_monospace" id="S3.E4.m1.14.14.2.2.1.1.2.2.cmml" xref="S3.E4.m1.14.14.2.2.1.1.2.2">with</mtext></ci><ci id="S3.E4.m1.14.14.2.2.1.1.2.3a.cmml" xref="S3.E4.m1.14.14.2.2.1.1.2.3"><mtext class="ltx_mathvariant_monospace" id="S3.E4.m1.14.14.2.2.1.1.2.3.cmml" xref="S3.E4.m1.14.14.2.2.1.1.2.3">vol</mtext></ci><ci id="S3.E4.m1.12.12.cmml" xref="S3.E4.m1.12.12">𝑆</ci></apply><apply id="S3.E4.m1.14.14.2.2.1.1.3.cmml" xref="S3.E4.m1.14.14.2.2.1.1.3"><apply id="S3.E4.m1.14.14.2.2.1.1.3.1.cmml" xref="S3.E4.m1.14.14.2.2.1.1.3.1"><csymbol cd="ambiguous" id="S3.E4.m1.14.14.2.2.1.1.3.1.1.cmml" xref="S3.E4.m1.14.14.2.2.1.1.3.1">subscript</csymbol><sum id="S3.E4.m1.14.14.2.2.1.1.3.1.2.cmml" xref="S3.E4.m1.14.14.2.2.1.1.3.1.2"></sum><apply id="S3.E4.m1.8.8.2.3.cmml" xref="S3.E4.m1.8.8.2.2"><csymbol cd="ambiguous" id="S3.E4.m1.8.8.2.3a.cmml" xref="S3.E4.m1.8.8.2.2.3">formulae-sequence</csymbol><apply id="S3.E4.m1.7.7.1.1.1.cmml" xref="S3.E4.m1.7.7.1.1.1"><in id="S3.E4.m1.7.7.1.1.1.1.cmml" xref="S3.E4.m1.7.7.1.1.1.1"></in><ci id="S3.E4.m1.7.7.1.1.1.2.cmml" xref="S3.E4.m1.7.7.1.1.1.2">𝑖</ci><ci id="S3.E4.m1.7.7.1.1.1.3.cmml" xref="S3.E4.m1.7.7.1.1.1.3">𝑆</ci></apply><apply id="S3.E4.m1.8.8.2.2.2.cmml" xref="S3.E4.m1.8.8.2.2.2"><in id="S3.E4.m1.8.8.2.2.2.1.cmml" xref="S3.E4.m1.8.8.2.2.2.1"></in><ci id="S3.E4.m1.8.8.2.2.2.2.cmml" xref="S3.E4.m1.8.8.2.2.2.2">𝑗</ci><ci id="S3.E4.m1.8.8.2.2.2.3.cmml" xref="S3.E4.m1.8.8.2.2.2.3">𝑆</ci></apply></apply></apply><apply id="S3.E4.m1.14.14.2.2.1.1.3.2.cmml" xref="S3.E4.m1.14.14.2.2.1.1.3.2"><csymbol cd="ambiguous" id="S3.E4.m1.14.14.2.2.1.1.3.2.1.cmml" xref="S3.E4.m1.14.14.2.2.1.1.3.2">subscript</csymbol><ci id="S3.E4.m1.14.14.2.2.1.1.3.2.2.cmml" xref="S3.E4.m1.14.14.2.2.1.1.3.2.2">𝐴</ci><apply id="S3.E4.m1.14.14.2.2.1.1.3.2.3.cmml" xref="S3.E4.m1.14.14.2.2.1.1.3.2.3"><times id="S3.E4.m1.14.14.2.2.1.1.3.2.3.1.cmml" xref="S3.E4.m1.14.14.2.2.1.1.3.2.3.1"></times><ci id="S3.E4.m1.14.14.2.2.1.1.3.2.3.2.cmml" xref="S3.E4.m1.14.14.2.2.1.1.3.2.3.2">𝑖</ci><ci id="S3.E4.m1.14.14.2.2.1.1.3.2.3.3.cmml" xref="S3.E4.m1.14.14.2.2.1.1.3.2.3.3">𝑗</ci></apply></apply></apply></apply><apply id="S3.E4.m1.14.14.2.2.2.2.cmml" xref="S3.E4.m1.14.14.2.2.2.2"><eq id="S3.E4.m1.14.14.2.2.2.2.2.cmml" xref="S3.E4.m1.14.14.2.2.2.2.2"></eq><apply id="S3.E4.m1.14.14.2.2.2.2.1.cmml" xref="S3.E4.m1.14.14.2.2.2.2.1"><times id="S3.E4.m1.14.14.2.2.2.2.1.2.cmml" xref="S3.E4.m1.14.14.2.2.2.2.1.2"></times><ci id="S3.E4.m1.14.14.2.2.2.2.1.3a.cmml" xref="S3.E4.m1.14.14.2.2.2.2.1.3"><mtext class="ltx_mathvariant_monospace" id="S3.E4.m1.14.14.2.2.2.2.1.3.cmml" xref="S3.E4.m1.14.14.2.2.2.2.1.3">vol</mtext></ci><apply id="S3.E4.m1.14.14.2.2.2.2.1.1.1.1.cmml" xref="S3.E4.m1.14.14.2.2.2.2.1.1.1"><setdiff id="S3.E4.m1.14.14.2.2.2.2.1.1.1.1.1.cmml" xref="S3.E4.m1.14.14.2.2.2.2.1.1.1.1.1"></setdiff><ci id="S3.E4.m1.14.14.2.2.2.2.1.1.1.1.2.cmml" xref="S3.E4.m1.14.14.2.2.2.2.1.1.1.1.2">𝑉</ci><ci id="S3.E4.m1.14.14.2.2.2.2.1.1.1.1.3.cmml" xref="S3.E4.m1.14.14.2.2.2.2.1.1.1.1.3">𝑆</ci></apply></apply><apply id="S3.E4.m1.14.14.2.2.2.2.3.cmml" xref="S3.E4.m1.14.14.2.2.2.2.3"><apply id="S3.E4.m1.14.14.2.2.2.2.3.1.cmml" xref="S3.E4.m1.14.14.2.2.2.2.3.1"><csymbol cd="ambiguous" id="S3.E4.m1.14.14.2.2.2.2.3.1.1.cmml" xref="S3.E4.m1.14.14.2.2.2.2.3.1">subscript</csymbol><sum id="S3.E4.m1.14.14.2.2.2.2.3.1.2.cmml" xref="S3.E4.m1.14.14.2.2.2.2.3.1.2"></sum><apply id="S3.E4.m1.10.10.2.3.cmml" xref="S3.E4.m1.10.10.2.2"><csymbol cd="ambiguous" id="S3.E4.m1.10.10.2.3a.cmml" xref="S3.E4.m1.10.10.2.2.3">formulae-sequence</csymbol><apply id="S3.E4.m1.9.9.1.1.1.cmml" xref="S3.E4.m1.9.9.1.1.1"><notin id="S3.E4.m1.9.9.1.1.1.1.cmml" xref="S3.E4.m1.9.9.1.1.1.1"></notin><ci id="S3.E4.m1.9.9.1.1.1.2.cmml" xref="S3.E4.m1.9.9.1.1.1.2">𝑖</ci><ci id="S3.E4.m1.9.9.1.1.1.3.cmml" xref="S3.E4.m1.9.9.1.1.1.3">𝑆</ci></apply><apply id="S3.E4.m1.10.10.2.2.2.cmml" xref="S3.E4.m1.10.10.2.2.2"><notin id="S3.E4.m1.10.10.2.2.2.1.cmml" xref="S3.E4.m1.10.10.2.2.2.1"></notin><ci id="S3.E4.m1.10.10.2.2.2.2.cmml" xref="S3.E4.m1.10.10.2.2.2.2">𝑗</ci><ci id="S3.E4.m1.10.10.2.2.2.3.cmml" xref="S3.E4.m1.10.10.2.2.2.3">𝑆</ci></apply></apply></apply><apply id="S3.E4.m1.14.14.2.2.2.2.3.2.cmml" xref="S3.E4.m1.14.14.2.2.2.2.3.2"><csymbol cd="ambiguous" id="S3.E4.m1.14.14.2.2.2.2.3.2.1.cmml" xref="S3.E4.m1.14.14.2.2.2.2.3.2">subscript</csymbol><ci id="S3.E4.m1.14.14.2.2.2.2.3.2.2.cmml" xref="S3.E4.m1.14.14.2.2.2.2.3.2.2">𝐴</ci><apply id="S3.E4.m1.14.14.2.2.2.2.3.2.3.cmml" xref="S3.E4.m1.14.14.2.2.2.2.3.2.3"><times id="S3.E4.m1.14.14.2.2.2.2.3.2.3.1.cmml" xref="S3.E4.m1.14.14.2.2.2.2.3.2.3.1"></times><ci id="S3.E4.m1.14.14.2.2.2.2.3.2.3.2.cmml" xref="S3.E4.m1.14.14.2.2.2.2.3.2.3.2">𝑖</ci><ci id="S3.E4.m1.14.14.2.2.2.2.3.2.3.3.cmml" xref="S3.E4.m1.14.14.2.2.2.2.3.2.3.3">𝑗</ci></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E4.m1.14c">f_{C}(A^{h},\mathcal{B})=\min_{S\in V}\frac{\sum_{i\in S,j\notin S}A^{h}_{ij}}{\min(\text{vol}(S),\text{vol}(V\setminus S))},\qquad\text{with}\;\text{vol}(S)=\sum_{i\in S,j\in S}A_{ij},\;\text{vol}(V\setminus S)=\sum_{i\notin S,j\notin S}A_{ij}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(4)</span></td>
</tr></tbody>
</table>
<p id="S3.SS3.p2.10" class="ltx_p"><span id="S3.SS3.p2.10.1" class="ltx_text ltx_font_typewriter">where </span><math id="S3.SS3.p2.8.m1.4" class="ltx_Math" alttext="S=\{b_{i},b_{i}+1,...,b_{i+1}\}" display="inline"><semantics id="S3.SS3.p2.8.m1.4a"><mrow id="S3.SS3.p2.8.m1.4.4" xref="S3.SS3.p2.8.m1.4.4.cmml"><mi id="S3.SS3.p2.8.m1.4.4.5" xref="S3.SS3.p2.8.m1.4.4.5.cmml">S</mi><mo id="S3.SS3.p2.8.m1.4.4.4" xref="S3.SS3.p2.8.m1.4.4.4.cmml">=</mo><mrow id="S3.SS3.p2.8.m1.4.4.3.3" xref="S3.SS3.p2.8.m1.4.4.3.4.cmml"><mo stretchy="false" id="S3.SS3.p2.8.m1.4.4.3.3.4" xref="S3.SS3.p2.8.m1.4.4.3.4.cmml">{</mo><msub id="S3.SS3.p2.8.m1.2.2.1.1.1" xref="S3.SS3.p2.8.m1.2.2.1.1.1.cmml"><mi id="S3.SS3.p2.8.m1.2.2.1.1.1.2" xref="S3.SS3.p2.8.m1.2.2.1.1.1.2.cmml">b</mi><mi id="S3.SS3.p2.8.m1.2.2.1.1.1.3" xref="S3.SS3.p2.8.m1.2.2.1.1.1.3.cmml">i</mi></msub><mo id="S3.SS3.p2.8.m1.4.4.3.3.5" xref="S3.SS3.p2.8.m1.4.4.3.4.cmml">,</mo><mrow id="S3.SS3.p2.8.m1.3.3.2.2.2" xref="S3.SS3.p2.8.m1.3.3.2.2.2.cmml"><msub id="S3.SS3.p2.8.m1.3.3.2.2.2.2" xref="S3.SS3.p2.8.m1.3.3.2.2.2.2.cmml"><mi id="S3.SS3.p2.8.m1.3.3.2.2.2.2.2" xref="S3.SS3.p2.8.m1.3.3.2.2.2.2.2.cmml">b</mi><mi id="S3.SS3.p2.8.m1.3.3.2.2.2.2.3" xref="S3.SS3.p2.8.m1.3.3.2.2.2.2.3.cmml">i</mi></msub><mo id="S3.SS3.p2.8.m1.3.3.2.2.2.1" xref="S3.SS3.p2.8.m1.3.3.2.2.2.1.cmml">+</mo><mn id="S3.SS3.p2.8.m1.3.3.2.2.2.3" xref="S3.SS3.p2.8.m1.3.3.2.2.2.3.cmml">1</mn></mrow><mo id="S3.SS3.p2.8.m1.4.4.3.3.6" xref="S3.SS3.p2.8.m1.4.4.3.4.cmml">,</mo><mi mathvariant="normal" id="S3.SS3.p2.8.m1.1.1" xref="S3.SS3.p2.8.m1.1.1.cmml">…</mi><mo id="S3.SS3.p2.8.m1.4.4.3.3.7" xref="S3.SS3.p2.8.m1.4.4.3.4.cmml">,</mo><msub id="S3.SS3.p2.8.m1.4.4.3.3.3" xref="S3.SS3.p2.8.m1.4.4.3.3.3.cmml"><mi id="S3.SS3.p2.8.m1.4.4.3.3.3.2" xref="S3.SS3.p2.8.m1.4.4.3.3.3.2.cmml">b</mi><mrow id="S3.SS3.p2.8.m1.4.4.3.3.3.3" xref="S3.SS3.p2.8.m1.4.4.3.3.3.3.cmml"><mi id="S3.SS3.p2.8.m1.4.4.3.3.3.3.2" xref="S3.SS3.p2.8.m1.4.4.3.3.3.3.2.cmml">i</mi><mo id="S3.SS3.p2.8.m1.4.4.3.3.3.3.1" xref="S3.SS3.p2.8.m1.4.4.3.3.3.3.1.cmml">+</mo><mn id="S3.SS3.p2.8.m1.4.4.3.3.3.3.3" xref="S3.SS3.p2.8.m1.4.4.3.3.3.3.3.cmml">1</mn></mrow></msub><mo stretchy="false" id="S3.SS3.p2.8.m1.4.4.3.3.8" xref="S3.SS3.p2.8.m1.4.4.3.4.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.8.m1.4b"><apply id="S3.SS3.p2.8.m1.4.4.cmml" xref="S3.SS3.p2.8.m1.4.4"><eq id="S3.SS3.p2.8.m1.4.4.4.cmml" xref="S3.SS3.p2.8.m1.4.4.4"></eq><ci id="S3.SS3.p2.8.m1.4.4.5.cmml" xref="S3.SS3.p2.8.m1.4.4.5">𝑆</ci><set id="S3.SS3.p2.8.m1.4.4.3.4.cmml" xref="S3.SS3.p2.8.m1.4.4.3.3"><apply id="S3.SS3.p2.8.m1.2.2.1.1.1.cmml" xref="S3.SS3.p2.8.m1.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.8.m1.2.2.1.1.1.1.cmml" xref="S3.SS3.p2.8.m1.2.2.1.1.1">subscript</csymbol><ci id="S3.SS3.p2.8.m1.2.2.1.1.1.2.cmml" xref="S3.SS3.p2.8.m1.2.2.1.1.1.2">𝑏</ci><ci id="S3.SS3.p2.8.m1.2.2.1.1.1.3.cmml" xref="S3.SS3.p2.8.m1.2.2.1.1.1.3">𝑖</ci></apply><apply id="S3.SS3.p2.8.m1.3.3.2.2.2.cmml" xref="S3.SS3.p2.8.m1.3.3.2.2.2"><plus id="S3.SS3.p2.8.m1.3.3.2.2.2.1.cmml" xref="S3.SS3.p2.8.m1.3.3.2.2.2.1"></plus><apply id="S3.SS3.p2.8.m1.3.3.2.2.2.2.cmml" xref="S3.SS3.p2.8.m1.3.3.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS3.p2.8.m1.3.3.2.2.2.2.1.cmml" xref="S3.SS3.p2.8.m1.3.3.2.2.2.2">subscript</csymbol><ci id="S3.SS3.p2.8.m1.3.3.2.2.2.2.2.cmml" xref="S3.SS3.p2.8.m1.3.3.2.2.2.2.2">𝑏</ci><ci id="S3.SS3.p2.8.m1.3.3.2.2.2.2.3.cmml" xref="S3.SS3.p2.8.m1.3.3.2.2.2.2.3">𝑖</ci></apply><cn type="integer" id="S3.SS3.p2.8.m1.3.3.2.2.2.3.cmml" xref="S3.SS3.p2.8.m1.3.3.2.2.2.3">1</cn></apply><ci id="S3.SS3.p2.8.m1.1.1.cmml" xref="S3.SS3.p2.8.m1.1.1">…</ci><apply id="S3.SS3.p2.8.m1.4.4.3.3.3.cmml" xref="S3.SS3.p2.8.m1.4.4.3.3.3"><csymbol cd="ambiguous" id="S3.SS3.p2.8.m1.4.4.3.3.3.1.cmml" xref="S3.SS3.p2.8.m1.4.4.3.3.3">subscript</csymbol><ci id="S3.SS3.p2.8.m1.4.4.3.3.3.2.cmml" xref="S3.SS3.p2.8.m1.4.4.3.3.3.2">𝑏</ci><apply id="S3.SS3.p2.8.m1.4.4.3.3.3.3.cmml" xref="S3.SS3.p2.8.m1.4.4.3.3.3.3"><plus id="S3.SS3.p2.8.m1.4.4.3.3.3.3.1.cmml" xref="S3.SS3.p2.8.m1.4.4.3.3.3.3.1"></plus><ci id="S3.SS3.p2.8.m1.4.4.3.3.3.3.2.cmml" xref="S3.SS3.p2.8.m1.4.4.3.3.3.3.2">𝑖</ci><cn type="integer" id="S3.SS3.p2.8.m1.4.4.3.3.3.3.3.cmml" xref="S3.SS3.p2.8.m1.4.4.3.3.3.3.3">1</cn></apply></apply></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.8.m1.4c">S=\{b_{i},b_{i}+1,...,b_{i+1}\}</annotation></semantics></math><span id="S3.SS3.p2.10.2" class="ltx_text ltx_font_typewriter"> is a subset of all nodes </span><math id="S3.SS3.p2.9.m2.4" class="ltx_Math" alttext="V=\{b_{1},b_{1}+1,...,b_{k}\}" display="inline"><semantics id="S3.SS3.p2.9.m2.4a"><mrow id="S3.SS3.p2.9.m2.4.4" xref="S3.SS3.p2.9.m2.4.4.cmml"><mi id="S3.SS3.p2.9.m2.4.4.5" xref="S3.SS3.p2.9.m2.4.4.5.cmml">V</mi><mo id="S3.SS3.p2.9.m2.4.4.4" xref="S3.SS3.p2.9.m2.4.4.4.cmml">=</mo><mrow id="S3.SS3.p2.9.m2.4.4.3.3" xref="S3.SS3.p2.9.m2.4.4.3.4.cmml"><mo stretchy="false" id="S3.SS3.p2.9.m2.4.4.3.3.4" xref="S3.SS3.p2.9.m2.4.4.3.4.cmml">{</mo><msub id="S3.SS3.p2.9.m2.2.2.1.1.1" xref="S3.SS3.p2.9.m2.2.2.1.1.1.cmml"><mi id="S3.SS3.p2.9.m2.2.2.1.1.1.2" xref="S3.SS3.p2.9.m2.2.2.1.1.1.2.cmml">b</mi><mn id="S3.SS3.p2.9.m2.2.2.1.1.1.3" xref="S3.SS3.p2.9.m2.2.2.1.1.1.3.cmml">1</mn></msub><mo id="S3.SS3.p2.9.m2.4.4.3.3.5" xref="S3.SS3.p2.9.m2.4.4.3.4.cmml">,</mo><mrow id="S3.SS3.p2.9.m2.3.3.2.2.2" xref="S3.SS3.p2.9.m2.3.3.2.2.2.cmml"><msub id="S3.SS3.p2.9.m2.3.3.2.2.2.2" xref="S3.SS3.p2.9.m2.3.3.2.2.2.2.cmml"><mi id="S3.SS3.p2.9.m2.3.3.2.2.2.2.2" xref="S3.SS3.p2.9.m2.3.3.2.2.2.2.2.cmml">b</mi><mn id="S3.SS3.p2.9.m2.3.3.2.2.2.2.3" xref="S3.SS3.p2.9.m2.3.3.2.2.2.2.3.cmml">1</mn></msub><mo id="S3.SS3.p2.9.m2.3.3.2.2.2.1" xref="S3.SS3.p2.9.m2.3.3.2.2.2.1.cmml">+</mo><mn id="S3.SS3.p2.9.m2.3.3.2.2.2.3" xref="S3.SS3.p2.9.m2.3.3.2.2.2.3.cmml">1</mn></mrow><mo id="S3.SS3.p2.9.m2.4.4.3.3.6" xref="S3.SS3.p2.9.m2.4.4.3.4.cmml">,</mo><mi mathvariant="normal" id="S3.SS3.p2.9.m2.1.1" xref="S3.SS3.p2.9.m2.1.1.cmml">…</mi><mo id="S3.SS3.p2.9.m2.4.4.3.3.7" xref="S3.SS3.p2.9.m2.4.4.3.4.cmml">,</mo><msub id="S3.SS3.p2.9.m2.4.4.3.3.3" xref="S3.SS3.p2.9.m2.4.4.3.3.3.cmml"><mi id="S3.SS3.p2.9.m2.4.4.3.3.3.2" xref="S3.SS3.p2.9.m2.4.4.3.3.3.2.cmml">b</mi><mi id="S3.SS3.p2.9.m2.4.4.3.3.3.3" xref="S3.SS3.p2.9.m2.4.4.3.3.3.3.cmml">k</mi></msub><mo stretchy="false" id="S3.SS3.p2.9.m2.4.4.3.3.8" xref="S3.SS3.p2.9.m2.4.4.3.4.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.9.m2.4b"><apply id="S3.SS3.p2.9.m2.4.4.cmml" xref="S3.SS3.p2.9.m2.4.4"><eq id="S3.SS3.p2.9.m2.4.4.4.cmml" xref="S3.SS3.p2.9.m2.4.4.4"></eq><ci id="S3.SS3.p2.9.m2.4.4.5.cmml" xref="S3.SS3.p2.9.m2.4.4.5">𝑉</ci><set id="S3.SS3.p2.9.m2.4.4.3.4.cmml" xref="S3.SS3.p2.9.m2.4.4.3.3"><apply id="S3.SS3.p2.9.m2.2.2.1.1.1.cmml" xref="S3.SS3.p2.9.m2.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.9.m2.2.2.1.1.1.1.cmml" xref="S3.SS3.p2.9.m2.2.2.1.1.1">subscript</csymbol><ci id="S3.SS3.p2.9.m2.2.2.1.1.1.2.cmml" xref="S3.SS3.p2.9.m2.2.2.1.1.1.2">𝑏</ci><cn type="integer" id="S3.SS3.p2.9.m2.2.2.1.1.1.3.cmml" xref="S3.SS3.p2.9.m2.2.2.1.1.1.3">1</cn></apply><apply id="S3.SS3.p2.9.m2.3.3.2.2.2.cmml" xref="S3.SS3.p2.9.m2.3.3.2.2.2"><plus id="S3.SS3.p2.9.m2.3.3.2.2.2.1.cmml" xref="S3.SS3.p2.9.m2.3.3.2.2.2.1"></plus><apply id="S3.SS3.p2.9.m2.3.3.2.2.2.2.cmml" xref="S3.SS3.p2.9.m2.3.3.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS3.p2.9.m2.3.3.2.2.2.2.1.cmml" xref="S3.SS3.p2.9.m2.3.3.2.2.2.2">subscript</csymbol><ci id="S3.SS3.p2.9.m2.3.3.2.2.2.2.2.cmml" xref="S3.SS3.p2.9.m2.3.3.2.2.2.2.2">𝑏</ci><cn type="integer" id="S3.SS3.p2.9.m2.3.3.2.2.2.2.3.cmml" xref="S3.SS3.p2.9.m2.3.3.2.2.2.2.3">1</cn></apply><cn type="integer" id="S3.SS3.p2.9.m2.3.3.2.2.2.3.cmml" xref="S3.SS3.p2.9.m2.3.3.2.2.2.3">1</cn></apply><ci id="S3.SS3.p2.9.m2.1.1.cmml" xref="S3.SS3.p2.9.m2.1.1">…</ci><apply id="S3.SS3.p2.9.m2.4.4.3.3.3.cmml" xref="S3.SS3.p2.9.m2.4.4.3.3.3"><csymbol cd="ambiguous" id="S3.SS3.p2.9.m2.4.4.3.3.3.1.cmml" xref="S3.SS3.p2.9.m2.4.4.3.3.3">subscript</csymbol><ci id="S3.SS3.p2.9.m2.4.4.3.3.3.2.cmml" xref="S3.SS3.p2.9.m2.4.4.3.3.3.2">𝑏</ci><ci id="S3.SS3.p2.9.m2.4.4.3.3.3.3.cmml" xref="S3.SS3.p2.9.m2.4.4.3.3.3.3">𝑘</ci></apply></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.9.m2.4c">V=\{b_{1},b_{1}+1,...,b_{k}\}</annotation></semantics></math><span id="S3.SS3.p2.10.3" class="ltx_text ltx_font_typewriter"> in the induced graph, with </span><math id="S3.SS3.p2.10.m3.1" class="ltx_Math" alttext="b_{i}\in\mathcal{B}" display="inline"><semantics id="S3.SS3.p2.10.m3.1a"><mrow id="S3.SS3.p2.10.m3.1.1" xref="S3.SS3.p2.10.m3.1.1.cmml"><msub id="S3.SS3.p2.10.m3.1.1.2" xref="S3.SS3.p2.10.m3.1.1.2.cmml"><mi id="S3.SS3.p2.10.m3.1.1.2.2" xref="S3.SS3.p2.10.m3.1.1.2.2.cmml">b</mi><mi id="S3.SS3.p2.10.m3.1.1.2.3" xref="S3.SS3.p2.10.m3.1.1.2.3.cmml">i</mi></msub><mo id="S3.SS3.p2.10.m3.1.1.1" xref="S3.SS3.p2.10.m3.1.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.10.m3.1.1.3" xref="S3.SS3.p2.10.m3.1.1.3.cmml">ℬ</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.10.m3.1b"><apply id="S3.SS3.p2.10.m3.1.1.cmml" xref="S3.SS3.p2.10.m3.1.1"><in id="S3.SS3.p2.10.m3.1.1.1.cmml" xref="S3.SS3.p2.10.m3.1.1.1"></in><apply id="S3.SS3.p2.10.m3.1.1.2.cmml" xref="S3.SS3.p2.10.m3.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.p2.10.m3.1.1.2.1.cmml" xref="S3.SS3.p2.10.m3.1.1.2">subscript</csymbol><ci id="S3.SS3.p2.10.m3.1.1.2.2.cmml" xref="S3.SS3.p2.10.m3.1.1.2.2">𝑏</ci><ci id="S3.SS3.p2.10.m3.1.1.2.3.cmml" xref="S3.SS3.p2.10.m3.1.1.2.3">𝑖</ci></apply><ci id="S3.SS3.p2.10.m3.1.1.3.cmml" xref="S3.SS3.p2.10.m3.1.1.3">ℬ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.10.m3.1c">b_{i}\in\mathcal{B}</annotation></semantics></math><span id="S3.SS3.p2.10.4" class="ltx_text ltx_font_typewriter">. Lower conductance values indicate better community structure.
Our boundary refinement algorithm iteratively adjusts the initial surprise-based boundaries to optimise these metric functions. While our best results were achieved using modularity, we also include comparisons with conductance-based boundary refinement to provide a comprehensive analysis.
The overall process can be summarized in Algorithm </span><a href="#alg1" title="Algorithm 1 ‣ 3.3 Boundary refinement ‣ 3 EM-LLM: LLM with Episodic Memory ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">1</span></a><span id="S3.SS3.p2.10.5" class="ltx_text ltx_font_typewriter">.</span></p>
</div>
<figure id="alg1" class="ltx_float ltx_float_algorithm ltx_framed ltx_framed_top">
<figcaption class="ltx_caption ltx_font_typewriter"><span class="ltx_tag ltx_tag_float"><span id="alg1.4.1.1" class="ltx_text ltx_font_serif ltx_font_bold">Algorithm 1</span> </span> Event segmentation in KV cache</figcaption>
<div id="alg1.5" class="ltx_listing ltx_listing">
<div id="alg1.l1" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">1:</span><span id="alg1.l1.1" class="ltx_text ltx_font_typewriter">tokens: List of tokens in the sequence
</span>
</div>
<div id="alg1.l2" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">2:</span><math id="alg1.l2.m1.1" class="ltx_Math" alttext="T" display="inline"><semantics id="alg1.l2.m1.1a"><mi id="alg1.l2.m1.1.1" xref="alg1.l2.m1.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="alg1.l2.m1.1b"><ci id="alg1.l2.m1.1.1.cmml" xref="alg1.l2.m1.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l2.m1.1c">T</annotation></semantics></math><span id="alg1.l2.1" class="ltx_text ltx_font_typewriter">: Threshold for surprisal to identify initial boundaries
</span>
</div>
<div id="alg1.l3" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">3:</span><math id="alg1.l3.m1.1" class="ltx_Math" alttext="f" display="inline"><semantics id="alg1.l3.m1.1a"><mi id="alg1.l3.m1.1.1" xref="alg1.l3.m1.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="alg1.l3.m1.1b"><ci id="alg1.l3.m1.1.1.cmml" xref="alg1.l3.m1.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l3.m1.1c">f</annotation></semantics></math><span id="alg1.l3.1" class="ltx_text ltx_font_typewriter">: Metric function to evaluate potential boundaries
</span>
</div>
<div id="alg1.l4" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">4:</span><math id="alg1.l4.m1.1" class="ltx_Math" alttext="\mathcal{B}" display="inline"><semantics id="alg1.l4.m1.1a"><mi class="ltx_font_mathcaligraphic" id="alg1.l4.m1.1.1" xref="alg1.l4.m1.1.1.cmml">ℬ</mi><annotation-xml encoding="MathML-Content" id="alg1.l4.m1.1b"><ci id="alg1.l4.m1.1.1.cmml" xref="alg1.l4.m1.1.1">ℬ</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l4.m1.1c">\mathcal{B}</annotation></semantics></math><span id="alg1.l4.1" class="ltx_text ltx_font_typewriter">: List of final boundary positions
</span>
</div>
<div id="alg1.l5" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">5:</span><math id="alg1.l5.m1.1" class="ltx_Math" alttext="\mathcal{B}\leftarrow" display="inline"><semantics id="alg1.l5.m1.1a"><mrow id="alg1.l5.m1.1.1" xref="alg1.l5.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="alg1.l5.m1.1.1.2" xref="alg1.l5.m1.1.1.2.cmml">ℬ</mi><mo stretchy="false" id="alg1.l5.m1.1.1.1" xref="alg1.l5.m1.1.1.1.cmml">←</mo><mi id="alg1.l5.m1.1.1.3" xref="alg1.l5.m1.1.1.3.cmml"></mi></mrow><annotation-xml encoding="MathML-Content" id="alg1.l5.m1.1b"><apply id="alg1.l5.m1.1.1.cmml" xref="alg1.l5.m1.1.1"><ci id="alg1.l5.m1.1.1.1.cmml" xref="alg1.l5.m1.1.1.1">←</ci><ci id="alg1.l5.m1.1.1.2.cmml" xref="alg1.l5.m1.1.1.2">ℬ</ci><csymbol cd="latexml" id="alg1.l5.m1.1.1.3.cmml" xref="alg1.l5.m1.1.1.3">absent</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l5.m1.1c">\mathcal{B}\leftarrow</annotation></semantics></math><span id="alg1.l5.2" class="ltx_text ltx_font_typewriter"> [i for i in range(length(tokens)) if </span><math id="alg1.l5.m2.3" class="ltx_Math" alttext="-\log(P(\texttt{tokens}[i]))&gt;" display="inline"><semantics id="alg1.l5.m2.3a"><mrow id="alg1.l5.m2.3.3" xref="alg1.l5.m2.3.3.cmml"><mrow id="alg1.l5.m2.3.3.1" xref="alg1.l5.m2.3.3.1.cmml"><mo rspace="0.167em" id="alg1.l5.m2.3.3.1a" xref="alg1.l5.m2.3.3.1.cmml">−</mo><mrow id="alg1.l5.m2.3.3.1.1.1" xref="alg1.l5.m2.3.3.1.1.2.cmml"><mi id="alg1.l5.m2.2.2" xref="alg1.l5.m2.2.2.cmml">log</mi><mo id="alg1.l5.m2.3.3.1.1.1a" xref="alg1.l5.m2.3.3.1.1.2.cmml">⁡</mo><mrow id="alg1.l5.m2.3.3.1.1.1.1" xref="alg1.l5.m2.3.3.1.1.2.cmml"><mo stretchy="false" id="alg1.l5.m2.3.3.1.1.1.1.2" xref="alg1.l5.m2.3.3.1.1.2.cmml">(</mo><mrow id="alg1.l5.m2.3.3.1.1.1.1.1" xref="alg1.l5.m2.3.3.1.1.1.1.1.cmml"><mi id="alg1.l5.m2.3.3.1.1.1.1.1.3" xref="alg1.l5.m2.3.3.1.1.1.1.1.3.cmml">P</mi><mo lspace="0em" rspace="0em" id="alg1.l5.m2.3.3.1.1.1.1.1.2" xref="alg1.l5.m2.3.3.1.1.1.1.1.2.cmml">​</mo><mrow id="alg1.l5.m2.3.3.1.1.1.1.1.1.1" xref="alg1.l5.m2.3.3.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="alg1.l5.m2.3.3.1.1.1.1.1.1.1.2" xref="alg1.l5.m2.3.3.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="alg1.l5.m2.3.3.1.1.1.1.1.1.1.1" xref="alg1.l5.m2.3.3.1.1.1.1.1.1.1.1.cmml"><mtext class="ltx_mathvariant_monospace" id="alg1.l5.m2.3.3.1.1.1.1.1.1.1.1.2" xref="alg1.l5.m2.3.3.1.1.1.1.1.1.1.1.2a.cmml">tokens</mtext><mo lspace="0em" rspace="0em" id="alg1.l5.m2.3.3.1.1.1.1.1.1.1.1.1" xref="alg1.l5.m2.3.3.1.1.1.1.1.1.1.1.1.cmml">​</mo><mrow id="alg1.l5.m2.3.3.1.1.1.1.1.1.1.1.3.2" xref="alg1.l5.m2.3.3.1.1.1.1.1.1.1.1.3.1.cmml"><mo stretchy="false" id="alg1.l5.m2.3.3.1.1.1.1.1.1.1.1.3.2.1" xref="alg1.l5.m2.3.3.1.1.1.1.1.1.1.1.3.1.1.cmml">[</mo><mi id="alg1.l5.m2.1.1" xref="alg1.l5.m2.1.1.cmml">i</mi><mo stretchy="false" id="alg1.l5.m2.3.3.1.1.1.1.1.1.1.1.3.2.2" xref="alg1.l5.m2.3.3.1.1.1.1.1.1.1.1.3.1.1.cmml">]</mo></mrow></mrow><mo stretchy="false" id="alg1.l5.m2.3.3.1.1.1.1.1.1.1.3" xref="alg1.l5.m2.3.3.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="alg1.l5.m2.3.3.1.1.1.1.3" xref="alg1.l5.m2.3.3.1.1.2.cmml">)</mo></mrow></mrow></mrow><mo id="alg1.l5.m2.3.3.2" xref="alg1.l5.m2.3.3.2.cmml">&gt;</mo><mi id="alg1.l5.m2.3.3.3" xref="alg1.l5.m2.3.3.3.cmml"></mi></mrow><annotation-xml encoding="MathML-Content" id="alg1.l5.m2.3b"><apply id="alg1.l5.m2.3.3.cmml" xref="alg1.l5.m2.3.3"><gt id="alg1.l5.m2.3.3.2.cmml" xref="alg1.l5.m2.3.3.2"></gt><apply id="alg1.l5.m2.3.3.1.cmml" xref="alg1.l5.m2.3.3.1"><minus id="alg1.l5.m2.3.3.1.2.cmml" xref="alg1.l5.m2.3.3.1"></minus><apply id="alg1.l5.m2.3.3.1.1.2.cmml" xref="alg1.l5.m2.3.3.1.1.1"><log id="alg1.l5.m2.2.2.cmml" xref="alg1.l5.m2.2.2"></log><apply id="alg1.l5.m2.3.3.1.1.1.1.1.cmml" xref="alg1.l5.m2.3.3.1.1.1.1.1"><times id="alg1.l5.m2.3.3.1.1.1.1.1.2.cmml" xref="alg1.l5.m2.3.3.1.1.1.1.1.2"></times><ci id="alg1.l5.m2.3.3.1.1.1.1.1.3.cmml" xref="alg1.l5.m2.3.3.1.1.1.1.1.3">𝑃</ci><apply id="alg1.l5.m2.3.3.1.1.1.1.1.1.1.1.cmml" xref="alg1.l5.m2.3.3.1.1.1.1.1.1.1"><times id="alg1.l5.m2.3.3.1.1.1.1.1.1.1.1.1.cmml" xref="alg1.l5.m2.3.3.1.1.1.1.1.1.1.1.1"></times><ci id="alg1.l5.m2.3.3.1.1.1.1.1.1.1.1.2a.cmml" xref="alg1.l5.m2.3.3.1.1.1.1.1.1.1.1.2"><mtext class="ltx_mathvariant_monospace" id="alg1.l5.m2.3.3.1.1.1.1.1.1.1.1.2.cmml" xref="alg1.l5.m2.3.3.1.1.1.1.1.1.1.1.2">tokens</mtext></ci><apply id="alg1.l5.m2.3.3.1.1.1.1.1.1.1.1.3.1.cmml" xref="alg1.l5.m2.3.3.1.1.1.1.1.1.1.1.3.2"><csymbol cd="latexml" id="alg1.l5.m2.3.3.1.1.1.1.1.1.1.1.3.1.1.cmml" xref="alg1.l5.m2.3.3.1.1.1.1.1.1.1.1.3.2.1">delimited-[]</csymbol><ci id="alg1.l5.m2.1.1.cmml" xref="alg1.l5.m2.1.1">𝑖</ci></apply></apply></apply></apply></apply><csymbol cd="latexml" id="alg1.l5.m2.3.3.3.cmml" xref="alg1.l5.m2.3.3.3">absent</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l5.m2.3c">-\log(P(\texttt{tokens}[i]))&gt;</annotation></semantics></math><span id="alg1.l5.3" class="ltx_text ltx_font_typewriter"> </span><math id="alg1.l5.m3.1" class="ltx_Math" alttext="T" display="inline"><semantics id="alg1.l5.m3.1a"><mi id="alg1.l5.m3.1.1" xref="alg1.l5.m3.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="alg1.l5.m3.1b"><ci id="alg1.l5.m3.1.1.cmml" xref="alg1.l5.m3.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l5.m3.1c">T</annotation></semantics></math><span id="alg1.l5.4" class="ltx_text ltx_font_typewriter">] </span><span id="alg1.l5.1" class="ltx_text ltx_font_typewriter" style="float:right;"><math id="alg1.l5.1.m1.1" class="ltx_Math" alttext="\triangleright" display="inline"><semantics id="alg1.l5.1.m1.1a"><mo id="alg1.l5.1.m1.1.1" xref="alg1.l5.1.m1.1.1.cmml">▷</mo><annotation-xml encoding="MathML-Content" id="alg1.l5.1.m1.1b"><ci id="alg1.l5.1.m1.1.1.cmml" xref="alg1.l5.1.m1.1.1">▷</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l5.1.m1.1c">\triangleright</annotation></semantics></math> Boundary identification
</span>
</div>
<div id="alg1.l6" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">6:</span><span id="alg1.l6.1" class="ltx_text ltx_font_bold">for</span><span id="alg1.l6.2" class="ltx_text ltx_font_typewriter"> i in range(length(</span><math id="alg1.l6.m1.1" class="ltx_Math" alttext="\mathcal{B}" display="inline"><semantics id="alg1.l6.m1.1a"><mi class="ltx_font_mathcaligraphic" id="alg1.l6.m1.1.1" xref="alg1.l6.m1.1.1.cmml">ℬ</mi><annotation-xml encoding="MathML-Content" id="alg1.l6.m1.1b"><ci id="alg1.l6.m1.1.1.cmml" xref="alg1.l6.m1.1.1">ℬ</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l6.m1.1c">\mathcal{B}</annotation></semantics></math><span id="alg1.l6.3" class="ltx_text ltx_font_typewriter">)) </span><span id="alg1.l6.4" class="ltx_text ltx_font_bold">do</span><span id="alg1.l6.5" class="ltx_text ltx_font_typewriter">
</span>
</div>
<div id="alg1.l7" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">7:</span><span id="alg1.l7.1" class="ltx_text ltx_font_typewriter">     </span><math id="alg1.l7.m1.5" class="ltx_Math" alttext="\alpha,\beta=\mathcal{B}[i],\mathcal{B}[i+1]" display="inline"><semantics id="alg1.l7.m1.5a"><mrow id="alg1.l7.m1.5.5.2" xref="alg1.l7.m1.5.5.3.cmml"><mrow id="alg1.l7.m1.4.4.1.1" xref="alg1.l7.m1.4.4.1.1.cmml"><mrow id="alg1.l7.m1.4.4.1.1.2.2" xref="alg1.l7.m1.4.4.1.1.2.1.cmml"><mi id="alg1.l7.m1.2.2" xref="alg1.l7.m1.2.2.cmml">α</mi><mo id="alg1.l7.m1.4.4.1.1.2.2.1" xref="alg1.l7.m1.4.4.1.1.2.1.cmml">,</mo><mi id="alg1.l7.m1.3.3" xref="alg1.l7.m1.3.3.cmml">β</mi></mrow><mo id="alg1.l7.m1.4.4.1.1.1" xref="alg1.l7.m1.4.4.1.1.1.cmml">=</mo><mrow id="alg1.l7.m1.4.4.1.1.3" xref="alg1.l7.m1.4.4.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="alg1.l7.m1.4.4.1.1.3.2" xref="alg1.l7.m1.4.4.1.1.3.2.cmml">ℬ</mi><mo lspace="0em" rspace="0em" id="alg1.l7.m1.4.4.1.1.3.1" xref="alg1.l7.m1.4.4.1.1.3.1.cmml">​</mo><mrow id="alg1.l7.m1.4.4.1.1.3.3.2" xref="alg1.l7.m1.4.4.1.1.3.3.1.cmml"><mo stretchy="false" id="alg1.l7.m1.4.4.1.1.3.3.2.1" xref="alg1.l7.m1.4.4.1.1.3.3.1.1.cmml">[</mo><mi id="alg1.l7.m1.1.1" xref="alg1.l7.m1.1.1.cmml">i</mi><mo stretchy="false" id="alg1.l7.m1.4.4.1.1.3.3.2.2" xref="alg1.l7.m1.4.4.1.1.3.3.1.1.cmml">]</mo></mrow></mrow></mrow><mo id="alg1.l7.m1.5.5.2.3" xref="alg1.l7.m1.5.5.3a.cmml">,</mo><mrow id="alg1.l7.m1.5.5.2.2" xref="alg1.l7.m1.5.5.2.2.cmml"><mi class="ltx_font_mathcaligraphic" id="alg1.l7.m1.5.5.2.2.3" xref="alg1.l7.m1.5.5.2.2.3.cmml">ℬ</mi><mo lspace="0em" rspace="0em" id="alg1.l7.m1.5.5.2.2.2" xref="alg1.l7.m1.5.5.2.2.2.cmml">​</mo><mrow id="alg1.l7.m1.5.5.2.2.1.1" xref="alg1.l7.m1.5.5.2.2.1.2.cmml"><mo stretchy="false" id="alg1.l7.m1.5.5.2.2.1.1.2" xref="alg1.l7.m1.5.5.2.2.1.2.1.cmml">[</mo><mrow id="alg1.l7.m1.5.5.2.2.1.1.1" xref="alg1.l7.m1.5.5.2.2.1.1.1.cmml"><mi id="alg1.l7.m1.5.5.2.2.1.1.1.2" xref="alg1.l7.m1.5.5.2.2.1.1.1.2.cmml">i</mi><mo id="alg1.l7.m1.5.5.2.2.1.1.1.1" xref="alg1.l7.m1.5.5.2.2.1.1.1.1.cmml">+</mo><mn id="alg1.l7.m1.5.5.2.2.1.1.1.3" xref="alg1.l7.m1.5.5.2.2.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="alg1.l7.m1.5.5.2.2.1.1.3" xref="alg1.l7.m1.5.5.2.2.1.2.1.cmml">]</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l7.m1.5b"><apply id="alg1.l7.m1.5.5.3.cmml" xref="alg1.l7.m1.5.5.2"><csymbol cd="ambiguous" id="alg1.l7.m1.5.5.3a.cmml" xref="alg1.l7.m1.5.5.2.3">formulae-sequence</csymbol><apply id="alg1.l7.m1.4.4.1.1.cmml" xref="alg1.l7.m1.4.4.1.1"><eq id="alg1.l7.m1.4.4.1.1.1.cmml" xref="alg1.l7.m1.4.4.1.1.1"></eq><list id="alg1.l7.m1.4.4.1.1.2.1.cmml" xref="alg1.l7.m1.4.4.1.1.2.2"><ci id="alg1.l7.m1.2.2.cmml" xref="alg1.l7.m1.2.2">𝛼</ci><ci id="alg1.l7.m1.3.3.cmml" xref="alg1.l7.m1.3.3">𝛽</ci></list><apply id="alg1.l7.m1.4.4.1.1.3.cmml" xref="alg1.l7.m1.4.4.1.1.3"><times id="alg1.l7.m1.4.4.1.1.3.1.cmml" xref="alg1.l7.m1.4.4.1.1.3.1"></times><ci id="alg1.l7.m1.4.4.1.1.3.2.cmml" xref="alg1.l7.m1.4.4.1.1.3.2">ℬ</ci><apply id="alg1.l7.m1.4.4.1.1.3.3.1.cmml" xref="alg1.l7.m1.4.4.1.1.3.3.2"><csymbol cd="latexml" id="alg1.l7.m1.4.4.1.1.3.3.1.1.cmml" xref="alg1.l7.m1.4.4.1.1.3.3.2.1">delimited-[]</csymbol><ci id="alg1.l7.m1.1.1.cmml" xref="alg1.l7.m1.1.1">𝑖</ci></apply></apply></apply><apply id="alg1.l7.m1.5.5.2.2.cmml" xref="alg1.l7.m1.5.5.2.2"><times id="alg1.l7.m1.5.5.2.2.2.cmml" xref="alg1.l7.m1.5.5.2.2.2"></times><ci id="alg1.l7.m1.5.5.2.2.3.cmml" xref="alg1.l7.m1.5.5.2.2.3">ℬ</ci><apply id="alg1.l7.m1.5.5.2.2.1.2.cmml" xref="alg1.l7.m1.5.5.2.2.1.1"><csymbol cd="latexml" id="alg1.l7.m1.5.5.2.2.1.2.1.cmml" xref="alg1.l7.m1.5.5.2.2.1.1.2">delimited-[]</csymbol><apply id="alg1.l7.m1.5.5.2.2.1.1.1.cmml" xref="alg1.l7.m1.5.5.2.2.1.1.1"><plus id="alg1.l7.m1.5.5.2.2.1.1.1.1.cmml" xref="alg1.l7.m1.5.5.2.2.1.1.1.1"></plus><ci id="alg1.l7.m1.5.5.2.2.1.1.1.2.cmml" xref="alg1.l7.m1.5.5.2.2.1.1.1.2">𝑖</ci><cn type="integer" id="alg1.l7.m1.5.5.2.2.1.1.1.3.cmml" xref="alg1.l7.m1.5.5.2.2.1.1.1.3">1</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l7.m1.5c">\alpha,\beta=\mathcal{B}[i],\mathcal{B}[i+1]</annotation></semantics></math><span id="alg1.l7.2" class="ltx_text ltx_font_typewriter">
</span>
</div>
<div id="alg1.l8" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">8:</span><span id="alg1.l8.2" class="ltx_text ltx_font_typewriter">     </span><math id="alg1.l8.m1.7" class="ltx_Math" alttext="\mathcal{B}[i+1]\leftarrow\arg\max_{\hat{\beta}\in(\alpha,\beta]}f(A,\{\alpha,\hat{\beta}\})" display="inline"><semantics id="alg1.l8.m1.7a"><mrow id="alg1.l8.m1.7.7" xref="alg1.l8.m1.7.7.cmml"><mrow id="alg1.l8.m1.6.6.1" xref="alg1.l8.m1.6.6.1.cmml"><mi class="ltx_font_mathcaligraphic" id="alg1.l8.m1.6.6.1.3" xref="alg1.l8.m1.6.6.1.3.cmml">ℬ</mi><mo lspace="0em" rspace="0em" id="alg1.l8.m1.6.6.1.2" xref="alg1.l8.m1.6.6.1.2.cmml">​</mo><mrow id="alg1.l8.m1.6.6.1.1.1" xref="alg1.l8.m1.6.6.1.1.2.cmml"><mo stretchy="false" id="alg1.l8.m1.6.6.1.1.1.2" xref="alg1.l8.m1.6.6.1.1.2.1.cmml">[</mo><mrow id="alg1.l8.m1.6.6.1.1.1.1" xref="alg1.l8.m1.6.6.1.1.1.1.cmml"><mi id="alg1.l8.m1.6.6.1.1.1.1.2" xref="alg1.l8.m1.6.6.1.1.1.1.2.cmml">i</mi><mo id="alg1.l8.m1.6.6.1.1.1.1.1" xref="alg1.l8.m1.6.6.1.1.1.1.1.cmml">+</mo><mn id="alg1.l8.m1.6.6.1.1.1.1.3" xref="alg1.l8.m1.6.6.1.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="alg1.l8.m1.6.6.1.1.1.3" xref="alg1.l8.m1.6.6.1.1.2.1.cmml">]</mo></mrow></mrow><mo stretchy="false" id="alg1.l8.m1.7.7.3" xref="alg1.l8.m1.7.7.3.cmml">←</mo><mrow id="alg1.l8.m1.7.7.2" xref="alg1.l8.m1.7.7.2.cmml"><mrow id="alg1.l8.m1.7.7.2.3" xref="alg1.l8.m1.7.7.2.3.cmml"><mi id="alg1.l8.m1.7.7.2.3.1" xref="alg1.l8.m1.7.7.2.3.1.cmml">arg</mi><mo lspace="0.167em" id="alg1.l8.m1.7.7.2.3a" xref="alg1.l8.m1.7.7.2.3.cmml">⁡</mo><mrow id="alg1.l8.m1.7.7.2.3.2" xref="alg1.l8.m1.7.7.2.3.2.cmml"><msub id="alg1.l8.m1.7.7.2.3.2.1" xref="alg1.l8.m1.7.7.2.3.2.1.cmml"><mi id="alg1.l8.m1.7.7.2.3.2.1.2" xref="alg1.l8.m1.7.7.2.3.2.1.2.cmml">max</mi><mrow id="alg1.l8.m1.2.2.2" xref="alg1.l8.m1.2.2.2.cmml"><mover accent="true" id="alg1.l8.m1.2.2.2.4" xref="alg1.l8.m1.2.2.2.4.cmml"><mi id="alg1.l8.m1.2.2.2.4.2" xref="alg1.l8.m1.2.2.2.4.2.cmml">β</mi><mo id="alg1.l8.m1.2.2.2.4.1" xref="alg1.l8.m1.2.2.2.4.1.cmml">^</mo></mover><mo id="alg1.l8.m1.2.2.2.3" xref="alg1.l8.m1.2.2.2.3.cmml">∈</mo><mrow id="alg1.l8.m1.2.2.2.5.2" xref="alg1.l8.m1.2.2.2.5.1.cmml"><mo stretchy="false" id="alg1.l8.m1.2.2.2.5.2.1" xref="alg1.l8.m1.2.2.2.5.1.cmml">(</mo><mi id="alg1.l8.m1.1.1.1.1" xref="alg1.l8.m1.1.1.1.1.cmml">α</mi><mo id="alg1.l8.m1.2.2.2.5.2.2" xref="alg1.l8.m1.2.2.2.5.1.cmml">,</mo><mi id="alg1.l8.m1.2.2.2.2" xref="alg1.l8.m1.2.2.2.2.cmml">β</mi><mo stretchy="false" id="alg1.l8.m1.2.2.2.5.2.3" xref="alg1.l8.m1.2.2.2.5.1.cmml">]</mo></mrow></mrow></msub><mo lspace="0.167em" id="alg1.l8.m1.7.7.2.3.2a" xref="alg1.l8.m1.7.7.2.3.2.cmml">⁡</mo><mi id="alg1.l8.m1.7.7.2.3.2.2" xref="alg1.l8.m1.7.7.2.3.2.2.cmml">f</mi></mrow></mrow><mo lspace="0em" rspace="0em" id="alg1.l8.m1.7.7.2.2" xref="alg1.l8.m1.7.7.2.2.cmml">​</mo><mrow id="alg1.l8.m1.7.7.2.1.1" xref="alg1.l8.m1.7.7.2.1.2.cmml"><mo stretchy="false" id="alg1.l8.m1.7.7.2.1.1.2" xref="alg1.l8.m1.7.7.2.1.2.cmml">(</mo><mi id="alg1.l8.m1.5.5" xref="alg1.l8.m1.5.5.cmml">A</mi><mo id="alg1.l8.m1.7.7.2.1.1.3" xref="alg1.l8.m1.7.7.2.1.2.cmml">,</mo><mrow id="alg1.l8.m1.7.7.2.1.1.1.2" xref="alg1.l8.m1.7.7.2.1.1.1.1.cmml"><mo stretchy="false" id="alg1.l8.m1.7.7.2.1.1.1.2.1" xref="alg1.l8.m1.7.7.2.1.1.1.1.cmml">{</mo><mi id="alg1.l8.m1.3.3" xref="alg1.l8.m1.3.3.cmml">α</mi><mo id="alg1.l8.m1.7.7.2.1.1.1.2.2" xref="alg1.l8.m1.7.7.2.1.1.1.1.cmml">,</mo><mover accent="true" id="alg1.l8.m1.4.4" xref="alg1.l8.m1.4.4.cmml"><mi id="alg1.l8.m1.4.4.2" xref="alg1.l8.m1.4.4.2.cmml">β</mi><mo id="alg1.l8.m1.4.4.1" xref="alg1.l8.m1.4.4.1.cmml">^</mo></mover><mo stretchy="false" id="alg1.l8.m1.7.7.2.1.1.1.2.3" xref="alg1.l8.m1.7.7.2.1.1.1.1.cmml">}</mo></mrow><mo stretchy="false" id="alg1.l8.m1.7.7.2.1.1.4" xref="alg1.l8.m1.7.7.2.1.2.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l8.m1.7b"><apply id="alg1.l8.m1.7.7.cmml" xref="alg1.l8.m1.7.7"><ci id="alg1.l8.m1.7.7.3.cmml" xref="alg1.l8.m1.7.7.3">←</ci><apply id="alg1.l8.m1.6.6.1.cmml" xref="alg1.l8.m1.6.6.1"><times id="alg1.l8.m1.6.6.1.2.cmml" xref="alg1.l8.m1.6.6.1.2"></times><ci id="alg1.l8.m1.6.6.1.3.cmml" xref="alg1.l8.m1.6.6.1.3">ℬ</ci><apply id="alg1.l8.m1.6.6.1.1.2.cmml" xref="alg1.l8.m1.6.6.1.1.1"><csymbol cd="latexml" id="alg1.l8.m1.6.6.1.1.2.1.cmml" xref="alg1.l8.m1.6.6.1.1.1.2">delimited-[]</csymbol><apply id="alg1.l8.m1.6.6.1.1.1.1.cmml" xref="alg1.l8.m1.6.6.1.1.1.1"><plus id="alg1.l8.m1.6.6.1.1.1.1.1.cmml" xref="alg1.l8.m1.6.6.1.1.1.1.1"></plus><ci id="alg1.l8.m1.6.6.1.1.1.1.2.cmml" xref="alg1.l8.m1.6.6.1.1.1.1.2">𝑖</ci><cn type="integer" id="alg1.l8.m1.6.6.1.1.1.1.3.cmml" xref="alg1.l8.m1.6.6.1.1.1.1.3">1</cn></apply></apply></apply><apply id="alg1.l8.m1.7.7.2.cmml" xref="alg1.l8.m1.7.7.2"><times id="alg1.l8.m1.7.7.2.2.cmml" xref="alg1.l8.m1.7.7.2.2"></times><apply id="alg1.l8.m1.7.7.2.3.cmml" xref="alg1.l8.m1.7.7.2.3"><arg id="alg1.l8.m1.7.7.2.3.1.cmml" xref="alg1.l8.m1.7.7.2.3.1"></arg><apply id="alg1.l8.m1.7.7.2.3.2.cmml" xref="alg1.l8.m1.7.7.2.3.2"><apply id="alg1.l8.m1.7.7.2.3.2.1.cmml" xref="alg1.l8.m1.7.7.2.3.2.1"><csymbol cd="ambiguous" id="alg1.l8.m1.7.7.2.3.2.1.1.cmml" xref="alg1.l8.m1.7.7.2.3.2.1">subscript</csymbol><max id="alg1.l8.m1.7.7.2.3.2.1.2.cmml" xref="alg1.l8.m1.7.7.2.3.2.1.2"></max><apply id="alg1.l8.m1.2.2.2.cmml" xref="alg1.l8.m1.2.2.2"><in id="alg1.l8.m1.2.2.2.3.cmml" xref="alg1.l8.m1.2.2.2.3"></in><apply id="alg1.l8.m1.2.2.2.4.cmml" xref="alg1.l8.m1.2.2.2.4"><ci id="alg1.l8.m1.2.2.2.4.1.cmml" xref="alg1.l8.m1.2.2.2.4.1">^</ci><ci id="alg1.l8.m1.2.2.2.4.2.cmml" xref="alg1.l8.m1.2.2.2.4.2">𝛽</ci></apply><interval closure="open-closed" id="alg1.l8.m1.2.2.2.5.1.cmml" xref="alg1.l8.m1.2.2.2.5.2"><ci id="alg1.l8.m1.1.1.1.1.cmml" xref="alg1.l8.m1.1.1.1.1">𝛼</ci><ci id="alg1.l8.m1.2.2.2.2.cmml" xref="alg1.l8.m1.2.2.2.2">𝛽</ci></interval></apply></apply><ci id="alg1.l8.m1.7.7.2.3.2.2.cmml" xref="alg1.l8.m1.7.7.2.3.2.2">𝑓</ci></apply></apply><interval closure="open" id="alg1.l8.m1.7.7.2.1.2.cmml" xref="alg1.l8.m1.7.7.2.1.1"><ci id="alg1.l8.m1.5.5.cmml" xref="alg1.l8.m1.5.5">𝐴</ci><set id="alg1.l8.m1.7.7.2.1.1.1.1.cmml" xref="alg1.l8.m1.7.7.2.1.1.1.2"><ci id="alg1.l8.m1.3.3.cmml" xref="alg1.l8.m1.3.3">𝛼</ci><apply id="alg1.l8.m1.4.4.cmml" xref="alg1.l8.m1.4.4"><ci id="alg1.l8.m1.4.4.1.cmml" xref="alg1.l8.m1.4.4.1">^</ci><ci id="alg1.l8.m1.4.4.2.cmml" xref="alg1.l8.m1.4.4.2">𝛽</ci></apply></set></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l8.m1.7c">\mathcal{B}[i+1]\leftarrow\arg\max_{\hat{\beta}\in(\alpha,\beta]}f(A,\{\alpha,\hat{\beta}\})</annotation></semantics></math><span id="alg1.l8.3" class="ltx_text ltx_font_typewriter"> </span><span id="alg1.l8.1" class="ltx_text ltx_font_typewriter" style="float:right;"><math id="alg1.l8.1.m1.1" class="ltx_Math" alttext="\triangleright" display="inline"><semantics id="alg1.l8.1.m1.1a"><mo id="alg1.l8.1.m1.1.1" xref="alg1.l8.1.m1.1.1.cmml">▷</mo><annotation-xml encoding="MathML-Content" id="alg1.l8.1.m1.1b"><ci id="alg1.l8.1.m1.1.1.cmml" xref="alg1.l8.1.m1.1.1">▷</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l8.1.m1.1c">\triangleright</annotation></semantics></math> Boundary refinement
</span>
</div>
<div id="alg1.l9" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">9:</span><span id="alg1.l9.1" class="ltx_text ltx_font_bold">end</span><span id="alg1.l9.2" class="ltx_text ltx_font_typewriter"> </span><span id="alg1.l9.3" class="ltx_text ltx_font_bold">for</span>
</div>
<div id="alg1.l10" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">10:</span><span id="alg1.l10.1" class="ltx_text ltx_font_bold">return</span><span id="alg1.l10.2" class="ltx_text ltx_font_typewriter"> </span><math id="alg1.l10.m1.1" class="ltx_Math" alttext="\mathcal{B}" display="inline"><semantics id="alg1.l10.m1.1a"><mi class="ltx_font_mathcaligraphic" id="alg1.l10.m1.1.1" xref="alg1.l10.m1.1.1.cmml">ℬ</mi><annotation-xml encoding="MathML-Content" id="alg1.l10.m1.1b"><ci id="alg1.l10.m1.1.1.cmml" xref="alg1.l10.m1.1.1">ℬ</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l10.m1.1c">\mathcal{B}</annotation></semantics></math><span id="alg1.l10.3" class="ltx_text ltx_font_typewriter">
</span>
</div>
</div>
</figure>
<div id="S3.SS3.p3" class="ltx_para ltx_noindent">
<p id="S3.SS3.p3.7" class="ltx_p"><span id="S3.SS3.p3.7.1" class="ltx_text ltx_font_typewriter">This algorithm first identifies initial boundaries based on the surprise threshold </span><math id="S3.SS3.p3.1.m1.1" class="ltx_Math" alttext="T" display="inline"><semantics id="S3.SS3.p3.1.m1.1a"><mi id="S3.SS3.p3.1.m1.1.1" xref="S3.SS3.p3.1.m1.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.1.m1.1b"><ci id="S3.SS3.p3.1.m1.1.1.cmml" xref="S3.SS3.p3.1.m1.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.1.m1.1c">T</annotation></semantics></math><span id="S3.SS3.p3.7.2" class="ltx_text ltx_font_typewriter">, then refines these boundaries by finding the optimal position </span><math id="S3.SS3.p3.2.m2.1" class="ltx_Math" alttext="\hat{\beta}" display="inline"><semantics id="S3.SS3.p3.2.m2.1a"><mover accent="true" id="S3.SS3.p3.2.m2.1.1" xref="S3.SS3.p3.2.m2.1.1.cmml"><mi id="S3.SS3.p3.2.m2.1.1.2" xref="S3.SS3.p3.2.m2.1.1.2.cmml">β</mi><mo id="S3.SS3.p3.2.m2.1.1.1" xref="S3.SS3.p3.2.m2.1.1.1.cmml">^</mo></mover><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.2.m2.1b"><apply id="S3.SS3.p3.2.m2.1.1.cmml" xref="S3.SS3.p3.2.m2.1.1"><ci id="S3.SS3.p3.2.m2.1.1.1.cmml" xref="S3.SS3.p3.2.m2.1.1.1">^</ci><ci id="S3.SS3.p3.2.m2.1.1.2.cmml" xref="S3.SS3.p3.2.m2.1.1.2">𝛽</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.2.m2.1c">\hat{\beta}</annotation></semantics></math><span id="S3.SS3.p3.7.3" class="ltx_text ltx_font_typewriter"> between each pair of consecutive initial boundaries </span><math id="S3.SS3.p3.3.m3.2" class="ltx_Math" alttext="(\alpha,\beta)" display="inline"><semantics id="S3.SS3.p3.3.m3.2a"><mrow id="S3.SS3.p3.3.m3.2.3.2" xref="S3.SS3.p3.3.m3.2.3.1.cmml"><mo stretchy="false" id="S3.SS3.p3.3.m3.2.3.2.1" xref="S3.SS3.p3.3.m3.2.3.1.cmml">(</mo><mi id="S3.SS3.p3.3.m3.1.1" xref="S3.SS3.p3.3.m3.1.1.cmml">α</mi><mo id="S3.SS3.p3.3.m3.2.3.2.2" xref="S3.SS3.p3.3.m3.2.3.1.cmml">,</mo><mi id="S3.SS3.p3.3.m3.2.2" xref="S3.SS3.p3.3.m3.2.2.cmml">β</mi><mo stretchy="false" id="S3.SS3.p3.3.m3.2.3.2.3" xref="S3.SS3.p3.3.m3.2.3.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.3.m3.2b"><interval closure="open" id="S3.SS3.p3.3.m3.2.3.1.cmml" xref="S3.SS3.p3.3.m3.2.3.2"><ci id="S3.SS3.p3.3.m3.1.1.cmml" xref="S3.SS3.p3.3.m3.1.1">𝛼</ci><ci id="S3.SS3.p3.3.m3.2.2.cmml" xref="S3.SS3.p3.3.m3.2.2">𝛽</ci></interval></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.3.m3.2c">(\alpha,\beta)</annotation></semantics></math><span id="S3.SS3.p3.7.4" class="ltx_text ltx_font_typewriter"> that optimises the chosen metric function </span><math id="S3.SS3.p3.4.m4.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S3.SS3.p3.4.m4.1a"><mi id="S3.SS3.p3.4.m4.1.1" xref="S3.SS3.p3.4.m4.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.4.m4.1b"><ci id="S3.SS3.p3.4.m4.1.1.cmml" xref="S3.SS3.p3.4.m4.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.4.m4.1c">f</annotation></semantics></math><span id="S3.SS3.p3.7.5" class="ltx_text ltx_font_typewriter"> (either maximising modularity or minimising conductance). This process ensures that the final segmentation (1) captures points of high surprise and (2) optimises for coherent information grouping. The boundary identification step incurs negligible computational cost, as it only evaluates existing LLM outputs. The time complexity of Algorithm </span><a href="#alg1" title="Algorithm 1 ‣ 3.3 Boundary refinement ‣ 3 EM-LLM: LLM with Episodic Memory ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">1</span></a><span id="S3.SS3.p3.7.6" class="ltx_text ltx_font_typewriter"> is dominated by the boundary refinement step, which has an overall complexity of </span><math id="S3.SS3.p3.5.m5.1" class="ltx_Math" alttext="\mathcal{O}(kn)" display="inline"><semantics id="S3.SS3.p3.5.m5.1a"><mrow id="S3.SS3.p3.5.m5.1.1" xref="S3.SS3.p3.5.m5.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p3.5.m5.1.1.3" xref="S3.SS3.p3.5.m5.1.1.3.cmml">𝒪</mi><mo lspace="0em" rspace="0em" id="S3.SS3.p3.5.m5.1.1.2" xref="S3.SS3.p3.5.m5.1.1.2.cmml">​</mo><mrow id="S3.SS3.p3.5.m5.1.1.1.1" xref="S3.SS3.p3.5.m5.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS3.p3.5.m5.1.1.1.1.2" xref="S3.SS3.p3.5.m5.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS3.p3.5.m5.1.1.1.1.1" xref="S3.SS3.p3.5.m5.1.1.1.1.1.cmml"><mi id="S3.SS3.p3.5.m5.1.1.1.1.1.2" xref="S3.SS3.p3.5.m5.1.1.1.1.1.2.cmml">k</mi><mo lspace="0em" rspace="0em" id="S3.SS3.p3.5.m5.1.1.1.1.1.1" xref="S3.SS3.p3.5.m5.1.1.1.1.1.1.cmml">​</mo><mi id="S3.SS3.p3.5.m5.1.1.1.1.1.3" xref="S3.SS3.p3.5.m5.1.1.1.1.1.3.cmml">n</mi></mrow><mo stretchy="false" id="S3.SS3.p3.5.m5.1.1.1.1.3" xref="S3.SS3.p3.5.m5.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.5.m5.1b"><apply id="S3.SS3.p3.5.m5.1.1.cmml" xref="S3.SS3.p3.5.m5.1.1"><times id="S3.SS3.p3.5.m5.1.1.2.cmml" xref="S3.SS3.p3.5.m5.1.1.2"></times><ci id="S3.SS3.p3.5.m5.1.1.3.cmml" xref="S3.SS3.p3.5.m5.1.1.3">𝒪</ci><apply id="S3.SS3.p3.5.m5.1.1.1.1.1.cmml" xref="S3.SS3.p3.5.m5.1.1.1.1"><times id="S3.SS3.p3.5.m5.1.1.1.1.1.1.cmml" xref="S3.SS3.p3.5.m5.1.1.1.1.1.1"></times><ci id="S3.SS3.p3.5.m5.1.1.1.1.1.2.cmml" xref="S3.SS3.p3.5.m5.1.1.1.1.1.2">𝑘</ci><ci id="S3.SS3.p3.5.m5.1.1.1.1.1.3.cmml" xref="S3.SS3.p3.5.m5.1.1.1.1.1.3">𝑛</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.5.m5.1c">\mathcal{O}(kn)</annotation></semantics></math><span id="S3.SS3.p3.7.7" class="ltx_text ltx_font_typewriter">, where </span><math id="S3.SS3.p3.6.m6.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.SS3.p3.6.m6.1a"><mi id="S3.SS3.p3.6.m6.1.1" xref="S3.SS3.p3.6.m6.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.6.m6.1b"><ci id="S3.SS3.p3.6.m6.1.1.cmml" xref="S3.SS3.p3.6.m6.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.6.m6.1c">k</annotation></semantics></math><span id="S3.SS3.p3.7.8" class="ltx_text ltx_font_typewriter"> is the number of initial boundaries and </span><math id="S3.SS3.p3.7.m7.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S3.SS3.p3.7.m7.1a"><mi id="S3.SS3.p3.7.m7.1.1" xref="S3.SS3.p3.7.m7.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.7.m7.1b"><ci id="S3.SS3.p3.7.m7.1.1.cmml" xref="S3.SS3.p3.7.m7.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.7.m7.1c">n</annotation></semantics></math><span id="S3.SS3.p3.7.9" class="ltx_text ltx_font_typewriter"> is the sequence length. A detailed analysis of the algorithm’s complexity, including the computation of modularity and conductance metrics, is provided in Appendix </span><a href="#A1.SS2" title="A.2 Complexity Analysis of EM-LLM Algorithm ‣ Appendix A Appendix / supplemental material ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">A.2</span></a><span id="S3.SS3.p3.7.10" class="ltx_text ltx_font_typewriter">. Despite this modest computational overhead, the resulting improvement in segment quality leads to significant performance gains in downstream tasks, particularly those requiring complex temporal reasoning.</span></p>
</div>
</section>
<section id="S3.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_font_typewriter ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="S3.SS4.1.1.1" class="ltx_text ltx_font_serif">3.4</span> </span>Memory Retrieval</h3>

<div id="S3.SS4.p1" class="ltx_para ltx_noindent">
<p id="S3.SS4.p1.4" class="ltx_p"><span id="S3.SS4.p1.4.1" class="ltx_text ltx_font_typewriter">When inferring a new token, a number of episodic events are selected and become a part of the (extended) context window of the underlying LLM. Our memory retrieval process employs a two-stage mechanism to select relevant episodic events for the LLM’s context window (Fig.</span><a href="#S2.F2" title="Figure 2 ‣ 2.2 Neural models of Episodic Memory and Event Cognition ‣ 2 Related work ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">2</span></a><span id="S3.SS4.p1.4.2" class="ltx_text ltx_font_typewriter">C). First, we retrieve </span><math id="S3.SS4.p1.1.m1.1" class="ltx_Math" alttext="k_{s}" display="inline"><semantics id="S3.SS4.p1.1.m1.1a"><msub id="S3.SS4.p1.1.m1.1.1" xref="S3.SS4.p1.1.m1.1.1.cmml"><mi id="S3.SS4.p1.1.m1.1.1.2" xref="S3.SS4.p1.1.m1.1.1.2.cmml">k</mi><mi id="S3.SS4.p1.1.m1.1.1.3" xref="S3.SS4.p1.1.m1.1.1.3.cmml">s</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS4.p1.1.m1.1b"><apply id="S3.SS4.p1.1.m1.1.1.cmml" xref="S3.SS4.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS4.p1.1.m1.1.1.1.cmml" xref="S3.SS4.p1.1.m1.1.1">subscript</csymbol><ci id="S3.SS4.p1.1.m1.1.1.2.cmml" xref="S3.SS4.p1.1.m1.1.1.2">𝑘</ci><ci id="S3.SS4.p1.1.m1.1.1.3.cmml" xref="S3.SS4.p1.1.m1.1.1.3">𝑠</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p1.1.m1.1c">k_{s}</annotation></semantics></math><span id="S3.SS4.p1.4.3" class="ltx_text ltx_font_typewriter"> events using </span><math id="S3.SS4.p1.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.SS4.p1.2.m2.1a"><mi id="S3.SS4.p1.2.m2.1.1" xref="S3.SS4.p1.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p1.2.m2.1b"><ci id="S3.SS4.p1.2.m2.1.1.cmml" xref="S3.SS4.p1.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p1.2.m2.1c">k</annotation></semantics></math><span id="S3.SS4.p1.4.4" class="ltx_text ltx_font_typewriter">-NN search based on dot product similarity between the current query and representative tokens of each event. These representatives, selected as per </span><cite class="ltx_cite ltx_citemacro_citet"><span class="ltx_text ltx_font_typewriter">Xiao et al.</span> <span id="S3.SS4.p1.4.5.1.1.1" class="ltx_text ltx_font_typewriter">[</span><a href="#bib.bib8" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024a</span></a><span id="S3.SS4.p1.4.6.2.2.1" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S3.SS4.p1.4.7" class="ltx_text ltx_font_typewriter">, are the most influential tokens within each event. For large memory stores, we utilise approximate </span><math id="S3.SS4.p1.3.m3.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.SS4.p1.3.m3.1a"><mi id="S3.SS4.p1.3.m3.1.1" xref="S3.SS4.p1.3.m3.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p1.3.m3.1b"><ci id="S3.SS4.p1.3.m3.1.1.cmml" xref="S3.SS4.p1.3.m3.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p1.3.m3.1c">k</annotation></semantics></math><span id="S3.SS4.p1.4.8" class="ltx_text ltx_font_typewriter">-NN </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S3.SS4.p1.4.9.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Douze et al.</span><span id="S3.SS4.p1.4.10.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib72" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024</span></a><span id="S3.SS4.p1.4.11.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S3.SS4.p1.4.12" class="ltx_text ltx_font_typewriter"> to maintain efficiency. These </span><math id="S3.SS4.p1.4.m4.1" class="ltx_Math" alttext="k_{s}" display="inline"><semantics id="S3.SS4.p1.4.m4.1a"><msub id="S3.SS4.p1.4.m4.1.1" xref="S3.SS4.p1.4.m4.1.1.cmml"><mi id="S3.SS4.p1.4.m4.1.1.2" xref="S3.SS4.p1.4.m4.1.1.2.cmml">k</mi><mi id="S3.SS4.p1.4.m4.1.1.3" xref="S3.SS4.p1.4.m4.1.1.3.cmml">s</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS4.p1.4.m4.1b"><apply id="S3.SS4.p1.4.m4.1.1.cmml" xref="S3.SS4.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS4.p1.4.m4.1.1.1.cmml" xref="S3.SS4.p1.4.m4.1.1">subscript</csymbol><ci id="S3.SS4.p1.4.m4.1.1.2.cmml" xref="S3.SS4.p1.4.m4.1.1.2">𝑘</ci><ci id="S3.SS4.p1.4.m4.1.1.3.cmml" xref="S3.SS4.p1.4.m4.1.1.3">𝑠</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p1.4.m4.1c">k_{s}</annotation></semantics></math><span id="S3.SS4.p1.4.13" class="ltx_text ltx_font_typewriter"> events, retrieved based on their similarity to the current query, form a part of the LLM’s context window that we refer to as the </span><span id="S3.SS4.p1.4.14" class="ltx_text ltx_font_italic">similarity buffer</span><span id="S3.SS4.p1.4.15" class="ltx_text ltx_font_typewriter">.</span></p>
</div>
<div id="S3.SS4.p2" class="ltx_para ltx_noindent">
<p id="S3.SS4.p2.3" class="ltx_p"><span id="S3.SS4.p2.3.1" class="ltx_text ltx_font_typewriter">The second stage of our retrieval process introduces another buffer, which we refer to as the </span><span id="S3.SS4.p2.3.2" class="ltx_text ltx_font_italic">contiguity buffer</span><span id="S3.SS4.p2.3.3" class="ltx_text ltx_font_typewriter">, designed to maintain temporal context. Implemented as a queue of size </span><math id="S3.SS4.p2.1.m1.1" class="ltx_Math" alttext="k_{c}" display="inline"><semantics id="S3.SS4.p2.1.m1.1a"><msub id="S3.SS4.p2.1.m1.1.1" xref="S3.SS4.p2.1.m1.1.1.cmml"><mi id="S3.SS4.p2.1.m1.1.1.2" xref="S3.SS4.p2.1.m1.1.1.2.cmml">k</mi><mi id="S3.SS4.p2.1.m1.1.1.3" xref="S3.SS4.p2.1.m1.1.1.3.cmml">c</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.1.m1.1b"><apply id="S3.SS4.p2.1.m1.1.1.cmml" xref="S3.SS4.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS4.p2.1.m1.1.1.1.cmml" xref="S3.SS4.p2.1.m1.1.1">subscript</csymbol><ci id="S3.SS4.p2.1.m1.1.1.2.cmml" xref="S3.SS4.p2.1.m1.1.1.2">𝑘</ci><ci id="S3.SS4.p2.1.m1.1.1.3.cmml" xref="S3.SS4.p2.1.m1.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.1.m1.1c">k_{c}</annotation></semantics></math><span id="S3.SS4.p2.3.4" class="ltx_text ltx_font_typewriter">, this buffer promotes temporal relationships in retrieval. When an event is retrieved, we also enqueue its neighboring events (within </span><math id="S3.SS4.p2.2.m2.1" class="ltx_Math" alttext="\pm n" display="inline"><semantics id="S3.SS4.p2.2.m2.1a"><mrow id="S3.SS4.p2.2.m2.1.1" xref="S3.SS4.p2.2.m2.1.1.cmml"><mo id="S3.SS4.p2.2.m2.1.1a" xref="S3.SS4.p2.2.m2.1.1.cmml">±</mo><mi id="S3.SS4.p2.2.m2.1.1.2" xref="S3.SS4.p2.2.m2.1.1.2.cmml">n</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.2.m2.1b"><apply id="S3.SS4.p2.2.m2.1.1.cmml" xref="S3.SS4.p2.2.m2.1.1"><csymbol cd="latexml" id="S3.SS4.p2.2.m2.1.1.1.cmml" xref="S3.SS4.p2.2.m2.1.1">plus-or-minus</csymbol><ci id="S3.SS4.p2.2.m2.1.1.2.cmml" xref="S3.SS4.p2.2.m2.1.1.2">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.2.m2.1c">\pm n</annotation></semantics></math><span id="S3.SS4.p2.3.5" class="ltx_text ltx_font_typewriter"> positions in the original sequence) into this buffer. This mechanism enables the LLM’s ‘‘induction’’ attention heads to exhibit the contiguity and asymmetry effects discussed in Section </span><a href="#S2.SS2" title="2.2 Neural models of Episodic Memory and Event Cognition ‣ 2 Related work ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">2.2</span></a><span id="S3.SS4.p2.3.6" class="ltx_text ltx_font_typewriter">. The queue structure allows for a natural decay of temporal context as new events are processed, with older or repeated events being dequeued as new ones are added. In total, </span><math id="S3.SS4.p2.3.m3.1" class="ltx_Math" alttext="k=k_{s}+k_{c}+2" display="inline"><semantics id="S3.SS4.p2.3.m3.1a"><mrow id="S3.SS4.p2.3.m3.1.1" xref="S3.SS4.p2.3.m3.1.1.cmml"><mi id="S3.SS4.p2.3.m3.1.1.2" xref="S3.SS4.p2.3.m3.1.1.2.cmml">k</mi><mo id="S3.SS4.p2.3.m3.1.1.1" xref="S3.SS4.p2.3.m3.1.1.1.cmml">=</mo><mrow id="S3.SS4.p2.3.m3.1.1.3" xref="S3.SS4.p2.3.m3.1.1.3.cmml"><msub id="S3.SS4.p2.3.m3.1.1.3.2" xref="S3.SS4.p2.3.m3.1.1.3.2.cmml"><mi id="S3.SS4.p2.3.m3.1.1.3.2.2" xref="S3.SS4.p2.3.m3.1.1.3.2.2.cmml">k</mi><mi id="S3.SS4.p2.3.m3.1.1.3.2.3" xref="S3.SS4.p2.3.m3.1.1.3.2.3.cmml">s</mi></msub><mo id="S3.SS4.p2.3.m3.1.1.3.1" xref="S3.SS4.p2.3.m3.1.1.3.1.cmml">+</mo><msub id="S3.SS4.p2.3.m3.1.1.3.3" xref="S3.SS4.p2.3.m3.1.1.3.3.cmml"><mi id="S3.SS4.p2.3.m3.1.1.3.3.2" xref="S3.SS4.p2.3.m3.1.1.3.3.2.cmml">k</mi><mi id="S3.SS4.p2.3.m3.1.1.3.3.3" xref="S3.SS4.p2.3.m3.1.1.3.3.3.cmml">c</mi></msub><mo id="S3.SS4.p2.3.m3.1.1.3.1a" xref="S3.SS4.p2.3.m3.1.1.3.1.cmml">+</mo><mn id="S3.SS4.p2.3.m3.1.1.3.4" xref="S3.SS4.p2.3.m3.1.1.3.4.cmml">2</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.3.m3.1b"><apply id="S3.SS4.p2.3.m3.1.1.cmml" xref="S3.SS4.p2.3.m3.1.1"><eq id="S3.SS4.p2.3.m3.1.1.1.cmml" xref="S3.SS4.p2.3.m3.1.1.1"></eq><ci id="S3.SS4.p2.3.m3.1.1.2.cmml" xref="S3.SS4.p2.3.m3.1.1.2">𝑘</ci><apply id="S3.SS4.p2.3.m3.1.1.3.cmml" xref="S3.SS4.p2.3.m3.1.1.3"><plus id="S3.SS4.p2.3.m3.1.1.3.1.cmml" xref="S3.SS4.p2.3.m3.1.1.3.1"></plus><apply id="S3.SS4.p2.3.m3.1.1.3.2.cmml" xref="S3.SS4.p2.3.m3.1.1.3.2"><csymbol cd="ambiguous" id="S3.SS4.p2.3.m3.1.1.3.2.1.cmml" xref="S3.SS4.p2.3.m3.1.1.3.2">subscript</csymbol><ci id="S3.SS4.p2.3.m3.1.1.3.2.2.cmml" xref="S3.SS4.p2.3.m3.1.1.3.2.2">𝑘</ci><ci id="S3.SS4.p2.3.m3.1.1.3.2.3.cmml" xref="S3.SS4.p2.3.m3.1.1.3.2.3">𝑠</ci></apply><apply id="S3.SS4.p2.3.m3.1.1.3.3.cmml" xref="S3.SS4.p2.3.m3.1.1.3.3"><csymbol cd="ambiguous" id="S3.SS4.p2.3.m3.1.1.3.3.1.cmml" xref="S3.SS4.p2.3.m3.1.1.3.3">subscript</csymbol><ci id="S3.SS4.p2.3.m3.1.1.3.3.2.cmml" xref="S3.SS4.p2.3.m3.1.1.3.3.2">𝑘</ci><ci id="S3.SS4.p2.3.m3.1.1.3.3.3.cmml" xref="S3.SS4.p2.3.m3.1.1.3.3.3">𝑐</ci></apply><cn type="integer" id="S3.SS4.p2.3.m3.1.1.3.4.cmml" xref="S3.SS4.p2.3.m3.1.1.3.4">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.3.m3.1c">k=k_{s}+k_{c}+2</annotation></semantics></math><span id="S3.SS4.p2.3.7" class="ltx_text ltx_font_typewriter"> events are added to the context window, striking a balance between relevance and temporal relationships in a manner analogous to human episodic memory retrieval.</span></p>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_font_typewriter ltx_title_section">
<span class="ltx_tag ltx_tag_section"><span id="S4.1.1.1" class="ltx_text ltx_font_serif">4</span> </span>Experiments</h2>

<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_font_typewriter ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="S4.SS1.1.1.1" class="ltx_text ltx_font_serif">4.1</span> </span>Performance of EM-LLM on long-context tasks</h3>

<div id="S4.SS1.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.p1.1" class="ltx_p"><span id="S4.SS1.p1.1.1" class="ltx_text ltx_font_typewriter">As previously mentioned, InfLLM is, at the time of writing, considered to achieve state-of-the-art performance on long-context benchmarks (</span><math id="S4.SS1.p1.1.m1.1" class="ltx_Math" alttext="\infty" display="inline"><semantics id="S4.SS1.p1.1.m1.1a"><mi mathvariant="normal" id="S4.SS1.p1.1.m1.1.1" xref="S4.SS1.p1.1.m1.1.1.cmml">∞</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.1.m1.1b"><infinity id="S4.SS1.p1.1.m1.1.1.cmml" xref="S4.SS1.p1.1.m1.1.1"></infinity></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.1.m1.1c">\infty</annotation></semantics></math><span id="S4.SS1.p1.1.2" class="ltx_text ltx_font_typewriter">-Bench, LongBench), as well as being the only method which uses group-based k-NN retrieval in LLMs on such benchmarks. We, therefore, employ this model as our baseline for comparison with our own methods on short context windows (4K+2K, as in </span><cite class="ltx_cite ltx_citemacro_citep"><span class="ltx_text ltx_font_typewriter">Xiao et al.</span><span id="S4.SS1.p1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib8" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024a</span></a></cite><span id="S4.SS1.p1.1.4" class="ltx_text ltx_font_typewriter">).</span></p>
</div>
<figure id="S4.T1" class="ltx_table">
<table id="S4.T1.16" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T1.16.17.1" class="ltx_tr">
<th id="S4.T1.16.17.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_r ltx_border_tt" style="padding-left:4.0pt;padding-right:4.0pt;" rowspan="2"><span id="S4.T1.16.17.1.1.1" class="ltx_text ltx_font_bold">Task</span></th>
<th id="S4.T1.16.17.1.2" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_r ltx_border_tt" style="padding-left:4.0pt;padding-right:4.0pt;" rowspan="2"><span id="S4.T1.16.17.1.2.1" class="ltx_text ltx_font_bold">InfLLM</span></th>
<th id="S4.T1.16.17.1.3" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_r ltx_border_tt" style="padding-left:4.0pt;padding-right:4.0pt;" rowspan="2"><span id="S4.T1.16.17.1.3.1" class="ltx_text ltx_font_bold">Max Imp.</span></th>
<th id="S4.T1.16.17.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-left:4.0pt;padding-right:4.0pt;" colspan="4"><span id="S4.T1.16.17.1.4.1" class="ltx_text ltx_font_bold">EM-LLM</span></th>
</tr>
<tr id="S4.T1.16.18.2" class="ltx_tr">
<th id="S4.T1.16.18.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.16.18.2.1.1" class="ltx_text ltx_font_bold">S</span></th>
<th id="S4.T1.16.18.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.16.18.2.2.1" class="ltx_text ltx_font_bold">SM</span></th>
<th id="S4.T1.16.18.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.16.18.2.3.1" class="ltx_text ltx_font_bold">S+C</span></th>
<th id="S4.T1.16.18.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.16.18.2.4.1" class="ltx_text ltx_font_bold">SM+C</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T1.1.1" class="ltx_tr">
<td id="S4.T1.1.1.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.1.1.2.1" class="ltx_text ltx_font_typewriter">NarrativeQA</span></td>
<td id="S4.T1.1.1.3" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.1.1.3.1" class="ltx_text ltx_font_typewriter">22.12</span></td>
<td id="S4.T1.1.1.1" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.1.1.1.1" class="ltx_text ltx_font_bold">+1.49<math id="S4.T1.1.1.1.1.m1.1" class="ltx_Math" alttext="\%" display="inline"><semantics id="S4.T1.1.1.1.1.m1.1a"><mo id="S4.T1.1.1.1.1.m1.1.1" xref="S4.T1.1.1.1.1.m1.1.1.cmml">%</mo><annotation-xml encoding="MathML-Content" id="S4.T1.1.1.1.1.m1.1b"><csymbol cd="latexml" id="S4.T1.1.1.1.1.m1.1.1.cmml" xref="S4.T1.1.1.1.1.m1.1.1">percent</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.1.1.1.1.m1.1c">\%</annotation></semantics></math></span></td>
<td id="S4.T1.1.1.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.1.1.4.1" class="ltx_text ltx_font_typewriter">21.32</span></td>
<td id="S4.T1.1.1.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.1.1.5.1" class="ltx_text ltx_font_typewriter">21.13</span></td>
<td id="S4.T1.1.1.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.1.1.6.1" class="ltx_text ltx_font_typewriter">21.80</span></td>
<td id="S4.T1.1.1.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.1.1.7.1" class="ltx_text ltx_font_bold">22.45</span></td>
</tr>
<tr id="S4.T1.2.2" class="ltx_tr">
<td id="S4.T1.2.2.2" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.2.2.2.1" class="ltx_text ltx_font_typewriter">MultiNews</span></td>
<td id="S4.T1.2.2.3" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.2.2.3.1" class="ltx_text ltx_font_bold">26.70</span></td>
<td id="S4.T1.2.2.1" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;">
<span id="S4.T1.2.2.1.1" class="ltx_text ltx_font_typewriter">-0.30</span><math id="S4.T1.2.2.1.m1.1" class="ltx_Math" alttext="\%" display="inline"><semantics id="S4.T1.2.2.1.m1.1a"><mo id="S4.T1.2.2.1.m1.1.1" xref="S4.T1.2.2.1.m1.1.1.cmml">%</mo><annotation-xml encoding="MathML-Content" id="S4.T1.2.2.1.m1.1b"><csymbol cd="latexml" id="S4.T1.2.2.1.m1.1.1.cmml" xref="S4.T1.2.2.1.m1.1.1">percent</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.2.2.1.m1.1c">\%</annotation></semantics></math>
</td>
<td id="S4.T1.2.2.4" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.2.2.4.1" class="ltx_text ltx_font_typewriter">26.52</span></td>
<td id="S4.T1.2.2.5" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.2.2.5.1" class="ltx_text ltx_font_typewriter">26.54</span></td>
<td id="S4.T1.2.2.6" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.2.2.6.1" class="ltx_text ltx_font_typewriter">26.69</span></td>
<td id="S4.T1.2.2.7" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.2.2.7.1" class="ltx_text ltx_font_typewriter">26.62</span></td>
</tr>
<tr id="S4.T1.3.3" class="ltx_tr">
<td id="S4.T1.3.3.2" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.3.3.2.1" class="ltx_text ltx_font_typewriter">Qasper</span></td>
<td id="S4.T1.3.3.3" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.3.3.3.1" class="ltx_text ltx_font_typewriter">29.33</span></td>
<td id="S4.T1.3.3.1" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.3.3.1.1" class="ltx_text ltx_font_bold">+0.17<math id="S4.T1.3.3.1.1.m1.1" class="ltx_Math" alttext="\%" display="inline"><semantics id="S4.T1.3.3.1.1.m1.1a"><mo id="S4.T1.3.3.1.1.m1.1.1" xref="S4.T1.3.3.1.1.m1.1.1.cmml">%</mo><annotation-xml encoding="MathML-Content" id="S4.T1.3.3.1.1.m1.1b"><csymbol cd="latexml" id="S4.T1.3.3.1.1.m1.1.1.cmml" xref="S4.T1.3.3.1.1.m1.1.1">percent</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.3.3.1.1.m1.1c">\%</annotation></semantics></math></span></td>
<td id="S4.T1.3.3.4" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.3.3.4.1" class="ltx_text ltx_font_typewriter">28.99</span></td>
<td id="S4.T1.3.3.5" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.3.3.5.1" class="ltx_text ltx_font_bold">29.38</span></td>
<td id="S4.T1.3.3.6" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.3.3.6.1" class="ltx_text ltx_font_typewriter">29.11</span></td>
<td id="S4.T1.3.3.7" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.3.3.7.1" class="ltx_text ltx_font_typewriter">28.68</span></td>
</tr>
<tr id="S4.T1.4.4" class="ltx_tr">
<td id="S4.T1.4.4.2" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.4.4.2.1" class="ltx_text ltx_font_typewriter">TREC</span></td>
<td id="S4.T1.4.4.3" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.4.4.3.1" class="ltx_text ltx_font_typewriter">69.00</span></td>
<td id="S4.T1.4.4.1" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.4.4.1.1" class="ltx_text ltx_font_bold">+2.17<math id="S4.T1.4.4.1.1.m1.1" class="ltx_Math" alttext="\%" display="inline"><semantics id="S4.T1.4.4.1.1.m1.1a"><mo id="S4.T1.4.4.1.1.m1.1.1" xref="S4.T1.4.4.1.1.m1.1.1.cmml">%</mo><annotation-xml encoding="MathML-Content" id="S4.T1.4.4.1.1.m1.1b"><csymbol cd="latexml" id="S4.T1.4.4.1.1.m1.1.1.cmml" xref="S4.T1.4.4.1.1.m1.1.1">percent</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.4.4.1.1.m1.1c">\%</annotation></semantics></math></span></td>
<td id="S4.T1.4.4.4" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.4.4.4.1" class="ltx_text ltx_font_typewriter">70.00</span></td>
<td id="S4.T1.4.4.5" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.4.4.5.1" class="ltx_text ltx_font_typewriter">70.00</span></td>
<td id="S4.T1.4.4.6" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.4.4.6.1" class="ltx_text ltx_font_bold">70.50</span></td>
<td id="S4.T1.4.4.7" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.4.4.7.1" class="ltx_text ltx_font_bold">70.50</span></td>
</tr>
<tr id="S4.T1.5.5" class="ltx_tr">
<td id="S4.T1.5.5.2" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.5.5.2.1" class="ltx_text ltx_font_typewriter">MultiFieldQA</span></td>
<td id="S4.T1.5.5.3" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.5.5.3.1" class="ltx_text ltx_font_typewriter">47.42</span></td>
<td id="S4.T1.5.5.1" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.5.5.1.1" class="ltx_text ltx_font_bold">+0.42<math id="S4.T1.5.5.1.1.m1.1" class="ltx_Math" alttext="\%" display="inline"><semantics id="S4.T1.5.5.1.1.m1.1a"><mo id="S4.T1.5.5.1.1.m1.1.1" xref="S4.T1.5.5.1.1.m1.1.1.cmml">%</mo><annotation-xml encoding="MathML-Content" id="S4.T1.5.5.1.1.m1.1b"><csymbol cd="latexml" id="S4.T1.5.5.1.1.m1.1.1.cmml" xref="S4.T1.5.5.1.1.m1.1.1">percent</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.5.5.1.1.m1.1c">\%</annotation></semantics></math></span></td>
<td id="S4.T1.5.5.4" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.5.5.4.1" class="ltx_text ltx_font_typewriter">47.49</span></td>
<td id="S4.T1.5.5.5" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.5.5.5.1" class="ltx_text ltx_font_typewriter">47.39</span></td>
<td id="S4.T1.5.5.6" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.5.5.6.1" class="ltx_text ltx_font_typewriter">47.46</span></td>
<td id="S4.T1.5.5.7" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.5.5.7.1" class="ltx_text ltx_font_bold">47.62</span></td>
</tr>
<tr id="S4.T1.6.6" class="ltx_tr">
<td id="S4.T1.6.6.2" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.6.6.2.1" class="ltx_text ltx_font_typewriter">TriviaQA</span></td>
<td id="S4.T1.6.6.3" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.6.6.3.1" class="ltx_text ltx_font_typewriter">86.67</span></td>
<td id="S4.T1.6.6.1" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.6.6.1.1" class="ltx_text ltx_font_bold">+1.10<math id="S4.T1.6.6.1.1.m1.1" class="ltx_Math" alttext="\%" display="inline"><semantics id="S4.T1.6.6.1.1.m1.1a"><mo id="S4.T1.6.6.1.1.m1.1.1" xref="S4.T1.6.6.1.1.m1.1.1.cmml">%</mo><annotation-xml encoding="MathML-Content" id="S4.T1.6.6.1.1.m1.1b"><csymbol cd="latexml" id="S4.T1.6.6.1.1.m1.1.1.cmml" xref="S4.T1.6.6.1.1.m1.1.1">percent</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.6.6.1.1.m1.1c">\%</annotation></semantics></math></span></td>
<td id="S4.T1.6.6.4" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.6.6.4.1" class="ltx_text ltx_font_typewriter">86.93</span></td>
<td id="S4.T1.6.6.5" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.6.6.5.1" class="ltx_text ltx_font_bold">87.62</span></td>
<td id="S4.T1.6.6.6" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.6.6.6.1" class="ltx_text ltx_font_typewriter">87.35</span></td>
<td id="S4.T1.6.6.7" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.6.6.7.1" class="ltx_text ltx_font_typewriter">87.47</span></td>
</tr>
<tr id="S4.T1.7.7" class="ltx_tr">
<td id="S4.T1.7.7.2" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.7.7.2.1" class="ltx_text ltx_font_typewriter">HotpotQA</span></td>
<td id="S4.T1.7.7.3" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.7.7.3.1" class="ltx_text ltx_font_typewriter">36.56</span></td>
<td id="S4.T1.7.7.1" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.7.7.1.1" class="ltx_text ltx_font_bold">+9.38<math id="S4.T1.7.7.1.1.m1.1" class="ltx_Math" alttext="\%" display="inline"><semantics id="S4.T1.7.7.1.1.m1.1a"><mo id="S4.T1.7.7.1.1.m1.1.1" xref="S4.T1.7.7.1.1.m1.1.1.cmml">%</mo><annotation-xml encoding="MathML-Content" id="S4.T1.7.7.1.1.m1.1b"><csymbol cd="latexml" id="S4.T1.7.7.1.1.m1.1.1.cmml" xref="S4.T1.7.7.1.1.m1.1.1">percent</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.7.7.1.1.m1.1c">\%</annotation></semantics></math></span></td>
<td id="S4.T1.7.7.4" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.7.7.4.1" class="ltx_text ltx_font_bold">39.99</span></td>
<td id="S4.T1.7.7.5" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.7.7.5.1" class="ltx_text ltx_font_typewriter">39.01</span></td>
<td id="S4.T1.7.7.6" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.7.7.6.1" class="ltx_text ltx_font_typewriter">39.05</span></td>
<td id="S4.T1.7.7.7" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.7.7.7.1" class="ltx_text ltx_font_typewriter">38.90</span></td>
</tr>
<tr id="S4.T1.8.8" class="ltx_tr">
<td id="S4.T1.8.8.2" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.8.8.2.1" class="ltx_text ltx_font_typewriter">SAMSum</span></td>
<td id="S4.T1.8.8.3" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.8.8.3.1" class="ltx_text ltx_font_typewriter">42.52</span></td>
<td id="S4.T1.8.8.1" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.8.8.1.1" class="ltx_text ltx_font_bold">+0.87<math id="S4.T1.8.8.1.1.m1.1" class="ltx_Math" alttext="\%" display="inline"><semantics id="S4.T1.8.8.1.1.m1.1a"><mo id="S4.T1.8.8.1.1.m1.1.1" xref="S4.T1.8.8.1.1.m1.1.1.cmml">%</mo><annotation-xml encoding="MathML-Content" id="S4.T1.8.8.1.1.m1.1b"><csymbol cd="latexml" id="S4.T1.8.8.1.1.m1.1.1.cmml" xref="S4.T1.8.8.1.1.m1.1.1">percent</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.8.8.1.1.m1.1c">\%</annotation></semantics></math></span></td>
<td id="S4.T1.8.8.4" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.8.8.4.1" class="ltx_text ltx_font_typewriter">42.34</span></td>
<td id="S4.T1.8.8.5" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.8.8.5.1" class="ltx_text ltx_font_typewriter">42.13</span></td>
<td id="S4.T1.8.8.6" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.8.8.6.1" class="ltx_text ltx_font_bold">42.89</span></td>
<td id="S4.T1.8.8.7" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.8.8.7.1" class="ltx_text ltx_font_typewriter">42.48</span></td>
</tr>
<tr id="S4.T1.9.9" class="ltx_tr">
<td id="S4.T1.9.9.2" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.9.9.2.1" class="ltx_text ltx_font_typewriter">2WikiMQA</span></td>
<td id="S4.T1.9.9.3" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.9.9.3.1" class="ltx_text ltx_font_typewriter">22.31</span></td>
<td id="S4.T1.9.9.1" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.9.9.1.1" class="ltx_text ltx_font_bold">+6.41<math id="S4.T1.9.9.1.1.m1.1" class="ltx_Math" alttext="\%" display="inline"><semantics id="S4.T1.9.9.1.1.m1.1a"><mo id="S4.T1.9.9.1.1.m1.1.1" xref="S4.T1.9.9.1.1.m1.1.1.cmml">%</mo><annotation-xml encoding="MathML-Content" id="S4.T1.9.9.1.1.m1.1b"><csymbol cd="latexml" id="S4.T1.9.9.1.1.m1.1.1.cmml" xref="S4.T1.9.9.1.1.m1.1.1">percent</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.9.9.1.1.m1.1c">\%</annotation></semantics></math></span></td>
<td id="S4.T1.9.9.4" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.9.9.4.1" class="ltx_text ltx_font_bold">23.74</span></td>
<td id="S4.T1.9.9.5" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.9.9.5.1" class="ltx_text ltx_font_typewriter">22.75</span></td>
<td id="S4.T1.9.9.6" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.9.9.6.1" class="ltx_text ltx_font_typewriter">22.65</span></td>
<td id="S4.T1.9.9.7" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.9.9.7.1" class="ltx_text ltx_font_typewriter">23.46</span></td>
</tr>
<tr id="S4.T1.10.10" class="ltx_tr">
<td id="S4.T1.10.10.2" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.10.10.2.1" class="ltx_text ltx_font_typewriter">PassageRetrieval</span></td>
<td id="S4.T1.10.10.3" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.10.10.3.1" class="ltx_text ltx_font_typewriter">64.00</span></td>
<td id="S4.T1.10.10.1" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.10.10.1.1" class="ltx_text ltx_font_bold">+33.47<math id="S4.T1.10.10.1.1.m1.1" class="ltx_Math" alttext="\%" display="inline"><semantics id="S4.T1.10.10.1.1.m1.1a"><mo id="S4.T1.10.10.1.1.m1.1.1" xref="S4.T1.10.10.1.1.m1.1.1.cmml">%</mo><annotation-xml encoding="MathML-Content" id="S4.T1.10.10.1.1.m1.1b"><csymbol cd="latexml" id="S4.T1.10.10.1.1.m1.1.1.cmml" xref="S4.T1.10.10.1.1.m1.1.1">percent</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.10.10.1.1.m1.1c">\%</annotation></semantics></math></span></td>
<td id="S4.T1.10.10.4" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.10.10.4.1" class="ltx_text ltx_font_bold">85.42</span></td>
<td id="S4.T1.10.10.5" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.10.10.5.1" class="ltx_text ltx_font_typewriter">78.92</span></td>
<td id="S4.T1.10.10.6" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.10.10.6.1" class="ltx_text ltx_font_typewriter">84.67</span></td>
<td id="S4.T1.10.10.7" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.10.10.7.1" class="ltx_text ltx_font_typewriter">84.08</span></td>
</tr>
<tr id="S4.T1.11.11" class="ltx_tr">
<td id="S4.T1.11.11.2" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.11.11.2.1" class="ltx_text ltx_font_typewriter">Musique</span></td>
<td id="S4.T1.11.11.3" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.11.11.3.1" class="ltx_text ltx_font_typewriter">17.68</span></td>
<td id="S4.T1.11.11.1" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.11.11.1.1" class="ltx_text ltx_font_bold">+6.17<math id="S4.T1.11.11.1.1.m1.1" class="ltx_Math" alttext="\%" display="inline"><semantics id="S4.T1.11.11.1.1.m1.1a"><mo id="S4.T1.11.11.1.1.m1.1.1" xref="S4.T1.11.11.1.1.m1.1.1.cmml">%</mo><annotation-xml encoding="MathML-Content" id="S4.T1.11.11.1.1.m1.1b"><csymbol cd="latexml" id="S4.T1.11.11.1.1.m1.1.1.cmml" xref="S4.T1.11.11.1.1.m1.1.1">percent</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.11.11.1.1.m1.1c">\%</annotation></semantics></math></span></td>
<td id="S4.T1.11.11.4" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.11.11.4.1" class="ltx_text ltx_font_typewriter">17.58</span></td>
<td id="S4.T1.11.11.5" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.11.11.5.1" class="ltx_text ltx_font_typewriter">17.82</span></td>
<td id="S4.T1.11.11.6" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.11.11.6.1" class="ltx_text ltx_font_typewriter">17.93</span></td>
<td id="S4.T1.11.11.7" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.11.11.7.1" class="ltx_text ltx_font_bold">18.77</span></td>
</tr>
<tr id="S4.T1.12.12" class="ltx_tr">
<td id="S4.T1.12.12.2" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.12.12.2.1" class="ltx_text ltx_font_typewriter">LCC</span></td>
<td id="S4.T1.12.12.3" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.12.12.3.1" class="ltx_text ltx_font_typewriter">56.67</span></td>
<td id="S4.T1.12.12.1" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.12.12.1.1" class="ltx_text ltx_font_bold">+0.63<math id="S4.T1.12.12.1.1.m1.1" class="ltx_Math" alttext="\%" display="inline"><semantics id="S4.T1.12.12.1.1.m1.1a"><mo id="S4.T1.12.12.1.1.m1.1.1" xref="S4.T1.12.12.1.1.m1.1.1.cmml">%</mo><annotation-xml encoding="MathML-Content" id="S4.T1.12.12.1.1.m1.1b"><csymbol cd="latexml" id="S4.T1.12.12.1.1.m1.1.1.cmml" xref="S4.T1.12.12.1.1.m1.1.1">percent</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.12.12.1.1.m1.1c">\%</annotation></semantics></math></span></td>
<td id="S4.T1.12.12.4" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.12.12.4.1" class="ltx_text ltx_font_typewriter">54.90</span></td>
<td id="S4.T1.12.12.5" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.12.12.5.1" class="ltx_text ltx_font_bold">57.03</span></td>
<td id="S4.T1.12.12.6" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.12.12.6.1" class="ltx_text ltx_font_typewriter">54.79</span></td>
<td id="S4.T1.12.12.7" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.12.12.7.1" class="ltx_text ltx_font_typewriter">56.79</span></td>
</tr>
<tr id="S4.T1.13.13" class="ltx_tr">
<td id="S4.T1.13.13.2" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.13.13.2.1" class="ltx_text ltx_font_typewriter">GovReport</span></td>
<td id="S4.T1.13.13.3" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.13.13.3.1" class="ltx_text ltx_font_typewriter">31.03</span></td>
<td id="S4.T1.13.13.1" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.13.13.1.1" class="ltx_text ltx_font_bold">+1.90<math id="S4.T1.13.13.1.1.m1.1" class="ltx_Math" alttext="\%" display="inline"><semantics id="S4.T1.13.13.1.1.m1.1a"><mo id="S4.T1.13.13.1.1.m1.1.1" xref="S4.T1.13.13.1.1.m1.1.1.cmml">%</mo><annotation-xml encoding="MathML-Content" id="S4.T1.13.13.1.1.m1.1b"><csymbol cd="latexml" id="S4.T1.13.13.1.1.m1.1.1.cmml" xref="S4.T1.13.13.1.1.m1.1.1">percent</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.13.13.1.1.m1.1c">\%</annotation></semantics></math></span></td>
<td id="S4.T1.13.13.4" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.13.13.4.1" class="ltx_text ltx_font_typewriter">31.24</span></td>
<td id="S4.T1.13.13.5" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.13.13.5.1" class="ltx_text ltx_font_bold">31.62</span></td>
<td id="S4.T1.13.13.6" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.13.13.6.1" class="ltx_text ltx_font_typewriter">31.34</span></td>
<td id="S4.T1.13.13.7" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.13.13.7.1" class="ltx_text ltx_font_typewriter">31.43</span></td>
</tr>
<tr id="S4.T1.14.14" class="ltx_tr">
<td id="S4.T1.14.14.2" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.14.14.2.1" class="ltx_text ltx_font_typewriter">RepoBench-P</span></td>
<td id="S4.T1.14.14.3" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.14.14.3.1" class="ltx_text ltx_font_typewriter">52.97</span></td>
<td id="S4.T1.14.14.1" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.14.14.1.1" class="ltx_text ltx_font_bold">+1.34<math id="S4.T1.14.14.1.1.m1.1" class="ltx_Math" alttext="\%" display="inline"><semantics id="S4.T1.14.14.1.1.m1.1a"><mo id="S4.T1.14.14.1.1.m1.1.1" xref="S4.T1.14.14.1.1.m1.1.1.cmml">%</mo><annotation-xml encoding="MathML-Content" id="S4.T1.14.14.1.1.m1.1b"><csymbol cd="latexml" id="S4.T1.14.14.1.1.m1.1.1.cmml" xref="S4.T1.14.14.1.1.m1.1.1">percent</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.14.14.1.1.m1.1c">\%</annotation></semantics></math></span></td>
<td id="S4.T1.14.14.4" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.14.14.4.1" class="ltx_text ltx_font_typewriter">50.76</span></td>
<td id="S4.T1.14.14.5" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.14.14.5.1" class="ltx_text ltx_font_bold">53.68</span></td>
<td id="S4.T1.14.14.6" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.14.14.6.1" class="ltx_text ltx_font_typewriter">51.34</span></td>
<td id="S4.T1.14.14.7" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.14.14.7.1" class="ltx_text ltx_font_typewriter">52.86</span></td>
</tr>
<tr id="S4.T1.15.15" class="ltx_tr">
<td id="S4.T1.15.15.2" class="ltx_td ltx_align_left ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.15.15.2.1" class="ltx_text ltx_font_typewriter">QMSum</span></td>
<td id="S4.T1.15.15.3" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.15.15.3.1" class="ltx_text ltx_font_typewriter">23.49</span></td>
<td id="S4.T1.15.15.1" class="ltx_td ltx_align_right ltx_border_r" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.15.15.1.1" class="ltx_text ltx_font_bold">+2.13<math id="S4.T1.15.15.1.1.m1.1" class="ltx_Math" alttext="\%" display="inline"><semantics id="S4.T1.15.15.1.1.m1.1a"><mo id="S4.T1.15.15.1.1.m1.1.1" xref="S4.T1.15.15.1.1.m1.1.1.cmml">%</mo><annotation-xml encoding="MathML-Content" id="S4.T1.15.15.1.1.m1.1b"><csymbol cd="latexml" id="S4.T1.15.15.1.1.m1.1.1.cmml" xref="S4.T1.15.15.1.1.m1.1.1">percent</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.15.15.1.1.m1.1c">\%</annotation></semantics></math></span></td>
<td id="S4.T1.15.15.4" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.15.15.4.1" class="ltx_text ltx_font_typewriter">23.82</span></td>
<td id="S4.T1.15.15.5" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.15.15.5.1" class="ltx_text ltx_font_typewriter">23.20</span></td>
<td id="S4.T1.15.15.6" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.15.15.6.1" class="ltx_text ltx_font_bold">23.99</span></td>
<td id="S4.T1.15.15.7" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.15.15.7.1" class="ltx_text ltx_font_typewriter">23.47</span></td>
</tr>
<tr id="S4.T1.16.16" class="ltx_tr">
<td id="S4.T1.16.16.2" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.16.16.2.1" class="ltx_text ltx_font_bold">Avg. score:</span></td>
<td id="S4.T1.16.16.3" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.16.16.3.1" class="ltx_text ltx_font_typewriter">41.90</span></td>
<td id="S4.T1.16.16.1" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.16.16.1.1" class="ltx_text ltx_font_bold">+4.30<math id="S4.T1.16.16.1.1.m1.1" class="ltx_Math" alttext="\%" display="inline"><semantics id="S4.T1.16.16.1.1.m1.1a"><mo id="S4.T1.16.16.1.1.m1.1.1" xref="S4.T1.16.16.1.1.m1.1.1.cmml">%</mo><annotation-xml encoding="MathML-Content" id="S4.T1.16.16.1.1.m1.1b"><csymbol cd="latexml" id="S4.T1.16.16.1.1.m1.1.1.cmml" xref="S4.T1.16.16.1.1.m1.1.1">percent</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.16.16.1.1.m1.1c">\%</annotation></semantics></math></span></td>
<td id="S4.T1.16.16.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.16.16.4.1" class="ltx_text ltx_font_typewriter">43.40</span></td>
<td id="S4.T1.16.16.5" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.16.16.5.1" class="ltx_text ltx_font_typewriter">43.22</span></td>
<td id="S4.T1.16.16.6" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.16.16.6.1" class="ltx_text ltx_font_typewriter">43.44</span></td>
<td id="S4.T1.16.16.7" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S4.T1.16.16.7.1" class="ltx_text ltx_font_bold">43.70</span></td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering ltx_font_typewriter"><span class="ltx_tag ltx_tag_table"><span id="S4.T1.27.1.1" class="ltx_text ltx_font_serif">Table 1</span>: </span>EM-LLM performance on LongBench compared to our baseline InfLLM. <span id="S4.T1.28.2" class="ltx_text ltx_font_serif ltx_font_bold">S</span>: surprise threshold, <span id="S4.T1.29.3" class="ltx_text ltx_font_serif ltx_font_bold">SM</span>: surprise threshold + refinement with modularity, <span id="S4.T1.30.4" class="ltx_text ltx_font_serif ltx_font_bold">S+C</span>: surprise threshold + contiguity buffer, <span id="S4.T1.31.5" class="ltx_text ltx_font_serif ltx_font_bold">SM+C</span>: surprise threshold + refinement with modularity + contiguity buffer. Max Imp. shows the maximum relative improvement over InfLLM across all EM-LLM variants.</figcaption>
</figure>
<div id="S4.SS1.p2" class="ltx_para ltx_noindent">
<p id="S4.SS1.p2.4" class="ltx_p"><span id="S4.SS1.p2.4.1" class="ltx_text ltx_font_typewriter">Results on the LongBench dataset (Table </span><a href="#S4.T1" title="Table 1 ‣ 4.1 Performance of EM-LLM on long-context tasks ‣ 4 Experiments ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">1</span></a><span id="S4.SS1.p2.4.2" class="ltx_text ltx_font_typewriter">) show that our method is able to improve on InfLLM in all but one task, with the best method achieving an overall increase in performance of </span><math id="S4.SS1.p2.1.m1.1" class="ltx_Math" alttext="1.8" display="inline"><semantics id="S4.SS1.p2.1.m1.1a"><mn id="S4.SS1.p2.1.m1.1.1" xref="S4.SS1.p2.1.m1.1.1.cmml">1.8</mn><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.1.m1.1b"><cn type="float" id="S4.SS1.p2.1.m1.1.1.cmml" xref="S4.SS1.p2.1.m1.1.1">1.8</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.1.m1.1c">1.8</annotation></semantics></math><span id="S4.SS1.p2.4.3" class="ltx_text ltx_font_typewriter"> percentage points (a relative improvement of </span><math id="S4.SS1.p2.2.m2.1" class="ltx_Math" alttext="4.3\%" display="inline"><semantics id="S4.SS1.p2.2.m2.1a"><mrow id="S4.SS1.p2.2.m2.1.1" xref="S4.SS1.p2.2.m2.1.1.cmml"><mn id="S4.SS1.p2.2.m2.1.1.2" xref="S4.SS1.p2.2.m2.1.1.2.cmml">4.3</mn><mo id="S4.SS1.p2.2.m2.1.1.1" xref="S4.SS1.p2.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.2.m2.1b"><apply id="S4.SS1.p2.2.m2.1.1.cmml" xref="S4.SS1.p2.2.m2.1.1"><csymbol cd="latexml" id="S4.SS1.p2.2.m2.1.1.1.cmml" xref="S4.SS1.p2.2.m2.1.1.1">percent</csymbol><cn type="float" id="S4.SS1.p2.2.m2.1.1.2.cmml" xref="S4.SS1.p2.2.m2.1.1.2">4.3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.2.m2.1c">4.3\%</annotation></semantics></math><span id="S4.SS1.p2.4.4" class="ltx_text ltx_font_typewriter">). Note that the table shows the best single method in terms of overall performance for each ablation. Looking at individual task performance across all experiments, we are able to beat InfLLM in all tasks (see Appendix </span><a href="#A1.SS1" title="A.1 Supplementary figures ‣ Appendix A Appendix / supplemental material ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">A.1</span></a><span id="S4.SS1.p2.4.5" class="ltx_text ltx_font_typewriter">).
Interestingly, we see an especially large jump in performance on the PassageRetrieval task across all ablations, with up to a </span><math id="S4.SS1.p2.3.m3.1" class="ltx_Math" alttext="33\%" display="inline"><semantics id="S4.SS1.p2.3.m3.1a"><mrow id="S4.SS1.p2.3.m3.1.1" xref="S4.SS1.p2.3.m3.1.1.cmml"><mn id="S4.SS1.p2.3.m3.1.1.2" xref="S4.SS1.p2.3.m3.1.1.2.cmml">33</mn><mo id="S4.SS1.p2.3.m3.1.1.1" xref="S4.SS1.p2.3.m3.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.3.m3.1b"><apply id="S4.SS1.p2.3.m3.1.1.cmml" xref="S4.SS1.p2.3.m3.1.1"><csymbol cd="latexml" id="S4.SS1.p2.3.m3.1.1.1.cmml" xref="S4.SS1.p2.3.m3.1.1.1">percent</csymbol><cn type="integer" id="S4.SS1.p2.3.m3.1.1.2.cmml" xref="S4.SS1.p2.3.m3.1.1.2">33</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.3.m3.1c">33\%</annotation></semantics></math><span id="S4.SS1.p2.4.6" class="ltx_text ltx_font_typewriter"> improvement on InfLLM.
This task requires the model to identify the original paragraph from a summary, a challenging task that tests the model’s ability to accurately recall a wide range of detailed information from a large context concurrently. The substantial improvement on this task highlights the effectiveness of our event segmentation method in enhancing long-term memory recall and retrieval accuracy in LLMs. Additionally, our method achieves a notable </span><math id="S4.SS1.p2.4.m4.1" class="ltx_Math" alttext="9.38\%" display="inline"><semantics id="S4.SS1.p2.4.m4.1a"><mrow id="S4.SS1.p2.4.m4.1.1" xref="S4.SS1.p2.4.m4.1.1.cmml"><mn id="S4.SS1.p2.4.m4.1.1.2" xref="S4.SS1.p2.4.m4.1.1.2.cmml">9.38</mn><mo id="S4.SS1.p2.4.m4.1.1.1" xref="S4.SS1.p2.4.m4.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.4.m4.1b"><apply id="S4.SS1.p2.4.m4.1.1.cmml" xref="S4.SS1.p2.4.m4.1.1"><csymbol cd="latexml" id="S4.SS1.p2.4.m4.1.1.1.cmml" xref="S4.SS1.p2.4.m4.1.1.1">percent</csymbol><cn type="float" id="S4.SS1.p2.4.m4.1.1.2.cmml" xref="S4.SS1.p2.4.m4.1.1.2">9.38</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.4.m4.1c">9.38\%</annotation></semantics></math><span id="S4.SS1.p2.4.7" class="ltx_text ltx_font_typewriter"> improvement on the HotpotQA task, which involves complex reasoning over multiple supporting documents, further emphasising the model’s ability to provide explanations for answers.</span></p>
</div>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_font_typewriter ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="S4.SS2.1.1.1" class="ltx_text ltx_font_serif">4.2</span> </span>Humans and LLM surprise cluster similar tokens together</h3>

<div id="S4.SS2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS2.p1.1" class="ltx_p"><span id="S4.SS2.p1.1.1" class="ltx_text ltx_font_typewriter">As mentioned in Section </span><a href="#S3.SS2" title="3.2 Memory formation via Surprise ‣ 3 EM-LLM: LLM with Episodic Memory ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">3.2</span></a><span id="S4.SS2.p1.1.2" class="ltx_text ltx_font_typewriter">, we employ modularity and conductance as two refinement objectives in our boundary refinement algorithm, due to their qualities in assessing the intra- and inter-event similarities between individual tokens. We will now use such metrics to compare various event segmentation methods, including human event segmentation data. Additionally, we introduce one further, simple metric for this experiment: the ratio between intra- and inter-community similarity (I/IS), calculated for each head and community </span><math id="S4.SS2.p1.1.m1.1" class="ltx_Math" alttext="S" display="inline"><semantics id="S4.SS2.p1.1.m1.1a"><mi id="S4.SS2.p1.1.m1.1.1" xref="S4.SS2.p1.1.m1.1.1.cmml">S</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.1.m1.1b"><ci id="S4.SS2.p1.1.m1.1.1.cmml" xref="S4.SS2.p1.1.m1.1.1">𝑆</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.1.m1.1c">S</annotation></semantics></math><span id="S4.SS2.p1.1.3" class="ltx_text ltx_font_typewriter"> as follows:</span></p>
<table id="S4.E5" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.E5.m1.6" class="ltx_Math" alttext="\text{intra}=\sum_{i\in S,j\in S}A_{ij},\qquad\text{inter}=\sum_{i\in S,j\notin S}A_{ij},\qquad\text{I/IS}\equiv\frac{\text{intra}}{\text{inter}}" display="block"><semantics id="S4.E5.m1.6a"><mrow id="S4.E5.m1.6.6.2" xref="S4.E5.m1.6.6.3.cmml"><mrow id="S4.E5.m1.5.5.1.1" xref="S4.E5.m1.5.5.1.1.cmml"><mtext class="ltx_mathvariant_monospace" id="S4.E5.m1.5.5.1.1.2" xref="S4.E5.m1.5.5.1.1.2a.cmml">intra</mtext><mo rspace="0.111em" id="S4.E5.m1.5.5.1.1.1" xref="S4.E5.m1.5.5.1.1.1.cmml">=</mo><mrow id="S4.E5.m1.5.5.1.1.3" xref="S4.E5.m1.5.5.1.1.3.cmml"><munder id="S4.E5.m1.5.5.1.1.3.1" xref="S4.E5.m1.5.5.1.1.3.1.cmml"><mo movablelimits="false" id="S4.E5.m1.5.5.1.1.3.1.2" xref="S4.E5.m1.5.5.1.1.3.1.2.cmml">∑</mo><mrow id="S4.E5.m1.2.2.2.2" xref="S4.E5.m1.2.2.2.3.cmml"><mrow id="S4.E5.m1.1.1.1.1.1" xref="S4.E5.m1.1.1.1.1.1.cmml"><mi id="S4.E5.m1.1.1.1.1.1.2" xref="S4.E5.m1.1.1.1.1.1.2.cmml">i</mi><mo id="S4.E5.m1.1.1.1.1.1.1" xref="S4.E5.m1.1.1.1.1.1.1.cmml">∈</mo><mi id="S4.E5.m1.1.1.1.1.1.3" xref="S4.E5.m1.1.1.1.1.1.3.cmml">S</mi></mrow><mo id="S4.E5.m1.2.2.2.2.3" xref="S4.E5.m1.2.2.2.3a.cmml">,</mo><mrow id="S4.E5.m1.2.2.2.2.2" xref="S4.E5.m1.2.2.2.2.2.cmml"><mi id="S4.E5.m1.2.2.2.2.2.2" xref="S4.E5.m1.2.2.2.2.2.2.cmml">j</mi><mo id="S4.E5.m1.2.2.2.2.2.1" xref="S4.E5.m1.2.2.2.2.2.1.cmml">∈</mo><mi id="S4.E5.m1.2.2.2.2.2.3" xref="S4.E5.m1.2.2.2.2.2.3.cmml">S</mi></mrow></mrow></munder><msub id="S4.E5.m1.5.5.1.1.3.2" xref="S4.E5.m1.5.5.1.1.3.2.cmml"><mi id="S4.E5.m1.5.5.1.1.3.2.2" xref="S4.E5.m1.5.5.1.1.3.2.2.cmml">A</mi><mrow id="S4.E5.m1.5.5.1.1.3.2.3" xref="S4.E5.m1.5.5.1.1.3.2.3.cmml"><mi id="S4.E5.m1.5.5.1.1.3.2.3.2" xref="S4.E5.m1.5.5.1.1.3.2.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S4.E5.m1.5.5.1.1.3.2.3.1" xref="S4.E5.m1.5.5.1.1.3.2.3.1.cmml">​</mo><mi id="S4.E5.m1.5.5.1.1.3.2.3.3" xref="S4.E5.m1.5.5.1.1.3.2.3.3.cmml">j</mi></mrow></msub></mrow></mrow><mo rspace="2.167em" id="S4.E5.m1.6.6.2.3" xref="S4.E5.m1.6.6.3a.cmml">,</mo><mrow id="S4.E5.m1.6.6.2.2.2" xref="S4.E5.m1.6.6.2.2.3.cmml"><mrow id="S4.E5.m1.6.6.2.2.1.1" xref="S4.E5.m1.6.6.2.2.1.1.cmml"><mtext class="ltx_mathvariant_monospace" id="S4.E5.m1.6.6.2.2.1.1.2" xref="S4.E5.m1.6.6.2.2.1.1.2a.cmml">inter</mtext><mo rspace="0.111em" id="S4.E5.m1.6.6.2.2.1.1.1" xref="S4.E5.m1.6.6.2.2.1.1.1.cmml">=</mo><mrow id="S4.E5.m1.6.6.2.2.1.1.3" xref="S4.E5.m1.6.6.2.2.1.1.3.cmml"><munder id="S4.E5.m1.6.6.2.2.1.1.3.1" xref="S4.E5.m1.6.6.2.2.1.1.3.1.cmml"><mo movablelimits="false" id="S4.E5.m1.6.6.2.2.1.1.3.1.2" xref="S4.E5.m1.6.6.2.2.1.1.3.1.2.cmml">∑</mo><mrow id="S4.E5.m1.4.4.2.2" xref="S4.E5.m1.4.4.2.3.cmml"><mrow id="S4.E5.m1.3.3.1.1.1" xref="S4.E5.m1.3.3.1.1.1.cmml"><mi id="S4.E5.m1.3.3.1.1.1.2" xref="S4.E5.m1.3.3.1.1.1.2.cmml">i</mi><mo id="S4.E5.m1.3.3.1.1.1.1" xref="S4.E5.m1.3.3.1.1.1.1.cmml">∈</mo><mi id="S4.E5.m1.3.3.1.1.1.3" xref="S4.E5.m1.3.3.1.1.1.3.cmml">S</mi></mrow><mo id="S4.E5.m1.4.4.2.2.3" xref="S4.E5.m1.4.4.2.3a.cmml">,</mo><mrow id="S4.E5.m1.4.4.2.2.2" xref="S4.E5.m1.4.4.2.2.2.cmml"><mi id="S4.E5.m1.4.4.2.2.2.2" xref="S4.E5.m1.4.4.2.2.2.2.cmml">j</mi><mo id="S4.E5.m1.4.4.2.2.2.1" xref="S4.E5.m1.4.4.2.2.2.1.cmml">∉</mo><mi id="S4.E5.m1.4.4.2.2.2.3" xref="S4.E5.m1.4.4.2.2.2.3.cmml">S</mi></mrow></mrow></munder><msub id="S4.E5.m1.6.6.2.2.1.1.3.2" xref="S4.E5.m1.6.6.2.2.1.1.3.2.cmml"><mi id="S4.E5.m1.6.6.2.2.1.1.3.2.2" xref="S4.E5.m1.6.6.2.2.1.1.3.2.2.cmml">A</mi><mrow id="S4.E5.m1.6.6.2.2.1.1.3.2.3" xref="S4.E5.m1.6.6.2.2.1.1.3.2.3.cmml"><mi id="S4.E5.m1.6.6.2.2.1.1.3.2.3.2" xref="S4.E5.m1.6.6.2.2.1.1.3.2.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S4.E5.m1.6.6.2.2.1.1.3.2.3.1" xref="S4.E5.m1.6.6.2.2.1.1.3.2.3.1.cmml">​</mo><mi id="S4.E5.m1.6.6.2.2.1.1.3.2.3.3" xref="S4.E5.m1.6.6.2.2.1.1.3.2.3.3.cmml">j</mi></mrow></msub></mrow></mrow><mo rspace="2.167em" id="S4.E5.m1.6.6.2.2.2.3" xref="S4.E5.m1.6.6.2.2.3a.cmml">,</mo><mrow id="S4.E5.m1.6.6.2.2.2.2" xref="S4.E5.m1.6.6.2.2.2.2.cmml"><mtext class="ltx_mathvariant_monospace" id="S4.E5.m1.6.6.2.2.2.2.2" xref="S4.E5.m1.6.6.2.2.2.2.2a.cmml">I/IS</mtext><mo id="S4.E5.m1.6.6.2.2.2.2.1" xref="S4.E5.m1.6.6.2.2.2.2.1.cmml">≡</mo><mfrac id="S4.E5.m1.6.6.2.2.2.2.3" xref="S4.E5.m1.6.6.2.2.2.2.3.cmml"><mtext class="ltx_mathvariant_monospace" id="S4.E5.m1.6.6.2.2.2.2.3.2" xref="S4.E5.m1.6.6.2.2.2.2.3.2a.cmml">intra</mtext><mtext class="ltx_mathvariant_monospace" id="S4.E5.m1.6.6.2.2.2.2.3.3" xref="S4.E5.m1.6.6.2.2.2.2.3.3a.cmml">inter</mtext></mfrac></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.E5.m1.6b"><apply id="S4.E5.m1.6.6.3.cmml" xref="S4.E5.m1.6.6.2"><csymbol cd="ambiguous" id="S4.E5.m1.6.6.3a.cmml" xref="S4.E5.m1.6.6.2.3">formulae-sequence</csymbol><apply id="S4.E5.m1.5.5.1.1.cmml" xref="S4.E5.m1.5.5.1.1"><eq id="S4.E5.m1.5.5.1.1.1.cmml" xref="S4.E5.m1.5.5.1.1.1"></eq><ci id="S4.E5.m1.5.5.1.1.2a.cmml" xref="S4.E5.m1.5.5.1.1.2"><mtext class="ltx_mathvariant_monospace" id="S4.E5.m1.5.5.1.1.2.cmml" xref="S4.E5.m1.5.5.1.1.2">intra</mtext></ci><apply id="S4.E5.m1.5.5.1.1.3.cmml" xref="S4.E5.m1.5.5.1.1.3"><apply id="S4.E5.m1.5.5.1.1.3.1.cmml" xref="S4.E5.m1.5.5.1.1.3.1"><csymbol cd="ambiguous" id="S4.E5.m1.5.5.1.1.3.1.1.cmml" xref="S4.E5.m1.5.5.1.1.3.1">subscript</csymbol><sum id="S4.E5.m1.5.5.1.1.3.1.2.cmml" xref="S4.E5.m1.5.5.1.1.3.1.2"></sum><apply id="S4.E5.m1.2.2.2.3.cmml" xref="S4.E5.m1.2.2.2.2"><csymbol cd="ambiguous" id="S4.E5.m1.2.2.2.3a.cmml" xref="S4.E5.m1.2.2.2.2.3">formulae-sequence</csymbol><apply id="S4.E5.m1.1.1.1.1.1.cmml" xref="S4.E5.m1.1.1.1.1.1"><in id="S4.E5.m1.1.1.1.1.1.1.cmml" xref="S4.E5.m1.1.1.1.1.1.1"></in><ci id="S4.E5.m1.1.1.1.1.1.2.cmml" xref="S4.E5.m1.1.1.1.1.1.2">𝑖</ci><ci id="S4.E5.m1.1.1.1.1.1.3.cmml" xref="S4.E5.m1.1.1.1.1.1.3">𝑆</ci></apply><apply id="S4.E5.m1.2.2.2.2.2.cmml" xref="S4.E5.m1.2.2.2.2.2"><in id="S4.E5.m1.2.2.2.2.2.1.cmml" xref="S4.E5.m1.2.2.2.2.2.1"></in><ci id="S4.E5.m1.2.2.2.2.2.2.cmml" xref="S4.E5.m1.2.2.2.2.2.2">𝑗</ci><ci id="S4.E5.m1.2.2.2.2.2.3.cmml" xref="S4.E5.m1.2.2.2.2.2.3">𝑆</ci></apply></apply></apply><apply id="S4.E5.m1.5.5.1.1.3.2.cmml" xref="S4.E5.m1.5.5.1.1.3.2"><csymbol cd="ambiguous" id="S4.E5.m1.5.5.1.1.3.2.1.cmml" xref="S4.E5.m1.5.5.1.1.3.2">subscript</csymbol><ci id="S4.E5.m1.5.5.1.1.3.2.2.cmml" xref="S4.E5.m1.5.5.1.1.3.2.2">𝐴</ci><apply id="S4.E5.m1.5.5.1.1.3.2.3.cmml" xref="S4.E5.m1.5.5.1.1.3.2.3"><times id="S4.E5.m1.5.5.1.1.3.2.3.1.cmml" xref="S4.E5.m1.5.5.1.1.3.2.3.1"></times><ci id="S4.E5.m1.5.5.1.1.3.2.3.2.cmml" xref="S4.E5.m1.5.5.1.1.3.2.3.2">𝑖</ci><ci id="S4.E5.m1.5.5.1.1.3.2.3.3.cmml" xref="S4.E5.m1.5.5.1.1.3.2.3.3">𝑗</ci></apply></apply></apply></apply><apply id="S4.E5.m1.6.6.2.2.3.cmml" xref="S4.E5.m1.6.6.2.2.2"><csymbol cd="ambiguous" id="S4.E5.m1.6.6.2.2.3a.cmml" xref="S4.E5.m1.6.6.2.2.2.3">formulae-sequence</csymbol><apply id="S4.E5.m1.6.6.2.2.1.1.cmml" xref="S4.E5.m1.6.6.2.2.1.1"><eq id="S4.E5.m1.6.6.2.2.1.1.1.cmml" xref="S4.E5.m1.6.6.2.2.1.1.1"></eq><ci id="S4.E5.m1.6.6.2.2.1.1.2a.cmml" xref="S4.E5.m1.6.6.2.2.1.1.2"><mtext class="ltx_mathvariant_monospace" id="S4.E5.m1.6.6.2.2.1.1.2.cmml" xref="S4.E5.m1.6.6.2.2.1.1.2">inter</mtext></ci><apply id="S4.E5.m1.6.6.2.2.1.1.3.cmml" xref="S4.E5.m1.6.6.2.2.1.1.3"><apply id="S4.E5.m1.6.6.2.2.1.1.3.1.cmml" xref="S4.E5.m1.6.6.2.2.1.1.3.1"><csymbol cd="ambiguous" id="S4.E5.m1.6.6.2.2.1.1.3.1.1.cmml" xref="S4.E5.m1.6.6.2.2.1.1.3.1">subscript</csymbol><sum id="S4.E5.m1.6.6.2.2.1.1.3.1.2.cmml" xref="S4.E5.m1.6.6.2.2.1.1.3.1.2"></sum><apply id="S4.E5.m1.4.4.2.3.cmml" xref="S4.E5.m1.4.4.2.2"><csymbol cd="ambiguous" id="S4.E5.m1.4.4.2.3a.cmml" xref="S4.E5.m1.4.4.2.2.3">formulae-sequence</csymbol><apply id="S4.E5.m1.3.3.1.1.1.cmml" xref="S4.E5.m1.3.3.1.1.1"><in id="S4.E5.m1.3.3.1.1.1.1.cmml" xref="S4.E5.m1.3.3.1.1.1.1"></in><ci id="S4.E5.m1.3.3.1.1.1.2.cmml" xref="S4.E5.m1.3.3.1.1.1.2">𝑖</ci><ci id="S4.E5.m1.3.3.1.1.1.3.cmml" xref="S4.E5.m1.3.3.1.1.1.3">𝑆</ci></apply><apply id="S4.E5.m1.4.4.2.2.2.cmml" xref="S4.E5.m1.4.4.2.2.2"><notin id="S4.E5.m1.4.4.2.2.2.1.cmml" xref="S4.E5.m1.4.4.2.2.2.1"></notin><ci id="S4.E5.m1.4.4.2.2.2.2.cmml" xref="S4.E5.m1.4.4.2.2.2.2">𝑗</ci><ci id="S4.E5.m1.4.4.2.2.2.3.cmml" xref="S4.E5.m1.4.4.2.2.2.3">𝑆</ci></apply></apply></apply><apply id="S4.E5.m1.6.6.2.2.1.1.3.2.cmml" xref="S4.E5.m1.6.6.2.2.1.1.3.2"><csymbol cd="ambiguous" id="S4.E5.m1.6.6.2.2.1.1.3.2.1.cmml" xref="S4.E5.m1.6.6.2.2.1.1.3.2">subscript</csymbol><ci id="S4.E5.m1.6.6.2.2.1.1.3.2.2.cmml" xref="S4.E5.m1.6.6.2.2.1.1.3.2.2">𝐴</ci><apply id="S4.E5.m1.6.6.2.2.1.1.3.2.3.cmml" xref="S4.E5.m1.6.6.2.2.1.1.3.2.3"><times id="S4.E5.m1.6.6.2.2.1.1.3.2.3.1.cmml" xref="S4.E5.m1.6.6.2.2.1.1.3.2.3.1"></times><ci id="S4.E5.m1.6.6.2.2.1.1.3.2.3.2.cmml" xref="S4.E5.m1.6.6.2.2.1.1.3.2.3.2">𝑖</ci><ci id="S4.E5.m1.6.6.2.2.1.1.3.2.3.3.cmml" xref="S4.E5.m1.6.6.2.2.1.1.3.2.3.3">𝑗</ci></apply></apply></apply></apply><apply id="S4.E5.m1.6.6.2.2.2.2.cmml" xref="S4.E5.m1.6.6.2.2.2.2"><equivalent id="S4.E5.m1.6.6.2.2.2.2.1.cmml" xref="S4.E5.m1.6.6.2.2.2.2.1"></equivalent><ci id="S4.E5.m1.6.6.2.2.2.2.2a.cmml" xref="S4.E5.m1.6.6.2.2.2.2.2"><mtext class="ltx_mathvariant_monospace" id="S4.E5.m1.6.6.2.2.2.2.2.cmml" xref="S4.E5.m1.6.6.2.2.2.2.2">I/IS</mtext></ci><apply id="S4.E5.m1.6.6.2.2.2.2.3.cmml" xref="S4.E5.m1.6.6.2.2.2.2.3"><divide id="S4.E5.m1.6.6.2.2.2.2.3.1.cmml" xref="S4.E5.m1.6.6.2.2.2.2.3"></divide><ci id="S4.E5.m1.6.6.2.2.2.2.3.2a.cmml" xref="S4.E5.m1.6.6.2.2.2.2.3.2"><mtext class="ltx_mathvariant_monospace" id="S4.E5.m1.6.6.2.2.2.2.3.2.cmml" xref="S4.E5.m1.6.6.2.2.2.2.3.2">intra</mtext></ci><ci id="S4.E5.m1.6.6.2.2.2.2.3.3a.cmml" xref="S4.E5.m1.6.6.2.2.2.2.3.3"><mtext class="ltx_mathvariant_monospace" id="S4.E5.m1.6.6.2.2.2.2.3.3.cmml" xref="S4.E5.m1.6.6.2.2.2.2.3.3">inter</mtext></ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E5.m1.6c">\text{intra}=\sum_{i\in S,j\in S}A_{ij},\qquad\text{inter}=\sum_{i\in S,j\notin S}A_{ij},\qquad\text{I/IS}\equiv\frac{\text{intra}}{\text{inter}}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(5)</span></td>
</tr></tbody>
</table>
</div>
<div id="S4.SS2.p2" class="ltx_para ltx_noindent">
<p id="S4.SS2.p2.1" class="ltx_p"><cite class="ltx_cite ltx_citemacro_citet"><span class="ltx_text ltx_font_typewriter">Kumar et al.</span> <span id="S4.SS2.p2.1.1.1.1.1" class="ltx_text ltx_font_typewriter">[</span><a href="#bib.bib20" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2023</span></a><span id="S4.SS2.p2.1.2.2.2.1" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S4.SS2.p2.1.3" class="ltx_text ltx_font_typewriter"> found strong correlations between human-perceived events and prediction errors across 3 short podcasts (7-30 minutes on average), when processing the corresponding transcript with an LLM. Taking advantage of such human data and results from previous works on this dataset </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S4.SS2.p2.1.4.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Michelmann et al.</span><span id="S4.SS2.p2.1.5.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib73" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2021</span></a>, <span class="ltx_text ltx_font_typewriter">Lositsky et al.</span><span id="S4.SS2.p2.1.5.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib74" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2016</span></a><span id="S4.SS2.p2.1.6.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S4.SS2.p2.1.7" class="ltx_text ltx_font_typewriter">, we compare the segmentation quality and correlation with human data for each of our methods (</span><a href="#S4.F3" title="Figure 3 ‣ 4.2 Humans and LLM surprise cluster similar tokens together ‣ 4 Experiments ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">3</span></a><span id="S4.SS2.p2.1.8" class="ltx_text ltx_font_typewriter">) using our similarity metrics.</span></p>
</div>
<figure id="S4.F3" class="ltx_figure"><img src="/html/2407.09450/assets/x3.png" id="S4.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="175" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering ltx_font_typewriter"><span class="ltx_tag ltx_tag_figure"><span id="S4.F3.22.1.1" class="ltx_text ltx_font_serif">Figure 3</span>: </span>Comparison of human event segmentation with different computational segmentation methods in two human-annotated audio datasets (see Appendix <a href="#A1.SS3" title="A.3 Analysis of human data ‣ Appendix A Appendix / supplemental material ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.3</span></a>). <span id="S4.F3.23.2" class="ltx_text ltx_font_serif ltx_font_bold">(A)</span> Difference in metrics for the cohesion and separation of KV cache of LLaMA2 attention heads. The graphs report the difference of each method with the corresponding random segmentation. <span id="S4.F3.24.3" class="ltx_text ltx_font_serif ltx_font_bold">(B)</span> Distance between human reports and different methods.
In both sets of results, fixed methods (<span id="S4.F3.25.4" class="ltx_text ltx_font_serif ltx_font_italic">F</span>, <span id="S4.F3.26.5" class="ltx_text ltx_font_serif ltx_font_italic">FM</span>, <span id="S4.F3.27.6" class="ltx_text ltx_font_serif ltx_font_italic">FC</span>) perform worse than their surprise-based counterparts (<span id="S4.F3.28.7" class="ltx_text ltx_font_serif ltx_font_italic">S</span>, <span id="S4.F3.29.8" class="ltx_text ltx_font_serif ltx_font_italic">SM</span>, <span id="S4.F3.30.9" class="ltx_text ltx_font_serif ltx_font_italic">SC</span>) with InfLLM’s method (<span id="S4.F3.31.10" class="ltx_text ltx_font_serif ltx_font_italic">F</span>) performing worse than random. </figcaption>
</figure>
<div id="S4.SS2.p3" class="ltx_para ltx_noindent">
<p id="S4.SS2.p3.8" class="ltx_p"><span id="S4.SS2.p3.8.1" class="ltx_text ltx_font_typewriter">As shown in Fig. </span><a href="#S4.F3" title="Figure 3 ‣ 4.2 Humans and LLM surprise cluster similar tokens together ‣ 4 Experiments ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">3</span></a><span id="S4.SS2.p3.8.2" class="ltx_text ltx_font_typewriter">A, human-perceived events achieve significantly higher scores in similarity metrics compared to fixed or random events, suggesting that surprise is indeed an important factor for humans in their own perception of events. Furthermore, surprise-only segmentation (</span><math id="S4.SS2.p3.1.m1.1" class="ltx_Math" alttext="\mathbf{S}" display="inline"><semantics id="S4.SS2.p3.1.m1.1a"><mi id="S4.SS2.p3.1.m1.1.1" xref="S4.SS2.p3.1.m1.1.1.cmml">𝐒</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p3.1.m1.1b"><ci id="S4.SS2.p3.1.m1.1.1.cmml" xref="S4.SS2.p3.1.m1.1.1">𝐒</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p3.1.m1.1c">\mathbf{S}</annotation></semantics></math><span id="S4.SS2.p3.8.3" class="ltx_text ltx_font_typewriter">) achieves very similar results to humans, while the addition of our refinement algorithm (</span><math id="S4.SS2.p3.2.m2.1" class="ltx_Math" alttext="\mathbf{SM}" display="inline"><semantics id="S4.SS2.p3.2.m2.1a"><mi id="S4.SS2.p3.2.m2.1.1" xref="S4.SS2.p3.2.m2.1.1.cmml">𝐒𝐌</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p3.2.m2.1b"><ci id="S4.SS2.p3.2.m2.1.1.cmml" xref="S4.SS2.p3.2.m2.1.1">𝐒𝐌</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p3.2.m2.1c">\mathbf{SM}</annotation></semantics></math><span id="S4.SS2.p3.8.4" class="ltx_text ltx_font_typewriter">, </span><math id="S4.SS2.p3.3.m3.1" class="ltx_Math" alttext="\mathbf{SC}" display="inline"><semantics id="S4.SS2.p3.3.m3.1a"><mi id="S4.SS2.p3.3.m3.1.1" xref="S4.SS2.p3.3.m3.1.1.cmml">𝐒𝐂</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p3.3.m3.1b"><ci id="S4.SS2.p3.3.m3.1.1.cmml" xref="S4.SS2.p3.3.m3.1.1">𝐒𝐂</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p3.3.m3.1c">\mathbf{SC}</annotation></semantics></math><span id="S4.SS2.p3.8.5" class="ltx_text ltx_font_typewriter">, </span><math id="S4.SS2.p3.4.m4.1" class="ltx_Math" alttext="\mathbf{FM}" display="inline"><semantics id="S4.SS2.p3.4.m4.1a"><mi id="S4.SS2.p3.4.m4.1.1" xref="S4.SS2.p3.4.m4.1.1.cmml">𝐅𝐌</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p3.4.m4.1b"><ci id="S4.SS2.p3.4.m4.1.1.cmml" xref="S4.SS2.p3.4.m4.1.1">𝐅𝐌</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p3.4.m4.1c">\mathbf{FM}</annotation></semantics></math><span id="S4.SS2.p3.8.6" class="ltx_text ltx_font_typewriter">, </span><math id="S4.SS2.p3.5.m5.1" class="ltx_Math" alttext="\mathbf{FC}" display="inline"><semantics id="S4.SS2.p3.5.m5.1a"><mi id="S4.SS2.p3.5.m5.1.1" xref="S4.SS2.p3.5.m5.1.1.cmml">𝐅𝐂</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p3.5.m5.1b"><ci id="S4.SS2.p3.5.m5.1.1.cmml" xref="S4.SS2.p3.5.m5.1.1">𝐅𝐂</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p3.5.m5.1c">\mathbf{FC}</annotation></semantics></math><span id="S4.SS2.p3.8.7" class="ltx_text ltx_font_typewriter">) significantly improves performance. Fig. </span><a href="#S4.F3" title="Figure 3 ‣ 4.2 Humans and LLM surprise cluster similar tokens together ‣ 4 Experiments ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">3</span></a><span id="S4.SS2.p3.8.8" class="ltx_text ltx_font_typewriter">B further shows that surprise-based methods (</span><math id="S4.SS2.p3.6.m6.1" class="ltx_Math" alttext="\mathbf{S}" display="inline"><semantics id="S4.SS2.p3.6.m6.1a"><mi id="S4.SS2.p3.6.m6.1.1" xref="S4.SS2.p3.6.m6.1.1.cmml">𝐒</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p3.6.m6.1b"><ci id="S4.SS2.p3.6.m6.1.1.cmml" xref="S4.SS2.p3.6.m6.1.1">𝐒</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p3.6.m6.1c">\mathbf{S}</annotation></semantics></math><span id="S4.SS2.p3.8.9" class="ltx_text ltx_font_typewriter">, </span><math id="S4.SS2.p3.7.m7.1" class="ltx_Math" alttext="\mathbf{SM}" display="inline"><semantics id="S4.SS2.p3.7.m7.1a"><mi id="S4.SS2.p3.7.m7.1.1" xref="S4.SS2.p3.7.m7.1.1.cmml">𝐒𝐌</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p3.7.m7.1b"><ci id="S4.SS2.p3.7.m7.1.1.cmml" xref="S4.SS2.p3.7.m7.1.1">𝐒𝐌</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p3.7.m7.1c">\mathbf{SM}</annotation></semantics></math><span id="S4.SS2.p3.8.10" class="ltx_text ltx_font_typewriter">, </span><math id="S4.SS2.p3.8.m8.1" class="ltx_Math" alttext="\mathbf{SC}" display="inline"><semantics id="S4.SS2.p3.8.m8.1a"><mi id="S4.SS2.p3.8.m8.1.1" xref="S4.SS2.p3.8.m8.1.1.cmml">𝐒𝐂</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p3.8.m8.1b"><ci id="S4.SS2.p3.8.m8.1.1.cmml" xref="S4.SS2.p3.8.m8.1.1">𝐒𝐂</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p3.8.m8.1c">\mathbf{SC}</annotation></semantics></math><span id="S4.SS2.p3.8.11" class="ltx_text ltx_font_typewriter">), consistently identify event boundaries that are closest to those perceived by humans.</span></p>
</div>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_font_typewriter ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="S4.SS3.1.1.1" class="ltx_text ltx_font_serif">4.3</span> </span>Comparing segmentation methods</h3>

<figure id="S4.T2" class="ltx_table">
<table id="S4.T2.63" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T2.63.64.1" class="ltx_tr">
<th id="S4.T2.63.64.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_th ltx_th_column ltx_th_row ltx_border_tt">
<span id="S4.T2.63.64.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T2.63.64.1.1.1.1" class="ltx_p" style="width:31.3pt;"><span id="S4.T2.63.64.1.1.1.1.1" class="ltx_text ltx_font_bold">LLM</span></span>
</span>
</th>
<th id="S4.T2.63.64.1.2" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_tt"><span id="S4.T2.63.64.1.2.1" class="ltx_text ltx_font_bold">Metric</span></th>
<th id="S4.T2.63.64.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T2.63.64.1.3.1" class="ltx_text ltx_font_bold">F</span></th>
<th id="S4.T2.63.64.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T2.63.64.1.4.1" class="ltx_text ltx_font_bold">FM</span></th>
<th id="S4.T2.63.64.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T2.63.64.1.5.1" class="ltx_text ltx_font_bold">FC</span></th>
<th id="S4.T2.63.64.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T2.63.64.1.6.1" class="ltx_text ltx_font_bold">S</span></th>
<th id="S4.T2.63.64.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T2.63.64.1.7.1" class="ltx_text ltx_font_bold">SM</span></th>
<th id="S4.T2.63.64.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T2.63.64.1.8.1" class="ltx_text ltx_font_bold">SC</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T2.7.7" class="ltx_tr">
<th id="S4.T2.7.7.8" class="ltx_td ltx_align_justify ltx_align_top ltx_th ltx_th_row ltx_border_t" rowspan="3">
<span id="S4.T2.7.7.8.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T2.7.7.8.1.1" class="ltx_p" style="width:31.3pt;"><span id="S4.T2.7.7.8.1.1.1" class="ltx_text ltx_font_typewriter">
<span id="S4.T2.7.7.8.1.1.1.1" class="ltx_inline-block ltx_parbox ltx_align_middle" style="width:31.3pt;">
<span id="S4.T2.7.7.8.1.1.1.1.1" class="ltx_p">Mistral-7B</span>
</span></span></span>
</span>
</th>
<td id="S4.T2.1.1.1" class="ltx_td ltx_align_right ltx_border_t">
<span id="S4.T2.1.1.1.1" class="ltx_text ltx_font_typewriter">Mod </span><math id="S4.T2.1.1.1.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S4.T2.1.1.1.m1.1a"><mo stretchy="false" id="S4.T2.1.1.1.m1.1.1" xref="S4.T2.1.1.1.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S4.T2.1.1.1.m1.1b"><ci id="S4.T2.1.1.1.m1.1.1.cmml" xref="S4.T2.1.1.1.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.1.1.1.m1.1c">\uparrow</annotation></semantics></math>
</td>
<td id="S4.T2.2.2.2" class="ltx_td ltx_align_right ltx_border_t">
<span id="S4.T2.2.2.2.1" class="ltx_text ltx_font_typewriter">-2.3 </span><math id="S4.T2.2.2.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.2.2.2.m1.1a"><mo id="S4.T2.2.2.2.m1.1.1" xref="S4.T2.2.2.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.2.2.2.m1.1b"><csymbol cd="latexml" id="S4.T2.2.2.2.m1.1.1.cmml" xref="S4.T2.2.2.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.2.2.2.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.2.2.2.2" class="ltx_text ltx_font_typewriter"> 4.1</span>
</td>
<td id="S4.T2.3.3.3" class="ltx_td ltx_align_right ltx_border_t">
<span id="S4.T2.3.3.3.1" class="ltx_text ltx_font_typewriter">29.2 </span><math id="S4.T2.3.3.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.3.3.3.m1.1a"><mo id="S4.T2.3.3.3.m1.1.1" xref="S4.T2.3.3.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.3.3.3.m1.1b"><csymbol cd="latexml" id="S4.T2.3.3.3.m1.1.1.cmml" xref="S4.T2.3.3.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.3.3.3.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.3.3.3.2" class="ltx_text ltx_font_typewriter"> 44.0</span>
</td>
<td id="S4.T2.4.4.4" class="ltx_td ltx_align_right ltx_border_t">
<span id="S4.T2.4.4.4.1" class="ltx_text ltx_font_typewriter">6.7 </span><math id="S4.T2.4.4.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.4.4.4.m1.1a"><mo id="S4.T2.4.4.4.m1.1.1" xref="S4.T2.4.4.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.4.4.4.m1.1b"><csymbol cd="latexml" id="S4.T2.4.4.4.m1.1.1.cmml" xref="S4.T2.4.4.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.4.4.4.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.4.4.4.2" class="ltx_text ltx_font_typewriter"> 25.9</span>
</td>
<td id="S4.T2.5.5.5" class="ltx_td ltx_align_right ltx_border_t">
<span id="S4.T2.5.5.5.1" class="ltx_text ltx_font_typewriter">18.6 </span><math id="S4.T2.5.5.5.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.5.5.5.m1.1a"><mo id="S4.T2.5.5.5.m1.1.1" xref="S4.T2.5.5.5.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.5.5.5.m1.1b"><csymbol cd="latexml" id="S4.T2.5.5.5.m1.1.1.cmml" xref="S4.T2.5.5.5.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.5.5.5.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.5.5.5.2" class="ltx_text ltx_font_typewriter"> 29.6</span>
</td>
<td id="S4.T2.6.6.6" class="ltx_td ltx_align_right ltx_border_t"><span id="S4.T2.6.6.6.1" class="ltx_text ltx_font_bold">39.9 <math id="S4.T2.6.6.6.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.6.6.6.1.m1.1a"><mo id="S4.T2.6.6.6.1.m1.1.1" xref="S4.T2.6.6.6.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.6.6.6.1.m1.1b"><csymbol cd="latexml" id="S4.T2.6.6.6.1.m1.1.1.cmml" xref="S4.T2.6.6.6.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.6.6.6.1.m1.1c">\pm</annotation></semantics></math> 55.5</span></td>
<td id="S4.T2.7.7.7" class="ltx_td ltx_align_right ltx_border_t">
<span id="S4.T2.7.7.7.1" class="ltx_text ltx_font_typewriter">29.5 </span><math id="S4.T2.7.7.7.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.7.7.7.m1.1a"><mo id="S4.T2.7.7.7.m1.1.1" xref="S4.T2.7.7.7.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.7.7.7.m1.1b"><csymbol cd="latexml" id="S4.T2.7.7.7.m1.1.1.cmml" xref="S4.T2.7.7.7.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.7.7.7.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.7.7.7.2" class="ltx_text ltx_font_typewriter"> 42.7</span>
</td>
</tr>
<tr id="S4.T2.14.14" class="ltx_tr">
<td id="S4.T2.8.8.1" class="ltx_td ltx_align_right">
<span id="S4.T2.8.8.1.1" class="ltx_text ltx_font_typewriter">Con </span><math id="S4.T2.8.8.1.m1.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="S4.T2.8.8.1.m1.1a"><mo stretchy="false" id="S4.T2.8.8.1.m1.1.1" xref="S4.T2.8.8.1.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="S4.T2.8.8.1.m1.1b"><ci id="S4.T2.8.8.1.m1.1.1.cmml" xref="S4.T2.8.8.1.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.8.8.1.m1.1c">\downarrow</annotation></semantics></math>
</td>
<td id="S4.T2.9.9.2" class="ltx_td ltx_align_right">
<span id="S4.T2.9.9.2.1" class="ltx_text ltx_font_typewriter">9.1 </span><math id="S4.T2.9.9.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.9.9.2.m1.1a"><mo id="S4.T2.9.9.2.m1.1.1" xref="S4.T2.9.9.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.9.9.2.m1.1b"><csymbol cd="latexml" id="S4.T2.9.9.2.m1.1.1.cmml" xref="S4.T2.9.9.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.9.9.2.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.9.9.2.2" class="ltx_text ltx_font_typewriter"> 8.7</span>
</td>
<td id="S4.T2.10.10.3" class="ltx_td ltx_align_right">
<span id="S4.T2.10.10.3.1" class="ltx_text ltx_font_typewriter">-16.9 </span><math id="S4.T2.10.10.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.10.10.3.m1.1a"><mo id="S4.T2.10.10.3.m1.1.1" xref="S4.T2.10.10.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.10.10.3.m1.1b"><csymbol cd="latexml" id="S4.T2.10.10.3.m1.1.1.cmml" xref="S4.T2.10.10.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.10.10.3.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.10.10.3.2" class="ltx_text ltx_font_typewriter"> 6.7</span>
</td>
<td id="S4.T2.11.11.4" class="ltx_td ltx_align_right">
<span id="S4.T2.11.11.4.1" class="ltx_text ltx_font_typewriter">-12.5 </span><math id="S4.T2.11.11.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.11.11.4.m1.1a"><mo id="S4.T2.11.11.4.m1.1.1" xref="S4.T2.11.11.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.11.11.4.m1.1b"><csymbol cd="latexml" id="S4.T2.11.11.4.m1.1.1.cmml" xref="S4.T2.11.11.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.11.11.4.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.11.11.4.2" class="ltx_text ltx_font_typewriter"> 9.6</span>
</td>
<td id="S4.T2.12.12.5" class="ltx_td ltx_align_right">
<span id="S4.T2.12.12.5.1" class="ltx_text ltx_font_typewriter">-23.6 </span><math id="S4.T2.12.12.5.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.12.12.5.m1.1a"><mo id="S4.T2.12.12.5.m1.1.1" xref="S4.T2.12.12.5.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.12.12.5.m1.1b"><csymbol cd="latexml" id="S4.T2.12.12.5.m1.1.1.cmml" xref="S4.T2.12.12.5.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.12.12.5.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.12.12.5.2" class="ltx_text ltx_font_typewriter"> 9.4</span>
</td>
<td id="S4.T2.13.13.6" class="ltx_td ltx_align_right">
<span id="S4.T2.13.13.6.1" class="ltx_text ltx_font_typewriter">-24.6 </span><math id="S4.T2.13.13.6.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.13.13.6.m1.1a"><mo id="S4.T2.13.13.6.m1.1.1" xref="S4.T2.13.13.6.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.13.13.6.m1.1b"><csymbol cd="latexml" id="S4.T2.13.13.6.m1.1.1.cmml" xref="S4.T2.13.13.6.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.13.13.6.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.13.13.6.2" class="ltx_text ltx_font_typewriter"> 9.3</span>
</td>
<td id="S4.T2.14.14.7" class="ltx_td ltx_align_right"><span id="S4.T2.14.14.7.1" class="ltx_text ltx_font_bold">-27.6 <math id="S4.T2.14.14.7.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.14.14.7.1.m1.1a"><mo id="S4.T2.14.14.7.1.m1.1.1" xref="S4.T2.14.14.7.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.14.14.7.1.m1.1b"><csymbol cd="latexml" id="S4.T2.14.14.7.1.m1.1.1.cmml" xref="S4.T2.14.14.7.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.14.14.7.1.m1.1c">\pm</annotation></semantics></math> 9.8</span></td>
</tr>
<tr id="S4.T2.21.21" class="ltx_tr">
<td id="S4.T2.15.15.1" class="ltx_td ltx_align_right">
<span id="S4.T2.15.15.1.1" class="ltx_text ltx_font_typewriter">I/IS </span><math id="S4.T2.15.15.1.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S4.T2.15.15.1.m1.1a"><mo stretchy="false" id="S4.T2.15.15.1.m1.1.1" xref="S4.T2.15.15.1.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S4.T2.15.15.1.m1.1b"><ci id="S4.T2.15.15.1.m1.1.1.cmml" xref="S4.T2.15.15.1.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.15.15.1.m1.1c">\uparrow</annotation></semantics></math>
</td>
<td id="S4.T2.16.16.2" class="ltx_td ltx_align_right">
<span id="S4.T2.16.16.2.1" class="ltx_text ltx_font_typewriter">-4.3 </span><math id="S4.T2.16.16.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.16.16.2.m1.1a"><mo id="S4.T2.16.16.2.m1.1.1" xref="S4.T2.16.16.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.16.16.2.m1.1b"><csymbol cd="latexml" id="S4.T2.16.16.2.m1.1.1.cmml" xref="S4.T2.16.16.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.16.16.2.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.16.16.2.2" class="ltx_text ltx_font_typewriter"> 4.0</span>
</td>
<td id="S4.T2.17.17.3" class="ltx_td ltx_align_right">
<span id="S4.T2.17.17.3.1" class="ltx_text ltx_font_typewriter">31.2 </span><math id="S4.T2.17.17.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.17.17.3.m1.1a"><mo id="S4.T2.17.17.3.m1.1.1" xref="S4.T2.17.17.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.17.17.3.m1.1b"><csymbol cd="latexml" id="S4.T2.17.17.3.m1.1.1.cmml" xref="S4.T2.17.17.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.17.17.3.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.17.17.3.2" class="ltx_text ltx_font_typewriter"> 21.4</span>
</td>
<td id="S4.T2.18.18.4" class="ltx_td ltx_align_right">
<span id="S4.T2.18.18.4.1" class="ltx_text ltx_font_typewriter">3.7 </span><math id="S4.T2.18.18.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.18.18.4.m1.1a"><mo id="S4.T2.18.18.4.m1.1.1" xref="S4.T2.18.18.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.18.18.4.m1.1b"><csymbol cd="latexml" id="S4.T2.18.18.4.m1.1.1.cmml" xref="S4.T2.18.18.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.18.18.4.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.18.18.4.2" class="ltx_text ltx_font_typewriter"> 14.9</span>
</td>
<td id="S4.T2.19.19.5" class="ltx_td ltx_align_right">
<span id="S4.T2.19.19.5.1" class="ltx_text ltx_font_typewriter">17.9 </span><math id="S4.T2.19.19.5.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.19.19.5.m1.1a"><mo id="S4.T2.19.19.5.m1.1.1" xref="S4.T2.19.19.5.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.19.19.5.m1.1b"><csymbol cd="latexml" id="S4.T2.19.19.5.m1.1.1.cmml" xref="S4.T2.19.19.5.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.19.19.5.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.19.19.5.2" class="ltx_text ltx_font_typewriter"> 17.0</span>
</td>
<td id="S4.T2.20.20.6" class="ltx_td ltx_align_right"><span id="S4.T2.20.20.6.1" class="ltx_text ltx_font_bold">35.3 <math id="S4.T2.20.20.6.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.20.20.6.1.m1.1a"><mo id="S4.T2.20.20.6.1.m1.1.1" xref="S4.T2.20.20.6.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.20.20.6.1.m1.1b"><csymbol cd="latexml" id="S4.T2.20.20.6.1.m1.1.1.cmml" xref="S4.T2.20.20.6.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.20.20.6.1.m1.1c">\pm</annotation></semantics></math> 27.7</span></td>
<td id="S4.T2.21.21.7" class="ltx_td ltx_align_right">
<span id="S4.T2.21.21.7.1" class="ltx_text ltx_font_typewriter">21.6 </span><math id="S4.T2.21.21.7.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.21.21.7.m1.1a"><mo id="S4.T2.21.21.7.m1.1.1" xref="S4.T2.21.21.7.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.21.21.7.m1.1b"><csymbol cd="latexml" id="S4.T2.21.21.7.m1.1.1.cmml" xref="S4.T2.21.21.7.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.21.21.7.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.21.21.7.2" class="ltx_text ltx_font_typewriter"> 22.4</span>
</td>
</tr>
<tr id="S4.T2.28.28" class="ltx_tr">
<th id="S4.T2.28.28.8" class="ltx_td ltx_align_justify ltx_align_top ltx_th ltx_th_row ltx_border_t" rowspan="3">
<span id="S4.T2.28.28.8.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T2.28.28.8.1.1" class="ltx_p" style="width:31.3pt;"><span id="S4.T2.28.28.8.1.1.1" class="ltx_text ltx_font_typewriter">
<span id="S4.T2.28.28.8.1.1.1.1" class="ltx_inline-block ltx_parbox ltx_align_middle" style="width:31.3pt;">
<span id="S4.T2.28.28.8.1.1.1.1.1" class="ltx_p">LLaMA2-7B</span>
</span></span></span>
</span>
</th>
<td id="S4.T2.22.22.1" class="ltx_td ltx_align_right ltx_border_t">
<span id="S4.T2.22.22.1.1" class="ltx_text ltx_font_typewriter">Mod </span><math id="S4.T2.22.22.1.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S4.T2.22.22.1.m1.1a"><mo stretchy="false" id="S4.T2.22.22.1.m1.1.1" xref="S4.T2.22.22.1.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S4.T2.22.22.1.m1.1b"><ci id="S4.T2.22.22.1.m1.1.1.cmml" xref="S4.T2.22.22.1.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.22.22.1.m1.1c">\uparrow</annotation></semantics></math>
</td>
<td id="S4.T2.23.23.2" class="ltx_td ltx_align_right ltx_border_t">
<span id="S4.T2.23.23.2.1" class="ltx_text ltx_font_typewriter">-1.1 </span><math id="S4.T2.23.23.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.23.23.2.m1.1a"><mo id="S4.T2.23.23.2.m1.1.1" xref="S4.T2.23.23.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.23.23.2.m1.1b"><csymbol cd="latexml" id="S4.T2.23.23.2.m1.1.1.cmml" xref="S4.T2.23.23.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.23.23.2.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.23.23.2.2" class="ltx_text ltx_font_typewriter"> 4.3</span>
</td>
<td id="S4.T2.24.24.3" class="ltx_td ltx_align_right ltx_border_t">
<span id="S4.T2.24.24.3.1" class="ltx_text ltx_font_typewriter">13.4 </span><math id="S4.T2.24.24.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.24.24.3.m1.1a"><mo id="S4.T2.24.24.3.m1.1.1" xref="S4.T2.24.24.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.24.24.3.m1.1b"><csymbol cd="latexml" id="S4.T2.24.24.3.m1.1.1.cmml" xref="S4.T2.24.24.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.24.24.3.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.24.24.3.2" class="ltx_text ltx_font_typewriter"> 19.5</span>
</td>
<td id="S4.T2.25.25.4" class="ltx_td ltx_align_right ltx_border_t">
<span id="S4.T2.25.25.4.1" class="ltx_text ltx_font_typewriter">0.6 </span><math id="S4.T2.25.25.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.25.25.4.m1.1a"><mo id="S4.T2.25.25.4.m1.1.1" xref="S4.T2.25.25.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.25.25.4.m1.1b"><csymbol cd="latexml" id="S4.T2.25.25.4.m1.1.1.cmml" xref="S4.T2.25.25.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.25.25.4.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.25.25.4.2" class="ltx_text ltx_font_typewriter"> 7.3</span>
</td>
<td id="S4.T2.26.26.5" class="ltx_td ltx_align_right ltx_border_t">
<span id="S4.T2.26.26.5.1" class="ltx_text ltx_font_typewriter">8.7 </span><math id="S4.T2.26.26.5.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.26.26.5.m1.1a"><mo id="S4.T2.26.26.5.m1.1.1" xref="S4.T2.26.26.5.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.26.26.5.m1.1b"><csymbol cd="latexml" id="S4.T2.26.26.5.m1.1.1.cmml" xref="S4.T2.26.26.5.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.26.26.5.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.26.26.5.2" class="ltx_text ltx_font_typewriter"> 16.0</span>
</td>
<td id="S4.T2.27.27.6" class="ltx_td ltx_align_right ltx_border_t"><span id="S4.T2.27.27.6.1" class="ltx_text ltx_font_bold">18.7 <math id="S4.T2.27.27.6.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.27.27.6.1.m1.1a"><mo id="S4.T2.27.27.6.1.m1.1.1" xref="S4.T2.27.27.6.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.27.27.6.1.m1.1b"><csymbol cd="latexml" id="S4.T2.27.27.6.1.m1.1.1.cmml" xref="S4.T2.27.27.6.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.27.27.6.1.m1.1c">\pm</annotation></semantics></math> 26.4</span></td>
<td id="S4.T2.28.28.7" class="ltx_td ltx_align_right ltx_border_t">
<span id="S4.T2.28.28.7.1" class="ltx_text ltx_font_typewriter">11.5 </span><math id="S4.T2.28.28.7.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.28.28.7.m1.1a"><mo id="S4.T2.28.28.7.m1.1.1" xref="S4.T2.28.28.7.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.28.28.7.m1.1b"><csymbol cd="latexml" id="S4.T2.28.28.7.m1.1.1.cmml" xref="S4.T2.28.28.7.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.28.28.7.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.28.28.7.2" class="ltx_text ltx_font_typewriter"> 19.4</span>
</td>
</tr>
<tr id="S4.T2.35.35" class="ltx_tr">
<td id="S4.T2.29.29.1" class="ltx_td ltx_align_right">
<span id="S4.T2.29.29.1.1" class="ltx_text ltx_font_typewriter">Con </span><math id="S4.T2.29.29.1.m1.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="S4.T2.29.29.1.m1.1a"><mo stretchy="false" id="S4.T2.29.29.1.m1.1.1" xref="S4.T2.29.29.1.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="S4.T2.29.29.1.m1.1b"><ci id="S4.T2.29.29.1.m1.1.1.cmml" xref="S4.T2.29.29.1.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.29.29.1.m1.1c">\downarrow</annotation></semantics></math>
</td>
<td id="S4.T2.30.30.2" class="ltx_td ltx_align_right">
<span id="S4.T2.30.30.2.1" class="ltx_text ltx_font_typewriter">11.9 </span><math id="S4.T2.30.30.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.30.30.2.m1.1a"><mo id="S4.T2.30.30.2.m1.1.1" xref="S4.T2.30.30.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.30.30.2.m1.1b"><csymbol cd="latexml" id="S4.T2.30.30.2.m1.1.1.cmml" xref="S4.T2.30.30.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.30.30.2.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.30.30.2.2" class="ltx_text ltx_font_typewriter"> 9.8</span>
</td>
<td id="S4.T2.31.31.3" class="ltx_td ltx_align_right">
<span id="S4.T2.31.31.3.1" class="ltx_text ltx_font_typewriter">-18.8 </span><math id="S4.T2.31.31.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.31.31.3.m1.1a"><mo id="S4.T2.31.31.3.m1.1.1" xref="S4.T2.31.31.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.31.31.3.m1.1b"><csymbol cd="latexml" id="S4.T2.31.31.3.m1.1.1.cmml" xref="S4.T2.31.31.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.31.31.3.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.31.31.3.2" class="ltx_text ltx_font_typewriter"> 7.4</span>
</td>
<td id="S4.T2.32.32.4" class="ltx_td ltx_align_right">
<span id="S4.T2.32.32.4.1" class="ltx_text ltx_font_typewriter">-13.7 </span><math id="S4.T2.32.32.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.32.32.4.m1.1a"><mo id="S4.T2.32.32.4.m1.1.1" xref="S4.T2.32.32.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.32.32.4.m1.1b"><csymbol cd="latexml" id="S4.T2.32.32.4.m1.1.1.cmml" xref="S4.T2.32.32.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.32.32.4.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.32.32.4.2" class="ltx_text ltx_font_typewriter"> 10.9</span>
</td>
<td id="S4.T2.33.33.5" class="ltx_td ltx_align_right">
<span id="S4.T2.33.33.5.1" class="ltx_text ltx_font_typewriter">-29.5 </span><math id="S4.T2.33.33.5.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.33.33.5.m1.1a"><mo id="S4.T2.33.33.5.m1.1.1" xref="S4.T2.33.33.5.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.33.33.5.m1.1b"><csymbol cd="latexml" id="S4.T2.33.33.5.m1.1.1.cmml" xref="S4.T2.33.33.5.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.33.33.5.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.33.33.5.2" class="ltx_text ltx_font_typewriter"> 10.2</span>
</td>
<td id="S4.T2.34.34.6" class="ltx_td ltx_align_right">
<span id="S4.T2.34.34.6.1" class="ltx_text ltx_font_typewriter">-29.7 </span><math id="S4.T2.34.34.6.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.34.34.6.m1.1a"><mo id="S4.T2.34.34.6.m1.1.1" xref="S4.T2.34.34.6.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.34.34.6.m1.1b"><csymbol cd="latexml" id="S4.T2.34.34.6.m1.1.1.cmml" xref="S4.T2.34.34.6.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.34.34.6.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.34.34.6.2" class="ltx_text ltx_font_typewriter"> 10.1</span>
</td>
<td id="S4.T2.35.35.7" class="ltx_td ltx_align_right"><span id="S4.T2.35.35.7.1" class="ltx_text ltx_font_bold">-33.3 <math id="S4.T2.35.35.7.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.35.35.7.1.m1.1a"><mo id="S4.T2.35.35.7.1.m1.1.1" xref="S4.T2.35.35.7.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.35.35.7.1.m1.1b"><csymbol cd="latexml" id="S4.T2.35.35.7.1.m1.1.1.cmml" xref="S4.T2.35.35.7.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.35.35.7.1.m1.1c">\pm</annotation></semantics></math> 10.3</span></td>
</tr>
<tr id="S4.T2.42.42" class="ltx_tr">
<td id="S4.T2.36.36.1" class="ltx_td ltx_align_right">
<span id="S4.T2.36.36.1.1" class="ltx_text ltx_font_typewriter">I/IS </span><math id="S4.T2.36.36.1.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S4.T2.36.36.1.m1.1a"><mo stretchy="false" id="S4.T2.36.36.1.m1.1.1" xref="S4.T2.36.36.1.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S4.T2.36.36.1.m1.1b"><ci id="S4.T2.36.36.1.m1.1.1.cmml" xref="S4.T2.36.36.1.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.36.36.1.m1.1c">\uparrow</annotation></semantics></math>
</td>
<td id="S4.T2.37.37.2" class="ltx_td ltx_align_right">
<span id="S4.T2.37.37.2.1" class="ltx_text ltx_font_typewriter">-3.8 </span><math id="S4.T2.37.37.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.37.37.2.m1.1a"><mo id="S4.T2.37.37.2.m1.1.1" xref="S4.T2.37.37.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.37.37.2.m1.1b"><csymbol cd="latexml" id="S4.T2.37.37.2.m1.1.1.cmml" xref="S4.T2.37.37.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.37.37.2.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.37.37.2.2" class="ltx_text ltx_font_typewriter"> 3.7</span>
</td>
<td id="S4.T2.38.38.3" class="ltx_td ltx_align_right">
<span id="S4.T2.38.38.3.1" class="ltx_text ltx_font_typewriter">20.7 </span><math id="S4.T2.38.38.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.38.38.3.m1.1a"><mo id="S4.T2.38.38.3.m1.1.1" xref="S4.T2.38.38.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.38.38.3.m1.1b"><csymbol cd="latexml" id="S4.T2.38.38.3.m1.1.1.cmml" xref="S4.T2.38.38.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.38.38.3.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.38.38.3.2" class="ltx_text ltx_font_typewriter"> 184.7</span>
</td>
<td id="S4.T2.39.39.4" class="ltx_td ltx_align_right">
<span id="S4.T2.39.39.4.1" class="ltx_text ltx_font_typewriter">-1.1 </span><math id="S4.T2.39.39.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.39.39.4.m1.1a"><mo id="S4.T2.39.39.4.m1.1.1" xref="S4.T2.39.39.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.39.39.4.m1.1b"><csymbol cd="latexml" id="S4.T2.39.39.4.m1.1.1.cmml" xref="S4.T2.39.39.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.39.39.4.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.39.39.4.2" class="ltx_text ltx_font_typewriter"> 6.8</span>
</td>
<td id="S4.T2.40.40.5" class="ltx_td ltx_align_right">
<span id="S4.T2.40.40.5.1" class="ltx_text ltx_font_typewriter">15.0 </span><math id="S4.T2.40.40.5.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.40.40.5.m1.1a"><mo id="S4.T2.40.40.5.m1.1.1" xref="S4.T2.40.40.5.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.40.40.5.m1.1b"><csymbol cd="latexml" id="S4.T2.40.40.5.m1.1.1.cmml" xref="S4.T2.40.40.5.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.40.40.5.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.40.40.5.2" class="ltx_text ltx_font_typewriter"> 880.0</span>
</td>
<td id="S4.T2.41.41.6" class="ltx_td ltx_align_right"><span id="S4.T2.41.41.6.1" class="ltx_text ltx_font_bold">25.0 <math id="S4.T2.41.41.6.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.41.41.6.1.m1.1a"><mo id="S4.T2.41.41.6.1.m1.1.1" xref="S4.T2.41.41.6.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.41.41.6.1.m1.1b"><csymbol cd="latexml" id="S4.T2.41.41.6.1.m1.1.1.cmml" xref="S4.T2.41.41.6.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.41.41.6.1.m1.1c">\pm</annotation></semantics></math> 19.9</span></td>
<td id="S4.T2.42.42.7" class="ltx_td ltx_align_right">
<span id="S4.T2.42.42.7.1" class="ltx_text ltx_font_typewriter">16.5 </span><math id="S4.T2.42.42.7.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.42.42.7.m1.1a"><mo id="S4.T2.42.42.7.m1.1.1" xref="S4.T2.42.42.7.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.42.42.7.m1.1b"><csymbol cd="latexml" id="S4.T2.42.42.7.m1.1.1.cmml" xref="S4.T2.42.42.7.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.42.42.7.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.42.42.7.2" class="ltx_text ltx_font_typewriter"> 15.4</span>
</td>
</tr>
<tr id="S4.T2.49.49" class="ltx_tr">
<th id="S4.T2.49.49.8" class="ltx_td ltx_align_justify ltx_align_top ltx_th ltx_th_row ltx_border_bb ltx_border_t" rowspan="3">
<span id="S4.T2.49.49.8.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T2.49.49.8.1.1" class="ltx_p" style="width:31.3pt;"><span id="S4.T2.49.49.8.1.1.1" class="ltx_text ltx_font_typewriter">
<span id="S4.T2.49.49.8.1.1.1.1" class="ltx_inline-block ltx_parbox ltx_align_middle" style="width:31.3pt;">
<span id="S4.T2.49.49.8.1.1.1.1.1" class="ltx_p">LLaMA3-8B</span>
</span></span></span>
</span>
</th>
<td id="S4.T2.43.43.1" class="ltx_td ltx_align_right ltx_border_t">
<span id="S4.T2.43.43.1.1" class="ltx_text ltx_font_typewriter">Mod </span><math id="S4.T2.43.43.1.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S4.T2.43.43.1.m1.1a"><mo stretchy="false" id="S4.T2.43.43.1.m1.1.1" xref="S4.T2.43.43.1.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S4.T2.43.43.1.m1.1b"><ci id="S4.T2.43.43.1.m1.1.1.cmml" xref="S4.T2.43.43.1.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.43.43.1.m1.1c">\uparrow</annotation></semantics></math>
</td>
<td id="S4.T2.44.44.2" class="ltx_td ltx_align_right ltx_border_t">
<span id="S4.T2.44.44.2.1" class="ltx_text ltx_font_typewriter">-1.6 </span><math id="S4.T2.44.44.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.44.44.2.m1.1a"><mo id="S4.T2.44.44.2.m1.1.1" xref="S4.T2.44.44.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.44.44.2.m1.1b"><csymbol cd="latexml" id="S4.T2.44.44.2.m1.1.1.cmml" xref="S4.T2.44.44.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.44.44.2.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.44.44.2.2" class="ltx_text ltx_font_typewriter"> 3.6</span>
</td>
<td id="S4.T2.45.45.3" class="ltx_td ltx_align_right ltx_border_t">
<span id="S4.T2.45.45.3.1" class="ltx_text ltx_font_typewriter">18.9 </span><math id="S4.T2.45.45.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.45.45.3.m1.1a"><mo id="S4.T2.45.45.3.m1.1.1" xref="S4.T2.45.45.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.45.45.3.m1.1b"><csymbol cd="latexml" id="S4.T2.45.45.3.m1.1.1.cmml" xref="S4.T2.45.45.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.45.45.3.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.45.45.3.2" class="ltx_text ltx_font_typewriter"> 25.6</span>
</td>
<td id="S4.T2.46.46.4" class="ltx_td ltx_align_right ltx_border_t">
<span id="S4.T2.46.46.4.1" class="ltx_text ltx_font_typewriter">0.9 </span><math id="S4.T2.46.46.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.46.46.4.m1.1a"><mo id="S4.T2.46.46.4.m1.1.1" xref="S4.T2.46.46.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.46.46.4.m1.1b"><csymbol cd="latexml" id="S4.T2.46.46.4.m1.1.1.cmml" xref="S4.T2.46.46.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.46.46.4.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.46.46.4.2" class="ltx_text ltx_font_typewriter"> 11.8</span>
</td>
<td id="S4.T2.47.47.5" class="ltx_td ltx_align_right ltx_border_t">
<span id="S4.T2.47.47.5.1" class="ltx_text ltx_font_typewriter">13.1 </span><math id="S4.T2.47.47.5.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.47.47.5.m1.1a"><mo id="S4.T2.47.47.5.m1.1.1" xref="S4.T2.47.47.5.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.47.47.5.m1.1b"><csymbol cd="latexml" id="S4.T2.47.47.5.m1.1.1.cmml" xref="S4.T2.47.47.5.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.47.47.5.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.47.47.5.2" class="ltx_text ltx_font_typewriter"> 21.5</span>
</td>
<td id="S4.T2.48.48.6" class="ltx_td ltx_align_right ltx_border_t"><span id="S4.T2.48.48.6.1" class="ltx_text ltx_font_bold">27.0 <math id="S4.T2.48.48.6.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.48.48.6.1.m1.1a"><mo id="S4.T2.48.48.6.1.m1.1.1" xref="S4.T2.48.48.6.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.48.48.6.1.m1.1b"><csymbol cd="latexml" id="S4.T2.48.48.6.1.m1.1.1.cmml" xref="S4.T2.48.48.6.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.48.48.6.1.m1.1c">\pm</annotation></semantics></math> 35.6</span></td>
<td id="S4.T2.49.49.7" class="ltx_td ltx_align_right ltx_border_t">
<span id="S4.T2.49.49.7.1" class="ltx_text ltx_font_typewriter">18.3 </span><math id="S4.T2.49.49.7.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.49.49.7.m1.1a"><mo id="S4.T2.49.49.7.m1.1.1" xref="S4.T2.49.49.7.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.49.49.7.m1.1b"><csymbol cd="latexml" id="S4.T2.49.49.7.m1.1.1.cmml" xref="S4.T2.49.49.7.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.49.49.7.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.49.49.7.2" class="ltx_text ltx_font_typewriter"> 28.5</span>
</td>
</tr>
<tr id="S4.T2.56.56" class="ltx_tr">
<td id="S4.T2.50.50.1" class="ltx_td ltx_align_right">
<span id="S4.T2.50.50.1.1" class="ltx_text ltx_font_typewriter">Con </span><math id="S4.T2.50.50.1.m1.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="S4.T2.50.50.1.m1.1a"><mo stretchy="false" id="S4.T2.50.50.1.m1.1.1" xref="S4.T2.50.50.1.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="S4.T2.50.50.1.m1.1b"><ci id="S4.T2.50.50.1.m1.1.1.cmml" xref="S4.T2.50.50.1.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.50.50.1.m1.1c">\downarrow</annotation></semantics></math>
</td>
<td id="S4.T2.51.51.2" class="ltx_td ltx_align_right">
<span id="S4.T2.51.51.2.1" class="ltx_text ltx_font_typewriter">11.3 </span><math id="S4.T2.51.51.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.51.51.2.m1.1a"><mo id="S4.T2.51.51.2.m1.1.1" xref="S4.T2.51.51.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.51.51.2.m1.1b"><csymbol cd="latexml" id="S4.T2.51.51.2.m1.1.1.cmml" xref="S4.T2.51.51.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.51.51.2.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.51.51.2.2" class="ltx_text ltx_font_typewriter"> 9.5</span>
</td>
<td id="S4.T2.52.52.3" class="ltx_td ltx_align_right">
<span id="S4.T2.52.52.3.1" class="ltx_text ltx_font_typewriter">-20.3 </span><math id="S4.T2.52.52.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.52.52.3.m1.1a"><mo id="S4.T2.52.52.3.m1.1.1" xref="S4.T2.52.52.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.52.52.3.m1.1b"><csymbol cd="latexml" id="S4.T2.52.52.3.m1.1.1.cmml" xref="S4.T2.52.52.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.52.52.3.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.52.52.3.2" class="ltx_text ltx_font_typewriter"> 6.9</span>
</td>
<td id="S4.T2.53.53.4" class="ltx_td ltx_align_right">
<span id="S4.T2.53.53.4.1" class="ltx_text ltx_font_typewriter">-14.6 </span><math id="S4.T2.53.53.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.53.53.4.m1.1a"><mo id="S4.T2.53.53.4.m1.1.1" xref="S4.T2.53.53.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.53.53.4.m1.1b"><csymbol cd="latexml" id="S4.T2.53.53.4.m1.1.1.cmml" xref="S4.T2.53.53.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.53.53.4.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.53.53.4.2" class="ltx_text ltx_font_typewriter"> 11.4</span>
</td>
<td id="S4.T2.54.54.5" class="ltx_td ltx_align_right">
<span id="S4.T2.54.54.5.1" class="ltx_text ltx_font_typewriter">-29.7 </span><math id="S4.T2.54.54.5.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.54.54.5.m1.1a"><mo id="S4.T2.54.54.5.m1.1.1" xref="S4.T2.54.54.5.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.54.54.5.m1.1b"><csymbol cd="latexml" id="S4.T2.54.54.5.m1.1.1.cmml" xref="S4.T2.54.54.5.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.54.54.5.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.54.54.5.2" class="ltx_text ltx_font_typewriter"> 9.2</span>
</td>
<td id="S4.T2.55.55.6" class="ltx_td ltx_align_right">
<span id="S4.T2.55.55.6.1" class="ltx_text ltx_font_typewriter">-30.6 </span><math id="S4.T2.55.55.6.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.55.55.6.m1.1a"><mo id="S4.T2.55.55.6.m1.1.1" xref="S4.T2.55.55.6.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.55.55.6.m1.1b"><csymbol cd="latexml" id="S4.T2.55.55.6.m1.1.1.cmml" xref="S4.T2.55.55.6.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.55.55.6.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.55.55.6.2" class="ltx_text ltx_font_typewriter"> 9.2</span>
</td>
<td id="S4.T2.56.56.7" class="ltx_td ltx_align_right"><span id="S4.T2.56.56.7.1" class="ltx_text ltx_font_bold">-33.9 <math id="S4.T2.56.56.7.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.56.56.7.1.m1.1a"><mo id="S4.T2.56.56.7.1.m1.1.1" xref="S4.T2.56.56.7.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.56.56.7.1.m1.1b"><csymbol cd="latexml" id="S4.T2.56.56.7.1.m1.1.1.cmml" xref="S4.T2.56.56.7.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.56.56.7.1.m1.1c">\pm</annotation></semantics></math> 9.6</span></td>
</tr>
<tr id="S4.T2.63.63" class="ltx_tr">
<td id="S4.T2.57.57.1" class="ltx_td ltx_align_right ltx_border_bb">
<span id="S4.T2.57.57.1.1" class="ltx_text ltx_font_typewriter">I/IS </span><math id="S4.T2.57.57.1.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S4.T2.57.57.1.m1.1a"><mo stretchy="false" id="S4.T2.57.57.1.m1.1.1" xref="S4.T2.57.57.1.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S4.T2.57.57.1.m1.1b"><ci id="S4.T2.57.57.1.m1.1.1.cmml" xref="S4.T2.57.57.1.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.57.57.1.m1.1c">\uparrow</annotation></semantics></math>
</td>
<td id="S4.T2.58.58.2" class="ltx_td ltx_align_right ltx_border_bb">
<span id="S4.T2.58.58.2.1" class="ltx_text ltx_font_typewriter">-3.8 </span><math id="S4.T2.58.58.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.58.58.2.m1.1a"><mo id="S4.T2.58.58.2.m1.1.1" xref="S4.T2.58.58.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.58.58.2.m1.1b"><csymbol cd="latexml" id="S4.T2.58.58.2.m1.1.1.cmml" xref="S4.T2.58.58.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.58.58.2.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.58.58.2.2" class="ltx_text ltx_font_typewriter"> 3.1</span>
</td>
<td id="S4.T2.59.59.3" class="ltx_td ltx_align_right ltx_border_bb">
<span id="S4.T2.59.59.3.1" class="ltx_text ltx_font_typewriter">24.5 </span><math id="S4.T2.59.59.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.59.59.3.m1.1a"><mo id="S4.T2.59.59.3.m1.1.1" xref="S4.T2.59.59.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.59.59.3.m1.1b"><csymbol cd="latexml" id="S4.T2.59.59.3.m1.1.1.cmml" xref="S4.T2.59.59.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.59.59.3.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.59.59.3.2" class="ltx_text ltx_font_typewriter"> 13.9</span>
</td>
<td id="S4.T2.60.60.4" class="ltx_td ltx_align_right ltx_border_bb">
<span id="S4.T2.60.60.4.1" class="ltx_text ltx_font_typewriter">-1.1 </span><math id="S4.T2.60.60.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.60.60.4.m1.1a"><mo id="S4.T2.60.60.4.m1.1.1" xref="S4.T2.60.60.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.60.60.4.m1.1b"><csymbol cd="latexml" id="S4.T2.60.60.4.m1.1.1.cmml" xref="S4.T2.60.60.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.60.60.4.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.60.60.4.2" class="ltx_text ltx_font_typewriter"> 5.8</span>
</td>
<td id="S4.T2.61.61.5" class="ltx_td ltx_align_right ltx_border_bb">
<span id="S4.T2.61.61.5.1" class="ltx_text ltx_font_typewriter">15.7 </span><math id="S4.T2.61.61.5.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.61.61.5.m1.1a"><mo id="S4.T2.61.61.5.m1.1.1" xref="S4.T2.61.61.5.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.61.61.5.m1.1b"><csymbol cd="latexml" id="S4.T2.61.61.5.m1.1.1.cmml" xref="S4.T2.61.61.5.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.61.61.5.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.61.61.5.2" class="ltx_text ltx_font_typewriter"> 11.0</span>
</td>
<td id="S4.T2.62.62.6" class="ltx_td ltx_align_right ltx_border_bb"><span id="S4.T2.62.62.6.1" class="ltx_text ltx_font_bold">28.1 <math id="S4.T2.62.62.6.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.62.62.6.1.m1.1a"><mo id="S4.T2.62.62.6.1.m1.1.1" xref="S4.T2.62.62.6.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.62.62.6.1.m1.1b"><csymbol cd="latexml" id="S4.T2.62.62.6.1.m1.1.1.cmml" xref="S4.T2.62.62.6.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.62.62.6.1.m1.1c">\pm</annotation></semantics></math> 16.1</span></td>
<td id="S4.T2.63.63.7" class="ltx_td ltx_align_right ltx_border_bb">
<span id="S4.T2.63.63.7.1" class="ltx_text ltx_font_typewriter">16.4 </span><math id="S4.T2.63.63.7.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.63.63.7.m1.1a"><mo id="S4.T2.63.63.7.m1.1.1" xref="S4.T2.63.63.7.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.63.63.7.m1.1b"><csymbol cd="latexml" id="S4.T2.63.63.7.m1.1.1.cmml" xref="S4.T2.63.63.7.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.63.63.7.m1.1c">\pm</annotation></semantics></math><span id="S4.T2.63.63.7.2" class="ltx_text ltx_font_typewriter"> 12.2</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_font_typewriter"><span class="ltx_tag ltx_tag_table"><span id="S4.T2.74.1.1" class="ltx_text ltx_font_serif">Table 2</span>: </span>Comparison with graph-theoretic metrics in the KV cache of different LLMs and segmentation methods using the PG-19 dataset. Reported values are the difference with the corresponding random segmentation. Mod: modularity<math id="S4.T2.67.m1.1" class="ltx_Math" alttext="\times 10^{5}" display="inline"><semantics id="S4.T2.67.m1.1b"><mrow id="S4.T2.67.m1.1.1" xref="S4.T2.67.m1.1.1.cmml"><mi id="S4.T2.67.m1.1.1.2" xref="S4.T2.67.m1.1.1.2.cmml"></mi><mo lspace="0.222em" rspace="0.222em" id="S4.T2.67.m1.1.1.1" xref="S4.T2.67.m1.1.1.1.cmml">×</mo><msup id="S4.T2.67.m1.1.1.3" xref="S4.T2.67.m1.1.1.3.cmml"><mn id="S4.T2.67.m1.1.1.3.2" xref="S4.T2.67.m1.1.1.3.2.cmml">10</mn><mn id="S4.T2.67.m1.1.1.3.3" xref="S4.T2.67.m1.1.1.3.3.cmml">5</mn></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.T2.67.m1.1c"><apply id="S4.T2.67.m1.1.1.cmml" xref="S4.T2.67.m1.1.1"><times id="S4.T2.67.m1.1.1.1.cmml" xref="S4.T2.67.m1.1.1.1"></times><csymbol cd="latexml" id="S4.T2.67.m1.1.1.2.cmml" xref="S4.T2.67.m1.1.1.2">absent</csymbol><apply id="S4.T2.67.m1.1.1.3.cmml" xref="S4.T2.67.m1.1.1.3"><csymbol cd="ambiguous" id="S4.T2.67.m1.1.1.3.1.cmml" xref="S4.T2.67.m1.1.1.3">superscript</csymbol><cn type="integer" id="S4.T2.67.m1.1.1.3.2.cmml" xref="S4.T2.67.m1.1.1.3.2">10</cn><cn type="integer" id="S4.T2.67.m1.1.1.3.3.cmml" xref="S4.T2.67.m1.1.1.3.3">5</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.67.m1.1d">\times 10^{5}</annotation></semantics></math>, Con: Conductance, I/IS: intra/inter-similarity <math id="S4.T2.68.m2.1" class="ltx_Math" alttext="\times 10^{3}" display="inline"><semantics id="S4.T2.68.m2.1b"><mrow id="S4.T2.68.m2.1.1" xref="S4.T2.68.m2.1.1.cmml"><mi id="S4.T2.68.m2.1.1.2" xref="S4.T2.68.m2.1.1.2.cmml"></mi><mo lspace="0.222em" rspace="0.222em" id="S4.T2.68.m2.1.1.1" xref="S4.T2.68.m2.1.1.1.cmml">×</mo><msup id="S4.T2.68.m2.1.1.3" xref="S4.T2.68.m2.1.1.3.cmml"><mn id="S4.T2.68.m2.1.1.3.2" xref="S4.T2.68.m2.1.1.3.2.cmml">10</mn><mn id="S4.T2.68.m2.1.1.3.3" xref="S4.T2.68.m2.1.1.3.3.cmml">3</mn></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.T2.68.m2.1c"><apply id="S4.T2.68.m2.1.1.cmml" xref="S4.T2.68.m2.1.1"><times id="S4.T2.68.m2.1.1.1.cmml" xref="S4.T2.68.m2.1.1.1"></times><csymbol cd="latexml" id="S4.T2.68.m2.1.1.2.cmml" xref="S4.T2.68.m2.1.1.2">absent</csymbol><apply id="S4.T2.68.m2.1.1.3.cmml" xref="S4.T2.68.m2.1.1.3"><csymbol cd="ambiguous" id="S4.T2.68.m2.1.1.3.1.cmml" xref="S4.T2.68.m2.1.1.3">superscript</csymbol><cn type="integer" id="S4.T2.68.m2.1.1.3.2.cmml" xref="S4.T2.68.m2.1.1.3.2">10</cn><cn type="integer" id="S4.T2.68.m2.1.1.3.3.cmml" xref="S4.T2.68.m2.1.1.3.3">3</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.68.m2.1d">\times 10^{3}</annotation></semantics></math>. Hyper-parameters: Surprise threshold = <math id="S4.T2.69.m3.1" class="ltx_Math" alttext="0.001" display="inline"><semantics id="S4.T2.69.m3.1b"><mn id="S4.T2.69.m3.1.1" xref="S4.T2.69.m3.1.1.cmml">0.001</mn><annotation-xml encoding="MathML-Content" id="S4.T2.69.m3.1c"><cn type="float" id="S4.T2.69.m3.1.1.cmml" xref="S4.T2.69.m3.1.1">0.001</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.69.m3.1d">0.001</annotation></semantics></math></figcaption>
</figure>
<div id="S4.SS3.p1" class="ltx_para ltx_noindent">
<p id="S4.SS3.p1.5" class="ltx_p"><span id="S4.SS3.p1.5.1" class="ltx_text ltx_font_typewriter">Looking at Table </span><a href="#S4.T2" title="Table 2 ‣ 4.3 Comparing segmentation methods ‣ 4 Experiments ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">2</span></a><span id="S4.SS3.p1.5.2" class="ltx_text ltx_font_typewriter">, it is clear that surprise-based segmentation with refinement (</span><math id="S4.SS3.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{SM}" display="inline"><semantics id="S4.SS3.p1.1.m1.1a"><mi id="S4.SS3.p1.1.m1.1.1" xref="S4.SS3.p1.1.m1.1.1.cmml">𝐒𝐌</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.1.m1.1b"><ci id="S4.SS3.p1.1.m1.1.1.cmml" xref="S4.SS3.p1.1.m1.1.1">𝐒𝐌</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.1.m1.1c">\mathbf{SM}</annotation></semantics></math><span id="S4.SS3.p1.5.3" class="ltx_text ltx_font_typewriter">, </span><math id="S4.SS3.p1.2.m2.1" class="ltx_Math" alttext="\mathbf{SC}" display="inline"><semantics id="S4.SS3.p1.2.m2.1a"><mi id="S4.SS3.p1.2.m2.1.1" xref="S4.SS3.p1.2.m2.1.1.cmml">𝐒𝐂</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.2.m2.1b"><ci id="S4.SS3.p1.2.m2.1.1.cmml" xref="S4.SS3.p1.2.m2.1.1">𝐒𝐂</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.2.m2.1c">\mathbf{SC}</annotation></semantics></math><span id="S4.SS3.p1.5.4" class="ltx_text ltx_font_typewriter">) provides the best results in terms of event similarity metrics, regardless of the base LLM used. While the surprise-only method (</span><math id="S4.SS3.p1.3.m3.1" class="ltx_Math" alttext="\mathbf{S}" display="inline"><semantics id="S4.SS3.p1.3.m3.1a"><mi id="S4.SS3.p1.3.m3.1.1" xref="S4.SS3.p1.3.m3.1.1.cmml">𝐒</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.3.m3.1b"><ci id="S4.SS3.p1.3.m3.1.1.cmml" xref="S4.SS3.p1.3.m3.1.1">𝐒</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.3.m3.1c">\mathbf{S}</annotation></semantics></math><span id="S4.SS3.p1.5.5" class="ltx_text ltx_font_typewriter">) achieves some good results, we observe that refinement is especially adept to improving this performance with regards to our metrics, as it is directly optimising for such an objective. Interestingly however, the fixed-based refinement methods (</span><math id="S4.SS3.p1.4.m4.1" class="ltx_Math" alttext="\mathbf{FM}" display="inline"><semantics id="S4.SS3.p1.4.m4.1a"><mi id="S4.SS3.p1.4.m4.1.1" xref="S4.SS3.p1.4.m4.1.1.cmml">𝐅𝐌</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.4.m4.1b"><ci id="S4.SS3.p1.4.m4.1.1.cmml" xref="S4.SS3.p1.4.m4.1.1">𝐅𝐌</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.4.m4.1c">\mathbf{FM}</annotation></semantics></math><span id="S4.SS3.p1.5.6" class="ltx_text ltx_font_typewriter">, </span><math id="S4.SS3.p1.5.m5.1" class="ltx_Math" alttext="\mathbf{FC}" display="inline"><semantics id="S4.SS3.p1.5.m5.1a"><mi id="S4.SS3.p1.5.m5.1.1" xref="S4.SS3.p1.5.m5.1.1.cmml">𝐅𝐂</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.5.m5.1b"><ci id="S4.SS3.p1.5.m5.1.1.cmml" xref="S4.SS3.p1.5.m5.1.1">𝐅𝐂</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.5.m5.1c">\mathbf{FC}</annotation></semantics></math><span id="S4.SS3.p1.5.7" class="ltx_text ltx_font_typewriter">) do not reach the same performance as their surprise-based counterparts, further showing that the initial segmentation with a surprise threshold is crucial to achieving the best possible balance in intra-/inter-similarity with our methods.</span></p>
</div>
</section>
<section id="S4.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_font_typewriter ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="S4.SS4.1.1.1" class="ltx_text ltx_font_serif">4.4</span> </span>Similarity, Contiguity, Recency and Temporal Order</h3>

<div id="S4.SS4.p1" class="ltx_para ltx_noindent">
<p id="S4.SS4.p1.1" class="ltx_p"><span id="S4.SS4.p1.1.1" class="ltx_text ltx_font_typewriter">As demonstrated in Tables </span><a href="#S4.T1" title="Table 1 ‣ 4.1 Performance of EM-LLM on long-context tasks ‣ 4 Experiments ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">1</span></a><span id="S4.SS4.p1.1.2" class="ltx_text ltx_font_typewriter"> and </span><a href="#S4.T2" title="Table 2 ‣ 4.3 Comparing segmentation methods ‣ 4 Experiments ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">2</span></a><span id="S4.SS4.p1.1.3" class="ltx_text ltx_font_typewriter">, along with Fig. </span><a href="#S4.F3" title="Figure 3 ‣ 4.2 Humans and LLM surprise cluster similar tokens together ‣ 4 Experiments ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">3</span></a><span id="S4.SS4.p1.1.4" class="ltx_text ltx_font_typewriter">, each of our ablations show various positive improvements on InfLLM (also see Appendix </span><a href="#A1.SS1" title="A.1 Supplementary figures ‣ Appendix A Appendix / supplemental material ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">A.1</span></a><span id="S4.SS4.p1.1.5" class="ltx_text ltx_font_typewriter">). As mentioned in Section </span><a href="#S4.SS3" title="4.3 Comparing segmentation methods ‣ 4 Experiments ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">4.3</span></a><span id="S4.SS4.p1.1.6" class="ltx_text ltx_font_typewriter">, refinement has a strong positive impact in improving our similarity metrics. This is seen to translate well to model performance in Table </span><a href="#S4.T1" title="Table 1 ‣ 4.1 Performance of EM-LLM on long-context tasks ‣ 4 Experiments ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">1</span></a><span id="S4.SS4.p1.1.7" class="ltx_text ltx_font_typewriter">, achieving the best performance in a third of the tasks, as well as agreeing with human data (Fig. </span><a href="#S4.F3" title="Figure 3 ‣ 4.2 Humans and LLM surprise cluster similar tokens together ‣ 4 Experiments ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">3</span></a><span id="S4.SS4.p1.1.8" class="ltx_text ltx_font_typewriter">). The effects of contiguity are also clearly demonstrated in this table, with the addition of our contiguity buffer achieving the best performance on three tasks, and the second-best overall score. Furthermore, these methods are shown to be generally complementary, achieving the best overall performance when combined.</span></p>
</div>
<div id="S4.SS4.p2" class="ltx_para ltx_noindent">
<p id="S4.SS4.p2.1" class="ltx_p"><span id="S4.SS4.p2.1.1" class="ltx_text ltx_font_typewriter">However, the fact that certain tasks still appear to benefit more from either surprise-only, refinement, or contiguity, is an interesting result. This is likely due to the nature of the tasks and the varying importance of contiguity across these tasks. For instance, in Supplementary Fig. </span><a href="#A1.F5" title="Figure 5 ‣ A.1 Supplementary figures ‣ Appendix A Appendix / supplemental material ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">5</span></a><span id="S4.SS4.p2.1.2" class="ltx_text ltx_font_typewriter">, the MultiNews task scores higher than our baseline only for a ratio of </span><math id="S4.SS4.p2.1.m1.1" class="ltx_Math" alttext="70\%" display="inline"><semantics id="S4.SS4.p2.1.m1.1a"><mrow id="S4.SS4.p2.1.m1.1.1" xref="S4.SS4.p2.1.m1.1.1.cmml"><mn id="S4.SS4.p2.1.m1.1.1.2" xref="S4.SS4.p2.1.m1.1.1.2.cmml">70</mn><mo id="S4.SS4.p2.1.m1.1.1.1" xref="S4.SS4.p2.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS4.p2.1.m1.1b"><apply id="S4.SS4.p2.1.m1.1.1.cmml" xref="S4.SS4.p2.1.m1.1.1"><csymbol cd="latexml" id="S4.SS4.p2.1.m1.1.1.1.cmml" xref="S4.SS4.p2.1.m1.1.1.1">percent</csymbol><cn type="integer" id="S4.SS4.p2.1.m1.1.1.2.cmml" xref="S4.SS4.p2.1.m1.1.1.2">70</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p2.1.m1.1c">70\%</annotation></semantics></math><span id="S4.SS4.p2.1.3" class="ltx_text ltx_font_typewriter"> contiguity to similarity buffers. Where contiguity is not crucial, adding such a buffer to our context window also reduces the size of the similarity buffer, and therefore provides potentially less directly relevant events. This is compatible with our own findings that a contiguity buffer that is as big or smaller than the similarity buffer yields the best results, suggesting that the similarity buffer, is still the most crucial part of our approach. This is especially the case when combined with refinement, which we expect is due to the improved similarity of refined events, hence further reducing the need for contiguous events.</span></p>
</div>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_font_typewriter ltx_title_section">
<span class="ltx_tag ltx_tag_section"><span id="S5.1.1.1" class="ltx_text ltx_font_serif">5</span> </span>Discussion</h2>

<section id="S5.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_font_typewriter ltx_title_paragraph">Human studies</h4>

<div id="S5.SS0.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S5.SS0.SSS0.Px1.p1.1" class="ltx_p"><span id="S5.SS0.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_typewriter">The surprise-based segmentation and boundary refinement processes in EM-LLM mirror key aspects of human event perception and memory formation. Our approach aligns with theories proposing that humans segment continuous experiences into discrete events based on prediction errors or moments of surprise </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S5.SS0.SSS0.Px1.p1.1.2.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Zacks et al.</span><span id="S5.SS0.SSS0.Px1.p1.1.3.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib13" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2007</span></a>, <span class="ltx_text ltx_font_typewriter">Fountas et al.</span><span id="S5.SS0.SSS0.Px1.p1.1.3.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib17" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2022</span></a><span id="S5.SS0.SSS0.Px1.p1.1.4.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S5.SS0.SSS0.Px1.p1.1.5" class="ltx_text ltx_font_typewriter">. This segmentation process is crucial for organising and later retrieving episodic memories efficiently. Indeed, significant correlations have been found between human event segmentation and prediction errors in both LLMs </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S5.SS0.SSS0.Px1.p1.1.6.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Kumar et al.</span><span id="S5.SS0.SSS0.Px1.p1.1.7.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib20" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2023</span></a><span id="S5.SS0.SSS0.Px1.p1.1.8.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S5.SS0.SSS0.Px1.p1.1.9" class="ltx_text ltx_font_typewriter"> and video models </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S5.SS0.SSS0.Px1.p1.1.10.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Fountas et al.</span><span id="S5.SS0.SSS0.Px1.p1.1.11.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib17" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2022</span></a>, <span class="ltx_text ltx_font_typewriter">Mariola et al.</span><span id="S5.SS0.SSS0.Px1.p1.1.11.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib75" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2022</span></a><span id="S5.SS0.SSS0.Px1.p1.1.12.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S5.SS0.SSS0.Px1.p1.1.13" class="ltx_text ltx_font_typewriter">. Our results add to this growing body of evidence, demonstrating that LLM-based surprise can serve as a proxy for human event segmentation, in multiple levels of hierarchical abstraction, and that the resulting event structure in EM-LLM’s attention heads correlates strongly with human-perceived events. This finding creates a more direct, low-level connection between LLM mechanisms and human cognitive processes. Furthermore, our model’s use of both similarity-based and temporally contiguous retrieval mechanisms parallels human memory retrieval patterns, allowing for the expression of robust phenomena found in human memory research </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S5.SS0.SSS0.Px1.p1.1.14.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Howard and Kahana</span><span id="S5.SS0.SSS0.Px1.p1.1.15.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib18" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2002</span></a><span id="S5.SS0.SSS0.Px1.p1.1.16.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S5.SS0.SSS0.Px1.p1.1.17" class="ltx_text ltx_font_typewriter">.</span></p>
</div>
<div id="S5.SS0.SSS0.Px1.p2" class="ltx_para ltx_noindent">
<p id="S5.SS0.SSS0.Px1.p2.1" class="ltx_p"><span id="S5.SS0.SSS0.Px1.p2.1.1" class="ltx_text ltx_font_typewriter">Furthermore, our model’s use of both similarity-based and temporally contiguous retrieval mechanisms parallels human memory retrieval patterns. The temporal contiguity effect, where items experienced close together in time are often recalled together, is a robust phenomenon in human memory research </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S5.SS0.SSS0.Px1.p2.1.2.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Howard and Kahana</span><span id="S5.SS0.SSS0.Px1.p2.1.3.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib18" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2002</span></a><span id="S5.SS0.SSS0.Px1.p2.1.4.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S5.SS0.SSS0.Px1.p2.1.5" class="ltx_text ltx_font_typewriter">.
Further experiments could deepen our understanding of the connections between EM-LLM and human episodic memory. Following </span><cite class="ltx_cite ltx_citemacro_cite"><span class="ltx_text ltx_font_typewriter">Michelmann et al.</span> <span id="S5.SS0.SSS0.Px1.p2.1.6.1.1.1" class="ltx_text ltx_font_typewriter">[</span><a href="#bib.bib76" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2023b</span></a><span id="S5.SS0.SSS0.Px1.p2.1.7.2.2.1" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S5.SS0.SSS0.Px1.p2.1.8" class="ltx_text ltx_font_typewriter">, one potential direction is to test whether the timing of the event boundaries or the degree of modularity per level that our method produces is closer on average to the human consensus, than individual human subjects. Second, we can explore the level at which different ratios of the contiguity buffer allow the human biases presented in Fig. </span><a href="#S2.F2" title="Figure 2 ‣ 2.2 Neural models of Episodic Memory and Event Cognition ‣ 2 Related work ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">2</span></a><span id="S5.SS0.SSS0.Px1.p2.1.9" class="ltx_text ltx_font_typewriter">A and the analysis in </span><cite class="ltx_cite ltx_citemacro_cite"><span class="ltx_text ltx_font_typewriter">Ji-An et al.</span> <span id="S5.SS0.SSS0.Px1.p2.1.10.1.1.1" class="ltx_text ltx_font_typewriter">[</span><a href="#bib.bib19" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2024</span></a><span id="S5.SS0.SSS0.Px1.p2.1.11.2.2.1" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S5.SS0.SSS0.Px1.p2.1.12" class="ltx_text ltx_font_typewriter"> to be more easily reproduced. Finally, we could investigate how skewing event recall based on recency and originally-recorded surprise affects model performance and to what extent it produces biased behaviour found in studies of free recall.</span></p>
</div>
<div id="S5.SS0.SSS0.Px1.p3" class="ltx_para ltx_noindent">
<p id="S5.SS0.SSS0.Px1.p3.1" class="ltx_p"><span id="S5.SS0.SSS0.Px1.p3.1.1" class="ltx_text ltx_font_typewriter">In addition, the architecture of EM-LLM, with its differentiated context handling described in Section </span><a href="#S3.SS1" title="3.1 Architecture ‣ 3 EM-LLM: LLM with Episodic Memory ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">3.1</span></a><span id="S5.SS0.SSS0.Px1.p3.1.2" class="ltx_text ltx_font_typewriter">, invites further interesting comparisons to cognitive models of human memory beyond episodic. The group of tokens forming the local context, which hold the most recent and task-relevant information, share characteristics with the concept of working memory. For instance, </span><cite class="ltx_cite ltx_citemacro_cite"><span class="ltx_text ltx_font_typewriter">Baddeley</span> <span id="S5.SS0.SSS0.Px1.p3.1.3.1.1.1" class="ltx_text ltx_font_typewriter">[</span><a href="#bib.bib77" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2003</span></a><span id="S5.SS0.SSS0.Px1.p3.1.4.2.2.1" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S5.SS0.SSS0.Px1.p3.1.5" class="ltx_text ltx_font_typewriter">’s influential model of working memory, which posits a limited-capacity system for temporary information storage and manipulation, bears similarities to our local context functionality. Yet, the analogy is not perfect. Our broader context window, including both local context and retrieved memories, might be more accurately compared to </span><cite class="ltx_cite ltx_citemacro_cite"><span class="ltx_text ltx_font_typewriter">Ericsson and Kintsch</span> <span id="S5.SS0.SSS0.Px1.p3.1.6.1.1.1" class="ltx_text ltx_font_typewriter">[</span><a href="#bib.bib78" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">1995</span></a><span id="S5.SS0.SSS0.Px1.p3.1.7.2.2.1" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S5.SS0.SSS0.Px1.p3.1.8" class="ltx_text ltx_font_typewriter">’s concept of long-term working memory, which proposes a mechanism for rapid access to relevant information in long-term memory, extending beyond the traditional capacity limits of working memory. Alternatively, our architecture aligns well with </span><cite class="ltx_cite ltx_citemacro_cite"><span class="ltx_text ltx_font_typewriter">Cowan</span> <span id="S5.SS0.SSS0.Px1.p3.1.9.1.1.1" class="ltx_text ltx_font_typewriter">[</span><a href="#bib.bib67" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2001</span></a><span id="S5.SS0.SSS0.Px1.p3.1.10.2.2.1" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S5.SS0.SSS0.Px1.p3.1.11" class="ltx_text ltx_font_typewriter">’s embedded-processes model, where our local context could be likened to the limited-capacity ‘‘focus of attention’’ within working memory, while the full context window parallels the activated portion of long-term memory. Future work could explore these analogies more deeply, providing a flexible test-bed for rapidly exploring hypotheses about human memory, and potentially informing debates about capacity limits in working memory.
Additionally, inspired by the multi-component nature of Baddeley’s model, one might explore the integration of modality-specific buffers within EM-LLM to enhance its performance on multi-modal tasks.</span></p>
</div>
</section>
<section id="S5.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_font_typewriter ltx_title_paragraph">Machine learning</h4>

<div id="S5.SS0.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S5.SS0.SSS0.Px2.p1.1" class="ltx_p"><span id="S5.SS0.SSS0.Px2.p1.1.1" class="ltx_text ltx_font_typewriter">In refining event boundaries, we utilized modularity and conductance as metrics for evaluating community structure in the similarity graph of attention keys. While effective in our experiments, we acknowledge that numerous other methods for graph clustering and sequence segmentation could potentially be applied </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S5.SS0.SSS0.Px2.p1.1.2.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Fortunato</span><span id="S5.SS0.SSS0.Px2.p1.1.3.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib79" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2010</span></a>, <span class="ltx_text ltx_font_typewriter">Yang et al.</span><span id="S5.SS0.SSS0.Px2.p1.1.3.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib80" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2016</span></a><span id="S5.SS0.SSS0.Px2.p1.1.4.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S5.SS0.SSS0.Px2.p1.1.5" class="ltx_text ltx_font_typewriter">. Our choice was motivated by their established theoretical foundations and computational efficiency, though comparative studies suggest performance can vary based on network characteristics </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S5.SS0.SSS0.Px2.p1.1.6.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Yang et al.</span><span id="S5.SS0.SSS0.Px2.p1.1.7.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib80" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2016</span></a><span id="S5.SS0.SSS0.Px2.p1.1.8.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S5.SS0.SSS0.Px2.p1.1.9" class="ltx_text ltx_font_typewriter">. Interestingly, our surprise-based initial boundary detection shares similarities with Bayesian online change-point detection </span><cite class="ltx_cite ltx_citemacro_citep"><span id="S5.SS0.SSS0.Px2.p1.1.10.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Adams and MacKay</span><span id="S5.SS0.SSS0.Px2.p1.1.11.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib81" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2007</span></a><span id="S5.SS0.SSS0.Px2.p1.1.12.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="S5.SS0.SSS0.Px2.p1.1.13" class="ltx_text ltx_font_typewriter">, suggesting potential avenues for integrating time series analysis techniques into LLM context processing. Future work could explore whether more sophisticated segmentation or clustering algorithms could improve EM-LLM’s performance, particularly for extremely long contexts or streaming data scenarios. Such investigations could enhance our model and contribute to understanding how information is structured and processed in LLMs, bridging the gap between traditional sequence analysis and LLM context processing.</span></p>
</div>
<div id="S5.SS0.SSS0.Px2.p2" class="ltx_para ltx_noindent">
<p id="S5.SS0.SSS0.Px2.p2.1" class="ltx_p"><span id="S5.SS0.SSS0.Px2.p2.1.1" class="ltx_text ltx_font_typewriter">Looking ahead, several more avenues for future research emerge from this work. One promising direction is to extend our surprise-based segmentation and boundary refinement processes to operate at each layer of the Transformer independently. This could lead to more nuanced and hierarchical representations of episodic memories, following the underlying semantic structure of the input more closely. Additionally, exploring how EM-LLM could be utilised to enable imagination and future thinking has great potential for advancing model-based reinforcement learning and continual learning techniques in LLMs. By leveraging its event-based structure to simulate potential future scenarios or recall past experiences in novel contexts, EM-LLM could enhance an LLM’s ability to plan, adapt, and learn continuously from new information.</span></p>
</div>
</section>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_font_typewriter ltx_title_section">
<span class="ltx_tag ltx_tag_section"><span id="S6.1.1.1" class="ltx_text ltx_font_serif">6</span> </span>Conclusion</h2>

<div id="S6.p1" class="ltx_para ltx_noindent">
<p id="S6.p1.1" class="ltx_p"><span id="S6.p1.1.1" class="ltx_text ltx_font_typewriter">In this work, we introduced EM-LLM, a novel and flexible architecture that integrates key aspects of human episodic memory and event cognition into transformer-based language models. Our approach enables LLMs to effectively process and utilise information from vastly extended contexts, far beyond their original training lengths. By combining surprise-based event segmentation with graph-theoretic boundary refinement, and a two-stage memory retrieval process, EM-LLM demonstrates superior performance on long-context tasks compared to state-of-the-art models. Crucially, our method requires no pre-training and can be readily applied to existing LLMs, offering a promising path towards virtually infinite context windows. This capability has the potential to revolutionise how we interact with LLMs, enabling continuous, personalized interactions over extended periods. Furthermore, the flexibility of our framework suggests it could serve as a viable alternative to traditional retrieval-augmented generation (RAG) techniques, especially when combined with efficient compression methods to reduce the memory requirements for the model’s KV cache.
</span></p>
</div>
<div id="S6.p2" class="ltx_para ltx_noindent">
<p id="S6.p2.1" class="ltx_p"><span id="S6.p2.1.1" class="ltx_text ltx_font_typewriter">In conclusion, EM-LLM represents a significant step forward in the development of language models with extended context-processing capabilities. By bridging insights from cognitive science with machine learning, our approach not only enhances the performance of LLMs on long-context tasks but also provides a scalable computational framework for testing hypotheses about human memory. We hope this study will inspire the community to expand research on the intersection between LLMs and human memory mechanisms.</span></p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_font_typewriter ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib1.5.5.1" class="ltx_text ltx_font_typewriter">Liu et al. [2024a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib1.7.1" class="ltx_text ltx_font_typewriter">
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib1.8.1" class="ltx_text ltx_font_typewriter">Lost in the middle: How language models use long contexts.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib1.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Transactions of the Association for Computational Linguistics</em><span id="bib.bib1.10.2" class="ltx_text ltx_font_typewriter">, 12:157--173, 2024a.
</span>
</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib2.5.5.1" class="ltx_text ltx_font_typewriter">Kazemnejad et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib2.7.1" class="ltx_text ltx_font_typewriter">
Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib2.8.1" class="ltx_text ltx_font_typewriter">The impact of positional encoding on length generalization in transformers.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib2.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Advances in Neural Information Processing Systems</em><span id="bib.bib2.10.2" class="ltx_text ltx_font_typewriter">, 36, 2024.
</span>
</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib3.5.5.1" class="ltx_text ltx_font_typewriter">Tworkowski et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib3.7.1" class="ltx_text ltx_font_typewriter">
Szymon Tworkowski, Konrad Staniszewski, Mikołaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Miłoś.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib3.8.1" class="ltx_text ltx_font_typewriter">Focused transformer: Contrastive training for context scaling.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib3.9.1" class="ltx_text ltx_font_typewriter">In </span><em id="bib.bib3.10.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">Thirty-seventh Conference on Neural Information Processing Systems</em><span id="bib.bib3.11.3" class="ltx_text ltx_font_typewriter">, 2023.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib3.12.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://openreview.net/forum?id=s1FjXzJ0jy" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=s1FjXzJ0jy</a><span id="bib.bib3.13.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib4.5.5.1" class="ltx_text ltx_font_typewriter">Lewis et al. [2020]</span></span>
<span class="ltx_bibblock"><span id="bib.bib4.7.1" class="ltx_text ltx_font_typewriter">
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib4.8.1" class="ltx_text ltx_font_typewriter">Retrieval-augmented generation for knowledge-intensive nlp tasks.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib4.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Advances in Neural Information Processing Systems</em><span id="bib.bib4.10.2" class="ltx_text ltx_font_typewriter">, 33:9459--9474, 2020.
</span>
</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib5.5.5.1" class="ltx_text ltx_font_typewriter">Gao et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib5.7.1" class="ltx_text ltx_font_typewriter">
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib5.8.1" class="ltx_text ltx_font_typewriter">Retrieval-augmented generation for large language models: A survey, 2024.
</span>
</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib6.5.5.1" class="ltx_text ltx_font_typewriter">Wu et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib6.7.1" class="ltx_text ltx_font_typewriter">
Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, and Christian Szegedy.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib6.8.1" class="ltx_text ltx_font_typewriter">Memorizing transformers.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib6.9.1" class="ltx_text ltx_font_typewriter">In </span><em id="bib.bib6.10.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">International Conference on Learning Representations</em><span id="bib.bib6.11.3" class="ltx_text ltx_font_typewriter">, 2022.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib6.12.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://openreview.net/forum?id=TrjbxzRcnf-" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=TrjbxzRcnf-</a><span id="bib.bib6.13.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib7.5.5.1" class="ltx_text ltx_font_typewriter">Bertsch et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib7.7.1" class="ltx_text ltx_font_typewriter">
Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew R. Gormley.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib7.8.1" class="ltx_text ltx_font_typewriter">Unlimiformer: Long-range transformers with unlimited length input.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib7.9.1" class="ltx_text ltx_font_typewriter">In </span><em id="bib.bib7.10.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">Thirty-seventh Conference on Neural Information Processing Systems</em><span id="bib.bib7.11.3" class="ltx_text ltx_font_typewriter">, 2023.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib7.12.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://openreview.net/forum?id=lJWUJWLCJo" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=lJWUJWLCJo</a><span id="bib.bib7.13.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib8.5.5.1" class="ltx_text ltx_font_typewriter">Xiao et al. [2024a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib8.7.1" class="ltx_text ltx_font_typewriter">
Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, Song Han, and Maosong Sun.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib8.8.1" class="ltx_text ltx_font_typewriter">Infllm: Unveiling the intrinsic capacity of llms for understanding extremely long sequences with training-free memory, 2024a.
</span>
</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib9.5.5.1" class="ltx_text ltx_font_typewriter">Clewett et al. [2019]</span></span>
<span class="ltx_bibblock"><span id="bib.bib9.7.1" class="ltx_text ltx_font_typewriter">
David Clewett, Sarah DuBrow, and Lila Davachi.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib9.8.1" class="ltx_text ltx_font_typewriter">Transcending time in the brain: How event memories are constructed from experience.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib9.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Hippocampus</em><span id="bib.bib9.10.2" class="ltx_text ltx_font_typewriter">, 29(3):162--183, 2019.
</span>
</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib10.4.4.1" class="ltx_text ltx_font_typewriter">Zacks [2020]</span></span>
<span class="ltx_bibblock"><span id="bib.bib10.6.1" class="ltx_text ltx_font_typewriter">
Jeffrey M Zacks.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib10.7.1" class="ltx_text ltx_font_typewriter">Event perception and memory.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib10.8.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Annual review of psychology</em><span id="bib.bib10.9.2" class="ltx_text ltx_font_typewriter">, 71:165--191, 2020.
</span>
</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib11.5.5.1" class="ltx_text ltx_font_typewriter">Baldassano et al. [2017]</span></span>
<span class="ltx_bibblock"><span id="bib.bib11.7.1" class="ltx_text ltx_font_typewriter">
Christopher Baldassano, Janice Chen, Asieh Zadbood, Jonathan W Pillow, Uri Hasson, and Kenneth A Norman.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib11.8.1" class="ltx_text ltx_font_typewriter">Discovering event structure in continuous narrative perception and memory.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib11.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Neuron</em><span id="bib.bib11.10.2" class="ltx_text ltx_font_typewriter">, 95(3):709--721, 2017.
</span>
</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib12.5.5.1" class="ltx_text ltx_font_typewriter">Michelmann et al. [2023a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib12.7.1" class="ltx_text ltx_font_typewriter">
Sebastian Michelmann, Uri Hasson, and Kenneth A. Norman.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib12.8.1" class="ltx_text ltx_font_typewriter">Evidence that event boundaries are access points for memory retrieval.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib12.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Psychological Science</em><span id="bib.bib12.10.2" class="ltx_text ltx_font_typewriter">, 34(3):326--344, 2023a.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib12.11.1" class="ltx_text ltx_font_typewriter">doi:</span><a target="_blank" href="https://doi.org/10.1177/09567976221128206" title="" class="ltx_ref ltx_href ltx_font_typewriter">10.1177/09567976221128206</a><span id="bib.bib12.12.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib12.13.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://doi.org/10.1177/09567976221128206" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1177/09567976221128206</a><span id="bib.bib12.14.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib12.15.1" class="ltx_text ltx_font_typewriter">PMID: 36595492.
</span>
</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib13.5.5.1" class="ltx_text ltx_font_typewriter">Zacks et al. [2007]</span></span>
<span class="ltx_bibblock"><span id="bib.bib13.7.1" class="ltx_text ltx_font_typewriter">
Jeffrey M Zacks, Nicole K Speer, Khena M Swallow, Todd S Braver, and Jeremy R Reynolds.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib13.8.1" class="ltx_text ltx_font_typewriter">Event perception: a mind-brain perspective.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib13.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Psychological bulletin</em><span id="bib.bib13.10.2" class="ltx_text ltx_font_typewriter">, 133(2):273, 2007.
</span>
</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib14.5.5.1" class="ltx_text ltx_font_typewriter">Zacks et al. [2011]</span></span>
<span class="ltx_bibblock"><span id="bib.bib14.7.1" class="ltx_text ltx_font_typewriter">
Jeffrey M Zacks, Christopher A Kurby, Michelle L Eisenberg, and Nayiri Haroutunian.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib14.8.1" class="ltx_text ltx_font_typewriter">Prediction error associated with the perceptual segmentation of naturalistic events.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib14.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Journal of cognitive neuroscience</em><span id="bib.bib14.10.2" class="ltx_text ltx_font_typewriter">, 23(12):4057--4066, 2011.
</span>
</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib15.5.5.1" class="ltx_text ltx_font_typewriter">Roseboom et al. [2019]</span></span>
<span class="ltx_bibblock"><span id="bib.bib15.7.1" class="ltx_text ltx_font_typewriter">
Warrick Roseboom, Zafeirios Fountas, Kyriacos Nikiforou, David Bhowmik, Murray Shanahan, and Anil K Seth.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib15.8.1" class="ltx_text ltx_font_typewriter">Activity in perceptual classification networks as a basis for human subjective time perception.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib15.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Nature communications</em><span id="bib.bib15.10.2" class="ltx_text ltx_font_typewriter">, 10(1):267, 2019.
</span>
</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib16.5.5.1" class="ltx_text ltx_font_typewriter">Sinclair et al. [2021]</span></span>
<span class="ltx_bibblock"><span id="bib.bib16.7.1" class="ltx_text ltx_font_typewriter">
Alyssa H. Sinclair, Grace M. Manalili, Iva K. Brunec, R. Alison Adcock, and Morgan D. Barense.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib16.8.1" class="ltx_text ltx_font_typewriter">Prediction errors disrupt hippocampal representations and update episodic memories.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib16.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Proceedings of the National Academy of Sciences</em><span id="bib.bib16.10.2" class="ltx_text ltx_font_typewriter">, 118(51):e2117625118, 2021.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib16.11.1" class="ltx_text ltx_font_typewriter">doi:</span><a target="_blank" href="https://doi.org/10.1073/pnas.2117625118" title="" class="ltx_ref ltx_href ltx_font_typewriter">10.1073/pnas.2117625118</a><span id="bib.bib16.12.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib16.13.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://www.pnas.org/doi/abs/10.1073/pnas.2117625118" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.pnas.org/doi/abs/10.1073/pnas.2117625118</a><span id="bib.bib16.14.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib17.5.5.1" class="ltx_text ltx_font_typewriter">Fountas et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib17.7.1" class="ltx_text ltx_font_typewriter">
Zafeirios Fountas, Anastasia Sylaidi, Kyriacos Nikiforou, Anil K. Seth, Murray Shanahan, and Warrick Roseboom.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib17.8.1" class="ltx_text ltx_font_typewriter">A Predictive Processing Model of Episodic Memory and Time Perception.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib17.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Neural Computation</em><span id="bib.bib17.10.2" class="ltx_text ltx_font_typewriter">, 34(7):1501--1544, 06 2022.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib17.11.1" class="ltx_text ltx_font_typewriter">ISSN 0899-7667.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib17.12.1" class="ltx_text ltx_font_typewriter">doi:</span><a target="_blank" href="https://doi.org/10.1162/neco_a_01514" title="" class="ltx_ref ltx_href ltx_font_typewriter">10.1162/neco_a_01514</a><span id="bib.bib17.13.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib17.14.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://doi.org/10.1162/neco_a_01514" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1162/neco_a_01514</a><span id="bib.bib17.15.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib18.4.4.1" class="ltx_text ltx_font_typewriter">Howard and Kahana [2002]</span></span>
<span class="ltx_bibblock"><span id="bib.bib18.6.1" class="ltx_text ltx_font_typewriter">
Marc W Howard and Michael J Kahana.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib18.7.1" class="ltx_text ltx_font_typewriter">A distributed representation of temporal context.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib18.8.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Journal of mathematical psychology</em><span id="bib.bib18.9.2" class="ltx_text ltx_font_typewriter">, 46(3):269--299, 2002.
</span>
</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib19.5.5.1" class="ltx_text ltx_font_typewriter">Ji-An et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib19.7.1" class="ltx_text ltx_font_typewriter">
Li Ji-An, Corey Y. Zhou, Marcus K. Benna, and Marcelo G. Mattar.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib19.8.1" class="ltx_text ltx_font_typewriter">Linking in-context learning in transformers to human episodic memory, 2024.
</span>
</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib20.5.5.1" class="ltx_text ltx_font_typewriter">Kumar et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib20.7.1" class="ltx_text ltx_font_typewriter">
Manoj Kumar, Ariel Goldstein, Sebastian Michelmann, Jeffrey M Zacks, Uri Hasson, and Kenneth A Norman.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib20.8.1" class="ltx_text ltx_font_typewriter">Bayesian surprise predicts human event segmentation in story listening.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib20.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Cognitive science</em><span id="bib.bib20.10.2" class="ltx_text ltx_font_typewriter">, 47(10):e13343, 2023.
</span>
</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib21.5.5.1" class="ltx_text ltx_font_typewriter">Rae et al. [2020]</span></span>
<span class="ltx_bibblock"><span id="bib.bib21.7.1" class="ltx_text ltx_font_typewriter">
Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib21.8.1" class="ltx_text ltx_font_typewriter">Compressive transformers for long-range sequence modelling.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib21.9.1" class="ltx_text ltx_font_typewriter">In </span><em id="bib.bib21.10.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">International Conference on Learning Representations</em><span id="bib.bib21.11.3" class="ltx_text ltx_font_typewriter">, 2020.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib21.12.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://openreview.net/forum?id=SylKikSYDH" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=SylKikSYDH</a><span id="bib.bib21.13.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib22.5.5.1" class="ltx_text ltx_font_typewriter">Bai et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib22.7.1" class="ltx_text ltx_font_typewriter">
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib22.8.1" class="ltx_text ltx_font_typewriter">Longbench: A bilingual, multitask benchmark for long context understanding.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib22.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">arXiv preprint arXiv:2308.14508</em><span id="bib.bib22.10.2" class="ltx_text ltx_font_typewriter">, 2023.
</span>
</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib23.5.5.1" class="ltx_text ltx_font_typewriter">Katharopoulos et al. [2020]</span></span>
<span class="ltx_bibblock"><span id="bib.bib23.7.1" class="ltx_text ltx_font_typewriter">
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib23.8.1" class="ltx_text ltx_font_typewriter">Transformers are rnns: Fast autoregressive transformers with linear attention.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib23.9.1" class="ltx_text ltx_font_typewriter">In </span><em id="bib.bib23.10.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">International conference on machine learning</em><span id="bib.bib23.11.3" class="ltx_text ltx_font_typewriter">, pages 5156--5165. PMLR, 2020.
</span>
</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib24.5.5.1" class="ltx_text ltx_font_typewriter">Munkhdalai et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib24.7.1" class="ltx_text ltx_font_typewriter">
Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib24.8.1" class="ltx_text ltx_font_typewriter">Leave no context behind: Efficient infinite context transformers with infini-attention, 2024.
</span>
</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib25.5.5.1" class="ltx_text ltx_font_typewriter">Vaswani et al. [2017]</span></span>
<span class="ltx_bibblock"><span id="bib.bib25.7.1" class="ltx_text ltx_font_typewriter">
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib25.8.1" class="ltx_text ltx_font_typewriter">Attention is all you need.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib25.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Advances in neural information processing systems</em><span id="bib.bib25.10.2" class="ltx_text ltx_font_typewriter">, 30, 2017.
</span>
</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib26.5.5.1" class="ltx_text ltx_font_typewriter">Su et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib26.7.1" class="ltx_text ltx_font_typewriter">
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib26.8.1" class="ltx_text ltx_font_typewriter">Roformer: Enhanced transformer with rotary position embedding.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib26.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Neurocomputing</em><span id="bib.bib26.10.2" class="ltx_text ltx_font_typewriter">, 568:127063, 2024.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib26.11.1" class="ltx_text ltx_font_typewriter">ISSN 0925-2312.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib26.12.1" class="ltx_text ltx_font_typewriter">doi:</span><a target="_blank" href="https://doi.org/https://doi.org/10.1016/j.neucom.2023.127063" title="" class="ltx_ref ltx_href ltx_font_typewriter">https://doi.org/10.1016/j.neucom.2023.127063</a><span id="bib.bib26.13.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib26.14.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://www.sciencedirect.com/science/article/pii/S0925231223011864" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.sciencedirect.com/science/article/pii/S0925231223011864</a><span id="bib.bib26.15.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib27.5.5.1" class="ltx_text ltx_font_typewriter">Chen et al. [2024a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib27.7.1" class="ltx_text ltx_font_typewriter">
Guanzheng Chen, Xin Li, Zaiqiao Meng, Shangsong Liang, and Lidong Bing.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib27.8.1" class="ltx_text ltx_font_typewriter">CLEX: Continuous length extrapolation for large language models.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib27.9.1" class="ltx_text ltx_font_typewriter">In </span><em id="bib.bib27.10.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">The Twelfth International Conference on Learning Representations</em><span id="bib.bib27.11.3" class="ltx_text ltx_font_typewriter">, 2024a.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib27.12.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://openreview.net/forum?id=wXpSidPpc5" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=wXpSidPpc5</a><span id="bib.bib27.13.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib28.5.5.1" class="ltx_text ltx_font_typewriter">Xiong et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib28.7.1" class="ltx_text ltx_font_typewriter">
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib28.8.1" class="ltx_text ltx_font_typewriter">Effective long-context scaling of foundation models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib28.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">arXiv preprint arXiv:2309.16039</em><span id="bib.bib28.10.2" class="ltx_text ltx_font_typewriter">, 2023.
</span>
</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib29.5.5.1" class="ltx_text ltx_font_typewriter">Liu et al. [2024b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib29.7.1" class="ltx_text ltx_font_typewriter">
Xiaoran Liu, Hang Yan, Chenxin An, Xipeng Qiu, and Dahua Lin.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib29.8.1" class="ltx_text ltx_font_typewriter">Scaling laws of roPE-based extrapolation.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib29.9.1" class="ltx_text ltx_font_typewriter">In </span><em id="bib.bib29.10.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">The Twelfth International Conference on Learning Representations</em><span id="bib.bib29.11.3" class="ltx_text ltx_font_typewriter">, 2024b.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib29.12.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://openreview.net/forum?id=JO7k0SJ5V6" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=JO7k0SJ5V6</a><span id="bib.bib29.13.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib30.5.5.1" class="ltx_text ltx_font_typewriter">Peng et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib30.7.1" class="ltx_text ltx_font_typewriter">
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib30.8.1" class="ltx_text ltx_font_typewriter">YaRN: Efficient context window extension of large language models.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib30.9.1" class="ltx_text ltx_font_typewriter">In </span><em id="bib.bib30.10.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">The Twelfth International Conference on Learning Representations</em><span id="bib.bib30.11.3" class="ltx_text ltx_font_typewriter">, 2024.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib30.12.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://openreview.net/forum?id=wHBfxhZu1u" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=wHBfxhZu1u</a><span id="bib.bib30.13.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib31.5.5.1" class="ltx_text ltx_font_typewriter">Ding et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib31.7.1" class="ltx_text ltx_font_typewriter">
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib31.8.1" class="ltx_text ltx_font_typewriter">Longrope: Extending llm context window beyond 2 million tokens.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib31.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">arXiv preprint arXiv:2402.13753</em><span id="bib.bib31.10.2" class="ltx_text ltx_font_typewriter">, 2024.
</span>
</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib32.5.5.1" class="ltx_text ltx_font_typewriter">Press et al. [2021]</span></span>
<span class="ltx_bibblock"><span id="bib.bib32.7.1" class="ltx_text ltx_font_typewriter">
Ofir Press, Noah A Smith, and Mike Lewis.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib32.8.1" class="ltx_text ltx_font_typewriter">Train short, test long: Attention with linear biases enables input length extrapolation.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib32.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">arXiv preprint arXiv:2108.12409</em><span id="bib.bib32.10.2" class="ltx_text ltx_font_typewriter">, 2021.
</span>
</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib33.5.5.1" class="ltx_text ltx_font_typewriter">Chen et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib33.7.1" class="ltx_text ltx_font_typewriter">
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib33.8.1" class="ltx_text ltx_font_typewriter">Extending context window of large language models via positional interpolation.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib33.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">arXiv preprint arXiv:2306.15595</em><span id="bib.bib33.10.2" class="ltx_text ltx_font_typewriter">, 2023.
</span>
</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib34.5.5.1" class="ltx_text ltx_font_typewriter">Jin et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib34.7.1" class="ltx_text ltx_font_typewriter">
Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib34.8.1" class="ltx_text ltx_font_typewriter">Llm maybe longlm: Self-extend llm context window without tuning, 2024.
</span>
</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib35.5.5.1" class="ltx_text ltx_font_typewriter">Chi et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib35.7.1" class="ltx_text ltx_font_typewriter">
Ta-Chung Chi, Ting-Han Fan, Peter Ramadge, and Alexander Rudnicky.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib35.8.1" class="ltx_text ltx_font_typewriter">KERPLE: Kernelized relative positional embedding for length extrapolation.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib35.9.1" class="ltx_text ltx_font_typewriter">In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, </span><em id="bib.bib35.10.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">Advances in Neural Information Processing Systems</em><span id="bib.bib35.11.3" class="ltx_text ltx_font_typewriter">, 2022.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib35.12.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://openreview.net/forum?id=hXzOqPlXDwm" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=hXzOqPlXDwm</a><span id="bib.bib35.13.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib36.5.5.1" class="ltx_text ltx_font_typewriter">Li et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib36.7.1" class="ltx_text ltx_font_typewriter">
Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontanon, Manzil Zaheer, Sumit Sanghai, Yiming Yang, Sanjiv Kumar, and Srinadh Bhojanapalli.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib36.8.1" class="ltx_text ltx_font_typewriter">Functional interpolation for relative positions improves long context transformers.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib36.9.1" class="ltx_text ltx_font_typewriter">In </span><em id="bib.bib36.10.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">The Twelfth International Conference on Learning Representations</em><span id="bib.bib36.11.3" class="ltx_text ltx_font_typewriter">, 2024.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib36.12.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://openreview.net/forum?id=rR03qFesqk" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=rR03qFesqk</a><span id="bib.bib36.13.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib37.5.5.1" class="ltx_text ltx_font_typewriter">Raffel et al. [2020]</span></span>
<span class="ltx_bibblock"><span id="bib.bib37.7.1" class="ltx_text ltx_font_typewriter">
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib37.8.1" class="ltx_text ltx_font_typewriter">Exploring the limits of transfer learning with a unified text-to-text transformer.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib37.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Journal of Machine Learning Research</em><span id="bib.bib37.10.2" class="ltx_text ltx_font_typewriter">, 21(140):1--67, 2020.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib37.11.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="http://jmlr.org/papers/v21/20-074.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://jmlr.org/papers/v21/20-074.html</a><span id="bib.bib37.12.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib38.4.4.1" class="ltx_text ltx_font_typewriter">Dao [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib38.6.1" class="ltx_text ltx_font_typewriter">
Tri Dao.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib38.7.1" class="ltx_text ltx_font_typewriter">Flashattention-2: Faster attention with better parallelism and work partitioning.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib38.8.1" class="ltx_text ltx_font_typewriter">In </span><em id="bib.bib38.9.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">The Twelfth International Conference on Learning Representations</em><span id="bib.bib38.10.3" class="ltx_text ltx_font_typewriter">, 2024.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib38.11.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://openreview.net/forum?id=mZn2Xyh9Ec" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=mZn2Xyh9Ec</a><span id="bib.bib38.12.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib39.5.5.1" class="ltx_text ltx_font_typewriter">Han et al. [2024a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib39.7.1" class="ltx_text ltx_font_typewriter">
Insu Han, Rajesh Jayaram, Amin Karbasi, Vahab Mirrokni, David Woodruff, and Amir Zandieh.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib39.8.1" class="ltx_text ltx_font_typewriter">Hyperattention: Long-context attention in near-linear time.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib39.9.1" class="ltx_text ltx_font_typewriter">In </span><em id="bib.bib39.10.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">The Twelfth International Conference on Learning Representations</em><span id="bib.bib39.11.3" class="ltx_text ltx_font_typewriter">, 2024a.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib39.12.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://openreview.net/forum?id=Eh0Od2BJIM" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=Eh0Od2BJIM</a><span id="bib.bib39.13.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib40.5.5.1" class="ltx_text ltx_font_typewriter">Aminabadi et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib40.7.1" class="ltx_text ltx_font_typewriter">
Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, and Yuxiong He.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib40.8.1" class="ltx_text ltx_font_typewriter">Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib40.9.1" class="ltx_text ltx_font_typewriter">In </span><em id="bib.bib40.10.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis</em><span id="bib.bib40.11.3" class="ltx_text ltx_font_typewriter">, SC ’22. IEEE Press, 2022.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib40.12.1" class="ltx_text ltx_font_typewriter">ISBN 9784665454445.
</span>
</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib41.5.5.1" class="ltx_text ltx_font_typewriter">Kwon et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib41.7.1" class="ltx_text ltx_font_typewriter">
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib41.8.1" class="ltx_text ltx_font_typewriter">Efficient memory management for large language model serving with pagedattention.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib41.9.1" class="ltx_text ltx_font_typewriter">In </span><em id="bib.bib41.10.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">Proceedings of the 29th Symposium on Operating Systems Principles</em><span id="bib.bib41.11.3" class="ltx_text ltx_font_typewriter">, SOSP ’23, page 611–626, New York, NY, USA, 2023. Association for Computing Machinery.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib41.12.1" class="ltx_text ltx_font_typewriter">ISBN 9798400702297.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib41.13.1" class="ltx_text ltx_font_typewriter">doi:</span><a target="_blank" href="https://doi.org/10.1145/3600006.3613165" title="" class="ltx_ref ltx_href ltx_font_typewriter">10.1145/3600006.3613165</a><span id="bib.bib41.14.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib41.15.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://doi.org/10.1145/3600006.3613165" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1145/3600006.3613165</a><span id="bib.bib41.16.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib42.5.5.1" class="ltx_text ltx_font_typewriter">Liu et al. [2024c]</span></span>
<span class="ltx_bibblock"><span id="bib.bib42.7.1" class="ltx_text ltx_font_typewriter">
Hao Liu, Matei Zaharia, and Pieter Abbeel.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib42.8.1" class="ltx_text ltx_font_typewriter">Ringattention with blockwise transformers for near-infinite context.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib42.9.1" class="ltx_text ltx_font_typewriter">In </span><em id="bib.bib42.10.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">The Twelfth International Conference on Learning Representations</em><span id="bib.bib42.11.3" class="ltx_text ltx_font_typewriter">, 2024c.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib42.12.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://openreview.net/forum?id=WsRHpHH4s0" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=WsRHpHH4s0</a><span id="bib.bib42.13.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib43.5.5.1" class="ltx_text ltx_font_typewriter">Brandon et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib43.7.1" class="ltx_text ltx_font_typewriter">
William Brandon, Aniruddha Nrusimha, Kevin Qian, Zachary Ankner, Tian Jin, Zhiye Song, and Jonathan Ragan-Kelley.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib43.8.1" class="ltx_text ltx_font_typewriter">Striped attention: Faster ring attention for causal transformers.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib43.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">arXiv preprint arXiv:2311.09431</em><span id="bib.bib43.10.2" class="ltx_text ltx_font_typewriter">, 2023.
</span>
</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib44.5.5.1" class="ltx_text ltx_font_typewriter">Nawrot et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib44.7.1" class="ltx_text ltx_font_typewriter">
Piotr Nawrot, Adrian Łańcucki, Marcin Chochowski, David Tarjan, and Edoardo M Ponti.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib44.8.1" class="ltx_text ltx_font_typewriter">Dynamic memory compression: Retrofitting llms for accelerated inference.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib44.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">arXiv preprint arXiv:2403.09636</em><span id="bib.bib44.10.2" class="ltx_text ltx_font_typewriter">, 2024.
</span>
</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib45.5.5.1" class="ltx_text ltx_font_typewriter">Zhang et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib45.7.1" class="ltx_text ltx_font_typewriter">
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Re, Clark Barrett, Zhangyang Wang, and Beidi Chen.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib45.8.1" class="ltx_text ltx_font_typewriter">H2o: Heavy-hitter oracle for efficient generative inference of large language models.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib45.9.1" class="ltx_text ltx_font_typewriter">In </span><em id="bib.bib45.10.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">Thirty-seventh Conference on Neural Information Processing Systems</em><span id="bib.bib45.11.3" class="ltx_text ltx_font_typewriter">, 2023.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib45.12.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://openreview.net/forum?id=RkRrPp7GKO" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=RkRrPp7GKO</a><span id="bib.bib45.13.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib46.5.5.1" class="ltx_text ltx_font_typewriter">Zhu et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib46.7.1" class="ltx_text ltx_font_typewriter">
Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib46.8.1" class="ltx_text ltx_font_typewriter">PoSE: Efficient context window extension of LLMs via positional skip-wise training.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib46.9.1" class="ltx_text ltx_font_typewriter">In </span><em id="bib.bib46.10.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">The Twelfth International Conference on Learning Representations</em><span id="bib.bib46.11.3" class="ltx_text ltx_font_typewriter">, 2024.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib46.12.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://openreview.net/forum?id=3Z1gxuAQrA" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=3Z1gxuAQrA</a><span id="bib.bib46.13.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib47.5.5.1" class="ltx_text ltx_font_typewriter">Chen et al. [2024b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib47.7.1" class="ltx_text ltx_font_typewriter">
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib47.8.1" class="ltx_text ltx_font_typewriter">LongloRA: Efficient fine-tuning of long-context large language models.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib47.9.1" class="ltx_text ltx_font_typewriter">In </span><em id="bib.bib47.10.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">The Twelfth International Conference on Learning Representations</em><span id="bib.bib47.11.3" class="ltx_text ltx_font_typewriter">, 2024b.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib47.12.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://openreview.net/forum?id=6PmJoRfdaK" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=6PmJoRfdaK</a><span id="bib.bib47.13.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib48.5.5.1" class="ltx_text ltx_font_typewriter">Wang et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib48.7.1" class="ltx_text ltx_font_typewriter">
Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib48.8.1" class="ltx_text ltx_font_typewriter">Augmenting language models with long-term memory.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib48.9.1" class="ltx_text ltx_font_typewriter">In </span><em id="bib.bib48.10.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">Thirty-seventh Conference on Neural Information Processing Systems</em><span id="bib.bib48.11.3" class="ltx_text ltx_font_typewriter">, 2023.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib48.12.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://openreview.net/forum?id=BryMFPQ4L6" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=BryMFPQ4L6</a><span id="bib.bib48.13.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib49.5.5.1" class="ltx_text ltx_font_typewriter">Ivgi et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib49.7.1" class="ltx_text ltx_font_typewriter">
Maor Ivgi, Uri Shaham, and Jonathan Berant.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib49.8.1" class="ltx_text ltx_font_typewriter">Efficient long-text understanding with short-text models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib49.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Transactions of the Association for Computational Linguistics</em><span id="bib.bib49.10.2" class="ltx_text ltx_font_typewriter">, 11:284--299, 2023.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib49.11.1" class="ltx_text ltx_font_typewriter">doi:</span><a target="_blank" href="https://doi.org/10.1162/tacl_a_00547" title="" class="ltx_ref ltx_href ltx_font_typewriter">10.1162/tacl_a_00547</a><span id="bib.bib49.12.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib49.13.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://aclanthology.org/2023.tacl-1.17" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/2023.tacl-1.17</a><span id="bib.bib49.14.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib50.5.5.1" class="ltx_text ltx_font_typewriter">Gershman et al. [2012]</span></span>
<span class="ltx_bibblock"><span id="bib.bib50.7.1" class="ltx_text ltx_font_typewriter">
Samuel J Gershman, Christopher D Moore, Michael T Todd, Kenneth A Norman, and Per B Sederberg.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib50.8.1" class="ltx_text ltx_font_typewriter">The successor representation and temporal context.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib50.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Neural Computation</em><span id="bib.bib50.10.2" class="ltx_text ltx_font_typewriter">, 24(6):1553--1568, 2012.
</span>
</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib51.4.4.1" class="ltx_text ltx_font_typewriter">Benna and Fusi [2021]</span></span>
<span class="ltx_bibblock"><span id="bib.bib51.6.1" class="ltx_text ltx_font_typewriter">
Marcus K Benna and Stefano Fusi.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib51.7.1" class="ltx_text ltx_font_typewriter">Place cells may simply be memory cells: Memory compression leads to spatial tuning and history dependence.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib51.8.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Proceedings of the National Academy of Sciences</em><span id="bib.bib51.9.2" class="ltx_text ltx_font_typewriter">, 118(51):e2018422118, 2021.
</span>
</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib52.5.5.1" class="ltx_text ltx_font_typewriter">Blundell et al. [2016]</span></span>
<span class="ltx_bibblock"><span id="bib.bib52.7.1" class="ltx_text ltx_font_typewriter">
Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z Leibo, Jack Rae, Daan Wierstra, and Demis Hassabis.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib52.8.1" class="ltx_text ltx_font_typewriter">Model-free episodic control.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib52.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">arXiv preprint arXiv:1606.04460</em><span id="bib.bib52.10.2" class="ltx_text ltx_font_typewriter">, 2016.
</span>
</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib53.5.5.1" class="ltx_text ltx_font_typewriter">Pritzel et al. [2017]</span></span>
<span class="ltx_bibblock"><span id="bib.bib53.7.1" class="ltx_text ltx_font_typewriter">
Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adrià Puigdomènech Badia, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib53.8.1" class="ltx_text ltx_font_typewriter">Neural episodic control.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib53.9.1" class="ltx_text ltx_font_typewriter">In Doina Precup and Yee Whye Teh, editors, </span><em id="bib.bib53.10.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">Proceedings of the 34th International Conference on Machine Learning</em><span id="bib.bib53.11.3" class="ltx_text ltx_font_typewriter">, volume 70 of </span><em id="bib.bib53.12.4" class="ltx_emph ltx_font_typewriter ltx_font_italic">Proceedings of Machine Learning Research</em><span id="bib.bib53.13.5" class="ltx_text ltx_font_typewriter">, pages 2827--2836. PMLR, 06--11 Aug 2017.
</span>
</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib54.5.5.1" class="ltx_text ltx_font_typewriter">Coda-Forno et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib54.7.1" class="ltx_text ltx_font_typewriter">
Julian Coda-Forno, Changmin Yu, Qinghai Guo, Zafeirios Fountas, and Neil Burgess.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib54.8.1" class="ltx_text ltx_font_typewriter">Leveraging episodic memory to improve world models for reinforcement learning.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib54.9.1" class="ltx_text ltx_font_typewriter">In </span><em id="bib.bib54.10.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">Memory in Artificial and Real Intelligence (MemARI) Workshop at 36th Conference on Neural Information Processing Systems (NeurIPS 2022)</em><span id="bib.bib54.11.3" class="ltx_text ltx_font_typewriter">, 2024.
</span>
</span>
</li>
<li id="bib.bib55" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib55.5.5.1" class="ltx_text ltx_font_typewriter">Kirkpatrick et al. [2017]</span></span>
<span class="ltx_bibblock"><span id="bib.bib55.7.1" class="ltx_text ltx_font_typewriter">
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib55.8.1" class="ltx_text ltx_font_typewriter">Overcoming catastrophic forgetting in neural networks.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib55.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Proceedings of the national academy of sciences</em><span id="bib.bib55.10.2" class="ltx_text ltx_font_typewriter">, 114(13):3521--3526, 2017.
</span>
</span>
</li>
<li id="bib.bib56" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib56.4.4.1" class="ltx_text ltx_font_typewriter">Lopez-Paz and Ranzato [2017]</span></span>
<span class="ltx_bibblock"><span id="bib.bib56.6.1" class="ltx_text ltx_font_typewriter">
David Lopez-Paz and Marc' Aurelio Ranzato.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib56.7.1" class="ltx_text ltx_font_typewriter">Gradient episodic memory for continual learning.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib56.8.1" class="ltx_text ltx_font_typewriter">In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, </span><em id="bib.bib56.9.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">Advances in Neural Information Processing Systems</em><span id="bib.bib56.10.3" class="ltx_text ltx_font_typewriter">, volume 30. Curran Associates, Inc., 2017.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib56.11.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://proceedings.neurips.cc/paper_files/paper/2017/file/f87522788a2be2d171666752f97ddebb-Paper.pdf" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.neurips.cc/paper_files/paper/2017/file/f87522788a2be2d171666752f97ddebb-Paper.pdf</a><span id="bib.bib56.12.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib57" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib57.5.5.1" class="ltx_text ltx_font_typewriter">Chaudhry et al. [2019]</span></span>
<span class="ltx_bibblock"><span id="bib.bib57.7.1" class="ltx_text ltx_font_typewriter">
Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib57.8.1" class="ltx_text ltx_font_typewriter">Efficient lifelong learning with a-GEM.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib57.9.1" class="ltx_text ltx_font_typewriter">In </span><em id="bib.bib57.10.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">International Conference on Learning Representations</em><span id="bib.bib57.11.3" class="ltx_text ltx_font_typewriter">, 2019.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib57.12.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://openreview.net/forum?id=Hkf2_sC5FX" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=Hkf2_sC5FX</a><span id="bib.bib57.13.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib58" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib58.5.5.1" class="ltx_text ltx_font_typewriter">Buzzega et al. [2020]</span></span>
<span class="ltx_bibblock"><span id="bib.bib58.7.1" class="ltx_text ltx_font_typewriter">
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and SIMONE CALDERARA.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib58.8.1" class="ltx_text ltx_font_typewriter">Dark experience for general continual learning: a strong, simple baseline.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib58.9.1" class="ltx_text ltx_font_typewriter">In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, </span><em id="bib.bib58.10.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">Advances in Neural Information Processing Systems</em><span id="bib.bib58.11.3" class="ltx_text ltx_font_typewriter">, volume 33, pages 15920--15930. Curran Associates, Inc., 2020.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib58.12.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://proceedings.neurips.cc/paper_files/paper/2020/file/b704ea2c39778f07c617f6b7ce480e9e-Paper.pdf" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.neurips.cc/paper_files/paper/2020/file/b704ea2c39778f07c617f6b7ce480e9e-Paper.pdf</a><span id="bib.bib58.13.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib59" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib59.5.5.1" class="ltx_text ltx_font_typewriter">Prabhu et al. [2020]</span></span>
<span class="ltx_bibblock"><span id="bib.bib59.7.1" class="ltx_text ltx_font_typewriter">
Ameya Prabhu, Philip H. S. Torr, and Puneet K. Dokania.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib59.8.1" class="ltx_text ltx_font_typewriter">Gdumb: A simple approach that questions our progress in continual learning.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib59.9.1" class="ltx_text ltx_font_typewriter">In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, </span><em id="bib.bib59.10.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">Computer Vision -- ECCV 2020</em><span id="bib.bib59.11.3" class="ltx_text ltx_font_typewriter">, pages 524--540, Cham, 2020. Springer International Publishing.
</span>
</span>
</li>
<li id="bib.bib60" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib60.5.5.1" class="ltx_text ltx_font_typewriter">Das et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib60.7.1" class="ltx_text ltx_font_typewriter">
Payel Das, Subhajit Chaudhury, Elliot Nelson, Igor Melnyk, Sarath Swaminathan, Sihui Dai, Aurélie Lozano, Georgios Kollias, Vijil Chenthamarakshan, Soham Dan, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib60.8.1" class="ltx_text ltx_font_typewriter">Larimar: Large language models with episodic memory control.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib60.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">arXiv preprint arXiv:2403.11901</em><span id="bib.bib60.10.2" class="ltx_text ltx_font_typewriter">, 2024.
</span>
</span>
</li>
<li id="bib.bib61" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib61.4.4.1" class="ltx_text ltx_font_typewriter">Spens and Burgess [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib61.6.1" class="ltx_text ltx_font_typewriter">
Eleanor Spens and Neil Burgess.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib61.7.1" class="ltx_text ltx_font_typewriter">A generative model of memory construction and consolidation.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib61.8.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Nature Human Behaviour</em><span id="bib.bib61.9.2" class="ltx_text ltx_font_typewriter">, pages 1--18, 2024.
</span>
</span>
</li>
<li id="bib.bib62" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib62.5.5.1" class="ltx_text ltx_font_typewriter">Lu et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib62.7.1" class="ltx_text ltx_font_typewriter">
Qihong Lu, Uri Hasson, and Kenneth A Norman.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib62.8.1" class="ltx_text ltx_font_typewriter">A neural network model of when to retrieve and encode episodic memories.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib62.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">elife</em><span id="bib.bib62.10.2" class="ltx_text ltx_font_typewriter">, 11:e74445, 2022.
</span>
</span>
</li>
<li id="bib.bib63" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib63.5.5.1" class="ltx_text ltx_font_typewriter">Sherman et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib63.7.1" class="ltx_text ltx_font_typewriter">
Maxine T Sherman, Zafeirios Fountas, Anil K Seth, and Warrick Roseboom.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib63.8.1" class="ltx_text ltx_font_typewriter">Trial-by-trial predictions of subjective time from human brain activity.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib63.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">PLOS Computational Biology</em><span id="bib.bib63.10.2" class="ltx_text ltx_font_typewriter">, 18(7):e1010223, 2022.
</span>
</span>
</li>
<li id="bib.bib64" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib64.5.5.1" class="ltx_text ltx_font_typewriter">Zakharov et al. [2022a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib64.7.1" class="ltx_text ltx_font_typewriter">
Alexey Zakharov, Qinghai Guo, and Zafeirios Fountas.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib64.8.1" class="ltx_text ltx_font_typewriter">Variational predictive routing with nested subjective timescales.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib64.9.1" class="ltx_text ltx_font_typewriter">In </span><em id="bib.bib64.10.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">International Conference on Learning Representations</em><span id="bib.bib64.11.3" class="ltx_text ltx_font_typewriter">, 2022a.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib64.12.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://openreview.net/forum?id=JxFgJbZ-wft" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=JxFgJbZ-wft</a><span id="bib.bib64.13.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib65" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib65.5.5.1" class="ltx_text ltx_font_typewriter">Zakharov et al. [2022b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib65.7.1" class="ltx_text ltx_font_typewriter">
Alexey Zakharov, Qinghai Guo, and Zafeirios Fountas.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib65.8.1" class="ltx_text ltx_font_typewriter">Long-horizon video prediction using a dynamic latent hierarchy.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib65.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">arXiv preprint arXiv:2212.14376</em><span id="bib.bib65.10.2" class="ltx_text ltx_font_typewriter">, 2022b.
</span>
</span>
</li>
<li id="bib.bib66" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib66.5.5.1" class="ltx_text ltx_font_typewriter">Zakharov et al. [2021]</span></span>
<span class="ltx_bibblock"><span id="bib.bib66.7.1" class="ltx_text ltx_font_typewriter">
Alexey Zakharov, Matthew Crosby, and Zafeirios Fountas.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib66.8.1" class="ltx_text ltx_font_typewriter">Episodic memory for subjective-timescale models.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib66.9.1" class="ltx_text ltx_font_typewriter">In </span><em id="bib.bib66.10.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">ICML 2021 Workshop on Unsupervised Reinforcement Learning</em><span id="bib.bib66.11.3" class="ltx_text ltx_font_typewriter">, 2021.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib66.12.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://openreview.net/forum?id=30lZDhrjonR" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=30lZDhrjonR</a><span id="bib.bib66.13.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib67" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib67.4.4.1" class="ltx_text ltx_font_typewriter">Cowan [2001]</span></span>
<span class="ltx_bibblock"><span id="bib.bib67.6.1" class="ltx_text ltx_font_typewriter">
Nelson Cowan.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib67.7.1" class="ltx_text ltx_font_typewriter">The magical number 4 in short-term memory: A reconsideration of mental storage capacity.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib67.8.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Behavioral and brain sciences</em><span id="bib.bib67.9.2" class="ltx_text ltx_font_typewriter">, 24(1):87--114, 2001.
</span>
</span>
</li>
<li id="bib.bib68" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib68.5.5.1" class="ltx_text ltx_font_typewriter">Xiao et al. [2024b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib68.7.1" class="ltx_text ltx_font_typewriter">
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib68.8.1" class="ltx_text ltx_font_typewriter">Efficient streaming language models with attention sinks.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib68.9.1" class="ltx_text ltx_font_typewriter">In </span><em id="bib.bib68.10.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">The Twelfth International Conference on Learning Representations</em><span id="bib.bib68.11.3" class="ltx_text ltx_font_typewriter">, 2024b.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib68.12.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://openreview.net/forum?id=NG7sS51zVF" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=NG7sS51zVF</a><span id="bib.bib68.13.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib69" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib69.5.5.1" class="ltx_text ltx_font_typewriter">Han et al. [2024b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib69.7.1" class="ltx_text ltx_font_typewriter">
Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib69.8.1" class="ltx_text ltx_font_typewriter">LM-infinite: Zero-shot extreme length generalization for large language models.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib69.9.1" class="ltx_text ltx_font_typewriter">In Kevin Duh, Helena Gomez, and Steven Bethard, editors, </span><em id="bib.bib69.10.2" class="ltx_emph ltx_font_typewriter ltx_font_italic">Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)</em><span id="bib.bib69.11.3" class="ltx_text ltx_font_typewriter">, pages 3991--4008, Mexico City, Mexico, June 2024b. Association for Computational Linguistics.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib69.12.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://aclanthology.org/2024.naacl-long.222" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/2024.naacl-long.222</a><span id="bib.bib69.13.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib70" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib70.5.5.1" class="ltx_text ltx_font_typewriter">Miasnikof et al. [2018]</span></span>
<span class="ltx_bibblock"><span id="bib.bib70.7.1" class="ltx_text ltx_font_typewriter">
Pierre Miasnikof, Alexander Shestopaloff, Anthony Bonner, and Yuri Lawryshyn.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib70.8.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">A Statistical Performance Analysis of Graph Clustering Algorithms</em><span id="bib.bib70.9.2" class="ltx_text ltx_font_typewriter">, pages 170--184.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib70.10.1" class="ltx_text ltx_font_typewriter">05 2018.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib70.11.1" class="ltx_text ltx_font_typewriter">ISBN 978-3-319-92870-8.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib70.12.1" class="ltx_text ltx_font_typewriter">doi:</span><a target="_blank" href="https://doi.org/10.1007/978-3-319-92871-5_11" title="" class="ltx_ref ltx_href ltx_font_typewriter">10.1007/978-3-319-92871-5_11</a><span id="bib.bib70.13.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
<li id="bib.bib71" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib71.4.4.1" class="ltx_text ltx_font_typewriter">Newman and Girvan [2004]</span></span>
<span class="ltx_bibblock"><span id="bib.bib71.6.1" class="ltx_text ltx_font_typewriter">
Mark EJ Newman and Michelle Girvan.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib71.7.1" class="ltx_text ltx_font_typewriter">Finding and evaluating community structure in networks.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib71.8.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Physical review E</em><span id="bib.bib71.9.2" class="ltx_text ltx_font_typewriter">, 69(2):026113, 2004.
</span>
</span>
</li>
<li id="bib.bib72" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib72.5.5.1" class="ltx_text ltx_font_typewriter">Douze et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib72.7.1" class="ltx_text ltx_font_typewriter">
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib72.8.1" class="ltx_text ltx_font_typewriter">The faiss library.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib72.9.1" class="ltx_text ltx_font_typewriter">2024.
</span>
</span>
</li>
<li id="bib.bib73" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib73.5.5.1" class="ltx_text ltx_font_typewriter">Michelmann et al. [2021]</span></span>
<span class="ltx_bibblock"><span id="bib.bib73.7.1" class="ltx_text ltx_font_typewriter">
Sebastian Michelmann, Amy R Price, Bobbi Aubrey, Camilla K Strauss, Werner K Doyle, Daniel Friedman, Patricia C Dugan, Orrin Devinsky, Sasha Devore, Adeen Flinker, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib73.8.1" class="ltx_text ltx_font_typewriter">Moment-by-moment tracking of naturalistic learning and its underlying hippocampo-cortical interactions.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib73.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Nature communications</em><span id="bib.bib73.10.2" class="ltx_text ltx_font_typewriter">, 12(1):5394, 2021.
</span>
</span>
</li>
<li id="bib.bib74" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib74.5.5.1" class="ltx_text ltx_font_typewriter">Lositsky et al. [2016]</span></span>
<span class="ltx_bibblock"><span id="bib.bib74.7.1" class="ltx_text ltx_font_typewriter">
Olga Lositsky, Janice Chen, Daniel Toker, Christopher J Honey, Michael Shvartsman, Jordan L Poppenk, Uri Hasson, and Kenneth A Norman.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib74.8.1" class="ltx_text ltx_font_typewriter">Neural pattern change during encoding of a narrative predicts retrospective duration estimates.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib74.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">elife</em><span id="bib.bib74.10.2" class="ltx_text ltx_font_typewriter">, 5:e16070, 2016.
</span>
</span>
</li>
<li id="bib.bib75" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib75.5.5.1" class="ltx_text ltx_font_typewriter">Mariola et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib75.7.1" class="ltx_text ltx_font_typewriter">
Alberto Mariola, Zafeirios Fountas, Lionel Barnett, and Warrick Roseboom.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib75.8.1" class="ltx_text ltx_font_typewriter">Event segmentation in continuous, naturalistic videos from model-based, data-driven, and human perspectives.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib75.9.1" class="ltx_text ltx_font_typewriter">2022.
</span>
</span>
</li>
<li id="bib.bib76" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib76.5.5.1" class="ltx_text ltx_font_typewriter">Michelmann et al. [2023b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib76.7.1" class="ltx_text ltx_font_typewriter">
Sebastian Michelmann, Manoj Kumar, Kenneth A Norman, and Mariya Toneva.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib76.8.1" class="ltx_text ltx_font_typewriter">Large language models can segment narrative events similarly to humans.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib76.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">arXiv preprint arXiv:2301.10297</em><span id="bib.bib76.10.2" class="ltx_text ltx_font_typewriter">, 2023b.
</span>
</span>
</li>
<li id="bib.bib77" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib77.4.4.1" class="ltx_text ltx_font_typewriter">Baddeley [2003]</span></span>
<span class="ltx_bibblock"><span id="bib.bib77.6.1" class="ltx_text ltx_font_typewriter">
Alan Baddeley.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib77.7.1" class="ltx_text ltx_font_typewriter">Working memory: looking back and looking forward.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib77.8.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Nature reviews neuroscience</em><span id="bib.bib77.9.2" class="ltx_text ltx_font_typewriter">, 4(10):829--839, 2003.
</span>
</span>
</li>
<li id="bib.bib78" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib78.4.4.1" class="ltx_text ltx_font_typewriter">Ericsson and Kintsch [1995]</span></span>
<span class="ltx_bibblock"><span id="bib.bib78.6.1" class="ltx_text ltx_font_typewriter">
K Anders Ericsson and Walter Kintsch.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib78.7.1" class="ltx_text ltx_font_typewriter">Long-term working memory.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib78.8.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Psychological review</em><span id="bib.bib78.9.2" class="ltx_text ltx_font_typewriter">, 102(2):211, 1995.
</span>
</span>
</li>
<li id="bib.bib79" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib79.4.4.1" class="ltx_text ltx_font_typewriter">Fortunato [2010]</span></span>
<span class="ltx_bibblock"><span id="bib.bib79.6.1" class="ltx_text ltx_font_typewriter">
Santo Fortunato.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib79.7.1" class="ltx_text ltx_font_typewriter">Community detection in graphs.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib79.8.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Physics reports</em><span id="bib.bib79.9.2" class="ltx_text ltx_font_typewriter">, 486(3-5):75--174, 2010.
</span>
</span>
</li>
<li id="bib.bib80" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib80.5.5.1" class="ltx_text ltx_font_typewriter">Yang et al. [2016]</span></span>
<span class="ltx_bibblock"><span id="bib.bib80.7.1" class="ltx_text ltx_font_typewriter">
Zhao Yang, René Algesheimer, and Claudio J Tessone.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib80.8.1" class="ltx_text ltx_font_typewriter">A comparative analysis of community detection algorithms on artificial networks.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib80.9.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Scientific reports</em><span id="bib.bib80.10.2" class="ltx_text ltx_font_typewriter">, 6(1):30750, 2016.
</span>
</span>
</li>
<li id="bib.bib81" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib81.4.4.1" class="ltx_text ltx_font_typewriter">Adams and MacKay [2007]</span></span>
<span class="ltx_bibblock"><span id="bib.bib81.6.1" class="ltx_text ltx_font_typewriter">
Ryan Prescott Adams and David JC MacKay.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib81.7.1" class="ltx_text ltx_font_typewriter">Bayesian online changepoint detection.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib81.8.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">arXiv preprint arXiv:0710.3742</em><span id="bib.bib81.9.2" class="ltx_text ltx_font_typewriter">, 2007.
</span>
</span>
</li>
<li id="bib.bib82" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib82.4.4.1" class="ltx_text ltx_font_typewriter">Newman [2004]</span></span>
<span class="ltx_bibblock"><span id="bib.bib82.6.1" class="ltx_text ltx_font_typewriter">
Mark EJ Newman.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib82.7.1" class="ltx_text ltx_font_typewriter">Fast algorithm for detecting community structure in networks.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib82.8.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Physical Review E—Statistical, Nonlinear, and Soft Matter Physics</em><span id="bib.bib82.9.2" class="ltx_text ltx_font_typewriter">, 69(6):066133, 2004.
</span>
</span>
</li>
<li id="bib.bib83" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib83.4.4.1" class="ltx_text ltx_font_typewriter">Panaretos and Zemel [2019]</span></span>
<span class="ltx_bibblock"><span id="bib.bib83.6.1" class="ltx_text ltx_font_typewriter">
Victor M. Panaretos and Yoav Zemel.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib83.7.1" class="ltx_text ltx_font_typewriter">Statistical aspects of wasserstein distances.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib83.8.1" class="ltx_emph ltx_font_typewriter ltx_font_italic">Annual Review of Statistics and Its Application</em><span id="bib.bib83.9.2" class="ltx_text ltx_font_typewriter">, 6(Volume 6, 2019):405--431, 2019.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib83.10.1" class="ltx_text ltx_font_typewriter">ISSN 2326-831X.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib83.11.1" class="ltx_text ltx_font_typewriter">doi:</span><a target="_blank" href="https://doi.org/https://doi.org/10.1146/annurev-statistics-030718-104938" title="" class="ltx_ref ltx_href ltx_font_typewriter">https://doi.org/10.1146/annurev-statistics-030718-104938</a><span id="bib.bib83.12.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib83.13.1" class="ltx_text ltx_font_typewriter">URL </span><a target="_blank" href="https://www.annualreviews.org/content/journals/10.1146/annurev-statistics-030718-104938" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.annualreviews.org/content/journals/10.1146/annurev-statistics-030718-104938</a><span id="bib.bib83.14.2" class="ltx_text ltx_font_typewriter">.
</span>
</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_font_typewriter ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix"><span id="A1.1.1.1" class="ltx_text ltx_font_serif">Appendix A</span> </span>Appendix / supplemental material</h2>

<section id="A1.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_font_typewriter ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="A1.SS1.1.1.1" class="ltx_text ltx_font_serif">A.1</span> </span>Supplementary figures</h3>

<figure id="A1.F4" class="ltx_figure"><img src="/html/2407.09450/assets/x4.png" id="A1.F4.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="461" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering ltx_font_typewriter"><span class="ltx_tag ltx_tag_figure"><span id="A1.F4.9.1.1" class="ltx_text ltx_font_serif">Figure 4</span>: </span>Ablation study in LongBench. Comparison of EM-LLM performance for different combinations of model features (represented by different colours) and different values of <math id="A1.F4.3.m1.1" class="ltx_Math" alttext="\gamma" display="inline"><semantics id="A1.F4.3.m1.1b"><mi id="A1.F4.3.m1.1.1" xref="A1.F4.3.m1.1.1.cmml">γ</mi><annotation-xml encoding="MathML-Content" id="A1.F4.3.m1.1c"><ci id="A1.F4.3.m1.1.1.cmml" xref="A1.F4.3.m1.1.1">𝛾</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.F4.3.m1.1d">\gamma</annotation></semantics></math> (the threshold’s scaling factor). Model variants are aligned on the x-axis based on the average number of block size that emerges for each case. The <math id="A1.F4.4.m2.1" class="ltx_Math" alttext="\gamma" display="inline"><semantics id="A1.F4.4.m2.1b"><mi id="A1.F4.4.m2.1.1" xref="A1.F4.4.m2.1.1.cmml">γ</mi><annotation-xml encoding="MathML-Content" id="A1.F4.4.m2.1c"><ci id="A1.F4.4.m2.1.1.cmml" xref="A1.F4.4.m2.1.1">𝛾</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.F4.4.m2.1d">\gamma</annotation></semantics></math> values for each model variant are shown in the first sub-plot. The corresponding InfLLM performance is also shown. </figcaption>
</figure>
<figure id="A1.F5" class="ltx_figure"><img src="/html/2407.09450/assets/x5.png" id="A1.F5.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="461" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering ltx_font_typewriter"><span class="ltx_tag ltx_tag_figure"><span id="A1.F5.9.1.1" class="ltx_text ltx_font_serif">Figure 5</span>: </span>Ablation study in LongBench. Comparison of EM-LLM performance for different ratios of the contiguity and similarity buffers (represented by different colours) and different values of <math id="A1.F5.3.m1.1" class="ltx_Math" alttext="\gamma" display="inline"><semantics id="A1.F5.3.m1.1b"><mi id="A1.F5.3.m1.1.1" xref="A1.F5.3.m1.1.1.cmml">γ</mi><annotation-xml encoding="MathML-Content" id="A1.F5.3.m1.1c"><ci id="A1.F5.3.m1.1.1.cmml" xref="A1.F5.3.m1.1.1">𝛾</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.F5.3.m1.1d">\gamma</annotation></semantics></math>. Model variants are aligned on the x-axis based on the average number of block size that emerges for each case. The <math id="A1.F5.4.m2.1" class="ltx_Math" alttext="\gamma" display="inline"><semantics id="A1.F5.4.m2.1b"><mi id="A1.F5.4.m2.1.1" xref="A1.F5.4.m2.1.1.cmml">γ</mi><annotation-xml encoding="MathML-Content" id="A1.F5.4.m2.1c"><ci id="A1.F5.4.m2.1.1.cmml" xref="A1.F5.4.m2.1.1">𝛾</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.F5.4.m2.1d">\gamma</annotation></semantics></math> values for each model variant are shown in the first sub-plot. The corresponding InfLLM performance is also shown. </figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A1.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_font_typewriter ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="A1.SS2.1.1.1" class="ltx_text ltx_font_serif">A.2</span> </span>Complexity Analysis of EM-LLM Algorithm</h3>

<div id="A1.SS2.p1" class="ltx_para ltx_noindent">
<p id="A1.SS2.p1.1" class="ltx_p"><span id="A1.SS2.p1.1.1" class="ltx_text ltx_font_typewriter">Here, we provide a detailed analysis of the computational complexity of our Algorithm </span><a href="#alg1" title="Algorithm 1 ‣ 3.3 Boundary refinement ‣ 3 EM-LLM: LLM with Episodic Memory ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">1</span></a><span id="A1.SS2.p1.1.2" class="ltx_text ltx_font_typewriter">, focusing on the boundary refinement step and the calculation of modularity and conductance metrics.</span></p>
</div>
<section id="A1.SS2.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_font_typewriter ltx_title_paragraph">Boundary Refinement Step</h4>

<div id="A1.SS2.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="A1.SS2.SSS0.Px1.p1.3" class="ltx_p"><span id="A1.SS2.SSS0.Px1.p1.3.1" class="ltx_text ltx_font_typewriter">The boundary refinement step involves finding the optimal position </span><math id="A1.SS2.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="\hat{\beta}" display="inline"><semantics id="A1.SS2.SSS0.Px1.p1.1.m1.1a"><mover accent="true" id="A1.SS2.SSS0.Px1.p1.1.m1.1.1" xref="A1.SS2.SSS0.Px1.p1.1.m1.1.1.cmml"><mi id="A1.SS2.SSS0.Px1.p1.1.m1.1.1.2" xref="A1.SS2.SSS0.Px1.p1.1.m1.1.1.2.cmml">β</mi><mo id="A1.SS2.SSS0.Px1.p1.1.m1.1.1.1" xref="A1.SS2.SSS0.Px1.p1.1.m1.1.1.1.cmml">^</mo></mover><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px1.p1.1.m1.1b"><apply id="A1.SS2.SSS0.Px1.p1.1.m1.1.1.cmml" xref="A1.SS2.SSS0.Px1.p1.1.m1.1.1"><ci id="A1.SS2.SSS0.Px1.p1.1.m1.1.1.1.cmml" xref="A1.SS2.SSS0.Px1.p1.1.m1.1.1.1">^</ci><ci id="A1.SS2.SSS0.Px1.p1.1.m1.1.1.2.cmml" xref="A1.SS2.SSS0.Px1.p1.1.m1.1.1.2">𝛽</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px1.p1.1.m1.1c">\hat{\beta}</annotation></semantics></math><span id="A1.SS2.SSS0.Px1.p1.3.2" class="ltx_text ltx_font_typewriter"> between each pair of consecutive initial boundaries </span><math id="A1.SS2.SSS0.Px1.p1.2.m2.2" class="ltx_Math" alttext="(\alpha,\beta)" display="inline"><semantics id="A1.SS2.SSS0.Px1.p1.2.m2.2a"><mrow id="A1.SS2.SSS0.Px1.p1.2.m2.2.3.2" xref="A1.SS2.SSS0.Px1.p1.2.m2.2.3.1.cmml"><mo stretchy="false" id="A1.SS2.SSS0.Px1.p1.2.m2.2.3.2.1" xref="A1.SS2.SSS0.Px1.p1.2.m2.2.3.1.cmml">(</mo><mi id="A1.SS2.SSS0.Px1.p1.2.m2.1.1" xref="A1.SS2.SSS0.Px1.p1.2.m2.1.1.cmml">α</mi><mo id="A1.SS2.SSS0.Px1.p1.2.m2.2.3.2.2" xref="A1.SS2.SSS0.Px1.p1.2.m2.2.3.1.cmml">,</mo><mi id="A1.SS2.SSS0.Px1.p1.2.m2.2.2" xref="A1.SS2.SSS0.Px1.p1.2.m2.2.2.cmml">β</mi><mo stretchy="false" id="A1.SS2.SSS0.Px1.p1.2.m2.2.3.2.3" xref="A1.SS2.SSS0.Px1.p1.2.m2.2.3.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px1.p1.2.m2.2b"><interval closure="open" id="A1.SS2.SSS0.Px1.p1.2.m2.2.3.1.cmml" xref="A1.SS2.SSS0.Px1.p1.2.m2.2.3.2"><ci id="A1.SS2.SSS0.Px1.p1.2.m2.1.1.cmml" xref="A1.SS2.SSS0.Px1.p1.2.m2.1.1">𝛼</ci><ci id="A1.SS2.SSS0.Px1.p1.2.m2.2.2.cmml" xref="A1.SS2.SSS0.Px1.p1.2.m2.2.2">𝛽</ci></interval></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px1.p1.2.m2.2c">(\alpha,\beta)</annotation></semantics></math><span id="A1.SS2.SSS0.Px1.p1.3.3" class="ltx_text ltx_font_typewriter"> that optimizes the chosen metric function </span><math id="A1.SS2.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="f" display="inline"><semantics id="A1.SS2.SSS0.Px1.p1.3.m3.1a"><mi id="A1.SS2.SSS0.Px1.p1.3.m3.1.1" xref="A1.SS2.SSS0.Px1.p1.3.m3.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px1.p1.3.m3.1b"><ci id="A1.SS2.SSS0.Px1.p1.3.m3.1.1.cmml" xref="A1.SS2.SSS0.Px1.p1.3.m3.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px1.p1.3.m3.1c">f</annotation></semantics></math><span id="A1.SS2.SSS0.Px1.p1.3.4" class="ltx_text ltx_font_typewriter">. This step has the following components:</span></p>
</div>
<div id="A1.SS2.SSS0.Px1.p2" class="ltx_para ltx_noindent">
<p id="A1.SS2.SSS0.Px1.p2.6" class="ltx_p"><span id="A1.SS2.SSS0.Px1.p2.6.1" class="ltx_text ltx_font_typewriter">Iteration over initial boundaries: </span><math id="A1.SS2.SSS0.Px1.p2.1.m1.1" class="ltx_Math" alttext="\mathcal{O}(k)" display="inline"><semantics id="A1.SS2.SSS0.Px1.p2.1.m1.1a"><mrow id="A1.SS2.SSS0.Px1.p2.1.m1.1.2" xref="A1.SS2.SSS0.Px1.p2.1.m1.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.SS2.SSS0.Px1.p2.1.m1.1.2.2" xref="A1.SS2.SSS0.Px1.p2.1.m1.1.2.2.cmml">𝒪</mi><mo lspace="0em" rspace="0em" id="A1.SS2.SSS0.Px1.p2.1.m1.1.2.1" xref="A1.SS2.SSS0.Px1.p2.1.m1.1.2.1.cmml">​</mo><mrow id="A1.SS2.SSS0.Px1.p2.1.m1.1.2.3.2" xref="A1.SS2.SSS0.Px1.p2.1.m1.1.2.cmml"><mo stretchy="false" id="A1.SS2.SSS0.Px1.p2.1.m1.1.2.3.2.1" xref="A1.SS2.SSS0.Px1.p2.1.m1.1.2.cmml">(</mo><mi id="A1.SS2.SSS0.Px1.p2.1.m1.1.1" xref="A1.SS2.SSS0.Px1.p2.1.m1.1.1.cmml">k</mi><mo stretchy="false" id="A1.SS2.SSS0.Px1.p2.1.m1.1.2.3.2.2" xref="A1.SS2.SSS0.Px1.p2.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px1.p2.1.m1.1b"><apply id="A1.SS2.SSS0.Px1.p2.1.m1.1.2.cmml" xref="A1.SS2.SSS0.Px1.p2.1.m1.1.2"><times id="A1.SS2.SSS0.Px1.p2.1.m1.1.2.1.cmml" xref="A1.SS2.SSS0.Px1.p2.1.m1.1.2.1"></times><ci id="A1.SS2.SSS0.Px1.p2.1.m1.1.2.2.cmml" xref="A1.SS2.SSS0.Px1.p2.1.m1.1.2.2">𝒪</ci><ci id="A1.SS2.SSS0.Px1.p2.1.m1.1.1.cmml" xref="A1.SS2.SSS0.Px1.p2.1.m1.1.1">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px1.p2.1.m1.1c">\mathcal{O}(k)</annotation></semantics></math><span id="A1.SS2.SSS0.Px1.p2.6.2" class="ltx_text ltx_font_typewriter">, where </span><math id="A1.SS2.SSS0.Px1.p2.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A1.SS2.SSS0.Px1.p2.2.m2.1a"><mi id="A1.SS2.SSS0.Px1.p2.2.m2.1.1" xref="A1.SS2.SSS0.Px1.p2.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px1.p2.2.m2.1b"><ci id="A1.SS2.SSS0.Px1.p2.2.m2.1.1.cmml" xref="A1.SS2.SSS0.Px1.p2.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px1.p2.2.m2.1c">k</annotation></semantics></math><span id="A1.SS2.SSS0.Px1.p2.6.3" class="ltx_text ltx_font_typewriter"> is the number of initial boundaries.
For each pair of boundaries, we compute the metric function </span><math id="A1.SS2.SSS0.Px1.p2.3.m3.1" class="ltx_Math" alttext="f" display="inline"><semantics id="A1.SS2.SSS0.Px1.p2.3.m3.1a"><mi id="A1.SS2.SSS0.Px1.p2.3.m3.1.1" xref="A1.SS2.SSS0.Px1.p2.3.m3.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px1.p2.3.m3.1b"><ci id="A1.SS2.SSS0.Px1.p2.3.m3.1.1.cmml" xref="A1.SS2.SSS0.Px1.p2.3.m3.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px1.p2.3.m3.1c">f</annotation></semantics></math><span id="A1.SS2.SSS0.Px1.p2.6.4" class="ltx_text ltx_font_typewriter"> for all positions between </span><math id="A1.SS2.SSS0.Px1.p2.4.m4.1" class="ltx_Math" alttext="\alpha" display="inline"><semantics id="A1.SS2.SSS0.Px1.p2.4.m4.1a"><mi id="A1.SS2.SSS0.Px1.p2.4.m4.1.1" xref="A1.SS2.SSS0.Px1.p2.4.m4.1.1.cmml">α</mi><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px1.p2.4.m4.1b"><ci id="A1.SS2.SSS0.Px1.p2.4.m4.1.1.cmml" xref="A1.SS2.SSS0.Px1.p2.4.m4.1.1">𝛼</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px1.p2.4.m4.1c">\alpha</annotation></semantics></math><span id="A1.SS2.SSS0.Px1.p2.6.5" class="ltx_text ltx_font_typewriter"> and </span><math id="A1.SS2.SSS0.Px1.p2.5.m5.1" class="ltx_Math" alttext="\beta" display="inline"><semantics id="A1.SS2.SSS0.Px1.p2.5.m5.1a"><mi id="A1.SS2.SSS0.Px1.p2.5.m5.1.1" xref="A1.SS2.SSS0.Px1.p2.5.m5.1.1.cmml">β</mi><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px1.p2.5.m5.1b"><ci id="A1.SS2.SSS0.Px1.p2.5.m5.1.1.cmml" xref="A1.SS2.SSS0.Px1.p2.5.m5.1.1">𝛽</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px1.p2.5.m5.1c">\beta</annotation></semantics></math><span id="A1.SS2.SSS0.Px1.p2.6.6" class="ltx_text ltx_font_typewriter">. In the worst case, this is </span><math id="A1.SS2.SSS0.Px1.p2.6.m6.1" class="ltx_Math" alttext="\mathcal{O}(n)" display="inline"><semantics id="A1.SS2.SSS0.Px1.p2.6.m6.1a"><mrow id="A1.SS2.SSS0.Px1.p2.6.m6.1.2" xref="A1.SS2.SSS0.Px1.p2.6.m6.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.SS2.SSS0.Px1.p2.6.m6.1.2.2" xref="A1.SS2.SSS0.Px1.p2.6.m6.1.2.2.cmml">𝒪</mi><mo lspace="0em" rspace="0em" id="A1.SS2.SSS0.Px1.p2.6.m6.1.2.1" xref="A1.SS2.SSS0.Px1.p2.6.m6.1.2.1.cmml">​</mo><mrow id="A1.SS2.SSS0.Px1.p2.6.m6.1.2.3.2" xref="A1.SS2.SSS0.Px1.p2.6.m6.1.2.cmml"><mo stretchy="false" id="A1.SS2.SSS0.Px1.p2.6.m6.1.2.3.2.1" xref="A1.SS2.SSS0.Px1.p2.6.m6.1.2.cmml">(</mo><mi id="A1.SS2.SSS0.Px1.p2.6.m6.1.1" xref="A1.SS2.SSS0.Px1.p2.6.m6.1.1.cmml">n</mi><mo stretchy="false" id="A1.SS2.SSS0.Px1.p2.6.m6.1.2.3.2.2" xref="A1.SS2.SSS0.Px1.p2.6.m6.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px1.p2.6.m6.1b"><apply id="A1.SS2.SSS0.Px1.p2.6.m6.1.2.cmml" xref="A1.SS2.SSS0.Px1.p2.6.m6.1.2"><times id="A1.SS2.SSS0.Px1.p2.6.m6.1.2.1.cmml" xref="A1.SS2.SSS0.Px1.p2.6.m6.1.2.1"></times><ci id="A1.SS2.SSS0.Px1.p2.6.m6.1.2.2.cmml" xref="A1.SS2.SSS0.Px1.p2.6.m6.1.2.2">𝒪</ci><ci id="A1.SS2.SSS0.Px1.p2.6.m6.1.1.cmml" xref="A1.SS2.SSS0.Px1.p2.6.m6.1.1">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px1.p2.6.m6.1c">\mathcal{O}(n)</annotation></semantics></math><span id="A1.SS2.SSS0.Px1.p2.6.7" class="ltx_text ltx_font_typewriter"> operations per boundary pair.</span></p>
</div>
<div id="A1.SS2.SSS0.Px1.p3" class="ltx_para ltx_noindent">
<p id="A1.SS2.SSS0.Px1.p3.1" class="ltx_p"><span id="A1.SS2.SSS0.Px1.p3.1.1" class="ltx_text ltx_font_typewriter">Therefore, the overall complexity of this step is </span><math id="A1.SS2.SSS0.Px1.p3.1.m1.1" class="ltx_Math" alttext="\mathcal{O}(kn)" display="inline"><semantics id="A1.SS2.SSS0.Px1.p3.1.m1.1a"><mrow id="A1.SS2.SSS0.Px1.p3.1.m1.1.1" xref="A1.SS2.SSS0.Px1.p3.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.SS2.SSS0.Px1.p3.1.m1.1.1.3" xref="A1.SS2.SSS0.Px1.p3.1.m1.1.1.3.cmml">𝒪</mi><mo lspace="0em" rspace="0em" id="A1.SS2.SSS0.Px1.p3.1.m1.1.1.2" xref="A1.SS2.SSS0.Px1.p3.1.m1.1.1.2.cmml">​</mo><mrow id="A1.SS2.SSS0.Px1.p3.1.m1.1.1.1.1" xref="A1.SS2.SSS0.Px1.p3.1.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.SS2.SSS0.Px1.p3.1.m1.1.1.1.1.2" xref="A1.SS2.SSS0.Px1.p3.1.m1.1.1.1.1.1.cmml">(</mo><mrow id="A1.SS2.SSS0.Px1.p3.1.m1.1.1.1.1.1" xref="A1.SS2.SSS0.Px1.p3.1.m1.1.1.1.1.1.cmml"><mi id="A1.SS2.SSS0.Px1.p3.1.m1.1.1.1.1.1.2" xref="A1.SS2.SSS0.Px1.p3.1.m1.1.1.1.1.1.2.cmml">k</mi><mo lspace="0em" rspace="0em" id="A1.SS2.SSS0.Px1.p3.1.m1.1.1.1.1.1.1" xref="A1.SS2.SSS0.Px1.p3.1.m1.1.1.1.1.1.1.cmml">​</mo><mi id="A1.SS2.SSS0.Px1.p3.1.m1.1.1.1.1.1.3" xref="A1.SS2.SSS0.Px1.p3.1.m1.1.1.1.1.1.3.cmml">n</mi></mrow><mo stretchy="false" id="A1.SS2.SSS0.Px1.p3.1.m1.1.1.1.1.3" xref="A1.SS2.SSS0.Px1.p3.1.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px1.p3.1.m1.1b"><apply id="A1.SS2.SSS0.Px1.p3.1.m1.1.1.cmml" xref="A1.SS2.SSS0.Px1.p3.1.m1.1.1"><times id="A1.SS2.SSS0.Px1.p3.1.m1.1.1.2.cmml" xref="A1.SS2.SSS0.Px1.p3.1.m1.1.1.2"></times><ci id="A1.SS2.SSS0.Px1.p3.1.m1.1.1.3.cmml" xref="A1.SS2.SSS0.Px1.p3.1.m1.1.1.3">𝒪</ci><apply id="A1.SS2.SSS0.Px1.p3.1.m1.1.1.1.1.1.cmml" xref="A1.SS2.SSS0.Px1.p3.1.m1.1.1.1.1"><times id="A1.SS2.SSS0.Px1.p3.1.m1.1.1.1.1.1.1.cmml" xref="A1.SS2.SSS0.Px1.p3.1.m1.1.1.1.1.1.1"></times><ci id="A1.SS2.SSS0.Px1.p3.1.m1.1.1.1.1.1.2.cmml" xref="A1.SS2.SSS0.Px1.p3.1.m1.1.1.1.1.1.2">𝑘</ci><ci id="A1.SS2.SSS0.Px1.p3.1.m1.1.1.1.1.1.3.cmml" xref="A1.SS2.SSS0.Px1.p3.1.m1.1.1.1.1.1.3">𝑛</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px1.p3.1.m1.1c">\mathcal{O}(kn)</annotation></semantics></math><span id="A1.SS2.SSS0.Px1.p3.1.2" class="ltx_text ltx_font_typewriter">.</span></p>
</div>
</section>
<section id="A1.SS2.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_font_typewriter ltx_title_paragraph">Metric Function Computation</h4>

<div id="A1.SS2.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="A1.SS2.SSS0.Px2.p1.1" class="ltx_p"><span id="A1.SS2.SSS0.Px2.p1.1.1" class="ltx_text ltx_font_typewriter">The metric functions (modularity or conductance) are computed at the level of individual memory units. For a memory unit of size </span><math id="A1.SS2.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="m" display="inline"><semantics id="A1.SS2.SSS0.Px2.p1.1.m1.1a"><mi id="A1.SS2.SSS0.Px2.p1.1.m1.1.1" xref="A1.SS2.SSS0.Px2.p1.1.m1.1.1.cmml">m</mi><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px2.p1.1.m1.1b"><ci id="A1.SS2.SSS0.Px2.p1.1.m1.1.1.cmml" xref="A1.SS2.SSS0.Px2.p1.1.m1.1.1">𝑚</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px2.p1.1.m1.1c">m</annotation></semantics></math><span id="A1.SS2.SSS0.Px2.p1.1.2" class="ltx_text ltx_font_typewriter">:</span></p>
<ul id="A1.I1" class="ltx_itemize">
<li id="A1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I1.i1.p1" class="ltx_para">
<p id="A1.I1.i1.p1.5" class="ltx_p"><span id="A1.I1.i1.p1.5.1" class="ltx_text ltx_font_italic">Modularity</span><span id="A1.I1.i1.p1.5.2" class="ltx_text ltx_font_typewriter">: The naive computation involves summing over all pairs of nodes within the unit, resulting in a worst-case complexity of </span><math id="A1.I1.i1.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{O}(m^{2})" display="inline"><semantics id="A1.I1.i1.p1.1.m1.1a"><mrow id="A1.I1.i1.p1.1.m1.1.1" xref="A1.I1.i1.p1.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.I1.i1.p1.1.m1.1.1.3" xref="A1.I1.i1.p1.1.m1.1.1.3.cmml">𝒪</mi><mo lspace="0em" rspace="0em" id="A1.I1.i1.p1.1.m1.1.1.2" xref="A1.I1.i1.p1.1.m1.1.1.2.cmml">​</mo><mrow id="A1.I1.i1.p1.1.m1.1.1.1.1" xref="A1.I1.i1.p1.1.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.I1.i1.p1.1.m1.1.1.1.1.2" xref="A1.I1.i1.p1.1.m1.1.1.1.1.1.cmml">(</mo><msup id="A1.I1.i1.p1.1.m1.1.1.1.1.1" xref="A1.I1.i1.p1.1.m1.1.1.1.1.1.cmml"><mi id="A1.I1.i1.p1.1.m1.1.1.1.1.1.2" xref="A1.I1.i1.p1.1.m1.1.1.1.1.1.2.cmml">m</mi><mn id="A1.I1.i1.p1.1.m1.1.1.1.1.1.3" xref="A1.I1.i1.p1.1.m1.1.1.1.1.1.3.cmml">2</mn></msup><mo stretchy="false" id="A1.I1.i1.p1.1.m1.1.1.1.1.3" xref="A1.I1.i1.p1.1.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.I1.i1.p1.1.m1.1b"><apply id="A1.I1.i1.p1.1.m1.1.1.cmml" xref="A1.I1.i1.p1.1.m1.1.1"><times id="A1.I1.i1.p1.1.m1.1.1.2.cmml" xref="A1.I1.i1.p1.1.m1.1.1.2"></times><ci id="A1.I1.i1.p1.1.m1.1.1.3.cmml" xref="A1.I1.i1.p1.1.m1.1.1.3">𝒪</ci><apply id="A1.I1.i1.p1.1.m1.1.1.1.1.1.cmml" xref="A1.I1.i1.p1.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="A1.I1.i1.p1.1.m1.1.1.1.1.1.1.cmml" xref="A1.I1.i1.p1.1.m1.1.1.1.1">superscript</csymbol><ci id="A1.I1.i1.p1.1.m1.1.1.1.1.1.2.cmml" xref="A1.I1.i1.p1.1.m1.1.1.1.1.1.2">𝑚</ci><cn type="integer" id="A1.I1.i1.p1.1.m1.1.1.1.1.1.3.cmml" xref="A1.I1.i1.p1.1.m1.1.1.1.1.1.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I1.i1.p1.1.m1.1c">\mathcal{O}(m^{2})</annotation></semantics></math><span id="A1.I1.i1.p1.5.3" class="ltx_text ltx_font_typewriter">. However, in practice, the similarity graph is often sparse, meaning many node pairs have negligible similarity. Leveraging this sparsity, more efficient implementations can achieve </span><math id="A1.I1.i1.p1.2.m2.1" class="ltx_Math" alttext="\mathcal{O}(l)" display="inline"><semantics id="A1.I1.i1.p1.2.m2.1a"><mrow id="A1.I1.i1.p1.2.m2.1.2" xref="A1.I1.i1.p1.2.m2.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.I1.i1.p1.2.m2.1.2.2" xref="A1.I1.i1.p1.2.m2.1.2.2.cmml">𝒪</mi><mo lspace="0em" rspace="0em" id="A1.I1.i1.p1.2.m2.1.2.1" xref="A1.I1.i1.p1.2.m2.1.2.1.cmml">​</mo><mrow id="A1.I1.i1.p1.2.m2.1.2.3.2" xref="A1.I1.i1.p1.2.m2.1.2.cmml"><mo stretchy="false" id="A1.I1.i1.p1.2.m2.1.2.3.2.1" xref="A1.I1.i1.p1.2.m2.1.2.cmml">(</mo><mi id="A1.I1.i1.p1.2.m2.1.1" xref="A1.I1.i1.p1.2.m2.1.1.cmml">l</mi><mo stretchy="false" id="A1.I1.i1.p1.2.m2.1.2.3.2.2" xref="A1.I1.i1.p1.2.m2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.I1.i1.p1.2.m2.1b"><apply id="A1.I1.i1.p1.2.m2.1.2.cmml" xref="A1.I1.i1.p1.2.m2.1.2"><times id="A1.I1.i1.p1.2.m2.1.2.1.cmml" xref="A1.I1.i1.p1.2.m2.1.2.1"></times><ci id="A1.I1.i1.p1.2.m2.1.2.2.cmml" xref="A1.I1.i1.p1.2.m2.1.2.2">𝒪</ci><ci id="A1.I1.i1.p1.2.m2.1.1.cmml" xref="A1.I1.i1.p1.2.m2.1.1">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I1.i1.p1.2.m2.1c">\mathcal{O}(l)</annotation></semantics></math><span id="A1.I1.i1.p1.5.4" class="ltx_text ltx_font_typewriter"> complexity, where </span><math id="A1.I1.i1.p1.3.m3.1" class="ltx_Math" alttext="l" display="inline"><semantics id="A1.I1.i1.p1.3.m3.1a"><mi id="A1.I1.i1.p1.3.m3.1.1" xref="A1.I1.i1.p1.3.m3.1.1.cmml">l</mi><annotation-xml encoding="MathML-Content" id="A1.I1.i1.p1.3.m3.1b"><ci id="A1.I1.i1.p1.3.m3.1.1.cmml" xref="A1.I1.i1.p1.3.m3.1.1">𝑙</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.I1.i1.p1.3.m3.1c">l</annotation></semantics></math><span id="A1.I1.i1.p1.5.5" class="ltx_text ltx_font_typewriter"> is the number of non-zero similarity edges within the unit </span><cite class="ltx_cite ltx_citemacro_citep"><span id="A1.I1.i1.p1.5.6.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Newman</span><span id="A1.I1.i1.p1.5.7.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib82" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2004</span></a><span id="A1.I1.i1.p1.5.8.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="A1.I1.i1.p1.5.9" class="ltx_text ltx_font_typewriter">. Typically, </span><math id="A1.I1.i1.p1.4.m4.1" class="ltx_Math" alttext="l" display="inline"><semantics id="A1.I1.i1.p1.4.m4.1a"><mi id="A1.I1.i1.p1.4.m4.1.1" xref="A1.I1.i1.p1.4.m4.1.1.cmml">l</mi><annotation-xml encoding="MathML-Content" id="A1.I1.i1.p1.4.m4.1b"><ci id="A1.I1.i1.p1.4.m4.1.1.cmml" xref="A1.I1.i1.p1.4.m4.1.1">𝑙</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.I1.i1.p1.4.m4.1c">l</annotation></semantics></math><span id="A1.I1.i1.p1.5.10" class="ltx_text ltx_font_typewriter"> is much smaller than </span><math id="A1.I1.i1.p1.5.m5.1" class="ltx_Math" alttext="m^{2}" display="inline"><semantics id="A1.I1.i1.p1.5.m5.1a"><msup id="A1.I1.i1.p1.5.m5.1.1" xref="A1.I1.i1.p1.5.m5.1.1.cmml"><mi id="A1.I1.i1.p1.5.m5.1.1.2" xref="A1.I1.i1.p1.5.m5.1.1.2.cmml">m</mi><mn id="A1.I1.i1.p1.5.m5.1.1.3" xref="A1.I1.i1.p1.5.m5.1.1.3.cmml">2</mn></msup><annotation-xml encoding="MathML-Content" id="A1.I1.i1.p1.5.m5.1b"><apply id="A1.I1.i1.p1.5.m5.1.1.cmml" xref="A1.I1.i1.p1.5.m5.1.1"><csymbol cd="ambiguous" id="A1.I1.i1.p1.5.m5.1.1.1.cmml" xref="A1.I1.i1.p1.5.m5.1.1">superscript</csymbol><ci id="A1.I1.i1.p1.5.m5.1.1.2.cmml" xref="A1.I1.i1.p1.5.m5.1.1.2">𝑚</ci><cn type="integer" id="A1.I1.i1.p1.5.m5.1.1.3.cmml" xref="A1.I1.i1.p1.5.m5.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I1.i1.p1.5.m5.1c">m^{2}</annotation></semantics></math><span id="A1.I1.i1.p1.5.11" class="ltx_text ltx_font_typewriter">, especially for larger units, leading to significant computational savings.</span></p>
</div>
</li>
<li id="A1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I1.i2.p1" class="ltx_para ltx_noindent">
<p id="A1.I1.i2.p1.1" class="ltx_p"><span id="A1.I1.i2.p1.1.1" class="ltx_text ltx_font_italic">Conductance</span><span id="A1.I1.i2.p1.1.2" class="ltx_text ltx_font_typewriter">: This requires computing the sum of edge weights crossing the boundary and the total volume of the unit, which can be done in </span><math id="A1.I1.i2.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{O}(m)" display="inline"><semantics id="A1.I1.i2.p1.1.m1.1a"><mrow id="A1.I1.i2.p1.1.m1.1.2" xref="A1.I1.i2.p1.1.m1.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.I1.i2.p1.1.m1.1.2.2" xref="A1.I1.i2.p1.1.m1.1.2.2.cmml">𝒪</mi><mo lspace="0em" rspace="0em" id="A1.I1.i2.p1.1.m1.1.2.1" xref="A1.I1.i2.p1.1.m1.1.2.1.cmml">​</mo><mrow id="A1.I1.i2.p1.1.m1.1.2.3.2" xref="A1.I1.i2.p1.1.m1.1.2.cmml"><mo stretchy="false" id="A1.I1.i2.p1.1.m1.1.2.3.2.1" xref="A1.I1.i2.p1.1.m1.1.2.cmml">(</mo><mi id="A1.I1.i2.p1.1.m1.1.1" xref="A1.I1.i2.p1.1.m1.1.1.cmml">m</mi><mo stretchy="false" id="A1.I1.i2.p1.1.m1.1.2.3.2.2" xref="A1.I1.i2.p1.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.I1.i2.p1.1.m1.1b"><apply id="A1.I1.i2.p1.1.m1.1.2.cmml" xref="A1.I1.i2.p1.1.m1.1.2"><times id="A1.I1.i2.p1.1.m1.1.2.1.cmml" xref="A1.I1.i2.p1.1.m1.1.2.1"></times><ci id="A1.I1.i2.p1.1.m1.1.2.2.cmml" xref="A1.I1.i2.p1.1.m1.1.2.2">𝒪</ci><ci id="A1.I1.i2.p1.1.m1.1.1.cmml" xref="A1.I1.i2.p1.1.m1.1.1">𝑚</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I1.i2.p1.1.m1.1c">\mathcal{O}(m)</annotation></semantics></math><span id="A1.I1.i2.p1.1.3" class="ltx_text ltx_font_typewriter"> time.</span></p>
</div>
</li>
</ul>
</div>
<div id="A1.SS2.SSS0.Px2.p2" class="ltx_para ltx_noindent">
<p id="A1.SS2.SSS0.Px2.p2.7" class="ltx_p"><span id="A1.SS2.SSS0.Px2.p2.7.1" class="ltx_text ltx_font_typewriter">Given that </span><math id="A1.SS2.SSS0.Px2.p2.1.m1.1" class="ltx_Math" alttext="m" display="inline"><semantics id="A1.SS2.SSS0.Px2.p2.1.m1.1a"><mi id="A1.SS2.SSS0.Px2.p2.1.m1.1.1" xref="A1.SS2.SSS0.Px2.p2.1.m1.1.1.cmml">m</mi><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px2.p2.1.m1.1b"><ci id="A1.SS2.SSS0.Px2.p2.1.m1.1.1.cmml" xref="A1.SS2.SSS0.Px2.p2.1.m1.1.1">𝑚</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px2.p2.1.m1.1c">m</annotation></semantics></math><span id="A1.SS2.SSS0.Px2.p2.7.2" class="ltx_text ltx_font_typewriter"> is typically much smaller than </span><math id="A1.SS2.SSS0.Px2.p2.2.m2.1" class="ltx_Math" alttext="n" display="inline"><semantics id="A1.SS2.SSS0.Px2.p2.2.m2.1a"><mi id="A1.SS2.SSS0.Px2.p2.2.m2.1.1" xref="A1.SS2.SSS0.Px2.p2.2.m2.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px2.p2.2.m2.1b"><ci id="A1.SS2.SSS0.Px2.p2.2.m2.1.1.cmml" xref="A1.SS2.SSS0.Px2.p2.2.m2.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px2.p2.2.m2.1c">n</annotation></semantics></math><span id="A1.SS2.SSS0.Px2.p2.7.3" class="ltx_text ltx_font_typewriter"> and varies for each unit, we can consider the average unit size </span><math id="A1.SS2.SSS0.Px2.p2.3.m3.1" class="ltx_Math" alttext="\bar{m}" display="inline"><semantics id="A1.SS2.SSS0.Px2.p2.3.m3.1a"><mover accent="true" id="A1.SS2.SSS0.Px2.p2.3.m3.1.1" xref="A1.SS2.SSS0.Px2.p2.3.m3.1.1.cmml"><mi id="A1.SS2.SSS0.Px2.p2.3.m3.1.1.2" xref="A1.SS2.SSS0.Px2.p2.3.m3.1.1.2.cmml">m</mi><mo id="A1.SS2.SSS0.Px2.p2.3.m3.1.1.1" xref="A1.SS2.SSS0.Px2.p2.3.m3.1.1.1.cmml">¯</mo></mover><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px2.p2.3.m3.1b"><apply id="A1.SS2.SSS0.Px2.p2.3.m3.1.1.cmml" xref="A1.SS2.SSS0.Px2.p2.3.m3.1.1"><ci id="A1.SS2.SSS0.Px2.p2.3.m3.1.1.1.cmml" xref="A1.SS2.SSS0.Px2.p2.3.m3.1.1.1">¯</ci><ci id="A1.SS2.SSS0.Px2.p2.3.m3.1.1.2.cmml" xref="A1.SS2.SSS0.Px2.p2.3.m3.1.1.2">𝑚</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px2.p2.3.m3.1c">\bar{m}</annotation></semantics></math><span id="A1.SS2.SSS0.Px2.p2.7.4" class="ltx_text ltx_font_typewriter"> and average number of non-zero similarity edges </span><math id="A1.SS2.SSS0.Px2.p2.4.m4.1" class="ltx_Math" alttext="\bar{l}" display="inline"><semantics id="A1.SS2.SSS0.Px2.p2.4.m4.1a"><mover accent="true" id="A1.SS2.SSS0.Px2.p2.4.m4.1.1" xref="A1.SS2.SSS0.Px2.p2.4.m4.1.1.cmml"><mi id="A1.SS2.SSS0.Px2.p2.4.m4.1.1.2" xref="A1.SS2.SSS0.Px2.p2.4.m4.1.1.2.cmml">l</mi><mo id="A1.SS2.SSS0.Px2.p2.4.m4.1.1.1" xref="A1.SS2.SSS0.Px2.p2.4.m4.1.1.1.cmml">¯</mo></mover><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px2.p2.4.m4.1b"><apply id="A1.SS2.SSS0.Px2.p2.4.m4.1.1.cmml" xref="A1.SS2.SSS0.Px2.p2.4.m4.1.1"><ci id="A1.SS2.SSS0.Px2.p2.4.m4.1.1.1.cmml" xref="A1.SS2.SSS0.Px2.p2.4.m4.1.1.1">¯</ci><ci id="A1.SS2.SSS0.Px2.p2.4.m4.1.1.2.cmml" xref="A1.SS2.SSS0.Px2.p2.4.m4.1.1.2">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px2.p2.4.m4.1c">\bar{l}</annotation></semantics></math><span id="A1.SS2.SSS0.Px2.p2.7.5" class="ltx_text ltx_font_typewriter">. The total complexity for computing metrics across all units is then </span><math id="A1.SS2.SSS0.Px2.p2.5.m5.1" class="ltx_Math" alttext="\mathcal{O}(k\bar{l})" display="inline"><semantics id="A1.SS2.SSS0.Px2.p2.5.m5.1a"><mrow id="A1.SS2.SSS0.Px2.p2.5.m5.1.1" xref="A1.SS2.SSS0.Px2.p2.5.m5.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.SS2.SSS0.Px2.p2.5.m5.1.1.3" xref="A1.SS2.SSS0.Px2.p2.5.m5.1.1.3.cmml">𝒪</mi><mo lspace="0em" rspace="0em" id="A1.SS2.SSS0.Px2.p2.5.m5.1.1.2" xref="A1.SS2.SSS0.Px2.p2.5.m5.1.1.2.cmml">​</mo><mrow id="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1" xref="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.2" xref="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1.cmml">(</mo><mrow id="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1" xref="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1.cmml"><mi id="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1.2" xref="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1.2.cmml">k</mi><mo lspace="0em" rspace="0em" id="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1.1" xref="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1.1.cmml">​</mo><mover accent="true" id="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1.3" xref="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1.3.cmml"><mi id="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1.3.2" xref="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1.3.2.cmml">l</mi><mo id="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1.3.1" xref="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1.3.1.cmml">¯</mo></mover></mrow><mo stretchy="false" id="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.3" xref="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px2.p2.5.m5.1b"><apply id="A1.SS2.SSS0.Px2.p2.5.m5.1.1.cmml" xref="A1.SS2.SSS0.Px2.p2.5.m5.1.1"><times id="A1.SS2.SSS0.Px2.p2.5.m5.1.1.2.cmml" xref="A1.SS2.SSS0.Px2.p2.5.m5.1.1.2"></times><ci id="A1.SS2.SSS0.Px2.p2.5.m5.1.1.3.cmml" xref="A1.SS2.SSS0.Px2.p2.5.m5.1.1.3">𝒪</ci><apply id="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1.cmml" xref="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1"><times id="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1.1.cmml" xref="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1.1"></times><ci id="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1.2.cmml" xref="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1.2">𝑘</ci><apply id="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1.3.cmml" xref="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1.3"><ci id="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1.3.1.cmml" xref="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1.3.1">¯</ci><ci id="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1.3.2.cmml" xref="A1.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1.3.2">𝑙</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px2.p2.5.m5.1c">\mathcal{O}(k\bar{l})</annotation></semantics></math><span id="A1.SS2.SSS0.Px2.p2.7.6" class="ltx_text ltx_font_typewriter"> for modularity (which in the worst case is </span><math id="A1.SS2.SSS0.Px2.p2.6.m6.1" class="ltx_Math" alttext="\mathcal{O}(k\bar{m}^{2})" display="inline"><semantics id="A1.SS2.SSS0.Px2.p2.6.m6.1a"><mrow id="A1.SS2.SSS0.Px2.p2.6.m6.1.1" xref="A1.SS2.SSS0.Px2.p2.6.m6.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.SS2.SSS0.Px2.p2.6.m6.1.1.3" xref="A1.SS2.SSS0.Px2.p2.6.m6.1.1.3.cmml">𝒪</mi><mo lspace="0em" rspace="0em" id="A1.SS2.SSS0.Px2.p2.6.m6.1.1.2" xref="A1.SS2.SSS0.Px2.p2.6.m6.1.1.2.cmml">​</mo><mrow id="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1" xref="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.2" xref="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.cmml">(</mo><mrow id="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1" xref="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.cmml"><mi id="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.2" xref="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.2.cmml">k</mi><mo lspace="0em" rspace="0em" id="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.1" xref="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.1.cmml">​</mo><msup id="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.3" xref="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.3.cmml"><mover accent="true" id="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.3.2" xref="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.3.2.cmml"><mi id="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.3.2.2" xref="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.3.2.2.cmml">m</mi><mo id="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.3.2.1" xref="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.3.2.1.cmml">¯</mo></mover><mn id="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.3.3" xref="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.3.3.cmml">2</mn></msup></mrow><mo stretchy="false" id="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.3" xref="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px2.p2.6.m6.1b"><apply id="A1.SS2.SSS0.Px2.p2.6.m6.1.1.cmml" xref="A1.SS2.SSS0.Px2.p2.6.m6.1.1"><times id="A1.SS2.SSS0.Px2.p2.6.m6.1.1.2.cmml" xref="A1.SS2.SSS0.Px2.p2.6.m6.1.1.2"></times><ci id="A1.SS2.SSS0.Px2.p2.6.m6.1.1.3.cmml" xref="A1.SS2.SSS0.Px2.p2.6.m6.1.1.3">𝒪</ci><apply id="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.cmml" xref="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1"><times id="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.1.cmml" xref="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.1"></times><ci id="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.2.cmml" xref="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.2">𝑘</ci><apply id="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.3.cmml" xref="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.3.1.cmml" xref="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.3">superscript</csymbol><apply id="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.3.2.cmml" xref="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.3.2"><ci id="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.3.2.1.cmml" xref="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.3.2.1">¯</ci><ci id="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.3.2.2.cmml" xref="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.3.2.2">𝑚</ci></apply><cn type="integer" id="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.3.3.cmml" xref="A1.SS2.SSS0.Px2.p2.6.m6.1.1.1.1.1.3.3">2</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px2.p2.6.m6.1c">\mathcal{O}(k\bar{m}^{2})</annotation></semantics></math><span id="A1.SS2.SSS0.Px2.p2.7.7" class="ltx_text ltx_font_typewriter">, but typically much lower) or </span><math id="A1.SS2.SSS0.Px2.p2.7.m7.1" class="ltx_Math" alttext="\mathcal{O}(k\bar{m})" display="inline"><semantics id="A1.SS2.SSS0.Px2.p2.7.m7.1a"><mrow id="A1.SS2.SSS0.Px2.p2.7.m7.1.1" xref="A1.SS2.SSS0.Px2.p2.7.m7.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.SS2.SSS0.Px2.p2.7.m7.1.1.3" xref="A1.SS2.SSS0.Px2.p2.7.m7.1.1.3.cmml">𝒪</mi><mo lspace="0em" rspace="0em" id="A1.SS2.SSS0.Px2.p2.7.m7.1.1.2" xref="A1.SS2.SSS0.Px2.p2.7.m7.1.1.2.cmml">​</mo><mrow id="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1" xref="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1.2" xref="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1.1.cmml">(</mo><mrow id="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1.1" xref="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1.1.cmml"><mi id="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1.1.2" xref="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1.1.2.cmml">k</mi><mo lspace="0em" rspace="0em" id="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1.1.1" xref="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1.1.1.cmml">​</mo><mover accent="true" id="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1.1.3" xref="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1.1.3.cmml"><mi id="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1.1.3.2" xref="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1.1.3.2.cmml">m</mi><mo id="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1.1.3.1" xref="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1.1.3.1.cmml">¯</mo></mover></mrow><mo stretchy="false" id="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1.3" xref="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px2.p2.7.m7.1b"><apply id="A1.SS2.SSS0.Px2.p2.7.m7.1.1.cmml" xref="A1.SS2.SSS0.Px2.p2.7.m7.1.1"><times id="A1.SS2.SSS0.Px2.p2.7.m7.1.1.2.cmml" xref="A1.SS2.SSS0.Px2.p2.7.m7.1.1.2"></times><ci id="A1.SS2.SSS0.Px2.p2.7.m7.1.1.3.cmml" xref="A1.SS2.SSS0.Px2.p2.7.m7.1.1.3">𝒪</ci><apply id="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1.1.cmml" xref="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1"><times id="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1.1.1.cmml" xref="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1.1.1"></times><ci id="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1.1.2.cmml" xref="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1.1.2">𝑘</ci><apply id="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1.1.3.cmml" xref="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1.1.3"><ci id="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1.1.3.1.cmml" xref="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1.1.3.1">¯</ci><ci id="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1.1.3.2.cmml" xref="A1.SS2.SSS0.Px2.p2.7.m7.1.1.1.1.1.3.2">𝑚</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px2.p2.7.m7.1c">\mathcal{O}(k\bar{m})</annotation></semantics></math><span id="A1.SS2.SSS0.Px2.p2.7.8" class="ltx_text ltx_font_typewriter"> for conductance.</span></p>
</div>
</section>
<section id="A1.SS2.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_font_typewriter ltx_title_paragraph">Overall Complexity</h4>

<div id="A1.SS2.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="A1.SS2.SSS0.Px3.p1.1" class="ltx_p"><span id="A1.SS2.SSS0.Px3.p1.1.1" class="ltx_text ltx_font_typewriter">Combining the boundary refinement step and metric computation, the overall complexity is:</span></p>
</div>
<div id="A1.SS2.SSS0.Px3.p2" class="ltx_para ltx_noindent">
<p id="A1.SS2.SSS0.Px3.p2.2" class="ltx_p"><span id="A1.SS2.SSS0.Px3.p2.2.1" class="ltx_text ltx_font_typewriter">For modularity: </span><math id="A1.SS2.SSS0.Px3.p2.1.m1.1" class="ltx_Math" alttext="\mathcal{O}(kn+k\bar{m}^{2})" display="inline"><semantics id="A1.SS2.SSS0.Px3.p2.1.m1.1a"><mrow id="A1.SS2.SSS0.Px3.p2.1.m1.1.1" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.3" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.3.cmml">𝒪</mi><mo lspace="0em" rspace="0em" id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.2" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.2.cmml">​</mo><mrow id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.2" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.cmml">(</mo><mrow id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.cmml"><mrow id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.2" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.2.cmml"><mi id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.2.2" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.2.2.cmml">k</mi><mo lspace="0em" rspace="0em" id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.2.1" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.2.1.cmml">​</mo><mi id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.2.3" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.2.3.cmml">n</mi></mrow><mo id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.1" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.1.cmml">+</mo><mrow id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.cmml"><mi id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.2" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.2.cmml">k</mi><mo lspace="0em" rspace="0em" id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.1" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.1.cmml">​</mo><msup id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.3" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.3.cmml"><mover accent="true" id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.3.2" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.3.2.cmml"><mi id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.3.2.2" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.3.2.2.cmml">m</mi><mo id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.3.2.1" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.3.2.1.cmml">¯</mo></mover><mn id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.3.3" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.3.3.cmml">2</mn></msup></mrow></mrow><mo stretchy="false" id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.3" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px3.p2.1.m1.1b"><apply id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.cmml" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1"><times id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.2.cmml" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.2"></times><ci id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.3.cmml" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.3">𝒪</ci><apply id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.cmml" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1"><plus id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.1.cmml" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.1"></plus><apply id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.2.cmml" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.2"><times id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.2.1.cmml" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.2.1"></times><ci id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.2.2.cmml" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.2.2">𝑘</ci><ci id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.2.3.cmml" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.2.3">𝑛</ci></apply><apply id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.cmml" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3"><times id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.1.cmml" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.1"></times><ci id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.2.cmml" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.2">𝑘</ci><apply id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.3.cmml" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.3.1.cmml" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.3">superscript</csymbol><apply id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.3.2.cmml" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.3.2"><ci id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.3.2.1.cmml" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.3.2.1">¯</ci><ci id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.3.2.2.cmml" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.3.2.2">𝑚</ci></apply><cn type="integer" id="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.3.3.cmml" xref="A1.SS2.SSS0.Px3.p2.1.m1.1.1.1.1.1.3.3.3">2</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px3.p2.1.m1.1c">\mathcal{O}(kn+k\bar{m}^{2})</annotation></semantics></math><span id="A1.SS2.SSS0.Px3.p2.2.2" class="ltx_text ltx_font_typewriter">
For conductance: </span><math id="A1.SS2.SSS0.Px3.p2.2.m2.1" class="ltx_Math" alttext="\mathcal{O}(kn+k\bar{m})" display="inline"><semantics id="A1.SS2.SSS0.Px3.p2.2.m2.1a"><mrow id="A1.SS2.SSS0.Px3.p2.2.m2.1.1" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.3" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.3.cmml">𝒪</mi><mo lspace="0em" rspace="0em" id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.2" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.2.cmml">​</mo><mrow id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.2" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.cmml">(</mo><mrow id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.cmml"><mrow id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.2" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.2.cmml"><mi id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.2.2" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.2.2.cmml">k</mi><mo lspace="0em" rspace="0em" id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.2.1" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.2.1.cmml">​</mo><mi id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.2.3" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.2.3.cmml">n</mi></mrow><mo id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.1" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.1.cmml">+</mo><mrow id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.3" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.3.cmml"><mi id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.3.2" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.3.2.cmml">k</mi><mo lspace="0em" rspace="0em" id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.3.1" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.3.1.cmml">​</mo><mover accent="true" id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.3.3" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.3.3.cmml"><mi id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.3.3.2" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.3.3.2.cmml">m</mi><mo id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.3.3.1" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.3.3.1.cmml">¯</mo></mover></mrow></mrow><mo stretchy="false" id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.3" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px3.p2.2.m2.1b"><apply id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.cmml" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1"><times id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.2.cmml" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.2"></times><ci id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.3.cmml" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.3">𝒪</ci><apply id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.cmml" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1"><plus id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.1.cmml" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.1"></plus><apply id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.2.cmml" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.2"><times id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.2.1.cmml" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.2.1"></times><ci id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.2.2.cmml" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.2.2">𝑘</ci><ci id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.2.3.cmml" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.2.3">𝑛</ci></apply><apply id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.3.cmml" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.3"><times id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.3.1.cmml" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.3.1"></times><ci id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.3.2.cmml" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.3.2">𝑘</ci><apply id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.3.3.cmml" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.3.3"><ci id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.3.3.1.cmml" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.3.3.1">¯</ci><ci id="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.3.3.2.cmml" xref="A1.SS2.SSS0.Px3.p2.2.m2.1.1.1.1.1.3.3.2">𝑚</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px3.p2.2.m2.1c">\mathcal{O}(kn+k\bar{m})</annotation></semantics></math><span id="A1.SS2.SSS0.Px3.p2.2.3" class="ltx_text ltx_font_typewriter"></span></p>
</div>
<div id="A1.SS2.SSS0.Px3.p3" class="ltx_para ltx_noindent">
<p id="A1.SS2.SSS0.Px3.p3.3" class="ltx_p"><span id="A1.SS2.SSS0.Px3.p3.3.1" class="ltx_text ltx_font_typewriter">Since typically </span><math id="A1.SS2.SSS0.Px3.p3.1.m1.1" class="ltx_Math" alttext="\bar{m}\ll n" display="inline"><semantics id="A1.SS2.SSS0.Px3.p3.1.m1.1a"><mrow id="A1.SS2.SSS0.Px3.p3.1.m1.1.1" xref="A1.SS2.SSS0.Px3.p3.1.m1.1.1.cmml"><mover accent="true" id="A1.SS2.SSS0.Px3.p3.1.m1.1.1.2" xref="A1.SS2.SSS0.Px3.p3.1.m1.1.1.2.cmml"><mi id="A1.SS2.SSS0.Px3.p3.1.m1.1.1.2.2" xref="A1.SS2.SSS0.Px3.p3.1.m1.1.1.2.2.cmml">m</mi><mo id="A1.SS2.SSS0.Px3.p3.1.m1.1.1.2.1" xref="A1.SS2.SSS0.Px3.p3.1.m1.1.1.2.1.cmml">¯</mo></mover><mo id="A1.SS2.SSS0.Px3.p3.1.m1.1.1.1" xref="A1.SS2.SSS0.Px3.p3.1.m1.1.1.1.cmml">≪</mo><mi id="A1.SS2.SSS0.Px3.p3.1.m1.1.1.3" xref="A1.SS2.SSS0.Px3.p3.1.m1.1.1.3.cmml">n</mi></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px3.p3.1.m1.1b"><apply id="A1.SS2.SSS0.Px3.p3.1.m1.1.1.cmml" xref="A1.SS2.SSS0.Px3.p3.1.m1.1.1"><csymbol cd="latexml" id="A1.SS2.SSS0.Px3.p3.1.m1.1.1.1.cmml" xref="A1.SS2.SSS0.Px3.p3.1.m1.1.1.1">much-less-than</csymbol><apply id="A1.SS2.SSS0.Px3.p3.1.m1.1.1.2.cmml" xref="A1.SS2.SSS0.Px3.p3.1.m1.1.1.2"><ci id="A1.SS2.SSS0.Px3.p3.1.m1.1.1.2.1.cmml" xref="A1.SS2.SSS0.Px3.p3.1.m1.1.1.2.1">¯</ci><ci id="A1.SS2.SSS0.Px3.p3.1.m1.1.1.2.2.cmml" xref="A1.SS2.SSS0.Px3.p3.1.m1.1.1.2.2">𝑚</ci></apply><ci id="A1.SS2.SSS0.Px3.p3.1.m1.1.1.3.cmml" xref="A1.SS2.SSS0.Px3.p3.1.m1.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px3.p3.1.m1.1c">\bar{m}\ll n</annotation></semantics></math><span id="A1.SS2.SSS0.Px3.p3.3.2" class="ltx_text ltx_font_typewriter">, the dominant term in both cases is </span><math id="A1.SS2.SSS0.Px3.p3.2.m2.1" class="ltx_Math" alttext="\mathcal{O}(kn)" display="inline"><semantics id="A1.SS2.SSS0.Px3.p3.2.m2.1a"><mrow id="A1.SS2.SSS0.Px3.p3.2.m2.1.1" xref="A1.SS2.SSS0.Px3.p3.2.m2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.SS2.SSS0.Px3.p3.2.m2.1.1.3" xref="A1.SS2.SSS0.Px3.p3.2.m2.1.1.3.cmml">𝒪</mi><mo lspace="0em" rspace="0em" id="A1.SS2.SSS0.Px3.p3.2.m2.1.1.2" xref="A1.SS2.SSS0.Px3.p3.2.m2.1.1.2.cmml">​</mo><mrow id="A1.SS2.SSS0.Px3.p3.2.m2.1.1.1.1" xref="A1.SS2.SSS0.Px3.p3.2.m2.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.SS2.SSS0.Px3.p3.2.m2.1.1.1.1.2" xref="A1.SS2.SSS0.Px3.p3.2.m2.1.1.1.1.1.cmml">(</mo><mrow id="A1.SS2.SSS0.Px3.p3.2.m2.1.1.1.1.1" xref="A1.SS2.SSS0.Px3.p3.2.m2.1.1.1.1.1.cmml"><mi id="A1.SS2.SSS0.Px3.p3.2.m2.1.1.1.1.1.2" xref="A1.SS2.SSS0.Px3.p3.2.m2.1.1.1.1.1.2.cmml">k</mi><mo lspace="0em" rspace="0em" id="A1.SS2.SSS0.Px3.p3.2.m2.1.1.1.1.1.1" xref="A1.SS2.SSS0.Px3.p3.2.m2.1.1.1.1.1.1.cmml">​</mo><mi id="A1.SS2.SSS0.Px3.p3.2.m2.1.1.1.1.1.3" xref="A1.SS2.SSS0.Px3.p3.2.m2.1.1.1.1.1.3.cmml">n</mi></mrow><mo stretchy="false" id="A1.SS2.SSS0.Px3.p3.2.m2.1.1.1.1.3" xref="A1.SS2.SSS0.Px3.p3.2.m2.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px3.p3.2.m2.1b"><apply id="A1.SS2.SSS0.Px3.p3.2.m2.1.1.cmml" xref="A1.SS2.SSS0.Px3.p3.2.m2.1.1"><times id="A1.SS2.SSS0.Px3.p3.2.m2.1.1.2.cmml" xref="A1.SS2.SSS0.Px3.p3.2.m2.1.1.2"></times><ci id="A1.SS2.SSS0.Px3.p3.2.m2.1.1.3.cmml" xref="A1.SS2.SSS0.Px3.p3.2.m2.1.1.3">𝒪</ci><apply id="A1.SS2.SSS0.Px3.p3.2.m2.1.1.1.1.1.cmml" xref="A1.SS2.SSS0.Px3.p3.2.m2.1.1.1.1"><times id="A1.SS2.SSS0.Px3.p3.2.m2.1.1.1.1.1.1.cmml" xref="A1.SS2.SSS0.Px3.p3.2.m2.1.1.1.1.1.1"></times><ci id="A1.SS2.SSS0.Px3.p3.2.m2.1.1.1.1.1.2.cmml" xref="A1.SS2.SSS0.Px3.p3.2.m2.1.1.1.1.1.2">𝑘</ci><ci id="A1.SS2.SSS0.Px3.p3.2.m2.1.1.1.1.1.3.cmml" xref="A1.SS2.SSS0.Px3.p3.2.m2.1.1.1.1.1.3">𝑛</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px3.p3.2.m2.1c">\mathcal{O}(kn)</annotation></semantics></math><span id="A1.SS2.SSS0.Px3.p3.3.3" class="ltx_text ltx_font_typewriter">. Therefore, we express the overall complexity of our algorithm as </span><math id="A1.SS2.SSS0.Px3.p3.3.m3.1" class="ltx_Math" alttext="\mathcal{O}(kn)" display="inline"><semantics id="A1.SS2.SSS0.Px3.p3.3.m3.1a"><mrow id="A1.SS2.SSS0.Px3.p3.3.m3.1.1" xref="A1.SS2.SSS0.Px3.p3.3.m3.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.SS2.SSS0.Px3.p3.3.m3.1.1.3" xref="A1.SS2.SSS0.Px3.p3.3.m3.1.1.3.cmml">𝒪</mi><mo lspace="0em" rspace="0em" id="A1.SS2.SSS0.Px3.p3.3.m3.1.1.2" xref="A1.SS2.SSS0.Px3.p3.3.m3.1.1.2.cmml">​</mo><mrow id="A1.SS2.SSS0.Px3.p3.3.m3.1.1.1.1" xref="A1.SS2.SSS0.Px3.p3.3.m3.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.SS2.SSS0.Px3.p3.3.m3.1.1.1.1.2" xref="A1.SS2.SSS0.Px3.p3.3.m3.1.1.1.1.1.cmml">(</mo><mrow id="A1.SS2.SSS0.Px3.p3.3.m3.1.1.1.1.1" xref="A1.SS2.SSS0.Px3.p3.3.m3.1.1.1.1.1.cmml"><mi id="A1.SS2.SSS0.Px3.p3.3.m3.1.1.1.1.1.2" xref="A1.SS2.SSS0.Px3.p3.3.m3.1.1.1.1.1.2.cmml">k</mi><mo lspace="0em" rspace="0em" id="A1.SS2.SSS0.Px3.p3.3.m3.1.1.1.1.1.1" xref="A1.SS2.SSS0.Px3.p3.3.m3.1.1.1.1.1.1.cmml">​</mo><mi id="A1.SS2.SSS0.Px3.p3.3.m3.1.1.1.1.1.3" xref="A1.SS2.SSS0.Px3.p3.3.m3.1.1.1.1.1.3.cmml">n</mi></mrow><mo stretchy="false" id="A1.SS2.SSS0.Px3.p3.3.m3.1.1.1.1.3" xref="A1.SS2.SSS0.Px3.p3.3.m3.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.SSS0.Px3.p3.3.m3.1b"><apply id="A1.SS2.SSS0.Px3.p3.3.m3.1.1.cmml" xref="A1.SS2.SSS0.Px3.p3.3.m3.1.1"><times id="A1.SS2.SSS0.Px3.p3.3.m3.1.1.2.cmml" xref="A1.SS2.SSS0.Px3.p3.3.m3.1.1.2"></times><ci id="A1.SS2.SSS0.Px3.p3.3.m3.1.1.3.cmml" xref="A1.SS2.SSS0.Px3.p3.3.m3.1.1.3">𝒪</ci><apply id="A1.SS2.SSS0.Px3.p3.3.m3.1.1.1.1.1.cmml" xref="A1.SS2.SSS0.Px3.p3.3.m3.1.1.1.1"><times id="A1.SS2.SSS0.Px3.p3.3.m3.1.1.1.1.1.1.cmml" xref="A1.SS2.SSS0.Px3.p3.3.m3.1.1.1.1.1.1"></times><ci id="A1.SS2.SSS0.Px3.p3.3.m3.1.1.1.1.1.2.cmml" xref="A1.SS2.SSS0.Px3.p3.3.m3.1.1.1.1.1.2">𝑘</ci><ci id="A1.SS2.SSS0.Px3.p3.3.m3.1.1.1.1.1.3.cmml" xref="A1.SS2.SSS0.Px3.p3.3.m3.1.1.1.1.1.3">𝑛</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.SSS0.Px3.p3.3.m3.1c">\mathcal{O}(kn)</annotation></semantics></math><span id="A1.SS2.SSS0.Px3.p3.3.4" class="ltx_text ltx_font_typewriter">.</span></p>
</div>
</section>
</section>
<section id="A1.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_font_typewriter ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="A1.SS3.1.1.1" class="ltx_text ltx_font_serif">A.3</span> </span>Analysis of human data</h3>

<div id="A1.SS3.p1" class="ltx_para ltx_noindent">
<p id="A1.SS3.p1.1" class="ltx_p"><span id="A1.SS3.p1.1.1" class="ltx_text ltx_font_typewriter">The human data released as part of </span><cite class="ltx_cite ltx_citemacro_citet"><span class="ltx_text ltx_font_typewriter">Kumar et al.</span> <span id="A1.SS3.p1.1.2.1.1.1" class="ltx_text ltx_font_typewriter">[</span><a href="#bib.bib20" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2023</span></a><span id="A1.SS3.p1.1.3.2.2.1" class="ltx_text ltx_font_typewriter">]</span></cite><span id="A1.SS3.p1.1.4" class="ltx_text ltx_font_typewriter"> used Gaussian smoothing on the average signal across participants to define a probability distribution of likely event boundary positions with respect to timestamps in the podcast. In order to calculate our similarity metrics, as shown in Fig. </span><a href="#S4.F3" title="Figure 3 ‣ 4.2 Humans and LLM surprise cluster similar tokens together ‣ 4 Experiments ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">3</span></a><span id="A1.SS3.p1.1.5" class="ltx_text ltx_font_typewriter">A, we need to express this data in terms of discrete event positions with respect to tokens in the transcript. For fair comparison, we therefore identified human-annotated positions by selecting as many of the most likely positions in the distribution as our initial surprise-based event segmentation had identified in the transcript. In the same process used by </span><cite class="ltx_cite ltx_citemacro_citet"><span class="ltx_text ltx_font_typewriter">Kumar et al.</span> <span id="A1.SS3.p1.1.6.1.1.1" class="ltx_text ltx_font_typewriter">[</span><a href="#bib.bib20" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2023</span></a><span id="A1.SS3.p1.1.7.2.2.1" class="ltx_text ltx_font_typewriter">]</span></cite><span id="A1.SS3.p1.1.8" class="ltx_text ltx_font_typewriter">, we then used their provided word onset times to translate these timestamps to token positions, allowing us to calculate our similarity metrics.</span></p>
</div>
<div id="A1.SS3.p2" class="ltx_para ltx_noindent">
<p id="A1.SS3.p2.1" class="ltx_p"><span id="A1.SS3.p2.1.1" class="ltx_text ltx_font_typewriter">In Fig. </span><a href="#S4.F3" title="Figure 3 ‣ 4.2 Humans and LLM surprise cluster similar tokens together ‣ 4 Experiments ‣ Human-like Episodic Memory for Infinite Context LLMs" class="ltx_ref ltx_font_typewriter"><span class="ltx_text ltx_ref_tag">3</span></a><span id="A1.SS3.p2.1.2" class="ltx_text ltx_font_typewriter">B, we use Wasserstein distance in order to compare the relative positions of event boundaries between human annotations and those found by our own methods. Wasserstein distance is a versatile metric used to compare two probability distributions </span><cite class="ltx_cite ltx_citemacro_citep"><span id="A1.SS3.p2.1.3.1" class="ltx_text ltx_font_typewriter">[</span><span class="ltx_text ltx_font_typewriter">Panaretos and Zemel</span><span id="A1.SS3.p2.1.4.2.1.1" class="ltx_text ltx_font_typewriter">, </span><a href="#bib.bib83" title="" class="ltx_ref"><span class="ltx_text ltx_font_typewriter">2019</span></a><span id="A1.SS3.p2.1.5.3" class="ltx_text ltx_font_typewriter">]</span></cite><span id="A1.SS3.p2.1.6" class="ltx_text ltx_font_typewriter">. We used such a metric to better capture the uncertainty present in the human data, and found it to give more meaningful results than standard correlation or discrete distance metrics, which showed very little differences between methods.
In order to calculate such a metric, we therefore need to convert our own discrete boundary positions to a distribution across token positions. We did so by defining a Mixture of Gaussians (MoG), with each Gaussian corresponding to a single position. Note that, for fair comparison with human data, we apply the same process to the discrete version of the human-annotated positions described above, and use this for comparison.</span></p>
</div>
</section>
<section id="A1.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_font_typewriter ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="A1.SS4.1.1.1" class="ltx_text ltx_font_serif">A.4</span> </span>Approximate equivalence of K-nearest neighbours and softmax attention</h3>

<div id="A1.SS4.p1" class="ltx_para ltx_noindent">
<p id="A1.SS4.p1.1" class="ltx_p"><span id="A1.SS4.p1.1.1" class="ltx_text ltx_font_typewriter">Here we will attempt to show that using a k-NN retrieval in a key-value cache as part of the attention mechanism in transformers is an approximation of applying softmax attention over the entire sequence of tokens.</span></p>
</div>
<div id="A1.SS4.p2" class="ltx_para ltx_noindent">
<p id="A1.SS4.p2.7" class="ltx_p"><span id="A1.SS4.p2.7.1" class="ltx_text ltx_font_typewriter">Let </span><math id="A1.SS4.p2.1.m1.1" class="ltx_Math" alttext="q" display="inline"><semantics id="A1.SS4.p2.1.m1.1a"><mi id="A1.SS4.p2.1.m1.1.1" xref="A1.SS4.p2.1.m1.1.1.cmml">q</mi><annotation-xml encoding="MathML-Content" id="A1.SS4.p2.1.m1.1b"><ci id="A1.SS4.p2.1.m1.1.1.cmml" xref="A1.SS4.p2.1.m1.1.1">𝑞</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p2.1.m1.1c">q</annotation></semantics></math><span id="A1.SS4.p2.7.2" class="ltx_text ltx_font_typewriter"> be a query vector and </span><math id="A1.SS4.p2.2.m2.4" class="ltx_Math" alttext="K=\{k_{1},k_{2},\dots,k_{n}\}" display="inline"><semantics id="A1.SS4.p2.2.m2.4a"><mrow id="A1.SS4.p2.2.m2.4.4" xref="A1.SS4.p2.2.m2.4.4.cmml"><mi id="A1.SS4.p2.2.m2.4.4.5" xref="A1.SS4.p2.2.m2.4.4.5.cmml">K</mi><mo id="A1.SS4.p2.2.m2.4.4.4" xref="A1.SS4.p2.2.m2.4.4.4.cmml">=</mo><mrow id="A1.SS4.p2.2.m2.4.4.3.3" xref="A1.SS4.p2.2.m2.4.4.3.4.cmml"><mo stretchy="false" id="A1.SS4.p2.2.m2.4.4.3.3.4" xref="A1.SS4.p2.2.m2.4.4.3.4.cmml">{</mo><msub id="A1.SS4.p2.2.m2.2.2.1.1.1" xref="A1.SS4.p2.2.m2.2.2.1.1.1.cmml"><mi id="A1.SS4.p2.2.m2.2.2.1.1.1.2" xref="A1.SS4.p2.2.m2.2.2.1.1.1.2.cmml">k</mi><mn id="A1.SS4.p2.2.m2.2.2.1.1.1.3" xref="A1.SS4.p2.2.m2.2.2.1.1.1.3.cmml">1</mn></msub><mo id="A1.SS4.p2.2.m2.4.4.3.3.5" xref="A1.SS4.p2.2.m2.4.4.3.4.cmml">,</mo><msub id="A1.SS4.p2.2.m2.3.3.2.2.2" xref="A1.SS4.p2.2.m2.3.3.2.2.2.cmml"><mi id="A1.SS4.p2.2.m2.3.3.2.2.2.2" xref="A1.SS4.p2.2.m2.3.3.2.2.2.2.cmml">k</mi><mn id="A1.SS4.p2.2.m2.3.3.2.2.2.3" xref="A1.SS4.p2.2.m2.3.3.2.2.2.3.cmml">2</mn></msub><mo id="A1.SS4.p2.2.m2.4.4.3.3.6" xref="A1.SS4.p2.2.m2.4.4.3.4.cmml">,</mo><mi mathvariant="normal" id="A1.SS4.p2.2.m2.1.1" xref="A1.SS4.p2.2.m2.1.1.cmml">…</mi><mo id="A1.SS4.p2.2.m2.4.4.3.3.7" xref="A1.SS4.p2.2.m2.4.4.3.4.cmml">,</mo><msub id="A1.SS4.p2.2.m2.4.4.3.3.3" xref="A1.SS4.p2.2.m2.4.4.3.3.3.cmml"><mi id="A1.SS4.p2.2.m2.4.4.3.3.3.2" xref="A1.SS4.p2.2.m2.4.4.3.3.3.2.cmml">k</mi><mi id="A1.SS4.p2.2.m2.4.4.3.3.3.3" xref="A1.SS4.p2.2.m2.4.4.3.3.3.3.cmml">n</mi></msub><mo stretchy="false" id="A1.SS4.p2.2.m2.4.4.3.3.8" xref="A1.SS4.p2.2.m2.4.4.3.4.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS4.p2.2.m2.4b"><apply id="A1.SS4.p2.2.m2.4.4.cmml" xref="A1.SS4.p2.2.m2.4.4"><eq id="A1.SS4.p2.2.m2.4.4.4.cmml" xref="A1.SS4.p2.2.m2.4.4.4"></eq><ci id="A1.SS4.p2.2.m2.4.4.5.cmml" xref="A1.SS4.p2.2.m2.4.4.5">𝐾</ci><set id="A1.SS4.p2.2.m2.4.4.3.4.cmml" xref="A1.SS4.p2.2.m2.4.4.3.3"><apply id="A1.SS4.p2.2.m2.2.2.1.1.1.cmml" xref="A1.SS4.p2.2.m2.2.2.1.1.1"><csymbol cd="ambiguous" id="A1.SS4.p2.2.m2.2.2.1.1.1.1.cmml" xref="A1.SS4.p2.2.m2.2.2.1.1.1">subscript</csymbol><ci id="A1.SS4.p2.2.m2.2.2.1.1.1.2.cmml" xref="A1.SS4.p2.2.m2.2.2.1.1.1.2">𝑘</ci><cn type="integer" id="A1.SS4.p2.2.m2.2.2.1.1.1.3.cmml" xref="A1.SS4.p2.2.m2.2.2.1.1.1.3">1</cn></apply><apply id="A1.SS4.p2.2.m2.3.3.2.2.2.cmml" xref="A1.SS4.p2.2.m2.3.3.2.2.2"><csymbol cd="ambiguous" id="A1.SS4.p2.2.m2.3.3.2.2.2.1.cmml" xref="A1.SS4.p2.2.m2.3.3.2.2.2">subscript</csymbol><ci id="A1.SS4.p2.2.m2.3.3.2.2.2.2.cmml" xref="A1.SS4.p2.2.m2.3.3.2.2.2.2">𝑘</ci><cn type="integer" id="A1.SS4.p2.2.m2.3.3.2.2.2.3.cmml" xref="A1.SS4.p2.2.m2.3.3.2.2.2.3">2</cn></apply><ci id="A1.SS4.p2.2.m2.1.1.cmml" xref="A1.SS4.p2.2.m2.1.1">…</ci><apply id="A1.SS4.p2.2.m2.4.4.3.3.3.cmml" xref="A1.SS4.p2.2.m2.4.4.3.3.3"><csymbol cd="ambiguous" id="A1.SS4.p2.2.m2.4.4.3.3.3.1.cmml" xref="A1.SS4.p2.2.m2.4.4.3.3.3">subscript</csymbol><ci id="A1.SS4.p2.2.m2.4.4.3.3.3.2.cmml" xref="A1.SS4.p2.2.m2.4.4.3.3.3.2">𝑘</ci><ci id="A1.SS4.p2.2.m2.4.4.3.3.3.3.cmml" xref="A1.SS4.p2.2.m2.4.4.3.3.3.3">𝑛</ci></apply></set></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p2.2.m2.4c">K=\{k_{1},k_{2},\dots,k_{n}\}</annotation></semantics></math><span id="A1.SS4.p2.7.3" class="ltx_text ltx_font_typewriter"> the set of key vectors in a transformer model with dimensionality </span><math id="A1.SS4.p2.3.m3.1" class="ltx_Math" alttext="d" display="inline"><semantics id="A1.SS4.p2.3.m3.1a"><mi id="A1.SS4.p2.3.m3.1.1" xref="A1.SS4.p2.3.m3.1.1.cmml">d</mi><annotation-xml encoding="MathML-Content" id="A1.SS4.p2.3.m3.1b"><ci id="A1.SS4.p2.3.m3.1.1.cmml" xref="A1.SS4.p2.3.m3.1.1">𝑑</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p2.3.m3.1c">d</annotation></semantics></math><span id="A1.SS4.p2.7.4" class="ltx_text ltx_font_typewriter">. Each key </span><math id="A1.SS4.p2.4.m4.1" class="ltx_Math" alttext="k_{i}" display="inline"><semantics id="A1.SS4.p2.4.m4.1a"><msub id="A1.SS4.p2.4.m4.1.1" xref="A1.SS4.p2.4.m4.1.1.cmml"><mi id="A1.SS4.p2.4.m4.1.1.2" xref="A1.SS4.p2.4.m4.1.1.2.cmml">k</mi><mi id="A1.SS4.p2.4.m4.1.1.3" xref="A1.SS4.p2.4.m4.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="A1.SS4.p2.4.m4.1b"><apply id="A1.SS4.p2.4.m4.1.1.cmml" xref="A1.SS4.p2.4.m4.1.1"><csymbol cd="ambiguous" id="A1.SS4.p2.4.m4.1.1.1.cmml" xref="A1.SS4.p2.4.m4.1.1">subscript</csymbol><ci id="A1.SS4.p2.4.m4.1.1.2.cmml" xref="A1.SS4.p2.4.m4.1.1.2">𝑘</ci><ci id="A1.SS4.p2.4.m4.1.1.3.cmml" xref="A1.SS4.p2.4.m4.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p2.4.m4.1c">k_{i}</annotation></semantics></math><span id="A1.SS4.p2.7.5" class="ltx_text ltx_font_typewriter"> has a corresponding value vector </span><math id="A1.SS4.p2.5.m5.1" class="ltx_Math" alttext="v_{i}" display="inline"><semantics id="A1.SS4.p2.5.m5.1a"><msub id="A1.SS4.p2.5.m5.1.1" xref="A1.SS4.p2.5.m5.1.1.cmml"><mi id="A1.SS4.p2.5.m5.1.1.2" xref="A1.SS4.p2.5.m5.1.1.2.cmml">v</mi><mi id="A1.SS4.p2.5.m5.1.1.3" xref="A1.SS4.p2.5.m5.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="A1.SS4.p2.5.m5.1b"><apply id="A1.SS4.p2.5.m5.1.1.cmml" xref="A1.SS4.p2.5.m5.1.1"><csymbol cd="ambiguous" id="A1.SS4.p2.5.m5.1.1.1.cmml" xref="A1.SS4.p2.5.m5.1.1">subscript</csymbol><ci id="A1.SS4.p2.5.m5.1.1.2.cmml" xref="A1.SS4.p2.5.m5.1.1.2">𝑣</ci><ci id="A1.SS4.p2.5.m5.1.1.3.cmml" xref="A1.SS4.p2.5.m5.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p2.5.m5.1c">v_{i}</annotation></semantics></math><span id="A1.SS4.p2.7.6" class="ltx_text ltx_font_typewriter">, with </span><math id="A1.SS4.p2.6.m6.4" class="ltx_Math" alttext="V=\{v_{1},v_{2},\dots,v_{n}\}" display="inline"><semantics id="A1.SS4.p2.6.m6.4a"><mrow id="A1.SS4.p2.6.m6.4.4" xref="A1.SS4.p2.6.m6.4.4.cmml"><mi id="A1.SS4.p2.6.m6.4.4.5" xref="A1.SS4.p2.6.m6.4.4.5.cmml">V</mi><mo id="A1.SS4.p2.6.m6.4.4.4" xref="A1.SS4.p2.6.m6.4.4.4.cmml">=</mo><mrow id="A1.SS4.p2.6.m6.4.4.3.3" xref="A1.SS4.p2.6.m6.4.4.3.4.cmml"><mo stretchy="false" id="A1.SS4.p2.6.m6.4.4.3.3.4" xref="A1.SS4.p2.6.m6.4.4.3.4.cmml">{</mo><msub id="A1.SS4.p2.6.m6.2.2.1.1.1" xref="A1.SS4.p2.6.m6.2.2.1.1.1.cmml"><mi id="A1.SS4.p2.6.m6.2.2.1.1.1.2" xref="A1.SS4.p2.6.m6.2.2.1.1.1.2.cmml">v</mi><mn id="A1.SS4.p2.6.m6.2.2.1.1.1.3" xref="A1.SS4.p2.6.m6.2.2.1.1.1.3.cmml">1</mn></msub><mo id="A1.SS4.p2.6.m6.4.4.3.3.5" xref="A1.SS4.p2.6.m6.4.4.3.4.cmml">,</mo><msub id="A1.SS4.p2.6.m6.3.3.2.2.2" xref="A1.SS4.p2.6.m6.3.3.2.2.2.cmml"><mi id="A1.SS4.p2.6.m6.3.3.2.2.2.2" xref="A1.SS4.p2.6.m6.3.3.2.2.2.2.cmml">v</mi><mn id="A1.SS4.p2.6.m6.3.3.2.2.2.3" xref="A1.SS4.p2.6.m6.3.3.2.2.2.3.cmml">2</mn></msub><mo id="A1.SS4.p2.6.m6.4.4.3.3.6" xref="A1.SS4.p2.6.m6.4.4.3.4.cmml">,</mo><mi mathvariant="normal" id="A1.SS4.p2.6.m6.1.1" xref="A1.SS4.p2.6.m6.1.1.cmml">…</mi><mo id="A1.SS4.p2.6.m6.4.4.3.3.7" xref="A1.SS4.p2.6.m6.4.4.3.4.cmml">,</mo><msub id="A1.SS4.p2.6.m6.4.4.3.3.3" xref="A1.SS4.p2.6.m6.4.4.3.3.3.cmml"><mi id="A1.SS4.p2.6.m6.4.4.3.3.3.2" xref="A1.SS4.p2.6.m6.4.4.3.3.3.2.cmml">v</mi><mi id="A1.SS4.p2.6.m6.4.4.3.3.3.3" xref="A1.SS4.p2.6.m6.4.4.3.3.3.3.cmml">n</mi></msub><mo stretchy="false" id="A1.SS4.p2.6.m6.4.4.3.3.8" xref="A1.SS4.p2.6.m6.4.4.3.4.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS4.p2.6.m6.4b"><apply id="A1.SS4.p2.6.m6.4.4.cmml" xref="A1.SS4.p2.6.m6.4.4"><eq id="A1.SS4.p2.6.m6.4.4.4.cmml" xref="A1.SS4.p2.6.m6.4.4.4"></eq><ci id="A1.SS4.p2.6.m6.4.4.5.cmml" xref="A1.SS4.p2.6.m6.4.4.5">𝑉</ci><set id="A1.SS4.p2.6.m6.4.4.3.4.cmml" xref="A1.SS4.p2.6.m6.4.4.3.3"><apply id="A1.SS4.p2.6.m6.2.2.1.1.1.cmml" xref="A1.SS4.p2.6.m6.2.2.1.1.1"><csymbol cd="ambiguous" id="A1.SS4.p2.6.m6.2.2.1.1.1.1.cmml" xref="A1.SS4.p2.6.m6.2.2.1.1.1">subscript</csymbol><ci id="A1.SS4.p2.6.m6.2.2.1.1.1.2.cmml" xref="A1.SS4.p2.6.m6.2.2.1.1.1.2">𝑣</ci><cn type="integer" id="A1.SS4.p2.6.m6.2.2.1.1.1.3.cmml" xref="A1.SS4.p2.6.m6.2.2.1.1.1.3">1</cn></apply><apply id="A1.SS4.p2.6.m6.3.3.2.2.2.cmml" xref="A1.SS4.p2.6.m6.3.3.2.2.2"><csymbol cd="ambiguous" id="A1.SS4.p2.6.m6.3.3.2.2.2.1.cmml" xref="A1.SS4.p2.6.m6.3.3.2.2.2">subscript</csymbol><ci id="A1.SS4.p2.6.m6.3.3.2.2.2.2.cmml" xref="A1.SS4.p2.6.m6.3.3.2.2.2.2">𝑣</ci><cn type="integer" id="A1.SS4.p2.6.m6.3.3.2.2.2.3.cmml" xref="A1.SS4.p2.6.m6.3.3.2.2.2.3">2</cn></apply><ci id="A1.SS4.p2.6.m6.1.1.cmml" xref="A1.SS4.p2.6.m6.1.1">…</ci><apply id="A1.SS4.p2.6.m6.4.4.3.3.3.cmml" xref="A1.SS4.p2.6.m6.4.4.3.3.3"><csymbol cd="ambiguous" id="A1.SS4.p2.6.m6.4.4.3.3.3.1.cmml" xref="A1.SS4.p2.6.m6.4.4.3.3.3">subscript</csymbol><ci id="A1.SS4.p2.6.m6.4.4.3.3.3.2.cmml" xref="A1.SS4.p2.6.m6.4.4.3.3.3.2">𝑣</ci><ci id="A1.SS4.p2.6.m6.4.4.3.3.3.3.cmml" xref="A1.SS4.p2.6.m6.4.4.3.3.3.3">𝑛</ci></apply></set></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p2.6.m6.4c">V=\{v_{1},v_{2},\dots,v_{n}\}</annotation></semantics></math><span id="A1.SS4.p2.7.7" class="ltx_text ltx_font_typewriter">. The softmax attention weights </span><math id="A1.SS4.p2.7.m7.1" class="ltx_Math" alttext="a_{i}" display="inline"><semantics id="A1.SS4.p2.7.m7.1a"><msub id="A1.SS4.p2.7.m7.1.1" xref="A1.SS4.p2.7.m7.1.1.cmml"><mi id="A1.SS4.p2.7.m7.1.1.2" xref="A1.SS4.p2.7.m7.1.1.2.cmml">a</mi><mi id="A1.SS4.p2.7.m7.1.1.3" xref="A1.SS4.p2.7.m7.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="A1.SS4.p2.7.m7.1b"><apply id="A1.SS4.p2.7.m7.1.1.cmml" xref="A1.SS4.p2.7.m7.1.1"><csymbol cd="ambiguous" id="A1.SS4.p2.7.m7.1.1.1.cmml" xref="A1.SS4.p2.7.m7.1.1">subscript</csymbol><ci id="A1.SS4.p2.7.m7.1.1.2.cmml" xref="A1.SS4.p2.7.m7.1.1.2">𝑎</ci><ci id="A1.SS4.p2.7.m7.1.1.3.cmml" xref="A1.SS4.p2.7.m7.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p2.7.m7.1c">a_{i}</annotation></semantics></math><span id="A1.SS4.p2.7.8" class="ltx_text ltx_font_typewriter"> are defined as:</span></p>
<table id="A1.E6" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.E6.m1.4" class="ltx_Math" alttext="a_{i}=\frac{\exp(q\cdot k_{i}~{}d^{-\frac{1}{2}})}{\sum_{j=1}^{n}\exp(q\cdot k_{j}~{}d^{-\frac{1}{2}})}" display="block"><semantics id="A1.E6.m1.4a"><mrow id="A1.E6.m1.4.5" xref="A1.E6.m1.4.5.cmml"><msub id="A1.E6.m1.4.5.2" xref="A1.E6.m1.4.5.2.cmml"><mi id="A1.E6.m1.4.5.2.2" xref="A1.E6.m1.4.5.2.2.cmml">a</mi><mi id="A1.E6.m1.4.5.2.3" xref="A1.E6.m1.4.5.2.3.cmml">i</mi></msub><mo id="A1.E6.m1.4.5.1" xref="A1.E6.m1.4.5.1.cmml">=</mo><mfrac id="A1.E6.m1.4.4" xref="A1.E6.m1.4.4.cmml"><mrow id="A1.E6.m1.2.2.2.2" xref="A1.E6.m1.2.2.2.3.cmml"><mi id="A1.E6.m1.1.1.1.1" xref="A1.E6.m1.1.1.1.1.cmml">exp</mi><mo id="A1.E6.m1.2.2.2.2a" xref="A1.E6.m1.2.2.2.3.cmml">⁡</mo><mrow id="A1.E6.m1.2.2.2.2.1" xref="A1.E6.m1.2.2.2.3.cmml"><mo stretchy="false" id="A1.E6.m1.2.2.2.2.1.2" xref="A1.E6.m1.2.2.2.3.cmml">(</mo><mrow id="A1.E6.m1.2.2.2.2.1.1" xref="A1.E6.m1.2.2.2.2.1.1.cmml"><mrow id="A1.E6.m1.2.2.2.2.1.1.2" xref="A1.E6.m1.2.2.2.2.1.1.2.cmml"><mi id="A1.E6.m1.2.2.2.2.1.1.2.2" xref="A1.E6.m1.2.2.2.2.1.1.2.2.cmml">q</mi><mo lspace="0.222em" rspace="0.222em" id="A1.E6.m1.2.2.2.2.1.1.2.1" xref="A1.E6.m1.2.2.2.2.1.1.2.1.cmml">⋅</mo><msub id="A1.E6.m1.2.2.2.2.1.1.2.3" xref="A1.E6.m1.2.2.2.2.1.1.2.3.cmml"><mi id="A1.E6.m1.2.2.2.2.1.1.2.3.2" xref="A1.E6.m1.2.2.2.2.1.1.2.3.2.cmml">k</mi><mi id="A1.E6.m1.2.2.2.2.1.1.2.3.3" xref="A1.E6.m1.2.2.2.2.1.1.2.3.3.cmml">i</mi></msub></mrow><mo lspace="0em" rspace="0em" id="A1.E6.m1.2.2.2.2.1.1.1" xref="A1.E6.m1.2.2.2.2.1.1.1.cmml">​</mo><msup id="A1.E6.m1.2.2.2.2.1.1.3" xref="A1.E6.m1.2.2.2.2.1.1.3.cmml"><mi id="A1.E6.m1.2.2.2.2.1.1.3.2" xref="A1.E6.m1.2.2.2.2.1.1.3.2.cmml">d</mi><mrow id="A1.E6.m1.2.2.2.2.1.1.3.3" xref="A1.E6.m1.2.2.2.2.1.1.3.3.cmml"><mo id="A1.E6.m1.2.2.2.2.1.1.3.3a" xref="A1.E6.m1.2.2.2.2.1.1.3.3.cmml">−</mo><mfrac id="A1.E6.m1.2.2.2.2.1.1.3.3.2" xref="A1.E6.m1.2.2.2.2.1.1.3.3.2.cmml"><mn id="A1.E6.m1.2.2.2.2.1.1.3.3.2.2" xref="A1.E6.m1.2.2.2.2.1.1.3.3.2.2.cmml">1</mn><mn id="A1.E6.m1.2.2.2.2.1.1.3.3.2.3" xref="A1.E6.m1.2.2.2.2.1.1.3.3.2.3.cmml">2</mn></mfrac></mrow></msup></mrow><mo stretchy="false" id="A1.E6.m1.2.2.2.2.1.3" xref="A1.E6.m1.2.2.2.3.cmml">)</mo></mrow></mrow><mrow id="A1.E6.m1.4.4.4" xref="A1.E6.m1.4.4.4.cmml"><msubsup id="A1.E6.m1.4.4.4.3" xref="A1.E6.m1.4.4.4.3.cmml"><mo id="A1.E6.m1.4.4.4.3.2.2" xref="A1.E6.m1.4.4.4.3.2.2.cmml">∑</mo><mrow id="A1.E6.m1.4.4.4.3.2.3" xref="A1.E6.m1.4.4.4.3.2.3.cmml"><mi id="A1.E6.m1.4.4.4.3.2.3.2" xref="A1.E6.m1.4.4.4.3.2.3.2.cmml">j</mi><mo id="A1.E6.m1.4.4.4.3.2.3.1" xref="A1.E6.m1.4.4.4.3.2.3.1.cmml">=</mo><mn id="A1.E6.m1.4.4.4.3.2.3.3" xref="A1.E6.m1.4.4.4.3.2.3.3.cmml">1</mn></mrow><mi id="A1.E6.m1.4.4.4.3.3" xref="A1.E6.m1.4.4.4.3.3.cmml">n</mi></msubsup><mrow id="A1.E6.m1.4.4.4.2.1" xref="A1.E6.m1.4.4.4.2.2.cmml"><mi id="A1.E6.m1.3.3.3.1" xref="A1.E6.m1.3.3.3.1.cmml">exp</mi><mo id="A1.E6.m1.4.4.4.2.1a" xref="A1.E6.m1.4.4.4.2.2.cmml">⁡</mo><mrow id="A1.E6.m1.4.4.4.2.1.1" xref="A1.E6.m1.4.4.4.2.2.cmml"><mo stretchy="false" id="A1.E6.m1.4.4.4.2.1.1.2" xref="A1.E6.m1.4.4.4.2.2.cmml">(</mo><mrow id="A1.E6.m1.4.4.4.2.1.1.1" xref="A1.E6.m1.4.4.4.2.1.1.1.cmml"><mrow id="A1.E6.m1.4.4.4.2.1.1.1.2" xref="A1.E6.m1.4.4.4.2.1.1.1.2.cmml"><mi id="A1.E6.m1.4.4.4.2.1.1.1.2.2" xref="A1.E6.m1.4.4.4.2.1.1.1.2.2.cmml">q</mi><mo lspace="0.222em" rspace="0.222em" id="A1.E6.m1.4.4.4.2.1.1.1.2.1" xref="A1.E6.m1.4.4.4.2.1.1.1.2.1.cmml">⋅</mo><msub id="A1.E6.m1.4.4.4.2.1.1.1.2.3" xref="A1.E6.m1.4.4.4.2.1.1.1.2.3.cmml"><mi id="A1.E6.m1.4.4.4.2.1.1.1.2.3.2" xref="A1.E6.m1.4.4.4.2.1.1.1.2.3.2.cmml">k</mi><mi id="A1.E6.m1.4.4.4.2.1.1.1.2.3.3" xref="A1.E6.m1.4.4.4.2.1.1.1.2.3.3.cmml">j</mi></msub></mrow><mo lspace="0em" rspace="0em" id="A1.E6.m1.4.4.4.2.1.1.1.1" xref="A1.E6.m1.4.4.4.2.1.1.1.1.cmml">​</mo><msup id="A1.E6.m1.4.4.4.2.1.1.1.3" xref="A1.E6.m1.4.4.4.2.1.1.1.3.cmml"><mi id="A1.E6.m1.4.4.4.2.1.1.1.3.2" xref="A1.E6.m1.4.4.4.2.1.1.1.3.2.cmml">d</mi><mrow id="A1.E6.m1.4.4.4.2.1.1.1.3.3" xref="A1.E6.m1.4.4.4.2.1.1.1.3.3.cmml"><mo id="A1.E6.m1.4.4.4.2.1.1.1.3.3a" xref="A1.E6.m1.4.4.4.2.1.1.1.3.3.cmml">−</mo><mfrac id="A1.E6.m1.4.4.4.2.1.1.1.3.3.2" xref="A1.E6.m1.4.4.4.2.1.1.1.3.3.2.cmml"><mn id="A1.E6.m1.4.4.4.2.1.1.1.3.3.2.2" xref="A1.E6.m1.4.4.4.2.1.1.1.3.3.2.2.cmml">1</mn><mn id="A1.E6.m1.4.4.4.2.1.1.1.3.3.2.3" xref="A1.E6.m1.4.4.4.2.1.1.1.3.3.2.3.cmml">2</mn></mfrac></mrow></msup></mrow><mo stretchy="false" id="A1.E6.m1.4.4.4.2.1.1.3" xref="A1.E6.m1.4.4.4.2.2.cmml">)</mo></mrow></mrow></mrow></mfrac></mrow><annotation-xml encoding="MathML-Content" id="A1.E6.m1.4b"><apply id="A1.E6.m1.4.5.cmml" xref="A1.E6.m1.4.5"><eq id="A1.E6.m1.4.5.1.cmml" xref="A1.E6.m1.4.5.1"></eq><apply id="A1.E6.m1.4.5.2.cmml" xref="A1.E6.m1.4.5.2"><csymbol cd="ambiguous" id="A1.E6.m1.4.5.2.1.cmml" xref="A1.E6.m1.4.5.2">subscript</csymbol><ci id="A1.E6.m1.4.5.2.2.cmml" xref="A1.E6.m1.4.5.2.2">𝑎</ci><ci id="A1.E6.m1.4.5.2.3.cmml" xref="A1.E6.m1.4.5.2.3">𝑖</ci></apply><apply id="A1.E6.m1.4.4.cmml" xref="A1.E6.m1.4.4"><divide id="A1.E6.m1.4.4.5.cmml" xref="A1.E6.m1.4.4"></divide><apply id="A1.E6.m1.2.2.2.3.cmml" xref="A1.E6.m1.2.2.2.2"><exp id="A1.E6.m1.1.1.1.1.cmml" xref="A1.E6.m1.1.1.1.1"></exp><apply id="A1.E6.m1.2.2.2.2.1.1.cmml" xref="A1.E6.m1.2.2.2.2.1.1"><times id="A1.E6.m1.2.2.2.2.1.1.1.cmml" xref="A1.E6.m1.2.2.2.2.1.1.1"></times><apply id="A1.E6.m1.2.2.2.2.1.1.2.cmml" xref="A1.E6.m1.2.2.2.2.1.1.2"><ci id="A1.E6.m1.2.2.2.2.1.1.2.1.cmml" xref="A1.E6.m1.2.2.2.2.1.1.2.1">⋅</ci><ci id="A1.E6.m1.2.2.2.2.1.1.2.2.cmml" xref="A1.E6.m1.2.2.2.2.1.1.2.2">𝑞</ci><apply id="A1.E6.m1.2.2.2.2.1.1.2.3.cmml" xref="A1.E6.m1.2.2.2.2.1.1.2.3"><csymbol cd="ambiguous" id="A1.E6.m1.2.2.2.2.1.1.2.3.1.cmml" xref="A1.E6.m1.2.2.2.2.1.1.2.3">subscript</csymbol><ci id="A1.E6.m1.2.2.2.2.1.1.2.3.2.cmml" xref="A1.E6.m1.2.2.2.2.1.1.2.3.2">𝑘</ci><ci id="A1.E6.m1.2.2.2.2.1.1.2.3.3.cmml" xref="A1.E6.m1.2.2.2.2.1.1.2.3.3">𝑖</ci></apply></apply><apply id="A1.E6.m1.2.2.2.2.1.1.3.cmml" xref="A1.E6.m1.2.2.2.2.1.1.3"><csymbol cd="ambiguous" id="A1.E6.m1.2.2.2.2.1.1.3.1.cmml" xref="A1.E6.m1.2.2.2.2.1.1.3">superscript</csymbol><ci id="A1.E6.m1.2.2.2.2.1.1.3.2.cmml" xref="A1.E6.m1.2.2.2.2.1.1.3.2">𝑑</ci><apply id="A1.E6.m1.2.2.2.2.1.1.3.3.cmml" xref="A1.E6.m1.2.2.2.2.1.1.3.3"><minus id="A1.E6.m1.2.2.2.2.1.1.3.3.1.cmml" xref="A1.E6.m1.2.2.2.2.1.1.3.3"></minus><apply id="A1.E6.m1.2.2.2.2.1.1.3.3.2.cmml" xref="A1.E6.m1.2.2.2.2.1.1.3.3.2"><divide id="A1.E6.m1.2.2.2.2.1.1.3.3.2.1.cmml" xref="A1.E6.m1.2.2.2.2.1.1.3.3.2"></divide><cn type="integer" id="A1.E6.m1.2.2.2.2.1.1.3.3.2.2.cmml" xref="A1.E6.m1.2.2.2.2.1.1.3.3.2.2">1</cn><cn type="integer" id="A1.E6.m1.2.2.2.2.1.1.3.3.2.3.cmml" xref="A1.E6.m1.2.2.2.2.1.1.3.3.2.3">2</cn></apply></apply></apply></apply></apply><apply id="A1.E6.m1.4.4.4.cmml" xref="A1.E6.m1.4.4.4"><apply id="A1.E6.m1.4.4.4.3.cmml" xref="A1.E6.m1.4.4.4.3"><csymbol cd="ambiguous" id="A1.E6.m1.4.4.4.3.1.cmml" xref="A1.E6.m1.4.4.4.3">superscript</csymbol><apply id="A1.E6.m1.4.4.4.3.2.cmml" xref="A1.E6.m1.4.4.4.3"><csymbol cd="ambiguous" id="A1.E6.m1.4.4.4.3.2.1.cmml" xref="A1.E6.m1.4.4.4.3">subscript</csymbol><sum id="A1.E6.m1.4.4.4.3.2.2.cmml" xref="A1.E6.m1.4.4.4.3.2.2"></sum><apply id="A1.E6.m1.4.4.4.3.2.3.cmml" xref="A1.E6.m1.4.4.4.3.2.3"><eq id="A1.E6.m1.4.4.4.3.2.3.1.cmml" xref="A1.E6.m1.4.4.4.3.2.3.1"></eq><ci id="A1.E6.m1.4.4.4.3.2.3.2.cmml" xref="A1.E6.m1.4.4.4.3.2.3.2">𝑗</ci><cn type="integer" id="A1.E6.m1.4.4.4.3.2.3.3.cmml" xref="A1.E6.m1.4.4.4.3.2.3.3">1</cn></apply></apply><ci id="A1.E6.m1.4.4.4.3.3.cmml" xref="A1.E6.m1.4.4.4.3.3">𝑛</ci></apply><apply id="A1.E6.m1.4.4.4.2.2.cmml" xref="A1.E6.m1.4.4.4.2.1"><exp id="A1.E6.m1.3.3.3.1.cmml" xref="A1.E6.m1.3.3.3.1"></exp><apply id="A1.E6.m1.4.4.4.2.1.1.1.cmml" xref="A1.E6.m1.4.4.4.2.1.1.1"><times id="A1.E6.m1.4.4.4.2.1.1.1.1.cmml" xref="A1.E6.m1.4.4.4.2.1.1.1.1"></times><apply id="A1.E6.m1.4.4.4.2.1.1.1.2.cmml" xref="A1.E6.m1.4.4.4.2.1.1.1.2"><ci id="A1.E6.m1.4.4.4.2.1.1.1.2.1.cmml" xref="A1.E6.m1.4.4.4.2.1.1.1.2.1">⋅</ci><ci id="A1.E6.m1.4.4.4.2.1.1.1.2.2.cmml" xref="A1.E6.m1.4.4.4.2.1.1.1.2.2">𝑞</ci><apply id="A1.E6.m1.4.4.4.2.1.1.1.2.3.cmml" xref="A1.E6.m1.4.4.4.2.1.1.1.2.3"><csymbol cd="ambiguous" id="A1.E6.m1.4.4.4.2.1.1.1.2.3.1.cmml" xref="A1.E6.m1.4.4.4.2.1.1.1.2.3">subscript</csymbol><ci id="A1.E6.m1.4.4.4.2.1.1.1.2.3.2.cmml" xref="A1.E6.m1.4.4.4.2.1.1.1.2.3.2">𝑘</ci><ci id="A1.E6.m1.4.4.4.2.1.1.1.2.3.3.cmml" xref="A1.E6.m1.4.4.4.2.1.1.1.2.3.3">𝑗</ci></apply></apply><apply id="A1.E6.m1.4.4.4.2.1.1.1.3.cmml" xref="A1.E6.m1.4.4.4.2.1.1.1.3"><csymbol cd="ambiguous" id="A1.E6.m1.4.4.4.2.1.1.1.3.1.cmml" xref="A1.E6.m1.4.4.4.2.1.1.1.3">superscript</csymbol><ci id="A1.E6.m1.4.4.4.2.1.1.1.3.2.cmml" xref="A1.E6.m1.4.4.4.2.1.1.1.3.2">𝑑</ci><apply id="A1.E6.m1.4.4.4.2.1.1.1.3.3.cmml" xref="A1.E6.m1.4.4.4.2.1.1.1.3.3"><minus id="A1.E6.m1.4.4.4.2.1.1.1.3.3.1.cmml" xref="A1.E6.m1.4.4.4.2.1.1.1.3.3"></minus><apply id="A1.E6.m1.4.4.4.2.1.1.1.3.3.2.cmml" xref="A1.E6.m1.4.4.4.2.1.1.1.3.3.2"><divide id="A1.E6.m1.4.4.4.2.1.1.1.3.3.2.1.cmml" xref="A1.E6.m1.4.4.4.2.1.1.1.3.3.2"></divide><cn type="integer" id="A1.E6.m1.4.4.4.2.1.1.1.3.3.2.2.cmml" xref="A1.E6.m1.4.4.4.2.1.1.1.3.3.2.2">1</cn><cn type="integer" id="A1.E6.m1.4.4.4.2.1.1.1.3.3.2.3.cmml" xref="A1.E6.m1.4.4.4.2.1.1.1.3.3.2.3">2</cn></apply></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E6.m1.4c">a_{i}=\frac{\exp(q\cdot k_{i}~{}d^{-\frac{1}{2}})}{\sum_{j=1}^{n}\exp(q\cdot k_{j}~{}d^{-\frac{1}{2}})}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(6)</span></td>
</tr></tbody>
</table>
<p id="A1.SS4.p2.8" class="ltx_p"><span id="A1.SS4.p2.8.1" class="ltx_text ltx_font_typewriter">The output vector </span><math id="A1.SS4.p2.8.m1.1" class="ltx_Math" alttext="u" display="inline"><semantics id="A1.SS4.p2.8.m1.1a"><mi id="A1.SS4.p2.8.m1.1.1" xref="A1.SS4.p2.8.m1.1.1.cmml">u</mi><annotation-xml encoding="MathML-Content" id="A1.SS4.p2.8.m1.1b"><ci id="A1.SS4.p2.8.m1.1.1.cmml" xref="A1.SS4.p2.8.m1.1.1">𝑢</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p2.8.m1.1c">u</annotation></semantics></math><span id="A1.SS4.p2.8.2" class="ltx_text ltx_font_typewriter"> is computed as:</span></p>
<table id="A1.E7" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.E7.m1.1" class="ltx_Math" alttext="u=\sum_{i=1}^{n}a_{i}v_{i}" display="block"><semantics id="A1.E7.m1.1a"><mrow id="A1.E7.m1.1.1" xref="A1.E7.m1.1.1.cmml"><mi id="A1.E7.m1.1.1.2" xref="A1.E7.m1.1.1.2.cmml">u</mi><mo rspace="0.111em" id="A1.E7.m1.1.1.1" xref="A1.E7.m1.1.1.1.cmml">=</mo><mrow id="A1.E7.m1.1.1.3" xref="A1.E7.m1.1.1.3.cmml"><munderover id="A1.E7.m1.1.1.3.1" xref="A1.E7.m1.1.1.3.1.cmml"><mo movablelimits="false" id="A1.E7.m1.1.1.3.1.2.2" xref="A1.E7.m1.1.1.3.1.2.2.cmml">∑</mo><mrow id="A1.E7.m1.1.1.3.1.2.3" xref="A1.E7.m1.1.1.3.1.2.3.cmml"><mi id="A1.E7.m1.1.1.3.1.2.3.2" xref="A1.E7.m1.1.1.3.1.2.3.2.cmml">i</mi><mo id="A1.E7.m1.1.1.3.1.2.3.1" xref="A1.E7.m1.1.1.3.1.2.3.1.cmml">=</mo><mn id="A1.E7.m1.1.1.3.1.2.3.3" xref="A1.E7.m1.1.1.3.1.2.3.3.cmml">1</mn></mrow><mi id="A1.E7.m1.1.1.3.1.3" xref="A1.E7.m1.1.1.3.1.3.cmml">n</mi></munderover><mrow id="A1.E7.m1.1.1.3.2" xref="A1.E7.m1.1.1.3.2.cmml"><msub id="A1.E7.m1.1.1.3.2.2" xref="A1.E7.m1.1.1.3.2.2.cmml"><mi id="A1.E7.m1.1.1.3.2.2.2" xref="A1.E7.m1.1.1.3.2.2.2.cmml">a</mi><mi id="A1.E7.m1.1.1.3.2.2.3" xref="A1.E7.m1.1.1.3.2.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="A1.E7.m1.1.1.3.2.1" xref="A1.E7.m1.1.1.3.2.1.cmml">​</mo><msub id="A1.E7.m1.1.1.3.2.3" xref="A1.E7.m1.1.1.3.2.3.cmml"><mi id="A1.E7.m1.1.1.3.2.3.2" xref="A1.E7.m1.1.1.3.2.3.2.cmml">v</mi><mi id="A1.E7.m1.1.1.3.2.3.3" xref="A1.E7.m1.1.1.3.2.3.3.cmml">i</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E7.m1.1b"><apply id="A1.E7.m1.1.1.cmml" xref="A1.E7.m1.1.1"><eq id="A1.E7.m1.1.1.1.cmml" xref="A1.E7.m1.1.1.1"></eq><ci id="A1.E7.m1.1.1.2.cmml" xref="A1.E7.m1.1.1.2">𝑢</ci><apply id="A1.E7.m1.1.1.3.cmml" xref="A1.E7.m1.1.1.3"><apply id="A1.E7.m1.1.1.3.1.cmml" xref="A1.E7.m1.1.1.3.1"><csymbol cd="ambiguous" id="A1.E7.m1.1.1.3.1.1.cmml" xref="A1.E7.m1.1.1.3.1">superscript</csymbol><apply id="A1.E7.m1.1.1.3.1.2.cmml" xref="A1.E7.m1.1.1.3.1"><csymbol cd="ambiguous" id="A1.E7.m1.1.1.3.1.2.1.cmml" xref="A1.E7.m1.1.1.3.1">subscript</csymbol><sum id="A1.E7.m1.1.1.3.1.2.2.cmml" xref="A1.E7.m1.1.1.3.1.2.2"></sum><apply id="A1.E7.m1.1.1.3.1.2.3.cmml" xref="A1.E7.m1.1.1.3.1.2.3"><eq id="A1.E7.m1.1.1.3.1.2.3.1.cmml" xref="A1.E7.m1.1.1.3.1.2.3.1"></eq><ci id="A1.E7.m1.1.1.3.1.2.3.2.cmml" xref="A1.E7.m1.1.1.3.1.2.3.2">𝑖</ci><cn type="integer" id="A1.E7.m1.1.1.3.1.2.3.3.cmml" xref="A1.E7.m1.1.1.3.1.2.3.3">1</cn></apply></apply><ci id="A1.E7.m1.1.1.3.1.3.cmml" xref="A1.E7.m1.1.1.3.1.3">𝑛</ci></apply><apply id="A1.E7.m1.1.1.3.2.cmml" xref="A1.E7.m1.1.1.3.2"><times id="A1.E7.m1.1.1.3.2.1.cmml" xref="A1.E7.m1.1.1.3.2.1"></times><apply id="A1.E7.m1.1.1.3.2.2.cmml" xref="A1.E7.m1.1.1.3.2.2"><csymbol cd="ambiguous" id="A1.E7.m1.1.1.3.2.2.1.cmml" xref="A1.E7.m1.1.1.3.2.2">subscript</csymbol><ci id="A1.E7.m1.1.1.3.2.2.2.cmml" xref="A1.E7.m1.1.1.3.2.2.2">𝑎</ci><ci id="A1.E7.m1.1.1.3.2.2.3.cmml" xref="A1.E7.m1.1.1.3.2.2.3">𝑖</ci></apply><apply id="A1.E7.m1.1.1.3.2.3.cmml" xref="A1.E7.m1.1.1.3.2.3"><csymbol cd="ambiguous" id="A1.E7.m1.1.1.3.2.3.1.cmml" xref="A1.E7.m1.1.1.3.2.3">subscript</csymbol><ci id="A1.E7.m1.1.1.3.2.3.2.cmml" xref="A1.E7.m1.1.1.3.2.3.2">𝑣</ci><ci id="A1.E7.m1.1.1.3.2.3.3.cmml" xref="A1.E7.m1.1.1.3.2.3.3">𝑖</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E7.m1.1c">u=\sum_{i=1}^{n}a_{i}v_{i}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(7)</span></td>
</tr></tbody>
</table>
<p id="A1.SS4.p2.12" class="ltx_p"><span id="A1.SS4.p2.12.1" class="ltx_text ltx_font_typewriter">In the k-NN approach, a subset </span><math id="A1.SS4.p2.9.m1.1" class="ltx_Math" alttext="K^{\prime}" display="inline"><semantics id="A1.SS4.p2.9.m1.1a"><msup id="A1.SS4.p2.9.m1.1.1" xref="A1.SS4.p2.9.m1.1.1.cmml"><mi id="A1.SS4.p2.9.m1.1.1.2" xref="A1.SS4.p2.9.m1.1.1.2.cmml">K</mi><mo id="A1.SS4.p2.9.m1.1.1.3" xref="A1.SS4.p2.9.m1.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="A1.SS4.p2.9.m1.1b"><apply id="A1.SS4.p2.9.m1.1.1.cmml" xref="A1.SS4.p2.9.m1.1.1"><csymbol cd="ambiguous" id="A1.SS4.p2.9.m1.1.1.1.cmml" xref="A1.SS4.p2.9.m1.1.1">superscript</csymbol><ci id="A1.SS4.p2.9.m1.1.1.2.cmml" xref="A1.SS4.p2.9.m1.1.1.2">𝐾</ci><ci id="A1.SS4.p2.9.m1.1.1.3.cmml" xref="A1.SS4.p2.9.m1.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p2.9.m1.1c">K^{\prime}</annotation></semantics></math><span id="A1.SS4.p2.12.2" class="ltx_text ltx_font_typewriter"> of size </span><math id="A1.SS4.p2.10.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A1.SS4.p2.10.m2.1a"><mi id="A1.SS4.p2.10.m2.1.1" xref="A1.SS4.p2.10.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A1.SS4.p2.10.m2.1b"><ci id="A1.SS4.p2.10.m2.1.1.cmml" xref="A1.SS4.p2.10.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p2.10.m2.1c">k</annotation></semantics></math><span id="A1.SS4.p2.12.3" class="ltx_text ltx_font_typewriter"> is selected, containing keys nearest to </span><math id="A1.SS4.p2.11.m3.1" class="ltx_Math" alttext="q" display="inline"><semantics id="A1.SS4.p2.11.m3.1a"><mi id="A1.SS4.p2.11.m3.1.1" xref="A1.SS4.p2.11.m3.1.1.cmml">q</mi><annotation-xml encoding="MathML-Content" id="A1.SS4.p2.11.m3.1b"><ci id="A1.SS4.p2.11.m3.1.1.cmml" xref="A1.SS4.p2.11.m3.1.1">𝑞</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p2.11.m3.1c">q</annotation></semantics></math><span id="A1.SS4.p2.12.4" class="ltx_text ltx_font_typewriter">. The approximated attention weights </span><math id="A1.SS4.p2.12.m4.1" class="ltx_Math" alttext="a^{\prime}_{i}" display="inline"><semantics id="A1.SS4.p2.12.m4.1a"><msubsup id="A1.SS4.p2.12.m4.1.1" xref="A1.SS4.p2.12.m4.1.1.cmml"><mi id="A1.SS4.p2.12.m4.1.1.2.2" xref="A1.SS4.p2.12.m4.1.1.2.2.cmml">a</mi><mi id="A1.SS4.p2.12.m4.1.1.3" xref="A1.SS4.p2.12.m4.1.1.3.cmml">i</mi><mo id="A1.SS4.p2.12.m4.1.1.2.3" xref="A1.SS4.p2.12.m4.1.1.2.3.cmml">′</mo></msubsup><annotation-xml encoding="MathML-Content" id="A1.SS4.p2.12.m4.1b"><apply id="A1.SS4.p2.12.m4.1.1.cmml" xref="A1.SS4.p2.12.m4.1.1"><csymbol cd="ambiguous" id="A1.SS4.p2.12.m4.1.1.1.cmml" xref="A1.SS4.p2.12.m4.1.1">subscript</csymbol><apply id="A1.SS4.p2.12.m4.1.1.2.cmml" xref="A1.SS4.p2.12.m4.1.1"><csymbol cd="ambiguous" id="A1.SS4.p2.12.m4.1.1.2.1.cmml" xref="A1.SS4.p2.12.m4.1.1">superscript</csymbol><ci id="A1.SS4.p2.12.m4.1.1.2.2.cmml" xref="A1.SS4.p2.12.m4.1.1.2.2">𝑎</ci><ci id="A1.SS4.p2.12.m4.1.1.2.3.cmml" xref="A1.SS4.p2.12.m4.1.1.2.3">′</ci></apply><ci id="A1.SS4.p2.12.m4.1.1.3.cmml" xref="A1.SS4.p2.12.m4.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p2.12.m4.1c">a^{\prime}_{i}</annotation></semantics></math><span id="A1.SS4.p2.12.5" class="ltx_text ltx_font_typewriter"> over this subset are:</span></p>
<table id="A1.E8" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.E8.m1.6" class="ltx_Math" alttext="a^{\prime}_{i}=\frac{\exp(q\cdot k_{i}~{}d^{-\frac{1}{2}})}{\sum_{j\in K^{\prime}}\exp(q\cdot k_{j}~{}d^{-\frac{1}{2}})}\quad\text{for }k_{i}\in K^{\prime}" display="block"><semantics id="A1.E8.m1.6a"><mrow id="A1.E8.m1.6.6.2" xref="A1.E8.m1.6.6.3.cmml"><mrow id="A1.E8.m1.5.5.1.1" xref="A1.E8.m1.5.5.1.1.cmml"><msubsup id="A1.E8.m1.5.5.1.1.2" xref="A1.E8.m1.5.5.1.1.2.cmml"><mi id="A1.E8.m1.5.5.1.1.2.2.2" xref="A1.E8.m1.5.5.1.1.2.2.2.cmml">a</mi><mi id="A1.E8.m1.5.5.1.1.2.3" xref="A1.E8.m1.5.5.1.1.2.3.cmml">i</mi><mo id="A1.E8.m1.5.5.1.1.2.2.3" xref="A1.E8.m1.5.5.1.1.2.2.3.cmml">′</mo></msubsup><mo id="A1.E8.m1.5.5.1.1.1" xref="A1.E8.m1.5.5.1.1.1.cmml">=</mo><mfrac id="A1.E8.m1.4.4" xref="A1.E8.m1.4.4.cmml"><mrow id="A1.E8.m1.2.2.2.2" xref="A1.E8.m1.2.2.2.3.cmml"><mi id="A1.E8.m1.1.1.1.1" xref="A1.E8.m1.1.1.1.1.cmml">exp</mi><mo id="A1.E8.m1.2.2.2.2a" xref="A1.E8.m1.2.2.2.3.cmml">⁡</mo><mrow id="A1.E8.m1.2.2.2.2.1" xref="A1.E8.m1.2.2.2.3.cmml"><mo stretchy="false" id="A1.E8.m1.2.2.2.2.1.2" xref="A1.E8.m1.2.2.2.3.cmml">(</mo><mrow id="A1.E8.m1.2.2.2.2.1.1" xref="A1.E8.m1.2.2.2.2.1.1.cmml"><mrow id="A1.E8.m1.2.2.2.2.1.1.2" xref="A1.E8.m1.2.2.2.2.1.1.2.cmml"><mi id="A1.E8.m1.2.2.2.2.1.1.2.2" xref="A1.E8.m1.2.2.2.2.1.1.2.2.cmml">q</mi><mo lspace="0.222em" rspace="0.222em" id="A1.E8.m1.2.2.2.2.1.1.2.1" xref="A1.E8.m1.2.2.2.2.1.1.2.1.cmml">⋅</mo><msub id="A1.E8.m1.2.2.2.2.1.1.2.3" xref="A1.E8.m1.2.2.2.2.1.1.2.3.cmml"><mi id="A1.E8.m1.2.2.2.2.1.1.2.3.2" xref="A1.E8.m1.2.2.2.2.1.1.2.3.2.cmml">k</mi><mi id="A1.E8.m1.2.2.2.2.1.1.2.3.3" xref="A1.E8.m1.2.2.2.2.1.1.2.3.3.cmml">i</mi></msub></mrow><mo lspace="0em" rspace="0em" id="A1.E8.m1.2.2.2.2.1.1.1" xref="A1.E8.m1.2.2.2.2.1.1.1.cmml">​</mo><msup id="A1.E8.m1.2.2.2.2.1.1.3" xref="A1.E8.m1.2.2.2.2.1.1.3.cmml"><mi id="A1.E8.m1.2.2.2.2.1.1.3.2" xref="A1.E8.m1.2.2.2.2.1.1.3.2.cmml">d</mi><mrow id="A1.E8.m1.2.2.2.2.1.1.3.3" xref="A1.E8.m1.2.2.2.2.1.1.3.3.cmml"><mo id="A1.E8.m1.2.2.2.2.1.1.3.3a" xref="A1.E8.m1.2.2.2.2.1.1.3.3.cmml">−</mo><mfrac id="A1.E8.m1.2.2.2.2.1.1.3.3.2" xref="A1.E8.m1.2.2.2.2.1.1.3.3.2.cmml"><mn id="A1.E8.m1.2.2.2.2.1.1.3.3.2.2" xref="A1.E8.m1.2.2.2.2.1.1.3.3.2.2.cmml">1</mn><mn id="A1.E8.m1.2.2.2.2.1.1.3.3.2.3" xref="A1.E8.m1.2.2.2.2.1.1.3.3.2.3.cmml">2</mn></mfrac></mrow></msup></mrow><mo stretchy="false" id="A1.E8.m1.2.2.2.2.1.3" xref="A1.E8.m1.2.2.2.3.cmml">)</mo></mrow></mrow><mrow id="A1.E8.m1.4.4.4" xref="A1.E8.m1.4.4.4.cmml"><msub id="A1.E8.m1.4.4.4.3" xref="A1.E8.m1.4.4.4.3.cmml"><mo id="A1.E8.m1.4.4.4.3.2" xref="A1.E8.m1.4.4.4.3.2.cmml">∑</mo><mrow id="A1.E8.m1.4.4.4.3.3" xref="A1.E8.m1.4.4.4.3.3.cmml"><mi id="A1.E8.m1.4.4.4.3.3.2" xref="A1.E8.m1.4.4.4.3.3.2.cmml">j</mi><mo id="A1.E8.m1.4.4.4.3.3.1" xref="A1.E8.m1.4.4.4.3.3.1.cmml">∈</mo><msup id="A1.E8.m1.4.4.4.3.3.3" xref="A1.E8.m1.4.4.4.3.3.3.cmml"><mi id="A1.E8.m1.4.4.4.3.3.3.2" xref="A1.E8.m1.4.4.4.3.3.3.2.cmml">K</mi><mo id="A1.E8.m1.4.4.4.3.3.3.3" xref="A1.E8.m1.4.4.4.3.3.3.3.cmml">′</mo></msup></mrow></msub><mrow id="A1.E8.m1.4.4.4.2.1" xref="A1.E8.m1.4.4.4.2.2.cmml"><mi id="A1.E8.m1.3.3.3.1" xref="A1.E8.m1.3.3.3.1.cmml">exp</mi><mo id="A1.E8.m1.4.4.4.2.1a" xref="A1.E8.m1.4.4.4.2.2.cmml">⁡</mo><mrow id="A1.E8.m1.4.4.4.2.1.1" xref="A1.E8.m1.4.4.4.2.2.cmml"><mo stretchy="false" id="A1.E8.m1.4.4.4.2.1.1.2" xref="A1.E8.m1.4.4.4.2.2.cmml">(</mo><mrow id="A1.E8.m1.4.4.4.2.1.1.1" xref="A1.E8.m1.4.4.4.2.1.1.1.cmml"><mrow id="A1.E8.m1.4.4.4.2.1.1.1.2" xref="A1.E8.m1.4.4.4.2.1.1.1.2.cmml"><mi id="A1.E8.m1.4.4.4.2.1.1.1.2.2" xref="A1.E8.m1.4.4.4.2.1.1.1.2.2.cmml">q</mi><mo lspace="0.222em" rspace="0.222em" id="A1.E8.m1.4.4.4.2.1.1.1.2.1" xref="A1.E8.m1.4.4.4.2.1.1.1.2.1.cmml">⋅</mo><msub id="A1.E8.m1.4.4.4.2.1.1.1.2.3" xref="A1.E8.m1.4.4.4.2.1.1.1.2.3.cmml"><mi id="A1.E8.m1.4.4.4.2.1.1.1.2.3.2" xref="A1.E8.m1.4.4.4.2.1.1.1.2.3.2.cmml">k</mi><mi id="A1.E8.m1.4.4.4.2.1.1.1.2.3.3" xref="A1.E8.m1.4.4.4.2.1.1.1.2.3.3.cmml">j</mi></msub></mrow><mo lspace="0em" rspace="0em" id="A1.E8.m1.4.4.4.2.1.1.1.1" xref="A1.E8.m1.4.4.4.2.1.1.1.1.cmml">​</mo><msup id="A1.E8.m1.4.4.4.2.1.1.1.3" xref="A1.E8.m1.4.4.4.2.1.1.1.3.cmml"><mi id="A1.E8.m1.4.4.4.2.1.1.1.3.2" xref="A1.E8.m1.4.4.4.2.1.1.1.3.2.cmml">d</mi><mrow id="A1.E8.m1.4.4.4.2.1.1.1.3.3" xref="A1.E8.m1.4.4.4.2.1.1.1.3.3.cmml"><mo id="A1.E8.m1.4.4.4.2.1.1.1.3.3a" xref="A1.E8.m1.4.4.4.2.1.1.1.3.3.cmml">−</mo><mfrac id="A1.E8.m1.4.4.4.2.1.1.1.3.3.2" xref="A1.E8.m1.4.4.4.2.1.1.1.3.3.2.cmml"><mn id="A1.E8.m1.4.4.4.2.1.1.1.3.3.2.2" xref="A1.E8.m1.4.4.4.2.1.1.1.3.3.2.2.cmml">1</mn><mn id="A1.E8.m1.4.4.4.2.1.1.1.3.3.2.3" xref="A1.E8.m1.4.4.4.2.1.1.1.3.3.2.3.cmml">2</mn></mfrac></mrow></msup></mrow><mo stretchy="false" id="A1.E8.m1.4.4.4.2.1.1.3" xref="A1.E8.m1.4.4.4.2.2.cmml">)</mo></mrow></mrow></mrow></mfrac></mrow><mspace width="1em" id="A1.E8.m1.6.6.2.3" xref="A1.E8.m1.6.6.3a.cmml"></mspace><mrow id="A1.E8.m1.6.6.2.2" xref="A1.E8.m1.6.6.2.2.cmml"><mrow id="A1.E8.m1.6.6.2.2.2" xref="A1.E8.m1.6.6.2.2.2.cmml"><mtext class="ltx_mathvariant_monospace" id="A1.E8.m1.6.6.2.2.2.2" xref="A1.E8.m1.6.6.2.2.2.2a.cmml">for </mtext><mo lspace="0em" rspace="0em" id="A1.E8.m1.6.6.2.2.2.1" xref="A1.E8.m1.6.6.2.2.2.1.cmml">​</mo><msub id="A1.E8.m1.6.6.2.2.2.3" xref="A1.E8.m1.6.6.2.2.2.3.cmml"><mi id="A1.E8.m1.6.6.2.2.2.3.2" xref="A1.E8.m1.6.6.2.2.2.3.2.cmml">k</mi><mi id="A1.E8.m1.6.6.2.2.2.3.3" xref="A1.E8.m1.6.6.2.2.2.3.3.cmml">i</mi></msub></mrow><mo id="A1.E8.m1.6.6.2.2.1" xref="A1.E8.m1.6.6.2.2.1.cmml">∈</mo><msup id="A1.E8.m1.6.6.2.2.3" xref="A1.E8.m1.6.6.2.2.3.cmml"><mi id="A1.E8.m1.6.6.2.2.3.2" xref="A1.E8.m1.6.6.2.2.3.2.cmml">K</mi><mo id="A1.E8.m1.6.6.2.2.3.3" xref="A1.E8.m1.6.6.2.2.3.3.cmml">′</mo></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E8.m1.6b"><apply id="A1.E8.m1.6.6.3.cmml" xref="A1.E8.m1.6.6.2"><csymbol cd="ambiguous" id="A1.E8.m1.6.6.3a.cmml" xref="A1.E8.m1.6.6.2.3">formulae-sequence</csymbol><apply id="A1.E8.m1.5.5.1.1.cmml" xref="A1.E8.m1.5.5.1.1"><eq id="A1.E8.m1.5.5.1.1.1.cmml" xref="A1.E8.m1.5.5.1.1.1"></eq><apply id="A1.E8.m1.5.5.1.1.2.cmml" xref="A1.E8.m1.5.5.1.1.2"><csymbol cd="ambiguous" id="A1.E8.m1.5.5.1.1.2.1.cmml" xref="A1.E8.m1.5.5.1.1.2">subscript</csymbol><apply id="A1.E8.m1.5.5.1.1.2.2.cmml" xref="A1.E8.m1.5.5.1.1.2"><csymbol cd="ambiguous" id="A1.E8.m1.5.5.1.1.2.2.1.cmml" xref="A1.E8.m1.5.5.1.1.2">superscript</csymbol><ci id="A1.E8.m1.5.5.1.1.2.2.2.cmml" xref="A1.E8.m1.5.5.1.1.2.2.2">𝑎</ci><ci id="A1.E8.m1.5.5.1.1.2.2.3.cmml" xref="A1.E8.m1.5.5.1.1.2.2.3">′</ci></apply><ci id="A1.E8.m1.5.5.1.1.2.3.cmml" xref="A1.E8.m1.5.5.1.1.2.3">𝑖</ci></apply><apply id="A1.E8.m1.4.4.cmml" xref="A1.E8.m1.4.4"><divide id="A1.E8.m1.4.4.5.cmml" xref="A1.E8.m1.4.4"></divide><apply id="A1.E8.m1.2.2.2.3.cmml" xref="A1.E8.m1.2.2.2.2"><exp id="A1.E8.m1.1.1.1.1.cmml" xref="A1.E8.m1.1.1.1.1"></exp><apply id="A1.E8.m1.2.2.2.2.1.1.cmml" xref="A1.E8.m1.2.2.2.2.1.1"><times id="A1.E8.m1.2.2.2.2.1.1.1.cmml" xref="A1.E8.m1.2.2.2.2.1.1.1"></times><apply id="A1.E8.m1.2.2.2.2.1.1.2.cmml" xref="A1.E8.m1.2.2.2.2.1.1.2"><ci id="A1.E8.m1.2.2.2.2.1.1.2.1.cmml" xref="A1.E8.m1.2.2.2.2.1.1.2.1">⋅</ci><ci id="A1.E8.m1.2.2.2.2.1.1.2.2.cmml" xref="A1.E8.m1.2.2.2.2.1.1.2.2">𝑞</ci><apply id="A1.E8.m1.2.2.2.2.1.1.2.3.cmml" xref="A1.E8.m1.2.2.2.2.1.1.2.3"><csymbol cd="ambiguous" id="A1.E8.m1.2.2.2.2.1.1.2.3.1.cmml" xref="A1.E8.m1.2.2.2.2.1.1.2.3">subscript</csymbol><ci id="A1.E8.m1.2.2.2.2.1.1.2.3.2.cmml" xref="A1.E8.m1.2.2.2.2.1.1.2.3.2">𝑘</ci><ci id="A1.E8.m1.2.2.2.2.1.1.2.3.3.cmml" xref="A1.E8.m1.2.2.2.2.1.1.2.3.3">𝑖</ci></apply></apply><apply id="A1.E8.m1.2.2.2.2.1.1.3.cmml" xref="A1.E8.m1.2.2.2.2.1.1.3"><csymbol cd="ambiguous" id="A1.E8.m1.2.2.2.2.1.1.3.1.cmml" xref="A1.E8.m1.2.2.2.2.1.1.3">superscript</csymbol><ci id="A1.E8.m1.2.2.2.2.1.1.3.2.cmml" xref="A1.E8.m1.2.2.2.2.1.1.3.2">𝑑</ci><apply id="A1.E8.m1.2.2.2.2.1.1.3.3.cmml" xref="A1.E8.m1.2.2.2.2.1.1.3.3"><minus id="A1.E8.m1.2.2.2.2.1.1.3.3.1.cmml" xref="A1.E8.m1.2.2.2.2.1.1.3.3"></minus><apply id="A1.E8.m1.2.2.2.2.1.1.3.3.2.cmml" xref="A1.E8.m1.2.2.2.2.1.1.3.3.2"><divide id="A1.E8.m1.2.2.2.2.1.1.3.3.2.1.cmml" xref="A1.E8.m1.2.2.2.2.1.1.3.3.2"></divide><cn type="integer" id="A1.E8.m1.2.2.2.2.1.1.3.3.2.2.cmml" xref="A1.E8.m1.2.2.2.2.1.1.3.3.2.2">1</cn><cn type="integer" id="A1.E8.m1.2.2.2.2.1.1.3.3.2.3.cmml" xref="A1.E8.m1.2.2.2.2.1.1.3.3.2.3">2</cn></apply></apply></apply></apply></apply><apply id="A1.E8.m1.4.4.4.cmml" xref="A1.E8.m1.4.4.4"><apply id="A1.E8.m1.4.4.4.3.cmml" xref="A1.E8.m1.4.4.4.3"><csymbol cd="ambiguous" id="A1.E8.m1.4.4.4.3.1.cmml" xref="A1.E8.m1.4.4.4.3">subscript</csymbol><sum id="A1.E8.m1.4.4.4.3.2.cmml" xref="A1.E8.m1.4.4.4.3.2"></sum><apply id="A1.E8.m1.4.4.4.3.3.cmml" xref="A1.E8.m1.4.4.4.3.3"><in id="A1.E8.m1.4.4.4.3.3.1.cmml" xref="A1.E8.m1.4.4.4.3.3.1"></in><ci id="A1.E8.m1.4.4.4.3.3.2.cmml" xref="A1.E8.m1.4.4.4.3.3.2">𝑗</ci><apply id="A1.E8.m1.4.4.4.3.3.3.cmml" xref="A1.E8.m1.4.4.4.3.3.3"><csymbol cd="ambiguous" id="A1.E8.m1.4.4.4.3.3.3.1.cmml" xref="A1.E8.m1.4.4.4.3.3.3">superscript</csymbol><ci id="A1.E8.m1.4.4.4.3.3.3.2.cmml" xref="A1.E8.m1.4.4.4.3.3.3.2">𝐾</ci><ci id="A1.E8.m1.4.4.4.3.3.3.3.cmml" xref="A1.E8.m1.4.4.4.3.3.3.3">′</ci></apply></apply></apply><apply id="A1.E8.m1.4.4.4.2.2.cmml" xref="A1.E8.m1.4.4.4.2.1"><exp id="A1.E8.m1.3.3.3.1.cmml" xref="A1.E8.m1.3.3.3.1"></exp><apply id="A1.E8.m1.4.4.4.2.1.1.1.cmml" xref="A1.E8.m1.4.4.4.2.1.1.1"><times id="A1.E8.m1.4.4.4.2.1.1.1.1.cmml" xref="A1.E8.m1.4.4.4.2.1.1.1.1"></times><apply id="A1.E8.m1.4.4.4.2.1.1.1.2.cmml" xref="A1.E8.m1.4.4.4.2.1.1.1.2"><ci id="A1.E8.m1.4.4.4.2.1.1.1.2.1.cmml" xref="A1.E8.m1.4.4.4.2.1.1.1.2.1">⋅</ci><ci id="A1.E8.m1.4.4.4.2.1.1.1.2.2.cmml" xref="A1.E8.m1.4.4.4.2.1.1.1.2.2">𝑞</ci><apply id="A1.E8.m1.4.4.4.2.1.1.1.2.3.cmml" xref="A1.E8.m1.4.4.4.2.1.1.1.2.3"><csymbol cd="ambiguous" id="A1.E8.m1.4.4.4.2.1.1.1.2.3.1.cmml" xref="A1.E8.m1.4.4.4.2.1.1.1.2.3">subscript</csymbol><ci id="A1.E8.m1.4.4.4.2.1.1.1.2.3.2.cmml" xref="A1.E8.m1.4.4.4.2.1.1.1.2.3.2">𝑘</ci><ci id="A1.E8.m1.4.4.4.2.1.1.1.2.3.3.cmml" xref="A1.E8.m1.4.4.4.2.1.1.1.2.3.3">𝑗</ci></apply></apply><apply id="A1.E8.m1.4.4.4.2.1.1.1.3.cmml" xref="A1.E8.m1.4.4.4.2.1.1.1.3"><csymbol cd="ambiguous" id="A1.E8.m1.4.4.4.2.1.1.1.3.1.cmml" xref="A1.E8.m1.4.4.4.2.1.1.1.3">superscript</csymbol><ci id="A1.E8.m1.4.4.4.2.1.1.1.3.2.cmml" xref="A1.E8.m1.4.4.4.2.1.1.1.3.2">𝑑</ci><apply id="A1.E8.m1.4.4.4.2.1.1.1.3.3.cmml" xref="A1.E8.m1.4.4.4.2.1.1.1.3.3"><minus id="A1.E8.m1.4.4.4.2.1.1.1.3.3.1.cmml" xref="A1.E8.m1.4.4.4.2.1.1.1.3.3"></minus><apply id="A1.E8.m1.4.4.4.2.1.1.1.3.3.2.cmml" xref="A1.E8.m1.4.4.4.2.1.1.1.3.3.2"><divide id="A1.E8.m1.4.4.4.2.1.1.1.3.3.2.1.cmml" xref="A1.E8.m1.4.4.4.2.1.1.1.3.3.2"></divide><cn type="integer" id="A1.E8.m1.4.4.4.2.1.1.1.3.3.2.2.cmml" xref="A1.E8.m1.4.4.4.2.1.1.1.3.3.2.2">1</cn><cn type="integer" id="A1.E8.m1.4.4.4.2.1.1.1.3.3.2.3.cmml" xref="A1.E8.m1.4.4.4.2.1.1.1.3.3.2.3">2</cn></apply></apply></apply></apply></apply></apply></apply></apply><apply id="A1.E8.m1.6.6.2.2.cmml" xref="A1.E8.m1.6.6.2.2"><in id="A1.E8.m1.6.6.2.2.1.cmml" xref="A1.E8.m1.6.6.2.2.1"></in><apply id="A1.E8.m1.6.6.2.2.2.cmml" xref="A1.E8.m1.6.6.2.2.2"><times id="A1.E8.m1.6.6.2.2.2.1.cmml" xref="A1.E8.m1.6.6.2.2.2.1"></times><ci id="A1.E8.m1.6.6.2.2.2.2a.cmml" xref="A1.E8.m1.6.6.2.2.2.2"><mtext class="ltx_mathvariant_monospace" id="A1.E8.m1.6.6.2.2.2.2.cmml" xref="A1.E8.m1.6.6.2.2.2.2">for </mtext></ci><apply id="A1.E8.m1.6.6.2.2.2.3.cmml" xref="A1.E8.m1.6.6.2.2.2.3"><csymbol cd="ambiguous" id="A1.E8.m1.6.6.2.2.2.3.1.cmml" xref="A1.E8.m1.6.6.2.2.2.3">subscript</csymbol><ci id="A1.E8.m1.6.6.2.2.2.3.2.cmml" xref="A1.E8.m1.6.6.2.2.2.3.2">𝑘</ci><ci id="A1.E8.m1.6.6.2.2.2.3.3.cmml" xref="A1.E8.m1.6.6.2.2.2.3.3">𝑖</ci></apply></apply><apply id="A1.E8.m1.6.6.2.2.3.cmml" xref="A1.E8.m1.6.6.2.2.3"><csymbol cd="ambiguous" id="A1.E8.m1.6.6.2.2.3.1.cmml" xref="A1.E8.m1.6.6.2.2.3">superscript</csymbol><ci id="A1.E8.m1.6.6.2.2.3.2.cmml" xref="A1.E8.m1.6.6.2.2.3.2">𝐾</ci><ci id="A1.E8.m1.6.6.2.2.3.3.cmml" xref="A1.E8.m1.6.6.2.2.3.3">′</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E8.m1.6c">a^{\prime}_{i}=\frac{\exp(q\cdot k_{i}~{}d^{-\frac{1}{2}})}{\sum_{j\in K^{\prime}}\exp(q\cdot k_{j}~{}d^{-\frac{1}{2}})}\quad\text{for }k_{i}\in K^{\prime}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(8)</span></td>
</tr></tbody>
</table>
<p id="A1.SS4.p2.13" class="ltx_p"><span id="A1.SS4.p2.13.1" class="ltx_text ltx_font_typewriter">The approximate output vector </span><math id="A1.SS4.p2.13.m1.1" class="ltx_Math" alttext="u^{\prime}" display="inline"><semantics id="A1.SS4.p2.13.m1.1a"><msup id="A1.SS4.p2.13.m1.1.1" xref="A1.SS4.p2.13.m1.1.1.cmml"><mi id="A1.SS4.p2.13.m1.1.1.2" xref="A1.SS4.p2.13.m1.1.1.2.cmml">u</mi><mo id="A1.SS4.p2.13.m1.1.1.3" xref="A1.SS4.p2.13.m1.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="A1.SS4.p2.13.m1.1b"><apply id="A1.SS4.p2.13.m1.1.1.cmml" xref="A1.SS4.p2.13.m1.1.1"><csymbol cd="ambiguous" id="A1.SS4.p2.13.m1.1.1.1.cmml" xref="A1.SS4.p2.13.m1.1.1">superscript</csymbol><ci id="A1.SS4.p2.13.m1.1.1.2.cmml" xref="A1.SS4.p2.13.m1.1.1.2">𝑢</ci><ci id="A1.SS4.p2.13.m1.1.1.3.cmml" xref="A1.SS4.p2.13.m1.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p2.13.m1.1c">u^{\prime}</annotation></semantics></math><span id="A1.SS4.p2.13.2" class="ltx_text ltx_font_typewriter"> is:</span></p>
<table id="A1.E9" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.E9.m1.1" class="ltx_Math" alttext="u^{\prime}=\sum_{k_{i}\in K^{\prime}}a^{\prime}_{i}v_{i}" display="block"><semantics id="A1.E9.m1.1a"><mrow id="A1.E9.m1.1.1" xref="A1.E9.m1.1.1.cmml"><msup id="A1.E9.m1.1.1.2" xref="A1.E9.m1.1.1.2.cmml"><mi id="A1.E9.m1.1.1.2.2" xref="A1.E9.m1.1.1.2.2.cmml">u</mi><mo id="A1.E9.m1.1.1.2.3" xref="A1.E9.m1.1.1.2.3.cmml">′</mo></msup><mo rspace="0.111em" id="A1.E9.m1.1.1.1" xref="A1.E9.m1.1.1.1.cmml">=</mo><mrow id="A1.E9.m1.1.1.3" xref="A1.E9.m1.1.1.3.cmml"><munder id="A1.E9.m1.1.1.3.1" xref="A1.E9.m1.1.1.3.1.cmml"><mo movablelimits="false" id="A1.E9.m1.1.1.3.1.2" xref="A1.E9.m1.1.1.3.1.2.cmml">∑</mo><mrow id="A1.E9.m1.1.1.3.1.3" xref="A1.E9.m1.1.1.3.1.3.cmml"><msub id="A1.E9.m1.1.1.3.1.3.2" xref="A1.E9.m1.1.1.3.1.3.2.cmml"><mi id="A1.E9.m1.1.1.3.1.3.2.2" xref="A1.E9.m1.1.1.3.1.3.2.2.cmml">k</mi><mi id="A1.E9.m1.1.1.3.1.3.2.3" xref="A1.E9.m1.1.1.3.1.3.2.3.cmml">i</mi></msub><mo id="A1.E9.m1.1.1.3.1.3.1" xref="A1.E9.m1.1.1.3.1.3.1.cmml">∈</mo><msup id="A1.E9.m1.1.1.3.1.3.3" xref="A1.E9.m1.1.1.3.1.3.3.cmml"><mi id="A1.E9.m1.1.1.3.1.3.3.2" xref="A1.E9.m1.1.1.3.1.3.3.2.cmml">K</mi><mo id="A1.E9.m1.1.1.3.1.3.3.3" xref="A1.E9.m1.1.1.3.1.3.3.3.cmml">′</mo></msup></mrow></munder><mrow id="A1.E9.m1.1.1.3.2" xref="A1.E9.m1.1.1.3.2.cmml"><msubsup id="A1.E9.m1.1.1.3.2.2" xref="A1.E9.m1.1.1.3.2.2.cmml"><mi id="A1.E9.m1.1.1.3.2.2.2.2" xref="A1.E9.m1.1.1.3.2.2.2.2.cmml">a</mi><mi id="A1.E9.m1.1.1.3.2.2.3" xref="A1.E9.m1.1.1.3.2.2.3.cmml">i</mi><mo id="A1.E9.m1.1.1.3.2.2.2.3" xref="A1.E9.m1.1.1.3.2.2.2.3.cmml">′</mo></msubsup><mo lspace="0em" rspace="0em" id="A1.E9.m1.1.1.3.2.1" xref="A1.E9.m1.1.1.3.2.1.cmml">​</mo><msub id="A1.E9.m1.1.1.3.2.3" xref="A1.E9.m1.1.1.3.2.3.cmml"><mi id="A1.E9.m1.1.1.3.2.3.2" xref="A1.E9.m1.1.1.3.2.3.2.cmml">v</mi><mi id="A1.E9.m1.1.1.3.2.3.3" xref="A1.E9.m1.1.1.3.2.3.3.cmml">i</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E9.m1.1b"><apply id="A1.E9.m1.1.1.cmml" xref="A1.E9.m1.1.1"><eq id="A1.E9.m1.1.1.1.cmml" xref="A1.E9.m1.1.1.1"></eq><apply id="A1.E9.m1.1.1.2.cmml" xref="A1.E9.m1.1.1.2"><csymbol cd="ambiguous" id="A1.E9.m1.1.1.2.1.cmml" xref="A1.E9.m1.1.1.2">superscript</csymbol><ci id="A1.E9.m1.1.1.2.2.cmml" xref="A1.E9.m1.1.1.2.2">𝑢</ci><ci id="A1.E9.m1.1.1.2.3.cmml" xref="A1.E9.m1.1.1.2.3">′</ci></apply><apply id="A1.E9.m1.1.1.3.cmml" xref="A1.E9.m1.1.1.3"><apply id="A1.E9.m1.1.1.3.1.cmml" xref="A1.E9.m1.1.1.3.1"><csymbol cd="ambiguous" id="A1.E9.m1.1.1.3.1.1.cmml" xref="A1.E9.m1.1.1.3.1">subscript</csymbol><sum id="A1.E9.m1.1.1.3.1.2.cmml" xref="A1.E9.m1.1.1.3.1.2"></sum><apply id="A1.E9.m1.1.1.3.1.3.cmml" xref="A1.E9.m1.1.1.3.1.3"><in id="A1.E9.m1.1.1.3.1.3.1.cmml" xref="A1.E9.m1.1.1.3.1.3.1"></in><apply id="A1.E9.m1.1.1.3.1.3.2.cmml" xref="A1.E9.m1.1.1.3.1.3.2"><csymbol cd="ambiguous" id="A1.E9.m1.1.1.3.1.3.2.1.cmml" xref="A1.E9.m1.1.1.3.1.3.2">subscript</csymbol><ci id="A1.E9.m1.1.1.3.1.3.2.2.cmml" xref="A1.E9.m1.1.1.3.1.3.2.2">𝑘</ci><ci id="A1.E9.m1.1.1.3.1.3.2.3.cmml" xref="A1.E9.m1.1.1.3.1.3.2.3">𝑖</ci></apply><apply id="A1.E9.m1.1.1.3.1.3.3.cmml" xref="A1.E9.m1.1.1.3.1.3.3"><csymbol cd="ambiguous" id="A1.E9.m1.1.1.3.1.3.3.1.cmml" xref="A1.E9.m1.1.1.3.1.3.3">superscript</csymbol><ci id="A1.E9.m1.1.1.3.1.3.3.2.cmml" xref="A1.E9.m1.1.1.3.1.3.3.2">𝐾</ci><ci id="A1.E9.m1.1.1.3.1.3.3.3.cmml" xref="A1.E9.m1.1.1.3.1.3.3.3">′</ci></apply></apply></apply><apply id="A1.E9.m1.1.1.3.2.cmml" xref="A1.E9.m1.1.1.3.2"><times id="A1.E9.m1.1.1.3.2.1.cmml" xref="A1.E9.m1.1.1.3.2.1"></times><apply id="A1.E9.m1.1.1.3.2.2.cmml" xref="A1.E9.m1.1.1.3.2.2"><csymbol cd="ambiguous" id="A1.E9.m1.1.1.3.2.2.1.cmml" xref="A1.E9.m1.1.1.3.2.2">subscript</csymbol><apply id="A1.E9.m1.1.1.3.2.2.2.cmml" xref="A1.E9.m1.1.1.3.2.2"><csymbol cd="ambiguous" id="A1.E9.m1.1.1.3.2.2.2.1.cmml" xref="A1.E9.m1.1.1.3.2.2">superscript</csymbol><ci id="A1.E9.m1.1.1.3.2.2.2.2.cmml" xref="A1.E9.m1.1.1.3.2.2.2.2">𝑎</ci><ci id="A1.E9.m1.1.1.3.2.2.2.3.cmml" xref="A1.E9.m1.1.1.3.2.2.2.3">′</ci></apply><ci id="A1.E9.m1.1.1.3.2.2.3.cmml" xref="A1.E9.m1.1.1.3.2.2.3">𝑖</ci></apply><apply id="A1.E9.m1.1.1.3.2.3.cmml" xref="A1.E9.m1.1.1.3.2.3"><csymbol cd="ambiguous" id="A1.E9.m1.1.1.3.2.3.1.cmml" xref="A1.E9.m1.1.1.3.2.3">subscript</csymbol><ci id="A1.E9.m1.1.1.3.2.3.2.cmml" xref="A1.E9.m1.1.1.3.2.3.2">𝑣</ci><ci id="A1.E9.m1.1.1.3.2.3.3.cmml" xref="A1.E9.m1.1.1.3.2.3.3">𝑖</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E9.m1.1c">u^{\prime}=\sum_{k_{i}\in K^{\prime}}a^{\prime}_{i}v_{i}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(9)</span></td>
</tr></tbody>
</table>
</div>
</section>
<section id="A1.SSx1" class="ltx_subsection">
<h3 class="ltx_title ltx_font_typewriter ltx_title_subsection">Assumptions</h3>

<div id="A1.SSx1.p1" class="ltx_para ltx_noindent">
<ol id="A1.I2" class="ltx_enumerate">
<li id="A1.I2.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="A1.I2.i1.p1" class="ltx_para">
<p id="A1.I2.i1.p1.1" class="ltx_p"><span id="A1.I2.i1.p1.1.1" class="ltx_text ltx_font_italic">Exponential Dominance</span><span id="A1.I2.i1.p1.1.2" class="ltx_text ltx_font_typewriter">: The exponential function in the softmax is sharply peaked, implying that keys with the highest similarities to </span><math id="A1.I2.i1.p1.1.m1.1" class="ltx_Math" alttext="q" display="inline"><semantics id="A1.I2.i1.p1.1.m1.1a"><mi id="A1.I2.i1.p1.1.m1.1.1" xref="A1.I2.i1.p1.1.m1.1.1.cmml">q</mi><annotation-xml encoding="MathML-Content" id="A1.I2.i1.p1.1.m1.1b"><ci id="A1.I2.i1.p1.1.m1.1.1.cmml" xref="A1.I2.i1.p1.1.m1.1.1">𝑞</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.I2.i1.p1.1.m1.1c">q</annotation></semantics></math><span id="A1.I2.i1.p1.1.3" class="ltx_text ltx_font_typewriter"> contribute significantly more to the sum than others.</span></p>
</div>
</li>
<li id="A1.I2.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="A1.I2.i2.p1" class="ltx_para ltx_noindent">
<p id="A1.I2.i2.p1.2" class="ltx_p"><span id="A1.I2.i2.p1.2.1" class="ltx_text ltx_font_italic">Representativeness of k-NN Subset</span><span id="A1.I2.i2.p1.2.2" class="ltx_text ltx_font_typewriter">: The subset </span><math id="A1.I2.i2.p1.1.m1.1" class="ltx_Math" alttext="K^{\prime}" display="inline"><semantics id="A1.I2.i2.p1.1.m1.1a"><msup id="A1.I2.i2.p1.1.m1.1.1" xref="A1.I2.i2.p1.1.m1.1.1.cmml"><mi id="A1.I2.i2.p1.1.m1.1.1.2" xref="A1.I2.i2.p1.1.m1.1.1.2.cmml">K</mi><mo id="A1.I2.i2.p1.1.m1.1.1.3" xref="A1.I2.i2.p1.1.m1.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="A1.I2.i2.p1.1.m1.1b"><apply id="A1.I2.i2.p1.1.m1.1.1.cmml" xref="A1.I2.i2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A1.I2.i2.p1.1.m1.1.1.1.cmml" xref="A1.I2.i2.p1.1.m1.1.1">superscript</csymbol><ci id="A1.I2.i2.p1.1.m1.1.1.2.cmml" xref="A1.I2.i2.p1.1.m1.1.1.2">𝐾</ci><ci id="A1.I2.i2.p1.1.m1.1.1.3.cmml" xref="A1.I2.i2.p1.1.m1.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I2.i2.p1.1.m1.1c">K^{\prime}</annotation></semantics></math><span id="A1.I2.i2.p1.2.3" class="ltx_text ltx_font_typewriter"> captures the majority of the attention weight from the full set </span><math id="A1.I2.i2.p1.2.m2.1" class="ltx_Math" alttext="K" display="inline"><semantics id="A1.I2.i2.p1.2.m2.1a"><mi id="A1.I2.i2.p1.2.m2.1.1" xref="A1.I2.i2.p1.2.m2.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="A1.I2.i2.p1.2.m2.1b"><ci id="A1.I2.i2.p1.2.m2.1.1.cmml" xref="A1.I2.i2.p1.2.m2.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.I2.i2.p1.2.m2.1c">K</annotation></semantics></math><span id="A1.I2.i2.p1.2.4" class="ltx_text ltx_font_typewriter">.</span></p>
</div>
</li>
</ol>
</div>
<section id="A1.SSx1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_font_typewriter ltx_title_paragraph">Lemma 1: Dominance of k-NN Subset</h4>

<div id="A1.SSx1.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="A1.SSx1.SSS0.Px1.p1.3" class="ltx_p"><span id="A1.SSx1.SSS0.Px1.p1.3.1" class="ltx_text ltx_font_typewriter">If </span><math id="A1.SSx1.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="K^{\prime}" display="inline"><semantics id="A1.SSx1.SSS0.Px1.p1.1.m1.1a"><msup id="A1.SSx1.SSS0.Px1.p1.1.m1.1.1" xref="A1.SSx1.SSS0.Px1.p1.1.m1.1.1.cmml"><mi id="A1.SSx1.SSS0.Px1.p1.1.m1.1.1.2" xref="A1.SSx1.SSS0.Px1.p1.1.m1.1.1.2.cmml">K</mi><mo id="A1.SSx1.SSS0.Px1.p1.1.m1.1.1.3" xref="A1.SSx1.SSS0.Px1.p1.1.m1.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="A1.SSx1.SSS0.Px1.p1.1.m1.1b"><apply id="A1.SSx1.SSS0.Px1.p1.1.m1.1.1.cmml" xref="A1.SSx1.SSS0.Px1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A1.SSx1.SSS0.Px1.p1.1.m1.1.1.1.cmml" xref="A1.SSx1.SSS0.Px1.p1.1.m1.1.1">superscript</csymbol><ci id="A1.SSx1.SSS0.Px1.p1.1.m1.1.1.2.cmml" xref="A1.SSx1.SSS0.Px1.p1.1.m1.1.1.2">𝐾</ci><ci id="A1.SSx1.SSS0.Px1.p1.1.m1.1.1.3.cmml" xref="A1.SSx1.SSS0.Px1.p1.1.m1.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SSx1.SSS0.Px1.p1.1.m1.1c">K^{\prime}</annotation></semantics></math><span id="A1.SSx1.SSS0.Px1.p1.3.2" class="ltx_text ltx_font_typewriter"> consists of the </span><math id="A1.SSx1.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A1.SSx1.SSS0.Px1.p1.2.m2.1a"><mi id="A1.SSx1.SSS0.Px1.p1.2.m2.1.1" xref="A1.SSx1.SSS0.Px1.p1.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A1.SSx1.SSS0.Px1.p1.2.m2.1b"><ci id="A1.SSx1.SSS0.Px1.p1.2.m2.1.1.cmml" xref="A1.SSx1.SSS0.Px1.p1.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SSx1.SSS0.Px1.p1.2.m2.1c">k</annotation></semantics></math><span id="A1.SSx1.SSS0.Px1.p1.3.3" class="ltx_text ltx_font_typewriter"> keys with the highest dot products </span><math id="A1.SSx1.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="q\cdot k_{i}" display="inline"><semantics id="A1.SSx1.SSS0.Px1.p1.3.m3.1a"><mrow id="A1.SSx1.SSS0.Px1.p1.3.m3.1.1" xref="A1.SSx1.SSS0.Px1.p1.3.m3.1.1.cmml"><mi id="A1.SSx1.SSS0.Px1.p1.3.m3.1.1.2" xref="A1.SSx1.SSS0.Px1.p1.3.m3.1.1.2.cmml">q</mi><mo lspace="0.222em" rspace="0.222em" id="A1.SSx1.SSS0.Px1.p1.3.m3.1.1.1" xref="A1.SSx1.SSS0.Px1.p1.3.m3.1.1.1.cmml">⋅</mo><msub id="A1.SSx1.SSS0.Px1.p1.3.m3.1.1.3" xref="A1.SSx1.SSS0.Px1.p1.3.m3.1.1.3.cmml"><mi id="A1.SSx1.SSS0.Px1.p1.3.m3.1.1.3.2" xref="A1.SSx1.SSS0.Px1.p1.3.m3.1.1.3.2.cmml">k</mi><mi id="A1.SSx1.SSS0.Px1.p1.3.m3.1.1.3.3" xref="A1.SSx1.SSS0.Px1.p1.3.m3.1.1.3.3.cmml">i</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="A1.SSx1.SSS0.Px1.p1.3.m3.1b"><apply id="A1.SSx1.SSS0.Px1.p1.3.m3.1.1.cmml" xref="A1.SSx1.SSS0.Px1.p1.3.m3.1.1"><ci id="A1.SSx1.SSS0.Px1.p1.3.m3.1.1.1.cmml" xref="A1.SSx1.SSS0.Px1.p1.3.m3.1.1.1">⋅</ci><ci id="A1.SSx1.SSS0.Px1.p1.3.m3.1.1.2.cmml" xref="A1.SSx1.SSS0.Px1.p1.3.m3.1.1.2">𝑞</ci><apply id="A1.SSx1.SSS0.Px1.p1.3.m3.1.1.3.cmml" xref="A1.SSx1.SSS0.Px1.p1.3.m3.1.1.3"><csymbol cd="ambiguous" id="A1.SSx1.SSS0.Px1.p1.3.m3.1.1.3.1.cmml" xref="A1.SSx1.SSS0.Px1.p1.3.m3.1.1.3">subscript</csymbol><ci id="A1.SSx1.SSS0.Px1.p1.3.m3.1.1.3.2.cmml" xref="A1.SSx1.SSS0.Px1.p1.3.m3.1.1.3.2">𝑘</ci><ci id="A1.SSx1.SSS0.Px1.p1.3.m3.1.1.3.3.cmml" xref="A1.SSx1.SSS0.Px1.p1.3.m3.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SSx1.SSS0.Px1.p1.3.m3.1c">q\cdot k_{i}</annotation></semantics></math><span id="A1.SSx1.SSS0.Px1.p1.3.4" class="ltx_text ltx_font_typewriter">, then:</span></p>
<table id="A1.E10" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.E10.m1.4" class="ltx_Math" alttext="\sum_{j\in K^{\prime}}\exp(q\cdot k_{j}~{}d^{-\frac{1}{2}})\geq\alpha\sum_{j=1}^{n}\exp(q\cdot k_{j}~{}d^{-\frac{1}{2}})" display="block"><semantics id="A1.E10.m1.4a"><mrow id="A1.E10.m1.4.4" xref="A1.E10.m1.4.4.cmml"><mrow id="A1.E10.m1.3.3.1" xref="A1.E10.m1.3.3.1.cmml"><munder id="A1.E10.m1.3.3.1.2" xref="A1.E10.m1.3.3.1.2.cmml"><mo movablelimits="false" id="A1.E10.m1.3.3.1.2.2" xref="A1.E10.m1.3.3.1.2.2.cmml">∑</mo><mrow id="A1.E10.m1.3.3.1.2.3" xref="A1.E10.m1.3.3.1.2.3.cmml"><mi id="A1.E10.m1.3.3.1.2.3.2" xref="A1.E10.m1.3.3.1.2.3.2.cmml">j</mi><mo id="A1.E10.m1.3.3.1.2.3.1" xref="A1.E10.m1.3.3.1.2.3.1.cmml">∈</mo><msup id="A1.E10.m1.3.3.1.2.3.3" xref="A1.E10.m1.3.3.1.2.3.3.cmml"><mi id="A1.E10.m1.3.3.1.2.3.3.2" xref="A1.E10.m1.3.3.1.2.3.3.2.cmml">K</mi><mo id="A1.E10.m1.3.3.1.2.3.3.3" xref="A1.E10.m1.3.3.1.2.3.3.3.cmml">′</mo></msup></mrow></munder><mrow id="A1.E10.m1.3.3.1.1.1" xref="A1.E10.m1.3.3.1.1.2.cmml"><mi id="A1.E10.m1.1.1" xref="A1.E10.m1.1.1.cmml">exp</mi><mo id="A1.E10.m1.3.3.1.1.1a" xref="A1.E10.m1.3.3.1.1.2.cmml">⁡</mo><mrow id="A1.E10.m1.3.3.1.1.1.1" xref="A1.E10.m1.3.3.1.1.2.cmml"><mo stretchy="false" id="A1.E10.m1.3.3.1.1.1.1.2" xref="A1.E10.m1.3.3.1.1.2.cmml">(</mo><mrow id="A1.E10.m1.3.3.1.1.1.1.1" xref="A1.E10.m1.3.3.1.1.1.1.1.cmml"><mrow id="A1.E10.m1.3.3.1.1.1.1.1.2" xref="A1.E10.m1.3.3.1.1.1.1.1.2.cmml"><mi id="A1.E10.m1.3.3.1.1.1.1.1.2.2" xref="A1.E10.m1.3.3.1.1.1.1.1.2.2.cmml">q</mi><mo lspace="0.222em" rspace="0.222em" id="A1.E10.m1.3.3.1.1.1.1.1.2.1" xref="A1.E10.m1.3.3.1.1.1.1.1.2.1.cmml">⋅</mo><msub id="A1.E10.m1.3.3.1.1.1.1.1.2.3" xref="A1.E10.m1.3.3.1.1.1.1.1.2.3.cmml"><mi id="A1.E10.m1.3.3.1.1.1.1.1.2.3.2" xref="A1.E10.m1.3.3.1.1.1.1.1.2.3.2.cmml">k</mi><mi id="A1.E10.m1.3.3.1.1.1.1.1.2.3.3" xref="A1.E10.m1.3.3.1.1.1.1.1.2.3.3.cmml">j</mi></msub></mrow><mo lspace="0em" rspace="0em" id="A1.E10.m1.3.3.1.1.1.1.1.1" xref="A1.E10.m1.3.3.1.1.1.1.1.1.cmml">​</mo><msup id="A1.E10.m1.3.3.1.1.1.1.1.3" xref="A1.E10.m1.3.3.1.1.1.1.1.3.cmml"><mi id="A1.E10.m1.3.3.1.1.1.1.1.3.2" xref="A1.E10.m1.3.3.1.1.1.1.1.3.2.cmml">d</mi><mrow id="A1.E10.m1.3.3.1.1.1.1.1.3.3" xref="A1.E10.m1.3.3.1.1.1.1.1.3.3.cmml"><mo id="A1.E10.m1.3.3.1.1.1.1.1.3.3a" xref="A1.E10.m1.3.3.1.1.1.1.1.3.3.cmml">−</mo><mfrac id="A1.E10.m1.3.3.1.1.1.1.1.3.3.2" xref="A1.E10.m1.3.3.1.1.1.1.1.3.3.2.cmml"><mn id="A1.E10.m1.3.3.1.1.1.1.1.3.3.2.2" xref="A1.E10.m1.3.3.1.1.1.1.1.3.3.2.2.cmml">1</mn><mn id="A1.E10.m1.3.3.1.1.1.1.1.3.3.2.3" xref="A1.E10.m1.3.3.1.1.1.1.1.3.3.2.3.cmml">2</mn></mfrac></mrow></msup></mrow><mo stretchy="false" id="A1.E10.m1.3.3.1.1.1.1.3" xref="A1.E10.m1.3.3.1.1.2.cmml">)</mo></mrow></mrow></mrow><mo id="A1.E10.m1.4.4.3" xref="A1.E10.m1.4.4.3.cmml">≥</mo><mrow id="A1.E10.m1.4.4.2" xref="A1.E10.m1.4.4.2.cmml"><mi id="A1.E10.m1.4.4.2.3" xref="A1.E10.m1.4.4.2.3.cmml">α</mi><mo lspace="0em" rspace="0em" id="A1.E10.m1.4.4.2.2" xref="A1.E10.m1.4.4.2.2.cmml">​</mo><mrow id="A1.E10.m1.4.4.2.1" xref="A1.E10.m1.4.4.2.1.cmml"><munderover id="A1.E10.m1.4.4.2.1.2" xref="A1.E10.m1.4.4.2.1.2.cmml"><mo movablelimits="false" id="A1.E10.m1.4.4.2.1.2.2.2" xref="A1.E10.m1.4.4.2.1.2.2.2.cmml">∑</mo><mrow id="A1.E10.m1.4.4.2.1.2.2.3" xref="A1.E10.m1.4.4.2.1.2.2.3.cmml"><mi id="A1.E10.m1.4.4.2.1.2.2.3.2" xref="A1.E10.m1.4.4.2.1.2.2.3.2.cmml">j</mi><mo id="A1.E10.m1.4.4.2.1.2.2.3.1" xref="A1.E10.m1.4.4.2.1.2.2.3.1.cmml">=</mo><mn id="A1.E10.m1.4.4.2.1.2.2.3.3" xref="A1.E10.m1.4.4.2.1.2.2.3.3.cmml">1</mn></mrow><mi id="A1.E10.m1.4.4.2.1.2.3" xref="A1.E10.m1.4.4.2.1.2.3.cmml">n</mi></munderover><mrow id="A1.E10.m1.4.4.2.1.1.1" xref="A1.E10.m1.4.4.2.1.1.2.cmml"><mi id="A1.E10.m1.2.2" xref="A1.E10.m1.2.2.cmml">exp</mi><mo id="A1.E10.m1.4.4.2.1.1.1a" xref="A1.E10.m1.4.4.2.1.1.2.cmml">⁡</mo><mrow id="A1.E10.m1.4.4.2.1.1.1.1" xref="A1.E10.m1.4.4.2.1.1.2.cmml"><mo stretchy="false" id="A1.E10.m1.4.4.2.1.1.1.1.2" xref="A1.E10.m1.4.4.2.1.1.2.cmml">(</mo><mrow id="A1.E10.m1.4.4.2.1.1.1.1.1" xref="A1.E10.m1.4.4.2.1.1.1.1.1.cmml"><mrow id="A1.E10.m1.4.4.2.1.1.1.1.1.2" xref="A1.E10.m1.4.4.2.1.1.1.1.1.2.cmml"><mi id="A1.E10.m1.4.4.2.1.1.1.1.1.2.2" xref="A1.E10.m1.4.4.2.1.1.1.1.1.2.2.cmml">q</mi><mo lspace="0.222em" rspace="0.222em" id="A1.E10.m1.4.4.2.1.1.1.1.1.2.1" xref="A1.E10.m1.4.4.2.1.1.1.1.1.2.1.cmml">⋅</mo><msub id="A1.E10.m1.4.4.2.1.1.1.1.1.2.3" xref="A1.E10.m1.4.4.2.1.1.1.1.1.2.3.cmml"><mi id="A1.E10.m1.4.4.2.1.1.1.1.1.2.3.2" xref="A1.E10.m1.4.4.2.1.1.1.1.1.2.3.2.cmml">k</mi><mi id="A1.E10.m1.4.4.2.1.1.1.1.1.2.3.3" xref="A1.E10.m1.4.4.2.1.1.1.1.1.2.3.3.cmml">j</mi></msub></mrow><mo lspace="0em" rspace="0em" id="A1.E10.m1.4.4.2.1.1.1.1.1.1" xref="A1.E10.m1.4.4.2.1.1.1.1.1.1.cmml">​</mo><msup id="A1.E10.m1.4.4.2.1.1.1.1.1.3" xref="A1.E10.m1.4.4.2.1.1.1.1.1.3.cmml"><mi id="A1.E10.m1.4.4.2.1.1.1.1.1.3.2" xref="A1.E10.m1.4.4.2.1.1.1.1.1.3.2.cmml">d</mi><mrow id="A1.E10.m1.4.4.2.1.1.1.1.1.3.3" xref="A1.E10.m1.4.4.2.1.1.1.1.1.3.3.cmml"><mo id="A1.E10.m1.4.4.2.1.1.1.1.1.3.3a" xref="A1.E10.m1.4.4.2.1.1.1.1.1.3.3.cmml">−</mo><mfrac id="A1.E10.m1.4.4.2.1.1.1.1.1.3.3.2" xref="A1.E10.m1.4.4.2.1.1.1.1.1.3.3.2.cmml"><mn id="A1.E10.m1.4.4.2.1.1.1.1.1.3.3.2.2" xref="A1.E10.m1.4.4.2.1.1.1.1.1.3.3.2.2.cmml">1</mn><mn id="A1.E10.m1.4.4.2.1.1.1.1.1.3.3.2.3" xref="A1.E10.m1.4.4.2.1.1.1.1.1.3.3.2.3.cmml">2</mn></mfrac></mrow></msup></mrow><mo stretchy="false" id="A1.E10.m1.4.4.2.1.1.1.1.3" xref="A1.E10.m1.4.4.2.1.1.2.cmml">)</mo></mrow></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E10.m1.4b"><apply id="A1.E10.m1.4.4.cmml" xref="A1.E10.m1.4.4"><geq id="A1.E10.m1.4.4.3.cmml" xref="A1.E10.m1.4.4.3"></geq><apply id="A1.E10.m1.3.3.1.cmml" xref="A1.E10.m1.3.3.1"><apply id="A1.E10.m1.3.3.1.2.cmml" xref="A1.E10.m1.3.3.1.2"><csymbol cd="ambiguous" id="A1.E10.m1.3.3.1.2.1.cmml" xref="A1.E10.m1.3.3.1.2">subscript</csymbol><sum id="A1.E10.m1.3.3.1.2.2.cmml" xref="A1.E10.m1.3.3.1.2.2"></sum><apply id="A1.E10.m1.3.3.1.2.3.cmml" xref="A1.E10.m1.3.3.1.2.3"><in id="A1.E10.m1.3.3.1.2.3.1.cmml" xref="A1.E10.m1.3.3.1.2.3.1"></in><ci id="A1.E10.m1.3.3.1.2.3.2.cmml" xref="A1.E10.m1.3.3.1.2.3.2">𝑗</ci><apply id="A1.E10.m1.3.3.1.2.3.3.cmml" xref="A1.E10.m1.3.3.1.2.3.3"><csymbol cd="ambiguous" id="A1.E10.m1.3.3.1.2.3.3.1.cmml" xref="A1.E10.m1.3.3.1.2.3.3">superscript</csymbol><ci id="A1.E10.m1.3.3.1.2.3.3.2.cmml" xref="A1.E10.m1.3.3.1.2.3.3.2">𝐾</ci><ci id="A1.E10.m1.3.3.1.2.3.3.3.cmml" xref="A1.E10.m1.3.3.1.2.3.3.3">′</ci></apply></apply></apply><apply id="A1.E10.m1.3.3.1.1.2.cmml" xref="A1.E10.m1.3.3.1.1.1"><exp id="A1.E10.m1.1.1.cmml" xref="A1.E10.m1.1.1"></exp><apply id="A1.E10.m1.3.3.1.1.1.1.1.cmml" xref="A1.E10.m1.3.3.1.1.1.1.1"><times id="A1.E10.m1.3.3.1.1.1.1.1.1.cmml" xref="A1.E10.m1.3.3.1.1.1.1.1.1"></times><apply id="A1.E10.m1.3.3.1.1.1.1.1.2.cmml" xref="A1.E10.m1.3.3.1.1.1.1.1.2"><ci id="A1.E10.m1.3.3.1.1.1.1.1.2.1.cmml" xref="A1.E10.m1.3.3.1.1.1.1.1.2.1">⋅</ci><ci id="A1.E10.m1.3.3.1.1.1.1.1.2.2.cmml" xref="A1.E10.m1.3.3.1.1.1.1.1.2.2">𝑞</ci><apply id="A1.E10.m1.3.3.1.1.1.1.1.2.3.cmml" xref="A1.E10.m1.3.3.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="A1.E10.m1.3.3.1.1.1.1.1.2.3.1.cmml" xref="A1.E10.m1.3.3.1.1.1.1.1.2.3">subscript</csymbol><ci id="A1.E10.m1.3.3.1.1.1.1.1.2.3.2.cmml" xref="A1.E10.m1.3.3.1.1.1.1.1.2.3.2">𝑘</ci><ci id="A1.E10.m1.3.3.1.1.1.1.1.2.3.3.cmml" xref="A1.E10.m1.3.3.1.1.1.1.1.2.3.3">𝑗</ci></apply></apply><apply id="A1.E10.m1.3.3.1.1.1.1.1.3.cmml" xref="A1.E10.m1.3.3.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.E10.m1.3.3.1.1.1.1.1.3.1.cmml" xref="A1.E10.m1.3.3.1.1.1.1.1.3">superscript</csymbol><ci id="A1.E10.m1.3.3.1.1.1.1.1.3.2.cmml" xref="A1.E10.m1.3.3.1.1.1.1.1.3.2">𝑑</ci><apply id="A1.E10.m1.3.3.1.1.1.1.1.3.3.cmml" xref="A1.E10.m1.3.3.1.1.1.1.1.3.3"><minus id="A1.E10.m1.3.3.1.1.1.1.1.3.3.1.cmml" xref="A1.E10.m1.3.3.1.1.1.1.1.3.3"></minus><apply id="A1.E10.m1.3.3.1.1.1.1.1.3.3.2.cmml" xref="A1.E10.m1.3.3.1.1.1.1.1.3.3.2"><divide id="A1.E10.m1.3.3.1.1.1.1.1.3.3.2.1.cmml" xref="A1.E10.m1.3.3.1.1.1.1.1.3.3.2"></divide><cn type="integer" id="A1.E10.m1.3.3.1.1.1.1.1.3.3.2.2.cmml" xref="A1.E10.m1.3.3.1.1.1.1.1.3.3.2.2">1</cn><cn type="integer" id="A1.E10.m1.3.3.1.1.1.1.1.3.3.2.3.cmml" xref="A1.E10.m1.3.3.1.1.1.1.1.3.3.2.3">2</cn></apply></apply></apply></apply></apply></apply><apply id="A1.E10.m1.4.4.2.cmml" xref="A1.E10.m1.4.4.2"><times id="A1.E10.m1.4.4.2.2.cmml" xref="A1.E10.m1.4.4.2.2"></times><ci id="A1.E10.m1.4.4.2.3.cmml" xref="A1.E10.m1.4.4.2.3">𝛼</ci><apply id="A1.E10.m1.4.4.2.1.cmml" xref="A1.E10.m1.4.4.2.1"><apply id="A1.E10.m1.4.4.2.1.2.cmml" xref="A1.E10.m1.4.4.2.1.2"><csymbol cd="ambiguous" id="A1.E10.m1.4.4.2.1.2.1.cmml" xref="A1.E10.m1.4.4.2.1.2">superscript</csymbol><apply id="A1.E10.m1.4.4.2.1.2.2.cmml" xref="A1.E10.m1.4.4.2.1.2"><csymbol cd="ambiguous" id="A1.E10.m1.4.4.2.1.2.2.1.cmml" xref="A1.E10.m1.4.4.2.1.2">subscript</csymbol><sum id="A1.E10.m1.4.4.2.1.2.2.2.cmml" xref="A1.E10.m1.4.4.2.1.2.2.2"></sum><apply id="A1.E10.m1.4.4.2.1.2.2.3.cmml" xref="A1.E10.m1.4.4.2.1.2.2.3"><eq id="A1.E10.m1.4.4.2.1.2.2.3.1.cmml" xref="A1.E10.m1.4.4.2.1.2.2.3.1"></eq><ci id="A1.E10.m1.4.4.2.1.2.2.3.2.cmml" xref="A1.E10.m1.4.4.2.1.2.2.3.2">𝑗</ci><cn type="integer" id="A1.E10.m1.4.4.2.1.2.2.3.3.cmml" xref="A1.E10.m1.4.4.2.1.2.2.3.3">1</cn></apply></apply><ci id="A1.E10.m1.4.4.2.1.2.3.cmml" xref="A1.E10.m1.4.4.2.1.2.3">𝑛</ci></apply><apply id="A1.E10.m1.4.4.2.1.1.2.cmml" xref="A1.E10.m1.4.4.2.1.1.1"><exp id="A1.E10.m1.2.2.cmml" xref="A1.E10.m1.2.2"></exp><apply id="A1.E10.m1.4.4.2.1.1.1.1.1.cmml" xref="A1.E10.m1.4.4.2.1.1.1.1.1"><times id="A1.E10.m1.4.4.2.1.1.1.1.1.1.cmml" xref="A1.E10.m1.4.4.2.1.1.1.1.1.1"></times><apply id="A1.E10.m1.4.4.2.1.1.1.1.1.2.cmml" xref="A1.E10.m1.4.4.2.1.1.1.1.1.2"><ci id="A1.E10.m1.4.4.2.1.1.1.1.1.2.1.cmml" xref="A1.E10.m1.4.4.2.1.1.1.1.1.2.1">⋅</ci><ci id="A1.E10.m1.4.4.2.1.1.1.1.1.2.2.cmml" xref="A1.E10.m1.4.4.2.1.1.1.1.1.2.2">𝑞</ci><apply id="A1.E10.m1.4.4.2.1.1.1.1.1.2.3.cmml" xref="A1.E10.m1.4.4.2.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="A1.E10.m1.4.4.2.1.1.1.1.1.2.3.1.cmml" xref="A1.E10.m1.4.4.2.1.1.1.1.1.2.3">subscript</csymbol><ci id="A1.E10.m1.4.4.2.1.1.1.1.1.2.3.2.cmml" xref="A1.E10.m1.4.4.2.1.1.1.1.1.2.3.2">𝑘</ci><ci id="A1.E10.m1.4.4.2.1.1.1.1.1.2.3.3.cmml" xref="A1.E10.m1.4.4.2.1.1.1.1.1.2.3.3">𝑗</ci></apply></apply><apply id="A1.E10.m1.4.4.2.1.1.1.1.1.3.cmml" xref="A1.E10.m1.4.4.2.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.E10.m1.4.4.2.1.1.1.1.1.3.1.cmml" xref="A1.E10.m1.4.4.2.1.1.1.1.1.3">superscript</csymbol><ci id="A1.E10.m1.4.4.2.1.1.1.1.1.3.2.cmml" xref="A1.E10.m1.4.4.2.1.1.1.1.1.3.2">𝑑</ci><apply id="A1.E10.m1.4.4.2.1.1.1.1.1.3.3.cmml" xref="A1.E10.m1.4.4.2.1.1.1.1.1.3.3"><minus id="A1.E10.m1.4.4.2.1.1.1.1.1.3.3.1.cmml" xref="A1.E10.m1.4.4.2.1.1.1.1.1.3.3"></minus><apply id="A1.E10.m1.4.4.2.1.1.1.1.1.3.3.2.cmml" xref="A1.E10.m1.4.4.2.1.1.1.1.1.3.3.2"><divide id="A1.E10.m1.4.4.2.1.1.1.1.1.3.3.2.1.cmml" xref="A1.E10.m1.4.4.2.1.1.1.1.1.3.3.2"></divide><cn type="integer" id="A1.E10.m1.4.4.2.1.1.1.1.1.3.3.2.2.cmml" xref="A1.E10.m1.4.4.2.1.1.1.1.1.3.3.2.2">1</cn><cn type="integer" id="A1.E10.m1.4.4.2.1.1.1.1.1.3.3.2.3.cmml" xref="A1.E10.m1.4.4.2.1.1.1.1.1.3.3.2.3">2</cn></apply></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E10.m1.4c">\sum_{j\in K^{\prime}}\exp(q\cdot k_{j}~{}d^{-\frac{1}{2}})\geq\alpha\sum_{j=1}^{n}\exp(q\cdot k_{j}~{}d^{-\frac{1}{2}})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(10)</span></td>
</tr></tbody>
</table>
<p id="A1.SSx1.SSS0.Px1.p1.4" class="ltx_p"><span id="A1.SSx1.SSS0.Px1.p1.4.1" class="ltx_text ltx_font_typewriter">for some </span><math id="A1.SSx1.SSS0.Px1.p1.4.m1.1" class="ltx_Math" alttext="\alpha\approx 1" display="inline"><semantics id="A1.SSx1.SSS0.Px1.p1.4.m1.1a"><mrow id="A1.SSx1.SSS0.Px1.p1.4.m1.1.1" xref="A1.SSx1.SSS0.Px1.p1.4.m1.1.1.cmml"><mi id="A1.SSx1.SSS0.Px1.p1.4.m1.1.1.2" xref="A1.SSx1.SSS0.Px1.p1.4.m1.1.1.2.cmml">α</mi><mo id="A1.SSx1.SSS0.Px1.p1.4.m1.1.1.1" xref="A1.SSx1.SSS0.Px1.p1.4.m1.1.1.1.cmml">≈</mo><mn id="A1.SSx1.SSS0.Px1.p1.4.m1.1.1.3" xref="A1.SSx1.SSS0.Px1.p1.4.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SSx1.SSS0.Px1.p1.4.m1.1b"><apply id="A1.SSx1.SSS0.Px1.p1.4.m1.1.1.cmml" xref="A1.SSx1.SSS0.Px1.p1.4.m1.1.1"><approx id="A1.SSx1.SSS0.Px1.p1.4.m1.1.1.1.cmml" xref="A1.SSx1.SSS0.Px1.p1.4.m1.1.1.1"></approx><ci id="A1.SSx1.SSS0.Px1.p1.4.m1.1.1.2.cmml" xref="A1.SSx1.SSS0.Px1.p1.4.m1.1.1.2">𝛼</ci><cn type="integer" id="A1.SSx1.SSS0.Px1.p1.4.m1.1.1.3.cmml" xref="A1.SSx1.SSS0.Px1.p1.4.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SSx1.SSS0.Px1.p1.4.m1.1c">\alpha\approx 1</annotation></semantics></math><span id="A1.SSx1.SSS0.Px1.p1.4.2" class="ltx_text ltx_font_typewriter">, typically very close to 1.</span></p>
</div>
<div id="A1.SSx1.SSS0.Px1.p2" class="ltx_para ltx_noindent">
<p id="A1.SSx1.SSS0.Px1.p2.1" class="ltx_p"><span id="A1.SSx1.SSS0.Px1.p2.1.1" class="ltx_text ltx_font_bold">Proof</span><span id="A1.SSx1.SSS0.Px1.p2.1.2" class="ltx_text ltx_font_typewriter">: This follows from the exponential dominance assumption and the nature of the exponential function, which is sharply peaked.</span></p>
</div>
</section>
<section id="A1.SSx1.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_font_typewriter ltx_title_paragraph">Lemma 2: Approximation of Output Vector</h4>

<div id="A1.SSx1.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="A1.SSx1.SSS0.Px2.p1.3" class="ltx_p"><span id="A1.SSx1.SSS0.Px2.p1.3.1" class="ltx_text ltx_font_typewriter">Given the dominance of </span><math id="A1.SSx1.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="K^{\prime}" display="inline"><semantics id="A1.SSx1.SSS0.Px2.p1.1.m1.1a"><msup id="A1.SSx1.SSS0.Px2.p1.1.m1.1.1" xref="A1.SSx1.SSS0.Px2.p1.1.m1.1.1.cmml"><mi id="A1.SSx1.SSS0.Px2.p1.1.m1.1.1.2" xref="A1.SSx1.SSS0.Px2.p1.1.m1.1.1.2.cmml">K</mi><mo id="A1.SSx1.SSS0.Px2.p1.1.m1.1.1.3" xref="A1.SSx1.SSS0.Px2.p1.1.m1.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="A1.SSx1.SSS0.Px2.p1.1.m1.1b"><apply id="A1.SSx1.SSS0.Px2.p1.1.m1.1.1.cmml" xref="A1.SSx1.SSS0.Px2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A1.SSx1.SSS0.Px2.p1.1.m1.1.1.1.cmml" xref="A1.SSx1.SSS0.Px2.p1.1.m1.1.1">superscript</csymbol><ci id="A1.SSx1.SSS0.Px2.p1.1.m1.1.1.2.cmml" xref="A1.SSx1.SSS0.Px2.p1.1.m1.1.1.2">𝐾</ci><ci id="A1.SSx1.SSS0.Px2.p1.1.m1.1.1.3.cmml" xref="A1.SSx1.SSS0.Px2.p1.1.m1.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SSx1.SSS0.Px2.p1.1.m1.1c">K^{\prime}</annotation></semantics></math><span id="A1.SSx1.SSS0.Px2.p1.3.2" class="ltx_text ltx_font_typewriter"> as shown in Lemma 1, the approximate output </span><math id="A1.SSx1.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="u^{\prime}" display="inline"><semantics id="A1.SSx1.SSS0.Px2.p1.2.m2.1a"><msup id="A1.SSx1.SSS0.Px2.p1.2.m2.1.1" xref="A1.SSx1.SSS0.Px2.p1.2.m2.1.1.cmml"><mi id="A1.SSx1.SSS0.Px2.p1.2.m2.1.1.2" xref="A1.SSx1.SSS0.Px2.p1.2.m2.1.1.2.cmml">u</mi><mo id="A1.SSx1.SSS0.Px2.p1.2.m2.1.1.3" xref="A1.SSx1.SSS0.Px2.p1.2.m2.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="A1.SSx1.SSS0.Px2.p1.2.m2.1b"><apply id="A1.SSx1.SSS0.Px2.p1.2.m2.1.1.cmml" xref="A1.SSx1.SSS0.Px2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="A1.SSx1.SSS0.Px2.p1.2.m2.1.1.1.cmml" xref="A1.SSx1.SSS0.Px2.p1.2.m2.1.1">superscript</csymbol><ci id="A1.SSx1.SSS0.Px2.p1.2.m2.1.1.2.cmml" xref="A1.SSx1.SSS0.Px2.p1.2.m2.1.1.2">𝑢</ci><ci id="A1.SSx1.SSS0.Px2.p1.2.m2.1.1.3.cmml" xref="A1.SSx1.SSS0.Px2.p1.2.m2.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SSx1.SSS0.Px2.p1.2.m2.1c">u^{\prime}</annotation></semantics></math><span id="A1.SSx1.SSS0.Px2.p1.3.3" class="ltx_text ltx_font_typewriter"> effectively represents the full output </span><math id="A1.SSx1.SSS0.Px2.p1.3.m3.1" class="ltx_Math" alttext="u" display="inline"><semantics id="A1.SSx1.SSS0.Px2.p1.3.m3.1a"><mi id="A1.SSx1.SSS0.Px2.p1.3.m3.1.1" xref="A1.SSx1.SSS0.Px2.p1.3.m3.1.1.cmml">u</mi><annotation-xml encoding="MathML-Content" id="A1.SSx1.SSS0.Px2.p1.3.m3.1b"><ci id="A1.SSx1.SSS0.Px2.p1.3.m3.1.1.cmml" xref="A1.SSx1.SSS0.Px2.p1.3.m3.1.1">𝑢</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SSx1.SSS0.Px2.p1.3.m3.1c">u</annotation></semantics></math><span id="A1.SSx1.SSS0.Px2.p1.3.4" class="ltx_text ltx_font_typewriter">:</span></p>
<table id="A1.E11" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.E11.m1.1" class="ltx_Math" alttext="\|u^{\prime}-u\|\leq\epsilon" display="block"><semantics id="A1.E11.m1.1a"><mrow id="A1.E11.m1.1.1" xref="A1.E11.m1.1.1.cmml"><mrow id="A1.E11.m1.1.1.1.1" xref="A1.E11.m1.1.1.1.2.cmml"><mo stretchy="false" id="A1.E11.m1.1.1.1.1.2" xref="A1.E11.m1.1.1.1.2.1.cmml">‖</mo><mrow id="A1.E11.m1.1.1.1.1.1" xref="A1.E11.m1.1.1.1.1.1.cmml"><msup id="A1.E11.m1.1.1.1.1.1.2" xref="A1.E11.m1.1.1.1.1.1.2.cmml"><mi id="A1.E11.m1.1.1.1.1.1.2.2" xref="A1.E11.m1.1.1.1.1.1.2.2.cmml">u</mi><mo id="A1.E11.m1.1.1.1.1.1.2.3" xref="A1.E11.m1.1.1.1.1.1.2.3.cmml">′</mo></msup><mo id="A1.E11.m1.1.1.1.1.1.1" xref="A1.E11.m1.1.1.1.1.1.1.cmml">−</mo><mi id="A1.E11.m1.1.1.1.1.1.3" xref="A1.E11.m1.1.1.1.1.1.3.cmml">u</mi></mrow><mo stretchy="false" id="A1.E11.m1.1.1.1.1.3" xref="A1.E11.m1.1.1.1.2.1.cmml">‖</mo></mrow><mo id="A1.E11.m1.1.1.2" xref="A1.E11.m1.1.1.2.cmml">≤</mo><mi id="A1.E11.m1.1.1.3" xref="A1.E11.m1.1.1.3.cmml">ϵ</mi></mrow><annotation-xml encoding="MathML-Content" id="A1.E11.m1.1b"><apply id="A1.E11.m1.1.1.cmml" xref="A1.E11.m1.1.1"><leq id="A1.E11.m1.1.1.2.cmml" xref="A1.E11.m1.1.1.2"></leq><apply id="A1.E11.m1.1.1.1.2.cmml" xref="A1.E11.m1.1.1.1.1"><csymbol cd="latexml" id="A1.E11.m1.1.1.1.2.1.cmml" xref="A1.E11.m1.1.1.1.1.2">norm</csymbol><apply id="A1.E11.m1.1.1.1.1.1.cmml" xref="A1.E11.m1.1.1.1.1.1"><minus id="A1.E11.m1.1.1.1.1.1.1.cmml" xref="A1.E11.m1.1.1.1.1.1.1"></minus><apply id="A1.E11.m1.1.1.1.1.1.2.cmml" xref="A1.E11.m1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A1.E11.m1.1.1.1.1.1.2.1.cmml" xref="A1.E11.m1.1.1.1.1.1.2">superscript</csymbol><ci id="A1.E11.m1.1.1.1.1.1.2.2.cmml" xref="A1.E11.m1.1.1.1.1.1.2.2">𝑢</ci><ci id="A1.E11.m1.1.1.1.1.1.2.3.cmml" xref="A1.E11.m1.1.1.1.1.1.2.3">′</ci></apply><ci id="A1.E11.m1.1.1.1.1.1.3.cmml" xref="A1.E11.m1.1.1.1.1.1.3">𝑢</ci></apply></apply><ci id="A1.E11.m1.1.1.3.cmml" xref="A1.E11.m1.1.1.3">italic-ϵ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E11.m1.1c">\|u^{\prime}-u\|\leq\epsilon</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(11)</span></td>
</tr></tbody>
</table>
<p id="A1.SSx1.SSS0.Px2.p1.4" class="ltx_p"><span id="A1.SSx1.SSS0.Px2.p1.4.1" class="ltx_text ltx_font_typewriter">where </span><math id="A1.SSx1.SSS0.Px2.p1.4.m1.1" class="ltx_Math" alttext="\epsilon" display="inline"><semantics id="A1.SSx1.SSS0.Px2.p1.4.m1.1a"><mi id="A1.SSx1.SSS0.Px2.p1.4.m1.1.1" xref="A1.SSx1.SSS0.Px2.p1.4.m1.1.1.cmml">ϵ</mi><annotation-xml encoding="MathML-Content" id="A1.SSx1.SSS0.Px2.p1.4.m1.1b"><ci id="A1.SSx1.SSS0.Px2.p1.4.m1.1.1.cmml" xref="A1.SSx1.SSS0.Px2.p1.4.m1.1.1">italic-ϵ</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SSx1.SSS0.Px2.p1.4.m1.1c">\epsilon</annotation></semantics></math><span id="A1.SSx1.SSS0.Px2.p1.4.2" class="ltx_text ltx_font_typewriter"> is a small error term.</span></p>
</div>
<div id="A1.SSx1.SSS0.Px2.p2" class="ltx_para ltx_noindent">
<p id="A1.SSx1.SSS0.Px2.p2.2" class="ltx_p"><span id="A1.SSx1.SSS0.Px2.p2.2.1" class="ltx_text ltx_font_bold">Proof</span><span id="A1.SSx1.SSS0.Px2.p2.2.2" class="ltx_text ltx_font_typewriter">: Follows from the weighted sum structure of </span><math id="A1.SSx1.SSS0.Px2.p2.1.m1.1" class="ltx_Math" alttext="u" display="inline"><semantics id="A1.SSx1.SSS0.Px2.p2.1.m1.1a"><mi id="A1.SSx1.SSS0.Px2.p2.1.m1.1.1" xref="A1.SSx1.SSS0.Px2.p2.1.m1.1.1.cmml">u</mi><annotation-xml encoding="MathML-Content" id="A1.SSx1.SSS0.Px2.p2.1.m1.1b"><ci id="A1.SSx1.SSS0.Px2.p2.1.m1.1.1.cmml" xref="A1.SSx1.SSS0.Px2.p2.1.m1.1.1">𝑢</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SSx1.SSS0.Px2.p2.1.m1.1c">u</annotation></semantics></math><span id="A1.SSx1.SSS0.Px2.p2.2.3" class="ltx_text ltx_font_typewriter"> and </span><math id="A1.SSx1.SSS0.Px2.p2.2.m2.1" class="ltx_Math" alttext="u^{\prime}" display="inline"><semantics id="A1.SSx1.SSS0.Px2.p2.2.m2.1a"><msup id="A1.SSx1.SSS0.Px2.p2.2.m2.1.1" xref="A1.SSx1.SSS0.Px2.p2.2.m2.1.1.cmml"><mi id="A1.SSx1.SSS0.Px2.p2.2.m2.1.1.2" xref="A1.SSx1.SSS0.Px2.p2.2.m2.1.1.2.cmml">u</mi><mo id="A1.SSx1.SSS0.Px2.p2.2.m2.1.1.3" xref="A1.SSx1.SSS0.Px2.p2.2.m2.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="A1.SSx1.SSS0.Px2.p2.2.m2.1b"><apply id="A1.SSx1.SSS0.Px2.p2.2.m2.1.1.cmml" xref="A1.SSx1.SSS0.Px2.p2.2.m2.1.1"><csymbol cd="ambiguous" id="A1.SSx1.SSS0.Px2.p2.2.m2.1.1.1.cmml" xref="A1.SSx1.SSS0.Px2.p2.2.m2.1.1">superscript</csymbol><ci id="A1.SSx1.SSS0.Px2.p2.2.m2.1.1.2.cmml" xref="A1.SSx1.SSS0.Px2.p2.2.m2.1.1.2">𝑢</ci><ci id="A1.SSx1.SSS0.Px2.p2.2.m2.1.1.3.cmml" xref="A1.SSx1.SSS0.Px2.p2.2.m2.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SSx1.SSS0.Px2.p2.2.m2.1c">u^{\prime}</annotation></semantics></math><span id="A1.SSx1.SSS0.Px2.p2.2.4" class="ltx_text ltx_font_typewriter">, using the bounds established in Lemma 1.</span></p>
</div>
<div id="A1.SSx1.SSS0.Px2.p3" class="ltx_para ltx_noindent">
<p id="A1.SSx1.SSS0.Px2.p3.1" class="ltx_p"><span id="A1.SSx1.SSS0.Px2.p3.1.1" class="ltx_text ltx_font_typewriter">Given the lemmas and under the stated assumptions, the k-NN retrieval mechanism within a key-value cache effectively approximates the softmax attention mechanism in transformers. This proof highlights the efficiency versus accuracy trade-off inherent in using approximate methods like k-NN retrieval.</span></p>
</div>
</section>
</section>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2407.09449" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2407.09450" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2407.09450">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2407.09450" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2407.09451" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Mon Aug  5 18:49:19 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2407.09450
Large language models (LLMs)  have shown remarkable capabilities, but still struggle with processing extensive contexts, limiting their ability to maintain coherence and accuracy over long sequences. In contrast, the human brain excels at organising and retrieving episodic experiences across vast temporal scales, spanning a lifetime. In this work, we introduce EM-LLM, a novel approach that integrates key aspects of human episodic memory and event cognition into LLMs, enabling them to effectively handle practically infinite context lengths while maintaining computational efficiency. EM-LLM organises sequences of tokens into coherent episodic events using a combination of Bayesian surprise and graph-theoretic boundary refinement in an on-line fashion. When needed, these events are retrieved through a two-stage memory process, combining similarity-based and temporally contiguous retrieval for efficient and human-like access to relevant information. Experiments on the LongBench dataset demonstrate EM-LLM’s superior performance, outperforming the state-of-the-art InfLLM model with an overall relative improvement ofacross various tasks, including aimprovement on the PassageRetrieval task. Furthermore, our analysis reveals strong correlations between EM-LLM’s event segmentation and human-perceived events, suggesting a bridge between this artificial system and its biological counterpart. This work not only advances LLM capabilities in processing extended contexts but also provides a computational framework for exploring human memory mechanisms, opening new avenues for interdisciplinary research in AI and cognitive science.
abstract
Human-like Episodic Memory for Infinite Context LLMs
Abstract
2407.09450
1IntroductionFor contemporary pre-trained large language models (LLMs) , the context window serves as the primary mechanism to incorporate domain-specific, private, or common up-to-date information. However, despite their remarkable and ever-expanding capabilities, LLMs still exhibit significant limitations when tasked with processing extensive contexts. These limitations stem from inherent challenges in Transformer-based architectures. Recent studies have shown that Transformers struggle with extrapolating to contexts longer than their training window size. On top of this, employing softmax attention over extended token sequences requires substantial computational resources for each token generation, and the resulting attention embeddings risk becoming excessively noisy and losing their distinctiveness.To mitigate those challenges, recent works have focused on retrieval-based methods, either in the form of in-context augmentation (e.g., RAG-based techniques)  or via retrieval of previously-inferred key-value pairs (KV)  within individual attention heads. Notably, state-of-the-art performance is achieved when KV pairs are initially organised into non-overlapping segments and then retrieved together as one block of sequential tokens. While such techniques present interesting avenues of research, results still indicate a significant gap between the performance of LLMs in short- vs long-context tasks, even when existing long-context architectures are employed.This work tackles the above challenges and attempts to bridge this performance gap by taking inspiration from the algorithmic interpretation ofepisodic memory in the human brain-- the memory system responsible for encoding, storing, and retrieving personal experiences and events. The brain makes sense of its continuous experience in the real world by segmenting it into discrete episodic events, which are organised in a hierarchical and nested-timescale structureand stored in long-term memory. Notably, the boundaries between such events are the access points when it comes to memory retrievaland are widely believed to correspond to points in time with high prediction errors between the brain’s generative model and its raw sensory input (a.k.a.,surprise) . In this context, surprise refers to moments when the brain’s predictions about incoming sensory information are significantly violated, leading to a mismatch between what is expected and what is actually perceived. These instances of high surprise are thought to signal important changes in the environment or narrative, prompting the brain to segment the ongoing experience into distinct events. Once segmented and stored, the brain can recall episodic memories based on their similarity to its current experience, recency, original temporal order, and their proximity to other recalled memories (temporal asymmetry and contiguity,) .Following these insights, we propose a novel architecture,EM-LLM, that integrates crucial aspects of event cognition and episodic memory into Transformer-based LLMs. For memory formation, we segment the sequence of the tokens presented to the underlying LLM into individual memory units representing episodic events. The boundaries, and thus the size of those events, are initially determined dynamically, based on the level of surprise of the model during inference, and then refined to maximise cohesion within memory units and separation of memory content across them (see Section3.2) . This refinement process leverages graph-theoretic metrics, treating the similarity between attention keys (the learned representations used in Transformer self-attention mechanisms)  as a weighted adjacency matrix, and aims to enhance the model’s ability to efficiently recall relevant information when addressing complex tasks with extended contexts.
Importantly, this memory formation process incurs minimal additional computational cost, with the surprise-based segmentation requiring no extra computation and the refinement step having a complexity of, whereis typically very small compared to the number of tokens.
With this efficient memory formation process, by grouping similar information in single units, we minimise the number of units needed to recall details around specific events.
For memory recall, our approach integrates similarity-based retrieval with mechanisms that facilitate temporal contiguity and asymmetry effects. By retrieving and buffering salient memory units, our model leverages and enhances the recently discovered propensity of LLMs to exhibit human-like patterns in sequential information retrieval. This method not only ensures efficient access to pertinent information but also mimics the temporal dynamics found in human free recall studies (such as) , further enhancing the model’s ability to handle complex tasks that require nuanced temporal reasoning.To prove our hypotheses, we first employ a series of human-annotated podcast scripts, where we show that information in LLM attention heads can be semantically grouped in a way that correlates with the event structure perceived by humans. Therefore, LLM-perceived surprise can indeed serve as a proxy for the cognitive signals that drive human event segmentation, as confirmed by previous works. Then, using the long-context PG-19 dataset, which comprises a diverse corpus of English books, we evaluate the effectiveness of both steps in our segmentation method for grouping relevant information, and assess the performance of different refinement objectives. Finally, we show that our method is scalable and significantly outperforms the state-of-the-art model InfLLMon the widely-used LongBench benchmarkfor long-context tasks, achieving an overall relative improvement of, including a substantialimprovement on the PassageRetrieval task.
introduction
Human-like Episodic Memory for Infinite Context LLMs
Section 1
2407.09450
2Related work2.1Long-context in LLMsRecently, several approaches have been proposed to extend the context window of Transformer-based models. Those include methods that address the limited representational capacity of softmax attention, and its quadratic computational and memory cost. Other methods target the poor extrapolation of typical positional encodings to out-of-distribution (OOD)  context lengths. The latter is evident in most widely used methods, including the original absolute positional encodingsand the more recent relative positional encodings, such as the popular Rotary Positional Embeddings (RoPE) . To address this, some approaches propose scaling of the rotation anglesor the base constant. Others, scale positions without affecting the embedding function, exploring alternative strategies such as KERPLEand FIREor adopt mechanisms from certain LMs like T5.Concerning computational efficiency and diluted attention, successful approaches propose methods for general improvements to the efficiency of Transformers through optimised computationsor compression techniques, as well as training methods tailored for long-context scenarios. Another direction is the utilisation of retrieval-based methods, the vast majority of which relies on a vector database that keeps a key-value cache and scalable approximations of k-nearest neighbors (k-NNs)  to perform lookups. Interestingly, since using a key-value cache with k-NN lookup can be seen as an approximation of applying softmax attention to the full token sequence (see AppendixA.4) , k-NN retrieval methods can be used without any fine-tuning. For an exception that does not rely on k-NNs, see.A recent and interesting variant of k-NN retrieval involves retrieving large groups of tokens, rather than individual ones. Models that rely on this approach include SLEDand the more recent InfLLM, which achieves state-of-the-art performance on long-context benchmarks. InfLLM segments the entire context length into fixed-size memory units and employs k-NN lookup using the tokens with the highest accumulated scores per unit. This can be seen as a form of hierarchical attention, as illustrated in Fig.1. While group-based retrieval represents a promising direction, our approach significantly advances this concept by dynamically determining token groupings in a manner akin to human memory formation, addressing a fundamental limitation of InfLLM’s fixed-size segmentation and enabling more adaptive and context-sensitive processing of extended information.Figure 1:Group-based-NN retrieval can be seen as a form of hierarchical episodic attention. Initially,groups of tokens are selected (left)  and then used for softmax attention (right) , as if all other similarity scores were forced to be zero (non-shaded areas of the left curve) . This framework can support multiple levels of episodic attention.2.2Neural models of Episodic Memory and Event CognitionThe concept of episodic memory, central to our approach, has been extensively studied in both theoretical neuroscience and machine learning. Neural models of episodic memory capture human behaviour and neuroimaging data, providing insights into how the brain processes and stores experiences and suggesting links between memory, efficient representations and navigation of physical and conceptual spaces.
In machine learning, episodic memory-inspired approaches have yielded significant improvements across various domains. For instance, episodic control has enhanced reinforcement learning agents’ performance and learning speed. In addition, models of memory construction and consolidation have been successful in alleviating catastrophic forgetting in neural networks, including LLMs, and appear to explain key features of human memory, such as imagination and future thinking.These models have revealed key aspects of episodic memory, particularly in describing how experiences are segmented into events, and when new memories are encoded and retrieved. Surprise plays a critical role in this process, triggering event boundaries and memory formation. This event-based structure is deeply intertwined with our perception of time, highlighting the interdependence of memory and temporal cognition. This insight has helped generative models for videoand reinforcement learningto capture temporal dynamics more accurately. In terms of memory retrieval, studies in human free recall have shown a distinctive increased likelihood of retrieving items encoded close together in time (temporal contiguity)  and in succession (temporal asymmetry)  (see Fig.2A) . Recently, it was shown that attention heads in transformer-based LLMs that are associated with in-context learning, already exhibit the same dynamic retrieval behaviour(Fig.2B)  including both contiguity and asymmetry effects. Therefore, transformers have the inherent ability to act as episodic memory retrieval models, if provided with the right information within their context window. Our work leverages these concepts of surprise-based event segmentation and LLMs’ inherent temporal contiguity and asymmetry effects to enable a new generation of Infinite Context-Length LLMs, capable of processing and understanding information over vastly extended timescales.Figure 2:(A)  Example of the temporal contiguity and asymmetry effect in human free recall. Data averaged over several large free recall studies (adopted from) . (B)  The attention scores of a GPT2 head averaged over all tokens tested. Figure adopted from. (C)  Schematic illustrating our proposed process for memory formation and retrieval: ① Input sequence with surprise-based segmentation (purple arrows indicate high surprise) . ② Formation of episodic memories: input is segmented into events and stored, with initial tokens and local context preserved. Note that the boundary refinement process is not shown here for clarity. ③ Memory retrieval via k-NN search, selecting contiguous events from episodic memory. ④ Final context window structure, comprising initial tokens, contiguity buffer (populated by neighboring events) , similarity buffer (from k-NN retrieval) , and local context.
body
Human-like Episodic Memory for Infinite Context LLMs
Section 2
2407.09450
3EM-LLM: LLM with Episodic Memory3.1ArchitectureEM-LLM is designed to be applied directly to pre-trained LLMs, enabling them to handle context lengths significantly larger than their original training length. Our architecture divides the context into three distinct groups: initial tokens, evicted tokens, and local context. This structure, while incorporating insights from recent work on token block retrieval, introduces novel elements inspired by human episodic memory.The local context represents the most recent tokens, maximising information about the current task, and fits within the typical context window of the underlying LLM. This group utilises full softmax attention and plays a role similar to the focus of attention in cognitive models of working memory, holding the most immediately relevant information for the current task. The evicted tokens typically comprise the majority of past tokens in a long-context scenario, extending far beyond the LLM’s original training length. These tokens are managed by our proposed memory model functioning similarly to short-term episodic memory in the brain. Finally, following previous work, we also maintain a group ofinitial tokens in the LLM context. These act asattention sinksand help recover the performance of window attention, as first observed byand later adopted by. For retrieved tokens, which are therefore discontinuous and outside the local context, we assign a fixed position embedding as in. This architecture enables EM-LLM to effectively process and utilise information from positions outside its pre-trained local context window, while maintaining the underlying LLM’s performance characteristics.3.2Memory formation via SurpriseIn the context of LLMs, we define episodic memory as the organised, event-based collection of past key-value pairs, analogous to the latent representations of personal experiences in human memory. Just as unexpected or novel information plays a crucial role in human memory formation, we posit that analogous indicators of novelty in LLMs can serve as an effective proxy for identifying significant ‘‘events’’ within the model’s experience. In Bayesian terms, surprise is quantified by the negative log-likelihood of observing the current, ground-truth token given the previous tokens in an auto-regressive model, with high values indicating the unpredictability or novelty of each new token within the context according to the model, i.e., it is ‘‘surprised’’ by the next token.
Following work on cognitive modelling, we employ a thresholding mechanism to perform an initial identification of event boundaries (used for the first time in LLMs) . Formally, a tokenis considered a potential boundary if its surprise value exceeds a threshold:(1) whereandare the mean and variance of surprise for a window offset, andis a scaling factor. The choice of thresholdis critical in balancing the granularity of segmentation with the model’s sensitivity to contextual shifts. If theis too high, we will identify very few event boundaries, especially if the local context contains few surprising tokens. Conversely, a lowresults in frequent boundary identification. Using a moving window ensures thatadapts to contextual shifts, minimizing the need for manual tuning while maintaining control over threshold sensitivity via. We also explored a fixed threshold approach () , though our primary focus remained on the dynamic threshold due to its adaptability to varying contexts. This initial segmentation results in a set of potential event boundaries, where eachrepresents the index of a token exceeding the surprise threshold. These boundaries serve as the starting point for our subsequent refinement process, which aims to optimise the intra-event coherence and inter-event distinctiveness of the resulting memory segments.3.3Boundary refinementWhile surprise-based segmentation provides an effective initial estimate of event boundaries, we make the key observation that the utility of elements within an event during memory recall depends on their likelihood of being utilised by the current query. Therefore, we theorise that memory recall will be most efficient with high intra-event similarity between keys while maintaining low inter-event similarity. For instance, see the similarity of groups in Fig.1. To further ensure this, we introduce a boundary refinement step which looks to optimise this objective. Such an objective is typically optimised in the context of graph-clustering, hence we will express this refinement process in a graph-theoretic manner. To achieve this, we treat the similarity matrix between all keys of an attention headwithin the local context window for tokensas an adjacency matrix. We define the adjacency matrixas(2) whereandare the key vectors corresponding to tokensand, respectively. The similarity function measures the closeness of two key vectors; in our implementation, we use dot product similaritydue to its effectiveness in capturing semantic relationships in high-dimensional spacesand to align with the mechanism of self-attention in Transformers.To evaluate the quality of potential boundaries, we define a metric function. This function quantifies the cohesion within events and separation between events based on the graph structure represented by the similarity matrixand event boundaries. We experimented with two widely-accepted graph-clustering metrics:modularityandconductance. Modularityprovides a measure of the quality of a particular division of a network into communities, with higher values indicating higher edge density in the identified cluster when compared to the density of edges expected in a random cluster. As our edge weights represent the similarity between two tokens, we seek to maximise this metric. Modularity is defined as:(3) whereis the total edge weight in the graph,is the community (episodic event)  to which nodeis assigned, andis the Kronecker delta function.
Conductance, on the other hand, measures the fraction of total weighted edges cut by a given community boundary, and is defined as:(4) whereis a subset of all nodesin the induced graph, with. Lower conductance values indicate better community structure.
Our boundary refinement algorithm iteratively adjusts the initial surprise-based boundaries to optimise these metric functions. While our best results were achieved using modularity, we also include comparisons with conductance-based boundary refinement to provide a comprehensive analysis.
The overall process can be summarized in Algorithm1.Algorithm 1Event segmentation in KV cache1:tokens: List of tokens in the sequence2:: Threshold for surprisal to identify initial boundaries3:: Metric function to evaluate potential boundaries4:: List of final boundary positions5:[i for i in range(length(tokens) )  if]Boundary identification6:fori in range(length() ) do7:8:Boundary refinement9:endfor10:returnThis algorithm first identifies initial boundaries based on the surprise threshold, then refines these boundaries by finding the optimal positionbetween each pair of consecutive initial boundariesthat optimises the chosen metric function(either maximising modularity or minimising conductance) . This process ensures that the final segmentation (1)  captures points of high surprise and (2)  optimises for coherent information grouping. The boundary identification step incurs negligible computational cost, as it only evaluates existing LLM outputs. The time complexity of Algorithm1is dominated by the boundary refinement step, which has an overall complexity of, whereis the number of initial boundaries andis the sequence length. A detailed analysis of the algorithm’s complexity, including the computation of modularity and conductance metrics, is provided in AppendixA.2. Despite this modest computational overhead, the resulting improvement in segment quality leads to significant performance gains in downstream tasks, particularly those requiring complex temporal reasoning.3.4Memory RetrievalWhen inferring a new token, a number of episodic events are selected and become a part of the (extended)  context window of the underlying LLM. Our memory retrieval process employs a two-stage mechanism to select relevant episodic events for the LLM’s context window (Fig.2C) . First, we retrieveevents using-NN search based on dot product similarity between the current query and representative tokens of each event. These representatives, selected as per, are the most influential tokens within each event. For large memory stores, we utilise approximate-NNto maintain efficiency. Theseevents, retrieved based on their similarity to the current query, form a part of the LLM’s context window that we refer to as thesimilarity buffer.The second stage of our retrieval process introduces another buffer, which we refer to as thecontiguity buffer, designed to maintain temporal context. Implemented as a queue of size, this buffer promotes temporal relationships in retrieval. When an event is retrieved, we also enqueue its neighboring events (withinpositions in the original sequence)  into this buffer. This mechanism enables the LLM’s ‘‘induction’’ attention heads to exhibit the contiguity and asymmetry effects discussed in Section2.2. The queue structure allows for a natural decay of temporal context as new events are processed, with older or repeated events being dequeued as new ones are added. In total,events are added to the context window, striking a balance between relevance and temporal relationships in a manner analogous to human episodic memory retrieval.
body
Human-like Episodic Memory for Infinite Context LLMs
Section 3
2407.09450
4Experiments4.1Performance of EM-LLM on long-context tasksAs previously mentioned, InfLLM is, at the time of writing, considered to achieve state-of-the-art performance on long-context benchmarks (-Bench, LongBench) , as well as being the only method which uses group-based k-NN retrieval in LLMs on such benchmarks. We, therefore, employ this model as our baseline for comparison with our own methods on short context windows (4K+2K, as in) .TaskInfLLMMax Imp.EM-LLMSSMS+CSM+CNarrativeQA22.12+1.4921.3221.1321.8022.45MultiNews26.70-0.3026.5226.5426.6926.62Qasper29.33+0.1728.9929.3829.1128.68TREC69.00+2.1770.0070.0070.5070.50MultiFieldQA47.42+0.4247.4947.3947.4647.62TriviaQA86.67+1.1086.9387.6287.3587.47HotpotQA36.56+9.3839.9939.0139.0538.90SAMSum42.52+0.8742.3442.1342.8942.482WikiMQA22.31+6.4123.7422.7522.6523.46PassageRetrieval64.00+33.4785.4278.9284.6784.08Musique17.68+6.1717.5817.8217.9318.77LCC56.67+0.6354.9057.0354.7956.79GovReport31.03+1.9031.2431.6231.3431.43RepoBench-P52.97+1.3450.7653.6851.3452.86QMSum23.49+2.1323.8223.2023.9923.47Avg. score:41.90+4.3043.4043.2243.4443.70Table 1:EM-LLM performance on LongBench compared to our baseline InfLLM.S: surprise threshold,SM: surprise threshold + refinement with modularity,S+C: surprise threshold + contiguity buffer,SM+C: surprise threshold + refinement with modularity + contiguity buffer. Max Imp. shows the maximum relative improvement over InfLLM across all EM-LLM variants.Results on the LongBench dataset (Table1)  show that our method is able to improve on InfLLM in all but one task, with the best method achieving an overall increase in performance ofpercentage points (a relative improvement of) . Note that the table shows the best single method in terms of overall performance for each ablation. Looking at individual task performance across all experiments, we are able to beat InfLLM in all tasks (see AppendixA.1) .
Interestingly, we see an especially large jump in performance on the PassageRetrieval task across all ablations, with up to aimprovement on InfLLM.
This task requires the model to identify the original paragraph from a summary, a challenging task that tests the model’s ability to accurately recall a wide range of detailed information from a large context concurrently. The substantial improvement on this task highlights the effectiveness of our event segmentation method in enhancing long-term memory recall and retrieval accuracy in LLMs. Additionally, our method achieves a notableimprovement on the HotpotQA task, which involves complex reasoning over multiple supporting documents, further emphasising the model’s ability to provide explanations for answers.4.2Humans and LLM surprise cluster similar tokens togetherAs mentioned in Section3.2, we employ modularity and conductance as two refinement objectives in our boundary refinement algorithm, due to their qualities in assessing the intra- and inter-event similarities between individual tokens. We will now use such metrics to compare various event segmentation methods, including human event segmentation data. Additionally, we introduce one further, simple metric for this experiment: the ratio between intra- and inter-community similarity (I/IS) , calculated for each head and communityas follows:(5) found strong correlations between human-perceived events and prediction errors across 3 short podcasts (7-30 minutes on average) , when processing the corresponding transcript with an LLM. Taking advantage of such human data and results from previous works on this dataset, we compare the segmentation quality and correlation with human data for each of our methods (3)  using our similarity metrics.Figure 3:Comparison of human event segmentation with different computational segmentation methods in two human-annotated audio datasets (see AppendixA.3) .(A) Difference in metrics for the cohesion and separation of KV cache of LLaMA2 attention heads. The graphs report the difference of each method with the corresponding random segmentation.(B) Distance between human reports and different methods.
In both sets of results, fixed methods (F,FM,FC)  perform worse than their surprise-based counterparts (S,SM,SC)  with InfLLM’s method (F)  performing worse than random.As shown in Fig.3A, human-perceived events achieve significantly higher scores in similarity metrics compared to fixed or random events, suggesting that surprise is indeed an important factor for humans in their own perception of events. Furthermore, surprise-only segmentation ()  achieves very similar results to humans, while the addition of our refinement algorithm (,,,)  significantly improves performance. Fig.3B further shows that surprise-based methods (,,) , consistently identify event boundaries that are closest to those perceived by humans.4.3Comparing segmentation methodsLLMMetricFFMFCSSMSCMistral-7BMod-2.34.129.244.06.725.918.629.639.955.529.542.7Con9.18.7-16.96.7-12.59.6-23.69.4-24.69.3-27.69.8I/IS-4.34.031.221.43.714.917.917.035.327.721.622.4LLaMA2-7BMod-1.14.313.419.50.67.38.716.018.726.411.519.4Con11.99.8-18.87.4-13.710.9-29.510.2-29.710.1-33.310.3I/IS-3.83.720.7184.7-1.16.815.0880.025.019.916.515.4LLaMA3-8BMod-1.63.618.925.60.911.813.121.527.035.618.328.5Con11.39.5-20.36.9-14.611.4-29.79.2-30.69.2-33.99.6I/IS-3.83.124.513.9-1.15.815.711.028.116.116.412.2Table 2:Comparison with graph-theoretic metrics in the KV cache of different LLMs and segmentation methods using the PG-19 dataset. Reported values are the difference with the corresponding random segmentation. Mod: modularity, Con: Conductance, I/IS: intra/inter-similarity. Hyper-parameters: Surprise threshold =Looking at Table2, it is clear that surprise-based segmentation with refinement (,)  provides the best results in terms of event similarity metrics, regardless of the base LLM used. While the surprise-only method ()  achieves some good results, we observe that refinement is especially adept to improving this performance with regards to our metrics, as it is directly optimising for such an objective. Interestingly however, the fixed-based refinement methods (,)  do not reach the same performance as their surprise-based counterparts, further showing that the initial segmentation with a surprise threshold is crucial to achieving the best possible balance in intra-/inter-similarity with our methods.4.4Similarity, Contiguity, Recency and Temporal OrderAs demonstrated in Tables1and2, along with Fig.3, each of our ablations show various positive improvements on InfLLM (also see AppendixA.1) . As mentioned in Section4.3, refinement has a strong positive impact in improving our similarity metrics. This is seen to translate well to model performance in Table1, achieving the best performance in a third of the tasks, as well as agreeing with human data (Fig.3) . The effects of contiguity are also clearly demonstrated in this table, with the addition of our contiguity buffer achieving the best performance on three tasks, and the second-best overall score. Furthermore, these methods are shown to be generally complementary, achieving the best overall performance when combined.However, the fact that certain tasks still appear to benefit more from either surprise-only, refinement, or contiguity, is an interesting result. This is likely due to the nature of the tasks and the varying importance of contiguity across these tasks. For instance, in Supplementary Fig.5, the MultiNews task scores higher than our baseline only for a ratio ofcontiguity to similarity buffers. Where contiguity is not crucial, adding such a buffer to our context window also reduces the size of the similarity buffer, and therefore provides potentially less directly relevant events. This is compatible with our own findings that a contiguity buffer that is as big or smaller than the similarity buffer yields the best results, suggesting that the similarity buffer, is still the most crucial part of our approach. This is especially the case when combined with refinement, which we expect is due to the improved similarity of refined events, hence further reducing the need for contiguous events.
body
Human-like Episodic Memory for Infinite Context LLMs
Section 4
2407.09450
5DiscussionHuman studiesThe surprise-based segmentation and boundary refinement processes in EM-LLM mirror key aspects of human event perception and memory formation. Our approach aligns with theories proposing that humans segment continuous experiences into discrete events based on prediction errors or moments of surprise. This segmentation process is crucial for organising and later retrieving episodic memories efficiently. Indeed, significant correlations have been found between human event segmentation and prediction errors in both LLMsand video models. Our results add to this growing body of evidence, demonstrating that LLM-based surprise can serve as a proxy for human event segmentation, in multiple levels of hierarchical abstraction, and that the resulting event structure in EM-LLM’s attention heads correlates strongly with human-perceived events. This finding creates a more direct, low-level connection between LLM mechanisms and human cognitive processes. Furthermore, our model’s use of both similarity-based and temporally contiguous retrieval mechanisms parallels human memory retrieval patterns, allowing for the expression of robust phenomena found in human memory research.Furthermore, our model’s use of both similarity-based and temporally contiguous retrieval mechanisms parallels human memory retrieval patterns. The temporal contiguity effect, where items experienced close together in time are often recalled together, is a robust phenomenon in human memory research.
Further experiments could deepen our understanding of the connections between EM-LLM and human episodic memory. Following, one potential direction is to test whether the timing of the event boundaries or the degree of modularity per level that our method produces is closer on average to the human consensus, than individual human subjects. Second, we can explore the level at which different ratios of the contiguity buffer allow the human biases presented in Fig.2A and the analysis into be more easily reproduced. Finally, we could investigate how skewing event recall based on recency and originally-recorded surprise affects model performance and to what extent it produces biased behaviour found in studies of free recall.In addition, the architecture of EM-LLM, with its differentiated context handling described in Section3.1, invites further interesting comparisons to cognitive models of human memory beyond episodic. The group of tokens forming the local context, which hold the most recent and task-relevant information, share characteristics with the concept of working memory. For instance,’s influential model of working memory, which posits a limited-capacity system for temporary information storage and manipulation, bears similarities to our local context functionality. Yet, the analogy is not perfect. Our broader context window, including both local context and retrieved memories, might be more accurately compared to’s concept of long-term working memory, which proposes a mechanism for rapid access to relevant information in long-term memory, extending beyond the traditional capacity limits of working memory. Alternatively, our architecture aligns well with’s embedded-processes model, where our local context could be likened to the limited-capacity ‘‘focus of attention’’ within working memory, while the full context window parallels the activated portion of long-term memory. Future work could explore these analogies more deeply, providing a flexible test-bed for rapidly exploring hypotheses about human memory, and potentially informing debates about capacity limits in working memory.
Additionally, inspired by the multi-component nature of Baddeley’s model, one might explore the integration of modality-specific buffers within EM-LLM to enhance its performance on multi-modal tasks.Machine learningIn refining event boundaries, we utilized modularity and conductance as metrics for evaluating community structure in the similarity graph of attention keys. While effective in our experiments, we acknowledge that numerous other methods for graph clustering and sequence segmentation could potentially be applied. Our choice was motivated by their established theoretical foundations and computational efficiency, though comparative studies suggest performance can vary based on network characteristics. Interestingly, our surprise-based initial boundary detection shares similarities with Bayesian online change-point detection, suggesting potential avenues for integrating time series analysis techniques into LLM context processing. Future work could explore whether more sophisticated segmentation or clustering algorithms could improve EM-LLM’s performance, particularly for extremely long contexts or streaming data scenarios. Such investigations could enhance our model and contribute to understanding how information is structured and processed in LLMs, bridging the gap between traditional sequence analysis and LLM context processing.Looking ahead, several more avenues for future research emerge from this work. One promising direction is to extend our surprise-based segmentation and boundary refinement processes to operate at each layer of the Transformer independently. This could lead to more nuanced and hierarchical representations of episodic memories, following the underlying semantic structure of the input more closely. Additionally, exploring how EM-LLM could be utilised to enable imagination and future thinking has great potential for advancing model-based reinforcement learning and continual learning techniques in LLMs. By leveraging its event-based structure to simulate potential future scenarios or recall past experiences in novel contexts, EM-LLM could enhance an LLM’s ability to plan, adapt, and learn continuously from new information.
body
Human-like Episodic Memory for Infinite Context LLMs
Section 5
2407.09450
6ConclusionIn this work, we introduced EM-LLM, a novel and flexible architecture that integrates key aspects of human episodic memory and event cognition into transformer-based language models. Our approach enables LLMs to effectively process and utilise information from vastly extended contexts, far beyond their original training lengths. By combining surprise-based event segmentation with graph-theoretic boundary refinement, and a two-stage memory retrieval process, EM-LLM demonstrates superior performance on long-context tasks compared to state-of-the-art models. Crucially, our method requires no pre-training and can be readily applied to existing LLMs, offering a promising path towards virtually infinite context windows. This capability has the potential to revolutionise how we interact with LLMs, enabling continuous, personalized interactions over extended periods. Furthermore, the flexibility of our framework suggests it could serve as a viable alternative to traditional retrieval-augmented generation (RAG)  techniques, especially when combined with efficient compression methods to reduce the memory requirements for the model’s KV cache.In conclusion, EM-LLM represents a significant step forward in the development of language models with extended context-processing capabilities. By bridging insights from cognitive science with machine learning, our approach not only enhances the performance of LLMs on long-context tasks but also provides a scalable computational framework for testing hypotheses about human memory. We hope this study will inspire the community to expand research on the intersection between LLMs and human memory mechanisms.
conclusion
Human-like Episodic Memory for Infinite Context LLMs
Section 6