<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2405.14366] MiniCache: KV Cache Compression in Depth Dimension for Large Language Models</title><meta property="og:description" content="A critical approach for efficiently deploying computationally demanding large language models (LLMs) is Key-Value (KV) caching. The KV cache stores key-value states of previously generated tokens, significantly reducin…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="MiniCache: KV Cache Compression in Depth Dimension for Large Language Models">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="MiniCache: KV Cache Compression in Depth Dimension for Large Language Models">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2405.14366">

<!--Generated on Wed Jun  5 15:10:19 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">

<span id="id1.1" class="ltx_inline-block ltx_minipage ltx_align_middle" style="width:30.4pt;"><img src="/html/2405.14366/assets/figures/minicache_logo_white_bg.png" id="id1.1.g1" class="ltx_graphics ltx_img_square" width="598" height="532" alt="[Uncaptioned image]">
</span>
MiniCache:
KV Cache Compression in Depth Dimension for Large Language Models
</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Akide Liu<sup id="id6.2.id1" class="ltx_sup">1</sup>
 Jing Liu<sup id="id7.3.id2" class="ltx_sup">1</sup>
 Zizheng Pan<sup id="id8.4.id3" class="ltx_sup">1</sup>
 Yefei He<sup id="id9.5.id4" class="ltx_sup">2</sup> 
<br class="ltx_break">
 <span id="id10.6.id5" class="ltx_text ltx_font_bold">Gholamreza Haffari<sup id="id10.6.id5.1" class="ltx_sup"><span id="id10.6.id5.1.1" class="ltx_text">1</span></sup></span>
 <span id="id11.7.id6" class="ltx_text ltx_font_bold">Bohan Zhuang<sup id="id11.7.id6.1" class="ltx_sup"><span id="id11.7.id6.1.1" class="ltx_text">1, 2</span></sup></span>

<br class="ltx_break">
<sup id="id12.8.id7" class="ltx_sup">1</sup>ZIP Lab, Monash University, Australia

<br class="ltx_break">
<sup id="id13.9.id8" class="ltx_sup">2</sup>ZIP Lab, Zhejiang University, China
</span><span class="ltx_author_notes">Corresponding author. Email: <math id="id2.1.m1.3" class="ltx_Math" alttext="\tt bohan.zhuang@gmail.com" display="inline"><semantics id="id2.1.m1.3a"><mrow id="id2.1.m1.3.3.1" xref="id2.1.m1.3.3.2.cmml"><mi id="id2.1.m1.1.1" xref="id2.1.m1.1.1.cmml">𝚋𝚘𝚑𝚊𝚗</mi><mo lspace="0em" rspace="0.167em" id="id2.1.m1.3.3.1.2" xref="id2.1.m1.3.3.2a.cmml">.</mo><mrow id="id2.1.m1.3.3.1.1" xref="id2.1.m1.3.3.1.1.cmml"><mi id="id2.1.m1.3.3.1.1.2" xref="id2.1.m1.3.3.1.1.2.cmml">𝚣𝚑𝚞𝚊𝚗𝚐</mi><mo lspace="0em" rspace="0em" id="id2.1.m1.3.3.1.1.1" xref="id2.1.m1.3.3.1.1.1.cmml">​</mo><mi mathvariant="normal" id="id2.1.m1.3.3.1.1.3" xref="id2.1.m1.3.3.1.1.3.cmml">@</mi><mo lspace="0em" rspace="0em" id="id2.1.m1.3.3.1.1.1a" xref="id2.1.m1.3.3.1.1.1.cmml">​</mo><mi id="id2.1.m1.3.3.1.1.4" xref="id2.1.m1.3.3.1.1.4.cmml">𝚐𝚖𝚊𝚒𝚕</mi></mrow><mo lspace="0em" rspace="0.167em" id="id2.1.m1.3.3.1.3" xref="id2.1.m1.3.3.2a.cmml">.</mo><mi id="id2.1.m1.2.2" xref="id2.1.m1.2.2.cmml">𝚌𝚘𝚖</mi></mrow><annotation-xml encoding="MathML-Content" id="id2.1.m1.3b"><apply id="id2.1.m1.3.3.2.cmml" xref="id2.1.m1.3.3.1"><csymbol cd="ambiguous" id="id2.1.m1.3.3.2a.cmml" xref="id2.1.m1.3.3.1.2">formulae-sequence</csymbol><ci id="id2.1.m1.1.1.cmml" xref="id2.1.m1.1.1">𝚋𝚘𝚑𝚊𝚗</ci><apply id="id2.1.m1.3.3.1.1.cmml" xref="id2.1.m1.3.3.1.1"><times id="id2.1.m1.3.3.1.1.1.cmml" xref="id2.1.m1.3.3.1.1.1"></times><ci id="id2.1.m1.3.3.1.1.2.cmml" xref="id2.1.m1.3.3.1.1.2">𝚣𝚑𝚞𝚊𝚗𝚐</ci><ci id="id2.1.m1.3.3.1.1.3.cmml" xref="id2.1.m1.3.3.1.1.3">@</ci><ci id="id2.1.m1.3.3.1.1.4.cmml" xref="id2.1.m1.3.3.1.1.4">𝚐𝚖𝚊𝚒𝚕</ci></apply><ci id="id2.1.m1.2.2.cmml" xref="id2.1.m1.2.2">𝚌𝚘𝚖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="id2.1.m1.3c">\tt bohan.zhuang@gmail.com</annotation></semantics></math></span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id5.3" class="ltx_p">A critical approach for efficiently deploying computationally demanding large language models (LLMs) is Key-Value (KV) caching. The KV cache stores key-value states of previously generated tokens, significantly reducing the need for repetitive computations and thereby lowering latency in autoregressive generation.
However, the size of the KV cache grows linearly with sequence length, posing challenges for applications requiring long context input and extensive sequence generation.
In this paper, we present a simple yet effective approach, called MiniCache, to compress the KV cache across layers from a novel depth perspective,
significantly reducing the memory footprint for LLM inference.
Our approach is based on the observation that KV cache states exhibit high similarity
between the adjacent layers
in the middle-to-deep portion of LLMs.
To facilitate merging, we propose disentangling the states into the magnitude and direction components, interpolating the directions of the state vectors while preserving their lengths unchanged. Furthermore, we introduce a token retention strategy to keep highly distinct state pairs unmerged, thus preserving the information with minimal additional storage overhead.
Our MiniCache is training-free and general, complementing existing KV cache compression strategies, such as quantization and sparsity.
We conduct a comprehensive evaluation of MiniCache utilizing various models including LLaMA-2, LLaMA-3, Phi-3, Mistral, and Mixtral across multiple benchmarks, demonstrating its exceptional performance in achieving superior compression ratios and high throughput.
On the ShareGPT dataset, LLaMA-2-7B with 4-bit MiniCache achieves a remarkable compression ratio of up to <math id="id3.1.m1.1" class="ltx_math_unparsed" alttext="5.02\times" display="inline"><semantics id="id3.1.m1.1a"><mrow id="id3.1.m1.1b"><mn id="id3.1.m1.1.1">5.02</mn><mo lspace="0.222em" id="id3.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="id3.1.m1.1c">5.02\times</annotation></semantics></math>, enhances inference throughput by approximately <math id="id4.2.m2.1" class="ltx_math_unparsed" alttext="5\times" display="inline"><semantics id="id4.2.m2.1a"><mrow id="id4.2.m2.1b"><mn id="id4.2.m2.1.1">5</mn><mo lspace="0.222em" id="id4.2.m2.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="id4.2.m2.1c">5\times</annotation></semantics></math>, and reduces the memory footprint by <math id="id5.3.m3.1" class="ltx_Math" alttext="41\%" display="inline"><semantics id="id5.3.m3.1a"><mrow id="id5.3.m3.1.1" xref="id5.3.m3.1.1.cmml"><mn id="id5.3.m3.1.1.2" xref="id5.3.m3.1.1.2.cmml">41</mn><mo id="id5.3.m3.1.1.1" xref="id5.3.m3.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="id5.3.m3.1b"><apply id="id5.3.m3.1.1.cmml" xref="id5.3.m3.1.1"><csymbol cd="latexml" id="id5.3.m3.1.1.1.cmml" xref="id5.3.m3.1.1.1">percent</csymbol><cn type="integer" id="id5.3.m3.1.1.2.cmml" xref="id5.3.m3.1.1.2">41</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="id5.3.m3.1c">41\%</annotation></semantics></math> compared to the FP16 full cache baseline, all while maintaining near-lossless performance.</p>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Large Language Models (LLMs), exemplified by the GPT series <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib1" title="" class="ltx_ref">1</a>, <a href="#bib.bib2" title="" class="ltx_ref">2</a>, <a href="#bib.bib3" title="" class="ltx_ref">3</a>]</cite> and the LLaMA series <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib4" title="" class="ltx_ref">4</a>, <a href="#bib.bib5" title="" class="ltx_ref">5</a>, <a href="#bib.bib6" title="" class="ltx_ref">6</a>]</cite>, have emerged as pivotal innovations within the artificial general intelligence, notably enhancing the capabilities of natural language processing. However, these models are meticulously trained using extensive computational resources <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib7" title="" class="ltx_ref">7</a>]</cite> and massive datasets <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib8" title="" class="ltx_ref">8</a>]</cite>, which enables them to produce text that effectively mimics human writing styles and conducts complex reasoning analysis, yet raises the challenge of efficient deployment and serving.
Within the inference framework of LLMs, KV caches <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib9" title="" class="ltx_ref">9</a>]</cite> are crucial for storing pre-computed keys and values, thus avoiding repeated calculation over the preceding context to substantially enhance the deployment efficiency of LLMs.
However, the increasing demand for longer sequence lengths results in cached states becoming voluminous, leading to significant memory consumption during generation.
For instance, a 175B GPT-3 model <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib2" title="" class="ltx_ref">2</a>]</cite>, with a batch size of 64 and a sequence length of 4,096 tokens (both prefilled and generated), requires approximately 1,208GB of GPU memory.
This requirement is <math id="S1.p1.1.m1.1" class="ltx_math_unparsed" alttext="3.45\times" display="inline"><semantics id="S1.p1.1.m1.1a"><mrow id="S1.p1.1.m1.1b"><mn id="S1.p1.1.m1.1.1">3.45</mn><mo lspace="0.222em" id="S1.p1.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S1.p1.1.m1.1c">3.45\times</annotation></semantics></math> greater than the memory used to store the model’s weights.
In this context, KV cache compression is of paramount importance due to its clear benefits: 1) largely reduces the memory footprint allowing for faster generation and larger batch serving; 2) significantly lowers the cost per token, demonstrating substantial commercial benefits.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2405.14366/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="305" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S1.F1.6.3.1" class="ltx_text" style="font-size:90%;">Figure 1</span>: </span><span id="S1.F1.4.2" class="ltx_text" style="font-size:90%;">Overview of our MiniCache strategy and example results:
(a) shows the observation that the KV cache states between two adjacent layers are highly similar, particularly across the middle to deep layers. The x-axis uses index/2 to represent the similarities for each pair of layers. (b) compares the performance of MiniCache, and the mean baseline, which simply averages the KV caches of two layers, using the LLaMA-3-70B model <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib6" title="" class="ltx_ref">6</a>]</cite> on the GSM8K dataset <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib10" title="" class="ltx_ref">10</a>]</cite>. MiniCache, which begins merging from the half-layer depth, achieves near-lossless performance.
(c) highlights the primary difference between MiniCache and previous approaches. MiniCache investigates the inter-layer redundancy of KV caches along the depth dimension of LLMs, an aspect previously overlooked by intra-layer-based methods. Here,
<math id="S1.F1.3.1.m1.1" class="ltx_Math" alttext="T" display="inline"><semantics id="S1.F1.3.1.m1.1b"><mi id="S1.F1.3.1.m1.1.1" xref="S1.F1.3.1.m1.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="S1.F1.3.1.m1.1c"><ci id="S1.F1.3.1.m1.1.1.cmml" xref="S1.F1.3.1.m1.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.3.1.m1.1d">T</annotation></semantics></math> refers to the last timestamp of pre-filling, and <math id="S1.F1.4.2.m2.1" class="ltx_Math" alttext="T+1" display="inline"><semantics id="S1.F1.4.2.m2.1b"><mrow id="S1.F1.4.2.m2.1.1" xref="S1.F1.4.2.m2.1.1.cmml"><mi id="S1.F1.4.2.m2.1.1.2" xref="S1.F1.4.2.m2.1.1.2.cmml">T</mi><mo id="S1.F1.4.2.m2.1.1.1" xref="S1.F1.4.2.m2.1.1.1.cmml">+</mo><mn id="S1.F1.4.2.m2.1.1.3" xref="S1.F1.4.2.m2.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S1.F1.4.2.m2.1c"><apply id="S1.F1.4.2.m2.1.1.cmml" xref="S1.F1.4.2.m2.1.1"><plus id="S1.F1.4.2.m2.1.1.1.cmml" xref="S1.F1.4.2.m2.1.1.1"></plus><ci id="S1.F1.4.2.m2.1.1.2.cmml" xref="S1.F1.4.2.m2.1.1.2">𝑇</ci><cn type="integer" id="S1.F1.4.2.m2.1.1.3.cmml" xref="S1.F1.4.2.m2.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.4.2.m2.1d">T+1</annotation></semantics></math> denotes to the first timestamp of decoding.
</span></figcaption>
</figure>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">Existing KV cache compression efforts can be roughly categorized into two types, namely quantization and sparsity. The quantization approaches <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib11" title="" class="ltx_ref">11</a>, <a href="#bib.bib12" title="" class="ltx_ref">12</a>]</cite> propose storing the KV states in low-bit numerical values. Typically, FlexGen <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib13" title="" class="ltx_ref">13</a>]</cite> demonstrates that 4-bit KV cache quantization can achieve lossless performance.
In contrast, sparsity-driven methods aim to retain only the salient tokens while evicting the rest, either heuristically <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib14" title="" class="ltx_ref">14</a>, <a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite> or adaptively <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib16" title="" class="ltx_ref">16</a>]</cite>.
Some approaches <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib11" title="" class="ltx_ref">11</a>]</cite> explore the intersection of these two types, by assigning high-bit to salient tokens and extremely low-bit to the rest of the tokens, achieving more aggressive memory gain.
Despite these innovations,
<em id="S1.p2.1.1" class="ltx_emph ltx_font_italic">existing literature merely consider the <span id="S1.p2.1.1.1" class="ltx_text ltx_font_bold">intra-layer redundancy</span>, while neglecting another important complementary direction – the <span id="S1.p2.1.1.2" class="ltx_text ltx_font_bold">inter-layer redundancy</span></em>, this motivation difference is illustrated in the <a href="#S1.F1" title="In 1 Introduction ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">1</span></a>(c).</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">Our analysis begins by exploring the redundancy of KV caches <span id="S1.p3.1.1" class="ltx_text ltx_font_bold">along the depth dimension</span>, as shown in Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>(a).
We observe that KV cache states exhibit high similarity between neighbouring layers in the middle-to-deep portions of LLMs. This intriguing property suggests that states paired by position between adjacent layers can be accurately merged into a single state space with a strong performance guarantee, as illustrated in Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>(b). This approach significantly reduces the memory footprint without the need to retain individual states for each attention layer.
Note that these observations are pertinent to
dynamic inference strategies such as mixture-of-depths <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib17" title="" class="ltx_ref">17</a>]</cite> and layer-wise early exiting <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib18" title="" class="ltx_ref">18</a>, <a href="#bib.bib19" title="" class="ltx_ref">19</a>]</cite>, which optimize computational paths by skipping non-critical layers
to enhance training and inference efficiency. Furthermore, layer pruning methods <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib20" title="" class="ltx_ref">20</a>]</cite> highlight considerable redundancy in deeper layers. However, despite these advancements,
the redundancy of KV caches along the depth dimension
has largely been overlooked.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">In this paper, we propose <span id="S1.p4.1.1" class="ltx_text ltx_font_bold">MiniCache</span>, a simple yet effective cross-layer KV cache compression method aimed at advancing the inference efficiency of LLMs.
MiniCache consists of two essential components.
Firstly, we introduce an accurate cache merging strategy, employing a reparameterization of state vectors that decompose them into the magnitude and direction components, akin to weight normalization <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib21" title="" class="ltx_ref">21</a>]</cite>. This approach allows for effective interpolation of the directional component in polar coordinates while preserving the original state norms to retain as much information as possible. This interpolation refers to the cross-layer merging as shown in the <a href="#S1.F1" title="In 1 Introduction ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">1</span></a>(c).
Secondly, we recognize that a small
subset of state pairs, characterized by low similarities but carrying largely distinct semantic meanings, are unsuitable for inter-layer merging.
To address this, we propose a token retention strategy to minimize performance degradation, which involves separately retaining these outlier pairs.
Our framework is notably memory-efficient, requiring storage for only a single high-dimensional directional component, along with minimal extra memory overhead. The overhead consists of a few unmergeable tokens and their corresponding indexes, as well as token-wise magnitudes to accurately restore the original states.</p>
</div>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.1" class="ltx_p">We conduct extensive experiments with representative LLMs,
including
Mixtral-8x7B <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib22" title="" class="ltx_ref">22</a>]</cite>, Phi-3-Mini <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib23" title="" class="ltx_ref">23</a>]</cite>, and LLaMA-3 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib6" title="" class="ltx_ref">6</a>]</cite> 8B and 70B, respectively. Our method is benchmarked across a diverse range of question answering and generation datasets <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib24" title="" class="ltx_ref">24</a>, <a href="#bib.bib25" title="" class="ltx_ref">25</a>, <a href="#bib.bib26" title="" class="ltx_ref">26</a>, <a href="#bib.bib27" title="" class="ltx_ref">27</a>, <a href="#bib.bib28" title="" class="ltx_ref">28</a>, <a href="#bib.bib29" title="" class="ltx_ref">29</a>, <a href="#bib.bib30" title="" class="ltx_ref">30</a>, <a href="#bib.bib31" title="" class="ltx_ref">31</a>]</cite> using the lm-eval-harness <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib32" title="" class="ltx_ref">32</a>]</cite>. Additionally, we evaluate our results on LongBench <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib33" title="" class="ltx_ref">33</a>]</cite> for long-sequence generation. The results demonstrate that MiniCache can reduce the memory footprint required for LLM inference by up to 41%, while simultaneously enhancing throughput by approximately <math id="S1.p5.1.m1.1" class="ltx_math_unparsed" alttext="5\times" display="inline"><semantics id="S1.p5.1.m1.1a"><mrow id="S1.p5.1.m1.1b"><mn id="S1.p5.1.m1.1.1">5</mn><mo lspace="0.222em" id="S1.p5.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S1.p5.1.m1.1c">5\times</annotation></semantics></math> compared to fully cached baseline, clearly surpassing existing methods <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib14" title="" class="ltx_ref">14</a>, <a href="#bib.bib12" title="" class="ltx_ref">12</a>, <a href="#bib.bib11" title="" class="ltx_ref">11</a>, <a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite>.</p>
</div>
<div id="S1.p6" class="ltx_para">
<p id="S1.p6.1" class="ltx_p">Our contributions are summarized as follows:</p>
<ul id="S1.I1" class="ltx_itemize">
<li id="S1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i1.p1" class="ltx_para">
<p id="S1.I1.i1.p1.1" class="ltx_p">We introduce MiniCache, a simple yet highly effective framework for KV cache compression. MiniCache pioneers the exploration of KV cache compression along the depth dimension, thereby significantly expanding its capabilities.</p>
</div>
</li>
<li id="S1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i2.p1" class="ltx_para">
<p id="S1.I1.i2.p1.1" class="ltx_p">We observe a fascinating characteristic of cross-layer KV cache states: they exhibit high similarity between adjacent layers in the middle to later stages of LLMs.
Additionally, we find that not all state pairs are suitable for merging.</p>
</div>
</li>
<li id="S1.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i3.p1" class="ltx_para">
<p id="S1.I1.i3.p1.1" class="ltx_p">We propose an accurate and memory-efficient method for cross-layer cache merging, comprising a reparameterization strategy and a token retention mechanism.
Our method complements existing KV cache compression approaches, further enhancing LLM serving efficiency.</p>
</div>
</li>
<li id="S1.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i4.p1" class="ltx_para">
<p id="S1.I1.i4.p1.2" class="ltx_p">Our MiniCache performs favourably against the state-of-the-art methods. Notably, our 4-bit MiniCache achieves a strong compression ratio of up to <math id="S1.I1.i4.p1.1.m1.1" class="ltx_math_unparsed" alttext="5.02\times" display="inline"><semantics id="S1.I1.i4.p1.1.m1.1a"><mrow id="S1.I1.i4.p1.1.m1.1b"><mn id="S1.I1.i4.p1.1.m1.1.1">5.02</mn><mo lspace="0.222em" id="S1.I1.i4.p1.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S1.I1.i4.p1.1.m1.1c">5.02\times</annotation></semantics></math>, <math id="S1.I1.i4.p1.2.m2.1" class="ltx_math_unparsed" alttext="5\times" display="inline"><semantics id="S1.I1.i4.p1.2.m2.1a"><mrow id="S1.I1.i4.p1.2.m2.1b"><mn id="S1.I1.i4.p1.2.m2.1.1">5</mn><mo lspace="0.222em" id="S1.I1.i4.p1.2.m2.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S1.I1.i4.p1.2.m2.1c">5\times</annotation></semantics></math> higher inference throughput, and 41% memory reduction compared to the FP16 full cache baseline with near-lossless performance.</p>
</div>
</li>
</ul>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related Work</h2>

<div id="S2.p1" class="ltx_para">
<p id="S2.p1.1" class="ltx_p"><span id="S2.p1.1.1" class="ltx_text ltx_font_bold">Efficient inference for LLMs.</span>
Large Language Models (LLMs) are constrained by considerable computational and memory requirements during inference, particularly in resource-constrained environments. To mitigate these challenges, a variety of efficient inference techniques have been developed. For instance, dynamic inference methods <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib34" title="" class="ltx_ref">34</a>, <a href="#bib.bib18" title="" class="ltx_ref">18</a>, <a href="#bib.bib35" title="" class="ltx_ref">35</a>, <a href="#bib.bib36" title="" class="ltx_ref">36</a>]</cite>, represented by mixture-of-experts (MoE) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib37" title="" class="ltx_ref">37</a>, <a href="#bib.bib38" title="" class="ltx_ref">38</a>, <a href="#bib.bib39" title="" class="ltx_ref">39</a>, <a href="#bib.bib40" title="" class="ltx_ref">40</a>, <a href="#bib.bib41" title="" class="ltx_ref">41</a>]</cite>, adaptively select specific sub-structures of the model during the inference process based on the input data, significantly improving inference efficiency while keeping model capacity. Techniques like Multi-Query Attention <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib42" title="" class="ltx_ref">42</a>, <a href="#bib.bib43" title="" class="ltx_ref">43</a>]</cite>, Kernel-driven attentions <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib44" title="" class="ltx_ref">44</a>, <a href="#bib.bib45" title="" class="ltx_ref">45</a>, <a href="#bib.bib46" title="" class="ltx_ref">46</a>, <a href="#bib.bib47" title="" class="ltx_ref">47</a>]</cite>, and low-rank attentions <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib48" title="" class="ltx_ref">48</a>, <a href="#bib.bib49" title="" class="ltx_ref">49</a>, <a href="#bib.bib50" title="" class="ltx_ref">50</a>, <a href="#bib.bib41" title="" class="ltx_ref">41</a>]</cite> approximate the functionality of traditional attention mechanisms but with more efficient implementations. Quantization strategies <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib51" title="" class="ltx_ref">51</a>, <a href="#bib.bib52" title="" class="ltx_ref">52</a>, <a href="#bib.bib53" title="" class="ltx_ref">53</a>, <a href="#bib.bib54" title="" class="ltx_ref">54</a>]</cite> involve converting the model’s weights and activations into a low bit-width format, thereby reducing memory footprint and computational intensity. Sparsification approaches <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib55" title="" class="ltx_ref">55</a>, <a href="#bib.bib56" title="" class="ltx_ref">56</a>, <a href="#bib.bib14" title="" class="ltx_ref">14</a>, <a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite> eliminate unnecessary elements in both model weights and token representations, further enhancing efficiency. Some closely related works, such as MoD <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib17" title="" class="ltx_ref">17</a>]</cite> and LayerSkips <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib19" title="" class="ltx_ref">19</a>]</cite>, considered the dynamic inference nature to ignore unimportant layers according to input; however, these methods require an additional fine-tuning process or carefully designed pre-training stages, which reduces the adaptivity of these methods. MiniCache relies on inter-layer similarity observations to perform cross-layer merging, significantly reducing memory demand.</p>
</div>
<div id="S2.p2" class="ltx_para">
<p id="S2.p2.1" class="ltx_p"><span id="S2.p2.1.1" class="ltx_text ltx_font_bold">Model merging.</span>
Merging compression involves the aggregation of a model’s parameters and activations at various granularities.
This process enhances the efficiency of inference in large models and facilitates huge redundancy <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib57" title="" class="ltx_ref">57</a>]</cite>. Linear Mode Connectivity (LMC) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib58" title="" class="ltx_ref">58</a>]</cite> enables the fine-tuning of models from a shared pre-trained base.
Commonly, weight averaging <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib59" title="" class="ltx_ref">59</a>]</cite> is employed as an efficient technique to perform merge compression; notably, Model Soup <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib60" title="" class="ltx_ref">60</a>]</cite> utilizes linear averaging in this context. Advanced methods like TIES Merging <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib61" title="" class="ltx_ref">61</a>]</cite>, Model Breadcrumbs <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib62" title="" class="ltx_ref">62</a>]</cite>, and DARE <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib63" title="" class="ltx_ref">63</a>]</cite> further enhance this process by sparsifying and combining
model parameters, enabling the merging of models without sacrificing performance capabilities. Additionally, Spherical Linear intERPolation (SLERP) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib64" title="" class="ltx_ref">64</a>]</cite> extends beyond simple weight averaging by interpolating between model parameters. The Fisher information matrix <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib65" title="" class="ltx_ref">65</a>]</cite> and RegMean-based methods <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib66" title="" class="ltx_ref">66</a>]</cite> further optimize merges to produce ideal weights with respect to the <math id="S2.p2.1.m1.1" class="ltx_Math" alttext="\ell_{2}" display="inline"><semantics id="S2.p2.1.m1.1a"><msub id="S2.p2.1.m1.1.1" xref="S2.p2.1.m1.1.1.cmml"><mi mathvariant="normal" id="S2.p2.1.m1.1.1.2" xref="S2.p2.1.m1.1.1.2.cmml">ℓ</mi><mn id="S2.p2.1.m1.1.1.3" xref="S2.p2.1.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S2.p2.1.m1.1b"><apply id="S2.p2.1.m1.1.1.cmml" xref="S2.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S2.p2.1.m1.1.1.1.cmml" xref="S2.p2.1.m1.1.1">subscript</csymbol><ci id="S2.p2.1.m1.1.1.2.cmml" xref="S2.p2.1.m1.1.1.2">ℓ</ci><cn type="integer" id="S2.p2.1.m1.1.1.3.cmml" xref="S2.p2.1.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p2.1.m1.1c">\ell_{2}</annotation></semantics></math> distance to generation outputs, while maintaining the privacy of training data. However, most existing works focus on merging model parameters, with the concept of depth-wise mergeability not being thoroughly explored in prior research. MiniCache focuses on the KV cache token merging in the depth dimensional of LLMs.</p>
</div>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Motivation</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.1" class="ltx_p">In the below, we present our new observations in a novel cross-layer perspective.</p>
</div>
<figure id="S3.F2" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="S3.F2.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2405.14366/assets/x2.png" id="S3.F2.sf1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="367" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S3.F2.sf1.3.1.1" class="ltx_text" style="font-size:90%;">(a)</span> </span><span id="S3.F2.sf1.4.2" class="ltx_text" style="font-size:90%;">Simple average baseline <span id="S3.F2.sf1.4.2.1" class="ltx_text ltx_font_italic">vs.</span> full cache on GSM8K</span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="S3.F2.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2405.14366/assets/x3.png" id="S3.F2.sf2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="354" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S3.F2.sf2.2.1.1" class="ltx_text" style="font-size:90%;">(b)</span> </span><span id="S3.F2.sf2.3.2" class="ltx_text" style="font-size:90%;">Pairwise similarity in adjacent layers KV cache</span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="S3.F2.sf3" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2405.14366/assets/x4.png" id="S3.F2.sf3.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="391" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S3.F2.sf3.2.1.1" class="ltx_text" style="font-size:90%;">(c)</span> </span><span id="S3.F2.sf3.3.2" class="ltx_text" style="font-size:90%;">Benchmark on five QA datasets</span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S3.F2.2.1.1" class="ltx_text" style="font-size:90%;">Figure 2</span>: </span><span id="S3.F2.3.2" class="ltx_text" style="font-size:90%;">Overall of our explorations and observations : (a) shows the strong baseline by performing average merging on the KV cache. (b) shows the pairwise similarity of cache states by position between adjacent layers. (c) compares between the MiniCache, simple average, and full cach baseline on five different datasets.
</span></figcaption>
</figure>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Cross-Layer Redundancy in KV Cache</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.1" class="ltx_p">Prior studies have revealed the ineffectiveness of middle-to-deep layers in LLMs <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib20" title="" class="ltx_ref">20</a>]</cite>. Thus, layer-wise early exiting in this case can effectively avoid the redundant computation with minor effect on LLM performance <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib67" title="" class="ltx_ref">67</a>, <a href="#bib.bib19" title="" class="ltx_ref">19</a>]</cite>.
Inspired by this, we explore a <span id="S3.SS1.p1.1.1" class="ltx_text ltx_font_italic">layer-wise</span> merging of KV cache in LLMs, starting with a simple baseline by averaging similar tokens across adjacent layers. We provide our key observations as follows.</p>
</div>
<div id="S3.SS1.p2" class="ltx_para">
<p id="S3.SS1.p2.1" class="ltx_p"><span id="S3.SS1.p2.1.1" class="ltx_text ltx_font_bold">Observation 1: KV cache shares a high similarity between adjacent layers.
</span> Based on LLaMA-3-70B <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib6" title="" class="ltx_ref">6</a>]</cite>, we conduct zero-shot inference on the validation sets of three widely recognized benchmarks: COQA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib68" title="" class="ltx_ref">68</a>]</cite>, GSM8K <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib10" title="" class="ltx_ref">10</a>]</cite> and TruthfulQA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib69" title="" class="ltx_ref">69</a>]</cite>. In general, we find that KV cache in the shallow layers exhibits low similarity, whereas those in the middle-to-deep layers are very similar to one another based on angular distance, as shown in <a href="#S1.F1" title="In 1 Introduction ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">1</span></a>(b).
Next, we merge the KV cache across adjacent layers by conducting five-shot inference with LLaMA-2-7B, LLaMA-2-13B <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib5" title="" class="ltx_ref">5</a>]</cite>, LLaMA-3-8B <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib6" title="" class="ltx_ref">6</a>]</cite>, and Mixtral-8x7B <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib22" title="" class="ltx_ref">22</a>]</cite> on GSM8K <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib10" title="" class="ltx_ref">10</a>]</cite>. Specifically, starting from the middle layer of each model, we merge the KV cache in the adjacent two layers.
As shown in <a href="#S3.F2" title="In 3 Motivation ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">2</span></a>(a), we observe a favourable performance for different LLMs,
which reveals the huge potential for efficiency improvement by sharing the KV cache across adjacent layers during LLM decoding.</p>
</div>
<div id="S3.SS1.p3" class="ltx_para">
<p id="S3.SS1.p3.1" class="ltx_p"><span id="S3.SS1.p3.1.1" class="ltx_text ltx_font_bold">Observation 2:
Not all tokens are equally important to merge, a few distinct pairs require retention.
</span>
Recent works <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib16" title="" class="ltx_ref">16</a>, <a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite> in KV cache compression have found that keeping a few salient tokens at each layer, which contribute the majority of attention scores, could be sufficient to maintain LLM performance.
In our case, we speculate that certain token pairs by position in adjacent layers
also exist outlier behaviours, showing strong semantic differences that make them unsuitable for merging.
Based on COQA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib68" title="" class="ltx_ref">68</a>]</cite> and LLaMA-2-7B <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib5" title="" class="ltx_ref">5</a>]</cite>, we investigate the similarities at the level of token pairs.
As shown in <a href="#S3.F2" title="In 3 Motivation ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">2</span></a>(b),
we find a significant portion of token pairs share high similarities across adjacent layers.
However, we observe a few outlier pairs, such as index 0 and index 15, which have large margins of difference. We consider these tokens non-mergeable due to their significant differences. We also show that merging these distinct tokens results in performance degradation, corresponding to <math id="S3.SS1.p3.1.m1.1" class="ltx_Math" alttext="\gamma=0" display="inline"><semantics id="S3.SS1.p3.1.m1.1a"><mrow id="S3.SS1.p3.1.m1.1.1" xref="S3.SS1.p3.1.m1.1.1.cmml"><mi id="S3.SS1.p3.1.m1.1.1.2" xref="S3.SS1.p3.1.m1.1.1.2.cmml">γ</mi><mo id="S3.SS1.p3.1.m1.1.1.1" xref="S3.SS1.p3.1.m1.1.1.1.cmml">=</mo><mn id="S3.SS1.p3.1.m1.1.1.3" xref="S3.SS1.p3.1.m1.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.1.m1.1b"><apply id="S3.SS1.p3.1.m1.1.1.cmml" xref="S3.SS1.p3.1.m1.1.1"><eq id="S3.SS1.p3.1.m1.1.1.1.cmml" xref="S3.SS1.p3.1.m1.1.1.1"></eq><ci id="S3.SS1.p3.1.m1.1.1.2.cmml" xref="S3.SS1.p3.1.m1.1.1.2">𝛾</ci><cn type="integer" id="S3.SS1.p3.1.m1.1.1.3.cmml" xref="S3.SS1.p3.1.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.1.m1.1c">\gamma=0</annotation></semantics></math> row in <a href="#S6.T2" title="In 6 Ablation Study ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">2</span></a>.
Thus, while cross-layer merging is a promising strategy for reducing memory burdens, it must be implemented with careful consideration of token-level similarities to ensure optimal performance, as shown in <a href="#S3.F2" title="In 3 Motivation ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">2</span></a>(c).</p>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Method</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.1" class="ltx_p">In this section, we introduce our MiniCache, a simple yet effective
method aimed at trimming the KV cache redundancy in the depth dimension. This framework exploits the high similarity of KV cache states between adjacent layers
and consists of two primary components: a reparameterization-based merging strategy
and a token retention mechanism.
The merging strategy compresses the KV cache states in adjacent layers to aggregate them into a single shared memory space, beginning from the middle of the model.
The token retention mechanism mitigates information lost by retaining the highly distinct state pairs with minimal additional memory cost. With the merged cache, retention tokens, and magnitudes, we can accurately restore the original cache states for token decoding.</p>
</div>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Cross-Layer Compression</h3>

<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.8" class="ltx_p">Our method commences with the identification of an optimal starting layer <math id="S4.SS1.p1.1.m1.1" class="ltx_Math" alttext="S" display="inline"><semantics id="S4.SS1.p1.1.m1.1a"><mi id="S4.SS1.p1.1.m1.1.1" xref="S4.SS1.p1.1.m1.1.1.cmml">S</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.1.m1.1b"><ci id="S4.SS1.p1.1.m1.1.1.cmml" xref="S4.SS1.p1.1.m1.1.1">𝑆</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.1.m1.1c">S</annotation></semantics></math>.
Observations in <a href="#S3.SS1" title="3.1 Cross-Layer Redundancy in KV Cache ‣ 3 Motivation ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">3.1</span></a> indicate that the KV cache from middle-to-deep layers exhibits explicit patterns characterized by high similarity across adjacent layers. Consequently, we select the starting layer from the middle of the LLM, specifically <math id="S4.SS1.p1.2.m2.1" class="ltx_Math" alttext="S=L/2" display="inline"><semantics id="S4.SS1.p1.2.m2.1a"><mrow id="S4.SS1.p1.2.m2.1.1" xref="S4.SS1.p1.2.m2.1.1.cmml"><mi id="S4.SS1.p1.2.m2.1.1.2" xref="S4.SS1.p1.2.m2.1.1.2.cmml">S</mi><mo id="S4.SS1.p1.2.m2.1.1.1" xref="S4.SS1.p1.2.m2.1.1.1.cmml">=</mo><mrow id="S4.SS1.p1.2.m2.1.1.3" xref="S4.SS1.p1.2.m2.1.1.3.cmml"><mi id="S4.SS1.p1.2.m2.1.1.3.2" xref="S4.SS1.p1.2.m2.1.1.3.2.cmml">L</mi><mo id="S4.SS1.p1.2.m2.1.1.3.1" xref="S4.SS1.p1.2.m2.1.1.3.1.cmml">/</mo><mn id="S4.SS1.p1.2.m2.1.1.3.3" xref="S4.SS1.p1.2.m2.1.1.3.3.cmml">2</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.2.m2.1b"><apply id="S4.SS1.p1.2.m2.1.1.cmml" xref="S4.SS1.p1.2.m2.1.1"><eq id="S4.SS1.p1.2.m2.1.1.1.cmml" xref="S4.SS1.p1.2.m2.1.1.1"></eq><ci id="S4.SS1.p1.2.m2.1.1.2.cmml" xref="S4.SS1.p1.2.m2.1.1.2">𝑆</ci><apply id="S4.SS1.p1.2.m2.1.1.3.cmml" xref="S4.SS1.p1.2.m2.1.1.3"><divide id="S4.SS1.p1.2.m2.1.1.3.1.cmml" xref="S4.SS1.p1.2.m2.1.1.3.1"></divide><ci id="S4.SS1.p1.2.m2.1.1.3.2.cmml" xref="S4.SS1.p1.2.m2.1.1.3.2">𝐿</ci><cn type="integer" id="S4.SS1.p1.2.m2.1.1.3.3.cmml" xref="S4.SS1.p1.2.m2.1.1.3.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.2.m2.1c">S=L/2</annotation></semantics></math>.
From this layer onward, the KV pairs are presumed to be sufficiently similar across adjacent layers to warrant their consolidation. Central to this approach is a merge function, <math id="S4.SS1.p1.3.m3.1" class="ltx_Math" alttext="F" display="inline"><semantics id="S4.SS1.p1.3.m3.1a"><mi id="S4.SS1.p1.3.m3.1.1" xref="S4.SS1.p1.3.m3.1.1.cmml">F</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.3.m3.1b"><ci id="S4.SS1.p1.3.m3.1.1.cmml" xref="S4.SS1.p1.3.m3.1.1">𝐹</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.3.m3.1c">F</annotation></semantics></math>, which is designed to integrate the KV caches
of consecutive layers into a single, unified cache.
We define <math id="S4.SS1.p1.4.m4.1" class="ltx_Math" alttext="{\bm{x}}" display="inline"><semantics id="S4.SS1.p1.4.m4.1a"><mi id="S4.SS1.p1.4.m4.1.1" xref="S4.SS1.p1.4.m4.1.1.cmml">𝒙</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.4.m4.1b"><ci id="S4.SS1.p1.4.m4.1.1.cmml" xref="S4.SS1.p1.4.m4.1.1">𝒙</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.4.m4.1c">{\bm{x}}</annotation></semantics></math> as the vectorized cache state of a single token, where the superscript indicates the layer index and the subscripts <math id="S4.SS1.p1.5.m5.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S4.SS1.p1.5.m5.1a"><mi id="S4.SS1.p1.5.m5.1.1" xref="S4.SS1.p1.5.m5.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.5.m5.1b"><ci id="S4.SS1.p1.5.m5.1.1.cmml" xref="S4.SS1.p1.5.m5.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.5.m5.1c">k</annotation></semantics></math> and <math id="S4.SS1.p1.6.m6.1" class="ltx_Math" alttext="v" display="inline"><semantics id="S4.SS1.p1.6.m6.1a"><mi id="S4.SS1.p1.6.m6.1.1" xref="S4.SS1.p1.6.m6.1.1.cmml">v</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.6.m6.1b"><ci id="S4.SS1.p1.6.m6.1.1.cmml" xref="S4.SS1.p1.6.m6.1.1">𝑣</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.6.m6.1c">v</annotation></semantics></math> denote the keys and values, respectively.
Specifically, for a pair of key/value tokens at the same position in layers <math id="S4.SS1.p1.7.m7.1" class="ltx_Math" alttext="l" display="inline"><semantics id="S4.SS1.p1.7.m7.1a"><mi id="S4.SS1.p1.7.m7.1.1" xref="S4.SS1.p1.7.m7.1.1.cmml">l</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.7.m7.1b"><ci id="S4.SS1.p1.7.m7.1.1.cmml" xref="S4.SS1.p1.7.m7.1.1">𝑙</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.7.m7.1c">l</annotation></semantics></math> and <math id="S4.SS1.p1.8.m8.1" class="ltx_Math" alttext="l-1" display="inline"><semantics id="S4.SS1.p1.8.m8.1a"><mrow id="S4.SS1.p1.8.m8.1.1" xref="S4.SS1.p1.8.m8.1.1.cmml"><mi id="S4.SS1.p1.8.m8.1.1.2" xref="S4.SS1.p1.8.m8.1.1.2.cmml">l</mi><mo id="S4.SS1.p1.8.m8.1.1.1" xref="S4.SS1.p1.8.m8.1.1.1.cmml">−</mo><mn id="S4.SS1.p1.8.m8.1.1.3" xref="S4.SS1.p1.8.m8.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.8.m8.1b"><apply id="S4.SS1.p1.8.m8.1.1.cmml" xref="S4.SS1.p1.8.m8.1.1"><minus id="S4.SS1.p1.8.m8.1.1.1.cmml" xref="S4.SS1.p1.8.m8.1.1.1"></minus><ci id="S4.SS1.p1.8.m8.1.1.2.cmml" xref="S4.SS1.p1.8.m8.1.1.2">𝑙</ci><cn type="integer" id="S4.SS1.p1.8.m8.1.1.3.cmml" xref="S4.SS1.p1.8.m8.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.8.m8.1c">l-1</annotation></semantics></math>, the merged cache is computed as</p>
<table id="S4.E1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.E1.m1.6" class="ltx_Math" alttext="\begin{array}[]{ll}\begin{aligned} {\bm{c}}_{k}^{l,l-1}=F({\bm{x}}_{k}^{l},{\bm{x}}_{k}^{l-1}),\\
{\bm{c}}_{v}^{l,l-1}=F({\bm{x}}_{v}^{l},{\bm{x}}_{v}^{l-1}).\end{aligned}\end{array}" display="block"><semantics id="S4.E1.m1.6a"><mtable columnspacing="5pt" displaystyle="true" id="S4.E1.m1.6.6" xref="S4.E1.m1.6.6.cmml"><mtr id="S4.E1.m1.6.6a" xref="S4.E1.m1.6.6.cmml"><mtd class="ltx_align_left" columnalign="left" id="S4.E1.m1.6.6b" xref="S4.E1.m1.6.6.cmml"><mtable displaystyle="true" rowspacing="0pt" id="S4.E1.m1.6.6.6.6.6" xref="S4.E1.m1.6.6.6.6.6.cmml"><mtr id="S4.E1.m1.6.6.6.6.6a" xref="S4.E1.m1.6.6.6.6.6.cmml"><mtd class="ltx_align_right" columnalign="right" id="S4.E1.m1.6.6.6.6.6b" xref="S4.E1.m1.6.6.6.6.6.cmml"><mrow id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.cmml"><mrow id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.cmml"><msubsup id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.4" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.4.cmml"><mi id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.4.2.2" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.4.2.2.cmml">𝒄</mi><mi id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.4.2.3" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.4.2.3.cmml">k</mi><mrow id="S4.E1.m1.2.2.2.2.2.2.2.2.2.2.2.2" xref="S4.E1.m1.2.2.2.2.2.2.2.2.2.2.2.3.cmml"><mi id="S4.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S4.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">l</mi><mo id="S4.E1.m1.2.2.2.2.2.2.2.2.2.2.2.2.2" xref="S4.E1.m1.2.2.2.2.2.2.2.2.2.2.2.3.cmml">,</mo><mrow id="S4.E1.m1.2.2.2.2.2.2.2.2.2.2.2.2.1" xref="S4.E1.m1.2.2.2.2.2.2.2.2.2.2.2.2.1.cmml"><mi id="S4.E1.m1.2.2.2.2.2.2.2.2.2.2.2.2.1.2" xref="S4.E1.m1.2.2.2.2.2.2.2.2.2.2.2.2.1.2.cmml">l</mi><mo id="S4.E1.m1.2.2.2.2.2.2.2.2.2.2.2.2.1.1" xref="S4.E1.m1.2.2.2.2.2.2.2.2.2.2.2.2.1.1.cmml">−</mo><mn id="S4.E1.m1.2.2.2.2.2.2.2.2.2.2.2.2.1.3" xref="S4.E1.m1.2.2.2.2.2.2.2.2.2.2.2.2.1.3.cmml">1</mn></mrow></mrow></msubsup><mo id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.3" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.3.cmml">=</mo><mrow id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.cmml"><mi id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.4" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.4.cmml">F</mi><mo lspace="0em" rspace="0em" id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.3" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.3.cmml">​</mo><mrow id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.3.cmml"><mo stretchy="false" id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.3" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.3.cmml">(</mo><msubsup id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.cmml"><mi id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.2.2" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.2.2.cmml">𝒙</mi><mi id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.2.3" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.2.3.cmml">k</mi><mi id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.3" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.3.cmml">l</mi></msubsup><mo id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.4" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.3.cmml">,</mo><msubsup id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.cmml"><mi id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.2.2" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.2.2.cmml">𝒙</mi><mi id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.2.3" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.2.3.cmml">k</mi><mrow id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.3" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.3.cmml"><mi id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.3.2" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.3.2.cmml">l</mi><mo id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.3.1" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.3.1.cmml">−</mo><mn id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.3.3" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.3.3.cmml">1</mn></mrow></msubsup><mo stretchy="false" id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.5" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.3.cmml">)</mo></mrow></mrow></mrow><mo id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.2" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.cmml">,</mo></mrow></mtd></mtr><mtr id="S4.E1.m1.6.6.6.6.6c" xref="S4.E1.m1.6.6.6.6.6.cmml"><mtd class="ltx_align_right" columnalign="right" id="S4.E1.m1.6.6.6.6.6d" xref="S4.E1.m1.6.6.6.6.6.cmml"><mrow id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.cmml"><mrow id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.cmml"><msubsup id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.4" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.4.cmml"><mi id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.4.2.2" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.4.2.2.cmml">𝒄</mi><mi id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.4.2.3" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.4.2.3.cmml">v</mi><mrow id="S4.E1.m1.5.5.5.5.5.5.5.2.2.2.2.2" xref="S4.E1.m1.5.5.5.5.5.5.5.2.2.2.2.3.cmml"><mi id="S4.E1.m1.4.4.4.4.4.4.4.1.1.1.1.1" xref="S4.E1.m1.4.4.4.4.4.4.4.1.1.1.1.1.cmml">l</mi><mo id="S4.E1.m1.5.5.5.5.5.5.5.2.2.2.2.2.2" xref="S4.E1.m1.5.5.5.5.5.5.5.2.2.2.2.3.cmml">,</mo><mrow id="S4.E1.m1.5.5.5.5.5.5.5.2.2.2.2.2.1" xref="S4.E1.m1.5.5.5.5.5.5.5.2.2.2.2.2.1.cmml"><mi id="S4.E1.m1.5.5.5.5.5.5.5.2.2.2.2.2.1.2" xref="S4.E1.m1.5.5.5.5.5.5.5.2.2.2.2.2.1.2.cmml">l</mi><mo id="S4.E1.m1.5.5.5.5.5.5.5.2.2.2.2.2.1.1" xref="S4.E1.m1.5.5.5.5.5.5.5.2.2.2.2.2.1.1.cmml">−</mo><mn id="S4.E1.m1.5.5.5.5.5.5.5.2.2.2.2.2.1.3" xref="S4.E1.m1.5.5.5.5.5.5.5.2.2.2.2.2.1.3.cmml">1</mn></mrow></mrow></msubsup><mo id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.3" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.3.cmml">=</mo><mrow id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.cmml"><mi id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.4" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.4.cmml">F</mi><mo lspace="0em" rspace="0em" id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.3" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.3.cmml">​</mo><mrow id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.3.cmml"><mo stretchy="false" id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.3" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.3.cmml">(</mo><msubsup id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.1.1.1.1" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.1.1.1.1.cmml"><mi id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.1.1.1.1.2.2" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.1.1.1.1.2.2.cmml">𝒙</mi><mi id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.1.1.1.1.2.3" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.1.1.1.1.2.3.cmml">v</mi><mi id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.1.1.1.1.3" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.1.1.1.1.3.cmml">l</mi></msubsup><mo id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.4" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.3.cmml">,</mo><msubsup id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.cmml"><mi id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.2.2" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.2.2.cmml">𝒙</mi><mi id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.2.3" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.2.3.cmml">v</mi><mrow id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.3" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.3.cmml"><mi id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.3.2" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.3.2.cmml">l</mi><mo id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.3.1" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.3.1.cmml">−</mo><mn id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.3.3" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.3.3.cmml">1</mn></mrow></msubsup><mo stretchy="false" id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.5" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.3.cmml">)</mo></mrow></mrow></mrow><mo lspace="0em" id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.2" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.cmml">.</mo></mrow></mtd></mtr></mtable></mtd><mtd id="S4.E1.m1.6.6c" xref="S4.E1.m1.6.6.cmml"></mtd></mtr></mtable><annotation-xml encoding="MathML-Content" id="S4.E1.m1.6b"><matrix id="S4.E1.m1.6.6.cmml" xref="S4.E1.m1.6.6"><matrixrow id="S4.E1.m1.6.6a.cmml" xref="S4.E1.m1.6.6"><matrix id="S4.E1.m1.6.6.6.6.6.cmml" xref="S4.E1.m1.6.6.6.6.6"><matrixrow id="S4.E1.m1.6.6.6.6.6a.cmml" xref="S4.E1.m1.6.6.6.6.6"><apply id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3"><eq id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.3.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.3"></eq><apply id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.4.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.4"><csymbol cd="ambiguous" id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.4.1.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.4">superscript</csymbol><apply id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.4.2.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.4"><csymbol cd="ambiguous" id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.4.2.1.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.4">subscript</csymbol><ci id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.4.2.2.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.4.2.2">𝒄</ci><ci id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.4.2.3.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.4.2.3">𝑘</ci></apply><list id="S4.E1.m1.2.2.2.2.2.2.2.2.2.2.2.3.cmml" xref="S4.E1.m1.2.2.2.2.2.2.2.2.2.2.2.2"><ci id="S4.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E1.m1.1.1.1.1.1.1.1.1.1.1.1.1">𝑙</ci><apply id="S4.E1.m1.2.2.2.2.2.2.2.2.2.2.2.2.1.cmml" xref="S4.E1.m1.2.2.2.2.2.2.2.2.2.2.2.2.1"><minus id="S4.E1.m1.2.2.2.2.2.2.2.2.2.2.2.2.1.1.cmml" xref="S4.E1.m1.2.2.2.2.2.2.2.2.2.2.2.2.1.1"></minus><ci id="S4.E1.m1.2.2.2.2.2.2.2.2.2.2.2.2.1.2.cmml" xref="S4.E1.m1.2.2.2.2.2.2.2.2.2.2.2.2.1.2">𝑙</ci><cn type="integer" id="S4.E1.m1.2.2.2.2.2.2.2.2.2.2.2.2.1.3.cmml" xref="S4.E1.m1.2.2.2.2.2.2.2.2.2.2.2.2.1.3">1</cn></apply></list></apply><apply id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2"><times id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.3.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.3"></times><ci id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.4.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.4">𝐹</ci><interval closure="open" id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.3.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2"><apply id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.1.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1">superscript</csymbol><apply id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.2.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.2.1.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1">subscript</csymbol><ci id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.2.2.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.2.2">𝒙</ci><ci id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.2.3.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.2.3">𝑘</ci></apply><ci id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.3.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.1.1.1.1.3">𝑙</ci></apply><apply id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2"><csymbol cd="ambiguous" id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.1.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2">superscript</csymbol><apply id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.2.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2"><csymbol cd="ambiguous" id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.2.1.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2">subscript</csymbol><ci id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.2.2.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.2.2">𝒙</ci><ci id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.2.3.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.2.3">𝑘</ci></apply><apply id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.3.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.3"><minus id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.3.1.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.3.1"></minus><ci id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.3.2.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.3.2">𝑙</ci><cn type="integer" id="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.3.3.cmml" xref="S4.E1.m1.3.3.3.3.3.3.3.3.3.3.1.2.2.2.2.3.3">1</cn></apply></apply></interval></apply></apply></matrixrow><matrixrow id="S4.E1.m1.6.6.6.6.6b.cmml" xref="S4.E1.m1.6.6.6.6.6"><apply id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3"><eq id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.3.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.3"></eq><apply id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.4.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.4"><csymbol cd="ambiguous" id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.4.1.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.4">superscript</csymbol><apply id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.4.2.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.4"><csymbol cd="ambiguous" id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.4.2.1.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.4">subscript</csymbol><ci id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.4.2.2.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.4.2.2">𝒄</ci><ci id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.4.2.3.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.4.2.3">𝑣</ci></apply><list id="S4.E1.m1.5.5.5.5.5.5.5.2.2.2.2.3.cmml" xref="S4.E1.m1.5.5.5.5.5.5.5.2.2.2.2.2"><ci id="S4.E1.m1.4.4.4.4.4.4.4.1.1.1.1.1.cmml" xref="S4.E1.m1.4.4.4.4.4.4.4.1.1.1.1.1">𝑙</ci><apply id="S4.E1.m1.5.5.5.5.5.5.5.2.2.2.2.2.1.cmml" xref="S4.E1.m1.5.5.5.5.5.5.5.2.2.2.2.2.1"><minus id="S4.E1.m1.5.5.5.5.5.5.5.2.2.2.2.2.1.1.cmml" xref="S4.E1.m1.5.5.5.5.5.5.5.2.2.2.2.2.1.1"></minus><ci id="S4.E1.m1.5.5.5.5.5.5.5.2.2.2.2.2.1.2.cmml" xref="S4.E1.m1.5.5.5.5.5.5.5.2.2.2.2.2.1.2">𝑙</ci><cn type="integer" id="S4.E1.m1.5.5.5.5.5.5.5.2.2.2.2.2.1.3.cmml" xref="S4.E1.m1.5.5.5.5.5.5.5.2.2.2.2.2.1.3">1</cn></apply></list></apply><apply id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2"><times id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.3.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.3"></times><ci id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.4.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.4">𝐹</ci><interval closure="open" id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.3.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2"><apply id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.1.1.1.1.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.1.1.1.1.1.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.1.1.1.1">superscript</csymbol><apply id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.1.1.1.1.2.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.1.1.1.1.2.1.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.1.1.1.1">subscript</csymbol><ci id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.1.1.1.1.2.2.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.1.1.1.1.2.2">𝒙</ci><ci id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.1.1.1.1.2.3.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.1.1.1.1.2.3">𝑣</ci></apply><ci id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.1.1.1.1.3.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.1.1.1.1.3">𝑙</ci></apply><apply id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2"><csymbol cd="ambiguous" id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.1.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2">superscript</csymbol><apply id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.2.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2"><csymbol cd="ambiguous" id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.2.1.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2">subscript</csymbol><ci id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.2.2.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.2.2">𝒙</ci><ci id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.2.3.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.2.3">𝑣</ci></apply><apply id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.3.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.3"><minus id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.3.1.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.3.1"></minus><ci id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.3.2.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.3.2">𝑙</ci><cn type="integer" id="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.3.3.cmml" xref="S4.E1.m1.6.6.6.6.6.6.6.3.3.3.1.2.2.2.2.3.3">1</cn></apply></apply></interval></apply></apply></matrixrow></matrix><cerror id="S4.E1.m1.6.6b.cmml" xref="S4.E1.m1.6.6"><csymbol cd="ambiguous" id="S4.E1.m1.6.6c.cmml" xref="S4.E1.m1.6.6">missing-subexpression</csymbol></cerror></matrixrow></matrix></annotation-xml><annotation encoding="application/x-tex" id="S4.E1.m1.6c">\begin{array}[]{ll}\begin{aligned} {\bm{c}}_{k}^{l,l-1}=F({\bm{x}}_{k}^{l},{\bm{x}}_{k}^{l-1}),\\
{\bm{c}}_{v}^{l,l-1}=F({\bm{x}}_{v}^{l},{\bm{x}}_{v}^{l-1}).\end{aligned}\end{array}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
<p id="S4.SS1.p1.9" class="ltx_p">This consolidation process effectively eliminates the need to store and process the original memory-intensive keys and values in each layer independently. Instead, it approximates a shared cache across the adjacent layers.</p>
</div>
<figure id="S4.F3" class="ltx_figure"><img src="/html/2405.14366/assets/x5.png" id="S4.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="170" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F3.17.8.1" class="ltx_text" style="font-size:90%;">Figure 3</span>: </span><span id="S4.F3.14.7" class="ltx_text" style="font-size:90%;">The illustration of the proposed method <span id="S4.F3.14.7.1" class="ltx_text ltx_font_bold">MiniCache</span>. (a) depicts the cross-layer compression process. We fetch the KV caches, from layers <math id="S4.F3.8.1.m1.1" class="ltx_Math" alttext="l" display="inline"><semantics id="S4.F3.8.1.m1.1b"><mi id="S4.F3.8.1.m1.1.1" xref="S4.F3.8.1.m1.1.1.cmml">l</mi><annotation-xml encoding="MathML-Content" id="S4.F3.8.1.m1.1c"><ci id="S4.F3.8.1.m1.1.1.cmml" xref="S4.F3.8.1.m1.1.1">𝑙</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.F3.8.1.m1.1d">l</annotation></semantics></math> and <math id="S4.F3.9.2.m2.1" class="ltx_Math" alttext="{l-1}" display="inline"><semantics id="S4.F3.9.2.m2.1b"><mrow id="S4.F3.9.2.m2.1.1" xref="S4.F3.9.2.m2.1.1.cmml"><mi id="S4.F3.9.2.m2.1.1.2" xref="S4.F3.9.2.m2.1.1.2.cmml">l</mi><mo id="S4.F3.9.2.m2.1.1.1" xref="S4.F3.9.2.m2.1.1.1.cmml">−</mo><mn id="S4.F3.9.2.m2.1.1.3" xref="S4.F3.9.2.m2.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.F3.9.2.m2.1c"><apply id="S4.F3.9.2.m2.1.1.cmml" xref="S4.F3.9.2.m2.1.1"><minus id="S4.F3.9.2.m2.1.1.1.cmml" xref="S4.F3.9.2.m2.1.1.1"></minus><ci id="S4.F3.9.2.m2.1.1.2.cmml" xref="S4.F3.9.2.m2.1.1.2">𝑙</ci><cn type="integer" id="S4.F3.9.2.m2.1.1.3.cmml" xref="S4.F3.9.2.m2.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.F3.9.2.m2.1d">{l-1}</annotation></semantics></math>, and merge them into shared states via Eq. (<a href="#S4.E3" title="Equation 3 ‣ 4.2 KV Cache Merging and Restoration ‣ 4 Method ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>). Additionally, we compute the <math id="S4.F3.10.3.m3.1" class="ltx_Math" alttext="\ell_{2}" display="inline"><semantics id="S4.F3.10.3.m3.1b"><msub id="S4.F3.10.3.m3.1.1" xref="S4.F3.10.3.m3.1.1.cmml"><mi mathvariant="normal" id="S4.F3.10.3.m3.1.1.2" xref="S4.F3.10.3.m3.1.1.2.cmml">ℓ</mi><mn id="S4.F3.10.3.m3.1.1.3" xref="S4.F3.10.3.m3.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S4.F3.10.3.m3.1c"><apply id="S4.F3.10.3.m3.1.1.cmml" xref="S4.F3.10.3.m3.1.1"><csymbol cd="ambiguous" id="S4.F3.10.3.m3.1.1.1.cmml" xref="S4.F3.10.3.m3.1.1">subscript</csymbol><ci id="S4.F3.10.3.m3.1.1.2.cmml" xref="S4.F3.10.3.m3.1.1.2">ℓ</ci><cn type="integer" id="S4.F3.10.3.m3.1.1.3.cmml" xref="S4.F3.10.3.m3.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.F3.10.3.m3.1d">\ell_{2}</annotation></semantics></math> norm for the caches to obtain their magnitudes. Furthermore, we select unmergable tokens for retention, then store merged cache, retention tokens, and magnitudes at layer <math id="S4.F3.11.4.m4.1" class="ltx_Math" alttext="l" display="inline"><semantics id="S4.F3.11.4.m4.1b"><mi id="S4.F3.11.4.m4.1.1" xref="S4.F3.11.4.m4.1.1.cmml">l</mi><annotation-xml encoding="MathML-Content" id="S4.F3.11.4.m4.1c"><ci id="S4.F3.11.4.m4.1.1.cmml" xref="S4.F3.11.4.m4.1.1">𝑙</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.F3.11.4.m4.1d">l</annotation></semantics></math> in <math id="S4.F3.12.5.m5.1" class="ltx_Math" alttext="{\bm{C}}" display="inline"><semantics id="S4.F3.12.5.m5.1b"><mi id="S4.F3.12.5.m5.1.1" xref="S4.F3.12.5.m5.1.1.cmml">𝑪</mi><annotation-xml encoding="MathML-Content" id="S4.F3.12.5.m5.1c"><ci id="S4.F3.12.5.m5.1.1.cmml" xref="S4.F3.12.5.m5.1.1">𝑪</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.F3.12.5.m5.1d">{\bm{C}}</annotation></semantics></math>. (b) demonstrates the restoration process for layers <math id="S4.F3.13.6.m6.1" class="ltx_Math" alttext="l" display="inline"><semantics id="S4.F3.13.6.m6.1b"><mi id="S4.F3.13.6.m6.1.1" xref="S4.F3.13.6.m6.1.1.cmml">l</mi><annotation-xml encoding="MathML-Content" id="S4.F3.13.6.m6.1c"><ci id="S4.F3.13.6.m6.1.1.cmml" xref="S4.F3.13.6.m6.1.1">𝑙</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.F3.13.6.m6.1d">l</annotation></semantics></math> and <math id="S4.F3.14.7.m7.1" class="ltx_Math" alttext="l-1" display="inline"><semantics id="S4.F3.14.7.m7.1b"><mrow id="S4.F3.14.7.m7.1.1" xref="S4.F3.14.7.m7.1.1.cmml"><mi id="S4.F3.14.7.m7.1.1.2" xref="S4.F3.14.7.m7.1.1.2.cmml">l</mi><mo id="S4.F3.14.7.m7.1.1.1" xref="S4.F3.14.7.m7.1.1.1.cmml">−</mo><mn id="S4.F3.14.7.m7.1.1.3" xref="S4.F3.14.7.m7.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.F3.14.7.m7.1c"><apply id="S4.F3.14.7.m7.1.1.cmml" xref="S4.F3.14.7.m7.1.1"><minus id="S4.F3.14.7.m7.1.1.1.cmml" xref="S4.F3.14.7.m7.1.1.1"></minus><ci id="S4.F3.14.7.m7.1.1.2.cmml" xref="S4.F3.14.7.m7.1.1.2">𝑙</ci><cn type="integer" id="S4.F3.14.7.m7.1.1.3.cmml" xref="S4.F3.14.7.m7.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.F3.14.7.m7.1d">l-1</annotation></semantics></math>, which includes magnitude rescaling in Eq. (<a href="#S4.E2" title="Equation 2 ‣ 4.2 KV Cache Merging and Restoration ‣ 4 Method ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>) and retention token recovery. </span></figcaption>
</figure>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>KV Cache Merging and Restoration</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.6" class="ltx_p"><span id="S4.SS2.p1.6.1" class="ltx_text ltx_font_bold">Reparameterization-based cache merging.</span> To perform the pairwise merging, one solution is to directly average a pair of KV tokens, analogous to model merging <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib61" title="" class="ltx_ref">61</a>, <a href="#bib.bib60" title="" class="ltx_ref">60</a>]</cite>. However, we observe that direct averaging can cause significant information loss. We conjecture that the distance between activations can be larger than that of weights due to the presence of outlier activation channels with extremely large magnitudes in LLMs <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib70" title="" class="ltx_ref">70</a>, <a href="#bib.bib71" title="" class="ltx_ref">71</a>]</cite>, while weights typically have relatively quite small magnitudes.
A potential method to compensate for this information loss is to project from <math id="S4.SS2.p1.1.m1.1" class="ltx_Math" alttext="{\bm{c}}" display="inline"><semantics id="S4.SS2.p1.1.m1.1a"><mi id="S4.SS2.p1.1.m1.1.1" xref="S4.SS2.p1.1.m1.1.1.cmml">𝒄</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.1.m1.1b"><ci id="S4.SS2.p1.1.m1.1.1.cmml" xref="S4.SS2.p1.1.m1.1.1">𝒄</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.1.m1.1c">{\bm{c}}</annotation></semantics></math> to <math id="S4.SS2.p1.2.m2.1" class="ltx_Math" alttext="{\bm{x}}^{l-1}" display="inline"><semantics id="S4.SS2.p1.2.m2.1a"><msup id="S4.SS2.p1.2.m2.1.1" xref="S4.SS2.p1.2.m2.1.1.cmml"><mi id="S4.SS2.p1.2.m2.1.1.2" xref="S4.SS2.p1.2.m2.1.1.2.cmml">𝒙</mi><mrow id="S4.SS2.p1.2.m2.1.1.3" xref="S4.SS2.p1.2.m2.1.1.3.cmml"><mi id="S4.SS2.p1.2.m2.1.1.3.2" xref="S4.SS2.p1.2.m2.1.1.3.2.cmml">l</mi><mo id="S4.SS2.p1.2.m2.1.1.3.1" xref="S4.SS2.p1.2.m2.1.1.3.1.cmml">−</mo><mn id="S4.SS2.p1.2.m2.1.1.3.3" xref="S4.SS2.p1.2.m2.1.1.3.3.cmml">1</mn></mrow></msup><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.2.m2.1b"><apply id="S4.SS2.p1.2.m2.1.1.cmml" xref="S4.SS2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.2.m2.1.1.1.cmml" xref="S4.SS2.p1.2.m2.1.1">superscript</csymbol><ci id="S4.SS2.p1.2.m2.1.1.2.cmml" xref="S4.SS2.p1.2.m2.1.1.2">𝒙</ci><apply id="S4.SS2.p1.2.m2.1.1.3.cmml" xref="S4.SS2.p1.2.m2.1.1.3"><minus id="S4.SS2.p1.2.m2.1.1.3.1.cmml" xref="S4.SS2.p1.2.m2.1.1.3.1"></minus><ci id="S4.SS2.p1.2.m2.1.1.3.2.cmml" xref="S4.SS2.p1.2.m2.1.1.3.2">𝑙</ci><cn type="integer" id="S4.SS2.p1.2.m2.1.1.3.3.cmml" xref="S4.SS2.p1.2.m2.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.2.m2.1c">{\bm{x}}^{l-1}</annotation></semantics></math> and <math id="S4.SS2.p1.3.m3.1" class="ltx_Math" alttext="{\bm{x}}^{l}" display="inline"><semantics id="S4.SS2.p1.3.m3.1a"><msup id="S4.SS2.p1.3.m3.1.1" xref="S4.SS2.p1.3.m3.1.1.cmml"><mi id="S4.SS2.p1.3.m3.1.1.2" xref="S4.SS2.p1.3.m3.1.1.2.cmml">𝒙</mi><mi id="S4.SS2.p1.3.m3.1.1.3" xref="S4.SS2.p1.3.m3.1.1.3.cmml">l</mi></msup><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.3.m3.1b"><apply id="S4.SS2.p1.3.m3.1.1.cmml" xref="S4.SS2.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.3.m3.1.1.1.cmml" xref="S4.SS2.p1.3.m3.1.1">superscript</csymbol><ci id="S4.SS2.p1.3.m3.1.1.2.cmml" xref="S4.SS2.p1.3.m3.1.1.2">𝒙</ci><ci id="S4.SS2.p1.3.m3.1.1.3.cmml" xref="S4.SS2.p1.3.m3.1.1.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.3.m3.1c">{\bm{x}}^{l}</annotation></semantics></math>, then rescale the projected vectors based on their relative magnitudes to exactly restore the original states. However, this approach requires extensive additional storage and computations; for example, restoring <math id="S4.SS2.p1.4.m4.1" class="ltx_Math" alttext="{\bm{x}}^{l-1}" display="inline"><semantics id="S4.SS2.p1.4.m4.1a"><msup id="S4.SS2.p1.4.m4.1.1" xref="S4.SS2.p1.4.m4.1.1.cmml"><mi id="S4.SS2.p1.4.m4.1.1.2" xref="S4.SS2.p1.4.m4.1.1.2.cmml">𝒙</mi><mrow id="S4.SS2.p1.4.m4.1.1.3" xref="S4.SS2.p1.4.m4.1.1.3.cmml"><mi id="S4.SS2.p1.4.m4.1.1.3.2" xref="S4.SS2.p1.4.m4.1.1.3.2.cmml">l</mi><mo id="S4.SS2.p1.4.m4.1.1.3.1" xref="S4.SS2.p1.4.m4.1.1.3.1.cmml">−</mo><mn id="S4.SS2.p1.4.m4.1.1.3.3" xref="S4.SS2.p1.4.m4.1.1.3.3.cmml">1</mn></mrow></msup><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.4.m4.1b"><apply id="S4.SS2.p1.4.m4.1.1.cmml" xref="S4.SS2.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.4.m4.1.1.1.cmml" xref="S4.SS2.p1.4.m4.1.1">superscript</csymbol><ci id="S4.SS2.p1.4.m4.1.1.2.cmml" xref="S4.SS2.p1.4.m4.1.1.2">𝒙</ci><apply id="S4.SS2.p1.4.m4.1.1.3.cmml" xref="S4.SS2.p1.4.m4.1.1.3"><minus id="S4.SS2.p1.4.m4.1.1.3.1.cmml" xref="S4.SS2.p1.4.m4.1.1.3.1"></minus><ci id="S4.SS2.p1.4.m4.1.1.3.2.cmml" xref="S4.SS2.p1.4.m4.1.1.3.2">𝑙</ci><cn type="integer" id="S4.SS2.p1.4.m4.1.1.3.3.cmml" xref="S4.SS2.p1.4.m4.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.4.m4.1c">{\bm{x}}^{l-1}</annotation></semantics></math> needs both <math id="S4.SS2.p1.5.m5.1" class="ltx_Math" alttext="{\bm{c}}" display="inline"><semantics id="S4.SS2.p1.5.m5.1a"><mi id="S4.SS2.p1.5.m5.1.1" xref="S4.SS2.p1.5.m5.1.1.cmml">𝒄</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.5.m5.1b"><ci id="S4.SS2.p1.5.m5.1.1.cmml" xref="S4.SS2.p1.5.m5.1.1">𝒄</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.5.m5.1c">{\bm{c}}</annotation></semantics></math> and <math id="S4.SS2.p1.6.m6.1" class="ltx_Math" alttext="{\bm{x}}^{l}" display="inline"><semantics id="S4.SS2.p1.6.m6.1a"><msup id="S4.SS2.p1.6.m6.1.1" xref="S4.SS2.p1.6.m6.1.1.cmml"><mi id="S4.SS2.p1.6.m6.1.1.2" xref="S4.SS2.p1.6.m6.1.1.2.cmml">𝒙</mi><mi id="S4.SS2.p1.6.m6.1.1.3" xref="S4.SS2.p1.6.m6.1.1.3.cmml">l</mi></msup><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.6.m6.1b"><apply id="S4.SS2.p1.6.m6.1.1.cmml" xref="S4.SS2.p1.6.m6.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.6.m6.1.1.1.cmml" xref="S4.SS2.p1.6.m6.1.1">superscript</csymbol><ci id="S4.SS2.p1.6.m6.1.1.2.cmml" xref="S4.SS2.p1.6.m6.1.1.2">𝒙</ci><ci id="S4.SS2.p1.6.m6.1.1.3.cmml" xref="S4.SS2.p1.6.m6.1.1.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.6.m6.1c">{\bm{x}}^{l}</annotation></semantics></math>, which undermines the benefits of cache merging.
To efficiently merge token pairs, we draw inspiration from weight normalization <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib21" title="" class="ltx_ref">21</a>]</cite>, which disentangles model parameters into the <em id="S4.SS2.p1.6.2" class="ltx_emph ltx_font_italic">magnitude</em> and <em id="S4.SS2.p1.6.3" class="ltx_emph ltx_font_italic">direction</em> components to accelerate the convergence of stochastic gradient descent. Additionally, we take cues from DoRA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib72" title="" class="ltx_ref">72</a>]</cite>, which employs a similar way to resemble the learning behavior of parameter-efficient fine-tuning compared to full fine-tuning.
In our case, the reparameterization can be formulated as follows:</p>
<table id="S4.E2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.E2.m1.13" class="ltx_Math" alttext="\begin{array}[]{ll}\hat{{\bm{x}}}^{l}={\bm{e}}^{l,l-1}\cdot\frac{\lVert{\bm{x}}^{l}\rVert}{\lVert{\bm{e}}^{l,l-1}\rVert},\ \hat{{\bm{x}}}^{l-1}={\bm{e}}^{l,l-1}\cdot\frac{\lVert{\bm{x}}^{l-1}\rVert}{\lVert{\bm{e}}^{l,l-1}\rVert},\end{array}" display="block"><semantics id="S4.E2.m1.13a"><mtable columnspacing="5pt" displaystyle="true" id="S4.E2.m1.13.13" xref="S4.E2.m1.13.13.cmml"><mtr id="S4.E2.m1.13.13a" xref="S4.E2.m1.13.13.cmml"><mtd class="ltx_align_left" columnalign="left" id="S4.E2.m1.13.13b" xref="S4.E2.m1.13.13.cmml"><mrow id="S4.E2.m1.13.13.13.13.13.13" xref="S4.E2.m1.13.13.cmml"><mrow id="S4.E2.m1.13.13.13.13.13.13.1.2" xref="S4.E2.m1.13.13.13.13.13.13.1.3.cmml"><mrow id="S4.E2.m1.13.13.13.13.13.13.1.1.1" xref="S4.E2.m1.13.13.13.13.13.13.1.1.1.cmml"><msup id="S4.E2.m1.13.13.13.13.13.13.1.1.1.2" xref="S4.E2.m1.13.13.13.13.13.13.1.1.1.2.cmml"><mover accent="true" id="S4.E2.m1.13.13.13.13.13.13.1.1.1.2.2" xref="S4.E2.m1.13.13.13.13.13.13.1.1.1.2.2.cmml"><mi id="S4.E2.m1.13.13.13.13.13.13.1.1.1.2.2.2" xref="S4.E2.m1.13.13.13.13.13.13.1.1.1.2.2.2.cmml">𝒙</mi><mo id="S4.E2.m1.13.13.13.13.13.13.1.1.1.2.2.1" xref="S4.E2.m1.13.13.13.13.13.13.1.1.1.2.2.1.cmml">^</mo></mover><mi id="S4.E2.m1.13.13.13.13.13.13.1.1.1.2.3" xref="S4.E2.m1.13.13.13.13.13.13.1.1.1.2.3.cmml">l</mi></msup><mo id="S4.E2.m1.13.13.13.13.13.13.1.1.1.1" xref="S4.E2.m1.13.13.13.13.13.13.1.1.1.1.cmml">=</mo><mrow id="S4.E2.m1.13.13.13.13.13.13.1.1.1.3" xref="S4.E2.m1.13.13.13.13.13.13.1.1.1.3.cmml"><msup id="S4.E2.m1.13.13.13.13.13.13.1.1.1.3.2" xref="S4.E2.m1.13.13.13.13.13.13.1.1.1.3.2.cmml"><mi id="S4.E2.m1.13.13.13.13.13.13.1.1.1.3.2.2" xref="S4.E2.m1.13.13.13.13.13.13.1.1.1.3.2.2.cmml">𝒆</mi><mrow id="S4.E2.m1.2.2.2.2.2.2.2.2" xref="S4.E2.m1.2.2.2.2.2.2.2.3.cmml"><mi id="S4.E2.m1.1.1.1.1.1.1.1.1" xref="S4.E2.m1.1.1.1.1.1.1.1.1.cmml">l</mi><mo id="S4.E2.m1.2.2.2.2.2.2.2.2.2" xref="S4.E2.m1.2.2.2.2.2.2.2.3.cmml">,</mo><mrow id="S4.E2.m1.2.2.2.2.2.2.2.2.1" xref="S4.E2.m1.2.2.2.2.2.2.2.2.1.cmml"><mi id="S4.E2.m1.2.2.2.2.2.2.2.2.1.2" xref="S4.E2.m1.2.2.2.2.2.2.2.2.1.2.cmml">l</mi><mo id="S4.E2.m1.2.2.2.2.2.2.2.2.1.1" xref="S4.E2.m1.2.2.2.2.2.2.2.2.1.1.cmml">−</mo><mn id="S4.E2.m1.2.2.2.2.2.2.2.2.1.3" xref="S4.E2.m1.2.2.2.2.2.2.2.2.1.3.cmml">1</mn></mrow></mrow></msup><mo lspace="0.222em" rspace="0.222em" id="S4.E2.m1.13.13.13.13.13.13.1.1.1.3.1" xref="S4.E2.m1.13.13.13.13.13.13.1.1.1.3.1.cmml">⋅</mo><mstyle displaystyle="false" id="S4.E2.m1.6.6.6.6.6.6" xref="S4.E2.m1.6.6.6.6.6.6.cmml"><mfrac id="S4.E2.m1.6.6.6.6.6.6a" xref="S4.E2.m1.6.6.6.6.6.6.cmml"><mrow id="S4.E2.m1.3.3.3.3.3.3.1.1" xref="S4.E2.m1.3.3.3.3.3.3.1.2.cmml"><mo fence="true" rspace="0em" id="S4.E2.m1.3.3.3.3.3.3.1.1.2" xref="S4.E2.m1.3.3.3.3.3.3.1.2.1.cmml">∥</mo><msup id="S4.E2.m1.3.3.3.3.3.3.1.1.1" xref="S4.E2.m1.3.3.3.3.3.3.1.1.1.cmml"><mi id="S4.E2.m1.3.3.3.3.3.3.1.1.1.2" xref="S4.E2.m1.3.3.3.3.3.3.1.1.1.2.cmml">𝒙</mi><mi id="S4.E2.m1.3.3.3.3.3.3.1.1.1.3" xref="S4.E2.m1.3.3.3.3.3.3.1.1.1.3.cmml">l</mi></msup><mo fence="true" lspace="0em" id="S4.E2.m1.3.3.3.3.3.3.1.1.3" xref="S4.E2.m1.3.3.3.3.3.3.1.2.1.cmml">∥</mo></mrow><mrow id="S4.E2.m1.6.6.6.6.6.6.4.3" xref="S4.E2.m1.6.6.6.6.6.6.4.4.cmml"><mo fence="true" rspace="0em" id="S4.E2.m1.6.6.6.6.6.6.4.3.2" xref="S4.E2.m1.6.6.6.6.6.6.4.4.1.cmml">∥</mo><msup id="S4.E2.m1.6.6.6.6.6.6.4.3.1" xref="S4.E2.m1.6.6.6.6.6.6.4.3.1.cmml"><mi id="S4.E2.m1.6.6.6.6.6.6.4.3.1.2" xref="S4.E2.m1.6.6.6.6.6.6.4.3.1.2.cmml">𝒆</mi><mrow id="S4.E2.m1.5.5.5.5.5.5.3.2.2.2" xref="S4.E2.m1.5.5.5.5.5.5.3.2.2.3.cmml"><mi id="S4.E2.m1.4.4.4.4.4.4.2.1.1.1" xref="S4.E2.m1.4.4.4.4.4.4.2.1.1.1.cmml">l</mi><mo id="S4.E2.m1.5.5.5.5.5.5.3.2.2.2.2" xref="S4.E2.m1.5.5.5.5.5.5.3.2.2.3.cmml">,</mo><mrow id="S4.E2.m1.5.5.5.5.5.5.3.2.2.2.1" xref="S4.E2.m1.5.5.5.5.5.5.3.2.2.2.1.cmml"><mi id="S4.E2.m1.5.5.5.5.5.5.3.2.2.2.1.2" xref="S4.E2.m1.5.5.5.5.5.5.3.2.2.2.1.2.cmml">l</mi><mo id="S4.E2.m1.5.5.5.5.5.5.3.2.2.2.1.1" xref="S4.E2.m1.5.5.5.5.5.5.3.2.2.2.1.1.cmml">−</mo><mn id="S4.E2.m1.5.5.5.5.5.5.3.2.2.2.1.3" xref="S4.E2.m1.5.5.5.5.5.5.3.2.2.2.1.3.cmml">1</mn></mrow></mrow></msup><mo fence="true" lspace="0em" id="S4.E2.m1.6.6.6.6.6.6.4.3.3" xref="S4.E2.m1.6.6.6.6.6.6.4.4.1.cmml">∥</mo></mrow></mfrac></mstyle></mrow></mrow><mo rspace="0.667em" id="S4.E2.m1.13.13.13.13.13.13.1.2.3" xref="S4.E2.m1.13.13.13.13.13.13.1.3a.cmml">,</mo><mrow id="S4.E2.m1.13.13.13.13.13.13.1.2.2" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.cmml"><msup id="S4.E2.m1.13.13.13.13.13.13.1.2.2.2" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.cmml"><mover accent="true" id="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.2" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.2.cmml"><mi id="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.2.2" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.2.2.cmml">𝒙</mi><mo id="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.2.1" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.2.1.cmml">^</mo></mover><mrow id="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.3" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.3.cmml"><mi id="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.3.2" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.3.2.cmml">l</mi><mo id="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.3.1" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.3.1.cmml">−</mo><mn id="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.3.3" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.3.3.cmml">1</mn></mrow></msup><mo id="S4.E2.m1.13.13.13.13.13.13.1.2.2.1" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.1.cmml">=</mo><mrow id="S4.E2.m1.13.13.13.13.13.13.1.2.2.3" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.3.cmml"><msup id="S4.E2.m1.13.13.13.13.13.13.1.2.2.3.2" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.3.2.cmml"><mi id="S4.E2.m1.13.13.13.13.13.13.1.2.2.3.2.2" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.3.2.2.cmml">𝒆</mi><mrow id="S4.E2.m1.8.8.8.8.8.8.2.2" xref="S4.E2.m1.8.8.8.8.8.8.2.3.cmml"><mi id="S4.E2.m1.7.7.7.7.7.7.1.1" xref="S4.E2.m1.7.7.7.7.7.7.1.1.cmml">l</mi><mo id="S4.E2.m1.8.8.8.8.8.8.2.2.2" xref="S4.E2.m1.8.8.8.8.8.8.2.3.cmml">,</mo><mrow id="S4.E2.m1.8.8.8.8.8.8.2.2.1" xref="S4.E2.m1.8.8.8.8.8.8.2.2.1.cmml"><mi id="S4.E2.m1.8.8.8.8.8.8.2.2.1.2" xref="S4.E2.m1.8.8.8.8.8.8.2.2.1.2.cmml">l</mi><mo id="S4.E2.m1.8.8.8.8.8.8.2.2.1.1" xref="S4.E2.m1.8.8.8.8.8.8.2.2.1.1.cmml">−</mo><mn id="S4.E2.m1.8.8.8.8.8.8.2.2.1.3" xref="S4.E2.m1.8.8.8.8.8.8.2.2.1.3.cmml">1</mn></mrow></mrow></msup><mo lspace="0.222em" rspace="0.222em" id="S4.E2.m1.13.13.13.13.13.13.1.2.2.3.1" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.3.1.cmml">⋅</mo><mstyle displaystyle="false" id="S4.E2.m1.12.12.12.12.12.12" xref="S4.E2.m1.12.12.12.12.12.12.cmml"><mfrac id="S4.E2.m1.12.12.12.12.12.12a" xref="S4.E2.m1.12.12.12.12.12.12.cmml"><mrow id="S4.E2.m1.9.9.9.9.9.9.1.1" xref="S4.E2.m1.9.9.9.9.9.9.1.2.cmml"><mo fence="true" rspace="0em" id="S4.E2.m1.9.9.9.9.9.9.1.1.2" xref="S4.E2.m1.9.9.9.9.9.9.1.2.1.cmml">∥</mo><msup id="S4.E2.m1.9.9.9.9.9.9.1.1.1" xref="S4.E2.m1.9.9.9.9.9.9.1.1.1.cmml"><mi id="S4.E2.m1.9.9.9.9.9.9.1.1.1.2" xref="S4.E2.m1.9.9.9.9.9.9.1.1.1.2.cmml">𝒙</mi><mrow id="S4.E2.m1.9.9.9.9.9.9.1.1.1.3" xref="S4.E2.m1.9.9.9.9.9.9.1.1.1.3.cmml"><mi id="S4.E2.m1.9.9.9.9.9.9.1.1.1.3.2" xref="S4.E2.m1.9.9.9.9.9.9.1.1.1.3.2.cmml">l</mi><mo id="S4.E2.m1.9.9.9.9.9.9.1.1.1.3.1" xref="S4.E2.m1.9.9.9.9.9.9.1.1.1.3.1.cmml">−</mo><mn id="S4.E2.m1.9.9.9.9.9.9.1.1.1.3.3" xref="S4.E2.m1.9.9.9.9.9.9.1.1.1.3.3.cmml">1</mn></mrow></msup><mo fence="true" lspace="0em" id="S4.E2.m1.9.9.9.9.9.9.1.1.3" xref="S4.E2.m1.9.9.9.9.9.9.1.2.1.cmml">∥</mo></mrow><mrow id="S4.E2.m1.12.12.12.12.12.12.4.3" xref="S4.E2.m1.12.12.12.12.12.12.4.4.cmml"><mo fence="true" rspace="0em" id="S4.E2.m1.12.12.12.12.12.12.4.3.2" xref="S4.E2.m1.12.12.12.12.12.12.4.4.1.cmml">∥</mo><msup id="S4.E2.m1.12.12.12.12.12.12.4.3.1" xref="S4.E2.m1.12.12.12.12.12.12.4.3.1.cmml"><mi id="S4.E2.m1.12.12.12.12.12.12.4.3.1.2" xref="S4.E2.m1.12.12.12.12.12.12.4.3.1.2.cmml">𝒆</mi><mrow id="S4.E2.m1.11.11.11.11.11.11.3.2.2.2" xref="S4.E2.m1.11.11.11.11.11.11.3.2.2.3.cmml"><mi id="S4.E2.m1.10.10.10.10.10.10.2.1.1.1" xref="S4.E2.m1.10.10.10.10.10.10.2.1.1.1.cmml">l</mi><mo id="S4.E2.m1.11.11.11.11.11.11.3.2.2.2.2" xref="S4.E2.m1.11.11.11.11.11.11.3.2.2.3.cmml">,</mo><mrow id="S4.E2.m1.11.11.11.11.11.11.3.2.2.2.1" xref="S4.E2.m1.11.11.11.11.11.11.3.2.2.2.1.cmml"><mi id="S4.E2.m1.11.11.11.11.11.11.3.2.2.2.1.2" xref="S4.E2.m1.11.11.11.11.11.11.3.2.2.2.1.2.cmml">l</mi><mo id="S4.E2.m1.11.11.11.11.11.11.3.2.2.2.1.1" xref="S4.E2.m1.11.11.11.11.11.11.3.2.2.2.1.1.cmml">−</mo><mn id="S4.E2.m1.11.11.11.11.11.11.3.2.2.2.1.3" xref="S4.E2.m1.11.11.11.11.11.11.3.2.2.2.1.3.cmml">1</mn></mrow></mrow></msup><mo fence="true" lspace="0em" id="S4.E2.m1.12.12.12.12.12.12.4.3.3" xref="S4.E2.m1.12.12.12.12.12.12.4.4.1.cmml">∥</mo></mrow></mfrac></mstyle></mrow></mrow></mrow><mo id="S4.E2.m1.13.13.13.13.13.13.2" xref="S4.E2.m1.13.13.cmml">,</mo></mrow></mtd><mtd id="S4.E2.m1.13.13c" xref="S4.E2.m1.13.13.cmml"></mtd></mtr></mtable><annotation-xml encoding="MathML-Content" id="S4.E2.m1.13b"><matrix id="S4.E2.m1.13.13.cmml" xref="S4.E2.m1.13.13"><matrixrow id="S4.E2.m1.13.13a.cmml" xref="S4.E2.m1.13.13"><apply id="S4.E2.m1.13.13.13.13.13.13.1.3.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.2"><csymbol cd="ambiguous" id="S4.E2.m1.13.13.13.13.13.13.1.3a.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.2.3">formulae-sequence</csymbol><apply id="S4.E2.m1.13.13.13.13.13.13.1.1.1.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.1.1"><eq id="S4.E2.m1.13.13.13.13.13.13.1.1.1.1.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.1.1.1"></eq><apply id="S4.E2.m1.13.13.13.13.13.13.1.1.1.2.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.1.1.2"><csymbol cd="ambiguous" id="S4.E2.m1.13.13.13.13.13.13.1.1.1.2.1.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.1.1.2">superscript</csymbol><apply id="S4.E2.m1.13.13.13.13.13.13.1.1.1.2.2.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.1.1.2.2"><ci id="S4.E2.m1.13.13.13.13.13.13.1.1.1.2.2.1.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.1.1.2.2.1">^</ci><ci id="S4.E2.m1.13.13.13.13.13.13.1.1.1.2.2.2.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.1.1.2.2.2">𝒙</ci></apply><ci id="S4.E2.m1.13.13.13.13.13.13.1.1.1.2.3.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.1.1.2.3">𝑙</ci></apply><apply id="S4.E2.m1.13.13.13.13.13.13.1.1.1.3.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.1.1.3"><ci id="S4.E2.m1.13.13.13.13.13.13.1.1.1.3.1.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.1.1.3.1">⋅</ci><apply id="S4.E2.m1.13.13.13.13.13.13.1.1.1.3.2.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.1.1.3.2"><csymbol cd="ambiguous" id="S4.E2.m1.13.13.13.13.13.13.1.1.1.3.2.1.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.1.1.3.2">superscript</csymbol><ci id="S4.E2.m1.13.13.13.13.13.13.1.1.1.3.2.2.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.1.1.3.2.2">𝒆</ci><list id="S4.E2.m1.2.2.2.2.2.2.2.3.cmml" xref="S4.E2.m1.2.2.2.2.2.2.2.2"><ci id="S4.E2.m1.1.1.1.1.1.1.1.1.cmml" xref="S4.E2.m1.1.1.1.1.1.1.1.1">𝑙</ci><apply id="S4.E2.m1.2.2.2.2.2.2.2.2.1.cmml" xref="S4.E2.m1.2.2.2.2.2.2.2.2.1"><minus id="S4.E2.m1.2.2.2.2.2.2.2.2.1.1.cmml" xref="S4.E2.m1.2.2.2.2.2.2.2.2.1.1"></minus><ci id="S4.E2.m1.2.2.2.2.2.2.2.2.1.2.cmml" xref="S4.E2.m1.2.2.2.2.2.2.2.2.1.2">𝑙</ci><cn type="integer" id="S4.E2.m1.2.2.2.2.2.2.2.2.1.3.cmml" xref="S4.E2.m1.2.2.2.2.2.2.2.2.1.3">1</cn></apply></list></apply><apply id="S4.E2.m1.6.6.6.6.6.6.cmml" xref="S4.E2.m1.6.6.6.6.6.6"><divide id="S4.E2.m1.6.6.6.6.6.6.5.cmml" xref="S4.E2.m1.6.6.6.6.6.6"></divide><apply id="S4.E2.m1.3.3.3.3.3.3.1.2.cmml" xref="S4.E2.m1.3.3.3.3.3.3.1.1"><csymbol cd="latexml" id="S4.E2.m1.3.3.3.3.3.3.1.2.1.cmml" xref="S4.E2.m1.3.3.3.3.3.3.1.1.2">delimited-∥∥</csymbol><apply id="S4.E2.m1.3.3.3.3.3.3.1.1.1.cmml" xref="S4.E2.m1.3.3.3.3.3.3.1.1.1"><csymbol cd="ambiguous" id="S4.E2.m1.3.3.3.3.3.3.1.1.1.1.cmml" xref="S4.E2.m1.3.3.3.3.3.3.1.1.1">superscript</csymbol><ci id="S4.E2.m1.3.3.3.3.3.3.1.1.1.2.cmml" xref="S4.E2.m1.3.3.3.3.3.3.1.1.1.2">𝒙</ci><ci id="S4.E2.m1.3.3.3.3.3.3.1.1.1.3.cmml" xref="S4.E2.m1.3.3.3.3.3.3.1.1.1.3">𝑙</ci></apply></apply><apply id="S4.E2.m1.6.6.6.6.6.6.4.4.cmml" xref="S4.E2.m1.6.6.6.6.6.6.4.3"><csymbol cd="latexml" id="S4.E2.m1.6.6.6.6.6.6.4.4.1.cmml" xref="S4.E2.m1.6.6.6.6.6.6.4.3.2">delimited-∥∥</csymbol><apply id="S4.E2.m1.6.6.6.6.6.6.4.3.1.cmml" xref="S4.E2.m1.6.6.6.6.6.6.4.3.1"><csymbol cd="ambiguous" id="S4.E2.m1.6.6.6.6.6.6.4.3.1.1.cmml" xref="S4.E2.m1.6.6.6.6.6.6.4.3.1">superscript</csymbol><ci id="S4.E2.m1.6.6.6.6.6.6.4.3.1.2.cmml" xref="S4.E2.m1.6.6.6.6.6.6.4.3.1.2">𝒆</ci><list id="S4.E2.m1.5.5.5.5.5.5.3.2.2.3.cmml" xref="S4.E2.m1.5.5.5.5.5.5.3.2.2.2"><ci id="S4.E2.m1.4.4.4.4.4.4.2.1.1.1.cmml" xref="S4.E2.m1.4.4.4.4.4.4.2.1.1.1">𝑙</ci><apply id="S4.E2.m1.5.5.5.5.5.5.3.2.2.2.1.cmml" xref="S4.E2.m1.5.5.5.5.5.5.3.2.2.2.1"><minus id="S4.E2.m1.5.5.5.5.5.5.3.2.2.2.1.1.cmml" xref="S4.E2.m1.5.5.5.5.5.5.3.2.2.2.1.1"></minus><ci id="S4.E2.m1.5.5.5.5.5.5.3.2.2.2.1.2.cmml" xref="S4.E2.m1.5.5.5.5.5.5.3.2.2.2.1.2">𝑙</ci><cn type="integer" id="S4.E2.m1.5.5.5.5.5.5.3.2.2.2.1.3.cmml" xref="S4.E2.m1.5.5.5.5.5.5.3.2.2.2.1.3">1</cn></apply></list></apply></apply></apply></apply></apply><apply id="S4.E2.m1.13.13.13.13.13.13.1.2.2.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2"><eq id="S4.E2.m1.13.13.13.13.13.13.1.2.2.1.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.1"></eq><apply id="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.2"><csymbol cd="ambiguous" id="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.1.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.2">superscript</csymbol><apply id="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.2.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.2"><ci id="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.2.1.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.2.1">^</ci><ci id="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.2.2.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.2.2">𝒙</ci></apply><apply id="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.3.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.3"><minus id="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.3.1.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.3.1"></minus><ci id="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.3.2.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.3.2">𝑙</ci><cn type="integer" id="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.3.3.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.2.3.3">1</cn></apply></apply><apply id="S4.E2.m1.13.13.13.13.13.13.1.2.2.3.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.3"><ci id="S4.E2.m1.13.13.13.13.13.13.1.2.2.3.1.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.3.1">⋅</ci><apply id="S4.E2.m1.13.13.13.13.13.13.1.2.2.3.2.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.3.2"><csymbol cd="ambiguous" id="S4.E2.m1.13.13.13.13.13.13.1.2.2.3.2.1.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.3.2">superscript</csymbol><ci id="S4.E2.m1.13.13.13.13.13.13.1.2.2.3.2.2.cmml" xref="S4.E2.m1.13.13.13.13.13.13.1.2.2.3.2.2">𝒆</ci><list id="S4.E2.m1.8.8.8.8.8.8.2.3.cmml" xref="S4.E2.m1.8.8.8.8.8.8.2.2"><ci id="S4.E2.m1.7.7.7.7.7.7.1.1.cmml" xref="S4.E2.m1.7.7.7.7.7.7.1.1">𝑙</ci><apply id="S4.E2.m1.8.8.8.8.8.8.2.2.1.cmml" xref="S4.E2.m1.8.8.8.8.8.8.2.2.1"><minus id="S4.E2.m1.8.8.8.8.8.8.2.2.1.1.cmml" xref="S4.E2.m1.8.8.8.8.8.8.2.2.1.1"></minus><ci id="S4.E2.m1.8.8.8.8.8.8.2.2.1.2.cmml" xref="S4.E2.m1.8.8.8.8.8.8.2.2.1.2">𝑙</ci><cn type="integer" id="S4.E2.m1.8.8.8.8.8.8.2.2.1.3.cmml" xref="S4.E2.m1.8.8.8.8.8.8.2.2.1.3">1</cn></apply></list></apply><apply id="S4.E2.m1.12.12.12.12.12.12.cmml" xref="S4.E2.m1.12.12.12.12.12.12"><divide id="S4.E2.m1.12.12.12.12.12.12.5.cmml" xref="S4.E2.m1.12.12.12.12.12.12"></divide><apply id="S4.E2.m1.9.9.9.9.9.9.1.2.cmml" xref="S4.E2.m1.9.9.9.9.9.9.1.1"><csymbol cd="latexml" id="S4.E2.m1.9.9.9.9.9.9.1.2.1.cmml" xref="S4.E2.m1.9.9.9.9.9.9.1.1.2">delimited-∥∥</csymbol><apply id="S4.E2.m1.9.9.9.9.9.9.1.1.1.cmml" xref="S4.E2.m1.9.9.9.9.9.9.1.1.1"><csymbol cd="ambiguous" id="S4.E2.m1.9.9.9.9.9.9.1.1.1.1.cmml" xref="S4.E2.m1.9.9.9.9.9.9.1.1.1">superscript</csymbol><ci id="S4.E2.m1.9.9.9.9.9.9.1.1.1.2.cmml" xref="S4.E2.m1.9.9.9.9.9.9.1.1.1.2">𝒙</ci><apply id="S4.E2.m1.9.9.9.9.9.9.1.1.1.3.cmml" xref="S4.E2.m1.9.9.9.9.9.9.1.1.1.3"><minus id="S4.E2.m1.9.9.9.9.9.9.1.1.1.3.1.cmml" xref="S4.E2.m1.9.9.9.9.9.9.1.1.1.3.1"></minus><ci id="S4.E2.m1.9.9.9.9.9.9.1.1.1.3.2.cmml" xref="S4.E2.m1.9.9.9.9.9.9.1.1.1.3.2">𝑙</ci><cn type="integer" id="S4.E2.m1.9.9.9.9.9.9.1.1.1.3.3.cmml" xref="S4.E2.m1.9.9.9.9.9.9.1.1.1.3.3">1</cn></apply></apply></apply><apply id="S4.E2.m1.12.12.12.12.12.12.4.4.cmml" xref="S4.E2.m1.12.12.12.12.12.12.4.3"><csymbol cd="latexml" id="S4.E2.m1.12.12.12.12.12.12.4.4.1.cmml" xref="S4.E2.m1.12.12.12.12.12.12.4.3.2">delimited-∥∥</csymbol><apply id="S4.E2.m1.12.12.12.12.12.12.4.3.1.cmml" xref="S4.E2.m1.12.12.12.12.12.12.4.3.1"><csymbol cd="ambiguous" id="S4.E2.m1.12.12.12.12.12.12.4.3.1.1.cmml" xref="S4.E2.m1.12.12.12.12.12.12.4.3.1">superscript</csymbol><ci id="S4.E2.m1.12.12.12.12.12.12.4.3.1.2.cmml" xref="S4.E2.m1.12.12.12.12.12.12.4.3.1.2">𝒆</ci><list id="S4.E2.m1.11.11.11.11.11.11.3.2.2.3.cmml" xref="S4.E2.m1.11.11.11.11.11.11.3.2.2.2"><ci id="S4.E2.m1.10.10.10.10.10.10.2.1.1.1.cmml" xref="S4.E2.m1.10.10.10.10.10.10.2.1.1.1">𝑙</ci><apply id="S4.E2.m1.11.11.11.11.11.11.3.2.2.2.1.cmml" xref="S4.E2.m1.11.11.11.11.11.11.3.2.2.2.1"><minus id="S4.E2.m1.11.11.11.11.11.11.3.2.2.2.1.1.cmml" xref="S4.E2.m1.11.11.11.11.11.11.3.2.2.2.1.1"></minus><ci id="S4.E2.m1.11.11.11.11.11.11.3.2.2.2.1.2.cmml" xref="S4.E2.m1.11.11.11.11.11.11.3.2.2.2.1.2">𝑙</ci><cn type="integer" id="S4.E2.m1.11.11.11.11.11.11.3.2.2.2.1.3.cmml" xref="S4.E2.m1.11.11.11.11.11.11.3.2.2.2.1.3">1</cn></apply></list></apply></apply></apply></apply></apply></apply><cerror id="S4.E2.m1.13.13b.cmml" xref="S4.E2.m1.13.13"><csymbol cd="ambiguous" id="S4.E2.m1.13.13c.cmml" xref="S4.E2.m1.13.13">missing-subexpression</csymbol></cerror></matrixrow></matrix></annotation-xml><annotation encoding="application/x-tex" id="S4.E2.m1.13c">\begin{array}[]{ll}\hat{{\bm{x}}}^{l}={\bm{e}}^{l,l-1}\cdot\frac{\lVert{\bm{x}}^{l}\rVert}{\lVert{\bm{e}}^{l,l-1}\rVert},\ \hat{{\bm{x}}}^{l-1}={\bm{e}}^{l,l-1}\cdot\frac{\lVert{\bm{x}}^{l-1}\rVert}{\lVert{\bm{e}}^{l,l-1}\rVert},\end{array}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
<p id="S4.SS2.p1.12" class="ltx_p">where <math id="S4.SS2.p1.7.m1.1" class="ltx_Math" alttext="{\bm{e}}" display="inline"><semantics id="S4.SS2.p1.7.m1.1a"><mi id="S4.SS2.p1.7.m1.1.1" xref="S4.SS2.p1.7.m1.1.1.cmml">𝒆</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.7.m1.1b"><ci id="S4.SS2.p1.7.m1.1.1.cmml" xref="S4.SS2.p1.7.m1.1.1">𝒆</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.7.m1.1c">{\bm{e}}</annotation></semantics></math> is the directional vector. This decomposition ensures that <math id="S4.SS2.p1.8.m2.5" class="ltx_Math" alttext="\frac{{\bm{e}}^{l,l-1}}{\lVert{\bm{e}}^{l,l-1}\rVert}" display="inline"><semantics id="S4.SS2.p1.8.m2.5a"><mfrac id="S4.SS2.p1.8.m2.5.5" xref="S4.SS2.p1.8.m2.5.5.cmml"><msup id="S4.SS2.p1.8.m2.2.2.2" xref="S4.SS2.p1.8.m2.2.2.2.cmml"><mi id="S4.SS2.p1.8.m2.2.2.2.4" xref="S4.SS2.p1.8.m2.2.2.2.4.cmml">𝒆</mi><mrow id="S4.SS2.p1.8.m2.2.2.2.2.2.2" xref="S4.SS2.p1.8.m2.2.2.2.2.2.3.cmml"><mi id="S4.SS2.p1.8.m2.1.1.1.1.1.1" xref="S4.SS2.p1.8.m2.1.1.1.1.1.1.cmml">l</mi><mo id="S4.SS2.p1.8.m2.2.2.2.2.2.2.2" xref="S4.SS2.p1.8.m2.2.2.2.2.2.3.cmml">,</mo><mrow id="S4.SS2.p1.8.m2.2.2.2.2.2.2.1" xref="S4.SS2.p1.8.m2.2.2.2.2.2.2.1.cmml"><mi id="S4.SS2.p1.8.m2.2.2.2.2.2.2.1.2" xref="S4.SS2.p1.8.m2.2.2.2.2.2.2.1.2.cmml">l</mi><mo id="S4.SS2.p1.8.m2.2.2.2.2.2.2.1.1" xref="S4.SS2.p1.8.m2.2.2.2.2.2.2.1.1.cmml">−</mo><mn id="S4.SS2.p1.8.m2.2.2.2.2.2.2.1.3" xref="S4.SS2.p1.8.m2.2.2.2.2.2.2.1.3.cmml">1</mn></mrow></mrow></msup><mrow id="S4.SS2.p1.8.m2.5.5.5.3" xref="S4.SS2.p1.8.m2.5.5.5.4.cmml"><mo fence="true" rspace="0em" id="S4.SS2.p1.8.m2.5.5.5.3.2" xref="S4.SS2.p1.8.m2.5.5.5.4.1.cmml">∥</mo><msup id="S4.SS2.p1.8.m2.5.5.5.3.1" xref="S4.SS2.p1.8.m2.5.5.5.3.1.cmml"><mi id="S4.SS2.p1.8.m2.5.5.5.3.1.2" xref="S4.SS2.p1.8.m2.5.5.5.3.1.2.cmml">𝒆</mi><mrow id="S4.SS2.p1.8.m2.4.4.4.2.2.2" xref="S4.SS2.p1.8.m2.4.4.4.2.2.3.cmml"><mi id="S4.SS2.p1.8.m2.3.3.3.1.1.1" xref="S4.SS2.p1.8.m2.3.3.3.1.1.1.cmml">l</mi><mo id="S4.SS2.p1.8.m2.4.4.4.2.2.2.2" xref="S4.SS2.p1.8.m2.4.4.4.2.2.3.cmml">,</mo><mrow id="S4.SS2.p1.8.m2.4.4.4.2.2.2.1" xref="S4.SS2.p1.8.m2.4.4.4.2.2.2.1.cmml"><mi id="S4.SS2.p1.8.m2.4.4.4.2.2.2.1.2" xref="S4.SS2.p1.8.m2.4.4.4.2.2.2.1.2.cmml">l</mi><mo id="S4.SS2.p1.8.m2.4.4.4.2.2.2.1.1" xref="S4.SS2.p1.8.m2.4.4.4.2.2.2.1.1.cmml">−</mo><mn id="S4.SS2.p1.8.m2.4.4.4.2.2.2.1.3" xref="S4.SS2.p1.8.m2.4.4.4.2.2.2.1.3.cmml">1</mn></mrow></mrow></msup><mo fence="true" lspace="0em" id="S4.SS2.p1.8.m2.5.5.5.3.3" xref="S4.SS2.p1.8.m2.5.5.5.4.1.cmml">∥</mo></mrow></mfrac><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.8.m2.5b"><apply id="S4.SS2.p1.8.m2.5.5.cmml" xref="S4.SS2.p1.8.m2.5.5"><divide id="S4.SS2.p1.8.m2.5.5.6.cmml" xref="S4.SS2.p1.8.m2.5.5"></divide><apply id="S4.SS2.p1.8.m2.2.2.2.cmml" xref="S4.SS2.p1.8.m2.2.2.2"><csymbol cd="ambiguous" id="S4.SS2.p1.8.m2.2.2.2.3.cmml" xref="S4.SS2.p1.8.m2.2.2.2">superscript</csymbol><ci id="S4.SS2.p1.8.m2.2.2.2.4.cmml" xref="S4.SS2.p1.8.m2.2.2.2.4">𝒆</ci><list id="S4.SS2.p1.8.m2.2.2.2.2.2.3.cmml" xref="S4.SS2.p1.8.m2.2.2.2.2.2.2"><ci id="S4.SS2.p1.8.m2.1.1.1.1.1.1.cmml" xref="S4.SS2.p1.8.m2.1.1.1.1.1.1">𝑙</ci><apply id="S4.SS2.p1.8.m2.2.2.2.2.2.2.1.cmml" xref="S4.SS2.p1.8.m2.2.2.2.2.2.2.1"><minus id="S4.SS2.p1.8.m2.2.2.2.2.2.2.1.1.cmml" xref="S4.SS2.p1.8.m2.2.2.2.2.2.2.1.1"></minus><ci id="S4.SS2.p1.8.m2.2.2.2.2.2.2.1.2.cmml" xref="S4.SS2.p1.8.m2.2.2.2.2.2.2.1.2">𝑙</ci><cn type="integer" id="S4.SS2.p1.8.m2.2.2.2.2.2.2.1.3.cmml" xref="S4.SS2.p1.8.m2.2.2.2.2.2.2.1.3">1</cn></apply></list></apply><apply id="S4.SS2.p1.8.m2.5.5.5.4.cmml" xref="S4.SS2.p1.8.m2.5.5.5.3"><csymbol cd="latexml" id="S4.SS2.p1.8.m2.5.5.5.4.1.cmml" xref="S4.SS2.p1.8.m2.5.5.5.3.2">delimited-∥∥</csymbol><apply id="S4.SS2.p1.8.m2.5.5.5.3.1.cmml" xref="S4.SS2.p1.8.m2.5.5.5.3.1"><csymbol cd="ambiguous" id="S4.SS2.p1.8.m2.5.5.5.3.1.1.cmml" xref="S4.SS2.p1.8.m2.5.5.5.3.1">superscript</csymbol><ci id="S4.SS2.p1.8.m2.5.5.5.3.1.2.cmml" xref="S4.SS2.p1.8.m2.5.5.5.3.1.2">𝒆</ci><list id="S4.SS2.p1.8.m2.4.4.4.2.2.3.cmml" xref="S4.SS2.p1.8.m2.4.4.4.2.2.2"><ci id="S4.SS2.p1.8.m2.3.3.3.1.1.1.cmml" xref="S4.SS2.p1.8.m2.3.3.3.1.1.1">𝑙</ci><apply id="S4.SS2.p1.8.m2.4.4.4.2.2.2.1.cmml" xref="S4.SS2.p1.8.m2.4.4.4.2.2.2.1"><minus id="S4.SS2.p1.8.m2.4.4.4.2.2.2.1.1.cmml" xref="S4.SS2.p1.8.m2.4.4.4.2.2.2.1.1"></minus><ci id="S4.SS2.p1.8.m2.4.4.4.2.2.2.1.2.cmml" xref="S4.SS2.p1.8.m2.4.4.4.2.2.2.1.2">𝑙</ci><cn type="integer" id="S4.SS2.p1.8.m2.4.4.4.2.2.2.1.3.cmml" xref="S4.SS2.p1.8.m2.4.4.4.2.2.2.1.3">1</cn></apply></list></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.8.m2.5c">\frac{{\bm{e}}^{l,l-1}}{\lVert{\bm{e}}^{l,l-1}\rVert}</annotation></semantics></math> is a unit vector, and allows the restored states to match the <math id="S4.SS2.p1.9.m3.1" class="ltx_Math" alttext="\ell_{2}" display="inline"><semantics id="S4.SS2.p1.9.m3.1a"><msub id="S4.SS2.p1.9.m3.1.1" xref="S4.SS2.p1.9.m3.1.1.cmml"><mi mathvariant="normal" id="S4.SS2.p1.9.m3.1.1.2" xref="S4.SS2.p1.9.m3.1.1.2.cmml">ℓ</mi><mn id="S4.SS2.p1.9.m3.1.1.3" xref="S4.SS2.p1.9.m3.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.9.m3.1b"><apply id="S4.SS2.p1.9.m3.1.1.cmml" xref="S4.SS2.p1.9.m3.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.9.m3.1.1.1.cmml" xref="S4.SS2.p1.9.m3.1.1">subscript</csymbol><ci id="S4.SS2.p1.9.m3.1.1.2.cmml" xref="S4.SS2.p1.9.m3.1.1.2">ℓ</ci><cn type="integer" id="S4.SS2.p1.9.m3.1.1.3.cmml" xref="S4.SS2.p1.9.m3.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.9.m3.1c">\ell_{2}</annotation></semantics></math> norm of the original states, thereby preserving the cache’s information as much as possible, the restoration is shown as <a href="#S4.F3" title="In 4.1 Cross-Layer Compression ‣ 4 Method ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">3</span></a>(b).
For brevity, we omit the subscripts <math id="S4.SS2.p1.10.m4.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S4.SS2.p1.10.m4.1a"><mi id="S4.SS2.p1.10.m4.1.1" xref="S4.SS2.p1.10.m4.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.10.m4.1b"><ci id="S4.SS2.p1.10.m4.1.1.cmml" xref="S4.SS2.p1.10.m4.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.10.m4.1c">k</annotation></semantics></math> and <math id="S4.SS2.p1.11.m5.1" class="ltx_Math" alttext="v" display="inline"><semantics id="S4.SS2.p1.11.m5.1a"><mi id="S4.SS2.p1.11.m5.1.1" xref="S4.SS2.p1.11.m5.1.1.cmml">v</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.11.m5.1b"><ci id="S4.SS2.p1.11.m5.1.1.cmml" xref="S4.SS2.p1.11.m5.1.1">𝑣</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.11.m5.1c">v</annotation></semantics></math>, as keys and values are decomposed in the same way.
For estimating the directional component <math id="S4.SS2.p1.12.m6.2" class="ltx_Math" alttext="{{\bm{e}}}^{l,l-1}" display="inline"><semantics id="S4.SS2.p1.12.m6.2a"><msup id="S4.SS2.p1.12.m6.2.3" xref="S4.SS2.p1.12.m6.2.3.cmml"><mi id="S4.SS2.p1.12.m6.2.3.2" xref="S4.SS2.p1.12.m6.2.3.2.cmml">𝒆</mi><mrow id="S4.SS2.p1.12.m6.2.2.2.2" xref="S4.SS2.p1.12.m6.2.2.2.3.cmml"><mi id="S4.SS2.p1.12.m6.1.1.1.1" xref="S4.SS2.p1.12.m6.1.1.1.1.cmml">l</mi><mo id="S4.SS2.p1.12.m6.2.2.2.2.2" xref="S4.SS2.p1.12.m6.2.2.2.3.cmml">,</mo><mrow id="S4.SS2.p1.12.m6.2.2.2.2.1" xref="S4.SS2.p1.12.m6.2.2.2.2.1.cmml"><mi id="S4.SS2.p1.12.m6.2.2.2.2.1.2" xref="S4.SS2.p1.12.m6.2.2.2.2.1.2.cmml">l</mi><mo id="S4.SS2.p1.12.m6.2.2.2.2.1.1" xref="S4.SS2.p1.12.m6.2.2.2.2.1.1.cmml">−</mo><mn id="S4.SS2.p1.12.m6.2.2.2.2.1.3" xref="S4.SS2.p1.12.m6.2.2.2.2.1.3.cmml">1</mn></mrow></mrow></msup><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.12.m6.2b"><apply id="S4.SS2.p1.12.m6.2.3.cmml" xref="S4.SS2.p1.12.m6.2.3"><csymbol cd="ambiguous" id="S4.SS2.p1.12.m6.2.3.1.cmml" xref="S4.SS2.p1.12.m6.2.3">superscript</csymbol><ci id="S4.SS2.p1.12.m6.2.3.2.cmml" xref="S4.SS2.p1.12.m6.2.3.2">𝒆</ci><list id="S4.SS2.p1.12.m6.2.2.2.3.cmml" xref="S4.SS2.p1.12.m6.2.2.2.2"><ci id="S4.SS2.p1.12.m6.1.1.1.1.cmml" xref="S4.SS2.p1.12.m6.1.1.1.1">𝑙</ci><apply id="S4.SS2.p1.12.m6.2.2.2.2.1.cmml" xref="S4.SS2.p1.12.m6.2.2.2.2.1"><minus id="S4.SS2.p1.12.m6.2.2.2.2.1.1.cmml" xref="S4.SS2.p1.12.m6.2.2.2.2.1.1"></minus><ci id="S4.SS2.p1.12.m6.2.2.2.2.1.2.cmml" xref="S4.SS2.p1.12.m6.2.2.2.2.1.2">𝑙</ci><cn type="integer" id="S4.SS2.p1.12.m6.2.2.2.2.1.3.cmml" xref="S4.SS2.p1.12.m6.2.2.2.2.1.3">1</cn></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.12.m6.2c">{{\bm{e}}}^{l,l-1}</annotation></semantics></math>, we follow SLERP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib64" title="" class="ltx_ref">64</a>]</cite>, which adaptively handles the interpolation, which often resembles rotation-like transformations.
The choice of SLERP as the merging function is strategic, as it facilitates interpolation along the shortest path on the unit sphere between two high-dimensional vectors, thereby preserving their geometric integrity, which refers to merge operation in <a href="#S4.F3" title="In 4.1 Cross-Layer Compression ‣ 4 Method ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">3</span></a>(a). This is crucial for maintaining the semantic and syntactic properties of the KV caches. The formula for SLERP in our context is:</p>
<table id="S4.E3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.E3.m1.21" class="ltx_Math" alttext="\begin{array}[]{ll}\begin{aligned} {{\bm{e}}}^{l,l-1}=\frac{\sin((1-t)\Omega^{l,l-1})}{\sin(\Omega^{l,l-1})}\cdot\frac{{\bm{x}}^{l-1}}{\lVert{\bm{x}}^{l-1}\rVert}+\frac{\sin(t\Omega^{l,l-1})}{\sin(\Omega^{l,l-1})}\cdot\frac{{\bm{x}}^{l}}{\lVert{\bm{x}}^{l}\rVert},\end{aligned}\end{array}" display="block"><semantics id="S4.E3.m1.21a"><mtable columnspacing="5pt" displaystyle="true" id="S4.E3.m1.21.21" xref="S4.E3.m1.21.21.cmml"><mtr id="S4.E3.m1.21.21a" xref="S4.E3.m1.21.21.cmml"><mtd class="ltx_align_left" columnalign="left" id="S4.E3.m1.21.21b" xref="S4.E3.m1.21.21.cmml"><mtable displaystyle="true" id="S4.E3.m1.21.21.21.21.21" xref="S4.E3.m1.21.21.21.21.21.cmml"><mtr id="S4.E3.m1.21.21.21.21.21a" xref="S4.E3.m1.21.21.21.21.21.cmml"><mtd class="ltx_align_right" columnalign="right" id="S4.E3.m1.21.21.21.21.21b" xref="S4.E3.m1.21.21.21.21.21.cmml"><mrow id="S4.E3.m1.21.21.21.21.21.21.21.21.21.21" xref="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.cmml"><mrow id="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1" xref="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.cmml"><msup id="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.2" xref="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.2.cmml"><mi id="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.2.2" xref="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.2.2.cmml">𝒆</mi><mrow id="S4.E3.m1.2.2.2.2.2.2.2.2.2.2.2.2" xref="S4.E3.m1.2.2.2.2.2.2.2.2.2.2.2.3.cmml"><mi id="S4.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S4.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">l</mi><mo id="S4.E3.m1.2.2.2.2.2.2.2.2.2.2.2.2.2" xref="S4.E3.m1.2.2.2.2.2.2.2.2.2.2.2.3.cmml">,</mo><mrow id="S4.E3.m1.2.2.2.2.2.2.2.2.2.2.2.2.1" xref="S4.E3.m1.2.2.2.2.2.2.2.2.2.2.2.2.1.cmml"><mi id="S4.E3.m1.2.2.2.2.2.2.2.2.2.2.2.2.1.2" xref="S4.E3.m1.2.2.2.2.2.2.2.2.2.2.2.2.1.2.cmml">l</mi><mo id="S4.E3.m1.2.2.2.2.2.2.2.2.2.2.2.2.1.1" xref="S4.E3.m1.2.2.2.2.2.2.2.2.2.2.2.2.1.1.cmml">−</mo><mn id="S4.E3.m1.2.2.2.2.2.2.2.2.2.2.2.2.1.3" xref="S4.E3.m1.2.2.2.2.2.2.2.2.2.2.2.2.1.3.cmml">1</mn></mrow></mrow></msup><mo id="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.1" xref="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.1.cmml">=</mo><mrow id="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.3" xref="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.3.cmml"><mrow id="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.3.2" xref="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.3.2.cmml"><mfrac id="S4.E3.m1.10.10.10.10.10.10.10.10.10.10" xref="S4.E3.m1.10.10.10.10.10.10.10.10.10.10.cmml"><mrow id="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4" xref="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.5.cmml"><mi id="S4.E3.m1.5.5.5.5.5.5.5.5.5.5.3.3" xref="S4.E3.m1.5.5.5.5.5.5.5.5.5.5.3.3.cmml">sin</mi><mo id="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4a" xref="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.5.cmml">⁡</mo><mrow id="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1" xref="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.5.cmml"><mo stretchy="false" id="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.2" xref="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.5.cmml">(</mo><mrow id="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1" xref="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.cmml"><mrow id="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.1.1" xref="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.1.1.2" xref="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.1.1.1.cmml">(</mo><mrow id="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.1.1.1" xref="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.1.1.1.cmml"><mn id="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.1.1.1.2" xref="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.1.1.1.2.cmml">1</mn><mo id="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.1.1.1.1" xref="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.1.1.1.1.cmml">−</mo><mi id="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.1.1.1.3" xref="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.1.1.1.3.cmml">t</mi></mrow><mo stretchy="false" id="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.1.1.3" xref="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.2" xref="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.2.cmml">​</mo><msup id="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.3" xref="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.3.cmml"><mi mathvariant="normal" id="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.3.2" xref="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.3.2.cmml">Ω</mi><mrow id="S4.E3.m1.4.4.4.4.4.4.4.4.4.4.2.2.2.2" xref="S4.E3.m1.4.4.4.4.4.4.4.4.4.4.2.2.2.3.cmml"><mi id="S4.E3.m1.3.3.3.3.3.3.3.3.3.3.1.1.1.1" xref="S4.E3.m1.3.3.3.3.3.3.3.3.3.3.1.1.1.1.cmml">l</mi><mo id="S4.E3.m1.4.4.4.4.4.4.4.4.4.4.2.2.2.2.2" xref="S4.E3.m1.4.4.4.4.4.4.4.4.4.4.2.2.2.3.cmml">,</mo><mrow id="S4.E3.m1.4.4.4.4.4.4.4.4.4.4.2.2.2.2.1" xref="S4.E3.m1.4.4.4.4.4.4.4.4.4.4.2.2.2.2.1.cmml"><mi id="S4.E3.m1.4.4.4.4.4.4.4.4.4.4.2.2.2.2.1.2" xref="S4.E3.m1.4.4.4.4.4.4.4.4.4.4.2.2.2.2.1.2.cmml">l</mi><mo id="S4.E3.m1.4.4.4.4.4.4.4.4.4.4.2.2.2.2.1.1" xref="S4.E3.m1.4.4.4.4.4.4.4.4.4.4.2.2.2.2.1.1.cmml">−</mo><mn id="S4.E3.m1.4.4.4.4.4.4.4.4.4.4.2.2.2.2.1.3" xref="S4.E3.m1.4.4.4.4.4.4.4.4.4.4.2.2.2.2.1.3.cmml">1</mn></mrow></mrow></msup></mrow><mo stretchy="false" id="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.3" xref="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.5.cmml">)</mo></mrow></mrow><mrow id="S4.E3.m1.10.10.10.10.10.10.10.10.10.10.8.4" xref="S4.E3.m1.10.10.10.10.10.10.10.10.10.10.8.5.cmml"><mi id="S4.E3.m1.9.9.9.9.9.9.9.9.9.9.7.3" xref="S4.E3.m1.9.9.9.9.9.9.9.9.9.9.7.3.cmml">sin</mi><mo id="S4.E3.m1.10.10.10.10.10.10.10.10.10.10.8.4a" xref="S4.E3.m1.10.10.10.10.10.10.10.10.10.10.8.5.cmml">⁡</mo><mrow id="S4.E3.m1.10.10.10.10.10.10.10.10.10.10.8.4.1" xref="S4.E3.m1.10.10.10.10.10.10.10.10.10.10.8.5.cmml"><mo stretchy="false" id="S4.E3.m1.10.10.10.10.10.10.10.10.10.10.8.4.1.2" xref="S4.E3.m1.10.10.10.10.10.10.10.10.10.10.8.5.cmml">(</mo><msup id="S4.E3.m1.10.10.10.10.10.10.10.10.10.10.8.4.1.1" xref="S4.E3.m1.10.10.10.10.10.10.10.10.10.10.8.4.1.1.cmml"><mi mathvariant="normal" id="S4.E3.m1.10.10.10.10.10.10.10.10.10.10.8.4.1.1.2" xref="S4.E3.m1.10.10.10.10.10.10.10.10.10.10.8.4.1.1.2.cmml">Ω</mi><mrow id="S4.E3.m1.8.8.8.8.8.8.8.8.8.8.6.2.2.2" xref="S4.E3.m1.8.8.8.8.8.8.8.8.8.8.6.2.2.3.cmml"><mi id="S4.E3.m1.7.7.7.7.7.7.7.7.7.7.5.1.1.1" xref="S4.E3.m1.7.7.7.7.7.7.7.7.7.7.5.1.1.1.cmml">l</mi><mo id="S4.E3.m1.8.8.8.8.8.8.8.8.8.8.6.2.2.2.2" xref="S4.E3.m1.8.8.8.8.8.8.8.8.8.8.6.2.2.3.cmml">,</mo><mrow id="S4.E3.m1.8.8.8.8.8.8.8.8.8.8.6.2.2.2.1" xref="S4.E3.m1.8.8.8.8.8.8.8.8.8.8.6.2.2.2.1.cmml"><mi id="S4.E3.m1.8.8.8.8.8.8.8.8.8.8.6.2.2.2.1.2" xref="S4.E3.m1.8.8.8.8.8.8.8.8.8.8.6.2.2.2.1.2.cmml">l</mi><mo id="S4.E3.m1.8.8.8.8.8.8.8.8.8.8.6.2.2.2.1.1" xref="S4.E3.m1.8.8.8.8.8.8.8.8.8.8.6.2.2.2.1.1.cmml">−</mo><mn id="S4.E3.m1.8.8.8.8.8.8.8.8.8.8.6.2.2.2.1.3" xref="S4.E3.m1.8.8.8.8.8.8.8.8.8.8.6.2.2.2.1.3.cmml">1</mn></mrow></mrow></msup><mo stretchy="false" id="S4.E3.m1.10.10.10.10.10.10.10.10.10.10.8.4.1.3" xref="S4.E3.m1.10.10.10.10.10.10.10.10.10.10.8.5.cmml">)</mo></mrow></mrow></mfrac><mo lspace="0.222em" rspace="0.222em" id="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.3.2.1" xref="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.3.2.1.cmml">⋅</mo><mfrac id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.cmml"><msup id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.3" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.3.cmml"><mi id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.3.2" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.3.2.cmml">𝒙</mi><mrow id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.3.3" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.3.3.cmml"><mi id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.3.3.2" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.3.3.2.cmml">l</mi><mo id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.3.3.1" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.3.3.1.cmml">−</mo><mn id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.3.3.3" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.3.3.3.cmml">1</mn></mrow></msup><mrow id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.2.cmml"><mo fence="true" rspace="0em" id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.2" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.2.1.cmml">∥</mo><msup id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.1" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.1.cmml"><mi id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.1.2" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.1.2.cmml">𝒙</mi><mrow id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.1.3" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.1.3.cmml"><mi id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.1.3.2" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.1.3.2.cmml">l</mi><mo id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.1.3.1" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.1.3.1.cmml">−</mo><mn id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.1.3.3" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.1.3.3.cmml">1</mn></mrow></msup><mo fence="true" lspace="0em" id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.3" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.2.1.cmml">∥</mo></mrow></mfrac></mrow><mo id="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.3.1" xref="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.3.1.cmml">+</mo><mrow id="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.3.3" xref="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.3.3.cmml"><mfrac id="S4.E3.m1.19.19.19.19.19.19.19.19.19.19" xref="S4.E3.m1.19.19.19.19.19.19.19.19.19.19.cmml"><mrow id="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.4" xref="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.5.cmml"><mi id="S4.E3.m1.14.14.14.14.14.14.14.14.14.14.3.3" xref="S4.E3.m1.14.14.14.14.14.14.14.14.14.14.3.3.cmml">sin</mi><mo id="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.4a" xref="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.5.cmml">⁡</mo><mrow id="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.4.1" xref="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.5.cmml"><mo stretchy="false" id="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.4.1.2" xref="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.5.cmml">(</mo><mrow id="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.4.1.1" xref="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.4.1.1.cmml"><mi id="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.4.1.1.2" xref="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.4.1.1.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.4.1.1.1" xref="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.4.1.1.1.cmml">​</mo><msup id="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.4.1.1.3" xref="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.4.1.1.3.cmml"><mi mathvariant="normal" id="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.4.1.1.3.2" xref="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.4.1.1.3.2.cmml">Ω</mi><mrow id="S4.E3.m1.13.13.13.13.13.13.13.13.13.13.2.2.2.2" xref="S4.E3.m1.13.13.13.13.13.13.13.13.13.13.2.2.2.3.cmml"><mi id="S4.E3.m1.12.12.12.12.12.12.12.12.12.12.1.1.1.1" xref="S4.E3.m1.12.12.12.12.12.12.12.12.12.12.1.1.1.1.cmml">l</mi><mo id="S4.E3.m1.13.13.13.13.13.13.13.13.13.13.2.2.2.2.2" xref="S4.E3.m1.13.13.13.13.13.13.13.13.13.13.2.2.2.3.cmml">,</mo><mrow id="S4.E3.m1.13.13.13.13.13.13.13.13.13.13.2.2.2.2.1" xref="S4.E3.m1.13.13.13.13.13.13.13.13.13.13.2.2.2.2.1.cmml"><mi id="S4.E3.m1.13.13.13.13.13.13.13.13.13.13.2.2.2.2.1.2" xref="S4.E3.m1.13.13.13.13.13.13.13.13.13.13.2.2.2.2.1.2.cmml">l</mi><mo id="S4.E3.m1.13.13.13.13.13.13.13.13.13.13.2.2.2.2.1.1" xref="S4.E3.m1.13.13.13.13.13.13.13.13.13.13.2.2.2.2.1.1.cmml">−</mo><mn id="S4.E3.m1.13.13.13.13.13.13.13.13.13.13.2.2.2.2.1.3" xref="S4.E3.m1.13.13.13.13.13.13.13.13.13.13.2.2.2.2.1.3.cmml">1</mn></mrow></mrow></msup></mrow><mo stretchy="false" id="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.4.1.3" xref="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.5.cmml">)</mo></mrow></mrow><mrow id="S4.E3.m1.19.19.19.19.19.19.19.19.19.19.8.4" xref="S4.E3.m1.19.19.19.19.19.19.19.19.19.19.8.5.cmml"><mi id="S4.E3.m1.18.18.18.18.18.18.18.18.18.18.7.3" xref="S4.E3.m1.18.18.18.18.18.18.18.18.18.18.7.3.cmml">sin</mi><mo id="S4.E3.m1.19.19.19.19.19.19.19.19.19.19.8.4a" xref="S4.E3.m1.19.19.19.19.19.19.19.19.19.19.8.5.cmml">⁡</mo><mrow id="S4.E3.m1.19.19.19.19.19.19.19.19.19.19.8.4.1" xref="S4.E3.m1.19.19.19.19.19.19.19.19.19.19.8.5.cmml"><mo stretchy="false" id="S4.E3.m1.19.19.19.19.19.19.19.19.19.19.8.4.1.2" xref="S4.E3.m1.19.19.19.19.19.19.19.19.19.19.8.5.cmml">(</mo><msup id="S4.E3.m1.19.19.19.19.19.19.19.19.19.19.8.4.1.1" xref="S4.E3.m1.19.19.19.19.19.19.19.19.19.19.8.4.1.1.cmml"><mi mathvariant="normal" id="S4.E3.m1.19.19.19.19.19.19.19.19.19.19.8.4.1.1.2" xref="S4.E3.m1.19.19.19.19.19.19.19.19.19.19.8.4.1.1.2.cmml">Ω</mi><mrow id="S4.E3.m1.17.17.17.17.17.17.17.17.17.17.6.2.2.2" xref="S4.E3.m1.17.17.17.17.17.17.17.17.17.17.6.2.2.3.cmml"><mi id="S4.E3.m1.16.16.16.16.16.16.16.16.16.16.5.1.1.1" xref="S4.E3.m1.16.16.16.16.16.16.16.16.16.16.5.1.1.1.cmml">l</mi><mo id="S4.E3.m1.17.17.17.17.17.17.17.17.17.17.6.2.2.2.2" xref="S4.E3.m1.17.17.17.17.17.17.17.17.17.17.6.2.2.3.cmml">,</mo><mrow id="S4.E3.m1.17.17.17.17.17.17.17.17.17.17.6.2.2.2.1" xref="S4.E3.m1.17.17.17.17.17.17.17.17.17.17.6.2.2.2.1.cmml"><mi id="S4.E3.m1.17.17.17.17.17.17.17.17.17.17.6.2.2.2.1.2" xref="S4.E3.m1.17.17.17.17.17.17.17.17.17.17.6.2.2.2.1.2.cmml">l</mi><mo id="S4.E3.m1.17.17.17.17.17.17.17.17.17.17.6.2.2.2.1.1" xref="S4.E3.m1.17.17.17.17.17.17.17.17.17.17.6.2.2.2.1.1.cmml">−</mo><mn id="S4.E3.m1.17.17.17.17.17.17.17.17.17.17.6.2.2.2.1.3" xref="S4.E3.m1.17.17.17.17.17.17.17.17.17.17.6.2.2.2.1.3.cmml">1</mn></mrow></mrow></msup><mo stretchy="false" id="S4.E3.m1.19.19.19.19.19.19.19.19.19.19.8.4.1.3" xref="S4.E3.m1.19.19.19.19.19.19.19.19.19.19.8.5.cmml">)</mo></mrow></mrow></mfrac><mo lspace="0.222em" rspace="0.222em" id="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.3.3.1" xref="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.3.3.1.cmml">⋅</mo><mfrac id="S4.E3.m1.20.20.20.20.20.20.20.20.20.20" xref="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.cmml"><msup id="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.3" xref="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.3.cmml"><mi id="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.3.2" xref="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.3.2.cmml">𝒙</mi><mi id="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.3.3" xref="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.3.3.cmml">l</mi></msup><mrow id="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.1.1" xref="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.1.2.cmml"><mo fence="true" rspace="0em" id="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.1.1.2" xref="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.1.2.1.cmml">∥</mo><msup id="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.1.1.1" xref="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.1.1.1.cmml"><mi id="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.1.1.1.2" xref="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.1.1.1.2.cmml">𝒙</mi><mi id="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.1.1.1.3" xref="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.1.1.1.3.cmml">l</mi></msup><mo fence="true" lspace="0em" id="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.1.1.3" xref="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.1.2.1.cmml">∥</mo></mrow></mfrac></mrow></mrow></mrow><mo id="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.2" xref="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.cmml">,</mo></mrow></mtd></mtr></mtable></mtd><mtd id="S4.E3.m1.21.21c" xref="S4.E3.m1.21.21.cmml"></mtd></mtr></mtable><annotation-xml encoding="MathML-Content" id="S4.E3.m1.21b"><matrix id="S4.E3.m1.21.21.cmml" xref="S4.E3.m1.21.21"><matrixrow id="S4.E3.m1.21.21a.cmml" xref="S4.E3.m1.21.21"><matrix id="S4.E3.m1.21.21.21.21.21.cmml" xref="S4.E3.m1.21.21.21.21.21"><matrixrow id="S4.E3.m1.21.21.21.21.21a.cmml" xref="S4.E3.m1.21.21.21.21.21"><apply id="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.cmml" xref="S4.E3.m1.21.21.21.21.21.21.21.21.21.21"><eq id="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.1.cmml" xref="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.1"></eq><apply id="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.2.cmml" xref="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.2"><csymbol cd="ambiguous" id="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.2.1.cmml" xref="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.2">superscript</csymbol><ci id="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.2.2.cmml" xref="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.2.2">𝒆</ci><list id="S4.E3.m1.2.2.2.2.2.2.2.2.2.2.2.3.cmml" xref="S4.E3.m1.2.2.2.2.2.2.2.2.2.2.2.2"><ci id="S4.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1">𝑙</ci><apply id="S4.E3.m1.2.2.2.2.2.2.2.2.2.2.2.2.1.cmml" xref="S4.E3.m1.2.2.2.2.2.2.2.2.2.2.2.2.1"><minus id="S4.E3.m1.2.2.2.2.2.2.2.2.2.2.2.2.1.1.cmml" xref="S4.E3.m1.2.2.2.2.2.2.2.2.2.2.2.2.1.1"></minus><ci id="S4.E3.m1.2.2.2.2.2.2.2.2.2.2.2.2.1.2.cmml" xref="S4.E3.m1.2.2.2.2.2.2.2.2.2.2.2.2.1.2">𝑙</ci><cn type="integer" id="S4.E3.m1.2.2.2.2.2.2.2.2.2.2.2.2.1.3.cmml" xref="S4.E3.m1.2.2.2.2.2.2.2.2.2.2.2.2.1.3">1</cn></apply></list></apply><apply id="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.3.cmml" xref="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.3"><plus id="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.3.1.cmml" xref="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.3.1"></plus><apply id="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.3.2.cmml" xref="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.3.2"><ci id="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.3.2.1.cmml" xref="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.3.2.1">⋅</ci><apply id="S4.E3.m1.10.10.10.10.10.10.10.10.10.10.cmml" xref="S4.E3.m1.10.10.10.10.10.10.10.10.10.10"><divide id="S4.E3.m1.10.10.10.10.10.10.10.10.10.10.9.cmml" xref="S4.E3.m1.10.10.10.10.10.10.10.10.10.10"></divide><apply id="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.5.cmml" xref="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4"><sin id="S4.E3.m1.5.5.5.5.5.5.5.5.5.5.3.3.cmml" xref="S4.E3.m1.5.5.5.5.5.5.5.5.5.5.3.3"></sin><apply id="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.cmml" xref="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1"><times id="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.2.cmml" xref="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.2"></times><apply id="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.1.1.1.cmml" xref="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.1.1"><minus id="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.1.1.1.1.cmml" xref="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.1.1.1.1"></minus><cn type="integer" id="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.1.1.1.2.cmml" xref="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.1.1.1.2">1</cn><ci id="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.1.1.1.3.cmml" xref="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.1.1.1.3">𝑡</ci></apply><apply id="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.3.cmml" xref="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.3"><csymbol cd="ambiguous" id="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.3.1.cmml" xref="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.3">superscript</csymbol><ci id="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.3.2.cmml" xref="S4.E3.m1.6.6.6.6.6.6.6.6.6.6.4.4.1.1.3.2">Ω</ci><list id="S4.E3.m1.4.4.4.4.4.4.4.4.4.4.2.2.2.3.cmml" xref="S4.E3.m1.4.4.4.4.4.4.4.4.4.4.2.2.2.2"><ci id="S4.E3.m1.3.3.3.3.3.3.3.3.3.3.1.1.1.1.cmml" xref="S4.E3.m1.3.3.3.3.3.3.3.3.3.3.1.1.1.1">𝑙</ci><apply id="S4.E3.m1.4.4.4.4.4.4.4.4.4.4.2.2.2.2.1.cmml" xref="S4.E3.m1.4.4.4.4.4.4.4.4.4.4.2.2.2.2.1"><minus id="S4.E3.m1.4.4.4.4.4.4.4.4.4.4.2.2.2.2.1.1.cmml" xref="S4.E3.m1.4.4.4.4.4.4.4.4.4.4.2.2.2.2.1.1"></minus><ci id="S4.E3.m1.4.4.4.4.4.4.4.4.4.4.2.2.2.2.1.2.cmml" xref="S4.E3.m1.4.4.4.4.4.4.4.4.4.4.2.2.2.2.1.2">𝑙</ci><cn type="integer" id="S4.E3.m1.4.4.4.4.4.4.4.4.4.4.2.2.2.2.1.3.cmml" xref="S4.E3.m1.4.4.4.4.4.4.4.4.4.4.2.2.2.2.1.3">1</cn></apply></list></apply></apply></apply><apply id="S4.E3.m1.10.10.10.10.10.10.10.10.10.10.8.5.cmml" xref="S4.E3.m1.10.10.10.10.10.10.10.10.10.10.8.4"><sin id="S4.E3.m1.9.9.9.9.9.9.9.9.9.9.7.3.cmml" xref="S4.E3.m1.9.9.9.9.9.9.9.9.9.9.7.3"></sin><apply id="S4.E3.m1.10.10.10.10.10.10.10.10.10.10.8.4.1.1.cmml" xref="S4.E3.m1.10.10.10.10.10.10.10.10.10.10.8.4.1.1"><csymbol cd="ambiguous" id="S4.E3.m1.10.10.10.10.10.10.10.10.10.10.8.4.1.1.1.cmml" xref="S4.E3.m1.10.10.10.10.10.10.10.10.10.10.8.4.1.1">superscript</csymbol><ci id="S4.E3.m1.10.10.10.10.10.10.10.10.10.10.8.4.1.1.2.cmml" xref="S4.E3.m1.10.10.10.10.10.10.10.10.10.10.8.4.1.1.2">Ω</ci><list id="S4.E3.m1.8.8.8.8.8.8.8.8.8.8.6.2.2.3.cmml" xref="S4.E3.m1.8.8.8.8.8.8.8.8.8.8.6.2.2.2"><ci id="S4.E3.m1.7.7.7.7.7.7.7.7.7.7.5.1.1.1.cmml" xref="S4.E3.m1.7.7.7.7.7.7.7.7.7.7.5.1.1.1">𝑙</ci><apply id="S4.E3.m1.8.8.8.8.8.8.8.8.8.8.6.2.2.2.1.cmml" xref="S4.E3.m1.8.8.8.8.8.8.8.8.8.8.6.2.2.2.1"><minus id="S4.E3.m1.8.8.8.8.8.8.8.8.8.8.6.2.2.2.1.1.cmml" xref="S4.E3.m1.8.8.8.8.8.8.8.8.8.8.6.2.2.2.1.1"></minus><ci id="S4.E3.m1.8.8.8.8.8.8.8.8.8.8.6.2.2.2.1.2.cmml" xref="S4.E3.m1.8.8.8.8.8.8.8.8.8.8.6.2.2.2.1.2">𝑙</ci><cn type="integer" id="S4.E3.m1.8.8.8.8.8.8.8.8.8.8.6.2.2.2.1.3.cmml" xref="S4.E3.m1.8.8.8.8.8.8.8.8.8.8.6.2.2.2.1.3">1</cn></apply></list></apply></apply></apply><apply id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.cmml" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11"><divide id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.2.cmml" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11"></divide><apply id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.3.cmml" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.3"><csymbol cd="ambiguous" id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.3.1.cmml" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.3">superscript</csymbol><ci id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.3.2.cmml" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.3.2">𝒙</ci><apply id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.3.3.cmml" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.3.3"><minus id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.3.3.1.cmml" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.3.3.1"></minus><ci id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.3.3.2.cmml" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.3.3.2">𝑙</ci><cn type="integer" id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.3.3.3.cmml" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.3.3.3">1</cn></apply></apply><apply id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.2.cmml" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1"><csymbol cd="latexml" id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.2.1.cmml" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.2">delimited-∥∥</csymbol><apply id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.1.cmml" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.1"><csymbol cd="ambiguous" id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.1.1.cmml" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.1">superscript</csymbol><ci id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.1.2.cmml" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.1.2">𝒙</ci><apply id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.1.3.cmml" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.1.3"><minus id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.1.3.1.cmml" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.1.3.1"></minus><ci id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.1.3.2.cmml" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.1.3.2">𝑙</ci><cn type="integer" id="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.1.3.3.cmml" xref="S4.E3.m1.11.11.11.11.11.11.11.11.11.11.1.1.1.3.3">1</cn></apply></apply></apply></apply></apply><apply id="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.3.3.cmml" xref="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.3.3"><ci id="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.3.3.1.cmml" xref="S4.E3.m1.21.21.21.21.21.21.21.21.21.21.1.3.3.1">⋅</ci><apply id="S4.E3.m1.19.19.19.19.19.19.19.19.19.19.cmml" xref="S4.E3.m1.19.19.19.19.19.19.19.19.19.19"><divide id="S4.E3.m1.19.19.19.19.19.19.19.19.19.19.9.cmml" xref="S4.E3.m1.19.19.19.19.19.19.19.19.19.19"></divide><apply id="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.5.cmml" xref="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.4"><sin id="S4.E3.m1.14.14.14.14.14.14.14.14.14.14.3.3.cmml" xref="S4.E3.m1.14.14.14.14.14.14.14.14.14.14.3.3"></sin><apply id="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.4.1.1.cmml" xref="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.4.1.1"><times id="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.4.1.1.1.cmml" xref="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.4.1.1.1"></times><ci id="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.4.1.1.2.cmml" xref="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.4.1.1.2">𝑡</ci><apply id="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.4.1.1.3.cmml" xref="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.4.1.1.3"><csymbol cd="ambiguous" id="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.4.1.1.3.1.cmml" xref="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.4.1.1.3">superscript</csymbol><ci id="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.4.1.1.3.2.cmml" xref="S4.E3.m1.15.15.15.15.15.15.15.15.15.15.4.4.1.1.3.2">Ω</ci><list id="S4.E3.m1.13.13.13.13.13.13.13.13.13.13.2.2.2.3.cmml" xref="S4.E3.m1.13.13.13.13.13.13.13.13.13.13.2.2.2.2"><ci id="S4.E3.m1.12.12.12.12.12.12.12.12.12.12.1.1.1.1.cmml" xref="S4.E3.m1.12.12.12.12.12.12.12.12.12.12.1.1.1.1">𝑙</ci><apply id="S4.E3.m1.13.13.13.13.13.13.13.13.13.13.2.2.2.2.1.cmml" xref="S4.E3.m1.13.13.13.13.13.13.13.13.13.13.2.2.2.2.1"><minus id="S4.E3.m1.13.13.13.13.13.13.13.13.13.13.2.2.2.2.1.1.cmml" xref="S4.E3.m1.13.13.13.13.13.13.13.13.13.13.2.2.2.2.1.1"></minus><ci id="S4.E3.m1.13.13.13.13.13.13.13.13.13.13.2.2.2.2.1.2.cmml" xref="S4.E3.m1.13.13.13.13.13.13.13.13.13.13.2.2.2.2.1.2">𝑙</ci><cn type="integer" id="S4.E3.m1.13.13.13.13.13.13.13.13.13.13.2.2.2.2.1.3.cmml" xref="S4.E3.m1.13.13.13.13.13.13.13.13.13.13.2.2.2.2.1.3">1</cn></apply></list></apply></apply></apply><apply id="S4.E3.m1.19.19.19.19.19.19.19.19.19.19.8.5.cmml" xref="S4.E3.m1.19.19.19.19.19.19.19.19.19.19.8.4"><sin id="S4.E3.m1.18.18.18.18.18.18.18.18.18.18.7.3.cmml" xref="S4.E3.m1.18.18.18.18.18.18.18.18.18.18.7.3"></sin><apply id="S4.E3.m1.19.19.19.19.19.19.19.19.19.19.8.4.1.1.cmml" xref="S4.E3.m1.19.19.19.19.19.19.19.19.19.19.8.4.1.1"><csymbol cd="ambiguous" id="S4.E3.m1.19.19.19.19.19.19.19.19.19.19.8.4.1.1.1.cmml" xref="S4.E3.m1.19.19.19.19.19.19.19.19.19.19.8.4.1.1">superscript</csymbol><ci id="S4.E3.m1.19.19.19.19.19.19.19.19.19.19.8.4.1.1.2.cmml" xref="S4.E3.m1.19.19.19.19.19.19.19.19.19.19.8.4.1.1.2">Ω</ci><list id="S4.E3.m1.17.17.17.17.17.17.17.17.17.17.6.2.2.3.cmml" xref="S4.E3.m1.17.17.17.17.17.17.17.17.17.17.6.2.2.2"><ci id="S4.E3.m1.16.16.16.16.16.16.16.16.16.16.5.1.1.1.cmml" xref="S4.E3.m1.16.16.16.16.16.16.16.16.16.16.5.1.1.1">𝑙</ci><apply id="S4.E3.m1.17.17.17.17.17.17.17.17.17.17.6.2.2.2.1.cmml" xref="S4.E3.m1.17.17.17.17.17.17.17.17.17.17.6.2.2.2.1"><minus id="S4.E3.m1.17.17.17.17.17.17.17.17.17.17.6.2.2.2.1.1.cmml" xref="S4.E3.m1.17.17.17.17.17.17.17.17.17.17.6.2.2.2.1.1"></minus><ci id="S4.E3.m1.17.17.17.17.17.17.17.17.17.17.6.2.2.2.1.2.cmml" xref="S4.E3.m1.17.17.17.17.17.17.17.17.17.17.6.2.2.2.1.2">𝑙</ci><cn type="integer" id="S4.E3.m1.17.17.17.17.17.17.17.17.17.17.6.2.2.2.1.3.cmml" xref="S4.E3.m1.17.17.17.17.17.17.17.17.17.17.6.2.2.2.1.3">1</cn></apply></list></apply></apply></apply><apply id="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.cmml" xref="S4.E3.m1.20.20.20.20.20.20.20.20.20.20"><divide id="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.2.cmml" xref="S4.E3.m1.20.20.20.20.20.20.20.20.20.20"></divide><apply id="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.3.cmml" xref="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.3"><csymbol cd="ambiguous" id="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.3.1.cmml" xref="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.3">superscript</csymbol><ci id="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.3.2.cmml" xref="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.3.2">𝒙</ci><ci id="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.3.3.cmml" xref="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.3.3">𝑙</ci></apply><apply id="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.1.2.cmml" xref="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.1.1"><csymbol cd="latexml" id="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.1.2.1.cmml" xref="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.1.1.2">delimited-∥∥</csymbol><apply id="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.1.1.1.cmml" xref="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.1.1.1"><csymbol cd="ambiguous" id="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.1.1.1.1.cmml" xref="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.1.1.1">superscript</csymbol><ci id="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.1.1.1.2.cmml" xref="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.1.1.1.2">𝒙</ci><ci id="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.1.1.1.3.cmml" xref="S4.E3.m1.20.20.20.20.20.20.20.20.20.20.1.1.1.3">𝑙</ci></apply></apply></apply></apply></apply></apply></matrixrow></matrix><cerror id="S4.E3.m1.21.21b.cmml" xref="S4.E3.m1.21.21"><csymbol cd="ambiguous" id="S4.E3.m1.21.21c.cmml" xref="S4.E3.m1.21.21">missing-subexpression</csymbol></cerror></matrixrow></matrix></annotation-xml><annotation encoding="application/x-tex" id="S4.E3.m1.21c">\begin{array}[]{ll}\begin{aligned} {{\bm{e}}}^{l,l-1}=\frac{\sin((1-t)\Omega^{l,l-1})}{\sin(\Omega^{l,l-1})}\cdot\frac{{\bm{x}}^{l-1}}{\lVert{\bm{x}}^{l-1}\rVert}+\frac{\sin(t\Omega^{l,l-1})}{\sin(\Omega^{l,l-1})}\cdot\frac{{\bm{x}}^{l}}{\lVert{\bm{x}}^{l}\rVert},\end{aligned}\end{array}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
</table>
<p id="S4.SS2.p1.21" class="ltx_p">where <math id="S4.SS2.p1.13.m1.5" class="ltx_Math" alttext="\Omega^{l,l-1}=\arccos\left(\frac{{\bm{x}}^{l}\cdot{\bm{x}}^{l-1}}{\|{\bm{x}}^{l}\|\|{\bm{x}}^{l-1}\|}\right)" display="inline"><semantics id="S4.SS2.p1.13.m1.5a"><mrow id="S4.SS2.p1.13.m1.5.6" xref="S4.SS2.p1.13.m1.5.6.cmml"><msup id="S4.SS2.p1.13.m1.5.6.2" xref="S4.SS2.p1.13.m1.5.6.2.cmml"><mi mathvariant="normal" id="S4.SS2.p1.13.m1.5.6.2.2" xref="S4.SS2.p1.13.m1.5.6.2.2.cmml">Ω</mi><mrow id="S4.SS2.p1.13.m1.2.2.2.2" xref="S4.SS2.p1.13.m1.2.2.2.3.cmml"><mi id="S4.SS2.p1.13.m1.1.1.1.1" xref="S4.SS2.p1.13.m1.1.1.1.1.cmml">l</mi><mo id="S4.SS2.p1.13.m1.2.2.2.2.2" xref="S4.SS2.p1.13.m1.2.2.2.3.cmml">,</mo><mrow id="S4.SS2.p1.13.m1.2.2.2.2.1" xref="S4.SS2.p1.13.m1.2.2.2.2.1.cmml"><mi id="S4.SS2.p1.13.m1.2.2.2.2.1.2" xref="S4.SS2.p1.13.m1.2.2.2.2.1.2.cmml">l</mi><mo id="S4.SS2.p1.13.m1.2.2.2.2.1.1" xref="S4.SS2.p1.13.m1.2.2.2.2.1.1.cmml">−</mo><mn id="S4.SS2.p1.13.m1.2.2.2.2.1.3" xref="S4.SS2.p1.13.m1.2.2.2.2.1.3.cmml">1</mn></mrow></mrow></msup><mo id="S4.SS2.p1.13.m1.5.6.1" xref="S4.SS2.p1.13.m1.5.6.1.cmml">=</mo><mrow id="S4.SS2.p1.13.m1.5.6.3.2" xref="S4.SS2.p1.13.m1.5.6.3.1.cmml"><mi id="S4.SS2.p1.13.m1.5.5" xref="S4.SS2.p1.13.m1.5.5.cmml">arccos</mi><mo id="S4.SS2.p1.13.m1.5.6.3.2a" xref="S4.SS2.p1.13.m1.5.6.3.1.cmml">⁡</mo><mrow id="S4.SS2.p1.13.m1.5.6.3.2.1" xref="S4.SS2.p1.13.m1.5.6.3.1.cmml"><mo id="S4.SS2.p1.13.m1.5.6.3.2.1.1" xref="S4.SS2.p1.13.m1.5.6.3.1.cmml">(</mo><mfrac id="S4.SS2.p1.13.m1.4.4" xref="S4.SS2.p1.13.m1.4.4.cmml"><mrow id="S4.SS2.p1.13.m1.4.4.4" xref="S4.SS2.p1.13.m1.4.4.4.cmml"><msup id="S4.SS2.p1.13.m1.4.4.4.2" xref="S4.SS2.p1.13.m1.4.4.4.2.cmml"><mi id="S4.SS2.p1.13.m1.4.4.4.2.2" xref="S4.SS2.p1.13.m1.4.4.4.2.2.cmml">𝒙</mi><mi id="S4.SS2.p1.13.m1.4.4.4.2.3" xref="S4.SS2.p1.13.m1.4.4.4.2.3.cmml">l</mi></msup><mo lspace="0.222em" rspace="0.222em" id="S4.SS2.p1.13.m1.4.4.4.1" xref="S4.SS2.p1.13.m1.4.4.4.1.cmml">⋅</mo><msup id="S4.SS2.p1.13.m1.4.4.4.3" xref="S4.SS2.p1.13.m1.4.4.4.3.cmml"><mi id="S4.SS2.p1.13.m1.4.4.4.3.2" xref="S4.SS2.p1.13.m1.4.4.4.3.2.cmml">𝒙</mi><mrow id="S4.SS2.p1.13.m1.4.4.4.3.3" xref="S4.SS2.p1.13.m1.4.4.4.3.3.cmml"><mi id="S4.SS2.p1.13.m1.4.4.4.3.3.2" xref="S4.SS2.p1.13.m1.4.4.4.3.3.2.cmml">l</mi><mo id="S4.SS2.p1.13.m1.4.4.4.3.3.1" xref="S4.SS2.p1.13.m1.4.4.4.3.3.1.cmml">−</mo><mn id="S4.SS2.p1.13.m1.4.4.4.3.3.3" xref="S4.SS2.p1.13.m1.4.4.4.3.3.3.cmml">1</mn></mrow></msup></mrow><mrow id="S4.SS2.p1.13.m1.4.4.2" xref="S4.SS2.p1.13.m1.4.4.2.cmml"><mrow id="S4.SS2.p1.13.m1.3.3.1.1.1" xref="S4.SS2.p1.13.m1.3.3.1.1.2.cmml"><mo stretchy="false" id="S4.SS2.p1.13.m1.3.3.1.1.1.2" xref="S4.SS2.p1.13.m1.3.3.1.1.2.1.cmml">‖</mo><msup id="S4.SS2.p1.13.m1.3.3.1.1.1.1" xref="S4.SS2.p1.13.m1.3.3.1.1.1.1.cmml"><mi id="S4.SS2.p1.13.m1.3.3.1.1.1.1.2" xref="S4.SS2.p1.13.m1.3.3.1.1.1.1.2.cmml">𝒙</mi><mi id="S4.SS2.p1.13.m1.3.3.1.1.1.1.3" xref="S4.SS2.p1.13.m1.3.3.1.1.1.1.3.cmml">l</mi></msup><mo stretchy="false" id="S4.SS2.p1.13.m1.3.3.1.1.1.3" xref="S4.SS2.p1.13.m1.3.3.1.1.2.1.cmml">‖</mo></mrow><mo lspace="0em" rspace="0em" id="S4.SS2.p1.13.m1.4.4.2.3" xref="S4.SS2.p1.13.m1.4.4.2.3.cmml">​</mo><mrow id="S4.SS2.p1.13.m1.4.4.2.2.1" xref="S4.SS2.p1.13.m1.4.4.2.2.2.cmml"><mo stretchy="false" id="S4.SS2.p1.13.m1.4.4.2.2.1.2" xref="S4.SS2.p1.13.m1.4.4.2.2.2.1.cmml">‖</mo><msup id="S4.SS2.p1.13.m1.4.4.2.2.1.1" xref="S4.SS2.p1.13.m1.4.4.2.2.1.1.cmml"><mi id="S4.SS2.p1.13.m1.4.4.2.2.1.1.2" xref="S4.SS2.p1.13.m1.4.4.2.2.1.1.2.cmml">𝒙</mi><mrow id="S4.SS2.p1.13.m1.4.4.2.2.1.1.3" xref="S4.SS2.p1.13.m1.4.4.2.2.1.1.3.cmml"><mi id="S4.SS2.p1.13.m1.4.4.2.2.1.1.3.2" xref="S4.SS2.p1.13.m1.4.4.2.2.1.1.3.2.cmml">l</mi><mo id="S4.SS2.p1.13.m1.4.4.2.2.1.1.3.1" xref="S4.SS2.p1.13.m1.4.4.2.2.1.1.3.1.cmml">−</mo><mn id="S4.SS2.p1.13.m1.4.4.2.2.1.1.3.3" xref="S4.SS2.p1.13.m1.4.4.2.2.1.1.3.3.cmml">1</mn></mrow></msup><mo stretchy="false" id="S4.SS2.p1.13.m1.4.4.2.2.1.3" xref="S4.SS2.p1.13.m1.4.4.2.2.2.1.cmml">‖</mo></mrow></mrow></mfrac><mo id="S4.SS2.p1.13.m1.5.6.3.2.1.2" xref="S4.SS2.p1.13.m1.5.6.3.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.13.m1.5b"><apply id="S4.SS2.p1.13.m1.5.6.cmml" xref="S4.SS2.p1.13.m1.5.6"><eq id="S4.SS2.p1.13.m1.5.6.1.cmml" xref="S4.SS2.p1.13.m1.5.6.1"></eq><apply id="S4.SS2.p1.13.m1.5.6.2.cmml" xref="S4.SS2.p1.13.m1.5.6.2"><csymbol cd="ambiguous" id="S4.SS2.p1.13.m1.5.6.2.1.cmml" xref="S4.SS2.p1.13.m1.5.6.2">superscript</csymbol><ci id="S4.SS2.p1.13.m1.5.6.2.2.cmml" xref="S4.SS2.p1.13.m1.5.6.2.2">Ω</ci><list id="S4.SS2.p1.13.m1.2.2.2.3.cmml" xref="S4.SS2.p1.13.m1.2.2.2.2"><ci id="S4.SS2.p1.13.m1.1.1.1.1.cmml" xref="S4.SS2.p1.13.m1.1.1.1.1">𝑙</ci><apply id="S4.SS2.p1.13.m1.2.2.2.2.1.cmml" xref="S4.SS2.p1.13.m1.2.2.2.2.1"><minus id="S4.SS2.p1.13.m1.2.2.2.2.1.1.cmml" xref="S4.SS2.p1.13.m1.2.2.2.2.1.1"></minus><ci id="S4.SS2.p1.13.m1.2.2.2.2.1.2.cmml" xref="S4.SS2.p1.13.m1.2.2.2.2.1.2">𝑙</ci><cn type="integer" id="S4.SS2.p1.13.m1.2.2.2.2.1.3.cmml" xref="S4.SS2.p1.13.m1.2.2.2.2.1.3">1</cn></apply></list></apply><apply id="S4.SS2.p1.13.m1.5.6.3.1.cmml" xref="S4.SS2.p1.13.m1.5.6.3.2"><arccos id="S4.SS2.p1.13.m1.5.5.cmml" xref="S4.SS2.p1.13.m1.5.5"></arccos><apply id="S4.SS2.p1.13.m1.4.4.cmml" xref="S4.SS2.p1.13.m1.4.4"><divide id="S4.SS2.p1.13.m1.4.4.3.cmml" xref="S4.SS2.p1.13.m1.4.4"></divide><apply id="S4.SS2.p1.13.m1.4.4.4.cmml" xref="S4.SS2.p1.13.m1.4.4.4"><ci id="S4.SS2.p1.13.m1.4.4.4.1.cmml" xref="S4.SS2.p1.13.m1.4.4.4.1">⋅</ci><apply id="S4.SS2.p1.13.m1.4.4.4.2.cmml" xref="S4.SS2.p1.13.m1.4.4.4.2"><csymbol cd="ambiguous" id="S4.SS2.p1.13.m1.4.4.4.2.1.cmml" xref="S4.SS2.p1.13.m1.4.4.4.2">superscript</csymbol><ci id="S4.SS2.p1.13.m1.4.4.4.2.2.cmml" xref="S4.SS2.p1.13.m1.4.4.4.2.2">𝒙</ci><ci id="S4.SS2.p1.13.m1.4.4.4.2.3.cmml" xref="S4.SS2.p1.13.m1.4.4.4.2.3">𝑙</ci></apply><apply id="S4.SS2.p1.13.m1.4.4.4.3.cmml" xref="S4.SS2.p1.13.m1.4.4.4.3"><csymbol cd="ambiguous" id="S4.SS2.p1.13.m1.4.4.4.3.1.cmml" xref="S4.SS2.p1.13.m1.4.4.4.3">superscript</csymbol><ci id="S4.SS2.p1.13.m1.4.4.4.3.2.cmml" xref="S4.SS2.p1.13.m1.4.4.4.3.2">𝒙</ci><apply id="S4.SS2.p1.13.m1.4.4.4.3.3.cmml" xref="S4.SS2.p1.13.m1.4.4.4.3.3"><minus id="S4.SS2.p1.13.m1.4.4.4.3.3.1.cmml" xref="S4.SS2.p1.13.m1.4.4.4.3.3.1"></minus><ci id="S4.SS2.p1.13.m1.4.4.4.3.3.2.cmml" xref="S4.SS2.p1.13.m1.4.4.4.3.3.2">𝑙</ci><cn type="integer" id="S4.SS2.p1.13.m1.4.4.4.3.3.3.cmml" xref="S4.SS2.p1.13.m1.4.4.4.3.3.3">1</cn></apply></apply></apply><apply id="S4.SS2.p1.13.m1.4.4.2.cmml" xref="S4.SS2.p1.13.m1.4.4.2"><times id="S4.SS2.p1.13.m1.4.4.2.3.cmml" xref="S4.SS2.p1.13.m1.4.4.2.3"></times><apply id="S4.SS2.p1.13.m1.3.3.1.1.2.cmml" xref="S4.SS2.p1.13.m1.3.3.1.1.1"><csymbol cd="latexml" id="S4.SS2.p1.13.m1.3.3.1.1.2.1.cmml" xref="S4.SS2.p1.13.m1.3.3.1.1.1.2">norm</csymbol><apply id="S4.SS2.p1.13.m1.3.3.1.1.1.1.cmml" xref="S4.SS2.p1.13.m1.3.3.1.1.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.13.m1.3.3.1.1.1.1.1.cmml" xref="S4.SS2.p1.13.m1.3.3.1.1.1.1">superscript</csymbol><ci id="S4.SS2.p1.13.m1.3.3.1.1.1.1.2.cmml" xref="S4.SS2.p1.13.m1.3.3.1.1.1.1.2">𝒙</ci><ci id="S4.SS2.p1.13.m1.3.3.1.1.1.1.3.cmml" xref="S4.SS2.p1.13.m1.3.3.1.1.1.1.3">𝑙</ci></apply></apply><apply id="S4.SS2.p1.13.m1.4.4.2.2.2.cmml" xref="S4.SS2.p1.13.m1.4.4.2.2.1"><csymbol cd="latexml" id="S4.SS2.p1.13.m1.4.4.2.2.2.1.cmml" xref="S4.SS2.p1.13.m1.4.4.2.2.1.2">norm</csymbol><apply id="S4.SS2.p1.13.m1.4.4.2.2.1.1.cmml" xref="S4.SS2.p1.13.m1.4.4.2.2.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.13.m1.4.4.2.2.1.1.1.cmml" xref="S4.SS2.p1.13.m1.4.4.2.2.1.1">superscript</csymbol><ci id="S4.SS2.p1.13.m1.4.4.2.2.1.1.2.cmml" xref="S4.SS2.p1.13.m1.4.4.2.2.1.1.2">𝒙</ci><apply id="S4.SS2.p1.13.m1.4.4.2.2.1.1.3.cmml" xref="S4.SS2.p1.13.m1.4.4.2.2.1.1.3"><minus id="S4.SS2.p1.13.m1.4.4.2.2.1.1.3.1.cmml" xref="S4.SS2.p1.13.m1.4.4.2.2.1.1.3.1"></minus><ci id="S4.SS2.p1.13.m1.4.4.2.2.1.1.3.2.cmml" xref="S4.SS2.p1.13.m1.4.4.2.2.1.1.3.2">𝑙</ci><cn type="integer" id="S4.SS2.p1.13.m1.4.4.2.2.1.1.3.3.cmml" xref="S4.SS2.p1.13.m1.4.4.2.2.1.1.3.3">1</cn></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.13.m1.5c">\Omega^{l,l-1}=\arccos\left(\frac{{\bm{x}}^{l}\cdot{\bm{x}}^{l-1}}{\|{\bm{x}}^{l}\|\|{\bm{x}}^{l-1}\|}\right)</annotation></semantics></math> represents the angle between vectors <math id="S4.SS2.p1.14.m2.1" class="ltx_Math" alttext="{\bm{x}}^{l}" display="inline"><semantics id="S4.SS2.p1.14.m2.1a"><msup id="S4.SS2.p1.14.m2.1.1" xref="S4.SS2.p1.14.m2.1.1.cmml"><mi id="S4.SS2.p1.14.m2.1.1.2" xref="S4.SS2.p1.14.m2.1.1.2.cmml">𝒙</mi><mi id="S4.SS2.p1.14.m2.1.1.3" xref="S4.SS2.p1.14.m2.1.1.3.cmml">l</mi></msup><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.14.m2.1b"><apply id="S4.SS2.p1.14.m2.1.1.cmml" xref="S4.SS2.p1.14.m2.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.14.m2.1.1.1.cmml" xref="S4.SS2.p1.14.m2.1.1">superscript</csymbol><ci id="S4.SS2.p1.14.m2.1.1.2.cmml" xref="S4.SS2.p1.14.m2.1.1.2">𝒙</ci><ci id="S4.SS2.p1.14.m2.1.1.3.cmml" xref="S4.SS2.p1.14.m2.1.1.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.14.m2.1c">{\bm{x}}^{l}</annotation></semantics></math> and <math id="S4.SS2.p1.15.m3.1" class="ltx_Math" alttext="{\bm{x}}^{l-1}" display="inline"><semantics id="S4.SS2.p1.15.m3.1a"><msup id="S4.SS2.p1.15.m3.1.1" xref="S4.SS2.p1.15.m3.1.1.cmml"><mi id="S4.SS2.p1.15.m3.1.1.2" xref="S4.SS2.p1.15.m3.1.1.2.cmml">𝒙</mi><mrow id="S4.SS2.p1.15.m3.1.1.3" xref="S4.SS2.p1.15.m3.1.1.3.cmml"><mi id="S4.SS2.p1.15.m3.1.1.3.2" xref="S4.SS2.p1.15.m3.1.1.3.2.cmml">l</mi><mo id="S4.SS2.p1.15.m3.1.1.3.1" xref="S4.SS2.p1.15.m3.1.1.3.1.cmml">−</mo><mn id="S4.SS2.p1.15.m3.1.1.3.3" xref="S4.SS2.p1.15.m3.1.1.3.3.cmml">1</mn></mrow></msup><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.15.m3.1b"><apply id="S4.SS2.p1.15.m3.1.1.cmml" xref="S4.SS2.p1.15.m3.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.15.m3.1.1.1.cmml" xref="S4.SS2.p1.15.m3.1.1">superscript</csymbol><ci id="S4.SS2.p1.15.m3.1.1.2.cmml" xref="S4.SS2.p1.15.m3.1.1.2">𝒙</ci><apply id="S4.SS2.p1.15.m3.1.1.3.cmml" xref="S4.SS2.p1.15.m3.1.1.3"><minus id="S4.SS2.p1.15.m3.1.1.3.1.cmml" xref="S4.SS2.p1.15.m3.1.1.3.1"></minus><ci id="S4.SS2.p1.15.m3.1.1.3.2.cmml" xref="S4.SS2.p1.15.m3.1.1.3.2">𝑙</ci><cn type="integer" id="S4.SS2.p1.15.m3.1.1.3.3.cmml" xref="S4.SS2.p1.15.m3.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.15.m3.1c">{\bm{x}}^{l-1}</annotation></semantics></math>, and <math id="S4.SS2.p1.16.m4.1" class="ltx_Math" alttext="\mathrm{sin}(\cdot)" display="inline"><semantics id="S4.SS2.p1.16.m4.1a"><mrow id="S4.SS2.p1.16.m4.1.2" xref="S4.SS2.p1.16.m4.1.2.cmml"><mi id="S4.SS2.p1.16.m4.1.2.2" xref="S4.SS2.p1.16.m4.1.2.2.cmml">sin</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p1.16.m4.1.2.1" xref="S4.SS2.p1.16.m4.1.2.1.cmml">​</mo><mrow id="S4.SS2.p1.16.m4.1.2.3.2" xref="S4.SS2.p1.16.m4.1.2.cmml"><mo stretchy="false" id="S4.SS2.p1.16.m4.1.2.3.2.1" xref="S4.SS2.p1.16.m4.1.2.cmml">(</mo><mo lspace="0em" rspace="0em" id="S4.SS2.p1.16.m4.1.1" xref="S4.SS2.p1.16.m4.1.1.cmml">⋅</mo><mo stretchy="false" id="S4.SS2.p1.16.m4.1.2.3.2.2" xref="S4.SS2.p1.16.m4.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.16.m4.1b"><apply id="S4.SS2.p1.16.m4.1.2.cmml" xref="S4.SS2.p1.16.m4.1.2"><times id="S4.SS2.p1.16.m4.1.2.1.cmml" xref="S4.SS2.p1.16.m4.1.2.1"></times><ci id="S4.SS2.p1.16.m4.1.2.2.cmml" xref="S4.SS2.p1.16.m4.1.2.2">sin</ci><ci id="S4.SS2.p1.16.m4.1.1.cmml" xref="S4.SS2.p1.16.m4.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.16.m4.1c">\mathrm{sin}(\cdot)</annotation></semantics></math> is the <math id="S4.SS2.p1.17.m5.1" class="ltx_Math" alttext="\mathrm{sine}" display="inline"><semantics id="S4.SS2.p1.17.m5.1a"><mi id="S4.SS2.p1.17.m5.1.1" xref="S4.SS2.p1.17.m5.1.1.cmml">sine</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.17.m5.1b"><ci id="S4.SS2.p1.17.m5.1.1.cmml" xref="S4.SS2.p1.17.m5.1.1">sine</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.17.m5.1c">\mathrm{sine}</annotation></semantics></math> function.
<math id="S4.SS2.p1.18.m6.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S4.SS2.p1.18.m6.1a"><mi id="S4.SS2.p1.18.m6.1.1" xref="S4.SS2.p1.18.m6.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.18.m6.1b"><ci id="S4.SS2.p1.18.m6.1.1.cmml" xref="S4.SS2.p1.18.m6.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.18.m6.1c">t</annotation></semantics></math> is an interpolation hyperparameter
that adjusts the relative influence of each vector on the resulting direction, tailored to the layer depth and specific characteristics of the KV pairs.
Note that when we set <math id="S4.SS2.p1.19.m7.1" class="ltx_Math" alttext="t=0.5" display="inline"><semantics id="S4.SS2.p1.19.m7.1a"><mrow id="S4.SS2.p1.19.m7.1.1" xref="S4.SS2.p1.19.m7.1.1.cmml"><mi id="S4.SS2.p1.19.m7.1.1.2" xref="S4.SS2.p1.19.m7.1.1.2.cmml">t</mi><mo id="S4.SS2.p1.19.m7.1.1.1" xref="S4.SS2.p1.19.m7.1.1.1.cmml">=</mo><mn id="S4.SS2.p1.19.m7.1.1.3" xref="S4.SS2.p1.19.m7.1.1.3.cmml">0.5</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.19.m7.1b"><apply id="S4.SS2.p1.19.m7.1.1.cmml" xref="S4.SS2.p1.19.m7.1.1"><eq id="S4.SS2.p1.19.m7.1.1.1.cmml" xref="S4.SS2.p1.19.m7.1.1.1"></eq><ci id="S4.SS2.p1.19.m7.1.1.2.cmml" xref="S4.SS2.p1.19.m7.1.1.2">𝑡</ci><cn type="float" id="S4.SS2.p1.19.m7.1.1.3.cmml" xref="S4.SS2.p1.19.m7.1.1.3">0.5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.19.m7.1c">t=0.5</annotation></semantics></math>, it will become an average merging, which we consider special cases in Eq. (<a href="#A3.E1" title="Equation A ‣ Appendix C Discussions and Limitations ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a>).
The merged cache for each token pair is then a concatenation of the directional vector, magnitude and <math id="S4.SS2.p1.20.m8.2" class="ltx_Math" alttext="\Omega^{l,l-1}" display="inline"><semantics id="S4.SS2.p1.20.m8.2a"><msup id="S4.SS2.p1.20.m8.2.3" xref="S4.SS2.p1.20.m8.2.3.cmml"><mi mathvariant="normal" id="S4.SS2.p1.20.m8.2.3.2" xref="S4.SS2.p1.20.m8.2.3.2.cmml">Ω</mi><mrow id="S4.SS2.p1.20.m8.2.2.2.2" xref="S4.SS2.p1.20.m8.2.2.2.3.cmml"><mi id="S4.SS2.p1.20.m8.1.1.1.1" xref="S4.SS2.p1.20.m8.1.1.1.1.cmml">l</mi><mo id="S4.SS2.p1.20.m8.2.2.2.2.2" xref="S4.SS2.p1.20.m8.2.2.2.3.cmml">,</mo><mrow id="S4.SS2.p1.20.m8.2.2.2.2.1" xref="S4.SS2.p1.20.m8.2.2.2.2.1.cmml"><mi id="S4.SS2.p1.20.m8.2.2.2.2.1.2" xref="S4.SS2.p1.20.m8.2.2.2.2.1.2.cmml">l</mi><mo id="S4.SS2.p1.20.m8.2.2.2.2.1.1" xref="S4.SS2.p1.20.m8.2.2.2.2.1.1.cmml">−</mo><mn id="S4.SS2.p1.20.m8.2.2.2.2.1.3" xref="S4.SS2.p1.20.m8.2.2.2.2.1.3.cmml">1</mn></mrow></mrow></msup><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.20.m8.2b"><apply id="S4.SS2.p1.20.m8.2.3.cmml" xref="S4.SS2.p1.20.m8.2.3"><csymbol cd="ambiguous" id="S4.SS2.p1.20.m8.2.3.1.cmml" xref="S4.SS2.p1.20.m8.2.3">superscript</csymbol><ci id="S4.SS2.p1.20.m8.2.3.2.cmml" xref="S4.SS2.p1.20.m8.2.3.2">Ω</ci><list id="S4.SS2.p1.20.m8.2.2.2.3.cmml" xref="S4.SS2.p1.20.m8.2.2.2.2"><ci id="S4.SS2.p1.20.m8.1.1.1.1.cmml" xref="S4.SS2.p1.20.m8.1.1.1.1">𝑙</ci><apply id="S4.SS2.p1.20.m8.2.2.2.2.1.cmml" xref="S4.SS2.p1.20.m8.2.2.2.2.1"><minus id="S4.SS2.p1.20.m8.2.2.2.2.1.1.cmml" xref="S4.SS2.p1.20.m8.2.2.2.2.1.1"></minus><ci id="S4.SS2.p1.20.m8.2.2.2.2.1.2.cmml" xref="S4.SS2.p1.20.m8.2.2.2.2.1.2">𝑙</ci><cn type="integer" id="S4.SS2.p1.20.m8.2.2.2.2.1.3.cmml" xref="S4.SS2.p1.20.m8.2.2.2.2.1.3">1</cn></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.20.m8.2c">\Omega^{l,l-1}</annotation></semantics></math>, denoting as <math id="S4.SS2.p1.21.m9.10" class="ltx_Math" alttext="{\bm{c}}^{l,l-1}=[{\bm{e}}^{l,l-1},{\lVert{\bm{x}}^{l-1}\rVert},{\lVert{\bm{x}}^{l}\rVert},\Omega^{l,l-1}]" display="inline"><semantics id="S4.SS2.p1.21.m9.10a"><mrow id="S4.SS2.p1.21.m9.10.10" xref="S4.SS2.p1.21.m9.10.10.cmml"><msup id="S4.SS2.p1.21.m9.10.10.6" xref="S4.SS2.p1.21.m9.10.10.6.cmml"><mi id="S4.SS2.p1.21.m9.10.10.6.2" xref="S4.SS2.p1.21.m9.10.10.6.2.cmml">𝒄</mi><mrow id="S4.SS2.p1.21.m9.2.2.2.2" xref="S4.SS2.p1.21.m9.2.2.2.3.cmml"><mi id="S4.SS2.p1.21.m9.1.1.1.1" xref="S4.SS2.p1.21.m9.1.1.1.1.cmml">l</mi><mo id="S4.SS2.p1.21.m9.2.2.2.2.2" xref="S4.SS2.p1.21.m9.2.2.2.3.cmml">,</mo><mrow id="S4.SS2.p1.21.m9.2.2.2.2.1" xref="S4.SS2.p1.21.m9.2.2.2.2.1.cmml"><mi id="S4.SS2.p1.21.m9.2.2.2.2.1.2" xref="S4.SS2.p1.21.m9.2.2.2.2.1.2.cmml">l</mi><mo id="S4.SS2.p1.21.m9.2.2.2.2.1.1" xref="S4.SS2.p1.21.m9.2.2.2.2.1.1.cmml">−</mo><mn id="S4.SS2.p1.21.m9.2.2.2.2.1.3" xref="S4.SS2.p1.21.m9.2.2.2.2.1.3.cmml">1</mn></mrow></mrow></msup><mo id="S4.SS2.p1.21.m9.10.10.5" xref="S4.SS2.p1.21.m9.10.10.5.cmml">=</mo><mrow id="S4.SS2.p1.21.m9.10.10.4.4" xref="S4.SS2.p1.21.m9.10.10.4.5.cmml"><mo stretchy="false" id="S4.SS2.p1.21.m9.10.10.4.4.5" xref="S4.SS2.p1.21.m9.10.10.4.5.cmml">[</mo><msup id="S4.SS2.p1.21.m9.7.7.1.1.1" xref="S4.SS2.p1.21.m9.7.7.1.1.1.cmml"><mi id="S4.SS2.p1.21.m9.7.7.1.1.1.2" xref="S4.SS2.p1.21.m9.7.7.1.1.1.2.cmml">𝒆</mi><mrow id="S4.SS2.p1.21.m9.4.4.2.2" xref="S4.SS2.p1.21.m9.4.4.2.3.cmml"><mi id="S4.SS2.p1.21.m9.3.3.1.1" xref="S4.SS2.p1.21.m9.3.3.1.1.cmml">l</mi><mo id="S4.SS2.p1.21.m9.4.4.2.2.2" xref="S4.SS2.p1.21.m9.4.4.2.3.cmml">,</mo><mrow id="S4.SS2.p1.21.m9.4.4.2.2.1" xref="S4.SS2.p1.21.m9.4.4.2.2.1.cmml"><mi id="S4.SS2.p1.21.m9.4.4.2.2.1.2" xref="S4.SS2.p1.21.m9.4.4.2.2.1.2.cmml">l</mi><mo id="S4.SS2.p1.21.m9.4.4.2.2.1.1" xref="S4.SS2.p1.21.m9.4.4.2.2.1.1.cmml">−</mo><mn id="S4.SS2.p1.21.m9.4.4.2.2.1.3" xref="S4.SS2.p1.21.m9.4.4.2.2.1.3.cmml">1</mn></mrow></mrow></msup><mo id="S4.SS2.p1.21.m9.10.10.4.4.6" xref="S4.SS2.p1.21.m9.10.10.4.5.cmml">,</mo><mrow id="S4.SS2.p1.21.m9.8.8.2.2.2.1" xref="S4.SS2.p1.21.m9.8.8.2.2.2.2.cmml"><mo fence="true" lspace="0em" rspace="0em" id="S4.SS2.p1.21.m9.8.8.2.2.2.1.2" xref="S4.SS2.p1.21.m9.8.8.2.2.2.2.1.cmml">∥</mo><msup id="S4.SS2.p1.21.m9.8.8.2.2.2.1.1" xref="S4.SS2.p1.21.m9.8.8.2.2.2.1.1.cmml"><mi id="S4.SS2.p1.21.m9.8.8.2.2.2.1.1.2" xref="S4.SS2.p1.21.m9.8.8.2.2.2.1.1.2.cmml">𝒙</mi><mrow id="S4.SS2.p1.21.m9.8.8.2.2.2.1.1.3" xref="S4.SS2.p1.21.m9.8.8.2.2.2.1.1.3.cmml"><mi id="S4.SS2.p1.21.m9.8.8.2.2.2.1.1.3.2" xref="S4.SS2.p1.21.m9.8.8.2.2.2.1.1.3.2.cmml">l</mi><mo id="S4.SS2.p1.21.m9.8.8.2.2.2.1.1.3.1" xref="S4.SS2.p1.21.m9.8.8.2.2.2.1.1.3.1.cmml">−</mo><mn id="S4.SS2.p1.21.m9.8.8.2.2.2.1.1.3.3" xref="S4.SS2.p1.21.m9.8.8.2.2.2.1.1.3.3.cmml">1</mn></mrow></msup><mo fence="true" lspace="0em" rspace="0em" id="S4.SS2.p1.21.m9.8.8.2.2.2.1.3" xref="S4.SS2.p1.21.m9.8.8.2.2.2.2.1.cmml">∥</mo></mrow><mo id="S4.SS2.p1.21.m9.10.10.4.4.7" xref="S4.SS2.p1.21.m9.10.10.4.5.cmml">,</mo><mrow id="S4.SS2.p1.21.m9.9.9.3.3.3.1" xref="S4.SS2.p1.21.m9.9.9.3.3.3.2.cmml"><mo fence="true" lspace="0em" rspace="0em" id="S4.SS2.p1.21.m9.9.9.3.3.3.1.2" xref="S4.SS2.p1.21.m9.9.9.3.3.3.2.1.cmml">∥</mo><msup id="S4.SS2.p1.21.m9.9.9.3.3.3.1.1" xref="S4.SS2.p1.21.m9.9.9.3.3.3.1.1.cmml"><mi id="S4.SS2.p1.21.m9.9.9.3.3.3.1.1.2" xref="S4.SS2.p1.21.m9.9.9.3.3.3.1.1.2.cmml">𝒙</mi><mi id="S4.SS2.p1.21.m9.9.9.3.3.3.1.1.3" xref="S4.SS2.p1.21.m9.9.9.3.3.3.1.1.3.cmml">l</mi></msup><mo fence="true" lspace="0em" rspace="0em" id="S4.SS2.p1.21.m9.9.9.3.3.3.1.3" xref="S4.SS2.p1.21.m9.9.9.3.3.3.2.1.cmml">∥</mo></mrow><mo id="S4.SS2.p1.21.m9.10.10.4.4.8" xref="S4.SS2.p1.21.m9.10.10.4.5.cmml">,</mo><msup id="S4.SS2.p1.21.m9.10.10.4.4.4" xref="S4.SS2.p1.21.m9.10.10.4.4.4.cmml"><mi mathvariant="normal" id="S4.SS2.p1.21.m9.10.10.4.4.4.2" xref="S4.SS2.p1.21.m9.10.10.4.4.4.2.cmml">Ω</mi><mrow id="S4.SS2.p1.21.m9.6.6.2.2" xref="S4.SS2.p1.21.m9.6.6.2.3.cmml"><mi id="S4.SS2.p1.21.m9.5.5.1.1" xref="S4.SS2.p1.21.m9.5.5.1.1.cmml">l</mi><mo id="S4.SS2.p1.21.m9.6.6.2.2.2" xref="S4.SS2.p1.21.m9.6.6.2.3.cmml">,</mo><mrow id="S4.SS2.p1.21.m9.6.6.2.2.1" xref="S4.SS2.p1.21.m9.6.6.2.2.1.cmml"><mi id="S4.SS2.p1.21.m9.6.6.2.2.1.2" xref="S4.SS2.p1.21.m9.6.6.2.2.1.2.cmml">l</mi><mo id="S4.SS2.p1.21.m9.6.6.2.2.1.1" xref="S4.SS2.p1.21.m9.6.6.2.2.1.1.cmml">−</mo><mn id="S4.SS2.p1.21.m9.6.6.2.2.1.3" xref="S4.SS2.p1.21.m9.6.6.2.2.1.3.cmml">1</mn></mrow></mrow></msup><mo stretchy="false" id="S4.SS2.p1.21.m9.10.10.4.4.9" xref="S4.SS2.p1.21.m9.10.10.4.5.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.21.m9.10b"><apply id="S4.SS2.p1.21.m9.10.10.cmml" xref="S4.SS2.p1.21.m9.10.10"><eq id="S4.SS2.p1.21.m9.10.10.5.cmml" xref="S4.SS2.p1.21.m9.10.10.5"></eq><apply id="S4.SS2.p1.21.m9.10.10.6.cmml" xref="S4.SS2.p1.21.m9.10.10.6"><csymbol cd="ambiguous" id="S4.SS2.p1.21.m9.10.10.6.1.cmml" xref="S4.SS2.p1.21.m9.10.10.6">superscript</csymbol><ci id="S4.SS2.p1.21.m9.10.10.6.2.cmml" xref="S4.SS2.p1.21.m9.10.10.6.2">𝒄</ci><list id="S4.SS2.p1.21.m9.2.2.2.3.cmml" xref="S4.SS2.p1.21.m9.2.2.2.2"><ci id="S4.SS2.p1.21.m9.1.1.1.1.cmml" xref="S4.SS2.p1.21.m9.1.1.1.1">𝑙</ci><apply id="S4.SS2.p1.21.m9.2.2.2.2.1.cmml" xref="S4.SS2.p1.21.m9.2.2.2.2.1"><minus id="S4.SS2.p1.21.m9.2.2.2.2.1.1.cmml" xref="S4.SS2.p1.21.m9.2.2.2.2.1.1"></minus><ci id="S4.SS2.p1.21.m9.2.2.2.2.1.2.cmml" xref="S4.SS2.p1.21.m9.2.2.2.2.1.2">𝑙</ci><cn type="integer" id="S4.SS2.p1.21.m9.2.2.2.2.1.3.cmml" xref="S4.SS2.p1.21.m9.2.2.2.2.1.3">1</cn></apply></list></apply><list id="S4.SS2.p1.21.m9.10.10.4.5.cmml" xref="S4.SS2.p1.21.m9.10.10.4.4"><apply id="S4.SS2.p1.21.m9.7.7.1.1.1.cmml" xref="S4.SS2.p1.21.m9.7.7.1.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.21.m9.7.7.1.1.1.1.cmml" xref="S4.SS2.p1.21.m9.7.7.1.1.1">superscript</csymbol><ci id="S4.SS2.p1.21.m9.7.7.1.1.1.2.cmml" xref="S4.SS2.p1.21.m9.7.7.1.1.1.2">𝒆</ci><list id="S4.SS2.p1.21.m9.4.4.2.3.cmml" xref="S4.SS2.p1.21.m9.4.4.2.2"><ci id="S4.SS2.p1.21.m9.3.3.1.1.cmml" xref="S4.SS2.p1.21.m9.3.3.1.1">𝑙</ci><apply id="S4.SS2.p1.21.m9.4.4.2.2.1.cmml" xref="S4.SS2.p1.21.m9.4.4.2.2.1"><minus id="S4.SS2.p1.21.m9.4.4.2.2.1.1.cmml" xref="S4.SS2.p1.21.m9.4.4.2.2.1.1"></minus><ci id="S4.SS2.p1.21.m9.4.4.2.2.1.2.cmml" xref="S4.SS2.p1.21.m9.4.4.2.2.1.2">𝑙</ci><cn type="integer" id="S4.SS2.p1.21.m9.4.4.2.2.1.3.cmml" xref="S4.SS2.p1.21.m9.4.4.2.2.1.3">1</cn></apply></list></apply><apply id="S4.SS2.p1.21.m9.8.8.2.2.2.2.cmml" xref="S4.SS2.p1.21.m9.8.8.2.2.2.1"><csymbol cd="latexml" id="S4.SS2.p1.21.m9.8.8.2.2.2.2.1.cmml" xref="S4.SS2.p1.21.m9.8.8.2.2.2.1.2">delimited-∥∥</csymbol><apply id="S4.SS2.p1.21.m9.8.8.2.2.2.1.1.cmml" xref="S4.SS2.p1.21.m9.8.8.2.2.2.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.21.m9.8.8.2.2.2.1.1.1.cmml" xref="S4.SS2.p1.21.m9.8.8.2.2.2.1.1">superscript</csymbol><ci id="S4.SS2.p1.21.m9.8.8.2.2.2.1.1.2.cmml" xref="S4.SS2.p1.21.m9.8.8.2.2.2.1.1.2">𝒙</ci><apply id="S4.SS2.p1.21.m9.8.8.2.2.2.1.1.3.cmml" xref="S4.SS2.p1.21.m9.8.8.2.2.2.1.1.3"><minus id="S4.SS2.p1.21.m9.8.8.2.2.2.1.1.3.1.cmml" xref="S4.SS2.p1.21.m9.8.8.2.2.2.1.1.3.1"></minus><ci id="S4.SS2.p1.21.m9.8.8.2.2.2.1.1.3.2.cmml" xref="S4.SS2.p1.21.m9.8.8.2.2.2.1.1.3.2">𝑙</ci><cn type="integer" id="S4.SS2.p1.21.m9.8.8.2.2.2.1.1.3.3.cmml" xref="S4.SS2.p1.21.m9.8.8.2.2.2.1.1.3.3">1</cn></apply></apply></apply><apply id="S4.SS2.p1.21.m9.9.9.3.3.3.2.cmml" xref="S4.SS2.p1.21.m9.9.9.3.3.3.1"><csymbol cd="latexml" id="S4.SS2.p1.21.m9.9.9.3.3.3.2.1.cmml" xref="S4.SS2.p1.21.m9.9.9.3.3.3.1.2">delimited-∥∥</csymbol><apply id="S4.SS2.p1.21.m9.9.9.3.3.3.1.1.cmml" xref="S4.SS2.p1.21.m9.9.9.3.3.3.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.21.m9.9.9.3.3.3.1.1.1.cmml" xref="S4.SS2.p1.21.m9.9.9.3.3.3.1.1">superscript</csymbol><ci id="S4.SS2.p1.21.m9.9.9.3.3.3.1.1.2.cmml" xref="S4.SS2.p1.21.m9.9.9.3.3.3.1.1.2">𝒙</ci><ci id="S4.SS2.p1.21.m9.9.9.3.3.3.1.1.3.cmml" xref="S4.SS2.p1.21.m9.9.9.3.3.3.1.1.3">𝑙</ci></apply></apply><apply id="S4.SS2.p1.21.m9.10.10.4.4.4.cmml" xref="S4.SS2.p1.21.m9.10.10.4.4.4"><csymbol cd="ambiguous" id="S4.SS2.p1.21.m9.10.10.4.4.4.1.cmml" xref="S4.SS2.p1.21.m9.10.10.4.4.4">superscript</csymbol><ci id="S4.SS2.p1.21.m9.10.10.4.4.4.2.cmml" xref="S4.SS2.p1.21.m9.10.10.4.4.4.2">Ω</ci><list id="S4.SS2.p1.21.m9.6.6.2.3.cmml" xref="S4.SS2.p1.21.m9.6.6.2.2"><ci id="S4.SS2.p1.21.m9.5.5.1.1.cmml" xref="S4.SS2.p1.21.m9.5.5.1.1">𝑙</ci><apply id="S4.SS2.p1.21.m9.6.6.2.2.1.cmml" xref="S4.SS2.p1.21.m9.6.6.2.2.1"><minus id="S4.SS2.p1.21.m9.6.6.2.2.1.1.cmml" xref="S4.SS2.p1.21.m9.6.6.2.2.1.1"></minus><ci id="S4.SS2.p1.21.m9.6.6.2.2.1.2.cmml" xref="S4.SS2.p1.21.m9.6.6.2.2.1.2">𝑙</ci><cn type="integer" id="S4.SS2.p1.21.m9.6.6.2.2.1.3.cmml" xref="S4.SS2.p1.21.m9.6.6.2.2.1.3">1</cn></apply></list></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.21.m9.10c">{\bm{c}}^{l,l-1}=[{\bm{e}}^{l,l-1},{\lVert{\bm{x}}^{l-1}\rVert},{\lVert{\bm{x}}^{l}\rVert},\Omega^{l,l-1}]</annotation></semantics></math>, cached components as shown in <a href="#S4.F3" title="In 4.1 Cross-Layer Compression ‣ 4 Method ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">3</span></a>(a).
Note that apart from storing the merged directional vector, we only need to store additional token-wise magnitude and angle scalars, which is memory efficient.
In this way,
we achieve substantial
memory efficiencies through reduced redundancy while ensuring the retention of the critical functional characteristics of the original KV pairs across transformer layers.</p>
</div>
<div id="S4.SS2.p2" class="ltx_para">
<p id="S4.SS2.p2.1" class="ltx_p"><span id="S4.SS2.p2.1.1" class="ltx_text ltx_font_bold">Unmergeable token retention.</span>
Highly distinct pairs are sensitive to merging operations, leading us to propose unmergeable token retention, as shown in <a href="#S4.F3" title="In 4.1 Cross-Layer Compression ‣ 4 Method ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">3</span></a>(a). Despite the high similarity between KV cache states across neighbouring layers, a few sensitive distinct pairs remain that are significantly difficult to merge and share. Aligned with previous studies, these distinct tokens carry substantial semantic meanings <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib15" title="" class="ltx_ref">15</a>, <a href="#bib.bib16" title="" class="ltx_ref">16</a>]</cite>.
We observe that merging sensitive tokens, which preserve layer-specific information, can lead to significant performance degradation. Therefore, it is crucial to properly disentangle the shared and unique information between adjacent layers. To address this issue, we designed a token retention strategy to selectively retain tokens that cannot be merged based on their angular distance, defined as:
<math id="S4.SS2.p2.1.m1.2" class="ltx_Math" alttext="d({\bm{x}}^{l},{\bm{x}}^{l-1})=\frac{1}{\pi}\Omega" display="inline"><semantics id="S4.SS2.p2.1.m1.2a"><mrow id="S4.SS2.p2.1.m1.2.2" xref="S4.SS2.p2.1.m1.2.2.cmml"><mrow id="S4.SS2.p2.1.m1.2.2.2" xref="S4.SS2.p2.1.m1.2.2.2.cmml"><mi id="S4.SS2.p2.1.m1.2.2.2.4" xref="S4.SS2.p2.1.m1.2.2.2.4.cmml">d</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p2.1.m1.2.2.2.3" xref="S4.SS2.p2.1.m1.2.2.2.3.cmml">​</mo><mrow id="S4.SS2.p2.1.m1.2.2.2.2.2" xref="S4.SS2.p2.1.m1.2.2.2.2.3.cmml"><mo stretchy="false" id="S4.SS2.p2.1.m1.2.2.2.2.2.3" xref="S4.SS2.p2.1.m1.2.2.2.2.3.cmml">(</mo><msup id="S4.SS2.p2.1.m1.1.1.1.1.1.1" xref="S4.SS2.p2.1.m1.1.1.1.1.1.1.cmml"><mi id="S4.SS2.p2.1.m1.1.1.1.1.1.1.2" xref="S4.SS2.p2.1.m1.1.1.1.1.1.1.2.cmml">𝒙</mi><mi id="S4.SS2.p2.1.m1.1.1.1.1.1.1.3" xref="S4.SS2.p2.1.m1.1.1.1.1.1.1.3.cmml">l</mi></msup><mo id="S4.SS2.p2.1.m1.2.2.2.2.2.4" xref="S4.SS2.p2.1.m1.2.2.2.2.3.cmml">,</mo><msup id="S4.SS2.p2.1.m1.2.2.2.2.2.2" xref="S4.SS2.p2.1.m1.2.2.2.2.2.2.cmml"><mi id="S4.SS2.p2.1.m1.2.2.2.2.2.2.2" xref="S4.SS2.p2.1.m1.2.2.2.2.2.2.2.cmml">𝒙</mi><mrow id="S4.SS2.p2.1.m1.2.2.2.2.2.2.3" xref="S4.SS2.p2.1.m1.2.2.2.2.2.2.3.cmml"><mi id="S4.SS2.p2.1.m1.2.2.2.2.2.2.3.2" xref="S4.SS2.p2.1.m1.2.2.2.2.2.2.3.2.cmml">l</mi><mo id="S4.SS2.p2.1.m1.2.2.2.2.2.2.3.1" xref="S4.SS2.p2.1.m1.2.2.2.2.2.2.3.1.cmml">−</mo><mn id="S4.SS2.p2.1.m1.2.2.2.2.2.2.3.3" xref="S4.SS2.p2.1.m1.2.2.2.2.2.2.3.3.cmml">1</mn></mrow></msup><mo stretchy="false" id="S4.SS2.p2.1.m1.2.2.2.2.2.5" xref="S4.SS2.p2.1.m1.2.2.2.2.3.cmml">)</mo></mrow></mrow><mo id="S4.SS2.p2.1.m1.2.2.3" xref="S4.SS2.p2.1.m1.2.2.3.cmml">=</mo><mrow id="S4.SS2.p2.1.m1.2.2.4" xref="S4.SS2.p2.1.m1.2.2.4.cmml"><mfrac id="S4.SS2.p2.1.m1.2.2.4.2" xref="S4.SS2.p2.1.m1.2.2.4.2.cmml"><mn id="S4.SS2.p2.1.m1.2.2.4.2.2" xref="S4.SS2.p2.1.m1.2.2.4.2.2.cmml">1</mn><mi id="S4.SS2.p2.1.m1.2.2.4.2.3" xref="S4.SS2.p2.1.m1.2.2.4.2.3.cmml">π</mi></mfrac><mo lspace="0em" rspace="0em" id="S4.SS2.p2.1.m1.2.2.4.1" xref="S4.SS2.p2.1.m1.2.2.4.1.cmml">​</mo><mi mathvariant="normal" id="S4.SS2.p2.1.m1.2.2.4.3" xref="S4.SS2.p2.1.m1.2.2.4.3.cmml">Ω</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.1.m1.2b"><apply id="S4.SS2.p2.1.m1.2.2.cmml" xref="S4.SS2.p2.1.m1.2.2"><eq id="S4.SS2.p2.1.m1.2.2.3.cmml" xref="S4.SS2.p2.1.m1.2.2.3"></eq><apply id="S4.SS2.p2.1.m1.2.2.2.cmml" xref="S4.SS2.p2.1.m1.2.2.2"><times id="S4.SS2.p2.1.m1.2.2.2.3.cmml" xref="S4.SS2.p2.1.m1.2.2.2.3"></times><ci id="S4.SS2.p2.1.m1.2.2.2.4.cmml" xref="S4.SS2.p2.1.m1.2.2.2.4">𝑑</ci><interval closure="open" id="S4.SS2.p2.1.m1.2.2.2.2.3.cmml" xref="S4.SS2.p2.1.m1.2.2.2.2.2"><apply id="S4.SS2.p2.1.m1.1.1.1.1.1.1.cmml" xref="S4.SS2.p2.1.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.SS2.p2.1.m1.1.1.1.1.1.1.1.cmml" xref="S4.SS2.p2.1.m1.1.1.1.1.1.1">superscript</csymbol><ci id="S4.SS2.p2.1.m1.1.1.1.1.1.1.2.cmml" xref="S4.SS2.p2.1.m1.1.1.1.1.1.1.2">𝒙</ci><ci id="S4.SS2.p2.1.m1.1.1.1.1.1.1.3.cmml" xref="S4.SS2.p2.1.m1.1.1.1.1.1.1.3">𝑙</ci></apply><apply id="S4.SS2.p2.1.m1.2.2.2.2.2.2.cmml" xref="S4.SS2.p2.1.m1.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S4.SS2.p2.1.m1.2.2.2.2.2.2.1.cmml" xref="S4.SS2.p2.1.m1.2.2.2.2.2.2">superscript</csymbol><ci id="S4.SS2.p2.1.m1.2.2.2.2.2.2.2.cmml" xref="S4.SS2.p2.1.m1.2.2.2.2.2.2.2">𝒙</ci><apply id="S4.SS2.p2.1.m1.2.2.2.2.2.2.3.cmml" xref="S4.SS2.p2.1.m1.2.2.2.2.2.2.3"><minus id="S4.SS2.p2.1.m1.2.2.2.2.2.2.3.1.cmml" xref="S4.SS2.p2.1.m1.2.2.2.2.2.2.3.1"></minus><ci id="S4.SS2.p2.1.m1.2.2.2.2.2.2.3.2.cmml" xref="S4.SS2.p2.1.m1.2.2.2.2.2.2.3.2">𝑙</ci><cn type="integer" id="S4.SS2.p2.1.m1.2.2.2.2.2.2.3.3.cmml" xref="S4.SS2.p2.1.m1.2.2.2.2.2.2.3.3">1</cn></apply></apply></interval></apply><apply id="S4.SS2.p2.1.m1.2.2.4.cmml" xref="S4.SS2.p2.1.m1.2.2.4"><times id="S4.SS2.p2.1.m1.2.2.4.1.cmml" xref="S4.SS2.p2.1.m1.2.2.4.1"></times><apply id="S4.SS2.p2.1.m1.2.2.4.2.cmml" xref="S4.SS2.p2.1.m1.2.2.4.2"><divide id="S4.SS2.p2.1.m1.2.2.4.2.1.cmml" xref="S4.SS2.p2.1.m1.2.2.4.2"></divide><cn type="integer" id="S4.SS2.p2.1.m1.2.2.4.2.2.cmml" xref="S4.SS2.p2.1.m1.2.2.4.2.2">1</cn><ci id="S4.SS2.p2.1.m1.2.2.4.2.3.cmml" xref="S4.SS2.p2.1.m1.2.2.4.2.3">𝜋</ci></apply><ci id="S4.SS2.p2.1.m1.2.2.4.3.cmml" xref="S4.SS2.p2.1.m1.2.2.4.3">Ω</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.1.m1.2c">d({\bm{x}}^{l},{\bm{x}}^{l-1})=\frac{1}{\pi}\Omega</annotation></semantics></math>.
For the KV caches, the minimum and maximum angular distances are determined to identify the unmergeable tokens.</p>
</div>
<div id="S4.SS2.p3" class="ltx_para">
<p id="S4.SS2.p3.1" class="ltx_p">The set of required token indices to keep, <math id="S4.SS2.p3.1.m1.1" class="ltx_Math" alttext="\mathbb{I}" display="inline"><semantics id="S4.SS2.p3.1.m1.1a"><mi id="S4.SS2.p3.1.m1.1.1" xref="S4.SS2.p3.1.m1.1.1.cmml">𝕀</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p3.1.m1.1b"><ci id="S4.SS2.p3.1.m1.1.1.cmml" xref="S4.SS2.p3.1.m1.1.1">𝕀</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p3.1.m1.1c">\mathbb{I}</annotation></semantics></math>, is obtained by:</p>
<table id="S4.E4" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="S4.E4X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S4.E4X.2.1.1.m1.2" class="ltx_Math" alttext="\displaystyle{\mathbb{I}}=\{i\mid d_{i}&lt;d_{\text{min}}+(d_{\text{max}}-d_{\text{min}})\cdot\gamma\}," display="inline"><semantics id="S4.E4X.2.1.1.m1.2a"><mrow id="S4.E4X.2.1.1.m1.2.2.1" xref="S4.E4X.2.1.1.m1.2.2.1.1.cmml"><mrow id="S4.E4X.2.1.1.m1.2.2.1.1" xref="S4.E4X.2.1.1.m1.2.2.1.1.cmml"><mi id="S4.E4X.2.1.1.m1.2.2.1.1.3" xref="S4.E4X.2.1.1.m1.2.2.1.1.3.cmml">𝕀</mi><mo id="S4.E4X.2.1.1.m1.2.2.1.1.2" xref="S4.E4X.2.1.1.m1.2.2.1.1.2.cmml">=</mo><mrow id="S4.E4X.2.1.1.m1.2.2.1.1.1.1" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.2.cmml"><mo stretchy="false" id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.2" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.2.1.cmml">{</mo><mi id="S4.E4X.2.1.1.m1.1.1" xref="S4.E4X.2.1.1.m1.1.1.cmml">i</mi><mo fence="true" lspace="0em" rspace="0em" id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.3" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.2.1.cmml">∣</mo><mrow id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.cmml"><msub id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.3" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.3.cmml"><mi id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.3.2" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.3.2.cmml">d</mi><mi id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.3.3" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.3.3.cmml">i</mi></msub><mo id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.2" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.2.cmml">&lt;</mo><mrow id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.cmml"><msub id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.3" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.3.cmml"><mi id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.3.2" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.3.2.cmml">d</mi><mtext id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.3.3" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.3.3a.cmml">min</mtext></msub><mo id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.2" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.2.cmml">+</mo><mrow id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.cmml"><mrow id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.2" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.cmml"><msub id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.2" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.2.cmml"><mi id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.2.2" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.2.2.cmml">d</mi><mtext id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.2.3" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.2.3a.cmml">max</mtext></msub><mo id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.1" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.1.cmml">−</mo><msub id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.3" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.3.2" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.3.2.cmml">d</mi><mtext id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.3.3" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.3.3a.cmml">min</mtext></msub></mrow><mo rspace="0.055em" stretchy="false" id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.3" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo rspace="0.222em" id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.2" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.2.cmml">⋅</mo><mi id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.3" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.cmml">γ</mi></mrow></mrow></mrow><mo stretchy="false" id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.4" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.2.1.cmml">}</mo></mrow></mrow><mo id="S4.E4X.2.1.1.m1.2.2.1.2" xref="S4.E4X.2.1.1.m1.2.2.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.E4X.2.1.1.m1.2b"><apply id="S4.E4X.2.1.1.m1.2.2.1.1.cmml" xref="S4.E4X.2.1.1.m1.2.2.1"><eq id="S4.E4X.2.1.1.m1.2.2.1.1.2.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.2"></eq><ci id="S4.E4X.2.1.1.m1.2.2.1.1.3.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.3">𝕀</ci><apply id="S4.E4X.2.1.1.m1.2.2.1.1.1.2.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1"><csymbol cd="latexml" id="S4.E4X.2.1.1.m1.2.2.1.1.1.2.1.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.2">conditional-set</csymbol><ci id="S4.E4X.2.1.1.m1.1.1.cmml" xref="S4.E4X.2.1.1.m1.1.1">𝑖</ci><apply id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1"><lt id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.2.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.2"></lt><apply id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.3.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.3.1.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.3">subscript</csymbol><ci id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.3.2.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.3.2">𝑑</ci><ci id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.3.3.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.3.3">𝑖</ci></apply><apply id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1"><plus id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.2.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.2"></plus><apply id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.3.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.3.1.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.3">subscript</csymbol><ci id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.3.2.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.3.2">𝑑</ci><ci id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.3.3a.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.3.3"><mtext mathsize="70%" id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.3.3.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.3.3">min</mtext></ci></apply><apply id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1"><ci id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.2.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.2">⋅</ci><apply id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1"><minus id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.1"></minus><apply id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.2.2">𝑑</ci><ci id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.2.3a.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.2.3"><mtext mathsize="70%" id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.2.3">max</mtext></ci></apply><apply id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.3.2">𝑑</ci><ci id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.3.3a.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.3.3"><mtext mathsize="70%" id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.1.1.1.3.3">min</mtext></ci></apply></apply><ci id="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.3.cmml" xref="S4.E4X.2.1.1.m1.2.2.1.1.1.1.1.1.1.3">𝛾</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E4X.2.1.1.m1.2c">\displaystyle{\mathbb{I}}=\{i\mid d_{i}&lt;d_{\text{min}}+(d_{\text{max}}-d_{\text{min}})\cdot\gamma\},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(4)</span></td>
</tr>
</tbody>
</table>
<p id="S4.SS2.p3.3" class="ltx_p">where <math id="S4.SS2.p3.2.m1.1" class="ltx_Math" alttext="\gamma" display="inline"><semantics id="S4.SS2.p3.2.m1.1a"><mi id="S4.SS2.p3.2.m1.1.1" xref="S4.SS2.p3.2.m1.1.1.cmml">γ</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p3.2.m1.1b"><ci id="S4.SS2.p3.2.m1.1.1.cmml" xref="S4.SS2.p3.2.m1.1.1">𝛾</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p3.2.m1.1c">\gamma</annotation></semantics></math> is a predefined hyperparameter that controls the retention threshold.
The tokens with indices in <math id="S4.SS2.p3.3.m2.1" class="ltx_Math" alttext="{\mathbb{I}}" display="inline"><semantics id="S4.SS2.p3.3.m2.1a"><mi id="S4.SS2.p3.3.m2.1.1" xref="S4.SS2.p3.3.m2.1.1.cmml">𝕀</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p3.3.m2.1b"><ci id="S4.SS2.p3.3.m2.1.1.cmml" xref="S4.SS2.p3.3.m2.1.1">𝕀</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p3.3.m2.1c">{\mathbb{I}}</annotation></semantics></math> are retained and not compressed during the merge, which ensures that performance does not decline by preventing the loss of unmergeable tokens.</p>
</div>
<div id="S4.SS2.p4" class="ltx_para">
<p id="S4.SS2.p4.9" class="ltx_p">Next, let <math id="S4.SS2.p4.1.m1.1" class="ltx_Math" alttext="{\bm{X}}\in{\mathbb{R}}^{n\times h}" display="inline"><semantics id="S4.SS2.p4.1.m1.1a"><mrow id="S4.SS2.p4.1.m1.1.1" xref="S4.SS2.p4.1.m1.1.1.cmml"><mi id="S4.SS2.p4.1.m1.1.1.2" xref="S4.SS2.p4.1.m1.1.1.2.cmml">𝑿</mi><mo id="S4.SS2.p4.1.m1.1.1.1" xref="S4.SS2.p4.1.m1.1.1.1.cmml">∈</mo><msup id="S4.SS2.p4.1.m1.1.1.3" xref="S4.SS2.p4.1.m1.1.1.3.cmml"><mi id="S4.SS2.p4.1.m1.1.1.3.2" xref="S4.SS2.p4.1.m1.1.1.3.2.cmml">ℝ</mi><mrow id="S4.SS2.p4.1.m1.1.1.3.3" xref="S4.SS2.p4.1.m1.1.1.3.3.cmml"><mi id="S4.SS2.p4.1.m1.1.1.3.3.2" xref="S4.SS2.p4.1.m1.1.1.3.3.2.cmml">n</mi><mo lspace="0.222em" rspace="0.222em" id="S4.SS2.p4.1.m1.1.1.3.3.1" xref="S4.SS2.p4.1.m1.1.1.3.3.1.cmml">×</mo><mi id="S4.SS2.p4.1.m1.1.1.3.3.3" xref="S4.SS2.p4.1.m1.1.1.3.3.3.cmml">h</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p4.1.m1.1b"><apply id="S4.SS2.p4.1.m1.1.1.cmml" xref="S4.SS2.p4.1.m1.1.1"><in id="S4.SS2.p4.1.m1.1.1.1.cmml" xref="S4.SS2.p4.1.m1.1.1.1"></in><ci id="S4.SS2.p4.1.m1.1.1.2.cmml" xref="S4.SS2.p4.1.m1.1.1.2">𝑿</ci><apply id="S4.SS2.p4.1.m1.1.1.3.cmml" xref="S4.SS2.p4.1.m1.1.1.3"><csymbol cd="ambiguous" id="S4.SS2.p4.1.m1.1.1.3.1.cmml" xref="S4.SS2.p4.1.m1.1.1.3">superscript</csymbol><ci id="S4.SS2.p4.1.m1.1.1.3.2.cmml" xref="S4.SS2.p4.1.m1.1.1.3.2">ℝ</ci><apply id="S4.SS2.p4.1.m1.1.1.3.3.cmml" xref="S4.SS2.p4.1.m1.1.1.3.3"><times id="S4.SS2.p4.1.m1.1.1.3.3.1.cmml" xref="S4.SS2.p4.1.m1.1.1.3.3.1"></times><ci id="S4.SS2.p4.1.m1.1.1.3.3.2.cmml" xref="S4.SS2.p4.1.m1.1.1.3.3.2">𝑛</ci><ci id="S4.SS2.p4.1.m1.1.1.3.3.3.cmml" xref="S4.SS2.p4.1.m1.1.1.3.3.3">ℎ</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p4.1.m1.1c">{\bm{X}}\in{\mathbb{R}}^{n\times h}</annotation></semantics></math> be either the key or value cache at one attention layer, where <math id="S4.SS2.p4.2.m2.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S4.SS2.p4.2.m2.1a"><mi id="S4.SS2.p4.2.m2.1.1" xref="S4.SS2.p4.2.m2.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p4.2.m2.1b"><ci id="S4.SS2.p4.2.m2.1.1.cmml" xref="S4.SS2.p4.2.m2.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p4.2.m2.1c">n</annotation></semantics></math> denotes the number of tokens and <math id="S4.SS2.p4.3.m3.1" class="ltx_Math" alttext="h" display="inline"><semantics id="S4.SS2.p4.3.m3.1a"><mi id="S4.SS2.p4.3.m3.1.1" xref="S4.SS2.p4.3.m3.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p4.3.m3.1b"><ci id="S4.SS2.p4.3.m3.1.1.cmml" xref="S4.SS2.p4.3.m3.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p4.3.m3.1c">h</annotation></semantics></math> is the number of hidden dimensions, and <math id="S4.SS2.p4.4.m4.1" class="ltx_Math" alttext="{\bm{E}}\in{\mathbb{R}}^{n\times h}" display="inline"><semantics id="S4.SS2.p4.4.m4.1a"><mrow id="S4.SS2.p4.4.m4.1.1" xref="S4.SS2.p4.4.m4.1.1.cmml"><mi id="S4.SS2.p4.4.m4.1.1.2" xref="S4.SS2.p4.4.m4.1.1.2.cmml">𝑬</mi><mo id="S4.SS2.p4.4.m4.1.1.1" xref="S4.SS2.p4.4.m4.1.1.1.cmml">∈</mo><msup id="S4.SS2.p4.4.m4.1.1.3" xref="S4.SS2.p4.4.m4.1.1.3.cmml"><mi id="S4.SS2.p4.4.m4.1.1.3.2" xref="S4.SS2.p4.4.m4.1.1.3.2.cmml">ℝ</mi><mrow id="S4.SS2.p4.4.m4.1.1.3.3" xref="S4.SS2.p4.4.m4.1.1.3.3.cmml"><mi id="S4.SS2.p4.4.m4.1.1.3.3.2" xref="S4.SS2.p4.4.m4.1.1.3.3.2.cmml">n</mi><mo lspace="0.222em" rspace="0.222em" id="S4.SS2.p4.4.m4.1.1.3.3.1" xref="S4.SS2.p4.4.m4.1.1.3.3.1.cmml">×</mo><mi id="S4.SS2.p4.4.m4.1.1.3.3.3" xref="S4.SS2.p4.4.m4.1.1.3.3.3.cmml">h</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p4.4.m4.1b"><apply id="S4.SS2.p4.4.m4.1.1.cmml" xref="S4.SS2.p4.4.m4.1.1"><in id="S4.SS2.p4.4.m4.1.1.1.cmml" xref="S4.SS2.p4.4.m4.1.1.1"></in><ci id="S4.SS2.p4.4.m4.1.1.2.cmml" xref="S4.SS2.p4.4.m4.1.1.2">𝑬</ci><apply id="S4.SS2.p4.4.m4.1.1.3.cmml" xref="S4.SS2.p4.4.m4.1.1.3"><csymbol cd="ambiguous" id="S4.SS2.p4.4.m4.1.1.3.1.cmml" xref="S4.SS2.p4.4.m4.1.1.3">superscript</csymbol><ci id="S4.SS2.p4.4.m4.1.1.3.2.cmml" xref="S4.SS2.p4.4.m4.1.1.3.2">ℝ</ci><apply id="S4.SS2.p4.4.m4.1.1.3.3.cmml" xref="S4.SS2.p4.4.m4.1.1.3.3"><times id="S4.SS2.p4.4.m4.1.1.3.3.1.cmml" xref="S4.SS2.p4.4.m4.1.1.3.3.1"></times><ci id="S4.SS2.p4.4.m4.1.1.3.3.2.cmml" xref="S4.SS2.p4.4.m4.1.1.3.3.2">𝑛</ci><ci id="S4.SS2.p4.4.m4.1.1.3.3.3.cmml" xref="S4.SS2.p4.4.m4.1.1.3.3.3">ℎ</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p4.4.m4.1c">{\bm{E}}\in{\mathbb{R}}^{n\times h}</annotation></semantics></math> be the shared KV cache states.
Then, for each pair of neighbouring two layers, the unmergeable tokens are selected along with the token dimension by <math id="S4.SS2.p4.5.m5.1" class="ltx_Math" alttext="{\bm{R}}^{l}={\bm{X}}^{l}[{\mathbb{I}}]" display="inline"><semantics id="S4.SS2.p4.5.m5.1a"><mrow id="S4.SS2.p4.5.m5.1.2" xref="S4.SS2.p4.5.m5.1.2.cmml"><msup id="S4.SS2.p4.5.m5.1.2.2" xref="S4.SS2.p4.5.m5.1.2.2.cmml"><mi id="S4.SS2.p4.5.m5.1.2.2.2" xref="S4.SS2.p4.5.m5.1.2.2.2.cmml">𝑹</mi><mi id="S4.SS2.p4.5.m5.1.2.2.3" xref="S4.SS2.p4.5.m5.1.2.2.3.cmml">l</mi></msup><mo id="S4.SS2.p4.5.m5.1.2.1" xref="S4.SS2.p4.5.m5.1.2.1.cmml">=</mo><mrow id="S4.SS2.p4.5.m5.1.2.3" xref="S4.SS2.p4.5.m5.1.2.3.cmml"><msup id="S4.SS2.p4.5.m5.1.2.3.2" xref="S4.SS2.p4.5.m5.1.2.3.2.cmml"><mi id="S4.SS2.p4.5.m5.1.2.3.2.2" xref="S4.SS2.p4.5.m5.1.2.3.2.2.cmml">𝑿</mi><mi id="S4.SS2.p4.5.m5.1.2.3.2.3" xref="S4.SS2.p4.5.m5.1.2.3.2.3.cmml">l</mi></msup><mo lspace="0em" rspace="0em" id="S4.SS2.p4.5.m5.1.2.3.1" xref="S4.SS2.p4.5.m5.1.2.3.1.cmml">​</mo><mrow id="S4.SS2.p4.5.m5.1.2.3.3.2" xref="S4.SS2.p4.5.m5.1.2.3.3.1.cmml"><mo stretchy="false" id="S4.SS2.p4.5.m5.1.2.3.3.2.1" xref="S4.SS2.p4.5.m5.1.2.3.3.1.1.cmml">[</mo><mi id="S4.SS2.p4.5.m5.1.1" xref="S4.SS2.p4.5.m5.1.1.cmml">𝕀</mi><mo stretchy="false" id="S4.SS2.p4.5.m5.1.2.3.3.2.2" xref="S4.SS2.p4.5.m5.1.2.3.3.1.1.cmml">]</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p4.5.m5.1b"><apply id="S4.SS2.p4.5.m5.1.2.cmml" xref="S4.SS2.p4.5.m5.1.2"><eq id="S4.SS2.p4.5.m5.1.2.1.cmml" xref="S4.SS2.p4.5.m5.1.2.1"></eq><apply id="S4.SS2.p4.5.m5.1.2.2.cmml" xref="S4.SS2.p4.5.m5.1.2.2"><csymbol cd="ambiguous" id="S4.SS2.p4.5.m5.1.2.2.1.cmml" xref="S4.SS2.p4.5.m5.1.2.2">superscript</csymbol><ci id="S4.SS2.p4.5.m5.1.2.2.2.cmml" xref="S4.SS2.p4.5.m5.1.2.2.2">𝑹</ci><ci id="S4.SS2.p4.5.m5.1.2.2.3.cmml" xref="S4.SS2.p4.5.m5.1.2.2.3">𝑙</ci></apply><apply id="S4.SS2.p4.5.m5.1.2.3.cmml" xref="S4.SS2.p4.5.m5.1.2.3"><times id="S4.SS2.p4.5.m5.1.2.3.1.cmml" xref="S4.SS2.p4.5.m5.1.2.3.1"></times><apply id="S4.SS2.p4.5.m5.1.2.3.2.cmml" xref="S4.SS2.p4.5.m5.1.2.3.2"><csymbol cd="ambiguous" id="S4.SS2.p4.5.m5.1.2.3.2.1.cmml" xref="S4.SS2.p4.5.m5.1.2.3.2">superscript</csymbol><ci id="S4.SS2.p4.5.m5.1.2.3.2.2.cmml" xref="S4.SS2.p4.5.m5.1.2.3.2.2">𝑿</ci><ci id="S4.SS2.p4.5.m5.1.2.3.2.3.cmml" xref="S4.SS2.p4.5.m5.1.2.3.2.3">𝑙</ci></apply><apply id="S4.SS2.p4.5.m5.1.2.3.3.1.cmml" xref="S4.SS2.p4.5.m5.1.2.3.3.2"><csymbol cd="latexml" id="S4.SS2.p4.5.m5.1.2.3.3.1.1.cmml" xref="S4.SS2.p4.5.m5.1.2.3.3.2.1">delimited-[]</csymbol><ci id="S4.SS2.p4.5.m5.1.1.cmml" xref="S4.SS2.p4.5.m5.1.1">𝕀</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p4.5.m5.1c">{\bm{R}}^{l}={\bm{X}}^{l}[{\mathbb{I}}]</annotation></semantics></math>, <math id="S4.SS2.p4.6.m6.1" class="ltx_Math" alttext="{\bm{R}}^{l-1}={\bm{X}}^{l-1}[{\mathbb{I}}]" display="inline"><semantics id="S4.SS2.p4.6.m6.1a"><mrow id="S4.SS2.p4.6.m6.1.2" xref="S4.SS2.p4.6.m6.1.2.cmml"><msup id="S4.SS2.p4.6.m6.1.2.2" xref="S4.SS2.p4.6.m6.1.2.2.cmml"><mi id="S4.SS2.p4.6.m6.1.2.2.2" xref="S4.SS2.p4.6.m6.1.2.2.2.cmml">𝑹</mi><mrow id="S4.SS2.p4.6.m6.1.2.2.3" xref="S4.SS2.p4.6.m6.1.2.2.3.cmml"><mi id="S4.SS2.p4.6.m6.1.2.2.3.2" xref="S4.SS2.p4.6.m6.1.2.2.3.2.cmml">l</mi><mo id="S4.SS2.p4.6.m6.1.2.2.3.1" xref="S4.SS2.p4.6.m6.1.2.2.3.1.cmml">−</mo><mn id="S4.SS2.p4.6.m6.1.2.2.3.3" xref="S4.SS2.p4.6.m6.1.2.2.3.3.cmml">1</mn></mrow></msup><mo id="S4.SS2.p4.6.m6.1.2.1" xref="S4.SS2.p4.6.m6.1.2.1.cmml">=</mo><mrow id="S4.SS2.p4.6.m6.1.2.3" xref="S4.SS2.p4.6.m6.1.2.3.cmml"><msup id="S4.SS2.p4.6.m6.1.2.3.2" xref="S4.SS2.p4.6.m6.1.2.3.2.cmml"><mi id="S4.SS2.p4.6.m6.1.2.3.2.2" xref="S4.SS2.p4.6.m6.1.2.3.2.2.cmml">𝑿</mi><mrow id="S4.SS2.p4.6.m6.1.2.3.2.3" xref="S4.SS2.p4.6.m6.1.2.3.2.3.cmml"><mi id="S4.SS2.p4.6.m6.1.2.3.2.3.2" xref="S4.SS2.p4.6.m6.1.2.3.2.3.2.cmml">l</mi><mo id="S4.SS2.p4.6.m6.1.2.3.2.3.1" xref="S4.SS2.p4.6.m6.1.2.3.2.3.1.cmml">−</mo><mn id="S4.SS2.p4.6.m6.1.2.3.2.3.3" xref="S4.SS2.p4.6.m6.1.2.3.2.3.3.cmml">1</mn></mrow></msup><mo lspace="0em" rspace="0em" id="S4.SS2.p4.6.m6.1.2.3.1" xref="S4.SS2.p4.6.m6.1.2.3.1.cmml">​</mo><mrow id="S4.SS2.p4.6.m6.1.2.3.3.2" xref="S4.SS2.p4.6.m6.1.2.3.3.1.cmml"><mo stretchy="false" id="S4.SS2.p4.6.m6.1.2.3.3.2.1" xref="S4.SS2.p4.6.m6.1.2.3.3.1.1.cmml">[</mo><mi id="S4.SS2.p4.6.m6.1.1" xref="S4.SS2.p4.6.m6.1.1.cmml">𝕀</mi><mo stretchy="false" id="S4.SS2.p4.6.m6.1.2.3.3.2.2" xref="S4.SS2.p4.6.m6.1.2.3.3.1.1.cmml">]</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p4.6.m6.1b"><apply id="S4.SS2.p4.6.m6.1.2.cmml" xref="S4.SS2.p4.6.m6.1.2"><eq id="S4.SS2.p4.6.m6.1.2.1.cmml" xref="S4.SS2.p4.6.m6.1.2.1"></eq><apply id="S4.SS2.p4.6.m6.1.2.2.cmml" xref="S4.SS2.p4.6.m6.1.2.2"><csymbol cd="ambiguous" id="S4.SS2.p4.6.m6.1.2.2.1.cmml" xref="S4.SS2.p4.6.m6.1.2.2">superscript</csymbol><ci id="S4.SS2.p4.6.m6.1.2.2.2.cmml" xref="S4.SS2.p4.6.m6.1.2.2.2">𝑹</ci><apply id="S4.SS2.p4.6.m6.1.2.2.3.cmml" xref="S4.SS2.p4.6.m6.1.2.2.3"><minus id="S4.SS2.p4.6.m6.1.2.2.3.1.cmml" xref="S4.SS2.p4.6.m6.1.2.2.3.1"></minus><ci id="S4.SS2.p4.6.m6.1.2.2.3.2.cmml" xref="S4.SS2.p4.6.m6.1.2.2.3.2">𝑙</ci><cn type="integer" id="S4.SS2.p4.6.m6.1.2.2.3.3.cmml" xref="S4.SS2.p4.6.m6.1.2.2.3.3">1</cn></apply></apply><apply id="S4.SS2.p4.6.m6.1.2.3.cmml" xref="S4.SS2.p4.6.m6.1.2.3"><times id="S4.SS2.p4.6.m6.1.2.3.1.cmml" xref="S4.SS2.p4.6.m6.1.2.3.1"></times><apply id="S4.SS2.p4.6.m6.1.2.3.2.cmml" xref="S4.SS2.p4.6.m6.1.2.3.2"><csymbol cd="ambiguous" id="S4.SS2.p4.6.m6.1.2.3.2.1.cmml" xref="S4.SS2.p4.6.m6.1.2.3.2">superscript</csymbol><ci id="S4.SS2.p4.6.m6.1.2.3.2.2.cmml" xref="S4.SS2.p4.6.m6.1.2.3.2.2">𝑿</ci><apply id="S4.SS2.p4.6.m6.1.2.3.2.3.cmml" xref="S4.SS2.p4.6.m6.1.2.3.2.3"><minus id="S4.SS2.p4.6.m6.1.2.3.2.3.1.cmml" xref="S4.SS2.p4.6.m6.1.2.3.2.3.1"></minus><ci id="S4.SS2.p4.6.m6.1.2.3.2.3.2.cmml" xref="S4.SS2.p4.6.m6.1.2.3.2.3.2">𝑙</ci><cn type="integer" id="S4.SS2.p4.6.m6.1.2.3.2.3.3.cmml" xref="S4.SS2.p4.6.m6.1.2.3.2.3.3">1</cn></apply></apply><apply id="S4.SS2.p4.6.m6.1.2.3.3.1.cmml" xref="S4.SS2.p4.6.m6.1.2.3.3.2"><csymbol cd="latexml" id="S4.SS2.p4.6.m6.1.2.3.3.1.1.cmml" xref="S4.SS2.p4.6.m6.1.2.3.3.2.1">delimited-[]</csymbol><ci id="S4.SS2.p4.6.m6.1.1.cmml" xref="S4.SS2.p4.6.m6.1.1">𝕀</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p4.6.m6.1c">{\bm{R}}^{l-1}={\bm{X}}^{l-1}[{\mathbb{I}}]</annotation></semantics></math>, then restoring to our compressed caches by
<math id="S4.SS2.p4.7.m7.1" class="ltx_Math" alttext="\hat{{\bm{X}}}^{l}[{\mathbb{I}}]={\bm{R}}^{l}" display="inline"><semantics id="S4.SS2.p4.7.m7.1a"><mrow id="S4.SS2.p4.7.m7.1.2" xref="S4.SS2.p4.7.m7.1.2.cmml"><mrow id="S4.SS2.p4.7.m7.1.2.2" xref="S4.SS2.p4.7.m7.1.2.2.cmml"><msup id="S4.SS2.p4.7.m7.1.2.2.2" xref="S4.SS2.p4.7.m7.1.2.2.2.cmml"><mover accent="true" id="S4.SS2.p4.7.m7.1.2.2.2.2" xref="S4.SS2.p4.7.m7.1.2.2.2.2.cmml"><mi id="S4.SS2.p4.7.m7.1.2.2.2.2.2" xref="S4.SS2.p4.7.m7.1.2.2.2.2.2.cmml">𝑿</mi><mo id="S4.SS2.p4.7.m7.1.2.2.2.2.1" xref="S4.SS2.p4.7.m7.1.2.2.2.2.1.cmml">^</mo></mover><mi id="S4.SS2.p4.7.m7.1.2.2.2.3" xref="S4.SS2.p4.7.m7.1.2.2.2.3.cmml">l</mi></msup><mo lspace="0em" rspace="0em" id="S4.SS2.p4.7.m7.1.2.2.1" xref="S4.SS2.p4.7.m7.1.2.2.1.cmml">​</mo><mrow id="S4.SS2.p4.7.m7.1.2.2.3.2" xref="S4.SS2.p4.7.m7.1.2.2.3.1.cmml"><mo stretchy="false" id="S4.SS2.p4.7.m7.1.2.2.3.2.1" xref="S4.SS2.p4.7.m7.1.2.2.3.1.1.cmml">[</mo><mi id="S4.SS2.p4.7.m7.1.1" xref="S4.SS2.p4.7.m7.1.1.cmml">𝕀</mi><mo stretchy="false" id="S4.SS2.p4.7.m7.1.2.2.3.2.2" xref="S4.SS2.p4.7.m7.1.2.2.3.1.1.cmml">]</mo></mrow></mrow><mo id="S4.SS2.p4.7.m7.1.2.1" xref="S4.SS2.p4.7.m7.1.2.1.cmml">=</mo><msup id="S4.SS2.p4.7.m7.1.2.3" xref="S4.SS2.p4.7.m7.1.2.3.cmml"><mi id="S4.SS2.p4.7.m7.1.2.3.2" xref="S4.SS2.p4.7.m7.1.2.3.2.cmml">𝑹</mi><mi id="S4.SS2.p4.7.m7.1.2.3.3" xref="S4.SS2.p4.7.m7.1.2.3.3.cmml">l</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p4.7.m7.1b"><apply id="S4.SS2.p4.7.m7.1.2.cmml" xref="S4.SS2.p4.7.m7.1.2"><eq id="S4.SS2.p4.7.m7.1.2.1.cmml" xref="S4.SS2.p4.7.m7.1.2.1"></eq><apply id="S4.SS2.p4.7.m7.1.2.2.cmml" xref="S4.SS2.p4.7.m7.1.2.2"><times id="S4.SS2.p4.7.m7.1.2.2.1.cmml" xref="S4.SS2.p4.7.m7.1.2.2.1"></times><apply id="S4.SS2.p4.7.m7.1.2.2.2.cmml" xref="S4.SS2.p4.7.m7.1.2.2.2"><csymbol cd="ambiguous" id="S4.SS2.p4.7.m7.1.2.2.2.1.cmml" xref="S4.SS2.p4.7.m7.1.2.2.2">superscript</csymbol><apply id="S4.SS2.p4.7.m7.1.2.2.2.2.cmml" xref="S4.SS2.p4.7.m7.1.2.2.2.2"><ci id="S4.SS2.p4.7.m7.1.2.2.2.2.1.cmml" xref="S4.SS2.p4.7.m7.1.2.2.2.2.1">^</ci><ci id="S4.SS2.p4.7.m7.1.2.2.2.2.2.cmml" xref="S4.SS2.p4.7.m7.1.2.2.2.2.2">𝑿</ci></apply><ci id="S4.SS2.p4.7.m7.1.2.2.2.3.cmml" xref="S4.SS2.p4.7.m7.1.2.2.2.3">𝑙</ci></apply><apply id="S4.SS2.p4.7.m7.1.2.2.3.1.cmml" xref="S4.SS2.p4.7.m7.1.2.2.3.2"><csymbol cd="latexml" id="S4.SS2.p4.7.m7.1.2.2.3.1.1.cmml" xref="S4.SS2.p4.7.m7.1.2.2.3.2.1">delimited-[]</csymbol><ci id="S4.SS2.p4.7.m7.1.1.cmml" xref="S4.SS2.p4.7.m7.1.1">𝕀</ci></apply></apply><apply id="S4.SS2.p4.7.m7.1.2.3.cmml" xref="S4.SS2.p4.7.m7.1.2.3"><csymbol cd="ambiguous" id="S4.SS2.p4.7.m7.1.2.3.1.cmml" xref="S4.SS2.p4.7.m7.1.2.3">superscript</csymbol><ci id="S4.SS2.p4.7.m7.1.2.3.2.cmml" xref="S4.SS2.p4.7.m7.1.2.3.2">𝑹</ci><ci id="S4.SS2.p4.7.m7.1.2.3.3.cmml" xref="S4.SS2.p4.7.m7.1.2.3.3">𝑙</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p4.7.m7.1c">\hat{{\bm{X}}}^{l}[{\mathbb{I}}]={\bm{R}}^{l}</annotation></semantics></math>, <math id="S4.SS2.p4.8.m8.1" class="ltx_Math" alttext="\hat{{\bm{X}}}^{l-1}[{\mathbb{I}}]={\bm{R}}^{l-1}" display="inline"><semantics id="S4.SS2.p4.8.m8.1a"><mrow id="S4.SS2.p4.8.m8.1.2" xref="S4.SS2.p4.8.m8.1.2.cmml"><mrow id="S4.SS2.p4.8.m8.1.2.2" xref="S4.SS2.p4.8.m8.1.2.2.cmml"><msup id="S4.SS2.p4.8.m8.1.2.2.2" xref="S4.SS2.p4.8.m8.1.2.2.2.cmml"><mover accent="true" id="S4.SS2.p4.8.m8.1.2.2.2.2" xref="S4.SS2.p4.8.m8.1.2.2.2.2.cmml"><mi id="S4.SS2.p4.8.m8.1.2.2.2.2.2" xref="S4.SS2.p4.8.m8.1.2.2.2.2.2.cmml">𝑿</mi><mo id="S4.SS2.p4.8.m8.1.2.2.2.2.1" xref="S4.SS2.p4.8.m8.1.2.2.2.2.1.cmml">^</mo></mover><mrow id="S4.SS2.p4.8.m8.1.2.2.2.3" xref="S4.SS2.p4.8.m8.1.2.2.2.3.cmml"><mi id="S4.SS2.p4.8.m8.1.2.2.2.3.2" xref="S4.SS2.p4.8.m8.1.2.2.2.3.2.cmml">l</mi><mo id="S4.SS2.p4.8.m8.1.2.2.2.3.1" xref="S4.SS2.p4.8.m8.1.2.2.2.3.1.cmml">−</mo><mn id="S4.SS2.p4.8.m8.1.2.2.2.3.3" xref="S4.SS2.p4.8.m8.1.2.2.2.3.3.cmml">1</mn></mrow></msup><mo lspace="0em" rspace="0em" id="S4.SS2.p4.8.m8.1.2.2.1" xref="S4.SS2.p4.8.m8.1.2.2.1.cmml">​</mo><mrow id="S4.SS2.p4.8.m8.1.2.2.3.2" xref="S4.SS2.p4.8.m8.1.2.2.3.1.cmml"><mo stretchy="false" id="S4.SS2.p4.8.m8.1.2.2.3.2.1" xref="S4.SS2.p4.8.m8.1.2.2.3.1.1.cmml">[</mo><mi id="S4.SS2.p4.8.m8.1.1" xref="S4.SS2.p4.8.m8.1.1.cmml">𝕀</mi><mo stretchy="false" id="S4.SS2.p4.8.m8.1.2.2.3.2.2" xref="S4.SS2.p4.8.m8.1.2.2.3.1.1.cmml">]</mo></mrow></mrow><mo id="S4.SS2.p4.8.m8.1.2.1" xref="S4.SS2.p4.8.m8.1.2.1.cmml">=</mo><msup id="S4.SS2.p4.8.m8.1.2.3" xref="S4.SS2.p4.8.m8.1.2.3.cmml"><mi id="S4.SS2.p4.8.m8.1.2.3.2" xref="S4.SS2.p4.8.m8.1.2.3.2.cmml">𝑹</mi><mrow id="S4.SS2.p4.8.m8.1.2.3.3" xref="S4.SS2.p4.8.m8.1.2.3.3.cmml"><mi id="S4.SS2.p4.8.m8.1.2.3.3.2" xref="S4.SS2.p4.8.m8.1.2.3.3.2.cmml">l</mi><mo id="S4.SS2.p4.8.m8.1.2.3.3.1" xref="S4.SS2.p4.8.m8.1.2.3.3.1.cmml">−</mo><mn id="S4.SS2.p4.8.m8.1.2.3.3.3" xref="S4.SS2.p4.8.m8.1.2.3.3.3.cmml">1</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p4.8.m8.1b"><apply id="S4.SS2.p4.8.m8.1.2.cmml" xref="S4.SS2.p4.8.m8.1.2"><eq id="S4.SS2.p4.8.m8.1.2.1.cmml" xref="S4.SS2.p4.8.m8.1.2.1"></eq><apply id="S4.SS2.p4.8.m8.1.2.2.cmml" xref="S4.SS2.p4.8.m8.1.2.2"><times id="S4.SS2.p4.8.m8.1.2.2.1.cmml" xref="S4.SS2.p4.8.m8.1.2.2.1"></times><apply id="S4.SS2.p4.8.m8.1.2.2.2.cmml" xref="S4.SS2.p4.8.m8.1.2.2.2"><csymbol cd="ambiguous" id="S4.SS2.p4.8.m8.1.2.2.2.1.cmml" xref="S4.SS2.p4.8.m8.1.2.2.2">superscript</csymbol><apply id="S4.SS2.p4.8.m8.1.2.2.2.2.cmml" xref="S4.SS2.p4.8.m8.1.2.2.2.2"><ci id="S4.SS2.p4.8.m8.1.2.2.2.2.1.cmml" xref="S4.SS2.p4.8.m8.1.2.2.2.2.1">^</ci><ci id="S4.SS2.p4.8.m8.1.2.2.2.2.2.cmml" xref="S4.SS2.p4.8.m8.1.2.2.2.2.2">𝑿</ci></apply><apply id="S4.SS2.p4.8.m8.1.2.2.2.3.cmml" xref="S4.SS2.p4.8.m8.1.2.2.2.3"><minus id="S4.SS2.p4.8.m8.1.2.2.2.3.1.cmml" xref="S4.SS2.p4.8.m8.1.2.2.2.3.1"></minus><ci id="S4.SS2.p4.8.m8.1.2.2.2.3.2.cmml" xref="S4.SS2.p4.8.m8.1.2.2.2.3.2">𝑙</ci><cn type="integer" id="S4.SS2.p4.8.m8.1.2.2.2.3.3.cmml" xref="S4.SS2.p4.8.m8.1.2.2.2.3.3">1</cn></apply></apply><apply id="S4.SS2.p4.8.m8.1.2.2.3.1.cmml" xref="S4.SS2.p4.8.m8.1.2.2.3.2"><csymbol cd="latexml" id="S4.SS2.p4.8.m8.1.2.2.3.1.1.cmml" xref="S4.SS2.p4.8.m8.1.2.2.3.2.1">delimited-[]</csymbol><ci id="S4.SS2.p4.8.m8.1.1.cmml" xref="S4.SS2.p4.8.m8.1.1">𝕀</ci></apply></apply><apply id="S4.SS2.p4.8.m8.1.2.3.cmml" xref="S4.SS2.p4.8.m8.1.2.3"><csymbol cd="ambiguous" id="S4.SS2.p4.8.m8.1.2.3.1.cmml" xref="S4.SS2.p4.8.m8.1.2.3">superscript</csymbol><ci id="S4.SS2.p4.8.m8.1.2.3.2.cmml" xref="S4.SS2.p4.8.m8.1.2.3.2">𝑹</ci><apply id="S4.SS2.p4.8.m8.1.2.3.3.cmml" xref="S4.SS2.p4.8.m8.1.2.3.3"><minus id="S4.SS2.p4.8.m8.1.2.3.3.1.cmml" xref="S4.SS2.p4.8.m8.1.2.3.3.1"></minus><ci id="S4.SS2.p4.8.m8.1.2.3.3.2.cmml" xref="S4.SS2.p4.8.m8.1.2.3.3.2">𝑙</ci><cn type="integer" id="S4.SS2.p4.8.m8.1.2.3.3.3.cmml" xref="S4.SS2.p4.8.m8.1.2.3.3.3">1</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p4.8.m8.1c">\hat{{\bm{X}}}^{l-1}[{\mathbb{I}}]={\bm{R}}^{l-1}</annotation></semantics></math>, as shown in <a href="#S4.F3" title="In 4.1 Cross-Layer Compression ‣ 4 Method ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">3</span></a>(b).
Overall, we share the final cache for the two layers as <math id="S4.SS2.p4.9.m9.10" class="ltx_Math" alttext="{\bm{C}}^{l,l-1}=[{\bm{E}}^{l,l-1},{\bm{R}}^{l},{\bm{R}}^{l-1},{\lVert{\bm{X}}^{l-1}\rVert},{\lVert{\bm{X}}^{l}\rVert},{\mathbb{I}}]" display="inline"><semantics id="S4.SS2.p4.9.m9.10a"><mrow id="S4.SS2.p4.9.m9.10.10" xref="S4.SS2.p4.9.m9.10.10.cmml"><msup id="S4.SS2.p4.9.m9.10.10.7" xref="S4.SS2.p4.9.m9.10.10.7.cmml"><mi id="S4.SS2.p4.9.m9.10.10.7.2" xref="S4.SS2.p4.9.m9.10.10.7.2.cmml">𝑪</mi><mrow id="S4.SS2.p4.9.m9.2.2.2.2" xref="S4.SS2.p4.9.m9.2.2.2.3.cmml"><mi id="S4.SS2.p4.9.m9.1.1.1.1" xref="S4.SS2.p4.9.m9.1.1.1.1.cmml">l</mi><mo id="S4.SS2.p4.9.m9.2.2.2.2.2" xref="S4.SS2.p4.9.m9.2.2.2.3.cmml">,</mo><mrow id="S4.SS2.p4.9.m9.2.2.2.2.1" xref="S4.SS2.p4.9.m9.2.2.2.2.1.cmml"><mi id="S4.SS2.p4.9.m9.2.2.2.2.1.2" xref="S4.SS2.p4.9.m9.2.2.2.2.1.2.cmml">l</mi><mo id="S4.SS2.p4.9.m9.2.2.2.2.1.1" xref="S4.SS2.p4.9.m9.2.2.2.2.1.1.cmml">−</mo><mn id="S4.SS2.p4.9.m9.2.2.2.2.1.3" xref="S4.SS2.p4.9.m9.2.2.2.2.1.3.cmml">1</mn></mrow></mrow></msup><mo id="S4.SS2.p4.9.m9.10.10.6" xref="S4.SS2.p4.9.m9.10.10.6.cmml">=</mo><mrow id="S4.SS2.p4.9.m9.10.10.5.5" xref="S4.SS2.p4.9.m9.10.10.5.6.cmml"><mo stretchy="false" id="S4.SS2.p4.9.m9.10.10.5.5.6" xref="S4.SS2.p4.9.m9.10.10.5.6.cmml">[</mo><msup id="S4.SS2.p4.9.m9.6.6.1.1.1" xref="S4.SS2.p4.9.m9.6.6.1.1.1.cmml"><mi id="S4.SS2.p4.9.m9.6.6.1.1.1.2" xref="S4.SS2.p4.9.m9.6.6.1.1.1.2.cmml">𝑬</mi><mrow id="S4.SS2.p4.9.m9.4.4.2.2" xref="S4.SS2.p4.9.m9.4.4.2.3.cmml"><mi id="S4.SS2.p4.9.m9.3.3.1.1" xref="S4.SS2.p4.9.m9.3.3.1.1.cmml">l</mi><mo id="S4.SS2.p4.9.m9.4.4.2.2.2" xref="S4.SS2.p4.9.m9.4.4.2.3.cmml">,</mo><mrow id="S4.SS2.p4.9.m9.4.4.2.2.1" xref="S4.SS2.p4.9.m9.4.4.2.2.1.cmml"><mi id="S4.SS2.p4.9.m9.4.4.2.2.1.2" xref="S4.SS2.p4.9.m9.4.4.2.2.1.2.cmml">l</mi><mo id="S4.SS2.p4.9.m9.4.4.2.2.1.1" xref="S4.SS2.p4.9.m9.4.4.2.2.1.1.cmml">−</mo><mn id="S4.SS2.p4.9.m9.4.4.2.2.1.3" xref="S4.SS2.p4.9.m9.4.4.2.2.1.3.cmml">1</mn></mrow></mrow></msup><mo id="S4.SS2.p4.9.m9.10.10.5.5.7" xref="S4.SS2.p4.9.m9.10.10.5.6.cmml">,</mo><msup id="S4.SS2.p4.9.m9.7.7.2.2.2" xref="S4.SS2.p4.9.m9.7.7.2.2.2.cmml"><mi id="S4.SS2.p4.9.m9.7.7.2.2.2.2" xref="S4.SS2.p4.9.m9.7.7.2.2.2.2.cmml">𝑹</mi><mi id="S4.SS2.p4.9.m9.7.7.2.2.2.3" xref="S4.SS2.p4.9.m9.7.7.2.2.2.3.cmml">l</mi></msup><mo id="S4.SS2.p4.9.m9.10.10.5.5.8" xref="S4.SS2.p4.9.m9.10.10.5.6.cmml">,</mo><msup id="S4.SS2.p4.9.m9.8.8.3.3.3" xref="S4.SS2.p4.9.m9.8.8.3.3.3.cmml"><mi id="S4.SS2.p4.9.m9.8.8.3.3.3.2" xref="S4.SS2.p4.9.m9.8.8.3.3.3.2.cmml">𝑹</mi><mrow id="S4.SS2.p4.9.m9.8.8.3.3.3.3" xref="S4.SS2.p4.9.m9.8.8.3.3.3.3.cmml"><mi id="S4.SS2.p4.9.m9.8.8.3.3.3.3.2" xref="S4.SS2.p4.9.m9.8.8.3.3.3.3.2.cmml">l</mi><mo id="S4.SS2.p4.9.m9.8.8.3.3.3.3.1" xref="S4.SS2.p4.9.m9.8.8.3.3.3.3.1.cmml">−</mo><mn id="S4.SS2.p4.9.m9.8.8.3.3.3.3.3" xref="S4.SS2.p4.9.m9.8.8.3.3.3.3.3.cmml">1</mn></mrow></msup><mo id="S4.SS2.p4.9.m9.10.10.5.5.9" xref="S4.SS2.p4.9.m9.10.10.5.6.cmml">,</mo><mrow id="S4.SS2.p4.9.m9.9.9.4.4.4.1" xref="S4.SS2.p4.9.m9.9.9.4.4.4.2.cmml"><mo fence="true" lspace="0em" rspace="0em" id="S4.SS2.p4.9.m9.9.9.4.4.4.1.2" xref="S4.SS2.p4.9.m9.9.9.4.4.4.2.1.cmml">∥</mo><msup id="S4.SS2.p4.9.m9.9.9.4.4.4.1.1" xref="S4.SS2.p4.9.m9.9.9.4.4.4.1.1.cmml"><mi id="S4.SS2.p4.9.m9.9.9.4.4.4.1.1.2" xref="S4.SS2.p4.9.m9.9.9.4.4.4.1.1.2.cmml">𝑿</mi><mrow id="S4.SS2.p4.9.m9.9.9.4.4.4.1.1.3" xref="S4.SS2.p4.9.m9.9.9.4.4.4.1.1.3.cmml"><mi id="S4.SS2.p4.9.m9.9.9.4.4.4.1.1.3.2" xref="S4.SS2.p4.9.m9.9.9.4.4.4.1.1.3.2.cmml">l</mi><mo id="S4.SS2.p4.9.m9.9.9.4.4.4.1.1.3.1" xref="S4.SS2.p4.9.m9.9.9.4.4.4.1.1.3.1.cmml">−</mo><mn id="S4.SS2.p4.9.m9.9.9.4.4.4.1.1.3.3" xref="S4.SS2.p4.9.m9.9.9.4.4.4.1.1.3.3.cmml">1</mn></mrow></msup><mo fence="true" lspace="0em" rspace="0em" id="S4.SS2.p4.9.m9.9.9.4.4.4.1.3" xref="S4.SS2.p4.9.m9.9.9.4.4.4.2.1.cmml">∥</mo></mrow><mo id="S4.SS2.p4.9.m9.10.10.5.5.10" xref="S4.SS2.p4.9.m9.10.10.5.6.cmml">,</mo><mrow id="S4.SS2.p4.9.m9.10.10.5.5.5.1" xref="S4.SS2.p4.9.m9.10.10.5.5.5.2.cmml"><mo fence="true" lspace="0em" rspace="0em" id="S4.SS2.p4.9.m9.10.10.5.5.5.1.2" xref="S4.SS2.p4.9.m9.10.10.5.5.5.2.1.cmml">∥</mo><msup id="S4.SS2.p4.9.m9.10.10.5.5.5.1.1" xref="S4.SS2.p4.9.m9.10.10.5.5.5.1.1.cmml"><mi id="S4.SS2.p4.9.m9.10.10.5.5.5.1.1.2" xref="S4.SS2.p4.9.m9.10.10.5.5.5.1.1.2.cmml">𝑿</mi><mi id="S4.SS2.p4.9.m9.10.10.5.5.5.1.1.3" xref="S4.SS2.p4.9.m9.10.10.5.5.5.1.1.3.cmml">l</mi></msup><mo fence="true" lspace="0em" rspace="0em" id="S4.SS2.p4.9.m9.10.10.5.5.5.1.3" xref="S4.SS2.p4.9.m9.10.10.5.5.5.2.1.cmml">∥</mo></mrow><mo id="S4.SS2.p4.9.m9.10.10.5.5.11" xref="S4.SS2.p4.9.m9.10.10.5.6.cmml">,</mo><mi id="S4.SS2.p4.9.m9.5.5" xref="S4.SS2.p4.9.m9.5.5.cmml">𝕀</mi><mo stretchy="false" id="S4.SS2.p4.9.m9.10.10.5.5.12" xref="S4.SS2.p4.9.m9.10.10.5.6.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p4.9.m9.10b"><apply id="S4.SS2.p4.9.m9.10.10.cmml" xref="S4.SS2.p4.9.m9.10.10"><eq id="S4.SS2.p4.9.m9.10.10.6.cmml" xref="S4.SS2.p4.9.m9.10.10.6"></eq><apply id="S4.SS2.p4.9.m9.10.10.7.cmml" xref="S4.SS2.p4.9.m9.10.10.7"><csymbol cd="ambiguous" id="S4.SS2.p4.9.m9.10.10.7.1.cmml" xref="S4.SS2.p4.9.m9.10.10.7">superscript</csymbol><ci id="S4.SS2.p4.9.m9.10.10.7.2.cmml" xref="S4.SS2.p4.9.m9.10.10.7.2">𝑪</ci><list id="S4.SS2.p4.9.m9.2.2.2.3.cmml" xref="S4.SS2.p4.9.m9.2.2.2.2"><ci id="S4.SS2.p4.9.m9.1.1.1.1.cmml" xref="S4.SS2.p4.9.m9.1.1.1.1">𝑙</ci><apply id="S4.SS2.p4.9.m9.2.2.2.2.1.cmml" xref="S4.SS2.p4.9.m9.2.2.2.2.1"><minus id="S4.SS2.p4.9.m9.2.2.2.2.1.1.cmml" xref="S4.SS2.p4.9.m9.2.2.2.2.1.1"></minus><ci id="S4.SS2.p4.9.m9.2.2.2.2.1.2.cmml" xref="S4.SS2.p4.9.m9.2.2.2.2.1.2">𝑙</ci><cn type="integer" id="S4.SS2.p4.9.m9.2.2.2.2.1.3.cmml" xref="S4.SS2.p4.9.m9.2.2.2.2.1.3">1</cn></apply></list></apply><list id="S4.SS2.p4.9.m9.10.10.5.6.cmml" xref="S4.SS2.p4.9.m9.10.10.5.5"><apply id="S4.SS2.p4.9.m9.6.6.1.1.1.cmml" xref="S4.SS2.p4.9.m9.6.6.1.1.1"><csymbol cd="ambiguous" id="S4.SS2.p4.9.m9.6.6.1.1.1.1.cmml" xref="S4.SS2.p4.9.m9.6.6.1.1.1">superscript</csymbol><ci id="S4.SS2.p4.9.m9.6.6.1.1.1.2.cmml" xref="S4.SS2.p4.9.m9.6.6.1.1.1.2">𝑬</ci><list id="S4.SS2.p4.9.m9.4.4.2.3.cmml" xref="S4.SS2.p4.9.m9.4.4.2.2"><ci id="S4.SS2.p4.9.m9.3.3.1.1.cmml" xref="S4.SS2.p4.9.m9.3.3.1.1">𝑙</ci><apply id="S4.SS2.p4.9.m9.4.4.2.2.1.cmml" xref="S4.SS2.p4.9.m9.4.4.2.2.1"><minus id="S4.SS2.p4.9.m9.4.4.2.2.1.1.cmml" xref="S4.SS2.p4.9.m9.4.4.2.2.1.1"></minus><ci id="S4.SS2.p4.9.m9.4.4.2.2.1.2.cmml" xref="S4.SS2.p4.9.m9.4.4.2.2.1.2">𝑙</ci><cn type="integer" id="S4.SS2.p4.9.m9.4.4.2.2.1.3.cmml" xref="S4.SS2.p4.9.m9.4.4.2.2.1.3">1</cn></apply></list></apply><apply id="S4.SS2.p4.9.m9.7.7.2.2.2.cmml" xref="S4.SS2.p4.9.m9.7.7.2.2.2"><csymbol cd="ambiguous" id="S4.SS2.p4.9.m9.7.7.2.2.2.1.cmml" xref="S4.SS2.p4.9.m9.7.7.2.2.2">superscript</csymbol><ci id="S4.SS2.p4.9.m9.7.7.2.2.2.2.cmml" xref="S4.SS2.p4.9.m9.7.7.2.2.2.2">𝑹</ci><ci id="S4.SS2.p4.9.m9.7.7.2.2.2.3.cmml" xref="S4.SS2.p4.9.m9.7.7.2.2.2.3">𝑙</ci></apply><apply id="S4.SS2.p4.9.m9.8.8.3.3.3.cmml" xref="S4.SS2.p4.9.m9.8.8.3.3.3"><csymbol cd="ambiguous" id="S4.SS2.p4.9.m9.8.8.3.3.3.1.cmml" xref="S4.SS2.p4.9.m9.8.8.3.3.3">superscript</csymbol><ci id="S4.SS2.p4.9.m9.8.8.3.3.3.2.cmml" xref="S4.SS2.p4.9.m9.8.8.3.3.3.2">𝑹</ci><apply id="S4.SS2.p4.9.m9.8.8.3.3.3.3.cmml" xref="S4.SS2.p4.9.m9.8.8.3.3.3.3"><minus id="S4.SS2.p4.9.m9.8.8.3.3.3.3.1.cmml" xref="S4.SS2.p4.9.m9.8.8.3.3.3.3.1"></minus><ci id="S4.SS2.p4.9.m9.8.8.3.3.3.3.2.cmml" xref="S4.SS2.p4.9.m9.8.8.3.3.3.3.2">𝑙</ci><cn type="integer" id="S4.SS2.p4.9.m9.8.8.3.3.3.3.3.cmml" xref="S4.SS2.p4.9.m9.8.8.3.3.3.3.3">1</cn></apply></apply><apply id="S4.SS2.p4.9.m9.9.9.4.4.4.2.cmml" xref="S4.SS2.p4.9.m9.9.9.4.4.4.1"><csymbol cd="latexml" id="S4.SS2.p4.9.m9.9.9.4.4.4.2.1.cmml" xref="S4.SS2.p4.9.m9.9.9.4.4.4.1.2">delimited-∥∥</csymbol><apply id="S4.SS2.p4.9.m9.9.9.4.4.4.1.1.cmml" xref="S4.SS2.p4.9.m9.9.9.4.4.4.1.1"><csymbol cd="ambiguous" id="S4.SS2.p4.9.m9.9.9.4.4.4.1.1.1.cmml" xref="S4.SS2.p4.9.m9.9.9.4.4.4.1.1">superscript</csymbol><ci id="S4.SS2.p4.9.m9.9.9.4.4.4.1.1.2.cmml" xref="S4.SS2.p4.9.m9.9.9.4.4.4.1.1.2">𝑿</ci><apply id="S4.SS2.p4.9.m9.9.9.4.4.4.1.1.3.cmml" xref="S4.SS2.p4.9.m9.9.9.4.4.4.1.1.3"><minus id="S4.SS2.p4.9.m9.9.9.4.4.4.1.1.3.1.cmml" xref="S4.SS2.p4.9.m9.9.9.4.4.4.1.1.3.1"></minus><ci id="S4.SS2.p4.9.m9.9.9.4.4.4.1.1.3.2.cmml" xref="S4.SS2.p4.9.m9.9.9.4.4.4.1.1.3.2">𝑙</ci><cn type="integer" id="S4.SS2.p4.9.m9.9.9.4.4.4.1.1.3.3.cmml" xref="S4.SS2.p4.9.m9.9.9.4.4.4.1.1.3.3">1</cn></apply></apply></apply><apply id="S4.SS2.p4.9.m9.10.10.5.5.5.2.cmml" xref="S4.SS2.p4.9.m9.10.10.5.5.5.1"><csymbol cd="latexml" id="S4.SS2.p4.9.m9.10.10.5.5.5.2.1.cmml" xref="S4.SS2.p4.9.m9.10.10.5.5.5.1.2">delimited-∥∥</csymbol><apply id="S4.SS2.p4.9.m9.10.10.5.5.5.1.1.cmml" xref="S4.SS2.p4.9.m9.10.10.5.5.5.1.1"><csymbol cd="ambiguous" id="S4.SS2.p4.9.m9.10.10.5.5.5.1.1.1.cmml" xref="S4.SS2.p4.9.m9.10.10.5.5.5.1.1">superscript</csymbol><ci id="S4.SS2.p4.9.m9.10.10.5.5.5.1.1.2.cmml" xref="S4.SS2.p4.9.m9.10.10.5.5.5.1.1.2">𝑿</ci><ci id="S4.SS2.p4.9.m9.10.10.5.5.5.1.1.3.cmml" xref="S4.SS2.p4.9.m9.10.10.5.5.5.1.1.3">𝑙</ci></apply></apply><ci id="S4.SS2.p4.9.m9.5.5.cmml" xref="S4.SS2.p4.9.m9.5.5">𝕀</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p4.9.m9.10c">{\bm{C}}^{l,l-1}=[{\bm{E}}^{l,l-1},{\bm{R}}^{l},{\bm{R}}^{l-1},{\lVert{\bm{X}}^{l-1}\rVert},{\lVert{\bm{X}}^{l}\rVert},{\mathbb{I}}]</annotation></semantics></math>. This cache includes the shared KV cache states, retention of unmerged tokens, magnitude vectors for each layer, and the token-keeping index, respectively.
These additional components are quite lightweight, thus compared to full-layer caches, our method remains memory-efficient, as discussed in Sec. <a href="#S4.SS3" title="4.3 Efficiency Discussion ‣ 4 Method ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.3</span></a>.</p>
</div>
<div id="S4.SS2.p5" class="ltx_para">
<p id="S4.SS2.p5.3" class="ltx_p"><span id="S4.SS2.p5.3.1" class="ltx_text ltx_font_bold">Cache restoration.</span> After obtaining the shared cache <math id="S4.SS2.p5.1.m1.2" class="ltx_Math" alttext="{\bm{C}}^{l,l-1}" display="inline"><semantics id="S4.SS2.p5.1.m1.2a"><msup id="S4.SS2.p5.1.m1.2.3" xref="S4.SS2.p5.1.m1.2.3.cmml"><mi id="S4.SS2.p5.1.m1.2.3.2" xref="S4.SS2.p5.1.m1.2.3.2.cmml">𝑪</mi><mrow id="S4.SS2.p5.1.m1.2.2.2.2" xref="S4.SS2.p5.1.m1.2.2.2.3.cmml"><mi id="S4.SS2.p5.1.m1.1.1.1.1" xref="S4.SS2.p5.1.m1.1.1.1.1.cmml">l</mi><mo id="S4.SS2.p5.1.m1.2.2.2.2.2" xref="S4.SS2.p5.1.m1.2.2.2.3.cmml">,</mo><mrow id="S4.SS2.p5.1.m1.2.2.2.2.1" xref="S4.SS2.p5.1.m1.2.2.2.2.1.cmml"><mi id="S4.SS2.p5.1.m1.2.2.2.2.1.2" xref="S4.SS2.p5.1.m1.2.2.2.2.1.2.cmml">l</mi><mo id="S4.SS2.p5.1.m1.2.2.2.2.1.1" xref="S4.SS2.p5.1.m1.2.2.2.2.1.1.cmml">−</mo><mn id="S4.SS2.p5.1.m1.2.2.2.2.1.3" xref="S4.SS2.p5.1.m1.2.2.2.2.1.3.cmml">1</mn></mrow></mrow></msup><annotation-xml encoding="MathML-Content" id="S4.SS2.p5.1.m1.2b"><apply id="S4.SS2.p5.1.m1.2.3.cmml" xref="S4.SS2.p5.1.m1.2.3"><csymbol cd="ambiguous" id="S4.SS2.p5.1.m1.2.3.1.cmml" xref="S4.SS2.p5.1.m1.2.3">superscript</csymbol><ci id="S4.SS2.p5.1.m1.2.3.2.cmml" xref="S4.SS2.p5.1.m1.2.3.2">𝑪</ci><list id="S4.SS2.p5.1.m1.2.2.2.3.cmml" xref="S4.SS2.p5.1.m1.2.2.2.2"><ci id="S4.SS2.p5.1.m1.1.1.1.1.cmml" xref="S4.SS2.p5.1.m1.1.1.1.1">𝑙</ci><apply id="S4.SS2.p5.1.m1.2.2.2.2.1.cmml" xref="S4.SS2.p5.1.m1.2.2.2.2.1"><minus id="S4.SS2.p5.1.m1.2.2.2.2.1.1.cmml" xref="S4.SS2.p5.1.m1.2.2.2.2.1.1"></minus><ci id="S4.SS2.p5.1.m1.2.2.2.2.1.2.cmml" xref="S4.SS2.p5.1.m1.2.2.2.2.1.2">𝑙</ci><cn type="integer" id="S4.SS2.p5.1.m1.2.2.2.2.1.3.cmml" xref="S4.SS2.p5.1.m1.2.2.2.2.1.3">1</cn></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p5.1.m1.2c">{\bm{C}}^{l,l-1}</annotation></semantics></math>, we further need to approximately restore the original cache states for the current token decoding, as shown in Fig. <a href="#S4.F3" title="Figure 3 ‣ 4.1 Cross-Layer Compression ‣ 4 Method ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>(b). Specifically, to restore <math id="S4.SS2.p5.2.m2.1" class="ltx_Math" alttext="{\bm{X}}^{l}" display="inline"><semantics id="S4.SS2.p5.2.m2.1a"><msup id="S4.SS2.p5.2.m2.1.1" xref="S4.SS2.p5.2.m2.1.1.cmml"><mi id="S4.SS2.p5.2.m2.1.1.2" xref="S4.SS2.p5.2.m2.1.1.2.cmml">𝑿</mi><mi id="S4.SS2.p5.2.m2.1.1.3" xref="S4.SS2.p5.2.m2.1.1.3.cmml">l</mi></msup><annotation-xml encoding="MathML-Content" id="S4.SS2.p5.2.m2.1b"><apply id="S4.SS2.p5.2.m2.1.1.cmml" xref="S4.SS2.p5.2.m2.1.1"><csymbol cd="ambiguous" id="S4.SS2.p5.2.m2.1.1.1.cmml" xref="S4.SS2.p5.2.m2.1.1">superscript</csymbol><ci id="S4.SS2.p5.2.m2.1.1.2.cmml" xref="S4.SS2.p5.2.m2.1.1.2">𝑿</ci><ci id="S4.SS2.p5.2.m2.1.1.3.cmml" xref="S4.SS2.p5.2.m2.1.1.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p5.2.m2.1c">{\bm{X}}^{l}</annotation></semantics></math>, we first rescale the directional shared states with the corresponding magnitude vector along the token dimension, denoted as <math id="S4.SS2.p5.3.m3.3" class="ltx_Math" alttext="{\bm{E}}^{l,l-1}{\lVert{\bm{X}}^{l}\rVert}" display="inline"><semantics id="S4.SS2.p5.3.m3.3a"><mrow id="S4.SS2.p5.3.m3.3.3" xref="S4.SS2.p5.3.m3.3.3.cmml"><msup id="S4.SS2.p5.3.m3.3.3.3" xref="S4.SS2.p5.3.m3.3.3.3.cmml"><mi id="S4.SS2.p5.3.m3.3.3.3.2" xref="S4.SS2.p5.3.m3.3.3.3.2.cmml">𝑬</mi><mrow id="S4.SS2.p5.3.m3.2.2.2.2" xref="S4.SS2.p5.3.m3.2.2.2.3.cmml"><mi id="S4.SS2.p5.3.m3.1.1.1.1" xref="S4.SS2.p5.3.m3.1.1.1.1.cmml">l</mi><mo id="S4.SS2.p5.3.m3.2.2.2.2.2" xref="S4.SS2.p5.3.m3.2.2.2.3.cmml">,</mo><mrow id="S4.SS2.p5.3.m3.2.2.2.2.1" xref="S4.SS2.p5.3.m3.2.2.2.2.1.cmml"><mi id="S4.SS2.p5.3.m3.2.2.2.2.1.2" xref="S4.SS2.p5.3.m3.2.2.2.2.1.2.cmml">l</mi><mo id="S4.SS2.p5.3.m3.2.2.2.2.1.1" xref="S4.SS2.p5.3.m3.2.2.2.2.1.1.cmml">−</mo><mn id="S4.SS2.p5.3.m3.2.2.2.2.1.3" xref="S4.SS2.p5.3.m3.2.2.2.2.1.3.cmml">1</mn></mrow></mrow></msup><mo lspace="0em" rspace="0em" id="S4.SS2.p5.3.m3.3.3.2" xref="S4.SS2.p5.3.m3.3.3.2.cmml">​</mo><mrow id="S4.SS2.p5.3.m3.3.3.1.1" xref="S4.SS2.p5.3.m3.3.3.1.2.cmml"><mo fence="true" rspace="0em" id="S4.SS2.p5.3.m3.3.3.1.1.2" xref="S4.SS2.p5.3.m3.3.3.1.2.1.cmml">∥</mo><msup id="S4.SS2.p5.3.m3.3.3.1.1.1" xref="S4.SS2.p5.3.m3.3.3.1.1.1.cmml"><mi id="S4.SS2.p5.3.m3.3.3.1.1.1.2" xref="S4.SS2.p5.3.m3.3.3.1.1.1.2.cmml">𝑿</mi><mi id="S4.SS2.p5.3.m3.3.3.1.1.1.3" xref="S4.SS2.p5.3.m3.3.3.1.1.1.3.cmml">l</mi></msup><mo fence="true" lspace="0em" id="S4.SS2.p5.3.m3.3.3.1.1.3" xref="S4.SS2.p5.3.m3.3.3.1.2.1.cmml">∥</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p5.3.m3.3b"><apply id="S4.SS2.p5.3.m3.3.3.cmml" xref="S4.SS2.p5.3.m3.3.3"><times id="S4.SS2.p5.3.m3.3.3.2.cmml" xref="S4.SS2.p5.3.m3.3.3.2"></times><apply id="S4.SS2.p5.3.m3.3.3.3.cmml" xref="S4.SS2.p5.3.m3.3.3.3"><csymbol cd="ambiguous" id="S4.SS2.p5.3.m3.3.3.3.1.cmml" xref="S4.SS2.p5.3.m3.3.3.3">superscript</csymbol><ci id="S4.SS2.p5.3.m3.3.3.3.2.cmml" xref="S4.SS2.p5.3.m3.3.3.3.2">𝑬</ci><list id="S4.SS2.p5.3.m3.2.2.2.3.cmml" xref="S4.SS2.p5.3.m3.2.2.2.2"><ci id="S4.SS2.p5.3.m3.1.1.1.1.cmml" xref="S4.SS2.p5.3.m3.1.1.1.1">𝑙</ci><apply id="S4.SS2.p5.3.m3.2.2.2.2.1.cmml" xref="S4.SS2.p5.3.m3.2.2.2.2.1"><minus id="S4.SS2.p5.3.m3.2.2.2.2.1.1.cmml" xref="S4.SS2.p5.3.m3.2.2.2.2.1.1"></minus><ci id="S4.SS2.p5.3.m3.2.2.2.2.1.2.cmml" xref="S4.SS2.p5.3.m3.2.2.2.2.1.2">𝑙</ci><cn type="integer" id="S4.SS2.p5.3.m3.2.2.2.2.1.3.cmml" xref="S4.SS2.p5.3.m3.2.2.2.2.1.3">1</cn></apply></list></apply><apply id="S4.SS2.p5.3.m3.3.3.1.2.cmml" xref="S4.SS2.p5.3.m3.3.3.1.1"><csymbol cd="latexml" id="S4.SS2.p5.3.m3.3.3.1.2.1.cmml" xref="S4.SS2.p5.3.m3.3.3.1.1.2">delimited-∥∥</csymbol><apply id="S4.SS2.p5.3.m3.3.3.1.1.1.cmml" xref="S4.SS2.p5.3.m3.3.3.1.1.1"><csymbol cd="ambiguous" id="S4.SS2.p5.3.m3.3.3.1.1.1.1.cmml" xref="S4.SS2.p5.3.m3.3.3.1.1.1">superscript</csymbol><ci id="S4.SS2.p5.3.m3.3.3.1.1.1.2.cmml" xref="S4.SS2.p5.3.m3.3.3.1.1.1.2">𝑿</ci><ci id="S4.SS2.p5.3.m3.3.3.1.1.1.3.cmml" xref="S4.SS2.p5.3.m3.3.3.1.1.1.3">𝑙</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p5.3.m3.3c">{\bm{E}}^{l,l-1}{\lVert{\bm{X}}^{l}\rVert}</annotation></semantics></math>. Subsequently, we perform retention token recovery by placing the sensitive tokens according to their token indices.</p>
</div>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Efficiency Discussion</h3>

<div id="S4.SS3.p1" class="ltx_para">
<p id="S4.SS3.p1.6" class="ltx_p"><span id="S4.SS3.p1.6.1" class="ltx_text ltx_font_bold">Compression efficiency.</span> We primarily analyze our memory efficiency in terms of the number of tokens used. Next, let <math id="S4.SS3.p1.1.m1.1" class="ltx_Math" alttext="r" display="inline"><semantics id="S4.SS3.p1.1.m1.1a"><mi id="S4.SS3.p1.1.m1.1.1" xref="S4.SS3.p1.1.m1.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.1.m1.1b"><ci id="S4.SS3.p1.1.m1.1.1.cmml" xref="S4.SS3.p1.1.m1.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.1.m1.1c">r</annotation></semantics></math> be the number of layers and
and <math id="S4.SS3.p1.2.m2.1" class="ltx_Math" alttext="b" display="inline"><semantics id="S4.SS3.p1.2.m2.1a"><mi id="S4.SS3.p1.2.m2.1.1" xref="S4.SS3.p1.2.m2.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.2.m2.1b"><ci id="S4.SS3.p1.2.m2.1.1.cmml" xref="S4.SS3.p1.2.m2.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.2.m2.1c">b</annotation></semantics></math> is the batch size, <math id="S4.SS3.p1.3.m3.1" class="ltx_Math" alttext="s" display="inline"><semantics id="S4.SS3.p1.3.m3.1a"><mi id="S4.SS3.p1.3.m3.1.1" xref="S4.SS3.p1.3.m3.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.3.m3.1b"><ci id="S4.SS3.p1.3.m3.1.1.cmml" xref="S4.SS3.p1.3.m3.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.3.m3.1c">s</annotation></semantics></math> and <math id="S4.SS3.p1.4.m4.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S4.SS3.p1.4.m4.1a"><mi id="S4.SS3.p1.4.m4.1.1" xref="S4.SS3.p1.4.m4.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.4.m4.1b"><ci id="S4.SS3.p1.4.m4.1.1.cmml" xref="S4.SS3.p1.4.m4.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.4.m4.1c">n</annotation></semantics></math> are input and output sequence length respectively. We consider the FP16 storage for KV cache. The full cache memory usage is given by
<math id="S4.SS3.p1.5.m5.1" class="ltx_Math" alttext="4brh(s+n)" display="inline"><semantics id="S4.SS3.p1.5.m5.1a"><mrow id="S4.SS3.p1.5.m5.1.1" xref="S4.SS3.p1.5.m5.1.1.cmml"><mn id="S4.SS3.p1.5.m5.1.1.3" xref="S4.SS3.p1.5.m5.1.1.3.cmml">4</mn><mo lspace="0em" rspace="0em" id="S4.SS3.p1.5.m5.1.1.2" xref="S4.SS3.p1.5.m5.1.1.2.cmml">​</mo><mi id="S4.SS3.p1.5.m5.1.1.4" xref="S4.SS3.p1.5.m5.1.1.4.cmml">b</mi><mo lspace="0em" rspace="0em" id="S4.SS3.p1.5.m5.1.1.2a" xref="S4.SS3.p1.5.m5.1.1.2.cmml">​</mo><mi id="S4.SS3.p1.5.m5.1.1.5" xref="S4.SS3.p1.5.m5.1.1.5.cmml">r</mi><mo lspace="0em" rspace="0em" id="S4.SS3.p1.5.m5.1.1.2b" xref="S4.SS3.p1.5.m5.1.1.2.cmml">​</mo><mi id="S4.SS3.p1.5.m5.1.1.6" xref="S4.SS3.p1.5.m5.1.1.6.cmml">h</mi><mo lspace="0em" rspace="0em" id="S4.SS3.p1.5.m5.1.1.2c" xref="S4.SS3.p1.5.m5.1.1.2.cmml">​</mo><mrow id="S4.SS3.p1.5.m5.1.1.1.1" xref="S4.SS3.p1.5.m5.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.SS3.p1.5.m5.1.1.1.1.2" xref="S4.SS3.p1.5.m5.1.1.1.1.1.cmml">(</mo><mrow id="S4.SS3.p1.5.m5.1.1.1.1.1" xref="S4.SS3.p1.5.m5.1.1.1.1.1.cmml"><mi id="S4.SS3.p1.5.m5.1.1.1.1.1.2" xref="S4.SS3.p1.5.m5.1.1.1.1.1.2.cmml">s</mi><mo id="S4.SS3.p1.5.m5.1.1.1.1.1.1" xref="S4.SS3.p1.5.m5.1.1.1.1.1.1.cmml">+</mo><mi id="S4.SS3.p1.5.m5.1.1.1.1.1.3" xref="S4.SS3.p1.5.m5.1.1.1.1.1.3.cmml">n</mi></mrow><mo stretchy="false" id="S4.SS3.p1.5.m5.1.1.1.1.3" xref="S4.SS3.p1.5.m5.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.5.m5.1b"><apply id="S4.SS3.p1.5.m5.1.1.cmml" xref="S4.SS3.p1.5.m5.1.1"><times id="S4.SS3.p1.5.m5.1.1.2.cmml" xref="S4.SS3.p1.5.m5.1.1.2"></times><cn type="integer" id="S4.SS3.p1.5.m5.1.1.3.cmml" xref="S4.SS3.p1.5.m5.1.1.3">4</cn><ci id="S4.SS3.p1.5.m5.1.1.4.cmml" xref="S4.SS3.p1.5.m5.1.1.4">𝑏</ci><ci id="S4.SS3.p1.5.m5.1.1.5.cmml" xref="S4.SS3.p1.5.m5.1.1.5">𝑟</ci><ci id="S4.SS3.p1.5.m5.1.1.6.cmml" xref="S4.SS3.p1.5.m5.1.1.6">ℎ</ci><apply id="S4.SS3.p1.5.m5.1.1.1.1.1.cmml" xref="S4.SS3.p1.5.m5.1.1.1.1"><plus id="S4.SS3.p1.5.m5.1.1.1.1.1.1.cmml" xref="S4.SS3.p1.5.m5.1.1.1.1.1.1"></plus><ci id="S4.SS3.p1.5.m5.1.1.1.1.1.2.cmml" xref="S4.SS3.p1.5.m5.1.1.1.1.1.2">𝑠</ci><ci id="S4.SS3.p1.5.m5.1.1.1.1.1.3.cmml" xref="S4.SS3.p1.5.m5.1.1.1.1.1.3">𝑛</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.5.m5.1c">4brh(s+n)</annotation></semantics></math>.
In our study, we begin merging layers from the middle to the deeper layers, consolidating the KV cache states of every two layers into a single shared state space. As a result, we effectively reduce the GPU memory usage in the decoding inference to <math id="S4.SS3.p1.6.m6.1" class="ltx_Math" alttext="3brh(s+n)" display="inline"><semantics id="S4.SS3.p1.6.m6.1a"><mrow id="S4.SS3.p1.6.m6.1.1" xref="S4.SS3.p1.6.m6.1.1.cmml"><mn id="S4.SS3.p1.6.m6.1.1.3" xref="S4.SS3.p1.6.m6.1.1.3.cmml">3</mn><mo lspace="0em" rspace="0em" id="S4.SS3.p1.6.m6.1.1.2" xref="S4.SS3.p1.6.m6.1.1.2.cmml">​</mo><mi id="S4.SS3.p1.6.m6.1.1.4" xref="S4.SS3.p1.6.m6.1.1.4.cmml">b</mi><mo lspace="0em" rspace="0em" id="S4.SS3.p1.6.m6.1.1.2a" xref="S4.SS3.p1.6.m6.1.1.2.cmml">​</mo><mi id="S4.SS3.p1.6.m6.1.1.5" xref="S4.SS3.p1.6.m6.1.1.5.cmml">r</mi><mo lspace="0em" rspace="0em" id="S4.SS3.p1.6.m6.1.1.2b" xref="S4.SS3.p1.6.m6.1.1.2.cmml">​</mo><mi id="S4.SS3.p1.6.m6.1.1.6" xref="S4.SS3.p1.6.m6.1.1.6.cmml">h</mi><mo lspace="0em" rspace="0em" id="S4.SS3.p1.6.m6.1.1.2c" xref="S4.SS3.p1.6.m6.1.1.2.cmml">​</mo><mrow id="S4.SS3.p1.6.m6.1.1.1.1" xref="S4.SS3.p1.6.m6.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.SS3.p1.6.m6.1.1.1.1.2" xref="S4.SS3.p1.6.m6.1.1.1.1.1.cmml">(</mo><mrow id="S4.SS3.p1.6.m6.1.1.1.1.1" xref="S4.SS3.p1.6.m6.1.1.1.1.1.cmml"><mi id="S4.SS3.p1.6.m6.1.1.1.1.1.2" xref="S4.SS3.p1.6.m6.1.1.1.1.1.2.cmml">s</mi><mo id="S4.SS3.p1.6.m6.1.1.1.1.1.1" xref="S4.SS3.p1.6.m6.1.1.1.1.1.1.cmml">+</mo><mi id="S4.SS3.p1.6.m6.1.1.1.1.1.3" xref="S4.SS3.p1.6.m6.1.1.1.1.1.3.cmml">n</mi></mrow><mo stretchy="false" id="S4.SS3.p1.6.m6.1.1.1.1.3" xref="S4.SS3.p1.6.m6.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.6.m6.1b"><apply id="S4.SS3.p1.6.m6.1.1.cmml" xref="S4.SS3.p1.6.m6.1.1"><times id="S4.SS3.p1.6.m6.1.1.2.cmml" xref="S4.SS3.p1.6.m6.1.1.2"></times><cn type="integer" id="S4.SS3.p1.6.m6.1.1.3.cmml" xref="S4.SS3.p1.6.m6.1.1.3">3</cn><ci id="S4.SS3.p1.6.m6.1.1.4.cmml" xref="S4.SS3.p1.6.m6.1.1.4">𝑏</ci><ci id="S4.SS3.p1.6.m6.1.1.5.cmml" xref="S4.SS3.p1.6.m6.1.1.5">𝑟</ci><ci id="S4.SS3.p1.6.m6.1.1.6.cmml" xref="S4.SS3.p1.6.m6.1.1.6">ℎ</ci><apply id="S4.SS3.p1.6.m6.1.1.1.1.1.cmml" xref="S4.SS3.p1.6.m6.1.1.1.1"><plus id="S4.SS3.p1.6.m6.1.1.1.1.1.1.cmml" xref="S4.SS3.p1.6.m6.1.1.1.1.1.1"></plus><ci id="S4.SS3.p1.6.m6.1.1.1.1.1.2.cmml" xref="S4.SS3.p1.6.m6.1.1.1.1.1.2">𝑠</ci><ci id="S4.SS3.p1.6.m6.1.1.1.1.1.3.cmml" xref="S4.SS3.p1.6.m6.1.1.1.1.1.3">𝑛</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.6.m6.1c">3brh(s+n)</annotation></semantics></math>, demonstrating a significant compression rate.</p>
</div>
<figure id="S4.F4" class="ltx_figure"><img src="/html/2405.14366/assets/x6.png" id="S4.F4.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="292" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F4.2.1.1" class="ltx_text" style="font-size:90%;">Figure 4</span>: </span><span id="S4.F4.3.2" class="ltx_text" style="font-size:90%;">Performance comparisons between our proposed MiniCache with the “averaging baseline” and the “unmerged full cache baseline” on multiple datasets with Phi3-Mini, Mixtral-8x7B, LLaMA-3-8B, and LLaMA-3-70B. More result details are shown in <a href="#A6" title="Appendix F Detailed Experiment Results ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Appendix</span> <span class="ltx_text ltx_ref_tag">F</span></a>. The x-axis indicates the number of layers merged. As more layers are merged, a greater reduction in memory usage is achieved.
</span></figcaption>
</figure>
<div id="S4.SS3.p2" class="ltx_para">
<p id="S4.SS3.p2.3" class="ltx_p"><span id="S4.SS3.p2.3.1" class="ltx_text ltx_font_bold">Restoration efficiency.</span>
We then analyze the additional memory cost incurred during the restoration process, which
During the magnitude rescaling phase, we save an additional norm vector for the corresponding layers in the KV cache. It is important to note that the norm vector is in the shape of <math id="S4.SS3.p2.1.m1.1" class="ltx_Math" alttext="\mathbb{R}^{b\times s\times 1}" display="inline"><semantics id="S4.SS3.p2.1.m1.1a"><msup id="S4.SS3.p2.1.m1.1.1" xref="S4.SS3.p2.1.m1.1.1.cmml"><mi id="S4.SS3.p2.1.m1.1.1.2" xref="S4.SS3.p2.1.m1.1.1.2.cmml">ℝ</mi><mrow id="S4.SS3.p2.1.m1.1.1.3" xref="S4.SS3.p2.1.m1.1.1.3.cmml"><mi id="S4.SS3.p2.1.m1.1.1.3.2" xref="S4.SS3.p2.1.m1.1.1.3.2.cmml">b</mi><mo lspace="0.222em" rspace="0.222em" id="S4.SS3.p2.1.m1.1.1.3.1" xref="S4.SS3.p2.1.m1.1.1.3.1.cmml">×</mo><mi id="S4.SS3.p2.1.m1.1.1.3.3" xref="S4.SS3.p2.1.m1.1.1.3.3.cmml">s</mi><mo lspace="0.222em" rspace="0.222em" id="S4.SS3.p2.1.m1.1.1.3.1a" xref="S4.SS3.p2.1.m1.1.1.3.1.cmml">×</mo><mn id="S4.SS3.p2.1.m1.1.1.3.4" xref="S4.SS3.p2.1.m1.1.1.3.4.cmml">1</mn></mrow></msup><annotation-xml encoding="MathML-Content" id="S4.SS3.p2.1.m1.1b"><apply id="S4.SS3.p2.1.m1.1.1.cmml" xref="S4.SS3.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S4.SS3.p2.1.m1.1.1.1.cmml" xref="S4.SS3.p2.1.m1.1.1">superscript</csymbol><ci id="S4.SS3.p2.1.m1.1.1.2.cmml" xref="S4.SS3.p2.1.m1.1.1.2">ℝ</ci><apply id="S4.SS3.p2.1.m1.1.1.3.cmml" xref="S4.SS3.p2.1.m1.1.1.3"><times id="S4.SS3.p2.1.m1.1.1.3.1.cmml" xref="S4.SS3.p2.1.m1.1.1.3.1"></times><ci id="S4.SS3.p2.1.m1.1.1.3.2.cmml" xref="S4.SS3.p2.1.m1.1.1.3.2">𝑏</ci><ci id="S4.SS3.p2.1.m1.1.1.3.3.cmml" xref="S4.SS3.p2.1.m1.1.1.3.3">𝑠</ci><cn type="integer" id="S4.SS3.p2.1.m1.1.1.3.4.cmml" xref="S4.SS3.p2.1.m1.1.1.3.4">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p2.1.m1.1c">\mathbb{R}^{b\times s\times 1}</annotation></semantics></math>, which has a single channel dimension compared to the fully ranked original KV states. Additionally, we suggest that the retention threshold can be set to 0.05. Therefore, we have <math id="S4.SS3.p2.2.m2.1" class="ltx_Math" alttext="brh(0.05(s+n))" display="inline"><semantics id="S4.SS3.p2.2.m2.1a"><mrow id="S4.SS3.p2.2.m2.1.1" xref="S4.SS3.p2.2.m2.1.1.cmml"><mi id="S4.SS3.p2.2.m2.1.1.3" xref="S4.SS3.p2.2.m2.1.1.3.cmml">b</mi><mo lspace="0em" rspace="0em" id="S4.SS3.p2.2.m2.1.1.2" xref="S4.SS3.p2.2.m2.1.1.2.cmml">​</mo><mi id="S4.SS3.p2.2.m2.1.1.4" xref="S4.SS3.p2.2.m2.1.1.4.cmml">r</mi><mo lspace="0em" rspace="0em" id="S4.SS3.p2.2.m2.1.1.2a" xref="S4.SS3.p2.2.m2.1.1.2.cmml">​</mo><mi id="S4.SS3.p2.2.m2.1.1.5" xref="S4.SS3.p2.2.m2.1.1.5.cmml">h</mi><mo lspace="0em" rspace="0em" id="S4.SS3.p2.2.m2.1.1.2b" xref="S4.SS3.p2.2.m2.1.1.2.cmml">​</mo><mrow id="S4.SS3.p2.2.m2.1.1.1.1" xref="S4.SS3.p2.2.m2.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.SS3.p2.2.m2.1.1.1.1.2" xref="S4.SS3.p2.2.m2.1.1.1.1.1.cmml">(</mo><mrow id="S4.SS3.p2.2.m2.1.1.1.1.1" xref="S4.SS3.p2.2.m2.1.1.1.1.1.cmml"><mn id="S4.SS3.p2.2.m2.1.1.1.1.1.3" xref="S4.SS3.p2.2.m2.1.1.1.1.1.3.cmml">0.05</mn><mo lspace="0em" rspace="0em" id="S4.SS3.p2.2.m2.1.1.1.1.1.2" xref="S4.SS3.p2.2.m2.1.1.1.1.1.2.cmml">​</mo><mrow id="S4.SS3.p2.2.m2.1.1.1.1.1.1.1" xref="S4.SS3.p2.2.m2.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.SS3.p2.2.m2.1.1.1.1.1.1.1.2" xref="S4.SS3.p2.2.m2.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S4.SS3.p2.2.m2.1.1.1.1.1.1.1.1" xref="S4.SS3.p2.2.m2.1.1.1.1.1.1.1.1.cmml"><mi id="S4.SS3.p2.2.m2.1.1.1.1.1.1.1.1.2" xref="S4.SS3.p2.2.m2.1.1.1.1.1.1.1.1.2.cmml">s</mi><mo id="S4.SS3.p2.2.m2.1.1.1.1.1.1.1.1.1" xref="S4.SS3.p2.2.m2.1.1.1.1.1.1.1.1.1.cmml">+</mo><mi id="S4.SS3.p2.2.m2.1.1.1.1.1.1.1.1.3" xref="S4.SS3.p2.2.m2.1.1.1.1.1.1.1.1.3.cmml">n</mi></mrow><mo stretchy="false" id="S4.SS3.p2.2.m2.1.1.1.1.1.1.1.3" xref="S4.SS3.p2.2.m2.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S4.SS3.p2.2.m2.1.1.1.1.3" xref="S4.SS3.p2.2.m2.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.p2.2.m2.1b"><apply id="S4.SS3.p2.2.m2.1.1.cmml" xref="S4.SS3.p2.2.m2.1.1"><times id="S4.SS3.p2.2.m2.1.1.2.cmml" xref="S4.SS3.p2.2.m2.1.1.2"></times><ci id="S4.SS3.p2.2.m2.1.1.3.cmml" xref="S4.SS3.p2.2.m2.1.1.3">𝑏</ci><ci id="S4.SS3.p2.2.m2.1.1.4.cmml" xref="S4.SS3.p2.2.m2.1.1.4">𝑟</ci><ci id="S4.SS3.p2.2.m2.1.1.5.cmml" xref="S4.SS3.p2.2.m2.1.1.5">ℎ</ci><apply id="S4.SS3.p2.2.m2.1.1.1.1.1.cmml" xref="S4.SS3.p2.2.m2.1.1.1.1"><times id="S4.SS3.p2.2.m2.1.1.1.1.1.2.cmml" xref="S4.SS3.p2.2.m2.1.1.1.1.1.2"></times><cn type="float" id="S4.SS3.p2.2.m2.1.1.1.1.1.3.cmml" xref="S4.SS3.p2.2.m2.1.1.1.1.1.3">0.05</cn><apply id="S4.SS3.p2.2.m2.1.1.1.1.1.1.1.1.cmml" xref="S4.SS3.p2.2.m2.1.1.1.1.1.1.1"><plus id="S4.SS3.p2.2.m2.1.1.1.1.1.1.1.1.1.cmml" xref="S4.SS3.p2.2.m2.1.1.1.1.1.1.1.1.1"></plus><ci id="S4.SS3.p2.2.m2.1.1.1.1.1.1.1.1.2.cmml" xref="S4.SS3.p2.2.m2.1.1.1.1.1.1.1.1.2">𝑠</ci><ci id="S4.SS3.p2.2.m2.1.1.1.1.1.1.1.1.3.cmml" xref="S4.SS3.p2.2.m2.1.1.1.1.1.1.1.1.3">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p2.2.m2.1c">brh(0.05(s+n))</annotation></semantics></math> tokens retained without compression. Finally, our overall memory requirement is given by
<math id="S4.SS3.p2.3.m3.2" class="ltx_Math" alttext="(3.1h+2)br(s+n)" display="inline"><semantics id="S4.SS3.p2.3.m3.2a"><mrow id="S4.SS3.p2.3.m3.2.2" xref="S4.SS3.p2.3.m3.2.2.cmml"><mrow id="S4.SS3.p2.3.m3.1.1.1.1" xref="S4.SS3.p2.3.m3.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.SS3.p2.3.m3.1.1.1.1.2" xref="S4.SS3.p2.3.m3.1.1.1.1.1.cmml">(</mo><mrow id="S4.SS3.p2.3.m3.1.1.1.1.1" xref="S4.SS3.p2.3.m3.1.1.1.1.1.cmml"><mrow id="S4.SS3.p2.3.m3.1.1.1.1.1.2" xref="S4.SS3.p2.3.m3.1.1.1.1.1.2.cmml"><mn id="S4.SS3.p2.3.m3.1.1.1.1.1.2.2" xref="S4.SS3.p2.3.m3.1.1.1.1.1.2.2.cmml">3.1</mn><mo lspace="0em" rspace="0em" id="S4.SS3.p2.3.m3.1.1.1.1.1.2.1" xref="S4.SS3.p2.3.m3.1.1.1.1.1.2.1.cmml">​</mo><mi id="S4.SS3.p2.3.m3.1.1.1.1.1.2.3" xref="S4.SS3.p2.3.m3.1.1.1.1.1.2.3.cmml">h</mi></mrow><mo id="S4.SS3.p2.3.m3.1.1.1.1.1.1" xref="S4.SS3.p2.3.m3.1.1.1.1.1.1.cmml">+</mo><mn id="S4.SS3.p2.3.m3.1.1.1.1.1.3" xref="S4.SS3.p2.3.m3.1.1.1.1.1.3.cmml">2</mn></mrow><mo stretchy="false" id="S4.SS3.p2.3.m3.1.1.1.1.3" xref="S4.SS3.p2.3.m3.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S4.SS3.p2.3.m3.2.2.3" xref="S4.SS3.p2.3.m3.2.2.3.cmml">​</mo><mi id="S4.SS3.p2.3.m3.2.2.4" xref="S4.SS3.p2.3.m3.2.2.4.cmml">b</mi><mo lspace="0em" rspace="0em" id="S4.SS3.p2.3.m3.2.2.3a" xref="S4.SS3.p2.3.m3.2.2.3.cmml">​</mo><mi id="S4.SS3.p2.3.m3.2.2.5" xref="S4.SS3.p2.3.m3.2.2.5.cmml">r</mi><mo lspace="0em" rspace="0em" id="S4.SS3.p2.3.m3.2.2.3b" xref="S4.SS3.p2.3.m3.2.2.3.cmml">​</mo><mrow id="S4.SS3.p2.3.m3.2.2.2.1" xref="S4.SS3.p2.3.m3.2.2.2.1.1.cmml"><mo stretchy="false" id="S4.SS3.p2.3.m3.2.2.2.1.2" xref="S4.SS3.p2.3.m3.2.2.2.1.1.cmml">(</mo><mrow id="S4.SS3.p2.3.m3.2.2.2.1.1" xref="S4.SS3.p2.3.m3.2.2.2.1.1.cmml"><mi id="S4.SS3.p2.3.m3.2.2.2.1.1.2" xref="S4.SS3.p2.3.m3.2.2.2.1.1.2.cmml">s</mi><mo id="S4.SS3.p2.3.m3.2.2.2.1.1.1" xref="S4.SS3.p2.3.m3.2.2.2.1.1.1.cmml">+</mo><mi id="S4.SS3.p2.3.m3.2.2.2.1.1.3" xref="S4.SS3.p2.3.m3.2.2.2.1.1.3.cmml">n</mi></mrow><mo stretchy="false" id="S4.SS3.p2.3.m3.2.2.2.1.3" xref="S4.SS3.p2.3.m3.2.2.2.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.p2.3.m3.2b"><apply id="S4.SS3.p2.3.m3.2.2.cmml" xref="S4.SS3.p2.3.m3.2.2"><times id="S4.SS3.p2.3.m3.2.2.3.cmml" xref="S4.SS3.p2.3.m3.2.2.3"></times><apply id="S4.SS3.p2.3.m3.1.1.1.1.1.cmml" xref="S4.SS3.p2.3.m3.1.1.1.1"><plus id="S4.SS3.p2.3.m3.1.1.1.1.1.1.cmml" xref="S4.SS3.p2.3.m3.1.1.1.1.1.1"></plus><apply id="S4.SS3.p2.3.m3.1.1.1.1.1.2.cmml" xref="S4.SS3.p2.3.m3.1.1.1.1.1.2"><times id="S4.SS3.p2.3.m3.1.1.1.1.1.2.1.cmml" xref="S4.SS3.p2.3.m3.1.1.1.1.1.2.1"></times><cn type="float" id="S4.SS3.p2.3.m3.1.1.1.1.1.2.2.cmml" xref="S4.SS3.p2.3.m3.1.1.1.1.1.2.2">3.1</cn><ci id="S4.SS3.p2.3.m3.1.1.1.1.1.2.3.cmml" xref="S4.SS3.p2.3.m3.1.1.1.1.1.2.3">ℎ</ci></apply><cn type="integer" id="S4.SS3.p2.3.m3.1.1.1.1.1.3.cmml" xref="S4.SS3.p2.3.m3.1.1.1.1.1.3">2</cn></apply><ci id="S4.SS3.p2.3.m3.2.2.4.cmml" xref="S4.SS3.p2.3.m3.2.2.4">𝑏</ci><ci id="S4.SS3.p2.3.m3.2.2.5.cmml" xref="S4.SS3.p2.3.m3.2.2.5">𝑟</ci><apply id="S4.SS3.p2.3.m3.2.2.2.1.1.cmml" xref="S4.SS3.p2.3.m3.2.2.2.1"><plus id="S4.SS3.p2.3.m3.2.2.2.1.1.1.cmml" xref="S4.SS3.p2.3.m3.2.2.2.1.1.1"></plus><ci id="S4.SS3.p2.3.m3.2.2.2.1.1.2.cmml" xref="S4.SS3.p2.3.m3.2.2.2.1.1.2">𝑠</ci><ci id="S4.SS3.p2.3.m3.2.2.2.1.1.3.cmml" xref="S4.SS3.p2.3.m3.2.2.2.1.1.3">𝑛</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p2.3.m3.2c">(3.1h+2)br(s+n)</annotation></semantics></math>. The detailed derivation is shown in the <a href="#A5" title="Appendix E Detailed Efficiency Derivation ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Appendix</span> <span class="ltx_text ltx_ref_tag">E</span></a>.</p>
</div>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Experiments</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">We demonstrated that our MiniCache can perform merging compression on the latter half of the layers of LLMs with minimal performance degradation.</p>
</div>
<div id="S5.p2" class="ltx_para">
<p id="S5.p2.1" class="ltx_p"><span id="S5.p2.1.1" class="ltx_text ltx_font_bold">Implementation details.</span> Our experiments are based on representative model families of LLMs, including a compact LLM Phi-3-Mini <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib23" title="" class="ltx_ref">23</a>]</cite> and an MoE LLM Mixtral-8x7B <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib22" title="" class="ltx_ref">22</a>]</cite>. Additionally, we adopt LLaMA-3 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib6" title="" class="ltx_ref">6</a>]</cite> 8B and 70B models to explore how our method generalizes to larger LLMs. We sample ten tasks from lm-eval-harness <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib32" title="" class="ltx_ref">32</a>]</cite>, including COPA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib24" title="" class="ltx_ref">24</a>]</cite>, MathQA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib25" title="" class="ltx_ref">25</a>]</cite>, OpenBookQA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib26" title="" class="ltx_ref">26</a>]</cite>, PiQA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib27" title="" class="ltx_ref">27</a>]</cite>, RTE <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib28" title="" class="ltx_ref">28</a>]</cite>, Winogrande <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib29" title="" class="ltx_ref">29</a>]</cite>, XSUM <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib30" title="" class="ltx_ref">30</a>]</cite>, and CNN/Daily Mail <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib31" title="" class="ltx_ref">31</a>]</cite>. We also evaluate long-sequence generation on LongBench <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib33" title="" class="ltx_ref">33</a>]</cite>. We compare ours with fully cached baseline, and other methods such as round-to-nearest quantization (RTN) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib73" title="" class="ltx_ref">73</a>]</cite>, SmoothQuant <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib70" title="" class="ltx_ref">70</a>]</cite> and KIVI <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib11" title="" class="ltx_ref">11</a>]</cite>.</p>
</div>
<div id="S5.p3" class="ltx_para">
<p id="S5.p3.2" class="ltx_p">For the proposed MiniCache, we set the interpolation parameter <math id="S5.p3.1.m1.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S5.p3.1.m1.1a"><mi id="S5.p3.1.m1.1.1" xref="S5.p3.1.m1.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S5.p3.1.m1.1b"><ci id="S5.p3.1.m1.1.1.cmml" xref="S5.p3.1.m1.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.p3.1.m1.1c">t</annotation></semantics></math> to 0.6, indicating that the merged results have a smaller rotation angle to the next layer. Furthermore, we set the token retention threshold <math id="S5.p3.2.m2.1" class="ltx_Math" alttext="\gamma" display="inline"><semantics id="S5.p3.2.m2.1a"><mi id="S5.p3.2.m2.1.1" xref="S5.p3.2.m2.1.1.cmml">γ</mi><annotation-xml encoding="MathML-Content" id="S5.p3.2.m2.1b"><ci id="S5.p3.2.m2.1.1.cmml" xref="S5.p3.2.m2.1.1">𝛾</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.p3.2.m2.1c">\gamma</annotation></semantics></math> to 0.05, according to the statistics of unmergeable tokens across multiple datasets.
In addition to our merging method, we also consider a strong baseline of average merging. For sequential loading of large models, we utilize NVIDIA 4 A100 80GB GPUs, more details refers to <a href="#A4" title="Appendix D Additional Implementation Details ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Appendix</span> <span class="ltx_text ltx_ref_tag">D</span></a>.</p>
</div>
<div id="S5.p4" class="ltx_para">
<p id="S5.p4.1" class="ltx_p"><span id="S5.p4.1.1" class="ltx_text ltx_font_bold">Main results.</span> We evaluate MiniCache by merging KV caches across all layers on GSM8K, COQA, and TruthfulQA. The results are shown in <a href="#S4.F4" title="In 4.3 Efficiency Discussion ‣ 4 Method ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">4</span></a>. In general, we demonstrate the general effectiveness of merging KV caches from middle-to-deep layers across different sized LLMs. Moreover, the proposed MiniCache demonstrates a consistent and significant advantage over the averaging baseline.
We also illustrate the performance of merging KV caches across half of the layers with the blue lines, where MiniCache still maintains a robust performance and achieves the best compression ratio.
Besides, we find that our method is even more effective for larger LLMs. For instance, based on LLaMA-3-70B, MiniCache shows nearly zero performance drop even with the KV cache in 87.5% of the layers
merged on the COQA dataset. This highlights the adaptability and efficiency of our approach in handling large-scale models while ensuring minimal performance degradation.</p>
</div>
<figure id="S5.T1" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="S5.T1.2.1.1" class="ltx_text" style="font-size:90%;">Table 1</span>: </span><span id="S5.T1.3.2" class="ltx_text" style="font-size:90%;">
Evaluation of different KV cache compression methods on LongBench.
MiniCache builds on top of 4-bit KIVI <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib11" title="" class="ltx_ref">11</a>]</cite> and achieves the best performance with the strongest compression rate.
</span></figcaption>
<div id="S5.T1.4" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:216.9pt;vertical-align:-0.5pt;"><span class="ltx_transformed_inner" style="transform:translate(-179.9pt,89.8pt) scale(0.546473618186747,0.546473618186747) ;">
<table id="S5.T1.4.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S5.T1.4.1.1.1" class="ltx_tr">
<th id="S5.T1.4.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt" rowspan="2"><span id="S5.T1.4.1.1.1.1.1" class="ltx_text ltx_font_bold">Model</span></th>
<th id="S5.T1.4.1.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt" rowspan="2"><span id="S5.T1.4.1.1.1.2.1" class="ltx_text ltx_font_bold">Method</span></th>
<th id="S5.T1.4.1.1.1.3" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt" rowspan="2"><span id="S5.T1.4.1.1.1.3.1" class="ltx_text ltx_font_bold">LCC</span></th>
<th id="S5.T1.4.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" rowspan="2"><span id="S5.T1.4.1.1.1.4.1" class="ltx_text ltx_font_bold">RepoBench-P</span></th>
<th id="S5.T1.4.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" rowspan="2"><span id="S5.T1.4.1.1.1.5.1" class="ltx_text ltx_font_bold">PR-en</span></th>
<th id="S5.T1.4.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" rowspan="2"><span id="S5.T1.4.1.1.1.6.1" class="ltx_text ltx_font_bold">TREC</span></th>
<th id="S5.T1.4.1.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" rowspan="2"><span id="S5.T1.4.1.1.1.7.1" class="ltx_text ltx_font_bold">2wikimqa</span></th>
<th id="S5.T1.4.1.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" rowspan="2"><span id="S5.T1.4.1.1.1.8.1" class="ltx_text ltx_font_bold">GovReport</span></th>
<th id="S5.T1.4.1.1.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" rowspan="2"><span id="S5.T1.4.1.1.1.9.1" class="ltx_text ltx_font_bold">MQA-zh</span></th>
<th id="S5.T1.4.1.1.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" rowspan="2"><span id="S5.T1.4.1.1.1.10.1" class="ltx_text ltx_font_bold">Average</span></th>
<th id="S5.T1.4.1.1.1.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S5.T1.4.1.1.1.11.1" class="ltx_text ltx_font_bold">Compression</span></th>
</tr>
<tr id="S5.T1.4.1.2.2" class="ltx_tr">
<th id="S5.T1.4.1.2.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="S5.T1.4.1.2.2.1.1" class="ltx_text ltx_font_bold">Ratio</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S5.T1.4.1.3.1" class="ltx_tr">
<th id="S5.T1.4.1.3.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="5"><span id="S5.T1.4.1.3.1.1.1" class="ltx_text">Llama-2-7B-Chat</span></th>
<th id="S5.T1.4.1.3.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Baseline</th>
<th id="S5.T1.4.1.3.1.3" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">58.16</th>
<td id="S5.T1.4.1.3.1.4" class="ltx_td ltx_align_center ltx_border_t">52.19</td>
<td id="S5.T1.4.1.3.1.5" class="ltx_td ltx_align_center ltx_border_t">10.12</td>
<td id="S5.T1.4.1.3.1.6" class="ltx_td ltx_align_center ltx_border_t">64.00</td>
<td id="S5.T1.4.1.3.1.7" class="ltx_td ltx_align_center ltx_border_t">31.12</td>
<td id="S5.T1.4.1.3.1.8" class="ltx_td ltx_align_center ltx_border_t">27.09</td>
<td id="S5.T1.4.1.3.1.9" class="ltx_td ltx_align_center ltx_border_t">10.12</td>
<td id="S5.T1.4.1.3.1.10" class="ltx_td ltx_align_center ltx_border_t">36.41</td>
<td id="S5.T1.4.1.3.1.11" class="ltx_td ltx_align_center ltx_border_t">1x</td>
</tr>
<tr id="S5.T1.4.1.4.2" class="ltx_tr">
<th id="S5.T1.4.1.4.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">RTN <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib73" title="" class="ltx_ref">73</a>]</cite>
</th>
<th id="S5.T1.4.1.4.2.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">15.44</th>
<td id="S5.T1.4.1.4.2.3" class="ltx_td ltx_align_center">8.76</td>
<td id="S5.T1.4.1.4.2.4" class="ltx_td ltx_align_center">0.79</td>
<td id="S5.T1.4.1.4.2.5" class="ltx_td ltx_align_center">4.00</td>
<td id="S5.T1.4.1.4.2.6" class="ltx_td ltx_align_center">0.30</td>
<td id="S5.T1.4.1.4.2.7" class="ltx_td ltx_align_center">1.93</td>
<td id="S5.T1.4.1.4.2.8" class="ltx_td ltx_align_center">0.07</td>
<td id="S5.T1.4.1.4.2.9" class="ltx_td ltx_align_center">4.90</td>
<td id="S5.T1.4.1.4.2.10" class="ltx_td ltx_align_center">3.21x</td>
</tr>
<tr id="S5.T1.4.1.5.3" class="ltx_tr">
<th id="S5.T1.4.1.5.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">SmoothQuant <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib70" title="" class="ltx_ref">70</a>]</cite>
</th>
<th id="S5.T1.4.1.5.3.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">35.31</th>
<td id="S5.T1.4.1.5.3.3" class="ltx_td ltx_align_center">32.18</td>
<td id="S5.T1.4.1.5.3.4" class="ltx_td ltx_align_center">0.79</td>
<td id="S5.T1.4.1.5.3.5" class="ltx_td ltx_align_center">28.75</td>
<td id="S5.T1.4.1.5.3.6" class="ltx_td ltx_align_center">7.45</td>
<td id="S5.T1.4.1.5.3.7" class="ltx_td ltx_align_center">11.83</td>
<td id="S5.T1.4.1.5.3.8" class="ltx_td ltx_align_center">1.68</td>
<td id="S5.T1.4.1.5.3.9" class="ltx_td ltx_align_center">16.28</td>
<td id="S5.T1.4.1.5.3.10" class="ltx_td ltx_align_center">2.15x</td>
</tr>
<tr id="S5.T1.4.1.6.4" class="ltx_tr">
<th id="S5.T1.4.1.6.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">KIVI-2 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib11" title="" class="ltx_ref">11</a>]</cite>
</th>
<th id="S5.T1.4.1.6.4.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">49.32</th>
<td id="S5.T1.4.1.6.4.3" class="ltx_td ltx_align_center">43.71</td>
<td id="S5.T1.4.1.6.4.4" class="ltx_td ltx_align_center">4.50</td>
<td id="S5.T1.4.1.6.4.5" class="ltx_td ltx_align_center">63.00</td>
<td id="S5.T1.4.1.6.4.6" class="ltx_td ltx_align_center">24.07</td>
<td id="S5.T1.4.1.6.4.7" class="ltx_td ltx_align_center">24.73</td>
<td id="S5.T1.4.1.6.4.8" class="ltx_td ltx_align_center">10.24</td>
<td id="S5.T1.4.1.6.4.9" class="ltx_td ltx_align_center">31.51</td>
<td id="S5.T1.4.1.6.4.10" class="ltx_td ltx_align_center">3.95x</td>
</tr>
<tr id="S5.T1.4.1.7.5" class="ltx_tr">
<th id="S5.T1.4.1.7.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S5.T1.4.1.7.5.1.1" class="ltx_text ltx_font_bold">MiniCache</span></th>
<th id="S5.T1.4.1.7.5.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">58.03</th>
<td id="S5.T1.4.1.7.5.3" class="ltx_td ltx_align_center">52.01</td>
<td id="S5.T1.4.1.7.5.4" class="ltx_td ltx_align_center">9.00</td>
<td id="S5.T1.4.1.7.5.5" class="ltx_td ltx_align_center">64.00</td>
<td id="S5.T1.4.1.7.5.6" class="ltx_td ltx_align_center">30.58</td>
<td id="S5.T1.4.1.7.5.7" class="ltx_td ltx_align_center">25.32</td>
<td id="S5.T1.4.1.7.5.8" class="ltx_td ltx_align_center">10.13</td>
<td id="S5.T1.4.1.7.5.9" class="ltx_td ltx_align_center"><span id="S5.T1.4.1.7.5.9.1" class="ltx_text ltx_font_bold">35.44</span></td>
<td id="S5.T1.4.1.7.5.10" class="ltx_td ltx_align_center"><span id="S5.T1.4.1.7.5.10.1" class="ltx_text ltx_font_bold">5.02x</span></td>
</tr>
<tr id="S5.T1.4.1.8.6" class="ltx_tr">
<th id="S5.T1.4.1.8.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="5"><span id="S5.T1.4.1.8.6.1.1" class="ltx_text">Llama-2-13B-Chat</span></th>
<th id="S5.T1.4.1.8.6.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Baseline</th>
<th id="S5.T1.4.1.8.6.3" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">48.06</th>
<td id="S5.T1.4.1.8.6.4" class="ltx_td ltx_align_center ltx_border_t">50.08</td>
<td id="S5.T1.4.1.8.6.5" class="ltx_td ltx_align_center ltx_border_t">14.25</td>
<td id="S5.T1.4.1.8.6.6" class="ltx_td ltx_align_center ltx_border_t">68.50</td>
<td id="S5.T1.4.1.8.6.7" class="ltx_td ltx_align_center ltx_border_t">13.09</td>
<td id="S5.T1.4.1.8.6.8" class="ltx_td ltx_align_center ltx_border_t">27.76</td>
<td id="S5.T1.4.1.8.6.9" class="ltx_td ltx_align_center ltx_border_t">7.23</td>
<td id="S5.T1.4.1.8.6.10" class="ltx_td ltx_align_center ltx_border_t">32.71</td>
<td id="S5.T1.4.1.8.6.11" class="ltx_td ltx_align_center ltx_border_t">1x</td>
</tr>
<tr id="S5.T1.4.1.9.7" class="ltx_tr">
<th id="S5.T1.4.1.9.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">RTN <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib73" title="" class="ltx_ref">73</a>]</cite>
</th>
<th id="S5.T1.4.1.9.7.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">20.89</th>
<td id="S5.T1.4.1.9.7.3" class="ltx_td ltx_align_center">18.62</td>
<td id="S5.T1.4.1.9.7.4" class="ltx_td ltx_align_center">0.33</td>
<td id="S5.T1.4.1.9.7.5" class="ltx_td ltx_align_center">0.00</td>
<td id="S5.T1.4.1.9.7.6" class="ltx_td ltx_align_center">0.52</td>
<td id="S5.T1.4.1.9.7.7" class="ltx_td ltx_align_center">1.68</td>
<td id="S5.T1.4.1.9.7.8" class="ltx_td ltx_align_center">0.16</td>
<td id="S5.T1.4.1.9.7.9" class="ltx_td ltx_align_center">6.03</td>
<td id="S5.T1.4.1.9.7.10" class="ltx_td ltx_align_center">3.21x</td>
</tr>
<tr id="S5.T1.4.1.10.8" class="ltx_tr">
<th id="S5.T1.4.1.10.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">SmoothQuant <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib70" title="" class="ltx_ref">70</a>]</cite>
</th>
<th id="S5.T1.4.1.10.8.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">32.17</th>
<td id="S5.T1.4.1.10.8.3" class="ltx_td ltx_align_center">33.86</td>
<td id="S5.T1.4.1.10.8.4" class="ltx_td ltx_align_center">2.65</td>
<td id="S5.T1.4.1.10.8.5" class="ltx_td ltx_align_center">48.00</td>
<td id="S5.T1.4.1.10.8.6" class="ltx_td ltx_align_center">3.53</td>
<td id="S5.T1.4.1.10.8.7" class="ltx_td ltx_align_center">12.47</td>
<td id="S5.T1.4.1.10.8.8" class="ltx_td ltx_align_center">0.47</td>
<td id="S5.T1.4.1.10.8.9" class="ltx_td ltx_align_center">19.16</td>
<td id="S5.T1.4.1.10.8.10" class="ltx_td ltx_align_center">2.15x</td>
</tr>
<tr id="S5.T1.4.1.11.9" class="ltx_tr">
<th id="S5.T1.4.1.11.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">KIVI-2 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib11" title="" class="ltx_ref">11</a>]</cite>
</th>
<th id="S5.T1.4.1.11.9.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">48.60</th>
<td id="S5.T1.4.1.11.9.3" class="ltx_td ltx_align_center">48.81</td>
<td id="S5.T1.4.1.11.9.4" class="ltx_td ltx_align_center">13.50</td>
<td id="S5.T1.4.1.11.9.5" class="ltx_td ltx_align_center">68.00</td>
<td id="S5.T1.4.1.11.9.6" class="ltx_td ltx_align_center">14.32</td>
<td id="S5.T1.4.1.11.9.7" class="ltx_td ltx_align_center">25.70</td>
<td id="S5.T1.4.1.11.9.8" class="ltx_td ltx_align_center">7.01</td>
<td id="S5.T1.4.1.11.9.9" class="ltx_td ltx_align_center">32.42</td>
<td id="S5.T1.4.1.11.9.10" class="ltx_td ltx_align_center">3.95x</td>
</tr>
<tr id="S5.T1.4.1.12.10" class="ltx_tr">
<th id="S5.T1.4.1.12.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S5.T1.4.1.12.10.1.1" class="ltx_text ltx_font_bold">MiniCache</span></th>
<th id="S5.T1.4.1.12.10.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">48.75</th>
<td id="S5.T1.4.1.12.10.3" class="ltx_td ltx_align_center">48.59</td>
<td id="S5.T1.4.1.12.10.4" class="ltx_td ltx_align_center">13.00</td>
<td id="S5.T1.4.1.12.10.5" class="ltx_td ltx_align_center">68.00</td>
<td id="S5.T1.4.1.12.10.6" class="ltx_td ltx_align_center">14.36</td>
<td id="S5.T1.4.1.12.10.7" class="ltx_td ltx_align_center">26.57</td>
<td id="S5.T1.4.1.12.10.8" class="ltx_td ltx_align_center">7.99</td>
<td id="S5.T1.4.1.12.10.9" class="ltx_td ltx_align_center"><span id="S5.T1.4.1.12.10.9.1" class="ltx_text ltx_font_bold">32.61</span></td>
<td id="S5.T1.4.1.12.10.10" class="ltx_td ltx_align_center"><span id="S5.T1.4.1.12.10.10.1" class="ltx_text ltx_font_bold">5.02x</span></td>
</tr>
<tr id="S5.T1.4.1.13.11" class="ltx_tr">
<th id="S5.T1.4.1.13.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="5"><span id="S5.T1.4.1.13.11.1.1" class="ltx_text">Mistral-7B</span></th>
<th id="S5.T1.4.1.13.11.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Baseline</th>
<th id="S5.T1.4.1.13.11.3" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">68.06</th>
<td id="S5.T1.4.1.13.11.4" class="ltx_td ltx_align_center ltx_border_t">60.46</td>
<td id="S5.T1.4.1.13.11.5" class="ltx_td ltx_align_center ltx_border_t">17.71</td>
<td id="S5.T1.4.1.13.11.6" class="ltx_td ltx_align_center ltx_border_t">68.00</td>
<td id="S5.T1.4.1.13.11.7" class="ltx_td ltx_align_center ltx_border_t">10.87</td>
<td id="S5.T1.4.1.13.11.8" class="ltx_td ltx_align_center ltx_border_t">20.09</td>
<td id="S5.T1.4.1.13.11.9" class="ltx_td ltx_align_center ltx_border_t">17.10</td>
<td id="S5.T1.4.1.13.11.10" class="ltx_td ltx_align_center ltx_border_t">37.33</td>
<td id="S5.T1.4.1.13.11.11" class="ltx_td ltx_align_center ltx_border_t">1x</td>
</tr>
<tr id="S5.T1.4.1.14.12" class="ltx_tr">
<th id="S5.T1.4.1.14.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">RTN <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib73" title="" class="ltx_ref">73</a>]</cite>
</th>
<th id="S5.T1.4.1.14.12.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">27.98</th>
<td id="S5.T1.4.1.14.12.3" class="ltx_td ltx_align_center">26.18</td>
<td id="S5.T1.4.1.14.12.4" class="ltx_td ltx_align_center">3.34</td>
<td id="S5.T1.4.1.14.12.5" class="ltx_td ltx_align_center">13.00</td>
<td id="S5.T1.4.1.14.12.6" class="ltx_td ltx_align_center">1.11</td>
<td id="S5.T1.4.1.14.12.7" class="ltx_td ltx_align_center">2.49</td>
<td id="S5.T1.4.1.14.12.8" class="ltx_td ltx_align_center">0.45</td>
<td id="S5.T1.4.1.14.12.9" class="ltx_td ltx_align_center">10.51</td>
<td id="S5.T1.4.1.14.12.10" class="ltx_td ltx_align_center">3.21x</td>
</tr>
<tr id="S5.T1.4.1.15.13" class="ltx_tr">
<th id="S5.T1.4.1.15.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">SmoothQuant <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib70" title="" class="ltx_ref">70</a>]</cite>
</th>
<th id="S5.T1.4.1.15.13.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">40.63</th>
<td id="S5.T1.4.1.15.13.3" class="ltx_td ltx_align_center">35.14</td>
<td id="S5.T1.4.1.15.13.4" class="ltx_td ltx_align_center">3.40</td>
<td id="S5.T1.4.1.15.13.5" class="ltx_td ltx_align_center">30.50</td>
<td id="S5.T1.4.1.15.13.6" class="ltx_td ltx_align_center">6.03</td>
<td id="S5.T1.4.1.15.13.7" class="ltx_td ltx_align_center">5.00</td>
<td id="S5.T1.4.1.15.13.8" class="ltx_td ltx_align_center">4.12</td>
<td id="S5.T1.4.1.15.13.9" class="ltx_td ltx_align_center">17.55</td>
<td id="S5.T1.4.1.15.13.10" class="ltx_td ltx_align_center">2.15x</td>
</tr>
<tr id="S5.T1.4.1.16.14" class="ltx_tr">
<th id="S5.T1.4.1.16.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">KIVI-2 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib11" title="" class="ltx_ref">11</a>]</cite>
</th>
<th id="S5.T1.4.1.16.14.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">65.16</th>
<td id="S5.T1.4.1.16.14.3" class="ltx_td ltx_align_center">58.33</td>
<td id="S5.T1.4.1.16.14.4" class="ltx_td ltx_align_center">12.43</td>
<td id="S5.T1.4.1.16.14.5" class="ltx_td ltx_align_center">65.00</td>
<td id="S5.T1.4.1.16.14.6" class="ltx_td ltx_align_center">11.03</td>
<td id="S5.T1.4.1.16.14.7" class="ltx_td ltx_align_center">13.22</td>
<td id="S5.T1.4.1.16.14.8" class="ltx_td ltx_align_center">13.87</td>
<td id="S5.T1.4.1.16.14.9" class="ltx_td ltx_align_center">33.43</td>
<td id="S5.T1.4.1.16.14.10" class="ltx_td ltx_align_center">3.95x</td>
</tr>
<tr id="S5.T1.4.1.17.15" class="ltx_tr">
<th id="S5.T1.4.1.17.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S5.T1.4.1.17.15.1.1" class="ltx_text ltx_font_bold">MiniCache</span></th>
<th id="S5.T1.4.1.17.15.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">68.89</th>
<td id="S5.T1.4.1.17.15.3" class="ltx_td ltx_align_center">60.98</td>
<td id="S5.T1.4.1.17.15.4" class="ltx_td ltx_align_center">13.92</td>
<td id="S5.T1.4.1.17.15.5" class="ltx_td ltx_align_center">67.00</td>
<td id="S5.T1.4.1.17.15.6" class="ltx_td ltx_align_center">10.50</td>
<td id="S5.T1.4.1.17.15.7" class="ltx_td ltx_align_center">18.06</td>
<td id="S5.T1.4.1.17.15.8" class="ltx_td ltx_align_center">7.88</td>
<td id="S5.T1.4.1.17.15.9" class="ltx_td ltx_align_center"><span id="S5.T1.4.1.17.15.9.1" class="ltx_text ltx_font_bold">35.75</span></td>
<td id="S5.T1.4.1.17.15.10" class="ltx_td ltx_align_center"><span id="S5.T1.4.1.17.15.10.1" class="ltx_text ltx_font_bold">5.02x</span></td>
</tr>
<tr id="S5.T1.4.1.18.16" class="ltx_tr">
<th id="S5.T1.4.1.18.16.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_t" rowspan="5"><span id="S5.T1.4.1.18.16.1.1" class="ltx_text">Mistral-7B-Instruct</span></th>
<th id="S5.T1.4.1.18.16.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Baseline</th>
<th id="S5.T1.4.1.18.16.3" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">55.51</th>
<td id="S5.T1.4.1.18.16.4" class="ltx_td ltx_align_center ltx_border_t">48.96</td>
<td id="S5.T1.4.1.18.16.5" class="ltx_td ltx_align_center ltx_border_t">60.00</td>
<td id="S5.T1.4.1.18.16.6" class="ltx_td ltx_align_center ltx_border_t">71.00</td>
<td id="S5.T1.4.1.18.16.7" class="ltx_td ltx_align_center ltx_border_t">27.33</td>
<td id="S5.T1.4.1.18.16.8" class="ltx_td ltx_align_center ltx_border_t">32.85</td>
<td id="S5.T1.4.1.18.16.9" class="ltx_td ltx_align_center ltx_border_t">42.74</td>
<td id="S5.T1.4.1.18.16.10" class="ltx_td ltx_align_center ltx_border_t">48.32</td>
<td id="S5.T1.4.1.18.16.11" class="ltx_td ltx_align_center ltx_border_t">1x</td>
</tr>
<tr id="S5.T1.4.1.19.17" class="ltx_tr">
<th id="S5.T1.4.1.19.17.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">RTN <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib73" title="" class="ltx_ref">73</a>]</cite>
</th>
<th id="S5.T1.4.1.19.17.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">32.36</th>
<td id="S5.T1.4.1.19.17.3" class="ltx_td ltx_align_center">33.23</td>
<td id="S5.T1.4.1.19.17.4" class="ltx_td ltx_align_center">0.67</td>
<td id="S5.T1.4.1.19.17.5" class="ltx_td ltx_align_center">1.00</td>
<td id="S5.T1.4.1.19.17.6" class="ltx_td ltx_align_center">2.25</td>
<td id="S5.T1.4.1.19.17.7" class="ltx_td ltx_align_center">10.03</td>
<td id="S5.T1.4.1.19.17.8" class="ltx_td ltx_align_center">2.30</td>
<td id="S5.T1.4.1.19.17.9" class="ltx_td ltx_align_center">11.55</td>
<td id="S5.T1.4.1.19.17.10" class="ltx_td ltx_align_center">3.21x</td>
</tr>
<tr id="S5.T1.4.1.20.18" class="ltx_tr">
<th id="S5.T1.4.1.20.18.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">SmoothQuant <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib70" title="" class="ltx_ref">70</a>]</cite>
</th>
<th id="S5.T1.4.1.20.18.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">43.84</th>
<td id="S5.T1.4.1.20.18.3" class="ltx_td ltx_align_center">38.63</td>
<td id="S5.T1.4.1.20.18.4" class="ltx_td ltx_align_center">4.79</td>
<td id="S5.T1.4.1.20.18.5" class="ltx_td ltx_align_center">39.50</td>
<td id="S5.T1.4.1.20.18.6" class="ltx_td ltx_align_center">10.34</td>
<td id="S5.T1.4.1.20.18.7" class="ltx_td ltx_align_center">23.61</td>
<td id="S5.T1.4.1.20.18.8" class="ltx_td ltx_align_center">8.33</td>
<td id="S5.T1.4.1.20.18.9" class="ltx_td ltx_align_center">24.43</td>
<td id="S5.T1.4.1.20.18.10" class="ltx_td ltx_align_center">2.15x</td>
</tr>
<tr id="S5.T1.4.1.21.19" class="ltx_tr">
<th id="S5.T1.4.1.21.19.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">KIVI-2 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib11" title="" class="ltx_ref">11</a>]</cite>
</th>
<th id="S5.T1.4.1.21.19.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">53.13</th>
<td id="S5.T1.4.1.21.19.3" class="ltx_td ltx_align_center">48.60</td>
<td id="S5.T1.4.1.21.19.4" class="ltx_td ltx_align_center">47.50</td>
<td id="S5.T1.4.1.21.19.5" class="ltx_td ltx_align_center">69.00</td>
<td id="S5.T1.4.1.21.19.6" class="ltx_td ltx_align_center">20.68</td>
<td id="S5.T1.4.1.21.19.7" class="ltx_td ltx_align_center">29.37</td>
<td id="S5.T1.4.1.21.19.8" class="ltx_td ltx_align_center">33.88</td>
<td id="S5.T1.4.1.21.19.9" class="ltx_td ltx_align_center">43.74</td>
<td id="S5.T1.4.1.21.19.10" class="ltx_td ltx_align_center">3.95x</td>
</tr>
<tr id="S5.T1.4.1.22.20" class="ltx_tr">
<th id="S5.T1.4.1.22.20.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb"><span id="S5.T1.4.1.22.20.1.1" class="ltx_text ltx_font_bold">MiniCache</span></th>
<th id="S5.T1.4.1.22.20.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">54.79</th>
<td id="S5.T1.4.1.22.20.3" class="ltx_td ltx_align_center ltx_border_bb">51.02</td>
<td id="S5.T1.4.1.22.20.4" class="ltx_td ltx_align_center ltx_border_bb">64.14</td>
<td id="S5.T1.4.1.22.20.5" class="ltx_td ltx_align_center ltx_border_bb">71.00</td>
<td id="S5.T1.4.1.22.20.6" class="ltx_td ltx_align_center ltx_border_bb">24.97</td>
<td id="S5.T1.4.1.22.20.7" class="ltx_td ltx_align_center ltx_border_bb">31.46</td>
<td id="S5.T1.4.1.22.20.8" class="ltx_td ltx_align_center ltx_border_bb">27.54</td>
<td id="S5.T1.4.1.22.20.9" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T1.4.1.22.20.9.1" class="ltx_text ltx_font_bold">46.99</span></td>
<td id="S5.T1.4.1.22.20.10" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T1.4.1.22.20.10.1" class="ltx_text ltx_font_bold">5.02x</span></td>
</tr>
</tbody>
</table>
</span></div>
</figure>
<div id="S5.p5" class="ltx_para">
<p id="S5.p5.1" class="ltx_p"><span id="S5.p5.1.1" class="ltx_text ltx_font_bold">LongBench.</span> We also conducted experiments to evaluate performance and quality in long-sequence generation using the LongBench dataset <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib33" title="" class="ltx_ref">33</a>]</cite>, shown in <a href="#S5.T1" title="In 5 Experiments ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">1</span></a>. Our experiments applied MiniCache over several models: LLaMA-2-7B-Chat, LLaMA-2-13B-Chat, Mistral-7B, and Mistral-7B-Instruct.
It is important to note that our MiniCache method maintains orthogonality with all existing quantization and sparsity (refers to <a href="#A1.T1" title="In Appendix A Additional Experiments Results ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">A</span></a>) methods at both the model and token-wise levels.
When combined with KIVI-4bit KV cache quantization, our approach achieved a compression rate of <math id="S5.p5.1.m1.1" class="ltx_math_unparsed" alttext="5.02\times" display="inline"><semantics id="S5.p5.1.m1.1a"><mrow id="S5.p5.1.m1.1b"><mn id="S5.p5.1.m1.1.1">5.02</mn><mo lspace="0.222em" id="S5.p5.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S5.p5.1.m1.1c">5.02\times</annotation></semantics></math>, with minimal impact on accuracy across various challenging long-context generation tasks.
The combination of MiniCache and KIVI-4bit KV cache quantization demonstrates significant memory savings without compromising the model’s ability to handle long sequences effectively. This highlights the potential of our method to optimize large language models for tasks requiring extensive context, making them more efficient and scalable for real-world applications.</p>
</div>
<div id="S5.p6" class="ltx_para">
<p id="S5.p6.2" class="ltx_p"><span id="S5.p6.2.2" class="ltx_text ltx_font_bold">Efficiency analysis.</span> To assess the acceleration capabilities of MiniCache, we conduct evaluations based on the methodologies employed in vLLM <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib74" title="" class="ltx_ref">74</a>]</cite> and KIVI <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib11" title="" class="ltx_ref">11</a>]</cite>. We generate synthetic workloads derived from ShareGPT, which include real input and output texts from LLM services. The dataset features an average input prompt length of 161 tokens and an output length of 338 tokens. Using the LLaMA-2-7B model on a single 80GB NVIDIA A100 GPU, we benchmark our method in a batch-serving scenario, comparing peak memory usage and throughput among 2-bit KIVI, 4-bit MiniCache, and an FP16 baseline. As illustrated in <a href="#S5.F6" title="In 5 Experiments ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">6</span></a>, with a batch size of 128, <span id="S5.p6.2.3" class="ltx_text ltx_font_bold">MiniCache reduces memory usage by 25GB, achieving a 41% memory saving</span>. In terms of throughput, <span id="S5.p6.1.1" class="ltx_text ltx_font_bold"> MiniCache outperforms the FP16 baseline by approximately 5<math id="S5.p6.1.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S5.p6.1.1.m1.1a"><mo id="S5.p6.1.1.m1.1.1" xref="S5.p6.1.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S5.p6.1.1.m1.1b"><times id="S5.p6.1.1.m1.1.1.cmml" xref="S5.p6.1.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S5.p6.1.1.m1.1c">\times</annotation></semantics></math></span>. Additionally, despite utilizing 4-bit quantization, MiniCache benefits from merging and sharing KV caches across adjacent layers, resulting in a 1.29<math id="S5.p6.2.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S5.p6.2.m1.1a"><mo id="S5.p6.2.m1.1.1" xref="S5.p6.2.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S5.p6.2.m1.1b"><times id="S5.p6.2.m1.1.1.cmml" xref="S5.p6.2.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S5.p6.2.m1.1c">\times</annotation></semantics></math> higher throughput compared to the 2-bit KIVI. These results demonstrate that MiniCache offers a state-of-the-art trade-off between efficiency and performance.</p>
</div>
<figure id="S5.F6" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S5.F6.fig1" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:277.5pt;">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="S5.F5.sf1" class="ltx_figure ltx_figure_panel"><img src="/html/2405.14366/assets/x7.png" id="S5.F5.sf1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="368" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S5.F5.sf1.3.1.1" class="ltx_text" style="font-size:90%;">(a)</span> </span><span id="S5.F5.sf1.4.2" class="ltx_text" style="font-size:90%;">BS. <span id="S5.F5.sf1.4.2.1" class="ltx_text ltx_font_italic">vs.</span> Peak Memory Usage</span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="S5.F5.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2405.14366/assets/x8.png" id="S5.F5.sf2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="362" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S5.F5.sf2.3.1.1" class="ltx_text" style="font-size:90%;">(b)</span> </span><span id="S5.F5.sf2.4.2" class="ltx_text" style="font-size:90%;">BS. <span id="S5.F5.sf2.4.2.1" class="ltx_text ltx_font_italic">vs.</span> Decoding Throughput</span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S5.F6.fig1.1.1.1" class="ltx_text" style="font-size:90%;">Figure 5</span>: </span><span id="S5.F6.fig1.2.2" class="ltx_text" style="font-size:90%;">Memory usage and throughput comparison between our 4-bit MiniCache, 2-bit KIVI, and 16-bit Baseline. MiniCache can achieve higher throughput by enabling a larger batch size while reducing memory footprints via LLaMA-2-7B <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib5" title="" class="ltx_ref">5</a>]</cite>.</span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S5.F6.3" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:143.1pt;"><img src="/html/2405.14366/assets/x9.png" id="S5.F6.1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="317" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S5.F6.3.3.2.1" class="ltx_text" style="font-size:90%;">Figure 6</span>: </span><span id="S5.F6.3.2.1" class="ltx_text" style="font-size:90%;">LLaMA-3-8B <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib6" title="" class="ltx_ref">6</a>]</cite> to experiment on the GSM8K <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib10" title="" class="ltx_ref">10</a>]</cite>.
The right axis is the normalized frequency of the relative magnitude ratio. Optional <math id="S5.F6.3.2.1.m1.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S5.F6.3.2.1.m1.1b"><mi id="S5.F6.3.2.1.m1.1.1" xref="S5.F6.3.2.1.m1.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S5.F6.3.2.1.m1.1c"><ci id="S5.F6.3.2.1.m1.1.1.cmml" xref="S5.F6.3.2.1.m1.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.F6.3.2.1.m1.1d">t</annotation></semantics></math> shows a strong correlation with frequency.
</span></figcaption>
</figure>
</div>
</div>
</figure>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Ablation Study</h2>

<figure id="S6.T2" class="ltx_table ltx_align_floatright">
<div id="S6.T2.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:151.8pt;height:105.8pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-14.5pt,10.1pt) scale(0.839858116108667,0.839858116108667) ;">
<table id="S6.T2.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S6.T2.1.1.1" class="ltx_tr">
<th id="S6.T2.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt"><math id="S6.T2.1.1.1.1.m1.1" class="ltx_Math" alttext="\gamma" display="inline"><semantics id="S6.T2.1.1.1.1.m1.1a"><mi id="S6.T2.1.1.1.1.m1.1.1" xref="S6.T2.1.1.1.1.m1.1.1.cmml">γ</mi><annotation-xml encoding="MathML-Content" id="S6.T2.1.1.1.1.m1.1b"><ci id="S6.T2.1.1.1.1.m1.1.1.cmml" xref="S6.T2.1.1.1.1.m1.1.1">𝛾</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.T2.1.1.1.1.m1.1c">\gamma</annotation></semantics></math></th>
<th id="S6.T2.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">COQA</th>
<th id="S6.T2.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">GSM8K</th>
<th id="S6.T2.1.1.1.4" class="ltx_td ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt">TruthfulQA</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S6.T2.1.1.2.1" class="ltx_tr">
<th id="S6.T2.1.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">0</th>
<td id="S6.T2.1.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t">0.603</td>
<td id="S6.T2.1.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t">0.108</td>
<td id="S6.T2.1.1.2.1.4" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">29.813</td>
</tr>
<tr id="S6.T2.1.1.3.2" class="ltx_tr">
<th id="S6.T2.1.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">0.01</th>
<td id="S6.T2.1.1.3.2.2" class="ltx_td ltx_align_center">0.620</td>
<td id="S6.T2.1.1.3.2.3" class="ltx_td ltx_align_center">0.126</td>
<td id="S6.T2.1.1.3.2.4" class="ltx_td ltx_nopad_r ltx_align_center">30.226</td>
</tr>
<tr id="S6.T2.1.1.4.3" class="ltx_tr">
<th id="S6.T2.1.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">0.02</th>
<td id="S6.T2.1.1.4.3.2" class="ltx_td ltx_align_center">0.630</td>
<td id="S6.T2.1.1.4.3.3" class="ltx_td ltx_align_center">0.143</td>
<td id="S6.T2.1.1.4.3.4" class="ltx_td ltx_nopad_r ltx_align_center">33.903</td>
</tr>
<tr id="S6.T2.1.1.5.4" class="ltx_tr" style="background-color:#E6E6E6;">
<th id="S6.T2.1.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S6.T2.1.1.5.4.1.1" class="ltx_text" style="background-color:#E6E6E6;">0.05</span></th>
<td id="S6.T2.1.1.5.4.2" class="ltx_td ltx_align_center"><span id="S6.T2.1.1.5.4.2.1" class="ltx_text" style="background-color:#E6E6E6;">0.647</span></td>
<td id="S6.T2.1.1.5.4.3" class="ltx_td ltx_align_center"><span id="S6.T2.1.1.5.4.3.1" class="ltx_text" style="background-color:#E6E6E6;">0.152</span></td>
<td id="S6.T2.1.1.5.4.4" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S6.T2.1.1.5.4.4.1" class="ltx_text" style="background-color:#E6E6E6;">33.213</span></td>
</tr>
<tr id="S6.T2.1.1.6.5" class="ltx_tr">
<th id="S6.T2.1.1.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">0.1</th>
<td id="S6.T2.1.1.6.5.2" class="ltx_td ltx_align_center">0.643</td>
<td id="S6.T2.1.1.6.5.3" class="ltx_td ltx_align_center">0.152</td>
<td id="S6.T2.1.1.6.5.4" class="ltx_td ltx_nopad_r ltx_align_center">33.903</td>
</tr>
<tr id="S6.T2.1.1.7.6" class="ltx_tr">
<th id="S6.T2.1.1.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">1</th>
<td id="S6.T2.1.1.7.6.2" class="ltx_td ltx_align_center ltx_border_bb">0.643</td>
<td id="S6.T2.1.1.7.6.3" class="ltx_td ltx_align_center ltx_border_bb">0.159</td>
<td id="S6.T2.1.1.7.6.4" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb">33.743</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S6.T2.5.2.1" class="ltx_text" style="font-size:90%;">Table 2</span>: </span><span id="S6.T2.3.1" class="ltx_text" style="font-size:90%;">Comparison of token retention threshold <math id="S6.T2.3.1.m1.1" class="ltx_Math" alttext="\gamma" display="inline"><semantics id="S6.T2.3.1.m1.1b"><mi id="S6.T2.3.1.m1.1.1" xref="S6.T2.3.1.m1.1.1.cmml">γ</mi><annotation-xml encoding="MathML-Content" id="S6.T2.3.1.m1.1c"><ci id="S6.T2.3.1.m1.1.1.cmml" xref="S6.T2.3.1.m1.1.1">𝛾</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.T2.3.1.m1.1d">\gamma</annotation></semantics></math> by LLaMA-2-7B <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib5" title="" class="ltx_ref">5</a>]</cite> on three benchmarks. </span></figcaption>
</figure>
<div id="S6.p1" class="ltx_para">
<p id="S6.p1.7" class="ltx_p"><span id="S6.p1.1.1" class="ltx_text ltx_font_bold">The effect of interpretation parameter <math id="S6.p1.1.1.m1.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S6.p1.1.1.m1.1a"><mi id="S6.p1.1.1.m1.1.1" xref="S6.p1.1.1.m1.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S6.p1.1.1.m1.1b"><ci id="S6.p1.1.1.m1.1.1.cmml" xref="S6.p1.1.1.m1.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.p1.1.1.m1.1c">t</annotation></semantics></math>.</span>

In our exploration of the effects of the interpretation parameter <math id="S6.p1.2.m1.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S6.p1.2.m1.1a"><mi id="S6.p1.2.m1.1.1" xref="S6.p1.2.m1.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S6.p1.2.m1.1b"><ci id="S6.p1.2.m1.1.1.cmml" xref="S6.p1.2.m1.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.p1.2.m1.1c">t</annotation></semantics></math> on performance, particularly in relation to the relative magnitude ratio of adjacent layers,
as shown in <a href="#S5.F6" title="In 5 Experiments ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">6</span></a>. We maintain all settings constant, starting from layer <math id="S6.p1.3.m2.1" class="ltx_Math" alttext="S=16" display="inline"><semantics id="S6.p1.3.m2.1a"><mrow id="S6.p1.3.m2.1.1" xref="S6.p1.3.m2.1.1.cmml"><mi id="S6.p1.3.m2.1.1.2" xref="S6.p1.3.m2.1.1.2.cmml">S</mi><mo id="S6.p1.3.m2.1.1.1" xref="S6.p1.3.m2.1.1.1.cmml">=</mo><mn id="S6.p1.3.m2.1.1.3" xref="S6.p1.3.m2.1.1.3.cmml">16</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.p1.3.m2.1b"><apply id="S6.p1.3.m2.1.1.cmml" xref="S6.p1.3.m2.1.1"><eq id="S6.p1.3.m2.1.1.1.cmml" xref="S6.p1.3.m2.1.1.1"></eq><ci id="S6.p1.3.m2.1.1.2.cmml" xref="S6.p1.3.m2.1.1.2">𝑆</ci><cn type="integer" id="S6.p1.3.m2.1.1.3.cmml" xref="S6.p1.3.m2.1.1.3">16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.p1.3.m2.1c">S=16</annotation></semantics></math> (halfway through the layers of LLaMA-3-8B), and vary the interpretation parameter <math id="S6.p1.4.m3.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S6.p1.4.m3.1a"><mi id="S6.p1.4.m3.1.1" xref="S6.p1.4.m3.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S6.p1.4.m3.1b"><ci id="S6.p1.4.m3.1.1.cmml" xref="S6.p1.4.m3.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.p1.4.m3.1c">t</annotation></semantics></math> from 0.3 to 0.7. Our findings reveal several key points:
When <math id="S6.p1.5.m4.1" class="ltx_Math" alttext="t=0.5" display="inline"><semantics id="S6.p1.5.m4.1a"><mrow id="S6.p1.5.m4.1.1" xref="S6.p1.5.m4.1.1.cmml"><mi id="S6.p1.5.m4.1.1.2" xref="S6.p1.5.m4.1.1.2.cmml">t</mi><mo id="S6.p1.5.m4.1.1.1" xref="S6.p1.5.m4.1.1.1.cmml">=</mo><mn id="S6.p1.5.m4.1.1.3" xref="S6.p1.5.m4.1.1.3.cmml">0.5</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.p1.5.m4.1b"><apply id="S6.p1.5.m4.1.1.cmml" xref="S6.p1.5.m4.1.1"><eq id="S6.p1.5.m4.1.1.1.cmml" xref="S6.p1.5.m4.1.1.1"></eq><ci id="S6.p1.5.m4.1.1.2.cmml" xref="S6.p1.5.m4.1.1.2">𝑡</ci><cn type="float" id="S6.p1.5.m4.1.1.3.cmml" xref="S6.p1.5.m4.1.1.3">0.5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.p1.5.m4.1c">t=0.5</annotation></semantics></math>, the process resembles average merging, which is less effective for cross-layer merging.
In contrast, when <math id="S6.p1.6.m5.1" class="ltx_Math" alttext="t=0.6" display="inline"><semantics id="S6.p1.6.m5.1a"><mrow id="S6.p1.6.m5.1.1" xref="S6.p1.6.m5.1.1.cmml"><mi id="S6.p1.6.m5.1.1.2" xref="S6.p1.6.m5.1.1.2.cmml">t</mi><mo id="S6.p1.6.m5.1.1.1" xref="S6.p1.6.m5.1.1.1.cmml">=</mo><mn id="S6.p1.6.m5.1.1.3" xref="S6.p1.6.m5.1.1.3.cmml">0.6</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.p1.6.m5.1b"><apply id="S6.p1.6.m5.1.1.cmml" xref="S6.p1.6.m5.1.1"><eq id="S6.p1.6.m5.1.1.1.cmml" xref="S6.p1.6.m5.1.1.1"></eq><ci id="S6.p1.6.m5.1.1.2.cmml" xref="S6.p1.6.m5.1.1.2">𝑡</ci><cn type="float" id="S6.p1.6.m5.1.1.3.cmml" xref="S6.p1.6.m5.1.1.3">0.6</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.p1.6.m5.1c">t=0.6</annotation></semantics></math> is optimal, the merged representation exhibits the most robust performance, while indicating that more information is derived from the second term (<math id="S6.p1.7.m6.1" class="ltx_Math" alttext="{\bm{x}}^{l}" display="inline"><semantics id="S6.p1.7.m6.1a"><msup id="S6.p1.7.m6.1.1" xref="S6.p1.7.m6.1.1.cmml"><mi id="S6.p1.7.m6.1.1.2" xref="S6.p1.7.m6.1.1.2.cmml">𝒙</mi><mi id="S6.p1.7.m6.1.1.3" xref="S6.p1.7.m6.1.1.3.cmml">l</mi></msup><annotation-xml encoding="MathML-Content" id="S6.p1.7.m6.1b"><apply id="S6.p1.7.m6.1.1.cmml" xref="S6.p1.7.m6.1.1"><csymbol cd="ambiguous" id="S6.p1.7.m6.1.1.1.cmml" xref="S6.p1.7.m6.1.1">superscript</csymbol><ci id="S6.p1.7.m6.1.1.2.cmml" xref="S6.p1.7.m6.1.1.2">𝒙</ci><ci id="S6.p1.7.m6.1.1.3.cmml" xref="S6.p1.7.m6.1.1.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.p1.7.m6.1c">{\bm{x}}^{l}</annotation></semantics></math>) of the SLERP.</p>
</div>
<div id="S6.p2" class="ltx_para">
<p id="S6.p2.4" class="ltx_p">The frequency results also indicate that the high frequencies are clustered around 0.4 and 0.6, corroborating our optimal <math id="S6.p2.1.m1.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S6.p2.1.m1.1a"><mi id="S6.p2.1.m1.1.1" xref="S6.p2.1.m1.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S6.p2.1.m1.1b"><ci id="S6.p2.1.m1.1.1.cmml" xref="S6.p2.1.m1.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.p2.1.m1.1c">t</annotation></semantics></math>. Moreover, there is a strong correlation between the optimal <math id="S6.p2.2.m2.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S6.p2.2.m2.1a"><mi id="S6.p2.2.m2.1.1" xref="S6.p2.2.m2.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S6.p2.2.m2.1b"><ci id="S6.p2.2.m2.1.1.cmml" xref="S6.p2.2.m2.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.p2.2.m2.1c">t</annotation></semantics></math> and the high frequency of the relative magnitude ratio of adjacent layers. This finding provides an opportunity to utilize the relative magnitude ratio to dynamically determine the interpretation parameter <math id="S6.p2.3.m3.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S6.p2.3.m3.1a"><mi id="S6.p2.3.m3.1.1" xref="S6.p2.3.m3.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S6.p2.3.m3.1b"><ci id="S6.p2.3.m3.1.1.cmml" xref="S6.p2.3.m3.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.p2.3.m3.1c">t</annotation></semantics></math>.
Dynamic <math id="S6.p2.4.m4.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S6.p2.4.m4.1a"><mi id="S6.p2.4.m4.1.1" xref="S6.p2.4.m4.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S6.p2.4.m4.1b"><ci id="S6.p2.4.m4.1.1.cmml" xref="S6.p2.4.m4.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.p2.4.m4.1c">t</annotation></semantics></math> allows for more flexible weight control in SLERP merging for each layer-wise operation, thereby showing potential for further exploration.</p>
</div>
<div id="S6.p3" class="ltx_para">
<p id="S6.p3.4" class="ltx_p"><span id="S6.p3.1.1" class="ltx_text ltx_font_bold">The effect of token retention threshold <math id="S6.p3.1.1.m1.1" class="ltx_Math" alttext="\gamma" display="inline"><semantics id="S6.p3.1.1.m1.1a"><mi id="S6.p3.1.1.m1.1.1" xref="S6.p3.1.1.m1.1.1.cmml">γ</mi><annotation-xml encoding="MathML-Content" id="S6.p3.1.1.m1.1b"><ci id="S6.p3.1.1.m1.1.1.cmml" xref="S6.p3.1.1.m1.1.1">𝛾</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.p3.1.1.m1.1c">\gamma</annotation></semantics></math>.</span>
We investigate the impact of the token retention threshold <math id="S6.p3.2.m1.1" class="ltx_Math" alttext="\gamma" display="inline"><semantics id="S6.p3.2.m1.1a"><mi id="S6.p3.2.m1.1.1" xref="S6.p3.2.m1.1.1.cmml">γ</mi><annotation-xml encoding="MathML-Content" id="S6.p3.2.m1.1b"><ci id="S6.p3.2.m1.1.1.cmml" xref="S6.p3.2.m1.1.1">𝛾</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.p3.2.m1.1c">\gamma</annotation></semantics></math> on model performance across the three datasets, as shown in <a href="#S6.T2" title="In 6 Ablation Study ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">2</span></a>.
A larger <math id="S6.p3.3.m2.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S6.p3.3.m2.1a"><mi id="S6.p3.3.m2.1.1" xref="S6.p3.3.m2.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S6.p3.3.m2.1b"><ci id="S6.p3.3.m2.1.1.cmml" xref="S6.p3.3.m2.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.p3.3.m2.1c">t</annotation></semantics></math> generally means retaining more tokens for improved performance, but this comes at the cost of increased memory demand.
The results suggest that setting <math id="S6.p3.4.m3.1" class="ltx_Math" alttext="\gamma" display="inline"><semantics id="S6.p3.4.m3.1a"><mi id="S6.p3.4.m3.1.1" xref="S6.p3.4.m3.1.1.cmml">γ</mi><annotation-xml encoding="MathML-Content" id="S6.p3.4.m3.1b"><ci id="S6.p3.4.m3.1.1.cmml" xref="S6.p3.4.m3.1.1">𝛾</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.p3.4.m3.1c">\gamma</annotation></semantics></math> to 0.05 achieves the best balance between performance and efficiency.</p>
</div>
</section>
<section id="S7" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">7 </span>Conclusion and Future Work</h2>

<div id="S7.p1" class="ltx_para">
<p id="S7.p1.1" class="ltx_p">This paper presents a pioneering exploration of KV cache compression in the depth dimension, addressing a significant memory bottleneck in LLMs. Our proposed MiniCache offers a simple, effective, and training-free approach to compressing KV caches by leveraging the notable high similarities between KV caches in neighboring layers, starting from the midpoint of LLMs.
We have demonstrated that MiniCache can significantly reduce the memory footprint required for LLM inference by up to 41%, while simultaneously enhancing throughput by approximately five times compared to the FP16 baseline.
In conclusion, MiniCache
significantly advances the field of KV cache compression,
offering a state-of-the-art balance between efficiency and performance. Future work will focus on enhancing the compression ratio by cross-multiple-layer merging, developing advanced merging algorithms such as Spherical Cubic Interpolation <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib75" title="" class="ltx_ref">75</a>]</cite>, and further optimizing memory usage for large-scale deployments in diverse application scenarios.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[1]</span>
<span class="ltx_bibblock">
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, <span id="bib.bib1.1.1" class="ltx_text ltx_font_italic">et al.</span>, “Language models are few-shot learners,” in <span id="bib.bib1.2.2" class="ltx_text ltx_font_italic">NeurIPS</span>, vol. 33, pp. 1877–1901, 2020.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[2]</span>
<span class="ltx_bibblock">
OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, and OTHERS, “Gpt-4 technical report,” 2023.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[3]</span>
<span class="ltx_bibblock">
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, <span id="bib.bib3.1.1" class="ltx_text ltx_font_italic">et al.</span>, “Training language models to follow instructions with human feedback,” in <span id="bib.bib3.2.2" class="ltx_text ltx_font_italic">NeurIPS</span>, vol. 35, pp. 27730–27744, 2022.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[4]</span>
<span class="ltx_bibblock">
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, <span id="bib.bib4.1.1" class="ltx_text ltx_font_italic">et al.</span>, “Llama: Open and efficient foundation language models,” <span id="bib.bib4.2.2" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2302.13971</span>, 2023.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[5]</span>
<span class="ltx_bibblock">
H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, <span id="bib.bib5.1.1" class="ltx_text ltx_font_italic">et al.</span>, “Llama 2: Open foundation and fine-tuned chat models,” <span id="bib.bib5.2.2" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2307.09288</span>, 2023.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[6]</span>
<span class="ltx_bibblock">
“Introducing meta llama 3: The most capable openly available llm to date.” <a target="_blank" href="https://ai.meta.com/blog/meta-llama-3/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://ai.meta.com/blog/meta-llama-3/</a>, 2024.

</span>
<span class="ltx_bibblock">Accessed: 2024-05-04.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[7]</span>
<span class="ltx_bibblock">
J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” <span id="bib.bib7.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2001.08361</span>, 2020.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[8]</span>
<span class="ltx_bibblock">
G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, <span id="bib.bib8.1.1" class="ltx_text ltx_font_italic">et al.</span>, “Gemini: a family of highly capable multimodal models,” <span id="bib.bib8.2.2" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2312.11805</span>, 2023.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[9]</span>
<span class="ltx_bibblock">
R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean, “Efficiently scaling transformer inference,” <span id="bib.bib9.1.1" class="ltx_text ltx_font_italic">Proceedings of Machine Learning and Systems</span>, vol. 5, 2023.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[10]</span>
<span class="ltx_bibblock">
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, <span id="bib.bib10.1.1" class="ltx_text ltx_font_italic">et al.</span>, “Training verifiers to solve math word problems,” <span id="bib.bib10.2.2" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2110.14168</span>, 2021.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[11]</span>
<span class="ltx_bibblock">
Z. Liu, J. Yuan, H. Jin, S. Zhong, Z. Xu, V. Braverman, B. Chen, and X. Hu, “Kivi: Plug-and-play 2bit kv cache quantization with streaming asymmetric quantization,” <span id="bib.bib11.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2402.02750</span>, 2024.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[12]</span>
<span class="ltx_bibblock">
H. Kang, Q. Zhang, S. Kundu, G. Jeong, Z. Liu, T. Krishna, and T. Zhao, “Gear: An efficient kv cache compression recipefor near-lossless generative inference of llm,” <span id="bib.bib12.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2403.05527</span>, 2024.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[13]</span>
<span class="ltx_bibblock">
Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. Ré, I. Stoica, and C. Zhang, “Flexgen: High-throughput generative inference of large language models with a single gpu,” in <span id="bib.bib13.1.1" class="ltx_text ltx_font_italic">ICML</span>, pp. 31094–31116, PMLR, 2023.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[14]</span>
<span class="ltx_bibblock">
Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett, <span id="bib.bib14.1.1" class="ltx_text ltx_font_italic">et al.</span>, “H2o: Heavy-hitter oracle for efficient generative inference of large language models,” in <span id="bib.bib14.2.2" class="ltx_text ltx_font_italic">NeurIPS</span>, vol. 36, 2024.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[15]</span>
<span class="ltx_bibblock">
G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis, “Efficient streaming language models with attention sinks,” in <span id="bib.bib15.1.1" class="ltx_text ltx_font_italic">ICLR</span>, 2024.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[16]</span>
<span class="ltx_bibblock">
S. Ge, Y. Zhang, L. Liu, M. Zhang, J. Han, and J. Gao, “Model tells you what to discard: Adaptive kv cache compression for llms,” <span id="bib.bib16.1.1" class="ltx_text ltx_font_italic">ICLR</span>, 2024.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[17]</span>
<span class="ltx_bibblock">
D. Raposo, S. Ritter, B. Richards, T. Lillicrap, P. C. Humphreys, and A. Santoro, “Mixture-of-depths: Dynamically allocating compute in transformer-based language models,” <span id="bib.bib17.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2404.02258</span>, 2024.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[18]</span>
<span class="ltx_bibblock">
W. Zhou, C. Xu, T. Ge, J. McAuley, K. Xu, and F. Wei, “Bert loses patience: Fast and robust inference with early exit,” in <span id="bib.bib18.1.1" class="ltx_text ltx_font_italic">NeurIPS</span>, vol. 33, pp. 18330–18341, 2020.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[19]</span>
<span class="ltx_bibblock">
M. Elhoushi, A. Shrivastava, D. Liskovich, B. Hosmer, B. Wasti, L. Lai, A. Mahmoud, B. Acun, S. Agarwal, A. Roman, <span id="bib.bib19.1.1" class="ltx_text ltx_font_italic">et al.</span>, “Layer skip: Enabling early exit inference and self-speculative decoding,” <span id="bib.bib19.2.2" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2404.16710</span>, 2024.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[20]</span>
<span class="ltx_bibblock">
A. Gromov, K. Tirumala, H. Shapourian, P. Glorioso, and D. A. Roberts, “The unreasonable ineffectiveness of the deeper layers,” <span id="bib.bib20.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2403.17887</span>, 2024.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[21]</span>
<span class="ltx_bibblock">
T. Salimans and D. P. Kingma, “Weight normalization: A simple reparameterization to accelerate training of deep neural networks,” in <span id="bib.bib21.1.1" class="ltx_text ltx_font_italic">NeurIPS</span>, vol. 29, 2016.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[22]</span>
<span class="ltx_bibblock">
A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand, <span id="bib.bib22.1.1" class="ltx_text ltx_font_italic">et al.</span>, “Mixtral of experts,” <span id="bib.bib22.2.2" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2401.04088</span>, 2024.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[23]</span>
<span class="ltx_bibblock">
M. Abdin, S. A. Jacobs, A. A. Awan, J. Aneja, A. Awadallah, H. Awadalla, N. Bach, A. Bahree, A. Bakhtiari, H. Behl, <span id="bib.bib23.1.1" class="ltx_text ltx_font_italic">et al.</span>, “Phi-3 technical report: A highly capable language model locally on your phone,” <span id="bib.bib23.2.2" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2404.14219</span>, 2024.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[24]</span>
<span class="ltx_bibblock">
M. Roemmele, C. A. Bejan, and A. S. Gordon, “Choice of plausible alternatives: An evaluation of commonsense causal reasoning.,” in <span id="bib.bib24.1.1" class="ltx_text ltx_font_italic">AAAI spring symposium: logical formalizations of commonsense reasoning</span>, pp. 90–95, 2011.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[25]</span>
<span class="ltx_bibblock">
A. Amini, S. Gabriel, S. Lin, R. Koncel-Kedziorski, Y. Choi, and H. Hajishirzi, “Mathqa: Towards interpretable math word problem solving with operation-based formalisms,” in <span id="bib.bib25.1.1" class="ltx_text ltx_font_italic">NAACL</span>, pp. 2357–2367, 2019.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[26]</span>
<span class="ltx_bibblock">
T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal, “Can a suit of armor conduct electricity? a new dataset for open book question answering,” in <span id="bib.bib26.1.1" class="ltx_text ltx_font_italic">EMNLP</span>, 2018.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[27]</span>
<span class="ltx_bibblock">
Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi, “Piqa: Reasoning about physical commonsense in natural language,” in <span id="bib.bib27.1.1" class="ltx_text ltx_font_italic">AAAI</span>, 2020.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[28]</span>
<span class="ltx_bibblock">
A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “Glue: A multi-task benchmark and analysis platform for natural language understanding,” <span id="bib.bib28.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:1804.07461</span>, 2018.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[29]</span>
<span class="ltx_bibblock">
K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi, “Winogrande: An adversarial winograd schema challenge at scale,” <span id="bib.bib29.1.1" class="ltx_text ltx_font_italic">Communications of the ACM</span>, vol. 64, no. 9, pp. 99–106, 2021.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[30]</span>
<span class="ltx_bibblock">
S. Narayan, S. B. Cohen, and M. Lapata, “Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization,” <span id="bib.bib30.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:1808.08745</span>, 2018.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[31]</span>
<span class="ltx_bibblock">
R. Nallapati, B. Zhou, C. Gulcehre, B. Xiang, <span id="bib.bib31.1.1" class="ltx_text ltx_font_italic">et al.</span>, “Abstractive text summarization using sequence-to-sequence rnns and beyond,” <span id="bib.bib31.2.2" class="ltx_text ltx_font_italic">arXiv preprint arXiv:1602.06023</span>, 2016.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[32]</span>
<span class="ltx_bibblock">
L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou, “A framework for few-shot language model evaluation,” 12 2023.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[33]</span>
<span class="ltx_bibblock">
Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, <span id="bib.bib33.1.1" class="ltx_text ltx_font_italic">et al.</span>, “Longbench: A bilingual, multitask benchmark for long context understanding,” <span id="bib.bib33.2.2" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2308.14508</span>, 2023.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[34]</span>
<span class="ltx_bibblock">
L. Del Corro, A. Del Giorno, S. Agarwal, B. Yu, A. Awadallah, and S. Mukherjee, “Skipdecode: Autoregressive skip decoding with batching and caching for efficient llm inference,” <span id="bib.bib34.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2307.02628</span>, 2023.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[35]</span>
<span class="ltx_bibblock">
T. Schuster, A. Fisch, J. Gupta, M. Dehghani, D. Bahri, V. Tran, Y. Tay, and D. Metzler, “Confident adaptive language modeling,” in <span id="bib.bib35.1.1" class="ltx_text ltx_font_italic">NeurIPS</span>, vol. 35, pp. 17456–17472, 2022.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[36]</span>
<span class="ltx_bibblock">
H. Wu and K. Tu, “Layer-condensed kv cache for efficient inference of large language models,” 2024.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[37]</span>
<span class="ltx_bibblock">
W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” <span id="bib.bib37.1.1" class="ltx_text ltx_font_italic">Journal of Machine Learning Research</span>, vol. 23, no. 120, pp. 1–39, 2022.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[38]</span>
<span class="ltx_bibblock">
D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen, “Gshard: Scaling giant models with conditional computation and automatic sharding,” <span id="bib.bib38.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2006.16668</span>, 2020.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[39]</span>
<span class="ltx_bibblock">
D. Dai, C. Deng, C. Zhao, R. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, <span id="bib.bib39.1.1" class="ltx_text ltx_font_italic">et al.</span>, “Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models,” <span id="bib.bib39.2.2" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2401.06066</span>, 2024.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[40]</span>
<span class="ltx_bibblock">
C. Hwang, W. Cui, Y. Xiong, Z. Yang, Z. Liu, H. Hu, Z. Wang, R. Salas, J. Jose, P. Ram, <span id="bib.bib40.1.1" class="ltx_text ltx_font_italic">et al.</span>, “Tutel: Adaptive mixture-of-experts at scale,” <span id="bib.bib40.2.2" class="ltx_text ltx_font_italic">Proceedings of Machine Learning and Systems</span>, vol. 5, 2023.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[41]</span>
<span class="ltx_bibblock">
DeepSeek-AI, “Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model,” <span id="bib.bib41.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2405.04434</span>, 2024.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[42]</span>
<span class="ltx_bibblock">
J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai, “Gqa: Training generalized multi-query transformer models from multi-head checkpoints,” <span id="bib.bib42.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2305.13245</span>, 2023.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[43]</span>
<span class="ltx_bibblock">
N. Shazeer, “Fast transformer decoding: One write-head is all you need,” <span id="bib.bib43.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:1911.02150</span>, 2019.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[44]</span>
<span class="ltx_bibblock">
K. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Davis, A. Mohiuddin, L. Kaiser, <span id="bib.bib44.1.1" class="ltx_text ltx_font_italic">et al.</span>, “Rethinking attention with performers,” <span id="bib.bib44.2.2" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2009.14794</span>, 2020.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[45]</span>
<span class="ltx_bibblock">
H. Peng, N. Pappas, D. Yogatama, R. Schwartz, N. A. Smith, and L. Kong, “Random feature attention,” in <span id="bib.bib45.1.1" class="ltx_text ltx_font_italic">ICLR</span>, 2021.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[46]</span>
<span class="ltx_bibblock">
T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré, “Flashattention: Fast and memory-efficient exact attention with io-awareness,” in <span id="bib.bib46.1.1" class="ltx_text ltx_font_italic">NeurIPS</span>, vol. 35, pp. 16344–16359, 2022.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[47]</span>
<span class="ltx_bibblock">
T. Dao, “Flashattention-2: Faster attention with better parallelism and work partitioning,” <span id="bib.bib47.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2307.08691</span>, 2023.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[48]</span>
<span class="ltx_bibblock">
S. Wang, B. Z. Li, M. Khabsa, H. Fang, and H. Ma, “Linformer: Self-attention with linear complexity,” <span id="bib.bib48.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2006.04768</span>, 2020.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[49]</span>
<span class="ltx_bibblock">
X. Ma, X. Kong, S. Wang, C. Zhou, J. May, H. Ma, and L. Zettlemoyer, “Luna: Linear unified nested attention,” in <span id="bib.bib49.1.1" class="ltx_text ltx_font_italic">NeurIPS</span>, vol. 34, pp. 2441–2453, 2021.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[50]</span>
<span class="ltx_bibblock">
J. Lee, Y. Lee, J. Kim, A. Kosiorek, S. Choi, and Y. W. Teh, “Set transformer: A framework for attention-based permutation-invariant neural networks,” in <span id="bib.bib50.1.1" class="ltx_text ltx_font_italic">ICML</span>, pp. 3744–3753, PMLR, 2019.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[51]</span>
<span class="ltx_bibblock">
J. Lin, J. Tang, H. Tang, S. Yang, X. Dang, and S. Han, “Awq: Activation-aware weight quantization for llm compression and acceleration,” <span id="bib.bib51.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2306.00978</span>, 2023.

</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[52]</span>
<span class="ltx_bibblock">
T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” in <span id="bib.bib52.1.1" class="ltx_text ltx_font_italic">NeurIPS</span>, vol. 36, 2023.

</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[53]</span>
<span class="ltx_bibblock">
T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer, “Llm.int8(): 8-bit matrix multiplication for transformers at scale,” in <span id="bib.bib53.1.1" class="ltx_text ltx_font_italic">NeurIPS</span>, vol. 35, pp. 30318–30332, 2022.

</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[54]</span>
<span class="ltx_bibblock">
J. Liu, R. Gong, X. Wei, Z. Dong, J. Cai, and B. Zhuang, “Qllm: Accurate and efficient low-bitwidth quantization for large language models,” in <span id="bib.bib54.1.1" class="ltx_text ltx_font_italic">ICLR</span>, 2024.

</span>
</li>
<li id="bib.bib55" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[55]</span>
<span class="ltx_bibblock">
M. Zhang, C. Shen, Z. Yang, L. Ou, X. Yu, B. Zhuang, <span id="bib.bib55.1.1" class="ltx_text ltx_font_italic">et al.</span>, “Loraprune: Pruning meets low-rank parameter-efficient fine-tuning,” in <span id="bib.bib55.2.2" class="ltx_text ltx_font_italic">ACL findings</span>, 2024.

</span>
</li>
<li id="bib.bib56" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[56]</span>
<span class="ltx_bibblock">
N. Kitaev, Ł. Kaiser, and A. Levskaya, “Reformer: The efficient transformer,” <span id="bib.bib56.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2001.04451</span>, 2020.

</span>
</li>
<li id="bib.bib57" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[57]</span>
<span class="ltx_bibblock">
S. K. Ainsworth, J. Hayase, and S. Srinivasa, “Git re-basin: Merging models modulo permutation symmetries,” in <span id="bib.bib57.1.1" class="ltx_text ltx_font_italic">ICLR</span>, 2023.

</span>
</li>
<li id="bib.bib58" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[58]</span>
<span class="ltx_bibblock">
R. Entezari, H. Sedghi, O. Saukh, and B. Neyshabur, “The role of permutation invariance in linear mode connectivity of neural networks,” <span id="bib.bib58.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2110.06296</span>, 2021.

</span>
</li>
<li id="bib.bib59" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[59]</span>
<span class="ltx_bibblock">
T. Garipov, P. Izmailov, D. Podoprikhin, D. P. Vetrov, and A. G. Wilson, “Loss surfaces, mode connectivity, and fast ensembling of dnns,” in <span id="bib.bib59.1.1" class="ltx_text ltx_font_italic">NeurIPS</span>, vol. 31, 2018.

</span>
</li>
<li id="bib.bib60" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[60]</span>
<span class="ltx_bibblock">
M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith, <span id="bib.bib60.1.1" class="ltx_text ltx_font_italic">et al.</span>, “Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time,” in <span id="bib.bib60.2.2" class="ltx_text ltx_font_italic">ICML</span>, pp. 23965–23998, PMLR, 2022.

</span>
</li>
<li id="bib.bib61" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[61]</span>
<span class="ltx_bibblock">
P. Yadav, D. Tam, L. Choshen, C. A. Raffel, and M. Bansal, “Ties-merging: Resolving interference when merging models,” in <span id="bib.bib61.1.1" class="ltx_text ltx_font_italic">NeurIPS</span>, vol. 36, 2023.

</span>
</li>
<li id="bib.bib62" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[62]</span>
<span class="ltx_bibblock">
M. Davari and E. Belilovsky, “Model breadcrumbs: Scaling multi-task model merging with sparse masks,” <span id="bib.bib62.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2312.06795</span>, 2023.

</span>
</li>
<li id="bib.bib63" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[63]</span>
<span class="ltx_bibblock">
L. Yu, B. Yu, H. Yu, F. Huang, and Y. Li, “Language models are super mario: Absorbing abilities from homologous models as a free lunch,” <span id="bib.bib63.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2311.03099</span>, 2023.

</span>
</li>
<li id="bib.bib64" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[64]</span>
<span class="ltx_bibblock">
K. Shoemake, “Animating rotation with quaternion curves,” in <span id="bib.bib64.1.1" class="ltx_text ltx_font_italic">Proceedings of the 12th annual conference on Computer graphics and interactive techniques</span>, pp. 245–254, 1985.

</span>
</li>
<li id="bib.bib65" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[65]</span>
<span class="ltx_bibblock">
M. S. Matena and C. A. Raffel, “Merging models with fisher-weighted averaging,” in <span id="bib.bib65.1.1" class="ltx_text ltx_font_italic">NeurIPS</span>, vol. 35, pp. 17703–17716, 2022.

</span>
</li>
<li id="bib.bib66" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[66]</span>
<span class="ltx_bibblock">
X. Jin, X. Ren, D. Preotiuc-Pietro, and P. Cheng, “Dataless knowledge fusion by merging weights of language models,” <span id="bib.bib66.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2212.09849</span>, 2022.

</span>
</li>
<li id="bib.bib67" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[67]</span>
<span class="ltx_bibblock">
Y. Chen, X. Pan, Y. Li, B. Ding, and J. Zhou, “Ee-llm: Large-scale training and inference of early-exit large language models with 3d parallelism,” <span id="bib.bib67.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2312.04916</span>, 2023.

</span>
</li>
<li id="bib.bib68" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[68]</span>
<span class="ltx_bibblock">
S. Reddy, D. Chen, and C. D. Manning, “Coqa: A conversational question answering challenge,” <span id="bib.bib68.1.1" class="ltx_text ltx_font_italic">Transactions of the Association for Computational Linguistics</span>, vol. 7, pp. 249–266, 2019.

</span>
</li>
<li id="bib.bib69" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[69]</span>
<span class="ltx_bibblock">
S. Lin, J. Hilton, and O. Evans, “Truthfulqa: Measuring how models mimic human falsehoods,” <span id="bib.bib69.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2109.07958</span>, 2021.

</span>
</li>
<li id="bib.bib70" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[70]</span>
<span class="ltx_bibblock">
G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “Smoothquant: Accurate and efficient post-training quantization for large language models,” in <span id="bib.bib70.1.1" class="ltx_text ltx_font_italic">International Conference on Machine Learning</span>, pp. 38087–38099, PMLR, 2023.

</span>
</li>
<li id="bib.bib71" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[71]</span>
<span class="ltx_bibblock">
J. Liu, R. Gong, X. Wei, Z. Dong, J. Cai, and B. Zhuang, “Qllm: Accurate and efficient low-bitwidth quantization for large language models,” in <span id="bib.bib71.1.1" class="ltx_text ltx_font_italic">ICLR</span>, 2024.

</span>
</li>
<li id="bib.bib72" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[72]</span>
<span class="ltx_bibblock">
S.-Y. Liu, C.-Y. Wang, H. Yin, P. Molchanov, Y.-C. F. Wang, K.-T. Cheng, and M.-H. Chen, “Dora: Weight-decomposed low-rank adaptation,” <span id="bib.bib72.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2402.09353</span>, 2024.

</span>
</li>
<li id="bib.bib73" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[73]</span>
<span class="ltx_bibblock">
M. Nagel, R. A. Amjad, M. Van Baalen, C. Louizos, and T. Blankevoort, “Up or down? adaptive rounding for post-training quantization,” in <span id="bib.bib73.1.1" class="ltx_text ltx_font_italic">International Conference on Machine Learning</span>, pp. 7197–7206, PMLR, 2020.

</span>
</li>
<li id="bib.bib74" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[74]</span>
<span class="ltx_bibblock">
W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in <span id="bib.bib74.1.1" class="ltx_text ltx_font_italic">Proceedings of the 29th Symposium on Operating Systems Principles</span>, pp. 611–626, 2023.

</span>
</li>
<li id="bib.bib75" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[75]</span>
<span class="ltx_bibblock">
D. H. Eberly, “Quaternion algebra and calculus,” 2002.

</span>
</li>
<li id="bib.bib76" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[76]</span>
<span class="ltx_bibblock">
“Stanford crfm.” <a target="_blank" href="https://crfm.stanford.edu/2023/10/12/flashdecoding.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://crfm.stanford.edu/2023/10/12/flashdecoding.html</a>, 2024.

</span>
<span class="ltx_bibblock">Accessed: 2024-05-04.

</span>
</li>
<li id="bib.bib77" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[77]</span>
<span class="ltx_bibblock">
Y. Liu, H. Li, K. Du, J. Yao, Y. Cheng, Y. Huang, S. Lu, M. Maire, H. Hoffmann, A. Holtzman, <span id="bib.bib77.1.1" class="ltx_text ltx_font_italic">et al.</span>, “Cachegen: Fast context loading for language model applications,” <span id="bib.bib77.2.2" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2310.07240</span>, 2023.

</span>
</li>
<li id="bib.bib78" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[78]</span>
<span class="ltx_bibblock">
L. Chen, H. Zhao, T. Liu, S. Bai, J. Lin, C. Zhou, and B. Chang, “An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models,” <span id="bib.bib78.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2403.06764</span>, 2024.

</span>
</li>
<li id="bib.bib79" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[79]</span>
<span class="ltx_bibblock">
S. Wei, T. Ye, S. Zhang, Y. Tang, and J. Liang, “Joint token pruning and squeezing towards more aggressive compression of vision transformers,” in <span id="bib.bib79.1.1" class="ltx_text ltx_font_italic">CVPR</span>, pp. 2092–2101, 2023.

</span>
</li>
<li id="bib.bib80" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[80]</span>
<span class="ltx_bibblock">
L. Gao, J. Tow, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, K. McDonell, N. Muennighoff, J. Phang, L. Reynolds, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou, “A framework for few-shot language model evaluation,” 2021.

</span>
</li>
<li id="bib.bib81" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[81]</span>
<span class="ltx_bibblock">
W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in <span id="bib.bib81.1.1" class="ltx_text ltx_font_italic">Proceedings of the 29th Symposium on Operating Systems Principles</span>, pp. 611–626, 2023.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<div id="p1" class="ltx_para">
<p id="p1.1" class="ltx_p ltx_align_center"><span id="p1.1.1" class="ltx_text ltx_font_bold" style="font-size:144%;">Appendix<span id="p1.1.1.1" class="ltx_text ltx_font_medium">
</span></span></p>
</div>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Additional Experiments Results</h2>

<div id="A1.p1" class="ltx_para">
<p id="A1.p1.1" class="ltx_p"><span id="A1.p1.1.1" class="ltx_text ltx_font_bold">Comparison with token sparsity methods</span>. We also compare MiniCache with the sparsity-based method H2O <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib14" title="" class="ltx_ref">14</a>]</cite>. Our method outperforms H2O in most tasks on LongBench. Notably, our approach addresses inter-layer redundancy, whereas H2O focuses on intra-layer redundancy. Our approaches are orthogonal to sparsity-based methods.</p>
</div>
<figure id="A1.T1" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="A1.T1.2.1.1" class="ltx_text" style="font-size:90%;">Table A</span>: </span><span id="A1.T1.3.2" class="ltx_text" style="font-size:90%;">Compare MiniCache with token sparsity based method H2O, using mistral-7B-instruct on LongBench dataset.</span></figcaption>
<div id="A1.T1.4" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:35.6pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-221.3pt,18.2pt) scale(0.494865687430815,0.494865687430815) ;">
<table id="A1.T1.4.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T1.4.1.1.1" class="ltx_tr">
<th id="A1.T1.4.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt"><span id="A1.T1.4.1.1.1.1.1" class="ltx_text ltx_font_bold">Methods</span></th>
<th id="A1.T1.4.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">NrtvQA</th>
<th id="A1.T1.4.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Qasper</th>
<th id="A1.T1.4.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">MF-en</th>
<th id="A1.T1.4.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">HotpotQA</th>
<th id="A1.T1.4.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">2WikiMQA</th>
<th id="A1.T1.4.1.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Musique</th>
<th id="A1.T1.4.1.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">GovReport</th>
<th id="A1.T1.4.1.1.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">QMSum</th>
<th id="A1.T1.4.1.1.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">MultiNews</th>
<th id="A1.T1.4.1.1.1.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">TREC</th>
<th id="A1.T1.4.1.1.1.12" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">TriviaQA</th>
<th id="A1.T1.4.1.1.1.13" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">SAMSum</th>
<th id="A1.T1.4.1.1.1.14" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">PCount</th>
<th id="A1.T1.4.1.1.1.15" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">PRe</th>
<th id="A1.T1.4.1.1.1.16" class="ltx_td ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt">Lcc</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T1.4.1.2.1" class="ltx_tr">
<th id="A1.T1.4.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Baseline</th>
<td id="A1.T1.4.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t">26.82</td>
<td id="A1.T1.4.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t">33.06</td>
<td id="A1.T1.4.1.2.1.4" class="ltx_td ltx_align_center ltx_border_t">49.28</td>
<td id="A1.T1.4.1.2.1.5" class="ltx_td ltx_align_center ltx_border_t">42.77</td>
<td id="A1.T1.4.1.2.1.6" class="ltx_td ltx_align_center ltx_border_t">27.33</td>
<td id="A1.T1.4.1.2.1.7" class="ltx_td ltx_align_center ltx_border_t">19.27</td>
<td id="A1.T1.4.1.2.1.8" class="ltx_td ltx_align_center ltx_border_t">32.85</td>
<td id="A1.T1.4.1.2.1.9" class="ltx_td ltx_align_center ltx_border_t">24.25</td>
<td id="A1.T1.4.1.2.1.10" class="ltx_td ltx_align_center ltx_border_t">27.06</td>
<td id="A1.T1.4.1.2.1.11" class="ltx_td ltx_align_center ltx_border_t">71.0</td>
<td id="A1.T1.4.1.2.1.12" class="ltx_td ltx_align_center ltx_border_t">86.23</td>
<td id="A1.T1.4.1.2.1.13" class="ltx_td ltx_align_center ltx_border_t">42.98</td>
<td id="A1.T1.4.1.2.1.14" class="ltx_td ltx_align_center ltx_border_t">2.75</td>
<td id="A1.T1.4.1.2.1.15" class="ltx_td ltx_align_center ltx_border_t">86.98</td>
<td id="A1.T1.4.1.2.1.16" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">55.51</td>
</tr>
<tr id="A1.T1.4.1.3.2" class="ltx_tr">
<th id="A1.T1.4.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">H2O<cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib14" title="" class="ltx_ref">14</a>]</cite>
</th>
<td id="A1.T1.4.1.3.2.2" class="ltx_td ltx_align_center">22.61</td>
<td id="A1.T1.4.1.3.2.3" class="ltx_td ltx_align_center">29.06</td>
<td id="A1.T1.4.1.3.2.4" class="ltx_td ltx_align_center">47.22</td>
<td id="A1.T1.4.1.3.2.5" class="ltx_td ltx_align_center">36.54</td>
<td id="A1.T1.4.1.3.2.6" class="ltx_td ltx_align_center">20.6</td>
<td id="A1.T1.4.1.3.2.7" class="ltx_td ltx_align_center">16.25</td>
<td id="A1.T1.4.1.3.2.8" class="ltx_td ltx_align_center">30.0</td>
<td id="A1.T1.4.1.3.2.9" class="ltx_td ltx_align_center">23.8</td>
<td id="A1.T1.4.1.3.2.10" class="ltx_td ltx_align_center">26.75</td>
<td id="A1.T1.4.1.3.2.11" class="ltx_td ltx_align_center">70.5</td>
<td id="A1.T1.4.1.3.2.12" class="ltx_td ltx_align_center">86.16</td>
<td id="A1.T1.4.1.3.2.13" class="ltx_td ltx_align_center">42.97</td>
<td id="A1.T1.4.1.3.2.14" class="ltx_td ltx_align_center">3.46</td>
<td id="A1.T1.4.1.3.2.15" class="ltx_td ltx_align_center">86.38</td>
<td id="A1.T1.4.1.3.2.16" class="ltx_td ltx_nopad_r ltx_align_center">53.72</td>
</tr>
<tr id="A1.T1.4.1.4.3" class="ltx_tr" style="background-color:#E6E6E6;">
<th id="A1.T1.4.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb"><span id="A1.T1.4.1.4.3.1.1" class="ltx_text" style="background-color:#E6E6E6;">MiniCache</span></th>
<td id="A1.T1.4.1.4.3.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="A1.T1.4.1.4.3.2.1" class="ltx_text" style="background-color:#E6E6E6;">27.04</span></td>
<td id="A1.T1.4.1.4.3.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="A1.T1.4.1.4.3.3.1" class="ltx_text" style="background-color:#E6E6E6;">32.59</span></td>
<td id="A1.T1.4.1.4.3.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="A1.T1.4.1.4.3.4.1" class="ltx_text" style="background-color:#E6E6E6;">49.38</span></td>
<td id="A1.T1.4.1.4.3.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="A1.T1.4.1.4.3.5.1" class="ltx_text" style="background-color:#E6E6E6;">43.91</span></td>
<td id="A1.T1.4.1.4.3.6" class="ltx_td ltx_align_center ltx_border_bb"><span id="A1.T1.4.1.4.3.6.1" class="ltx_text" style="background-color:#E6E6E6;">24.97</span></td>
<td id="A1.T1.4.1.4.3.7" class="ltx_td ltx_align_center ltx_border_bb"><span id="A1.T1.4.1.4.3.7.1" class="ltx_text" style="background-color:#E6E6E6;">18.3</span></td>
<td id="A1.T1.4.1.4.3.8" class="ltx_td ltx_align_center ltx_border_bb"><span id="A1.T1.4.1.4.3.8.1" class="ltx_text" style="background-color:#E6E6E6;">31.46</span></td>
<td id="A1.T1.4.1.4.3.9" class="ltx_td ltx_align_center ltx_border_bb"><span id="A1.T1.4.1.4.3.9.1" class="ltx_text" style="background-color:#E6E6E6;">23.85</span></td>
<td id="A1.T1.4.1.4.3.10" class="ltx_td ltx_align_center ltx_border_bb"><span id="A1.T1.4.1.4.3.10.1" class="ltx_text" style="background-color:#E6E6E6;">26.64</span></td>
<td id="A1.T1.4.1.4.3.11" class="ltx_td ltx_align_center ltx_border_bb"><span id="A1.T1.4.1.4.3.11.1" class="ltx_text" style="background-color:#E6E6E6;">71.0</span></td>
<td id="A1.T1.4.1.4.3.12" class="ltx_td ltx_align_center ltx_border_bb"><span id="A1.T1.4.1.4.3.12.1" class="ltx_text" style="background-color:#E6E6E6;">86.93</span></td>
<td id="A1.T1.4.1.4.3.13" class="ltx_td ltx_align_center ltx_border_bb"><span id="A1.T1.4.1.4.3.13.1" class="ltx_text" style="background-color:#E6E6E6;">43.6</span></td>
<td id="A1.T1.4.1.4.3.14" class="ltx_td ltx_align_center ltx_border_bb"><span id="A1.T1.4.1.4.3.14.1" class="ltx_text" style="background-color:#E6E6E6;">3.04</span></td>
<td id="A1.T1.4.1.4.3.15" class="ltx_td ltx_align_center ltx_border_bb"><span id="A1.T1.4.1.4.3.15.1" class="ltx_text" style="background-color:#E6E6E6;">79.56</span></td>
<td id="A1.T1.4.1.4.3.16" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb"><span id="A1.T1.4.1.4.3.16.1" class="ltx_text" style="background-color:#E6E6E6;">54.79</span></td>
</tr>
</tbody>
</table>
</span></div>
</figure>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Additional Related Works</h2>

<div id="A2.p1" class="ltx_para">
<p id="A2.p1.1" class="ltx_p"><span id="A2.p1.1.1" class="ltx_text ltx_font_bold">Preliminaries of KV cache in LLMs.</span>
LLM inference has been significantly improved in recent works <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib9" title="" class="ltx_ref">9</a>, <a href="#bib.bib74" title="" class="ltx_ref">74</a>, <a href="#bib.bib13" title="" class="ltx_ref">13</a>]</cite>
by optimizing the KV cache management. Overall, this line of research is typically done in two steps: 1) First, at the <span id="A2.p1.1.2" class="ltx_text ltx_font_italic">prefilling</span> stage, LLM calculates the initial KV caches at each attention layer based on the input sequence and decodes the first token. Second, 2) at the <span id="A2.p1.1.3" class="ltx_text ltx_font_italic">decoding</span> Stage, LLM autoregressively predicts the next token, where its KV cache at each attention layer is added to the overall caches. While existing works have compressed KV cache in different aspects (e.g., quantization <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib11" title="" class="ltx_ref">11</a>, <a href="#bib.bib12" title="" class="ltx_ref">12</a>]</cite>, token pruning
<cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib14" title="" class="ltx_ref">14</a>, <a href="#bib.bib16" title="" class="ltx_ref">16</a>]</cite>
).</p>
</div>
<div id="A2.p2" class="ltx_para">
<p id="A2.p2.1" class="ltx_p"><span id="A2.p2.1.1" class="ltx_text ltx_font_bold">KV cache compression.</span> In the prior study, various strategies for enhancing efficient transformer architectures are discussed, covering a spectrum of techniques aimed at optimizing performance and managing resource constraints. These methods include attention optimization <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib46" title="" class="ltx_ref">46</a>, <a href="#bib.bib47" title="" class="ltx_ref">47</a>, <a href="#bib.bib76" title="" class="ltx_ref">76</a>]</cite>, grouping queries <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib43" title="" class="ltx_ref">43</a>, <a href="#bib.bib42" title="" class="ltx_ref">42</a>]</cite>, sparse KV caching <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib16" title="" class="ltx_ref">16</a>, <a href="#bib.bib77" title="" class="ltx_ref">77</a>, <a href="#bib.bib78" title="" class="ltx_ref">78</a>]</cite>, shrinking tokens <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib79" title="" class="ltx_ref">79</a>, <a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite>, and improving long-context generation. Significant contributions come from projects such as H2O <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite>, GEAR <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite>, and KIVI <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib11" title="" class="ltx_ref">11</a>]</cite>. Additionally, efforts to alleviate KV cache bottlenecks include strategies like multi-query attention <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib42" title="" class="ltx_ref">42</a>]</cite> and multi-group attention <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib43" title="" class="ltx_ref">43</a>]</cite>, which propose reducing the number of heads in the KV cache.
However, these methods often necessitate retraining or fine-tuning models from scratch.
Other approaches focus on diminishing the size of the KV cache by selectively evicting less important tokens <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib14" title="" class="ltx_ref">14</a>]</cite> and enhancing the system architecture by offloading the KV cache <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib80" title="" class="ltx_ref">80</a>]</cite> or integrating techniques such as virtual memory and paging <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib81" title="" class="ltx_ref">81</a>]</cite> into the attention mechanism.</p>
</div>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Discussions and Limitations</h2>

<div id="A3.p1" class="ltx_para ltx_noindent">
<p id="A3.p1.1" class="ltx_p"><span id="A3.p1.1.1" class="ltx_text ltx_font_bold">Alternative merging function.</span>
During our preliminary exploration, we first consider another simpler merge function maximum norm-preserving interpolation for cross-layer compression.
This function is designed to maintain the maximum norm of the vectors involved, ensuring that the most significant features are preserved during the merging process. The maximum norm-preserving interpolation in terms of <math id="A3.p1.1.m1.1" class="ltx_Math" alttext="F_{\mathrm{max}}" display="inline"><semantics id="A3.p1.1.m1.1a"><msub id="A3.p1.1.m1.1.1" xref="A3.p1.1.m1.1.1.cmml"><mi id="A3.p1.1.m1.1.1.2" xref="A3.p1.1.m1.1.1.2.cmml">F</mi><mi id="A3.p1.1.m1.1.1.3" xref="A3.p1.1.m1.1.1.3.cmml">max</mi></msub><annotation-xml encoding="MathML-Content" id="A3.p1.1.m1.1b"><apply id="A3.p1.1.m1.1.1.cmml" xref="A3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A3.p1.1.m1.1.1.1.cmml" xref="A3.p1.1.m1.1.1">subscript</csymbol><ci id="A3.p1.1.m1.1.1.2.cmml" xref="A3.p1.1.m1.1.1.2">𝐹</ci><ci id="A3.p1.1.m1.1.1.3.cmml" xref="A3.p1.1.m1.1.1.3">max</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p1.1.m1.1c">F_{\mathrm{max}}</annotation></semantics></math> can be defined as follows:</p>
</div>
<div id="A3.p2" class="ltx_para">
<table id="A3.E1" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="A3.E1X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A3.E1X.2.1.1.m1.9" class="ltx_Math" alttext="\displaystyle F_{\mathrm{max}}({\bm{x}}^{l},{{\bm{x}}}^{l-1})=\frac{\bar{{\bm{x}}}^{l,l-1}}{\lVert\bar{{\bm{x}}}^{l,l-1}\rVert}\cdot\mathrm{Max}(\lVert{\bm{x}}^{l}\rVert,\lVert{\bm{x}}^{l-1}\rVert)" display="inline"><semantics id="A3.E1X.2.1.1.m1.9a"><mrow id="A3.E1X.2.1.1.m1.9.9" xref="A3.E1X.2.1.1.m1.9.9.cmml"><mrow id="A3.E1X.2.1.1.m1.7.7.2" xref="A3.E1X.2.1.1.m1.7.7.2.cmml"><msub id="A3.E1X.2.1.1.m1.7.7.2.4" xref="A3.E1X.2.1.1.m1.7.7.2.4.cmml"><mi id="A3.E1X.2.1.1.m1.7.7.2.4.2" xref="A3.E1X.2.1.1.m1.7.7.2.4.2.cmml">F</mi><mi id="A3.E1X.2.1.1.m1.7.7.2.4.3" xref="A3.E1X.2.1.1.m1.7.7.2.4.3.cmml">max</mi></msub><mo lspace="0em" rspace="0em" id="A3.E1X.2.1.1.m1.7.7.2.3" xref="A3.E1X.2.1.1.m1.7.7.2.3.cmml">​</mo><mrow id="A3.E1X.2.1.1.m1.7.7.2.2.2" xref="A3.E1X.2.1.1.m1.7.7.2.2.3.cmml"><mo stretchy="false" id="A3.E1X.2.1.1.m1.7.7.2.2.2.3" xref="A3.E1X.2.1.1.m1.7.7.2.2.3.cmml">(</mo><msup id="A3.E1X.2.1.1.m1.6.6.1.1.1.1" xref="A3.E1X.2.1.1.m1.6.6.1.1.1.1.cmml"><mi id="A3.E1X.2.1.1.m1.6.6.1.1.1.1.2" xref="A3.E1X.2.1.1.m1.6.6.1.1.1.1.2.cmml">𝒙</mi><mi id="A3.E1X.2.1.1.m1.6.6.1.1.1.1.3" xref="A3.E1X.2.1.1.m1.6.6.1.1.1.1.3.cmml">l</mi></msup><mo id="A3.E1X.2.1.1.m1.7.7.2.2.2.4" xref="A3.E1X.2.1.1.m1.7.7.2.2.3.cmml">,</mo><msup id="A3.E1X.2.1.1.m1.7.7.2.2.2.2" xref="A3.E1X.2.1.1.m1.7.7.2.2.2.2.cmml"><mi id="A3.E1X.2.1.1.m1.7.7.2.2.2.2.2" xref="A3.E1X.2.1.1.m1.7.7.2.2.2.2.2.cmml">𝒙</mi><mrow id="A3.E1X.2.1.1.m1.7.7.2.2.2.2.3" xref="A3.E1X.2.1.1.m1.7.7.2.2.2.2.3.cmml"><mi id="A3.E1X.2.1.1.m1.7.7.2.2.2.2.3.2" xref="A3.E1X.2.1.1.m1.7.7.2.2.2.2.3.2.cmml">l</mi><mo id="A3.E1X.2.1.1.m1.7.7.2.2.2.2.3.1" xref="A3.E1X.2.1.1.m1.7.7.2.2.2.2.3.1.cmml">−</mo><mn id="A3.E1X.2.1.1.m1.7.7.2.2.2.2.3.3" xref="A3.E1X.2.1.1.m1.7.7.2.2.2.2.3.3.cmml">1</mn></mrow></msup><mo stretchy="false" id="A3.E1X.2.1.1.m1.7.7.2.2.2.5" xref="A3.E1X.2.1.1.m1.7.7.2.2.3.cmml">)</mo></mrow></mrow><mo id="A3.E1X.2.1.1.m1.9.9.5" xref="A3.E1X.2.1.1.m1.9.9.5.cmml">=</mo><mrow id="A3.E1X.2.1.1.m1.9.9.4" xref="A3.E1X.2.1.1.m1.9.9.4.cmml"><mrow id="A3.E1X.2.1.1.m1.9.9.4.4" xref="A3.E1X.2.1.1.m1.9.9.4.4.cmml"><mstyle displaystyle="true" id="A3.E1X.2.1.1.m1.5.5" xref="A3.E1X.2.1.1.m1.5.5.cmml"><mfrac id="A3.E1X.2.1.1.m1.5.5a" xref="A3.E1X.2.1.1.m1.5.5.cmml"><msup id="A3.E1X.2.1.1.m1.2.2.2" xref="A3.E1X.2.1.1.m1.2.2.2.cmml"><mover accent="true" id="A3.E1X.2.1.1.m1.2.2.2.4" xref="A3.E1X.2.1.1.m1.2.2.2.4.cmml"><mi id="A3.E1X.2.1.1.m1.2.2.2.4.2" xref="A3.E1X.2.1.1.m1.2.2.2.4.2.cmml">𝒙</mi><mo id="A3.E1X.2.1.1.m1.2.2.2.4.1" xref="A3.E1X.2.1.1.m1.2.2.2.4.1.cmml">¯</mo></mover><mrow id="A3.E1X.2.1.1.m1.2.2.2.2.2.2" xref="A3.E1X.2.1.1.m1.2.2.2.2.2.3.cmml"><mi id="A3.E1X.2.1.1.m1.1.1.1.1.1.1" xref="A3.E1X.2.1.1.m1.1.1.1.1.1.1.cmml">l</mi><mo id="A3.E1X.2.1.1.m1.2.2.2.2.2.2.2" xref="A3.E1X.2.1.1.m1.2.2.2.2.2.3.cmml">,</mo><mrow id="A3.E1X.2.1.1.m1.2.2.2.2.2.2.1" xref="A3.E1X.2.1.1.m1.2.2.2.2.2.2.1.cmml"><mi id="A3.E1X.2.1.1.m1.2.2.2.2.2.2.1.2" xref="A3.E1X.2.1.1.m1.2.2.2.2.2.2.1.2.cmml">l</mi><mo id="A3.E1X.2.1.1.m1.2.2.2.2.2.2.1.1" xref="A3.E1X.2.1.1.m1.2.2.2.2.2.2.1.1.cmml">−</mo><mn id="A3.E1X.2.1.1.m1.2.2.2.2.2.2.1.3" xref="A3.E1X.2.1.1.m1.2.2.2.2.2.2.1.3.cmml">1</mn></mrow></mrow></msup><mrow id="A3.E1X.2.1.1.m1.5.5.5.3" xref="A3.E1X.2.1.1.m1.5.5.5.4.cmml"><mo fence="true" rspace="0em" id="A3.E1X.2.1.1.m1.5.5.5.3.2" xref="A3.E1X.2.1.1.m1.5.5.5.4.1.cmml">∥</mo><msup id="A3.E1X.2.1.1.m1.5.5.5.3.1" xref="A3.E1X.2.1.1.m1.5.5.5.3.1.cmml"><mover accent="true" id="A3.E1X.2.1.1.m1.5.5.5.3.1.2" xref="A3.E1X.2.1.1.m1.5.5.5.3.1.2.cmml"><mi id="A3.E1X.2.1.1.m1.5.5.5.3.1.2.2" xref="A3.E1X.2.1.1.m1.5.5.5.3.1.2.2.cmml">𝒙</mi><mo id="A3.E1X.2.1.1.m1.5.5.5.3.1.2.1" xref="A3.E1X.2.1.1.m1.5.5.5.3.1.2.1.cmml">¯</mo></mover><mrow id="A3.E1X.2.1.1.m1.4.4.4.2.2.2" xref="A3.E1X.2.1.1.m1.4.4.4.2.2.3.cmml"><mi id="A3.E1X.2.1.1.m1.3.3.3.1.1.1" xref="A3.E1X.2.1.1.m1.3.3.3.1.1.1.cmml">l</mi><mo id="A3.E1X.2.1.1.m1.4.4.4.2.2.2.2" xref="A3.E1X.2.1.1.m1.4.4.4.2.2.3.cmml">,</mo><mrow id="A3.E1X.2.1.1.m1.4.4.4.2.2.2.1" xref="A3.E1X.2.1.1.m1.4.4.4.2.2.2.1.cmml"><mi id="A3.E1X.2.1.1.m1.4.4.4.2.2.2.1.2" xref="A3.E1X.2.1.1.m1.4.4.4.2.2.2.1.2.cmml">l</mi><mo id="A3.E1X.2.1.1.m1.4.4.4.2.2.2.1.1" xref="A3.E1X.2.1.1.m1.4.4.4.2.2.2.1.1.cmml">−</mo><mn id="A3.E1X.2.1.1.m1.4.4.4.2.2.2.1.3" xref="A3.E1X.2.1.1.m1.4.4.4.2.2.2.1.3.cmml">1</mn></mrow></mrow></msup><mo fence="true" lspace="0em" id="A3.E1X.2.1.1.m1.5.5.5.3.3" xref="A3.E1X.2.1.1.m1.5.5.5.4.1.cmml">∥</mo></mrow></mfrac></mstyle><mo lspace="0.222em" rspace="0.222em" id="A3.E1X.2.1.1.m1.9.9.4.4.1" xref="A3.E1X.2.1.1.m1.9.9.4.4.1.cmml">⋅</mo><mi id="A3.E1X.2.1.1.m1.9.9.4.4.2" xref="A3.E1X.2.1.1.m1.9.9.4.4.2.cmml">Max</mi></mrow><mo lspace="0em" rspace="0em" id="A3.E1X.2.1.1.m1.9.9.4.3" xref="A3.E1X.2.1.1.m1.9.9.4.3.cmml">​</mo><mrow id="A3.E1X.2.1.1.m1.9.9.4.2.2" xref="A3.E1X.2.1.1.m1.9.9.4.2.3.cmml"><mo stretchy="false" id="A3.E1X.2.1.1.m1.9.9.4.2.2.3" xref="A3.E1X.2.1.1.m1.9.9.4.2.3.cmml">(</mo><mrow id="A3.E1X.2.1.1.m1.8.8.3.1.1.1.1" xref="A3.E1X.2.1.1.m1.8.8.3.1.1.1.2.cmml"><mo fence="true" lspace="0em" rspace="0em" id="A3.E1X.2.1.1.m1.8.8.3.1.1.1.1.2" xref="A3.E1X.2.1.1.m1.8.8.3.1.1.1.2.1.cmml">∥</mo><msup id="A3.E1X.2.1.1.m1.8.8.3.1.1.1.1.1" xref="A3.E1X.2.1.1.m1.8.8.3.1.1.1.1.1.cmml"><mi id="A3.E1X.2.1.1.m1.8.8.3.1.1.1.1.1.2" xref="A3.E1X.2.1.1.m1.8.8.3.1.1.1.1.1.2.cmml">𝒙</mi><mi id="A3.E1X.2.1.1.m1.8.8.3.1.1.1.1.1.3" xref="A3.E1X.2.1.1.m1.8.8.3.1.1.1.1.1.3.cmml">l</mi></msup><mo fence="true" lspace="0em" rspace="0em" id="A3.E1X.2.1.1.m1.8.8.3.1.1.1.1.3" xref="A3.E1X.2.1.1.m1.8.8.3.1.1.1.2.1.cmml">∥</mo></mrow><mo id="A3.E1X.2.1.1.m1.9.9.4.2.2.4" xref="A3.E1X.2.1.1.m1.9.9.4.2.3.cmml">,</mo><mrow id="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1" xref="A3.E1X.2.1.1.m1.9.9.4.2.2.2.2.cmml"><mo fence="true" lspace="0em" rspace="0em" id="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.2" xref="A3.E1X.2.1.1.m1.9.9.4.2.2.2.2.1.cmml">∥</mo><msup id="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.1" xref="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.1.cmml"><mi id="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.1.2" xref="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.1.2.cmml">𝒙</mi><mrow id="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.1.3" xref="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.1.3.cmml"><mi id="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.1.3.2" xref="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.1.3.2.cmml">l</mi><mo id="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.1.3.1" xref="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.1.3.1.cmml">−</mo><mn id="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.1.3.3" xref="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.1.3.3.cmml">1</mn></mrow></msup><mo fence="true" lspace="0em" rspace="0em" id="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.3" xref="A3.E1X.2.1.1.m1.9.9.4.2.2.2.2.1.cmml">∥</mo></mrow><mo stretchy="false" id="A3.E1X.2.1.1.m1.9.9.4.2.2.5" xref="A3.E1X.2.1.1.m1.9.9.4.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.E1X.2.1.1.m1.9b"><apply id="A3.E1X.2.1.1.m1.9.9.cmml" xref="A3.E1X.2.1.1.m1.9.9"><eq id="A3.E1X.2.1.1.m1.9.9.5.cmml" xref="A3.E1X.2.1.1.m1.9.9.5"></eq><apply id="A3.E1X.2.1.1.m1.7.7.2.cmml" xref="A3.E1X.2.1.1.m1.7.7.2"><times id="A3.E1X.2.1.1.m1.7.7.2.3.cmml" xref="A3.E1X.2.1.1.m1.7.7.2.3"></times><apply id="A3.E1X.2.1.1.m1.7.7.2.4.cmml" xref="A3.E1X.2.1.1.m1.7.7.2.4"><csymbol cd="ambiguous" id="A3.E1X.2.1.1.m1.7.7.2.4.1.cmml" xref="A3.E1X.2.1.1.m1.7.7.2.4">subscript</csymbol><ci id="A3.E1X.2.1.1.m1.7.7.2.4.2.cmml" xref="A3.E1X.2.1.1.m1.7.7.2.4.2">𝐹</ci><ci id="A3.E1X.2.1.1.m1.7.7.2.4.3.cmml" xref="A3.E1X.2.1.1.m1.7.7.2.4.3">max</ci></apply><interval closure="open" id="A3.E1X.2.1.1.m1.7.7.2.2.3.cmml" xref="A3.E1X.2.1.1.m1.7.7.2.2.2"><apply id="A3.E1X.2.1.1.m1.6.6.1.1.1.1.cmml" xref="A3.E1X.2.1.1.m1.6.6.1.1.1.1"><csymbol cd="ambiguous" id="A3.E1X.2.1.1.m1.6.6.1.1.1.1.1.cmml" xref="A3.E1X.2.1.1.m1.6.6.1.1.1.1">superscript</csymbol><ci id="A3.E1X.2.1.1.m1.6.6.1.1.1.1.2.cmml" xref="A3.E1X.2.1.1.m1.6.6.1.1.1.1.2">𝒙</ci><ci id="A3.E1X.2.1.1.m1.6.6.1.1.1.1.3.cmml" xref="A3.E1X.2.1.1.m1.6.6.1.1.1.1.3">𝑙</ci></apply><apply id="A3.E1X.2.1.1.m1.7.7.2.2.2.2.cmml" xref="A3.E1X.2.1.1.m1.7.7.2.2.2.2"><csymbol cd="ambiguous" id="A3.E1X.2.1.1.m1.7.7.2.2.2.2.1.cmml" xref="A3.E1X.2.1.1.m1.7.7.2.2.2.2">superscript</csymbol><ci id="A3.E1X.2.1.1.m1.7.7.2.2.2.2.2.cmml" xref="A3.E1X.2.1.1.m1.7.7.2.2.2.2.2">𝒙</ci><apply id="A3.E1X.2.1.1.m1.7.7.2.2.2.2.3.cmml" xref="A3.E1X.2.1.1.m1.7.7.2.2.2.2.3"><minus id="A3.E1X.2.1.1.m1.7.7.2.2.2.2.3.1.cmml" xref="A3.E1X.2.1.1.m1.7.7.2.2.2.2.3.1"></minus><ci id="A3.E1X.2.1.1.m1.7.7.2.2.2.2.3.2.cmml" xref="A3.E1X.2.1.1.m1.7.7.2.2.2.2.3.2">𝑙</ci><cn type="integer" id="A3.E1X.2.1.1.m1.7.7.2.2.2.2.3.3.cmml" xref="A3.E1X.2.1.1.m1.7.7.2.2.2.2.3.3">1</cn></apply></apply></interval></apply><apply id="A3.E1X.2.1.1.m1.9.9.4.cmml" xref="A3.E1X.2.1.1.m1.9.9.4"><times id="A3.E1X.2.1.1.m1.9.9.4.3.cmml" xref="A3.E1X.2.1.1.m1.9.9.4.3"></times><apply id="A3.E1X.2.1.1.m1.9.9.4.4.cmml" xref="A3.E1X.2.1.1.m1.9.9.4.4"><ci id="A3.E1X.2.1.1.m1.9.9.4.4.1.cmml" xref="A3.E1X.2.1.1.m1.9.9.4.4.1">⋅</ci><apply id="A3.E1X.2.1.1.m1.5.5.cmml" xref="A3.E1X.2.1.1.m1.5.5"><divide id="A3.E1X.2.1.1.m1.5.5.6.cmml" xref="A3.E1X.2.1.1.m1.5.5"></divide><apply id="A3.E1X.2.1.1.m1.2.2.2.cmml" xref="A3.E1X.2.1.1.m1.2.2.2"><csymbol cd="ambiguous" id="A3.E1X.2.1.1.m1.2.2.2.3.cmml" xref="A3.E1X.2.1.1.m1.2.2.2">superscript</csymbol><apply id="A3.E1X.2.1.1.m1.2.2.2.4.cmml" xref="A3.E1X.2.1.1.m1.2.2.2.4"><ci id="A3.E1X.2.1.1.m1.2.2.2.4.1.cmml" xref="A3.E1X.2.1.1.m1.2.2.2.4.1">¯</ci><ci id="A3.E1X.2.1.1.m1.2.2.2.4.2.cmml" xref="A3.E1X.2.1.1.m1.2.2.2.4.2">𝒙</ci></apply><list id="A3.E1X.2.1.1.m1.2.2.2.2.2.3.cmml" xref="A3.E1X.2.1.1.m1.2.2.2.2.2.2"><ci id="A3.E1X.2.1.1.m1.1.1.1.1.1.1.cmml" xref="A3.E1X.2.1.1.m1.1.1.1.1.1.1">𝑙</ci><apply id="A3.E1X.2.1.1.m1.2.2.2.2.2.2.1.cmml" xref="A3.E1X.2.1.1.m1.2.2.2.2.2.2.1"><minus id="A3.E1X.2.1.1.m1.2.2.2.2.2.2.1.1.cmml" xref="A3.E1X.2.1.1.m1.2.2.2.2.2.2.1.1"></minus><ci id="A3.E1X.2.1.1.m1.2.2.2.2.2.2.1.2.cmml" xref="A3.E1X.2.1.1.m1.2.2.2.2.2.2.1.2">𝑙</ci><cn type="integer" id="A3.E1X.2.1.1.m1.2.2.2.2.2.2.1.3.cmml" xref="A3.E1X.2.1.1.m1.2.2.2.2.2.2.1.3">1</cn></apply></list></apply><apply id="A3.E1X.2.1.1.m1.5.5.5.4.cmml" xref="A3.E1X.2.1.1.m1.5.5.5.3"><csymbol cd="latexml" id="A3.E1X.2.1.1.m1.5.5.5.4.1.cmml" xref="A3.E1X.2.1.1.m1.5.5.5.3.2">delimited-∥∥</csymbol><apply id="A3.E1X.2.1.1.m1.5.5.5.3.1.cmml" xref="A3.E1X.2.1.1.m1.5.5.5.3.1"><csymbol cd="ambiguous" id="A3.E1X.2.1.1.m1.5.5.5.3.1.1.cmml" xref="A3.E1X.2.1.1.m1.5.5.5.3.1">superscript</csymbol><apply id="A3.E1X.2.1.1.m1.5.5.5.3.1.2.cmml" xref="A3.E1X.2.1.1.m1.5.5.5.3.1.2"><ci id="A3.E1X.2.1.1.m1.5.5.5.3.1.2.1.cmml" xref="A3.E1X.2.1.1.m1.5.5.5.3.1.2.1">¯</ci><ci id="A3.E1X.2.1.1.m1.5.5.5.3.1.2.2.cmml" xref="A3.E1X.2.1.1.m1.5.5.5.3.1.2.2">𝒙</ci></apply><list id="A3.E1X.2.1.1.m1.4.4.4.2.2.3.cmml" xref="A3.E1X.2.1.1.m1.4.4.4.2.2.2"><ci id="A3.E1X.2.1.1.m1.3.3.3.1.1.1.cmml" xref="A3.E1X.2.1.1.m1.3.3.3.1.1.1">𝑙</ci><apply id="A3.E1X.2.1.1.m1.4.4.4.2.2.2.1.cmml" xref="A3.E1X.2.1.1.m1.4.4.4.2.2.2.1"><minus id="A3.E1X.2.1.1.m1.4.4.4.2.2.2.1.1.cmml" xref="A3.E1X.2.1.1.m1.4.4.4.2.2.2.1.1"></minus><ci id="A3.E1X.2.1.1.m1.4.4.4.2.2.2.1.2.cmml" xref="A3.E1X.2.1.1.m1.4.4.4.2.2.2.1.2">𝑙</ci><cn type="integer" id="A3.E1X.2.1.1.m1.4.4.4.2.2.2.1.3.cmml" xref="A3.E1X.2.1.1.m1.4.4.4.2.2.2.1.3">1</cn></apply></list></apply></apply></apply><ci id="A3.E1X.2.1.1.m1.9.9.4.4.2.cmml" xref="A3.E1X.2.1.1.m1.9.9.4.4.2">Max</ci></apply><interval closure="open" id="A3.E1X.2.1.1.m1.9.9.4.2.3.cmml" xref="A3.E1X.2.1.1.m1.9.9.4.2.2"><apply id="A3.E1X.2.1.1.m1.8.8.3.1.1.1.2.cmml" xref="A3.E1X.2.1.1.m1.8.8.3.1.1.1.1"><csymbol cd="latexml" id="A3.E1X.2.1.1.m1.8.8.3.1.1.1.2.1.cmml" xref="A3.E1X.2.1.1.m1.8.8.3.1.1.1.1.2">delimited-∥∥</csymbol><apply id="A3.E1X.2.1.1.m1.8.8.3.1.1.1.1.1.cmml" xref="A3.E1X.2.1.1.m1.8.8.3.1.1.1.1.1"><csymbol cd="ambiguous" id="A3.E1X.2.1.1.m1.8.8.3.1.1.1.1.1.1.cmml" xref="A3.E1X.2.1.1.m1.8.8.3.1.1.1.1.1">superscript</csymbol><ci id="A3.E1X.2.1.1.m1.8.8.3.1.1.1.1.1.2.cmml" xref="A3.E1X.2.1.1.m1.8.8.3.1.1.1.1.1.2">𝒙</ci><ci id="A3.E1X.2.1.1.m1.8.8.3.1.1.1.1.1.3.cmml" xref="A3.E1X.2.1.1.m1.8.8.3.1.1.1.1.1.3">𝑙</ci></apply></apply><apply id="A3.E1X.2.1.1.m1.9.9.4.2.2.2.2.cmml" xref="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1"><csymbol cd="latexml" id="A3.E1X.2.1.1.m1.9.9.4.2.2.2.2.1.cmml" xref="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.2">delimited-∥∥</csymbol><apply id="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.1.cmml" xref="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.1"><csymbol cd="ambiguous" id="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.1.1.cmml" xref="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.1">superscript</csymbol><ci id="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.1.2.cmml" xref="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.1.2">𝒙</ci><apply id="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.1.3.cmml" xref="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.1.3"><minus id="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.1.3.1.cmml" xref="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.1.3.1"></minus><ci id="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.1.3.2.cmml" xref="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.1.3.2">𝑙</ci><cn type="integer" id="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.1.3.3.cmml" xref="A3.E1X.2.1.1.m1.9.9.4.2.2.2.1.1.3.3">1</cn></apply></apply></apply></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.E1X.2.1.1.m1.9c">\displaystyle F_{\mathrm{max}}({\bm{x}}^{l},{{\bm{x}}}^{l-1})=\frac{\bar{{\bm{x}}}^{l,l-1}}{\lVert\bar{{\bm{x}}}^{l,l-1}\rVert}\cdot\mathrm{Max}(\lVert{\bm{x}}^{l}\rVert,\lVert{\bm{x}}^{l-1}\rVert)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(A)</span></td>
</tr>
</tbody>
</table>
</div>
<div id="A3.p3" class="ltx_para">
<p id="A3.p3.7" class="ltx_p">Here
<math id="A3.p3.1.m1.2" class="ltx_Math" alttext="\bar{{\bm{x}}}^{l,l-1}" display="inline"><semantics id="A3.p3.1.m1.2a"><msup id="A3.p3.1.m1.2.3" xref="A3.p3.1.m1.2.3.cmml"><mover accent="true" id="A3.p3.1.m1.2.3.2" xref="A3.p3.1.m1.2.3.2.cmml"><mi id="A3.p3.1.m1.2.3.2.2" xref="A3.p3.1.m1.2.3.2.2.cmml">𝒙</mi><mo id="A3.p3.1.m1.2.3.2.1" xref="A3.p3.1.m1.2.3.2.1.cmml">¯</mo></mover><mrow id="A3.p3.1.m1.2.2.2.2" xref="A3.p3.1.m1.2.2.2.3.cmml"><mi id="A3.p3.1.m1.1.1.1.1" xref="A3.p3.1.m1.1.1.1.1.cmml">l</mi><mo id="A3.p3.1.m1.2.2.2.2.2" xref="A3.p3.1.m1.2.2.2.3.cmml">,</mo><mrow id="A3.p3.1.m1.2.2.2.2.1" xref="A3.p3.1.m1.2.2.2.2.1.cmml"><mi id="A3.p3.1.m1.2.2.2.2.1.2" xref="A3.p3.1.m1.2.2.2.2.1.2.cmml">l</mi><mo id="A3.p3.1.m1.2.2.2.2.1.1" xref="A3.p3.1.m1.2.2.2.2.1.1.cmml">−</mo><mn id="A3.p3.1.m1.2.2.2.2.1.3" xref="A3.p3.1.m1.2.2.2.2.1.3.cmml">1</mn></mrow></mrow></msup><annotation-xml encoding="MathML-Content" id="A3.p3.1.m1.2b"><apply id="A3.p3.1.m1.2.3.cmml" xref="A3.p3.1.m1.2.3"><csymbol cd="ambiguous" id="A3.p3.1.m1.2.3.1.cmml" xref="A3.p3.1.m1.2.3">superscript</csymbol><apply id="A3.p3.1.m1.2.3.2.cmml" xref="A3.p3.1.m1.2.3.2"><ci id="A3.p3.1.m1.2.3.2.1.cmml" xref="A3.p3.1.m1.2.3.2.1">¯</ci><ci id="A3.p3.1.m1.2.3.2.2.cmml" xref="A3.p3.1.m1.2.3.2.2">𝒙</ci></apply><list id="A3.p3.1.m1.2.2.2.3.cmml" xref="A3.p3.1.m1.2.2.2.2"><ci id="A3.p3.1.m1.1.1.1.1.cmml" xref="A3.p3.1.m1.1.1.1.1">𝑙</ci><apply id="A3.p3.1.m1.2.2.2.2.1.cmml" xref="A3.p3.1.m1.2.2.2.2.1"><minus id="A3.p3.1.m1.2.2.2.2.1.1.cmml" xref="A3.p3.1.m1.2.2.2.2.1.1"></minus><ci id="A3.p3.1.m1.2.2.2.2.1.2.cmml" xref="A3.p3.1.m1.2.2.2.2.1.2">𝑙</ci><cn type="integer" id="A3.p3.1.m1.2.2.2.2.1.3.cmml" xref="A3.p3.1.m1.2.2.2.2.1.3">1</cn></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p3.1.m1.2c">\bar{{\bm{x}}}^{l,l-1}</annotation></semantics></math> represents the average vector between <math id="A3.p3.2.m2.1" class="ltx_Math" alttext="{\bm{x}}^{l}" display="inline"><semantics id="A3.p3.2.m2.1a"><msup id="A3.p3.2.m2.1.1" xref="A3.p3.2.m2.1.1.cmml"><mi id="A3.p3.2.m2.1.1.2" xref="A3.p3.2.m2.1.1.2.cmml">𝒙</mi><mi id="A3.p3.2.m2.1.1.3" xref="A3.p3.2.m2.1.1.3.cmml">l</mi></msup><annotation-xml encoding="MathML-Content" id="A3.p3.2.m2.1b"><apply id="A3.p3.2.m2.1.1.cmml" xref="A3.p3.2.m2.1.1"><csymbol cd="ambiguous" id="A3.p3.2.m2.1.1.1.cmml" xref="A3.p3.2.m2.1.1">superscript</csymbol><ci id="A3.p3.2.m2.1.1.2.cmml" xref="A3.p3.2.m2.1.1.2">𝒙</ci><ci id="A3.p3.2.m2.1.1.3.cmml" xref="A3.p3.2.m2.1.1.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p3.2.m2.1c">{\bm{x}}^{l}</annotation></semantics></math> and <math id="A3.p3.3.m3.1" class="ltx_Math" alttext="{\bm{x}}^{l-1}" display="inline"><semantics id="A3.p3.3.m3.1a"><msup id="A3.p3.3.m3.1.1" xref="A3.p3.3.m3.1.1.cmml"><mi id="A3.p3.3.m3.1.1.2" xref="A3.p3.3.m3.1.1.2.cmml">𝒙</mi><mrow id="A3.p3.3.m3.1.1.3" xref="A3.p3.3.m3.1.1.3.cmml"><mi id="A3.p3.3.m3.1.1.3.2" xref="A3.p3.3.m3.1.1.3.2.cmml">l</mi><mo id="A3.p3.3.m3.1.1.3.1" xref="A3.p3.3.m3.1.1.3.1.cmml">−</mo><mn id="A3.p3.3.m3.1.1.3.3" xref="A3.p3.3.m3.1.1.3.3.cmml">1</mn></mrow></msup><annotation-xml encoding="MathML-Content" id="A3.p3.3.m3.1b"><apply id="A3.p3.3.m3.1.1.cmml" xref="A3.p3.3.m3.1.1"><csymbol cd="ambiguous" id="A3.p3.3.m3.1.1.1.cmml" xref="A3.p3.3.m3.1.1">superscript</csymbol><ci id="A3.p3.3.m3.1.1.2.cmml" xref="A3.p3.3.m3.1.1.2">𝒙</ci><apply id="A3.p3.3.m3.1.1.3.cmml" xref="A3.p3.3.m3.1.1.3"><minus id="A3.p3.3.m3.1.1.3.1.cmml" xref="A3.p3.3.m3.1.1.3.1"></minus><ci id="A3.p3.3.m3.1.1.3.2.cmml" xref="A3.p3.3.m3.1.1.3.2">𝑙</ci><cn type="integer" id="A3.p3.3.m3.1.1.3.3.cmml" xref="A3.p3.3.m3.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p3.3.m3.1c">{\bm{x}}^{l-1}</annotation></semantics></math>. The function <math id="A3.p3.4.m4.1" class="ltx_Math" alttext="F_{\mathrm{max}}" display="inline"><semantics id="A3.p3.4.m4.1a"><msub id="A3.p3.4.m4.1.1" xref="A3.p3.4.m4.1.1.cmml"><mi id="A3.p3.4.m4.1.1.2" xref="A3.p3.4.m4.1.1.2.cmml">F</mi><mi id="A3.p3.4.m4.1.1.3" xref="A3.p3.4.m4.1.1.3.cmml">max</mi></msub><annotation-xml encoding="MathML-Content" id="A3.p3.4.m4.1b"><apply id="A3.p3.4.m4.1.1.cmml" xref="A3.p3.4.m4.1.1"><csymbol cd="ambiguous" id="A3.p3.4.m4.1.1.1.cmml" xref="A3.p3.4.m4.1.1">subscript</csymbol><ci id="A3.p3.4.m4.1.1.2.cmml" xref="A3.p3.4.m4.1.1.2">𝐹</ci><ci id="A3.p3.4.m4.1.1.3.cmml" xref="A3.p3.4.m4.1.1.3">max</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p3.4.m4.1c">F_{\mathrm{max}}</annotation></semantics></math> ensures that the merged vector preserves the direction of the average vector while scaling it to the maximum norm of the original KV states. Compared to the SLERP-based merge function, <math id="A3.p3.5.m5.1" class="ltx_Math" alttext="F_{\mathrm{max}}" display="inline"><semantics id="A3.p3.5.m5.1a"><msub id="A3.p3.5.m5.1.1" xref="A3.p3.5.m5.1.1.cmml"><mi id="A3.p3.5.m5.1.1.2" xref="A3.p3.5.m5.1.1.2.cmml">F</mi><mi id="A3.p3.5.m5.1.1.3" xref="A3.p3.5.m5.1.1.3.cmml">max</mi></msub><annotation-xml encoding="MathML-Content" id="A3.p3.5.m5.1b"><apply id="A3.p3.5.m5.1.1.cmml" xref="A3.p3.5.m5.1.1"><csymbol cd="ambiguous" id="A3.p3.5.m5.1.1.1.cmml" xref="A3.p3.5.m5.1.1">subscript</csymbol><ci id="A3.p3.5.m5.1.1.2.cmml" xref="A3.p3.5.m5.1.1.2">𝐹</ci><ci id="A3.p3.5.m5.1.1.3.cmml" xref="A3.p3.5.m5.1.1.3">max</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p3.5.m5.1c">F_{\mathrm{max}}</annotation></semantics></math> has less computational overhead and lower memory consumption. However, it is less accurate than SLERP. The choice between using <math id="A3.p3.6.m6.1" class="ltx_Math" alttext="F_{\mathrm{SLERP}}" display="inline"><semantics id="A3.p3.6.m6.1a"><msub id="A3.p3.6.m6.1.1" xref="A3.p3.6.m6.1.1.cmml"><mi id="A3.p3.6.m6.1.1.2" xref="A3.p3.6.m6.1.1.2.cmml">F</mi><mi id="A3.p3.6.m6.1.1.3" xref="A3.p3.6.m6.1.1.3.cmml">SLERP</mi></msub><annotation-xml encoding="MathML-Content" id="A3.p3.6.m6.1b"><apply id="A3.p3.6.m6.1.1.cmml" xref="A3.p3.6.m6.1.1"><csymbol cd="ambiguous" id="A3.p3.6.m6.1.1.1.cmml" xref="A3.p3.6.m6.1.1">subscript</csymbol><ci id="A3.p3.6.m6.1.1.2.cmml" xref="A3.p3.6.m6.1.1.2">𝐹</ci><ci id="A3.p3.6.m6.1.1.3.cmml" xref="A3.p3.6.m6.1.1.3">SLERP</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p3.6.m6.1c">F_{\mathrm{SLERP}}</annotation></semantics></math> or <math id="A3.p3.7.m7.1" class="ltx_Math" alttext="F_{\mathrm{max}}" display="inline"><semantics id="A3.p3.7.m7.1a"><msub id="A3.p3.7.m7.1.1" xref="A3.p3.7.m7.1.1.cmml"><mi id="A3.p3.7.m7.1.1.2" xref="A3.p3.7.m7.1.1.2.cmml">F</mi><mi id="A3.p3.7.m7.1.1.3" xref="A3.p3.7.m7.1.1.3.cmml">max</mi></msub><annotation-xml encoding="MathML-Content" id="A3.p3.7.m7.1b"><apply id="A3.p3.7.m7.1.1.cmml" xref="A3.p3.7.m7.1.1"><csymbol cd="ambiguous" id="A3.p3.7.m7.1.1.1.cmml" xref="A3.p3.7.m7.1.1">subscript</csymbol><ci id="A3.p3.7.m7.1.1.2.cmml" xref="A3.p3.7.m7.1.1.2">𝐹</ci><ci id="A3.p3.7.m7.1.1.3.cmml" xref="A3.p3.7.m7.1.1.3">max</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p3.7.m7.1c">F_{\mathrm{max}}</annotation></semantics></math> depends on the specific requirements of the application. In our study, we primarily use SLERP to maximize performance.</p>
</div>
<div id="A3.p4" class="ltx_para">
<p id="A3.p4.1" class="ltx_p"><span id="A3.p4.1.1" class="ltx_text ltx_font_bold">Societal impact.</span> Our work represents a preliminary exploration of KV cache Compression in the depth dimension, an area that has been relatively unexplored and yet presents a significant bottleneck in large language models (LLMs). The proposed MiniCache provides a solution to improve the efficiency of LLM generation and is adaptable to existing intra-layer-based KV cache pruning and quantization technologies. Additionally, we proposed a simple yet effective approach that merges similar KV cache states in a cross-layer manner and effectively restores performance through a novel restoration technique. Our observations indicate that cross-layer similarity and mergeability can be applied in broad-efficient applications for post-training optimization in low-resource scenarios, such as deployment on mobile devices. Moreover, with effective optimization of KV cache memory demand, MiniCache further enhances long-context generation, which is a crucial paradigm for real-world applications, such as understanding concepts in textbooks. We hope our work supports the two key challenges of batch inference and long-context generation in the LLM industry and further research.</p>
</div>
<div id="A3.p5" class="ltx_para">
<p id="A3.p5.1" class="ltx_p">Furthermore, in addition to the significant contributions of MiniCache for KV cache Compression, several challenges remain that are common to large language models (LLMs). Issues such as the truthfulness and security of LLMs are still unresolved. Ensuring the accuracy and reliability of generated content is critical, as LLMs can sometimes produce plausible but incorrect or misleading information. Additionally, safeguarding against security vulnerabilities, such as adversarial attacks or data leakage, is paramount to maintaining the integrity and confidentiality of user interactions. Addressing these challenges requires ongoing research and development to enhance the robustness and trustworthiness of LLMs, alongside the advancements in computational efficiency and performance provided by innovations like MiniCache.</p>
</div>
<div id="A3.p6" class="ltx_para">
<p id="A3.p6.1" class="ltx_p"><span id="A3.p6.1.1" class="ltx_text ltx_font_bold">Limitations.</span> The current merging algorithm based on Spherical Linear Interpolation (SLERP) has its limitations. SLERP is suitable only for merging two vectors and uses an interpolation approach that restricts our algorithm from merging multiple layers simultaneously and maximizing the compression ratio in further states. This limitation impacts the overall efficiency of KV cache Compression and underscores the need for advanced techniques capable of handling more complex merging scenarios. Future research should focus on developing more sophisticated algorithms that can overcome these constraints, thereby enhancing the compression capabilities and overall performance of LLMs.</p>
</div>
</section>
<section id="A4" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>Additional Implementation Details</h2>

<div id="A4.p1" class="ltx_para">
<p id="A4.p1.5" class="ltx_p"><span id="A4.p1.5.1" class="ltx_text ltx_font_bold">Overview of the inference algorithm.</span> The MiniCache inference implementation, as shown in Alg. <a href="#alg1" title="Algorithm 1 ‣ Appendix D Additional Implementation Details ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, involves several key steps. When the interface starts, In the prefilling phase, we define the merging starting layer <math id="A4.p1.1.m1.1" class="ltx_Math" alttext="S" display="inline"><semantics id="A4.p1.1.m1.1a"><mi id="A4.p1.1.m1.1.1" xref="A4.p1.1.m1.1.1.cmml">S</mi><annotation-xml encoding="MathML-Content" id="A4.p1.1.m1.1b"><ci id="A4.p1.1.m1.1.1.cmml" xref="A4.p1.1.m1.1.1">𝑆</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.1.m1.1c">S</annotation></semantics></math>.
Before reaching layer <math id="A4.p1.2.m2.1" class="ltx_Math" alttext="S" display="inline"><semantics id="A4.p1.2.m2.1a"><mi id="A4.p1.2.m2.1.1" xref="A4.p1.2.m2.1.1.cmml">S</mi><annotation-xml encoding="MathML-Content" id="A4.p1.2.m2.1b"><ci id="A4.p1.2.m2.1.1.cmml" xref="A4.p1.2.m2.1.1">𝑆</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.2.m2.1c">S</annotation></semantics></math>, the inference uses the original attention and cache logic. From layer <math id="A4.p1.3.m3.1" class="ltx_Math" alttext="S" display="inline"><semantics id="A4.p1.3.m3.1a"><mi id="A4.p1.3.m3.1.1" xref="A4.p1.3.m3.1.1.cmml">S</mi><annotation-xml encoding="MathML-Content" id="A4.p1.3.m3.1b"><ci id="A4.p1.3.m3.1.1.cmml" xref="A4.p1.3.m3.1.1">𝑆</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.3.m3.1c">S</annotation></semantics></math> onward, we implement our merging algorithm, which operates in a cross-layer manner and is applied only at odd-numbered layers.
During merging, we fetch the KV cache from the previous layer and save the merged shared states into the current layer’s KV cache. To reduce memory usage, we then remove the cache of the previous layer. Additionally, we store the magnitudes vector and retention-sensitive tokens for each layer.
During the decoding phase, two scenarios arise after layer <math id="A4.p1.4.m4.1" class="ltx_Math" alttext="S" display="inline"><semantics id="A4.p1.4.m4.1a"><mi id="A4.p1.4.m4.1.1" xref="A4.p1.4.m4.1.1.cmml">S</mi><annotation-xml encoding="MathML-Content" id="A4.p1.4.m4.1b"><ci id="A4.p1.4.m4.1.1.cmml" xref="A4.p1.4.m4.1.1">𝑆</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.4.m4.1c">S</annotation></semantics></math>. For even-numbered layers (first round), since the KV cache has been removed during the prefilling phase, we refer to the next layer (<math id="A4.p1.5.m5.1" class="ltx_Math" alttext="l+1" display="inline"><semantics id="A4.p1.5.m5.1a"><mrow id="A4.p1.5.m5.1.1" xref="A4.p1.5.m5.1.1.cmml"><mi id="A4.p1.5.m5.1.1.2" xref="A4.p1.5.m5.1.1.2.cmml">l</mi><mo id="A4.p1.5.m5.1.1.1" xref="A4.p1.5.m5.1.1.1.cmml">+</mo><mn id="A4.p1.5.m5.1.1.3" xref="A4.p1.5.m5.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="A4.p1.5.m5.1b"><apply id="A4.p1.5.m5.1.1.cmml" xref="A4.p1.5.m5.1.1"><plus id="A4.p1.5.m5.1.1.1.cmml" xref="A4.p1.5.m5.1.1.1"></plus><ci id="A4.p1.5.m5.1.1.2.cmml" xref="A4.p1.5.m5.1.1.2">𝑙</ci><cn type="integer" id="A4.p1.5.m5.1.1.3.cmml" xref="A4.p1.5.m5.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.5.m5.1c">l+1</annotation></semantics></math>) to fetch the shared KV cache states. We then perform approximated scale restoration and retention token recovery. The new KV states from this phase are stored for use in the next round.
In the second round, which involves odd-numbered layers, we use the new KV tokens from both the previous and current layers. After the restoration phase, we perform the merge operations and update the shared KV cache states in the stack.</p>
</div>
<div id="A4.p2" class="ltx_para">
<p id="A4.p2.27" class="ltx_p"><span id="A4.p2.27.1" class="ltx_text ltx_font_bold">Cross-Layer merging and restoration algorithm.</span> As outlined in Alg. <a href="#alg2" title="Algorithm 2 ‣ Appendix D Additional Implementation Details ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, involves several crucial steps to ensure efficient memory usage while maintaining the integrity of the Key-Value (KV) Cache states. Initially, given the KV cache (<math id="A4.p2.1.m1.2" class="ltx_Math" alttext="{\bm{E}}_{k,v}^{l}" display="inline"><semantics id="A4.p2.1.m1.2a"><msubsup id="A4.p2.1.m1.2.3" xref="A4.p2.1.m1.2.3.cmml"><mi id="A4.p2.1.m1.2.3.2.2" xref="A4.p2.1.m1.2.3.2.2.cmml">𝑬</mi><mrow id="A4.p2.1.m1.2.2.2.4" xref="A4.p2.1.m1.2.2.2.3.cmml"><mi id="A4.p2.1.m1.1.1.1.1" xref="A4.p2.1.m1.1.1.1.1.cmml">k</mi><mo id="A4.p2.1.m1.2.2.2.4.1" xref="A4.p2.1.m1.2.2.2.3.cmml">,</mo><mi id="A4.p2.1.m1.2.2.2.2" xref="A4.p2.1.m1.2.2.2.2.cmml">v</mi></mrow><mi id="A4.p2.1.m1.2.3.3" xref="A4.p2.1.m1.2.3.3.cmml">l</mi></msubsup><annotation-xml encoding="MathML-Content" id="A4.p2.1.m1.2b"><apply id="A4.p2.1.m1.2.3.cmml" xref="A4.p2.1.m1.2.3"><csymbol cd="ambiguous" id="A4.p2.1.m1.2.3.1.cmml" xref="A4.p2.1.m1.2.3">superscript</csymbol><apply id="A4.p2.1.m1.2.3.2.cmml" xref="A4.p2.1.m1.2.3"><csymbol cd="ambiguous" id="A4.p2.1.m1.2.3.2.1.cmml" xref="A4.p2.1.m1.2.3">subscript</csymbol><ci id="A4.p2.1.m1.2.3.2.2.cmml" xref="A4.p2.1.m1.2.3.2.2">𝑬</ci><list id="A4.p2.1.m1.2.2.2.3.cmml" xref="A4.p2.1.m1.2.2.2.4"><ci id="A4.p2.1.m1.1.1.1.1.cmml" xref="A4.p2.1.m1.1.1.1.1">𝑘</ci><ci id="A4.p2.1.m1.2.2.2.2.cmml" xref="A4.p2.1.m1.2.2.2.2">𝑣</ci></list></apply><ci id="A4.p2.1.m1.2.3.3.cmml" xref="A4.p2.1.m1.2.3.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.1.m1.2c">{\bm{E}}_{k,v}^{l}</annotation></semantics></math>), norm values (<math id="A4.p2.2.m2.3" class="ltx_Math" alttext="\lVert{\bm{X}}_{k,v}^{l}\rVert" display="inline"><semantics id="A4.p2.2.m2.3a"><mrow id="A4.p2.2.m2.3.3.1" xref="A4.p2.2.m2.3.3.2.cmml"><mo fence="true" rspace="0em" id="A4.p2.2.m2.3.3.1.2" xref="A4.p2.2.m2.3.3.2.1.cmml">∥</mo><msubsup id="A4.p2.2.m2.3.3.1.1" xref="A4.p2.2.m2.3.3.1.1.cmml"><mi id="A4.p2.2.m2.3.3.1.1.2.2" xref="A4.p2.2.m2.3.3.1.1.2.2.cmml">𝑿</mi><mrow id="A4.p2.2.m2.2.2.2.4" xref="A4.p2.2.m2.2.2.2.3.cmml"><mi id="A4.p2.2.m2.1.1.1.1" xref="A4.p2.2.m2.1.1.1.1.cmml">k</mi><mo id="A4.p2.2.m2.2.2.2.4.1" xref="A4.p2.2.m2.2.2.2.3.cmml">,</mo><mi id="A4.p2.2.m2.2.2.2.2" xref="A4.p2.2.m2.2.2.2.2.cmml">v</mi></mrow><mi id="A4.p2.2.m2.3.3.1.1.3" xref="A4.p2.2.m2.3.3.1.1.3.cmml">l</mi></msubsup><mo fence="true" lspace="0em" id="A4.p2.2.m2.3.3.1.3" xref="A4.p2.2.m2.3.3.2.1.cmml">∥</mo></mrow><annotation-xml encoding="MathML-Content" id="A4.p2.2.m2.3b"><apply id="A4.p2.2.m2.3.3.2.cmml" xref="A4.p2.2.m2.3.3.1"><csymbol cd="latexml" id="A4.p2.2.m2.3.3.2.1.cmml" xref="A4.p2.2.m2.3.3.1.2">delimited-∥∥</csymbol><apply id="A4.p2.2.m2.3.3.1.1.cmml" xref="A4.p2.2.m2.3.3.1.1"><csymbol cd="ambiguous" id="A4.p2.2.m2.3.3.1.1.1.cmml" xref="A4.p2.2.m2.3.3.1.1">superscript</csymbol><apply id="A4.p2.2.m2.3.3.1.1.2.cmml" xref="A4.p2.2.m2.3.3.1.1"><csymbol cd="ambiguous" id="A4.p2.2.m2.3.3.1.1.2.1.cmml" xref="A4.p2.2.m2.3.3.1.1">subscript</csymbol><ci id="A4.p2.2.m2.3.3.1.1.2.2.cmml" xref="A4.p2.2.m2.3.3.1.1.2.2">𝑿</ci><list id="A4.p2.2.m2.2.2.2.3.cmml" xref="A4.p2.2.m2.2.2.2.4"><ci id="A4.p2.2.m2.1.1.1.1.cmml" xref="A4.p2.2.m2.1.1.1.1">𝑘</ci><ci id="A4.p2.2.m2.2.2.2.2.cmml" xref="A4.p2.2.m2.2.2.2.2">𝑣</ci></list></apply><ci id="A4.p2.2.m2.3.3.1.1.3.cmml" xref="A4.p2.2.m2.3.3.1.1.3">𝑙</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.2.m2.3c">\lVert{\bm{X}}_{k,v}^{l}\rVert</annotation></semantics></math>), unmerged tokens (<math id="A4.p2.3.m3.2" class="ltx_Math" alttext="{\bm{R}}_{k,v}^{l}" display="inline"><semantics id="A4.p2.3.m3.2a"><msubsup id="A4.p2.3.m3.2.3" xref="A4.p2.3.m3.2.3.cmml"><mi id="A4.p2.3.m3.2.3.2.2" xref="A4.p2.3.m3.2.3.2.2.cmml">𝑹</mi><mrow id="A4.p2.3.m3.2.2.2.4" xref="A4.p2.3.m3.2.2.2.3.cmml"><mi id="A4.p2.3.m3.1.1.1.1" xref="A4.p2.3.m3.1.1.1.1.cmml">k</mi><mo id="A4.p2.3.m3.2.2.2.4.1" xref="A4.p2.3.m3.2.2.2.3.cmml">,</mo><mi id="A4.p2.3.m3.2.2.2.2" xref="A4.p2.3.m3.2.2.2.2.cmml">v</mi></mrow><mi id="A4.p2.3.m3.2.3.3" xref="A4.p2.3.m3.2.3.3.cmml">l</mi></msubsup><annotation-xml encoding="MathML-Content" id="A4.p2.3.m3.2b"><apply id="A4.p2.3.m3.2.3.cmml" xref="A4.p2.3.m3.2.3"><csymbol cd="ambiguous" id="A4.p2.3.m3.2.3.1.cmml" xref="A4.p2.3.m3.2.3">superscript</csymbol><apply id="A4.p2.3.m3.2.3.2.cmml" xref="A4.p2.3.m3.2.3"><csymbol cd="ambiguous" id="A4.p2.3.m3.2.3.2.1.cmml" xref="A4.p2.3.m3.2.3">subscript</csymbol><ci id="A4.p2.3.m3.2.3.2.2.cmml" xref="A4.p2.3.m3.2.3.2.2">𝑹</ci><list id="A4.p2.3.m3.2.2.2.3.cmml" xref="A4.p2.3.m3.2.2.2.4"><ci id="A4.p2.3.m3.1.1.1.1.cmml" xref="A4.p2.3.m3.1.1.1.1">𝑘</ci><ci id="A4.p2.3.m3.2.2.2.2.cmml" xref="A4.p2.3.m3.2.2.2.2">𝑣</ci></list></apply><ci id="A4.p2.3.m3.2.3.3.cmml" xref="A4.p2.3.m3.2.3.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.3.m3.2c">{\bm{R}}_{k,v}^{l}</annotation></semantics></math>), retention indices (<math id="A4.p2.4.m4.2" class="ltx_Math" alttext="{\mathbb{I}}_{k,v}" display="inline"><semantics id="A4.p2.4.m4.2a"><msub id="A4.p2.4.m4.2.3" xref="A4.p2.4.m4.2.3.cmml"><mi id="A4.p2.4.m4.2.3.2" xref="A4.p2.4.m4.2.3.2.cmml">𝕀</mi><mrow id="A4.p2.4.m4.2.2.2.4" xref="A4.p2.4.m4.2.2.2.3.cmml"><mi id="A4.p2.4.m4.1.1.1.1" xref="A4.p2.4.m4.1.1.1.1.cmml">k</mi><mo id="A4.p2.4.m4.2.2.2.4.1" xref="A4.p2.4.m4.2.2.2.3.cmml">,</mo><mi id="A4.p2.4.m4.2.2.2.2" xref="A4.p2.4.m4.2.2.2.2.cmml">v</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A4.p2.4.m4.2b"><apply id="A4.p2.4.m4.2.3.cmml" xref="A4.p2.4.m4.2.3"><csymbol cd="ambiguous" id="A4.p2.4.m4.2.3.1.cmml" xref="A4.p2.4.m4.2.3">subscript</csymbol><ci id="A4.p2.4.m4.2.3.2.cmml" xref="A4.p2.4.m4.2.3.2">𝕀</ci><list id="A4.p2.4.m4.2.2.2.3.cmml" xref="A4.p2.4.m4.2.2.2.4"><ci id="A4.p2.4.m4.1.1.1.1.cmml" xref="A4.p2.4.m4.1.1.1.1">𝑘</ci><ci id="A4.p2.4.m4.2.2.2.2.cmml" xref="A4.p2.4.m4.2.2.2.2">𝑣</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.4.m4.2c">{\mathbb{I}}_{k,v}</annotation></semantics></math>), and the next tokens (<math id="A4.p2.5.m5.1" class="ltx_Math" alttext="{\bm{t}}^{l}" display="inline"><semantics id="A4.p2.5.m5.1a"><msup id="A4.p2.5.m5.1.1" xref="A4.p2.5.m5.1.1.cmml"><mi id="A4.p2.5.m5.1.1.2" xref="A4.p2.5.m5.1.1.2.cmml">𝒕</mi><mi id="A4.p2.5.m5.1.1.3" xref="A4.p2.5.m5.1.1.3.cmml">l</mi></msup><annotation-xml encoding="MathML-Content" id="A4.p2.5.m5.1b"><apply id="A4.p2.5.m5.1.1.cmml" xref="A4.p2.5.m5.1.1"><csymbol cd="ambiguous" id="A4.p2.5.m5.1.1.1.cmml" xref="A4.p2.5.m5.1.1">superscript</csymbol><ci id="A4.p2.5.m5.1.1.2.cmml" xref="A4.p2.5.m5.1.1.2">𝒕</ci><ci id="A4.p2.5.m5.1.1.3.cmml" xref="A4.p2.5.m5.1.1.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.5.m5.1c">{\bm{t}}^{l}</annotation></semantics></math>, <math id="A4.p2.6.m6.1" class="ltx_Math" alttext="{\bm{t}}^{l-1}" display="inline"><semantics id="A4.p2.6.m6.1a"><msup id="A4.p2.6.m6.1.1" xref="A4.p2.6.m6.1.1.cmml"><mi id="A4.p2.6.m6.1.1.2" xref="A4.p2.6.m6.1.1.2.cmml">𝒕</mi><mrow id="A4.p2.6.m6.1.1.3" xref="A4.p2.6.m6.1.1.3.cmml"><mi id="A4.p2.6.m6.1.1.3.2" xref="A4.p2.6.m6.1.1.3.2.cmml">l</mi><mo id="A4.p2.6.m6.1.1.3.1" xref="A4.p2.6.m6.1.1.3.1.cmml">−</mo><mn id="A4.p2.6.m6.1.1.3.3" xref="A4.p2.6.m6.1.1.3.3.cmml">1</mn></mrow></msup><annotation-xml encoding="MathML-Content" id="A4.p2.6.m6.1b"><apply id="A4.p2.6.m6.1.1.cmml" xref="A4.p2.6.m6.1.1"><csymbol cd="ambiguous" id="A4.p2.6.m6.1.1.1.cmml" xref="A4.p2.6.m6.1.1">superscript</csymbol><ci id="A4.p2.6.m6.1.1.2.cmml" xref="A4.p2.6.m6.1.1.2">𝒕</ci><apply id="A4.p2.6.m6.1.1.3.cmml" xref="A4.p2.6.m6.1.1.3"><minus id="A4.p2.6.m6.1.1.3.1.cmml" xref="A4.p2.6.m6.1.1.3.1"></minus><ci id="A4.p2.6.m6.1.1.3.2.cmml" xref="A4.p2.6.m6.1.1.3.2">𝑙</ci><cn type="integer" id="A4.p2.6.m6.1.1.3.3.cmml" xref="A4.p2.6.m6.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.6.m6.1c">{\bm{t}}^{l-1}</annotation></semantics></math>), the algorithm proceeds by rescaling the magnitude of the KV pairs. Specifically, <math id="A4.p2.7.m7.1" class="ltx_Math" alttext="\hat{{\bm{X}}}_{k}^{l}" display="inline"><semantics id="A4.p2.7.m7.1a"><msubsup id="A4.p2.7.m7.1.1" xref="A4.p2.7.m7.1.1.cmml"><mover accent="true" id="A4.p2.7.m7.1.1.2.2" xref="A4.p2.7.m7.1.1.2.2.cmml"><mi id="A4.p2.7.m7.1.1.2.2.2" xref="A4.p2.7.m7.1.1.2.2.2.cmml">𝑿</mi><mo id="A4.p2.7.m7.1.1.2.2.1" xref="A4.p2.7.m7.1.1.2.2.1.cmml">^</mo></mover><mi id="A4.p2.7.m7.1.1.2.3" xref="A4.p2.7.m7.1.1.2.3.cmml">k</mi><mi id="A4.p2.7.m7.1.1.3" xref="A4.p2.7.m7.1.1.3.cmml">l</mi></msubsup><annotation-xml encoding="MathML-Content" id="A4.p2.7.m7.1b"><apply id="A4.p2.7.m7.1.1.cmml" xref="A4.p2.7.m7.1.1"><csymbol cd="ambiguous" id="A4.p2.7.m7.1.1.1.cmml" xref="A4.p2.7.m7.1.1">superscript</csymbol><apply id="A4.p2.7.m7.1.1.2.cmml" xref="A4.p2.7.m7.1.1"><csymbol cd="ambiguous" id="A4.p2.7.m7.1.1.2.1.cmml" xref="A4.p2.7.m7.1.1">subscript</csymbol><apply id="A4.p2.7.m7.1.1.2.2.cmml" xref="A4.p2.7.m7.1.1.2.2"><ci id="A4.p2.7.m7.1.1.2.2.1.cmml" xref="A4.p2.7.m7.1.1.2.2.1">^</ci><ci id="A4.p2.7.m7.1.1.2.2.2.cmml" xref="A4.p2.7.m7.1.1.2.2.2">𝑿</ci></apply><ci id="A4.p2.7.m7.1.1.2.3.cmml" xref="A4.p2.7.m7.1.1.2.3">𝑘</ci></apply><ci id="A4.p2.7.m7.1.1.3.cmml" xref="A4.p2.7.m7.1.1.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.7.m7.1c">\hat{{\bm{X}}}_{k}^{l}</annotation></semantics></math> and <math id="A4.p2.8.m8.1" class="ltx_Math" alttext="\hat{{\bm{X}}}_{v}^{l}" display="inline"><semantics id="A4.p2.8.m8.1a"><msubsup id="A4.p2.8.m8.1.1" xref="A4.p2.8.m8.1.1.cmml"><mover accent="true" id="A4.p2.8.m8.1.1.2.2" xref="A4.p2.8.m8.1.1.2.2.cmml"><mi id="A4.p2.8.m8.1.1.2.2.2" xref="A4.p2.8.m8.1.1.2.2.2.cmml">𝑿</mi><mo id="A4.p2.8.m8.1.1.2.2.1" xref="A4.p2.8.m8.1.1.2.2.1.cmml">^</mo></mover><mi id="A4.p2.8.m8.1.1.2.3" xref="A4.p2.8.m8.1.1.2.3.cmml">v</mi><mi id="A4.p2.8.m8.1.1.3" xref="A4.p2.8.m8.1.1.3.cmml">l</mi></msubsup><annotation-xml encoding="MathML-Content" id="A4.p2.8.m8.1b"><apply id="A4.p2.8.m8.1.1.cmml" xref="A4.p2.8.m8.1.1"><csymbol cd="ambiguous" id="A4.p2.8.m8.1.1.1.cmml" xref="A4.p2.8.m8.1.1">superscript</csymbol><apply id="A4.p2.8.m8.1.1.2.cmml" xref="A4.p2.8.m8.1.1"><csymbol cd="ambiguous" id="A4.p2.8.m8.1.1.2.1.cmml" xref="A4.p2.8.m8.1.1">subscript</csymbol><apply id="A4.p2.8.m8.1.1.2.2.cmml" xref="A4.p2.8.m8.1.1.2.2"><ci id="A4.p2.8.m8.1.1.2.2.1.cmml" xref="A4.p2.8.m8.1.1.2.2.1">^</ci><ci id="A4.p2.8.m8.1.1.2.2.2.cmml" xref="A4.p2.8.m8.1.1.2.2.2">𝑿</ci></apply><ci id="A4.p2.8.m8.1.1.2.3.cmml" xref="A4.p2.8.m8.1.1.2.3">𝑣</ci></apply><ci id="A4.p2.8.m8.1.1.3.cmml" xref="A4.p2.8.m8.1.1.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.8.m8.1c">\hat{{\bm{X}}}_{v}^{l}</annotation></semantics></math> are computed by multiplying the normalized KV pairs <math id="A4.p2.9.m9.2" class="ltx_Math" alttext="{\bm{E}}_{k,v}^{l}" display="inline"><semantics id="A4.p2.9.m9.2a"><msubsup id="A4.p2.9.m9.2.3" xref="A4.p2.9.m9.2.3.cmml"><mi id="A4.p2.9.m9.2.3.2.2" xref="A4.p2.9.m9.2.3.2.2.cmml">𝑬</mi><mrow id="A4.p2.9.m9.2.2.2.4" xref="A4.p2.9.m9.2.2.2.3.cmml"><mi id="A4.p2.9.m9.1.1.1.1" xref="A4.p2.9.m9.1.1.1.1.cmml">k</mi><mo id="A4.p2.9.m9.2.2.2.4.1" xref="A4.p2.9.m9.2.2.2.3.cmml">,</mo><mi id="A4.p2.9.m9.2.2.2.2" xref="A4.p2.9.m9.2.2.2.2.cmml">v</mi></mrow><mi id="A4.p2.9.m9.2.3.3" xref="A4.p2.9.m9.2.3.3.cmml">l</mi></msubsup><annotation-xml encoding="MathML-Content" id="A4.p2.9.m9.2b"><apply id="A4.p2.9.m9.2.3.cmml" xref="A4.p2.9.m9.2.3"><csymbol cd="ambiguous" id="A4.p2.9.m9.2.3.1.cmml" xref="A4.p2.9.m9.2.3">superscript</csymbol><apply id="A4.p2.9.m9.2.3.2.cmml" xref="A4.p2.9.m9.2.3"><csymbol cd="ambiguous" id="A4.p2.9.m9.2.3.2.1.cmml" xref="A4.p2.9.m9.2.3">subscript</csymbol><ci id="A4.p2.9.m9.2.3.2.2.cmml" xref="A4.p2.9.m9.2.3.2.2">𝑬</ci><list id="A4.p2.9.m9.2.2.2.3.cmml" xref="A4.p2.9.m9.2.2.2.4"><ci id="A4.p2.9.m9.1.1.1.1.cmml" xref="A4.p2.9.m9.1.1.1.1">𝑘</ci><ci id="A4.p2.9.m9.2.2.2.2.cmml" xref="A4.p2.9.m9.2.2.2.2">𝑣</ci></list></apply><ci id="A4.p2.9.m9.2.3.3.cmml" xref="A4.p2.9.m9.2.3.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.9.m9.2c">{\bm{E}}_{k,v}^{l}</annotation></semantics></math> with their respective magnitude norms <math id="A4.p2.10.m10.3" class="ltx_Math" alttext="\lVert{\bm{X}}_{k,v}^{l}\rVert" display="inline"><semantics id="A4.p2.10.m10.3a"><mrow id="A4.p2.10.m10.3.3.1" xref="A4.p2.10.m10.3.3.2.cmml"><mo fence="true" rspace="0em" id="A4.p2.10.m10.3.3.1.2" xref="A4.p2.10.m10.3.3.2.1.cmml">∥</mo><msubsup id="A4.p2.10.m10.3.3.1.1" xref="A4.p2.10.m10.3.3.1.1.cmml"><mi id="A4.p2.10.m10.3.3.1.1.2.2" xref="A4.p2.10.m10.3.3.1.1.2.2.cmml">𝑿</mi><mrow id="A4.p2.10.m10.2.2.2.4" xref="A4.p2.10.m10.2.2.2.3.cmml"><mi id="A4.p2.10.m10.1.1.1.1" xref="A4.p2.10.m10.1.1.1.1.cmml">k</mi><mo id="A4.p2.10.m10.2.2.2.4.1" xref="A4.p2.10.m10.2.2.2.3.cmml">,</mo><mi id="A4.p2.10.m10.2.2.2.2" xref="A4.p2.10.m10.2.2.2.2.cmml">v</mi></mrow><mi id="A4.p2.10.m10.3.3.1.1.3" xref="A4.p2.10.m10.3.3.1.1.3.cmml">l</mi></msubsup><mo fence="true" lspace="0em" id="A4.p2.10.m10.3.3.1.3" xref="A4.p2.10.m10.3.3.2.1.cmml">∥</mo></mrow><annotation-xml encoding="MathML-Content" id="A4.p2.10.m10.3b"><apply id="A4.p2.10.m10.3.3.2.cmml" xref="A4.p2.10.m10.3.3.1"><csymbol cd="latexml" id="A4.p2.10.m10.3.3.2.1.cmml" xref="A4.p2.10.m10.3.3.1.2">delimited-∥∥</csymbol><apply id="A4.p2.10.m10.3.3.1.1.cmml" xref="A4.p2.10.m10.3.3.1.1"><csymbol cd="ambiguous" id="A4.p2.10.m10.3.3.1.1.1.cmml" xref="A4.p2.10.m10.3.3.1.1">superscript</csymbol><apply id="A4.p2.10.m10.3.3.1.1.2.cmml" xref="A4.p2.10.m10.3.3.1.1"><csymbol cd="ambiguous" id="A4.p2.10.m10.3.3.1.1.2.1.cmml" xref="A4.p2.10.m10.3.3.1.1">subscript</csymbol><ci id="A4.p2.10.m10.3.3.1.1.2.2.cmml" xref="A4.p2.10.m10.3.3.1.1.2.2">𝑿</ci><list id="A4.p2.10.m10.2.2.2.3.cmml" xref="A4.p2.10.m10.2.2.2.4"><ci id="A4.p2.10.m10.1.1.1.1.cmml" xref="A4.p2.10.m10.1.1.1.1">𝑘</ci><ci id="A4.p2.10.m10.2.2.2.2.cmml" xref="A4.p2.10.m10.2.2.2.2">𝑣</ci></list></apply><ci id="A4.p2.10.m10.3.3.1.1.3.cmml" xref="A4.p2.10.m10.3.3.1.1.3">𝑙</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.10.m10.3c">\lVert{\bm{X}}_{k,v}^{l}\rVert</annotation></semantics></math>. Following this, the algorithm restores unmerged tokens using the retention indices, updating <math id="A4.p2.11.m11.1" class="ltx_Math" alttext="\hat{{\bm{X}}}_{k}^{l}" display="inline"><semantics id="A4.p2.11.m11.1a"><msubsup id="A4.p2.11.m11.1.1" xref="A4.p2.11.m11.1.1.cmml"><mover accent="true" id="A4.p2.11.m11.1.1.2.2" xref="A4.p2.11.m11.1.1.2.2.cmml"><mi id="A4.p2.11.m11.1.1.2.2.2" xref="A4.p2.11.m11.1.1.2.2.2.cmml">𝑿</mi><mo id="A4.p2.11.m11.1.1.2.2.1" xref="A4.p2.11.m11.1.1.2.2.1.cmml">^</mo></mover><mi id="A4.p2.11.m11.1.1.2.3" xref="A4.p2.11.m11.1.1.2.3.cmml">k</mi><mi id="A4.p2.11.m11.1.1.3" xref="A4.p2.11.m11.1.1.3.cmml">l</mi></msubsup><annotation-xml encoding="MathML-Content" id="A4.p2.11.m11.1b"><apply id="A4.p2.11.m11.1.1.cmml" xref="A4.p2.11.m11.1.1"><csymbol cd="ambiguous" id="A4.p2.11.m11.1.1.1.cmml" xref="A4.p2.11.m11.1.1">superscript</csymbol><apply id="A4.p2.11.m11.1.1.2.cmml" xref="A4.p2.11.m11.1.1"><csymbol cd="ambiguous" id="A4.p2.11.m11.1.1.2.1.cmml" xref="A4.p2.11.m11.1.1">subscript</csymbol><apply id="A4.p2.11.m11.1.1.2.2.cmml" xref="A4.p2.11.m11.1.1.2.2"><ci id="A4.p2.11.m11.1.1.2.2.1.cmml" xref="A4.p2.11.m11.1.1.2.2.1">^</ci><ci id="A4.p2.11.m11.1.1.2.2.2.cmml" xref="A4.p2.11.m11.1.1.2.2.2">𝑿</ci></apply><ci id="A4.p2.11.m11.1.1.2.3.cmml" xref="A4.p2.11.m11.1.1.2.3">𝑘</ci></apply><ci id="A4.p2.11.m11.1.1.3.cmml" xref="A4.p2.11.m11.1.1.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.11.m11.1c">\hat{{\bm{X}}}_{k}^{l}</annotation></semantics></math> and <math id="A4.p2.12.m12.1" class="ltx_Math" alttext="\hat{{\bm{X}}}_{v}^{l}" display="inline"><semantics id="A4.p2.12.m12.1a"><msubsup id="A4.p2.12.m12.1.1" xref="A4.p2.12.m12.1.1.cmml"><mover accent="true" id="A4.p2.12.m12.1.1.2.2" xref="A4.p2.12.m12.1.1.2.2.cmml"><mi id="A4.p2.12.m12.1.1.2.2.2" xref="A4.p2.12.m12.1.1.2.2.2.cmml">𝑿</mi><mo id="A4.p2.12.m12.1.1.2.2.1" xref="A4.p2.12.m12.1.1.2.2.1.cmml">^</mo></mover><mi id="A4.p2.12.m12.1.1.2.3" xref="A4.p2.12.m12.1.1.2.3.cmml">v</mi><mi id="A4.p2.12.m12.1.1.3" xref="A4.p2.12.m12.1.1.3.cmml">l</mi></msubsup><annotation-xml encoding="MathML-Content" id="A4.p2.12.m12.1b"><apply id="A4.p2.12.m12.1.1.cmml" xref="A4.p2.12.m12.1.1"><csymbol cd="ambiguous" id="A4.p2.12.m12.1.1.1.cmml" xref="A4.p2.12.m12.1.1">superscript</csymbol><apply id="A4.p2.12.m12.1.1.2.cmml" xref="A4.p2.12.m12.1.1"><csymbol cd="ambiguous" id="A4.p2.12.m12.1.1.2.1.cmml" xref="A4.p2.12.m12.1.1">subscript</csymbol><apply id="A4.p2.12.m12.1.1.2.2.cmml" xref="A4.p2.12.m12.1.1.2.2"><ci id="A4.p2.12.m12.1.1.2.2.1.cmml" xref="A4.p2.12.m12.1.1.2.2.1">^</ci><ci id="A4.p2.12.m12.1.1.2.2.2.cmml" xref="A4.p2.12.m12.1.1.2.2.2">𝑿</ci></apply><ci id="A4.p2.12.m12.1.1.2.3.cmml" xref="A4.p2.12.m12.1.1.2.3">𝑣</ci></apply><ci id="A4.p2.12.m12.1.1.3.cmml" xref="A4.p2.12.m12.1.1.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.12.m12.1c">\hat{{\bm{X}}}_{v}^{l}</annotation></semantics></math> accordingly.
Next, the new tokens <math id="A4.p2.13.m13.1" class="ltx_Math" alttext="{\bm{t}}_{k}" display="inline"><semantics id="A4.p2.13.m13.1a"><msub id="A4.p2.13.m13.1.1" xref="A4.p2.13.m13.1.1.cmml"><mi id="A4.p2.13.m13.1.1.2" xref="A4.p2.13.m13.1.1.2.cmml">𝒕</mi><mi id="A4.p2.13.m13.1.1.3" xref="A4.p2.13.m13.1.1.3.cmml">k</mi></msub><annotation-xml encoding="MathML-Content" id="A4.p2.13.m13.1b"><apply id="A4.p2.13.m13.1.1.cmml" xref="A4.p2.13.m13.1.1"><csymbol cd="ambiguous" id="A4.p2.13.m13.1.1.1.cmml" xref="A4.p2.13.m13.1.1">subscript</csymbol><ci id="A4.p2.13.m13.1.1.2.cmml" xref="A4.p2.13.m13.1.1.2">𝒕</ci><ci id="A4.p2.13.m13.1.1.3.cmml" xref="A4.p2.13.m13.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.13.m13.1c">{\bm{t}}_{k}</annotation></semantics></math> and <math id="A4.p2.14.m14.1" class="ltx_Math" alttext="{\bm{t}}_{v}" display="inline"><semantics id="A4.p2.14.m14.1a"><msub id="A4.p2.14.m14.1.1" xref="A4.p2.14.m14.1.1.cmml"><mi id="A4.p2.14.m14.1.1.2" xref="A4.p2.14.m14.1.1.2.cmml">𝒕</mi><mi id="A4.p2.14.m14.1.1.3" xref="A4.p2.14.m14.1.1.3.cmml">v</mi></msub><annotation-xml encoding="MathML-Content" id="A4.p2.14.m14.1b"><apply id="A4.p2.14.m14.1.1.cmml" xref="A4.p2.14.m14.1.1"><csymbol cd="ambiguous" id="A4.p2.14.m14.1.1.1.cmml" xref="A4.p2.14.m14.1.1">subscript</csymbol><ci id="A4.p2.14.m14.1.1.2.cmml" xref="A4.p2.14.m14.1.1.2">𝒕</ci><ci id="A4.p2.14.m14.1.1.3.cmml" xref="A4.p2.14.m14.1.1.3">𝑣</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.14.m14.1c">{\bm{t}}_{v}</annotation></semantics></math> are concatenated to the rescaled KV pairs along the token dimension. This augmented KV cache undergoes a softmax attention mechanism where the attention scores <math id="A4.p2.15.m15.1" class="ltx_Math" alttext="{\bm{A}}" display="inline"><semantics id="A4.p2.15.m15.1a"><mi id="A4.p2.15.m15.1.1" xref="A4.p2.15.m15.1.1.cmml">𝑨</mi><annotation-xml encoding="MathML-Content" id="A4.p2.15.m15.1b"><ci id="A4.p2.15.m15.1.1.cmml" xref="A4.p2.15.m15.1.1">𝑨</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.15.m15.1c">{\bm{A}}</annotation></semantics></math> are computed by taking the dot product of the query token <math id="A4.p2.16.m16.1" class="ltx_Math" alttext="\bm{t}_{q}" display="inline"><semantics id="A4.p2.16.m16.1a"><msub id="A4.p2.16.m16.1.1" xref="A4.p2.16.m16.1.1.cmml"><mi id="A4.p2.16.m16.1.1.2" xref="A4.p2.16.m16.1.1.2.cmml">𝒕</mi><mi id="A4.p2.16.m16.1.1.3" xref="A4.p2.16.m16.1.1.3.cmml">q</mi></msub><annotation-xml encoding="MathML-Content" id="A4.p2.16.m16.1b"><apply id="A4.p2.16.m16.1.1.cmml" xref="A4.p2.16.m16.1.1"><csymbol cd="ambiguous" id="A4.p2.16.m16.1.1.1.cmml" xref="A4.p2.16.m16.1.1">subscript</csymbol><ci id="A4.p2.16.m16.1.1.2.cmml" xref="A4.p2.16.m16.1.1.2">𝒕</ci><ci id="A4.p2.16.m16.1.1.3.cmml" xref="A4.p2.16.m16.1.1.3">𝑞</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.16.m16.1c">\bm{t}_{q}</annotation></semantics></math> with the transposed keys <math id="A4.p2.17.m17.1" class="ltx_Math" alttext="(\hat{{\bm{X}}}_{k}^{l})^{\top}" display="inline"><semantics id="A4.p2.17.m17.1a"><msup id="A4.p2.17.m17.1.1" xref="A4.p2.17.m17.1.1.cmml"><mrow id="A4.p2.17.m17.1.1.1.1" xref="A4.p2.17.m17.1.1.1.1.1.cmml"><mo stretchy="false" id="A4.p2.17.m17.1.1.1.1.2" xref="A4.p2.17.m17.1.1.1.1.1.cmml">(</mo><msubsup id="A4.p2.17.m17.1.1.1.1.1" xref="A4.p2.17.m17.1.1.1.1.1.cmml"><mover accent="true" id="A4.p2.17.m17.1.1.1.1.1.2.2" xref="A4.p2.17.m17.1.1.1.1.1.2.2.cmml"><mi id="A4.p2.17.m17.1.1.1.1.1.2.2.2" xref="A4.p2.17.m17.1.1.1.1.1.2.2.2.cmml">𝑿</mi><mo id="A4.p2.17.m17.1.1.1.1.1.2.2.1" xref="A4.p2.17.m17.1.1.1.1.1.2.2.1.cmml">^</mo></mover><mi id="A4.p2.17.m17.1.1.1.1.1.2.3" xref="A4.p2.17.m17.1.1.1.1.1.2.3.cmml">k</mi><mi id="A4.p2.17.m17.1.1.1.1.1.3" xref="A4.p2.17.m17.1.1.1.1.1.3.cmml">l</mi></msubsup><mo stretchy="false" id="A4.p2.17.m17.1.1.1.1.3" xref="A4.p2.17.m17.1.1.1.1.1.cmml">)</mo></mrow><mo id="A4.p2.17.m17.1.1.3" xref="A4.p2.17.m17.1.1.3.cmml">⊤</mo></msup><annotation-xml encoding="MathML-Content" id="A4.p2.17.m17.1b"><apply id="A4.p2.17.m17.1.1.cmml" xref="A4.p2.17.m17.1.1"><csymbol cd="ambiguous" id="A4.p2.17.m17.1.1.2.cmml" xref="A4.p2.17.m17.1.1">superscript</csymbol><apply id="A4.p2.17.m17.1.1.1.1.1.cmml" xref="A4.p2.17.m17.1.1.1.1"><csymbol cd="ambiguous" id="A4.p2.17.m17.1.1.1.1.1.1.cmml" xref="A4.p2.17.m17.1.1.1.1">superscript</csymbol><apply id="A4.p2.17.m17.1.1.1.1.1.2.cmml" xref="A4.p2.17.m17.1.1.1.1"><csymbol cd="ambiguous" id="A4.p2.17.m17.1.1.1.1.1.2.1.cmml" xref="A4.p2.17.m17.1.1.1.1">subscript</csymbol><apply id="A4.p2.17.m17.1.1.1.1.1.2.2.cmml" xref="A4.p2.17.m17.1.1.1.1.1.2.2"><ci id="A4.p2.17.m17.1.1.1.1.1.2.2.1.cmml" xref="A4.p2.17.m17.1.1.1.1.1.2.2.1">^</ci><ci id="A4.p2.17.m17.1.1.1.1.1.2.2.2.cmml" xref="A4.p2.17.m17.1.1.1.1.1.2.2.2">𝑿</ci></apply><ci id="A4.p2.17.m17.1.1.1.1.1.2.3.cmml" xref="A4.p2.17.m17.1.1.1.1.1.2.3">𝑘</ci></apply><ci id="A4.p2.17.m17.1.1.1.1.1.3.cmml" xref="A4.p2.17.m17.1.1.1.1.1.3">𝑙</ci></apply><csymbol cd="latexml" id="A4.p2.17.m17.1.1.3.cmml" xref="A4.p2.17.m17.1.1.3">top</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.17.m17.1c">(\hat{{\bm{X}}}_{k}^{l})^{\top}</annotation></semantics></math>. The output token <math id="A4.p2.18.m18.1" class="ltx_Math" alttext="{\bm{t}}_{O}" display="inline"><semantics id="A4.p2.18.m18.1a"><msub id="A4.p2.18.m18.1.1" xref="A4.p2.18.m18.1.1.cmml"><mi id="A4.p2.18.m18.1.1.2" xref="A4.p2.18.m18.1.1.2.cmml">𝒕</mi><mi id="A4.p2.18.m18.1.1.3" xref="A4.p2.18.m18.1.1.3.cmml">O</mi></msub><annotation-xml encoding="MathML-Content" id="A4.p2.18.m18.1b"><apply id="A4.p2.18.m18.1.1.cmml" xref="A4.p2.18.m18.1.1"><csymbol cd="ambiguous" id="A4.p2.18.m18.1.1.1.cmml" xref="A4.p2.18.m18.1.1">subscript</csymbol><ci id="A4.p2.18.m18.1.1.2.cmml" xref="A4.p2.18.m18.1.1.2">𝒕</ci><ci id="A4.p2.18.m18.1.1.3.cmml" xref="A4.p2.18.m18.1.1.3">𝑂</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.18.m18.1c">{\bm{t}}_{O}</annotation></semantics></math> is then obtained by multiplying the attention scores <math id="A4.p2.19.m19.1" class="ltx_Math" alttext="{\bm{A}}" display="inline"><semantics id="A4.p2.19.m19.1a"><mi id="A4.p2.19.m19.1.1" xref="A4.p2.19.m19.1.1.cmml">𝑨</mi><annotation-xml encoding="MathML-Content" id="A4.p2.19.m19.1b"><ci id="A4.p2.19.m19.1.1.cmml" xref="A4.p2.19.m19.1.1">𝑨</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.19.m19.1c">{\bm{A}}</annotation></semantics></math> with the values <math id="A4.p2.20.m20.1" class="ltx_Math" alttext="\hat{{\bm{X}}}_{v}^{l}" display="inline"><semantics id="A4.p2.20.m20.1a"><msubsup id="A4.p2.20.m20.1.1" xref="A4.p2.20.m20.1.1.cmml"><mover accent="true" id="A4.p2.20.m20.1.1.2.2" xref="A4.p2.20.m20.1.1.2.2.cmml"><mi id="A4.p2.20.m20.1.1.2.2.2" xref="A4.p2.20.m20.1.1.2.2.2.cmml">𝑿</mi><mo id="A4.p2.20.m20.1.1.2.2.1" xref="A4.p2.20.m20.1.1.2.2.1.cmml">^</mo></mover><mi id="A4.p2.20.m20.1.1.2.3" xref="A4.p2.20.m20.1.1.2.3.cmml">v</mi><mi id="A4.p2.20.m20.1.1.3" xref="A4.p2.20.m20.1.1.3.cmml">l</mi></msubsup><annotation-xml encoding="MathML-Content" id="A4.p2.20.m20.1b"><apply id="A4.p2.20.m20.1.1.cmml" xref="A4.p2.20.m20.1.1"><csymbol cd="ambiguous" id="A4.p2.20.m20.1.1.1.cmml" xref="A4.p2.20.m20.1.1">superscript</csymbol><apply id="A4.p2.20.m20.1.1.2.cmml" xref="A4.p2.20.m20.1.1"><csymbol cd="ambiguous" id="A4.p2.20.m20.1.1.2.1.cmml" xref="A4.p2.20.m20.1.1">subscript</csymbol><apply id="A4.p2.20.m20.1.1.2.2.cmml" xref="A4.p2.20.m20.1.1.2.2"><ci id="A4.p2.20.m20.1.1.2.2.1.cmml" xref="A4.p2.20.m20.1.1.2.2.1">^</ci><ci id="A4.p2.20.m20.1.1.2.2.2.cmml" xref="A4.p2.20.m20.1.1.2.2.2">𝑿</ci></apply><ci id="A4.p2.20.m20.1.1.2.3.cmml" xref="A4.p2.20.m20.1.1.2.3">𝑣</ci></apply><ci id="A4.p2.20.m20.1.1.3.cmml" xref="A4.p2.20.m20.1.1.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.20.m20.1c">\hat{{\bm{X}}}_{v}^{l}</annotation></semantics></math>.
In cases where the previous token <math id="A4.p2.21.m21.1" class="ltx_Math" alttext="{\bm{t}}^{l-1}" display="inline"><semantics id="A4.p2.21.m21.1a"><msup id="A4.p2.21.m21.1.1" xref="A4.p2.21.m21.1.1.cmml"><mi id="A4.p2.21.m21.1.1.2" xref="A4.p2.21.m21.1.1.2.cmml">𝒕</mi><mrow id="A4.p2.21.m21.1.1.3" xref="A4.p2.21.m21.1.1.3.cmml"><mi id="A4.p2.21.m21.1.1.3.2" xref="A4.p2.21.m21.1.1.3.2.cmml">l</mi><mo id="A4.p2.21.m21.1.1.3.1" xref="A4.p2.21.m21.1.1.3.1.cmml">−</mo><mn id="A4.p2.21.m21.1.1.3.3" xref="A4.p2.21.m21.1.1.3.3.cmml">1</mn></mrow></msup><annotation-xml encoding="MathML-Content" id="A4.p2.21.m21.1b"><apply id="A4.p2.21.m21.1.1.cmml" xref="A4.p2.21.m21.1.1"><csymbol cd="ambiguous" id="A4.p2.21.m21.1.1.1.cmml" xref="A4.p2.21.m21.1.1">superscript</csymbol><ci id="A4.p2.21.m21.1.1.2.cmml" xref="A4.p2.21.m21.1.1.2">𝒕</ci><apply id="A4.p2.21.m21.1.1.3.cmml" xref="A4.p2.21.m21.1.1.3"><minus id="A4.p2.21.m21.1.1.3.1.cmml" xref="A4.p2.21.m21.1.1.3.1"></minus><ci id="A4.p2.21.m21.1.1.3.2.cmml" xref="A4.p2.21.m21.1.1.3.2">𝑙</ci><cn type="integer" id="A4.p2.21.m21.1.1.3.3.cmml" xref="A4.p2.21.m21.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.21.m21.1c">{\bm{t}}^{l-1}</annotation></semantics></math> exists, the algorithm performs a compression step. It concatenates the existing KV cache <math id="A4.p2.22.m22.2" class="ltx_Math" alttext="{\bm{E}}_{k,v}^{l}" display="inline"><semantics id="A4.p2.22.m22.2a"><msubsup id="A4.p2.22.m22.2.3" xref="A4.p2.22.m22.2.3.cmml"><mi id="A4.p2.22.m22.2.3.2.2" xref="A4.p2.22.m22.2.3.2.2.cmml">𝑬</mi><mrow id="A4.p2.22.m22.2.2.2.4" xref="A4.p2.22.m22.2.2.2.3.cmml"><mi id="A4.p2.22.m22.1.1.1.1" xref="A4.p2.22.m22.1.1.1.1.cmml">k</mi><mo id="A4.p2.22.m22.2.2.2.4.1" xref="A4.p2.22.m22.2.2.2.3.cmml">,</mo><mi id="A4.p2.22.m22.2.2.2.2" xref="A4.p2.22.m22.2.2.2.2.cmml">v</mi></mrow><mi id="A4.p2.22.m22.2.3.3" xref="A4.p2.22.m22.2.3.3.cmml">l</mi></msubsup><annotation-xml encoding="MathML-Content" id="A4.p2.22.m22.2b"><apply id="A4.p2.22.m22.2.3.cmml" xref="A4.p2.22.m22.2.3"><csymbol cd="ambiguous" id="A4.p2.22.m22.2.3.1.cmml" xref="A4.p2.22.m22.2.3">superscript</csymbol><apply id="A4.p2.22.m22.2.3.2.cmml" xref="A4.p2.22.m22.2.3"><csymbol cd="ambiguous" id="A4.p2.22.m22.2.3.2.1.cmml" xref="A4.p2.22.m22.2.3">subscript</csymbol><ci id="A4.p2.22.m22.2.3.2.2.cmml" xref="A4.p2.22.m22.2.3.2.2">𝑬</ci><list id="A4.p2.22.m22.2.2.2.3.cmml" xref="A4.p2.22.m22.2.2.2.4"><ci id="A4.p2.22.m22.1.1.1.1.cmml" xref="A4.p2.22.m22.1.1.1.1">𝑘</ci><ci id="A4.p2.22.m22.2.2.2.2.cmml" xref="A4.p2.22.m22.2.2.2.2">𝑣</ci></list></apply><ci id="A4.p2.22.m22.2.3.3.cmml" xref="A4.p2.22.m22.2.3.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.22.m22.2c">{\bm{E}}_{k,v}^{l}</annotation></semantics></math> with the merged tokens resulting from the current and previous layers, effectively reducing redundancy and optimizing memory. If <math id="A4.p2.23.m23.1" class="ltx_Math" alttext="{\bm{t}}^{l-1}" display="inline"><semantics id="A4.p2.23.m23.1a"><msup id="A4.p2.23.m23.1.1" xref="A4.p2.23.m23.1.1.cmml"><mi id="A4.p2.23.m23.1.1.2" xref="A4.p2.23.m23.1.1.2.cmml">𝒕</mi><mrow id="A4.p2.23.m23.1.1.3" xref="A4.p2.23.m23.1.1.3.cmml"><mi id="A4.p2.23.m23.1.1.3.2" xref="A4.p2.23.m23.1.1.3.2.cmml">l</mi><mo id="A4.p2.23.m23.1.1.3.1" xref="A4.p2.23.m23.1.1.3.1.cmml">−</mo><mn id="A4.p2.23.m23.1.1.3.3" xref="A4.p2.23.m23.1.1.3.3.cmml">1</mn></mrow></msup><annotation-xml encoding="MathML-Content" id="A4.p2.23.m23.1b"><apply id="A4.p2.23.m23.1.1.cmml" xref="A4.p2.23.m23.1.1"><csymbol cd="ambiguous" id="A4.p2.23.m23.1.1.1.cmml" xref="A4.p2.23.m23.1.1">superscript</csymbol><ci id="A4.p2.23.m23.1.1.2.cmml" xref="A4.p2.23.m23.1.1.2">𝒕</ci><apply id="A4.p2.23.m23.1.1.3.cmml" xref="A4.p2.23.m23.1.1.3"><minus id="A4.p2.23.m23.1.1.3.1.cmml" xref="A4.p2.23.m23.1.1.3.1"></minus><ci id="A4.p2.23.m23.1.1.3.2.cmml" xref="A4.p2.23.m23.1.1.3.2">𝑙</ci><cn type="integer" id="A4.p2.23.m23.1.1.3.3.cmml" xref="A4.p2.23.m23.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.23.m23.1c">{\bm{t}}^{l-1}</annotation></semantics></math> is not available, the KV cache is updated by simply concatenating <math id="A4.p2.24.m24.2" class="ltx_Math" alttext="{\bm{E}}_{k,v}^{l}" display="inline"><semantics id="A4.p2.24.m24.2a"><msubsup id="A4.p2.24.m24.2.3" xref="A4.p2.24.m24.2.3.cmml"><mi id="A4.p2.24.m24.2.3.2.2" xref="A4.p2.24.m24.2.3.2.2.cmml">𝑬</mi><mrow id="A4.p2.24.m24.2.2.2.4" xref="A4.p2.24.m24.2.2.2.3.cmml"><mi id="A4.p2.24.m24.1.1.1.1" xref="A4.p2.24.m24.1.1.1.1.cmml">k</mi><mo id="A4.p2.24.m24.2.2.2.4.1" xref="A4.p2.24.m24.2.2.2.3.cmml">,</mo><mi id="A4.p2.24.m24.2.2.2.2" xref="A4.p2.24.m24.2.2.2.2.cmml">v</mi></mrow><mi id="A4.p2.24.m24.2.3.3" xref="A4.p2.24.m24.2.3.3.cmml">l</mi></msubsup><annotation-xml encoding="MathML-Content" id="A4.p2.24.m24.2b"><apply id="A4.p2.24.m24.2.3.cmml" xref="A4.p2.24.m24.2.3"><csymbol cd="ambiguous" id="A4.p2.24.m24.2.3.1.cmml" xref="A4.p2.24.m24.2.3">superscript</csymbol><apply id="A4.p2.24.m24.2.3.2.cmml" xref="A4.p2.24.m24.2.3"><csymbol cd="ambiguous" id="A4.p2.24.m24.2.3.2.1.cmml" xref="A4.p2.24.m24.2.3">subscript</csymbol><ci id="A4.p2.24.m24.2.3.2.2.cmml" xref="A4.p2.24.m24.2.3.2.2">𝑬</ci><list id="A4.p2.24.m24.2.2.2.3.cmml" xref="A4.p2.24.m24.2.2.2.4"><ci id="A4.p2.24.m24.1.1.1.1.cmml" xref="A4.p2.24.m24.1.1.1.1">𝑘</ci><ci id="A4.p2.24.m24.2.2.2.2.cmml" xref="A4.p2.24.m24.2.2.2.2">𝑣</ci></list></apply><ci id="A4.p2.24.m24.2.3.3.cmml" xref="A4.p2.24.m24.2.3.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.24.m24.2c">{\bm{E}}_{k,v}^{l}</annotation></semantics></math> with the new tokens <math id="A4.p2.25.m25.2" class="ltx_Math" alttext="{\bm{t}}_{k,v}^{l}" display="inline"><semantics id="A4.p2.25.m25.2a"><msubsup id="A4.p2.25.m25.2.3" xref="A4.p2.25.m25.2.3.cmml"><mi id="A4.p2.25.m25.2.3.2.2" xref="A4.p2.25.m25.2.3.2.2.cmml">𝒕</mi><mrow id="A4.p2.25.m25.2.2.2.4" xref="A4.p2.25.m25.2.2.2.3.cmml"><mi id="A4.p2.25.m25.1.1.1.1" xref="A4.p2.25.m25.1.1.1.1.cmml">k</mi><mo id="A4.p2.25.m25.2.2.2.4.1" xref="A4.p2.25.m25.2.2.2.3.cmml">,</mo><mi id="A4.p2.25.m25.2.2.2.2" xref="A4.p2.25.m25.2.2.2.2.cmml">v</mi></mrow><mi id="A4.p2.25.m25.2.3.3" xref="A4.p2.25.m25.2.3.3.cmml">l</mi></msubsup><annotation-xml encoding="MathML-Content" id="A4.p2.25.m25.2b"><apply id="A4.p2.25.m25.2.3.cmml" xref="A4.p2.25.m25.2.3"><csymbol cd="ambiguous" id="A4.p2.25.m25.2.3.1.cmml" xref="A4.p2.25.m25.2.3">superscript</csymbol><apply id="A4.p2.25.m25.2.3.2.cmml" xref="A4.p2.25.m25.2.3"><csymbol cd="ambiguous" id="A4.p2.25.m25.2.3.2.1.cmml" xref="A4.p2.25.m25.2.3">subscript</csymbol><ci id="A4.p2.25.m25.2.3.2.2.cmml" xref="A4.p2.25.m25.2.3.2.2">𝒕</ci><list id="A4.p2.25.m25.2.2.2.3.cmml" xref="A4.p2.25.m25.2.2.2.4"><ci id="A4.p2.25.m25.1.1.1.1.cmml" xref="A4.p2.25.m25.1.1.1.1">𝑘</ci><ci id="A4.p2.25.m25.2.2.2.2.cmml" xref="A4.p2.25.m25.2.2.2.2">𝑣</ci></list></apply><ci id="A4.p2.25.m25.2.3.3.cmml" xref="A4.p2.25.m25.2.3.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.25.m25.2c">{\bm{t}}_{k,v}^{l}</annotation></semantics></math>, deferring the compression until the next iteration. The final output token <math id="A4.p2.26.m26.1" class="ltx_Math" alttext="{\bm{t}}_{O}" display="inline"><semantics id="A4.p2.26.m26.1a"><msub id="A4.p2.26.m26.1.1" xref="A4.p2.26.m26.1.1.cmml"><mi id="A4.p2.26.m26.1.1.2" xref="A4.p2.26.m26.1.1.2.cmml">𝒕</mi><mi id="A4.p2.26.m26.1.1.3" xref="A4.p2.26.m26.1.1.3.cmml">O</mi></msub><annotation-xml encoding="MathML-Content" id="A4.p2.26.m26.1b"><apply id="A4.p2.26.m26.1.1.cmml" xref="A4.p2.26.m26.1.1"><csymbol cd="ambiguous" id="A4.p2.26.m26.1.1.1.cmml" xref="A4.p2.26.m26.1.1">subscript</csymbol><ci id="A4.p2.26.m26.1.1.2.cmml" xref="A4.p2.26.m26.1.1.2">𝒕</ci><ci id="A4.p2.26.m26.1.1.3.cmml" xref="A4.p2.26.m26.1.1.3">𝑂</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.26.m26.1c">{\bm{t}}_{O}</annotation></semantics></math> is then returned, concluding the decoding process.
In the merging function, the algorithm normalizes the KV pairs from both the current and previous layers. It calculates the angular distance <math id="A4.p2.27.m27.1" class="ltx_Math" alttext="\Omega" display="inline"><semantics id="A4.p2.27.m27.1a"><mi mathvariant="normal" id="A4.p2.27.m27.1.1" xref="A4.p2.27.m27.1.1.cmml">Ω</mi><annotation-xml encoding="MathML-Content" id="A4.p2.27.m27.1b"><ci id="A4.p2.27.m27.1.1.cmml" xref="A4.p2.27.m27.1.1">Ω</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.p2.27.m27.1c">\Omega</annotation></semantics></math> between the normalized vectors, ensuring that the interpolation occurs along the shortest path on the unit sphere. The merged KV cache is then obtained by weighted interpolation of the normalized vectors, preserving the geometric and semantic integrity of the original states. This comprehensive process allows MiniCache to achieve substantial memory efficiencies while maintaining the functional characteristics of the KV pairs across transformer layers.</p>
</div>
<figure id="A4.F1" class="ltx_figure"><img src="/html/2405.14366/assets/x10.png" id="A4.F1.g1" class="ltx_graphics ltx_centering ltx_img_portrait" width="368" height="494" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A4.F1.3.1.1" class="ltx_text" style="font-size:90%;">Figure A</span>: </span><span id="A4.F1.4.2" class="ltx_text" style="font-size:90%;">Overall prefilling and decoding logic for <span id="A4.F1.4.2.1" class="ltx_text ltx_font_bold">MiniCache</span> involves performing cross-layer merging and recovery within our framework.</span></figcaption>
</figure>
<div id="A4.p3" class="ltx_para">
<p id="A4.p3.2" class="ltx_p"><span id="A4.p3.2.1" class="ltx_text ltx_font_bold">MiniCache execution flow.</span>
<a href="#A4.F1" title="In Appendix D Additional Implementation Details ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">A</span></a> delineates the pre-filling and decoding logic for the MiniCache framework, which incorporates cross-layer merging and error suppression to achieve memory efficiency and maintain functional integrity. Initially, in Step 1, the KV cache is fetched from the previous layer (Layer L-1) during the pre-filling phase. In Step 2, the fetched KV pairs from the current layer (<math id="A4.p3.1.m1.1" class="ltx_Math" alttext="\chi^{L}" display="inline"><semantics id="A4.p3.1.m1.1a"><msup id="A4.p3.1.m1.1.1" xref="A4.p3.1.m1.1.1.cmml"><mi id="A4.p3.1.m1.1.1.2" xref="A4.p3.1.m1.1.1.2.cmml">χ</mi><mi id="A4.p3.1.m1.1.1.3" xref="A4.p3.1.m1.1.1.3.cmml">L</mi></msup><annotation-xml encoding="MathML-Content" id="A4.p3.1.m1.1b"><apply id="A4.p3.1.m1.1.1.cmml" xref="A4.p3.1.m1.1.1"><csymbol cd="ambiguous" id="A4.p3.1.m1.1.1.1.cmml" xref="A4.p3.1.m1.1.1">superscript</csymbol><ci id="A4.p3.1.m1.1.1.2.cmml" xref="A4.p3.1.m1.1.1.2">𝜒</ci><ci id="A4.p3.1.m1.1.1.3.cmml" xref="A4.p3.1.m1.1.1.3">𝐿</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p3.1.m1.1c">\chi^{L}</annotation></semantics></math>) are merged with the KV pairs from the preceding layer (<math id="A4.p3.2.m2.1" class="ltx_Math" alttext="\chi^{L-1}" display="inline"><semantics id="A4.p3.2.m2.1a"><msup id="A4.p3.2.m2.1.1" xref="A4.p3.2.m2.1.1.cmml"><mi id="A4.p3.2.m2.1.1.2" xref="A4.p3.2.m2.1.1.2.cmml">χ</mi><mrow id="A4.p3.2.m2.1.1.3" xref="A4.p3.2.m2.1.1.3.cmml"><mi id="A4.p3.2.m2.1.1.3.2" xref="A4.p3.2.m2.1.1.3.2.cmml">L</mi><mo id="A4.p3.2.m2.1.1.3.1" xref="A4.p3.2.m2.1.1.3.1.cmml">−</mo><mn id="A4.p3.2.m2.1.1.3.3" xref="A4.p3.2.m2.1.1.3.3.cmml">1</mn></mrow></msup><annotation-xml encoding="MathML-Content" id="A4.p3.2.m2.1b"><apply id="A4.p3.2.m2.1.1.cmml" xref="A4.p3.2.m2.1.1"><csymbol cd="ambiguous" id="A4.p3.2.m2.1.1.1.cmml" xref="A4.p3.2.m2.1.1">superscript</csymbol><ci id="A4.p3.2.m2.1.1.2.cmml" xref="A4.p3.2.m2.1.1.2">𝜒</ci><apply id="A4.p3.2.m2.1.1.3.cmml" xref="A4.p3.2.m2.1.1.3"><minus id="A4.p3.2.m2.1.1.3.1.cmml" xref="A4.p3.2.m2.1.1.3.1"></minus><ci id="A4.p3.2.m2.1.1.3.2.cmml" xref="A4.p3.2.m2.1.1.3.2">𝐿</ci><cn type="integer" id="A4.p3.2.m2.1.1.3.3.cmml" xref="A4.p3.2.m2.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p3.2.m2.1c">\chi^{L-1}</annotation></semantics></math>), reducing redundancy through a merge function. Subsequently, in Step 3, the merged KV pairs are cached for future use, representing a consolidated data set from multiple layers.
During the decoding phase, Step 4 involves deleting unnecessary or redundant KV pairs to optimize memory usage. In Step 5, the decoder fetches the required KV pairs from the cache for output generation. Step 6 applies error suppression mechanisms to the fetched KV pairs, including rescaling and retention recovery, to minimize errors introduced during the merging and compression processes. Finally, in Step 7, the cache is updated with the final KV pairs post-error suppression and adjustments, ensuring the cache contains the most accurate and efficient representation of the KV pairs for subsequent layers. This comprehensive approach guarantees substantial memory efficiencies while preserving the critical functional characteristics of the original KV pairs across transformer layers.</p>
</div>
<figure id="alg1" class="ltx_float ltx_algorithm">
<div id="alg1.32" class="ltx_listing ltx_lst_numbers_left ltx_listing">
<div id="alg1.32.33" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">1</span>




</div>
<div id="alg1.32.34" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">2</span><span id="alg1.32.34.1" class="ltx_text ltx_font_bold">procedure</span> <em id="alg1.32.34.2" class="ltx_emph ltx_font_typewriter">MiniCache Inference</em><span id="alg1.32.34.3" class="ltx_text ltx_font_bold">:</span>
</div>
<div id="alg1.3.3" class="ltx_listingline">  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<span id="alg1.3.3.1" class="ltx_text"><span id="alg1.3.3.1.1" class="ltx_text ltx_font_bold">Input:</span> </span>Input Tokens: <math id="alg1.1.1.m1.1" class="ltx_Math" alttext="{\bm{T}}\in\mathbb{R}^{t_{\text{input}}\times d}" display="inline"><semantics id="alg1.1.1.m1.1a"><mrow id="alg1.1.1.m1.1.1" xref="alg1.1.1.m1.1.1.cmml"><mi id="alg1.1.1.m1.1.1.2" xref="alg1.1.1.m1.1.1.2.cmml">𝑻</mi><mo id="alg1.1.1.m1.1.1.1" xref="alg1.1.1.m1.1.1.1.cmml">∈</mo><msup id="alg1.1.1.m1.1.1.3" xref="alg1.1.1.m1.1.1.3.cmml"><mi id="alg1.1.1.m1.1.1.3.2" xref="alg1.1.1.m1.1.1.3.2.cmml">ℝ</mi><mrow id="alg1.1.1.m1.1.1.3.3" xref="alg1.1.1.m1.1.1.3.3.cmml"><msub id="alg1.1.1.m1.1.1.3.3.2" xref="alg1.1.1.m1.1.1.3.3.2.cmml"><mi id="alg1.1.1.m1.1.1.3.3.2.2" xref="alg1.1.1.m1.1.1.3.3.2.2.cmml">t</mi><mtext id="alg1.1.1.m1.1.1.3.3.2.3" xref="alg1.1.1.m1.1.1.3.3.2.3a.cmml">input</mtext></msub><mo lspace="0.222em" rspace="0.222em" id="alg1.1.1.m1.1.1.3.3.1" xref="alg1.1.1.m1.1.1.3.3.1.cmml">×</mo><mi id="alg1.1.1.m1.1.1.3.3.3" xref="alg1.1.1.m1.1.1.3.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="alg1.1.1.m1.1b"><apply id="alg1.1.1.m1.1.1.cmml" xref="alg1.1.1.m1.1.1"><in id="alg1.1.1.m1.1.1.1.cmml" xref="alg1.1.1.m1.1.1.1"></in><ci id="alg1.1.1.m1.1.1.2.cmml" xref="alg1.1.1.m1.1.1.2">𝑻</ci><apply id="alg1.1.1.m1.1.1.3.cmml" xref="alg1.1.1.m1.1.1.3"><csymbol cd="ambiguous" id="alg1.1.1.m1.1.1.3.1.cmml" xref="alg1.1.1.m1.1.1.3">superscript</csymbol><ci id="alg1.1.1.m1.1.1.3.2.cmml" xref="alg1.1.1.m1.1.1.3.2">ℝ</ci><apply id="alg1.1.1.m1.1.1.3.3.cmml" xref="alg1.1.1.m1.1.1.3.3"><times id="alg1.1.1.m1.1.1.3.3.1.cmml" xref="alg1.1.1.m1.1.1.3.3.1"></times><apply id="alg1.1.1.m1.1.1.3.3.2.cmml" xref="alg1.1.1.m1.1.1.3.3.2"><csymbol cd="ambiguous" id="alg1.1.1.m1.1.1.3.3.2.1.cmml" xref="alg1.1.1.m1.1.1.3.3.2">subscript</csymbol><ci id="alg1.1.1.m1.1.1.3.3.2.2.cmml" xref="alg1.1.1.m1.1.1.3.3.2.2">𝑡</ci><ci id="alg1.1.1.m1.1.1.3.3.2.3a.cmml" xref="alg1.1.1.m1.1.1.3.3.2.3"><mtext mathsize="50%" id="alg1.1.1.m1.1.1.3.3.2.3.cmml" xref="alg1.1.1.m1.1.1.3.3.2.3">input</mtext></ci></apply><ci id="alg1.1.1.m1.1.1.3.3.3.cmml" xref="alg1.1.1.m1.1.1.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.1.1.m1.1c">{\bm{T}}\in\mathbb{R}^{t_{\text{input}}\times d}</annotation></semantics></math>, number of layers <math id="alg1.2.2.m2.1" class="ltx_Math" alttext="L" display="inline"><semantics id="alg1.2.2.m2.1a"><mi id="alg1.2.2.m2.1.1" xref="alg1.2.2.m2.1.1.cmml">L</mi><annotation-xml encoding="MathML-Content" id="alg1.2.2.m2.1b"><ci id="alg1.2.2.m2.1.1.cmml" xref="alg1.2.2.m2.1.1">𝐿</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.2.2.m2.1c">L</annotation></semantics></math>, merging beginning layer <math id="alg1.3.3.m3.1" class="ltx_Math" alttext="S" display="inline"><semantics id="alg1.3.3.m3.1a"><mi id="alg1.3.3.m3.1.1" xref="alg1.3.3.m3.1.1.cmml">S</mi><annotation-xml encoding="MathML-Content" id="alg1.3.3.m3.1b"><ci id="alg1.3.3.m3.1.1.cmml" xref="alg1.3.3.m3.1.1">𝑆</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.3.3.m3.1c">S</annotation></semantics></math>
</div>
<div id="alg1.4.4" class="ltx_listingline">  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<span id="alg1.4.4.1" class="ltx_text"><span id="alg1.4.4.1.1" class="ltx_text ltx_font_bold">Output:</span> </span>Output Tokens: <math id="alg1.4.4.m1.1" class="ltx_Math" alttext="{\bm{O}}\in\mathbb{R}^{t_{\text{output}}\times d}" display="inline"><semantics id="alg1.4.4.m1.1a"><mrow id="alg1.4.4.m1.1.1" xref="alg1.4.4.m1.1.1.cmml"><mi id="alg1.4.4.m1.1.1.2" xref="alg1.4.4.m1.1.1.2.cmml">𝑶</mi><mo id="alg1.4.4.m1.1.1.1" xref="alg1.4.4.m1.1.1.1.cmml">∈</mo><msup id="alg1.4.4.m1.1.1.3" xref="alg1.4.4.m1.1.1.3.cmml"><mi id="alg1.4.4.m1.1.1.3.2" xref="alg1.4.4.m1.1.1.3.2.cmml">ℝ</mi><mrow id="alg1.4.4.m1.1.1.3.3" xref="alg1.4.4.m1.1.1.3.3.cmml"><msub id="alg1.4.4.m1.1.1.3.3.2" xref="alg1.4.4.m1.1.1.3.3.2.cmml"><mi id="alg1.4.4.m1.1.1.3.3.2.2" xref="alg1.4.4.m1.1.1.3.3.2.2.cmml">t</mi><mtext id="alg1.4.4.m1.1.1.3.3.2.3" xref="alg1.4.4.m1.1.1.3.3.2.3a.cmml">output</mtext></msub><mo lspace="0.222em" rspace="0.222em" id="alg1.4.4.m1.1.1.3.3.1" xref="alg1.4.4.m1.1.1.3.3.1.cmml">×</mo><mi id="alg1.4.4.m1.1.1.3.3.3" xref="alg1.4.4.m1.1.1.3.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="alg1.4.4.m1.1b"><apply id="alg1.4.4.m1.1.1.cmml" xref="alg1.4.4.m1.1.1"><in id="alg1.4.4.m1.1.1.1.cmml" xref="alg1.4.4.m1.1.1.1"></in><ci id="alg1.4.4.m1.1.1.2.cmml" xref="alg1.4.4.m1.1.1.2">𝑶</ci><apply id="alg1.4.4.m1.1.1.3.cmml" xref="alg1.4.4.m1.1.1.3"><csymbol cd="ambiguous" id="alg1.4.4.m1.1.1.3.1.cmml" xref="alg1.4.4.m1.1.1.3">superscript</csymbol><ci id="alg1.4.4.m1.1.1.3.2.cmml" xref="alg1.4.4.m1.1.1.3.2">ℝ</ci><apply id="alg1.4.4.m1.1.1.3.3.cmml" xref="alg1.4.4.m1.1.1.3.3"><times id="alg1.4.4.m1.1.1.3.3.1.cmml" xref="alg1.4.4.m1.1.1.3.3.1"></times><apply id="alg1.4.4.m1.1.1.3.3.2.cmml" xref="alg1.4.4.m1.1.1.3.3.2"><csymbol cd="ambiguous" id="alg1.4.4.m1.1.1.3.3.2.1.cmml" xref="alg1.4.4.m1.1.1.3.3.2">subscript</csymbol><ci id="alg1.4.4.m1.1.1.3.3.2.2.cmml" xref="alg1.4.4.m1.1.1.3.3.2.2">𝑡</ci><ci id="alg1.4.4.m1.1.1.3.3.2.3a.cmml" xref="alg1.4.4.m1.1.1.3.3.2.3"><mtext mathsize="50%" id="alg1.4.4.m1.1.1.3.3.2.3.cmml" xref="alg1.4.4.m1.1.1.3.3.2.3">output</mtext></ci></apply><ci id="alg1.4.4.m1.1.1.3.3.3.cmml" xref="alg1.4.4.m1.1.1.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.4.4.m1.1c">{\bm{O}}\in\mathbb{R}^{t_{\text{output}}\times d}</annotation></semantics></math>
</div>
<div id="alg1.32.35" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">3</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   

</div>
<div id="alg1.6.6" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">4</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   <span id="alg1.6.6.3" class="ltx_text ltx_font_bold">for</span> <em id="alg1.6.6.2" class="ltx_emph ltx_font_italic"><math id="alg1.5.5.1.m1.1" class="ltx_Math" alttext="l\leftarrow 0" display="inline"><semantics id="alg1.5.5.1.m1.1a"><mrow id="alg1.5.5.1.m1.1.1" xref="alg1.5.5.1.m1.1.1.cmml"><mi id="alg1.5.5.1.m1.1.1.2" xref="alg1.5.5.1.m1.1.1.2.cmml">l</mi><mo stretchy="false" id="alg1.5.5.1.m1.1.1.1" xref="alg1.5.5.1.m1.1.1.1.cmml">←</mo><mn id="alg1.5.5.1.m1.1.1.3" xref="alg1.5.5.1.m1.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="alg1.5.5.1.m1.1b"><apply id="alg1.5.5.1.m1.1.1.cmml" xref="alg1.5.5.1.m1.1.1"><ci id="alg1.5.5.1.m1.1.1.1.cmml" xref="alg1.5.5.1.m1.1.1.1">←</ci><ci id="alg1.5.5.1.m1.1.1.2.cmml" xref="alg1.5.5.1.m1.1.1.2">𝑙</ci><cn type="integer" id="alg1.5.5.1.m1.1.1.3.cmml" xref="alg1.5.5.1.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.5.5.1.m1.1c">l\leftarrow 0</annotation></semantics></math> <span id="alg1.6.6.2.1" class="ltx_text ltx_font_bold ltx_font_upright">to</span> <math id="alg1.6.6.2.m2.1" class="ltx_Math" alttext="S-1" display="inline"><semantics id="alg1.6.6.2.m2.1a"><mrow id="alg1.6.6.2.m2.1.1" xref="alg1.6.6.2.m2.1.1.cmml"><mi id="alg1.6.6.2.m2.1.1.2" xref="alg1.6.6.2.m2.1.1.2.cmml">S</mi><mo id="alg1.6.6.2.m2.1.1.1" xref="alg1.6.6.2.m2.1.1.1.cmml">−</mo><mn id="alg1.6.6.2.m2.1.1.3" xref="alg1.6.6.2.m2.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="alg1.6.6.2.m2.1b"><apply id="alg1.6.6.2.m2.1.1.cmml" xref="alg1.6.6.2.m2.1.1"><minus id="alg1.6.6.2.m2.1.1.1.cmml" xref="alg1.6.6.2.m2.1.1.1"></minus><ci id="alg1.6.6.2.m2.1.1.2.cmml" xref="alg1.6.6.2.m2.1.1.2">𝑆</ci><cn type="integer" id="alg1.6.6.2.m2.1.1.3.cmml" xref="alg1.6.6.2.m2.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.6.6.2.m2.1c">S-1</annotation></semantics></math></em> <span id="alg1.6.6.4" class="ltx_text ltx_font_bold">do</span>
</div>
<div id="alg1.32.36" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">5</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<span id="alg1.32.36.1" class="ltx_text ltx_font_bold">procedure</span> <em id="alg1.32.36.2" class="ltx_emph ltx_font_typewriter">Standard Prefill</em><span id="alg1.32.36.3" class="ltx_text ltx_font_bold">:</span>
</div>
<div id="alg1.32.37" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">6</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
Standard Attention &amp; Standard Cache

</div>
<div id="alg1.32.38" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">7</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   <span id="alg1.32.38.1" class="ltx_text ltx_font_bold">procedure</span> <em id="alg1.32.38.2" class="ltx_emph ltx_font_typewriter">Standard Decoding</em><span id="alg1.32.38.3" class="ltx_text ltx_font_bold">:</span>
</div>
<div id="alg1.32.39" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">8</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
Standard Attention &amp; Standard Cache

</div>
<div id="alg1.32.40" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">9</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
</div>
<div id="alg1.32.41" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">10</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
</div>
<div id="alg1.7.7" class="ltx_listingline">  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   <span id="alg1.7.7.2" class="ltx_text ltx_font_typewriter">// </span><span id="alg1.7.7.1" class="ltx_text ltx_font_typewriter">Start Merging from layer <math id="alg1.7.7.1.m1.1" class="ltx_Math" alttext="S" display="inline"><semantics id="alg1.7.7.1.m1.1a"><mi id="alg1.7.7.1.m1.1.1" xref="alg1.7.7.1.m1.1.1.cmml">S</mi><annotation-xml encoding="MathML-Content" id="alg1.7.7.1.m1.1b"><ci id="alg1.7.7.1.m1.1.1.cmml" xref="alg1.7.7.1.m1.1.1">𝑆</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.7.7.1.m1.1c">S</annotation></semantics></math> </span>
</div>
<div id="alg1.9.9" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">11</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<span id="alg1.9.9.3" class="ltx_text ltx_font_bold">for</span> <em id="alg1.9.9.2" class="ltx_emph ltx_font_italic"><math id="alg1.8.8.1.m1.1" class="ltx_Math" alttext="l\leftarrow S" display="inline"><semantics id="alg1.8.8.1.m1.1a"><mrow id="alg1.8.8.1.m1.1.1" xref="alg1.8.8.1.m1.1.1.cmml"><mi id="alg1.8.8.1.m1.1.1.2" xref="alg1.8.8.1.m1.1.1.2.cmml">l</mi><mo stretchy="false" id="alg1.8.8.1.m1.1.1.1" xref="alg1.8.8.1.m1.1.1.1.cmml">←</mo><mi id="alg1.8.8.1.m1.1.1.3" xref="alg1.8.8.1.m1.1.1.3.cmml">S</mi></mrow><annotation-xml encoding="MathML-Content" id="alg1.8.8.1.m1.1b"><apply id="alg1.8.8.1.m1.1.1.cmml" xref="alg1.8.8.1.m1.1.1"><ci id="alg1.8.8.1.m1.1.1.1.cmml" xref="alg1.8.8.1.m1.1.1.1">←</ci><ci id="alg1.8.8.1.m1.1.1.2.cmml" xref="alg1.8.8.1.m1.1.1.2">𝑙</ci><ci id="alg1.8.8.1.m1.1.1.3.cmml" xref="alg1.8.8.1.m1.1.1.3">𝑆</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.8.8.1.m1.1c">l\leftarrow S</annotation></semantics></math> <span id="alg1.9.9.2.1" class="ltx_text ltx_font_bold ltx_font_upright">to</span> <math id="alg1.9.9.2.m2.1" class="ltx_Math" alttext="L" display="inline"><semantics id="alg1.9.9.2.m2.1a"><mi id="alg1.9.9.2.m2.1.1" xref="alg1.9.9.2.m2.1.1.cmml">L</mi><annotation-xml encoding="MathML-Content" id="alg1.9.9.2.m2.1b"><ci id="alg1.9.9.2.m2.1.1.cmml" xref="alg1.9.9.2.m2.1.1">𝐿</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.9.9.2.m2.1c">L</annotation></semantics></math></em> <span id="alg1.9.9.4" class="ltx_text ltx_font_bold">do</span>
</div>
<div id="alg1.10.10" class="ltx_listingline">  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<span id="alg1.10.10.2" class="ltx_text ltx_font_typewriter">// </span><span id="alg1.10.10.1" class="ltx_text ltx_font_typewriter">Perform Merging in every two layers <math id="alg1.10.10.1.m1.1" class="ltx_math_unparsed" alttext="l\%2==1" display="inline"><semantics id="alg1.10.10.1.m1.1a"><mrow id="alg1.10.10.1.m1.1b"><mi id="alg1.10.10.1.m1.1.1">l</mi><mo id="alg1.10.10.1.m1.1.2">%</mo><mn id="alg1.10.10.1.m1.1.3">2</mn><mo rspace="0em" id="alg1.10.10.1.m1.1.4">=</mo><mo lspace="0em" id="alg1.10.10.1.m1.1.5">=</mo><mn id="alg1.10.10.1.m1.1.6">1</mn></mrow><annotation encoding="application/x-tex" id="alg1.10.10.1.m1.1c">l\%2==1</annotation></semantics></math> </span>
</div>
<div id="alg1.11.11" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">12</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<span id="alg1.11.11.2" class="ltx_text ltx_font_bold">if</span> <em id="alg1.11.11.1" class="ltx_emph ltx_font_italic"><math id="alg1.11.11.1.m1.1" class="ltx_math_unparsed" alttext="l\%2==1" display="inline"><semantics id="alg1.11.11.1.m1.1a"><mrow id="alg1.11.11.1.m1.1b"><mi id="alg1.11.11.1.m1.1.1">l</mi><mo id="alg1.11.11.1.m1.1.2">%</mo><mn id="alg1.11.11.1.m1.1.3">2</mn><mo rspace="0em" id="alg1.11.11.1.m1.1.4">=</mo><mo lspace="0em" id="alg1.11.11.1.m1.1.5">=</mo><mn id="alg1.11.11.1.m1.1.6">1</mn></mrow><annotation encoding="application/x-tex" id="alg1.11.11.1.m1.1c">l\%2==1</annotation></semantics></math> <span id="alg1.11.11.1.1" class="ltx_text">and prefilling</span></em> <span id="alg1.11.11.3" class="ltx_text ltx_font_bold">then</span>
</div>
<div id="alg1.32.42" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">13</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<span id="alg1.32.42.1" class="ltx_text ltx_font_bold">procedure</span> <em id="alg1.32.42.2" class="ltx_emph ltx_font_typewriter">MiniCache Prefill</em><span id="alg1.32.42.3" class="ltx_text ltx_font_bold">:</span>
</div>
<div id="alg1.16.16" class="ltx_listingline">  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<span id="alg1.16.16.1" class="ltx_text"><span id="alg1.16.16.1.1" class="ltx_text ltx_font_bold">Input:</span> </span>KV cache from Current layer <math id="alg1.12.12.m1.1" class="ltx_Math" alttext="l" display="inline"><semantics id="alg1.12.12.m1.1a"><mi id="alg1.12.12.m1.1.1" xref="alg1.12.12.m1.1.1.cmml">l</mi><annotation-xml encoding="MathML-Content" id="alg1.12.12.m1.1b"><ci id="alg1.12.12.m1.1.1.cmml" xref="alg1.12.12.m1.1.1">𝑙</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.12.12.m1.1c">l</annotation></semantics></math>: <math id="alg1.13.13.m2.2" class="ltx_Math" alttext="{\bm{X}}_{k,v}^{l}" display="inline"><semantics id="alg1.13.13.m2.2a"><msubsup id="alg1.13.13.m2.2.3" xref="alg1.13.13.m2.2.3.cmml"><mi id="alg1.13.13.m2.2.3.2.2" xref="alg1.13.13.m2.2.3.2.2.cmml">𝑿</mi><mrow id="alg1.13.13.m2.2.2.2.4" xref="alg1.13.13.m2.2.2.2.3.cmml"><mi id="alg1.13.13.m2.1.1.1.1" xref="alg1.13.13.m2.1.1.1.1.cmml">k</mi><mo id="alg1.13.13.m2.2.2.2.4.1" xref="alg1.13.13.m2.2.2.2.3.cmml">,</mo><mi id="alg1.13.13.m2.2.2.2.2" xref="alg1.13.13.m2.2.2.2.2.cmml">v</mi></mrow><mi id="alg1.13.13.m2.2.3.3" xref="alg1.13.13.m2.2.3.3.cmml">l</mi></msubsup><annotation-xml encoding="MathML-Content" id="alg1.13.13.m2.2b"><apply id="alg1.13.13.m2.2.3.cmml" xref="alg1.13.13.m2.2.3"><csymbol cd="ambiguous" id="alg1.13.13.m2.2.3.1.cmml" xref="alg1.13.13.m2.2.3">superscript</csymbol><apply id="alg1.13.13.m2.2.3.2.cmml" xref="alg1.13.13.m2.2.3"><csymbol cd="ambiguous" id="alg1.13.13.m2.2.3.2.1.cmml" xref="alg1.13.13.m2.2.3">subscript</csymbol><ci id="alg1.13.13.m2.2.3.2.2.cmml" xref="alg1.13.13.m2.2.3.2.2">𝑿</ci><list id="alg1.13.13.m2.2.2.2.3.cmml" xref="alg1.13.13.m2.2.2.2.4"><ci id="alg1.13.13.m2.1.1.1.1.cmml" xref="alg1.13.13.m2.1.1.1.1">𝑘</ci><ci id="alg1.13.13.m2.2.2.2.2.cmml" xref="alg1.13.13.m2.2.2.2.2">𝑣</ci></list></apply><ci id="alg1.13.13.m2.2.3.3.cmml" xref="alg1.13.13.m2.2.3.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.13.13.m2.2c">{\bm{X}}_{k,v}^{l}</annotation></semantics></math>, KV cache from Previous layer <math id="alg1.14.14.m3.1" class="ltx_Math" alttext="l-1" display="inline"><semantics id="alg1.14.14.m3.1a"><mrow id="alg1.14.14.m3.1.1" xref="alg1.14.14.m3.1.1.cmml"><mi id="alg1.14.14.m3.1.1.2" xref="alg1.14.14.m3.1.1.2.cmml">l</mi><mo id="alg1.14.14.m3.1.1.1" xref="alg1.14.14.m3.1.1.1.cmml">−</mo><mn id="alg1.14.14.m3.1.1.3" xref="alg1.14.14.m3.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="alg1.14.14.m3.1b"><apply id="alg1.14.14.m3.1.1.cmml" xref="alg1.14.14.m3.1.1"><minus id="alg1.14.14.m3.1.1.1.cmml" xref="alg1.14.14.m3.1.1.1"></minus><ci id="alg1.14.14.m3.1.1.2.cmml" xref="alg1.14.14.m3.1.1.2">𝑙</ci><cn type="integer" id="alg1.14.14.m3.1.1.3.cmml" xref="alg1.14.14.m3.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.14.14.m3.1c">l-1</annotation></semantics></math>: <math id="alg1.15.15.m4.2" class="ltx_Math" alttext="{\bm{X}}_{k,v}^{l-1}" display="inline"><semantics id="alg1.15.15.m4.2a"><msubsup id="alg1.15.15.m4.2.3" xref="alg1.15.15.m4.2.3.cmml"><mi id="alg1.15.15.m4.2.3.2.2" xref="alg1.15.15.m4.2.3.2.2.cmml">𝑿</mi><mrow id="alg1.15.15.m4.2.2.2.4" xref="alg1.15.15.m4.2.2.2.3.cmml"><mi id="alg1.15.15.m4.1.1.1.1" xref="alg1.15.15.m4.1.1.1.1.cmml">k</mi><mo id="alg1.15.15.m4.2.2.2.4.1" xref="alg1.15.15.m4.2.2.2.3.cmml">,</mo><mi id="alg1.15.15.m4.2.2.2.2" xref="alg1.15.15.m4.2.2.2.2.cmml">v</mi></mrow><mrow id="alg1.15.15.m4.2.3.3" xref="alg1.15.15.m4.2.3.3.cmml"><mi id="alg1.15.15.m4.2.3.3.2" xref="alg1.15.15.m4.2.3.3.2.cmml">l</mi><mo id="alg1.15.15.m4.2.3.3.1" xref="alg1.15.15.m4.2.3.3.1.cmml">−</mo><mn id="alg1.15.15.m4.2.3.3.3" xref="alg1.15.15.m4.2.3.3.3.cmml">1</mn></mrow></msubsup><annotation-xml encoding="MathML-Content" id="alg1.15.15.m4.2b"><apply id="alg1.15.15.m4.2.3.cmml" xref="alg1.15.15.m4.2.3"><csymbol cd="ambiguous" id="alg1.15.15.m4.2.3.1.cmml" xref="alg1.15.15.m4.2.3">superscript</csymbol><apply id="alg1.15.15.m4.2.3.2.cmml" xref="alg1.15.15.m4.2.3"><csymbol cd="ambiguous" id="alg1.15.15.m4.2.3.2.1.cmml" xref="alg1.15.15.m4.2.3">subscript</csymbol><ci id="alg1.15.15.m4.2.3.2.2.cmml" xref="alg1.15.15.m4.2.3.2.2">𝑿</ci><list id="alg1.15.15.m4.2.2.2.3.cmml" xref="alg1.15.15.m4.2.2.2.4"><ci id="alg1.15.15.m4.1.1.1.1.cmml" xref="alg1.15.15.m4.1.1.1.1">𝑘</ci><ci id="alg1.15.15.m4.2.2.2.2.cmml" xref="alg1.15.15.m4.2.2.2.2">𝑣</ci></list></apply><apply id="alg1.15.15.m4.2.3.3.cmml" xref="alg1.15.15.m4.2.3.3"><minus id="alg1.15.15.m4.2.3.3.1.cmml" xref="alg1.15.15.m4.2.3.3.1"></minus><ci id="alg1.15.15.m4.2.3.3.2.cmml" xref="alg1.15.15.m4.2.3.3.2">𝑙</ci><cn type="integer" id="alg1.15.15.m4.2.3.3.3.cmml" xref="alg1.15.15.m4.2.3.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.15.15.m4.2c">{\bm{X}}_{k,v}^{l-1}</annotation></semantics></math>, token retention threshold: <math id="alg1.16.16.m5.1" class="ltx_Math" alttext="\gamma" display="inline"><semantics id="alg1.16.16.m5.1a"><mi id="alg1.16.16.m5.1.1" xref="alg1.16.16.m5.1.1.cmml">γ</mi><annotation-xml encoding="MathML-Content" id="alg1.16.16.m5.1b"><ci id="alg1.16.16.m5.1.1.cmml" xref="alg1.16.16.m5.1.1">𝛾</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.16.16.m5.1c">\gamma</annotation></semantics></math>
</div>
<div id="alg1.32.43" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">14</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   

</div>
<div id="alg1.18.18" class="ltx_listingline">  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   <span id="alg1.18.18.2" class="ltx_text ltx_font_bold">Delete</span> KV cache of the <math id="alg1.17.17.m1.1" class="ltx_Math" alttext="l-1" display="inline"><semantics id="alg1.17.17.m1.1a"><mrow id="alg1.17.17.m1.1.1" xref="alg1.17.17.m1.1.1.cmml"><mi id="alg1.17.17.m1.1.1.2" xref="alg1.17.17.m1.1.1.2.cmml">l</mi><mo id="alg1.17.17.m1.1.1.1" xref="alg1.17.17.m1.1.1.1.cmml">−</mo><mn id="alg1.17.17.m1.1.1.3" xref="alg1.17.17.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="alg1.17.17.m1.1b"><apply id="alg1.17.17.m1.1.1.cmml" xref="alg1.17.17.m1.1.1"><minus id="alg1.17.17.m1.1.1.1.cmml" xref="alg1.17.17.m1.1.1.1"></minus><ci id="alg1.17.17.m1.1.1.2.cmml" xref="alg1.17.17.m1.1.1.2">𝑙</ci><cn type="integer" id="alg1.17.17.m1.1.1.3.cmml" xref="alg1.17.17.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.17.17.m1.1c">l-1</annotation></semantics></math>-th layer <span id="alg1.18.18.3" class="ltx_text ltx_font_typewriter">// </span><span id="alg1.18.18.1" class="ltx_text ltx_font_typewriter">layer <math id="alg1.18.18.1.m1.2" class="ltx_Math" alttext="l,l-1" display="inline"><semantics id="alg1.18.18.1.m1.2a"><mrow id="alg1.18.18.1.m1.2.2.1" xref="alg1.18.18.1.m1.2.2.2.cmml"><mi id="alg1.18.18.1.m1.1.1" xref="alg1.18.18.1.m1.1.1.cmml">l</mi><mo id="alg1.18.18.1.m1.2.2.1.2" xref="alg1.18.18.1.m1.2.2.2.cmml">,</mo><mrow id="alg1.18.18.1.m1.2.2.1.1" xref="alg1.18.18.1.m1.2.2.1.1.cmml"><mi id="alg1.18.18.1.m1.2.2.1.1.2" xref="alg1.18.18.1.m1.2.2.1.1.2.cmml">l</mi><mo id="alg1.18.18.1.m1.2.2.1.1.1" xref="alg1.18.18.1.m1.2.2.1.1.1.cmml">−</mo><mn id="alg1.18.18.1.m1.2.2.1.1.3" xref="alg1.18.18.1.m1.2.2.1.1.3.cmml">1</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.18.18.1.m1.2b"><list id="alg1.18.18.1.m1.2.2.2.cmml" xref="alg1.18.18.1.m1.2.2.1"><ci id="alg1.18.18.1.m1.1.1.cmml" xref="alg1.18.18.1.m1.1.1">𝑙</ci><apply id="alg1.18.18.1.m1.2.2.1.1.cmml" xref="alg1.18.18.1.m1.2.2.1.1"><minus id="alg1.18.18.1.m1.2.2.1.1.1.cmml" xref="alg1.18.18.1.m1.2.2.1.1.1"></minus><ci id="alg1.18.18.1.m1.2.2.1.1.2.cmml" xref="alg1.18.18.1.m1.2.2.1.1.2">𝑙</ci><cn type="integer" id="alg1.18.18.1.m1.2.2.1.1.3.cmml" xref="alg1.18.18.1.m1.2.2.1.1.3">1</cn></apply></list></annotation-xml><annotation encoding="application/x-tex" id="alg1.18.18.1.m1.2c">l,l-1</annotation></semantics></math> shares one Cache </span>
</div>
<div id="alg1.32.44" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">15</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   

</div>
<div id="alg1.32.45" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">16</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   Standard Attention &amp; Standard Cache

</div>
<div id="alg1.32.46" class="ltx_listingline">  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   <span id="alg1.32.46.1" class="ltx_text ltx_font_typewriter">// </span><span id="alg1.32.46.2" class="ltx_text ltx_font_typewriter">Perform Merging in the second layer </span>
</div>
<div id="alg1.32.47" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">17</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<span id="alg1.32.47.1" class="ltx_text ltx_font_bold">if</span> <em id="alg1.32.47.2" class="ltx_emph ltx_font_italic">and decoding</em> <span id="alg1.32.47.3" class="ltx_text ltx_font_bold">then</span>
</div>
<div id="alg1.19.19" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">18</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<span id="alg1.19.19.2" class="ltx_text ltx_font_bold">if</span> <em id="alg1.19.19.1" class="ltx_emph ltx_font_italic"><math id="alg1.19.19.1.m1.1" class="ltx_math_unparsed" alttext="l\%2==0" display="inline"><semantics id="alg1.19.19.1.m1.1a"><mrow id="alg1.19.19.1.m1.1b"><mi id="alg1.19.19.1.m1.1.1">l</mi><mo id="alg1.19.19.1.m1.1.2">%</mo><mn id="alg1.19.19.1.m1.1.3">2</mn><mo rspace="0em" id="alg1.19.19.1.m1.1.4">=</mo><mo lspace="0em" id="alg1.19.19.1.m1.1.5">=</mo><mn id="alg1.19.19.1.m1.1.6">0</mn></mrow><annotation encoding="application/x-tex" id="alg1.19.19.1.m1.1c">l\%2==0</annotation></semantics></math> </em> <span id="alg1.19.19.3" class="ltx_text ltx_font_bold">then</span>
</div>
<div id="alg1.20.20" class="ltx_listingline">  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<span id="alg1.20.20.2" class="ltx_text ltx_font_typewriter">// </span><span id="alg1.20.20.1" class="ltx_text ltx_font_typewriter">first round in the cross-layer merging, fetch shared KV cache states from <math id="alg1.20.20.1.m1.2" class="ltx_Math" alttext="{\bm{C}}_{k,v}^{l+1}" display="inline"><semantics id="alg1.20.20.1.m1.2a"><msubsup id="alg1.20.20.1.m1.2.3" xref="alg1.20.20.1.m1.2.3.cmml"><mi id="alg1.20.20.1.m1.2.3.2.2" xref="alg1.20.20.1.m1.2.3.2.2.cmml">𝑪</mi><mrow id="alg1.20.20.1.m1.2.2.2.4" xref="alg1.20.20.1.m1.2.2.2.3.cmml"><mi id="alg1.20.20.1.m1.1.1.1.1" xref="alg1.20.20.1.m1.1.1.1.1.cmml">k</mi><mo id="alg1.20.20.1.m1.2.2.2.4.1" xref="alg1.20.20.1.m1.2.2.2.3.cmml">,</mo><mi id="alg1.20.20.1.m1.2.2.2.2" xref="alg1.20.20.1.m1.2.2.2.2.cmml">v</mi></mrow><mrow id="alg1.20.20.1.m1.2.3.3" xref="alg1.20.20.1.m1.2.3.3.cmml"><mi id="alg1.20.20.1.m1.2.3.3.2" xref="alg1.20.20.1.m1.2.3.3.2.cmml">l</mi><mo id="alg1.20.20.1.m1.2.3.3.1" xref="alg1.20.20.1.m1.2.3.3.1.cmml">+</mo><mn id="alg1.20.20.1.m1.2.3.3.3" xref="alg1.20.20.1.m1.2.3.3.3.cmml">1</mn></mrow></msubsup><annotation-xml encoding="MathML-Content" id="alg1.20.20.1.m1.2b"><apply id="alg1.20.20.1.m1.2.3.cmml" xref="alg1.20.20.1.m1.2.3"><csymbol cd="ambiguous" id="alg1.20.20.1.m1.2.3.1.cmml" xref="alg1.20.20.1.m1.2.3">superscript</csymbol><apply id="alg1.20.20.1.m1.2.3.2.cmml" xref="alg1.20.20.1.m1.2.3"><csymbol cd="ambiguous" id="alg1.20.20.1.m1.2.3.2.1.cmml" xref="alg1.20.20.1.m1.2.3">subscript</csymbol><ci id="alg1.20.20.1.m1.2.3.2.2.cmml" xref="alg1.20.20.1.m1.2.3.2.2">𝑪</ci><list id="alg1.20.20.1.m1.2.2.2.3.cmml" xref="alg1.20.20.1.m1.2.2.2.4"><ci id="alg1.20.20.1.m1.1.1.1.1.cmml" xref="alg1.20.20.1.m1.1.1.1.1">𝑘</ci><ci id="alg1.20.20.1.m1.2.2.2.2.cmml" xref="alg1.20.20.1.m1.2.2.2.2">𝑣</ci></list></apply><apply id="alg1.20.20.1.m1.2.3.3.cmml" xref="alg1.20.20.1.m1.2.3.3"><plus id="alg1.20.20.1.m1.2.3.3.1.cmml" xref="alg1.20.20.1.m1.2.3.3.1"></plus><ci id="alg1.20.20.1.m1.2.3.3.2.cmml" xref="alg1.20.20.1.m1.2.3.3.2">𝑙</ci><cn type="integer" id="alg1.20.20.1.m1.2.3.3.3.cmml" xref="alg1.20.20.1.m1.2.3.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.20.20.1.m1.2c">{\bm{C}}_{k,v}^{l+1}</annotation></semantics></math> </span>
</div>
<div id="alg1.32.48" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">19</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<span id="alg1.32.48.1" class="ltx_text ltx_font_bold">procedure</span> <em id="alg1.32.48.2" class="ltx_emph ltx_font_typewriter">MiniCache Decoding</em><span id="alg1.32.48.3" class="ltx_text ltx_font_bold">:</span>
</div>
<div id="alg1.25.25" class="ltx_listingline">  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<span id="alg1.25.25.1" class="ltx_text"><span id="alg1.25.25.1.1" class="ltx_text ltx_font_bold">Input:</span> </span>KV cache: <math id="alg1.21.21.m1.2" class="ltx_Math" alttext="{\bm{C}}_{k,v}^{l+1}" display="inline"><semantics id="alg1.21.21.m1.2a"><msubsup id="alg1.21.21.m1.2.3" xref="alg1.21.21.m1.2.3.cmml"><mi id="alg1.21.21.m1.2.3.2.2" xref="alg1.21.21.m1.2.3.2.2.cmml">𝑪</mi><mrow id="alg1.21.21.m1.2.2.2.4" xref="alg1.21.21.m1.2.2.2.3.cmml"><mi id="alg1.21.21.m1.1.1.1.1" xref="alg1.21.21.m1.1.1.1.1.cmml">k</mi><mo id="alg1.21.21.m1.2.2.2.4.1" xref="alg1.21.21.m1.2.2.2.3.cmml">,</mo><mi id="alg1.21.21.m1.2.2.2.2" xref="alg1.21.21.m1.2.2.2.2.cmml">v</mi></mrow><mrow id="alg1.21.21.m1.2.3.3" xref="alg1.21.21.m1.2.3.3.cmml"><mi id="alg1.21.21.m1.2.3.3.2" xref="alg1.21.21.m1.2.3.3.2.cmml">l</mi><mo id="alg1.21.21.m1.2.3.3.1" xref="alg1.21.21.m1.2.3.3.1.cmml">+</mo><mn id="alg1.21.21.m1.2.3.3.3" xref="alg1.21.21.m1.2.3.3.3.cmml">1</mn></mrow></msubsup><annotation-xml encoding="MathML-Content" id="alg1.21.21.m1.2b"><apply id="alg1.21.21.m1.2.3.cmml" xref="alg1.21.21.m1.2.3"><csymbol cd="ambiguous" id="alg1.21.21.m1.2.3.1.cmml" xref="alg1.21.21.m1.2.3">superscript</csymbol><apply id="alg1.21.21.m1.2.3.2.cmml" xref="alg1.21.21.m1.2.3"><csymbol cd="ambiguous" id="alg1.21.21.m1.2.3.2.1.cmml" xref="alg1.21.21.m1.2.3">subscript</csymbol><ci id="alg1.21.21.m1.2.3.2.2.cmml" xref="alg1.21.21.m1.2.3.2.2">𝑪</ci><list id="alg1.21.21.m1.2.2.2.3.cmml" xref="alg1.21.21.m1.2.2.2.4"><ci id="alg1.21.21.m1.1.1.1.1.cmml" xref="alg1.21.21.m1.1.1.1.1">𝑘</ci><ci id="alg1.21.21.m1.2.2.2.2.cmml" xref="alg1.21.21.m1.2.2.2.2">𝑣</ci></list></apply><apply id="alg1.21.21.m1.2.3.3.cmml" xref="alg1.21.21.m1.2.3.3"><plus id="alg1.21.21.m1.2.3.3.1.cmml" xref="alg1.21.21.m1.2.3.3.1"></plus><ci id="alg1.21.21.m1.2.3.3.2.cmml" xref="alg1.21.21.m1.2.3.3.2">𝑙</ci><cn type="integer" id="alg1.21.21.m1.2.3.3.3.cmml" xref="alg1.21.21.m1.2.3.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.21.21.m1.2c">{\bm{C}}_{k,v}^{l+1}</annotation></semantics></math>, Norm: <math id="alg1.22.22.m2.3" class="ltx_Math" alttext="\lVert{\bm{X}}_{k,v}^{l}\rVert" display="inline"><semantics id="alg1.22.22.m2.3a"><mrow id="alg1.22.22.m2.3.3.1" xref="alg1.22.22.m2.3.3.2.cmml"><mo fence="true" rspace="0em" id="alg1.22.22.m2.3.3.1.2" xref="alg1.22.22.m2.3.3.2.1.cmml">∥</mo><msubsup id="alg1.22.22.m2.3.3.1.1" xref="alg1.22.22.m2.3.3.1.1.cmml"><mi id="alg1.22.22.m2.3.3.1.1.2.2" xref="alg1.22.22.m2.3.3.1.1.2.2.cmml">𝑿</mi><mrow id="alg1.22.22.m2.2.2.2.4" xref="alg1.22.22.m2.2.2.2.3.cmml"><mi id="alg1.22.22.m2.1.1.1.1" xref="alg1.22.22.m2.1.1.1.1.cmml">k</mi><mo id="alg1.22.22.m2.2.2.2.4.1" xref="alg1.22.22.m2.2.2.2.3.cmml">,</mo><mi id="alg1.22.22.m2.2.2.2.2" xref="alg1.22.22.m2.2.2.2.2.cmml">v</mi></mrow><mi id="alg1.22.22.m2.3.3.1.1.3" xref="alg1.22.22.m2.3.3.1.1.3.cmml">l</mi></msubsup><mo fence="true" lspace="0em" id="alg1.22.22.m2.3.3.1.3" xref="alg1.22.22.m2.3.3.2.1.cmml">∥</mo></mrow><annotation-xml encoding="MathML-Content" id="alg1.22.22.m2.3b"><apply id="alg1.22.22.m2.3.3.2.cmml" xref="alg1.22.22.m2.3.3.1"><csymbol cd="latexml" id="alg1.22.22.m2.3.3.2.1.cmml" xref="alg1.22.22.m2.3.3.1.2">delimited-∥∥</csymbol><apply id="alg1.22.22.m2.3.3.1.1.cmml" xref="alg1.22.22.m2.3.3.1.1"><csymbol cd="ambiguous" id="alg1.22.22.m2.3.3.1.1.1.cmml" xref="alg1.22.22.m2.3.3.1.1">superscript</csymbol><apply id="alg1.22.22.m2.3.3.1.1.2.cmml" xref="alg1.22.22.m2.3.3.1.1"><csymbol cd="ambiguous" id="alg1.22.22.m2.3.3.1.1.2.1.cmml" xref="alg1.22.22.m2.3.3.1.1">subscript</csymbol><ci id="alg1.22.22.m2.3.3.1.1.2.2.cmml" xref="alg1.22.22.m2.3.3.1.1.2.2">𝑿</ci><list id="alg1.22.22.m2.2.2.2.3.cmml" xref="alg1.22.22.m2.2.2.2.4"><ci id="alg1.22.22.m2.1.1.1.1.cmml" xref="alg1.22.22.m2.1.1.1.1">𝑘</ci><ci id="alg1.22.22.m2.2.2.2.2.cmml" xref="alg1.22.22.m2.2.2.2.2">𝑣</ci></list></apply><ci id="alg1.22.22.m2.3.3.1.1.3.cmml" xref="alg1.22.22.m2.3.3.1.1.3">𝑙</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.22.22.m2.3c">\lVert{\bm{X}}_{k,v}^{l}\rVert</annotation></semantics></math>, Unmerged Tokens: <math id="alg1.23.23.m3.2" class="ltx_Math" alttext="{\bm{R}}_{k,v}^{l}" display="inline"><semantics id="alg1.23.23.m3.2a"><msubsup id="alg1.23.23.m3.2.3" xref="alg1.23.23.m3.2.3.cmml"><mi id="alg1.23.23.m3.2.3.2.2" xref="alg1.23.23.m3.2.3.2.2.cmml">𝑹</mi><mrow id="alg1.23.23.m3.2.2.2.4" xref="alg1.23.23.m3.2.2.2.3.cmml"><mi id="alg1.23.23.m3.1.1.1.1" xref="alg1.23.23.m3.1.1.1.1.cmml">k</mi><mo id="alg1.23.23.m3.2.2.2.4.1" xref="alg1.23.23.m3.2.2.2.3.cmml">,</mo><mi id="alg1.23.23.m3.2.2.2.2" xref="alg1.23.23.m3.2.2.2.2.cmml">v</mi></mrow><mi id="alg1.23.23.m3.2.3.3" xref="alg1.23.23.m3.2.3.3.cmml">l</mi></msubsup><annotation-xml encoding="MathML-Content" id="alg1.23.23.m3.2b"><apply id="alg1.23.23.m3.2.3.cmml" xref="alg1.23.23.m3.2.3"><csymbol cd="ambiguous" id="alg1.23.23.m3.2.3.1.cmml" xref="alg1.23.23.m3.2.3">superscript</csymbol><apply id="alg1.23.23.m3.2.3.2.cmml" xref="alg1.23.23.m3.2.3"><csymbol cd="ambiguous" id="alg1.23.23.m3.2.3.2.1.cmml" xref="alg1.23.23.m3.2.3">subscript</csymbol><ci id="alg1.23.23.m3.2.3.2.2.cmml" xref="alg1.23.23.m3.2.3.2.2">𝑹</ci><list id="alg1.23.23.m3.2.2.2.3.cmml" xref="alg1.23.23.m3.2.2.2.4"><ci id="alg1.23.23.m3.1.1.1.1.cmml" xref="alg1.23.23.m3.1.1.1.1">𝑘</ci><ci id="alg1.23.23.m3.2.2.2.2.cmml" xref="alg1.23.23.m3.2.2.2.2">𝑣</ci></list></apply><ci id="alg1.23.23.m3.2.3.3.cmml" xref="alg1.23.23.m3.2.3.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.23.23.m3.2c">{\bm{R}}_{k,v}^{l}</annotation></semantics></math>, Retention indices: <math id="alg1.24.24.m4.2" class="ltx_Math" alttext="{\mathbb{I}}_{k,v}" display="inline"><semantics id="alg1.24.24.m4.2a"><msub id="alg1.24.24.m4.2.3" xref="alg1.24.24.m4.2.3.cmml"><mi id="alg1.24.24.m4.2.3.2" xref="alg1.24.24.m4.2.3.2.cmml">𝕀</mi><mrow id="alg1.24.24.m4.2.2.2.4" xref="alg1.24.24.m4.2.2.2.3.cmml"><mi id="alg1.24.24.m4.1.1.1.1" xref="alg1.24.24.m4.1.1.1.1.cmml">k</mi><mo id="alg1.24.24.m4.2.2.2.4.1" xref="alg1.24.24.m4.2.2.2.3.cmml">,</mo><mi id="alg1.24.24.m4.2.2.2.2" xref="alg1.24.24.m4.2.2.2.2.cmml">v</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="alg1.24.24.m4.2b"><apply id="alg1.24.24.m4.2.3.cmml" xref="alg1.24.24.m4.2.3"><csymbol cd="ambiguous" id="alg1.24.24.m4.2.3.1.cmml" xref="alg1.24.24.m4.2.3">subscript</csymbol><ci id="alg1.24.24.m4.2.3.2.cmml" xref="alg1.24.24.m4.2.3.2">𝕀</ci><list id="alg1.24.24.m4.2.2.2.3.cmml" xref="alg1.24.24.m4.2.2.2.4"><ci id="alg1.24.24.m4.1.1.1.1.cmml" xref="alg1.24.24.m4.1.1.1.1">𝑘</ci><ci id="alg1.24.24.m4.2.2.2.2.cmml" xref="alg1.24.24.m4.2.2.2.2">𝑣</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.24.24.m4.2c">{\mathbb{I}}_{k,v}</annotation></semantics></math>, Next Token: <math id="alg1.25.25.m5.1" class="ltx_Math" alttext="t^{l}" display="inline"><semantics id="alg1.25.25.m5.1a"><msup id="alg1.25.25.m5.1.1" xref="alg1.25.25.m5.1.1.cmml"><mi id="alg1.25.25.m5.1.1.2" xref="alg1.25.25.m5.1.1.2.cmml">t</mi><mi id="alg1.25.25.m5.1.1.3" xref="alg1.25.25.m5.1.1.3.cmml">l</mi></msup><annotation-xml encoding="MathML-Content" id="alg1.25.25.m5.1b"><apply id="alg1.25.25.m5.1.1.cmml" xref="alg1.25.25.m5.1.1"><csymbol cd="ambiguous" id="alg1.25.25.m5.1.1.1.cmml" xref="alg1.25.25.m5.1.1">superscript</csymbol><ci id="alg1.25.25.m5.1.1.2.cmml" xref="alg1.25.25.m5.1.1.2">𝑡</ci><ci id="alg1.25.25.m5.1.1.3.cmml" xref="alg1.25.25.m5.1.1.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.25.25.m5.1c">t^{l}</annotation></semantics></math>
</div>
<div id="alg1.32.49" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">20</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
</div>
<div id="alg1.32.50" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">21</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
</div>
<div id="alg1.32.51" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">22</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   <span id="alg1.32.51.1" class="ltx_text ltx_font_bold">else</span>
</div>
<div id="alg1.26.26" class="ltx_listingline">  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<span id="alg1.26.26.2" class="ltx_text ltx_font_typewriter">// </span><span id="alg1.26.26.1" class="ltx_text ltx_font_typewriter">second round in cross-layer merging, while <math id="alg1.26.26.1.m1.1" class="ltx_Math" alttext="t^{l-1}" display="inline"><semantics id="alg1.26.26.1.m1.1a"><msup id="alg1.26.26.1.m1.1.1" xref="alg1.26.26.1.m1.1.1.cmml"><mi id="alg1.26.26.1.m1.1.1.2" xref="alg1.26.26.1.m1.1.1.2.cmml">t</mi><mrow id="alg1.26.26.1.m1.1.1.3" xref="alg1.26.26.1.m1.1.1.3.cmml"><mi id="alg1.26.26.1.m1.1.1.3.2" xref="alg1.26.26.1.m1.1.1.3.2.cmml">l</mi><mo id="alg1.26.26.1.m1.1.1.3.1" xref="alg1.26.26.1.m1.1.1.3.1.cmml">−</mo><mn id="alg1.26.26.1.m1.1.1.3.3" xref="alg1.26.26.1.m1.1.1.3.3.cmml">1</mn></mrow></msup><annotation-xml encoding="MathML-Content" id="alg1.26.26.1.m1.1b"><apply id="alg1.26.26.1.m1.1.1.cmml" xref="alg1.26.26.1.m1.1.1"><csymbol cd="ambiguous" id="alg1.26.26.1.m1.1.1.1.cmml" xref="alg1.26.26.1.m1.1.1">superscript</csymbol><ci id="alg1.26.26.1.m1.1.1.2.cmml" xref="alg1.26.26.1.m1.1.1.2">𝑡</ci><apply id="alg1.26.26.1.m1.1.1.3.cmml" xref="alg1.26.26.1.m1.1.1.3"><minus id="alg1.26.26.1.m1.1.1.3.1.cmml" xref="alg1.26.26.1.m1.1.1.3.1"></minus><ci id="alg1.26.26.1.m1.1.1.3.2.cmml" xref="alg1.26.26.1.m1.1.1.3.2">𝑙</ci><cn type="integer" id="alg1.26.26.1.m1.1.1.3.3.cmml" xref="alg1.26.26.1.m1.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.26.26.1.m1.1c">t^{l-1}</annotation></semantics></math> exist </span>
</div>
<div id="alg1.32.52" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">23</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<span id="alg1.32.52.1" class="ltx_text ltx_font_bold">procedure</span> <em id="alg1.32.52.2" class="ltx_emph ltx_font_typewriter">MiniCache Decoding</em><span id="alg1.32.52.3" class="ltx_text ltx_font_bold">:</span>
</div>
<div id="alg1.31.31" class="ltx_listingline">  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<span id="alg1.31.31.1" class="ltx_text"><span id="alg1.31.31.1.1" class="ltx_text ltx_font_bold">Input:</span> </span>KV cache: <math id="alg1.27.27.m1.2" class="ltx_Math" alttext="{\bm{C}}_{k,v}^{l}" display="inline"><semantics id="alg1.27.27.m1.2a"><msubsup id="alg1.27.27.m1.2.3" xref="alg1.27.27.m1.2.3.cmml"><mi id="alg1.27.27.m1.2.3.2.2" xref="alg1.27.27.m1.2.3.2.2.cmml">𝑪</mi><mrow id="alg1.27.27.m1.2.2.2.4" xref="alg1.27.27.m1.2.2.2.3.cmml"><mi id="alg1.27.27.m1.1.1.1.1" xref="alg1.27.27.m1.1.1.1.1.cmml">k</mi><mo id="alg1.27.27.m1.2.2.2.4.1" xref="alg1.27.27.m1.2.2.2.3.cmml">,</mo><mi id="alg1.27.27.m1.2.2.2.2" xref="alg1.27.27.m1.2.2.2.2.cmml">v</mi></mrow><mi id="alg1.27.27.m1.2.3.3" xref="alg1.27.27.m1.2.3.3.cmml">l</mi></msubsup><annotation-xml encoding="MathML-Content" id="alg1.27.27.m1.2b"><apply id="alg1.27.27.m1.2.3.cmml" xref="alg1.27.27.m1.2.3"><csymbol cd="ambiguous" id="alg1.27.27.m1.2.3.1.cmml" xref="alg1.27.27.m1.2.3">superscript</csymbol><apply id="alg1.27.27.m1.2.3.2.cmml" xref="alg1.27.27.m1.2.3"><csymbol cd="ambiguous" id="alg1.27.27.m1.2.3.2.1.cmml" xref="alg1.27.27.m1.2.3">subscript</csymbol><ci id="alg1.27.27.m1.2.3.2.2.cmml" xref="alg1.27.27.m1.2.3.2.2">𝑪</ci><list id="alg1.27.27.m1.2.2.2.3.cmml" xref="alg1.27.27.m1.2.2.2.4"><ci id="alg1.27.27.m1.1.1.1.1.cmml" xref="alg1.27.27.m1.1.1.1.1">𝑘</ci><ci id="alg1.27.27.m1.2.2.2.2.cmml" xref="alg1.27.27.m1.2.2.2.2">𝑣</ci></list></apply><ci id="alg1.27.27.m1.2.3.3.cmml" xref="alg1.27.27.m1.2.3.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.27.27.m1.2c">{\bm{C}}_{k,v}^{l}</annotation></semantics></math>, Norm: <math id="alg1.28.28.m2.3" class="ltx_Math" alttext="\lVert{\bm{X}}_{k,v}^{l}\rVert" display="inline"><semantics id="alg1.28.28.m2.3a"><mrow id="alg1.28.28.m2.3.3.1" xref="alg1.28.28.m2.3.3.2.cmml"><mo fence="true" rspace="0em" id="alg1.28.28.m2.3.3.1.2" xref="alg1.28.28.m2.3.3.2.1.cmml">∥</mo><msubsup id="alg1.28.28.m2.3.3.1.1" xref="alg1.28.28.m2.3.3.1.1.cmml"><mi id="alg1.28.28.m2.3.3.1.1.2.2" xref="alg1.28.28.m2.3.3.1.1.2.2.cmml">𝑿</mi><mrow id="alg1.28.28.m2.2.2.2.4" xref="alg1.28.28.m2.2.2.2.3.cmml"><mi id="alg1.28.28.m2.1.1.1.1" xref="alg1.28.28.m2.1.1.1.1.cmml">k</mi><mo id="alg1.28.28.m2.2.2.2.4.1" xref="alg1.28.28.m2.2.2.2.3.cmml">,</mo><mi id="alg1.28.28.m2.2.2.2.2" xref="alg1.28.28.m2.2.2.2.2.cmml">v</mi></mrow><mi id="alg1.28.28.m2.3.3.1.1.3" xref="alg1.28.28.m2.3.3.1.1.3.cmml">l</mi></msubsup><mo fence="true" lspace="0em" id="alg1.28.28.m2.3.3.1.3" xref="alg1.28.28.m2.3.3.2.1.cmml">∥</mo></mrow><annotation-xml encoding="MathML-Content" id="alg1.28.28.m2.3b"><apply id="alg1.28.28.m2.3.3.2.cmml" xref="alg1.28.28.m2.3.3.1"><csymbol cd="latexml" id="alg1.28.28.m2.3.3.2.1.cmml" xref="alg1.28.28.m2.3.3.1.2">delimited-∥∥</csymbol><apply id="alg1.28.28.m2.3.3.1.1.cmml" xref="alg1.28.28.m2.3.3.1.1"><csymbol cd="ambiguous" id="alg1.28.28.m2.3.3.1.1.1.cmml" xref="alg1.28.28.m2.3.3.1.1">superscript</csymbol><apply id="alg1.28.28.m2.3.3.1.1.2.cmml" xref="alg1.28.28.m2.3.3.1.1"><csymbol cd="ambiguous" id="alg1.28.28.m2.3.3.1.1.2.1.cmml" xref="alg1.28.28.m2.3.3.1.1">subscript</csymbol><ci id="alg1.28.28.m2.3.3.1.1.2.2.cmml" xref="alg1.28.28.m2.3.3.1.1.2.2">𝑿</ci><list id="alg1.28.28.m2.2.2.2.3.cmml" xref="alg1.28.28.m2.2.2.2.4"><ci id="alg1.28.28.m2.1.1.1.1.cmml" xref="alg1.28.28.m2.1.1.1.1">𝑘</ci><ci id="alg1.28.28.m2.2.2.2.2.cmml" xref="alg1.28.28.m2.2.2.2.2">𝑣</ci></list></apply><ci id="alg1.28.28.m2.3.3.1.1.3.cmml" xref="alg1.28.28.m2.3.3.1.1.3">𝑙</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.28.28.m2.3c">\lVert{\bm{X}}_{k,v}^{l}\rVert</annotation></semantics></math>, Unmerged Tokens: <math id="alg1.29.29.m3.2" class="ltx_Math" alttext="{\bm{R}}_{k,v}^{l}" display="inline"><semantics id="alg1.29.29.m3.2a"><msubsup id="alg1.29.29.m3.2.3" xref="alg1.29.29.m3.2.3.cmml"><mi id="alg1.29.29.m3.2.3.2.2" xref="alg1.29.29.m3.2.3.2.2.cmml">𝑹</mi><mrow id="alg1.29.29.m3.2.2.2.4" xref="alg1.29.29.m3.2.2.2.3.cmml"><mi id="alg1.29.29.m3.1.1.1.1" xref="alg1.29.29.m3.1.1.1.1.cmml">k</mi><mo id="alg1.29.29.m3.2.2.2.4.1" xref="alg1.29.29.m3.2.2.2.3.cmml">,</mo><mi id="alg1.29.29.m3.2.2.2.2" xref="alg1.29.29.m3.2.2.2.2.cmml">v</mi></mrow><mi id="alg1.29.29.m3.2.3.3" xref="alg1.29.29.m3.2.3.3.cmml">l</mi></msubsup><annotation-xml encoding="MathML-Content" id="alg1.29.29.m3.2b"><apply id="alg1.29.29.m3.2.3.cmml" xref="alg1.29.29.m3.2.3"><csymbol cd="ambiguous" id="alg1.29.29.m3.2.3.1.cmml" xref="alg1.29.29.m3.2.3">superscript</csymbol><apply id="alg1.29.29.m3.2.3.2.cmml" xref="alg1.29.29.m3.2.3"><csymbol cd="ambiguous" id="alg1.29.29.m3.2.3.2.1.cmml" xref="alg1.29.29.m3.2.3">subscript</csymbol><ci id="alg1.29.29.m3.2.3.2.2.cmml" xref="alg1.29.29.m3.2.3.2.2">𝑹</ci><list id="alg1.29.29.m3.2.2.2.3.cmml" xref="alg1.29.29.m3.2.2.2.4"><ci id="alg1.29.29.m3.1.1.1.1.cmml" xref="alg1.29.29.m3.1.1.1.1">𝑘</ci><ci id="alg1.29.29.m3.2.2.2.2.cmml" xref="alg1.29.29.m3.2.2.2.2">𝑣</ci></list></apply><ci id="alg1.29.29.m3.2.3.3.cmml" xref="alg1.29.29.m3.2.3.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.29.29.m3.2c">{\bm{R}}_{k,v}^{l}</annotation></semantics></math>, Retention indices: <math id="alg1.30.30.m4.2" class="ltx_Math" alttext="{\mathbb{I}}_{k,v}" display="inline"><semantics id="alg1.30.30.m4.2a"><msub id="alg1.30.30.m4.2.3" xref="alg1.30.30.m4.2.3.cmml"><mi id="alg1.30.30.m4.2.3.2" xref="alg1.30.30.m4.2.3.2.cmml">𝕀</mi><mrow id="alg1.30.30.m4.2.2.2.4" xref="alg1.30.30.m4.2.2.2.3.cmml"><mi id="alg1.30.30.m4.1.1.1.1" xref="alg1.30.30.m4.1.1.1.1.cmml">k</mi><mo id="alg1.30.30.m4.2.2.2.4.1" xref="alg1.30.30.m4.2.2.2.3.cmml">,</mo><mi id="alg1.30.30.m4.2.2.2.2" xref="alg1.30.30.m4.2.2.2.2.cmml">v</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="alg1.30.30.m4.2b"><apply id="alg1.30.30.m4.2.3.cmml" xref="alg1.30.30.m4.2.3"><csymbol cd="ambiguous" id="alg1.30.30.m4.2.3.1.cmml" xref="alg1.30.30.m4.2.3">subscript</csymbol><ci id="alg1.30.30.m4.2.3.2.cmml" xref="alg1.30.30.m4.2.3.2">𝕀</ci><list id="alg1.30.30.m4.2.2.2.3.cmml" xref="alg1.30.30.m4.2.2.2.4"><ci id="alg1.30.30.m4.1.1.1.1.cmml" xref="alg1.30.30.m4.1.1.1.1">𝑘</ci><ci id="alg1.30.30.m4.2.2.2.2.cmml" xref="alg1.30.30.m4.2.2.2.2">𝑣</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.30.30.m4.2c">{\mathbb{I}}_{k,v}</annotation></semantics></math>, Next Token: <math id="alg1.31.31.m5.2" class="ltx_Math" alttext="t^{l},t^{l-1}" display="inline"><semantics id="alg1.31.31.m5.2a"><mrow id="alg1.31.31.m5.2.2.2" xref="alg1.31.31.m5.2.2.3.cmml"><msup id="alg1.31.31.m5.1.1.1.1" xref="alg1.31.31.m5.1.1.1.1.cmml"><mi id="alg1.31.31.m5.1.1.1.1.2" xref="alg1.31.31.m5.1.1.1.1.2.cmml">t</mi><mi id="alg1.31.31.m5.1.1.1.1.3" xref="alg1.31.31.m5.1.1.1.1.3.cmml">l</mi></msup><mo id="alg1.31.31.m5.2.2.2.3" xref="alg1.31.31.m5.2.2.3.cmml">,</mo><msup id="alg1.31.31.m5.2.2.2.2" xref="alg1.31.31.m5.2.2.2.2.cmml"><mi id="alg1.31.31.m5.2.2.2.2.2" xref="alg1.31.31.m5.2.2.2.2.2.cmml">t</mi><mrow id="alg1.31.31.m5.2.2.2.2.3" xref="alg1.31.31.m5.2.2.2.2.3.cmml"><mi id="alg1.31.31.m5.2.2.2.2.3.2" xref="alg1.31.31.m5.2.2.2.2.3.2.cmml">l</mi><mo id="alg1.31.31.m5.2.2.2.2.3.1" xref="alg1.31.31.m5.2.2.2.2.3.1.cmml">−</mo><mn id="alg1.31.31.m5.2.2.2.2.3.3" xref="alg1.31.31.m5.2.2.2.2.3.3.cmml">1</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="alg1.31.31.m5.2b"><list id="alg1.31.31.m5.2.2.3.cmml" xref="alg1.31.31.m5.2.2.2"><apply id="alg1.31.31.m5.1.1.1.1.cmml" xref="alg1.31.31.m5.1.1.1.1"><csymbol cd="ambiguous" id="alg1.31.31.m5.1.1.1.1.1.cmml" xref="alg1.31.31.m5.1.1.1.1">superscript</csymbol><ci id="alg1.31.31.m5.1.1.1.1.2.cmml" xref="alg1.31.31.m5.1.1.1.1.2">𝑡</ci><ci id="alg1.31.31.m5.1.1.1.1.3.cmml" xref="alg1.31.31.m5.1.1.1.1.3">𝑙</ci></apply><apply id="alg1.31.31.m5.2.2.2.2.cmml" xref="alg1.31.31.m5.2.2.2.2"><csymbol cd="ambiguous" id="alg1.31.31.m5.2.2.2.2.1.cmml" xref="alg1.31.31.m5.2.2.2.2">superscript</csymbol><ci id="alg1.31.31.m5.2.2.2.2.2.cmml" xref="alg1.31.31.m5.2.2.2.2.2">𝑡</ci><apply id="alg1.31.31.m5.2.2.2.2.3.cmml" xref="alg1.31.31.m5.2.2.2.2.3"><minus id="alg1.31.31.m5.2.2.2.2.3.1.cmml" xref="alg1.31.31.m5.2.2.2.2.3.1"></minus><ci id="alg1.31.31.m5.2.2.2.2.3.2.cmml" xref="alg1.31.31.m5.2.2.2.2.3.2">𝑙</ci><cn type="integer" id="alg1.31.31.m5.2.2.2.2.3.3.cmml" xref="alg1.31.31.m5.2.2.2.2.3.3">1</cn></apply></apply></list></annotation-xml><annotation encoding="application/x-tex" id="alg1.31.31.m5.2c">t^{l},t^{l-1}</annotation></semantics></math>
</div>
<div id="alg1.32.53" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">24</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   

</div>
<div id="alg1.32.54" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">25</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
</div>
<div id="alg1.32.55" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">26</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
</div>
<div id="alg1.32.56" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">27</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
</div>
<div id="alg1.32.57" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">28</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
</div>
<div id="alg1.32.32" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">29</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   <span id="alg1.32.32.2" class="ltx_text ltx_font_bold">return</span> <em id="alg1.32.32.1" class="ltx_emph ltx_font_italic"><math id="alg1.32.32.1.m1.1" class="ltx_Math" alttext="{\bm{O}}" display="inline"><semantics id="alg1.32.32.1.m1.1a"><mi id="alg1.32.32.1.m1.1.1" xref="alg1.32.32.1.m1.1.1.cmml">𝐎</mi><annotation-xml encoding="MathML-Content" id="alg1.32.32.1.m1.1b"><ci id="alg1.32.32.1.m1.1.1.cmml" xref="alg1.32.32.1.m1.1.1">𝐎</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.32.32.1.m1.1c">{\bm{O}}</annotation></semantics></math></em>

</div>
<div id="alg1.32.58" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">30</span>
</div>
<div id="alg1.32.59" class="ltx_listingline">
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float"><span id="alg1.34.1.1" class="ltx_text ltx_font_bold">Algorithm 1</span> </span>The MiniCache Inference Algorithm</figcaption>
</figure>
<figure id="alg2" class="ltx_float ltx_algorithm">
<div id="alg2.38" class="ltx_listing ltx_lst_numbers_left ltx_listing">
<div id="alg2.2.2" class="ltx_listingline">

<span id="alg2.2.2.1" class="ltx_text"><span id="alg2.2.2.1.1" class="ltx_text ltx_font_bold">Hyperparameter:</span> </span>number of layers <math id="alg2.1.1.m1.1" class="ltx_Math" alttext="L" display="inline"><semantics id="alg2.1.1.m1.1a"><mi id="alg2.1.1.m1.1.1" xref="alg2.1.1.m1.1.1.cmml">L</mi><annotation-xml encoding="MathML-Content" id="alg2.1.1.m1.1b"><ci id="alg2.1.1.m1.1.1.cmml" xref="alg2.1.1.m1.1.1">𝐿</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.1.1.m1.1c">L</annotation></semantics></math>, merging beginning layer <math id="alg2.2.2.m2.1" class="ltx_Math" alttext="S" display="inline"><semantics id="alg2.2.2.m2.1a"><mi id="alg2.2.2.m2.1.1" xref="alg2.2.2.m2.1.1.cmml">S</mi><annotation-xml encoding="MathML-Content" id="alg2.2.2.m2.1b"><ci id="alg2.2.2.m2.1.1.cmml" xref="alg2.2.2.m2.1.1">𝑆</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.2.2.m2.1c">S</annotation></semantics></math>
</div>
<div id="alg2.38.39" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">1</span>




</div>
<div id="alg2.38.40" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">2</span><span id="alg2.38.40.1" class="ltx_text ltx_font_bold">procedure</span> <em id="alg2.38.40.2" class="ltx_emph ltx_font_typewriter">MiniCache Prefill</em><span id="alg2.38.40.3" class="ltx_text ltx_font_bold">:</span>
</div>
<div id="alg2.7.7" class="ltx_listingline">  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<span id="alg2.7.7.1" class="ltx_text"><span id="alg2.7.7.1.1" class="ltx_text ltx_font_bold">Input:</span> </span>KV cache from current layer <math id="alg2.3.3.m1.1" class="ltx_Math" alttext="l" display="inline"><semantics id="alg2.3.3.m1.1a"><mi id="alg2.3.3.m1.1.1" xref="alg2.3.3.m1.1.1.cmml">l</mi><annotation-xml encoding="MathML-Content" id="alg2.3.3.m1.1b"><ci id="alg2.3.3.m1.1.1.cmml" xref="alg2.3.3.m1.1.1">𝑙</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.3.3.m1.1c">l</annotation></semantics></math>: <math id="alg2.4.4.m2.2" class="ltx_Math" alttext="{\bm{X}}_{k,v}^{l}\in\mathbb{R}^{2\times t_{\text{prompt}}\times d}" display="inline"><semantics id="alg2.4.4.m2.2a"><mrow id="alg2.4.4.m2.2.3" xref="alg2.4.4.m2.2.3.cmml"><msubsup id="alg2.4.4.m2.2.3.2" xref="alg2.4.4.m2.2.3.2.cmml"><mi id="alg2.4.4.m2.2.3.2.2.2" xref="alg2.4.4.m2.2.3.2.2.2.cmml">𝑿</mi><mrow id="alg2.4.4.m2.2.2.2.4" xref="alg2.4.4.m2.2.2.2.3.cmml"><mi id="alg2.4.4.m2.1.1.1.1" xref="alg2.4.4.m2.1.1.1.1.cmml">k</mi><mo id="alg2.4.4.m2.2.2.2.4.1" xref="alg2.4.4.m2.2.2.2.3.cmml">,</mo><mi id="alg2.4.4.m2.2.2.2.2" xref="alg2.4.4.m2.2.2.2.2.cmml">v</mi></mrow><mi id="alg2.4.4.m2.2.3.2.3" xref="alg2.4.4.m2.2.3.2.3.cmml">l</mi></msubsup><mo id="alg2.4.4.m2.2.3.1" xref="alg2.4.4.m2.2.3.1.cmml">∈</mo><msup id="alg2.4.4.m2.2.3.3" xref="alg2.4.4.m2.2.3.3.cmml"><mi id="alg2.4.4.m2.2.3.3.2" xref="alg2.4.4.m2.2.3.3.2.cmml">ℝ</mi><mrow id="alg2.4.4.m2.2.3.3.3" xref="alg2.4.4.m2.2.3.3.3.cmml"><mn id="alg2.4.4.m2.2.3.3.3.2" xref="alg2.4.4.m2.2.3.3.3.2.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="alg2.4.4.m2.2.3.3.3.1" xref="alg2.4.4.m2.2.3.3.3.1.cmml">×</mo><msub id="alg2.4.4.m2.2.3.3.3.3" xref="alg2.4.4.m2.2.3.3.3.3.cmml"><mi id="alg2.4.4.m2.2.3.3.3.3.2" xref="alg2.4.4.m2.2.3.3.3.3.2.cmml">t</mi><mtext id="alg2.4.4.m2.2.3.3.3.3.3" xref="alg2.4.4.m2.2.3.3.3.3.3a.cmml">prompt</mtext></msub><mo lspace="0.222em" rspace="0.222em" id="alg2.4.4.m2.2.3.3.3.1a" xref="alg2.4.4.m2.2.3.3.3.1.cmml">×</mo><mi id="alg2.4.4.m2.2.3.3.3.4" xref="alg2.4.4.m2.2.3.3.3.4.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="alg2.4.4.m2.2b"><apply id="alg2.4.4.m2.2.3.cmml" xref="alg2.4.4.m2.2.3"><in id="alg2.4.4.m2.2.3.1.cmml" xref="alg2.4.4.m2.2.3.1"></in><apply id="alg2.4.4.m2.2.3.2.cmml" xref="alg2.4.4.m2.2.3.2"><csymbol cd="ambiguous" id="alg2.4.4.m2.2.3.2.1.cmml" xref="alg2.4.4.m2.2.3.2">superscript</csymbol><apply id="alg2.4.4.m2.2.3.2.2.cmml" xref="alg2.4.4.m2.2.3.2"><csymbol cd="ambiguous" id="alg2.4.4.m2.2.3.2.2.1.cmml" xref="alg2.4.4.m2.2.3.2">subscript</csymbol><ci id="alg2.4.4.m2.2.3.2.2.2.cmml" xref="alg2.4.4.m2.2.3.2.2.2">𝑿</ci><list id="alg2.4.4.m2.2.2.2.3.cmml" xref="alg2.4.4.m2.2.2.2.4"><ci id="alg2.4.4.m2.1.1.1.1.cmml" xref="alg2.4.4.m2.1.1.1.1">𝑘</ci><ci id="alg2.4.4.m2.2.2.2.2.cmml" xref="alg2.4.4.m2.2.2.2.2">𝑣</ci></list></apply><ci id="alg2.4.4.m2.2.3.2.3.cmml" xref="alg2.4.4.m2.2.3.2.3">𝑙</ci></apply><apply id="alg2.4.4.m2.2.3.3.cmml" xref="alg2.4.4.m2.2.3.3"><csymbol cd="ambiguous" id="alg2.4.4.m2.2.3.3.1.cmml" xref="alg2.4.4.m2.2.3.3">superscript</csymbol><ci id="alg2.4.4.m2.2.3.3.2.cmml" xref="alg2.4.4.m2.2.3.3.2">ℝ</ci><apply id="alg2.4.4.m2.2.3.3.3.cmml" xref="alg2.4.4.m2.2.3.3.3"><times id="alg2.4.4.m2.2.3.3.3.1.cmml" xref="alg2.4.4.m2.2.3.3.3.1"></times><cn type="integer" id="alg2.4.4.m2.2.3.3.3.2.cmml" xref="alg2.4.4.m2.2.3.3.3.2">2</cn><apply id="alg2.4.4.m2.2.3.3.3.3.cmml" xref="alg2.4.4.m2.2.3.3.3.3"><csymbol cd="ambiguous" id="alg2.4.4.m2.2.3.3.3.3.1.cmml" xref="alg2.4.4.m2.2.3.3.3.3">subscript</csymbol><ci id="alg2.4.4.m2.2.3.3.3.3.2.cmml" xref="alg2.4.4.m2.2.3.3.3.3.2">𝑡</ci><ci id="alg2.4.4.m2.2.3.3.3.3.3a.cmml" xref="alg2.4.4.m2.2.3.3.3.3.3"><mtext mathsize="50%" id="alg2.4.4.m2.2.3.3.3.3.3.cmml" xref="alg2.4.4.m2.2.3.3.3.3.3">prompt</mtext></ci></apply><ci id="alg2.4.4.m2.2.3.3.3.4.cmml" xref="alg2.4.4.m2.2.3.3.3.4">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.4.4.m2.2c">{\bm{X}}_{k,v}^{l}\in\mathbb{R}^{2\times t_{\text{prompt}}\times d}</annotation></semantics></math>, KV cache from previous layer <math id="alg2.5.5.m3.1" class="ltx_Math" alttext="l-1" display="inline"><semantics id="alg2.5.5.m3.1a"><mrow id="alg2.5.5.m3.1.1" xref="alg2.5.5.m3.1.1.cmml"><mi id="alg2.5.5.m3.1.1.2" xref="alg2.5.5.m3.1.1.2.cmml">l</mi><mo id="alg2.5.5.m3.1.1.1" xref="alg2.5.5.m3.1.1.1.cmml">−</mo><mn id="alg2.5.5.m3.1.1.3" xref="alg2.5.5.m3.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="alg2.5.5.m3.1b"><apply id="alg2.5.5.m3.1.1.cmml" xref="alg2.5.5.m3.1.1"><minus id="alg2.5.5.m3.1.1.1.cmml" xref="alg2.5.5.m3.1.1.1"></minus><ci id="alg2.5.5.m3.1.1.2.cmml" xref="alg2.5.5.m3.1.1.2">𝑙</ci><cn type="integer" id="alg2.5.5.m3.1.1.3.cmml" xref="alg2.5.5.m3.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.5.5.m3.1c">l-1</annotation></semantics></math>: <math id="alg2.6.6.m4.2" class="ltx_Math" alttext="{\bm{X}}_{k,v}^{l-1}\in\mathbb{R}^{2\times t_{\text{prompt}}\times d}" display="inline"><semantics id="alg2.6.6.m4.2a"><mrow id="alg2.6.6.m4.2.3" xref="alg2.6.6.m4.2.3.cmml"><msubsup id="alg2.6.6.m4.2.3.2" xref="alg2.6.6.m4.2.3.2.cmml"><mi id="alg2.6.6.m4.2.3.2.2.2" xref="alg2.6.6.m4.2.3.2.2.2.cmml">𝑿</mi><mrow id="alg2.6.6.m4.2.2.2.4" xref="alg2.6.6.m4.2.2.2.3.cmml"><mi id="alg2.6.6.m4.1.1.1.1" xref="alg2.6.6.m4.1.1.1.1.cmml">k</mi><mo id="alg2.6.6.m4.2.2.2.4.1" xref="alg2.6.6.m4.2.2.2.3.cmml">,</mo><mi id="alg2.6.6.m4.2.2.2.2" xref="alg2.6.6.m4.2.2.2.2.cmml">v</mi></mrow><mrow id="alg2.6.6.m4.2.3.2.3" xref="alg2.6.6.m4.2.3.2.3.cmml"><mi id="alg2.6.6.m4.2.3.2.3.2" xref="alg2.6.6.m4.2.3.2.3.2.cmml">l</mi><mo id="alg2.6.6.m4.2.3.2.3.1" xref="alg2.6.6.m4.2.3.2.3.1.cmml">−</mo><mn id="alg2.6.6.m4.2.3.2.3.3" xref="alg2.6.6.m4.2.3.2.3.3.cmml">1</mn></mrow></msubsup><mo id="alg2.6.6.m4.2.3.1" xref="alg2.6.6.m4.2.3.1.cmml">∈</mo><msup id="alg2.6.6.m4.2.3.3" xref="alg2.6.6.m4.2.3.3.cmml"><mi id="alg2.6.6.m4.2.3.3.2" xref="alg2.6.6.m4.2.3.3.2.cmml">ℝ</mi><mrow id="alg2.6.6.m4.2.3.3.3" xref="alg2.6.6.m4.2.3.3.3.cmml"><mn id="alg2.6.6.m4.2.3.3.3.2" xref="alg2.6.6.m4.2.3.3.3.2.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="alg2.6.6.m4.2.3.3.3.1" xref="alg2.6.6.m4.2.3.3.3.1.cmml">×</mo><msub id="alg2.6.6.m4.2.3.3.3.3" xref="alg2.6.6.m4.2.3.3.3.3.cmml"><mi id="alg2.6.6.m4.2.3.3.3.3.2" xref="alg2.6.6.m4.2.3.3.3.3.2.cmml">t</mi><mtext id="alg2.6.6.m4.2.3.3.3.3.3" xref="alg2.6.6.m4.2.3.3.3.3.3a.cmml">prompt</mtext></msub><mo lspace="0.222em" rspace="0.222em" id="alg2.6.6.m4.2.3.3.3.1a" xref="alg2.6.6.m4.2.3.3.3.1.cmml">×</mo><mi id="alg2.6.6.m4.2.3.3.3.4" xref="alg2.6.6.m4.2.3.3.3.4.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="alg2.6.6.m4.2b"><apply id="alg2.6.6.m4.2.3.cmml" xref="alg2.6.6.m4.2.3"><in id="alg2.6.6.m4.2.3.1.cmml" xref="alg2.6.6.m4.2.3.1"></in><apply id="alg2.6.6.m4.2.3.2.cmml" xref="alg2.6.6.m4.2.3.2"><csymbol cd="ambiguous" id="alg2.6.6.m4.2.3.2.1.cmml" xref="alg2.6.6.m4.2.3.2">superscript</csymbol><apply id="alg2.6.6.m4.2.3.2.2.cmml" xref="alg2.6.6.m4.2.3.2"><csymbol cd="ambiguous" id="alg2.6.6.m4.2.3.2.2.1.cmml" xref="alg2.6.6.m4.2.3.2">subscript</csymbol><ci id="alg2.6.6.m4.2.3.2.2.2.cmml" xref="alg2.6.6.m4.2.3.2.2.2">𝑿</ci><list id="alg2.6.6.m4.2.2.2.3.cmml" xref="alg2.6.6.m4.2.2.2.4"><ci id="alg2.6.6.m4.1.1.1.1.cmml" xref="alg2.6.6.m4.1.1.1.1">𝑘</ci><ci id="alg2.6.6.m4.2.2.2.2.cmml" xref="alg2.6.6.m4.2.2.2.2">𝑣</ci></list></apply><apply id="alg2.6.6.m4.2.3.2.3.cmml" xref="alg2.6.6.m4.2.3.2.3"><minus id="alg2.6.6.m4.2.3.2.3.1.cmml" xref="alg2.6.6.m4.2.3.2.3.1"></minus><ci id="alg2.6.6.m4.2.3.2.3.2.cmml" xref="alg2.6.6.m4.2.3.2.3.2">𝑙</ci><cn type="integer" id="alg2.6.6.m4.2.3.2.3.3.cmml" xref="alg2.6.6.m4.2.3.2.3.3">1</cn></apply></apply><apply id="alg2.6.6.m4.2.3.3.cmml" xref="alg2.6.6.m4.2.3.3"><csymbol cd="ambiguous" id="alg2.6.6.m4.2.3.3.1.cmml" xref="alg2.6.6.m4.2.3.3">superscript</csymbol><ci id="alg2.6.6.m4.2.3.3.2.cmml" xref="alg2.6.6.m4.2.3.3.2">ℝ</ci><apply id="alg2.6.6.m4.2.3.3.3.cmml" xref="alg2.6.6.m4.2.3.3.3"><times id="alg2.6.6.m4.2.3.3.3.1.cmml" xref="alg2.6.6.m4.2.3.3.3.1"></times><cn type="integer" id="alg2.6.6.m4.2.3.3.3.2.cmml" xref="alg2.6.6.m4.2.3.3.3.2">2</cn><apply id="alg2.6.6.m4.2.3.3.3.3.cmml" xref="alg2.6.6.m4.2.3.3.3.3"><csymbol cd="ambiguous" id="alg2.6.6.m4.2.3.3.3.3.1.cmml" xref="alg2.6.6.m4.2.3.3.3.3">subscript</csymbol><ci id="alg2.6.6.m4.2.3.3.3.3.2.cmml" xref="alg2.6.6.m4.2.3.3.3.3.2">𝑡</ci><ci id="alg2.6.6.m4.2.3.3.3.3.3a.cmml" xref="alg2.6.6.m4.2.3.3.3.3.3"><mtext mathsize="50%" id="alg2.6.6.m4.2.3.3.3.3.3.cmml" xref="alg2.6.6.m4.2.3.3.3.3.3">prompt</mtext></ci></apply><ci id="alg2.6.6.m4.2.3.3.3.4.cmml" xref="alg2.6.6.m4.2.3.3.3.4">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.6.6.m4.2c">{\bm{X}}_{k,v}^{l-1}\in\mathbb{R}^{2\times t_{\text{prompt}}\times d}</annotation></semantics></math>, token retention threshold: <math id="alg2.7.7.m5.1" class="ltx_Math" alttext="\gamma" display="inline"><semantics id="alg2.7.7.m5.1a"><mi id="alg2.7.7.m5.1.1" xref="alg2.7.7.m5.1.1.cmml">γ</mi><annotation-xml encoding="MathML-Content" id="alg2.7.7.m5.1b"><ci id="alg2.7.7.m5.1.1.cmml" xref="alg2.7.7.m5.1.1">𝛾</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.7.7.m5.1c">\gamma</annotation></semantics></math>
</div>
<div id="alg2.8.8" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">3</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<math id="alg2.8.8.m1.6" class="ltx_Math" alttext="{\bm{E}}_{k}^{l},\lVert{\bm{X}}_{k}^{l}\rVert,\lVert{\bm{X}}_{k}^{l-1}\rVert,\Omega_{k}\leftarrow\text{Merge}({\bm{X}}_{k}^{l},{\bm{X}}_{k}^{l-1})" display="inline"><semantics id="alg2.8.8.m1.6a"><mrow id="alg2.8.8.m1.6.6" xref="alg2.8.8.m1.6.6.cmml"><mrow id="alg2.8.8.m1.4.4.4.4" xref="alg2.8.8.m1.4.4.4.5.cmml"><msubsup id="alg2.8.8.m1.1.1.1.1.1" xref="alg2.8.8.m1.1.1.1.1.1.cmml"><mi id="alg2.8.8.m1.1.1.1.1.1.2.2" xref="alg2.8.8.m1.1.1.1.1.1.2.2.cmml">𝑬</mi><mi id="alg2.8.8.m1.1.1.1.1.1.2.3" xref="alg2.8.8.m1.1.1.1.1.1.2.3.cmml">k</mi><mi id="alg2.8.8.m1.1.1.1.1.1.3" xref="alg2.8.8.m1.1.1.1.1.1.3.cmml">l</mi></msubsup><mo id="alg2.8.8.m1.4.4.4.4.5" xref="alg2.8.8.m1.4.4.4.5.cmml">,</mo><mrow id="alg2.8.8.m1.2.2.2.2.2.1" xref="alg2.8.8.m1.2.2.2.2.2.2.cmml"><mo fence="true" lspace="0em" rspace="0em" id="alg2.8.8.m1.2.2.2.2.2.1.2" xref="alg2.8.8.m1.2.2.2.2.2.2.1.cmml">∥</mo><msubsup id="alg2.8.8.m1.2.2.2.2.2.1.1" xref="alg2.8.8.m1.2.2.2.2.2.1.1.cmml"><mi id="alg2.8.8.m1.2.2.2.2.2.1.1.2.2" xref="alg2.8.8.m1.2.2.2.2.2.1.1.2.2.cmml">𝑿</mi><mi id="alg2.8.8.m1.2.2.2.2.2.1.1.2.3" xref="alg2.8.8.m1.2.2.2.2.2.1.1.2.3.cmml">k</mi><mi id="alg2.8.8.m1.2.2.2.2.2.1.1.3" xref="alg2.8.8.m1.2.2.2.2.2.1.1.3.cmml">l</mi></msubsup><mo fence="true" lspace="0em" rspace="0em" id="alg2.8.8.m1.2.2.2.2.2.1.3" xref="alg2.8.8.m1.2.2.2.2.2.2.1.cmml">∥</mo></mrow><mo id="alg2.8.8.m1.4.4.4.4.6" xref="alg2.8.8.m1.4.4.4.5.cmml">,</mo><mrow id="alg2.8.8.m1.3.3.3.3.3.1" xref="alg2.8.8.m1.3.3.3.3.3.2.cmml"><mo fence="true" lspace="0em" rspace="0em" id="alg2.8.8.m1.3.3.3.3.3.1.2" xref="alg2.8.8.m1.3.3.3.3.3.2.1.cmml">∥</mo><msubsup id="alg2.8.8.m1.3.3.3.3.3.1.1" xref="alg2.8.8.m1.3.3.3.3.3.1.1.cmml"><mi id="alg2.8.8.m1.3.3.3.3.3.1.1.2.2" xref="alg2.8.8.m1.3.3.3.3.3.1.1.2.2.cmml">𝑿</mi><mi id="alg2.8.8.m1.3.3.3.3.3.1.1.2.3" xref="alg2.8.8.m1.3.3.3.3.3.1.1.2.3.cmml">k</mi><mrow id="alg2.8.8.m1.3.3.3.3.3.1.1.3" xref="alg2.8.8.m1.3.3.3.3.3.1.1.3.cmml"><mi id="alg2.8.8.m1.3.3.3.3.3.1.1.3.2" xref="alg2.8.8.m1.3.3.3.3.3.1.1.3.2.cmml">l</mi><mo id="alg2.8.8.m1.3.3.3.3.3.1.1.3.1" xref="alg2.8.8.m1.3.3.3.3.3.1.1.3.1.cmml">−</mo><mn id="alg2.8.8.m1.3.3.3.3.3.1.1.3.3" xref="alg2.8.8.m1.3.3.3.3.3.1.1.3.3.cmml">1</mn></mrow></msubsup><mo fence="true" lspace="0em" rspace="0em" id="alg2.8.8.m1.3.3.3.3.3.1.3" xref="alg2.8.8.m1.3.3.3.3.3.2.1.cmml">∥</mo></mrow><mo id="alg2.8.8.m1.4.4.4.4.7" xref="alg2.8.8.m1.4.4.4.5.cmml">,</mo><msub id="alg2.8.8.m1.4.4.4.4.4" xref="alg2.8.8.m1.4.4.4.4.4.cmml"><mi mathvariant="normal" id="alg2.8.8.m1.4.4.4.4.4.2" xref="alg2.8.8.m1.4.4.4.4.4.2.cmml">Ω</mi><mi id="alg2.8.8.m1.4.4.4.4.4.3" xref="alg2.8.8.m1.4.4.4.4.4.3.cmml">k</mi></msub></mrow><mo stretchy="false" id="alg2.8.8.m1.6.6.7" xref="alg2.8.8.m1.6.6.7.cmml">←</mo><mrow id="alg2.8.8.m1.6.6.6" xref="alg2.8.8.m1.6.6.6.cmml"><mtext id="alg2.8.8.m1.6.6.6.4" xref="alg2.8.8.m1.6.6.6.4a.cmml">Merge</mtext><mo lspace="0em" rspace="0em" id="alg2.8.8.m1.6.6.6.3" xref="alg2.8.8.m1.6.6.6.3.cmml">​</mo><mrow id="alg2.8.8.m1.6.6.6.2.2" xref="alg2.8.8.m1.6.6.6.2.3.cmml"><mo stretchy="false" id="alg2.8.8.m1.6.6.6.2.2.3" xref="alg2.8.8.m1.6.6.6.2.3.cmml">(</mo><msubsup id="alg2.8.8.m1.5.5.5.1.1.1" xref="alg2.8.8.m1.5.5.5.1.1.1.cmml"><mi id="alg2.8.8.m1.5.5.5.1.1.1.2.2" xref="alg2.8.8.m1.5.5.5.1.1.1.2.2.cmml">𝑿</mi><mi id="alg2.8.8.m1.5.5.5.1.1.1.2.3" xref="alg2.8.8.m1.5.5.5.1.1.1.2.3.cmml">k</mi><mi id="alg2.8.8.m1.5.5.5.1.1.1.3" xref="alg2.8.8.m1.5.5.5.1.1.1.3.cmml">l</mi></msubsup><mo id="alg2.8.8.m1.6.6.6.2.2.4" xref="alg2.8.8.m1.6.6.6.2.3.cmml">,</mo><msubsup id="alg2.8.8.m1.6.6.6.2.2.2" xref="alg2.8.8.m1.6.6.6.2.2.2.cmml"><mi id="alg2.8.8.m1.6.6.6.2.2.2.2.2" xref="alg2.8.8.m1.6.6.6.2.2.2.2.2.cmml">𝑿</mi><mi id="alg2.8.8.m1.6.6.6.2.2.2.2.3" xref="alg2.8.8.m1.6.6.6.2.2.2.2.3.cmml">k</mi><mrow id="alg2.8.8.m1.6.6.6.2.2.2.3" xref="alg2.8.8.m1.6.6.6.2.2.2.3.cmml"><mi id="alg2.8.8.m1.6.6.6.2.2.2.3.2" xref="alg2.8.8.m1.6.6.6.2.2.2.3.2.cmml">l</mi><mo id="alg2.8.8.m1.6.6.6.2.2.2.3.1" xref="alg2.8.8.m1.6.6.6.2.2.2.3.1.cmml">−</mo><mn id="alg2.8.8.m1.6.6.6.2.2.2.3.3" xref="alg2.8.8.m1.6.6.6.2.2.2.3.3.cmml">1</mn></mrow></msubsup><mo stretchy="false" id="alg2.8.8.m1.6.6.6.2.2.5" xref="alg2.8.8.m1.6.6.6.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.8.8.m1.6b"><apply id="alg2.8.8.m1.6.6.cmml" xref="alg2.8.8.m1.6.6"><ci id="alg2.8.8.m1.6.6.7.cmml" xref="alg2.8.8.m1.6.6.7">←</ci><list id="alg2.8.8.m1.4.4.4.5.cmml" xref="alg2.8.8.m1.4.4.4.4"><apply id="alg2.8.8.m1.1.1.1.1.1.cmml" xref="alg2.8.8.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.8.8.m1.1.1.1.1.1.1.cmml" xref="alg2.8.8.m1.1.1.1.1.1">superscript</csymbol><apply id="alg2.8.8.m1.1.1.1.1.1.2.cmml" xref="alg2.8.8.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.8.8.m1.1.1.1.1.1.2.1.cmml" xref="alg2.8.8.m1.1.1.1.1.1">subscript</csymbol><ci id="alg2.8.8.m1.1.1.1.1.1.2.2.cmml" xref="alg2.8.8.m1.1.1.1.1.1.2.2">𝑬</ci><ci id="alg2.8.8.m1.1.1.1.1.1.2.3.cmml" xref="alg2.8.8.m1.1.1.1.1.1.2.3">𝑘</ci></apply><ci id="alg2.8.8.m1.1.1.1.1.1.3.cmml" xref="alg2.8.8.m1.1.1.1.1.1.3">𝑙</ci></apply><apply id="alg2.8.8.m1.2.2.2.2.2.2.cmml" xref="alg2.8.8.m1.2.2.2.2.2.1"><csymbol cd="latexml" id="alg2.8.8.m1.2.2.2.2.2.2.1.cmml" xref="alg2.8.8.m1.2.2.2.2.2.1.2">delimited-∥∥</csymbol><apply id="alg2.8.8.m1.2.2.2.2.2.1.1.cmml" xref="alg2.8.8.m1.2.2.2.2.2.1.1"><csymbol cd="ambiguous" id="alg2.8.8.m1.2.2.2.2.2.1.1.1.cmml" xref="alg2.8.8.m1.2.2.2.2.2.1.1">superscript</csymbol><apply id="alg2.8.8.m1.2.2.2.2.2.1.1.2.cmml" xref="alg2.8.8.m1.2.2.2.2.2.1.1"><csymbol cd="ambiguous" id="alg2.8.8.m1.2.2.2.2.2.1.1.2.1.cmml" xref="alg2.8.8.m1.2.2.2.2.2.1.1">subscript</csymbol><ci id="alg2.8.8.m1.2.2.2.2.2.1.1.2.2.cmml" xref="alg2.8.8.m1.2.2.2.2.2.1.1.2.2">𝑿</ci><ci id="alg2.8.8.m1.2.2.2.2.2.1.1.2.3.cmml" xref="alg2.8.8.m1.2.2.2.2.2.1.1.2.3">𝑘</ci></apply><ci id="alg2.8.8.m1.2.2.2.2.2.1.1.3.cmml" xref="alg2.8.8.m1.2.2.2.2.2.1.1.3">𝑙</ci></apply></apply><apply id="alg2.8.8.m1.3.3.3.3.3.2.cmml" xref="alg2.8.8.m1.3.3.3.3.3.1"><csymbol cd="latexml" id="alg2.8.8.m1.3.3.3.3.3.2.1.cmml" xref="alg2.8.8.m1.3.3.3.3.3.1.2">delimited-∥∥</csymbol><apply id="alg2.8.8.m1.3.3.3.3.3.1.1.cmml" xref="alg2.8.8.m1.3.3.3.3.3.1.1"><csymbol cd="ambiguous" id="alg2.8.8.m1.3.3.3.3.3.1.1.1.cmml" xref="alg2.8.8.m1.3.3.3.3.3.1.1">superscript</csymbol><apply id="alg2.8.8.m1.3.3.3.3.3.1.1.2.cmml" xref="alg2.8.8.m1.3.3.3.3.3.1.1"><csymbol cd="ambiguous" id="alg2.8.8.m1.3.3.3.3.3.1.1.2.1.cmml" xref="alg2.8.8.m1.3.3.3.3.3.1.1">subscript</csymbol><ci id="alg2.8.8.m1.3.3.3.3.3.1.1.2.2.cmml" xref="alg2.8.8.m1.3.3.3.3.3.1.1.2.2">𝑿</ci><ci id="alg2.8.8.m1.3.3.3.3.3.1.1.2.3.cmml" xref="alg2.8.8.m1.3.3.3.3.3.1.1.2.3">𝑘</ci></apply><apply id="alg2.8.8.m1.3.3.3.3.3.1.1.3.cmml" xref="alg2.8.8.m1.3.3.3.3.3.1.1.3"><minus id="alg2.8.8.m1.3.3.3.3.3.1.1.3.1.cmml" xref="alg2.8.8.m1.3.3.3.3.3.1.1.3.1"></minus><ci id="alg2.8.8.m1.3.3.3.3.3.1.1.3.2.cmml" xref="alg2.8.8.m1.3.3.3.3.3.1.1.3.2">𝑙</ci><cn type="integer" id="alg2.8.8.m1.3.3.3.3.3.1.1.3.3.cmml" xref="alg2.8.8.m1.3.3.3.3.3.1.1.3.3">1</cn></apply></apply></apply><apply id="alg2.8.8.m1.4.4.4.4.4.cmml" xref="alg2.8.8.m1.4.4.4.4.4"><csymbol cd="ambiguous" id="alg2.8.8.m1.4.4.4.4.4.1.cmml" xref="alg2.8.8.m1.4.4.4.4.4">subscript</csymbol><ci id="alg2.8.8.m1.4.4.4.4.4.2.cmml" xref="alg2.8.8.m1.4.4.4.4.4.2">Ω</ci><ci id="alg2.8.8.m1.4.4.4.4.4.3.cmml" xref="alg2.8.8.m1.4.4.4.4.4.3">𝑘</ci></apply></list><apply id="alg2.8.8.m1.6.6.6.cmml" xref="alg2.8.8.m1.6.6.6"><times id="alg2.8.8.m1.6.6.6.3.cmml" xref="alg2.8.8.m1.6.6.6.3"></times><ci id="alg2.8.8.m1.6.6.6.4a.cmml" xref="alg2.8.8.m1.6.6.6.4"><mtext id="alg2.8.8.m1.6.6.6.4.cmml" xref="alg2.8.8.m1.6.6.6.4">Merge</mtext></ci><interval closure="open" id="alg2.8.8.m1.6.6.6.2.3.cmml" xref="alg2.8.8.m1.6.6.6.2.2"><apply id="alg2.8.8.m1.5.5.5.1.1.1.cmml" xref="alg2.8.8.m1.5.5.5.1.1.1"><csymbol cd="ambiguous" id="alg2.8.8.m1.5.5.5.1.1.1.1.cmml" xref="alg2.8.8.m1.5.5.5.1.1.1">superscript</csymbol><apply id="alg2.8.8.m1.5.5.5.1.1.1.2.cmml" xref="alg2.8.8.m1.5.5.5.1.1.1"><csymbol cd="ambiguous" id="alg2.8.8.m1.5.5.5.1.1.1.2.1.cmml" xref="alg2.8.8.m1.5.5.5.1.1.1">subscript</csymbol><ci id="alg2.8.8.m1.5.5.5.1.1.1.2.2.cmml" xref="alg2.8.8.m1.5.5.5.1.1.1.2.2">𝑿</ci><ci id="alg2.8.8.m1.5.5.5.1.1.1.2.3.cmml" xref="alg2.8.8.m1.5.5.5.1.1.1.2.3">𝑘</ci></apply><ci id="alg2.8.8.m1.5.5.5.1.1.1.3.cmml" xref="alg2.8.8.m1.5.5.5.1.1.1.3">𝑙</ci></apply><apply id="alg2.8.8.m1.6.6.6.2.2.2.cmml" xref="alg2.8.8.m1.6.6.6.2.2.2"><csymbol cd="ambiguous" id="alg2.8.8.m1.6.6.6.2.2.2.1.cmml" xref="alg2.8.8.m1.6.6.6.2.2.2">superscript</csymbol><apply id="alg2.8.8.m1.6.6.6.2.2.2.2.cmml" xref="alg2.8.8.m1.6.6.6.2.2.2"><csymbol cd="ambiguous" id="alg2.8.8.m1.6.6.6.2.2.2.2.1.cmml" xref="alg2.8.8.m1.6.6.6.2.2.2">subscript</csymbol><ci id="alg2.8.8.m1.6.6.6.2.2.2.2.2.cmml" xref="alg2.8.8.m1.6.6.6.2.2.2.2.2">𝑿</ci><ci id="alg2.8.8.m1.6.6.6.2.2.2.2.3.cmml" xref="alg2.8.8.m1.6.6.6.2.2.2.2.3">𝑘</ci></apply><apply id="alg2.8.8.m1.6.6.6.2.2.2.3.cmml" xref="alg2.8.8.m1.6.6.6.2.2.2.3"><minus id="alg2.8.8.m1.6.6.6.2.2.2.3.1.cmml" xref="alg2.8.8.m1.6.6.6.2.2.2.3.1"></minus><ci id="alg2.8.8.m1.6.6.6.2.2.2.3.2.cmml" xref="alg2.8.8.m1.6.6.6.2.2.2.3.2">𝑙</ci><cn type="integer" id="alg2.8.8.m1.6.6.6.2.2.2.3.3.cmml" xref="alg2.8.8.m1.6.6.6.2.2.2.3.3">1</cn></apply></apply></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.8.8.m1.6c">{\bm{E}}_{k}^{l},\lVert{\bm{X}}_{k}^{l}\rVert,\lVert{\bm{X}}_{k}^{l-1}\rVert,\Omega_{k}\leftarrow\text{Merge}({\bm{X}}_{k}^{l},{\bm{X}}_{k}^{l-1})</annotation></semantics></math> ;
</div>
<div id="alg2.9.9" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">4</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<math id="alg2.9.9.m1.6" class="ltx_Math" alttext="{\bm{E}}_{v}^{l},\lVert{\bm{X}}_{v}^{l}\rVert,\lVert{\bm{X}}_{v}^{l-1}\rVert,\Omega_{v}\leftarrow\text{Merge}({\bm{X}}_{v}^{l},{\bm{X}}_{v}^{l-1})" display="inline"><semantics id="alg2.9.9.m1.6a"><mrow id="alg2.9.9.m1.6.6" xref="alg2.9.9.m1.6.6.cmml"><mrow id="alg2.9.9.m1.4.4.4.4" xref="alg2.9.9.m1.4.4.4.5.cmml"><msubsup id="alg2.9.9.m1.1.1.1.1.1" xref="alg2.9.9.m1.1.1.1.1.1.cmml"><mi id="alg2.9.9.m1.1.1.1.1.1.2.2" xref="alg2.9.9.m1.1.1.1.1.1.2.2.cmml">𝑬</mi><mi id="alg2.9.9.m1.1.1.1.1.1.2.3" xref="alg2.9.9.m1.1.1.1.1.1.2.3.cmml">v</mi><mi id="alg2.9.9.m1.1.1.1.1.1.3" xref="alg2.9.9.m1.1.1.1.1.1.3.cmml">l</mi></msubsup><mo id="alg2.9.9.m1.4.4.4.4.5" xref="alg2.9.9.m1.4.4.4.5.cmml">,</mo><mrow id="alg2.9.9.m1.2.2.2.2.2.1" xref="alg2.9.9.m1.2.2.2.2.2.2.cmml"><mo fence="true" lspace="0em" rspace="0em" id="alg2.9.9.m1.2.2.2.2.2.1.2" xref="alg2.9.9.m1.2.2.2.2.2.2.1.cmml">∥</mo><msubsup id="alg2.9.9.m1.2.2.2.2.2.1.1" xref="alg2.9.9.m1.2.2.2.2.2.1.1.cmml"><mi id="alg2.9.9.m1.2.2.2.2.2.1.1.2.2" xref="alg2.9.9.m1.2.2.2.2.2.1.1.2.2.cmml">𝑿</mi><mi id="alg2.9.9.m1.2.2.2.2.2.1.1.2.3" xref="alg2.9.9.m1.2.2.2.2.2.1.1.2.3.cmml">v</mi><mi id="alg2.9.9.m1.2.2.2.2.2.1.1.3" xref="alg2.9.9.m1.2.2.2.2.2.1.1.3.cmml">l</mi></msubsup><mo fence="true" lspace="0em" rspace="0em" id="alg2.9.9.m1.2.2.2.2.2.1.3" xref="alg2.9.9.m1.2.2.2.2.2.2.1.cmml">∥</mo></mrow><mo id="alg2.9.9.m1.4.4.4.4.6" xref="alg2.9.9.m1.4.4.4.5.cmml">,</mo><mrow id="alg2.9.9.m1.3.3.3.3.3.1" xref="alg2.9.9.m1.3.3.3.3.3.2.cmml"><mo fence="true" lspace="0em" rspace="0em" id="alg2.9.9.m1.3.3.3.3.3.1.2" xref="alg2.9.9.m1.3.3.3.3.3.2.1.cmml">∥</mo><msubsup id="alg2.9.9.m1.3.3.3.3.3.1.1" xref="alg2.9.9.m1.3.3.3.3.3.1.1.cmml"><mi id="alg2.9.9.m1.3.3.3.3.3.1.1.2.2" xref="alg2.9.9.m1.3.3.3.3.3.1.1.2.2.cmml">𝑿</mi><mi id="alg2.9.9.m1.3.3.3.3.3.1.1.2.3" xref="alg2.9.9.m1.3.3.3.3.3.1.1.2.3.cmml">v</mi><mrow id="alg2.9.9.m1.3.3.3.3.3.1.1.3" xref="alg2.9.9.m1.3.3.3.3.3.1.1.3.cmml"><mi id="alg2.9.9.m1.3.3.3.3.3.1.1.3.2" xref="alg2.9.9.m1.3.3.3.3.3.1.1.3.2.cmml">l</mi><mo id="alg2.9.9.m1.3.3.3.3.3.1.1.3.1" xref="alg2.9.9.m1.3.3.3.3.3.1.1.3.1.cmml">−</mo><mn id="alg2.9.9.m1.3.3.3.3.3.1.1.3.3" xref="alg2.9.9.m1.3.3.3.3.3.1.1.3.3.cmml">1</mn></mrow></msubsup><mo fence="true" lspace="0em" rspace="0em" id="alg2.9.9.m1.3.3.3.3.3.1.3" xref="alg2.9.9.m1.3.3.3.3.3.2.1.cmml">∥</mo></mrow><mo id="alg2.9.9.m1.4.4.4.4.7" xref="alg2.9.9.m1.4.4.4.5.cmml">,</mo><msub id="alg2.9.9.m1.4.4.4.4.4" xref="alg2.9.9.m1.4.4.4.4.4.cmml"><mi mathvariant="normal" id="alg2.9.9.m1.4.4.4.4.4.2" xref="alg2.9.9.m1.4.4.4.4.4.2.cmml">Ω</mi><mi id="alg2.9.9.m1.4.4.4.4.4.3" xref="alg2.9.9.m1.4.4.4.4.4.3.cmml">v</mi></msub></mrow><mo stretchy="false" id="alg2.9.9.m1.6.6.7" xref="alg2.9.9.m1.6.6.7.cmml">←</mo><mrow id="alg2.9.9.m1.6.6.6" xref="alg2.9.9.m1.6.6.6.cmml"><mtext id="alg2.9.9.m1.6.6.6.4" xref="alg2.9.9.m1.6.6.6.4a.cmml">Merge</mtext><mo lspace="0em" rspace="0em" id="alg2.9.9.m1.6.6.6.3" xref="alg2.9.9.m1.6.6.6.3.cmml">​</mo><mrow id="alg2.9.9.m1.6.6.6.2.2" xref="alg2.9.9.m1.6.6.6.2.3.cmml"><mo stretchy="false" id="alg2.9.9.m1.6.6.6.2.2.3" xref="alg2.9.9.m1.6.6.6.2.3.cmml">(</mo><msubsup id="alg2.9.9.m1.5.5.5.1.1.1" xref="alg2.9.9.m1.5.5.5.1.1.1.cmml"><mi id="alg2.9.9.m1.5.5.5.1.1.1.2.2" xref="alg2.9.9.m1.5.5.5.1.1.1.2.2.cmml">𝑿</mi><mi id="alg2.9.9.m1.5.5.5.1.1.1.2.3" xref="alg2.9.9.m1.5.5.5.1.1.1.2.3.cmml">v</mi><mi id="alg2.9.9.m1.5.5.5.1.1.1.3" xref="alg2.9.9.m1.5.5.5.1.1.1.3.cmml">l</mi></msubsup><mo id="alg2.9.9.m1.6.6.6.2.2.4" xref="alg2.9.9.m1.6.6.6.2.3.cmml">,</mo><msubsup id="alg2.9.9.m1.6.6.6.2.2.2" xref="alg2.9.9.m1.6.6.6.2.2.2.cmml"><mi id="alg2.9.9.m1.6.6.6.2.2.2.2.2" xref="alg2.9.9.m1.6.6.6.2.2.2.2.2.cmml">𝑿</mi><mi id="alg2.9.9.m1.6.6.6.2.2.2.2.3" xref="alg2.9.9.m1.6.6.6.2.2.2.2.3.cmml">v</mi><mrow id="alg2.9.9.m1.6.6.6.2.2.2.3" xref="alg2.9.9.m1.6.6.6.2.2.2.3.cmml"><mi id="alg2.9.9.m1.6.6.6.2.2.2.3.2" xref="alg2.9.9.m1.6.6.6.2.2.2.3.2.cmml">l</mi><mo id="alg2.9.9.m1.6.6.6.2.2.2.3.1" xref="alg2.9.9.m1.6.6.6.2.2.2.3.1.cmml">−</mo><mn id="alg2.9.9.m1.6.6.6.2.2.2.3.3" xref="alg2.9.9.m1.6.6.6.2.2.2.3.3.cmml">1</mn></mrow></msubsup><mo stretchy="false" id="alg2.9.9.m1.6.6.6.2.2.5" xref="alg2.9.9.m1.6.6.6.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.9.9.m1.6b"><apply id="alg2.9.9.m1.6.6.cmml" xref="alg2.9.9.m1.6.6"><ci id="alg2.9.9.m1.6.6.7.cmml" xref="alg2.9.9.m1.6.6.7">←</ci><list id="alg2.9.9.m1.4.4.4.5.cmml" xref="alg2.9.9.m1.4.4.4.4"><apply id="alg2.9.9.m1.1.1.1.1.1.cmml" xref="alg2.9.9.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.9.9.m1.1.1.1.1.1.1.cmml" xref="alg2.9.9.m1.1.1.1.1.1">superscript</csymbol><apply id="alg2.9.9.m1.1.1.1.1.1.2.cmml" xref="alg2.9.9.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.9.9.m1.1.1.1.1.1.2.1.cmml" xref="alg2.9.9.m1.1.1.1.1.1">subscript</csymbol><ci id="alg2.9.9.m1.1.1.1.1.1.2.2.cmml" xref="alg2.9.9.m1.1.1.1.1.1.2.2">𝑬</ci><ci id="alg2.9.9.m1.1.1.1.1.1.2.3.cmml" xref="alg2.9.9.m1.1.1.1.1.1.2.3">𝑣</ci></apply><ci id="alg2.9.9.m1.1.1.1.1.1.3.cmml" xref="alg2.9.9.m1.1.1.1.1.1.3">𝑙</ci></apply><apply id="alg2.9.9.m1.2.2.2.2.2.2.cmml" xref="alg2.9.9.m1.2.2.2.2.2.1"><csymbol cd="latexml" id="alg2.9.9.m1.2.2.2.2.2.2.1.cmml" xref="alg2.9.9.m1.2.2.2.2.2.1.2">delimited-∥∥</csymbol><apply id="alg2.9.9.m1.2.2.2.2.2.1.1.cmml" xref="alg2.9.9.m1.2.2.2.2.2.1.1"><csymbol cd="ambiguous" id="alg2.9.9.m1.2.2.2.2.2.1.1.1.cmml" xref="alg2.9.9.m1.2.2.2.2.2.1.1">superscript</csymbol><apply id="alg2.9.9.m1.2.2.2.2.2.1.1.2.cmml" xref="alg2.9.9.m1.2.2.2.2.2.1.1"><csymbol cd="ambiguous" id="alg2.9.9.m1.2.2.2.2.2.1.1.2.1.cmml" xref="alg2.9.9.m1.2.2.2.2.2.1.1">subscript</csymbol><ci id="alg2.9.9.m1.2.2.2.2.2.1.1.2.2.cmml" xref="alg2.9.9.m1.2.2.2.2.2.1.1.2.2">𝑿</ci><ci id="alg2.9.9.m1.2.2.2.2.2.1.1.2.3.cmml" xref="alg2.9.9.m1.2.2.2.2.2.1.1.2.3">𝑣</ci></apply><ci id="alg2.9.9.m1.2.2.2.2.2.1.1.3.cmml" xref="alg2.9.9.m1.2.2.2.2.2.1.1.3">𝑙</ci></apply></apply><apply id="alg2.9.9.m1.3.3.3.3.3.2.cmml" xref="alg2.9.9.m1.3.3.3.3.3.1"><csymbol cd="latexml" id="alg2.9.9.m1.3.3.3.3.3.2.1.cmml" xref="alg2.9.9.m1.3.3.3.3.3.1.2">delimited-∥∥</csymbol><apply id="alg2.9.9.m1.3.3.3.3.3.1.1.cmml" xref="alg2.9.9.m1.3.3.3.3.3.1.1"><csymbol cd="ambiguous" id="alg2.9.9.m1.3.3.3.3.3.1.1.1.cmml" xref="alg2.9.9.m1.3.3.3.3.3.1.1">superscript</csymbol><apply id="alg2.9.9.m1.3.3.3.3.3.1.1.2.cmml" xref="alg2.9.9.m1.3.3.3.3.3.1.1"><csymbol cd="ambiguous" id="alg2.9.9.m1.3.3.3.3.3.1.1.2.1.cmml" xref="alg2.9.9.m1.3.3.3.3.3.1.1">subscript</csymbol><ci id="alg2.9.9.m1.3.3.3.3.3.1.1.2.2.cmml" xref="alg2.9.9.m1.3.3.3.3.3.1.1.2.2">𝑿</ci><ci id="alg2.9.9.m1.3.3.3.3.3.1.1.2.3.cmml" xref="alg2.9.9.m1.3.3.3.3.3.1.1.2.3">𝑣</ci></apply><apply id="alg2.9.9.m1.3.3.3.3.3.1.1.3.cmml" xref="alg2.9.9.m1.3.3.3.3.3.1.1.3"><minus id="alg2.9.9.m1.3.3.3.3.3.1.1.3.1.cmml" xref="alg2.9.9.m1.3.3.3.3.3.1.1.3.1"></minus><ci id="alg2.9.9.m1.3.3.3.3.3.1.1.3.2.cmml" xref="alg2.9.9.m1.3.3.3.3.3.1.1.3.2">𝑙</ci><cn type="integer" id="alg2.9.9.m1.3.3.3.3.3.1.1.3.3.cmml" xref="alg2.9.9.m1.3.3.3.3.3.1.1.3.3">1</cn></apply></apply></apply><apply id="alg2.9.9.m1.4.4.4.4.4.cmml" xref="alg2.9.9.m1.4.4.4.4.4"><csymbol cd="ambiguous" id="alg2.9.9.m1.4.4.4.4.4.1.cmml" xref="alg2.9.9.m1.4.4.4.4.4">subscript</csymbol><ci id="alg2.9.9.m1.4.4.4.4.4.2.cmml" xref="alg2.9.9.m1.4.4.4.4.4.2">Ω</ci><ci id="alg2.9.9.m1.4.4.4.4.4.3.cmml" xref="alg2.9.9.m1.4.4.4.4.4.3">𝑣</ci></apply></list><apply id="alg2.9.9.m1.6.6.6.cmml" xref="alg2.9.9.m1.6.6.6"><times id="alg2.9.9.m1.6.6.6.3.cmml" xref="alg2.9.9.m1.6.6.6.3"></times><ci id="alg2.9.9.m1.6.6.6.4a.cmml" xref="alg2.9.9.m1.6.6.6.4"><mtext id="alg2.9.9.m1.6.6.6.4.cmml" xref="alg2.9.9.m1.6.6.6.4">Merge</mtext></ci><interval closure="open" id="alg2.9.9.m1.6.6.6.2.3.cmml" xref="alg2.9.9.m1.6.6.6.2.2"><apply id="alg2.9.9.m1.5.5.5.1.1.1.cmml" xref="alg2.9.9.m1.5.5.5.1.1.1"><csymbol cd="ambiguous" id="alg2.9.9.m1.5.5.5.1.1.1.1.cmml" xref="alg2.9.9.m1.5.5.5.1.1.1">superscript</csymbol><apply id="alg2.9.9.m1.5.5.5.1.1.1.2.cmml" xref="alg2.9.9.m1.5.5.5.1.1.1"><csymbol cd="ambiguous" id="alg2.9.9.m1.5.5.5.1.1.1.2.1.cmml" xref="alg2.9.9.m1.5.5.5.1.1.1">subscript</csymbol><ci id="alg2.9.9.m1.5.5.5.1.1.1.2.2.cmml" xref="alg2.9.9.m1.5.5.5.1.1.1.2.2">𝑿</ci><ci id="alg2.9.9.m1.5.5.5.1.1.1.2.3.cmml" xref="alg2.9.9.m1.5.5.5.1.1.1.2.3">𝑣</ci></apply><ci id="alg2.9.9.m1.5.5.5.1.1.1.3.cmml" xref="alg2.9.9.m1.5.5.5.1.1.1.3">𝑙</ci></apply><apply id="alg2.9.9.m1.6.6.6.2.2.2.cmml" xref="alg2.9.9.m1.6.6.6.2.2.2"><csymbol cd="ambiguous" id="alg2.9.9.m1.6.6.6.2.2.2.1.cmml" xref="alg2.9.9.m1.6.6.6.2.2.2">superscript</csymbol><apply id="alg2.9.9.m1.6.6.6.2.2.2.2.cmml" xref="alg2.9.9.m1.6.6.6.2.2.2"><csymbol cd="ambiguous" id="alg2.9.9.m1.6.6.6.2.2.2.2.1.cmml" xref="alg2.9.9.m1.6.6.6.2.2.2">subscript</csymbol><ci id="alg2.9.9.m1.6.6.6.2.2.2.2.2.cmml" xref="alg2.9.9.m1.6.6.6.2.2.2.2.2">𝑿</ci><ci id="alg2.9.9.m1.6.6.6.2.2.2.2.3.cmml" xref="alg2.9.9.m1.6.6.6.2.2.2.2.3">𝑣</ci></apply><apply id="alg2.9.9.m1.6.6.6.2.2.2.3.cmml" xref="alg2.9.9.m1.6.6.6.2.2.2.3"><minus id="alg2.9.9.m1.6.6.6.2.2.2.3.1.cmml" xref="alg2.9.9.m1.6.6.6.2.2.2.3.1"></minus><ci id="alg2.9.9.m1.6.6.6.2.2.2.3.2.cmml" xref="alg2.9.9.m1.6.6.6.2.2.2.3.2">𝑙</ci><cn type="integer" id="alg2.9.9.m1.6.6.6.2.2.2.3.3.cmml" xref="alg2.9.9.m1.6.6.6.2.2.2.3.3">1</cn></apply></apply></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.9.9.m1.6c">{\bm{E}}_{v}^{l},\lVert{\bm{X}}_{v}^{l}\rVert,\lVert{\bm{X}}_{v}^{l-1}\rVert,\Omega_{v}\leftarrow\text{Merge}({\bm{X}}_{v}^{l},{\bm{X}}_{v}^{l-1})</annotation></semantics></math> ;
</div>
<div id="alg2.10.10" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">5</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<math id="alg2.10.10.m1.9" class="ltx_Math" alttext="\underbrace{{\bm{E}}_{k,v}^{l}\in\mathbb{R}^{t_{\text{prompt}}\times d}}_{\text{compression output}},\underbrace{\lVert{\bm{X}}_{k,v}^{l,l-1}\rVert\in\mathbb{R}^{4\times t_{\text{prompt}}\times 1}}_{\text{norms for rescaling}}" display="inline"><semantics id="alg2.10.10.m1.9a"><mrow id="alg2.10.10.m1.9.9.2" xref="alg2.10.10.m1.9.9.3.cmml"><munder id="alg2.10.10.m1.8.8.1.1" xref="alg2.10.10.m1.8.8.1.1.cmml"><munder accentunder="true" id="alg2.10.10.m1.2.2" xref="alg2.10.10.m1.2.2.cmml"><mrow id="alg2.10.10.m1.2.2.2" xref="alg2.10.10.m1.2.2.2.cmml"><msubsup id="alg2.10.10.m1.2.2.2.4" xref="alg2.10.10.m1.2.2.2.4.cmml"><mi id="alg2.10.10.m1.2.2.2.4.2.2" xref="alg2.10.10.m1.2.2.2.4.2.2.cmml">𝑬</mi><mrow id="alg2.10.10.m1.2.2.2.2.2.4" xref="alg2.10.10.m1.2.2.2.2.2.3.cmml"><mi id="alg2.10.10.m1.1.1.1.1.1.1" xref="alg2.10.10.m1.1.1.1.1.1.1.cmml">k</mi><mo id="alg2.10.10.m1.2.2.2.2.2.4.1" xref="alg2.10.10.m1.2.2.2.2.2.3.cmml">,</mo><mi id="alg2.10.10.m1.2.2.2.2.2.2" xref="alg2.10.10.m1.2.2.2.2.2.2.cmml">v</mi></mrow><mi id="alg2.10.10.m1.2.2.2.4.3" xref="alg2.10.10.m1.2.2.2.4.3.cmml">l</mi></msubsup><mo id="alg2.10.10.m1.2.2.2.3" xref="alg2.10.10.m1.2.2.2.3.cmml">∈</mo><msup id="alg2.10.10.m1.2.2.2.5" xref="alg2.10.10.m1.2.2.2.5.cmml"><mi id="alg2.10.10.m1.2.2.2.5.2" xref="alg2.10.10.m1.2.2.2.5.2.cmml">ℝ</mi><mrow id="alg2.10.10.m1.2.2.2.5.3" xref="alg2.10.10.m1.2.2.2.5.3.cmml"><msub id="alg2.10.10.m1.2.2.2.5.3.2" xref="alg2.10.10.m1.2.2.2.5.3.2.cmml"><mi id="alg2.10.10.m1.2.2.2.5.3.2.2" xref="alg2.10.10.m1.2.2.2.5.3.2.2.cmml">t</mi><mtext id="alg2.10.10.m1.2.2.2.5.3.2.3" xref="alg2.10.10.m1.2.2.2.5.3.2.3a.cmml">prompt</mtext></msub><mo lspace="0.222em" rspace="0.222em" id="alg2.10.10.m1.2.2.2.5.3.1" xref="alg2.10.10.m1.2.2.2.5.3.1.cmml">×</mo><mi id="alg2.10.10.m1.2.2.2.5.3.3" xref="alg2.10.10.m1.2.2.2.5.3.3.cmml">d</mi></mrow></msup></mrow><mo id="alg2.10.10.m1.2.2.3" xref="alg2.10.10.m1.2.2.3.cmml">⏟</mo></munder><mtext id="alg2.10.10.m1.8.8.1.1.2" xref="alg2.10.10.m1.8.8.1.1.2a.cmml">compression output</mtext></munder><mo id="alg2.10.10.m1.9.9.2.3" xref="alg2.10.10.m1.9.9.3.cmml">,</mo><munder id="alg2.10.10.m1.9.9.2.2" xref="alg2.10.10.m1.9.9.2.2.cmml"><munder accentunder="true" id="alg2.10.10.m1.7.7" xref="alg2.10.10.m1.7.7.cmml"><mrow id="alg2.10.10.m1.7.7.5" xref="alg2.10.10.m1.7.7.5.cmml"><mrow id="alg2.10.10.m1.7.7.5.5.1" xref="alg2.10.10.m1.7.7.5.5.2.cmml"><mo fence="true" lspace="0em" rspace="0em" id="alg2.10.10.m1.7.7.5.5.1.2" xref="alg2.10.10.m1.7.7.5.5.2.1.cmml">∥</mo><msubsup id="alg2.10.10.m1.7.7.5.5.1.1" xref="alg2.10.10.m1.7.7.5.5.1.1.cmml"><mi id="alg2.10.10.m1.7.7.5.5.1.1.2.2" xref="alg2.10.10.m1.7.7.5.5.1.1.2.2.cmml">𝑿</mi><mrow id="alg2.10.10.m1.4.4.2.2.2.4" xref="alg2.10.10.m1.4.4.2.2.2.3.cmml"><mi id="alg2.10.10.m1.3.3.1.1.1.1" xref="alg2.10.10.m1.3.3.1.1.1.1.cmml">k</mi><mo id="alg2.10.10.m1.4.4.2.2.2.4.1" xref="alg2.10.10.m1.4.4.2.2.2.3.cmml">,</mo><mi id="alg2.10.10.m1.4.4.2.2.2.2" xref="alg2.10.10.m1.4.4.2.2.2.2.cmml">v</mi></mrow><mrow id="alg2.10.10.m1.6.6.4.4.2.2" xref="alg2.10.10.m1.6.6.4.4.2.3.cmml"><mi id="alg2.10.10.m1.5.5.3.3.1.1" xref="alg2.10.10.m1.5.5.3.3.1.1.cmml">l</mi><mo id="alg2.10.10.m1.6.6.4.4.2.2.2" xref="alg2.10.10.m1.6.6.4.4.2.3.cmml">,</mo><mrow id="alg2.10.10.m1.6.6.4.4.2.2.1" xref="alg2.10.10.m1.6.6.4.4.2.2.1.cmml"><mi id="alg2.10.10.m1.6.6.4.4.2.2.1.2" xref="alg2.10.10.m1.6.6.4.4.2.2.1.2.cmml">l</mi><mo id="alg2.10.10.m1.6.6.4.4.2.2.1.1" xref="alg2.10.10.m1.6.6.4.4.2.2.1.1.cmml">−</mo><mn id="alg2.10.10.m1.6.6.4.4.2.2.1.3" xref="alg2.10.10.m1.6.6.4.4.2.2.1.3.cmml">1</mn></mrow></mrow></msubsup><mo fence="true" lspace="0em" rspace="0.1389em" id="alg2.10.10.m1.7.7.5.5.1.3" xref="alg2.10.10.m1.7.7.5.5.2.1.cmml">∥</mo></mrow><mo lspace="0.1389em" id="alg2.10.10.m1.7.7.5.6" xref="alg2.10.10.m1.7.7.5.6.cmml">∈</mo><msup id="alg2.10.10.m1.7.7.5.7" xref="alg2.10.10.m1.7.7.5.7.cmml"><mi id="alg2.10.10.m1.7.7.5.7.2" xref="alg2.10.10.m1.7.7.5.7.2.cmml">ℝ</mi><mrow id="alg2.10.10.m1.7.7.5.7.3" xref="alg2.10.10.m1.7.7.5.7.3.cmml"><mn id="alg2.10.10.m1.7.7.5.7.3.2" xref="alg2.10.10.m1.7.7.5.7.3.2.cmml">4</mn><mo lspace="0.222em" rspace="0.222em" id="alg2.10.10.m1.7.7.5.7.3.1" xref="alg2.10.10.m1.7.7.5.7.3.1.cmml">×</mo><msub id="alg2.10.10.m1.7.7.5.7.3.3" xref="alg2.10.10.m1.7.7.5.7.3.3.cmml"><mi id="alg2.10.10.m1.7.7.5.7.3.3.2" xref="alg2.10.10.m1.7.7.5.7.3.3.2.cmml">t</mi><mtext id="alg2.10.10.m1.7.7.5.7.3.3.3" xref="alg2.10.10.m1.7.7.5.7.3.3.3a.cmml">prompt</mtext></msub><mo lspace="0.222em" rspace="0.222em" id="alg2.10.10.m1.7.7.5.7.3.1a" xref="alg2.10.10.m1.7.7.5.7.3.1.cmml">×</mo><mn id="alg2.10.10.m1.7.7.5.7.3.4" xref="alg2.10.10.m1.7.7.5.7.3.4.cmml">1</mn></mrow></msup></mrow><mo id="alg2.10.10.m1.7.7.6" xref="alg2.10.10.m1.7.7.6.cmml">⏟</mo></munder><mtext id="alg2.10.10.m1.9.9.2.2.2" xref="alg2.10.10.m1.9.9.2.2.2a.cmml">norms for rescaling</mtext></munder></mrow><annotation-xml encoding="MathML-Content" id="alg2.10.10.m1.9b"><list id="alg2.10.10.m1.9.9.3.cmml" xref="alg2.10.10.m1.9.9.2"><apply id="alg2.10.10.m1.8.8.1.1.cmml" xref="alg2.10.10.m1.8.8.1.1"><csymbol cd="ambiguous" id="alg2.10.10.m1.8.8.1.1.1.cmml" xref="alg2.10.10.m1.8.8.1.1">subscript</csymbol><apply id="alg2.10.10.m1.2.2.cmml" xref="alg2.10.10.m1.2.2"><ci id="alg2.10.10.m1.2.2.3.cmml" xref="alg2.10.10.m1.2.2.3">⏟</ci><apply id="alg2.10.10.m1.2.2.2.cmml" xref="alg2.10.10.m1.2.2.2"><in id="alg2.10.10.m1.2.2.2.3.cmml" xref="alg2.10.10.m1.2.2.2.3"></in><apply id="alg2.10.10.m1.2.2.2.4.cmml" xref="alg2.10.10.m1.2.2.2.4"><csymbol cd="ambiguous" id="alg2.10.10.m1.2.2.2.4.1.cmml" xref="alg2.10.10.m1.2.2.2.4">superscript</csymbol><apply id="alg2.10.10.m1.2.2.2.4.2.cmml" xref="alg2.10.10.m1.2.2.2.4"><csymbol cd="ambiguous" id="alg2.10.10.m1.2.2.2.4.2.1.cmml" xref="alg2.10.10.m1.2.2.2.4">subscript</csymbol><ci id="alg2.10.10.m1.2.2.2.4.2.2.cmml" xref="alg2.10.10.m1.2.2.2.4.2.2">𝑬</ci><list id="alg2.10.10.m1.2.2.2.2.2.3.cmml" xref="alg2.10.10.m1.2.2.2.2.2.4"><ci id="alg2.10.10.m1.1.1.1.1.1.1.cmml" xref="alg2.10.10.m1.1.1.1.1.1.1">𝑘</ci><ci id="alg2.10.10.m1.2.2.2.2.2.2.cmml" xref="alg2.10.10.m1.2.2.2.2.2.2">𝑣</ci></list></apply><ci id="alg2.10.10.m1.2.2.2.4.3.cmml" xref="alg2.10.10.m1.2.2.2.4.3">𝑙</ci></apply><apply id="alg2.10.10.m1.2.2.2.5.cmml" xref="alg2.10.10.m1.2.2.2.5"><csymbol cd="ambiguous" id="alg2.10.10.m1.2.2.2.5.1.cmml" xref="alg2.10.10.m1.2.2.2.5">superscript</csymbol><ci id="alg2.10.10.m1.2.2.2.5.2.cmml" xref="alg2.10.10.m1.2.2.2.5.2">ℝ</ci><apply id="alg2.10.10.m1.2.2.2.5.3.cmml" xref="alg2.10.10.m1.2.2.2.5.3"><times id="alg2.10.10.m1.2.2.2.5.3.1.cmml" xref="alg2.10.10.m1.2.2.2.5.3.1"></times><apply id="alg2.10.10.m1.2.2.2.5.3.2.cmml" xref="alg2.10.10.m1.2.2.2.5.3.2"><csymbol cd="ambiguous" id="alg2.10.10.m1.2.2.2.5.3.2.1.cmml" xref="alg2.10.10.m1.2.2.2.5.3.2">subscript</csymbol><ci id="alg2.10.10.m1.2.2.2.5.3.2.2.cmml" xref="alg2.10.10.m1.2.2.2.5.3.2.2">𝑡</ci><ci id="alg2.10.10.m1.2.2.2.5.3.2.3a.cmml" xref="alg2.10.10.m1.2.2.2.5.3.2.3"><mtext mathsize="50%" id="alg2.10.10.m1.2.2.2.5.3.2.3.cmml" xref="alg2.10.10.m1.2.2.2.5.3.2.3">prompt</mtext></ci></apply><ci id="alg2.10.10.m1.2.2.2.5.3.3.cmml" xref="alg2.10.10.m1.2.2.2.5.3.3">𝑑</ci></apply></apply></apply></apply><ci id="alg2.10.10.m1.8.8.1.1.2a.cmml" xref="alg2.10.10.m1.8.8.1.1.2"><mtext mathsize="70%" id="alg2.10.10.m1.8.8.1.1.2.cmml" xref="alg2.10.10.m1.8.8.1.1.2">compression output</mtext></ci></apply><apply id="alg2.10.10.m1.9.9.2.2.cmml" xref="alg2.10.10.m1.9.9.2.2"><csymbol cd="ambiguous" id="alg2.10.10.m1.9.9.2.2.1.cmml" xref="alg2.10.10.m1.9.9.2.2">subscript</csymbol><apply id="alg2.10.10.m1.7.7.cmml" xref="alg2.10.10.m1.7.7"><ci id="alg2.10.10.m1.7.7.6.cmml" xref="alg2.10.10.m1.7.7.6">⏟</ci><apply id="alg2.10.10.m1.7.7.5.cmml" xref="alg2.10.10.m1.7.7.5"><in id="alg2.10.10.m1.7.7.5.6.cmml" xref="alg2.10.10.m1.7.7.5.6"></in><apply id="alg2.10.10.m1.7.7.5.5.2.cmml" xref="alg2.10.10.m1.7.7.5.5.1"><csymbol cd="latexml" id="alg2.10.10.m1.7.7.5.5.2.1.cmml" xref="alg2.10.10.m1.7.7.5.5.1.2">delimited-∥∥</csymbol><apply id="alg2.10.10.m1.7.7.5.5.1.1.cmml" xref="alg2.10.10.m1.7.7.5.5.1.1"><csymbol cd="ambiguous" id="alg2.10.10.m1.7.7.5.5.1.1.1.cmml" xref="alg2.10.10.m1.7.7.5.5.1.1">superscript</csymbol><apply id="alg2.10.10.m1.7.7.5.5.1.1.2.cmml" xref="alg2.10.10.m1.7.7.5.5.1.1"><csymbol cd="ambiguous" id="alg2.10.10.m1.7.7.5.5.1.1.2.1.cmml" xref="alg2.10.10.m1.7.7.5.5.1.1">subscript</csymbol><ci id="alg2.10.10.m1.7.7.5.5.1.1.2.2.cmml" xref="alg2.10.10.m1.7.7.5.5.1.1.2.2">𝑿</ci><list id="alg2.10.10.m1.4.4.2.2.2.3.cmml" xref="alg2.10.10.m1.4.4.2.2.2.4"><ci id="alg2.10.10.m1.3.3.1.1.1.1.cmml" xref="alg2.10.10.m1.3.3.1.1.1.1">𝑘</ci><ci id="alg2.10.10.m1.4.4.2.2.2.2.cmml" xref="alg2.10.10.m1.4.4.2.2.2.2">𝑣</ci></list></apply><list id="alg2.10.10.m1.6.6.4.4.2.3.cmml" xref="alg2.10.10.m1.6.6.4.4.2.2"><ci id="alg2.10.10.m1.5.5.3.3.1.1.cmml" xref="alg2.10.10.m1.5.5.3.3.1.1">𝑙</ci><apply id="alg2.10.10.m1.6.6.4.4.2.2.1.cmml" xref="alg2.10.10.m1.6.6.4.4.2.2.1"><minus id="alg2.10.10.m1.6.6.4.4.2.2.1.1.cmml" xref="alg2.10.10.m1.6.6.4.4.2.2.1.1"></minus><ci id="alg2.10.10.m1.6.6.4.4.2.2.1.2.cmml" xref="alg2.10.10.m1.6.6.4.4.2.2.1.2">𝑙</ci><cn type="integer" id="alg2.10.10.m1.6.6.4.4.2.2.1.3.cmml" xref="alg2.10.10.m1.6.6.4.4.2.2.1.3">1</cn></apply></list></apply></apply><apply id="alg2.10.10.m1.7.7.5.7.cmml" xref="alg2.10.10.m1.7.7.5.7"><csymbol cd="ambiguous" id="alg2.10.10.m1.7.7.5.7.1.cmml" xref="alg2.10.10.m1.7.7.5.7">superscript</csymbol><ci id="alg2.10.10.m1.7.7.5.7.2.cmml" xref="alg2.10.10.m1.7.7.5.7.2">ℝ</ci><apply id="alg2.10.10.m1.7.7.5.7.3.cmml" xref="alg2.10.10.m1.7.7.5.7.3"><times id="alg2.10.10.m1.7.7.5.7.3.1.cmml" xref="alg2.10.10.m1.7.7.5.7.3.1"></times><cn type="integer" id="alg2.10.10.m1.7.7.5.7.3.2.cmml" xref="alg2.10.10.m1.7.7.5.7.3.2">4</cn><apply id="alg2.10.10.m1.7.7.5.7.3.3.cmml" xref="alg2.10.10.m1.7.7.5.7.3.3"><csymbol cd="ambiguous" id="alg2.10.10.m1.7.7.5.7.3.3.1.cmml" xref="alg2.10.10.m1.7.7.5.7.3.3">subscript</csymbol><ci id="alg2.10.10.m1.7.7.5.7.3.3.2.cmml" xref="alg2.10.10.m1.7.7.5.7.3.3.2">𝑡</ci><ci id="alg2.10.10.m1.7.7.5.7.3.3.3a.cmml" xref="alg2.10.10.m1.7.7.5.7.3.3.3"><mtext mathsize="50%" id="alg2.10.10.m1.7.7.5.7.3.3.3.cmml" xref="alg2.10.10.m1.7.7.5.7.3.3.3">prompt</mtext></ci></apply><cn type="integer" id="alg2.10.10.m1.7.7.5.7.3.4.cmml" xref="alg2.10.10.m1.7.7.5.7.3.4">1</cn></apply></apply></apply></apply><ci id="alg2.10.10.m1.9.9.2.2.2a.cmml" xref="alg2.10.10.m1.9.9.2.2.2"><mtext mathsize="70%" id="alg2.10.10.m1.9.9.2.2.2.cmml" xref="alg2.10.10.m1.9.9.2.2.2">norms for rescaling</mtext></ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="alg2.10.10.m1.9c">\underbrace{{\bm{E}}_{k,v}^{l}\in\mathbb{R}^{t_{\text{prompt}}\times d}}_{\text{compression output}},\underbrace{\lVert{\bm{X}}_{k,v}^{l,l-1}\rVert\in\mathbb{R}^{4\times t_{\text{prompt}}\times 1}}_{\text{norms for rescaling}}</annotation></semantics></math> ;
</div>
<div id="alg2.11.11" class="ltx_listingline">  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<math id="alg2.11.11.m1.6" class="ltx_Math" alttext="d({\bm{X}}^{l},{\bm{X}}^{l-1})_{k,v}=\frac{1}{\pi}\cdot\Omega_{k,v}" display="inline"><semantics id="alg2.11.11.m1.6a"><mrow id="alg2.11.11.m1.6.6" xref="alg2.11.11.m1.6.6.cmml"><mrow id="alg2.11.11.m1.6.6.2" xref="alg2.11.11.m1.6.6.2.cmml"><mi id="alg2.11.11.m1.6.6.2.4" xref="alg2.11.11.m1.6.6.2.4.cmml">d</mi><mo lspace="0em" rspace="0em" id="alg2.11.11.m1.6.6.2.3" xref="alg2.11.11.m1.6.6.2.3.cmml">​</mo><msub id="alg2.11.11.m1.6.6.2.2" xref="alg2.11.11.m1.6.6.2.2.cmml"><mrow id="alg2.11.11.m1.6.6.2.2.2.2" xref="alg2.11.11.m1.6.6.2.2.2.3.cmml"><mo stretchy="false" id="alg2.11.11.m1.6.6.2.2.2.2.3" xref="alg2.11.11.m1.6.6.2.2.2.3.cmml">(</mo><msup id="alg2.11.11.m1.5.5.1.1.1.1.1" xref="alg2.11.11.m1.5.5.1.1.1.1.1.cmml"><mi id="alg2.11.11.m1.5.5.1.1.1.1.1.2" xref="alg2.11.11.m1.5.5.1.1.1.1.1.2.cmml">𝑿</mi><mi id="alg2.11.11.m1.5.5.1.1.1.1.1.3" xref="alg2.11.11.m1.5.5.1.1.1.1.1.3.cmml">l</mi></msup><mo id="alg2.11.11.m1.6.6.2.2.2.2.4" xref="alg2.11.11.m1.6.6.2.2.2.3.cmml">,</mo><msup id="alg2.11.11.m1.6.6.2.2.2.2.2" xref="alg2.11.11.m1.6.6.2.2.2.2.2.cmml"><mi id="alg2.11.11.m1.6.6.2.2.2.2.2.2" xref="alg2.11.11.m1.6.6.2.2.2.2.2.2.cmml">𝑿</mi><mrow id="alg2.11.11.m1.6.6.2.2.2.2.2.3" xref="alg2.11.11.m1.6.6.2.2.2.2.2.3.cmml"><mi id="alg2.11.11.m1.6.6.2.2.2.2.2.3.2" xref="alg2.11.11.m1.6.6.2.2.2.2.2.3.2.cmml">l</mi><mo id="alg2.11.11.m1.6.6.2.2.2.2.2.3.1" xref="alg2.11.11.m1.6.6.2.2.2.2.2.3.1.cmml">−</mo><mn id="alg2.11.11.m1.6.6.2.2.2.2.2.3.3" xref="alg2.11.11.m1.6.6.2.2.2.2.2.3.3.cmml">1</mn></mrow></msup><mo stretchy="false" id="alg2.11.11.m1.6.6.2.2.2.2.5" xref="alg2.11.11.m1.6.6.2.2.2.3.cmml">)</mo></mrow><mrow id="alg2.11.11.m1.2.2.2.4" xref="alg2.11.11.m1.2.2.2.3.cmml"><mi id="alg2.11.11.m1.1.1.1.1" xref="alg2.11.11.m1.1.1.1.1.cmml">k</mi><mo id="alg2.11.11.m1.2.2.2.4.1" xref="alg2.11.11.m1.2.2.2.3.cmml">,</mo><mi id="alg2.11.11.m1.2.2.2.2" xref="alg2.11.11.m1.2.2.2.2.cmml">v</mi></mrow></msub></mrow><mo id="alg2.11.11.m1.6.6.3" xref="alg2.11.11.m1.6.6.3.cmml">=</mo><mrow id="alg2.11.11.m1.6.6.4" xref="alg2.11.11.m1.6.6.4.cmml"><mfrac id="alg2.11.11.m1.6.6.4.2" xref="alg2.11.11.m1.6.6.4.2.cmml"><mn id="alg2.11.11.m1.6.6.4.2.2" xref="alg2.11.11.m1.6.6.4.2.2.cmml">1</mn><mi id="alg2.11.11.m1.6.6.4.2.3" xref="alg2.11.11.m1.6.6.4.2.3.cmml">π</mi></mfrac><mo lspace="0.222em" rspace="0.222em" id="alg2.11.11.m1.6.6.4.1" xref="alg2.11.11.m1.6.6.4.1.cmml">⋅</mo><msub id="alg2.11.11.m1.6.6.4.3" xref="alg2.11.11.m1.6.6.4.3.cmml"><mi mathvariant="normal" id="alg2.11.11.m1.6.6.4.3.2" xref="alg2.11.11.m1.6.6.4.3.2.cmml">Ω</mi><mrow id="alg2.11.11.m1.4.4.2.4" xref="alg2.11.11.m1.4.4.2.3.cmml"><mi id="alg2.11.11.m1.3.3.1.1" xref="alg2.11.11.m1.3.3.1.1.cmml">k</mi><mo id="alg2.11.11.m1.4.4.2.4.1" xref="alg2.11.11.m1.4.4.2.3.cmml">,</mo><mi id="alg2.11.11.m1.4.4.2.2" xref="alg2.11.11.m1.4.4.2.2.cmml">v</mi></mrow></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.11.11.m1.6b"><apply id="alg2.11.11.m1.6.6.cmml" xref="alg2.11.11.m1.6.6"><eq id="alg2.11.11.m1.6.6.3.cmml" xref="alg2.11.11.m1.6.6.3"></eq><apply id="alg2.11.11.m1.6.6.2.cmml" xref="alg2.11.11.m1.6.6.2"><times id="alg2.11.11.m1.6.6.2.3.cmml" xref="alg2.11.11.m1.6.6.2.3"></times><ci id="alg2.11.11.m1.6.6.2.4.cmml" xref="alg2.11.11.m1.6.6.2.4">𝑑</ci><apply id="alg2.11.11.m1.6.6.2.2.cmml" xref="alg2.11.11.m1.6.6.2.2"><csymbol cd="ambiguous" id="alg2.11.11.m1.6.6.2.2.3.cmml" xref="alg2.11.11.m1.6.6.2.2">subscript</csymbol><interval closure="open" id="alg2.11.11.m1.6.6.2.2.2.3.cmml" xref="alg2.11.11.m1.6.6.2.2.2.2"><apply id="alg2.11.11.m1.5.5.1.1.1.1.1.cmml" xref="alg2.11.11.m1.5.5.1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.11.11.m1.5.5.1.1.1.1.1.1.cmml" xref="alg2.11.11.m1.5.5.1.1.1.1.1">superscript</csymbol><ci id="alg2.11.11.m1.5.5.1.1.1.1.1.2.cmml" xref="alg2.11.11.m1.5.5.1.1.1.1.1.2">𝑿</ci><ci id="alg2.11.11.m1.5.5.1.1.1.1.1.3.cmml" xref="alg2.11.11.m1.5.5.1.1.1.1.1.3">𝑙</ci></apply><apply id="alg2.11.11.m1.6.6.2.2.2.2.2.cmml" xref="alg2.11.11.m1.6.6.2.2.2.2.2"><csymbol cd="ambiguous" id="alg2.11.11.m1.6.6.2.2.2.2.2.1.cmml" xref="alg2.11.11.m1.6.6.2.2.2.2.2">superscript</csymbol><ci id="alg2.11.11.m1.6.6.2.2.2.2.2.2.cmml" xref="alg2.11.11.m1.6.6.2.2.2.2.2.2">𝑿</ci><apply id="alg2.11.11.m1.6.6.2.2.2.2.2.3.cmml" xref="alg2.11.11.m1.6.6.2.2.2.2.2.3"><minus id="alg2.11.11.m1.6.6.2.2.2.2.2.3.1.cmml" xref="alg2.11.11.m1.6.6.2.2.2.2.2.3.1"></minus><ci id="alg2.11.11.m1.6.6.2.2.2.2.2.3.2.cmml" xref="alg2.11.11.m1.6.6.2.2.2.2.2.3.2">𝑙</ci><cn type="integer" id="alg2.11.11.m1.6.6.2.2.2.2.2.3.3.cmml" xref="alg2.11.11.m1.6.6.2.2.2.2.2.3.3">1</cn></apply></apply></interval><list id="alg2.11.11.m1.2.2.2.3.cmml" xref="alg2.11.11.m1.2.2.2.4"><ci id="alg2.11.11.m1.1.1.1.1.cmml" xref="alg2.11.11.m1.1.1.1.1">𝑘</ci><ci id="alg2.11.11.m1.2.2.2.2.cmml" xref="alg2.11.11.m1.2.2.2.2">𝑣</ci></list></apply></apply><apply id="alg2.11.11.m1.6.6.4.cmml" xref="alg2.11.11.m1.6.6.4"><ci id="alg2.11.11.m1.6.6.4.1.cmml" xref="alg2.11.11.m1.6.6.4.1">⋅</ci><apply id="alg2.11.11.m1.6.6.4.2.cmml" xref="alg2.11.11.m1.6.6.4.2"><divide id="alg2.11.11.m1.6.6.4.2.1.cmml" xref="alg2.11.11.m1.6.6.4.2"></divide><cn type="integer" id="alg2.11.11.m1.6.6.4.2.2.cmml" xref="alg2.11.11.m1.6.6.4.2.2">1</cn><ci id="alg2.11.11.m1.6.6.4.2.3.cmml" xref="alg2.11.11.m1.6.6.4.2.3">𝜋</ci></apply><apply id="alg2.11.11.m1.6.6.4.3.cmml" xref="alg2.11.11.m1.6.6.4.3"><csymbol cd="ambiguous" id="alg2.11.11.m1.6.6.4.3.1.cmml" xref="alg2.11.11.m1.6.6.4.3">subscript</csymbol><ci id="alg2.11.11.m1.6.6.4.3.2.cmml" xref="alg2.11.11.m1.6.6.4.3.2">Ω</ci><list id="alg2.11.11.m1.4.4.2.3.cmml" xref="alg2.11.11.m1.4.4.2.4"><ci id="alg2.11.11.m1.3.3.1.1.cmml" xref="alg2.11.11.m1.3.3.1.1">𝑘</ci><ci id="alg2.11.11.m1.4.4.2.2.cmml" xref="alg2.11.11.m1.4.4.2.2">𝑣</ci></list></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.11.11.m1.6c">d({\bm{X}}^{l},{\bm{X}}^{l-1})_{k,v}=\frac{1}{\pi}\cdot\Omega_{k,v}</annotation></semantics></math> <span id="alg2.11.11.1" class="ltx_text ltx_font_typewriter">/* </span><span id="alg2.11.11.2" class="ltx_text ltx_font_typewriter">distance metrics  */</span>
</div>
<div id="alg2.38.41" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">6</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>    ;
</div>
<div id="alg2.12.12" class="ltx_listingline">  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<math id="alg2.12.12.m1.4" class="ltx_Math" alttext="{\mathbb{I}}_{k,v}=\{i\mid d_{i}&lt;d_{\text{min}}+(d_{\text{max}}-d_{\text{min}})\cdot\gamma\}" display="inline"><semantics id="alg2.12.12.m1.4a"><mrow id="alg2.12.12.m1.4.4" xref="alg2.12.12.m1.4.4.cmml"><msub id="alg2.12.12.m1.4.4.3" xref="alg2.12.12.m1.4.4.3.cmml"><mi id="alg2.12.12.m1.4.4.3.2" xref="alg2.12.12.m1.4.4.3.2.cmml">𝕀</mi><mrow id="alg2.12.12.m1.2.2.2.4" xref="alg2.12.12.m1.2.2.2.3.cmml"><mi id="alg2.12.12.m1.1.1.1.1" xref="alg2.12.12.m1.1.1.1.1.cmml">k</mi><mo id="alg2.12.12.m1.2.2.2.4.1" xref="alg2.12.12.m1.2.2.2.3.cmml">,</mo><mi id="alg2.12.12.m1.2.2.2.2" xref="alg2.12.12.m1.2.2.2.2.cmml">v</mi></mrow></msub><mo id="alg2.12.12.m1.4.4.2" xref="alg2.12.12.m1.4.4.2.cmml">=</mo><mrow id="alg2.12.12.m1.4.4.1.1" xref="alg2.12.12.m1.4.4.1.2.cmml"><mo stretchy="false" id="alg2.12.12.m1.4.4.1.1.2" xref="alg2.12.12.m1.4.4.1.2.1.cmml">{</mo><mi id="alg2.12.12.m1.3.3" xref="alg2.12.12.m1.3.3.cmml">i</mi><mo fence="true" lspace="0em" rspace="0em" id="alg2.12.12.m1.4.4.1.1.3" xref="alg2.12.12.m1.4.4.1.2.1.cmml">∣</mo><mrow id="alg2.12.12.m1.4.4.1.1.1" xref="alg2.12.12.m1.4.4.1.1.1.cmml"><msub id="alg2.12.12.m1.4.4.1.1.1.3" xref="alg2.12.12.m1.4.4.1.1.1.3.cmml"><mi id="alg2.12.12.m1.4.4.1.1.1.3.2" xref="alg2.12.12.m1.4.4.1.1.1.3.2.cmml">d</mi><mi id="alg2.12.12.m1.4.4.1.1.1.3.3" xref="alg2.12.12.m1.4.4.1.1.1.3.3.cmml">i</mi></msub><mo id="alg2.12.12.m1.4.4.1.1.1.2" xref="alg2.12.12.m1.4.4.1.1.1.2.cmml">&lt;</mo><mrow id="alg2.12.12.m1.4.4.1.1.1.1" xref="alg2.12.12.m1.4.4.1.1.1.1.cmml"><msub id="alg2.12.12.m1.4.4.1.1.1.1.3" xref="alg2.12.12.m1.4.4.1.1.1.1.3.cmml"><mi id="alg2.12.12.m1.4.4.1.1.1.1.3.2" xref="alg2.12.12.m1.4.4.1.1.1.1.3.2.cmml">d</mi><mtext id="alg2.12.12.m1.4.4.1.1.1.1.3.3" xref="alg2.12.12.m1.4.4.1.1.1.1.3.3a.cmml">min</mtext></msub><mo id="alg2.12.12.m1.4.4.1.1.1.1.2" xref="alg2.12.12.m1.4.4.1.1.1.1.2.cmml">+</mo><mrow id="alg2.12.12.m1.4.4.1.1.1.1.1" xref="alg2.12.12.m1.4.4.1.1.1.1.1.cmml"><mrow id="alg2.12.12.m1.4.4.1.1.1.1.1.1.1" xref="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.2" xref="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1" xref="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.cmml"><msub id="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.2" xref="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.2.cmml"><mi id="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.2.2" xref="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.2.2.cmml">d</mi><mtext id="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.2.3" xref="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.2.3a.cmml">max</mtext></msub><mo id="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.1" xref="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.1.cmml">−</mo><msub id="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.3" xref="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.3.cmml"><mi id="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.3.2" xref="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.3.2.cmml">d</mi><mtext id="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.3.3" xref="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.3.3a.cmml">min</mtext></msub></mrow><mo rspace="0.055em" stretchy="false" id="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.3" xref="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo rspace="0.222em" id="alg2.12.12.m1.4.4.1.1.1.1.1.2" xref="alg2.12.12.m1.4.4.1.1.1.1.1.2.cmml">⋅</mo><mi id="alg2.12.12.m1.4.4.1.1.1.1.1.3" xref="alg2.12.12.m1.4.4.1.1.1.1.1.3.cmml">γ</mi></mrow></mrow></mrow><mo stretchy="false" id="alg2.12.12.m1.4.4.1.1.4" xref="alg2.12.12.m1.4.4.1.2.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.12.12.m1.4b"><apply id="alg2.12.12.m1.4.4.cmml" xref="alg2.12.12.m1.4.4"><eq id="alg2.12.12.m1.4.4.2.cmml" xref="alg2.12.12.m1.4.4.2"></eq><apply id="alg2.12.12.m1.4.4.3.cmml" xref="alg2.12.12.m1.4.4.3"><csymbol cd="ambiguous" id="alg2.12.12.m1.4.4.3.1.cmml" xref="alg2.12.12.m1.4.4.3">subscript</csymbol><ci id="alg2.12.12.m1.4.4.3.2.cmml" xref="alg2.12.12.m1.4.4.3.2">𝕀</ci><list id="alg2.12.12.m1.2.2.2.3.cmml" xref="alg2.12.12.m1.2.2.2.4"><ci id="alg2.12.12.m1.1.1.1.1.cmml" xref="alg2.12.12.m1.1.1.1.1">𝑘</ci><ci id="alg2.12.12.m1.2.2.2.2.cmml" xref="alg2.12.12.m1.2.2.2.2">𝑣</ci></list></apply><apply id="alg2.12.12.m1.4.4.1.2.cmml" xref="alg2.12.12.m1.4.4.1.1"><csymbol cd="latexml" id="alg2.12.12.m1.4.4.1.2.1.cmml" xref="alg2.12.12.m1.4.4.1.1.2">conditional-set</csymbol><ci id="alg2.12.12.m1.3.3.cmml" xref="alg2.12.12.m1.3.3">𝑖</ci><apply id="alg2.12.12.m1.4.4.1.1.1.cmml" xref="alg2.12.12.m1.4.4.1.1.1"><lt id="alg2.12.12.m1.4.4.1.1.1.2.cmml" xref="alg2.12.12.m1.4.4.1.1.1.2"></lt><apply id="alg2.12.12.m1.4.4.1.1.1.3.cmml" xref="alg2.12.12.m1.4.4.1.1.1.3"><csymbol cd="ambiguous" id="alg2.12.12.m1.4.4.1.1.1.3.1.cmml" xref="alg2.12.12.m1.4.4.1.1.1.3">subscript</csymbol><ci id="alg2.12.12.m1.4.4.1.1.1.3.2.cmml" xref="alg2.12.12.m1.4.4.1.1.1.3.2">𝑑</ci><ci id="alg2.12.12.m1.4.4.1.1.1.3.3.cmml" xref="alg2.12.12.m1.4.4.1.1.1.3.3">𝑖</ci></apply><apply id="alg2.12.12.m1.4.4.1.1.1.1.cmml" xref="alg2.12.12.m1.4.4.1.1.1.1"><plus id="alg2.12.12.m1.4.4.1.1.1.1.2.cmml" xref="alg2.12.12.m1.4.4.1.1.1.1.2"></plus><apply id="alg2.12.12.m1.4.4.1.1.1.1.3.cmml" xref="alg2.12.12.m1.4.4.1.1.1.1.3"><csymbol cd="ambiguous" id="alg2.12.12.m1.4.4.1.1.1.1.3.1.cmml" xref="alg2.12.12.m1.4.4.1.1.1.1.3">subscript</csymbol><ci id="alg2.12.12.m1.4.4.1.1.1.1.3.2.cmml" xref="alg2.12.12.m1.4.4.1.1.1.1.3.2">𝑑</ci><ci id="alg2.12.12.m1.4.4.1.1.1.1.3.3a.cmml" xref="alg2.12.12.m1.4.4.1.1.1.1.3.3"><mtext mathsize="70%" id="alg2.12.12.m1.4.4.1.1.1.1.3.3.cmml" xref="alg2.12.12.m1.4.4.1.1.1.1.3.3">min</mtext></ci></apply><apply id="alg2.12.12.m1.4.4.1.1.1.1.1.cmml" xref="alg2.12.12.m1.4.4.1.1.1.1.1"><ci id="alg2.12.12.m1.4.4.1.1.1.1.1.2.cmml" xref="alg2.12.12.m1.4.4.1.1.1.1.1.2">⋅</ci><apply id="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.cmml" xref="alg2.12.12.m1.4.4.1.1.1.1.1.1.1"><minus id="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.1.cmml" xref="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.1"></minus><apply id="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.2.cmml" xref="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.2.1.cmml" xref="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.2.2.cmml" xref="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.2.2">𝑑</ci><ci id="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.2.3a.cmml" xref="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.2.3"><mtext mathsize="70%" id="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.2.3.cmml" xref="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.2.3">max</mtext></ci></apply><apply id="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.3.cmml" xref="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.3.1.cmml" xref="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.3.2.cmml" xref="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.3.2">𝑑</ci><ci id="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.3.3a.cmml" xref="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.3.3"><mtext mathsize="70%" id="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.3.3.cmml" xref="alg2.12.12.m1.4.4.1.1.1.1.1.1.1.1.3.3">min</mtext></ci></apply></apply><ci id="alg2.12.12.m1.4.4.1.1.1.1.1.3.cmml" xref="alg2.12.12.m1.4.4.1.1.1.1.1.3">𝛾</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.12.12.m1.4c">{\mathbb{I}}_{k,v}=\{i\mid d_{i}&lt;d_{\text{min}}+(d_{\text{max}}-d_{\text{min}})\cdot\gamma\}</annotation></semantics></math> <span id="alg2.12.12.1" class="ltx_text ltx_font_typewriter">/* </span><span id="alg2.12.12.2" class="ltx_text ltx_font_typewriter">retention indices  */</span>
</div>
<div id="alg2.38.42" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">7</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   ;
</div>
<div id="alg2.13.13" class="ltx_listingline">  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<math id="alg2.13.13.m1.10" class="ltx_Math" alttext="{\bm{R}}_{k}^{l,l-1}\leftarrow{\bm{X}}_{k}^{l,l-1}[{\mathbb{I}}_{k}],{\bm{R}}_{v}^{l,l-1}\leftarrow{\bm{X}}_{v}^{l,l-1}[{\mathbb{I}}_{v}]" display="inline"><semantics id="alg2.13.13.m1.10a"><mrow id="alg2.13.13.m1.10.10.2" xref="alg2.13.13.m1.10.10.3.cmml"><mrow id="alg2.13.13.m1.9.9.1.1" xref="alg2.13.13.m1.9.9.1.1.cmml"><msubsup id="alg2.13.13.m1.9.9.1.1.3" xref="alg2.13.13.m1.9.9.1.1.3.cmml"><mi id="alg2.13.13.m1.9.9.1.1.3.2.2" xref="alg2.13.13.m1.9.9.1.1.3.2.2.cmml">𝑹</mi><mi id="alg2.13.13.m1.9.9.1.1.3.2.3" xref="alg2.13.13.m1.9.9.1.1.3.2.3.cmml">k</mi><mrow id="alg2.13.13.m1.2.2.2.2" xref="alg2.13.13.m1.2.2.2.3.cmml"><mi id="alg2.13.13.m1.1.1.1.1" xref="alg2.13.13.m1.1.1.1.1.cmml">l</mi><mo id="alg2.13.13.m1.2.2.2.2.2" xref="alg2.13.13.m1.2.2.2.3.cmml">,</mo><mrow id="alg2.13.13.m1.2.2.2.2.1" xref="alg2.13.13.m1.2.2.2.2.1.cmml"><mi id="alg2.13.13.m1.2.2.2.2.1.2" xref="alg2.13.13.m1.2.2.2.2.1.2.cmml">l</mi><mo id="alg2.13.13.m1.2.2.2.2.1.1" xref="alg2.13.13.m1.2.2.2.2.1.1.cmml">−</mo><mn id="alg2.13.13.m1.2.2.2.2.1.3" xref="alg2.13.13.m1.2.2.2.2.1.3.cmml">1</mn></mrow></mrow></msubsup><mo stretchy="false" id="alg2.13.13.m1.9.9.1.1.2" xref="alg2.13.13.m1.9.9.1.1.2.cmml">←</mo><mrow id="alg2.13.13.m1.9.9.1.1.1" xref="alg2.13.13.m1.9.9.1.1.1.cmml"><msubsup id="alg2.13.13.m1.9.9.1.1.1.3" xref="alg2.13.13.m1.9.9.1.1.1.3.cmml"><mi id="alg2.13.13.m1.9.9.1.1.1.3.2.2" xref="alg2.13.13.m1.9.9.1.1.1.3.2.2.cmml">𝑿</mi><mi id="alg2.13.13.m1.9.9.1.1.1.3.2.3" xref="alg2.13.13.m1.9.9.1.1.1.3.2.3.cmml">k</mi><mrow id="alg2.13.13.m1.4.4.2.2" xref="alg2.13.13.m1.4.4.2.3.cmml"><mi id="alg2.13.13.m1.3.3.1.1" xref="alg2.13.13.m1.3.3.1.1.cmml">l</mi><mo id="alg2.13.13.m1.4.4.2.2.2" xref="alg2.13.13.m1.4.4.2.3.cmml">,</mo><mrow id="alg2.13.13.m1.4.4.2.2.1" xref="alg2.13.13.m1.4.4.2.2.1.cmml"><mi id="alg2.13.13.m1.4.4.2.2.1.2" xref="alg2.13.13.m1.4.4.2.2.1.2.cmml">l</mi><mo id="alg2.13.13.m1.4.4.2.2.1.1" xref="alg2.13.13.m1.4.4.2.2.1.1.cmml">−</mo><mn id="alg2.13.13.m1.4.4.2.2.1.3" xref="alg2.13.13.m1.4.4.2.2.1.3.cmml">1</mn></mrow></mrow></msubsup><mo lspace="0em" rspace="0em" id="alg2.13.13.m1.9.9.1.1.1.2" xref="alg2.13.13.m1.9.9.1.1.1.2.cmml">​</mo><mrow id="alg2.13.13.m1.9.9.1.1.1.1.1" xref="alg2.13.13.m1.9.9.1.1.1.1.2.cmml"><mo stretchy="false" id="alg2.13.13.m1.9.9.1.1.1.1.1.2" xref="alg2.13.13.m1.9.9.1.1.1.1.2.1.cmml">[</mo><msub id="alg2.13.13.m1.9.9.1.1.1.1.1.1" xref="alg2.13.13.m1.9.9.1.1.1.1.1.1.cmml"><mi id="alg2.13.13.m1.9.9.1.1.1.1.1.1.2" xref="alg2.13.13.m1.9.9.1.1.1.1.1.1.2.cmml">𝕀</mi><mi id="alg2.13.13.m1.9.9.1.1.1.1.1.1.3" xref="alg2.13.13.m1.9.9.1.1.1.1.1.1.3.cmml">k</mi></msub><mo stretchy="false" id="alg2.13.13.m1.9.9.1.1.1.1.1.3" xref="alg2.13.13.m1.9.9.1.1.1.1.2.1.cmml">]</mo></mrow></mrow></mrow><mo id="alg2.13.13.m1.10.10.2.3" xref="alg2.13.13.m1.10.10.3a.cmml">,</mo><mrow id="alg2.13.13.m1.10.10.2.2" xref="alg2.13.13.m1.10.10.2.2.cmml"><msubsup id="alg2.13.13.m1.10.10.2.2.3" xref="alg2.13.13.m1.10.10.2.2.3.cmml"><mi id="alg2.13.13.m1.10.10.2.2.3.2.2" xref="alg2.13.13.m1.10.10.2.2.3.2.2.cmml">𝑹</mi><mi id="alg2.13.13.m1.10.10.2.2.3.2.3" xref="alg2.13.13.m1.10.10.2.2.3.2.3.cmml">v</mi><mrow id="alg2.13.13.m1.6.6.2.2" xref="alg2.13.13.m1.6.6.2.3.cmml"><mi id="alg2.13.13.m1.5.5.1.1" xref="alg2.13.13.m1.5.5.1.1.cmml">l</mi><mo id="alg2.13.13.m1.6.6.2.2.2" xref="alg2.13.13.m1.6.6.2.3.cmml">,</mo><mrow id="alg2.13.13.m1.6.6.2.2.1" xref="alg2.13.13.m1.6.6.2.2.1.cmml"><mi id="alg2.13.13.m1.6.6.2.2.1.2" xref="alg2.13.13.m1.6.6.2.2.1.2.cmml">l</mi><mo id="alg2.13.13.m1.6.6.2.2.1.1" xref="alg2.13.13.m1.6.6.2.2.1.1.cmml">−</mo><mn id="alg2.13.13.m1.6.6.2.2.1.3" xref="alg2.13.13.m1.6.6.2.2.1.3.cmml">1</mn></mrow></mrow></msubsup><mo stretchy="false" id="alg2.13.13.m1.10.10.2.2.2" xref="alg2.13.13.m1.10.10.2.2.2.cmml">←</mo><mrow id="alg2.13.13.m1.10.10.2.2.1" xref="alg2.13.13.m1.10.10.2.2.1.cmml"><msubsup id="alg2.13.13.m1.10.10.2.2.1.3" xref="alg2.13.13.m1.10.10.2.2.1.3.cmml"><mi id="alg2.13.13.m1.10.10.2.2.1.3.2.2" xref="alg2.13.13.m1.10.10.2.2.1.3.2.2.cmml">𝑿</mi><mi id="alg2.13.13.m1.10.10.2.2.1.3.2.3" xref="alg2.13.13.m1.10.10.2.2.1.3.2.3.cmml">v</mi><mrow id="alg2.13.13.m1.8.8.2.2" xref="alg2.13.13.m1.8.8.2.3.cmml"><mi id="alg2.13.13.m1.7.7.1.1" xref="alg2.13.13.m1.7.7.1.1.cmml">l</mi><mo id="alg2.13.13.m1.8.8.2.2.2" xref="alg2.13.13.m1.8.8.2.3.cmml">,</mo><mrow id="alg2.13.13.m1.8.8.2.2.1" xref="alg2.13.13.m1.8.8.2.2.1.cmml"><mi id="alg2.13.13.m1.8.8.2.2.1.2" xref="alg2.13.13.m1.8.8.2.2.1.2.cmml">l</mi><mo id="alg2.13.13.m1.8.8.2.2.1.1" xref="alg2.13.13.m1.8.8.2.2.1.1.cmml">−</mo><mn id="alg2.13.13.m1.8.8.2.2.1.3" xref="alg2.13.13.m1.8.8.2.2.1.3.cmml">1</mn></mrow></mrow></msubsup><mo lspace="0em" rspace="0em" id="alg2.13.13.m1.10.10.2.2.1.2" xref="alg2.13.13.m1.10.10.2.2.1.2.cmml">​</mo><mrow id="alg2.13.13.m1.10.10.2.2.1.1.1" xref="alg2.13.13.m1.10.10.2.2.1.1.2.cmml"><mo stretchy="false" id="alg2.13.13.m1.10.10.2.2.1.1.1.2" xref="alg2.13.13.m1.10.10.2.2.1.1.2.1.cmml">[</mo><msub id="alg2.13.13.m1.10.10.2.2.1.1.1.1" xref="alg2.13.13.m1.10.10.2.2.1.1.1.1.cmml"><mi id="alg2.13.13.m1.10.10.2.2.1.1.1.1.2" xref="alg2.13.13.m1.10.10.2.2.1.1.1.1.2.cmml">𝕀</mi><mi id="alg2.13.13.m1.10.10.2.2.1.1.1.1.3" xref="alg2.13.13.m1.10.10.2.2.1.1.1.1.3.cmml">v</mi></msub><mo stretchy="false" id="alg2.13.13.m1.10.10.2.2.1.1.1.3" xref="alg2.13.13.m1.10.10.2.2.1.1.2.1.cmml">]</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.13.13.m1.10b"><apply id="alg2.13.13.m1.10.10.3.cmml" xref="alg2.13.13.m1.10.10.2"><csymbol cd="ambiguous" id="alg2.13.13.m1.10.10.3a.cmml" xref="alg2.13.13.m1.10.10.2.3">formulae-sequence</csymbol><apply id="alg2.13.13.m1.9.9.1.1.cmml" xref="alg2.13.13.m1.9.9.1.1"><ci id="alg2.13.13.m1.9.9.1.1.2.cmml" xref="alg2.13.13.m1.9.9.1.1.2">←</ci><apply id="alg2.13.13.m1.9.9.1.1.3.cmml" xref="alg2.13.13.m1.9.9.1.1.3"><csymbol cd="ambiguous" id="alg2.13.13.m1.9.9.1.1.3.1.cmml" xref="alg2.13.13.m1.9.9.1.1.3">superscript</csymbol><apply id="alg2.13.13.m1.9.9.1.1.3.2.cmml" xref="alg2.13.13.m1.9.9.1.1.3"><csymbol cd="ambiguous" id="alg2.13.13.m1.9.9.1.1.3.2.1.cmml" xref="alg2.13.13.m1.9.9.1.1.3">subscript</csymbol><ci id="alg2.13.13.m1.9.9.1.1.3.2.2.cmml" xref="alg2.13.13.m1.9.9.1.1.3.2.2">𝑹</ci><ci id="alg2.13.13.m1.9.9.1.1.3.2.3.cmml" xref="alg2.13.13.m1.9.9.1.1.3.2.3">𝑘</ci></apply><list id="alg2.13.13.m1.2.2.2.3.cmml" xref="alg2.13.13.m1.2.2.2.2"><ci id="alg2.13.13.m1.1.1.1.1.cmml" xref="alg2.13.13.m1.1.1.1.1">𝑙</ci><apply id="alg2.13.13.m1.2.2.2.2.1.cmml" xref="alg2.13.13.m1.2.2.2.2.1"><minus id="alg2.13.13.m1.2.2.2.2.1.1.cmml" xref="alg2.13.13.m1.2.2.2.2.1.1"></minus><ci id="alg2.13.13.m1.2.2.2.2.1.2.cmml" xref="alg2.13.13.m1.2.2.2.2.1.2">𝑙</ci><cn type="integer" id="alg2.13.13.m1.2.2.2.2.1.3.cmml" xref="alg2.13.13.m1.2.2.2.2.1.3">1</cn></apply></list></apply><apply id="alg2.13.13.m1.9.9.1.1.1.cmml" xref="alg2.13.13.m1.9.9.1.1.1"><times id="alg2.13.13.m1.9.9.1.1.1.2.cmml" xref="alg2.13.13.m1.9.9.1.1.1.2"></times><apply id="alg2.13.13.m1.9.9.1.1.1.3.cmml" xref="alg2.13.13.m1.9.9.1.1.1.3"><csymbol cd="ambiguous" id="alg2.13.13.m1.9.9.1.1.1.3.1.cmml" xref="alg2.13.13.m1.9.9.1.1.1.3">superscript</csymbol><apply id="alg2.13.13.m1.9.9.1.1.1.3.2.cmml" xref="alg2.13.13.m1.9.9.1.1.1.3"><csymbol cd="ambiguous" id="alg2.13.13.m1.9.9.1.1.1.3.2.1.cmml" xref="alg2.13.13.m1.9.9.1.1.1.3">subscript</csymbol><ci id="alg2.13.13.m1.9.9.1.1.1.3.2.2.cmml" xref="alg2.13.13.m1.9.9.1.1.1.3.2.2">𝑿</ci><ci id="alg2.13.13.m1.9.9.1.1.1.3.2.3.cmml" xref="alg2.13.13.m1.9.9.1.1.1.3.2.3">𝑘</ci></apply><list id="alg2.13.13.m1.4.4.2.3.cmml" xref="alg2.13.13.m1.4.4.2.2"><ci id="alg2.13.13.m1.3.3.1.1.cmml" xref="alg2.13.13.m1.3.3.1.1">𝑙</ci><apply id="alg2.13.13.m1.4.4.2.2.1.cmml" xref="alg2.13.13.m1.4.4.2.2.1"><minus id="alg2.13.13.m1.4.4.2.2.1.1.cmml" xref="alg2.13.13.m1.4.4.2.2.1.1"></minus><ci id="alg2.13.13.m1.4.4.2.2.1.2.cmml" xref="alg2.13.13.m1.4.4.2.2.1.2">𝑙</ci><cn type="integer" id="alg2.13.13.m1.4.4.2.2.1.3.cmml" xref="alg2.13.13.m1.4.4.2.2.1.3">1</cn></apply></list></apply><apply id="alg2.13.13.m1.9.9.1.1.1.1.2.cmml" xref="alg2.13.13.m1.9.9.1.1.1.1.1"><csymbol cd="latexml" id="alg2.13.13.m1.9.9.1.1.1.1.2.1.cmml" xref="alg2.13.13.m1.9.9.1.1.1.1.1.2">delimited-[]</csymbol><apply id="alg2.13.13.m1.9.9.1.1.1.1.1.1.cmml" xref="alg2.13.13.m1.9.9.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.13.13.m1.9.9.1.1.1.1.1.1.1.cmml" xref="alg2.13.13.m1.9.9.1.1.1.1.1.1">subscript</csymbol><ci id="alg2.13.13.m1.9.9.1.1.1.1.1.1.2.cmml" xref="alg2.13.13.m1.9.9.1.1.1.1.1.1.2">𝕀</ci><ci id="alg2.13.13.m1.9.9.1.1.1.1.1.1.3.cmml" xref="alg2.13.13.m1.9.9.1.1.1.1.1.1.3">𝑘</ci></apply></apply></apply></apply><apply id="alg2.13.13.m1.10.10.2.2.cmml" xref="alg2.13.13.m1.10.10.2.2"><ci id="alg2.13.13.m1.10.10.2.2.2.cmml" xref="alg2.13.13.m1.10.10.2.2.2">←</ci><apply id="alg2.13.13.m1.10.10.2.2.3.cmml" xref="alg2.13.13.m1.10.10.2.2.3"><csymbol cd="ambiguous" id="alg2.13.13.m1.10.10.2.2.3.1.cmml" xref="alg2.13.13.m1.10.10.2.2.3">superscript</csymbol><apply id="alg2.13.13.m1.10.10.2.2.3.2.cmml" xref="alg2.13.13.m1.10.10.2.2.3"><csymbol cd="ambiguous" id="alg2.13.13.m1.10.10.2.2.3.2.1.cmml" xref="alg2.13.13.m1.10.10.2.2.3">subscript</csymbol><ci id="alg2.13.13.m1.10.10.2.2.3.2.2.cmml" xref="alg2.13.13.m1.10.10.2.2.3.2.2">𝑹</ci><ci id="alg2.13.13.m1.10.10.2.2.3.2.3.cmml" xref="alg2.13.13.m1.10.10.2.2.3.2.3">𝑣</ci></apply><list id="alg2.13.13.m1.6.6.2.3.cmml" xref="alg2.13.13.m1.6.6.2.2"><ci id="alg2.13.13.m1.5.5.1.1.cmml" xref="alg2.13.13.m1.5.5.1.1">𝑙</ci><apply id="alg2.13.13.m1.6.6.2.2.1.cmml" xref="alg2.13.13.m1.6.6.2.2.1"><minus id="alg2.13.13.m1.6.6.2.2.1.1.cmml" xref="alg2.13.13.m1.6.6.2.2.1.1"></minus><ci id="alg2.13.13.m1.6.6.2.2.1.2.cmml" xref="alg2.13.13.m1.6.6.2.2.1.2">𝑙</ci><cn type="integer" id="alg2.13.13.m1.6.6.2.2.1.3.cmml" xref="alg2.13.13.m1.6.6.2.2.1.3">1</cn></apply></list></apply><apply id="alg2.13.13.m1.10.10.2.2.1.cmml" xref="alg2.13.13.m1.10.10.2.2.1"><times id="alg2.13.13.m1.10.10.2.2.1.2.cmml" xref="alg2.13.13.m1.10.10.2.2.1.2"></times><apply id="alg2.13.13.m1.10.10.2.2.1.3.cmml" xref="alg2.13.13.m1.10.10.2.2.1.3"><csymbol cd="ambiguous" id="alg2.13.13.m1.10.10.2.2.1.3.1.cmml" xref="alg2.13.13.m1.10.10.2.2.1.3">superscript</csymbol><apply id="alg2.13.13.m1.10.10.2.2.1.3.2.cmml" xref="alg2.13.13.m1.10.10.2.2.1.3"><csymbol cd="ambiguous" id="alg2.13.13.m1.10.10.2.2.1.3.2.1.cmml" xref="alg2.13.13.m1.10.10.2.2.1.3">subscript</csymbol><ci id="alg2.13.13.m1.10.10.2.2.1.3.2.2.cmml" xref="alg2.13.13.m1.10.10.2.2.1.3.2.2">𝑿</ci><ci id="alg2.13.13.m1.10.10.2.2.1.3.2.3.cmml" xref="alg2.13.13.m1.10.10.2.2.1.3.2.3">𝑣</ci></apply><list id="alg2.13.13.m1.8.8.2.3.cmml" xref="alg2.13.13.m1.8.8.2.2"><ci id="alg2.13.13.m1.7.7.1.1.cmml" xref="alg2.13.13.m1.7.7.1.1">𝑙</ci><apply id="alg2.13.13.m1.8.8.2.2.1.cmml" xref="alg2.13.13.m1.8.8.2.2.1"><minus id="alg2.13.13.m1.8.8.2.2.1.1.cmml" xref="alg2.13.13.m1.8.8.2.2.1.1"></minus><ci id="alg2.13.13.m1.8.8.2.2.1.2.cmml" xref="alg2.13.13.m1.8.8.2.2.1.2">𝑙</ci><cn type="integer" id="alg2.13.13.m1.8.8.2.2.1.3.cmml" xref="alg2.13.13.m1.8.8.2.2.1.3">1</cn></apply></list></apply><apply id="alg2.13.13.m1.10.10.2.2.1.1.2.cmml" xref="alg2.13.13.m1.10.10.2.2.1.1.1"><csymbol cd="latexml" id="alg2.13.13.m1.10.10.2.2.1.1.2.1.cmml" xref="alg2.13.13.m1.10.10.2.2.1.1.1.2">delimited-[]</csymbol><apply id="alg2.13.13.m1.10.10.2.2.1.1.1.1.cmml" xref="alg2.13.13.m1.10.10.2.2.1.1.1.1"><csymbol cd="ambiguous" id="alg2.13.13.m1.10.10.2.2.1.1.1.1.1.cmml" xref="alg2.13.13.m1.10.10.2.2.1.1.1.1">subscript</csymbol><ci id="alg2.13.13.m1.10.10.2.2.1.1.1.1.2.cmml" xref="alg2.13.13.m1.10.10.2.2.1.1.1.1.2">𝕀</ci><ci id="alg2.13.13.m1.10.10.2.2.1.1.1.1.3.cmml" xref="alg2.13.13.m1.10.10.2.2.1.1.1.1.3">𝑣</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.13.13.m1.10c">{\bm{R}}_{k}^{l,l-1}\leftarrow{\bm{X}}_{k}^{l,l-1}[{\mathbb{I}}_{k}],{\bm{R}}_{v}^{l,l-1}\leftarrow{\bm{X}}_{v}^{l,l-1}[{\mathbb{I}}_{v}]</annotation></semantics></math> <span id="alg2.13.13.1" class="ltx_text ltx_font_typewriter">/* </span><span id="alg2.13.13.2" class="ltx_text ltx_font_typewriter">unmerged tokens  */</span>
</div>
<div id="alg2.38.43" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">8</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>    ;
</div>
<div id="alg2.14.14" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">9</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<span id="alg2.14.14.2" class="ltx_text ltx_font_bold">return</span> <em id="alg2.14.14.1" class="ltx_emph ltx_font_italic"><math id="alg2.14.14.1.m1.17" class="ltx_Math" alttext="{\bm{E}}_{k,v}^{l},\lVert{\bm{X}}_{k,v}^{l,l-1}\rVert,{\bm{R}}_{k}^{l,l-1},{\bm{R}}_{v}^{l,l-1},{\mathbb{I}}_{k,v}" display="inline"><semantics id="alg2.14.14.1.m1.17a"><mrow id="alg2.14.14.1.m1.17.17.5" xref="alg2.14.14.1.m1.17.17.6.cmml"><msubsup id="alg2.14.14.1.m1.13.13.1.1" xref="alg2.14.14.1.m1.13.13.1.1.cmml"><mi id="alg2.14.14.1.m1.13.13.1.1.2.2" xref="alg2.14.14.1.m1.13.13.1.1.2.2.cmml">𝐄</mi><mrow id="alg2.14.14.1.m1.2.2.2.4" xref="alg2.14.14.1.m1.2.2.2.3.cmml"><mi id="alg2.14.14.1.m1.1.1.1.1" xref="alg2.14.14.1.m1.1.1.1.1.cmml">k</mi><mo id="alg2.14.14.1.m1.2.2.2.4.1" xref="alg2.14.14.1.m1.2.2.2.3.cmml">,</mo><mi id="alg2.14.14.1.m1.2.2.2.2" xref="alg2.14.14.1.m1.2.2.2.2.cmml">v</mi></mrow><mi id="alg2.14.14.1.m1.13.13.1.1.3" xref="alg2.14.14.1.m1.13.13.1.1.3.cmml">l</mi></msubsup><mo id="alg2.14.14.1.m1.17.17.5.6" xref="alg2.14.14.1.m1.17.17.6.cmml">,</mo><mrow id="alg2.14.14.1.m1.14.14.2.2.1" xref="alg2.14.14.1.m1.14.14.2.2.2.cmml"><mo fence="true" lspace="0em" rspace="0em" id="alg2.14.14.1.m1.14.14.2.2.1.2" xref="alg2.14.14.1.m1.14.14.2.2.2.1.cmml">∥</mo><msubsup id="alg2.14.14.1.m1.14.14.2.2.1.1" xref="alg2.14.14.1.m1.14.14.2.2.1.1.cmml"><mi id="alg2.14.14.1.m1.14.14.2.2.1.1.2.2" xref="alg2.14.14.1.m1.14.14.2.2.1.1.2.2.cmml">𝐗</mi><mrow id="alg2.14.14.1.m1.4.4.2.4" xref="alg2.14.14.1.m1.4.4.2.3.cmml"><mi id="alg2.14.14.1.m1.3.3.1.1" xref="alg2.14.14.1.m1.3.3.1.1.cmml">k</mi><mo id="alg2.14.14.1.m1.4.4.2.4.1" xref="alg2.14.14.1.m1.4.4.2.3.cmml">,</mo><mi id="alg2.14.14.1.m1.4.4.2.2" xref="alg2.14.14.1.m1.4.4.2.2.cmml">v</mi></mrow><mrow id="alg2.14.14.1.m1.6.6.2.2" xref="alg2.14.14.1.m1.6.6.2.3.cmml"><mi id="alg2.14.14.1.m1.5.5.1.1" xref="alg2.14.14.1.m1.5.5.1.1.cmml">l</mi><mo id="alg2.14.14.1.m1.6.6.2.2.2" xref="alg2.14.14.1.m1.6.6.2.3.cmml">,</mo><mrow id="alg2.14.14.1.m1.6.6.2.2.1" xref="alg2.14.14.1.m1.6.6.2.2.1.cmml"><mi id="alg2.14.14.1.m1.6.6.2.2.1.2" xref="alg2.14.14.1.m1.6.6.2.2.1.2.cmml">l</mi><mo id="alg2.14.14.1.m1.6.6.2.2.1.1" xref="alg2.14.14.1.m1.6.6.2.2.1.1.cmml">−</mo><mn id="alg2.14.14.1.m1.6.6.2.2.1.3" xref="alg2.14.14.1.m1.6.6.2.2.1.3.cmml">1</mn></mrow></mrow></msubsup><mo fence="true" lspace="0em" rspace="0em" id="alg2.14.14.1.m1.14.14.2.2.1.3" xref="alg2.14.14.1.m1.14.14.2.2.2.1.cmml">∥</mo></mrow><mo id="alg2.14.14.1.m1.17.17.5.7" xref="alg2.14.14.1.m1.17.17.6.cmml">,</mo><msubsup id="alg2.14.14.1.m1.15.15.3.3" xref="alg2.14.14.1.m1.15.15.3.3.cmml"><mi id="alg2.14.14.1.m1.15.15.3.3.2.2" xref="alg2.14.14.1.m1.15.15.3.3.2.2.cmml">𝐑</mi><mi id="alg2.14.14.1.m1.15.15.3.3.2.3" xref="alg2.14.14.1.m1.15.15.3.3.2.3.cmml">k</mi><mrow id="alg2.14.14.1.m1.8.8.2.2" xref="alg2.14.14.1.m1.8.8.2.3.cmml"><mi id="alg2.14.14.1.m1.7.7.1.1" xref="alg2.14.14.1.m1.7.7.1.1.cmml">l</mi><mo id="alg2.14.14.1.m1.8.8.2.2.2" xref="alg2.14.14.1.m1.8.8.2.3.cmml">,</mo><mrow id="alg2.14.14.1.m1.8.8.2.2.1" xref="alg2.14.14.1.m1.8.8.2.2.1.cmml"><mi id="alg2.14.14.1.m1.8.8.2.2.1.2" xref="alg2.14.14.1.m1.8.8.2.2.1.2.cmml">l</mi><mo id="alg2.14.14.1.m1.8.8.2.2.1.1" xref="alg2.14.14.1.m1.8.8.2.2.1.1.cmml">−</mo><mn id="alg2.14.14.1.m1.8.8.2.2.1.3" xref="alg2.14.14.1.m1.8.8.2.2.1.3.cmml">1</mn></mrow></mrow></msubsup><mo id="alg2.14.14.1.m1.17.17.5.8" xref="alg2.14.14.1.m1.17.17.6.cmml">,</mo><msubsup id="alg2.14.14.1.m1.16.16.4.4" xref="alg2.14.14.1.m1.16.16.4.4.cmml"><mi id="alg2.14.14.1.m1.16.16.4.4.2.2" xref="alg2.14.14.1.m1.16.16.4.4.2.2.cmml">𝐑</mi><mi id="alg2.14.14.1.m1.16.16.4.4.2.3" xref="alg2.14.14.1.m1.16.16.4.4.2.3.cmml">v</mi><mrow id="alg2.14.14.1.m1.10.10.2.2" xref="alg2.14.14.1.m1.10.10.2.3.cmml"><mi id="alg2.14.14.1.m1.9.9.1.1" xref="alg2.14.14.1.m1.9.9.1.1.cmml">l</mi><mo id="alg2.14.14.1.m1.10.10.2.2.2" xref="alg2.14.14.1.m1.10.10.2.3.cmml">,</mo><mrow id="alg2.14.14.1.m1.10.10.2.2.1" xref="alg2.14.14.1.m1.10.10.2.2.1.cmml"><mi id="alg2.14.14.1.m1.10.10.2.2.1.2" xref="alg2.14.14.1.m1.10.10.2.2.1.2.cmml">l</mi><mo id="alg2.14.14.1.m1.10.10.2.2.1.1" xref="alg2.14.14.1.m1.10.10.2.2.1.1.cmml">−</mo><mn id="alg2.14.14.1.m1.10.10.2.2.1.3" xref="alg2.14.14.1.m1.10.10.2.2.1.3.cmml">1</mn></mrow></mrow></msubsup><mo id="alg2.14.14.1.m1.17.17.5.9" xref="alg2.14.14.1.m1.17.17.6.cmml">,</mo><msub id="alg2.14.14.1.m1.17.17.5.5" xref="alg2.14.14.1.m1.17.17.5.5.cmml"><mi id="alg2.14.14.1.m1.17.17.5.5.2" xref="alg2.14.14.1.m1.17.17.5.5.2.cmml">𝕀</mi><mrow id="alg2.14.14.1.m1.12.12.2.4" xref="alg2.14.14.1.m1.12.12.2.3.cmml"><mi id="alg2.14.14.1.m1.11.11.1.1" xref="alg2.14.14.1.m1.11.11.1.1.cmml">k</mi><mo id="alg2.14.14.1.m1.12.12.2.4.1" xref="alg2.14.14.1.m1.12.12.2.3.cmml">,</mo><mi id="alg2.14.14.1.m1.12.12.2.2" xref="alg2.14.14.1.m1.12.12.2.2.cmml">v</mi></mrow></msub></mrow><annotation-xml encoding="MathML-Content" id="alg2.14.14.1.m1.17b"><list id="alg2.14.14.1.m1.17.17.6.cmml" xref="alg2.14.14.1.m1.17.17.5"><apply id="alg2.14.14.1.m1.13.13.1.1.cmml" xref="alg2.14.14.1.m1.13.13.1.1"><csymbol cd="ambiguous" id="alg2.14.14.1.m1.13.13.1.1.1.cmml" xref="alg2.14.14.1.m1.13.13.1.1">superscript</csymbol><apply id="alg2.14.14.1.m1.13.13.1.1.2.cmml" xref="alg2.14.14.1.m1.13.13.1.1"><csymbol cd="ambiguous" id="alg2.14.14.1.m1.13.13.1.1.2.1.cmml" xref="alg2.14.14.1.m1.13.13.1.1">subscript</csymbol><ci id="alg2.14.14.1.m1.13.13.1.1.2.2.cmml" xref="alg2.14.14.1.m1.13.13.1.1.2.2">𝐄</ci><list id="alg2.14.14.1.m1.2.2.2.3.cmml" xref="alg2.14.14.1.m1.2.2.2.4"><ci id="alg2.14.14.1.m1.1.1.1.1.cmml" xref="alg2.14.14.1.m1.1.1.1.1">𝑘</ci><ci id="alg2.14.14.1.m1.2.2.2.2.cmml" xref="alg2.14.14.1.m1.2.2.2.2">𝑣</ci></list></apply><ci id="alg2.14.14.1.m1.13.13.1.1.3.cmml" xref="alg2.14.14.1.m1.13.13.1.1.3">𝑙</ci></apply><apply id="alg2.14.14.1.m1.14.14.2.2.2.cmml" xref="alg2.14.14.1.m1.14.14.2.2.1"><csymbol cd="latexml" id="alg2.14.14.1.m1.14.14.2.2.2.1.cmml" xref="alg2.14.14.1.m1.14.14.2.2.1.2">delimited-∥∥</csymbol><apply id="alg2.14.14.1.m1.14.14.2.2.1.1.cmml" xref="alg2.14.14.1.m1.14.14.2.2.1.1"><csymbol cd="ambiguous" id="alg2.14.14.1.m1.14.14.2.2.1.1.1.cmml" xref="alg2.14.14.1.m1.14.14.2.2.1.1">superscript</csymbol><apply id="alg2.14.14.1.m1.14.14.2.2.1.1.2.cmml" xref="alg2.14.14.1.m1.14.14.2.2.1.1"><csymbol cd="ambiguous" id="alg2.14.14.1.m1.14.14.2.2.1.1.2.1.cmml" xref="alg2.14.14.1.m1.14.14.2.2.1.1">subscript</csymbol><ci id="alg2.14.14.1.m1.14.14.2.2.1.1.2.2.cmml" xref="alg2.14.14.1.m1.14.14.2.2.1.1.2.2">𝐗</ci><list id="alg2.14.14.1.m1.4.4.2.3.cmml" xref="alg2.14.14.1.m1.4.4.2.4"><ci id="alg2.14.14.1.m1.3.3.1.1.cmml" xref="alg2.14.14.1.m1.3.3.1.1">𝑘</ci><ci id="alg2.14.14.1.m1.4.4.2.2.cmml" xref="alg2.14.14.1.m1.4.4.2.2">𝑣</ci></list></apply><list id="alg2.14.14.1.m1.6.6.2.3.cmml" xref="alg2.14.14.1.m1.6.6.2.2"><ci id="alg2.14.14.1.m1.5.5.1.1.cmml" xref="alg2.14.14.1.m1.5.5.1.1">𝑙</ci><apply id="alg2.14.14.1.m1.6.6.2.2.1.cmml" xref="alg2.14.14.1.m1.6.6.2.2.1"><minus id="alg2.14.14.1.m1.6.6.2.2.1.1.cmml" xref="alg2.14.14.1.m1.6.6.2.2.1.1"></minus><ci id="alg2.14.14.1.m1.6.6.2.2.1.2.cmml" xref="alg2.14.14.1.m1.6.6.2.2.1.2">𝑙</ci><cn type="integer" id="alg2.14.14.1.m1.6.6.2.2.1.3.cmml" xref="alg2.14.14.1.m1.6.6.2.2.1.3">1</cn></apply></list></apply></apply><apply id="alg2.14.14.1.m1.15.15.3.3.cmml" xref="alg2.14.14.1.m1.15.15.3.3"><csymbol cd="ambiguous" id="alg2.14.14.1.m1.15.15.3.3.1.cmml" xref="alg2.14.14.1.m1.15.15.3.3">superscript</csymbol><apply id="alg2.14.14.1.m1.15.15.3.3.2.cmml" xref="alg2.14.14.1.m1.15.15.3.3"><csymbol cd="ambiguous" id="alg2.14.14.1.m1.15.15.3.3.2.1.cmml" xref="alg2.14.14.1.m1.15.15.3.3">subscript</csymbol><ci id="alg2.14.14.1.m1.15.15.3.3.2.2.cmml" xref="alg2.14.14.1.m1.15.15.3.3.2.2">𝐑</ci><ci id="alg2.14.14.1.m1.15.15.3.3.2.3.cmml" xref="alg2.14.14.1.m1.15.15.3.3.2.3">𝑘</ci></apply><list id="alg2.14.14.1.m1.8.8.2.3.cmml" xref="alg2.14.14.1.m1.8.8.2.2"><ci id="alg2.14.14.1.m1.7.7.1.1.cmml" xref="alg2.14.14.1.m1.7.7.1.1">𝑙</ci><apply id="alg2.14.14.1.m1.8.8.2.2.1.cmml" xref="alg2.14.14.1.m1.8.8.2.2.1"><minus id="alg2.14.14.1.m1.8.8.2.2.1.1.cmml" xref="alg2.14.14.1.m1.8.8.2.2.1.1"></minus><ci id="alg2.14.14.1.m1.8.8.2.2.1.2.cmml" xref="alg2.14.14.1.m1.8.8.2.2.1.2">𝑙</ci><cn type="integer" id="alg2.14.14.1.m1.8.8.2.2.1.3.cmml" xref="alg2.14.14.1.m1.8.8.2.2.1.3">1</cn></apply></list></apply><apply id="alg2.14.14.1.m1.16.16.4.4.cmml" xref="alg2.14.14.1.m1.16.16.4.4"><csymbol cd="ambiguous" id="alg2.14.14.1.m1.16.16.4.4.1.cmml" xref="alg2.14.14.1.m1.16.16.4.4">superscript</csymbol><apply id="alg2.14.14.1.m1.16.16.4.4.2.cmml" xref="alg2.14.14.1.m1.16.16.4.4"><csymbol cd="ambiguous" id="alg2.14.14.1.m1.16.16.4.4.2.1.cmml" xref="alg2.14.14.1.m1.16.16.4.4">subscript</csymbol><ci id="alg2.14.14.1.m1.16.16.4.4.2.2.cmml" xref="alg2.14.14.1.m1.16.16.4.4.2.2">𝐑</ci><ci id="alg2.14.14.1.m1.16.16.4.4.2.3.cmml" xref="alg2.14.14.1.m1.16.16.4.4.2.3">𝑣</ci></apply><list id="alg2.14.14.1.m1.10.10.2.3.cmml" xref="alg2.14.14.1.m1.10.10.2.2"><ci id="alg2.14.14.1.m1.9.9.1.1.cmml" xref="alg2.14.14.1.m1.9.9.1.1">𝑙</ci><apply id="alg2.14.14.1.m1.10.10.2.2.1.cmml" xref="alg2.14.14.1.m1.10.10.2.2.1"><minus id="alg2.14.14.1.m1.10.10.2.2.1.1.cmml" xref="alg2.14.14.1.m1.10.10.2.2.1.1"></minus><ci id="alg2.14.14.1.m1.10.10.2.2.1.2.cmml" xref="alg2.14.14.1.m1.10.10.2.2.1.2">𝑙</ci><cn type="integer" id="alg2.14.14.1.m1.10.10.2.2.1.3.cmml" xref="alg2.14.14.1.m1.10.10.2.2.1.3">1</cn></apply></list></apply><apply id="alg2.14.14.1.m1.17.17.5.5.cmml" xref="alg2.14.14.1.m1.17.17.5.5"><csymbol cd="ambiguous" id="alg2.14.14.1.m1.17.17.5.5.1.cmml" xref="alg2.14.14.1.m1.17.17.5.5">subscript</csymbol><ci id="alg2.14.14.1.m1.17.17.5.5.2.cmml" xref="alg2.14.14.1.m1.17.17.5.5.2">𝕀</ci><list id="alg2.14.14.1.m1.12.12.2.3.cmml" xref="alg2.14.14.1.m1.12.12.2.4"><ci id="alg2.14.14.1.m1.11.11.1.1.cmml" xref="alg2.14.14.1.m1.11.11.1.1">𝑘</ci><ci id="alg2.14.14.1.m1.12.12.2.2.cmml" xref="alg2.14.14.1.m1.12.12.2.2">𝑣</ci></list></apply></list></annotation-xml><annotation encoding="application/x-tex" id="alg2.14.14.1.m1.17c">{\bm{E}}_{k,v}^{l},\lVert{\bm{X}}_{k,v}^{l,l-1}\rVert,{\bm{R}}_{k}^{l,l-1},{\bm{R}}_{v}^{l,l-1},{\mathbb{I}}_{k,v}</annotation></semantics></math></em>

</div>
<div id="alg2.38.44" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">10</span>
</div>
<div id="alg2.38.45" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">11</span><span id="alg2.38.45.1" class="ltx_text ltx_font_bold">procedure</span> <em id="alg2.38.45.2" class="ltx_emph ltx_font_typewriter">MiniCache Decoding</em><span id="alg2.38.45.3" class="ltx_text ltx_font_bold">:</span>
</div>
<div id="alg2.20.20" class="ltx_listingline">  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<span id="alg2.20.20.1" class="ltx_text"><span id="alg2.20.20.1.1" class="ltx_text ltx_font_bold">Input:</span> </span>KV cache: <math id="alg2.15.15.m1.2" class="ltx_Math" alttext="{\bm{E}}_{k,v}^{l}\in\mathbb{R}^{2\times t_{\text{prompt}}\times d}" display="inline"><semantics id="alg2.15.15.m1.2a"><mrow id="alg2.15.15.m1.2.3" xref="alg2.15.15.m1.2.3.cmml"><msubsup id="alg2.15.15.m1.2.3.2" xref="alg2.15.15.m1.2.3.2.cmml"><mi id="alg2.15.15.m1.2.3.2.2.2" xref="alg2.15.15.m1.2.3.2.2.2.cmml">𝑬</mi><mrow id="alg2.15.15.m1.2.2.2.4" xref="alg2.15.15.m1.2.2.2.3.cmml"><mi id="alg2.15.15.m1.1.1.1.1" xref="alg2.15.15.m1.1.1.1.1.cmml">k</mi><mo id="alg2.15.15.m1.2.2.2.4.1" xref="alg2.15.15.m1.2.2.2.3.cmml">,</mo><mi id="alg2.15.15.m1.2.2.2.2" xref="alg2.15.15.m1.2.2.2.2.cmml">v</mi></mrow><mi id="alg2.15.15.m1.2.3.2.3" xref="alg2.15.15.m1.2.3.2.3.cmml">l</mi></msubsup><mo id="alg2.15.15.m1.2.3.1" xref="alg2.15.15.m1.2.3.1.cmml">∈</mo><msup id="alg2.15.15.m1.2.3.3" xref="alg2.15.15.m1.2.3.3.cmml"><mi id="alg2.15.15.m1.2.3.3.2" xref="alg2.15.15.m1.2.3.3.2.cmml">ℝ</mi><mrow id="alg2.15.15.m1.2.3.3.3" xref="alg2.15.15.m1.2.3.3.3.cmml"><mn id="alg2.15.15.m1.2.3.3.3.2" xref="alg2.15.15.m1.2.3.3.3.2.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="alg2.15.15.m1.2.3.3.3.1" xref="alg2.15.15.m1.2.3.3.3.1.cmml">×</mo><msub id="alg2.15.15.m1.2.3.3.3.3" xref="alg2.15.15.m1.2.3.3.3.3.cmml"><mi id="alg2.15.15.m1.2.3.3.3.3.2" xref="alg2.15.15.m1.2.3.3.3.3.2.cmml">t</mi><mtext id="alg2.15.15.m1.2.3.3.3.3.3" xref="alg2.15.15.m1.2.3.3.3.3.3a.cmml">prompt</mtext></msub><mo lspace="0.222em" rspace="0.222em" id="alg2.15.15.m1.2.3.3.3.1a" xref="alg2.15.15.m1.2.3.3.3.1.cmml">×</mo><mi id="alg2.15.15.m1.2.3.3.3.4" xref="alg2.15.15.m1.2.3.3.3.4.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="alg2.15.15.m1.2b"><apply id="alg2.15.15.m1.2.3.cmml" xref="alg2.15.15.m1.2.3"><in id="alg2.15.15.m1.2.3.1.cmml" xref="alg2.15.15.m1.2.3.1"></in><apply id="alg2.15.15.m1.2.3.2.cmml" xref="alg2.15.15.m1.2.3.2"><csymbol cd="ambiguous" id="alg2.15.15.m1.2.3.2.1.cmml" xref="alg2.15.15.m1.2.3.2">superscript</csymbol><apply id="alg2.15.15.m1.2.3.2.2.cmml" xref="alg2.15.15.m1.2.3.2"><csymbol cd="ambiguous" id="alg2.15.15.m1.2.3.2.2.1.cmml" xref="alg2.15.15.m1.2.3.2">subscript</csymbol><ci id="alg2.15.15.m1.2.3.2.2.2.cmml" xref="alg2.15.15.m1.2.3.2.2.2">𝑬</ci><list id="alg2.15.15.m1.2.2.2.3.cmml" xref="alg2.15.15.m1.2.2.2.4"><ci id="alg2.15.15.m1.1.1.1.1.cmml" xref="alg2.15.15.m1.1.1.1.1">𝑘</ci><ci id="alg2.15.15.m1.2.2.2.2.cmml" xref="alg2.15.15.m1.2.2.2.2">𝑣</ci></list></apply><ci id="alg2.15.15.m1.2.3.2.3.cmml" xref="alg2.15.15.m1.2.3.2.3">𝑙</ci></apply><apply id="alg2.15.15.m1.2.3.3.cmml" xref="alg2.15.15.m1.2.3.3"><csymbol cd="ambiguous" id="alg2.15.15.m1.2.3.3.1.cmml" xref="alg2.15.15.m1.2.3.3">superscript</csymbol><ci id="alg2.15.15.m1.2.3.3.2.cmml" xref="alg2.15.15.m1.2.3.3.2">ℝ</ci><apply id="alg2.15.15.m1.2.3.3.3.cmml" xref="alg2.15.15.m1.2.3.3.3"><times id="alg2.15.15.m1.2.3.3.3.1.cmml" xref="alg2.15.15.m1.2.3.3.3.1"></times><cn type="integer" id="alg2.15.15.m1.2.3.3.3.2.cmml" xref="alg2.15.15.m1.2.3.3.3.2">2</cn><apply id="alg2.15.15.m1.2.3.3.3.3.cmml" xref="alg2.15.15.m1.2.3.3.3.3"><csymbol cd="ambiguous" id="alg2.15.15.m1.2.3.3.3.3.1.cmml" xref="alg2.15.15.m1.2.3.3.3.3">subscript</csymbol><ci id="alg2.15.15.m1.2.3.3.3.3.2.cmml" xref="alg2.15.15.m1.2.3.3.3.3.2">𝑡</ci><ci id="alg2.15.15.m1.2.3.3.3.3.3a.cmml" xref="alg2.15.15.m1.2.3.3.3.3.3"><mtext mathsize="50%" id="alg2.15.15.m1.2.3.3.3.3.3.cmml" xref="alg2.15.15.m1.2.3.3.3.3.3">prompt</mtext></ci></apply><ci id="alg2.15.15.m1.2.3.3.3.4.cmml" xref="alg2.15.15.m1.2.3.3.3.4">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.15.15.m1.2c">{\bm{E}}_{k,v}^{l}\in\mathbb{R}^{2\times t_{\text{prompt}}\times d}</annotation></semantics></math>, Norm: <math id="alg2.16.16.m2.3" class="ltx_Math" alttext="\lVert{\bm{X}}_{k,v}^{l}\rVert\in\mathbb{R}^{2\times t_{\text{prompt}}\times 1}" display="inline"><semantics id="alg2.16.16.m2.3a"><mrow id="alg2.16.16.m2.3.3" xref="alg2.16.16.m2.3.3.cmml"><mrow id="alg2.16.16.m2.3.3.1.1" xref="alg2.16.16.m2.3.3.1.2.cmml"><mo fence="true" rspace="0em" id="alg2.16.16.m2.3.3.1.1.2" xref="alg2.16.16.m2.3.3.1.2.1.cmml">∥</mo><msubsup id="alg2.16.16.m2.3.3.1.1.1" xref="alg2.16.16.m2.3.3.1.1.1.cmml"><mi id="alg2.16.16.m2.3.3.1.1.1.2.2" xref="alg2.16.16.m2.3.3.1.1.1.2.2.cmml">𝑿</mi><mrow id="alg2.16.16.m2.2.2.2.4" xref="alg2.16.16.m2.2.2.2.3.cmml"><mi id="alg2.16.16.m2.1.1.1.1" xref="alg2.16.16.m2.1.1.1.1.cmml">k</mi><mo id="alg2.16.16.m2.2.2.2.4.1" xref="alg2.16.16.m2.2.2.2.3.cmml">,</mo><mi id="alg2.16.16.m2.2.2.2.2" xref="alg2.16.16.m2.2.2.2.2.cmml">v</mi></mrow><mi id="alg2.16.16.m2.3.3.1.1.1.3" xref="alg2.16.16.m2.3.3.1.1.1.3.cmml">l</mi></msubsup><mo fence="true" lspace="0em" rspace="0.1389em" id="alg2.16.16.m2.3.3.1.1.3" xref="alg2.16.16.m2.3.3.1.2.1.cmml">∥</mo></mrow><mo lspace="0.1389em" id="alg2.16.16.m2.3.3.2" xref="alg2.16.16.m2.3.3.2.cmml">∈</mo><msup id="alg2.16.16.m2.3.3.3" xref="alg2.16.16.m2.3.3.3.cmml"><mi id="alg2.16.16.m2.3.3.3.2" xref="alg2.16.16.m2.3.3.3.2.cmml">ℝ</mi><mrow id="alg2.16.16.m2.3.3.3.3" xref="alg2.16.16.m2.3.3.3.3.cmml"><mn id="alg2.16.16.m2.3.3.3.3.2" xref="alg2.16.16.m2.3.3.3.3.2.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="alg2.16.16.m2.3.3.3.3.1" xref="alg2.16.16.m2.3.3.3.3.1.cmml">×</mo><msub id="alg2.16.16.m2.3.3.3.3.3" xref="alg2.16.16.m2.3.3.3.3.3.cmml"><mi id="alg2.16.16.m2.3.3.3.3.3.2" xref="alg2.16.16.m2.3.3.3.3.3.2.cmml">t</mi><mtext id="alg2.16.16.m2.3.3.3.3.3.3" xref="alg2.16.16.m2.3.3.3.3.3.3a.cmml">prompt</mtext></msub><mo lspace="0.222em" rspace="0.222em" id="alg2.16.16.m2.3.3.3.3.1a" xref="alg2.16.16.m2.3.3.3.3.1.cmml">×</mo><mn id="alg2.16.16.m2.3.3.3.3.4" xref="alg2.16.16.m2.3.3.3.3.4.cmml">1</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="alg2.16.16.m2.3b"><apply id="alg2.16.16.m2.3.3.cmml" xref="alg2.16.16.m2.3.3"><in id="alg2.16.16.m2.3.3.2.cmml" xref="alg2.16.16.m2.3.3.2"></in><apply id="alg2.16.16.m2.3.3.1.2.cmml" xref="alg2.16.16.m2.3.3.1.1"><csymbol cd="latexml" id="alg2.16.16.m2.3.3.1.2.1.cmml" xref="alg2.16.16.m2.3.3.1.1.2">delimited-∥∥</csymbol><apply id="alg2.16.16.m2.3.3.1.1.1.cmml" xref="alg2.16.16.m2.3.3.1.1.1"><csymbol cd="ambiguous" id="alg2.16.16.m2.3.3.1.1.1.1.cmml" xref="alg2.16.16.m2.3.3.1.1.1">superscript</csymbol><apply id="alg2.16.16.m2.3.3.1.1.1.2.cmml" xref="alg2.16.16.m2.3.3.1.1.1"><csymbol cd="ambiguous" id="alg2.16.16.m2.3.3.1.1.1.2.1.cmml" xref="alg2.16.16.m2.3.3.1.1.1">subscript</csymbol><ci id="alg2.16.16.m2.3.3.1.1.1.2.2.cmml" xref="alg2.16.16.m2.3.3.1.1.1.2.2">𝑿</ci><list id="alg2.16.16.m2.2.2.2.3.cmml" xref="alg2.16.16.m2.2.2.2.4"><ci id="alg2.16.16.m2.1.1.1.1.cmml" xref="alg2.16.16.m2.1.1.1.1">𝑘</ci><ci id="alg2.16.16.m2.2.2.2.2.cmml" xref="alg2.16.16.m2.2.2.2.2">𝑣</ci></list></apply><ci id="alg2.16.16.m2.3.3.1.1.1.3.cmml" xref="alg2.16.16.m2.3.3.1.1.1.3">𝑙</ci></apply></apply><apply id="alg2.16.16.m2.3.3.3.cmml" xref="alg2.16.16.m2.3.3.3"><csymbol cd="ambiguous" id="alg2.16.16.m2.3.3.3.1.cmml" xref="alg2.16.16.m2.3.3.3">superscript</csymbol><ci id="alg2.16.16.m2.3.3.3.2.cmml" xref="alg2.16.16.m2.3.3.3.2">ℝ</ci><apply id="alg2.16.16.m2.3.3.3.3.cmml" xref="alg2.16.16.m2.3.3.3.3"><times id="alg2.16.16.m2.3.3.3.3.1.cmml" xref="alg2.16.16.m2.3.3.3.3.1"></times><cn type="integer" id="alg2.16.16.m2.3.3.3.3.2.cmml" xref="alg2.16.16.m2.3.3.3.3.2">2</cn><apply id="alg2.16.16.m2.3.3.3.3.3.cmml" xref="alg2.16.16.m2.3.3.3.3.3"><csymbol cd="ambiguous" id="alg2.16.16.m2.3.3.3.3.3.1.cmml" xref="alg2.16.16.m2.3.3.3.3.3">subscript</csymbol><ci id="alg2.16.16.m2.3.3.3.3.3.2.cmml" xref="alg2.16.16.m2.3.3.3.3.3.2">𝑡</ci><ci id="alg2.16.16.m2.3.3.3.3.3.3a.cmml" xref="alg2.16.16.m2.3.3.3.3.3.3"><mtext mathsize="50%" id="alg2.16.16.m2.3.3.3.3.3.3.cmml" xref="alg2.16.16.m2.3.3.3.3.3.3">prompt</mtext></ci></apply><cn type="integer" id="alg2.16.16.m2.3.3.3.3.4.cmml" xref="alg2.16.16.m2.3.3.3.3.4">1</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.16.16.m2.3c">\lVert{\bm{X}}_{k,v}^{l}\rVert\in\mathbb{R}^{2\times t_{\text{prompt}}\times 1}</annotation></semantics></math>, Unmerged Tokens: <math id="alg2.17.17.m3.2" class="ltx_Math" alttext="{\bm{R}}_{k,v}^{l}\in\mathbb{R}^{2\times\gamma\cdot t_{\text{prompt}}\times d}" display="inline"><semantics id="alg2.17.17.m3.2a"><mrow id="alg2.17.17.m3.2.3" xref="alg2.17.17.m3.2.3.cmml"><msubsup id="alg2.17.17.m3.2.3.2" xref="alg2.17.17.m3.2.3.2.cmml"><mi id="alg2.17.17.m3.2.3.2.2.2" xref="alg2.17.17.m3.2.3.2.2.2.cmml">𝑹</mi><mrow id="alg2.17.17.m3.2.2.2.4" xref="alg2.17.17.m3.2.2.2.3.cmml"><mi id="alg2.17.17.m3.1.1.1.1" xref="alg2.17.17.m3.1.1.1.1.cmml">k</mi><mo id="alg2.17.17.m3.2.2.2.4.1" xref="alg2.17.17.m3.2.2.2.3.cmml">,</mo><mi id="alg2.17.17.m3.2.2.2.2" xref="alg2.17.17.m3.2.2.2.2.cmml">v</mi></mrow><mi id="alg2.17.17.m3.2.3.2.3" xref="alg2.17.17.m3.2.3.2.3.cmml">l</mi></msubsup><mo id="alg2.17.17.m3.2.3.1" xref="alg2.17.17.m3.2.3.1.cmml">∈</mo><msup id="alg2.17.17.m3.2.3.3" xref="alg2.17.17.m3.2.3.3.cmml"><mi id="alg2.17.17.m3.2.3.3.2" xref="alg2.17.17.m3.2.3.3.2.cmml">ℝ</mi><mrow id="alg2.17.17.m3.2.3.3.3" xref="alg2.17.17.m3.2.3.3.3.cmml"><mrow id="alg2.17.17.m3.2.3.3.3.2" xref="alg2.17.17.m3.2.3.3.3.2.cmml"><mrow id="alg2.17.17.m3.2.3.3.3.2.2" xref="alg2.17.17.m3.2.3.3.3.2.2.cmml"><mn id="alg2.17.17.m3.2.3.3.3.2.2.2" xref="alg2.17.17.m3.2.3.3.3.2.2.2.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="alg2.17.17.m3.2.3.3.3.2.2.1" xref="alg2.17.17.m3.2.3.3.3.2.2.1.cmml">×</mo><mi id="alg2.17.17.m3.2.3.3.3.2.2.3" xref="alg2.17.17.m3.2.3.3.3.2.2.3.cmml">γ</mi></mrow><mo lspace="0.222em" rspace="0.222em" id="alg2.17.17.m3.2.3.3.3.2.1" xref="alg2.17.17.m3.2.3.3.3.2.1.cmml">⋅</mo><msub id="alg2.17.17.m3.2.3.3.3.2.3" xref="alg2.17.17.m3.2.3.3.3.2.3.cmml"><mi id="alg2.17.17.m3.2.3.3.3.2.3.2" xref="alg2.17.17.m3.2.3.3.3.2.3.2.cmml">t</mi><mtext id="alg2.17.17.m3.2.3.3.3.2.3.3" xref="alg2.17.17.m3.2.3.3.3.2.3.3a.cmml">prompt</mtext></msub></mrow><mo lspace="0.222em" rspace="0.222em" id="alg2.17.17.m3.2.3.3.3.1" xref="alg2.17.17.m3.2.3.3.3.1.cmml">×</mo><mi id="alg2.17.17.m3.2.3.3.3.3" xref="alg2.17.17.m3.2.3.3.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="alg2.17.17.m3.2b"><apply id="alg2.17.17.m3.2.3.cmml" xref="alg2.17.17.m3.2.3"><in id="alg2.17.17.m3.2.3.1.cmml" xref="alg2.17.17.m3.2.3.1"></in><apply id="alg2.17.17.m3.2.3.2.cmml" xref="alg2.17.17.m3.2.3.2"><csymbol cd="ambiguous" id="alg2.17.17.m3.2.3.2.1.cmml" xref="alg2.17.17.m3.2.3.2">superscript</csymbol><apply id="alg2.17.17.m3.2.3.2.2.cmml" xref="alg2.17.17.m3.2.3.2"><csymbol cd="ambiguous" id="alg2.17.17.m3.2.3.2.2.1.cmml" xref="alg2.17.17.m3.2.3.2">subscript</csymbol><ci id="alg2.17.17.m3.2.3.2.2.2.cmml" xref="alg2.17.17.m3.2.3.2.2.2">𝑹</ci><list id="alg2.17.17.m3.2.2.2.3.cmml" xref="alg2.17.17.m3.2.2.2.4"><ci id="alg2.17.17.m3.1.1.1.1.cmml" xref="alg2.17.17.m3.1.1.1.1">𝑘</ci><ci id="alg2.17.17.m3.2.2.2.2.cmml" xref="alg2.17.17.m3.2.2.2.2">𝑣</ci></list></apply><ci id="alg2.17.17.m3.2.3.2.3.cmml" xref="alg2.17.17.m3.2.3.2.3">𝑙</ci></apply><apply id="alg2.17.17.m3.2.3.3.cmml" xref="alg2.17.17.m3.2.3.3"><csymbol cd="ambiguous" id="alg2.17.17.m3.2.3.3.1.cmml" xref="alg2.17.17.m3.2.3.3">superscript</csymbol><ci id="alg2.17.17.m3.2.3.3.2.cmml" xref="alg2.17.17.m3.2.3.3.2">ℝ</ci><apply id="alg2.17.17.m3.2.3.3.3.cmml" xref="alg2.17.17.m3.2.3.3.3"><times id="alg2.17.17.m3.2.3.3.3.1.cmml" xref="alg2.17.17.m3.2.3.3.3.1"></times><apply id="alg2.17.17.m3.2.3.3.3.2.cmml" xref="alg2.17.17.m3.2.3.3.3.2"><ci id="alg2.17.17.m3.2.3.3.3.2.1.cmml" xref="alg2.17.17.m3.2.3.3.3.2.1">⋅</ci><apply id="alg2.17.17.m3.2.3.3.3.2.2.cmml" xref="alg2.17.17.m3.2.3.3.3.2.2"><times id="alg2.17.17.m3.2.3.3.3.2.2.1.cmml" xref="alg2.17.17.m3.2.3.3.3.2.2.1"></times><cn type="integer" id="alg2.17.17.m3.2.3.3.3.2.2.2.cmml" xref="alg2.17.17.m3.2.3.3.3.2.2.2">2</cn><ci id="alg2.17.17.m3.2.3.3.3.2.2.3.cmml" xref="alg2.17.17.m3.2.3.3.3.2.2.3">𝛾</ci></apply><apply id="alg2.17.17.m3.2.3.3.3.2.3.cmml" xref="alg2.17.17.m3.2.3.3.3.2.3"><csymbol cd="ambiguous" id="alg2.17.17.m3.2.3.3.3.2.3.1.cmml" xref="alg2.17.17.m3.2.3.3.3.2.3">subscript</csymbol><ci id="alg2.17.17.m3.2.3.3.3.2.3.2.cmml" xref="alg2.17.17.m3.2.3.3.3.2.3.2">𝑡</ci><ci id="alg2.17.17.m3.2.3.3.3.2.3.3a.cmml" xref="alg2.17.17.m3.2.3.3.3.2.3.3"><mtext mathsize="50%" id="alg2.17.17.m3.2.3.3.3.2.3.3.cmml" xref="alg2.17.17.m3.2.3.3.3.2.3.3">prompt</mtext></ci></apply></apply><ci id="alg2.17.17.m3.2.3.3.3.3.cmml" xref="alg2.17.17.m3.2.3.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.17.17.m3.2c">{\bm{R}}_{k,v}^{l}\in\mathbb{R}^{2\times\gamma\cdot t_{\text{prompt}}\times d}</annotation></semantics></math>, Retention indices: <math id="alg2.18.18.m4.2" class="ltx_Math" alttext="{\mathbb{I}}_{k,v}\in\mathbb{R}^{2\times\gamma\cdot t_{\text{prompt}}\times 1}" display="inline"><semantics id="alg2.18.18.m4.2a"><mrow id="alg2.18.18.m4.2.3" xref="alg2.18.18.m4.2.3.cmml"><msub id="alg2.18.18.m4.2.3.2" xref="alg2.18.18.m4.2.3.2.cmml"><mi id="alg2.18.18.m4.2.3.2.2" xref="alg2.18.18.m4.2.3.2.2.cmml">𝕀</mi><mrow id="alg2.18.18.m4.2.2.2.4" xref="alg2.18.18.m4.2.2.2.3.cmml"><mi id="alg2.18.18.m4.1.1.1.1" xref="alg2.18.18.m4.1.1.1.1.cmml">k</mi><mo id="alg2.18.18.m4.2.2.2.4.1" xref="alg2.18.18.m4.2.2.2.3.cmml">,</mo><mi id="alg2.18.18.m4.2.2.2.2" xref="alg2.18.18.m4.2.2.2.2.cmml">v</mi></mrow></msub><mo id="alg2.18.18.m4.2.3.1" xref="alg2.18.18.m4.2.3.1.cmml">∈</mo><msup id="alg2.18.18.m4.2.3.3" xref="alg2.18.18.m4.2.3.3.cmml"><mi id="alg2.18.18.m4.2.3.3.2" xref="alg2.18.18.m4.2.3.3.2.cmml">ℝ</mi><mrow id="alg2.18.18.m4.2.3.3.3" xref="alg2.18.18.m4.2.3.3.3.cmml"><mrow id="alg2.18.18.m4.2.3.3.3.2" xref="alg2.18.18.m4.2.3.3.3.2.cmml"><mrow id="alg2.18.18.m4.2.3.3.3.2.2" xref="alg2.18.18.m4.2.3.3.3.2.2.cmml"><mn id="alg2.18.18.m4.2.3.3.3.2.2.2" xref="alg2.18.18.m4.2.3.3.3.2.2.2.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="alg2.18.18.m4.2.3.3.3.2.2.1" xref="alg2.18.18.m4.2.3.3.3.2.2.1.cmml">×</mo><mi id="alg2.18.18.m4.2.3.3.3.2.2.3" xref="alg2.18.18.m4.2.3.3.3.2.2.3.cmml">γ</mi></mrow><mo lspace="0.222em" rspace="0.222em" id="alg2.18.18.m4.2.3.3.3.2.1" xref="alg2.18.18.m4.2.3.3.3.2.1.cmml">⋅</mo><msub id="alg2.18.18.m4.2.3.3.3.2.3" xref="alg2.18.18.m4.2.3.3.3.2.3.cmml"><mi id="alg2.18.18.m4.2.3.3.3.2.3.2" xref="alg2.18.18.m4.2.3.3.3.2.3.2.cmml">t</mi><mtext id="alg2.18.18.m4.2.3.3.3.2.3.3" xref="alg2.18.18.m4.2.3.3.3.2.3.3a.cmml">prompt</mtext></msub></mrow><mo lspace="0.222em" rspace="0.222em" id="alg2.18.18.m4.2.3.3.3.1" xref="alg2.18.18.m4.2.3.3.3.1.cmml">×</mo><mn id="alg2.18.18.m4.2.3.3.3.3" xref="alg2.18.18.m4.2.3.3.3.3.cmml">1</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="alg2.18.18.m4.2b"><apply id="alg2.18.18.m4.2.3.cmml" xref="alg2.18.18.m4.2.3"><in id="alg2.18.18.m4.2.3.1.cmml" xref="alg2.18.18.m4.2.3.1"></in><apply id="alg2.18.18.m4.2.3.2.cmml" xref="alg2.18.18.m4.2.3.2"><csymbol cd="ambiguous" id="alg2.18.18.m4.2.3.2.1.cmml" xref="alg2.18.18.m4.2.3.2">subscript</csymbol><ci id="alg2.18.18.m4.2.3.2.2.cmml" xref="alg2.18.18.m4.2.3.2.2">𝕀</ci><list id="alg2.18.18.m4.2.2.2.3.cmml" xref="alg2.18.18.m4.2.2.2.4"><ci id="alg2.18.18.m4.1.1.1.1.cmml" xref="alg2.18.18.m4.1.1.1.1">𝑘</ci><ci id="alg2.18.18.m4.2.2.2.2.cmml" xref="alg2.18.18.m4.2.2.2.2">𝑣</ci></list></apply><apply id="alg2.18.18.m4.2.3.3.cmml" xref="alg2.18.18.m4.2.3.3"><csymbol cd="ambiguous" id="alg2.18.18.m4.2.3.3.1.cmml" xref="alg2.18.18.m4.2.3.3">superscript</csymbol><ci id="alg2.18.18.m4.2.3.3.2.cmml" xref="alg2.18.18.m4.2.3.3.2">ℝ</ci><apply id="alg2.18.18.m4.2.3.3.3.cmml" xref="alg2.18.18.m4.2.3.3.3"><times id="alg2.18.18.m4.2.3.3.3.1.cmml" xref="alg2.18.18.m4.2.3.3.3.1"></times><apply id="alg2.18.18.m4.2.3.3.3.2.cmml" xref="alg2.18.18.m4.2.3.3.3.2"><ci id="alg2.18.18.m4.2.3.3.3.2.1.cmml" xref="alg2.18.18.m4.2.3.3.3.2.1">⋅</ci><apply id="alg2.18.18.m4.2.3.3.3.2.2.cmml" xref="alg2.18.18.m4.2.3.3.3.2.2"><times id="alg2.18.18.m4.2.3.3.3.2.2.1.cmml" xref="alg2.18.18.m4.2.3.3.3.2.2.1"></times><cn type="integer" id="alg2.18.18.m4.2.3.3.3.2.2.2.cmml" xref="alg2.18.18.m4.2.3.3.3.2.2.2">2</cn><ci id="alg2.18.18.m4.2.3.3.3.2.2.3.cmml" xref="alg2.18.18.m4.2.3.3.3.2.2.3">𝛾</ci></apply><apply id="alg2.18.18.m4.2.3.3.3.2.3.cmml" xref="alg2.18.18.m4.2.3.3.3.2.3"><csymbol cd="ambiguous" id="alg2.18.18.m4.2.3.3.3.2.3.1.cmml" xref="alg2.18.18.m4.2.3.3.3.2.3">subscript</csymbol><ci id="alg2.18.18.m4.2.3.3.3.2.3.2.cmml" xref="alg2.18.18.m4.2.3.3.3.2.3.2">𝑡</ci><ci id="alg2.18.18.m4.2.3.3.3.2.3.3a.cmml" xref="alg2.18.18.m4.2.3.3.3.2.3.3"><mtext mathsize="50%" id="alg2.18.18.m4.2.3.3.3.2.3.3.cmml" xref="alg2.18.18.m4.2.3.3.3.2.3.3">prompt</mtext></ci></apply></apply><cn type="integer" id="alg2.18.18.m4.2.3.3.3.3.cmml" xref="alg2.18.18.m4.2.3.3.3.3">1</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.18.18.m4.2c">{\mathbb{I}}_{k,v}\in\mathbb{R}^{2\times\gamma\cdot t_{\text{prompt}}\times 1}</annotation></semantics></math>, Next Token: <math id="alg2.19.19.m5.1" class="ltx_Math" alttext="{\bm{t}}^{l}\in\mathbb{R}^{1\times d}" display="inline"><semantics id="alg2.19.19.m5.1a"><mrow id="alg2.19.19.m5.1.1" xref="alg2.19.19.m5.1.1.cmml"><msup id="alg2.19.19.m5.1.1.2" xref="alg2.19.19.m5.1.1.2.cmml"><mi id="alg2.19.19.m5.1.1.2.2" xref="alg2.19.19.m5.1.1.2.2.cmml">𝒕</mi><mi id="alg2.19.19.m5.1.1.2.3" xref="alg2.19.19.m5.1.1.2.3.cmml">l</mi></msup><mo id="alg2.19.19.m5.1.1.1" xref="alg2.19.19.m5.1.1.1.cmml">∈</mo><msup id="alg2.19.19.m5.1.1.3" xref="alg2.19.19.m5.1.1.3.cmml"><mi id="alg2.19.19.m5.1.1.3.2" xref="alg2.19.19.m5.1.1.3.2.cmml">ℝ</mi><mrow id="alg2.19.19.m5.1.1.3.3" xref="alg2.19.19.m5.1.1.3.3.cmml"><mn id="alg2.19.19.m5.1.1.3.3.2" xref="alg2.19.19.m5.1.1.3.3.2.cmml">1</mn><mo lspace="0.222em" rspace="0.222em" id="alg2.19.19.m5.1.1.3.3.1" xref="alg2.19.19.m5.1.1.3.3.1.cmml">×</mo><mi id="alg2.19.19.m5.1.1.3.3.3" xref="alg2.19.19.m5.1.1.3.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="alg2.19.19.m5.1b"><apply id="alg2.19.19.m5.1.1.cmml" xref="alg2.19.19.m5.1.1"><in id="alg2.19.19.m5.1.1.1.cmml" xref="alg2.19.19.m5.1.1.1"></in><apply id="alg2.19.19.m5.1.1.2.cmml" xref="alg2.19.19.m5.1.1.2"><csymbol cd="ambiguous" id="alg2.19.19.m5.1.1.2.1.cmml" xref="alg2.19.19.m5.1.1.2">superscript</csymbol><ci id="alg2.19.19.m5.1.1.2.2.cmml" xref="alg2.19.19.m5.1.1.2.2">𝒕</ci><ci id="alg2.19.19.m5.1.1.2.3.cmml" xref="alg2.19.19.m5.1.1.2.3">𝑙</ci></apply><apply id="alg2.19.19.m5.1.1.3.cmml" xref="alg2.19.19.m5.1.1.3"><csymbol cd="ambiguous" id="alg2.19.19.m5.1.1.3.1.cmml" xref="alg2.19.19.m5.1.1.3">superscript</csymbol><ci id="alg2.19.19.m5.1.1.3.2.cmml" xref="alg2.19.19.m5.1.1.3.2">ℝ</ci><apply id="alg2.19.19.m5.1.1.3.3.cmml" xref="alg2.19.19.m5.1.1.3.3"><times id="alg2.19.19.m5.1.1.3.3.1.cmml" xref="alg2.19.19.m5.1.1.3.3.1"></times><cn type="integer" id="alg2.19.19.m5.1.1.3.3.2.cmml" xref="alg2.19.19.m5.1.1.3.3.2">1</cn><ci id="alg2.19.19.m5.1.1.3.3.3.cmml" xref="alg2.19.19.m5.1.1.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.19.19.m5.1c">{\bm{t}}^{l}\in\mathbb{R}^{1\times d}</annotation></semantics></math>, <math id="alg2.20.20.m6.1" class="ltx_Math" alttext="{\bm{t}}^{l-1}\in\mathbb{R}^{1\times d}" display="inline"><semantics id="alg2.20.20.m6.1a"><mrow id="alg2.20.20.m6.1.1" xref="alg2.20.20.m6.1.1.cmml"><msup id="alg2.20.20.m6.1.1.2" xref="alg2.20.20.m6.1.1.2.cmml"><mi id="alg2.20.20.m6.1.1.2.2" xref="alg2.20.20.m6.1.1.2.2.cmml">𝒕</mi><mrow id="alg2.20.20.m6.1.1.2.3" xref="alg2.20.20.m6.1.1.2.3.cmml"><mi id="alg2.20.20.m6.1.1.2.3.2" xref="alg2.20.20.m6.1.1.2.3.2.cmml">l</mi><mo id="alg2.20.20.m6.1.1.2.3.1" xref="alg2.20.20.m6.1.1.2.3.1.cmml">−</mo><mn id="alg2.20.20.m6.1.1.2.3.3" xref="alg2.20.20.m6.1.1.2.3.3.cmml">1</mn></mrow></msup><mo id="alg2.20.20.m6.1.1.1" xref="alg2.20.20.m6.1.1.1.cmml">∈</mo><msup id="alg2.20.20.m6.1.1.3" xref="alg2.20.20.m6.1.1.3.cmml"><mi id="alg2.20.20.m6.1.1.3.2" xref="alg2.20.20.m6.1.1.3.2.cmml">ℝ</mi><mrow id="alg2.20.20.m6.1.1.3.3" xref="alg2.20.20.m6.1.1.3.3.cmml"><mn id="alg2.20.20.m6.1.1.3.3.2" xref="alg2.20.20.m6.1.1.3.3.2.cmml">1</mn><mo lspace="0.222em" rspace="0.222em" id="alg2.20.20.m6.1.1.3.3.1" xref="alg2.20.20.m6.1.1.3.3.1.cmml">×</mo><mi id="alg2.20.20.m6.1.1.3.3.3" xref="alg2.20.20.m6.1.1.3.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="alg2.20.20.m6.1b"><apply id="alg2.20.20.m6.1.1.cmml" xref="alg2.20.20.m6.1.1"><in id="alg2.20.20.m6.1.1.1.cmml" xref="alg2.20.20.m6.1.1.1"></in><apply id="alg2.20.20.m6.1.1.2.cmml" xref="alg2.20.20.m6.1.1.2"><csymbol cd="ambiguous" id="alg2.20.20.m6.1.1.2.1.cmml" xref="alg2.20.20.m6.1.1.2">superscript</csymbol><ci id="alg2.20.20.m6.1.1.2.2.cmml" xref="alg2.20.20.m6.1.1.2.2">𝒕</ci><apply id="alg2.20.20.m6.1.1.2.3.cmml" xref="alg2.20.20.m6.1.1.2.3"><minus id="alg2.20.20.m6.1.1.2.3.1.cmml" xref="alg2.20.20.m6.1.1.2.3.1"></minus><ci id="alg2.20.20.m6.1.1.2.3.2.cmml" xref="alg2.20.20.m6.1.1.2.3.2">𝑙</ci><cn type="integer" id="alg2.20.20.m6.1.1.2.3.3.cmml" xref="alg2.20.20.m6.1.1.2.3.3">1</cn></apply></apply><apply id="alg2.20.20.m6.1.1.3.cmml" xref="alg2.20.20.m6.1.1.3"><csymbol cd="ambiguous" id="alg2.20.20.m6.1.1.3.1.cmml" xref="alg2.20.20.m6.1.1.3">superscript</csymbol><ci id="alg2.20.20.m6.1.1.3.2.cmml" xref="alg2.20.20.m6.1.1.3.2">ℝ</ci><apply id="alg2.20.20.m6.1.1.3.3.cmml" xref="alg2.20.20.m6.1.1.3.3"><times id="alg2.20.20.m6.1.1.3.3.1.cmml" xref="alg2.20.20.m6.1.1.3.3.1"></times><cn type="integer" id="alg2.20.20.m6.1.1.3.3.2.cmml" xref="alg2.20.20.m6.1.1.3.3.2">1</cn><ci id="alg2.20.20.m6.1.1.3.3.3.cmml" xref="alg2.20.20.m6.1.1.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.20.20.m6.1c">{\bm{t}}^{l-1}\in\mathbb{R}^{1\times d}</annotation></semantics></math>
</div>
<div id="alg2.22.22" class="ltx_listingline">  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<math id="alg2.21.21.m1.2" class="ltx_Math" alttext="\hat{{\bm{X}}}_{k}^{l}\leftarrow{\bm{E}}^{l}_{k}\cdot\frac{\lVert{\bm{X}}_{k}^{l}\rVert}{\lVert{\bm{E}}^{l}_{k}\rVert}" display="inline"><semantics id="alg2.21.21.m1.2a"><mrow id="alg2.21.21.m1.2.3" xref="alg2.21.21.m1.2.3.cmml"><msubsup id="alg2.21.21.m1.2.3.2" xref="alg2.21.21.m1.2.3.2.cmml"><mover accent="true" id="alg2.21.21.m1.2.3.2.2.2" xref="alg2.21.21.m1.2.3.2.2.2.cmml"><mi id="alg2.21.21.m1.2.3.2.2.2.2" xref="alg2.21.21.m1.2.3.2.2.2.2.cmml">𝑿</mi><mo id="alg2.21.21.m1.2.3.2.2.2.1" xref="alg2.21.21.m1.2.3.2.2.2.1.cmml">^</mo></mover><mi id="alg2.21.21.m1.2.3.2.2.3" xref="alg2.21.21.m1.2.3.2.2.3.cmml">k</mi><mi id="alg2.21.21.m1.2.3.2.3" xref="alg2.21.21.m1.2.3.2.3.cmml">l</mi></msubsup><mo stretchy="false" id="alg2.21.21.m1.2.3.1" xref="alg2.21.21.m1.2.3.1.cmml">←</mo><mrow id="alg2.21.21.m1.2.3.3" xref="alg2.21.21.m1.2.3.3.cmml"><msubsup id="alg2.21.21.m1.2.3.3.2" xref="alg2.21.21.m1.2.3.3.2.cmml"><mi id="alg2.21.21.m1.2.3.3.2.2.2" xref="alg2.21.21.m1.2.3.3.2.2.2.cmml">𝑬</mi><mi id="alg2.21.21.m1.2.3.3.2.3" xref="alg2.21.21.m1.2.3.3.2.3.cmml">k</mi><mi id="alg2.21.21.m1.2.3.3.2.2.3" xref="alg2.21.21.m1.2.3.3.2.2.3.cmml">l</mi></msubsup><mo lspace="0.222em" rspace="0.222em" id="alg2.21.21.m1.2.3.3.1" xref="alg2.21.21.m1.2.3.3.1.cmml">⋅</mo><mfrac id="alg2.21.21.m1.2.2" xref="alg2.21.21.m1.2.2.cmml"><mrow id="alg2.21.21.m1.1.1.1.1" xref="alg2.21.21.m1.1.1.1.2.cmml"><mo fence="true" rspace="0em" id="alg2.21.21.m1.1.1.1.1.2" xref="alg2.21.21.m1.1.1.1.2.1.cmml">∥</mo><msubsup id="alg2.21.21.m1.1.1.1.1.1" xref="alg2.21.21.m1.1.1.1.1.1.cmml"><mi id="alg2.21.21.m1.1.1.1.1.1.2.2" xref="alg2.21.21.m1.1.1.1.1.1.2.2.cmml">𝑿</mi><mi id="alg2.21.21.m1.1.1.1.1.1.2.3" xref="alg2.21.21.m1.1.1.1.1.1.2.3.cmml">k</mi><mi id="alg2.21.21.m1.1.1.1.1.1.3" xref="alg2.21.21.m1.1.1.1.1.1.3.cmml">l</mi></msubsup><mo fence="true" lspace="0em" id="alg2.21.21.m1.1.1.1.1.3" xref="alg2.21.21.m1.1.1.1.2.1.cmml">∥</mo></mrow><mrow id="alg2.21.21.m1.2.2.2.1" xref="alg2.21.21.m1.2.2.2.2.cmml"><mo fence="true" rspace="0em" id="alg2.21.21.m1.2.2.2.1.2" xref="alg2.21.21.m1.2.2.2.2.1.cmml">∥</mo><msubsup id="alg2.21.21.m1.2.2.2.1.1" xref="alg2.21.21.m1.2.2.2.1.1.cmml"><mi id="alg2.21.21.m1.2.2.2.1.1.2.2" xref="alg2.21.21.m1.2.2.2.1.1.2.2.cmml">𝑬</mi><mi id="alg2.21.21.m1.2.2.2.1.1.3" xref="alg2.21.21.m1.2.2.2.1.1.3.cmml">k</mi><mi id="alg2.21.21.m1.2.2.2.1.1.2.3" xref="alg2.21.21.m1.2.2.2.1.1.2.3.cmml">l</mi></msubsup><mo fence="true" lspace="0em" id="alg2.21.21.m1.2.2.2.1.3" xref="alg2.21.21.m1.2.2.2.2.1.cmml">∥</mo></mrow></mfrac></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.21.21.m1.2b"><apply id="alg2.21.21.m1.2.3.cmml" xref="alg2.21.21.m1.2.3"><ci id="alg2.21.21.m1.2.3.1.cmml" xref="alg2.21.21.m1.2.3.1">←</ci><apply id="alg2.21.21.m1.2.3.2.cmml" xref="alg2.21.21.m1.2.3.2"><csymbol cd="ambiguous" id="alg2.21.21.m1.2.3.2.1.cmml" xref="alg2.21.21.m1.2.3.2">superscript</csymbol><apply id="alg2.21.21.m1.2.3.2.2.cmml" xref="alg2.21.21.m1.2.3.2"><csymbol cd="ambiguous" id="alg2.21.21.m1.2.3.2.2.1.cmml" xref="alg2.21.21.m1.2.3.2">subscript</csymbol><apply id="alg2.21.21.m1.2.3.2.2.2.cmml" xref="alg2.21.21.m1.2.3.2.2.2"><ci id="alg2.21.21.m1.2.3.2.2.2.1.cmml" xref="alg2.21.21.m1.2.3.2.2.2.1">^</ci><ci id="alg2.21.21.m1.2.3.2.2.2.2.cmml" xref="alg2.21.21.m1.2.3.2.2.2.2">𝑿</ci></apply><ci id="alg2.21.21.m1.2.3.2.2.3.cmml" xref="alg2.21.21.m1.2.3.2.2.3">𝑘</ci></apply><ci id="alg2.21.21.m1.2.3.2.3.cmml" xref="alg2.21.21.m1.2.3.2.3">𝑙</ci></apply><apply id="alg2.21.21.m1.2.3.3.cmml" xref="alg2.21.21.m1.2.3.3"><ci id="alg2.21.21.m1.2.3.3.1.cmml" xref="alg2.21.21.m1.2.3.3.1">⋅</ci><apply id="alg2.21.21.m1.2.3.3.2.cmml" xref="alg2.21.21.m1.2.3.3.2"><csymbol cd="ambiguous" id="alg2.21.21.m1.2.3.3.2.1.cmml" xref="alg2.21.21.m1.2.3.3.2">subscript</csymbol><apply id="alg2.21.21.m1.2.3.3.2.2.cmml" xref="alg2.21.21.m1.2.3.3.2"><csymbol cd="ambiguous" id="alg2.21.21.m1.2.3.3.2.2.1.cmml" xref="alg2.21.21.m1.2.3.3.2">superscript</csymbol><ci id="alg2.21.21.m1.2.3.3.2.2.2.cmml" xref="alg2.21.21.m1.2.3.3.2.2.2">𝑬</ci><ci id="alg2.21.21.m1.2.3.3.2.2.3.cmml" xref="alg2.21.21.m1.2.3.3.2.2.3">𝑙</ci></apply><ci id="alg2.21.21.m1.2.3.3.2.3.cmml" xref="alg2.21.21.m1.2.3.3.2.3">𝑘</ci></apply><apply id="alg2.21.21.m1.2.2.cmml" xref="alg2.21.21.m1.2.2"><divide id="alg2.21.21.m1.2.2.3.cmml" xref="alg2.21.21.m1.2.2"></divide><apply id="alg2.21.21.m1.1.1.1.2.cmml" xref="alg2.21.21.m1.1.1.1.1"><csymbol cd="latexml" id="alg2.21.21.m1.1.1.1.2.1.cmml" xref="alg2.21.21.m1.1.1.1.1.2">delimited-∥∥</csymbol><apply id="alg2.21.21.m1.1.1.1.1.1.cmml" xref="alg2.21.21.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.21.21.m1.1.1.1.1.1.1.cmml" xref="alg2.21.21.m1.1.1.1.1.1">superscript</csymbol><apply id="alg2.21.21.m1.1.1.1.1.1.2.cmml" xref="alg2.21.21.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.21.21.m1.1.1.1.1.1.2.1.cmml" xref="alg2.21.21.m1.1.1.1.1.1">subscript</csymbol><ci id="alg2.21.21.m1.1.1.1.1.1.2.2.cmml" xref="alg2.21.21.m1.1.1.1.1.1.2.2">𝑿</ci><ci id="alg2.21.21.m1.1.1.1.1.1.2.3.cmml" xref="alg2.21.21.m1.1.1.1.1.1.2.3">𝑘</ci></apply><ci id="alg2.21.21.m1.1.1.1.1.1.3.cmml" xref="alg2.21.21.m1.1.1.1.1.1.3">𝑙</ci></apply></apply><apply id="alg2.21.21.m1.2.2.2.2.cmml" xref="alg2.21.21.m1.2.2.2.1"><csymbol cd="latexml" id="alg2.21.21.m1.2.2.2.2.1.cmml" xref="alg2.21.21.m1.2.2.2.1.2">delimited-∥∥</csymbol><apply id="alg2.21.21.m1.2.2.2.1.1.cmml" xref="alg2.21.21.m1.2.2.2.1.1"><csymbol cd="ambiguous" id="alg2.21.21.m1.2.2.2.1.1.1.cmml" xref="alg2.21.21.m1.2.2.2.1.1">subscript</csymbol><apply id="alg2.21.21.m1.2.2.2.1.1.2.cmml" xref="alg2.21.21.m1.2.2.2.1.1"><csymbol cd="ambiguous" id="alg2.21.21.m1.2.2.2.1.1.2.1.cmml" xref="alg2.21.21.m1.2.2.2.1.1">superscript</csymbol><ci id="alg2.21.21.m1.2.2.2.1.1.2.2.cmml" xref="alg2.21.21.m1.2.2.2.1.1.2.2">𝑬</ci><ci id="alg2.21.21.m1.2.2.2.1.1.2.3.cmml" xref="alg2.21.21.m1.2.2.2.1.1.2.3">𝑙</ci></apply><ci id="alg2.21.21.m1.2.2.2.1.1.3.cmml" xref="alg2.21.21.m1.2.2.2.1.1.3">𝑘</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.21.21.m1.2c">\hat{{\bm{X}}}_{k}^{l}\leftarrow{\bm{E}}^{l}_{k}\cdot\frac{\lVert{\bm{X}}_{k}^{l}\rVert}{\lVert{\bm{E}}^{l}_{k}\rVert}</annotation></semantics></math>
<math id="alg2.22.22.m2.2" class="ltx_Math" alttext="\hat{{\bm{X}}}_{v}^{l}\leftarrow{\bm{E}}^{l}_{v}\cdot\frac{\lVert{\bm{X}}_{v}^{l}\rVert}{\lVert{\bm{E}}^{l}_{v}\rVert}" display="inline"><semantics id="alg2.22.22.m2.2a"><mrow id="alg2.22.22.m2.2.3" xref="alg2.22.22.m2.2.3.cmml"><msubsup id="alg2.22.22.m2.2.3.2" xref="alg2.22.22.m2.2.3.2.cmml"><mover accent="true" id="alg2.22.22.m2.2.3.2.2.2" xref="alg2.22.22.m2.2.3.2.2.2.cmml"><mi id="alg2.22.22.m2.2.3.2.2.2.2" xref="alg2.22.22.m2.2.3.2.2.2.2.cmml">𝑿</mi><mo id="alg2.22.22.m2.2.3.2.2.2.1" xref="alg2.22.22.m2.2.3.2.2.2.1.cmml">^</mo></mover><mi id="alg2.22.22.m2.2.3.2.2.3" xref="alg2.22.22.m2.2.3.2.2.3.cmml">v</mi><mi id="alg2.22.22.m2.2.3.2.3" xref="alg2.22.22.m2.2.3.2.3.cmml">l</mi></msubsup><mo stretchy="false" id="alg2.22.22.m2.2.3.1" xref="alg2.22.22.m2.2.3.1.cmml">←</mo><mrow id="alg2.22.22.m2.2.3.3" xref="alg2.22.22.m2.2.3.3.cmml"><msubsup id="alg2.22.22.m2.2.3.3.2" xref="alg2.22.22.m2.2.3.3.2.cmml"><mi id="alg2.22.22.m2.2.3.3.2.2.2" xref="alg2.22.22.m2.2.3.3.2.2.2.cmml">𝑬</mi><mi id="alg2.22.22.m2.2.3.3.2.3" xref="alg2.22.22.m2.2.3.3.2.3.cmml">v</mi><mi id="alg2.22.22.m2.2.3.3.2.2.3" xref="alg2.22.22.m2.2.3.3.2.2.3.cmml">l</mi></msubsup><mo lspace="0.222em" rspace="0.222em" id="alg2.22.22.m2.2.3.3.1" xref="alg2.22.22.m2.2.3.3.1.cmml">⋅</mo><mfrac id="alg2.22.22.m2.2.2" xref="alg2.22.22.m2.2.2.cmml"><mrow id="alg2.22.22.m2.1.1.1.1" xref="alg2.22.22.m2.1.1.1.2.cmml"><mo fence="true" rspace="0em" id="alg2.22.22.m2.1.1.1.1.2" xref="alg2.22.22.m2.1.1.1.2.1.cmml">∥</mo><msubsup id="alg2.22.22.m2.1.1.1.1.1" xref="alg2.22.22.m2.1.1.1.1.1.cmml"><mi id="alg2.22.22.m2.1.1.1.1.1.2.2" xref="alg2.22.22.m2.1.1.1.1.1.2.2.cmml">𝑿</mi><mi id="alg2.22.22.m2.1.1.1.1.1.2.3" xref="alg2.22.22.m2.1.1.1.1.1.2.3.cmml">v</mi><mi id="alg2.22.22.m2.1.1.1.1.1.3" xref="alg2.22.22.m2.1.1.1.1.1.3.cmml">l</mi></msubsup><mo fence="true" lspace="0em" id="alg2.22.22.m2.1.1.1.1.3" xref="alg2.22.22.m2.1.1.1.2.1.cmml">∥</mo></mrow><mrow id="alg2.22.22.m2.2.2.2.1" xref="alg2.22.22.m2.2.2.2.2.cmml"><mo fence="true" rspace="0em" id="alg2.22.22.m2.2.2.2.1.2" xref="alg2.22.22.m2.2.2.2.2.1.cmml">∥</mo><msubsup id="alg2.22.22.m2.2.2.2.1.1" xref="alg2.22.22.m2.2.2.2.1.1.cmml"><mi id="alg2.22.22.m2.2.2.2.1.1.2.2" xref="alg2.22.22.m2.2.2.2.1.1.2.2.cmml">𝑬</mi><mi id="alg2.22.22.m2.2.2.2.1.1.3" xref="alg2.22.22.m2.2.2.2.1.1.3.cmml">v</mi><mi id="alg2.22.22.m2.2.2.2.1.1.2.3" xref="alg2.22.22.m2.2.2.2.1.1.2.3.cmml">l</mi></msubsup><mo fence="true" lspace="0em" id="alg2.22.22.m2.2.2.2.1.3" xref="alg2.22.22.m2.2.2.2.2.1.cmml">∥</mo></mrow></mfrac></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.22.22.m2.2b"><apply id="alg2.22.22.m2.2.3.cmml" xref="alg2.22.22.m2.2.3"><ci id="alg2.22.22.m2.2.3.1.cmml" xref="alg2.22.22.m2.2.3.1">←</ci><apply id="alg2.22.22.m2.2.3.2.cmml" xref="alg2.22.22.m2.2.3.2"><csymbol cd="ambiguous" id="alg2.22.22.m2.2.3.2.1.cmml" xref="alg2.22.22.m2.2.3.2">superscript</csymbol><apply id="alg2.22.22.m2.2.3.2.2.cmml" xref="alg2.22.22.m2.2.3.2"><csymbol cd="ambiguous" id="alg2.22.22.m2.2.3.2.2.1.cmml" xref="alg2.22.22.m2.2.3.2">subscript</csymbol><apply id="alg2.22.22.m2.2.3.2.2.2.cmml" xref="alg2.22.22.m2.2.3.2.2.2"><ci id="alg2.22.22.m2.2.3.2.2.2.1.cmml" xref="alg2.22.22.m2.2.3.2.2.2.1">^</ci><ci id="alg2.22.22.m2.2.3.2.2.2.2.cmml" xref="alg2.22.22.m2.2.3.2.2.2.2">𝑿</ci></apply><ci id="alg2.22.22.m2.2.3.2.2.3.cmml" xref="alg2.22.22.m2.2.3.2.2.3">𝑣</ci></apply><ci id="alg2.22.22.m2.2.3.2.3.cmml" xref="alg2.22.22.m2.2.3.2.3">𝑙</ci></apply><apply id="alg2.22.22.m2.2.3.3.cmml" xref="alg2.22.22.m2.2.3.3"><ci id="alg2.22.22.m2.2.3.3.1.cmml" xref="alg2.22.22.m2.2.3.3.1">⋅</ci><apply id="alg2.22.22.m2.2.3.3.2.cmml" xref="alg2.22.22.m2.2.3.3.2"><csymbol cd="ambiguous" id="alg2.22.22.m2.2.3.3.2.1.cmml" xref="alg2.22.22.m2.2.3.3.2">subscript</csymbol><apply id="alg2.22.22.m2.2.3.3.2.2.cmml" xref="alg2.22.22.m2.2.3.3.2"><csymbol cd="ambiguous" id="alg2.22.22.m2.2.3.3.2.2.1.cmml" xref="alg2.22.22.m2.2.3.3.2">superscript</csymbol><ci id="alg2.22.22.m2.2.3.3.2.2.2.cmml" xref="alg2.22.22.m2.2.3.3.2.2.2">𝑬</ci><ci id="alg2.22.22.m2.2.3.3.2.2.3.cmml" xref="alg2.22.22.m2.2.3.3.2.2.3">𝑙</ci></apply><ci id="alg2.22.22.m2.2.3.3.2.3.cmml" xref="alg2.22.22.m2.2.3.3.2.3">𝑣</ci></apply><apply id="alg2.22.22.m2.2.2.cmml" xref="alg2.22.22.m2.2.2"><divide id="alg2.22.22.m2.2.2.3.cmml" xref="alg2.22.22.m2.2.2"></divide><apply id="alg2.22.22.m2.1.1.1.2.cmml" xref="alg2.22.22.m2.1.1.1.1"><csymbol cd="latexml" id="alg2.22.22.m2.1.1.1.2.1.cmml" xref="alg2.22.22.m2.1.1.1.1.2">delimited-∥∥</csymbol><apply id="alg2.22.22.m2.1.1.1.1.1.cmml" xref="alg2.22.22.m2.1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.22.22.m2.1.1.1.1.1.1.cmml" xref="alg2.22.22.m2.1.1.1.1.1">superscript</csymbol><apply id="alg2.22.22.m2.1.1.1.1.1.2.cmml" xref="alg2.22.22.m2.1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.22.22.m2.1.1.1.1.1.2.1.cmml" xref="alg2.22.22.m2.1.1.1.1.1">subscript</csymbol><ci id="alg2.22.22.m2.1.1.1.1.1.2.2.cmml" xref="alg2.22.22.m2.1.1.1.1.1.2.2">𝑿</ci><ci id="alg2.22.22.m2.1.1.1.1.1.2.3.cmml" xref="alg2.22.22.m2.1.1.1.1.1.2.3">𝑣</ci></apply><ci id="alg2.22.22.m2.1.1.1.1.1.3.cmml" xref="alg2.22.22.m2.1.1.1.1.1.3">𝑙</ci></apply></apply><apply id="alg2.22.22.m2.2.2.2.2.cmml" xref="alg2.22.22.m2.2.2.2.1"><csymbol cd="latexml" id="alg2.22.22.m2.2.2.2.2.1.cmml" xref="alg2.22.22.m2.2.2.2.1.2">delimited-∥∥</csymbol><apply id="alg2.22.22.m2.2.2.2.1.1.cmml" xref="alg2.22.22.m2.2.2.2.1.1"><csymbol cd="ambiguous" id="alg2.22.22.m2.2.2.2.1.1.1.cmml" xref="alg2.22.22.m2.2.2.2.1.1">subscript</csymbol><apply id="alg2.22.22.m2.2.2.2.1.1.2.cmml" xref="alg2.22.22.m2.2.2.2.1.1"><csymbol cd="ambiguous" id="alg2.22.22.m2.2.2.2.1.1.2.1.cmml" xref="alg2.22.22.m2.2.2.2.1.1">superscript</csymbol><ci id="alg2.22.22.m2.2.2.2.1.1.2.2.cmml" xref="alg2.22.22.m2.2.2.2.1.1.2.2">𝑬</ci><ci id="alg2.22.22.m2.2.2.2.1.1.2.3.cmml" xref="alg2.22.22.m2.2.2.2.1.1.2.3">𝑙</ci></apply><ci id="alg2.22.22.m2.2.2.2.1.1.3.cmml" xref="alg2.22.22.m2.2.2.2.1.1.3">𝑣</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.22.22.m2.2c">\hat{{\bm{X}}}_{v}^{l}\leftarrow{\bm{E}}^{l}_{v}\cdot\frac{\lVert{\bm{X}}_{v}^{l}\rVert}{\lVert{\bm{E}}^{l}_{v}\rVert}</annotation></semantics></math> <span id="alg2.22.22.1" class="ltx_text ltx_font_typewriter">// </span><span id="alg2.22.22.2" class="ltx_text ltx_font_typewriter">magnitude rescale </span>
</div>
<div id="alg2.24.24" class="ltx_listingline">  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<math id="alg2.23.23.m1.1" class="ltx_Math" alttext="\hat{{\bm{X}}}_{k}^{l}[{\mathbb{I}}_{k}]={\bm{R}}_{k}^{l}" display="inline"><semantics id="alg2.23.23.m1.1a"><mrow id="alg2.23.23.m1.1.1" xref="alg2.23.23.m1.1.1.cmml"><mrow id="alg2.23.23.m1.1.1.1" xref="alg2.23.23.m1.1.1.1.cmml"><msubsup id="alg2.23.23.m1.1.1.1.3" xref="alg2.23.23.m1.1.1.1.3.cmml"><mover accent="true" id="alg2.23.23.m1.1.1.1.3.2.2" xref="alg2.23.23.m1.1.1.1.3.2.2.cmml"><mi id="alg2.23.23.m1.1.1.1.3.2.2.2" xref="alg2.23.23.m1.1.1.1.3.2.2.2.cmml">𝑿</mi><mo id="alg2.23.23.m1.1.1.1.3.2.2.1" xref="alg2.23.23.m1.1.1.1.3.2.2.1.cmml">^</mo></mover><mi id="alg2.23.23.m1.1.1.1.3.2.3" xref="alg2.23.23.m1.1.1.1.3.2.3.cmml">k</mi><mi id="alg2.23.23.m1.1.1.1.3.3" xref="alg2.23.23.m1.1.1.1.3.3.cmml">l</mi></msubsup><mo lspace="0em" rspace="0em" id="alg2.23.23.m1.1.1.1.2" xref="alg2.23.23.m1.1.1.1.2.cmml">​</mo><mrow id="alg2.23.23.m1.1.1.1.1.1" xref="alg2.23.23.m1.1.1.1.1.2.cmml"><mo stretchy="false" id="alg2.23.23.m1.1.1.1.1.1.2" xref="alg2.23.23.m1.1.1.1.1.2.1.cmml">[</mo><msub id="alg2.23.23.m1.1.1.1.1.1.1" xref="alg2.23.23.m1.1.1.1.1.1.1.cmml"><mi id="alg2.23.23.m1.1.1.1.1.1.1.2" xref="alg2.23.23.m1.1.1.1.1.1.1.2.cmml">𝕀</mi><mi id="alg2.23.23.m1.1.1.1.1.1.1.3" xref="alg2.23.23.m1.1.1.1.1.1.1.3.cmml">k</mi></msub><mo stretchy="false" id="alg2.23.23.m1.1.1.1.1.1.3" xref="alg2.23.23.m1.1.1.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="alg2.23.23.m1.1.1.2" xref="alg2.23.23.m1.1.1.2.cmml">=</mo><msubsup id="alg2.23.23.m1.1.1.3" xref="alg2.23.23.m1.1.1.3.cmml"><mi id="alg2.23.23.m1.1.1.3.2.2" xref="alg2.23.23.m1.1.1.3.2.2.cmml">𝑹</mi><mi id="alg2.23.23.m1.1.1.3.2.3" xref="alg2.23.23.m1.1.1.3.2.3.cmml">k</mi><mi id="alg2.23.23.m1.1.1.3.3" xref="alg2.23.23.m1.1.1.3.3.cmml">l</mi></msubsup></mrow><annotation-xml encoding="MathML-Content" id="alg2.23.23.m1.1b"><apply id="alg2.23.23.m1.1.1.cmml" xref="alg2.23.23.m1.1.1"><eq id="alg2.23.23.m1.1.1.2.cmml" xref="alg2.23.23.m1.1.1.2"></eq><apply id="alg2.23.23.m1.1.1.1.cmml" xref="alg2.23.23.m1.1.1.1"><times id="alg2.23.23.m1.1.1.1.2.cmml" xref="alg2.23.23.m1.1.1.1.2"></times><apply id="alg2.23.23.m1.1.1.1.3.cmml" xref="alg2.23.23.m1.1.1.1.3"><csymbol cd="ambiguous" id="alg2.23.23.m1.1.1.1.3.1.cmml" xref="alg2.23.23.m1.1.1.1.3">superscript</csymbol><apply id="alg2.23.23.m1.1.1.1.3.2.cmml" xref="alg2.23.23.m1.1.1.1.3"><csymbol cd="ambiguous" id="alg2.23.23.m1.1.1.1.3.2.1.cmml" xref="alg2.23.23.m1.1.1.1.3">subscript</csymbol><apply id="alg2.23.23.m1.1.1.1.3.2.2.cmml" xref="alg2.23.23.m1.1.1.1.3.2.2"><ci id="alg2.23.23.m1.1.1.1.3.2.2.1.cmml" xref="alg2.23.23.m1.1.1.1.3.2.2.1">^</ci><ci id="alg2.23.23.m1.1.1.1.3.2.2.2.cmml" xref="alg2.23.23.m1.1.1.1.3.2.2.2">𝑿</ci></apply><ci id="alg2.23.23.m1.1.1.1.3.2.3.cmml" xref="alg2.23.23.m1.1.1.1.3.2.3">𝑘</ci></apply><ci id="alg2.23.23.m1.1.1.1.3.3.cmml" xref="alg2.23.23.m1.1.1.1.3.3">𝑙</ci></apply><apply id="alg2.23.23.m1.1.1.1.1.2.cmml" xref="alg2.23.23.m1.1.1.1.1.1"><csymbol cd="latexml" id="alg2.23.23.m1.1.1.1.1.2.1.cmml" xref="alg2.23.23.m1.1.1.1.1.1.2">delimited-[]</csymbol><apply id="alg2.23.23.m1.1.1.1.1.1.1.cmml" xref="alg2.23.23.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.23.23.m1.1.1.1.1.1.1.1.cmml" xref="alg2.23.23.m1.1.1.1.1.1.1">subscript</csymbol><ci id="alg2.23.23.m1.1.1.1.1.1.1.2.cmml" xref="alg2.23.23.m1.1.1.1.1.1.1.2">𝕀</ci><ci id="alg2.23.23.m1.1.1.1.1.1.1.3.cmml" xref="alg2.23.23.m1.1.1.1.1.1.1.3">𝑘</ci></apply></apply></apply><apply id="alg2.23.23.m1.1.1.3.cmml" xref="alg2.23.23.m1.1.1.3"><csymbol cd="ambiguous" id="alg2.23.23.m1.1.1.3.1.cmml" xref="alg2.23.23.m1.1.1.3">superscript</csymbol><apply id="alg2.23.23.m1.1.1.3.2.cmml" xref="alg2.23.23.m1.1.1.3"><csymbol cd="ambiguous" id="alg2.23.23.m1.1.1.3.2.1.cmml" xref="alg2.23.23.m1.1.1.3">subscript</csymbol><ci id="alg2.23.23.m1.1.1.3.2.2.cmml" xref="alg2.23.23.m1.1.1.3.2.2">𝑹</ci><ci id="alg2.23.23.m1.1.1.3.2.3.cmml" xref="alg2.23.23.m1.1.1.3.2.3">𝑘</ci></apply><ci id="alg2.23.23.m1.1.1.3.3.cmml" xref="alg2.23.23.m1.1.1.3.3">𝑙</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.23.23.m1.1c">\hat{{\bm{X}}}_{k}^{l}[{\mathbb{I}}_{k}]={\bm{R}}_{k}^{l}</annotation></semantics></math>
<math id="alg2.24.24.m2.1" class="ltx_Math" alttext="\hat{{\bm{X}}}_{v}^{l}[{\mathbb{I}}_{v}]={\bm{R}}_{v}^{l}" display="inline"><semantics id="alg2.24.24.m2.1a"><mrow id="alg2.24.24.m2.1.1" xref="alg2.24.24.m2.1.1.cmml"><mrow id="alg2.24.24.m2.1.1.1" xref="alg2.24.24.m2.1.1.1.cmml"><msubsup id="alg2.24.24.m2.1.1.1.3" xref="alg2.24.24.m2.1.1.1.3.cmml"><mover accent="true" id="alg2.24.24.m2.1.1.1.3.2.2" xref="alg2.24.24.m2.1.1.1.3.2.2.cmml"><mi id="alg2.24.24.m2.1.1.1.3.2.2.2" xref="alg2.24.24.m2.1.1.1.3.2.2.2.cmml">𝑿</mi><mo id="alg2.24.24.m2.1.1.1.3.2.2.1" xref="alg2.24.24.m2.1.1.1.3.2.2.1.cmml">^</mo></mover><mi id="alg2.24.24.m2.1.1.1.3.2.3" xref="alg2.24.24.m2.1.1.1.3.2.3.cmml">v</mi><mi id="alg2.24.24.m2.1.1.1.3.3" xref="alg2.24.24.m2.1.1.1.3.3.cmml">l</mi></msubsup><mo lspace="0em" rspace="0em" id="alg2.24.24.m2.1.1.1.2" xref="alg2.24.24.m2.1.1.1.2.cmml">​</mo><mrow id="alg2.24.24.m2.1.1.1.1.1" xref="alg2.24.24.m2.1.1.1.1.2.cmml"><mo stretchy="false" id="alg2.24.24.m2.1.1.1.1.1.2" xref="alg2.24.24.m2.1.1.1.1.2.1.cmml">[</mo><msub id="alg2.24.24.m2.1.1.1.1.1.1" xref="alg2.24.24.m2.1.1.1.1.1.1.cmml"><mi id="alg2.24.24.m2.1.1.1.1.1.1.2" xref="alg2.24.24.m2.1.1.1.1.1.1.2.cmml">𝕀</mi><mi id="alg2.24.24.m2.1.1.1.1.1.1.3" xref="alg2.24.24.m2.1.1.1.1.1.1.3.cmml">v</mi></msub><mo stretchy="false" id="alg2.24.24.m2.1.1.1.1.1.3" xref="alg2.24.24.m2.1.1.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="alg2.24.24.m2.1.1.2" xref="alg2.24.24.m2.1.1.2.cmml">=</mo><msubsup id="alg2.24.24.m2.1.1.3" xref="alg2.24.24.m2.1.1.3.cmml"><mi id="alg2.24.24.m2.1.1.3.2.2" xref="alg2.24.24.m2.1.1.3.2.2.cmml">𝑹</mi><mi id="alg2.24.24.m2.1.1.3.2.3" xref="alg2.24.24.m2.1.1.3.2.3.cmml">v</mi><mi id="alg2.24.24.m2.1.1.3.3" xref="alg2.24.24.m2.1.1.3.3.cmml">l</mi></msubsup></mrow><annotation-xml encoding="MathML-Content" id="alg2.24.24.m2.1b"><apply id="alg2.24.24.m2.1.1.cmml" xref="alg2.24.24.m2.1.1"><eq id="alg2.24.24.m2.1.1.2.cmml" xref="alg2.24.24.m2.1.1.2"></eq><apply id="alg2.24.24.m2.1.1.1.cmml" xref="alg2.24.24.m2.1.1.1"><times id="alg2.24.24.m2.1.1.1.2.cmml" xref="alg2.24.24.m2.1.1.1.2"></times><apply id="alg2.24.24.m2.1.1.1.3.cmml" xref="alg2.24.24.m2.1.1.1.3"><csymbol cd="ambiguous" id="alg2.24.24.m2.1.1.1.3.1.cmml" xref="alg2.24.24.m2.1.1.1.3">superscript</csymbol><apply id="alg2.24.24.m2.1.1.1.3.2.cmml" xref="alg2.24.24.m2.1.1.1.3"><csymbol cd="ambiguous" id="alg2.24.24.m2.1.1.1.3.2.1.cmml" xref="alg2.24.24.m2.1.1.1.3">subscript</csymbol><apply id="alg2.24.24.m2.1.1.1.3.2.2.cmml" xref="alg2.24.24.m2.1.1.1.3.2.2"><ci id="alg2.24.24.m2.1.1.1.3.2.2.1.cmml" xref="alg2.24.24.m2.1.1.1.3.2.2.1">^</ci><ci id="alg2.24.24.m2.1.1.1.3.2.2.2.cmml" xref="alg2.24.24.m2.1.1.1.3.2.2.2">𝑿</ci></apply><ci id="alg2.24.24.m2.1.1.1.3.2.3.cmml" xref="alg2.24.24.m2.1.1.1.3.2.3">𝑣</ci></apply><ci id="alg2.24.24.m2.1.1.1.3.3.cmml" xref="alg2.24.24.m2.1.1.1.3.3">𝑙</ci></apply><apply id="alg2.24.24.m2.1.1.1.1.2.cmml" xref="alg2.24.24.m2.1.1.1.1.1"><csymbol cd="latexml" id="alg2.24.24.m2.1.1.1.1.2.1.cmml" xref="alg2.24.24.m2.1.1.1.1.1.2">delimited-[]</csymbol><apply id="alg2.24.24.m2.1.1.1.1.1.1.cmml" xref="alg2.24.24.m2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.24.24.m2.1.1.1.1.1.1.1.cmml" xref="alg2.24.24.m2.1.1.1.1.1.1">subscript</csymbol><ci id="alg2.24.24.m2.1.1.1.1.1.1.2.cmml" xref="alg2.24.24.m2.1.1.1.1.1.1.2">𝕀</ci><ci id="alg2.24.24.m2.1.1.1.1.1.1.3.cmml" xref="alg2.24.24.m2.1.1.1.1.1.1.3">𝑣</ci></apply></apply></apply><apply id="alg2.24.24.m2.1.1.3.cmml" xref="alg2.24.24.m2.1.1.3"><csymbol cd="ambiguous" id="alg2.24.24.m2.1.1.3.1.cmml" xref="alg2.24.24.m2.1.1.3">superscript</csymbol><apply id="alg2.24.24.m2.1.1.3.2.cmml" xref="alg2.24.24.m2.1.1.3"><csymbol cd="ambiguous" id="alg2.24.24.m2.1.1.3.2.1.cmml" xref="alg2.24.24.m2.1.1.3">subscript</csymbol><ci id="alg2.24.24.m2.1.1.3.2.2.cmml" xref="alg2.24.24.m2.1.1.3.2.2">𝑹</ci><ci id="alg2.24.24.m2.1.1.3.2.3.cmml" xref="alg2.24.24.m2.1.1.3.2.3">𝑣</ci></apply><ci id="alg2.24.24.m2.1.1.3.3.cmml" xref="alg2.24.24.m2.1.1.3.3">𝑙</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.24.24.m2.1c">\hat{{\bm{X}}}_{v}^{l}[{\mathbb{I}}_{v}]={\bm{R}}_{v}^{l}</annotation></semantics></math> <span id="alg2.24.24.1" class="ltx_text ltx_font_typewriter">// </span><span id="alg2.24.24.2" class="ltx_text ltx_font_typewriter">token restoration </span>
</div>
<div id="alg2.29.29" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">12</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<math id="alg2.25.25.m1.3" class="ltx_Math" alttext="\hat{{\bm{X}}}_{k}^{l}\leftarrow\text{Concat}(\hat{{\bm{X}}}_{k}^{l},{\bm{t}}_{k},\text{dim=token})" display="inline"><semantics id="alg2.25.25.m1.3a"><mrow id="alg2.25.25.m1.3.3" xref="alg2.25.25.m1.3.3.cmml"><msubsup id="alg2.25.25.m1.3.3.4" xref="alg2.25.25.m1.3.3.4.cmml"><mover accent="true" id="alg2.25.25.m1.3.3.4.2.2" xref="alg2.25.25.m1.3.3.4.2.2.cmml"><mi id="alg2.25.25.m1.3.3.4.2.2.2" xref="alg2.25.25.m1.3.3.4.2.2.2.cmml">𝑿</mi><mo id="alg2.25.25.m1.3.3.4.2.2.1" xref="alg2.25.25.m1.3.3.4.2.2.1.cmml">^</mo></mover><mi id="alg2.25.25.m1.3.3.4.2.3" xref="alg2.25.25.m1.3.3.4.2.3.cmml">k</mi><mi id="alg2.25.25.m1.3.3.4.3" xref="alg2.25.25.m1.3.3.4.3.cmml">l</mi></msubsup><mo stretchy="false" id="alg2.25.25.m1.3.3.3" xref="alg2.25.25.m1.3.3.3.cmml">←</mo><mrow id="alg2.25.25.m1.3.3.2" xref="alg2.25.25.m1.3.3.2.cmml"><mtext id="alg2.25.25.m1.3.3.2.4" xref="alg2.25.25.m1.3.3.2.4a.cmml">Concat</mtext><mo lspace="0em" rspace="0em" id="alg2.25.25.m1.3.3.2.3" xref="alg2.25.25.m1.3.3.2.3.cmml">​</mo><mrow id="alg2.25.25.m1.3.3.2.2.2" xref="alg2.25.25.m1.3.3.2.2.3.cmml"><mo stretchy="false" id="alg2.25.25.m1.3.3.2.2.2.3" xref="alg2.25.25.m1.3.3.2.2.3.cmml">(</mo><msubsup id="alg2.25.25.m1.2.2.1.1.1.1" xref="alg2.25.25.m1.2.2.1.1.1.1.cmml"><mover accent="true" id="alg2.25.25.m1.2.2.1.1.1.1.2.2" xref="alg2.25.25.m1.2.2.1.1.1.1.2.2.cmml"><mi id="alg2.25.25.m1.2.2.1.1.1.1.2.2.2" xref="alg2.25.25.m1.2.2.1.1.1.1.2.2.2.cmml">𝑿</mi><mo id="alg2.25.25.m1.2.2.1.1.1.1.2.2.1" xref="alg2.25.25.m1.2.2.1.1.1.1.2.2.1.cmml">^</mo></mover><mi id="alg2.25.25.m1.2.2.1.1.1.1.2.3" xref="alg2.25.25.m1.2.2.1.1.1.1.2.3.cmml">k</mi><mi id="alg2.25.25.m1.2.2.1.1.1.1.3" xref="alg2.25.25.m1.2.2.1.1.1.1.3.cmml">l</mi></msubsup><mo id="alg2.25.25.m1.3.3.2.2.2.4" xref="alg2.25.25.m1.3.3.2.2.3.cmml">,</mo><msub id="alg2.25.25.m1.3.3.2.2.2.2" xref="alg2.25.25.m1.3.3.2.2.2.2.cmml"><mi id="alg2.25.25.m1.3.3.2.2.2.2.2" xref="alg2.25.25.m1.3.3.2.2.2.2.2.cmml">𝒕</mi><mi id="alg2.25.25.m1.3.3.2.2.2.2.3" xref="alg2.25.25.m1.3.3.2.2.2.2.3.cmml">k</mi></msub><mo id="alg2.25.25.m1.3.3.2.2.2.5" xref="alg2.25.25.m1.3.3.2.2.3.cmml">,</mo><mtext id="alg2.25.25.m1.1.1" xref="alg2.25.25.m1.1.1a.cmml">dim=token</mtext><mo stretchy="false" id="alg2.25.25.m1.3.3.2.2.2.6" xref="alg2.25.25.m1.3.3.2.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.25.25.m1.3b"><apply id="alg2.25.25.m1.3.3.cmml" xref="alg2.25.25.m1.3.3"><ci id="alg2.25.25.m1.3.3.3.cmml" xref="alg2.25.25.m1.3.3.3">←</ci><apply id="alg2.25.25.m1.3.3.4.cmml" xref="alg2.25.25.m1.3.3.4"><csymbol cd="ambiguous" id="alg2.25.25.m1.3.3.4.1.cmml" xref="alg2.25.25.m1.3.3.4">superscript</csymbol><apply id="alg2.25.25.m1.3.3.4.2.cmml" xref="alg2.25.25.m1.3.3.4"><csymbol cd="ambiguous" id="alg2.25.25.m1.3.3.4.2.1.cmml" xref="alg2.25.25.m1.3.3.4">subscript</csymbol><apply id="alg2.25.25.m1.3.3.4.2.2.cmml" xref="alg2.25.25.m1.3.3.4.2.2"><ci id="alg2.25.25.m1.3.3.4.2.2.1.cmml" xref="alg2.25.25.m1.3.3.4.2.2.1">^</ci><ci id="alg2.25.25.m1.3.3.4.2.2.2.cmml" xref="alg2.25.25.m1.3.3.4.2.2.2">𝑿</ci></apply><ci id="alg2.25.25.m1.3.3.4.2.3.cmml" xref="alg2.25.25.m1.3.3.4.2.3">𝑘</ci></apply><ci id="alg2.25.25.m1.3.3.4.3.cmml" xref="alg2.25.25.m1.3.3.4.3">𝑙</ci></apply><apply id="alg2.25.25.m1.3.3.2.cmml" xref="alg2.25.25.m1.3.3.2"><times id="alg2.25.25.m1.3.3.2.3.cmml" xref="alg2.25.25.m1.3.3.2.3"></times><ci id="alg2.25.25.m1.3.3.2.4a.cmml" xref="alg2.25.25.m1.3.3.2.4"><mtext id="alg2.25.25.m1.3.3.2.4.cmml" xref="alg2.25.25.m1.3.3.2.4">Concat</mtext></ci><vector id="alg2.25.25.m1.3.3.2.2.3.cmml" xref="alg2.25.25.m1.3.3.2.2.2"><apply id="alg2.25.25.m1.2.2.1.1.1.1.cmml" xref="alg2.25.25.m1.2.2.1.1.1.1"><csymbol cd="ambiguous" id="alg2.25.25.m1.2.2.1.1.1.1.1.cmml" xref="alg2.25.25.m1.2.2.1.1.1.1">superscript</csymbol><apply id="alg2.25.25.m1.2.2.1.1.1.1.2.cmml" xref="alg2.25.25.m1.2.2.1.1.1.1"><csymbol cd="ambiguous" id="alg2.25.25.m1.2.2.1.1.1.1.2.1.cmml" xref="alg2.25.25.m1.2.2.1.1.1.1">subscript</csymbol><apply id="alg2.25.25.m1.2.2.1.1.1.1.2.2.cmml" xref="alg2.25.25.m1.2.2.1.1.1.1.2.2"><ci id="alg2.25.25.m1.2.2.1.1.1.1.2.2.1.cmml" xref="alg2.25.25.m1.2.2.1.1.1.1.2.2.1">^</ci><ci id="alg2.25.25.m1.2.2.1.1.1.1.2.2.2.cmml" xref="alg2.25.25.m1.2.2.1.1.1.1.2.2.2">𝑿</ci></apply><ci id="alg2.25.25.m1.2.2.1.1.1.1.2.3.cmml" xref="alg2.25.25.m1.2.2.1.1.1.1.2.3">𝑘</ci></apply><ci id="alg2.25.25.m1.2.2.1.1.1.1.3.cmml" xref="alg2.25.25.m1.2.2.1.1.1.1.3">𝑙</ci></apply><apply id="alg2.25.25.m1.3.3.2.2.2.2.cmml" xref="alg2.25.25.m1.3.3.2.2.2.2"><csymbol cd="ambiguous" id="alg2.25.25.m1.3.3.2.2.2.2.1.cmml" xref="alg2.25.25.m1.3.3.2.2.2.2">subscript</csymbol><ci id="alg2.25.25.m1.3.3.2.2.2.2.2.cmml" xref="alg2.25.25.m1.3.3.2.2.2.2.2">𝒕</ci><ci id="alg2.25.25.m1.3.3.2.2.2.2.3.cmml" xref="alg2.25.25.m1.3.3.2.2.2.2.3">𝑘</ci></apply><ci id="alg2.25.25.m1.1.1a.cmml" xref="alg2.25.25.m1.1.1"><mtext id="alg2.25.25.m1.1.1.cmml" xref="alg2.25.25.m1.1.1">dim=token</mtext></ci></vector></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.25.25.m1.3c">\hat{{\bm{X}}}_{k}^{l}\leftarrow\text{Concat}(\hat{{\bm{X}}}_{k}^{l},{\bm{t}}_{k},\text{dim=token})</annotation></semantics></math>
<math id="alg2.26.26.m2.3" class="ltx_Math" alttext="\hat{{\bm{X}}}_{v}^{l}\leftarrow\text{Concat}(\hat{{\bm{X}}}_{v}^{l},{\bm{t}}_{v},\text{dim=token})" display="inline"><semantics id="alg2.26.26.m2.3a"><mrow id="alg2.26.26.m2.3.3" xref="alg2.26.26.m2.3.3.cmml"><msubsup id="alg2.26.26.m2.3.3.4" xref="alg2.26.26.m2.3.3.4.cmml"><mover accent="true" id="alg2.26.26.m2.3.3.4.2.2" xref="alg2.26.26.m2.3.3.4.2.2.cmml"><mi id="alg2.26.26.m2.3.3.4.2.2.2" xref="alg2.26.26.m2.3.3.4.2.2.2.cmml">𝑿</mi><mo id="alg2.26.26.m2.3.3.4.2.2.1" xref="alg2.26.26.m2.3.3.4.2.2.1.cmml">^</mo></mover><mi id="alg2.26.26.m2.3.3.4.2.3" xref="alg2.26.26.m2.3.3.4.2.3.cmml">v</mi><mi id="alg2.26.26.m2.3.3.4.3" xref="alg2.26.26.m2.3.3.4.3.cmml">l</mi></msubsup><mo stretchy="false" id="alg2.26.26.m2.3.3.3" xref="alg2.26.26.m2.3.3.3.cmml">←</mo><mrow id="alg2.26.26.m2.3.3.2" xref="alg2.26.26.m2.3.3.2.cmml"><mtext id="alg2.26.26.m2.3.3.2.4" xref="alg2.26.26.m2.3.3.2.4a.cmml">Concat</mtext><mo lspace="0em" rspace="0em" id="alg2.26.26.m2.3.3.2.3" xref="alg2.26.26.m2.3.3.2.3.cmml">​</mo><mrow id="alg2.26.26.m2.3.3.2.2.2" xref="alg2.26.26.m2.3.3.2.2.3.cmml"><mo stretchy="false" id="alg2.26.26.m2.3.3.2.2.2.3" xref="alg2.26.26.m2.3.3.2.2.3.cmml">(</mo><msubsup id="alg2.26.26.m2.2.2.1.1.1.1" xref="alg2.26.26.m2.2.2.1.1.1.1.cmml"><mover accent="true" id="alg2.26.26.m2.2.2.1.1.1.1.2.2" xref="alg2.26.26.m2.2.2.1.1.1.1.2.2.cmml"><mi id="alg2.26.26.m2.2.2.1.1.1.1.2.2.2" xref="alg2.26.26.m2.2.2.1.1.1.1.2.2.2.cmml">𝑿</mi><mo id="alg2.26.26.m2.2.2.1.1.1.1.2.2.1" xref="alg2.26.26.m2.2.2.1.1.1.1.2.2.1.cmml">^</mo></mover><mi id="alg2.26.26.m2.2.2.1.1.1.1.2.3" xref="alg2.26.26.m2.2.2.1.1.1.1.2.3.cmml">v</mi><mi id="alg2.26.26.m2.2.2.1.1.1.1.3" xref="alg2.26.26.m2.2.2.1.1.1.1.3.cmml">l</mi></msubsup><mo id="alg2.26.26.m2.3.3.2.2.2.4" xref="alg2.26.26.m2.3.3.2.2.3.cmml">,</mo><msub id="alg2.26.26.m2.3.3.2.2.2.2" xref="alg2.26.26.m2.3.3.2.2.2.2.cmml"><mi id="alg2.26.26.m2.3.3.2.2.2.2.2" xref="alg2.26.26.m2.3.3.2.2.2.2.2.cmml">𝒕</mi><mi id="alg2.26.26.m2.3.3.2.2.2.2.3" xref="alg2.26.26.m2.3.3.2.2.2.2.3.cmml">v</mi></msub><mo id="alg2.26.26.m2.3.3.2.2.2.5" xref="alg2.26.26.m2.3.3.2.2.3.cmml">,</mo><mtext id="alg2.26.26.m2.1.1" xref="alg2.26.26.m2.1.1a.cmml">dim=token</mtext><mo stretchy="false" id="alg2.26.26.m2.3.3.2.2.2.6" xref="alg2.26.26.m2.3.3.2.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.26.26.m2.3b"><apply id="alg2.26.26.m2.3.3.cmml" xref="alg2.26.26.m2.3.3"><ci id="alg2.26.26.m2.3.3.3.cmml" xref="alg2.26.26.m2.3.3.3">←</ci><apply id="alg2.26.26.m2.3.3.4.cmml" xref="alg2.26.26.m2.3.3.4"><csymbol cd="ambiguous" id="alg2.26.26.m2.3.3.4.1.cmml" xref="alg2.26.26.m2.3.3.4">superscript</csymbol><apply id="alg2.26.26.m2.3.3.4.2.cmml" xref="alg2.26.26.m2.3.3.4"><csymbol cd="ambiguous" id="alg2.26.26.m2.3.3.4.2.1.cmml" xref="alg2.26.26.m2.3.3.4">subscript</csymbol><apply id="alg2.26.26.m2.3.3.4.2.2.cmml" xref="alg2.26.26.m2.3.3.4.2.2"><ci id="alg2.26.26.m2.3.3.4.2.2.1.cmml" xref="alg2.26.26.m2.3.3.4.2.2.1">^</ci><ci id="alg2.26.26.m2.3.3.4.2.2.2.cmml" xref="alg2.26.26.m2.3.3.4.2.2.2">𝑿</ci></apply><ci id="alg2.26.26.m2.3.3.4.2.3.cmml" xref="alg2.26.26.m2.3.3.4.2.3">𝑣</ci></apply><ci id="alg2.26.26.m2.3.3.4.3.cmml" xref="alg2.26.26.m2.3.3.4.3">𝑙</ci></apply><apply id="alg2.26.26.m2.3.3.2.cmml" xref="alg2.26.26.m2.3.3.2"><times id="alg2.26.26.m2.3.3.2.3.cmml" xref="alg2.26.26.m2.3.3.2.3"></times><ci id="alg2.26.26.m2.3.3.2.4a.cmml" xref="alg2.26.26.m2.3.3.2.4"><mtext id="alg2.26.26.m2.3.3.2.4.cmml" xref="alg2.26.26.m2.3.3.2.4">Concat</mtext></ci><vector id="alg2.26.26.m2.3.3.2.2.3.cmml" xref="alg2.26.26.m2.3.3.2.2.2"><apply id="alg2.26.26.m2.2.2.1.1.1.1.cmml" xref="alg2.26.26.m2.2.2.1.1.1.1"><csymbol cd="ambiguous" id="alg2.26.26.m2.2.2.1.1.1.1.1.cmml" xref="alg2.26.26.m2.2.2.1.1.1.1">superscript</csymbol><apply id="alg2.26.26.m2.2.2.1.1.1.1.2.cmml" xref="alg2.26.26.m2.2.2.1.1.1.1"><csymbol cd="ambiguous" id="alg2.26.26.m2.2.2.1.1.1.1.2.1.cmml" xref="alg2.26.26.m2.2.2.1.1.1.1">subscript</csymbol><apply id="alg2.26.26.m2.2.2.1.1.1.1.2.2.cmml" xref="alg2.26.26.m2.2.2.1.1.1.1.2.2"><ci id="alg2.26.26.m2.2.2.1.1.1.1.2.2.1.cmml" xref="alg2.26.26.m2.2.2.1.1.1.1.2.2.1">^</ci><ci id="alg2.26.26.m2.2.2.1.1.1.1.2.2.2.cmml" xref="alg2.26.26.m2.2.2.1.1.1.1.2.2.2">𝑿</ci></apply><ci id="alg2.26.26.m2.2.2.1.1.1.1.2.3.cmml" xref="alg2.26.26.m2.2.2.1.1.1.1.2.3">𝑣</ci></apply><ci id="alg2.26.26.m2.2.2.1.1.1.1.3.cmml" xref="alg2.26.26.m2.2.2.1.1.1.1.3">𝑙</ci></apply><apply id="alg2.26.26.m2.3.3.2.2.2.2.cmml" xref="alg2.26.26.m2.3.3.2.2.2.2"><csymbol cd="ambiguous" id="alg2.26.26.m2.3.3.2.2.2.2.1.cmml" xref="alg2.26.26.m2.3.3.2.2.2.2">subscript</csymbol><ci id="alg2.26.26.m2.3.3.2.2.2.2.2.cmml" xref="alg2.26.26.m2.3.3.2.2.2.2.2">𝒕</ci><ci id="alg2.26.26.m2.3.3.2.2.2.2.3.cmml" xref="alg2.26.26.m2.3.3.2.2.2.2.3">𝑣</ci></apply><ci id="alg2.26.26.m2.1.1a.cmml" xref="alg2.26.26.m2.1.1"><mtext id="alg2.26.26.m2.1.1.cmml" xref="alg2.26.26.m2.1.1">dim=token</mtext></ci></vector></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.26.26.m2.3c">\hat{{\bm{X}}}_{v}^{l}\leftarrow\text{Concat}(\hat{{\bm{X}}}_{v}^{l},{\bm{t}}_{v},\text{dim=token})</annotation></semantics></math>
<math id="alg2.27.27.m3.1" class="ltx_Math" alttext="{\bm{A}}\leftarrow\text{Softmax}(\bm{t}_{q}\cdot(\hat{{\bm{X}}}_{k}^{l})^{\top})" display="inline"><semantics id="alg2.27.27.m3.1a"><mrow id="alg2.27.27.m3.1.1" xref="alg2.27.27.m3.1.1.cmml"><mi id="alg2.27.27.m3.1.1.3" xref="alg2.27.27.m3.1.1.3.cmml">𝑨</mi><mo stretchy="false" id="alg2.27.27.m3.1.1.2" xref="alg2.27.27.m3.1.1.2.cmml">←</mo><mrow id="alg2.27.27.m3.1.1.1" xref="alg2.27.27.m3.1.1.1.cmml"><mtext id="alg2.27.27.m3.1.1.1.3" xref="alg2.27.27.m3.1.1.1.3a.cmml">Softmax</mtext><mo lspace="0em" rspace="0em" id="alg2.27.27.m3.1.1.1.2" xref="alg2.27.27.m3.1.1.1.2.cmml">​</mo><mrow id="alg2.27.27.m3.1.1.1.1.1" xref="alg2.27.27.m3.1.1.1.1.1.1.cmml"><mo stretchy="false" id="alg2.27.27.m3.1.1.1.1.1.2" xref="alg2.27.27.m3.1.1.1.1.1.1.cmml">(</mo><mrow id="alg2.27.27.m3.1.1.1.1.1.1" xref="alg2.27.27.m3.1.1.1.1.1.1.cmml"><msub id="alg2.27.27.m3.1.1.1.1.1.1.3" xref="alg2.27.27.m3.1.1.1.1.1.1.3.cmml"><mi id="alg2.27.27.m3.1.1.1.1.1.1.3.2" xref="alg2.27.27.m3.1.1.1.1.1.1.3.2.cmml">𝒕</mi><mi id="alg2.27.27.m3.1.1.1.1.1.1.3.3" xref="alg2.27.27.m3.1.1.1.1.1.1.3.3.cmml">q</mi></msub><mo lspace="0.222em" rspace="0.222em" id="alg2.27.27.m3.1.1.1.1.1.1.2" xref="alg2.27.27.m3.1.1.1.1.1.1.2.cmml">⋅</mo><msup id="alg2.27.27.m3.1.1.1.1.1.1.1" xref="alg2.27.27.m3.1.1.1.1.1.1.1.cmml"><mrow id="alg2.27.27.m3.1.1.1.1.1.1.1.1.1" xref="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.2" xref="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><msubsup id="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1" xref="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1.cmml"><mover accent="true" id="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1.2.2" xref="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1.2.2.cmml"><mi id="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1.2.2.2" xref="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1.2.2.2.cmml">𝑿</mi><mo id="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1.2.2.1" xref="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1.2.2.1.cmml">^</mo></mover><mi id="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1.2.3" xref="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1.2.3.cmml">k</mi><mi id="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1.3" xref="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1.3.cmml">l</mi></msubsup><mo stretchy="false" id="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.3" xref="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo id="alg2.27.27.m3.1.1.1.1.1.1.1.3" xref="alg2.27.27.m3.1.1.1.1.1.1.1.3.cmml">⊤</mo></msup></mrow><mo stretchy="false" id="alg2.27.27.m3.1.1.1.1.1.3" xref="alg2.27.27.m3.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.27.27.m3.1b"><apply id="alg2.27.27.m3.1.1.cmml" xref="alg2.27.27.m3.1.1"><ci id="alg2.27.27.m3.1.1.2.cmml" xref="alg2.27.27.m3.1.1.2">←</ci><ci id="alg2.27.27.m3.1.1.3.cmml" xref="alg2.27.27.m3.1.1.3">𝑨</ci><apply id="alg2.27.27.m3.1.1.1.cmml" xref="alg2.27.27.m3.1.1.1"><times id="alg2.27.27.m3.1.1.1.2.cmml" xref="alg2.27.27.m3.1.1.1.2"></times><ci id="alg2.27.27.m3.1.1.1.3a.cmml" xref="alg2.27.27.m3.1.1.1.3"><mtext id="alg2.27.27.m3.1.1.1.3.cmml" xref="alg2.27.27.m3.1.1.1.3">Softmax</mtext></ci><apply id="alg2.27.27.m3.1.1.1.1.1.1.cmml" xref="alg2.27.27.m3.1.1.1.1.1"><ci id="alg2.27.27.m3.1.1.1.1.1.1.2.cmml" xref="alg2.27.27.m3.1.1.1.1.1.1.2">⋅</ci><apply id="alg2.27.27.m3.1.1.1.1.1.1.3.cmml" xref="alg2.27.27.m3.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="alg2.27.27.m3.1.1.1.1.1.1.3.1.cmml" xref="alg2.27.27.m3.1.1.1.1.1.1.3">subscript</csymbol><ci id="alg2.27.27.m3.1.1.1.1.1.1.3.2.cmml" xref="alg2.27.27.m3.1.1.1.1.1.1.3.2">𝒕</ci><ci id="alg2.27.27.m3.1.1.1.1.1.1.3.3.cmml" xref="alg2.27.27.m3.1.1.1.1.1.1.3.3">𝑞</ci></apply><apply id="alg2.27.27.m3.1.1.1.1.1.1.1.cmml" xref="alg2.27.27.m3.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.27.27.m3.1.1.1.1.1.1.1.2.cmml" xref="alg2.27.27.m3.1.1.1.1.1.1.1">superscript</csymbol><apply id="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1.cmml" xref="alg2.27.27.m3.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="alg2.27.27.m3.1.1.1.1.1.1.1.1.1">superscript</csymbol><apply id="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="alg2.27.27.m3.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="alg2.27.27.m3.1.1.1.1.1.1.1.1.1">subscript</csymbol><apply id="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1.2.2"><ci id="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1.2.2.1.cmml" xref="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1.2.2.1">^</ci><ci id="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1.2.2.2.cmml" xref="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1.2.2.2">𝑿</ci></apply><ci id="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1.2.3">𝑘</ci></apply><ci id="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="alg2.27.27.m3.1.1.1.1.1.1.1.1.1.1.3">𝑙</ci></apply><csymbol cd="latexml" id="alg2.27.27.m3.1.1.1.1.1.1.1.3.cmml" xref="alg2.27.27.m3.1.1.1.1.1.1.1.3">top</csymbol></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.27.27.m3.1c">{\bm{A}}\leftarrow\text{Softmax}(\bm{t}_{q}\cdot(\hat{{\bm{X}}}_{k}^{l})^{\top})</annotation></semantics></math>
<math id="alg2.28.28.m4.1" class="ltx_Math" alttext="{\bm{t}}_{O}\leftarrow{\bm{A}}\cdot\hat{{\bm{X}}}_{v}^{l}" display="inline"><semantics id="alg2.28.28.m4.1a"><mrow id="alg2.28.28.m4.1.1" xref="alg2.28.28.m4.1.1.cmml"><msub id="alg2.28.28.m4.1.1.2" xref="alg2.28.28.m4.1.1.2.cmml"><mi id="alg2.28.28.m4.1.1.2.2" xref="alg2.28.28.m4.1.1.2.2.cmml">𝒕</mi><mi id="alg2.28.28.m4.1.1.2.3" xref="alg2.28.28.m4.1.1.2.3.cmml">O</mi></msub><mo stretchy="false" id="alg2.28.28.m4.1.1.1" xref="alg2.28.28.m4.1.1.1.cmml">←</mo><mrow id="alg2.28.28.m4.1.1.3" xref="alg2.28.28.m4.1.1.3.cmml"><mi id="alg2.28.28.m4.1.1.3.2" xref="alg2.28.28.m4.1.1.3.2.cmml">𝑨</mi><mo lspace="0.222em" rspace="0.222em" id="alg2.28.28.m4.1.1.3.1" xref="alg2.28.28.m4.1.1.3.1.cmml">⋅</mo><msubsup id="alg2.28.28.m4.1.1.3.3" xref="alg2.28.28.m4.1.1.3.3.cmml"><mover accent="true" id="alg2.28.28.m4.1.1.3.3.2.2" xref="alg2.28.28.m4.1.1.3.3.2.2.cmml"><mi id="alg2.28.28.m4.1.1.3.3.2.2.2" xref="alg2.28.28.m4.1.1.3.3.2.2.2.cmml">𝑿</mi><mo id="alg2.28.28.m4.1.1.3.3.2.2.1" xref="alg2.28.28.m4.1.1.3.3.2.2.1.cmml">^</mo></mover><mi id="alg2.28.28.m4.1.1.3.3.2.3" xref="alg2.28.28.m4.1.1.3.3.2.3.cmml">v</mi><mi id="alg2.28.28.m4.1.1.3.3.3" xref="alg2.28.28.m4.1.1.3.3.3.cmml">l</mi></msubsup></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.28.28.m4.1b"><apply id="alg2.28.28.m4.1.1.cmml" xref="alg2.28.28.m4.1.1"><ci id="alg2.28.28.m4.1.1.1.cmml" xref="alg2.28.28.m4.1.1.1">←</ci><apply id="alg2.28.28.m4.1.1.2.cmml" xref="alg2.28.28.m4.1.1.2"><csymbol cd="ambiguous" id="alg2.28.28.m4.1.1.2.1.cmml" xref="alg2.28.28.m4.1.1.2">subscript</csymbol><ci id="alg2.28.28.m4.1.1.2.2.cmml" xref="alg2.28.28.m4.1.1.2.2">𝒕</ci><ci id="alg2.28.28.m4.1.1.2.3.cmml" xref="alg2.28.28.m4.1.1.2.3">𝑂</ci></apply><apply id="alg2.28.28.m4.1.1.3.cmml" xref="alg2.28.28.m4.1.1.3"><ci id="alg2.28.28.m4.1.1.3.1.cmml" xref="alg2.28.28.m4.1.1.3.1">⋅</ci><ci id="alg2.28.28.m4.1.1.3.2.cmml" xref="alg2.28.28.m4.1.1.3.2">𝑨</ci><apply id="alg2.28.28.m4.1.1.3.3.cmml" xref="alg2.28.28.m4.1.1.3.3"><csymbol cd="ambiguous" id="alg2.28.28.m4.1.1.3.3.1.cmml" xref="alg2.28.28.m4.1.1.3.3">superscript</csymbol><apply id="alg2.28.28.m4.1.1.3.3.2.cmml" xref="alg2.28.28.m4.1.1.3.3"><csymbol cd="ambiguous" id="alg2.28.28.m4.1.1.3.3.2.1.cmml" xref="alg2.28.28.m4.1.1.3.3">subscript</csymbol><apply id="alg2.28.28.m4.1.1.3.3.2.2.cmml" xref="alg2.28.28.m4.1.1.3.3.2.2"><ci id="alg2.28.28.m4.1.1.3.3.2.2.1.cmml" xref="alg2.28.28.m4.1.1.3.3.2.2.1">^</ci><ci id="alg2.28.28.m4.1.1.3.3.2.2.2.cmml" xref="alg2.28.28.m4.1.1.3.3.2.2.2">𝑿</ci></apply><ci id="alg2.28.28.m4.1.1.3.3.2.3.cmml" xref="alg2.28.28.m4.1.1.3.3.2.3">𝑣</ci></apply><ci id="alg2.28.28.m4.1.1.3.3.3.cmml" xref="alg2.28.28.m4.1.1.3.3.3">𝑙</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.28.28.m4.1c">{\bm{t}}_{O}\leftarrow{\bm{A}}\cdot\hat{{\bm{X}}}_{v}^{l}</annotation></semantics></math>
<span id="alg2.29.29.2" class="ltx_text ltx_font_bold">if</span> <em id="alg2.29.29.1" class="ltx_emph ltx_font_italic"><math id="alg2.29.29.1.m1.1" class="ltx_Math" alttext="{\bm{t}}^{l-1}" display="inline"><semantics id="alg2.29.29.1.m1.1a"><msup id="alg2.29.29.1.m1.1.1" xref="alg2.29.29.1.m1.1.1.cmml"><mi id="alg2.29.29.1.m1.1.1.2" xref="alg2.29.29.1.m1.1.1.2.cmml">𝐭</mi><mrow id="alg2.29.29.1.m1.1.1.3" xref="alg2.29.29.1.m1.1.1.3.cmml"><mi id="alg2.29.29.1.m1.1.1.3.2" xref="alg2.29.29.1.m1.1.1.3.2.cmml">l</mi><mo id="alg2.29.29.1.m1.1.1.3.1" xref="alg2.29.29.1.m1.1.1.3.1.cmml">−</mo><mn id="alg2.29.29.1.m1.1.1.3.3" xref="alg2.29.29.1.m1.1.1.3.3.cmml">1</mn></mrow></msup><annotation-xml encoding="MathML-Content" id="alg2.29.29.1.m1.1b"><apply id="alg2.29.29.1.m1.1.1.cmml" xref="alg2.29.29.1.m1.1.1"><csymbol cd="ambiguous" id="alg2.29.29.1.m1.1.1.1.cmml" xref="alg2.29.29.1.m1.1.1">superscript</csymbol><ci id="alg2.29.29.1.m1.1.1.2.cmml" xref="alg2.29.29.1.m1.1.1.2">𝐭</ci><apply id="alg2.29.29.1.m1.1.1.3.cmml" xref="alg2.29.29.1.m1.1.1.3"><minus id="alg2.29.29.1.m1.1.1.3.1.cmml" xref="alg2.29.29.1.m1.1.1.3.1"></minus><ci id="alg2.29.29.1.m1.1.1.3.2.cmml" xref="alg2.29.29.1.m1.1.1.3.2">𝑙</ci><cn type="integer" id="alg2.29.29.1.m1.1.1.3.3.cmml" xref="alg2.29.29.1.m1.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.29.29.1.m1.1c">{\bm{t}}^{l-1}</annotation></semantics></math> exists</em> <span id="alg2.29.29.3" class="ltx_text ltx_font_bold">then</span>
</div>
<div id="alg2.30.30" class="ltx_listingline">  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
KV cache <math id="alg2.30.30.m1.9" class="ltx_Math" alttext="\leftarrow\text{Concat}({\bm{E}}_{k,v}^{l},\text{Merge}({\bm{t}}_{k,v}^{l},{\bm{t}}_{k,v}^{l-1}),\text{dim=token})" display="inline"><semantics id="alg2.30.30.m1.9a"><mrow id="alg2.30.30.m1.9.9" xref="alg2.30.30.m1.9.9.cmml"><mi id="alg2.30.30.m1.9.9.4" xref="alg2.30.30.m1.9.9.4.cmml"></mi><mo stretchy="false" id="alg2.30.30.m1.9.9.3" xref="alg2.30.30.m1.9.9.3.cmml">←</mo><mrow id="alg2.30.30.m1.9.9.2" xref="alg2.30.30.m1.9.9.2.cmml"><mtext id="alg2.30.30.m1.9.9.2.4" xref="alg2.30.30.m1.9.9.2.4a.cmml">Concat</mtext><mo lspace="0em" rspace="0em" id="alg2.30.30.m1.9.9.2.3" xref="alg2.30.30.m1.9.9.2.3.cmml">​</mo><mrow id="alg2.30.30.m1.9.9.2.2.2" xref="alg2.30.30.m1.9.9.2.2.3.cmml"><mo stretchy="false" id="alg2.30.30.m1.9.9.2.2.2.3" xref="alg2.30.30.m1.9.9.2.2.3.cmml">(</mo><msubsup id="alg2.30.30.m1.8.8.1.1.1.1" xref="alg2.30.30.m1.8.8.1.1.1.1.cmml"><mi id="alg2.30.30.m1.8.8.1.1.1.1.2.2" xref="alg2.30.30.m1.8.8.1.1.1.1.2.2.cmml">𝑬</mi><mrow id="alg2.30.30.m1.2.2.2.4" xref="alg2.30.30.m1.2.2.2.3.cmml"><mi id="alg2.30.30.m1.1.1.1.1" xref="alg2.30.30.m1.1.1.1.1.cmml">k</mi><mo id="alg2.30.30.m1.2.2.2.4.1" xref="alg2.30.30.m1.2.2.2.3.cmml">,</mo><mi id="alg2.30.30.m1.2.2.2.2" xref="alg2.30.30.m1.2.2.2.2.cmml">v</mi></mrow><mi id="alg2.30.30.m1.8.8.1.1.1.1.3" xref="alg2.30.30.m1.8.8.1.1.1.1.3.cmml">l</mi></msubsup><mo id="alg2.30.30.m1.9.9.2.2.2.4" xref="alg2.30.30.m1.9.9.2.2.3.cmml">,</mo><mrow id="alg2.30.30.m1.9.9.2.2.2.2" xref="alg2.30.30.m1.9.9.2.2.2.2.cmml"><mtext id="alg2.30.30.m1.9.9.2.2.2.2.4" xref="alg2.30.30.m1.9.9.2.2.2.2.4a.cmml">Merge</mtext><mo lspace="0em" rspace="0em" id="alg2.30.30.m1.9.9.2.2.2.2.3" xref="alg2.30.30.m1.9.9.2.2.2.2.3.cmml">​</mo><mrow id="alg2.30.30.m1.9.9.2.2.2.2.2.2" xref="alg2.30.30.m1.9.9.2.2.2.2.2.3.cmml"><mo stretchy="false" id="alg2.30.30.m1.9.9.2.2.2.2.2.2.3" xref="alg2.30.30.m1.9.9.2.2.2.2.2.3.cmml">(</mo><msubsup id="alg2.30.30.m1.9.9.2.2.2.2.1.1.1" xref="alg2.30.30.m1.9.9.2.2.2.2.1.1.1.cmml"><mi id="alg2.30.30.m1.9.9.2.2.2.2.1.1.1.2.2" xref="alg2.30.30.m1.9.9.2.2.2.2.1.1.1.2.2.cmml">𝒕</mi><mrow id="alg2.30.30.m1.4.4.2.4" xref="alg2.30.30.m1.4.4.2.3.cmml"><mi id="alg2.30.30.m1.3.3.1.1" xref="alg2.30.30.m1.3.3.1.1.cmml">k</mi><mo id="alg2.30.30.m1.4.4.2.4.1" xref="alg2.30.30.m1.4.4.2.3.cmml">,</mo><mi id="alg2.30.30.m1.4.4.2.2" xref="alg2.30.30.m1.4.4.2.2.cmml">v</mi></mrow><mi id="alg2.30.30.m1.9.9.2.2.2.2.1.1.1.3" xref="alg2.30.30.m1.9.9.2.2.2.2.1.1.1.3.cmml">l</mi></msubsup><mo id="alg2.30.30.m1.9.9.2.2.2.2.2.2.4" xref="alg2.30.30.m1.9.9.2.2.2.2.2.3.cmml">,</mo><msubsup id="alg2.30.30.m1.9.9.2.2.2.2.2.2.2" xref="alg2.30.30.m1.9.9.2.2.2.2.2.2.2.cmml"><mi id="alg2.30.30.m1.9.9.2.2.2.2.2.2.2.2.2" xref="alg2.30.30.m1.9.9.2.2.2.2.2.2.2.2.2.cmml">𝒕</mi><mrow id="alg2.30.30.m1.6.6.2.4" xref="alg2.30.30.m1.6.6.2.3.cmml"><mi id="alg2.30.30.m1.5.5.1.1" xref="alg2.30.30.m1.5.5.1.1.cmml">k</mi><mo id="alg2.30.30.m1.6.6.2.4.1" xref="alg2.30.30.m1.6.6.2.3.cmml">,</mo><mi id="alg2.30.30.m1.6.6.2.2" xref="alg2.30.30.m1.6.6.2.2.cmml">v</mi></mrow><mrow id="alg2.30.30.m1.9.9.2.2.2.2.2.2.2.3" xref="alg2.30.30.m1.9.9.2.2.2.2.2.2.2.3.cmml"><mi id="alg2.30.30.m1.9.9.2.2.2.2.2.2.2.3.2" xref="alg2.30.30.m1.9.9.2.2.2.2.2.2.2.3.2.cmml">l</mi><mo id="alg2.30.30.m1.9.9.2.2.2.2.2.2.2.3.1" xref="alg2.30.30.m1.9.9.2.2.2.2.2.2.2.3.1.cmml">−</mo><mn id="alg2.30.30.m1.9.9.2.2.2.2.2.2.2.3.3" xref="alg2.30.30.m1.9.9.2.2.2.2.2.2.2.3.3.cmml">1</mn></mrow></msubsup><mo stretchy="false" id="alg2.30.30.m1.9.9.2.2.2.2.2.2.5" xref="alg2.30.30.m1.9.9.2.2.2.2.2.3.cmml">)</mo></mrow></mrow><mo id="alg2.30.30.m1.9.9.2.2.2.5" xref="alg2.30.30.m1.9.9.2.2.3.cmml">,</mo><mtext id="alg2.30.30.m1.7.7" xref="alg2.30.30.m1.7.7a.cmml">dim=token</mtext><mo stretchy="false" id="alg2.30.30.m1.9.9.2.2.2.6" xref="alg2.30.30.m1.9.9.2.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.30.30.m1.9b"><apply id="alg2.30.30.m1.9.9.cmml" xref="alg2.30.30.m1.9.9"><ci id="alg2.30.30.m1.9.9.3.cmml" xref="alg2.30.30.m1.9.9.3">←</ci><csymbol cd="latexml" id="alg2.30.30.m1.9.9.4.cmml" xref="alg2.30.30.m1.9.9.4">absent</csymbol><apply id="alg2.30.30.m1.9.9.2.cmml" xref="alg2.30.30.m1.9.9.2"><times id="alg2.30.30.m1.9.9.2.3.cmml" xref="alg2.30.30.m1.9.9.2.3"></times><ci id="alg2.30.30.m1.9.9.2.4a.cmml" xref="alg2.30.30.m1.9.9.2.4"><mtext id="alg2.30.30.m1.9.9.2.4.cmml" xref="alg2.30.30.m1.9.9.2.4">Concat</mtext></ci><vector id="alg2.30.30.m1.9.9.2.2.3.cmml" xref="alg2.30.30.m1.9.9.2.2.2"><apply id="alg2.30.30.m1.8.8.1.1.1.1.cmml" xref="alg2.30.30.m1.8.8.1.1.1.1"><csymbol cd="ambiguous" id="alg2.30.30.m1.8.8.1.1.1.1.1.cmml" xref="alg2.30.30.m1.8.8.1.1.1.1">superscript</csymbol><apply id="alg2.30.30.m1.8.8.1.1.1.1.2.cmml" xref="alg2.30.30.m1.8.8.1.1.1.1"><csymbol cd="ambiguous" id="alg2.30.30.m1.8.8.1.1.1.1.2.1.cmml" xref="alg2.30.30.m1.8.8.1.1.1.1">subscript</csymbol><ci id="alg2.30.30.m1.8.8.1.1.1.1.2.2.cmml" xref="alg2.30.30.m1.8.8.1.1.1.1.2.2">𝑬</ci><list id="alg2.30.30.m1.2.2.2.3.cmml" xref="alg2.30.30.m1.2.2.2.4"><ci id="alg2.30.30.m1.1.1.1.1.cmml" xref="alg2.30.30.m1.1.1.1.1">𝑘</ci><ci id="alg2.30.30.m1.2.2.2.2.cmml" xref="alg2.30.30.m1.2.2.2.2">𝑣</ci></list></apply><ci id="alg2.30.30.m1.8.8.1.1.1.1.3.cmml" xref="alg2.30.30.m1.8.8.1.1.1.1.3">𝑙</ci></apply><apply id="alg2.30.30.m1.9.9.2.2.2.2.cmml" xref="alg2.30.30.m1.9.9.2.2.2.2"><times id="alg2.30.30.m1.9.9.2.2.2.2.3.cmml" xref="alg2.30.30.m1.9.9.2.2.2.2.3"></times><ci id="alg2.30.30.m1.9.9.2.2.2.2.4a.cmml" xref="alg2.30.30.m1.9.9.2.2.2.2.4"><mtext id="alg2.30.30.m1.9.9.2.2.2.2.4.cmml" xref="alg2.30.30.m1.9.9.2.2.2.2.4">Merge</mtext></ci><interval closure="open" id="alg2.30.30.m1.9.9.2.2.2.2.2.3.cmml" xref="alg2.30.30.m1.9.9.2.2.2.2.2.2"><apply id="alg2.30.30.m1.9.9.2.2.2.2.1.1.1.cmml" xref="alg2.30.30.m1.9.9.2.2.2.2.1.1.1"><csymbol cd="ambiguous" id="alg2.30.30.m1.9.9.2.2.2.2.1.1.1.1.cmml" xref="alg2.30.30.m1.9.9.2.2.2.2.1.1.1">superscript</csymbol><apply id="alg2.30.30.m1.9.9.2.2.2.2.1.1.1.2.cmml" xref="alg2.30.30.m1.9.9.2.2.2.2.1.1.1"><csymbol cd="ambiguous" id="alg2.30.30.m1.9.9.2.2.2.2.1.1.1.2.1.cmml" xref="alg2.30.30.m1.9.9.2.2.2.2.1.1.1">subscript</csymbol><ci id="alg2.30.30.m1.9.9.2.2.2.2.1.1.1.2.2.cmml" xref="alg2.30.30.m1.9.9.2.2.2.2.1.1.1.2.2">𝒕</ci><list id="alg2.30.30.m1.4.4.2.3.cmml" xref="alg2.30.30.m1.4.4.2.4"><ci id="alg2.30.30.m1.3.3.1.1.cmml" xref="alg2.30.30.m1.3.3.1.1">𝑘</ci><ci id="alg2.30.30.m1.4.4.2.2.cmml" xref="alg2.30.30.m1.4.4.2.2">𝑣</ci></list></apply><ci id="alg2.30.30.m1.9.9.2.2.2.2.1.1.1.3.cmml" xref="alg2.30.30.m1.9.9.2.2.2.2.1.1.1.3">𝑙</ci></apply><apply id="alg2.30.30.m1.9.9.2.2.2.2.2.2.2.cmml" xref="alg2.30.30.m1.9.9.2.2.2.2.2.2.2"><csymbol cd="ambiguous" id="alg2.30.30.m1.9.9.2.2.2.2.2.2.2.1.cmml" xref="alg2.30.30.m1.9.9.2.2.2.2.2.2.2">superscript</csymbol><apply id="alg2.30.30.m1.9.9.2.2.2.2.2.2.2.2.cmml" xref="alg2.30.30.m1.9.9.2.2.2.2.2.2.2"><csymbol cd="ambiguous" id="alg2.30.30.m1.9.9.2.2.2.2.2.2.2.2.1.cmml" xref="alg2.30.30.m1.9.9.2.2.2.2.2.2.2">subscript</csymbol><ci id="alg2.30.30.m1.9.9.2.2.2.2.2.2.2.2.2.cmml" xref="alg2.30.30.m1.9.9.2.2.2.2.2.2.2.2.2">𝒕</ci><list id="alg2.30.30.m1.6.6.2.3.cmml" xref="alg2.30.30.m1.6.6.2.4"><ci id="alg2.30.30.m1.5.5.1.1.cmml" xref="alg2.30.30.m1.5.5.1.1">𝑘</ci><ci id="alg2.30.30.m1.6.6.2.2.cmml" xref="alg2.30.30.m1.6.6.2.2">𝑣</ci></list></apply><apply id="alg2.30.30.m1.9.9.2.2.2.2.2.2.2.3.cmml" xref="alg2.30.30.m1.9.9.2.2.2.2.2.2.2.3"><minus id="alg2.30.30.m1.9.9.2.2.2.2.2.2.2.3.1.cmml" xref="alg2.30.30.m1.9.9.2.2.2.2.2.2.2.3.1"></minus><ci id="alg2.30.30.m1.9.9.2.2.2.2.2.2.2.3.2.cmml" xref="alg2.30.30.m1.9.9.2.2.2.2.2.2.2.3.2">𝑙</ci><cn type="integer" id="alg2.30.30.m1.9.9.2.2.2.2.2.2.2.3.3.cmml" xref="alg2.30.30.m1.9.9.2.2.2.2.2.2.2.3.3">1</cn></apply></apply></interval></apply><ci id="alg2.30.30.m1.7.7a.cmml" xref="alg2.30.30.m1.7.7"><mtext id="alg2.30.30.m1.7.7.cmml" xref="alg2.30.30.m1.7.7">dim=token</mtext></ci></vector></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.30.30.m1.9c">\leftarrow\text{Concat}({\bm{E}}_{k,v}^{l},\text{Merge}({\bm{t}}_{k,v}^{l},{\bm{t}}_{k,v}^{l-1}),\text{dim=token})</annotation></semantics></math> <span id="alg2.30.30.1" class="ltx_text ltx_font_typewriter">// </span><span id="alg2.30.30.2" class="ltx_text ltx_font_typewriter">perform compression </span>
</div>
<div id="alg2.38.46" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">13</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   

</div>
<div id="alg2.38.47" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">14</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   <span id="alg2.38.47.1" class="ltx_text ltx_font_bold">else</span>
</div>
<div id="alg2.31.31" class="ltx_listingline">  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
KV cache <math id="alg2.31.31.m1.7" class="ltx_Math" alttext="\leftarrow\text{Concat}({\bm{E}}_{k,v}^{l},{\bm{t}}_{k,v}^{l},\text{dim=token})" display="inline"><semantics id="alg2.31.31.m1.7a"><mrow id="alg2.31.31.m1.7.7" xref="alg2.31.31.m1.7.7.cmml"><mi id="alg2.31.31.m1.7.7.4" xref="alg2.31.31.m1.7.7.4.cmml"></mi><mo stretchy="false" id="alg2.31.31.m1.7.7.3" xref="alg2.31.31.m1.7.7.3.cmml">←</mo><mrow id="alg2.31.31.m1.7.7.2" xref="alg2.31.31.m1.7.7.2.cmml"><mtext id="alg2.31.31.m1.7.7.2.4" xref="alg2.31.31.m1.7.7.2.4a.cmml">Concat</mtext><mo lspace="0em" rspace="0em" id="alg2.31.31.m1.7.7.2.3" xref="alg2.31.31.m1.7.7.2.3.cmml">​</mo><mrow id="alg2.31.31.m1.7.7.2.2.2" xref="alg2.31.31.m1.7.7.2.2.3.cmml"><mo stretchy="false" id="alg2.31.31.m1.7.7.2.2.2.3" xref="alg2.31.31.m1.7.7.2.2.3.cmml">(</mo><msubsup id="alg2.31.31.m1.6.6.1.1.1.1" xref="alg2.31.31.m1.6.6.1.1.1.1.cmml"><mi id="alg2.31.31.m1.6.6.1.1.1.1.2.2" xref="alg2.31.31.m1.6.6.1.1.1.1.2.2.cmml">𝑬</mi><mrow id="alg2.31.31.m1.2.2.2.4" xref="alg2.31.31.m1.2.2.2.3.cmml"><mi id="alg2.31.31.m1.1.1.1.1" xref="alg2.31.31.m1.1.1.1.1.cmml">k</mi><mo id="alg2.31.31.m1.2.2.2.4.1" xref="alg2.31.31.m1.2.2.2.3.cmml">,</mo><mi id="alg2.31.31.m1.2.2.2.2" xref="alg2.31.31.m1.2.2.2.2.cmml">v</mi></mrow><mi id="alg2.31.31.m1.6.6.1.1.1.1.3" xref="alg2.31.31.m1.6.6.1.1.1.1.3.cmml">l</mi></msubsup><mo id="alg2.31.31.m1.7.7.2.2.2.4" xref="alg2.31.31.m1.7.7.2.2.3.cmml">,</mo><msubsup id="alg2.31.31.m1.7.7.2.2.2.2" xref="alg2.31.31.m1.7.7.2.2.2.2.cmml"><mi id="alg2.31.31.m1.7.7.2.2.2.2.2.2" xref="alg2.31.31.m1.7.7.2.2.2.2.2.2.cmml">𝒕</mi><mrow id="alg2.31.31.m1.4.4.2.4" xref="alg2.31.31.m1.4.4.2.3.cmml"><mi id="alg2.31.31.m1.3.3.1.1" xref="alg2.31.31.m1.3.3.1.1.cmml">k</mi><mo id="alg2.31.31.m1.4.4.2.4.1" xref="alg2.31.31.m1.4.4.2.3.cmml">,</mo><mi id="alg2.31.31.m1.4.4.2.2" xref="alg2.31.31.m1.4.4.2.2.cmml">v</mi></mrow><mi id="alg2.31.31.m1.7.7.2.2.2.2.3" xref="alg2.31.31.m1.7.7.2.2.2.2.3.cmml">l</mi></msubsup><mo id="alg2.31.31.m1.7.7.2.2.2.5" xref="alg2.31.31.m1.7.7.2.2.3.cmml">,</mo><mtext id="alg2.31.31.m1.5.5" xref="alg2.31.31.m1.5.5a.cmml">dim=token</mtext><mo stretchy="false" id="alg2.31.31.m1.7.7.2.2.2.6" xref="alg2.31.31.m1.7.7.2.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.31.31.m1.7b"><apply id="alg2.31.31.m1.7.7.cmml" xref="alg2.31.31.m1.7.7"><ci id="alg2.31.31.m1.7.7.3.cmml" xref="alg2.31.31.m1.7.7.3">←</ci><csymbol cd="latexml" id="alg2.31.31.m1.7.7.4.cmml" xref="alg2.31.31.m1.7.7.4">absent</csymbol><apply id="alg2.31.31.m1.7.7.2.cmml" xref="alg2.31.31.m1.7.7.2"><times id="alg2.31.31.m1.7.7.2.3.cmml" xref="alg2.31.31.m1.7.7.2.3"></times><ci id="alg2.31.31.m1.7.7.2.4a.cmml" xref="alg2.31.31.m1.7.7.2.4"><mtext id="alg2.31.31.m1.7.7.2.4.cmml" xref="alg2.31.31.m1.7.7.2.4">Concat</mtext></ci><vector id="alg2.31.31.m1.7.7.2.2.3.cmml" xref="alg2.31.31.m1.7.7.2.2.2"><apply id="alg2.31.31.m1.6.6.1.1.1.1.cmml" xref="alg2.31.31.m1.6.6.1.1.1.1"><csymbol cd="ambiguous" id="alg2.31.31.m1.6.6.1.1.1.1.1.cmml" xref="alg2.31.31.m1.6.6.1.1.1.1">superscript</csymbol><apply id="alg2.31.31.m1.6.6.1.1.1.1.2.cmml" xref="alg2.31.31.m1.6.6.1.1.1.1"><csymbol cd="ambiguous" id="alg2.31.31.m1.6.6.1.1.1.1.2.1.cmml" xref="alg2.31.31.m1.6.6.1.1.1.1">subscript</csymbol><ci id="alg2.31.31.m1.6.6.1.1.1.1.2.2.cmml" xref="alg2.31.31.m1.6.6.1.1.1.1.2.2">𝑬</ci><list id="alg2.31.31.m1.2.2.2.3.cmml" xref="alg2.31.31.m1.2.2.2.4"><ci id="alg2.31.31.m1.1.1.1.1.cmml" xref="alg2.31.31.m1.1.1.1.1">𝑘</ci><ci id="alg2.31.31.m1.2.2.2.2.cmml" xref="alg2.31.31.m1.2.2.2.2">𝑣</ci></list></apply><ci id="alg2.31.31.m1.6.6.1.1.1.1.3.cmml" xref="alg2.31.31.m1.6.6.1.1.1.1.3">𝑙</ci></apply><apply id="alg2.31.31.m1.7.7.2.2.2.2.cmml" xref="alg2.31.31.m1.7.7.2.2.2.2"><csymbol cd="ambiguous" id="alg2.31.31.m1.7.7.2.2.2.2.1.cmml" xref="alg2.31.31.m1.7.7.2.2.2.2">superscript</csymbol><apply id="alg2.31.31.m1.7.7.2.2.2.2.2.cmml" xref="alg2.31.31.m1.7.7.2.2.2.2"><csymbol cd="ambiguous" id="alg2.31.31.m1.7.7.2.2.2.2.2.1.cmml" xref="alg2.31.31.m1.7.7.2.2.2.2">subscript</csymbol><ci id="alg2.31.31.m1.7.7.2.2.2.2.2.2.cmml" xref="alg2.31.31.m1.7.7.2.2.2.2.2.2">𝒕</ci><list id="alg2.31.31.m1.4.4.2.3.cmml" xref="alg2.31.31.m1.4.4.2.4"><ci id="alg2.31.31.m1.3.3.1.1.cmml" xref="alg2.31.31.m1.3.3.1.1">𝑘</ci><ci id="alg2.31.31.m1.4.4.2.2.cmml" xref="alg2.31.31.m1.4.4.2.2">𝑣</ci></list></apply><ci id="alg2.31.31.m1.7.7.2.2.2.2.3.cmml" xref="alg2.31.31.m1.7.7.2.2.2.2.3">𝑙</ci></apply><ci id="alg2.31.31.m1.5.5a.cmml" xref="alg2.31.31.m1.5.5"><mtext id="alg2.31.31.m1.5.5.cmml" xref="alg2.31.31.m1.5.5">dim=token</mtext></ci></vector></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.31.31.m1.7c">\leftarrow\text{Concat}({\bm{E}}_{k,v}^{l},{\bm{t}}_{k,v}^{l},\text{dim=token})</annotation></semantics></math> <span id="alg2.31.31.1" class="ltx_text ltx_font_typewriter">// </span><span id="alg2.31.31.2" class="ltx_text ltx_font_typewriter">wait for compression </span>
</div>
<div id="alg2.38.48" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">15</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>     <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   

</div>
<div id="alg2.32.32" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">16</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   <span id="alg2.32.32.2" class="ltx_text ltx_font_bold">return</span> <em id="alg2.32.32.1" class="ltx_emph ltx_font_italic"><math id="alg2.32.32.1.m1.1" class="ltx_Math" alttext="{\bm{t}}_{O}" display="inline"><semantics id="alg2.32.32.1.m1.1a"><msub id="alg2.32.32.1.m1.1.1" xref="alg2.32.32.1.m1.1.1.cmml"><mi id="alg2.32.32.1.m1.1.1.2" xref="alg2.32.32.1.m1.1.1.2.cmml">𝐭</mi><mi id="alg2.32.32.1.m1.1.1.3" xref="alg2.32.32.1.m1.1.1.3.cmml">O</mi></msub><annotation-xml encoding="MathML-Content" id="alg2.32.32.1.m1.1b"><apply id="alg2.32.32.1.m1.1.1.cmml" xref="alg2.32.32.1.m1.1.1"><csymbol cd="ambiguous" id="alg2.32.32.1.m1.1.1.1.cmml" xref="alg2.32.32.1.m1.1.1">subscript</csymbol><ci id="alg2.32.32.1.m1.1.1.2.cmml" xref="alg2.32.32.1.m1.1.1.2">𝐭</ci><ci id="alg2.32.32.1.m1.1.1.3.cmml" xref="alg2.32.32.1.m1.1.1.3">𝑂</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.32.32.1.m1.1c">{\bm{t}}_{O}</annotation></semantics></math></em>

</div>
<div id="alg2.38.49" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">17</span>
</div>
<div id="alg2.33.33" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">18</span><span id="alg2.33.33.2" class="ltx_text ltx_font_bold">function</span> <em id="alg2.33.33.1" class="ltx_emph ltx_font_italic"><span id="alg2.33.33.1.1" class="ltx_text ltx_font_typewriter ltx_font_upright">MiniCache Merge</span>(<math id="alg2.33.33.1.m1.3" class="ltx_Math" alttext="{\bm{X}}^{l},{\bm{X}}^{l-1},t" display="inline"><semantics id="alg2.33.33.1.m1.3a"><mrow id="alg2.33.33.1.m1.3.3.2" xref="alg2.33.33.1.m1.3.3.3.cmml"><msup id="alg2.33.33.1.m1.2.2.1.1" xref="alg2.33.33.1.m1.2.2.1.1.cmml"><mi id="alg2.33.33.1.m1.2.2.1.1.2" xref="alg2.33.33.1.m1.2.2.1.1.2.cmml">𝐗</mi><mi id="alg2.33.33.1.m1.2.2.1.1.3" xref="alg2.33.33.1.m1.2.2.1.1.3.cmml">l</mi></msup><mo id="alg2.33.33.1.m1.3.3.2.3" xref="alg2.33.33.1.m1.3.3.3.cmml">,</mo><msup id="alg2.33.33.1.m1.3.3.2.2" xref="alg2.33.33.1.m1.3.3.2.2.cmml"><mi id="alg2.33.33.1.m1.3.3.2.2.2" xref="alg2.33.33.1.m1.3.3.2.2.2.cmml">𝐗</mi><mrow id="alg2.33.33.1.m1.3.3.2.2.3" xref="alg2.33.33.1.m1.3.3.2.2.3.cmml"><mi id="alg2.33.33.1.m1.3.3.2.2.3.2" xref="alg2.33.33.1.m1.3.3.2.2.3.2.cmml">l</mi><mo id="alg2.33.33.1.m1.3.3.2.2.3.1" xref="alg2.33.33.1.m1.3.3.2.2.3.1.cmml">−</mo><mn id="alg2.33.33.1.m1.3.3.2.2.3.3" xref="alg2.33.33.1.m1.3.3.2.2.3.3.cmml">1</mn></mrow></msup><mo id="alg2.33.33.1.m1.3.3.2.4" xref="alg2.33.33.1.m1.3.3.3.cmml">,</mo><mi id="alg2.33.33.1.m1.1.1" xref="alg2.33.33.1.m1.1.1.cmml">t</mi></mrow><annotation-xml encoding="MathML-Content" id="alg2.33.33.1.m1.3b"><list id="alg2.33.33.1.m1.3.3.3.cmml" xref="alg2.33.33.1.m1.3.3.2"><apply id="alg2.33.33.1.m1.2.2.1.1.cmml" xref="alg2.33.33.1.m1.2.2.1.1"><csymbol cd="ambiguous" id="alg2.33.33.1.m1.2.2.1.1.1.cmml" xref="alg2.33.33.1.m1.2.2.1.1">superscript</csymbol><ci id="alg2.33.33.1.m1.2.2.1.1.2.cmml" xref="alg2.33.33.1.m1.2.2.1.1.2">𝐗</ci><ci id="alg2.33.33.1.m1.2.2.1.1.3.cmml" xref="alg2.33.33.1.m1.2.2.1.1.3">𝑙</ci></apply><apply id="alg2.33.33.1.m1.3.3.2.2.cmml" xref="alg2.33.33.1.m1.3.3.2.2"><csymbol cd="ambiguous" id="alg2.33.33.1.m1.3.3.2.2.1.cmml" xref="alg2.33.33.1.m1.3.3.2.2">superscript</csymbol><ci id="alg2.33.33.1.m1.3.3.2.2.2.cmml" xref="alg2.33.33.1.m1.3.3.2.2.2">𝐗</ci><apply id="alg2.33.33.1.m1.3.3.2.2.3.cmml" xref="alg2.33.33.1.m1.3.3.2.2.3"><minus id="alg2.33.33.1.m1.3.3.2.2.3.1.cmml" xref="alg2.33.33.1.m1.3.3.2.2.3.1"></minus><ci id="alg2.33.33.1.m1.3.3.2.2.3.2.cmml" xref="alg2.33.33.1.m1.3.3.2.2.3.2">𝑙</ci><cn type="integer" id="alg2.33.33.1.m1.3.3.2.2.3.3.cmml" xref="alg2.33.33.1.m1.3.3.2.2.3.3">1</cn></apply></apply><ci id="alg2.33.33.1.m1.1.1.cmml" xref="alg2.33.33.1.m1.1.1">𝑡</ci></list></annotation-xml><annotation encoding="application/x-tex" id="alg2.33.33.1.m1.3c">{\bm{X}}^{l},{\bm{X}}^{l-1},t</annotation></semantics></math>)</em><span id="alg2.33.33.3" class="ltx_text ltx_font_bold">:</span>
</div>
<div id="alg2.34.34" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">19</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<math id="alg2.34.34.m1.1" class="ltx_Math" alttext="\vec{{\bm{X}}}^{l}\leftarrow\frac{{\bm{X}}^{l}}{\lVert{\bm{X}}^{l}\rVert}" display="inline"><semantics id="alg2.34.34.m1.1a"><mrow id="alg2.34.34.m1.1.2" xref="alg2.34.34.m1.1.2.cmml"><msup id="alg2.34.34.m1.1.2.2" xref="alg2.34.34.m1.1.2.2.cmml"><mover accent="true" id="alg2.34.34.m1.1.2.2.2" xref="alg2.34.34.m1.1.2.2.2.cmml"><mi id="alg2.34.34.m1.1.2.2.2.2" xref="alg2.34.34.m1.1.2.2.2.2.cmml">𝑿</mi><mo stretchy="false" id="alg2.34.34.m1.1.2.2.2.1" xref="alg2.34.34.m1.1.2.2.2.1.cmml">→</mo></mover><mi id="alg2.34.34.m1.1.2.2.3" xref="alg2.34.34.m1.1.2.2.3.cmml">l</mi></msup><mo stretchy="false" id="alg2.34.34.m1.1.2.1" xref="alg2.34.34.m1.1.2.1.cmml">←</mo><mfrac id="alg2.34.34.m1.1.1" xref="alg2.34.34.m1.1.1.cmml"><msup id="alg2.34.34.m1.1.1.3" xref="alg2.34.34.m1.1.1.3.cmml"><mi id="alg2.34.34.m1.1.1.3.2" xref="alg2.34.34.m1.1.1.3.2.cmml">𝑿</mi><mi id="alg2.34.34.m1.1.1.3.3" xref="alg2.34.34.m1.1.1.3.3.cmml">l</mi></msup><mrow id="alg2.34.34.m1.1.1.1.1" xref="alg2.34.34.m1.1.1.1.2.cmml"><mo fence="true" rspace="0em" id="alg2.34.34.m1.1.1.1.1.2" xref="alg2.34.34.m1.1.1.1.2.1.cmml">∥</mo><msup id="alg2.34.34.m1.1.1.1.1.1" xref="alg2.34.34.m1.1.1.1.1.1.cmml"><mi id="alg2.34.34.m1.1.1.1.1.1.2" xref="alg2.34.34.m1.1.1.1.1.1.2.cmml">𝑿</mi><mi id="alg2.34.34.m1.1.1.1.1.1.3" xref="alg2.34.34.m1.1.1.1.1.1.3.cmml">l</mi></msup><mo fence="true" lspace="0em" id="alg2.34.34.m1.1.1.1.1.3" xref="alg2.34.34.m1.1.1.1.2.1.cmml">∥</mo></mrow></mfrac></mrow><annotation-xml encoding="MathML-Content" id="alg2.34.34.m1.1b"><apply id="alg2.34.34.m1.1.2.cmml" xref="alg2.34.34.m1.1.2"><ci id="alg2.34.34.m1.1.2.1.cmml" xref="alg2.34.34.m1.1.2.1">←</ci><apply id="alg2.34.34.m1.1.2.2.cmml" xref="alg2.34.34.m1.1.2.2"><csymbol cd="ambiguous" id="alg2.34.34.m1.1.2.2.1.cmml" xref="alg2.34.34.m1.1.2.2">superscript</csymbol><apply id="alg2.34.34.m1.1.2.2.2.cmml" xref="alg2.34.34.m1.1.2.2.2"><ci id="alg2.34.34.m1.1.2.2.2.1.cmml" xref="alg2.34.34.m1.1.2.2.2.1">→</ci><ci id="alg2.34.34.m1.1.2.2.2.2.cmml" xref="alg2.34.34.m1.1.2.2.2.2">𝑿</ci></apply><ci id="alg2.34.34.m1.1.2.2.3.cmml" xref="alg2.34.34.m1.1.2.2.3">𝑙</ci></apply><apply id="alg2.34.34.m1.1.1.cmml" xref="alg2.34.34.m1.1.1"><divide id="alg2.34.34.m1.1.1.2.cmml" xref="alg2.34.34.m1.1.1"></divide><apply id="alg2.34.34.m1.1.1.3.cmml" xref="alg2.34.34.m1.1.1.3"><csymbol cd="ambiguous" id="alg2.34.34.m1.1.1.3.1.cmml" xref="alg2.34.34.m1.1.1.3">superscript</csymbol><ci id="alg2.34.34.m1.1.1.3.2.cmml" xref="alg2.34.34.m1.1.1.3.2">𝑿</ci><ci id="alg2.34.34.m1.1.1.3.3.cmml" xref="alg2.34.34.m1.1.1.3.3">𝑙</ci></apply><apply id="alg2.34.34.m1.1.1.1.2.cmml" xref="alg2.34.34.m1.1.1.1.1"><csymbol cd="latexml" id="alg2.34.34.m1.1.1.1.2.1.cmml" xref="alg2.34.34.m1.1.1.1.1.2">delimited-∥∥</csymbol><apply id="alg2.34.34.m1.1.1.1.1.1.cmml" xref="alg2.34.34.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.34.34.m1.1.1.1.1.1.1.cmml" xref="alg2.34.34.m1.1.1.1.1.1">superscript</csymbol><ci id="alg2.34.34.m1.1.1.1.1.1.2.cmml" xref="alg2.34.34.m1.1.1.1.1.1.2">𝑿</ci><ci id="alg2.34.34.m1.1.1.1.1.1.3.cmml" xref="alg2.34.34.m1.1.1.1.1.1.3">𝑙</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.34.34.m1.1c">\vec{{\bm{X}}}^{l}\leftarrow\frac{{\bm{X}}^{l}}{\lVert{\bm{X}}^{l}\rVert}</annotation></semantics></math> 
</div>
<div id="alg2.35.35" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">20</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<math id="alg2.35.35.m1.1" class="ltx_Math" alttext="\vec{{\bm{X}}}^{l-1}\leftarrow\frac{{\bm{X}}^{l-1}}{\lVert{\bm{X}}^{l-1}\rVert}" display="inline"><semantics id="alg2.35.35.m1.1a"><mrow id="alg2.35.35.m1.1.2" xref="alg2.35.35.m1.1.2.cmml"><msup id="alg2.35.35.m1.1.2.2" xref="alg2.35.35.m1.1.2.2.cmml"><mover accent="true" id="alg2.35.35.m1.1.2.2.2" xref="alg2.35.35.m1.1.2.2.2.cmml"><mi id="alg2.35.35.m1.1.2.2.2.2" xref="alg2.35.35.m1.1.2.2.2.2.cmml">𝑿</mi><mo stretchy="false" id="alg2.35.35.m1.1.2.2.2.1" xref="alg2.35.35.m1.1.2.2.2.1.cmml">→</mo></mover><mrow id="alg2.35.35.m1.1.2.2.3" xref="alg2.35.35.m1.1.2.2.3.cmml"><mi id="alg2.35.35.m1.1.2.2.3.2" xref="alg2.35.35.m1.1.2.2.3.2.cmml">l</mi><mo id="alg2.35.35.m1.1.2.2.3.1" xref="alg2.35.35.m1.1.2.2.3.1.cmml">−</mo><mn id="alg2.35.35.m1.1.2.2.3.3" xref="alg2.35.35.m1.1.2.2.3.3.cmml">1</mn></mrow></msup><mo stretchy="false" id="alg2.35.35.m1.1.2.1" xref="alg2.35.35.m1.1.2.1.cmml">←</mo><mfrac id="alg2.35.35.m1.1.1" xref="alg2.35.35.m1.1.1.cmml"><msup id="alg2.35.35.m1.1.1.3" xref="alg2.35.35.m1.1.1.3.cmml"><mi id="alg2.35.35.m1.1.1.3.2" xref="alg2.35.35.m1.1.1.3.2.cmml">𝑿</mi><mrow id="alg2.35.35.m1.1.1.3.3" xref="alg2.35.35.m1.1.1.3.3.cmml"><mi id="alg2.35.35.m1.1.1.3.3.2" xref="alg2.35.35.m1.1.1.3.3.2.cmml">l</mi><mo id="alg2.35.35.m1.1.1.3.3.1" xref="alg2.35.35.m1.1.1.3.3.1.cmml">−</mo><mn id="alg2.35.35.m1.1.1.3.3.3" xref="alg2.35.35.m1.1.1.3.3.3.cmml">1</mn></mrow></msup><mrow id="alg2.35.35.m1.1.1.1.1" xref="alg2.35.35.m1.1.1.1.2.cmml"><mo fence="true" rspace="0em" id="alg2.35.35.m1.1.1.1.1.2" xref="alg2.35.35.m1.1.1.1.2.1.cmml">∥</mo><msup id="alg2.35.35.m1.1.1.1.1.1" xref="alg2.35.35.m1.1.1.1.1.1.cmml"><mi id="alg2.35.35.m1.1.1.1.1.1.2" xref="alg2.35.35.m1.1.1.1.1.1.2.cmml">𝑿</mi><mrow id="alg2.35.35.m1.1.1.1.1.1.3" xref="alg2.35.35.m1.1.1.1.1.1.3.cmml"><mi id="alg2.35.35.m1.1.1.1.1.1.3.2" xref="alg2.35.35.m1.1.1.1.1.1.3.2.cmml">l</mi><mo id="alg2.35.35.m1.1.1.1.1.1.3.1" xref="alg2.35.35.m1.1.1.1.1.1.3.1.cmml">−</mo><mn id="alg2.35.35.m1.1.1.1.1.1.3.3" xref="alg2.35.35.m1.1.1.1.1.1.3.3.cmml">1</mn></mrow></msup><mo fence="true" lspace="0em" id="alg2.35.35.m1.1.1.1.1.3" xref="alg2.35.35.m1.1.1.1.2.1.cmml">∥</mo></mrow></mfrac></mrow><annotation-xml encoding="MathML-Content" id="alg2.35.35.m1.1b"><apply id="alg2.35.35.m1.1.2.cmml" xref="alg2.35.35.m1.1.2"><ci id="alg2.35.35.m1.1.2.1.cmml" xref="alg2.35.35.m1.1.2.1">←</ci><apply id="alg2.35.35.m1.1.2.2.cmml" xref="alg2.35.35.m1.1.2.2"><csymbol cd="ambiguous" id="alg2.35.35.m1.1.2.2.1.cmml" xref="alg2.35.35.m1.1.2.2">superscript</csymbol><apply id="alg2.35.35.m1.1.2.2.2.cmml" xref="alg2.35.35.m1.1.2.2.2"><ci id="alg2.35.35.m1.1.2.2.2.1.cmml" xref="alg2.35.35.m1.1.2.2.2.1">→</ci><ci id="alg2.35.35.m1.1.2.2.2.2.cmml" xref="alg2.35.35.m1.1.2.2.2.2">𝑿</ci></apply><apply id="alg2.35.35.m1.1.2.2.3.cmml" xref="alg2.35.35.m1.1.2.2.3"><minus id="alg2.35.35.m1.1.2.2.3.1.cmml" xref="alg2.35.35.m1.1.2.2.3.1"></minus><ci id="alg2.35.35.m1.1.2.2.3.2.cmml" xref="alg2.35.35.m1.1.2.2.3.2">𝑙</ci><cn type="integer" id="alg2.35.35.m1.1.2.2.3.3.cmml" xref="alg2.35.35.m1.1.2.2.3.3">1</cn></apply></apply><apply id="alg2.35.35.m1.1.1.cmml" xref="alg2.35.35.m1.1.1"><divide id="alg2.35.35.m1.1.1.2.cmml" xref="alg2.35.35.m1.1.1"></divide><apply id="alg2.35.35.m1.1.1.3.cmml" xref="alg2.35.35.m1.1.1.3"><csymbol cd="ambiguous" id="alg2.35.35.m1.1.1.3.1.cmml" xref="alg2.35.35.m1.1.1.3">superscript</csymbol><ci id="alg2.35.35.m1.1.1.3.2.cmml" xref="alg2.35.35.m1.1.1.3.2">𝑿</ci><apply id="alg2.35.35.m1.1.1.3.3.cmml" xref="alg2.35.35.m1.1.1.3.3"><minus id="alg2.35.35.m1.1.1.3.3.1.cmml" xref="alg2.35.35.m1.1.1.3.3.1"></minus><ci id="alg2.35.35.m1.1.1.3.3.2.cmml" xref="alg2.35.35.m1.1.1.3.3.2">𝑙</ci><cn type="integer" id="alg2.35.35.m1.1.1.3.3.3.cmml" xref="alg2.35.35.m1.1.1.3.3.3">1</cn></apply></apply><apply id="alg2.35.35.m1.1.1.1.2.cmml" xref="alg2.35.35.m1.1.1.1.1"><csymbol cd="latexml" id="alg2.35.35.m1.1.1.1.2.1.cmml" xref="alg2.35.35.m1.1.1.1.1.2">delimited-∥∥</csymbol><apply id="alg2.35.35.m1.1.1.1.1.1.cmml" xref="alg2.35.35.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.35.35.m1.1.1.1.1.1.1.cmml" xref="alg2.35.35.m1.1.1.1.1.1">superscript</csymbol><ci id="alg2.35.35.m1.1.1.1.1.1.2.cmml" xref="alg2.35.35.m1.1.1.1.1.1.2">𝑿</ci><apply id="alg2.35.35.m1.1.1.1.1.1.3.cmml" xref="alg2.35.35.m1.1.1.1.1.1.3"><minus id="alg2.35.35.m1.1.1.1.1.1.3.1.cmml" xref="alg2.35.35.m1.1.1.1.1.1.3.1"></minus><ci id="alg2.35.35.m1.1.1.1.1.1.3.2.cmml" xref="alg2.35.35.m1.1.1.1.1.1.3.2">𝑙</ci><cn type="integer" id="alg2.35.35.m1.1.1.1.1.1.3.3.cmml" xref="alg2.35.35.m1.1.1.1.1.1.3.3">1</cn></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.35.35.m1.1c">\vec{{\bm{X}}}^{l-1}\leftarrow\frac{{\bm{X}}^{l-1}}{\lVert{\bm{X}}^{l-1}\rVert}</annotation></semantics></math> 
</div>
<div id="alg2.36.36" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">21</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<math id="alg2.36.36.m1.3" class="ltx_Math" alttext="\Omega\leftarrow\arccos\left(\frac{{\bm{X}}^{l}_{T}\cdot{\bm{X}}^{l-1}_{T}}{\|{\bm{X}}^{l}_{T}\|\|{\bm{X}}^{l-1}_{T}\|}\right)" display="inline"><semantics id="alg2.36.36.m1.3a"><mrow id="alg2.36.36.m1.3.4" xref="alg2.36.36.m1.3.4.cmml"><mi mathvariant="normal" id="alg2.36.36.m1.3.4.2" xref="alg2.36.36.m1.3.4.2.cmml">Ω</mi><mo stretchy="false" id="alg2.36.36.m1.3.4.1" xref="alg2.36.36.m1.3.4.1.cmml">←</mo><mrow id="alg2.36.36.m1.3.4.3.2" xref="alg2.36.36.m1.3.4.3.1.cmml"><mi id="alg2.36.36.m1.3.3" xref="alg2.36.36.m1.3.3.cmml">arccos</mi><mo id="alg2.36.36.m1.3.4.3.2a" xref="alg2.36.36.m1.3.4.3.1.cmml">⁡</mo><mrow id="alg2.36.36.m1.3.4.3.2.1" xref="alg2.36.36.m1.3.4.3.1.cmml"><mo id="alg2.36.36.m1.3.4.3.2.1.1" xref="alg2.36.36.m1.3.4.3.1.cmml">(</mo><mfrac id="alg2.36.36.m1.2.2" xref="alg2.36.36.m1.2.2.cmml"><mrow id="alg2.36.36.m1.2.2.4" xref="alg2.36.36.m1.2.2.4.cmml"><msubsup id="alg2.36.36.m1.2.2.4.2" xref="alg2.36.36.m1.2.2.4.2.cmml"><mi id="alg2.36.36.m1.2.2.4.2.2.2" xref="alg2.36.36.m1.2.2.4.2.2.2.cmml">𝑿</mi><mi id="alg2.36.36.m1.2.2.4.2.3" xref="alg2.36.36.m1.2.2.4.2.3.cmml">T</mi><mi id="alg2.36.36.m1.2.2.4.2.2.3" xref="alg2.36.36.m1.2.2.4.2.2.3.cmml">l</mi></msubsup><mo lspace="0.222em" rspace="0.222em" id="alg2.36.36.m1.2.2.4.1" xref="alg2.36.36.m1.2.2.4.1.cmml">⋅</mo><msubsup id="alg2.36.36.m1.2.2.4.3" xref="alg2.36.36.m1.2.2.4.3.cmml"><mi id="alg2.36.36.m1.2.2.4.3.2.2" xref="alg2.36.36.m1.2.2.4.3.2.2.cmml">𝑿</mi><mi id="alg2.36.36.m1.2.2.4.3.3" xref="alg2.36.36.m1.2.2.4.3.3.cmml">T</mi><mrow id="alg2.36.36.m1.2.2.4.3.2.3" xref="alg2.36.36.m1.2.2.4.3.2.3.cmml"><mi id="alg2.36.36.m1.2.2.4.3.2.3.2" xref="alg2.36.36.m1.2.2.4.3.2.3.2.cmml">l</mi><mo id="alg2.36.36.m1.2.2.4.3.2.3.1" xref="alg2.36.36.m1.2.2.4.3.2.3.1.cmml">−</mo><mn id="alg2.36.36.m1.2.2.4.3.2.3.3" xref="alg2.36.36.m1.2.2.4.3.2.3.3.cmml">1</mn></mrow></msubsup></mrow><mrow id="alg2.36.36.m1.2.2.2" xref="alg2.36.36.m1.2.2.2.cmml"><mrow id="alg2.36.36.m1.1.1.1.1.1" xref="alg2.36.36.m1.1.1.1.1.2.cmml"><mo stretchy="false" id="alg2.36.36.m1.1.1.1.1.1.2" xref="alg2.36.36.m1.1.1.1.1.2.1.cmml">‖</mo><msubsup id="alg2.36.36.m1.1.1.1.1.1.1" xref="alg2.36.36.m1.1.1.1.1.1.1.cmml"><mi id="alg2.36.36.m1.1.1.1.1.1.1.2.2" xref="alg2.36.36.m1.1.1.1.1.1.1.2.2.cmml">𝑿</mi><mi id="alg2.36.36.m1.1.1.1.1.1.1.3" xref="alg2.36.36.m1.1.1.1.1.1.1.3.cmml">T</mi><mi id="alg2.36.36.m1.1.1.1.1.1.1.2.3" xref="alg2.36.36.m1.1.1.1.1.1.1.2.3.cmml">l</mi></msubsup><mo stretchy="false" id="alg2.36.36.m1.1.1.1.1.1.3" xref="alg2.36.36.m1.1.1.1.1.2.1.cmml">‖</mo></mrow><mo lspace="0em" rspace="0em" id="alg2.36.36.m1.2.2.2.3" xref="alg2.36.36.m1.2.2.2.3.cmml">​</mo><mrow id="alg2.36.36.m1.2.2.2.2.1" xref="alg2.36.36.m1.2.2.2.2.2.cmml"><mo stretchy="false" id="alg2.36.36.m1.2.2.2.2.1.2" xref="alg2.36.36.m1.2.2.2.2.2.1.cmml">‖</mo><msubsup id="alg2.36.36.m1.2.2.2.2.1.1" xref="alg2.36.36.m1.2.2.2.2.1.1.cmml"><mi id="alg2.36.36.m1.2.2.2.2.1.1.2.2" xref="alg2.36.36.m1.2.2.2.2.1.1.2.2.cmml">𝑿</mi><mi id="alg2.36.36.m1.2.2.2.2.1.1.3" xref="alg2.36.36.m1.2.2.2.2.1.1.3.cmml">T</mi><mrow id="alg2.36.36.m1.2.2.2.2.1.1.2.3" xref="alg2.36.36.m1.2.2.2.2.1.1.2.3.cmml"><mi id="alg2.36.36.m1.2.2.2.2.1.1.2.3.2" xref="alg2.36.36.m1.2.2.2.2.1.1.2.3.2.cmml">l</mi><mo id="alg2.36.36.m1.2.2.2.2.1.1.2.3.1" xref="alg2.36.36.m1.2.2.2.2.1.1.2.3.1.cmml">−</mo><mn id="alg2.36.36.m1.2.2.2.2.1.1.2.3.3" xref="alg2.36.36.m1.2.2.2.2.1.1.2.3.3.cmml">1</mn></mrow></msubsup><mo stretchy="false" id="alg2.36.36.m1.2.2.2.2.1.3" xref="alg2.36.36.m1.2.2.2.2.2.1.cmml">‖</mo></mrow></mrow></mfrac><mo id="alg2.36.36.m1.3.4.3.2.1.2" xref="alg2.36.36.m1.3.4.3.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.36.36.m1.3b"><apply id="alg2.36.36.m1.3.4.cmml" xref="alg2.36.36.m1.3.4"><ci id="alg2.36.36.m1.3.4.1.cmml" xref="alg2.36.36.m1.3.4.1">←</ci><ci id="alg2.36.36.m1.3.4.2.cmml" xref="alg2.36.36.m1.3.4.2">Ω</ci><apply id="alg2.36.36.m1.3.4.3.1.cmml" xref="alg2.36.36.m1.3.4.3.2"><arccos id="alg2.36.36.m1.3.3.cmml" xref="alg2.36.36.m1.3.3"></arccos><apply id="alg2.36.36.m1.2.2.cmml" xref="alg2.36.36.m1.2.2"><divide id="alg2.36.36.m1.2.2.3.cmml" xref="alg2.36.36.m1.2.2"></divide><apply id="alg2.36.36.m1.2.2.4.cmml" xref="alg2.36.36.m1.2.2.4"><ci id="alg2.36.36.m1.2.2.4.1.cmml" xref="alg2.36.36.m1.2.2.4.1">⋅</ci><apply id="alg2.36.36.m1.2.2.4.2.cmml" xref="alg2.36.36.m1.2.2.4.2"><csymbol cd="ambiguous" id="alg2.36.36.m1.2.2.4.2.1.cmml" xref="alg2.36.36.m1.2.2.4.2">subscript</csymbol><apply id="alg2.36.36.m1.2.2.4.2.2.cmml" xref="alg2.36.36.m1.2.2.4.2"><csymbol cd="ambiguous" id="alg2.36.36.m1.2.2.4.2.2.1.cmml" xref="alg2.36.36.m1.2.2.4.2">superscript</csymbol><ci id="alg2.36.36.m1.2.2.4.2.2.2.cmml" xref="alg2.36.36.m1.2.2.4.2.2.2">𝑿</ci><ci id="alg2.36.36.m1.2.2.4.2.2.3.cmml" xref="alg2.36.36.m1.2.2.4.2.2.3">𝑙</ci></apply><ci id="alg2.36.36.m1.2.2.4.2.3.cmml" xref="alg2.36.36.m1.2.2.4.2.3">𝑇</ci></apply><apply id="alg2.36.36.m1.2.2.4.3.cmml" xref="alg2.36.36.m1.2.2.4.3"><csymbol cd="ambiguous" id="alg2.36.36.m1.2.2.4.3.1.cmml" xref="alg2.36.36.m1.2.2.4.3">subscript</csymbol><apply id="alg2.36.36.m1.2.2.4.3.2.cmml" xref="alg2.36.36.m1.2.2.4.3"><csymbol cd="ambiguous" id="alg2.36.36.m1.2.2.4.3.2.1.cmml" xref="alg2.36.36.m1.2.2.4.3">superscript</csymbol><ci id="alg2.36.36.m1.2.2.4.3.2.2.cmml" xref="alg2.36.36.m1.2.2.4.3.2.2">𝑿</ci><apply id="alg2.36.36.m1.2.2.4.3.2.3.cmml" xref="alg2.36.36.m1.2.2.4.3.2.3"><minus id="alg2.36.36.m1.2.2.4.3.2.3.1.cmml" xref="alg2.36.36.m1.2.2.4.3.2.3.1"></minus><ci id="alg2.36.36.m1.2.2.4.3.2.3.2.cmml" xref="alg2.36.36.m1.2.2.4.3.2.3.2">𝑙</ci><cn type="integer" id="alg2.36.36.m1.2.2.4.3.2.3.3.cmml" xref="alg2.36.36.m1.2.2.4.3.2.3.3">1</cn></apply></apply><ci id="alg2.36.36.m1.2.2.4.3.3.cmml" xref="alg2.36.36.m1.2.2.4.3.3">𝑇</ci></apply></apply><apply id="alg2.36.36.m1.2.2.2.cmml" xref="alg2.36.36.m1.2.2.2"><times id="alg2.36.36.m1.2.2.2.3.cmml" xref="alg2.36.36.m1.2.2.2.3"></times><apply id="alg2.36.36.m1.1.1.1.1.2.cmml" xref="alg2.36.36.m1.1.1.1.1.1"><csymbol cd="latexml" id="alg2.36.36.m1.1.1.1.1.2.1.cmml" xref="alg2.36.36.m1.1.1.1.1.1.2">norm</csymbol><apply id="alg2.36.36.m1.1.1.1.1.1.1.cmml" xref="alg2.36.36.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.36.36.m1.1.1.1.1.1.1.1.cmml" xref="alg2.36.36.m1.1.1.1.1.1.1">subscript</csymbol><apply id="alg2.36.36.m1.1.1.1.1.1.1.2.cmml" xref="alg2.36.36.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.36.36.m1.1.1.1.1.1.1.2.1.cmml" xref="alg2.36.36.m1.1.1.1.1.1.1">superscript</csymbol><ci id="alg2.36.36.m1.1.1.1.1.1.1.2.2.cmml" xref="alg2.36.36.m1.1.1.1.1.1.1.2.2">𝑿</ci><ci id="alg2.36.36.m1.1.1.1.1.1.1.2.3.cmml" xref="alg2.36.36.m1.1.1.1.1.1.1.2.3">𝑙</ci></apply><ci id="alg2.36.36.m1.1.1.1.1.1.1.3.cmml" xref="alg2.36.36.m1.1.1.1.1.1.1.3">𝑇</ci></apply></apply><apply id="alg2.36.36.m1.2.2.2.2.2.cmml" xref="alg2.36.36.m1.2.2.2.2.1"><csymbol cd="latexml" id="alg2.36.36.m1.2.2.2.2.2.1.cmml" xref="alg2.36.36.m1.2.2.2.2.1.2">norm</csymbol><apply id="alg2.36.36.m1.2.2.2.2.1.1.cmml" xref="alg2.36.36.m1.2.2.2.2.1.1"><csymbol cd="ambiguous" id="alg2.36.36.m1.2.2.2.2.1.1.1.cmml" xref="alg2.36.36.m1.2.2.2.2.1.1">subscript</csymbol><apply id="alg2.36.36.m1.2.2.2.2.1.1.2.cmml" xref="alg2.36.36.m1.2.2.2.2.1.1"><csymbol cd="ambiguous" id="alg2.36.36.m1.2.2.2.2.1.1.2.1.cmml" xref="alg2.36.36.m1.2.2.2.2.1.1">superscript</csymbol><ci id="alg2.36.36.m1.2.2.2.2.1.1.2.2.cmml" xref="alg2.36.36.m1.2.2.2.2.1.1.2.2">𝑿</ci><apply id="alg2.36.36.m1.2.2.2.2.1.1.2.3.cmml" xref="alg2.36.36.m1.2.2.2.2.1.1.2.3"><minus id="alg2.36.36.m1.2.2.2.2.1.1.2.3.1.cmml" xref="alg2.36.36.m1.2.2.2.2.1.1.2.3.1"></minus><ci id="alg2.36.36.m1.2.2.2.2.1.1.2.3.2.cmml" xref="alg2.36.36.m1.2.2.2.2.1.1.2.3.2">𝑙</ci><cn type="integer" id="alg2.36.36.m1.2.2.2.2.1.1.2.3.3.cmml" xref="alg2.36.36.m1.2.2.2.2.1.1.2.3.3">1</cn></apply></apply><ci id="alg2.36.36.m1.2.2.2.2.1.1.3.cmml" xref="alg2.36.36.m1.2.2.2.2.1.1.3">𝑇</ci></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.36.36.m1.3c">\Omega\leftarrow\arccos\left(\frac{{\bm{X}}^{l}_{T}\cdot{\bm{X}}^{l-1}_{T}}{\|{\bm{X}}^{l}_{T}\|\|{\bm{X}}^{l-1}_{T}\|}\right)</annotation></semantics></math> 
</div>
<div id="alg2.37.37" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">22</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<math id="alg2.37.37.m1.8" class="ltx_Math" alttext="{\bm{E}}\leftarrow\frac{\sin((1-t)\Omega)}{\sin(\Omega)}\vec{{\bm{X}}}^{\,l}+\frac{\sin(t\Omega)}{\sin(\Omega)}\vec{{\bm{X}}}^{\,l-1}" display="inline"><semantics id="alg2.37.37.m1.8a"><mrow id="alg2.37.37.m1.8.9" xref="alg2.37.37.m1.8.9.cmml"><mi id="alg2.37.37.m1.8.9.2" xref="alg2.37.37.m1.8.9.2.cmml">𝑬</mi><mo stretchy="false" id="alg2.37.37.m1.8.9.1" xref="alg2.37.37.m1.8.9.1.cmml">←</mo><mrow id="alg2.37.37.m1.8.9.3" xref="alg2.37.37.m1.8.9.3.cmml"><mrow id="alg2.37.37.m1.8.9.3.2" xref="alg2.37.37.m1.8.9.3.2.cmml"><mfrac id="alg2.37.37.m1.4.4" xref="alg2.37.37.m1.4.4.cmml"><mrow id="alg2.37.37.m1.2.2.2.2" xref="alg2.37.37.m1.2.2.2.3.cmml"><mi id="alg2.37.37.m1.1.1.1.1" xref="alg2.37.37.m1.1.1.1.1.cmml">sin</mi><mo id="alg2.37.37.m1.2.2.2.2a" xref="alg2.37.37.m1.2.2.2.3.cmml">⁡</mo><mrow id="alg2.37.37.m1.2.2.2.2.1" xref="alg2.37.37.m1.2.2.2.3.cmml"><mo stretchy="false" id="alg2.37.37.m1.2.2.2.2.1.2" xref="alg2.37.37.m1.2.2.2.3.cmml">(</mo><mrow id="alg2.37.37.m1.2.2.2.2.1.1" xref="alg2.37.37.m1.2.2.2.2.1.1.cmml"><mrow id="alg2.37.37.m1.2.2.2.2.1.1.1.1" xref="alg2.37.37.m1.2.2.2.2.1.1.1.1.1.cmml"><mo stretchy="false" id="alg2.37.37.m1.2.2.2.2.1.1.1.1.2" xref="alg2.37.37.m1.2.2.2.2.1.1.1.1.1.cmml">(</mo><mrow id="alg2.37.37.m1.2.2.2.2.1.1.1.1.1" xref="alg2.37.37.m1.2.2.2.2.1.1.1.1.1.cmml"><mn id="alg2.37.37.m1.2.2.2.2.1.1.1.1.1.2" xref="alg2.37.37.m1.2.2.2.2.1.1.1.1.1.2.cmml">1</mn><mo id="alg2.37.37.m1.2.2.2.2.1.1.1.1.1.1" xref="alg2.37.37.m1.2.2.2.2.1.1.1.1.1.1.cmml">−</mo><mi id="alg2.37.37.m1.2.2.2.2.1.1.1.1.1.3" xref="alg2.37.37.m1.2.2.2.2.1.1.1.1.1.3.cmml">t</mi></mrow><mo stretchy="false" id="alg2.37.37.m1.2.2.2.2.1.1.1.1.3" xref="alg2.37.37.m1.2.2.2.2.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="alg2.37.37.m1.2.2.2.2.1.1.2" xref="alg2.37.37.m1.2.2.2.2.1.1.2.cmml">​</mo><mi mathvariant="normal" id="alg2.37.37.m1.2.2.2.2.1.1.3" xref="alg2.37.37.m1.2.2.2.2.1.1.3.cmml">Ω</mi></mrow><mo stretchy="false" id="alg2.37.37.m1.2.2.2.2.1.3" xref="alg2.37.37.m1.2.2.2.3.cmml">)</mo></mrow></mrow><mrow id="alg2.37.37.m1.4.4.4.4" xref="alg2.37.37.m1.4.4.4.3.cmml"><mi id="alg2.37.37.m1.3.3.3.1" xref="alg2.37.37.m1.3.3.3.1.cmml">sin</mi><mo id="alg2.37.37.m1.4.4.4.4a" xref="alg2.37.37.m1.4.4.4.3.cmml">⁡</mo><mrow id="alg2.37.37.m1.4.4.4.4.1" xref="alg2.37.37.m1.4.4.4.3.cmml"><mo stretchy="false" id="alg2.37.37.m1.4.4.4.4.1.1" xref="alg2.37.37.m1.4.4.4.3.cmml">(</mo><mi mathvariant="normal" id="alg2.37.37.m1.4.4.4.2" xref="alg2.37.37.m1.4.4.4.2.cmml">Ω</mi><mo stretchy="false" id="alg2.37.37.m1.4.4.4.4.1.2" xref="alg2.37.37.m1.4.4.4.3.cmml">)</mo></mrow></mrow></mfrac><mo lspace="0em" rspace="0em" id="alg2.37.37.m1.8.9.3.2.1" xref="alg2.37.37.m1.8.9.3.2.1.cmml">​</mo><msup id="alg2.37.37.m1.8.9.3.2.2" xref="alg2.37.37.m1.8.9.3.2.2.cmml"><mover accent="true" id="alg2.37.37.m1.8.9.3.2.2.2" xref="alg2.37.37.m1.8.9.3.2.2.2.cmml"><mi id="alg2.37.37.m1.8.9.3.2.2.2.2" xref="alg2.37.37.m1.8.9.3.2.2.2.2.cmml">𝑿</mi><mo stretchy="false" id="alg2.37.37.m1.8.9.3.2.2.2.1" xref="alg2.37.37.m1.8.9.3.2.2.2.1.cmml">→</mo></mover><mi id="alg2.37.37.m1.8.9.3.2.2.3" xref="alg2.37.37.m1.8.9.3.2.2.3.cmml">l</mi></msup></mrow><mo id="alg2.37.37.m1.8.9.3.1" xref="alg2.37.37.m1.8.9.3.1.cmml">+</mo><mrow id="alg2.37.37.m1.8.9.3.3" xref="alg2.37.37.m1.8.9.3.3.cmml"><mfrac id="alg2.37.37.m1.8.8" xref="alg2.37.37.m1.8.8.cmml"><mrow id="alg2.37.37.m1.6.6.2.2" xref="alg2.37.37.m1.6.6.2.3.cmml"><mi id="alg2.37.37.m1.5.5.1.1" xref="alg2.37.37.m1.5.5.1.1.cmml">sin</mi><mo id="alg2.37.37.m1.6.6.2.2a" xref="alg2.37.37.m1.6.6.2.3.cmml">⁡</mo><mrow id="alg2.37.37.m1.6.6.2.2.1" xref="alg2.37.37.m1.6.6.2.3.cmml"><mo stretchy="false" id="alg2.37.37.m1.6.6.2.2.1.2" xref="alg2.37.37.m1.6.6.2.3.cmml">(</mo><mrow id="alg2.37.37.m1.6.6.2.2.1.1" xref="alg2.37.37.m1.6.6.2.2.1.1.cmml"><mi id="alg2.37.37.m1.6.6.2.2.1.1.2" xref="alg2.37.37.m1.6.6.2.2.1.1.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg2.37.37.m1.6.6.2.2.1.1.1" xref="alg2.37.37.m1.6.6.2.2.1.1.1.cmml">​</mo><mi mathvariant="normal" id="alg2.37.37.m1.6.6.2.2.1.1.3" xref="alg2.37.37.m1.6.6.2.2.1.1.3.cmml">Ω</mi></mrow><mo stretchy="false" id="alg2.37.37.m1.6.6.2.2.1.3" xref="alg2.37.37.m1.6.6.2.3.cmml">)</mo></mrow></mrow><mrow id="alg2.37.37.m1.8.8.4.4" xref="alg2.37.37.m1.8.8.4.3.cmml"><mi id="alg2.37.37.m1.7.7.3.1" xref="alg2.37.37.m1.7.7.3.1.cmml">sin</mi><mo id="alg2.37.37.m1.8.8.4.4a" xref="alg2.37.37.m1.8.8.4.3.cmml">⁡</mo><mrow id="alg2.37.37.m1.8.8.4.4.1" xref="alg2.37.37.m1.8.8.4.3.cmml"><mo stretchy="false" id="alg2.37.37.m1.8.8.4.4.1.1" xref="alg2.37.37.m1.8.8.4.3.cmml">(</mo><mi mathvariant="normal" id="alg2.37.37.m1.8.8.4.2" xref="alg2.37.37.m1.8.8.4.2.cmml">Ω</mi><mo stretchy="false" id="alg2.37.37.m1.8.8.4.4.1.2" xref="alg2.37.37.m1.8.8.4.3.cmml">)</mo></mrow></mrow></mfrac><mo lspace="0em" rspace="0em" id="alg2.37.37.m1.8.9.3.3.1" xref="alg2.37.37.m1.8.9.3.3.1.cmml">​</mo><msup id="alg2.37.37.m1.8.9.3.3.2" xref="alg2.37.37.m1.8.9.3.3.2.cmml"><mover accent="true" id="alg2.37.37.m1.8.9.3.3.2.2" xref="alg2.37.37.m1.8.9.3.3.2.2.cmml"><mi id="alg2.37.37.m1.8.9.3.3.2.2.2" xref="alg2.37.37.m1.8.9.3.3.2.2.2.cmml">𝑿</mi><mo stretchy="false" id="alg2.37.37.m1.8.9.3.3.2.2.1" xref="alg2.37.37.m1.8.9.3.3.2.2.1.cmml">→</mo></mover><mrow id="alg2.37.37.m1.8.9.3.3.2.3" xref="alg2.37.37.m1.8.9.3.3.2.3.cmml"><mi id="alg2.37.37.m1.8.9.3.3.2.3.2" xref="alg2.37.37.m1.8.9.3.3.2.3.2.cmml">l</mi><mo id="alg2.37.37.m1.8.9.3.3.2.3.1" xref="alg2.37.37.m1.8.9.3.3.2.3.1.cmml">−</mo><mn id="alg2.37.37.m1.8.9.3.3.2.3.3" xref="alg2.37.37.m1.8.9.3.3.2.3.3.cmml">1</mn></mrow></msup></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.37.37.m1.8b"><apply id="alg2.37.37.m1.8.9.cmml" xref="alg2.37.37.m1.8.9"><ci id="alg2.37.37.m1.8.9.1.cmml" xref="alg2.37.37.m1.8.9.1">←</ci><ci id="alg2.37.37.m1.8.9.2.cmml" xref="alg2.37.37.m1.8.9.2">𝑬</ci><apply id="alg2.37.37.m1.8.9.3.cmml" xref="alg2.37.37.m1.8.9.3"><plus id="alg2.37.37.m1.8.9.3.1.cmml" xref="alg2.37.37.m1.8.9.3.1"></plus><apply id="alg2.37.37.m1.8.9.3.2.cmml" xref="alg2.37.37.m1.8.9.3.2"><times id="alg2.37.37.m1.8.9.3.2.1.cmml" xref="alg2.37.37.m1.8.9.3.2.1"></times><apply id="alg2.37.37.m1.4.4.cmml" xref="alg2.37.37.m1.4.4"><divide id="alg2.37.37.m1.4.4.5.cmml" xref="alg2.37.37.m1.4.4"></divide><apply id="alg2.37.37.m1.2.2.2.3.cmml" xref="alg2.37.37.m1.2.2.2.2"><sin id="alg2.37.37.m1.1.1.1.1.cmml" xref="alg2.37.37.m1.1.1.1.1"></sin><apply id="alg2.37.37.m1.2.2.2.2.1.1.cmml" xref="alg2.37.37.m1.2.2.2.2.1.1"><times id="alg2.37.37.m1.2.2.2.2.1.1.2.cmml" xref="alg2.37.37.m1.2.2.2.2.1.1.2"></times><apply id="alg2.37.37.m1.2.2.2.2.1.1.1.1.1.cmml" xref="alg2.37.37.m1.2.2.2.2.1.1.1.1"><minus id="alg2.37.37.m1.2.2.2.2.1.1.1.1.1.1.cmml" xref="alg2.37.37.m1.2.2.2.2.1.1.1.1.1.1"></minus><cn type="integer" id="alg2.37.37.m1.2.2.2.2.1.1.1.1.1.2.cmml" xref="alg2.37.37.m1.2.2.2.2.1.1.1.1.1.2">1</cn><ci id="alg2.37.37.m1.2.2.2.2.1.1.1.1.1.3.cmml" xref="alg2.37.37.m1.2.2.2.2.1.1.1.1.1.3">𝑡</ci></apply><ci id="alg2.37.37.m1.2.2.2.2.1.1.3.cmml" xref="alg2.37.37.m1.2.2.2.2.1.1.3">Ω</ci></apply></apply><apply id="alg2.37.37.m1.4.4.4.3.cmml" xref="alg2.37.37.m1.4.4.4.4"><sin id="alg2.37.37.m1.3.3.3.1.cmml" xref="alg2.37.37.m1.3.3.3.1"></sin><ci id="alg2.37.37.m1.4.4.4.2.cmml" xref="alg2.37.37.m1.4.4.4.2">Ω</ci></apply></apply><apply id="alg2.37.37.m1.8.9.3.2.2.cmml" xref="alg2.37.37.m1.8.9.3.2.2"><csymbol cd="ambiguous" id="alg2.37.37.m1.8.9.3.2.2.1.cmml" xref="alg2.37.37.m1.8.9.3.2.2">superscript</csymbol><apply id="alg2.37.37.m1.8.9.3.2.2.2.cmml" xref="alg2.37.37.m1.8.9.3.2.2.2"><ci id="alg2.37.37.m1.8.9.3.2.2.2.1.cmml" xref="alg2.37.37.m1.8.9.3.2.2.2.1">→</ci><ci id="alg2.37.37.m1.8.9.3.2.2.2.2.cmml" xref="alg2.37.37.m1.8.9.3.2.2.2.2">𝑿</ci></apply><ci id="alg2.37.37.m1.8.9.3.2.2.3.cmml" xref="alg2.37.37.m1.8.9.3.2.2.3">𝑙</ci></apply></apply><apply id="alg2.37.37.m1.8.9.3.3.cmml" xref="alg2.37.37.m1.8.9.3.3"><times id="alg2.37.37.m1.8.9.3.3.1.cmml" xref="alg2.37.37.m1.8.9.3.3.1"></times><apply id="alg2.37.37.m1.8.8.cmml" xref="alg2.37.37.m1.8.8"><divide id="alg2.37.37.m1.8.8.5.cmml" xref="alg2.37.37.m1.8.8"></divide><apply id="alg2.37.37.m1.6.6.2.3.cmml" xref="alg2.37.37.m1.6.6.2.2"><sin id="alg2.37.37.m1.5.5.1.1.cmml" xref="alg2.37.37.m1.5.5.1.1"></sin><apply id="alg2.37.37.m1.6.6.2.2.1.1.cmml" xref="alg2.37.37.m1.6.6.2.2.1.1"><times id="alg2.37.37.m1.6.6.2.2.1.1.1.cmml" xref="alg2.37.37.m1.6.6.2.2.1.1.1"></times><ci id="alg2.37.37.m1.6.6.2.2.1.1.2.cmml" xref="alg2.37.37.m1.6.6.2.2.1.1.2">𝑡</ci><ci id="alg2.37.37.m1.6.6.2.2.1.1.3.cmml" xref="alg2.37.37.m1.6.6.2.2.1.1.3">Ω</ci></apply></apply><apply id="alg2.37.37.m1.8.8.4.3.cmml" xref="alg2.37.37.m1.8.8.4.4"><sin id="alg2.37.37.m1.7.7.3.1.cmml" xref="alg2.37.37.m1.7.7.3.1"></sin><ci id="alg2.37.37.m1.8.8.4.2.cmml" xref="alg2.37.37.m1.8.8.4.2">Ω</ci></apply></apply><apply id="alg2.37.37.m1.8.9.3.3.2.cmml" xref="alg2.37.37.m1.8.9.3.3.2"><csymbol cd="ambiguous" id="alg2.37.37.m1.8.9.3.3.2.1.cmml" xref="alg2.37.37.m1.8.9.3.3.2">superscript</csymbol><apply id="alg2.37.37.m1.8.9.3.3.2.2.cmml" xref="alg2.37.37.m1.8.9.3.3.2.2"><ci id="alg2.37.37.m1.8.9.3.3.2.2.1.cmml" xref="alg2.37.37.m1.8.9.3.3.2.2.1">→</ci><ci id="alg2.37.37.m1.8.9.3.3.2.2.2.cmml" xref="alg2.37.37.m1.8.9.3.3.2.2.2">𝑿</ci></apply><apply id="alg2.37.37.m1.8.9.3.3.2.3.cmml" xref="alg2.37.37.m1.8.9.3.3.2.3"><minus id="alg2.37.37.m1.8.9.3.3.2.3.1.cmml" xref="alg2.37.37.m1.8.9.3.3.2.3.1"></minus><ci id="alg2.37.37.m1.8.9.3.3.2.3.2.cmml" xref="alg2.37.37.m1.8.9.3.3.2.3.2">𝑙</ci><cn type="integer" id="alg2.37.37.m1.8.9.3.3.2.3.3.cmml" xref="alg2.37.37.m1.8.9.3.3.2.3.3">1</cn></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.37.37.m1.8c">{\bm{E}}\leftarrow\frac{\sin((1-t)\Omega)}{\sin(\Omega)}\vec{{\bm{X}}}^{\,l}+\frac{\sin(t\Omega)}{\sin(\Omega)}\vec{{\bm{X}}}^{\,l-1}</annotation></semantics></math> 
</div>
<div id="alg2.38.38" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">23</span>  <span class="ltx_rule" style="width:1px;height:100%;background:black;display:inline-block;"> </span>   
<span id="alg2.38.38.2" class="ltx_text ltx_font_bold">return</span> <em id="alg2.38.38.1" class="ltx_emph ltx_font_italic"><math id="alg2.38.38.1.m1.4" class="ltx_Math" alttext="{\bm{E}},\lVert{\bm{X}}^{l}\rVert,\lVert{\bm{X}}^{l-1}\rVert,\Omega" display="inline"><semantics id="alg2.38.38.1.m1.4a"><mrow id="alg2.38.38.1.m1.4.4.2" xref="alg2.38.38.1.m1.4.4.3.cmml"><mi id="alg2.38.38.1.m1.1.1" xref="alg2.38.38.1.m1.1.1.cmml">𝐄</mi><mo id="alg2.38.38.1.m1.4.4.2.3" xref="alg2.38.38.1.m1.4.4.3.cmml">,</mo><mrow id="alg2.38.38.1.m1.3.3.1.1.1" xref="alg2.38.38.1.m1.3.3.1.1.2.cmml"><mo fence="true" lspace="0em" rspace="0em" id="alg2.38.38.1.m1.3.3.1.1.1.2" xref="alg2.38.38.1.m1.3.3.1.1.2.1.cmml">∥</mo><msup id="alg2.38.38.1.m1.3.3.1.1.1.1" xref="alg2.38.38.1.m1.3.3.1.1.1.1.cmml"><mi id="alg2.38.38.1.m1.3.3.1.1.1.1.2" xref="alg2.38.38.1.m1.3.3.1.1.1.1.2.cmml">𝐗</mi><mi id="alg2.38.38.1.m1.3.3.1.1.1.1.3" xref="alg2.38.38.1.m1.3.3.1.1.1.1.3.cmml">l</mi></msup><mo fence="true" lspace="0em" rspace="0em" id="alg2.38.38.1.m1.3.3.1.1.1.3" xref="alg2.38.38.1.m1.3.3.1.1.2.1.cmml">∥</mo></mrow><mo id="alg2.38.38.1.m1.4.4.2.4" xref="alg2.38.38.1.m1.4.4.3.cmml">,</mo><mrow id="alg2.38.38.1.m1.4.4.2.2.1" xref="alg2.38.38.1.m1.4.4.2.2.2.cmml"><mo fence="true" lspace="0em" rspace="0em" id="alg2.38.38.1.m1.4.4.2.2.1.2" xref="alg2.38.38.1.m1.4.4.2.2.2.1.cmml">∥</mo><msup id="alg2.38.38.1.m1.4.4.2.2.1.1" xref="alg2.38.38.1.m1.4.4.2.2.1.1.cmml"><mi id="alg2.38.38.1.m1.4.4.2.2.1.1.2" xref="alg2.38.38.1.m1.4.4.2.2.1.1.2.cmml">𝐗</mi><mrow id="alg2.38.38.1.m1.4.4.2.2.1.1.3" xref="alg2.38.38.1.m1.4.4.2.2.1.1.3.cmml"><mi id="alg2.38.38.1.m1.4.4.2.2.1.1.3.2" xref="alg2.38.38.1.m1.4.4.2.2.1.1.3.2.cmml">l</mi><mo id="alg2.38.38.1.m1.4.4.2.2.1.1.3.1" xref="alg2.38.38.1.m1.4.4.2.2.1.1.3.1.cmml">−</mo><mn id="alg2.38.38.1.m1.4.4.2.2.1.1.3.3" xref="alg2.38.38.1.m1.4.4.2.2.1.1.3.3.cmml">1</mn></mrow></msup><mo fence="true" lspace="0em" rspace="0em" id="alg2.38.38.1.m1.4.4.2.2.1.3" xref="alg2.38.38.1.m1.4.4.2.2.2.1.cmml">∥</mo></mrow><mo id="alg2.38.38.1.m1.4.4.2.5" xref="alg2.38.38.1.m1.4.4.3.cmml">,</mo><mi mathvariant="normal" id="alg2.38.38.1.m1.2.2" xref="alg2.38.38.1.m1.2.2.cmml">Ω</mi></mrow><annotation-xml encoding="MathML-Content" id="alg2.38.38.1.m1.4b"><list id="alg2.38.38.1.m1.4.4.3.cmml" xref="alg2.38.38.1.m1.4.4.2"><ci id="alg2.38.38.1.m1.1.1.cmml" xref="alg2.38.38.1.m1.1.1">𝐄</ci><apply id="alg2.38.38.1.m1.3.3.1.1.2.cmml" xref="alg2.38.38.1.m1.3.3.1.1.1"><csymbol cd="latexml" id="alg2.38.38.1.m1.3.3.1.1.2.1.cmml" xref="alg2.38.38.1.m1.3.3.1.1.1.2">delimited-∥∥</csymbol><apply id="alg2.38.38.1.m1.3.3.1.1.1.1.cmml" xref="alg2.38.38.1.m1.3.3.1.1.1.1"><csymbol cd="ambiguous" id="alg2.38.38.1.m1.3.3.1.1.1.1.1.cmml" xref="alg2.38.38.1.m1.3.3.1.1.1.1">superscript</csymbol><ci id="alg2.38.38.1.m1.3.3.1.1.1.1.2.cmml" xref="alg2.38.38.1.m1.3.3.1.1.1.1.2">𝐗</ci><ci id="alg2.38.38.1.m1.3.3.1.1.1.1.3.cmml" xref="alg2.38.38.1.m1.3.3.1.1.1.1.3">𝑙</ci></apply></apply><apply id="alg2.38.38.1.m1.4.4.2.2.2.cmml" xref="alg2.38.38.1.m1.4.4.2.2.1"><csymbol cd="latexml" id="alg2.38.38.1.m1.4.4.2.2.2.1.cmml" xref="alg2.38.38.1.m1.4.4.2.2.1.2">delimited-∥∥</csymbol><apply id="alg2.38.38.1.m1.4.4.2.2.1.1.cmml" xref="alg2.38.38.1.m1.4.4.2.2.1.1"><csymbol cd="ambiguous" id="alg2.38.38.1.m1.4.4.2.2.1.1.1.cmml" xref="alg2.38.38.1.m1.4.4.2.2.1.1">superscript</csymbol><ci id="alg2.38.38.1.m1.4.4.2.2.1.1.2.cmml" xref="alg2.38.38.1.m1.4.4.2.2.1.1.2">𝐗</ci><apply id="alg2.38.38.1.m1.4.4.2.2.1.1.3.cmml" xref="alg2.38.38.1.m1.4.4.2.2.1.1.3"><minus id="alg2.38.38.1.m1.4.4.2.2.1.1.3.1.cmml" xref="alg2.38.38.1.m1.4.4.2.2.1.1.3.1"></minus><ci id="alg2.38.38.1.m1.4.4.2.2.1.1.3.2.cmml" xref="alg2.38.38.1.m1.4.4.2.2.1.1.3.2">𝑙</ci><cn type="integer" id="alg2.38.38.1.m1.4.4.2.2.1.1.3.3.cmml" xref="alg2.38.38.1.m1.4.4.2.2.1.1.3.3">1</cn></apply></apply></apply><ci id="alg2.38.38.1.m1.2.2.cmml" xref="alg2.38.38.1.m1.2.2">Ω</ci></list></annotation-xml><annotation encoding="application/x-tex" id="alg2.38.38.1.m1.4c">{\bm{E}},\lVert{\bm{X}}^{l}\rVert,\lVert{\bm{X}}^{l-1}\rVert,\Omega</annotation></semantics></math></em>

</div>
<div id="alg2.38.50" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">24</span>
</div>
<div id="alg2.38.51" class="ltx_listingline">
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float"><span id="alg2.40.1.1" class="ltx_text ltx_font_bold">Algorithm 2</span> </span>The MiniCache Prefill &amp; Decoding Compression Algorithm</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A5" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix E </span>Detailed Efficiency Derivation</h2>

<div id="A5.p1" class="ltx_para">
<p id="A5.p1.1" class="ltx_p">In this section, we provide a detailed derivation of the memory efficiency improvements outlined in Section <a href="#S4.SS3" title="4.3 Efficiency Discussion ‣ 4 Method ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.3</span></a>.</p>
</div>
<div id="A5.p2" class="ltx_para">
<p id="A5.p2.1" class="ltx_p">First, we consider the original KV cache memory usage, which is given by:</p>
</div>
<div id="A5.p3" class="ltx_para">
<table id="A5.Ex1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A5.Ex1.m1.1" class="ltx_Math" alttext="4brh(s+n)" display="block"><semantics id="A5.Ex1.m1.1a"><mrow id="A5.Ex1.m1.1.1" xref="A5.Ex1.m1.1.1.cmml"><mn id="A5.Ex1.m1.1.1.3" xref="A5.Ex1.m1.1.1.3.cmml">4</mn><mo lspace="0em" rspace="0em" id="A5.Ex1.m1.1.1.2" xref="A5.Ex1.m1.1.1.2.cmml">​</mo><mi id="A5.Ex1.m1.1.1.4" xref="A5.Ex1.m1.1.1.4.cmml">b</mi><mo lspace="0em" rspace="0em" id="A5.Ex1.m1.1.1.2a" xref="A5.Ex1.m1.1.1.2.cmml">​</mo><mi id="A5.Ex1.m1.1.1.5" xref="A5.Ex1.m1.1.1.5.cmml">r</mi><mo lspace="0em" rspace="0em" id="A5.Ex1.m1.1.1.2b" xref="A5.Ex1.m1.1.1.2.cmml">​</mo><mi id="A5.Ex1.m1.1.1.6" xref="A5.Ex1.m1.1.1.6.cmml">h</mi><mo lspace="0em" rspace="0em" id="A5.Ex1.m1.1.1.2c" xref="A5.Ex1.m1.1.1.2.cmml">​</mo><mrow id="A5.Ex1.m1.1.1.1.1" xref="A5.Ex1.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="A5.Ex1.m1.1.1.1.1.2" xref="A5.Ex1.m1.1.1.1.1.1.cmml">(</mo><mrow id="A5.Ex1.m1.1.1.1.1.1" xref="A5.Ex1.m1.1.1.1.1.1.cmml"><mi id="A5.Ex1.m1.1.1.1.1.1.2" xref="A5.Ex1.m1.1.1.1.1.1.2.cmml">s</mi><mo id="A5.Ex1.m1.1.1.1.1.1.1" xref="A5.Ex1.m1.1.1.1.1.1.1.cmml">+</mo><mi id="A5.Ex1.m1.1.1.1.1.1.3" xref="A5.Ex1.m1.1.1.1.1.1.3.cmml">n</mi></mrow><mo stretchy="false" id="A5.Ex1.m1.1.1.1.1.3" xref="A5.Ex1.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.Ex1.m1.1b"><apply id="A5.Ex1.m1.1.1.cmml" xref="A5.Ex1.m1.1.1"><times id="A5.Ex1.m1.1.1.2.cmml" xref="A5.Ex1.m1.1.1.2"></times><cn type="integer" id="A5.Ex1.m1.1.1.3.cmml" xref="A5.Ex1.m1.1.1.3">4</cn><ci id="A5.Ex1.m1.1.1.4.cmml" xref="A5.Ex1.m1.1.1.4">𝑏</ci><ci id="A5.Ex1.m1.1.1.5.cmml" xref="A5.Ex1.m1.1.1.5">𝑟</ci><ci id="A5.Ex1.m1.1.1.6.cmml" xref="A5.Ex1.m1.1.1.6">ℎ</ci><apply id="A5.Ex1.m1.1.1.1.1.1.cmml" xref="A5.Ex1.m1.1.1.1.1"><plus id="A5.Ex1.m1.1.1.1.1.1.1.cmml" xref="A5.Ex1.m1.1.1.1.1.1.1"></plus><ci id="A5.Ex1.m1.1.1.1.1.1.2.cmml" xref="A5.Ex1.m1.1.1.1.1.1.2">𝑠</ci><ci id="A5.Ex1.m1.1.1.1.1.1.3.cmml" xref="A5.Ex1.m1.1.1.1.1.1.3">𝑛</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.Ex1.m1.1c">4brh(s+n)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="A5.p4" class="ltx_para">
<p id="A5.p4.7" class="ltx_p">Here, <math id="A5.p4.1.m1.1" class="ltx_Math" alttext="r" display="inline"><semantics id="A5.p4.1.m1.1a"><mi id="A5.p4.1.m1.1.1" xref="A5.p4.1.m1.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="A5.p4.1.m1.1b"><ci id="A5.p4.1.m1.1.1.cmml" xref="A5.p4.1.m1.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.p4.1.m1.1c">r</annotation></semantics></math> is the number of layers, <math id="A5.p4.2.m2.1" class="ltx_Math" alttext="b" display="inline"><semantics id="A5.p4.2.m2.1a"><mi id="A5.p4.2.m2.1.1" xref="A5.p4.2.m2.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="A5.p4.2.m2.1b"><ci id="A5.p4.2.m2.1.1.cmml" xref="A5.p4.2.m2.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.p4.2.m2.1c">b</annotation></semantics></math> is the batch size, <math id="A5.p4.3.m3.1" class="ltx_Math" alttext="h" display="inline"><semantics id="A5.p4.3.m3.1a"><mi id="A5.p4.3.m3.1.1" xref="A5.p4.3.m3.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="A5.p4.3.m3.1b"><ci id="A5.p4.3.m3.1.1.cmml" xref="A5.p4.3.m3.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.p4.3.m3.1c">h</annotation></semantics></math> is the hidden size, <math id="A5.p4.4.m4.1" class="ltx_Math" alttext="s" display="inline"><semantics id="A5.p4.4.m4.1a"><mi id="A5.p4.4.m4.1.1" xref="A5.p4.4.m4.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="A5.p4.4.m4.1b"><ci id="A5.p4.4.m4.1.1.cmml" xref="A5.p4.4.m4.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.p4.4.m4.1c">s</annotation></semantics></math> is the input sequence length, and <math id="A5.p4.5.m5.1" class="ltx_Math" alttext="n" display="inline"><semantics id="A5.p4.5.m5.1a"><mi id="A5.p4.5.m5.1.1" xref="A5.p4.5.m5.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="A5.p4.5.m5.1b"><ci id="A5.p4.5.m5.1.1.cmml" xref="A5.p4.5.m5.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.p4.5.m5.1c">n</annotation></semantics></math> is the output sequence length.
To improve efficiency, we begin merging layers starting from the midpoint, <math id="A5.p4.6.m6.1" class="ltx_Math" alttext="S=\frac{1}{2}r" display="inline"><semantics id="A5.p4.6.m6.1a"><mrow id="A5.p4.6.m6.1.1" xref="A5.p4.6.m6.1.1.cmml"><mi id="A5.p4.6.m6.1.1.2" xref="A5.p4.6.m6.1.1.2.cmml">S</mi><mo id="A5.p4.6.m6.1.1.1" xref="A5.p4.6.m6.1.1.1.cmml">=</mo><mrow id="A5.p4.6.m6.1.1.3" xref="A5.p4.6.m6.1.1.3.cmml"><mfrac id="A5.p4.6.m6.1.1.3.2" xref="A5.p4.6.m6.1.1.3.2.cmml"><mn id="A5.p4.6.m6.1.1.3.2.2" xref="A5.p4.6.m6.1.1.3.2.2.cmml">1</mn><mn id="A5.p4.6.m6.1.1.3.2.3" xref="A5.p4.6.m6.1.1.3.2.3.cmml">2</mn></mfrac><mo lspace="0em" rspace="0em" id="A5.p4.6.m6.1.1.3.1" xref="A5.p4.6.m6.1.1.3.1.cmml">​</mo><mi id="A5.p4.6.m6.1.1.3.3" xref="A5.p4.6.m6.1.1.3.3.cmml">r</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.p4.6.m6.1b"><apply id="A5.p4.6.m6.1.1.cmml" xref="A5.p4.6.m6.1.1"><eq id="A5.p4.6.m6.1.1.1.cmml" xref="A5.p4.6.m6.1.1.1"></eq><ci id="A5.p4.6.m6.1.1.2.cmml" xref="A5.p4.6.m6.1.1.2">𝑆</ci><apply id="A5.p4.6.m6.1.1.3.cmml" xref="A5.p4.6.m6.1.1.3"><times id="A5.p4.6.m6.1.1.3.1.cmml" xref="A5.p4.6.m6.1.1.3.1"></times><apply id="A5.p4.6.m6.1.1.3.2.cmml" xref="A5.p4.6.m6.1.1.3.2"><divide id="A5.p4.6.m6.1.1.3.2.1.cmml" xref="A5.p4.6.m6.1.1.3.2"></divide><cn type="integer" id="A5.p4.6.m6.1.1.3.2.2.cmml" xref="A5.p4.6.m6.1.1.3.2.2">1</cn><cn type="integer" id="A5.p4.6.m6.1.1.3.2.3.cmml" xref="A5.p4.6.m6.1.1.3.2.3">2</cn></apply><ci id="A5.p4.6.m6.1.1.3.3.cmml" xref="A5.p4.6.m6.1.1.3.3">𝑟</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p4.6.m6.1c">S=\frac{1}{2}r</annotation></semantics></math>, by consolidating the KV cache states of every two layers into a single shared state space. The memory usage derivation proceeds as follows:
For the unmerged part of the cache (from layer 1 to <math id="A5.p4.7.m7.1" class="ltx_Math" alttext="S" display="inline"><semantics id="A5.p4.7.m7.1a"><mi id="A5.p4.7.m7.1.1" xref="A5.p4.7.m7.1.1.cmml">S</mi><annotation-xml encoding="MathML-Content" id="A5.p4.7.m7.1b"><ci id="A5.p4.7.m7.1.1.cmml" xref="A5.p4.7.m7.1.1">𝑆</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.p4.7.m7.1c">S</annotation></semantics></math>):</p>
</div>
<div id="A5.p5" class="ltx_para">
<table id="A5.Ex2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A5.Ex2.m1.2" class="ltx_Math" alttext="4brh(s+n)\cdot\frac{1}{2}=2brh(s+n)" display="block"><semantics id="A5.Ex2.m1.2a"><mrow id="A5.Ex2.m1.2.2" xref="A5.Ex2.m1.2.2.cmml"><mrow id="A5.Ex2.m1.1.1.1" xref="A5.Ex2.m1.1.1.1.cmml"><mrow id="A5.Ex2.m1.1.1.1.1" xref="A5.Ex2.m1.1.1.1.1.cmml"><mn id="A5.Ex2.m1.1.1.1.1.3" xref="A5.Ex2.m1.1.1.1.1.3.cmml">4</mn><mo lspace="0em" rspace="0em" id="A5.Ex2.m1.1.1.1.1.2" xref="A5.Ex2.m1.1.1.1.1.2.cmml">​</mo><mi id="A5.Ex2.m1.1.1.1.1.4" xref="A5.Ex2.m1.1.1.1.1.4.cmml">b</mi><mo lspace="0em" rspace="0em" id="A5.Ex2.m1.1.1.1.1.2a" xref="A5.Ex2.m1.1.1.1.1.2.cmml">​</mo><mi id="A5.Ex2.m1.1.1.1.1.5" xref="A5.Ex2.m1.1.1.1.1.5.cmml">r</mi><mo lspace="0em" rspace="0em" id="A5.Ex2.m1.1.1.1.1.2b" xref="A5.Ex2.m1.1.1.1.1.2.cmml">​</mo><mi id="A5.Ex2.m1.1.1.1.1.6" xref="A5.Ex2.m1.1.1.1.1.6.cmml">h</mi><mo lspace="0em" rspace="0em" id="A5.Ex2.m1.1.1.1.1.2c" xref="A5.Ex2.m1.1.1.1.1.2.cmml">​</mo><mrow id="A5.Ex2.m1.1.1.1.1.1.1" xref="A5.Ex2.m1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A5.Ex2.m1.1.1.1.1.1.1.2" xref="A5.Ex2.m1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A5.Ex2.m1.1.1.1.1.1.1.1" xref="A5.Ex2.m1.1.1.1.1.1.1.1.cmml"><mi id="A5.Ex2.m1.1.1.1.1.1.1.1.2" xref="A5.Ex2.m1.1.1.1.1.1.1.1.2.cmml">s</mi><mo id="A5.Ex2.m1.1.1.1.1.1.1.1.1" xref="A5.Ex2.m1.1.1.1.1.1.1.1.1.cmml">+</mo><mi id="A5.Ex2.m1.1.1.1.1.1.1.1.3" xref="A5.Ex2.m1.1.1.1.1.1.1.1.3.cmml">n</mi></mrow><mo rspace="0.055em" stretchy="false" id="A5.Ex2.m1.1.1.1.1.1.1.3" xref="A5.Ex2.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo rspace="0.222em" id="A5.Ex2.m1.1.1.1.2" xref="A5.Ex2.m1.1.1.1.2.cmml">⋅</mo><mfrac id="A5.Ex2.m1.1.1.1.3" xref="A5.Ex2.m1.1.1.1.3.cmml"><mn id="A5.Ex2.m1.1.1.1.3.2" xref="A5.Ex2.m1.1.1.1.3.2.cmml">1</mn><mn id="A5.Ex2.m1.1.1.1.3.3" xref="A5.Ex2.m1.1.1.1.3.3.cmml">2</mn></mfrac></mrow><mo id="A5.Ex2.m1.2.2.3" xref="A5.Ex2.m1.2.2.3.cmml">=</mo><mrow id="A5.Ex2.m1.2.2.2" xref="A5.Ex2.m1.2.2.2.cmml"><mn id="A5.Ex2.m1.2.2.2.3" xref="A5.Ex2.m1.2.2.2.3.cmml">2</mn><mo lspace="0em" rspace="0em" id="A5.Ex2.m1.2.2.2.2" xref="A5.Ex2.m1.2.2.2.2.cmml">​</mo><mi id="A5.Ex2.m1.2.2.2.4" xref="A5.Ex2.m1.2.2.2.4.cmml">b</mi><mo lspace="0em" rspace="0em" id="A5.Ex2.m1.2.2.2.2a" xref="A5.Ex2.m1.2.2.2.2.cmml">​</mo><mi id="A5.Ex2.m1.2.2.2.5" xref="A5.Ex2.m1.2.2.2.5.cmml">r</mi><mo lspace="0em" rspace="0em" id="A5.Ex2.m1.2.2.2.2b" xref="A5.Ex2.m1.2.2.2.2.cmml">​</mo><mi id="A5.Ex2.m1.2.2.2.6" xref="A5.Ex2.m1.2.2.2.6.cmml">h</mi><mo lspace="0em" rspace="0em" id="A5.Ex2.m1.2.2.2.2c" xref="A5.Ex2.m1.2.2.2.2.cmml">​</mo><mrow id="A5.Ex2.m1.2.2.2.1.1" xref="A5.Ex2.m1.2.2.2.1.1.1.cmml"><mo stretchy="false" id="A5.Ex2.m1.2.2.2.1.1.2" xref="A5.Ex2.m1.2.2.2.1.1.1.cmml">(</mo><mrow id="A5.Ex2.m1.2.2.2.1.1.1" xref="A5.Ex2.m1.2.2.2.1.1.1.cmml"><mi id="A5.Ex2.m1.2.2.2.1.1.1.2" xref="A5.Ex2.m1.2.2.2.1.1.1.2.cmml">s</mi><mo id="A5.Ex2.m1.2.2.2.1.1.1.1" xref="A5.Ex2.m1.2.2.2.1.1.1.1.cmml">+</mo><mi id="A5.Ex2.m1.2.2.2.1.1.1.3" xref="A5.Ex2.m1.2.2.2.1.1.1.3.cmml">n</mi></mrow><mo stretchy="false" id="A5.Ex2.m1.2.2.2.1.1.3" xref="A5.Ex2.m1.2.2.2.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.Ex2.m1.2b"><apply id="A5.Ex2.m1.2.2.cmml" xref="A5.Ex2.m1.2.2"><eq id="A5.Ex2.m1.2.2.3.cmml" xref="A5.Ex2.m1.2.2.3"></eq><apply id="A5.Ex2.m1.1.1.1.cmml" xref="A5.Ex2.m1.1.1.1"><ci id="A5.Ex2.m1.1.1.1.2.cmml" xref="A5.Ex2.m1.1.1.1.2">⋅</ci><apply id="A5.Ex2.m1.1.1.1.1.cmml" xref="A5.Ex2.m1.1.1.1.1"><times id="A5.Ex2.m1.1.1.1.1.2.cmml" xref="A5.Ex2.m1.1.1.1.1.2"></times><cn type="integer" id="A5.Ex2.m1.1.1.1.1.3.cmml" xref="A5.Ex2.m1.1.1.1.1.3">4</cn><ci id="A5.Ex2.m1.1.1.1.1.4.cmml" xref="A5.Ex2.m1.1.1.1.1.4">𝑏</ci><ci id="A5.Ex2.m1.1.1.1.1.5.cmml" xref="A5.Ex2.m1.1.1.1.1.5">𝑟</ci><ci id="A5.Ex2.m1.1.1.1.1.6.cmml" xref="A5.Ex2.m1.1.1.1.1.6">ℎ</ci><apply id="A5.Ex2.m1.1.1.1.1.1.1.1.cmml" xref="A5.Ex2.m1.1.1.1.1.1.1"><plus id="A5.Ex2.m1.1.1.1.1.1.1.1.1.cmml" xref="A5.Ex2.m1.1.1.1.1.1.1.1.1"></plus><ci id="A5.Ex2.m1.1.1.1.1.1.1.1.2.cmml" xref="A5.Ex2.m1.1.1.1.1.1.1.1.2">𝑠</ci><ci id="A5.Ex2.m1.1.1.1.1.1.1.1.3.cmml" xref="A5.Ex2.m1.1.1.1.1.1.1.1.3">𝑛</ci></apply></apply><apply id="A5.Ex2.m1.1.1.1.3.cmml" xref="A5.Ex2.m1.1.1.1.3"><divide id="A5.Ex2.m1.1.1.1.3.1.cmml" xref="A5.Ex2.m1.1.1.1.3"></divide><cn type="integer" id="A5.Ex2.m1.1.1.1.3.2.cmml" xref="A5.Ex2.m1.1.1.1.3.2">1</cn><cn type="integer" id="A5.Ex2.m1.1.1.1.3.3.cmml" xref="A5.Ex2.m1.1.1.1.3.3">2</cn></apply></apply><apply id="A5.Ex2.m1.2.2.2.cmml" xref="A5.Ex2.m1.2.2.2"><times id="A5.Ex2.m1.2.2.2.2.cmml" xref="A5.Ex2.m1.2.2.2.2"></times><cn type="integer" id="A5.Ex2.m1.2.2.2.3.cmml" xref="A5.Ex2.m1.2.2.2.3">2</cn><ci id="A5.Ex2.m1.2.2.2.4.cmml" xref="A5.Ex2.m1.2.2.2.4">𝑏</ci><ci id="A5.Ex2.m1.2.2.2.5.cmml" xref="A5.Ex2.m1.2.2.2.5">𝑟</ci><ci id="A5.Ex2.m1.2.2.2.6.cmml" xref="A5.Ex2.m1.2.2.2.6">ℎ</ci><apply id="A5.Ex2.m1.2.2.2.1.1.1.cmml" xref="A5.Ex2.m1.2.2.2.1.1"><plus id="A5.Ex2.m1.2.2.2.1.1.1.1.cmml" xref="A5.Ex2.m1.2.2.2.1.1.1.1"></plus><ci id="A5.Ex2.m1.2.2.2.1.1.1.2.cmml" xref="A5.Ex2.m1.2.2.2.1.1.1.2">𝑠</ci><ci id="A5.Ex2.m1.2.2.2.1.1.1.3.cmml" xref="A5.Ex2.m1.2.2.2.1.1.1.3">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.Ex2.m1.2c">4brh(s+n)\cdot\frac{1}{2}=2brh(s+n)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="A5.p6" class="ltx_para">
<p id="A5.p6.2" class="ltx_p">For the merged part of the cache (from layer <math id="A5.p6.1.m1.1" class="ltx_Math" alttext="S+1" display="inline"><semantics id="A5.p6.1.m1.1a"><mrow id="A5.p6.1.m1.1.1" xref="A5.p6.1.m1.1.1.cmml"><mi id="A5.p6.1.m1.1.1.2" xref="A5.p6.1.m1.1.1.2.cmml">S</mi><mo id="A5.p6.1.m1.1.1.1" xref="A5.p6.1.m1.1.1.1.cmml">+</mo><mn id="A5.p6.1.m1.1.1.3" xref="A5.p6.1.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.p6.1.m1.1b"><apply id="A5.p6.1.m1.1.1.cmml" xref="A5.p6.1.m1.1.1"><plus id="A5.p6.1.m1.1.1.1.cmml" xref="A5.p6.1.m1.1.1.1"></plus><ci id="A5.p6.1.m1.1.1.2.cmml" xref="A5.p6.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.p6.1.m1.1.1.3.cmml" xref="A5.p6.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p6.1.m1.1c">S+1</annotation></semantics></math> to <math id="A5.p6.2.m2.1" class="ltx_Math" alttext="r" display="inline"><semantics id="A5.p6.2.m2.1a"><mi id="A5.p6.2.m2.1.1" xref="A5.p6.2.m2.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="A5.p6.2.m2.1b"><ci id="A5.p6.2.m2.1.1.cmml" xref="A5.p6.2.m2.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.p6.2.m2.1c">r</annotation></semantics></math>):</p>
</div>
<div id="A5.p7" class="ltx_para">
<table id="A5.Ex3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A5.Ex3.m1.2" class="ltx_Math" alttext="4brh(s+n)\cdot\frac{1}{2}\cdot\frac{1}{2}=brh(s+n)" display="block"><semantics id="A5.Ex3.m1.2a"><mrow id="A5.Ex3.m1.2.2" xref="A5.Ex3.m1.2.2.cmml"><mrow id="A5.Ex3.m1.1.1.1" xref="A5.Ex3.m1.1.1.1.cmml"><mrow id="A5.Ex3.m1.1.1.1.1" xref="A5.Ex3.m1.1.1.1.1.cmml"><mn id="A5.Ex3.m1.1.1.1.1.3" xref="A5.Ex3.m1.1.1.1.1.3.cmml">4</mn><mo lspace="0em" rspace="0em" id="A5.Ex3.m1.1.1.1.1.2" xref="A5.Ex3.m1.1.1.1.1.2.cmml">​</mo><mi id="A5.Ex3.m1.1.1.1.1.4" xref="A5.Ex3.m1.1.1.1.1.4.cmml">b</mi><mo lspace="0em" rspace="0em" id="A5.Ex3.m1.1.1.1.1.2a" xref="A5.Ex3.m1.1.1.1.1.2.cmml">​</mo><mi id="A5.Ex3.m1.1.1.1.1.5" xref="A5.Ex3.m1.1.1.1.1.5.cmml">r</mi><mo lspace="0em" rspace="0em" id="A5.Ex3.m1.1.1.1.1.2b" xref="A5.Ex3.m1.1.1.1.1.2.cmml">​</mo><mi id="A5.Ex3.m1.1.1.1.1.6" xref="A5.Ex3.m1.1.1.1.1.6.cmml">h</mi><mo lspace="0em" rspace="0em" id="A5.Ex3.m1.1.1.1.1.2c" xref="A5.Ex3.m1.1.1.1.1.2.cmml">​</mo><mrow id="A5.Ex3.m1.1.1.1.1.1.1" xref="A5.Ex3.m1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A5.Ex3.m1.1.1.1.1.1.1.2" xref="A5.Ex3.m1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A5.Ex3.m1.1.1.1.1.1.1.1" xref="A5.Ex3.m1.1.1.1.1.1.1.1.cmml"><mi id="A5.Ex3.m1.1.1.1.1.1.1.1.2" xref="A5.Ex3.m1.1.1.1.1.1.1.1.2.cmml">s</mi><mo id="A5.Ex3.m1.1.1.1.1.1.1.1.1" xref="A5.Ex3.m1.1.1.1.1.1.1.1.1.cmml">+</mo><mi id="A5.Ex3.m1.1.1.1.1.1.1.1.3" xref="A5.Ex3.m1.1.1.1.1.1.1.1.3.cmml">n</mi></mrow><mo rspace="0.055em" stretchy="false" id="A5.Ex3.m1.1.1.1.1.1.1.3" xref="A5.Ex3.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo rspace="0.222em" id="A5.Ex3.m1.1.1.1.2" xref="A5.Ex3.m1.1.1.1.2.cmml">⋅</mo><mfrac id="A5.Ex3.m1.1.1.1.3" xref="A5.Ex3.m1.1.1.1.3.cmml"><mn id="A5.Ex3.m1.1.1.1.3.2" xref="A5.Ex3.m1.1.1.1.3.2.cmml">1</mn><mn id="A5.Ex3.m1.1.1.1.3.3" xref="A5.Ex3.m1.1.1.1.3.3.cmml">2</mn></mfrac><mo lspace="0.222em" rspace="0.222em" id="A5.Ex3.m1.1.1.1.2a" xref="A5.Ex3.m1.1.1.1.2.cmml">⋅</mo><mfrac id="A5.Ex3.m1.1.1.1.4" xref="A5.Ex3.m1.1.1.1.4.cmml"><mn id="A5.Ex3.m1.1.1.1.4.2" xref="A5.Ex3.m1.1.1.1.4.2.cmml">1</mn><mn id="A5.Ex3.m1.1.1.1.4.3" xref="A5.Ex3.m1.1.1.1.4.3.cmml">2</mn></mfrac></mrow><mo id="A5.Ex3.m1.2.2.3" xref="A5.Ex3.m1.2.2.3.cmml">=</mo><mrow id="A5.Ex3.m1.2.2.2" xref="A5.Ex3.m1.2.2.2.cmml"><mi id="A5.Ex3.m1.2.2.2.3" xref="A5.Ex3.m1.2.2.2.3.cmml">b</mi><mo lspace="0em" rspace="0em" id="A5.Ex3.m1.2.2.2.2" xref="A5.Ex3.m1.2.2.2.2.cmml">​</mo><mi id="A5.Ex3.m1.2.2.2.4" xref="A5.Ex3.m1.2.2.2.4.cmml">r</mi><mo lspace="0em" rspace="0em" id="A5.Ex3.m1.2.2.2.2a" xref="A5.Ex3.m1.2.2.2.2.cmml">​</mo><mi id="A5.Ex3.m1.2.2.2.5" xref="A5.Ex3.m1.2.2.2.5.cmml">h</mi><mo lspace="0em" rspace="0em" id="A5.Ex3.m1.2.2.2.2b" xref="A5.Ex3.m1.2.2.2.2.cmml">​</mo><mrow id="A5.Ex3.m1.2.2.2.1.1" xref="A5.Ex3.m1.2.2.2.1.1.1.cmml"><mo stretchy="false" id="A5.Ex3.m1.2.2.2.1.1.2" xref="A5.Ex3.m1.2.2.2.1.1.1.cmml">(</mo><mrow id="A5.Ex3.m1.2.2.2.1.1.1" xref="A5.Ex3.m1.2.2.2.1.1.1.cmml"><mi id="A5.Ex3.m1.2.2.2.1.1.1.2" xref="A5.Ex3.m1.2.2.2.1.1.1.2.cmml">s</mi><mo id="A5.Ex3.m1.2.2.2.1.1.1.1" xref="A5.Ex3.m1.2.2.2.1.1.1.1.cmml">+</mo><mi id="A5.Ex3.m1.2.2.2.1.1.1.3" xref="A5.Ex3.m1.2.2.2.1.1.1.3.cmml">n</mi></mrow><mo stretchy="false" id="A5.Ex3.m1.2.2.2.1.1.3" xref="A5.Ex3.m1.2.2.2.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.Ex3.m1.2b"><apply id="A5.Ex3.m1.2.2.cmml" xref="A5.Ex3.m1.2.2"><eq id="A5.Ex3.m1.2.2.3.cmml" xref="A5.Ex3.m1.2.2.3"></eq><apply id="A5.Ex3.m1.1.1.1.cmml" xref="A5.Ex3.m1.1.1.1"><ci id="A5.Ex3.m1.1.1.1.2.cmml" xref="A5.Ex3.m1.1.1.1.2">⋅</ci><apply id="A5.Ex3.m1.1.1.1.1.cmml" xref="A5.Ex3.m1.1.1.1.1"><times id="A5.Ex3.m1.1.1.1.1.2.cmml" xref="A5.Ex3.m1.1.1.1.1.2"></times><cn type="integer" id="A5.Ex3.m1.1.1.1.1.3.cmml" xref="A5.Ex3.m1.1.1.1.1.3">4</cn><ci id="A5.Ex3.m1.1.1.1.1.4.cmml" xref="A5.Ex3.m1.1.1.1.1.4">𝑏</ci><ci id="A5.Ex3.m1.1.1.1.1.5.cmml" xref="A5.Ex3.m1.1.1.1.1.5">𝑟</ci><ci id="A5.Ex3.m1.1.1.1.1.6.cmml" xref="A5.Ex3.m1.1.1.1.1.6">ℎ</ci><apply id="A5.Ex3.m1.1.1.1.1.1.1.1.cmml" xref="A5.Ex3.m1.1.1.1.1.1.1"><plus id="A5.Ex3.m1.1.1.1.1.1.1.1.1.cmml" xref="A5.Ex3.m1.1.1.1.1.1.1.1.1"></plus><ci id="A5.Ex3.m1.1.1.1.1.1.1.1.2.cmml" xref="A5.Ex3.m1.1.1.1.1.1.1.1.2">𝑠</ci><ci id="A5.Ex3.m1.1.1.1.1.1.1.1.3.cmml" xref="A5.Ex3.m1.1.1.1.1.1.1.1.3">𝑛</ci></apply></apply><apply id="A5.Ex3.m1.1.1.1.3.cmml" xref="A5.Ex3.m1.1.1.1.3"><divide id="A5.Ex3.m1.1.1.1.3.1.cmml" xref="A5.Ex3.m1.1.1.1.3"></divide><cn type="integer" id="A5.Ex3.m1.1.1.1.3.2.cmml" xref="A5.Ex3.m1.1.1.1.3.2">1</cn><cn type="integer" id="A5.Ex3.m1.1.1.1.3.3.cmml" xref="A5.Ex3.m1.1.1.1.3.3">2</cn></apply><apply id="A5.Ex3.m1.1.1.1.4.cmml" xref="A5.Ex3.m1.1.1.1.4"><divide id="A5.Ex3.m1.1.1.1.4.1.cmml" xref="A5.Ex3.m1.1.1.1.4"></divide><cn type="integer" id="A5.Ex3.m1.1.1.1.4.2.cmml" xref="A5.Ex3.m1.1.1.1.4.2">1</cn><cn type="integer" id="A5.Ex3.m1.1.1.1.4.3.cmml" xref="A5.Ex3.m1.1.1.1.4.3">2</cn></apply></apply><apply id="A5.Ex3.m1.2.2.2.cmml" xref="A5.Ex3.m1.2.2.2"><times id="A5.Ex3.m1.2.2.2.2.cmml" xref="A5.Ex3.m1.2.2.2.2"></times><ci id="A5.Ex3.m1.2.2.2.3.cmml" xref="A5.Ex3.m1.2.2.2.3">𝑏</ci><ci id="A5.Ex3.m1.2.2.2.4.cmml" xref="A5.Ex3.m1.2.2.2.4">𝑟</ci><ci id="A5.Ex3.m1.2.2.2.5.cmml" xref="A5.Ex3.m1.2.2.2.5">ℎ</ci><apply id="A5.Ex3.m1.2.2.2.1.1.1.cmml" xref="A5.Ex3.m1.2.2.2.1.1"><plus id="A5.Ex3.m1.2.2.2.1.1.1.1.cmml" xref="A5.Ex3.m1.2.2.2.1.1.1.1"></plus><ci id="A5.Ex3.m1.2.2.2.1.1.1.2.cmml" xref="A5.Ex3.m1.2.2.2.1.1.1.2">𝑠</ci><ci id="A5.Ex3.m1.2.2.2.1.1.1.3.cmml" xref="A5.Ex3.m1.2.2.2.1.1.1.3">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.Ex3.m1.2c">4brh(s+n)\cdot\frac{1}{2}\cdot\frac{1}{2}=brh(s+n)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="A5.p8" class="ltx_para">
<p id="A5.p8.1" class="ltx_p">Combining these two parts, the total memory usage is:</p>
</div>
<div id="A5.p9" class="ltx_para">
<table id="A5.Ex4" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A5.Ex4.m1.3" class="ltx_Math" alttext="2brh(s+n)+brh(s+n)=3brh(s+n)" display="block"><semantics id="A5.Ex4.m1.3a"><mrow id="A5.Ex4.m1.3.3" xref="A5.Ex4.m1.3.3.cmml"><mrow id="A5.Ex4.m1.2.2.2" xref="A5.Ex4.m1.2.2.2.cmml"><mrow id="A5.Ex4.m1.1.1.1.1" xref="A5.Ex4.m1.1.1.1.1.cmml"><mn id="A5.Ex4.m1.1.1.1.1.3" xref="A5.Ex4.m1.1.1.1.1.3.cmml">2</mn><mo lspace="0em" rspace="0em" id="A5.Ex4.m1.1.1.1.1.2" xref="A5.Ex4.m1.1.1.1.1.2.cmml">​</mo><mi id="A5.Ex4.m1.1.1.1.1.4" xref="A5.Ex4.m1.1.1.1.1.4.cmml">b</mi><mo lspace="0em" rspace="0em" id="A5.Ex4.m1.1.1.1.1.2a" xref="A5.Ex4.m1.1.1.1.1.2.cmml">​</mo><mi id="A5.Ex4.m1.1.1.1.1.5" xref="A5.Ex4.m1.1.1.1.1.5.cmml">r</mi><mo lspace="0em" rspace="0em" id="A5.Ex4.m1.1.1.1.1.2b" xref="A5.Ex4.m1.1.1.1.1.2.cmml">​</mo><mi id="A5.Ex4.m1.1.1.1.1.6" xref="A5.Ex4.m1.1.1.1.1.6.cmml">h</mi><mo lspace="0em" rspace="0em" id="A5.Ex4.m1.1.1.1.1.2c" xref="A5.Ex4.m1.1.1.1.1.2.cmml">​</mo><mrow id="A5.Ex4.m1.1.1.1.1.1.1" xref="A5.Ex4.m1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A5.Ex4.m1.1.1.1.1.1.1.2" xref="A5.Ex4.m1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A5.Ex4.m1.1.1.1.1.1.1.1" xref="A5.Ex4.m1.1.1.1.1.1.1.1.cmml"><mi id="A5.Ex4.m1.1.1.1.1.1.1.1.2" xref="A5.Ex4.m1.1.1.1.1.1.1.1.2.cmml">s</mi><mo id="A5.Ex4.m1.1.1.1.1.1.1.1.1" xref="A5.Ex4.m1.1.1.1.1.1.1.1.1.cmml">+</mo><mi id="A5.Ex4.m1.1.1.1.1.1.1.1.3" xref="A5.Ex4.m1.1.1.1.1.1.1.1.3.cmml">n</mi></mrow><mo stretchy="false" id="A5.Ex4.m1.1.1.1.1.1.1.3" xref="A5.Ex4.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="A5.Ex4.m1.2.2.2.3" xref="A5.Ex4.m1.2.2.2.3.cmml">+</mo><mrow id="A5.Ex4.m1.2.2.2.2" xref="A5.Ex4.m1.2.2.2.2.cmml"><mi id="A5.Ex4.m1.2.2.2.2.3" xref="A5.Ex4.m1.2.2.2.2.3.cmml">b</mi><mo lspace="0em" rspace="0em" id="A5.Ex4.m1.2.2.2.2.2" xref="A5.Ex4.m1.2.2.2.2.2.cmml">​</mo><mi id="A5.Ex4.m1.2.2.2.2.4" xref="A5.Ex4.m1.2.2.2.2.4.cmml">r</mi><mo lspace="0em" rspace="0em" id="A5.Ex4.m1.2.2.2.2.2a" xref="A5.Ex4.m1.2.2.2.2.2.cmml">​</mo><mi id="A5.Ex4.m1.2.2.2.2.5" xref="A5.Ex4.m1.2.2.2.2.5.cmml">h</mi><mo lspace="0em" rspace="0em" id="A5.Ex4.m1.2.2.2.2.2b" xref="A5.Ex4.m1.2.2.2.2.2.cmml">​</mo><mrow id="A5.Ex4.m1.2.2.2.2.1.1" xref="A5.Ex4.m1.2.2.2.2.1.1.1.cmml"><mo stretchy="false" id="A5.Ex4.m1.2.2.2.2.1.1.2" xref="A5.Ex4.m1.2.2.2.2.1.1.1.cmml">(</mo><mrow id="A5.Ex4.m1.2.2.2.2.1.1.1" xref="A5.Ex4.m1.2.2.2.2.1.1.1.cmml"><mi id="A5.Ex4.m1.2.2.2.2.1.1.1.2" xref="A5.Ex4.m1.2.2.2.2.1.1.1.2.cmml">s</mi><mo id="A5.Ex4.m1.2.2.2.2.1.1.1.1" xref="A5.Ex4.m1.2.2.2.2.1.1.1.1.cmml">+</mo><mi id="A5.Ex4.m1.2.2.2.2.1.1.1.3" xref="A5.Ex4.m1.2.2.2.2.1.1.1.3.cmml">n</mi></mrow><mo stretchy="false" id="A5.Ex4.m1.2.2.2.2.1.1.3" xref="A5.Ex4.m1.2.2.2.2.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo id="A5.Ex4.m1.3.3.4" xref="A5.Ex4.m1.3.3.4.cmml">=</mo><mrow id="A5.Ex4.m1.3.3.3" xref="A5.Ex4.m1.3.3.3.cmml"><mn id="A5.Ex4.m1.3.3.3.3" xref="A5.Ex4.m1.3.3.3.3.cmml">3</mn><mo lspace="0em" rspace="0em" id="A5.Ex4.m1.3.3.3.2" xref="A5.Ex4.m1.3.3.3.2.cmml">​</mo><mi id="A5.Ex4.m1.3.3.3.4" xref="A5.Ex4.m1.3.3.3.4.cmml">b</mi><mo lspace="0em" rspace="0em" id="A5.Ex4.m1.3.3.3.2a" xref="A5.Ex4.m1.3.3.3.2.cmml">​</mo><mi id="A5.Ex4.m1.3.3.3.5" xref="A5.Ex4.m1.3.3.3.5.cmml">r</mi><mo lspace="0em" rspace="0em" id="A5.Ex4.m1.3.3.3.2b" xref="A5.Ex4.m1.3.3.3.2.cmml">​</mo><mi id="A5.Ex4.m1.3.3.3.6" xref="A5.Ex4.m1.3.3.3.6.cmml">h</mi><mo lspace="0em" rspace="0em" id="A5.Ex4.m1.3.3.3.2c" xref="A5.Ex4.m1.3.3.3.2.cmml">​</mo><mrow id="A5.Ex4.m1.3.3.3.1.1" xref="A5.Ex4.m1.3.3.3.1.1.1.cmml"><mo stretchy="false" id="A5.Ex4.m1.3.3.3.1.1.2" xref="A5.Ex4.m1.3.3.3.1.1.1.cmml">(</mo><mrow id="A5.Ex4.m1.3.3.3.1.1.1" xref="A5.Ex4.m1.3.3.3.1.1.1.cmml"><mi id="A5.Ex4.m1.3.3.3.1.1.1.2" xref="A5.Ex4.m1.3.3.3.1.1.1.2.cmml">s</mi><mo id="A5.Ex4.m1.3.3.3.1.1.1.1" xref="A5.Ex4.m1.3.3.3.1.1.1.1.cmml">+</mo><mi id="A5.Ex4.m1.3.3.3.1.1.1.3" xref="A5.Ex4.m1.3.3.3.1.1.1.3.cmml">n</mi></mrow><mo stretchy="false" id="A5.Ex4.m1.3.3.3.1.1.3" xref="A5.Ex4.m1.3.3.3.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.Ex4.m1.3b"><apply id="A5.Ex4.m1.3.3.cmml" xref="A5.Ex4.m1.3.3"><eq id="A5.Ex4.m1.3.3.4.cmml" xref="A5.Ex4.m1.3.3.4"></eq><apply id="A5.Ex4.m1.2.2.2.cmml" xref="A5.Ex4.m1.2.2.2"><plus id="A5.Ex4.m1.2.2.2.3.cmml" xref="A5.Ex4.m1.2.2.2.3"></plus><apply id="A5.Ex4.m1.1.1.1.1.cmml" xref="A5.Ex4.m1.1.1.1.1"><times id="A5.Ex4.m1.1.1.1.1.2.cmml" xref="A5.Ex4.m1.1.1.1.1.2"></times><cn type="integer" id="A5.Ex4.m1.1.1.1.1.3.cmml" xref="A5.Ex4.m1.1.1.1.1.3">2</cn><ci id="A5.Ex4.m1.1.1.1.1.4.cmml" xref="A5.Ex4.m1.1.1.1.1.4">𝑏</ci><ci id="A5.Ex4.m1.1.1.1.1.5.cmml" xref="A5.Ex4.m1.1.1.1.1.5">𝑟</ci><ci id="A5.Ex4.m1.1.1.1.1.6.cmml" xref="A5.Ex4.m1.1.1.1.1.6">ℎ</ci><apply id="A5.Ex4.m1.1.1.1.1.1.1.1.cmml" xref="A5.Ex4.m1.1.1.1.1.1.1"><plus id="A5.Ex4.m1.1.1.1.1.1.1.1.1.cmml" xref="A5.Ex4.m1.1.1.1.1.1.1.1.1"></plus><ci id="A5.Ex4.m1.1.1.1.1.1.1.1.2.cmml" xref="A5.Ex4.m1.1.1.1.1.1.1.1.2">𝑠</ci><ci id="A5.Ex4.m1.1.1.1.1.1.1.1.3.cmml" xref="A5.Ex4.m1.1.1.1.1.1.1.1.3">𝑛</ci></apply></apply><apply id="A5.Ex4.m1.2.2.2.2.cmml" xref="A5.Ex4.m1.2.2.2.2"><times id="A5.Ex4.m1.2.2.2.2.2.cmml" xref="A5.Ex4.m1.2.2.2.2.2"></times><ci id="A5.Ex4.m1.2.2.2.2.3.cmml" xref="A5.Ex4.m1.2.2.2.2.3">𝑏</ci><ci id="A5.Ex4.m1.2.2.2.2.4.cmml" xref="A5.Ex4.m1.2.2.2.2.4">𝑟</ci><ci id="A5.Ex4.m1.2.2.2.2.5.cmml" xref="A5.Ex4.m1.2.2.2.2.5">ℎ</ci><apply id="A5.Ex4.m1.2.2.2.2.1.1.1.cmml" xref="A5.Ex4.m1.2.2.2.2.1.1"><plus id="A5.Ex4.m1.2.2.2.2.1.1.1.1.cmml" xref="A5.Ex4.m1.2.2.2.2.1.1.1.1"></plus><ci id="A5.Ex4.m1.2.2.2.2.1.1.1.2.cmml" xref="A5.Ex4.m1.2.2.2.2.1.1.1.2">𝑠</ci><ci id="A5.Ex4.m1.2.2.2.2.1.1.1.3.cmml" xref="A5.Ex4.m1.2.2.2.2.1.1.1.3">𝑛</ci></apply></apply></apply><apply id="A5.Ex4.m1.3.3.3.cmml" xref="A5.Ex4.m1.3.3.3"><times id="A5.Ex4.m1.3.3.3.2.cmml" xref="A5.Ex4.m1.3.3.3.2"></times><cn type="integer" id="A5.Ex4.m1.3.3.3.3.cmml" xref="A5.Ex4.m1.3.3.3.3">3</cn><ci id="A5.Ex4.m1.3.3.3.4.cmml" xref="A5.Ex4.m1.3.3.3.4">𝑏</ci><ci id="A5.Ex4.m1.3.3.3.5.cmml" xref="A5.Ex4.m1.3.3.3.5">𝑟</ci><ci id="A5.Ex4.m1.3.3.3.6.cmml" xref="A5.Ex4.m1.3.3.3.6">ℎ</ci><apply id="A5.Ex4.m1.3.3.3.1.1.1.cmml" xref="A5.Ex4.m1.3.3.3.1.1"><plus id="A5.Ex4.m1.3.3.3.1.1.1.1.cmml" xref="A5.Ex4.m1.3.3.3.1.1.1.1"></plus><ci id="A5.Ex4.m1.3.3.3.1.1.1.2.cmml" xref="A5.Ex4.m1.3.3.3.1.1.1.2">𝑠</ci><ci id="A5.Ex4.m1.3.3.3.1.1.1.3.cmml" xref="A5.Ex4.m1.3.3.3.1.1.1.3">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.Ex4.m1.3c">2brh(s+n)+brh(s+n)=3brh(s+n)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="A5.p10" class="ltx_para">
<p id="A5.p10.1" class="ltx_p">Next, we consider the additional memory cost incurred during the restoration process. During this phase, we save additional normalized vectors for the layers in the KV cache. These vectors are of shape <math id="A5.p10.1.m1.1" class="ltx_Math" alttext="\mathbb{R}^{b\times s\times 1}" display="inline"><semantics id="A5.p10.1.m1.1a"><msup id="A5.p10.1.m1.1.1" xref="A5.p10.1.m1.1.1.cmml"><mi id="A5.p10.1.m1.1.1.2" xref="A5.p10.1.m1.1.1.2.cmml">ℝ</mi><mrow id="A5.p10.1.m1.1.1.3" xref="A5.p10.1.m1.1.1.3.cmml"><mi id="A5.p10.1.m1.1.1.3.2" xref="A5.p10.1.m1.1.1.3.2.cmml">b</mi><mo lspace="0.222em" rspace="0.222em" id="A5.p10.1.m1.1.1.3.1" xref="A5.p10.1.m1.1.1.3.1.cmml">×</mo><mi id="A5.p10.1.m1.1.1.3.3" xref="A5.p10.1.m1.1.1.3.3.cmml">s</mi><mo lspace="0.222em" rspace="0.222em" id="A5.p10.1.m1.1.1.3.1a" xref="A5.p10.1.m1.1.1.3.1.cmml">×</mo><mn id="A5.p10.1.m1.1.1.3.4" xref="A5.p10.1.m1.1.1.3.4.cmml">1</mn></mrow></msup><annotation-xml encoding="MathML-Content" id="A5.p10.1.m1.1b"><apply id="A5.p10.1.m1.1.1.cmml" xref="A5.p10.1.m1.1.1"><csymbol cd="ambiguous" id="A5.p10.1.m1.1.1.1.cmml" xref="A5.p10.1.m1.1.1">superscript</csymbol><ci id="A5.p10.1.m1.1.1.2.cmml" xref="A5.p10.1.m1.1.1.2">ℝ</ci><apply id="A5.p10.1.m1.1.1.3.cmml" xref="A5.p10.1.m1.1.1.3"><times id="A5.p10.1.m1.1.1.3.1.cmml" xref="A5.p10.1.m1.1.1.3.1"></times><ci id="A5.p10.1.m1.1.1.3.2.cmml" xref="A5.p10.1.m1.1.1.3.2">𝑏</ci><ci id="A5.p10.1.m1.1.1.3.3.cmml" xref="A5.p10.1.m1.1.1.3.3">𝑠</ci><cn type="integer" id="A5.p10.1.m1.1.1.3.4.cmml" xref="A5.p10.1.m1.1.1.3.4">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p10.1.m1.1c">\mathbb{R}^{b\times s\times 1}</annotation></semantics></math>, which means they have a single channel dimension compared to the fully ranked original KV states.</p>
</div>
<div id="A5.p11" class="ltx_para">
<p id="A5.p11.1" class="ltx_p">The additional normalized vectors for layers from <math id="A5.p11.1.m1.1" class="ltx_Math" alttext="S" display="inline"><semantics id="A5.p11.1.m1.1a"><mi id="A5.p11.1.m1.1.1" xref="A5.p11.1.m1.1.1.cmml">S</mi><annotation-xml encoding="MathML-Content" id="A5.p11.1.m1.1b"><ci id="A5.p11.1.m1.1.1.cmml" xref="A5.p11.1.m1.1.1">𝑆</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.p11.1.m1.1c">S</annotation></semantics></math> onwards are given by:</p>
</div>
<div id="A5.p12" class="ltx_para">
<table id="A5.Ex5" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A5.Ex5.m1.2" class="ltx_Math" alttext="br(s+n)\cdot 2=2br(s+n)" display="block"><semantics id="A5.Ex5.m1.2a"><mrow id="A5.Ex5.m1.2.2" xref="A5.Ex5.m1.2.2.cmml"><mrow id="A5.Ex5.m1.1.1.1" xref="A5.Ex5.m1.1.1.1.cmml"><mrow id="A5.Ex5.m1.1.1.1.1" xref="A5.Ex5.m1.1.1.1.1.cmml"><mi id="A5.Ex5.m1.1.1.1.1.3" xref="A5.Ex5.m1.1.1.1.1.3.cmml">b</mi><mo lspace="0em" rspace="0em" id="A5.Ex5.m1.1.1.1.1.2" xref="A5.Ex5.m1.1.1.1.1.2.cmml">​</mo><mi id="A5.Ex5.m1.1.1.1.1.4" xref="A5.Ex5.m1.1.1.1.1.4.cmml">r</mi><mo lspace="0em" rspace="0em" id="A5.Ex5.m1.1.1.1.1.2a" xref="A5.Ex5.m1.1.1.1.1.2.cmml">​</mo><mrow id="A5.Ex5.m1.1.1.1.1.1.1" xref="A5.Ex5.m1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A5.Ex5.m1.1.1.1.1.1.1.2" xref="A5.Ex5.m1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A5.Ex5.m1.1.1.1.1.1.1.1" xref="A5.Ex5.m1.1.1.1.1.1.1.1.cmml"><mi id="A5.Ex5.m1.1.1.1.1.1.1.1.2" xref="A5.Ex5.m1.1.1.1.1.1.1.1.2.cmml">s</mi><mo id="A5.Ex5.m1.1.1.1.1.1.1.1.1" xref="A5.Ex5.m1.1.1.1.1.1.1.1.1.cmml">+</mo><mi id="A5.Ex5.m1.1.1.1.1.1.1.1.3" xref="A5.Ex5.m1.1.1.1.1.1.1.1.3.cmml">n</mi></mrow><mo rspace="0.055em" stretchy="false" id="A5.Ex5.m1.1.1.1.1.1.1.3" xref="A5.Ex5.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo rspace="0.222em" id="A5.Ex5.m1.1.1.1.2" xref="A5.Ex5.m1.1.1.1.2.cmml">⋅</mo><mn id="A5.Ex5.m1.1.1.1.3" xref="A5.Ex5.m1.1.1.1.3.cmml">2</mn></mrow><mo id="A5.Ex5.m1.2.2.3" xref="A5.Ex5.m1.2.2.3.cmml">=</mo><mrow id="A5.Ex5.m1.2.2.2" xref="A5.Ex5.m1.2.2.2.cmml"><mn id="A5.Ex5.m1.2.2.2.3" xref="A5.Ex5.m1.2.2.2.3.cmml">2</mn><mo lspace="0em" rspace="0em" id="A5.Ex5.m1.2.2.2.2" xref="A5.Ex5.m1.2.2.2.2.cmml">​</mo><mi id="A5.Ex5.m1.2.2.2.4" xref="A5.Ex5.m1.2.2.2.4.cmml">b</mi><mo lspace="0em" rspace="0em" id="A5.Ex5.m1.2.2.2.2a" xref="A5.Ex5.m1.2.2.2.2.cmml">​</mo><mi id="A5.Ex5.m1.2.2.2.5" xref="A5.Ex5.m1.2.2.2.5.cmml">r</mi><mo lspace="0em" rspace="0em" id="A5.Ex5.m1.2.2.2.2b" xref="A5.Ex5.m1.2.2.2.2.cmml">​</mo><mrow id="A5.Ex5.m1.2.2.2.1.1" xref="A5.Ex5.m1.2.2.2.1.1.1.cmml"><mo stretchy="false" id="A5.Ex5.m1.2.2.2.1.1.2" xref="A5.Ex5.m1.2.2.2.1.1.1.cmml">(</mo><mrow id="A5.Ex5.m1.2.2.2.1.1.1" xref="A5.Ex5.m1.2.2.2.1.1.1.cmml"><mi id="A5.Ex5.m1.2.2.2.1.1.1.2" xref="A5.Ex5.m1.2.2.2.1.1.1.2.cmml">s</mi><mo id="A5.Ex5.m1.2.2.2.1.1.1.1" xref="A5.Ex5.m1.2.2.2.1.1.1.1.cmml">+</mo><mi id="A5.Ex5.m1.2.2.2.1.1.1.3" xref="A5.Ex5.m1.2.2.2.1.1.1.3.cmml">n</mi></mrow><mo stretchy="false" id="A5.Ex5.m1.2.2.2.1.1.3" xref="A5.Ex5.m1.2.2.2.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.Ex5.m1.2b"><apply id="A5.Ex5.m1.2.2.cmml" xref="A5.Ex5.m1.2.2"><eq id="A5.Ex5.m1.2.2.3.cmml" xref="A5.Ex5.m1.2.2.3"></eq><apply id="A5.Ex5.m1.1.1.1.cmml" xref="A5.Ex5.m1.1.1.1"><ci id="A5.Ex5.m1.1.1.1.2.cmml" xref="A5.Ex5.m1.1.1.1.2">⋅</ci><apply id="A5.Ex5.m1.1.1.1.1.cmml" xref="A5.Ex5.m1.1.1.1.1"><times id="A5.Ex5.m1.1.1.1.1.2.cmml" xref="A5.Ex5.m1.1.1.1.1.2"></times><ci id="A5.Ex5.m1.1.1.1.1.3.cmml" xref="A5.Ex5.m1.1.1.1.1.3">𝑏</ci><ci id="A5.Ex5.m1.1.1.1.1.4.cmml" xref="A5.Ex5.m1.1.1.1.1.4">𝑟</ci><apply id="A5.Ex5.m1.1.1.1.1.1.1.1.cmml" xref="A5.Ex5.m1.1.1.1.1.1.1"><plus id="A5.Ex5.m1.1.1.1.1.1.1.1.1.cmml" xref="A5.Ex5.m1.1.1.1.1.1.1.1.1"></plus><ci id="A5.Ex5.m1.1.1.1.1.1.1.1.2.cmml" xref="A5.Ex5.m1.1.1.1.1.1.1.1.2">𝑠</ci><ci id="A5.Ex5.m1.1.1.1.1.1.1.1.3.cmml" xref="A5.Ex5.m1.1.1.1.1.1.1.1.3">𝑛</ci></apply></apply><cn type="integer" id="A5.Ex5.m1.1.1.1.3.cmml" xref="A5.Ex5.m1.1.1.1.3">2</cn></apply><apply id="A5.Ex5.m1.2.2.2.cmml" xref="A5.Ex5.m1.2.2.2"><times id="A5.Ex5.m1.2.2.2.2.cmml" xref="A5.Ex5.m1.2.2.2.2"></times><cn type="integer" id="A5.Ex5.m1.2.2.2.3.cmml" xref="A5.Ex5.m1.2.2.2.3">2</cn><ci id="A5.Ex5.m1.2.2.2.4.cmml" xref="A5.Ex5.m1.2.2.2.4">𝑏</ci><ci id="A5.Ex5.m1.2.2.2.5.cmml" xref="A5.Ex5.m1.2.2.2.5">𝑟</ci><apply id="A5.Ex5.m1.2.2.2.1.1.1.cmml" xref="A5.Ex5.m1.2.2.2.1.1"><plus id="A5.Ex5.m1.2.2.2.1.1.1.1.cmml" xref="A5.Ex5.m1.2.2.2.1.1.1.1"></plus><ci id="A5.Ex5.m1.2.2.2.1.1.1.2.cmml" xref="A5.Ex5.m1.2.2.2.1.1.1.2">𝑠</ci><ci id="A5.Ex5.m1.2.2.2.1.1.1.3.cmml" xref="A5.Ex5.m1.2.2.2.1.1.1.3">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.Ex5.m1.2c">br(s+n)\cdot 2=2br(s+n)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="A5.p13" class="ltx_para">
<p id="A5.p13.1" class="ltx_p">We also introduce a retention threshold, which we set to 0.05. This means that 5% of the KV cache tokens are retained without compression:</p>
</div>
<div id="A5.p14" class="ltx_para">
<table id="A5.Ex6" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A5.Ex6.m1.1" class="ltx_Math" alttext="brh(0.05(s+n))" display="block"><semantics id="A5.Ex6.m1.1a"><mrow id="A5.Ex6.m1.1.1" xref="A5.Ex6.m1.1.1.cmml"><mi id="A5.Ex6.m1.1.1.3" xref="A5.Ex6.m1.1.1.3.cmml">b</mi><mo lspace="0em" rspace="0em" id="A5.Ex6.m1.1.1.2" xref="A5.Ex6.m1.1.1.2.cmml">​</mo><mi id="A5.Ex6.m1.1.1.4" xref="A5.Ex6.m1.1.1.4.cmml">r</mi><mo lspace="0em" rspace="0em" id="A5.Ex6.m1.1.1.2a" xref="A5.Ex6.m1.1.1.2.cmml">​</mo><mi id="A5.Ex6.m1.1.1.5" xref="A5.Ex6.m1.1.1.5.cmml">h</mi><mo lspace="0em" rspace="0em" id="A5.Ex6.m1.1.1.2b" xref="A5.Ex6.m1.1.1.2.cmml">​</mo><mrow id="A5.Ex6.m1.1.1.1.1" xref="A5.Ex6.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="A5.Ex6.m1.1.1.1.1.2" xref="A5.Ex6.m1.1.1.1.1.1.cmml">(</mo><mrow id="A5.Ex6.m1.1.1.1.1.1" xref="A5.Ex6.m1.1.1.1.1.1.cmml"><mn id="A5.Ex6.m1.1.1.1.1.1.3" xref="A5.Ex6.m1.1.1.1.1.1.3.cmml">0.05</mn><mo lspace="0em" rspace="0em" id="A5.Ex6.m1.1.1.1.1.1.2" xref="A5.Ex6.m1.1.1.1.1.1.2.cmml">​</mo><mrow id="A5.Ex6.m1.1.1.1.1.1.1.1" xref="A5.Ex6.m1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A5.Ex6.m1.1.1.1.1.1.1.1.2" xref="A5.Ex6.m1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A5.Ex6.m1.1.1.1.1.1.1.1.1" xref="A5.Ex6.m1.1.1.1.1.1.1.1.1.cmml"><mi id="A5.Ex6.m1.1.1.1.1.1.1.1.1.2" xref="A5.Ex6.m1.1.1.1.1.1.1.1.1.2.cmml">s</mi><mo id="A5.Ex6.m1.1.1.1.1.1.1.1.1.1" xref="A5.Ex6.m1.1.1.1.1.1.1.1.1.1.cmml">+</mo><mi id="A5.Ex6.m1.1.1.1.1.1.1.1.1.3" xref="A5.Ex6.m1.1.1.1.1.1.1.1.1.3.cmml">n</mi></mrow><mo stretchy="false" id="A5.Ex6.m1.1.1.1.1.1.1.1.3" xref="A5.Ex6.m1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="A5.Ex6.m1.1.1.1.1.3" xref="A5.Ex6.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.Ex6.m1.1b"><apply id="A5.Ex6.m1.1.1.cmml" xref="A5.Ex6.m1.1.1"><times id="A5.Ex6.m1.1.1.2.cmml" xref="A5.Ex6.m1.1.1.2"></times><ci id="A5.Ex6.m1.1.1.3.cmml" xref="A5.Ex6.m1.1.1.3">𝑏</ci><ci id="A5.Ex6.m1.1.1.4.cmml" xref="A5.Ex6.m1.1.1.4">𝑟</ci><ci id="A5.Ex6.m1.1.1.5.cmml" xref="A5.Ex6.m1.1.1.5">ℎ</ci><apply id="A5.Ex6.m1.1.1.1.1.1.cmml" xref="A5.Ex6.m1.1.1.1.1"><times id="A5.Ex6.m1.1.1.1.1.1.2.cmml" xref="A5.Ex6.m1.1.1.1.1.1.2"></times><cn type="float" id="A5.Ex6.m1.1.1.1.1.1.3.cmml" xref="A5.Ex6.m1.1.1.1.1.1.3">0.05</cn><apply id="A5.Ex6.m1.1.1.1.1.1.1.1.1.cmml" xref="A5.Ex6.m1.1.1.1.1.1.1.1"><plus id="A5.Ex6.m1.1.1.1.1.1.1.1.1.1.cmml" xref="A5.Ex6.m1.1.1.1.1.1.1.1.1.1"></plus><ci id="A5.Ex6.m1.1.1.1.1.1.1.1.1.2.cmml" xref="A5.Ex6.m1.1.1.1.1.1.1.1.1.2">𝑠</ci><ci id="A5.Ex6.m1.1.1.1.1.1.1.1.1.3.cmml" xref="A5.Ex6.m1.1.1.1.1.1.1.1.1.3">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.Ex6.m1.1c">brh(0.05(s+n))</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="A5.p15" class="ltx_para">
<p id="A5.p15.1" class="ltx_p">Combining these terms, the total additional memory for the restoration process is:</p>
</div>
<div id="A5.p16" class="ltx_para">
<table id="A5.Ex7" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A5.Ex7.m1.2" class="ltx_Math" alttext="2br(s+n)+0.1brh(s+n)" display="block"><semantics id="A5.Ex7.m1.2a"><mrow id="A5.Ex7.m1.2.2" xref="A5.Ex7.m1.2.2.cmml"><mrow id="A5.Ex7.m1.1.1.1" xref="A5.Ex7.m1.1.1.1.cmml"><mn id="A5.Ex7.m1.1.1.1.3" xref="A5.Ex7.m1.1.1.1.3.cmml">2</mn><mo lspace="0em" rspace="0em" id="A5.Ex7.m1.1.1.1.2" xref="A5.Ex7.m1.1.1.1.2.cmml">​</mo><mi id="A5.Ex7.m1.1.1.1.4" xref="A5.Ex7.m1.1.1.1.4.cmml">b</mi><mo lspace="0em" rspace="0em" id="A5.Ex7.m1.1.1.1.2a" xref="A5.Ex7.m1.1.1.1.2.cmml">​</mo><mi id="A5.Ex7.m1.1.1.1.5" xref="A5.Ex7.m1.1.1.1.5.cmml">r</mi><mo lspace="0em" rspace="0em" id="A5.Ex7.m1.1.1.1.2b" xref="A5.Ex7.m1.1.1.1.2.cmml">​</mo><mrow id="A5.Ex7.m1.1.1.1.1.1" xref="A5.Ex7.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A5.Ex7.m1.1.1.1.1.1.2" xref="A5.Ex7.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="A5.Ex7.m1.1.1.1.1.1.1" xref="A5.Ex7.m1.1.1.1.1.1.1.cmml"><mi id="A5.Ex7.m1.1.1.1.1.1.1.2" xref="A5.Ex7.m1.1.1.1.1.1.1.2.cmml">s</mi><mo id="A5.Ex7.m1.1.1.1.1.1.1.1" xref="A5.Ex7.m1.1.1.1.1.1.1.1.cmml">+</mo><mi id="A5.Ex7.m1.1.1.1.1.1.1.3" xref="A5.Ex7.m1.1.1.1.1.1.1.3.cmml">n</mi></mrow><mo stretchy="false" id="A5.Ex7.m1.1.1.1.1.1.3" xref="A5.Ex7.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="A5.Ex7.m1.2.2.3" xref="A5.Ex7.m1.2.2.3.cmml">+</mo><mrow id="A5.Ex7.m1.2.2.2" xref="A5.Ex7.m1.2.2.2.cmml"><mn id="A5.Ex7.m1.2.2.2.3" xref="A5.Ex7.m1.2.2.2.3.cmml">0.1</mn><mo lspace="0em" rspace="0em" id="A5.Ex7.m1.2.2.2.2" xref="A5.Ex7.m1.2.2.2.2.cmml">​</mo><mi id="A5.Ex7.m1.2.2.2.4" xref="A5.Ex7.m1.2.2.2.4.cmml">b</mi><mo lspace="0em" rspace="0em" id="A5.Ex7.m1.2.2.2.2a" xref="A5.Ex7.m1.2.2.2.2.cmml">​</mo><mi id="A5.Ex7.m1.2.2.2.5" xref="A5.Ex7.m1.2.2.2.5.cmml">r</mi><mo lspace="0em" rspace="0em" id="A5.Ex7.m1.2.2.2.2b" xref="A5.Ex7.m1.2.2.2.2.cmml">​</mo><mi id="A5.Ex7.m1.2.2.2.6" xref="A5.Ex7.m1.2.2.2.6.cmml">h</mi><mo lspace="0em" rspace="0em" id="A5.Ex7.m1.2.2.2.2c" xref="A5.Ex7.m1.2.2.2.2.cmml">​</mo><mrow id="A5.Ex7.m1.2.2.2.1.1" xref="A5.Ex7.m1.2.2.2.1.1.1.cmml"><mo stretchy="false" id="A5.Ex7.m1.2.2.2.1.1.2" xref="A5.Ex7.m1.2.2.2.1.1.1.cmml">(</mo><mrow id="A5.Ex7.m1.2.2.2.1.1.1" xref="A5.Ex7.m1.2.2.2.1.1.1.cmml"><mi id="A5.Ex7.m1.2.2.2.1.1.1.2" xref="A5.Ex7.m1.2.2.2.1.1.1.2.cmml">s</mi><mo id="A5.Ex7.m1.2.2.2.1.1.1.1" xref="A5.Ex7.m1.2.2.2.1.1.1.1.cmml">+</mo><mi id="A5.Ex7.m1.2.2.2.1.1.1.3" xref="A5.Ex7.m1.2.2.2.1.1.1.3.cmml">n</mi></mrow><mo stretchy="false" id="A5.Ex7.m1.2.2.2.1.1.3" xref="A5.Ex7.m1.2.2.2.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.Ex7.m1.2b"><apply id="A5.Ex7.m1.2.2.cmml" xref="A5.Ex7.m1.2.2"><plus id="A5.Ex7.m1.2.2.3.cmml" xref="A5.Ex7.m1.2.2.3"></plus><apply id="A5.Ex7.m1.1.1.1.cmml" xref="A5.Ex7.m1.1.1.1"><times id="A5.Ex7.m1.1.1.1.2.cmml" xref="A5.Ex7.m1.1.1.1.2"></times><cn type="integer" id="A5.Ex7.m1.1.1.1.3.cmml" xref="A5.Ex7.m1.1.1.1.3">2</cn><ci id="A5.Ex7.m1.1.1.1.4.cmml" xref="A5.Ex7.m1.1.1.1.4">𝑏</ci><ci id="A5.Ex7.m1.1.1.1.5.cmml" xref="A5.Ex7.m1.1.1.1.5">𝑟</ci><apply id="A5.Ex7.m1.1.1.1.1.1.1.cmml" xref="A5.Ex7.m1.1.1.1.1.1"><plus id="A5.Ex7.m1.1.1.1.1.1.1.1.cmml" xref="A5.Ex7.m1.1.1.1.1.1.1.1"></plus><ci id="A5.Ex7.m1.1.1.1.1.1.1.2.cmml" xref="A5.Ex7.m1.1.1.1.1.1.1.2">𝑠</ci><ci id="A5.Ex7.m1.1.1.1.1.1.1.3.cmml" xref="A5.Ex7.m1.1.1.1.1.1.1.3">𝑛</ci></apply></apply><apply id="A5.Ex7.m1.2.2.2.cmml" xref="A5.Ex7.m1.2.2.2"><times id="A5.Ex7.m1.2.2.2.2.cmml" xref="A5.Ex7.m1.2.2.2.2"></times><cn type="float" id="A5.Ex7.m1.2.2.2.3.cmml" xref="A5.Ex7.m1.2.2.2.3">0.1</cn><ci id="A5.Ex7.m1.2.2.2.4.cmml" xref="A5.Ex7.m1.2.2.2.4">𝑏</ci><ci id="A5.Ex7.m1.2.2.2.5.cmml" xref="A5.Ex7.m1.2.2.2.5">𝑟</ci><ci id="A5.Ex7.m1.2.2.2.6.cmml" xref="A5.Ex7.m1.2.2.2.6">ℎ</ci><apply id="A5.Ex7.m1.2.2.2.1.1.1.cmml" xref="A5.Ex7.m1.2.2.2.1.1"><plus id="A5.Ex7.m1.2.2.2.1.1.1.1.cmml" xref="A5.Ex7.m1.2.2.2.1.1.1.1"></plus><ci id="A5.Ex7.m1.2.2.2.1.1.1.2.cmml" xref="A5.Ex7.m1.2.2.2.1.1.1.2">𝑠</ci><ci id="A5.Ex7.m1.2.2.2.1.1.1.3.cmml" xref="A5.Ex7.m1.2.2.2.1.1.1.3">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.Ex7.m1.2c">2br(s+n)+0.1brh(s+n)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="A5.p17" class="ltx_para">
<p id="A5.p17.1" class="ltx_p">Finally, summing the compressed memory usage and the restoration memory cost, the overall memory requirement is:</p>
</div>
<div id="A5.p18" class="ltx_para">
<table id="A5.Ex8" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A5.Ex8.m1.3" class="ltx_Math" alttext="3brh(s+n)+2br(s+n)+0.1brh(s+n)" display="block"><semantics id="A5.Ex8.m1.3a"><mrow id="A5.Ex8.m1.3.3" xref="A5.Ex8.m1.3.3.cmml"><mrow id="A5.Ex8.m1.1.1.1" xref="A5.Ex8.m1.1.1.1.cmml"><mn id="A5.Ex8.m1.1.1.1.3" xref="A5.Ex8.m1.1.1.1.3.cmml">3</mn><mo lspace="0em" rspace="0em" id="A5.Ex8.m1.1.1.1.2" xref="A5.Ex8.m1.1.1.1.2.cmml">​</mo><mi id="A5.Ex8.m1.1.1.1.4" xref="A5.Ex8.m1.1.1.1.4.cmml">b</mi><mo lspace="0em" rspace="0em" id="A5.Ex8.m1.1.1.1.2a" xref="A5.Ex8.m1.1.1.1.2.cmml">​</mo><mi id="A5.Ex8.m1.1.1.1.5" xref="A5.Ex8.m1.1.1.1.5.cmml">r</mi><mo lspace="0em" rspace="0em" id="A5.Ex8.m1.1.1.1.2b" xref="A5.Ex8.m1.1.1.1.2.cmml">​</mo><mi id="A5.Ex8.m1.1.1.1.6" xref="A5.Ex8.m1.1.1.1.6.cmml">h</mi><mo lspace="0em" rspace="0em" id="A5.Ex8.m1.1.1.1.2c" xref="A5.Ex8.m1.1.1.1.2.cmml">​</mo><mrow id="A5.Ex8.m1.1.1.1.1.1" xref="A5.Ex8.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A5.Ex8.m1.1.1.1.1.1.2" xref="A5.Ex8.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="A5.Ex8.m1.1.1.1.1.1.1" xref="A5.Ex8.m1.1.1.1.1.1.1.cmml"><mi id="A5.Ex8.m1.1.1.1.1.1.1.2" xref="A5.Ex8.m1.1.1.1.1.1.1.2.cmml">s</mi><mo id="A5.Ex8.m1.1.1.1.1.1.1.1" xref="A5.Ex8.m1.1.1.1.1.1.1.1.cmml">+</mo><mi id="A5.Ex8.m1.1.1.1.1.1.1.3" xref="A5.Ex8.m1.1.1.1.1.1.1.3.cmml">n</mi></mrow><mo stretchy="false" id="A5.Ex8.m1.1.1.1.1.1.3" xref="A5.Ex8.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="A5.Ex8.m1.3.3.4" xref="A5.Ex8.m1.3.3.4.cmml">+</mo><mrow id="A5.Ex8.m1.2.2.2" xref="A5.Ex8.m1.2.2.2.cmml"><mn id="A5.Ex8.m1.2.2.2.3" xref="A5.Ex8.m1.2.2.2.3.cmml">2</mn><mo lspace="0em" rspace="0em" id="A5.Ex8.m1.2.2.2.2" xref="A5.Ex8.m1.2.2.2.2.cmml">​</mo><mi id="A5.Ex8.m1.2.2.2.4" xref="A5.Ex8.m1.2.2.2.4.cmml">b</mi><mo lspace="0em" rspace="0em" id="A5.Ex8.m1.2.2.2.2a" xref="A5.Ex8.m1.2.2.2.2.cmml">​</mo><mi id="A5.Ex8.m1.2.2.2.5" xref="A5.Ex8.m1.2.2.2.5.cmml">r</mi><mo lspace="0em" rspace="0em" id="A5.Ex8.m1.2.2.2.2b" xref="A5.Ex8.m1.2.2.2.2.cmml">​</mo><mrow id="A5.Ex8.m1.2.2.2.1.1" xref="A5.Ex8.m1.2.2.2.1.1.1.cmml"><mo stretchy="false" id="A5.Ex8.m1.2.2.2.1.1.2" xref="A5.Ex8.m1.2.2.2.1.1.1.cmml">(</mo><mrow id="A5.Ex8.m1.2.2.2.1.1.1" xref="A5.Ex8.m1.2.2.2.1.1.1.cmml"><mi id="A5.Ex8.m1.2.2.2.1.1.1.2" xref="A5.Ex8.m1.2.2.2.1.1.1.2.cmml">s</mi><mo id="A5.Ex8.m1.2.2.2.1.1.1.1" xref="A5.Ex8.m1.2.2.2.1.1.1.1.cmml">+</mo><mi id="A5.Ex8.m1.2.2.2.1.1.1.3" xref="A5.Ex8.m1.2.2.2.1.1.1.3.cmml">n</mi></mrow><mo stretchy="false" id="A5.Ex8.m1.2.2.2.1.1.3" xref="A5.Ex8.m1.2.2.2.1.1.1.cmml">)</mo></mrow></mrow><mo id="A5.Ex8.m1.3.3.4a" xref="A5.Ex8.m1.3.3.4.cmml">+</mo><mrow id="A5.Ex8.m1.3.3.3" xref="A5.Ex8.m1.3.3.3.cmml"><mn id="A5.Ex8.m1.3.3.3.3" xref="A5.Ex8.m1.3.3.3.3.cmml">0.1</mn><mo lspace="0em" rspace="0em" id="A5.Ex8.m1.3.3.3.2" xref="A5.Ex8.m1.3.3.3.2.cmml">​</mo><mi id="A5.Ex8.m1.3.3.3.4" xref="A5.Ex8.m1.3.3.3.4.cmml">b</mi><mo lspace="0em" rspace="0em" id="A5.Ex8.m1.3.3.3.2a" xref="A5.Ex8.m1.3.3.3.2.cmml">​</mo><mi id="A5.Ex8.m1.3.3.3.5" xref="A5.Ex8.m1.3.3.3.5.cmml">r</mi><mo lspace="0em" rspace="0em" id="A5.Ex8.m1.3.3.3.2b" xref="A5.Ex8.m1.3.3.3.2.cmml">​</mo><mi id="A5.Ex8.m1.3.3.3.6" xref="A5.Ex8.m1.3.3.3.6.cmml">h</mi><mo lspace="0em" rspace="0em" id="A5.Ex8.m1.3.3.3.2c" xref="A5.Ex8.m1.3.3.3.2.cmml">​</mo><mrow id="A5.Ex8.m1.3.3.3.1.1" xref="A5.Ex8.m1.3.3.3.1.1.1.cmml"><mo stretchy="false" id="A5.Ex8.m1.3.3.3.1.1.2" xref="A5.Ex8.m1.3.3.3.1.1.1.cmml">(</mo><mrow id="A5.Ex8.m1.3.3.3.1.1.1" xref="A5.Ex8.m1.3.3.3.1.1.1.cmml"><mi id="A5.Ex8.m1.3.3.3.1.1.1.2" xref="A5.Ex8.m1.3.3.3.1.1.1.2.cmml">s</mi><mo id="A5.Ex8.m1.3.3.3.1.1.1.1" xref="A5.Ex8.m1.3.3.3.1.1.1.1.cmml">+</mo><mi id="A5.Ex8.m1.3.3.3.1.1.1.3" xref="A5.Ex8.m1.3.3.3.1.1.1.3.cmml">n</mi></mrow><mo stretchy="false" id="A5.Ex8.m1.3.3.3.1.1.3" xref="A5.Ex8.m1.3.3.3.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.Ex8.m1.3b"><apply id="A5.Ex8.m1.3.3.cmml" xref="A5.Ex8.m1.3.3"><plus id="A5.Ex8.m1.3.3.4.cmml" xref="A5.Ex8.m1.3.3.4"></plus><apply id="A5.Ex8.m1.1.1.1.cmml" xref="A5.Ex8.m1.1.1.1"><times id="A5.Ex8.m1.1.1.1.2.cmml" xref="A5.Ex8.m1.1.1.1.2"></times><cn type="integer" id="A5.Ex8.m1.1.1.1.3.cmml" xref="A5.Ex8.m1.1.1.1.3">3</cn><ci id="A5.Ex8.m1.1.1.1.4.cmml" xref="A5.Ex8.m1.1.1.1.4">𝑏</ci><ci id="A5.Ex8.m1.1.1.1.5.cmml" xref="A5.Ex8.m1.1.1.1.5">𝑟</ci><ci id="A5.Ex8.m1.1.1.1.6.cmml" xref="A5.Ex8.m1.1.1.1.6">ℎ</ci><apply id="A5.Ex8.m1.1.1.1.1.1.1.cmml" xref="A5.Ex8.m1.1.1.1.1.1"><plus id="A5.Ex8.m1.1.1.1.1.1.1.1.cmml" xref="A5.Ex8.m1.1.1.1.1.1.1.1"></plus><ci id="A5.Ex8.m1.1.1.1.1.1.1.2.cmml" xref="A5.Ex8.m1.1.1.1.1.1.1.2">𝑠</ci><ci id="A5.Ex8.m1.1.1.1.1.1.1.3.cmml" xref="A5.Ex8.m1.1.1.1.1.1.1.3">𝑛</ci></apply></apply><apply id="A5.Ex8.m1.2.2.2.cmml" xref="A5.Ex8.m1.2.2.2"><times id="A5.Ex8.m1.2.2.2.2.cmml" xref="A5.Ex8.m1.2.2.2.2"></times><cn type="integer" id="A5.Ex8.m1.2.2.2.3.cmml" xref="A5.Ex8.m1.2.2.2.3">2</cn><ci id="A5.Ex8.m1.2.2.2.4.cmml" xref="A5.Ex8.m1.2.2.2.4">𝑏</ci><ci id="A5.Ex8.m1.2.2.2.5.cmml" xref="A5.Ex8.m1.2.2.2.5">𝑟</ci><apply id="A5.Ex8.m1.2.2.2.1.1.1.cmml" xref="A5.Ex8.m1.2.2.2.1.1"><plus id="A5.Ex8.m1.2.2.2.1.1.1.1.cmml" xref="A5.Ex8.m1.2.2.2.1.1.1.1"></plus><ci id="A5.Ex8.m1.2.2.2.1.1.1.2.cmml" xref="A5.Ex8.m1.2.2.2.1.1.1.2">𝑠</ci><ci id="A5.Ex8.m1.2.2.2.1.1.1.3.cmml" xref="A5.Ex8.m1.2.2.2.1.1.1.3">𝑛</ci></apply></apply><apply id="A5.Ex8.m1.3.3.3.cmml" xref="A5.Ex8.m1.3.3.3"><times id="A5.Ex8.m1.3.3.3.2.cmml" xref="A5.Ex8.m1.3.3.3.2"></times><cn type="float" id="A5.Ex8.m1.3.3.3.3.cmml" xref="A5.Ex8.m1.3.3.3.3">0.1</cn><ci id="A5.Ex8.m1.3.3.3.4.cmml" xref="A5.Ex8.m1.3.3.3.4">𝑏</ci><ci id="A5.Ex8.m1.3.3.3.5.cmml" xref="A5.Ex8.m1.3.3.3.5">𝑟</ci><ci id="A5.Ex8.m1.3.3.3.6.cmml" xref="A5.Ex8.m1.3.3.3.6">ℎ</ci><apply id="A5.Ex8.m1.3.3.3.1.1.1.cmml" xref="A5.Ex8.m1.3.3.3.1.1"><plus id="A5.Ex8.m1.3.3.3.1.1.1.1.cmml" xref="A5.Ex8.m1.3.3.3.1.1.1.1"></plus><ci id="A5.Ex8.m1.3.3.3.1.1.1.2.cmml" xref="A5.Ex8.m1.3.3.3.1.1.1.2">𝑠</ci><ci id="A5.Ex8.m1.3.3.3.1.1.1.3.cmml" xref="A5.Ex8.m1.3.3.3.1.1.1.3">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.Ex8.m1.3c">3brh(s+n)+2br(s+n)+0.1brh(s+n)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="A5.p19" class="ltx_para">
<p id="A5.p19.1" class="ltx_p">This can be simplified by grouping the common factors:</p>
</div>
<div id="A5.p20" class="ltx_para">
<table id="A5.Ex9" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A5.Ex9.m1.2" class="ltx_Math" alttext="br(s+n)\left(3h+2+0.1h\right)" display="block"><semantics id="A5.Ex9.m1.2a"><mrow id="A5.Ex9.m1.2.2" xref="A5.Ex9.m1.2.2.cmml"><mi id="A5.Ex9.m1.2.2.4" xref="A5.Ex9.m1.2.2.4.cmml">b</mi><mo lspace="0em" rspace="0em" id="A5.Ex9.m1.2.2.3" xref="A5.Ex9.m1.2.2.3.cmml">​</mo><mi id="A5.Ex9.m1.2.2.5" xref="A5.Ex9.m1.2.2.5.cmml">r</mi><mo lspace="0em" rspace="0em" id="A5.Ex9.m1.2.2.3a" xref="A5.Ex9.m1.2.2.3.cmml">​</mo><mrow id="A5.Ex9.m1.1.1.1.1" xref="A5.Ex9.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="A5.Ex9.m1.1.1.1.1.2" xref="A5.Ex9.m1.1.1.1.1.1.cmml">(</mo><mrow id="A5.Ex9.m1.1.1.1.1.1" xref="A5.Ex9.m1.1.1.1.1.1.cmml"><mi id="A5.Ex9.m1.1.1.1.1.1.2" xref="A5.Ex9.m1.1.1.1.1.1.2.cmml">s</mi><mo id="A5.Ex9.m1.1.1.1.1.1.1" xref="A5.Ex9.m1.1.1.1.1.1.1.cmml">+</mo><mi id="A5.Ex9.m1.1.1.1.1.1.3" xref="A5.Ex9.m1.1.1.1.1.1.3.cmml">n</mi></mrow><mo stretchy="false" id="A5.Ex9.m1.1.1.1.1.3" xref="A5.Ex9.m1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A5.Ex9.m1.2.2.3b" xref="A5.Ex9.m1.2.2.3.cmml">​</mo><mrow id="A5.Ex9.m1.2.2.2.1" xref="A5.Ex9.m1.2.2.2.1.1.cmml"><mo id="A5.Ex9.m1.2.2.2.1.2" xref="A5.Ex9.m1.2.2.2.1.1.cmml">(</mo><mrow id="A5.Ex9.m1.2.2.2.1.1" xref="A5.Ex9.m1.2.2.2.1.1.cmml"><mrow id="A5.Ex9.m1.2.2.2.1.1.2" xref="A5.Ex9.m1.2.2.2.1.1.2.cmml"><mn id="A5.Ex9.m1.2.2.2.1.1.2.2" xref="A5.Ex9.m1.2.2.2.1.1.2.2.cmml">3</mn><mo lspace="0em" rspace="0em" id="A5.Ex9.m1.2.2.2.1.1.2.1" xref="A5.Ex9.m1.2.2.2.1.1.2.1.cmml">​</mo><mi id="A5.Ex9.m1.2.2.2.1.1.2.3" xref="A5.Ex9.m1.2.2.2.1.1.2.3.cmml">h</mi></mrow><mo id="A5.Ex9.m1.2.2.2.1.1.1" xref="A5.Ex9.m1.2.2.2.1.1.1.cmml">+</mo><mn id="A5.Ex9.m1.2.2.2.1.1.3" xref="A5.Ex9.m1.2.2.2.1.1.3.cmml">2</mn><mo id="A5.Ex9.m1.2.2.2.1.1.1a" xref="A5.Ex9.m1.2.2.2.1.1.1.cmml">+</mo><mrow id="A5.Ex9.m1.2.2.2.1.1.4" xref="A5.Ex9.m1.2.2.2.1.1.4.cmml"><mn id="A5.Ex9.m1.2.2.2.1.1.4.2" xref="A5.Ex9.m1.2.2.2.1.1.4.2.cmml">0.1</mn><mo lspace="0em" rspace="0em" id="A5.Ex9.m1.2.2.2.1.1.4.1" xref="A5.Ex9.m1.2.2.2.1.1.4.1.cmml">​</mo><mi id="A5.Ex9.m1.2.2.2.1.1.4.3" xref="A5.Ex9.m1.2.2.2.1.1.4.3.cmml">h</mi></mrow></mrow><mo id="A5.Ex9.m1.2.2.2.1.3" xref="A5.Ex9.m1.2.2.2.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.Ex9.m1.2b"><apply id="A5.Ex9.m1.2.2.cmml" xref="A5.Ex9.m1.2.2"><times id="A5.Ex9.m1.2.2.3.cmml" xref="A5.Ex9.m1.2.2.3"></times><ci id="A5.Ex9.m1.2.2.4.cmml" xref="A5.Ex9.m1.2.2.4">𝑏</ci><ci id="A5.Ex9.m1.2.2.5.cmml" xref="A5.Ex9.m1.2.2.5">𝑟</ci><apply id="A5.Ex9.m1.1.1.1.1.1.cmml" xref="A5.Ex9.m1.1.1.1.1"><plus id="A5.Ex9.m1.1.1.1.1.1.1.cmml" xref="A5.Ex9.m1.1.1.1.1.1.1"></plus><ci id="A5.Ex9.m1.1.1.1.1.1.2.cmml" xref="A5.Ex9.m1.1.1.1.1.1.2">𝑠</ci><ci id="A5.Ex9.m1.1.1.1.1.1.3.cmml" xref="A5.Ex9.m1.1.1.1.1.1.3">𝑛</ci></apply><apply id="A5.Ex9.m1.2.2.2.1.1.cmml" xref="A5.Ex9.m1.2.2.2.1"><plus id="A5.Ex9.m1.2.2.2.1.1.1.cmml" xref="A5.Ex9.m1.2.2.2.1.1.1"></plus><apply id="A5.Ex9.m1.2.2.2.1.1.2.cmml" xref="A5.Ex9.m1.2.2.2.1.1.2"><times id="A5.Ex9.m1.2.2.2.1.1.2.1.cmml" xref="A5.Ex9.m1.2.2.2.1.1.2.1"></times><cn type="integer" id="A5.Ex9.m1.2.2.2.1.1.2.2.cmml" xref="A5.Ex9.m1.2.2.2.1.1.2.2">3</cn><ci id="A5.Ex9.m1.2.2.2.1.1.2.3.cmml" xref="A5.Ex9.m1.2.2.2.1.1.2.3">ℎ</ci></apply><cn type="integer" id="A5.Ex9.m1.2.2.2.1.1.3.cmml" xref="A5.Ex9.m1.2.2.2.1.1.3">2</cn><apply id="A5.Ex9.m1.2.2.2.1.1.4.cmml" xref="A5.Ex9.m1.2.2.2.1.1.4"><times id="A5.Ex9.m1.2.2.2.1.1.4.1.cmml" xref="A5.Ex9.m1.2.2.2.1.1.4.1"></times><cn type="float" id="A5.Ex9.m1.2.2.2.1.1.4.2.cmml" xref="A5.Ex9.m1.2.2.2.1.1.4.2">0.1</cn><ci id="A5.Ex9.m1.2.2.2.1.1.4.3.cmml" xref="A5.Ex9.m1.2.2.2.1.1.4.3">ℎ</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.Ex9.m1.2c">br(s+n)\left(3h+2+0.1h\right)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="A5.p21" class="ltx_para">
<p id="A5.p21.1" class="ltx_p">Simplifying the expression inside the parentheses, we get:</p>
</div>
<div id="A5.p22" class="ltx_para">
<table id="A5.Ex10" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A5.Ex10.m1.2" class="ltx_Math" alttext="br(s+n)\left(3.1h+2\right)" display="block"><semantics id="A5.Ex10.m1.2a"><mrow id="A5.Ex10.m1.2.2" xref="A5.Ex10.m1.2.2.cmml"><mi id="A5.Ex10.m1.2.2.4" xref="A5.Ex10.m1.2.2.4.cmml">b</mi><mo lspace="0em" rspace="0em" id="A5.Ex10.m1.2.2.3" xref="A5.Ex10.m1.2.2.3.cmml">​</mo><mi id="A5.Ex10.m1.2.2.5" xref="A5.Ex10.m1.2.2.5.cmml">r</mi><mo lspace="0em" rspace="0em" id="A5.Ex10.m1.2.2.3a" xref="A5.Ex10.m1.2.2.3.cmml">​</mo><mrow id="A5.Ex10.m1.1.1.1.1" xref="A5.Ex10.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="A5.Ex10.m1.1.1.1.1.2" xref="A5.Ex10.m1.1.1.1.1.1.cmml">(</mo><mrow id="A5.Ex10.m1.1.1.1.1.1" xref="A5.Ex10.m1.1.1.1.1.1.cmml"><mi id="A5.Ex10.m1.1.1.1.1.1.2" xref="A5.Ex10.m1.1.1.1.1.1.2.cmml">s</mi><mo id="A5.Ex10.m1.1.1.1.1.1.1" xref="A5.Ex10.m1.1.1.1.1.1.1.cmml">+</mo><mi id="A5.Ex10.m1.1.1.1.1.1.3" xref="A5.Ex10.m1.1.1.1.1.1.3.cmml">n</mi></mrow><mo stretchy="false" id="A5.Ex10.m1.1.1.1.1.3" xref="A5.Ex10.m1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A5.Ex10.m1.2.2.3b" xref="A5.Ex10.m1.2.2.3.cmml">​</mo><mrow id="A5.Ex10.m1.2.2.2.1" xref="A5.Ex10.m1.2.2.2.1.1.cmml"><mo id="A5.Ex10.m1.2.2.2.1.2" xref="A5.Ex10.m1.2.2.2.1.1.cmml">(</mo><mrow id="A5.Ex10.m1.2.2.2.1.1" xref="A5.Ex10.m1.2.2.2.1.1.cmml"><mrow id="A5.Ex10.m1.2.2.2.1.1.2" xref="A5.Ex10.m1.2.2.2.1.1.2.cmml"><mn id="A5.Ex10.m1.2.2.2.1.1.2.2" xref="A5.Ex10.m1.2.2.2.1.1.2.2.cmml">3.1</mn><mo lspace="0em" rspace="0em" id="A5.Ex10.m1.2.2.2.1.1.2.1" xref="A5.Ex10.m1.2.2.2.1.1.2.1.cmml">​</mo><mi id="A5.Ex10.m1.2.2.2.1.1.2.3" xref="A5.Ex10.m1.2.2.2.1.1.2.3.cmml">h</mi></mrow><mo id="A5.Ex10.m1.2.2.2.1.1.1" xref="A5.Ex10.m1.2.2.2.1.1.1.cmml">+</mo><mn id="A5.Ex10.m1.2.2.2.1.1.3" xref="A5.Ex10.m1.2.2.2.1.1.3.cmml">2</mn></mrow><mo id="A5.Ex10.m1.2.2.2.1.3" xref="A5.Ex10.m1.2.2.2.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.Ex10.m1.2b"><apply id="A5.Ex10.m1.2.2.cmml" xref="A5.Ex10.m1.2.2"><times id="A5.Ex10.m1.2.2.3.cmml" xref="A5.Ex10.m1.2.2.3"></times><ci id="A5.Ex10.m1.2.2.4.cmml" xref="A5.Ex10.m1.2.2.4">𝑏</ci><ci id="A5.Ex10.m1.2.2.5.cmml" xref="A5.Ex10.m1.2.2.5">𝑟</ci><apply id="A5.Ex10.m1.1.1.1.1.1.cmml" xref="A5.Ex10.m1.1.1.1.1"><plus id="A5.Ex10.m1.1.1.1.1.1.1.cmml" xref="A5.Ex10.m1.1.1.1.1.1.1"></plus><ci id="A5.Ex10.m1.1.1.1.1.1.2.cmml" xref="A5.Ex10.m1.1.1.1.1.1.2">𝑠</ci><ci id="A5.Ex10.m1.1.1.1.1.1.3.cmml" xref="A5.Ex10.m1.1.1.1.1.1.3">𝑛</ci></apply><apply id="A5.Ex10.m1.2.2.2.1.1.cmml" xref="A5.Ex10.m1.2.2.2.1"><plus id="A5.Ex10.m1.2.2.2.1.1.1.cmml" xref="A5.Ex10.m1.2.2.2.1.1.1"></plus><apply id="A5.Ex10.m1.2.2.2.1.1.2.cmml" xref="A5.Ex10.m1.2.2.2.1.1.2"><times id="A5.Ex10.m1.2.2.2.1.1.2.1.cmml" xref="A5.Ex10.m1.2.2.2.1.1.2.1"></times><cn type="float" id="A5.Ex10.m1.2.2.2.1.1.2.2.cmml" xref="A5.Ex10.m1.2.2.2.1.1.2.2">3.1</cn><ci id="A5.Ex10.m1.2.2.2.1.1.2.3.cmml" xref="A5.Ex10.m1.2.2.2.1.1.2.3">ℎ</ci></apply><cn type="integer" id="A5.Ex10.m1.2.2.2.1.1.3.cmml" xref="A5.Ex10.m1.2.2.2.1.1.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.Ex10.m1.2c">br(s+n)\left(3.1h+2\right)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="A5.p23" class="ltx_para">
<p id="A5.p23.1" class="ltx_p">Therefore, the total memory cost for the KV cache in the MiniCache Framework is:</p>
</div>
<div id="A5.p24" class="ltx_para">
<table id="A5.Ex11" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A5.Ex11.m1.2" class="ltx_Math" alttext="br(s+n)(3.1h+2)" display="block"><semantics id="A5.Ex11.m1.2a"><mrow id="A5.Ex11.m1.2.2" xref="A5.Ex11.m1.2.2.cmml"><mi id="A5.Ex11.m1.2.2.4" xref="A5.Ex11.m1.2.2.4.cmml">b</mi><mo lspace="0em" rspace="0em" id="A5.Ex11.m1.2.2.3" xref="A5.Ex11.m1.2.2.3.cmml">​</mo><mi id="A5.Ex11.m1.2.2.5" xref="A5.Ex11.m1.2.2.5.cmml">r</mi><mo lspace="0em" rspace="0em" id="A5.Ex11.m1.2.2.3a" xref="A5.Ex11.m1.2.2.3.cmml">​</mo><mrow id="A5.Ex11.m1.1.1.1.1" xref="A5.Ex11.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="A5.Ex11.m1.1.1.1.1.2" xref="A5.Ex11.m1.1.1.1.1.1.cmml">(</mo><mrow id="A5.Ex11.m1.1.1.1.1.1" xref="A5.Ex11.m1.1.1.1.1.1.cmml"><mi id="A5.Ex11.m1.1.1.1.1.1.2" xref="A5.Ex11.m1.1.1.1.1.1.2.cmml">s</mi><mo id="A5.Ex11.m1.1.1.1.1.1.1" xref="A5.Ex11.m1.1.1.1.1.1.1.cmml">+</mo><mi id="A5.Ex11.m1.1.1.1.1.1.3" xref="A5.Ex11.m1.1.1.1.1.1.3.cmml">n</mi></mrow><mo stretchy="false" id="A5.Ex11.m1.1.1.1.1.3" xref="A5.Ex11.m1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="A5.Ex11.m1.2.2.3b" xref="A5.Ex11.m1.2.2.3.cmml">​</mo><mrow id="A5.Ex11.m1.2.2.2.1" xref="A5.Ex11.m1.2.2.2.1.1.cmml"><mo stretchy="false" id="A5.Ex11.m1.2.2.2.1.2" xref="A5.Ex11.m1.2.2.2.1.1.cmml">(</mo><mrow id="A5.Ex11.m1.2.2.2.1.1" xref="A5.Ex11.m1.2.2.2.1.1.cmml"><mrow id="A5.Ex11.m1.2.2.2.1.1.2" xref="A5.Ex11.m1.2.2.2.1.1.2.cmml"><mn id="A5.Ex11.m1.2.2.2.1.1.2.2" xref="A5.Ex11.m1.2.2.2.1.1.2.2.cmml">3.1</mn><mo lspace="0em" rspace="0em" id="A5.Ex11.m1.2.2.2.1.1.2.1" xref="A5.Ex11.m1.2.2.2.1.1.2.1.cmml">​</mo><mi id="A5.Ex11.m1.2.2.2.1.1.2.3" xref="A5.Ex11.m1.2.2.2.1.1.2.3.cmml">h</mi></mrow><mo id="A5.Ex11.m1.2.2.2.1.1.1" xref="A5.Ex11.m1.2.2.2.1.1.1.cmml">+</mo><mn id="A5.Ex11.m1.2.2.2.1.1.3" xref="A5.Ex11.m1.2.2.2.1.1.3.cmml">2</mn></mrow><mo stretchy="false" id="A5.Ex11.m1.2.2.2.1.3" xref="A5.Ex11.m1.2.2.2.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.Ex11.m1.2b"><apply id="A5.Ex11.m1.2.2.cmml" xref="A5.Ex11.m1.2.2"><times id="A5.Ex11.m1.2.2.3.cmml" xref="A5.Ex11.m1.2.2.3"></times><ci id="A5.Ex11.m1.2.2.4.cmml" xref="A5.Ex11.m1.2.2.4">𝑏</ci><ci id="A5.Ex11.m1.2.2.5.cmml" xref="A5.Ex11.m1.2.2.5">𝑟</ci><apply id="A5.Ex11.m1.1.1.1.1.1.cmml" xref="A5.Ex11.m1.1.1.1.1"><plus id="A5.Ex11.m1.1.1.1.1.1.1.cmml" xref="A5.Ex11.m1.1.1.1.1.1.1"></plus><ci id="A5.Ex11.m1.1.1.1.1.1.2.cmml" xref="A5.Ex11.m1.1.1.1.1.1.2">𝑠</ci><ci id="A5.Ex11.m1.1.1.1.1.1.3.cmml" xref="A5.Ex11.m1.1.1.1.1.1.3">𝑛</ci></apply><apply id="A5.Ex11.m1.2.2.2.1.1.cmml" xref="A5.Ex11.m1.2.2.2.1"><plus id="A5.Ex11.m1.2.2.2.1.1.1.cmml" xref="A5.Ex11.m1.2.2.2.1.1.1"></plus><apply id="A5.Ex11.m1.2.2.2.1.1.2.cmml" xref="A5.Ex11.m1.2.2.2.1.1.2"><times id="A5.Ex11.m1.2.2.2.1.1.2.1.cmml" xref="A5.Ex11.m1.2.2.2.1.1.2.1"></times><cn type="float" id="A5.Ex11.m1.2.2.2.1.1.2.2.cmml" xref="A5.Ex11.m1.2.2.2.1.1.2.2">3.1</cn><ci id="A5.Ex11.m1.2.2.2.1.1.2.3.cmml" xref="A5.Ex11.m1.2.2.2.1.1.2.3">ℎ</ci></apply><cn type="integer" id="A5.Ex11.m1.2.2.2.1.1.3.cmml" xref="A5.Ex11.m1.2.2.2.1.1.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.Ex11.m1.2c">br(s+n)(3.1h+2)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="A5.p25" class="ltx_para">
<p id="A5.p25.1" class="ltx_p">This detailed derivation confirms the efficiency improvements discussed in Section <a href="#S4.SS3" title="4.3 Efficiency Discussion ‣ 4 Method ‣ MiniCache: KV Cache Compression in Depth Dimension for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.3</span></a>, highlighting the significant reduction in memory usage achieved through our layer merging and restoration strategies.</p>
</div>
</section>
<section id="A6" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix F </span>Detailed Experiment Results</h2>

<figure id="A6.T2" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="A6.T2.2.1.1" class="ltx_text" style="font-size:90%;">Table B</span>: </span><span id="A6.T2.3.2" class="ltx_text" style="font-size:90%;">Detailed performance comparison on GSM8K dataset with LLaMA-3-70B</span></figcaption>
<table id="A6.T2.4" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A6.T2.4.1.1" class="ltx_tr">
<th id="A6.T2.4.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">Layer</th>
<th id="A6.T2.4.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">LLaMA-3-70B MiniCache</th>
<th id="A6.T2.4.1.1.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt">LLaMA-3-70B Mean</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A6.T2.4.2.1" class="ltx_tr">
<th id="A6.T2.4.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">0</th>
<td id="A6.T2.4.2.1.2" class="ltx_td ltx_align_center ltx_border_t">0.799</td>
<td id="A6.T2.4.2.1.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">0.729</td>
</tr>
<tr id="A6.T2.4.3.2" class="ltx_tr">
<th id="A6.T2.4.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">5</th>
<td id="A6.T2.4.3.2.2" class="ltx_td ltx_align_center">0.800</td>
<td id="A6.T2.4.3.2.3" class="ltx_td ltx_nopad_r ltx_align_center">0.640</td>
</tr>
<tr id="A6.T2.4.4.3" class="ltx_tr">
<th id="A6.T2.4.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">10</th>
<td id="A6.T2.4.4.3.2" class="ltx_td ltx_align_center">0.792</td>
<td id="A6.T2.4.4.3.3" class="ltx_td ltx_nopad_r ltx_align_center">0.578</td>
</tr>
<tr id="A6.T2.4.5.4" class="ltx_tr">
<th id="A6.T2.4.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">15</th>
<td id="A6.T2.4.5.4.2" class="ltx_td ltx_align_center">0.815</td>
<td id="A6.T2.4.5.4.3" class="ltx_td ltx_nopad_r ltx_align_center">0.545</td>
</tr>
<tr id="A6.T2.4.6.5" class="ltx_tr">
<th id="A6.T2.4.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">20</th>
<td id="A6.T2.4.6.5.2" class="ltx_td ltx_align_center">0.801</td>
<td id="A6.T2.4.6.5.3" class="ltx_td ltx_nopad_r ltx_align_center">0.560</td>
</tr>
<tr id="A6.T2.4.7.6" class="ltx_tr">
<th id="A6.T2.4.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">25</th>
<td id="A6.T2.4.7.6.2" class="ltx_td ltx_align_center">0.812</td>
<td id="A6.T2.4.7.6.3" class="ltx_td ltx_nopad_r ltx_align_center">0.544</td>
</tr>
<tr id="A6.T2.4.8.7" class="ltx_tr">
<th id="A6.T2.4.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">30</th>
<td id="A6.T2.4.8.7.2" class="ltx_td ltx_align_center">0.799</td>
<td id="A6.T2.4.8.7.3" class="ltx_td ltx_nopad_r ltx_align_center">0.556</td>
</tr>
<tr id="A6.T2.4.9.8" class="ltx_tr">
<th id="A6.T2.4.9.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">35</th>
<td id="A6.T2.4.9.8.2" class="ltx_td ltx_align_center">0.810</td>
<td id="A6.T2.4.9.8.3" class="ltx_td ltx_nopad_r ltx_align_center">0.557</td>
</tr>
<tr id="A6.T2.4.10.9" class="ltx_tr">
<th id="A6.T2.4.10.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">40</th>
<td id="A6.T2.4.10.9.2" class="ltx_td ltx_align_center">0.790</td>
<td id="A6.T2.4.10.9.3" class="ltx_td ltx_nopad_r ltx_align_center">0.551</td>
</tr>
<tr id="A6.T2.4.11.10" class="ltx_tr">
<th id="A6.T2.4.11.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">45</th>
<td id="A6.T2.4.11.10.2" class="ltx_td ltx_align_center">0.725</td>
<td id="A6.T2.4.11.10.3" class="ltx_td ltx_nopad_r ltx_align_center">0.539</td>
</tr>
<tr id="A6.T2.4.12.11" class="ltx_tr">
<th id="A6.T2.4.12.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">50</th>
<td id="A6.T2.4.12.11.2" class="ltx_td ltx_align_center">0.638</td>
<td id="A6.T2.4.12.11.3" class="ltx_td ltx_nopad_r ltx_align_center">0.541</td>
</tr>
<tr id="A6.T2.4.13.12" class="ltx_tr">
<th id="A6.T2.4.13.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">55</th>
<td id="A6.T2.4.13.12.2" class="ltx_td ltx_align_center">0.638</td>
<td id="A6.T2.4.13.12.3" class="ltx_td ltx_nopad_r ltx_align_center">0.501</td>
</tr>
<tr id="A6.T2.4.14.13" class="ltx_tr">
<th id="A6.T2.4.14.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">60</th>
<td id="A6.T2.4.14.13.2" class="ltx_td ltx_align_center">0.625</td>
<td id="A6.T2.4.14.13.3" class="ltx_td ltx_nopad_r ltx_align_center">0.497</td>
</tr>
<tr id="A6.T2.4.15.14" class="ltx_tr">
<th id="A6.T2.4.15.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">65</th>
<td id="A6.T2.4.15.14.2" class="ltx_td ltx_align_center">0.635</td>
<td id="A6.T2.4.15.14.3" class="ltx_td ltx_nopad_r ltx_align_center">0.511</td>
</tr>
<tr id="A6.T2.4.16.15" class="ltx_tr">
<th id="A6.T2.4.16.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">70</th>
<td id="A6.T2.4.16.15.2" class="ltx_td ltx_align_center">0.623</td>
<td id="A6.T2.4.16.15.3" class="ltx_td ltx_nopad_r ltx_align_center">0.497</td>
</tr>
<tr id="A6.T2.4.17.16" class="ltx_tr">
<th id="A6.T2.4.17.16.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">75</th>
<td id="A6.T2.4.17.16.2" class="ltx_td ltx_align_center ltx_border_bb">0.615</td>
<td id="A6.T2.4.17.16.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb">0.493</td>
</tr>
</tbody>
</table>
</figure>
<figure id="A6.T3" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="A6.T3.2.1.1" class="ltx_text" style="font-size:90%;">Table C</span>: </span><span id="A6.T3.3.2" class="ltx_text" style="font-size:90%;">Detailed performance comparison on COQA dataset with LLaMA-3-70B</span></figcaption>
<table id="A6.T3.4" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A6.T3.4.1.1" class="ltx_tr">
<th id="A6.T3.4.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">Layer</th>
<th id="A6.T3.4.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">LLaMA-3-70B MiniCache</th>
<th id="A6.T3.4.1.1.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt">LLaMA-3-70B Mean</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A6.T3.4.2.1" class="ltx_tr">
<th id="A6.T3.4.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">0</th>
<td id="A6.T3.4.2.1.2" class="ltx_td ltx_align_center ltx_border_t">0.705</td>
<td id="A6.T3.4.2.1.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">0.706</td>
</tr>
<tr id="A6.T3.4.3.2" class="ltx_tr">
<th id="A6.T3.4.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">4</th>
<td id="A6.T3.4.3.2.2" class="ltx_td ltx_align_center">0.705</td>
<td id="A6.T3.4.3.2.3" class="ltx_td ltx_nopad_r ltx_align_center">0.699</td>
</tr>
<tr id="A6.T3.4.4.3" class="ltx_tr">
<th id="A6.T3.4.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">8</th>
<td id="A6.T3.4.4.3.2" class="ltx_td ltx_align_center">0.706</td>
<td id="A6.T3.4.4.3.3" class="ltx_td ltx_nopad_r ltx_align_center">0.696</td>
</tr>
<tr id="A6.T3.4.5.4" class="ltx_tr">
<th id="A6.T3.4.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">12</th>
<td id="A6.T3.4.5.4.2" class="ltx_td ltx_align_center">0.704</td>
<td id="A6.T3.4.5.4.3" class="ltx_td ltx_nopad_r ltx_align_center">0.691</td>
</tr>
<tr id="A6.T3.4.6.5" class="ltx_tr">
<th id="A6.T3.4.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">16</th>
<td id="A6.T3.4.6.5.2" class="ltx_td ltx_align_center">0.704</td>
<td id="A6.T3.4.6.5.3" class="ltx_td ltx_nopad_r ltx_align_center">0.690</td>
</tr>
<tr id="A6.T3.4.7.6" class="ltx_tr">
<th id="A6.T3.4.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">20</th>
<td id="A6.T3.4.7.6.2" class="ltx_td ltx_align_center">0.703</td>
<td id="A6.T3.4.7.6.3" class="ltx_td ltx_nopad_r ltx_align_center">0.690</td>
</tr>
<tr id="A6.T3.4.8.7" class="ltx_tr">
<th id="A6.T3.4.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">24</th>
<td id="A6.T3.4.8.7.2" class="ltx_td ltx_align_center">0.701</td>
<td id="A6.T3.4.8.7.3" class="ltx_td ltx_nopad_r ltx_align_center">0.690</td>
</tr>
<tr id="A6.T3.4.9.8" class="ltx_tr">
<th id="A6.T3.4.9.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">28</th>
<td id="A6.T3.4.9.8.2" class="ltx_td ltx_align_center">0.702</td>
<td id="A6.T3.4.9.8.3" class="ltx_td ltx_nopad_r ltx_align_center">0.690</td>
</tr>
<tr id="A6.T3.4.10.9" class="ltx_tr">
<th id="A6.T3.4.10.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">32</th>
<td id="A6.T3.4.10.9.2" class="ltx_td ltx_align_center">0.702</td>
<td id="A6.T3.4.10.9.3" class="ltx_td ltx_nopad_r ltx_align_center">0.688</td>
</tr>
<tr id="A6.T3.4.11.10" class="ltx_tr">
<th id="A6.T3.4.11.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">36</th>
<td id="A6.T3.4.11.10.2" class="ltx_td ltx_align_center">0.703</td>
<td id="A6.T3.4.11.10.3" class="ltx_td ltx_nopad_r ltx_align_center">0.688</td>
</tr>
<tr id="A6.T3.4.12.11" class="ltx_tr">
<th id="A6.T3.4.12.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">40</th>
<td id="A6.T3.4.12.11.2" class="ltx_td ltx_align_center">0.697</td>
<td id="A6.T3.4.12.11.3" class="ltx_td ltx_nopad_r ltx_align_center">0.687</td>
</tr>
<tr id="A6.T3.4.13.12" class="ltx_tr">
<th id="A6.T3.4.13.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">44</th>
<td id="A6.T3.4.13.12.2" class="ltx_td ltx_align_center">0.698</td>
<td id="A6.T3.4.13.12.3" class="ltx_td ltx_nopad_r ltx_align_center">0.685</td>
</tr>
<tr id="A6.T3.4.14.13" class="ltx_tr">
<th id="A6.T3.4.14.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">48</th>
<td id="A6.T3.4.14.13.2" class="ltx_td ltx_align_center">0.699</td>
<td id="A6.T3.4.14.13.3" class="ltx_td ltx_nopad_r ltx_align_center">0.678</td>
</tr>
<tr id="A6.T3.4.15.14" class="ltx_tr">
<th id="A6.T3.4.15.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">52</th>
<td id="A6.T3.4.15.14.2" class="ltx_td ltx_align_center">0.699</td>
<td id="A6.T3.4.15.14.3" class="ltx_td ltx_nopad_r ltx_align_center">0.672</td>
</tr>
<tr id="A6.T3.4.16.15" class="ltx_tr">
<th id="A6.T3.4.16.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">56</th>
<td id="A6.T3.4.16.15.2" class="ltx_td ltx_align_center">0.701</td>
<td id="A6.T3.4.16.15.3" class="ltx_td ltx_nopad_r ltx_align_center">0.668</td>
</tr>
<tr id="A6.T3.4.17.16" class="ltx_tr">
<th id="A6.T3.4.17.16.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">60</th>
<td id="A6.T3.4.17.16.2" class="ltx_td ltx_align_center">0.704</td>
<td id="A6.T3.4.17.16.3" class="ltx_td ltx_nopad_r ltx_align_center">0.657</td>
</tr>
<tr id="A6.T3.4.18.17" class="ltx_tr">
<th id="A6.T3.4.18.17.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">64</th>
<td id="A6.T3.4.18.17.2" class="ltx_td ltx_align_center">0.706</td>
<td id="A6.T3.4.18.17.3" class="ltx_td ltx_nopad_r ltx_align_center">0.635</td>
</tr>
<tr id="A6.T3.4.19.18" class="ltx_tr">
<th id="A6.T3.4.19.18.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">68</th>
<td id="A6.T3.4.19.18.2" class="ltx_td ltx_align_center">0.691</td>
<td id="A6.T3.4.19.18.3" class="ltx_td ltx_nopad_r ltx_align_center">0.611</td>
</tr>
<tr id="A6.T3.4.20.19" class="ltx_tr">
<th id="A6.T3.4.20.19.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">72</th>
<td id="A6.T3.4.20.19.2" class="ltx_td ltx_align_center">0.689</td>
<td id="A6.T3.4.20.19.3" class="ltx_td ltx_nopad_r ltx_align_center">0.565</td>
</tr>
<tr id="A6.T3.4.21.20" class="ltx_tr">
<th id="A6.T3.4.21.20.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">76</th>
<td id="A6.T3.4.21.20.2" class="ltx_td ltx_align_center ltx_border_bb">0.641</td>
<td id="A6.T3.4.21.20.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb">0.526</td>
</tr>
</tbody>
</table>
</figure>
<figure id="A6.T4" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="A6.T4.2.1.1" class="ltx_text" style="font-size:90%;">Table D</span>: </span><span id="A6.T4.3.2" class="ltx_text" style="font-size:90%;">Detailed performance comparison on TruthfulQA dataset with LLaMA-3-70B</span></figcaption>
<table id="A6.T4.4" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A6.T4.4.1.1" class="ltx_tr">
<th id="A6.T4.4.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">Layer</th>
<th id="A6.T4.4.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">LLaMA-3-70B MiniCache</th>
<th id="A6.T4.4.1.1.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt">LLaMA-3-70B Mean</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A6.T4.4.2.1" class="ltx_tr">
<th id="A6.T4.4.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">0</th>
<td id="A6.T4.4.2.1.2" class="ltx_td ltx_align_center ltx_border_t">22.615</td>
<td id="A6.T4.4.2.1.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">22.130</td>
</tr>
<tr id="A6.T4.4.3.2" class="ltx_tr">
<th id="A6.T4.4.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">4</th>
<td id="A6.T4.4.3.2.2" class="ltx_td ltx_align_center">22.512</td>
<td id="A6.T4.4.3.2.3" class="ltx_td ltx_nopad_r ltx_align_center">22.005</td>
</tr>
<tr id="A6.T4.4.4.3" class="ltx_tr">
<th id="A6.T4.4.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">8</th>
<td id="A6.T4.4.4.3.2" class="ltx_td ltx_align_center">22.451</td>
<td id="A6.T4.4.4.3.3" class="ltx_td ltx_nopad_r ltx_align_center">21.876</td>
</tr>
<tr id="A6.T4.4.5.4" class="ltx_tr">
<th id="A6.T4.4.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">12</th>
<td id="A6.T4.4.5.4.2" class="ltx_td ltx_align_center">22.413</td>
<td id="A6.T4.4.5.4.3" class="ltx_td ltx_nopad_r ltx_align_center">21.303</td>
</tr>
<tr id="A6.T4.4.6.5" class="ltx_tr">
<th id="A6.T4.4.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">16</th>
<td id="A6.T4.4.6.5.2" class="ltx_td ltx_align_center">22.387</td>
<td id="A6.T4.4.6.5.3" class="ltx_td ltx_nopad_r ltx_align_center">21.209</td>
</tr>
<tr id="A6.T4.4.7.6" class="ltx_tr">
<th id="A6.T4.4.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">20</th>
<td id="A6.T4.4.7.6.2" class="ltx_td ltx_align_center">22.387</td>
<td id="A6.T4.4.7.6.3" class="ltx_td ltx_nopad_r ltx_align_center">20.752</td>
</tr>
<tr id="A6.T4.4.8.7" class="ltx_tr">
<th id="A6.T4.4.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">24</th>
<td id="A6.T4.4.8.7.2" class="ltx_td ltx_align_center">22.387</td>
<td id="A6.T4.4.8.7.3" class="ltx_td ltx_nopad_r ltx_align_center">20.657</td>
</tr>
<tr id="A6.T4.4.9.8" class="ltx_tr">
<th id="A6.T4.4.9.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">28</th>
<td id="A6.T4.4.9.8.2" class="ltx_td ltx_align_center">22.276</td>
<td id="A6.T4.4.9.8.3" class="ltx_td ltx_nopad_r ltx_align_center">20.501</td>
</tr>
<tr id="A6.T4.4.10.9" class="ltx_tr">
<th id="A6.T4.4.10.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">32</th>
<td id="A6.T4.4.10.9.2" class="ltx_td ltx_align_center">22.130</td>
<td id="A6.T4.4.10.9.3" class="ltx_td ltx_nopad_r ltx_align_center">20.479</td>
</tr>
<tr id="A6.T4.4.11.10" class="ltx_tr">
<th id="A6.T4.4.11.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">36</th>
<td id="A6.T4.4.11.10.2" class="ltx_td ltx_align_center">22.130</td>
<td id="A6.T4.4.11.10.3" class="ltx_td ltx_nopad_r ltx_align_center">20.335</td>
</tr>
<tr id="A6.T4.4.12.11" class="ltx_tr">
<th id="A6.T4.4.12.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">40</th>
<td id="A6.T4.4.12.11.2" class="ltx_td ltx_align_center">22.073</td>
<td id="A6.T4.4.12.11.3" class="ltx_td ltx_nopad_r ltx_align_center">19.834</td>
</tr>
<tr id="A6.T4.4.13.12" class="ltx_tr">
<th id="A6.T4.4.13.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">44</th>
<td id="A6.T4.4.13.12.2" class="ltx_td ltx_align_center">21.356</td>
<td id="A6.T4.4.13.12.3" class="ltx_td ltx_nopad_r ltx_align_center">17.024</td>
</tr>
<tr id="A6.T4.4.14.13" class="ltx_tr">
<th id="A6.T4.4.14.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">48</th>
<td id="A6.T4.4.14.13.2" class="ltx_td ltx_align_center">21.356</td>
<td id="A6.T4.4.14.13.3" class="ltx_td ltx_nopad_r ltx_align_center">12.440</td>
</tr>
<tr id="A6.T4.4.15.14" class="ltx_tr">
<th id="A6.T4.4.15.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">52</th>
<td id="A6.T4.4.15.14.2" class="ltx_td ltx_align_center">21.333</td>
<td id="A6.T4.4.15.14.3" class="ltx_td ltx_nopad_r ltx_align_center">9.127</td>
</tr>
<tr id="A6.T4.4.16.15" class="ltx_tr">
<th id="A6.T4.4.16.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">56</th>
<td id="A6.T4.4.16.15.2" class="ltx_td ltx_align_center">21.316</td>
<td id="A6.T4.4.16.15.3" class="ltx_td ltx_nopad_r ltx_align_center">3.255</td>
</tr>
<tr id="A6.T4.4.17.16" class="ltx_tr">
<th id="A6.T4.4.17.16.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">60</th>
<td id="A6.T4.4.17.16.2" class="ltx_td ltx_align_center">21.172</td>
<td id="A6.T4.4.17.16.3" class="ltx_td ltx_nopad_r ltx_align_center">2.349</td>
</tr>
<tr id="A6.T4.4.18.17" class="ltx_tr">
<th id="A6.T4.4.18.17.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">64</th>
<td id="A6.T4.4.18.17.2" class="ltx_td ltx_align_center">21.153</td>
<td id="A6.T4.4.18.17.3" class="ltx_td ltx_nopad_r ltx_align_center">2.250</td>
</tr>
<tr id="A6.T4.4.19.18" class="ltx_tr">
<th id="A6.T4.4.19.18.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">68</th>
<td id="A6.T4.4.19.18.2" class="ltx_td ltx_align_center">21.002</td>
<td id="A6.T4.4.19.18.3" class="ltx_td ltx_nopad_r ltx_align_center">1.721</td>
</tr>
<tr id="A6.T4.4.20.19" class="ltx_tr">
<th id="A6.T4.4.20.19.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">72</th>
<td id="A6.T4.4.20.19.2" class="ltx_td ltx_align_center">20.940</td>
<td id="A6.T4.4.20.19.3" class="ltx_td ltx_nopad_r ltx_align_center">1.119</td>
</tr>
<tr id="A6.T4.4.21.20" class="ltx_tr">
<th id="A6.T4.4.21.20.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">76</th>
<td id="A6.T4.4.21.20.2" class="ltx_td ltx_align_center ltx_border_bb">20.683</td>
<td id="A6.T4.4.21.20.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb">0.784</td>
</tr>
</tbody>
</table>
</figure>
<figure id="A6.T5" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="A6.T5.2.1.1" class="ltx_text" style="font-size:90%;">Table E</span>: </span><span id="A6.T5.3.2" class="ltx_text" style="font-size:90%;">Detailed performance comparison on GSM8K dataset with LLaMA-3-8B</span></figcaption>
<table id="A6.T5.4" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A6.T5.4.1.1" class="ltx_tr">
<th id="A6.T5.4.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">Layer</th>
<th id="A6.T5.4.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">LLaMA-3-8B MiniCache</th>
<th id="A6.T5.4.1.1.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt">LLaMA-3-8B Mean</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A6.T5.4.2.1" class="ltx_tr">
<th id="A6.T5.4.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">0</th>
<td id="A6.T5.4.2.1.2" class="ltx_td ltx_align_center ltx_border_t">0.488</td>
<td id="A6.T5.4.2.1.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">0.467</td>
</tr>
<tr id="A6.T5.4.3.2" class="ltx_tr">
<th id="A6.T5.4.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">2</th>
<td id="A6.T5.4.3.2.2" class="ltx_td ltx_align_center">0.476</td>
<td id="A6.T5.4.3.2.3" class="ltx_td ltx_nopad_r ltx_align_center">0.369</td>
</tr>
<tr id="A6.T5.4.4.3" class="ltx_tr">
<th id="A6.T5.4.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">4</th>
<td id="A6.T5.4.4.3.2" class="ltx_td ltx_align_center">0.489</td>
<td id="A6.T5.4.4.3.3" class="ltx_td ltx_nopad_r ltx_align_center">0.388</td>
</tr>
<tr id="A6.T5.4.5.4" class="ltx_tr">
<th id="A6.T5.4.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">6</th>
<td id="A6.T5.4.5.4.2" class="ltx_td ltx_align_center">0.487</td>
<td id="A6.T5.4.5.4.3" class="ltx_td ltx_nopad_r ltx_align_center">0.387</td>
</tr>
<tr id="A6.T5.4.6.5" class="ltx_tr">
<th id="A6.T5.4.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">8</th>
<td id="A6.T5.4.6.5.2" class="ltx_td ltx_align_center">0.489</td>
<td id="A6.T5.4.6.5.3" class="ltx_td ltx_nopad_r ltx_align_center">0.359</td>
</tr>
<tr id="A6.T5.4.7.6" class="ltx_tr">
<th id="A6.T5.4.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">10</th>
<td id="A6.T5.4.7.6.2" class="ltx_td ltx_align_center">0.479</td>
<td id="A6.T5.4.7.6.3" class="ltx_td ltx_nopad_r ltx_align_center">0.388</td>
</tr>
<tr id="A6.T5.4.8.7" class="ltx_tr">
<th id="A6.T5.4.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">12</th>
<td id="A6.T5.4.8.7.2" class="ltx_td ltx_align_center">0.486</td>
<td id="A6.T5.4.8.7.3" class="ltx_td ltx_nopad_r ltx_align_center">0.384</td>
</tr>
<tr id="A6.T5.4.9.8" class="ltx_tr">
<th id="A6.T5.4.9.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">14</th>
<td id="A6.T5.4.9.8.2" class="ltx_td ltx_align_center">0.472</td>
<td id="A6.T5.4.9.8.3" class="ltx_td ltx_nopad_r ltx_align_center">0.368</td>
</tr>
<tr id="A6.T5.4.10.9" class="ltx_tr">
<th id="A6.T5.4.10.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">16</th>
<td id="A6.T5.4.10.9.2" class="ltx_td ltx_align_center">0.477</td>
<td id="A6.T5.4.10.9.3" class="ltx_td ltx_nopad_r ltx_align_center">0.343</td>
</tr>
<tr id="A6.T5.4.11.10" class="ltx_tr">
<th id="A6.T5.4.11.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">18</th>
<td id="A6.T5.4.11.10.2" class="ltx_td ltx_align_center">0.446</td>
<td id="A6.T5.4.11.10.3" class="ltx_td ltx_nopad_r ltx_align_center">0.291</td>
</tr>
<tr id="A6.T5.4.12.11" class="ltx_tr">
<th id="A6.T5.4.12.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">20</th>
<td id="A6.T5.4.12.11.2" class="ltx_td ltx_align_center">0.447</td>
<td id="A6.T5.4.12.11.3" class="ltx_td ltx_nopad_r ltx_align_center">0.271</td>
</tr>
<tr id="A6.T5.4.13.12" class="ltx_tr">
<th id="A6.T5.4.13.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">22</th>
<td id="A6.T5.4.13.12.2" class="ltx_td ltx_align_center">0.433</td>
<td id="A6.T5.4.13.12.3" class="ltx_td ltx_nopad_r ltx_align_center">0.234</td>
</tr>
<tr id="A6.T5.4.14.13" class="ltx_tr">
<th id="A6.T5.4.14.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">24</th>
<td id="A6.T5.4.14.13.2" class="ltx_td ltx_align_center">0.399</td>
<td id="A6.T5.4.14.13.3" class="ltx_td ltx_nopad_r ltx_align_center">0.155</td>
</tr>
<tr id="A6.T5.4.15.14" class="ltx_tr">
<th id="A6.T5.4.15.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">26</th>
<td id="A6.T5.4.15.14.2" class="ltx_td ltx_align_center">0.396</td>
<td id="A6.T5.4.15.14.3" class="ltx_td ltx_nopad_r ltx_align_center">0.140</td>
</tr>
<tr id="A6.T5.4.16.15" class="ltx_tr">
<th id="A6.T5.4.16.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">28</th>
<td id="A6.T5.4.16.15.2" class="ltx_td ltx_align_center">0.395</td>
<td id="A6.T5.4.16.15.3" class="ltx_td ltx_nopad_r ltx_align_center">0.052</td>
</tr>
<tr id="A6.T5.4.17.16" class="ltx_tr">
<th id="A6.T5.4.17.16.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">30</th>
<td id="A6.T5.4.17.16.2" class="ltx_td ltx_align_center">0.391</td>
<td id="A6.T5.4.17.16.3" class="ltx_td ltx_nopad_r ltx_align_center">0.024</td>
</tr>
<tr id="A6.T5.4.18.17" class="ltx_tr">
<th id="A6.T5.4.18.17.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">32</th>
<td id="A6.T5.4.18.17.2" class="ltx_td ltx_align_center ltx_border_bb">0.397</td>
<td id="A6.T5.4.18.17.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb">0.025</td>
</tr>
</tbody>
</table>
</figure>
<figure id="A6.T6" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="A6.T6.2.1.1" class="ltx_text" style="font-size:90%;">Table F</span>: </span><span id="A6.T6.3.2" class="ltx_text" style="font-size:90%;">Detailed performance comparison on COQA dataset with LLaMA-3-8B</span></figcaption>
<table id="A6.T6.4" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A6.T6.4.1.1" class="ltx_tr">
<th id="A6.T6.4.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">Layer</th>
<th id="A6.T6.4.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">LLaMA-3-8B MiniCache</th>
<th id="A6.T6.4.1.1.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt">LLaMA-3-8B Mean</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A6.T6.4.2.1" class="ltx_tr">
<th id="A6.T6.4.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">0</th>
<td id="A6.T6.4.2.1.2" class="ltx_td ltx_align_center ltx_border_t">0.676</td>
<td id="A6.T6.4.2.1.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">0.676</td>
</tr>
<tr id="A6.T6.4.3.2" class="ltx_tr">
<th id="A6.T6.4.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">2</th>
<td id="A6.T6.4.3.2.2" class="ltx_td ltx_align_center">0.676</td>
<td id="A6.T6.4.3.2.3" class="ltx_td ltx_nopad_r ltx_align_center">0.571</td>
</tr>
<tr id="A6.T6.4.4.3" class="ltx_tr">
<th id="A6.T6.4.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">4</th>
<td id="A6.T6.4.4.3.2" class="ltx_td ltx_align_center">0.675</td>
<td id="A6.T6.4.4.3.3" class="ltx_td ltx_nopad_r ltx_align_center">0.566</td>
</tr>
<tr id="A6.T6.4.5.4" class="ltx_tr">
<th id="A6.T6.4.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">6</th>
<td id="A6.T6.4.5.4.2" class="ltx_td ltx_align_center">0.674</td>
<td id="A6.T6.4.5.4.3" class="ltx_td ltx_nopad_r ltx_align_center">0.564</td>
</tr>
<tr id="A6.T6.4.6.5" class="ltx_tr">
<th id="A6.T6.4.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">8</th>
<td id="A6.T6.4.6.5.2" class="ltx_td ltx_align_center">0.674</td>
<td id="A6.T6.4.6.5.3" class="ltx_td ltx_nopad_r ltx_align_center">0.561</td>
</tr>
<tr id="A6.T6.4.7.6" class="ltx_tr">
<th id="A6.T6.4.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">10</th>
<td id="A6.T6.4.7.6.2" class="ltx_td ltx_align_center">0.673</td>
<td id="A6.T6.4.7.6.3" class="ltx_td ltx_nopad_r ltx_align_center">0.560</td>
</tr>
<tr id="A6.T6.4.8.7" class="ltx_tr">
<th id="A6.T6.4.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">12</th>
<td id="A6.T6.4.8.7.2" class="ltx_td ltx_align_center">0.672</td>
<td id="A6.T6.4.8.7.3" class="ltx_td ltx_nopad_r ltx_align_center">0.560</td>
</tr>
<tr id="A6.T6.4.9.8" class="ltx_tr">
<th id="A6.T6.4.9.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">14</th>
<td id="A6.T6.4.9.8.2" class="ltx_td ltx_align_center">0.670</td>
<td id="A6.T6.4.9.8.3" class="ltx_td ltx_nopad_r ltx_align_center">0.559</td>
</tr>
<tr id="A6.T6.4.10.9" class="ltx_tr">
<th id="A6.T6.4.10.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">16</th>
<td id="A6.T6.4.10.9.2" class="ltx_td ltx_align_center">0.670</td>
<td id="A6.T6.4.10.9.3" class="ltx_td ltx_nopad_r ltx_align_center">0.558</td>
</tr>
<tr id="A6.T6.4.11.10" class="ltx_tr">
<th id="A6.T6.4.11.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">18</th>
<td id="A6.T6.4.11.10.2" class="ltx_td ltx_align_center">0.669</td>
<td id="A6.T6.4.11.10.3" class="ltx_td ltx_nopad_r ltx_align_center">0.555</td>
</tr>
<tr id="A6.T6.4.12.11" class="ltx_tr">
<th id="A6.T6.4.12.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">20</th>
<td id="A6.T6.4.12.11.2" class="ltx_td ltx_align_center">0.669</td>
<td id="A6.T6.4.12.11.3" class="ltx_td ltx_nopad_r ltx_align_center">0.552</td>
</tr>
<tr id="A6.T6.4.13.12" class="ltx_tr">
<th id="A6.T6.4.13.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">22</th>
<td id="A6.T6.4.13.12.2" class="ltx_td ltx_align_center">0.668</td>
<td id="A6.T6.4.13.12.3" class="ltx_td ltx_nopad_r ltx_align_center">0.549</td>
</tr>
<tr id="A6.T6.4.14.13" class="ltx_tr">
<th id="A6.T6.4.14.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">24</th>
<td id="A6.T6.4.14.13.2" class="ltx_td ltx_align_center">0.667</td>
<td id="A6.T6.4.14.13.3" class="ltx_td ltx_nopad_r ltx_align_center">0.543</td>
</tr>
<tr id="A6.T6.4.15.14" class="ltx_tr">
<th id="A6.T6.4.15.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">26</th>
<td id="A6.T6.4.15.14.2" class="ltx_td ltx_align_center">0.667</td>
<td id="A6.T6.4.15.14.3" class="ltx_td ltx_nopad_r ltx_align_center">0.537</td>
</tr>
<tr id="A6.T6.4.16.15" class="ltx_tr">
<th id="A6.T6.4.16.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">28</th>
<td id="A6.T6.4.16.15.2" class="ltx_td ltx_align_center">0.666</td>
<td id="A6.T6.4.16.15.3" class="ltx_td ltx_nopad_r ltx_align_center">0.536</td>
</tr>
<tr id="A6.T6.4.17.16" class="ltx_tr">
<th id="A6.T6.4.17.16.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">30</th>
<td id="A6.T6.4.17.16.2" class="ltx_td ltx_align_center">0.666</td>
<td id="A6.T6.4.17.16.3" class="ltx_td ltx_nopad_r ltx_align_center">0.531</td>
</tr>
<tr id="A6.T6.4.18.17" class="ltx_tr">
<th id="A6.T6.4.18.17.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">32</th>
<td id="A6.T6.4.18.17.2" class="ltx_td ltx_align_center ltx_border_bb">0.665</td>
<td id="A6.T6.4.18.17.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb">0.528</td>
</tr>
</tbody>
</table>
</figure>
<figure id="A6.T7" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="A6.T7.2.1.1" class="ltx_text" style="font-size:90%;">Table G</span>: </span><span id="A6.T7.3.2" class="ltx_text" style="font-size:90%;">Detailed performance comparison on TruthfulQA dataset with LLaMA-3-8B</span></figcaption>
<table id="A6.T7.4" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A6.T7.4.1.1" class="ltx_tr">
<th id="A6.T7.4.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">Layer</th>
<th id="A6.T7.4.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">LLaMA-3-8B MiniCache</th>
<th id="A6.T7.4.1.1.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt">LLaMA-3-8B Mean</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A6.T7.4.2.1" class="ltx_tr">
<th id="A6.T7.4.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">0</th>
<td id="A6.T7.4.2.1.2" class="ltx_td ltx_align_center ltx_border_t">32.520</td>
<td id="A6.T7.4.2.1.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">32.524</td>
</tr>
<tr id="A6.T7.4.3.2" class="ltx_tr">
<th id="A6.T7.4.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">2</th>
<td id="A6.T7.4.3.2.2" class="ltx_td ltx_align_center">32.231</td>
<td id="A6.T7.4.3.2.3" class="ltx_td ltx_nopad_r ltx_align_center">28.431</td>
</tr>
<tr id="A6.T7.4.4.3" class="ltx_tr">
<th id="A6.T7.4.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">4</th>
<td id="A6.T7.4.4.3.2" class="ltx_td ltx_align_center">31.645</td>
<td id="A6.T7.4.4.3.3" class="ltx_td ltx_nopad_r ltx_align_center">28.197</td>
</tr>
<tr id="A6.T7.4.5.4" class="ltx_tr">
<th id="A6.T7.4.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">6</th>
<td id="A6.T7.4.5.4.2" class="ltx_td ltx_align_center">31.485</td>
<td id="A6.T7.4.5.4.3" class="ltx_td ltx_nopad_r ltx_align_center">27.894</td>
</tr>
<tr id="A6.T7.4.6.5" class="ltx_tr">
<th id="A6.T7.4.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">8</th>
<td id="A6.T7.4.6.5.2" class="ltx_td ltx_align_center">31.008</td>
<td id="A6.T7.4.6.5.3" class="ltx_td ltx_nopad_r ltx_align_center">27.796</td>
</tr>
<tr id="A6.T7.4.7.6" class="ltx_tr">
<th id="A6.T7.4.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">10</th>
<td id="A6.T7.4.7.6.2" class="ltx_td ltx_align_center">30.964</td>
<td id="A6.T7.4.7.6.3" class="ltx_td ltx_nopad_r ltx_align_center">27.704</td>
</tr>
<tr id="A6.T7.4.8.7" class="ltx_tr">
<th id="A6.T7.4.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">12</th>
<td id="A6.T7.4.8.7.2" class="ltx_td ltx_align_center">30.798</td>
<td id="A6.T7.4.8.7.3" class="ltx_td ltx_nopad_r ltx_align_center">27.371</td>
</tr>
<tr id="A6.T7.4.9.8" class="ltx_tr">
<th id="A6.T7.4.9.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">14</th>
<td id="A6.T7.4.9.8.2" class="ltx_td ltx_align_center">30.798</td>
<td id="A6.T7.4.9.8.3" class="ltx_td ltx_nopad_r ltx_align_center">27.093</td>
</tr>
<tr id="A6.T7.4.10.9" class="ltx_tr">
<th id="A6.T7.4.10.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">16</th>
<td id="A6.T7.4.10.9.2" class="ltx_td ltx_align_center">30.798</td>
<td id="A6.T7.4.10.9.3" class="ltx_td ltx_nopad_r ltx_align_center">26.643</td>
</tr>
<tr id="A6.T7.4.11.10" class="ltx_tr">
<th id="A6.T7.4.11.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">18</th>
<td id="A6.T7.4.11.10.2" class="ltx_td ltx_align_center">30.798</td>
<td id="A6.T7.4.11.10.3" class="ltx_td ltx_nopad_r ltx_align_center">26.517</td>
</tr>
<tr id="A6.T7.4.12.11" class="ltx_tr">
<th id="A6.T7.4.12.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">20</th>
<td id="A6.T7.4.12.11.2" class="ltx_td ltx_align_center">30.798</td>
<td id="A6.T7.4.12.11.3" class="ltx_td ltx_nopad_r ltx_align_center">26.355</td>
</tr>
<tr id="A6.T7.4.13.12" class="ltx_tr">
<th id="A6.T7.4.13.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">22</th>
<td id="A6.T7.4.13.12.2" class="ltx_td ltx_align_center">30.798</td>
<td id="A6.T7.4.13.12.3" class="ltx_td ltx_nopad_r ltx_align_center">26.011</td>
</tr>
<tr id="A6.T7.4.14.13" class="ltx_tr">
<th id="A6.T7.4.14.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">24</th>
<td id="A6.T7.4.14.13.2" class="ltx_td ltx_align_center">30.798</td>
<td id="A6.T7.4.14.13.3" class="ltx_td ltx_nopad_r ltx_align_center">25.044</td>
</tr>
<tr id="A6.T7.4.15.14" class="ltx_tr">
<th id="A6.T7.4.15.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">26</th>
<td id="A6.T7.4.15.14.2" class="ltx_td ltx_align_center">30.798</td>
<td id="A6.T7.4.15.14.3" class="ltx_td ltx_nopad_r ltx_align_center">15.254</td>
</tr>
<tr id="A6.T7.4.16.15" class="ltx_tr">
<th id="A6.T7.4.16.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">28</th>
<td id="A6.T7.4.16.15.2" class="ltx_td ltx_align_center">30.798</td>
<td id="A6.T7.4.16.15.3" class="ltx_td ltx_nopad_r ltx_align_center">14.791</td>
</tr>
<tr id="A6.T7.4.17.16" class="ltx_tr">
<th id="A6.T7.4.17.16.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">30</th>
<td id="A6.T7.4.17.16.2" class="ltx_td ltx_align_center">30.765</td>
<td id="A6.T7.4.17.16.3" class="ltx_td ltx_nopad_r ltx_align_center">9.419</td>
</tr>
<tr id="A6.T7.4.18.17" class="ltx_tr">
<th id="A6.T7.4.18.17.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">32</th>
<td id="A6.T7.4.18.17.2" class="ltx_td ltx_align_center ltx_border_bb">30.390</td>
<td id="A6.T7.4.18.17.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb">6.068</td>
</tr>
</tbody>
</table>
</figure>
<figure id="A6.T8" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="A6.T8.2.1.1" class="ltx_text" style="font-size:90%;">Table H</span>: </span><span id="A6.T8.3.2" class="ltx_text" style="font-size:90%;">Detailed performance comparison on GSM8K dataset with Mixtral-8x7B</span></figcaption>
<table id="A6.T8.4" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A6.T8.4.1.1" class="ltx_tr">
<th id="A6.T8.4.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">Layer</th>
<th id="A6.T8.4.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Mixtral-8x7B MiniCache</th>
<th id="A6.T8.4.1.1.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt">Mixtral-8x7B Mean</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A6.T8.4.2.1" class="ltx_tr">
<th id="A6.T8.4.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">0</th>
<td id="A6.T8.4.2.1.2" class="ltx_td ltx_align_center ltx_border_t">0.589</td>
<td id="A6.T8.4.2.1.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">0.575</td>
</tr>
<tr id="A6.T8.4.3.2" class="ltx_tr">
<th id="A6.T8.4.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">2</th>
<td id="A6.T8.4.3.2.2" class="ltx_td ltx_align_center">0.592</td>
<td id="A6.T8.4.3.2.3" class="ltx_td ltx_nopad_r ltx_align_center">0.480</td>
</tr>
<tr id="A6.T8.4.4.3" class="ltx_tr">
<th id="A6.T8.4.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">4</th>
<td id="A6.T8.4.4.3.2" class="ltx_td ltx_align_center">0.593</td>
<td id="A6.T8.4.4.3.3" class="ltx_td ltx_nopad_r ltx_align_center">0.491</td>
</tr>
<tr id="A6.T8.4.5.4" class="ltx_tr">
<th id="A6.T8.4.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">6</th>
<td id="A6.T8.4.5.4.2" class="ltx_td ltx_align_center">0.591</td>
<td id="A6.T8.4.5.4.3" class="ltx_td ltx_nopad_r ltx_align_center">0.469</td>
</tr>
<tr id="A6.T8.4.6.5" class="ltx_tr">
<th id="A6.T8.4.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">8</th>
<td id="A6.T8.4.6.5.2" class="ltx_td ltx_align_center">0.580</td>
<td id="A6.T8.4.6.5.3" class="ltx_td ltx_nopad_r ltx_align_center">0.472</td>
</tr>
<tr id="A6.T8.4.7.6" class="ltx_tr">
<th id="A6.T8.4.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">10</th>
<td id="A6.T8.4.7.6.2" class="ltx_td ltx_align_center">0.592</td>
<td id="A6.T8.4.7.6.3" class="ltx_td ltx_nopad_r ltx_align_center">0.492</td>
</tr>
<tr id="A6.T8.4.8.7" class="ltx_tr">
<th id="A6.T8.4.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">12</th>
<td id="A6.T8.4.8.7.2" class="ltx_td ltx_align_center">0.582</td>
<td id="A6.T8.4.8.7.3" class="ltx_td ltx_nopad_r ltx_align_center">0.485</td>
</tr>
<tr id="A6.T8.4.9.8" class="ltx_tr">
<th id="A6.T8.4.9.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">14</th>
<td id="A6.T8.4.9.8.2" class="ltx_td ltx_align_center">0.572</td>
<td id="A6.T8.4.9.8.3" class="ltx_td ltx_nopad_r ltx_align_center">0.480</td>
</tr>
<tr id="A6.T8.4.10.9" class="ltx_tr">
<th id="A6.T8.4.10.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">16</th>
<td id="A6.T8.4.10.9.2" class="ltx_td ltx_align_center">0.562</td>
<td id="A6.T8.4.10.9.3" class="ltx_td ltx_nopad_r ltx_align_center">0.462</td>
</tr>
<tr id="A6.T8.4.11.10" class="ltx_tr">
<th id="A6.T8.4.11.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">18</th>
<td id="A6.T8.4.11.10.2" class="ltx_td ltx_align_center">0.522</td>
<td id="A6.T8.4.11.10.3" class="ltx_td ltx_nopad_r ltx_align_center">0.432</td>
</tr>
<tr id="A6.T8.4.12.11" class="ltx_tr">
<th id="A6.T8.4.12.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">20</th>
<td id="A6.T8.4.12.11.2" class="ltx_td ltx_align_center">0.526</td>
<td id="A6.T8.4.12.11.3" class="ltx_td ltx_nopad_r ltx_align_center">0.426</td>
</tr>
<tr id="A6.T8.4.13.12" class="ltx_tr">
<th id="A6.T8.4.13.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">22</th>
<td id="A6.T8.4.13.12.2" class="ltx_td ltx_align_center">0.540</td>
<td id="A6.T8.4.13.12.3" class="ltx_td ltx_nopad_r ltx_align_center">0.416</td>
</tr>
<tr id="A6.T8.4.14.13" class="ltx_tr">
<th id="A6.T8.4.14.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">24</th>
<td id="A6.T8.4.14.13.2" class="ltx_td ltx_align_center">0.519</td>
<td id="A6.T8.4.14.13.3" class="ltx_td ltx_nopad_r ltx_align_center">0.398</td>
</tr>
<tr id="A6.T8.4.15.14" class="ltx_tr">
<th id="A6.T8.4.15.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">26</th>
<td id="A6.T8.4.15.14.2" class="ltx_td ltx_align_center">0.515</td>
<td id="A6.T8.4.15.14.3" class="ltx_td ltx_nopad_r ltx_align_center">0.436</td>
</tr>
<tr id="A6.T8.4.16.15" class="ltx_tr">
<th id="A6.T8.4.16.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">28</th>
<td id="A6.T8.4.16.15.2" class="ltx_td ltx_align_center">0.502</td>
<td id="A6.T8.4.16.15.3" class="ltx_td ltx_nopad_r ltx_align_center">0.401</td>
</tr>
<tr id="A6.T8.4.17.16" class="ltx_tr">
<th id="A6.T8.4.17.16.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">30</th>
<td id="A6.T8.4.17.16.2" class="ltx_td ltx_align_center">0.515</td>
<td id="A6.T8.4.17.16.3" class="ltx_td ltx_nopad_r ltx_align_center">0.386</td>
</tr>
<tr id="A6.T8.4.18.17" class="ltx_tr">
<th id="A6.T8.4.18.17.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">32</th>
<td id="A6.T8.4.18.17.2" class="ltx_td ltx_align_center ltx_border_bb">0.490</td>
<td id="A6.T8.4.18.17.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb">0.258</td>
</tr>
</tbody>
</table>
</figure>
<figure id="A6.T9" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="A6.T9.2.1.1" class="ltx_text" style="font-size:90%;">Table I</span>: </span><span id="A6.T9.3.2" class="ltx_text" style="font-size:90%;">Detailed performance comparison on COQA dataset with Mixtral-8x7B</span></figcaption>
<table id="A6.T9.4" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A6.T9.4.1.1" class="ltx_tr">
<th id="A6.T9.4.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">Layer</th>
<th id="A6.T9.4.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Mixtral-8x7B MiniCache</th>
<th id="A6.T9.4.1.1.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt">Mixtral-8x7B Mean</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A6.T9.4.2.1" class="ltx_tr">
<th id="A6.T9.4.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">0</th>
<td id="A6.T9.4.2.1.2" class="ltx_td ltx_align_center ltx_border_t">0.672</td>
<td id="A6.T9.4.2.1.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">0.675</td>
</tr>
<tr id="A6.T9.4.3.2" class="ltx_tr">
<th id="A6.T9.4.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">2</th>
<td id="A6.T9.4.3.2.2" class="ltx_td ltx_align_center">0.671</td>
<td id="A6.T9.4.3.2.3" class="ltx_td ltx_nopad_r ltx_align_center">0.612</td>
</tr>
<tr id="A6.T9.4.4.3" class="ltx_tr">
<th id="A6.T9.4.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">4</th>
<td id="A6.T9.4.4.3.2" class="ltx_td ltx_align_center">0.670</td>
<td id="A6.T9.4.4.3.3" class="ltx_td ltx_nopad_r ltx_align_center">0.601</td>
</tr>
<tr id="A6.T9.4.5.4" class="ltx_tr">
<th id="A6.T9.4.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">6</th>
<td id="A6.T9.4.5.4.2" class="ltx_td ltx_align_center">0.672</td>
<td id="A6.T9.4.5.4.3" class="ltx_td ltx_nopad_r ltx_align_center">0.590</td>
</tr>
<tr id="A6.T9.4.6.5" class="ltx_tr">
<th id="A6.T9.4.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">8</th>
<td id="A6.T9.4.6.5.2" class="ltx_td ltx_align_center">0.674</td>
<td id="A6.T9.4.6.5.3" class="ltx_td ltx_nopad_r ltx_align_center">0.582</td>
</tr>
<tr id="A6.T9.4.7.6" class="ltx_tr">
<th id="A6.T9.4.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">10</th>
<td id="A6.T9.4.7.6.2" class="ltx_td ltx_align_center">0.671</td>
<td id="A6.T9.4.7.6.3" class="ltx_td ltx_nopad_r ltx_align_center">0.571</td>
</tr>
<tr id="A6.T9.4.8.7" class="ltx_tr">
<th id="A6.T9.4.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">12</th>
<td id="A6.T9.4.8.7.2" class="ltx_td ltx_align_center">0.674</td>
<td id="A6.T9.4.8.7.3" class="ltx_td ltx_nopad_r ltx_align_center">0.561</td>
</tr>
<tr id="A6.T9.4.9.8" class="ltx_tr">
<th id="A6.T9.4.9.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">14</th>
<td id="A6.T9.4.9.8.2" class="ltx_td ltx_align_center">0.670</td>
<td id="A6.T9.4.9.8.3" class="ltx_td ltx_nopad_r ltx_align_center">0.546</td>
</tr>
<tr id="A6.T9.4.10.9" class="ltx_tr">
<th id="A6.T9.4.10.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">16</th>
<td id="A6.T9.4.10.9.2" class="ltx_td ltx_align_center">0.672</td>
<td id="A6.T9.4.10.9.3" class="ltx_td ltx_nopad_r ltx_align_center">0.544</td>
</tr>
<tr id="A6.T9.4.11.10" class="ltx_tr">
<th id="A6.T9.4.11.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">18</th>
<td id="A6.T9.4.11.10.2" class="ltx_td ltx_align_center">0.672</td>
<td id="A6.T9.4.11.10.3" class="ltx_td ltx_nopad_r ltx_align_center">0.530</td>
</tr>
<tr id="A6.T9.4.12.11" class="ltx_tr">
<th id="A6.T9.4.12.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">20</th>
<td id="A6.T9.4.12.11.2" class="ltx_td ltx_align_center">0.675</td>
<td id="A6.T9.4.12.11.3" class="ltx_td ltx_nopad_r ltx_align_center">0.522</td>
</tr>
<tr id="A6.T9.4.13.12" class="ltx_tr">
<th id="A6.T9.4.13.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">22</th>
<td id="A6.T9.4.13.12.2" class="ltx_td ltx_align_center">0.671</td>
<td id="A6.T9.4.13.12.3" class="ltx_td ltx_nopad_r ltx_align_center">0.512</td>
</tr>
<tr id="A6.T9.4.14.13" class="ltx_tr">
<th id="A6.T9.4.14.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">24</th>
<td id="A6.T9.4.14.13.2" class="ltx_td ltx_align_center">0.660</td>
<td id="A6.T9.4.14.13.3" class="ltx_td ltx_nopad_r ltx_align_center">0.455</td>
</tr>
<tr id="A6.T9.4.15.14" class="ltx_tr">
<th id="A6.T9.4.15.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">26</th>
<td id="A6.T9.4.15.14.2" class="ltx_td ltx_align_center">0.657</td>
<td id="A6.T9.4.15.14.3" class="ltx_td ltx_nopad_r ltx_align_center">0.447</td>
</tr>
<tr id="A6.T9.4.16.15" class="ltx_tr">
<th id="A6.T9.4.16.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">28</th>
<td id="A6.T9.4.16.15.2" class="ltx_td ltx_align_center">0.640</td>
<td id="A6.T9.4.16.15.3" class="ltx_td ltx_nopad_r ltx_align_center">0.440</td>
</tr>
<tr id="A6.T9.4.17.16" class="ltx_tr">
<th id="A6.T9.4.17.16.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">30</th>
<td id="A6.T9.4.17.16.2" class="ltx_td ltx_align_center">0.634</td>
<td id="A6.T9.4.17.16.3" class="ltx_td ltx_nopad_r ltx_align_center">0.424</td>
</tr>
<tr id="A6.T9.4.18.17" class="ltx_tr">
<th id="A6.T9.4.18.17.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">32</th>
<td id="A6.T9.4.18.17.2" class="ltx_td ltx_align_center ltx_border_bb">0.459</td>
<td id="A6.T9.4.18.17.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb">0.430</td>
</tr>
</tbody>
</table>
</figure>
<figure id="A6.T10" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="A6.T10.2.1.1" class="ltx_text" style="font-size:90%;">Table J</span>: </span><span id="A6.T10.3.2" class="ltx_text" style="font-size:90%;">Detailed performance comparison on TruthfulQA dataset with Mixtral-8x7B</span></figcaption>
<table id="A6.T10.4" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A6.T10.4.1.1" class="ltx_tr">
<th id="A6.T10.4.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">Layer</th>
<th id="A6.T10.4.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Mixtral-8x7B MiniCache</th>
<th id="A6.T10.4.1.1.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt">Mixtral-8x7B Mean</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A6.T10.4.2.1" class="ltx_tr">
<th id="A6.T10.4.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">0</th>
<td id="A6.T10.4.2.1.2" class="ltx_td ltx_align_center ltx_border_t">21.686</td>
<td id="A6.T10.4.2.1.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">19.465</td>
</tr>
<tr id="A6.T10.4.3.2" class="ltx_tr">
<th id="A6.T10.4.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">2</th>
<td id="A6.T10.4.3.2.2" class="ltx_td ltx_align_center">21.385</td>
<td id="A6.T10.4.3.2.3" class="ltx_td ltx_nopad_r ltx_align_center">19.405</td>
</tr>
<tr id="A6.T10.4.4.3" class="ltx_tr">
<th id="A6.T10.4.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">4</th>
<td id="A6.T10.4.4.3.2" class="ltx_td ltx_align_center">21.368</td>
<td id="A6.T10.4.4.3.3" class="ltx_td ltx_nopad_r ltx_align_center">19.251</td>
</tr>
<tr id="A6.T10.4.5.4" class="ltx_tr">
<th id="A6.T10.4.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">6</th>
<td id="A6.T10.4.5.4.2" class="ltx_td ltx_align_center">21.038</td>
<td id="A6.T10.4.5.4.3" class="ltx_td ltx_nopad_r ltx_align_center">19.094</td>
</tr>
<tr id="A6.T10.4.6.5" class="ltx_tr">
<th id="A6.T10.4.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">8</th>
<td id="A6.T10.4.6.5.2" class="ltx_td ltx_align_center">21.038</td>
<td id="A6.T10.4.6.5.3" class="ltx_td ltx_nopad_r ltx_align_center">18.265</td>
</tr>
<tr id="A6.T10.4.7.6" class="ltx_tr">
<th id="A6.T10.4.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">10</th>
<td id="A6.T10.4.7.6.2" class="ltx_td ltx_align_center">20.216</td>
<td id="A6.T10.4.7.6.3" class="ltx_td ltx_nopad_r ltx_align_center">17.019</td>
</tr>
<tr id="A6.T10.4.8.7" class="ltx_tr">
<th id="A6.T10.4.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">12</th>
<td id="A6.T10.4.8.7.2" class="ltx_td ltx_align_center">20.026</td>
<td id="A6.T10.4.8.7.3" class="ltx_td ltx_nopad_r ltx_align_center">15.902</td>
</tr>
<tr id="A6.T10.4.9.8" class="ltx_tr">
<th id="A6.T10.4.9.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">14</th>
<td id="A6.T10.4.9.8.2" class="ltx_td ltx_align_center">19.723</td>
<td id="A6.T10.4.9.8.3" class="ltx_td ltx_nopad_r ltx_align_center">15.505</td>
</tr>
<tr id="A6.T10.4.10.9" class="ltx_tr">
<th id="A6.T10.4.10.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">16</th>
<td id="A6.T10.4.10.9.2" class="ltx_td ltx_align_center">19.641</td>
<td id="A6.T10.4.10.9.3" class="ltx_td ltx_nopad_r ltx_align_center">15.028</td>
</tr>
<tr id="A6.T10.4.11.10" class="ltx_tr">
<th id="A6.T10.4.11.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">18</th>
<td id="A6.T10.4.11.10.2" class="ltx_td ltx_align_center">19.641</td>
<td id="A6.T10.4.11.10.3" class="ltx_td ltx_nopad_r ltx_align_center">14.723</td>
</tr>
<tr id="A6.T10.4.12.11" class="ltx_tr">
<th id="A6.T10.4.12.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">20</th>
<td id="A6.T10.4.12.11.2" class="ltx_td ltx_align_center">19.546</td>
<td id="A6.T10.4.12.11.3" class="ltx_td ltx_nopad_r ltx_align_center">14.543</td>
</tr>
<tr id="A6.T10.4.13.12" class="ltx_tr">
<th id="A6.T10.4.13.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">22</th>
<td id="A6.T10.4.13.12.2" class="ltx_td ltx_align_center">18.756</td>
<td id="A6.T10.4.13.12.3" class="ltx_td ltx_nopad_r ltx_align_center">14.122</td>
</tr>
<tr id="A6.T10.4.14.13" class="ltx_tr">
<th id="A6.T10.4.14.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">24</th>
<td id="A6.T10.4.14.13.2" class="ltx_td ltx_align_center">18.402</td>
<td id="A6.T10.4.14.13.3" class="ltx_td ltx_nopad_r ltx_align_center">13.834</td>
</tr>
<tr id="A6.T10.4.15.14" class="ltx_tr">
<th id="A6.T10.4.15.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">26</th>
<td id="A6.T10.4.15.14.2" class="ltx_td ltx_align_center">18.366</td>
<td id="A6.T10.4.15.14.3" class="ltx_td ltx_nopad_r ltx_align_center">13.789</td>
</tr>
<tr id="A6.T10.4.16.15" class="ltx_tr">
<th id="A6.T10.4.16.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">28</th>
<td id="A6.T10.4.16.15.2" class="ltx_td ltx_align_center">17.738</td>
<td id="A6.T10.4.16.15.3" class="ltx_td ltx_nopad_r ltx_align_center">12.091</td>
</tr>
<tr id="A6.T10.4.17.16" class="ltx_tr">
<th id="A6.T10.4.17.16.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">30</th>
<td id="A6.T10.4.17.16.2" class="ltx_td ltx_align_center">16.827</td>
<td id="A6.T10.4.17.16.3" class="ltx_td ltx_nopad_r ltx_align_center">12.008</td>
</tr>
<tr id="A6.T10.4.18.17" class="ltx_tr">
<th id="A6.T10.4.18.17.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">32</th>
<td id="A6.T10.4.18.17.2" class="ltx_td ltx_align_center ltx_border_bb">16.635</td>
<td id="A6.T10.4.18.17.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb">0.430</td>
</tr>
</tbody>
</table>
</figure>
<figure id="A6.T11" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="A6.T11.2.1.1" class="ltx_text" style="font-size:90%;">Table K</span>: </span><span id="A6.T11.3.2" class="ltx_text" style="font-size:90%;">Detailed performance comparison on GSM8K dataset with Phi-3-Mini</span></figcaption>
<table id="A6.T11.4" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A6.T11.4.1.1" class="ltx_tr">
<th id="A6.T11.4.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">Layer</th>
<th id="A6.T11.4.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Phi-3-Mini MiniCache</th>
<th id="A6.T11.4.1.1.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt">Phi-3-Mini Mean</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A6.T11.4.2.1" class="ltx_tr">
<th id="A6.T11.4.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">0</th>
<td id="A6.T11.4.2.1.2" class="ltx_td ltx_align_center ltx_border_t">0.774</td>
<td id="A6.T11.4.2.1.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">0.774</td>
</tr>
<tr id="A6.T11.4.3.2" class="ltx_tr">
<th id="A6.T11.4.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">2</th>
<td id="A6.T11.4.3.2.2" class="ltx_td ltx_align_center">0.765</td>
<td id="A6.T11.4.3.2.3" class="ltx_td ltx_nopad_r ltx_align_center">0.667</td>
</tr>
<tr id="A6.T11.4.4.3" class="ltx_tr">
<th id="A6.T11.4.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">4</th>
<td id="A6.T11.4.4.3.2" class="ltx_td ltx_align_center">0.757</td>
<td id="A6.T11.4.4.3.3" class="ltx_td ltx_nopad_r ltx_align_center">0.661</td>
</tr>
<tr id="A6.T11.4.5.4" class="ltx_tr">
<th id="A6.T11.4.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">6</th>
<td id="A6.T11.4.5.4.2" class="ltx_td ltx_align_center">0.754</td>
<td id="A6.T11.4.5.4.3" class="ltx_td ltx_nopad_r ltx_align_center">0.659</td>
</tr>
<tr id="A6.T11.4.6.5" class="ltx_tr">
<th id="A6.T11.4.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">8</th>
<td id="A6.T11.4.6.5.2" class="ltx_td ltx_align_center">0.748</td>
<td id="A6.T11.4.6.5.3" class="ltx_td ltx_nopad_r ltx_align_center">0.657</td>
</tr>
<tr id="A6.T11.4.7.6" class="ltx_tr">
<th id="A6.T11.4.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">10</th>
<td id="A6.T11.4.7.6.2" class="ltx_td ltx_align_center">0.750</td>
<td id="A6.T11.4.7.6.3" class="ltx_td ltx_nopad_r ltx_align_center">0.645</td>
</tr>
<tr id="A6.T11.4.8.7" class="ltx_tr">
<th id="A6.T11.4.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">12</th>
<td id="A6.T11.4.8.7.2" class="ltx_td ltx_align_center">0.750</td>
<td id="A6.T11.4.8.7.3" class="ltx_td ltx_nopad_r ltx_align_center">0.616</td>
</tr>
<tr id="A6.T11.4.9.8" class="ltx_tr">
<th id="A6.T11.4.9.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">14</th>
<td id="A6.T11.4.9.8.2" class="ltx_td ltx_align_center">0.752</td>
<td id="A6.T11.4.9.8.3" class="ltx_td ltx_nopad_r ltx_align_center">0.575</td>
</tr>
<tr id="A6.T11.4.10.9" class="ltx_tr">
<th id="A6.T11.4.10.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">16</th>
<td id="A6.T11.4.10.9.2" class="ltx_td ltx_align_center">0.739</td>
<td id="A6.T11.4.10.9.3" class="ltx_td ltx_nopad_r ltx_align_center">0.491</td>
</tr>
<tr id="A6.T11.4.11.10" class="ltx_tr">
<th id="A6.T11.4.11.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">18</th>
<td id="A6.T11.4.11.10.2" class="ltx_td ltx_align_center">0.742</td>
<td id="A6.T11.4.11.10.3" class="ltx_td ltx_nopad_r ltx_align_center">0.417</td>
</tr>
<tr id="A6.T11.4.12.11" class="ltx_tr">
<th id="A6.T11.4.12.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">20</th>
<td id="A6.T11.4.12.11.2" class="ltx_td ltx_align_center">0.692</td>
<td id="A6.T11.4.12.11.3" class="ltx_td ltx_nopad_r ltx_align_center">0.272</td>
</tr>
<tr id="A6.T11.4.13.12" class="ltx_tr">
<th id="A6.T11.4.13.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">22</th>
<td id="A6.T11.4.13.12.2" class="ltx_td ltx_align_center">0.685</td>
<td id="A6.T11.4.13.12.3" class="ltx_td ltx_nopad_r ltx_align_center">0.206</td>
</tr>
<tr id="A6.T11.4.14.13" class="ltx_tr">
<th id="A6.T11.4.14.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">24</th>
<td id="A6.T11.4.14.13.2" class="ltx_td ltx_align_center">0.640</td>
<td id="A6.T11.4.14.13.3" class="ltx_td ltx_nopad_r ltx_align_center">0.110</td>
</tr>
<tr id="A6.T11.4.15.14" class="ltx_tr">
<th id="A6.T11.4.15.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">26</th>
<td id="A6.T11.4.15.14.2" class="ltx_td ltx_align_center">0.545</td>
<td id="A6.T11.4.15.14.3" class="ltx_td ltx_nopad_r ltx_align_center">0.061</td>
</tr>
<tr id="A6.T11.4.16.15" class="ltx_tr">
<th id="A6.T11.4.16.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">28</th>
<td id="A6.T11.4.16.15.2" class="ltx_td ltx_align_center">0.500</td>
<td id="A6.T11.4.16.15.3" class="ltx_td ltx_nopad_r ltx_align_center">0.039</td>
</tr>
<tr id="A6.T11.4.17.16" class="ltx_tr">
<th id="A6.T11.4.17.16.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">30</th>
<td id="A6.T11.4.17.16.2" class="ltx_td ltx_align_center">0.460</td>
<td id="A6.T11.4.17.16.3" class="ltx_td ltx_nopad_r ltx_align_center">0.036</td>
</tr>
<tr id="A6.T11.4.18.17" class="ltx_tr">
<th id="A6.T11.4.18.17.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">32</th>
<td id="A6.T11.4.18.17.2" class="ltx_td ltx_align_center ltx_border_bb">0.447</td>
<td id="A6.T11.4.18.17.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb">0.028</td>
</tr>
</tbody>
</table>
</figure>
<figure id="A6.T12" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="A6.T12.2.1.1" class="ltx_text" style="font-size:90%;">Table L</span>: </span><span id="A6.T12.3.2" class="ltx_text" style="font-size:90%;">Detailed performance comparison on COQA dataset with Phi-3-Mini</span></figcaption>
<table id="A6.T12.4" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A6.T12.4.1.1" class="ltx_tr">
<th id="A6.T12.4.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">Layer</th>
<th id="A6.T12.4.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Phi-3-Mini MiniCache</th>
<th id="A6.T12.4.1.1.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt">Phi-3-Mini Mean</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A6.T12.4.2.1" class="ltx_tr">
<th id="A6.T12.4.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">0</th>
<td id="A6.T12.4.2.1.2" class="ltx_td ltx_align_center ltx_border_t">0.665</td>
<td id="A6.T12.4.2.1.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">0.665</td>
</tr>
<tr id="A6.T12.4.3.2" class="ltx_tr">
<th id="A6.T12.4.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">2</th>
<td id="A6.T12.4.3.2.2" class="ltx_td ltx_align_center">0.662</td>
<td id="A6.T12.4.3.2.3" class="ltx_td ltx_nopad_r ltx_align_center">0.562</td>
</tr>
<tr id="A6.T12.4.4.3" class="ltx_tr">
<th id="A6.T12.4.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">4</th>
<td id="A6.T12.4.4.3.2" class="ltx_td ltx_align_center">0.657</td>
<td id="A6.T12.4.4.3.3" class="ltx_td ltx_nopad_r ltx_align_center">0.557</td>
</tr>
<tr id="A6.T12.4.5.4" class="ltx_tr">
<th id="A6.T12.4.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">6</th>
<td id="A6.T12.4.5.4.2" class="ltx_td ltx_align_center">0.656</td>
<td id="A6.T12.4.5.4.3" class="ltx_td ltx_nopad_r ltx_align_center">0.556</td>
</tr>
<tr id="A6.T12.4.6.5" class="ltx_tr">
<th id="A6.T12.4.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">8</th>
<td id="A6.T12.4.6.5.2" class="ltx_td ltx_align_center">0.656</td>
<td id="A6.T12.4.6.5.3" class="ltx_td ltx_nopad_r ltx_align_center">0.556</td>
</tr>
<tr id="A6.T12.4.7.6" class="ltx_tr">
<th id="A6.T12.4.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">10</th>
<td id="A6.T12.4.7.6.2" class="ltx_td ltx_align_center">0.654</td>
<td id="A6.T12.4.7.6.3" class="ltx_td ltx_nopad_r ltx_align_center">0.554</td>
</tr>
<tr id="A6.T12.4.8.7" class="ltx_tr">
<th id="A6.T12.4.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">12</th>
<td id="A6.T12.4.8.7.2" class="ltx_td ltx_align_center">0.646</td>
<td id="A6.T12.4.8.7.3" class="ltx_td ltx_nopad_r ltx_align_center">0.546</td>
</tr>
<tr id="A6.T12.4.9.8" class="ltx_tr">
<th id="A6.T12.4.9.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">14</th>
<td id="A6.T12.4.9.8.2" class="ltx_td ltx_align_center">0.648</td>
<td id="A6.T12.4.9.8.3" class="ltx_td ltx_nopad_r ltx_align_center">0.538</td>
</tr>
<tr id="A6.T12.4.10.9" class="ltx_tr">
<th id="A6.T12.4.10.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">16</th>
<td id="A6.T12.4.10.9.2" class="ltx_td ltx_align_center">0.647</td>
<td id="A6.T12.4.10.9.3" class="ltx_td ltx_nopad_r ltx_align_center">0.537</td>
</tr>
<tr id="A6.T12.4.11.10" class="ltx_tr">
<th id="A6.T12.4.11.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">18</th>
<td id="A6.T12.4.11.10.2" class="ltx_td ltx_align_center">0.637</td>
<td id="A6.T12.4.11.10.3" class="ltx_td ltx_nopad_r ltx_align_center">0.527</td>
</tr>
<tr id="A6.T12.4.12.11" class="ltx_tr">
<th id="A6.T12.4.12.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">20</th>
<td id="A6.T12.4.12.11.2" class="ltx_td ltx_align_center">0.627</td>
<td id="A6.T12.4.12.11.3" class="ltx_td ltx_nopad_r ltx_align_center">0.487</td>
</tr>
<tr id="A6.T12.4.13.12" class="ltx_tr">
<th id="A6.T12.4.13.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">22</th>
<td id="A6.T12.4.13.12.2" class="ltx_td ltx_align_center">0.591</td>
<td id="A6.T12.4.13.12.3" class="ltx_td ltx_nopad_r ltx_align_center">0.461</td>
</tr>
<tr id="A6.T12.4.14.13" class="ltx_tr">
<th id="A6.T12.4.14.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">24</th>
<td id="A6.T12.4.14.13.2" class="ltx_td ltx_align_center">0.567</td>
<td id="A6.T12.4.14.13.3" class="ltx_td ltx_nopad_r ltx_align_center">0.437</td>
</tr>
<tr id="A6.T12.4.15.14" class="ltx_tr">
<th id="A6.T12.4.15.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">26</th>
<td id="A6.T12.4.15.14.2" class="ltx_td ltx_align_center">0.548</td>
<td id="A6.T12.4.15.14.3" class="ltx_td ltx_nopad_r ltx_align_center">0.408</td>
</tr>
<tr id="A6.T12.4.16.15" class="ltx_tr">
<th id="A6.T12.4.16.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">28</th>
<td id="A6.T12.4.16.15.2" class="ltx_td ltx_align_center">0.527</td>
<td id="A6.T12.4.16.15.3" class="ltx_td ltx_nopad_r ltx_align_center">0.407</td>
</tr>
<tr id="A6.T12.4.17.16" class="ltx_tr">
<th id="A6.T12.4.17.16.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">30</th>
<td id="A6.T12.4.17.16.2" class="ltx_td ltx_align_center">0.506</td>
<td id="A6.T12.4.17.16.3" class="ltx_td ltx_nopad_r ltx_align_center">0.406</td>
</tr>
<tr id="A6.T12.4.18.17" class="ltx_tr">
<th id="A6.T12.4.18.17.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">32</th>
<td id="A6.T12.4.18.17.2" class="ltx_td ltx_align_center ltx_border_bb">0.503</td>
<td id="A6.T12.4.18.17.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb">0.403</td>
</tr>
</tbody>
</table>
</figure>
<figure id="A6.T13" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="A6.T13.2.1.1" class="ltx_text" style="font-size:90%;">Table M</span>: </span><span id="A6.T13.3.2" class="ltx_text" style="font-size:90%;">Detailed performance comparison on TruthfulQA dataset with Phi-3-Mini</span></figcaption>
<table id="A6.T13.4" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A6.T13.4.1.1" class="ltx_tr">
<th id="A6.T13.4.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">Layer</th>
<th id="A6.T13.4.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Phi-3-Mini MiniCache</th>
<th id="A6.T13.4.1.1.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt">Phi-3-Mini Mean</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A6.T13.4.2.1" class="ltx_tr">
<th id="A6.T13.4.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">0</th>
<td id="A6.T13.4.2.1.2" class="ltx_td ltx_align_center ltx_border_t">19.686</td>
<td id="A6.T13.4.2.1.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">19.465</td>
</tr>
<tr id="A6.T13.4.3.2" class="ltx_tr">
<th id="A6.T13.4.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">2</th>
<td id="A6.T13.4.3.2.2" class="ltx_td ltx_align_center">19.385</td>
<td id="A6.T13.4.3.2.3" class="ltx_td ltx_nopad_r ltx_align_center">19.365</td>
</tr>
<tr id="A6.T13.4.4.3" class="ltx_tr">
<th id="A6.T13.4.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">4</th>
<td id="A6.T13.4.4.3.2" class="ltx_td ltx_align_center">19.368</td>
<td id="A6.T13.4.4.3.3" class="ltx_td ltx_nopad_r ltx_align_center">19.221</td>
</tr>
<tr id="A6.T13.4.5.4" class="ltx_tr">
<th id="A6.T13.4.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">6</th>
<td id="A6.T13.4.5.4.2" class="ltx_td ltx_align_center">19.100</td>
<td id="A6.T13.4.5.4.3" class="ltx_td ltx_nopad_r ltx_align_center">18.255</td>
</tr>
<tr id="A6.T13.4.6.5" class="ltx_tr">
<th id="A6.T13.4.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">8</th>
<td id="A6.T13.4.6.5.2" class="ltx_td ltx_align_center">19.038</td>
<td id="A6.T13.4.6.5.3" class="ltx_td ltx_nopad_r ltx_align_center">17.019</td>
</tr>
<tr id="A6.T13.4.7.6" class="ltx_tr">
<th id="A6.T13.4.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">10</th>
<td id="A6.T13.4.7.6.2" class="ltx_td ltx_align_center">19.500</td>
<td id="A6.T13.4.7.6.3" class="ltx_td ltx_nopad_r ltx_align_center">15.912</td>
</tr>
<tr id="A6.T13.4.8.7" class="ltx_tr">
<th id="A6.T13.4.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">12</th>
<td id="A6.T13.4.8.7.2" class="ltx_td ltx_align_center">19.216</td>
<td id="A6.T13.4.8.7.3" class="ltx_td ltx_nopad_r ltx_align_center">15.525</td>
</tr>
<tr id="A6.T13.4.9.8" class="ltx_tr">
<th id="A6.T13.4.9.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">14</th>
<td id="A6.T13.4.9.8.2" class="ltx_td ltx_align_center">20.026</td>
<td id="A6.T13.4.9.8.3" class="ltx_td ltx_nopad_r ltx_align_center">15.195</td>
</tr>
<tr id="A6.T13.4.10.9" class="ltx_tr">
<th id="A6.T13.4.10.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">16</th>
<td id="A6.T13.4.10.9.2" class="ltx_td ltx_align_center">19.641</td>
<td id="A6.T13.4.10.9.3" class="ltx_td ltx_nopad_r ltx_align_center">15.058</td>
</tr>
<tr id="A6.T13.4.11.10" class="ltx_tr">
<th id="A6.T13.4.11.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">18</th>
<td id="A6.T13.4.11.10.2" class="ltx_td ltx_align_center">18.756</td>
<td id="A6.T13.4.11.10.3" class="ltx_td ltx_nopad_r ltx_align_center">14.763</td>
</tr>
<tr id="A6.T13.4.12.11" class="ltx_tr">
<th id="A6.T13.4.12.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">20</th>
<td id="A6.T13.4.12.11.2" class="ltx_td ltx_align_center">17.738</td>
<td id="A6.T13.4.12.11.3" class="ltx_td ltx_nopad_r ltx_align_center">14.593</td>
</tr>
<tr id="A6.T13.4.13.12" class="ltx_tr">
<th id="A6.T13.4.13.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">22</th>
<td id="A6.T13.4.13.12.2" class="ltx_td ltx_align_center">19.546</td>
<td id="A6.T13.4.13.12.3" class="ltx_td ltx_nopad_r ltx_align_center">14.182</td>
</tr>
<tr id="A6.T13.4.14.13" class="ltx_tr">
<th id="A6.T13.4.14.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">24</th>
<td id="A6.T13.4.14.13.2" class="ltx_td ltx_align_center">19.723</td>
<td id="A6.T13.4.14.13.3" class="ltx_td ltx_nopad_r ltx_align_center">13.954</td>
</tr>
<tr id="A6.T13.4.15.14" class="ltx_tr">
<th id="A6.T13.4.15.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">26</th>
<td id="A6.T13.4.15.14.2" class="ltx_td ltx_align_center">18.366</td>
<td id="A6.T13.4.15.14.3" class="ltx_td ltx_nopad_r ltx_align_center">13.919</td>
</tr>
<tr id="A6.T13.4.16.15" class="ltx_tr">
<th id="A6.T13.4.16.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">28</th>
<td id="A6.T13.4.16.15.2" class="ltx_td ltx_align_center">18.402</td>
<td id="A6.T13.4.16.15.3" class="ltx_td ltx_nopad_r ltx_align_center">12.231</td>
</tr>
<tr id="A6.T13.4.17.16" class="ltx_tr">
<th id="A6.T13.4.17.16.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">30</th>
<td id="A6.T13.4.17.16.2" class="ltx_td ltx_align_center">16.827</td>
<td id="A6.T13.4.17.16.3" class="ltx_td ltx_nopad_r ltx_align_center">12.158</td>
</tr>
<tr id="A6.T13.4.18.17" class="ltx_tr">
<th id="A6.T13.4.18.17.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">32</th>
<td id="A6.T13.4.18.17.2" class="ltx_td ltx_align_center ltx_border_bb">16.635</td>
<td id="A6.T13.4.18.17.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb">10.333</td>
</tr>
</tbody>
</table>
</figure>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2405.14365" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2405.14366" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2405.14366">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2405.14366" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2405.14367" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Wed Jun  5 15:10:19 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2405.14366
A critical approach for efficiently deploying computationally demanding large language models (LLMs)  is Key-Value (KV)  caching. The KV cache stores key-value states of previously generated tokens, significantly reducing the need for repetitive computations and thereby lowering latency in autoregressive generation.
However, the size of the KV cache grows linearly with sequence length, posing challenges for applications requiring long context input and extensive sequence generation.
In this paper, we present a simple yet effective approach, called MiniCache, to compress the KV cache across layers from a novel depth perspective,
significantly reducing the memory footprint for LLM inference.
Our approach is based on the observation that KV cache states exhibit high similarity
between the adjacent layers
in the middle-to-deep portion of LLMs.
To facilitate merging, we propose disentangling the states into the magnitude and direction components, interpolating the directions of the state vectors while preserving their lengths unchanged. Furthermore, we introduce a token retention strategy to keep highly distinct state pairs unmerged, thus preserving the information with minimal additional storage overhead.
Our MiniCache is training-free and general, complementing existing KV cache compression strategies, such as quantization and sparsity.
We conduct a comprehensive evaluation of MiniCache utilizing various models including LLaMA-2, LLaMA-3, Phi-3, Mistral, and Mixtral across multiple benchmarks, demonstrating its exceptional performance in achieving superior compression ratios and high throughput.
On the ShareGPT dataset, LLaMA-2-7B with 4-bit MiniCache achieves a remarkable compression ratio of up to, enhances inference throughput by approximately, and reduces the memory footprint bycompared to the FP16 full cache baseline, all while maintaining near-lossless performance.
abstract
MiniCache:
KV Cache Compression in Depth Dimension for Large Language Models
Abstract
2405.14366
1IntroductionLarge Language Models (LLMs) , exemplified by the GPT seriesand the LLaMA series, have emerged as pivotal innovations within the artificial general intelligence, notably enhancing the capabilities of natural language processing. However, these models are meticulously trained using extensive computational resourcesand massive datasets, which enables them to produce text that effectively mimics human writing styles and conducts complex reasoning analysis, yet raises the challenge of efficient deployment and serving.
Within the inference framework of LLMs, KV cachesare crucial for storing pre-computed keys and values, thus avoiding repeated calculation over the preceding context to substantially enhance the deployment efficiency of LLMs.
However, the increasing demand for longer sequence lengths results in cached states becoming voluminous, leading to significant memory consumption during generation.
For instance, a 175B GPT-3 model, with a batch size of 64 and a sequence length of 4,096 tokens (both prefilled and generated) , requires approximately 1,208GB of GPU memory.
This requirement isgreater than the memory used to store the model’s weights.
In this context, KV cache compression is of paramount importance due to its clear benefits: 1)  largely reduces the memory footprint allowing for faster generation and larger batch serving; 2)  significantly lowers the cost per token, demonstrating substantial commercial benefits.Figure 1:Overview of our MiniCache strategy and example results:
(a)  shows the observation that the KV cache states between two adjacent layers are highly similar, particularly across the middle to deep layers. The x-axis uses index/2 to represent the similarities for each pair of layers. (b)  compares the performance of MiniCache, and the mean baseline, which simply averages the KV caches of two layers, using the LLaMA-3-70B modelon the GSM8K dataset. MiniCache, which begins merging from the half-layer depth, achieves near-lossless performance.
(c)  highlights the primary difference between MiniCache and previous approaches. MiniCache investigates the inter-layer redundancy of KV caches along the depth dimension of LLMs, an aspect previously overlooked by intra-layer-based methods. Here,refers to the last timestamp of pre-filling, anddenotes to the first timestamp of decoding.Existing KV cache compression efforts can be roughly categorized into two types, namely quantization and sparsity. The quantization approachespropose storing the KV states in low-bit numerical values. Typically, FlexGendemonstrates that 4-bit KV cache quantization can achieve lossless performance.
In contrast, sparsity-driven methods aim to retain only the salient tokens while evicting the rest, either heuristicallyor adaptively.
Some approachesexplore the intersection of these two types, by assigning high-bit to salient tokens and extremely low-bit to the rest of the tokens, achieving more aggressive memory gain.
Despite these innovations,existing literature merely consider theintra-layer redundancy, while neglecting another important complementary direction – theinter-layer redundancy, this motivation difference is illustrated in theFigure1(c) .Our analysis begins by exploring the redundancy of KV cachesalong the depth dimension, as shown in Figure1(a) .
We observe that KV cache states exhibit high similarity between neighbouring layers in the middle-to-deep portions of LLMs. This intriguing property suggests that states paired by position between adjacent layers can be accurately merged into a single state space with a strong performance guarantee, as illustrated in Figure1(b) . This approach significantly reduces the memory footprint without the need to retain individual states for each attention layer.
Note that these observations are pertinent to
dynamic inference strategies such as mixture-of-depthsand layer-wise early exiting, which optimize computational paths by skipping non-critical layers
to enhance training and inference efficiency. Furthermore, layer pruning methodshighlight considerable redundancy in deeper layers. However, despite these advancements,
the redundancy of KV caches along the depth dimension
has largely been overlooked.In this paper, we proposeMiniCache, a simple yet effective cross-layer KV cache compression method aimed at advancing the inference efficiency of LLMs.
MiniCache consists of two essential components.
Firstly, we introduce an accurate cache merging strategy, employing a reparameterization of state vectors that decompose them into the magnitude and direction components, akin to weight normalization. This approach allows for effective interpolation of the directional component in polar coordinates while preserving the original state norms to retain as much information as possible. This interpolation refers to the cross-layer merging as shown in theFigure1(c) .
Secondly, we recognize that a small
subset of state pairs, characterized by low similarities but carrying largely distinct semantic meanings, are unsuitable for inter-layer merging.
To address this, we propose a token retention strategy to minimize performance degradation, which involves separately retaining these outlier pairs.
Our framework is notably memory-efficient, requiring storage for only a single high-dimensional directional component, along with minimal extra memory overhead. The overhead consists of a few unmergeable tokens and their corresponding indexes, as well as token-wise magnitudes to accurately restore the original states.We conduct extensive experiments with representative LLMs,
including
Mixtral-8x7B, Phi-3-Mini, and LLaMA-38B and 70B, respectively. Our method is benchmarked across a diverse range of question answering and generation datasetsusing the lm-eval-harness. Additionally, we evaluate our results on LongBenchfor long-sequence generation. The results demonstrate that MiniCache can reduce the memory footprint required for LLM inference by up to 41%, while simultaneously enhancing throughput by approximatelycompared to fully cached baseline, clearly surpassing existing methods.Our contributions are summarized as follows:•We introduce MiniCache, a simple yet highly effective framework for KV cache compression. MiniCache pioneers the exploration of KV cache compression along the depth dimension, thereby significantly expanding its capabilities.•We observe a fascinating characteristic of cross-layer KV cache states: they exhibit high similarity between adjacent layers in the middle to later stages of LLMs.
Additionally, we find that not all state pairs are suitable for merging.•We propose an accurate and memory-efficient method for cross-layer cache merging, comprising a reparameterization strategy and a token retention mechanism.
Our method complements existing KV cache compression approaches, further enhancing LLM serving efficiency.•Our MiniCache performs favourably against the state-of-the-art methods. Notably, our 4-bit MiniCache achieves a strong compression ratio of up to,higher inference throughput, and 41% memory reduction compared to the FP16 full cache baseline with near-lossless performance.
introduction
MiniCache:
KV Cache Compression in Depth Dimension for Large Language Models
1Introduction
2405.14366
2Related WorkEfficient inference for LLMs.Large Language Models (LLMs)  are constrained by considerable computational and memory requirements during inference, particularly in resource-constrained environments. To mitigate these challenges, a variety of efficient inference techniques have been developed. For instance, dynamic inference methods, represented by mixture-of-experts (MoE) , adaptively select specific sub-structures of the model during the inference process based on the input data, significantly improving inference efficiency while keeping model capacity. Techniques like Multi-Query Attention, Kernel-driven attentions, and low-rank attentionsapproximate the functionality of traditional attention mechanisms but with more efficient implementations. Quantization strategiesinvolve converting the model’s weights and activations into a low bit-width format, thereby reducing memory footprint and computational intensity. Sparsification approacheseliminate unnecessary elements in both model weights and token representations, further enhancing efficiency. Some closely related works, such as MoDand LayerSkips, considered the dynamic inference nature to ignore unimportant layers according to input; however, these methods require an additional fine-tuning process or carefully designed pre-training stages, which reduces the adaptivity of these methods. MiniCache relies on inter-layer similarity observations to perform cross-layer merging, significantly reducing memory demand.Model merging.Merging compression involves the aggregation of a model’s parameters and activations at various granularities.
This process enhances the efficiency of inference in large models and facilitates huge redundancy. Linear Mode Connectivity (LMC) enables the fine-tuning of models from a shared pre-trained base.
Commonly, weight averagingis employed as an efficient technique to perform merge compression; notably, Model Souputilizes linear averaging in this context. Advanced methods like TIES Merging, Model Breadcrumbs, and DAREfurther enhance this process by sparsifying and combining
model parameters, enabling the merging of models without sacrificing performance capabilities. Additionally, Spherical Linear intERPolation (SLERP) extends beyond simple weight averaging by interpolating between model parameters. The Fisher information matrixand RegMean-based methodsfurther optimize merges to produce ideal weights with respect to thedistance to generation outputs, while maintaining the privacy of training data. However, most existing works focus on merging model parameters, with the concept of depth-wise mergeability not being thoroughly explored in prior research. MiniCache focuses on the KV cache token merging in the depth dimensional of LLMs.
body
MiniCache:
KV Cache Compression in Depth Dimension for Large Language Models
2Related Work
2405.14366
3MotivationIn the below, we present our new observations in a novel cross-layer perspective.(a) Simple average baselinevs.full cache on GSM8K(b) Pairwise similarity in adjacent layers KV cache(c) Benchmark on five QA datasetsFigure 2:Overall of our explorations and observations : (a)  shows the strong baseline by performing average merging on the KV cache. (b)  shows the pairwise similarity of cache states by position between adjacent layers. (c)  compares between the MiniCache, simple average, and full cach baseline on five different datasets.3.1Cross-Layer Redundancy in KV CachePrior studies have revealed the ineffectiveness of middle-to-deep layers in LLMs. Thus, layer-wise early exiting in this case can effectively avoid the redundant computation with minor effect on LLM performance.
Inspired by this, we explore alayer-wisemerging of KV cache in LLMs, starting with a simple baseline by averaging similar tokens across adjacent layers. We provide our key observations as follows.Observation 1: KV cache shares a high similarity between adjacent layers.Based on LLaMA-3-70B, we conduct zero-shot inference on the validation sets of three widely recognized benchmarks: COQA, GSM8Kand TruthfulQA. In general, we find that KV cache in the shallow layers exhibits low similarity, whereas those in the middle-to-deep layers are very similar to one another based on angular distance, as shown inFigure1(b) .
Next, we merge the KV cache across adjacent layers by conducting five-shot inference with LLaMA-2-7B, LLaMA-2-13B, LLaMA-3-8B, and Mixtral-8x7Bon GSM8K. Specifically, starting from the middle layer of each model, we merge the KV cache in the adjacent two layers.
As shown inFigure2(a) , we observe a favourable performance for different LLMs,
which reveals the huge potential for efficiency improvement by sharing the KV cache across adjacent layers during LLM decoding.Observation 2:
Not all tokens are equally important to merge, a few distinct pairs require retention.Recent worksin KV cache compression have found that keeping a few salient tokens at each layer, which contribute the majority of attention scores, could be sufficient to maintain LLM performance.
In our case, we speculate that certain token pairs by position in adjacent layers
also exist outlier behaviours, showing strong semantic differences that make them unsuitable for merging.
Based on COQAand LLaMA-2-7B, we investigate the similarities at the level of token pairs.
As shown inFigure2(b) ,
we find a significant portion of token pairs share high similarities across adjacent layers.
However, we observe a few outlier pairs, such as index 0 and index 15, which have large margins of difference. We consider these tokens non-mergeable due to their significant differences. We also show that merging these distinct tokens results in performance degradation, corresponding torow inTable2.
Thus, while cross-layer merging is a promising strategy for reducing memory burdens, it must be implemented with careful consideration of token-level similarities to ensure optimal performance, as shown inFigure2(c) .
body
MiniCache:
KV Cache Compression in Depth Dimension for Large Language Models
3Motivation
2405.14366
4MethodIn this section, we introduce our MiniCache, a simple yet effective
method aimed at trimming the KV cache redundancy in the depth dimension. This framework exploits the high similarity of KV cache states between adjacent layers
and consists of two primary components: a reparameterization-based merging strategy
and a token retention mechanism.
The merging strategy compresses the KV cache states in adjacent layers to aggregate them into a single shared memory space, beginning from the middle of the model.
The token retention mechanism mitigates information lost by retaining the highly distinct state pairs with minimal additional memory cost. With the merged cache, retention tokens, and magnitudes, we can accurately restore the original cache states for token decoding.4.1Cross-Layer CompressionOur method commences with the identification of an optimal starting layer.
Observations inSection3.1indicate that the KV cache from middle-to-deep layers exhibits explicit patterns characterized by high similarity across adjacent layers. Consequently, we select the starting layer from the middle of the LLM, specifically.
From this layer onward, the KV pairs are presumed to be sufficiently similar across adjacent layers to warrant their consolidation. Central to this approach is a merge function,, which is designed to integrate the KV caches
of consecutive layers into a single, unified cache.
We defineas the vectorized cache state of a single token, where the superscript indicates the layer index and the subscriptsanddenote the keys and values, respectively.
Specifically, for a pair of key/value tokens at the same position in layersand, the merged cache is computed as(1) This consolidation process effectively eliminates the need to store and process the original memory-intensive keys and values in each layer independently. Instead, it approximates a shared cache across the adjacent layers.Figure 3:The illustration of the proposed methodMiniCache. (a)  depicts the cross-layer compression process. We fetch the KV caches, from layersand, and merge them into shared states via Eq. (3) . Additionally, we compute thenorm for the caches to obtain their magnitudes. Furthermore, we select unmergable tokens for retention, then store merged cache, retention tokens, and magnitudes at layerin. (b)  demonstrates the restoration process for layersand, which includes magnitude rescaling in Eq. (2)  and retention token recovery.4.2KV Cache Merging and RestorationReparameterization-based cache merging.To perform the pairwise merging, one solution is to directly average a pair of KV tokens, analogous to model merging. However, we observe that direct averaging can cause significant information loss. We conjecture that the distance between activations can be larger than that of weights due to the presence of outlier activation channels with extremely large magnitudes in LLMs, while weights typically have relatively quite small magnitudes.
A potential method to compensate for this information loss is to project fromtoand, then rescale the projected vectors based on their relative magnitudes to exactly restore the original states. However, this approach requires extensive additional storage and computations; for example, restoringneeds bothand, which undermines the benefits of cache merging.
To efficiently merge token pairs, we draw inspiration from weight normalization, which disentangles model parameters into themagnitudeanddirectioncomponents to accelerate the convergence of stochastic gradient descent. Additionally, we take cues from DoRA, which employs a similar way to resemble the learning behavior of parameter-efficient fine-tuning compared to full fine-tuning.
In our case, the reparameterization can be formulated as follows:(2) whereis the directional vector. This decomposition ensures thatis a unit vector, and allows the restored states to match thenorm of the original states, thereby preserving the cache’s information as much as possible, the restoration is shown asFigure3(b) .
For brevity, we omit the subscriptsand, as keys and values are decomposed in the same way.
For estimating the directional component, we follow SLERP, which adaptively handles the interpolation, which often resembles rotation-like transformations.
The choice of SLERP as the merging function is strategic, as it facilitates interpolation along the shortest path on the unit sphere between two high-dimensional vectors, thereby preserving their geometric integrity, which refers to merge operation inFigure3(a) . This is crucial for maintaining the semantic and syntactic properties of the KV caches. The formula for SLERP in our context is:(3) whererepresents the angle between vectorsand, andis thefunction.is an interpolation hyperparameter
that adjusts the relative influence of each vector on the resulting direction, tailored to the layer depth and specific characteristics of the KV pairs.
Note that when we set, it will become an average merging, which we consider special cases in Eq. (A) .
The merged cache for each token pair is then a concatenation of the directional vector, magnitude and, denoting as, cached components as shown inFigure3(a) .
Note that apart from storing the merged directional vector, we only need to store additional token-wise magnitude and angle scalars, which is memory efficient.
In this way,
we achieve substantial
memory efficiencies through reduced redundancy while ensuring the retention of the critical functional characteristics of the original KV pairs across transformer layers.Unmergeable token retention.Highly distinct pairs are sensitive to merging operations, leading us to propose unmergeable token retention, as shown inFigure3(a) . Despite the high similarity between KV cache states across neighbouring layers, a few sensitive distinct pairs remain that are significantly difficult to merge and share. Aligned with previous studies, these distinct tokens carry substantial semantic meanings.
We observe that merging sensitive tokens, which preserve layer-specific information, can lead to significant performance degradation. Therefore, it is crucial to properly disentangle the shared and unique information between adjacent layers. To address this issue, we designed a token retention strategy to selectively retain tokens that cannot be merged based on their angular distance, defined as:.
For the KV caches, the minimum and maximum angular distances are determined to identify the unmergeable tokens.The set of required token indices to keep,, is obtained by:(4) whereis a predefined hyperparameter that controls the retention threshold.
The tokens with indices inare retained and not compressed during the merge, which ensures that performance does not decline by preventing the loss of unmergeable tokens.Next, letbe either the key or value cache at one attention layer, wheredenotes the number of tokens andis the number of hidden dimensions, andbe the shared KV cache states.
Then, for each pair of neighbouring two layers, the unmergeable tokens are selected along with the token dimension by,, then restoring to our compressed caches by,, as shown inFigure3(b) .
Overall, we share the final cache for the two layers as. This cache includes the shared KV cache states, retention of unmerged tokens, magnitude vectors for each layer, and the token-keeping index, respectively.
These additional components are quite lightweight, thus compared to full-layer caches, our method remains memory-efficient, as discussed in Sec.4.3.Cache restoration.After obtaining the shared cache, we further need to approximately restore the original cache states for the current token decoding, as shown in Fig.3(b) . Specifically, to restore, we first rescale the directional shared states with the corresponding magnitude vector along the token dimension, denoted as. Subsequently, we perform retention token recovery by placing the sensitive tokens according to their token indices.4.3Efficiency DiscussionCompression efficiency.We primarily analyze our memory efficiency in terms of the number of tokens used. Next, letbe the number of layers and
andis the batch size,andare input and output sequence length respectively. We consider the FP16 storage for KV cache. The full cache memory usage is given by.
In our study, we begin merging layers from the middle to the deeper layers, consolidating the KV cache states of every two layers into a single shared state space. As a result, we effectively reduce the GPU memory usage in the decoding inference to, demonstrating a significant compression rate.Figure 4:Performance comparisons between our proposed MiniCache with the “averaging baseline” and the “unmerged full cache baseline” on multiple datasets with Phi3-Mini, Mixtral-8x7B, LLaMA-3-8B, and LLaMA-3-70B. More result details are shown inAppendixF. The x-axis indicates the number of layers merged. As more layers are merged, a greater reduction in memory usage is achieved.Restoration efficiency.We then analyze the additional memory cost incurred during the restoration process, which
During the magnitude rescaling phase, we save an additional norm vector for the corresponding layers in the KV cache. It is important to note that the norm vector is in the shape of, which has a single channel dimension compared to the fully ranked original KV states. Additionally, we suggest that the retention threshold can be set to 0.05. Therefore, we havetokens retained without compression. Finally, our overall memory requirement is given by. The detailed derivation is shown in theAppendixE.
body
MiniCache:
KV Cache Compression in Depth Dimension for Large Language Models
4Method
2405.14366
5ExperimentsWe demonstrated that our MiniCache can perform merging compression on the latter half of the layers of LLMs with minimal performance degradation.Implementation details.Our experiments are based on representative model families of LLMs, including a compact LLM Phi-3-Miniand an MoE LLM Mixtral-8x7B. Additionally, we adopt LLaMA-38B and 70B models to explore how our method generalizes to larger LLMs. We sample ten tasks from lm-eval-harness, including COPA, MathQA, OpenBookQA, PiQA, RTE, Winogrande, XSUM, and CNN/Daily Mail. We also evaluate long-sequence generation on LongBench. We compare ours with fully cached baseline, and other methods such as round-to-nearest quantization (RTN) , SmoothQuantand KIVI.For the proposed MiniCache, we set the interpolation parameterto 0.6, indicating that the merged results have a smaller rotation angle to the next layer. Furthermore, we set the token retention thresholdto 0.05, according to the statistics of unmergeable tokens across multiple datasets.
In addition to our merging method, we also consider a strong baseline of average merging. For sequential loading of large models, we utilize NVIDIA 4 A100 80GB GPUs, more details refers toAppendixD.Main results.We evaluate MiniCache by merging KV caches across all layers on GSM8K, COQA, and TruthfulQA. The results are shown inFigure4. In general, we demonstrate the general effectiveness of merging KV caches from middle-to-deep layers across different sized LLMs. Moreover, the proposed MiniCache demonstrates a consistent and significant advantage over the averaging baseline.
We also illustrate the performance of merging KV caches across half of the layers with the blue lines, where MiniCache still maintains a robust performance and achieves the best compression ratio.
Besides, we find that our method is even more effective for larger LLMs. For instance, based on LLaMA-3-70B, MiniCache shows nearly zero performance drop even with the KV cache in 87.5% of the layers
merged on the COQA dataset. This highlights the adaptability and efficiency of our approach in handling large-scale models while ensuring minimal performance degradation.Table 1:Evaluation of different KV cache compression methods on LongBench.
MiniCache builds on top of 4-bit KIVIand achieves the best performance with the strongest compression rate.ModelMethodLCCRepoBench-PPR-enTREC2wikimqaGovReportMQA-zhAverageCompressionRatioLlama-2-7B-ChatBaseline58.1652.1910.1264.0031.1227.0910.1236.411xRTN15.448.760.794.000.301.930.074.903.21xSmoothQuant35.3132.180.7928.757.4511.831.6816.282.15xKIVI-249.3243.714.5063.0024.0724.7310.2431.513.95xMiniCache58.0352.019.0064.0030.5825.3210.1335.445.02xLlama-2-13B-ChatBaseline48.0650.0814.2568.5013.0927.767.2332.711xRTN20.8918.620.330.000.521.680.166.033.21xSmoothQuant32.1733.862.6548.003.5312.470.4719.162.15xKIVI-248.6048.8113.5068.0014.3225.707.0132.423.95xMiniCache48.7548.5913.0068.0014.3626.577.9932.615.02xMistral-7BBaseline68.0660.4617.7168.0010.8720.0917.1037.331xRTN27.9826.183.3413.001.112.490.4510.513.21xSmoothQuant40.6335.143.4030.506.035.004.1217.552.15xKIVI-265.1658.3312.4365.0011.0313.2213.8733.433.95xMiniCache68.8960.9813.9267.0010.5018.067.8835.755.02xMistral-7B-InstructBaseline55.5148.9660.0071.0027.3332.8542.7448.321xRTN32.3633.230.671.002.2510.032.3011.553.21xSmoothQuant43.8438.634.7939.5010.3423.618.3324.432.15xKIVI-253.1348.6047.5069.0020.6829.3733.8843.743.95xMiniCache54.7951.0264.1471.0024.9731.4627.5446.995.02xLongBench.We also conducted experiments to evaluate performance and quality in long-sequence generation using the LongBench dataset, shown inTable1. Our experiments applied MiniCache over several models: LLaMA-2-7B-Chat, LLaMA-2-13B-Chat, Mistral-7B, and Mistral-7B-Instruct.
It is important to note that our MiniCache method maintains orthogonality with all existing quantization and sparsity (refers toTableA)  methods at both the model and token-wise levels.
When combined with KIVI-4bit KV cache quantization, our approach achieved a compression rate of, with minimal impact on accuracy across various challenging long-context generation tasks.
The combination of MiniCache and KIVI-4bit KV cache quantization demonstrates significant memory savings without compromising the model’s ability to handle long sequences effectively. This highlights the potential of our method to optimize large language models for tasks requiring extensive context, making them more efficient and scalable for real-world applications.Efficiency analysis.To assess the acceleration capabilities of MiniCache, we conduct evaluations based on the methodologies employed in vLLMand KIVI. We generate synthetic workloads derived from ShareGPT, which include real input and output texts from LLM services. The dataset features an average input prompt length of 161 tokens and an output length of 338 tokens. Using the LLaMA-2-7B model on a single 80GB NVIDIA A100 GPU, we benchmark our method in a batch-serving scenario, comparing peak memory usage and throughput among 2-bit KIVI, 4-bit MiniCache, and an FP16 baseline. As illustrated inFigure6, with a batch size of 128,MiniCache reduces memory usage by 25GB, achieving a 41% memory saving. In terms of throughput,MiniCache outperforms the FP16 baseline by approximately 5. Additionally, despite utilizing 4-bit quantization, MiniCache benefits from merging and sharing KV caches across adjacent layers, resulting in a 1.29higher throughput compared to the 2-bit KIVI. These results demonstrate that MiniCache offers a state-of-the-art trade-off between efficiency and performance.(a) BS.vs.Peak Memory Usage(b) BS.vs.Decoding ThroughputFigure 5:Memory usage and throughput comparison between our 4-bit MiniCache, 2-bit KIVI, and 16-bit Baseline. MiniCache can achieve higher throughput by enabling a larger batch size while reducing memory footprints via LLaMA-2-7B.Figure 6:LLaMA-3-8Bto experiment on the GSM8K.
The right axis is the normalized frequency of the relative magnitude ratio. Optionalshows a strong correlation with frequency.
body
MiniCache:
KV Cache Compression in Depth Dimension for Large Language Models
5Experiments
2405.14366
6Ablation StudyCOQAGSM8KTruthfulQA00.6030.10829.8130.010.6200.12630.2260.020.6300.14333.9030.050.6470.15233.2130.10.6430.15233.90310.6430.15933.743Table 2:Comparison of token retention thresholdby LLaMA-2-7Bon three benchmarks.The effect of interpretation parameter.In our exploration of the effects of the interpretation parameteron performance, particularly in relation to the relative magnitude ratio of adjacent layers,
as shown inFigure6. We maintain all settings constant, starting from layer(halfway through the layers of LLaMA-3-8B) , and vary the interpretation parameterfrom 0.3 to 0.7. Our findings reveal several key points:
When, the process resembles average merging, which is less effective for cross-layer merging.
In contrast, whenis optimal, the merged representation exhibits the most robust performance, while indicating that more information is derived from the second term ()  of the SLERP.The frequency results also indicate that the high frequencies are clustered around 0.4 and 0.6, corroborating our optimal. Moreover, there is a strong correlation between the optimaland the high frequency of the relative magnitude ratio of adjacent layers. This finding provides an opportunity to utilize the relative magnitude ratio to dynamically determine the interpretation parameter.
Dynamicallows for more flexible weight control in SLERP merging for each layer-wise operation, thereby showing potential for further exploration.The effect of token retention threshold.We investigate the impact of the token retention thresholdon model performance across the three datasets, as shown inTable2.
A largergenerally means retaining more tokens for improved performance, but this comes at the cost of increased memory demand.
The results suggest that settingto 0.05 achieves the best balance between performance and efficiency.
body
MiniCache:
KV Cache Compression in Depth Dimension for Large Language Models
6Ablation Study
2405.14366
7Conclusion and Future WorkThis paper presents a pioneering exploration of KV cache compression in the depth dimension, addressing a significant memory bottleneck in LLMs. Our proposed MiniCache offers a simple, effective, and training-free approach to compressing KV caches by leveraging the notable high similarities between KV caches in neighboring layers, starting from the midpoint of LLMs.
We have demonstrated that MiniCache can significantly reduce the memory footprint required for LLM inference by up to 41%, while simultaneously enhancing throughput by approximately five times compared to the FP16 baseline.
In conclusion, MiniCache
significantly advances the field of KV cache compression,
offering a state-of-the-art balance between efficiency and performance. Future work will focus on enhancing the compression ratio by cross-multiple-layer merging, developing advanced merging algorithms such as Spherical Cubic Interpolation, and further optimizing memory usage for large-scale deployments in diverse application scenarios.
conclusion
MiniCache:
KV Cache Compression in Depth Dimension for Large Language Models
7Conclusion and Future Work