<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2407.21018] ThinK: Thinner Key Cache by Query-Driven Pruning</title><meta property="og:description" content="Large Language Models (LLMs) have revolutionized the field of natural language processing, achieving unprecedented performance across a variety of applications by leveraging increased model sizes and sequence lengths.
…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="ThinK: Thinner Key Cache by Query-Driven Pruning">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="ThinK: Thinner Key Cache by Query-Driven Pruning">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2407.21018">

<!--Generated on Mon Aug  5 17:29:37 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<div id="p1" class="ltx_para">
<span id="p1.1" class="ltx_ERROR undefined">\pdfcolInitStack</span>
<p id="p1.2" class="ltx_p">tcb@breakable</p>
</div>
<h1 class="ltx_title ltx_title_document">
<span id="id12.id1" class="ltx_text ltx_font_bold">ThinK:</span> <span id="id13.id2" class="ltx_text ltx_framed ltx_framed_underline">Thin</span>ner <span id="id14.id3" class="ltx_text ltx_framed ltx_framed_underline">K</span>ey Cache by Query-Driven Pruning </h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Yuhui Xu<sup id="id15.12.id1" class="ltx_sup">1</sup>   Zhanming Jie<sup id="id16.13.id2" class="ltx_sup"><span id="id16.13.id2.1" class="ltx_text ltx_font_italic">1</span></sup>   Hanze Dong<sup id="id17.14.id3" class="ltx_sup">1</sup>   Lei Wang<sup id="id18.15.id4" class="ltx_sup">1</sup>
<br class="ltx_break">
<br class="ltx_break">Xudong Lu<sup id="id19.16.id5" class="ltx_sup">2</sup>  Aojun Zhou<sup id="id20.17.id6" class="ltx_sup">2</sup>   Amrita Saha<sup id="id21.18.id7" class="ltx_sup">1</sup>   Caiming Xiong<sup id="id22.19.id8" class="ltx_sup">1</sup>  Doyen Sahoo<sup id="id23.20.id9" class="ltx_sup">1</sup>

<br class="ltx_break">
<br class="ltx_break"><sup id="id24.21.id10" class="ltx_sup">1</sup>Salesforce AI Research   <sup id="id25.22.id11" class="ltx_sup">2</sup> The Chinese University of Hong Kong
</span><span class="ltx_author_notes">Email: <span id="id26.23.id1" class="ltx_text ltx_font_typewriter">{yuhui.xu, ajie, hanze.dong, leiwang, amrita.saha, cxiong, dsahoo}@salesforce.com</span></span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id27.id1" class="ltx_p">Large Language Models (LLMs) have revolutionized the field of natural language processing, achieving unprecedented performance across a variety of applications by leveraging increased model sizes and sequence lengths.
However, the associated rise in computational and memory costs poses significant challenges, particularly in managing long sequences due to the quadratic complexity of the transformer attention mechanism.
This paper focuses on the long-context scenario, addressing the inefficiencies in KV cache memory consumption during inference.
Unlike existing approaches that optimize the memory based on the sequence lengths, we uncover that the channel dimension of the KV cache exhibits significant redundancy, characterized by unbalanced magnitude distribution and low-rank structure in attention weights.
Based on these observations, we propose ThinK, a novel query-dependent KV cache pruning method designed to minimize attention weight loss while selectively pruning the least significant channels. Our approach not only maintains or enhances model accuracy but also achieves a reduction in memory costs by over 20% compared with vanilla KV cache eviction methods. Extensive evaluations on the LLaMA3 and Mistral models across various long-sequence datasets confirm the efficacy of ThinK, setting a new precedent for efficient LLM deployment without compromising performance. We also outline the potential of extending our method to value cache pruning, demonstrating ThinK’s versatility and broad applicability in reducing both memory and computational overheads.</p>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
<nav class="ltx_TOC ltx_list_toc ltx_toc_toc"><h6 class="ltx_title ltx_title_contents">Contents</h6>
<ol class="ltx_toclist">
<li class="ltx_tocentry ltx_tocentry_section"><a href="#S1" title="In ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">1 </span>Introduction</span></a></li>
<li class="ltx_tocentry ltx_tocentry_section"><a href="#S2" title="In ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">2 </span>Observations</span></a></li>
<li class="ltx_tocentry ltx_tocentry_section">
<a href="#S3" title="In ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3 </span>ThinK</span></a>
<ol class="ltx_toclist ltx_toclist_section">
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#S3.SS1" title="In 3 ThinK ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3.1 </span>Preliminary Study of KV Cache Optimization</span></a></li>
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#S3.SS2" title="In 3 ThinK ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3.2 </span>Query-Driven Pruning</span></a></li>
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#S3.SS3" title="In 3 ThinK ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3.3 </span>Implementations</span></a></li>
</ol>
</li>
<li class="ltx_tocentry ltx_tocentry_section">
<a href="#S4" title="In ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4 </span>Experiment Results</span></a>
<ol class="ltx_toclist ltx_toclist_section">
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#S4.SS1" title="In 4 Experiment Results ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4.1 </span>Settings</span></a></li>
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#S4.SS2" title="In 4 Experiment Results ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4.2 </span>Results on LongBench</span></a></li>
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#S4.SS3" title="In 4 Experiment Results ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4.3 </span>Results on Needle-in-a-Haystack</span></a></li>
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#S4.SS4" title="In 4 Experiment Results ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4.4 </span>Ablation Studies</span></a></li>
</ol>
</li>
<li class="ltx_tocentry ltx_tocentry_section"><a href="#S5" title="In ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">5 </span>Related Work</span></a></li>
<li class="ltx_tocentry ltx_tocentry_section"><a href="#S6" title="In ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">6 </span>End Note and Future Direction</span></a></li>
<li class="ltx_tocentry ltx_tocentry_appendix">
<a href="#A1" title="In ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">A </span>Additional Experimental Details</span></a>
<ol class="ltx_toclist ltx_toclist_appendix">
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#A1.SS1" title="In Appendix A Additional Experimental Details ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">A.1 </span>Value Cache Pruning</span></a></li>
</ol>
</li>
</ol></nav>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para ltx_noindent">
<p id="S1.p1.1" class="ltx_p">Large language models (LLMs) <cite class="ltx_cite ltx_citemacro_citep">(Hadi et al.,, <a href="#bib.bib19" title="" class="ltx_ref">2023</a>; Brown et al.,, <a href="#bib.bib7" title="" class="ltx_ref">2020</a>; OpenAI,, <a href="#bib.bib36" title="" class="ltx_ref">2023</a>; <a href="#bib.bib41" title="" class="ltx_ref">Touvron et al., 2023a, </a>; <a href="#bib.bib42" title="" class="ltx_ref">Touvron et al., 2023b, </a>; Scao et al.,, <a href="#bib.bib38" title="" class="ltx_ref">2022</a>; Reid et al.,, <a href="#bib.bib37" title="" class="ltx_ref">2024</a>)</cite> have emerged as a dominant paradigm in natural language processing, achieving state-of-the-art performance across various tasks. A key principle, the Scaling Law <cite class="ltx_cite ltx_citemacro_citep">(Kaplan et al.,, <a href="#bib.bib26" title="" class="ltx_ref">2020</a>)</cite>, suggests that LLMs exhibit emergent abilities as model size increases, enhancing their capacity to understand context and handle long sequences <cite class="ltx_cite ltx_citemacro_citep">(Xiong et al.,, <a href="#bib.bib50" title="" class="ltx_ref">2023</a>)</cite>.
This capacity growth allows LLMs to generate coherent and contextually accurate responses and
enables various downstream applications, such as document summarization <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al.,, <a href="#bib.bib54" title="" class="ltx_ref">2019</a>; <a href="#bib.bib53" title="" class="ltx_ref">Zhang et al., 2024a, </a>)</cite>, code generation <cite class="ltx_cite ltx_citemacro_citep">(<a href="#bib.bib9" title="" class="ltx_ref">Chen et al., 2021b, </a>)</cite>, and conversational AI <cite class="ltx_cite ltx_citemacro_citep">(Bordes et al.,, <a href="#bib.bib5" title="" class="ltx_ref">2016</a>; OpenAI,, <a href="#bib.bib35" title="" class="ltx_ref">2022</a>)</cite>, .</p>
</div>
<div id="S1.p2" class="ltx_para ltx_noindent">
<p id="S1.p2.1" class="ltx_p">Despite their success in various applications, the generation of LLMs incurs significant expenses, which escalate with increasing model size and sequence length. Notably, both the training <cite class="ltx_cite ltx_citemacro_citep">(Strubell et al.,, <a href="#bib.bib40" title="" class="ltx_ref">2020</a>; Hoffmann et al.,, <a href="#bib.bib20" title="" class="ltx_ref">2022</a>; <a href="#bib.bib14" title="" class="ltx_ref">Dong et al., 2024a, </a>)</cite> and inference <cite class="ltx_cite ltx_citemacro_citep">(Ainslie et al.,, <a href="#bib.bib1" title="" class="ltx_ref">2023</a>)</cite> stages involve frequent generation by LLMs, further contributing to these costs. Consequently, efficient LLMs have gained popularity in recent years <cite class="ltx_cite ltx_citemacro_citep">(Hu et al.,, <a href="#bib.bib22" title="" class="ltx_ref">2021</a>; Wan et al.,, <a href="#bib.bib44" title="" class="ltx_ref">2023</a>)</cite>.
To address these challenges, quantization <cite class="ltx_cite ltx_citemacro_citep">(Frantar et al.,, <a href="#bib.bib17" title="" class="ltx_ref">2022</a>; Lin et al.,, <a href="#bib.bib30" title="" class="ltx_ref">2024</a>; Dettmers et al.,, <a href="#bib.bib12" title="" class="ltx_ref">2024</a>; Xu et al.,, <a href="#bib.bib51" title="" class="ltx_ref">2023</a>)</cite> and pruning methods <cite class="ltx_cite ltx_citemacro_citep">(Frankle and Carbin,, <a href="#bib.bib16" title="" class="ltx_ref">2018</a>; Blalock et al.,, <a href="#bib.bib4" title="" class="ltx_ref">2020</a>)</cite> are employed to reduce model size. Additionally, managing long sequences presents another cost due to the transformer attention mechanism. The quadratic complexity of the attention mechanism results in substantial computational burden when dealing with long sequences, scaling poorly with sequence length. Therefore, effective management of long sequences is essential for the practical deployment of LLMs. In this paper, we focus on the long-context scenario and aim to reduce the memory consumption associated with lengthy sequences.</p>
</div>
<div id="S1.p3" class="ltx_para ltx_noindent">
<p id="S1.p3.11" class="ltx_p">Specifically, the number of KV cache parameters is the product of batch size <math id="S1.p3.1.m1.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S1.p3.1.m1.1a"><mi id="S1.p3.1.m1.1.1" xref="S1.p3.1.m1.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S1.p3.1.m1.1b"><ci id="S1.p3.1.m1.1.1.cmml" xref="S1.p3.1.m1.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p3.1.m1.1c">B</annotation></semantics></math>, sequence length <math id="S1.p3.2.m2.1" class="ltx_Math" alttext="S" display="inline"><semantics id="S1.p3.2.m2.1a"><mi id="S1.p3.2.m2.1.1" xref="S1.p3.2.m2.1.1.cmml">S</mi><annotation-xml encoding="MathML-Content" id="S1.p3.2.m2.1b"><ci id="S1.p3.2.m2.1.1.cmml" xref="S1.p3.2.m2.1.1">𝑆</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p3.2.m2.1c">S</annotation></semantics></math>, number of layers <math id="S1.p3.3.m3.1" class="ltx_Math" alttext="L" display="inline"><semantics id="S1.p3.3.m3.1a"><mi id="S1.p3.3.m3.1.1" xref="S1.p3.3.m3.1.1.cmml">L</mi><annotation-xml encoding="MathML-Content" id="S1.p3.3.m3.1b"><ci id="S1.p3.3.m3.1.1.cmml" xref="S1.p3.3.m3.1.1">𝐿</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p3.3.m3.1c">L</annotation></semantics></math>, number of heads <math id="S1.p3.4.m4.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S1.p3.4.m4.1a"><mi id="S1.p3.4.m4.1.1" xref="S1.p3.4.m4.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S1.p3.4.m4.1b"><ci id="S1.p3.4.m4.1.1.cmml" xref="S1.p3.4.m4.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p3.4.m4.1c">N</annotation></semantics></math>, channel size of each head <math id="S1.p3.5.m5.1" class="ltx_Math" alttext="D" display="inline"><semantics id="S1.p3.5.m5.1a"><mi id="S1.p3.5.m5.1.1" xref="S1.p3.5.m5.1.1.cmml">D</mi><annotation-xml encoding="MathML-Content" id="S1.p3.5.m5.1b"><ci id="S1.p3.5.m5.1.1.cmml" xref="S1.p3.5.m5.1.1">𝐷</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p3.5.m5.1c">D</annotation></semantics></math>, i.e., <math id="S1.p3.6.m6.2" class="ltx_Math" alttext="\mathbf{K},\mathbf{V}\in\mathbb{R}^{B\times S\times L\times N\times D}" display="inline"><semantics id="S1.p3.6.m6.2a"><mrow id="S1.p3.6.m6.2.3" xref="S1.p3.6.m6.2.3.cmml"><mrow id="S1.p3.6.m6.2.3.2.2" xref="S1.p3.6.m6.2.3.2.1.cmml"><mi id="S1.p3.6.m6.1.1" xref="S1.p3.6.m6.1.1.cmml">𝐊</mi><mo id="S1.p3.6.m6.2.3.2.2.1" xref="S1.p3.6.m6.2.3.2.1.cmml">,</mo><mi id="S1.p3.6.m6.2.2" xref="S1.p3.6.m6.2.2.cmml">𝐕</mi></mrow><mo id="S1.p3.6.m6.2.3.1" xref="S1.p3.6.m6.2.3.1.cmml">∈</mo><msup id="S1.p3.6.m6.2.3.3" xref="S1.p3.6.m6.2.3.3.cmml"><mi id="S1.p3.6.m6.2.3.3.2" xref="S1.p3.6.m6.2.3.3.2.cmml">ℝ</mi><mrow id="S1.p3.6.m6.2.3.3.3" xref="S1.p3.6.m6.2.3.3.3.cmml"><mi id="S1.p3.6.m6.2.3.3.3.2" xref="S1.p3.6.m6.2.3.3.3.2.cmml">B</mi><mo lspace="0.222em" rspace="0.222em" id="S1.p3.6.m6.2.3.3.3.1" xref="S1.p3.6.m6.2.3.3.3.1.cmml">×</mo><mi id="S1.p3.6.m6.2.3.3.3.3" xref="S1.p3.6.m6.2.3.3.3.3.cmml">S</mi><mo lspace="0.222em" rspace="0.222em" id="S1.p3.6.m6.2.3.3.3.1a" xref="S1.p3.6.m6.2.3.3.3.1.cmml">×</mo><mi id="S1.p3.6.m6.2.3.3.3.4" xref="S1.p3.6.m6.2.3.3.3.4.cmml">L</mi><mo lspace="0.222em" rspace="0.222em" id="S1.p3.6.m6.2.3.3.3.1b" xref="S1.p3.6.m6.2.3.3.3.1.cmml">×</mo><mi id="S1.p3.6.m6.2.3.3.3.5" xref="S1.p3.6.m6.2.3.3.3.5.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="S1.p3.6.m6.2.3.3.3.1c" xref="S1.p3.6.m6.2.3.3.3.1.cmml">×</mo><mi id="S1.p3.6.m6.2.3.3.3.6" xref="S1.p3.6.m6.2.3.3.3.6.cmml">D</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S1.p3.6.m6.2b"><apply id="S1.p3.6.m6.2.3.cmml" xref="S1.p3.6.m6.2.3"><in id="S1.p3.6.m6.2.3.1.cmml" xref="S1.p3.6.m6.2.3.1"></in><list id="S1.p3.6.m6.2.3.2.1.cmml" xref="S1.p3.6.m6.2.3.2.2"><ci id="S1.p3.6.m6.1.1.cmml" xref="S1.p3.6.m6.1.1">𝐊</ci><ci id="S1.p3.6.m6.2.2.cmml" xref="S1.p3.6.m6.2.2">𝐕</ci></list><apply id="S1.p3.6.m6.2.3.3.cmml" xref="S1.p3.6.m6.2.3.3"><csymbol cd="ambiguous" id="S1.p3.6.m6.2.3.3.1.cmml" xref="S1.p3.6.m6.2.3.3">superscript</csymbol><ci id="S1.p3.6.m6.2.3.3.2.cmml" xref="S1.p3.6.m6.2.3.3.2">ℝ</ci><apply id="S1.p3.6.m6.2.3.3.3.cmml" xref="S1.p3.6.m6.2.3.3.3"><times id="S1.p3.6.m6.2.3.3.3.1.cmml" xref="S1.p3.6.m6.2.3.3.3.1"></times><ci id="S1.p3.6.m6.2.3.3.3.2.cmml" xref="S1.p3.6.m6.2.3.3.3.2">𝐵</ci><ci id="S1.p3.6.m6.2.3.3.3.3.cmml" xref="S1.p3.6.m6.2.3.3.3.3">𝑆</ci><ci id="S1.p3.6.m6.2.3.3.3.4.cmml" xref="S1.p3.6.m6.2.3.3.3.4">𝐿</ci><ci id="S1.p3.6.m6.2.3.3.3.5.cmml" xref="S1.p3.6.m6.2.3.3.3.5">𝑁</ci><ci id="S1.p3.6.m6.2.3.3.3.6.cmml" xref="S1.p3.6.m6.2.3.3.3.6">𝐷</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p3.6.m6.2c">\mathbf{K},\mathbf{V}\in\mathbb{R}^{B\times S\times L\times N\times D}</annotation></semantics></math>, which need to be stored in the GPU memory during inference.
To reduce memory and computational costs during inference, efficiency can only be achieved by pruning the dimensions across <math id="S1.p3.7.m7.4" class="ltx_Math" alttext="S,L,N,D" display="inline"><semantics id="S1.p3.7.m7.4a"><mrow id="S1.p3.7.m7.4.5.2" xref="S1.p3.7.m7.4.5.1.cmml"><mi id="S1.p3.7.m7.1.1" xref="S1.p3.7.m7.1.1.cmml">S</mi><mo id="S1.p3.7.m7.4.5.2.1" xref="S1.p3.7.m7.4.5.1.cmml">,</mo><mi id="S1.p3.7.m7.2.2" xref="S1.p3.7.m7.2.2.cmml">L</mi><mo id="S1.p3.7.m7.4.5.2.2" xref="S1.p3.7.m7.4.5.1.cmml">,</mo><mi id="S1.p3.7.m7.3.3" xref="S1.p3.7.m7.3.3.cmml">N</mi><mo id="S1.p3.7.m7.4.5.2.3" xref="S1.p3.7.m7.4.5.1.cmml">,</mo><mi id="S1.p3.7.m7.4.4" xref="S1.p3.7.m7.4.4.cmml">D</mi></mrow><annotation-xml encoding="MathML-Content" id="S1.p3.7.m7.4b"><list id="S1.p3.7.m7.4.5.1.cmml" xref="S1.p3.7.m7.4.5.2"><ci id="S1.p3.7.m7.1.1.cmml" xref="S1.p3.7.m7.1.1">𝑆</ci><ci id="S1.p3.7.m7.2.2.cmml" xref="S1.p3.7.m7.2.2">𝐿</ci><ci id="S1.p3.7.m7.3.3.cmml" xref="S1.p3.7.m7.3.3">𝑁</ci><ci id="S1.p3.7.m7.4.4.cmml" xref="S1.p3.7.m7.4.4">𝐷</ci></list></annotation-xml><annotation encoding="application/x-tex" id="S1.p3.7.m7.4c">S,L,N,D</annotation></semantics></math> or performing quantization over the caches. It is well-acknowledged that token importance tends to be sparse. Consequently, KV eviction algorithms have been proposed to minimize the KV cache memory cost from dimension <math id="S1.p3.8.m8.1" class="ltx_Math" alttext="S" display="inline"><semantics id="S1.p3.8.m8.1a"><mi id="S1.p3.8.m8.1.1" xref="S1.p3.8.m8.1.1.cmml">S</mi><annotation-xml encoding="MathML-Content" id="S1.p3.8.m8.1b"><ci id="S1.p3.8.m8.1.1.cmml" xref="S1.p3.8.m8.1.1">𝑆</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p3.8.m8.1c">S</annotation></semantics></math> <cite class="ltx_cite ltx_citemacro_citep">(<a href="#bib.bib49" title="" class="ltx_ref">Xiao et al., 2023b, </a>; Li et al.,, <a href="#bib.bib29" title="" class="ltx_ref">2024</a>; <a href="#bib.bib56" title="" class="ltx_ref">Zhang et al., 2024c, </a>; Leviathan et al.,, <a href="#bib.bib28" title="" class="ltx_ref">2023</a>)</cite>. Additionally, inter-layer redundancy has been explored <cite class="ltx_cite ltx_citemacro_citep">(<a href="#bib.bib31" title="" class="ltx_ref">Liu et al., 2024a, </a>; Wu and Tu,, <a href="#bib.bib47" title="" class="ltx_ref">2024</a>; Brandon et al.,, <a href="#bib.bib6" title="" class="ltx_ref">2024</a>)</cite> to address the layer dimension <math id="S1.p3.9.m9.1" class="ltx_Math" alttext="L" display="inline"><semantics id="S1.p3.9.m9.1a"><mi id="S1.p3.9.m9.1.1" xref="S1.p3.9.m9.1.1.cmml">L</mi><annotation-xml encoding="MathML-Content" id="S1.p3.9.m9.1b"><ci id="S1.p3.9.m9.1.1.cmml" xref="S1.p3.9.m9.1.1">𝐿</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p3.9.m9.1c">L</annotation></semantics></math>. Despite these advances, existing methods have largely overlooked the channel dimension <math id="S1.p3.10.m10.1" class="ltx_Math" alttext="D" display="inline"><semantics id="S1.p3.10.m10.1a"><mi id="S1.p3.10.m10.1.1" xref="S1.p3.10.m10.1.1.cmml">D</mi><annotation-xml encoding="MathML-Content" id="S1.p3.10.m10.1b"><ci id="S1.p3.10.m10.1.1.cmml" xref="S1.p3.10.m10.1.1">𝐷</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p3.10.m10.1c">D</annotation></semantics></math>. In this paper, we highlight that the magnitude across Key cache channel dimensions is significantly unbalanced. Additionally, we observe that the attention weights exhibit a low-rank structure. Based on these findings, we hypothesize that the channel dimension of the key cache exhibits redundancy. Consequently, we explore the redundancy of KV cache tensors specifically from dimension <math id="S1.p3.11.m11.1" class="ltx_Math" alttext="D" display="inline"><semantics id="S1.p3.11.m11.1a"><mi id="S1.p3.11.m11.1.1" xref="S1.p3.11.m11.1.1.cmml">D</mi><annotation-xml encoding="MathML-Content" id="S1.p3.11.m11.1b"><ci id="S1.p3.11.m11.1.1.cmml" xref="S1.p3.11.m11.1.1">𝐷</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p3.11.m11.1c">D</annotation></semantics></math>, aiming to devise effective strategies that reduce costs without compromising performance.</p>
</div>
<div id="S1.p4" class="ltx_para ltx_noindent">
<p id="S1.p4.1" class="ltx_p">In our paper, we propose a simple yet effective method – <span id="S1.p4.1.1" class="ltx_text ltx_font_bold">ThinK</span>, for KV cache pruning. To pinpoint the least significant channels, we formulate the task as an optimization problem, aiming to minimize the loss in attention weights attributable to pruning. To effectively address this problem, we establish a novel query-dependent criterion that assesses the importance of each channel. Using this criterion, we then select the most critical channels in a greedy fashion. We evaluate <span id="S1.p4.1.2" class="ltx_text ltx_font_bold">ThinK</span> using the LLaMA3 <cite class="ltx_cite ltx_citemacro_citep">(Meta,, <a href="#bib.bib34" title="" class="ltx_ref">2024</a>)</cite> and Mistral <cite class="ltx_cite ltx_citemacro_citep">(Jiang et al.,, <a href="#bib.bib24" title="" class="ltx_ref">2023</a>)</cite> models and validate its effectiveness across various long-sequence datasets. The results indicate that when paired with token eviction methods, ThinK not only achieves comparable or superior accuracy but also reduces KV cache memory costs by more than 20%.</p>
</div>
<section id="S1.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Contributions.</h4>

<div id="S1.SS0.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S1.SS0.SSS0.Px1.p1.1" class="ltx_p">This work pioneers the investigation into the sparsity structure of the channels in the KV cache. Specifically, we discovered that the activated key cache is sparse given a specific query. This insight allows us to prune the key cache channels using a query-induced norm. Inspired by this phenomenon, we introduce a plug-and-play method ThinK, the first channel pruning method specifically designed for KV cache. ThinK reduces the dimensionality of the cache channel, leading to linear savings in both memory and computational requirements. As plug-and-play techniques, our proposed ThinK is orthogonal to other KV cache compression schemes (<span id="S1.SS0.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_italic">e.g.</span> KV cache eviction, quantization). Our extensive experiments demonstrate the remarkable efficiency of ThinK on LLaMA-3 and Mistral models. Furthermore, we explore the promising potential of extending ThinK to value cache pruning (ThinKV) in specific scenarios, highlighting the broad applicability of our method.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2407.21018/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="178" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S1.F1.2.1.1" class="ltx_text" style="font-size:90%;">Figure 1</span>: </span><span id="S1.F1.3.2" class="ltx_text" style="font-size:90%;">An illustration of the pruning procedure of ThinK. Within each head, scores are calculated for each channel, and only the top T channels out of D are selected for retention. A binary channel mask, along with the pruned keys, is subsequently stored in the cache memory.</span></figcaption>
</figure>
</section>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Observations</h2>

<div id="S2.p1" class="ltx_para ltx_noindent">
<p id="S2.p1.1" class="ltx_p">We identify several key observations that motivate our approach to prune the channel of KV-Cache.
Specifically, we visualize the magnitude of KV-cache and perform singular value decomposition of the attention in LLaMA3-8B model.
We show the visualization for certain heads and layers for illustration purposes.</p>
</div>
<section id="S2.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Magnitude of KV-Cache Channel</h4>

<div id="S2.SS0.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px1.p1.5" class="ltx_p">Figure <a href="#S2.F2" title="Figure 2 ‣ Magnitude of KV-Cache Channel ‣ 2 Observations ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> visualizes the absolute values of key and value cache across the tokens in each channel<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>We use the visualization code from <a target="_blank" href="https://github.com/jy-yuan/KIVI/tree/main/vis" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/jy-yuan/KIVI/tree/main/vis</a>.</span></span></span>.
Consistent with previous findings <cite class="ltx_cite ltx_citemacro_citep">(Lin et al.,, <a href="#bib.bib30" title="" class="ltx_ref">2024</a>; <a href="#bib.bib48" title="" class="ltx_ref">Xiao et al., 2023a, </a>; <a href="#bib.bib32" title="" class="ltx_ref">Liu et al., 2024b, </a>)</cite>, we observe that only certain channels have significant magnitudes in the key cache, whereas the value cache lacks obvious patterns.
For instance, the absolute value around <math id="S2.SS0.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="50^{th}" display="inline"><semantics id="S2.SS0.SSS0.Px1.p1.1.m1.1a"><msup id="S2.SS0.SSS0.Px1.p1.1.m1.1.1" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.cmml"><mn id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.2" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.2.cmml">50</mn><mrow id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.cmml"><mi id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.2" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.1" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.3" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.cmml">h</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p1.1.m1.1b"><apply id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1">superscript</csymbol><cn type="integer" id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.2">50</cn><apply id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3"><times id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.1"></times><ci id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.2">𝑡</ci><ci id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.3">ℎ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p1.1.m1.1c">50^{th}</annotation></semantics></math> channel of key cache has much larger magnitudes than other channels for all tokens in layer <math id="S2.SS0.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="14" display="inline"><semantics id="S2.SS0.SSS0.Px1.p1.2.m2.1a"><mn id="S2.SS0.SSS0.Px1.p1.2.m2.1.1" xref="S2.SS0.SSS0.Px1.p1.2.m2.1.1.cmml">14</mn><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p1.2.m2.1b"><cn type="integer" id="S2.SS0.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.2.m2.1.1">14</cn></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p1.2.m2.1c">14</annotation></semantics></math> (Figure <a href="#S2.F2" title="Figure 2 ‣ Magnitude of KV-Cache Channel ‣ 2 Observations ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> (a)).
The same observation applies to <math id="S2.SS0.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="50^{th}" display="inline"><semantics id="S2.SS0.SSS0.Px1.p1.3.m3.1a"><msup id="S2.SS0.SSS0.Px1.p1.3.m3.1.1" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.cmml"><mn id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.2" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.2.cmml">50</mn><mrow id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3.cmml"><mi id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3.2" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3.1" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3.3" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3.3.cmml">h</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p1.3.m3.1b"><apply id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1">superscript</csymbol><cn type="integer" id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.2">50</cn><apply id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3"><times id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3.1"></times><ci id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3.2">𝑡</ci><ci id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3.3">ℎ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p1.3.m3.1c">50^{th}</annotation></semantics></math> and <math id="S2.SS0.SSS0.Px1.p1.4.m4.1" class="ltx_Math" alttext="150^{th}" display="inline"><semantics id="S2.SS0.SSS0.Px1.p1.4.m4.1a"><msup id="S2.SS0.SSS0.Px1.p1.4.m4.1.1" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.cmml"><mn id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.2" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.2.cmml">150</mn><mrow id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.cmml"><mi id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.2" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.1" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.3" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.3.cmml">h</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p1.4.m4.1b"><apply id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1">superscript</csymbol><cn type="integer" id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.2">150</cn><apply id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.cmml" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3"><times id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.1"></times><ci id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.2">𝑡</ci><ci id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.3">ℎ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p1.4.m4.1c">150^{th}</annotation></semantics></math> channels of the first head in layer <math id="S2.SS0.SSS0.Px1.p1.5.m5.1" class="ltx_Math" alttext="20" display="inline"><semantics id="S2.SS0.SSS0.Px1.p1.5.m5.1a"><mn id="S2.SS0.SSS0.Px1.p1.5.m5.1.1" xref="S2.SS0.SSS0.Px1.p1.5.m5.1.1.cmml">20</mn><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p1.5.m5.1b"><cn type="integer" id="S2.SS0.SSS0.Px1.p1.5.m5.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.5.m5.1.1">20</cn></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p1.5.m5.1c">20</annotation></semantics></math> (Figure <a href="#S2.F2" title="Figure 2 ‣ Magnitude of KV-Cache Channel ‣ 2 Observations ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> (c)).
Given such an observation, <cite class="ltx_cite ltx_citemacro_citet"><a href="#bib.bib32" title="" class="ltx_ref">Liu et al., 2024b </a></cite> proposed to perform quantization over the channels of the key cache based on such observations.
In addition to channel quantization, the observation also suggests that it is feasible to prune away certain key cache channels that contribute less to the attention mechanism.
Moreover, channel quantization and pruning techniques are orthogonal such that they can be utilized concurrently to enhance model efficiency.</p>
</div>
<figure id="S2.F2" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_4">
<div id="S2.F2.2" class="ltx_block ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:104.1pt;">
<img src="/html/2407.21018/assets/x2.png" id="S2.F2.1.g1" class="ltx_graphics ltx_img_square" width="461" height="455" alt="Refer to caption">
<p id="S2.F2.2.1" class="ltx_p ltx_align_center"><span id="S2.F2.2.1.1" class="ltx_text" style="font-size:80%;">(a) Key Cache (<math id="S2.F2.2.1.1.m1.1" class="ltx_Math" alttext="14^{th}" display="inline"><semantics id="S2.F2.2.1.1.m1.1a"><msup id="S2.F2.2.1.1.m1.1.1" xref="S2.F2.2.1.1.m1.1.1.cmml"><mn id="S2.F2.2.1.1.m1.1.1.2" xref="S2.F2.2.1.1.m1.1.1.2.cmml">14</mn><mrow id="S2.F2.2.1.1.m1.1.1.3" xref="S2.F2.2.1.1.m1.1.1.3.cmml"><mi id="S2.F2.2.1.1.m1.1.1.3.2" xref="S2.F2.2.1.1.m1.1.1.3.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="S2.F2.2.1.1.m1.1.1.3.1" xref="S2.F2.2.1.1.m1.1.1.3.1.cmml">​</mo><mi id="S2.F2.2.1.1.m1.1.1.3.3" xref="S2.F2.2.1.1.m1.1.1.3.3.cmml">h</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="S2.F2.2.1.1.m1.1b"><apply id="S2.F2.2.1.1.m1.1.1.cmml" xref="S2.F2.2.1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.F2.2.1.1.m1.1.1.1.cmml" xref="S2.F2.2.1.1.m1.1.1">superscript</csymbol><cn type="integer" id="S2.F2.2.1.1.m1.1.1.2.cmml" xref="S2.F2.2.1.1.m1.1.1.2">14</cn><apply id="S2.F2.2.1.1.m1.1.1.3.cmml" xref="S2.F2.2.1.1.m1.1.1.3"><times id="S2.F2.2.1.1.m1.1.1.3.1.cmml" xref="S2.F2.2.1.1.m1.1.1.3.1"></times><ci id="S2.F2.2.1.1.m1.1.1.3.2.cmml" xref="S2.F2.2.1.1.m1.1.1.3.2">𝑡</ci><ci id="S2.F2.2.1.1.m1.1.1.3.3.cmml" xref="S2.F2.2.1.1.m1.1.1.3.3">ℎ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.F2.2.1.1.m1.1c">14^{th}</annotation></semantics></math> layer)</span></p>
</div>
</div>
<div class="ltx_flex_cell ltx_flex_size_4">
<div id="S2.F2.4" class="ltx_block ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:104.1pt;">
<img src="/html/2407.21018/assets/x3.png" id="S2.F2.3.g1" class="ltx_graphics ltx_img_square" width="462" height="449" alt="Refer to caption">
<p id="S2.F2.4.1" class="ltx_p ltx_align_center"><span id="S2.F2.4.1.1" class="ltx_text" style="font-size:80%;">(b) Value Cache (<math id="S2.F2.4.1.1.m1.1" class="ltx_Math" alttext="14^{th}" display="inline"><semantics id="S2.F2.4.1.1.m1.1a"><msup id="S2.F2.4.1.1.m1.1.1" xref="S2.F2.4.1.1.m1.1.1.cmml"><mn id="S2.F2.4.1.1.m1.1.1.2" xref="S2.F2.4.1.1.m1.1.1.2.cmml">14</mn><mrow id="S2.F2.4.1.1.m1.1.1.3" xref="S2.F2.4.1.1.m1.1.1.3.cmml"><mi id="S2.F2.4.1.1.m1.1.1.3.2" xref="S2.F2.4.1.1.m1.1.1.3.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="S2.F2.4.1.1.m1.1.1.3.1" xref="S2.F2.4.1.1.m1.1.1.3.1.cmml">​</mo><mi id="S2.F2.4.1.1.m1.1.1.3.3" xref="S2.F2.4.1.1.m1.1.1.3.3.cmml">h</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="S2.F2.4.1.1.m1.1b"><apply id="S2.F2.4.1.1.m1.1.1.cmml" xref="S2.F2.4.1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.F2.4.1.1.m1.1.1.1.cmml" xref="S2.F2.4.1.1.m1.1.1">superscript</csymbol><cn type="integer" id="S2.F2.4.1.1.m1.1.1.2.cmml" xref="S2.F2.4.1.1.m1.1.1.2">14</cn><apply id="S2.F2.4.1.1.m1.1.1.3.cmml" xref="S2.F2.4.1.1.m1.1.1.3"><times id="S2.F2.4.1.1.m1.1.1.3.1.cmml" xref="S2.F2.4.1.1.m1.1.1.3.1"></times><ci id="S2.F2.4.1.1.m1.1.1.3.2.cmml" xref="S2.F2.4.1.1.m1.1.1.3.2">𝑡</ci><ci id="S2.F2.4.1.1.m1.1.1.3.3.cmml" xref="S2.F2.4.1.1.m1.1.1.3.3">ℎ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.F2.4.1.1.m1.1c">14^{th}</annotation></semantics></math> layer)</span></p>
</div>
</div>
<div class="ltx_flex_cell ltx_flex_size_4">
<div id="S2.F2.6" class="ltx_block ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:104.1pt;">
<img src="/html/2407.21018/assets/x4.png" id="S2.F2.5.g1" class="ltx_graphics ltx_img_square" width="461" height="455" alt="Refer to caption">
<p id="S2.F2.6.1" class="ltx_p ltx_align_center"><span id="S2.F2.6.1.1" class="ltx_text" style="font-size:80%;">(c) Key Cache (<math id="S2.F2.6.1.1.m1.1" class="ltx_Math" alttext="20^{th}" display="inline"><semantics id="S2.F2.6.1.1.m1.1a"><msup id="S2.F2.6.1.1.m1.1.1" xref="S2.F2.6.1.1.m1.1.1.cmml"><mn id="S2.F2.6.1.1.m1.1.1.2" xref="S2.F2.6.1.1.m1.1.1.2.cmml">20</mn><mrow id="S2.F2.6.1.1.m1.1.1.3" xref="S2.F2.6.1.1.m1.1.1.3.cmml"><mi id="S2.F2.6.1.1.m1.1.1.3.2" xref="S2.F2.6.1.1.m1.1.1.3.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="S2.F2.6.1.1.m1.1.1.3.1" xref="S2.F2.6.1.1.m1.1.1.3.1.cmml">​</mo><mi id="S2.F2.6.1.1.m1.1.1.3.3" xref="S2.F2.6.1.1.m1.1.1.3.3.cmml">h</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="S2.F2.6.1.1.m1.1b"><apply id="S2.F2.6.1.1.m1.1.1.cmml" xref="S2.F2.6.1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.F2.6.1.1.m1.1.1.1.cmml" xref="S2.F2.6.1.1.m1.1.1">superscript</csymbol><cn type="integer" id="S2.F2.6.1.1.m1.1.1.2.cmml" xref="S2.F2.6.1.1.m1.1.1.2">20</cn><apply id="S2.F2.6.1.1.m1.1.1.3.cmml" xref="S2.F2.6.1.1.m1.1.1.3"><times id="S2.F2.6.1.1.m1.1.1.3.1.cmml" xref="S2.F2.6.1.1.m1.1.1.3.1"></times><ci id="S2.F2.6.1.1.m1.1.1.3.2.cmml" xref="S2.F2.6.1.1.m1.1.1.3.2">𝑡</ci><ci id="S2.F2.6.1.1.m1.1.1.3.3.cmml" xref="S2.F2.6.1.1.m1.1.1.3.3">ℎ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.F2.6.1.1.m1.1c">20^{th}</annotation></semantics></math> layer)</span></p>
</div>
</div>
<div class="ltx_flex_cell ltx_flex_size_4">
<div id="S2.F2.8" class="ltx_block ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:104.1pt;">
<img src="/html/2407.21018/assets/x5.png" id="S2.F2.7.g1" class="ltx_graphics ltx_img_square" width="461" height="464" alt="Refer to caption">
<p id="S2.F2.8.1" class="ltx_p ltx_align_center"><span id="S2.F2.8.1.1" class="ltx_text" style="font-size:80%;">(d) Value Cache (<math id="S2.F2.8.1.1.m1.1" class="ltx_Math" alttext="20^{th}" display="inline"><semantics id="S2.F2.8.1.1.m1.1a"><msup id="S2.F2.8.1.1.m1.1.1" xref="S2.F2.8.1.1.m1.1.1.cmml"><mn id="S2.F2.8.1.1.m1.1.1.2" xref="S2.F2.8.1.1.m1.1.1.2.cmml">20</mn><mrow id="S2.F2.8.1.1.m1.1.1.3" xref="S2.F2.8.1.1.m1.1.1.3.cmml"><mi id="S2.F2.8.1.1.m1.1.1.3.2" xref="S2.F2.8.1.1.m1.1.1.3.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="S2.F2.8.1.1.m1.1.1.3.1" xref="S2.F2.8.1.1.m1.1.1.3.1.cmml">​</mo><mi id="S2.F2.8.1.1.m1.1.1.3.3" xref="S2.F2.8.1.1.m1.1.1.3.3.cmml">h</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="S2.F2.8.1.1.m1.1b"><apply id="S2.F2.8.1.1.m1.1.1.cmml" xref="S2.F2.8.1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.F2.8.1.1.m1.1.1.1.cmml" xref="S2.F2.8.1.1.m1.1.1">superscript</csymbol><cn type="integer" id="S2.F2.8.1.1.m1.1.1.2.cmml" xref="S2.F2.8.1.1.m1.1.1.2">20</cn><apply id="S2.F2.8.1.1.m1.1.1.3.cmml" xref="S2.F2.8.1.1.m1.1.1.3"><times id="S2.F2.8.1.1.m1.1.1.3.1.cmml" xref="S2.F2.8.1.1.m1.1.1.3.1"></times><ci id="S2.F2.8.1.1.m1.1.1.3.2.cmml" xref="S2.F2.8.1.1.m1.1.1.3.2">𝑡</ci><ci id="S2.F2.8.1.1.m1.1.1.3.3.cmml" xref="S2.F2.8.1.1.m1.1.1.3.3">ℎ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.F2.8.1.1.m1.1c">20^{th}</annotation></semantics></math> layer)</span></p>
</div>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S2.F2.10.1.1" class="ltx_text" style="font-size:90%;">Figure 2</span>: </span><span id="S2.F2.11.2" class="ltx_text" style="font-size:90%;">Magnitude of key and value cache for LLaMA3-8B. The first head of layer 14 and layer 20 of LLaMA3-8B is selected to visualize the magnitude of the key and value caches. We observe that the magnitudes of the key cache channels vary differently, whereas the channels of the value cache do not exhibit such variation.</span></figcaption>
</figure>
</section>
<section id="S2.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Singular Value Analysis</h4>

<div id="S2.SS0.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px2.p1.1" class="ltx_p">We conducted singular value decomposition (SVD) <cite class="ltx_cite ltx_citemacro_citep">(Demmel,, <a href="#bib.bib11" title="" class="ltx_ref">1997</a>)</cite> on the attention of the specified layers to investigate their principal components.
The singular values derived from SVD capture the effective rank of the attention matrix, indicating how the information is distributed across different components.</p>
</div>
<div id="S2.SS0.SSS0.Px2.p2" class="ltx_para ltx_noindent">
<table id="A1.EGx1" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S2.E1"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S2.E1.m1.3" class="ltx_Math" alttext="\displaystyle\mathbf{U},\mathbf{\Sigma},\mathbf{V}" display="inline"><semantics id="S2.E1.m1.3a"><mrow id="S2.E1.m1.3.4.2" xref="S2.E1.m1.3.4.1.cmml"><mi id="S2.E1.m1.1.1" xref="S2.E1.m1.1.1.cmml">𝐔</mi><mo id="S2.E1.m1.3.4.2.1" xref="S2.E1.m1.3.4.1.cmml">,</mo><mi id="S2.E1.m1.2.2" xref="S2.E1.m1.2.2.cmml">𝚺</mi><mo id="S2.E1.m1.3.4.2.2" xref="S2.E1.m1.3.4.1.cmml">,</mo><mi id="S2.E1.m1.3.3" xref="S2.E1.m1.3.3.cmml">𝐕</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.E1.m1.3b"><list id="S2.E1.m1.3.4.1.cmml" xref="S2.E1.m1.3.4.2"><ci id="S2.E1.m1.1.1.cmml" xref="S2.E1.m1.1.1">𝐔</ci><ci id="S2.E1.m1.2.2.cmml" xref="S2.E1.m1.2.2">𝚺</ci><ci id="S2.E1.m1.3.3.cmml" xref="S2.E1.m1.3.3">𝐕</ci></list></annotation-xml><annotation encoding="application/x-tex" id="S2.E1.m1.3c">\displaystyle\mathbf{U},\mathbf{\Sigma},\mathbf{V}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S2.E1.m2.3" class="ltx_Math" alttext="\displaystyle=\text{SVD}\bigg{(}\text{softmax}\Big{(}\frac{\textbf{Q}\textbf{K}^{T}}{\sqrt{D}}\Big{)}\bigg{)},\quad\text{Energy}_{i}=\frac{\sigma_{i}^{2}}{\sum_{i}\sigma_{i}^{2}}" display="inline"><semantics id="S2.E1.m2.3a"><mrow id="S2.E1.m2.3.3.2" xref="S2.E1.m2.3.3.3.cmml"><mrow id="S2.E1.m2.2.2.1.1" xref="S2.E1.m2.2.2.1.1.cmml"><mi id="S2.E1.m2.2.2.1.1.3" xref="S2.E1.m2.2.2.1.1.3.cmml"></mi><mo id="S2.E1.m2.2.2.1.1.2" xref="S2.E1.m2.2.2.1.1.2.cmml">=</mo><mrow id="S2.E1.m2.2.2.1.1.1" xref="S2.E1.m2.2.2.1.1.1.cmml"><mtext id="S2.E1.m2.2.2.1.1.1.3" xref="S2.E1.m2.2.2.1.1.1.3a.cmml">SVD</mtext><mo lspace="0em" rspace="0em" id="S2.E1.m2.2.2.1.1.1.2" xref="S2.E1.m2.2.2.1.1.1.2.cmml">​</mo><mrow id="S2.E1.m2.2.2.1.1.1.1.1" xref="S2.E1.m2.2.2.1.1.1.1.1.1.cmml"><mo maxsize="210%" minsize="210%" id="S2.E1.m2.2.2.1.1.1.1.1.2" xref="S2.E1.m2.2.2.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.E1.m2.2.2.1.1.1.1.1.1" xref="S2.E1.m2.2.2.1.1.1.1.1.1.cmml"><mtext id="S2.E1.m2.2.2.1.1.1.1.1.1.2" xref="S2.E1.m2.2.2.1.1.1.1.1.1.2a.cmml">softmax</mtext><mo lspace="0em" rspace="0em" id="S2.E1.m2.2.2.1.1.1.1.1.1.1" xref="S2.E1.m2.2.2.1.1.1.1.1.1.1.cmml">​</mo><mrow id="S2.E1.m2.2.2.1.1.1.1.1.1.3.2" xref="S2.E1.m2.1.1.cmml"><mo maxsize="160%" minsize="160%" id="S2.E1.m2.2.2.1.1.1.1.1.1.3.2.1" xref="S2.E1.m2.1.1.cmml">(</mo><mstyle displaystyle="true" id="S2.E1.m2.1.1" xref="S2.E1.m2.1.1.cmml"><mfrac id="S2.E1.m2.1.1a" xref="S2.E1.m2.1.1.cmml"><msup id="S2.E1.m2.1.1.2" xref="S2.E1.m2.1.1.2.cmml"><mrow id="S2.E1.m2.1.1.2.2" xref="S2.E1.m2.1.1.2.2c.cmml"><mtext class="ltx_mathvariant_bold" id="S2.E1.m2.1.1.2.2a" xref="S2.E1.m2.1.1.2.2c.cmml">Q</mtext><mtext class="ltx_mathvariant_bold" id="S2.E1.m2.1.1.2.2b" xref="S2.E1.m2.1.1.2.2c.cmml">K</mtext></mrow><mi id="S2.E1.m2.1.1.2.3" xref="S2.E1.m2.1.1.2.3.cmml">T</mi></msup><msqrt id="S2.E1.m2.1.1.3" xref="S2.E1.m2.1.1.3.cmml"><mi id="S2.E1.m2.1.1.3.2" xref="S2.E1.m2.1.1.3.2.cmml">D</mi></msqrt></mfrac></mstyle><mo maxsize="160%" minsize="160%" id="S2.E1.m2.2.2.1.1.1.1.1.1.3.2.2" xref="S2.E1.m2.1.1.cmml">)</mo></mrow></mrow><mo maxsize="210%" minsize="210%" id="S2.E1.m2.2.2.1.1.1.1.1.3" xref="S2.E1.m2.2.2.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo rspace="1.167em" id="S2.E1.m2.3.3.2.3" xref="S2.E1.m2.3.3.3a.cmml">,</mo><mrow id="S2.E1.m2.3.3.2.2" xref="S2.E1.m2.3.3.2.2.cmml"><msub id="S2.E1.m2.3.3.2.2.2" xref="S2.E1.m2.3.3.2.2.2.cmml"><mtext id="S2.E1.m2.3.3.2.2.2.2" xref="S2.E1.m2.3.3.2.2.2.2a.cmml">Energy</mtext><mi id="S2.E1.m2.3.3.2.2.2.3" xref="S2.E1.m2.3.3.2.2.2.3.cmml">i</mi></msub><mo id="S2.E1.m2.3.3.2.2.1" xref="S2.E1.m2.3.3.2.2.1.cmml">=</mo><mstyle displaystyle="true" id="S2.E1.m2.3.3.2.2.3" xref="S2.E1.m2.3.3.2.2.3.cmml"><mfrac id="S2.E1.m2.3.3.2.2.3a" xref="S2.E1.m2.3.3.2.2.3.cmml"><msubsup id="S2.E1.m2.3.3.2.2.3.2" xref="S2.E1.m2.3.3.2.2.3.2.cmml"><mi id="S2.E1.m2.3.3.2.2.3.2.2.2" xref="S2.E1.m2.3.3.2.2.3.2.2.2.cmml">σ</mi><mi id="S2.E1.m2.3.3.2.2.3.2.2.3" xref="S2.E1.m2.3.3.2.2.3.2.2.3.cmml">i</mi><mn id="S2.E1.m2.3.3.2.2.3.2.3" xref="S2.E1.m2.3.3.2.2.3.2.3.cmml">2</mn></msubsup><mrow id="S2.E1.m2.3.3.2.2.3.3" xref="S2.E1.m2.3.3.2.2.3.3.cmml"><msub id="S2.E1.m2.3.3.2.2.3.3.1" xref="S2.E1.m2.3.3.2.2.3.3.1.cmml"><mo id="S2.E1.m2.3.3.2.2.3.3.1.2" xref="S2.E1.m2.3.3.2.2.3.3.1.2.cmml">∑</mo><mi id="S2.E1.m2.3.3.2.2.3.3.1.3" xref="S2.E1.m2.3.3.2.2.3.3.1.3.cmml">i</mi></msub><msubsup id="S2.E1.m2.3.3.2.2.3.3.2" xref="S2.E1.m2.3.3.2.2.3.3.2.cmml"><mi id="S2.E1.m2.3.3.2.2.3.3.2.2.2" xref="S2.E1.m2.3.3.2.2.3.3.2.2.2.cmml">σ</mi><mi id="S2.E1.m2.3.3.2.2.3.3.2.2.3" xref="S2.E1.m2.3.3.2.2.3.3.2.2.3.cmml">i</mi><mn id="S2.E1.m2.3.3.2.2.3.3.2.3" xref="S2.E1.m2.3.3.2.2.3.3.2.3.cmml">2</mn></msubsup></mrow></mfrac></mstyle></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E1.m2.3b"><apply id="S2.E1.m2.3.3.3.cmml" xref="S2.E1.m2.3.3.2"><csymbol cd="ambiguous" id="S2.E1.m2.3.3.3a.cmml" xref="S2.E1.m2.3.3.2.3">formulae-sequence</csymbol><apply id="S2.E1.m2.2.2.1.1.cmml" xref="S2.E1.m2.2.2.1.1"><eq id="S2.E1.m2.2.2.1.1.2.cmml" xref="S2.E1.m2.2.2.1.1.2"></eq><csymbol cd="latexml" id="S2.E1.m2.2.2.1.1.3.cmml" xref="S2.E1.m2.2.2.1.1.3">absent</csymbol><apply id="S2.E1.m2.2.2.1.1.1.cmml" xref="S2.E1.m2.2.2.1.1.1"><times id="S2.E1.m2.2.2.1.1.1.2.cmml" xref="S2.E1.m2.2.2.1.1.1.2"></times><ci id="S2.E1.m2.2.2.1.1.1.3a.cmml" xref="S2.E1.m2.2.2.1.1.1.3"><mtext id="S2.E1.m2.2.2.1.1.1.3.cmml" xref="S2.E1.m2.2.2.1.1.1.3">SVD</mtext></ci><apply id="S2.E1.m2.2.2.1.1.1.1.1.1.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1"><times id="S2.E1.m2.2.2.1.1.1.1.1.1.1.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.1.1"></times><ci id="S2.E1.m2.2.2.1.1.1.1.1.1.2a.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.1.2"><mtext id="S2.E1.m2.2.2.1.1.1.1.1.1.2.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.1.2">softmax</mtext></ci><apply id="S2.E1.m2.1.1.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.1.3.2"><divide id="S2.E1.m2.1.1.1.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.1.3.2"></divide><apply id="S2.E1.m2.1.1.2.cmml" xref="S2.E1.m2.1.1.2"><csymbol cd="ambiguous" id="S2.E1.m2.1.1.2.1.cmml" xref="S2.E1.m2.1.1.2">superscript</csymbol><ci id="S2.E1.m2.1.1.2.2c.cmml" xref="S2.E1.m2.1.1.2.2"><mrow id="S2.E1.m2.1.1.2.2.cmml" xref="S2.E1.m2.1.1.2.2"><mtext class="ltx_mathvariant_bold" id="S2.E1.m2.1.1.2.2a.cmml" xref="S2.E1.m2.1.1.2.2">Q</mtext><mtext class="ltx_mathvariant_bold" id="S2.E1.m2.1.1.2.2b.cmml" xref="S2.E1.m2.1.1.2.2">K</mtext></mrow></ci><ci id="S2.E1.m2.1.1.2.3.cmml" xref="S2.E1.m2.1.1.2.3">𝑇</ci></apply><apply id="S2.E1.m2.1.1.3.cmml" xref="S2.E1.m2.1.1.3"><root id="S2.E1.m2.1.1.3a.cmml" xref="S2.E1.m2.1.1.3"></root><ci id="S2.E1.m2.1.1.3.2.cmml" xref="S2.E1.m2.1.1.3.2">𝐷</ci></apply></apply></apply></apply></apply><apply id="S2.E1.m2.3.3.2.2.cmml" xref="S2.E1.m2.3.3.2.2"><eq id="S2.E1.m2.3.3.2.2.1.cmml" xref="S2.E1.m2.3.3.2.2.1"></eq><apply id="S2.E1.m2.3.3.2.2.2.cmml" xref="S2.E1.m2.3.3.2.2.2"><csymbol cd="ambiguous" id="S2.E1.m2.3.3.2.2.2.1.cmml" xref="S2.E1.m2.3.3.2.2.2">subscript</csymbol><ci id="S2.E1.m2.3.3.2.2.2.2a.cmml" xref="S2.E1.m2.3.3.2.2.2.2"><mtext id="S2.E1.m2.3.3.2.2.2.2.cmml" xref="S2.E1.m2.3.3.2.2.2.2">Energy</mtext></ci><ci id="S2.E1.m2.3.3.2.2.2.3.cmml" xref="S2.E1.m2.3.3.2.2.2.3">𝑖</ci></apply><apply id="S2.E1.m2.3.3.2.2.3.cmml" xref="S2.E1.m2.3.3.2.2.3"><divide id="S2.E1.m2.3.3.2.2.3.1.cmml" xref="S2.E1.m2.3.3.2.2.3"></divide><apply id="S2.E1.m2.3.3.2.2.3.2.cmml" xref="S2.E1.m2.3.3.2.2.3.2"><csymbol cd="ambiguous" id="S2.E1.m2.3.3.2.2.3.2.1.cmml" xref="S2.E1.m2.3.3.2.2.3.2">superscript</csymbol><apply id="S2.E1.m2.3.3.2.2.3.2.2.cmml" xref="S2.E1.m2.3.3.2.2.3.2"><csymbol cd="ambiguous" id="S2.E1.m2.3.3.2.2.3.2.2.1.cmml" xref="S2.E1.m2.3.3.2.2.3.2">subscript</csymbol><ci id="S2.E1.m2.3.3.2.2.3.2.2.2.cmml" xref="S2.E1.m2.3.3.2.2.3.2.2.2">𝜎</ci><ci id="S2.E1.m2.3.3.2.2.3.2.2.3.cmml" xref="S2.E1.m2.3.3.2.2.3.2.2.3">𝑖</ci></apply><cn type="integer" id="S2.E1.m2.3.3.2.2.3.2.3.cmml" xref="S2.E1.m2.3.3.2.2.3.2.3">2</cn></apply><apply id="S2.E1.m2.3.3.2.2.3.3.cmml" xref="S2.E1.m2.3.3.2.2.3.3"><apply id="S2.E1.m2.3.3.2.2.3.3.1.cmml" xref="S2.E1.m2.3.3.2.2.3.3.1"><csymbol cd="ambiguous" id="S2.E1.m2.3.3.2.2.3.3.1.1.cmml" xref="S2.E1.m2.3.3.2.2.3.3.1">subscript</csymbol><sum id="S2.E1.m2.3.3.2.2.3.3.1.2.cmml" xref="S2.E1.m2.3.3.2.2.3.3.1.2"></sum><ci id="S2.E1.m2.3.3.2.2.3.3.1.3.cmml" xref="S2.E1.m2.3.3.2.2.3.3.1.3">𝑖</ci></apply><apply id="S2.E1.m2.3.3.2.2.3.3.2.cmml" xref="S2.E1.m2.3.3.2.2.3.3.2"><csymbol cd="ambiguous" id="S2.E1.m2.3.3.2.2.3.3.2.1.cmml" xref="S2.E1.m2.3.3.2.2.3.3.2">superscript</csymbol><apply id="S2.E1.m2.3.3.2.2.3.3.2.2.cmml" xref="S2.E1.m2.3.3.2.2.3.3.2"><csymbol cd="ambiguous" id="S2.E1.m2.3.3.2.2.3.3.2.2.1.cmml" xref="S2.E1.m2.3.3.2.2.3.3.2">subscript</csymbol><ci id="S2.E1.m2.3.3.2.2.3.3.2.2.2.cmml" xref="S2.E1.m2.3.3.2.2.3.3.2.2.2">𝜎</ci><ci id="S2.E1.m2.3.3.2.2.3.3.2.2.3.cmml" xref="S2.E1.m2.3.3.2.2.3.3.2.2.3">𝑖</ci></apply><cn type="integer" id="S2.E1.m2.3.3.2.2.3.3.2.3.cmml" xref="S2.E1.m2.3.3.2.2.3.3.2.3">2</cn></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E1.m2.3c">\displaystyle=\text{SVD}\bigg{(}\text{softmax}\Big{(}\frac{\textbf{Q}\textbf{K}^{T}}{\sqrt{D}}\Big{)}\bigg{)},\quad\text{Energy}_{i}=\frac{\sigma_{i}^{2}}{\sum_{i}\sigma_{i}^{2}}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
</div>
<div id="S2.SS0.SSS0.Px2.p3" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px2.p3.1" class="ltx_p">Figure <a href="#S2.F3" title="Figure 3 ‣ Singular Value Analysis ‣ 2 Observations ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> (a) illustrates the energy distribution of the singular values, plotted in log scale to enhance the visibility of differences among them.
Notably, only a few singular values exhibit high energy levels exceeding <math id="S2.SS0.SSS0.Px2.p3.1.m1.1" class="ltx_Math" alttext="0.01" display="inline"><semantics id="S2.SS0.SSS0.Px2.p3.1.m1.1a"><mn id="S2.SS0.SSS0.Px2.p3.1.m1.1.1" xref="S2.SS0.SSS0.Px2.p3.1.m1.1.1.cmml">0.01</mn><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p3.1.m1.1b"><cn type="float" id="S2.SS0.SSS0.Px2.p3.1.m1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p3.1.m1.1.1">0.01</cn></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p3.1.m1.1c">0.01</annotation></semantics></math> across all heads and layers, highlighting their relative significance.
This observation aligns with previous findings <cite class="ltx_cite ltx_citemacro_citep">(Bhojanapalli et al.,, <a href="#bib.bib3" title="" class="ltx_ref">2021</a>)</cite>, where a small subset of singular values often captures most of the information in attention mechanisms.
In addition, the rapid decay of the energy suggests that a low-rank approximation can effectively capture the essential information in the key cache.</p>
</div>
<div id="S2.SS0.SSS0.Px2.p4" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px2.p4.2" class="ltx_p">Figure <a href="#S2.F3" title="Figure 3 ‣ Singular Value Analysis ‣ 2 Observations ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> (b), the normalized cumulative energy sum reveals that the top <math id="S2.SS0.SSS0.Px2.p4.1.m1.1" class="ltx_Math" alttext="50" display="inline"><semantics id="S2.SS0.SSS0.Px2.p4.1.m1.1a"><mn id="S2.SS0.SSS0.Px2.p4.1.m1.1.1" xref="S2.SS0.SSS0.Px2.p4.1.m1.1.1.cmml">50</mn><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p4.1.m1.1b"><cn type="integer" id="S2.SS0.SSS0.Px2.p4.1.m1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p4.1.m1.1.1">50</cn></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p4.1.m1.1c">50</annotation></semantics></math> singular values account for over <math id="S2.SS0.SSS0.Px2.p4.2.m2.1" class="ltx_Math" alttext="90" display="inline"><semantics id="S2.SS0.SSS0.Px2.p4.2.m2.1a"><mn id="S2.SS0.SSS0.Px2.p4.2.m2.1.1" xref="S2.SS0.SSS0.Px2.p4.2.m2.1.1.cmml">90</mn><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p4.2.m2.1b"><cn type="integer" id="S2.SS0.SSS0.Px2.p4.2.m2.1.1.cmml" xref="S2.SS0.SSS0.Px2.p4.2.m2.1.1">90</cn></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p4.2.m2.1c">90</annotation></semantics></math>% of the total energy.
These findings suggest that the attention matrix is inherently low-rank <cite class="ltx_cite ltx_citemacro_citep">(Wang et al.,, <a href="#bib.bib45" title="" class="ltx_ref">2020</a>; <a href="#bib.bib8" title="" class="ltx_ref">Chen et al., 2021a, </a>; <a href="#bib.bib15" title="" class="ltx_ref">Dong et al., 2024b, </a>)</cite>, indicating that the key cache can be approximated using low-dimensional vectors <cite class="ltx_cite ltx_citemacro_citep">(Singhania et al.,, <a href="#bib.bib39" title="" class="ltx_ref">2024</a>)</cite>.</p>
</div>
<figure id="S2.F3" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<div id="S2.F3.1" class="ltx_block ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:208.1pt;">
<img src="/html/2407.21018/assets/x6.png" id="S2.F3.1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="272" alt="Refer to caption">
<p id="S2.F3.1.1" class="ltx_p ltx_align_center">(a) Energy of singular values</p>
</div>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<div id="S2.F3.2" class="ltx_block ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:208.1pt;">
<img src="/html/2407.21018/assets/x7.png" id="S2.F3.2.g1" class="ltx_graphics ltx_img_landscape" width="461" height="280" alt="Refer to caption">
<p id="S2.F3.2.1" class="ltx_p ltx_align_center">(b) Cumulative energy of singular values</p>
</div>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S2.F3.4.1.1" class="ltx_text" style="font-size:90%;">Figure 3</span>: </span><span id="S2.F3.5.2" class="ltx_text" style="font-size:90%;">The energy and cumulative energy of the singular values.</span></figcaption>
</figure>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>ThinK</h2>

<section id="S3.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Notations.</h4>

<div id="S3.SS0.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S3.SS0.SSS0.Px1.p1.9" class="ltx_p">We use uppercase letters (<span id="S3.SS0.SSS0.Px1.p1.9.1" class="ltx_text ltx_font_italic">e.g.</span>, <math id="S3.SS0.SSS0.Px1.p1.1.m1.2" class="ltx_Math" alttext="X,Y" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.1.m1.2a"><mrow id="S3.SS0.SSS0.Px1.p1.1.m1.2.3.2" xref="S3.SS0.SSS0.Px1.p1.1.m1.2.3.1.cmml"><mi id="S3.SS0.SSS0.Px1.p1.1.m1.1.1" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1.cmml">X</mi><mo id="S3.SS0.SSS0.Px1.p1.1.m1.2.3.2.1" xref="S3.SS0.SSS0.Px1.p1.1.m1.2.3.1.cmml">,</mo><mi id="S3.SS0.SSS0.Px1.p1.1.m1.2.2" xref="S3.SS0.SSS0.Px1.p1.1.m1.2.2.cmml">Y</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.1.m1.2b"><list id="S3.SS0.SSS0.Px1.p1.1.m1.2.3.1.cmml" xref="S3.SS0.SSS0.Px1.p1.1.m1.2.3.2"><ci id="S3.SS0.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1">𝑋</ci><ci id="S3.SS0.SSS0.Px1.p1.1.m1.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.1.m1.2.2">𝑌</ci></list></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.1.m1.2c">X,Y</annotation></semantics></math>) to denote scalar values and boldface uppercase letters (<span id="S3.SS0.SSS0.Px1.p1.9.2" class="ltx_text ltx_font_italic">e.g.</span>, <math id="S3.SS0.SSS0.Px1.p1.2.m2.2" class="ltx_Math" alttext="\mathbf{Q},\mathbf{K}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.2.m2.2a"><mrow id="S3.SS0.SSS0.Px1.p1.2.m2.2.3.2" xref="S3.SS0.SSS0.Px1.p1.2.m2.2.3.1.cmml"><mi id="S3.SS0.SSS0.Px1.p1.2.m2.1.1" xref="S3.SS0.SSS0.Px1.p1.2.m2.1.1.cmml">𝐐</mi><mo id="S3.SS0.SSS0.Px1.p1.2.m2.2.3.2.1" xref="S3.SS0.SSS0.Px1.p1.2.m2.2.3.1.cmml">,</mo><mi id="S3.SS0.SSS0.Px1.p1.2.m2.2.2" xref="S3.SS0.SSS0.Px1.p1.2.m2.2.2.cmml">𝐊</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.2.m2.2b"><list id="S3.SS0.SSS0.Px1.p1.2.m2.2.3.1.cmml" xref="S3.SS0.SSS0.Px1.p1.2.m2.2.3.2"><ci id="S3.SS0.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.2.m2.1.1">𝐐</ci><ci id="S3.SS0.SSS0.Px1.p1.2.m2.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.2.m2.2.2">𝐊</ci></list></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.2.m2.2c">\mathbf{Q},\mathbf{K}</annotation></semantics></math>) to denote matrices and tensors. The notation <math id="S3.SS0.SSS0.Px1.p1.3.m3.1" class="ltx_math_unparsed" alttext="\|\cdot\|_{p}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.3.m3.1a"><mrow id="S3.SS0.SSS0.Px1.p1.3.m3.1b"><mo rspace="0em" id="S3.SS0.SSS0.Px1.p1.3.m3.1.1">∥</mo><mo lspace="0em" rspace="0em" id="S3.SS0.SSS0.Px1.p1.3.m3.1.2">⋅</mo><msub id="S3.SS0.SSS0.Px1.p1.3.m3.1.3"><mo lspace="0em" id="S3.SS0.SSS0.Px1.p1.3.m3.1.3.2">∥</mo><mi id="S3.SS0.SSS0.Px1.p1.3.m3.1.3.3">p</mi></msub></mrow><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.3.m3.1c">\|\cdot\|_{p}</annotation></semantics></math> denotes the <math id="S3.SS0.SSS0.Px1.p1.4.m4.1" class="ltx_Math" alttext="l_{p}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.4.m4.1a"><msub id="S3.SS0.SSS0.Px1.p1.4.m4.1.1" xref="S3.SS0.SSS0.Px1.p1.4.m4.1.1.cmml"><mi id="S3.SS0.SSS0.Px1.p1.4.m4.1.1.2" xref="S3.SS0.SSS0.Px1.p1.4.m4.1.1.2.cmml">l</mi><mi id="S3.SS0.SSS0.Px1.p1.4.m4.1.1.3" xref="S3.SS0.SSS0.Px1.p1.4.m4.1.1.3.cmml">p</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.4.m4.1b"><apply id="S3.SS0.SSS0.Px1.p1.4.m4.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.4.m4.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.4.m4.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.4.m4.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.4.m4.1.1.2">𝑙</ci><ci id="S3.SS0.SSS0.Px1.p1.4.m4.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p1.4.m4.1.1.3">𝑝</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.4.m4.1c">l_{p}</annotation></semantics></math>-norm for vectors. Unless otherwise specified, <math id="S3.SS0.SSS0.Px1.p1.5.m5.1" class="ltx_math_unparsed" alttext="\|\cdot\|" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.5.m5.1a"><mrow id="S3.SS0.SSS0.Px1.p1.5.m5.1b"><mo rspace="0em" id="S3.SS0.SSS0.Px1.p1.5.m5.1.1">∥</mo><mo lspace="0em" rspace="0em" id="S3.SS0.SSS0.Px1.p1.5.m5.1.2">⋅</mo><mo lspace="0em" id="S3.SS0.SSS0.Px1.p1.5.m5.1.3">∥</mo></mrow><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.5.m5.1c">\|\cdot\|</annotation></semantics></math> denotes the <math id="S3.SS0.SSS0.Px1.p1.6.m6.1" class="ltx_Math" alttext="l_{2}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.6.m6.1a"><msub id="S3.SS0.SSS0.Px1.p1.6.m6.1.1" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.1.cmml"><mi id="S3.SS0.SSS0.Px1.p1.6.m6.1.1.2" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.1.2.cmml">l</mi><mn id="S3.SS0.SSS0.Px1.p1.6.m6.1.1.3" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.6.m6.1b"><apply id="S3.SS0.SSS0.Px1.p1.6.m6.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.6.m6.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.6.m6.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.1.2">𝑙</ci><cn type="integer" id="S3.SS0.SSS0.Px1.p1.6.m6.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.6.m6.1c">l_{2}</annotation></semantics></math>-norm. The Frobenius norm is denoted by <math id="S3.SS0.SSS0.Px1.p1.7.m7.1" class="ltx_math_unparsed" alttext="\|\cdot\|_{F}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.7.m7.1a"><mrow id="S3.SS0.SSS0.Px1.p1.7.m7.1b"><mo rspace="0em" id="S3.SS0.SSS0.Px1.p1.7.m7.1.1">∥</mo><mo lspace="0em" rspace="0em" id="S3.SS0.SSS0.Px1.p1.7.m7.1.2">⋅</mo><msub id="S3.SS0.SSS0.Px1.p1.7.m7.1.3"><mo lspace="0em" id="S3.SS0.SSS0.Px1.p1.7.m7.1.3.2">∥</mo><mi id="S3.SS0.SSS0.Px1.p1.7.m7.1.3.3">F</mi></msub></mrow><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.7.m7.1c">\|\cdot\|_{F}</annotation></semantics></math>. The floor function is denoted by <math id="S3.SS0.SSS0.Px1.p1.8.m8.1" class="ltx_Math" alttext="\lfloor\cdot\rfloor" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.8.m8.1a"><mrow id="S3.SS0.SSS0.Px1.p1.8.m8.1.2.2" xref="S3.SS0.SSS0.Px1.p1.8.m8.1.2.1.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.8.m8.1.2.2.1" xref="S3.SS0.SSS0.Px1.p1.8.m8.1.2.1.1.cmml">⌊</mo><mo lspace="0em" rspace="0em" id="S3.SS0.SSS0.Px1.p1.8.m8.1.1" xref="S3.SS0.SSS0.Px1.p1.8.m8.1.1.cmml">⋅</mo><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.8.m8.1.2.2.2" xref="S3.SS0.SSS0.Px1.p1.8.m8.1.2.1.1.cmml">⌋</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.8.m8.1b"><apply id="S3.SS0.SSS0.Px1.p1.8.m8.1.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m8.1.2.2"><floor id="S3.SS0.SSS0.Px1.p1.8.m8.1.2.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m8.1.2.2.1"></floor><ci id="S3.SS0.SSS0.Px1.p1.8.m8.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m8.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.8.m8.1c">\lfloor\cdot\rfloor</annotation></semantics></math>, and the ceiling function is denoted by <math id="S3.SS0.SSS0.Px1.p1.9.m9.1" class="ltx_Math" alttext="\lceil\cdot\rceil" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.9.m9.1a"><mrow id="S3.SS0.SSS0.Px1.p1.9.m9.1.2.2" xref="S3.SS0.SSS0.Px1.p1.9.m9.1.2.1.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.9.m9.1.2.2.1" xref="S3.SS0.SSS0.Px1.p1.9.m9.1.2.1.1.cmml">⌈</mo><mo lspace="0em" rspace="0em" id="S3.SS0.SSS0.Px1.p1.9.m9.1.1" xref="S3.SS0.SSS0.Px1.p1.9.m9.1.1.cmml">⋅</mo><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.9.m9.1.2.2.2" xref="S3.SS0.SSS0.Px1.p1.9.m9.1.2.1.1.cmml">⌉</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.9.m9.1b"><apply id="S3.SS0.SSS0.Px1.p1.9.m9.1.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.1.2.2"><ceiling id="S3.SS0.SSS0.Px1.p1.9.m9.1.2.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.1.2.2.1"></ceiling><ci id="S3.SS0.SSS0.Px1.p1.9.m9.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.9.m9.1c">\lceil\cdot\rceil</annotation></semantics></math>.</p>
</div>
</section>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Preliminary Study of KV Cache Optimization</h3>

<div id="S3.SS1.p1" class="ltx_para ltx_noindent">
<p id="S3.SS1.p1.1" class="ltx_p">In scenarios with extended contexts or batch processing, the main limitations in terms of memory and speed are due to the handling of the KV cache size. Considering a batch of requests to a Large Language Model (LLM) service that provides a long input prompt consisting of tokens <math id="S3.SS1.p1.1.m1.3" class="ltx_Math" alttext="[x_{B1},...,x_{BS}]" display="inline"><semantics id="S3.SS1.p1.1.m1.3a"><mrow id="S3.SS1.p1.1.m1.3.3.2" xref="S3.SS1.p1.1.m1.3.3.3.cmml"><mo stretchy="false" id="S3.SS1.p1.1.m1.3.3.2.3" xref="S3.SS1.p1.1.m1.3.3.3.cmml">[</mo><msub id="S3.SS1.p1.1.m1.2.2.1.1" xref="S3.SS1.p1.1.m1.2.2.1.1.cmml"><mi id="S3.SS1.p1.1.m1.2.2.1.1.2" xref="S3.SS1.p1.1.m1.2.2.1.1.2.cmml">x</mi><mrow id="S3.SS1.p1.1.m1.2.2.1.1.3" xref="S3.SS1.p1.1.m1.2.2.1.1.3.cmml"><mi id="S3.SS1.p1.1.m1.2.2.1.1.3.2" xref="S3.SS1.p1.1.m1.2.2.1.1.3.2.cmml">B</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.1.m1.2.2.1.1.3.1" xref="S3.SS1.p1.1.m1.2.2.1.1.3.1.cmml">​</mo><mn id="S3.SS1.p1.1.m1.2.2.1.1.3.3" xref="S3.SS1.p1.1.m1.2.2.1.1.3.3.cmml">1</mn></mrow></msub><mo id="S3.SS1.p1.1.m1.3.3.2.4" xref="S3.SS1.p1.1.m1.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S3.SS1.p1.1.m1.1.1" xref="S3.SS1.p1.1.m1.1.1.cmml">…</mi><mo id="S3.SS1.p1.1.m1.3.3.2.5" xref="S3.SS1.p1.1.m1.3.3.3.cmml">,</mo><msub id="S3.SS1.p1.1.m1.3.3.2.2" xref="S3.SS1.p1.1.m1.3.3.2.2.cmml"><mi id="S3.SS1.p1.1.m1.3.3.2.2.2" xref="S3.SS1.p1.1.m1.3.3.2.2.2.cmml">x</mi><mrow id="S3.SS1.p1.1.m1.3.3.2.2.3" xref="S3.SS1.p1.1.m1.3.3.2.2.3.cmml"><mi id="S3.SS1.p1.1.m1.3.3.2.2.3.2" xref="S3.SS1.p1.1.m1.3.3.2.2.3.2.cmml">B</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.1.m1.3.3.2.2.3.1" xref="S3.SS1.p1.1.m1.3.3.2.2.3.1.cmml">​</mo><mi id="S3.SS1.p1.1.m1.3.3.2.2.3.3" xref="S3.SS1.p1.1.m1.3.3.2.2.3.3.cmml">S</mi></mrow></msub><mo stretchy="false" id="S3.SS1.p1.1.m1.3.3.2.6" xref="S3.SS1.p1.1.m1.3.3.3.cmml">]</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.1.m1.3b"><list id="S3.SS1.p1.1.m1.3.3.3.cmml" xref="S3.SS1.p1.1.m1.3.3.2"><apply id="S3.SS1.p1.1.m1.2.2.1.1.cmml" xref="S3.SS1.p1.1.m1.2.2.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.1.m1.2.2.1.1.1.cmml" xref="S3.SS1.p1.1.m1.2.2.1.1">subscript</csymbol><ci id="S3.SS1.p1.1.m1.2.2.1.1.2.cmml" xref="S3.SS1.p1.1.m1.2.2.1.1.2">𝑥</ci><apply id="S3.SS1.p1.1.m1.2.2.1.1.3.cmml" xref="S3.SS1.p1.1.m1.2.2.1.1.3"><times id="S3.SS1.p1.1.m1.2.2.1.1.3.1.cmml" xref="S3.SS1.p1.1.m1.2.2.1.1.3.1"></times><ci id="S3.SS1.p1.1.m1.2.2.1.1.3.2.cmml" xref="S3.SS1.p1.1.m1.2.2.1.1.3.2">𝐵</ci><cn type="integer" id="S3.SS1.p1.1.m1.2.2.1.1.3.3.cmml" xref="S3.SS1.p1.1.m1.2.2.1.1.3.3">1</cn></apply></apply><ci id="S3.SS1.p1.1.m1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1">…</ci><apply id="S3.SS1.p1.1.m1.3.3.2.2.cmml" xref="S3.SS1.p1.1.m1.3.3.2.2"><csymbol cd="ambiguous" id="S3.SS1.p1.1.m1.3.3.2.2.1.cmml" xref="S3.SS1.p1.1.m1.3.3.2.2">subscript</csymbol><ci id="S3.SS1.p1.1.m1.3.3.2.2.2.cmml" xref="S3.SS1.p1.1.m1.3.3.2.2.2">𝑥</ci><apply id="S3.SS1.p1.1.m1.3.3.2.2.3.cmml" xref="S3.SS1.p1.1.m1.3.3.2.2.3"><times id="S3.SS1.p1.1.m1.3.3.2.2.3.1.cmml" xref="S3.SS1.p1.1.m1.3.3.2.2.3.1"></times><ci id="S3.SS1.p1.1.m1.3.3.2.2.3.2.cmml" xref="S3.SS1.p1.1.m1.3.3.2.2.3.2">𝐵</ci><ci id="S3.SS1.p1.1.m1.3.3.2.2.3.3.cmml" xref="S3.SS1.p1.1.m1.3.3.2.2.3.3">𝑆</ci></apply></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.1.m1.3c">[x_{B1},...,x_{BS}]</annotation></semantics></math>, the total KV cache size can be computed as follows:</p>
<table id="S3.E2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E2.m1.1" class="ltx_Math" alttext="2\times B\times S\times L\times N\times D," display="block"><semantics id="S3.E2.m1.1a"><mrow id="S3.E2.m1.1.1.1" xref="S3.E2.m1.1.1.1.1.cmml"><mrow id="S3.E2.m1.1.1.1.1" xref="S3.E2.m1.1.1.1.1.cmml"><mn id="S3.E2.m1.1.1.1.1.2" xref="S3.E2.m1.1.1.1.1.2.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="S3.E2.m1.1.1.1.1.1" xref="S3.E2.m1.1.1.1.1.1.cmml">×</mo><mi id="S3.E2.m1.1.1.1.1.3" xref="S3.E2.m1.1.1.1.1.3.cmml">B</mi><mo lspace="0.222em" rspace="0.222em" id="S3.E2.m1.1.1.1.1.1a" xref="S3.E2.m1.1.1.1.1.1.cmml">×</mo><mi id="S3.E2.m1.1.1.1.1.4" xref="S3.E2.m1.1.1.1.1.4.cmml">S</mi><mo lspace="0.222em" rspace="0.222em" id="S3.E2.m1.1.1.1.1.1b" xref="S3.E2.m1.1.1.1.1.1.cmml">×</mo><mi id="S3.E2.m1.1.1.1.1.5" xref="S3.E2.m1.1.1.1.1.5.cmml">L</mi><mo lspace="0.222em" rspace="0.222em" id="S3.E2.m1.1.1.1.1.1c" xref="S3.E2.m1.1.1.1.1.1.cmml">×</mo><mi id="S3.E2.m1.1.1.1.1.6" xref="S3.E2.m1.1.1.1.1.6.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="S3.E2.m1.1.1.1.1.1d" xref="S3.E2.m1.1.1.1.1.1.cmml">×</mo><mi id="S3.E2.m1.1.1.1.1.7" xref="S3.E2.m1.1.1.1.1.7.cmml">D</mi></mrow><mo id="S3.E2.m1.1.1.1.2" xref="S3.E2.m1.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E2.m1.1b"><apply id="S3.E2.m1.1.1.1.1.cmml" xref="S3.E2.m1.1.1.1"><times id="S3.E2.m1.1.1.1.1.1.cmml" xref="S3.E2.m1.1.1.1.1.1"></times><cn type="integer" id="S3.E2.m1.1.1.1.1.2.cmml" xref="S3.E2.m1.1.1.1.1.2">2</cn><ci id="S3.E2.m1.1.1.1.1.3.cmml" xref="S3.E2.m1.1.1.1.1.3">𝐵</ci><ci id="S3.E2.m1.1.1.1.1.4.cmml" xref="S3.E2.m1.1.1.1.1.4">𝑆</ci><ci id="S3.E2.m1.1.1.1.1.5.cmml" xref="S3.E2.m1.1.1.1.1.5">𝐿</ci><ci id="S3.E2.m1.1.1.1.1.6.cmml" xref="S3.E2.m1.1.1.1.1.6">𝑁</ci><ci id="S3.E2.m1.1.1.1.1.7.cmml" xref="S3.E2.m1.1.1.1.1.7">𝐷</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E2.m1.1c">2\times B\times S\times L\times N\times D,</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
<p id="S3.SS1.p1.11" class="ltx_p">where <math id="S3.SS1.p1.2.m1.1" class="ltx_Math" alttext="L" display="inline"><semantics id="S3.SS1.p1.2.m1.1a"><mi id="S3.SS1.p1.2.m1.1.1" xref="S3.SS1.p1.2.m1.1.1.cmml">L</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.2.m1.1b"><ci id="S3.SS1.p1.2.m1.1.1.cmml" xref="S3.SS1.p1.2.m1.1.1">𝐿</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.2.m1.1c">L</annotation></semantics></math> is the number of layers, <math id="S3.SS1.p1.3.m2.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S3.SS1.p1.3.m2.1a"><mi id="S3.SS1.p1.3.m2.1.1" xref="S3.SS1.p1.3.m2.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.3.m2.1b"><ci id="S3.SS1.p1.3.m2.1.1.cmml" xref="S3.SS1.p1.3.m2.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.3.m2.1c">N</annotation></semantics></math> is the number of heads, <math id="S3.SS1.p1.4.m3.1" class="ltx_Math" alttext="D" display="inline"><semantics id="S3.SS1.p1.4.m3.1a"><mi id="S3.SS1.p1.4.m3.1.1" xref="S3.SS1.p1.4.m3.1.1.cmml">D</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.4.m3.1b"><ci id="S3.SS1.p1.4.m3.1.1.cmml" xref="S3.SS1.p1.4.m3.1.1">𝐷</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.4.m3.1c">D</annotation></semantics></math> is the head dimension. The KV cach size grows linearly as the batch size <math id="S3.SS1.p1.5.m4.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S3.SS1.p1.5.m4.1a"><mi id="S3.SS1.p1.5.m4.1.1" xref="S3.SS1.p1.5.m4.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.5.m4.1b"><ci id="S3.SS1.p1.5.m4.1.1.cmml" xref="S3.SS1.p1.5.m4.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.5.m4.1c">B</annotation></semantics></math> and sequence length <math id="S3.SS1.p1.6.m5.1" class="ltx_Math" alttext="S" display="inline"><semantics id="S3.SS1.p1.6.m5.1a"><mi id="S3.SS1.p1.6.m5.1.1" xref="S3.SS1.p1.6.m5.1.1.cmml">S</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.6.m5.1b"><ci id="S3.SS1.p1.6.m5.1.1.cmml" xref="S3.SS1.p1.6.m5.1.1">𝑆</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.6.m5.1c">S</annotation></semantics></math>. For
a model with multihead attention (MHA) <cite class="ltx_cite ltx_citemacro_citep">(Vaswani et al.,, <a href="#bib.bib43" title="" class="ltx_ref">2017</a>)</cite>, such as LLaMA2-7B <cite class="ltx_cite ltx_citemacro_citep">(<a href="#bib.bib42" title="" class="ltx_ref">Touvron et al., 2023b, </a>)</cite>, a context length of <math id="S3.SS1.p1.7.m6.1" class="ltx_Math" alttext="2048" display="inline"><semantics id="S3.SS1.p1.7.m6.1a"><mn id="S3.SS1.p1.7.m6.1.1" xref="S3.SS1.p1.7.m6.1.1.cmml">2048</mn><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.7.m6.1b"><cn type="integer" id="S3.SS1.p1.7.m6.1.1.cmml" xref="S3.SS1.p1.7.m6.1.1">2048</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.7.m6.1c">2048</annotation></semantics></math> and a batch size of <math id="S3.SS1.p1.8.m7.1" class="ltx_Math" alttext="13" display="inline"><semantics id="S3.SS1.p1.8.m7.1a"><mn id="S3.SS1.p1.8.m7.1.1" xref="S3.SS1.p1.8.m7.1.1.cmml">13</mn><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.8.m7.1b"><cn type="integer" id="S3.SS1.p1.8.m7.1.1.cmml" xref="S3.SS1.p1.8.m7.1.1">13</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.8.m7.1c">13</annotation></semantics></math> require storing a <math id="S3.SS1.p1.9.m8.1" class="ltx_Math" alttext="13" display="inline"><semantics id="S3.SS1.p1.9.m8.1a"><mn id="S3.SS1.p1.9.m8.1.1" xref="S3.SS1.p1.9.m8.1.1.cmml">13</mn><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.9.m8.1b"><cn type="integer" id="S3.SS1.p1.9.m8.1.1.cmml" xref="S3.SS1.p1.9.m8.1.1">13</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.9.m8.1c">13</annotation></semantics></math> GB KV cache, which is equivalent to the size of the model parameters.
The KV cache must be transferred from off-chip memory (HBM) <cite class="ltx_cite ltx_citemacro_citep">(Jia et al.,, <a href="#bib.bib23" title="" class="ltx_ref">2018</a>)</cite> to on-chip memory (cache) for each token generated, leading to a memory bottleneck.
This substantial memory demand highlights the challenges in managing large-scale models and the need for efficient memory utilization strategies. Current methods optimize the KV cache based on the sequence length <math id="S3.SS1.p1.10.m9.1" class="ltx_Math" alttext="S" display="inline"><semantics id="S3.SS1.p1.10.m9.1a"><mi id="S3.SS1.p1.10.m9.1.1" xref="S3.SS1.p1.10.m9.1.1.cmml">S</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.10.m9.1b"><ci id="S3.SS1.p1.10.m9.1.1.cmml" xref="S3.SS1.p1.10.m9.1.1">𝑆</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.10.m9.1c">S</annotation></semantics></math> <cite class="ltx_cite ltx_citemacro_citep">(<a href="#bib.bib49" title="" class="ltx_ref">Xiao et al., 2023b, </a>; <a href="#bib.bib56" title="" class="ltx_ref">Zhang et al., 2024c, </a>; Li et al.,, <a href="#bib.bib29" title="" class="ltx_ref">2024</a>)</cite> and precision <cite class="ltx_cite ltx_citemacro_citep">(Hooper et al.,, <a href="#bib.bib21" title="" class="ltx_ref">2024</a>; <a href="#bib.bib32" title="" class="ltx_ref">Liu et al., 2024b, </a>)</cite>.
We will introduce a new method, <span id="S3.SS1.p1.11.1" class="ltx_text ltx_font_bold">ThinK</span>, to optimize it from the perspective of the number of head dimensions <math id="S3.SS1.p1.11.m10.1" class="ltx_Math" alttext="D" display="inline"><semantics id="S3.SS1.p1.11.m10.1a"><mi id="S3.SS1.p1.11.m10.1.1" xref="S3.SS1.p1.11.m10.1.1.cmml">D</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.11.m10.1b"><ci id="S3.SS1.p1.11.m10.1.1.cmml" xref="S3.SS1.p1.11.m10.1.1">𝐷</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.11.m10.1c">D</annotation></semantics></math>.</p>
</div>
<figure id="S3.T1" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S3.T1.13.2.1" class="ltx_text" style="font-size:90%;">Table 1</span>: </span><span id="S3.T1.2.1" class="ltx_text" style="font-size:90%;">Performance comparison of pruning K cache by <math id="S3.T1.2.1.m1.1" class="ltx_Math" alttext="l_{p}" display="inline"><semantics id="S3.T1.2.1.m1.1b"><msub id="S3.T1.2.1.m1.1.1" xref="S3.T1.2.1.m1.1.1.cmml"><mi id="S3.T1.2.1.m1.1.1.2" xref="S3.T1.2.1.m1.1.1.2.cmml">l</mi><mi id="S3.T1.2.1.m1.1.1.3" xref="S3.T1.2.1.m1.1.1.3.cmml">p</mi></msub><annotation-xml encoding="MathML-Content" id="S3.T1.2.1.m1.1c"><apply id="S3.T1.2.1.m1.1.1.cmml" xref="S3.T1.2.1.m1.1.1"><csymbol cd="ambiguous" id="S3.T1.2.1.m1.1.1.1.cmml" xref="S3.T1.2.1.m1.1.1">subscript</csymbol><ci id="S3.T1.2.1.m1.1.1.2.cmml" xref="S3.T1.2.1.m1.1.1.2">𝑙</ci><ci id="S3.T1.2.1.m1.1.1.3.cmml" xref="S3.T1.2.1.m1.1.1.3">𝑝</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.2.1.m1.1d">l_{p}</annotation></semantics></math> norm on LongBench.</span></figcaption>
<div id="S3.T1.11" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:148pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-121.6pt,41.5pt) scale(0.64065237982904,0.64065237982904) ;">
<table id="S3.T1.11.9" class="ltx_tabular ltx_align_middle">
<tr id="S3.T1.3.1.1" class="ltx_tr">
<td id="S3.T1.3.1.1.2" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt" rowspan="2"><span id="S3.T1.3.1.1.2.1" class="ltx_text">Method</span></td>
<td id="S3.T1.3.1.1.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_tt" rowspan="2"><span id="S3.T1.3.1.1.1.1" class="ltx_text"><math id="S3.T1.3.1.1.1.1.m1.1" class="ltx_Math" alttext="\lambda" display="inline"><semantics id="S3.T1.3.1.1.1.1.m1.1a"><mi id="S3.T1.3.1.1.1.1.m1.1.1" xref="S3.T1.3.1.1.1.1.m1.1.1.cmml">λ</mi><annotation-xml encoding="MathML-Content" id="S3.T1.3.1.1.1.1.m1.1b"><ci id="S3.T1.3.1.1.1.1.m1.1.1.cmml" xref="S3.T1.3.1.1.1.1.m1.1.1">𝜆</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.3.1.1.1.1.m1.1c">\lambda</annotation></semantics></math></span></td>
<td id="S3.T1.3.1.1.3" class="ltx_td ltx_align_center ltx_border_tt" colspan="3">Single-Document QA</td>
<td id="S3.T1.3.1.1.4" class="ltx_td ltx_align_center ltx_border_tt" colspan="3">Multi-Document QA</td>
<td id="S3.T1.3.1.1.5" class="ltx_td ltx_align_center ltx_border_tt" colspan="3">Summarization</td>
<td id="S3.T1.3.1.1.6" class="ltx_td ltx_align_center ltx_border_tt" colspan="3">Few-shot Learning</td>
<td id="S3.T1.3.1.1.7" class="ltx_td ltx_align_center ltx_border_tt" colspan="2">Synthetic</td>
<td id="S3.T1.3.1.1.8" class="ltx_td ltx_align_center ltx_border_tt" colspan="2">Code</td>
<td id="S3.T1.3.1.1.9" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_tt" rowspan="2"><span id="S3.T1.3.1.1.9.1" class="ltx_text">Avg.</span></td>
</tr>
<tr id="S3.T1.11.9.10" class="ltx_tr">
<td id="S3.T1.11.9.10.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S3.T1.11.9.10.1.1" class="ltx_inline-block ltx_transformed_outer" style="width:35.4pt;height:25.5pt;vertical-align:-10.3pt;"><span class="ltx_transformed_inner" style="width:35.9pt;transform:translate(-0.21pt,2.92pt) rotate(-30deg) ;">
<p id="S3.T1.11.9.10.1.1.1" class="ltx_p">NrtvQA</p>
</span></div>
</td>
<td id="S3.T1.11.9.10.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S3.T1.11.9.10.2.1" class="ltx_inline-block ltx_transformed_outer" style="width:31.2pt;height:23.1pt;vertical-align:-9.1pt;"><span class="ltx_transformed_inner" style="width:30.9pt;transform:translate(0.12pt,2.92pt) rotate(-30deg) ;">
<p id="S3.T1.11.9.10.2.1.1" class="ltx_p">Qasper</p>
</span></div>
</td>
<td id="S3.T1.11.9.10.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S3.T1.11.9.10.3.1" class="ltx_inline-block ltx_transformed_outer" style="width:28.6pt;height:20.4pt;vertical-align:-6.8pt;"><span class="ltx_transformed_inner" style="width:29.0pt;transform:translate(-0.24pt,0pt) rotate(-30deg) ;">
<p id="S3.T1.11.9.10.3.1.1" class="ltx_p">MF-en</p>
</span></div>
</td>
<td id="S3.T1.11.9.10.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S3.T1.11.9.10.4.1" class="ltx_inline-block ltx_transformed_outer" style="width:44.6pt;height:30.8pt;vertical-align:-13.0pt;"><span class="ltx_transformed_inner" style="width:46.4pt;transform:translate(-0.91pt,2.92pt) rotate(-30deg) ;">
<p id="S3.T1.11.9.10.4.1.1" class="ltx_p">HotpotQA</p>
</span></div>
</td>
<td id="S3.T1.11.9.10.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S3.T1.11.9.10.5.1" class="ltx_inline-block ltx_transformed_outer" style="width:48.2pt;height:33pt;vertical-align:-14.0pt;"><span class="ltx_transformed_inner" style="width:50.6pt;transform:translate(-1.16pt,2.92pt) rotate(-30deg) ;">
<p id="S3.T1.11.9.10.5.1.1" class="ltx_p">2WikiMQA</p>
</span></div>
</td>
<td id="S3.T1.11.9.10.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S3.T1.11.9.10.6.1" class="ltx_inline-block ltx_transformed_outer" style="width:36.2pt;height:25.9pt;vertical-align:-10.5pt;"><span class="ltx_transformed_inner" style="width:36.7pt;transform:translate(-0.27pt,2.92pt) rotate(-30deg) ;">
<p id="S3.T1.11.9.10.6.1.1" class="ltx_p">Musique</p>
</span></div>
</td>
<td id="S3.T1.11.9.10.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S3.T1.11.9.10.7.1" class="ltx_inline-block ltx_transformed_outer" style="width:46.2pt;height:31.7pt;vertical-align:-13.4pt;"><span class="ltx_transformed_inner" style="width:48.3pt;transform:translate(-1.04pt,2.92pt) rotate(-30deg) ;">
<p id="S3.T1.11.9.10.7.1.1" class="ltx_p">GovReport</p>
</span></div>
</td>
<td id="S3.T1.11.9.10.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S3.T1.11.9.10.8.1" class="ltx_inline-block ltx_transformed_outer" style="width:35.9pt;height:25.8pt;vertical-align:-10.5pt;"><span class="ltx_transformed_inner" style="width:36.4pt;transform:translate(-0.24pt,2.92pt) rotate(-30deg) ;">
<p id="S3.T1.11.9.10.8.1.1" class="ltx_p">QMSum</p>
</span></div>
</td>
<td id="S3.T1.11.9.10.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S3.T1.11.9.10.9.1" class="ltx_inline-block ltx_transformed_outer" style="width:44.4pt;height:29.7pt;vertical-align:-11.4pt;"><span class="ltx_transformed_inner" style="width:47.3pt;transform:translate(-1.43pt,0pt) rotate(-30deg) ;">
<p id="S3.T1.11.9.10.9.1.1" class="ltx_p">MultiNews</p>
</span></div>
</td>
<td id="S3.T1.11.9.10.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S3.T1.11.9.10.10.1" class="ltx_inline-block ltx_transformed_outer" style="width:28.2pt;height:20.2pt;vertical-align:-6.7pt;"><span class="ltx_transformed_inner" style="width:28.6pt;transform:translate(-0.21pt,0pt) rotate(-30deg) ;">
<p id="S3.T1.11.9.10.10.1.1" class="ltx_p">TREC</p>
</span></div>
</td>
<td id="S3.T1.11.9.10.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S3.T1.11.9.10.11.1" class="ltx_inline-block ltx_transformed_outer" style="width:40.3pt;height:28.3pt;vertical-align:-11.7pt;"><span class="ltx_transformed_inner" style="width:41.4pt;transform:translate(-0.58pt,2.92pt) rotate(-30deg) ;">
<p id="S3.T1.11.9.10.11.1.1" class="ltx_p">TriviaQA</p>
</span></div>
</td>
<td id="S3.T1.11.9.10.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S3.T1.11.9.10.12.1" class="ltx_inline-block ltx_transformed_outer" style="width:39.5pt;height:26.8pt;vertical-align:-10.0pt;"><span class="ltx_transformed_inner" style="width:41.7pt;transform:translate(-1.08pt,0pt) rotate(-30deg) ;">
<p id="S3.T1.11.9.10.12.1.1" class="ltx_p">SAMSum</p>
</span></div>
</td>
<td id="S3.T1.11.9.10.13" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_t">
<div id="S3.T1.11.9.10.13.1" class="ltx_inline-block ltx_transformed_outer" style="width:32.6pt;height:22.8pt;vertical-align:-8.0pt;"><span class="ltx_transformed_inner" style="width:33.8pt;transform:translate(-0.55pt,0pt) rotate(-30deg) ;">
<p id="S3.T1.11.9.10.13.1.1" class="ltx_p">PCount</p>
</span></div>
</td>
<td id="S3.T1.11.9.10.14" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_t">
<div id="S3.T1.11.9.10.14.1" class="ltx_inline-block ltx_transformed_outer" style="width:28.2pt;height:20.2pt;vertical-align:-6.7pt;"><span class="ltx_transformed_inner" style="width:28.6pt;transform:translate(-0.21pt,0pt) rotate(-30deg) ;">
<p id="S3.T1.11.9.10.14.1.1" class="ltx_p">PRe</p>
</span></div>
</td>
<td id="S3.T1.11.9.10.15" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_t">
<div id="S3.T1.11.9.10.15.1" class="ltx_inline-block ltx_transformed_outer" style="width:25.2pt;height:18.5pt;vertical-align:-5.8pt;"><span class="ltx_transformed_inner" style="width:25.1pt;transform:translate(0.03pt,0pt) rotate(-30deg) ;">
<p id="S3.T1.11.9.10.15.1.1" class="ltx_p">Lcc</p>
</span></div>
</td>
<td id="S3.T1.11.9.10.16" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_t">
<div id="S3.T1.11.9.10.16.1" class="ltx_inline-block ltx_transformed_outer" style="width:30.5pt;height:21.6pt;vertical-align:-7.4pt;"><span class="ltx_transformed_inner" style="width:31.2pt;transform:translate(-0.38pt,0pt) rotate(-30deg) ;">
<p id="S3.T1.11.9.10.16.1.1" class="ltx_p">RB-P</p>
</span></div>
</td>
</tr>
<tr id="S3.T1.11.9.11" class="ltx_tr">
<td id="S3.T1.11.9.11.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">H2O</td>
<td id="S3.T1.11.9.11.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">0</td>
<td id="S3.T1.11.9.11.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">23.52</td>
<td id="S3.T1.11.9.11.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">17.93</td>
<td id="S3.T1.11.9.11.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">34.68</td>
<td id="S3.T1.11.9.11.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">42.11</td>
<td id="S3.T1.11.9.11.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">33.52</td>
<td id="S3.T1.11.9.11.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">19.92</td>
<td id="S3.T1.11.9.11.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">22.11</td>
<td id="S3.T1.11.9.11.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">22.56</td>
<td id="S3.T1.11.9.11.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">23.82</td>
<td id="S3.T1.11.9.11.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">41.00</td>
<td id="S3.T1.11.9.11.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">90.46</td>
<td id="S3.T1.11.9.11.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">40.20</td>
<td id="S3.T1.11.9.11.15" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_t">5.87</td>
<td id="S3.T1.11.9.11.16" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_t">69.50</td>
<td id="S3.T1.11.9.11.17" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_t">56.71</td>
<td id="S3.T1.11.9.11.18" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_t">51.69</td>
<td id="S3.T1.11.9.11.19" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_t">37.23</td>
</tr>
<tr id="S3.T1.4.2.2" class="ltx_tr">
<td id="S3.T1.4.2.2.1" class="ltx_td ltx_nopad_r ltx_align_left">+<math id="S3.T1.4.2.2.1.m1.1" class="ltx_Math" alttext="l_{1}" display="inline"><semantics id="S3.T1.4.2.2.1.m1.1a"><msub id="S3.T1.4.2.2.1.m1.1.1" xref="S3.T1.4.2.2.1.m1.1.1.cmml"><mi id="S3.T1.4.2.2.1.m1.1.1.2" xref="S3.T1.4.2.2.1.m1.1.1.2.cmml">l</mi><mn id="S3.T1.4.2.2.1.m1.1.1.3" xref="S3.T1.4.2.2.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S3.T1.4.2.2.1.m1.1b"><apply id="S3.T1.4.2.2.1.m1.1.1.cmml" xref="S3.T1.4.2.2.1.m1.1.1"><csymbol cd="ambiguous" id="S3.T1.4.2.2.1.m1.1.1.1.cmml" xref="S3.T1.4.2.2.1.m1.1.1">subscript</csymbol><ci id="S3.T1.4.2.2.1.m1.1.1.2.cmml" xref="S3.T1.4.2.2.1.m1.1.1.2">𝑙</ci><cn type="integer" id="S3.T1.4.2.2.1.m1.1.1.3.cmml" xref="S3.T1.4.2.2.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.4.2.2.1.m1.1c">l_{1}</annotation></semantics></math>
</td>
<td id="S3.T1.4.2.2.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">30</td>
<td id="S3.T1.4.2.2.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">23.38</td>
<td id="S3.T1.4.2.2.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">17.15</td>
<td id="S3.T1.4.2.2.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">34.99</td>
<td id="S3.T1.4.2.2.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">40.56</td>
<td id="S3.T1.4.2.2.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">31.49</td>
<td id="S3.T1.4.2.2.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">19.90</td>
<td id="S3.T1.4.2.2.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">21.37</td>
<td id="S3.T1.4.2.2.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">22.13</td>
<td id="S3.T1.4.2.2.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">23.44</td>
<td id="S3.T1.4.2.2.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">40.50</td>
<td id="S3.T1.4.2.2.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">90.10</td>
<td id="S3.T1.4.2.2.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">40.65</td>
<td id="S3.T1.4.2.2.15" class="ltx_td ltx_nopad_l ltx_align_center">5.41</td>
<td id="S3.T1.4.2.2.16" class="ltx_td ltx_nopad_l ltx_align_center">69.00</td>
<td id="S3.T1.4.2.2.17" class="ltx_td ltx_nopad_l ltx_align_center">58.64</td>
<td id="S3.T1.4.2.2.18" class="ltx_td ltx_nopad_l ltx_align_center">54.99</td>
<td id="S3.T1.4.2.2.19" class="ltx_td ltx_nopad_l ltx_align_center">37.11</td>
</tr>
<tr id="S3.T1.5.3.3" class="ltx_tr">
<td id="S3.T1.5.3.3.1" class="ltx_td ltx_nopad_r ltx_align_left">+<math id="S3.T1.5.3.3.1.m1.1" class="ltx_Math" alttext="l_{1}" display="inline"><semantics id="S3.T1.5.3.3.1.m1.1a"><msub id="S3.T1.5.3.3.1.m1.1.1" xref="S3.T1.5.3.3.1.m1.1.1.cmml"><mi id="S3.T1.5.3.3.1.m1.1.1.2" xref="S3.T1.5.3.3.1.m1.1.1.2.cmml">l</mi><mn id="S3.T1.5.3.3.1.m1.1.1.3" xref="S3.T1.5.3.3.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S3.T1.5.3.3.1.m1.1b"><apply id="S3.T1.5.3.3.1.m1.1.1.cmml" xref="S3.T1.5.3.3.1.m1.1.1"><csymbol cd="ambiguous" id="S3.T1.5.3.3.1.m1.1.1.1.cmml" xref="S3.T1.5.3.3.1.m1.1.1">subscript</csymbol><ci id="S3.T1.5.3.3.1.m1.1.1.2.cmml" xref="S3.T1.5.3.3.1.m1.1.1.2">𝑙</ci><cn type="integer" id="S3.T1.5.3.3.1.m1.1.1.3.cmml" xref="S3.T1.5.3.3.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.5.3.3.1.m1.1c">l_{1}</annotation></semantics></math>
</td>
<td id="S3.T1.5.3.3.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">40</td>
<td id="S3.T1.5.3.3.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">23.51</td>
<td id="S3.T1.5.3.3.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">15.40</td>
<td id="S3.T1.5.3.3.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">34.37</td>
<td id="S3.T1.5.3.3.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">40.71</td>
<td id="S3.T1.5.3.3.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">31.28</td>
<td id="S3.T1.5.3.3.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">20.24</td>
<td id="S3.T1.5.3.3.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">21.25</td>
<td id="S3.T1.5.3.3.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">22.29</td>
<td id="S3.T1.5.3.3.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">22.54</td>
<td id="S3.T1.5.3.3.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">38.50</td>
<td id="S3.T1.5.3.3.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">89.22</td>
<td id="S3.T1.5.3.3.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">39.27</td>
<td id="S3.T1.5.3.3.15" class="ltx_td ltx_nopad_l ltx_align_center">5.87</td>
<td id="S3.T1.5.3.3.16" class="ltx_td ltx_nopad_l ltx_align_center">68.33</td>
<td id="S3.T1.5.3.3.17" class="ltx_td ltx_nopad_l ltx_align_center">58.47</td>
<td id="S3.T1.5.3.3.18" class="ltx_td ltx_nopad_l ltx_align_center">54.33</td>
<td id="S3.T1.5.3.3.19" class="ltx_td ltx_nopad_l ltx_align_center">36.60</td>
</tr>
<tr id="S3.T1.6.4.4" class="ltx_tr">
<td id="S3.T1.6.4.4.1" class="ltx_td ltx_nopad_r ltx_align_left">+<math id="S3.T1.6.4.4.1.m1.1" class="ltx_Math" alttext="l_{2}" display="inline"><semantics id="S3.T1.6.4.4.1.m1.1a"><msub id="S3.T1.6.4.4.1.m1.1.1" xref="S3.T1.6.4.4.1.m1.1.1.cmml"><mi id="S3.T1.6.4.4.1.m1.1.1.2" xref="S3.T1.6.4.4.1.m1.1.1.2.cmml">l</mi><mn id="S3.T1.6.4.4.1.m1.1.1.3" xref="S3.T1.6.4.4.1.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S3.T1.6.4.4.1.m1.1b"><apply id="S3.T1.6.4.4.1.m1.1.1.cmml" xref="S3.T1.6.4.4.1.m1.1.1"><csymbol cd="ambiguous" id="S3.T1.6.4.4.1.m1.1.1.1.cmml" xref="S3.T1.6.4.4.1.m1.1.1">subscript</csymbol><ci id="S3.T1.6.4.4.1.m1.1.1.2.cmml" xref="S3.T1.6.4.4.1.m1.1.1.2">𝑙</ci><cn type="integer" id="S3.T1.6.4.4.1.m1.1.1.3.cmml" xref="S3.T1.6.4.4.1.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.6.4.4.1.m1.1c">l_{2}</annotation></semantics></math>
</td>
<td id="S3.T1.6.4.4.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">30</td>
<td id="S3.T1.6.4.4.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">23.98</td>
<td id="S3.T1.6.4.4.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">17.04</td>
<td id="S3.T1.6.4.4.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">35.19</td>
<td id="S3.T1.6.4.4.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">39.27</td>
<td id="S3.T1.6.4.4.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">31.29</td>
<td id="S3.T1.6.4.4.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">20.4</td>
<td id="S3.T1.6.4.4.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">21.62</td>
<td id="S3.T1.6.4.4.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">22.46</td>
<td id="S3.T1.6.4.4.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">23.34</td>
<td id="S3.T1.6.4.4.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">40.5</td>
<td id="S3.T1.6.4.4.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">89.75</td>
<td id="S3.T1.6.4.4.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">40.71</td>
<td id="S3.T1.6.4.4.15" class="ltx_td ltx_nopad_l ltx_align_center">5.54</td>
<td id="S3.T1.6.4.4.16" class="ltx_td ltx_nopad_l ltx_align_center">68.67</td>
<td id="S3.T1.6.4.4.17" class="ltx_td ltx_nopad_l ltx_align_center">60.12</td>
<td id="S3.T1.6.4.4.18" class="ltx_td ltx_nopad_l ltx_align_center">58.52</td>
<td id="S3.T1.6.4.4.19" class="ltx_td ltx_nopad_l ltx_align_center">37.40</td>
</tr>
<tr id="S3.T1.7.5.5" class="ltx_tr">
<td id="S3.T1.7.5.5.1" class="ltx_td ltx_nopad_r ltx_align_left">+<math id="S3.T1.7.5.5.1.m1.1" class="ltx_Math" alttext="l_{2}" display="inline"><semantics id="S3.T1.7.5.5.1.m1.1a"><msub id="S3.T1.7.5.5.1.m1.1.1" xref="S3.T1.7.5.5.1.m1.1.1.cmml"><mi id="S3.T1.7.5.5.1.m1.1.1.2" xref="S3.T1.7.5.5.1.m1.1.1.2.cmml">l</mi><mn id="S3.T1.7.5.5.1.m1.1.1.3" xref="S3.T1.7.5.5.1.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S3.T1.7.5.5.1.m1.1b"><apply id="S3.T1.7.5.5.1.m1.1.1.cmml" xref="S3.T1.7.5.5.1.m1.1.1"><csymbol cd="ambiguous" id="S3.T1.7.5.5.1.m1.1.1.1.cmml" xref="S3.T1.7.5.5.1.m1.1.1">subscript</csymbol><ci id="S3.T1.7.5.5.1.m1.1.1.2.cmml" xref="S3.T1.7.5.5.1.m1.1.1.2">𝑙</ci><cn type="integer" id="S3.T1.7.5.5.1.m1.1.1.3.cmml" xref="S3.T1.7.5.5.1.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.7.5.5.1.m1.1c">l_{2}</annotation></semantics></math>
</td>
<td id="S3.T1.7.5.5.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">40</td>
<td id="S3.T1.7.5.5.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">23.76</td>
<td id="S3.T1.7.5.5.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">16.23</td>
<td id="S3.T1.7.5.5.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">32.19</td>
<td id="S3.T1.7.5.5.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">40.23</td>
<td id="S3.T1.7.5.5.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">32.13</td>
<td id="S3.T1.7.5.5.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">20.69</td>
<td id="S3.T1.7.5.5.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">21.3</td>
<td id="S3.T1.7.5.5.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">22.25</td>
<td id="S3.T1.7.5.5.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">23.2</td>
<td id="S3.T1.7.5.5.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">39.5</td>
<td id="S3.T1.7.5.5.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">89.61</td>
<td id="S3.T1.7.5.5.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">40.24</td>
<td id="S3.T1.7.5.5.15" class="ltx_td ltx_nopad_l ltx_align_center">5.66</td>
<td id="S3.T1.7.5.5.16" class="ltx_td ltx_nopad_l ltx_align_center">69.0</td>
<td id="S3.T1.7.5.5.17" class="ltx_td ltx_nopad_l ltx_align_center">60.09</td>
<td id="S3.T1.7.5.5.18" class="ltx_td ltx_nopad_l ltx_align_center">59.45</td>
<td id="S3.T1.7.5.5.19" class="ltx_td ltx_nopad_l ltx_align_center">37.22</td>
</tr>
<tr id="S3.T1.11.9.12" class="ltx_tr">
<td id="S3.T1.11.9.12.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">SnapKV</td>
<td id="S3.T1.11.9.12.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">0</td>
<td id="S3.T1.11.9.12.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">24.84</td>
<td id="S3.T1.11.9.12.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">23.96</td>
<td id="S3.T1.11.9.12.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">38.77</td>
<td id="S3.T1.11.9.12.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">42.75</td>
<td id="S3.T1.11.9.12.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">34.55</td>
<td id="S3.T1.11.9.12.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">20.87</td>
<td id="S3.T1.11.9.12.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">22.26</td>
<td id="S3.T1.11.9.12.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">22.61</td>
<td id="S3.T1.11.9.12.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">23.97</td>
<td id="S3.T1.11.9.12.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">70.00</td>
<td id="S3.T1.11.9.12.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">90.52</td>
<td id="S3.T1.11.9.12.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">40.29</td>
<td id="S3.T1.11.9.12.15" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_t">5.81</td>
<td id="S3.T1.11.9.12.16" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_t">69.50</td>
<td id="S3.T1.11.9.12.17" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_t">59.04</td>
<td id="S3.T1.11.9.12.18" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_t">51.81</td>
<td id="S3.T1.11.9.12.19" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_t">40.10</td>
</tr>
<tr id="S3.T1.8.6.6" class="ltx_tr">
<td id="S3.T1.8.6.6.1" class="ltx_td ltx_nopad_r ltx_align_left">+<math id="S3.T1.8.6.6.1.m1.1" class="ltx_Math" alttext="l_{1}" display="inline"><semantics id="S3.T1.8.6.6.1.m1.1a"><msub id="S3.T1.8.6.6.1.m1.1.1" xref="S3.T1.8.6.6.1.m1.1.1.cmml"><mi id="S3.T1.8.6.6.1.m1.1.1.2" xref="S3.T1.8.6.6.1.m1.1.1.2.cmml">l</mi><mn id="S3.T1.8.6.6.1.m1.1.1.3" xref="S3.T1.8.6.6.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S3.T1.8.6.6.1.m1.1b"><apply id="S3.T1.8.6.6.1.m1.1.1.cmml" xref="S3.T1.8.6.6.1.m1.1.1"><csymbol cd="ambiguous" id="S3.T1.8.6.6.1.m1.1.1.1.cmml" xref="S3.T1.8.6.6.1.m1.1.1">subscript</csymbol><ci id="S3.T1.8.6.6.1.m1.1.1.2.cmml" xref="S3.T1.8.6.6.1.m1.1.1.2">𝑙</ci><cn type="integer" id="S3.T1.8.6.6.1.m1.1.1.3.cmml" xref="S3.T1.8.6.6.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.8.6.6.1.m1.1c">l_{1}</annotation></semantics></math>
</td>
<td id="S3.T1.8.6.6.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">30</td>
<td id="S3.T1.8.6.6.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">24.43</td>
<td id="S3.T1.8.6.6.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">24.63</td>
<td id="S3.T1.8.6.6.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">40.11</td>
<td id="S3.T1.8.6.6.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">41.83</td>
<td id="S3.T1.8.6.6.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">33.47</td>
<td id="S3.T1.8.6.6.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">21.22</td>
<td id="S3.T1.8.6.6.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">21.47</td>
<td id="S3.T1.8.6.6.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">22.41</td>
<td id="S3.T1.8.6.6.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">23.73</td>
<td id="S3.T1.8.6.6.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">66.50</td>
<td id="S3.T1.8.6.6.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">90.39</td>
<td id="S3.T1.8.6.6.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">40.20</td>
<td id="S3.T1.8.6.6.15" class="ltx_td ltx_nopad_l ltx_align_center">5.70</td>
<td id="S3.T1.8.6.6.16" class="ltx_td ltx_nopad_l ltx_align_center">68.10</td>
<td id="S3.T1.8.6.6.17" class="ltx_td ltx_nopad_l ltx_align_center">61.04</td>
<td id="S3.T1.8.6.6.18" class="ltx_td ltx_nopad_l ltx_align_center">55.37</td>
<td id="S3.T1.8.6.6.19" class="ltx_td ltx_nopad_l ltx_align_center">40.04</td>
</tr>
<tr id="S3.T1.9.7.7" class="ltx_tr">
<td id="S3.T1.9.7.7.1" class="ltx_td ltx_nopad_r ltx_align_left">+<math id="S3.T1.9.7.7.1.m1.1" class="ltx_Math" alttext="l_{1}" display="inline"><semantics id="S3.T1.9.7.7.1.m1.1a"><msub id="S3.T1.9.7.7.1.m1.1.1" xref="S3.T1.9.7.7.1.m1.1.1.cmml"><mi id="S3.T1.9.7.7.1.m1.1.1.2" xref="S3.T1.9.7.7.1.m1.1.1.2.cmml">l</mi><mn id="S3.T1.9.7.7.1.m1.1.1.3" xref="S3.T1.9.7.7.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S3.T1.9.7.7.1.m1.1b"><apply id="S3.T1.9.7.7.1.m1.1.1.cmml" xref="S3.T1.9.7.7.1.m1.1.1"><csymbol cd="ambiguous" id="S3.T1.9.7.7.1.m1.1.1.1.cmml" xref="S3.T1.9.7.7.1.m1.1.1">subscript</csymbol><ci id="S3.T1.9.7.7.1.m1.1.1.2.cmml" xref="S3.T1.9.7.7.1.m1.1.1.2">𝑙</ci><cn type="integer" id="S3.T1.9.7.7.1.m1.1.1.3.cmml" xref="S3.T1.9.7.7.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.9.7.7.1.m1.1c">l_{1}</annotation></semantics></math>
</td>
<td id="S3.T1.9.7.7.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">40</td>
<td id="S3.T1.9.7.7.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">24.58</td>
<td id="S3.T1.9.7.7.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">24.87</td>
<td id="S3.T1.9.7.7.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">39.30</td>
<td id="S3.T1.9.7.7.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">42.76</td>
<td id="S3.T1.9.7.7.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">31.95</td>
<td id="S3.T1.9.7.7.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">20.47</td>
<td id="S3.T1.9.7.7.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">20.95</td>
<td id="S3.T1.9.7.7.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">22.22</td>
<td id="S3.T1.9.7.7.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">23.42</td>
<td id="S3.T1.9.7.7.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">55.50</td>
<td id="S3.T1.9.7.7.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">90.22</td>
<td id="S3.T1.9.7.7.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">39.13</td>
<td id="S3.T1.9.7.7.15" class="ltx_td ltx_nopad_l ltx_align_center">5.82</td>
<td id="S3.T1.9.7.7.16" class="ltx_td ltx_nopad_l ltx_align_center">68.39</td>
<td id="S3.T1.9.7.7.17" class="ltx_td ltx_nopad_l ltx_align_center">60.71</td>
<td id="S3.T1.9.7.7.18" class="ltx_td ltx_nopad_l ltx_align_center">56.10</td>
<td id="S3.T1.9.7.7.19" class="ltx_td ltx_nopad_l ltx_align_center">39.15</td>
</tr>
<tr id="S3.T1.10.8.8" class="ltx_tr">
<td id="S3.T1.10.8.8.1" class="ltx_td ltx_nopad_r ltx_align_left">+<math id="S3.T1.10.8.8.1.m1.1" class="ltx_Math" alttext="l_{2}" display="inline"><semantics id="S3.T1.10.8.8.1.m1.1a"><msub id="S3.T1.10.8.8.1.m1.1.1" xref="S3.T1.10.8.8.1.m1.1.1.cmml"><mi id="S3.T1.10.8.8.1.m1.1.1.2" xref="S3.T1.10.8.8.1.m1.1.1.2.cmml">l</mi><mn id="S3.T1.10.8.8.1.m1.1.1.3" xref="S3.T1.10.8.8.1.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S3.T1.10.8.8.1.m1.1b"><apply id="S3.T1.10.8.8.1.m1.1.1.cmml" xref="S3.T1.10.8.8.1.m1.1.1"><csymbol cd="ambiguous" id="S3.T1.10.8.8.1.m1.1.1.1.cmml" xref="S3.T1.10.8.8.1.m1.1.1">subscript</csymbol><ci id="S3.T1.10.8.8.1.m1.1.1.2.cmml" xref="S3.T1.10.8.8.1.m1.1.1.2">𝑙</ci><cn type="integer" id="S3.T1.10.8.8.1.m1.1.1.3.cmml" xref="S3.T1.10.8.8.1.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.10.8.8.1.m1.1c">l_{2}</annotation></semantics></math>
</td>
<td id="S3.T1.10.8.8.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">30</td>
<td id="S3.T1.10.8.8.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">24.47</td>
<td id="S3.T1.10.8.8.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">24.73</td>
<td id="S3.T1.10.8.8.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">38.16</td>
<td id="S3.T1.10.8.8.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">41.86</td>
<td id="S3.T1.10.8.8.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">32.23</td>
<td id="S3.T1.10.8.8.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">20.23</td>
<td id="S3.T1.10.8.8.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">21.59</td>
<td id="S3.T1.10.8.8.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">22.45</td>
<td id="S3.T1.10.8.8.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">23.77</td>
<td id="S3.T1.10.8.8.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">67.5</td>
<td id="S3.T1.10.8.8.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">90.33</td>
<td id="S3.T1.10.8.8.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center">40.31</td>
<td id="S3.T1.10.8.8.15" class="ltx_td ltx_nopad_l ltx_align_center">5.7</td>
<td id="S3.T1.10.8.8.16" class="ltx_td ltx_nopad_l ltx_align_center">68.42</td>
<td id="S3.T1.10.8.8.17" class="ltx_td ltx_nopad_l ltx_align_center">62.65</td>
<td id="S3.T1.10.8.8.18" class="ltx_td ltx_nopad_l ltx_align_center">60.07</td>
<td id="S3.T1.10.8.8.19" class="ltx_td ltx_nopad_l ltx_align_center">40.28</td>
</tr>
<tr id="S3.T1.11.9.9" class="ltx_tr">
<td id="S3.T1.11.9.9.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">+<math id="S3.T1.11.9.9.1.m1.1" class="ltx_Math" alttext="l_{2}" display="inline"><semantics id="S3.T1.11.9.9.1.m1.1a"><msub id="S3.T1.11.9.9.1.m1.1.1" xref="S3.T1.11.9.9.1.m1.1.1.cmml"><mi id="S3.T1.11.9.9.1.m1.1.1.2" xref="S3.T1.11.9.9.1.m1.1.1.2.cmml">l</mi><mn id="S3.T1.11.9.9.1.m1.1.1.3" xref="S3.T1.11.9.9.1.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S3.T1.11.9.9.1.m1.1b"><apply id="S3.T1.11.9.9.1.m1.1.1.cmml" xref="S3.T1.11.9.9.1.m1.1.1"><csymbol cd="ambiguous" id="S3.T1.11.9.9.1.m1.1.1.1.cmml" xref="S3.T1.11.9.9.1.m1.1.1">subscript</csymbol><ci id="S3.T1.11.9.9.1.m1.1.1.2.cmml" xref="S3.T1.11.9.9.1.m1.1.1.2">𝑙</ci><cn type="integer" id="S3.T1.11.9.9.1.m1.1.1.3.cmml" xref="S3.T1.11.9.9.1.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.11.9.9.1.m1.1c">l_{2}</annotation></semantics></math>
</td>
<td id="S3.T1.11.9.9.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb">40</td>
<td id="S3.T1.11.9.9.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb">24.52</td>
<td id="S3.T1.11.9.9.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb">23.75</td>
<td id="S3.T1.11.9.9.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb">38.35</td>
<td id="S3.T1.11.9.9.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb">42.42</td>
<td id="S3.T1.11.9.9.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb">32.96</td>
<td id="S3.T1.11.9.9.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb">20.39</td>
<td id="S3.T1.11.9.9.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb">21.21</td>
<td id="S3.T1.11.9.9.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb">22.28</td>
<td id="S3.T1.11.9.9.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb">23.41</td>
<td id="S3.T1.11.9.9.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb">60.00</td>
<td id="S3.T1.11.9.9.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb">90.20</td>
<td id="S3.T1.11.9.9.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb">39.59</td>
<td id="S3.T1.11.9.9.15" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_bb">5.75</td>
<td id="S3.T1.11.9.9.16" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_bb">68.29</td>
<td id="S3.T1.11.9.9.17" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_bb">61.96</td>
<td id="S3.T1.11.9.9.18" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_bb">60.59</td>
<td id="S3.T1.11.9.9.19" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_bb">39.74</td>
</tr>
</table>
</span></div>
</figure>
<div id="S3.SS1.p2" class="ltx_para ltx_noindent">
<p id="S3.SS1.p2.11" class="ltx_p"><span id="S3.SS1.p2.11.1" class="ltx_text ltx_font_bold">Magnitude base Pruning:</span> Based on the observations in Figure <a href="#S2.F2" title="Figure 2 ‣ Magnitude of KV-Cache Channel ‣ 2 Observations ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> which depicts the significant variation in the magnitudes across different channels, one straightforward criterion is to use the norm of the magnitude to measure the importance of different channels in key cache.</p>
<table id="S3.E3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E3.m1.6" class="ltx_Math" alttext="M_{n,d}=\big{\|}K[n,:,d]\big{\|}_{p}." display="block"><semantics id="S3.E3.m1.6a"><mrow id="S3.E3.m1.6.6.1" xref="S3.E3.m1.6.6.1.1.cmml"><mrow id="S3.E3.m1.6.6.1.1" xref="S3.E3.m1.6.6.1.1.cmml"><msub id="S3.E3.m1.6.6.1.1.3" xref="S3.E3.m1.6.6.1.1.3.cmml"><mi id="S3.E3.m1.6.6.1.1.3.2" xref="S3.E3.m1.6.6.1.1.3.2.cmml">M</mi><mrow id="S3.E3.m1.2.2.2.4" xref="S3.E3.m1.2.2.2.3.cmml"><mi id="S3.E3.m1.1.1.1.1" xref="S3.E3.m1.1.1.1.1.cmml">n</mi><mo id="S3.E3.m1.2.2.2.4.1" xref="S3.E3.m1.2.2.2.3.cmml">,</mo><mi id="S3.E3.m1.2.2.2.2" xref="S3.E3.m1.2.2.2.2.cmml">d</mi></mrow></msub><mo id="S3.E3.m1.6.6.1.1.2" xref="S3.E3.m1.6.6.1.1.2.cmml">=</mo><msub id="S3.E3.m1.6.6.1.1.1" xref="S3.E3.m1.6.6.1.1.1.cmml"><mrow id="S3.E3.m1.6.6.1.1.1.1.1" xref="S3.E3.m1.6.6.1.1.1.1.2.cmml"><mo maxsize="120%" minsize="120%" id="S3.E3.m1.6.6.1.1.1.1.1.2" xref="S3.E3.m1.6.6.1.1.1.1.2.1.cmml">‖</mo><mrow id="S3.E3.m1.6.6.1.1.1.1.1.1" xref="S3.E3.m1.6.6.1.1.1.1.1.1.cmml"><mi id="S3.E3.m1.6.6.1.1.1.1.1.1.2" xref="S3.E3.m1.6.6.1.1.1.1.1.1.2.cmml">K</mi><mo lspace="0em" rspace="0em" id="S3.E3.m1.6.6.1.1.1.1.1.1.1" xref="S3.E3.m1.6.6.1.1.1.1.1.1.1.cmml">​</mo><mrow id="S3.E3.m1.6.6.1.1.1.1.1.1.3.2" xref="S3.E3.m1.6.6.1.1.1.1.1.1.3.1.cmml"><mo stretchy="false" id="S3.E3.m1.6.6.1.1.1.1.1.1.3.2.1" xref="S3.E3.m1.6.6.1.1.1.1.1.1.3.1.cmml">[</mo><mi id="S3.E3.m1.3.3" xref="S3.E3.m1.3.3.cmml">n</mi><mo id="S3.E3.m1.6.6.1.1.1.1.1.1.3.2.2" xref="S3.E3.m1.6.6.1.1.1.1.1.1.3.1.cmml">,</mo><mo rspace="0em" id="S3.E3.m1.4.4" xref="S3.E3.m1.4.4.cmml">:</mo><mo id="S3.E3.m1.6.6.1.1.1.1.1.1.3.2.3" xref="S3.E3.m1.6.6.1.1.1.1.1.1.3.1.cmml">,</mo><mi id="S3.E3.m1.5.5" xref="S3.E3.m1.5.5.cmml">d</mi><mo stretchy="false" id="S3.E3.m1.6.6.1.1.1.1.1.1.3.2.4" xref="S3.E3.m1.6.6.1.1.1.1.1.1.3.1.cmml">]</mo></mrow></mrow><mo maxsize="120%" minsize="120%" id="S3.E3.m1.6.6.1.1.1.1.1.3" xref="S3.E3.m1.6.6.1.1.1.1.2.1.cmml">‖</mo></mrow><mi id="S3.E3.m1.6.6.1.1.1.3" xref="S3.E3.m1.6.6.1.1.1.3.cmml">p</mi></msub></mrow><mo lspace="0em" id="S3.E3.m1.6.6.1.2" xref="S3.E3.m1.6.6.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E3.m1.6b"><apply id="S3.E3.m1.6.6.1.1.cmml" xref="S3.E3.m1.6.6.1"><eq id="S3.E3.m1.6.6.1.1.2.cmml" xref="S3.E3.m1.6.6.1.1.2"></eq><apply id="S3.E3.m1.6.6.1.1.3.cmml" xref="S3.E3.m1.6.6.1.1.3"><csymbol cd="ambiguous" id="S3.E3.m1.6.6.1.1.3.1.cmml" xref="S3.E3.m1.6.6.1.1.3">subscript</csymbol><ci id="S3.E3.m1.6.6.1.1.3.2.cmml" xref="S3.E3.m1.6.6.1.1.3.2">𝑀</ci><list id="S3.E3.m1.2.2.2.3.cmml" xref="S3.E3.m1.2.2.2.4"><ci id="S3.E3.m1.1.1.1.1.cmml" xref="S3.E3.m1.1.1.1.1">𝑛</ci><ci id="S3.E3.m1.2.2.2.2.cmml" xref="S3.E3.m1.2.2.2.2">𝑑</ci></list></apply><apply id="S3.E3.m1.6.6.1.1.1.cmml" xref="S3.E3.m1.6.6.1.1.1"><csymbol cd="ambiguous" id="S3.E3.m1.6.6.1.1.1.2.cmml" xref="S3.E3.m1.6.6.1.1.1">subscript</csymbol><apply id="S3.E3.m1.6.6.1.1.1.1.2.cmml" xref="S3.E3.m1.6.6.1.1.1.1.1"><csymbol cd="latexml" id="S3.E3.m1.6.6.1.1.1.1.2.1.cmml" xref="S3.E3.m1.6.6.1.1.1.1.1.2">norm</csymbol><apply id="S3.E3.m1.6.6.1.1.1.1.1.1.cmml" xref="S3.E3.m1.6.6.1.1.1.1.1.1"><times id="S3.E3.m1.6.6.1.1.1.1.1.1.1.cmml" xref="S3.E3.m1.6.6.1.1.1.1.1.1.1"></times><ci id="S3.E3.m1.6.6.1.1.1.1.1.1.2.cmml" xref="S3.E3.m1.6.6.1.1.1.1.1.1.2">𝐾</ci><list id="S3.E3.m1.6.6.1.1.1.1.1.1.3.1.cmml" xref="S3.E3.m1.6.6.1.1.1.1.1.1.3.2"><ci id="S3.E3.m1.3.3.cmml" xref="S3.E3.m1.3.3">𝑛</ci><ci id="S3.E3.m1.4.4.cmml" xref="S3.E3.m1.4.4">:</ci><ci id="S3.E3.m1.5.5.cmml" xref="S3.E3.m1.5.5">𝑑</ci></list></apply></apply><ci id="S3.E3.m1.6.6.1.1.1.3.cmml" xref="S3.E3.m1.6.6.1.1.1.3">𝑝</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E3.m1.6c">M_{n,d}=\big{\|}K[n,:,d]\big{\|}_{p}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
</table>
<p id="S3.SS1.p2.3" class="ltx_p">Given pruning ratio <math id="S3.SS1.p2.1.m1.1" class="ltx_Math" alttext="\lambda" display="inline"><semantics id="S3.SS1.p2.1.m1.1a"><mi id="S3.SS1.p2.1.m1.1.1" xref="S3.SS1.p2.1.m1.1.1.cmml">λ</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.1.m1.1b"><ci id="S3.SS1.p2.1.m1.1.1.cmml" xref="S3.SS1.p2.1.m1.1.1">𝜆</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.1.m1.1c">\lambda</annotation></semantics></math>, We only keep <math id="S3.SS1.p2.2.m2.1" class="ltx_Math" alttext="T=\lfloor(1-\lambda)D\rfloor" display="inline"><semantics id="S3.SS1.p2.2.m2.1a"><mrow id="S3.SS1.p2.2.m2.1.1" xref="S3.SS1.p2.2.m2.1.1.cmml"><mi id="S3.SS1.p2.2.m2.1.1.3" xref="S3.SS1.p2.2.m2.1.1.3.cmml">T</mi><mo id="S3.SS1.p2.2.m2.1.1.2" xref="S3.SS1.p2.2.m2.1.1.2.cmml">=</mo><mrow id="S3.SS1.p2.2.m2.1.1.1.1" xref="S3.SS1.p2.2.m2.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS1.p2.2.m2.1.1.1.1.2" xref="S3.SS1.p2.2.m2.1.1.1.2.1.cmml">⌊</mo><mrow id="S3.SS1.p2.2.m2.1.1.1.1.1" xref="S3.SS1.p2.2.m2.1.1.1.1.1.cmml"><mrow id="S3.SS1.p2.2.m2.1.1.1.1.1.1.1" xref="S3.SS1.p2.2.m2.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p2.2.m2.1.1.1.1.1.1.1.2" xref="S3.SS1.p2.2.m2.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS1.p2.2.m2.1.1.1.1.1.1.1.1" xref="S3.SS1.p2.2.m2.1.1.1.1.1.1.1.1.cmml"><mn id="S3.SS1.p2.2.m2.1.1.1.1.1.1.1.1.2" xref="S3.SS1.p2.2.m2.1.1.1.1.1.1.1.1.2.cmml">1</mn><mo id="S3.SS1.p2.2.m2.1.1.1.1.1.1.1.1.1" xref="S3.SS1.p2.2.m2.1.1.1.1.1.1.1.1.1.cmml">−</mo><mi id="S3.SS1.p2.2.m2.1.1.1.1.1.1.1.1.3" xref="S3.SS1.p2.2.m2.1.1.1.1.1.1.1.1.3.cmml">λ</mi></mrow><mo stretchy="false" id="S3.SS1.p2.2.m2.1.1.1.1.1.1.1.3" xref="S3.SS1.p2.2.m2.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S3.SS1.p2.2.m2.1.1.1.1.1.2" xref="S3.SS1.p2.2.m2.1.1.1.1.1.2.cmml">​</mo><mi id="S3.SS1.p2.2.m2.1.1.1.1.1.3" xref="S3.SS1.p2.2.m2.1.1.1.1.1.3.cmml">D</mi></mrow><mo stretchy="false" id="S3.SS1.p2.2.m2.1.1.1.1.3" xref="S3.SS1.p2.2.m2.1.1.1.2.1.cmml">⌋</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.2.m2.1b"><apply id="S3.SS1.p2.2.m2.1.1.cmml" xref="S3.SS1.p2.2.m2.1.1"><eq id="S3.SS1.p2.2.m2.1.1.2.cmml" xref="S3.SS1.p2.2.m2.1.1.2"></eq><ci id="S3.SS1.p2.2.m2.1.1.3.cmml" xref="S3.SS1.p2.2.m2.1.1.3">𝑇</ci><apply id="S3.SS1.p2.2.m2.1.1.1.2.cmml" xref="S3.SS1.p2.2.m2.1.1.1.1"><floor id="S3.SS1.p2.2.m2.1.1.1.2.1.cmml" xref="S3.SS1.p2.2.m2.1.1.1.1.2"></floor><apply id="S3.SS1.p2.2.m2.1.1.1.1.1.cmml" xref="S3.SS1.p2.2.m2.1.1.1.1.1"><times id="S3.SS1.p2.2.m2.1.1.1.1.1.2.cmml" xref="S3.SS1.p2.2.m2.1.1.1.1.1.2"></times><apply id="S3.SS1.p2.2.m2.1.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p2.2.m2.1.1.1.1.1.1.1"><minus id="S3.SS1.p2.2.m2.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p2.2.m2.1.1.1.1.1.1.1.1.1"></minus><cn type="integer" id="S3.SS1.p2.2.m2.1.1.1.1.1.1.1.1.2.cmml" xref="S3.SS1.p2.2.m2.1.1.1.1.1.1.1.1.2">1</cn><ci id="S3.SS1.p2.2.m2.1.1.1.1.1.1.1.1.3.cmml" xref="S3.SS1.p2.2.m2.1.1.1.1.1.1.1.1.3">𝜆</ci></apply><ci id="S3.SS1.p2.2.m2.1.1.1.1.1.3.cmml" xref="S3.SS1.p2.2.m2.1.1.1.1.1.3">𝐷</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.2.m2.1c">T=\lfloor(1-\lambda)D\rfloor</annotation></semantics></math> most important channels among the <math id="S3.SS1.p2.3.m3.1" class="ltx_Math" alttext="D" display="inline"><semantics id="S3.SS1.p2.3.m3.1a"><mi id="S3.SS1.p2.3.m3.1.1" xref="S3.SS1.p2.3.m3.1.1.cmml">D</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.3.m3.1b"><ci id="S3.SS1.p2.3.m3.1.1.cmml" xref="S3.SS1.p2.3.m3.1.1">𝐷</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.3.m3.1c">D</annotation></semantics></math> channels of each head:</p>
<table id="S3.E4" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E4.m1.2" class="ltx_Math" alttext="I=\textbf{Top}_{T}(M,T)" display="block"><semantics id="S3.E4.m1.2a"><mrow id="S3.E4.m1.2.3" xref="S3.E4.m1.2.3.cmml"><mi id="S3.E4.m1.2.3.2" xref="S3.E4.m1.2.3.2.cmml">I</mi><mo id="S3.E4.m1.2.3.1" xref="S3.E4.m1.2.3.1.cmml">=</mo><mrow id="S3.E4.m1.2.3.3" xref="S3.E4.m1.2.3.3.cmml"><msub id="S3.E4.m1.2.3.3.2" xref="S3.E4.m1.2.3.3.2.cmml"><mtext class="ltx_mathvariant_bold" id="S3.E4.m1.2.3.3.2.2" xref="S3.E4.m1.2.3.3.2.2a.cmml">Top</mtext><mi id="S3.E4.m1.2.3.3.2.3" xref="S3.E4.m1.2.3.3.2.3.cmml">T</mi></msub><mo lspace="0em" rspace="0em" id="S3.E4.m1.2.3.3.1" xref="S3.E4.m1.2.3.3.1.cmml">​</mo><mrow id="S3.E4.m1.2.3.3.3.2" xref="S3.E4.m1.2.3.3.3.1.cmml"><mo stretchy="false" id="S3.E4.m1.2.3.3.3.2.1" xref="S3.E4.m1.2.3.3.3.1.cmml">(</mo><mi id="S3.E4.m1.1.1" xref="S3.E4.m1.1.1.cmml">M</mi><mo id="S3.E4.m1.2.3.3.3.2.2" xref="S3.E4.m1.2.3.3.3.1.cmml">,</mo><mi id="S3.E4.m1.2.2" xref="S3.E4.m1.2.2.cmml">T</mi><mo stretchy="false" id="S3.E4.m1.2.3.3.3.2.3" xref="S3.E4.m1.2.3.3.3.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E4.m1.2b"><apply id="S3.E4.m1.2.3.cmml" xref="S3.E4.m1.2.3"><eq id="S3.E4.m1.2.3.1.cmml" xref="S3.E4.m1.2.3.1"></eq><ci id="S3.E4.m1.2.3.2.cmml" xref="S3.E4.m1.2.3.2">𝐼</ci><apply id="S3.E4.m1.2.3.3.cmml" xref="S3.E4.m1.2.3.3"><times id="S3.E4.m1.2.3.3.1.cmml" xref="S3.E4.m1.2.3.3.1"></times><apply id="S3.E4.m1.2.3.3.2.cmml" xref="S3.E4.m1.2.3.3.2"><csymbol cd="ambiguous" id="S3.E4.m1.2.3.3.2.1.cmml" xref="S3.E4.m1.2.3.3.2">subscript</csymbol><ci id="S3.E4.m1.2.3.3.2.2a.cmml" xref="S3.E4.m1.2.3.3.2.2"><mtext class="ltx_mathvariant_bold" id="S3.E4.m1.2.3.3.2.2.cmml" xref="S3.E4.m1.2.3.3.2.2">Top</mtext></ci><ci id="S3.E4.m1.2.3.3.2.3.cmml" xref="S3.E4.m1.2.3.3.2.3">𝑇</ci></apply><interval closure="open" id="S3.E4.m1.2.3.3.3.1.cmml" xref="S3.E4.m1.2.3.3.3.2"><ci id="S3.E4.m1.1.1.cmml" xref="S3.E4.m1.1.1">𝑀</ci><ci id="S3.E4.m1.2.2.cmml" xref="S3.E4.m1.2.2">𝑇</ci></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E4.m1.2c">I=\textbf{Top}_{T}(M,T)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(4)</span></td>
</tr></tbody>
</table>
<p id="S3.SS1.p2.10" class="ltx_p">where <math id="S3.SS1.p2.4.m1.1" class="ltx_math_unparsed" alttext="\|\cdot\|_{p}" display="inline"><semantics id="S3.SS1.p2.4.m1.1a"><mrow id="S3.SS1.p2.4.m1.1b"><mo rspace="0em" id="S3.SS1.p2.4.m1.1.1">∥</mo><mo lspace="0em" rspace="0em" id="S3.SS1.p2.4.m1.1.2">⋅</mo><msub id="S3.SS1.p2.4.m1.1.3"><mo lspace="0em" id="S3.SS1.p2.4.m1.1.3.2">∥</mo><mi id="S3.SS1.p2.4.m1.1.3.3">p</mi></msub></mrow><annotation encoding="application/x-tex" id="S3.SS1.p2.4.m1.1c">\|\cdot\|_{p}</annotation></semantics></math> is the <math id="S3.SS1.p2.5.m2.1" class="ltx_Math" alttext="l_{p}" display="inline"><semantics id="S3.SS1.p2.5.m2.1a"><msub id="S3.SS1.p2.5.m2.1.1" xref="S3.SS1.p2.5.m2.1.1.cmml"><mi id="S3.SS1.p2.5.m2.1.1.2" xref="S3.SS1.p2.5.m2.1.1.2.cmml">l</mi><mi id="S3.SS1.p2.5.m2.1.1.3" xref="S3.SS1.p2.5.m2.1.1.3.cmml">p</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.5.m2.1b"><apply id="S3.SS1.p2.5.m2.1.1.cmml" xref="S3.SS1.p2.5.m2.1.1"><csymbol cd="ambiguous" id="S3.SS1.p2.5.m2.1.1.1.cmml" xref="S3.SS1.p2.5.m2.1.1">subscript</csymbol><ci id="S3.SS1.p2.5.m2.1.1.2.cmml" xref="S3.SS1.p2.5.m2.1.1.2">𝑙</ci><ci id="S3.SS1.p2.5.m2.1.1.3.cmml" xref="S3.SS1.p2.5.m2.1.1.3">𝑝</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.5.m2.1c">l_{p}</annotation></semantics></math> norm of each channel. <math id="S3.SS1.p2.6.m3.2" class="ltx_Math" alttext="n\in[1,N]" display="inline"><semantics id="S3.SS1.p2.6.m3.2a"><mrow id="S3.SS1.p2.6.m3.2.3" xref="S3.SS1.p2.6.m3.2.3.cmml"><mi id="S3.SS1.p2.6.m3.2.3.2" xref="S3.SS1.p2.6.m3.2.3.2.cmml">n</mi><mo id="S3.SS1.p2.6.m3.2.3.1" xref="S3.SS1.p2.6.m3.2.3.1.cmml">∈</mo><mrow id="S3.SS1.p2.6.m3.2.3.3.2" xref="S3.SS1.p2.6.m3.2.3.3.1.cmml"><mo stretchy="false" id="S3.SS1.p2.6.m3.2.3.3.2.1" xref="S3.SS1.p2.6.m3.2.3.3.1.cmml">[</mo><mn id="S3.SS1.p2.6.m3.1.1" xref="S3.SS1.p2.6.m3.1.1.cmml">1</mn><mo id="S3.SS1.p2.6.m3.2.3.3.2.2" xref="S3.SS1.p2.6.m3.2.3.3.1.cmml">,</mo><mi id="S3.SS1.p2.6.m3.2.2" xref="S3.SS1.p2.6.m3.2.2.cmml">N</mi><mo stretchy="false" id="S3.SS1.p2.6.m3.2.3.3.2.3" xref="S3.SS1.p2.6.m3.2.3.3.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.6.m3.2b"><apply id="S3.SS1.p2.6.m3.2.3.cmml" xref="S3.SS1.p2.6.m3.2.3"><in id="S3.SS1.p2.6.m3.2.3.1.cmml" xref="S3.SS1.p2.6.m3.2.3.1"></in><ci id="S3.SS1.p2.6.m3.2.3.2.cmml" xref="S3.SS1.p2.6.m3.2.3.2">𝑛</ci><interval closure="closed" id="S3.SS1.p2.6.m3.2.3.3.1.cmml" xref="S3.SS1.p2.6.m3.2.3.3.2"><cn type="integer" id="S3.SS1.p2.6.m3.1.1.cmml" xref="S3.SS1.p2.6.m3.1.1">1</cn><ci id="S3.SS1.p2.6.m3.2.2.cmml" xref="S3.SS1.p2.6.m3.2.2">𝑁</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.6.m3.2c">n\in[1,N]</annotation></semantics></math> and <math id="S3.SS1.p2.7.m4.2" class="ltx_Math" alttext="d\in[1,D]" display="inline"><semantics id="S3.SS1.p2.7.m4.2a"><mrow id="S3.SS1.p2.7.m4.2.3" xref="S3.SS1.p2.7.m4.2.3.cmml"><mi id="S3.SS1.p2.7.m4.2.3.2" xref="S3.SS1.p2.7.m4.2.3.2.cmml">d</mi><mo id="S3.SS1.p2.7.m4.2.3.1" xref="S3.SS1.p2.7.m4.2.3.1.cmml">∈</mo><mrow id="S3.SS1.p2.7.m4.2.3.3.2" xref="S3.SS1.p2.7.m4.2.3.3.1.cmml"><mo stretchy="false" id="S3.SS1.p2.7.m4.2.3.3.2.1" xref="S3.SS1.p2.7.m4.2.3.3.1.cmml">[</mo><mn id="S3.SS1.p2.7.m4.1.1" xref="S3.SS1.p2.7.m4.1.1.cmml">1</mn><mo id="S3.SS1.p2.7.m4.2.3.3.2.2" xref="S3.SS1.p2.7.m4.2.3.3.1.cmml">,</mo><mi id="S3.SS1.p2.7.m4.2.2" xref="S3.SS1.p2.7.m4.2.2.cmml">D</mi><mo stretchy="false" id="S3.SS1.p2.7.m4.2.3.3.2.3" xref="S3.SS1.p2.7.m4.2.3.3.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.7.m4.2b"><apply id="S3.SS1.p2.7.m4.2.3.cmml" xref="S3.SS1.p2.7.m4.2.3"><in id="S3.SS1.p2.7.m4.2.3.1.cmml" xref="S3.SS1.p2.7.m4.2.3.1"></in><ci id="S3.SS1.p2.7.m4.2.3.2.cmml" xref="S3.SS1.p2.7.m4.2.3.2">𝑑</ci><interval closure="closed" id="S3.SS1.p2.7.m4.2.3.3.1.cmml" xref="S3.SS1.p2.7.m4.2.3.3.2"><cn type="integer" id="S3.SS1.p2.7.m4.1.1.cmml" xref="S3.SS1.p2.7.m4.1.1">1</cn><ci id="S3.SS1.p2.7.m4.2.2.cmml" xref="S3.SS1.p2.7.m4.2.2">𝐷</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.7.m4.2c">d\in[1,D]</annotation></semantics></math> are indicators of heads and channels in key cache. <math id="S3.SS1.p2.8.m5.1" class="ltx_Math" alttext="I\in\mathbb{(Z^{+})}^{N\times T}" display="inline"><semantics id="S3.SS1.p2.8.m5.1a"><mrow id="S3.SS1.p2.8.m5.1.1" xref="S3.SS1.p2.8.m5.1.1.cmml"><mi id="S3.SS1.p2.8.m5.1.1.3" xref="S3.SS1.p2.8.m5.1.1.3.cmml">I</mi><mo id="S3.SS1.p2.8.m5.1.1.2" xref="S3.SS1.p2.8.m5.1.1.2.cmml">∈</mo><msup id="S3.SS1.p2.8.m5.1.1.1" xref="S3.SS1.p2.8.m5.1.1.1.cmml"><mrow id="S3.SS1.p2.8.m5.1.1.1.1.1" xref="S3.SS1.p2.8.m5.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p2.8.m5.1.1.1.1.1.2" xref="S3.SS1.p2.8.m5.1.1.1.1.1.1.cmml">(</mo><msup id="S3.SS1.p2.8.m5.1.1.1.1.1.1" xref="S3.SS1.p2.8.m5.1.1.1.1.1.1.cmml"><mi id="S3.SS1.p2.8.m5.1.1.1.1.1.1.2" xref="S3.SS1.p2.8.m5.1.1.1.1.1.1.2.cmml">ℤ</mi><mo id="S3.SS1.p2.8.m5.1.1.1.1.1.1.3" xref="S3.SS1.p2.8.m5.1.1.1.1.1.1.3.cmml">+</mo></msup><mo stretchy="false" id="S3.SS1.p2.8.m5.1.1.1.1.1.3" xref="S3.SS1.p2.8.m5.1.1.1.1.1.1.cmml">)</mo></mrow><mrow id="S3.SS1.p2.8.m5.1.1.1.3" xref="S3.SS1.p2.8.m5.1.1.1.3.cmml"><mi id="S3.SS1.p2.8.m5.1.1.1.3.2" xref="S3.SS1.p2.8.m5.1.1.1.3.2.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS1.p2.8.m5.1.1.1.3.1" xref="S3.SS1.p2.8.m5.1.1.1.3.1.cmml">×</mo><mi id="S3.SS1.p2.8.m5.1.1.1.3.3" xref="S3.SS1.p2.8.m5.1.1.1.3.3.cmml">T</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.8.m5.1b"><apply id="S3.SS1.p2.8.m5.1.1.cmml" xref="S3.SS1.p2.8.m5.1.1"><in id="S3.SS1.p2.8.m5.1.1.2.cmml" xref="S3.SS1.p2.8.m5.1.1.2"></in><ci id="S3.SS1.p2.8.m5.1.1.3.cmml" xref="S3.SS1.p2.8.m5.1.1.3">𝐼</ci><apply id="S3.SS1.p2.8.m5.1.1.1.cmml" xref="S3.SS1.p2.8.m5.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p2.8.m5.1.1.1.2.cmml" xref="S3.SS1.p2.8.m5.1.1.1">superscript</csymbol><apply id="S3.SS1.p2.8.m5.1.1.1.1.1.1.cmml" xref="S3.SS1.p2.8.m5.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p2.8.m5.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p2.8.m5.1.1.1.1.1">superscript</csymbol><ci id="S3.SS1.p2.8.m5.1.1.1.1.1.1.2.cmml" xref="S3.SS1.p2.8.m5.1.1.1.1.1.1.2">ℤ</ci><plus id="S3.SS1.p2.8.m5.1.1.1.1.1.1.3.cmml" xref="S3.SS1.p2.8.m5.1.1.1.1.1.1.3"></plus></apply><apply id="S3.SS1.p2.8.m5.1.1.1.3.cmml" xref="S3.SS1.p2.8.m5.1.1.1.3"><times id="S3.SS1.p2.8.m5.1.1.1.3.1.cmml" xref="S3.SS1.p2.8.m5.1.1.1.3.1"></times><ci id="S3.SS1.p2.8.m5.1.1.1.3.2.cmml" xref="S3.SS1.p2.8.m5.1.1.1.3.2">𝑁</ci><ci id="S3.SS1.p2.8.m5.1.1.1.3.3.cmml" xref="S3.SS1.p2.8.m5.1.1.1.3.3">𝑇</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.8.m5.1c">I\in\mathbb{(Z^{+})}^{N\times T}</annotation></semantics></math> stores the indicators of the top <math id="S3.SS1.p2.9.m6.1" class="ltx_Math" alttext="T" display="inline"><semantics id="S3.SS1.p2.9.m6.1a"><mi id="S3.SS1.p2.9.m6.1.1" xref="S3.SS1.p2.9.m6.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.9.m6.1b"><ci id="S3.SS1.p2.9.m6.1.1.cmml" xref="S3.SS1.p2.9.m6.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.9.m6.1c">T</annotation></semantics></math> values in tensor <math id="S3.SS1.p2.10.m7.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S3.SS1.p2.10.m7.1a"><mi id="S3.SS1.p2.10.m7.1.1" xref="S3.SS1.p2.10.m7.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.10.m7.1b"><ci id="S3.SS1.p2.10.m7.1.1.cmml" xref="S3.SS1.p2.10.m7.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.10.m7.1c">M</annotation></semantics></math> per head.</p>
</div>
<div id="S3.SS1.p3" class="ltx_para ltx_noindent">
<p id="S3.SS1.p3.5" class="ltx_p">In Table <a href="#S3.T1" title="Table 1 ‣ 3.1 Preliminary Study of KV Cache Optimization ‣ 3 ThinK ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, we present the results of key cache pruning with various pruning ratios applied to the LLaMA3-8B model. We utilize the <math id="S3.SS1.p3.1.m1.1" class="ltx_Math" alttext="l_{1}" display="inline"><semantics id="S3.SS1.p3.1.m1.1a"><msub id="S3.SS1.p3.1.m1.1.1" xref="S3.SS1.p3.1.m1.1.1.cmml"><mi id="S3.SS1.p3.1.m1.1.1.2" xref="S3.SS1.p3.1.m1.1.1.2.cmml">l</mi><mn id="S3.SS1.p3.1.m1.1.1.3" xref="S3.SS1.p3.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.1.m1.1b"><apply id="S3.SS1.p3.1.m1.1.1.cmml" xref="S3.SS1.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.1.m1.1.1.1.cmml" xref="S3.SS1.p3.1.m1.1.1">subscript</csymbol><ci id="S3.SS1.p3.1.m1.1.1.2.cmml" xref="S3.SS1.p3.1.m1.1.1.2">𝑙</ci><cn type="integer" id="S3.SS1.p3.1.m1.1.1.3.cmml" xref="S3.SS1.p3.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.1.m1.1c">l_{1}</annotation></semantics></math> and <math id="S3.SS1.p3.2.m2.1" class="ltx_Math" alttext="l_{2}" display="inline"><semantics id="S3.SS1.p3.2.m2.1a"><msub id="S3.SS1.p3.2.m2.1.1" xref="S3.SS1.p3.2.m2.1.1.cmml"><mi id="S3.SS1.p3.2.m2.1.1.2" xref="S3.SS1.p3.2.m2.1.1.2.cmml">l</mi><mn id="S3.SS1.p3.2.m2.1.1.3" xref="S3.SS1.p3.2.m2.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.2.m2.1b"><apply id="S3.SS1.p3.2.m2.1.1.cmml" xref="S3.SS1.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.2.m2.1.1.1.cmml" xref="S3.SS1.p3.2.m2.1.1">subscript</csymbol><ci id="S3.SS1.p3.2.m2.1.1.2.cmml" xref="S3.SS1.p3.2.m2.1.1.2">𝑙</ci><cn type="integer" id="S3.SS1.p3.2.m2.1.1.3.cmml" xref="S3.SS1.p3.2.m2.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.2.m2.1c">l_{2}</annotation></semantics></math> norms as criteria for evaluation, and validate performance using the LongBench benchmark <cite class="ltx_cite ltx_citemacro_citep">(Bai et al.,, <a href="#bib.bib2" title="" class="ltx_ref">2023</a>)</cite>. Compared to the baseline methods, H2O <cite class="ltx_cite ltx_citemacro_citep">(<a href="#bib.bib56" title="" class="ltx_ref">Zhang et al., 2024c, </a>)</cite> and SnapKV <cite class="ltx_cite ltx_citemacro_citep">(Li et al.,, <a href="#bib.bib29" title="" class="ltx_ref">2024</a>)</cite>, both with a KV length of 512, we further prune the channels of the key cache. A <math id="S3.SS1.p3.3.m3.1" class="ltx_Math" alttext="30\%" display="inline"><semantics id="S3.SS1.p3.3.m3.1a"><mrow id="S3.SS1.p3.3.m3.1.1" xref="S3.SS1.p3.3.m3.1.1.cmml"><mn id="S3.SS1.p3.3.m3.1.1.2" xref="S3.SS1.p3.3.m3.1.1.2.cmml">30</mn><mo id="S3.SS1.p3.3.m3.1.1.1" xref="S3.SS1.p3.3.m3.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.3.m3.1b"><apply id="S3.SS1.p3.3.m3.1.1.cmml" xref="S3.SS1.p3.3.m3.1.1"><csymbol cd="latexml" id="S3.SS1.p3.3.m3.1.1.1.cmml" xref="S3.SS1.p3.3.m3.1.1.1">percent</csymbol><cn type="integer" id="S3.SS1.p3.3.m3.1.1.2.cmml" xref="S3.SS1.p3.3.m3.1.1.2">30</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.3.m3.1c">30\%</annotation></semantics></math> pruning ratio can maintain accuracy; however, increasing it to <math id="S3.SS1.p3.4.m4.1" class="ltx_Math" alttext="40\%" display="inline"><semantics id="S3.SS1.p3.4.m4.1a"><mrow id="S3.SS1.p3.4.m4.1.1" xref="S3.SS1.p3.4.m4.1.1.cmml"><mn id="S3.SS1.p3.4.m4.1.1.2" xref="S3.SS1.p3.4.m4.1.1.2.cmml">40</mn><mo id="S3.SS1.p3.4.m4.1.1.1" xref="S3.SS1.p3.4.m4.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.4.m4.1b"><apply id="S3.SS1.p3.4.m4.1.1.cmml" xref="S3.SS1.p3.4.m4.1.1"><csymbol cd="latexml" id="S3.SS1.p3.4.m4.1.1.1.cmml" xref="S3.SS1.p3.4.m4.1.1.1">percent</csymbol><cn type="integer" id="S3.SS1.p3.4.m4.1.1.2.cmml" xref="S3.SS1.p3.4.m4.1.1.2">40</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.4.m4.1c">40\%</annotation></semantics></math> results in significant performance degradation, especially for <math id="S3.SS1.p3.5.m5.1" class="ltx_Math" alttext="l_{1}" display="inline"><semantics id="S3.SS1.p3.5.m5.1a"><msub id="S3.SS1.p3.5.m5.1.1" xref="S3.SS1.p3.5.m5.1.1.cmml"><mi id="S3.SS1.p3.5.m5.1.1.2" xref="S3.SS1.p3.5.m5.1.1.2.cmml">l</mi><mn id="S3.SS1.p3.5.m5.1.1.3" xref="S3.SS1.p3.5.m5.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.5.m5.1b"><apply id="S3.SS1.p3.5.m5.1.1.cmml" xref="S3.SS1.p3.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.5.m5.1.1.1.cmml" xref="S3.SS1.p3.5.m5.1.1">subscript</csymbol><ci id="S3.SS1.p3.5.m5.1.1.2.cmml" xref="S3.SS1.p3.5.m5.1.1.2">𝑙</ci><cn type="integer" id="S3.SS1.p3.5.m5.1.1.3.cmml" xref="S3.SS1.p3.5.m5.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.5.m5.1c">l_{1}</annotation></semantics></math> norm based pruning. The results of magnitude-based pruning support our assumption that the key cache is redundant in the channel dimension. These results also indicate the need for a better pruning matrix to achieve higher pruning ratios effectively.</p>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Query-Driven Pruning</h3>

<div id="S3.SS2.p1" class="ltx_para ltx_noindent">
<p id="S3.SS2.p1.1" class="ltx_p">For each head, the attention scores are computed using the queries and keys, and then applied to the values. The formula for the attention for head <math id="S3.SS2.p1.1.m1.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S3.SS2.p1.1.m1.1a"><mi id="S3.SS2.p1.1.m1.1.1" xref="S3.SS2.p1.1.m1.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.1.m1.1b"><ci id="S3.SS2.p1.1.m1.1.1.cmml" xref="S3.SS2.p1.1.m1.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.1.m1.1c">i</annotation></semantics></math> is:</p>
<table id="S3.E5" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E5.m1.4" class="ltx_Math" alttext="\text{Attention}(\textbf{Q}_{i},\textbf{K}_{i},\textbf{V}_{i})=\text{softmax}(\frac{\textbf{Q}_{i}\textbf{K}_{i}^{T}}{\sqrt{D}})\textbf{V}_{i}" display="block"><semantics id="S3.E5.m1.4a"><mrow id="S3.E5.m1.4.4" xref="S3.E5.m1.4.4.cmml"><mrow id="S3.E5.m1.4.4.3" xref="S3.E5.m1.4.4.3.cmml"><mtext id="S3.E5.m1.4.4.3.5" xref="S3.E5.m1.4.4.3.5a.cmml">Attention</mtext><mo lspace="0em" rspace="0em" id="S3.E5.m1.4.4.3.4" xref="S3.E5.m1.4.4.3.4.cmml">​</mo><mrow id="S3.E5.m1.4.4.3.3.3" xref="S3.E5.m1.4.4.3.3.4.cmml"><mo stretchy="false" id="S3.E5.m1.4.4.3.3.3.4" xref="S3.E5.m1.4.4.3.3.4.cmml">(</mo><msub id="S3.E5.m1.2.2.1.1.1.1" xref="S3.E5.m1.2.2.1.1.1.1.cmml"><mtext class="ltx_mathvariant_bold" id="S3.E5.m1.2.2.1.1.1.1.2" xref="S3.E5.m1.2.2.1.1.1.1.2a.cmml">Q</mtext><mi id="S3.E5.m1.2.2.1.1.1.1.3" xref="S3.E5.m1.2.2.1.1.1.1.3.cmml">i</mi></msub><mo id="S3.E5.m1.4.4.3.3.3.5" xref="S3.E5.m1.4.4.3.3.4.cmml">,</mo><msub id="S3.E5.m1.3.3.2.2.2.2" xref="S3.E5.m1.3.3.2.2.2.2.cmml"><mtext class="ltx_mathvariant_bold" id="S3.E5.m1.3.3.2.2.2.2.2" xref="S3.E5.m1.3.3.2.2.2.2.2a.cmml">K</mtext><mi id="S3.E5.m1.3.3.2.2.2.2.3" xref="S3.E5.m1.3.3.2.2.2.2.3.cmml">i</mi></msub><mo id="S3.E5.m1.4.4.3.3.3.6" xref="S3.E5.m1.4.4.3.3.4.cmml">,</mo><msub id="S3.E5.m1.4.4.3.3.3.3" xref="S3.E5.m1.4.4.3.3.3.3.cmml"><mtext class="ltx_mathvariant_bold" id="S3.E5.m1.4.4.3.3.3.3.2" xref="S3.E5.m1.4.4.3.3.3.3.2a.cmml">V</mtext><mi id="S3.E5.m1.4.4.3.3.3.3.3" xref="S3.E5.m1.4.4.3.3.3.3.3.cmml">i</mi></msub><mo stretchy="false" id="S3.E5.m1.4.4.3.3.3.7" xref="S3.E5.m1.4.4.3.3.4.cmml">)</mo></mrow></mrow><mo id="S3.E5.m1.4.4.4" xref="S3.E5.m1.4.4.4.cmml">=</mo><mrow id="S3.E5.m1.4.4.5" xref="S3.E5.m1.4.4.5.cmml"><mtext id="S3.E5.m1.4.4.5.2" xref="S3.E5.m1.4.4.5.2a.cmml">softmax</mtext><mo lspace="0em" rspace="0em" id="S3.E5.m1.4.4.5.1" xref="S3.E5.m1.4.4.5.1.cmml">​</mo><mrow id="S3.E5.m1.4.4.5.3.2" xref="S3.E5.m1.1.1.cmml"><mo stretchy="false" id="S3.E5.m1.4.4.5.3.2.1" xref="S3.E5.m1.1.1.cmml">(</mo><mfrac id="S3.E5.m1.1.1" xref="S3.E5.m1.1.1.cmml"><mrow id="S3.E5.m1.1.1.2" xref="S3.E5.m1.1.1.2.cmml"><msub id="S3.E5.m1.1.1.2.2" xref="S3.E5.m1.1.1.2.2.cmml"><mtext class="ltx_mathvariant_bold" id="S3.E5.m1.1.1.2.2.2" xref="S3.E5.m1.1.1.2.2.2a.cmml">Q</mtext><mi id="S3.E5.m1.1.1.2.2.3" xref="S3.E5.m1.1.1.2.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S3.E5.m1.1.1.2.1" xref="S3.E5.m1.1.1.2.1.cmml">​</mo><msubsup id="S3.E5.m1.1.1.2.3" xref="S3.E5.m1.1.1.2.3.cmml"><mtext class="ltx_mathvariant_bold" id="S3.E5.m1.1.1.2.3.2.2" xref="S3.E5.m1.1.1.2.3.2.2a.cmml">K</mtext><mi id="S3.E5.m1.1.1.2.3.2.3" xref="S3.E5.m1.1.1.2.3.2.3.cmml">i</mi><mi id="S3.E5.m1.1.1.2.3.3" xref="S3.E5.m1.1.1.2.3.3.cmml">T</mi></msubsup></mrow><msqrt id="S3.E5.m1.1.1.3" xref="S3.E5.m1.1.1.3.cmml"><mi id="S3.E5.m1.1.1.3.2" xref="S3.E5.m1.1.1.3.2.cmml">D</mi></msqrt></mfrac><mo stretchy="false" id="S3.E5.m1.4.4.5.3.2.2" xref="S3.E5.m1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S3.E5.m1.4.4.5.1a" xref="S3.E5.m1.4.4.5.1.cmml">​</mo><msub id="S3.E5.m1.4.4.5.4" xref="S3.E5.m1.4.4.5.4.cmml"><mtext class="ltx_mathvariant_bold" id="S3.E5.m1.4.4.5.4.2" xref="S3.E5.m1.4.4.5.4.2a.cmml">V</mtext><mi id="S3.E5.m1.4.4.5.4.3" xref="S3.E5.m1.4.4.5.4.3.cmml">i</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E5.m1.4b"><apply id="S3.E5.m1.4.4.cmml" xref="S3.E5.m1.4.4"><eq id="S3.E5.m1.4.4.4.cmml" xref="S3.E5.m1.4.4.4"></eq><apply id="S3.E5.m1.4.4.3.cmml" xref="S3.E5.m1.4.4.3"><times id="S3.E5.m1.4.4.3.4.cmml" xref="S3.E5.m1.4.4.3.4"></times><ci id="S3.E5.m1.4.4.3.5a.cmml" xref="S3.E5.m1.4.4.3.5"><mtext id="S3.E5.m1.4.4.3.5.cmml" xref="S3.E5.m1.4.4.3.5">Attention</mtext></ci><vector id="S3.E5.m1.4.4.3.3.4.cmml" xref="S3.E5.m1.4.4.3.3.3"><apply id="S3.E5.m1.2.2.1.1.1.1.cmml" xref="S3.E5.m1.2.2.1.1.1.1"><csymbol cd="ambiguous" id="S3.E5.m1.2.2.1.1.1.1.1.cmml" xref="S3.E5.m1.2.2.1.1.1.1">subscript</csymbol><ci id="S3.E5.m1.2.2.1.1.1.1.2a.cmml" xref="S3.E5.m1.2.2.1.1.1.1.2"><mtext class="ltx_mathvariant_bold" id="S3.E5.m1.2.2.1.1.1.1.2.cmml" xref="S3.E5.m1.2.2.1.1.1.1.2">Q</mtext></ci><ci id="S3.E5.m1.2.2.1.1.1.1.3.cmml" xref="S3.E5.m1.2.2.1.1.1.1.3">𝑖</ci></apply><apply id="S3.E5.m1.3.3.2.2.2.2.cmml" xref="S3.E5.m1.3.3.2.2.2.2"><csymbol cd="ambiguous" id="S3.E5.m1.3.3.2.2.2.2.1.cmml" xref="S3.E5.m1.3.3.2.2.2.2">subscript</csymbol><ci id="S3.E5.m1.3.3.2.2.2.2.2a.cmml" xref="S3.E5.m1.3.3.2.2.2.2.2"><mtext class="ltx_mathvariant_bold" id="S3.E5.m1.3.3.2.2.2.2.2.cmml" xref="S3.E5.m1.3.3.2.2.2.2.2">K</mtext></ci><ci id="S3.E5.m1.3.3.2.2.2.2.3.cmml" xref="S3.E5.m1.3.3.2.2.2.2.3">𝑖</ci></apply><apply id="S3.E5.m1.4.4.3.3.3.3.cmml" xref="S3.E5.m1.4.4.3.3.3.3"><csymbol cd="ambiguous" id="S3.E5.m1.4.4.3.3.3.3.1.cmml" xref="S3.E5.m1.4.4.3.3.3.3">subscript</csymbol><ci id="S3.E5.m1.4.4.3.3.3.3.2a.cmml" xref="S3.E5.m1.4.4.3.3.3.3.2"><mtext class="ltx_mathvariant_bold" id="S3.E5.m1.4.4.3.3.3.3.2.cmml" xref="S3.E5.m1.4.4.3.3.3.3.2">V</mtext></ci><ci id="S3.E5.m1.4.4.3.3.3.3.3.cmml" xref="S3.E5.m1.4.4.3.3.3.3.3">𝑖</ci></apply></vector></apply><apply id="S3.E5.m1.4.4.5.cmml" xref="S3.E5.m1.4.4.5"><times id="S3.E5.m1.4.4.5.1.cmml" xref="S3.E5.m1.4.4.5.1"></times><ci id="S3.E5.m1.4.4.5.2a.cmml" xref="S3.E5.m1.4.4.5.2"><mtext id="S3.E5.m1.4.4.5.2.cmml" xref="S3.E5.m1.4.4.5.2">softmax</mtext></ci><apply id="S3.E5.m1.1.1.cmml" xref="S3.E5.m1.4.4.5.3.2"><divide id="S3.E5.m1.1.1.1.cmml" xref="S3.E5.m1.4.4.5.3.2"></divide><apply id="S3.E5.m1.1.1.2.cmml" xref="S3.E5.m1.1.1.2"><times id="S3.E5.m1.1.1.2.1.cmml" xref="S3.E5.m1.1.1.2.1"></times><apply id="S3.E5.m1.1.1.2.2.cmml" xref="S3.E5.m1.1.1.2.2"><csymbol cd="ambiguous" id="S3.E5.m1.1.1.2.2.1.cmml" xref="S3.E5.m1.1.1.2.2">subscript</csymbol><ci id="S3.E5.m1.1.1.2.2.2a.cmml" xref="S3.E5.m1.1.1.2.2.2"><mtext class="ltx_mathvariant_bold" id="S3.E5.m1.1.1.2.2.2.cmml" xref="S3.E5.m1.1.1.2.2.2">Q</mtext></ci><ci id="S3.E5.m1.1.1.2.2.3.cmml" xref="S3.E5.m1.1.1.2.2.3">𝑖</ci></apply><apply id="S3.E5.m1.1.1.2.3.cmml" xref="S3.E5.m1.1.1.2.3"><csymbol cd="ambiguous" id="S3.E5.m1.1.1.2.3.1.cmml" xref="S3.E5.m1.1.1.2.3">superscript</csymbol><apply id="S3.E5.m1.1.1.2.3.2.cmml" xref="S3.E5.m1.1.1.2.3"><csymbol cd="ambiguous" id="S3.E5.m1.1.1.2.3.2.1.cmml" xref="S3.E5.m1.1.1.2.3">subscript</csymbol><ci id="S3.E5.m1.1.1.2.3.2.2a.cmml" xref="S3.E5.m1.1.1.2.3.2.2"><mtext class="ltx_mathvariant_bold" id="S3.E5.m1.1.1.2.3.2.2.cmml" xref="S3.E5.m1.1.1.2.3.2.2">K</mtext></ci><ci id="S3.E5.m1.1.1.2.3.2.3.cmml" xref="S3.E5.m1.1.1.2.3.2.3">𝑖</ci></apply><ci id="S3.E5.m1.1.1.2.3.3.cmml" xref="S3.E5.m1.1.1.2.3.3">𝑇</ci></apply></apply><apply id="S3.E5.m1.1.1.3.cmml" xref="S3.E5.m1.1.1.3"><root id="S3.E5.m1.1.1.3a.cmml" xref="S3.E5.m1.1.1.3"></root><ci id="S3.E5.m1.1.1.3.2.cmml" xref="S3.E5.m1.1.1.3.2">𝐷</ci></apply></apply><apply id="S3.E5.m1.4.4.5.4.cmml" xref="S3.E5.m1.4.4.5.4"><csymbol cd="ambiguous" id="S3.E5.m1.4.4.5.4.1.cmml" xref="S3.E5.m1.4.4.5.4">subscript</csymbol><ci id="S3.E5.m1.4.4.5.4.2a.cmml" xref="S3.E5.m1.4.4.5.4.2"><mtext class="ltx_mathvariant_bold" id="S3.E5.m1.4.4.5.4.2.cmml" xref="S3.E5.m1.4.4.5.4.2">V</mtext></ci><ci id="S3.E5.m1.4.4.5.4.3.cmml" xref="S3.E5.m1.4.4.5.4.3">𝑖</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E5.m1.4c">\text{Attention}(\textbf{Q}_{i},\textbf{K}_{i},\textbf{V}_{i})=\text{softmax}(\frac{\textbf{Q}_{i}\textbf{K}_{i}^{T}}{\sqrt{D}})\textbf{V}_{i}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(5)</span></td>
</tr></tbody>
</table>
<p id="S3.SS2.p1.8" class="ltx_p">where <math id="S3.SS2.p1.2.m1.3" class="ltx_Math" alttext="\textbf{Q}_{i},\textbf{K}_{i},\textbf{V}_{i}\in\mathbb{R}^{S\times D}" display="inline"><semantics id="S3.SS2.p1.2.m1.3a"><mrow id="S3.SS2.p1.2.m1.3.3" xref="S3.SS2.p1.2.m1.3.3.cmml"><mrow id="S3.SS2.p1.2.m1.3.3.3.3" xref="S3.SS2.p1.2.m1.3.3.3.4.cmml"><msub id="S3.SS2.p1.2.m1.1.1.1.1.1" xref="S3.SS2.p1.2.m1.1.1.1.1.1.cmml"><mtext class="ltx_mathvariant_bold" id="S3.SS2.p1.2.m1.1.1.1.1.1.2" xref="S3.SS2.p1.2.m1.1.1.1.1.1.2a.cmml">Q</mtext><mi id="S3.SS2.p1.2.m1.1.1.1.1.1.3" xref="S3.SS2.p1.2.m1.1.1.1.1.1.3.cmml">i</mi></msub><mo id="S3.SS2.p1.2.m1.3.3.3.3.4" xref="S3.SS2.p1.2.m1.3.3.3.4.cmml">,</mo><msub id="S3.SS2.p1.2.m1.2.2.2.2.2" xref="S3.SS2.p1.2.m1.2.2.2.2.2.cmml"><mtext class="ltx_mathvariant_bold" id="S3.SS2.p1.2.m1.2.2.2.2.2.2" xref="S3.SS2.p1.2.m1.2.2.2.2.2.2a.cmml">K</mtext><mi id="S3.SS2.p1.2.m1.2.2.2.2.2.3" xref="S3.SS2.p1.2.m1.2.2.2.2.2.3.cmml">i</mi></msub><mo id="S3.SS2.p1.2.m1.3.3.3.3.5" xref="S3.SS2.p1.2.m1.3.3.3.4.cmml">,</mo><msub id="S3.SS2.p1.2.m1.3.3.3.3.3" xref="S3.SS2.p1.2.m1.3.3.3.3.3.cmml"><mtext class="ltx_mathvariant_bold" id="S3.SS2.p1.2.m1.3.3.3.3.3.2" xref="S3.SS2.p1.2.m1.3.3.3.3.3.2a.cmml">V</mtext><mi id="S3.SS2.p1.2.m1.3.3.3.3.3.3" xref="S3.SS2.p1.2.m1.3.3.3.3.3.3.cmml">i</mi></msub></mrow><mo id="S3.SS2.p1.2.m1.3.3.4" xref="S3.SS2.p1.2.m1.3.3.4.cmml">∈</mo><msup id="S3.SS2.p1.2.m1.3.3.5" xref="S3.SS2.p1.2.m1.3.3.5.cmml"><mi id="S3.SS2.p1.2.m1.3.3.5.2" xref="S3.SS2.p1.2.m1.3.3.5.2.cmml">ℝ</mi><mrow id="S3.SS2.p1.2.m1.3.3.5.3" xref="S3.SS2.p1.2.m1.3.3.5.3.cmml"><mi id="S3.SS2.p1.2.m1.3.3.5.3.2" xref="S3.SS2.p1.2.m1.3.3.5.3.2.cmml">S</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.p1.2.m1.3.3.5.3.1" xref="S3.SS2.p1.2.m1.3.3.5.3.1.cmml">×</mo><mi id="S3.SS2.p1.2.m1.3.3.5.3.3" xref="S3.SS2.p1.2.m1.3.3.5.3.3.cmml">D</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.2.m1.3b"><apply id="S3.SS2.p1.2.m1.3.3.cmml" xref="S3.SS2.p1.2.m1.3.3"><in id="S3.SS2.p1.2.m1.3.3.4.cmml" xref="S3.SS2.p1.2.m1.3.3.4"></in><list id="S3.SS2.p1.2.m1.3.3.3.4.cmml" xref="S3.SS2.p1.2.m1.3.3.3.3"><apply id="S3.SS2.p1.2.m1.1.1.1.1.1.cmml" xref="S3.SS2.p1.2.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.2.m1.1.1.1.1.1.1.cmml" xref="S3.SS2.p1.2.m1.1.1.1.1.1">subscript</csymbol><ci id="S3.SS2.p1.2.m1.1.1.1.1.1.2a.cmml" xref="S3.SS2.p1.2.m1.1.1.1.1.1.2"><mtext class="ltx_mathvariant_bold" id="S3.SS2.p1.2.m1.1.1.1.1.1.2.cmml" xref="S3.SS2.p1.2.m1.1.1.1.1.1.2">Q</mtext></ci><ci id="S3.SS2.p1.2.m1.1.1.1.1.1.3.cmml" xref="S3.SS2.p1.2.m1.1.1.1.1.1.3">𝑖</ci></apply><apply id="S3.SS2.p1.2.m1.2.2.2.2.2.cmml" xref="S3.SS2.p1.2.m1.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS2.p1.2.m1.2.2.2.2.2.1.cmml" xref="S3.SS2.p1.2.m1.2.2.2.2.2">subscript</csymbol><ci id="S3.SS2.p1.2.m1.2.2.2.2.2.2a.cmml" xref="S3.SS2.p1.2.m1.2.2.2.2.2.2"><mtext class="ltx_mathvariant_bold" id="S3.SS2.p1.2.m1.2.2.2.2.2.2.cmml" xref="S3.SS2.p1.2.m1.2.2.2.2.2.2">K</mtext></ci><ci id="S3.SS2.p1.2.m1.2.2.2.2.2.3.cmml" xref="S3.SS2.p1.2.m1.2.2.2.2.2.3">𝑖</ci></apply><apply id="S3.SS2.p1.2.m1.3.3.3.3.3.cmml" xref="S3.SS2.p1.2.m1.3.3.3.3.3"><csymbol cd="ambiguous" id="S3.SS2.p1.2.m1.3.3.3.3.3.1.cmml" xref="S3.SS2.p1.2.m1.3.3.3.3.3">subscript</csymbol><ci id="S3.SS2.p1.2.m1.3.3.3.3.3.2a.cmml" xref="S3.SS2.p1.2.m1.3.3.3.3.3.2"><mtext class="ltx_mathvariant_bold" id="S3.SS2.p1.2.m1.3.3.3.3.3.2.cmml" xref="S3.SS2.p1.2.m1.3.3.3.3.3.2">V</mtext></ci><ci id="S3.SS2.p1.2.m1.3.3.3.3.3.3.cmml" xref="S3.SS2.p1.2.m1.3.3.3.3.3.3">𝑖</ci></apply></list><apply id="S3.SS2.p1.2.m1.3.3.5.cmml" xref="S3.SS2.p1.2.m1.3.3.5"><csymbol cd="ambiguous" id="S3.SS2.p1.2.m1.3.3.5.1.cmml" xref="S3.SS2.p1.2.m1.3.3.5">superscript</csymbol><ci id="S3.SS2.p1.2.m1.3.3.5.2.cmml" xref="S3.SS2.p1.2.m1.3.3.5.2">ℝ</ci><apply id="S3.SS2.p1.2.m1.3.3.5.3.cmml" xref="S3.SS2.p1.2.m1.3.3.5.3"><times id="S3.SS2.p1.2.m1.3.3.5.3.1.cmml" xref="S3.SS2.p1.2.m1.3.3.5.3.1"></times><ci id="S3.SS2.p1.2.m1.3.3.5.3.2.cmml" xref="S3.SS2.p1.2.m1.3.3.5.3.2">𝑆</ci><ci id="S3.SS2.p1.2.m1.3.3.5.3.3.cmml" xref="S3.SS2.p1.2.m1.3.3.5.3.3">𝐷</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.2.m1.3c">\textbf{Q}_{i},\textbf{K}_{i},\textbf{V}_{i}\in\mathbb{R}^{S\times D}</annotation></semantics></math>. When pruning one channels of <math id="S3.SS2.p1.3.m2.1" class="ltx_Math" alttext="\textbf{K}_{i}" display="inline"><semantics id="S3.SS2.p1.3.m2.1a"><msub id="S3.SS2.p1.3.m2.1.1" xref="S3.SS2.p1.3.m2.1.1.cmml"><mtext class="ltx_mathvariant_bold" id="S3.SS2.p1.3.m2.1.1.2" xref="S3.SS2.p1.3.m2.1.1.2a.cmml">K</mtext><mi id="S3.SS2.p1.3.m2.1.1.3" xref="S3.SS2.p1.3.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.3.m2.1b"><apply id="S3.SS2.p1.3.m2.1.1.cmml" xref="S3.SS2.p1.3.m2.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.3.m2.1.1.1.cmml" xref="S3.SS2.p1.3.m2.1.1">subscript</csymbol><ci id="S3.SS2.p1.3.m2.1.1.2a.cmml" xref="S3.SS2.p1.3.m2.1.1.2"><mtext class="ltx_mathvariant_bold" id="S3.SS2.p1.3.m2.1.1.2.cmml" xref="S3.SS2.p1.3.m2.1.1.2">K</mtext></ci><ci id="S3.SS2.p1.3.m2.1.1.3.cmml" xref="S3.SS2.p1.3.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.3.m2.1c">\textbf{K}_{i}</annotation></semantics></math>, the corresponding channels in <math id="S3.SS2.p1.4.m3.1" class="ltx_Math" alttext="\textbf{Q}_{i}" display="inline"><semantics id="S3.SS2.p1.4.m3.1a"><msub id="S3.SS2.p1.4.m3.1.1" xref="S3.SS2.p1.4.m3.1.1.cmml"><mtext class="ltx_mathvariant_bold" id="S3.SS2.p1.4.m3.1.1.2" xref="S3.SS2.p1.4.m3.1.1.2a.cmml">Q</mtext><mi id="S3.SS2.p1.4.m3.1.1.3" xref="S3.SS2.p1.4.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.4.m3.1b"><apply id="S3.SS2.p1.4.m3.1.1.cmml" xref="S3.SS2.p1.4.m3.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.4.m3.1.1.1.cmml" xref="S3.SS2.p1.4.m3.1.1">subscript</csymbol><ci id="S3.SS2.p1.4.m3.1.1.2a.cmml" xref="S3.SS2.p1.4.m3.1.1.2"><mtext class="ltx_mathvariant_bold" id="S3.SS2.p1.4.m3.1.1.2.cmml" xref="S3.SS2.p1.4.m3.1.1.2">Q</mtext></ci><ci id="S3.SS2.p1.4.m3.1.1.3.cmml" xref="S3.SS2.p1.4.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.4.m3.1c">\textbf{Q}_{i}</annotation></semantics></math> will also be removed. We aim to find the optimal subset of channels to prune, denoted by the selection matrix <math id="S3.SS2.p1.5.m4.2" class="ltx_Math" alttext="\textbf{S}\in\{0,1\}^{D\times D}" display="inline"><semantics id="S3.SS2.p1.5.m4.2a"><mrow id="S3.SS2.p1.5.m4.2.3" xref="S3.SS2.p1.5.m4.2.3.cmml"><mtext class="ltx_mathvariant_bold" id="S3.SS2.p1.5.m4.2.3.2" xref="S3.SS2.p1.5.m4.2.3.2a.cmml">S</mtext><mo id="S3.SS2.p1.5.m4.2.3.1" xref="S3.SS2.p1.5.m4.2.3.1.cmml">∈</mo><msup id="S3.SS2.p1.5.m4.2.3.3" xref="S3.SS2.p1.5.m4.2.3.3.cmml"><mrow id="S3.SS2.p1.5.m4.2.3.3.2.2" xref="S3.SS2.p1.5.m4.2.3.3.2.1.cmml"><mo stretchy="false" id="S3.SS2.p1.5.m4.2.3.3.2.2.1" xref="S3.SS2.p1.5.m4.2.3.3.2.1.cmml">{</mo><mn id="S3.SS2.p1.5.m4.1.1" xref="S3.SS2.p1.5.m4.1.1.cmml">0</mn><mo id="S3.SS2.p1.5.m4.2.3.3.2.2.2" xref="S3.SS2.p1.5.m4.2.3.3.2.1.cmml">,</mo><mn id="S3.SS2.p1.5.m4.2.2" xref="S3.SS2.p1.5.m4.2.2.cmml">1</mn><mo stretchy="false" id="S3.SS2.p1.5.m4.2.3.3.2.2.3" xref="S3.SS2.p1.5.m4.2.3.3.2.1.cmml">}</mo></mrow><mrow id="S3.SS2.p1.5.m4.2.3.3.3" xref="S3.SS2.p1.5.m4.2.3.3.3.cmml"><mi id="S3.SS2.p1.5.m4.2.3.3.3.2" xref="S3.SS2.p1.5.m4.2.3.3.3.2.cmml">D</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.p1.5.m4.2.3.3.3.1" xref="S3.SS2.p1.5.m4.2.3.3.3.1.cmml">×</mo><mi id="S3.SS2.p1.5.m4.2.3.3.3.3" xref="S3.SS2.p1.5.m4.2.3.3.3.3.cmml">D</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.5.m4.2b"><apply id="S3.SS2.p1.5.m4.2.3.cmml" xref="S3.SS2.p1.5.m4.2.3"><in id="S3.SS2.p1.5.m4.2.3.1.cmml" xref="S3.SS2.p1.5.m4.2.3.1"></in><ci id="S3.SS2.p1.5.m4.2.3.2a.cmml" xref="S3.SS2.p1.5.m4.2.3.2"><mtext class="ltx_mathvariant_bold" id="S3.SS2.p1.5.m4.2.3.2.cmml" xref="S3.SS2.p1.5.m4.2.3.2">S</mtext></ci><apply id="S3.SS2.p1.5.m4.2.3.3.cmml" xref="S3.SS2.p1.5.m4.2.3.3"><csymbol cd="ambiguous" id="S3.SS2.p1.5.m4.2.3.3.1.cmml" xref="S3.SS2.p1.5.m4.2.3.3">superscript</csymbol><set id="S3.SS2.p1.5.m4.2.3.3.2.1.cmml" xref="S3.SS2.p1.5.m4.2.3.3.2.2"><cn type="integer" id="S3.SS2.p1.5.m4.1.1.cmml" xref="S3.SS2.p1.5.m4.1.1">0</cn><cn type="integer" id="S3.SS2.p1.5.m4.2.2.cmml" xref="S3.SS2.p1.5.m4.2.2">1</cn></set><apply id="S3.SS2.p1.5.m4.2.3.3.3.cmml" xref="S3.SS2.p1.5.m4.2.3.3.3"><times id="S3.SS2.p1.5.m4.2.3.3.3.1.cmml" xref="S3.SS2.p1.5.m4.2.3.3.3.1"></times><ci id="S3.SS2.p1.5.m4.2.3.3.3.2.cmml" xref="S3.SS2.p1.5.m4.2.3.3.3.2">𝐷</ci><ci id="S3.SS2.p1.5.m4.2.3.3.3.3.cmml" xref="S3.SS2.p1.5.m4.2.3.3.3.3">𝐷</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.5.m4.2c">\textbf{S}\in\{0,1\}^{D\times D}</annotation></semantics></math>, where
<span id="S3.SS2.p1.8.1" class="ltx_text ltx_markedasmath ltx_font_bold">S</span> is a diagonal matrix with binary entries (<math id="S3.SS2.p1.7.m6.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S3.SS2.p1.7.m6.1a"><mn id="S3.SS2.p1.7.m6.1.1" xref="S3.SS2.p1.7.m6.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.7.m6.1b"><cn type="integer" id="S3.SS2.p1.7.m6.1.1.cmml" xref="S3.SS2.p1.7.m6.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.7.m6.1c">1</annotation></semantics></math> for keeping a channel, <math id="S3.SS2.p1.8.m7.1" class="ltx_Math" alttext="0" display="inline"><semantics id="S3.SS2.p1.8.m7.1a"><mn id="S3.SS2.p1.8.m7.1.1" xref="S3.SS2.p1.8.m7.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.8.m7.1b"><cn type="integer" id="S3.SS2.p1.8.m7.1.1.cmml" xref="S3.SS2.p1.8.m7.1.1">0</cn></annotation-xml></semantics></math> for pruning it). To better maintain the performance after pruning the channels, we minimize the Frobenius norm of the difference between the original and pruned attention weights:</p>
<table id="S3.E6" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E6.m1.1" class="ltx_Math" alttext="\min_{\textbf{S}}\|\textbf{Q}_{i}\textbf{K}^{T}_{i}-\textbf{Q}_{i}\textbf{S}(\textbf{K}_{i}\textbf{S})^{T}\|_{F}" display="block"><semantics id="S3.E6.m1.1a"><mrow id="S3.E6.m1.1.1" xref="S3.E6.m1.1.1.cmml"><munder id="S3.E6.m1.1.1.2" xref="S3.E6.m1.1.1.2.cmml"><mi id="S3.E6.m1.1.1.2.2" xref="S3.E6.m1.1.1.2.2.cmml">min</mi><mtext class="ltx_mathvariant_bold" id="S3.E6.m1.1.1.2.3" xref="S3.E6.m1.1.1.2.3a.cmml">S</mtext></munder><mo id="S3.E6.m1.1.1a" xref="S3.E6.m1.1.1.cmml">⁡</mo><msub id="S3.E6.m1.1.1.1" xref="S3.E6.m1.1.1.1.cmml"><mrow id="S3.E6.m1.1.1.1.1.1" xref="S3.E6.m1.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.E6.m1.1.1.1.1.1.2" xref="S3.E6.m1.1.1.1.1.2.1.cmml">‖</mo><mrow id="S3.E6.m1.1.1.1.1.1.1" xref="S3.E6.m1.1.1.1.1.1.1.cmml"><mrow id="S3.E6.m1.1.1.1.1.1.1.3" xref="S3.E6.m1.1.1.1.1.1.1.3.cmml"><msub id="S3.E6.m1.1.1.1.1.1.1.3.2" xref="S3.E6.m1.1.1.1.1.1.1.3.2.cmml"><mtext class="ltx_mathvariant_bold" id="S3.E6.m1.1.1.1.1.1.1.3.2.2" xref="S3.E6.m1.1.1.1.1.1.1.3.2.2a.cmml">Q</mtext><mi id="S3.E6.m1.1.1.1.1.1.1.3.2.3" xref="S3.E6.m1.1.1.1.1.1.1.3.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S3.E6.m1.1.1.1.1.1.1.3.1" xref="S3.E6.m1.1.1.1.1.1.1.3.1.cmml">​</mo><msubsup id="S3.E6.m1.1.1.1.1.1.1.3.3" xref="S3.E6.m1.1.1.1.1.1.1.3.3.cmml"><mtext class="ltx_mathvariant_bold" id="S3.E6.m1.1.1.1.1.1.1.3.3.2.2" xref="S3.E6.m1.1.1.1.1.1.1.3.3.2.2a.cmml">K</mtext><mi id="S3.E6.m1.1.1.1.1.1.1.3.3.3" xref="S3.E6.m1.1.1.1.1.1.1.3.3.3.cmml">i</mi><mi id="S3.E6.m1.1.1.1.1.1.1.3.3.2.3" xref="S3.E6.m1.1.1.1.1.1.1.3.3.2.3.cmml">T</mi></msubsup></mrow><mo id="S3.E6.m1.1.1.1.1.1.1.2" xref="S3.E6.m1.1.1.1.1.1.1.2.cmml">−</mo><mrow id="S3.E6.m1.1.1.1.1.1.1.1" xref="S3.E6.m1.1.1.1.1.1.1.1.cmml"><msub id="S3.E6.m1.1.1.1.1.1.1.1.3" xref="S3.E6.m1.1.1.1.1.1.1.1.3.cmml"><mtext class="ltx_mathvariant_bold" id="S3.E6.m1.1.1.1.1.1.1.1.3.2" xref="S3.E6.m1.1.1.1.1.1.1.1.3.2a.cmml">Q</mtext><mi id="S3.E6.m1.1.1.1.1.1.1.1.3.3" xref="S3.E6.m1.1.1.1.1.1.1.1.3.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S3.E6.m1.1.1.1.1.1.1.1.2" xref="S3.E6.m1.1.1.1.1.1.1.1.2.cmml">​</mo><mtext class="ltx_mathvariant_bold" id="S3.E6.m1.1.1.1.1.1.1.1.4" xref="S3.E6.m1.1.1.1.1.1.1.1.4a.cmml">S</mtext><mo lspace="0em" rspace="0em" id="S3.E6.m1.1.1.1.1.1.1.1.2a" xref="S3.E6.m1.1.1.1.1.1.1.1.2.cmml">​</mo><msup id="S3.E6.m1.1.1.1.1.1.1.1.1" xref="S3.E6.m1.1.1.1.1.1.1.1.1.cmml"><mrow id="S3.E6.m1.1.1.1.1.1.1.1.1.1.1" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.2" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.cmml"><msub id="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2.cmml"><mtext class="ltx_mathvariant_bold" id="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2.2" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2.2a.cmml">K</mtext><mi id="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2.3" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">​</mo><mtext class="ltx_mathvariant_bold" id="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.3a.cmml">S</mtext></mrow><mo stretchy="false" id="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.3" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mi id="S3.E6.m1.1.1.1.1.1.1.1.1.3" xref="S3.E6.m1.1.1.1.1.1.1.1.1.3.cmml">T</mi></msup></mrow></mrow><mo stretchy="false" id="S3.E6.m1.1.1.1.1.1.3" xref="S3.E6.m1.1.1.1.1.2.1.cmml">‖</mo></mrow><mi id="S3.E6.m1.1.1.1.3" xref="S3.E6.m1.1.1.1.3.cmml">F</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.E6.m1.1b"><apply id="S3.E6.m1.1.1.cmml" xref="S3.E6.m1.1.1"><apply id="S3.E6.m1.1.1.2.cmml" xref="S3.E6.m1.1.1.2"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.2.1.cmml" xref="S3.E6.m1.1.1.2">subscript</csymbol><min id="S3.E6.m1.1.1.2.2.cmml" xref="S3.E6.m1.1.1.2.2"></min><ci id="S3.E6.m1.1.1.2.3a.cmml" xref="S3.E6.m1.1.1.2.3"><mtext class="ltx_mathvariant_bold" mathsize="70%" id="S3.E6.m1.1.1.2.3.cmml" xref="S3.E6.m1.1.1.2.3">S</mtext></ci></apply><apply id="S3.E6.m1.1.1.1.cmml" xref="S3.E6.m1.1.1.1"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.2.cmml" xref="S3.E6.m1.1.1.1">subscript</csymbol><apply id="S3.E6.m1.1.1.1.1.2.cmml" xref="S3.E6.m1.1.1.1.1.1"><csymbol cd="latexml" id="S3.E6.m1.1.1.1.1.2.1.cmml" xref="S3.E6.m1.1.1.1.1.1.2">norm</csymbol><apply id="S3.E6.m1.1.1.1.1.1.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1"><minus id="S3.E6.m1.1.1.1.1.1.1.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.2"></minus><apply id="S3.E6.m1.1.1.1.1.1.1.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.3"><times id="S3.E6.m1.1.1.1.1.1.1.3.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.3.1"></times><apply id="S3.E6.m1.1.1.1.1.1.1.3.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.1.1.3.2.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.3.2">subscript</csymbol><ci id="S3.E6.m1.1.1.1.1.1.1.3.2.2a.cmml" xref="S3.E6.m1.1.1.1.1.1.1.3.2.2"><mtext class="ltx_mathvariant_bold" id="S3.E6.m1.1.1.1.1.1.1.3.2.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.3.2.2">Q</mtext></ci><ci id="S3.E6.m1.1.1.1.1.1.1.3.2.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.3.2.3">𝑖</ci></apply><apply id="S3.E6.m1.1.1.1.1.1.1.3.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.1.1.3.3.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.3.3">subscript</csymbol><apply id="S3.E6.m1.1.1.1.1.1.1.3.3.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.1.1.3.3.2.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.3.3">superscript</csymbol><ci id="S3.E6.m1.1.1.1.1.1.1.3.3.2.2a.cmml" xref="S3.E6.m1.1.1.1.1.1.1.3.3.2.2"><mtext class="ltx_mathvariant_bold" id="S3.E6.m1.1.1.1.1.1.1.3.3.2.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.3.3.2.2">K</mtext></ci><ci id="S3.E6.m1.1.1.1.1.1.1.3.3.2.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.3.3.2.3">𝑇</ci></apply><ci id="S3.E6.m1.1.1.1.1.1.1.3.3.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.3.3.3">𝑖</ci></apply></apply><apply id="S3.E6.m1.1.1.1.1.1.1.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1"><times id="S3.E6.m1.1.1.1.1.1.1.1.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.2"></times><apply id="S3.E6.m1.1.1.1.1.1.1.1.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.E6.m1.1.1.1.1.1.1.1.3.2a.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.3.2"><mtext class="ltx_mathvariant_bold" id="S3.E6.m1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.3.2">Q</mtext></ci><ci id="S3.E6.m1.1.1.1.1.1.1.1.3.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.3.3">𝑖</ci></apply><ci id="S3.E6.m1.1.1.1.1.1.1.1.4a.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.4"><mtext class="ltx_mathvariant_bold" id="S3.E6.m1.1.1.1.1.1.1.1.4.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.4">S</mtext></ci><apply id="S3.E6.m1.1.1.1.1.1.1.1.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.1">superscript</csymbol><apply id="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.1"><times id="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1"></times><apply id="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2.2a.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2.2"><mtext class="ltx_mathvariant_bold" id="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2.2">K</mtext></ci><ci id="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2.3">𝑖</ci></apply><ci id="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.3a.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.3"><mtext class="ltx_mathvariant_bold" id="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.1.3">S</mtext></ci></apply><ci id="S3.E6.m1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.1.3">𝑇</ci></apply></apply></apply></apply><ci id="S3.E6.m1.1.1.1.3.cmml" xref="S3.E6.m1.1.1.1.3">𝐹</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E6.m1.1c">\min_{\textbf{S}}\|\textbf{Q}_{i}\textbf{K}^{T}_{i}-\textbf{Q}_{i}\textbf{S}(\textbf{K}_{i}\textbf{S})^{T}\|_{F}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(6)</span></td>
</tr></tbody>
</table>
</div>
<div id="S3.SS2.p2" class="ltx_para ltx_noindent">
<p id="S3.SS2.p2.1" class="ltx_p">Given a pruning ratio <math id="S3.SS2.p2.1.m1.1" class="ltx_Math" alttext="\lambda" display="inline"><semantics id="S3.SS2.p2.1.m1.1a"><mi id="S3.SS2.p2.1.m1.1.1" xref="S3.SS2.p2.1.m1.1.1.cmml">λ</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.1.m1.1b"><ci id="S3.SS2.p2.1.m1.1.1.cmml" xref="S3.SS2.p2.1.m1.1.1">𝜆</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.1.m1.1c">\lambda</annotation></semantics></math>, it can further expanded as:</p>
<table id="A1.EGx2" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S3.Ex1"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.Ex1.m1.1" class="ltx_Math" alttext="\displaystyle\min_{\mathbf{S}}" display="inline"><semantics id="S3.Ex1.m1.1a"><munder id="S3.Ex1.m1.1.1" xref="S3.Ex1.m1.1.1.cmml"><mi id="S3.Ex1.m1.1.1.2" xref="S3.Ex1.m1.1.1.2.cmml">min</mi><mi id="S3.Ex1.m1.1.1.3" xref="S3.Ex1.m1.1.1.3.cmml">𝐒</mi></munder><annotation-xml encoding="MathML-Content" id="S3.Ex1.m1.1b"><apply id="S3.Ex1.m1.1.1.cmml" xref="S3.Ex1.m1.1.1"><csymbol cd="ambiguous" id="S3.Ex1.m1.1.1.1.cmml" xref="S3.Ex1.m1.1.1">subscript</csymbol><min id="S3.Ex1.m1.1.1.2.cmml" xref="S3.Ex1.m1.1.1.2"></min><ci id="S3.Ex1.m1.1.1.3.cmml" xref="S3.Ex1.m1.1.1.3">𝐒</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.Ex1.m1.1c">\displaystyle\min_{\mathbf{S}}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.Ex1.m2.1" class="ltx_Math" alttext="\displaystyle\quad\left\|\mathbf{Q}_{i}\mathbf{K}_{i}^{T}-\mathbf{Q}_{i}\mathbf{S}\mathbf{K}_{i}^{T}\right\|_{F}" display="inline"><semantics id="S3.Ex1.m2.1a"><msub id="S3.Ex1.m2.1.1" xref="S3.Ex1.m2.1.1.cmml"><mrow id="S3.Ex1.m2.1.1.1.1" xref="S3.Ex1.m2.1.1.1.2.cmml"><mo id="S3.Ex1.m2.1.1.1.1.2" xref="S3.Ex1.m2.1.1.1.2.1.cmml">‖</mo><mrow id="S3.Ex1.m2.1.1.1.1.1" xref="S3.Ex1.m2.1.1.1.1.1.cmml"><mrow id="S3.Ex1.m2.1.1.1.1.1.2" xref="S3.Ex1.m2.1.1.1.1.1.2.cmml"><msub id="S3.Ex1.m2.1.1.1.1.1.2.2" xref="S3.Ex1.m2.1.1.1.1.1.2.2.cmml"><mi id="S3.Ex1.m2.1.1.1.1.1.2.2.2" xref="S3.Ex1.m2.1.1.1.1.1.2.2.2.cmml">𝐐</mi><mi id="S3.Ex1.m2.1.1.1.1.1.2.2.3" xref="S3.Ex1.m2.1.1.1.1.1.2.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S3.Ex1.m2.1.1.1.1.1.2.1" xref="S3.Ex1.m2.1.1.1.1.1.2.1.cmml">​</mo><msubsup id="S3.Ex1.m2.1.1.1.1.1.2.3" xref="S3.Ex1.m2.1.1.1.1.1.2.3.cmml"><mi id="S3.Ex1.m2.1.1.1.1.1.2.3.2.2" xref="S3.Ex1.m2.1.1.1.1.1.2.3.2.2.cmml">𝐊</mi><mi id="S3.Ex1.m2.1.1.1.1.1.2.3.2.3" xref="S3.Ex1.m2.1.1.1.1.1.2.3.2.3.cmml">i</mi><mi id="S3.Ex1.m2.1.1.1.1.1.2.3.3" xref="S3.Ex1.m2.1.1.1.1.1.2.3.3.cmml">T</mi></msubsup></mrow><mo id="S3.Ex1.m2.1.1.1.1.1.1" xref="S3.Ex1.m2.1.1.1.1.1.1.cmml">−</mo><mrow id="S3.Ex1.m2.1.1.1.1.1.3" xref="S3.Ex1.m2.1.1.1.1.1.3.cmml"><msub id="S3.Ex1.m2.1.1.1.1.1.3.2" xref="S3.Ex1.m2.1.1.1.1.1.3.2.cmml"><mi id="S3.Ex1.m2.1.1.1.1.1.3.2.2" xref="S3.Ex1.m2.1.1.1.1.1.3.2.2.cmml">𝐐</mi><mi id="S3.Ex1.m2.1.1.1.1.1.3.2.3" xref="S3.Ex1.m2.1.1.1.1.1.3.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S3.Ex1.m2.1.1.1.1.1.3.1" xref="S3.Ex1.m2.1.1.1.1.1.3.1.cmml">​</mo><msubsup id="S3.Ex1.m2.1.1.1.1.1.3.3" xref="S3.Ex1.m2.1.1.1.1.1.3.3.cmml"><mi id="S3.Ex1.m2.1.1.1.1.1.3.3.2.2" xref="S3.Ex1.m2.1.1.1.1.1.3.3.2.2.cmml">𝐒𝐊</mi><mi id="S3.Ex1.m2.1.1.1.1.1.3.3.2.3" xref="S3.Ex1.m2.1.1.1.1.1.3.3.2.3.cmml">i</mi><mi id="S3.Ex1.m2.1.1.1.1.1.3.3.3" xref="S3.Ex1.m2.1.1.1.1.1.3.3.3.cmml">T</mi></msubsup></mrow></mrow><mo id="S3.Ex1.m2.1.1.1.1.3" xref="S3.Ex1.m2.1.1.1.2.1.cmml">‖</mo></mrow><mi id="S3.Ex1.m2.1.1.3" xref="S3.Ex1.m2.1.1.3.cmml">F</mi></msub><annotation-xml encoding="MathML-Content" id="S3.Ex1.m2.1b"><apply id="S3.Ex1.m2.1.1.cmml" xref="S3.Ex1.m2.1.1"><csymbol cd="ambiguous" id="S3.Ex1.m2.1.1.2.cmml" xref="S3.Ex1.m2.1.1">subscript</csymbol><apply id="S3.Ex1.m2.1.1.1.2.cmml" xref="S3.Ex1.m2.1.1.1.1"><csymbol cd="latexml" id="S3.Ex1.m2.1.1.1.2.1.cmml" xref="S3.Ex1.m2.1.1.1.1.2">norm</csymbol><apply id="S3.Ex1.m2.1.1.1.1.1.cmml" xref="S3.Ex1.m2.1.1.1.1.1"><minus id="S3.Ex1.m2.1.1.1.1.1.1.cmml" xref="S3.Ex1.m2.1.1.1.1.1.1"></minus><apply id="S3.Ex1.m2.1.1.1.1.1.2.cmml" xref="S3.Ex1.m2.1.1.1.1.1.2"><times id="S3.Ex1.m2.1.1.1.1.1.2.1.cmml" xref="S3.Ex1.m2.1.1.1.1.1.2.1"></times><apply id="S3.Ex1.m2.1.1.1.1.1.2.2.cmml" xref="S3.Ex1.m2.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S3.Ex1.m2.1.1.1.1.1.2.2.1.cmml" xref="S3.Ex1.m2.1.1.1.1.1.2.2">subscript</csymbol><ci id="S3.Ex1.m2.1.1.1.1.1.2.2.2.cmml" xref="S3.Ex1.m2.1.1.1.1.1.2.2.2">𝐐</ci><ci id="S3.Ex1.m2.1.1.1.1.1.2.2.3.cmml" xref="S3.Ex1.m2.1.1.1.1.1.2.2.3">𝑖</ci></apply><apply id="S3.Ex1.m2.1.1.1.1.1.2.3.cmml" xref="S3.Ex1.m2.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="S3.Ex1.m2.1.1.1.1.1.2.3.1.cmml" xref="S3.Ex1.m2.1.1.1.1.1.2.3">superscript</csymbol><apply id="S3.Ex1.m2.1.1.1.1.1.2.3.2.cmml" xref="S3.Ex1.m2.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="S3.Ex1.m2.1.1.1.1.1.2.3.2.1.cmml" xref="S3.Ex1.m2.1.1.1.1.1.2.3">subscript</csymbol><ci id="S3.Ex1.m2.1.1.1.1.1.2.3.2.2.cmml" xref="S3.Ex1.m2.1.1.1.1.1.2.3.2.2">𝐊</ci><ci id="S3.Ex1.m2.1.1.1.1.1.2.3.2.3.cmml" xref="S3.Ex1.m2.1.1.1.1.1.2.3.2.3">𝑖</ci></apply><ci id="S3.Ex1.m2.1.1.1.1.1.2.3.3.cmml" xref="S3.Ex1.m2.1.1.1.1.1.2.3.3">𝑇</ci></apply></apply><apply id="S3.Ex1.m2.1.1.1.1.1.3.cmml" xref="S3.Ex1.m2.1.1.1.1.1.3"><times id="S3.Ex1.m2.1.1.1.1.1.3.1.cmml" xref="S3.Ex1.m2.1.1.1.1.1.3.1"></times><apply id="S3.Ex1.m2.1.1.1.1.1.3.2.cmml" xref="S3.Ex1.m2.1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S3.Ex1.m2.1.1.1.1.1.3.2.1.cmml" xref="S3.Ex1.m2.1.1.1.1.1.3.2">subscript</csymbol><ci id="S3.Ex1.m2.1.1.1.1.1.3.2.2.cmml" xref="S3.Ex1.m2.1.1.1.1.1.3.2.2">𝐐</ci><ci id="S3.Ex1.m2.1.1.1.1.1.3.2.3.cmml" xref="S3.Ex1.m2.1.1.1.1.1.3.2.3">𝑖</ci></apply><apply id="S3.Ex1.m2.1.1.1.1.1.3.3.cmml" xref="S3.Ex1.m2.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S3.Ex1.m2.1.1.1.1.1.3.3.1.cmml" xref="S3.Ex1.m2.1.1.1.1.1.3.3">superscript</csymbol><apply id="S3.Ex1.m2.1.1.1.1.1.3.3.2.cmml" xref="S3.Ex1.m2.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S3.Ex1.m2.1.1.1.1.1.3.3.2.1.cmml" xref="S3.Ex1.m2.1.1.1.1.1.3.3">subscript</csymbol><ci id="S3.Ex1.m2.1.1.1.1.1.3.3.2.2.cmml" xref="S3.Ex1.m2.1.1.1.1.1.3.3.2.2">𝐒𝐊</ci><ci id="S3.Ex1.m2.1.1.1.1.1.3.3.2.3.cmml" xref="S3.Ex1.m2.1.1.1.1.1.3.3.2.3">𝑖</ci></apply><ci id="S3.Ex1.m2.1.1.1.1.1.3.3.3.cmml" xref="S3.Ex1.m2.1.1.1.1.1.3.3.3">𝑇</ci></apply></apply></apply></apply><ci id="S3.Ex1.m2.1.1.3.cmml" xref="S3.Ex1.m2.1.1.3">𝐹</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.Ex1.m2.1c">\displaystyle\quad\left\|\mathbf{Q}_{i}\mathbf{K}_{i}^{T}-\mathbf{Q}_{i}\mathbf{S}\mathbf{K}_{i}^{T}\right\|_{F}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="S3.Ex2"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><span id="S3.Ex2.2.1.1.1" class="ltx_text ltx_markedasmath">subject to</span></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.Ex2.m2.2" class="ltx_Math" alttext="\displaystyle\quad\text{trace}(\mathbf{S})=\lfloor(1-\lambda)D\rfloor" display="inline"><semantics id="S3.Ex2.m2.2a"><mrow id="S3.Ex2.m2.2.2" xref="S3.Ex2.m2.2.2.cmml"><mrow id="S3.Ex2.m2.2.2.3" xref="S3.Ex2.m2.2.2.3.cmml"><mtext id="S3.Ex2.m2.2.2.3.2" xref="S3.Ex2.m2.2.2.3.2a.cmml">trace</mtext><mo lspace="0em" rspace="0em" id="S3.Ex2.m2.2.2.3.1" xref="S3.Ex2.m2.2.2.3.1.cmml">​</mo><mrow id="S3.Ex2.m2.2.2.3.3.2" xref="S3.Ex2.m2.2.2.3.cmml"><mo stretchy="false" id="S3.Ex2.m2.2.2.3.3.2.1" xref="S3.Ex2.m2.2.2.3.cmml">(</mo><mi id="S3.Ex2.m2.1.1" xref="S3.Ex2.m2.1.1.cmml">𝐒</mi><mo stretchy="false" id="S3.Ex2.m2.2.2.3.3.2.2" xref="S3.Ex2.m2.2.2.3.cmml">)</mo></mrow></mrow><mo id="S3.Ex2.m2.2.2.2" xref="S3.Ex2.m2.2.2.2.cmml">=</mo><mrow id="S3.Ex2.m2.2.2.1.1" xref="S3.Ex2.m2.2.2.1.2.cmml"><mo stretchy="false" id="S3.Ex2.m2.2.2.1.1.2" xref="S3.Ex2.m2.2.2.1.2.1.cmml">⌊</mo><mrow id="S3.Ex2.m2.2.2.1.1.1" xref="S3.Ex2.m2.2.2.1.1.1.cmml"><mrow id="S3.Ex2.m2.2.2.1.1.1.1.1" xref="S3.Ex2.m2.2.2.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.Ex2.m2.2.2.1.1.1.1.1.2" xref="S3.Ex2.m2.2.2.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.Ex2.m2.2.2.1.1.1.1.1.1" xref="S3.Ex2.m2.2.2.1.1.1.1.1.1.cmml"><mn id="S3.Ex2.m2.2.2.1.1.1.1.1.1.2" xref="S3.Ex2.m2.2.2.1.1.1.1.1.1.2.cmml">1</mn><mo id="S3.Ex2.m2.2.2.1.1.1.1.1.1.1" xref="S3.Ex2.m2.2.2.1.1.1.1.1.1.1.cmml">−</mo><mi id="S3.Ex2.m2.2.2.1.1.1.1.1.1.3" xref="S3.Ex2.m2.2.2.1.1.1.1.1.1.3.cmml">λ</mi></mrow><mo stretchy="false" id="S3.Ex2.m2.2.2.1.1.1.1.1.3" xref="S3.Ex2.m2.2.2.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S3.Ex2.m2.2.2.1.1.1.2" xref="S3.Ex2.m2.2.2.1.1.1.2.cmml">​</mo><mi id="S3.Ex2.m2.2.2.1.1.1.3" xref="S3.Ex2.m2.2.2.1.1.1.3.cmml">D</mi></mrow><mo stretchy="false" id="S3.Ex2.m2.2.2.1.1.3" xref="S3.Ex2.m2.2.2.1.2.1.cmml">⌋</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.Ex2.m2.2b"><apply id="S3.Ex2.m2.2.2.cmml" xref="S3.Ex2.m2.2.2"><eq id="S3.Ex2.m2.2.2.2.cmml" xref="S3.Ex2.m2.2.2.2"></eq><apply id="S3.Ex2.m2.2.2.3.cmml" xref="S3.Ex2.m2.2.2.3"><times id="S3.Ex2.m2.2.2.3.1.cmml" xref="S3.Ex2.m2.2.2.3.1"></times><ci id="S3.Ex2.m2.2.2.3.2a.cmml" xref="S3.Ex2.m2.2.2.3.2"><mtext id="S3.Ex2.m2.2.2.3.2.cmml" xref="S3.Ex2.m2.2.2.3.2">trace</mtext></ci><ci id="S3.Ex2.m2.1.1.cmml" xref="S3.Ex2.m2.1.1">𝐒</ci></apply><apply id="S3.Ex2.m2.2.2.1.2.cmml" xref="S3.Ex2.m2.2.2.1.1"><floor id="S3.Ex2.m2.2.2.1.2.1.cmml" xref="S3.Ex2.m2.2.2.1.1.2"></floor><apply id="S3.Ex2.m2.2.2.1.1.1.cmml" xref="S3.Ex2.m2.2.2.1.1.1"><times id="S3.Ex2.m2.2.2.1.1.1.2.cmml" xref="S3.Ex2.m2.2.2.1.1.1.2"></times><apply id="S3.Ex2.m2.2.2.1.1.1.1.1.1.cmml" xref="S3.Ex2.m2.2.2.1.1.1.1.1"><minus id="S3.Ex2.m2.2.2.1.1.1.1.1.1.1.cmml" xref="S3.Ex2.m2.2.2.1.1.1.1.1.1.1"></minus><cn type="integer" id="S3.Ex2.m2.2.2.1.1.1.1.1.1.2.cmml" xref="S3.Ex2.m2.2.2.1.1.1.1.1.1.2">1</cn><ci id="S3.Ex2.m2.2.2.1.1.1.1.1.1.3.cmml" xref="S3.Ex2.m2.2.2.1.1.1.1.1.1.3">𝜆</ci></apply><ci id="S3.Ex2.m2.2.2.1.1.1.3.cmml" xref="S3.Ex2.m2.2.2.1.1.1.3">𝐷</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.Ex2.m2.2c">\displaystyle\quad\text{trace}(\mathbf{S})=\lfloor(1-\lambda)D\rfloor</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="S3.E7"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E7.m1.5" class="ltx_Math" alttext="\displaystyle\quad\mathbf{S}=\text{diag}(s_{1},s_{2},\ldots,s_{D}),\text{ where }s_{j}\in\{0,1\}" display="inline"><semantics id="S3.E7.m1.5a"><mrow id="S3.E7.m1.5.5.2" xref="S3.E7.m1.5.5.3.cmml"><mrow id="S3.E7.m1.4.4.1.1" xref="S3.E7.m1.4.4.1.1.cmml"><mi id="S3.E7.m1.4.4.1.1.5" xref="S3.E7.m1.4.4.1.1.5.cmml">𝐒</mi><mo id="S3.E7.m1.4.4.1.1.4" xref="S3.E7.m1.4.4.1.1.4.cmml">=</mo><mrow id="S3.E7.m1.4.4.1.1.3" xref="S3.E7.m1.4.4.1.1.3.cmml"><mtext id="S3.E7.m1.4.4.1.1.3.5" xref="S3.E7.m1.4.4.1.1.3.5a.cmml">diag</mtext><mo lspace="0em" rspace="0em" id="S3.E7.m1.4.4.1.1.3.4" xref="S3.E7.m1.4.4.1.1.3.4.cmml">​</mo><mrow id="S3.E7.m1.4.4.1.1.3.3.3" xref="S3.E7.m1.4.4.1.1.3.3.4.cmml"><mo stretchy="false" id="S3.E7.m1.4.4.1.1.3.3.3.4" xref="S3.E7.m1.4.4.1.1.3.3.4.cmml">(</mo><msub id="S3.E7.m1.4.4.1.1.1.1.1.1" xref="S3.E7.m1.4.4.1.1.1.1.1.1.cmml"><mi id="S3.E7.m1.4.4.1.1.1.1.1.1.2" xref="S3.E7.m1.4.4.1.1.1.1.1.1.2.cmml">s</mi><mn id="S3.E7.m1.4.4.1.1.1.1.1.1.3" xref="S3.E7.m1.4.4.1.1.1.1.1.1.3.cmml">1</mn></msub><mo id="S3.E7.m1.4.4.1.1.3.3.3.5" xref="S3.E7.m1.4.4.1.1.3.3.4.cmml">,</mo><msub id="S3.E7.m1.4.4.1.1.2.2.2.2" xref="S3.E7.m1.4.4.1.1.2.2.2.2.cmml"><mi id="S3.E7.m1.4.4.1.1.2.2.2.2.2" xref="S3.E7.m1.4.4.1.1.2.2.2.2.2.cmml">s</mi><mn id="S3.E7.m1.4.4.1.1.2.2.2.2.3" xref="S3.E7.m1.4.4.1.1.2.2.2.2.3.cmml">2</mn></msub><mo id="S3.E7.m1.4.4.1.1.3.3.3.6" xref="S3.E7.m1.4.4.1.1.3.3.4.cmml">,</mo><mi mathvariant="normal" id="S3.E7.m1.1.1" xref="S3.E7.m1.1.1.cmml">…</mi><mo id="S3.E7.m1.4.4.1.1.3.3.3.7" xref="S3.E7.m1.4.4.1.1.3.3.4.cmml">,</mo><msub id="S3.E7.m1.4.4.1.1.3.3.3.3" xref="S3.E7.m1.4.4.1.1.3.3.3.3.cmml"><mi id="S3.E7.m1.4.4.1.1.3.3.3.3.2" xref="S3.E7.m1.4.4.1.1.3.3.3.3.2.cmml">s</mi><mi id="S3.E7.m1.4.4.1.1.3.3.3.3.3" xref="S3.E7.m1.4.4.1.1.3.3.3.3.3.cmml">D</mi></msub><mo stretchy="false" id="S3.E7.m1.4.4.1.1.3.3.3.8" xref="S3.E7.m1.4.4.1.1.3.3.4.cmml">)</mo></mrow></mrow></mrow><mo id="S3.E7.m1.5.5.2.3" xref="S3.E7.m1.5.5.3a.cmml">,</mo><mrow id="S3.E7.m1.5.5.2.2" xref="S3.E7.m1.5.5.2.2.cmml"><mrow id="S3.E7.m1.5.5.2.2.2" xref="S3.E7.m1.5.5.2.2.2.cmml"><mtext id="S3.E7.m1.5.5.2.2.2.2" xref="S3.E7.m1.5.5.2.2.2.2a.cmml"> where </mtext><mo lspace="0em" rspace="0em" id="S3.E7.m1.5.5.2.2.2.1" xref="S3.E7.m1.5.5.2.2.2.1.cmml">​</mo><msub id="S3.E7.m1.5.5.2.2.2.3" xref="S3.E7.m1.5.5.2.2.2.3.cmml"><mi id="S3.E7.m1.5.5.2.2.2.3.2" xref="S3.E7.m1.5.5.2.2.2.3.2.cmml">s</mi><mi id="S3.E7.m1.5.5.2.2.2.3.3" xref="S3.E7.m1.5.5.2.2.2.3.3.cmml">j</mi></msub></mrow><mo id="S3.E7.m1.5.5.2.2.1" xref="S3.E7.m1.5.5.2.2.1.cmml">∈</mo><mrow id="S3.E7.m1.5.5.2.2.3.2" xref="S3.E7.m1.5.5.2.2.3.1.cmml"><mo stretchy="false" id="S3.E7.m1.5.5.2.2.3.2.1" xref="S3.E7.m1.5.5.2.2.3.1.cmml">{</mo><mn id="S3.E7.m1.2.2" xref="S3.E7.m1.2.2.cmml">0</mn><mo id="S3.E7.m1.5.5.2.2.3.2.2" xref="S3.E7.m1.5.5.2.2.3.1.cmml">,</mo><mn id="S3.E7.m1.3.3" xref="S3.E7.m1.3.3.cmml">1</mn><mo stretchy="false" id="S3.E7.m1.5.5.2.2.3.2.3" xref="S3.E7.m1.5.5.2.2.3.1.cmml">}</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E7.m1.5b"><apply id="S3.E7.m1.5.5.3.cmml" xref="S3.E7.m1.5.5.2"><csymbol cd="ambiguous" id="S3.E7.m1.5.5.3a.cmml" xref="S3.E7.m1.5.5.2.3">formulae-sequence</csymbol><apply id="S3.E7.m1.4.4.1.1.cmml" xref="S3.E7.m1.4.4.1.1"><eq id="S3.E7.m1.4.4.1.1.4.cmml" xref="S3.E7.m1.4.4.1.1.4"></eq><ci id="S3.E7.m1.4.4.1.1.5.cmml" xref="S3.E7.m1.4.4.1.1.5">𝐒</ci><apply id="S3.E7.m1.4.4.1.1.3.cmml" xref="S3.E7.m1.4.4.1.1.3"><times id="S3.E7.m1.4.4.1.1.3.4.cmml" xref="S3.E7.m1.4.4.1.1.3.4"></times><ci id="S3.E7.m1.4.4.1.1.3.5a.cmml" xref="S3.E7.m1.4.4.1.1.3.5"><mtext id="S3.E7.m1.4.4.1.1.3.5.cmml" xref="S3.E7.m1.4.4.1.1.3.5">diag</mtext></ci><vector id="S3.E7.m1.4.4.1.1.3.3.4.cmml" xref="S3.E7.m1.4.4.1.1.3.3.3"><apply id="S3.E7.m1.4.4.1.1.1.1.1.1.cmml" xref="S3.E7.m1.4.4.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E7.m1.4.4.1.1.1.1.1.1.1.cmml" xref="S3.E7.m1.4.4.1.1.1.1.1.1">subscript</csymbol><ci id="S3.E7.m1.4.4.1.1.1.1.1.1.2.cmml" xref="S3.E7.m1.4.4.1.1.1.1.1.1.2">𝑠</ci><cn type="integer" id="S3.E7.m1.4.4.1.1.1.1.1.1.3.cmml" xref="S3.E7.m1.4.4.1.1.1.1.1.1.3">1</cn></apply><apply id="S3.E7.m1.4.4.1.1.2.2.2.2.cmml" xref="S3.E7.m1.4.4.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S3.E7.m1.4.4.1.1.2.2.2.2.1.cmml" xref="S3.E7.m1.4.4.1.1.2.2.2.2">subscript</csymbol><ci id="S3.E7.m1.4.4.1.1.2.2.2.2.2.cmml" xref="S3.E7.m1.4.4.1.1.2.2.2.2.2">𝑠</ci><cn type="integer" id="S3.E7.m1.4.4.1.1.2.2.2.2.3.cmml" xref="S3.E7.m1.4.4.1.1.2.2.2.2.3">2</cn></apply><ci id="S3.E7.m1.1.1.cmml" xref="S3.E7.m1.1.1">…</ci><apply id="S3.E7.m1.4.4.1.1.3.3.3.3.cmml" xref="S3.E7.m1.4.4.1.1.3.3.3.3"><csymbol cd="ambiguous" id="S3.E7.m1.4.4.1.1.3.3.3.3.1.cmml" xref="S3.E7.m1.4.4.1.1.3.3.3.3">subscript</csymbol><ci id="S3.E7.m1.4.4.1.1.3.3.3.3.2.cmml" xref="S3.E7.m1.4.4.1.1.3.3.3.3.2">𝑠</ci><ci id="S3.E7.m1.4.4.1.1.3.3.3.3.3.cmml" xref="S3.E7.m1.4.4.1.1.3.3.3.3.3">𝐷</ci></apply></vector></apply></apply><apply id="S3.E7.m1.5.5.2.2.cmml" xref="S3.E7.m1.5.5.2.2"><in id="S3.E7.m1.5.5.2.2.1.cmml" xref="S3.E7.m1.5.5.2.2.1"></in><apply id="S3.E7.m1.5.5.2.2.2.cmml" xref="S3.E7.m1.5.5.2.2.2"><times id="S3.E7.m1.5.5.2.2.2.1.cmml" xref="S3.E7.m1.5.5.2.2.2.1"></times><ci id="S3.E7.m1.5.5.2.2.2.2a.cmml" xref="S3.E7.m1.5.5.2.2.2.2"><mtext id="S3.E7.m1.5.5.2.2.2.2.cmml" xref="S3.E7.m1.5.5.2.2.2.2"> where </mtext></ci><apply id="S3.E7.m1.5.5.2.2.2.3.cmml" xref="S3.E7.m1.5.5.2.2.2.3"><csymbol cd="ambiguous" id="S3.E7.m1.5.5.2.2.2.3.1.cmml" xref="S3.E7.m1.5.5.2.2.2.3">subscript</csymbol><ci id="S3.E7.m1.5.5.2.2.2.3.2.cmml" xref="S3.E7.m1.5.5.2.2.2.3.2">𝑠</ci><ci id="S3.E7.m1.5.5.2.2.2.3.3.cmml" xref="S3.E7.m1.5.5.2.2.2.3.3">𝑗</ci></apply></apply><set id="S3.E7.m1.5.5.2.2.3.1.cmml" xref="S3.E7.m1.5.5.2.2.3.2"><cn type="integer" id="S3.E7.m1.2.2.cmml" xref="S3.E7.m1.2.2">0</cn><cn type="integer" id="S3.E7.m1.3.3.cmml" xref="S3.E7.m1.3.3">1</cn></set></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E7.m1.5c">\displaystyle\quad\mathbf{S}=\text{diag}(s_{1},s_{2},\ldots,s_{D}),\text{ where }s_{j}\in\{0,1\}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(7)</span></td>
</tr></tbody>
</table>
</div>
<div id="S3.SS2.p3" class="ltx_para ltx_noindent">
<p id="S3.SS2.p3.1" class="ltx_p">For simplicity, we use greedy algorithm to optimize <math id="S3.SS2.p3.1.m1.1" class="ltx_Math" alttext="\mathbf{S}" display="inline"><semantics id="S3.SS2.p3.1.m1.1a"><mi id="S3.SS2.p3.1.m1.1.1" xref="S3.SS2.p3.1.m1.1.1.cmml">𝐒</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.1.m1.1b"><ci id="S3.SS2.p3.1.m1.1.1.cmml" xref="S3.SS2.p3.1.m1.1.1">𝐒</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.1.m1.1c">\mathbf{S}</annotation></semantics></math>. To achieve the pruning goal, we define a criterion for evaluating the importance of each channel and greedily select the channels with largest scores:</p>
<table id="S3.E8" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E8.m1.6" class="ltx_Math" alttext="\text{Score}_{i}[j]=\big{\|}\textbf{Q}_{i}[:,j]\textbf{K}_{i}[:,j]^{T}\big{\|}_{F}" display="block"><semantics id="S3.E8.m1.6a"><mrow id="S3.E8.m1.6.6" xref="S3.E8.m1.6.6.cmml"><mrow id="S3.E8.m1.6.6.3" xref="S3.E8.m1.6.6.3.cmml"><msub id="S3.E8.m1.6.6.3.2" xref="S3.E8.m1.6.6.3.2.cmml"><mtext id="S3.E8.m1.6.6.3.2.2" xref="S3.E8.m1.6.6.3.2.2a.cmml">Score</mtext><mi id="S3.E8.m1.6.6.3.2.3" xref="S3.E8.m1.6.6.3.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S3.E8.m1.6.6.3.1" xref="S3.E8.m1.6.6.3.1.cmml">​</mo><mrow id="S3.E8.m1.6.6.3.3.2" xref="S3.E8.m1.6.6.3.3.1.cmml"><mo stretchy="false" id="S3.E8.m1.6.6.3.3.2.1" xref="S3.E8.m1.6.6.3.3.1.1.cmml">[</mo><mi id="S3.E8.m1.1.1" xref="S3.E8.m1.1.1.cmml">j</mi><mo stretchy="false" id="S3.E8.m1.6.6.3.3.2.2" xref="S3.E8.m1.6.6.3.3.1.1.cmml">]</mo></mrow></mrow><mo id="S3.E8.m1.6.6.2" xref="S3.E8.m1.6.6.2.cmml">=</mo><msub id="S3.E8.m1.6.6.1" xref="S3.E8.m1.6.6.1.cmml"><mrow id="S3.E8.m1.6.6.1.1.1" xref="S3.E8.m1.6.6.1.1.2.cmml"><mo maxsize="120%" minsize="120%" id="S3.E8.m1.6.6.1.1.1.2" xref="S3.E8.m1.6.6.1.1.2.1.cmml">‖</mo><mrow id="S3.E8.m1.6.6.1.1.1.1" xref="S3.E8.m1.6.6.1.1.1.1.cmml"><msub id="S3.E8.m1.6.6.1.1.1.1.2" xref="S3.E8.m1.6.6.1.1.1.1.2.cmml"><mtext class="ltx_mathvariant_bold" id="S3.E8.m1.6.6.1.1.1.1.2.2" xref="S3.E8.m1.6.6.1.1.1.1.2.2a.cmml">Q</mtext><mi id="S3.E8.m1.6.6.1.1.1.1.2.3" xref="S3.E8.m1.6.6.1.1.1.1.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S3.E8.m1.6.6.1.1.1.1.1" xref="S3.E8.m1.6.6.1.1.1.1.1.cmml">​</mo><mrow id="S3.E8.m1.6.6.1.1.1.1.3.2" xref="S3.E8.m1.6.6.1.1.1.1.3.1.cmml"><mo stretchy="false" id="S3.E8.m1.6.6.1.1.1.1.3.2.1" xref="S3.E8.m1.6.6.1.1.1.1.3.1.cmml">[</mo><mo rspace="0em" id="S3.E8.m1.2.2" xref="S3.E8.m1.2.2.cmml">:</mo><mo id="S3.E8.m1.6.6.1.1.1.1.3.2.2" xref="S3.E8.m1.6.6.1.1.1.1.3.1.cmml">,</mo><mi id="S3.E8.m1.3.3" xref="S3.E8.m1.3.3.cmml">j</mi><mo stretchy="false" id="S3.E8.m1.6.6.1.1.1.1.3.2.3" xref="S3.E8.m1.6.6.1.1.1.1.3.1.cmml">]</mo></mrow><mo lspace="0em" rspace="0em" id="S3.E8.m1.6.6.1.1.1.1.1a" xref="S3.E8.m1.6.6.1.1.1.1.1.cmml">​</mo><msub id="S3.E8.m1.6.6.1.1.1.1.4" xref="S3.E8.m1.6.6.1.1.1.1.4.cmml"><mtext class="ltx_mathvariant_bold" id="S3.E8.m1.6.6.1.1.1.1.4.2" xref="S3.E8.m1.6.6.1.1.1.1.4.2a.cmml">K</mtext><mi id="S3.E8.m1.6.6.1.1.1.1.4.3" xref="S3.E8.m1.6.6.1.1.1.1.4.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S3.E8.m1.6.6.1.1.1.1.1b" xref="S3.E8.m1.6.6.1.1.1.1.1.cmml">​</mo><msup id="S3.E8.m1.6.6.1.1.1.1.5" xref="S3.E8.m1.6.6.1.1.1.1.5.cmml"><mrow id="S3.E8.m1.6.6.1.1.1.1.5.2.2" xref="S3.E8.m1.6.6.1.1.1.1.5.2.1.cmml"><mo stretchy="false" id="S3.E8.m1.6.6.1.1.1.1.5.2.2.1" xref="S3.E8.m1.6.6.1.1.1.1.5.2.1.cmml">[</mo><mo rspace="0em" id="S3.E8.m1.4.4" xref="S3.E8.m1.4.4.cmml">:</mo><mo id="S3.E8.m1.6.6.1.1.1.1.5.2.2.2" xref="S3.E8.m1.6.6.1.1.1.1.5.2.1.cmml">,</mo><mi id="S3.E8.m1.5.5" xref="S3.E8.m1.5.5.cmml">j</mi><mo stretchy="false" id="S3.E8.m1.6.6.1.1.1.1.5.2.2.3" xref="S3.E8.m1.6.6.1.1.1.1.5.2.1.cmml">]</mo></mrow><mi id="S3.E8.m1.6.6.1.1.1.1.5.3" xref="S3.E8.m1.6.6.1.1.1.1.5.3.cmml">T</mi></msup></mrow><mo maxsize="120%" minsize="120%" id="S3.E8.m1.6.6.1.1.1.3" xref="S3.E8.m1.6.6.1.1.2.1.cmml">‖</mo></mrow><mi id="S3.E8.m1.6.6.1.3" xref="S3.E8.m1.6.6.1.3.cmml">F</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.E8.m1.6b"><apply id="S3.E8.m1.6.6.cmml" xref="S3.E8.m1.6.6"><eq id="S3.E8.m1.6.6.2.cmml" xref="S3.E8.m1.6.6.2"></eq><apply id="S3.E8.m1.6.6.3.cmml" xref="S3.E8.m1.6.6.3"><times id="S3.E8.m1.6.6.3.1.cmml" xref="S3.E8.m1.6.6.3.1"></times><apply id="S3.E8.m1.6.6.3.2.cmml" xref="S3.E8.m1.6.6.3.2"><csymbol cd="ambiguous" id="S3.E8.m1.6.6.3.2.1.cmml" xref="S3.E8.m1.6.6.3.2">subscript</csymbol><ci id="S3.E8.m1.6.6.3.2.2a.cmml" xref="S3.E8.m1.6.6.3.2.2"><mtext id="S3.E8.m1.6.6.3.2.2.cmml" xref="S3.E8.m1.6.6.3.2.2">Score</mtext></ci><ci id="S3.E8.m1.6.6.3.2.3.cmml" xref="S3.E8.m1.6.6.3.2.3">𝑖</ci></apply><apply id="S3.E8.m1.6.6.3.3.1.cmml" xref="S3.E8.m1.6.6.3.3.2"><csymbol cd="latexml" id="S3.E8.m1.6.6.3.3.1.1.cmml" xref="S3.E8.m1.6.6.3.3.2.1">delimited-[]</csymbol><ci id="S3.E8.m1.1.1.cmml" xref="S3.E8.m1.1.1">𝑗</ci></apply></apply><apply id="S3.E8.m1.6.6.1.cmml" xref="S3.E8.m1.6.6.1"><csymbol cd="ambiguous" id="S3.E8.m1.6.6.1.2.cmml" xref="S3.E8.m1.6.6.1">subscript</csymbol><apply id="S3.E8.m1.6.6.1.1.2.cmml" xref="S3.E8.m1.6.6.1.1.1"><csymbol cd="latexml" id="S3.E8.m1.6.6.1.1.2.1.cmml" xref="S3.E8.m1.6.6.1.1.1.2">norm</csymbol><apply id="S3.E8.m1.6.6.1.1.1.1.cmml" xref="S3.E8.m1.6.6.1.1.1.1"><times id="S3.E8.m1.6.6.1.1.1.1.1.cmml" xref="S3.E8.m1.6.6.1.1.1.1.1"></times><apply id="S3.E8.m1.6.6.1.1.1.1.2.cmml" xref="S3.E8.m1.6.6.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E8.m1.6.6.1.1.1.1.2.1.cmml" xref="S3.E8.m1.6.6.1.1.1.1.2">subscript</csymbol><ci id="S3.E8.m1.6.6.1.1.1.1.2.2a.cmml" xref="S3.E8.m1.6.6.1.1.1.1.2.2"><mtext class="ltx_mathvariant_bold" id="S3.E8.m1.6.6.1.1.1.1.2.2.cmml" xref="S3.E8.m1.6.6.1.1.1.1.2.2">Q</mtext></ci><ci id="S3.E8.m1.6.6.1.1.1.1.2.3.cmml" xref="S3.E8.m1.6.6.1.1.1.1.2.3">𝑖</ci></apply><interval closure="closed" id="S3.E8.m1.6.6.1.1.1.1.3.1.cmml" xref="S3.E8.m1.6.6.1.1.1.1.3.2"><ci id="S3.E8.m1.2.2.cmml" xref="S3.E8.m1.2.2">:</ci><ci id="S3.E8.m1.3.3.cmml" xref="S3.E8.m1.3.3">𝑗</ci></interval><apply id="S3.E8.m1.6.6.1.1.1.1.4.cmml" xref="S3.E8.m1.6.6.1.1.1.1.4"><csymbol cd="ambiguous" id="S3.E8.m1.6.6.1.1.1.1.4.1.cmml" xref="S3.E8.m1.6.6.1.1.1.1.4">subscript</csymbol><ci id="S3.E8.m1.6.6.1.1.1.1.4.2a.cmml" xref="S3.E8.m1.6.6.1.1.1.1.4.2"><mtext class="ltx_mathvariant_bold" id="S3.E8.m1.6.6.1.1.1.1.4.2.cmml" xref="S3.E8.m1.6.6.1.1.1.1.4.2">K</mtext></ci><ci id="S3.E8.m1.6.6.1.1.1.1.4.3.cmml" xref="S3.E8.m1.6.6.1.1.1.1.4.3">𝑖</ci></apply><apply id="S3.E8.m1.6.6.1.1.1.1.5.cmml" xref="S3.E8.m1.6.6.1.1.1.1.5"><csymbol cd="ambiguous" id="S3.E8.m1.6.6.1.1.1.1.5.1.cmml" xref="S3.E8.m1.6.6.1.1.1.1.5">superscript</csymbol><interval closure="closed" id="S3.E8.m1.6.6.1.1.1.1.5.2.1.cmml" xref="S3.E8.m1.6.6.1.1.1.1.5.2.2"><ci id="S3.E8.m1.4.4.cmml" xref="S3.E8.m1.4.4">:</ci><ci id="S3.E8.m1.5.5.cmml" xref="S3.E8.m1.5.5">𝑗</ci></interval><ci id="S3.E8.m1.6.6.1.1.1.1.5.3.cmml" xref="S3.E8.m1.6.6.1.1.1.1.5.3">𝑇</ci></apply></apply></apply><ci id="S3.E8.m1.6.6.1.3.cmml" xref="S3.E8.m1.6.6.1.3">𝐹</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E8.m1.6c">\text{Score}_{i}[j]=\big{\|}\textbf{Q}_{i}[:,j]\textbf{K}_{i}[:,j]^{T}\big{\|}_{F}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(8)</span></td>
</tr></tbody>
</table>
<table id="S3.E9" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E9.m1.2" class="ltx_Math" alttext="I_{i}=\textbf{Top}_{T}(\text{Score}_{i},T)" display="block"><semantics id="S3.E9.m1.2a"><mrow id="S3.E9.m1.2.2" xref="S3.E9.m1.2.2.cmml"><msub id="S3.E9.m1.2.2.3" xref="S3.E9.m1.2.2.3.cmml"><mi id="S3.E9.m1.2.2.3.2" xref="S3.E9.m1.2.2.3.2.cmml">I</mi><mi id="S3.E9.m1.2.2.3.3" xref="S3.E9.m1.2.2.3.3.cmml">i</mi></msub><mo id="S3.E9.m1.2.2.2" xref="S3.E9.m1.2.2.2.cmml">=</mo><mrow id="S3.E9.m1.2.2.1" xref="S3.E9.m1.2.2.1.cmml"><msub id="S3.E9.m1.2.2.1.3" xref="S3.E9.m1.2.2.1.3.cmml"><mtext class="ltx_mathvariant_bold" id="S3.E9.m1.2.2.1.3.2" xref="S3.E9.m1.2.2.1.3.2a.cmml">Top</mtext><mi id="S3.E9.m1.2.2.1.3.3" xref="S3.E9.m1.2.2.1.3.3.cmml">T</mi></msub><mo lspace="0em" rspace="0em" id="S3.E9.m1.2.2.1.2" xref="S3.E9.m1.2.2.1.2.cmml">​</mo><mrow id="S3.E9.m1.2.2.1.1.1" xref="S3.E9.m1.2.2.1.1.2.cmml"><mo stretchy="false" id="S3.E9.m1.2.2.1.1.1.2" xref="S3.E9.m1.2.2.1.1.2.cmml">(</mo><msub id="S3.E9.m1.2.2.1.1.1.1" xref="S3.E9.m1.2.2.1.1.1.1.cmml"><mtext id="S3.E9.m1.2.2.1.1.1.1.2" xref="S3.E9.m1.2.2.1.1.1.1.2a.cmml">Score</mtext><mi id="S3.E9.m1.2.2.1.1.1.1.3" xref="S3.E9.m1.2.2.1.1.1.1.3.cmml">i</mi></msub><mo id="S3.E9.m1.2.2.1.1.1.3" xref="S3.E9.m1.2.2.1.1.2.cmml">,</mo><mi id="S3.E9.m1.1.1" xref="S3.E9.m1.1.1.cmml">T</mi><mo stretchy="false" id="S3.E9.m1.2.2.1.1.1.4" xref="S3.E9.m1.2.2.1.1.2.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E9.m1.2b"><apply id="S3.E9.m1.2.2.cmml" xref="S3.E9.m1.2.2"><eq id="S3.E9.m1.2.2.2.cmml" xref="S3.E9.m1.2.2.2"></eq><apply id="S3.E9.m1.2.2.3.cmml" xref="S3.E9.m1.2.2.3"><csymbol cd="ambiguous" id="S3.E9.m1.2.2.3.1.cmml" xref="S3.E9.m1.2.2.3">subscript</csymbol><ci id="S3.E9.m1.2.2.3.2.cmml" xref="S3.E9.m1.2.2.3.2">𝐼</ci><ci id="S3.E9.m1.2.2.3.3.cmml" xref="S3.E9.m1.2.2.3.3">𝑖</ci></apply><apply id="S3.E9.m1.2.2.1.cmml" xref="S3.E9.m1.2.2.1"><times id="S3.E9.m1.2.2.1.2.cmml" xref="S3.E9.m1.2.2.1.2"></times><apply id="S3.E9.m1.2.2.1.3.cmml" xref="S3.E9.m1.2.2.1.3"><csymbol cd="ambiguous" id="S3.E9.m1.2.2.1.3.1.cmml" xref="S3.E9.m1.2.2.1.3">subscript</csymbol><ci id="S3.E9.m1.2.2.1.3.2a.cmml" xref="S3.E9.m1.2.2.1.3.2"><mtext class="ltx_mathvariant_bold" id="S3.E9.m1.2.2.1.3.2.cmml" xref="S3.E9.m1.2.2.1.3.2">Top</mtext></ci><ci id="S3.E9.m1.2.2.1.3.3.cmml" xref="S3.E9.m1.2.2.1.3.3">𝑇</ci></apply><interval closure="open" id="S3.E9.m1.2.2.1.1.2.cmml" xref="S3.E9.m1.2.2.1.1.1"><apply id="S3.E9.m1.2.2.1.1.1.1.cmml" xref="S3.E9.m1.2.2.1.1.1.1"><csymbol cd="ambiguous" id="S3.E9.m1.2.2.1.1.1.1.1.cmml" xref="S3.E9.m1.2.2.1.1.1.1">subscript</csymbol><ci id="S3.E9.m1.2.2.1.1.1.1.2a.cmml" xref="S3.E9.m1.2.2.1.1.1.1.2"><mtext id="S3.E9.m1.2.2.1.1.1.1.2.cmml" xref="S3.E9.m1.2.2.1.1.1.1.2">Score</mtext></ci><ci id="S3.E9.m1.2.2.1.1.1.1.3.cmml" xref="S3.E9.m1.2.2.1.1.1.1.3">𝑖</ci></apply><ci id="S3.E9.m1.1.1.cmml" xref="S3.E9.m1.1.1">𝑇</ci></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E9.m1.2c">I_{i}=\textbf{Top}_{T}(\text{Score}_{i},T)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(9)</span></td>
</tr></tbody>
</table>
<p id="S3.SS2.p3.4" class="ltx_p">Here’s a detailed explanation of why it optimizes the selection matrix. The <math id="S3.SS2.p3.2.m1.1" class="ltx_Math" alttext="\text{score}_{i}[j]" display="inline"><semantics id="S3.SS2.p3.2.m1.1a"><mrow id="S3.SS2.p3.2.m1.1.2" xref="S3.SS2.p3.2.m1.1.2.cmml"><msub id="S3.SS2.p3.2.m1.1.2.2" xref="S3.SS2.p3.2.m1.1.2.2.cmml"><mtext id="S3.SS2.p3.2.m1.1.2.2.2" xref="S3.SS2.p3.2.m1.1.2.2.2a.cmml">score</mtext><mi id="S3.SS2.p3.2.m1.1.2.2.3" xref="S3.SS2.p3.2.m1.1.2.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S3.SS2.p3.2.m1.1.2.1" xref="S3.SS2.p3.2.m1.1.2.1.cmml">​</mo><mrow id="S3.SS2.p3.2.m1.1.2.3.2" xref="S3.SS2.p3.2.m1.1.2.3.1.cmml"><mo stretchy="false" id="S3.SS2.p3.2.m1.1.2.3.2.1" xref="S3.SS2.p3.2.m1.1.2.3.1.1.cmml">[</mo><mi id="S3.SS2.p3.2.m1.1.1" xref="S3.SS2.p3.2.m1.1.1.cmml">j</mi><mo stretchy="false" id="S3.SS2.p3.2.m1.1.2.3.2.2" xref="S3.SS2.p3.2.m1.1.2.3.1.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.2.m1.1b"><apply id="S3.SS2.p3.2.m1.1.2.cmml" xref="S3.SS2.p3.2.m1.1.2"><times id="S3.SS2.p3.2.m1.1.2.1.cmml" xref="S3.SS2.p3.2.m1.1.2.1"></times><apply id="S3.SS2.p3.2.m1.1.2.2.cmml" xref="S3.SS2.p3.2.m1.1.2.2"><csymbol cd="ambiguous" id="S3.SS2.p3.2.m1.1.2.2.1.cmml" xref="S3.SS2.p3.2.m1.1.2.2">subscript</csymbol><ci id="S3.SS2.p3.2.m1.1.2.2.2a.cmml" xref="S3.SS2.p3.2.m1.1.2.2.2"><mtext id="S3.SS2.p3.2.m1.1.2.2.2.cmml" xref="S3.SS2.p3.2.m1.1.2.2.2">score</mtext></ci><ci id="S3.SS2.p3.2.m1.1.2.2.3.cmml" xref="S3.SS2.p3.2.m1.1.2.2.3">𝑖</ci></apply><apply id="S3.SS2.p3.2.m1.1.2.3.1.cmml" xref="S3.SS2.p3.2.m1.1.2.3.2"><csymbol cd="latexml" id="S3.SS2.p3.2.m1.1.2.3.1.1.cmml" xref="S3.SS2.p3.2.m1.1.2.3.2.1">delimited-[]</csymbol><ci id="S3.SS2.p3.2.m1.1.1.cmml" xref="S3.SS2.p3.2.m1.1.1">𝑗</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.2.m1.1c">\text{score}_{i}[j]</annotation></semantics></math> measures the magnitude of the interaction between the query and key vectors for channel <math id="S3.SS2.p3.3.m2.1" class="ltx_Math" alttext="j" display="inline"><semantics id="S3.SS2.p3.3.m2.1a"><mi id="S3.SS2.p3.3.m2.1.1" xref="S3.SS2.p3.3.m2.1.1.cmml">j</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.3.m2.1b"><ci id="S3.SS2.p3.3.m2.1.1.cmml" xref="S3.SS2.p3.3.m2.1.1">𝑗</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.3.m2.1c">j</annotation></semantics></math> in each head <math id="S3.SS2.p3.4.m3.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S3.SS2.p3.4.m3.1a"><mi id="S3.SS2.p3.4.m3.1.1" xref="S3.SS2.p3.4.m3.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.4.m3.1b"><ci id="S3.SS2.p3.4.m3.1.1.cmml" xref="S3.SS2.p3.4.m3.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.4.m3.1c">i</annotation></semantics></math>. By selecting channels with the highest interaction magnitudes, we aim to retain the most significant contributions to the attention mechanism. This criterion ensures that the selected channels preserve the primary information flow in the attention computation, thereby minimizing the loss of important information.</p>
</div>
<div id="S3.SS2.p4" class="ltx_para ltx_noindent">
<p id="S3.SS2.p4.2" class="ltx_p"><span id="S3.SS2.p4.2.1" class="ltx_text ltx_font_bold">Observation Window.</span> Based on the observations in SnapKV <cite class="ltx_cite ltx_citemacro_citep">(Li et al.,, <a href="#bib.bib29" title="" class="ltx_ref">2024</a>)</cite> that the last window of input sequence recognizes highly similar attention allocation
pattern with the actual generation. To reduce the computation cost, we only use the last <math id="S3.SS2.p4.1.m1.1" class="ltx_Math" alttext="S_{obs}" display="inline"><semantics id="S3.SS2.p4.1.m1.1a"><msub id="S3.SS2.p4.1.m1.1.1" xref="S3.SS2.p4.1.m1.1.1.cmml"><mi id="S3.SS2.p4.1.m1.1.1.2" xref="S3.SS2.p4.1.m1.1.1.2.cmml">S</mi><mrow id="S3.SS2.p4.1.m1.1.1.3" xref="S3.SS2.p4.1.m1.1.1.3.cmml"><mi id="S3.SS2.p4.1.m1.1.1.3.2" xref="S3.SS2.p4.1.m1.1.1.3.2.cmml">o</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p4.1.m1.1.1.3.1" xref="S3.SS2.p4.1.m1.1.1.3.1.cmml">​</mo><mi id="S3.SS2.p4.1.m1.1.1.3.3" xref="S3.SS2.p4.1.m1.1.1.3.3.cmml">b</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p4.1.m1.1.1.3.1a" xref="S3.SS2.p4.1.m1.1.1.3.1.cmml">​</mo><mi id="S3.SS2.p4.1.m1.1.1.3.4" xref="S3.SS2.p4.1.m1.1.1.3.4.cmml">s</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.1.m1.1b"><apply id="S3.SS2.p4.1.m1.1.1.cmml" xref="S3.SS2.p4.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p4.1.m1.1.1.1.cmml" xref="S3.SS2.p4.1.m1.1.1">subscript</csymbol><ci id="S3.SS2.p4.1.m1.1.1.2.cmml" xref="S3.SS2.p4.1.m1.1.1.2">𝑆</ci><apply id="S3.SS2.p4.1.m1.1.1.3.cmml" xref="S3.SS2.p4.1.m1.1.1.3"><times id="S3.SS2.p4.1.m1.1.1.3.1.cmml" xref="S3.SS2.p4.1.m1.1.1.3.1"></times><ci id="S3.SS2.p4.1.m1.1.1.3.2.cmml" xref="S3.SS2.p4.1.m1.1.1.3.2">𝑜</ci><ci id="S3.SS2.p4.1.m1.1.1.3.3.cmml" xref="S3.SS2.p4.1.m1.1.1.3.3">𝑏</ci><ci id="S3.SS2.p4.1.m1.1.1.3.4.cmml" xref="S3.SS2.p4.1.m1.1.1.3.4">𝑠</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.1.m1.1c">S_{obs}</annotation></semantics></math> window to calculate the score: <math id="S3.SS2.p4.2.m2.1" class="ltx_math_unparsed" alttext="\|\textbf{Q}_{i}[-S_{\text{obs}}:,j]\textbf{K}_{i}[:,j]^{T}\|_{F}" display="inline"><semantics id="S3.SS2.p4.2.m2.1a"><mrow id="S3.SS2.p4.2.m2.1b"><mo rspace="0.167em" id="S3.SS2.p4.2.m2.1.1">∥</mo><msub id="S3.SS2.p4.2.m2.1.2"><mtext class="ltx_mathvariant_bold" id="S3.SS2.p4.2.m2.1.2.2">Q</mtext><mi id="S3.SS2.p4.2.m2.1.2.3">i</mi></msub><mrow id="S3.SS2.p4.2.m2.1.3"><mo stretchy="false" id="S3.SS2.p4.2.m2.1.3.1">[</mo><mo lspace="0em" id="S3.SS2.p4.2.m2.1.3.2">−</mo><msub id="S3.SS2.p4.2.m2.1.3.3"><mi id="S3.SS2.p4.2.m2.1.3.3.2">S</mi><mtext id="S3.SS2.p4.2.m2.1.3.3.3">obs</mtext></msub><mo lspace="0.278em" rspace="0em" id="S3.SS2.p4.2.m2.1.3.4">:</mo><mo id="S3.SS2.p4.2.m2.1.3.5">,</mo><mi id="S3.SS2.p4.2.m2.1.3.6">j</mi><mo stretchy="false" id="S3.SS2.p4.2.m2.1.3.7">]</mo></mrow><msub id="S3.SS2.p4.2.m2.1.4"><mtext class="ltx_mathvariant_bold" id="S3.SS2.p4.2.m2.1.4.2">K</mtext><mi id="S3.SS2.p4.2.m2.1.4.3">i</mi></msub><msup id="S3.SS2.p4.2.m2.1.5"><mrow id="S3.SS2.p4.2.m2.1.5.2"><mo stretchy="false" id="S3.SS2.p4.2.m2.1.5.2.1">[</mo><mo rspace="0em" id="S3.SS2.p4.2.m2.1.5.2.2">:</mo><mo id="S3.SS2.p4.2.m2.1.5.2.3">,</mo><mi id="S3.SS2.p4.2.m2.1.5.2.4">j</mi><mo stretchy="false" id="S3.SS2.p4.2.m2.1.5.2.5">]</mo></mrow><mi id="S3.SS2.p4.2.m2.1.5.3">T</mi></msup><msub id="S3.SS2.p4.2.m2.1.6"><mo lspace="0em" id="S3.SS2.p4.2.m2.1.6.2">∥</mo><mi id="S3.SS2.p4.2.m2.1.6.3">F</mi></msub></mrow><annotation encoding="application/x-tex" id="S3.SS2.p4.2.m2.1c">\|\textbf{Q}_{i}[-S_{\text{obs}}:,j]\textbf{K}_{i}[:,j]^{T}\|_{F}</annotation></semantics></math>.</p>
</div>
<figure id="S3.F4" class="ltx_figure ltx_align_floatright"><img src="/html/2407.21018/assets/x8.png" id="S3.F4.g1" class="ltx_graphics ltx_centering ltx_img_square" width="184" height="182" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S3.F4.2.1.1" class="ltx_text" style="font-size:90%;">Figure 4</span>: </span><span id="S3.F4.3.2" class="ltx_text" style="font-size:90%;">Implementation during decoding.</span></figcaption>
</figure>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Implementations</h3>

<div id="S3.SS3.p1" class="ltx_para ltx_noindent">
<p id="S3.SS3.p1.1" class="ltx_p">Following SnapKV, we focus on the long context input scenario. We opt not to prune the most recent tokens and newly generated keys. Consequently, our key-value (KV) cache will store two distinct categories of keys: one subset consists of pruned keys with a reduced channel size, while the other retains keys at their original size. Additionally, we maintain a binary mask to indicate which channels have been pruned. Note that the memory overhead associated with this mask is negligible. Figure <a href="#S3.F4" title="Figure 4 ‣ 3.2 Query-Driven Pruning ‣ 3 ThinK ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> illustrates one implementation of our method during the decoding stage. The pruned keys are first zero-filled using the mask to restore them to their original size, and then concatenated with the unpruned keys. This implementation can be effectively integrated with optimization techniques such as FlashAttention <cite class="ltx_cite ltx_citemacro_citep">(Dao,, <a href="#bib.bib10" title="" class="ltx_ref">2023</a>)</cite>. The second implementation involves initially pruning the Query using the mask. The pruned Query is then multiplied by the pruned Key, while the unpruned Query is applied to the unpruned Key. Subsequently, the two outputs are concatenated. In theory, this implementation has lower computational costs.</p>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Experiment Results</h2>

<div id="S4.p1" class="ltx_para ltx_noindent">
<p id="S4.p1.1" class="ltx_p">In this section, we conduct extensive experiments to evaluate the effectiveness of <span id="S4.p1.1.1" class="ltx_text ltx_font_bold">ThinK</span> on performance and memory reduction. First, we introduce benchmark datasets, baseline methods, and implementation details in Section <a href="#S4.SS1" title="4.1 Settings ‣ 4 Experiment Results ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.1</span></a>. Next, we report the performance of our KV pruning method with two different KV compression methods in long-context scenarios in Section <a href="#S4.SS2" title="4.2 Results on LongBench ‣ 4 Experiment Results ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.2</span></a>. Finally, we present the results of ablation experiments in Section <a href="#S4.SS4" title="4.4 Ablation Studies ‣ 4 Experiment Results ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.4</span></a>.</p>
</div>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Settings</h3>

<section id="S4.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Benchmark Datasets.</h4>

<div id="S4.SS1.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px1.p1.1" class="ltx_p">We perform evaluations of our method with state-of-the-art KV cache compression methods on two widely used benchmarks: LongBench and Needle-in-a-Haystack.
<span id="S4.SS1.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_bold">LongBench</span> <cite class="ltx_cite ltx_citemacro_citep">(Bai et al.,, <a href="#bib.bib2" title="" class="ltx_ref">2023</a>)</cite> is designed to comprehensively evaluate LLM’s long context understanding capabilities. It includes 17 datasets covering six different tasks: single-document QA, multi-document QA, summarization, few-shot learning, synthetic tasks, and code completion. The average input length of LongBench is 6,711 words, which necessitates reducing the KV cache to lower memory usage for inference.
<span id="S4.SS1.SSS0.Px1.p1.1.2" class="ltx_text ltx_font_bold">Needle-in-a-Haystack</span> <cite class="ltx_cite ltx_citemacro_citep">(Kamradt,, <a href="#bib.bib25" title="" class="ltx_ref">2023</a>)</cite> is a recent popular test challenge that requires models to accurately identify a small piece of information (“needle”) in a long document (“haystack”), where the needle is placed at a random position.
This challenge can test if KV cache compression methods still retain the small piece of critical information.</p>
</div>
</section>
<section id="S4.SS1.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Baseline Approaches.</h4>

<div id="S4.SS1.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px2.p1.1" class="ltx_p">The baseline methods include Heavy Hitter Oracle (H2O) and SnapKV, both of which are the state-of-the-art KV cache compression methods but use different strategies for selecting KV cache.
<span id="S4.SS1.SSS0.Px2.p1.1.1" class="ltx_text ltx_font_bold">H2O</span> <cite class="ltx_cite ltx_citemacro_citep">(<a href="#bib.bib56" title="" class="ltx_ref">Zhang et al., 2024c, </a>)</cite> is designed to reduce memory usage by dynamically balancing recent tokens and Heavy Hitter (H2) tokens, where H2 tokens are a small set of tokens that contribute most of the value when computing attention scores.
<span id="S4.SS1.SSS0.Px2.p1.1.2" class="ltx_text ltx_font_bold">SnapKV</span> <cite class="ltx_cite ltx_citemacro_citep">(Li et al.,, <a href="#bib.bib29" title="" class="ltx_ref">2024</a>)</cite> automatically compresses KV caches by selecting clustered important KV positions for each attention head. This is inspired by the fact that each attention head has a specific pattern, focusing on certain attention features during generation.</p>
</div>
<figure id="S4.T2" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table"><span id="S4.T2.4.1.1" class="ltx_text" style="font-size:129%;">Table 2</span>: </span><span id="S4.T2.5.2" class="ltx_text" style="font-size:129%;">Performance comparison of Key cache pruning on LLaMA3-8B by ThinK on LongBench.</span></figcaption>
<div id="S4.T2.6" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:614.8pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-26.2pt,37.1pt) scale(0.892235798599881,0.892235798599881) ;">
<table id="S4.T2.6.1" class="ltx_tabular ltx_align_middle">
<tr id="S4.T2.6.1.1" class="ltx_tr">
<td id="S4.T2.6.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt" rowspan="2"><span id="S4.T2.6.1.1.1.1" class="ltx_text" style="font-size:70%;">Method</span></td>
<td id="S4.T2.6.1.1.2" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="S4.T2.6.1.1.2.1" class="ltx_text" style="font-size:70%;">Single-Document QA</span></td>
<td id="S4.T2.6.1.1.3" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="S4.T2.6.1.1.3.1" class="ltx_text" style="font-size:70%;">Multi-Document QA</span></td>
<td id="S4.T2.6.1.1.4" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="S4.T2.6.1.1.4.1" class="ltx_text" style="font-size:70%;">Summarization</span></td>
<td id="S4.T2.6.1.1.5" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="S4.T2.6.1.1.5.1" class="ltx_text" style="font-size:70%;">Few-shot Learning</span></td>
<td id="S4.T2.6.1.1.6" class="ltx_td ltx_align_center ltx_border_tt" colspan="2"><span id="S4.T2.6.1.1.6.1" class="ltx_text" style="font-size:70%;">Synthetic</span></td>
<td id="S4.T2.6.1.1.7" class="ltx_td ltx_align_center ltx_border_tt" colspan="2"><span id="S4.T2.6.1.1.7.1" class="ltx_text" style="font-size:70%;">Code</span></td>
<td id="S4.T2.6.1.1.8" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_tt" rowspan="3"><span id="S4.T2.6.1.1.8.1" class="ltx_text" style="font-size:70%;">Avg.</span></td>
</tr>
<tr id="S4.T2.6.1.2" class="ltx_tr">
<td id="S4.T2.6.1.2.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T2.6.1.2.1.1" class="ltx_inline-block ltx_transformed_outer" style="width:24.8pt;height:17.8pt;vertical-align:-7.2pt;"><span class="ltx_transformed_inner" style="width:25.1pt;transform:translate(-0.15pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T2.6.1.2.1.1.1" class="ltx_p"><span id="S4.T2.6.1.2.1.1.1.1" class="ltx_text" style="font-size:70%;">NrtvQA</span></p>
</span></div>
</td>
<td id="S4.T2.6.1.2.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T2.6.1.2.2.1" class="ltx_inline-block ltx_transformed_outer" style="width:21.8pt;height:16.2pt;vertical-align:-6.4pt;"><span class="ltx_transformed_inner" style="width:21.6pt;transform:translate(0.09pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T2.6.1.2.2.1.1" class="ltx_p"><span id="S4.T2.6.1.2.2.1.1.1" class="ltx_text" style="font-size:70%;">Qasper</span></p>
</span></div>
</td>
<td id="S4.T2.6.1.2.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T2.6.1.2.3.1" class="ltx_inline-block ltx_transformed_outer" style="width:20.0pt;height:14.3pt;vertical-align:-4.8pt;"><span class="ltx_transformed_inner" style="width:20.3pt;transform:translate(-0.17pt,0pt) rotate(-30deg) ;">
<p id="S4.T2.6.1.2.3.1.1" class="ltx_p"><span id="S4.T2.6.1.2.3.1.1.1" class="ltx_text" style="font-size:70%;">MF-en</span></p>
</span></div>
</td>
<td id="S4.T2.6.1.2.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T2.6.1.2.4.1" class="ltx_inline-block ltx_transformed_outer" style="width:31.2pt;height:21.6pt;vertical-align:-9.1pt;"><span class="ltx_transformed_inner" style="width:32.5pt;transform:translate(-0.64pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T2.6.1.2.4.1.1" class="ltx_p"><span id="S4.T2.6.1.2.4.1.1.1" class="ltx_text" style="font-size:70%;">HotpotQA</span></p>
</span></div>
</td>
<td id="S4.T2.6.1.2.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T2.6.1.2.5.1" class="ltx_inline-block ltx_transformed_outer" style="width:33.8pt;height:23.1pt;vertical-align:-9.8pt;"><span class="ltx_transformed_inner" style="width:35.4pt;transform:translate(-0.82pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T2.6.1.2.5.1.1" class="ltx_p"><span id="S4.T2.6.1.2.5.1.1.1" class="ltx_text" style="font-size:70%;">2WikiMQA</span></p>
</span></div>
</td>
<td id="S4.T2.6.1.2.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T2.6.1.2.6.1" class="ltx_inline-block ltx_transformed_outer" style="width:25.3pt;height:18.2pt;vertical-align:-7.4pt;"><span class="ltx_transformed_inner" style="width:25.7pt;transform:translate(-0.19pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T2.6.1.2.6.1.1" class="ltx_p"><span id="S4.T2.6.1.2.6.1.1.1" class="ltx_text" style="font-size:70%;">Musique</span></p>
</span></div>
</td>
<td id="S4.T2.6.1.2.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T2.6.1.2.7.1" class="ltx_inline-block ltx_transformed_outer" style="width:32.3pt;height:22.2pt;vertical-align:-9.4pt;"><span class="ltx_transformed_inner" style="width:33.8pt;transform:translate(-0.73pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T2.6.1.2.7.1.1" class="ltx_p"><span id="S4.T2.6.1.2.7.1.1.1" class="ltx_text" style="font-size:70%;">GovReport</span></p>
</span></div>
</td>
<td id="S4.T2.6.1.2.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T2.6.1.2.8.1" class="ltx_inline-block ltx_transformed_outer" style="width:25.1pt;height:18pt;vertical-align:-7.3pt;"><span class="ltx_transformed_inner" style="width:25.5pt;transform:translate(-0.17pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T2.6.1.2.8.1.1" class="ltx_p"><span id="S4.T2.6.1.2.8.1.1.1" class="ltx_text" style="font-size:70%;">QMSum</span></p>
</span></div>
</td>
<td id="S4.T2.6.1.2.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T2.6.1.2.9.1" class="ltx_inline-block ltx_transformed_outer" style="width:31.1pt;height:20.7pt;vertical-align:-7.9pt;"><span class="ltx_transformed_inner" style="width:33.1pt;transform:translate(-1pt,0pt) rotate(-30deg) ;">
<p id="S4.T2.6.1.2.9.1.1" class="ltx_p"><span id="S4.T2.6.1.2.9.1.1.1" class="ltx_text" style="font-size:70%;">MultiNews</span></p>
</span></div>
</td>
<td id="S4.T2.6.1.2.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T2.6.1.2.10.1" class="ltx_inline-block ltx_transformed_outer" style="width:19.7pt;height:14.2pt;vertical-align:-4.7pt;"><span class="ltx_transformed_inner" style="width:20.0pt;transform:translate(-0.15pt,0pt) rotate(-30deg) ;">
<p id="S4.T2.6.1.2.10.1.1" class="ltx_p"><span id="S4.T2.6.1.2.10.1.1.1" class="ltx_text" style="font-size:70%;">TREC</span></p>
</span></div>
</td>
<td id="S4.T2.6.1.2.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T2.6.1.2.11.1" class="ltx_inline-block ltx_transformed_outer" style="width:28.2pt;height:19.8pt;vertical-align:-8.2pt;"><span class="ltx_transformed_inner" style="width:29.0pt;transform:translate(-0.41pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T2.6.1.2.11.1.1" class="ltx_p"><span id="S4.T2.6.1.2.11.1.1.1" class="ltx_text" style="font-size:70%;">TriviaQA</span></p>
</span></div>
</td>
<td id="S4.T2.6.1.2.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T2.6.1.2.12.1" class="ltx_inline-block ltx_transformed_outer" style="width:27.7pt;height:18.8pt;vertical-align:-7.0pt;"><span class="ltx_transformed_inner" style="width:29.2pt;transform:translate(-0.76pt,0pt) rotate(-30deg) ;">
<p id="S4.T2.6.1.2.12.1.1" class="ltx_p"><span id="S4.T2.6.1.2.12.1.1.1" class="ltx_text" style="font-size:70%;">SAMSum</span></p>
</span></div>
</td>
<td id="S4.T2.6.1.2.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T2.6.1.2.13.1" class="ltx_inline-block ltx_transformed_outer" style="width:22.9pt;height:16pt;vertical-align:-5.6pt;"><span class="ltx_transformed_inner" style="width:23.6pt;transform:translate(-0.39pt,0pt) rotate(-30deg) ;">
<p id="S4.T2.6.1.2.13.1.1" class="ltx_p"><span id="S4.T2.6.1.2.13.1.1.1" class="ltx_text" style="font-size:70%;">PCount</span></p>
</span></div>
</td>
<td id="S4.T2.6.1.2.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T2.6.1.2.14.1" class="ltx_inline-block ltx_transformed_outer" style="width:19.7pt;height:14.2pt;vertical-align:-4.7pt;"><span class="ltx_transformed_inner" style="width:20.0pt;transform:translate(-0.15pt,0pt) rotate(-30deg) ;">
<p id="S4.T2.6.1.2.14.1.1" class="ltx_p"><span id="S4.T2.6.1.2.14.1.1.1" class="ltx_text" style="font-size:70%;">PRe</span></p>
</span></div>
</td>
<td id="S4.T2.6.1.2.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T2.6.1.2.15.1" class="ltx_inline-block ltx_transformed_outer" style="width:17.6pt;height:13pt;vertical-align:-4.1pt;"><span class="ltx_transformed_inner" style="width:17.6pt;transform:translate(0.02pt,0pt) rotate(-30deg) ;">
<p id="S4.T2.6.1.2.15.1.1" class="ltx_p"><span id="S4.T2.6.1.2.15.1.1.1" class="ltx_text" style="font-size:70%;">Lcc</span></p>
</span></div>
</td>
<td id="S4.T2.6.1.2.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T2.6.1.2.16.1" class="ltx_inline-block ltx_transformed_outer" style="width:21.3pt;height:15pt;vertical-align:-5.1pt;"><span class="ltx_transformed_inner" style="width:21.9pt;transform:translate(-0.27pt,0pt) rotate(-30deg) ;">
<p id="S4.T2.6.1.2.16.1.1" class="ltx_p"><span id="S4.T2.6.1.2.16.1.1.1" class="ltx_text" style="font-size:70%;">RB-P</span></p>
</span></div>
</td>
</tr>
<tr id="S4.T2.6.1.3" class="ltx_tr">
<td id="S4.T2.6.1.3.1" class="ltx_td ltx_align_center" colspan="18"><span id="S4.T2.6.1.3.1.1" class="ltx_text" style="font-size:70%;">KV-size Full</span></td>
</tr>
<tr id="S4.T2.6.1.4" class="ltx_tr">
<td id="S4.T2.6.1.4.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T2.6.1.4.1.1" class="ltx_text" style="font-size:70%;">ALL KV</span></td>
<td id="S4.T2.6.1.4.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.4.2.1" class="ltx_text" style="font-size:70%;">25.56</span></td>
<td id="S4.T2.6.1.4.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.4.3.1" class="ltx_text" style="font-size:70%;">32.27</span></td>
<td id="S4.T2.6.1.4.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.4.4.1" class="ltx_text" style="font-size:70%;">39.71</span></td>
<td id="S4.T2.6.1.4.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.4.5.1" class="ltx_text" style="font-size:70%;">43.56</span></td>
<td id="S4.T2.6.1.4.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.4.6.1" class="ltx_text" style="font-size:70%;">35.09</span></td>
<td id="S4.T2.6.1.4.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.4.7.1" class="ltx_text" style="font-size:70%;">21.18</span></td>
<td id="S4.T2.6.1.4.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.4.8.1" class="ltx_text" style="font-size:70%;">28.71</span></td>
<td id="S4.T2.6.1.4.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.4.9.1" class="ltx_text" style="font-size:70%;">23.26</span></td>
<td id="S4.T2.6.1.4.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.4.10.1" class="ltx_text" style="font-size:70%;">26.64</span></td>
<td id="S4.T2.6.1.4.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.4.11.1" class="ltx_text" style="font-size:70%;">73.50</span></td>
<td id="S4.T2.6.1.4.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.4.12.1" class="ltx_text" style="font-size:70%;">90.48</span></td>
<td id="S4.T2.6.1.4.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.4.13.1" class="ltx_text" style="font-size:70%;">42.33</span></td>
<td id="S4.T2.6.1.4.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.4.14.1" class="ltx_text" style="font-size:70%;">4.80</span></td>
<td id="S4.T2.6.1.4.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.4.15.1" class="ltx_text" style="font-size:70%;">69.25</span></td>
<td id="S4.T2.6.1.4.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.4.16.1" class="ltx_text" style="font-size:70%;">59.29</span></td>
<td id="S4.T2.6.1.4.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.4.17.1" class="ltx_text" style="font-size:70%;">54.05</span></td>
<td id="S4.T2.6.1.4.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.4.18.1" class="ltx_text" style="font-size:70%;">41.86</span></td>
</tr>
<tr id="S4.T2.6.1.5" class="ltx_tr">
<td id="S4.T2.6.1.5.1" class="ltx_td ltx_align_center ltx_border_t" colspan="18"><span id="S4.T2.6.1.5.1.1" class="ltx_text" style="font-size:70%;">KV-size 128</span></td>
</tr>
<tr id="S4.T2.6.1.6" class="ltx_tr">
<td id="S4.T2.6.1.6.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T2.6.1.6.1.1" class="ltx_text" style="font-size:70%;">H2O</span></td>
<td id="S4.T2.6.1.6.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.6.2.1" class="ltx_text" style="font-size:70%;">22.12</span></td>
<td id="S4.T2.6.1.6.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.6.3.1" class="ltx_text" style="font-size:70%;">13.20</span></td>
<td id="S4.T2.6.1.6.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.6.4.1" class="ltx_text" style="font-size:70%;">31.61</span></td>
<td id="S4.T2.6.1.6.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.6.5.1" class="ltx_text" style="font-size:70%;">37.79</span></td>
<td id="S4.T2.6.1.6.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.6.6.1" class="ltx_text" style="font-size:70%;">32.71</span></td>
<td id="S4.T2.6.1.6.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.6.7.1" class="ltx_text" style="font-size:70%;">18.45</span></td>
<td id="S4.T2.6.1.6.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.6.8.1" class="ltx_text" style="font-size:70%;">20.32</span></td>
<td id="S4.T2.6.1.6.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.6.9.1" class="ltx_text" style="font-size:70%;">22.02</span></td>
<td id="S4.T2.6.1.6.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.6.10.1" class="ltx_text" style="font-size:70%;">21.10</span></td>
<td id="S4.T2.6.1.6.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.6.11.1" class="ltx_text" style="font-size:70%;">38.50</span></td>
<td id="S4.T2.6.1.6.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.6.12.1" class="ltx_text" style="font-size:70%;">87.75</span></td>
<td id="S4.T2.6.1.6.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.6.13.1" class="ltx_text" style="font-size:70%;">39.14</span></td>
<td id="S4.T2.6.1.6.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.6.14.1" class="ltx_text" style="font-size:70%;">5.83</span></td>
<td id="S4.T2.6.1.6.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.6.15.1" class="ltx_text" style="font-size:70%;">69.50</span></td>
<td id="S4.T2.6.1.6.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.6.16.1" class="ltx_text" style="font-size:70%;">55.06</span></td>
<td id="S4.T2.6.1.6.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.6.17.1" class="ltx_text" style="font-size:70%;">50.97</span></td>
<td id="S4.T2.6.1.6.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.6.18.1" class="ltx_text" style="font-size:70%;">35.38</span></td>
</tr>
<tr id="S4.T2.6.1.7" class="ltx_tr">
<td id="S4.T2.6.1.7.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T2.6.1.7.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T2.6.1.7.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T2.6.1.7.1.3" class="ltx_text" style="font-size:70%;">(40)</span>
</td>
<td id="S4.T2.6.1.7.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.7.2.1" class="ltx_text" style="font-size:70%;">22.85</span></td>
<td id="S4.T2.6.1.7.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.7.3.1" class="ltx_text" style="font-size:70%;">14.55</span></td>
<td id="S4.T2.6.1.7.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.7.4.1" class="ltx_text" style="font-size:70%;">29.49</span></td>
<td id="S4.T2.6.1.7.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.7.5.1" class="ltx_text" style="font-size:70%;">38.63</span></td>
<td id="S4.T2.6.1.7.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.7.6.1" class="ltx_text" style="font-size:70%;">30.84</span></td>
<td id="S4.T2.6.1.7.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.7.7.1" class="ltx_text" style="font-size:70%;">18.90</span></td>
<td id="S4.T2.6.1.7.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.7.8.1" class="ltx_text" style="font-size:70%;">20.12</span></td>
<td id="S4.T2.6.1.7.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.7.9.1" class="ltx_text" style="font-size:70%;">21.96</span></td>
<td id="S4.T2.6.1.7.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.7.10.1" class="ltx_text" style="font-size:70%;">20.68</span></td>
<td id="S4.T2.6.1.7.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.7.11.1" class="ltx_text" style="font-size:70%;">38.50</span></td>
<td id="S4.T2.6.1.7.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.7.12.1" class="ltx_text" style="font-size:70%;">86.38</span></td>
<td id="S4.T2.6.1.7.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.7.13.1" class="ltx_text" style="font-size:70%;">38.40</span></td>
<td id="S4.T2.6.1.7.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.7.14.1" class="ltx_text" style="font-size:70%;">5.50</span></td>
<td id="S4.T2.6.1.7.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.7.15.1" class="ltx_text" style="font-size:70%;">69.17</span></td>
<td id="S4.T2.6.1.7.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.7.16.1" class="ltx_text" style="font-size:70%;">57.93</span></td>
<td id="S4.T2.6.1.7.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.7.17.1" class="ltx_text" style="font-size:70%;">56.12</span></td>
<td id="S4.T2.6.1.7.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.7.18.1" class="ltx_text ltx_font_bold" style="font-size:70%;">35.63</span></td>
</tr>
<tr id="S4.T2.6.1.8" class="ltx_tr">
<td id="S4.T2.6.1.8.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T2.6.1.8.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T2.6.1.8.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T2.6.1.8.1.3" class="ltx_text" style="font-size:70%;">(50)</span>
</td>
<td id="S4.T2.6.1.8.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.8.2.1" class="ltx_text" style="font-size:70%;">23.47</span></td>
<td id="S4.T2.6.1.8.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.8.3.1" class="ltx_text" style="font-size:70%;">14.06</span></td>
<td id="S4.T2.6.1.8.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.8.4.1" class="ltx_text" style="font-size:70%;">28.67</span></td>
<td id="S4.T2.6.1.8.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.8.5.1" class="ltx_text" style="font-size:70%;">38.35</span></td>
<td id="S4.T2.6.1.8.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.8.6.1" class="ltx_text" style="font-size:70%;">30.21</span></td>
<td id="S4.T2.6.1.8.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.8.7.1" class="ltx_text" style="font-size:70%;">17.87</span></td>
<td id="S4.T2.6.1.8.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.8.8.1" class="ltx_text" style="font-size:70%;">19.69</span></td>
<td id="S4.T2.6.1.8.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.8.9.1" class="ltx_text" style="font-size:70%;">21.94</span></td>
<td id="S4.T2.6.1.8.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.8.10.1" class="ltx_text" style="font-size:70%;">19.95</span></td>
<td id="S4.T2.6.1.8.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.8.11.1" class="ltx_text" style="font-size:70%;">38.50</span></td>
<td id="S4.T2.6.1.8.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.8.12.1" class="ltx_text" style="font-size:70%;">87.14</span></td>
<td id="S4.T2.6.1.8.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.8.13.1" class="ltx_text" style="font-size:70%;">38.07</span></td>
<td id="S4.T2.6.1.8.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.8.14.1" class="ltx_text" style="font-size:70%;">4.92</span></td>
<td id="S4.T2.6.1.8.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.8.15.1" class="ltx_text" style="font-size:70%;">69.50</span></td>
<td id="S4.T2.6.1.8.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.8.16.1" class="ltx_text" style="font-size:70%;">57.99</span></td>
<td id="S4.T2.6.1.8.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.8.17.1" class="ltx_text" style="font-size:70%;">56.66</span></td>
<td id="S4.T2.6.1.8.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.8.18.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:70%;">35.44</span></td>
</tr>
<tr id="S4.T2.6.1.9" class="ltx_tr">
<td id="S4.T2.6.1.9.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T2.6.1.9.1.1" class="ltx_ERROR undefined">\cdashline</span><span id="S4.T2.6.1.9.1.2" class="ltx_text" style="font-size:70%;">1-18
SnapKV</span>
</td>
<td id="S4.T2.6.1.9.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.9.2.1" class="ltx_text" style="font-size:70%;">21.19</span></td>
<td id="S4.T2.6.1.9.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.9.3.1" class="ltx_text" style="font-size:70%;">13.55</span></td>
<td id="S4.T2.6.1.9.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.9.4.1" class="ltx_text" style="font-size:70%;">32.64</span></td>
<td id="S4.T2.6.1.9.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.9.5.1" class="ltx_text" style="font-size:70%;">38.75</span></td>
<td id="S4.T2.6.1.9.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.9.6.1" class="ltx_text" style="font-size:70%;">29.64</span></td>
<td id="S4.T2.6.1.9.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.9.7.1" class="ltx_text" style="font-size:70%;">18.73</span></td>
<td id="S4.T2.6.1.9.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.9.8.1" class="ltx_text" style="font-size:70%;">18.98</span></td>
<td id="S4.T2.6.1.9.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.9.9.1" class="ltx_text" style="font-size:70%;">21.62</span></td>
<td id="S4.T2.6.1.9.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.9.10.1" class="ltx_text" style="font-size:70%;">20.26</span></td>
<td id="S4.T2.6.1.9.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.9.11.1" class="ltx_text" style="font-size:70%;">45.00</span></td>
<td id="S4.T2.6.1.9.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.9.12.1" class="ltx_text" style="font-size:70%;">88.36</span></td>
<td id="S4.T2.6.1.9.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.9.13.1" class="ltx_text" style="font-size:70%;">37.64</span></td>
<td id="S4.T2.6.1.9.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.9.14.1" class="ltx_text" style="font-size:70%;">5.13</span></td>
<td id="S4.T2.6.1.9.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.9.15.1" class="ltx_text" style="font-size:70%;">68.85</span></td>
<td id="S4.T2.6.1.9.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.9.16.1" class="ltx_text" style="font-size:70%;">55.84</span></td>
<td id="S4.T2.6.1.9.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.9.17.1" class="ltx_text" style="font-size:70%;">51.82</span></td>
<td id="S4.T2.6.1.9.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.9.18.1" class="ltx_text" style="font-size:70%;">35.50</span></td>
</tr>
<tr id="S4.T2.6.1.10" class="ltx_tr">
<td id="S4.T2.6.1.10.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T2.6.1.10.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T2.6.1.10.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T2.6.1.10.1.3" class="ltx_text" style="font-size:70%;">(40)</span>
</td>
<td id="S4.T2.6.1.10.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.10.2.1" class="ltx_text" style="font-size:70%;">22.11</span></td>
<td id="S4.T2.6.1.10.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.10.3.1" class="ltx_text" style="font-size:70%;">14.67</span></td>
<td id="S4.T2.6.1.10.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.10.4.1" class="ltx_text" style="font-size:70%;">32.49</span></td>
<td id="S4.T2.6.1.10.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.10.5.1" class="ltx_text" style="font-size:70%;">36.25</span></td>
<td id="S4.T2.6.1.10.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.10.6.1" class="ltx_text" style="font-size:70%;">28.63</span></td>
<td id="S4.T2.6.1.10.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.10.7.1" class="ltx_text" style="font-size:70%;">18.80</span></td>
<td id="S4.T2.6.1.10.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.10.8.1" class="ltx_text" style="font-size:70%;">18.93</span></td>
<td id="S4.T2.6.1.10.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.10.9.1" class="ltx_text" style="font-size:70%;">21.49</span></td>
<td id="S4.T2.6.1.10.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.10.10.1" class="ltx_text" style="font-size:70%;">20.14</span></td>
<td id="S4.T2.6.1.10.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.10.11.1" class="ltx_text" style="font-size:70%;">44.50</span></td>
<td id="S4.T2.6.1.10.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.10.12.1" class="ltx_text" style="font-size:70%;">88.11</span></td>
<td id="S4.T2.6.1.10.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.10.13.1" class="ltx_text" style="font-size:70%;">38.32</span></td>
<td id="S4.T2.6.1.10.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.10.14.1" class="ltx_text" style="font-size:70%;">5.75</span></td>
<td id="S4.T2.6.1.10.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.10.15.1" class="ltx_text" style="font-size:70%;">69.17</span></td>
<td id="S4.T2.6.1.10.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.10.16.1" class="ltx_text" style="font-size:70%;">58.21</span></td>
<td id="S4.T2.6.1.10.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.10.17.1" class="ltx_text" style="font-size:70%;">55.89</span></td>
<td id="S4.T2.6.1.10.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.10.18.1" class="ltx_text ltx_font_bold" style="font-size:70%;">35.84</span></td>
</tr>
<tr id="S4.T2.6.1.11" class="ltx_tr">
<td id="S4.T2.6.1.11.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T2.6.1.11.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T2.6.1.11.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T2.6.1.11.1.3" class="ltx_text" style="font-size:70%;">(50)</span>
</td>
<td id="S4.T2.6.1.11.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.11.2.1" class="ltx_text" style="font-size:70%;">21.79</span></td>
<td id="S4.T2.6.1.11.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.11.3.1" class="ltx_text" style="font-size:70%;">14.73</span></td>
<td id="S4.T2.6.1.11.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.11.4.1" class="ltx_text" style="font-size:70%;">32.03</span></td>
<td id="S4.T2.6.1.11.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.11.5.1" class="ltx_text" style="font-size:70%;">37.52</span></td>
<td id="S4.T2.6.1.11.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.11.6.1" class="ltx_text" style="font-size:70%;">27.86</span></td>
<td id="S4.T2.6.1.11.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.11.7.1" class="ltx_text" style="font-size:70%;">18.28</span></td>
<td id="S4.T2.6.1.11.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.11.8.1" class="ltx_text" style="font-size:70%;">18.50</span></td>
<td id="S4.T2.6.1.11.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.11.9.1" class="ltx_text" style="font-size:70%;">21.52</span></td>
<td id="S4.T2.6.1.11.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.11.10.1" class="ltx_text" style="font-size:70%;">19.71</span></td>
<td id="S4.T2.6.1.11.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.11.11.1" class="ltx_text" style="font-size:70%;">43.50</span></td>
<td id="S4.T2.6.1.11.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.11.12.1" class="ltx_text" style="font-size:70%;">86.00</span></td>
<td id="S4.T2.6.1.11.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.11.13.1" class="ltx_text" style="font-size:70%;">38.35</span></td>
<td id="S4.T2.6.1.11.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.11.14.1" class="ltx_text" style="font-size:70%;">5.59</span></td>
<td id="S4.T2.6.1.11.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.11.15.1" class="ltx_text" style="font-size:70%;">69.50</span></td>
<td id="S4.T2.6.1.11.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.11.16.1" class="ltx_text" style="font-size:70%;">57.96</span></td>
<td id="S4.T2.6.1.11.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.11.17.1" class="ltx_text" style="font-size:70%;">56.96</span></td>
<td id="S4.T2.6.1.11.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.11.18.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:70%;">35.61</span></td>
</tr>
<tr id="S4.T2.6.1.12" class="ltx_tr">
<td id="S4.T2.6.1.12.1" class="ltx_td ltx_align_center ltx_border_t" colspan="18"><span id="S4.T2.6.1.12.1.1" class="ltx_text" style="font-size:70%;">KV-size 512</span></td>
</tr>
<tr id="S4.T2.6.1.13" class="ltx_tr">
<td id="S4.T2.6.1.13.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T2.6.1.13.1.1" class="ltx_text" style="font-size:70%;">H2O</span></td>
<td id="S4.T2.6.1.13.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.13.2.1" class="ltx_text" style="font-size:70%;">23.52</span></td>
<td id="S4.T2.6.1.13.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.13.3.1" class="ltx_text" style="font-size:70%;">17.93</span></td>
<td id="S4.T2.6.1.13.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.13.4.1" class="ltx_text" style="font-size:70%;">34.68</span></td>
<td id="S4.T2.6.1.13.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.13.5.1" class="ltx_text" style="font-size:70%;">42.11</span></td>
<td id="S4.T2.6.1.13.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.13.6.1" class="ltx_text" style="font-size:70%;">33.52</span></td>
<td id="S4.T2.6.1.13.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.13.7.1" class="ltx_text" style="font-size:70%;">19.92</span></td>
<td id="S4.T2.6.1.13.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.13.8.1" class="ltx_text" style="font-size:70%;">22.11</span></td>
<td id="S4.T2.6.1.13.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.13.9.1" class="ltx_text" style="font-size:70%;">22.56</span></td>
<td id="S4.T2.6.1.13.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.13.10.1" class="ltx_text" style="font-size:70%;">23.82</span></td>
<td id="S4.T2.6.1.13.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.13.11.1" class="ltx_text" style="font-size:70%;">41.00</span></td>
<td id="S4.T2.6.1.13.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.13.12.1" class="ltx_text" style="font-size:70%;">90.46</span></td>
<td id="S4.T2.6.1.13.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.13.13.1" class="ltx_text" style="font-size:70%;">40.20</span></td>
<td id="S4.T2.6.1.13.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.13.14.1" class="ltx_text" style="font-size:70%;">5.87</span></td>
<td id="S4.T2.6.1.13.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.13.15.1" class="ltx_text" style="font-size:70%;">69.50</span></td>
<td id="S4.T2.6.1.13.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.13.16.1" class="ltx_text" style="font-size:70%;">56.71</span></td>
<td id="S4.T2.6.1.13.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.13.17.1" class="ltx_text" style="font-size:70%;">51.69</span></td>
<td id="S4.T2.6.1.13.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.13.18.1" class="ltx_text" style="font-size:70%;">37.23</span></td>
</tr>
<tr id="S4.T2.6.1.14" class="ltx_tr">
<td id="S4.T2.6.1.14.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T2.6.1.14.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T2.6.1.14.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T2.6.1.14.1.3" class="ltx_text" style="font-size:70%;">(40)</span>
</td>
<td id="S4.T2.6.1.14.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.14.2.1" class="ltx_text" style="font-size:70%;">23.76</span></td>
<td id="S4.T2.6.1.14.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.14.3.1" class="ltx_text" style="font-size:70%;">17.80</span></td>
<td id="S4.T2.6.1.14.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.14.4.1" class="ltx_text" style="font-size:70%;">33.80</span></td>
<td id="S4.T2.6.1.14.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.14.5.1" class="ltx_text" style="font-size:70%;">40.39</span></td>
<td id="S4.T2.6.1.14.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.14.6.1" class="ltx_text" style="font-size:70%;">30.70</span></td>
<td id="S4.T2.6.1.14.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.14.7.1" class="ltx_text" style="font-size:70%;">19.09</span></td>
<td id="S4.T2.6.1.14.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.14.8.1" class="ltx_text" style="font-size:70%;">21.82</span></td>
<td id="S4.T2.6.1.14.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.14.9.1" class="ltx_text" style="font-size:70%;">22.51</span></td>
<td id="S4.T2.6.1.14.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.14.10.1" class="ltx_text" style="font-size:70%;">23.78</span></td>
<td id="S4.T2.6.1.14.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.14.11.1" class="ltx_text" style="font-size:70%;">41.00</span></td>
<td id="S4.T2.6.1.14.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.14.12.1" class="ltx_text" style="font-size:70%;">90.16</span></td>
<td id="S4.T2.6.1.14.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.14.13.1" class="ltx_text" style="font-size:70%;">40.67</span></td>
<td id="S4.T2.6.1.14.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.14.14.1" class="ltx_text" style="font-size:70%;">5.15</span></td>
<td id="S4.T2.6.1.14.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.14.15.1" class="ltx_text" style="font-size:70%;">69.25</span></td>
<td id="S4.T2.6.1.14.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.14.16.1" class="ltx_text" style="font-size:70%;">60.77</span></td>
<td id="S4.T2.6.1.14.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.14.17.1" class="ltx_text" style="font-size:70%;">57.58</span></td>
<td id="S4.T2.6.1.14.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.14.18.1" class="ltx_text ltx_font_bold" style="font-size:70%;">37.39</span></td>
</tr>
<tr id="S4.T2.6.1.15" class="ltx_tr">
<td id="S4.T2.6.1.15.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T2.6.1.15.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T2.6.1.15.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T2.6.1.15.1.3" class="ltx_text" style="font-size:70%;">(50)</span>
</td>
<td id="S4.T2.6.1.15.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.15.2.1" class="ltx_text" style="font-size:70%;">24.17</span></td>
<td id="S4.T2.6.1.15.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.15.3.1" class="ltx_text" style="font-size:70%;">16.96</span></td>
<td id="S4.T2.6.1.15.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.15.4.1" class="ltx_text" style="font-size:70%;">35.76</span></td>
<td id="S4.T2.6.1.15.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.15.5.1" class="ltx_text" style="font-size:70%;">39.47</span></td>
<td id="S4.T2.6.1.15.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.15.6.1" class="ltx_text" style="font-size:70%;">30.29</span></td>
<td id="S4.T2.6.1.15.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.15.7.1" class="ltx_text" style="font-size:70%;">18.67</span></td>
<td id="S4.T2.6.1.15.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.15.8.1" class="ltx_text" style="font-size:70%;">21.39</span></td>
<td id="S4.T2.6.1.15.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.15.9.1" class="ltx_text" style="font-size:70%;">22.59</span></td>
<td id="S4.T2.6.1.15.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.15.10.1" class="ltx_text" style="font-size:70%;">23.06</span></td>
<td id="S4.T2.6.1.15.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.15.11.1" class="ltx_text" style="font-size:70%;">41.00</span></td>
<td id="S4.T2.6.1.15.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.15.12.1" class="ltx_text" style="font-size:70%;">89.81</span></td>
<td id="S4.T2.6.1.15.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.15.13.1" class="ltx_text" style="font-size:70%;">40.35</span></td>
<td id="S4.T2.6.1.15.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.15.14.1" class="ltx_text" style="font-size:70%;">5.23</span></td>
<td id="S4.T2.6.1.15.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.15.15.1" class="ltx_text" style="font-size:70%;">69.33</span></td>
<td id="S4.T2.6.1.15.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.15.16.1" class="ltx_text" style="font-size:70%;">60.20</span></td>
<td id="S4.T2.6.1.15.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.15.17.1" class="ltx_text" style="font-size:70%;">58.34</span></td>
<td id="S4.T2.6.1.15.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.15.18.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:70%;">37.29</span></td>
</tr>
<tr id="S4.T2.6.1.16" class="ltx_tr">
<td id="S4.T2.6.1.16.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T2.6.1.16.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T2.6.1.16.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T2.6.1.16.1.3" class="ltx_text" style="font-size:70%;">(60)</span>
</td>
<td id="S4.T2.6.1.16.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.16.2.1" class="ltx_text" style="font-size:70%;">23.40</span></td>
<td id="S4.T2.6.1.16.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.16.3.1" class="ltx_text" style="font-size:70%;">14.83</span></td>
<td id="S4.T2.6.1.16.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.16.4.1" class="ltx_text" style="font-size:70%;">32.62</span></td>
<td id="S4.T2.6.1.16.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.16.5.1" class="ltx_text" style="font-size:70%;">38.47</span></td>
<td id="S4.T2.6.1.16.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.16.6.1" class="ltx_text" style="font-size:70%;">30.97</span></td>
<td id="S4.T2.6.1.16.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.16.7.1" class="ltx_text" style="font-size:70%;">19.81</span></td>
<td id="S4.T2.6.1.16.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.16.8.1" class="ltx_text" style="font-size:70%;">20.80</span></td>
<td id="S4.T2.6.1.16.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.16.9.1" class="ltx_text" style="font-size:70%;">22.04</span></td>
<td id="S4.T2.6.1.16.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.16.10.1" class="ltx_text" style="font-size:70%;">21.60</span></td>
<td id="S4.T2.6.1.16.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.16.11.1" class="ltx_text" style="font-size:70%;">40.00</span></td>
<td id="S4.T2.6.1.16.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.16.12.1" class="ltx_text" style="font-size:70%;">88.79</span></td>
<td id="S4.T2.6.1.16.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.16.13.1" class="ltx_text" style="font-size:70%;">38.90</span></td>
<td id="S4.T2.6.1.16.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.16.14.1" class="ltx_text" style="font-size:70%;">5.36</span></td>
<td id="S4.T2.6.1.16.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.16.15.1" class="ltx_text" style="font-size:70%;">69.50</span></td>
<td id="S4.T2.6.1.16.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.16.16.1" class="ltx_text" style="font-size:70%;">58.28</span></td>
<td id="S4.T2.6.1.16.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.16.17.1" class="ltx_text" style="font-size:70%;">57.65</span></td>
<td id="S4.T2.6.1.16.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.16.18.1" class="ltx_text" style="font-size:70%;">36.44</span></td>
</tr>
<tr id="S4.T2.6.1.17" class="ltx_tr">
<td id="S4.T2.6.1.17.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T2.6.1.17.1.1" class="ltx_ERROR undefined">\cdashline</span><span id="S4.T2.6.1.17.1.2" class="ltx_text" style="font-size:70%;">1-18
SnapKV</span>
</td>
<td id="S4.T2.6.1.17.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.17.2.1" class="ltx_text" style="font-size:70%;">24.84</span></td>
<td id="S4.T2.6.1.17.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.17.3.1" class="ltx_text" style="font-size:70%;">23.96</span></td>
<td id="S4.T2.6.1.17.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.17.4.1" class="ltx_text" style="font-size:70%;">38.77</span></td>
<td id="S4.T2.6.1.17.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.17.5.1" class="ltx_text" style="font-size:70%;">42.75</span></td>
<td id="S4.T2.6.1.17.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.17.6.1" class="ltx_text" style="font-size:70%;">34.55</span></td>
<td id="S4.T2.6.1.17.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.17.7.1" class="ltx_text" style="font-size:70%;">20.87</span></td>
<td id="S4.T2.6.1.17.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.17.8.1" class="ltx_text" style="font-size:70%;">22.26</span></td>
<td id="S4.T2.6.1.17.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.17.9.1" class="ltx_text" style="font-size:70%;">22.61</span></td>
<td id="S4.T2.6.1.17.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.17.10.1" class="ltx_text" style="font-size:70%;">23.97</span></td>
<td id="S4.T2.6.1.17.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.17.11.1" class="ltx_text" style="font-size:70%;">70.00</span></td>
<td id="S4.T2.6.1.17.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.17.12.1" class="ltx_text" style="font-size:70%;">90.52</span></td>
<td id="S4.T2.6.1.17.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.17.13.1" class="ltx_text" style="font-size:70%;">40.29</span></td>
<td id="S4.T2.6.1.17.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.17.14.1" class="ltx_text" style="font-size:70%;">5.81</span></td>
<td id="S4.T2.6.1.17.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.17.15.1" class="ltx_text" style="font-size:70%;">69.50</span></td>
<td id="S4.T2.6.1.17.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.17.16.1" class="ltx_text" style="font-size:70%;">59.04</span></td>
<td id="S4.T2.6.1.17.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.17.17.1" class="ltx_text" style="font-size:70%;">51.81</span></td>
<td id="S4.T2.6.1.17.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.17.18.1" class="ltx_text" style="font-size:70%;">40.10</span></td>
</tr>
<tr id="S4.T2.6.1.18" class="ltx_tr">
<td id="S4.T2.6.1.18.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T2.6.1.18.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T2.6.1.18.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T2.6.1.18.1.3" class="ltx_text" style="font-size:70%;">(40)</span>
</td>
<td id="S4.T2.6.1.18.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.18.2.1" class="ltx_text" style="font-size:70%;">24.58</span></td>
<td id="S4.T2.6.1.18.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.18.3.1" class="ltx_text" style="font-size:70%;">25.44</span></td>
<td id="S4.T2.6.1.18.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.18.4.1" class="ltx_text" style="font-size:70%;">37.03</span></td>
<td id="S4.T2.6.1.18.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.18.5.1" class="ltx_text" style="font-size:70%;">41.87</span></td>
<td id="S4.T2.6.1.18.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.18.6.1" class="ltx_text" style="font-size:70%;">33.45</span></td>
<td id="S4.T2.6.1.18.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.18.7.1" class="ltx_text" style="font-size:70%;">20.58</span></td>
<td id="S4.T2.6.1.18.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.18.8.1" class="ltx_text" style="font-size:70%;">21.77</span></td>
<td id="S4.T2.6.1.18.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.18.9.1" class="ltx_text" style="font-size:70%;">22.42</span></td>
<td id="S4.T2.6.1.18.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.18.10.1" class="ltx_text" style="font-size:70%;">24.16</span></td>
<td id="S4.T2.6.1.18.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.18.11.1" class="ltx_text" style="font-size:70%;">70.00</span></td>
<td id="S4.T2.6.1.18.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.18.12.1" class="ltx_text" style="font-size:70%;">90.39</span></td>
<td id="S4.T2.6.1.18.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.18.13.1" class="ltx_text" style="font-size:70%;">40.29</span></td>
<td id="S4.T2.6.1.18.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.18.14.1" class="ltx_text" style="font-size:70%;">6.06</span></td>
<td id="S4.T2.6.1.18.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.18.15.1" class="ltx_text" style="font-size:70%;">69.50</span></td>
<td id="S4.T2.6.1.18.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.18.16.1" class="ltx_text" style="font-size:70%;">62.05</span></td>
<td id="S4.T2.6.1.18.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.18.17.1" class="ltx_text" style="font-size:70%;">59.23</span></td>
<td id="S4.T2.6.1.18.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.18.18.1" class="ltx_text ltx_font_bold" style="font-size:70%;">40.55</span></td>
</tr>
<tr id="S4.T2.6.1.19" class="ltx_tr">
<td id="S4.T2.6.1.19.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T2.6.1.19.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T2.6.1.19.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T2.6.1.19.1.3" class="ltx_text" style="font-size:70%;">(50)</span>
</td>
<td id="S4.T2.6.1.19.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.19.2.1" class="ltx_text" style="font-size:70%;">24.85</span></td>
<td id="S4.T2.6.1.19.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.19.3.1" class="ltx_text" style="font-size:70%;">25.10</span></td>
<td id="S4.T2.6.1.19.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.19.4.1" class="ltx_text" style="font-size:70%;">37.06</span></td>
<td id="S4.T2.6.1.19.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.19.5.1" class="ltx_text" style="font-size:70%;">41.58</span></td>
<td id="S4.T2.6.1.19.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.19.6.1" class="ltx_text" style="font-size:70%;">32.34</span></td>
<td id="S4.T2.6.1.19.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.19.7.1" class="ltx_text" style="font-size:70%;">20.60</span></td>
<td id="S4.T2.6.1.19.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.19.8.1" class="ltx_text" style="font-size:70%;">21.61</span></td>
<td id="S4.T2.6.1.19.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.19.9.1" class="ltx_text" style="font-size:70%;">22.44</span></td>
<td id="S4.T2.6.1.19.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.19.10.1" class="ltx_text" style="font-size:70%;">23.66</span></td>
<td id="S4.T2.6.1.19.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.19.11.1" class="ltx_text" style="font-size:70%;">69.50</span></td>
<td id="S4.T2.6.1.19.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.19.12.1" class="ltx_text" style="font-size:70%;">90.39</span></td>
<td id="S4.T2.6.1.19.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.19.13.1" class="ltx_text" style="font-size:70%;">39.70</span></td>
<td id="S4.T2.6.1.19.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.19.14.1" class="ltx_text" style="font-size:70%;">5.84</span></td>
<td id="S4.T2.6.1.19.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.19.15.1" class="ltx_text" style="font-size:70%;">69.79</span></td>
<td id="S4.T2.6.1.19.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.19.16.1" class="ltx_text" style="font-size:70%;">61.57</span></td>
<td id="S4.T2.6.1.19.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.19.17.1" class="ltx_text" style="font-size:70%;">59.42</span></td>
<td id="S4.T2.6.1.19.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.19.18.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:70%;">40.34</span></td>
</tr>
<tr id="S4.T2.6.1.20" class="ltx_tr">
<td id="S4.T2.6.1.20.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T2.6.1.20.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T2.6.1.20.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T2.6.1.20.1.3" class="ltx_text" style="font-size:70%;">(60)</span>
</td>
<td id="S4.T2.6.1.20.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.20.2.1" class="ltx_text" style="font-size:70%;">25.98</span></td>
<td id="S4.T2.6.1.20.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.20.3.1" class="ltx_text" style="font-size:70%;">22.77</span></td>
<td id="S4.T2.6.1.20.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.20.4.1" class="ltx_text" style="font-size:70%;">38.37</span></td>
<td id="S4.T2.6.1.20.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.20.5.1" class="ltx_text" style="font-size:70%;">40.44</span></td>
<td id="S4.T2.6.1.20.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.20.6.1" class="ltx_text" style="font-size:70%;">33.19</span></td>
<td id="S4.T2.6.1.20.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.20.7.1" class="ltx_text" style="font-size:70%;">19.90</span></td>
<td id="S4.T2.6.1.20.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.20.8.1" class="ltx_text" style="font-size:70%;">20.84</span></td>
<td id="S4.T2.6.1.20.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.20.9.1" class="ltx_text" style="font-size:70%;">22.21</span></td>
<td id="S4.T2.6.1.20.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.20.10.1" class="ltx_text" style="font-size:70%;">22.55</span></td>
<td id="S4.T2.6.1.20.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.20.11.1" class="ltx_text" style="font-size:70%;">59.00</span></td>
<td id="S4.T2.6.1.20.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.20.12.1" class="ltx_text" style="font-size:70%;">90.32</span></td>
<td id="S4.T2.6.1.20.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.20.13.1" class="ltx_text" style="font-size:70%;">38.12</span></td>
<td id="S4.T2.6.1.20.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.20.14.1" class="ltx_text" style="font-size:70%;">6.39</span></td>
<td id="S4.T2.6.1.20.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.20.15.1" class="ltx_text" style="font-size:70%;">69.50</span></td>
<td id="S4.T2.6.1.20.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.20.16.1" class="ltx_text" style="font-size:70%;">59.14</span></td>
<td id="S4.T2.6.1.20.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.20.17.1" class="ltx_text" style="font-size:70%;">58.40</span></td>
<td id="S4.T2.6.1.20.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.20.18.1" class="ltx_text" style="font-size:70%;">39.20</span></td>
</tr>
<tr id="S4.T2.6.1.21" class="ltx_tr">
<td id="S4.T2.6.1.21.1" class="ltx_td ltx_align_center ltx_border_t" colspan="18"><span id="S4.T2.6.1.21.1.1" class="ltx_text" style="font-size:70%;">KV-size 1024</span></td>
</tr>
<tr id="S4.T2.6.1.22" class="ltx_tr">
<td id="S4.T2.6.1.22.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T2.6.1.22.1.1" class="ltx_text" style="font-size:70%;">H2O</span></td>
<td id="S4.T2.6.1.22.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.22.2.1" class="ltx_text" style="font-size:70%;">25.62</span></td>
<td id="S4.T2.6.1.22.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.22.3.1" class="ltx_text" style="font-size:70%;">22.16</span></td>
<td id="S4.T2.6.1.22.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.22.4.1" class="ltx_text" style="font-size:70%;">36.81</span></td>
<td id="S4.T2.6.1.22.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.22.5.1" class="ltx_text" style="font-size:70%;">41.01</span></td>
<td id="S4.T2.6.1.22.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.22.6.1" class="ltx_text" style="font-size:70%;">33.53</span></td>
<td id="S4.T2.6.1.22.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.22.7.1" class="ltx_text" style="font-size:70%;">19.41</span></td>
<td id="S4.T2.6.1.22.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.22.8.1" class="ltx_text" style="font-size:70%;">23.28</span></td>
<td id="S4.T2.6.1.22.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.22.9.1" class="ltx_text" style="font-size:70%;">22.65</span></td>
<td id="S4.T2.6.1.22.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.22.10.1" class="ltx_text" style="font-size:70%;">25.41</span></td>
<td id="S4.T2.6.1.22.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.22.11.1" class="ltx_text" style="font-size:70%;">46.50</span></td>
<td id="S4.T2.6.1.22.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.22.12.1" class="ltx_text" style="font-size:70%;">90.82</span></td>
<td id="S4.T2.6.1.22.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.22.13.1" class="ltx_text" style="font-size:70%;">41.78</span></td>
<td id="S4.T2.6.1.22.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.22.14.1" class="ltx_text" style="font-size:70%;">5.79</span></td>
<td id="S4.T2.6.1.22.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.22.15.1" class="ltx_text" style="font-size:70%;">69.25</span></td>
<td id="S4.T2.6.1.22.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.22.16.1" class="ltx_text" style="font-size:70%;">59.69</span></td>
<td id="S4.T2.6.1.22.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.22.17.1" class="ltx_text" style="font-size:70%;">55.50</span></td>
<td id="S4.T2.6.1.22.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.22.18.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:70%;">38.70</span></td>
</tr>
<tr id="S4.T2.6.1.23" class="ltx_tr">
<td id="S4.T2.6.1.23.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T2.6.1.23.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T2.6.1.23.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T2.6.1.23.1.3" class="ltx_text" style="font-size:70%;">(40)</span>
</td>
<td id="S4.T2.6.1.23.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.23.2.1" class="ltx_text" style="font-size:70%;">25.52</span></td>
<td id="S4.T2.6.1.23.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.23.3.1" class="ltx_text" style="font-size:70%;">21.93</span></td>
<td id="S4.T2.6.1.23.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.23.4.1" class="ltx_text" style="font-size:70%;">37.17</span></td>
<td id="S4.T2.6.1.23.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.23.5.1" class="ltx_text" style="font-size:70%;">41.56</span></td>
<td id="S4.T2.6.1.23.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.23.6.1" class="ltx_text" style="font-size:70%;">31.22</span></td>
<td id="S4.T2.6.1.23.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.23.7.1" class="ltx_text" style="font-size:70%;">20.17</span></td>
<td id="S4.T2.6.1.23.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.23.8.1" class="ltx_text" style="font-size:70%;">22.89</span></td>
<td id="S4.T2.6.1.23.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.23.9.1" class="ltx_text" style="font-size:70%;">22.95</span></td>
<td id="S4.T2.6.1.23.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.23.10.1" class="ltx_text" style="font-size:70%;">25.21</span></td>
<td id="S4.T2.6.1.23.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.23.11.1" class="ltx_text" style="font-size:70%;">47.00</span></td>
<td id="S4.T2.6.1.23.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.23.12.1" class="ltx_text" style="font-size:70%;">90.74</span></td>
<td id="S4.T2.6.1.23.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.23.13.1" class="ltx_text" style="font-size:70%;">41.34</span></td>
<td id="S4.T2.6.1.23.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.23.14.1" class="ltx_text" style="font-size:70%;">5.57</span></td>
<td id="S4.T2.6.1.23.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.23.15.1" class="ltx_text" style="font-size:70%;">69.50</span></td>
<td id="S4.T2.6.1.23.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.23.16.1" class="ltx_text" style="font-size:70%;">62.58</span></td>
<td id="S4.T2.6.1.23.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.23.17.1" class="ltx_text" style="font-size:70%;">58.67</span></td>
<td id="S4.T2.6.1.23.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.23.18.1" class="ltx_text ltx_font_bold" style="font-size:70%;">39.00</span></td>
</tr>
<tr id="S4.T2.6.1.24" class="ltx_tr">
<td id="S4.T2.6.1.24.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T2.6.1.24.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T2.6.1.24.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T2.6.1.24.1.3" class="ltx_text" style="font-size:70%;">(50)</span>
</td>
<td id="S4.T2.6.1.24.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.24.2.1" class="ltx_text" style="font-size:70%;">25.41</span></td>
<td id="S4.T2.6.1.24.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.24.3.1" class="ltx_text" style="font-size:70%;">22.19</span></td>
<td id="S4.T2.6.1.24.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.24.4.1" class="ltx_text" style="font-size:70%;">37.64</span></td>
<td id="S4.T2.6.1.24.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.24.5.1" class="ltx_text" style="font-size:70%;">40.92</span></td>
<td id="S4.T2.6.1.24.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.24.6.1" class="ltx_text" style="font-size:70%;">31.27</span></td>
<td id="S4.T2.6.1.24.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.24.7.1" class="ltx_text" style="font-size:70%;">18.66</span></td>
<td id="S4.T2.6.1.24.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.24.8.1" class="ltx_text" style="font-size:70%;">22.17</span></td>
<td id="S4.T2.6.1.24.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.24.9.1" class="ltx_text" style="font-size:70%;">22.22</span></td>
<td id="S4.T2.6.1.24.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.24.10.1" class="ltx_text" style="font-size:70%;">24.84</span></td>
<td id="S4.T2.6.1.24.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.24.11.1" class="ltx_text" style="font-size:70%;">46.50</span></td>
<td id="S4.T2.6.1.24.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.24.12.1" class="ltx_text" style="font-size:70%;">90.34</span></td>
<td id="S4.T2.6.1.24.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.24.13.1" class="ltx_text" style="font-size:70%;">40.59</span></td>
<td id="S4.T2.6.1.24.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.24.14.1" class="ltx_text" style="font-size:70%;">5.20</span></td>
<td id="S4.T2.6.1.24.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.24.15.1" class="ltx_text" style="font-size:70%;">69.50</span></td>
<td id="S4.T2.6.1.24.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.24.16.1" class="ltx_text" style="font-size:70%;">61.71</span></td>
<td id="S4.T2.6.1.24.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.24.17.1" class="ltx_text" style="font-size:70%;">57.99</span></td>
<td id="S4.T2.6.1.24.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.24.18.1" class="ltx_text" style="font-size:70%;">38.57</span></td>
</tr>
<tr id="S4.T2.6.1.25" class="ltx_tr">
<td id="S4.T2.6.1.25.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T2.6.1.25.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T2.6.1.25.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T2.6.1.25.1.3" class="ltx_text" style="font-size:70%;">(60)</span>
</td>
<td id="S4.T2.6.1.25.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.25.2.1" class="ltx_text" style="font-size:70%;">24.06</span></td>
<td id="S4.T2.6.1.25.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.25.3.1" class="ltx_text" style="font-size:70%;">17.80</span></td>
<td id="S4.T2.6.1.25.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.25.4.1" class="ltx_text" style="font-size:70%;">37.85</span></td>
<td id="S4.T2.6.1.25.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.25.5.1" class="ltx_text" style="font-size:70%;">38.63</span></td>
<td id="S4.T2.6.1.25.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.25.6.1" class="ltx_text" style="font-size:70%;">29.98</span></td>
<td id="S4.T2.6.1.25.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.25.7.1" class="ltx_text" style="font-size:70%;">19.40</span></td>
<td id="S4.T2.6.1.25.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.25.8.1" class="ltx_text" style="font-size:70%;">21.41</span></td>
<td id="S4.T2.6.1.25.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.25.9.1" class="ltx_text" style="font-size:70%;">22.32</span></td>
<td id="S4.T2.6.1.25.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.25.10.1" class="ltx_text" style="font-size:70%;">23.42</span></td>
<td id="S4.T2.6.1.25.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.25.11.1" class="ltx_text" style="font-size:70%;">44.50</span></td>
<td id="S4.T2.6.1.25.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.25.12.1" class="ltx_text" style="font-size:70%;">90.16</span></td>
<td id="S4.T2.6.1.25.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.25.13.1" class="ltx_text" style="font-size:70%;">39.43</span></td>
<td id="S4.T2.6.1.25.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.25.14.1" class="ltx_text" style="font-size:70%;">5.84</span></td>
<td id="S4.T2.6.1.25.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.25.15.1" class="ltx_text" style="font-size:70%;">69.50</span></td>
<td id="S4.T2.6.1.25.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.25.16.1" class="ltx_text" style="font-size:70%;">58.31</span></td>
<td id="S4.T2.6.1.25.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.25.17.1" class="ltx_text" style="font-size:70%;">58.73</span></td>
<td id="S4.T2.6.1.25.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.25.18.1" class="ltx_text" style="font-size:70%;">37.58</span></td>
</tr>
<tr id="S4.T2.6.1.26" class="ltx_tr">
<td id="S4.T2.6.1.26.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T2.6.1.26.1.1" class="ltx_ERROR undefined">\cdashline</span><span id="S4.T2.6.1.26.1.2" class="ltx_text" style="font-size:70%;">1-18
SnapKV</span>
</td>
<td id="S4.T2.6.1.26.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.26.2.1" class="ltx_text" style="font-size:70%;">24.62</span></td>
<td id="S4.T2.6.1.26.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.26.3.1" class="ltx_text" style="font-size:70%;">25.99</span></td>
<td id="S4.T2.6.1.26.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.26.4.1" class="ltx_text" style="font-size:70%;">37.64</span></td>
<td id="S4.T2.6.1.26.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.26.5.1" class="ltx_text" style="font-size:70%;">43.84</span></td>
<td id="S4.T2.6.1.26.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.26.6.1" class="ltx_text" style="font-size:70%;">34.99</span></td>
<td id="S4.T2.6.1.26.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.26.7.1" class="ltx_text" style="font-size:70%;">20.00</span></td>
<td id="S4.T2.6.1.26.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.26.8.1" class="ltx_text" style="font-size:70%;">24.28</span></td>
<td id="S4.T2.6.1.26.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.26.9.1" class="ltx_text" style="font-size:70%;">22.39</span></td>
<td id="S4.T2.6.1.26.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.26.10.1" class="ltx_text" style="font-size:70%;">25.63</span></td>
<td id="S4.T2.6.1.26.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.26.11.1" class="ltx_text" style="font-size:70%;">72.5</span></td>
<td id="S4.T2.6.1.26.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.26.12.1" class="ltx_text" style="font-size:70%;">90.56</span></td>
<td id="S4.T2.6.1.26.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.26.13.1" class="ltx_text" style="font-size:70%;">40.41</span></td>
<td id="S4.T2.6.1.26.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.26.14.1" class="ltx_text" style="font-size:70%;">5.36</span></td>
<td id="S4.T2.6.1.26.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.26.15.1" class="ltx_text" style="font-size:70%;">69.25</span></td>
<td id="S4.T2.6.1.26.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.26.16.1" class="ltx_text" style="font-size:70%;">60.57</span></td>
<td id="S4.T2.6.1.26.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.26.17.1" class="ltx_text" style="font-size:70%;">56.11</span></td>
<td id="S4.T2.6.1.26.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.26.18.1" class="ltx_text" style="font-size:70%;">40.88</span></td>
</tr>
<tr id="S4.T2.6.1.27" class="ltx_tr">
<td id="S4.T2.6.1.27.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T2.6.1.27.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T2.6.1.27.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T2.6.1.27.1.3" class="ltx_text" style="font-size:70%;">(40)</span>
</td>
<td id="S4.T2.6.1.27.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.27.2.1" class="ltx_text" style="font-size:70%;">24.88</span></td>
<td id="S4.T2.6.1.27.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.27.3.1" class="ltx_text" style="font-size:70%;">27.72</span></td>
<td id="S4.T2.6.1.27.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.27.4.1" class="ltx_text" style="font-size:70%;">38.60</span></td>
<td id="S4.T2.6.1.27.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.27.5.1" class="ltx_text" style="font-size:70%;">43.16</span></td>
<td id="S4.T2.6.1.27.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.27.6.1" class="ltx_text" style="font-size:70%;">32.44</span></td>
<td id="S4.T2.6.1.27.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.27.7.1" class="ltx_text" style="font-size:70%;">20.67</span></td>
<td id="S4.T2.6.1.27.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.27.8.1" class="ltx_text" style="font-size:70%;">24.21</span></td>
<td id="S4.T2.6.1.27.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.27.9.1" class="ltx_text" style="font-size:70%;">22.79</span></td>
<td id="S4.T2.6.1.27.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.27.10.1" class="ltx_text" style="font-size:70%;">25.56</span></td>
<td id="S4.T2.6.1.27.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.27.11.1" class="ltx_text" style="font-size:70%;">71.50</span></td>
<td id="S4.T2.6.1.27.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.27.12.1" class="ltx_text" style="font-size:70%;">90.45</span></td>
<td id="S4.T2.6.1.27.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.27.13.1" class="ltx_text" style="font-size:70%;">40.94</span></td>
<td id="S4.T2.6.1.27.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.27.14.1" class="ltx_text" style="font-size:70%;">5.93</span></td>
<td id="S4.T2.6.1.27.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.27.15.1" class="ltx_text" style="font-size:70%;">69.50</span></td>
<td id="S4.T2.6.1.27.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.27.16.1" class="ltx_text" style="font-size:70%;">62.77</span></td>
<td id="S4.T2.6.1.27.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.27.17.1" class="ltx_text" style="font-size:70%;">59.45</span></td>
<td id="S4.T2.6.1.27.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.27.18.1" class="ltx_text ltx_font_bold" style="font-size:70%;">41.29</span></td>
</tr>
<tr id="S4.T2.6.1.28" class="ltx_tr">
<td id="S4.T2.6.1.28.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T2.6.1.28.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T2.6.1.28.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T2.6.1.28.1.3" class="ltx_text" style="font-size:70%;">(50)</span>
</td>
<td id="S4.T2.6.1.28.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.28.2.1" class="ltx_text" style="font-size:70%;">24.82</span></td>
<td id="S4.T2.6.1.28.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.28.3.1" class="ltx_text" style="font-size:70%;">27.26</span></td>
<td id="S4.T2.6.1.28.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.28.4.1" class="ltx_text" style="font-size:70%;">39.66</span></td>
<td id="S4.T2.6.1.28.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.28.5.1" class="ltx_text" style="font-size:70%;">42.82</span></td>
<td id="S4.T2.6.1.28.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.28.6.1" class="ltx_text" style="font-size:70%;">32.09</span></td>
<td id="S4.T2.6.1.28.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.28.7.1" class="ltx_text" style="font-size:70%;">19.56</span></td>
<td id="S4.T2.6.1.28.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.28.8.1" class="ltx_text" style="font-size:70%;">23.52</span></td>
<td id="S4.T2.6.1.28.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.28.9.1" class="ltx_text" style="font-size:70%;">22.48</span></td>
<td id="S4.T2.6.1.28.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.28.10.1" class="ltx_text" style="font-size:70%;">25.34</span></td>
<td id="S4.T2.6.1.28.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.28.11.1" class="ltx_text" style="font-size:70%;">71.50</span></td>
<td id="S4.T2.6.1.28.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.28.12.1" class="ltx_text" style="font-size:70%;">90.43</span></td>
<td id="S4.T2.6.1.28.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.28.13.1" class="ltx_text" style="font-size:70%;">40.74</span></td>
<td id="S4.T2.6.1.28.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.28.14.1" class="ltx_text" style="font-size:70%;">5.20</span></td>
<td id="S4.T2.6.1.28.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.28.15.1" class="ltx_text" style="font-size:70%;">69.50</span></td>
<td id="S4.T2.6.1.28.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.28.16.1" class="ltx_text" style="font-size:70%;">62.46</span></td>
<td id="S4.T2.6.1.28.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.28.17.1" class="ltx_text" style="font-size:70%;">59.75</span></td>
<td id="S4.T2.6.1.28.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.28.18.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:70%;">41.07</span></td>
</tr>
<tr id="S4.T2.6.1.29" class="ltx_tr">
<td id="S4.T2.6.1.29.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T2.6.1.29.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T2.6.1.29.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T2.6.1.29.1.3" class="ltx_text" style="font-size:70%;">(60)</span>
</td>
<td id="S4.T2.6.1.29.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.29.2.1" class="ltx_text" style="font-size:70%;">24.46</span></td>
<td id="S4.T2.6.1.29.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.29.3.1" class="ltx_text" style="font-size:70%;">27.35</span></td>
<td id="S4.T2.6.1.29.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.29.4.1" class="ltx_text" style="font-size:70%;">38.22</span></td>
<td id="S4.T2.6.1.29.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.29.5.1" class="ltx_text" style="font-size:70%;">41.96</span></td>
<td id="S4.T2.6.1.29.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.29.6.1" class="ltx_text" style="font-size:70%;">31.64</span></td>
<td id="S4.T2.6.1.29.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.29.7.1" class="ltx_text" style="font-size:70%;">20.18</span></td>
<td id="S4.T2.6.1.29.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.29.8.1" class="ltx_text" style="font-size:70%;">21.89</span></td>
<td id="S4.T2.6.1.29.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.29.9.1" class="ltx_text" style="font-size:70%;">22.83</span></td>
<td id="S4.T2.6.1.29.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.29.10.1" class="ltx_text" style="font-size:70%;">23.68</span></td>
<td id="S4.T2.6.1.29.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.29.11.1" class="ltx_text" style="font-size:70%;">70.00</span></td>
<td id="S4.T2.6.1.29.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.29.12.1" class="ltx_text" style="font-size:70%;">90.19</span></td>
<td id="S4.T2.6.1.29.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.29.13.1" class="ltx_text" style="font-size:70%;">38.69</span></td>
<td id="S4.T2.6.1.29.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.29.14.1" class="ltx_text" style="font-size:70%;">6.10</span></td>
<td id="S4.T2.6.1.29.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.29.15.1" class="ltx_text" style="font-size:70%;">69.50</span></td>
<td id="S4.T2.6.1.29.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.29.16.1" class="ltx_text" style="font-size:70%;">58.87</span></td>
<td id="S4.T2.6.1.29.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.29.17.1" class="ltx_text" style="font-size:70%;">59.26</span></td>
<td id="S4.T2.6.1.29.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.29.18.1" class="ltx_text" style="font-size:70%;">40.30</span></td>
</tr>
<tr id="S4.T2.6.1.30" class="ltx_tr">
<td id="S4.T2.6.1.30.1" class="ltx_td ltx_align_center ltx_border_t" colspan="18"><span id="S4.T2.6.1.30.1.1" class="ltx_text" style="font-size:70%;">KV-size 2048</span></td>
</tr>
<tr id="S4.T2.6.1.31" class="ltx_tr">
<td id="S4.T2.6.1.31.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T2.6.1.31.1.1" class="ltx_text" style="font-size:70%;">H2O</span></td>
<td id="S4.T2.6.1.31.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.31.2.1" class="ltx_text" style="font-size:70%;">25.56</span></td>
<td id="S4.T2.6.1.31.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.31.3.1" class="ltx_text" style="font-size:70%;">26.85</span></td>
<td id="S4.T2.6.1.31.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.31.4.1" class="ltx_text" style="font-size:70%;">39.54</span></td>
<td id="S4.T2.6.1.31.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.31.5.1" class="ltx_text" style="font-size:70%;">44.30</span></td>
<td id="S4.T2.6.1.31.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.31.6.1" class="ltx_text" style="font-size:70%;">32.92</span></td>
<td id="S4.T2.6.1.31.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.31.7.1" class="ltx_text" style="font-size:70%;">21.09</span></td>
<td id="S4.T2.6.1.31.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.31.8.1" class="ltx_text" style="font-size:70%;">24.68</span></td>
<td id="S4.T2.6.1.31.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.31.9.1" class="ltx_text" style="font-size:70%;">23.01</span></td>
<td id="S4.T2.6.1.31.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.31.10.1" class="ltx_text" style="font-size:70%;">26.16</span></td>
<td id="S4.T2.6.1.31.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.31.11.1" class="ltx_text" style="font-size:70%;">53.00</span></td>
<td id="S4.T2.6.1.31.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.31.12.1" class="ltx_text" style="font-size:70%;">90.65</span></td>
<td id="S4.T2.6.1.31.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.31.13.1" class="ltx_text" style="font-size:70%;">41.84</span></td>
<td id="S4.T2.6.1.31.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.31.14.1" class="ltx_text" style="font-size:70%;">4.91</span></td>
<td id="S4.T2.6.1.31.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.31.15.1" class="ltx_text" style="font-size:70%;">69.25</span></td>
<td id="S4.T2.6.1.31.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.31.16.1" class="ltx_text" style="font-size:70%;">58.43</span></td>
<td id="S4.T2.6.1.31.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.31.17.1" class="ltx_text" style="font-size:70%;">51.31</span></td>
<td id="S4.T2.6.1.31.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.31.18.1" class="ltx_text" style="font-size:70%;">39.59</span></td>
</tr>
<tr id="S4.T2.6.1.32" class="ltx_tr">
<td id="S4.T2.6.1.32.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T2.6.1.32.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T2.6.1.32.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T2.6.1.32.1.3" class="ltx_text" style="font-size:70%;">(40)</span>
</td>
<td id="S4.T2.6.1.32.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.32.2.1" class="ltx_text" style="font-size:70%;">25.56</span></td>
<td id="S4.T2.6.1.32.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.32.3.1" class="ltx_text" style="font-size:70%;">26.31</span></td>
<td id="S4.T2.6.1.32.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.32.4.1" class="ltx_text" style="font-size:70%;">39.20</span></td>
<td id="S4.T2.6.1.32.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.32.5.1" class="ltx_text" style="font-size:70%;">42.96</span></td>
<td id="S4.T2.6.1.32.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.32.6.1" class="ltx_text" style="font-size:70%;">31.81</span></td>
<td id="S4.T2.6.1.32.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.32.7.1" class="ltx_text" style="font-size:70%;">20.53</span></td>
<td id="S4.T2.6.1.32.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.32.8.1" class="ltx_text" style="font-size:70%;">24.23</span></td>
<td id="S4.T2.6.1.32.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.32.9.1" class="ltx_text" style="font-size:70%;">23.35</span></td>
<td id="S4.T2.6.1.32.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.32.10.1" class="ltx_text" style="font-size:70%;">25.90</span></td>
<td id="S4.T2.6.1.32.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.32.11.1" class="ltx_text" style="font-size:70%;">53.50</span></td>
<td id="S4.T2.6.1.32.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.32.12.1" class="ltx_text" style="font-size:70%;">90.56</span></td>
<td id="S4.T2.6.1.32.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.32.13.1" class="ltx_text" style="font-size:70%;">41.03</span></td>
<td id="S4.T2.6.1.32.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.32.14.1" class="ltx_text" style="font-size:70%;">5.52</span></td>
<td id="S4.T2.6.1.32.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.32.15.1" class="ltx_text" style="font-size:70%;">69.25</span></td>
<td id="S4.T2.6.1.32.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.32.16.1" class="ltx_text" style="font-size:70%;">62.10</span></td>
<td id="S4.T2.6.1.32.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.32.17.1" class="ltx_text" style="font-size:70%;">59.00</span></td>
<td id="S4.T2.6.1.32.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.32.18.1" class="ltx_text ltx_font_bold" style="font-size:70%;">40.05</span></td>
</tr>
<tr id="S4.T2.6.1.33" class="ltx_tr">
<td id="S4.T2.6.1.33.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T2.6.1.33.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T2.6.1.33.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T2.6.1.33.1.3" class="ltx_text" style="font-size:70%;">(50)</span>
</td>
<td id="S4.T2.6.1.33.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.33.2.1" class="ltx_text" style="font-size:70%;">25.01</span></td>
<td id="S4.T2.6.1.33.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.33.3.1" class="ltx_text" style="font-size:70%;">25.37</span></td>
<td id="S4.T2.6.1.33.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.33.4.1" class="ltx_text" style="font-size:70%;">38.82</span></td>
<td id="S4.T2.6.1.33.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.33.5.1" class="ltx_text" style="font-size:70%;">42.32</span></td>
<td id="S4.T2.6.1.33.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.33.6.1" class="ltx_text" style="font-size:70%;">31.27</span></td>
<td id="S4.T2.6.1.33.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.33.7.1" class="ltx_text" style="font-size:70%;">20.50</span></td>
<td id="S4.T2.6.1.33.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.33.8.1" class="ltx_text" style="font-size:70%;">23.78</span></td>
<td id="S4.T2.6.1.33.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.33.9.1" class="ltx_text" style="font-size:70%;">23.21</span></td>
<td id="S4.T2.6.1.33.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.33.10.1" class="ltx_text" style="font-size:70%;">26.03</span></td>
<td id="S4.T2.6.1.33.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.33.11.1" class="ltx_text" style="font-size:70%;">53.00</span></td>
<td id="S4.T2.6.1.33.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.33.12.1" class="ltx_text" style="font-size:70%;">90.37</span></td>
<td id="S4.T2.6.1.33.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.33.13.1" class="ltx_text" style="font-size:70%;">40.86</span></td>
<td id="S4.T2.6.1.33.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.33.14.1" class="ltx_text" style="font-size:70%;">5.13</span></td>
<td id="S4.T2.6.1.33.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.33.15.1" class="ltx_text" style="font-size:70%;">69.50</span></td>
<td id="S4.T2.6.1.33.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.33.16.1" class="ltx_text" style="font-size:70%;">61.91</span></td>
<td id="S4.T2.6.1.33.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.33.17.1" class="ltx_text" style="font-size:70%;">58.95</span></td>
<td id="S4.T2.6.1.33.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.33.18.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:70%;">39.75</span></td>
</tr>
<tr id="S4.T2.6.1.34" class="ltx_tr">
<td id="S4.T2.6.1.34.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T2.6.1.34.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T2.6.1.34.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T2.6.1.34.1.3" class="ltx_text" style="font-size:70%;">(60)</span>
</td>
<td id="S4.T2.6.1.34.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.34.2.1" class="ltx_text" style="font-size:70%;">24.37</span></td>
<td id="S4.T2.6.1.34.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.34.3.1" class="ltx_text" style="font-size:70%;">22.14</span></td>
<td id="S4.T2.6.1.34.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.34.4.1" class="ltx_text" style="font-size:70%;">37.77</span></td>
<td id="S4.T2.6.1.34.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.34.5.1" class="ltx_text" style="font-size:70%;">40.13</span></td>
<td id="S4.T2.6.1.34.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.34.6.1" class="ltx_text" style="font-size:70%;">29.50</span></td>
<td id="S4.T2.6.1.34.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.34.7.1" class="ltx_text" style="font-size:70%;">20.26</span></td>
<td id="S4.T2.6.1.34.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.34.8.1" class="ltx_text" style="font-size:70%;">22.09</span></td>
<td id="S4.T2.6.1.34.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.34.9.1" class="ltx_text" style="font-size:70%;">22.76</span></td>
<td id="S4.T2.6.1.34.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.34.10.1" class="ltx_text" style="font-size:70%;">24.78</span></td>
<td id="S4.T2.6.1.34.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.34.11.1" class="ltx_text" style="font-size:70%;">49.50</span></td>
<td id="S4.T2.6.1.34.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.34.12.1" class="ltx_text" style="font-size:70%;">90.16</span></td>
<td id="S4.T2.6.1.34.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.34.13.1" class="ltx_text" style="font-size:70%;">39.69</span></td>
<td id="S4.T2.6.1.34.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.34.14.1" class="ltx_text" style="font-size:70%;">5.56</span></td>
<td id="S4.T2.6.1.34.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.34.15.1" class="ltx_text" style="font-size:70%;">69.50</span></td>
<td id="S4.T2.6.1.34.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.34.16.1" class="ltx_text" style="font-size:70%;">59.24</span></td>
<td id="S4.T2.6.1.34.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.34.17.1" class="ltx_text" style="font-size:70%;">58.78</span></td>
<td id="S4.T2.6.1.34.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.34.18.1" class="ltx_text" style="font-size:70%;">38.51</span></td>
</tr>
<tr id="S4.T2.6.1.35" class="ltx_tr">
<td id="S4.T2.6.1.35.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T2.6.1.35.1.1" class="ltx_ERROR undefined">\cdashline</span><span id="S4.T2.6.1.35.1.2" class="ltx_text" style="font-size:70%;">1-18
SnapKV</span>
</td>
<td id="S4.T2.6.1.35.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.35.2.1" class="ltx_text" style="font-size:70%;">25.86</span></td>
<td id="S4.T2.6.1.35.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.35.3.1" class="ltx_text" style="font-size:70%;">29.55</span></td>
<td id="S4.T2.6.1.35.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.35.4.1" class="ltx_text" style="font-size:70%;">41.10</span></td>
<td id="S4.T2.6.1.35.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.35.5.1" class="ltx_text" style="font-size:70%;">44.99</span></td>
<td id="S4.T2.6.1.35.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.35.6.1" class="ltx_text" style="font-size:70%;">35.80</span></td>
<td id="S4.T2.6.1.35.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.35.7.1" class="ltx_text" style="font-size:70%;">21.81</span></td>
<td id="S4.T2.6.1.35.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.35.8.1" class="ltx_text" style="font-size:70%;">25.98</span></td>
<td id="S4.T2.6.1.35.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.35.9.1" class="ltx_text" style="font-size:70%;">23.40</span></td>
<td id="S4.T2.6.1.35.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.35.10.1" class="ltx_text" style="font-size:70%;">26.46</span></td>
<td id="S4.T2.6.1.35.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.35.11.1" class="ltx_text" style="font-size:70%;">73.50</span></td>
<td id="S4.T2.6.1.35.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.35.12.1" class="ltx_text" style="font-size:70%;">90.56</span></td>
<td id="S4.T2.6.1.35.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.35.13.1" class="ltx_text" style="font-size:70%;">41.66</span></td>
<td id="S4.T2.6.1.35.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.35.14.1" class="ltx_text" style="font-size:70%;">5.17</span></td>
<td id="S4.T2.6.1.35.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.35.15.1" class="ltx_text" style="font-size:70%;">69.25</span></td>
<td id="S4.T2.6.1.35.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.35.16.1" class="ltx_text" style="font-size:70%;">58.67</span></td>
<td id="S4.T2.6.1.35.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.35.17.1" class="ltx_text" style="font-size:70%;">51.52</span></td>
<td id="S4.T2.6.1.35.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.35.18.1" class="ltx_text" style="font-size:70%;">41.58</span></td>
</tr>
<tr id="S4.T2.6.1.36" class="ltx_tr">
<td id="S4.T2.6.1.36.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T2.6.1.36.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T2.6.1.36.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T2.6.1.36.1.3" class="ltx_text" style="font-size:70%;">(40)</span>
</td>
<td id="S4.T2.6.1.36.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.36.2.1" class="ltx_text" style="font-size:70%;">25.41</span></td>
<td id="S4.T2.6.1.36.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.36.3.1" class="ltx_text" style="font-size:70%;">29.79</span></td>
<td id="S4.T2.6.1.36.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.36.4.1" class="ltx_text" style="font-size:70%;">39.21</span></td>
<td id="S4.T2.6.1.36.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.36.5.1" class="ltx_text" style="font-size:70%;">43.35</span></td>
<td id="S4.T2.6.1.36.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.36.6.1" class="ltx_text" style="font-size:70%;">33.96</span></td>
<td id="S4.T2.6.1.36.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.36.7.1" class="ltx_text" style="font-size:70%;">21.49</span></td>
<td id="S4.T2.6.1.36.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.36.8.1" class="ltx_text" style="font-size:70%;">25.78</span></td>
<td id="S4.T2.6.1.36.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.36.9.1" class="ltx_text" style="font-size:70%;">23.11</span></td>
<td id="S4.T2.6.1.36.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.36.10.1" class="ltx_text" style="font-size:70%;">26.23</span></td>
<td id="S4.T2.6.1.36.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.36.11.1" class="ltx_text" style="font-size:70%;">73.00</span></td>
<td id="S4.T2.6.1.36.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.36.12.1" class="ltx_text" style="font-size:70%;">90.56</span></td>
<td id="S4.T2.6.1.36.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.36.13.1" class="ltx_text" style="font-size:70%;">41.79</span></td>
<td id="S4.T2.6.1.36.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.36.14.1" class="ltx_text" style="font-size:70%;">5.81</span></td>
<td id="S4.T2.6.1.36.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.36.15.1" class="ltx_text" style="font-size:70%;">69.50</span></td>
<td id="S4.T2.6.1.36.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.36.16.1" class="ltx_text" style="font-size:70%;">62.45</span></td>
<td id="S4.T2.6.1.36.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.36.17.1" class="ltx_text" style="font-size:70%;">59.19</span></td>
<td id="S4.T2.6.1.36.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.36.18.1" class="ltx_text ltx_font_bold" style="font-size:70%;">41.91</span></td>
</tr>
<tr id="S4.T2.6.1.37" class="ltx_tr">
<td id="S4.T2.6.1.37.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T2.6.1.37.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T2.6.1.37.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T2.6.1.37.1.3" class="ltx_text" style="font-size:70%;">(50)</span>
</td>
<td id="S4.T2.6.1.37.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.37.2.1" class="ltx_text" style="font-size:70%;">25.00</span></td>
<td id="S4.T2.6.1.37.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.37.3.1" class="ltx_text" style="font-size:70%;">30.25</span></td>
<td id="S4.T2.6.1.37.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.37.4.1" class="ltx_text" style="font-size:70%;">39.27</span></td>
<td id="S4.T2.6.1.37.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.37.5.1" class="ltx_text" style="font-size:70%;">43.23</span></td>
<td id="S4.T2.6.1.37.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.37.6.1" class="ltx_text" style="font-size:70%;">32.93</span></td>
<td id="S4.T2.6.1.37.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.37.7.1" class="ltx_text" style="font-size:70%;">21.24</span></td>
<td id="S4.T2.6.1.37.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.37.8.1" class="ltx_text" style="font-size:70%;">25.16</span></td>
<td id="S4.T2.6.1.37.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.37.9.1" class="ltx_text" style="font-size:70%;">23.01</span></td>
<td id="S4.T2.6.1.37.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.37.10.1" class="ltx_text" style="font-size:70%;">26.5</span></td>
<td id="S4.T2.6.1.37.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.37.11.1" class="ltx_text" style="font-size:70%;">73.00</span></td>
<td id="S4.T2.6.1.37.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.37.12.1" class="ltx_text" style="font-size:70%;">90.37</span></td>
<td id="S4.T2.6.1.37.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.37.13.1" class="ltx_text" style="font-size:70%;">41.26</span></td>
<td id="S4.T2.6.1.37.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.37.14.1" class="ltx_text" style="font-size:70%;">5.45</span></td>
<td id="S4.T2.6.1.37.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.37.15.1" class="ltx_text" style="font-size:70%;">69.50</span></td>
<td id="S4.T2.6.1.37.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.37.16.1" class="ltx_text" style="font-size:70%;">62.3</span></td>
<td id="S4.T2.6.1.37.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T2.6.1.37.17.1" class="ltx_text" style="font-size:70%;">59.84</span></td>
<td id="S4.T2.6.1.37.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T2.6.1.37.18.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:70%;">41.77</span></td>
</tr>
<tr id="S4.T2.6.1.38" class="ltx_tr">
<td id="S4.T2.6.1.38.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">
<span id="S4.T2.6.1.38.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T2.6.1.38.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T2.6.1.38.1.3" class="ltx_text" style="font-size:70%;">(60)</span>
</td>
<td id="S4.T2.6.1.38.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T2.6.1.38.2.1" class="ltx_text" style="font-size:70%;">24.89</span></td>
<td id="S4.T2.6.1.38.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T2.6.1.38.3.1" class="ltx_text" style="font-size:70%;">28.88</span></td>
<td id="S4.T2.6.1.38.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T2.6.1.38.4.1" class="ltx_text" style="font-size:70%;">40.44</span></td>
<td id="S4.T2.6.1.38.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T2.6.1.38.5.1" class="ltx_text" style="font-size:70%;">41.30</span></td>
<td id="S4.T2.6.1.38.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T2.6.1.38.6.1" class="ltx_text" style="font-size:70%;">29.99</span></td>
<td id="S4.T2.6.1.38.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T2.6.1.38.7.1" class="ltx_text" style="font-size:70%;">21.34</span></td>
<td id="S4.T2.6.1.38.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T2.6.1.38.8.1" class="ltx_text" style="font-size:70%;">23.48</span></td>
<td id="S4.T2.6.1.38.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T2.6.1.38.9.1" class="ltx_text" style="font-size:70%;">22.9</span></td>
<td id="S4.T2.6.1.38.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T2.6.1.38.10.1" class="ltx_text" style="font-size:70%;">24.99</span></td>
<td id="S4.T2.6.1.38.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T2.6.1.38.11.1" class="ltx_text" style="font-size:70%;">72.50</span></td>
<td id="S4.T2.6.1.38.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T2.6.1.38.12.1" class="ltx_text" style="font-size:70%;">90.36</span></td>
<td id="S4.T2.6.1.38.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T2.6.1.38.13.1" class="ltx_text" style="font-size:70%;">38.5</span></td>
<td id="S4.T2.6.1.38.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T2.6.1.38.14.1" class="ltx_text" style="font-size:70%;">5.71</span></td>
<td id="S4.T2.6.1.38.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T2.6.1.38.15.1" class="ltx_text" style="font-size:70%;">69.50</span></td>
<td id="S4.T2.6.1.38.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T2.6.1.38.16.1" class="ltx_text" style="font-size:70%;">59.77</span></td>
<td id="S4.T2.6.1.38.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T2.6.1.38.17.1" class="ltx_text" style="font-size:70%;">59.50</span></td>
<td id="S4.T2.6.1.38.18" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_bb"><span id="S4.T2.6.1.38.18.1" class="ltx_text" style="font-size:70%;">40.88</span></td>
</tr>
</table>
</span></div>
</figure>
</section>
<section id="S4.SS1.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Implementation Details.</h4>

<div id="S4.SS1.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px3.p1.1" class="ltx_p">In this paper, we use LLaMA3-8B-Instruct <cite class="ltx_cite ltx_citemacro_citep">(Meta,, <a href="#bib.bib34" title="" class="ltx_ref">2024</a>)</cite> and Mistral-7B-Instruct-v0.2 <cite class="ltx_cite ltx_citemacro_citep">(Jiang et al.,, <a href="#bib.bib24" title="" class="ltx_ref">2023</a>)</cite> as the backbone LLMs, both accessible via HuggingFace <cite class="ltx_cite ltx_citemacro_citep">(Wolf et al.,, <a href="#bib.bib46" title="" class="ltx_ref">2020</a>)</cite>.
Our <span id="S4.SS1.SSS0.Px3.p1.1.1" class="ltx_text ltx_font_bold">ThinK</span> aims to prune channels of the KV, which is agnostic to KV cache compression methods.
If there is no other statement, we prune the key cache by default. All the experiments are conducted using one Nvidia A100.
To fairly compare KV cache compression methods and KV cache compression integrated with <span id="S4.SS1.SSS0.Px3.p1.1.2" class="ltx_text ltx_font_bold">ThinK</span>, we use the same hyperparameters for both.
For example, when comparing SnapKV and SnapKV integrated with <span id="S4.SS1.SSS0.Px3.p1.1.3" class="ltx_text ltx_font_bold">ThinK</span>, we set the maximum pooling kernel size to 7 and the observation window size to 32, using the same KV-size for both.</p>
</div>
<figure id="S4.T3" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table"><span id="S4.T3.4.1.1" class="ltx_text" style="font-size:129%;">Table 3</span>: </span><span id="S4.T3.5.2" class="ltx_text" style="font-size:129%;">Performance comparison of Key cache pruning on Mistral-7B by ThinK on LongBench.</span></figcaption>
<div id="S4.T3.6" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:646.9pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-26.2pt,39.1pt) scale(0.892235798599881,0.892235798599881) ;">
<table id="S4.T3.6.1" class="ltx_tabular ltx_align_middle">
<tr id="S4.T3.6.1.1" class="ltx_tr">
<td id="S4.T3.6.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt" rowspan="2"><span id="S4.T3.6.1.1.1.1" class="ltx_text" style="font-size:70%;">Method</span></td>
<td id="S4.T3.6.1.1.2" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="S4.T3.6.1.1.2.1" class="ltx_text" style="font-size:70%;">Single-Document QA</span></td>
<td id="S4.T3.6.1.1.3" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="S4.T3.6.1.1.3.1" class="ltx_text" style="font-size:70%;">Multi-Document QA</span></td>
<td id="S4.T3.6.1.1.4" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="S4.T3.6.1.1.4.1" class="ltx_text" style="font-size:70%;">Summarization</span></td>
<td id="S4.T3.6.1.1.5" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="S4.T3.6.1.1.5.1" class="ltx_text" style="font-size:70%;">Few-shot Learning</span></td>
<td id="S4.T3.6.1.1.6" class="ltx_td ltx_align_center ltx_border_tt" colspan="2"><span id="S4.T3.6.1.1.6.1" class="ltx_text" style="font-size:70%;">Synthetic</span></td>
<td id="S4.T3.6.1.1.7" class="ltx_td ltx_align_center ltx_border_tt" colspan="2"><span id="S4.T3.6.1.1.7.1" class="ltx_text" style="font-size:70%;">Code</span></td>
<td id="S4.T3.6.1.1.8" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_tt" rowspan="3"><span id="S4.T3.6.1.1.8.1" class="ltx_text" style="font-size:70%;">Avg.</span></td>
</tr>
<tr id="S4.T3.6.1.2" class="ltx_tr">
<td id="S4.T3.6.1.2.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T3.6.1.2.1.1" class="ltx_inline-block ltx_transformed_outer" style="width:24.8pt;height:17.8pt;vertical-align:-7.2pt;"><span class="ltx_transformed_inner" style="width:25.1pt;transform:translate(-0.15pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T3.6.1.2.1.1.1" class="ltx_p"><span id="S4.T3.6.1.2.1.1.1.1" class="ltx_text" style="font-size:70%;">NrtvQA</span></p>
</span></div>
</td>
<td id="S4.T3.6.1.2.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T3.6.1.2.2.1" class="ltx_inline-block ltx_transformed_outer" style="width:21.8pt;height:16.2pt;vertical-align:-6.4pt;"><span class="ltx_transformed_inner" style="width:21.6pt;transform:translate(0.09pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T3.6.1.2.2.1.1" class="ltx_p"><span id="S4.T3.6.1.2.2.1.1.1" class="ltx_text" style="font-size:70%;">Qasper</span></p>
</span></div>
</td>
<td id="S4.T3.6.1.2.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T3.6.1.2.3.1" class="ltx_inline-block ltx_transformed_outer" style="width:20.0pt;height:14.3pt;vertical-align:-4.8pt;"><span class="ltx_transformed_inner" style="width:20.3pt;transform:translate(-0.17pt,0pt) rotate(-30deg) ;">
<p id="S4.T3.6.1.2.3.1.1" class="ltx_p"><span id="S4.T3.6.1.2.3.1.1.1" class="ltx_text" style="font-size:70%;">MF-en</span></p>
</span></div>
</td>
<td id="S4.T3.6.1.2.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T3.6.1.2.4.1" class="ltx_inline-block ltx_transformed_outer" style="width:31.2pt;height:21.6pt;vertical-align:-9.1pt;"><span class="ltx_transformed_inner" style="width:32.5pt;transform:translate(-0.64pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T3.6.1.2.4.1.1" class="ltx_p"><span id="S4.T3.6.1.2.4.1.1.1" class="ltx_text" style="font-size:70%;">HotpotQA</span></p>
</span></div>
</td>
<td id="S4.T3.6.1.2.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T3.6.1.2.5.1" class="ltx_inline-block ltx_transformed_outer" style="width:33.8pt;height:23.1pt;vertical-align:-9.8pt;"><span class="ltx_transformed_inner" style="width:35.4pt;transform:translate(-0.82pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T3.6.1.2.5.1.1" class="ltx_p"><span id="S4.T3.6.1.2.5.1.1.1" class="ltx_text" style="font-size:70%;">2WikiMQA</span></p>
</span></div>
</td>
<td id="S4.T3.6.1.2.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T3.6.1.2.6.1" class="ltx_inline-block ltx_transformed_outer" style="width:25.3pt;height:18.2pt;vertical-align:-7.4pt;"><span class="ltx_transformed_inner" style="width:25.7pt;transform:translate(-0.19pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T3.6.1.2.6.1.1" class="ltx_p"><span id="S4.T3.6.1.2.6.1.1.1" class="ltx_text" style="font-size:70%;">Musique</span></p>
</span></div>
</td>
<td id="S4.T3.6.1.2.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T3.6.1.2.7.1" class="ltx_inline-block ltx_transformed_outer" style="width:32.3pt;height:22.2pt;vertical-align:-9.4pt;"><span class="ltx_transformed_inner" style="width:33.8pt;transform:translate(-0.73pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T3.6.1.2.7.1.1" class="ltx_p"><span id="S4.T3.6.1.2.7.1.1.1" class="ltx_text" style="font-size:70%;">GovReport</span></p>
</span></div>
</td>
<td id="S4.T3.6.1.2.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T3.6.1.2.8.1" class="ltx_inline-block ltx_transformed_outer" style="width:25.1pt;height:18pt;vertical-align:-7.3pt;"><span class="ltx_transformed_inner" style="width:25.5pt;transform:translate(-0.17pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T3.6.1.2.8.1.1" class="ltx_p"><span id="S4.T3.6.1.2.8.1.1.1" class="ltx_text" style="font-size:70%;">QMSum</span></p>
</span></div>
</td>
<td id="S4.T3.6.1.2.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T3.6.1.2.9.1" class="ltx_inline-block ltx_transformed_outer" style="width:31.1pt;height:20.7pt;vertical-align:-7.9pt;"><span class="ltx_transformed_inner" style="width:33.1pt;transform:translate(-1pt,0pt) rotate(-30deg) ;">
<p id="S4.T3.6.1.2.9.1.1" class="ltx_p"><span id="S4.T3.6.1.2.9.1.1.1" class="ltx_text" style="font-size:70%;">MultiNews</span></p>
</span></div>
</td>
<td id="S4.T3.6.1.2.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T3.6.1.2.10.1" class="ltx_inline-block ltx_transformed_outer" style="width:19.7pt;height:14.2pt;vertical-align:-4.7pt;"><span class="ltx_transformed_inner" style="width:20.0pt;transform:translate(-0.15pt,0pt) rotate(-30deg) ;">
<p id="S4.T3.6.1.2.10.1.1" class="ltx_p"><span id="S4.T3.6.1.2.10.1.1.1" class="ltx_text" style="font-size:70%;">TREC</span></p>
</span></div>
</td>
<td id="S4.T3.6.1.2.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T3.6.1.2.11.1" class="ltx_inline-block ltx_transformed_outer" style="width:28.2pt;height:19.8pt;vertical-align:-8.2pt;"><span class="ltx_transformed_inner" style="width:29.0pt;transform:translate(-0.41pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T3.6.1.2.11.1.1" class="ltx_p"><span id="S4.T3.6.1.2.11.1.1.1" class="ltx_text" style="font-size:70%;">TriviaQA</span></p>
</span></div>
</td>
<td id="S4.T3.6.1.2.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T3.6.1.2.12.1" class="ltx_inline-block ltx_transformed_outer" style="width:27.7pt;height:18.8pt;vertical-align:-7.0pt;"><span class="ltx_transformed_inner" style="width:29.2pt;transform:translate(-0.76pt,0pt) rotate(-30deg) ;">
<p id="S4.T3.6.1.2.12.1.1" class="ltx_p"><span id="S4.T3.6.1.2.12.1.1.1" class="ltx_text" style="font-size:70%;">SAMSum</span></p>
</span></div>
</td>
<td id="S4.T3.6.1.2.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T3.6.1.2.13.1" class="ltx_inline-block ltx_transformed_outer" style="width:22.9pt;height:16pt;vertical-align:-5.6pt;"><span class="ltx_transformed_inner" style="width:23.6pt;transform:translate(-0.39pt,0pt) rotate(-30deg) ;">
<p id="S4.T3.6.1.2.13.1.1" class="ltx_p"><span id="S4.T3.6.1.2.13.1.1.1" class="ltx_text" style="font-size:70%;">PCount</span></p>
</span></div>
</td>
<td id="S4.T3.6.1.2.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T3.6.1.2.14.1" class="ltx_inline-block ltx_transformed_outer" style="width:19.7pt;height:14.2pt;vertical-align:-4.7pt;"><span class="ltx_transformed_inner" style="width:20.0pt;transform:translate(-0.15pt,0pt) rotate(-30deg) ;">
<p id="S4.T3.6.1.2.14.1.1" class="ltx_p"><span id="S4.T3.6.1.2.14.1.1.1" class="ltx_text" style="font-size:70%;">PRe</span></p>
</span></div>
</td>
<td id="S4.T3.6.1.2.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T3.6.1.2.15.1" class="ltx_inline-block ltx_transformed_outer" style="width:17.6pt;height:13pt;vertical-align:-4.1pt;"><span class="ltx_transformed_inner" style="width:17.6pt;transform:translate(0.02pt,0pt) rotate(-30deg) ;">
<p id="S4.T3.6.1.2.15.1.1" class="ltx_p"><span id="S4.T3.6.1.2.15.1.1.1" class="ltx_text" style="font-size:70%;">Lcc</span></p>
</span></div>
</td>
<td id="S4.T3.6.1.2.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T3.6.1.2.16.1" class="ltx_inline-block ltx_transformed_outer" style="width:21.3pt;height:15pt;vertical-align:-5.1pt;"><span class="ltx_transformed_inner" style="width:21.9pt;transform:translate(-0.27pt,0pt) rotate(-30deg) ;">
<p id="S4.T3.6.1.2.16.1.1" class="ltx_p"><span id="S4.T3.6.1.2.16.1.1.1" class="ltx_text" style="font-size:70%;">RB-P</span></p>
</span></div>
</td>
</tr>
<tr id="S4.T3.6.1.3" class="ltx_tr">
<td id="S4.T3.6.1.3.1" class="ltx_td ltx_align_center" colspan="18"><span id="S4.T3.6.1.3.1.1" class="ltx_text" style="font-size:70%;">KV-size Full</span></td>
</tr>
<tr id="S4.T3.6.1.4" class="ltx_tr">
<td id="S4.T3.6.1.4.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T3.6.1.4.1.1" class="ltx_text" style="font-size:70%;">ALL KV</span></td>
<td id="S4.T3.6.1.4.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.4.2.1" class="ltx_text" style="font-size:70%;">26.63</span></td>
<td id="S4.T3.6.1.4.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.4.3.1" class="ltx_text" style="font-size:70%;">32.99</span></td>
<td id="S4.T3.6.1.4.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.4.4.1" class="ltx_text" style="font-size:70%;">49.34</span></td>
<td id="S4.T3.6.1.4.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.4.5.1" class="ltx_text" style="font-size:70%;">42.77</span></td>
<td id="S4.T3.6.1.4.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.4.6.1" class="ltx_text" style="font-size:70%;">27.35</span></td>
<td id="S4.T3.6.1.4.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.4.7.1" class="ltx_text" style="font-size:70%;">18.77</span></td>
<td id="S4.T3.6.1.4.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.4.8.1" class="ltx_text" style="font-size:70%;">32.87</span></td>
<td id="S4.T3.6.1.4.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.4.9.1" class="ltx_text" style="font-size:70%;">24.24</span></td>
<td id="S4.T3.6.1.4.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.4.10.1" class="ltx_text" style="font-size:70%;">27.10</span></td>
<td id="S4.T3.6.1.4.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.4.11.1" class="ltx_text" style="font-size:70%;">71.00</span></td>
<td id="S4.T3.6.1.4.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.4.12.1" class="ltx_text" style="font-size:70%;">86.23</span></td>
<td id="S4.T3.6.1.4.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.4.13.1" class="ltx_text" style="font-size:70%;">42.96</span></td>
<td id="S4.T3.6.1.4.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.4.14.1" class="ltx_text" style="font-size:70%;">2.75</span></td>
<td id="S4.T3.6.1.4.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.4.15.1" class="ltx_text" style="font-size:70%;">86.98</span></td>
<td id="S4.T3.6.1.4.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.4.16.1" class="ltx_text" style="font-size:70%;">56.93</span></td>
<td id="S4.T3.6.1.4.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.4.17.1" class="ltx_text" style="font-size:70%;">54.49</span></td>
<td id="S4.T3.6.1.4.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.4.18.1" class="ltx_text" style="font-size:70%;">42.71</span></td>
</tr>
<tr id="S4.T3.6.1.5" class="ltx_tr">
<td id="S4.T3.6.1.5.1" class="ltx_td ltx_align_center ltx_border_t" colspan="18"><span id="S4.T3.6.1.5.1.1" class="ltx_text" style="font-size:70%;">KV-size 128</span></td>
</tr>
<tr id="S4.T3.6.1.6" class="ltx_tr">
<td id="S4.T3.6.1.6.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T3.6.1.6.1.1" class="ltx_text" style="font-size:70%;">H2O</span></td>
<td id="S4.T3.6.1.6.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.6.2.1" class="ltx_text" style="font-size:70%;">21.21</span></td>
<td id="S4.T3.6.1.6.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.6.3.1" class="ltx_text" style="font-size:70%;">21.81</span></td>
<td id="S4.T3.6.1.6.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.6.4.1" class="ltx_text" style="font-size:70%;">38.87</span></td>
<td id="S4.T3.6.1.6.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.6.5.1" class="ltx_text" style="font-size:70%;">30.42</span></td>
<td id="S4.T3.6.1.6.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.6.6.1" class="ltx_text" style="font-size:70%;">20.36</span></td>
<td id="S4.T3.6.1.6.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.6.7.1" class="ltx_text" style="font-size:70%;">12.30</span></td>
<td id="S4.T3.6.1.6.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.6.8.1" class="ltx_text" style="font-size:70%;">20.58</span></td>
<td id="S4.T3.6.1.6.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.6.9.1" class="ltx_text" style="font-size:70%;">22.61</span></td>
<td id="S4.T3.6.1.6.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.6.10.1" class="ltx_text" style="font-size:70%;">22.10</span></td>
<td id="S4.T3.6.1.6.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.6.11.1" class="ltx_text" style="font-size:70%;">39.00</span></td>
<td id="S4.T3.6.1.6.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.6.12.1" class="ltx_text" style="font-size:70%;">82.37</span></td>
<td id="S4.T3.6.1.6.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.6.13.1" class="ltx_text" style="font-size:70%;">40.44</span></td>
<td id="S4.T3.6.1.6.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.6.14.1" class="ltx_text" style="font-size:70%;">2.90</span></td>
<td id="S4.T3.6.1.6.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.6.15.1" class="ltx_text" style="font-size:70%;">79.56</span></td>
<td id="S4.T3.6.1.6.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.6.16.1" class="ltx_text" style="font-size:70%;">51.22</span></td>
<td id="S4.T3.6.1.6.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.6.17.1" class="ltx_text" style="font-size:70%;">48.38</span></td>
<td id="S4.T3.6.1.6.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.6.18.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:70%;">34.63</span></td>
</tr>
<tr id="S4.T3.6.1.7" class="ltx_tr">
<td id="S4.T3.6.1.7.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T3.6.1.7.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T3.6.1.7.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T3.6.1.7.1.3" class="ltx_text" style="font-size:70%;">(40)</span>
</td>
<td id="S4.T3.6.1.7.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.7.2.1" class="ltx_text" style="font-size:70%;">21.17</span></td>
<td id="S4.T3.6.1.7.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.7.3.1" class="ltx_text" style="font-size:70%;">21.90</span></td>
<td id="S4.T3.6.1.7.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.7.4.1" class="ltx_text" style="font-size:70%;">39.29</span></td>
<td id="S4.T3.6.1.7.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.7.5.1" class="ltx_text" style="font-size:70%;">29.92</span></td>
<td id="S4.T3.6.1.7.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.7.6.1" class="ltx_text" style="font-size:70%;">20.99</span></td>
<td id="S4.T3.6.1.7.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.7.7.1" class="ltx_text" style="font-size:70%;">12.30</span></td>
<td id="S4.T3.6.1.7.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.7.8.1" class="ltx_text" style="font-size:70%;">20.84</span></td>
<td id="S4.T3.6.1.7.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.7.9.1" class="ltx_text" style="font-size:70%;">22.91</span></td>
<td id="S4.T3.6.1.7.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.7.10.1" class="ltx_text" style="font-size:70%;">21.92</span></td>
<td id="S4.T3.6.1.7.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.7.11.1" class="ltx_text" style="font-size:70%;">39.00</span></td>
<td id="S4.T3.6.1.7.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.7.12.1" class="ltx_text" style="font-size:70%;">82.70</span></td>
<td id="S4.T3.6.1.7.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.7.13.1" class="ltx_text" style="font-size:70%;">40.35</span></td>
<td id="S4.T3.6.1.7.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.7.14.1" class="ltx_text" style="font-size:70%;">2.97</span></td>
<td id="S4.T3.6.1.7.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.7.15.1" class="ltx_text" style="font-size:70%;">79.21</span></td>
<td id="S4.T3.6.1.7.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.7.16.1" class="ltx_text" style="font-size:70%;">51.19</span></td>
<td id="S4.T3.6.1.7.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.7.17.1" class="ltx_text" style="font-size:70%;">48.32</span></td>
<td id="S4.T3.6.1.7.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.7.18.1" class="ltx_text ltx_font_bold" style="font-size:70%;">34.69</span></td>
</tr>
<tr id="S4.T3.6.1.8" class="ltx_tr">
<td id="S4.T3.6.1.8.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T3.6.1.8.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T3.6.1.8.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T3.6.1.8.1.3" class="ltx_text" style="font-size:70%;">(50)</span>
</td>
<td id="S4.T3.6.1.8.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.8.2.1" class="ltx_text" style="font-size:70%;">21.67</span></td>
<td id="S4.T3.6.1.8.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.8.3.1" class="ltx_text" style="font-size:70%;">21.80</span></td>
<td id="S4.T3.6.1.8.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.8.4.1" class="ltx_text" style="font-size:70%;">39.48</span></td>
<td id="S4.T3.6.1.8.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.8.5.1" class="ltx_text" style="font-size:70%;">28.74</span></td>
<td id="S4.T3.6.1.8.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.8.6.1" class="ltx_text" style="font-size:70%;">20.65</span></td>
<td id="S4.T3.6.1.8.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.8.7.1" class="ltx_text" style="font-size:70%;">13.14</span></td>
<td id="S4.T3.6.1.8.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.8.8.1" class="ltx_text" style="font-size:70%;">20.57</span></td>
<td id="S4.T3.6.1.8.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.8.9.1" class="ltx_text" style="font-size:70%;">22.83</span></td>
<td id="S4.T3.6.1.8.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.8.10.1" class="ltx_text" style="font-size:70%;">21.78</span></td>
<td id="S4.T3.6.1.8.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.8.11.1" class="ltx_text" style="font-size:70%;">39.00</span></td>
<td id="S4.T3.6.1.8.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.8.12.1" class="ltx_text" style="font-size:70%;">82.54</span></td>
<td id="S4.T3.6.1.8.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.8.13.1" class="ltx_text" style="font-size:70%;">40.12</span></td>
<td id="S4.T3.6.1.8.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.8.14.1" class="ltx_text" style="font-size:70%;">3.61</span></td>
<td id="S4.T3.6.1.8.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.8.15.1" class="ltx_text" style="font-size:70%;">78.39</span></td>
<td id="S4.T3.6.1.8.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.8.16.1" class="ltx_text" style="font-size:70%;">50.27</span></td>
<td id="S4.T3.6.1.8.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.8.17.1" class="ltx_text" style="font-size:70%;">48.4</span></td>
<td id="S4.T3.6.1.8.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.8.18.1" class="ltx_text" style="font-size:70%;">34.56</span></td>
</tr>
<tr id="S4.T3.6.1.9" class="ltx_tr">
<td id="S4.T3.6.1.9.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T3.6.1.9.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T3.6.1.9.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T3.6.1.9.1.3" class="ltx_text" style="font-size:70%;">(60)</span>
</td>
<td id="S4.T3.6.1.9.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.9.2.1" class="ltx_text" style="font-size:70%;">21.04</span></td>
<td id="S4.T3.6.1.9.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.9.3.1" class="ltx_text" style="font-size:70%;">21.30</span></td>
<td id="S4.T3.6.1.9.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.9.4.1" class="ltx_text" style="font-size:70%;">39.56</span></td>
<td id="S4.T3.6.1.9.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.9.5.1" class="ltx_text" style="font-size:70%;">28.68</span></td>
<td id="S4.T3.6.1.9.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.9.6.1" class="ltx_text" style="font-size:70%;">21.29</span></td>
<td id="S4.T3.6.1.9.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.9.7.1" class="ltx_text" style="font-size:70%;">13.97</span></td>
<td id="S4.T3.6.1.9.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.9.8.1" class="ltx_text" style="font-size:70%;">20.13</span></td>
<td id="S4.T3.6.1.9.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.9.9.1" class="ltx_text" style="font-size:70%;">22.52</span></td>
<td id="S4.T3.6.1.9.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.9.10.1" class="ltx_text" style="font-size:70%;">21.81</span></td>
<td id="S4.T3.6.1.9.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.9.11.1" class="ltx_text" style="font-size:70%;">39.50</span></td>
<td id="S4.T3.6.1.9.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.9.12.1" class="ltx_text" style="font-size:70%;">82.05</span></td>
<td id="S4.T3.6.1.9.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.9.13.1" class="ltx_text" style="font-size:70%;">39.14</span></td>
<td id="S4.T3.6.1.9.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.9.14.1" class="ltx_text" style="font-size:70%;">4.16</span></td>
<td id="S4.T3.6.1.9.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.9.15.1" class="ltx_text" style="font-size:70%;">74.23</span></td>
<td id="S4.T3.6.1.9.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.9.16.1" class="ltx_text" style="font-size:70%;">49.83</span></td>
<td id="S4.T3.6.1.9.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.9.17.1" class="ltx_text" style="font-size:70%;">47.67</span></td>
<td id="S4.T3.6.1.9.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.9.18.1" class="ltx_text" style="font-size:70%;">34.18</span></td>
</tr>
<tr id="S4.T3.6.1.10" class="ltx_tr">
<td id="S4.T3.6.1.10.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T3.6.1.10.1.1" class="ltx_ERROR undefined">\cdashline</span><span id="S4.T3.6.1.10.1.2" class="ltx_text" style="font-size:70%;">1-18
SnapKV</span>
</td>
<td id="S4.T3.6.1.10.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.10.2.1" class="ltx_text" style="font-size:70%;">19.17</span></td>
<td id="S4.T3.6.1.10.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.10.3.1" class="ltx_text" style="font-size:70%;">21.40</span></td>
<td id="S4.T3.6.1.10.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.10.4.1" class="ltx_text" style="font-size:70%;">42.93</span></td>
<td id="S4.T3.6.1.10.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.10.5.1" class="ltx_text" style="font-size:70%;">36.76</span></td>
<td id="S4.T3.6.1.10.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.10.6.1" class="ltx_text" style="font-size:70%;">22.44</span></td>
<td id="S4.T3.6.1.10.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.10.7.1" class="ltx_text" style="font-size:70%;">15.86</span></td>
<td id="S4.T3.6.1.10.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.10.8.1" class="ltx_text" style="font-size:70%;">19.16</span></td>
<td id="S4.T3.6.1.10.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.10.9.1" class="ltx_text" style="font-size:70%;">21.84</span></td>
<td id="S4.T3.6.1.10.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.10.10.1" class="ltx_text" style="font-size:70%;">21.55</span></td>
<td id="S4.T3.6.1.10.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.10.11.1" class="ltx_text" style="font-size:70%;">47.50</span></td>
<td id="S4.T3.6.1.10.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.10.12.1" class="ltx_text" style="font-size:70%;">84.15</span></td>
<td id="S4.T3.6.1.10.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.10.13.1" class="ltx_text" style="font-size:70%;">40.24</span></td>
<td id="S4.T3.6.1.10.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.10.14.1" class="ltx_text" style="font-size:70%;">2.30</span></td>
<td id="S4.T3.6.1.10.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.10.15.1" class="ltx_text" style="font-size:70%;">68.26</span></td>
<td id="S4.T3.6.1.10.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.10.16.1" class="ltx_text" style="font-size:70%;">52.31</span></td>
<td id="S4.T3.6.1.10.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.10.17.1" class="ltx_text" style="font-size:70%;">48.80</span></td>
<td id="S4.T3.6.1.10.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.10.18.1" class="ltx_text ltx_font_bold" style="font-size:70%;">35.29</span></td>
</tr>
<tr id="S4.T3.6.1.11" class="ltx_tr">
<td id="S4.T3.6.1.11.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T3.6.1.11.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T3.6.1.11.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T3.6.1.11.1.3" class="ltx_text" style="font-size:70%;">(40)</span>
</td>
<td id="S4.T3.6.1.11.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.11.2.1" class="ltx_text" style="font-size:70%;">20.52</span></td>
<td id="S4.T3.6.1.11.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.11.3.1" class="ltx_text" style="font-size:70%;">21.00</span></td>
<td id="S4.T3.6.1.11.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.11.4.1" class="ltx_text" style="font-size:70%;">42.65</span></td>
<td id="S4.T3.6.1.11.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.11.5.1" class="ltx_text" style="font-size:70%;">37.58</span></td>
<td id="S4.T3.6.1.11.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.11.6.1" class="ltx_text" style="font-size:70%;">22.09</span></td>
<td id="S4.T3.6.1.11.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.11.7.1" class="ltx_text" style="font-size:70%;">15.23</span></td>
<td id="S4.T3.6.1.11.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.11.8.1" class="ltx_text" style="font-size:70%;">19.29</span></td>
<td id="S4.T3.6.1.11.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.11.9.1" class="ltx_text" style="font-size:70%;">22.01</span></td>
<td id="S4.T3.6.1.11.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.11.10.1" class="ltx_text" style="font-size:70%;">21.22</span></td>
<td id="S4.T3.6.1.11.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.11.11.1" class="ltx_text" style="font-size:70%;">47.00</span></td>
<td id="S4.T3.6.1.11.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.11.12.1" class="ltx_text" style="font-size:70%;">83.85</span></td>
<td id="S4.T3.6.1.11.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.11.13.1" class="ltx_text" style="font-size:70%;">39.64</span></td>
<td id="S4.T3.6.1.11.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.11.14.1" class="ltx_text" style="font-size:70%;">3.20</span></td>
<td id="S4.T3.6.1.11.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.11.15.1" class="ltx_text" style="font-size:70%;">67.45</span></td>
<td id="S4.T3.6.1.11.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.11.16.1" class="ltx_text" style="font-size:70%;">51.48</span></td>
<td id="S4.T3.6.1.11.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.11.17.1" class="ltx_text" style="font-size:70%;">48.31</span></td>
<td id="S4.T3.6.1.11.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.11.18.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:70%;">35.16</span></td>
</tr>
<tr id="S4.T3.6.1.12" class="ltx_tr">
<td id="S4.T3.6.1.12.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T3.6.1.12.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T3.6.1.12.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T3.6.1.12.1.3" class="ltx_text" style="font-size:70%;">(50)</span>
</td>
<td id="S4.T3.6.1.12.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.12.2.1" class="ltx_text" style="font-size:70%;">20.67</span></td>
<td id="S4.T3.6.1.12.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.12.3.1" class="ltx_text" style="font-size:70%;">20.60</span></td>
<td id="S4.T3.6.1.12.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.12.4.1" class="ltx_text" style="font-size:70%;">43.37</span></td>
<td id="S4.T3.6.1.12.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.12.5.1" class="ltx_text" style="font-size:70%;">37.27</span></td>
<td id="S4.T3.6.1.12.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.12.6.1" class="ltx_text" style="font-size:70%;">21.58</span></td>
<td id="S4.T3.6.1.12.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.12.7.1" class="ltx_text" style="font-size:70%;">15.66</span></td>
<td id="S4.T3.6.1.12.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.12.8.1" class="ltx_text" style="font-size:70%;">19.06</span></td>
<td id="S4.T3.6.1.12.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.12.9.1" class="ltx_text" style="font-size:70%;">21.79</span></td>
<td id="S4.T3.6.1.12.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.12.10.1" class="ltx_text" style="font-size:70%;">21.02</span></td>
<td id="S4.T3.6.1.12.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.12.11.1" class="ltx_text" style="font-size:70%;">47.00</span></td>
<td id="S4.T3.6.1.12.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.12.12.1" class="ltx_text" style="font-size:70%;">83.38</span></td>
<td id="S4.T3.6.1.12.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.12.13.1" class="ltx_text" style="font-size:70%;">39.77</span></td>
<td id="S4.T3.6.1.12.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.12.14.1" class="ltx_text" style="font-size:70%;">3.65</span></td>
<td id="S4.T3.6.1.12.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.12.15.1" class="ltx_text" style="font-size:70%;">67.06</span></td>
<td id="S4.T3.6.1.12.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.12.16.1" class="ltx_text" style="font-size:70%;">50.80</span></td>
<td id="S4.T3.6.1.12.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.12.17.1" class="ltx_text" style="font-size:70%;">48.35</span></td>
<td id="S4.T3.6.1.12.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.12.18.1" class="ltx_text" style="font-size:70%;">35.06</span></td>
</tr>
<tr id="S4.T3.6.1.13" class="ltx_tr">
<td id="S4.T3.6.1.13.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T3.6.1.13.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T3.6.1.13.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T3.6.1.13.1.3" class="ltx_text" style="font-size:70%;">(60)</span>
</td>
<td id="S4.T3.6.1.13.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.13.2.1" class="ltx_text" style="font-size:70%;">21.25</span></td>
<td id="S4.T3.6.1.13.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.13.3.1" class="ltx_text" style="font-size:70%;">20.82</span></td>
<td id="S4.T3.6.1.13.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.13.4.1" class="ltx_text" style="font-size:70%;">44.20</span></td>
<td id="S4.T3.6.1.13.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.13.5.1" class="ltx_text" style="font-size:70%;">36.21</span></td>
<td id="S4.T3.6.1.13.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.13.6.1" class="ltx_text" style="font-size:70%;">21.68</span></td>
<td id="S4.T3.6.1.13.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.13.7.1" class="ltx_text" style="font-size:70%;">16.47</span></td>
<td id="S4.T3.6.1.13.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.13.8.1" class="ltx_text" style="font-size:70%;">19.05</span></td>
<td id="S4.T3.6.1.13.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.13.9.1" class="ltx_text" style="font-size:70%;">21.99</span></td>
<td id="S4.T3.6.1.13.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.13.10.1" class="ltx_text" style="font-size:70%;">20.73</span></td>
<td id="S4.T3.6.1.13.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.13.11.1" class="ltx_text" style="font-size:70%;">45.00</span></td>
<td id="S4.T3.6.1.13.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.13.12.1" class="ltx_text" style="font-size:70%;">83.81</span></td>
<td id="S4.T3.6.1.13.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.13.13.1" class="ltx_text" style="font-size:70%;">38.79</span></td>
<td id="S4.T3.6.1.13.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.13.14.1" class="ltx_text" style="font-size:70%;">4.19</span></td>
<td id="S4.T3.6.1.13.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.13.15.1" class="ltx_text" style="font-size:70%;">66.90</span></td>
<td id="S4.T3.6.1.13.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.13.16.1" class="ltx_text" style="font-size:70%;">49.99</span></td>
<td id="S4.T3.6.1.13.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.13.17.1" class="ltx_text" style="font-size:70%;">47.61</span></td>
<td id="S4.T3.6.1.13.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.13.18.1" class="ltx_text" style="font-size:70%;">34.92</span></td>
</tr>
<tr id="S4.T3.6.1.14" class="ltx_tr">
<td id="S4.T3.6.1.14.1" class="ltx_td ltx_align_center ltx_border_t" colspan="18"><span id="S4.T3.6.1.14.1.1" class="ltx_text" style="font-size:70%;">KV-size 512</span></td>
</tr>
<tr id="S4.T3.6.1.15" class="ltx_tr">
<td id="S4.T3.6.1.15.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T3.6.1.15.1.1" class="ltx_text" style="font-size:70%;">H2O</span></td>
<td id="S4.T3.6.1.15.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.15.2.1" class="ltx_text" style="font-size:70%;">21.83</span></td>
<td id="S4.T3.6.1.15.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.15.3.1" class="ltx_text" style="font-size:70%;">26.00</span></td>
<td id="S4.T3.6.1.15.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.15.4.1" class="ltx_text" style="font-size:70%;">44.69</span></td>
<td id="S4.T3.6.1.15.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.15.5.1" class="ltx_text" style="font-size:70%;">32.46</span></td>
<td id="S4.T3.6.1.15.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.15.6.1" class="ltx_text" style="font-size:70%;">23.05</span></td>
<td id="S4.T3.6.1.15.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.15.7.1" class="ltx_text" style="font-size:70%;">14.69</span></td>
<td id="S4.T3.6.1.15.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.15.8.1" class="ltx_text" style="font-size:70%;">23.53</span></td>
<td id="S4.T3.6.1.15.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.15.9.1" class="ltx_text" style="font-size:70%;">23.06</span></td>
<td id="S4.T3.6.1.15.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.15.10.1" class="ltx_text" style="font-size:70%;">24.59</span></td>
<td id="S4.T3.6.1.15.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.15.11.1" class="ltx_text" style="font-size:70%;">42.00</span></td>
<td id="S4.T3.6.1.15.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.15.12.1" class="ltx_text" style="font-size:70%;">85.22</span></td>
<td id="S4.T3.6.1.15.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.15.13.1" class="ltx_text" style="font-size:70%;">41.49</span></td>
<td id="S4.T3.6.1.15.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.15.14.1" class="ltx_text" style="font-size:70%;">3.40</span></td>
<td id="S4.T3.6.1.15.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.15.15.1" class="ltx_text" style="font-size:70%;">86.20</span></td>
<td id="S4.T3.6.1.15.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.15.16.1" class="ltx_text" style="font-size:70%;">54.78</span></td>
<td id="S4.T3.6.1.15.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.15.17.1" class="ltx_text" style="font-size:70%;">51.09</span></td>
<td id="S4.T3.6.1.15.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.15.18.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:70%;">37.38</span></td>
</tr>
<tr id="S4.T3.6.1.16" class="ltx_tr">
<td id="S4.T3.6.1.16.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T3.6.1.16.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T3.6.1.16.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T3.6.1.16.1.3" class="ltx_text" style="font-size:70%;">(40)</span>
</td>
<td id="S4.T3.6.1.16.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.16.2.1" class="ltx_text" style="font-size:70%;">21.58</span></td>
<td id="S4.T3.6.1.16.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.16.3.1" class="ltx_text" style="font-size:70%;">26.15</span></td>
<td id="S4.T3.6.1.16.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.16.4.1" class="ltx_text" style="font-size:70%;">44.49</span></td>
<td id="S4.T3.6.1.16.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.16.5.1" class="ltx_text" style="font-size:70%;">32.73</span></td>
<td id="S4.T3.6.1.16.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.16.6.1" class="ltx_text" style="font-size:70%;">23.99</span></td>
<td id="S4.T3.6.1.16.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.16.7.1" class="ltx_text" style="font-size:70%;">15.09</span></td>
<td id="S4.T3.6.1.16.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.16.8.1" class="ltx_text" style="font-size:70%;">23.56</span></td>
<td id="S4.T3.6.1.16.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.16.9.1" class="ltx_text" style="font-size:70%;">23.28</span></td>
<td id="S4.T3.6.1.16.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.16.10.1" class="ltx_text" style="font-size:70%;">24.45</span></td>
<td id="S4.T3.6.1.16.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.16.11.1" class="ltx_text" style="font-size:70%;">42.00</span></td>
<td id="S4.T3.6.1.16.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.16.12.1" class="ltx_text" style="font-size:70%;">85.58</span></td>
<td id="S4.T3.6.1.16.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.16.13.1" class="ltx_text" style="font-size:70%;">42.58</span></td>
<td id="S4.T3.6.1.16.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.16.14.1" class="ltx_text" style="font-size:70%;">3.18</span></td>
<td id="S4.T3.6.1.16.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.16.15.1" class="ltx_text" style="font-size:70%;">85.7</span></td>
<td id="S4.T3.6.1.16.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.16.16.1" class="ltx_text" style="font-size:70%;">54.39</span></td>
<td id="S4.T3.6.1.16.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.16.17.1" class="ltx_text" style="font-size:70%;">51.15</span></td>
<td id="S4.T3.6.1.16.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.16.18.1" class="ltx_text ltx_font_bold" style="font-size:70%;">37.49</span></td>
</tr>
<tr id="S4.T3.6.1.17" class="ltx_tr">
<td id="S4.T3.6.1.17.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T3.6.1.17.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T3.6.1.17.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T3.6.1.17.1.3" class="ltx_text" style="font-size:70%;">(50)</span>
</td>
<td id="S4.T3.6.1.17.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.17.2.1" class="ltx_text" style="font-size:70%;">22.76</span></td>
<td id="S4.T3.6.1.17.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.17.3.1" class="ltx_text" style="font-size:70%;">25.74</span></td>
<td id="S4.T3.6.1.17.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.17.4.1" class="ltx_text" style="font-size:70%;">44.61</span></td>
<td id="S4.T3.6.1.17.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.17.5.1" class="ltx_text" style="font-size:70%;">31.74</span></td>
<td id="S4.T3.6.1.17.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.17.6.1" class="ltx_text" style="font-size:70%;">23.25</span></td>
<td id="S4.T3.6.1.17.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.17.7.1" class="ltx_text" style="font-size:70%;">13.91</span></td>
<td id="S4.T3.6.1.17.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.17.8.1" class="ltx_text" style="font-size:70%;">23.31</span></td>
<td id="S4.T3.6.1.17.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.17.9.1" class="ltx_text" style="font-size:70%;">23.13</span></td>
<td id="S4.T3.6.1.17.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.17.10.1" class="ltx_text" style="font-size:70%;">24.34</span></td>
<td id="S4.T3.6.1.17.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.17.11.1" class="ltx_text" style="font-size:70%;">41.00</span></td>
<td id="S4.T3.6.1.17.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.17.12.1" class="ltx_text" style="font-size:70%;">85.39</span></td>
<td id="S4.T3.6.1.17.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.17.13.1" class="ltx_text" style="font-size:70%;">41.85</span></td>
<td id="S4.T3.6.1.17.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.17.14.1" class="ltx_text" style="font-size:70%;">2.82</span></td>
<td id="S4.T3.6.1.17.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.17.15.1" class="ltx_text" style="font-size:70%;">84.36</span></td>
<td id="S4.T3.6.1.17.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.17.16.1" class="ltx_text" style="font-size:70%;">54.69</span></td>
<td id="S4.T3.6.1.17.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.17.17.1" class="ltx_text" style="font-size:70%;">50.88</span></td>
<td id="S4.T3.6.1.17.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.17.18.1" class="ltx_text" style="font-size:70%;">37.11</span></td>
</tr>
<tr id="S4.T3.6.1.18" class="ltx_tr">
<td id="S4.T3.6.1.18.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T3.6.1.18.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T3.6.1.18.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T3.6.1.18.1.3" class="ltx_text" style="font-size:70%;">(60)</span>
</td>
<td id="S4.T3.6.1.18.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.18.2.1" class="ltx_text" style="font-size:70%;">22.91</span></td>
<td id="S4.T3.6.1.18.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.18.3.1" class="ltx_text" style="font-size:70%;">25.57</span></td>
<td id="S4.T3.6.1.18.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.18.4.1" class="ltx_text" style="font-size:70%;">44.04</span></td>
<td id="S4.T3.6.1.18.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.18.5.1" class="ltx_text" style="font-size:70%;">29.48</span></td>
<td id="S4.T3.6.1.18.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.18.6.1" class="ltx_text" style="font-size:70%;">22.88</span></td>
<td id="S4.T3.6.1.18.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.18.7.1" class="ltx_text" style="font-size:70%;">13.67</span></td>
<td id="S4.T3.6.1.18.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.18.8.1" class="ltx_text" style="font-size:70%;">23.31</span></td>
<td id="S4.T3.6.1.18.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.18.9.1" class="ltx_text" style="font-size:70%;">22.64</span></td>
<td id="S4.T3.6.1.18.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.18.10.1" class="ltx_text" style="font-size:70%;">24.10</span></td>
<td id="S4.T3.6.1.18.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.18.11.1" class="ltx_text" style="font-size:70%;">41.00</span></td>
<td id="S4.T3.6.1.18.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.18.12.1" class="ltx_text" style="font-size:70%;">85.31</span></td>
<td id="S4.T3.6.1.18.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.18.13.1" class="ltx_text" style="font-size:70%;">41.15</span></td>
<td id="S4.T3.6.1.18.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.18.14.1" class="ltx_text" style="font-size:70%;">2.98</span></td>
<td id="S4.T3.6.1.18.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.18.15.1" class="ltx_text" style="font-size:70%;">82.34</span></td>
<td id="S4.T3.6.1.18.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.18.16.1" class="ltx_text" style="font-size:70%;">53.70</span></td>
<td id="S4.T3.6.1.18.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.18.17.1" class="ltx_text" style="font-size:70%;">50.25</span></td>
<td id="S4.T3.6.1.18.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.18.18.1" class="ltx_text" style="font-size:70%;">36.58</span></td>
</tr>
<tr id="S4.T3.6.1.19" class="ltx_tr">
<td id="S4.T3.6.1.19.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T3.6.1.19.1.1" class="ltx_ERROR undefined">\cdashline</span><span id="S4.T3.6.1.19.1.2" class="ltx_text" style="font-size:70%;">1-18
SnapKV</span>
</td>
<td id="S4.T3.6.1.19.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.19.2.1" class="ltx_text" style="font-size:70%;">24.44</span></td>
<td id="S4.T3.6.1.19.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.19.3.1" class="ltx_text" style="font-size:70%;">27.81</span></td>
<td id="S4.T3.6.1.19.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.19.4.1" class="ltx_text" style="font-size:70%;">48.98</span></td>
<td id="S4.T3.6.1.19.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.19.5.1" class="ltx_text" style="font-size:70%;">39.46</span></td>
<td id="S4.T3.6.1.19.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.19.6.1" class="ltx_text" style="font-size:70%;">25.25</span></td>
<td id="S4.T3.6.1.19.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.19.7.1" class="ltx_text" style="font-size:70%;">16.98</span></td>
<td id="S4.T3.6.1.19.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.19.8.1" class="ltx_text" style="font-size:70%;">23.70</span></td>
<td id="S4.T3.6.1.19.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.19.9.1" class="ltx_text" style="font-size:70%;">22.96</span></td>
<td id="S4.T3.6.1.19.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.19.10.1" class="ltx_text" style="font-size:70%;">24.37</span></td>
<td id="S4.T3.6.1.19.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.19.11.1" class="ltx_text" style="font-size:70%;">67.00</span></td>
<td id="S4.T3.6.1.19.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.19.12.1" class="ltx_text" style="font-size:70%;">85.88</span></td>
<td id="S4.T3.6.1.19.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.19.13.1" class="ltx_text" style="font-size:70%;">41.26</span></td>
<td id="S4.T3.6.1.19.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.19.14.1" class="ltx_text" style="font-size:70%;">2.78</span></td>
<td id="S4.T3.6.1.19.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.19.15.1" class="ltx_text" style="font-size:70%;">86.56</span></td>
<td id="S4.T3.6.1.19.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.19.16.1" class="ltx_text" style="font-size:70%;">56.46</span></td>
<td id="S4.T3.6.1.19.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.19.17.1" class="ltx_text" style="font-size:70%;">53.41</span></td>
<td id="S4.T3.6.1.19.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.19.18.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:70%;">40.46</span></td>
</tr>
<tr id="S4.T3.6.1.20" class="ltx_tr">
<td id="S4.T3.6.1.20.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T3.6.1.20.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T3.6.1.20.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T3.6.1.20.1.3" class="ltx_text" style="font-size:70%;">(40)</span>
</td>
<td id="S4.T3.6.1.20.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.20.2.1" class="ltx_text" style="font-size:70%;">24.27</span></td>
<td id="S4.T3.6.1.20.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.20.3.1" class="ltx_text" style="font-size:70%;">28.46</span></td>
<td id="S4.T3.6.1.20.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.20.4.1" class="ltx_text" style="font-size:70%;">49.26</span></td>
<td id="S4.T3.6.1.20.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.20.5.1" class="ltx_text" style="font-size:70%;">38.13</span></td>
<td id="S4.T3.6.1.20.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.20.6.1" class="ltx_text" style="font-size:70%;">24.22</span></td>
<td id="S4.T3.6.1.20.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.20.7.1" class="ltx_text" style="font-size:70%;">16.92</span></td>
<td id="S4.T3.6.1.20.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.20.8.1" class="ltx_text" style="font-size:70%;">23.59</span></td>
<td id="S4.T3.6.1.20.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.20.9.1" class="ltx_text" style="font-size:70%;">23.70</span></td>
<td id="S4.T3.6.1.20.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.20.10.1" class="ltx_text" style="font-size:70%;">24.46</span></td>
<td id="S4.T3.6.1.20.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.20.11.1" class="ltx_text" style="font-size:70%;">67.50</span></td>
<td id="S4.T3.6.1.20.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.20.12.1" class="ltx_text" style="font-size:70%;">85.9</span></td>
<td id="S4.T3.6.1.20.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.20.13.1" class="ltx_text" style="font-size:70%;">42.51</span></td>
<td id="S4.T3.6.1.20.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.20.14.1" class="ltx_text" style="font-size:70%;">2.92</span></td>
<td id="S4.T3.6.1.20.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.20.15.1" class="ltx_text" style="font-size:70%;">85.32</span></td>
<td id="S4.T3.6.1.20.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.20.16.1" class="ltx_text" style="font-size:70%;">55.89</span></td>
<td id="S4.T3.6.1.20.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.20.17.1" class="ltx_text" style="font-size:70%;">53.35</span></td>
<td id="S4.T3.6.1.20.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.20.18.1" class="ltx_text" style="font-size:70%;">40.40</span></td>
</tr>
<tr id="S4.T3.6.1.21" class="ltx_tr">
<td id="S4.T3.6.1.21.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T3.6.1.21.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T3.6.1.21.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T3.6.1.21.1.3" class="ltx_text" style="font-size:70%;">(50)</span>
</td>
<td id="S4.T3.6.1.21.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.21.2.1" class="ltx_text" style="font-size:70%;">24.56</span></td>
<td id="S4.T3.6.1.21.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.21.3.1" class="ltx_text" style="font-size:70%;">29.22</span></td>
<td id="S4.T3.6.1.21.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.21.4.1" class="ltx_text" style="font-size:70%;">48.59</span></td>
<td id="S4.T3.6.1.21.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.21.5.1" class="ltx_text" style="font-size:70%;">37.70</span></td>
<td id="S4.T3.6.1.21.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.21.6.1" class="ltx_text" style="font-size:70%;">24.27</span></td>
<td id="S4.T3.6.1.21.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.21.7.1" class="ltx_text" style="font-size:70%;">17.39</span></td>
<td id="S4.T3.6.1.21.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.21.8.1" class="ltx_text" style="font-size:70%;">23.68</span></td>
<td id="S4.T3.6.1.21.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.21.9.1" class="ltx_text" style="font-size:70%;">23.65</span></td>
<td id="S4.T3.6.1.21.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.21.10.1" class="ltx_text" style="font-size:70%;">24.58</span></td>
<td id="S4.T3.6.1.21.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.21.11.1" class="ltx_text" style="font-size:70%;">67.50</span></td>
<td id="S4.T3.6.1.21.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.21.12.1" class="ltx_text" style="font-size:70%;">86.05</span></td>
<td id="S4.T3.6.1.21.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.21.13.1" class="ltx_text" style="font-size:70%;">42.01</span></td>
<td id="S4.T3.6.1.21.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.21.14.1" class="ltx_text" style="font-size:70%;">3.07</span></td>
<td id="S4.T3.6.1.21.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.21.15.1" class="ltx_text" style="font-size:70%;">86.30</span></td>
<td id="S4.T3.6.1.21.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.21.16.1" class="ltx_text" style="font-size:70%;">56.49</span></td>
<td id="S4.T3.6.1.21.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.21.17.1" class="ltx_text" style="font-size:70%;">53.29</span></td>
<td id="S4.T3.6.1.21.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.21.18.1" class="ltx_text ltx_font_bold" style="font-size:70%;">40.52</span></td>
</tr>
<tr id="S4.T3.6.1.22" class="ltx_tr">
<td id="S4.T3.6.1.22.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T3.6.1.22.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T3.6.1.22.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T3.6.1.22.1.3" class="ltx_text" style="font-size:70%;">(60)</span>
</td>
<td id="S4.T3.6.1.22.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.22.2.1" class="ltx_text" style="font-size:70%;">24.07</span></td>
<td id="S4.T3.6.1.22.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.22.3.1" class="ltx_text" style="font-size:70%;">28.27</span></td>
<td id="S4.T3.6.1.22.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.22.4.1" class="ltx_text" style="font-size:70%;">49.10</span></td>
<td id="S4.T3.6.1.22.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.22.5.1" class="ltx_text" style="font-size:70%;">38.65</span></td>
<td id="S4.T3.6.1.22.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.22.6.1" class="ltx_text" style="font-size:70%;">24.31</span></td>
<td id="S4.T3.6.1.22.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.22.7.1" class="ltx_text" style="font-size:70%;">17.52</span></td>
<td id="S4.T3.6.1.22.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.22.8.1" class="ltx_text" style="font-size:70%;">23.16</span></td>
<td id="S4.T3.6.1.22.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.22.9.1" class="ltx_text" style="font-size:70%;">23.51</span></td>
<td id="S4.T3.6.1.22.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.22.10.1" class="ltx_text" style="font-size:70%;">24.23</span></td>
<td id="S4.T3.6.1.22.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.22.11.1" class="ltx_text" style="font-size:70%;">67.00</span></td>
<td id="S4.T3.6.1.22.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.22.12.1" class="ltx_text" style="font-size:70%;">86.33</span></td>
<td id="S4.T3.6.1.22.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.22.13.1" class="ltx_text" style="font-size:70%;">40.78</span></td>
<td id="S4.T3.6.1.22.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.22.14.1" class="ltx_text" style="font-size:70%;">3.69</span></td>
<td id="S4.T3.6.1.22.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.22.15.1" class="ltx_text" style="font-size:70%;">83.74</span></td>
<td id="S4.T3.6.1.22.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.22.16.1" class="ltx_text" style="font-size:70%;">54.94</span></td>
<td id="S4.T3.6.1.22.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.22.17.1" class="ltx_text" style="font-size:70%;">52.23</span></td>
<td id="S4.T3.6.1.22.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.22.18.1" class="ltx_text" style="font-size:70%;">40.10</span></td>
</tr>
<tr id="S4.T3.6.1.23" class="ltx_tr">
<td id="S4.T3.6.1.23.1" class="ltx_td ltx_align_center ltx_border_t" colspan="18"><span id="S4.T3.6.1.23.1.1" class="ltx_text" style="font-size:70%;">KV-size 1024</span></td>
</tr>
<tr id="S4.T3.6.1.24" class="ltx_tr">
<td id="S4.T3.6.1.24.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T3.6.1.24.1.1" class="ltx_text" style="font-size:70%;">H2O</span></td>
<td id="S4.T3.6.1.24.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.24.2.1" class="ltx_text" style="font-size:70%;">23.67</span></td>
<td id="S4.T3.6.1.24.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.24.3.1" class="ltx_text" style="font-size:70%;">28.55</span></td>
<td id="S4.T3.6.1.24.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.24.4.1" class="ltx_text" style="font-size:70%;">46.40</span></td>
<td id="S4.T3.6.1.24.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.24.5.1" class="ltx_text" style="font-size:70%;">36.99</span></td>
<td id="S4.T3.6.1.24.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.24.6.1" class="ltx_text" style="font-size:70%;">24.82</span></td>
<td id="S4.T3.6.1.24.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.24.7.1" class="ltx_text" style="font-size:70%;">15.02</span></td>
<td id="S4.T3.6.1.24.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.24.8.1" class="ltx_text" style="font-size:70%;">25.21</span></td>
<td id="S4.T3.6.1.24.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.24.9.1" class="ltx_text" style="font-size:70%;">23.04</span></td>
<td id="S4.T3.6.1.24.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.24.10.1" class="ltx_text" style="font-size:70%;">25.77</span></td>
<td id="S4.T3.6.1.24.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.24.11.1" class="ltx_text" style="font-size:70%;">46.00</span></td>
<td id="S4.T3.6.1.24.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.24.12.1" class="ltx_text" style="font-size:70%;">85.93</span></td>
<td id="S4.T3.6.1.24.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.24.13.1" class="ltx_text" style="font-size:70%;">41.98</span></td>
<td id="S4.T3.6.1.24.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.24.14.1" class="ltx_text" style="font-size:70%;">3.24</span></td>
<td id="S4.T3.6.1.24.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.24.15.1" class="ltx_text" style="font-size:70%;">86.57</span></td>
<td id="S4.T3.6.1.24.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.24.16.1" class="ltx_text" style="font-size:70%;">56.40</span></td>
<td id="S4.T3.6.1.24.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.24.17.1" class="ltx_text" style="font-size:70%;">52.75</span></td>
<td id="S4.T3.6.1.24.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.24.18.1" class="ltx_text ltx_font_bold" style="font-size:70%;">38.90</span></td>
</tr>
<tr id="S4.T3.6.1.25" class="ltx_tr">
<td id="S4.T3.6.1.25.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T3.6.1.25.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T3.6.1.25.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T3.6.1.25.1.3" class="ltx_text" style="font-size:70%;">(40)</span>
</td>
<td id="S4.T3.6.1.25.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.25.2.1" class="ltx_text" style="font-size:70%;">23.97</span></td>
<td id="S4.T3.6.1.25.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.25.3.1" class="ltx_text" style="font-size:70%;">28.91</span></td>
<td id="S4.T3.6.1.25.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.25.4.1" class="ltx_text" style="font-size:70%;">45.84</span></td>
<td id="S4.T3.6.1.25.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.25.5.1" class="ltx_text" style="font-size:70%;">35.78</span></td>
<td id="S4.T3.6.1.25.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.25.6.1" class="ltx_text" style="font-size:70%;">24.88</span></td>
<td id="S4.T3.6.1.25.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.25.7.1" class="ltx_text" style="font-size:70%;">14.55</span></td>
<td id="S4.T3.6.1.25.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.25.8.1" class="ltx_text" style="font-size:70%;">25.11</span></td>
<td id="S4.T3.6.1.25.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.25.9.1" class="ltx_text" style="font-size:70%;">23.35</span></td>
<td id="S4.T3.6.1.25.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.25.10.1" class="ltx_text" style="font-size:70%;">25.83</span></td>
<td id="S4.T3.6.1.25.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.25.11.1" class="ltx_text" style="font-size:70%;">45.50</span></td>
<td id="S4.T3.6.1.25.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.25.12.1" class="ltx_text" style="font-size:70%;">86.11</span></td>
<td id="S4.T3.6.1.25.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.25.13.1" class="ltx_text" style="font-size:70%;">42.44</span></td>
<td id="S4.T3.6.1.25.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.25.14.1" class="ltx_text" style="font-size:70%;">3.23</span></td>
<td id="S4.T3.6.1.25.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.25.15.1" class="ltx_text" style="font-size:70%;">84.82</span></td>
<td id="S4.T3.6.1.25.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.25.16.1" class="ltx_text" style="font-size:70%;">56.21</span></td>
<td id="S4.T3.6.1.25.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.25.17.1" class="ltx_text" style="font-size:70%;">53.02</span></td>
<td id="S4.T3.6.1.25.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.25.18.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:70%;">38.72</span></td>
</tr>
<tr id="S4.T3.6.1.26" class="ltx_tr">
<td id="S4.T3.6.1.26.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T3.6.1.26.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T3.6.1.26.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T3.6.1.26.1.3" class="ltx_text" style="font-size:70%;">(50)</span>
</td>
<td id="S4.T3.6.1.26.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.26.2.1" class="ltx_text" style="font-size:70%;">23.89</span></td>
<td id="S4.T3.6.1.26.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.26.3.1" class="ltx_text" style="font-size:70%;">28.40</span></td>
<td id="S4.T3.6.1.26.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.26.4.1" class="ltx_text" style="font-size:70%;">46.60</span></td>
<td id="S4.T3.6.1.26.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.26.5.1" class="ltx_text" style="font-size:70%;">35.57</span></td>
<td id="S4.T3.6.1.26.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.26.6.1" class="ltx_text" style="font-size:70%;">24.26</span></td>
<td id="S4.T3.6.1.26.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.26.7.1" class="ltx_text" style="font-size:70%;">14.78</span></td>
<td id="S4.T3.6.1.26.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.26.8.1" class="ltx_text" style="font-size:70%;">24.98</span></td>
<td id="S4.T3.6.1.26.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.26.9.1" class="ltx_text" style="font-size:70%;">23.31</span></td>
<td id="S4.T3.6.1.26.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.26.10.1" class="ltx_text" style="font-size:70%;">25.68</span></td>
<td id="S4.T3.6.1.26.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.26.11.1" class="ltx_text" style="font-size:70%;">44.50</span></td>
<td id="S4.T3.6.1.26.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.26.12.1" class="ltx_text" style="font-size:70%;">86.16</span></td>
<td id="S4.T3.6.1.26.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.26.13.1" class="ltx_text" style="font-size:70%;">42.72</span></td>
<td id="S4.T3.6.1.26.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.26.14.1" class="ltx_text" style="font-size:70%;">3.38</span></td>
<td id="S4.T3.6.1.26.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.26.15.1" class="ltx_text" style="font-size:70%;">83.20</span></td>
<td id="S4.T3.6.1.26.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.26.16.1" class="ltx_text" style="font-size:70%;">55.88</span></td>
<td id="S4.T3.6.1.26.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.26.17.1" class="ltx_text" style="font-size:70%;">52.63</span></td>
<td id="S4.T3.6.1.26.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.26.18.1" class="ltx_text" style="font-size:70%;">38.50</span></td>
</tr>
<tr id="S4.T3.6.1.27" class="ltx_tr">
<td id="S4.T3.6.1.27.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T3.6.1.27.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T3.6.1.27.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T3.6.1.27.1.3" class="ltx_text" style="font-size:70%;">(60)</span>
</td>
<td id="S4.T3.6.1.27.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.27.2.1" class="ltx_text" style="font-size:70%;">23.87</span></td>
<td id="S4.T3.6.1.27.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.27.3.1" class="ltx_text" style="font-size:70%;">27.76</span></td>
<td id="S4.T3.6.1.27.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.27.4.1" class="ltx_text" style="font-size:70%;">46.25</span></td>
<td id="S4.T3.6.1.27.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.27.5.1" class="ltx_text" style="font-size:70%;">35.28</span></td>
<td id="S4.T3.6.1.27.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.27.6.1" class="ltx_text" style="font-size:70%;">24.38</span></td>
<td id="S4.T3.6.1.27.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.27.7.1" class="ltx_text" style="font-size:70%;">14.74</span></td>
<td id="S4.T3.6.1.27.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.27.8.1" class="ltx_text" style="font-size:70%;">24.35</span></td>
<td id="S4.T3.6.1.27.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.27.9.1" class="ltx_text" style="font-size:70%;">23.35</span></td>
<td id="S4.T3.6.1.27.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.27.10.1" class="ltx_text" style="font-size:70%;">25.50</span></td>
<td id="S4.T3.6.1.27.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.27.11.1" class="ltx_text" style="font-size:70%;">44.50</span></td>
<td id="S4.T3.6.1.27.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.27.12.1" class="ltx_text" style="font-size:70%;">85.38</span></td>
<td id="S4.T3.6.1.27.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.27.13.1" class="ltx_text" style="font-size:70%;">41.37</span></td>
<td id="S4.T3.6.1.27.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.27.14.1" class="ltx_text" style="font-size:70%;">3.34</span></td>
<td id="S4.T3.6.1.27.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.27.15.1" class="ltx_text" style="font-size:70%;">81.42</span></td>
<td id="S4.T3.6.1.27.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.27.16.1" class="ltx_text" style="font-size:70%;">55.21</span></td>
<td id="S4.T3.6.1.27.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.27.17.1" class="ltx_text" style="font-size:70%;">51.89</span></td>
<td id="S4.T3.6.1.27.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.27.18.1" class="ltx_text" style="font-size:70%;">38.04</span></td>
</tr>
<tr id="S4.T3.6.1.28" class="ltx_tr">
<td id="S4.T3.6.1.28.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T3.6.1.28.1.1" class="ltx_ERROR undefined">\cdashline</span><span id="S4.T3.6.1.28.1.2" class="ltx_text" style="font-size:70%;">1-18
SnapKV</span>
</td>
<td id="S4.T3.6.1.28.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.28.2.1" class="ltx_text" style="font-size:70%;">25.47</span></td>
<td id="S4.T3.6.1.28.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.28.3.1" class="ltx_text" style="font-size:70%;">29.57</span></td>
<td id="S4.T3.6.1.28.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.28.4.1" class="ltx_text" style="font-size:70%;">49.33</span></td>
<td id="S4.T3.6.1.28.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.28.5.1" class="ltx_text" style="font-size:70%;">40.90</span></td>
<td id="S4.T3.6.1.28.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.28.6.1" class="ltx_text" style="font-size:70%;">25.53</span></td>
<td id="S4.T3.6.1.28.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.28.7.1" class="ltx_text" style="font-size:70%;">19.01</span></td>
<td id="S4.T3.6.1.28.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.28.8.1" class="ltx_text" style="font-size:70%;">25.94</span></td>
<td id="S4.T3.6.1.28.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.28.9.1" class="ltx_text" style="font-size:70%;">23.89</span></td>
<td id="S4.T3.6.1.28.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.28.10.1" class="ltx_text" style="font-size:70%;">26.21</span></td>
<td id="S4.T3.6.1.28.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.28.11.1" class="ltx_text" style="font-size:70%;">69.50</span></td>
<td id="S4.T3.6.1.28.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.28.12.1" class="ltx_text" style="font-size:70%;">86.48</span></td>
<td id="S4.T3.6.1.28.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.28.13.1" class="ltx_text" style="font-size:70%;">42.10</span></td>
<td id="S4.T3.6.1.28.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.28.14.1" class="ltx_text" style="font-size:70%;">2.98</span></td>
<td id="S4.T3.6.1.28.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.28.15.1" class="ltx_text" style="font-size:70%;">88.56</span></td>
<td id="S4.T3.6.1.28.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.28.16.1" class="ltx_text" style="font-size:70%;">57.19</span></td>
<td id="S4.T3.6.1.28.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.28.17.1" class="ltx_text" style="font-size:70%;">53.60</span></td>
<td id="S4.T3.6.1.28.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.28.18.1" class="ltx_text ltx_font_bold" style="font-size:70%;">41.64</span></td>
</tr>
<tr id="S4.T3.6.1.29" class="ltx_tr">
<td id="S4.T3.6.1.29.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T3.6.1.29.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T3.6.1.29.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T3.6.1.29.1.3" class="ltx_text" style="font-size:70%;">(40)</span>
</td>
<td id="S4.T3.6.1.29.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.29.2.1" class="ltx_text" style="font-size:70%;">25.22</span></td>
<td id="S4.T3.6.1.29.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.29.3.1" class="ltx_text" style="font-size:70%;">30.48</span></td>
<td id="S4.T3.6.1.29.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.29.4.1" class="ltx_text" style="font-size:70%;">48.58</span></td>
<td id="S4.T3.6.1.29.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.29.5.1" class="ltx_text" style="font-size:70%;">41.11</span></td>
<td id="S4.T3.6.1.29.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.29.6.1" class="ltx_text" style="font-size:70%;">25.28</span></td>
<td id="S4.T3.6.1.29.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.29.7.1" class="ltx_text" style="font-size:70%;">18.99</span></td>
<td id="S4.T3.6.1.29.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.29.8.1" class="ltx_text" style="font-size:70%;">25.91</span></td>
<td id="S4.T3.6.1.29.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.29.9.1" class="ltx_text" style="font-size:70%;">24.00</span></td>
<td id="S4.T3.6.1.29.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.29.10.1" class="ltx_text" style="font-size:70%;">26.13</span></td>
<td id="S4.T3.6.1.29.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.29.11.1" class="ltx_text" style="font-size:70%;">70.00</span></td>
<td id="S4.T3.6.1.29.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.29.12.1" class="ltx_text" style="font-size:70%;">86.64</span></td>
<td id="S4.T3.6.1.29.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.29.13.1" class="ltx_text" style="font-size:70%;">43.35</span></td>
<td id="S4.T3.6.1.29.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.29.14.1" class="ltx_text" style="font-size:70%;">2.98</span></td>
<td id="S4.T3.6.1.29.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.29.15.1" class="ltx_text" style="font-size:70%;">86.3</span></td>
<td id="S4.T3.6.1.29.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.29.16.1" class="ltx_text" style="font-size:70%;">56.71</span></td>
<td id="S4.T3.6.1.29.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.29.17.1" class="ltx_text" style="font-size:70%;">54.19</span></td>
<td id="S4.T3.6.1.29.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.29.18.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:70%;">41.62</span></td>
</tr>
<tr id="S4.T3.6.1.30" class="ltx_tr">
<td id="S4.T3.6.1.30.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T3.6.1.30.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T3.6.1.30.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T3.6.1.30.1.3" class="ltx_text" style="font-size:70%;">(50)</span>
</td>
<td id="S4.T3.6.1.30.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.30.2.1" class="ltx_text" style="font-size:70%;">25.63</span></td>
<td id="S4.T3.6.1.30.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.30.3.1" class="ltx_text" style="font-size:70%;">30.08</span></td>
<td id="S4.T3.6.1.30.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.30.4.1" class="ltx_text" style="font-size:70%;">49.41</span></td>
<td id="S4.T3.6.1.30.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.30.5.1" class="ltx_text" style="font-size:70%;">40.59</span></td>
<td id="S4.T3.6.1.30.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.30.6.1" class="ltx_text" style="font-size:70%;">25.13</span></td>
<td id="S4.T3.6.1.30.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.30.7.1" class="ltx_text" style="font-size:70%;">19.58</span></td>
<td id="S4.T3.6.1.30.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.30.8.1" class="ltx_text" style="font-size:70%;">25.47</span></td>
<td id="S4.T3.6.1.30.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.30.9.1" class="ltx_text" style="font-size:70%;">24.23</span></td>
<td id="S4.T3.6.1.30.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.30.10.1" class="ltx_text" style="font-size:70%;">25.92</span></td>
<td id="S4.T3.6.1.30.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.30.11.1" class="ltx_text" style="font-size:70%;">69.5</span></td>
<td id="S4.T3.6.1.30.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.30.12.1" class="ltx_text" style="font-size:70%;">86.67</span></td>
<td id="S4.T3.6.1.30.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.30.13.1" class="ltx_text" style="font-size:70%;">42.31</span></td>
<td id="S4.T3.6.1.30.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.30.14.1" class="ltx_text" style="font-size:70%;">2.74</span></td>
<td id="S4.T3.6.1.30.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.30.15.1" class="ltx_text" style="font-size:70%;">84.78</span></td>
<td id="S4.T3.6.1.30.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.30.16.1" class="ltx_text" style="font-size:70%;">57.43</span></td>
<td id="S4.T3.6.1.30.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.30.17.1" class="ltx_text" style="font-size:70%;">53.59</span></td>
<td id="S4.T3.6.1.30.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.30.18.1" class="ltx_text" style="font-size:70%;">41.44</span></td>
</tr>
<tr id="S4.T3.6.1.31" class="ltx_tr">
<td id="S4.T3.6.1.31.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T3.6.1.31.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T3.6.1.31.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T3.6.1.31.1.3" class="ltx_text" style="font-size:70%;">(60)</span>
</td>
<td id="S4.T3.6.1.31.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.31.2.1" class="ltx_text" style="font-size:70%;">24.69</span></td>
<td id="S4.T3.6.1.31.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.31.3.1" class="ltx_text" style="font-size:70%;">29.3</span></td>
<td id="S4.T3.6.1.31.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.31.4.1" class="ltx_text" style="font-size:70%;">48.90</span></td>
<td id="S4.T3.6.1.31.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.31.5.1" class="ltx_text" style="font-size:70%;">40.44</span></td>
<td id="S4.T3.6.1.31.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.31.6.1" class="ltx_text" style="font-size:70%;">25.33</span></td>
<td id="S4.T3.6.1.31.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.31.7.1" class="ltx_text" style="font-size:70%;">19.58</span></td>
<td id="S4.T3.6.1.31.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.31.8.1" class="ltx_text" style="font-size:70%;">25.23</span></td>
<td id="S4.T3.6.1.31.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.31.9.1" class="ltx_text" style="font-size:70%;">23.6</span></td>
<td id="S4.T3.6.1.31.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.31.10.1" class="ltx_text" style="font-size:70%;">25.25</span></td>
<td id="S4.T3.6.1.31.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.31.11.1" class="ltx_text" style="font-size:70%;">69.00</span></td>
<td id="S4.T3.6.1.31.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.31.12.1" class="ltx_text" style="font-size:70%;">86.85</span></td>
<td id="S4.T3.6.1.31.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.31.13.1" class="ltx_text" style="font-size:70%;">40.86</span></td>
<td id="S4.T3.6.1.31.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.31.14.1" class="ltx_text" style="font-size:70%;">3.19</span></td>
<td id="S4.T3.6.1.31.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.31.15.1" class="ltx_text" style="font-size:70%;">83.70</span></td>
<td id="S4.T3.6.1.31.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.31.16.1" class="ltx_text" style="font-size:70%;">56.3</span></td>
<td id="S4.T3.6.1.31.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.31.17.1" class="ltx_text" style="font-size:70%;">53.30</span></td>
<td id="S4.T3.6.1.31.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.31.18.1" class="ltx_text" style="font-size:70%;">40.97</span></td>
</tr>
<tr id="S4.T3.6.1.32" class="ltx_tr">
<td id="S4.T3.6.1.32.1" class="ltx_td ltx_align_center ltx_border_t" colspan="18"><span id="S4.T3.6.1.32.1.1" class="ltx_text" style="font-size:70%;">KV-size 2048</span></td>
</tr>
<tr id="S4.T3.6.1.33" class="ltx_tr">
<td id="S4.T3.6.1.33.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T3.6.1.33.1.1" class="ltx_text" style="font-size:70%;">H2O</span></td>
<td id="S4.T3.6.1.33.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.33.2.1" class="ltx_text" style="font-size:70%;">25.76</span></td>
<td id="S4.T3.6.1.33.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.33.3.1" class="ltx_text" style="font-size:70%;">31.10</span></td>
<td id="S4.T3.6.1.33.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.33.4.1" class="ltx_text" style="font-size:70%;">49.06</span></td>
<td id="S4.T3.6.1.33.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.33.5.1" class="ltx_text" style="font-size:70%;">40.38</span></td>
<td id="S4.T3.6.1.33.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.33.6.1" class="ltx_text" style="font-size:70%;">26.43</span></td>
<td id="S4.T3.6.1.33.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.33.7.1" class="ltx_text" style="font-size:70%;">16.78</span></td>
<td id="S4.T3.6.1.33.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.33.8.1" class="ltx_text" style="font-size:70%;">27.17</span></td>
<td id="S4.T3.6.1.33.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.33.9.1" class="ltx_text" style="font-size:70%;">23.64</span></td>
<td id="S4.T3.6.1.33.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.33.10.1" class="ltx_text" style="font-size:70%;">26.69</span></td>
<td id="S4.T3.6.1.33.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.33.11.1" class="ltx_text" style="font-size:70%;">55.00</span></td>
<td id="S4.T3.6.1.33.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.33.12.1" class="ltx_text" style="font-size:70%;">86.35</span></td>
<td id="S4.T3.6.1.33.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.33.13.1" class="ltx_text" style="font-size:70%;">42.48</span></td>
<td id="S4.T3.6.1.33.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.33.14.1" class="ltx_text" style="font-size:70%;">2.72</span></td>
<td id="S4.T3.6.1.33.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.33.15.1" class="ltx_text" style="font-size:70%;">86.64</span></td>
<td id="S4.T3.6.1.33.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.33.16.1" class="ltx_text" style="font-size:70%;">56.98</span></td>
<td id="S4.T3.6.1.33.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.33.17.1" class="ltx_text" style="font-size:70%;">53.91</span></td>
<td id="S4.T3.6.1.33.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.33.18.1" class="ltx_text ltx_font_bold" style="font-size:70%;">40.69</span></td>
</tr>
<tr id="S4.T3.6.1.34" class="ltx_tr">
<td id="S4.T3.6.1.34.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T3.6.1.34.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T3.6.1.34.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T3.6.1.34.1.3" class="ltx_text" style="font-size:70%;">(40)</span>
</td>
<td id="S4.T3.6.1.34.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.34.2.1" class="ltx_text" style="font-size:70%;">25.40</span></td>
<td id="S4.T3.6.1.34.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.34.3.1" class="ltx_text" style="font-size:70%;">30.80</span></td>
<td id="S4.T3.6.1.34.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.34.4.1" class="ltx_text" style="font-size:70%;">48.45</span></td>
<td id="S4.T3.6.1.34.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.34.5.1" class="ltx_text" style="font-size:70%;">39.64</span></td>
<td id="S4.T3.6.1.34.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.34.6.1" class="ltx_text" style="font-size:70%;">26.08</span></td>
<td id="S4.T3.6.1.34.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.34.7.1" class="ltx_text" style="font-size:70%;">16.82</span></td>
<td id="S4.T3.6.1.34.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.34.8.1" class="ltx_text" style="font-size:70%;">27.12</span></td>
<td id="S4.T3.6.1.34.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.34.9.1" class="ltx_text" style="font-size:70%;">23.79</span></td>
<td id="S4.T3.6.1.34.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.34.10.1" class="ltx_text" style="font-size:70%;">26.65</span></td>
<td id="S4.T3.6.1.34.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.34.11.1" class="ltx_text" style="font-size:70%;">53.50</span></td>
<td id="S4.T3.6.1.34.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.34.12.1" class="ltx_text" style="font-size:70%;">86.39</span></td>
<td id="S4.T3.6.1.34.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.34.13.1" class="ltx_text" style="font-size:70%;">43.03</span></td>
<td id="S4.T3.6.1.34.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.34.14.1" class="ltx_text" style="font-size:70%;">3.29</span></td>
<td id="S4.T3.6.1.34.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.34.15.1" class="ltx_text" style="font-size:70%;">86.39</span></td>
<td id="S4.T3.6.1.34.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.34.16.1" class="ltx_text" style="font-size:70%;">56.61</span></td>
<td id="S4.T3.6.1.34.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.34.17.1" class="ltx_text" style="font-size:70%;">53.60</span></td>
<td id="S4.T3.6.1.34.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.34.18.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:70%;">40.47</span></td>
</tr>
<tr id="S4.T3.6.1.35" class="ltx_tr">
<td id="S4.T3.6.1.35.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T3.6.1.35.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T3.6.1.35.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T3.6.1.35.1.3" class="ltx_text" style="font-size:70%;">(50)</span>
</td>
<td id="S4.T3.6.1.35.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.35.2.1" class="ltx_text" style="font-size:70%;">25.68</span></td>
<td id="S4.T3.6.1.35.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.35.3.1" class="ltx_text" style="font-size:70%;">31.24</span></td>
<td id="S4.T3.6.1.35.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.35.4.1" class="ltx_text" style="font-size:70%;">48.69</span></td>
<td id="S4.T3.6.1.35.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.35.5.1" class="ltx_text" style="font-size:70%;">39.65</span></td>
<td id="S4.T3.6.1.35.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.35.6.1" class="ltx_text" style="font-size:70%;">25.84</span></td>
<td id="S4.T3.6.1.35.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.35.7.1" class="ltx_text" style="font-size:70%;">16.72</span></td>
<td id="S4.T3.6.1.35.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.35.8.1" class="ltx_text" style="font-size:70%;">26.69</span></td>
<td id="S4.T3.6.1.35.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.35.9.1" class="ltx_text" style="font-size:70%;">23.57</span></td>
<td id="S4.T3.6.1.35.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.35.10.1" class="ltx_text" style="font-size:70%;">26.78</span></td>
<td id="S4.T3.6.1.35.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.35.11.1" class="ltx_text" style="font-size:70%;">52.00</span></td>
<td id="S4.T3.6.1.35.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.35.12.1" class="ltx_text" style="font-size:70%;">86.74</span></td>
<td id="S4.T3.6.1.35.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.35.13.1" class="ltx_text" style="font-size:70%;">42.85</span></td>
<td id="S4.T3.6.1.35.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.35.14.1" class="ltx_text" style="font-size:70%;">4.01</span></td>
<td id="S4.T3.6.1.35.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.35.15.1" class="ltx_text" style="font-size:70%;">83.46</span></td>
<td id="S4.T3.6.1.35.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.35.16.1" class="ltx_text" style="font-size:70%;">57.12</span></td>
<td id="S4.T3.6.1.35.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.35.17.1" class="ltx_text" style="font-size:70%;">53.67</span></td>
<td id="S4.T3.6.1.35.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.35.18.1" class="ltx_text" style="font-size:70%;">40.29</span></td>
</tr>
<tr id="S4.T3.6.1.36" class="ltx_tr">
<td id="S4.T3.6.1.36.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T3.6.1.36.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T3.6.1.36.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T3.6.1.36.1.3" class="ltx_text" style="font-size:70%;">(60)</span>
</td>
<td id="S4.T3.6.1.36.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.36.2.1" class="ltx_text" style="font-size:70%;">25.83</span></td>
<td id="S4.T3.6.1.36.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.36.3.1" class="ltx_text" style="font-size:70%;">31.00</span></td>
<td id="S4.T3.6.1.36.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.36.4.1" class="ltx_text" style="font-size:70%;">48.23</span></td>
<td id="S4.T3.6.1.36.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.36.5.1" class="ltx_text" style="font-size:70%;">38.58</span></td>
<td id="S4.T3.6.1.36.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.36.6.1" class="ltx_text" style="font-size:70%;">25.71</span></td>
<td id="S4.T3.6.1.36.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.36.7.1" class="ltx_text" style="font-size:70%;">16.54</span></td>
<td id="S4.T3.6.1.36.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.36.8.1" class="ltx_text" style="font-size:70%;">26.51</span></td>
<td id="S4.T3.6.1.36.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.36.9.1" class="ltx_text" style="font-size:70%;">23.81</span></td>
<td id="S4.T3.6.1.36.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.36.10.1" class="ltx_text" style="font-size:70%;">26.28</span></td>
<td id="S4.T3.6.1.36.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.36.11.1" class="ltx_text" style="font-size:70%;">50.50</span></td>
<td id="S4.T3.6.1.36.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.36.12.1" class="ltx_text" style="font-size:70%;">86.57</span></td>
<td id="S4.T3.6.1.36.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.36.13.1" class="ltx_text" style="font-size:70%;">42.05</span></td>
<td id="S4.T3.6.1.36.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.36.14.1" class="ltx_text" style="font-size:70%;">3.36</span></td>
<td id="S4.T3.6.1.36.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.36.15.1" class="ltx_text" style="font-size:70%;">82.49</span></td>
<td id="S4.T3.6.1.36.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.36.16.1" class="ltx_text" style="font-size:70%;">56.04</span></td>
<td id="S4.T3.6.1.36.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.36.17.1" class="ltx_text" style="font-size:70%;">52.67</span></td>
<td id="S4.T3.6.1.36.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.36.18.1" class="ltx_text" style="font-size:70%;">39.76</span></td>
</tr>
<tr id="S4.T3.6.1.37" class="ltx_tr">
<td id="S4.T3.6.1.37.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T3.6.1.37.1.1" class="ltx_ERROR undefined">\cdashline</span><span id="S4.T3.6.1.37.1.2" class="ltx_text" style="font-size:70%;">1-18
SnapKV</span>
</td>
<td id="S4.T3.6.1.37.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.37.2.1" class="ltx_text" style="font-size:70%;">25.89</span></td>
<td id="S4.T3.6.1.37.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.37.3.1" class="ltx_text" style="font-size:70%;">32.56</span></td>
<td id="S4.T3.6.1.37.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.37.4.1" class="ltx_text" style="font-size:70%;">48.55</span></td>
<td id="S4.T3.6.1.37.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.37.5.1" class="ltx_text" style="font-size:70%;">41.68</span></td>
<td id="S4.T3.6.1.37.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.37.6.1" class="ltx_text" style="font-size:70%;">27.24</span></td>
<td id="S4.T3.6.1.37.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.37.7.1" class="ltx_text" style="font-size:70%;">18.75</span></td>
<td id="S4.T3.6.1.37.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.37.8.1" class="ltx_text" style="font-size:70%;">28.90</span></td>
<td id="S4.T3.6.1.37.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.37.9.1" class="ltx_text" style="font-size:70%;">24.47</span></td>
<td id="S4.T3.6.1.37.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.37.10.1" class="ltx_text" style="font-size:70%;">26.63</span></td>
<td id="S4.T3.6.1.37.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.37.11.1" class="ltx_text" style="font-size:70%;">70.00</span></td>
<td id="S4.T3.6.1.37.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.37.12.1" class="ltx_text" style="font-size:70%;">86.27</span></td>
<td id="S4.T3.6.1.37.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.37.13.1" class="ltx_text" style="font-size:70%;">42.57</span></td>
<td id="S4.T3.6.1.37.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.37.14.1" class="ltx_text" style="font-size:70%;">3.09</span></td>
<td id="S4.T3.6.1.37.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.37.15.1" class="ltx_text" style="font-size:70%;">86.93</span></td>
<td id="S4.T3.6.1.37.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.37.16.1" class="ltx_text" style="font-size:70%;">57.44</span></td>
<td id="S4.T3.6.1.37.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.37.17.1" class="ltx_text" style="font-size:70%;">53.83</span></td>
<td id="S4.T3.6.1.37.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.37.18.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:70%;">42.18</span></td>
</tr>
<tr id="S4.T3.6.1.38" class="ltx_tr">
<td id="S4.T3.6.1.38.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T3.6.1.38.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T3.6.1.38.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T3.6.1.38.1.3" class="ltx_text" style="font-size:70%;">(40)</span>
</td>
<td id="S4.T3.6.1.38.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.38.2.1" class="ltx_text" style="font-size:70%;">25.77</span></td>
<td id="S4.T3.6.1.38.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.38.3.1" class="ltx_text" style="font-size:70%;">32.67</span></td>
<td id="S4.T3.6.1.38.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.38.4.1" class="ltx_text" style="font-size:70%;">48.70</span></td>
<td id="S4.T3.6.1.38.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.38.5.1" class="ltx_text" style="font-size:70%;">41.06</span></td>
<td id="S4.T3.6.1.38.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.38.6.1" class="ltx_text" style="font-size:70%;">27.07</span></td>
<td id="S4.T3.6.1.38.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.38.7.1" class="ltx_text" style="font-size:70%;">19.14</span></td>
<td id="S4.T3.6.1.38.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.38.8.1" class="ltx_text" style="font-size:70%;">28.91</span></td>
<td id="S4.T3.6.1.38.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.38.9.1" class="ltx_text" style="font-size:70%;">24.37</span></td>
<td id="S4.T3.6.1.38.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.38.10.1" class="ltx_text" style="font-size:70%;">26.88</span></td>
<td id="S4.T3.6.1.38.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.38.11.1" class="ltx_text" style="font-size:70%;">70.00</span></td>
<td id="S4.T3.6.1.38.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.38.12.1" class="ltx_text" style="font-size:70%;">86.37</span></td>
<td id="S4.T3.6.1.38.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.38.13.1" class="ltx_text" style="font-size:70%;">42.75</span></td>
<td id="S4.T3.6.1.38.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.38.14.1" class="ltx_text" style="font-size:70%;">3.61</span></td>
<td id="S4.T3.6.1.38.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.38.15.1" class="ltx_text" style="font-size:70%;">87.38</span></td>
<td id="S4.T3.6.1.38.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.38.16.1" class="ltx_text" style="font-size:70%;">57.21</span></td>
<td id="S4.T3.6.1.38.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.38.17.1" class="ltx_text" style="font-size:70%;">54.44</span></td>
<td id="S4.T3.6.1.38.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.38.18.1" class="ltx_text ltx_font_bold" style="font-size:70%;">42.27</span></td>
</tr>
<tr id="S4.T3.6.1.39" class="ltx_tr">
<td id="S4.T3.6.1.39.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T3.6.1.39.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T3.6.1.39.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T3.6.1.39.1.3" class="ltx_text" style="font-size:70%;">(50)</span>
</td>
<td id="S4.T3.6.1.39.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.39.2.1" class="ltx_text" style="font-size:70%;">26.44</span></td>
<td id="S4.T3.6.1.39.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.39.3.1" class="ltx_text" style="font-size:70%;">32.94</span></td>
<td id="S4.T3.6.1.39.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.39.4.1" class="ltx_text" style="font-size:70%;">49.02</span></td>
<td id="S4.T3.6.1.39.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.39.5.1" class="ltx_text" style="font-size:70%;">40.86</span></td>
<td id="S4.T3.6.1.39.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.39.6.1" class="ltx_text" style="font-size:70%;">26.84</span></td>
<td id="S4.T3.6.1.39.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.39.7.1" class="ltx_text" style="font-size:70%;">19.49</span></td>
<td id="S4.T3.6.1.39.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.39.8.1" class="ltx_text" style="font-size:70%;">28.46</span></td>
<td id="S4.T3.6.1.39.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.39.9.1" class="ltx_text" style="font-size:70%;">24.51</span></td>
<td id="S4.T3.6.1.39.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.39.10.1" class="ltx_text" style="font-size:70%;">26.72</span></td>
<td id="S4.T3.6.1.39.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.39.11.1" class="ltx_text" style="font-size:70%;">70.00</span></td>
<td id="S4.T3.6.1.39.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.39.12.1" class="ltx_text" style="font-size:70%;">86.50</span></td>
<td id="S4.T3.6.1.39.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.39.13.1" class="ltx_text" style="font-size:70%;">41.75</span></td>
<td id="S4.T3.6.1.39.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.39.14.1" class="ltx_text" style="font-size:70%;">2.78</span></td>
<td id="S4.T3.6.1.39.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.39.15.1" class="ltx_text" style="font-size:70%;">84.70</span></td>
<td id="S4.T3.6.1.39.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.39.16.1" class="ltx_text" style="font-size:70%;">56.47</span></td>
<td id="S4.T3.6.1.39.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T3.6.1.39.17.1" class="ltx_text" style="font-size:70%;">54.15</span></td>
<td id="S4.T3.6.1.39.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T3.6.1.39.18.1" class="ltx_text" style="font-size:70%;">41.98</span></td>
</tr>
<tr id="S4.T3.6.1.40" class="ltx_tr">
<td id="S4.T3.6.1.40.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">
<span id="S4.T3.6.1.40.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T3.6.1.40.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T3.6.1.40.1.3" class="ltx_text" style="font-size:70%;">(60)</span>
</td>
<td id="S4.T3.6.1.40.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T3.6.1.40.2.1" class="ltx_text" style="font-size:70%;">26.00</span></td>
<td id="S4.T3.6.1.40.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T3.6.1.40.3.1" class="ltx_text" style="font-size:70%;">32.53</span></td>
<td id="S4.T3.6.1.40.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T3.6.1.40.4.1" class="ltx_text" style="font-size:70%;">48.73</span></td>
<td id="S4.T3.6.1.40.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T3.6.1.40.5.1" class="ltx_text" style="font-size:70%;">40.95</span></td>
<td id="S4.T3.6.1.40.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T3.6.1.40.6.1" class="ltx_text" style="font-size:70%;">26.77</span></td>
<td id="S4.T3.6.1.40.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T3.6.1.40.7.1" class="ltx_text" style="font-size:70%;">18.92</span></td>
<td id="S4.T3.6.1.40.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T3.6.1.40.8.1" class="ltx_text" style="font-size:70%;">27.40</span></td>
<td id="S4.T3.6.1.40.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T3.6.1.40.9.1" class="ltx_text" style="font-size:70%;">23.97</span></td>
<td id="S4.T3.6.1.40.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T3.6.1.40.10.1" class="ltx_text" style="font-size:70%;">26.37</span></td>
<td id="S4.T3.6.1.40.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T3.6.1.40.11.1" class="ltx_text" style="font-size:70%;">70.00</span></td>
<td id="S4.T3.6.1.40.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T3.6.1.40.12.1" class="ltx_text" style="font-size:70%;">86.45</span></td>
<td id="S4.T3.6.1.40.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T3.6.1.40.13.1" class="ltx_text" style="font-size:70%;">41.12</span></td>
<td id="S4.T3.6.1.40.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T3.6.1.40.14.1" class="ltx_text" style="font-size:70%;">3.31</span></td>
<td id="S4.T3.6.1.40.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T3.6.1.40.15.1" class="ltx_text" style="font-size:70%;">82.24</span></td>
<td id="S4.T3.6.1.40.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T3.6.1.40.16.1" class="ltx_text" style="font-size:70%;">56.01</span></td>
<td id="S4.T3.6.1.40.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T3.6.1.40.17.1" class="ltx_text" style="font-size:70%;">53.53</span></td>
<td id="S4.T3.6.1.40.18" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_bb"><span id="S4.T3.6.1.40.18.1" class="ltx_text" style="font-size:70%;">41.52</span></td>
</tr>
</table>
</span></div>
</figure>
</section>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Results on LongBench</h3>

<div id="S4.SS2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS2.p1.11" class="ltx_p">Tables <a href="#S4.T2" title="Table 2 ‣ Baseline Approaches. ‣ 4.1 Settings ‣ 4 Experiment Results ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> and  <a href="#S4.T3" title="Table 3 ‣ Implementation Details. ‣ 4.1 Settings ‣ 4 Experiment Results ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> present the results of KV compression methods and KV compression methods integrated with our channel pruning method of Key cache (<span id="S4.SS2.p1.11.1" class="ltx_text ltx_font_bold">ThinK</span>) over two different base LLMs across various KV-sizes on LongBench.
The following observations can be drawn:
(1) Our method can further prune the channels of the key cache after compressing the KV cache with H2O and SnapKV. For the base model LLaMA3-8B, our method reduces memory usage and slightly improves performance for both H2O and SnapKV. For the base model Mistral-7B, our method reduces memory with only a slight drop in performance in some cases.
(2) Comparing the results of SnapKV or H2O integrated with <span id="S4.SS2.p1.11.2" class="ltx_text ltx_font_bold">ThinK</span> in Table <a href="#S4.T2" title="Table 2 ‣ Baseline Approaches. ‣ 4.1 Settings ‣ 4 Experiment Results ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> to SnapKV or H2O integrated with <math id="S4.SS2.p1.1.m1.1" class="ltx_Math" alttext="l_{1}" display="inline"><semantics id="S4.SS2.p1.1.m1.1a"><msub id="S4.SS2.p1.1.m1.1.1" xref="S4.SS2.p1.1.m1.1.1.cmml"><mi id="S4.SS2.p1.1.m1.1.1.2" xref="S4.SS2.p1.1.m1.1.1.2.cmml">l</mi><mn id="S4.SS2.p1.1.m1.1.1.3" xref="S4.SS2.p1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.1.m1.1b"><apply id="S4.SS2.p1.1.m1.1.1.cmml" xref="S4.SS2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.1.m1.1.1.1.cmml" xref="S4.SS2.p1.1.m1.1.1">subscript</csymbol><ci id="S4.SS2.p1.1.m1.1.1.2.cmml" xref="S4.SS2.p1.1.m1.1.1.2">𝑙</ci><cn type="integer" id="S4.SS2.p1.1.m1.1.1.3.cmml" xref="S4.SS2.p1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.1.m1.1c">l_{1}</annotation></semantics></math> or <math id="S4.SS2.p1.2.m2.1" class="ltx_Math" alttext="l_{2}" display="inline"><semantics id="S4.SS2.p1.2.m2.1a"><msub id="S4.SS2.p1.2.m2.1.1" xref="S4.SS2.p1.2.m2.1.1.cmml"><mi id="S4.SS2.p1.2.m2.1.1.2" xref="S4.SS2.p1.2.m2.1.1.2.cmml">l</mi><mn id="S4.SS2.p1.2.m2.1.1.3" xref="S4.SS2.p1.2.m2.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.2.m2.1b"><apply id="S4.SS2.p1.2.m2.1.1.cmml" xref="S4.SS2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.2.m2.1.1.1.cmml" xref="S4.SS2.p1.2.m2.1.1">subscript</csymbol><ci id="S4.SS2.p1.2.m2.1.1.2.cmml" xref="S4.SS2.p1.2.m2.1.1.2">𝑙</ci><cn type="integer" id="S4.SS2.p1.2.m2.1.1.3.cmml" xref="S4.SS2.p1.2.m2.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.2.m2.1c">l_{2}</annotation></semantics></math> norm shown in Table <a href="#S3.T1" title="Table 1 ‣ 3.1 Preliminary Study of KV Cache Optimization ‣ 3 ThinK ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, our proposed query-based channel pruning shows better performance when the pruning ratio is <math id="S4.SS2.p1.3.m3.1" class="ltx_Math" alttext="40\%" display="inline"><semantics id="S4.SS2.p1.3.m3.1a"><mrow id="S4.SS2.p1.3.m3.1.1" xref="S4.SS2.p1.3.m3.1.1.cmml"><mn id="S4.SS2.p1.3.m3.1.1.2" xref="S4.SS2.p1.3.m3.1.1.2.cmml">40</mn><mo id="S4.SS2.p1.3.m3.1.1.1" xref="S4.SS2.p1.3.m3.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.3.m3.1b"><apply id="S4.SS2.p1.3.m3.1.1.cmml" xref="S4.SS2.p1.3.m3.1.1"><csymbol cd="latexml" id="S4.SS2.p1.3.m3.1.1.1.cmml" xref="S4.SS2.p1.3.m3.1.1.1">percent</csymbol><cn type="integer" id="S4.SS2.p1.3.m3.1.1.2.cmml" xref="S4.SS2.p1.3.m3.1.1.2">40</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.3.m3.1c">40\%</annotation></semantics></math> compared to no pruning. However, <math id="S4.SS2.p1.4.m4.1" class="ltx_Math" alttext="l_{1}" display="inline"><semantics id="S4.SS2.p1.4.m4.1a"><msub id="S4.SS2.p1.4.m4.1.1" xref="S4.SS2.p1.4.m4.1.1.cmml"><mi id="S4.SS2.p1.4.m4.1.1.2" xref="S4.SS2.p1.4.m4.1.1.2.cmml">l</mi><mn id="S4.SS2.p1.4.m4.1.1.3" xref="S4.SS2.p1.4.m4.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.4.m4.1b"><apply id="S4.SS2.p1.4.m4.1.1.cmml" xref="S4.SS2.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.4.m4.1.1.1.cmml" xref="S4.SS2.p1.4.m4.1.1">subscript</csymbol><ci id="S4.SS2.p1.4.m4.1.1.2.cmml" xref="S4.SS2.p1.4.m4.1.1.2">𝑙</ci><cn type="integer" id="S4.SS2.p1.4.m4.1.1.3.cmml" xref="S4.SS2.p1.4.m4.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.4.m4.1c">l_{1}</annotation></semantics></math> and <math id="S4.SS2.p1.5.m5.1" class="ltx_Math" alttext="l_{2}" display="inline"><semantics id="S4.SS2.p1.5.m5.1a"><msub id="S4.SS2.p1.5.m5.1.1" xref="S4.SS2.p1.5.m5.1.1.cmml"><mi id="S4.SS2.p1.5.m5.1.1.2" xref="S4.SS2.p1.5.m5.1.1.2.cmml">l</mi><mn id="S4.SS2.p1.5.m5.1.1.3" xref="S4.SS2.p1.5.m5.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.5.m5.1b"><apply id="S4.SS2.p1.5.m5.1.1.cmml" xref="S4.SS2.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.5.m5.1.1.1.cmml" xref="S4.SS2.p1.5.m5.1.1">subscript</csymbol><ci id="S4.SS2.p1.5.m5.1.1.2.cmml" xref="S4.SS2.p1.5.m5.1.1.2">𝑙</ci><cn type="integer" id="S4.SS2.p1.5.m5.1.1.3.cmml" xref="S4.SS2.p1.5.m5.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.5.m5.1c">l_{2}</annotation></semantics></math> norm methods cannot outperform the KV compression methods without pruning.
(3) When the pruning ratio is small, performance tends to be better than when the pruning ratio is large.
(4) When the KV-size is increased from <math id="S4.SS2.p1.6.m6.1" class="ltx_Math" alttext="128" display="inline"><semantics id="S4.SS2.p1.6.m6.1a"><mn id="S4.SS2.p1.6.m6.1.1" xref="S4.SS2.p1.6.m6.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.6.m6.1b"><cn type="integer" id="S4.SS2.p1.6.m6.1.1.cmml" xref="S4.SS2.p1.6.m6.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.6.m6.1c">128</annotation></semantics></math> to <math id="S4.SS2.p1.7.m7.1" class="ltx_Math" alttext="2048" display="inline"><semantics id="S4.SS2.p1.7.m7.1a"><mn id="S4.SS2.p1.7.m7.1.1" xref="S4.SS2.p1.7.m7.1.1.cmml">2048</mn><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.7.m7.1b"><cn type="integer" id="S4.SS2.p1.7.m7.1.1.cmml" xref="S4.SS2.p1.7.m7.1.1">2048</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.7.m7.1c">2048</annotation></semantics></math>, the performance of our channel pruning method improves. Notably, with a KV cache size of <math id="S4.SS2.p1.8.m8.1" class="ltx_Math" alttext="2048" display="inline"><semantics id="S4.SS2.p1.8.m8.1a"><mn id="S4.SS2.p1.8.m8.1.1" xref="S4.SS2.p1.8.m8.1.1.cmml">2048</mn><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.8.m8.1b"><cn type="integer" id="S4.SS2.p1.8.m8.1.1.cmml" xref="S4.SS2.p1.8.m8.1.1">2048</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.8.m8.1c">2048</annotation></semantics></math> and a pruning ratio of <math id="S4.SS2.p1.9.m9.1" class="ltx_Math" alttext="40\%" display="inline"><semantics id="S4.SS2.p1.9.m9.1a"><mrow id="S4.SS2.p1.9.m9.1.1" xref="S4.SS2.p1.9.m9.1.1.cmml"><mn id="S4.SS2.p1.9.m9.1.1.2" xref="S4.SS2.p1.9.m9.1.1.2.cmml">40</mn><mo id="S4.SS2.p1.9.m9.1.1.1" xref="S4.SS2.p1.9.m9.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.9.m9.1b"><apply id="S4.SS2.p1.9.m9.1.1.cmml" xref="S4.SS2.p1.9.m9.1.1"><csymbol cd="latexml" id="S4.SS2.p1.9.m9.1.1.1.cmml" xref="S4.SS2.p1.9.m9.1.1.1">percent</csymbol><cn type="integer" id="S4.SS2.p1.9.m9.1.1.2.cmml" xref="S4.SS2.p1.9.m9.1.1.2">40</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.9.m9.1c">40\%</annotation></semantics></math>, our method can even outperform the LLaMA3-8B with a full KV cache.
The above observations indicate that our method is agnostic to existing KV cache compression methods and can further improve their performance and memory reduction. Additionally, query-based pruning is more effective than <math id="S4.SS2.p1.10.m10.1" class="ltx_Math" alttext="l_{1}" display="inline"><semantics id="S4.SS2.p1.10.m10.1a"><msub id="S4.SS2.p1.10.m10.1.1" xref="S4.SS2.p1.10.m10.1.1.cmml"><mi id="S4.SS2.p1.10.m10.1.1.2" xref="S4.SS2.p1.10.m10.1.1.2.cmml">l</mi><mn id="S4.SS2.p1.10.m10.1.1.3" xref="S4.SS2.p1.10.m10.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.10.m10.1b"><apply id="S4.SS2.p1.10.m10.1.1.cmml" xref="S4.SS2.p1.10.m10.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.10.m10.1.1.1.cmml" xref="S4.SS2.p1.10.m10.1.1">subscript</csymbol><ci id="S4.SS2.p1.10.m10.1.1.2.cmml" xref="S4.SS2.p1.10.m10.1.1.2">𝑙</ci><cn type="integer" id="S4.SS2.p1.10.m10.1.1.3.cmml" xref="S4.SS2.p1.10.m10.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.10.m10.1c">l_{1}</annotation></semantics></math> and <math id="S4.SS2.p1.11.m11.1" class="ltx_Math" alttext="l_{2}" display="inline"><semantics id="S4.SS2.p1.11.m11.1a"><msub id="S4.SS2.p1.11.m11.1.1" xref="S4.SS2.p1.11.m11.1.1.cmml"><mi id="S4.SS2.p1.11.m11.1.1.2" xref="S4.SS2.p1.11.m11.1.1.2.cmml">l</mi><mn id="S4.SS2.p1.11.m11.1.1.3" xref="S4.SS2.p1.11.m11.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.11.m11.1b"><apply id="S4.SS2.p1.11.m11.1.1.cmml" xref="S4.SS2.p1.11.m11.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.11.m11.1.1.1.cmml" xref="S4.SS2.p1.11.m11.1.1">subscript</csymbol><ci id="S4.SS2.p1.11.m11.1.1.2.cmml" xref="S4.SS2.p1.11.m11.1.1.2">𝑙</ci><cn type="integer" id="S4.SS2.p1.11.m11.1.1.3.cmml" xref="S4.SS2.p1.11.m11.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.11.m11.1c">l_{2}</annotation></semantics></math> norm for channel pruning in LLMs.</p>
</div>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Results on Needle-in-a-Haystack</h3>

<div id="S4.SS3.p1" class="ltx_para ltx_noindent">
<p id="S4.SS3.p1.8" class="ltx_p">Table <a href="#S4.T4" title="Table 4 ‣ 4.3 Results on Needle-in-a-Haystack ‣ 4 Experiment Results ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> presents the results of the Needle-in-a-Haystack test based on the SnapKV <cite class="ltx_cite ltx_citemacro_citep">(Li et al.,, <a href="#bib.bib29" title="" class="ltx_ref">2024</a>)</cite> approach with KV cache size of input prompt from <math id="S4.SS3.p1.1.m1.1" class="ltx_Math" alttext="128" display="inline"><semantics id="S4.SS3.p1.1.m1.1a"><mn id="S4.SS3.p1.1.m1.1.1" xref="S4.SS3.p1.1.m1.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.1.m1.1b"><cn type="integer" id="S4.SS3.p1.1.m1.1.1.cmml" xref="S4.SS3.p1.1.m1.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.1.m1.1c">128</annotation></semantics></math> to <math id="S4.SS3.p1.2.m2.1" class="ltx_Math" alttext="2048" display="inline"><semantics id="S4.SS3.p1.2.m2.1a"><mn id="S4.SS3.p1.2.m2.1.1" xref="S4.SS3.p1.2.m2.1.1.cmml">2048</mn><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.2.m2.1b"><cn type="integer" id="S4.SS3.p1.2.m2.1.1.cmml" xref="S4.SS3.p1.2.m2.1.1">2048</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.2.m2.1c">2048</annotation></semantics></math>.
With a modest pruning ratio <math id="S4.SS3.p1.3.m3.1" class="ltx_Math" alttext="\lambda=40" display="inline"><semantics id="S4.SS3.p1.3.m3.1a"><mrow id="S4.SS3.p1.3.m3.1.1" xref="S4.SS3.p1.3.m3.1.1.cmml"><mi id="S4.SS3.p1.3.m3.1.1.2" xref="S4.SS3.p1.3.m3.1.1.2.cmml">λ</mi><mo id="S4.SS3.p1.3.m3.1.1.1" xref="S4.SS3.p1.3.m3.1.1.1.cmml">=</mo><mn id="S4.SS3.p1.3.m3.1.1.3" xref="S4.SS3.p1.3.m3.1.1.3.cmml">40</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.3.m3.1b"><apply id="S4.SS3.p1.3.m3.1.1.cmml" xref="S4.SS3.p1.3.m3.1.1"><eq id="S4.SS3.p1.3.m3.1.1.1.cmml" xref="S4.SS3.p1.3.m3.1.1.1"></eq><ci id="S4.SS3.p1.3.m3.1.1.2.cmml" xref="S4.SS3.p1.3.m3.1.1.2">𝜆</ci><cn type="integer" id="S4.SS3.p1.3.m3.1.1.3.cmml" xref="S4.SS3.p1.3.m3.1.1.3">40</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.3.m3.1c">\lambda=40</annotation></semantics></math>%, the accuracy of <span id="S4.SS3.p1.8.1" class="ltx_text ltx_font_bold">ThinK</span> is consistently better or comparable than the original SnapKV across both LLaMA3 and Mistral with different KV cache sizes.
Such comparisons demonstrate that the proposed query-driven channel pruning can effectively retain the informative channels and discard the noisy ones.
With a pruning ratio of <math id="S4.SS3.p1.4.m4.1" class="ltx_Math" alttext="\lambda\geq 50" display="inline"><semantics id="S4.SS3.p1.4.m4.1a"><mrow id="S4.SS3.p1.4.m4.1.1" xref="S4.SS3.p1.4.m4.1.1.cmml"><mi id="S4.SS3.p1.4.m4.1.1.2" xref="S4.SS3.p1.4.m4.1.1.2.cmml">λ</mi><mo id="S4.SS3.p1.4.m4.1.1.1" xref="S4.SS3.p1.4.m4.1.1.1.cmml">≥</mo><mn id="S4.SS3.p1.4.m4.1.1.3" xref="S4.SS3.p1.4.m4.1.1.3.cmml">50</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.4.m4.1b"><apply id="S4.SS3.p1.4.m4.1.1.cmml" xref="S4.SS3.p1.4.m4.1.1"><geq id="S4.SS3.p1.4.m4.1.1.1.cmml" xref="S4.SS3.p1.4.m4.1.1.1"></geq><ci id="S4.SS3.p1.4.m4.1.1.2.cmml" xref="S4.SS3.p1.4.m4.1.1.2">𝜆</ci><cn type="integer" id="S4.SS3.p1.4.m4.1.1.3.cmml" xref="S4.SS3.p1.4.m4.1.1.3">50</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.4.m4.1c">\lambda\geq 50</annotation></semantics></math>%, we observe an accuracy drop with a small KV-cache size, particularly for <math id="S4.SS3.p1.5.m5.1" class="ltx_Math" alttext="128" display="inline"><semantics id="S4.SS3.p1.5.m5.1a"><mn id="S4.SS3.p1.5.m5.1.1" xref="S4.SS3.p1.5.m5.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.5.m5.1b"><cn type="integer" id="S4.SS3.p1.5.m5.1.1.cmml" xref="S4.SS3.p1.5.m5.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.5.m5.1c">128</annotation></semantics></math> and <math id="S4.SS3.p1.6.m6.1" class="ltx_Math" alttext="512" display="inline"><semantics id="S4.SS3.p1.6.m6.1a"><mn id="S4.SS3.p1.6.m6.1.1" xref="S4.SS3.p1.6.m6.1.1.cmml">512</mn><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.6.m6.1b"><cn type="integer" id="S4.SS3.p1.6.m6.1.1.cmml" xref="S4.SS3.p1.6.m6.1.1">512</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.6.m6.1c">512</annotation></semantics></math> across both LLaMA3 and Mistral.
However, <span id="S4.SS3.p1.8.2" class="ltx_text ltx_font_bold">ThinK</span> is still able to obtain comparable performance with SnapKV when we have a larger KV cache size (i.e., <math id="S4.SS3.p1.7.m7.1" class="ltx_Math" alttext="1024" display="inline"><semantics id="S4.SS3.p1.7.m7.1a"><mn id="S4.SS3.p1.7.m7.1.1" xref="S4.SS3.p1.7.m7.1.1.cmml">1024</mn><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.7.m7.1b"><cn type="integer" id="S4.SS3.p1.7.m7.1.1.cmml" xref="S4.SS3.p1.7.m7.1.1">1024</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.7.m7.1c">1024</annotation></semantics></math> and <math id="S4.SS3.p1.8.m8.1" class="ltx_Math" alttext="2048" display="inline"><semantics id="S4.SS3.p1.8.m8.1a"><mn id="S4.SS3.p1.8.m8.1.1" xref="S4.SS3.p1.8.m8.1.1.cmml">2048</mn><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.8.m8.1b"><cn type="integer" id="S4.SS3.p1.8.m8.1.1.cmml" xref="S4.SS3.p1.8.m8.1.1">2048</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.8.m8.1c">2048</annotation></semantics></math>).
Intuitively, a larger pruning ratio with a smaller KV cache size could tend to lose more important information compared to a larger KV cache size.
In addition, the performance on larger KV cache size also suggests <span id="S4.SS3.p1.8.3" class="ltx_text ltx_font_bold">ThinK</span> is robust for long context retrieval tasks.</p>
</div>
<div id="S4.SS3.p2" class="ltx_para ltx_noindent">
<p id="S4.SS3.p2.9" class="ltx_p">Figure <a href="#S4.F6" title="Figure 6 ‣ Pruning Channels of Both Key and Value Cache. ‣ 4.4 Ablation Studies ‣ 4 Experiment Results ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> (a)-(d) visualize the Needle-in-a-Haystack test accuracy against different token lengths and depth percent.
The KV cache size is configured to <math id="S4.SS3.p2.1.m1.1" class="ltx_Math" alttext="128" display="inline"><semantics id="S4.SS3.p2.1.m1.1a"><mn id="S4.SS3.p2.1.m1.1.1" xref="S4.SS3.p2.1.m1.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="S4.SS3.p2.1.m1.1b"><cn type="integer" id="S4.SS3.p2.1.m1.1.1.cmml" xref="S4.SS3.p2.1.m1.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p2.1.m1.1c">128</annotation></semantics></math> and <math id="S4.SS3.p2.2.m2.1" class="ltx_Math" alttext="1024" display="inline"><semantics id="S4.SS3.p2.2.m2.1a"><mn id="S4.SS3.p2.2.m2.1.1" xref="S4.SS3.p2.2.m2.1.1.cmml">1024</mn><annotation-xml encoding="MathML-Content" id="S4.SS3.p2.2.m2.1b"><cn type="integer" id="S4.SS3.p2.2.m2.1.1.cmml" xref="S4.SS3.p2.2.m2.1.1">1024</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p2.2.m2.1c">1024</annotation></semantics></math>, with pruning ratio <math id="S4.SS3.p2.3.m3.1" class="ltx_Math" alttext="\lambda" display="inline"><semantics id="S4.SS3.p2.3.m3.1a"><mi id="S4.SS3.p2.3.m3.1.1" xref="S4.SS3.p2.3.m3.1.1.cmml">λ</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p2.3.m3.1b"><ci id="S4.SS3.p2.3.m3.1.1.cmml" xref="S4.SS3.p2.3.m3.1.1">𝜆</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p2.3.m3.1c">\lambda</annotation></semantics></math> equal to <math id="S4.SS3.p2.4.m4.1" class="ltx_Math" alttext="40" display="inline"><semantics id="S4.SS3.p2.4.m4.1a"><mn id="S4.SS3.p2.4.m4.1.1" xref="S4.SS3.p2.4.m4.1.1.cmml">40</mn><annotation-xml encoding="MathML-Content" id="S4.SS3.p2.4.m4.1b"><cn type="integer" id="S4.SS3.p2.4.m4.1.1.cmml" xref="S4.SS3.p2.4.m4.1.1">40</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p2.4.m4.1c">40</annotation></semantics></math>% and <math id="S4.SS3.p2.5.m5.1" class="ltx_Math" alttext="50" display="inline"><semantics id="S4.SS3.p2.5.m5.1a"><mn id="S4.SS3.p2.5.m5.1.1" xref="S4.SS3.p2.5.m5.1.1.cmml">50</mn><annotation-xml encoding="MathML-Content" id="S4.SS3.p2.5.m5.1b"><cn type="integer" id="S4.SS3.p2.5.m5.1.1.cmml" xref="S4.SS3.p2.5.m5.1.1">50</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p2.5.m5.1c">50</annotation></semantics></math>%, respectively.
First, <span id="S4.SS3.p2.9.1" class="ltx_text ltx_font_bold">ThinK</span> can maintain the original retrieval capability of SnapKV though the overall numerical accuracy is different (i.e., <math id="S4.SS3.p2.6.m6.1" class="ltx_Math" alttext="77.8" display="inline"><semantics id="S4.SS3.p2.6.m6.1a"><mn id="S4.SS3.p2.6.m6.1.1" xref="S4.SS3.p2.6.m6.1.1.cmml">77.8</mn><annotation-xml encoding="MathML-Content" id="S4.SS3.p2.6.m6.1b"><cn type="float" id="S4.SS3.p2.6.m6.1.1.cmml" xref="S4.SS3.p2.6.m6.1.1">77.8</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p2.6.m6.1c">77.8</annotation></semantics></math> vs <math id="S4.SS3.p2.7.m7.1" class="ltx_Math" alttext="78.6" display="inline"><semantics id="S4.SS3.p2.7.m7.1a"><mn id="S4.SS3.p2.7.m7.1.1" xref="S4.SS3.p2.7.m7.1.1.cmml">78.6</mn><annotation-xml encoding="MathML-Content" id="S4.SS3.p2.7.m7.1b"><cn type="float" id="S4.SS3.p2.7.m7.1.1.cmml" xref="S4.SS3.p2.7.m7.1.1">78.6</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p2.7.m7.1c">78.6</annotation></semantics></math> and <math id="S4.SS3.p2.8.m8.1" class="ltx_Math" alttext="90.4" display="inline"><semantics id="S4.SS3.p2.8.m8.1a"><mn id="S4.SS3.p2.8.m8.1.1" xref="S4.SS3.p2.8.m8.1.1.cmml">90.4</mn><annotation-xml encoding="MathML-Content" id="S4.SS3.p2.8.m8.1b"><cn type="float" id="S4.SS3.p2.8.m8.1.1.cmml" xref="S4.SS3.p2.8.m8.1.1">90.4</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p2.8.m8.1c">90.4</annotation></semantics></math> vs <math id="S4.SS3.p2.9.m9.1" class="ltx_Math" alttext="90.6" display="inline"><semantics id="S4.SS3.p2.9.m9.1a"><mn id="S4.SS3.p2.9.m9.1.1" xref="S4.SS3.p2.9.m9.1.1.cmml">90.6</mn><annotation-xml encoding="MathML-Content" id="S4.SS3.p2.9.m9.1b"><cn type="float" id="S4.SS3.p2.9.m9.1.1.cmml" xref="S4.SS3.p2.9.m9.1.1">90.6</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p2.9.m9.1c">90.6</annotation></semantics></math>).
Because <span id="S4.SS3.p2.9.2" class="ltx_text ltx_font_bold">ThinK</span>achieves the same accuracy at most of the entries with different token limits and depths as Snap KV.
In addition, <span id="S4.SS3.p2.9.3" class="ltx_text ltx_font_bold">ThinK</span> also successfully retrieves some “needles” where SnapKV fails, leading to higher overall accuracy.
The visualization results further demonstrate the robustness of <span id="S4.SS3.p2.9.4" class="ltx_text ltx_font_bold">ThinK</span> in a fine-grained perspective and <span id="S4.SS3.p2.9.5" class="ltx_text ltx_font_bold">ThinK</span> can enhance the original quantization approach.</p>
</div>
<figure id="S4.T4" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table"><span id="S4.T4.5.1.1" class="ltx_text" style="font-size:129%;">Table 4</span>: </span><span id="S4.T4.6.2" class="ltx_text" style="font-size:129%;">Needles-in-a-Haystack Test Results</span></figcaption>
<div id="S4.T4.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:233.6pt;height:162pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(0.0pt,0.0pt) scale(1.0,1.0) ;">
<table id="S4.T4.1.1" class="ltx_tabular ltx_align_middle">
<tr id="S4.T4.1.1.1" class="ltx_tr">
<td id="S4.T4.1.1.1.2" class="ltx_td ltx_align_left ltx_border_tt" rowspan="2"><span id="S4.T4.1.1.1.2.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Model</span></td>
<td id="S4.T4.1.1.1.3" class="ltx_td ltx_align_left ltx_border_tt" rowspan="2"><span id="S4.T4.1.1.1.3.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Method</span></td>
<td id="S4.T4.1.1.1.1" class="ltx_td ltx_align_center ltx_border_tt" rowspan="2"><span id="S4.T4.1.1.1.1.1" class="ltx_text" style="font-size:70%;"><math id="S4.T4.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\bm{\lambda}" display="inline"><semantics id="S4.T4.1.1.1.1.1.m1.1a"><mi id="S4.T4.1.1.1.1.1.m1.1.1" xref="S4.T4.1.1.1.1.1.m1.1.1.cmml">𝝀</mi><annotation-xml encoding="MathML-Content" id="S4.T4.1.1.1.1.1.m1.1b"><ci id="S4.T4.1.1.1.1.1.m1.1.1.cmml" xref="S4.T4.1.1.1.1.1.m1.1.1">𝝀</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T4.1.1.1.1.1.m1.1c">\bm{\lambda}</annotation></semantics></math></span></td>
<td id="S4.T4.1.1.1.4" class="ltx_td ltx_align_center ltx_border_tt" colspan="4"><span id="S4.T4.1.1.1.4.1" class="ltx_text ltx_font_bold" style="font-size:70%;">KV-size</span></td>
</tr>
<tr id="S4.T4.1.1.2" class="ltx_tr">
<td id="S4.T4.1.1.2.1" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T4.1.1.2.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">128</span></td>
<td id="S4.T4.1.1.2.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T4.1.1.2.2.1" class="ltx_text ltx_font_bold" style="font-size:70%;">512</span></td>
<td id="S4.T4.1.1.2.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T4.1.1.2.3.1" class="ltx_text ltx_font_bold" style="font-size:70%;">1024</span></td>
<td id="S4.T4.1.1.2.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T4.1.1.2.4.1" class="ltx_text ltx_font_bold" style="font-size:70%;">2048</span></td>
</tr>
<tr id="S4.T4.1.1.3" class="ltx_tr">
<td id="S4.T4.1.1.3.1" class="ltx_td ltx_align_left ltx_border_t" rowspan="3"><span id="S4.T4.1.1.3.1.1" class="ltx_text" style="font-size:70%;">LLaMA3-8B</span></td>
<td id="S4.T4.1.1.3.2" class="ltx_td ltx_align_left ltx_border_t"><span id="S4.T4.1.1.3.2.1" class="ltx_text" style="font-size:70%;">SnapKV</span></td>
<td id="S4.T4.1.1.3.3" class="ltx_td ltx_align_center ltx_border_t">
<span id="S4.T4.1.1.3.3.1" class="ltx_text" style="font-size:70%;color:#FFFFFF;">0</span><span id="S4.T4.1.1.3.3.2" class="ltx_text" style="font-size:70%;">0</span>
</td>
<td id="S4.T4.1.1.3.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T4.1.1.3.4.1" class="ltx_text" style="font-size:70%;">79.6</span></td>
<td id="S4.T4.1.1.3.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T4.1.1.3.5.1" class="ltx_text" style="font-size:70%;">90.2</span></td>
<td id="S4.T4.1.1.3.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T4.1.1.3.6.1" class="ltx_text" style="font-size:70%;">91.2</span></td>
<td id="S4.T4.1.1.3.7" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T4.1.1.3.7.1" class="ltx_text" style="font-size:70%;">91.7</span></td>
</tr>
<tr id="S4.T4.1.1.4" class="ltx_tr">
<td id="S4.T4.1.1.4.1" class="ltx_td ltx_align_left">
<span id="S4.T4.1.1.4.1.1" class="ltx_text" style="font-size:70%;">SnapKV+</span><span id="S4.T4.1.1.4.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span>
</td>
<td id="S4.T4.1.1.4.2" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.4.2.1" class="ltx_text" style="font-size:70%;">40</span></td>
<td id="S4.T4.1.1.4.3" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.4.3.1" class="ltx_text" style="font-size:70%;">79.6</span></td>
<td id="S4.T4.1.1.4.4" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.4.4.1" class="ltx_text" style="font-size:70%;">90.3</span></td>
<td id="S4.T4.1.1.4.5" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.4.5.1" class="ltx_text" style="font-size:70%;">91.2</span></td>
<td id="S4.T4.1.1.4.6" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.4.6.1" class="ltx_text" style="font-size:70%;">91.7</span></td>
</tr>
<tr id="S4.T4.1.1.5" class="ltx_tr">
<td id="S4.T4.1.1.5.1" class="ltx_td ltx_align_left">
<span id="S4.T4.1.1.5.1.1" class="ltx_text" style="font-size:70%;">SnapKV+</span><span id="S4.T4.1.1.5.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span>
</td>
<td id="S4.T4.1.1.5.2" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.5.2.1" class="ltx_text" style="font-size:70%;">50</span></td>
<td id="S4.T4.1.1.5.3" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.5.3.1" class="ltx_text" style="font-size:70%;">77.4</span></td>
<td id="S4.T4.1.1.5.4" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.5.4.1" class="ltx_text" style="font-size:70%;">89.6</span></td>
<td id="S4.T4.1.1.5.5" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.5.5.1" class="ltx_text" style="font-size:70%;">91.0</span></td>
<td id="S4.T4.1.1.5.6" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.5.6.1" class="ltx_text" style="font-size:70%;">91.7</span></td>
</tr>
<tr id="S4.T4.1.1.6" class="ltx_tr">
<td id="S4.T4.1.1.6.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t" rowspan="4"><span id="S4.T4.1.1.6.1.1" class="ltx_text" style="font-size:70%;">Mistral-7B</span></td>
<td id="S4.T4.1.1.6.2" class="ltx_td ltx_align_left ltx_border_t"><span id="S4.T4.1.1.6.2.1" class="ltx_text" style="font-size:70%;">SnapKV</span></td>
<td id="S4.T4.1.1.6.3" class="ltx_td ltx_align_center ltx_border_t">
<span id="S4.T4.1.1.6.3.1" class="ltx_text" style="font-size:70%;color:#FFFFFF;">0</span><span id="S4.T4.1.1.6.3.2" class="ltx_text" style="font-size:70%;">0</span>
</td>
<td id="S4.T4.1.1.6.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T4.1.1.6.4.1" class="ltx_text" style="font-size:70%;">77.8</span></td>
<td id="S4.T4.1.1.6.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T4.1.1.6.5.1" class="ltx_text" style="font-size:70%;">89.5</span></td>
<td id="S4.T4.1.1.6.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T4.1.1.6.6.1" class="ltx_text" style="font-size:70%;">90.4</span></td>
<td id="S4.T4.1.1.6.7" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T4.1.1.6.7.1" class="ltx_text" style="font-size:70%;">90.8</span></td>
</tr>
<tr id="S4.T4.1.1.7" class="ltx_tr">
<td id="S4.T4.1.1.7.1" class="ltx_td ltx_align_left">
<span id="S4.T4.1.1.7.1.1" class="ltx_text" style="font-size:70%;">SnapKV+</span><span id="S4.T4.1.1.7.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span>
</td>
<td id="S4.T4.1.1.7.2" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.7.2.1" class="ltx_text" style="font-size:70%;">40</span></td>
<td id="S4.T4.1.1.7.3" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.7.3.1" class="ltx_text" style="font-size:70%;">78.6</span></td>
<td id="S4.T4.1.1.7.4" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.7.4.1" class="ltx_text" style="font-size:70%;">90.1</span></td>
<td id="S4.T4.1.1.7.5" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.7.5.1" class="ltx_text" style="font-size:70%;">90.6</span></td>
<td id="S4.T4.1.1.7.6" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.7.6.1" class="ltx_text" style="font-size:70%;">90.9</span></td>
</tr>
<tr id="S4.T4.1.1.8" class="ltx_tr">
<td id="S4.T4.1.1.8.1" class="ltx_td ltx_align_left">
<span id="S4.T4.1.1.8.1.1" class="ltx_text" style="font-size:70%;">SnapKV+</span><span id="S4.T4.1.1.8.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span>
</td>
<td id="S4.T4.1.1.8.2" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.8.2.1" class="ltx_text" style="font-size:70%;">50</span></td>
<td id="S4.T4.1.1.8.3" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.8.3.1" class="ltx_text" style="font-size:70%;">78.1</span></td>
<td id="S4.T4.1.1.8.4" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.8.4.1" class="ltx_text" style="font-size:70%;">90.1</span></td>
<td id="S4.T4.1.1.8.5" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.8.5.1" class="ltx_text" style="font-size:70%;">90.8</span></td>
<td id="S4.T4.1.1.8.6" class="ltx_td ltx_align_center"><span id="S4.T4.1.1.8.6.1" class="ltx_text" style="font-size:70%;">91.1</span></td>
</tr>
<tr id="S4.T4.1.1.9" class="ltx_tr">
<td id="S4.T4.1.1.9.1" class="ltx_td ltx_align_left ltx_border_bb">
<span id="S4.T4.1.1.9.1.1" class="ltx_text" style="font-size:70%;">SnapKV+</span><span id="S4.T4.1.1.9.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span>
</td>
<td id="S4.T4.1.1.9.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T4.1.1.9.2.1" class="ltx_text" style="font-size:70%;">60</span></td>
<td id="S4.T4.1.1.9.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T4.1.1.9.3.1" class="ltx_text" style="font-size:70%;">75.9</span></td>
<td id="S4.T4.1.1.9.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T4.1.1.9.4.1" class="ltx_text" style="font-size:70%;">89.2</span></td>
<td id="S4.T4.1.1.9.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T4.1.1.9.5.1" class="ltx_text" style="font-size:70%;">90.6</span></td>
<td id="S4.T4.1.1.9.6" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T4.1.1.9.6.1" class="ltx_text" style="font-size:70%;">91.1</span></td>
</tr>
</table>
</span></div>
</figure>
</section>
<section id="S4.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.4 </span>Ablation Studies</h3>

<figure id="S4.T5" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table"><span id="S4.T5.6.1.1" class="ltx_text" style="font-size:129%;">Table 5</span>: </span><span id="S4.T5.7.2" class="ltx_text" style="font-size:129%;">Performance comparison of pruning K cache using different recent-size.</span></figcaption>
<div id="S4.T5.2" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:169pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-20.7pt,8.1pt) scale(0.912871707848879,0.912871707848879) ;">
<table id="S4.T5.2.2" class="ltx_tabular ltx_align_middle">
<tr id="S4.T5.2.2.3" class="ltx_tr">
<td id="S4.T5.2.2.3.1" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_tt" rowspan="2"><span id="S4.T5.2.2.3.1.1" class="ltx_text" style="font-size:70%;">Recent-Size</span></td>
<td id="S4.T5.2.2.3.2" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="S4.T5.2.2.3.2.1" class="ltx_text" style="font-size:70%;">Single-Document QA</span></td>
<td id="S4.T5.2.2.3.3" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="S4.T5.2.2.3.3.1" class="ltx_text" style="font-size:70%;">Multi-Document QA</span></td>
<td id="S4.T5.2.2.3.4" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="S4.T5.2.2.3.4.1" class="ltx_text" style="font-size:70%;">Summarization</span></td>
<td id="S4.T5.2.2.3.5" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="S4.T5.2.2.3.5.1" class="ltx_text" style="font-size:70%;">Few-shot Learning</span></td>
<td id="S4.T5.2.2.3.6" class="ltx_td ltx_align_center ltx_border_tt" colspan="2"><span id="S4.T5.2.2.3.6.1" class="ltx_text" style="font-size:70%;">Synthetic</span></td>
<td id="S4.T5.2.2.3.7" class="ltx_td ltx_align_center ltx_border_tt" colspan="2"><span id="S4.T5.2.2.3.7.1" class="ltx_text" style="font-size:70%;">Code</span></td>
<td id="S4.T5.2.2.3.8" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_tt" rowspan="3"><span id="S4.T5.2.2.3.8.1" class="ltx_text" style="font-size:70%;">Avg.</span></td>
</tr>
<tr id="S4.T5.2.2.4" class="ltx_tr">
<td id="S4.T5.2.2.4.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T5.2.2.4.1.1" class="ltx_inline-block ltx_transformed_outer" style="width:24.8pt;height:17.8pt;vertical-align:-7.2pt;"><span class="ltx_transformed_inner" style="width:25.1pt;transform:translate(-0.15pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T5.2.2.4.1.1.1" class="ltx_p"><span id="S4.T5.2.2.4.1.1.1.1" class="ltx_text" style="font-size:70%;">NrtvQA</span></p>
</span></div>
</td>
<td id="S4.T5.2.2.4.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T5.2.2.4.2.1" class="ltx_inline-block ltx_transformed_outer" style="width:21.8pt;height:16.2pt;vertical-align:-6.4pt;"><span class="ltx_transformed_inner" style="width:21.6pt;transform:translate(0.09pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T5.2.2.4.2.1.1" class="ltx_p"><span id="S4.T5.2.2.4.2.1.1.1" class="ltx_text" style="font-size:70%;">Qasper</span></p>
</span></div>
</td>
<td id="S4.T5.2.2.4.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T5.2.2.4.3.1" class="ltx_inline-block ltx_transformed_outer" style="width:20.0pt;height:14.3pt;vertical-align:-4.8pt;"><span class="ltx_transformed_inner" style="width:20.3pt;transform:translate(-0.17pt,0pt) rotate(-30deg) ;">
<p id="S4.T5.2.2.4.3.1.1" class="ltx_p"><span id="S4.T5.2.2.4.3.1.1.1" class="ltx_text" style="font-size:70%;">MF-en</span></p>
</span></div>
</td>
<td id="S4.T5.2.2.4.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T5.2.2.4.4.1" class="ltx_inline-block ltx_transformed_outer" style="width:31.2pt;height:21.6pt;vertical-align:-9.1pt;"><span class="ltx_transformed_inner" style="width:32.5pt;transform:translate(-0.64pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T5.2.2.4.4.1.1" class="ltx_p"><span id="S4.T5.2.2.4.4.1.1.1" class="ltx_text" style="font-size:70%;">HotpotQA</span></p>
</span></div>
</td>
<td id="S4.T5.2.2.4.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T5.2.2.4.5.1" class="ltx_inline-block ltx_transformed_outer" style="width:33.8pt;height:23.1pt;vertical-align:-9.8pt;"><span class="ltx_transformed_inner" style="width:35.4pt;transform:translate(-0.82pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T5.2.2.4.5.1.1" class="ltx_p"><span id="S4.T5.2.2.4.5.1.1.1" class="ltx_text" style="font-size:70%;">2WikiMQA</span></p>
</span></div>
</td>
<td id="S4.T5.2.2.4.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T5.2.2.4.6.1" class="ltx_inline-block ltx_transformed_outer" style="width:25.3pt;height:18.2pt;vertical-align:-7.4pt;"><span class="ltx_transformed_inner" style="width:25.7pt;transform:translate(-0.19pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T5.2.2.4.6.1.1" class="ltx_p"><span id="S4.T5.2.2.4.6.1.1.1" class="ltx_text" style="font-size:70%;">Musique</span></p>
</span></div>
</td>
<td id="S4.T5.2.2.4.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T5.2.2.4.7.1" class="ltx_inline-block ltx_transformed_outer" style="width:32.3pt;height:22.2pt;vertical-align:-9.4pt;"><span class="ltx_transformed_inner" style="width:33.8pt;transform:translate(-0.73pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T5.2.2.4.7.1.1" class="ltx_p"><span id="S4.T5.2.2.4.7.1.1.1" class="ltx_text" style="font-size:70%;">GovReport</span></p>
</span></div>
</td>
<td id="S4.T5.2.2.4.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T5.2.2.4.8.1" class="ltx_inline-block ltx_transformed_outer" style="width:25.1pt;height:18pt;vertical-align:-7.3pt;"><span class="ltx_transformed_inner" style="width:25.5pt;transform:translate(-0.17pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T5.2.2.4.8.1.1" class="ltx_p"><span id="S4.T5.2.2.4.8.1.1.1" class="ltx_text" style="font-size:70%;">QMSum</span></p>
</span></div>
</td>
<td id="S4.T5.2.2.4.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T5.2.2.4.9.1" class="ltx_inline-block ltx_transformed_outer" style="width:31.1pt;height:20.7pt;vertical-align:-7.9pt;"><span class="ltx_transformed_inner" style="width:33.1pt;transform:translate(-1pt,0pt) rotate(-30deg) ;">
<p id="S4.T5.2.2.4.9.1.1" class="ltx_p"><span id="S4.T5.2.2.4.9.1.1.1" class="ltx_text" style="font-size:70%;">MultiNews</span></p>
</span></div>
</td>
<td id="S4.T5.2.2.4.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T5.2.2.4.10.1" class="ltx_inline-block ltx_transformed_outer" style="width:19.7pt;height:14.2pt;vertical-align:-4.7pt;"><span class="ltx_transformed_inner" style="width:20.0pt;transform:translate(-0.15pt,0pt) rotate(-30deg) ;">
<p id="S4.T5.2.2.4.10.1.1" class="ltx_p"><span id="S4.T5.2.2.4.10.1.1.1" class="ltx_text" style="font-size:70%;">TREC</span></p>
</span></div>
</td>
<td id="S4.T5.2.2.4.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T5.2.2.4.11.1" class="ltx_inline-block ltx_transformed_outer" style="width:28.2pt;height:19.8pt;vertical-align:-8.2pt;"><span class="ltx_transformed_inner" style="width:29.0pt;transform:translate(-0.41pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T5.2.2.4.11.1.1" class="ltx_p"><span id="S4.T5.2.2.4.11.1.1.1" class="ltx_text" style="font-size:70%;">TriviaQA</span></p>
</span></div>
</td>
<td id="S4.T5.2.2.4.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T5.2.2.4.12.1" class="ltx_inline-block ltx_transformed_outer" style="width:27.7pt;height:18.8pt;vertical-align:-7.0pt;"><span class="ltx_transformed_inner" style="width:29.2pt;transform:translate(-0.76pt,0pt) rotate(-30deg) ;">
<p id="S4.T5.2.2.4.12.1.1" class="ltx_p"><span id="S4.T5.2.2.4.12.1.1.1" class="ltx_text" style="font-size:70%;">SAMSum</span></p>
</span></div>
</td>
<td id="S4.T5.2.2.4.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T5.2.2.4.13.1" class="ltx_inline-block ltx_transformed_outer" style="width:22.9pt;height:16pt;vertical-align:-5.6pt;"><span class="ltx_transformed_inner" style="width:23.6pt;transform:translate(-0.39pt,0pt) rotate(-30deg) ;">
<p id="S4.T5.2.2.4.13.1.1" class="ltx_p"><span id="S4.T5.2.2.4.13.1.1.1" class="ltx_text" style="font-size:70%;">PCount</span></p>
</span></div>
</td>
<td id="S4.T5.2.2.4.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T5.2.2.4.14.1" class="ltx_inline-block ltx_transformed_outer" style="width:19.7pt;height:14.2pt;vertical-align:-4.7pt;"><span class="ltx_transformed_inner" style="width:20.0pt;transform:translate(-0.15pt,0pt) rotate(-30deg) ;">
<p id="S4.T5.2.2.4.14.1.1" class="ltx_p"><span id="S4.T5.2.2.4.14.1.1.1" class="ltx_text" style="font-size:70%;">PRe</span></p>
</span></div>
</td>
<td id="S4.T5.2.2.4.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T5.2.2.4.15.1" class="ltx_inline-block ltx_transformed_outer" style="width:17.6pt;height:13pt;vertical-align:-4.1pt;"><span class="ltx_transformed_inner" style="width:17.6pt;transform:translate(0.02pt,0pt) rotate(-30deg) ;">
<p id="S4.T5.2.2.4.15.1.1" class="ltx_p"><span id="S4.T5.2.2.4.15.1.1.1" class="ltx_text" style="font-size:70%;">Lcc</span></p>
</span></div>
</td>
<td id="S4.T5.2.2.4.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T5.2.2.4.16.1" class="ltx_inline-block ltx_transformed_outer" style="width:21.3pt;height:15pt;vertical-align:-5.1pt;"><span class="ltx_transformed_inner" style="width:21.9pt;transform:translate(-0.27pt,0pt) rotate(-30deg) ;">
<p id="S4.T5.2.2.4.16.1.1" class="ltx_p"><span id="S4.T5.2.2.4.16.1.1.1" class="ltx_text" style="font-size:70%;">RB-P</span></p>
</span></div>
</td>
</tr>
<tr id="S4.T5.1.1.1" class="ltx_tr">
<td id="S4.T5.1.1.1.1" class="ltx_td ltx_align_center ltx_border_t" colspan="17">
<span id="S4.T5.1.1.1.1.1" class="ltx_text" style="font-size:70%;">H2O + </span><span id="S4.T5.1.1.1.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T5.1.1.1.1.3" class="ltx_text" style="font-size:70%;">(</span><math id="S4.T5.1.1.1.1.m1.1" class="ltx_Math" alttext="\lambda=40\%" display="inline"><semantics id="S4.T5.1.1.1.1.m1.1a"><mrow id="S4.T5.1.1.1.1.m1.1.1" xref="S4.T5.1.1.1.1.m1.1.1.cmml"><mi mathsize="70%" id="S4.T5.1.1.1.1.m1.1.1.2" xref="S4.T5.1.1.1.1.m1.1.1.2.cmml">λ</mi><mo mathsize="70%" id="S4.T5.1.1.1.1.m1.1.1.1" xref="S4.T5.1.1.1.1.m1.1.1.1.cmml">=</mo><mrow id="S4.T5.1.1.1.1.m1.1.1.3" xref="S4.T5.1.1.1.1.m1.1.1.3.cmml"><mn mathsize="70%" id="S4.T5.1.1.1.1.m1.1.1.3.2" xref="S4.T5.1.1.1.1.m1.1.1.3.2.cmml">40</mn><mo mathsize="70%" id="S4.T5.1.1.1.1.m1.1.1.3.1" xref="S4.T5.1.1.1.1.m1.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.T5.1.1.1.1.m1.1b"><apply id="S4.T5.1.1.1.1.m1.1.1.cmml" xref="S4.T5.1.1.1.1.m1.1.1"><eq id="S4.T5.1.1.1.1.m1.1.1.1.cmml" xref="S4.T5.1.1.1.1.m1.1.1.1"></eq><ci id="S4.T5.1.1.1.1.m1.1.1.2.cmml" xref="S4.T5.1.1.1.1.m1.1.1.2">𝜆</ci><apply id="S4.T5.1.1.1.1.m1.1.1.3.cmml" xref="S4.T5.1.1.1.1.m1.1.1.3"><csymbol cd="latexml" id="S4.T5.1.1.1.1.m1.1.1.3.1.cmml" xref="S4.T5.1.1.1.1.m1.1.1.3.1">percent</csymbol><cn type="integer" id="S4.T5.1.1.1.1.m1.1.1.3.2.cmml" xref="S4.T5.1.1.1.1.m1.1.1.3.2">40</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T5.1.1.1.1.m1.1c">\lambda=40\%</annotation></semantics></math><span id="S4.T5.1.1.1.1.4" class="ltx_text" style="font-size:70%;">)</span>
</td>
</tr>
<tr id="S4.T5.2.2.5" class="ltx_tr">
<td id="S4.T5.2.2.5.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.5.1.1" class="ltx_text" style="font-size:70%;">0</span></td>
<td id="S4.T5.2.2.5.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.5.2.1" class="ltx_text" style="font-size:70%;">24.40</span></td>
<td id="S4.T5.2.2.5.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.5.3.1" class="ltx_text" style="font-size:70%;">27.50</span></td>
<td id="S4.T5.2.2.5.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.5.4.1" class="ltx_text" style="font-size:70%;">45.42</span></td>
<td id="S4.T5.2.2.5.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.5.5.1" class="ltx_text" style="font-size:70%;">35.17</span></td>
<td id="S4.T5.2.2.5.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.5.6.1" class="ltx_text" style="font-size:70%;">24.45</span></td>
<td id="S4.T5.2.2.5.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.5.7.1" class="ltx_text" style="font-size:70%;">13.02</span></td>
<td id="S4.T5.2.2.5.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.5.8.1" class="ltx_text" style="font-size:70%;">27.65</span></td>
<td id="S4.T5.2.2.5.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.5.9.1" class="ltx_text" style="font-size:70%;">23.88</span></td>
<td id="S4.T5.2.2.5.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.5.10.1" class="ltx_text" style="font-size:70%;">26.86</span></td>
<td id="S4.T5.2.2.5.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.5.11.1" class="ltx_text" style="font-size:70%;">53.50</span></td>
<td id="S4.T5.2.2.5.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.5.12.1" class="ltx_text" style="font-size:70%;">86.06</span></td>
<td id="S4.T5.2.2.5.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.5.13.1" class="ltx_text" style="font-size:70%;">41.73</span></td>
<td id="S4.T5.2.2.5.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.5.14.1" class="ltx_text" style="font-size:70%;">3.01</span></td>
<td id="S4.T5.2.2.5.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.5.15.1" class="ltx_text" style="font-size:70%;">83.42</span></td>
<td id="S4.T5.2.2.5.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.5.16.1" class="ltx_text" style="font-size:70%;">55.12</span></td>
<td id="S4.T5.2.2.5.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.5.17.1" class="ltx_text" style="font-size:70%;">51.32</span></td>
<td id="S4.T5.2.2.5.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T5.2.2.5.18.1" class="ltx_text" style="font-size:70%;">38.91</span></td>
</tr>
<tr id="S4.T5.2.2.6" class="ltx_tr">
<td id="S4.T5.2.2.6.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.6.1.1" class="ltx_text" style="font-size:70%;">32</span></td>
<td id="S4.T5.2.2.6.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.6.2.1" class="ltx_text" style="font-size:70%;">25.40</span></td>
<td id="S4.T5.2.2.6.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.6.3.1" class="ltx_text" style="font-size:70%;">30.80</span></td>
<td id="S4.T5.2.2.6.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.6.4.1" class="ltx_text" style="font-size:70%;">48.45</span></td>
<td id="S4.T5.2.2.6.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.6.5.1" class="ltx_text" style="font-size:70%;">39.64</span></td>
<td id="S4.T5.2.2.6.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.6.6.1" class="ltx_text" style="font-size:70%;">26.08</span></td>
<td id="S4.T5.2.2.6.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.6.7.1" class="ltx_text" style="font-size:70%;">16.82</span></td>
<td id="S4.T5.2.2.6.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.6.8.1" class="ltx_text" style="font-size:70%;">27.12</span></td>
<td id="S4.T5.2.2.6.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.6.9.1" class="ltx_text" style="font-size:70%;">23.79</span></td>
<td id="S4.T5.2.2.6.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.6.10.1" class="ltx_text" style="font-size:70%;">26.65</span></td>
<td id="S4.T5.2.2.6.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.6.11.1" class="ltx_text" style="font-size:70%;">53.50</span></td>
<td id="S4.T5.2.2.6.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.6.12.1" class="ltx_text" style="font-size:70%;">86.39</span></td>
<td id="S4.T5.2.2.6.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.6.13.1" class="ltx_text" style="font-size:70%;">43.03</span></td>
<td id="S4.T5.2.2.6.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.6.14.1" class="ltx_text" style="font-size:70%;">3.29</span></td>
<td id="S4.T5.2.2.6.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.6.15.1" class="ltx_text" style="font-size:70%;">86.39</span></td>
<td id="S4.T5.2.2.6.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.6.16.1" class="ltx_text" style="font-size:70%;">56.61</span></td>
<td id="S4.T5.2.2.6.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.6.17.1" class="ltx_text" style="font-size:70%;">53.60</span></td>
<td id="S4.T5.2.2.6.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T5.2.2.6.18.1" class="ltx_text" style="font-size:70%;">40.47</span></td>
</tr>
<tr id="S4.T5.2.2.7" class="ltx_tr">
<td id="S4.T5.2.2.7.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.7.1.1" class="ltx_text" style="font-size:70%;">128</span></td>
<td id="S4.T5.2.2.7.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.7.2.1" class="ltx_text" style="font-size:70%;">25.69</span></td>
<td id="S4.T5.2.2.7.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.7.3.1" class="ltx_text" style="font-size:70%;">30.93</span></td>
<td id="S4.T5.2.2.7.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.7.4.1" class="ltx_text" style="font-size:70%;">48.32</span></td>
<td id="S4.T5.2.2.7.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.7.5.1" class="ltx_text" style="font-size:70%;">39.63</span></td>
<td id="S4.T5.2.2.7.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.7.6.1" class="ltx_text" style="font-size:70%;">26.08</span></td>
<td id="S4.T5.2.2.7.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.7.7.1" class="ltx_text" style="font-size:70%;">16.82</span></td>
<td id="S4.T5.2.2.7.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.7.8.1" class="ltx_text" style="font-size:70%;">27.18</span></td>
<td id="S4.T5.2.2.7.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.7.9.1" class="ltx_text" style="font-size:70%;">23.92</span></td>
<td id="S4.T5.2.2.7.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.7.10.1" class="ltx_text" style="font-size:70%;">26.62</span></td>
<td id="S4.T5.2.2.7.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.7.11.1" class="ltx_text" style="font-size:70%;">53.50</span></td>
<td id="S4.T5.2.2.7.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.7.12.1" class="ltx_text" style="font-size:70%;">86.39</span></td>
<td id="S4.T5.2.2.7.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.7.13.1" class="ltx_text" style="font-size:70%;">42.96</span></td>
<td id="S4.T5.2.2.7.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.7.14.1" class="ltx_text" style="font-size:70%;">3.29</span></td>
<td id="S4.T5.2.2.7.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.7.15.1" class="ltx_text" style="font-size:70%;">86.39</span></td>
<td id="S4.T5.2.2.7.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.7.16.1" class="ltx_text" style="font-size:70%;">56.77</span></td>
<td id="S4.T5.2.2.7.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.7.17.1" class="ltx_text" style="font-size:70%;">53.60</span></td>
<td id="S4.T5.2.2.7.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T5.2.2.7.18.1" class="ltx_text ltx_font_bold" style="font-size:70%;">40.51</span></td>
</tr>
<tr id="S4.T5.2.2.2" class="ltx_tr">
<td id="S4.T5.2.2.2.1" class="ltx_td ltx_align_center ltx_border_t" colspan="17">
<span id="S4.T5.2.2.2.1.1" class="ltx_text" style="font-size:70%;">SnapKV + </span><span id="S4.T5.2.2.2.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T5.2.2.2.1.3" class="ltx_text" style="font-size:70%;">(</span><math id="S4.T5.2.2.2.1.m1.1" class="ltx_Math" alttext="\lambda=40\%" display="inline"><semantics id="S4.T5.2.2.2.1.m1.1a"><mrow id="S4.T5.2.2.2.1.m1.1.1" xref="S4.T5.2.2.2.1.m1.1.1.cmml"><mi mathsize="70%" id="S4.T5.2.2.2.1.m1.1.1.2" xref="S4.T5.2.2.2.1.m1.1.1.2.cmml">λ</mi><mo mathsize="70%" id="S4.T5.2.2.2.1.m1.1.1.1" xref="S4.T5.2.2.2.1.m1.1.1.1.cmml">=</mo><mrow id="S4.T5.2.2.2.1.m1.1.1.3" xref="S4.T5.2.2.2.1.m1.1.1.3.cmml"><mn mathsize="70%" id="S4.T5.2.2.2.1.m1.1.1.3.2" xref="S4.T5.2.2.2.1.m1.1.1.3.2.cmml">40</mn><mo mathsize="70%" id="S4.T5.2.2.2.1.m1.1.1.3.1" xref="S4.T5.2.2.2.1.m1.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.T5.2.2.2.1.m1.1b"><apply id="S4.T5.2.2.2.1.m1.1.1.cmml" xref="S4.T5.2.2.2.1.m1.1.1"><eq id="S4.T5.2.2.2.1.m1.1.1.1.cmml" xref="S4.T5.2.2.2.1.m1.1.1.1"></eq><ci id="S4.T5.2.2.2.1.m1.1.1.2.cmml" xref="S4.T5.2.2.2.1.m1.1.1.2">𝜆</ci><apply id="S4.T5.2.2.2.1.m1.1.1.3.cmml" xref="S4.T5.2.2.2.1.m1.1.1.3"><csymbol cd="latexml" id="S4.T5.2.2.2.1.m1.1.1.3.1.cmml" xref="S4.T5.2.2.2.1.m1.1.1.3.1">percent</csymbol><cn type="integer" id="S4.T5.2.2.2.1.m1.1.1.3.2.cmml" xref="S4.T5.2.2.2.1.m1.1.1.3.2">40</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T5.2.2.2.1.m1.1c">\lambda=40\%</annotation></semantics></math><span id="S4.T5.2.2.2.1.4" class="ltx_text" style="font-size:70%;">)</span>
</td>
<td id="S4.T5.2.2.2.2" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="S4.T5.2.2.8" class="ltx_tr">
<td id="S4.T5.2.2.8.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.8.1.1" class="ltx_text" style="font-size:70%;">0</span></td>
<td id="S4.T5.2.2.8.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.8.2.1" class="ltx_text" style="font-size:70%;">24.94</span></td>
<td id="S4.T5.2.2.8.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.8.3.1" class="ltx_text" style="font-size:70%;">28.58</span></td>
<td id="S4.T5.2.2.8.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.8.4.1" class="ltx_text" style="font-size:70%;">45.78</span></td>
<td id="S4.T5.2.2.8.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.8.5.1" class="ltx_text" style="font-size:70%;">39.59</span></td>
<td id="S4.T5.2.2.8.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.8.6.1" class="ltx_text" style="font-size:70%;">25.40</span></td>
<td id="S4.T5.2.2.8.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.8.7.1" class="ltx_text" style="font-size:70%;">15.92</span></td>
<td id="S4.T5.2.2.8.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.8.8.1" class="ltx_text" style="font-size:70%;">29.50</span></td>
<td id="S4.T5.2.2.8.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.8.9.1" class="ltx_text" style="font-size:70%;">24.05</span></td>
<td id="S4.T5.2.2.8.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.8.10.1" class="ltx_text" style="font-size:70%;">26.72</span></td>
<td id="S4.T5.2.2.8.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.8.11.1" class="ltx_text" style="font-size:70%;">70.00</span></td>
<td id="S4.T5.2.2.8.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.8.12.1" class="ltx_text" style="font-size:70%;">85.60</span></td>
<td id="S4.T5.2.2.8.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.8.13.1" class="ltx_text" style="font-size:70%;">41.38</span></td>
<td id="S4.T5.2.2.8.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.8.14.1" class="ltx_text" style="font-size:70%;">2.97</span></td>
<td id="S4.T5.2.2.8.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.8.15.1" class="ltx_text" style="font-size:70%;">84.00</span></td>
<td id="S4.T5.2.2.8.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.8.16.1" class="ltx_text" style="font-size:70%;">55.27</span></td>
<td id="S4.T5.2.2.8.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.8.17.1" class="ltx_text" style="font-size:70%;">52.39</span></td>
<td id="S4.T5.2.2.8.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T5.2.2.8.18.1" class="ltx_text" style="font-size:70%;">40.76</span></td>
</tr>
<tr id="S4.T5.2.2.9" class="ltx_tr">
<td id="S4.T5.2.2.9.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.9.1.1" class="ltx_text" style="font-size:70%;">32</span></td>
<td id="S4.T5.2.2.9.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.9.2.1" class="ltx_text" style="font-size:70%;">25.77</span></td>
<td id="S4.T5.2.2.9.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.9.3.1" class="ltx_text" style="font-size:70%;">32.67</span></td>
<td id="S4.T5.2.2.9.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.9.4.1" class="ltx_text" style="font-size:70%;">48.70</span></td>
<td id="S4.T5.2.2.9.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.9.5.1" class="ltx_text" style="font-size:70%;">41.06</span></td>
<td id="S4.T5.2.2.9.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.9.6.1" class="ltx_text" style="font-size:70%;">27.07</span></td>
<td id="S4.T5.2.2.9.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.9.7.1" class="ltx_text" style="font-size:70%;">19.14</span></td>
<td id="S4.T5.2.2.9.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.9.8.1" class="ltx_text" style="font-size:70%;">28.91</span></td>
<td id="S4.T5.2.2.9.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.9.9.1" class="ltx_text" style="font-size:70%;">24.37</span></td>
<td id="S4.T5.2.2.9.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.9.10.1" class="ltx_text" style="font-size:70%;">26.88</span></td>
<td id="S4.T5.2.2.9.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.9.11.1" class="ltx_text" style="font-size:70%;">70.00</span></td>
<td id="S4.T5.2.2.9.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.9.12.1" class="ltx_text" style="font-size:70%;">86.37</span></td>
<td id="S4.T5.2.2.9.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.9.13.1" class="ltx_text" style="font-size:70%;">42.75</span></td>
<td id="S4.T5.2.2.9.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.9.14.1" class="ltx_text" style="font-size:70%;">3.61</span></td>
<td id="S4.T5.2.2.9.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.9.15.1" class="ltx_text" style="font-size:70%;">87.38</span></td>
<td id="S4.T5.2.2.9.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.9.16.1" class="ltx_text" style="font-size:70%;">57.21</span></td>
<td id="S4.T5.2.2.9.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T5.2.2.9.17.1" class="ltx_text" style="font-size:70%;">54.44</span></td>
<td id="S4.T5.2.2.9.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T5.2.2.9.18.1" class="ltx_text ltx_font_bold" style="font-size:70%;">42.27</span></td>
</tr>
<tr id="S4.T5.2.2.10" class="ltx_tr">
<td id="S4.T5.2.2.10.1" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T5.2.2.10.1.1" class="ltx_text" style="font-size:70%;">128</span></td>
<td id="S4.T5.2.2.10.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T5.2.2.10.2.1" class="ltx_text" style="font-size:70%;">25.75</span></td>
<td id="S4.T5.2.2.10.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T5.2.2.10.3.1" class="ltx_text" style="font-size:70%;">32.49</span></td>
<td id="S4.T5.2.2.10.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T5.2.2.10.4.1" class="ltx_text" style="font-size:70%;">48.61</span></td>
<td id="S4.T5.2.2.10.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T5.2.2.10.5.1" class="ltx_text" style="font-size:70%;">41.01</span></td>
<td id="S4.T5.2.2.10.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T5.2.2.10.6.1" class="ltx_text" style="font-size:70%;">27.18</span></td>
<td id="S4.T5.2.2.10.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T5.2.2.10.7.1" class="ltx_text" style="font-size:70%;">19.14</span></td>
<td id="S4.T5.2.2.10.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T5.2.2.10.8.1" class="ltx_text" style="font-size:70%;">28.79</span></td>
<td id="S4.T5.2.2.10.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T5.2.2.10.9.1" class="ltx_text" style="font-size:70%;">24.64</span></td>
<td id="S4.T5.2.2.10.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T5.2.2.10.10.1" class="ltx_text" style="font-size:70%;">26.77</span></td>
<td id="S4.T5.2.2.10.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T5.2.2.10.11.1" class="ltx_text" style="font-size:70%;">70.00</span></td>
<td id="S4.T5.2.2.10.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T5.2.2.10.12.1" class="ltx_text" style="font-size:70%;">86.37</span></td>
<td id="S4.T5.2.2.10.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T5.2.2.10.13.1" class="ltx_text" style="font-size:70%;">42.77</span></td>
<td id="S4.T5.2.2.10.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T5.2.2.10.14.1" class="ltx_text" style="font-size:70%;">3.61</span></td>
<td id="S4.T5.2.2.10.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T5.2.2.10.15.1" class="ltx_text" style="font-size:70%;">87.13</span></td>
<td id="S4.T5.2.2.10.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T5.2.2.10.16.1" class="ltx_text" style="font-size:70%;">57.19</span></td>
<td id="S4.T5.2.2.10.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T5.2.2.10.17.1" class="ltx_text" style="font-size:70%;">54.39</span></td>
<td id="S4.T5.2.2.10.18" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_bb"><span id="S4.T5.2.2.10.18.1" class="ltx_text" style="font-size:70%;">42.24</span></td>
</tr>
</table>
</span></div>
</figure>
<figure id="S4.T6" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table"><span id="S4.T6.4.1.1" class="ltx_text" style="font-size:129%;">Table 6</span>: </span><span id="S4.T6.5.2" class="ltx_text" style="font-size:129%;">ablations with the same memory consumption.</span></figcaption>
<div id="S4.T6.6" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:313pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-36.1pt,26.1pt) scale(0.857230629709038,0.857230629709038) ;">
<table id="S4.T6.6.1" class="ltx_tabular ltx_align_middle">
<tr id="S4.T6.6.1.1" class="ltx_tr">
<td id="S4.T6.6.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt" rowspan="2"><span id="S4.T6.6.1.1.1.1" class="ltx_text" style="font-size:70%;">Methods</span></td>
<td id="S4.T6.6.1.1.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_tt" rowspan="3"><span id="S4.T6.6.1.1.2.1" class="ltx_text" style="font-size:70%;">Memory(M)</span></td>
<td id="S4.T6.6.1.1.3" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="S4.T6.6.1.1.3.1" class="ltx_text" style="font-size:70%;">Single-Document QA</span></td>
<td id="S4.T6.6.1.1.4" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="S4.T6.6.1.1.4.1" class="ltx_text" style="font-size:70%;">Multi-Document QA</span></td>
<td id="S4.T6.6.1.1.5" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="S4.T6.6.1.1.5.1" class="ltx_text" style="font-size:70%;">Summarization</span></td>
<td id="S4.T6.6.1.1.6" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="S4.T6.6.1.1.6.1" class="ltx_text" style="font-size:70%;">Few-shot Learning</span></td>
<td id="S4.T6.6.1.1.7" class="ltx_td ltx_align_center ltx_border_tt" colspan="2"><span id="S4.T6.6.1.1.7.1" class="ltx_text" style="font-size:70%;">Synthetic</span></td>
<td id="S4.T6.6.1.1.8" class="ltx_td ltx_align_center ltx_border_tt" colspan="2"><span id="S4.T6.6.1.1.8.1" class="ltx_text" style="font-size:70%;">Code</span></td>
<td id="S4.T6.6.1.1.9" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_tt" rowspan="3"><span id="S4.T6.6.1.1.9.1" class="ltx_text" style="font-size:70%;">Avg.</span></td>
</tr>
<tr id="S4.T6.6.1.2" class="ltx_tr">
<td id="S4.T6.6.1.2.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T6.6.1.2.1.1" class="ltx_inline-block ltx_transformed_outer" style="width:24.8pt;height:17.8pt;vertical-align:-7.2pt;"><span class="ltx_transformed_inner" style="width:25.1pt;transform:translate(-0.15pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T6.6.1.2.1.1.1" class="ltx_p"><span id="S4.T6.6.1.2.1.1.1.1" class="ltx_text" style="font-size:70%;">NrtvQA</span></p>
</span></div>
</td>
<td id="S4.T6.6.1.2.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T6.6.1.2.2.1" class="ltx_inline-block ltx_transformed_outer" style="width:21.8pt;height:16.2pt;vertical-align:-6.4pt;"><span class="ltx_transformed_inner" style="width:21.6pt;transform:translate(0.09pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T6.6.1.2.2.1.1" class="ltx_p"><span id="S4.T6.6.1.2.2.1.1.1" class="ltx_text" style="font-size:70%;">Qasper</span></p>
</span></div>
</td>
<td id="S4.T6.6.1.2.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T6.6.1.2.3.1" class="ltx_inline-block ltx_transformed_outer" style="width:20.0pt;height:14.3pt;vertical-align:-4.8pt;"><span class="ltx_transformed_inner" style="width:20.3pt;transform:translate(-0.17pt,0pt) rotate(-30deg) ;">
<p id="S4.T6.6.1.2.3.1.1" class="ltx_p"><span id="S4.T6.6.1.2.3.1.1.1" class="ltx_text" style="font-size:70%;">MF-en</span></p>
</span></div>
</td>
<td id="S4.T6.6.1.2.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T6.6.1.2.4.1" class="ltx_inline-block ltx_transformed_outer" style="width:31.2pt;height:21.6pt;vertical-align:-9.1pt;"><span class="ltx_transformed_inner" style="width:32.5pt;transform:translate(-0.64pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T6.6.1.2.4.1.1" class="ltx_p"><span id="S4.T6.6.1.2.4.1.1.1" class="ltx_text" style="font-size:70%;">HotpotQA</span></p>
</span></div>
</td>
<td id="S4.T6.6.1.2.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T6.6.1.2.5.1" class="ltx_inline-block ltx_transformed_outer" style="width:33.8pt;height:23.1pt;vertical-align:-9.8pt;"><span class="ltx_transformed_inner" style="width:35.4pt;transform:translate(-0.82pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T6.6.1.2.5.1.1" class="ltx_p"><span id="S4.T6.6.1.2.5.1.1.1" class="ltx_text" style="font-size:70%;">2WikiMQA</span></p>
</span></div>
</td>
<td id="S4.T6.6.1.2.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T6.6.1.2.6.1" class="ltx_inline-block ltx_transformed_outer" style="width:25.3pt;height:18.2pt;vertical-align:-7.4pt;"><span class="ltx_transformed_inner" style="width:25.7pt;transform:translate(-0.19pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T6.6.1.2.6.1.1" class="ltx_p"><span id="S4.T6.6.1.2.6.1.1.1" class="ltx_text" style="font-size:70%;">Musique</span></p>
</span></div>
</td>
<td id="S4.T6.6.1.2.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T6.6.1.2.7.1" class="ltx_inline-block ltx_transformed_outer" style="width:32.3pt;height:22.2pt;vertical-align:-9.4pt;"><span class="ltx_transformed_inner" style="width:33.8pt;transform:translate(-0.73pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T6.6.1.2.7.1.1" class="ltx_p"><span id="S4.T6.6.1.2.7.1.1.1" class="ltx_text" style="font-size:70%;">GovReport</span></p>
</span></div>
</td>
<td id="S4.T6.6.1.2.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T6.6.1.2.8.1" class="ltx_inline-block ltx_transformed_outer" style="width:25.1pt;height:18pt;vertical-align:-7.3pt;"><span class="ltx_transformed_inner" style="width:25.5pt;transform:translate(-0.17pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T6.6.1.2.8.1.1" class="ltx_p"><span id="S4.T6.6.1.2.8.1.1.1" class="ltx_text" style="font-size:70%;">QMSum</span></p>
</span></div>
</td>
<td id="S4.T6.6.1.2.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T6.6.1.2.9.1" class="ltx_inline-block ltx_transformed_outer" style="width:31.1pt;height:20.7pt;vertical-align:-7.9pt;"><span class="ltx_transformed_inner" style="width:33.1pt;transform:translate(-1pt,0pt) rotate(-30deg) ;">
<p id="S4.T6.6.1.2.9.1.1" class="ltx_p"><span id="S4.T6.6.1.2.9.1.1.1" class="ltx_text" style="font-size:70%;">MultiNews</span></p>
</span></div>
</td>
<td id="S4.T6.6.1.2.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T6.6.1.2.10.1" class="ltx_inline-block ltx_transformed_outer" style="width:19.7pt;height:14.2pt;vertical-align:-4.7pt;"><span class="ltx_transformed_inner" style="width:20.0pt;transform:translate(-0.15pt,0pt) rotate(-30deg) ;">
<p id="S4.T6.6.1.2.10.1.1" class="ltx_p"><span id="S4.T6.6.1.2.10.1.1.1" class="ltx_text" style="font-size:70%;">TREC</span></p>
</span></div>
</td>
<td id="S4.T6.6.1.2.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T6.6.1.2.11.1" class="ltx_inline-block ltx_transformed_outer" style="width:28.2pt;height:19.8pt;vertical-align:-8.2pt;"><span class="ltx_transformed_inner" style="width:29.0pt;transform:translate(-0.41pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T6.6.1.2.11.1.1" class="ltx_p"><span id="S4.T6.6.1.2.11.1.1.1" class="ltx_text" style="font-size:70%;">TriviaQA</span></p>
</span></div>
</td>
<td id="S4.T6.6.1.2.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T6.6.1.2.12.1" class="ltx_inline-block ltx_transformed_outer" style="width:27.7pt;height:18.8pt;vertical-align:-7.0pt;"><span class="ltx_transformed_inner" style="width:29.2pt;transform:translate(-0.76pt,0pt) rotate(-30deg) ;">
<p id="S4.T6.6.1.2.12.1.1" class="ltx_p"><span id="S4.T6.6.1.2.12.1.1.1" class="ltx_text" style="font-size:70%;">SAMSum</span></p>
</span></div>
</td>
<td id="S4.T6.6.1.2.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T6.6.1.2.13.1" class="ltx_inline-block ltx_transformed_outer" style="width:22.9pt;height:16pt;vertical-align:-5.6pt;"><span class="ltx_transformed_inner" style="width:23.6pt;transform:translate(-0.39pt,0pt) rotate(-30deg) ;">
<p id="S4.T6.6.1.2.13.1.1" class="ltx_p"><span id="S4.T6.6.1.2.13.1.1.1" class="ltx_text" style="font-size:70%;">PCount</span></p>
</span></div>
</td>
<td id="S4.T6.6.1.2.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T6.6.1.2.14.1" class="ltx_inline-block ltx_transformed_outer" style="width:19.7pt;height:14.2pt;vertical-align:-4.7pt;"><span class="ltx_transformed_inner" style="width:20.0pt;transform:translate(-0.15pt,0pt) rotate(-30deg) ;">
<p id="S4.T6.6.1.2.14.1.1" class="ltx_p"><span id="S4.T6.6.1.2.14.1.1.1" class="ltx_text" style="font-size:70%;">PRe</span></p>
</span></div>
</td>
<td id="S4.T6.6.1.2.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T6.6.1.2.15.1" class="ltx_inline-block ltx_transformed_outer" style="width:17.6pt;height:13pt;vertical-align:-4.1pt;"><span class="ltx_transformed_inner" style="width:17.6pt;transform:translate(0.02pt,0pt) rotate(-30deg) ;">
<p id="S4.T6.6.1.2.15.1.1" class="ltx_p"><span id="S4.T6.6.1.2.15.1.1.1" class="ltx_text" style="font-size:70%;">Lcc</span></p>
</span></div>
</td>
<td id="S4.T6.6.1.2.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T6.6.1.2.16.1" class="ltx_inline-block ltx_transformed_outer" style="width:21.3pt;height:15pt;vertical-align:-5.1pt;"><span class="ltx_transformed_inner" style="width:21.9pt;transform:translate(-0.27pt,0pt) rotate(-30deg) ;">
<p id="S4.T6.6.1.2.16.1.1" class="ltx_p"><span id="S4.T6.6.1.2.16.1.1.1" class="ltx_text" style="font-size:70%;">RB-P</span></p>
</span></div>
</td>
</tr>
<tr id="S4.T6.6.1.3" class="ltx_tr">
<td id="S4.T6.6.1.3.1" class="ltx_td ltx_align_center ltx_border_t" colspan="18"><span id="S4.T6.6.1.3.1.1" class="ltx_text" style="font-size:70%;">H2O</span></td>
</tr>
<tr id="S4.T6.6.1.4" class="ltx_tr">
<td id="S4.T6.6.1.4.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T6.6.1.4.1.1" class="ltx_text" style="font-size:70%;">Vanilla</span></td>
<td id="S4.T6.6.1.4.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.4.2.1" class="ltx_text" style="font-size:70%;">54.5</span></td>
<td id="S4.T6.6.1.4.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.4.3.1" class="ltx_text" style="font-size:70%;">21.29</span></td>
<td id="S4.T6.6.1.4.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.4.4.1" class="ltx_text" style="font-size:70%;">20.69</span></td>
<td id="S4.T6.6.1.4.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.4.5.1" class="ltx_text" style="font-size:70%;">37.66</span></td>
<td id="S4.T6.6.1.4.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.4.6.1" class="ltx_text" style="font-size:70%;">28.65</span></td>
<td id="S4.T6.6.1.4.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.4.7.1" class="ltx_text" style="font-size:70%;">21.08</span></td>
<td id="S4.T6.6.1.4.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.4.8.1" class="ltx_text" style="font-size:70%;">14.01</span></td>
<td id="S4.T6.6.1.4.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.4.9.1" class="ltx_text" style="font-size:70%;">20.20</span></td>
<td id="S4.T6.6.1.4.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.4.10.1" class="ltx_text" style="font-size:70%;">22.11</span></td>
<td id="S4.T6.6.1.4.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.4.11.1" class="ltx_text" style="font-size:70%;">21.33</span></td>
<td id="S4.T6.6.1.4.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.4.12.1" class="ltx_text" style="font-size:70%;">38.50</span></td>
<td id="S4.T6.6.1.4.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.4.13.1" class="ltx_text" style="font-size:70%;">82.55</span></td>
<td id="S4.T6.6.1.4.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.4.14.1" class="ltx_text" style="font-size:70%;">39.87</span></td>
<td id="S4.T6.6.1.4.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.4.15.1" class="ltx_text" style="font-size:70%;">3.66</span></td>
<td id="S4.T6.6.1.4.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.4.16.1" class="ltx_text" style="font-size:70%;">78.14</span></td>
<td id="S4.T6.6.1.4.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.4.17.1" class="ltx_text" style="font-size:70%;">50.32</span></td>
<td id="S4.T6.6.1.4.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.4.18.1" class="ltx_text" style="font-size:70%;">48.54</span></td>
<td id="S4.T6.6.1.4.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T6.6.1.4.19.1" class="ltx_text" style="font-size:70%;">34.29</span></td>
</tr>
<tr id="S4.T6.6.1.5" class="ltx_tr">
<td id="S4.T6.6.1.5.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T6.6.1.5.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span></td>
<td id="S4.T6.6.1.5.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.5.2.1" class="ltx_text" style="font-size:70%;">54.4</span></td>
<td id="S4.T6.6.1.5.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.5.3.1" class="ltx_text" style="font-size:70%;">21.17</span></td>
<td id="S4.T6.6.1.5.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.5.4.1" class="ltx_text" style="font-size:70%;">21.90</span></td>
<td id="S4.T6.6.1.5.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.5.5.1" class="ltx_text" style="font-size:70%;">39.29</span></td>
<td id="S4.T6.6.1.5.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.5.6.1" class="ltx_text" style="font-size:70%;">29.92</span></td>
<td id="S4.T6.6.1.5.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.5.7.1" class="ltx_text" style="font-size:70%;">20.99</span></td>
<td id="S4.T6.6.1.5.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.5.8.1" class="ltx_text" style="font-size:70%;">12.30</span></td>
<td id="S4.T6.6.1.5.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.5.9.1" class="ltx_text" style="font-size:70%;">20.84</span></td>
<td id="S4.T6.6.1.5.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.5.10.1" class="ltx_text" style="font-size:70%;">22.91</span></td>
<td id="S4.T6.6.1.5.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.5.11.1" class="ltx_text" style="font-size:70%;">21.92</span></td>
<td id="S4.T6.6.1.5.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.5.12.1" class="ltx_text" style="font-size:70%;">39.00</span></td>
<td id="S4.T6.6.1.5.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.5.13.1" class="ltx_text" style="font-size:70%;">82.70</span></td>
<td id="S4.T6.6.1.5.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.5.14.1" class="ltx_text" style="font-size:70%;">40.35</span></td>
<td id="S4.T6.6.1.5.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.5.15.1" class="ltx_text" style="font-size:70%;">2.97</span></td>
<td id="S4.T6.6.1.5.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.5.16.1" class="ltx_text" style="font-size:70%;">79.21</span></td>
<td id="S4.T6.6.1.5.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.5.17.1" class="ltx_text" style="font-size:70%;">51.19</span></td>
<td id="S4.T6.6.1.5.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.5.18.1" class="ltx_text" style="font-size:70%;">48.32</span></td>
<td id="S4.T6.6.1.5.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T6.6.1.5.19.1" class="ltx_text ltx_font_bold" style="font-size:70%;">34.69</span></td>
</tr>
<tr id="S4.T6.6.1.6" class="ltx_tr">
<td id="S4.T6.6.1.6.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T6.6.1.6.1.1" class="ltx_text" style="font-size:70%;">Vanilla</span></td>
<td id="S4.T6.6.1.6.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.6.2.1" class="ltx_text" style="font-size:70%;">208.0</span></td>
<td id="S4.T6.6.1.6.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.6.3.1" class="ltx_text" style="font-size:70%;">22.13</span></td>
<td id="S4.T6.6.1.6.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.6.4.1" class="ltx_text" style="font-size:70%;">23.83</span></td>
<td id="S4.T6.6.1.6.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.6.5.1" class="ltx_text" style="font-size:70%;">43.24</span></td>
<td id="S4.T6.6.1.6.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.6.6.1" class="ltx_text" style="font-size:70%;">30.92</span></td>
<td id="S4.T6.6.1.6.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.6.7.1" class="ltx_text" style="font-size:70%;">23.36</span></td>
<td id="S4.T6.6.1.6.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.6.8.1" class="ltx_text" style="font-size:70%;">14.56</span></td>
<td id="S4.T6.6.1.6.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.6.9.1" class="ltx_text" style="font-size:70%;">22.92</span></td>
<td id="S4.T6.6.1.6.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.6.10.1" class="ltx_text" style="font-size:70%;">22.77</span></td>
<td id="S4.T6.6.1.6.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.6.11.1" class="ltx_text" style="font-size:70%;">24.23</span></td>
<td id="S4.T6.6.1.6.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.6.12.1" class="ltx_text" style="font-size:70%;">41.50</span></td>
<td id="S4.T6.6.1.6.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.6.13.1" class="ltx_text" style="font-size:70%;">85.04</span></td>
<td id="S4.T6.6.1.6.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.6.14.1" class="ltx_text" style="font-size:70%;">41.26</span></td>
<td id="S4.T6.6.1.6.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.6.15.1" class="ltx_text" style="font-size:70%;">3.02</span></td>
<td id="S4.T6.6.1.6.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.6.16.1" class="ltx_text" style="font-size:70%;">86.03</span></td>
<td id="S4.T6.6.1.6.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.6.17.1" class="ltx_text" style="font-size:70%;">54.91</span></td>
<td id="S4.T6.6.1.6.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.6.18.1" class="ltx_text" style="font-size:70%;">50.50</span></td>
<td id="S4.T6.6.1.6.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T6.6.1.6.19.1" class="ltx_text" style="font-size:70%;">36.89</span></td>
</tr>
<tr id="S4.T6.6.1.7" class="ltx_tr">
<td id="S4.T6.6.1.7.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T6.6.1.7.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span></td>
<td id="S4.T6.6.1.7.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.7.2.1" class="ltx_text" style="font-size:70%;">208.0</span></td>
<td id="S4.T6.6.1.7.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.7.3.1" class="ltx_text" style="font-size:70%;">21.58</span></td>
<td id="S4.T6.6.1.7.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.7.4.1" class="ltx_text" style="font-size:70%;">26.15</span></td>
<td id="S4.T6.6.1.7.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.7.5.1" class="ltx_text" style="font-size:70%;">44.49</span></td>
<td id="S4.T6.6.1.7.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.7.6.1" class="ltx_text" style="font-size:70%;">32.73</span></td>
<td id="S4.T6.6.1.7.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.7.7.1" class="ltx_text" style="font-size:70%;">23.99</span></td>
<td id="S4.T6.6.1.7.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.7.8.1" class="ltx_text" style="font-size:70%;">15.09</span></td>
<td id="S4.T6.6.1.7.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.7.9.1" class="ltx_text" style="font-size:70%;">23.56</span></td>
<td id="S4.T6.6.1.7.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.7.10.1" class="ltx_text" style="font-size:70%;">23.28</span></td>
<td id="S4.T6.6.1.7.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.7.11.1" class="ltx_text" style="font-size:70%;">24.45</span></td>
<td id="S4.T6.6.1.7.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.7.12.1" class="ltx_text" style="font-size:70%;">42.00</span></td>
<td id="S4.T6.6.1.7.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.7.13.1" class="ltx_text" style="font-size:70%;">85.58</span></td>
<td id="S4.T6.6.1.7.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.7.14.1" class="ltx_text" style="font-size:70%;">42.58</span></td>
<td id="S4.T6.6.1.7.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.7.15.1" class="ltx_text" style="font-size:70%;">3.18</span></td>
<td id="S4.T6.6.1.7.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.7.16.1" class="ltx_text" style="font-size:70%;">85.7</span></td>
<td id="S4.T6.6.1.7.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.7.17.1" class="ltx_text" style="font-size:70%;">54.39</span></td>
<td id="S4.T6.6.1.7.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.7.18.1" class="ltx_text" style="font-size:70%;">51.15</span></td>
<td id="S4.T6.6.1.7.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T6.6.1.7.19.1" class="ltx_text ltx_font_bold" style="font-size:70%;">37.49</span></td>
</tr>
<tr id="S4.T6.6.1.8" class="ltx_tr">
<td id="S4.T6.6.1.8.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T6.6.1.8.1.1" class="ltx_text" style="font-size:70%;">Vanilla</span></td>
<td id="S4.T6.6.1.8.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.8.2.1" class="ltx_text" style="font-size:70%;">413.0</span></td>
<td id="S4.T6.6.1.8.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.8.3.1" class="ltx_text" style="font-size:70%;">22.90</span></td>
<td id="S4.T6.6.1.8.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.8.4.1" class="ltx_text" style="font-size:70%;">28.45</span></td>
<td id="S4.T6.6.1.8.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.8.5.1" class="ltx_text" style="font-size:70%;">46.16</span></td>
<td id="S4.T6.6.1.8.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.8.6.1" class="ltx_text" style="font-size:70%;">35.57</span></td>
<td id="S4.T6.6.1.8.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.8.7.1" class="ltx_text" style="font-size:70%;">23.86</span></td>
<td id="S4.T6.6.1.8.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.8.8.1" class="ltx_text" style="font-size:70%;">13.74</span></td>
<td id="S4.T6.6.1.8.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.8.9.1" class="ltx_text" style="font-size:70%;">24.90</span></td>
<td id="S4.T6.6.1.8.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.8.10.1" class="ltx_text" style="font-size:70%;">23.19</span></td>
<td id="S4.T6.6.1.8.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.8.11.1" class="ltx_text" style="font-size:70%;">25.77</span></td>
<td id="S4.T6.6.1.8.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.8.12.1" class="ltx_text" style="font-size:70%;">44.50</span></td>
<td id="S4.T6.6.1.8.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.8.13.1" class="ltx_text" style="font-size:70%;">85.54</span></td>
<td id="S4.T6.6.1.8.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.8.14.1" class="ltx_text" style="font-size:70%;">41.97</span></td>
<td id="S4.T6.6.1.8.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.8.15.1" class="ltx_text" style="font-size:70%;">3.22</span></td>
<td id="S4.T6.6.1.8.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.8.16.1" class="ltx_text" style="font-size:70%;">85.82</span></td>
<td id="S4.T6.6.1.8.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.8.17.1" class="ltx_text" style="font-size:70%;">55.96</span></td>
<td id="S4.T6.6.1.8.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.8.18.1" class="ltx_text" style="font-size:70%;">52.33</span></td>
<td id="S4.T6.6.1.8.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T6.6.1.8.19.1" class="ltx_text" style="font-size:70%;">38.37</span></td>
</tr>
<tr id="S4.T6.6.1.9" class="ltx_tr">
<td id="S4.T6.6.1.9.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T6.6.1.9.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span></td>
<td id="S4.T6.6.1.9.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.9.2.1" class="ltx_text" style="font-size:70%;">412.8</span></td>
<td id="S4.T6.6.1.9.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.9.3.1" class="ltx_text" style="font-size:70%;">23.97</span></td>
<td id="S4.T6.6.1.9.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.9.4.1" class="ltx_text" style="font-size:70%;">28.91</span></td>
<td id="S4.T6.6.1.9.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.9.5.1" class="ltx_text" style="font-size:70%;">45.84</span></td>
<td id="S4.T6.6.1.9.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.9.6.1" class="ltx_text" style="font-size:70%;">35.78</span></td>
<td id="S4.T6.6.1.9.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.9.7.1" class="ltx_text" style="font-size:70%;">24.88</span></td>
<td id="S4.T6.6.1.9.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.9.8.1" class="ltx_text" style="font-size:70%;">14.55</span></td>
<td id="S4.T6.6.1.9.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.9.9.1" class="ltx_text" style="font-size:70%;">25.11</span></td>
<td id="S4.T6.6.1.9.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.9.10.1" class="ltx_text" style="font-size:70%;">23.35</span></td>
<td id="S4.T6.6.1.9.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.9.11.1" class="ltx_text" style="font-size:70%;">25.83</span></td>
<td id="S4.T6.6.1.9.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.9.12.1" class="ltx_text" style="font-size:70%;">45.50</span></td>
<td id="S4.T6.6.1.9.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.9.13.1" class="ltx_text" style="font-size:70%;">86.11</span></td>
<td id="S4.T6.6.1.9.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.9.14.1" class="ltx_text" style="font-size:70%;">42.44</span></td>
<td id="S4.T6.6.1.9.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.9.15.1" class="ltx_text" style="font-size:70%;">3.23</span></td>
<td id="S4.T6.6.1.9.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.9.16.1" class="ltx_text" style="font-size:70%;">84.82</span></td>
<td id="S4.T6.6.1.9.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.9.17.1" class="ltx_text" style="font-size:70%;">56.21</span></td>
<td id="S4.T6.6.1.9.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.9.18.1" class="ltx_text" style="font-size:70%;">53.02</span></td>
<td id="S4.T6.6.1.9.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T6.6.1.9.19.1" class="ltx_text ltx_font_bold" style="font-size:70%;">38.72</span></td>
</tr>
<tr id="S4.T6.6.1.10" class="ltx_tr">
<td id="S4.T6.6.1.10.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T6.6.1.10.1.1" class="ltx_text" style="font-size:70%;">Vanilla</span></td>
<td id="S4.T6.6.1.10.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.10.2.1" class="ltx_text" style="font-size:70%;">822.5</span></td>
<td id="S4.T6.6.1.10.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.10.3.1" class="ltx_text" style="font-size:70%;">25.51</span></td>
<td id="S4.T6.6.1.10.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.10.4.1" class="ltx_text" style="font-size:70%;">30.23</span></td>
<td id="S4.T6.6.1.10.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.10.5.1" class="ltx_text" style="font-size:70%;">48.23</span></td>
<td id="S4.T6.6.1.10.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.10.6.1" class="ltx_text" style="font-size:70%;">39.72</span></td>
<td id="S4.T6.6.1.10.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.10.7.1" class="ltx_text" style="font-size:70%;">25.56</span></td>
<td id="S4.T6.6.1.10.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.10.8.1" class="ltx_text" style="font-size:70%;">16.75</span></td>
<td id="S4.T6.6.1.10.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.10.9.1" class="ltx_text" style="font-size:70%;">26.98</span></td>
<td id="S4.T6.6.1.10.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.10.10.1" class="ltx_text" style="font-size:70%;">23.81</span></td>
<td id="S4.T6.6.1.10.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.10.11.1" class="ltx_text" style="font-size:70%;">26.47</span></td>
<td id="S4.T6.6.1.10.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.10.12.1" class="ltx_text" style="font-size:70%;">50.50</span></td>
<td id="S4.T6.6.1.10.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.10.13.1" class="ltx_text" style="font-size:70%;">86.43</span></td>
<td id="S4.T6.6.1.10.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.10.14.1" class="ltx_text" style="font-size:70%;">42.09</span></td>
<td id="S4.T6.6.1.10.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.10.15.1" class="ltx_text" style="font-size:70%;">2.78</span></td>
<td id="S4.T6.6.1.10.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.10.16.1" class="ltx_text" style="font-size:70%;">85.57</span></td>
<td id="S4.T6.6.1.10.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.10.17.1" class="ltx_text" style="font-size:70%;">57.4</span></td>
<td id="S4.T6.6.1.10.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.10.18.1" class="ltx_text" style="font-size:70%;">53.42</span></td>
<td id="S4.T6.6.1.10.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T6.6.1.10.19.1" class="ltx_text" style="font-size:70%;">40.09</span></td>
</tr>
<tr id="S4.T6.6.1.11" class="ltx_tr">
<td id="S4.T6.6.1.11.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T6.6.1.11.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span></td>
<td id="S4.T6.6.1.11.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.11.2.1" class="ltx_text" style="font-size:70%;">822.4</span></td>
<td id="S4.T6.6.1.11.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.11.3.1" class="ltx_text" style="font-size:70%;">25.40</span></td>
<td id="S4.T6.6.1.11.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.11.4.1" class="ltx_text" style="font-size:70%;">30.80</span></td>
<td id="S4.T6.6.1.11.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.11.5.1" class="ltx_text" style="font-size:70%;">48.45</span></td>
<td id="S4.T6.6.1.11.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.11.6.1" class="ltx_text" style="font-size:70%;">39.64</span></td>
<td id="S4.T6.6.1.11.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.11.7.1" class="ltx_text" style="font-size:70%;">26.08</span></td>
<td id="S4.T6.6.1.11.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.11.8.1" class="ltx_text" style="font-size:70%;">16.82</span></td>
<td id="S4.T6.6.1.11.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.11.9.1" class="ltx_text" style="font-size:70%;">27.12</span></td>
<td id="S4.T6.6.1.11.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.11.10.1" class="ltx_text" style="font-size:70%;">23.79</span></td>
<td id="S4.T6.6.1.11.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.11.11.1" class="ltx_text" style="font-size:70%;">26.65</span></td>
<td id="S4.T6.6.1.11.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.11.12.1" class="ltx_text" style="font-size:70%;">53.50</span></td>
<td id="S4.T6.6.1.11.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.11.13.1" class="ltx_text" style="font-size:70%;">86.39</span></td>
<td id="S4.T6.6.1.11.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.11.14.1" class="ltx_text" style="font-size:70%;">43.03</span></td>
<td id="S4.T6.6.1.11.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.11.15.1" class="ltx_text" style="font-size:70%;">3.29</span></td>
<td id="S4.T6.6.1.11.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.11.16.1" class="ltx_text" style="font-size:70%;">86.39</span></td>
<td id="S4.T6.6.1.11.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.11.17.1" class="ltx_text" style="font-size:70%;">56.61</span></td>
<td id="S4.T6.6.1.11.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.11.18.1" class="ltx_text" style="font-size:70%;">53.60</span></td>
<td id="S4.T6.6.1.11.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T6.6.1.11.19.1" class="ltx_text ltx_font_bold" style="font-size:70%;">40.47</span></td>
</tr>
<tr id="S4.T6.6.1.12" class="ltx_tr">
<td id="S4.T6.6.1.12.1" class="ltx_td ltx_align_center ltx_border_t" colspan="18"><span id="S4.T6.6.1.12.1.1" class="ltx_text" style="font-size:70%;">SnapKV</span></td>
<td id="S4.T6.6.1.12.2" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="S4.T6.6.1.13" class="ltx_tr">
<td id="S4.T6.6.1.13.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T6.6.1.13.1.1" class="ltx_text" style="font-size:70%;">Vanilla</span></td>
<td id="S4.T6.6.1.13.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.13.2.1" class="ltx_text" style="font-size:70%;">54.5</span></td>
<td id="S4.T6.6.1.13.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.13.3.1" class="ltx_text" style="font-size:70%;">19.25</span></td>
<td id="S4.T6.6.1.13.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.13.4.1" class="ltx_text" style="font-size:70%;">19.95</span></td>
<td id="S4.T6.6.1.13.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.13.5.1" class="ltx_text" style="font-size:70%;">42.80</span></td>
<td id="S4.T6.6.1.13.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.13.6.1" class="ltx_text" style="font-size:70%;">35.88</span></td>
<td id="S4.T6.6.1.13.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.13.7.1" class="ltx_text" style="font-size:70%;">21.96</span></td>
<td id="S4.T6.6.1.13.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.13.8.1" class="ltx_text" style="font-size:70%;">14.59</span></td>
<td id="S4.T6.6.1.13.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.13.9.1" class="ltx_text" style="font-size:70%;">18.76</span></td>
<td id="S4.T6.6.1.13.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.13.10.1" class="ltx_text" style="font-size:70%;">21.71</span></td>
<td id="S4.T6.6.1.13.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.13.11.1" class="ltx_text" style="font-size:70%;">20.70</span></td>
<td id="S4.T6.6.1.13.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.13.12.1" class="ltx_text" style="font-size:70%;">46.00</span></td>
<td id="S4.T6.6.1.13.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.13.13.1" class="ltx_text" style="font-size:70%;">84.12</span></td>
<td id="S4.T6.6.1.13.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.13.14.1" class="ltx_text" style="font-size:70%;">39.43</span></td>
<td id="S4.T6.6.1.13.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.13.15.1" class="ltx_text" style="font-size:70%;">2.59</span></td>
<td id="S4.T6.6.1.13.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.13.16.1" class="ltx_text" style="font-size:70%;">65.36</span></td>
<td id="S4.T6.6.1.13.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.13.17.1" class="ltx_text" style="font-size:70%;">51.39</span></td>
<td id="S4.T6.6.1.13.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.13.18.1" class="ltx_text" style="font-size:70%;">47.81</span></td>
<td id="S4.T6.6.1.13.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T6.6.1.13.19.1" class="ltx_text" style="font-size:70%;">34.52</span></td>
</tr>
<tr id="S4.T6.6.1.14" class="ltx_tr">
<td id="S4.T6.6.1.14.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T6.6.1.14.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span></td>
<td id="S4.T6.6.1.14.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.14.2.1" class="ltx_text" style="font-size:70%;">54.4</span></td>
<td id="S4.T6.6.1.14.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.14.3.1" class="ltx_text" style="font-size:70%;">20.52</span></td>
<td id="S4.T6.6.1.14.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.14.4.1" class="ltx_text" style="font-size:70%;">21.00</span></td>
<td id="S4.T6.6.1.14.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.14.5.1" class="ltx_text" style="font-size:70%;">42.65</span></td>
<td id="S4.T6.6.1.14.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.14.6.1" class="ltx_text" style="font-size:70%;">37.58</span></td>
<td id="S4.T6.6.1.14.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.14.7.1" class="ltx_text" style="font-size:70%;">22.09</span></td>
<td id="S4.T6.6.1.14.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.14.8.1" class="ltx_text" style="font-size:70%;">15.23</span></td>
<td id="S4.T6.6.1.14.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.14.9.1" class="ltx_text" style="font-size:70%;">19.29</span></td>
<td id="S4.T6.6.1.14.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.14.10.1" class="ltx_text" style="font-size:70%;">22.01</span></td>
<td id="S4.T6.6.1.14.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.14.11.1" class="ltx_text" style="font-size:70%;">21.22</span></td>
<td id="S4.T6.6.1.14.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.14.12.1" class="ltx_text" style="font-size:70%;">47.00</span></td>
<td id="S4.T6.6.1.14.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.14.13.1" class="ltx_text" style="font-size:70%;">83.85</span></td>
<td id="S4.T6.6.1.14.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.14.14.1" class="ltx_text" style="font-size:70%;">39.64</span></td>
<td id="S4.T6.6.1.14.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.14.15.1" class="ltx_text" style="font-size:70%;">3.20</span></td>
<td id="S4.T6.6.1.14.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.14.16.1" class="ltx_text" style="font-size:70%;">67.45</span></td>
<td id="S4.T6.6.1.14.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.14.17.1" class="ltx_text" style="font-size:70%;">51.48</span></td>
<td id="S4.T6.6.1.14.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.14.18.1" class="ltx_text" style="font-size:70%;">48.31</span></td>
<td id="S4.T6.6.1.14.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T6.6.1.14.19.1" class="ltx_text ltx_font_bold" style="font-size:70%;">35.16</span></td>
</tr>
<tr id="S4.T6.6.1.15" class="ltx_tr">
<td id="S4.T6.6.1.15.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T6.6.1.15.1.1" class="ltx_text" style="font-size:70%;">Vanilla</span></td>
<td id="S4.T6.6.1.15.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.15.2.1" class="ltx_text" style="font-size:70%;">208.0</span></td>
<td id="S4.T6.6.1.15.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.15.3.1" class="ltx_text" style="font-size:70%;">23.31</span></td>
<td id="S4.T6.6.1.15.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.15.4.1" class="ltx_text" style="font-size:70%;">27.45</span></td>
<td id="S4.T6.6.1.15.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.15.5.1" class="ltx_text" style="font-size:70%;">48.85</span></td>
<td id="S4.T6.6.1.15.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.15.6.1" class="ltx_text" style="font-size:70%;">38.77</span></td>
<td id="S4.T6.6.1.15.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.15.7.1" class="ltx_text" style="font-size:70%;">23.93</span></td>
<td id="S4.T6.6.1.15.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.15.8.1" class="ltx_text" style="font-size:70%;">16.50</span></td>
<td id="S4.T6.6.1.15.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.15.9.1" class="ltx_text" style="font-size:70%;">23.44</span></td>
<td id="S4.T6.6.1.15.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.15.10.1" class="ltx_text" style="font-size:70%;">23.63</span></td>
<td id="S4.T6.6.1.15.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.15.11.1" class="ltx_text" style="font-size:70%;">24.13</span></td>
<td id="S4.T6.6.1.15.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.15.12.1" class="ltx_text" style="font-size:70%;">66.00</span></td>
<td id="S4.T6.6.1.15.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.15.13.1" class="ltx_text" style="font-size:70%;">86.05</span></td>
<td id="S4.T6.6.1.15.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.15.14.1" class="ltx_text" style="font-size:70%;">41.00</span></td>
<td id="S4.T6.6.1.15.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.15.15.1" class="ltx_text" style="font-size:70%;">2.62</span></td>
<td id="S4.T6.6.1.15.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.15.16.1" class="ltx_text" style="font-size:70%;">87.01</span></td>
<td id="S4.T6.6.1.15.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.15.17.1" class="ltx_text" style="font-size:70%;">56.13</span></td>
<td id="S4.T6.6.1.15.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.15.18.1" class="ltx_text" style="font-size:70%;">52.60</span></td>
<td id="S4.T6.6.1.15.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T6.6.1.15.19.1" class="ltx_text" style="font-size:70%;">40.09</span></td>
</tr>
<tr id="S4.T6.6.1.16" class="ltx_tr">
<td id="S4.T6.6.1.16.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T6.6.1.16.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span></td>
<td id="S4.T6.6.1.16.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.16.2.1" class="ltx_text" style="font-size:70%;">208.0</span></td>
<td id="S4.T6.6.1.16.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.16.3.1" class="ltx_text" style="font-size:70%;">24.27</span></td>
<td id="S4.T6.6.1.16.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.16.4.1" class="ltx_text" style="font-size:70%;">28.46</span></td>
<td id="S4.T6.6.1.16.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.16.5.1" class="ltx_text" style="font-size:70%;">49.26</span></td>
<td id="S4.T6.6.1.16.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.16.6.1" class="ltx_text" style="font-size:70%;">38.13</span></td>
<td id="S4.T6.6.1.16.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.16.7.1" class="ltx_text" style="font-size:70%;">24.22</span></td>
<td id="S4.T6.6.1.16.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.16.8.1" class="ltx_text" style="font-size:70%;">16.92</span></td>
<td id="S4.T6.6.1.16.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.16.9.1" class="ltx_text" style="font-size:70%;">23.59</span></td>
<td id="S4.T6.6.1.16.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.16.10.1" class="ltx_text" style="font-size:70%;">23.70</span></td>
<td id="S4.T6.6.1.16.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.16.11.1" class="ltx_text" style="font-size:70%;">24.46</span></td>
<td id="S4.T6.6.1.16.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.16.12.1" class="ltx_text" style="font-size:70%;">67.50</span></td>
<td id="S4.T6.6.1.16.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.16.13.1" class="ltx_text" style="font-size:70%;">85.90</span></td>
<td id="S4.T6.6.1.16.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.16.14.1" class="ltx_text" style="font-size:70%;">42.51</span></td>
<td id="S4.T6.6.1.16.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.16.15.1" class="ltx_text" style="font-size:70%;">2.92</span></td>
<td id="S4.T6.6.1.16.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.16.16.1" class="ltx_text" style="font-size:70%;">85.32</span></td>
<td id="S4.T6.6.1.16.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.16.17.1" class="ltx_text" style="font-size:70%;">55.89</span></td>
<td id="S4.T6.6.1.16.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.16.18.1" class="ltx_text" style="font-size:70%;">53.35</span></td>
<td id="S4.T6.6.1.16.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T6.6.1.16.19.1" class="ltx_text ltx_font_bold" style="font-size:70%;">40.40</span></td>
</tr>
<tr id="S4.T6.6.1.17" class="ltx_tr">
<td id="S4.T6.6.1.17.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T6.6.1.17.1.1" class="ltx_text" style="font-size:70%;">Vanilla</span></td>
<td id="S4.T6.6.1.17.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.17.2.1" class="ltx_text" style="font-size:70%;">413.0</span></td>
<td id="S4.T6.6.1.17.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.17.3.1" class="ltx_text" style="font-size:70%;">24.24</span></td>
<td id="S4.T6.6.1.17.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.17.4.1" class="ltx_text" style="font-size:70%;">29.53</span></td>
<td id="S4.T6.6.1.17.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.17.5.1" class="ltx_text" style="font-size:70%;">49.13</span></td>
<td id="S4.T6.6.1.17.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.17.6.1" class="ltx_text" style="font-size:70%;">40.48</span></td>
<td id="S4.T6.6.1.17.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.17.7.1" class="ltx_text" style="font-size:70%;">25.05</span></td>
<td id="S4.T6.6.1.17.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.17.8.1" class="ltx_text" style="font-size:70%;">18.74</span></td>
<td id="S4.T6.6.1.17.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.17.9.1" class="ltx_text" style="font-size:70%;">25.46</span></td>
<td id="S4.T6.6.1.17.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.17.10.1" class="ltx_text" style="font-size:70%;">23.64</span></td>
<td id="S4.T6.6.1.17.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.17.11.1" class="ltx_text" style="font-size:70%;">25.60</span></td>
<td id="S4.T6.6.1.17.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.17.12.1" class="ltx_text" style="font-size:70%;">68.00</span></td>
<td id="S4.T6.6.1.17.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.17.13.1" class="ltx_text" style="font-size:70%;">86.14</span></td>
<td id="S4.T6.6.1.17.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.17.14.1" class="ltx_text" style="font-size:70%;">41.42</span></td>
<td id="S4.T6.6.1.17.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.17.15.1" class="ltx_text" style="font-size:70%;">3.03</span></td>
<td id="S4.T6.6.1.17.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.17.16.1" class="ltx_text" style="font-size:70%;">88.55</span></td>
<td id="S4.T6.6.1.17.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.17.17.1" class="ltx_text" style="font-size:70%;">57.08</span></td>
<td id="S4.T6.6.1.17.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.17.18.1" class="ltx_text" style="font-size:70%;">53.86</span></td>
<td id="S4.T6.6.1.17.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T6.6.1.17.19.1" class="ltx_text" style="font-size:70%;">41.25</span></td>
</tr>
<tr id="S4.T6.6.1.18" class="ltx_tr">
<td id="S4.T6.6.1.18.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T6.6.1.18.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span></td>
<td id="S4.T6.6.1.18.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.18.2.1" class="ltx_text" style="font-size:70%;">412.8</span></td>
<td id="S4.T6.6.1.18.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.18.3.1" class="ltx_text" style="font-size:70%;">25.22</span></td>
<td id="S4.T6.6.1.18.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.18.4.1" class="ltx_text" style="font-size:70%;">30.48</span></td>
<td id="S4.T6.6.1.18.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.18.5.1" class="ltx_text" style="font-size:70%;">48.58</span></td>
<td id="S4.T6.6.1.18.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.18.6.1" class="ltx_text" style="font-size:70%;">41.11</span></td>
<td id="S4.T6.6.1.18.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.18.7.1" class="ltx_text" style="font-size:70%;">25.28</span></td>
<td id="S4.T6.6.1.18.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.18.8.1" class="ltx_text" style="font-size:70%;">18.99</span></td>
<td id="S4.T6.6.1.18.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.18.9.1" class="ltx_text" style="font-size:70%;">25.91</span></td>
<td id="S4.T6.6.1.18.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.18.10.1" class="ltx_text" style="font-size:70%;">24.00</span></td>
<td id="S4.T6.6.1.18.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.18.11.1" class="ltx_text" style="font-size:70%;">26.13</span></td>
<td id="S4.T6.6.1.18.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.18.12.1" class="ltx_text" style="font-size:70%;">70.00</span></td>
<td id="S4.T6.6.1.18.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.18.13.1" class="ltx_text" style="font-size:70%;">86.64</span></td>
<td id="S4.T6.6.1.18.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.18.14.1" class="ltx_text" style="font-size:70%;">43.35</span></td>
<td id="S4.T6.6.1.18.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.18.15.1" class="ltx_text" style="font-size:70%;">2.98</span></td>
<td id="S4.T6.6.1.18.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.18.16.1" class="ltx_text" style="font-size:70%;">86.30</span></td>
<td id="S4.T6.6.1.18.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.18.17.1" class="ltx_text" style="font-size:70%;">56.71</span></td>
<td id="S4.T6.6.1.18.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.18.18.1" class="ltx_text" style="font-size:70%;">54.19</span></td>
<td id="S4.T6.6.1.18.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T6.6.1.18.19.1" class="ltx_text ltx_font_bold" style="font-size:70%;">41.62</span></td>
</tr>
<tr id="S4.T6.6.1.19" class="ltx_tr">
<td id="S4.T6.6.1.19.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T6.6.1.19.1.1" class="ltx_text" style="font-size:70%;">Vanilla</span></td>
<td id="S4.T6.6.1.19.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.19.2.1" class="ltx_text" style="font-size:70%;">822.5</span></td>
<td id="S4.T6.6.1.19.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.19.3.1" class="ltx_text" style="font-size:70%;">24.84</span></td>
<td id="S4.T6.6.1.19.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.19.4.1" class="ltx_text" style="font-size:70%;">31.90</span></td>
<td id="S4.T6.6.1.19.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.19.5.1" class="ltx_text" style="font-size:70%;">48.16</span></td>
<td id="S4.T6.6.1.19.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.19.6.1" class="ltx_text" style="font-size:70%;">41.32</span></td>
<td id="S4.T6.6.1.19.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.19.7.1" class="ltx_text" style="font-size:70%;">26.77</span></td>
<td id="S4.T6.6.1.19.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.19.8.1" class="ltx_text" style="font-size:70%;">19.49</span></td>
<td id="S4.T6.6.1.19.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.19.9.1" class="ltx_text" style="font-size:70%;">28.23</span></td>
<td id="S4.T6.6.1.19.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.19.10.1" class="ltx_text" style="font-size:70%;">24.63</span></td>
<td id="S4.T6.6.1.19.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.19.11.1" class="ltx_text" style="font-size:70%;">26.41</span></td>
<td id="S4.T6.6.1.19.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.19.12.1" class="ltx_text" style="font-size:70%;">70.00</span></td>
<td id="S4.T6.6.1.19.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.19.13.1" class="ltx_text" style="font-size:70%;">86.32</span></td>
<td id="S4.T6.6.1.19.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.19.14.1" class="ltx_text" style="font-size:70%;">41.83</span></td>
<td id="S4.T6.6.1.19.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.19.15.1" class="ltx_text" style="font-size:70%;">2.91</span></td>
<td id="S4.T6.6.1.19.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.19.16.1" class="ltx_text" style="font-size:70%;">88.06</span></td>
<td id="S4.T6.6.1.19.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.19.17.1" class="ltx_text" style="font-size:70%;">56.98</span></td>
<td id="S4.T6.6.1.19.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T6.6.1.19.18.1" class="ltx_text" style="font-size:70%;">53.74</span></td>
<td id="S4.T6.6.1.19.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T6.6.1.19.19.1" class="ltx_text" style="font-size:70%;">41.97</span></td>
</tr>
<tr id="S4.T6.6.1.20" class="ltx_tr">
<td id="S4.T6.6.1.20.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb"><span id="S4.T6.6.1.20.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span></td>
<td id="S4.T6.6.1.20.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T6.6.1.20.2.1" class="ltx_text" style="font-size:70%;">822.4</span></td>
<td id="S4.T6.6.1.20.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T6.6.1.20.3.1" class="ltx_text" style="font-size:70%;">25.77</span></td>
<td id="S4.T6.6.1.20.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T6.6.1.20.4.1" class="ltx_text" style="font-size:70%;">32.67</span></td>
<td id="S4.T6.6.1.20.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T6.6.1.20.5.1" class="ltx_text" style="font-size:70%;">48.7</span></td>
<td id="S4.T6.6.1.20.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T6.6.1.20.6.1" class="ltx_text" style="font-size:70%;">41.06</span></td>
<td id="S4.T6.6.1.20.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T6.6.1.20.7.1" class="ltx_text" style="font-size:70%;">27.07</span></td>
<td id="S4.T6.6.1.20.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T6.6.1.20.8.1" class="ltx_text" style="font-size:70%;">19.14</span></td>
<td id="S4.T6.6.1.20.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T6.6.1.20.9.1" class="ltx_text" style="font-size:70%;">28.91</span></td>
<td id="S4.T6.6.1.20.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T6.6.1.20.10.1" class="ltx_text" style="font-size:70%;">24.37</span></td>
<td id="S4.T6.6.1.20.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T6.6.1.20.11.1" class="ltx_text" style="font-size:70%;">26.88</span></td>
<td id="S4.T6.6.1.20.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T6.6.1.20.12.1" class="ltx_text" style="font-size:70%;">70.00</span></td>
<td id="S4.T6.6.1.20.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T6.6.1.20.13.1" class="ltx_text" style="font-size:70%;">86.37</span></td>
<td id="S4.T6.6.1.20.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T6.6.1.20.14.1" class="ltx_text" style="font-size:70%;">42.75</span></td>
<td id="S4.T6.6.1.20.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T6.6.1.20.15.1" class="ltx_text" style="font-size:70%;">3.61</span></td>
<td id="S4.T6.6.1.20.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T6.6.1.20.16.1" class="ltx_text" style="font-size:70%;">87.38</span></td>
<td id="S4.T6.6.1.20.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T6.6.1.20.17.1" class="ltx_text" style="font-size:70%;">57.21</span></td>
<td id="S4.T6.6.1.20.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T6.6.1.20.18.1" class="ltx_text" style="font-size:70%;">54.44</span></td>
<td id="S4.T6.6.1.20.19" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_bb"><span id="S4.T6.6.1.20.19.1" class="ltx_text ltx_font_bold" style="font-size:70%;">42.27</span></td>
</tr>
</table>
</span></div>
</figure>
<section id="S4.SS4.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Impact of Different Recent Sizes.</h4>

<div id="S4.SS4.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S4.SS4.SSS0.Px1.p1.6" class="ltx_p">Preserve the most recent KV embeddings <cite class="ltx_cite ltx_citemacro_citep">(<a href="#bib.bib56" title="" class="ltx_ref">Zhang et al., 2024c, </a>; Li et al.,, <a href="#bib.bib29" title="" class="ltx_ref">2024</a>)</cite> is important for maintaining the performance of LLMs after KV cache compression. Note that a tradeoff exists: increasing the recent-size allows more infomation to be propagated, while increasing the cache size to be stored. To study its impacts, we evaluate the performance produced by three recent-size, namely <math id="S4.SS4.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="0" display="inline"><semantics id="S4.SS4.SSS0.Px1.p1.1.m1.1a"><mn id="S4.SS4.SSS0.Px1.p1.1.m1.1.1" xref="S4.SS4.SSS0.Px1.p1.1.m1.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="S4.SS4.SSS0.Px1.p1.1.m1.1b"><cn type="integer" id="S4.SS4.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S4.SS4.SSS0.Px1.p1.1.m1.1.1">0</cn></annotation-xml></semantics></math>, <math id="S4.SS4.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="32" display="inline"><semantics id="S4.SS4.SSS0.Px1.p1.2.m2.1a"><mn id="S4.SS4.SSS0.Px1.p1.2.m2.1.1" xref="S4.SS4.SSS0.Px1.p1.2.m2.1.1.cmml">32</mn><annotation-xml encoding="MathML-Content" id="S4.SS4.SSS0.Px1.p1.2.m2.1b"><cn type="integer" id="S4.SS4.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S4.SS4.SSS0.Px1.p1.2.m2.1.1">32</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.SSS0.Px1.p1.2.m2.1c">32</annotation></semantics></math> and <math id="S4.SS4.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="128" display="inline"><semantics id="S4.SS4.SSS0.Px1.p1.3.m3.1a"><mn id="S4.SS4.SSS0.Px1.p1.3.m3.1.1" xref="S4.SS4.SSS0.Px1.p1.3.m3.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="S4.SS4.SSS0.Px1.p1.3.m3.1b"><cn type="integer" id="S4.SS4.SSS0.Px1.p1.3.m3.1.1.cmml" xref="S4.SS4.SSS0.Px1.p1.3.m3.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.SSS0.Px1.p1.3.m3.1c">128</annotation></semantics></math>, on LongBench, using Mistral-7B-Instruct-v0.2 as the basleline model. The results are summarized in Table <a href="#S4.T5" title="Table 5 ‣ 4.4 Ablation Studies ‣ 4 Experiment Results ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>. One can observe that a recent-size of 32 yields superior performance than <math id="S4.SS4.SSS0.Px1.p1.4.m4.1" class="ltx_Math" alttext="0" display="inline"><semantics id="S4.SS4.SSS0.Px1.p1.4.m4.1a"><mn id="S4.SS4.SSS0.Px1.p1.4.m4.1.1" xref="S4.SS4.SSS0.Px1.p1.4.m4.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="S4.SS4.SSS0.Px1.p1.4.m4.1b"><cn type="integer" id="S4.SS4.SSS0.Px1.p1.4.m4.1.1.cmml" xref="S4.SS4.SSS0.Px1.p1.4.m4.1.1">0</cn></annotation-xml></semantics></math> in terms of averaged score on LongBench which demonstrates the importance of keeping the most recent KVs. On the other hand, the performance of <math id="S4.SS4.SSS0.Px1.p1.5.m5.1" class="ltx_Math" alttext="32" display="inline"><semantics id="S4.SS4.SSS0.Px1.p1.5.m5.1a"><mn id="S4.SS4.SSS0.Px1.p1.5.m5.1.1" xref="S4.SS4.SSS0.Px1.p1.5.m5.1.1.cmml">32</mn><annotation-xml encoding="MathML-Content" id="S4.SS4.SSS0.Px1.p1.5.m5.1b"><cn type="integer" id="S4.SS4.SSS0.Px1.p1.5.m5.1.1.cmml" xref="S4.SS4.SSS0.Px1.p1.5.m5.1.1">32</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.SSS0.Px1.p1.5.m5.1c">32</annotation></semantics></math> and <math id="S4.SS4.SSS0.Px1.p1.6.m6.1" class="ltx_Math" alttext="128" display="inline"><semantics id="S4.SS4.SSS0.Px1.p1.6.m6.1a"><mn id="S4.SS4.SSS0.Px1.p1.6.m6.1.1" xref="S4.SS4.SSS0.Px1.p1.6.m6.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="S4.SS4.SSS0.Px1.p1.6.m6.1b"><cn type="integer" id="S4.SS4.SSS0.Px1.p1.6.m6.1.1.cmml" xref="S4.SS4.SSS0.Px1.p1.6.m6.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.SSS0.Px1.p1.6.m6.1c">128</annotation></semantics></math> is negligible, suggesting that maintaining the most recent 32 KVs suffices to preserve optimal performance.</p>
</div>
<figure id="S4.F5" class="ltx_figure ltx_align_floatright"><img src="/html/2407.21018/assets/x9.png" id="S4.F5.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="207" height="145" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F5.2.1.1" class="ltx_text" style="font-size:90%;">Figure 5</span>: </span><span id="S4.F5.3.2" class="ltx_text" style="font-size:90%;">Performance comparisons under the same KV cache memory consumption.</span></figcaption>
</figure>
</section>
<section id="S4.SS4.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Comparison Under the Same Memory Consumption.</h4>

<div id="S4.SS4.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS4.SSS0.Px2.p1.4" class="ltx_p">To ensure a fair comparison, we adjust the KV cache size of H2O or SnapKV to match the memory usage of H2O with ThinK or SnapKV with ThinK.
For example, the KV cache size of H2O with ThinK is <math id="S4.SS4.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="128" display="inline"><semantics id="S4.SS4.SSS0.Px2.p1.1.m1.1a"><mn id="S4.SS4.SSS0.Px2.p1.1.m1.1.1" xref="S4.SS4.SSS0.Px2.p1.1.m1.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="S4.SS4.SSS0.Px2.p1.1.m1.1b"><cn type="integer" id="S4.SS4.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S4.SS4.SSS0.Px2.p1.1.m1.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.SSS0.Px2.p1.1.m1.1c">128</annotation></semantics></math>. Due to channel pruning of the key cache, the memory consumption of H2O with ThinK at a <math id="S4.SS4.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="128" display="inline"><semantics id="S4.SS4.SSS0.Px2.p1.2.m2.1a"><mn id="S4.SS4.SSS0.Px2.p1.2.m2.1.1" xref="S4.SS4.SSS0.Px2.p1.2.m2.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="S4.SS4.SSS0.Px2.p1.2.m2.1b"><cn type="integer" id="S4.SS4.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S4.SS4.SSS0.Px2.p1.2.m2.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.SSS0.Px2.p1.2.m2.1c">128</annotation></semantics></math> KV-size is lower than H2O at a 128 KV-size.
Therefore, the KV-size of H2O is adjusted from <math id="S4.SS4.SSS0.Px2.p1.3.m3.1" class="ltx_Math" alttext="128" display="inline"><semantics id="S4.SS4.SSS0.Px2.p1.3.m3.1a"><mn id="S4.SS4.SSS0.Px2.p1.3.m3.1.1" xref="S4.SS4.SSS0.Px2.p1.3.m3.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="S4.SS4.SSS0.Px2.p1.3.m3.1b"><cn type="integer" id="S4.SS4.SSS0.Px2.p1.3.m3.1.1.cmml" xref="S4.SS4.SSS0.Px2.p1.3.m3.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.SSS0.Px2.p1.3.m3.1c">128</annotation></semantics></math> to <math id="S4.SS4.SSS0.Px2.p1.4.m4.1" class="ltx_Math" alttext="109" display="inline"><semantics id="S4.SS4.SSS0.Px2.p1.4.m4.1a"><mn id="S4.SS4.SSS0.Px2.p1.4.m4.1.1" xref="S4.SS4.SSS0.Px2.p1.4.m4.1.1.cmml">109</mn><annotation-xml encoding="MathML-Content" id="S4.SS4.SSS0.Px2.p1.4.m4.1b"><cn type="integer" id="S4.SS4.SSS0.Px2.p1.4.m4.1.1.cmml" xref="S4.SS4.SSS0.Px2.p1.4.m4.1.1">109</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.SSS0.Px2.p1.4.m4.1c">109</annotation></semantics></math>.
Table <a href="#S4.T6" title="Table 6 ‣ 4.4 Ablation Studies ‣ 4 Experiment Results ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> and Figure <a href="#S4.F5" title="Figure 5 ‣ Impact of Different Recent Sizes. ‣ 4.4 Ablation Studies ‣ 4 Experiment Results ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> present the results of these comparisons on LongBench.
As the results show, H2O or SnapKV with ThinK deliver superior performance compared to H2O or SnapKV without ThinK while maintaining the same memory usage.
This demonstrates that KV cache compression methods, combined with our query-based channel pruning, can compress the KV cache more effectively and utilize memory more efficiently.</p>
</div>
</section>
<section id="S4.SS4.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Pruning Channels of Both Key and Value Cache.</h4>

<div id="S4.SS4.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="S4.SS4.SSS0.Px3.p1.1" class="ltx_p">In our previous study, we evaluate methods with pruning channels of the key cache. In this part, we explore the impact of pruning channels of the value cache (Appendix <a href="#A1.SS1" title="A.1 Value Cache Pruning ‣ Appendix A Additional Experimental Details ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.1</span></a>). Specifically, for KV cache compression methods, we prune channels of the key cache at one pruning ratio and channels of the value cache at another pruning ratio.
Table <a href="#S4.T7" title="Table 7 ‣ Pruning Channels of Both Key and Value Cache. ‣ 4.4 Ablation Studies ‣ 4 Experiment Results ‣ ThinK: Thinner Key Cache by Query-Driven Pruning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a> presents the results with LLaMA3-8B and Mistral-7B on LongBench.
When testing on the base model LLaMA3-8B, H2O or SnapKV with key and value channel pruning can perform on par with H2O or SnapKV without channel pruning. In some cases, H2O or SnapKV with key and value channel pruning may even perform better. Pruning the channels of the value cache over the base model Mistral-7B experiences a slight performance drop. Despite this, note that KV cache with additional value channel pruning can further reduce memory usage.</p>
</div>
<figure id="S4.T7" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table"><span id="S4.T7.19.6.1" class="ltx_text" style="font-size:129%;">Table 7</span>: </span><span id="S4.T7.10.5" class="ltx_text" style="font-size:129%;">Performance comparison of pruning both K and V cache with different pruning ratios on LongBench.
H2O <math id="S4.T7.6.1.m1.1" class="ltx_Math" alttext="+" display="inline"><semantics id="S4.T7.6.1.m1.1b"><mo id="S4.T7.6.1.m1.1.1" xref="S4.T7.6.1.m1.1.1.cmml">+</mo><annotation-xml encoding="MathML-Content" id="S4.T7.6.1.m1.1c"><plus id="S4.T7.6.1.m1.1.1.cmml" xref="S4.T7.6.1.m1.1.1"></plus></annotation-xml><annotation encoding="application/x-tex" id="S4.T7.6.1.m1.1d">+</annotation></semantics></math> ThinK(<math id="S4.T7.7.2.m2.1" class="ltx_Math" alttext="\lambda_{1}" display="inline"><semantics id="S4.T7.7.2.m2.1b"><msub id="S4.T7.7.2.m2.1.1" xref="S4.T7.7.2.m2.1.1.cmml"><mi id="S4.T7.7.2.m2.1.1.2" xref="S4.T7.7.2.m2.1.1.2.cmml">λ</mi><mn id="S4.T7.7.2.m2.1.1.3" xref="S4.T7.7.2.m2.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T7.7.2.m2.1c"><apply id="S4.T7.7.2.m2.1.1.cmml" xref="S4.T7.7.2.m2.1.1"><csymbol cd="ambiguous" id="S4.T7.7.2.m2.1.1.1.cmml" xref="S4.T7.7.2.m2.1.1">subscript</csymbol><ci id="S4.T7.7.2.m2.1.1.2.cmml" xref="S4.T7.7.2.m2.1.1.2">𝜆</ci><cn type="integer" id="S4.T7.7.2.m2.1.1.3.cmml" xref="S4.T7.7.2.m2.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T7.7.2.m2.1d">\lambda_{1}</annotation></semantics></math>+<math id="S4.T7.8.3.m3.1" class="ltx_Math" alttext="\lambda_{2}" display="inline"><semantics id="S4.T7.8.3.m3.1b"><msub id="S4.T7.8.3.m3.1.1" xref="S4.T7.8.3.m3.1.1.cmml"><mi id="S4.T7.8.3.m3.1.1.2" xref="S4.T7.8.3.m3.1.1.2.cmml">λ</mi><mn id="S4.T7.8.3.m3.1.1.3" xref="S4.T7.8.3.m3.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T7.8.3.m3.1c"><apply id="S4.T7.8.3.m3.1.1.cmml" xref="S4.T7.8.3.m3.1.1"><csymbol cd="ambiguous" id="S4.T7.8.3.m3.1.1.1.cmml" xref="S4.T7.8.3.m3.1.1">subscript</csymbol><ci id="S4.T7.8.3.m3.1.1.2.cmml" xref="S4.T7.8.3.m3.1.1.2">𝜆</ci><cn type="integer" id="S4.T7.8.3.m3.1.1.3.cmml" xref="S4.T7.8.3.m3.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T7.8.3.m3.1d">\lambda_{2}</annotation></semantics></math>) indicates that H2O has <math id="S4.T7.9.4.m4.1" class="ltx_Math" alttext="\lambda_{1}" display="inline"><semantics id="S4.T7.9.4.m4.1b"><msub id="S4.T7.9.4.m4.1.1" xref="S4.T7.9.4.m4.1.1.cmml"><mi id="S4.T7.9.4.m4.1.1.2" xref="S4.T7.9.4.m4.1.1.2.cmml">λ</mi><mn id="S4.T7.9.4.m4.1.1.3" xref="S4.T7.9.4.m4.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T7.9.4.m4.1c"><apply id="S4.T7.9.4.m4.1.1.cmml" xref="S4.T7.9.4.m4.1.1"><csymbol cd="ambiguous" id="S4.T7.9.4.m4.1.1.1.cmml" xref="S4.T7.9.4.m4.1.1">subscript</csymbol><ci id="S4.T7.9.4.m4.1.1.2.cmml" xref="S4.T7.9.4.m4.1.1.2">𝜆</ci><cn type="integer" id="S4.T7.9.4.m4.1.1.3.cmml" xref="S4.T7.9.4.m4.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T7.9.4.m4.1d">\lambda_{1}</annotation></semantics></math>% of the Key cache channels pruned and <math id="S4.T7.10.5.m5.1" class="ltx_Math" alttext="\lambda_{2}" display="inline"><semantics id="S4.T7.10.5.m5.1b"><msub id="S4.T7.10.5.m5.1.1" xref="S4.T7.10.5.m5.1.1.cmml"><mi id="S4.T7.10.5.m5.1.1.2" xref="S4.T7.10.5.m5.1.1.2.cmml">λ</mi><mn id="S4.T7.10.5.m5.1.1.3" xref="S4.T7.10.5.m5.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T7.10.5.m5.1c"><apply id="S4.T7.10.5.m5.1.1.cmml" xref="S4.T7.10.5.m5.1.1"><csymbol cd="ambiguous" id="S4.T7.10.5.m5.1.1.1.cmml" xref="S4.T7.10.5.m5.1.1">subscript</csymbol><ci id="S4.T7.10.5.m5.1.1.2.cmml" xref="S4.T7.10.5.m5.1.1.2">𝜆</ci><cn type="integer" id="S4.T7.10.5.m5.1.1.3.cmml" xref="S4.T7.10.5.m5.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T7.10.5.m5.1d">\lambda_{2}</annotation></semantics></math>% of the Value cache channels pruned.
</span></figcaption>
<div id="S4.T7.20" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:679.2pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-43.4pt,67.9pt) scale(0.833317718368972,0.833317718368972) ;">
<table id="S4.T7.20.1" class="ltx_tabular ltx_align_middle">
<tr id="S4.T7.20.1.1" class="ltx_tr">
<td id="S4.T7.20.1.1.1" class="ltx_td ltx_border_r ltx_border_tt"></td>
<td id="S4.T7.20.1.1.2" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt" rowspan="2"><span id="S4.T7.20.1.1.2.1" class="ltx_text" style="font-size:70%;">Method</span></td>
<td id="S4.T7.20.1.1.3" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="S4.T7.20.1.1.3.1" class="ltx_text" style="font-size:70%;">Single-Document QA</span></td>
<td id="S4.T7.20.1.1.4" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="S4.T7.20.1.1.4.1" class="ltx_text" style="font-size:70%;">Multi-Document QA</span></td>
<td id="S4.T7.20.1.1.5" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="S4.T7.20.1.1.5.1" class="ltx_text" style="font-size:70%;">Summarization</span></td>
<td id="S4.T7.20.1.1.6" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="S4.T7.20.1.1.6.1" class="ltx_text" style="font-size:70%;">Few-shot Learning</span></td>
<td id="S4.T7.20.1.1.7" class="ltx_td ltx_align_center ltx_border_tt" colspan="2"><span id="S4.T7.20.1.1.7.1" class="ltx_text" style="font-size:70%;">Synthetic</span></td>
<td id="S4.T7.20.1.1.8" class="ltx_td ltx_align_center ltx_border_tt" colspan="2"><span id="S4.T7.20.1.1.8.1" class="ltx_text" style="font-size:70%;">Code</span></td>
<td id="S4.T7.20.1.1.9" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_tt" rowspan="3"><span id="S4.T7.20.1.1.9.1" class="ltx_text" style="font-size:70%;">Avg.</span></td>
</tr>
<tr id="S4.T7.20.1.2" class="ltx_tr">
<td id="S4.T7.20.1.2.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.2.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T7.20.1.2.2.1" class="ltx_inline-block ltx_transformed_outer" style="width:24.8pt;height:17.8pt;vertical-align:-7.2pt;"><span class="ltx_transformed_inner" style="width:25.1pt;transform:translate(-0.15pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T7.20.1.2.2.1.1" class="ltx_p"><span id="S4.T7.20.1.2.2.1.1.1" class="ltx_text" style="font-size:70%;">NrtvQA</span></p>
</span></div>
</td>
<td id="S4.T7.20.1.2.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T7.20.1.2.3.1" class="ltx_inline-block ltx_transformed_outer" style="width:21.8pt;height:16.2pt;vertical-align:-6.4pt;"><span class="ltx_transformed_inner" style="width:21.6pt;transform:translate(0.09pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T7.20.1.2.3.1.1" class="ltx_p"><span id="S4.T7.20.1.2.3.1.1.1" class="ltx_text" style="font-size:70%;">Qasper</span></p>
</span></div>
</td>
<td id="S4.T7.20.1.2.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T7.20.1.2.4.1" class="ltx_inline-block ltx_transformed_outer" style="width:20.0pt;height:14.3pt;vertical-align:-4.8pt;"><span class="ltx_transformed_inner" style="width:20.3pt;transform:translate(-0.17pt,0pt) rotate(-30deg) ;">
<p id="S4.T7.20.1.2.4.1.1" class="ltx_p"><span id="S4.T7.20.1.2.4.1.1.1" class="ltx_text" style="font-size:70%;">MF-en</span></p>
</span></div>
</td>
<td id="S4.T7.20.1.2.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T7.20.1.2.5.1" class="ltx_inline-block ltx_transformed_outer" style="width:31.2pt;height:21.6pt;vertical-align:-9.1pt;"><span class="ltx_transformed_inner" style="width:32.5pt;transform:translate(-0.64pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T7.20.1.2.5.1.1" class="ltx_p"><span id="S4.T7.20.1.2.5.1.1.1" class="ltx_text" style="font-size:70%;">HotpotQA</span></p>
</span></div>
</td>
<td id="S4.T7.20.1.2.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T7.20.1.2.6.1" class="ltx_inline-block ltx_transformed_outer" style="width:33.8pt;height:23.1pt;vertical-align:-9.8pt;"><span class="ltx_transformed_inner" style="width:35.4pt;transform:translate(-0.82pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T7.20.1.2.6.1.1" class="ltx_p"><span id="S4.T7.20.1.2.6.1.1.1" class="ltx_text" style="font-size:70%;">2WikiMQA</span></p>
</span></div>
</td>
<td id="S4.T7.20.1.2.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T7.20.1.2.7.1" class="ltx_inline-block ltx_transformed_outer" style="width:25.3pt;height:18.2pt;vertical-align:-7.4pt;"><span class="ltx_transformed_inner" style="width:25.7pt;transform:translate(-0.19pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T7.20.1.2.7.1.1" class="ltx_p"><span id="S4.T7.20.1.2.7.1.1.1" class="ltx_text" style="font-size:70%;">Musique</span></p>
</span></div>
</td>
<td id="S4.T7.20.1.2.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T7.20.1.2.8.1" class="ltx_inline-block ltx_transformed_outer" style="width:32.3pt;height:22.2pt;vertical-align:-9.4pt;"><span class="ltx_transformed_inner" style="width:33.8pt;transform:translate(-0.73pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T7.20.1.2.8.1.1" class="ltx_p"><span id="S4.T7.20.1.2.8.1.1.1" class="ltx_text" style="font-size:70%;">GovReport</span></p>
</span></div>
</td>
<td id="S4.T7.20.1.2.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T7.20.1.2.9.1" class="ltx_inline-block ltx_transformed_outer" style="width:25.1pt;height:18pt;vertical-align:-7.3pt;"><span class="ltx_transformed_inner" style="width:25.5pt;transform:translate(-0.17pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T7.20.1.2.9.1.1" class="ltx_p"><span id="S4.T7.20.1.2.9.1.1.1" class="ltx_text" style="font-size:70%;">QMSum</span></p>
</span></div>
</td>
<td id="S4.T7.20.1.2.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T7.20.1.2.10.1" class="ltx_inline-block ltx_transformed_outer" style="width:31.1pt;height:20.7pt;vertical-align:-7.9pt;"><span class="ltx_transformed_inner" style="width:33.1pt;transform:translate(-1pt,0pt) rotate(-30deg) ;">
<p id="S4.T7.20.1.2.10.1.1" class="ltx_p"><span id="S4.T7.20.1.2.10.1.1.1" class="ltx_text" style="font-size:70%;">MultiNews</span></p>
</span></div>
</td>
<td id="S4.T7.20.1.2.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T7.20.1.2.11.1" class="ltx_inline-block ltx_transformed_outer" style="width:19.7pt;height:14.2pt;vertical-align:-4.7pt;"><span class="ltx_transformed_inner" style="width:20.0pt;transform:translate(-0.15pt,0pt) rotate(-30deg) ;">
<p id="S4.T7.20.1.2.11.1.1" class="ltx_p"><span id="S4.T7.20.1.2.11.1.1.1" class="ltx_text" style="font-size:70%;">TREC</span></p>
</span></div>
</td>
<td id="S4.T7.20.1.2.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T7.20.1.2.12.1" class="ltx_inline-block ltx_transformed_outer" style="width:28.2pt;height:19.8pt;vertical-align:-8.2pt;"><span class="ltx_transformed_inner" style="width:29.0pt;transform:translate(-0.41pt,2.04pt) rotate(-30deg) ;">
<p id="S4.T7.20.1.2.12.1.1" class="ltx_p"><span id="S4.T7.20.1.2.12.1.1.1" class="ltx_text" style="font-size:70%;">TriviaQA</span></p>
</span></div>
</td>
<td id="S4.T7.20.1.2.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T7.20.1.2.13.1" class="ltx_inline-block ltx_transformed_outer" style="width:27.7pt;height:18.8pt;vertical-align:-7.0pt;"><span class="ltx_transformed_inner" style="width:29.2pt;transform:translate(-0.76pt,0pt) rotate(-30deg) ;">
<p id="S4.T7.20.1.2.13.1.1" class="ltx_p"><span id="S4.T7.20.1.2.13.1.1.1" class="ltx_text" style="font-size:70%;">SAMSum</span></p>
</span></div>
</td>
<td id="S4.T7.20.1.2.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T7.20.1.2.14.1" class="ltx_inline-block ltx_transformed_outer" style="width:22.9pt;height:16pt;vertical-align:-5.6pt;"><span class="ltx_transformed_inner" style="width:23.6pt;transform:translate(-0.39pt,0pt) rotate(-30deg) ;">
<p id="S4.T7.20.1.2.14.1.1" class="ltx_p"><span id="S4.T7.20.1.2.14.1.1.1" class="ltx_text" style="font-size:70%;">PCount</span></p>
</span></div>
</td>
<td id="S4.T7.20.1.2.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T7.20.1.2.15.1" class="ltx_inline-block ltx_transformed_outer" style="width:19.7pt;height:14.2pt;vertical-align:-4.7pt;"><span class="ltx_transformed_inner" style="width:20.0pt;transform:translate(-0.15pt,0pt) rotate(-30deg) ;">
<p id="S4.T7.20.1.2.15.1.1" class="ltx_p"><span id="S4.T7.20.1.2.15.1.1.1" class="ltx_text" style="font-size:70%;">PRe</span></p>
</span></div>
</td>
<td id="S4.T7.20.1.2.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T7.20.1.2.16.1" class="ltx_inline-block ltx_transformed_outer" style="width:17.6pt;height:13pt;vertical-align:-4.1pt;"><span class="ltx_transformed_inner" style="width:17.6pt;transform:translate(0.02pt,0pt) rotate(-30deg) ;">
<p id="S4.T7.20.1.2.16.1.1" class="ltx_p"><span id="S4.T7.20.1.2.16.1.1.1" class="ltx_text" style="font-size:70%;">Lcc</span></p>
</span></div>
</td>
<td id="S4.T7.20.1.2.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t">
<div id="S4.T7.20.1.2.17.1" class="ltx_inline-block ltx_transformed_outer" style="width:21.3pt;height:15pt;vertical-align:-5.1pt;"><span class="ltx_transformed_inner" style="width:21.9pt;transform:translate(-0.27pt,0pt) rotate(-30deg) ;">
<p id="S4.T7.20.1.2.17.1.1" class="ltx_p"><span id="S4.T7.20.1.2.17.1.1.1" class="ltx_text" style="font-size:70%;">RB-P</span></p>
</span></div>
</td>
</tr>
<tr id="S4.T7.20.1.3" class="ltx_tr">
<td id="S4.T7.20.1.3.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" rowspan="3"><span id="S4.T7.20.1.3.1.1" class="ltx_text" style="font-size:70%;">
<span id="S4.T7.20.1.3.1.1.1" class="ltx_inline-block ltx_transformed_outer" style="width:4.8pt;height:40.6pt;vertical-align:-17.9pt;"><span class="ltx_transformed_inner" style="width:40.5pt;transform:translate(-17.88pt,0pt) rotate(-90deg) ;">
<span id="S4.T7.20.1.3.1.1.1.1" class="ltx_p">LLaMA3 8B</span>
</span></span></span></td>
<td id="S4.T7.20.1.3.2" class="ltx_td ltx_align_center ltx_border_t" colspan="18"><span id="S4.T7.20.1.3.2.1" class="ltx_text" style="font-size:70%;">KV-size 128</span></td>
</tr>
<tr id="S4.T7.20.1.4" class="ltx_tr">
<td id="S4.T7.20.1.4.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T7.20.1.4.1.1" class="ltx_text" style="font-size:70%;">H2O</span></td>
<td id="S4.T7.20.1.4.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.4.2.1" class="ltx_text" style="font-size:70%;">22.12</span></td>
<td id="S4.T7.20.1.4.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.4.3.1" class="ltx_text" style="font-size:70%;">13.20</span></td>
<td id="S4.T7.20.1.4.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.4.4.1" class="ltx_text" style="font-size:70%;">31.61</span></td>
<td id="S4.T7.20.1.4.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.4.5.1" class="ltx_text" style="font-size:70%;">37.79</span></td>
<td id="S4.T7.20.1.4.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.4.6.1" class="ltx_text" style="font-size:70%;">32.71</span></td>
<td id="S4.T7.20.1.4.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.4.7.1" class="ltx_text" style="font-size:70%;">18.45</span></td>
<td id="S4.T7.20.1.4.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.4.8.1" class="ltx_text" style="font-size:70%;">20.32</span></td>
<td id="S4.T7.20.1.4.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.4.9.1" class="ltx_text" style="font-size:70%;">22.02</span></td>
<td id="S4.T7.20.1.4.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.4.10.1" class="ltx_text" style="font-size:70%;">21.10</span></td>
<td id="S4.T7.20.1.4.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.4.11.1" class="ltx_text" style="font-size:70%;">38.50</span></td>
<td id="S4.T7.20.1.4.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.4.12.1" class="ltx_text" style="font-size:70%;">87.75</span></td>
<td id="S4.T7.20.1.4.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.4.13.1" class="ltx_text" style="font-size:70%;">39.14</span></td>
<td id="S4.T7.20.1.4.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.4.14.1" class="ltx_text" style="font-size:70%;">5.83</span></td>
<td id="S4.T7.20.1.4.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.4.15.1" class="ltx_text" style="font-size:70%;">69.50</span></td>
<td id="S4.T7.20.1.4.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.4.16.1" class="ltx_text" style="font-size:70%;">55.06</span></td>
<td id="S4.T7.20.1.4.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.4.17.1" class="ltx_text" style="font-size:70%;">50.97</span></td>
<td id="S4.T7.20.1.4.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.4.18.1" class="ltx_text" style="font-size:70%;">35.38</span></td>
</tr>
<tr id="S4.T7.20.1.5" class="ltx_tr">
<td id="S4.T7.20.1.5.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T7.20.1.5.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T7.20.1.5.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T7.20.1.5.1.3" class="ltx_text" style="font-size:70%;">(30+30)</span>
</td>
<td id="S4.T7.20.1.5.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.5.2.1" class="ltx_text" style="font-size:70%;">23.71</span></td>
<td id="S4.T7.20.1.5.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.5.3.1" class="ltx_text" style="font-size:70%;">13.65</span></td>
<td id="S4.T7.20.1.5.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.5.4.1" class="ltx_text" style="font-size:70%;">33.08</span></td>
<td id="S4.T7.20.1.5.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.5.5.1" class="ltx_text" style="font-size:70%;">41.86</span></td>
<td id="S4.T7.20.1.5.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.5.6.1" class="ltx_text" style="font-size:70%;">29.88</span></td>
<td id="S4.T7.20.1.5.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.5.7.1" class="ltx_text" style="font-size:70%;">18.04</span></td>
<td id="S4.T7.20.1.5.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.5.8.1" class="ltx_text" style="font-size:70%;">19.60</span></td>
<td id="S4.T7.20.1.5.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.5.9.1" class="ltx_text" style="font-size:70%;">21.65</span></td>
<td id="S4.T7.20.1.5.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.5.10.1" class="ltx_text" style="font-size:70%;">20.26</span></td>
<td id="S4.T7.20.1.5.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.5.11.1" class="ltx_text" style="font-size:70%;">38.00</span></td>
<td id="S4.T7.20.1.5.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.5.12.1" class="ltx_text" style="font-size:70%;">86.08</span></td>
<td id="S4.T7.20.1.5.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.5.13.1" class="ltx_text" style="font-size:70%;">38.61</span></td>
<td id="S4.T7.20.1.5.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.5.14.1" class="ltx_text" style="font-size:70%;">5.16</span></td>
<td id="S4.T7.20.1.5.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.5.15.1" class="ltx_text" style="font-size:70%;">69.50</span></td>
<td id="S4.T7.20.1.5.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.5.16.1" class="ltx_text" style="font-size:70%;">57.59</span></td>
<td id="S4.T7.20.1.5.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.5.17.1" class="ltx_text" style="font-size:70%;">55.19</span></td>
<td id="S4.T7.20.1.5.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.5.18.1" class="ltx_text" style="font-size:70%;">35.74</span></td>
</tr>
<tr id="S4.T7.20.1.6" class="ltx_tr">
<td id="S4.T7.20.1.6.1" class="ltx_td ltx_align_left ltx_border_r">
<span id="S4.T7.20.1.6.1.1" class="ltx_ERROR undefined">\cdashline</span><span id="S4.T7.20.1.6.1.2" class="ltx_text" style="font-size:70%;">2-19</span>
</td>
<td id="S4.T7.20.1.6.2" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T7.20.1.6.2.1" class="ltx_text" style="font-size:70%;">SnapKV</span></td>
<td id="S4.T7.20.1.6.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.6.3.1" class="ltx_text" style="font-size:70%;">21.19</span></td>
<td id="S4.T7.20.1.6.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.6.4.1" class="ltx_text" style="font-size:70%;">13.55</span></td>
<td id="S4.T7.20.1.6.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.6.5.1" class="ltx_text" style="font-size:70%;">32.64</span></td>
<td id="S4.T7.20.1.6.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.6.6.1" class="ltx_text" style="font-size:70%;">38.75</span></td>
<td id="S4.T7.20.1.6.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.6.7.1" class="ltx_text" style="font-size:70%;">29.64</span></td>
<td id="S4.T7.20.1.6.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.6.8.1" class="ltx_text" style="font-size:70%;">18.73</span></td>
<td id="S4.T7.20.1.6.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.6.9.1" class="ltx_text" style="font-size:70%;">18.98</span></td>
<td id="S4.T7.20.1.6.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.6.10.1" class="ltx_text" style="font-size:70%;">21.62</span></td>
<td id="S4.T7.20.1.6.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.6.11.1" class="ltx_text" style="font-size:70%;">20.26</span></td>
<td id="S4.T7.20.1.6.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.6.12.1" class="ltx_text" style="font-size:70%;">45.00</span></td>
<td id="S4.T7.20.1.6.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.6.13.1" class="ltx_text" style="font-size:70%;">88.36</span></td>
<td id="S4.T7.20.1.6.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.6.14.1" class="ltx_text" style="font-size:70%;">37.64</span></td>
<td id="S4.T7.20.1.6.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.6.15.1" class="ltx_text" style="font-size:70%;">5.13</span></td>
<td id="S4.T7.20.1.6.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.6.16.1" class="ltx_text" style="font-size:70%;">68.85</span></td>
<td id="S4.T7.20.1.6.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.6.17.1" class="ltx_text" style="font-size:70%;">55.84</span></td>
<td id="S4.T7.20.1.6.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.6.18.1" class="ltx_text" style="font-size:70%;">51.82</span></td>
<td id="S4.T7.20.1.6.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.6.19.1" class="ltx_text" style="font-size:70%;">35.50</span></td>
</tr>
<tr id="S4.T7.20.1.7" class="ltx_tr">
<td id="S4.T7.20.1.7.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.7.2" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T7.20.1.7.2.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T7.20.1.7.2.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T7.20.1.7.2.3" class="ltx_text" style="font-size:70%;">(30+30)</span>
</td>
<td id="S4.T7.20.1.7.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.7.3.1" class="ltx_text" style="font-size:70%;">21.86</span></td>
<td id="S4.T7.20.1.7.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.7.4.1" class="ltx_text" style="font-size:70%;">13.79</span></td>
<td id="S4.T7.20.1.7.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.7.5.1" class="ltx_text" style="font-size:70%;">33.26</span></td>
<td id="S4.T7.20.1.7.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.7.6.1" class="ltx_text" style="font-size:70%;">40.93</span></td>
<td id="S4.T7.20.1.7.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.7.7.1" class="ltx_text" style="font-size:70%;">29.39</span></td>
<td id="S4.T7.20.1.7.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.7.8.1" class="ltx_text" style="font-size:70%;">19.22</span></td>
<td id="S4.T7.20.1.7.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.7.9.1" class="ltx_text" style="font-size:70%;">18.81</span></td>
<td id="S4.T7.20.1.7.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.7.10.1" class="ltx_text" style="font-size:70%;">21.30</span></td>
<td id="S4.T7.20.1.7.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.7.11.1" class="ltx_text" style="font-size:70%;">19.26</span></td>
<td id="S4.T7.20.1.7.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.7.12.1" class="ltx_text" style="font-size:70%;">41.50</span></td>
<td id="S4.T7.20.1.7.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.7.13.1" class="ltx_text" style="font-size:70%;">87.00</span></td>
<td id="S4.T7.20.1.7.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.7.14.1" class="ltx_text" style="font-size:70%;">37.95</span></td>
<td id="S4.T7.20.1.7.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.7.15.1" class="ltx_text" style="font-size:70%;">5.78</span></td>
<td id="S4.T7.20.1.7.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.7.16.1" class="ltx_text" style="font-size:70%;">69.50</span></td>
<td id="S4.T7.20.1.7.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.7.17.1" class="ltx_text" style="font-size:70%;">57.84</span></td>
<td id="S4.T7.20.1.7.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.7.18.1" class="ltx_text" style="font-size:70%;">55.62</span></td>
<td id="S4.T7.20.1.7.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.7.19.1" class="ltx_text" style="font-size:70%;">35.81</span></td>
</tr>
<tr id="S4.T7.20.1.8" class="ltx_tr">
<td id="S4.T7.20.1.8.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.8.2" class="ltx_td ltx_align_center ltx_border_t" colspan="18"><span id="S4.T7.20.1.8.2.1" class="ltx_text" style="font-size:70%;">KV-size 512</span></td>
</tr>
<tr id="S4.T7.20.1.9" class="ltx_tr">
<td id="S4.T7.20.1.9.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.9.2" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T7.20.1.9.2.1" class="ltx_text" style="font-size:70%;">H2O</span></td>
<td id="S4.T7.20.1.9.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.9.3.1" class="ltx_text" style="font-size:70%;">23.52</span></td>
<td id="S4.T7.20.1.9.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.9.4.1" class="ltx_text" style="font-size:70%;">17.93</span></td>
<td id="S4.T7.20.1.9.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.9.5.1" class="ltx_text" style="font-size:70%;">34.68</span></td>
<td id="S4.T7.20.1.9.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.9.6.1" class="ltx_text" style="font-size:70%;">42.11</span></td>
<td id="S4.T7.20.1.9.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.9.7.1" class="ltx_text" style="font-size:70%;">33.52</span></td>
<td id="S4.T7.20.1.9.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.9.8.1" class="ltx_text" style="font-size:70%;">19.92</span></td>
<td id="S4.T7.20.1.9.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.9.9.1" class="ltx_text" style="font-size:70%;">22.11</span></td>
<td id="S4.T7.20.1.9.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.9.10.1" class="ltx_text" style="font-size:70%;">22.56</span></td>
<td id="S4.T7.20.1.9.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.9.11.1" class="ltx_text" style="font-size:70%;">23.82</span></td>
<td id="S4.T7.20.1.9.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.9.12.1" class="ltx_text" style="font-size:70%;">41.00</span></td>
<td id="S4.T7.20.1.9.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.9.13.1" class="ltx_text" style="font-size:70%;">90.46</span></td>
<td id="S4.T7.20.1.9.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.9.14.1" class="ltx_text" style="font-size:70%;">40.20</span></td>
<td id="S4.T7.20.1.9.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.9.15.1" class="ltx_text" style="font-size:70%;">5.87</span></td>
<td id="S4.T7.20.1.9.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.9.16.1" class="ltx_text" style="font-size:70%;">69.50</span></td>
<td id="S4.T7.20.1.9.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.9.17.1" class="ltx_text" style="font-size:70%;">56.71</span></td>
<td id="S4.T7.20.1.9.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.9.18.1" class="ltx_text" style="font-size:70%;">51.69</span></td>
<td id="S4.T7.20.1.9.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.9.19.1" class="ltx_text" style="font-size:70%;">37.23</span></td>
</tr>
<tr id="S4.T7.20.1.10" class="ltx_tr">
<td id="S4.T7.20.1.10.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.10.2" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T7.20.1.10.2.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T7.20.1.10.2.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T7.20.1.10.2.3" class="ltx_text" style="font-size:70%;">(30+30)</span>
</td>
<td id="S4.T7.20.1.10.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.10.3.1" class="ltx_text" style="font-size:70%;">22.83</span></td>
<td id="S4.T7.20.1.10.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.10.4.1" class="ltx_text" style="font-size:70%;">17.57</span></td>
<td id="S4.T7.20.1.10.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.10.5.1" class="ltx_text" style="font-size:70%;">34.18</span></td>
<td id="S4.T7.20.1.10.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.10.6.1" class="ltx_text" style="font-size:70%;">42.67</span></td>
<td id="S4.T7.20.1.10.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.10.7.1" class="ltx_text" style="font-size:70%;">33.52</span></td>
<td id="S4.T7.20.1.10.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.10.8.1" class="ltx_text" style="font-size:70%;">19.95</span></td>
<td id="S4.T7.20.1.10.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.10.9.1" class="ltx_text" style="font-size:70%;">21.17</span></td>
<td id="S4.T7.20.1.10.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.10.10.1" class="ltx_text" style="font-size:70%;">22.23</span></td>
<td id="S4.T7.20.1.10.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.10.11.1" class="ltx_text" style="font-size:70%;">22.82</span></td>
<td id="S4.T7.20.1.10.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.10.12.1" class="ltx_text" style="font-size:70%;">38.50</span></td>
<td id="S4.T7.20.1.10.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.10.13.1" class="ltx_text" style="font-size:70%;">90.11</span></td>
<td id="S4.T7.20.1.10.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.10.14.1" class="ltx_text" style="font-size:70%;">39.08</span></td>
<td id="S4.T7.20.1.10.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.10.15.1" class="ltx_text" style="font-size:70%;">5.21</span></td>
<td id="S4.T7.20.1.10.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.10.16.1" class="ltx_text" style="font-size:70%;">69.0</span></td>
<td id="S4.T7.20.1.10.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.10.17.1" class="ltx_text" style="font-size:70%;">59.99</span></td>
<td id="S4.T7.20.1.10.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.10.18.1" class="ltx_text" style="font-size:70%;">56.83</span></td>
<td id="S4.T7.20.1.10.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.10.19.1" class="ltx_text" style="font-size:70%;">37.23</span></td>
</tr>
<tr id="S4.T7.20.1.11" class="ltx_tr">
<td id="S4.T7.20.1.11.1" class="ltx_td ltx_align_left ltx_border_r">
<span id="S4.T7.20.1.11.1.1" class="ltx_ERROR undefined">\cdashline</span><span id="S4.T7.20.1.11.1.2" class="ltx_text" style="font-size:70%;">2-19</span>
</td>
<td id="S4.T7.20.1.11.2" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T7.20.1.11.2.1" class="ltx_text" style="font-size:70%;">SnapKV</span></td>
<td id="S4.T7.20.1.11.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.11.3.1" class="ltx_text" style="font-size:70%;">24.84</span></td>
<td id="S4.T7.20.1.11.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.11.4.1" class="ltx_text" style="font-size:70%;">23.96</span></td>
<td id="S4.T7.20.1.11.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.11.5.1" class="ltx_text" style="font-size:70%;">38.77</span></td>
<td id="S4.T7.20.1.11.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.11.6.1" class="ltx_text" style="font-size:70%;">42.75</span></td>
<td id="S4.T7.20.1.11.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.11.7.1" class="ltx_text" style="font-size:70%;">34.55</span></td>
<td id="S4.T7.20.1.11.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.11.8.1" class="ltx_text" style="font-size:70%;">20.87</span></td>
<td id="S4.T7.20.1.11.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.11.9.1" class="ltx_text" style="font-size:70%;">22.26</span></td>
<td id="S4.T7.20.1.11.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.11.10.1" class="ltx_text" style="font-size:70%;">22.61</span></td>
<td id="S4.T7.20.1.11.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.11.11.1" class="ltx_text" style="font-size:70%;">23.97</span></td>
<td id="S4.T7.20.1.11.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.11.12.1" class="ltx_text" style="font-size:70%;">70.00</span></td>
<td id="S4.T7.20.1.11.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.11.13.1" class="ltx_text" style="font-size:70%;">90.52</span></td>
<td id="S4.T7.20.1.11.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.11.14.1" class="ltx_text" style="font-size:70%;">40.29</span></td>
<td id="S4.T7.20.1.11.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.11.15.1" class="ltx_text" style="font-size:70%;">5.81</span></td>
<td id="S4.T7.20.1.11.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.11.16.1" class="ltx_text" style="font-size:70%;">69.50</span></td>
<td id="S4.T7.20.1.11.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.11.17.1" class="ltx_text" style="font-size:70%;">59.04</span></td>
<td id="S4.T7.20.1.11.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.11.18.1" class="ltx_text" style="font-size:70%;">51.81</span></td>
<td id="S4.T7.20.1.11.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.11.19.1" class="ltx_text" style="font-size:70%;">40.10</span></td>
</tr>
<tr id="S4.T7.20.1.12" class="ltx_tr">
<td id="S4.T7.20.1.12.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.12.2" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T7.20.1.12.2.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T7.20.1.12.2.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T7.20.1.12.2.3" class="ltx_text" style="font-size:70%;">(30+30)</span>
</td>
<td id="S4.T7.20.1.12.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.12.3.1" class="ltx_text" style="font-size:70%;">24.57</span></td>
<td id="S4.T7.20.1.12.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.12.4.1" class="ltx_text" style="font-size:70%;">24.59</span></td>
<td id="S4.T7.20.1.12.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.12.5.1" class="ltx_text" style="font-size:70%;">38.09</span></td>
<td id="S4.T7.20.1.12.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.12.6.1" class="ltx_text" style="font-size:70%;">44.61</span></td>
<td id="S4.T7.20.1.12.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.12.7.1" class="ltx_text" style="font-size:70%;">34.37</span></td>
<td id="S4.T7.20.1.12.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.12.8.1" class="ltx_text" style="font-size:70%;">20.37</span></td>
<td id="S4.T7.20.1.12.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.12.9.1" class="ltx_text" style="font-size:70%;">21.23</span></td>
<td id="S4.T7.20.1.12.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.12.10.1" class="ltx_text" style="font-size:70%;">21.95</span></td>
<td id="S4.T7.20.1.12.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.12.11.1" class="ltx_text" style="font-size:70%;">23.30</span></td>
<td id="S4.T7.20.1.12.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.12.12.1" class="ltx_text" style="font-size:70%;">66.00</span></td>
<td id="S4.T7.20.1.12.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.12.13.1" class="ltx_text" style="font-size:70%;">90.69</span></td>
<td id="S4.T7.20.1.12.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.12.14.1" class="ltx_text" style="font-size:70%;">39.38</span></td>
<td id="S4.T7.20.1.12.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.12.15.1" class="ltx_text" style="font-size:70%;">5.60</span></td>
<td id="S4.T7.20.1.12.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.12.16.1" class="ltx_text" style="font-size:70%;">69.00</span></td>
<td id="S4.T7.20.1.12.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.12.17.1" class="ltx_text" style="font-size:70%;">61.75</span></td>
<td id="S4.T7.20.1.12.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.12.18.1" class="ltx_text" style="font-size:70%;">58.46</span></td>
<td id="S4.T7.20.1.12.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.12.19.1" class="ltx_text" style="font-size:70%;">40.25</span></td>
</tr>
<tr id="S4.T7.20.1.13" class="ltx_tr">
<td id="S4.T7.20.1.13.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.13.2" class="ltx_td ltx_align_center ltx_border_t" colspan="18"><span id="S4.T7.20.1.13.2.1" class="ltx_text" style="font-size:70%;">KV-size 2048</span></td>
</tr>
<tr id="S4.T7.20.1.14" class="ltx_tr">
<td id="S4.T7.20.1.14.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.14.2" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T7.20.1.14.2.1" class="ltx_text" style="font-size:70%;">H2O</span></td>
<td id="S4.T7.20.1.14.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.14.3.1" class="ltx_text" style="font-size:70%;">25.56</span></td>
<td id="S4.T7.20.1.14.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.14.4.1" class="ltx_text" style="font-size:70%;">26.85</span></td>
<td id="S4.T7.20.1.14.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.14.5.1" class="ltx_text" style="font-size:70%;">39.54</span></td>
<td id="S4.T7.20.1.14.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.14.6.1" class="ltx_text" style="font-size:70%;">44.30</span></td>
<td id="S4.T7.20.1.14.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.14.7.1" class="ltx_text" style="font-size:70%;">32.92</span></td>
<td id="S4.T7.20.1.14.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.14.8.1" class="ltx_text" style="font-size:70%;">21.09</span></td>
<td id="S4.T7.20.1.14.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.14.9.1" class="ltx_text" style="font-size:70%;">24.68</span></td>
<td id="S4.T7.20.1.14.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.14.10.1" class="ltx_text" style="font-size:70%;">23.01</span></td>
<td id="S4.T7.20.1.14.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.14.11.1" class="ltx_text" style="font-size:70%;">26.16</span></td>
<td id="S4.T7.20.1.14.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.14.12.1" class="ltx_text" style="font-size:70%;">53.00</span></td>
<td id="S4.T7.20.1.14.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.14.13.1" class="ltx_text" style="font-size:70%;">90.65</span></td>
<td id="S4.T7.20.1.14.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.14.14.1" class="ltx_text" style="font-size:70%;">41.84</span></td>
<td id="S4.T7.20.1.14.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.14.15.1" class="ltx_text" style="font-size:70%;">4.91</span></td>
<td id="S4.T7.20.1.14.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.14.16.1" class="ltx_text" style="font-size:70%;">69.25</span></td>
<td id="S4.T7.20.1.14.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.14.17.1" class="ltx_text" style="font-size:70%;">58.43</span></td>
<td id="S4.T7.20.1.14.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.14.18.1" class="ltx_text" style="font-size:70%;">51.31</span></td>
<td id="S4.T7.20.1.14.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.14.19.1" class="ltx_text" style="font-size:70%;">39.59</span></td>
</tr>
<tr id="S4.T7.20.1.15" class="ltx_tr">
<td id="S4.T7.20.1.15.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.15.2" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T7.20.1.15.2.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T7.20.1.15.2.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T7.20.1.15.2.3" class="ltx_text" style="font-size:70%;">(30+30)</span>
</td>
<td id="S4.T7.20.1.15.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.15.3.1" class="ltx_text" style="font-size:70%;">25.03</span></td>
<td id="S4.T7.20.1.15.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.15.4.1" class="ltx_text" style="font-size:70%;">26.77</span></td>
<td id="S4.T7.20.1.15.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.15.5.1" class="ltx_text" style="font-size:70%;">39.68</span></td>
<td id="S4.T7.20.1.15.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.15.6.1" class="ltx_text" style="font-size:70%;">42.12</span></td>
<td id="S4.T7.20.1.15.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.15.7.1" class="ltx_text" style="font-size:70%;">33.08</span></td>
<td id="S4.T7.20.1.15.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.15.8.1" class="ltx_text" style="font-size:70%;">19.59</span></td>
<td id="S4.T7.20.1.15.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.15.9.1" class="ltx_text" style="font-size:70%;">23.00</span></td>
<td id="S4.T7.20.1.15.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.15.10.1" class="ltx_text" style="font-size:70%;">22.89</span></td>
<td id="S4.T7.20.1.15.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.15.11.1" class="ltx_text" style="font-size:70%;">25.27</span></td>
<td id="S4.T7.20.1.15.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.15.12.1" class="ltx_text" style="font-size:70%;">51.00</span></td>
<td id="S4.T7.20.1.15.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.15.13.1" class="ltx_text" style="font-size:70%;">91.11</span></td>
<td id="S4.T7.20.1.15.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.15.14.1" class="ltx_text" style="font-size:70%;">40.58</span></td>
<td id="S4.T7.20.1.15.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.15.15.1" class="ltx_text" style="font-size:70%;">5.23</span></td>
<td id="S4.T7.20.1.15.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.15.16.1" class="ltx_text" style="font-size:70%;">69.00</span></td>
<td id="S4.T7.20.1.15.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.15.17.1" class="ltx_text" style="font-size:70%;">61.12</span></td>
<td id="S4.T7.20.1.15.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.15.18.1" class="ltx_text" style="font-size:70%;">57.95</span></td>
<td id="S4.T7.20.1.15.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.15.19.1" class="ltx_text" style="font-size:70%;">39.59</span></td>
</tr>
<tr id="S4.T7.20.1.16" class="ltx_tr">
<td id="S4.T7.20.1.16.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.16.2" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T7.20.1.16.2.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T7.20.1.16.2.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T7.20.1.16.2.3" class="ltx_text" style="font-size:70%;">(40+40)</span>
</td>
<td id="S4.T7.20.1.16.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.16.3.1" class="ltx_text" style="font-size:70%;">24.87</span></td>
<td id="S4.T7.20.1.16.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.16.4.1" class="ltx_text" style="font-size:70%;">24.31</span></td>
<td id="S4.T7.20.1.16.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.16.5.1" class="ltx_text" style="font-size:70%;">37.77</span></td>
<td id="S4.T7.20.1.16.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.16.6.1" class="ltx_text" style="font-size:70%;">43.13</span></td>
<td id="S4.T7.20.1.16.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.16.7.1" class="ltx_text" style="font-size:70%;">34.42</span></td>
<td id="S4.T7.20.1.16.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.16.8.1" class="ltx_text" style="font-size:70%;">19.60</span></td>
<td id="S4.T7.20.1.16.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.16.9.1" class="ltx_text" style="font-size:70%;">21.67</span></td>
<td id="S4.T7.20.1.16.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.16.10.1" class="ltx_text" style="font-size:70%;">22.70</span></td>
<td id="S4.T7.20.1.16.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.16.11.1" class="ltx_text" style="font-size:70%;">24.52</span></td>
<td id="S4.T7.20.1.16.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.16.12.1" class="ltx_text" style="font-size:70%;">49.00</span></td>
<td id="S4.T7.20.1.16.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.16.13.1" class="ltx_text" style="font-size:70%;">90.81</span></td>
<td id="S4.T7.20.1.16.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.16.14.1" class="ltx_text" style="font-size:70%;">39.28</span></td>
<td id="S4.T7.20.1.16.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.16.15.1" class="ltx_text" style="font-size:70%;">6.00</span></td>
<td id="S4.T7.20.1.16.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.16.16.1" class="ltx_text" style="font-size:70%;">69.00</span></td>
<td id="S4.T7.20.1.16.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.16.17.1" class="ltx_text" style="font-size:70%;">61.81</span></td>
<td id="S4.T7.20.1.16.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.16.18.1" class="ltx_text" style="font-size:70%;">58.08</span></td>
<td id="S4.T7.20.1.16.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.16.19.1" class="ltx_text" style="font-size:70%;">39.19</span></td>
</tr>
<tr id="S4.T7.20.1.17" class="ltx_tr">
<td id="S4.T7.20.1.17.1" class="ltx_td ltx_align_left ltx_border_r">
<span id="S4.T7.20.1.17.1.1" class="ltx_ERROR undefined">\cdashline</span><span id="S4.T7.20.1.17.1.2" class="ltx_text" style="font-size:70%;">2-19</span>
</td>
<td id="S4.T7.20.1.17.2" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T7.20.1.17.2.1" class="ltx_text" style="font-size:70%;">SnapKV</span></td>
<td id="S4.T7.20.1.17.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.17.3.1" class="ltx_text" style="font-size:70%;">25.86</span></td>
<td id="S4.T7.20.1.17.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.17.4.1" class="ltx_text" style="font-size:70%;">29.55</span></td>
<td id="S4.T7.20.1.17.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.17.5.1" class="ltx_text" style="font-size:70%;">41.10</span></td>
<td id="S4.T7.20.1.17.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.17.6.1" class="ltx_text" style="font-size:70%;">44.99</span></td>
<td id="S4.T7.20.1.17.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.17.7.1" class="ltx_text" style="font-size:70%;">35.80</span></td>
<td id="S4.T7.20.1.17.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.17.8.1" class="ltx_text" style="font-size:70%;">21.81</span></td>
<td id="S4.T7.20.1.17.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.17.9.1" class="ltx_text" style="font-size:70%;">25.98</span></td>
<td id="S4.T7.20.1.17.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.17.10.1" class="ltx_text" style="font-size:70%;">23.40</span></td>
<td id="S4.T7.20.1.17.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.17.11.1" class="ltx_text" style="font-size:70%;">26.46</span></td>
<td id="S4.T7.20.1.17.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.17.12.1" class="ltx_text" style="font-size:70%;">73.50</span></td>
<td id="S4.T7.20.1.17.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.17.13.1" class="ltx_text" style="font-size:70%;">90.56</span></td>
<td id="S4.T7.20.1.17.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.17.14.1" class="ltx_text" style="font-size:70%;">41.66</span></td>
<td id="S4.T7.20.1.17.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.17.15.1" class="ltx_text" style="font-size:70%;">5.17</span></td>
<td id="S4.T7.20.1.17.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.17.16.1" class="ltx_text" style="font-size:70%;">69.25</span></td>
<td id="S4.T7.20.1.17.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.17.17.1" class="ltx_text" style="font-size:70%;">58.67</span></td>
<td id="S4.T7.20.1.17.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.17.18.1" class="ltx_text" style="font-size:70%;">51.52</span></td>
<td id="S4.T7.20.1.17.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.17.19.1" class="ltx_text" style="font-size:70%;">41.58</span></td>
</tr>
<tr id="S4.T7.20.1.18" class="ltx_tr">
<td id="S4.T7.20.1.18.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.18.2" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T7.20.1.18.2.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T7.20.1.18.2.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T7.20.1.18.2.3" class="ltx_text" style="font-size:70%;">(30+30)</span>
</td>
<td id="S4.T7.20.1.18.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.18.3.1" class="ltx_text" style="font-size:70%;">25.13</span></td>
<td id="S4.T7.20.1.18.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.18.4.1" class="ltx_text" style="font-size:70%;">29.97</span></td>
<td id="S4.T7.20.1.18.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.18.5.1" class="ltx_text" style="font-size:70%;">40.35</span></td>
<td id="S4.T7.20.1.18.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.18.6.1" class="ltx_text" style="font-size:70%;">44.12</span></td>
<td id="S4.T7.20.1.18.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.18.7.1" class="ltx_text" style="font-size:70%;">34.64</span></td>
<td id="S4.T7.20.1.18.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.18.8.1" class="ltx_text" style="font-size:70%;">19.94</span></td>
<td id="S4.T7.20.1.18.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.18.9.1" class="ltx_text" style="font-size:70%;">23.62</span></td>
<td id="S4.T7.20.1.18.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.18.10.1" class="ltx_text" style="font-size:70%;">23.03</span></td>
<td id="S4.T7.20.1.18.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.18.11.1" class="ltx_text" style="font-size:70%;">25.30</span></td>
<td id="S4.T7.20.1.18.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.18.12.1" class="ltx_text" style="font-size:70%;">72.50</span></td>
<td id="S4.T7.20.1.18.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.18.13.1" class="ltx_text" style="font-size:70%;">90.78</span></td>
<td id="S4.T7.20.1.18.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.18.14.1" class="ltx_text" style="font-size:70%;">39.46</span></td>
<td id="S4.T7.20.1.18.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.18.15.1" class="ltx_text" style="font-size:70%;">5.35</span></td>
<td id="S4.T7.20.1.18.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.18.16.1" class="ltx_text" style="font-size:70%;">69.00</span></td>
<td id="S4.T7.20.1.18.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.18.17.1" class="ltx_text" style="font-size:70%;">61.50</span></td>
<td id="S4.T7.20.1.18.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.18.18.1" class="ltx_text" style="font-size:70%;">57.91</span></td>
<td id="S4.T7.20.1.18.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.18.19.1" class="ltx_text" style="font-size:70%;">41.41</span></td>
</tr>
<tr id="S4.T7.20.1.19" class="ltx_tr">
<td id="S4.T7.20.1.19.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.19.2" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T7.20.1.19.2.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T7.20.1.19.2.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T7.20.1.19.2.3" class="ltx_text" style="font-size:70%;">(40+40)</span>
</td>
<td id="S4.T7.20.1.19.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.19.3.1" class="ltx_text" style="font-size:70%;">25.13</span></td>
<td id="S4.T7.20.1.19.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.19.4.1" class="ltx_text" style="font-size:70%;">28.85</span></td>
<td id="S4.T7.20.1.19.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.19.5.1" class="ltx_text" style="font-size:70%;">40.70</span></td>
<td id="S4.T7.20.1.19.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.19.6.1" class="ltx_text" style="font-size:70%;">44.21</span></td>
<td id="S4.T7.20.1.19.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.19.7.1" class="ltx_text" style="font-size:70%;">36.36</span></td>
<td id="S4.T7.20.1.19.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.19.8.1" class="ltx_text" style="font-size:70%;">21.07</span></td>
<td id="S4.T7.20.1.19.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.19.9.1" class="ltx_text" style="font-size:70%;">22.31</span></td>
<td id="S4.T7.20.1.19.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.19.10.1" class="ltx_text" style="font-size:70%;">22.89</span></td>
<td id="S4.T7.20.1.19.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.19.11.1" class="ltx_text" style="font-size:70%;">24.80</span></td>
<td id="S4.T7.20.1.19.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.19.12.1" class="ltx_text" style="font-size:70%;">72.50</span></td>
<td id="S4.T7.20.1.19.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.19.13.1" class="ltx_text" style="font-size:70%;">90.88</span></td>
<td id="S4.T7.20.1.19.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.19.14.1" class="ltx_text" style="font-size:70%;">38.77</span></td>
<td id="S4.T7.20.1.19.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.19.15.1" class="ltx_text" style="font-size:70%;">6.41</span></td>
<td id="S4.T7.20.1.19.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.19.16.1" class="ltx_text" style="font-size:70%;">69.00</span></td>
<td id="S4.T7.20.1.19.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.19.17.1" class="ltx_text" style="font-size:70%;">61.49</span></td>
<td id="S4.T7.20.1.19.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.19.18.1" class="ltx_text" style="font-size:70%;">58.87</span></td>
<td id="S4.T7.20.1.19.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.19.19.1" class="ltx_text" style="font-size:70%;">41.52</span></td>
</tr>
<tr id="S4.T7.20.1.20" class="ltx_tr">
<td id="S4.T7.20.1.20.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_tt" rowspan="3"><span id="S4.T7.20.1.20.1.1" class="ltx_text" style="font-size:70%;">
<span id="S4.T7.20.1.20.1.1.1" class="ltx_inline-block ltx_transformed_outer" style="width:4.9pt;height:35.1pt;vertical-align:-15.1pt;"><span class="ltx_transformed_inner" style="width:35.2pt;transform:translate(-15.15pt,0pt) rotate(-90deg) ;">
<span id="S4.T7.20.1.20.1.1.1.1" class="ltx_p">Mistral 7B</span>
</span></span></span></td>
<td id="S4.T7.20.1.20.2" class="ltx_td ltx_align_center ltx_border_tt" colspan="18"><span id="S4.T7.20.1.20.2.1" class="ltx_text" style="font-size:70%;">KV-size 128</span></td>
</tr>
<tr id="S4.T7.20.1.21" class="ltx_tr">
<td id="S4.T7.20.1.21.1" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T7.20.1.21.1.1" class="ltx_text" style="font-size:70%;">H2O</span></td>
<td id="S4.T7.20.1.21.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.21.2.1" class="ltx_text" style="font-size:70%;">21.21</span></td>
<td id="S4.T7.20.1.21.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.21.3.1" class="ltx_text" style="font-size:70%;">21.81</span></td>
<td id="S4.T7.20.1.21.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.21.4.1" class="ltx_text" style="font-size:70%;">38.87</span></td>
<td id="S4.T7.20.1.21.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.21.5.1" class="ltx_text" style="font-size:70%;">30.42</span></td>
<td id="S4.T7.20.1.21.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.21.6.1" class="ltx_text" style="font-size:70%;">20.36</span></td>
<td id="S4.T7.20.1.21.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.21.7.1" class="ltx_text" style="font-size:70%;">12.30</span></td>
<td id="S4.T7.20.1.21.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.21.8.1" class="ltx_text" style="font-size:70%;">20.58</span></td>
<td id="S4.T7.20.1.21.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.21.9.1" class="ltx_text" style="font-size:70%;">22.61</span></td>
<td id="S4.T7.20.1.21.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.21.10.1" class="ltx_text" style="font-size:70%;">22.10</span></td>
<td id="S4.T7.20.1.21.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.21.11.1" class="ltx_text" style="font-size:70%;">39.00</span></td>
<td id="S4.T7.20.1.21.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.21.12.1" class="ltx_text" style="font-size:70%;">82.37</span></td>
<td id="S4.T7.20.1.21.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.21.13.1" class="ltx_text" style="font-size:70%;">40.44</span></td>
<td id="S4.T7.20.1.21.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.21.14.1" class="ltx_text" style="font-size:70%;">2.90</span></td>
<td id="S4.T7.20.1.21.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.21.15.1" class="ltx_text" style="font-size:70%;">79.56</span></td>
<td id="S4.T7.20.1.21.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.21.16.1" class="ltx_text" style="font-size:70%;">51.22</span></td>
<td id="S4.T7.20.1.21.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.21.17.1" class="ltx_text" style="font-size:70%;">48.38</span></td>
<td id="S4.T7.20.1.21.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.21.18.1" class="ltx_text" style="font-size:70%;">34.63</span></td>
</tr>
<tr id="S4.T7.20.1.22" class="ltx_tr">
<td id="S4.T7.20.1.22.1" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T7.20.1.22.1.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T7.20.1.22.1.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T7.20.1.22.1.3" class="ltx_text" style="font-size:70%;">(30+30)</span>
</td>
<td id="S4.T7.20.1.22.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.22.2.1" class="ltx_text" style="font-size:70%;">20.71</span></td>
<td id="S4.T7.20.1.22.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.22.3.1" class="ltx_text" style="font-size:70%;">21.49</span></td>
<td id="S4.T7.20.1.22.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.22.4.1" class="ltx_text" style="font-size:70%;">38.01</span></td>
<td id="S4.T7.20.1.22.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.22.5.1" class="ltx_text" style="font-size:70%;">30.66</span></td>
<td id="S4.T7.20.1.22.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.22.6.1" class="ltx_text" style="font-size:70%;">22.28</span></td>
<td id="S4.T7.20.1.22.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.22.7.1" class="ltx_text" style="font-size:70%;">13.87</span></td>
<td id="S4.T7.20.1.22.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.22.8.1" class="ltx_text" style="font-size:70%;">20.13</span></td>
<td id="S4.T7.20.1.22.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.22.9.1" class="ltx_text" style="font-size:70%;">22.45</span></td>
<td id="S4.T7.20.1.22.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.22.10.1" class="ltx_text" style="font-size:70%;">21.07</span></td>
<td id="S4.T7.20.1.22.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.22.11.1" class="ltx_text" style="font-size:70%;">38.50</span></td>
<td id="S4.T7.20.1.22.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.22.12.1" class="ltx_text" style="font-size:70%;">82.20</span></td>
<td id="S4.T7.20.1.22.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.22.13.1" class="ltx_text" style="font-size:70%;">38.69</span></td>
<td id="S4.T7.20.1.22.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.22.14.1" class="ltx_text" style="font-size:70%;">2.94</span></td>
<td id="S4.T7.20.1.22.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.22.15.1" class="ltx_text" style="font-size:70%;">78.56</span></td>
<td id="S4.T7.20.1.22.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.22.16.1" class="ltx_text" style="font-size:70%;">51.55</span></td>
<td id="S4.T7.20.1.22.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.22.17.1" class="ltx_text" style="font-size:70%;">48.28</span></td>
<td id="S4.T7.20.1.22.18" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.22.18.1" class="ltx_text" style="font-size:70%;">34.46</span></td>
</tr>
<tr id="S4.T7.20.1.23" class="ltx_tr">
<td id="S4.T7.20.1.23.1" class="ltx_td ltx_align_left ltx_border_r">
<span id="S4.T7.20.1.23.1.1" class="ltx_ERROR undefined">\cdashline</span><span id="S4.T7.20.1.23.1.2" class="ltx_text" style="font-size:70%;">2-19</span>
</td>
<td id="S4.T7.20.1.23.2" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T7.20.1.23.2.1" class="ltx_text" style="font-size:70%;">SnapKV</span></td>
<td id="S4.T7.20.1.23.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.23.3.1" class="ltx_text" style="font-size:70%;">19.17</span></td>
<td id="S4.T7.20.1.23.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.23.4.1" class="ltx_text" style="font-size:70%;">21.40</span></td>
<td id="S4.T7.20.1.23.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.23.5.1" class="ltx_text" style="font-size:70%;">42.93</span></td>
<td id="S4.T7.20.1.23.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.23.6.1" class="ltx_text" style="font-size:70%;">36.76</span></td>
<td id="S4.T7.20.1.23.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.23.7.1" class="ltx_text" style="font-size:70%;">22.44</span></td>
<td id="S4.T7.20.1.23.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.23.8.1" class="ltx_text" style="font-size:70%;">15.86</span></td>
<td id="S4.T7.20.1.23.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.23.9.1" class="ltx_text" style="font-size:70%;">19.16</span></td>
<td id="S4.T7.20.1.23.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.23.10.1" class="ltx_text" style="font-size:70%;">21.84</span></td>
<td id="S4.T7.20.1.23.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.23.11.1" class="ltx_text" style="font-size:70%;">21.55</span></td>
<td id="S4.T7.20.1.23.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.23.12.1" class="ltx_text" style="font-size:70%;">47.50</span></td>
<td id="S4.T7.20.1.23.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.23.13.1" class="ltx_text" style="font-size:70%;">84.15</span></td>
<td id="S4.T7.20.1.23.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.23.14.1" class="ltx_text" style="font-size:70%;">40.24</span></td>
<td id="S4.T7.20.1.23.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.23.15.1" class="ltx_text" style="font-size:70%;">2.30</span></td>
<td id="S4.T7.20.1.23.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.23.16.1" class="ltx_text" style="font-size:70%;">68.26</span></td>
<td id="S4.T7.20.1.23.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.23.17.1" class="ltx_text" style="font-size:70%;">52.31</span></td>
<td id="S4.T7.20.1.23.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.23.18.1" class="ltx_text" style="font-size:70%;">48.80</span></td>
<td id="S4.T7.20.1.23.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.23.19.1" class="ltx_text" style="font-size:70%;">35.29</span></td>
</tr>
<tr id="S4.T7.20.1.24" class="ltx_tr">
<td id="S4.T7.20.1.24.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.24.2" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T7.20.1.24.2.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T7.20.1.24.2.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T7.20.1.24.2.3" class="ltx_text" style="font-size:70%;">(30+30)</span>
</td>
<td id="S4.T7.20.1.24.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.24.3.1" class="ltx_text" style="font-size:70%;">19.92</span></td>
<td id="S4.T7.20.1.24.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.24.4.1" class="ltx_text" style="font-size:70%;">20.61</span></td>
<td id="S4.T7.20.1.24.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.24.5.1" class="ltx_text" style="font-size:70%;">42.68</span></td>
<td id="S4.T7.20.1.24.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.24.6.1" class="ltx_text" style="font-size:70%;">37.63</span></td>
<td id="S4.T7.20.1.24.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.24.7.1" class="ltx_text" style="font-size:70%;">23.19</span></td>
<td id="S4.T7.20.1.24.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.24.8.1" class="ltx_text" style="font-size:70%;">15.09</span></td>
<td id="S4.T7.20.1.24.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.24.9.1" class="ltx_text" style="font-size:70%;">18.97</span></td>
<td id="S4.T7.20.1.24.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.24.10.1" class="ltx_text" style="font-size:70%;">21.93</span></td>
<td id="S4.T7.20.1.24.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.24.11.1" class="ltx_text" style="font-size:70%;">20.55</span></td>
<td id="S4.T7.20.1.24.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.24.12.1" class="ltx_text" style="font-size:70%;">45.00</span></td>
<td id="S4.T7.20.1.24.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.24.13.1" class="ltx_text" style="font-size:70%;">84.06</span></td>
<td id="S4.T7.20.1.24.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.24.14.1" class="ltx_text" style="font-size:70%;">39.33</span></td>
<td id="S4.T7.20.1.24.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.24.15.1" class="ltx_text" style="font-size:70%;">2.99</span></td>
<td id="S4.T7.20.1.24.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.24.16.1" class="ltx_text" style="font-size:70%;">66.00</span></td>
<td id="S4.T7.20.1.24.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.24.17.1" class="ltx_text" style="font-size:70%;">51.51</span></td>
<td id="S4.T7.20.1.24.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.24.18.1" class="ltx_text" style="font-size:70%;">47.51</span></td>
<td id="S4.T7.20.1.24.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.24.19.1" class="ltx_text" style="font-size:70%;">34.81</span></td>
</tr>
<tr id="S4.T7.20.1.25" class="ltx_tr">
<td id="S4.T7.20.1.25.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.25.2" class="ltx_td ltx_align_center ltx_border_t" colspan="18"><span id="S4.T7.20.1.25.2.1" class="ltx_text" style="font-size:70%;">KV-size 512</span></td>
</tr>
<tr id="S4.T7.20.1.26" class="ltx_tr">
<td id="S4.T7.20.1.26.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.26.2" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T7.20.1.26.2.1" class="ltx_text" style="font-size:70%;">H2O</span></td>
<td id="S4.T7.20.1.26.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.26.3.1" class="ltx_text" style="font-size:70%;">21.83</span></td>
<td id="S4.T7.20.1.26.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.26.4.1" class="ltx_text" style="font-size:70%;">26.00</span></td>
<td id="S4.T7.20.1.26.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.26.5.1" class="ltx_text" style="font-size:70%;">44.69</span></td>
<td id="S4.T7.20.1.26.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.26.6.1" class="ltx_text" style="font-size:70%;">32.46</span></td>
<td id="S4.T7.20.1.26.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.26.7.1" class="ltx_text" style="font-size:70%;">23.05</span></td>
<td id="S4.T7.20.1.26.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.26.8.1" class="ltx_text" style="font-size:70%;">14.69</span></td>
<td id="S4.T7.20.1.26.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.26.9.1" class="ltx_text" style="font-size:70%;">23.53</span></td>
<td id="S4.T7.20.1.26.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.26.10.1" class="ltx_text" style="font-size:70%;">23.06</span></td>
<td id="S4.T7.20.1.26.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.26.11.1" class="ltx_text" style="font-size:70%;">24.59</span></td>
<td id="S4.T7.20.1.26.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.26.12.1" class="ltx_text" style="font-size:70%;">42.00</span></td>
<td id="S4.T7.20.1.26.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.26.13.1" class="ltx_text" style="font-size:70%;">85.22</span></td>
<td id="S4.T7.20.1.26.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.26.14.1" class="ltx_text" style="font-size:70%;">41.49</span></td>
<td id="S4.T7.20.1.26.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.26.15.1" class="ltx_text" style="font-size:70%;">3.40</span></td>
<td id="S4.T7.20.1.26.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.26.16.1" class="ltx_text" style="font-size:70%;">86.20</span></td>
<td id="S4.T7.20.1.26.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.26.17.1" class="ltx_text" style="font-size:70%;">54.78</span></td>
<td id="S4.T7.20.1.26.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.26.18.1" class="ltx_text" style="font-size:70%;">51.09</span></td>
<td id="S4.T7.20.1.26.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.26.19.1" class="ltx_text" style="font-size:70%;">37.38</span></td>
</tr>
<tr id="S4.T7.20.1.27" class="ltx_tr">
<td id="S4.T7.20.1.27.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.27.2" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T7.20.1.27.2.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T7.20.1.27.2.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T7.20.1.27.2.3" class="ltx_text" style="font-size:70%;">(30+30)</span>
</td>
<td id="S4.T7.20.1.27.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.27.3.1" class="ltx_text" style="font-size:70%;">22.36</span></td>
<td id="S4.T7.20.1.27.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.27.4.1" class="ltx_text" style="font-size:70%;">24.26</span></td>
<td id="S4.T7.20.1.27.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.27.5.1" class="ltx_text" style="font-size:70%;">44.77</span></td>
<td id="S4.T7.20.1.27.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.27.6.1" class="ltx_text" style="font-size:70%;">30.47</span></td>
<td id="S4.T7.20.1.27.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.27.7.1" class="ltx_text" style="font-size:70%;">22.94</span></td>
<td id="S4.T7.20.1.27.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.27.8.1" class="ltx_text" style="font-size:70%;">14.96</span></td>
<td id="S4.T7.20.1.27.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.27.9.1" class="ltx_text" style="font-size:70%;">22.63</span></td>
<td id="S4.T7.20.1.27.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.27.10.1" class="ltx_text" style="font-size:70%;">22.90</span></td>
<td id="S4.T7.20.1.27.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.27.11.1" class="ltx_text" style="font-size:70%;">23.73</span></td>
<td id="S4.T7.20.1.27.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.27.12.1" class="ltx_text" style="font-size:70%;">41.50</span></td>
<td id="S4.T7.20.1.27.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.27.13.1" class="ltx_text" style="font-size:70%;">85.30</span></td>
<td id="S4.T7.20.1.27.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.27.14.1" class="ltx_text" style="font-size:70%;">40.21</span></td>
<td id="S4.T7.20.1.27.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.27.15.1" class="ltx_text" style="font-size:70%;">3.08</span></td>
<td id="S4.T7.20.1.27.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.27.16.1" class="ltx_text" style="font-size:70%;">80.07</span></td>
<td id="S4.T7.20.1.27.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.27.17.1" class="ltx_text" style="font-size:70%;">54.48</span></td>
<td id="S4.T7.20.1.27.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.27.18.1" class="ltx_text" style="font-size:70%;">50.96</span></td>
<td id="S4.T7.20.1.27.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.27.19.1" class="ltx_text" style="font-size:70%;">36.54</span></td>
</tr>
<tr id="S4.T7.20.1.28" class="ltx_tr">
<td id="S4.T7.20.1.28.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.28.2" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T7.20.1.28.2.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T7.20.1.28.2.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T7.20.1.28.2.3" class="ltx_text" style="font-size:70%;">(30+10)</span>
</td>
<td id="S4.T7.20.1.28.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.28.3.1" class="ltx_text" style="font-size:70%;">22.14</span></td>
<td id="S4.T7.20.1.28.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.28.4.1" class="ltx_text" style="font-size:70%;">25.15</span></td>
<td id="S4.T7.20.1.28.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.28.5.1" class="ltx_text" style="font-size:70%;">45.29</span></td>
<td id="S4.T7.20.1.28.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.28.6.1" class="ltx_text" style="font-size:70%;">31.78</span></td>
<td id="S4.T7.20.1.28.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.28.7.1" class="ltx_text" style="font-size:70%;">23.21</span></td>
<td id="S4.T7.20.1.28.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.28.8.1" class="ltx_text" style="font-size:70%;">14.62</span></td>
<td id="S4.T7.20.1.28.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.28.9.1" class="ltx_text" style="font-size:70%;">23.36</span></td>
<td id="S4.T7.20.1.28.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.28.10.1" class="ltx_text" style="font-size:70%;">22.70</span></td>
<td id="S4.T7.20.1.28.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.28.11.1" class="ltx_text" style="font-size:70%;">24.51</span></td>
<td id="S4.T7.20.1.28.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.28.12.1" class="ltx_text" style="font-size:70%;">41.50</span></td>
<td id="S4.T7.20.1.28.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.28.13.1" class="ltx_text" style="font-size:70%;">85.61</span></td>
<td id="S4.T7.20.1.28.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.28.14.1" class="ltx_text" style="font-size:70%;">41.58</span></td>
<td id="S4.T7.20.1.28.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.28.15.1" class="ltx_text" style="font-size:70%;">2.75</span></td>
<td id="S4.T7.20.1.28.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.28.16.1" class="ltx_text" style="font-size:70%;">84.03</span></td>
<td id="S4.T7.20.1.28.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.28.17.1" class="ltx_text" style="font-size:70%;">54.50</span></td>
<td id="S4.T7.20.1.28.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.28.18.1" class="ltx_text" style="font-size:70%;">51.09</span></td>
<td id="S4.T7.20.1.28.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.28.19.1" class="ltx_text" style="font-size:70%;">37.11</span></td>
</tr>
<tr id="S4.T7.20.1.29" class="ltx_tr">
<td id="S4.T7.20.1.29.1" class="ltx_td ltx_align_left ltx_border_r">
<span id="S4.T7.20.1.29.1.1" class="ltx_ERROR undefined">\cdashline</span><span id="S4.T7.20.1.29.1.2" class="ltx_text" style="font-size:70%;">2-19</span>
</td>
<td id="S4.T7.20.1.29.2" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T7.20.1.29.2.1" class="ltx_text" style="font-size:70%;">SnapKV</span></td>
<td id="S4.T7.20.1.29.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.29.3.1" class="ltx_text" style="font-size:70%;">24.44</span></td>
<td id="S4.T7.20.1.29.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.29.4.1" class="ltx_text" style="font-size:70%;">27.81</span></td>
<td id="S4.T7.20.1.29.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.29.5.1" class="ltx_text" style="font-size:70%;">48.98</span></td>
<td id="S4.T7.20.1.29.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.29.6.1" class="ltx_text" style="font-size:70%;">39.46</span></td>
<td id="S4.T7.20.1.29.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.29.7.1" class="ltx_text" style="font-size:70%;">25.25</span></td>
<td id="S4.T7.20.1.29.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.29.8.1" class="ltx_text" style="font-size:70%;">16.98</span></td>
<td id="S4.T7.20.1.29.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.29.9.1" class="ltx_text" style="font-size:70%;">23.70</span></td>
<td id="S4.T7.20.1.29.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.29.10.1" class="ltx_text" style="font-size:70%;">22.96</span></td>
<td id="S4.T7.20.1.29.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.29.11.1" class="ltx_text" style="font-size:70%;">24.37</span></td>
<td id="S4.T7.20.1.29.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.29.12.1" class="ltx_text" style="font-size:70%;">67.0</span></td>
<td id="S4.T7.20.1.29.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.29.13.1" class="ltx_text" style="font-size:70%;">85.88</span></td>
<td id="S4.T7.20.1.29.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.29.14.1" class="ltx_text" style="font-size:70%;">41.26</span></td>
<td id="S4.T7.20.1.29.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.29.15.1" class="ltx_text" style="font-size:70%;">2.78</span></td>
<td id="S4.T7.20.1.29.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.29.16.1" class="ltx_text" style="font-size:70%;">86.56</span></td>
<td id="S4.T7.20.1.29.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.29.17.1" class="ltx_text" style="font-size:70%;">56.46</span></td>
<td id="S4.T7.20.1.29.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.29.18.1" class="ltx_text" style="font-size:70%;">53.41</span></td>
<td id="S4.T7.20.1.29.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.29.19.1" class="ltx_text" style="font-size:70%;">40.46</span></td>
</tr>
<tr id="S4.T7.20.1.30" class="ltx_tr">
<td id="S4.T7.20.1.30.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.30.2" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T7.20.1.30.2.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T7.20.1.30.2.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T7.20.1.30.2.3" class="ltx_text" style="font-size:70%;">(30+30)</span>
</td>
<td id="S4.T7.20.1.30.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.30.3.1" class="ltx_text" style="font-size:70%;">24.10</span></td>
<td id="S4.T7.20.1.30.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.30.4.1" class="ltx_text" style="font-size:70%;">27.04</span></td>
<td id="S4.T7.20.1.30.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.30.5.1" class="ltx_text" style="font-size:70%;">47.76</span></td>
<td id="S4.T7.20.1.30.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.30.6.1" class="ltx_text" style="font-size:70%;">38.66</span></td>
<td id="S4.T7.20.1.30.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.30.7.1" class="ltx_text" style="font-size:70%;">25.45</span></td>
<td id="S4.T7.20.1.30.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.30.8.1" class="ltx_text" style="font-size:70%;">17.51</span></td>
<td id="S4.T7.20.1.30.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.30.9.1" class="ltx_text" style="font-size:70%;">22.64</span></td>
<td id="S4.T7.20.1.30.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.30.10.1" class="ltx_text" style="font-size:70%;">22.81</span></td>
<td id="S4.T7.20.1.30.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.30.11.1" class="ltx_text" style="font-size:70%;">23.91</span></td>
<td id="S4.T7.20.1.30.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.30.12.1" class="ltx_text" style="font-size:70%;">66.00</span></td>
<td id="S4.T7.20.1.30.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.30.13.1" class="ltx_text" style="font-size:70%;">86.62</span></td>
<td id="S4.T7.20.1.30.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.30.14.1" class="ltx_text" style="font-size:70%;">39.91</span></td>
<td id="S4.T7.20.1.30.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.30.15.1" class="ltx_text" style="font-size:70%;">3.36</span></td>
<td id="S4.T7.20.1.30.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.30.16.1" class="ltx_text" style="font-size:70%;">82.24</span></td>
<td id="S4.T7.20.1.30.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.30.17.1" class="ltx_text" style="font-size:70%;">55.96</span></td>
<td id="S4.T7.20.1.30.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.30.18.1" class="ltx_text" style="font-size:70%;">52.81</span></td>
<td id="S4.T7.20.1.30.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.30.19.1" class="ltx_text" style="font-size:70%;">39.80</span></td>
</tr>
<tr id="S4.T7.20.1.31" class="ltx_tr">
<td id="S4.T7.20.1.31.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.31.2" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T7.20.1.31.2.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T7.20.1.31.2.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T7.20.1.31.2.3" class="ltx_text" style="font-size:70%;">(30+10)</span>
</td>
<td id="S4.T7.20.1.31.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.31.3.1" class="ltx_text" style="font-size:70%;">23.90</span></td>
<td id="S4.T7.20.1.31.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.31.4.1" class="ltx_text" style="font-size:70%;">28.14</span></td>
<td id="S4.T7.20.1.31.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.31.5.1" class="ltx_text" style="font-size:70%;">48.35</span></td>
<td id="S4.T7.20.1.31.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.31.6.1" class="ltx_text" style="font-size:70%;">39.03</span></td>
<td id="S4.T7.20.1.31.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.31.7.1" class="ltx_text" style="font-size:70%;">24.83</span></td>
<td id="S4.T7.20.1.31.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.31.8.1" class="ltx_text" style="font-size:70%;">16.68</span></td>
<td id="S4.T7.20.1.31.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.31.9.1" class="ltx_text" style="font-size:70%;">23.51</span></td>
<td id="S4.T7.20.1.31.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.31.10.1" class="ltx_text" style="font-size:70%;">23.12</span></td>
<td id="S4.T7.20.1.31.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.31.11.1" class="ltx_text" style="font-size:70%;">24.34</span></td>
<td id="S4.T7.20.1.31.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.31.12.1" class="ltx_text" style="font-size:70%;">67.50</span></td>
<td id="S4.T7.20.1.31.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.31.13.1" class="ltx_text" style="font-size:70%;">86.09</span></td>
<td id="S4.T7.20.1.31.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.31.14.1" class="ltx_text" style="font-size:70%;">41.69</span></td>
<td id="S4.T7.20.1.31.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.31.15.1" class="ltx_text" style="font-size:70%;">2.65</span></td>
<td id="S4.T7.20.1.31.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.31.16.1" class="ltx_text" style="font-size:70%;">84.34</span></td>
<td id="S4.T7.20.1.31.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.31.17.1" class="ltx_text" style="font-size:70%;">57.29</span></td>
<td id="S4.T7.20.1.31.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.31.18.1" class="ltx_text" style="font-size:70%;">53.22</span></td>
<td id="S4.T7.20.1.31.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.31.19.1" class="ltx_text" style="font-size:70%;">40.29</span></td>
</tr>
<tr id="S4.T7.20.1.32" class="ltx_tr">
<td id="S4.T7.20.1.32.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.32.2" class="ltx_td ltx_align_center ltx_border_t" colspan="18"><span id="S4.T7.20.1.32.2.1" class="ltx_text" style="font-size:70%;">KV-size 1024</span></td>
</tr>
<tr id="S4.T7.20.1.33" class="ltx_tr">
<td id="S4.T7.20.1.33.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.33.2" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T7.20.1.33.2.1" class="ltx_text" style="font-size:70%;">H2O</span></td>
<td id="S4.T7.20.1.33.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.33.3.1" class="ltx_text" style="font-size:70%;">23.67</span></td>
<td id="S4.T7.20.1.33.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.33.4.1" class="ltx_text" style="font-size:70%;">28.55</span></td>
<td id="S4.T7.20.1.33.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.33.5.1" class="ltx_text" style="font-size:70%;">46.4</span></td>
<td id="S4.T7.20.1.33.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.33.6.1" class="ltx_text" style="font-size:70%;">36.99</span></td>
<td id="S4.T7.20.1.33.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.33.7.1" class="ltx_text" style="font-size:70%;">24.82</span></td>
<td id="S4.T7.20.1.33.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.33.8.1" class="ltx_text" style="font-size:70%;">15.02</span></td>
<td id="S4.T7.20.1.33.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.33.9.1" class="ltx_text" style="font-size:70%;">25.21</span></td>
<td id="S4.T7.20.1.33.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.33.10.1" class="ltx_text" style="font-size:70%;">23.04</span></td>
<td id="S4.T7.20.1.33.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.33.11.1" class="ltx_text" style="font-size:70%;">25.77</span></td>
<td id="S4.T7.20.1.33.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.33.12.1" class="ltx_text" style="font-size:70%;">46.00</span></td>
<td id="S4.T7.20.1.33.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.33.13.1" class="ltx_text" style="font-size:70%;">85.93</span></td>
<td id="S4.T7.20.1.33.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.33.14.1" class="ltx_text" style="font-size:70%;">41.98</span></td>
<td id="S4.T7.20.1.33.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.33.15.1" class="ltx_text" style="font-size:70%;">3.24</span></td>
<td id="S4.T7.20.1.33.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.33.16.1" class="ltx_text" style="font-size:70%;">86.57</span></td>
<td id="S4.T7.20.1.33.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.33.17.1" class="ltx_text" style="font-size:70%;">56.40</span></td>
<td id="S4.T7.20.1.33.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.33.18.1" class="ltx_text" style="font-size:70%;">52.75</span></td>
<td id="S4.T7.20.1.33.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.33.19.1" class="ltx_text" style="font-size:70%;">38.90</span></td>
</tr>
<tr id="S4.T7.20.1.34" class="ltx_tr">
<td id="S4.T7.20.1.34.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.34.2" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T7.20.1.34.2.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T7.20.1.34.2.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T7.20.1.34.2.3" class="ltx_text" style="font-size:70%;">(30+30)</span>
</td>
<td id="S4.T7.20.1.34.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.34.3.1" class="ltx_text" style="font-size:70%;">23.65</span></td>
<td id="S4.T7.20.1.34.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.34.4.1" class="ltx_text" style="font-size:70%;">26.54</span></td>
<td id="S4.T7.20.1.34.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.34.5.1" class="ltx_text" style="font-size:70%;">47.00</span></td>
<td id="S4.T7.20.1.34.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.34.6.1" class="ltx_text" style="font-size:70%;">35.52</span></td>
<td id="S4.T7.20.1.34.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.34.7.1" class="ltx_text" style="font-size:70%;">24.79</span></td>
<td id="S4.T7.20.1.34.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.34.8.1" class="ltx_text" style="font-size:70%;">17.15</span></td>
<td id="S4.T7.20.1.34.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.34.9.1" class="ltx_text" style="font-size:70%;">23.64</span></td>
<td id="S4.T7.20.1.34.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.34.10.1" class="ltx_text" style="font-size:70%;">23.12</span></td>
<td id="S4.T7.20.1.34.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.34.11.1" class="ltx_text" style="font-size:70%;">25.20</span></td>
<td id="S4.T7.20.1.34.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.34.12.1" class="ltx_text" style="font-size:70%;">44.00</span></td>
<td id="S4.T7.20.1.34.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.34.13.1" class="ltx_text" style="font-size:70%;">86.38</span></td>
<td id="S4.T7.20.1.34.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.34.14.1" class="ltx_text" style="font-size:70%;">41.67</span></td>
<td id="S4.T7.20.1.34.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.34.15.1" class="ltx_text" style="font-size:70%;">3.46</span></td>
<td id="S4.T7.20.1.34.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.34.16.1" class="ltx_text" style="font-size:70%;">80.14</span></td>
<td id="S4.T7.20.1.34.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.34.17.1" class="ltx_text" style="font-size:70%;">56.53</span></td>
<td id="S4.T7.20.1.34.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.34.18.1" class="ltx_text" style="font-size:70%;">52.86</span></td>
<td id="S4.T7.20.1.34.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.34.19.1" class="ltx_text" style="font-size:70%;">38.23</span></td>
</tr>
<tr id="S4.T7.20.1.35" class="ltx_tr">
<td id="S4.T7.20.1.35.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.35.2" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T7.20.1.35.2.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T7.20.1.35.2.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T7.20.1.35.2.3" class="ltx_text" style="font-size:70%;">(30+10)</span>
</td>
<td id="S4.T7.20.1.35.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.35.3.1" class="ltx_text" style="font-size:70%;">24.13</span></td>
<td id="S4.T7.20.1.35.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.35.4.1" class="ltx_text" style="font-size:70%;">28.57</span></td>
<td id="S4.T7.20.1.35.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.35.5.1" class="ltx_text" style="font-size:70%;">46.31</span></td>
<td id="S4.T7.20.1.35.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.35.6.1" class="ltx_text" style="font-size:70%;">35.59</span></td>
<td id="S4.T7.20.1.35.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.35.7.1" class="ltx_text" style="font-size:70%;">24.92</span></td>
<td id="S4.T7.20.1.35.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.35.8.1" class="ltx_text" style="font-size:70%;">15.34</span></td>
<td id="S4.T7.20.1.35.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.35.9.1" class="ltx_text" style="font-size:70%;">24.58</span></td>
<td id="S4.T7.20.1.35.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.35.10.1" class="ltx_text" style="font-size:70%;">23.33</span></td>
<td id="S4.T7.20.1.35.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.35.11.1" class="ltx_text" style="font-size:70%;">25.93</span></td>
<td id="S4.T7.20.1.35.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.35.12.1" class="ltx_text" style="font-size:70%;">45.50</span></td>
<td id="S4.T7.20.1.35.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.35.13.1" class="ltx_text" style="font-size:70%;">85.91</span></td>
<td id="S4.T7.20.1.35.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.35.14.1" class="ltx_text" style="font-size:70%;">42.97</span></td>
<td id="S4.T7.20.1.35.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.35.15.1" class="ltx_text" style="font-size:70%;">2.57</span></td>
<td id="S4.T7.20.1.35.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.35.16.1" class="ltx_text" style="font-size:70%;">83.64</span></td>
<td id="S4.T7.20.1.35.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.35.17.1" class="ltx_text" style="font-size:70%;">55.39</span></td>
<td id="S4.T7.20.1.35.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.35.18.1" class="ltx_text" style="font-size:70%;">52.73</span></td>
<td id="S4.T7.20.1.35.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.35.19.1" class="ltx_text" style="font-size:70%;">38.59</span></td>
</tr>
<tr id="S4.T7.20.1.36" class="ltx_tr">
<td id="S4.T7.20.1.36.1" class="ltx_td ltx_align_left ltx_border_r">
<span id="S4.T7.20.1.36.1.1" class="ltx_ERROR undefined">\cdashline</span><span id="S4.T7.20.1.36.1.2" class="ltx_text" style="font-size:70%;">2-19</span>
</td>
<td id="S4.T7.20.1.36.2" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T7.20.1.36.2.1" class="ltx_text" style="font-size:70%;">SnapKV</span></td>
<td id="S4.T7.20.1.36.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.36.3.1" class="ltx_text" style="font-size:70%;">25.47</span></td>
<td id="S4.T7.20.1.36.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.36.4.1" class="ltx_text" style="font-size:70%;">29.57</span></td>
<td id="S4.T7.20.1.36.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.36.5.1" class="ltx_text" style="font-size:70%;">49.33</span></td>
<td id="S4.T7.20.1.36.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.36.6.1" class="ltx_text" style="font-size:70%;">40.90</span></td>
<td id="S4.T7.20.1.36.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.36.7.1" class="ltx_text" style="font-size:70%;">25.53</span></td>
<td id="S4.T7.20.1.36.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.36.8.1" class="ltx_text" style="font-size:70%;">19.01</span></td>
<td id="S4.T7.20.1.36.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.36.9.1" class="ltx_text" style="font-size:70%;">25.94</span></td>
<td id="S4.T7.20.1.36.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.36.10.1" class="ltx_text" style="font-size:70%;">23.89</span></td>
<td id="S4.T7.20.1.36.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.36.11.1" class="ltx_text" style="font-size:70%;">26.21</span></td>
<td id="S4.T7.20.1.36.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.36.12.1" class="ltx_text" style="font-size:70%;">69.50</span></td>
<td id="S4.T7.20.1.36.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.36.13.1" class="ltx_text" style="font-size:70%;">86.48</span></td>
<td id="S4.T7.20.1.36.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.36.14.1" class="ltx_text" style="font-size:70%;">42.10</span></td>
<td id="S4.T7.20.1.36.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.36.15.1" class="ltx_text" style="font-size:70%;">2.98</span></td>
<td id="S4.T7.20.1.36.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.36.16.1" class="ltx_text" style="font-size:70%;">88.56</span></td>
<td id="S4.T7.20.1.36.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.36.17.1" class="ltx_text" style="font-size:70%;">57.19</span></td>
<td id="S4.T7.20.1.36.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.36.18.1" class="ltx_text" style="font-size:70%;">53.60</span></td>
<td id="S4.T7.20.1.36.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.36.19.1" class="ltx_text" style="font-size:70%;">41.64</span></td>
</tr>
<tr id="S4.T7.20.1.37" class="ltx_tr">
<td id="S4.T7.20.1.37.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.37.2" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T7.20.1.37.2.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T7.20.1.37.2.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T7.20.1.37.2.3" class="ltx_text" style="font-size:70%;">(30+30)</span>
</td>
<td id="S4.T7.20.1.37.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.37.3.1" class="ltx_text" style="font-size:70%;">25.29</span></td>
<td id="S4.T7.20.1.37.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.37.4.1" class="ltx_text" style="font-size:70%;">29.25</span></td>
<td id="S4.T7.20.1.37.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.37.5.1" class="ltx_text" style="font-size:70%;">49.17</span></td>
<td id="S4.T7.20.1.37.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.37.6.1" class="ltx_text" style="font-size:70%;">41.25</span></td>
<td id="S4.T7.20.1.37.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.37.7.1" class="ltx_text" style="font-size:70%;">25.75</span></td>
<td id="S4.T7.20.1.37.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.37.8.1" class="ltx_text" style="font-size:70%;">19.37</span></td>
<td id="S4.T7.20.1.37.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.37.9.1" class="ltx_text" style="font-size:70%;">24.64</span></td>
<td id="S4.T7.20.1.37.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.37.10.1" class="ltx_text" style="font-size:70%;">23.02</span></td>
<td id="S4.T7.20.1.37.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.37.11.1" class="ltx_text" style="font-size:70%;">25.27</span></td>
<td id="S4.T7.20.1.37.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.37.12.1" class="ltx_text" style="font-size:70%;">69.00</span></td>
<td id="S4.T7.20.1.37.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.37.13.1" class="ltx_text" style="font-size:70%;">86.70</span></td>
<td id="S4.T7.20.1.37.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.37.14.1" class="ltx_text" style="font-size:70%;">40.92</span></td>
<td id="S4.T7.20.1.37.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.37.15.1" class="ltx_text" style="font-size:70%;">3.29</span></td>
<td id="S4.T7.20.1.37.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.37.16.1" class="ltx_text" style="font-size:70%;">82.06</span></td>
<td id="S4.T7.20.1.37.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.37.17.1" class="ltx_text" style="font-size:70%;">57.15</span></td>
<td id="S4.T7.20.1.37.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.37.18.1" class="ltx_text" style="font-size:70%;">54.15</span></td>
<td id="S4.T7.20.1.37.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.37.19.1" class="ltx_text" style="font-size:70%;">41.02</span></td>
</tr>
<tr id="S4.T7.20.1.38" class="ltx_tr">
<td id="S4.T7.20.1.38.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.38.2" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T7.20.1.38.2.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T7.20.1.38.2.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T7.20.1.38.2.3" class="ltx_text" style="font-size:70%;">(30+10)</span>
</td>
<td id="S4.T7.20.1.38.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.38.3.1" class="ltx_text" style="font-size:70%;">25.84</span></td>
<td id="S4.T7.20.1.38.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.38.4.1" class="ltx_text" style="font-size:70%;">29.30</span></td>
<td id="S4.T7.20.1.38.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.38.5.1" class="ltx_text" style="font-size:70%;">49.56</span></td>
<td id="S4.T7.20.1.38.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.38.6.1" class="ltx_text" style="font-size:70%;">41.44</span></td>
<td id="S4.T7.20.1.38.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.38.7.1" class="ltx_text" style="font-size:70%;">25.29</span></td>
<td id="S4.T7.20.1.38.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.38.8.1" class="ltx_text" style="font-size:70%;">19.02</span></td>
<td id="S4.T7.20.1.38.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.38.9.1" class="ltx_text" style="font-size:70%;">25.21</span></td>
<td id="S4.T7.20.1.38.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.38.10.1" class="ltx_text" style="font-size:70%;">23.73</span></td>
<td id="S4.T7.20.1.38.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.38.11.1" class="ltx_text" style="font-size:70%;">25.72</span></td>
<td id="S4.T7.20.1.38.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.38.12.1" class="ltx_text" style="font-size:70%;">69.00</span></td>
<td id="S4.T7.20.1.38.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.38.13.1" class="ltx_text" style="font-size:70%;">86.69</span></td>
<td id="S4.T7.20.1.38.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.38.14.1" class="ltx_text" style="font-size:70%;">42.55</span></td>
<td id="S4.T7.20.1.38.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.38.15.1" class="ltx_text" style="font-size:70%;">2.44</span></td>
<td id="S4.T7.20.1.38.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.38.16.1" class="ltx_text" style="font-size:70%;">85.76</span></td>
<td id="S4.T7.20.1.38.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.38.17.1" class="ltx_text" style="font-size:70%;">57.55</span></td>
<td id="S4.T7.20.1.38.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.38.18.1" class="ltx_text" style="font-size:70%;">54.10</span></td>
<td id="S4.T7.20.1.38.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.38.19.1" class="ltx_text" style="font-size:70%;">41.45</span></td>
</tr>
<tr id="S4.T7.20.1.39" class="ltx_tr">
<td id="S4.T7.20.1.39.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.39.2" class="ltx_td ltx_align_center ltx_border_t" colspan="18"><span id="S4.T7.20.1.39.2.1" class="ltx_text" style="font-size:70%;">KV-size 2048</span></td>
</tr>
<tr id="S4.T7.20.1.40" class="ltx_tr">
<td id="S4.T7.20.1.40.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.40.2" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T7.20.1.40.2.1" class="ltx_text" style="font-size:70%;">H2O</span></td>
<td id="S4.T7.20.1.40.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.40.3.1" class="ltx_text" style="font-size:70%;">25.76</span></td>
<td id="S4.T7.20.1.40.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.40.4.1" class="ltx_text" style="font-size:70%;">31.10</span></td>
<td id="S4.T7.20.1.40.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.40.5.1" class="ltx_text" style="font-size:70%;">49.06</span></td>
<td id="S4.T7.20.1.40.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.40.6.1" class="ltx_text" style="font-size:70%;">40.38</span></td>
<td id="S4.T7.20.1.40.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.40.7.1" class="ltx_text" style="font-size:70%;">26.43</span></td>
<td id="S4.T7.20.1.40.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.40.8.1" class="ltx_text" style="font-size:70%;">16.78</span></td>
<td id="S4.T7.20.1.40.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.40.9.1" class="ltx_text" style="font-size:70%;">27.17</span></td>
<td id="S4.T7.20.1.40.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.40.10.1" class="ltx_text" style="font-size:70%;">23.64</span></td>
<td id="S4.T7.20.1.40.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.40.11.1" class="ltx_text" style="font-size:70%;">26.69</span></td>
<td id="S4.T7.20.1.40.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.40.12.1" class="ltx_text" style="font-size:70%;">55.0</span></td>
<td id="S4.T7.20.1.40.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.40.13.1" class="ltx_text" style="font-size:70%;">86.35</span></td>
<td id="S4.T7.20.1.40.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.40.14.1" class="ltx_text" style="font-size:70%;">42.48</span></td>
<td id="S4.T7.20.1.40.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.40.15.1" class="ltx_text" style="font-size:70%;">2.72</span></td>
<td id="S4.T7.20.1.40.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.40.16.1" class="ltx_text" style="font-size:70%;">86.64</span></td>
<td id="S4.T7.20.1.40.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.40.17.1" class="ltx_text" style="font-size:70%;">56.98</span></td>
<td id="S4.T7.20.1.40.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.40.18.1" class="ltx_text" style="font-size:70%;">53.91</span></td>
<td id="S4.T7.20.1.40.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.40.19.1" class="ltx_text" style="font-size:70%;">40.69</span></td>
</tr>
<tr id="S4.T7.20.1.41" class="ltx_tr">
<td id="S4.T7.20.1.41.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.41.2" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T7.20.1.41.2.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T7.20.1.41.2.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T7.20.1.41.2.3" class="ltx_text" style="font-size:70%;">(30+30)</span>
</td>
<td id="S4.T7.20.1.41.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.41.3.1" class="ltx_text" style="font-size:70%;">25.60</span></td>
<td id="S4.T7.20.1.41.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.41.4.1" class="ltx_text" style="font-size:70%;">28.74</span></td>
<td id="S4.T7.20.1.41.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.41.5.1" class="ltx_text" style="font-size:70%;">47.54</span></td>
<td id="S4.T7.20.1.41.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.41.6.1" class="ltx_text" style="font-size:70%;">38.67</span></td>
<td id="S4.T7.20.1.41.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.41.7.1" class="ltx_text" style="font-size:70%;">26.25</span></td>
<td id="S4.T7.20.1.41.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.41.8.1" class="ltx_text" style="font-size:70%;">17.35</span></td>
<td id="S4.T7.20.1.41.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.41.9.1" class="ltx_text" style="font-size:70%;">24.54</span></td>
<td id="S4.T7.20.1.41.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.41.10.1" class="ltx_text" style="font-size:70%;">23.27</span></td>
<td id="S4.T7.20.1.41.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.41.11.1" class="ltx_text" style="font-size:70%;">26.15</span></td>
<td id="S4.T7.20.1.41.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.41.12.1" class="ltx_text" style="font-size:70%;">51.00</span></td>
<td id="S4.T7.20.1.41.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.41.13.1" class="ltx_text" style="font-size:70%;">87.01</span></td>
<td id="S4.T7.20.1.41.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.41.14.1" class="ltx_text" style="font-size:70%;">43.02</span></td>
<td id="S4.T7.20.1.41.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.41.15.1" class="ltx_text" style="font-size:70%;">2.94</span></td>
<td id="S4.T7.20.1.41.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.41.16.1" class="ltx_text" style="font-size:70%;">81.46</span></td>
<td id="S4.T7.20.1.41.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.41.17.1" class="ltx_text" style="font-size:70%;">56.41</span></td>
<td id="S4.T7.20.1.41.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.41.18.1" class="ltx_text" style="font-size:70%;">54.26</span></td>
<td id="S4.T7.20.1.41.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.41.19.1" class="ltx_text" style="font-size:70%;">39.64</span></td>
</tr>
<tr id="S4.T7.20.1.42" class="ltx_tr">
<td id="S4.T7.20.1.42.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.42.2" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T7.20.1.42.2.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T7.20.1.42.2.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T7.20.1.42.2.3" class="ltx_text" style="font-size:70%;">(30+10)</span>
</td>
<td id="S4.T7.20.1.42.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.42.3.1" class="ltx_text" style="font-size:70%;">25.64</span></td>
<td id="S4.T7.20.1.42.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.42.4.1" class="ltx_text" style="font-size:70%;">30.65</span></td>
<td id="S4.T7.20.1.42.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.42.5.1" class="ltx_text" style="font-size:70%;">48.95</span></td>
<td id="S4.T7.20.1.42.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.42.6.1" class="ltx_text" style="font-size:70%;">40.42</span></td>
<td id="S4.T7.20.1.42.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.42.7.1" class="ltx_text" style="font-size:70%;">26.43</span></td>
<td id="S4.T7.20.1.42.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.42.8.1" class="ltx_text" style="font-size:70%;">16.65</span></td>
<td id="S4.T7.20.1.42.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.42.9.1" class="ltx_text" style="font-size:70%;">26.76</span></td>
<td id="S4.T7.20.1.42.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.42.10.1" class="ltx_text" style="font-size:70%;">23.51</span></td>
<td id="S4.T7.20.1.42.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.42.11.1" class="ltx_text" style="font-size:70%;">26.59</span></td>
<td id="S4.T7.20.1.42.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.42.12.1" class="ltx_text" style="font-size:70%;">52.50</span></td>
<td id="S4.T7.20.1.42.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.42.13.1" class="ltx_text" style="font-size:70%;">86.53</span></td>
<td id="S4.T7.20.1.42.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.42.14.1" class="ltx_text" style="font-size:70%;">43.45</span></td>
<td id="S4.T7.20.1.42.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.42.15.1" class="ltx_text" style="font-size:70%;">2.66</span></td>
<td id="S4.T7.20.1.42.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.42.16.1" class="ltx_text" style="font-size:70%;">83.96</span></td>
<td id="S4.T7.20.1.42.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.42.17.1" class="ltx_text" style="font-size:70%;">56.55</span></td>
<td id="S4.T7.20.1.42.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.42.18.1" class="ltx_text" style="font-size:70%;">53.83</span></td>
<td id="S4.T7.20.1.42.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.42.19.1" class="ltx_text" style="font-size:70%;">40.32</span></td>
</tr>
<tr id="S4.T7.20.1.43" class="ltx_tr">
<td id="S4.T7.20.1.43.1" class="ltx_td ltx_align_left ltx_border_r">
<span id="S4.T7.20.1.43.1.1" class="ltx_ERROR undefined">\cdashline</span><span id="S4.T7.20.1.43.1.2" class="ltx_text" style="font-size:70%;">2-19</span>
</td>
<td id="S4.T7.20.1.43.2" class="ltx_td ltx_nopad_r ltx_align_left"><span id="S4.T7.20.1.43.2.1" class="ltx_text" style="font-size:70%;">SnapKV</span></td>
<td id="S4.T7.20.1.43.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.43.3.1" class="ltx_text" style="font-size:70%;">25.89</span></td>
<td id="S4.T7.20.1.43.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.43.4.1" class="ltx_text" style="font-size:70%;">32.56</span></td>
<td id="S4.T7.20.1.43.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.43.5.1" class="ltx_text" style="font-size:70%;">48.55</span></td>
<td id="S4.T7.20.1.43.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.43.6.1" class="ltx_text" style="font-size:70%;">41.68</span></td>
<td id="S4.T7.20.1.43.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.43.7.1" class="ltx_text" style="font-size:70%;">27.24</span></td>
<td id="S4.T7.20.1.43.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.43.8.1" class="ltx_text" style="font-size:70%;">18.75</span></td>
<td id="S4.T7.20.1.43.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.43.9.1" class="ltx_text" style="font-size:70%;">28.90</span></td>
<td id="S4.T7.20.1.43.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.43.10.1" class="ltx_text" style="font-size:70%;">24.47</span></td>
<td id="S4.T7.20.1.43.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.43.11.1" class="ltx_text" style="font-size:70%;">26.63</span></td>
<td id="S4.T7.20.1.43.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.43.12.1" class="ltx_text" style="font-size:70%;">70.00</span></td>
<td id="S4.T7.20.1.43.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.43.13.1" class="ltx_text" style="font-size:70%;">86.27</span></td>
<td id="S4.T7.20.1.43.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.43.14.1" class="ltx_text" style="font-size:70%;">42.57</span></td>
<td id="S4.T7.20.1.43.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.43.15.1" class="ltx_text" style="font-size:70%;">3.09</span></td>
<td id="S4.T7.20.1.43.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.43.16.1" class="ltx_text" style="font-size:70%;">86.93</span></td>
<td id="S4.T7.20.1.43.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.43.17.1" class="ltx_text" style="font-size:70%;">57.44</span></td>
<td id="S4.T7.20.1.43.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.43.18.1" class="ltx_text" style="font-size:70%;">53.83</span></td>
<td id="S4.T7.20.1.43.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.43.19.1" class="ltx_text" style="font-size:70%;">42.18</span></td>
</tr>
<tr id="S4.T7.20.1.44" class="ltx_tr">
<td id="S4.T7.20.1.44.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T7.20.1.44.2" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="S4.T7.20.1.44.2.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T7.20.1.44.2.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T7.20.1.44.2.3" class="ltx_text" style="font-size:70%;">(30+30)</span>
</td>
<td id="S4.T7.20.1.44.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.44.3.1" class="ltx_text" style="font-size:70%;">27.01</span></td>
<td id="S4.T7.20.1.44.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.44.4.1" class="ltx_text" style="font-size:70%;">30.72</span></td>
<td id="S4.T7.20.1.44.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.44.5.1" class="ltx_text" style="font-size:70%;">48.81</span></td>
<td id="S4.T7.20.1.44.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.44.6.1" class="ltx_text" style="font-size:70%;">41.15</span></td>
<td id="S4.T7.20.1.44.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.44.7.1" class="ltx_text" style="font-size:70%;">26.93</span></td>
<td id="S4.T7.20.1.44.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.44.8.1" class="ltx_text" style="font-size:70%;">18.93</span></td>
<td id="S4.T7.20.1.44.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.44.9.1" class="ltx_text" style="font-size:70%;">25.81</span></td>
<td id="S4.T7.20.1.44.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.44.10.1" class="ltx_text" style="font-size:70%;">23.59</span></td>
<td id="S4.T7.20.1.44.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.44.11.1" class="ltx_text" style="font-size:70%;">26.42</span></td>
<td id="S4.T7.20.1.44.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.44.12.1" class="ltx_text" style="font-size:70%;">70.00</span></td>
<td id="S4.T7.20.1.44.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.44.13.1" class="ltx_text" style="font-size:70%;">86.82</span></td>
<td id="S4.T7.20.1.44.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.44.14.1" class="ltx_text" style="font-size:70%;">41.91</span></td>
<td id="S4.T7.20.1.44.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.44.15.1" class="ltx_text" style="font-size:70%;">3.05</span></td>
<td id="S4.T7.20.1.44.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.44.16.1" class="ltx_text" style="font-size:70%;">82.65</span></td>
<td id="S4.T7.20.1.44.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.44.17.1" class="ltx_text" style="font-size:70%;">57.01</span></td>
<td id="S4.T7.20.1.44.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center"><span id="S4.T7.20.1.44.18.1" class="ltx_text" style="font-size:70%;">54.25</span></td>
<td id="S4.T7.20.1.44.19" class="ltx_td ltx_nopad_l ltx_align_center"><span id="S4.T7.20.1.44.19.1" class="ltx_text" style="font-size:70%;">41.57</span></td>
</tr>
<tr id="S4.T7.20.1.45" class="ltx_tr">
<td id="S4.T7.20.1.45.1" class="ltx_td ltx_border_bb ltx_border_r"></td>
<td id="S4.T7.20.1.45.2" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">
<span id="S4.T7.20.1.45.2.1" class="ltx_text" style="font-size:70%;">+</span><span id="S4.T7.20.1.45.2.2" class="ltx_text ltx_font_bold" style="font-size:70%;">ThinK</span><span id="S4.T7.20.1.45.2.3" class="ltx_text" style="font-size:70%;">(30+10)</span>
</td>
<td id="S4.T7.20.1.45.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T7.20.1.45.3.1" class="ltx_text" style="font-size:70%;">26.22</span></td>
<td id="S4.T7.20.1.45.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T7.20.1.45.4.1" class="ltx_text" style="font-size:70%;">32.69</span></td>
<td id="S4.T7.20.1.45.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T7.20.1.45.5.1" class="ltx_text" style="font-size:70%;">48.96</span></td>
<td id="S4.T7.20.1.45.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T7.20.1.45.6.1" class="ltx_text" style="font-size:70%;">40.83</span></td>
<td id="S4.T7.20.1.45.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T7.20.1.45.7.1" class="ltx_text" style="font-size:70%;">26.70</span></td>
<td id="S4.T7.20.1.45.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T7.20.1.45.8.1" class="ltx_text" style="font-size:70%;">19.02</span></td>
<td id="S4.T7.20.1.45.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T7.20.1.45.9.1" class="ltx_text" style="font-size:70%;">27.87</span></td>
<td id="S4.T7.20.1.45.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T7.20.1.45.10.1" class="ltx_text" style="font-size:70%;">24.23</span></td>
<td id="S4.T7.20.1.45.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T7.20.1.45.11.1" class="ltx_text" style="font-size:70%;">26.64</span></td>
<td id="S4.T7.20.1.45.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T7.20.1.45.12.1" class="ltx_text" style="font-size:70%;">70.00</span></td>
<td id="S4.T7.20.1.45.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T7.20.1.45.13.1" class="ltx_text" style="font-size:70%;">86.65</span></td>
<td id="S4.T7.20.1.45.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T7.20.1.45.14.1" class="ltx_text" style="font-size:70%;">42.63</span></td>
<td id="S4.T7.20.1.45.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T7.20.1.45.15.1" class="ltx_text" style="font-size:70%;">2.22</span></td>
<td id="S4.T7.20.1.45.16" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T7.20.1.45.16.1" class="ltx_text" style="font-size:70%;">85.13</span></td>
<td id="S4.T7.20.1.45.17" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T7.20.1.45.17.1" class="ltx_text" style="font-size:70%;">57.00</span></td>
<td id="S4.T7.20.1.45.18" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S4.T7.20.1.45.18.1" class="ltx_text" style="font-size:70%;">54.28</span></td>
<td id="S4.T7.20.1.45.19" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_bb"><span id="S4.T7.20.1.45.19.1" class="ltx_text" style="font-size:70%;">41.94</span></td>
</tr>
</table>
</span></div>
</figure>
<figure id="S4.F6" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="S4.F6.1" class="ltx_block ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:433.6pt;">
<img src="/html/2407.21018/assets/x10.png" id="S4.F6.1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="110" alt="Refer to caption">
<p id="S4.F6.1.1" class="ltx_p ltx_align_center">(a) KV-size 128, SnapKV, Acc.77.8</p>
</div>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="S4.F6.3" class="ltx_block ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:433.6pt;">
<img src="/html/2407.21018/assets/x11.png" id="S4.F6.2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="110" alt="Refer to caption">
<p id="S4.F6.3.1" class="ltx_p ltx_align_center">(b) KV-size 128, SnapKV+ThinK(<math id="S4.F6.3.1.m1.1" class="ltx_Math" alttext="40" display="inline"><semantics id="S4.F6.3.1.m1.1a"><mn id="S4.F6.3.1.m1.1.1" xref="S4.F6.3.1.m1.1.1.cmml">40</mn><annotation-xml encoding="MathML-Content" id="S4.F6.3.1.m1.1b"><cn type="integer" id="S4.F6.3.1.m1.1.1.cmml" xref="S4.F6.3.1.m1.1.1">40</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.F6.3.1.m1.1c">40</annotation></semantics></math>) Acc.78.6</p>
</div>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="S4.F6.4" class="ltx_block ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:433.6pt;">
<img src="/html/2407.21018/assets/x12.png" id="S4.F6.4.g1" class="ltx_graphics ltx_img_landscape" width="461" height="110" alt="Refer to caption">
<p id="S4.F6.4.1" class="ltx_p ltx_align_center">(c) KV-size 1024, SnapKV, Acc.90.4</p>
</div>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="S4.F6.6" class="ltx_block ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:433.6pt;">
<img src="/html/2407.21018/assets/x13.png" id="S4.F6.5.g1" class="ltx_graphics ltx_img_landscape" width="461" height="110" alt="Refer to caption">
<p id="S4.F6.6.1" class="ltx_p ltx_align_center">(d) KV-size 1024, SnapKV+ThinK(<math id="S4.F6.6.1.m1.1" class="ltx_Math" alttext="50" display="inline"><semantics id="S4.F6.6.1.m1.1a"><mn id="S4.F6.6.1.m1.1.1" xref="S4.F6.6.1.m1.1.1.cmml">50</mn><annotation-xml encoding="MathML-Content" id="S4.F6.6.1.m1.1b"><cn type="integer" id="S4.F6.6.1.m1.1.1.cmml" xref="S4.F6.6.1.m1.1.1">50</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.F6.6.1.m1.1c">50</annotation></semantics></math>), Acc.90.8</p>
</div>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F6.8.1.1" class="ltx_text" style="font-size:90%;">Figure 6</span>: </span><span id="S4.F6.9.2" class="ltx_text" style="font-size:90%;">Needle-in-a-Haystack test performance comparison with Mistral-7B-Instruct-v0.2.</span></figcaption>
</figure>
</section>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Related Work</h2>

<div id="S5.p1" class="ltx_para ltx_noindent">
<p id="S5.p1.1" class="ltx_p">In scenarios involving long contexts, the most significant computational burden from the attention mechanism is the key-value (KV) cache. Reducing the KV cache is a high priority for optimizing deployment efficiency. System-level optimizations, such as FlashAttention <cite class="ltx_cite ltx_citemacro_citep">(Dao,, <a href="#bib.bib10" title="" class="ltx_ref">2023</a>)</cite> and PagedAttention <cite class="ltx_cite ltx_citemacro_citep">(Kwon et al.,, <a href="#bib.bib27" title="" class="ltx_ref">2023</a>)</cite>, have been developed to address this issue. Additionally, algorithm-level optimizations are being explored to further enhance efficiency.</p>
</div>
<div id="S5.p2" class="ltx_para ltx_noindent">
<p id="S5.p2.1" class="ltx_p"><span id="S5.p2.1.1" class="ltx_text ltx_font_bold">KV Cache Eviction.</span> StreamingLLM <cite class="ltx_cite ltx_citemacro_citep">(<a href="#bib.bib49" title="" class="ltx_ref">Xiao et al., 2023b, </a>)</cite> maintains a few initial tokens and some recent tokens based on the observation of attention sink, which may result in the loss of important information carried by the dropped tokens. H2O <cite class="ltx_cite ltx_citemacro_citep">(<a href="#bib.bib56" title="" class="ltx_ref">Zhang et al., 2024c, </a>)</cite> retains only a small portion of the tokens by greedily dropping tokens based on their contributions to the cumulative attention. SnapKV <cite class="ltx_cite ltx_citemacro_citep">(Li et al.,, <a href="#bib.bib29" title="" class="ltx_ref">2024</a>)</cite> selects clustered important KV positions for each attention head from an ‘observation’ window located at the end of the prompts. FastGen <cite class="ltx_cite ltx_citemacro_citep">(Ge et al.,, <a href="#bib.bib18" title="" class="ltx_ref">2023</a>)</cite> adaptively evicts tokens from attention heads that emphasize local contexts. This approach focuses on discarding non-special tokens centered on special tokens, while the standard KV cache is used only for attention heads that broadly attend to all tokens. PyramidKV <cite class="ltx_cite ltx_citemacro_citep">(<a href="#bib.bib55" title="" class="ltx_ref">Zhang et al., 2024b, </a>)</cite> and PyramidInfer <cite class="ltx_cite ltx_citemacro_citep">(Yang et al.,, <a href="#bib.bib52" title="" class="ltx_ref">2024</a>)</cite> considers adjusting the KV cache size across different layers by allocating more cache in the lower layers and less in the higher ones.</p>
</div>
<div id="S5.p3" class="ltx_para ltx_noindent">
<p id="S5.p3.1" class="ltx_p"><span id="S5.p3.1.1" class="ltx_text ltx_font_bold">KV Cache Quantization.</span> SmoothQuant <cite class="ltx_cite ltx_citemacro_citep">(<a href="#bib.bib48" title="" class="ltx_ref">Xiao et al., 2023a, </a>)</cite> can quantize the KV cache to 8-bit with minimal performance degradation. Q-Hitter <cite class="ltx_cite ltx_citemacro_citep">(<a href="#bib.bib56" title="" class="ltx_ref">Zhang et al., 2024c, </a>)</cite> uses accumulated attention scores and ‘Quantization Friendliness’ metrics to identify tokens that are essential for maintaining the generalization capabilities of LLMs and are suitable for KV cache quantization. Some studies have found that the key cache and value cache should be quantized differently <cite class="ltx_cite ltx_citemacro_citep">(<a href="#bib.bib32" title="" class="ltx_ref">Liu et al., 2024b, </a>; Hooper et al.,, <a href="#bib.bib21" title="" class="ltx_ref">2024</a>)</cite>: the key cache should be quantized per-channel, while the value cache should be quantized per-token.</p>
</div>
<div id="S5.p4" class="ltx_para ltx_noindent">
<p id="S5.p4.1" class="ltx_p"><span id="S5.p4.1.1" class="ltx_text ltx_font_bold">Structured Pruning of LLMs.</span> Structured pruning  <cite class="ltx_cite ltx_citemacro_citep">(Ma et al.,, <a href="#bib.bib33" title="" class="ltx_ref">2023</a>; Ding et al.,, <a href="#bib.bib13" title="" class="ltx_ref">2023</a>)</cite> of LLMs traditionally focuses on removing unimportant layers, heads, and hidden dimensions, which often results in significant performance degradation. In contrast, our methodology preserves the original architecture of the LLM and specifically targets the channel dimension within each head’s key cache. By dynamically identifying unimportant channels based on data dependant criterion, our approach greatly reduce the key cache-size with negligible performance loss.</p>
</div>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>End Note and Future Direction</h2>

<div id="S6.p1" class="ltx_para ltx_noindent">
<p id="S6.p1.1" class="ltx_p">Motivated by the observation that certain channels have significantly larger magnitudes compared to others, and the singular value analysis indicates that the key cache is inherently low-rank <cite class="ltx_cite ltx_citemacro_citep">(Singhania et al.,, <a href="#bib.bib39" title="" class="ltx_ref">2024</a>)</cite>, we propose <span id="S6.p1.1.1" class="ltx_text ltx_font_bold">ThinK</span> to perform pruning over the key cache channel.
The proposed pruning strategy is query-dependant and optimized based on the attention scores, ensuring that essential information is retained for each input query.
In addition, <span id="S6.p1.1.2" class="ltx_text ltx_font_bold">ThinK</span> can be seamlessly integrated with other popular token-level KV cache quantization techniques <cite class="ltx_cite ltx_citemacro_citep">(Li et al.,, <a href="#bib.bib29" title="" class="ltx_ref">2024</a>; <a href="#bib.bib56" title="" class="ltx_ref">Zhang et al., 2024c, </a>)</cite>, further enhancing inference efficiency.
Extensive experiments on LongBench <cite class="ltx_cite ltx_citemacro_citep">(Bai et al.,, <a href="#bib.bib2" title="" class="ltx_ref">2023</a>)</cite> and Needle-in-a-Haystack tests with two foundation models demonstrate the effectiveness and robustness of our query-dependent channel pruning method.
Our approach achieves comparable or superior performance to baseline methods while reducing the key cache size by <math id="S6.p1.1.m1.1" class="ltx_Math" alttext="40" display="inline"><semantics id="S6.p1.1.m1.1a"><mn id="S6.p1.1.m1.1.1" xref="S6.p1.1.m1.1.1.cmml">40</mn><annotation-xml encoding="MathML-Content" id="S6.p1.1.m1.1b"><cn type="integer" id="S6.p1.1.m1.1.1.cmml" xref="S6.p1.1.m1.1.1">40</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.p1.1.m1.1c">40</annotation></semantics></math>%.
Our analysis indicates that <span id="S6.p1.1.3" class="ltx_text ltx_font_bold">ThinK</span> can maintain superior performance over baselines with a smaller KV cache size under equivalent memory consumption conditions.</p>
</div>
<div id="S6.p2" class="ltx_para ltx_noindent">
<p id="S6.p2.1" class="ltx_p">Future work will focus on strategies to increase the pruning ratio without compromising model performance, and exploring pruning techniques for value caches.
We also plan to evaluate channel importance from a more sophisticated and compositional perspective, incorporating both token-level and channel-level information to calculate importance.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ainslie et al.,  (2023)</span>
<span class="ltx_bibblock">
Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. (2023).

</span>
<span class="ltx_bibblock">Gqa: Training generalized multi-query transformer models from multi-head checkpoints.

</span>
<span class="ltx_bibblock"><span id="bib.bib1.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2305.13245</span>.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bai et al.,  (2023)</span>
<span class="ltx_bibblock">
Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., et al. (2023).

</span>
<span class="ltx_bibblock">Longbench: A bilingual, multitask benchmark for long context understanding.

</span>
<span class="ltx_bibblock"><span id="bib.bib2.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2308.14508</span>.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bhojanapalli et al.,  (2021)</span>
<span class="ltx_bibblock">
Bhojanapalli, S., Chakrabarti, A., Jain, H., Kumar, S., Lukasik, M., and Veit, A. (2021).

</span>
<span class="ltx_bibblock">Eigen analysis of self-attention and its reconstruction from partial computation.

</span>
<span class="ltx_bibblock"><span id="bib.bib3.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2106.08823</span>.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Blalock et al.,  (2020)</span>
<span class="ltx_bibblock">
Blalock, D., Gonzalez Ortiz, J. J., Frankle, J., and Guttag, J. (2020).

</span>
<span class="ltx_bibblock">What is the state of neural network pruning?

</span>
<span class="ltx_bibblock"><span id="bib.bib4.1.1" class="ltx_text ltx_font_italic">Proceedings of machine learning and systems</span>, 2:129–146.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bordes et al.,  (2016)</span>
<span class="ltx_bibblock">
Bordes, A., Boureau, Y.-L., and Weston, J. (2016).

</span>
<span class="ltx_bibblock">Learning end-to-end goal-oriented dialog.

</span>
<span class="ltx_bibblock"><span id="bib.bib5.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:1605.07683</span>.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Brandon et al.,  (2024)</span>
<span class="ltx_bibblock">
Brandon, W., Mishra, M., Nrusimha, A., Panda, R., and Kelly, J. R. (2024).

</span>
<span class="ltx_bibblock">Reducing transformer key-value cache size with cross-layer attention.

</span>
<span class="ltx_bibblock"><span id="bib.bib6.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2405.12981</span>.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Brown et al.,  (2020)</span>
<span class="ltx_bibblock">
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020).

</span>
<span class="ltx_bibblock">Language models are few-shot learners.

</span>
<span class="ltx_bibblock"><span id="bib.bib7.1.1" class="ltx_text ltx_font_italic">Advances in neural information processing systems</span>, 33:1877–1901.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">(8)</span>
<span class="ltx_bibblock">
Chen, B., Dao, T., Winsor, E., Song, Z., Rudra, A., and Ré, C. (2021a).

</span>
<span class="ltx_bibblock">Scatterbrain: Unifying sparse and low-rank attention.

</span>
<span class="ltx_bibblock"><span id="bib.bib8.1.1" class="ltx_text ltx_font_italic">Advances in Neural Information Processing Systems</span>, 34:17413–17426.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">(9)</span>
<span class="ltx_bibblock">
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021b).

</span>
<span class="ltx_bibblock">Evaluating large language models trained on code.

</span>
<span class="ltx_bibblock"><span id="bib.bib9.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2107.03374</span>.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dao,  (2023)</span>
<span class="ltx_bibblock">
Dao, T. (2023).

</span>
<span class="ltx_bibblock">Flashattention-2: Faster attention with better parallelism and work partitioning.

</span>
<span class="ltx_bibblock"><span id="bib.bib10.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2307.08691</span>.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Demmel,  (1997)</span>
<span class="ltx_bibblock">
Demmel, J. W. (1997).

</span>
<span class="ltx_bibblock"><span id="bib.bib11.1.1" class="ltx_text ltx_font_italic">Applied numerical linear algebra</span>.

</span>
<span class="ltx_bibblock">SIAM.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dettmers et al.,  (2024)</span>
<span class="ltx_bibblock">
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. (2024).

</span>
<span class="ltx_bibblock">Qlora: Efficient finetuning of quantized llms.

</span>
<span class="ltx_bibblock"><span id="bib.bib12.1.1" class="ltx_text ltx_font_italic">Advances in Neural Information Processing Systems</span>, 36.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ding et al.,  (2023)</span>
<span class="ltx_bibblock">
Ding, N., Chen, Y., Xu, B., Qin, Y., Zheng, Z., Hu, S., Liu, Z., Sun, M., and Zhou, B. (2023).

</span>
<span class="ltx_bibblock">Enhancing chat language models by scaling high-quality instructional conversations.

</span>
<span class="ltx_bibblock"><span id="bib.bib13.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2305.14233</span>.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">(14)</span>
<span class="ltx_bibblock">
Dong, H., Xiong, W., Pang, B., Wang, H., Zhao, H., Zhou, Y., Jiang, N., Sahoo, D., Xiong, C., and Zhang, T. (2024a).

</span>
<span class="ltx_bibblock">Rlhf workflow: From reward modeling to online rlhf.

</span>
<span class="ltx_bibblock"><span id="bib.bib14.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2405.07863</span>.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">(15)</span>
<span class="ltx_bibblock">
Dong, H., Yang, X., Zhang, Z., Wang, Z., Chi, Y., and Chen, B. (2024b).

</span>
<span class="ltx_bibblock">Get more with less: Synthesizing recurrence with kv cache compression for efficient llm inference.

</span>
<span class="ltx_bibblock"><span id="bib.bib15.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2402.09398</span>.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Frankle and Carbin,  (2018)</span>
<span class="ltx_bibblock">
Frankle, J. and Carbin, M. (2018).

</span>
<span class="ltx_bibblock">The lottery ticket hypothesis: Finding sparse, trainable neural networks.

</span>
<span class="ltx_bibblock"><span id="bib.bib16.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:1803.03635</span>.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Frantar et al.,  (2022)</span>
<span class="ltx_bibblock">
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. (2022).

</span>
<span class="ltx_bibblock">Gptq: Accurate post-training quantization for generative pre-trained transformers.

</span>
<span class="ltx_bibblock"><span id="bib.bib17.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2210.17323</span>.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ge et al.,  (2023)</span>
<span class="ltx_bibblock">
Ge, S., Zhang, Y., Liu, L., Zhang, M., Han, J., and Gao, J. (2023).

</span>
<span class="ltx_bibblock">Model tells you what to discard: Adaptive kv cache compression for llms.

</span>
<span class="ltx_bibblock"><span id="bib.bib18.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2310.01801</span>.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hadi et al.,  (2023)</span>
<span class="ltx_bibblock">
Hadi, M. U., Qureshi, R., Shah, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Wu, J., Mirjalili, S., et al. (2023).

</span>
<span class="ltx_bibblock">Large language models: A comprehensive survey of its applications, challenges, limitations, and future prospects.

</span>
<span class="ltx_bibblock"><span id="bib.bib19.1.1" class="ltx_text ltx_font_italic">TechRxiv</span>.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hoffmann et al.,  (2022)</span>
<span class="ltx_bibblock">
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022).

</span>
<span class="ltx_bibblock">Training compute-optimal large language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib20.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2203.15556</span>.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hooper et al.,  (2024)</span>
<span class="ltx_bibblock">
Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y. S., Keutzer, K., and Gholami, A. (2024).

</span>
<span class="ltx_bibblock">Kvquant: Towards 10 million context length llm inference with kv cache quantization.

</span>
<span class="ltx_bibblock"><span id="bib.bib21.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2401.18079</span>.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hu et al.,  (2021)</span>
<span class="ltx_bibblock">
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2021).

</span>
<span class="ltx_bibblock">Lora: Low-rank adaptation of large language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib22.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2106.09685</span>.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jia et al.,  (2018)</span>
<span class="ltx_bibblock">
Jia, Z., Maggioni, M., Staiger, B., and Scarpazza, D. P. (2018).

</span>
<span class="ltx_bibblock">Dissecting the nvidia volta gpu architecture via microbenchmarking.

</span>
<span class="ltx_bibblock"><span id="bib.bib23.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:1804.06826</span>.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jiang et al.,  (2023)</span>
<span class="ltx_bibblock">
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. (2023).

</span>
<span class="ltx_bibblock">Mistral 7b.

</span>
<span class="ltx_bibblock"><span id="bib.bib24.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2310.06825</span>.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kamradt,  (2023)</span>
<span class="ltx_bibblock">
Kamradt, G. (2023).

</span>
<span class="ltx_bibblock">Needle In A Haystack - pressure testing LLMs.

</span>
<span class="ltx_bibblock"><span id="bib.bib25.1.1" class="ltx_text ltx_font_italic">Github</span>.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kaplan et al.,  (2020)</span>
<span class="ltx_bibblock">
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020).

</span>
<span class="ltx_bibblock">Scaling laws for neural language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib26.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2001.08361</span>.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kwon et al.,  (2023)</span>
<span class="ltx_bibblock">
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. (2023).

</span>
<span class="ltx_bibblock">Efficient memory management for large language model serving with pagedattention.

</span>
<span class="ltx_bibblock">In <span id="bib.bib27.1.1" class="ltx_text ltx_font_italic">Proceedings of the 29th Symposium on Operating Systems Principles</span>, pages 611–626.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Leviathan et al.,  (2023)</span>
<span class="ltx_bibblock">
Leviathan, Y., Kalman, M., and Matias, Y. (2023).

</span>
<span class="ltx_bibblock">Fast inference from transformers via speculative decoding.

</span>
<span class="ltx_bibblock">In <span id="bib.bib28.1.1" class="ltx_text ltx_font_italic">International Conference on Machine Learning</span>, pages 19274–19286. PMLR.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al.,  (2024)</span>
<span class="ltx_bibblock">
Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P., and Chen, D. (2024).

</span>
<span class="ltx_bibblock">Snapkv: Llm knows what you are looking for before generation.

</span>
<span class="ltx_bibblock"><span id="bib.bib29.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2404.14469</span>.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lin et al.,  (2024)</span>
<span class="ltx_bibblock">
Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. (2024).

</span>
<span class="ltx_bibblock">Awq: Activation-aware weight quantization for on-device llm compression and acceleration.

</span>
<span class="ltx_bibblock"><span id="bib.bib30.1.1" class="ltx_text ltx_font_italic">Proceedings of Machine Learning and Systems</span>, 6:87–100.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">(31)</span>
<span class="ltx_bibblock">
Liu, A., Liu, J., Pan, Z., He, Y., Haffari, G., and Zhuang, B. (2024a).

</span>
<span class="ltx_bibblock">Minicache: Kv cache compression in depth dimension for large language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib31.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2405.14366</span>.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">(32)</span>
<span class="ltx_bibblock">
Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., and Hu, X. (2024b).

</span>
<span class="ltx_bibblock">Kivi: A tuning-free asymmetric 2bit quantization for kv cache.

</span>
<span class="ltx_bibblock"><span id="bib.bib32.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2402.02750</span>.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ma et al.,  (2023)</span>
<span class="ltx_bibblock">
Ma, X., Fang, G., and Wang, X. (2023).

</span>
<span class="ltx_bibblock">Llm-pruner: On the structural pruning of large language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib33.1.1" class="ltx_text ltx_font_italic">Advances in neural information processing systems</span>, 36:21702–21720.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Meta,  (2024)</span>
<span class="ltx_bibblock">
Meta, A. (2024).

</span>
<span class="ltx_bibblock">Introducing meta llama 3: The most capable openly available llm to date.

</span>
<span class="ltx_bibblock"><span id="bib.bib34.1.1" class="ltx_text ltx_font_italic">Meta AI</span>.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">OpenAI,  (2022)</span>
<span class="ltx_bibblock">
OpenAI (2022).

</span>
<span class="ltx_bibblock">OpenAI: Introducing ChatGPT.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">OpenAI,  (2023)</span>
<span class="ltx_bibblock">
OpenAI (2023).

</span>
<span class="ltx_bibblock">Gpt-4 technical report.

</span>
<span class="ltx_bibblock"><span id="bib.bib36.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2303.08774</span>.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Reid et al.,  (2024)</span>
<span class="ltx_bibblock">
Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lillicrap, T., Alayrac, J.-b., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., et al. (2024).

</span>
<span class="ltx_bibblock">Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.

</span>
<span class="ltx_bibblock"><span id="bib.bib37.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2403.05530</span>.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Scao et al.,  (2022)</span>
<span class="ltx_bibblock">
Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., Gallé, M., et al. (2022).

</span>
<span class="ltx_bibblock">Bloom: A 176b-parameter open-access multilingual language model.

</span>
<span class="ltx_bibblock"><span id="bib.bib38.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2211.05100</span>.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Singhania et al.,  (2024)</span>
<span class="ltx_bibblock">
Singhania, P., Singh, S., He, S., Feizi, S., and Bhatele, A. (2024).

</span>
<span class="ltx_bibblock">Loki: Low-rank keys for efficient sparse attention.

</span>
<span class="ltx_bibblock"><span id="bib.bib39.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2406.02542</span>.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Strubell et al.,  (2020)</span>
<span class="ltx_bibblock">
Strubell, E., Ganesh, A., and McCallum, A. (2020).

</span>
<span class="ltx_bibblock">Energy and policy considerations for modern deep learning research.

</span>
<span class="ltx_bibblock">In <span id="bib.bib40.1.1" class="ltx_text ltx_font_italic">Proceedings of the AAAI conference on artificial intelligence</span>, volume 34, pages 13693–13696.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">(41)</span>
<span class="ltx_bibblock">
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023a).

</span>
<span class="ltx_bibblock">Llama: Open and efficient foundation language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib41.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2302.13971</span>.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">(42)</span>
<span class="ltx_bibblock">
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023b).

</span>
<span class="ltx_bibblock">Llama 2: Open foundation and fine-tuned chat models.

</span>
<span class="ltx_bibblock"><span id="bib.bib42.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2307.09288</span>.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Vaswani et al.,  (2017)</span>
<span class="ltx_bibblock">
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017).

</span>
<span class="ltx_bibblock">Attention is all you need.

</span>
<span class="ltx_bibblock"><span id="bib.bib43.1.1" class="ltx_text ltx_font_italic">Advances in neural information processing systems</span>, 30.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wan et al.,  (2023)</span>
<span class="ltx_bibblock">
Wan, Z., Wang, X., Liu, C., Alam, S., Zheng, Y., Qu, Z., Yan, S., Zhu, Y., Zhang, Q., Chowdhury, M., et al. (2023).

</span>
<span class="ltx_bibblock">Efficient large language models: A survey.

</span>
<span class="ltx_bibblock"><span id="bib.bib44.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2312.03863</span>, 1.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al.,  (2020)</span>
<span class="ltx_bibblock">
Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H. (2020).

</span>
<span class="ltx_bibblock">Linformer: Self-attention with linear complexity.

</span>
<span class="ltx_bibblock"><span id="bib.bib45.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2006.04768</span>.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wolf et al.,  (2020)</span>
<span class="ltx_bibblock">
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020).

</span>
<span class="ltx_bibblock">Transformers: State-of-the-art natural language processing.

</span>
<span class="ltx_bibblock">In <span id="bib.bib46.1.1" class="ltx_text ltx_font_italic">Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations</span>, pages 38–45.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wu and Tu,  (2024)</span>
<span class="ltx_bibblock">
Wu, H. and Tu, K. (2024).

</span>
<span class="ltx_bibblock">Layer-condensed kv cache for efficient inference of large language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib47.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2405.10637</span>.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">(48)</span>
<span class="ltx_bibblock">
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. (2023a).

</span>
<span class="ltx_bibblock">Smoothquant: Accurate and efficient post-training quantization for large language models.

</span>
<span class="ltx_bibblock">In <span id="bib.bib48.1.1" class="ltx_text ltx_font_italic">International Conference on Machine Learning</span>, pages 38087–38099. PMLR.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">(49)</span>
<span class="ltx_bibblock">
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. (2023b).

</span>
<span class="ltx_bibblock">Efficient streaming language models with attention sinks.

</span>
<span class="ltx_bibblock"><span id="bib.bib49.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2309.17453</span>.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xiong et al.,  (2023)</span>
<span class="ltx_bibblock">
Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., et al. (2023).

</span>
<span class="ltx_bibblock">Effective long-context scaling of foundation models.

</span>
<span class="ltx_bibblock"><span id="bib.bib50.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2309.16039</span>.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xu et al.,  (2023)</span>
<span class="ltx_bibblock">
Xu, Y., Xie, L., Gu, X., Chen, X., Chang, H., Zhang, H., Chen, Z., Zhang, X., and Tian, Q. (2023).

</span>
<span class="ltx_bibblock">Qa-lora: Quantization-aware low-rank adaptation of large language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib51.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2309.14717</span>.

</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yang et al.,  (2024)</span>
<span class="ltx_bibblock">
Yang, D., Han, X., Gao, Y., Hu, Y., Zhang, S., and Zhao, H. (2024).

</span>
<span class="ltx_bibblock">Pyramidinfer: Pyramid kv cache compression for high-throughput llm inference.

</span>
<span class="ltx_bibblock"><span id="bib.bib52.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2405.12532</span>.

</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">(53)</span>
<span class="ltx_bibblock">
Zhang, T., Ladhak, F., Durmus, E., Liang, P., McKeown, K., and Hashimoto, T. B. (2024a).

</span>
<span class="ltx_bibblock">Benchmarking large language models for news summarization.

</span>
<span class="ltx_bibblock"><span id="bib.bib53.1.1" class="ltx_text ltx_font_italic">Transactions of the Association for Computational Linguistics</span>, 12:39–57.

</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al.,  (2019)</span>
<span class="ltx_bibblock">
Zhang, X., Wei, F., and Zhou, M. (2019).

</span>
<span class="ltx_bibblock">Hibert: Document level pre-training of hierarchical bidirectional transformers for document summarization.

</span>
<span class="ltx_bibblock"><span id="bib.bib54.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:1905.06566</span>.

</span>
</li>
<li id="bib.bib55" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">(55)</span>
<span class="ltx_bibblock">
Zhang, Y., Gao, B., Liu, T., Lu, K., Xiong, W., Dong, Y., Chang, B., Hu, J., Xiao, W., et al. (2024b).

</span>
<span class="ltx_bibblock">Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling.

</span>
<span class="ltx_bibblock"><span id="bib.bib55.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2406.02069</span>.

</span>
</li>
<li id="bib.bib56" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">(56)</span>
<span class="ltx_bibblock">
Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., et al. (2024c).

</span>
<span class="ltx_bibblock">H2o: Heavy-hitter oracle for efficient generative inference of large language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib56.1.1" class="ltx_text ltx_font_italic">Advances in Neural Information Processing Systems</span>, 36.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Additional Experimental Details</h2>

<section id="A1.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.1 </span>Value Cache Pruning</h3>

<div id="A1.SS1.p1" class="ltx_para ltx_noindent">
<p id="A1.SS1.p1.4" class="ltx_p">Similar to the approach used for the Key cache, the pruning of channels in the Value cache can be guided by two primary criteria: magnitude-based pruning and query-driven pruning. We find that query-driven pruning is still better than magnitude based pruning.</p>
<table id="A1.E10" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.E10.m1.10" class="ltx_math_unparsed" alttext="\text{Score}_{v,i}(\textbf{Q}_{i},\textbf{K}_{i},\textbf{V}_{i})[j]=\|\text{softmax}(\frac{\textbf{Q}_{i}[-S_{\text{obs}}:]\textbf{K}_{i}^{T}}{\sqrt{D}})\textbf{V}_{i}[:,j]\|_{F}" display="block"><semantics id="A1.E10.m1.10a"><mrow id="A1.E10.m1.10.10"><mrow id="A1.E10.m1.9.9.3"><msub id="A1.E10.m1.9.9.3.5"><mtext id="A1.E10.m1.9.9.3.5.2">Score</mtext><mrow id="A1.E10.m1.2.2.2.4"><mi id="A1.E10.m1.1.1.1.1">v</mi><mo id="A1.E10.m1.2.2.2.4.1">,</mo><mi id="A1.E10.m1.2.2.2.2">i</mi></mrow></msub><mo lspace="0em" rspace="0em" id="A1.E10.m1.9.9.3.4">​</mo><mrow id="A1.E10.m1.9.9.3.3.3"><mo stretchy="false" id="A1.E10.m1.9.9.3.3.3.4">(</mo><msub id="A1.E10.m1.7.7.1.1.1.1"><mtext class="ltx_mathvariant_bold" id="A1.E10.m1.7.7.1.1.1.1.2">Q</mtext><mi id="A1.E10.m1.7.7.1.1.1.1.3">i</mi></msub><mo id="A1.E10.m1.9.9.3.3.3.5">,</mo><msub id="A1.E10.m1.8.8.2.2.2.2"><mtext class="ltx_mathvariant_bold" id="A1.E10.m1.8.8.2.2.2.2.2">K</mtext><mi id="A1.E10.m1.8.8.2.2.2.2.3">i</mi></msub><mo id="A1.E10.m1.9.9.3.3.3.6">,</mo><msub id="A1.E10.m1.9.9.3.3.3.3"><mtext class="ltx_mathvariant_bold" id="A1.E10.m1.9.9.3.3.3.3.2">V</mtext><mi id="A1.E10.m1.9.9.3.3.3.3.3">i</mi></msub><mo stretchy="false" id="A1.E10.m1.9.9.3.3.3.7">)</mo></mrow><mo lspace="0em" rspace="0em" id="A1.E10.m1.9.9.3.4a">​</mo><mrow id="A1.E10.m1.9.9.3.6.2"><mo stretchy="false" id="A1.E10.m1.9.9.3.6.2.1">[</mo><mi id="A1.E10.m1.3.3">j</mi><mo stretchy="false" id="A1.E10.m1.9.9.3.6.2.2">]</mo></mrow></mrow><mo id="A1.E10.m1.10.10.5">=</mo><msub id="A1.E10.m1.10.10.4"><mrow id="A1.E10.m1.10.10.4.1.1"><mo stretchy="false" id="A1.E10.m1.10.10.4.1.1.2">‖</mo><mrow id="A1.E10.m1.10.10.4.1.1.1"><mtext id="A1.E10.m1.10.10.4.1.1.1.2">softmax</mtext><mo lspace="0em" rspace="0em" id="A1.E10.m1.10.10.4.1.1.1.1">​</mo><mrow id="A1.E10.m1.10.10.4.1.1.1.3.2"><mo stretchy="false" id="A1.E10.m1.10.10.4.1.1.1.3.2.1">(</mo><mfrac id="A1.E10.m1.4.4"><mrow id="A1.E10.m1.4.4.2"><msub id="A1.E10.m1.4.4.2.1"><mtext class="ltx_mathvariant_bold" id="A1.E10.m1.4.4.2.1.2">Q</mtext><mi id="A1.E10.m1.4.4.2.1.3">i</mi></msub><mrow id="A1.E10.m1.4.4.2.2"><mo stretchy="false" id="A1.E10.m1.4.4.2.2.1">[</mo><mo lspace="0em" id="A1.E10.m1.4.4.2.2.2">−</mo><msub id="A1.E10.m1.4.4.2.2.3"><mi id="A1.E10.m1.4.4.2.2.3.2">S</mi><mtext id="A1.E10.m1.4.4.2.2.3.3">obs</mtext></msub><mo lspace="0.278em" rspace="0em" id="A1.E10.m1.4.4.2.2.4">:</mo><mo stretchy="false" id="A1.E10.m1.4.4.2.2.5">]</mo></mrow><msubsup id="A1.E10.m1.4.4.2.3"><mtext class="ltx_mathvariant_bold" id="A1.E10.m1.4.4.2.3.2.2">K</mtext><mi id="A1.E10.m1.4.4.2.3.2.3">i</mi><mi id="A1.E10.m1.4.4.2.3.3">T</mi></msubsup></mrow><msqrt id="A1.E10.m1.4.4.3"><mi id="A1.E10.m1.4.4.3.2">D</mi></msqrt></mfrac><mo stretchy="false" id="A1.E10.m1.10.10.4.1.1.1.3.2.2">)</mo></mrow><mo lspace="0em" rspace="0em" id="A1.E10.m1.10.10.4.1.1.1.1a">​</mo><msub id="A1.E10.m1.10.10.4.1.1.1.4"><mtext class="ltx_mathvariant_bold" id="A1.E10.m1.10.10.4.1.1.1.4.2">V</mtext><mi id="A1.E10.m1.10.10.4.1.1.1.4.3">i</mi></msub><mo lspace="0em" rspace="0em" id="A1.E10.m1.10.10.4.1.1.1.1b">​</mo><mrow id="A1.E10.m1.10.10.4.1.1.1.5.2"><mo stretchy="false" id="A1.E10.m1.10.10.4.1.1.1.5.2.1">[</mo><mo rspace="0em" id="A1.E10.m1.5.5">:</mo><mo id="A1.E10.m1.10.10.4.1.1.1.5.2.2">,</mo><mi id="A1.E10.m1.6.6">j</mi><mo stretchy="false" id="A1.E10.m1.10.10.4.1.1.1.5.2.3">]</mo></mrow></mrow><mo stretchy="false" id="A1.E10.m1.10.10.4.1.1.3">‖</mo></mrow><mi id="A1.E10.m1.10.10.4.3">F</mi></msub></mrow><annotation encoding="application/x-tex" id="A1.E10.m1.10b">\text{Score}_{v,i}(\textbf{Q}_{i},\textbf{K}_{i},\textbf{V}_{i})[j]=\|\text{softmax}(\frac{\textbf{Q}_{i}[-S_{\text{obs}}:]\textbf{K}_{i}^{T}}{\sqrt{D}})\textbf{V}_{i}[:,j]\|_{F}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(10)</span></td>
</tr></tbody>
</table>
<table id="A1.E11" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.E11.m1.4" class="ltx_Math" alttext="I_{i}=\textbf{Top}_{T}(\text{Score}_{v,i},T)" display="block"><semantics id="A1.E11.m1.4a"><mrow id="A1.E11.m1.4.4" xref="A1.E11.m1.4.4.cmml"><msub id="A1.E11.m1.4.4.3" xref="A1.E11.m1.4.4.3.cmml"><mi id="A1.E11.m1.4.4.3.2" xref="A1.E11.m1.4.4.3.2.cmml">I</mi><mi id="A1.E11.m1.4.4.3.3" xref="A1.E11.m1.4.4.3.3.cmml">i</mi></msub><mo id="A1.E11.m1.4.4.2" xref="A1.E11.m1.4.4.2.cmml">=</mo><mrow id="A1.E11.m1.4.4.1" xref="A1.E11.m1.4.4.1.cmml"><msub id="A1.E11.m1.4.4.1.3" xref="A1.E11.m1.4.4.1.3.cmml"><mtext class="ltx_mathvariant_bold" id="A1.E11.m1.4.4.1.3.2" xref="A1.E11.m1.4.4.1.3.2a.cmml">Top</mtext><mi id="A1.E11.m1.4.4.1.3.3" xref="A1.E11.m1.4.4.1.3.3.cmml">T</mi></msub><mo lspace="0em" rspace="0em" id="A1.E11.m1.4.4.1.2" xref="A1.E11.m1.4.4.1.2.cmml">​</mo><mrow id="A1.E11.m1.4.4.1.1.1" xref="A1.E11.m1.4.4.1.1.2.cmml"><mo stretchy="false" id="A1.E11.m1.4.4.1.1.1.2" xref="A1.E11.m1.4.4.1.1.2.cmml">(</mo><msub id="A1.E11.m1.4.4.1.1.1.1" xref="A1.E11.m1.4.4.1.1.1.1.cmml"><mtext id="A1.E11.m1.4.4.1.1.1.1.2" xref="A1.E11.m1.4.4.1.1.1.1.2a.cmml">Score</mtext><mrow id="A1.E11.m1.2.2.2.4" xref="A1.E11.m1.2.2.2.3.cmml"><mi id="A1.E11.m1.1.1.1.1" xref="A1.E11.m1.1.1.1.1.cmml">v</mi><mo id="A1.E11.m1.2.2.2.4.1" xref="A1.E11.m1.2.2.2.3.cmml">,</mo><mi id="A1.E11.m1.2.2.2.2" xref="A1.E11.m1.2.2.2.2.cmml">i</mi></mrow></msub><mo id="A1.E11.m1.4.4.1.1.1.3" xref="A1.E11.m1.4.4.1.1.2.cmml">,</mo><mi id="A1.E11.m1.3.3" xref="A1.E11.m1.3.3.cmml">T</mi><mo stretchy="false" id="A1.E11.m1.4.4.1.1.1.4" xref="A1.E11.m1.4.4.1.1.2.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E11.m1.4b"><apply id="A1.E11.m1.4.4.cmml" xref="A1.E11.m1.4.4"><eq id="A1.E11.m1.4.4.2.cmml" xref="A1.E11.m1.4.4.2"></eq><apply id="A1.E11.m1.4.4.3.cmml" xref="A1.E11.m1.4.4.3"><csymbol cd="ambiguous" id="A1.E11.m1.4.4.3.1.cmml" xref="A1.E11.m1.4.4.3">subscript</csymbol><ci id="A1.E11.m1.4.4.3.2.cmml" xref="A1.E11.m1.4.4.3.2">𝐼</ci><ci id="A1.E11.m1.4.4.3.3.cmml" xref="A1.E11.m1.4.4.3.3">𝑖</ci></apply><apply id="A1.E11.m1.4.4.1.cmml" xref="A1.E11.m1.4.4.1"><times id="A1.E11.m1.4.4.1.2.cmml" xref="A1.E11.m1.4.4.1.2"></times><apply id="A1.E11.m1.4.4.1.3.cmml" xref="A1.E11.m1.4.4.1.3"><csymbol cd="ambiguous" id="A1.E11.m1.4.4.1.3.1.cmml" xref="A1.E11.m1.4.4.1.3">subscript</csymbol><ci id="A1.E11.m1.4.4.1.3.2a.cmml" xref="A1.E11.m1.4.4.1.3.2"><mtext class="ltx_mathvariant_bold" id="A1.E11.m1.4.4.1.3.2.cmml" xref="A1.E11.m1.4.4.1.3.2">Top</mtext></ci><ci id="A1.E11.m1.4.4.1.3.3.cmml" xref="A1.E11.m1.4.4.1.3.3">𝑇</ci></apply><interval closure="open" id="A1.E11.m1.4.4.1.1.2.cmml" xref="A1.E11.m1.4.4.1.1.1"><apply id="A1.E11.m1.4.4.1.1.1.1.cmml" xref="A1.E11.m1.4.4.1.1.1.1"><csymbol cd="ambiguous" id="A1.E11.m1.4.4.1.1.1.1.1.cmml" xref="A1.E11.m1.4.4.1.1.1.1">subscript</csymbol><ci id="A1.E11.m1.4.4.1.1.1.1.2a.cmml" xref="A1.E11.m1.4.4.1.1.1.1.2"><mtext id="A1.E11.m1.4.4.1.1.1.1.2.cmml" xref="A1.E11.m1.4.4.1.1.1.1.2">Score</mtext></ci><list id="A1.E11.m1.2.2.2.3.cmml" xref="A1.E11.m1.2.2.2.4"><ci id="A1.E11.m1.1.1.1.1.cmml" xref="A1.E11.m1.1.1.1.1">𝑣</ci><ci id="A1.E11.m1.2.2.2.2.cmml" xref="A1.E11.m1.2.2.2.2">𝑖</ci></list></apply><ci id="A1.E11.m1.3.3.cmml" xref="A1.E11.m1.3.3">𝑇</ci></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E11.m1.4c">I_{i}=\textbf{Top}_{T}(\text{Score}_{v,i},T)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(11)</span></td>
</tr></tbody>
</table>
<p id="A1.SS1.p1.3" class="ltx_p">where <math id="A1.SS1.p1.1.m1.3" class="ltx_Math" alttext="\textbf{Q}_{i},\textbf{K}_{i},\textbf{V}_{i}\in\mathbb{R}^{S\times D}" display="inline"><semantics id="A1.SS1.p1.1.m1.3a"><mrow id="A1.SS1.p1.1.m1.3.3" xref="A1.SS1.p1.1.m1.3.3.cmml"><mrow id="A1.SS1.p1.1.m1.3.3.3.3" xref="A1.SS1.p1.1.m1.3.3.3.4.cmml"><msub id="A1.SS1.p1.1.m1.1.1.1.1.1" xref="A1.SS1.p1.1.m1.1.1.1.1.1.cmml"><mtext class="ltx_mathvariant_bold" id="A1.SS1.p1.1.m1.1.1.1.1.1.2" xref="A1.SS1.p1.1.m1.1.1.1.1.1.2a.cmml">Q</mtext><mi id="A1.SS1.p1.1.m1.1.1.1.1.1.3" xref="A1.SS1.p1.1.m1.1.1.1.1.1.3.cmml">i</mi></msub><mo id="A1.SS1.p1.1.m1.3.3.3.3.4" xref="A1.SS1.p1.1.m1.3.3.3.4.cmml">,</mo><msub id="A1.SS1.p1.1.m1.2.2.2.2.2" xref="A1.SS1.p1.1.m1.2.2.2.2.2.cmml"><mtext class="ltx_mathvariant_bold" id="A1.SS1.p1.1.m1.2.2.2.2.2.2" xref="A1.SS1.p1.1.m1.2.2.2.2.2.2a.cmml">K</mtext><mi id="A1.SS1.p1.1.m1.2.2.2.2.2.3" xref="A1.SS1.p1.1.m1.2.2.2.2.2.3.cmml">i</mi></msub><mo id="A1.SS1.p1.1.m1.3.3.3.3.5" xref="A1.SS1.p1.1.m1.3.3.3.4.cmml">,</mo><msub id="A1.SS1.p1.1.m1.3.3.3.3.3" xref="A1.SS1.p1.1.m1.3.3.3.3.3.cmml"><mtext class="ltx_mathvariant_bold" id="A1.SS1.p1.1.m1.3.3.3.3.3.2" xref="A1.SS1.p1.1.m1.3.3.3.3.3.2a.cmml">V</mtext><mi id="A1.SS1.p1.1.m1.3.3.3.3.3.3" xref="A1.SS1.p1.1.m1.3.3.3.3.3.3.cmml">i</mi></msub></mrow><mo id="A1.SS1.p1.1.m1.3.3.4" xref="A1.SS1.p1.1.m1.3.3.4.cmml">∈</mo><msup id="A1.SS1.p1.1.m1.3.3.5" xref="A1.SS1.p1.1.m1.3.3.5.cmml"><mi id="A1.SS1.p1.1.m1.3.3.5.2" xref="A1.SS1.p1.1.m1.3.3.5.2.cmml">ℝ</mi><mrow id="A1.SS1.p1.1.m1.3.3.5.3" xref="A1.SS1.p1.1.m1.3.3.5.3.cmml"><mi id="A1.SS1.p1.1.m1.3.3.5.3.2" xref="A1.SS1.p1.1.m1.3.3.5.3.2.cmml">S</mi><mo lspace="0.222em" rspace="0.222em" id="A1.SS1.p1.1.m1.3.3.5.3.1" xref="A1.SS1.p1.1.m1.3.3.5.3.1.cmml">×</mo><mi id="A1.SS1.p1.1.m1.3.3.5.3.3" xref="A1.SS1.p1.1.m1.3.3.5.3.3.cmml">D</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A1.SS1.p1.1.m1.3b"><apply id="A1.SS1.p1.1.m1.3.3.cmml" xref="A1.SS1.p1.1.m1.3.3"><in id="A1.SS1.p1.1.m1.3.3.4.cmml" xref="A1.SS1.p1.1.m1.3.3.4"></in><list id="A1.SS1.p1.1.m1.3.3.3.4.cmml" xref="A1.SS1.p1.1.m1.3.3.3.3"><apply id="A1.SS1.p1.1.m1.1.1.1.1.1.cmml" xref="A1.SS1.p1.1.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.SS1.p1.1.m1.1.1.1.1.1.1.cmml" xref="A1.SS1.p1.1.m1.1.1.1.1.1">subscript</csymbol><ci id="A1.SS1.p1.1.m1.1.1.1.1.1.2a.cmml" xref="A1.SS1.p1.1.m1.1.1.1.1.1.2"><mtext class="ltx_mathvariant_bold" id="A1.SS1.p1.1.m1.1.1.1.1.1.2.cmml" xref="A1.SS1.p1.1.m1.1.1.1.1.1.2">Q</mtext></ci><ci id="A1.SS1.p1.1.m1.1.1.1.1.1.3.cmml" xref="A1.SS1.p1.1.m1.1.1.1.1.1.3">𝑖</ci></apply><apply id="A1.SS1.p1.1.m1.2.2.2.2.2.cmml" xref="A1.SS1.p1.1.m1.2.2.2.2.2"><csymbol cd="ambiguous" id="A1.SS1.p1.1.m1.2.2.2.2.2.1.cmml" xref="A1.SS1.p1.1.m1.2.2.2.2.2">subscript</csymbol><ci id="A1.SS1.p1.1.m1.2.2.2.2.2.2a.cmml" xref="A1.SS1.p1.1.m1.2.2.2.2.2.2"><mtext class="ltx_mathvariant_bold" id="A1.SS1.p1.1.m1.2.2.2.2.2.2.cmml" xref="A1.SS1.p1.1.m1.2.2.2.2.2.2">K</mtext></ci><ci id="A1.SS1.p1.1.m1.2.2.2.2.2.3.cmml" xref="A1.SS1.p1.1.m1.2.2.2.2.2.3">𝑖</ci></apply><apply id="A1.SS1.p1.1.m1.3.3.3.3.3.cmml" xref="A1.SS1.p1.1.m1.3.3.3.3.3"><csymbol cd="ambiguous" id="A1.SS1.p1.1.m1.3.3.3.3.3.1.cmml" xref="A1.SS1.p1.1.m1.3.3.3.3.3">subscript</csymbol><ci id="A1.SS1.p1.1.m1.3.3.3.3.3.2a.cmml" xref="A1.SS1.p1.1.m1.3.3.3.3.3.2"><mtext class="ltx_mathvariant_bold" id="A1.SS1.p1.1.m1.3.3.3.3.3.2.cmml" xref="A1.SS1.p1.1.m1.3.3.3.3.3.2">V</mtext></ci><ci id="A1.SS1.p1.1.m1.3.3.3.3.3.3.cmml" xref="A1.SS1.p1.1.m1.3.3.3.3.3.3">𝑖</ci></apply></list><apply id="A1.SS1.p1.1.m1.3.3.5.cmml" xref="A1.SS1.p1.1.m1.3.3.5"><csymbol cd="ambiguous" id="A1.SS1.p1.1.m1.3.3.5.1.cmml" xref="A1.SS1.p1.1.m1.3.3.5">superscript</csymbol><ci id="A1.SS1.p1.1.m1.3.3.5.2.cmml" xref="A1.SS1.p1.1.m1.3.3.5.2">ℝ</ci><apply id="A1.SS1.p1.1.m1.3.3.5.3.cmml" xref="A1.SS1.p1.1.m1.3.3.5.3"><times id="A1.SS1.p1.1.m1.3.3.5.3.1.cmml" xref="A1.SS1.p1.1.m1.3.3.5.3.1"></times><ci id="A1.SS1.p1.1.m1.3.3.5.3.2.cmml" xref="A1.SS1.p1.1.m1.3.3.5.3.2">𝑆</ci><ci id="A1.SS1.p1.1.m1.3.3.5.3.3.cmml" xref="A1.SS1.p1.1.m1.3.3.5.3.3">𝐷</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p1.1.m1.3c">\textbf{Q}_{i},\textbf{K}_{i},\textbf{V}_{i}\in\mathbb{R}^{S\times D}</annotation></semantics></math>. We define a criterion <math id="A1.SS1.p1.2.m2.2" class="ltx_Math" alttext="\text{Score}_{v,i}" display="inline"><semantics id="A1.SS1.p1.2.m2.2a"><msub id="A1.SS1.p1.2.m2.2.3" xref="A1.SS1.p1.2.m2.2.3.cmml"><mtext id="A1.SS1.p1.2.m2.2.3.2" xref="A1.SS1.p1.2.m2.2.3.2a.cmml">Score</mtext><mrow id="A1.SS1.p1.2.m2.2.2.2.4" xref="A1.SS1.p1.2.m2.2.2.2.3.cmml"><mi id="A1.SS1.p1.2.m2.1.1.1.1" xref="A1.SS1.p1.2.m2.1.1.1.1.cmml">v</mi><mo id="A1.SS1.p1.2.m2.2.2.2.4.1" xref="A1.SS1.p1.2.m2.2.2.2.3.cmml">,</mo><mi id="A1.SS1.p1.2.m2.2.2.2.2" xref="A1.SS1.p1.2.m2.2.2.2.2.cmml">i</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A1.SS1.p1.2.m2.2b"><apply id="A1.SS1.p1.2.m2.2.3.cmml" xref="A1.SS1.p1.2.m2.2.3"><csymbol cd="ambiguous" id="A1.SS1.p1.2.m2.2.3.1.cmml" xref="A1.SS1.p1.2.m2.2.3">subscript</csymbol><ci id="A1.SS1.p1.2.m2.2.3.2a.cmml" xref="A1.SS1.p1.2.m2.2.3.2"><mtext id="A1.SS1.p1.2.m2.2.3.2.cmml" xref="A1.SS1.p1.2.m2.2.3.2">Score</mtext></ci><list id="A1.SS1.p1.2.m2.2.2.2.3.cmml" xref="A1.SS1.p1.2.m2.2.2.2.4"><ci id="A1.SS1.p1.2.m2.1.1.1.1.cmml" xref="A1.SS1.p1.2.m2.1.1.1.1">𝑣</ci><ci id="A1.SS1.p1.2.m2.2.2.2.2.cmml" xref="A1.SS1.p1.2.m2.2.2.2.2">𝑖</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p1.2.m2.2c">\text{Score}_{v,i}</annotation></semantics></math> to indicate the importance of each channel in the head <math id="A1.SS1.p1.3.m3.1" class="ltx_Math" alttext="i" display="inline"><semantics id="A1.SS1.p1.3.m3.1a"><mi id="A1.SS1.p1.3.m3.1.1" xref="A1.SS1.p1.3.m3.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="A1.SS1.p1.3.m3.1b"><ci id="A1.SS1.p1.3.m3.1.1.cmml" xref="A1.SS1.p1.3.m3.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p1.3.m3.1c">i</annotation></semantics></math> of value cache. Then, only top T channels are retained.</p>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2407.21017" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2407.21018" class="ar5iv-text-button ar5iv-severity-error">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2407.21018">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2407.21018" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2407.21019" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Mon Aug  5 17:29:37 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2407.21018
Large Language Models (LLMs)  have revolutionized the field of natural language processing, achieving unprecedented performance across a variety of applications by leveraging increased model sizes and sequence lengths.
However, the associated rise in computational and memory costs poses significant challenges, particularly in managing long sequences due to the quadratic complexity of the transformer attention mechanism.
This paper focuses on the long-context scenario, addressing the inefficiencies in KV cache memory consumption during inference.
Unlike existing approaches that optimize the memory based on the sequence lengths, we uncover that the channel dimension of the KV cache exhibits significant redundancy, characterized by unbalanced magnitude distribution and low-rank structure in attention weights.
Based on these observations, we propose ThinK, a novel query-dependent KV cache pruning method designed to minimize attention weight loss while selectively pruning the least significant channels. Our approach not only maintains or enhances model accuracy but also achieves a reduction in memory costs by over 20% compared with vanilla KV cache eviction methods. Extensive evaluations on the LLaMA3 and Mistral models across various long-sequence datasets confirm the efficacy of ThinK, setting a new precedent for efficient LLM deployment without compromising performance. We also outline the potential of extending our method to value cache pruning, demonstrating ThinK’s versatility and broad applicability in reducing both memory and computational overheads.
abstract
ThinK:ThinnerKey Cache by Query-Driven Pruning
Abstract
2407.21018
1IntroductionLarge language models (LLMs) have emerged as a dominant paradigm in natural language processing, achieving state-of-the-art performance across various tasks. A key principle, the Scaling Law, suggests that LLMs exhibit emergent abilities as model size increases, enhancing their capacity to understand context and handle long sequences.
This capacity growth allows LLMs to generate coherent and contextually accurate responses and
enables various downstream applications, such as document summarization, code generation, and conversational AI, .Despite their success in various applications, the generation of LLMs incurs significant expenses, which escalate with increasing model size and sequence length. Notably, both the trainingand inferencestages involve frequent generation by LLMs, further contributing to these costs. Consequently, efficient LLMs have gained popularity in recent years.
To address these challenges, quantizationand pruning methodsare employed to reduce model size. Additionally, managing long sequences presents another cost due to the transformer attention mechanism. The quadratic complexity of the attention mechanism results in substantial computational burden when dealing with long sequences, scaling poorly with sequence length. Therefore, effective management of long sequences is essential for the practical deployment of LLMs. In this paper, we focus on the long-context scenario and aim to reduce the memory consumption associated with lengthy sequences.Specifically, the number of KV cache parameters is the product of batch size, sequence length, number of layers, number of heads, channel size of each head, i.e.,, which need to be stored in the GPU memory during inference.
To reduce memory and computational costs during inference, efficiency can only be achieved by pruning the dimensions acrossor performing quantization over the caches. It is well-acknowledged that token importance tends to be sparse. Consequently, KV eviction algorithms have been proposed to minimize the KV cache memory cost from dimension. Additionally, inter-layer redundancy has been exploredto address the layer dimension. Despite these advances, existing methods have largely overlooked the channel dimension. In this paper, we highlight that the magnitude across Key cache channel dimensions is significantly unbalanced. Additionally, we observe that the attention weights exhibit a low-rank structure. Based on these findings, we hypothesize that the channel dimension of the key cache exhibits redundancy. Consequently, we explore the redundancy of KV cache tensors specifically from dimension, aiming to devise effective strategies that reduce costs without compromising performance.In our paper, we propose a simple yet effective method –ThinK, for KV cache pruning. To pinpoint the least significant channels, we formulate the task as an optimization problem, aiming to minimize the loss in attention weights attributable to pruning. To effectively address this problem, we establish a novel query-dependent criterion that assesses the importance of each channel. Using this criterion, we then select the most critical channels in a greedy fashion. We evaluateThinKusing the LLaMA3and Mistralmodels and validate its effectiveness across various long-sequence datasets. The results indicate that when paired with token eviction methods, ThinK not only achieves comparable or superior accuracy but also reduces KV cache memory costs by more than 20%.Contributions.This work pioneers the investigation into the sparsity structure of the channels in the KV cache. Specifically, we discovered that the activated key cache is sparse given a specific query. This insight allows us to prune the key cache channels using a query-induced norm. Inspired by this phenomenon, we introduce a plug-and-play method ThinK, the first channel pruning method specifically designed for KV cache. ThinK reduces the dimensionality of the cache channel, leading to linear savings in both memory and computational requirements. As plug-and-play techniques, our proposed ThinK is orthogonal to other KV cache compression schemes (e.g.KV cache eviction, quantization) . Our extensive experiments demonstrate the remarkable efficiency of ThinK on LLaMA-3 and Mistral models. Furthermore, we explore the promising potential of extending ThinK to value cache pruning (ThinKV)  in specific scenarios, highlighting the broad applicability of our method.Figure 1:An illustration of the pruning procedure of ThinK. Within each head, scores are calculated for each channel, and only the top T channels out of D are selected for retention. A binary channel mask, along with the pruned keys, is subsequently stored in the cache memory.
introduction
ThinK:ThinnerKey Cache by Query-Driven Pruning
1Introduction
2407.21018
2ObservationsWe identify several key observations that motivate our approach to prune the channel of KV-Cache.
Specifically, we visualize the magnitude of KV-cache and perform singular value decomposition of the attention in LLaMA3-8B model.
We show the visualization for certain heads and layers for illustration purposes.Magnitude of KV-Cache ChannelFigure2visualizes the absolute values of key and value cache across the tokens in each channel111We use the visualization code fromhttps://github.com/jy-yuan/KIVI/tree/main/vis..
Consistent with previous findings, we observe that only certain channels have significant magnitudes in the key cache, whereas the value cache lacks obvious patterns.
For instance, the absolute value aroundchannel of key cache has much larger magnitudes than other channels for all tokens in layer(Figure2(a) ) .
The same observation applies toandchannels of the first head in layer(Figure2(c) ) .
Given such an observation,proposed to perform quantization over the channels of the key cache based on such observations.
In addition to channel quantization, the observation also suggests that it is feasible to prune away certain key cache channels that contribute less to the attention mechanism.
Moreover, channel quantization and pruning techniques are orthogonal such that they can be utilized concurrently to enhance model efficiency.(a)  Key Cache (layer) (b)  Value Cache (layer) (c)  Key Cache (layer) (d)  Value Cache (layer) Figure 2:Magnitude of key and value cache for LLaMA3-8B. The first head of layer 14 and layer 20 of LLaMA3-8B is selected to visualize the magnitude of the key and value caches. We observe that the magnitudes of the key cache channels vary differently, whereas the channels of the value cache do not exhibit such variation.Singular Value AnalysisWe conducted singular value decomposition (SVD) on the attention of the specified layers to investigate their principal components.
The singular values derived from SVD capture the effective rank of the attention matrix, indicating how the information is distributed across different components.(1) Figure3(a)  illustrates the energy distribution of the singular values, plotted in log scale to enhance the visibility of differences among them.
Notably, only a few singular values exhibit high energy levels exceedingacross all heads and layers, highlighting their relative significance.
This observation aligns with previous findings, where a small subset of singular values often captures most of the information in attention mechanisms.
In addition, the rapid decay of the energy suggests that a low-rank approximation can effectively capture the essential information in the key cache.Figure3(b) , the normalized cumulative energy sum reveals that the topsingular values account for over% of the total energy.
These findings suggest that the attention matrix is inherently low-rank, indicating that the key cache can be approximated using low-dimensional vectors.(a)  Energy of singular values(b)  Cumulative energy of singular valuesFigure 3:The energy and cumulative energy of the singular values.
body
ThinK:ThinnerKey Cache by Query-Driven Pruning
2Observations
2407.21018
3ThinKNotations.We use uppercase letters (e.g.,)  to denote scalar values and boldface uppercase letters (e.g.,)  to denote matrices and tensors. The notationdenotes the-norm for vectors. Unless otherwise specified,denotes the-norm. The Frobenius norm is denoted by. The floor function is denoted by, and the ceiling function is denoted by.3.1Preliminary Study of KV Cache OptimizationIn scenarios with extended contexts or batch processing, the main limitations in terms of memory and speed are due to the handling of the KV cache size. Considering a batch of requests to a Large Language Model (LLM)  service that provides a long input prompt consisting of tokens, the total KV cache size can be computed as follows:(2) whereis the number of layers,is the number of heads,is the head dimension. The KV cach size grows linearly as the batch sizeand sequence length. For
a model with multihead attention (MHA) , such as LLaMA2-7B, a context length ofand a batch size ofrequire storing aGB KV cache, which is equivalent to the size of the model parameters.
The KV cache must be transferred from off-chip memory (HBM) to on-chip memory (cache)  for each token generated, leading to a memory bottleneck.
This substantial memory demand highlights the challenges in managing large-scale models and the need for efficient memory utilization strategies. Current methods optimize the KV cache based on the sequence lengthand precision.
We will introduce a new method,ThinK, to optimize it from the perspective of the number of head dimensions.Table 1:Performance comparison of pruning K cache bynorm on LongBench.MethodSingle-Document QAMulti-Document QASummarizationFew-shot LearningSyntheticCodeAvg.NrtvQAQasperMF-enHotpotQA2WikiMQAMusiqueGovReportQMSumMultiNewsTRECTriviaQASAMSumPCountPReLccRB-PH2O023.5217.9334.6842.1133.5219.9222.1122.5623.8241.0090.4640.205.8769.5056.7151.6937.23+3023.3817.1534.9940.5631.4919.9021.3722.1323.4440.5090.1040.655.4169.0058.6454.9937.11+4023.5115.4034.3740.7131.2820.2421.2522.2922.5438.5089.2239.275.8768.3358.4754.3336.60+3023.9817.0435.1939.2731.2920.421.6222.4623.3440.589.7540.715.5468.6760.1258.5237.40+4023.7616.2332.1940.2332.1320.6921.322.2523.239.589.6140.245.6669.060.0959.4537.22SnapKV024.8423.9638.7742.7534.5520.8722.2622.6123.9770.0090.5240.295.8169.5059.0451.8140.10+3024.4324.6340.1141.8333.4721.2221.4722.4123.7366.5090.3940.205.7068.1061.0455.3740.04+4024.5824.8739.3042.7631.9520.4720.9522.2223.4255.5090.2239.135.8268.3960.7156.1039.15+3024.4724.7338.1641.8632.2320.2321.5922.4523.7767.590.3340.315.768.4262.6560.0740.28+4024.5223.7538.3542.4232.9620.3921.2122.2823.4160.0090.2039.595.7568.2961.9660.5939.74Magnitude base Pruning:Based on the observations in Figure2which depicts the significant variation in the magnitudes across different channels, one straightforward criterion is to use the norm of the magnitude to measure the importance of different channels in key cache.(3) Given pruning ratio, We only keepmost important channels among thechannels of each head:(4) whereis thenorm of each channel.andare indicators of heads and channels in key cache.stores the indicators of the topvalues in tensorper head.In Table1, we present the results of key cache pruning with various pruning ratios applied to the LLaMA3-8B model. We utilize theandnorms as criteria for evaluation, and validate performance using the LongBench benchmark. Compared to the baseline methods, H2Oand SnapKV, both with a KV length of 512, we further prune the channels of the key cache. Apruning ratio can maintain accuracy; however, increasing it toresults in significant performance degradation, especially fornorm based pruning. The results of magnitude-based pruning support our assumption that the key cache is redundant in the channel dimension. These results also indicate the need for a better pruning matrix to achieve higher pruning ratios effectively.3.2Query-Driven PruningFor each head, the attention scores are computed using the queries and keys, and then applied to the values. The formula for the attention for headis:(5) where. When pruning one channels of, the corresponding channels inwill also be removed. We aim to find the optimal subset of channels to prune, denoted by the selection matrix, whereSis a diagonal matrix with binary entries (for keeping a channel,for pruning it) . To better maintain the performance after pruning the channels, we minimize the Frobenius norm of the difference between the original and pruned attention weights:(6) Given a pruning ratio, it can further expanded as:subject to(7) For simplicity, we use greedy algorithm to optimize. To achieve the pruning goal, we define a criterion for evaluating the importance of each channel and greedily select the channels with largest scores:(8) (9) Here’s a detailed explanation of why it optimizes the selection matrix. Themeasures the magnitude of the interaction between the query and key vectors for channelin each head. By selecting channels with the highest interaction magnitudes, we aim to retain the most significant contributions to the attention mechanism. This criterion ensures that the selected channels preserve the primary information flow in the attention computation, thereby minimizing the loss of important information.Observation Window.Based on the observations in SnapKVthat the last window of input sequence recognizes highly similar attention allocation
pattern with the actual generation. To reduce the computation cost, we only use the lastwindow to calculate the score:.Figure 4:Implementation during decoding.3.3ImplementationsFollowing SnapKV, we focus on the long context input scenario. We opt not to prune the most recent tokens and newly generated keys. Consequently, our key-value (KV)  cache will store two distinct categories of keys: one subset consists of pruned keys with a reduced channel size, while the other retains keys at their original size. Additionally, we maintain a binary mask to indicate which channels have been pruned. Note that the memory overhead associated with this mask is negligible. Figure4illustrates one implementation of our method during the decoding stage. The pruned keys are first zero-filled using the mask to restore them to their original size, and then concatenated with the unpruned keys. This implementation can be effectively integrated with optimization techniques such as FlashAttention. The second implementation involves initially pruning the Query using the mask. The pruned Query is then multiplied by the pruned Key, while the unpruned Query is applied to the unpruned Key. Subsequently, the two outputs are concatenated. In theory, this implementation has lower computational costs.
body
ThinK:ThinnerKey Cache by Query-Driven Pruning
3ThinK
2407.21018
4Experiment ResultsIn this section, we conduct extensive experiments to evaluate the effectiveness ofThinKon performance and memory reduction. First, we introduce benchmark datasets, baseline methods, and implementation details in Section4.1. Next, we report the performance of our KV pruning method with two different KV compression methods in long-context scenarios in Section4.2. Finally, we present the results of ablation experiments in Section4.4.4.1SettingsBenchmark Datasets.We perform evaluations of our method with state-of-the-art KV cache compression methods on two widely used benchmarks: LongBench and Needle-in-a-Haystack.LongBenchis designed to comprehensively evaluate LLM’s long context understanding capabilities. It includes 17 datasets covering six different tasks: single-document QA, multi-document QA, summarization, few-shot learning, synthetic tasks, and code completion. The average input length of LongBench is 6,711 words, which necessitates reducing the KV cache to lower memory usage for inference.Needle-in-a-Haystackis a recent popular test challenge that requires models to accurately identify a small piece of information (“needle”)  in a long document (“haystack”) , where the needle is placed at a random position.
This challenge can test if KV cache compression methods still retain the small piece of critical information.Baseline Approaches.The baseline methods include Heavy Hitter Oracle (H2O)  and SnapKV, both of which are the state-of-the-art KV cache compression methods but use different strategies for selecting KV cache.H2Ois designed to reduce memory usage by dynamically balancing recent tokens and Heavy Hitter (H2)  tokens, where H2 tokens are a small set of tokens that contribute most of the value when computing attention scores.SnapKVautomatically compresses KV caches by selecting clustered important KV positions for each attention head. This is inspired by the fact that each attention head has a specific pattern, focusing on certain attention features during generation.Table 2:Performance comparison of Key cache pruning on LLaMA3-8B by ThinK on LongBench.MethodSingle-Document QAMulti-Document QASummarizationFew-shot LearningSyntheticCodeAvg.NrtvQAQasperMF-enHotpotQA2WikiMQAMusiqueGovReportQMSumMultiNewsTRECTriviaQASAMSumPCountPReLccRB-PKV-size FullALL KV25.5632.2739.7143.5635.0921.1828.7123.2626.6473.5090.4842.334.8069.2559.2954.0541.86KV-size 128H2O22.1213.2031.6137.7932.7118.4520.3222.0221.1038.5087.7539.145.8369.5055.0650.9735.38+ThinK(40) 22.8514.5529.4938.6330.8418.9020.1221.9620.6838.5086.3838.405.5069.1757.9356.1235.63+ThinK(50) 23.4714.0628.6738.3530.2117.8719.6921.9419.9538.5087.1438.074.9269.5057.9956.6635.44\cdashline1-18
SnapKV21.1913.5532.6438.7529.6418.7318.9821.6220.2645.0088.3637.645.1368.8555.8451.8235.50+ThinK(40) 22.1114.6732.4936.2528.6318.8018.9321.4920.1444.5088.1138.325.7569.1758.2155.8935.84+ThinK(50) 21.7914.7332.0337.5227.8618.2818.5021.5219.7143.5086.0038.355.5969.5057.9656.9635.61KV-size 512H2O23.5217.9334.6842.1133.5219.9222.1122.5623.8241.0090.4640.205.8769.5056.7151.6937.23+ThinK(40) 23.7617.8033.8040.3930.7019.0921.8222.5123.7841.0090.1640.675.1569.2560.7757.5837.39+ThinK(50) 24.1716.9635.7639.4730.2918.6721.3922.5923.0641.0089.8140.355.2369.3360.2058.3437.29+ThinK(60) 23.4014.8332.6238.4730.9719.8120.8022.0421.6040.0088.7938.905.3669.5058.2857.6536.44\cdashline1-18
SnapKV24.8423.9638.7742.7534.5520.8722.2622.6123.9770.0090.5240.295.8169.5059.0451.8140.10+ThinK(40) 24.5825.4437.0341.8733.4520.5821.7722.4224.1670.0090.3940.296.0669.5062.0559.2340.55+ThinK(50) 24.8525.1037.0641.5832.3420.6021.6122.4423.6669.5090.3939.705.8469.7961.5759.4240.34+ThinK(60) 25.9822.7738.3740.4433.1919.9020.8422.2122.5559.0090.3238.126.3969.5059.1458.4039.20KV-size 1024H2O25.6222.1636.8141.0133.5319.4123.2822.6525.4146.5090.8241.785.7969.2559.6955.5038.70+ThinK(40) 25.5221.9337.1741.5631.2220.1722.8922.9525.2147.0090.7441.345.5769.5062.5858.6739.00+ThinK(50) 25.4122.1937.6440.9231.2718.6622.1722.2224.8446.5090.3440.595.2069.5061.7157.9938.57+ThinK(60) 24.0617.8037.8538.6329.9819.4021.4122.3223.4244.5090.1639.435.8469.5058.3158.7337.58\cdashline1-18
SnapKV24.6225.9937.6443.8434.9920.0024.2822.3925.6372.590.5640.415.3669.2560.5756.1140.88+ThinK(40) 24.8827.7238.6043.1632.4420.6724.2122.7925.5671.5090.4540.945.9369.5062.7759.4541.29+ThinK(50) 24.8227.2639.6642.8232.0919.5623.5222.4825.3471.5090.4340.745.2069.5062.4659.7541.07+ThinK(60) 24.4627.3538.2241.9631.6420.1821.8922.8323.6870.0090.1938.696.1069.5058.8759.2640.30KV-size 2048H2O25.5626.8539.5444.3032.9221.0924.6823.0126.1653.0090.6541.844.9169.2558.4351.3139.59+ThinK(40) 25.5626.3139.2042.9631.8120.5324.2323.3525.9053.5090.5641.035.5269.2562.1059.0040.05+ThinK(50) 25.0125.3738.8242.3231.2720.5023.7823.2126.0353.0090.3740.865.1369.5061.9158.9539.75+ThinK(60) 24.3722.1437.7740.1329.5020.2622.0922.7624.7849.5090.1639.695.5669.5059.2458.7838.51\cdashline1-18
SnapKV25.8629.5541.1044.9935.8021.8125.9823.4026.4673.5090.5641.665.1769.2558.6751.5241.58+ThinK(40) 25.4129.7939.2143.3533.9621.4925.7823.1126.2373.0090.5641.795.8169.5062.4559.1941.91+ThinK(50) 25.0030.2539.2743.2332.9321.2425.1623.0126.573.0090.3741.265.4569.5062.359.8441.77+ThinK(60) 24.8928.8840.4441.3029.9921.3423.4822.924.9972.5090.3638.55.7169.5059.7759.5040.88Implementation Details.In this paper, we use LLaMA3-8B-Instructand Mistral-7B-Instruct-v0.2as the backbone LLMs, both accessible via HuggingFace.
OurThinKaims to prune channels of the KV, which is agnostic to KV cache compression methods.
If there is no other statement, we prune the key cache by default. All the experiments are conducted using one Nvidia A100.
To fairly compare KV cache compression methods and KV cache compression integrated withThinK, we use the same hyperparameters for both.
For example, when comparing SnapKV and SnapKV integrated withThinK, we set the maximum pooling kernel size to 7 and the observation window size to 32, using the same KV-size for both.Table 3:Performance comparison of Key cache pruning on Mistral-7B by ThinK on LongBench.MethodSingle-Document QAMulti-Document QASummarizationFew-shot LearningSyntheticCodeAvg.NrtvQAQasperMF-enHotpotQA2WikiMQAMusiqueGovReportQMSumMultiNewsTRECTriviaQASAMSumPCountPReLccRB-PKV-size FullALL KV26.6332.9949.3442.7727.3518.7732.8724.2427.1071.0086.2342.962.7586.9856.9354.4942.71KV-size 128H2O21.2121.8138.8730.4220.3612.3020.5822.6122.1039.0082.3740.442.9079.5651.2248.3834.63+ThinK(40) 21.1721.9039.2929.9220.9912.3020.8422.9121.9239.0082.7040.352.9779.2151.1948.3234.69+ThinK(50) 21.6721.8039.4828.7420.6513.1420.5722.8321.7839.0082.5440.123.6178.3950.2748.434.56+ThinK(60) 21.0421.3039.5628.6821.2913.9720.1322.5221.8139.5082.0539.144.1674.2349.8347.6734.18\cdashline1-18
SnapKV19.1721.4042.9336.7622.4415.8619.1621.8421.5547.5084.1540.242.3068.2652.3148.8035.29+ThinK(40) 20.5221.0042.6537.5822.0915.2319.2922.0121.2247.0083.8539.643.2067.4551.4848.3135.16+ThinK(50) 20.6720.6043.3737.2721.5815.6619.0621.7921.0247.0083.3839.773.6567.0650.8048.3535.06+ThinK(60) 21.2520.8244.2036.2121.6816.4719.0521.9920.7345.0083.8138.794.1966.9049.9947.6134.92KV-size 512H2O21.8326.0044.6932.4623.0514.6923.5323.0624.5942.0085.2241.493.4086.2054.7851.0937.38+ThinK(40) 21.5826.1544.4932.7323.9915.0923.5623.2824.4542.0085.5842.583.1885.754.3951.1537.49+ThinK(50) 22.7625.7444.6131.7423.2513.9123.3123.1324.3441.0085.3941.852.8284.3654.6950.8837.11+ThinK(60) 22.9125.5744.0429.4822.8813.6723.3122.6424.1041.0085.3141.152.9882.3453.7050.2536.58\cdashline1-18
SnapKV24.4427.8148.9839.4625.2516.9823.7022.9624.3767.0085.8841.262.7886.5656.4653.4140.46+ThinK(40) 24.2728.4649.2638.1324.2216.9223.5923.7024.4667.5085.942.512.9285.3255.8953.3540.40+ThinK(50) 24.5629.2248.5937.7024.2717.3923.6823.6524.5867.5086.0542.013.0786.3056.4953.2940.52+ThinK(60) 24.0728.2749.1038.6524.3117.5223.1623.5124.2367.0086.3340.783.6983.7454.9452.2340.10KV-size 1024H2O23.6728.5546.4036.9924.8215.0225.2123.0425.7746.0085.9341.983.2486.5756.4052.7538.90+ThinK(40) 23.9728.9145.8435.7824.8814.5525.1123.3525.8345.5086.1142.443.2384.8256.2153.0238.72+ThinK(50) 23.8928.4046.6035.5724.2614.7824.9823.3125.6844.5086.1642.723.3883.2055.8852.6338.50+ThinK(60) 23.8727.7646.2535.2824.3814.7424.3523.3525.5044.5085.3841.373.3481.4255.2151.8938.04\cdashline1-18
SnapKV25.4729.5749.3340.9025.5319.0125.9423.8926.2169.5086.4842.102.9888.5657.1953.6041.64+ThinK(40) 25.2230.4848.5841.1125.2818.9925.9124.0026.1370.0086.6443.352.9886.356.7154.1941.62+ThinK(50) 25.6330.0849.4140.5925.1319.5825.4724.2325.9269.586.6742.312.7484.7857.4353.5941.44+ThinK(60) 24.6929.348.9040.4425.3319.5825.2323.625.2569.0086.8540.863.1983.7056.353.3040.97KV-size 2048H2O25.7631.1049.0640.3826.4316.7827.1723.6426.6955.0086.3542.482.7286.6456.9853.9140.69+ThinK(40) 25.4030.8048.4539.6426.0816.8227.1223.7926.6553.5086.3943.033.2986.3956.6153.6040.47+ThinK(50) 25.6831.2448.6939.6525.8416.7226.6923.5726.7852.0086.7442.854.0183.4657.1253.6740.29+ThinK(60) 25.8331.0048.2338.5825.7116.5426.5123.8126.2850.5086.5742.053.3682.4956.0452.6739.76\cdashline1-18
SnapKV25.8932.5648.5541.6827.2418.7528.9024.4726.6370.0086.2742.573.0986.9357.4453.8342.18+ThinK(40) 25.7732.6748.7041.0627.0719.1428.9124.3726.8870.0086.3742.753.6187.3857.2154.4442.27+ThinK(50) 26.4432.9449.0240.8626.8419.4928.4624.5126.7270.0086.5041.752.7884.7056.4754.1541.98+ThinK(60) 26.0032.5348.7340.9526.7718.9227.4023.9726.3770.0086.4541.123.3182.2456.0153.5341.524.2Results on LongBenchTables2and3present the results of KV compression methods and KV compression methods integrated with our channel pruning method of Key cache (ThinK)  over two different base LLMs across various KV-sizes on LongBench.
The following observations can be drawn:
(1)  Our method can further prune the channels of the key cache after compressing the KV cache with H2O and SnapKV. For the base model LLaMA3-8B, our method reduces memory usage and slightly improves performance for both H2O and SnapKV. For the base model Mistral-7B, our method reduces memory with only a slight drop in performance in some cases.
(2)  Comparing the results of SnapKV or H2O integrated withThinKin Table2to SnapKV or H2O integrated withornorm shown in Table1, our proposed query-based channel pruning shows better performance when the pruning ratio iscompared to no pruning. However,andnorm methods cannot outperform the KV compression methods without pruning.
(3)  When the pruning ratio is small, performance tends to be better than when the pruning ratio is large.
(4)  When the KV-size is increased fromto, the performance of our channel pruning method improves. Notably, with a KV cache size ofand a pruning ratio of, our method can even outperform the LLaMA3-8B with a full KV cache.
The above observations indicate that our method is agnostic to existing KV cache compression methods and can further improve their performance and memory reduction. Additionally, query-based pruning is more effective thanandnorm for channel pruning in LLMs.4.3Results on Needle-in-a-HaystackTable4presents the results of the Needle-in-a-Haystack test based on the SnapKVapproach with KV cache size of input prompt fromto.
With a modest pruning ratio%, the accuracy ofThinKis consistently better or comparable than the original SnapKV across both LLaMA3 and Mistral with different KV cache sizes.
Such comparisons demonstrate that the proposed query-driven channel pruning can effectively retain the informative channels and discard the noisy ones.
With a pruning ratio of%, we observe an accuracy drop with a small KV-cache size, particularly forandacross both LLaMA3 and Mistral.
However,ThinKis still able to obtain comparable performance with SnapKV when we have a larger KV cache size (i.e.,and) .
Intuitively, a larger pruning ratio with a smaller KV cache size could tend to lose more important information compared to a larger KV cache size.
In addition, the performance on larger KV cache size also suggestsThinKis robust for long context retrieval tasks.Figure6(a) -(d)  visualize the Needle-in-a-Haystack test accuracy against different token lengths and depth percent.
The KV cache size is configured toand, with pruning ratioequal to% and%, respectively.
First,ThinKcan maintain the original retrieval capability of SnapKV though the overall numerical accuracy is different (i.e.,vsandvs) .
BecauseThinKachieves the same accuracy at most of the entries with different token limits and depths as Snap KV.
In addition,ThinKalso successfully retrieves some “needles” where SnapKV fails, leading to higher overall accuracy.
The visualization results further demonstrate the robustness ofThinKin a fine-grained perspective andThinKcan enhance the original quantization approach.Table 4:Needles-in-a-Haystack Test ResultsModelMethodKV-size12851210242048LLaMA3-8BSnapKV0079.690.291.291.7SnapKV+ThinK4079.690.391.291.7SnapKV+ThinK5077.489.691.091.7Mistral-7BSnapKV0077.889.590.490.8SnapKV+ThinK4078.690.190.690.9SnapKV+ThinK5078.190.190.891.1SnapKV+ThinK6075.989.290.691.14.4Ablation StudiesTable 5:Performance comparison of pruning K cache using different recent-size.Recent-SizeSingle-Document QAMulti-Document QASummarizationFew-shot LearningSyntheticCodeAvg.NrtvQAQasperMF-enHotpotQA2WikiMQAMusiqueGovReportQMSumMultiNewsTRECTriviaQASAMSumPCountPReLccRB-PH2O +ThinK() 024.4027.5045.4235.1724.4513.0227.6523.8826.8653.5086.0641.733.0183.4255.1251.3238.913225.4030.8048.4539.6426.0816.8227.1223.7926.6553.5086.3943.033.2986.3956.6153.6040.4712825.6930.9348.3239.6326.0816.8227.1823.9226.6253.5086.3942.963.2986.3956.7753.6040.51SnapKV +ThinK() 024.9428.5845.7839.5925.4015.9229.5024.0526.7270.0085.6041.382.9784.0055.2752.3940.763225.7732.6748.7041.0627.0719.1428.9124.3726.8870.0086.3742.753.6187.3857.2154.4442.2712825.7532.4948.6141.0127.1819.1428.7924.6426.7770.0086.3742.773.6187.1357.1954.3942.24Table 6:ablations with the same memory consumption.MethodsMemory(M) Single-Document QAMulti-Document QASummarizationFew-shot LearningSyntheticCodeAvg.NrtvQAQasperMF-enHotpotQA2WikiMQAMusiqueGovReportQMSumMultiNewsTRECTriviaQASAMSumPCountPReLccRB-PH2OVanilla54.521.2920.6937.6628.6521.0814.0120.2022.1121.3338.5082.5539.873.6678.1450.3248.5434.29ThinK54.421.1721.9039.2929.9220.9912.3020.8422.9121.9239.0082.7040.352.9779.2151.1948.3234.69Vanilla208.022.1323.8343.2430.9223.3614.5622.9222.7724.2341.5085.0441.263.0286.0354.9150.5036.89ThinK208.021.5826.1544.4932.7323.9915.0923.5623.2824.4542.0085.5842.583.1885.754.3951.1537.49Vanilla413.022.9028.4546.1635.5723.8613.7424.9023.1925.7744.5085.5441.973.2285.8255.9652.3338.37ThinK412.823.9728.9145.8435.7824.8814.5525.1123.3525.8345.5086.1142.443.2384.8256.2153.0238.72Vanilla822.525.5130.2348.2339.7225.5616.7526.9823.8126.4750.5086.4342.092.7885.5757.453.4240.09ThinK822.425.4030.8048.4539.6426.0816.8227.1223.7926.6553.5086.3943.033.2986.3956.6153.6040.47SnapKVVanilla54.519.2519.9542.8035.8821.9614.5918.7621.7120.7046.0084.1239.432.5965.3651.3947.8134.52ThinK54.420.5221.0042.6537.5822.0915.2319.2922.0121.2247.0083.8539.643.2067.4551.4848.3135.16Vanilla208.023.3127.4548.8538.7723.9316.5023.4423.6324.1366.0086.0541.002.6287.0156.1352.6040.09ThinK208.024.2728.4649.2638.1324.2216.9223.5923.7024.4667.5085.9042.512.9285.3255.8953.3540.40Vanilla413.024.2429.5349.1340.4825.0518.7425.4623.6425.6068.0086.1441.423.0388.5557.0853.8641.25ThinK412.825.2230.4848.5841.1125.2818.9925.9124.0026.1370.0086.6443.352.9886.3056.7154.1941.62Vanilla822.524.8431.9048.1641.3226.7719.4928.2324.6326.4170.0086.3241.832.9188.0656.9853.7441.97ThinK822.425.7732.6748.741.0627.0719.1428.9124.3726.8870.0086.3742.753.6187.3857.2154.4442.27Impact of Different Recent Sizes.Preserve the most recent KV embeddingsis important for maintaining the performance of LLMs after KV cache compression. Note that a tradeoff exists: increasing the recent-size allows more infomation to be propagated, while increasing the cache size to be stored. To study its impacts, we evaluate the performance produced by three recent-size, namely,and, on LongBench, using Mistral-7B-Instruct-v0.2 as the basleline model. The results are summarized in Table5. One can observe that a recent-size of 32 yields superior performance thanin terms of averaged score on LongBench which demonstrates the importance of keeping the most recent KVs. On the other hand, the performance ofandis negligible, suggesting that maintaining the most recent 32 KVs suffices to preserve optimal performance.Figure 5:Performance comparisons under the same KV cache memory consumption.Comparison Under the Same Memory Consumption.To ensure a fair comparison, we adjust the KV cache size of H2O or SnapKV to match the memory usage of H2O with ThinK or SnapKV with ThinK.
For example, the KV cache size of H2O with ThinK is. Due to channel pruning of the key cache, the memory consumption of H2O with ThinK at aKV-size is lower than H2O at a 128 KV-size.
Therefore, the KV-size of H2O is adjusted fromto.
Table6and Figure5present the results of these comparisons on LongBench.
As the results show, H2O or SnapKV with ThinK deliver superior performance compared to H2O or SnapKV without ThinK while maintaining the same memory usage.
This demonstrates that KV cache compression methods, combined with our query-based channel pruning, can compress the KV cache more effectively and utilize memory more efficiently.Pruning Channels of Both Key and Value Cache.In our previous study, we evaluate methods with pruning channels of the key cache. In this part, we explore the impact of pruning channels of the value cache (AppendixA.1) . Specifically, for KV cache compression methods, we prune channels of the key cache at one pruning ratio and channels of the value cache at another pruning ratio.
Table7presents the results with LLaMA3-8B and Mistral-7B on LongBench.
When testing on the base model LLaMA3-8B, H2O or SnapKV with key and value channel pruning can perform on par with H2O or SnapKV without channel pruning. In some cases, H2O or SnapKV with key and value channel pruning may even perform better. Pruning the channels of the value cache over the base model Mistral-7B experiences a slight performance drop. Despite this, note that KV cache with additional value channel pruning can further reduce memory usage.Table 7:Performance comparison of pruning both K and V cache with different pruning ratios on LongBench.
H2OThinK(+)  indicates that H2O has% of the Key cache channels pruned and% of the Value cache channels pruned.MethodSingle-Document QAMulti-Document QASummarizationFew-shot LearningSyntheticCodeAvg.NrtvQAQasperMF-enHotpotQA2WikiMQAMusiqueGovReportQMSumMultiNewsTRECTriviaQASAMSumPCountPReLccRB-PLLaMA3 8BKV-size 128H2O22.1213.2031.6137.7932.7118.4520.3222.0221.1038.5087.7539.145.8369.5055.0650.9735.38+ThinK(30+30) 23.7113.6533.0841.8629.8818.0419.6021.6520.2638.0086.0838.615.1669.5057.5955.1935.74\cdashline2-19SnapKV21.1913.5532.6438.7529.6418.7318.9821.6220.2645.0088.3637.645.1368.8555.8451.8235.50+ThinK(30+30) 21.8613.7933.2640.9329.3919.2218.8121.3019.2641.5087.0037.955.7869.5057.8455.6235.81KV-size 512H2O23.5217.9334.6842.1133.5219.9222.1122.5623.8241.0090.4640.205.8769.5056.7151.6937.23+ThinK(30+30) 22.8317.5734.1842.6733.5219.9521.1722.2322.8238.5090.1139.085.2169.059.9956.8337.23\cdashline2-19SnapKV24.8423.9638.7742.7534.5520.8722.2622.6123.9770.0090.5240.295.8169.5059.0451.8140.10+ThinK(30+30) 24.5724.5938.0944.6134.3720.3721.2321.9523.3066.0090.6939.385.6069.0061.7558.4640.25KV-size 2048H2O25.5626.8539.5444.3032.9221.0924.6823.0126.1653.0090.6541.844.9169.2558.4351.3139.59+ThinK(30+30) 25.0326.7739.6842.1233.0819.5923.0022.8925.2751.0091.1140.585.2369.0061.1257.9539.59+ThinK(40+40) 24.8724.3137.7743.1334.4219.6021.6722.7024.5249.0090.8139.286.0069.0061.8158.0839.19\cdashline2-19SnapKV25.8629.5541.1044.9935.8021.8125.9823.4026.4673.5090.5641.665.1769.2558.6751.5241.58+ThinK(30+30) 25.1329.9740.3544.1234.6419.9423.6223.0325.3072.5090.7839.465.3569.0061.5057.9141.41+ThinK(40+40) 25.1328.8540.7044.2136.3621.0722.3122.8924.8072.5090.8838.776.4169.0061.4958.8741.52Mistral 7BKV-size 128H2O21.2121.8138.8730.4220.3612.3020.5822.6122.1039.0082.3740.442.9079.5651.2248.3834.63+ThinK(30+30) 20.7121.4938.0130.6622.2813.8720.1322.4521.0738.5082.2038.692.9478.5651.5548.2834.46\cdashline2-19SnapKV19.1721.4042.9336.7622.4415.8619.1621.8421.5547.5084.1540.242.3068.2652.3148.8035.29+ThinK(30+30) 19.9220.6142.6837.6323.1915.0918.9721.9320.5545.0084.0639.332.9966.0051.5147.5134.81KV-size 512H2O21.8326.0044.6932.4623.0514.6923.5323.0624.5942.0085.2241.493.4086.2054.7851.0937.38+ThinK(30+30) 22.3624.2644.7730.4722.9414.9622.6322.9023.7341.5085.3040.213.0880.0754.4850.9636.54+ThinK(30+10) 22.1425.1545.2931.7823.2114.6223.3622.7024.5141.5085.6141.582.7584.0354.5051.0937.11\cdashline2-19SnapKV24.4427.8148.9839.4625.2516.9823.7022.9624.3767.085.8841.262.7886.5656.4653.4140.46+ThinK(30+30) 24.1027.0447.7638.6625.4517.5122.6422.8123.9166.0086.6239.913.3682.2455.9652.8139.80+ThinK(30+10) 23.9028.1448.3539.0324.8316.6823.5123.1224.3467.5086.0941.692.6584.3457.2953.2240.29KV-size 1024H2O23.6728.5546.436.9924.8215.0225.2123.0425.7746.0085.9341.983.2486.5756.4052.7538.90+ThinK(30+30) 23.6526.5447.0035.5224.7917.1523.6423.1225.2044.0086.3841.673.4680.1456.5352.8638.23+ThinK(30+10) 24.1328.5746.3135.5924.9215.3424.5823.3325.9345.5085.9142.972.5783.6455.3952.7338.59\cdashline2-19SnapKV25.4729.5749.3340.9025.5319.0125.9423.8926.2169.5086.4842.102.9888.5657.1953.6041.64+ThinK(30+30) 25.2929.2549.1741.2525.7519.3724.6423.0225.2769.0086.7040.923.2982.0657.1554.1541.02+ThinK(30+10) 25.8429.3049.5641.4425.2919.0225.2123.7325.7269.0086.6942.552.4485.7657.5554.1041.45KV-size 2048H2O25.7631.1049.0640.3826.4316.7827.1723.6426.6955.086.3542.482.7286.6456.9853.9140.69+ThinK(30+30) 25.6028.7447.5438.6726.2517.3524.5423.2726.1551.0087.0143.022.9481.4656.4154.2639.64+ThinK(30+10) 25.6430.6548.9540.4226.4316.6526.7623.5126.5952.5086.5343.452.6683.9656.5553.8340.32\cdashline2-19SnapKV25.8932.5648.5541.6827.2418.7528.9024.4726.6370.0086.2742.573.0986.9357.4453.8342.18+ThinK(30+30) 27.0130.7248.8141.1526.9318.9325.8123.5926.4270.0086.8241.913.0582.6557.0154.2541.57+ThinK(30+10) 26.2232.6948.9640.8326.7019.0227.8724.2326.6470.0086.6542.632.2285.1357.0054.2841.94(a)  KV-size 128, SnapKV, Acc.77.8(b)  KV-size 128, SnapKV+ThinK()  Acc.78.6(c)  KV-size 1024, SnapKV, Acc.90.4(d)  KV-size 1024, SnapKV+ThinK() , Acc.90.8Figure 6:Needle-in-a-Haystack test performance comparison with Mistral-7B-Instruct-v0.2.
body
ThinK:ThinnerKey Cache by Query-Driven Pruning
4Experiment Results
2407.21018
5Related WorkIn scenarios involving long contexts, the most significant computational burden from the attention mechanism is the key-value (KV)  cache. Reducing the KV cache is a high priority for optimizing deployment efficiency. System-level optimizations, such as FlashAttentionand PagedAttention, have been developed to address this issue. Additionally, algorithm-level optimizations are being explored to further enhance efficiency.KV Cache Eviction.StreamingLLMmaintains a few initial tokens and some recent tokens based on the observation of attention sink, which may result in the loss of important information carried by the dropped tokens. H2Oretains only a small portion of the tokens by greedily dropping tokens based on their contributions to the cumulative attention. SnapKVselects clustered important KV positions for each attention head from an ‘observation’ window located at the end of the prompts. FastGenadaptively evicts tokens from attention heads that emphasize local contexts. This approach focuses on discarding non-special tokens centered on special tokens, while the standard KV cache is used only for attention heads that broadly attend to all tokens. PyramidKVand PyramidInferconsiders adjusting the KV cache size across different layers by allocating more cache in the lower layers and less in the higher ones.KV Cache Quantization.SmoothQuantcan quantize the KV cache to 8-bit with minimal performance degradation. Q-Hitteruses accumulated attention scores and ‘Quantization Friendliness’ metrics to identify tokens that are essential for maintaining the generalization capabilities of LLMs and are suitable for KV cache quantization. Some studies have found that the key cache and value cache should be quantized differently: the key cache should be quantized per-channel, while the value cache should be quantized per-token.Structured Pruning of LLMs.Structured pruningof LLMs traditionally focuses on removing unimportant layers, heads, and hidden dimensions, which often results in significant performance degradation. In contrast, our methodology preserves the original architecture of the LLM and specifically targets the channel dimension within each head’s key cache. By dynamically identifying unimportant channels based on data dependant criterion, our approach greatly reduce the key cache-size with negligible performance loss.
body
ThinK:ThinnerKey Cache by Query-Driven Pruning
5Related Work
2407.21018
6End Note and Future DirectionMotivated by the observation that certain channels have significantly larger magnitudes compared to others, and the singular value analysis indicates that the key cache is inherently low-rank, we proposeThinKto perform pruning over the key cache channel.
The proposed pruning strategy is query-dependant and optimized based on the attention scores, ensuring that essential information is retained for each input query.
In addition,ThinKcan be seamlessly integrated with other popular token-level KV cache quantization techniques, further enhancing inference efficiency.
Extensive experiments on LongBenchand Needle-in-a-Haystack tests with two foundation models demonstrate the effectiveness and robustness of our query-dependent channel pruning method.
Our approach achieves comparable or superior performance to baseline methods while reducing the key cache size by%.
Our analysis indicates thatThinKcan maintain superior performance over baselines with a smaller KV cache size under equivalent memory consumption conditions.Future work will focus on strategies to increase the pruning ratio without compromising model performance, and exploring pruning techniques for value caches.
We also plan to evaluate channel importance from a more sophisticated and compositional perspective, incorporating both token-level and channel-level information to calculate importance.
conclusion
ThinK:ThinnerKey Cache by Query-Driven Pruning
6End Note and Future Direction