<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2407.21118] Palu: Compressing KV-Cache with Low-Rank Projection</title><meta property="og:description" content="KV-Cache compression methods generally sample a KV-Cache of effectual tokens or quantize it into lower bits.
However, these methods cannot exploit the redundancy of the hidden dimension of KV tensors.
This paper invest…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Palu: Compressing KV-Cache with Low-Rank Projection">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Palu: Compressing KV-Cache with Low-Rank Projection">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2407.21118">

<!--Generated on Mon Aug  5 13:10:25 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document">
<h1 class="ltx_title ltx_title_document">Palu: Compressing KV-Cache with Low-Rank Projection</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Chi-Chih Chang<sup id="id19.18.id1" class="ltx_sup"><span id="id19.18.id1.1" class="ltx_text ltx_font_italic">1,∗</span></sup>, Wei-Cheng Lin<sup id="id20.19.id2" class="ltx_sup"><span id="id20.19.id2.1" class="ltx_text ltx_font_italic">1,∗</span></sup>, Chien-Yu Lin<sup id="id21.20.id3" class="ltx_sup"><span id="id21.20.id3.1" class="ltx_text ltx_font_italic">2,∗</span></sup>, Chong-Yan Chen<sup id="id22.21.id4" class="ltx_sup"><span id="id22.21.id4.1" class="ltx_text ltx_font_italic">1</span></sup>, 
<br class="ltx_break"><span id="id8.8.4" class="ltx_text ltx_font_bold">Yu-Fang Hu<sup id="id8.8.4.1" class="ltx_sup"><span id="id8.8.4.1.1" class="ltx_text ltx_font_medium ltx_font_italic">1</span></sup>, Pei-Shuo Wang<sup id="id8.8.4.2" class="ltx_sup"><span id="id8.8.4.2.1" class="ltx_text ltx_font_medium ltx_font_italic">1</span></sup>, Ning-Chi Huang<sup id="id8.8.4.3" class="ltx_sup"><span id="id8.8.4.3.1" class="ltx_text ltx_font_medium ltx_font_italic">1</span></sup>, Luis Ceze<sup id="id8.8.4.4" class="ltx_sup"><span id="id8.8.4.4.1" class="ltx_text ltx_font_medium ltx_font_italic">2</span></sup></span> and <span id="id9.9.5" class="ltx_text ltx_font_bold">Kai-Chiang Wu<sup id="id9.9.5.1" class="ltx_sup"><span id="id9.9.5.1.1" class="ltx_text ltx_font_medium ltx_font_italic">1</span></sup></span> 
<br class="ltx_break"><sup id="id23.22.id5" class="ltx_sup">1</sup>National Yang Ming Chiao Tung University, <sup id="id24.23.id6" class="ltx_sup">2</sup>University of Washington 
<br class="ltx_break"><math id="id12.12.m7.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="id12.12.m7.1a"><mo stretchy="false" id="id12.12.m7.1.1" xref="id12.12.m7.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="id12.12.m7.1b"><ci id="id12.12.m7.1.1.cmml" xref="id12.12.m7.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="id12.12.m7.1c">\{</annotation></semantics></math><span id="id15.15.8" class="ltx_text ltx_font_typewriter">brian1009.en08, weicheng.lin.cs11, a.cs09, yuko29.cs11, scottlr306.cs12<math id="id13.13.6.m1.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="id13.13.6.m1.1a"><mo stretchy="false" id="id13.13.6.m1.1.1" xref="id13.13.6.m1.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="id13.13.6.m1.1b"><ci id="id13.13.6.m1.1.1.cmml" xref="id13.13.6.m1.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="id13.13.6.m1.1c">\}</annotation></semantics></math>@nycu.edu.tw
<br class="ltx_break"><math id="id14.14.7.m2.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="id14.14.7.m2.1a"><mo stretchy="false" id="id14.14.7.m2.1.1" xref="id14.14.7.m2.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="id14.14.7.m2.1b"><ci id="id14.14.7.m2.1.1.cmml" xref="id14.14.7.m2.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="id14.14.7.m2.1c">\{</annotation></semantics></math>nchuang.cs, kcw.cs<math id="id15.15.8.m3.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="id15.15.8.m3.1a"><mo stretchy="false" id="id15.15.8.m3.1.1" xref="id15.15.8.m3.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="id15.15.8.m3.1b"><ci id="id15.15.8.m3.1.1.cmml" xref="id15.15.8.m3.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="id15.15.8.m3.1c">\}</annotation></semantics></math>@nycu.edu.tw</span>,
<math id="id16.16.m8.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="id16.16.m8.1a"><mo stretchy="false" id="id16.16.m8.1.1" xref="id16.16.m8.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="id16.16.m8.1b"><ci id="id16.16.m8.1.1.cmml" xref="id16.16.m8.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="id16.16.m8.1c">\{</annotation></semantics></math><span id="id17.17.9" class="ltx_text ltx_font_typewriter">cyulin, luisceze<math id="id17.17.9.m1.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="id17.17.9.m1.1a"><mo stretchy="false" id="id17.17.9.m1.1.1" xref="id17.17.9.m1.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="id17.17.9.m1.1b"><ci id="id17.17.9.m1.1.1.cmml" xref="id17.17.9.m1.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="id17.17.9.m1.1c">\}</annotation></semantics></math>@cs.washington.edu</span>
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id18.1" class="ltx_p">KV-Cache compression methods generally sample a KV-Cache of effectual tokens or quantize it into lower bits.
However, these methods cannot exploit the redundancy of the hidden dimension of KV tensors.
This paper investigates a unique hidden dimension approach called <span id="id18.1.2" class="ltx_text ltx_font_italic">Palu</span>, a novel KV-Cache compression framework that utilizes low-rank projection. Palu decomposes the linear layers into low-rank matrices, caches the smaller intermediate states, and reconstructs the full keys and values on the fly. To improve accuracy, compression rate, and efficiency, <span id="id18.1.3" class="ltx_text ltx_font_italic">Palu</span> further encompasses (1) a medium-grained low-rank decomposition scheme, (2) an efficient rank search algorithm, (3) a low-rank-aware quantization algorithm, and (4) matrix fusion with optimized GPU kernels.
Our extensive experiments with popular LLMs show that <span id="id18.1.4" class="ltx_text ltx_font_italic">Palu</span> can compress KV-Cache by more than <span id="id18.1.5" class="ltx_text ltx_font_bold">91.25%</span> while maintaining a significantly <span id="id18.1.6" class="ltx_text ltx_font_bold">better accuracy (up to 1.19 lower perplexity)</span> than state-of-the-art KV-Cache quantization methods at a similar or even higher memory usage. When compressing KV-Cache for 50%, <span id="id18.1.7" class="ltx_text ltx_font_italic">Palu</span> delivers up to <span id="id18.1.1" class="ltx_text ltx_font_bold">1.61<math id="id18.1.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="id18.1.1.m1.1a"><mo id="id18.1.1.m1.1.1" xref="id18.1.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="id18.1.1.m1.1b"><times id="id18.1.1.m1.1.1.cmml" xref="id18.1.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="id18.1.1.m1.1c">\times</annotation></semantics></math> end-to-end speedup</span> for the attention module. Our code is publicly available at: <a target="_blank" href="https://github.com/shadowpa0327/Palu" title="" class="ltx_ref ltx_href">https://github.com/shadowpa0327/Palu</a>.</p>
</div>
<div id="p1" class="ltx_para ltx_noindent">
<div id="p1.17" class="ltx_block ltx_align_bottom">
<p id="p1.17.18" class="ltx_p"><span id="p1.17.18.1" class="ltx_text ltx_font_bold">Palu: Compressing KV-Cache with Low-Rank Projection</span></p>
<br class="ltx_break ltx_centering">
<p id="p1.17.17" class="ltx_p ltx_align_center" style="width:433.6pt;"><span id="p1.17.17.17" class="ltx_text ltx_inline-block" style="width:0.0pt;">
<span id="p1.17.17.17.17" class="ltx_tabular ltx_align_top">
<span class="ltx_tbody">
<span id="p1.4.4.4.4.4" class="ltx_tr">
<span id="p1.4.4.4.4.4.4" class="ltx_td ltx_align_center"><span id="p1.4.4.4.4.4.4.4" class="ltx_text ltx_font_bold">Chi-Chih Chang<sup id="p1.4.4.4.4.4.4.4.1" class="ltx_sup"><span id="p1.4.4.4.4.4.4.4.1.1" class="ltx_text ltx_font_medium ltx_font_italic">1,∗</span></sup>, Wei-Cheng Lin<sup id="p1.4.4.4.4.4.4.4.2" class="ltx_sup"><span id="p1.4.4.4.4.4.4.4.2.1" class="ltx_text ltx_font_medium ltx_font_italic">1,∗</span></sup>, Chien-Yu Lin<sup id="p1.4.4.4.4.4.4.4.3" class="ltx_sup"><span id="p1.4.4.4.4.4.4.4.3.1" class="ltx_text ltx_font_medium ltx_font_italic">2,∗</span></sup>, Chong-Yan Chen<sup id="p1.4.4.4.4.4.4.4.4" class="ltx_sup"><span id="p1.4.4.4.4.4.4.4.4.1" class="ltx_text ltx_font_medium ltx_font_italic">1</span></sup>,</span></span></span>
<span id="p1.9.9.9.9.9" class="ltx_tr">
<span id="p1.9.9.9.9.9.5" class="ltx_td ltx_align_center"><span id="p1.8.8.8.8.8.4.4" class="ltx_text ltx_font_bold">Yu-Fang Hu<sup id="p1.8.8.8.8.8.4.4.1" class="ltx_sup"><span id="p1.8.8.8.8.8.4.4.1.1" class="ltx_text ltx_font_medium ltx_font_italic">1</span></sup>, Pei-Shuo Wang<sup id="p1.8.8.8.8.8.4.4.2" class="ltx_sup"><span id="p1.8.8.8.8.8.4.4.2.1" class="ltx_text ltx_font_medium ltx_font_italic">1</span></sup>, Ning-Chi Huang<sup id="p1.8.8.8.8.8.4.4.3" class="ltx_sup"><span id="p1.8.8.8.8.8.4.4.3.1" class="ltx_text ltx_font_medium ltx_font_italic">1</span></sup>, Luis Ceze<sup id="p1.8.8.8.8.8.4.4.4" class="ltx_sup"><span id="p1.8.8.8.8.8.4.4.4.1" class="ltx_text ltx_font_medium ltx_font_italic">2</span></sup></span> and <span id="p1.9.9.9.9.9.5.5" class="ltx_text ltx_font_bold">Kai-Chiang Wu<sup id="p1.9.9.9.9.9.5.5.1" class="ltx_sup"><span id="p1.9.9.9.9.9.5.5.1.1" class="ltx_text ltx_font_medium ltx_font_italic">1</span></sup></span></span></span>
<span id="p1.11.11.11.11.11" class="ltx_tr">
<span id="p1.11.11.11.11.11.2" class="ltx_td ltx_align_center"><sup id="p1.11.11.11.11.11.2.1" class="ltx_sup">1</sup>National Yang Ming Chiao Tung University, <sup id="p1.11.11.11.11.11.2.2" class="ltx_sup">2</sup>University of Washington</span></span>
<span id="p1.13.13.13.13.13" class="ltx_tr">
<span id="p1.13.13.13.13.13.2" class="ltx_td ltx_align_center"><math id="p1.12.12.12.12.12.1.m1.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="p1.12.12.12.12.12.1.m1.1a"><mo stretchy="false" id="p1.12.12.12.12.12.1.m1.1.1" xref="p1.12.12.12.12.12.1.m1.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="p1.12.12.12.12.12.1.m1.1b"><ci id="p1.12.12.12.12.12.1.m1.1.1.cmml" xref="p1.12.12.12.12.12.1.m1.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="p1.12.12.12.12.12.1.m1.1c">\{</annotation></semantics></math><span id="p1.13.13.13.13.13.2.1" class="ltx_text ltx_font_typewriter">brian1009.en08, weicheng.lin.cs11, a.cs09, yuko29.cs11, scottlr306.cs12<math id="p1.13.13.13.13.13.2.1.m1.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="p1.13.13.13.13.13.2.1.m1.1a"><mo stretchy="false" id="p1.13.13.13.13.13.2.1.m1.1.1" xref="p1.13.13.13.13.13.2.1.m1.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="p1.13.13.13.13.13.2.1.m1.1b"><ci id="p1.13.13.13.13.13.2.1.m1.1.1.cmml" xref="p1.13.13.13.13.13.2.1.m1.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="p1.13.13.13.13.13.2.1.m1.1c">\}</annotation></semantics></math>@nycu.edu.tw</span></span></span>
<span id="p1.17.17.17.17.17" class="ltx_tr">
<span id="p1.17.17.17.17.17.4" class="ltx_td ltx_align_center"><math id="p1.14.14.14.14.14.1.m1.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="p1.14.14.14.14.14.1.m1.1a"><mo stretchy="false" id="p1.14.14.14.14.14.1.m1.1.1" xref="p1.14.14.14.14.14.1.m1.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="p1.14.14.14.14.14.1.m1.1b"><ci id="p1.14.14.14.14.14.1.m1.1.1.cmml" xref="p1.14.14.14.14.14.1.m1.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="p1.14.14.14.14.14.1.m1.1c">\{</annotation></semantics></math><span id="p1.15.15.15.15.15.2.1" class="ltx_text ltx_font_typewriter">nchuang.cs, kcw.cs<math id="p1.15.15.15.15.15.2.1.m1.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="p1.15.15.15.15.15.2.1.m1.1a"><mo stretchy="false" id="p1.15.15.15.15.15.2.1.m1.1.1" xref="p1.15.15.15.15.15.2.1.m1.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="p1.15.15.15.15.15.2.1.m1.1b"><ci id="p1.15.15.15.15.15.2.1.m1.1.1.cmml" xref="p1.15.15.15.15.15.2.1.m1.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="p1.15.15.15.15.15.2.1.m1.1c">\}</annotation></semantics></math>@nycu.edu.tw</span>,
<math id="p1.16.16.16.16.16.3.m2.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="p1.16.16.16.16.16.3.m2.1a"><mo stretchy="false" id="p1.16.16.16.16.16.3.m2.1.1" xref="p1.16.16.16.16.16.3.m2.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="p1.16.16.16.16.16.3.m2.1b"><ci id="p1.16.16.16.16.16.3.m2.1.1.cmml" xref="p1.16.16.16.16.16.3.m2.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="p1.16.16.16.16.16.3.m2.1c">\{</annotation></semantics></math><span id="p1.17.17.17.17.17.4.2" class="ltx_text ltx_font_typewriter">cyulin, luisceze<math id="p1.17.17.17.17.17.4.2.m1.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="p1.17.17.17.17.17.4.2.m1.1a"><mo stretchy="false" id="p1.17.17.17.17.17.4.2.m1.1.1" xref="p1.17.17.17.17.17.4.2.m1.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="p1.17.17.17.17.17.4.2.m1.1b"><ci id="p1.17.17.17.17.17.4.2.m1.1.1.cmml" xref="p1.17.17.17.17.17.4.2.m1.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="p1.17.17.17.17.17.4.2.m1.1c">\}</annotation></semantics></math>@cs.washington.edu</span></span></span>
</span>
</span></span></p>
<br class="ltx_break ltx_centering">
</div>
</div>
<span id="footnotex1" class="ltx_note ltx_role_footnotetext"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">footnotetext: </span><sup id="footnotex1.1" class="ltx_sup">∗</sup>These authors contributed equally to this work.</span></span></span>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Large language models (LLMs) are revolutionizing the AI industry and providing a high-level intelligence that previous machine learning (ML) models could not achieve.
To speed inference, caching key-value states (KV-Cache) in memory is a simple yet effective technique. However, the size of the KV-Cache can grow rapidly, straining memory capacity and bandwidth; further, the memory-bounded characteristics of the decoding stage also limit inference speed when loading KV-Cache data. Therefore, KV-Cache compression has become a central research topic for building efficient LLMs.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2407.21118/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="184" height="116" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>
<span id="S1.F1.9.1" class="ltx_text ltx_font_italic">Palu</span>’s <span id="S1.F1.10.2" class="ltx_text ltx_font_italic">low-rank projection method</span> for KV-Cache reduction. A weight matrix <span id="S1.F1.11.3" class="ltx_text ltx_font_bold">W</span> of linear projection is decomposed into two low-rank matrices. Input <span id="S1.F1.12.4" class="ltx_text ltx_font_bold">X</span> is down-projected to latent representation <span id="S1.F1.13.5" class="ltx_text ltx_font_bold">H</span>, which is cached. <span id="S1.F1.14.6" class="ltx_text ltx_font_bold">Y</span> is reconstructed from <span id="S1.F1.15.7" class="ltx_text ltx_font_bold">H</span> on-the-fly using the up-projection matrix <span id="S1.F1.16.8" class="ltx_text ltx_font_bold">B</span>.
</figcaption>
</figure>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">Research on KV-Cache compression falls into two categories: quantization and token eviction. Quantization methods aim to use less bit-width to represent each datum; token eviction techniques seek ways to retain only a partial set of KV-Cache. However, both categories fail to explore the hidden dimensions of KV-Cache where high redundancy often occurs.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">This work examines the problem from a fundamentally different perspective, i.e., ways to reduce the hidden dimensions. We introduce <span id="S1.p3.1.1" class="ltx_text ltx_font_italic">Palu</span>, a KV-Cache compression framework that reduces the size of the KV-Cache through<span id="S1.p3.1.2" class="ltx_text ltx_font_italic"> low-rank projection</span>. Although low-rank projection has been shown to improve LLM efficiency, e.g., by reducing model size <cite class="ltx_cite ltx_citemacro_citep">(Yuan et al., <a href="#bib.bib44" title="" class="ltx_ref">2023</a>; Wang et al., <a href="#bib.bib42" title="" class="ltx_ref">2024</a>)</cite> or minimizing memory footprint for finetuning <cite class="ltx_cite ltx_citemacro_citep">(Hu et al., <a href="#bib.bib18" title="" class="ltx_ref">2022</a>; Dettmers et al., <a href="#bib.bib10" title="" class="ltx_ref">2023</a>)</cite>, it has not been studied for compressing the KV-Cache, especially in post-training.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">A naive way to compress the KV-Cache with low-rank projection is by directly mapping cached matrices into low-rank space <cite class="ltx_cite ltx_citemacro_citep">(Jolliffe and Cadima, <a href="#bib.bib20" title="" class="ltx_ref">2016</a>; Zhao et al., <a href="#bib.bib47" title="" class="ltx_ref">2024</a>)</cite>. However, this approach imposes an unacceptably heavy overhead of computing the decomposition matrices during runtime, after the prefill stage. To avoid this, <span id="S1.p4.1.1" class="ltx_text ltx_font_italic">Palu</span> <span id="S1.p4.1.2" class="ltx_text ltx_font_italic">statically decomposes the key and value-projection weight matrices</span> and <span id="S1.p4.1.3" class="ltx_text ltx_font_italic">caches the latent representations of the low-rank decomposition</span> (Fig. <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>), instead of directly decomposing the KV-Cache. This innovative design enables <span id="S1.p4.1.4" class="ltx_text ltx_font_italic">Palu</span> to reduce memory while substantially mitigating runtime overhead.</p>
</div>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.1" class="ltx_p">Despite its promising memory reduction, we found that naively decomposing the key and value projection matrices along the head dimension causes a substantial loss in accuracy. Although we found that decomposing the entire matrix at one time can maintain accuracy, it significantly increases reconstruction overhead.
To find a balance, <span id="S1.p5.1.1" class="ltx_text ltx_font_italic">Palu</span> proposes a <span id="S1.p5.1.2" class="ltx_text ltx_font_italic">medium-grained, group-head low-rank decomposition optimization</span> that is both accurate and less costly in terms of reconstruction.</p>
</div>
<div id="S1.p6" class="ltx_para">
<p id="S1.p6.1" class="ltx_p">For LLMs, each linear projection module has a different sensitivity to compression <cite class="ltx_cite ltx_citemacro_cite">Sharma et al. (<a href="#bib.bib35" title="" class="ltx_ref">2023</a>); Yuan et al. (<a href="#bib.bib44" title="" class="ltx_ref">2023</a>)</cite>. To exploit the sensitivity and improve accuracy, we design an efficient <span id="S1.p6.1.1" class="ltx_text ltx_font_italic">rank search algorithm</span> based on Fisher information. Our algorithm automatically assigns a larger rank for important matrices and smaller ranks for less critical ones, boosting accuracy at the same compression rate.</p>
</div>
<div id="S1.p7" class="ltx_para">
<p id="S1.p7.1" class="ltx_p">In addition to low-rank decomposition, we integrate quantization into <span id="S1.p7.1.1" class="ltx_text ltx_font_italic">Palu</span>. We found that low-rank decomposition often induces severe outliers in the latent representation. Inspired by recent LLM quantization methods <cite class="ltx_cite ltx_citemacro_cite">Tseng et al. (<a href="#bib.bib40" title="" class="ltx_ref">2024</a>); Ashkboos et al. (<a href="#bib.bib2" title="" class="ltx_ref">2024b</a>)</cite>, we use Hadamard transformation matrices with a <span id="S1.p7.1.2" class="ltx_text ltx_font_italic">low-rank-aware quantization algorithm</span> to eliminate outliers and increase quantization accuracy. To avoid runtime overhead, <span id="S1.p7.1.3" class="ltx_text ltx_font_italic">Palu</span> fuses the transformation matrices into the forward and backward decomposed matrices, concealing the overhead completely.</p>
</div>
<div id="S1.p8" class="ltx_para">
<p id="S1.p8.1" class="ltx_p">To enhance inference latency, we integrate the decomposed matrix into the subsequent linear layer, thereby mitigating the online reconstruction overhead.
In scenarios where RoPE is applied and matrix fusion is infeasible, we developed an optimized GPU kernel to accelerate the reconstruction.
</p>
</div>
<div id="S1.p9" class="ltx_para">
<p id="S1.p9.2" class="ltx_p">We evaluate <span id="S1.p9.2.3" class="ltx_text ltx_font_italic">Palu</span> on popular LLMs and benchmarks. Our experiments show that <span id="S1.p9.2.4" class="ltx_text ltx_font_italic">Palu</span> maintains strong zero-shot accuracy and perplexity with up to 50% low-rank compression. Combining low-rank compression and quantization, <span id="S1.p9.2.5" class="ltx_text ltx_font_italic">Palu</span> compresses KV-Cache by <span id="S1.p9.1.1" class="ltx_text ltx_font_bold">over 91.25% (11.4<math id="S1.p9.1.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S1.p9.1.1.m1.1a"><mo id="S1.p9.1.1.m1.1.1" xref="S1.p9.1.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S1.p9.1.1.m1.1b"><times id="S1.p9.1.1.m1.1.1.cmml" xref="S1.p9.1.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S1.p9.1.1.m1.1c">\times</annotation></semantics></math> compression)</span> while maintaining perplexity. Compared to a state-of-the-art KV-Cache quantization <cite class="ltx_cite ltx_citemacro_citep">(Hooper et al., <a href="#bib.bib16" title="" class="ltx_ref">2024</a>)</cite> with 87.5% compression, <span id="S1.p9.2.6" class="ltx_text ltx_font_italic">Palu</span> achieves a <span id="S1.p9.2.7" class="ltx_text ltx_font_italic">significantly lower perplexity of 1.19</span>. Additionally, with 50% compression without quantization, <span id="S1.p9.2.8" class="ltx_text ltx_font_italic">Palu</span> demonstrates a <span id="S1.p9.2.2" class="ltx_text ltx_font_italic">1.61<math id="S1.p9.2.2.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S1.p9.2.2.m1.1a"><mo id="S1.p9.2.2.m1.1.1" xref="S1.p9.2.2.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S1.p9.2.2.m1.1b"><times id="S1.p9.2.2.m1.1.1.cmml" xref="S1.p9.2.2.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S1.p9.2.2.m1.1c">\times</annotation></semantics></math> end-to-end speedup</span>, highlighting <span id="S1.p9.2.9" class="ltx_text ltx_font_italic">Palu</span>’s ability to reduce memory usage and enhance speed for LLMs.</p>
</div>
<div id="S1.p10" class="ltx_para">
<p id="S1.p10.1" class="ltx_p">Our key contributions include:</p>
<ul id="S1.I1" class="ltx_itemize">
<li id="S1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i1.p1" class="ltx_para">
<p id="S1.I1.i1.p1.1" class="ltx_p"><span id="S1.I1.i1.p1.1.1" class="ltx_text ltx_font_italic">Palu</span>, a new post-training KV-Cache compression framework that caches <span id="S1.I1.i1.p1.1.2" class="ltx_text ltx_font_italic">low-rank latent representations.</span></p>
</div>
</li>
<li id="S1.I1.i2" class="ltx_item" style="list-style-type:none;padding-top:-2.0pt;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i2.p1" class="ltx_para">
<p id="S1.I1.i2.p1.1" class="ltx_p"><span id="S1.I1.i2.p1.1.1" class="ltx_text ltx_font_italic">Group-head low-rank decomposition (G-LRD)</span>, an optimization for balancing accuracy and efficiency.</p>
</div>
</li>
<li id="S1.I1.i3" class="ltx_item" style="list-style-type:none;padding-top:-2.0pt;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i3.p1" class="ltx_para">
<p id="S1.I1.i3.p1.1" class="ltx_p">An <span id="S1.I1.i3.p1.1.1" class="ltx_text ltx_font_italic">automated rank search</span> <span id="S1.I1.i3.p1.1.2" class="ltx_text ltx_font_italic">algorithm</span> for adaptively assigning ranks to each decomposed matrix, given a target compression rate.</p>
</div>
</li>
<li id="S1.I1.i4" class="ltx_item" style="list-style-type:none;padding-top:-2.0pt;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i4.p1" class="ltx_para">
<p id="S1.I1.i4.p1.1" class="ltx_p">A <span id="S1.I1.i4.p1.1.1" class="ltx_text ltx_font_italic">low-rank-aware quantization algorithm</span> that eliminates low-rank-induced outliers and imposes zero runtime overhead.</p>
</div>
</li>
</ul>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Background</h2>

<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>Multi-Head Attention Mechanism</h3>

<div id="S2.SS1.p1" class="ltx_para">
<p id="S2.SS1.p1.6" class="ltx_p">The multi-head attention (MHA) mechanism <cite class="ltx_cite ltx_citemacro_citep">(Vaswani et al., <a href="#bib.bib41" title="" class="ltx_ref">2017</a>)</cite> is a core component of the transformer architecture. Given a new input token <math id="S2.SS1.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{x}\in\mathbb{R}^{d}" display="inline"><semantics id="S2.SS1.p1.1.m1.1a"><mrow id="S2.SS1.p1.1.m1.1.1" xref="S2.SS1.p1.1.m1.1.1.cmml"><mi id="S2.SS1.p1.1.m1.1.1.2" xref="S2.SS1.p1.1.m1.1.1.2.cmml">𝐱</mi><mo id="S2.SS1.p1.1.m1.1.1.1" xref="S2.SS1.p1.1.m1.1.1.1.cmml">∈</mo><msup id="S2.SS1.p1.1.m1.1.1.3" xref="S2.SS1.p1.1.m1.1.1.3.cmml"><mi id="S2.SS1.p1.1.m1.1.1.3.2" xref="S2.SS1.p1.1.m1.1.1.3.2.cmml">ℝ</mi><mi id="S2.SS1.p1.1.m1.1.1.3.3" xref="S2.SS1.p1.1.m1.1.1.3.3.cmml">d</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.1.m1.1b"><apply id="S2.SS1.p1.1.m1.1.1.cmml" xref="S2.SS1.p1.1.m1.1.1"><in id="S2.SS1.p1.1.m1.1.1.1.cmml" xref="S2.SS1.p1.1.m1.1.1.1"></in><ci id="S2.SS1.p1.1.m1.1.1.2.cmml" xref="S2.SS1.p1.1.m1.1.1.2">𝐱</ci><apply id="S2.SS1.p1.1.m1.1.1.3.cmml" xref="S2.SS1.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S2.SS1.p1.1.m1.1.1.3.1.cmml" xref="S2.SS1.p1.1.m1.1.1.3">superscript</csymbol><ci id="S2.SS1.p1.1.m1.1.1.3.2.cmml" xref="S2.SS1.p1.1.m1.1.1.3.2">ℝ</ci><ci id="S2.SS1.p1.1.m1.1.1.3.3.cmml" xref="S2.SS1.p1.1.m1.1.1.3.3">𝑑</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.1.m1.1c">\mathbf{x}\in\mathbb{R}^{d}</annotation></semantics></math>, an MHA with <math id="S2.SS1.p1.2.m2.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S2.SS1.p1.2.m2.1a"><mi id="S2.SS1.p1.2.m2.1.1" xref="S2.SS1.p1.2.m2.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.2.m2.1b"><ci id="S2.SS1.p1.2.m2.1.1.cmml" xref="S2.SS1.p1.2.m2.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.2.m2.1c">n</annotation></semantics></math> heads projects the input into multiple queries, keys, and values using weight matrices <math id="S2.SS1.p1.3.m3.1" class="ltx_Math" alttext="\mathbf{W}^{q}_{i}" display="inline"><semantics id="S2.SS1.p1.3.m3.1a"><msubsup id="S2.SS1.p1.3.m3.1.1" xref="S2.SS1.p1.3.m3.1.1.cmml"><mi id="S2.SS1.p1.3.m3.1.1.2.2" xref="S2.SS1.p1.3.m3.1.1.2.2.cmml">𝐖</mi><mi id="S2.SS1.p1.3.m3.1.1.3" xref="S2.SS1.p1.3.m3.1.1.3.cmml">i</mi><mi id="S2.SS1.p1.3.m3.1.1.2.3" xref="S2.SS1.p1.3.m3.1.1.2.3.cmml">q</mi></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.3.m3.1b"><apply id="S2.SS1.p1.3.m3.1.1.cmml" xref="S2.SS1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.3.m3.1.1.1.cmml" xref="S2.SS1.p1.3.m3.1.1">subscript</csymbol><apply id="S2.SS1.p1.3.m3.1.1.2.cmml" xref="S2.SS1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.3.m3.1.1.2.1.cmml" xref="S2.SS1.p1.3.m3.1.1">superscript</csymbol><ci id="S2.SS1.p1.3.m3.1.1.2.2.cmml" xref="S2.SS1.p1.3.m3.1.1.2.2">𝐖</ci><ci id="S2.SS1.p1.3.m3.1.1.2.3.cmml" xref="S2.SS1.p1.3.m3.1.1.2.3">𝑞</ci></apply><ci id="S2.SS1.p1.3.m3.1.1.3.cmml" xref="S2.SS1.p1.3.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.3.m3.1c">\mathbf{W}^{q}_{i}</annotation></semantics></math>, <math id="S2.SS1.p1.4.m4.1" class="ltx_Math" alttext="\mathbf{W}_{i}^{k}" display="inline"><semantics id="S2.SS1.p1.4.m4.1a"><msubsup id="S2.SS1.p1.4.m4.1.1" xref="S2.SS1.p1.4.m4.1.1.cmml"><mi id="S2.SS1.p1.4.m4.1.1.2.2" xref="S2.SS1.p1.4.m4.1.1.2.2.cmml">𝐖</mi><mi id="S2.SS1.p1.4.m4.1.1.2.3" xref="S2.SS1.p1.4.m4.1.1.2.3.cmml">i</mi><mi id="S2.SS1.p1.4.m4.1.1.3" xref="S2.SS1.p1.4.m4.1.1.3.cmml">k</mi></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.4.m4.1b"><apply id="S2.SS1.p1.4.m4.1.1.cmml" xref="S2.SS1.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.4.m4.1.1.1.cmml" xref="S2.SS1.p1.4.m4.1.1">superscript</csymbol><apply id="S2.SS1.p1.4.m4.1.1.2.cmml" xref="S2.SS1.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.4.m4.1.1.2.1.cmml" xref="S2.SS1.p1.4.m4.1.1">subscript</csymbol><ci id="S2.SS1.p1.4.m4.1.1.2.2.cmml" xref="S2.SS1.p1.4.m4.1.1.2.2">𝐖</ci><ci id="S2.SS1.p1.4.m4.1.1.2.3.cmml" xref="S2.SS1.p1.4.m4.1.1.2.3">𝑖</ci></apply><ci id="S2.SS1.p1.4.m4.1.1.3.cmml" xref="S2.SS1.p1.4.m4.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.4.m4.1c">\mathbf{W}_{i}^{k}</annotation></semantics></math>, and <math id="S2.SS1.p1.5.m5.1" class="ltx_Math" alttext="\mathbf{W}_{i}^{v}" display="inline"><semantics id="S2.SS1.p1.5.m5.1a"><msubsup id="S2.SS1.p1.5.m5.1.1" xref="S2.SS1.p1.5.m5.1.1.cmml"><mi id="S2.SS1.p1.5.m5.1.1.2.2" xref="S2.SS1.p1.5.m5.1.1.2.2.cmml">𝐖</mi><mi id="S2.SS1.p1.5.m5.1.1.2.3" xref="S2.SS1.p1.5.m5.1.1.2.3.cmml">i</mi><mi id="S2.SS1.p1.5.m5.1.1.3" xref="S2.SS1.p1.5.m5.1.1.3.cmml">v</mi></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.5.m5.1b"><apply id="S2.SS1.p1.5.m5.1.1.cmml" xref="S2.SS1.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.5.m5.1.1.1.cmml" xref="S2.SS1.p1.5.m5.1.1">superscript</csymbol><apply id="S2.SS1.p1.5.m5.1.1.2.cmml" xref="S2.SS1.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.5.m5.1.1.2.1.cmml" xref="S2.SS1.p1.5.m5.1.1">subscript</csymbol><ci id="S2.SS1.p1.5.m5.1.1.2.2.cmml" xref="S2.SS1.p1.5.m5.1.1.2.2">𝐖</ci><ci id="S2.SS1.p1.5.m5.1.1.2.3.cmml" xref="S2.SS1.p1.5.m5.1.1.2.3">𝑖</ci></apply><ci id="S2.SS1.p1.5.m5.1.1.3.cmml" xref="S2.SS1.p1.5.m5.1.1.3">𝑣</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.5.m5.1c">\mathbf{W}_{i}^{v}</annotation></semantics></math>, respectively, for each head <math id="S2.SS1.p1.6.m6.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S2.SS1.p1.6.m6.1a"><mi id="S2.SS1.p1.6.m6.1.1" xref="S2.SS1.p1.6.m6.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.6.m6.1b"><ci id="S2.SS1.p1.6.m6.1.1.cmml" xref="S2.SS1.p1.6.m6.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.6.m6.1c">i</annotation></semantics></math>, as shown by</p>
</div>
<div id="S2.SS1.p2" class="ltx_para">
<table id="S2.E1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E1.m1.1" class="ltx_Math" alttext="\mathbf{q}_{i}=\mathbf{x}\mathbf{W}_{i}^{q},\hskip 5.0pt\mathbf{k}_{i}=\mathbf{x}\mathbf{W}^{k}_{i},\hskip 5.0pt\mathbf{v}_{i}=\mathbf{x}\mathbf{W}^{v}_{i}." display="block"><semantics id="S2.E1.m1.1a"><mrow id="S2.E1.m1.1.1.1"><mrow id="S2.E1.m1.1.1.1.1.2" xref="S2.E1.m1.1.1.1.1.3.cmml"><mrow id="S2.E1.m1.1.1.1.1.1.1" xref="S2.E1.m1.1.1.1.1.1.1.cmml"><msub id="S2.E1.m1.1.1.1.1.1.1.2" xref="S2.E1.m1.1.1.1.1.1.1.2.cmml"><mi id="S2.E1.m1.1.1.1.1.1.1.2.2" xref="S2.E1.m1.1.1.1.1.1.1.2.2.cmml">𝐪</mi><mi id="S2.E1.m1.1.1.1.1.1.1.2.3" xref="S2.E1.m1.1.1.1.1.1.1.2.3.cmml">i</mi></msub><mo id="S2.E1.m1.1.1.1.1.1.1.1" xref="S2.E1.m1.1.1.1.1.1.1.1.cmml">=</mo><msubsup id="S2.E1.m1.1.1.1.1.1.1.3" xref="S2.E1.m1.1.1.1.1.1.1.3.cmml"><mi id="S2.E1.m1.1.1.1.1.1.1.3.2.2" xref="S2.E1.m1.1.1.1.1.1.1.3.2.2.cmml">𝐱𝐖</mi><mi id="S2.E1.m1.1.1.1.1.1.1.3.2.3" xref="S2.E1.m1.1.1.1.1.1.1.3.2.3.cmml">i</mi><mi id="S2.E1.m1.1.1.1.1.1.1.3.3" xref="S2.E1.m1.1.1.1.1.1.1.3.3.cmml">q</mi></msubsup></mrow><mo rspace="0.667em" id="S2.E1.m1.1.1.1.1.2.3" xref="S2.E1.m1.1.1.1.1.3a.cmml">,</mo><mrow id="S2.E1.m1.1.1.1.1.2.2.2" xref="S2.E1.m1.1.1.1.1.2.2.3.cmml"><mrow id="S2.E1.m1.1.1.1.1.2.2.1.1" xref="S2.E1.m1.1.1.1.1.2.2.1.1.cmml"><msub id="S2.E1.m1.1.1.1.1.2.2.1.1.2" xref="S2.E1.m1.1.1.1.1.2.2.1.1.2.cmml"><mi id="S2.E1.m1.1.1.1.1.2.2.1.1.2.2" xref="S2.E1.m1.1.1.1.1.2.2.1.1.2.2.cmml">𝐤</mi><mi id="S2.E1.m1.1.1.1.1.2.2.1.1.2.3" xref="S2.E1.m1.1.1.1.1.2.2.1.1.2.3.cmml">i</mi></msub><mo id="S2.E1.m1.1.1.1.1.2.2.1.1.1" xref="S2.E1.m1.1.1.1.1.2.2.1.1.1.cmml">=</mo><msubsup id="S2.E1.m1.1.1.1.1.2.2.1.1.3" xref="S2.E1.m1.1.1.1.1.2.2.1.1.3.cmml"><mi id="S2.E1.m1.1.1.1.1.2.2.1.1.3.2.2" xref="S2.E1.m1.1.1.1.1.2.2.1.1.3.2.2.cmml">𝐱𝐖</mi><mi id="S2.E1.m1.1.1.1.1.2.2.1.1.3.3" xref="S2.E1.m1.1.1.1.1.2.2.1.1.3.3.cmml">i</mi><mi id="S2.E1.m1.1.1.1.1.2.2.1.1.3.2.3" xref="S2.E1.m1.1.1.1.1.2.2.1.1.3.2.3.cmml">k</mi></msubsup></mrow><mo rspace="0.667em" id="S2.E1.m1.1.1.1.1.2.2.2.3" xref="S2.E1.m1.1.1.1.1.2.2.3a.cmml">,</mo><mrow id="S2.E1.m1.1.1.1.1.2.2.2.2" xref="S2.E1.m1.1.1.1.1.2.2.2.2.cmml"><msub id="S2.E1.m1.1.1.1.1.2.2.2.2.2" xref="S2.E1.m1.1.1.1.1.2.2.2.2.2.cmml"><mi id="S2.E1.m1.1.1.1.1.2.2.2.2.2.2" xref="S2.E1.m1.1.1.1.1.2.2.2.2.2.2.cmml">𝐯</mi><mi id="S2.E1.m1.1.1.1.1.2.2.2.2.2.3" xref="S2.E1.m1.1.1.1.1.2.2.2.2.2.3.cmml">i</mi></msub><mo id="S2.E1.m1.1.1.1.1.2.2.2.2.1" xref="S2.E1.m1.1.1.1.1.2.2.2.2.1.cmml">=</mo><msubsup id="S2.E1.m1.1.1.1.1.2.2.2.2.3" xref="S2.E1.m1.1.1.1.1.2.2.2.2.3.cmml"><mi id="S2.E1.m1.1.1.1.1.2.2.2.2.3.2.2" xref="S2.E1.m1.1.1.1.1.2.2.2.2.3.2.2.cmml">𝐱𝐖</mi><mi id="S2.E1.m1.1.1.1.1.2.2.2.2.3.3" xref="S2.E1.m1.1.1.1.1.2.2.2.2.3.3.cmml">i</mi><mi id="S2.E1.m1.1.1.1.1.2.2.2.2.3.2.3" xref="S2.E1.m1.1.1.1.1.2.2.2.2.3.2.3.cmml">v</mi></msubsup></mrow></mrow></mrow><mo lspace="0em" id="S2.E1.m1.1.1.1.2">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E1.m1.1b"><apply id="S2.E1.m1.1.1.1.1.3.cmml" xref="S2.E1.m1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.1.3a.cmml" xref="S2.E1.m1.1.1.1.1.2.3">formulae-sequence</csymbol><apply id="S2.E1.m1.1.1.1.1.1.1.cmml" xref="S2.E1.m1.1.1.1.1.1.1"><eq id="S2.E1.m1.1.1.1.1.1.1.1.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1"></eq><apply id="S2.E1.m1.1.1.1.1.1.1.2.cmml" xref="S2.E1.m1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.1.1.1.2.1.cmml" xref="S2.E1.m1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S2.E1.m1.1.1.1.1.1.1.2.2.cmml" xref="S2.E1.m1.1.1.1.1.1.1.2.2">𝐪</ci><ci id="S2.E1.m1.1.1.1.1.1.1.2.3.cmml" xref="S2.E1.m1.1.1.1.1.1.1.2.3">𝑖</ci></apply><apply id="S2.E1.m1.1.1.1.1.1.1.3.cmml" xref="S2.E1.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.1.1.1.3.1.cmml" xref="S2.E1.m1.1.1.1.1.1.1.3">superscript</csymbol><apply id="S2.E1.m1.1.1.1.1.1.1.3.2.cmml" xref="S2.E1.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.1.1.1.3.2.1.cmml" xref="S2.E1.m1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.E1.m1.1.1.1.1.1.1.3.2.2.cmml" xref="S2.E1.m1.1.1.1.1.1.1.3.2.2">𝐱𝐖</ci><ci id="S2.E1.m1.1.1.1.1.1.1.3.2.3.cmml" xref="S2.E1.m1.1.1.1.1.1.1.3.2.3">𝑖</ci></apply><ci id="S2.E1.m1.1.1.1.1.1.1.3.3.cmml" xref="S2.E1.m1.1.1.1.1.1.1.3.3">𝑞</ci></apply></apply><apply id="S2.E1.m1.1.1.1.1.2.2.3.cmml" xref="S2.E1.m1.1.1.1.1.2.2.2"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.1.2.2.3a.cmml" xref="S2.E1.m1.1.1.1.1.2.2.2.3">formulae-sequence</csymbol><apply id="S2.E1.m1.1.1.1.1.2.2.1.1.cmml" xref="S2.E1.m1.1.1.1.1.2.2.1.1"><eq id="S2.E1.m1.1.1.1.1.2.2.1.1.1.cmml" xref="S2.E1.m1.1.1.1.1.2.2.1.1.1"></eq><apply id="S2.E1.m1.1.1.1.1.2.2.1.1.2.cmml" xref="S2.E1.m1.1.1.1.1.2.2.1.1.2"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.1.2.2.1.1.2.1.cmml" xref="S2.E1.m1.1.1.1.1.2.2.1.1.2">subscript</csymbol><ci id="S2.E1.m1.1.1.1.1.2.2.1.1.2.2.cmml" xref="S2.E1.m1.1.1.1.1.2.2.1.1.2.2">𝐤</ci><ci id="S2.E1.m1.1.1.1.1.2.2.1.1.2.3.cmml" xref="S2.E1.m1.1.1.1.1.2.2.1.1.2.3">𝑖</ci></apply><apply id="S2.E1.m1.1.1.1.1.2.2.1.1.3.cmml" xref="S2.E1.m1.1.1.1.1.2.2.1.1.3"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.1.2.2.1.1.3.1.cmml" xref="S2.E1.m1.1.1.1.1.2.2.1.1.3">subscript</csymbol><apply id="S2.E1.m1.1.1.1.1.2.2.1.1.3.2.cmml" xref="S2.E1.m1.1.1.1.1.2.2.1.1.3"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.1.2.2.1.1.3.2.1.cmml" xref="S2.E1.m1.1.1.1.1.2.2.1.1.3">superscript</csymbol><ci id="S2.E1.m1.1.1.1.1.2.2.1.1.3.2.2.cmml" xref="S2.E1.m1.1.1.1.1.2.2.1.1.3.2.2">𝐱𝐖</ci><ci id="S2.E1.m1.1.1.1.1.2.2.1.1.3.2.3.cmml" xref="S2.E1.m1.1.1.1.1.2.2.1.1.3.2.3">𝑘</ci></apply><ci id="S2.E1.m1.1.1.1.1.2.2.1.1.3.3.cmml" xref="S2.E1.m1.1.1.1.1.2.2.1.1.3.3">𝑖</ci></apply></apply><apply id="S2.E1.m1.1.1.1.1.2.2.2.2.cmml" xref="S2.E1.m1.1.1.1.1.2.2.2.2"><eq id="S2.E1.m1.1.1.1.1.2.2.2.2.1.cmml" xref="S2.E1.m1.1.1.1.1.2.2.2.2.1"></eq><apply id="S2.E1.m1.1.1.1.1.2.2.2.2.2.cmml" xref="S2.E1.m1.1.1.1.1.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.1.2.2.2.2.2.1.cmml" xref="S2.E1.m1.1.1.1.1.2.2.2.2.2">subscript</csymbol><ci id="S2.E1.m1.1.1.1.1.2.2.2.2.2.2.cmml" xref="S2.E1.m1.1.1.1.1.2.2.2.2.2.2">𝐯</ci><ci id="S2.E1.m1.1.1.1.1.2.2.2.2.2.3.cmml" xref="S2.E1.m1.1.1.1.1.2.2.2.2.2.3">𝑖</ci></apply><apply id="S2.E1.m1.1.1.1.1.2.2.2.2.3.cmml" xref="S2.E1.m1.1.1.1.1.2.2.2.2.3"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.1.2.2.2.2.3.1.cmml" xref="S2.E1.m1.1.1.1.1.2.2.2.2.3">subscript</csymbol><apply id="S2.E1.m1.1.1.1.1.2.2.2.2.3.2.cmml" xref="S2.E1.m1.1.1.1.1.2.2.2.2.3"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.1.2.2.2.2.3.2.1.cmml" xref="S2.E1.m1.1.1.1.1.2.2.2.2.3">superscript</csymbol><ci id="S2.E1.m1.1.1.1.1.2.2.2.2.3.2.2.cmml" xref="S2.E1.m1.1.1.1.1.2.2.2.2.3.2.2">𝐱𝐖</ci><ci id="S2.E1.m1.1.1.1.1.2.2.2.2.3.2.3.cmml" xref="S2.E1.m1.1.1.1.1.2.2.2.2.3.2.3">𝑣</ci></apply><ci id="S2.E1.m1.1.1.1.1.2.2.2.2.3.3.cmml" xref="S2.E1.m1.1.1.1.1.2.2.2.2.3.3">𝑖</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E1.m1.1c">\mathbf{q}_{i}=\mathbf{x}\mathbf{W}_{i}^{q},\hskip 5.0pt\mathbf{k}_{i}=\mathbf{x}\mathbf{W}^{k}_{i},\hskip 5.0pt\mathbf{v}_{i}=\mathbf{x}\mathbf{W}^{v}_{i}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
</div>
<div id="S2.SS1.p3" class="ltx_para">
<p id="S2.SS1.p3.5" class="ltx_p">Here, <math id="S2.SS1.p3.1.m1.1" class="ltx_Math" alttext="\mathbf{k}_{i}" display="inline"><semantics id="S2.SS1.p3.1.m1.1a"><msub id="S2.SS1.p3.1.m1.1.1" xref="S2.SS1.p3.1.m1.1.1.cmml"><mi id="S2.SS1.p3.1.m1.1.1.2" xref="S2.SS1.p3.1.m1.1.1.2.cmml">𝐤</mi><mi id="S2.SS1.p3.1.m1.1.1.3" xref="S2.SS1.p3.1.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.1.m1.1b"><apply id="S2.SS1.p3.1.m1.1.1.cmml" xref="S2.SS1.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS1.p3.1.m1.1.1.1.cmml" xref="S2.SS1.p3.1.m1.1.1">subscript</csymbol><ci id="S2.SS1.p3.1.m1.1.1.2.cmml" xref="S2.SS1.p3.1.m1.1.1.2">𝐤</ci><ci id="S2.SS1.p3.1.m1.1.1.3.cmml" xref="S2.SS1.p3.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.1.m1.1c">\mathbf{k}_{i}</annotation></semantics></math> and <math id="S2.SS1.p3.2.m2.1" class="ltx_Math" alttext="\mathbf{v}_{i}" display="inline"><semantics id="S2.SS1.p3.2.m2.1a"><msub id="S2.SS1.p3.2.m2.1.1" xref="S2.SS1.p3.2.m2.1.1.cmml"><mi id="S2.SS1.p3.2.m2.1.1.2" xref="S2.SS1.p3.2.m2.1.1.2.cmml">𝐯</mi><mi id="S2.SS1.p3.2.m2.1.1.3" xref="S2.SS1.p3.2.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.2.m2.1b"><apply id="S2.SS1.p3.2.m2.1.1.cmml" xref="S2.SS1.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS1.p3.2.m2.1.1.1.cmml" xref="S2.SS1.p3.2.m2.1.1">subscript</csymbol><ci id="S2.SS1.p3.2.m2.1.1.2.cmml" xref="S2.SS1.p3.2.m2.1.1.2">𝐯</ci><ci id="S2.SS1.p3.2.m2.1.1.3.cmml" xref="S2.SS1.p3.2.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.2.m2.1c">\mathbf{v}_{i}</annotation></semantics></math> represent the key and value at time step <math id="S2.SS1.p3.3.m3.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S2.SS1.p3.3.m3.1a"><mi id="S2.SS1.p3.3.m3.1.1" xref="S2.SS1.p3.3.m3.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.3.m3.1b"><ci id="S2.SS1.p3.3.m3.1.1.cmml" xref="S2.SS1.p3.3.m3.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.3.m3.1c">t</annotation></semantics></math> for head <math id="S2.SS1.p3.4.m4.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S2.SS1.p3.4.m4.1a"><mi id="S2.SS1.p3.4.m4.1.1" xref="S2.SS1.p3.4.m4.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.4.m4.1b"><ci id="S2.SS1.p3.4.m4.1.1.cmml" xref="S2.SS1.p3.4.m4.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.4.m4.1c">i</annotation></semantics></math>. We can then compute the attention score for each head <math id="S2.SS1.p3.5.m5.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S2.SS1.p3.5.m5.1a"><mi id="S2.SS1.p3.5.m5.1.1" xref="S2.SS1.p3.5.m5.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.5.m5.1b"><ci id="S2.SS1.p3.5.m5.1.1.cmml" xref="S2.SS1.p3.5.m5.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.5.m5.1c">i</annotation></semantics></math> and the corresponding attention output as</p>
<table id="S2.E2" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="S2.E2X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S2.E2X.2.1.1.m1.4" class="ltx_Math" alttext="\displaystyle\mathbf{p}_{t,i}=\text{Softmax}\left(\frac{{\mathbf{q}_{i}}\mathbf{K}_{i}^{T}}{\sqrt{d_{h}}}\right),\hskip 5.0pt\mathbf{a}_{i}=\mathbf{p}_{i}\mathbf{V}_{i}," display="inline"><semantics id="S2.E2X.2.1.1.m1.4a"><mrow id="S2.E2X.2.1.1.m1.4.4.1"><mrow id="S2.E2X.2.1.1.m1.4.4.1.1.2" xref="S2.E2X.2.1.1.m1.4.4.1.1.3.cmml"><mrow id="S2.E2X.2.1.1.m1.4.4.1.1.1.1" xref="S2.E2X.2.1.1.m1.4.4.1.1.1.1.cmml"><msub id="S2.E2X.2.1.1.m1.4.4.1.1.1.1.2" xref="S2.E2X.2.1.1.m1.4.4.1.1.1.1.2.cmml"><mi id="S2.E2X.2.1.1.m1.4.4.1.1.1.1.2.2" xref="S2.E2X.2.1.1.m1.4.4.1.1.1.1.2.2.cmml">𝐩</mi><mrow id="S2.E2X.2.1.1.m1.2.2.2.4" xref="S2.E2X.2.1.1.m1.2.2.2.3.cmml"><mi id="S2.E2X.2.1.1.m1.1.1.1.1" xref="S2.E2X.2.1.1.m1.1.1.1.1.cmml">t</mi><mo id="S2.E2X.2.1.1.m1.2.2.2.4.1" xref="S2.E2X.2.1.1.m1.2.2.2.3.cmml">,</mo><mi id="S2.E2X.2.1.1.m1.2.2.2.2" xref="S2.E2X.2.1.1.m1.2.2.2.2.cmml">i</mi></mrow></msub><mo id="S2.E2X.2.1.1.m1.4.4.1.1.1.1.1" xref="S2.E2X.2.1.1.m1.4.4.1.1.1.1.1.cmml">=</mo><mrow id="S2.E2X.2.1.1.m1.4.4.1.1.1.1.3" xref="S2.E2X.2.1.1.m1.4.4.1.1.1.1.3.cmml"><mtext id="S2.E2X.2.1.1.m1.4.4.1.1.1.1.3.2" xref="S2.E2X.2.1.1.m1.4.4.1.1.1.1.3.2a.cmml">Softmax</mtext><mo lspace="0em" rspace="0em" id="S2.E2X.2.1.1.m1.4.4.1.1.1.1.3.1" xref="S2.E2X.2.1.1.m1.4.4.1.1.1.1.3.1.cmml">​</mo><mrow id="S2.E2X.2.1.1.m1.4.4.1.1.1.1.3.3.2" xref="S2.E2X.2.1.1.m1.3.3.cmml"><mo id="S2.E2X.2.1.1.m1.4.4.1.1.1.1.3.3.2.1" xref="S2.E2X.2.1.1.m1.3.3.cmml">(</mo><mstyle displaystyle="true" id="S2.E2X.2.1.1.m1.3.3" xref="S2.E2X.2.1.1.m1.3.3.cmml"><mfrac id="S2.E2X.2.1.1.m1.3.3a" xref="S2.E2X.2.1.1.m1.3.3.cmml"><mrow id="S2.E2X.2.1.1.m1.3.3.2" xref="S2.E2X.2.1.1.m1.3.3.2.cmml"><msub id="S2.E2X.2.1.1.m1.3.3.2.2" xref="S2.E2X.2.1.1.m1.3.3.2.2.cmml"><mi id="S2.E2X.2.1.1.m1.3.3.2.2.2" xref="S2.E2X.2.1.1.m1.3.3.2.2.2.cmml">𝐪</mi><mi id="S2.E2X.2.1.1.m1.3.3.2.2.3" xref="S2.E2X.2.1.1.m1.3.3.2.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S2.E2X.2.1.1.m1.3.3.2.1" xref="S2.E2X.2.1.1.m1.3.3.2.1.cmml">​</mo><msubsup id="S2.E2X.2.1.1.m1.3.3.2.3" xref="S2.E2X.2.1.1.m1.3.3.2.3.cmml"><mi id="S2.E2X.2.1.1.m1.3.3.2.3.2.2" xref="S2.E2X.2.1.1.m1.3.3.2.3.2.2.cmml">𝐊</mi><mi id="S2.E2X.2.1.1.m1.3.3.2.3.2.3" xref="S2.E2X.2.1.1.m1.3.3.2.3.2.3.cmml">i</mi><mi id="S2.E2X.2.1.1.m1.3.3.2.3.3" xref="S2.E2X.2.1.1.m1.3.3.2.3.3.cmml">T</mi></msubsup></mrow><msqrt id="S2.E2X.2.1.1.m1.3.3.3" xref="S2.E2X.2.1.1.m1.3.3.3.cmml"><msub id="S2.E2X.2.1.1.m1.3.3.3.2" xref="S2.E2X.2.1.1.m1.3.3.3.2.cmml"><mi id="S2.E2X.2.1.1.m1.3.3.3.2.2" xref="S2.E2X.2.1.1.m1.3.3.3.2.2.cmml">d</mi><mi id="S2.E2X.2.1.1.m1.3.3.3.2.3" xref="S2.E2X.2.1.1.m1.3.3.3.2.3.cmml">h</mi></msub></msqrt></mfrac></mstyle><mo id="S2.E2X.2.1.1.m1.4.4.1.1.1.1.3.3.2.2" xref="S2.E2X.2.1.1.m1.3.3.cmml">)</mo></mrow></mrow></mrow><mo rspace="0.667em" id="S2.E2X.2.1.1.m1.4.4.1.1.2.3" xref="S2.E2X.2.1.1.m1.4.4.1.1.3a.cmml">,</mo><mrow id="S2.E2X.2.1.1.m1.4.4.1.1.2.2" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2.cmml"><msub id="S2.E2X.2.1.1.m1.4.4.1.1.2.2.2" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2.2.cmml"><mi id="S2.E2X.2.1.1.m1.4.4.1.1.2.2.2.2" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2.2.2.cmml">𝐚</mi><mi id="S2.E2X.2.1.1.m1.4.4.1.1.2.2.2.3" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2.2.3.cmml">i</mi></msub><mo id="S2.E2X.2.1.1.m1.4.4.1.1.2.2.1" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2.1.cmml">=</mo><mrow id="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.cmml"><msub id="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.2" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.2.cmml"><mi id="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.2.2" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.2.2.cmml">𝐩</mi><mi id="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.2.3" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.1" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.1.cmml">​</mo><msub id="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.3" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.3.cmml"><mi id="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.3.2" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.3.2.cmml">𝐕</mi><mi id="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.3.3" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.3.3.cmml">i</mi></msub></mrow></mrow></mrow><mo id="S2.E2X.2.1.1.m1.4.4.1.2">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E2X.2.1.1.m1.4b"><apply id="S2.E2X.2.1.1.m1.4.4.1.1.3.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.2"><csymbol cd="ambiguous" id="S2.E2X.2.1.1.m1.4.4.1.1.3a.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.3">formulae-sequence</csymbol><apply id="S2.E2X.2.1.1.m1.4.4.1.1.1.1.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.1.1"><eq id="S2.E2X.2.1.1.m1.4.4.1.1.1.1.1.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.1.1.1"></eq><apply id="S2.E2X.2.1.1.m1.4.4.1.1.1.1.2.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E2X.2.1.1.m1.4.4.1.1.1.1.2.1.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.1.1.2">subscript</csymbol><ci id="S2.E2X.2.1.1.m1.4.4.1.1.1.1.2.2.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.1.1.2.2">𝐩</ci><list id="S2.E2X.2.1.1.m1.2.2.2.3.cmml" xref="S2.E2X.2.1.1.m1.2.2.2.4"><ci id="S2.E2X.2.1.1.m1.1.1.1.1.cmml" xref="S2.E2X.2.1.1.m1.1.1.1.1">𝑡</ci><ci id="S2.E2X.2.1.1.m1.2.2.2.2.cmml" xref="S2.E2X.2.1.1.m1.2.2.2.2">𝑖</ci></list></apply><apply id="S2.E2X.2.1.1.m1.4.4.1.1.1.1.3.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.1.1.3"><times id="S2.E2X.2.1.1.m1.4.4.1.1.1.1.3.1.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.1.1.3.1"></times><ci id="S2.E2X.2.1.1.m1.4.4.1.1.1.1.3.2a.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.1.1.3.2"><mtext id="S2.E2X.2.1.1.m1.4.4.1.1.1.1.3.2.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.1.1.3.2">Softmax</mtext></ci><apply id="S2.E2X.2.1.1.m1.3.3.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.1.1.3.3.2"><divide id="S2.E2X.2.1.1.m1.3.3.1.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.1.1.3.3.2"></divide><apply id="S2.E2X.2.1.1.m1.3.3.2.cmml" xref="S2.E2X.2.1.1.m1.3.3.2"><times id="S2.E2X.2.1.1.m1.3.3.2.1.cmml" xref="S2.E2X.2.1.1.m1.3.3.2.1"></times><apply id="S2.E2X.2.1.1.m1.3.3.2.2.cmml" xref="S2.E2X.2.1.1.m1.3.3.2.2"><csymbol cd="ambiguous" id="S2.E2X.2.1.1.m1.3.3.2.2.1.cmml" xref="S2.E2X.2.1.1.m1.3.3.2.2">subscript</csymbol><ci id="S2.E2X.2.1.1.m1.3.3.2.2.2.cmml" xref="S2.E2X.2.1.1.m1.3.3.2.2.2">𝐪</ci><ci id="S2.E2X.2.1.1.m1.3.3.2.2.3.cmml" xref="S2.E2X.2.1.1.m1.3.3.2.2.3">𝑖</ci></apply><apply id="S2.E2X.2.1.1.m1.3.3.2.3.cmml" xref="S2.E2X.2.1.1.m1.3.3.2.3"><csymbol cd="ambiguous" id="S2.E2X.2.1.1.m1.3.3.2.3.1.cmml" xref="S2.E2X.2.1.1.m1.3.3.2.3">superscript</csymbol><apply id="S2.E2X.2.1.1.m1.3.3.2.3.2.cmml" xref="S2.E2X.2.1.1.m1.3.3.2.3"><csymbol cd="ambiguous" id="S2.E2X.2.1.1.m1.3.3.2.3.2.1.cmml" xref="S2.E2X.2.1.1.m1.3.3.2.3">subscript</csymbol><ci id="S2.E2X.2.1.1.m1.3.3.2.3.2.2.cmml" xref="S2.E2X.2.1.1.m1.3.3.2.3.2.2">𝐊</ci><ci id="S2.E2X.2.1.1.m1.3.3.2.3.2.3.cmml" xref="S2.E2X.2.1.1.m1.3.3.2.3.2.3">𝑖</ci></apply><ci id="S2.E2X.2.1.1.m1.3.3.2.3.3.cmml" xref="S2.E2X.2.1.1.m1.3.3.2.3.3">𝑇</ci></apply></apply><apply id="S2.E2X.2.1.1.m1.3.3.3.cmml" xref="S2.E2X.2.1.1.m1.3.3.3"><root id="S2.E2X.2.1.1.m1.3.3.3a.cmml" xref="S2.E2X.2.1.1.m1.3.3.3"></root><apply id="S2.E2X.2.1.1.m1.3.3.3.2.cmml" xref="S2.E2X.2.1.1.m1.3.3.3.2"><csymbol cd="ambiguous" id="S2.E2X.2.1.1.m1.3.3.3.2.1.cmml" xref="S2.E2X.2.1.1.m1.3.3.3.2">subscript</csymbol><ci id="S2.E2X.2.1.1.m1.3.3.3.2.2.cmml" xref="S2.E2X.2.1.1.m1.3.3.3.2.2">𝑑</ci><ci id="S2.E2X.2.1.1.m1.3.3.3.2.3.cmml" xref="S2.E2X.2.1.1.m1.3.3.3.2.3">ℎ</ci></apply></apply></apply></apply></apply><apply id="S2.E2X.2.1.1.m1.4.4.1.1.2.2.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2"><eq id="S2.E2X.2.1.1.m1.4.4.1.1.2.2.1.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2.1"></eq><apply id="S2.E2X.2.1.1.m1.4.4.1.1.2.2.2.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2.2"><csymbol cd="ambiguous" id="S2.E2X.2.1.1.m1.4.4.1.1.2.2.2.1.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2.2">subscript</csymbol><ci id="S2.E2X.2.1.1.m1.4.4.1.1.2.2.2.2.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2.2.2">𝐚</ci><ci id="S2.E2X.2.1.1.m1.4.4.1.1.2.2.2.3.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2.2.3">𝑖</ci></apply><apply id="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3"><times id="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.1.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.1"></times><apply id="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.2.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.2"><csymbol cd="ambiguous" id="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.2.1.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.2">subscript</csymbol><ci id="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.2.2.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.2.2">𝐩</ci><ci id="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.2.3.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.2.3">𝑖</ci></apply><apply id="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.3.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.3"><csymbol cd="ambiguous" id="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.3.1.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.3">subscript</csymbol><ci id="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.3.2.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.3.2">𝐕</ci><ci id="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.3.3.cmml" xref="S2.E2X.2.1.1.m1.4.4.1.1.2.2.3.3.3">𝑖</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E2X.2.1.1.m1.4c">\displaystyle\mathbf{p}_{t,i}=\text{Softmax}\left(\frac{{\mathbf{q}_{i}}\mathbf{K}_{i}^{T}}{\sqrt{d_{h}}}\right),\hskip 5.0pt\mathbf{a}_{i}=\mathbf{p}_{i}\mathbf{V}_{i},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(2)</span></td>
</tr>
</tbody>
</table>
<p id="S2.SS1.p3.9" class="ltx_p">where <math id="S2.SS1.p3.6.m1.1" class="ltx_Math" alttext="\mathbf{K}_{i}" display="inline"><semantics id="S2.SS1.p3.6.m1.1a"><msub id="S2.SS1.p3.6.m1.1.1" xref="S2.SS1.p3.6.m1.1.1.cmml"><mi id="S2.SS1.p3.6.m1.1.1.2" xref="S2.SS1.p3.6.m1.1.1.2.cmml">𝐊</mi><mi id="S2.SS1.p3.6.m1.1.1.3" xref="S2.SS1.p3.6.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.6.m1.1b"><apply id="S2.SS1.p3.6.m1.1.1.cmml" xref="S2.SS1.p3.6.m1.1.1"><csymbol cd="ambiguous" id="S2.SS1.p3.6.m1.1.1.1.cmml" xref="S2.SS1.p3.6.m1.1.1">subscript</csymbol><ci id="S2.SS1.p3.6.m1.1.1.2.cmml" xref="S2.SS1.p3.6.m1.1.1.2">𝐊</ci><ci id="S2.SS1.p3.6.m1.1.1.3.cmml" xref="S2.SS1.p3.6.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.6.m1.1c">\mathbf{K}_{i}</annotation></semantics></math> and <math id="S2.SS1.p3.7.m2.1" class="ltx_Math" alttext="\mathbf{V}_{i}" display="inline"><semantics id="S2.SS1.p3.7.m2.1a"><msub id="S2.SS1.p3.7.m2.1.1" xref="S2.SS1.p3.7.m2.1.1.cmml"><mi id="S2.SS1.p3.7.m2.1.1.2" xref="S2.SS1.p3.7.m2.1.1.2.cmml">𝐕</mi><mi id="S2.SS1.p3.7.m2.1.1.3" xref="S2.SS1.p3.7.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.7.m2.1b"><apply id="S2.SS1.p3.7.m2.1.1.cmml" xref="S2.SS1.p3.7.m2.1.1"><csymbol cd="ambiguous" id="S2.SS1.p3.7.m2.1.1.1.cmml" xref="S2.SS1.p3.7.m2.1.1">subscript</csymbol><ci id="S2.SS1.p3.7.m2.1.1.2.cmml" xref="S2.SS1.p3.7.m2.1.1.2">𝐕</ci><ci id="S2.SS1.p3.7.m2.1.1.3.cmml" xref="S2.SS1.p3.7.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.7.m2.1c">\mathbf{V}_{i}</annotation></semantics></math> denote the concatenation of current and all previous keys and values corresponding to the <math id="S2.SS1.p3.8.m3.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S2.SS1.p3.8.m3.1a"><mi id="S2.SS1.p3.8.m3.1.1" xref="S2.SS1.p3.8.m3.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.8.m3.1b"><ci id="S2.SS1.p3.8.m3.1.1.cmml" xref="S2.SS1.p3.8.m3.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.8.m3.1c">i</annotation></semantics></math>-th head.
The final MHA output is obtained by concatenating the outputs of all heads and then applying the out-projection layer <math id="S2.SS1.p3.9.m4.1" class="ltx_Math" alttext="\mathbf{W}_{o}" display="inline"><semantics id="S2.SS1.p3.9.m4.1a"><msub id="S2.SS1.p3.9.m4.1.1" xref="S2.SS1.p3.9.m4.1.1.cmml"><mi id="S2.SS1.p3.9.m4.1.1.2" xref="S2.SS1.p3.9.m4.1.1.2.cmml">𝐖</mi><mi id="S2.SS1.p3.9.m4.1.1.3" xref="S2.SS1.p3.9.m4.1.1.3.cmml">o</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.9.m4.1b"><apply id="S2.SS1.p3.9.m4.1.1.cmml" xref="S2.SS1.p3.9.m4.1.1"><csymbol cd="ambiguous" id="S2.SS1.p3.9.m4.1.1.1.cmml" xref="S2.SS1.p3.9.m4.1.1">subscript</csymbol><ci id="S2.SS1.p3.9.m4.1.1.2.cmml" xref="S2.SS1.p3.9.m4.1.1.2">𝐖</ci><ci id="S2.SS1.p3.9.m4.1.1.3.cmml" xref="S2.SS1.p3.9.m4.1.1.3">𝑜</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.9.m4.1c">\mathbf{W}_{o}</annotation></semantics></math>, as shown by</p>
</div>
<div id="S2.SS1.p4" class="ltx_para">
<table id="S2.E3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E3.m1.2" class="ltx_Math" alttext="\text{MHA}(\mathbf{x})=\sum_{i=1}^{h}\mathbf{a}_{i}\mathbf{W}^{o}_{i}=\sum_{i=1}^{h}(\mathbf{p}_{i}\mathbf{V}_{i})\mathbf{W}^{o}_{i}," display="block"><semantics id="S2.E3.m1.2a"><mrow id="S2.E3.m1.2.2.1" xref="S2.E3.m1.2.2.1.1.cmml"><mrow id="S2.E3.m1.2.2.1.1" xref="S2.E3.m1.2.2.1.1.cmml"><mrow id="S2.E3.m1.2.2.1.1.3" xref="S2.E3.m1.2.2.1.1.3.cmml"><mtext id="S2.E3.m1.2.2.1.1.3.2" xref="S2.E3.m1.2.2.1.1.3.2a.cmml">MHA</mtext><mo lspace="0em" rspace="0em" id="S2.E3.m1.2.2.1.1.3.1" xref="S2.E3.m1.2.2.1.1.3.1.cmml">​</mo><mrow id="S2.E3.m1.2.2.1.1.3.3.2" xref="S2.E3.m1.2.2.1.1.3.cmml"><mo stretchy="false" id="S2.E3.m1.2.2.1.1.3.3.2.1" xref="S2.E3.m1.2.2.1.1.3.cmml">(</mo><mi id="S2.E3.m1.1.1" xref="S2.E3.m1.1.1.cmml">𝐱</mi><mo stretchy="false" id="S2.E3.m1.2.2.1.1.3.3.2.2" xref="S2.E3.m1.2.2.1.1.3.cmml">)</mo></mrow></mrow><mo rspace="0.111em" id="S2.E3.m1.2.2.1.1.4" xref="S2.E3.m1.2.2.1.1.4.cmml">=</mo><mrow id="S2.E3.m1.2.2.1.1.5" xref="S2.E3.m1.2.2.1.1.5.cmml"><munderover id="S2.E3.m1.2.2.1.1.5.1" xref="S2.E3.m1.2.2.1.1.5.1.cmml"><mo movablelimits="false" id="S2.E3.m1.2.2.1.1.5.1.2.2" xref="S2.E3.m1.2.2.1.1.5.1.2.2.cmml">∑</mo><mrow id="S2.E3.m1.2.2.1.1.5.1.2.3" xref="S2.E3.m1.2.2.1.1.5.1.2.3.cmml"><mi id="S2.E3.m1.2.2.1.1.5.1.2.3.2" xref="S2.E3.m1.2.2.1.1.5.1.2.3.2.cmml">i</mi><mo id="S2.E3.m1.2.2.1.1.5.1.2.3.1" xref="S2.E3.m1.2.2.1.1.5.1.2.3.1.cmml">=</mo><mn id="S2.E3.m1.2.2.1.1.5.1.2.3.3" xref="S2.E3.m1.2.2.1.1.5.1.2.3.3.cmml">1</mn></mrow><mi id="S2.E3.m1.2.2.1.1.5.1.3" xref="S2.E3.m1.2.2.1.1.5.1.3.cmml">h</mi></munderover><mrow id="S2.E3.m1.2.2.1.1.5.2" xref="S2.E3.m1.2.2.1.1.5.2.cmml"><msub id="S2.E3.m1.2.2.1.1.5.2.2" xref="S2.E3.m1.2.2.1.1.5.2.2.cmml"><mi id="S2.E3.m1.2.2.1.1.5.2.2.2" xref="S2.E3.m1.2.2.1.1.5.2.2.2.cmml">𝐚</mi><mi id="S2.E3.m1.2.2.1.1.5.2.2.3" xref="S2.E3.m1.2.2.1.1.5.2.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S2.E3.m1.2.2.1.1.5.2.1" xref="S2.E3.m1.2.2.1.1.5.2.1.cmml">​</mo><msubsup id="S2.E3.m1.2.2.1.1.5.2.3" xref="S2.E3.m1.2.2.1.1.5.2.3.cmml"><mi id="S2.E3.m1.2.2.1.1.5.2.3.2.2" xref="S2.E3.m1.2.2.1.1.5.2.3.2.2.cmml">𝐖</mi><mi id="S2.E3.m1.2.2.1.1.5.2.3.3" xref="S2.E3.m1.2.2.1.1.5.2.3.3.cmml">i</mi><mi id="S2.E3.m1.2.2.1.1.5.2.3.2.3" xref="S2.E3.m1.2.2.1.1.5.2.3.2.3.cmml">o</mi></msubsup></mrow></mrow><mo rspace="0.111em" id="S2.E3.m1.2.2.1.1.6" xref="S2.E3.m1.2.2.1.1.6.cmml">=</mo><mrow id="S2.E3.m1.2.2.1.1.1" xref="S2.E3.m1.2.2.1.1.1.cmml"><munderover id="S2.E3.m1.2.2.1.1.1.2" xref="S2.E3.m1.2.2.1.1.1.2.cmml"><mo movablelimits="false" rspace="0em" id="S2.E3.m1.2.2.1.1.1.2.2.2" xref="S2.E3.m1.2.2.1.1.1.2.2.2.cmml">∑</mo><mrow id="S2.E3.m1.2.2.1.1.1.2.2.3" xref="S2.E3.m1.2.2.1.1.1.2.2.3.cmml"><mi id="S2.E3.m1.2.2.1.1.1.2.2.3.2" xref="S2.E3.m1.2.2.1.1.1.2.2.3.2.cmml">i</mi><mo id="S2.E3.m1.2.2.1.1.1.2.2.3.1" xref="S2.E3.m1.2.2.1.1.1.2.2.3.1.cmml">=</mo><mn id="S2.E3.m1.2.2.1.1.1.2.2.3.3" xref="S2.E3.m1.2.2.1.1.1.2.2.3.3.cmml">1</mn></mrow><mi id="S2.E3.m1.2.2.1.1.1.2.3" xref="S2.E3.m1.2.2.1.1.1.2.3.cmml">h</mi></munderover><mrow id="S2.E3.m1.2.2.1.1.1.1" xref="S2.E3.m1.2.2.1.1.1.1.cmml"><mrow id="S2.E3.m1.2.2.1.1.1.1.1.1" xref="S2.E3.m1.2.2.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.E3.m1.2.2.1.1.1.1.1.1.2" xref="S2.E3.m1.2.2.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.E3.m1.2.2.1.1.1.1.1.1.1" xref="S2.E3.m1.2.2.1.1.1.1.1.1.1.cmml"><msub id="S2.E3.m1.2.2.1.1.1.1.1.1.1.2" xref="S2.E3.m1.2.2.1.1.1.1.1.1.1.2.cmml"><mi id="S2.E3.m1.2.2.1.1.1.1.1.1.1.2.2" xref="S2.E3.m1.2.2.1.1.1.1.1.1.1.2.2.cmml">𝐩</mi><mi id="S2.E3.m1.2.2.1.1.1.1.1.1.1.2.3" xref="S2.E3.m1.2.2.1.1.1.1.1.1.1.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S2.E3.m1.2.2.1.1.1.1.1.1.1.1" xref="S2.E3.m1.2.2.1.1.1.1.1.1.1.1.cmml">​</mo><msub id="S2.E3.m1.2.2.1.1.1.1.1.1.1.3" xref="S2.E3.m1.2.2.1.1.1.1.1.1.1.3.cmml"><mi id="S2.E3.m1.2.2.1.1.1.1.1.1.1.3.2" xref="S2.E3.m1.2.2.1.1.1.1.1.1.1.3.2.cmml">𝐕</mi><mi id="S2.E3.m1.2.2.1.1.1.1.1.1.1.3.3" xref="S2.E3.m1.2.2.1.1.1.1.1.1.1.3.3.cmml">i</mi></msub></mrow><mo stretchy="false" id="S2.E3.m1.2.2.1.1.1.1.1.1.3" xref="S2.E3.m1.2.2.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S2.E3.m1.2.2.1.1.1.1.2" xref="S2.E3.m1.2.2.1.1.1.1.2.cmml">​</mo><msubsup id="S2.E3.m1.2.2.1.1.1.1.3" xref="S2.E3.m1.2.2.1.1.1.1.3.cmml"><mi id="S2.E3.m1.2.2.1.1.1.1.3.2.2" xref="S2.E3.m1.2.2.1.1.1.1.3.2.2.cmml">𝐖</mi><mi id="S2.E3.m1.2.2.1.1.1.1.3.3" xref="S2.E3.m1.2.2.1.1.1.1.3.3.cmml">i</mi><mi id="S2.E3.m1.2.2.1.1.1.1.3.2.3" xref="S2.E3.m1.2.2.1.1.1.1.3.2.3.cmml">o</mi></msubsup></mrow></mrow></mrow><mo id="S2.E3.m1.2.2.1.2" xref="S2.E3.m1.2.2.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E3.m1.2b"><apply id="S2.E3.m1.2.2.1.1.cmml" xref="S2.E3.m1.2.2.1"><and id="S2.E3.m1.2.2.1.1a.cmml" xref="S2.E3.m1.2.2.1"></and><apply id="S2.E3.m1.2.2.1.1b.cmml" xref="S2.E3.m1.2.2.1"><eq id="S2.E3.m1.2.2.1.1.4.cmml" xref="S2.E3.m1.2.2.1.1.4"></eq><apply id="S2.E3.m1.2.2.1.1.3.cmml" xref="S2.E3.m1.2.2.1.1.3"><times id="S2.E3.m1.2.2.1.1.3.1.cmml" xref="S2.E3.m1.2.2.1.1.3.1"></times><ci id="S2.E3.m1.2.2.1.1.3.2a.cmml" xref="S2.E3.m1.2.2.1.1.3.2"><mtext id="S2.E3.m1.2.2.1.1.3.2.cmml" xref="S2.E3.m1.2.2.1.1.3.2">MHA</mtext></ci><ci id="S2.E3.m1.1.1.cmml" xref="S2.E3.m1.1.1">𝐱</ci></apply><apply id="S2.E3.m1.2.2.1.1.5.cmml" xref="S2.E3.m1.2.2.1.1.5"><apply id="S2.E3.m1.2.2.1.1.5.1.cmml" xref="S2.E3.m1.2.2.1.1.5.1"><csymbol cd="ambiguous" id="S2.E3.m1.2.2.1.1.5.1.1.cmml" xref="S2.E3.m1.2.2.1.1.5.1">superscript</csymbol><apply id="S2.E3.m1.2.2.1.1.5.1.2.cmml" xref="S2.E3.m1.2.2.1.1.5.1"><csymbol cd="ambiguous" id="S2.E3.m1.2.2.1.1.5.1.2.1.cmml" xref="S2.E3.m1.2.2.1.1.5.1">subscript</csymbol><sum id="S2.E3.m1.2.2.1.1.5.1.2.2.cmml" xref="S2.E3.m1.2.2.1.1.5.1.2.2"></sum><apply id="S2.E3.m1.2.2.1.1.5.1.2.3.cmml" xref="S2.E3.m1.2.2.1.1.5.1.2.3"><eq id="S2.E3.m1.2.2.1.1.5.1.2.3.1.cmml" xref="S2.E3.m1.2.2.1.1.5.1.2.3.1"></eq><ci id="S2.E3.m1.2.2.1.1.5.1.2.3.2.cmml" xref="S2.E3.m1.2.2.1.1.5.1.2.3.2">𝑖</ci><cn type="integer" id="S2.E3.m1.2.2.1.1.5.1.2.3.3.cmml" xref="S2.E3.m1.2.2.1.1.5.1.2.3.3">1</cn></apply></apply><ci id="S2.E3.m1.2.2.1.1.5.1.3.cmml" xref="S2.E3.m1.2.2.1.1.5.1.3">ℎ</ci></apply><apply id="S2.E3.m1.2.2.1.1.5.2.cmml" xref="S2.E3.m1.2.2.1.1.5.2"><times id="S2.E3.m1.2.2.1.1.5.2.1.cmml" xref="S2.E3.m1.2.2.1.1.5.2.1"></times><apply id="S2.E3.m1.2.2.1.1.5.2.2.cmml" xref="S2.E3.m1.2.2.1.1.5.2.2"><csymbol cd="ambiguous" id="S2.E3.m1.2.2.1.1.5.2.2.1.cmml" xref="S2.E3.m1.2.2.1.1.5.2.2">subscript</csymbol><ci id="S2.E3.m1.2.2.1.1.5.2.2.2.cmml" xref="S2.E3.m1.2.2.1.1.5.2.2.2">𝐚</ci><ci id="S2.E3.m1.2.2.1.1.5.2.2.3.cmml" xref="S2.E3.m1.2.2.1.1.5.2.2.3">𝑖</ci></apply><apply id="S2.E3.m1.2.2.1.1.5.2.3.cmml" xref="S2.E3.m1.2.2.1.1.5.2.3"><csymbol cd="ambiguous" id="S2.E3.m1.2.2.1.1.5.2.3.1.cmml" xref="S2.E3.m1.2.2.1.1.5.2.3">subscript</csymbol><apply id="S2.E3.m1.2.2.1.1.5.2.3.2.cmml" xref="S2.E3.m1.2.2.1.1.5.2.3"><csymbol cd="ambiguous" id="S2.E3.m1.2.2.1.1.5.2.3.2.1.cmml" xref="S2.E3.m1.2.2.1.1.5.2.3">superscript</csymbol><ci id="S2.E3.m1.2.2.1.1.5.2.3.2.2.cmml" xref="S2.E3.m1.2.2.1.1.5.2.3.2.2">𝐖</ci><ci id="S2.E3.m1.2.2.1.1.5.2.3.2.3.cmml" xref="S2.E3.m1.2.2.1.1.5.2.3.2.3">𝑜</ci></apply><ci id="S2.E3.m1.2.2.1.1.5.2.3.3.cmml" xref="S2.E3.m1.2.2.1.1.5.2.3.3">𝑖</ci></apply></apply></apply></apply><apply id="S2.E3.m1.2.2.1.1c.cmml" xref="S2.E3.m1.2.2.1"><eq id="S2.E3.m1.2.2.1.1.6.cmml" xref="S2.E3.m1.2.2.1.1.6"></eq><share href="#S2.E3.m1.2.2.1.1.5.cmml" id="S2.E3.m1.2.2.1.1d.cmml" xref="S2.E3.m1.2.2.1"></share><apply id="S2.E3.m1.2.2.1.1.1.cmml" xref="S2.E3.m1.2.2.1.1.1"><apply id="S2.E3.m1.2.2.1.1.1.2.cmml" xref="S2.E3.m1.2.2.1.1.1.2"><csymbol cd="ambiguous" id="S2.E3.m1.2.2.1.1.1.2.1.cmml" xref="S2.E3.m1.2.2.1.1.1.2">superscript</csymbol><apply id="S2.E3.m1.2.2.1.1.1.2.2.cmml" xref="S2.E3.m1.2.2.1.1.1.2"><csymbol cd="ambiguous" id="S2.E3.m1.2.2.1.1.1.2.2.1.cmml" xref="S2.E3.m1.2.2.1.1.1.2">subscript</csymbol><sum id="S2.E3.m1.2.2.1.1.1.2.2.2.cmml" xref="S2.E3.m1.2.2.1.1.1.2.2.2"></sum><apply id="S2.E3.m1.2.2.1.1.1.2.2.3.cmml" xref="S2.E3.m1.2.2.1.1.1.2.2.3"><eq id="S2.E3.m1.2.2.1.1.1.2.2.3.1.cmml" xref="S2.E3.m1.2.2.1.1.1.2.2.3.1"></eq><ci id="S2.E3.m1.2.2.1.1.1.2.2.3.2.cmml" xref="S2.E3.m1.2.2.1.1.1.2.2.3.2">𝑖</ci><cn type="integer" id="S2.E3.m1.2.2.1.1.1.2.2.3.3.cmml" xref="S2.E3.m1.2.2.1.1.1.2.2.3.3">1</cn></apply></apply><ci id="S2.E3.m1.2.2.1.1.1.2.3.cmml" xref="S2.E3.m1.2.2.1.1.1.2.3">ℎ</ci></apply><apply id="S2.E3.m1.2.2.1.1.1.1.cmml" xref="S2.E3.m1.2.2.1.1.1.1"><times id="S2.E3.m1.2.2.1.1.1.1.2.cmml" xref="S2.E3.m1.2.2.1.1.1.1.2"></times><apply id="S2.E3.m1.2.2.1.1.1.1.1.1.1.cmml" xref="S2.E3.m1.2.2.1.1.1.1.1.1"><times id="S2.E3.m1.2.2.1.1.1.1.1.1.1.1.cmml" xref="S2.E3.m1.2.2.1.1.1.1.1.1.1.1"></times><apply id="S2.E3.m1.2.2.1.1.1.1.1.1.1.2.cmml" xref="S2.E3.m1.2.2.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E3.m1.2.2.1.1.1.1.1.1.1.2.1.cmml" xref="S2.E3.m1.2.2.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S2.E3.m1.2.2.1.1.1.1.1.1.1.2.2.cmml" xref="S2.E3.m1.2.2.1.1.1.1.1.1.1.2.2">𝐩</ci><ci id="S2.E3.m1.2.2.1.1.1.1.1.1.1.2.3.cmml" xref="S2.E3.m1.2.2.1.1.1.1.1.1.1.2.3">𝑖</ci></apply><apply id="S2.E3.m1.2.2.1.1.1.1.1.1.1.3.cmml" xref="S2.E3.m1.2.2.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E3.m1.2.2.1.1.1.1.1.1.1.3.1.cmml" xref="S2.E3.m1.2.2.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.E3.m1.2.2.1.1.1.1.1.1.1.3.2.cmml" xref="S2.E3.m1.2.2.1.1.1.1.1.1.1.3.2">𝐕</ci><ci id="S2.E3.m1.2.2.1.1.1.1.1.1.1.3.3.cmml" xref="S2.E3.m1.2.2.1.1.1.1.1.1.1.3.3">𝑖</ci></apply></apply><apply id="S2.E3.m1.2.2.1.1.1.1.3.cmml" xref="S2.E3.m1.2.2.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E3.m1.2.2.1.1.1.1.3.1.cmml" xref="S2.E3.m1.2.2.1.1.1.1.3">subscript</csymbol><apply id="S2.E3.m1.2.2.1.1.1.1.3.2.cmml" xref="S2.E3.m1.2.2.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E3.m1.2.2.1.1.1.1.3.2.1.cmml" xref="S2.E3.m1.2.2.1.1.1.1.3">superscript</csymbol><ci id="S2.E3.m1.2.2.1.1.1.1.3.2.2.cmml" xref="S2.E3.m1.2.2.1.1.1.1.3.2.2">𝐖</ci><ci id="S2.E3.m1.2.2.1.1.1.1.3.2.3.cmml" xref="S2.E3.m1.2.2.1.1.1.1.3.2.3">𝑜</ci></apply><ci id="S2.E3.m1.2.2.1.1.1.1.3.3.cmml" xref="S2.E3.m1.2.2.1.1.1.1.3.3">𝑖</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E3.m1.2c">\text{MHA}(\mathbf{x})=\sum_{i=1}^{h}\mathbf{a}_{i}\mathbf{W}^{o}_{i}=\sum_{i=1}^{h}(\mathbf{p}_{i}\mathbf{V}_{i})\mathbf{W}^{o}_{i},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
</table>
<p id="S2.SS1.p4.2" class="ltx_p">where <math id="S2.SS1.p4.1.m1.1" class="ltx_Math" alttext="\mathbf{W}^{o}_{i}\in\mathbb{R}^{d_{h}\times d}" display="inline"><semantics id="S2.SS1.p4.1.m1.1a"><mrow id="S2.SS1.p4.1.m1.1.1" xref="S2.SS1.p4.1.m1.1.1.cmml"><msubsup id="S2.SS1.p4.1.m1.1.1.2" xref="S2.SS1.p4.1.m1.1.1.2.cmml"><mi id="S2.SS1.p4.1.m1.1.1.2.2.2" xref="S2.SS1.p4.1.m1.1.1.2.2.2.cmml">𝐖</mi><mi id="S2.SS1.p4.1.m1.1.1.2.3" xref="S2.SS1.p4.1.m1.1.1.2.3.cmml">i</mi><mi id="S2.SS1.p4.1.m1.1.1.2.2.3" xref="S2.SS1.p4.1.m1.1.1.2.2.3.cmml">o</mi></msubsup><mo id="S2.SS1.p4.1.m1.1.1.1" xref="S2.SS1.p4.1.m1.1.1.1.cmml">∈</mo><msup id="S2.SS1.p4.1.m1.1.1.3" xref="S2.SS1.p4.1.m1.1.1.3.cmml"><mi id="S2.SS1.p4.1.m1.1.1.3.2" xref="S2.SS1.p4.1.m1.1.1.3.2.cmml">ℝ</mi><mrow id="S2.SS1.p4.1.m1.1.1.3.3" xref="S2.SS1.p4.1.m1.1.1.3.3.cmml"><msub id="S2.SS1.p4.1.m1.1.1.3.3.2" xref="S2.SS1.p4.1.m1.1.1.3.3.2.cmml"><mi id="S2.SS1.p4.1.m1.1.1.3.3.2.2" xref="S2.SS1.p4.1.m1.1.1.3.3.2.2.cmml">d</mi><mi id="S2.SS1.p4.1.m1.1.1.3.3.2.3" xref="S2.SS1.p4.1.m1.1.1.3.3.2.3.cmml">h</mi></msub><mo lspace="0.222em" rspace="0.222em" id="S2.SS1.p4.1.m1.1.1.3.3.1" xref="S2.SS1.p4.1.m1.1.1.3.3.1.cmml">×</mo><mi id="S2.SS1.p4.1.m1.1.1.3.3.3" xref="S2.SS1.p4.1.m1.1.1.3.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.1.m1.1b"><apply id="S2.SS1.p4.1.m1.1.1.cmml" xref="S2.SS1.p4.1.m1.1.1"><in id="S2.SS1.p4.1.m1.1.1.1.cmml" xref="S2.SS1.p4.1.m1.1.1.1"></in><apply id="S2.SS1.p4.1.m1.1.1.2.cmml" xref="S2.SS1.p4.1.m1.1.1.2"><csymbol cd="ambiguous" id="S2.SS1.p4.1.m1.1.1.2.1.cmml" xref="S2.SS1.p4.1.m1.1.1.2">subscript</csymbol><apply id="S2.SS1.p4.1.m1.1.1.2.2.cmml" xref="S2.SS1.p4.1.m1.1.1.2"><csymbol cd="ambiguous" id="S2.SS1.p4.1.m1.1.1.2.2.1.cmml" xref="S2.SS1.p4.1.m1.1.1.2">superscript</csymbol><ci id="S2.SS1.p4.1.m1.1.1.2.2.2.cmml" xref="S2.SS1.p4.1.m1.1.1.2.2.2">𝐖</ci><ci id="S2.SS1.p4.1.m1.1.1.2.2.3.cmml" xref="S2.SS1.p4.1.m1.1.1.2.2.3">𝑜</ci></apply><ci id="S2.SS1.p4.1.m1.1.1.2.3.cmml" xref="S2.SS1.p4.1.m1.1.1.2.3">𝑖</ci></apply><apply id="S2.SS1.p4.1.m1.1.1.3.cmml" xref="S2.SS1.p4.1.m1.1.1.3"><csymbol cd="ambiguous" id="S2.SS1.p4.1.m1.1.1.3.1.cmml" xref="S2.SS1.p4.1.m1.1.1.3">superscript</csymbol><ci id="S2.SS1.p4.1.m1.1.1.3.2.cmml" xref="S2.SS1.p4.1.m1.1.1.3.2">ℝ</ci><apply id="S2.SS1.p4.1.m1.1.1.3.3.cmml" xref="S2.SS1.p4.1.m1.1.1.3.3"><times id="S2.SS1.p4.1.m1.1.1.3.3.1.cmml" xref="S2.SS1.p4.1.m1.1.1.3.3.1"></times><apply id="S2.SS1.p4.1.m1.1.1.3.3.2.cmml" xref="S2.SS1.p4.1.m1.1.1.3.3.2"><csymbol cd="ambiguous" id="S2.SS1.p4.1.m1.1.1.3.3.2.1.cmml" xref="S2.SS1.p4.1.m1.1.1.3.3.2">subscript</csymbol><ci id="S2.SS1.p4.1.m1.1.1.3.3.2.2.cmml" xref="S2.SS1.p4.1.m1.1.1.3.3.2.2">𝑑</ci><ci id="S2.SS1.p4.1.m1.1.1.3.3.2.3.cmml" xref="S2.SS1.p4.1.m1.1.1.3.3.2.3">ℎ</ci></apply><ci id="S2.SS1.p4.1.m1.1.1.3.3.3.cmml" xref="S2.SS1.p4.1.m1.1.1.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.1.m1.1c">\mathbf{W}^{o}_{i}\in\mathbb{R}^{d_{h}\times d}</annotation></semantics></math> represents the submatrices of the out-projection matrix for each head <math id="S2.SS1.p4.2.m2.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S2.SS1.p4.2.m2.1a"><mi id="S2.SS1.p4.2.m2.1.1" xref="S2.SS1.p4.2.m2.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.2.m2.1b"><ci id="S2.SS1.p4.2.m2.1.1.cmml" xref="S2.SS1.p4.2.m2.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.2.m2.1c">i</annotation></semantics></math>.</p>
</div>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>Singular Value Decomposition (SVD)</h3>

<div id="S2.SS2.p1" class="ltx_para">
<p id="S2.SS2.p1.1" class="ltx_p">SVD <cite class="ltx_cite ltx_citemacro_citep">(Jolliffe and Cadima, <a href="#bib.bib20" title="" class="ltx_ref">2016</a>)</cite> is a commonly used technique for computing the low-rank approximation for a given matrix.
We now introduce our specific use of SVD for <span id="S2.SS2.p1.1.1" class="ltx_text ltx_font_italic">Palu</span>’s default low-rank decomposition method.</p>
</div>
<div id="S2.SS2.p2" class="ltx_para">
<p id="S2.SS2.p2.6" class="ltx_p">Given a weight matrix <math id="S2.SS2.p2.1.m1.1" class="ltx_Math" alttext="\mathbf{W}\in\mathbb{R}^{m\times n}" display="inline"><semantics id="S2.SS2.p2.1.m1.1a"><mrow id="S2.SS2.p2.1.m1.1.1" xref="S2.SS2.p2.1.m1.1.1.cmml"><mi id="S2.SS2.p2.1.m1.1.1.2" xref="S2.SS2.p2.1.m1.1.1.2.cmml">𝐖</mi><mo id="S2.SS2.p2.1.m1.1.1.1" xref="S2.SS2.p2.1.m1.1.1.1.cmml">∈</mo><msup id="S2.SS2.p2.1.m1.1.1.3" xref="S2.SS2.p2.1.m1.1.1.3.cmml"><mi id="S2.SS2.p2.1.m1.1.1.3.2" xref="S2.SS2.p2.1.m1.1.1.3.2.cmml">ℝ</mi><mrow id="S2.SS2.p2.1.m1.1.1.3.3" xref="S2.SS2.p2.1.m1.1.1.3.3.cmml"><mi id="S2.SS2.p2.1.m1.1.1.3.3.2" xref="S2.SS2.p2.1.m1.1.1.3.3.2.cmml">m</mi><mo lspace="0.222em" rspace="0.222em" id="S2.SS2.p2.1.m1.1.1.3.3.1" xref="S2.SS2.p2.1.m1.1.1.3.3.1.cmml">×</mo><mi id="S2.SS2.p2.1.m1.1.1.3.3.3" xref="S2.SS2.p2.1.m1.1.1.3.3.3.cmml">n</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.1.m1.1b"><apply id="S2.SS2.p2.1.m1.1.1.cmml" xref="S2.SS2.p2.1.m1.1.1"><in id="S2.SS2.p2.1.m1.1.1.1.cmml" xref="S2.SS2.p2.1.m1.1.1.1"></in><ci id="S2.SS2.p2.1.m1.1.1.2.cmml" xref="S2.SS2.p2.1.m1.1.1.2">𝐖</ci><apply id="S2.SS2.p2.1.m1.1.1.3.cmml" xref="S2.SS2.p2.1.m1.1.1.3"><csymbol cd="ambiguous" id="S2.SS2.p2.1.m1.1.1.3.1.cmml" xref="S2.SS2.p2.1.m1.1.1.3">superscript</csymbol><ci id="S2.SS2.p2.1.m1.1.1.3.2.cmml" xref="S2.SS2.p2.1.m1.1.1.3.2">ℝ</ci><apply id="S2.SS2.p2.1.m1.1.1.3.3.cmml" xref="S2.SS2.p2.1.m1.1.1.3.3"><times id="S2.SS2.p2.1.m1.1.1.3.3.1.cmml" xref="S2.SS2.p2.1.m1.1.1.3.3.1"></times><ci id="S2.SS2.p2.1.m1.1.1.3.3.2.cmml" xref="S2.SS2.p2.1.m1.1.1.3.3.2">𝑚</ci><ci id="S2.SS2.p2.1.m1.1.1.3.3.3.cmml" xref="S2.SS2.p2.1.m1.1.1.3.3.3">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.1.m1.1c">\mathbf{W}\in\mathbb{R}^{m\times n}</annotation></semantics></math>, SVD decomposes <math id="S2.SS2.p2.2.m2.1" class="ltx_Math" alttext="\mathbf{W}" display="inline"><semantics id="S2.SS2.p2.2.m2.1a"><mi id="S2.SS2.p2.2.m2.1.1" xref="S2.SS2.p2.2.m2.1.1.cmml">𝐖</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.2.m2.1b"><ci id="S2.SS2.p2.2.m2.1.1.cmml" xref="S2.SS2.p2.2.m2.1.1">𝐖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.2.m2.1c">\mathbf{W}</annotation></semantics></math> into three matrices: <math id="S2.SS2.p2.3.m3.1" class="ltx_Math" alttext="\mathbf{W}=\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{T}" display="inline"><semantics id="S2.SS2.p2.3.m3.1a"><mrow id="S2.SS2.p2.3.m3.1.1" xref="S2.SS2.p2.3.m3.1.1.cmml"><mi id="S2.SS2.p2.3.m3.1.1.2" xref="S2.SS2.p2.3.m3.1.1.2.cmml">𝐖</mi><mo id="S2.SS2.p2.3.m3.1.1.1" xref="S2.SS2.p2.3.m3.1.1.1.cmml">=</mo><mrow id="S2.SS2.p2.3.m3.1.1.3" xref="S2.SS2.p2.3.m3.1.1.3.cmml"><mi id="S2.SS2.p2.3.m3.1.1.3.2" xref="S2.SS2.p2.3.m3.1.1.3.2.cmml">𝐔</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p2.3.m3.1.1.3.1" xref="S2.SS2.p2.3.m3.1.1.3.1.cmml">​</mo><mi id="S2.SS2.p2.3.m3.1.1.3.3" xref="S2.SS2.p2.3.m3.1.1.3.3.cmml">𝚺</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p2.3.m3.1.1.3.1a" xref="S2.SS2.p2.3.m3.1.1.3.1.cmml">​</mo><msup id="S2.SS2.p2.3.m3.1.1.3.4" xref="S2.SS2.p2.3.m3.1.1.3.4.cmml"><mi id="S2.SS2.p2.3.m3.1.1.3.4.2" xref="S2.SS2.p2.3.m3.1.1.3.4.2.cmml">𝐕</mi><mi id="S2.SS2.p2.3.m3.1.1.3.4.3" xref="S2.SS2.p2.3.m3.1.1.3.4.3.cmml">T</mi></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.3.m3.1b"><apply id="S2.SS2.p2.3.m3.1.1.cmml" xref="S2.SS2.p2.3.m3.1.1"><eq id="S2.SS2.p2.3.m3.1.1.1.cmml" xref="S2.SS2.p2.3.m3.1.1.1"></eq><ci id="S2.SS2.p2.3.m3.1.1.2.cmml" xref="S2.SS2.p2.3.m3.1.1.2">𝐖</ci><apply id="S2.SS2.p2.3.m3.1.1.3.cmml" xref="S2.SS2.p2.3.m3.1.1.3"><times id="S2.SS2.p2.3.m3.1.1.3.1.cmml" xref="S2.SS2.p2.3.m3.1.1.3.1"></times><ci id="S2.SS2.p2.3.m3.1.1.3.2.cmml" xref="S2.SS2.p2.3.m3.1.1.3.2">𝐔</ci><ci id="S2.SS2.p2.3.m3.1.1.3.3.cmml" xref="S2.SS2.p2.3.m3.1.1.3.3">𝚺</ci><apply id="S2.SS2.p2.3.m3.1.1.3.4.cmml" xref="S2.SS2.p2.3.m3.1.1.3.4"><csymbol cd="ambiguous" id="S2.SS2.p2.3.m3.1.1.3.4.1.cmml" xref="S2.SS2.p2.3.m3.1.1.3.4">superscript</csymbol><ci id="S2.SS2.p2.3.m3.1.1.3.4.2.cmml" xref="S2.SS2.p2.3.m3.1.1.3.4.2">𝐕</ci><ci id="S2.SS2.p2.3.m3.1.1.3.4.3.cmml" xref="S2.SS2.p2.3.m3.1.1.3.4.3">𝑇</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.3.m3.1c">\mathbf{W}=\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{T}</annotation></semantics></math>. Here, <math id="S2.SS2.p2.4.m4.1" class="ltx_Math" alttext="\mathbf{U}" display="inline"><semantics id="S2.SS2.p2.4.m4.1a"><mi id="S2.SS2.p2.4.m4.1.1" xref="S2.SS2.p2.4.m4.1.1.cmml">𝐔</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.4.m4.1b"><ci id="S2.SS2.p2.4.m4.1.1.cmml" xref="S2.SS2.p2.4.m4.1.1">𝐔</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.4.m4.1c">\mathbf{U}</annotation></semantics></math> and <math id="S2.SS2.p2.5.m5.1" class="ltx_Math" alttext="\mathbf{V}" display="inline"><semantics id="S2.SS2.p2.5.m5.1a"><mi id="S2.SS2.p2.5.m5.1.1" xref="S2.SS2.p2.5.m5.1.1.cmml">𝐕</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.5.m5.1b"><ci id="S2.SS2.p2.5.m5.1.1.cmml" xref="S2.SS2.p2.5.m5.1.1">𝐕</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.5.m5.1c">\mathbf{V}</annotation></semantics></math> are orthogonal matrices containing the left and right singular vectors, respectively. The matrix <math id="S2.SS2.p2.6.m6.1" class="ltx_Math" alttext="\boldsymbol{\Sigma}" display="inline"><semantics id="S2.SS2.p2.6.m6.1a"><mi id="S2.SS2.p2.6.m6.1.1" xref="S2.SS2.p2.6.m6.1.1.cmml">𝚺</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.6.m6.1b"><ci id="S2.SS2.p2.6.m6.1.1.cmml" xref="S2.SS2.p2.6.m6.1.1">𝚺</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.6.m6.1c">\boldsymbol{\Sigma}</annotation></semantics></math> is a diagonal matrix that consists of singular values.
We describe the decomposition as</p>
</div>
<div id="S2.SS2.p3" class="ltx_para">
<table id="S2.Ex1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.Ex1.m1.1" class="ltx_Math" alttext="\mathbf{W}\approx\mathbf{A}\mathbf{B},\hskip 5.0pt\mathbf{A}=\mathbf{U}_{r}\sqrt{\boldsymbol{\Sigma}_{r}},\hskip 5.0pt\mathbf{B}=\sqrt{\boldsymbol{\Sigma}_{r}}\mathbf{V}_{r}^{T}," display="block"><semantics id="S2.Ex1.m1.1a"><mrow id="S2.Ex1.m1.1.1.1"><mrow id="S2.Ex1.m1.1.1.1.1.2" xref="S2.Ex1.m1.1.1.1.1.3.cmml"><mrow id="S2.Ex1.m1.1.1.1.1.1.1" xref="S2.Ex1.m1.1.1.1.1.1.1.cmml"><mi id="S2.Ex1.m1.1.1.1.1.1.1.2" xref="S2.Ex1.m1.1.1.1.1.1.1.2.cmml">𝐖</mi><mo id="S2.Ex1.m1.1.1.1.1.1.1.1" xref="S2.Ex1.m1.1.1.1.1.1.1.1.cmml">≈</mo><mi id="S2.Ex1.m1.1.1.1.1.1.1.3" xref="S2.Ex1.m1.1.1.1.1.1.1.3.cmml">𝐀𝐁</mi></mrow><mo rspace="0.667em" id="S2.Ex1.m1.1.1.1.1.2.3" xref="S2.Ex1.m1.1.1.1.1.3a.cmml">,</mo><mrow id="S2.Ex1.m1.1.1.1.1.2.2.2" xref="S2.Ex1.m1.1.1.1.1.2.2.3.cmml"><mrow id="S2.Ex1.m1.1.1.1.1.2.2.1.1" xref="S2.Ex1.m1.1.1.1.1.2.2.1.1.cmml"><mi id="S2.Ex1.m1.1.1.1.1.2.2.1.1.2" xref="S2.Ex1.m1.1.1.1.1.2.2.1.1.2.cmml">𝐀</mi><mo id="S2.Ex1.m1.1.1.1.1.2.2.1.1.1" xref="S2.Ex1.m1.1.1.1.1.2.2.1.1.1.cmml">=</mo><mrow id="S2.Ex1.m1.1.1.1.1.2.2.1.1.3" xref="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.cmml"><msub id="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.2" xref="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.2.cmml"><mi id="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.2.2" xref="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.2.2.cmml">𝐔</mi><mi id="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.2.3" xref="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.2.3.cmml">r</mi></msub><mo lspace="0em" rspace="0em" id="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.1" xref="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.1.cmml">​</mo><msqrt id="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.3" xref="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.3.cmml"><msub id="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.3.2" xref="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.3.2.cmml"><mi id="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.3.2.2" xref="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.3.2.2.cmml">𝚺</mi><mi id="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.3.2.3" xref="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.3.2.3.cmml">r</mi></msub></msqrt></mrow></mrow><mo rspace="0.667em" id="S2.Ex1.m1.1.1.1.1.2.2.2.3" xref="S2.Ex1.m1.1.1.1.1.2.2.3a.cmml">,</mo><mrow id="S2.Ex1.m1.1.1.1.1.2.2.2.2" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.cmml"><mi id="S2.Ex1.m1.1.1.1.1.2.2.2.2.2" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.2.cmml">𝐁</mi><mo id="S2.Ex1.m1.1.1.1.1.2.2.2.2.1" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.1.cmml">=</mo><mrow id="S2.Ex1.m1.1.1.1.1.2.2.2.2.3" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.cmml"><msqrt id="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.2" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.2.cmml"><msub id="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.2.2" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.2.2.cmml"><mi id="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.2.2.2" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.2.2.2.cmml">𝚺</mi><mi id="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.2.2.3" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.2.2.3.cmml">r</mi></msub></msqrt><mo lspace="0em" rspace="0em" id="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.1" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.1.cmml">​</mo><msubsup id="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.3" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.3.cmml"><mi id="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.3.2.2" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.3.2.2.cmml">𝐕</mi><mi id="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.3.2.3" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.3.2.3.cmml">r</mi><mi id="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.3.3" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.3.3.cmml">T</mi></msubsup></mrow></mrow></mrow></mrow><mo id="S2.Ex1.m1.1.1.1.2">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.Ex1.m1.1b"><apply id="S2.Ex1.m1.1.1.1.1.3.cmml" xref="S2.Ex1.m1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.Ex1.m1.1.1.1.1.3a.cmml" xref="S2.Ex1.m1.1.1.1.1.2.3">formulae-sequence</csymbol><apply id="S2.Ex1.m1.1.1.1.1.1.1.cmml" xref="S2.Ex1.m1.1.1.1.1.1.1"><approx id="S2.Ex1.m1.1.1.1.1.1.1.1.cmml" xref="S2.Ex1.m1.1.1.1.1.1.1.1"></approx><ci id="S2.Ex1.m1.1.1.1.1.1.1.2.cmml" xref="S2.Ex1.m1.1.1.1.1.1.1.2">𝐖</ci><ci id="S2.Ex1.m1.1.1.1.1.1.1.3.cmml" xref="S2.Ex1.m1.1.1.1.1.1.1.3">𝐀𝐁</ci></apply><apply id="S2.Ex1.m1.1.1.1.1.2.2.3.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.2"><csymbol cd="ambiguous" id="S2.Ex1.m1.1.1.1.1.2.2.3a.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.2.3">formulae-sequence</csymbol><apply id="S2.Ex1.m1.1.1.1.1.2.2.1.1.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.1.1"><eq id="S2.Ex1.m1.1.1.1.1.2.2.1.1.1.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.1.1.1"></eq><ci id="S2.Ex1.m1.1.1.1.1.2.2.1.1.2.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.1.1.2">𝐀</ci><apply id="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.1.1.3"><times id="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.1.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.1"></times><apply id="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.2.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.2"><csymbol cd="ambiguous" id="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.2.1.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.2">subscript</csymbol><ci id="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.2.2.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.2.2">𝐔</ci><ci id="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.2.3.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.2.3">𝑟</ci></apply><apply id="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.3.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.3"><root id="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.3a.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.3"></root><apply id="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.3.2.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.3.2"><csymbol cd="ambiguous" id="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.3.2.1.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.3.2">subscript</csymbol><ci id="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.3.2.2.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.3.2.2">𝚺</ci><ci id="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.3.2.3.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.1.1.3.3.2.3">𝑟</ci></apply></apply></apply></apply><apply id="S2.Ex1.m1.1.1.1.1.2.2.2.2.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2"><eq id="S2.Ex1.m1.1.1.1.1.2.2.2.2.1.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.1"></eq><ci id="S2.Ex1.m1.1.1.1.1.2.2.2.2.2.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.2">𝐁</ci><apply id="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.3"><times id="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.1.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.1"></times><apply id="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.2.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.2"><root id="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.2a.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.2"></root><apply id="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.2.2.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.2.2"><csymbol cd="ambiguous" id="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.2.2.1.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.2.2">subscript</csymbol><ci id="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.2.2.2.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.2.2.2">𝚺</ci><ci id="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.2.2.3.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.2.2.3">𝑟</ci></apply></apply><apply id="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.3.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.3"><csymbol cd="ambiguous" id="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.3.1.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.3">superscript</csymbol><apply id="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.3.2.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.3"><csymbol cd="ambiguous" id="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.3.2.1.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.3">subscript</csymbol><ci id="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.3.2.2.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.3.2.2">𝐕</ci><ci id="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.3.2.3.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.3.2.3">𝑟</ci></apply><ci id="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.3.3.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.3.3">𝑇</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.Ex1.m1.1c">\mathbf{W}\approx\mathbf{A}\mathbf{B},\hskip 5.0pt\mathbf{A}=\mathbf{U}_{r}\sqrt{\boldsymbol{\Sigma}_{r}},\hskip 5.0pt\mathbf{B}=\sqrt{\boldsymbol{\Sigma}_{r}}\mathbf{V}_{r}^{T},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S2.SS2.p3.13" class="ltx_p">where <math id="S2.SS2.p3.1.m1.1" class="ltx_Math" alttext="\mathbf{A}\in\mathbb{R}^{m\times r}" display="inline"><semantics id="S2.SS2.p3.1.m1.1a"><mrow id="S2.SS2.p3.1.m1.1.1" xref="S2.SS2.p3.1.m1.1.1.cmml"><mi id="S2.SS2.p3.1.m1.1.1.2" xref="S2.SS2.p3.1.m1.1.1.2.cmml">𝐀</mi><mo id="S2.SS2.p3.1.m1.1.1.1" xref="S2.SS2.p3.1.m1.1.1.1.cmml">∈</mo><msup id="S2.SS2.p3.1.m1.1.1.3" xref="S2.SS2.p3.1.m1.1.1.3.cmml"><mi id="S2.SS2.p3.1.m1.1.1.3.2" xref="S2.SS2.p3.1.m1.1.1.3.2.cmml">ℝ</mi><mrow id="S2.SS2.p3.1.m1.1.1.3.3" xref="S2.SS2.p3.1.m1.1.1.3.3.cmml"><mi id="S2.SS2.p3.1.m1.1.1.3.3.2" xref="S2.SS2.p3.1.m1.1.1.3.3.2.cmml">m</mi><mo lspace="0.222em" rspace="0.222em" id="S2.SS2.p3.1.m1.1.1.3.3.1" xref="S2.SS2.p3.1.m1.1.1.3.3.1.cmml">×</mo><mi id="S2.SS2.p3.1.m1.1.1.3.3.3" xref="S2.SS2.p3.1.m1.1.1.3.3.3.cmml">r</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.1.m1.1b"><apply id="S2.SS2.p3.1.m1.1.1.cmml" xref="S2.SS2.p3.1.m1.1.1"><in id="S2.SS2.p3.1.m1.1.1.1.cmml" xref="S2.SS2.p3.1.m1.1.1.1"></in><ci id="S2.SS2.p3.1.m1.1.1.2.cmml" xref="S2.SS2.p3.1.m1.1.1.2">𝐀</ci><apply id="S2.SS2.p3.1.m1.1.1.3.cmml" xref="S2.SS2.p3.1.m1.1.1.3"><csymbol cd="ambiguous" id="S2.SS2.p3.1.m1.1.1.3.1.cmml" xref="S2.SS2.p3.1.m1.1.1.3">superscript</csymbol><ci id="S2.SS2.p3.1.m1.1.1.3.2.cmml" xref="S2.SS2.p3.1.m1.1.1.3.2">ℝ</ci><apply id="S2.SS2.p3.1.m1.1.1.3.3.cmml" xref="S2.SS2.p3.1.m1.1.1.3.3"><times id="S2.SS2.p3.1.m1.1.1.3.3.1.cmml" xref="S2.SS2.p3.1.m1.1.1.3.3.1"></times><ci id="S2.SS2.p3.1.m1.1.1.3.3.2.cmml" xref="S2.SS2.p3.1.m1.1.1.3.3.2">𝑚</ci><ci id="S2.SS2.p3.1.m1.1.1.3.3.3.cmml" xref="S2.SS2.p3.1.m1.1.1.3.3.3">𝑟</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.1.m1.1c">\mathbf{A}\in\mathbb{R}^{m\times r}</annotation></semantics></math>, <math id="S2.SS2.p3.2.m2.1" class="ltx_Math" alttext="\mathbf{B}\in\mathbb{R}^{r\times n}" display="inline"><semantics id="S2.SS2.p3.2.m2.1a"><mrow id="S2.SS2.p3.2.m2.1.1" xref="S2.SS2.p3.2.m2.1.1.cmml"><mi id="S2.SS2.p3.2.m2.1.1.2" xref="S2.SS2.p3.2.m2.1.1.2.cmml">𝐁</mi><mo id="S2.SS2.p3.2.m2.1.1.1" xref="S2.SS2.p3.2.m2.1.1.1.cmml">∈</mo><msup id="S2.SS2.p3.2.m2.1.1.3" xref="S2.SS2.p3.2.m2.1.1.3.cmml"><mi id="S2.SS2.p3.2.m2.1.1.3.2" xref="S2.SS2.p3.2.m2.1.1.3.2.cmml">ℝ</mi><mrow id="S2.SS2.p3.2.m2.1.1.3.3" xref="S2.SS2.p3.2.m2.1.1.3.3.cmml"><mi id="S2.SS2.p3.2.m2.1.1.3.3.2" xref="S2.SS2.p3.2.m2.1.1.3.3.2.cmml">r</mi><mo lspace="0.222em" rspace="0.222em" id="S2.SS2.p3.2.m2.1.1.3.3.1" xref="S2.SS2.p3.2.m2.1.1.3.3.1.cmml">×</mo><mi id="S2.SS2.p3.2.m2.1.1.3.3.3" xref="S2.SS2.p3.2.m2.1.1.3.3.3.cmml">n</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.2.m2.1b"><apply id="S2.SS2.p3.2.m2.1.1.cmml" xref="S2.SS2.p3.2.m2.1.1"><in id="S2.SS2.p3.2.m2.1.1.1.cmml" xref="S2.SS2.p3.2.m2.1.1.1"></in><ci id="S2.SS2.p3.2.m2.1.1.2.cmml" xref="S2.SS2.p3.2.m2.1.1.2">𝐁</ci><apply id="S2.SS2.p3.2.m2.1.1.3.cmml" xref="S2.SS2.p3.2.m2.1.1.3"><csymbol cd="ambiguous" id="S2.SS2.p3.2.m2.1.1.3.1.cmml" xref="S2.SS2.p3.2.m2.1.1.3">superscript</csymbol><ci id="S2.SS2.p3.2.m2.1.1.3.2.cmml" xref="S2.SS2.p3.2.m2.1.1.3.2">ℝ</ci><apply id="S2.SS2.p3.2.m2.1.1.3.3.cmml" xref="S2.SS2.p3.2.m2.1.1.3.3"><times id="S2.SS2.p3.2.m2.1.1.3.3.1.cmml" xref="S2.SS2.p3.2.m2.1.1.3.3.1"></times><ci id="S2.SS2.p3.2.m2.1.1.3.3.2.cmml" xref="S2.SS2.p3.2.m2.1.1.3.3.2">𝑟</ci><ci id="S2.SS2.p3.2.m2.1.1.3.3.3.cmml" xref="S2.SS2.p3.2.m2.1.1.3.3.3">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.2.m2.1c">\mathbf{B}\in\mathbb{R}^{r\times n}</annotation></semantics></math>,
<math id="S2.SS2.p3.3.m3.1" class="ltx_Math" alttext="\boldsymbol{\Sigma}_{r}\in\mathbb{R}^{r\times r}" display="inline"><semantics id="S2.SS2.p3.3.m3.1a"><mrow id="S2.SS2.p3.3.m3.1.1" xref="S2.SS2.p3.3.m3.1.1.cmml"><msub id="S2.SS2.p3.3.m3.1.1.2" xref="S2.SS2.p3.3.m3.1.1.2.cmml"><mi id="S2.SS2.p3.3.m3.1.1.2.2" xref="S2.SS2.p3.3.m3.1.1.2.2.cmml">𝚺</mi><mi id="S2.SS2.p3.3.m3.1.1.2.3" xref="S2.SS2.p3.3.m3.1.1.2.3.cmml">r</mi></msub><mo id="S2.SS2.p3.3.m3.1.1.1" xref="S2.SS2.p3.3.m3.1.1.1.cmml">∈</mo><msup id="S2.SS2.p3.3.m3.1.1.3" xref="S2.SS2.p3.3.m3.1.1.3.cmml"><mi id="S2.SS2.p3.3.m3.1.1.3.2" xref="S2.SS2.p3.3.m3.1.1.3.2.cmml">ℝ</mi><mrow id="S2.SS2.p3.3.m3.1.1.3.3" xref="S2.SS2.p3.3.m3.1.1.3.3.cmml"><mi id="S2.SS2.p3.3.m3.1.1.3.3.2" xref="S2.SS2.p3.3.m3.1.1.3.3.2.cmml">r</mi><mo lspace="0.222em" rspace="0.222em" id="S2.SS2.p3.3.m3.1.1.3.3.1" xref="S2.SS2.p3.3.m3.1.1.3.3.1.cmml">×</mo><mi id="S2.SS2.p3.3.m3.1.1.3.3.3" xref="S2.SS2.p3.3.m3.1.1.3.3.3.cmml">r</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.3.m3.1b"><apply id="S2.SS2.p3.3.m3.1.1.cmml" xref="S2.SS2.p3.3.m3.1.1"><in id="S2.SS2.p3.3.m3.1.1.1.cmml" xref="S2.SS2.p3.3.m3.1.1.1"></in><apply id="S2.SS2.p3.3.m3.1.1.2.cmml" xref="S2.SS2.p3.3.m3.1.1.2"><csymbol cd="ambiguous" id="S2.SS2.p3.3.m3.1.1.2.1.cmml" xref="S2.SS2.p3.3.m3.1.1.2">subscript</csymbol><ci id="S2.SS2.p3.3.m3.1.1.2.2.cmml" xref="S2.SS2.p3.3.m3.1.1.2.2">𝚺</ci><ci id="S2.SS2.p3.3.m3.1.1.2.3.cmml" xref="S2.SS2.p3.3.m3.1.1.2.3">𝑟</ci></apply><apply id="S2.SS2.p3.3.m3.1.1.3.cmml" xref="S2.SS2.p3.3.m3.1.1.3"><csymbol cd="ambiguous" id="S2.SS2.p3.3.m3.1.1.3.1.cmml" xref="S2.SS2.p3.3.m3.1.1.3">superscript</csymbol><ci id="S2.SS2.p3.3.m3.1.1.3.2.cmml" xref="S2.SS2.p3.3.m3.1.1.3.2">ℝ</ci><apply id="S2.SS2.p3.3.m3.1.1.3.3.cmml" xref="S2.SS2.p3.3.m3.1.1.3.3"><times id="S2.SS2.p3.3.m3.1.1.3.3.1.cmml" xref="S2.SS2.p3.3.m3.1.1.3.3.1"></times><ci id="S2.SS2.p3.3.m3.1.1.3.3.2.cmml" xref="S2.SS2.p3.3.m3.1.1.3.3.2">𝑟</ci><ci id="S2.SS2.p3.3.m3.1.1.3.3.3.cmml" xref="S2.SS2.p3.3.m3.1.1.3.3.3">𝑟</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.3.m3.1c">\boldsymbol{\Sigma}_{r}\in\mathbb{R}^{r\times r}</annotation></semantics></math>. <math id="S2.SS2.p3.4.m4.1" class="ltx_Math" alttext="\boldsymbol{\Sigma}_{r}" display="inline"><semantics id="S2.SS2.p3.4.m4.1a"><msub id="S2.SS2.p3.4.m4.1.1" xref="S2.SS2.p3.4.m4.1.1.cmml"><mi id="S2.SS2.p3.4.m4.1.1.2" xref="S2.SS2.p3.4.m4.1.1.2.cmml">𝚺</mi><mi id="S2.SS2.p3.4.m4.1.1.3" xref="S2.SS2.p3.4.m4.1.1.3.cmml">r</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.4.m4.1b"><apply id="S2.SS2.p3.4.m4.1.1.cmml" xref="S2.SS2.p3.4.m4.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.4.m4.1.1.1.cmml" xref="S2.SS2.p3.4.m4.1.1">subscript</csymbol><ci id="S2.SS2.p3.4.m4.1.1.2.cmml" xref="S2.SS2.p3.4.m4.1.1.2">𝚺</ci><ci id="S2.SS2.p3.4.m4.1.1.3.cmml" xref="S2.SS2.p3.4.m4.1.1.3">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.4.m4.1c">\boldsymbol{\Sigma}_{r}</annotation></semantics></math> is a diagonal matrix containing the largest <math id="S2.SS2.p3.5.m5.1" class="ltx_Math" alttext="r" display="inline"><semantics id="S2.SS2.p3.5.m5.1a"><mi id="S2.SS2.p3.5.m5.1.1" xref="S2.SS2.p3.5.m5.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.5.m5.1b"><ci id="S2.SS2.p3.5.m5.1.1.cmml" xref="S2.SS2.p3.5.m5.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.5.m5.1c">r</annotation></semantics></math> singular values, and <math id="S2.SS2.p3.6.m6.1" class="ltx_Math" alttext="\mathbf{U}_{r}" display="inline"><semantics id="S2.SS2.p3.6.m6.1a"><msub id="S2.SS2.p3.6.m6.1.1" xref="S2.SS2.p3.6.m6.1.1.cmml"><mi id="S2.SS2.p3.6.m6.1.1.2" xref="S2.SS2.p3.6.m6.1.1.2.cmml">𝐔</mi><mi id="S2.SS2.p3.6.m6.1.1.3" xref="S2.SS2.p3.6.m6.1.1.3.cmml">r</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.6.m6.1b"><apply id="S2.SS2.p3.6.m6.1.1.cmml" xref="S2.SS2.p3.6.m6.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.6.m6.1.1.1.cmml" xref="S2.SS2.p3.6.m6.1.1">subscript</csymbol><ci id="S2.SS2.p3.6.m6.1.1.2.cmml" xref="S2.SS2.p3.6.m6.1.1.2">𝐔</ci><ci id="S2.SS2.p3.6.m6.1.1.3.cmml" xref="S2.SS2.p3.6.m6.1.1.3">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.6.m6.1c">\mathbf{U}_{r}</annotation></semantics></math>, <math id="S2.SS2.p3.7.m7.1" class="ltx_Math" alttext="\mathbf{V}_{r}^{T}" display="inline"><semantics id="S2.SS2.p3.7.m7.1a"><msubsup id="S2.SS2.p3.7.m7.1.1" xref="S2.SS2.p3.7.m7.1.1.cmml"><mi id="S2.SS2.p3.7.m7.1.1.2.2" xref="S2.SS2.p3.7.m7.1.1.2.2.cmml">𝐕</mi><mi id="S2.SS2.p3.7.m7.1.1.2.3" xref="S2.SS2.p3.7.m7.1.1.2.3.cmml">r</mi><mi id="S2.SS2.p3.7.m7.1.1.3" xref="S2.SS2.p3.7.m7.1.1.3.cmml">T</mi></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.7.m7.1b"><apply id="S2.SS2.p3.7.m7.1.1.cmml" xref="S2.SS2.p3.7.m7.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.7.m7.1.1.1.cmml" xref="S2.SS2.p3.7.m7.1.1">superscript</csymbol><apply id="S2.SS2.p3.7.m7.1.1.2.cmml" xref="S2.SS2.p3.7.m7.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.7.m7.1.1.2.1.cmml" xref="S2.SS2.p3.7.m7.1.1">subscript</csymbol><ci id="S2.SS2.p3.7.m7.1.1.2.2.cmml" xref="S2.SS2.p3.7.m7.1.1.2.2">𝐕</ci><ci id="S2.SS2.p3.7.m7.1.1.2.3.cmml" xref="S2.SS2.p3.7.m7.1.1.2.3">𝑟</ci></apply><ci id="S2.SS2.p3.7.m7.1.1.3.cmml" xref="S2.SS2.p3.7.m7.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.7.m7.1c">\mathbf{V}_{r}^{T}</annotation></semantics></math> are corresponding singular vectors truncated from <math id="S2.SS2.p3.8.m8.1" class="ltx_Math" alttext="\mathbf{U}" display="inline"><semantics id="S2.SS2.p3.8.m8.1a"><mi id="S2.SS2.p3.8.m8.1.1" xref="S2.SS2.p3.8.m8.1.1.cmml">𝐔</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.8.m8.1b"><ci id="S2.SS2.p3.8.m8.1.1.cmml" xref="S2.SS2.p3.8.m8.1.1">𝐔</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.8.m8.1c">\mathbf{U}</annotation></semantics></math> and <math id="S2.SS2.p3.9.m9.1" class="ltx_Math" alttext="\mathbf{V}^{T}" display="inline"><semantics id="S2.SS2.p3.9.m9.1a"><msup id="S2.SS2.p3.9.m9.1.1" xref="S2.SS2.p3.9.m9.1.1.cmml"><mi id="S2.SS2.p3.9.m9.1.1.2" xref="S2.SS2.p3.9.m9.1.1.2.cmml">𝐕</mi><mi id="S2.SS2.p3.9.m9.1.1.3" xref="S2.SS2.p3.9.m9.1.1.3.cmml">T</mi></msup><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.9.m9.1b"><apply id="S2.SS2.p3.9.m9.1.1.cmml" xref="S2.SS2.p3.9.m9.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.9.m9.1.1.1.cmml" xref="S2.SS2.p3.9.m9.1.1">superscript</csymbol><ci id="S2.SS2.p3.9.m9.1.1.2.cmml" xref="S2.SS2.p3.9.m9.1.1.2">𝐕</ci><ci id="S2.SS2.p3.9.m9.1.1.3.cmml" xref="S2.SS2.p3.9.m9.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.9.m9.1c">\mathbf{V}^{T}</annotation></semantics></math>. This truncation and subsequent matrix formation let us approximate weight matrix <math id="S2.SS2.p3.10.m10.1" class="ltx_Math" alttext="\mathbf{W}" display="inline"><semantics id="S2.SS2.p3.10.m10.1a"><mi id="S2.SS2.p3.10.m10.1.1" xref="S2.SS2.p3.10.m10.1.1.cmml">𝐖</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.10.m10.1b"><ci id="S2.SS2.p3.10.m10.1.1.cmml" xref="S2.SS2.p3.10.m10.1.1">𝐖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.10.m10.1c">\mathbf{W}</annotation></semantics></math> with two low-rank matrices <math id="S2.SS2.p3.11.m11.1" class="ltx_Math" alttext="\mathbf{A}" display="inline"><semantics id="S2.SS2.p3.11.m11.1a"><mi id="S2.SS2.p3.11.m11.1.1" xref="S2.SS2.p3.11.m11.1.1.cmml">𝐀</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.11.m11.1b"><ci id="S2.SS2.p3.11.m11.1.1.cmml" xref="S2.SS2.p3.11.m11.1.1">𝐀</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.11.m11.1c">\mathbf{A}</annotation></semantics></math> and <math id="S2.SS2.p3.12.m12.1" class="ltx_Math" alttext="\mathbf{B}" display="inline"><semantics id="S2.SS2.p3.12.m12.1a"><mi id="S2.SS2.p3.12.m12.1.1" xref="S2.SS2.p3.12.m12.1.1.cmml">𝐁</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.12.m12.1b"><ci id="S2.SS2.p3.12.m12.1.1.cmml" xref="S2.SS2.p3.12.m12.1.1">𝐁</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.12.m12.1c">\mathbf{B}</annotation></semantics></math>, thereby reducing the storage by <math id="S2.SS2.p3.13.m13.1" class="ltx_Math" alttext="\frac{mr+rn}{mn}" display="inline"><semantics id="S2.SS2.p3.13.m13.1a"><mfrac id="S2.SS2.p3.13.m13.1.1" xref="S2.SS2.p3.13.m13.1.1.cmml"><mrow id="S2.SS2.p3.13.m13.1.1.2" xref="S2.SS2.p3.13.m13.1.1.2.cmml"><mrow id="S2.SS2.p3.13.m13.1.1.2.2" xref="S2.SS2.p3.13.m13.1.1.2.2.cmml"><mi id="S2.SS2.p3.13.m13.1.1.2.2.2" xref="S2.SS2.p3.13.m13.1.1.2.2.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p3.13.m13.1.1.2.2.1" xref="S2.SS2.p3.13.m13.1.1.2.2.1.cmml">​</mo><mi id="S2.SS2.p3.13.m13.1.1.2.2.3" xref="S2.SS2.p3.13.m13.1.1.2.2.3.cmml">r</mi></mrow><mo id="S2.SS2.p3.13.m13.1.1.2.1" xref="S2.SS2.p3.13.m13.1.1.2.1.cmml">+</mo><mrow id="S2.SS2.p3.13.m13.1.1.2.3" xref="S2.SS2.p3.13.m13.1.1.2.3.cmml"><mi id="S2.SS2.p3.13.m13.1.1.2.3.2" xref="S2.SS2.p3.13.m13.1.1.2.3.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p3.13.m13.1.1.2.3.1" xref="S2.SS2.p3.13.m13.1.1.2.3.1.cmml">​</mo><mi id="S2.SS2.p3.13.m13.1.1.2.3.3" xref="S2.SS2.p3.13.m13.1.1.2.3.3.cmml">n</mi></mrow></mrow><mrow id="S2.SS2.p3.13.m13.1.1.3" xref="S2.SS2.p3.13.m13.1.1.3.cmml"><mi id="S2.SS2.p3.13.m13.1.1.3.2" xref="S2.SS2.p3.13.m13.1.1.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p3.13.m13.1.1.3.1" xref="S2.SS2.p3.13.m13.1.1.3.1.cmml">​</mo><mi id="S2.SS2.p3.13.m13.1.1.3.3" xref="S2.SS2.p3.13.m13.1.1.3.3.cmml">n</mi></mrow></mfrac><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.13.m13.1b"><apply id="S2.SS2.p3.13.m13.1.1.cmml" xref="S2.SS2.p3.13.m13.1.1"><divide id="S2.SS2.p3.13.m13.1.1.1.cmml" xref="S2.SS2.p3.13.m13.1.1"></divide><apply id="S2.SS2.p3.13.m13.1.1.2.cmml" xref="S2.SS2.p3.13.m13.1.1.2"><plus id="S2.SS2.p3.13.m13.1.1.2.1.cmml" xref="S2.SS2.p3.13.m13.1.1.2.1"></plus><apply id="S2.SS2.p3.13.m13.1.1.2.2.cmml" xref="S2.SS2.p3.13.m13.1.1.2.2"><times id="S2.SS2.p3.13.m13.1.1.2.2.1.cmml" xref="S2.SS2.p3.13.m13.1.1.2.2.1"></times><ci id="S2.SS2.p3.13.m13.1.1.2.2.2.cmml" xref="S2.SS2.p3.13.m13.1.1.2.2.2">𝑚</ci><ci id="S2.SS2.p3.13.m13.1.1.2.2.3.cmml" xref="S2.SS2.p3.13.m13.1.1.2.2.3">𝑟</ci></apply><apply id="S2.SS2.p3.13.m13.1.1.2.3.cmml" xref="S2.SS2.p3.13.m13.1.1.2.3"><times id="S2.SS2.p3.13.m13.1.1.2.3.1.cmml" xref="S2.SS2.p3.13.m13.1.1.2.3.1"></times><ci id="S2.SS2.p3.13.m13.1.1.2.3.2.cmml" xref="S2.SS2.p3.13.m13.1.1.2.3.2">𝑟</ci><ci id="S2.SS2.p3.13.m13.1.1.2.3.3.cmml" xref="S2.SS2.p3.13.m13.1.1.2.3.3">𝑛</ci></apply></apply><apply id="S2.SS2.p3.13.m13.1.1.3.cmml" xref="S2.SS2.p3.13.m13.1.1.3"><times id="S2.SS2.p3.13.m13.1.1.3.1.cmml" xref="S2.SS2.p3.13.m13.1.1.3.1"></times><ci id="S2.SS2.p3.13.m13.1.1.3.2.cmml" xref="S2.SS2.p3.13.m13.1.1.3.2">𝑚</ci><ci id="S2.SS2.p3.13.m13.1.1.3.3.cmml" xref="S2.SS2.p3.13.m13.1.1.3.3">𝑛</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.13.m13.1c">\frac{mr+rn}{mn}</annotation></semantics></math>.</p>
</div>
<figure id="S2.F2" class="ltx_figure"><img src="/html/2407.21118/assets/x2.png" id="S2.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="406" height="148" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span><span id="S2.F2.14.1" class="ltx_text ltx_font_italic">Palu</span> uses low-rank decomposition <math id="S2.F2.7.m1.1" class="ltx_Math" alttext="(\mathbf{W}\approx\mathbf{A}\mathbf{B})" display="inline"><semantics id="S2.F2.7.m1.1b"><mrow id="S2.F2.7.m1.1.1.1" xref="S2.F2.7.m1.1.1.1.1.cmml"><mo stretchy="false" id="S2.F2.7.m1.1.1.1.2" xref="S2.F2.7.m1.1.1.1.1.cmml">(</mo><mrow id="S2.F2.7.m1.1.1.1.1" xref="S2.F2.7.m1.1.1.1.1.cmml"><mi id="S2.F2.7.m1.1.1.1.1.2" xref="S2.F2.7.m1.1.1.1.1.2.cmml">𝐖</mi><mo id="S2.F2.7.m1.1.1.1.1.1" xref="S2.F2.7.m1.1.1.1.1.1.cmml">≈</mo><mi id="S2.F2.7.m1.1.1.1.1.3" xref="S2.F2.7.m1.1.1.1.1.3.cmml">𝐀𝐁</mi></mrow><mo stretchy="false" id="S2.F2.7.m1.1.1.1.3" xref="S2.F2.7.m1.1.1.1.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.F2.7.m1.1c"><apply id="S2.F2.7.m1.1.1.1.1.cmml" xref="S2.F2.7.m1.1.1.1"><approx id="S2.F2.7.m1.1.1.1.1.1.cmml" xref="S2.F2.7.m1.1.1.1.1.1"></approx><ci id="S2.F2.7.m1.1.1.1.1.2.cmml" xref="S2.F2.7.m1.1.1.1.1.2">𝐖</ci><ci id="S2.F2.7.m1.1.1.1.1.3.cmml" xref="S2.F2.7.m1.1.1.1.1.3">𝐀𝐁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.F2.7.m1.1d">(\mathbf{W}\approx\mathbf{A}\mathbf{B})</annotation></semantics></math> to project the key (or value) to a lower-dimensional latent representation <math id="S2.F2.8.m2.1" class="ltx_Math" alttext="(\mathbf{h})" display="inline"><semantics id="S2.F2.8.m2.1b"><mrow id="S2.F2.8.m2.1.2.2"><mo stretchy="false" id="S2.F2.8.m2.1.2.2.1">(</mo><mi id="S2.F2.8.m2.1.1" xref="S2.F2.8.m2.1.1.cmml">𝐡</mi><mo stretchy="false" id="S2.F2.8.m2.1.2.2.2">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.F2.8.m2.1c"><ci id="S2.F2.8.m2.1.1.cmml" xref="S2.F2.8.m2.1.1">𝐡</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.F2.8.m2.1d">(\mathbf{h})</annotation></semantics></math>, thereby reducing the size of the KV-Cache.
The original key <math id="S2.F2.9.m3.1" class="ltx_Math" alttext="(\mathbf{K}_{t})" display="inline"><semantics id="S2.F2.9.m3.1b"><mrow id="S2.F2.9.m3.1.1.1" xref="S2.F2.9.m3.1.1.1.1.cmml"><mo stretchy="false" id="S2.F2.9.m3.1.1.1.2" xref="S2.F2.9.m3.1.1.1.1.cmml">(</mo><msub id="S2.F2.9.m3.1.1.1.1" xref="S2.F2.9.m3.1.1.1.1.cmml"><mi id="S2.F2.9.m3.1.1.1.1.2" xref="S2.F2.9.m3.1.1.1.1.2.cmml">𝐊</mi><mi id="S2.F2.9.m3.1.1.1.1.3" xref="S2.F2.9.m3.1.1.1.1.3.cmml">t</mi></msub><mo stretchy="false" id="S2.F2.9.m3.1.1.1.3" xref="S2.F2.9.m3.1.1.1.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.F2.9.m3.1c"><apply id="S2.F2.9.m3.1.1.1.1.cmml" xref="S2.F2.9.m3.1.1.1"><csymbol cd="ambiguous" id="S2.F2.9.m3.1.1.1.1.1.cmml" xref="S2.F2.9.m3.1.1.1">subscript</csymbol><ci id="S2.F2.9.m3.1.1.1.1.2.cmml" xref="S2.F2.9.m3.1.1.1.1.2">𝐊</ci><ci id="S2.F2.9.m3.1.1.1.1.3.cmml" xref="S2.F2.9.m3.1.1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.F2.9.m3.1d">(\mathbf{K}_{t})</annotation></semantics></math> is reconstructed on-the-fly with <math id="S2.F2.10.m4.1" class="ltx_Math" alttext="\mathbf{B}^{k}" display="inline"><semantics id="S2.F2.10.m4.1b"><msup id="S2.F2.10.m4.1.1" xref="S2.F2.10.m4.1.1.cmml"><mi id="S2.F2.10.m4.1.1.2" xref="S2.F2.10.m4.1.1.2.cmml">𝐁</mi><mi id="S2.F2.10.m4.1.1.3" xref="S2.F2.10.m4.1.1.3.cmml">k</mi></msup><annotation-xml encoding="MathML-Content" id="S2.F2.10.m4.1c"><apply id="S2.F2.10.m4.1.1.cmml" xref="S2.F2.10.m4.1.1"><csymbol cd="ambiguous" id="S2.F2.10.m4.1.1.1.cmml" xref="S2.F2.10.m4.1.1">superscript</csymbol><ci id="S2.F2.10.m4.1.1.2.cmml" xref="S2.F2.10.m4.1.1.2">𝐁</ci><ci id="S2.F2.10.m4.1.1.3.cmml" xref="S2.F2.10.m4.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.F2.10.m4.1d">\mathbf{B}^{k}</annotation></semantics></math>, and <math id="S2.F2.11.m5.1" class="ltx_Math" alttext="\mathbf{B}^{v}" display="inline"><semantics id="S2.F2.11.m5.1b"><msup id="S2.F2.11.m5.1.1" xref="S2.F2.11.m5.1.1.cmml"><mi id="S2.F2.11.m5.1.1.2" xref="S2.F2.11.m5.1.1.2.cmml">𝐁</mi><mi id="S2.F2.11.m5.1.1.3" xref="S2.F2.11.m5.1.1.3.cmml">v</mi></msup><annotation-xml encoding="MathML-Content" id="S2.F2.11.m5.1c"><apply id="S2.F2.11.m5.1.1.cmml" xref="S2.F2.11.m5.1.1"><csymbol cd="ambiguous" id="S2.F2.11.m5.1.1.1.cmml" xref="S2.F2.11.m5.1.1">superscript</csymbol><ci id="S2.F2.11.m5.1.1.2.cmml" xref="S2.F2.11.m5.1.1.2">𝐁</ci><ci id="S2.F2.11.m5.1.1.3.cmml" xref="S2.F2.11.m5.1.1.3">𝑣</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.F2.11.m5.1d">\mathbf{B}^{v}</annotation></semantics></math> is fused into <math id="S2.F2.12.m6.1" class="ltx_Math" alttext="\mathbf{W}^{o}" display="inline"><semantics id="S2.F2.12.m6.1b"><msup id="S2.F2.12.m6.1.1" xref="S2.F2.12.m6.1.1.cmml"><mi id="S2.F2.12.m6.1.1.2" xref="S2.F2.12.m6.1.1.2.cmml">𝐖</mi><mi id="S2.F2.12.m6.1.1.3" xref="S2.F2.12.m6.1.1.3.cmml">o</mi></msup><annotation-xml encoding="MathML-Content" id="S2.F2.12.m6.1c"><apply id="S2.F2.12.m6.1.1.cmml" xref="S2.F2.12.m6.1.1"><csymbol cd="ambiguous" id="S2.F2.12.m6.1.1.1.cmml" xref="S2.F2.12.m6.1.1">superscript</csymbol><ci id="S2.F2.12.m6.1.1.2.cmml" xref="S2.F2.12.m6.1.1.2">𝐖</ci><ci id="S2.F2.12.m6.1.1.3.cmml" xref="S2.F2.12.m6.1.1.3">𝑜</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.F2.12.m6.1d">\mathbf{W}^{o}</annotation></semantics></math> to avoid reconstruction overhead. The fusion also reduces the computational burden for output projection.</figcaption>
</figure>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>The Palu Framework</h2>

<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Compressing the KV-Cache via Low-Rank Projection</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.1" class="ltx_p">As noted, a naive approach to compress the KV-Cache with
low-rank projection is to apply SVD directly on the KV-Cache and store the top-<math id="S3.SS1.p1.1.m1.1" class="ltx_Math" alttext="r" display="inline"><semantics id="S3.SS1.p1.1.m1.1a"><mi id="S3.SS1.p1.1.m1.1.1" xref="S3.SS1.p1.1.m1.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.1.m1.1b"><ci id="S3.SS1.p1.1.m1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.1.m1.1c">r</annotation></semantics></math> singular vectors.
However, this approach poses significant computational challenges during runtime that make it impractical for deployments (see Section <a href="#S1" title="1 Introduction ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>).</p>
</div>
<div id="S3.SS1.p2" class="ltx_para">
<p id="S3.SS1.p2.6" class="ltx_p">To apply low-rank projection more efficiently, <span id="S3.SS1.p2.6.1" class="ltx_text ltx_font_italic">Palu</span> uses SVD to decompose the key and value projection weight matrices.
This approach is based on the observation that low-rank decomposition rewrites the linear projection layer from <math id="S3.SS1.p2.1.m1.1" class="ltx_Math" alttext="\mathbf{y}=\mathbf{x}\mathbf{W}" display="inline"><semantics id="S3.SS1.p2.1.m1.1a"><mrow id="S3.SS1.p2.1.m1.1.1" xref="S3.SS1.p2.1.m1.1.1.cmml"><mi id="S3.SS1.p2.1.m1.1.1.2" xref="S3.SS1.p2.1.m1.1.1.2.cmml">𝐲</mi><mo id="S3.SS1.p2.1.m1.1.1.1" xref="S3.SS1.p2.1.m1.1.1.1.cmml">=</mo><mi id="S3.SS1.p2.1.m1.1.1.3" xref="S3.SS1.p2.1.m1.1.1.3.cmml">𝐱𝐖</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.1.m1.1b"><apply id="S3.SS1.p2.1.m1.1.1.cmml" xref="S3.SS1.p2.1.m1.1.1"><eq id="S3.SS1.p2.1.m1.1.1.1.cmml" xref="S3.SS1.p2.1.m1.1.1.1"></eq><ci id="S3.SS1.p2.1.m1.1.1.2.cmml" xref="S3.SS1.p2.1.m1.1.1.2">𝐲</ci><ci id="S3.SS1.p2.1.m1.1.1.3.cmml" xref="S3.SS1.p2.1.m1.1.1.3">𝐱𝐖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.1.m1.1c">\mathbf{y}=\mathbf{x}\mathbf{W}</annotation></semantics></math> into
<math id="S3.SS1.p2.2.m2.1" class="ltx_Math" alttext="\mathbf{y}=\mathbf{x}\mathbf{A}\mathbf{B}" display="inline"><semantics id="S3.SS1.p2.2.m2.1a"><mrow id="S3.SS1.p2.2.m2.1.1" xref="S3.SS1.p2.2.m2.1.1.cmml"><mi id="S3.SS1.p2.2.m2.1.1.2" xref="S3.SS1.p2.2.m2.1.1.2.cmml">𝐲</mi><mo id="S3.SS1.p2.2.m2.1.1.1" xref="S3.SS1.p2.2.m2.1.1.1.cmml">=</mo><mi id="S3.SS1.p2.2.m2.1.1.3" xref="S3.SS1.p2.2.m2.1.1.3.cmml">𝐱𝐀𝐁</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.2.m2.1b"><apply id="S3.SS1.p2.2.m2.1.1.cmml" xref="S3.SS1.p2.2.m2.1.1"><eq id="S3.SS1.p2.2.m2.1.1.1.cmml" xref="S3.SS1.p2.2.m2.1.1.1"></eq><ci id="S3.SS1.p2.2.m2.1.1.2.cmml" xref="S3.SS1.p2.2.m2.1.1.2">𝐲</ci><ci id="S3.SS1.p2.2.m2.1.1.3.cmml" xref="S3.SS1.p2.2.m2.1.1.3">𝐱𝐀𝐁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.2.m2.1c">\mathbf{y}=\mathbf{x}\mathbf{A}\mathbf{B}</annotation></semantics></math>.
Here, <math id="S3.SS1.p2.3.m3.1" class="ltx_Math" alttext="\mathbf{A}\in\mathbb{R}^{d\times r}" display="inline"><semantics id="S3.SS1.p2.3.m3.1a"><mrow id="S3.SS1.p2.3.m3.1.1" xref="S3.SS1.p2.3.m3.1.1.cmml"><mi id="S3.SS1.p2.3.m3.1.1.2" xref="S3.SS1.p2.3.m3.1.1.2.cmml">𝐀</mi><mo id="S3.SS1.p2.3.m3.1.1.1" xref="S3.SS1.p2.3.m3.1.1.1.cmml">∈</mo><msup id="S3.SS1.p2.3.m3.1.1.3" xref="S3.SS1.p2.3.m3.1.1.3.cmml"><mi id="S3.SS1.p2.3.m3.1.1.3.2" xref="S3.SS1.p2.3.m3.1.1.3.2.cmml">ℝ</mi><mrow id="S3.SS1.p2.3.m3.1.1.3.3" xref="S3.SS1.p2.3.m3.1.1.3.3.cmml"><mi id="S3.SS1.p2.3.m3.1.1.3.3.2" xref="S3.SS1.p2.3.m3.1.1.3.3.2.cmml">d</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS1.p2.3.m3.1.1.3.3.1" xref="S3.SS1.p2.3.m3.1.1.3.3.1.cmml">×</mo><mi id="S3.SS1.p2.3.m3.1.1.3.3.3" xref="S3.SS1.p2.3.m3.1.1.3.3.3.cmml">r</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.3.m3.1b"><apply id="S3.SS1.p2.3.m3.1.1.cmml" xref="S3.SS1.p2.3.m3.1.1"><in id="S3.SS1.p2.3.m3.1.1.1.cmml" xref="S3.SS1.p2.3.m3.1.1.1"></in><ci id="S3.SS1.p2.3.m3.1.1.2.cmml" xref="S3.SS1.p2.3.m3.1.1.2">𝐀</ci><apply id="S3.SS1.p2.3.m3.1.1.3.cmml" xref="S3.SS1.p2.3.m3.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p2.3.m3.1.1.3.1.cmml" xref="S3.SS1.p2.3.m3.1.1.3">superscript</csymbol><ci id="S3.SS1.p2.3.m3.1.1.3.2.cmml" xref="S3.SS1.p2.3.m3.1.1.3.2">ℝ</ci><apply id="S3.SS1.p2.3.m3.1.1.3.3.cmml" xref="S3.SS1.p2.3.m3.1.1.3.3"><times id="S3.SS1.p2.3.m3.1.1.3.3.1.cmml" xref="S3.SS1.p2.3.m3.1.1.3.3.1"></times><ci id="S3.SS1.p2.3.m3.1.1.3.3.2.cmml" xref="S3.SS1.p2.3.m3.1.1.3.3.2">𝑑</ci><ci id="S3.SS1.p2.3.m3.1.1.3.3.3.cmml" xref="S3.SS1.p2.3.m3.1.1.3.3.3">𝑟</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.3.m3.1c">\mathbf{A}\in\mathbb{R}^{d\times r}</annotation></semantics></math> is the low-rank projection matrix, and <math id="S3.SS1.p2.4.m4.1" class="ltx_Math" alttext="\mathbf{B}\in\mathbb{R}^{r\times d}" display="inline"><semantics id="S3.SS1.p2.4.m4.1a"><mrow id="S3.SS1.p2.4.m4.1.1" xref="S3.SS1.p2.4.m4.1.1.cmml"><mi id="S3.SS1.p2.4.m4.1.1.2" xref="S3.SS1.p2.4.m4.1.1.2.cmml">𝐁</mi><mo id="S3.SS1.p2.4.m4.1.1.1" xref="S3.SS1.p2.4.m4.1.1.1.cmml">∈</mo><msup id="S3.SS1.p2.4.m4.1.1.3" xref="S3.SS1.p2.4.m4.1.1.3.cmml"><mi id="S3.SS1.p2.4.m4.1.1.3.2" xref="S3.SS1.p2.4.m4.1.1.3.2.cmml">ℝ</mi><mrow id="S3.SS1.p2.4.m4.1.1.3.3" xref="S3.SS1.p2.4.m4.1.1.3.3.cmml"><mi id="S3.SS1.p2.4.m4.1.1.3.3.2" xref="S3.SS1.p2.4.m4.1.1.3.3.2.cmml">r</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS1.p2.4.m4.1.1.3.3.1" xref="S3.SS1.p2.4.m4.1.1.3.3.1.cmml">×</mo><mi id="S3.SS1.p2.4.m4.1.1.3.3.3" xref="S3.SS1.p2.4.m4.1.1.3.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.4.m4.1b"><apply id="S3.SS1.p2.4.m4.1.1.cmml" xref="S3.SS1.p2.4.m4.1.1"><in id="S3.SS1.p2.4.m4.1.1.1.cmml" xref="S3.SS1.p2.4.m4.1.1.1"></in><ci id="S3.SS1.p2.4.m4.1.1.2.cmml" xref="S3.SS1.p2.4.m4.1.1.2">𝐁</ci><apply id="S3.SS1.p2.4.m4.1.1.3.cmml" xref="S3.SS1.p2.4.m4.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p2.4.m4.1.1.3.1.cmml" xref="S3.SS1.p2.4.m4.1.1.3">superscript</csymbol><ci id="S3.SS1.p2.4.m4.1.1.3.2.cmml" xref="S3.SS1.p2.4.m4.1.1.3.2">ℝ</ci><apply id="S3.SS1.p2.4.m4.1.1.3.3.cmml" xref="S3.SS1.p2.4.m4.1.1.3.3"><times id="S3.SS1.p2.4.m4.1.1.3.3.1.cmml" xref="S3.SS1.p2.4.m4.1.1.3.3.1"></times><ci id="S3.SS1.p2.4.m4.1.1.3.3.2.cmml" xref="S3.SS1.p2.4.m4.1.1.3.3.2">𝑟</ci><ci id="S3.SS1.p2.4.m4.1.1.3.3.3.cmml" xref="S3.SS1.p2.4.m4.1.1.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.4.m4.1c">\mathbf{B}\in\mathbb{R}^{r\times d}</annotation></semantics></math> is the reconstruction matrix derived by SVD.
The forward process first <span id="S3.SS1.p2.6.2" class="ltx_text ltx_font_italic">down-projects</span> the input token <math id="S3.SS1.p2.5.m5.1" class="ltx_Math" alttext="\mathbf{x}\in\mathbb{R}^{d}" display="inline"><semantics id="S3.SS1.p2.5.m5.1a"><mrow id="S3.SS1.p2.5.m5.1.1" xref="S3.SS1.p2.5.m5.1.1.cmml"><mi id="S3.SS1.p2.5.m5.1.1.2" xref="S3.SS1.p2.5.m5.1.1.2.cmml">𝐱</mi><mo id="S3.SS1.p2.5.m5.1.1.1" xref="S3.SS1.p2.5.m5.1.1.1.cmml">∈</mo><msup id="S3.SS1.p2.5.m5.1.1.3" xref="S3.SS1.p2.5.m5.1.1.3.cmml"><mi id="S3.SS1.p2.5.m5.1.1.3.2" xref="S3.SS1.p2.5.m5.1.1.3.2.cmml">ℝ</mi><mi id="S3.SS1.p2.5.m5.1.1.3.3" xref="S3.SS1.p2.5.m5.1.1.3.3.cmml">d</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.5.m5.1b"><apply id="S3.SS1.p2.5.m5.1.1.cmml" xref="S3.SS1.p2.5.m5.1.1"><in id="S3.SS1.p2.5.m5.1.1.1.cmml" xref="S3.SS1.p2.5.m5.1.1.1"></in><ci id="S3.SS1.p2.5.m5.1.1.2.cmml" xref="S3.SS1.p2.5.m5.1.1.2">𝐱</ci><apply id="S3.SS1.p2.5.m5.1.1.3.cmml" xref="S3.SS1.p2.5.m5.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p2.5.m5.1.1.3.1.cmml" xref="S3.SS1.p2.5.m5.1.1.3">superscript</csymbol><ci id="S3.SS1.p2.5.m5.1.1.3.2.cmml" xref="S3.SS1.p2.5.m5.1.1.3.2">ℝ</ci><ci id="S3.SS1.p2.5.m5.1.1.3.3.cmml" xref="S3.SS1.p2.5.m5.1.1.3.3">𝑑</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.5.m5.1c">\mathbf{x}\in\mathbb{R}^{d}</annotation></semantics></math> into a low-dimensional latent space <math id="S3.SS1.p2.6.m6.1" class="ltx_Math" alttext="\mathbf{h}\in\mathbb{R}^{r}" display="inline"><semantics id="S3.SS1.p2.6.m6.1a"><mrow id="S3.SS1.p2.6.m6.1.1" xref="S3.SS1.p2.6.m6.1.1.cmml"><mi id="S3.SS1.p2.6.m6.1.1.2" xref="S3.SS1.p2.6.m6.1.1.2.cmml">𝐡</mi><mo id="S3.SS1.p2.6.m6.1.1.1" xref="S3.SS1.p2.6.m6.1.1.1.cmml">∈</mo><msup id="S3.SS1.p2.6.m6.1.1.3" xref="S3.SS1.p2.6.m6.1.1.3.cmml"><mi id="S3.SS1.p2.6.m6.1.1.3.2" xref="S3.SS1.p2.6.m6.1.1.3.2.cmml">ℝ</mi><mi id="S3.SS1.p2.6.m6.1.1.3.3" xref="S3.SS1.p2.6.m6.1.1.3.3.cmml">r</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.6.m6.1b"><apply id="S3.SS1.p2.6.m6.1.1.cmml" xref="S3.SS1.p2.6.m6.1.1"><in id="S3.SS1.p2.6.m6.1.1.1.cmml" xref="S3.SS1.p2.6.m6.1.1.1"></in><ci id="S3.SS1.p2.6.m6.1.1.2.cmml" xref="S3.SS1.p2.6.m6.1.1.2">𝐡</ci><apply id="S3.SS1.p2.6.m6.1.1.3.cmml" xref="S3.SS1.p2.6.m6.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p2.6.m6.1.1.3.1.cmml" xref="S3.SS1.p2.6.m6.1.1.3">superscript</csymbol><ci id="S3.SS1.p2.6.m6.1.1.3.2.cmml" xref="S3.SS1.p2.6.m6.1.1.3.2">ℝ</ci><ci id="S3.SS1.p2.6.m6.1.1.3.3.cmml" xref="S3.SS1.p2.6.m6.1.1.3.3">𝑟</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.6.m6.1c">\mathbf{h}\in\mathbb{R}^{r}</annotation></semantics></math> and then <span id="S3.SS1.p2.6.3" class="ltx_text ltx_font_italic">up-projects</span> it back to the original space:</p>
<table id="S3.E4" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E4.m1.2" class="ltx_Math" alttext="\mathbf{h}=\mathbf{A}\mathbf{x},\quad\mathbf{y}=\mathbf{B}\mathbf{h}" display="block"><semantics id="S3.E4.m1.2a"><mrow id="S3.E4.m1.2.2.2" xref="S3.E4.m1.2.2.3.cmml"><mrow id="S3.E4.m1.1.1.1.1" xref="S3.E4.m1.1.1.1.1.cmml"><mi id="S3.E4.m1.1.1.1.1.2" xref="S3.E4.m1.1.1.1.1.2.cmml">𝐡</mi><mo id="S3.E4.m1.1.1.1.1.1" xref="S3.E4.m1.1.1.1.1.1.cmml">=</mo><mi id="S3.E4.m1.1.1.1.1.3" xref="S3.E4.m1.1.1.1.1.3.cmml">𝐀𝐱</mi></mrow><mo rspace="1.167em" id="S3.E4.m1.2.2.2.3" xref="S3.E4.m1.2.2.3a.cmml">,</mo><mrow id="S3.E4.m1.2.2.2.2" xref="S3.E4.m1.2.2.2.2.cmml"><mi id="S3.E4.m1.2.2.2.2.2" xref="S3.E4.m1.2.2.2.2.2.cmml">𝐲</mi><mo id="S3.E4.m1.2.2.2.2.1" xref="S3.E4.m1.2.2.2.2.1.cmml">=</mo><mi id="S3.E4.m1.2.2.2.2.3" xref="S3.E4.m1.2.2.2.2.3.cmml">𝐁𝐡</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E4.m1.2b"><apply id="S3.E4.m1.2.2.3.cmml" xref="S3.E4.m1.2.2.2"><csymbol cd="ambiguous" id="S3.E4.m1.2.2.3a.cmml" xref="S3.E4.m1.2.2.2.3">formulae-sequence</csymbol><apply id="S3.E4.m1.1.1.1.1.cmml" xref="S3.E4.m1.1.1.1.1"><eq id="S3.E4.m1.1.1.1.1.1.cmml" xref="S3.E4.m1.1.1.1.1.1"></eq><ci id="S3.E4.m1.1.1.1.1.2.cmml" xref="S3.E4.m1.1.1.1.1.2">𝐡</ci><ci id="S3.E4.m1.1.1.1.1.3.cmml" xref="S3.E4.m1.1.1.1.1.3">𝐀𝐱</ci></apply><apply id="S3.E4.m1.2.2.2.2.cmml" xref="S3.E4.m1.2.2.2.2"><eq id="S3.E4.m1.2.2.2.2.1.cmml" xref="S3.E4.m1.2.2.2.2.1"></eq><ci id="S3.E4.m1.2.2.2.2.2.cmml" xref="S3.E4.m1.2.2.2.2.2">𝐲</ci><ci id="S3.E4.m1.2.2.2.2.3.cmml" xref="S3.E4.m1.2.2.2.2.3">𝐁𝐡</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E4.m1.2c">\mathbf{h}=\mathbf{A}\mathbf{x},\quad\mathbf{y}=\mathbf{B}\mathbf{h}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(4)</span></td>
</tr></tbody>
</table>
<p id="S3.SS1.p2.7" class="ltx_p">This two-step process lets <span id="S3.SS1.p2.7.1" class="ltx_text ltx_font_italic">Palu</span> (1) store the lower dimension latent representation instead of the origin keys and values states, and (2) reconstruct them during decoding.</p>
</div>
<section id="S3.SS1.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.1.1 </span>Integration with the Attention Mechanism and Offline Matrix Fusion</h4>

<div id="S3.SS1.SSS1.p1" class="ltx_para">
<p id="S3.SS1.SSS1.p1.5" class="ltx_p">We now describe how <span id="S3.SS1.SSS1.p1.5.1" class="ltx_text ltx_font_italic">Palu</span> decomposes the key and value linear layers for the attention mechanism.
For each attention head <math id="S3.SS1.SSS1.p1.1.m1.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S3.SS1.SSS1.p1.1.m1.1a"><mi id="S3.SS1.SSS1.p1.1.m1.1.1" xref="S3.SS1.SSS1.p1.1.m1.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS1.p1.1.m1.1b"><ci id="S3.SS1.SSS1.p1.1.m1.1.1.cmml" xref="S3.SS1.SSS1.p1.1.m1.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS1.p1.1.m1.1c">i</annotation></semantics></math>, <span id="S3.SS1.SSS1.p1.5.2" class="ltx_text ltx_font_italic">Palu</span> applies SVD and maps the key-projection matrix <math id="S3.SS1.SSS1.p1.2.m2.1" class="ltx_Math" alttext="\mathbf{W}^{k}_{i}" display="inline"><semantics id="S3.SS1.SSS1.p1.2.m2.1a"><msubsup id="S3.SS1.SSS1.p1.2.m2.1.1" xref="S3.SS1.SSS1.p1.2.m2.1.1.cmml"><mi id="S3.SS1.SSS1.p1.2.m2.1.1.2.2" xref="S3.SS1.SSS1.p1.2.m2.1.1.2.2.cmml">𝐖</mi><mi id="S3.SS1.SSS1.p1.2.m2.1.1.3" xref="S3.SS1.SSS1.p1.2.m2.1.1.3.cmml">i</mi><mi id="S3.SS1.SSS1.p1.2.m2.1.1.2.3" xref="S3.SS1.SSS1.p1.2.m2.1.1.2.3.cmml">k</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS1.p1.2.m2.1b"><apply id="S3.SS1.SSS1.p1.2.m2.1.1.cmml" xref="S3.SS1.SSS1.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p1.2.m2.1.1.1.cmml" xref="S3.SS1.SSS1.p1.2.m2.1.1">subscript</csymbol><apply id="S3.SS1.SSS1.p1.2.m2.1.1.2.cmml" xref="S3.SS1.SSS1.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p1.2.m2.1.1.2.1.cmml" xref="S3.SS1.SSS1.p1.2.m2.1.1">superscript</csymbol><ci id="S3.SS1.SSS1.p1.2.m2.1.1.2.2.cmml" xref="S3.SS1.SSS1.p1.2.m2.1.1.2.2">𝐖</ci><ci id="S3.SS1.SSS1.p1.2.m2.1.1.2.3.cmml" xref="S3.SS1.SSS1.p1.2.m2.1.1.2.3">𝑘</ci></apply><ci id="S3.SS1.SSS1.p1.2.m2.1.1.3.cmml" xref="S3.SS1.SSS1.p1.2.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS1.p1.2.m2.1c">\mathbf{W}^{k}_{i}</annotation></semantics></math> and value-projection matrix <math id="S3.SS1.SSS1.p1.3.m3.1" class="ltx_Math" alttext="\mathbf{W}^{v}_{i}" display="inline"><semantics id="S3.SS1.SSS1.p1.3.m3.1a"><msubsup id="S3.SS1.SSS1.p1.3.m3.1.1" xref="S3.SS1.SSS1.p1.3.m3.1.1.cmml"><mi id="S3.SS1.SSS1.p1.3.m3.1.1.2.2" xref="S3.SS1.SSS1.p1.3.m3.1.1.2.2.cmml">𝐖</mi><mi id="S3.SS1.SSS1.p1.3.m3.1.1.3" xref="S3.SS1.SSS1.p1.3.m3.1.1.3.cmml">i</mi><mi id="S3.SS1.SSS1.p1.3.m3.1.1.2.3" xref="S3.SS1.SSS1.p1.3.m3.1.1.2.3.cmml">v</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS1.p1.3.m3.1b"><apply id="S3.SS1.SSS1.p1.3.m3.1.1.cmml" xref="S3.SS1.SSS1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p1.3.m3.1.1.1.cmml" xref="S3.SS1.SSS1.p1.3.m3.1.1">subscript</csymbol><apply id="S3.SS1.SSS1.p1.3.m3.1.1.2.cmml" xref="S3.SS1.SSS1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p1.3.m3.1.1.2.1.cmml" xref="S3.SS1.SSS1.p1.3.m3.1.1">superscript</csymbol><ci id="S3.SS1.SSS1.p1.3.m3.1.1.2.2.cmml" xref="S3.SS1.SSS1.p1.3.m3.1.1.2.2">𝐖</ci><ci id="S3.SS1.SSS1.p1.3.m3.1.1.2.3.cmml" xref="S3.SS1.SSS1.p1.3.m3.1.1.2.3">𝑣</ci></apply><ci id="S3.SS1.SSS1.p1.3.m3.1.1.3.cmml" xref="S3.SS1.SSS1.p1.3.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS1.p1.3.m3.1c">\mathbf{W}^{v}_{i}</annotation></semantics></math> into <math id="S3.SS1.SSS1.p1.4.m4.1" class="ltx_Math" alttext="\mathbf{A}^{k}_{i}\mathbf{B}^{k}_{i}" display="inline"><semantics id="S3.SS1.SSS1.p1.4.m4.1a"><mrow id="S3.SS1.SSS1.p1.4.m4.1.1" xref="S3.SS1.SSS1.p1.4.m4.1.1.cmml"><msubsup id="S3.SS1.SSS1.p1.4.m4.1.1.2" xref="S3.SS1.SSS1.p1.4.m4.1.1.2.cmml"><mi id="S3.SS1.SSS1.p1.4.m4.1.1.2.2.2" xref="S3.SS1.SSS1.p1.4.m4.1.1.2.2.2.cmml">𝐀</mi><mi id="S3.SS1.SSS1.p1.4.m4.1.1.2.3" xref="S3.SS1.SSS1.p1.4.m4.1.1.2.3.cmml">i</mi><mi id="S3.SS1.SSS1.p1.4.m4.1.1.2.2.3" xref="S3.SS1.SSS1.p1.4.m4.1.1.2.2.3.cmml">k</mi></msubsup><mo lspace="0em" rspace="0em" id="S3.SS1.SSS1.p1.4.m4.1.1.1" xref="S3.SS1.SSS1.p1.4.m4.1.1.1.cmml">​</mo><msubsup id="S3.SS1.SSS1.p1.4.m4.1.1.3" xref="S3.SS1.SSS1.p1.4.m4.1.1.3.cmml"><mi id="S3.SS1.SSS1.p1.4.m4.1.1.3.2.2" xref="S3.SS1.SSS1.p1.4.m4.1.1.3.2.2.cmml">𝐁</mi><mi id="S3.SS1.SSS1.p1.4.m4.1.1.3.3" xref="S3.SS1.SSS1.p1.4.m4.1.1.3.3.cmml">i</mi><mi id="S3.SS1.SSS1.p1.4.m4.1.1.3.2.3" xref="S3.SS1.SSS1.p1.4.m4.1.1.3.2.3.cmml">k</mi></msubsup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS1.p1.4.m4.1b"><apply id="S3.SS1.SSS1.p1.4.m4.1.1.cmml" xref="S3.SS1.SSS1.p1.4.m4.1.1"><times id="S3.SS1.SSS1.p1.4.m4.1.1.1.cmml" xref="S3.SS1.SSS1.p1.4.m4.1.1.1"></times><apply id="S3.SS1.SSS1.p1.4.m4.1.1.2.cmml" xref="S3.SS1.SSS1.p1.4.m4.1.1.2"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p1.4.m4.1.1.2.1.cmml" xref="S3.SS1.SSS1.p1.4.m4.1.1.2">subscript</csymbol><apply id="S3.SS1.SSS1.p1.4.m4.1.1.2.2.cmml" xref="S3.SS1.SSS1.p1.4.m4.1.1.2"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p1.4.m4.1.1.2.2.1.cmml" xref="S3.SS1.SSS1.p1.4.m4.1.1.2">superscript</csymbol><ci id="S3.SS1.SSS1.p1.4.m4.1.1.2.2.2.cmml" xref="S3.SS1.SSS1.p1.4.m4.1.1.2.2.2">𝐀</ci><ci id="S3.SS1.SSS1.p1.4.m4.1.1.2.2.3.cmml" xref="S3.SS1.SSS1.p1.4.m4.1.1.2.2.3">𝑘</ci></apply><ci id="S3.SS1.SSS1.p1.4.m4.1.1.2.3.cmml" xref="S3.SS1.SSS1.p1.4.m4.1.1.2.3">𝑖</ci></apply><apply id="S3.SS1.SSS1.p1.4.m4.1.1.3.cmml" xref="S3.SS1.SSS1.p1.4.m4.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p1.4.m4.1.1.3.1.cmml" xref="S3.SS1.SSS1.p1.4.m4.1.1.3">subscript</csymbol><apply id="S3.SS1.SSS1.p1.4.m4.1.1.3.2.cmml" xref="S3.SS1.SSS1.p1.4.m4.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p1.4.m4.1.1.3.2.1.cmml" xref="S3.SS1.SSS1.p1.4.m4.1.1.3">superscript</csymbol><ci id="S3.SS1.SSS1.p1.4.m4.1.1.3.2.2.cmml" xref="S3.SS1.SSS1.p1.4.m4.1.1.3.2.2">𝐁</ci><ci id="S3.SS1.SSS1.p1.4.m4.1.1.3.2.3.cmml" xref="S3.SS1.SSS1.p1.4.m4.1.1.3.2.3">𝑘</ci></apply><ci id="S3.SS1.SSS1.p1.4.m4.1.1.3.3.cmml" xref="S3.SS1.SSS1.p1.4.m4.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS1.p1.4.m4.1c">\mathbf{A}^{k}_{i}\mathbf{B}^{k}_{i}</annotation></semantics></math> and <math id="S3.SS1.SSS1.p1.5.m5.1" class="ltx_Math" alttext="\mathbf{A}^{v}_{i}\mathbf{B}^{v}_{i}" display="inline"><semantics id="S3.SS1.SSS1.p1.5.m5.1a"><mrow id="S3.SS1.SSS1.p1.5.m5.1.1" xref="S3.SS1.SSS1.p1.5.m5.1.1.cmml"><msubsup id="S3.SS1.SSS1.p1.5.m5.1.1.2" xref="S3.SS1.SSS1.p1.5.m5.1.1.2.cmml"><mi id="S3.SS1.SSS1.p1.5.m5.1.1.2.2.2" xref="S3.SS1.SSS1.p1.5.m5.1.1.2.2.2.cmml">𝐀</mi><mi id="S3.SS1.SSS1.p1.5.m5.1.1.2.3" xref="S3.SS1.SSS1.p1.5.m5.1.1.2.3.cmml">i</mi><mi id="S3.SS1.SSS1.p1.5.m5.1.1.2.2.3" xref="S3.SS1.SSS1.p1.5.m5.1.1.2.2.3.cmml">v</mi></msubsup><mo lspace="0em" rspace="0em" id="S3.SS1.SSS1.p1.5.m5.1.1.1" xref="S3.SS1.SSS1.p1.5.m5.1.1.1.cmml">​</mo><msubsup id="S3.SS1.SSS1.p1.5.m5.1.1.3" xref="S3.SS1.SSS1.p1.5.m5.1.1.3.cmml"><mi id="S3.SS1.SSS1.p1.5.m5.1.1.3.2.2" xref="S3.SS1.SSS1.p1.5.m5.1.1.3.2.2.cmml">𝐁</mi><mi id="S3.SS1.SSS1.p1.5.m5.1.1.3.3" xref="S3.SS1.SSS1.p1.5.m5.1.1.3.3.cmml">i</mi><mi id="S3.SS1.SSS1.p1.5.m5.1.1.3.2.3" xref="S3.SS1.SSS1.p1.5.m5.1.1.3.2.3.cmml">v</mi></msubsup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS1.p1.5.m5.1b"><apply id="S3.SS1.SSS1.p1.5.m5.1.1.cmml" xref="S3.SS1.SSS1.p1.5.m5.1.1"><times id="S3.SS1.SSS1.p1.5.m5.1.1.1.cmml" xref="S3.SS1.SSS1.p1.5.m5.1.1.1"></times><apply id="S3.SS1.SSS1.p1.5.m5.1.1.2.cmml" xref="S3.SS1.SSS1.p1.5.m5.1.1.2"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p1.5.m5.1.1.2.1.cmml" xref="S3.SS1.SSS1.p1.5.m5.1.1.2">subscript</csymbol><apply id="S3.SS1.SSS1.p1.5.m5.1.1.2.2.cmml" xref="S3.SS1.SSS1.p1.5.m5.1.1.2"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p1.5.m5.1.1.2.2.1.cmml" xref="S3.SS1.SSS1.p1.5.m5.1.1.2">superscript</csymbol><ci id="S3.SS1.SSS1.p1.5.m5.1.1.2.2.2.cmml" xref="S3.SS1.SSS1.p1.5.m5.1.1.2.2.2">𝐀</ci><ci id="S3.SS1.SSS1.p1.5.m5.1.1.2.2.3.cmml" xref="S3.SS1.SSS1.p1.5.m5.1.1.2.2.3">𝑣</ci></apply><ci id="S3.SS1.SSS1.p1.5.m5.1.1.2.3.cmml" xref="S3.SS1.SSS1.p1.5.m5.1.1.2.3">𝑖</ci></apply><apply id="S3.SS1.SSS1.p1.5.m5.1.1.3.cmml" xref="S3.SS1.SSS1.p1.5.m5.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p1.5.m5.1.1.3.1.cmml" xref="S3.SS1.SSS1.p1.5.m5.1.1.3">subscript</csymbol><apply id="S3.SS1.SSS1.p1.5.m5.1.1.3.2.cmml" xref="S3.SS1.SSS1.p1.5.m5.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p1.5.m5.1.1.3.2.1.cmml" xref="S3.SS1.SSS1.p1.5.m5.1.1.3">superscript</csymbol><ci id="S3.SS1.SSS1.p1.5.m5.1.1.3.2.2.cmml" xref="S3.SS1.SSS1.p1.5.m5.1.1.3.2.2">𝐁</ci><ci id="S3.SS1.SSS1.p1.5.m5.1.1.3.2.3.cmml" xref="S3.SS1.SSS1.p1.5.m5.1.1.3.2.3">𝑣</ci></apply><ci id="S3.SS1.SSS1.p1.5.m5.1.1.3.3.cmml" xref="S3.SS1.SSS1.p1.5.m5.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS1.p1.5.m5.1c">\mathbf{A}^{v}_{i}\mathbf{B}^{v}_{i}</annotation></semantics></math>.</p>
</div>
<div id="S3.SS1.SSS1.p2" class="ltx_para">
<p id="S3.SS1.SSS1.p2.2" class="ltx_p">Based on the formula of attention output in equation <a href="#S2.E2" title="In 2.1 Multi-Head Attention Mechanism ‣ 2 Background ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, <span id="S3.SS1.SSS1.p2.2.1" class="ltx_text ltx_font_italic">Palu</span> absorbs the reconstruction matrix <math id="S3.SS1.SSS1.p2.1.m1.1" class="ltx_Math" alttext="\mathbf{B}^{v}_{i}" display="inline"><semantics id="S3.SS1.SSS1.p2.1.m1.1a"><msubsup id="S3.SS1.SSS1.p2.1.m1.1.1" xref="S3.SS1.SSS1.p2.1.m1.1.1.cmml"><mi id="S3.SS1.SSS1.p2.1.m1.1.1.2.2" xref="S3.SS1.SSS1.p2.1.m1.1.1.2.2.cmml">𝐁</mi><mi id="S3.SS1.SSS1.p2.1.m1.1.1.3" xref="S3.SS1.SSS1.p2.1.m1.1.1.3.cmml">i</mi><mi id="S3.SS1.SSS1.p2.1.m1.1.1.2.3" xref="S3.SS1.SSS1.p2.1.m1.1.1.2.3.cmml">v</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS1.p2.1.m1.1b"><apply id="S3.SS1.SSS1.p2.1.m1.1.1.cmml" xref="S3.SS1.SSS1.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p2.1.m1.1.1.1.cmml" xref="S3.SS1.SSS1.p2.1.m1.1.1">subscript</csymbol><apply id="S3.SS1.SSS1.p2.1.m1.1.1.2.cmml" xref="S3.SS1.SSS1.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p2.1.m1.1.1.2.1.cmml" xref="S3.SS1.SSS1.p2.1.m1.1.1">superscript</csymbol><ci id="S3.SS1.SSS1.p2.1.m1.1.1.2.2.cmml" xref="S3.SS1.SSS1.p2.1.m1.1.1.2.2">𝐁</ci><ci id="S3.SS1.SSS1.p2.1.m1.1.1.2.3.cmml" xref="S3.SS1.SSS1.p2.1.m1.1.1.2.3">𝑣</ci></apply><ci id="S3.SS1.SSS1.p2.1.m1.1.1.3.cmml" xref="S3.SS1.SSS1.p2.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS1.p2.1.m1.1c">\mathbf{B}^{v}_{i}</annotation></semantics></math> into the output projection matrix <math id="S3.SS1.SSS1.p2.2.m2.1" class="ltx_Math" alttext="\mathbf{W}_{i}^{o}" display="inline"><semantics id="S3.SS1.SSS1.p2.2.m2.1a"><msubsup id="S3.SS1.SSS1.p2.2.m2.1.1" xref="S3.SS1.SSS1.p2.2.m2.1.1.cmml"><mi id="S3.SS1.SSS1.p2.2.m2.1.1.2.2" xref="S3.SS1.SSS1.p2.2.m2.1.1.2.2.cmml">𝐖</mi><mi id="S3.SS1.SSS1.p2.2.m2.1.1.2.3" xref="S3.SS1.SSS1.p2.2.m2.1.1.2.3.cmml">i</mi><mi id="S3.SS1.SSS1.p2.2.m2.1.1.3" xref="S3.SS1.SSS1.p2.2.m2.1.1.3.cmml">o</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS1.p2.2.m2.1b"><apply id="S3.SS1.SSS1.p2.2.m2.1.1.cmml" xref="S3.SS1.SSS1.p2.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p2.2.m2.1.1.1.cmml" xref="S3.SS1.SSS1.p2.2.m2.1.1">superscript</csymbol><apply id="S3.SS1.SSS1.p2.2.m2.1.1.2.cmml" xref="S3.SS1.SSS1.p2.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p2.2.m2.1.1.2.1.cmml" xref="S3.SS1.SSS1.p2.2.m2.1.1">subscript</csymbol><ci id="S3.SS1.SSS1.p2.2.m2.1.1.2.2.cmml" xref="S3.SS1.SSS1.p2.2.m2.1.1.2.2">𝐖</ci><ci id="S3.SS1.SSS1.p2.2.m2.1.1.2.3.cmml" xref="S3.SS1.SSS1.p2.2.m2.1.1.2.3">𝑖</ci></apply><ci id="S3.SS1.SSS1.p2.2.m2.1.1.3.cmml" xref="S3.SS1.SSS1.p2.2.m2.1.1.3">𝑜</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS1.p2.2.m2.1c">\mathbf{W}_{i}^{o}</annotation></semantics></math> offline:</p>
</div>
<div id="S3.SS1.SSS1.p3" class="ltx_para">
<table id="S3.E5" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="S3.E5X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E5X.2.1.1.m1.1" class="ltx_Math" alttext="\displaystyle\mathbf{a}_{i}\mathbf{W}^{o}_{i}" display="inline"><semantics id="S3.E5X.2.1.1.m1.1a"><mrow id="S3.E5X.2.1.1.m1.1.1" xref="S3.E5X.2.1.1.m1.1.1.cmml"><msub id="S3.E5X.2.1.1.m1.1.1.2" xref="S3.E5X.2.1.1.m1.1.1.2.cmml"><mi id="S3.E5X.2.1.1.m1.1.1.2.2" xref="S3.E5X.2.1.1.m1.1.1.2.2.cmml">𝐚</mi><mi id="S3.E5X.2.1.1.m1.1.1.2.3" xref="S3.E5X.2.1.1.m1.1.1.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S3.E5X.2.1.1.m1.1.1.1" xref="S3.E5X.2.1.1.m1.1.1.1.cmml">​</mo><msubsup id="S3.E5X.2.1.1.m1.1.1.3" xref="S3.E5X.2.1.1.m1.1.1.3.cmml"><mi id="S3.E5X.2.1.1.m1.1.1.3.2.2" xref="S3.E5X.2.1.1.m1.1.1.3.2.2.cmml">𝐖</mi><mi id="S3.E5X.2.1.1.m1.1.1.3.3" xref="S3.E5X.2.1.1.m1.1.1.3.3.cmml">i</mi><mi id="S3.E5X.2.1.1.m1.1.1.3.2.3" xref="S3.E5X.2.1.1.m1.1.1.3.2.3.cmml">o</mi></msubsup></mrow><annotation-xml encoding="MathML-Content" id="S3.E5X.2.1.1.m1.1b"><apply id="S3.E5X.2.1.1.m1.1.1.cmml" xref="S3.E5X.2.1.1.m1.1.1"><times id="S3.E5X.2.1.1.m1.1.1.1.cmml" xref="S3.E5X.2.1.1.m1.1.1.1"></times><apply id="S3.E5X.2.1.1.m1.1.1.2.cmml" xref="S3.E5X.2.1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.E5X.2.1.1.m1.1.1.2.1.cmml" xref="S3.E5X.2.1.1.m1.1.1.2">subscript</csymbol><ci id="S3.E5X.2.1.1.m1.1.1.2.2.cmml" xref="S3.E5X.2.1.1.m1.1.1.2.2">𝐚</ci><ci id="S3.E5X.2.1.1.m1.1.1.2.3.cmml" xref="S3.E5X.2.1.1.m1.1.1.2.3">𝑖</ci></apply><apply id="S3.E5X.2.1.1.m1.1.1.3.cmml" xref="S3.E5X.2.1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.E5X.2.1.1.m1.1.1.3.1.cmml" xref="S3.E5X.2.1.1.m1.1.1.3">subscript</csymbol><apply id="S3.E5X.2.1.1.m1.1.1.3.2.cmml" xref="S3.E5X.2.1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.E5X.2.1.1.m1.1.1.3.2.1.cmml" xref="S3.E5X.2.1.1.m1.1.1.3">superscript</csymbol><ci id="S3.E5X.2.1.1.m1.1.1.3.2.2.cmml" xref="S3.E5X.2.1.1.m1.1.1.3.2.2">𝐖</ci><ci id="S3.E5X.2.1.1.m1.1.1.3.2.3.cmml" xref="S3.E5X.2.1.1.m1.1.1.3.2.3">𝑜</ci></apply><ci id="S3.E5X.2.1.1.m1.1.1.3.3.cmml" xref="S3.E5X.2.1.1.m1.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E5X.2.1.1.m1.1c">\displaystyle\mathbf{a}_{i}\mathbf{W}^{o}_{i}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E5X.3.2.2.m1.1" class="ltx_Math" alttext="\displaystyle=(\mathbf{p}_{i}\mathbf{V}_{i})\mathbf{W}^{o}_{i}" display="inline"><semantics id="S3.E5X.3.2.2.m1.1a"><mrow id="S3.E5X.3.2.2.m1.1.1" xref="S3.E5X.3.2.2.m1.1.1.cmml"><mi id="S3.E5X.3.2.2.m1.1.1.3" xref="S3.E5X.3.2.2.m1.1.1.3.cmml"></mi><mo id="S3.E5X.3.2.2.m1.1.1.2" xref="S3.E5X.3.2.2.m1.1.1.2.cmml">=</mo><mrow id="S3.E5X.3.2.2.m1.1.1.1" xref="S3.E5X.3.2.2.m1.1.1.1.cmml"><mrow id="S3.E5X.3.2.2.m1.1.1.1.1.1" xref="S3.E5X.3.2.2.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E5X.3.2.2.m1.1.1.1.1.1.2" xref="S3.E5X.3.2.2.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E5X.3.2.2.m1.1.1.1.1.1.1" xref="S3.E5X.3.2.2.m1.1.1.1.1.1.1.cmml"><msub id="S3.E5X.3.2.2.m1.1.1.1.1.1.1.2" xref="S3.E5X.3.2.2.m1.1.1.1.1.1.1.2.cmml"><mi id="S3.E5X.3.2.2.m1.1.1.1.1.1.1.2.2" xref="S3.E5X.3.2.2.m1.1.1.1.1.1.1.2.2.cmml">𝐩</mi><mi id="S3.E5X.3.2.2.m1.1.1.1.1.1.1.2.3" xref="S3.E5X.3.2.2.m1.1.1.1.1.1.1.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S3.E5X.3.2.2.m1.1.1.1.1.1.1.1" xref="S3.E5X.3.2.2.m1.1.1.1.1.1.1.1.cmml">​</mo><msub id="S3.E5X.3.2.2.m1.1.1.1.1.1.1.3" xref="S3.E5X.3.2.2.m1.1.1.1.1.1.1.3.cmml"><mi id="S3.E5X.3.2.2.m1.1.1.1.1.1.1.3.2" xref="S3.E5X.3.2.2.m1.1.1.1.1.1.1.3.2.cmml">𝐕</mi><mi id="S3.E5X.3.2.2.m1.1.1.1.1.1.1.3.3" xref="S3.E5X.3.2.2.m1.1.1.1.1.1.1.3.3.cmml">i</mi></msub></mrow><mo stretchy="false" id="S3.E5X.3.2.2.m1.1.1.1.1.1.3" xref="S3.E5X.3.2.2.m1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S3.E5X.3.2.2.m1.1.1.1.2" xref="S3.E5X.3.2.2.m1.1.1.1.2.cmml">​</mo><msubsup id="S3.E5X.3.2.2.m1.1.1.1.3" xref="S3.E5X.3.2.2.m1.1.1.1.3.cmml"><mi id="S3.E5X.3.2.2.m1.1.1.1.3.2.2" xref="S3.E5X.3.2.2.m1.1.1.1.3.2.2.cmml">𝐖</mi><mi id="S3.E5X.3.2.2.m1.1.1.1.3.3" xref="S3.E5X.3.2.2.m1.1.1.1.3.3.cmml">i</mi><mi id="S3.E5X.3.2.2.m1.1.1.1.3.2.3" xref="S3.E5X.3.2.2.m1.1.1.1.3.2.3.cmml">o</mi></msubsup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E5X.3.2.2.m1.1b"><apply id="S3.E5X.3.2.2.m1.1.1.cmml" xref="S3.E5X.3.2.2.m1.1.1"><eq id="S3.E5X.3.2.2.m1.1.1.2.cmml" xref="S3.E5X.3.2.2.m1.1.1.2"></eq><csymbol cd="latexml" id="S3.E5X.3.2.2.m1.1.1.3.cmml" xref="S3.E5X.3.2.2.m1.1.1.3">absent</csymbol><apply id="S3.E5X.3.2.2.m1.1.1.1.cmml" xref="S3.E5X.3.2.2.m1.1.1.1"><times id="S3.E5X.3.2.2.m1.1.1.1.2.cmml" xref="S3.E5X.3.2.2.m1.1.1.1.2"></times><apply id="S3.E5X.3.2.2.m1.1.1.1.1.1.1.cmml" xref="S3.E5X.3.2.2.m1.1.1.1.1.1"><times id="S3.E5X.3.2.2.m1.1.1.1.1.1.1.1.cmml" xref="S3.E5X.3.2.2.m1.1.1.1.1.1.1.1"></times><apply id="S3.E5X.3.2.2.m1.1.1.1.1.1.1.2.cmml" xref="S3.E5X.3.2.2.m1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E5X.3.2.2.m1.1.1.1.1.1.1.2.1.cmml" xref="S3.E5X.3.2.2.m1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.E5X.3.2.2.m1.1.1.1.1.1.1.2.2.cmml" xref="S3.E5X.3.2.2.m1.1.1.1.1.1.1.2.2">𝐩</ci><ci id="S3.E5X.3.2.2.m1.1.1.1.1.1.1.2.3.cmml" xref="S3.E5X.3.2.2.m1.1.1.1.1.1.1.2.3">𝑖</ci></apply><apply id="S3.E5X.3.2.2.m1.1.1.1.1.1.1.3.cmml" xref="S3.E5X.3.2.2.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E5X.3.2.2.m1.1.1.1.1.1.1.3.1.cmml" xref="S3.E5X.3.2.2.m1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.E5X.3.2.2.m1.1.1.1.1.1.1.3.2.cmml" xref="S3.E5X.3.2.2.m1.1.1.1.1.1.1.3.2">𝐕</ci><ci id="S3.E5X.3.2.2.m1.1.1.1.1.1.1.3.3.cmml" xref="S3.E5X.3.2.2.m1.1.1.1.1.1.1.3.3">𝑖</ci></apply></apply><apply id="S3.E5X.3.2.2.m1.1.1.1.3.cmml" xref="S3.E5X.3.2.2.m1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E5X.3.2.2.m1.1.1.1.3.1.cmml" xref="S3.E5X.3.2.2.m1.1.1.1.3">subscript</csymbol><apply id="S3.E5X.3.2.2.m1.1.1.1.3.2.cmml" xref="S3.E5X.3.2.2.m1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E5X.3.2.2.m1.1.1.1.3.2.1.cmml" xref="S3.E5X.3.2.2.m1.1.1.1.3">superscript</csymbol><ci id="S3.E5X.3.2.2.m1.1.1.1.3.2.2.cmml" xref="S3.E5X.3.2.2.m1.1.1.1.3.2.2">𝐖</ci><ci id="S3.E5X.3.2.2.m1.1.1.1.3.2.3.cmml" xref="S3.E5X.3.2.2.m1.1.1.1.3.2.3">𝑜</ci></apply><ci id="S3.E5X.3.2.2.m1.1.1.1.3.3.cmml" xref="S3.E5X.3.2.2.m1.1.1.1.3.3">𝑖</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E5X.3.2.2.m1.1c">\displaystyle=(\mathbf{p}_{i}\mathbf{V}_{i})\mathbf{W}^{o}_{i}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="3" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(5)</span></td>
</tr>
<tr id="S3.E5Xa" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E5Xa.2.1.1.m1.1" class="ltx_Math" alttext="\displaystyle=(\mathbf{p}_{i}\mathbf{H}^{v}_{i}\mathbf{B}^{v}_{i})\mathbf{W}^{o}_{i}" display="inline"><semantics id="S3.E5Xa.2.1.1.m1.1a"><mrow id="S3.E5Xa.2.1.1.m1.1.1" xref="S3.E5Xa.2.1.1.m1.1.1.cmml"><mi id="S3.E5Xa.2.1.1.m1.1.1.3" xref="S3.E5Xa.2.1.1.m1.1.1.3.cmml"></mi><mo id="S3.E5Xa.2.1.1.m1.1.1.2" xref="S3.E5Xa.2.1.1.m1.1.1.2.cmml">=</mo><mrow id="S3.E5Xa.2.1.1.m1.1.1.1" xref="S3.E5Xa.2.1.1.m1.1.1.1.cmml"><mrow id="S3.E5Xa.2.1.1.m1.1.1.1.1.1" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.2" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.cmml"><msub id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.2" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.2.cmml"><mi id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.2.2" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.2.2.cmml">𝐩</mi><mi id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.2.3" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.1" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.1.cmml">​</mo><msubsup id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.3" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.3.cmml"><mi id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.3.2.2" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.3.2.2.cmml">𝐇</mi><mi id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.3.3" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.3.3.cmml">i</mi><mi id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.3.2.3" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.3.2.3.cmml">v</mi></msubsup><mo lspace="0em" rspace="0em" id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.1a" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.1.cmml">​</mo><msubsup id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.4" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.4.cmml"><mi id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.4.2.2" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.4.2.2.cmml">𝐁</mi><mi id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.4.3" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.4.3.cmml">i</mi><mi id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.4.2.3" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.4.2.3.cmml">v</mi></msubsup></mrow><mo stretchy="false" id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.3" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S3.E5Xa.2.1.1.m1.1.1.1.2" xref="S3.E5Xa.2.1.1.m1.1.1.1.2.cmml">​</mo><msubsup id="S3.E5Xa.2.1.1.m1.1.1.1.3" xref="S3.E5Xa.2.1.1.m1.1.1.1.3.cmml"><mi id="S3.E5Xa.2.1.1.m1.1.1.1.3.2.2" xref="S3.E5Xa.2.1.1.m1.1.1.1.3.2.2.cmml">𝐖</mi><mi id="S3.E5Xa.2.1.1.m1.1.1.1.3.3" xref="S3.E5Xa.2.1.1.m1.1.1.1.3.3.cmml">i</mi><mi id="S3.E5Xa.2.1.1.m1.1.1.1.3.2.3" xref="S3.E5Xa.2.1.1.m1.1.1.1.3.2.3.cmml">o</mi></msubsup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E5Xa.2.1.1.m1.1b"><apply id="S3.E5Xa.2.1.1.m1.1.1.cmml" xref="S3.E5Xa.2.1.1.m1.1.1"><eq id="S3.E5Xa.2.1.1.m1.1.1.2.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.2"></eq><csymbol cd="latexml" id="S3.E5Xa.2.1.1.m1.1.1.3.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.3">absent</csymbol><apply id="S3.E5Xa.2.1.1.m1.1.1.1.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1"><times id="S3.E5Xa.2.1.1.m1.1.1.1.2.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1.2"></times><apply id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1"><times id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.1.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.1"></times><apply id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.2.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.2.1.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.2.2.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.2.2">𝐩</ci><ci id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.2.3.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.2.3">𝑖</ci></apply><apply id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.3.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.3.1.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.3">subscript</csymbol><apply id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.3.2.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.3.2.1.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.3">superscript</csymbol><ci id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.3.2.2.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.3.2.2">𝐇</ci><ci id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.3.2.3.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.3.2.3">𝑣</ci></apply><ci id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.3.3.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.3.3">𝑖</ci></apply><apply id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.4.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.4"><csymbol cd="ambiguous" id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.4.1.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.4">subscript</csymbol><apply id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.4.2.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.4"><csymbol cd="ambiguous" id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.4.2.1.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.4">superscript</csymbol><ci id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.4.2.2.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.4.2.2">𝐁</ci><ci id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.4.2.3.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.4.2.3">𝑣</ci></apply><ci id="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.4.3.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1.1.1.1.4.3">𝑖</ci></apply></apply><apply id="S3.E5Xa.2.1.1.m1.1.1.1.3.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E5Xa.2.1.1.m1.1.1.1.3.1.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1.3">subscript</csymbol><apply id="S3.E5Xa.2.1.1.m1.1.1.1.3.2.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E5Xa.2.1.1.m1.1.1.1.3.2.1.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1.3">superscript</csymbol><ci id="S3.E5Xa.2.1.1.m1.1.1.1.3.2.2.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1.3.2.2">𝐖</ci><ci id="S3.E5Xa.2.1.1.m1.1.1.1.3.2.3.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1.3.2.3">𝑜</ci></apply><ci id="S3.E5Xa.2.1.1.m1.1.1.1.3.3.cmml" xref="S3.E5Xa.2.1.1.m1.1.1.1.3.3">𝑖</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E5Xa.2.1.1.m1.1c">\displaystyle=(\mathbf{p}_{i}\mathbf{H}^{v}_{i}\mathbf{B}^{v}_{i})\mathbf{W}^{o}_{i}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr>
<tr id="S3.E5Xb" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E5Xb.2.1.1.m1.1" class="ltx_Math" alttext="\displaystyle=\mathbf{p}_{i}\mathbf{H}^{v}_{i}(\mathbf{B}^{v}_{i}\mathbf{W}^{o}_{i})" display="inline"><semantics id="S3.E5Xb.2.1.1.m1.1a"><mrow id="S3.E5Xb.2.1.1.m1.1.1" xref="S3.E5Xb.2.1.1.m1.1.1.cmml"><mi id="S3.E5Xb.2.1.1.m1.1.1.3" xref="S3.E5Xb.2.1.1.m1.1.1.3.cmml"></mi><mo id="S3.E5Xb.2.1.1.m1.1.1.2" xref="S3.E5Xb.2.1.1.m1.1.1.2.cmml">=</mo><mrow id="S3.E5Xb.2.1.1.m1.1.1.1" xref="S3.E5Xb.2.1.1.m1.1.1.1.cmml"><msub id="S3.E5Xb.2.1.1.m1.1.1.1.3" xref="S3.E5Xb.2.1.1.m1.1.1.1.3.cmml"><mi id="S3.E5Xb.2.1.1.m1.1.1.1.3.2" xref="S3.E5Xb.2.1.1.m1.1.1.1.3.2.cmml">𝐩</mi><mi id="S3.E5Xb.2.1.1.m1.1.1.1.3.3" xref="S3.E5Xb.2.1.1.m1.1.1.1.3.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S3.E5Xb.2.1.1.m1.1.1.1.2" xref="S3.E5Xb.2.1.1.m1.1.1.1.2.cmml">​</mo><msubsup id="S3.E5Xb.2.1.1.m1.1.1.1.4" xref="S3.E5Xb.2.1.1.m1.1.1.1.4.cmml"><mi id="S3.E5Xb.2.1.1.m1.1.1.1.4.2.2" xref="S3.E5Xb.2.1.1.m1.1.1.1.4.2.2.cmml">𝐇</mi><mi id="S3.E5Xb.2.1.1.m1.1.1.1.4.3" xref="S3.E5Xb.2.1.1.m1.1.1.1.4.3.cmml">i</mi><mi id="S3.E5Xb.2.1.1.m1.1.1.1.4.2.3" xref="S3.E5Xb.2.1.1.m1.1.1.1.4.2.3.cmml">v</mi></msubsup><mo lspace="0em" rspace="0em" id="S3.E5Xb.2.1.1.m1.1.1.1.2a" xref="S3.E5Xb.2.1.1.m1.1.1.1.2.cmml">​</mo><mrow id="S3.E5Xb.2.1.1.m1.1.1.1.1.1" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E5Xb.2.1.1.m1.1.1.1.1.1.2" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.cmml"><msubsup id="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.2" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.2.cmml"><mi id="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.2.2.2" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.2.2.2.cmml">𝐁</mi><mi id="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.2.3" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.2.3.cmml">i</mi><mi id="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.2.2.3" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.2.2.3.cmml">v</mi></msubsup><mo lspace="0em" rspace="0em" id="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.1" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.1.cmml">​</mo><msubsup id="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.3" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.3.cmml"><mi id="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.3.2.2" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.3.2.2.cmml">𝐖</mi><mi id="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.3.3" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.3.3.cmml">i</mi><mi id="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.3.2.3" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.3.2.3.cmml">o</mi></msubsup></mrow><mo stretchy="false" id="S3.E5Xb.2.1.1.m1.1.1.1.1.1.3" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E5Xb.2.1.1.m1.1b"><apply id="S3.E5Xb.2.1.1.m1.1.1.cmml" xref="S3.E5Xb.2.1.1.m1.1.1"><eq id="S3.E5Xb.2.1.1.m1.1.1.2.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.2"></eq><csymbol cd="latexml" id="S3.E5Xb.2.1.1.m1.1.1.3.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.3">absent</csymbol><apply id="S3.E5Xb.2.1.1.m1.1.1.1.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1"><times id="S3.E5Xb.2.1.1.m1.1.1.1.2.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1.2"></times><apply id="S3.E5Xb.2.1.1.m1.1.1.1.3.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E5Xb.2.1.1.m1.1.1.1.3.1.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1.3">subscript</csymbol><ci id="S3.E5Xb.2.1.1.m1.1.1.1.3.2.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1.3.2">𝐩</ci><ci id="S3.E5Xb.2.1.1.m1.1.1.1.3.3.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1.3.3">𝑖</ci></apply><apply id="S3.E5Xb.2.1.1.m1.1.1.1.4.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1.4"><csymbol cd="ambiguous" id="S3.E5Xb.2.1.1.m1.1.1.1.4.1.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1.4">subscript</csymbol><apply id="S3.E5Xb.2.1.1.m1.1.1.1.4.2.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1.4"><csymbol cd="ambiguous" id="S3.E5Xb.2.1.1.m1.1.1.1.4.2.1.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1.4">superscript</csymbol><ci id="S3.E5Xb.2.1.1.m1.1.1.1.4.2.2.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1.4.2.2">𝐇</ci><ci id="S3.E5Xb.2.1.1.m1.1.1.1.4.2.3.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1.4.2.3">𝑣</ci></apply><ci id="S3.E5Xb.2.1.1.m1.1.1.1.4.3.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1.4.3">𝑖</ci></apply><apply id="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1"><times id="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.1.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.1"></times><apply id="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.2.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.2.1.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.2">subscript</csymbol><apply id="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.2.2.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.2.2.1.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.2">superscript</csymbol><ci id="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.2.2.2.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.2.2.2">𝐁</ci><ci id="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.2.2.3.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.2.2.3">𝑣</ci></apply><ci id="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.2.3.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.2.3">𝑖</ci></apply><apply id="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.3.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.3.1.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.3">subscript</csymbol><apply id="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.3.2.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.3.2.1.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.3">superscript</csymbol><ci id="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.3.2.2.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.3.2.2">𝐖</ci><ci id="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.3.2.3.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.3.2.3">𝑜</ci></apply><ci id="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.3.3.cmml" xref="S3.E5Xb.2.1.1.m1.1.1.1.1.1.1.3.3">𝑖</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E5Xb.2.1.1.m1.1c">\displaystyle=\mathbf{p}_{i}\mathbf{H}^{v}_{i}(\mathbf{B}^{v}_{i}\mathbf{W}^{o}_{i})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr>
</tbody>
</table>
</div>
<div id="S3.SS1.SSS1.p4" class="ltx_para">
<p id="S3.SS1.SSS1.p4.2" class="ltx_p">Such fusion lets <span id="S3.SS1.SSS1.p4.2.1" class="ltx_text ltx_font_italic">Palu</span> skip the explicit reconstruction of the full value vectors, reduce the number of matrix multiplications, and improve efficiency.
A similar approach applies for calculating attention scores.
Matrix <math id="S3.SS1.SSS1.p4.1.m1.1" class="ltx_Math" alttext="\mathbf{B}^{k}_{i}" display="inline"><semantics id="S3.SS1.SSS1.p4.1.m1.1a"><msubsup id="S3.SS1.SSS1.p4.1.m1.1.1" xref="S3.SS1.SSS1.p4.1.m1.1.1.cmml"><mi id="S3.SS1.SSS1.p4.1.m1.1.1.2.2" xref="S3.SS1.SSS1.p4.1.m1.1.1.2.2.cmml">𝐁</mi><mi id="S3.SS1.SSS1.p4.1.m1.1.1.3" xref="S3.SS1.SSS1.p4.1.m1.1.1.3.cmml">i</mi><mi id="S3.SS1.SSS1.p4.1.m1.1.1.2.3" xref="S3.SS1.SSS1.p4.1.m1.1.1.2.3.cmml">k</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS1.p4.1.m1.1b"><apply id="S3.SS1.SSS1.p4.1.m1.1.1.cmml" xref="S3.SS1.SSS1.p4.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p4.1.m1.1.1.1.cmml" xref="S3.SS1.SSS1.p4.1.m1.1.1">subscript</csymbol><apply id="S3.SS1.SSS1.p4.1.m1.1.1.2.cmml" xref="S3.SS1.SSS1.p4.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p4.1.m1.1.1.2.1.cmml" xref="S3.SS1.SSS1.p4.1.m1.1.1">superscript</csymbol><ci id="S3.SS1.SSS1.p4.1.m1.1.1.2.2.cmml" xref="S3.SS1.SSS1.p4.1.m1.1.1.2.2">𝐁</ci><ci id="S3.SS1.SSS1.p4.1.m1.1.1.2.3.cmml" xref="S3.SS1.SSS1.p4.1.m1.1.1.2.3">𝑘</ci></apply><ci id="S3.SS1.SSS1.p4.1.m1.1.1.3.cmml" xref="S3.SS1.SSS1.p4.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS1.p4.1.m1.1c">\mathbf{B}^{k}_{i}</annotation></semantics></math> can be fused into the query projection matrix <math id="S3.SS1.SSS1.p4.2.m2.1" class="ltx_Math" alttext="\mathbf{W}_{i}^{q}" display="inline"><semantics id="S3.SS1.SSS1.p4.2.m2.1a"><msubsup id="S3.SS1.SSS1.p4.2.m2.1.1" xref="S3.SS1.SSS1.p4.2.m2.1.1.cmml"><mi id="S3.SS1.SSS1.p4.2.m2.1.1.2.2" xref="S3.SS1.SSS1.p4.2.m2.1.1.2.2.cmml">𝐖</mi><mi id="S3.SS1.SSS1.p4.2.m2.1.1.2.3" xref="S3.SS1.SSS1.p4.2.m2.1.1.2.3.cmml">i</mi><mi id="S3.SS1.SSS1.p4.2.m2.1.1.3" xref="S3.SS1.SSS1.p4.2.m2.1.1.3.cmml">q</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS1.p4.2.m2.1b"><apply id="S3.SS1.SSS1.p4.2.m2.1.1.cmml" xref="S3.SS1.SSS1.p4.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p4.2.m2.1.1.1.cmml" xref="S3.SS1.SSS1.p4.2.m2.1.1">superscript</csymbol><apply id="S3.SS1.SSS1.p4.2.m2.1.1.2.cmml" xref="S3.SS1.SSS1.p4.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p4.2.m2.1.1.2.1.cmml" xref="S3.SS1.SSS1.p4.2.m2.1.1">subscript</csymbol><ci id="S3.SS1.SSS1.p4.2.m2.1.1.2.2.cmml" xref="S3.SS1.SSS1.p4.2.m2.1.1.2.2">𝐖</ci><ci id="S3.SS1.SSS1.p4.2.m2.1.1.2.3.cmml" xref="S3.SS1.SSS1.p4.2.m2.1.1.2.3">𝑖</ci></apply><ci id="S3.SS1.SSS1.p4.2.m2.1.1.3.cmml" xref="S3.SS1.SSS1.p4.2.m2.1.1.3">𝑞</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS1.p4.2.m2.1c">\mathbf{W}_{i}^{q}</annotation></semantics></math> offline, as shown by</p>
</div>
<div id="S3.SS1.SSS1.p5" class="ltx_para">
<table id="S3.E6" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="S3.E6X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E6X.2.1.1.m1.1" class="ltx_Math" alttext="\displaystyle\mathbf{q}_{i}\mathbf{K}_{i}^{T}" display="inline"><semantics id="S3.E6X.2.1.1.m1.1a"><mrow id="S3.E6X.2.1.1.m1.1.1" xref="S3.E6X.2.1.1.m1.1.1.cmml"><msub id="S3.E6X.2.1.1.m1.1.1.2" xref="S3.E6X.2.1.1.m1.1.1.2.cmml"><mi id="S3.E6X.2.1.1.m1.1.1.2.2" xref="S3.E6X.2.1.1.m1.1.1.2.2.cmml">𝐪</mi><mi id="S3.E6X.2.1.1.m1.1.1.2.3" xref="S3.E6X.2.1.1.m1.1.1.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S3.E6X.2.1.1.m1.1.1.1" xref="S3.E6X.2.1.1.m1.1.1.1.cmml">​</mo><msubsup id="S3.E6X.2.1.1.m1.1.1.3" xref="S3.E6X.2.1.1.m1.1.1.3.cmml"><mi id="S3.E6X.2.1.1.m1.1.1.3.2.2" xref="S3.E6X.2.1.1.m1.1.1.3.2.2.cmml">𝐊</mi><mi id="S3.E6X.2.1.1.m1.1.1.3.2.3" xref="S3.E6X.2.1.1.m1.1.1.3.2.3.cmml">i</mi><mi id="S3.E6X.2.1.1.m1.1.1.3.3" xref="S3.E6X.2.1.1.m1.1.1.3.3.cmml">T</mi></msubsup></mrow><annotation-xml encoding="MathML-Content" id="S3.E6X.2.1.1.m1.1b"><apply id="S3.E6X.2.1.1.m1.1.1.cmml" xref="S3.E6X.2.1.1.m1.1.1"><times id="S3.E6X.2.1.1.m1.1.1.1.cmml" xref="S3.E6X.2.1.1.m1.1.1.1"></times><apply id="S3.E6X.2.1.1.m1.1.1.2.cmml" xref="S3.E6X.2.1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.E6X.2.1.1.m1.1.1.2.1.cmml" xref="S3.E6X.2.1.1.m1.1.1.2">subscript</csymbol><ci id="S3.E6X.2.1.1.m1.1.1.2.2.cmml" xref="S3.E6X.2.1.1.m1.1.1.2.2">𝐪</ci><ci id="S3.E6X.2.1.1.m1.1.1.2.3.cmml" xref="S3.E6X.2.1.1.m1.1.1.2.3">𝑖</ci></apply><apply id="S3.E6X.2.1.1.m1.1.1.3.cmml" xref="S3.E6X.2.1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.E6X.2.1.1.m1.1.1.3.1.cmml" xref="S3.E6X.2.1.1.m1.1.1.3">superscript</csymbol><apply id="S3.E6X.2.1.1.m1.1.1.3.2.cmml" xref="S3.E6X.2.1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.E6X.2.1.1.m1.1.1.3.2.1.cmml" xref="S3.E6X.2.1.1.m1.1.1.3">subscript</csymbol><ci id="S3.E6X.2.1.1.m1.1.1.3.2.2.cmml" xref="S3.E6X.2.1.1.m1.1.1.3.2.2">𝐊</ci><ci id="S3.E6X.2.1.1.m1.1.1.3.2.3.cmml" xref="S3.E6X.2.1.1.m1.1.1.3.2.3">𝑖</ci></apply><ci id="S3.E6X.2.1.1.m1.1.1.3.3.cmml" xref="S3.E6X.2.1.1.m1.1.1.3.3">𝑇</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E6X.2.1.1.m1.1c">\displaystyle\mathbf{q}_{i}\mathbf{K}_{i}^{T}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E6X.3.2.2.m1.1" class="ltx_Math" alttext="\displaystyle=\mathbf{q}_{i}(\mathbf{H}^{k}_{i}\mathbf{B}^{k}_{i})^{T}" display="inline"><semantics id="S3.E6X.3.2.2.m1.1a"><mrow id="S3.E6X.3.2.2.m1.1.1" xref="S3.E6X.3.2.2.m1.1.1.cmml"><mi id="S3.E6X.3.2.2.m1.1.1.3" xref="S3.E6X.3.2.2.m1.1.1.3.cmml"></mi><mo id="S3.E6X.3.2.2.m1.1.1.2" xref="S3.E6X.3.2.2.m1.1.1.2.cmml">=</mo><mrow id="S3.E6X.3.2.2.m1.1.1.1" xref="S3.E6X.3.2.2.m1.1.1.1.cmml"><msub id="S3.E6X.3.2.2.m1.1.1.1.3" xref="S3.E6X.3.2.2.m1.1.1.1.3.cmml"><mi id="S3.E6X.3.2.2.m1.1.1.1.3.2" xref="S3.E6X.3.2.2.m1.1.1.1.3.2.cmml">𝐪</mi><mi id="S3.E6X.3.2.2.m1.1.1.1.3.3" xref="S3.E6X.3.2.2.m1.1.1.1.3.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S3.E6X.3.2.2.m1.1.1.1.2" xref="S3.E6X.3.2.2.m1.1.1.1.2.cmml">​</mo><msup id="S3.E6X.3.2.2.m1.1.1.1.1" xref="S3.E6X.3.2.2.m1.1.1.1.1.cmml"><mrow id="S3.E6X.3.2.2.m1.1.1.1.1.1.1" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E6X.3.2.2.m1.1.1.1.1.1.1.2" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.cmml"><msubsup id="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.2" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.2.cmml"><mi id="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.2.2.2" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.2.2.2.cmml">𝐇</mi><mi id="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.2.3" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.2.3.cmml">i</mi><mi id="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.2.2.3" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.2.2.3.cmml">k</mi></msubsup><mo lspace="0em" rspace="0em" id="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.1" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.1.cmml">​</mo><msubsup id="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.3" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.3.cmml"><mi id="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.3.2.2" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.3.2.2.cmml">𝐁</mi><mi id="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.3.3" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.3.3.cmml">i</mi><mi id="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.3.2.3" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.3.2.3.cmml">k</mi></msubsup></mrow><mo stretchy="false" id="S3.E6X.3.2.2.m1.1.1.1.1.1.1.3" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mi id="S3.E6X.3.2.2.m1.1.1.1.1.3" xref="S3.E6X.3.2.2.m1.1.1.1.1.3.cmml">T</mi></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E6X.3.2.2.m1.1b"><apply id="S3.E6X.3.2.2.m1.1.1.cmml" xref="S3.E6X.3.2.2.m1.1.1"><eq id="S3.E6X.3.2.2.m1.1.1.2.cmml" xref="S3.E6X.3.2.2.m1.1.1.2"></eq><csymbol cd="latexml" id="S3.E6X.3.2.2.m1.1.1.3.cmml" xref="S3.E6X.3.2.2.m1.1.1.3">absent</csymbol><apply id="S3.E6X.3.2.2.m1.1.1.1.cmml" xref="S3.E6X.3.2.2.m1.1.1.1"><times id="S3.E6X.3.2.2.m1.1.1.1.2.cmml" xref="S3.E6X.3.2.2.m1.1.1.1.2"></times><apply id="S3.E6X.3.2.2.m1.1.1.1.3.cmml" xref="S3.E6X.3.2.2.m1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E6X.3.2.2.m1.1.1.1.3.1.cmml" xref="S3.E6X.3.2.2.m1.1.1.1.3">subscript</csymbol><ci id="S3.E6X.3.2.2.m1.1.1.1.3.2.cmml" xref="S3.E6X.3.2.2.m1.1.1.1.3.2">𝐪</ci><ci id="S3.E6X.3.2.2.m1.1.1.1.3.3.cmml" xref="S3.E6X.3.2.2.m1.1.1.1.3.3">𝑖</ci></apply><apply id="S3.E6X.3.2.2.m1.1.1.1.1.cmml" xref="S3.E6X.3.2.2.m1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E6X.3.2.2.m1.1.1.1.1.2.cmml" xref="S3.E6X.3.2.2.m1.1.1.1.1">superscript</csymbol><apply id="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.cmml" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1"><times id="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.1.cmml" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.1"></times><apply id="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.2.cmml" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.2">subscript</csymbol><apply id="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.2.2.1.cmml" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.2">superscript</csymbol><ci id="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.2.2.2.cmml" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.2.2.2">𝐇</ci><ci id="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.2.2.3.cmml" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.2.2.3">𝑘</ci></apply><ci id="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.2.3">𝑖</ci></apply><apply id="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.3.cmml" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.3">subscript</csymbol><apply id="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.3.2.1.cmml" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.3">superscript</csymbol><ci id="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.3.2.2.cmml" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.3.2.2">𝐁</ci><ci id="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.3.2.3.cmml" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.3.2.3">𝑘</ci></apply><ci id="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.3.3.cmml" xref="S3.E6X.3.2.2.m1.1.1.1.1.1.1.1.3.3">𝑖</ci></apply></apply><ci id="S3.E6X.3.2.2.m1.1.1.1.1.3.cmml" xref="S3.E6X.3.2.2.m1.1.1.1.1.3">𝑇</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E6X.3.2.2.m1.1c">\displaystyle=\mathbf{q}_{i}(\mathbf{H}^{k}_{i}\mathbf{B}^{k}_{i})^{T}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="3" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(6)</span></td>
</tr>
<tr id="S3.E6Xa" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E6Xa.2.1.1.m1.2" class="ltx_Math" alttext="\displaystyle=\mathbf{x}_{t}\mathbf{W}^{q}_{i}(\mathbf{B}^{k}_{i})^{T}(\mathbf{H}^{k}_{i})^{T}" display="inline"><semantics id="S3.E6Xa.2.1.1.m1.2a"><mrow id="S3.E6Xa.2.1.1.m1.2.2" xref="S3.E6Xa.2.1.1.m1.2.2.cmml"><mi id="S3.E6Xa.2.1.1.m1.2.2.4" xref="S3.E6Xa.2.1.1.m1.2.2.4.cmml"></mi><mo id="S3.E6Xa.2.1.1.m1.2.2.3" xref="S3.E6Xa.2.1.1.m1.2.2.3.cmml">=</mo><mrow id="S3.E6Xa.2.1.1.m1.2.2.2" xref="S3.E6Xa.2.1.1.m1.2.2.2.cmml"><msub id="S3.E6Xa.2.1.1.m1.2.2.2.4" xref="S3.E6Xa.2.1.1.m1.2.2.2.4.cmml"><mi id="S3.E6Xa.2.1.1.m1.2.2.2.4.2" xref="S3.E6Xa.2.1.1.m1.2.2.2.4.2.cmml">𝐱</mi><mi id="S3.E6Xa.2.1.1.m1.2.2.2.4.3" xref="S3.E6Xa.2.1.1.m1.2.2.2.4.3.cmml">t</mi></msub><mo lspace="0em" rspace="0em" id="S3.E6Xa.2.1.1.m1.2.2.2.3" xref="S3.E6Xa.2.1.1.m1.2.2.2.3.cmml">​</mo><msubsup id="S3.E6Xa.2.1.1.m1.2.2.2.5" xref="S3.E6Xa.2.1.1.m1.2.2.2.5.cmml"><mi id="S3.E6Xa.2.1.1.m1.2.2.2.5.2.2" xref="S3.E6Xa.2.1.1.m1.2.2.2.5.2.2.cmml">𝐖</mi><mi id="S3.E6Xa.2.1.1.m1.2.2.2.5.3" xref="S3.E6Xa.2.1.1.m1.2.2.2.5.3.cmml">i</mi><mi id="S3.E6Xa.2.1.1.m1.2.2.2.5.2.3" xref="S3.E6Xa.2.1.1.m1.2.2.2.5.2.3.cmml">q</mi></msubsup><mo lspace="0em" rspace="0em" id="S3.E6Xa.2.1.1.m1.2.2.2.3a" xref="S3.E6Xa.2.1.1.m1.2.2.2.3.cmml">​</mo><msup id="S3.E6Xa.2.1.1.m1.1.1.1.1" xref="S3.E6Xa.2.1.1.m1.1.1.1.1.cmml"><mrow id="S3.E6Xa.2.1.1.m1.1.1.1.1.1.1" xref="S3.E6Xa.2.1.1.m1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E6Xa.2.1.1.m1.1.1.1.1.1.1.2" xref="S3.E6Xa.2.1.1.m1.1.1.1.1.1.1.1.cmml">(</mo><msubsup id="S3.E6Xa.2.1.1.m1.1.1.1.1.1.1.1" xref="S3.E6Xa.2.1.1.m1.1.1.1.1.1.1.1.cmml"><mi id="S3.E6Xa.2.1.1.m1.1.1.1.1.1.1.1.2.2" xref="S3.E6Xa.2.1.1.m1.1.1.1.1.1.1.1.2.2.cmml">𝐁</mi><mi id="S3.E6Xa.2.1.1.m1.1.1.1.1.1.1.1.3" xref="S3.E6Xa.2.1.1.m1.1.1.1.1.1.1.1.3.cmml">i</mi><mi id="S3.E6Xa.2.1.1.m1.1.1.1.1.1.1.1.2.3" xref="S3.E6Xa.2.1.1.m1.1.1.1.1.1.1.1.2.3.cmml">k</mi></msubsup><mo stretchy="false" id="S3.E6Xa.2.1.1.m1.1.1.1.1.1.1.3" xref="S3.E6Xa.2.1.1.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mi id="S3.E6Xa.2.1.1.m1.1.1.1.1.3" xref="S3.E6Xa.2.1.1.m1.1.1.1.1.3.cmml">T</mi></msup><mo lspace="0em" rspace="0em" id="S3.E6Xa.2.1.1.m1.2.2.2.3b" xref="S3.E6Xa.2.1.1.m1.2.2.2.3.cmml">​</mo><msup id="S3.E6Xa.2.1.1.m1.2.2.2.2" xref="S3.E6Xa.2.1.1.m1.2.2.2.2.cmml"><mrow id="S3.E6Xa.2.1.1.m1.2.2.2.2.1.1" xref="S3.E6Xa.2.1.1.m1.2.2.2.2.1.1.1.cmml"><mo stretchy="false" id="S3.E6Xa.2.1.1.m1.2.2.2.2.1.1.2" xref="S3.E6Xa.2.1.1.m1.2.2.2.2.1.1.1.cmml">(</mo><msubsup id="S3.E6Xa.2.1.1.m1.2.2.2.2.1.1.1" xref="S3.E6Xa.2.1.1.m1.2.2.2.2.1.1.1.cmml"><mi id="S3.E6Xa.2.1.1.m1.2.2.2.2.1.1.1.2.2" xref="S3.E6Xa.2.1.1.m1.2.2.2.2.1.1.1.2.2.cmml">𝐇</mi><mi id="S3.E6Xa.2.1.1.m1.2.2.2.2.1.1.1.3" xref="S3.E6Xa.2.1.1.m1.2.2.2.2.1.1.1.3.cmml">i</mi><mi id="S3.E6Xa.2.1.1.m1.2.2.2.2.1.1.1.2.3" xref="S3.E6Xa.2.1.1.m1.2.2.2.2.1.1.1.2.3.cmml">k</mi></msubsup><mo stretchy="false" id="S3.E6Xa.2.1.1.m1.2.2.2.2.1.1.3" xref="S3.E6Xa.2.1.1.m1.2.2.2.2.1.1.1.cmml">)</mo></mrow><mi id="S3.E6Xa.2.1.1.m1.2.2.2.2.3" xref="S3.E6Xa.2.1.1.m1.2.2.2.2.3.cmml">T</mi></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E6Xa.2.1.1.m1.2b"><apply id="S3.E6Xa.2.1.1.m1.2.2.cmml" xref="S3.E6Xa.2.1.1.m1.2.2"><eq id="S3.E6Xa.2.1.1.m1.2.2.3.cmml" xref="S3.E6Xa.2.1.1.m1.2.2.3"></eq><csymbol cd="latexml" id="S3.E6Xa.2.1.1.m1.2.2.4.cmml" xref="S3.E6Xa.2.1.1.m1.2.2.4">absent</csymbol><apply id="S3.E6Xa.2.1.1.m1.2.2.2.cmml" xref="S3.E6Xa.2.1.1.m1.2.2.2"><times id="S3.E6Xa.2.1.1.m1.2.2.2.3.cmml" xref="S3.E6Xa.2.1.1.m1.2.2.2.3"></times><apply id="S3.E6Xa.2.1.1.m1.2.2.2.4.cmml" xref="S3.E6Xa.2.1.1.m1.2.2.2.4"><csymbol cd="ambiguous" id="S3.E6Xa.2.1.1.m1.2.2.2.4.1.cmml" xref="S3.E6Xa.2.1.1.m1.2.2.2.4">subscript</csymbol><ci id="S3.E6Xa.2.1.1.m1.2.2.2.4.2.cmml" xref="S3.E6Xa.2.1.1.m1.2.2.2.4.2">𝐱</ci><ci id="S3.E6Xa.2.1.1.m1.2.2.2.4.3.cmml" xref="S3.E6Xa.2.1.1.m1.2.2.2.4.3">𝑡</ci></apply><apply id="S3.E6Xa.2.1.1.m1.2.2.2.5.cmml" xref="S3.E6Xa.2.1.1.m1.2.2.2.5"><csymbol cd="ambiguous" id="S3.E6Xa.2.1.1.m1.2.2.2.5.1.cmml" xref="S3.E6Xa.2.1.1.m1.2.2.2.5">subscript</csymbol><apply id="S3.E6Xa.2.1.1.m1.2.2.2.5.2.cmml" xref="S3.E6Xa.2.1.1.m1.2.2.2.5"><csymbol cd="ambiguous" id="S3.E6Xa.2.1.1.m1.2.2.2.5.2.1.cmml" xref="S3.E6Xa.2.1.1.m1.2.2.2.5">superscript</csymbol><ci id="S3.E6Xa.2.1.1.m1.2.2.2.5.2.2.cmml" xref="S3.E6Xa.2.1.1.m1.2.2.2.5.2.2">𝐖</ci><ci id="S3.E6Xa.2.1.1.m1.2.2.2.5.2.3.cmml" xref="S3.E6Xa.2.1.1.m1.2.2.2.5.2.3">𝑞</ci></apply><ci id="S3.E6Xa.2.1.1.m1.2.2.2.5.3.cmml" xref="S3.E6Xa.2.1.1.m1.2.2.2.5.3">𝑖</ci></apply><apply id="S3.E6Xa.2.1.1.m1.1.1.1.1.cmml" xref="S3.E6Xa.2.1.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E6Xa.2.1.1.m1.1.1.1.1.2.cmml" xref="S3.E6Xa.2.1.1.m1.1.1.1.1">superscript</csymbol><apply id="S3.E6Xa.2.1.1.m1.1.1.1.1.1.1.1.cmml" xref="S3.E6Xa.2.1.1.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E6Xa.2.1.1.m1.1.1.1.1.1.1.1.1.cmml" xref="S3.E6Xa.2.1.1.m1.1.1.1.1.1.1">subscript</csymbol><apply id="S3.E6Xa.2.1.1.m1.1.1.1.1.1.1.1.2.cmml" xref="S3.E6Xa.2.1.1.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E6Xa.2.1.1.m1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.E6Xa.2.1.1.m1.1.1.1.1.1.1">superscript</csymbol><ci id="S3.E6Xa.2.1.1.m1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.E6Xa.2.1.1.m1.1.1.1.1.1.1.1.2.2">𝐁</ci><ci id="S3.E6Xa.2.1.1.m1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.E6Xa.2.1.1.m1.1.1.1.1.1.1.1.2.3">𝑘</ci></apply><ci id="S3.E6Xa.2.1.1.m1.1.1.1.1.1.1.1.3.cmml" xref="S3.E6Xa.2.1.1.m1.1.1.1.1.1.1.1.3">𝑖</ci></apply><ci id="S3.E6Xa.2.1.1.m1.1.1.1.1.3.cmml" xref="S3.E6Xa.2.1.1.m1.1.1.1.1.3">𝑇</ci></apply><apply id="S3.E6Xa.2.1.1.m1.2.2.2.2.cmml" xref="S3.E6Xa.2.1.1.m1.2.2.2.2"><csymbol cd="ambiguous" id="S3.E6Xa.2.1.1.m1.2.2.2.2.2.cmml" xref="S3.E6Xa.2.1.1.m1.2.2.2.2">superscript</csymbol><apply id="S3.E6Xa.2.1.1.m1.2.2.2.2.1.1.1.cmml" xref="S3.E6Xa.2.1.1.m1.2.2.2.2.1.1"><csymbol cd="ambiguous" id="S3.E6Xa.2.1.1.m1.2.2.2.2.1.1.1.1.cmml" xref="S3.E6Xa.2.1.1.m1.2.2.2.2.1.1">subscript</csymbol><apply id="S3.E6Xa.2.1.1.m1.2.2.2.2.1.1.1.2.cmml" xref="S3.E6Xa.2.1.1.m1.2.2.2.2.1.1"><csymbol cd="ambiguous" id="S3.E6Xa.2.1.1.m1.2.2.2.2.1.1.1.2.1.cmml" xref="S3.E6Xa.2.1.1.m1.2.2.2.2.1.1">superscript</csymbol><ci id="S3.E6Xa.2.1.1.m1.2.2.2.2.1.1.1.2.2.cmml" xref="S3.E6Xa.2.1.1.m1.2.2.2.2.1.1.1.2.2">𝐇</ci><ci id="S3.E6Xa.2.1.1.m1.2.2.2.2.1.1.1.2.3.cmml" xref="S3.E6Xa.2.1.1.m1.2.2.2.2.1.1.1.2.3">𝑘</ci></apply><ci id="S3.E6Xa.2.1.1.m1.2.2.2.2.1.1.1.3.cmml" xref="S3.E6Xa.2.1.1.m1.2.2.2.2.1.1.1.3">𝑖</ci></apply><ci id="S3.E6Xa.2.1.1.m1.2.2.2.2.3.cmml" xref="S3.E6Xa.2.1.1.m1.2.2.2.2.3">𝑇</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E6Xa.2.1.1.m1.2c">\displaystyle=\mathbf{x}_{t}\mathbf{W}^{q}_{i}(\mathbf{B}^{k}_{i})^{T}(\mathbf{H}^{k}_{i})^{T}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr>
<tr id="S3.E6Xb" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E6Xb.2.1.1.m1.1" class="ltx_Math" alttext="\displaystyle=\mathbf{x}_{t}\Bigl{(}\mathbf{W}^{q}_{i}(\mathbf{B}^{k}_{i})^{T}\Bigr{)}(\mathbf{H}^{k}_{i})^{T}." display="inline"><semantics id="S3.E6Xb.2.1.1.m1.1a"><mrow id="S3.E6Xb.2.1.1.m1.1.1.1" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.cmml"><mrow id="S3.E6Xb.2.1.1.m1.1.1.1.1" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.cmml"><mi id="S3.E6Xb.2.1.1.m1.1.1.1.1.4" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.4.cmml"></mi><mo id="S3.E6Xb.2.1.1.m1.1.1.1.1.3" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.3.cmml">=</mo><mrow id="S3.E6Xb.2.1.1.m1.1.1.1.1.2" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.cmml"><msub id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.4" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.4.cmml"><mi id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.4.2" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.4.2.cmml">𝐱</mi><mi id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.4.3" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.4.3.cmml">t</mi></msub><mo lspace="0em" rspace="0em" id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.3" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.3.cmml">​</mo><mrow id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.cmml"><mo maxsize="160%" minsize="160%" id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.2" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.cmml"><msubsup id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.3" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.3.2.2" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.3.2.2.cmml">𝐖</mi><mi id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.3.3" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.3.3.cmml">i</mi><mi id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.3.2.3" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.3.2.3.cmml">q</mi></msubsup><mo lspace="0em" rspace="0em" id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.2" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.2.cmml">​</mo><msup id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.cmml"><mrow id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><msubsup id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.2" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml">𝐁</mi><mi id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml">i</mi><mi id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.3" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml">k</mi></msubsup><mo stretchy="false" id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mi id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.3" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.3.cmml">T</mi></msup></mrow><mo maxsize="160%" minsize="160%" id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.3" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.3a" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.3.cmml">​</mo><msup id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.cmml"><mrow id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.1.1" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.1.1.1.cmml"><mo stretchy="false" id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.1.1.2" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.1.1.1.cmml">(</mo><msubsup id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.1.1.1" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.1.1.1.cmml"><mi id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.1.1.1.2.2" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.1.1.1.2.2.cmml">𝐇</mi><mi id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.1.1.1.3" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.1.1.1.3.cmml">i</mi><mi id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.1.1.1.2.3" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.1.1.1.2.3.cmml">k</mi></msubsup><mo stretchy="false" id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.1.1.3" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.1.1.1.cmml">)</mo></mrow><mi id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.3" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.3.cmml">T</mi></msup></mrow></mrow><mo lspace="0em" id="S3.E6Xb.2.1.1.m1.1.1.1.2" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E6Xb.2.1.1.m1.1b"><apply id="S3.E6Xb.2.1.1.m1.1.1.1.1.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1"><eq id="S3.E6Xb.2.1.1.m1.1.1.1.1.3.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.3"></eq><csymbol cd="latexml" id="S3.E6Xb.2.1.1.m1.1.1.1.1.4.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.4">absent</csymbol><apply id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2"><times id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.3.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.3"></times><apply id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.4.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.4"><csymbol cd="ambiguous" id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.4.1.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.4">subscript</csymbol><ci id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.4.2.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.4.2">𝐱</ci><ci id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.4.3.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.4.3">𝑡</ci></apply><apply id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1"><times id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.2"></times><apply id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.3">subscript</csymbol><apply id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.3.2.1.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.3">superscript</csymbol><ci id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.3.2.2.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.3.2.2">𝐖</ci><ci id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.3.2.3.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.3.2.3">𝑞</ci></apply><ci id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.3.3">𝑖</ci></apply><apply id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1">superscript</csymbol><apply id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1">subscript</csymbol><apply id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1">superscript</csymbol><ci id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.2">𝐁</ci><ci id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.3">𝑘</ci></apply><ci id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.3">𝑖</ci></apply><ci id="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.1.1.1.1.1.3">𝑇</ci></apply></apply><apply id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.2.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2">superscript</csymbol><apply id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.1.1.1.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.1.1"><csymbol cd="ambiguous" id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.1.1.1.1.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.1.1">subscript</csymbol><apply id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.1.1.1.2.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.1.1"><csymbol cd="ambiguous" id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.1.1.1.2.1.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.1.1">superscript</csymbol><ci id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.1.1.1.2.2.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.1.1.1.2.2">𝐇</ci><ci id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.1.1.1.2.3.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.1.1.1.2.3">𝑘</ci></apply><ci id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.1.1.1.3.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.1.1.1.3">𝑖</ci></apply><ci id="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.3.cmml" xref="S3.E6Xb.2.1.1.m1.1.1.1.1.2.2.3">𝑇</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E6Xb.2.1.1.m1.1c">\displaystyle=\mathbf{x}_{t}\Bigl{(}\mathbf{W}^{q}_{i}(\mathbf{B}^{k}_{i})^{T}\Bigr{)}(\mathbf{H}^{k}_{i})^{T}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr>
</tbody>
</table>
<p id="S3.SS1.SSS1.p5.4" class="ltx_p">Here, <math id="S3.SS1.SSS1.p5.1.m1.1" class="ltx_Math" alttext="\mathbf{B}^{k}_{i}\in\mathbb{R}^{r\times d_{h}}" display="inline"><semantics id="S3.SS1.SSS1.p5.1.m1.1a"><mrow id="S3.SS1.SSS1.p5.1.m1.1.1" xref="S3.SS1.SSS1.p5.1.m1.1.1.cmml"><msubsup id="S3.SS1.SSS1.p5.1.m1.1.1.2" xref="S3.SS1.SSS1.p5.1.m1.1.1.2.cmml"><mi id="S3.SS1.SSS1.p5.1.m1.1.1.2.2.2" xref="S3.SS1.SSS1.p5.1.m1.1.1.2.2.2.cmml">𝐁</mi><mi id="S3.SS1.SSS1.p5.1.m1.1.1.2.3" xref="S3.SS1.SSS1.p5.1.m1.1.1.2.3.cmml">i</mi><mi id="S3.SS1.SSS1.p5.1.m1.1.1.2.2.3" xref="S3.SS1.SSS1.p5.1.m1.1.1.2.2.3.cmml">k</mi></msubsup><mo id="S3.SS1.SSS1.p5.1.m1.1.1.1" xref="S3.SS1.SSS1.p5.1.m1.1.1.1.cmml">∈</mo><msup id="S3.SS1.SSS1.p5.1.m1.1.1.3" xref="S3.SS1.SSS1.p5.1.m1.1.1.3.cmml"><mi id="S3.SS1.SSS1.p5.1.m1.1.1.3.2" xref="S3.SS1.SSS1.p5.1.m1.1.1.3.2.cmml">ℝ</mi><mrow id="S3.SS1.SSS1.p5.1.m1.1.1.3.3" xref="S3.SS1.SSS1.p5.1.m1.1.1.3.3.cmml"><mi id="S3.SS1.SSS1.p5.1.m1.1.1.3.3.2" xref="S3.SS1.SSS1.p5.1.m1.1.1.3.3.2.cmml">r</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS1.SSS1.p5.1.m1.1.1.3.3.1" xref="S3.SS1.SSS1.p5.1.m1.1.1.3.3.1.cmml">×</mo><msub id="S3.SS1.SSS1.p5.1.m1.1.1.3.3.3" xref="S3.SS1.SSS1.p5.1.m1.1.1.3.3.3.cmml"><mi id="S3.SS1.SSS1.p5.1.m1.1.1.3.3.3.2" xref="S3.SS1.SSS1.p5.1.m1.1.1.3.3.3.2.cmml">d</mi><mi id="S3.SS1.SSS1.p5.1.m1.1.1.3.3.3.3" xref="S3.SS1.SSS1.p5.1.m1.1.1.3.3.3.3.cmml">h</mi></msub></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS1.p5.1.m1.1b"><apply id="S3.SS1.SSS1.p5.1.m1.1.1.cmml" xref="S3.SS1.SSS1.p5.1.m1.1.1"><in id="S3.SS1.SSS1.p5.1.m1.1.1.1.cmml" xref="S3.SS1.SSS1.p5.1.m1.1.1.1"></in><apply id="S3.SS1.SSS1.p5.1.m1.1.1.2.cmml" xref="S3.SS1.SSS1.p5.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p5.1.m1.1.1.2.1.cmml" xref="S3.SS1.SSS1.p5.1.m1.1.1.2">subscript</csymbol><apply id="S3.SS1.SSS1.p5.1.m1.1.1.2.2.cmml" xref="S3.SS1.SSS1.p5.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p5.1.m1.1.1.2.2.1.cmml" xref="S3.SS1.SSS1.p5.1.m1.1.1.2">superscript</csymbol><ci id="S3.SS1.SSS1.p5.1.m1.1.1.2.2.2.cmml" xref="S3.SS1.SSS1.p5.1.m1.1.1.2.2.2">𝐁</ci><ci id="S3.SS1.SSS1.p5.1.m1.1.1.2.2.3.cmml" xref="S3.SS1.SSS1.p5.1.m1.1.1.2.2.3">𝑘</ci></apply><ci id="S3.SS1.SSS1.p5.1.m1.1.1.2.3.cmml" xref="S3.SS1.SSS1.p5.1.m1.1.1.2.3">𝑖</ci></apply><apply id="S3.SS1.SSS1.p5.1.m1.1.1.3.cmml" xref="S3.SS1.SSS1.p5.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p5.1.m1.1.1.3.1.cmml" xref="S3.SS1.SSS1.p5.1.m1.1.1.3">superscript</csymbol><ci id="S3.SS1.SSS1.p5.1.m1.1.1.3.2.cmml" xref="S3.SS1.SSS1.p5.1.m1.1.1.3.2">ℝ</ci><apply id="S3.SS1.SSS1.p5.1.m1.1.1.3.3.cmml" xref="S3.SS1.SSS1.p5.1.m1.1.1.3.3"><times id="S3.SS1.SSS1.p5.1.m1.1.1.3.3.1.cmml" xref="S3.SS1.SSS1.p5.1.m1.1.1.3.3.1"></times><ci id="S3.SS1.SSS1.p5.1.m1.1.1.3.3.2.cmml" xref="S3.SS1.SSS1.p5.1.m1.1.1.3.3.2">𝑟</ci><apply id="S3.SS1.SSS1.p5.1.m1.1.1.3.3.3.cmml" xref="S3.SS1.SSS1.p5.1.m1.1.1.3.3.3"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p5.1.m1.1.1.3.3.3.1.cmml" xref="S3.SS1.SSS1.p5.1.m1.1.1.3.3.3">subscript</csymbol><ci id="S3.SS1.SSS1.p5.1.m1.1.1.3.3.3.2.cmml" xref="S3.SS1.SSS1.p5.1.m1.1.1.3.3.3.2">𝑑</ci><ci id="S3.SS1.SSS1.p5.1.m1.1.1.3.3.3.3.cmml" xref="S3.SS1.SSS1.p5.1.m1.1.1.3.3.3.3">ℎ</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS1.p5.1.m1.1c">\mathbf{B}^{k}_{i}\in\mathbb{R}^{r\times d_{h}}</annotation></semantics></math> and <math id="S3.SS1.SSS1.p5.2.m2.1" class="ltx_Math" alttext="\mathbf{W}^{q}_{i}\in\mathbb{R}^{d\times d_{h}}" display="inline"><semantics id="S3.SS1.SSS1.p5.2.m2.1a"><mrow id="S3.SS1.SSS1.p5.2.m2.1.1" xref="S3.SS1.SSS1.p5.2.m2.1.1.cmml"><msubsup id="S3.SS1.SSS1.p5.2.m2.1.1.2" xref="S3.SS1.SSS1.p5.2.m2.1.1.2.cmml"><mi id="S3.SS1.SSS1.p5.2.m2.1.1.2.2.2" xref="S3.SS1.SSS1.p5.2.m2.1.1.2.2.2.cmml">𝐖</mi><mi id="S3.SS1.SSS1.p5.2.m2.1.1.2.3" xref="S3.SS1.SSS1.p5.2.m2.1.1.2.3.cmml">i</mi><mi id="S3.SS1.SSS1.p5.2.m2.1.1.2.2.3" xref="S3.SS1.SSS1.p5.2.m2.1.1.2.2.3.cmml">q</mi></msubsup><mo id="S3.SS1.SSS1.p5.2.m2.1.1.1" xref="S3.SS1.SSS1.p5.2.m2.1.1.1.cmml">∈</mo><msup id="S3.SS1.SSS1.p5.2.m2.1.1.3" xref="S3.SS1.SSS1.p5.2.m2.1.1.3.cmml"><mi id="S3.SS1.SSS1.p5.2.m2.1.1.3.2" xref="S3.SS1.SSS1.p5.2.m2.1.1.3.2.cmml">ℝ</mi><mrow id="S3.SS1.SSS1.p5.2.m2.1.1.3.3" xref="S3.SS1.SSS1.p5.2.m2.1.1.3.3.cmml"><mi id="S3.SS1.SSS1.p5.2.m2.1.1.3.3.2" xref="S3.SS1.SSS1.p5.2.m2.1.1.3.3.2.cmml">d</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS1.SSS1.p5.2.m2.1.1.3.3.1" xref="S3.SS1.SSS1.p5.2.m2.1.1.3.3.1.cmml">×</mo><msub id="S3.SS1.SSS1.p5.2.m2.1.1.3.3.3" xref="S3.SS1.SSS1.p5.2.m2.1.1.3.3.3.cmml"><mi id="S3.SS1.SSS1.p5.2.m2.1.1.3.3.3.2" xref="S3.SS1.SSS1.p5.2.m2.1.1.3.3.3.2.cmml">d</mi><mi id="S3.SS1.SSS1.p5.2.m2.1.1.3.3.3.3" xref="S3.SS1.SSS1.p5.2.m2.1.1.3.3.3.3.cmml">h</mi></msub></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS1.p5.2.m2.1b"><apply id="S3.SS1.SSS1.p5.2.m2.1.1.cmml" xref="S3.SS1.SSS1.p5.2.m2.1.1"><in id="S3.SS1.SSS1.p5.2.m2.1.1.1.cmml" xref="S3.SS1.SSS1.p5.2.m2.1.1.1"></in><apply id="S3.SS1.SSS1.p5.2.m2.1.1.2.cmml" xref="S3.SS1.SSS1.p5.2.m2.1.1.2"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p5.2.m2.1.1.2.1.cmml" xref="S3.SS1.SSS1.p5.2.m2.1.1.2">subscript</csymbol><apply id="S3.SS1.SSS1.p5.2.m2.1.1.2.2.cmml" xref="S3.SS1.SSS1.p5.2.m2.1.1.2"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p5.2.m2.1.1.2.2.1.cmml" xref="S3.SS1.SSS1.p5.2.m2.1.1.2">superscript</csymbol><ci id="S3.SS1.SSS1.p5.2.m2.1.1.2.2.2.cmml" xref="S3.SS1.SSS1.p5.2.m2.1.1.2.2.2">𝐖</ci><ci id="S3.SS1.SSS1.p5.2.m2.1.1.2.2.3.cmml" xref="S3.SS1.SSS1.p5.2.m2.1.1.2.2.3">𝑞</ci></apply><ci id="S3.SS1.SSS1.p5.2.m2.1.1.2.3.cmml" xref="S3.SS1.SSS1.p5.2.m2.1.1.2.3">𝑖</ci></apply><apply id="S3.SS1.SSS1.p5.2.m2.1.1.3.cmml" xref="S3.SS1.SSS1.p5.2.m2.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p5.2.m2.1.1.3.1.cmml" xref="S3.SS1.SSS1.p5.2.m2.1.1.3">superscript</csymbol><ci id="S3.SS1.SSS1.p5.2.m2.1.1.3.2.cmml" xref="S3.SS1.SSS1.p5.2.m2.1.1.3.2">ℝ</ci><apply id="S3.SS1.SSS1.p5.2.m2.1.1.3.3.cmml" xref="S3.SS1.SSS1.p5.2.m2.1.1.3.3"><times id="S3.SS1.SSS1.p5.2.m2.1.1.3.3.1.cmml" xref="S3.SS1.SSS1.p5.2.m2.1.1.3.3.1"></times><ci id="S3.SS1.SSS1.p5.2.m2.1.1.3.3.2.cmml" xref="S3.SS1.SSS1.p5.2.m2.1.1.3.3.2">𝑑</ci><apply id="S3.SS1.SSS1.p5.2.m2.1.1.3.3.3.cmml" xref="S3.SS1.SSS1.p5.2.m2.1.1.3.3.3"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p5.2.m2.1.1.3.3.3.1.cmml" xref="S3.SS1.SSS1.p5.2.m2.1.1.3.3.3">subscript</csymbol><ci id="S3.SS1.SSS1.p5.2.m2.1.1.3.3.3.2.cmml" xref="S3.SS1.SSS1.p5.2.m2.1.1.3.3.3.2">𝑑</ci><ci id="S3.SS1.SSS1.p5.2.m2.1.1.3.3.3.3.cmml" xref="S3.SS1.SSS1.p5.2.m2.1.1.3.3.3.3">ℎ</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS1.p5.2.m2.1c">\mathbf{W}^{q}_{i}\in\mathbb{R}^{d\times d_{h}}</annotation></semantics></math>, so the fused matrix <math id="S3.SS1.SSS1.p5.3.m3.1" class="ltx_Math" alttext="(\mathbf{W}^{q}_{i}(\mathbf{B}^{k}_{i})^{T})" display="inline"><semantics id="S3.SS1.SSS1.p5.3.m3.1a"><mrow id="S3.SS1.SSS1.p5.3.m3.1.1.1" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.SSS1.p5.3.m3.1.1.1.2" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.cmml">(</mo><mrow id="S3.SS1.SSS1.p5.3.m3.1.1.1.1" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.cmml"><msubsup id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.3" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.3.cmml"><mi id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.3.2.2" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.3.2.2.cmml">𝐖</mi><mi id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.3.3" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.3.3.cmml">i</mi><mi id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.3.2.3" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.3.2.3.cmml">q</mi></msubsup><mo lspace="0em" rspace="0em" id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.2" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.2.cmml">​</mo><msup id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.cmml"><mrow id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.1.1" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.1.1.2" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.1.1.1.cmml">(</mo><msubsup id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.1.1.1" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.1.1.1.cmml"><mi id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.1.1.1.2.2" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.1.1.1.2.2.cmml">𝐁</mi><mi id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.1.1.1.3" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.1.1.1.3.cmml">i</mi><mi id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.1.1.1.2.3" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.1.1.1.2.3.cmml">k</mi></msubsup><mo stretchy="false" id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.1.1.3" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mi id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.3" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.3.cmml">T</mi></msup></mrow><mo stretchy="false" id="S3.SS1.SSS1.p5.3.m3.1.1.1.3" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS1.p5.3.m3.1b"><apply id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.cmml" xref="S3.SS1.SSS1.p5.3.m3.1.1.1"><times id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.2.cmml" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.2"></times><apply id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.3.cmml" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.3.1.cmml" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.3">subscript</csymbol><apply id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.3.2.cmml" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.3.2.1.cmml" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.3">superscript</csymbol><ci id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.3.2.2.cmml" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.3.2.2">𝐖</ci><ci id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.3.2.3.cmml" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.3.2.3">𝑞</ci></apply><ci id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.3.3.cmml" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.3.3">𝑖</ci></apply><apply id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.cmml" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.2.cmml" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1">superscript</csymbol><apply id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.1.1.1.cmml" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.1.1">subscript</csymbol><apply id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.1.1.1.2.cmml" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.1.1">superscript</csymbol><ci id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.1.1.1.2.2">𝐁</ci><ci id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.1.1.1.2.3">𝑘</ci></apply><ci id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.1.1.1.3.cmml" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.1.1.1.3">𝑖</ci></apply><ci id="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.3.cmml" xref="S3.SS1.SSS1.p5.3.m3.1.1.1.1.1.3">𝑇</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS1.p5.3.m3.1c">(\mathbf{W}^{q}_{i}(\mathbf{B}^{k}_{i})^{T})</annotation></semantics></math> has size <math id="S3.SS1.SSS1.p5.4.m4.1" class="ltx_Math" alttext="\mathbb{R}^{d\times r}" display="inline"><semantics id="S3.SS1.SSS1.p5.4.m4.1a"><msup id="S3.SS1.SSS1.p5.4.m4.1.1" xref="S3.SS1.SSS1.p5.4.m4.1.1.cmml"><mi id="S3.SS1.SSS1.p5.4.m4.1.1.2" xref="S3.SS1.SSS1.p5.4.m4.1.1.2.cmml">ℝ</mi><mrow id="S3.SS1.SSS1.p5.4.m4.1.1.3" xref="S3.SS1.SSS1.p5.4.m4.1.1.3.cmml"><mi id="S3.SS1.SSS1.p5.4.m4.1.1.3.2" xref="S3.SS1.SSS1.p5.4.m4.1.1.3.2.cmml">d</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS1.SSS1.p5.4.m4.1.1.3.1" xref="S3.SS1.SSS1.p5.4.m4.1.1.3.1.cmml">×</mo><mi id="S3.SS1.SSS1.p5.4.m4.1.1.3.3" xref="S3.SS1.SSS1.p5.4.m4.1.1.3.3.cmml">r</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS1.p5.4.m4.1b"><apply id="S3.SS1.SSS1.p5.4.m4.1.1.cmml" xref="S3.SS1.SSS1.p5.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS1.SSS1.p5.4.m4.1.1.1.cmml" xref="S3.SS1.SSS1.p5.4.m4.1.1">superscript</csymbol><ci id="S3.SS1.SSS1.p5.4.m4.1.1.2.cmml" xref="S3.SS1.SSS1.p5.4.m4.1.1.2">ℝ</ci><apply id="S3.SS1.SSS1.p5.4.m4.1.1.3.cmml" xref="S3.SS1.SSS1.p5.4.m4.1.1.3"><times id="S3.SS1.SSS1.p5.4.m4.1.1.3.1.cmml" xref="S3.SS1.SSS1.p5.4.m4.1.1.3.1"></times><ci id="S3.SS1.SSS1.p5.4.m4.1.1.3.2.cmml" xref="S3.SS1.SSS1.p5.4.m4.1.1.3.2">𝑑</ci><ci id="S3.SS1.SSS1.p5.4.m4.1.1.3.3.cmml" xref="S3.SS1.SSS1.p5.4.m4.1.1.3.3">𝑟</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS1.p5.4.m4.1c">\mathbb{R}^{d\times r}</annotation></semantics></math>. This fusion boosts computational efficiency by reducing the matrix dimension during attention score calculation.</p>
</div>
</section>
<section id="S3.SS1.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.1.2 </span>Mechanism with Positional Embedding</h4>

<div id="S3.SS1.SSS2.p1" class="ltx_para">
<p id="S3.SS1.SSS2.p1.1" class="ltx_p">Recent LLMs, such as the Llama family, apply positional embedding (<span id="S3.SS1.SSS2.p1.1.1" class="ltx_text ltx_font_italic">i.e.,</span> RoPE <cite class="ltx_cite ltx_citemacro_citep">(Su et al., <a href="#bib.bib37" title="" class="ltx_ref">2021</a>)</cite>) onto the query and key states before multiplication.
The non-linear operations of adding positional embedding prevent the fusion for the attention score, as described in equation <a href="#S3.E6" title="In 3.1.1 Integration with the Attention Mechanism and Offline Matrix Fusion ‣ 3.1 Compressing the KV-Cache via Low-Rank Projection ‣ 3 The Palu Framework ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>.
In this case, <span id="S3.SS1.SSS2.p1.1.2" class="ltx_text ltx_font_italic">Palu</span> reconstructs the keys from latent representations on-the-fly during decoding.
We improve reconstruction efficiency by designing an optimized kernel, which we present in Sec. <a href="#S3.SS5" title="3.5 Efficient Inference Implementation ‣ 3 The Palu Framework ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.5</span></a>.</p>
</div>
<div id="S3.SS1.SSS2.p2" class="ltx_para">
<p id="S3.SS1.SSS2.p2.1" class="ltx_p">Note that for some positional embedding methods, such as ALiBi <cite class="ltx_cite ltx_citemacro_cite">Press et al. (<a href="#bib.bib32" title="" class="ltx_ref">2022</a>)</cite>, or new attention mechanisms, such as MLA <cite class="ltx_cite ltx_citemacro_cite">DeepSeek-AI et al. (<a href="#bib.bib9" title="" class="ltx_ref">2024</a>)</cite>, positional embedding is not directly applied on the key states; therefore, the fusion of equation <a href="#S3.E6" title="In 3.1.1 Integration with the Attention Mechanism and Offline Matrix Fusion ‣ 3.1 Compressing the KV-Cache via Low-Rank Projection ‣ 3 The Palu Framework ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> remains valid.
We expect the end-to-end speedup to be larger on these models.
Fig. <a href="#S2.F2" title="Figure 2 ‣ 2.2 Singular Value Decomposition (SVD) ‣ 2 Background ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> presents a detailed workflow of <span id="S3.SS1.SSS2.p2.1.1" class="ltx_text ltx_font_italic">Palu</span> based on Llama’s attention with RoPE.</p>
</div>
</section>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Decomposition Granularity</h3>

<section id="S3.SS2.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.2.1 </span>Multi-Head Low-Rank Decomposition</h4>

<div id="S3.SS2.SSS1.p1" class="ltx_para">
<p id="S3.SS2.SSS1.p1.1" class="ltx_p">We name the per-head decomposition scheme in Sec.<a href="#S3.SS1.SSS1" title="3.1.1 Integration with the Attention Mechanism and Offline Matrix Fusion ‣ 3.1 Compressing the KV-Cache via Low-Rank Projection ‣ 3 The Palu Framework ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.1.1</span></a> as <span id="S3.SS2.SSS1.p1.1.1" class="ltx_text ltx_font_italic">multi-head low-rank decomposition (M-LRD)</span>.
We found M-LRD often causes a non-negligible accuracy degradation (discussed further in Sec.<a href="#S4.SS1" title="4.1 Main Results ‣ 4 Experiments ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.1</span></a>), possibly because SVD fails to capture the common information shared across heads.
Therefore, alternative approaches are needed for preserving model accuracy.</p>
</div>
</section>
<section id="S3.SS2.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.2.2 </span>Joint-Head Low-Rank Decomposition</h4>

<div id="S3.SS2.SSS2.p1" class="ltx_para">
<p id="S3.SS2.SSS2.p1.4" class="ltx_p">An alternative approach is to jointly decompose weight matrices for all heads.
By considering the combined weight matrix <math id="S3.SS2.SSS2.p1.1.m1.5" class="ltx_Math" alttext="\mathbf{W}_{\text{joint}}=[\mathbf{W}_{1},\mathbf{W}_{2},\dots,\mathbf{W}_{n}]\in\mathbb{R}^{d\times(d_{h}\cdot n_{h})}" display="inline"><semantics id="S3.SS2.SSS2.p1.1.m1.5a"><mrow id="S3.SS2.SSS2.p1.1.m1.5.5" xref="S3.SS2.SSS2.p1.1.m1.5.5.cmml"><msub id="S3.SS2.SSS2.p1.1.m1.5.5.5" xref="S3.SS2.SSS2.p1.1.m1.5.5.5.cmml"><mi id="S3.SS2.SSS2.p1.1.m1.5.5.5.2" xref="S3.SS2.SSS2.p1.1.m1.5.5.5.2.cmml">𝐖</mi><mtext id="S3.SS2.SSS2.p1.1.m1.5.5.5.3" xref="S3.SS2.SSS2.p1.1.m1.5.5.5.3a.cmml">joint</mtext></msub><mo id="S3.SS2.SSS2.p1.1.m1.5.5.6" xref="S3.SS2.SSS2.p1.1.m1.5.5.6.cmml">=</mo><mrow id="S3.SS2.SSS2.p1.1.m1.5.5.3.3" xref="S3.SS2.SSS2.p1.1.m1.5.5.3.4.cmml"><mo stretchy="false" id="S3.SS2.SSS2.p1.1.m1.5.5.3.3.4" xref="S3.SS2.SSS2.p1.1.m1.5.5.3.4.cmml">[</mo><msub id="S3.SS2.SSS2.p1.1.m1.3.3.1.1.1" xref="S3.SS2.SSS2.p1.1.m1.3.3.1.1.1.cmml"><mi id="S3.SS2.SSS2.p1.1.m1.3.3.1.1.1.2" xref="S3.SS2.SSS2.p1.1.m1.3.3.1.1.1.2.cmml">𝐖</mi><mn id="S3.SS2.SSS2.p1.1.m1.3.3.1.1.1.3" xref="S3.SS2.SSS2.p1.1.m1.3.3.1.1.1.3.cmml">1</mn></msub><mo id="S3.SS2.SSS2.p1.1.m1.5.5.3.3.5" xref="S3.SS2.SSS2.p1.1.m1.5.5.3.4.cmml">,</mo><msub id="S3.SS2.SSS2.p1.1.m1.4.4.2.2.2" xref="S3.SS2.SSS2.p1.1.m1.4.4.2.2.2.cmml"><mi id="S3.SS2.SSS2.p1.1.m1.4.4.2.2.2.2" xref="S3.SS2.SSS2.p1.1.m1.4.4.2.2.2.2.cmml">𝐖</mi><mn id="S3.SS2.SSS2.p1.1.m1.4.4.2.2.2.3" xref="S3.SS2.SSS2.p1.1.m1.4.4.2.2.2.3.cmml">2</mn></msub><mo id="S3.SS2.SSS2.p1.1.m1.5.5.3.3.6" xref="S3.SS2.SSS2.p1.1.m1.5.5.3.4.cmml">,</mo><mi mathvariant="normal" id="S3.SS2.SSS2.p1.1.m1.2.2" xref="S3.SS2.SSS2.p1.1.m1.2.2.cmml">…</mi><mo id="S3.SS2.SSS2.p1.1.m1.5.5.3.3.7" xref="S3.SS2.SSS2.p1.1.m1.5.5.3.4.cmml">,</mo><msub id="S3.SS2.SSS2.p1.1.m1.5.5.3.3.3" xref="S3.SS2.SSS2.p1.1.m1.5.5.3.3.3.cmml"><mi id="S3.SS2.SSS2.p1.1.m1.5.5.3.3.3.2" xref="S3.SS2.SSS2.p1.1.m1.5.5.3.3.3.2.cmml">𝐖</mi><mi id="S3.SS2.SSS2.p1.1.m1.5.5.3.3.3.3" xref="S3.SS2.SSS2.p1.1.m1.5.5.3.3.3.3.cmml">n</mi></msub><mo stretchy="false" id="S3.SS2.SSS2.p1.1.m1.5.5.3.3.8" xref="S3.SS2.SSS2.p1.1.m1.5.5.3.4.cmml">]</mo></mrow><mo id="S3.SS2.SSS2.p1.1.m1.5.5.7" xref="S3.SS2.SSS2.p1.1.m1.5.5.7.cmml">∈</mo><msup id="S3.SS2.SSS2.p1.1.m1.5.5.8" xref="S3.SS2.SSS2.p1.1.m1.5.5.8.cmml"><mi id="S3.SS2.SSS2.p1.1.m1.5.5.8.2" xref="S3.SS2.SSS2.p1.1.m1.5.5.8.2.cmml">ℝ</mi><mrow id="S3.SS2.SSS2.p1.1.m1.1.1.1" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.cmml"><mi id="S3.SS2.SSS2.p1.1.m1.1.1.1.3" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.3.cmml">d</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.SSS2.p1.1.m1.1.1.1.2" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.2.cmml">×</mo><mrow id="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.2" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.cmml"><msub id="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.2" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.2.cmml"><mi id="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.2.2" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.2.2.cmml">d</mi><mi id="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.2.3" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.2.3.cmml">h</mi></msub><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.1" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.1.cmml">⋅</mo><msub id="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.3" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.3.cmml"><mi id="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.3.2" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.3.2.cmml">n</mi><mi id="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.3.3" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.3.3.cmml">h</mi></msub></mrow><mo stretchy="false" id="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.3" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p1.1.m1.5b"><apply id="S3.SS2.SSS2.p1.1.m1.5.5.cmml" xref="S3.SS2.SSS2.p1.1.m1.5.5"><and id="S3.SS2.SSS2.p1.1.m1.5.5a.cmml" xref="S3.SS2.SSS2.p1.1.m1.5.5"></and><apply id="S3.SS2.SSS2.p1.1.m1.5.5b.cmml" xref="S3.SS2.SSS2.p1.1.m1.5.5"><eq id="S3.SS2.SSS2.p1.1.m1.5.5.6.cmml" xref="S3.SS2.SSS2.p1.1.m1.5.5.6"></eq><apply id="S3.SS2.SSS2.p1.1.m1.5.5.5.cmml" xref="S3.SS2.SSS2.p1.1.m1.5.5.5"><csymbol cd="ambiguous" id="S3.SS2.SSS2.p1.1.m1.5.5.5.1.cmml" xref="S3.SS2.SSS2.p1.1.m1.5.5.5">subscript</csymbol><ci id="S3.SS2.SSS2.p1.1.m1.5.5.5.2.cmml" xref="S3.SS2.SSS2.p1.1.m1.5.5.5.2">𝐖</ci><ci id="S3.SS2.SSS2.p1.1.m1.5.5.5.3a.cmml" xref="S3.SS2.SSS2.p1.1.m1.5.5.5.3"><mtext mathsize="70%" id="S3.SS2.SSS2.p1.1.m1.5.5.5.3.cmml" xref="S3.SS2.SSS2.p1.1.m1.5.5.5.3">joint</mtext></ci></apply><list id="S3.SS2.SSS2.p1.1.m1.5.5.3.4.cmml" xref="S3.SS2.SSS2.p1.1.m1.5.5.3.3"><apply id="S3.SS2.SSS2.p1.1.m1.3.3.1.1.1.cmml" xref="S3.SS2.SSS2.p1.1.m1.3.3.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS2.p1.1.m1.3.3.1.1.1.1.cmml" xref="S3.SS2.SSS2.p1.1.m1.3.3.1.1.1">subscript</csymbol><ci id="S3.SS2.SSS2.p1.1.m1.3.3.1.1.1.2.cmml" xref="S3.SS2.SSS2.p1.1.m1.3.3.1.1.1.2">𝐖</ci><cn type="integer" id="S3.SS2.SSS2.p1.1.m1.3.3.1.1.1.3.cmml" xref="S3.SS2.SSS2.p1.1.m1.3.3.1.1.1.3">1</cn></apply><apply id="S3.SS2.SSS2.p1.1.m1.4.4.2.2.2.cmml" xref="S3.SS2.SSS2.p1.1.m1.4.4.2.2.2"><csymbol cd="ambiguous" id="S3.SS2.SSS2.p1.1.m1.4.4.2.2.2.1.cmml" xref="S3.SS2.SSS2.p1.1.m1.4.4.2.2.2">subscript</csymbol><ci id="S3.SS2.SSS2.p1.1.m1.4.4.2.2.2.2.cmml" xref="S3.SS2.SSS2.p1.1.m1.4.4.2.2.2.2">𝐖</ci><cn type="integer" id="S3.SS2.SSS2.p1.1.m1.4.4.2.2.2.3.cmml" xref="S3.SS2.SSS2.p1.1.m1.4.4.2.2.2.3">2</cn></apply><ci id="S3.SS2.SSS2.p1.1.m1.2.2.cmml" xref="S3.SS2.SSS2.p1.1.m1.2.2">…</ci><apply id="S3.SS2.SSS2.p1.1.m1.5.5.3.3.3.cmml" xref="S3.SS2.SSS2.p1.1.m1.5.5.3.3.3"><csymbol cd="ambiguous" id="S3.SS2.SSS2.p1.1.m1.5.5.3.3.3.1.cmml" xref="S3.SS2.SSS2.p1.1.m1.5.5.3.3.3">subscript</csymbol><ci id="S3.SS2.SSS2.p1.1.m1.5.5.3.3.3.2.cmml" xref="S3.SS2.SSS2.p1.1.m1.5.5.3.3.3.2">𝐖</ci><ci id="S3.SS2.SSS2.p1.1.m1.5.5.3.3.3.3.cmml" xref="S3.SS2.SSS2.p1.1.m1.5.5.3.3.3.3">𝑛</ci></apply></list></apply><apply id="S3.SS2.SSS2.p1.1.m1.5.5c.cmml" xref="S3.SS2.SSS2.p1.1.m1.5.5"><in id="S3.SS2.SSS2.p1.1.m1.5.5.7.cmml" xref="S3.SS2.SSS2.p1.1.m1.5.5.7"></in><share href="#S3.SS2.SSS2.p1.1.m1.5.5.3.cmml" id="S3.SS2.SSS2.p1.1.m1.5.5d.cmml" xref="S3.SS2.SSS2.p1.1.m1.5.5"></share><apply id="S3.SS2.SSS2.p1.1.m1.5.5.8.cmml" xref="S3.SS2.SSS2.p1.1.m1.5.5.8"><csymbol cd="ambiguous" id="S3.SS2.SSS2.p1.1.m1.5.5.8.1.cmml" xref="S3.SS2.SSS2.p1.1.m1.5.5.8">superscript</csymbol><ci id="S3.SS2.SSS2.p1.1.m1.5.5.8.2.cmml" xref="S3.SS2.SSS2.p1.1.m1.5.5.8.2">ℝ</ci><apply id="S3.SS2.SSS2.p1.1.m1.1.1.1.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1.1"><times id="S3.SS2.SSS2.p1.1.m1.1.1.1.2.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.2"></times><ci id="S3.SS2.SSS2.p1.1.m1.1.1.1.3.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.3">𝑑</ci><apply id="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1"><ci id="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.1.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.1">⋅</ci><apply id="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.2.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.2.1.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.2.2.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.2.2">𝑑</ci><ci id="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.2.3.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.2.3">ℎ</ci></apply><apply id="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.3.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.3.1.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.3.2.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.3.2">𝑛</ci><ci id="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.3.3.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.1.1.1.3.3">ℎ</ci></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p1.1.m1.5c">\mathbf{W}_{\text{joint}}=[\mathbf{W}_{1},\mathbf{W}_{2},\dots,\mathbf{W}_{n}]\in\mathbb{R}^{d\times(d_{h}\cdot n_{h})}</annotation></semantics></math>,
we can perform a single low-rank decomposition
<math id="S3.SS2.SSS2.p1.2.m2.1" class="ltx_Math" alttext="\mathbf{W}_{\text{joint}}\approx\mathbf{A}_{\text{joint}}\mathbf{B}_{\text{joint}}" display="inline"><semantics id="S3.SS2.SSS2.p1.2.m2.1a"><mrow id="S3.SS2.SSS2.p1.2.m2.1.1" xref="S3.SS2.SSS2.p1.2.m2.1.1.cmml"><msub id="S3.SS2.SSS2.p1.2.m2.1.1.2" xref="S3.SS2.SSS2.p1.2.m2.1.1.2.cmml"><mi id="S3.SS2.SSS2.p1.2.m2.1.1.2.2" xref="S3.SS2.SSS2.p1.2.m2.1.1.2.2.cmml">𝐖</mi><mtext id="S3.SS2.SSS2.p1.2.m2.1.1.2.3" xref="S3.SS2.SSS2.p1.2.m2.1.1.2.3a.cmml">joint</mtext></msub><mo id="S3.SS2.SSS2.p1.2.m2.1.1.1" xref="S3.SS2.SSS2.p1.2.m2.1.1.1.cmml">≈</mo><mrow id="S3.SS2.SSS2.p1.2.m2.1.1.3" xref="S3.SS2.SSS2.p1.2.m2.1.1.3.cmml"><msub id="S3.SS2.SSS2.p1.2.m2.1.1.3.2" xref="S3.SS2.SSS2.p1.2.m2.1.1.3.2.cmml"><mi id="S3.SS2.SSS2.p1.2.m2.1.1.3.2.2" xref="S3.SS2.SSS2.p1.2.m2.1.1.3.2.2.cmml">𝐀</mi><mtext id="S3.SS2.SSS2.p1.2.m2.1.1.3.2.3" xref="S3.SS2.SSS2.p1.2.m2.1.1.3.2.3a.cmml">joint</mtext></msub><mo lspace="0em" rspace="0em" id="S3.SS2.SSS2.p1.2.m2.1.1.3.1" xref="S3.SS2.SSS2.p1.2.m2.1.1.3.1.cmml">​</mo><msub id="S3.SS2.SSS2.p1.2.m2.1.1.3.3" xref="S3.SS2.SSS2.p1.2.m2.1.1.3.3.cmml"><mi id="S3.SS2.SSS2.p1.2.m2.1.1.3.3.2" xref="S3.SS2.SSS2.p1.2.m2.1.1.3.3.2.cmml">𝐁</mi><mtext id="S3.SS2.SSS2.p1.2.m2.1.1.3.3.3" xref="S3.SS2.SSS2.p1.2.m2.1.1.3.3.3a.cmml">joint</mtext></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p1.2.m2.1b"><apply id="S3.SS2.SSS2.p1.2.m2.1.1.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1"><approx id="S3.SS2.SSS2.p1.2.m2.1.1.1.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1.1"></approx><apply id="S3.SS2.SSS2.p1.2.m2.1.1.2.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS2.p1.2.m2.1.1.2.1.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS2.p1.2.m2.1.1.2.2.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1.2.2">𝐖</ci><ci id="S3.SS2.SSS2.p1.2.m2.1.1.2.3a.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1.2.3"><mtext mathsize="70%" id="S3.SS2.SSS2.p1.2.m2.1.1.2.3.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1.2.3">joint</mtext></ci></apply><apply id="S3.SS2.SSS2.p1.2.m2.1.1.3.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1.3"><times id="S3.SS2.SSS2.p1.2.m2.1.1.3.1.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1.3.1"></times><apply id="S3.SS2.SSS2.p1.2.m2.1.1.3.2.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1.3.2"><csymbol cd="ambiguous" id="S3.SS2.SSS2.p1.2.m2.1.1.3.2.1.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1.3.2">subscript</csymbol><ci id="S3.SS2.SSS2.p1.2.m2.1.1.3.2.2.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1.3.2.2">𝐀</ci><ci id="S3.SS2.SSS2.p1.2.m2.1.1.3.2.3a.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1.3.2.3"><mtext mathsize="70%" id="S3.SS2.SSS2.p1.2.m2.1.1.3.2.3.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1.3.2.3">joint</mtext></ci></apply><apply id="S3.SS2.SSS2.p1.2.m2.1.1.3.3.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1.3.3"><csymbol cd="ambiguous" id="S3.SS2.SSS2.p1.2.m2.1.1.3.3.1.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1.3.3">subscript</csymbol><ci id="S3.SS2.SSS2.p1.2.m2.1.1.3.3.2.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1.3.3.2">𝐁</ci><ci id="S3.SS2.SSS2.p1.2.m2.1.1.3.3.3a.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1.3.3.3"><mtext mathsize="70%" id="S3.SS2.SSS2.p1.2.m2.1.1.3.3.3.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1.3.3.3">joint</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p1.2.m2.1c">\mathbf{W}_{\text{joint}}\approx\mathbf{A}_{\text{joint}}\mathbf{B}_{\text{joint}}</annotation></semantics></math>,
where <math id="S3.SS2.SSS2.p1.3.m3.1" class="ltx_Math" alttext="\mathbf{A}_{\text{joint}}\in\mathbb{R}^{d\times r_{\text{joint}}}" display="inline"><semantics id="S3.SS2.SSS2.p1.3.m3.1a"><mrow id="S3.SS2.SSS2.p1.3.m3.1.1" xref="S3.SS2.SSS2.p1.3.m3.1.1.cmml"><msub id="S3.SS2.SSS2.p1.3.m3.1.1.2" xref="S3.SS2.SSS2.p1.3.m3.1.1.2.cmml"><mi id="S3.SS2.SSS2.p1.3.m3.1.1.2.2" xref="S3.SS2.SSS2.p1.3.m3.1.1.2.2.cmml">𝐀</mi><mtext id="S3.SS2.SSS2.p1.3.m3.1.1.2.3" xref="S3.SS2.SSS2.p1.3.m3.1.1.2.3a.cmml">joint</mtext></msub><mo id="S3.SS2.SSS2.p1.3.m3.1.1.1" xref="S3.SS2.SSS2.p1.3.m3.1.1.1.cmml">∈</mo><msup id="S3.SS2.SSS2.p1.3.m3.1.1.3" xref="S3.SS2.SSS2.p1.3.m3.1.1.3.cmml"><mi id="S3.SS2.SSS2.p1.3.m3.1.1.3.2" xref="S3.SS2.SSS2.p1.3.m3.1.1.3.2.cmml">ℝ</mi><mrow id="S3.SS2.SSS2.p1.3.m3.1.1.3.3" xref="S3.SS2.SSS2.p1.3.m3.1.1.3.3.cmml"><mi id="S3.SS2.SSS2.p1.3.m3.1.1.3.3.2" xref="S3.SS2.SSS2.p1.3.m3.1.1.3.3.2.cmml">d</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.SSS2.p1.3.m3.1.1.3.3.1" xref="S3.SS2.SSS2.p1.3.m3.1.1.3.3.1.cmml">×</mo><msub id="S3.SS2.SSS2.p1.3.m3.1.1.3.3.3" xref="S3.SS2.SSS2.p1.3.m3.1.1.3.3.3.cmml"><mi id="S3.SS2.SSS2.p1.3.m3.1.1.3.3.3.2" xref="S3.SS2.SSS2.p1.3.m3.1.1.3.3.3.2.cmml">r</mi><mtext id="S3.SS2.SSS2.p1.3.m3.1.1.3.3.3.3" xref="S3.SS2.SSS2.p1.3.m3.1.1.3.3.3.3a.cmml">joint</mtext></msub></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p1.3.m3.1b"><apply id="S3.SS2.SSS2.p1.3.m3.1.1.cmml" xref="S3.SS2.SSS2.p1.3.m3.1.1"><in id="S3.SS2.SSS2.p1.3.m3.1.1.1.cmml" xref="S3.SS2.SSS2.p1.3.m3.1.1.1"></in><apply id="S3.SS2.SSS2.p1.3.m3.1.1.2.cmml" xref="S3.SS2.SSS2.p1.3.m3.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS2.p1.3.m3.1.1.2.1.cmml" xref="S3.SS2.SSS2.p1.3.m3.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS2.p1.3.m3.1.1.2.2.cmml" xref="S3.SS2.SSS2.p1.3.m3.1.1.2.2">𝐀</ci><ci id="S3.SS2.SSS2.p1.3.m3.1.1.2.3a.cmml" xref="S3.SS2.SSS2.p1.3.m3.1.1.2.3"><mtext mathsize="70%" id="S3.SS2.SSS2.p1.3.m3.1.1.2.3.cmml" xref="S3.SS2.SSS2.p1.3.m3.1.1.2.3">joint</mtext></ci></apply><apply id="S3.SS2.SSS2.p1.3.m3.1.1.3.cmml" xref="S3.SS2.SSS2.p1.3.m3.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.SSS2.p1.3.m3.1.1.3.1.cmml" xref="S3.SS2.SSS2.p1.3.m3.1.1.3">superscript</csymbol><ci id="S3.SS2.SSS2.p1.3.m3.1.1.3.2.cmml" xref="S3.SS2.SSS2.p1.3.m3.1.1.3.2">ℝ</ci><apply id="S3.SS2.SSS2.p1.3.m3.1.1.3.3.cmml" xref="S3.SS2.SSS2.p1.3.m3.1.1.3.3"><times id="S3.SS2.SSS2.p1.3.m3.1.1.3.3.1.cmml" xref="S3.SS2.SSS2.p1.3.m3.1.1.3.3.1"></times><ci id="S3.SS2.SSS2.p1.3.m3.1.1.3.3.2.cmml" xref="S3.SS2.SSS2.p1.3.m3.1.1.3.3.2">𝑑</ci><apply id="S3.SS2.SSS2.p1.3.m3.1.1.3.3.3.cmml" xref="S3.SS2.SSS2.p1.3.m3.1.1.3.3.3"><csymbol cd="ambiguous" id="S3.SS2.SSS2.p1.3.m3.1.1.3.3.3.1.cmml" xref="S3.SS2.SSS2.p1.3.m3.1.1.3.3.3">subscript</csymbol><ci id="S3.SS2.SSS2.p1.3.m3.1.1.3.3.3.2.cmml" xref="S3.SS2.SSS2.p1.3.m3.1.1.3.3.3.2">𝑟</ci><ci id="S3.SS2.SSS2.p1.3.m3.1.1.3.3.3.3a.cmml" xref="S3.SS2.SSS2.p1.3.m3.1.1.3.3.3.3"><mtext mathsize="50%" id="S3.SS2.SSS2.p1.3.m3.1.1.3.3.3.3.cmml" xref="S3.SS2.SSS2.p1.3.m3.1.1.3.3.3.3">joint</mtext></ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p1.3.m3.1c">\mathbf{A}_{\text{joint}}\in\mathbb{R}^{d\times r_{\text{joint}}}</annotation></semantics></math> and <math id="S3.SS2.SSS2.p1.4.m4.1" class="ltx_Math" alttext="\mathbf{B}_{\text{joint}}\in\mathbb{R}^{r_{\text{joint}}\times(d_{h}\cdot n_{h})}" display="inline"><semantics id="S3.SS2.SSS2.p1.4.m4.1a"><mrow id="S3.SS2.SSS2.p1.4.m4.1.2" xref="S3.SS2.SSS2.p1.4.m4.1.2.cmml"><msub id="S3.SS2.SSS2.p1.4.m4.1.2.2" xref="S3.SS2.SSS2.p1.4.m4.1.2.2.cmml"><mi id="S3.SS2.SSS2.p1.4.m4.1.2.2.2" xref="S3.SS2.SSS2.p1.4.m4.1.2.2.2.cmml">𝐁</mi><mtext id="S3.SS2.SSS2.p1.4.m4.1.2.2.3" xref="S3.SS2.SSS2.p1.4.m4.1.2.2.3a.cmml">joint</mtext></msub><mo id="S3.SS2.SSS2.p1.4.m4.1.2.1" xref="S3.SS2.SSS2.p1.4.m4.1.2.1.cmml">∈</mo><msup id="S3.SS2.SSS2.p1.4.m4.1.2.3" xref="S3.SS2.SSS2.p1.4.m4.1.2.3.cmml"><mi id="S3.SS2.SSS2.p1.4.m4.1.2.3.2" xref="S3.SS2.SSS2.p1.4.m4.1.2.3.2.cmml">ℝ</mi><mrow id="S3.SS2.SSS2.p1.4.m4.1.1.1" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.cmml"><msub id="S3.SS2.SSS2.p1.4.m4.1.1.1.3" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.3.cmml"><mi id="S3.SS2.SSS2.p1.4.m4.1.1.1.3.2" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.3.2.cmml">r</mi><mtext id="S3.SS2.SSS2.p1.4.m4.1.1.1.3.3" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.3.3a.cmml">joint</mtext></msub><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.SSS2.p1.4.m4.1.1.1.2" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.2.cmml">×</mo><mrow id="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.2" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.cmml"><msub id="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.2" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.2.cmml"><mi id="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.2.2" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.2.2.cmml">d</mi><mi id="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.2.3" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.2.3.cmml">h</mi></msub><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.1" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.1.cmml">⋅</mo><msub id="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.3" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.3.cmml"><mi id="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.3.2" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.3.2.cmml">n</mi><mi id="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.3.3" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.3.3.cmml">h</mi></msub></mrow><mo stretchy="false" id="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.3" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p1.4.m4.1b"><apply id="S3.SS2.SSS2.p1.4.m4.1.2.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.2"><in id="S3.SS2.SSS2.p1.4.m4.1.2.1.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.2.1"></in><apply id="S3.SS2.SSS2.p1.4.m4.1.2.2.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.2.2"><csymbol cd="ambiguous" id="S3.SS2.SSS2.p1.4.m4.1.2.2.1.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.2.2">subscript</csymbol><ci id="S3.SS2.SSS2.p1.4.m4.1.2.2.2.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.2.2.2">𝐁</ci><ci id="S3.SS2.SSS2.p1.4.m4.1.2.2.3a.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.2.2.3"><mtext mathsize="70%" id="S3.SS2.SSS2.p1.4.m4.1.2.2.3.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.2.2.3">joint</mtext></ci></apply><apply id="S3.SS2.SSS2.p1.4.m4.1.2.3.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.2.3"><csymbol cd="ambiguous" id="S3.SS2.SSS2.p1.4.m4.1.2.3.1.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.2.3">superscript</csymbol><ci id="S3.SS2.SSS2.p1.4.m4.1.2.3.2.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.2.3.2">ℝ</ci><apply id="S3.SS2.SSS2.p1.4.m4.1.1.1.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.1.1"><times id="S3.SS2.SSS2.p1.4.m4.1.1.1.2.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.2"></times><apply id="S3.SS2.SSS2.p1.4.m4.1.1.1.3.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.SSS2.p1.4.m4.1.1.1.3.1.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.3">subscript</csymbol><ci id="S3.SS2.SSS2.p1.4.m4.1.1.1.3.2.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.3.2">𝑟</ci><ci id="S3.SS2.SSS2.p1.4.m4.1.1.1.3.3a.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.3.3"><mtext mathsize="50%" id="S3.SS2.SSS2.p1.4.m4.1.1.1.3.3.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.3.3">joint</mtext></ci></apply><apply id="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1"><ci id="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.1.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.1">⋅</ci><apply id="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.2.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.2.1.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.2.2.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.2.2">𝑑</ci><ci id="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.2.3.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.2.3">ℎ</ci></apply><apply id="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.3.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.3.1.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.3.2.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.3.2">𝑛</ci><ci id="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.3.3.cmml" xref="S3.SS2.SSS2.p1.4.m4.1.1.1.1.1.1.3.3">ℎ</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p1.4.m4.1c">\mathbf{B}_{\text{joint}}\in\mathbb{R}^{r_{\text{joint}}\times(d_{h}\cdot n_{h})}</annotation></semantics></math>.
We call this scheme <span id="S3.SS2.SSS2.p1.4.1" class="ltx_text ltx_font_italic">joint-head low-rank decomposition (J-LRD)</span>.</p>
</div>
<div id="S3.SS2.SSS2.p2" class="ltx_para">
<p id="S3.SS2.SSS2.p2.1" class="ltx_p">J-LRD has the advantage of preserving the common principal components shared among different heads.
This occurs because SVD is particularly effective at capturing the dominant components when applied to a larger, combined matrix, resulting in a more accurate approximation.</p>
</div>
<div id="S3.SS2.SSS2.p3" class="ltx_para">
<p id="S3.SS2.SSS2.p3.1" class="ltx_p">For J-LRD, the joint latent representation shared among all heads can be computed with
<math id="S3.SS2.SSS2.p3.1.m1.1" class="ltx_Math" alttext="\mathbf{h}_{\text{joint}}=\mathbf{x}\mathbf{A}_{\text{joint}}" display="inline"><semantics id="S3.SS2.SSS2.p3.1.m1.1a"><mrow id="S3.SS2.SSS2.p3.1.m1.1.1" xref="S3.SS2.SSS2.p3.1.m1.1.1.cmml"><msub id="S3.SS2.SSS2.p3.1.m1.1.1.2" xref="S3.SS2.SSS2.p3.1.m1.1.1.2.cmml"><mi id="S3.SS2.SSS2.p3.1.m1.1.1.2.2" xref="S3.SS2.SSS2.p3.1.m1.1.1.2.2.cmml">𝐡</mi><mtext id="S3.SS2.SSS2.p3.1.m1.1.1.2.3" xref="S3.SS2.SSS2.p3.1.m1.1.1.2.3a.cmml">joint</mtext></msub><mo id="S3.SS2.SSS2.p3.1.m1.1.1.1" xref="S3.SS2.SSS2.p3.1.m1.1.1.1.cmml">=</mo><msub id="S3.SS2.SSS2.p3.1.m1.1.1.3" xref="S3.SS2.SSS2.p3.1.m1.1.1.3.cmml"><mi id="S3.SS2.SSS2.p3.1.m1.1.1.3.2" xref="S3.SS2.SSS2.p3.1.m1.1.1.3.2.cmml">𝐱𝐀</mi><mtext id="S3.SS2.SSS2.p3.1.m1.1.1.3.3" xref="S3.SS2.SSS2.p3.1.m1.1.1.3.3a.cmml">joint</mtext></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p3.1.m1.1b"><apply id="S3.SS2.SSS2.p3.1.m1.1.1.cmml" xref="S3.SS2.SSS2.p3.1.m1.1.1"><eq id="S3.SS2.SSS2.p3.1.m1.1.1.1.cmml" xref="S3.SS2.SSS2.p3.1.m1.1.1.1"></eq><apply id="S3.SS2.SSS2.p3.1.m1.1.1.2.cmml" xref="S3.SS2.SSS2.p3.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS2.p3.1.m1.1.1.2.1.cmml" xref="S3.SS2.SSS2.p3.1.m1.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS2.p3.1.m1.1.1.2.2.cmml" xref="S3.SS2.SSS2.p3.1.m1.1.1.2.2">𝐡</ci><ci id="S3.SS2.SSS2.p3.1.m1.1.1.2.3a.cmml" xref="S3.SS2.SSS2.p3.1.m1.1.1.2.3"><mtext mathsize="70%" id="S3.SS2.SSS2.p3.1.m1.1.1.2.3.cmml" xref="S3.SS2.SSS2.p3.1.m1.1.1.2.3">joint</mtext></ci></apply><apply id="S3.SS2.SSS2.p3.1.m1.1.1.3.cmml" xref="S3.SS2.SSS2.p3.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.SSS2.p3.1.m1.1.1.3.1.cmml" xref="S3.SS2.SSS2.p3.1.m1.1.1.3">subscript</csymbol><ci id="S3.SS2.SSS2.p3.1.m1.1.1.3.2.cmml" xref="S3.SS2.SSS2.p3.1.m1.1.1.3.2">𝐱𝐀</ci><ci id="S3.SS2.SSS2.p3.1.m1.1.1.3.3a.cmml" xref="S3.SS2.SSS2.p3.1.m1.1.1.3.3"><mtext mathsize="70%" id="S3.SS2.SSS2.p3.1.m1.1.1.3.3.cmml" xref="S3.SS2.SSS2.p3.1.m1.1.1.3.3">joint</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p3.1.m1.1c">\mathbf{h}_{\text{joint}}=\mathbf{x}\mathbf{A}_{\text{joint}}</annotation></semantics></math>.
During decoding, the original states for each head can be reconstructed via</p>
<table id="S3.Ex2" class="ltx_equationgroup ltx_eqn_table">

<tbody id="S3.Ex2X"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.Ex2X.2.1.1.m1.3" class="ltx_Math" alttext="\displaystyle\bigl{[}\mathbf{y}_{1},\dots,\mathbf{y}_{n}\bigr{]}" display="inline"><semantics id="S3.Ex2X.2.1.1.m1.3a"><mrow id="S3.Ex2X.2.1.1.m1.3.3.2" xref="S3.Ex2X.2.1.1.m1.3.3.3.cmml"><mo maxsize="120%" minsize="120%" id="S3.Ex2X.2.1.1.m1.3.3.2.3" xref="S3.Ex2X.2.1.1.m1.3.3.3.cmml">[</mo><msub id="S3.Ex2X.2.1.1.m1.2.2.1.1" xref="S3.Ex2X.2.1.1.m1.2.2.1.1.cmml"><mi id="S3.Ex2X.2.1.1.m1.2.2.1.1.2" xref="S3.Ex2X.2.1.1.m1.2.2.1.1.2.cmml">𝐲</mi><mn id="S3.Ex2X.2.1.1.m1.2.2.1.1.3" xref="S3.Ex2X.2.1.1.m1.2.2.1.1.3.cmml">1</mn></msub><mo id="S3.Ex2X.2.1.1.m1.3.3.2.4" xref="S3.Ex2X.2.1.1.m1.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S3.Ex2X.2.1.1.m1.1.1" xref="S3.Ex2X.2.1.1.m1.1.1.cmml">…</mi><mo id="S3.Ex2X.2.1.1.m1.3.3.2.5" xref="S3.Ex2X.2.1.1.m1.3.3.3.cmml">,</mo><msub id="S3.Ex2X.2.1.1.m1.3.3.2.2" xref="S3.Ex2X.2.1.1.m1.3.3.2.2.cmml"><mi id="S3.Ex2X.2.1.1.m1.3.3.2.2.2" xref="S3.Ex2X.2.1.1.m1.3.3.2.2.2.cmml">𝐲</mi><mi id="S3.Ex2X.2.1.1.m1.3.3.2.2.3" xref="S3.Ex2X.2.1.1.m1.3.3.2.2.3.cmml">n</mi></msub><mo maxsize="120%" minsize="120%" id="S3.Ex2X.2.1.1.m1.3.3.2.6" xref="S3.Ex2X.2.1.1.m1.3.3.3.cmml">]</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.Ex2X.2.1.1.m1.3b"><list id="S3.Ex2X.2.1.1.m1.3.3.3.cmml" xref="S3.Ex2X.2.1.1.m1.3.3.2"><apply id="S3.Ex2X.2.1.1.m1.2.2.1.1.cmml" xref="S3.Ex2X.2.1.1.m1.2.2.1.1"><csymbol cd="ambiguous" id="S3.Ex2X.2.1.1.m1.2.2.1.1.1.cmml" xref="S3.Ex2X.2.1.1.m1.2.2.1.1">subscript</csymbol><ci id="S3.Ex2X.2.1.1.m1.2.2.1.1.2.cmml" xref="S3.Ex2X.2.1.1.m1.2.2.1.1.2">𝐲</ci><cn type="integer" id="S3.Ex2X.2.1.1.m1.2.2.1.1.3.cmml" xref="S3.Ex2X.2.1.1.m1.2.2.1.1.3">1</cn></apply><ci id="S3.Ex2X.2.1.1.m1.1.1.cmml" xref="S3.Ex2X.2.1.1.m1.1.1">…</ci><apply id="S3.Ex2X.2.1.1.m1.3.3.2.2.cmml" xref="S3.Ex2X.2.1.1.m1.3.3.2.2"><csymbol cd="ambiguous" id="S3.Ex2X.2.1.1.m1.3.3.2.2.1.cmml" xref="S3.Ex2X.2.1.1.m1.3.3.2.2">subscript</csymbol><ci id="S3.Ex2X.2.1.1.m1.3.3.2.2.2.cmml" xref="S3.Ex2X.2.1.1.m1.3.3.2.2.2">𝐲</ci><ci id="S3.Ex2X.2.1.1.m1.3.3.2.2.3.cmml" xref="S3.Ex2X.2.1.1.m1.3.3.2.2.3">𝑛</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S3.Ex2X.2.1.1.m1.3c">\displaystyle\bigl{[}\mathbf{y}_{1},\dots,\mathbf{y}_{n}\bigr{]}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.Ex2X.3.2.2.m1.1" class="ltx_Math" alttext="\displaystyle=\mathbf{h}_{\text{joint}}\mathbf{B}_{\text{joint}}." display="inline"><semantics id="S3.Ex2X.3.2.2.m1.1a"><mrow id="S3.Ex2X.3.2.2.m1.1.1.1" xref="S3.Ex2X.3.2.2.m1.1.1.1.1.cmml"><mrow id="S3.Ex2X.3.2.2.m1.1.1.1.1" xref="S3.Ex2X.3.2.2.m1.1.1.1.1.cmml"><mi id="S3.Ex2X.3.2.2.m1.1.1.1.1.2" xref="S3.Ex2X.3.2.2.m1.1.1.1.1.2.cmml"></mi><mo id="S3.Ex2X.3.2.2.m1.1.1.1.1.1" xref="S3.Ex2X.3.2.2.m1.1.1.1.1.1.cmml">=</mo><mrow id="S3.Ex2X.3.2.2.m1.1.1.1.1.3" xref="S3.Ex2X.3.2.2.m1.1.1.1.1.3.cmml"><msub id="S3.Ex2X.3.2.2.m1.1.1.1.1.3.2" xref="S3.Ex2X.3.2.2.m1.1.1.1.1.3.2.cmml"><mi id="S3.Ex2X.3.2.2.m1.1.1.1.1.3.2.2" xref="S3.Ex2X.3.2.2.m1.1.1.1.1.3.2.2.cmml">𝐡</mi><mtext id="S3.Ex2X.3.2.2.m1.1.1.1.1.3.2.3" xref="S3.Ex2X.3.2.2.m1.1.1.1.1.3.2.3a.cmml">joint</mtext></msub><mo lspace="0em" rspace="0em" id="S3.Ex2X.3.2.2.m1.1.1.1.1.3.1" xref="S3.Ex2X.3.2.2.m1.1.1.1.1.3.1.cmml">​</mo><msub id="S3.Ex2X.3.2.2.m1.1.1.1.1.3.3" xref="S3.Ex2X.3.2.2.m1.1.1.1.1.3.3.cmml"><mi id="S3.Ex2X.3.2.2.m1.1.1.1.1.3.3.2" xref="S3.Ex2X.3.2.2.m1.1.1.1.1.3.3.2.cmml">𝐁</mi><mtext id="S3.Ex2X.3.2.2.m1.1.1.1.1.3.3.3" xref="S3.Ex2X.3.2.2.m1.1.1.1.1.3.3.3a.cmml">joint</mtext></msub></mrow></mrow><mo lspace="0em" id="S3.Ex2X.3.2.2.m1.1.1.1.2" xref="S3.Ex2X.3.2.2.m1.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.Ex2X.3.2.2.m1.1b"><apply id="S3.Ex2X.3.2.2.m1.1.1.1.1.cmml" xref="S3.Ex2X.3.2.2.m1.1.1.1"><eq id="S3.Ex2X.3.2.2.m1.1.1.1.1.1.cmml" xref="S3.Ex2X.3.2.2.m1.1.1.1.1.1"></eq><csymbol cd="latexml" id="S3.Ex2X.3.2.2.m1.1.1.1.1.2.cmml" xref="S3.Ex2X.3.2.2.m1.1.1.1.1.2">absent</csymbol><apply id="S3.Ex2X.3.2.2.m1.1.1.1.1.3.cmml" xref="S3.Ex2X.3.2.2.m1.1.1.1.1.3"><times id="S3.Ex2X.3.2.2.m1.1.1.1.1.3.1.cmml" xref="S3.Ex2X.3.2.2.m1.1.1.1.1.3.1"></times><apply id="S3.Ex2X.3.2.2.m1.1.1.1.1.3.2.cmml" xref="S3.Ex2X.3.2.2.m1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S3.Ex2X.3.2.2.m1.1.1.1.1.3.2.1.cmml" xref="S3.Ex2X.3.2.2.m1.1.1.1.1.3.2">subscript</csymbol><ci id="S3.Ex2X.3.2.2.m1.1.1.1.1.3.2.2.cmml" xref="S3.Ex2X.3.2.2.m1.1.1.1.1.3.2.2">𝐡</ci><ci id="S3.Ex2X.3.2.2.m1.1.1.1.1.3.2.3a.cmml" xref="S3.Ex2X.3.2.2.m1.1.1.1.1.3.2.3"><mtext mathsize="70%" id="S3.Ex2X.3.2.2.m1.1.1.1.1.3.2.3.cmml" xref="S3.Ex2X.3.2.2.m1.1.1.1.1.3.2.3">joint</mtext></ci></apply><apply id="S3.Ex2X.3.2.2.m1.1.1.1.1.3.3.cmml" xref="S3.Ex2X.3.2.2.m1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S3.Ex2X.3.2.2.m1.1.1.1.1.3.3.1.cmml" xref="S3.Ex2X.3.2.2.m1.1.1.1.1.3.3">subscript</csymbol><ci id="S3.Ex2X.3.2.2.m1.1.1.1.1.3.3.2.cmml" xref="S3.Ex2X.3.2.2.m1.1.1.1.1.3.3.2">𝐁</ci><ci id="S3.Ex2X.3.2.2.m1.1.1.1.1.3.3.3a.cmml" xref="S3.Ex2X.3.2.2.m1.1.1.1.1.3.3.3"><mtext mathsize="70%" id="S3.Ex2X.3.2.2.m1.1.1.1.1.3.3.3.cmml" xref="S3.Ex2X.3.2.2.m1.1.1.1.1.3.3.3">joint</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.Ex2X.3.2.2.m1.1c">\displaystyle=\mathbf{h}_{\text{joint}}\mathbf{B}_{\text{joint}}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<section id="S3.SS2.SSS2.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">High Inference Overhead.</h5>

<div id="S3.SS2.SSS2.Px1.p1" class="ltx_para">
<p id="S3.SS2.SSS2.Px1.p1.6" class="ltx_p">Despite better preserving model information, J-LRD introduces <span id="S3.SS2.SSS2.Px1.p1.6.1" class="ltx_text ltx_font_italic">significant computational and memory overhead</span> during decoding. Specifically, the total number of floating point operations (FLOPs) to reconstruct the key or values state of one head now becomes <math id="S3.SS2.SSS2.Px1.p1.1.m1.1" class="ltx_Math" alttext="r_{\text{joint}}\cdot d_{h}\cdot n" display="inline"><semantics id="S3.SS2.SSS2.Px1.p1.1.m1.1a"><mrow id="S3.SS2.SSS2.Px1.p1.1.m1.1.1" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.cmml"><msub id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.2" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.2.cmml"><mi id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.2.2" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.2.2.cmml">r</mi><mtext id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.2.3" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.2.3a.cmml">joint</mtext></msub><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.1" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.1.cmml">⋅</mo><msub id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.cmml"><mi id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.2" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.2.cmml">d</mi><mi id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.3" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.3.cmml">h</mi></msub><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.1a" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.1.cmml">⋅</mo><mi id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.4" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.4.cmml">n</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.Px1.p1.1.m1.1b"><apply id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.cmml" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1"><ci id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.1.cmml" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.1">⋅</ci><apply id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.2.cmml" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.2.1.cmml" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.2.2.cmml" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.2.2">𝑟</ci><ci id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.2.3a.cmml" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.2.3"><mtext mathsize="70%" id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.2.3.cmml" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.2.3">joint</mtext></ci></apply><apply id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.cmml" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.1.cmml" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3">subscript</csymbol><ci id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.2.cmml" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.2">𝑑</ci><ci id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.3.cmml" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.3">ℎ</ci></apply><ci id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.4.cmml" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.4">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.Px1.p1.1.m1.1c">r_{\text{joint}}\cdot d_{h}\cdot n</annotation></semantics></math>. Assuming the same size as the total low-rank latent representations (<span id="S3.SS2.SSS2.Px1.p1.6.2" class="ltx_text ltx_font_italic">i.e.,</span> <math id="S3.SS2.SSS2.Px1.p1.2.m2.1" class="ltx_Math" alttext="r_{\text{joint}}=\sum_{i=1}^{n}{r_{i}}" display="inline"><semantics id="S3.SS2.SSS2.Px1.p1.2.m2.1a"><mrow id="S3.SS2.SSS2.Px1.p1.2.m2.1.1" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.cmml"><msub id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.2" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.2.cmml"><mi id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.2.2" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.2.2.cmml">r</mi><mtext id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.2.3" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.2.3a.cmml">joint</mtext></msub><mo rspace="0.111em" id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.1" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.1.cmml">=</mo><mrow id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.cmml"><msubsup id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.cmml"><mo id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.2.2" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.2.2.cmml">∑</mo><mrow id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.2.3" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.2.3.cmml"><mi id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.2.3.2" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.2.3.2.cmml">i</mi><mo id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.2.3.1" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.2.3.1.cmml">=</mo><mn id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.2.3.3" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.2.3.3.cmml">1</mn></mrow><mi id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.3" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.3.cmml">n</mi></msubsup><msub id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.2" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.2.cmml"><mi id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.2.2" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.2.2.cmml">r</mi><mi id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.2.3" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.2.3.cmml">i</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.Px1.p1.2.m2.1b"><apply id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.cmml" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1"><eq id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.1.cmml" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.1"></eq><apply id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.2.cmml" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.2.1.cmml" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.2.2.cmml" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.2.2">𝑟</ci><ci id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.2.3a.cmml" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.2.3"><mtext mathsize="70%" id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.2.3.cmml" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.2.3">joint</mtext></ci></apply><apply id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.cmml" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3"><apply id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.cmml" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1"><csymbol cd="ambiguous" id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.1.cmml" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1">superscript</csymbol><apply id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.2.cmml" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1"><csymbol cd="ambiguous" id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.2.1.cmml" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1">subscript</csymbol><sum id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.2.2.cmml" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.2.2"></sum><apply id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.2.3.cmml" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.2.3"><eq id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.2.3.1.cmml" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.2.3.1"></eq><ci id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.2.3.2.cmml" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.2.3.2">𝑖</ci><cn type="integer" id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.2.3.3.cmml" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.2.3.3">1</cn></apply></apply><ci id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.3.cmml" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.1.3">𝑛</ci></apply><apply id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.2.cmml" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.2"><csymbol cd="ambiguous" id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.2.1.cmml" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.2">subscript</csymbol><ci id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.2.2.cmml" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.2.2">𝑟</ci><ci id="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.2.3.cmml" xref="S3.SS2.SSS2.Px1.p1.2.m2.1.1.3.2.3">𝑖</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.Px1.p1.2.m2.1c">r_{\text{joint}}=\sum_{i=1}^{n}{r_{i}}</annotation></semantics></math>),
the total reconstruction cost is <math id="S3.SS2.SSS2.Px1.p1.3.m3.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S3.SS2.SSS2.Px1.p1.3.m3.1a"><mi id="S3.SS2.SSS2.Px1.p1.3.m3.1.1" xref="S3.SS2.SSS2.Px1.p1.3.m3.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.Px1.p1.3.m3.1b"><ci id="S3.SS2.SSS2.Px1.p1.3.m3.1.1.cmml" xref="S3.SS2.SSS2.Px1.p1.3.m3.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.Px1.p1.3.m3.1c">n</annotation></semantics></math> times higher than M-LRD, whose total FLOPs is <math id="S3.SS2.SSS2.Px1.p1.4.m4.1" class="ltx_Math" alttext="r_{i}\cdot d_{h}\cdot n" display="inline"><semantics id="S3.SS2.SSS2.Px1.p1.4.m4.1a"><mrow id="S3.SS2.SSS2.Px1.p1.4.m4.1.1" xref="S3.SS2.SSS2.Px1.p1.4.m4.1.1.cmml"><msub id="S3.SS2.SSS2.Px1.p1.4.m4.1.1.2" xref="S3.SS2.SSS2.Px1.p1.4.m4.1.1.2.cmml"><mi id="S3.SS2.SSS2.Px1.p1.4.m4.1.1.2.2" xref="S3.SS2.SSS2.Px1.p1.4.m4.1.1.2.2.cmml">r</mi><mi id="S3.SS2.SSS2.Px1.p1.4.m4.1.1.2.3" xref="S3.SS2.SSS2.Px1.p1.4.m4.1.1.2.3.cmml">i</mi></msub><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.SSS2.Px1.p1.4.m4.1.1.1" xref="S3.SS2.SSS2.Px1.p1.4.m4.1.1.1.cmml">⋅</mo><msub id="S3.SS2.SSS2.Px1.p1.4.m4.1.1.3" xref="S3.SS2.SSS2.Px1.p1.4.m4.1.1.3.cmml"><mi id="S3.SS2.SSS2.Px1.p1.4.m4.1.1.3.2" xref="S3.SS2.SSS2.Px1.p1.4.m4.1.1.3.2.cmml">d</mi><mi id="S3.SS2.SSS2.Px1.p1.4.m4.1.1.3.3" xref="S3.SS2.SSS2.Px1.p1.4.m4.1.1.3.3.cmml">h</mi></msub><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.SSS2.Px1.p1.4.m4.1.1.1a" xref="S3.SS2.SSS2.Px1.p1.4.m4.1.1.1.cmml">⋅</mo><mi id="S3.SS2.SSS2.Px1.p1.4.m4.1.1.4" xref="S3.SS2.SSS2.Px1.p1.4.m4.1.1.4.cmml">n</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.Px1.p1.4.m4.1b"><apply id="S3.SS2.SSS2.Px1.p1.4.m4.1.1.cmml" xref="S3.SS2.SSS2.Px1.p1.4.m4.1.1"><ci id="S3.SS2.SSS2.Px1.p1.4.m4.1.1.1.cmml" xref="S3.SS2.SSS2.Px1.p1.4.m4.1.1.1">⋅</ci><apply id="S3.SS2.SSS2.Px1.p1.4.m4.1.1.2.cmml" xref="S3.SS2.SSS2.Px1.p1.4.m4.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS2.Px1.p1.4.m4.1.1.2.1.cmml" xref="S3.SS2.SSS2.Px1.p1.4.m4.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS2.Px1.p1.4.m4.1.1.2.2.cmml" xref="S3.SS2.SSS2.Px1.p1.4.m4.1.1.2.2">𝑟</ci><ci id="S3.SS2.SSS2.Px1.p1.4.m4.1.1.2.3.cmml" xref="S3.SS2.SSS2.Px1.p1.4.m4.1.1.2.3">𝑖</ci></apply><apply id="S3.SS2.SSS2.Px1.p1.4.m4.1.1.3.cmml" xref="S3.SS2.SSS2.Px1.p1.4.m4.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.SSS2.Px1.p1.4.m4.1.1.3.1.cmml" xref="S3.SS2.SSS2.Px1.p1.4.m4.1.1.3">subscript</csymbol><ci id="S3.SS2.SSS2.Px1.p1.4.m4.1.1.3.2.cmml" xref="S3.SS2.SSS2.Px1.p1.4.m4.1.1.3.2">𝑑</ci><ci id="S3.SS2.SSS2.Px1.p1.4.m4.1.1.3.3.cmml" xref="S3.SS2.SSS2.Px1.p1.4.m4.1.1.3.3">ℎ</ci></apply><ci id="S3.SS2.SSS2.Px1.p1.4.m4.1.1.4.cmml" xref="S3.SS2.SSS2.Px1.p1.4.m4.1.1.4">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.Px1.p1.4.m4.1c">r_{i}\cdot d_{h}\cdot n</annotation></semantics></math>.
When considering the matrix fusion in Sec.<a href="#S3.SS1.SSS1" title="3.1.1 Integration with the Attention Mechanism and Offline Matrix Fusion ‣ 3.1 Compressing the KV-Cache via Low-Rank Projection ‣ 3 The Palu Framework ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.1.1</span></a>,
the fused matrix size of J-LRD has a size of <math id="S3.SS2.SSS2.Px1.p1.5.m5.1" class="ltx_Math" alttext="r_{joint}\cdot d\cdot n" display="inline"><semantics id="S3.SS2.SSS2.Px1.p1.5.m5.1a"><mrow id="S3.SS2.SSS2.Px1.p1.5.m5.1.1" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.cmml"><msub id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.cmml"><mi id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.2" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.2.cmml">r</mi><mrow id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.cmml"><mi id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.2" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.2.cmml">j</mi><mo lspace="0em" rspace="0em" id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.1" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.1.cmml">​</mo><mi id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.3" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.1a" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.1.cmml">​</mo><mi id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.4" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.4.cmml">i</mi><mo lspace="0em" rspace="0em" id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.1b" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.1.cmml">​</mo><mi id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.5" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.5.cmml">n</mi><mo lspace="0em" rspace="0em" id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.1c" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.1.cmml">​</mo><mi id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.6" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.6.cmml">t</mi></mrow></msub><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.1" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.1.cmml">⋅</mo><mi id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.3" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.3.cmml">d</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.1a" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.1.cmml">⋅</mo><mi id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.4" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.4.cmml">n</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.Px1.p1.5.m5.1b"><apply id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.cmml" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1"><ci id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.1.cmml" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.1">⋅</ci><apply id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.cmml" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.1.cmml" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.2.cmml" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.2">𝑟</ci><apply id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.cmml" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3"><times id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.1.cmml" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.1"></times><ci id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.2.cmml" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.2">𝑗</ci><ci id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.3.cmml" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.3">𝑜</ci><ci id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.4.cmml" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.4">𝑖</ci><ci id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.5.cmml" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.5">𝑛</ci><ci id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.6.cmml" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.2.3.6">𝑡</ci></apply></apply><ci id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.3.cmml" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.3">𝑑</ci><ci id="S3.SS2.SSS2.Px1.p1.5.m5.1.1.4.cmml" xref="S3.SS2.SSS2.Px1.p1.5.m5.1.1.4">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.Px1.p1.5.m5.1c">r_{joint}\cdot d\cdot n</annotation></semantics></math>, which is also <math id="S3.SS2.SSS2.Px1.p1.6.m6.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S3.SS2.SSS2.Px1.p1.6.m6.1a"><mi id="S3.SS2.SSS2.Px1.p1.6.m6.1.1" xref="S3.SS2.SSS2.Px1.p1.6.m6.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.Px1.p1.6.m6.1b"><ci id="S3.SS2.SSS2.Px1.p1.6.m6.1.1.cmml" xref="S3.SS2.SSS2.Px1.p1.6.m6.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.Px1.p1.6.m6.1c">n</annotation></semantics></math> times larger than M-LRD, leading to a substantial higher memory consumption.</p>
</div>
<figure id="S3.F3" class="ltx_figure"><img src="/html/2407.21118/assets/x3.png" id="S3.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="415" height="191" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>Performing decomposition at different granularities. Jointly decomposing multiple heads can achieve higher accuracy. Assuming the same total size of the latent representations (<span id="S3.F3.4.1" class="ltx_text ltx_font_italic">i.e.</span>, <math id="S3.F3.2.m1.1" class="ltx_Math" alttext="4\cdot r_{i}=2\cdot r_{g}=r_{\text{joint}}" display="inline"><semantics id="S3.F3.2.m1.1b"><mrow id="S3.F3.2.m1.1.1" xref="S3.F3.2.m1.1.1.cmml"><mrow id="S3.F3.2.m1.1.1.2" xref="S3.F3.2.m1.1.1.2.cmml"><mn id="S3.F3.2.m1.1.1.2.2" xref="S3.F3.2.m1.1.1.2.2.cmml">4</mn><mo lspace="0.222em" rspace="0.222em" id="S3.F3.2.m1.1.1.2.1" xref="S3.F3.2.m1.1.1.2.1.cmml">⋅</mo><msub id="S3.F3.2.m1.1.1.2.3" xref="S3.F3.2.m1.1.1.2.3.cmml"><mi id="S3.F3.2.m1.1.1.2.3.2" xref="S3.F3.2.m1.1.1.2.3.2.cmml">r</mi><mi id="S3.F3.2.m1.1.1.2.3.3" xref="S3.F3.2.m1.1.1.2.3.3.cmml">i</mi></msub></mrow><mo id="S3.F3.2.m1.1.1.3" xref="S3.F3.2.m1.1.1.3.cmml">=</mo><mrow id="S3.F3.2.m1.1.1.4" xref="S3.F3.2.m1.1.1.4.cmml"><mn id="S3.F3.2.m1.1.1.4.2" xref="S3.F3.2.m1.1.1.4.2.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="S3.F3.2.m1.1.1.4.1" xref="S3.F3.2.m1.1.1.4.1.cmml">⋅</mo><msub id="S3.F3.2.m1.1.1.4.3" xref="S3.F3.2.m1.1.1.4.3.cmml"><mi id="S3.F3.2.m1.1.1.4.3.2" xref="S3.F3.2.m1.1.1.4.3.2.cmml">r</mi><mi id="S3.F3.2.m1.1.1.4.3.3" xref="S3.F3.2.m1.1.1.4.3.3.cmml">g</mi></msub></mrow><mo id="S3.F3.2.m1.1.1.5" xref="S3.F3.2.m1.1.1.5.cmml">=</mo><msub id="S3.F3.2.m1.1.1.6" xref="S3.F3.2.m1.1.1.6.cmml"><mi id="S3.F3.2.m1.1.1.6.2" xref="S3.F3.2.m1.1.1.6.2.cmml">r</mi><mtext id="S3.F3.2.m1.1.1.6.3" xref="S3.F3.2.m1.1.1.6.3a.cmml">joint</mtext></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.F3.2.m1.1c"><apply id="S3.F3.2.m1.1.1.cmml" xref="S3.F3.2.m1.1.1"><and id="S3.F3.2.m1.1.1a.cmml" xref="S3.F3.2.m1.1.1"></and><apply id="S3.F3.2.m1.1.1b.cmml" xref="S3.F3.2.m1.1.1"><eq id="S3.F3.2.m1.1.1.3.cmml" xref="S3.F3.2.m1.1.1.3"></eq><apply id="S3.F3.2.m1.1.1.2.cmml" xref="S3.F3.2.m1.1.1.2"><ci id="S3.F3.2.m1.1.1.2.1.cmml" xref="S3.F3.2.m1.1.1.2.1">⋅</ci><cn type="integer" id="S3.F3.2.m1.1.1.2.2.cmml" xref="S3.F3.2.m1.1.1.2.2">4</cn><apply id="S3.F3.2.m1.1.1.2.3.cmml" xref="S3.F3.2.m1.1.1.2.3"><csymbol cd="ambiguous" id="S3.F3.2.m1.1.1.2.3.1.cmml" xref="S3.F3.2.m1.1.1.2.3">subscript</csymbol><ci id="S3.F3.2.m1.1.1.2.3.2.cmml" xref="S3.F3.2.m1.1.1.2.3.2">𝑟</ci><ci id="S3.F3.2.m1.1.1.2.3.3.cmml" xref="S3.F3.2.m1.1.1.2.3.3">𝑖</ci></apply></apply><apply id="S3.F3.2.m1.1.1.4.cmml" xref="S3.F3.2.m1.1.1.4"><ci id="S3.F3.2.m1.1.1.4.1.cmml" xref="S3.F3.2.m1.1.1.4.1">⋅</ci><cn type="integer" id="S3.F3.2.m1.1.1.4.2.cmml" xref="S3.F3.2.m1.1.1.4.2">2</cn><apply id="S3.F3.2.m1.1.1.4.3.cmml" xref="S3.F3.2.m1.1.1.4.3"><csymbol cd="ambiguous" id="S3.F3.2.m1.1.1.4.3.1.cmml" xref="S3.F3.2.m1.1.1.4.3">subscript</csymbol><ci id="S3.F3.2.m1.1.1.4.3.2.cmml" xref="S3.F3.2.m1.1.1.4.3.2">𝑟</ci><ci id="S3.F3.2.m1.1.1.4.3.3.cmml" xref="S3.F3.2.m1.1.1.4.3.3">𝑔</ci></apply></apply></apply><apply id="S3.F3.2.m1.1.1c.cmml" xref="S3.F3.2.m1.1.1"><eq id="S3.F3.2.m1.1.1.5.cmml" xref="S3.F3.2.m1.1.1.5"></eq><share href="#S3.F3.2.m1.1.1.4.cmml" id="S3.F3.2.m1.1.1d.cmml" xref="S3.F3.2.m1.1.1"></share><apply id="S3.F3.2.m1.1.1.6.cmml" xref="S3.F3.2.m1.1.1.6"><csymbol cd="ambiguous" id="S3.F3.2.m1.1.1.6.1.cmml" xref="S3.F3.2.m1.1.1.6">subscript</csymbol><ci id="S3.F3.2.m1.1.1.6.2.cmml" xref="S3.F3.2.m1.1.1.6.2">𝑟</ci><ci id="S3.F3.2.m1.1.1.6.3a.cmml" xref="S3.F3.2.m1.1.1.6.3"><mtext mathsize="70%" id="S3.F3.2.m1.1.1.6.3.cmml" xref="S3.F3.2.m1.1.1.6.3">joint</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.2.m1.1d">4\cdot r_{i}=2\cdot r_{g}=r_{\text{joint}}</annotation></semantics></math>), the FLOPs for reconstruction overhead in joint-head decomposition schemes are 4 times larger than those in multi-head ones.</figcaption>
</figure>
</section>
</section>
<section id="S3.SS2.SSS3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.2.3 </span>Group-Head Low-Rank Decomposition</h4>

<div id="S3.SS2.SSS3.p1" class="ltx_para">
<p id="S3.SS2.SSS3.p1.1" class="ltx_p">To balance the trade-off between accuracy and reconstruction cost, we propose <span id="S3.SS2.SSS3.p1.1.1" class="ltx_text ltx_font_italic">group-head low-rank decomposition</span>(G-LRD).
G-LRD decomposes the matrices for a group of heads together.
With combined weight matrices, it captures shared information within each group while limiting computational overhead and preserving accuracy.</p>
</div>
<div id="S3.SS2.SSS3.p2" class="ltx_para">
<p id="S3.SS2.SSS3.p2.7" class="ltx_p">To illustrate the G-LRD process, consider the weight matrices for a group of <math id="S3.SS2.SSS3.p2.1.m1.1" class="ltx_Math" alttext="s" display="inline"><semantics id="S3.SS2.SSS3.p2.1.m1.1a"><mi id="S3.SS2.SSS3.p2.1.m1.1.1" xref="S3.SS2.SSS3.p2.1.m1.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS3.p2.1.m1.1b"><ci id="S3.SS2.SSS3.p2.1.m1.1.1.cmml" xref="S3.SS2.SSS3.p2.1.m1.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS3.p2.1.m1.1c">s</annotation></semantics></math> heads,
<math id="S3.SS2.SSS3.p2.2.m2.5" class="ltx_Math" alttext="\mathbf{W}_{g_{j}}=\bigl{[}\mathbf{W}_{j,1}\dots\mathbf{W}_{j,s}\bigr{]}" display="inline"><semantics id="S3.SS2.SSS3.p2.2.m2.5a"><mrow id="S3.SS2.SSS3.p2.2.m2.5.5" xref="S3.SS2.SSS3.p2.2.m2.5.5.cmml"><msub id="S3.SS2.SSS3.p2.2.m2.5.5.3" xref="S3.SS2.SSS3.p2.2.m2.5.5.3.cmml"><mi id="S3.SS2.SSS3.p2.2.m2.5.5.3.2" xref="S3.SS2.SSS3.p2.2.m2.5.5.3.2.cmml">𝐖</mi><msub id="S3.SS2.SSS3.p2.2.m2.5.5.3.3" xref="S3.SS2.SSS3.p2.2.m2.5.5.3.3.cmml"><mi id="S3.SS2.SSS3.p2.2.m2.5.5.3.3.2" xref="S3.SS2.SSS3.p2.2.m2.5.5.3.3.2.cmml">g</mi><mi id="S3.SS2.SSS3.p2.2.m2.5.5.3.3.3" xref="S3.SS2.SSS3.p2.2.m2.5.5.3.3.3.cmml">j</mi></msub></msub><mo id="S3.SS2.SSS3.p2.2.m2.5.5.2" xref="S3.SS2.SSS3.p2.2.m2.5.5.2.cmml">=</mo><mrow id="S3.SS2.SSS3.p2.2.m2.5.5.1.1" xref="S3.SS2.SSS3.p2.2.m2.5.5.1.2.cmml"><mo maxsize="120%" minsize="120%" id="S3.SS2.SSS3.p2.2.m2.5.5.1.1.2" xref="S3.SS2.SSS3.p2.2.m2.5.5.1.2.1.cmml">[</mo><mrow id="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1" xref="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.cmml"><msub id="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.2" xref="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.2.cmml"><mi id="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.2.2" xref="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.2.2.cmml">𝐖</mi><mrow id="S3.SS2.SSS3.p2.2.m2.2.2.2.4" xref="S3.SS2.SSS3.p2.2.m2.2.2.2.3.cmml"><mi id="S3.SS2.SSS3.p2.2.m2.1.1.1.1" xref="S3.SS2.SSS3.p2.2.m2.1.1.1.1.cmml">j</mi><mo id="S3.SS2.SSS3.p2.2.m2.2.2.2.4.1" xref="S3.SS2.SSS3.p2.2.m2.2.2.2.3.cmml">,</mo><mn id="S3.SS2.SSS3.p2.2.m2.2.2.2.2" xref="S3.SS2.SSS3.p2.2.m2.2.2.2.2.cmml">1</mn></mrow></msub><mo lspace="0em" rspace="0em" id="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.1" xref="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.1.cmml">​</mo><mi mathvariant="normal" id="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.3" xref="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.3.cmml">…</mi><mo lspace="0em" rspace="0em" id="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.1a" xref="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.1.cmml">​</mo><msub id="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.4" xref="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.4.cmml"><mi id="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.4.2" xref="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.4.2.cmml">𝐖</mi><mrow id="S3.SS2.SSS3.p2.2.m2.4.4.2.4" xref="S3.SS2.SSS3.p2.2.m2.4.4.2.3.cmml"><mi id="S3.SS2.SSS3.p2.2.m2.3.3.1.1" xref="S3.SS2.SSS3.p2.2.m2.3.3.1.1.cmml">j</mi><mo id="S3.SS2.SSS3.p2.2.m2.4.4.2.4.1" xref="S3.SS2.SSS3.p2.2.m2.4.4.2.3.cmml">,</mo><mi id="S3.SS2.SSS3.p2.2.m2.4.4.2.2" xref="S3.SS2.SSS3.p2.2.m2.4.4.2.2.cmml">s</mi></mrow></msub></mrow><mo maxsize="120%" minsize="120%" id="S3.SS2.SSS3.p2.2.m2.5.5.1.1.3" xref="S3.SS2.SSS3.p2.2.m2.5.5.1.2.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS3.p2.2.m2.5b"><apply id="S3.SS2.SSS3.p2.2.m2.5.5.cmml" xref="S3.SS2.SSS3.p2.2.m2.5.5"><eq id="S3.SS2.SSS3.p2.2.m2.5.5.2.cmml" xref="S3.SS2.SSS3.p2.2.m2.5.5.2"></eq><apply id="S3.SS2.SSS3.p2.2.m2.5.5.3.cmml" xref="S3.SS2.SSS3.p2.2.m2.5.5.3"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p2.2.m2.5.5.3.1.cmml" xref="S3.SS2.SSS3.p2.2.m2.5.5.3">subscript</csymbol><ci id="S3.SS2.SSS3.p2.2.m2.5.5.3.2.cmml" xref="S3.SS2.SSS3.p2.2.m2.5.5.3.2">𝐖</ci><apply id="S3.SS2.SSS3.p2.2.m2.5.5.3.3.cmml" xref="S3.SS2.SSS3.p2.2.m2.5.5.3.3"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p2.2.m2.5.5.3.3.1.cmml" xref="S3.SS2.SSS3.p2.2.m2.5.5.3.3">subscript</csymbol><ci id="S3.SS2.SSS3.p2.2.m2.5.5.3.3.2.cmml" xref="S3.SS2.SSS3.p2.2.m2.5.5.3.3.2">𝑔</ci><ci id="S3.SS2.SSS3.p2.2.m2.5.5.3.3.3.cmml" xref="S3.SS2.SSS3.p2.2.m2.5.5.3.3.3">𝑗</ci></apply></apply><apply id="S3.SS2.SSS3.p2.2.m2.5.5.1.2.cmml" xref="S3.SS2.SSS3.p2.2.m2.5.5.1.1"><csymbol cd="latexml" id="S3.SS2.SSS3.p2.2.m2.5.5.1.2.1.cmml" xref="S3.SS2.SSS3.p2.2.m2.5.5.1.1.2">delimited-[]</csymbol><apply id="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.cmml" xref="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1"><times id="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.1.cmml" xref="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.1"></times><apply id="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.2.cmml" xref="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.2.1.cmml" xref="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.2.2.cmml" xref="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.2.2">𝐖</ci><list id="S3.SS2.SSS3.p2.2.m2.2.2.2.3.cmml" xref="S3.SS2.SSS3.p2.2.m2.2.2.2.4"><ci id="S3.SS2.SSS3.p2.2.m2.1.1.1.1.cmml" xref="S3.SS2.SSS3.p2.2.m2.1.1.1.1">𝑗</ci><cn type="integer" id="S3.SS2.SSS3.p2.2.m2.2.2.2.2.cmml" xref="S3.SS2.SSS3.p2.2.m2.2.2.2.2">1</cn></list></apply><ci id="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.3.cmml" xref="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.3">…</ci><apply id="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.4.cmml" xref="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.4"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.4.1.cmml" xref="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.4">subscript</csymbol><ci id="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.4.2.cmml" xref="S3.SS2.SSS3.p2.2.m2.5.5.1.1.1.4.2">𝐖</ci><list id="S3.SS2.SSS3.p2.2.m2.4.4.2.3.cmml" xref="S3.SS2.SSS3.p2.2.m2.4.4.2.4"><ci id="S3.SS2.SSS3.p2.2.m2.3.3.1.1.cmml" xref="S3.SS2.SSS3.p2.2.m2.3.3.1.1">𝑗</ci><ci id="S3.SS2.SSS3.p2.2.m2.4.4.2.2.cmml" xref="S3.SS2.SSS3.p2.2.m2.4.4.2.2">𝑠</ci></list></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS3.p2.2.m2.5c">\mathbf{W}_{g_{j}}=\bigl{[}\mathbf{W}_{j,1}\dots\mathbf{W}_{j,s}\bigr{]}</annotation></semantics></math>, where <math id="S3.SS2.SSS3.p2.3.m3.1" class="ltx_Math" alttext="\mathbf{W}_{g_{j}}\in\mathbb{R}^{d\times(d_{h}\cdot s)}" display="inline"><semantics id="S3.SS2.SSS3.p2.3.m3.1a"><mrow id="S3.SS2.SSS3.p2.3.m3.1.2" xref="S3.SS2.SSS3.p2.3.m3.1.2.cmml"><msub id="S3.SS2.SSS3.p2.3.m3.1.2.2" xref="S3.SS2.SSS3.p2.3.m3.1.2.2.cmml"><mi id="S3.SS2.SSS3.p2.3.m3.1.2.2.2" xref="S3.SS2.SSS3.p2.3.m3.1.2.2.2.cmml">𝐖</mi><msub id="S3.SS2.SSS3.p2.3.m3.1.2.2.3" xref="S3.SS2.SSS3.p2.3.m3.1.2.2.3.cmml"><mi id="S3.SS2.SSS3.p2.3.m3.1.2.2.3.2" xref="S3.SS2.SSS3.p2.3.m3.1.2.2.3.2.cmml">g</mi><mi id="S3.SS2.SSS3.p2.3.m3.1.2.2.3.3" xref="S3.SS2.SSS3.p2.3.m3.1.2.2.3.3.cmml">j</mi></msub></msub><mo id="S3.SS2.SSS3.p2.3.m3.1.2.1" xref="S3.SS2.SSS3.p2.3.m3.1.2.1.cmml">∈</mo><msup id="S3.SS2.SSS3.p2.3.m3.1.2.3" xref="S3.SS2.SSS3.p2.3.m3.1.2.3.cmml"><mi id="S3.SS2.SSS3.p2.3.m3.1.2.3.2" xref="S3.SS2.SSS3.p2.3.m3.1.2.3.2.cmml">ℝ</mi><mrow id="S3.SS2.SSS3.p2.3.m3.1.1.1" xref="S3.SS2.SSS3.p2.3.m3.1.1.1.cmml"><mi id="S3.SS2.SSS3.p2.3.m3.1.1.1.3" xref="S3.SS2.SSS3.p2.3.m3.1.1.1.3.cmml">d</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.SSS3.p2.3.m3.1.1.1.2" xref="S3.SS2.SSS3.p2.3.m3.1.1.1.2.cmml">×</mo><mrow id="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1" xref="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.2" xref="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.1" xref="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.1.cmml"><msub id="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.1.2" xref="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.1.2.cmml"><mi id="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.1.2.2" xref="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.1.2.2.cmml">d</mi><mi id="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.1.2.3" xref="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.1.2.3.cmml">h</mi></msub><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.1.1" xref="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.1.1.cmml">⋅</mo><mi id="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.1.3" xref="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.1.3.cmml">s</mi></mrow><mo stretchy="false" id="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.3" xref="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS3.p2.3.m3.1b"><apply id="S3.SS2.SSS3.p2.3.m3.1.2.cmml" xref="S3.SS2.SSS3.p2.3.m3.1.2"><in id="S3.SS2.SSS3.p2.3.m3.1.2.1.cmml" xref="S3.SS2.SSS3.p2.3.m3.1.2.1"></in><apply id="S3.SS2.SSS3.p2.3.m3.1.2.2.cmml" xref="S3.SS2.SSS3.p2.3.m3.1.2.2"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p2.3.m3.1.2.2.1.cmml" xref="S3.SS2.SSS3.p2.3.m3.1.2.2">subscript</csymbol><ci id="S3.SS2.SSS3.p2.3.m3.1.2.2.2.cmml" xref="S3.SS2.SSS3.p2.3.m3.1.2.2.2">𝐖</ci><apply id="S3.SS2.SSS3.p2.3.m3.1.2.2.3.cmml" xref="S3.SS2.SSS3.p2.3.m3.1.2.2.3"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p2.3.m3.1.2.2.3.1.cmml" xref="S3.SS2.SSS3.p2.3.m3.1.2.2.3">subscript</csymbol><ci id="S3.SS2.SSS3.p2.3.m3.1.2.2.3.2.cmml" xref="S3.SS2.SSS3.p2.3.m3.1.2.2.3.2">𝑔</ci><ci id="S3.SS2.SSS3.p2.3.m3.1.2.2.3.3.cmml" xref="S3.SS2.SSS3.p2.3.m3.1.2.2.3.3">𝑗</ci></apply></apply><apply id="S3.SS2.SSS3.p2.3.m3.1.2.3.cmml" xref="S3.SS2.SSS3.p2.3.m3.1.2.3"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p2.3.m3.1.2.3.1.cmml" xref="S3.SS2.SSS3.p2.3.m3.1.2.3">superscript</csymbol><ci id="S3.SS2.SSS3.p2.3.m3.1.2.3.2.cmml" xref="S3.SS2.SSS3.p2.3.m3.1.2.3.2">ℝ</ci><apply id="S3.SS2.SSS3.p2.3.m3.1.1.1.cmml" xref="S3.SS2.SSS3.p2.3.m3.1.1.1"><times id="S3.SS2.SSS3.p2.3.m3.1.1.1.2.cmml" xref="S3.SS2.SSS3.p2.3.m3.1.1.1.2"></times><ci id="S3.SS2.SSS3.p2.3.m3.1.1.1.3.cmml" xref="S3.SS2.SSS3.p2.3.m3.1.1.1.3">𝑑</ci><apply id="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.1.cmml" xref="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1"><ci id="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.1.1.cmml" xref="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.1.1">⋅</ci><apply id="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.1.2.cmml" xref="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.1.2.1.cmml" xref="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.1.2.2.cmml" xref="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.1.2.2">𝑑</ci><ci id="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.1.2.3.cmml" xref="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.1.2.3">ℎ</ci></apply><ci id="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.1.3.cmml" xref="S3.SS2.SSS3.p2.3.m3.1.1.1.1.1.1.3">𝑠</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS3.p2.3.m3.1c">\mathbf{W}_{g_{j}}\in\mathbb{R}^{d\times(d_{h}\cdot s)}</annotation></semantics></math>.
We low-rank decompose <span id="S3.SS2.SSS3.p2.4.1" class="ltx_text"><math id="S3.SS2.SSS3.p2.4.1.m1.1" class="ltx_Math" alttext="\mathbf{W}_{g_{j}}\approx\mathbf{A}_{g_{j}}\mathbf{B}_{g_{j}}" display="inline"><semantics id="S3.SS2.SSS3.p2.4.1.m1.1a"><mrow id="S3.SS2.SSS3.p2.4.1.m1.1.1" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.cmml"><msub id="S3.SS2.SSS3.p2.4.1.m1.1.1.2" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.2.cmml"><mi id="S3.SS2.SSS3.p2.4.1.m1.1.1.2.2" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.2.2.cmml">𝐖</mi><msub id="S3.SS2.SSS3.p2.4.1.m1.1.1.2.3" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.2.3.cmml"><mi id="S3.SS2.SSS3.p2.4.1.m1.1.1.2.3.2" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.2.3.2.cmml">g</mi><mi id="S3.SS2.SSS3.p2.4.1.m1.1.1.2.3.3" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.2.3.3.cmml">j</mi></msub></msub><mo id="S3.SS2.SSS3.p2.4.1.m1.1.1.1" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.1.cmml">≈</mo><mrow id="S3.SS2.SSS3.p2.4.1.m1.1.1.3" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.3.cmml"><msub id="S3.SS2.SSS3.p2.4.1.m1.1.1.3.2" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.3.2.cmml"><mi id="S3.SS2.SSS3.p2.4.1.m1.1.1.3.2.2" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.3.2.2.cmml">𝐀</mi><msub id="S3.SS2.SSS3.p2.4.1.m1.1.1.3.2.3" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.3.2.3.cmml"><mi id="S3.SS2.SSS3.p2.4.1.m1.1.1.3.2.3.2" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.3.2.3.2.cmml">g</mi><mi id="S3.SS2.SSS3.p2.4.1.m1.1.1.3.2.3.3" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.3.2.3.3.cmml">j</mi></msub></msub><mo lspace="0em" rspace="0em" id="S3.SS2.SSS3.p2.4.1.m1.1.1.3.1" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.3.1.cmml">​</mo><msub id="S3.SS2.SSS3.p2.4.1.m1.1.1.3.3" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.3.3.cmml"><mi id="S3.SS2.SSS3.p2.4.1.m1.1.1.3.3.2" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.3.3.2.cmml">𝐁</mi><msub id="S3.SS2.SSS3.p2.4.1.m1.1.1.3.3.3" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.3.3.3.cmml"><mi id="S3.SS2.SSS3.p2.4.1.m1.1.1.3.3.3.2" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.3.3.3.2.cmml">g</mi><mi id="S3.SS2.SSS3.p2.4.1.m1.1.1.3.3.3.3" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.3.3.3.3.cmml">j</mi></msub></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS3.p2.4.1.m1.1b"><apply id="S3.SS2.SSS3.p2.4.1.m1.1.1.cmml" xref="S3.SS2.SSS3.p2.4.1.m1.1.1"><approx id="S3.SS2.SSS3.p2.4.1.m1.1.1.1.cmml" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.1"></approx><apply id="S3.SS2.SSS3.p2.4.1.m1.1.1.2.cmml" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p2.4.1.m1.1.1.2.1.cmml" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS3.p2.4.1.m1.1.1.2.2.cmml" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.2.2">𝐖</ci><apply id="S3.SS2.SSS3.p2.4.1.m1.1.1.2.3.cmml" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.2.3"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p2.4.1.m1.1.1.2.3.1.cmml" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.2.3">subscript</csymbol><ci id="S3.SS2.SSS3.p2.4.1.m1.1.1.2.3.2.cmml" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.2.3.2">𝑔</ci><ci id="S3.SS2.SSS3.p2.4.1.m1.1.1.2.3.3.cmml" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.2.3.3">𝑗</ci></apply></apply><apply id="S3.SS2.SSS3.p2.4.1.m1.1.1.3.cmml" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.3"><times id="S3.SS2.SSS3.p2.4.1.m1.1.1.3.1.cmml" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.3.1"></times><apply id="S3.SS2.SSS3.p2.4.1.m1.1.1.3.2.cmml" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.3.2"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p2.4.1.m1.1.1.3.2.1.cmml" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.3.2">subscript</csymbol><ci id="S3.SS2.SSS3.p2.4.1.m1.1.1.3.2.2.cmml" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.3.2.2">𝐀</ci><apply id="S3.SS2.SSS3.p2.4.1.m1.1.1.3.2.3.cmml" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.3.2.3"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p2.4.1.m1.1.1.3.2.3.1.cmml" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.3.2.3">subscript</csymbol><ci id="S3.SS2.SSS3.p2.4.1.m1.1.1.3.2.3.2.cmml" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.3.2.3.2">𝑔</ci><ci id="S3.SS2.SSS3.p2.4.1.m1.1.1.3.2.3.3.cmml" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.3.2.3.3">𝑗</ci></apply></apply><apply id="S3.SS2.SSS3.p2.4.1.m1.1.1.3.3.cmml" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.3.3"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p2.4.1.m1.1.1.3.3.1.cmml" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.3.3">subscript</csymbol><ci id="S3.SS2.SSS3.p2.4.1.m1.1.1.3.3.2.cmml" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.3.3.2">𝐁</ci><apply id="S3.SS2.SSS3.p2.4.1.m1.1.1.3.3.3.cmml" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.3.3.3"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p2.4.1.m1.1.1.3.3.3.1.cmml" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.3.3.3">subscript</csymbol><ci id="S3.SS2.SSS3.p2.4.1.m1.1.1.3.3.3.2.cmml" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.3.3.3.2">𝑔</ci><ci id="S3.SS2.SSS3.p2.4.1.m1.1.1.3.3.3.3.cmml" xref="S3.SS2.SSS3.p2.4.1.m1.1.1.3.3.3.3">𝑗</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS3.p2.4.1.m1.1c">\mathbf{W}_{g_{j}}\approx\mathbf{A}_{g_{j}}\mathbf{B}_{g_{j}}</annotation></semantics></math></span>,
where <math id="S3.SS2.SSS3.p2.5.m4.1" class="ltx_Math" alttext="\mathbf{A}_{g_{j}}\in\mathbb{R}^{d\times r_{g}}" display="inline"><semantics id="S3.SS2.SSS3.p2.5.m4.1a"><mrow id="S3.SS2.SSS3.p2.5.m4.1.1" xref="S3.SS2.SSS3.p2.5.m4.1.1.cmml"><msub id="S3.SS2.SSS3.p2.5.m4.1.1.2" xref="S3.SS2.SSS3.p2.5.m4.1.1.2.cmml"><mi id="S3.SS2.SSS3.p2.5.m4.1.1.2.2" xref="S3.SS2.SSS3.p2.5.m4.1.1.2.2.cmml">𝐀</mi><msub id="S3.SS2.SSS3.p2.5.m4.1.1.2.3" xref="S3.SS2.SSS3.p2.5.m4.1.1.2.3.cmml"><mi id="S3.SS2.SSS3.p2.5.m4.1.1.2.3.2" xref="S3.SS2.SSS3.p2.5.m4.1.1.2.3.2.cmml">g</mi><mi id="S3.SS2.SSS3.p2.5.m4.1.1.2.3.3" xref="S3.SS2.SSS3.p2.5.m4.1.1.2.3.3.cmml">j</mi></msub></msub><mo id="S3.SS2.SSS3.p2.5.m4.1.1.1" xref="S3.SS2.SSS3.p2.5.m4.1.1.1.cmml">∈</mo><msup id="S3.SS2.SSS3.p2.5.m4.1.1.3" xref="S3.SS2.SSS3.p2.5.m4.1.1.3.cmml"><mi id="S3.SS2.SSS3.p2.5.m4.1.1.3.2" xref="S3.SS2.SSS3.p2.5.m4.1.1.3.2.cmml">ℝ</mi><mrow id="S3.SS2.SSS3.p2.5.m4.1.1.3.3" xref="S3.SS2.SSS3.p2.5.m4.1.1.3.3.cmml"><mi id="S3.SS2.SSS3.p2.5.m4.1.1.3.3.2" xref="S3.SS2.SSS3.p2.5.m4.1.1.3.3.2.cmml">d</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.SSS3.p2.5.m4.1.1.3.3.1" xref="S3.SS2.SSS3.p2.5.m4.1.1.3.3.1.cmml">×</mo><msub id="S3.SS2.SSS3.p2.5.m4.1.1.3.3.3" xref="S3.SS2.SSS3.p2.5.m4.1.1.3.3.3.cmml"><mi id="S3.SS2.SSS3.p2.5.m4.1.1.3.3.3.2" xref="S3.SS2.SSS3.p2.5.m4.1.1.3.3.3.2.cmml">r</mi><mi id="S3.SS2.SSS3.p2.5.m4.1.1.3.3.3.3" xref="S3.SS2.SSS3.p2.5.m4.1.1.3.3.3.3.cmml">g</mi></msub></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS3.p2.5.m4.1b"><apply id="S3.SS2.SSS3.p2.5.m4.1.1.cmml" xref="S3.SS2.SSS3.p2.5.m4.1.1"><in id="S3.SS2.SSS3.p2.5.m4.1.1.1.cmml" xref="S3.SS2.SSS3.p2.5.m4.1.1.1"></in><apply id="S3.SS2.SSS3.p2.5.m4.1.1.2.cmml" xref="S3.SS2.SSS3.p2.5.m4.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p2.5.m4.1.1.2.1.cmml" xref="S3.SS2.SSS3.p2.5.m4.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS3.p2.5.m4.1.1.2.2.cmml" xref="S3.SS2.SSS3.p2.5.m4.1.1.2.2">𝐀</ci><apply id="S3.SS2.SSS3.p2.5.m4.1.1.2.3.cmml" xref="S3.SS2.SSS3.p2.5.m4.1.1.2.3"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p2.5.m4.1.1.2.3.1.cmml" xref="S3.SS2.SSS3.p2.5.m4.1.1.2.3">subscript</csymbol><ci id="S3.SS2.SSS3.p2.5.m4.1.1.2.3.2.cmml" xref="S3.SS2.SSS3.p2.5.m4.1.1.2.3.2">𝑔</ci><ci id="S3.SS2.SSS3.p2.5.m4.1.1.2.3.3.cmml" xref="S3.SS2.SSS3.p2.5.m4.1.1.2.3.3">𝑗</ci></apply></apply><apply id="S3.SS2.SSS3.p2.5.m4.1.1.3.cmml" xref="S3.SS2.SSS3.p2.5.m4.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p2.5.m4.1.1.3.1.cmml" xref="S3.SS2.SSS3.p2.5.m4.1.1.3">superscript</csymbol><ci id="S3.SS2.SSS3.p2.5.m4.1.1.3.2.cmml" xref="S3.SS2.SSS3.p2.5.m4.1.1.3.2">ℝ</ci><apply id="S3.SS2.SSS3.p2.5.m4.1.1.3.3.cmml" xref="S3.SS2.SSS3.p2.5.m4.1.1.3.3"><times id="S3.SS2.SSS3.p2.5.m4.1.1.3.3.1.cmml" xref="S3.SS2.SSS3.p2.5.m4.1.1.3.3.1"></times><ci id="S3.SS2.SSS3.p2.5.m4.1.1.3.3.2.cmml" xref="S3.SS2.SSS3.p2.5.m4.1.1.3.3.2">𝑑</ci><apply id="S3.SS2.SSS3.p2.5.m4.1.1.3.3.3.cmml" xref="S3.SS2.SSS3.p2.5.m4.1.1.3.3.3"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p2.5.m4.1.1.3.3.3.1.cmml" xref="S3.SS2.SSS3.p2.5.m4.1.1.3.3.3">subscript</csymbol><ci id="S3.SS2.SSS3.p2.5.m4.1.1.3.3.3.2.cmml" xref="S3.SS2.SSS3.p2.5.m4.1.1.3.3.3.2">𝑟</ci><ci id="S3.SS2.SSS3.p2.5.m4.1.1.3.3.3.3.cmml" xref="S3.SS2.SSS3.p2.5.m4.1.1.3.3.3.3">𝑔</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS3.p2.5.m4.1c">\mathbf{A}_{g_{j}}\in\mathbb{R}^{d\times r_{g}}</annotation></semantics></math> and <math id="S3.SS2.SSS3.p2.6.m5.1" class="ltx_Math" alttext="\mathbf{B}_{g_{j}}\in\mathbb{R}^{r_{g}\times(d_{h}\cdot s)}" display="inline"><semantics id="S3.SS2.SSS3.p2.6.m5.1a"><mrow id="S3.SS2.SSS3.p2.6.m5.1.2" xref="S3.SS2.SSS3.p2.6.m5.1.2.cmml"><msub id="S3.SS2.SSS3.p2.6.m5.1.2.2" xref="S3.SS2.SSS3.p2.6.m5.1.2.2.cmml"><mi id="S3.SS2.SSS3.p2.6.m5.1.2.2.2" xref="S3.SS2.SSS3.p2.6.m5.1.2.2.2.cmml">𝐁</mi><msub id="S3.SS2.SSS3.p2.6.m5.1.2.2.3" xref="S3.SS2.SSS3.p2.6.m5.1.2.2.3.cmml"><mi id="S3.SS2.SSS3.p2.6.m5.1.2.2.3.2" xref="S3.SS2.SSS3.p2.6.m5.1.2.2.3.2.cmml">g</mi><mi id="S3.SS2.SSS3.p2.6.m5.1.2.2.3.3" xref="S3.SS2.SSS3.p2.6.m5.1.2.2.3.3.cmml">j</mi></msub></msub><mo id="S3.SS2.SSS3.p2.6.m5.1.2.1" xref="S3.SS2.SSS3.p2.6.m5.1.2.1.cmml">∈</mo><msup id="S3.SS2.SSS3.p2.6.m5.1.2.3" xref="S3.SS2.SSS3.p2.6.m5.1.2.3.cmml"><mi id="S3.SS2.SSS3.p2.6.m5.1.2.3.2" xref="S3.SS2.SSS3.p2.6.m5.1.2.3.2.cmml">ℝ</mi><mrow id="S3.SS2.SSS3.p2.6.m5.1.1.1" xref="S3.SS2.SSS3.p2.6.m5.1.1.1.cmml"><msub id="S3.SS2.SSS3.p2.6.m5.1.1.1.3" xref="S3.SS2.SSS3.p2.6.m5.1.1.1.3.cmml"><mi id="S3.SS2.SSS3.p2.6.m5.1.1.1.3.2" xref="S3.SS2.SSS3.p2.6.m5.1.1.1.3.2.cmml">r</mi><mi id="S3.SS2.SSS3.p2.6.m5.1.1.1.3.3" xref="S3.SS2.SSS3.p2.6.m5.1.1.1.3.3.cmml">g</mi></msub><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.SSS3.p2.6.m5.1.1.1.2" xref="S3.SS2.SSS3.p2.6.m5.1.1.1.2.cmml">×</mo><mrow id="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1" xref="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.2" xref="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.1" xref="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.1.cmml"><msub id="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.1.2" xref="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.1.2.cmml"><mi id="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.1.2.2" xref="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.1.2.2.cmml">d</mi><mi id="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.1.2.3" xref="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.1.2.3.cmml">h</mi></msub><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.1.1" xref="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.1.1.cmml">⋅</mo><mi id="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.1.3" xref="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.1.3.cmml">s</mi></mrow><mo stretchy="false" id="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.3" xref="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS3.p2.6.m5.1b"><apply id="S3.SS2.SSS3.p2.6.m5.1.2.cmml" xref="S3.SS2.SSS3.p2.6.m5.1.2"><in id="S3.SS2.SSS3.p2.6.m5.1.2.1.cmml" xref="S3.SS2.SSS3.p2.6.m5.1.2.1"></in><apply id="S3.SS2.SSS3.p2.6.m5.1.2.2.cmml" xref="S3.SS2.SSS3.p2.6.m5.1.2.2"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p2.6.m5.1.2.2.1.cmml" xref="S3.SS2.SSS3.p2.6.m5.1.2.2">subscript</csymbol><ci id="S3.SS2.SSS3.p2.6.m5.1.2.2.2.cmml" xref="S3.SS2.SSS3.p2.6.m5.1.2.2.2">𝐁</ci><apply id="S3.SS2.SSS3.p2.6.m5.1.2.2.3.cmml" xref="S3.SS2.SSS3.p2.6.m5.1.2.2.3"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p2.6.m5.1.2.2.3.1.cmml" xref="S3.SS2.SSS3.p2.6.m5.1.2.2.3">subscript</csymbol><ci id="S3.SS2.SSS3.p2.6.m5.1.2.2.3.2.cmml" xref="S3.SS2.SSS3.p2.6.m5.1.2.2.3.2">𝑔</ci><ci id="S3.SS2.SSS3.p2.6.m5.1.2.2.3.3.cmml" xref="S3.SS2.SSS3.p2.6.m5.1.2.2.3.3">𝑗</ci></apply></apply><apply id="S3.SS2.SSS3.p2.6.m5.1.2.3.cmml" xref="S3.SS2.SSS3.p2.6.m5.1.2.3"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p2.6.m5.1.2.3.1.cmml" xref="S3.SS2.SSS3.p2.6.m5.1.2.3">superscript</csymbol><ci id="S3.SS2.SSS3.p2.6.m5.1.2.3.2.cmml" xref="S3.SS2.SSS3.p2.6.m5.1.2.3.2">ℝ</ci><apply id="S3.SS2.SSS3.p2.6.m5.1.1.1.cmml" xref="S3.SS2.SSS3.p2.6.m5.1.1.1"><times id="S3.SS2.SSS3.p2.6.m5.1.1.1.2.cmml" xref="S3.SS2.SSS3.p2.6.m5.1.1.1.2"></times><apply id="S3.SS2.SSS3.p2.6.m5.1.1.1.3.cmml" xref="S3.SS2.SSS3.p2.6.m5.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p2.6.m5.1.1.1.3.1.cmml" xref="S3.SS2.SSS3.p2.6.m5.1.1.1.3">subscript</csymbol><ci id="S3.SS2.SSS3.p2.6.m5.1.1.1.3.2.cmml" xref="S3.SS2.SSS3.p2.6.m5.1.1.1.3.2">𝑟</ci><ci id="S3.SS2.SSS3.p2.6.m5.1.1.1.3.3.cmml" xref="S3.SS2.SSS3.p2.6.m5.1.1.1.3.3">𝑔</ci></apply><apply id="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.1.cmml" xref="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1"><ci id="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.1.1.cmml" xref="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.1.1">⋅</ci><apply id="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.1.2.cmml" xref="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.1.2.1.cmml" xref="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.1.2.2.cmml" xref="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.1.2.2">𝑑</ci><ci id="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.1.2.3.cmml" xref="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.1.2.3">ℎ</ci></apply><ci id="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.1.3.cmml" xref="S3.SS2.SSS3.p2.6.m5.1.1.1.1.1.1.3">𝑠</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS3.p2.6.m5.1c">\mathbf{B}_{g_{j}}\in\mathbb{R}^{r_{g}\times(d_{h}\cdot s)}</annotation></semantics></math>.
The latent representation shared among attention heads in the same group can be computed as <math id="S3.SS2.SSS3.p2.7.m6.1" class="ltx_Math" alttext="\mathbf{h}_{\text{g}_{j}}=\mathbf{x}\mathbf{U}_{\text{g}_{j}}" display="inline"><semantics id="S3.SS2.SSS3.p2.7.m6.1a"><mrow id="S3.SS2.SSS3.p2.7.m6.1.1" xref="S3.SS2.SSS3.p2.7.m6.1.1.cmml"><msub id="S3.SS2.SSS3.p2.7.m6.1.1.2" xref="S3.SS2.SSS3.p2.7.m6.1.1.2.cmml"><mi id="S3.SS2.SSS3.p2.7.m6.1.1.2.2" xref="S3.SS2.SSS3.p2.7.m6.1.1.2.2.cmml">𝐡</mi><msub id="S3.SS2.SSS3.p2.7.m6.1.1.2.3" xref="S3.SS2.SSS3.p2.7.m6.1.1.2.3.cmml"><mtext id="S3.SS2.SSS3.p2.7.m6.1.1.2.3.2" xref="S3.SS2.SSS3.p2.7.m6.1.1.2.3.2a.cmml">g</mtext><mi id="S3.SS2.SSS3.p2.7.m6.1.1.2.3.3" xref="S3.SS2.SSS3.p2.7.m6.1.1.2.3.3.cmml">j</mi></msub></msub><mo id="S3.SS2.SSS3.p2.7.m6.1.1.1" xref="S3.SS2.SSS3.p2.7.m6.1.1.1.cmml">=</mo><msub id="S3.SS2.SSS3.p2.7.m6.1.1.3" xref="S3.SS2.SSS3.p2.7.m6.1.1.3.cmml"><mi id="S3.SS2.SSS3.p2.7.m6.1.1.3.2" xref="S3.SS2.SSS3.p2.7.m6.1.1.3.2.cmml">𝐱𝐔</mi><msub id="S3.SS2.SSS3.p2.7.m6.1.1.3.3" xref="S3.SS2.SSS3.p2.7.m6.1.1.3.3.cmml"><mtext id="S3.SS2.SSS3.p2.7.m6.1.1.3.3.2" xref="S3.SS2.SSS3.p2.7.m6.1.1.3.3.2a.cmml">g</mtext><mi id="S3.SS2.SSS3.p2.7.m6.1.1.3.3.3" xref="S3.SS2.SSS3.p2.7.m6.1.1.3.3.3.cmml">j</mi></msub></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS3.p2.7.m6.1b"><apply id="S3.SS2.SSS3.p2.7.m6.1.1.cmml" xref="S3.SS2.SSS3.p2.7.m6.1.1"><eq id="S3.SS2.SSS3.p2.7.m6.1.1.1.cmml" xref="S3.SS2.SSS3.p2.7.m6.1.1.1"></eq><apply id="S3.SS2.SSS3.p2.7.m6.1.1.2.cmml" xref="S3.SS2.SSS3.p2.7.m6.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p2.7.m6.1.1.2.1.cmml" xref="S3.SS2.SSS3.p2.7.m6.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS3.p2.7.m6.1.1.2.2.cmml" xref="S3.SS2.SSS3.p2.7.m6.1.1.2.2">𝐡</ci><apply id="S3.SS2.SSS3.p2.7.m6.1.1.2.3.cmml" xref="S3.SS2.SSS3.p2.7.m6.1.1.2.3"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p2.7.m6.1.1.2.3.1.cmml" xref="S3.SS2.SSS3.p2.7.m6.1.1.2.3">subscript</csymbol><ci id="S3.SS2.SSS3.p2.7.m6.1.1.2.3.2a.cmml" xref="S3.SS2.SSS3.p2.7.m6.1.1.2.3.2"><mtext mathsize="70%" id="S3.SS2.SSS3.p2.7.m6.1.1.2.3.2.cmml" xref="S3.SS2.SSS3.p2.7.m6.1.1.2.3.2">g</mtext></ci><ci id="S3.SS2.SSS3.p2.7.m6.1.1.2.3.3.cmml" xref="S3.SS2.SSS3.p2.7.m6.1.1.2.3.3">𝑗</ci></apply></apply><apply id="S3.SS2.SSS3.p2.7.m6.1.1.3.cmml" xref="S3.SS2.SSS3.p2.7.m6.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p2.7.m6.1.1.3.1.cmml" xref="S3.SS2.SSS3.p2.7.m6.1.1.3">subscript</csymbol><ci id="S3.SS2.SSS3.p2.7.m6.1.1.3.2.cmml" xref="S3.SS2.SSS3.p2.7.m6.1.1.3.2">𝐱𝐔</ci><apply id="S3.SS2.SSS3.p2.7.m6.1.1.3.3.cmml" xref="S3.SS2.SSS3.p2.7.m6.1.1.3.3"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p2.7.m6.1.1.3.3.1.cmml" xref="S3.SS2.SSS3.p2.7.m6.1.1.3.3">subscript</csymbol><ci id="S3.SS2.SSS3.p2.7.m6.1.1.3.3.2a.cmml" xref="S3.SS2.SSS3.p2.7.m6.1.1.3.3.2"><mtext mathsize="70%" id="S3.SS2.SSS3.p2.7.m6.1.1.3.3.2.cmml" xref="S3.SS2.SSS3.p2.7.m6.1.1.3.3.2">g</mtext></ci><ci id="S3.SS2.SSS3.p2.7.m6.1.1.3.3.3.cmml" xref="S3.SS2.SSS3.p2.7.m6.1.1.3.3.3">𝑗</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS3.p2.7.m6.1c">\mathbf{h}_{\text{g}_{j}}=\mathbf{x}\mathbf{U}_{\text{g}_{j}}</annotation></semantics></math>.
During decoding, the original key or value for each head can be reconstructed via</p>
<table id="S3.Ex3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.Ex3.m1.5" class="ltx_Math" alttext="[\mathbf{y}_{j,1}\dots\mathbf{y}_{j,s}]=\mathbf{h}_{\text{g}_{j}}\mathbf{B}_{\text{g}_{j}}." display="block"><semantics id="S3.Ex3.m1.5a"><mrow id="S3.Ex3.m1.5.5.1" xref="S3.Ex3.m1.5.5.1.1.cmml"><mrow id="S3.Ex3.m1.5.5.1.1" xref="S3.Ex3.m1.5.5.1.1.cmml"><mrow id="S3.Ex3.m1.5.5.1.1.1.1" xref="S3.Ex3.m1.5.5.1.1.1.2.cmml"><mo stretchy="false" id="S3.Ex3.m1.5.5.1.1.1.1.2" xref="S3.Ex3.m1.5.5.1.1.1.2.1.cmml">[</mo><mrow id="S3.Ex3.m1.5.5.1.1.1.1.1" xref="S3.Ex3.m1.5.5.1.1.1.1.1.cmml"><msub id="S3.Ex3.m1.5.5.1.1.1.1.1.2" xref="S3.Ex3.m1.5.5.1.1.1.1.1.2.cmml"><mi id="S3.Ex3.m1.5.5.1.1.1.1.1.2.2" xref="S3.Ex3.m1.5.5.1.1.1.1.1.2.2.cmml">𝐲</mi><mrow id="S3.Ex3.m1.2.2.2.4" xref="S3.Ex3.m1.2.2.2.3.cmml"><mi id="S3.Ex3.m1.1.1.1.1" xref="S3.Ex3.m1.1.1.1.1.cmml">j</mi><mo id="S3.Ex3.m1.2.2.2.4.1" xref="S3.Ex3.m1.2.2.2.3.cmml">,</mo><mn id="S3.Ex3.m1.2.2.2.2" xref="S3.Ex3.m1.2.2.2.2.cmml">1</mn></mrow></msub><mo lspace="0em" rspace="0em" id="S3.Ex3.m1.5.5.1.1.1.1.1.1" xref="S3.Ex3.m1.5.5.1.1.1.1.1.1.cmml">​</mo><mi mathvariant="normal" id="S3.Ex3.m1.5.5.1.1.1.1.1.3" xref="S3.Ex3.m1.5.5.1.1.1.1.1.3.cmml">…</mi><mo lspace="0em" rspace="0em" id="S3.Ex3.m1.5.5.1.1.1.1.1.1a" xref="S3.Ex3.m1.5.5.1.1.1.1.1.1.cmml">​</mo><msub id="S3.Ex3.m1.5.5.1.1.1.1.1.4" xref="S3.Ex3.m1.5.5.1.1.1.1.1.4.cmml"><mi id="S3.Ex3.m1.5.5.1.1.1.1.1.4.2" xref="S3.Ex3.m1.5.5.1.1.1.1.1.4.2.cmml">𝐲</mi><mrow id="S3.Ex3.m1.4.4.2.4" xref="S3.Ex3.m1.4.4.2.3.cmml"><mi id="S3.Ex3.m1.3.3.1.1" xref="S3.Ex3.m1.3.3.1.1.cmml">j</mi><mo id="S3.Ex3.m1.4.4.2.4.1" xref="S3.Ex3.m1.4.4.2.3.cmml">,</mo><mi id="S3.Ex3.m1.4.4.2.2" xref="S3.Ex3.m1.4.4.2.2.cmml">s</mi></mrow></msub></mrow><mo stretchy="false" id="S3.Ex3.m1.5.5.1.1.1.1.3" xref="S3.Ex3.m1.5.5.1.1.1.2.1.cmml">]</mo></mrow><mo id="S3.Ex3.m1.5.5.1.1.2" xref="S3.Ex3.m1.5.5.1.1.2.cmml">=</mo><mrow id="S3.Ex3.m1.5.5.1.1.3" xref="S3.Ex3.m1.5.5.1.1.3.cmml"><msub id="S3.Ex3.m1.5.5.1.1.3.2" xref="S3.Ex3.m1.5.5.1.1.3.2.cmml"><mi id="S3.Ex3.m1.5.5.1.1.3.2.2" xref="S3.Ex3.m1.5.5.1.1.3.2.2.cmml">𝐡</mi><msub id="S3.Ex3.m1.5.5.1.1.3.2.3" xref="S3.Ex3.m1.5.5.1.1.3.2.3.cmml"><mtext id="S3.Ex3.m1.5.5.1.1.3.2.3.2" xref="S3.Ex3.m1.5.5.1.1.3.2.3.2a.cmml">g</mtext><mi id="S3.Ex3.m1.5.5.1.1.3.2.3.3" xref="S3.Ex3.m1.5.5.1.1.3.2.3.3.cmml">j</mi></msub></msub><mo lspace="0em" rspace="0em" id="S3.Ex3.m1.5.5.1.1.3.1" xref="S3.Ex3.m1.5.5.1.1.3.1.cmml">​</mo><msub id="S3.Ex3.m1.5.5.1.1.3.3" xref="S3.Ex3.m1.5.5.1.1.3.3.cmml"><mi id="S3.Ex3.m1.5.5.1.1.3.3.2" xref="S3.Ex3.m1.5.5.1.1.3.3.2.cmml">𝐁</mi><msub id="S3.Ex3.m1.5.5.1.1.3.3.3" xref="S3.Ex3.m1.5.5.1.1.3.3.3.cmml"><mtext id="S3.Ex3.m1.5.5.1.1.3.3.3.2" xref="S3.Ex3.m1.5.5.1.1.3.3.3.2a.cmml">g</mtext><mi id="S3.Ex3.m1.5.5.1.1.3.3.3.3" xref="S3.Ex3.m1.5.5.1.1.3.3.3.3.cmml">j</mi></msub></msub></mrow></mrow><mo lspace="0em" id="S3.Ex3.m1.5.5.1.2" xref="S3.Ex3.m1.5.5.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.Ex3.m1.5b"><apply id="S3.Ex3.m1.5.5.1.1.cmml" xref="S3.Ex3.m1.5.5.1"><eq id="S3.Ex3.m1.5.5.1.1.2.cmml" xref="S3.Ex3.m1.5.5.1.1.2"></eq><apply id="S3.Ex3.m1.5.5.1.1.1.2.cmml" xref="S3.Ex3.m1.5.5.1.1.1.1"><csymbol cd="latexml" id="S3.Ex3.m1.5.5.1.1.1.2.1.cmml" xref="S3.Ex3.m1.5.5.1.1.1.1.2">delimited-[]</csymbol><apply id="S3.Ex3.m1.5.5.1.1.1.1.1.cmml" xref="S3.Ex3.m1.5.5.1.1.1.1.1"><times id="S3.Ex3.m1.5.5.1.1.1.1.1.1.cmml" xref="S3.Ex3.m1.5.5.1.1.1.1.1.1"></times><apply id="S3.Ex3.m1.5.5.1.1.1.1.1.2.cmml" xref="S3.Ex3.m1.5.5.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.Ex3.m1.5.5.1.1.1.1.1.2.1.cmml" xref="S3.Ex3.m1.5.5.1.1.1.1.1.2">subscript</csymbol><ci id="S3.Ex3.m1.5.5.1.1.1.1.1.2.2.cmml" xref="S3.Ex3.m1.5.5.1.1.1.1.1.2.2">𝐲</ci><list id="S3.Ex3.m1.2.2.2.3.cmml" xref="S3.Ex3.m1.2.2.2.4"><ci id="S3.Ex3.m1.1.1.1.1.cmml" xref="S3.Ex3.m1.1.1.1.1">𝑗</ci><cn type="integer" id="S3.Ex3.m1.2.2.2.2.cmml" xref="S3.Ex3.m1.2.2.2.2">1</cn></list></apply><ci id="S3.Ex3.m1.5.5.1.1.1.1.1.3.cmml" xref="S3.Ex3.m1.5.5.1.1.1.1.1.3">…</ci><apply id="S3.Ex3.m1.5.5.1.1.1.1.1.4.cmml" xref="S3.Ex3.m1.5.5.1.1.1.1.1.4"><csymbol cd="ambiguous" id="S3.Ex3.m1.5.5.1.1.1.1.1.4.1.cmml" xref="S3.Ex3.m1.5.5.1.1.1.1.1.4">subscript</csymbol><ci id="S3.Ex3.m1.5.5.1.1.1.1.1.4.2.cmml" xref="S3.Ex3.m1.5.5.1.1.1.1.1.4.2">𝐲</ci><list id="S3.Ex3.m1.4.4.2.3.cmml" xref="S3.Ex3.m1.4.4.2.4"><ci id="S3.Ex3.m1.3.3.1.1.cmml" xref="S3.Ex3.m1.3.3.1.1">𝑗</ci><ci id="S3.Ex3.m1.4.4.2.2.cmml" xref="S3.Ex3.m1.4.4.2.2">𝑠</ci></list></apply></apply></apply><apply id="S3.Ex3.m1.5.5.1.1.3.cmml" xref="S3.Ex3.m1.5.5.1.1.3"><times id="S3.Ex3.m1.5.5.1.1.3.1.cmml" xref="S3.Ex3.m1.5.5.1.1.3.1"></times><apply id="S3.Ex3.m1.5.5.1.1.3.2.cmml" xref="S3.Ex3.m1.5.5.1.1.3.2"><csymbol cd="ambiguous" id="S3.Ex3.m1.5.5.1.1.3.2.1.cmml" xref="S3.Ex3.m1.5.5.1.1.3.2">subscript</csymbol><ci id="S3.Ex3.m1.5.5.1.1.3.2.2.cmml" xref="S3.Ex3.m1.5.5.1.1.3.2.2">𝐡</ci><apply id="S3.Ex3.m1.5.5.1.1.3.2.3.cmml" xref="S3.Ex3.m1.5.5.1.1.3.2.3"><csymbol cd="ambiguous" id="S3.Ex3.m1.5.5.1.1.3.2.3.1.cmml" xref="S3.Ex3.m1.5.5.1.1.3.2.3">subscript</csymbol><ci id="S3.Ex3.m1.5.5.1.1.3.2.3.2a.cmml" xref="S3.Ex3.m1.5.5.1.1.3.2.3.2"><mtext mathsize="70%" id="S3.Ex3.m1.5.5.1.1.3.2.3.2.cmml" xref="S3.Ex3.m1.5.5.1.1.3.2.3.2">g</mtext></ci><ci id="S3.Ex3.m1.5.5.1.1.3.2.3.3.cmml" xref="S3.Ex3.m1.5.5.1.1.3.2.3.3">𝑗</ci></apply></apply><apply id="S3.Ex3.m1.5.5.1.1.3.3.cmml" xref="S3.Ex3.m1.5.5.1.1.3.3"><csymbol cd="ambiguous" id="S3.Ex3.m1.5.5.1.1.3.3.1.cmml" xref="S3.Ex3.m1.5.5.1.1.3.3">subscript</csymbol><ci id="S3.Ex3.m1.5.5.1.1.3.3.2.cmml" xref="S3.Ex3.m1.5.5.1.1.3.3.2">𝐁</ci><apply id="S3.Ex3.m1.5.5.1.1.3.3.3.cmml" xref="S3.Ex3.m1.5.5.1.1.3.3.3"><csymbol cd="ambiguous" id="S3.Ex3.m1.5.5.1.1.3.3.3.1.cmml" xref="S3.Ex3.m1.5.5.1.1.3.3.3">subscript</csymbol><ci id="S3.Ex3.m1.5.5.1.1.3.3.3.2a.cmml" xref="S3.Ex3.m1.5.5.1.1.3.3.3.2"><mtext mathsize="70%" id="S3.Ex3.m1.5.5.1.1.3.3.3.2.cmml" xref="S3.Ex3.m1.5.5.1.1.3.3.3.2">g</mtext></ci><ci id="S3.Ex3.m1.5.5.1.1.3.3.3.3.cmml" xref="S3.Ex3.m1.5.5.1.1.3.3.3.3">𝑗</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.Ex3.m1.5c">[\mathbf{y}_{j,1}\dots\mathbf{y}_{j,s}]=\mathbf{h}_{\text{g}_{j}}\mathbf{B}_{\text{g}_{j}}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="S3.SS2.SSS3.p3" class="ltx_para">
<p id="S3.SS2.SSS3.p3.5" class="ltx_p">The FLOPs for reconstructing the keys and values for each head in G-LRD is <math id="S3.SS2.SSS3.p3.1.m1.1" class="ltx_Math" alttext="r_{g}\cdot d_{h}\cdot n_{g}" display="inline"><semantics id="S3.SS2.SSS3.p3.1.m1.1a"><mrow id="S3.SS2.SSS3.p3.1.m1.1.1" xref="S3.SS2.SSS3.p3.1.m1.1.1.cmml"><msub id="S3.SS2.SSS3.p3.1.m1.1.1.2" xref="S3.SS2.SSS3.p3.1.m1.1.1.2.cmml"><mi id="S3.SS2.SSS3.p3.1.m1.1.1.2.2" xref="S3.SS2.SSS3.p3.1.m1.1.1.2.2.cmml">r</mi><mi id="S3.SS2.SSS3.p3.1.m1.1.1.2.3" xref="S3.SS2.SSS3.p3.1.m1.1.1.2.3.cmml">g</mi></msub><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.SSS3.p3.1.m1.1.1.1" xref="S3.SS2.SSS3.p3.1.m1.1.1.1.cmml">⋅</mo><msub id="S3.SS2.SSS3.p3.1.m1.1.1.3" xref="S3.SS2.SSS3.p3.1.m1.1.1.3.cmml"><mi id="S3.SS2.SSS3.p3.1.m1.1.1.3.2" xref="S3.SS2.SSS3.p3.1.m1.1.1.3.2.cmml">d</mi><mi id="S3.SS2.SSS3.p3.1.m1.1.1.3.3" xref="S3.SS2.SSS3.p3.1.m1.1.1.3.3.cmml">h</mi></msub><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.SSS3.p3.1.m1.1.1.1a" xref="S3.SS2.SSS3.p3.1.m1.1.1.1.cmml">⋅</mo><msub id="S3.SS2.SSS3.p3.1.m1.1.1.4" xref="S3.SS2.SSS3.p3.1.m1.1.1.4.cmml"><mi id="S3.SS2.SSS3.p3.1.m1.1.1.4.2" xref="S3.SS2.SSS3.p3.1.m1.1.1.4.2.cmml">n</mi><mi id="S3.SS2.SSS3.p3.1.m1.1.1.4.3" xref="S3.SS2.SSS3.p3.1.m1.1.1.4.3.cmml">g</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS3.p3.1.m1.1b"><apply id="S3.SS2.SSS3.p3.1.m1.1.1.cmml" xref="S3.SS2.SSS3.p3.1.m1.1.1"><ci id="S3.SS2.SSS3.p3.1.m1.1.1.1.cmml" xref="S3.SS2.SSS3.p3.1.m1.1.1.1">⋅</ci><apply id="S3.SS2.SSS3.p3.1.m1.1.1.2.cmml" xref="S3.SS2.SSS3.p3.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p3.1.m1.1.1.2.1.cmml" xref="S3.SS2.SSS3.p3.1.m1.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS3.p3.1.m1.1.1.2.2.cmml" xref="S3.SS2.SSS3.p3.1.m1.1.1.2.2">𝑟</ci><ci id="S3.SS2.SSS3.p3.1.m1.1.1.2.3.cmml" xref="S3.SS2.SSS3.p3.1.m1.1.1.2.3">𝑔</ci></apply><apply id="S3.SS2.SSS3.p3.1.m1.1.1.3.cmml" xref="S3.SS2.SSS3.p3.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p3.1.m1.1.1.3.1.cmml" xref="S3.SS2.SSS3.p3.1.m1.1.1.3">subscript</csymbol><ci id="S3.SS2.SSS3.p3.1.m1.1.1.3.2.cmml" xref="S3.SS2.SSS3.p3.1.m1.1.1.3.2">𝑑</ci><ci id="S3.SS2.SSS3.p3.1.m1.1.1.3.3.cmml" xref="S3.SS2.SSS3.p3.1.m1.1.1.3.3">ℎ</ci></apply><apply id="S3.SS2.SSS3.p3.1.m1.1.1.4.cmml" xref="S3.SS2.SSS3.p3.1.m1.1.1.4"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p3.1.m1.1.1.4.1.cmml" xref="S3.SS2.SSS3.p3.1.m1.1.1.4">subscript</csymbol><ci id="S3.SS2.SSS3.p3.1.m1.1.1.4.2.cmml" xref="S3.SS2.SSS3.p3.1.m1.1.1.4.2">𝑛</ci><ci id="S3.SS2.SSS3.p3.1.m1.1.1.4.3.cmml" xref="S3.SS2.SSS3.p3.1.m1.1.1.4.3">𝑔</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS3.p3.1.m1.1c">r_{g}\cdot d_{h}\cdot n_{g}</annotation></semantics></math>,
where <math id="S3.SS2.SSS3.p3.2.m2.1" class="ltx_Math" alttext="n_{g}=\frac{n}{s}" display="inline"><semantics id="S3.SS2.SSS3.p3.2.m2.1a"><mrow id="S3.SS2.SSS3.p3.2.m2.1.1" xref="S3.SS2.SSS3.p3.2.m2.1.1.cmml"><msub id="S3.SS2.SSS3.p3.2.m2.1.1.2" xref="S3.SS2.SSS3.p3.2.m2.1.1.2.cmml"><mi id="S3.SS2.SSS3.p3.2.m2.1.1.2.2" xref="S3.SS2.SSS3.p3.2.m2.1.1.2.2.cmml">n</mi><mi id="S3.SS2.SSS3.p3.2.m2.1.1.2.3" xref="S3.SS2.SSS3.p3.2.m2.1.1.2.3.cmml">g</mi></msub><mo id="S3.SS2.SSS3.p3.2.m2.1.1.1" xref="S3.SS2.SSS3.p3.2.m2.1.1.1.cmml">=</mo><mfrac id="S3.SS2.SSS3.p3.2.m2.1.1.3" xref="S3.SS2.SSS3.p3.2.m2.1.1.3.cmml"><mi id="S3.SS2.SSS3.p3.2.m2.1.1.3.2" xref="S3.SS2.SSS3.p3.2.m2.1.1.3.2.cmml">n</mi><mi id="S3.SS2.SSS3.p3.2.m2.1.1.3.3" xref="S3.SS2.SSS3.p3.2.m2.1.1.3.3.cmml">s</mi></mfrac></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS3.p3.2.m2.1b"><apply id="S3.SS2.SSS3.p3.2.m2.1.1.cmml" xref="S3.SS2.SSS3.p3.2.m2.1.1"><eq id="S3.SS2.SSS3.p3.2.m2.1.1.1.cmml" xref="S3.SS2.SSS3.p3.2.m2.1.1.1"></eq><apply id="S3.SS2.SSS3.p3.2.m2.1.1.2.cmml" xref="S3.SS2.SSS3.p3.2.m2.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p3.2.m2.1.1.2.1.cmml" xref="S3.SS2.SSS3.p3.2.m2.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS3.p3.2.m2.1.1.2.2.cmml" xref="S3.SS2.SSS3.p3.2.m2.1.1.2.2">𝑛</ci><ci id="S3.SS2.SSS3.p3.2.m2.1.1.2.3.cmml" xref="S3.SS2.SSS3.p3.2.m2.1.1.2.3">𝑔</ci></apply><apply id="S3.SS2.SSS3.p3.2.m2.1.1.3.cmml" xref="S3.SS2.SSS3.p3.2.m2.1.1.3"><divide id="S3.SS2.SSS3.p3.2.m2.1.1.3.1.cmml" xref="S3.SS2.SSS3.p3.2.m2.1.1.3"></divide><ci id="S3.SS2.SSS3.p3.2.m2.1.1.3.2.cmml" xref="S3.SS2.SSS3.p3.2.m2.1.1.3.2">𝑛</ci><ci id="S3.SS2.SSS3.p3.2.m2.1.1.3.3.cmml" xref="S3.SS2.SSS3.p3.2.m2.1.1.3.3">𝑠</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS3.p3.2.m2.1c">n_{g}=\frac{n}{s}</annotation></semantics></math> is the number of groups.
Comparing the cost to J-LRD and assuming the same total rank size (<math id="S3.SS2.SSS3.p3.3.m3.1" class="ltx_Math" alttext="r_{g}\cdot n_{g}=r_{\text{joint}}" display="inline"><semantics id="S3.SS2.SSS3.p3.3.m3.1a"><mrow id="S3.SS2.SSS3.p3.3.m3.1.1" xref="S3.SS2.SSS3.p3.3.m3.1.1.cmml"><mrow id="S3.SS2.SSS3.p3.3.m3.1.1.2" xref="S3.SS2.SSS3.p3.3.m3.1.1.2.cmml"><msub id="S3.SS2.SSS3.p3.3.m3.1.1.2.2" xref="S3.SS2.SSS3.p3.3.m3.1.1.2.2.cmml"><mi id="S3.SS2.SSS3.p3.3.m3.1.1.2.2.2" xref="S3.SS2.SSS3.p3.3.m3.1.1.2.2.2.cmml">r</mi><mi id="S3.SS2.SSS3.p3.3.m3.1.1.2.2.3" xref="S3.SS2.SSS3.p3.3.m3.1.1.2.2.3.cmml">g</mi></msub><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.SSS3.p3.3.m3.1.1.2.1" xref="S3.SS2.SSS3.p3.3.m3.1.1.2.1.cmml">⋅</mo><msub id="S3.SS2.SSS3.p3.3.m3.1.1.2.3" xref="S3.SS2.SSS3.p3.3.m3.1.1.2.3.cmml"><mi id="S3.SS2.SSS3.p3.3.m3.1.1.2.3.2" xref="S3.SS2.SSS3.p3.3.m3.1.1.2.3.2.cmml">n</mi><mi id="S3.SS2.SSS3.p3.3.m3.1.1.2.3.3" xref="S3.SS2.SSS3.p3.3.m3.1.1.2.3.3.cmml">g</mi></msub></mrow><mo id="S3.SS2.SSS3.p3.3.m3.1.1.1" xref="S3.SS2.SSS3.p3.3.m3.1.1.1.cmml">=</mo><msub id="S3.SS2.SSS3.p3.3.m3.1.1.3" xref="S3.SS2.SSS3.p3.3.m3.1.1.3.cmml"><mi id="S3.SS2.SSS3.p3.3.m3.1.1.3.2" xref="S3.SS2.SSS3.p3.3.m3.1.1.3.2.cmml">r</mi><mtext id="S3.SS2.SSS3.p3.3.m3.1.1.3.3" xref="S3.SS2.SSS3.p3.3.m3.1.1.3.3a.cmml">joint</mtext></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS3.p3.3.m3.1b"><apply id="S3.SS2.SSS3.p3.3.m3.1.1.cmml" xref="S3.SS2.SSS3.p3.3.m3.1.1"><eq id="S3.SS2.SSS3.p3.3.m3.1.1.1.cmml" xref="S3.SS2.SSS3.p3.3.m3.1.1.1"></eq><apply id="S3.SS2.SSS3.p3.3.m3.1.1.2.cmml" xref="S3.SS2.SSS3.p3.3.m3.1.1.2"><ci id="S3.SS2.SSS3.p3.3.m3.1.1.2.1.cmml" xref="S3.SS2.SSS3.p3.3.m3.1.1.2.1">⋅</ci><apply id="S3.SS2.SSS3.p3.3.m3.1.1.2.2.cmml" xref="S3.SS2.SSS3.p3.3.m3.1.1.2.2"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p3.3.m3.1.1.2.2.1.cmml" xref="S3.SS2.SSS3.p3.3.m3.1.1.2.2">subscript</csymbol><ci id="S3.SS2.SSS3.p3.3.m3.1.1.2.2.2.cmml" xref="S3.SS2.SSS3.p3.3.m3.1.1.2.2.2">𝑟</ci><ci id="S3.SS2.SSS3.p3.3.m3.1.1.2.2.3.cmml" xref="S3.SS2.SSS3.p3.3.m3.1.1.2.2.3">𝑔</ci></apply><apply id="S3.SS2.SSS3.p3.3.m3.1.1.2.3.cmml" xref="S3.SS2.SSS3.p3.3.m3.1.1.2.3"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p3.3.m3.1.1.2.3.1.cmml" xref="S3.SS2.SSS3.p3.3.m3.1.1.2.3">subscript</csymbol><ci id="S3.SS2.SSS3.p3.3.m3.1.1.2.3.2.cmml" xref="S3.SS2.SSS3.p3.3.m3.1.1.2.3.2">𝑛</ci><ci id="S3.SS2.SSS3.p3.3.m3.1.1.2.3.3.cmml" xref="S3.SS2.SSS3.p3.3.m3.1.1.2.3.3">𝑔</ci></apply></apply><apply id="S3.SS2.SSS3.p3.3.m3.1.1.3.cmml" xref="S3.SS2.SSS3.p3.3.m3.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p3.3.m3.1.1.3.1.cmml" xref="S3.SS2.SSS3.p3.3.m3.1.1.3">subscript</csymbol><ci id="S3.SS2.SSS3.p3.3.m3.1.1.3.2.cmml" xref="S3.SS2.SSS3.p3.3.m3.1.1.3.2">𝑟</ci><ci id="S3.SS2.SSS3.p3.3.m3.1.1.3.3a.cmml" xref="S3.SS2.SSS3.p3.3.m3.1.1.3.3"><mtext mathsize="70%" id="S3.SS2.SSS3.p3.3.m3.1.1.3.3.cmml" xref="S3.SS2.SSS3.p3.3.m3.1.1.3.3">joint</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS3.p3.3.m3.1c">r_{g}\cdot n_{g}=r_{\text{joint}}</annotation></semantics></math>), G-LRD reduces the reconstruction cost by <math id="S3.SS2.SSS3.p3.4.m4.1" class="ltx_Math" alttext="n_{g}" display="inline"><semantics id="S3.SS2.SSS3.p3.4.m4.1a"><msub id="S3.SS2.SSS3.p3.4.m4.1.1" xref="S3.SS2.SSS3.p3.4.m4.1.1.cmml"><mi id="S3.SS2.SSS3.p3.4.m4.1.1.2" xref="S3.SS2.SSS3.p3.4.m4.1.1.2.cmml">n</mi><mi id="S3.SS2.SSS3.p3.4.m4.1.1.3" xref="S3.SS2.SSS3.p3.4.m4.1.1.3.cmml">g</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS3.p3.4.m4.1b"><apply id="S3.SS2.SSS3.p3.4.m4.1.1.cmml" xref="S3.SS2.SSS3.p3.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p3.4.m4.1.1.1.cmml" xref="S3.SS2.SSS3.p3.4.m4.1.1">subscript</csymbol><ci id="S3.SS2.SSS3.p3.4.m4.1.1.2.cmml" xref="S3.SS2.SSS3.p3.4.m4.1.1.2">𝑛</ci><ci id="S3.SS2.SSS3.p3.4.m4.1.1.3.cmml" xref="S3.SS2.SSS3.p3.4.m4.1.1.3">𝑔</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS3.p3.4.m4.1c">n_{g}</annotation></semantics></math>.
Similarly, G-LRD also reduces the fused matrix size by <math id="S3.SS2.SSS3.p3.5.m5.1" class="ltx_Math" alttext="n_{g}" display="inline"><semantics id="S3.SS2.SSS3.p3.5.m5.1a"><msub id="S3.SS2.SSS3.p3.5.m5.1.1" xref="S3.SS2.SSS3.p3.5.m5.1.1.cmml"><mi id="S3.SS2.SSS3.p3.5.m5.1.1.2" xref="S3.SS2.SSS3.p3.5.m5.1.1.2.cmml">n</mi><mi id="S3.SS2.SSS3.p3.5.m5.1.1.3" xref="S3.SS2.SSS3.p3.5.m5.1.1.3.cmml">g</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS3.p3.5.m5.1b"><apply id="S3.SS2.SSS3.p3.5.m5.1.1.cmml" xref="S3.SS2.SSS3.p3.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p3.5.m5.1.1.1.cmml" xref="S3.SS2.SSS3.p3.5.m5.1.1">subscript</csymbol><ci id="S3.SS2.SSS3.p3.5.m5.1.1.2.cmml" xref="S3.SS2.SSS3.p3.5.m5.1.1.2">𝑛</ci><ci id="S3.SS2.SSS3.p3.5.m5.1.1.3.cmml" xref="S3.SS2.SSS3.p3.5.m5.1.1.3">𝑔</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS3.p3.5.m5.1c">n_{g}</annotation></semantics></math>.
To sum up, G-LRD offers a middle ground between computation overhead and approximation accuracy.
We illustrate M-LRD, J-LRD and G-LRD in Fig.<a href="#S3.F3" title="Figure 3 ‣ High Inference Overhead. ‣ 3.2.2 Joint-Head Low-Rank Decomposition ‣ 3.2 Decomposition Granularity ‣ 3 The Palu Framework ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>.</p>
</div>
<div id="S3.SS2.SSS3.p4" class="ltx_para">
<p id="S3.SS2.SSS3.p4.1" class="ltx_p">For further discussion on decomposed weight matrix size, please refer to Appendix <a href="#A2" title="Appendix B Discussion regarding memory usage ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a>, where more detailed information is provided.</p>
</div>
</section>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Automatic Rank Allocation</h3>

<div id="S3.SS3.p1" class="ltx_para">
<p id="S3.SS3.p1.1" class="ltx_p">To allocate an ideal rank size to the decomposition target, it is crucial to accurately estimate the importance of the target matrix (<span id="S3.SS3.p1.1.1" class="ltx_text ltx_font_italic">e.g.,</span> grouped weights). In <span id="S3.SS3.p1.1.2" class="ltx_text ltx_font_italic">Palu</span>, we identify <span id="S3.SS3.p1.1.3" class="ltx_text ltx_font_bold">Fisher information</span> <cite class="ltx_cite ltx_citemacro_citep">(Ly et al., <a href="#bib.bib27" title="" class="ltx_ref">2017</a>; Liu et al., <a href="#bib.bib24" title="" class="ltx_ref">2021</a>)</cite> as an accurate approximator since it can quantify the amount of information for each parameter.
We then employ the sum of Fisher information to estimate the importance of the weight matrix of each linear layer.</p>
</div>
<div id="S3.SS3.p2" class="ltx_para">
<p id="S3.SS3.p2.1" class="ltx_p">Assuming that the compression sensitivity is proportional to Fisher information, we determine the rank for each weight matrix by computing the ratio of its Fisher information to the total Fisher information across all decomposition targets.
We use this ratio to allocate the compression rate (<span id="S3.SS3.p2.1.1" class="ltx_text ltx_font_italic">i.e.,</span> rank level <math id="S3.SS3.p2.1.m1.1" class="ltx_Math" alttext="r" display="inline"><semantics id="S3.SS3.p2.1.m1.1a"><mi id="S3.SS3.p2.1.m1.1.1" xref="S3.SS3.p2.1.m1.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.1.m1.1b"><ci id="S3.SS3.p2.1.m1.1.1.cmml" xref="S3.SS3.p2.1.m1.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.1.m1.1c">r</annotation></semantics></math>), ensuring that more important layers retain higher rank levels.</p>
</div>
<div id="S3.SS3.p3" class="ltx_para">
<p id="S3.SS3.p3.1" class="ltx_p">For a detailed ablation study on our Automatic Rank Allocation method, please refer to Appendix <a href="#A8.SS3" title="H.3 Automatic Rank Allocation vs. Uniform Rank Allocation ‣ Appendix H Ablation Study ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">H.3</span></a>, where we analyze the impact of our strategy on model performance.</p>
</div>
</section>
<section id="S3.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.4 </span>Low-Rank-Aware Quantization</h3>

<figure id="S3.F4" class="ltx_figure"><img src="/html/2407.21118/assets/x4.png" id="S3.F4.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="219" height="99" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>Activation distribution of the low-rank compressed key caches at the <math id="S3.F4.2.m1.1" class="ltx_Math" alttext="4^{th}" display="inline"><semantics id="S3.F4.2.m1.1b"><msup id="S3.F4.2.m1.1.1" xref="S3.F4.2.m1.1.1.cmml"><mn id="S3.F4.2.m1.1.1.2" xref="S3.F4.2.m1.1.1.2.cmml">4</mn><mrow id="S3.F4.2.m1.1.1.3" xref="S3.F4.2.m1.1.1.3.cmml"><mi id="S3.F4.2.m1.1.1.3.2" xref="S3.F4.2.m1.1.1.3.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="S3.F4.2.m1.1.1.3.1" xref="S3.F4.2.m1.1.1.3.1.cmml">​</mo><mi id="S3.F4.2.m1.1.1.3.3" xref="S3.F4.2.m1.1.1.3.3.cmml">h</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="S3.F4.2.m1.1c"><apply id="S3.F4.2.m1.1.1.cmml" xref="S3.F4.2.m1.1.1"><csymbol cd="ambiguous" id="S3.F4.2.m1.1.1.1.cmml" xref="S3.F4.2.m1.1.1">superscript</csymbol><cn type="integer" id="S3.F4.2.m1.1.1.2.cmml" xref="S3.F4.2.m1.1.1.2">4</cn><apply id="S3.F4.2.m1.1.1.3.cmml" xref="S3.F4.2.m1.1.1.3"><times id="S3.F4.2.m1.1.1.3.1.cmml" xref="S3.F4.2.m1.1.1.3.1"></times><ci id="S3.F4.2.m1.1.1.3.2.cmml" xref="S3.F4.2.m1.1.1.3.2">𝑡</ci><ci id="S3.F4.2.m1.1.1.3.3.cmml" xref="S3.F4.2.m1.1.1.3.3">ℎ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F4.2.m1.1d">4^{th}</annotation></semantics></math> Llama-2 attention layer.</figcaption>
</figure>
<div id="S3.SS4.p1" class="ltx_para">
<p id="S3.SS4.p1.1" class="ltx_p">We integrate quantization into <span id="S3.SS4.p1.1.1" class="ltx_text ltx_font_italic">Palu</span> to further compress the KV-Cache. However, we observe that low-rank compressed latent representations have severe outliers, which limit quantization applicability in <span id="S3.SS4.p1.1.2" class="ltx_text ltx_font_italic">Palu</span>. Unlike natural outliers described in previous KV-Cache quantization literature <cite class="ltx_cite ltx_citemacro_citep">(Liu et al., <a href="#bib.bib26" title="" class="ltx_ref">2024</a>; Hooper et al., <a href="#bib.bib16" title="" class="ltx_ref">2024</a>)</cite>, these outliers are induced by SVD-based low-rank factorization.</p>
</div>
<div id="S3.SS4.p2" class="ltx_para">
<p id="S3.SS4.p2.1" class="ltx_p">Fig. <a href="#S3.F4" title="Figure 4 ‣ 3.4 Low-Rank-Aware Quantization ‣ 3 The Palu Framework ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> (a) shows the distribution of low-rank compressed key states from a layer of Llama-2 with G-LRD. Repeating outlier patterns appear at the beginning of each decomposed group because SVD arranges larger eigenvalues in the initial rows or columns, resulting in rapidly descending values in the latent representation. This pattern stretches the data distribution and hurts quantization accuracy.</p>
</div>
<div id="S3.SS4.p3" class="ltx_para">
<p id="S3.SS4.p3.1" class="ltx_p">Inspired by recent LLM quantization literature <cite class="ltx_cite ltx_citemacro_citep">(Ashkboos et al., <a href="#bib.bib2" title="" class="ltx_ref">2024b</a>; Tseng et al., <a href="#bib.bib40" title="" class="ltx_ref">2024</a>)</cite>, we apply the Walsh-Hadamard transform (WHT, <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib12" title="" class="ltx_ref">Fino and Algazi</a></cite>) to eliminate outliers (Fig. <a href="#S3.F4" title="Figure 4 ‣ 3.4 Low-Rank-Aware Quantization ‣ 3 The Palu Framework ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> (b)),
enabling a high quantization accuracy.
However, this transformation introduces an extra matrix multiplication with associated runtime overhead.
Unlike earlier methods <cite class="ltx_cite ltx_citemacro_citep">(Ashkboos et al., <a href="#bib.bib2" title="" class="ltx_ref">2024b</a>)</cite> that must apply online WHT when quantizing KV-Cache, we optimize this process by integrating the Hadamard matrix into low-rank decomposed weights with no additional overhead, as described by</p>
</div>
<div id="S3.SS4.p4" class="ltx_para">
<table id="S3.E7" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E7.m1.2" class="ltx_Math" alttext="\mathbf{W}\approx\mathbf{AB}=(\mathbf{AR})(\mathbf{R}^{T}\mathbf{B})=\mathbf{\hat{A}\hat{B}}," display="block"><semantics id="S3.E7.m1.2a"><mrow id="S3.E7.m1.2.2.1" xref="S3.E7.m1.2.2.1.1.cmml"><mrow id="S3.E7.m1.2.2.1.1" xref="S3.E7.m1.2.2.1.1.cmml"><mi id="S3.E7.m1.2.2.1.1.3" xref="S3.E7.m1.2.2.1.1.3.cmml">𝐖</mi><mo id="S3.E7.m1.2.2.1.1.4" xref="S3.E7.m1.2.2.1.1.4.cmml">≈</mo><mi id="S3.E7.m1.2.2.1.1.5" xref="S3.E7.m1.2.2.1.1.5.cmml">𝐀𝐁</mi><mo id="S3.E7.m1.2.2.1.1.6" xref="S3.E7.m1.2.2.1.1.6.cmml">=</mo><mrow id="S3.E7.m1.2.2.1.1.1" xref="S3.E7.m1.2.2.1.1.1.cmml"><mrow id="S3.E7.m1.2.2.1.1.1.3.2" xref="S3.E7.m1.2.2.1.1.1.cmml"><mo stretchy="false" id="S3.E7.m1.2.2.1.1.1.3.2.1" xref="S3.E7.m1.2.2.1.1.1.cmml">(</mo><mi id="S3.E7.m1.1.1" xref="S3.E7.m1.1.1.cmml">𝐀𝐑</mi><mo stretchy="false" id="S3.E7.m1.2.2.1.1.1.3.2.2" xref="S3.E7.m1.2.2.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S3.E7.m1.2.2.1.1.1.2" xref="S3.E7.m1.2.2.1.1.1.2.cmml">​</mo><mrow id="S3.E7.m1.2.2.1.1.1.1.1" xref="S3.E7.m1.2.2.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E7.m1.2.2.1.1.1.1.1.2" xref="S3.E7.m1.2.2.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E7.m1.2.2.1.1.1.1.1.1" xref="S3.E7.m1.2.2.1.1.1.1.1.1.cmml"><msup id="S3.E7.m1.2.2.1.1.1.1.1.1.2" xref="S3.E7.m1.2.2.1.1.1.1.1.1.2.cmml"><mi id="S3.E7.m1.2.2.1.1.1.1.1.1.2.2" xref="S3.E7.m1.2.2.1.1.1.1.1.1.2.2.cmml">𝐑</mi><mi id="S3.E7.m1.2.2.1.1.1.1.1.1.2.3" xref="S3.E7.m1.2.2.1.1.1.1.1.1.2.3.cmml">T</mi></msup><mo lspace="0em" rspace="0em" id="S3.E7.m1.2.2.1.1.1.1.1.1.1" xref="S3.E7.m1.2.2.1.1.1.1.1.1.1.cmml">​</mo><mi id="S3.E7.m1.2.2.1.1.1.1.1.1.3" xref="S3.E7.m1.2.2.1.1.1.1.1.1.3.cmml">𝐁</mi></mrow><mo stretchy="false" id="S3.E7.m1.2.2.1.1.1.1.1.3" xref="S3.E7.m1.2.2.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.E7.m1.2.2.1.1.7" xref="S3.E7.m1.2.2.1.1.7.cmml">=</mo><mrow id="S3.E7.m1.2.2.1.1.8" xref="S3.E7.m1.2.2.1.1.8.cmml"><mover accent="true" id="S3.E7.m1.2.2.1.1.8.2" xref="S3.E7.m1.2.2.1.1.8.2.cmml"><mi id="S3.E7.m1.2.2.1.1.8.2.2" xref="S3.E7.m1.2.2.1.1.8.2.2.cmml">𝐀</mi><mo id="S3.E7.m1.2.2.1.1.8.2.1" xref="S3.E7.m1.2.2.1.1.8.2.1.cmml">^</mo></mover><mo lspace="0em" rspace="0em" id="S3.E7.m1.2.2.1.1.8.1" xref="S3.E7.m1.2.2.1.1.8.1.cmml">​</mo><mover accent="true" id="S3.E7.m1.2.2.1.1.8.3" xref="S3.E7.m1.2.2.1.1.8.3.cmml"><mi id="S3.E7.m1.2.2.1.1.8.3.2" xref="S3.E7.m1.2.2.1.1.8.3.2.cmml">𝐁</mi><mo id="S3.E7.m1.2.2.1.1.8.3.1" xref="S3.E7.m1.2.2.1.1.8.3.1.cmml">^</mo></mover></mrow></mrow><mo id="S3.E7.m1.2.2.1.2" xref="S3.E7.m1.2.2.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E7.m1.2b"><apply id="S3.E7.m1.2.2.1.1.cmml" xref="S3.E7.m1.2.2.1"><and id="S3.E7.m1.2.2.1.1a.cmml" xref="S3.E7.m1.2.2.1"></and><apply id="S3.E7.m1.2.2.1.1b.cmml" xref="S3.E7.m1.2.2.1"><approx id="S3.E7.m1.2.2.1.1.4.cmml" xref="S3.E7.m1.2.2.1.1.4"></approx><ci id="S3.E7.m1.2.2.1.1.3.cmml" xref="S3.E7.m1.2.2.1.1.3">𝐖</ci><ci id="S3.E7.m1.2.2.1.1.5.cmml" xref="S3.E7.m1.2.2.1.1.5">𝐀𝐁</ci></apply><apply id="S3.E7.m1.2.2.1.1c.cmml" xref="S3.E7.m1.2.2.1"><eq id="S3.E7.m1.2.2.1.1.6.cmml" xref="S3.E7.m1.2.2.1.1.6"></eq><share href="#S3.E7.m1.2.2.1.1.5.cmml" id="S3.E7.m1.2.2.1.1d.cmml" xref="S3.E7.m1.2.2.1"></share><apply id="S3.E7.m1.2.2.1.1.1.cmml" xref="S3.E7.m1.2.2.1.1.1"><times id="S3.E7.m1.2.2.1.1.1.2.cmml" xref="S3.E7.m1.2.2.1.1.1.2"></times><ci id="S3.E7.m1.1.1.cmml" xref="S3.E7.m1.1.1">𝐀𝐑</ci><apply id="S3.E7.m1.2.2.1.1.1.1.1.1.cmml" xref="S3.E7.m1.2.2.1.1.1.1.1"><times id="S3.E7.m1.2.2.1.1.1.1.1.1.1.cmml" xref="S3.E7.m1.2.2.1.1.1.1.1.1.1"></times><apply id="S3.E7.m1.2.2.1.1.1.1.1.1.2.cmml" xref="S3.E7.m1.2.2.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E7.m1.2.2.1.1.1.1.1.1.2.1.cmml" xref="S3.E7.m1.2.2.1.1.1.1.1.1.2">superscript</csymbol><ci id="S3.E7.m1.2.2.1.1.1.1.1.1.2.2.cmml" xref="S3.E7.m1.2.2.1.1.1.1.1.1.2.2">𝐑</ci><ci id="S3.E7.m1.2.2.1.1.1.1.1.1.2.3.cmml" xref="S3.E7.m1.2.2.1.1.1.1.1.1.2.3">𝑇</ci></apply><ci id="S3.E7.m1.2.2.1.1.1.1.1.1.3.cmml" xref="S3.E7.m1.2.2.1.1.1.1.1.1.3">𝐁</ci></apply></apply></apply><apply id="S3.E7.m1.2.2.1.1e.cmml" xref="S3.E7.m1.2.2.1"><eq id="S3.E7.m1.2.2.1.1.7.cmml" xref="S3.E7.m1.2.2.1.1.7"></eq><share href="#S3.E7.m1.2.2.1.1.1.cmml" id="S3.E7.m1.2.2.1.1f.cmml" xref="S3.E7.m1.2.2.1"></share><apply id="S3.E7.m1.2.2.1.1.8.cmml" xref="S3.E7.m1.2.2.1.1.8"><times id="S3.E7.m1.2.2.1.1.8.1.cmml" xref="S3.E7.m1.2.2.1.1.8.1"></times><apply id="S3.E7.m1.2.2.1.1.8.2.cmml" xref="S3.E7.m1.2.2.1.1.8.2"><ci id="S3.E7.m1.2.2.1.1.8.2.1.cmml" xref="S3.E7.m1.2.2.1.1.8.2.1">^</ci><ci id="S3.E7.m1.2.2.1.1.8.2.2.cmml" xref="S3.E7.m1.2.2.1.1.8.2.2">𝐀</ci></apply><apply id="S3.E7.m1.2.2.1.1.8.3.cmml" xref="S3.E7.m1.2.2.1.1.8.3"><ci id="S3.E7.m1.2.2.1.1.8.3.1.cmml" xref="S3.E7.m1.2.2.1.1.8.3.1">^</ci><ci id="S3.E7.m1.2.2.1.1.8.3.2.cmml" xref="S3.E7.m1.2.2.1.1.8.3.2">𝐁</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E7.m1.2c">\mathbf{W}\approx\mathbf{AB}=(\mathbf{AR})(\mathbf{R}^{T}\mathbf{B})=\mathbf{\hat{A}\hat{B}},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(7)</span></td>
</tr></tbody>
</table>
<p id="S3.SS4.p4.1" class="ltx_p">where <math id="S3.SS4.p4.1.m1.1" class="ltx_Math" alttext="\mathbf{R}" display="inline"><semantics id="S3.SS4.p4.1.m1.1a"><mi id="S3.SS4.p4.1.m1.1.1" xref="S3.SS4.p4.1.m1.1.1.cmml">𝐑</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p4.1.m1.1b"><ci id="S3.SS4.p4.1.m1.1.1.cmml" xref="S3.SS4.p4.1.m1.1.1">𝐑</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p4.1.m1.1c">\mathbf{R}</annotation></semantics></math> is the Hadamard matrix.
This optimization allows <span id="S3.SS4.p4.1.1" class="ltx_text ltx_font_italic">Palu</span> can integrate the proposed low-rank compression technique with low-bit quantization.
Our experiments show that, on top of the low-rank compression, our quantization method only <span id="S3.SS4.p4.1.2" class="ltx_text ltx_font_italic">negligibly increases perplexity</span>, even at extreme levels such as <span id="S3.SS4.p4.1.3" class="ltx_text ltx_font_italic">3-bit or 2-bit</span> (see Sec.<a href="#S4.SS2" title="4.2 Integrating Palu with Quantization ‣ 4 Experiments ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.2</span></a>).</p>
</div>
</section>
<section id="S3.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.5 </span>Efficient Inference Implementation</h3>

<section id="S3.SS5.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Support for RoPE-based Attention.</h5>

<div id="S3.SS5.SSS0.Px1.p1" class="ltx_para">
<p id="S3.SS5.SSS0.Px1.p1.1" class="ltx_p">To support RoPE-based attention, we fuse the reconstruction of original keys, applying positional embedding (<span id="S3.SS5.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_italic">i.e.</span>, RoPE <cite class="ltx_cite ltx_citemacro_citep">(Su et al., <a href="#bib.bib36" title="" class="ltx_ref">2024</a>)</cite>) and the final matrix-vector multiplication of query and reconstructed keys by implementing customized GPU kernels in Triton <cite class="ltx_cite ltx_citemacro_cite">Tillet et al. (<a href="#bib.bib38" title="" class="ltx_ref">2019</a>)</cite>. We leverage the tiling techniques akin to FlashAttention <cite class="ltx_cite ltx_citemacro_citep">(Dao et al., <a href="#bib.bib7" title="" class="ltx_ref">2022</a>)</cite> to minimize the memory reads/writes of GPU high bandwidth memory.</p>
</div>
<figure id="S3.T1" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>Perplexity and zero-shot accuracy of <span id="S3.T1.4.1" class="ltx_text ltx_font_italic">Palu</span>, with different decomposition strategies at 50% compression rate.
</figcaption>
<div id="S3.T1.2" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:370.0pt;height:144.6pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-45.3pt,17.7pt) scale(0.803425980865702,0.803425980865702) ;">
<table id="S3.T1.2.2" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S3.T1.2.2.2" class="ltx_tr">
<td id="S3.T1.2.2.2.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" rowspan="2"><span id="S3.T1.2.2.2.3.1" class="ltx_text">Model</span></td>
<td id="S3.T1.2.2.2.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" rowspan="2"><span id="S3.T1.2.2.2.4.1" class="ltx_text">Method</span></td>
<td id="S3.T1.1.1.1.1" class="ltx_td ltx_align_right ltx_border_r ltx_border_tt" colspan="2">Perplexity <math id="S3.T1.1.1.1.1.m1.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="S3.T1.1.1.1.1.m1.1a"><mo stretchy="false" id="S3.T1.1.1.1.1.m1.1.1" xref="S3.T1.1.1.1.1.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="S3.T1.1.1.1.1.m1.1b"><ci id="S3.T1.1.1.1.1.m1.1.1.cmml" xref="S3.T1.1.1.1.1.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.1.1.1.1.m1.1c">\downarrow</annotation></semantics></math>
</td>
<td id="S3.T1.2.2.2.2" class="ltx_td ltx_align_center ltx_border_tt" colspan="7">Zero-Shot Accuracy (%) <math id="S3.T1.2.2.2.2.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S3.T1.2.2.2.2.m1.1a"><mo stretchy="false" id="S3.T1.2.2.2.2.m1.1.1" xref="S3.T1.2.2.2.2.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S3.T1.2.2.2.2.m1.1b"><ci id="S3.T1.2.2.2.2.m1.1.1.cmml" xref="S3.T1.2.2.2.2.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.2.2.2.2.m1.1c">\uparrow</annotation></semantics></math>
</td>
</tr>
<tr id="S3.T1.2.2.3.1" class="ltx_tr">
<td id="S3.T1.2.2.3.1.1" class="ltx_td ltx_align_right">Wiki2</td>
<td id="S3.T1.2.2.3.1.2" class="ltx_td ltx_align_right ltx_border_r">C4</td>
<td id="S3.T1.2.2.3.1.3" class="ltx_td ltx_align_center">OBQA</td>
<td id="S3.T1.2.2.3.1.4" class="ltx_td ltx_align_center">Hella</td>
<td id="S3.T1.2.2.3.1.5" class="ltx_td ltx_align_center">PIQA</td>
<td id="S3.T1.2.2.3.1.6" class="ltx_td ltx_align_center">ARC-e</td>
<td id="S3.T1.2.2.3.1.7" class="ltx_td ltx_align_center">ARC-c</td>
<td id="S3.T1.2.2.3.1.8" class="ltx_td ltx_align_center">Wino</td>
<td id="S3.T1.2.2.3.1.9" class="ltx_td ltx_align_center">Avg.</td>
</tr>
<tr id="S3.T1.2.2.4.2" class="ltx_tr">
<td id="S3.T1.2.2.4.2.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" rowspan="4"><span id="S3.T1.2.2.4.2.1.1" class="ltx_text">Llama2-7B</span></td>
<td id="S3.T1.2.2.4.2.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">Baseline</td>
<td id="S3.T1.2.2.4.2.3" class="ltx_td ltx_align_right ltx_border_t">5.47</td>
<td id="S3.T1.2.2.4.2.4" class="ltx_td ltx_align_right ltx_border_r ltx_border_t">7.26</td>
<td id="S3.T1.2.2.4.2.5" class="ltx_td ltx_align_center ltx_border_t">44.20</td>
<td id="S3.T1.2.2.4.2.6" class="ltx_td ltx_align_center ltx_border_t">76.00</td>
<td id="S3.T1.2.2.4.2.7" class="ltx_td ltx_align_center ltx_border_t">78.07</td>
<td id="S3.T1.2.2.4.2.8" class="ltx_td ltx_align_center ltx_border_t">76.30</td>
<td id="S3.T1.2.2.4.2.9" class="ltx_td ltx_align_center ltx_border_t">46.42</td>
<td id="S3.T1.2.2.4.2.10" class="ltx_td ltx_align_center ltx_border_t">69.30</td>
<td id="S3.T1.2.2.4.2.11" class="ltx_td ltx_align_center ltx_border_t">65.05</td>
</tr>
<tr id="S3.T1.2.2.5.3" class="ltx_tr">
<td id="S3.T1.2.2.5.3.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">J-LRD</td>
<td id="S3.T1.2.2.5.3.2" class="ltx_td ltx_align_right ltx_border_t">5.62</td>
<td id="S3.T1.2.2.5.3.3" class="ltx_td ltx_align_right ltx_border_r ltx_border_t">7.75</td>
<td id="S3.T1.2.2.5.3.4" class="ltx_td ltx_align_center ltx_border_t">45.40</td>
<td id="S3.T1.2.2.5.3.5" class="ltx_td ltx_align_center ltx_border_t">75.57</td>
<td id="S3.T1.2.2.5.3.6" class="ltx_td ltx_align_center ltx_border_t">77.48</td>
<td id="S3.T1.2.2.5.3.7" class="ltx_td ltx_align_center ltx_border_t">75.97</td>
<td id="S3.T1.2.2.5.3.8" class="ltx_td ltx_align_center ltx_border_t">45.31</td>
<td id="S3.T1.2.2.5.3.9" class="ltx_td ltx_align_center ltx_border_t">69.22</td>
<td id="S3.T1.2.2.5.3.10" class="ltx_td ltx_align_center ltx_border_t">64.82</td>
</tr>
<tr id="S3.T1.2.2.6.4" class="ltx_tr">
<td id="S3.T1.2.2.6.4.1" class="ltx_td ltx_align_left ltx_border_r">G-LRD</td>
<td id="S3.T1.2.2.6.4.2" class="ltx_td ltx_align_right">6.01</td>
<td id="S3.T1.2.2.6.4.3" class="ltx_td ltx_align_right ltx_border_r">9.82</td>
<td id="S3.T1.2.2.6.4.4" class="ltx_td ltx_align_center">43.60</td>
<td id="S3.T1.2.2.6.4.5" class="ltx_td ltx_align_center">73.39</td>
<td id="S3.T1.2.2.6.4.6" class="ltx_td ltx_align_center">76.33</td>
<td id="S3.T1.2.2.6.4.7" class="ltx_td ltx_align_center">73.02</td>
<td id="S3.T1.2.2.6.4.8" class="ltx_td ltx_align_center">42.57</td>
<td id="S3.T1.2.2.6.4.9" class="ltx_td ltx_align_center">66.77</td>
<td id="S3.T1.2.2.6.4.10" class="ltx_td ltx_align_center">62.61</td>
</tr>
<tr id="S3.T1.2.2.7.5" class="ltx_tr">
<td id="S3.T1.2.2.7.5.1" class="ltx_td ltx_align_left ltx_border_r">M-LRD</td>
<td id="S3.T1.2.2.7.5.2" class="ltx_td ltx_align_right">6.75</td>
<td id="S3.T1.2.2.7.5.3" class="ltx_td ltx_align_right ltx_border_r">12.01</td>
<td id="S3.T1.2.2.7.5.4" class="ltx_td ltx_align_center">39.60</td>
<td id="S3.T1.2.2.7.5.5" class="ltx_td ltx_align_center">65.35</td>
<td id="S3.T1.2.2.7.5.6" class="ltx_td ltx_align_center">74.76</td>
<td id="S3.T1.2.2.7.5.7" class="ltx_td ltx_align_center">67.17</td>
<td id="S3.T1.2.2.7.5.8" class="ltx_td ltx_align_center">35.24</td>
<td id="S3.T1.2.2.7.5.9" class="ltx_td ltx_align_center">64.64</td>
<td id="S3.T1.2.2.7.5.10" class="ltx_td ltx_align_center">57.79</td>
</tr>
<tr id="S3.T1.2.2.8.6" class="ltx_tr">
<td id="S3.T1.2.2.8.6.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r ltx_border_t" rowspan="4"><span id="S3.T1.2.2.8.6.1.1" class="ltx_text">Llama2-13B</span></td>
<td id="S3.T1.2.2.8.6.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">Baseline</td>
<td id="S3.T1.2.2.8.6.3" class="ltx_td ltx_align_right ltx_border_t">4.88</td>
<td id="S3.T1.2.2.8.6.4" class="ltx_td ltx_align_right ltx_border_r ltx_border_t">6.70</td>
<td id="S3.T1.2.2.8.6.5" class="ltx_td ltx_align_center ltx_border_t">45.20</td>
<td id="S3.T1.2.2.8.6.6" class="ltx_td ltx_align_center ltx_border_t">79.39</td>
<td id="S3.T1.2.2.8.6.7" class="ltx_td ltx_align_center ltx_border_t">79.11</td>
<td id="S3.T1.2.2.8.6.8" class="ltx_td ltx_align_center ltx_border_t">79.42</td>
<td id="S3.T1.2.2.8.6.9" class="ltx_td ltx_align_center ltx_border_t">49.06</td>
<td id="S3.T1.2.2.8.6.10" class="ltx_td ltx_align_center ltx_border_t">72.38</td>
<td id="S3.T1.2.2.8.6.11" class="ltx_td ltx_align_center ltx_border_t">67.43</td>
</tr>
<tr id="S3.T1.2.2.9.7" class="ltx_tr">
<td id="S3.T1.2.2.9.7.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">J-LRD</td>
<td id="S3.T1.2.2.9.7.2" class="ltx_td ltx_align_right ltx_border_t">4.97</td>
<td id="S3.T1.2.2.9.7.3" class="ltx_td ltx_align_right ltx_border_r ltx_border_t">6.92</td>
<td id="S3.T1.2.2.9.7.4" class="ltx_td ltx_align_center ltx_border_t">46.40</td>
<td id="S3.T1.2.2.9.7.5" class="ltx_td ltx_align_center ltx_border_t">79.48</td>
<td id="S3.T1.2.2.9.7.6" class="ltx_td ltx_align_center ltx_border_t">78.62</td>
<td id="S3.T1.2.2.9.7.7" class="ltx_td ltx_align_center ltx_border_t">79.29</td>
<td id="S3.T1.2.2.9.7.8" class="ltx_td ltx_align_center ltx_border_t">49.91</td>
<td id="S3.T1.2.2.9.7.9" class="ltx_td ltx_align_center ltx_border_t">70.56</td>
<td id="S3.T1.2.2.9.7.10" class="ltx_td ltx_align_center ltx_border_t">67.38</td>
</tr>
<tr id="S3.T1.2.2.10.8" class="ltx_tr">
<td id="S3.T1.2.2.10.8.1" class="ltx_td ltx_align_left ltx_border_r">G-LRD</td>
<td id="S3.T1.2.2.10.8.2" class="ltx_td ltx_align_right">5.31</td>
<td id="S3.T1.2.2.10.8.3" class="ltx_td ltx_align_right ltx_border_r">7.76</td>
<td id="S3.T1.2.2.10.8.4" class="ltx_td ltx_align_center">45.60</td>
<td id="S3.T1.2.2.10.8.5" class="ltx_td ltx_align_center">77.29</td>
<td id="S3.T1.2.2.10.8.6" class="ltx_td ltx_align_center">77.42</td>
<td id="S3.T1.2.2.10.8.7" class="ltx_td ltx_align_center">76.05</td>
<td id="S3.T1.2.2.10.8.8" class="ltx_td ltx_align_center">45.99</td>
<td id="S3.T1.2.2.10.8.9" class="ltx_td ltx_align_center">72.45</td>
<td id="S3.T1.2.2.10.8.10" class="ltx_td ltx_align_center">65.80</td>
</tr>
<tr id="S3.T1.2.2.11.9" class="ltx_tr">
<td id="S3.T1.2.2.11.9.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r">M-LRD</td>
<td id="S3.T1.2.2.11.9.2" class="ltx_td ltx_align_right ltx_border_bb">5.65</td>
<td id="S3.T1.2.2.11.9.3" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r">8.34</td>
<td id="S3.T1.2.2.11.9.4" class="ltx_td ltx_align_center ltx_border_bb">43.20</td>
<td id="S3.T1.2.2.11.9.5" class="ltx_td ltx_align_center ltx_border_bb">74.34</td>
<td id="S3.T1.2.2.11.9.6" class="ltx_td ltx_align_center ltx_border_bb">77.53</td>
<td id="S3.T1.2.2.11.9.7" class="ltx_td ltx_align_center ltx_border_bb">75.76</td>
<td id="S3.T1.2.2.11.9.8" class="ltx_td ltx_align_center ltx_border_bb">45.39</td>
<td id="S3.T1.2.2.11.9.9" class="ltx_td ltx_align_center ltx_border_bb">68.98</td>
<td id="S3.T1.2.2.11.9.10" class="ltx_td ltx_align_center ltx_border_bb">64.20</td>
</tr>
</tbody>
</table>
</span></div>
</figure>
</section>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Experiments</h2>

<figure id="S4.T2" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>Accuracy of <span id="S4.T2.2.1" class="ltx_text ltx_font_italic">Palu</span> with 30% and 50% compression rate on 8 tasks in LongBench.</figcaption>
<div id="S4.T2.3" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:370.0pt;height:91.3pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-70.4pt,17.4pt) scale(0.724267301715405,0.724267301715405) ;">
<table id="S4.T2.3.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S4.T2.3.1.1.1" class="ltx_tr">
<th id="S4.T2.3.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_tt">Model</th>
<th id="S4.T2.3.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_tt">Method</th>
<th id="S4.T2.3.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">SAMSum</th>
<th id="S4.T2.3.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">LCC</th>
<th id="S4.T2.3.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">M-News</th>
<th id="S4.T2.3.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Qasper</th>
<th id="S4.T2.3.1.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">TREC</th>
<th id="S4.T2.3.1.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">QmSum</th>
<th id="S4.T2.3.1.1.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">R-Bench</th>
<th id="S4.T2.3.1.1.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">TriQA</th>
<th id="S4.T2.3.1.1.1.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Avg.</th>
</tr>
<tr id="S4.T2.3.1.2.2" class="ltx_tr">
<th id="S4.T2.3.1.2.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_t" rowspan="3"><span id="S4.T2.3.1.2.2.1.1" class="ltx_text">Vicuna-7B</span></th>
<th id="S4.T2.3.1.2.2.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_t">Baseline</th>
<th id="S4.T2.3.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">40.73</th>
<th id="S4.T2.3.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">49.20</th>
<th id="S4.T2.3.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">27.98</th>
<th id="S4.T2.3.1.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">25.57</th>
<th id="S4.T2.3.1.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">69.00</th>
<th id="S4.T2.3.1.2.2.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">22.54</th>
<th id="S4.T2.3.1.2.2.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">43.50</th>
<th id="S4.T2.3.1.2.2.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">86.42</th>
<th id="S4.T2.3.1.2.2.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">45.62</th>
</tr>
<tr id="S4.T2.3.1.3.3" class="ltx_tr">
<th id="S4.T2.3.1.3.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Palu-30%</th>
<td id="S4.T2.3.1.3.3.2" class="ltx_td ltx_align_center ltx_border_t">40.42</td>
<td id="S4.T2.3.1.3.3.3" class="ltx_td ltx_align_center ltx_border_t">49.00</td>
<td id="S4.T2.3.1.3.3.4" class="ltx_td ltx_align_center ltx_border_t">26.96</td>
<td id="S4.T2.3.1.3.3.5" class="ltx_td ltx_align_center ltx_border_t">18.54</td>
<td id="S4.T2.3.1.3.3.6" class="ltx_td ltx_align_center ltx_border_t">65.50</td>
<td id="S4.T2.3.1.3.3.7" class="ltx_td ltx_align_center ltx_border_t">21.78</td>
<td id="S4.T2.3.1.3.3.8" class="ltx_td ltx_align_center ltx_border_t">45.77</td>
<td id="S4.T2.3.1.3.3.9" class="ltx_td ltx_align_center ltx_border_t">85.22</td>
<td id="S4.T2.3.1.3.3.10" class="ltx_td ltx_align_center ltx_border_t">44.15</td>
</tr>
<tr id="S4.T2.3.1.4.4" class="ltx_tr">
<th id="S4.T2.3.1.4.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Palu-50%</th>
<td id="S4.T2.3.1.4.4.2" class="ltx_td ltx_align_center">38.36</td>
<td id="S4.T2.3.1.4.4.3" class="ltx_td ltx_align_center">34.59</td>
<td id="S4.T2.3.1.4.4.4" class="ltx_td ltx_align_center">24.78</td>
<td id="S4.T2.3.1.4.4.5" class="ltx_td ltx_align_center">19.77</td>
<td id="S4.T2.3.1.4.4.6" class="ltx_td ltx_align_center">63.50</td>
<td id="S4.T2.3.1.4.4.7" class="ltx_td ltx_align_center">21.44</td>
<td id="S4.T2.3.1.4.4.8" class="ltx_td ltx_align_center">35.42</td>
<td id="S4.T2.3.1.4.4.9" class="ltx_td ltx_align_center">79.74</td>
<td id="S4.T2.3.1.4.4.10" class="ltx_td ltx_align_center">39.70</td>
</tr>
<tr id="S4.T2.3.1.5.5" class="ltx_tr">
<th id="S4.T2.3.1.5.5.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb ltx_border_t" rowspan="3"><span id="S4.T2.3.1.5.5.1.1" class="ltx_text">Mistral-7B</span></th>
<th id="S4.T2.3.1.5.5.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Baseline</th>
<td id="S4.T2.3.1.5.5.3" class="ltx_td ltx_align_center ltx_border_t">41.71</td>
<td id="S4.T2.3.1.5.5.4" class="ltx_td ltx_align_center ltx_border_t">66.59</td>
<td id="S4.T2.3.1.5.5.5" class="ltx_td ltx_align_center ltx_border_t">20.26</td>
<td id="S4.T2.3.1.5.5.6" class="ltx_td ltx_align_center ltx_border_t">8.13</td>
<td id="S4.T2.3.1.5.5.7" class="ltx_td ltx_align_center ltx_border_t">67.00</td>
<td id="S4.T2.3.1.5.5.8" class="ltx_td ltx_align_center ltx_border_t">19.99</td>
<td id="S4.T2.3.1.5.5.9" class="ltx_td ltx_align_center ltx_border_t">59.10</td>
<td id="S4.T2.3.1.5.5.10" class="ltx_td ltx_align_center ltx_border_t">89.90</td>
<td id="S4.T2.3.1.5.5.11" class="ltx_td ltx_align_center ltx_border_t">46.59</td>
</tr>
<tr id="S4.T2.3.1.6.6" class="ltx_tr">
<th id="S4.T2.3.1.6.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Palu-30%</th>
<td id="S4.T2.3.1.6.6.2" class="ltx_td ltx_align_center ltx_border_t">43.23</td>
<td id="S4.T2.3.1.6.6.3" class="ltx_td ltx_align_center ltx_border_t">64.23</td>
<td id="S4.T2.3.1.6.6.4" class="ltx_td ltx_align_center ltx_border_t">23.50</td>
<td id="S4.T2.3.1.6.6.5" class="ltx_td ltx_align_center ltx_border_t">10.21</td>
<td id="S4.T2.3.1.6.6.6" class="ltx_td ltx_align_center ltx_border_t">72.50</td>
<td id="S4.T2.3.1.6.6.7" class="ltx_td ltx_align_center ltx_border_t">21.16</td>
<td id="S4.T2.3.1.6.6.8" class="ltx_td ltx_align_center ltx_border_t">56.80</td>
<td id="S4.T2.3.1.6.6.9" class="ltx_td ltx_align_center ltx_border_t">89.82</td>
<td id="S4.T2.3.1.6.6.10" class="ltx_td ltx_align_center ltx_border_t">47.68</td>
</tr>
<tr id="S4.T2.3.1.7.7" class="ltx_tr">
<th id="S4.T2.3.1.7.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">Palu-50%</th>
<td id="S4.T2.3.1.7.7.2" class="ltx_td ltx_align_center ltx_border_bb">36.43</td>
<td id="S4.T2.3.1.7.7.3" class="ltx_td ltx_align_center ltx_border_bb">50.22</td>
<td id="S4.T2.3.1.7.7.4" class="ltx_td ltx_align_center ltx_border_bb">19.23</td>
<td id="S4.T2.3.1.7.7.5" class="ltx_td ltx_align_center ltx_border_bb">7.54</td>
<td id="S4.T2.3.1.7.7.6" class="ltx_td ltx_align_center ltx_border_bb">67.50</td>
<td id="S4.T2.3.1.7.7.7" class="ltx_td ltx_align_center ltx_border_bb">19.56</td>
<td id="S4.T2.3.1.7.7.8" class="ltx_td ltx_align_center ltx_border_bb">44.76</td>
<td id="S4.T2.3.1.7.7.9" class="ltx_td ltx_align_center ltx_border_bb">85.37</td>
<td id="S4.T2.3.1.7.7.10" class="ltx_td ltx_align_center ltx_border_bb">41.33</td>
</tr>
</tbody>
</table>
</span></div>
</figure>
<section id="S4.SS0.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Models and Tasks.</h5>

<div id="S4.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS0.SSS0.Px1.p1.1" class="ltx_p">We evaluate <span id="S4.SS0.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_italic">Palu</span> on three LLM families, Llama2<cite class="ltx_cite ltx_citemacro_citep">(Touvron et al., <a href="#bib.bib39" title="" class="ltx_ref">2023</a>)</cite>, Mistral<cite class="ltx_cite ltx_citemacro_citep">(Jiang et al., <a href="#bib.bib19" title="" class="ltx_ref">2023</a>)</cite> and Vicuna<cite class="ltx_cite ltx_citemacro_citep">(Chiang et al., <a href="#bib.bib5" title="" class="ltx_ref">2023</a>)</cite>.
For accuracy evaluation, we measure perplexity on
the WikiText-2 <cite class="ltx_cite ltx_citemacro_citep">(Merity et al., <a href="#bib.bib29" title="" class="ltx_ref">2016</a>)</cite> and C4 <cite class="ltx_cite ltx_citemacro_citep">(Raffel et al., <a href="#bib.bib33" title="" class="ltx_ref">2020</a>)</cite> datasets and use LM-Evaluation-Harness <cite class="ltx_cite ltx_citemacro_citep">(Gao et al., <a href="#bib.bib13" title="" class="ltx_ref">2023</a>)</cite> to measure zero-shot accuracy on six common sense tasks.
We also evaluate long context accuracy on 8 tasks in LongBench <cite class="ltx_cite ltx_citemacro_citep">(Bai et al., <a href="#bib.bib3" title="" class="ltx_ref">2023</a>)</cite>.
See Appendix <a href="#A9" title="Appendix I Experiment Details ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">I</span></a> for further details on the dataset and settings.</p>
</div>
</section>
<section id="S4.SS0.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Compression Settings.</h5>

<div id="S4.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S4.SS0.SSS0.Px2.p1.1" class="ltx_p">We implemented <span id="S4.SS0.SSS0.Px2.p1.1.1" class="ltx_text ltx_font_italic">Palu</span> using the Huggingface library <cite class="ltx_cite ltx_citemacro_cite">Wolf et al. (<a href="#bib.bib43" title="" class="ltx_ref">2020</a>)</cite>. Decomposition of the key and value projection layers was performed using the truncation-aware SVD method proposed by SVD-LLM <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib42" title="" class="ltx_ref">2024</a>)</cite>. Unless otherwise specified, results for <span id="S4.SS0.SSS0.Px2.p1.1.2" class="ltx_text ltx_font_italic">Palu</span> are reported using G-LRD with a group size (gs) of 4 with equal rank size for each group. To calculate Fisher information in rank searching, we used 2048 samples from Wikitext-2, each with a sequence length of 1024. For quantization, we aligned our settings with Atom <cite class="ltx_cite ltx_citemacro_citep">(Zhao et al., <a href="#bib.bib48" title="" class="ltx_ref">2023</a>)</cite>, and use a simple per-token, asymmetric quantization to quantize the low-rank latent representation of both key and value states.</p>
</div>
</section>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Main Results</h3>

<section id="S4.SS1.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Perplexity Evaluation.</h5>

<div id="S4.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS1.SSS0.Px1.p1.1" class="ltx_p">We first present perplexity evaluation results on Wikitext-2 and C4 in Table <a href="#S3.T1" title="Table 1 ‣ Support for RoPE-based Attention. ‣ 3.5 Efficient Inference Implementation ‣ 3 The Palu Framework ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>. For the Llama2-7B model, we observe that the M-LRD method fails to maintain low perplexity at a 50% compression rate. In contrast, the J-LRD method performs significantly better given the same compression rate, achieving perplexity values of 5.62 on Wikitext-2. The G-LRD method, with a group size of 4, achieves an optimal balance between perplexity and computational cost, yielding perplexity values of 6.01 on Wikitext-2 at a 50% compression rate. The same trend is consistent across the Llama2-13B model.</p>
</div>
</section>
<section id="S4.SS1.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Zero-shot Evaluation Results.</h5>

<div id="S4.SS1.SSS0.Px2.p1" class="ltx_para">
<p id="S4.SS1.SSS0.Px2.p1.1" class="ltx_p">We further evaluate <span id="S4.SS1.SSS0.Px2.p1.1.1" class="ltx_text ltx_font_italic">Palu</span> on six common-sense zero-shot tasks, as shown in Table <a href="#S3.T1" title="Table 1 ‣ Support for RoPE-based Attention. ‣ 3.5 Efficient Inference Implementation ‣ 3 The Palu Framework ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>. Similar to the perplexity evaluation, the J-LRD method demonstrates the best performance on Llama2-7B, with only a 0.23% average accuracy degradation, while the M-LRD method results in the lowest average performance, with a 7.26% drop in accuracy compared to the baseline. In comparison, G-LRD experiences only a 2.4% average accuracy decline.</p>
</div>
</section>
<section id="S4.SS1.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">LongBench Results.</h5>

<div id="S4.SS1.SSS0.Px3.p1" class="ltx_para">
<p id="S4.SS1.SSS0.Px3.p1.1" class="ltx_p">To assess whether <span id="S4.SS1.SSS0.Px3.p1.1.1" class="ltx_text ltx_font_italic">Palu</span> can effectively compress the KV-Cache for long-context scenarios, we present the performance of <span id="S4.SS1.SSS0.Px3.p1.1.2" class="ltx_text ltx_font_italic">Palu</span> across eight tasks from LongBench.
We select Mistral-7B and Vicuna-7B <cite class="ltx_cite ltx_citemacro_citep">(Chiang et al., <a href="#bib.bib5" title="" class="ltx_ref">2023</a>)</cite> as our evaluation targets, which support sequences up to 8k and 16k, respectively. Table <a href="#S4.T2" title="Table 2 ‣ 4 Experiments ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> reports experiment results. We find that at a 50% compression level, it is relatively difficult to fully preserve accuracy.
However, when compressing KV-Cache for 30%, <span id="S4.SS1.SSS0.Px3.p1.1.3" class="ltx_text ltx_font_italic">Palu</span> only has a 1.47% averaged accuracy degradation for Vicuna-7B, and even achieves a 1.09% enhancement for Mistral-7B. These results evidence of <span id="S4.SS1.SSS0.Px3.p1.1.4" class="ltx_text ltx_font_italic">Palu</span>’s ability to shrink the KV-Cache while maintaining performance.</p>
</div>
<div id="S4.SS1.SSS0.Px3.p2" class="ltx_para">
<p id="S4.SS1.SSS0.Px3.p2.1" class="ltx_p">For detailed information on the integration of <span id="S4.SS1.SSS0.Px3.p2.1.1" class="ltx_text ltx_font_italic">Palu</span> with quantization and the performance measurements on LongBench, please refer to Appendix <a href="#A5" title="Appendix E Extended Experiments on LongBench and comparisons to quantization baselines ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">E</span></a>.</p>
</div>
</section>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Integrating <span id="S4.SS2.1.1" class="ltx_text ltx_font_italic">Palu</span> with Quantization</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.1" class="ltx_p">Table <a href="#S4.T3" title="Table 3 ‣ GEMV with Online Reconstruction. ‣ 4.3 Latency Evaluation ‣ 4 Experiments ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> shows the results of integrating quantization with <span id="S4.SS2.p1.1.1" class="ltx_text ltx_font_italic">Palu</span>. With 3-bit quantization, <span id="S4.SS2.p1.1.2" class="ltx_text ltx_font_italic">Palu</span> exhibits only a negligible perplexity increase of 0.08 at a 30% compression rate and a minimal 0.23 at a 50% compression rate, both compared to <span id="S4.SS2.p1.1.3" class="ltx_text ltx_font_italic">Palu</span> at 16-bit. This indicates a minimal trade-off for substantial compression of integrating quantization.
Furthermore, at 2-bit quantization, <span id="S4.SS2.p1.1.4" class="ltx_text ltx_font_italic">Palu</span> significantly outperforms KVQuant, which is a SOTA quantization-based method. <span id="S4.SS2.p1.1.5" class="ltx_text ltx_font_italic">Palu</span> demonstrates a compelling reduction in perplexity by 1.19 and 0.54 points, while consuming 30% and 50% less memory, respectively. These results underscore the effectiveness of <span id="S4.SS2.p1.1.6" class="ltx_text ltx_font_italic">Palu</span>, establishing it as a superior KV-Cache compression strategy.</p>
</div>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Latency Evaluation</h3>

<div id="S4.SS3.p1" class="ltx_para">
<p id="S4.SS3.p1.1" class="ltx_p">We now provide the latency evaluation for <span id="S4.SS3.p1.1.1" class="ltx_text ltx_font_italic">Palu</span>. In our experiment, latency is measured on an A6000 GPU, and the averaged results over 100 runs are reported.
For baseline, we use the default implementation from HuggingFace in PyTorch.
We do not apply FlashAttention <cite class="ltx_cite ltx_citemacro_citep">(Dao et al., <a href="#bib.bib7" title="" class="ltx_ref">2022</a>)</cite> for an apple-to-apple comparison, as our current implementation does not fuse the Softmax operation.
We leave the integration with FlashAttention as the future work.</p>
</div>
<section id="S4.SS3.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">GEMV with Online Reconstruction.</h5>

<div id="S4.SS3.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS3.SSS0.Px1.p1.1" class="ltx_p">We evaluate the performance of our fused reconstruction kernel (Sec. <a href="#S3.SS5" title="3.5 Efficient Inference Implementation ‣ 3 The Palu Framework ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.5</span></a>) for computing RoPE-based attention score.
Specifically, we measure the latency from <span id="S4.SS3.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_bold">pre-RoPE query vector to the post-GEMV attention score</span>
(refer to Fig. <a href="#S2.F2" title="Figure 2 ‣ 2.2 Singular Value Decomposition (SVD) ‣ 2 Background ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> for details).
Here, we set the compression rate for our kernel as 75%.
It’s because we observe the key states often achieve such compression rate when setting a 50% overall compression rate (see Appendix <a href="#A8.SS3" title="H.3 Automatic Rank Allocation vs. Uniform Rank Allocation ‣ Appendix H Ablation Study ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">H.3</span></a> for detailed distribution).</p>
</div>
<div id="S4.SS3.SSS0.Px1.p2" class="ltx_para">
<p id="S4.SS3.SSS0.Px1.p2.4" class="ltx_p">We compare <span id="S4.SS3.SSS0.Px1.p2.4.1" class="ltx_text ltx_font_italic">Palu</span>’s runtime against the baseline, which is simply a GEMV, and report the speedup in Fig. <a href="#S4.F5" title="Figure 5 ‣ End-to-end attention module. ‣ 4.3 Latency Evaluation ‣ 4 Experiments ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> (a). For group size 32 (J-LRD), although it’s the most accurate decomposition, the high reconstruction leads to a significant 7.09<math id="S4.SS3.SSS0.Px1.p2.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S4.SS3.SSS0.Px1.p2.1.m1.1a"><mo id="S4.SS3.SSS0.Px1.p2.1.m1.1.1" xref="S4.SS3.SSS0.Px1.p2.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px1.p2.1.m1.1b"><times id="S4.SS3.SSS0.Px1.p2.1.m1.1.1.cmml" xref="S4.SS3.SSS0.Px1.p2.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px1.p2.1.m1.1c">\times</annotation></semantics></math> slowdown at sequence length 64K .
For group size 1 (M-LRD), our fused kernel achieves up to a 2.48<math id="S4.SS3.SSS0.Px1.p2.2.m2.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S4.SS3.SSS0.Px1.p2.2.m2.1a"><mo id="S4.SS3.SSS0.Px1.p2.2.m2.1.1" xref="S4.SS3.SSS0.Px1.p2.2.m2.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px1.p2.2.m2.1b"><times id="S4.SS3.SSS0.Px1.p2.2.m2.1.1.cmml" xref="S4.SS3.SSS0.Px1.p2.2.m2.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px1.p2.2.m2.1c">\times</annotation></semantics></math> speedup at a sequence length 16k, showcasing a high speedup when more accuracy loss is acceptable.
For group size 2 and 4 (G-LRD), our fused kernel demonstrates 1.75<math id="S4.SS3.SSS0.Px1.p2.3.m3.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S4.SS3.SSS0.Px1.p2.3.m3.1a"><mo id="S4.SS3.SSS0.Px1.p2.3.m3.1.1" xref="S4.SS3.SSS0.Px1.p2.3.m3.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px1.p2.3.m3.1b"><times id="S4.SS3.SSS0.Px1.p2.3.m3.1.1.cmml" xref="S4.SS3.SSS0.Px1.p2.3.m3.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px1.p2.3.m3.1c">\times</annotation></semantics></math> and 1.16<math id="S4.SS3.SSS0.Px1.p2.4.m4.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S4.SS3.SSS0.Px1.p2.4.m4.1a"><mo id="S4.SS3.SSS0.Px1.p2.4.m4.1.1" xref="S4.SS3.SSS0.Px1.p2.4.m4.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px1.p2.4.m4.1b"><times id="S4.SS3.SSS0.Px1.p2.4.m4.1.1.cmml" xref="S4.SS3.SSS0.Px1.p2.4.m4.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px1.p2.4.m4.1c">\times</annotation></semantics></math> speedup at sequence length 16K.
These results highlight the importance of exploring trade-off between decomposition granularity, reconstruction cost, and accuracy.
Furthermore, we observe our kernel speedup decreases for sequence lengths over 16k, due to the increasing reconstruction cost shifts the attention operation from memory- to compute-bounded.</p>
</div>
<figure id="S4.T3" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 3: </span>Perplexity and KV-Cache size comparison of <span id="S4.T3.2.1" class="ltx_text ltx_font_italic">Palu</span> and other quantization-based methods for Llama2-7B on WikiText-2. For perplexity, sequence length is set to 4096, aligning with baselines. KV-Cache size is demonstrated for a sequence length of 128K. </figcaption>
<div id="S4.T3.3" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:170.0pt;height:174.5pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-31.6pt,32.5pt) scale(0.728831693433125,0.728831693433125) ;">
<table id="S4.T3.3.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T3.3.1.1.1" class="ltx_tr">
<th id="S4.T3.3.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_tt">Method</th>
<th id="S4.T3.3.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_tt">Bit</th>
<th id="S4.T3.3.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt">PPL</th>
<th id="S4.T3.3.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt">
<table id="S4.T3.3.1.1.1.4.1" class="ltx_tabular ltx_align_middle">
<tr id="S4.T3.3.1.1.1.4.1.1" class="ltx_tr">
<td id="S4.T3.3.1.1.1.4.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">KV-Cache</td>
</tr>
<tr id="S4.T3.3.1.1.1.4.1.2" class="ltx_tr">
<td id="S4.T3.3.1.1.1.4.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">Size (GB)</td>
</tr>
</table>
</th>
<th id="S4.T3.3.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">
<table id="S4.T3.3.1.1.1.5.1" class="ltx_tabular ltx_align_middle">
<tr id="S4.T3.3.1.1.1.5.1.1" class="ltx_tr">
<td id="S4.T3.3.1.1.1.5.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Comp.</td>
</tr>
<tr id="S4.T3.3.1.1.1.5.1.2" class="ltx_tr">
<td id="S4.T3.3.1.1.1.5.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">Rate</td>
</tr>
</table>
</th>
</tr>
<tr id="S4.T3.3.1.2.2" class="ltx_tr">
<th id="S4.T3.3.1.2.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_t">Baseline</th>
<th id="S4.T3.3.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_t">16</th>
<th id="S4.T3.3.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t">5.12</th>
<th id="S4.T3.3.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t">64.0</th>
<th id="S4.T3.3.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">-</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T3.3.1.3.1" class="ltx_tr">
<th id="S4.T3.3.1.3.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">Palu-30%</th>
<th id="S4.T3.3.1.3.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">16</th>
<td id="S4.T3.3.1.3.1.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">5.25</td>
<td id="S4.T3.3.1.3.1.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">44.8</td>
<td id="S4.T3.3.1.3.1.5" class="ltx_td ltx_align_center ltx_border_t">30%</td>
</tr>
<tr id="S4.T3.3.1.4.2" class="ltx_tr">
<th id="S4.T3.3.1.4.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">Palu-50%</th>
<th id="S4.T3.3.1.4.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">16</th>
<td id="S4.T3.3.1.4.2.3" class="ltx_td ltx_align_center ltx_border_r">5.63</td>
<td id="S4.T3.3.1.4.2.4" class="ltx_td ltx_align_center ltx_border_r">32.0</td>
<td id="S4.T3.3.1.4.2.5" class="ltx_td ltx_align_center">50%</td>
</tr>
<tr id="S4.T3.3.1.5.3" class="ltx_tr">
<th id="S4.T3.3.1.5.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">Atom</th>
<th id="S4.T3.3.1.5.3.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">3</th>
<td id="S4.T3.3.1.5.3.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">6.15</td>
<td id="S4.T3.3.1.5.3.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">12.6</td>
<td id="S4.T3.3.1.5.3.5" class="ltx_td ltx_align_center ltx_border_t">80.32%</td>
</tr>
<tr id="S4.T3.3.1.6.4" class="ltx_tr">
<th id="S4.T3.3.1.6.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">KVQuant</th>
<th id="S4.T3.3.1.6.4.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">3</th>
<td id="S4.T3.3.1.6.4.3" class="ltx_td ltx_align_center ltx_border_r">5.35</td>
<td id="S4.T3.3.1.6.4.4" class="ltx_td ltx_align_center ltx_border_r">12.0</td>
<td id="S4.T3.3.1.6.4.5" class="ltx_td ltx_align_center">81.25%</td>
</tr>
<tr id="S4.T3.3.1.7.5" class="ltx_tr">
<th id="S4.T3.3.1.7.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">Palu-30%</th>
<th id="S4.T3.3.1.7.5.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">3</th>
<td id="S4.T3.3.1.7.5.3" class="ltx_td ltx_align_center ltx_border_r"><span id="S4.T3.3.1.7.5.3.1" class="ltx_text ltx_font_bold">5.33</span></td>
<td id="S4.T3.3.1.7.5.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S4.T3.3.1.7.5.4.1" class="ltx_text ltx_font_bold">8.4</span></td>
<td id="S4.T3.3.1.7.5.5" class="ltx_td ltx_align_center"><span id="S4.T3.3.1.7.5.5.1" class="ltx_text ltx_font_bold">86.87%</span></td>
</tr>
<tr id="S4.T3.3.1.8.6" class="ltx_tr">
<th id="S4.T3.3.1.8.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">Palu-50%</th>
<th id="S4.T3.3.1.8.6.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">3</th>
<td id="S4.T3.3.1.8.6.3" class="ltx_td ltx_align_center ltx_border_r">5.77</td>
<td id="S4.T3.3.1.8.6.4" class="ltx_td ltx_align_center ltx_border_r">6.0</td>
<td id="S4.T3.3.1.8.6.5" class="ltx_td ltx_align_center">90.63%</td>
</tr>
<tr id="S4.T3.3.1.9.7" class="ltx_tr">
<th id="S4.T3.3.1.9.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">Atom</th>
<th id="S4.T3.3.1.9.7.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">2</th>
<td id="S4.T3.3.1.9.7.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">117.88</td>
<td id="S4.T3.3.1.9.7.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">8.6</td>
<td id="S4.T3.3.1.9.7.5" class="ltx_td ltx_align_center ltx_border_t">86.56%</td>
</tr>
<tr id="S4.T3.3.1.10.8" class="ltx_tr">
<th id="S4.T3.3.1.10.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">KVQuant</th>
<th id="S4.T3.3.1.10.8.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">2</th>
<td id="S4.T3.3.1.10.8.3" class="ltx_td ltx_align_center ltx_border_r">6.95</td>
<td id="S4.T3.3.1.10.8.4" class="ltx_td ltx_align_center ltx_border_r">8.0</td>
<td id="S4.T3.3.1.10.8.5" class="ltx_td ltx_align_center">87.50%</td>
</tr>
<tr id="S4.T3.3.1.11.9" class="ltx_tr">
<th id="S4.T3.3.1.11.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">Palu-30%</th>
<th id="S4.T3.3.1.11.9.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">2</th>
<td id="S4.T3.3.1.11.9.3" class="ltx_td ltx_align_center ltx_border_r"><span id="S4.T3.3.1.11.9.3.1" class="ltx_text ltx_font_bold">5.76</span></td>
<td id="S4.T3.3.1.11.9.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S4.T3.3.1.11.9.4.1" class="ltx_text ltx_font_bold">5.6</span></td>
<td id="S4.T3.3.1.11.9.5" class="ltx_td ltx_align_center"><span id="S4.T3.3.1.11.9.5.1" class="ltx_text ltx_font_bold">91.25%</span></td>
</tr>
<tr id="S4.T3.3.1.12.10" class="ltx_tr">
<th id="S4.T3.3.1.12.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_r">Palu-50%</th>
<th id="S4.T3.3.1.12.10.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb ltx_border_r">2</th>
<td id="S4.T3.3.1.12.10.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r"><span id="S4.T3.3.1.12.10.3.1" class="ltx_text ltx_font_bold">6.41</span></td>
<td id="S4.T3.3.1.12.10.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r"><span id="S4.T3.3.1.12.10.4.1" class="ltx_text ltx_font_bold">4.0</span></td>
<td id="S4.T3.3.1.12.10.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T3.3.1.12.10.5.1" class="ltx_text ltx_font_bold">93.75%</span></td>
</tr>
</tbody>
</table>
</span></div>
</figure>
</section>
<section id="S4.SS3.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">End-to-end attention module.</h5>

<div id="S4.SS3.SSS0.Px2.p1" class="ltx_para">
<p id="S4.SS3.SSS0.Px2.p1.3" class="ltx_p">We measure the end-to-end latency of the entire RoPE-based attention module, with the integration of our fused reconstruction kernel for key states (Sec. <a href="#S3.SS5" title="3.5 Efficient Inference Implementation ‣ 3 The Palu Framework ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.5</span></a>) and offline matrix fusion for value states (Eq. <a href="#S3.E5" title="In 3.1.1 Integration with the Attention Mechanism and Offline Matrix Fusion ‣ 3.1 Compressing the KV-Cache via Low-Rank Projection ‣ 3 The Palu Framework ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>).
Here, we set the compression rate for key and value as 75% and 25%, for an overall 50% compression rate.
The detailed forward pass operations are presented in Fig. <a href="#S2.F2" title="Figure 2 ‣ 2.2 Singular Value Decomposition (SVD) ‣ 2 Background ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.
We measure the decoding runtime for sequence length from 4K to 64K, and report the runtime of <span id="S4.SS3.SSS0.Px2.p1.3.2" class="ltx_text ltx_font_italic">Palu</span> and baseline in Fig. <a href="#S4.F5" title="Figure 5 ‣ End-to-end attention module. ‣ 4.3 Latency Evaluation ‣ 4 Experiments ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> (b).
As Fig. <a href="#S4.F5" title="Figure 5 ‣ End-to-end attention module. ‣ 4.3 Latency Evaluation ‣ 4 Experiments ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> (b), <span id="S4.SS3.SSS0.Px2.p1.3.3" class="ltx_text ltx_font_italic">Palu</span> with smaller group sizes consistently delivers more than 1.43<math id="S4.SS3.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S4.SS3.SSS0.Px2.p1.1.m1.1a"><mo id="S4.SS3.SSS0.Px2.p1.1.m1.1.1" xref="S4.SS3.SSS0.Px2.p1.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px2.p1.1.m1.1b"><times id="S4.SS3.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S4.SS3.SSS0.Px2.p1.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px2.p1.1.m1.1c">\times</annotation></semantics></math> speedup across all sequence lengths.
For group size 1 (M-LRD), <span id="S4.SS3.SSS0.Px2.p1.3.4" class="ltx_text ltx_font_italic">Palu</span> achieves a 1.98<math id="S4.SS3.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S4.SS3.SSS0.Px2.p1.2.m2.1a"><mo id="S4.SS3.SSS0.Px2.p1.2.m2.1.1" xref="S4.SS3.SSS0.Px2.p1.2.m2.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px2.p1.2.m2.1b"><times id="S4.SS3.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S4.SS3.SSS0.Px2.p1.2.m2.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px2.p1.2.m2.1c">\times</annotation></semantics></math>, close to optimal (for 50% compression rate), speedup for sequence length 16K.
For group size 4 (G-LRD), which is the sweet spot for maintaining strong accuracy and limiting reconstruction cost,
<span id="S4.SS3.SSS0.Px2.p1.3.5" class="ltx_text ltx_font_italic">Palu</span> achieves up to <span id="S4.SS3.SSS0.Px2.p1.3.1" class="ltx_text ltx_font_bold">1.61<math id="S4.SS3.SSS0.Px2.p1.3.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S4.SS3.SSS0.Px2.p1.3.1.m1.1a"><mo id="S4.SS3.SSS0.Px2.p1.3.1.m1.1.1" xref="S4.SS3.SSS0.Px2.p1.3.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px2.p1.3.1.m1.1b"><times id="S4.SS3.SSS0.Px2.p1.3.1.m1.1.1.cmml" xref="S4.SS3.SSS0.Px2.p1.3.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px2.p1.3.1.m1.1c">\times</annotation></semantics></math> end-to-end speedup</span> at sequence length 64K.</p>
</div>
<div id="S4.SS3.SSS0.Px2.p2" class="ltx_para">
<p id="S4.SS3.SSS0.Px2.p2.1" class="ltx_p">We observe increasing speedup as sequence length increases from 4k to 64k due to continuously reducing latency from compressing value states. However, speedup is expected to converge or slightly decrease with further length increases due to higher reconstruction costs and the compute-bound nature of reconstruction kernels.
We expect the speedup of <span id="S4.SS3.SSS0.Px2.p2.1.1" class="ltx_text ltx_font_italic">Palu</span> will be larger for non-RoPE-based attention, as the reconstruction overhead can be entirely eliminated by applying Eq. <a href="#S3.E6" title="In 3.1.1 Integration with the Attention Mechanism and Offline Matrix Fusion ‣ 3.1 Compressing the KV-Cache via Low-Rank Projection ‣ 3 The Palu Framework ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>.</p>
</div>
<div id="S4.SS3.SSS0.Px2.p3" class="ltx_para">
<p id="S4.SS3.SSS0.Px2.p3.1" class="ltx_p">For the end-to-end model latency, please refer to Appendix <a href="#A4" title="Appendix D End-to-End Latency Measurements ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">D</span></a>, where more detailed information and data are provided.</p>
</div>
<figure id="S4.F5" class="ltx_figure"><img src="/html/2407.21118/assets/x5.png" id="S4.F5.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="193" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 5: </span>
Normalized speedup for (a) reconstruction kernel at 75% compression and (b) RoPE-based attention module with an overall 50% compression under different group sizes.
</figcaption>
</figure>
</section>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Related Work</h2>

<section id="S5.SS0.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">SVD for LLM Compression.</h5>

<div id="S5.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S5.SS0.SSS0.Px1.p1.1" class="ltx_p">There are several related works applying SVD for LLM compression.
An early work <cite class="ltx_cite ltx_citemacro_citep">(Noach and Goldberg, <a href="#bib.bib31" title="" class="ltx_ref">2020</a>)</cite> simply compresses the weight matrices using standard SVD, resulting in high compression errors.
To address this, FWSVD <cite class="ltx_cite ltx_citemacro_citep">(Hsu et al., <a href="#bib.bib17" title="" class="ltx_ref">2022</a>)</cite> incorporates Fisher information to prioritize parameters. Additionally, ASVD <cite class="ltx_cite ltx_citemacro_citep">(Yuan et al., <a href="#bib.bib44" title="" class="ltx_ref">2023</a>)</cite> taken the activation outlier phenomenon in LLM into consideration, and propose and scaling techniques to mitigate the compression error. SVD-LLM <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib42" title="" class="ltx_ref">2024</a>)</cite> further advances this work by understanding the compression loss from each singular value, improving compression accuracy. Unlike previous related SVD works that focus on compressing model weights, <span id="S5.SS0.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_italic">Palu</span> focus on using SVD to reduce KV-Cache size.</p>
</div>
</section>
<section id="S5.SS0.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">KV-Cache Quantization.</h5>

<div id="S5.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S5.SS0.SSS0.Px2.p1.1" class="ltx_p">Quantization appears to be an effective compression technique to compress KV-Cache.
Atom <cite class="ltx_cite ltx_citemacro_citep">(Zhao et al., <a href="#bib.bib48" title="" class="ltx_ref">2023</a>)</cite> uses a simple per-token quantization scheme to compress KV-Cache.
WKVQuant <cite class="ltx_cite ltx_citemacro_citep">(Yue et al., <a href="#bib.bib45" title="" class="ltx_ref">2024</a>)</cite> proposes a two-level quantization scheme to improve quantization accuracy.
KIVI <cite class="ltx_cite ltx_citemacro_citep">(Liu et al., <a href="#bib.bib26" title="" class="ltx_ref">2024</a>)</cite> utilizes different quantization schemes, per-channel and per-token, for key and value states respectively.
KVQuant <cite class="ltx_cite ltx_citemacro_citep">(Hooper et al., <a href="#bib.bib16" title="" class="ltx_ref">2024</a>)</cite> follow a similar setting but adopts non-uniform quantization and sparse matrix for preserving outliers.
In <span id="S5.SS0.SSS0.Px2.p1.1.1" class="ltx_text ltx_font_italic">Palu</span>, we eliminate outliers with WHT to enhance quantization friendliness and, therefore, only need a simple per-token quantization and achieve a remarkable accuracy.</p>
</div>
</section>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Conclusion</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">We introduce <span id="S6.p1.1.1" class="ltx_text ltx_font_italic">Palu</span>, a novel KV-Cache compression framework that decomposes key and value linear projection weight matrices and caches the compressed latent representations. We propose various optimizations, including group-head low-rank decomposition, rank search algorithm, low-rank-aware quantization, matrix fusion, and kernel fusion. With these optimizations, <span id="S6.p1.1.2" class="ltx_text ltx_font_italic">Palu</span> can maintain accuracy while achieving significant memory reduction and faster inference. Experiments show that <span id="S6.p1.1.3" class="ltx_text ltx_font_italic">Palu</span> can accelerate the end-to-end attention module by up to 1.61<math id="S6.p1.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S6.p1.1.m1.1a"><mo id="S6.p1.1.m1.1.1" xref="S6.p1.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S6.p1.1.m1.1b"><times id="S6.p1.1.m1.1.1.cmml" xref="S6.p1.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S6.p1.1.m1.1c">\times</annotation></semantics></math> with 50% KV-Cache compression; when combined with 2-bit quantization, <span id="S6.p1.1.4" class="ltx_text ltx_font_italic">Palu</span> achieves a remarkable 91.25% compression rate, both maintaining strong accuracy.</p>
</div>
</section>
<section id="Sx1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Limitations and Future Work</h2>

<div id="Sx1.p1" class="ltx_para">
<p id="Sx1.p1.1" class="ltx_p">We introduce a novel KV-Cache compression framework based on low-rank projection. However, our experiments cover model variants with up to only 13B parameters due to limited computational resources. Future work needs to establish how <span id="Sx1.p1.1.1" class="ltx_text ltx_font_italic">Palu</span> performs for models with larger sizes (e.g., 70B or beyond). Also, we leave for future work the evaluation of end-to-end speedup when combining low-rank compression and quantization. It would also be interesting to investigate how to combine <span id="Sx1.p1.1.2" class="ltx_text ltx_font_italic">Palu</span> with other efficient LLM methods, such as token eviction or weight quantization, to achieve the highest possible acceleration.</p>
</div>
</section>
<section id="Sx2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Acknowledgement</h2>

<div id="Sx2.p1" class="ltx_para">
<p id="Sx2.p1.1" class="ltx_p">We would like to thank Sandy Kaplan for her valuable feedback and support in refining the writing of this paper. Her insights and suggestions significantly contributed to the clarity and quality of the final manuscript.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ashkboos et al. (2024a)</span>
<span class="ltx_bibblock">
Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. 2024a.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=vXxardq6db" title="" class="ltx_ref ltx_href">SliceGPT: Compress large language models by deleting rows and columns</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">The Twelfth International Conference on Learning Representations</em>.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ashkboos et al. (2024b)</span>
<span class="ltx_bibblock">
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. 2024b.

</span>
<span class="ltx_bibblock">Quarot: Outlier-free 4-bit inference in rotated llms.

</span>
<span class="ltx_bibblock"><em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2404.00456</em>.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bai et al. (2023)</span>
<span class="ltx_bibblock">
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2023.

</span>
<span class="ltx_bibblock">Longbench: A bilingual, multitask benchmark for long context understanding.

</span>
<span class="ltx_bibblock"><em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2308.14508</em>.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bisk et al. (2020)</span>
<span class="ltx_bibblock">
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020.

</span>
<span class="ltx_bibblock">Piqa: Reasoning about physical commonsense in natural language.

</span>
<span class="ltx_bibblock">In <em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">Proceedings of the AAAI conference on artificial intelligence</em>, volume 34, pages 7432–7439.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chiang et al. (2023)</span>
<span class="ltx_bibblock">
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://lmsys.org/blog/2023-03-30-vicuna/" title="" class="ltx_ref ltx_href">Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality</a>.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Clark et al. (2018)</span>
<span class="ltx_bibblock">
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.

</span>
<span class="ltx_bibblock">Think you have solved question answering? try arc, the ai2 reasoning challenge.

</span>
<span class="ltx_bibblock"><em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">arXiv:1803.05457v1</em>.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dao et al. (2022)</span>
<span class="ltx_bibblock">
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022.

</span>
<span class="ltx_bibblock">Flashattention: Fast and memory-efficient exact attention with io-awareness.

</span>
<span class="ltx_bibblock"><em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 35:16344–16359.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dasigi et al. (2021)</span>
<span class="ltx_bibblock">
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2021.naacl-main.365" title="" class="ltx_ref ltx_href">A dataset of information-seeking questions and answers anchored in research papers</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies</em>, pages 4599–4610, Online. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">DeepSeek-AI et al. (2024)</span>
<span class="ltx_bibblock">
DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie Qiu, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruizhe Pan, Runxin Xu, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Size Zheng, T. Wang, Tian Pei, Tian Yuan, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wei An, Wen
Liu, Wenfeng Liang, Wenjun Gao, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Liu, Xin Xie, Xingkai Yu, Xinnan Song, Xinyi Zhou, Xinyu Yang, Xuan Lu, Xuecheng Su, Y. Wu, Y. K. Li, Y. X. Wei, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Zheng, Yichao Zhang, Yiliang Xiong, Yilong Zhao, Ying He, Ying Tang, Yishi Piao, Yixin Dong, Yixuan Tan, Yiyuan Liu, Yongji Wang, Yongqiang Guo, Yuchen Zhu, Yuduan Wang, Yuheng Zou, Yukun Zha, Yunxian Ma, Yuting Yan, Yuxiang You, Yuxuan Liu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhewen Hao, Zhihong Shao, Zhiniu Wen, Zhipeng Xu, Zhongyu Zhang, Zhuoshu Li, Zihan Wang, Zihui Gu, Zilin Li, and Ziwei Xie. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2405.04434" title="" class="ltx_ref ltx_href">Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2405.04434.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dettmers et al. (2023)</span>
<span class="ltx_bibblock">
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023.

</span>
<span class="ltx_bibblock">Qlora: Efficient finetuning of quantized llms.

</span>
<span class="ltx_bibblock"><em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.14314</em>.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Fabbri et al. (2019)</span>
<span class="ltx_bibblock">
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/P19-1102" title="" class="ltx_ref ltx_href">Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics</em>, pages 1074–1084, Florence, Italy. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Fino and Algazi (1976)</span>
<span class="ltx_bibblock">
Fino and Algazi. 1976.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.1109/TC.1976.1674569" title="" class="ltx_ref ltx_href">Unified matrix treatment of the fast walsh-hadamard transform</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">IEEE Transactions on Computers</em>, C-25(11):1142–1146.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gao et al. (2023)</span>
<span class="ltx_bibblock">
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.5281/zenodo.10256836" title="" class="ltx_ref ltx_href">A framework for few-shot language model evaluation</a>.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gliwa et al. (2019)</span>
<span class="ltx_bibblock">
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/1911.12237" title="" class="ltx_ref ltx_href">Samsum corpus: A human-annotated dialogue dataset for abstractive summarization</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/1911.12237.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Guo et al. (2023)</span>
<span class="ltx_bibblock">
Daya Guo, Canwen Xu, Nan Duan, Jian Yin, and Julian J. McAuley. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://proceedings.mlr.press/v202/guo23j.html" title="" class="ltx_ref ltx_href">Longcoder: A long-range pre-trained language model for code completion</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA</em>, volume 202 of <em id="bib.bib15.2.2" class="ltx_emph ltx_font_italic">Proceedings of Machine Learning Research</em>, pages 12098–12107. PMLR.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hooper et al. (2024)</span>
<span class="ltx_bibblock">
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. 2024.

</span>
<span class="ltx_bibblock">Kvquant: Towards 10 million context length llm inference with kv cache quantization.

</span>
<span class="ltx_bibblock"><em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2401.18079</em>.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hsu et al. (2022)</span>
<span class="ltx_bibblock">
Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=uPv9Y3gmAI5" title="" class="ltx_ref ltx_href">Language model compression with weighted low-rank factorization</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hu et al. (2022)</span>
<span class="ltx_bibblock">
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=nZeVKeeFYf9" title="" class="ltx_ref ltx_href">LoRA: Low-rank adaptation of large language models</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jiang et al. (2023)</span>
<span class="ltx_bibblock">
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2310.06825" title="" class="ltx_ref ltx_href">Mistral 7b</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2310.06825.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jolliffe and Cadima (2016)</span>
<span class="ltx_bibblock">
Ian T Jolliffe and Jorge Cadima. 2016.

</span>
<span class="ltx_bibblock">Principal component analysis: a review and recent developments.

</span>
<span class="ltx_bibblock"><em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">Philosophical transactions of the royal society A: Mathematical, Physical and Engineering Sciences</em>, 374(2065):20150202.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Joshi et al. (2017)</span>
<span class="ltx_bibblock">
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/P17-1147" title="" class="ltx_ref ltx_href">TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)</em>, pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2023)</span>
<span class="ltx_bibblock">
Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=LywifFNXV5" title="" class="ltx_ref ltx_href">How long can context length of open-source LLMs truly promise?</a>

</span>
<span class="ltx_bibblock">In <em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following</em>.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li and Roth (2002)</span>
<span class="ltx_bibblock">
Xin Li and Dan Roth. 2002.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://aclanthology.org/C02-1150/" title="" class="ltx_ref ltx_href">Learning question classifiers</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">19th International Conference on Computational Linguistics, COLING 2002, Howard International House and Academia Sinica, Taipei, Taiwan, August 24 - September 1, 2002</em>.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2021)</span>
<span class="ltx_bibblock">
Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://proceedings.mlr.press/v139/liu21ab.html" title="" class="ltx_ref ltx_href">Group fisher pruning for practical network compression</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event</em>, volume 139 of <em id="bib.bib24.2.2" class="ltx_emph ltx_font_italic">Proceedings of Machine Learning Research</em>, pages 7021–7032. PMLR.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2023)</span>
<span class="ltx_bibblock">
Tianyang Liu, Canwen Xu, and Julian McAuley. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2306.03091" title="" class="ltx_ref ltx_href">Repobench: Benchmarking repository-level code auto-completion systems</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2306.03091.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2024)</span>
<span class="ltx_bibblock">
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024.

</span>
<span class="ltx_bibblock">Kivi: A tuning-free asymmetric 2bit quantization for kv cache.

</span>
<span class="ltx_bibblock"><em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.02750</em>.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ly et al. (2017)</span>
<span class="ltx_bibblock">
Alexander Ly, Maarten Marsman, Josine Verhagen, Raoul Grasman, and Eric-Jan Wagenmakers. 2017.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/1705.01064" title="" class="ltx_ref ltx_href">A tutorial on fisher information</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:1705.01064.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ma et al. (2023)</span>
<span class="ltx_bibblock">
Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2305.11627" title="" class="ltx_ref ltx_href">Llm-pruner: On the structural pruning of large language models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2305.11627.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Merity et al. (2016)</span>
<span class="ltx_bibblock">
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/1609.07843" title="" class="ltx_ref ltx_href">Pointer sentinel mixture models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:1609.07843.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mihaylov et al. (2018)</span>
<span class="ltx_bibblock">
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018.

</span>
<span class="ltx_bibblock">Can a suit of armor conduct electricity? a new dataset for open book question answering.

</span>
<span class="ltx_bibblock"><em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1809.02789</em>.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Noach and Goldberg (2020)</span>
<span class="ltx_bibblock">
Matan Ben Noach and Yoav Goldberg. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://aclanthology.org/2020.aacl-main.88/" title="" class="ltx_ref ltx_href">Compressing pre-trained language models by matrix decomposition</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, AACL/IJCNLP 2020, Suzhou, China, December 4-7, 2020</em>, pages 884–889. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Press et al. (2022)</span>
<span class="ltx_bibblock">
Ofir Press, Noah Smith, and Mike Lewis. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=R8sQPpGCv0" title="" class="ltx_ref ltx_href">Train short, test long: Attention with linear biases enables input length extrapolation</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Raffel et al. (2020)</span>
<span class="ltx_bibblock">
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.

</span>
<span class="ltx_bibblock">Exploring the limits of transfer learning with a unified text-to-text transformer.

</span>
<span class="ltx_bibblock"><em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">Journal of machine learning research</em>, 21(140):1–67.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sakaguchi et al. (2021)</span>
<span class="ltx_bibblock">
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021.

</span>
<span class="ltx_bibblock">Winogrande: An adversarial winograd schema challenge at scale.

</span>
<span class="ltx_bibblock"><em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">Communications of the ACM</em>, 64(9):99–106.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sharma et al. (2023)</span>
<span class="ltx_bibblock">
Pratyusha Sharma, Jordan T. Ash, and Dipendra Misra. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2312.13558" title="" class="ltx_ref ltx_href">The truth is in there: Improving reasoning in language models with layer-selective rank reduction</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2312.13558.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Su et al. (2024)</span>
<span class="ltx_bibblock">
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024.

</span>
<span class="ltx_bibblock">Roformer: Enhanced transformer with rotary position embedding.

</span>
<span class="ltx_bibblock"><em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">Neurocomputing</em>, 568:127063.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Su et al. (2021)</span>
<span class="ltx_bibblock">
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2104.09864" title="" class="ltx_ref ltx_href">Roformer: Enhanced transformer with rotary position embedding</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2104.09864.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Tillet et al. (2019)</span>
<span class="ltx_bibblock">
Philippe Tillet, Hsiang-Tsung Kung, and David Cox. 2019.

</span>
<span class="ltx_bibblock">Triton: an intermediate language and compiler for tiled neural network computations.

</span>
<span class="ltx_bibblock">In <em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages</em>, pages 10–19.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. (2023)</span>
<span class="ltx_bibblock">
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas
Scialom. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2307.09288" title="" class="ltx_ref ltx_href">Llama 2: Open foundation and fine-tuned chat models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2307.09288.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Tseng et al. (2024)</span>
<span class="ltx_bibblock">
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. 2024.

</span>
<span class="ltx_bibblock">Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks.

</span>
<span class="ltx_bibblock"><em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.04396</em>.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Vaswani et al. (2017)</span>
<span class="ltx_bibblock">
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html" title="" class="ltx_ref ltx_href">Attention is all you need</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA</em>, pages 5998–6008.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2024)</span>
<span class="ltx_bibblock">
Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. 2024.

</span>
<span class="ltx_bibblock">Svd-llm: Truncation-aware singular value decomposition for large language model compression.

</span>
<span class="ltx_bibblock"><em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2403.07378</em>.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wolf et al. (2020)</span>
<span class="ltx_bibblock">
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2020.emnlp-demos.6" title="" class="ltx_ref ltx_href">Transformers: State-of-the-art natural language processing</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations</em>, pages 38–45, Online. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yuan et al. (2023)</span>
<span class="ltx_bibblock">
Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2312.05821" title="" class="ltx_ref ltx_href">Asvd: Activation-aware singular value decomposition for compressing large language models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib44.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2312.05821.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yue et al. (2024)</span>
<span class="ltx_bibblock">
Yuxuan Yue, Zhihang Yuan, Haojie Duanmu, Sifan Zhou, Jianlong Wu, and Liqiang Nie. 2024.

</span>
<span class="ltx_bibblock">Wkvquant: Quantizing weight and key/value cache for large language models gains more.

</span>
<span class="ltx_bibblock"><em id="bib.bib45.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.12065</em>.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zellers et al. (2019)</span>
<span class="ltx_bibblock">
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.

</span>
<span class="ltx_bibblock">Hellaswag: Can a machine really finish your sentence?

</span>
<span class="ltx_bibblock"><em id="bib.bib46.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1905.07830</em>.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhao et al. (2024)</span>
<span class="ltx_bibblock">
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.48550/ARXIV.2403.03507" title="" class="ltx_ref ltx_href">Galore: Memory-efficient LLM training by gradient low-rank projection</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib47.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2403.03507.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhao et al. (2023)</span>
<span class="ltx_bibblock">
Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zhenga, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. 2023.

</span>
<span class="ltx_bibblock">Atom: Low-bit quantization for efficient and accurate llm serving.

</span>
<span class="ltx_bibblock"><em id="bib.bib48.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2310.19102</em>.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhong et al. (2021)</span>
<span class="ltx_bibblock">
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2021.naacl-main.472" title="" class="ltx_ref ltx_href">QMSum: A new benchmark for query-based multi-domain meeting summarization</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib49.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies</em>, pages 5905–5921, Online. Association for Computational Linguistics.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="Ax1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">Appendix</h2>

</section>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Quantization Basics</h2>

<div id="A1.p1" class="ltx_para">
<p id="A1.p1.6" class="ltx_p">Quantization techniques use discrete low-bit values to approximate high-precision floating points. The general asymmetric uniform quantization function is defined as:</p>
<table id="A1.E8" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.E8.m1.1" class="ltx_math_unparsed" alttext="\overline{\mathbf{X}}=\text{clamp}\Big{(}\left\lfloor\frac{\mathbf{X}}{s}\right\rceil+z,0,2^{B}-1)," display="block"><semantics id="A1.E8.m1.1a"><mrow id="A1.E8.m1.1b"><mover accent="true" id="A1.E8.m1.1.1"><mi id="A1.E8.m1.1.1.2">𝐗</mi><mo id="A1.E8.m1.1.1.1">¯</mo></mover><mo id="A1.E8.m1.1.2">=</mo><mtext id="A1.E8.m1.1.3">clamp</mtext><mrow id="A1.E8.m1.1.4"><mo maxsize="160%" minsize="160%" id="A1.E8.m1.1.4.1">(</mo><mrow id="A1.E8.m1.1.4.2"><mo id="A1.E8.m1.1.4.2.1">⌊</mo><mfrac id="A1.E8.m1.1.4.2.2"><mi id="A1.E8.m1.1.4.2.2.2">𝐗</mi><mi id="A1.E8.m1.1.4.2.2.3">s</mi></mfrac><mo id="A1.E8.m1.1.4.2.3">⌉</mo></mrow><mo id="A1.E8.m1.1.4.3">+</mo><mi id="A1.E8.m1.1.4.4">z</mi><mo id="A1.E8.m1.1.4.5">,</mo><mn id="A1.E8.m1.1.4.6">0</mn><mo id="A1.E8.m1.1.4.7">,</mo><msup id="A1.E8.m1.1.4.8"><mn id="A1.E8.m1.1.4.8.2">2</mn><mi id="A1.E8.m1.1.4.8.3">B</mi></msup><mo id="A1.E8.m1.1.4.9">−</mo><mn id="A1.E8.m1.1.4.10">1</mn><mo stretchy="false" id="A1.E8.m1.1.4.11">)</mo></mrow><mo id="A1.E8.m1.1.5">,</mo></mrow><annotation encoding="application/x-tex" id="A1.E8.m1.1c">\overline{\mathbf{X}}=\text{clamp}\Big{(}\left\lfloor\frac{\mathbf{X}}{s}\right\rceil+z,0,2^{B}-1),</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(8)</span></td>
</tr></tbody>
</table>
<p id="A1.p1.5" class="ltx_p">where <math id="A1.p1.1.m1.1" class="ltx_Math" alttext="\overline{\mathbf{X}}" display="inline"><semantics id="A1.p1.1.m1.1a"><mover accent="true" id="A1.p1.1.m1.1.1" xref="A1.p1.1.m1.1.1.cmml"><mi id="A1.p1.1.m1.1.1.2" xref="A1.p1.1.m1.1.1.2.cmml">𝐗</mi><mo id="A1.p1.1.m1.1.1.1" xref="A1.p1.1.m1.1.1.1.cmml">¯</mo></mover><annotation-xml encoding="MathML-Content" id="A1.p1.1.m1.1b"><apply id="A1.p1.1.m1.1.1.cmml" xref="A1.p1.1.m1.1.1"><ci id="A1.p1.1.m1.1.1.1.cmml" xref="A1.p1.1.m1.1.1.1">¯</ci><ci id="A1.p1.1.m1.1.1.2.cmml" xref="A1.p1.1.m1.1.1.2">𝐗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.1.m1.1c">\overline{\mathbf{X}}</annotation></semantics></math> denotes the approximated tensor with low-bit representations (i.e., 4-bit integers), <math id="A1.p1.2.m2.1" class="ltx_Math" alttext="\mathbf{X}" display="inline"><semantics id="A1.p1.2.m2.1a"><mi id="A1.p1.2.m2.1.1" xref="A1.p1.2.m2.1.1.cmml">𝐗</mi><annotation-xml encoding="MathML-Content" id="A1.p1.2.m2.1b"><ci id="A1.p1.2.m2.1.1.cmml" xref="A1.p1.2.m2.1.1">𝐗</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.2.m2.1c">\mathbf{X}</annotation></semantics></math> is the floating-point tensor, <math id="A1.p1.3.m3.1" class="ltx_Math" alttext="s=\frac{\mathbf{X}_{\text{max}}-\mathbf{X}_{\text{min}}}{2^{B}-1}" display="inline"><semantics id="A1.p1.3.m3.1a"><mrow id="A1.p1.3.m3.1.1" xref="A1.p1.3.m3.1.1.cmml"><mi id="A1.p1.3.m3.1.1.2" xref="A1.p1.3.m3.1.1.2.cmml">s</mi><mo id="A1.p1.3.m3.1.1.1" xref="A1.p1.3.m3.1.1.1.cmml">=</mo><mfrac id="A1.p1.3.m3.1.1.3" xref="A1.p1.3.m3.1.1.3.cmml"><mrow id="A1.p1.3.m3.1.1.3.2" xref="A1.p1.3.m3.1.1.3.2.cmml"><msub id="A1.p1.3.m3.1.1.3.2.2" xref="A1.p1.3.m3.1.1.3.2.2.cmml"><mi id="A1.p1.3.m3.1.1.3.2.2.2" xref="A1.p1.3.m3.1.1.3.2.2.2.cmml">𝐗</mi><mtext id="A1.p1.3.m3.1.1.3.2.2.3" xref="A1.p1.3.m3.1.1.3.2.2.3a.cmml">max</mtext></msub><mo id="A1.p1.3.m3.1.1.3.2.1" xref="A1.p1.3.m3.1.1.3.2.1.cmml">−</mo><msub id="A1.p1.3.m3.1.1.3.2.3" xref="A1.p1.3.m3.1.1.3.2.3.cmml"><mi id="A1.p1.3.m3.1.1.3.2.3.2" xref="A1.p1.3.m3.1.1.3.2.3.2.cmml">𝐗</mi><mtext id="A1.p1.3.m3.1.1.3.2.3.3" xref="A1.p1.3.m3.1.1.3.2.3.3a.cmml">min</mtext></msub></mrow><mrow id="A1.p1.3.m3.1.1.3.3" xref="A1.p1.3.m3.1.1.3.3.cmml"><msup id="A1.p1.3.m3.1.1.3.3.2" xref="A1.p1.3.m3.1.1.3.3.2.cmml"><mn id="A1.p1.3.m3.1.1.3.3.2.2" xref="A1.p1.3.m3.1.1.3.3.2.2.cmml">2</mn><mi id="A1.p1.3.m3.1.1.3.3.2.3" xref="A1.p1.3.m3.1.1.3.3.2.3.cmml">B</mi></msup><mo id="A1.p1.3.m3.1.1.3.3.1" xref="A1.p1.3.m3.1.1.3.3.1.cmml">−</mo><mn id="A1.p1.3.m3.1.1.3.3.3" xref="A1.p1.3.m3.1.1.3.3.3.cmml">1</mn></mrow></mfrac></mrow><annotation-xml encoding="MathML-Content" id="A1.p1.3.m3.1b"><apply id="A1.p1.3.m3.1.1.cmml" xref="A1.p1.3.m3.1.1"><eq id="A1.p1.3.m3.1.1.1.cmml" xref="A1.p1.3.m3.1.1.1"></eq><ci id="A1.p1.3.m3.1.1.2.cmml" xref="A1.p1.3.m3.1.1.2">𝑠</ci><apply id="A1.p1.3.m3.1.1.3.cmml" xref="A1.p1.3.m3.1.1.3"><divide id="A1.p1.3.m3.1.1.3.1.cmml" xref="A1.p1.3.m3.1.1.3"></divide><apply id="A1.p1.3.m3.1.1.3.2.cmml" xref="A1.p1.3.m3.1.1.3.2"><minus id="A1.p1.3.m3.1.1.3.2.1.cmml" xref="A1.p1.3.m3.1.1.3.2.1"></minus><apply id="A1.p1.3.m3.1.1.3.2.2.cmml" xref="A1.p1.3.m3.1.1.3.2.2"><csymbol cd="ambiguous" id="A1.p1.3.m3.1.1.3.2.2.1.cmml" xref="A1.p1.3.m3.1.1.3.2.2">subscript</csymbol><ci id="A1.p1.3.m3.1.1.3.2.2.2.cmml" xref="A1.p1.3.m3.1.1.3.2.2.2">𝐗</ci><ci id="A1.p1.3.m3.1.1.3.2.2.3a.cmml" xref="A1.p1.3.m3.1.1.3.2.2.3"><mtext mathsize="50%" id="A1.p1.3.m3.1.1.3.2.2.3.cmml" xref="A1.p1.3.m3.1.1.3.2.2.3">max</mtext></ci></apply><apply id="A1.p1.3.m3.1.1.3.2.3.cmml" xref="A1.p1.3.m3.1.1.3.2.3"><csymbol cd="ambiguous" id="A1.p1.3.m3.1.1.3.2.3.1.cmml" xref="A1.p1.3.m3.1.1.3.2.3">subscript</csymbol><ci id="A1.p1.3.m3.1.1.3.2.3.2.cmml" xref="A1.p1.3.m3.1.1.3.2.3.2">𝐗</ci><ci id="A1.p1.3.m3.1.1.3.2.3.3a.cmml" xref="A1.p1.3.m3.1.1.3.2.3.3"><mtext mathsize="50%" id="A1.p1.3.m3.1.1.3.2.3.3.cmml" xref="A1.p1.3.m3.1.1.3.2.3.3">min</mtext></ci></apply></apply><apply id="A1.p1.3.m3.1.1.3.3.cmml" xref="A1.p1.3.m3.1.1.3.3"><minus id="A1.p1.3.m3.1.1.3.3.1.cmml" xref="A1.p1.3.m3.1.1.3.3.1"></minus><apply id="A1.p1.3.m3.1.1.3.3.2.cmml" xref="A1.p1.3.m3.1.1.3.3.2"><csymbol cd="ambiguous" id="A1.p1.3.m3.1.1.3.3.2.1.cmml" xref="A1.p1.3.m3.1.1.3.3.2">superscript</csymbol><cn type="integer" id="A1.p1.3.m3.1.1.3.3.2.2.cmml" xref="A1.p1.3.m3.1.1.3.3.2.2">2</cn><ci id="A1.p1.3.m3.1.1.3.3.2.3.cmml" xref="A1.p1.3.m3.1.1.3.3.2.3">𝐵</ci></apply><cn type="integer" id="A1.p1.3.m3.1.1.3.3.3.cmml" xref="A1.p1.3.m3.1.1.3.3.3">1</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.3.m3.1c">s=\frac{\mathbf{X}_{\text{max}}-\mathbf{X}_{\text{min}}}{2^{B}-1}</annotation></semantics></math> is the scaling factor, and
<math id="A1.p1.4.m4.1" class="ltx_math_unparsed" alttext="z=\left\lfloor\frac{-\mathbf{X}_{\text{min}}}{s}\right\rceil" display="inline"><semantics id="A1.p1.4.m4.1a"><mrow id="A1.p1.4.m4.1b"><mi id="A1.p1.4.m4.1.1">z</mi><mo id="A1.p1.4.m4.1.2">=</mo><mrow id="A1.p1.4.m4.1.3"><mo id="A1.p1.4.m4.1.3.1">⌊</mo><mfrac id="A1.p1.4.m4.1.3.2"><mrow id="A1.p1.4.m4.1.3.2.2"><mo id="A1.p1.4.m4.1.3.2.2a">−</mo><msub id="A1.p1.4.m4.1.3.2.2.2"><mi id="A1.p1.4.m4.1.3.2.2.2.2">𝐗</mi><mtext id="A1.p1.4.m4.1.3.2.2.2.3">min</mtext></msub></mrow><mi id="A1.p1.4.m4.1.3.2.3">s</mi></mfrac><mo id="A1.p1.4.m4.1.3.3">⌉</mo></mrow></mrow><annotation encoding="application/x-tex" id="A1.p1.4.m4.1c">z=\left\lfloor\frac{-\mathbf{X}_{\text{min}}}{s}\right\rceil</annotation></semantics></math> is a zero-point. The <math id="A1.p1.5.m5.1" class="ltx_Math" alttext="\left\lfloor\cdot\right\rceil" display="inline"><semantics id="A1.p1.5.m5.1a"><mrow id="A1.p1.5.m5.1.2.2" xref="A1.p1.5.m5.1.2.1.cmml"><mo id="A1.p1.5.m5.1.2.2.1" xref="A1.p1.5.m5.1.2.1.1.cmml">⌊</mo><mo lspace="0em" rspace="0em" id="A1.p1.5.m5.1.1" xref="A1.p1.5.m5.1.1.cmml">⋅</mo><mo id="A1.p1.5.m5.1.2.2.2" xref="A1.p1.5.m5.1.2.1.1.cmml">⌉</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.p1.5.m5.1b"><apply id="A1.p1.5.m5.1.2.1.cmml" xref="A1.p1.5.m5.1.2.2"><csymbol cd="latexml" id="A1.p1.5.m5.1.2.1.1.cmml" xref="A1.p1.5.m5.1.2.2.1">delimited-⌊⌉</csymbol><ci id="A1.p1.5.m5.1.1.cmml" xref="A1.p1.5.m5.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.5.m5.1c">\left\lfloor\cdot\right\rceil</annotation></semantics></math> is the rounding operation.</p>
</div>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Discussion regarding memory usage</h2>

<section id="A2.SS0.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph"></h5>

<div id="A2.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="A2.SS0.SSS0.Px1.p1.3" class="ltx_p">In this work, the experimental results focus on the compression rate of the KV-Cache as a key metric. However, it is crucial to consider overall memory savings as a more significant factor. For instance, as demonstrated in Sec. <a href="#S2.SS2" title="2.2 Singular Value Decomposition (SVD) ‣ 2 Background ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.2</span></a>, a typical compression rate of 30% can lead to an increase in weight size by approximately 40%. This increase is calculated under the assumption that <math id="A2.SS0.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="m=n" display="inline"><semantics id="A2.SS0.SSS0.Px1.p1.1.m1.1a"><mrow id="A2.SS0.SSS0.Px1.p1.1.m1.1.1" xref="A2.SS0.SSS0.Px1.p1.1.m1.1.1.cmml"><mi id="A2.SS0.SSS0.Px1.p1.1.m1.1.1.2" xref="A2.SS0.SSS0.Px1.p1.1.m1.1.1.2.cmml">m</mi><mo id="A2.SS0.SSS0.Px1.p1.1.m1.1.1.1" xref="A2.SS0.SSS0.Px1.p1.1.m1.1.1.1.cmml">=</mo><mi id="A2.SS0.SSS0.Px1.p1.1.m1.1.1.3" xref="A2.SS0.SSS0.Px1.p1.1.m1.1.1.3.cmml">n</mi></mrow><annotation-xml encoding="MathML-Content" id="A2.SS0.SSS0.Px1.p1.1.m1.1b"><apply id="A2.SS0.SSS0.Px1.p1.1.m1.1.1.cmml" xref="A2.SS0.SSS0.Px1.p1.1.m1.1.1"><eq id="A2.SS0.SSS0.Px1.p1.1.m1.1.1.1.cmml" xref="A2.SS0.SSS0.Px1.p1.1.m1.1.1.1"></eq><ci id="A2.SS0.SSS0.Px1.p1.1.m1.1.1.2.cmml" xref="A2.SS0.SSS0.Px1.p1.1.m1.1.1.2">𝑚</ci><ci id="A2.SS0.SSS0.Px1.p1.1.m1.1.1.3.cmml" xref="A2.SS0.SSS0.Px1.p1.1.m1.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS0.SSS0.Px1.p1.1.m1.1c">m=n</annotation></semantics></math> and <math id="A2.SS0.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="r=0.7n" display="inline"><semantics id="A2.SS0.SSS0.Px1.p1.2.m2.1a"><mrow id="A2.SS0.SSS0.Px1.p1.2.m2.1.1" xref="A2.SS0.SSS0.Px1.p1.2.m2.1.1.cmml"><mi id="A2.SS0.SSS0.Px1.p1.2.m2.1.1.2" xref="A2.SS0.SSS0.Px1.p1.2.m2.1.1.2.cmml">r</mi><mo id="A2.SS0.SSS0.Px1.p1.2.m2.1.1.1" xref="A2.SS0.SSS0.Px1.p1.2.m2.1.1.1.cmml">=</mo><mrow id="A2.SS0.SSS0.Px1.p1.2.m2.1.1.3" xref="A2.SS0.SSS0.Px1.p1.2.m2.1.1.3.cmml"><mn id="A2.SS0.SSS0.Px1.p1.2.m2.1.1.3.2" xref="A2.SS0.SSS0.Px1.p1.2.m2.1.1.3.2.cmml">0.7</mn><mo lspace="0em" rspace="0em" id="A2.SS0.SSS0.Px1.p1.2.m2.1.1.3.1" xref="A2.SS0.SSS0.Px1.p1.2.m2.1.1.3.1.cmml">​</mo><mi id="A2.SS0.SSS0.Px1.p1.2.m2.1.1.3.3" xref="A2.SS0.SSS0.Px1.p1.2.m2.1.1.3.3.cmml">n</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.SS0.SSS0.Px1.p1.2.m2.1b"><apply id="A2.SS0.SSS0.Px1.p1.2.m2.1.1.cmml" xref="A2.SS0.SSS0.Px1.p1.2.m2.1.1"><eq id="A2.SS0.SSS0.Px1.p1.2.m2.1.1.1.cmml" xref="A2.SS0.SSS0.Px1.p1.2.m2.1.1.1"></eq><ci id="A2.SS0.SSS0.Px1.p1.2.m2.1.1.2.cmml" xref="A2.SS0.SSS0.Px1.p1.2.m2.1.1.2">𝑟</ci><apply id="A2.SS0.SSS0.Px1.p1.2.m2.1.1.3.cmml" xref="A2.SS0.SSS0.Px1.p1.2.m2.1.1.3"><times id="A2.SS0.SSS0.Px1.p1.2.m2.1.1.3.1.cmml" xref="A2.SS0.SSS0.Px1.p1.2.m2.1.1.3.1"></times><cn type="float" id="A2.SS0.SSS0.Px1.p1.2.m2.1.1.3.2.cmml" xref="A2.SS0.SSS0.Px1.p1.2.m2.1.1.3.2">0.7</cn><ci id="A2.SS0.SSS0.Px1.p1.2.m2.1.1.3.3.cmml" xref="A2.SS0.SSS0.Px1.p1.2.m2.1.1.3.3">𝑛</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS0.SSS0.Px1.p1.2.m2.1c">r=0.7n</annotation></semantics></math>, resulting in the equation <math id="A2.SS0.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="\frac{mr+nr}{mn}=1.4" display="inline"><semantics id="A2.SS0.SSS0.Px1.p1.3.m3.1a"><mrow id="A2.SS0.SSS0.Px1.p1.3.m3.1.1" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.cmml"><mfrac id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.cmml"><mrow id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.cmml"><mrow id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.2" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.2.cmml"><mi id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.2.2" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.2.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.2.1" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.2.1.cmml">​</mo><mi id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.2.3" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.2.3.cmml">r</mi></mrow><mo id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.1" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.1.cmml">+</mo><mrow id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.3" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.3.cmml"><mi id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.3.2" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.3.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.3.1" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.3.1.cmml">​</mo><mi id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.3.3" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.3.3.cmml">r</mi></mrow></mrow><mrow id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.3" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.3.cmml"><mi id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.3.2" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.3.1" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.3.1.cmml">​</mo><mi id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.3.3" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.3.3.cmml">n</mi></mrow></mfrac><mo id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.1" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.1.cmml">=</mo><mn id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.3" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.3.cmml">1.4</mn></mrow><annotation-xml encoding="MathML-Content" id="A2.SS0.SSS0.Px1.p1.3.m3.1b"><apply id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.cmml" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1"><eq id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.1.cmml" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.1"></eq><apply id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.cmml" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2"><divide id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.1.cmml" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2"></divide><apply id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.cmml" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2"><plus id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.1.cmml" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.1"></plus><apply id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.2.cmml" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.2"><times id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.2.1.cmml" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.2.1"></times><ci id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.2.2.cmml" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.2.2">𝑚</ci><ci id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.2.3.cmml" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.2.3">𝑟</ci></apply><apply id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.3.cmml" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.3"><times id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.3.1.cmml" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.3.1"></times><ci id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.3.2.cmml" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.3.2">𝑛</ci><ci id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.3.3.cmml" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.3.3">𝑟</ci></apply></apply><apply id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.3.cmml" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.3"><times id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.3.1.cmml" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.3.1"></times><ci id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.3.2.cmml" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.3.2">𝑚</ci><ci id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.3.3.cmml" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.2.3.3">𝑛</ci></apply></apply><cn type="float" id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.3.cmml" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.3">1.4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS0.SSS0.Px1.p1.3.m3.1c">\frac{mr+nr}{mn}=1.4</annotation></semantics></math>. Such an increase indicates substantial extra memory usage.</p>
</div>
</section>
<section id="A2.SS0.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph"></h5>

<div id="A2.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="A2.SS0.SSS0.Px2.p1.4" class="ltx_p">This issue primarily arises in J-LRD decomposition schemes, where the projections of all heads are decomposed jointly. In contrast, our M-LRD decomposition schemes and optimized G-LRD schemes involve non-square target matrices. For example, in the G-LRD scheme with a group size of 4, the target matrix is formed by concatenating the original projection matrices of each attention head in the group. In the Llama-2-7b model, with an embedding dimension of 4096 and head dimensions of 128, each projection matrix is 4096x128, resulting in a concatenated matrix of size 4096x512. In this case, the dimensions should be considered as <math id="A2.SS0.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="m=8n" display="inline"><semantics id="A2.SS0.SSS0.Px2.p1.1.m1.1a"><mrow id="A2.SS0.SSS0.Px2.p1.1.m1.1.1" xref="A2.SS0.SSS0.Px2.p1.1.m1.1.1.cmml"><mi id="A2.SS0.SSS0.Px2.p1.1.m1.1.1.2" xref="A2.SS0.SSS0.Px2.p1.1.m1.1.1.2.cmml">m</mi><mo id="A2.SS0.SSS0.Px2.p1.1.m1.1.1.1" xref="A2.SS0.SSS0.Px2.p1.1.m1.1.1.1.cmml">=</mo><mrow id="A2.SS0.SSS0.Px2.p1.1.m1.1.1.3" xref="A2.SS0.SSS0.Px2.p1.1.m1.1.1.3.cmml"><mn id="A2.SS0.SSS0.Px2.p1.1.m1.1.1.3.2" xref="A2.SS0.SSS0.Px2.p1.1.m1.1.1.3.2.cmml">8</mn><mo lspace="0em" rspace="0em" id="A2.SS0.SSS0.Px2.p1.1.m1.1.1.3.1" xref="A2.SS0.SSS0.Px2.p1.1.m1.1.1.3.1.cmml">​</mo><mi id="A2.SS0.SSS0.Px2.p1.1.m1.1.1.3.3" xref="A2.SS0.SSS0.Px2.p1.1.m1.1.1.3.3.cmml">n</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.SS0.SSS0.Px2.p1.1.m1.1b"><apply id="A2.SS0.SSS0.Px2.p1.1.m1.1.1.cmml" xref="A2.SS0.SSS0.Px2.p1.1.m1.1.1"><eq id="A2.SS0.SSS0.Px2.p1.1.m1.1.1.1.cmml" xref="A2.SS0.SSS0.Px2.p1.1.m1.1.1.1"></eq><ci id="A2.SS0.SSS0.Px2.p1.1.m1.1.1.2.cmml" xref="A2.SS0.SSS0.Px2.p1.1.m1.1.1.2">𝑚</ci><apply id="A2.SS0.SSS0.Px2.p1.1.m1.1.1.3.cmml" xref="A2.SS0.SSS0.Px2.p1.1.m1.1.1.3"><times id="A2.SS0.SSS0.Px2.p1.1.m1.1.1.3.1.cmml" xref="A2.SS0.SSS0.Px2.p1.1.m1.1.1.3.1"></times><cn type="integer" id="A2.SS0.SSS0.Px2.p1.1.m1.1.1.3.2.cmml" xref="A2.SS0.SSS0.Px2.p1.1.m1.1.1.3.2">8</cn><ci id="A2.SS0.SSS0.Px2.p1.1.m1.1.1.3.3.cmml" xref="A2.SS0.SSS0.Px2.p1.1.m1.1.1.3.3">𝑛</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS0.SSS0.Px2.p1.1.m1.1c">m=8n</annotation></semantics></math>. Applying the referenced equation <math id="A2.SS0.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="\frac{mr+nr}{mn}" display="inline"><semantics id="A2.SS0.SSS0.Px2.p1.2.m2.1a"><mfrac id="A2.SS0.SSS0.Px2.p1.2.m2.1.1" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.cmml"><mrow id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.cmml"><mrow id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.2" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.2.cmml"><mi id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.2.2" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.2.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.2.1" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.2.1.cmml">​</mo><mi id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.2.3" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.2.3.cmml">r</mi></mrow><mo id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.1" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.1.cmml">+</mo><mrow id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.3" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.3.cmml"><mi id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.3.2" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.3.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.3.1" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.3.1.cmml">​</mo><mi id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.3.3" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.3.3.cmml">r</mi></mrow></mrow><mrow id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3.cmml"><mi id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3.2" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3.1" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3.1.cmml">​</mo><mi id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.cmml">n</mi></mrow></mfrac><annotation-xml encoding="MathML-Content" id="A2.SS0.SSS0.Px2.p1.2.m2.1b"><apply id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.cmml" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1"><divide id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.1.cmml" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1"></divide><apply id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.cmml" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2"><plus id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.1.cmml" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.1"></plus><apply id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.2.cmml" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.2"><times id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.2.1.cmml" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.2.1"></times><ci id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.2.2.cmml" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.2.2">𝑚</ci><ci id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.2.3.cmml" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.2.3">𝑟</ci></apply><apply id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.3.cmml" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.3"><times id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.3.1.cmml" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.3.1"></times><ci id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.3.2.cmml" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.3.2">𝑛</ci><ci id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.3.3.cmml" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.3.3">𝑟</ci></apply></apply><apply id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3.cmml" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3"><times id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3.1.cmml" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3.1"></times><ci id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3.2.cmml" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3.2">𝑚</ci><ci id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.cmml" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3">𝑛</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS0.SSS0.Px2.p1.2.m2.1c">\frac{mr+nr}{mn}</annotation></semantics></math> with <math id="A2.SS0.SSS0.Px2.p1.3.m3.1" class="ltx_Math" alttext="r=0.7n" display="inline"><semantics id="A2.SS0.SSS0.Px2.p1.3.m3.1a"><mrow id="A2.SS0.SSS0.Px2.p1.3.m3.1.1" xref="A2.SS0.SSS0.Px2.p1.3.m3.1.1.cmml"><mi id="A2.SS0.SSS0.Px2.p1.3.m3.1.1.2" xref="A2.SS0.SSS0.Px2.p1.3.m3.1.1.2.cmml">r</mi><mo id="A2.SS0.SSS0.Px2.p1.3.m3.1.1.1" xref="A2.SS0.SSS0.Px2.p1.3.m3.1.1.1.cmml">=</mo><mrow id="A2.SS0.SSS0.Px2.p1.3.m3.1.1.3" xref="A2.SS0.SSS0.Px2.p1.3.m3.1.1.3.cmml"><mn id="A2.SS0.SSS0.Px2.p1.3.m3.1.1.3.2" xref="A2.SS0.SSS0.Px2.p1.3.m3.1.1.3.2.cmml">0.7</mn><mo lspace="0em" rspace="0em" id="A2.SS0.SSS0.Px2.p1.3.m3.1.1.3.1" xref="A2.SS0.SSS0.Px2.p1.3.m3.1.1.3.1.cmml">​</mo><mi id="A2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3" xref="A2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.cmml">n</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.SS0.SSS0.Px2.p1.3.m3.1b"><apply id="A2.SS0.SSS0.Px2.p1.3.m3.1.1.cmml" xref="A2.SS0.SSS0.Px2.p1.3.m3.1.1"><eq id="A2.SS0.SSS0.Px2.p1.3.m3.1.1.1.cmml" xref="A2.SS0.SSS0.Px2.p1.3.m3.1.1.1"></eq><ci id="A2.SS0.SSS0.Px2.p1.3.m3.1.1.2.cmml" xref="A2.SS0.SSS0.Px2.p1.3.m3.1.1.2">𝑟</ci><apply id="A2.SS0.SSS0.Px2.p1.3.m3.1.1.3.cmml" xref="A2.SS0.SSS0.Px2.p1.3.m3.1.1.3"><times id="A2.SS0.SSS0.Px2.p1.3.m3.1.1.3.1.cmml" xref="A2.SS0.SSS0.Px2.p1.3.m3.1.1.3.1"></times><cn type="float" id="A2.SS0.SSS0.Px2.p1.3.m3.1.1.3.2.cmml" xref="A2.SS0.SSS0.Px2.p1.3.m3.1.1.3.2">0.7</cn><ci id="A2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.cmml" xref="A2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3">𝑛</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS0.SSS0.Px2.p1.3.m3.1c">r=0.7n</annotation></semantics></math>, we find that <math id="A2.SS0.SSS0.Px2.p1.4.m4.1" class="ltx_Math" alttext="\frac{mr+nr}{mn}=0.7875" display="inline"><semantics id="A2.SS0.SSS0.Px2.p1.4.m4.1a"><mrow id="A2.SS0.SSS0.Px2.p1.4.m4.1.1" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.cmml"><mfrac id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.cmml"><mrow id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.cmml"><mrow id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.2" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.2.cmml"><mi id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.2.2" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.2.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.2.1" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.2.1.cmml">​</mo><mi id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.2.3" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.2.3.cmml">r</mi></mrow><mo id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.1" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.1.cmml">+</mo><mrow id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.3" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.3.cmml"><mi id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.3.2" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.3.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.3.1" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.3.1.cmml">​</mo><mi id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.3.3" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.3.3.cmml">r</mi></mrow></mrow><mrow id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.3" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.3.cmml"><mi id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.3.2" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.3.1" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.3.1.cmml">​</mo><mi id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.3.3" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.3.3.cmml">n</mi></mrow></mfrac><mo id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.1" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.1.cmml">=</mo><mn id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.3" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.3.cmml">0.7875</mn></mrow><annotation-xml encoding="MathML-Content" id="A2.SS0.SSS0.Px2.p1.4.m4.1b"><apply id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1"><eq id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.1.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.1"></eq><apply id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2"><divide id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.1.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2"></divide><apply id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2"><plus id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.1.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.1"></plus><apply id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.2.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.2"><times id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.2.1.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.2.1"></times><ci id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.2.2.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.2.2">𝑚</ci><ci id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.2.3.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.2.3">𝑟</ci></apply><apply id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.3.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.3"><times id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.3.1.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.3.1"></times><ci id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.3.2.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.3.2">𝑛</ci><ci id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.3.3.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.3.3">𝑟</ci></apply></apply><apply id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.3.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.3"><times id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.3.1.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.3.1"></times><ci id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.3.2.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.3.2">𝑚</ci><ci id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.3.3.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.3.3">𝑛</ci></apply></apply><cn type="float" id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.3.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.3">0.7875</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS0.SSS0.Px2.p1.4.m4.1c">\frac{mr+nr}{mn}=0.7875</annotation></semantics></math>, indicating no additional storage cost and, in fact, achieving an additional 21.25% memory savings.</p>
</div>
</section>
<section id="A2.SS0.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph"></h5>

<div id="A2.SS0.SSS0.Px3.p1" class="ltx_para">
<p id="A2.SS0.SSS0.Px3.p1.1" class="ltx_p">Furthermore, it is important to highlight that the weights associated with the K and V projections account for only 2 out of 7 linear layers within transformer blocks, comprising merely 16% of the parameters in Llama-2-7b models. This limits the overall impact on memory usage. Thus, while J-LRD may incur overhead, the M-LRD and G-LRD schemes provide efficient alternatives that do not lead to increased memory usage, making them viable options for practical applications.</p>
</div>
</section>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Overall Memory Reduction</h2>

<div id="A3.p1" class="ltx_para">
<p id="A3.p1.2" class="ltx_p">In Fig. <a href="#A3.F6" title="Figure 6 ‣ Appendix C Overall Memory Reduction ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>, we show the total memory usage, including model weights and KV-Cache under different sequence lengths. We observe that KV-Cache quickly becomes the memory bottleneck as sequence length grows.
At a sequence length of 64k, the KV-Cache constitutes 78% of the total memory consumption.
With 50% low-rank compression, <span id="A3.p1.2.1" class="ltx_text ltx_font_italic">Palu</span> effectively reduces memory usage by 1.7<math id="A3.p1.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="A3.p1.1.m1.1a"><mo id="A3.p1.1.m1.1.1" xref="A3.p1.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="A3.p1.1.m1.1b"><times id="A3.p1.1.m1.1.1.cmml" xref="A3.p1.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="A3.p1.1.m1.1c">\times</annotation></semantics></math>, and when combined with 2-bit quantization, it further decreases total memory usage by 4.6<math id="A3.p1.2.m2.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="A3.p1.2.m2.1a"><mo id="A3.p1.2.m2.1.1" xref="A3.p1.2.m2.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="A3.p1.2.m2.1b"><times id="A3.p1.2.m2.1.1.cmml" xref="A3.p1.2.m2.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="A3.p1.2.m2.1c">\times</annotation></semantics></math>.</p>
</div>
<figure id="A3.F6" class="ltx_figure">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 6: </span>
Total memory usage for various sequence lengths in Llama2-7B. low-rank compression is at 50% for <span id="A3.F6.2.1" class="ltx_text ltx_font_italic">Palu</span>.
</figcaption><img src="/html/2407.21118/assets/x6.png" id="A3.F6.g1" class="ltx_graphics ltx_centering ltx_img_square" width="161" height="133" alt="Refer to caption">
</figure>
</section>
<section id="A4" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>End-to-End Latency Measurements</h2>

<div id="A4.p1" class="ltx_para">
<p id="A4.p1.1" class="ltx_p">We provide the latency evaluation of integrating our optimized attention modules into the Llama-2 models in Table <a href="#A4.T4" title="Table 4 ‣ Appendix D End-to-End Latency Measurements ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>. Measurement times are expressed in milliseconds (ms). Our measurements indicated that <span id="A4.p1.1.1" class="ltx_text ltx_font_italic">Palu</span> achieved significant end-to-end speedup during the decoding phase, especially with longer sequence lengths. Specifically, for a sequence length of 32K, the speedups were 1.40x, 1.55x, and 1.71x for gs=4, gs=2, and gs=1, respectively. The results demonstrate the effectiveness of our methods and the potential to explore the redundancy in the hidden dimension.</p>
</div>
<figure id="A4.T4" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 4: </span>End-to-end decoding latency of Llama-2-7B with <span id="A4.T4.2.1" class="ltx_text ltx_font_italic">Palu</span> measured on A6000.</figcaption>
<div id="A4.T4.3" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:200.0pt;height:102.7pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-22.7pt,11.6pt) scale(0.815170993659162,0.815170993659162) ;">
<table id="A4.T4.3.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A4.T4.3.1.1.1" class="ltx_tr">
<th id="A4.T4.3.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_tt" rowspan="2"><span id="A4.T4.3.1.1.1.1.1" class="ltx_text">Method</span></th>
<th id="A4.T4.3.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_tt">Group</th>
<th id="A4.T4.3.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="4">Sequence Length</th>
</tr>
<tr id="A4.T4.3.1.2.2" class="ltx_tr">
<th id="A4.T4.3.1.2.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_r">Size</th>
<th id="A4.T4.3.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column">4k</th>
<th id="A4.T4.3.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column">8k</th>
<th id="A4.T4.3.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column">16k</th>
<th id="A4.T4.3.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column">32k</th>
</tr>
<tr id="A4.T4.3.1.3.3" class="ltx_tr">
<th id="A4.T4.3.1.3.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_t">Baseline</th>
<th id="A4.T4.3.1.3.3.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_t">-</th>
<th id="A4.T4.3.1.3.3.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">36.07</th>
<th id="A4.T4.3.1.3.3.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">44.03</th>
<th id="A4.T4.3.1.3.3.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">69.28</th>
<th id="A4.T4.3.1.3.3.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">112.75</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A4.T4.3.1.4.1" class="ltx_tr">
<th id="A4.T4.3.1.4.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">J-LRD</th>
<th id="A4.T4.3.1.4.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">32</th>
<td id="A4.T4.3.1.4.1.3" class="ltx_td ltx_align_center ltx_border_t">86.39</td>
<td id="A4.T4.3.1.4.1.4" class="ltx_td ltx_align_center ltx_border_t">103.41</td>
<td id="A4.T4.3.1.4.1.5" class="ltx_td ltx_align_center ltx_border_t">140.85</td>
<td id="A4.T4.3.1.4.1.6" class="ltx_td ltx_align_center ltx_border_t">OOM</td>
</tr>
<tr id="A4.T4.3.1.5.2" class="ltx_tr">
<th id="A4.T4.3.1.5.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">G-LRD</th>
<th id="A4.T4.3.1.5.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">4</th>
<td id="A4.T4.3.1.5.2.3" class="ltx_td ltx_align_center">32.12</td>
<td id="A4.T4.3.1.5.2.4" class="ltx_td ltx_align_center">38.89</td>
<td id="A4.T4.3.1.5.2.5" class="ltx_td ltx_align_center">52.69</td>
<td id="A4.T4.3.1.5.2.6" class="ltx_td ltx_align_center">80.81</td>
</tr>
<tr id="A4.T4.3.1.6.3" class="ltx_tr">
<th id="A4.T4.3.1.6.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">G-LRD</th>
<th id="A4.T4.3.1.6.3.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">2</th>
<td id="A4.T4.3.1.6.3.3" class="ltx_td ltx_align_center">28.82</td>
<td id="A4.T4.3.1.6.3.4" class="ltx_td ltx_align_center">34.94</td>
<td id="A4.T4.3.1.6.3.5" class="ltx_td ltx_align_center">47.45</td>
<td id="A4.T4.3.1.6.3.6" class="ltx_td ltx_align_center">72.89</td>
</tr>
<tr id="A4.T4.3.1.7.4" class="ltx_tr">
<th id="A4.T4.3.1.7.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_r">M-LRD</th>
<th id="A4.T4.3.1.7.4.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb ltx_border_r">1</th>
<td id="A4.T4.3.1.7.4.3" class="ltx_td ltx_align_center ltx_border_bb">27.00</td>
<td id="A4.T4.3.1.7.4.4" class="ltx_td ltx_align_center ltx_border_bb">32.46</td>
<td id="A4.T4.3.1.7.4.5" class="ltx_td ltx_align_center ltx_border_bb">43.56</td>
<td id="A4.T4.3.1.7.4.6" class="ltx_td ltx_align_center ltx_border_bb">66.13</td>
</tr>
</tbody>
</table>
</span></div>
</figure>
</section>
<section id="A5" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix E </span>Extended Experiments on LongBench and comparisons to quantization baselines</h2>

<section id="A5.SS0.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph"></h5>

<div id="A5.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="A5.SS0.SSS0.Px1.p1.1" class="ltx_p">We conducted extensive experiments using the Mistral-Instruct-7B-v0.2 <cite class="ltx_cite ltx_citemacro_citep">(Jiang et al., <a href="#bib.bib19" title="" class="ltx_ref">2023</a>)</cite> and LongChat-7b-v1.5-32k <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib22" title="" class="ltx_ref">2023</a>)</cite> models, both with extended context lengths of 32k tokens. We included 16 available English datasets from LongBench to ensure a comprehensive evaluation. We report the average score for each type of task separately, as well as the overall average across all 16 tasks. This evaluation aimed to broadly assess the effectiveness of <span id="A5.SS0.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_italic">Palu</span> in long-context scenarios. For reference, we also include comparisons with KIVI.</p>
</div>
</section>
<section id="A5.SS0.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph"></h5>

<div id="A5.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="A5.SS0.SSS0.Px2.p1.1" class="ltx_p">Our findings indicate that at a 30% compression level, <span id="A5.SS0.SSS0.Px2.p1.1.1" class="ltx_text ltx_font_italic">Palu</span> achieves only a minor average accuracy drop (&lt;1%) compared to the baseline for both models. These impressive results are attributed to our effective use of low-rank projection methods to explore redundancy in hidden dimensions. Furthermore, by fusing the Hadamard matrix offline to improve quantization friendliness, we were able to quantize the low-rank projected KV-Cache down to 3 bits. This quantization process resulted in less than a 1% accuracy degradation, achieved with a straightforward per-token quantization approach. When compared to KIVI, our framework demonstrates the ability to harness an additional 30% of redundancy. Notably, <span id="A5.SS0.SSS0.Px2.p1.1.2" class="ltx_text ltx_font_italic">Palu</span> does not require the complex grouped quantization and mixed-precision techniques employed by KIVI, highlighting the quantization friendliness that <span id="A5.SS0.SSS0.Px2.p1.1.3" class="ltx_text ltx_font_italic">Palu</span> offers.</p>
</div>
<figure id="A5.T5" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 5: </span>Experiment results on Longbench. Average bit widths are estimated for each approach, assuming 10K context length. Each column represents the average score for the tasks of each type. Multi-QA and Single-QA denote Multi-Document and Single-Document Question Answering, respectively, while Summ refers to Summarization.</figcaption>
<div id="A5.T5.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:400.0pt;height:181.9pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-77.1pt,35.1pt) scale(0.721659596508926,0.721659596508926) ;">
<table id="A5.T5.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A5.T5.1.1.1.1" class="ltx_tr">
<th id="A5.T5.1.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_tt" rowspan="2"><span id="A5.T5.1.1.1.1.1.1" class="ltx_text">Model</span></th>
<th id="A5.T5.1.1.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_tt" rowspan="2"><span id="A5.T5.1.1.1.1.2.1" class="ltx_text">Method</span></th>
<th id="A5.T5.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_tt">Avg.</th>
<td id="A5.T5.1.1.1.1.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt">Comp.</td>
<td id="A5.T5.1.1.1.1.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt">Multi-</td>
<td id="A5.T5.1.1.1.1.6" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt">Single-</td>
<td id="A5.T5.1.1.1.1.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" rowspan="2"><span id="A5.T5.1.1.1.1.7.1" class="ltx_text">Summ</span></td>
<td id="A5.T5.1.1.1.1.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" rowspan="2"><span id="A5.T5.1.1.1.1.8.1" class="ltx_text">Few-Shot</span></td>
<td id="A5.T5.1.1.1.1.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" rowspan="2"><span id="A5.T5.1.1.1.1.9.1" class="ltx_text">Code</span></td>
<td id="A5.T5.1.1.1.1.10" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" rowspan="2"><span id="A5.T5.1.1.1.1.10.1" class="ltx_text">Synthetic</span></td>
<td id="A5.T5.1.1.1.1.11" class="ltx_td ltx_align_center ltx_border_tt" rowspan="2"><span id="A5.T5.1.1.1.1.11.1" class="ltx_text">Avg.</span></td>
</tr>
<tr id="A5.T5.1.1.2.2" class="ltx_tr">
<th id="A5.T5.1.1.2.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Bits</th>
<td id="A5.T5.1.1.2.2.2" class="ltx_td ltx_align_center ltx_border_r">Ratio</td>
<td id="A5.T5.1.1.2.2.3" class="ltx_td ltx_align_center ltx_border_r">QA</td>
<td id="A5.T5.1.1.2.2.4" class="ltx_td ltx_align_center ltx_border_r">QA</td>
</tr>
<tr id="A5.T5.1.1.3.3" class="ltx_tr">
<th id="A5.T5.1.1.3.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t" rowspan="6"><span id="A5.T5.1.1.3.3.1.1" class="ltx_text">Mistral-7B-v0.2</span></th>
<th id="A5.T5.1.1.3.3.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">Baseline</th>
<th id="A5.T5.1.1.3.3.3" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">16</th>
<td id="A5.T5.1.1.3.3.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">1.00x</td>
<td id="A5.T5.1.1.3.3.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">29.63</td>
<td id="A5.T5.1.1.3.3.6" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">36.43</td>
<td id="A5.T5.1.1.3.3.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">28.10</td>
<td id="A5.T5.1.1.3.3.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">66.71</td>
<td id="A5.T5.1.1.3.3.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">54.16</td>
<td id="A5.T5.1.1.3.3.10" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">44.87</td>
<td id="A5.T5.1.1.3.3.11" class="ltx_td ltx_align_center ltx_border_t">42.54</td>
</tr>
<tr id="A5.T5.1.1.4.4" class="ltx_tr">
<th id="A5.T5.1.1.4.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">Palu-30%</th>
<th id="A5.T5.1.1.4.4.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">16</th>
<td id="A5.T5.1.1.4.4.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">1.43x</td>
<td id="A5.T5.1.1.4.4.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">29.83</td>
<td id="A5.T5.1.1.4.4.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">36.52</td>
<td id="A5.T5.1.1.4.4.6" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">27.48</td>
<td id="A5.T5.1.1.4.4.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">65.70</td>
<td id="A5.T5.1.1.4.4.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">55.16</td>
<td id="A5.T5.1.1.4.4.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">37.92</td>
<td id="A5.T5.1.1.4.4.10" class="ltx_td ltx_align_center ltx_border_t">41.55</td>
</tr>
<tr id="A5.T5.1.1.5.5" class="ltx_tr">
<th id="A5.T5.1.1.5.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">Palu-50%</th>
<th id="A5.T5.1.1.5.5.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">16</th>
<td id="A5.T5.1.1.5.5.3" class="ltx_td ltx_align_center ltx_border_r">2.00x</td>
<td id="A5.T5.1.1.5.5.4" class="ltx_td ltx_align_center ltx_border_r">26.92</td>
<td id="A5.T5.1.1.5.5.5" class="ltx_td ltx_align_center ltx_border_r">35.33</td>
<td id="A5.T5.1.1.5.5.6" class="ltx_td ltx_align_center ltx_border_r">26.01</td>
<td id="A5.T5.1.1.5.5.7" class="ltx_td ltx_align_center ltx_border_r">64.04</td>
<td id="A5.T5.1.1.5.5.8" class="ltx_td ltx_align_center ltx_border_r">44.54</td>
<td id="A5.T5.1.1.5.5.9" class="ltx_td ltx_align_center ltx_border_r">16.88</td>
<td id="A5.T5.1.1.5.5.10" class="ltx_td ltx_align_center">36.23</td>
</tr>
<tr id="A5.T5.1.1.6.6" class="ltx_tr">
<th id="A5.T5.1.1.6.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">KIVI-2</th>
<th id="A5.T5.1.1.6.6.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">3.16</th>
<td id="A5.T5.1.1.6.6.3" class="ltx_td ltx_align_center ltx_border_r">5.05x</td>
<td id="A5.T5.1.1.6.6.4" class="ltx_td ltx_align_center ltx_border_r">28.81</td>
<td id="A5.T5.1.1.6.6.5" class="ltx_td ltx_align_center ltx_border_r">35.07</td>
<td id="A5.T5.1.1.6.6.6" class="ltx_td ltx_align_center ltx_border_r">27.60</td>
<td id="A5.T5.1.1.6.6.7" class="ltx_td ltx_align_center ltx_border_r">66.45</td>
<td id="A5.T5.1.1.6.6.8" class="ltx_td ltx_align_center ltx_border_r">54.47</td>
<td id="A5.T5.1.1.6.6.9" class="ltx_td ltx_align_center ltx_border_r">40.28</td>
<td id="A5.T5.1.1.6.6.10" class="ltx_td ltx_align_center">41.45</td>
</tr>
<tr id="A5.T5.1.1.7.7" class="ltx_tr">
<th id="A5.T5.1.1.7.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">Palu-30% (3 bits)</th>
<th id="A5.T5.1.1.7.7.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">3</th>
<td id="A5.T5.1.1.7.7.3" class="ltx_td ltx_align_center ltx_border_r">7.59x</td>
<td id="A5.T5.1.1.7.7.4" class="ltx_td ltx_align_center ltx_border_r">29.48</td>
<td id="A5.T5.1.1.7.7.5" class="ltx_td ltx_align_center ltx_border_r">36.40</td>
<td id="A5.T5.1.1.7.7.6" class="ltx_td ltx_align_center ltx_border_r">27.20</td>
<td id="A5.T5.1.1.7.7.7" class="ltx_td ltx_align_center ltx_border_r">65.73</td>
<td id="A5.T5.1.1.7.7.8" class="ltx_td ltx_align_center ltx_border_r">53.19</td>
<td id="A5.T5.1.1.7.7.9" class="ltx_td ltx_align_center ltx_border_r">34.74</td>
<td id="A5.T5.1.1.7.7.10" class="ltx_td ltx_align_center">40.77</td>
</tr>
<tr id="A5.T5.1.1.8.8" class="ltx_tr">
<th id="A5.T5.1.1.8.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">Palu-50% (3 bits)</th>
<th id="A5.T5.1.1.8.8.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">3</th>
<td id="A5.T5.1.1.8.8.3" class="ltx_td ltx_align_center ltx_border_r">10.6x</td>
<td id="A5.T5.1.1.8.8.4" class="ltx_td ltx_align_center ltx_border_r">26.73</td>
<td id="A5.T5.1.1.8.8.5" class="ltx_td ltx_align_center ltx_border_r">32.72</td>
<td id="A5.T5.1.1.8.8.6" class="ltx_td ltx_align_center ltx_border_r">25.73</td>
<td id="A5.T5.1.1.8.8.7" class="ltx_td ltx_align_center ltx_border_r">63.25</td>
<td id="A5.T5.1.1.8.8.8" class="ltx_td ltx_align_center ltx_border_r">44.43</td>
<td id="A5.T5.1.1.8.8.9" class="ltx_td ltx_align_center ltx_border_r">18.57</td>
<td id="A5.T5.1.1.8.8.10" class="ltx_td ltx_align_center">35.71</td>
</tr>
<tr id="A5.T5.1.1.9.9" class="ltx_tr">
<th id="A5.T5.1.1.9.9.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb ltx_border_r ltx_border_t" rowspan="6"><span id="A5.T5.1.1.9.9.1.1" class="ltx_text">LongChat-7B-v1.5</span></th>
<th id="A5.T5.1.1.9.9.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">Baseline</th>
<th id="A5.T5.1.1.9.9.3" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">16</th>
<td id="A5.T5.1.1.9.9.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">1.00x</td>
<td id="A5.T5.1.1.9.9.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">23.95</td>
<td id="A5.T5.1.1.9.9.6" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">31.12</td>
<td id="A5.T5.1.1.9.9.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">26.74</td>
<td id="A5.T5.1.1.9.9.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">63.80</td>
<td id="A5.T5.1.1.9.9.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">56.91</td>
<td id="A5.T5.1.1.9.9.10" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">15.25</td>
<td id="A5.T5.1.1.9.9.11" class="ltx_td ltx_align_center ltx_border_t">36.32</td>
</tr>
<tr id="A5.T5.1.1.10.10" class="ltx_tr">
<th id="A5.T5.1.1.10.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">Palu-30%</th>
<th id="A5.T5.1.1.10.10.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">16</th>
<td id="A5.T5.1.1.10.10.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">1.43x</td>
<td id="A5.T5.1.1.10.10.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">22.42</td>
<td id="A5.T5.1.1.10.10.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">29.43</td>
<td id="A5.T5.1.1.10.10.6" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">25.52</td>
<td id="A5.T5.1.1.10.10.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">62.87</td>
<td id="A5.T5.1.1.10.10.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">58.99</td>
<td id="A5.T5.1.1.10.10.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">14.25</td>
<td id="A5.T5.1.1.10.10.10" class="ltx_td ltx_align_center ltx_border_t">35.45</td>
</tr>
<tr id="A5.T5.1.1.11.11" class="ltx_tr">
<th id="A5.T5.1.1.11.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">Palu-50%</th>
<th id="A5.T5.1.1.11.11.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">16</th>
<td id="A5.T5.1.1.11.11.3" class="ltx_td ltx_align_center ltx_border_r">2.00x</td>
<td id="A5.T5.1.1.11.11.4" class="ltx_td ltx_align_center ltx_border_r">22.61</td>
<td id="A5.T5.1.1.11.11.5" class="ltx_td ltx_align_center ltx_border_r">25.33</td>
<td id="A5.T5.1.1.11.11.6" class="ltx_td ltx_align_center ltx_border_r">22.73</td>
<td id="A5.T5.1.1.11.11.7" class="ltx_td ltx_align_center ltx_border_r">60.12</td>
<td id="A5.T5.1.1.11.11.8" class="ltx_td ltx_align_center ltx_border_r">43.52</td>
<td id="A5.T5.1.1.11.11.9" class="ltx_td ltx_align_center ltx_border_r">6.84</td>
<td id="A5.T5.1.1.11.11.10" class="ltx_td ltx_align_center">30.82</td>
</tr>
<tr id="A5.T5.1.1.12.12" class="ltx_tr">
<th id="A5.T5.1.1.12.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">KIVI-2</th>
<th id="A5.T5.1.1.12.12.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">3.16</th>
<td id="A5.T5.1.1.12.12.3" class="ltx_td ltx_align_center ltx_border_r">5.06x</td>
<td id="A5.T5.1.1.12.12.4" class="ltx_td ltx_align_center ltx_border_r">23.24</td>
<td id="A5.T5.1.1.12.12.5" class="ltx_td ltx_align_center ltx_border_r">30.19</td>
<td id="A5.T5.1.1.12.12.6" class="ltx_td ltx_align_center ltx_border_r">26.47</td>
<td id="A5.T5.1.1.12.12.7" class="ltx_td ltx_align_center ltx_border_r">63.54</td>
<td id="A5.T5.1.1.12.12.8" class="ltx_td ltx_align_center ltx_border_r">53.51</td>
<td id="A5.T5.1.1.12.12.9" class="ltx_td ltx_align_center ltx_border_r">16.13</td>
<td id="A5.T5.1.1.12.12.10" class="ltx_td ltx_align_center">35.60</td>
</tr>
<tr id="A5.T5.1.1.13.13" class="ltx_tr">
<th id="A5.T5.1.1.13.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">Palu-30% (3 bits)</th>
<th id="A5.T5.1.1.13.13.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">3</th>
<td id="A5.T5.1.1.13.13.3" class="ltx_td ltx_align_center ltx_border_r">7.59x</td>
<td id="A5.T5.1.1.13.13.4" class="ltx_td ltx_align_center ltx_border_r">23.12</td>
<td id="A5.T5.1.1.13.13.5" class="ltx_td ltx_align_center ltx_border_r">29.21</td>
<td id="A5.T5.1.1.13.13.6" class="ltx_td ltx_align_center ltx_border_r">25.04</td>
<td id="A5.T5.1.1.13.13.7" class="ltx_td ltx_align_center ltx_border_r">61.99</td>
<td id="A5.T5.1.1.13.13.8" class="ltx_td ltx_align_center ltx_border_r">54.38</td>
<td id="A5.T5.1.1.13.13.9" class="ltx_td ltx_align_center ltx_border_r">11.25</td>
<td id="A5.T5.1.1.13.13.10" class="ltx_td ltx_align_center">34.33</td>
</tr>
<tr id="A5.T5.1.1.14.14" class="ltx_tr">
<th id="A5.T5.1.1.14.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_r">Palu-50% (3 bits)</th>
<th id="A5.T5.1.1.14.14.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb ltx_border_r">3</th>
<td id="A5.T5.1.1.14.14.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r">10.6x</td>
<td id="A5.T5.1.1.14.14.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r">18.56</td>
<td id="A5.T5.1.1.14.14.5" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r">24.14</td>
<td id="A5.T5.1.1.14.14.6" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r">22.35</td>
<td id="A5.T5.1.1.14.14.7" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r">58.76</td>
<td id="A5.T5.1.1.14.14.8" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r">40.50</td>
<td id="A5.T5.1.1.14.14.9" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r">6.02</td>
<td id="A5.T5.1.1.14.14.10" class="ltx_td ltx_align_center ltx_border_bb">29.03</td>
</tr>
</tbody>
</table>
</span></div>
</figure>
<figure id="A5.T6" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 6: </span>Evaluation integrating LoRA with Palu on Llama2-7B.</figcaption>
<div id="A5.T6.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:380.0pt;height:127.8pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-50.8pt,17.1pt) scale(0.789086690887061,0.789086690887061) ;">
<table id="A5.T6.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A5.T6.1.1.1" class="ltx_tr">
<th id="A5.T6.1.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_tt" rowspan="2"><span id="A5.T6.1.1.1.2.1" class="ltx_text">Comp. Rate</span></th>
<th id="A5.T6.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_tt" rowspan="2"><span id="A5.T6.1.1.1.3.1" class="ltx_text">Method</span></th>
<th id="A5.T6.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="8">Zero-Shot Accuracy (%) <math id="A5.T6.1.1.1.1.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="A5.T6.1.1.1.1.m1.1a"><mo stretchy="false" id="A5.T6.1.1.1.1.m1.1.1" xref="A5.T6.1.1.1.1.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="A5.T6.1.1.1.1.m1.1b"><ci id="A5.T6.1.1.1.1.m1.1.1.cmml" xref="A5.T6.1.1.1.1.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.T6.1.1.1.1.m1.1c">\uparrow</annotation></semantics></math>
</th>
</tr>
<tr id="A5.T6.1.1.2.1" class="ltx_tr">
<th id="A5.T6.1.1.2.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column">OBQA</th>
<th id="A5.T6.1.1.2.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column">HellaSwag</th>
<th id="A5.T6.1.1.2.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column">PIQA</th>
<th id="A5.T6.1.1.2.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column">ARC-e</th>
<th id="A5.T6.1.1.2.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column">ARC-c</th>
<th id="A5.T6.1.1.2.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column">WinoGrande</th>
<th id="A5.T6.1.1.2.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column">Avg.</th>
<th id="A5.T6.1.1.2.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column">Diff.</th>
</tr>
<tr id="A5.T6.1.1.3.2" class="ltx_tr">
<th id="A5.T6.1.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_t">Rate = 0%</th>
<th id="A5.T6.1.1.3.2.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_t">baseline</th>
<th id="A5.T6.1.1.3.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">44.20</th>
<th id="A5.T6.1.1.3.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">76.00</th>
<th id="A5.T6.1.1.3.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">78.07</th>
<th id="A5.T6.1.1.3.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">76.30</th>
<th id="A5.T6.1.1.3.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">46.42</th>
<th id="A5.T6.1.1.3.2.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">69.30</th>
<th id="A5.T6.1.1.3.2.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">65.05</th>
<th id="A5.T6.1.1.3.2.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">-</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A5.T6.1.1.4.1" class="ltx_tr">
<th id="A5.T6.1.1.4.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t" rowspan="3"><span id="A5.T6.1.1.4.1.1.1" class="ltx_text">
<span id="A5.T6.1.1.4.1.1.1.1" class="ltx_tabular ltx_align_middle">
<span id="A5.T6.1.1.4.1.1.1.1.1" class="ltx_tr">
<span id="A5.T6.1.1.4.1.1.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left">Rate = 50%</span></span>
<span id="A5.T6.1.1.4.1.1.1.1.2" class="ltx_tr">
<span id="A5.T6.1.1.4.1.1.1.1.2.1" class="ltx_td ltx_nopad_r ltx_align_left">w/o LoRA</span></span>
</span></span></th>
<th id="A5.T6.1.1.4.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">J-LRD</th>
<td id="A5.T6.1.1.4.1.3" class="ltx_td ltx_align_center ltx_border_t">45.40</td>
<td id="A5.T6.1.1.4.1.4" class="ltx_td ltx_align_center ltx_border_t">75.57</td>
<td id="A5.T6.1.1.4.1.5" class="ltx_td ltx_align_center ltx_border_t">77.48</td>
<td id="A5.T6.1.1.4.1.6" class="ltx_td ltx_align_center ltx_border_t">75.97</td>
<td id="A5.T6.1.1.4.1.7" class="ltx_td ltx_align_center ltx_border_t">45.31</td>
<td id="A5.T6.1.1.4.1.8" class="ltx_td ltx_align_center ltx_border_t">69.22</td>
<td id="A5.T6.1.1.4.1.9" class="ltx_td ltx_align_center ltx_border_t">64.83</td>
<td id="A5.T6.1.1.4.1.10" class="ltx_td ltx_align_center ltx_border_t">-0.22</td>
</tr>
<tr id="A5.T6.1.1.5.2" class="ltx_tr">
<th id="A5.T6.1.1.5.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">G-LRD</th>
<td id="A5.T6.1.1.5.2.2" class="ltx_td ltx_align_center">43.60</td>
<td id="A5.T6.1.1.5.2.3" class="ltx_td ltx_align_center">73.39</td>
<td id="A5.T6.1.1.5.2.4" class="ltx_td ltx_align_center">76.33</td>
<td id="A5.T6.1.1.5.2.5" class="ltx_td ltx_align_center">73.02</td>
<td id="A5.T6.1.1.5.2.6" class="ltx_td ltx_align_center">42.57</td>
<td id="A5.T6.1.1.5.2.7" class="ltx_td ltx_align_center">66.77</td>
<td id="A5.T6.1.1.5.2.8" class="ltx_td ltx_align_center">62.61</td>
<td id="A5.T6.1.1.5.2.9" class="ltx_td ltx_align_center">-2.44</td>
</tr>
<tr id="A5.T6.1.1.6.3" class="ltx_tr">
<th id="A5.T6.1.1.6.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">M-LRD</th>
<td id="A5.T6.1.1.6.3.2" class="ltx_td ltx_align_center">39.60</td>
<td id="A5.T6.1.1.6.3.3" class="ltx_td ltx_align_center">65.35</td>
<td id="A5.T6.1.1.6.3.4" class="ltx_td ltx_align_center">74.76</td>
<td id="A5.T6.1.1.6.3.5" class="ltx_td ltx_align_center">67.17</td>
<td id="A5.T6.1.1.6.3.6" class="ltx_td ltx_align_center">35.24</td>
<td id="A5.T6.1.1.6.3.7" class="ltx_td ltx_align_center">64.64</td>
<td id="A5.T6.1.1.6.3.8" class="ltx_td ltx_align_center">57.79</td>
<td id="A5.T6.1.1.6.3.9" class="ltx_td ltx_align_center">-7.26</td>
</tr>
<tr id="A5.T6.1.1.7.4" class="ltx_tr">
<th id="A5.T6.1.1.7.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_r ltx_border_t" rowspan="3"><span id="A5.T6.1.1.7.4.1.1" class="ltx_text">
<span id="A5.T6.1.1.7.4.1.1.1" class="ltx_tabular ltx_align_middle">
<span id="A5.T6.1.1.7.4.1.1.1.1" class="ltx_tr">
<span id="A5.T6.1.1.7.4.1.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left">Rate = 50%</span></span>
<span id="A5.T6.1.1.7.4.1.1.1.2" class="ltx_tr">
<span id="A5.T6.1.1.7.4.1.1.1.2.1" class="ltx_td ltx_nopad_r ltx_align_left">w/ LoRA</span></span>
</span></span></th>
<th id="A5.T6.1.1.7.4.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">J-LRD</th>
<td id="A5.T6.1.1.7.4.3" class="ltx_td ltx_align_center ltx_border_t">44.20</td>
<td id="A5.T6.1.1.7.4.4" class="ltx_td ltx_align_center ltx_border_t">74.09</td>
<td id="A5.T6.1.1.7.4.5" class="ltx_td ltx_align_center ltx_border_t">78.51</td>
<td id="A5.T6.1.1.7.4.6" class="ltx_td ltx_align_center ltx_border_t">77.27</td>
<td id="A5.T6.1.1.7.4.7" class="ltx_td ltx_align_center ltx_border_t">48.81</td>
<td id="A5.T6.1.1.7.4.8" class="ltx_td ltx_align_center ltx_border_t">71.03</td>
<td id="A5.T6.1.1.7.4.9" class="ltx_td ltx_align_center ltx_border_t">65.65</td>
<td id="A5.T6.1.1.7.4.10" class="ltx_td ltx_align_center ltx_border_t">+0.60</td>
</tr>
<tr id="A5.T6.1.1.8.5" class="ltx_tr">
<th id="A5.T6.1.1.8.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">G-LRD</th>
<td id="A5.T6.1.1.8.5.2" class="ltx_td ltx_align_center">43.40</td>
<td id="A5.T6.1.1.8.5.3" class="ltx_td ltx_align_center">73.08</td>
<td id="A5.T6.1.1.8.5.4" class="ltx_td ltx_align_center">78.56</td>
<td id="A5.T6.1.1.8.5.5" class="ltx_td ltx_align_center">75.72</td>
<td id="A5.T6.1.1.8.5.6" class="ltx_td ltx_align_center">47.10</td>
<td id="A5.T6.1.1.8.5.7" class="ltx_td ltx_align_center">69.85</td>
<td id="A5.T6.1.1.8.5.8" class="ltx_td ltx_align_center">64.62</td>
<td id="A5.T6.1.1.8.5.9" class="ltx_td ltx_align_center">-0.43</td>
</tr>
<tr id="A5.T6.1.1.9.6" class="ltx_tr">
<th id="A5.T6.1.1.9.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_r">M-LRD</th>
<td id="A5.T6.1.1.9.6.2" class="ltx_td ltx_align_center ltx_border_bb">41.80</td>
<td id="A5.T6.1.1.9.6.3" class="ltx_td ltx_align_center ltx_border_bb">70.78</td>
<td id="A5.T6.1.1.9.6.4" class="ltx_td ltx_align_center ltx_border_bb">78.02</td>
<td id="A5.T6.1.1.9.6.5" class="ltx_td ltx_align_center ltx_border_bb">73.86</td>
<td id="A5.T6.1.1.9.6.6" class="ltx_td ltx_align_center ltx_border_bb">43.86</td>
<td id="A5.T6.1.1.9.6.7" class="ltx_td ltx_align_center ltx_border_bb">69.22</td>
<td id="A5.T6.1.1.9.6.8" class="ltx_td ltx_align_center ltx_border_bb">62.92</td>
<td id="A5.T6.1.1.9.6.9" class="ltx_td ltx_align_center ltx_border_bb">-2.12</td>
</tr>
</tbody>
</table>
</span></div>
</figure>
<figure id="A5.T7" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 7: </span>
Zero-shot accuracy of Palu on LLama2 models, with different decomposition strategies at 30% compression rate.
</figcaption>
<div id="A5.T7.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:380.0pt;height:160.9pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-22.6pt,9.6pt) scale(0.893708748848849,0.893708748848849) ;">
<table id="A5.T7.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A5.T7.1.1.1" class="ltx_tr">
<th id="A5.T7.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_tt" rowspan="2"><span id="A5.T7.1.1.1.2.1" class="ltx_text">Model</span></th>
<th id="A5.T7.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_tt" rowspan="2"><span id="A5.T7.1.1.1.3.1" class="ltx_text">Method</span></th>
<th id="A5.T7.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="7">Zero-Shot Accuracy (%) <math id="A5.T7.1.1.1.1.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="A5.T7.1.1.1.1.m1.1a"><mo stretchy="false" id="A5.T7.1.1.1.1.m1.1.1" xref="A5.T7.1.1.1.1.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="A5.T7.1.1.1.1.m1.1b"><ci id="A5.T7.1.1.1.1.m1.1.1.cmml" xref="A5.T7.1.1.1.1.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.T7.1.1.1.1.m1.1c">\uparrow</annotation></semantics></math>
</th>
<th id="A5.T7.1.1.1.4" class="ltx_td ltx_th ltx_th_column ltx_border_tt"></th>
</tr>
<tr id="A5.T7.1.1.2.1" class="ltx_tr">
<th id="A5.T7.1.1.2.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column">OBQA</th>
<th id="A5.T7.1.1.2.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column">Hella</th>
<th id="A5.T7.1.1.2.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column">PIQA</th>
<th id="A5.T7.1.1.2.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column">ARC-e</th>
<th id="A5.T7.1.1.2.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column">ARC-c</th>
<th id="A5.T7.1.1.2.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column">Wino</th>
<th id="A5.T7.1.1.2.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column">Avg.</th>
<th id="A5.T7.1.1.2.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column">Diff.</th>
</tr>
<tr id="A5.T7.1.1.3.2" class="ltx_tr">
<th id="A5.T7.1.1.3.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_t" rowspan="4"><span id="A5.T7.1.1.3.2.1.1" class="ltx_text">Llama2-7B</span></th>
<th id="A5.T7.1.1.3.2.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_t">Baseline</th>
<th id="A5.T7.1.1.3.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">44.20</th>
<th id="A5.T7.1.1.3.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">76.00</th>
<th id="A5.T7.1.1.3.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">78.07</th>
<th id="A5.T7.1.1.3.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">76.30</th>
<th id="A5.T7.1.1.3.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">46.42</th>
<th id="A5.T7.1.1.3.2.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">69.30</th>
<th id="A5.T7.1.1.3.2.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">65.05</th>
<th id="A5.T7.1.1.3.2.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">-</th>
</tr>
<tr id="A5.T7.1.1.4.3" class="ltx_tr">
<th id="A5.T7.1.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">J-LRD</th>
<td id="A5.T7.1.1.4.3.2" class="ltx_td ltx_align_center ltx_border_t">44.40</td>
<td id="A5.T7.1.1.4.3.3" class="ltx_td ltx_align_center ltx_border_t">76.01</td>
<td id="A5.T7.1.1.4.3.4" class="ltx_td ltx_align_center ltx_border_t">77.75</td>
<td id="A5.T7.1.1.4.3.5" class="ltx_td ltx_align_center ltx_border_t">76.01</td>
<td id="A5.T7.1.1.4.3.6" class="ltx_td ltx_align_center ltx_border_t">45.31</td>
<td id="A5.T7.1.1.4.3.7" class="ltx_td ltx_align_center ltx_border_t">69.38</td>
<td id="A5.T7.1.1.4.3.8" class="ltx_td ltx_align_center ltx_border_t">64.81</td>
<td id="A5.T7.1.1.4.3.9" class="ltx_td ltx_align_center ltx_border_t">-0.24</td>
</tr>
<tr id="A5.T7.1.1.5.4" class="ltx_tr">
<th id="A5.T7.1.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">G-LRD</th>
<td id="A5.T7.1.1.5.4.2" class="ltx_td ltx_align_center">45.20</td>
<td id="A5.T7.1.1.5.4.3" class="ltx_td ltx_align_center">76.07</td>
<td id="A5.T7.1.1.5.4.4" class="ltx_td ltx_align_center">78.56</td>
<td id="A5.T7.1.1.5.4.5" class="ltx_td ltx_align_center">75.93</td>
<td id="A5.T7.1.1.5.4.6" class="ltx_td ltx_align_center">44.97</td>
<td id="A5.T7.1.1.5.4.7" class="ltx_td ltx_align_center">68.75</td>
<td id="A5.T7.1.1.5.4.8" class="ltx_td ltx_align_center">64.91</td>
<td id="A5.T7.1.1.5.4.9" class="ltx_td ltx_align_center">-0.14</td>
</tr>
<tr id="A5.T7.1.1.6.5" class="ltx_tr">
<th id="A5.T7.1.1.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">M-LRD</th>
<td id="A5.T7.1.1.6.5.2" class="ltx_td ltx_align_center">43.20</td>
<td id="A5.T7.1.1.6.5.3" class="ltx_td ltx_align_center">73.50</td>
<td id="A5.T7.1.1.6.5.4" class="ltx_td ltx_align_center">77.42</td>
<td id="A5.T7.1.1.6.5.5" class="ltx_td ltx_align_center">74.16</td>
<td id="A5.T7.1.1.6.5.6" class="ltx_td ltx_align_center">42.58</td>
<td id="A5.T7.1.1.6.5.7" class="ltx_td ltx_align_center">68.90</td>
<td id="A5.T7.1.1.6.5.8" class="ltx_td ltx_align_center">63.29</td>
<td id="A5.T7.1.1.6.5.9" class="ltx_td ltx_align_center">-1.76</td>
</tr>
<tr id="A5.T7.1.1.7.6" class="ltx_tr">
<th id="A5.T7.1.1.7.6.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb ltx_border_r ltx_border_t" rowspan="4"><span id="A5.T7.1.1.7.6.1.1" class="ltx_text">Llama2-13B</span></th>
<th id="A5.T7.1.1.7.6.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">Baseline</th>
<td id="A5.T7.1.1.7.6.3" class="ltx_td ltx_align_center ltx_border_t">45.20</td>
<td id="A5.T7.1.1.7.6.4" class="ltx_td ltx_align_center ltx_border_t">79.39</td>
<td id="A5.T7.1.1.7.6.5" class="ltx_td ltx_align_center ltx_border_t">79.11</td>
<td id="A5.T7.1.1.7.6.6" class="ltx_td ltx_align_center ltx_border_t">79.42</td>
<td id="A5.T7.1.1.7.6.7" class="ltx_td ltx_align_center ltx_border_t">49.06</td>
<td id="A5.T7.1.1.7.6.8" class="ltx_td ltx_align_center ltx_border_t">72.38</td>
<td id="A5.T7.1.1.7.6.9" class="ltx_td ltx_align_center ltx_border_t">67.43</td>
<td id="A5.T7.1.1.7.6.10" class="ltx_td ltx_align_center ltx_border_t">-</td>
</tr>
<tr id="A5.T7.1.1.8.7" class="ltx_tr">
<th id="A5.T7.1.1.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">J-LRD</th>
<td id="A5.T7.1.1.8.7.2" class="ltx_td ltx_align_center ltx_border_t">45.20</td>
<td id="A5.T7.1.1.8.7.3" class="ltx_td ltx_align_center ltx_border_t">79.58</td>
<td id="A5.T7.1.1.8.7.4" class="ltx_td ltx_align_center ltx_border_t">80.57</td>
<td id="A5.T7.1.1.8.7.5" class="ltx_td ltx_align_center ltx_border_t">79.67</td>
<td id="A5.T7.1.1.8.7.6" class="ltx_td ltx_align_center ltx_border_t">49.40</td>
<td id="A5.T7.1.1.8.7.7" class="ltx_td ltx_align_center ltx_border_t">71.74</td>
<td id="A5.T7.1.1.8.7.8" class="ltx_td ltx_align_center ltx_border_t">67.69</td>
<td id="A5.T7.1.1.8.7.9" class="ltx_td ltx_align_center ltx_border_t">+0.26</td>
</tr>
<tr id="A5.T7.1.1.9.8" class="ltx_tr">
<th id="A5.T7.1.1.9.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">G-LRD</th>
<td id="A5.T7.1.1.9.8.2" class="ltx_td ltx_align_center">46.60</td>
<td id="A5.T7.1.1.9.8.3" class="ltx_td ltx_align_center">79.04</td>
<td id="A5.T7.1.1.9.8.4" class="ltx_td ltx_align_center">78.23</td>
<td id="A5.T7.1.1.9.8.5" class="ltx_td ltx_align_center">79.42</td>
<td id="A5.T7.1.1.9.8.6" class="ltx_td ltx_align_center">48.38</td>
<td id="A5.T7.1.1.9.8.7" class="ltx_td ltx_align_center">71.59</td>
<td id="A5.T7.1.1.9.8.8" class="ltx_td ltx_align_center">67.21</td>
<td id="A5.T7.1.1.9.8.9" class="ltx_td ltx_align_center">-0.22</td>
</tr>
<tr id="A5.T7.1.1.10.9" class="ltx_tr">
<th id="A5.T7.1.1.10.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_r">M-LRD</th>
<td id="A5.T7.1.1.10.9.2" class="ltx_td ltx_align_center ltx_border_bb">45.40</td>
<td id="A5.T7.1.1.10.9.3" class="ltx_td ltx_align_center ltx_border_bb">78.57</td>
<td id="A5.T7.1.1.10.9.4" class="ltx_td ltx_align_center ltx_border_bb">78.78</td>
<td id="A5.T7.1.1.10.9.5" class="ltx_td ltx_align_center ltx_border_bb">78.83</td>
<td id="A5.T7.1.1.10.9.6" class="ltx_td ltx_align_center ltx_border_bb">47.61</td>
<td id="A5.T7.1.1.10.9.7" class="ltx_td ltx_align_center ltx_border_bb">72.06</td>
<td id="A5.T7.1.1.10.9.8" class="ltx_td ltx_align_center ltx_border_bb">66.88</td>
<td id="A5.T7.1.1.10.9.9" class="ltx_td ltx_align_center ltx_border_bb">-0.55</td>
</tr>
</tbody>
</table>
</span></div>
</figure>
</section>
</section>
<section id="A6" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix F </span>Integrating <span id="A6.1.1" class="ltx_text ltx_font_italic">Palu</span> with LoRA Finetune</h2>

<div id="A6.p1" class="ltx_para">
<p id="A6.p1.1" class="ltx_p">LoRA <cite class="ltx_cite ltx_citemacro_citep">(Hu et al., <a href="#bib.bib18" title="" class="ltx_ref">2022</a>)</cite> has become one of the most widely used efficient fine-tuning techniques for adapting models to particular tasks or domains with limited data. It has also been applied with LLM compression approaches <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib42" title="" class="ltx_ref">2024</a>; Ma et al., <a href="#bib.bib28" title="" class="ltx_ref">2023</a>)</cite> as a post-compression recovery technique to recover information loss after compression. In <span id="A6.p1.1.1" class="ltx_text ltx_font_italic">Palu</span>, LoRA is also applicable to boost the accuracy further.</p>
</div>
<div id="A6.p2" class="ltx_para">
<p id="A6.p2.2" class="ltx_p">To integrate LoRA with <span id="A6.p2.2.1" class="ltx_text ltx_font_italic">Palu</span>, we introduce additional low-rank matrices <math id="A6.p2.1.m1.1" class="ltx_Math" alttext="\mathbf{A}_{r}^{\prime}\in\mathbb{R}^{d\times r^{\prime}}" display="inline"><semantics id="A6.p2.1.m1.1a"><mrow id="A6.p2.1.m1.1.1" xref="A6.p2.1.m1.1.1.cmml"><msubsup id="A6.p2.1.m1.1.1.2" xref="A6.p2.1.m1.1.1.2.cmml"><mi id="A6.p2.1.m1.1.1.2.2.2" xref="A6.p2.1.m1.1.1.2.2.2.cmml">𝐀</mi><mi id="A6.p2.1.m1.1.1.2.2.3" xref="A6.p2.1.m1.1.1.2.2.3.cmml">r</mi><mo id="A6.p2.1.m1.1.1.2.3" xref="A6.p2.1.m1.1.1.2.3.cmml">′</mo></msubsup><mo id="A6.p2.1.m1.1.1.1" xref="A6.p2.1.m1.1.1.1.cmml">∈</mo><msup id="A6.p2.1.m1.1.1.3" xref="A6.p2.1.m1.1.1.3.cmml"><mi id="A6.p2.1.m1.1.1.3.2" xref="A6.p2.1.m1.1.1.3.2.cmml">ℝ</mi><mrow id="A6.p2.1.m1.1.1.3.3" xref="A6.p2.1.m1.1.1.3.3.cmml"><mi id="A6.p2.1.m1.1.1.3.3.2" xref="A6.p2.1.m1.1.1.3.3.2.cmml">d</mi><mo lspace="0.222em" rspace="0.222em" id="A6.p2.1.m1.1.1.3.3.1" xref="A6.p2.1.m1.1.1.3.3.1.cmml">×</mo><msup id="A6.p2.1.m1.1.1.3.3.3" xref="A6.p2.1.m1.1.1.3.3.3.cmml"><mi id="A6.p2.1.m1.1.1.3.3.3.2" xref="A6.p2.1.m1.1.1.3.3.3.2.cmml">r</mi><mo id="A6.p2.1.m1.1.1.3.3.3.3" xref="A6.p2.1.m1.1.1.3.3.3.3.cmml">′</mo></msup></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A6.p2.1.m1.1b"><apply id="A6.p2.1.m1.1.1.cmml" xref="A6.p2.1.m1.1.1"><in id="A6.p2.1.m1.1.1.1.cmml" xref="A6.p2.1.m1.1.1.1"></in><apply id="A6.p2.1.m1.1.1.2.cmml" xref="A6.p2.1.m1.1.1.2"><csymbol cd="ambiguous" id="A6.p2.1.m1.1.1.2.1.cmml" xref="A6.p2.1.m1.1.1.2">superscript</csymbol><apply id="A6.p2.1.m1.1.1.2.2.cmml" xref="A6.p2.1.m1.1.1.2"><csymbol cd="ambiguous" id="A6.p2.1.m1.1.1.2.2.1.cmml" xref="A6.p2.1.m1.1.1.2">subscript</csymbol><ci id="A6.p2.1.m1.1.1.2.2.2.cmml" xref="A6.p2.1.m1.1.1.2.2.2">𝐀</ci><ci id="A6.p2.1.m1.1.1.2.2.3.cmml" xref="A6.p2.1.m1.1.1.2.2.3">𝑟</ci></apply><ci id="A6.p2.1.m1.1.1.2.3.cmml" xref="A6.p2.1.m1.1.1.2.3">′</ci></apply><apply id="A6.p2.1.m1.1.1.3.cmml" xref="A6.p2.1.m1.1.1.3"><csymbol cd="ambiguous" id="A6.p2.1.m1.1.1.3.1.cmml" xref="A6.p2.1.m1.1.1.3">superscript</csymbol><ci id="A6.p2.1.m1.1.1.3.2.cmml" xref="A6.p2.1.m1.1.1.3.2">ℝ</ci><apply id="A6.p2.1.m1.1.1.3.3.cmml" xref="A6.p2.1.m1.1.1.3.3"><times id="A6.p2.1.m1.1.1.3.3.1.cmml" xref="A6.p2.1.m1.1.1.3.3.1"></times><ci id="A6.p2.1.m1.1.1.3.3.2.cmml" xref="A6.p2.1.m1.1.1.3.3.2">𝑑</ci><apply id="A6.p2.1.m1.1.1.3.3.3.cmml" xref="A6.p2.1.m1.1.1.3.3.3"><csymbol cd="ambiguous" id="A6.p2.1.m1.1.1.3.3.3.1.cmml" xref="A6.p2.1.m1.1.1.3.3.3">superscript</csymbol><ci id="A6.p2.1.m1.1.1.3.3.3.2.cmml" xref="A6.p2.1.m1.1.1.3.3.3.2">𝑟</ci><ci id="A6.p2.1.m1.1.1.3.3.3.3.cmml" xref="A6.p2.1.m1.1.1.3.3.3.3">′</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.p2.1.m1.1c">\mathbf{A}_{r}^{\prime}\in\mathbb{R}^{d\times r^{\prime}}</annotation></semantics></math> and <math id="A6.p2.2.m2.1" class="ltx_Math" alttext="\mathbf{B}_{r}^{\prime}\in\mathbb{R}^{r^{\prime}\times d}" display="inline"><semantics id="A6.p2.2.m2.1a"><mrow id="A6.p2.2.m2.1.1" xref="A6.p2.2.m2.1.1.cmml"><msubsup id="A6.p2.2.m2.1.1.2" xref="A6.p2.2.m2.1.1.2.cmml"><mi id="A6.p2.2.m2.1.1.2.2.2" xref="A6.p2.2.m2.1.1.2.2.2.cmml">𝐁</mi><mi id="A6.p2.2.m2.1.1.2.2.3" xref="A6.p2.2.m2.1.1.2.2.3.cmml">r</mi><mo id="A6.p2.2.m2.1.1.2.3" xref="A6.p2.2.m2.1.1.2.3.cmml">′</mo></msubsup><mo id="A6.p2.2.m2.1.1.1" xref="A6.p2.2.m2.1.1.1.cmml">∈</mo><msup id="A6.p2.2.m2.1.1.3" xref="A6.p2.2.m2.1.1.3.cmml"><mi id="A6.p2.2.m2.1.1.3.2" xref="A6.p2.2.m2.1.1.3.2.cmml">ℝ</mi><mrow id="A6.p2.2.m2.1.1.3.3" xref="A6.p2.2.m2.1.1.3.3.cmml"><msup id="A6.p2.2.m2.1.1.3.3.2" xref="A6.p2.2.m2.1.1.3.3.2.cmml"><mi id="A6.p2.2.m2.1.1.3.3.2.2" xref="A6.p2.2.m2.1.1.3.3.2.2.cmml">r</mi><mo id="A6.p2.2.m2.1.1.3.3.2.3" xref="A6.p2.2.m2.1.1.3.3.2.3.cmml">′</mo></msup><mo lspace="0.222em" rspace="0.222em" id="A6.p2.2.m2.1.1.3.3.1" xref="A6.p2.2.m2.1.1.3.3.1.cmml">×</mo><mi id="A6.p2.2.m2.1.1.3.3.3" xref="A6.p2.2.m2.1.1.3.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A6.p2.2.m2.1b"><apply id="A6.p2.2.m2.1.1.cmml" xref="A6.p2.2.m2.1.1"><in id="A6.p2.2.m2.1.1.1.cmml" xref="A6.p2.2.m2.1.1.1"></in><apply id="A6.p2.2.m2.1.1.2.cmml" xref="A6.p2.2.m2.1.1.2"><csymbol cd="ambiguous" id="A6.p2.2.m2.1.1.2.1.cmml" xref="A6.p2.2.m2.1.1.2">superscript</csymbol><apply id="A6.p2.2.m2.1.1.2.2.cmml" xref="A6.p2.2.m2.1.1.2"><csymbol cd="ambiguous" id="A6.p2.2.m2.1.1.2.2.1.cmml" xref="A6.p2.2.m2.1.1.2">subscript</csymbol><ci id="A6.p2.2.m2.1.1.2.2.2.cmml" xref="A6.p2.2.m2.1.1.2.2.2">𝐁</ci><ci id="A6.p2.2.m2.1.1.2.2.3.cmml" xref="A6.p2.2.m2.1.1.2.2.3">𝑟</ci></apply><ci id="A6.p2.2.m2.1.1.2.3.cmml" xref="A6.p2.2.m2.1.1.2.3">′</ci></apply><apply id="A6.p2.2.m2.1.1.3.cmml" xref="A6.p2.2.m2.1.1.3"><csymbol cd="ambiguous" id="A6.p2.2.m2.1.1.3.1.cmml" xref="A6.p2.2.m2.1.1.3">superscript</csymbol><ci id="A6.p2.2.m2.1.1.3.2.cmml" xref="A6.p2.2.m2.1.1.3.2">ℝ</ci><apply id="A6.p2.2.m2.1.1.3.3.cmml" xref="A6.p2.2.m2.1.1.3.3"><times id="A6.p2.2.m2.1.1.3.3.1.cmml" xref="A6.p2.2.m2.1.1.3.3.1"></times><apply id="A6.p2.2.m2.1.1.3.3.2.cmml" xref="A6.p2.2.m2.1.1.3.3.2"><csymbol cd="ambiguous" id="A6.p2.2.m2.1.1.3.3.2.1.cmml" xref="A6.p2.2.m2.1.1.3.3.2">superscript</csymbol><ci id="A6.p2.2.m2.1.1.3.3.2.2.cmml" xref="A6.p2.2.m2.1.1.3.3.2.2">𝑟</ci><ci id="A6.p2.2.m2.1.1.3.3.2.3.cmml" xref="A6.p2.2.m2.1.1.3.3.2.3">′</ci></apply><ci id="A6.p2.2.m2.1.1.3.3.3.cmml" xref="A6.p2.2.m2.1.1.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.p2.2.m2.1c">\mathbf{B}_{r}^{\prime}\in\mathbb{R}^{r^{\prime}\times d}</annotation></semantics></math> to refine the original low-rank projection as below:</p>
<table id="A6.E9" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A6.E9.m1.1" class="ltx_Math" alttext="\mathbf{h}=\mathbf{A}\mathbf{x}+\mathbf{A}_{\mathbf{r^{\prime}}}\mathbf{B}_{\mathbf{r^{\prime}}}\mathbf{x}" display="block"><semantics id="A6.E9.m1.1a"><mrow id="A6.E9.m1.1.1" xref="A6.E9.m1.1.1.cmml"><mi id="A6.E9.m1.1.1.2" xref="A6.E9.m1.1.1.2.cmml">𝐡</mi><mo id="A6.E9.m1.1.1.1" xref="A6.E9.m1.1.1.1.cmml">=</mo><mrow id="A6.E9.m1.1.1.3" xref="A6.E9.m1.1.1.3.cmml"><mi id="A6.E9.m1.1.1.3.2" xref="A6.E9.m1.1.1.3.2.cmml">𝐀𝐱</mi><mo id="A6.E9.m1.1.1.3.1" xref="A6.E9.m1.1.1.3.1.cmml">+</mo><mrow id="A6.E9.m1.1.1.3.3" xref="A6.E9.m1.1.1.3.3.cmml"><msub id="A6.E9.m1.1.1.3.3.2" xref="A6.E9.m1.1.1.3.3.2.cmml"><mi id="A6.E9.m1.1.1.3.3.2.2" xref="A6.E9.m1.1.1.3.3.2.2.cmml">𝐀</mi><msup id="A6.E9.m1.1.1.3.3.2.3" xref="A6.E9.m1.1.1.3.3.2.3.cmml"><mi id="A6.E9.m1.1.1.3.3.2.3.2" xref="A6.E9.m1.1.1.3.3.2.3.2.cmml">𝐫</mi><mo id="A6.E9.m1.1.1.3.3.2.3.3" xref="A6.E9.m1.1.1.3.3.2.3.3.cmml">′</mo></msup></msub><mo lspace="0em" rspace="0em" id="A6.E9.m1.1.1.3.3.1" xref="A6.E9.m1.1.1.3.3.1.cmml">​</mo><msub id="A6.E9.m1.1.1.3.3.3" xref="A6.E9.m1.1.1.3.3.3.cmml"><mi id="A6.E9.m1.1.1.3.3.3.2" xref="A6.E9.m1.1.1.3.3.3.2.cmml">𝐁</mi><msup id="A6.E9.m1.1.1.3.3.3.3" xref="A6.E9.m1.1.1.3.3.3.3.cmml"><mi id="A6.E9.m1.1.1.3.3.3.3.2" xref="A6.E9.m1.1.1.3.3.3.3.2.cmml">𝐫</mi><mo id="A6.E9.m1.1.1.3.3.3.3.3" xref="A6.E9.m1.1.1.3.3.3.3.3.cmml">′</mo></msup></msub><mo lspace="0em" rspace="0em" id="A6.E9.m1.1.1.3.3.1a" xref="A6.E9.m1.1.1.3.3.1.cmml">​</mo><mi id="A6.E9.m1.1.1.3.3.4" xref="A6.E9.m1.1.1.3.3.4.cmml">𝐱</mi></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A6.E9.m1.1b"><apply id="A6.E9.m1.1.1.cmml" xref="A6.E9.m1.1.1"><eq id="A6.E9.m1.1.1.1.cmml" xref="A6.E9.m1.1.1.1"></eq><ci id="A6.E9.m1.1.1.2.cmml" xref="A6.E9.m1.1.1.2">𝐡</ci><apply id="A6.E9.m1.1.1.3.cmml" xref="A6.E9.m1.1.1.3"><plus id="A6.E9.m1.1.1.3.1.cmml" xref="A6.E9.m1.1.1.3.1"></plus><ci id="A6.E9.m1.1.1.3.2.cmml" xref="A6.E9.m1.1.1.3.2">𝐀𝐱</ci><apply id="A6.E9.m1.1.1.3.3.cmml" xref="A6.E9.m1.1.1.3.3"><times id="A6.E9.m1.1.1.3.3.1.cmml" xref="A6.E9.m1.1.1.3.3.1"></times><apply id="A6.E9.m1.1.1.3.3.2.cmml" xref="A6.E9.m1.1.1.3.3.2"><csymbol cd="ambiguous" id="A6.E9.m1.1.1.3.3.2.1.cmml" xref="A6.E9.m1.1.1.3.3.2">subscript</csymbol><ci id="A6.E9.m1.1.1.3.3.2.2.cmml" xref="A6.E9.m1.1.1.3.3.2.2">𝐀</ci><apply id="A6.E9.m1.1.1.3.3.2.3.cmml" xref="A6.E9.m1.1.1.3.3.2.3"><csymbol cd="ambiguous" id="A6.E9.m1.1.1.3.3.2.3.1.cmml" xref="A6.E9.m1.1.1.3.3.2.3">superscript</csymbol><ci id="A6.E9.m1.1.1.3.3.2.3.2.cmml" xref="A6.E9.m1.1.1.3.3.2.3.2">𝐫</ci><ci id="A6.E9.m1.1.1.3.3.2.3.3.cmml" xref="A6.E9.m1.1.1.3.3.2.3.3">′</ci></apply></apply><apply id="A6.E9.m1.1.1.3.3.3.cmml" xref="A6.E9.m1.1.1.3.3.3"><csymbol cd="ambiguous" id="A6.E9.m1.1.1.3.3.3.1.cmml" xref="A6.E9.m1.1.1.3.3.3">subscript</csymbol><ci id="A6.E9.m1.1.1.3.3.3.2.cmml" xref="A6.E9.m1.1.1.3.3.3.2">𝐁</ci><apply id="A6.E9.m1.1.1.3.3.3.3.cmml" xref="A6.E9.m1.1.1.3.3.3.3"><csymbol cd="ambiguous" id="A6.E9.m1.1.1.3.3.3.3.1.cmml" xref="A6.E9.m1.1.1.3.3.3.3">superscript</csymbol><ci id="A6.E9.m1.1.1.3.3.3.3.2.cmml" xref="A6.E9.m1.1.1.3.3.3.3.2">𝐫</ci><ci id="A6.E9.m1.1.1.3.3.3.3.3.cmml" xref="A6.E9.m1.1.1.3.3.3.3.3">′</ci></apply></apply><ci id="A6.E9.m1.1.1.3.3.4.cmml" xref="A6.E9.m1.1.1.3.3.4">𝐱</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.E9.m1.1c">\mathbf{h}=\mathbf{A}\mathbf{x}+\mathbf{A}_{\mathbf{r^{\prime}}}\mathbf{B}_{\mathbf{r^{\prime}}}\mathbf{x}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(9)</span></td>
</tr></tbody>
</table>
<p id="A6.p2.5" class="ltx_p">Here, <math id="A6.p2.3.m1.1" class="ltx_Math" alttext="\mathbf{A}" display="inline"><semantics id="A6.p2.3.m1.1a"><mi id="A6.p2.3.m1.1.1" xref="A6.p2.3.m1.1.1.cmml">𝐀</mi><annotation-xml encoding="MathML-Content" id="A6.p2.3.m1.1b"><ci id="A6.p2.3.m1.1.1.cmml" xref="A6.p2.3.m1.1.1">𝐀</ci></annotation-xml><annotation encoding="application/x-tex" id="A6.p2.3.m1.1c">\mathbf{A}</annotation></semantics></math> will be fixed parameters derived from low-rank decomposition from pre-trained weights of linear layers, while <math id="A6.p2.4.m2.1" class="ltx_Math" alttext="\mathbf{A}_{r}^{\prime}" display="inline"><semantics id="A6.p2.4.m2.1a"><msubsup id="A6.p2.4.m2.1.1" xref="A6.p2.4.m2.1.1.cmml"><mi id="A6.p2.4.m2.1.1.2.2" xref="A6.p2.4.m2.1.1.2.2.cmml">𝐀</mi><mi id="A6.p2.4.m2.1.1.2.3" xref="A6.p2.4.m2.1.1.2.3.cmml">r</mi><mo id="A6.p2.4.m2.1.1.3" xref="A6.p2.4.m2.1.1.3.cmml">′</mo></msubsup><annotation-xml encoding="MathML-Content" id="A6.p2.4.m2.1b"><apply id="A6.p2.4.m2.1.1.cmml" xref="A6.p2.4.m2.1.1"><csymbol cd="ambiguous" id="A6.p2.4.m2.1.1.1.cmml" xref="A6.p2.4.m2.1.1">superscript</csymbol><apply id="A6.p2.4.m2.1.1.2.cmml" xref="A6.p2.4.m2.1.1"><csymbol cd="ambiguous" id="A6.p2.4.m2.1.1.2.1.cmml" xref="A6.p2.4.m2.1.1">subscript</csymbol><ci id="A6.p2.4.m2.1.1.2.2.cmml" xref="A6.p2.4.m2.1.1.2.2">𝐀</ci><ci id="A6.p2.4.m2.1.1.2.3.cmml" xref="A6.p2.4.m2.1.1.2.3">𝑟</ci></apply><ci id="A6.p2.4.m2.1.1.3.cmml" xref="A6.p2.4.m2.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.p2.4.m2.1c">\mathbf{A}_{r}^{\prime}</annotation></semantics></math> and <math id="A6.p2.5.m3.1" class="ltx_Math" alttext="\mathbf{B}_{r}^{\prime}" display="inline"><semantics id="A6.p2.5.m3.1a"><msubsup id="A6.p2.5.m3.1.1" xref="A6.p2.5.m3.1.1.cmml"><mi id="A6.p2.5.m3.1.1.2.2" xref="A6.p2.5.m3.1.1.2.2.cmml">𝐁</mi><mi id="A6.p2.5.m3.1.1.2.3" xref="A6.p2.5.m3.1.1.2.3.cmml">r</mi><mo id="A6.p2.5.m3.1.1.3" xref="A6.p2.5.m3.1.1.3.cmml">′</mo></msubsup><annotation-xml encoding="MathML-Content" id="A6.p2.5.m3.1b"><apply id="A6.p2.5.m3.1.1.cmml" xref="A6.p2.5.m3.1.1"><csymbol cd="ambiguous" id="A6.p2.5.m3.1.1.1.cmml" xref="A6.p2.5.m3.1.1">superscript</csymbol><apply id="A6.p2.5.m3.1.1.2.cmml" xref="A6.p2.5.m3.1.1"><csymbol cd="ambiguous" id="A6.p2.5.m3.1.1.2.1.cmml" xref="A6.p2.5.m3.1.1">subscript</csymbol><ci id="A6.p2.5.m3.1.1.2.2.cmml" xref="A6.p2.5.m3.1.1.2.2">𝐁</ci><ci id="A6.p2.5.m3.1.1.2.3.cmml" xref="A6.p2.5.m3.1.1.2.3">𝑟</ci></apply><ci id="A6.p2.5.m3.1.1.3.cmml" xref="A6.p2.5.m3.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.p2.5.m3.1c">\mathbf{B}_{r}^{\prime}</annotation></semantics></math> are trainable parameters to capture the task-specific nuances and recovers the information lost during the compression.</p>
</div>
<section id="A6.SS0.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Setup.</h5>

<div id="A6.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="A6.SS0.SSS0.Px1.p1.3" class="ltx_p">Following <cite class="ltx_cite ltx_citemacro_citet">Ashkboos et al. <a href="#bib.bib1" title="" class="ltx_ref">2024a</a></cite>, we sample 8k samples from the Alpaca training dataset as a fine-tuning dataset and apply LoRA with rank <math id="A6.SS0.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="r^{\prime}=32" display="inline"><semantics id="A6.SS0.SSS0.Px1.p1.1.m1.1a"><mrow id="A6.SS0.SSS0.Px1.p1.1.m1.1.1" xref="A6.SS0.SSS0.Px1.p1.1.m1.1.1.cmml"><msup id="A6.SS0.SSS0.Px1.p1.1.m1.1.1.2" xref="A6.SS0.SSS0.Px1.p1.1.m1.1.1.2.cmml"><mi id="A6.SS0.SSS0.Px1.p1.1.m1.1.1.2.2" xref="A6.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.cmml">r</mi><mo id="A6.SS0.SSS0.Px1.p1.1.m1.1.1.2.3" xref="A6.SS0.SSS0.Px1.p1.1.m1.1.1.2.3.cmml">′</mo></msup><mo id="A6.SS0.SSS0.Px1.p1.1.m1.1.1.1" xref="A6.SS0.SSS0.Px1.p1.1.m1.1.1.1.cmml">=</mo><mn id="A6.SS0.SSS0.Px1.p1.1.m1.1.1.3" xref="A6.SS0.SSS0.Px1.p1.1.m1.1.1.3.cmml">32</mn></mrow><annotation-xml encoding="MathML-Content" id="A6.SS0.SSS0.Px1.p1.1.m1.1b"><apply id="A6.SS0.SSS0.Px1.p1.1.m1.1.1.cmml" xref="A6.SS0.SSS0.Px1.p1.1.m1.1.1"><eq id="A6.SS0.SSS0.Px1.p1.1.m1.1.1.1.cmml" xref="A6.SS0.SSS0.Px1.p1.1.m1.1.1.1"></eq><apply id="A6.SS0.SSS0.Px1.p1.1.m1.1.1.2.cmml" xref="A6.SS0.SSS0.Px1.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="A6.SS0.SSS0.Px1.p1.1.m1.1.1.2.1.cmml" xref="A6.SS0.SSS0.Px1.p1.1.m1.1.1.2">superscript</csymbol><ci id="A6.SS0.SSS0.Px1.p1.1.m1.1.1.2.2.cmml" xref="A6.SS0.SSS0.Px1.p1.1.m1.1.1.2.2">𝑟</ci><ci id="A6.SS0.SSS0.Px1.p1.1.m1.1.1.2.3.cmml" xref="A6.SS0.SSS0.Px1.p1.1.m1.1.1.2.3">′</ci></apply><cn type="integer" id="A6.SS0.SSS0.Px1.p1.1.m1.1.1.3.cmml" xref="A6.SS0.SSS0.Px1.p1.1.m1.1.1.3">32</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS0.SSS0.Px1.p1.1.m1.1c">r^{\prime}=32</annotation></semantics></math> and <math id="A6.SS0.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="\alpha=32" display="inline"><semantics id="A6.SS0.SSS0.Px1.p1.2.m2.1a"><mrow id="A6.SS0.SSS0.Px1.p1.2.m2.1.1" xref="A6.SS0.SSS0.Px1.p1.2.m2.1.1.cmml"><mi id="A6.SS0.SSS0.Px1.p1.2.m2.1.1.2" xref="A6.SS0.SSS0.Px1.p1.2.m2.1.1.2.cmml">α</mi><mo id="A6.SS0.SSS0.Px1.p1.2.m2.1.1.1" xref="A6.SS0.SSS0.Px1.p1.2.m2.1.1.1.cmml">=</mo><mn id="A6.SS0.SSS0.Px1.p1.2.m2.1.1.3" xref="A6.SS0.SSS0.Px1.p1.2.m2.1.1.3.cmml">32</mn></mrow><annotation-xml encoding="MathML-Content" id="A6.SS0.SSS0.Px1.p1.2.m2.1b"><apply id="A6.SS0.SSS0.Px1.p1.2.m2.1.1.cmml" xref="A6.SS0.SSS0.Px1.p1.2.m2.1.1"><eq id="A6.SS0.SSS0.Px1.p1.2.m2.1.1.1.cmml" xref="A6.SS0.SSS0.Px1.p1.2.m2.1.1.1"></eq><ci id="A6.SS0.SSS0.Px1.p1.2.m2.1.1.2.cmml" xref="A6.SS0.SSS0.Px1.p1.2.m2.1.1.2">𝛼</ci><cn type="integer" id="A6.SS0.SSS0.Px1.p1.2.m2.1.1.3.cmml" xref="A6.SS0.SSS0.Px1.p1.2.m2.1.1.3">32</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS0.SSS0.Px1.p1.2.m2.1c">\alpha=32</annotation></semantics></math>. All other hyper-parameters are aligned with <cite class="ltx_cite ltx_citemacro_citet">Ashkboos et al. (<a href="#bib.bib1" title="" class="ltx_ref">2024a</a>)</cite>, except for the learning rate <math id="A6.SS0.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="2e-4" display="inline"><semantics id="A6.SS0.SSS0.Px1.p1.3.m3.1a"><mrow id="A6.SS0.SSS0.Px1.p1.3.m3.1.1" xref="A6.SS0.SSS0.Px1.p1.3.m3.1.1.cmml"><mrow id="A6.SS0.SSS0.Px1.p1.3.m3.1.1.2" xref="A6.SS0.SSS0.Px1.p1.3.m3.1.1.2.cmml"><mn id="A6.SS0.SSS0.Px1.p1.3.m3.1.1.2.2" xref="A6.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A6.SS0.SSS0.Px1.p1.3.m3.1.1.2.1" xref="A6.SS0.SSS0.Px1.p1.3.m3.1.1.2.1.cmml">​</mo><mi id="A6.SS0.SSS0.Px1.p1.3.m3.1.1.2.3" xref="A6.SS0.SSS0.Px1.p1.3.m3.1.1.2.3.cmml">e</mi></mrow><mo id="A6.SS0.SSS0.Px1.p1.3.m3.1.1.1" xref="A6.SS0.SSS0.Px1.p1.3.m3.1.1.1.cmml">−</mo><mn id="A6.SS0.SSS0.Px1.p1.3.m3.1.1.3" xref="A6.SS0.SSS0.Px1.p1.3.m3.1.1.3.cmml">4</mn></mrow><annotation-xml encoding="MathML-Content" id="A6.SS0.SSS0.Px1.p1.3.m3.1b"><apply id="A6.SS0.SSS0.Px1.p1.3.m3.1.1.cmml" xref="A6.SS0.SSS0.Px1.p1.3.m3.1.1"><minus id="A6.SS0.SSS0.Px1.p1.3.m3.1.1.1.cmml" xref="A6.SS0.SSS0.Px1.p1.3.m3.1.1.1"></minus><apply id="A6.SS0.SSS0.Px1.p1.3.m3.1.1.2.cmml" xref="A6.SS0.SSS0.Px1.p1.3.m3.1.1.2"><times id="A6.SS0.SSS0.Px1.p1.3.m3.1.1.2.1.cmml" xref="A6.SS0.SSS0.Px1.p1.3.m3.1.1.2.1"></times><cn type="integer" id="A6.SS0.SSS0.Px1.p1.3.m3.1.1.2.2.cmml" xref="A6.SS0.SSS0.Px1.p1.3.m3.1.1.2.2">2</cn><ci id="A6.SS0.SSS0.Px1.p1.3.m3.1.1.2.3.cmml" xref="A6.SS0.SSS0.Px1.p1.3.m3.1.1.2.3">𝑒</ci></apply><cn type="integer" id="A6.SS0.SSS0.Px1.p1.3.m3.1.1.3.cmml" xref="A6.SS0.SSS0.Px1.p1.3.m3.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A6.SS0.SSS0.Px1.p1.3.m3.1c">2e-4</annotation></semantics></math>, and the use of a cosine learning rate scheduler.</p>
</div>
</section>
<section id="A6.SS0.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Experiment Results.</h5>

<div id="A6.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="A6.SS0.SSS0.Px2.p1.1" class="ltx_p">We present the experiment results with LoRA in Table <a href="#A5.T6" title="Table 6 ‣ Appendix E Extended Experiments on LongBench and comparisons to quantization baselines ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>. Following <cite class="ltx_cite ltx_citemacro_citet">Ashkboos et al. <a href="#bib.bib1" title="" class="ltx_ref">2024a</a></cite> With LoRA incorporated, J-LRD continues to show minimal performance degradation with an average drop of 1.00%. G-LRD (gs=4) and M-LRD show improved results compared to their non-LoRA counterparts, with average drops of 2.01% and 5.14%, respectively. Notably, with LoRA integration, G-LRD shows only a 1.03% accuracy difference compared to J-LRD.</p>
</div>
</section>
</section>
<section id="A7" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix G </span>More Results on Zero-shot Accuracy</h2>

<div id="A7.p1" class="ltx_para">
<p id="A7.p1.1" class="ltx_p">Following up Sec. <a href="#S4.SS1" title="4.1 Main Results ‣ 4 Experiments ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.1</span></a>, we further report the zero-shot evaluation results of <span id="A7.p1.1.1" class="ltx_text ltx_font_italic">Palu</span> on the LLama2 family at 30% compression rate. As shown
in Table. <a href="#A5.T7" title="Table 7 ‣ Appendix E Extended Experiments on LongBench and comparisons to quantization baselines ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>, we observe that <span id="A7.p1.1.2" class="ltx_text ltx_font_italic">Palu</span> achieve nearly no accuracy drops using either J-LRD, G-LRD, or M-LRD. Thus, the users may adopt M-LRD first to optimize the efficiency further.</p>
</div>
<figure id="A7.T8" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 8: </span>
Ablation study of low-rank decomposition group size on perplexity for the Llama2-7B model at a 50% compression rate using Wikitext-2.
</figcaption>
<div id="A7.T8.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:130.0pt;height:106.9pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-22.5pt,18.5pt) scale(0.742562644650227,0.742562644650227) ;">
<table id="A7.T8.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A7.T8.1.1.1.1" class="ltx_tr">
<th id="A7.T8.1.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Method</th>
<th id="A7.T8.1.1.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">Group Size</th>
<th id="A7.T8.1.1.1.1.3" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">Perplexity</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A7.T8.1.1.2.1" class="ltx_tr">
<td id="A7.T8.1.1.2.1.1" class="ltx_td ltx_align_center ltx_border_t">Baseline</td>
<td id="A7.T8.1.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t">-</td>
<td id="A7.T8.1.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t">5.47</td>
</tr>
<tr id="A7.T8.1.1.3.2" class="ltx_tr">
<td id="A7.T8.1.1.3.2.1" class="ltx_td ltx_align_center ltx_border_t">J-LRD</td>
<td id="A7.T8.1.1.3.2.2" class="ltx_td ltx_align_center ltx_border_t">32</td>
<td id="A7.T8.1.1.3.2.3" class="ltx_td ltx_align_center ltx_border_t">5.62</td>
</tr>
<tr id="A7.T8.1.1.4.3" class="ltx_tr">
<td id="A7.T8.1.1.4.3.1" class="ltx_td ltx_align_center ltx_border_t" rowspan="4"><span id="A7.T8.1.1.4.3.1.1" class="ltx_text">G-LRD</span></td>
<td id="A7.T8.1.1.4.3.2" class="ltx_td ltx_align_center ltx_border_t">16</td>
<td id="A7.T8.1.1.4.3.3" class="ltx_td ltx_align_center ltx_border_t">5.74</td>
</tr>
<tr id="A7.T8.1.1.5.4" class="ltx_tr">
<td id="A7.T8.1.1.5.4.1" class="ltx_td ltx_align_center">8</td>
<td id="A7.T8.1.1.5.4.2" class="ltx_td ltx_align_center">5.88</td>
</tr>
<tr id="A7.T8.1.1.6.5" class="ltx_tr">
<td id="A7.T8.1.1.6.5.1" class="ltx_td ltx_align_center">4</td>
<td id="A7.T8.1.1.6.5.2" class="ltx_td ltx_align_center">6.01</td>
</tr>
<tr id="A7.T8.1.1.7.6" class="ltx_tr">
<td id="A7.T8.1.1.7.6.1" class="ltx_td ltx_align_center">2</td>
<td id="A7.T8.1.1.7.6.2" class="ltx_td ltx_align_center">6.42</td>
</tr>
<tr id="A7.T8.1.1.8.7" class="ltx_tr">
<td id="A7.T8.1.1.8.7.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">M-LRD</td>
<td id="A7.T8.1.1.8.7.2" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">1</td>
<td id="A7.T8.1.1.8.7.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">6.81</td>
</tr>
</tbody>
</table>
</span></div>
</figure>
</section>
<section id="A8" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix H </span>Ablation Study</h2>

<section id="A8.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">H.1 </span>Influence of Different Group Size</h3>

<div id="A8.SS1.p1" class="ltx_para">
<p id="A8.SS1.p1.1" class="ltx_p">Since our proposed G-LRD method allows for balancing performance and efficiency by adjusting the group size, we conducted an ablation study on group size. As seen in Table <a href="#A7.T8" title="Table 8 ‣ Appendix G More Results on Zero-shot Accuracy ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a>, as the group size increases, the amount of shared information also increases, leading to improved performance.</p>
</div>
</section>
<section id="A8.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">H.2 </span>Influence of Walsh-Hadamard Transform</h3>

<div id="A8.SS2.p1" class="ltx_para">
<p id="A8.SS2.p1.1" class="ltx_p">We conduct the ablation study to profile the benefits of applying the Walsh-Hadamard Transform (WHT). Experiment results are reported at Table <a href="#A8.T9" title="Table 9 ‣ H.3 Automatic Rank Allocation vs. Uniform Rank Allocation ‣ Appendix H Ablation Study ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>. On the 3-bit quantization level, we observe that the Hadamard Transform only brings a slight amount of perplexity. However, when we quantize the low-rank representation more extremely (<span id="A8.SS2.p1.1.1" class="ltx_text ltx_font_italic">i.e.,</span> 2-bit), we can observe a notable 4.17 perplexity enhancements. It’s worth re-emphasizing that Hadamard Transform will not bring extra overhead during inference, as <span id="A8.SS2.p1.1.2" class="ltx_text ltx_font_italic">Palu</span> optimizes the WHT process via offline preprocessing. The reader may refer to Sec. <a href="#S3.SS4" title="3.4 Low-Rank-Aware Quantization ‣ 3 The Palu Framework ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.4</span></a> for more details.</p>
</div>
</section>
<section id="A8.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">H.3 </span>Automatic Rank Allocation vs. Uniform Rank Allocation</h3>

<div id="A8.SS3.p1" class="ltx_para">
<p id="A8.SS3.p1.1" class="ltx_p">Table <a href="#A8.T10" title="Table 10 ‣ H.3 Automatic Rank Allocation vs. Uniform Rank Allocation ‣ Appendix H Ablation Study ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">10</span></a> presents the ablation study on the impact of different rank allocation schemes on the model’s accuracy. Applying rank searching results in a notable performance improvement. For instance, at a compression rate of 50%, there is a significant reduction in perplexity by 2.18. Fig. <a href="#A8.F7" title="Figure 7 ‣ H.3 Automatic Rank Allocation vs. Uniform Rank Allocation ‣ Appendix H Ablation Study ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a> visualizes the rank allocation across different transformer blocks for key and value projection layers. The results clearly demonstrate a non-uniform allocation result. Specifically, we observe that the value is generally allocated a higher rank compared to the key. Additionally, the first half of the layers are assigned higher ranks, indicating their greater importance in preserving model performance. This visualization underscores the effectiveness of our rank search algorithm in identifying and allocating appropriate ranks to different components, thereby optimizing the balance between compression and accuracy.</p>
</div>
<figure id="A8.T9" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 9: </span>Ablation Study on different quantization settings for quantizing low-rank latent representations. Same as Sec. <a href="#S4.SS2" title="4.2 Integrating Palu with Quantization ‣ 4 Experiments ‣ Palu: Compressing KV-Cache with Low-Rank Projection" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.2</span></a>, we use the WikiText-2 with sequence length set to 4096 as the evaluation benchmark.
</figcaption>
<div id="A8.T9.5" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:180.0pt;height:193.7pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-27.1pt,29.2pt) scale(0.768635396159566,0.768635396159566) ;">
<table id="A8.T9.5.5" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A8.T9.1.1.1" class="ltx_tr">
<th id="A8.T9.1.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt">Method</th>
<td id="A8.T9.1.1.1.1" class="ltx_td ltx_nopad_l ltx_align_left ltx_border_tt">Wikitext-2 PPL <math id="A8.T9.1.1.1.1.m1.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="A8.T9.1.1.1.1.m1.1a"><mo stretchy="false" id="A8.T9.1.1.1.1.m1.1.1" xref="A8.T9.1.1.1.1.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="A8.T9.1.1.1.1.m1.1b"><ci id="A8.T9.1.1.1.1.m1.1.1.cmml" xref="A8.T9.1.1.1.1.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.1.1.1.1.m1.1c">\downarrow</annotation></semantics></math>
</td>
</tr>
<tr id="A8.T9.5.5.6.1" class="ltx_tr">
<th id="A8.T9.5.5.6.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Llama2-7B</th>
<td id="A8.T9.5.5.6.1.2" class="ltx_td ltx_nopad_l ltx_align_left ltx_border_t">5.12</td>
</tr>
<tr id="A8.T9.5.5.7.2" class="ltx_tr">
<th id="A8.T9.5.5.7.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Palu-30% (FP16)</th>
<td id="A8.T9.5.5.7.2.2" class="ltx_td ltx_nopad_l ltx_align_left ltx_border_t">5.25</td>
</tr>
<tr id="A8.T9.5.5.8.3" class="ltx_tr">
<th id="A8.T9.5.5.8.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">      + 3-bits w/o Hadamard</th>
<td id="A8.T9.5.5.8.3.2" class="ltx_td ltx_nopad_l ltx_align_left">5.52</td>
</tr>
<tr id="A8.T9.2.2.2" class="ltx_tr">
<th id="A8.T9.2.2.2.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">      + 3-bits w Hadamard</th>
<td id="A8.T9.2.2.2.1" class="ltx_td ltx_nopad_l ltx_align_left">5.33 (0.19<math id="A8.T9.2.2.2.1.m1.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="A8.T9.2.2.2.1.m1.1a"><mo stretchy="false" id="A8.T9.2.2.2.1.m1.1.1" xref="A8.T9.2.2.2.1.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="A8.T9.2.2.2.1.m1.1b"><ci id="A8.T9.2.2.2.1.m1.1.1.cmml" xref="A8.T9.2.2.2.1.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.2.2.2.1.m1.1c">\downarrow</annotation></semantics></math>)</td>
</tr>
<tr id="A8.T9.5.5.9.4" class="ltx_tr">
<th id="A8.T9.5.5.9.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" colspan="2">     <span class="ltx_rule" style="width:433.6pt;height:0.1pt;background:black;display:inline-block;"> </span>
</th>
</tr>
<tr id="A8.T9.5.5.10.5" class="ltx_tr">
<th id="A8.T9.5.5.10.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">      + 2-bits w/o Hadamard</th>
<td id="A8.T9.5.5.10.5.2" class="ltx_td ltx_nopad_l ltx_align_left">9.48</td>
</tr>
<tr id="A8.T9.3.3.3" class="ltx_tr">
<th id="A8.T9.3.3.3.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">      + 2-bits w Hadamard</th>
<td id="A8.T9.3.3.3.1" class="ltx_td ltx_nopad_l ltx_align_left">5.77 (3.71<math id="A8.T9.3.3.3.1.m1.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="A8.T9.3.3.3.1.m1.1a"><mo stretchy="false" id="A8.T9.3.3.3.1.m1.1.1" xref="A8.T9.3.3.3.1.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="A8.T9.3.3.3.1.m1.1b"><ci id="A8.T9.3.3.3.1.m1.1.1.cmml" xref="A8.T9.3.3.3.1.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.3.3.3.1.m1.1c">\downarrow</annotation></semantics></math>)</td>
</tr>
<tr id="A8.T9.5.5.11.6" class="ltx_tr">
<th id="A8.T9.5.5.11.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Palu-50% (FP16)</th>
<td id="A8.T9.5.5.11.6.2" class="ltx_td ltx_nopad_l ltx_align_left ltx_border_t">5.63</td>
</tr>
<tr id="A8.T9.5.5.12.7" class="ltx_tr">
<th id="A8.T9.5.5.12.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">      + 3-bits w/o Hadamard</th>
<td id="A8.T9.5.5.12.7.2" class="ltx_td ltx_nopad_l ltx_align_left">5.99</td>
</tr>
<tr id="A8.T9.4.4.4" class="ltx_tr">
<th id="A8.T9.4.4.4.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">      + 3-bits w Hadamard</th>
<td id="A8.T9.4.4.4.1" class="ltx_td ltx_nopad_l ltx_align_left">5.77 (0.22<math id="A8.T9.4.4.4.1.m1.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="A8.T9.4.4.4.1.m1.1a"><mo stretchy="false" id="A8.T9.4.4.4.1.m1.1.1" xref="A8.T9.4.4.4.1.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="A8.T9.4.4.4.1.m1.1b"><ci id="A8.T9.4.4.4.1.m1.1.1.cmml" xref="A8.T9.4.4.4.1.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.4.4.4.1.m1.1c">\downarrow</annotation></semantics></math>)</td>
</tr>
<tr id="A8.T9.5.5.13.8" class="ltx_tr">
<th id="A8.T9.5.5.13.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" colspan="2">     <span class="ltx_rule" style="width:433.6pt;height:0.1pt;background:black;display:inline-block;"> </span>
</th>
</tr>
<tr id="A8.T9.5.5.14.9" class="ltx_tr">
<th id="A8.T9.5.5.14.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">      + 2-bits w/o Hadamard</th>
<td id="A8.T9.5.5.14.9.2" class="ltx_td ltx_nopad_l ltx_align_left">10.58</td>
</tr>
<tr id="A8.T9.5.5.5" class="ltx_tr">
<th id="A8.T9.5.5.5.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">      + 2-bits w Hadamard</th>
<td id="A8.T9.5.5.5.1" class="ltx_td ltx_nopad_l ltx_align_left ltx_border_bb">6.41 (4.17<math id="A8.T9.5.5.5.1.m1.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="A8.T9.5.5.5.1.m1.1a"><mo stretchy="false" id="A8.T9.5.5.5.1.m1.1.1" xref="A8.T9.5.5.5.1.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="A8.T9.5.5.5.1.m1.1b"><ci id="A8.T9.5.5.5.1.m1.1.1.cmml" xref="A8.T9.5.5.5.1.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.5.5.5.1.m1.1c">\downarrow</annotation></semantics></math>)</td>
</tr>
</tbody>
</table>
</span></div>
</figure>
<figure id="A8.F7" class="ltx_figure"><img src="/html/2407.21118/assets/x7.png" id="A8.F7.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="217" height="116" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 7: </span>
Visualization of layer-wise low-rank compression rate on Llama2-7B with 50% of overall compression rate. Here, compression rates (<span id="A8.F7.2.1" class="ltx_text ltx_font_italic">i.e.,</span> rank) are allocated using the proposed Fisher Information-based automated rank allocation algorithm.
</figcaption>
</figure>
<figure id="A8.T10" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 10: </span>Ablation study on w/ and w/o rank search. We use Llama2-7b and Wikitext-2 with sequence length 2048 as the benchmark.
</figcaption>
<div id="A8.T10.3" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:200.0pt;height:37.9pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-42.4pt,8.0pt) scale(0.702484164626253,0.702484164626253) ;">
<table id="A8.T10.3.3" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A8.T10.3.3.4.1" class="ltx_tr">
<th id="A8.T10.3.3.4.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="A8.T10.3.3.4.1.2" class="ltx_td ltx_nopad_l ltx_align_left ltx_th ltx_th_column ltx_border_tt">Rate=30%</th>
<th id="A8.T10.3.3.4.1.3" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">Rate=50%</th>
<th id="A8.T10.3.3.4.1.4" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">Rate=70%</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A8.T10.3.3.5.1" class="ltx_tr">
<th id="A8.T10.3.3.5.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Uniform</th>
<td id="A8.T10.3.3.5.1.2" class="ltx_td ltx_nopad_l ltx_align_left ltx_border_t">6.34</td>
<td id="A8.T10.3.3.5.1.3" class="ltx_td ltx_align_left ltx_border_t">7.36</td>
<td id="A8.T10.3.3.5.1.4" class="ltx_td ltx_align_left ltx_border_t">10.77</td>
</tr>
<tr id="A8.T10.3.3.3" class="ltx_tr">
<th id="A8.T10.3.3.3.4" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">Automatic (ours)</th>
<td id="A8.T10.1.1.1.1" class="ltx_td ltx_nopad_l ltx_align_left ltx_border_bb">5.62 (0.72<math id="A8.T10.1.1.1.1.m1.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="A8.T10.1.1.1.1.m1.1a"><mo stretchy="false" id="A8.T10.1.1.1.1.m1.1.1" xref="A8.T10.1.1.1.1.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="A8.T10.1.1.1.1.m1.1b"><ci id="A8.T10.1.1.1.1.m1.1.1.cmml" xref="A8.T10.1.1.1.1.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="A8.T10.1.1.1.1.m1.1c">\downarrow</annotation></semantics></math>)</td>
<td id="A8.T10.2.2.2.2" class="ltx_td ltx_align_left ltx_border_bb">6.02 (1.36<math id="A8.T10.2.2.2.2.m1.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="A8.T10.2.2.2.2.m1.1a"><mo stretchy="false" id="A8.T10.2.2.2.2.m1.1.1" xref="A8.T10.2.2.2.2.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="A8.T10.2.2.2.2.m1.1b"><ci id="A8.T10.2.2.2.2.m1.1.1.cmml" xref="A8.T10.2.2.2.2.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="A8.T10.2.2.2.2.m1.1c">\downarrow</annotation></semantics></math>)</td>
<td id="A8.T10.3.3.3.3" class="ltx_td ltx_align_left ltx_border_bb">8.59 (2.18<math id="A8.T10.3.3.3.3.m1.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="A8.T10.3.3.3.3.m1.1a"><mo stretchy="false" id="A8.T10.3.3.3.3.m1.1.1" xref="A8.T10.3.3.3.3.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="A8.T10.3.3.3.3.m1.1b"><ci id="A8.T10.3.3.3.3.m1.1.1.cmml" xref="A8.T10.3.3.3.3.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="A8.T10.3.3.3.3.m1.1c">\downarrow</annotation></semantics></math>)</td>
</tr>
</tbody>
</table>
</span></div>
</figure>
</section>
</section>
<section id="A9" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix I </span>Experiment Details</h2>

<section id="A9.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">I.1 </span>Zero-Shot Evaluation Details</h3>

<div id="A9.SS1.p1" class="ltx_para">
<p id="A9.SS1.p1.1" class="ltx_p">We selected six zero-shot tasks from the LM-eval benchmark to evaluate <span id="A9.SS1.p1.1.1" class="ltx_text ltx_font_italic">Palu</span>:</p>
<ul id="A9.I1" class="ltx_itemize">
<li id="A9.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A9.I1.i1.p1" class="ltx_para">
<p id="A9.I1.i1.p1.1" class="ltx_p">OpenBookQA (accuracy, <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib30" title="" class="ltx_ref">Mihaylov et al.</a></cite>)</p>
</div>
</li>
<li id="A9.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A9.I1.i2.p1" class="ltx_para">
<p id="A9.I1.i2.p1.1" class="ltx_p">HellaSwag (acc_norm, <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib46" title="" class="ltx_ref">Zellers et al.</a></cite>)</p>
</div>
</li>
<li id="A9.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A9.I1.i3.p1" class="ltx_para">
<p id="A9.I1.i3.p1.1" class="ltx_p">PIQA (accuracy, <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib4" title="" class="ltx_ref">Bisk et al.</a></cite>)</p>
</div>
</li>
<li id="A9.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A9.I1.i4.p1" class="ltx_para">
<p id="A9.I1.i4.p1.1" class="ltx_p">ARC-Easy (accuracy, <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib6" title="" class="ltx_ref">Clark et al.</a></cite>)</p>
</div>
</li>
<li id="A9.I1.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A9.I1.i5.p1" class="ltx_para">
<p id="A9.I1.i5.p1.1" class="ltx_p">ARC-Challenge (acc_norm, <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib6" title="" class="ltx_ref">Clark et al.</a></cite>)</p>
</div>
</li>
<li id="A9.I1.i6" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A9.I1.i6.p1" class="ltx_para">
<p id="A9.I1.i6.p1.1" class="ltx_p">WinoGrande (accuracy, <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib34" title="" class="ltx_ref">Sakaguchi et al.</a></cite>)</p>
</div>
</li>
</ul>
<p id="A9.SS1.p1.2" class="ltx_p">We report accuracy for WinoGrande, PIQA, and ARC-Easy, and accuracy normalized by sequence length (acc_norm) for HellaSwag and ARC-Challenge.</p>
</div>
</section>
<section id="A9.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">I.2 </span>LongBench Evaluation Details</h3>

<div id="A9.SS2.p1" class="ltx_para">
<p id="A9.SS2.p1.1" class="ltx_p">For the LongBench evaluation in the manuscript, we selected eight tasks from four subgroups, ensuring a comprehensive evaluation of <span id="A9.SS2.p1.1.1" class="ltx_text ltx_font_italic">Palu</span>. The tasks and their corresponding metrics are detailed below:</p>
</div>
<div id="A9.SS2.p2" class="ltx_para">
<ul id="A9.I2" class="ltx_itemize">
<li id="A9.I2.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A9.I2.i1.p1" class="ltx_para">
<p id="A9.I2.i1.p1.1" class="ltx_p">Single-Document QA:</p>
<ul id="A9.I2.i1.I1" class="ltx_itemize">
<li id="A9.I2.i1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><span id="A9.I2.i1.I1.i1.1.1.1" class="ltx_text ltx_font_bold">–</span></span> 
<div id="A9.I2.i1.I1.i1.p1" class="ltx_para">
<p id="A9.I2.i1.I1.i1.p1.1" class="ltx_p">Qasper (F1 score, <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib8" title="" class="ltx_ref">Dasigi et al.</a></cite>)</p>
</div>
</li>
</ul>
</div>
</li>
<li id="A9.I2.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A9.I2.i2.p1" class="ltx_para">
<p id="A9.I2.i2.p1.1" class="ltx_p">Summarization:</p>
<ul id="A9.I2.i2.I1" class="ltx_itemize">
<li id="A9.I2.i2.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><span id="A9.I2.i2.I1.i1.1.1.1" class="ltx_text ltx_font_bold">–</span></span> 
<div id="A9.I2.i2.I1.i1.p1" class="ltx_para">
<p id="A9.I2.i2.I1.i1.p1.1" class="ltx_p">QMSum (ROUGE score, <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib49" title="" class="ltx_ref">Zhong et al.</a></cite>)</p>
</div>
</li>
<li id="A9.I2.i2.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><span id="A9.I2.i2.I1.i2.1.1.1" class="ltx_text ltx_font_bold">–</span></span> 
<div id="A9.I2.i2.I1.i2.p1" class="ltx_para">
<p id="A9.I2.i2.I1.i2.p1.1" class="ltx_p">MultiNews (ROUGE score, <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib11" title="" class="ltx_ref">Fabbri et al.</a></cite>)</p>
</div>
</li>
</ul>
</div>
</li>
<li id="A9.I2.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A9.I2.i3.p1" class="ltx_para">
<p id="A9.I2.i3.p1.1" class="ltx_p">Few-shot Learning:</p>
<ul id="A9.I2.i3.I1" class="ltx_itemize">
<li id="A9.I2.i3.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><span id="A9.I2.i3.I1.i1.1.1.1" class="ltx_text ltx_font_bold">–</span></span> 
<div id="A9.I2.i3.I1.i1.p1" class="ltx_para">
<p id="A9.I2.i3.I1.i1.p1.1" class="ltx_p">TREC (classification score, <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib23" title="" class="ltx_ref">Li and Roth</a></cite>)</p>
</div>
</li>
<li id="A9.I2.i3.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><span id="A9.I2.i3.I1.i2.1.1.1" class="ltx_text ltx_font_bold">–</span></span> 
<div id="A9.I2.i3.I1.i2.p1" class="ltx_para">
<p id="A9.I2.i3.I1.i2.p1.1" class="ltx_p">TriviaQA (F1 score, <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib21" title="" class="ltx_ref">Joshi et al.</a></cite>)</p>
</div>
</li>
<li id="A9.I2.i3.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><span id="A9.I2.i3.I1.i3.1.1.1" class="ltx_text ltx_font_bold">–</span></span> 
<div id="A9.I2.i3.I1.i3.p1" class="ltx_para">
<p id="A9.I2.i3.I1.i3.p1.1" class="ltx_p">SAMSum (ROUGE score, <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib14" title="" class="ltx_ref">Gliwa et al.</a></cite>)</p>
</div>
</li>
</ul>
</div>
</li>
<li id="A9.I2.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A9.I2.i4.p1" class="ltx_para">
<p id="A9.I2.i4.p1.1" class="ltx_p">Code Completion:</p>
<ul id="A9.I2.i4.I1" class="ltx_itemize">
<li id="A9.I2.i4.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><span id="A9.I2.i4.I1.i1.1.1.1" class="ltx_text ltx_font_bold">–</span></span> 
<div id="A9.I2.i4.I1.i1.p1" class="ltx_para">
<p id="A9.I2.i4.I1.i1.p1.1" class="ltx_p">LCC (similarity score, <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib15" title="" class="ltx_ref">Guo et al.</a></cite>)</p>
</div>
</li>
<li id="A9.I2.i4.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"><span id="A9.I2.i4.I1.i2.1.1.1" class="ltx_text ltx_font_bold">–</span></span> 
<div id="A9.I2.i4.I1.i2.p1" class="ltx_para">
<p id="A9.I2.i4.I1.i2.p1.1" class="ltx_p">RepoBench-P (similarity score, <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib25" title="" class="ltx_ref">Liu et al.</a></cite>)</p>
</div>
</li>
</ul>
</div>
</li>
</ul>
<p id="A9.SS2.p2.1" class="ltx_p">During evaluation, we set the maximum sequence length to 8192 for the Mistral model and 15500 for Vicuna.</p>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2407.21117" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2407.21118" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2407.21118">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2407.21118" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2407.21119" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Mon Aug  5 13:10:25 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2407.21118
KV-Cache compression methods generally sample a KV-Cache of effectual tokens or quantize it into lower bits.
However, these methods cannot exploit the redundancy of the hidden dimension of KV tensors.
This paper investigates a unique hidden dimension approach calledPalu, a novel KV-Cache compression framework that utilizes low-rank projection. Palu decomposes the linear layers into low-rank matrices, caches the smaller intermediate states, and reconstructs the full keys and values on the fly. To improve accuracy, compression rate, and efficiency,Palufurther encompasses (1)  a medium-grained low-rank decomposition scheme, (2)  an efficient rank search algorithm, (3)  a low-rank-aware quantization algorithm, and (4)  matrix fusion with optimized GPU kernels.
Our extensive experiments with popular LLMs show thatPalucan compress KV-Cache by more than91.25%while maintaining a significantlybetter accuracy (up to 1.19 lower perplexity) than state-of-the-art KV-Cache quantization methods at a similar or even higher memory usage. When compressing KV-Cache for 50%,Paludelivers up to1.61end-to-end speedupfor the attention module. Our code is publicly available at:https://github.com/shadowpa0327/Palu.
abstract
Palu: Compressing KV-Cache with Low-Rank Projection
Abstract
2407.21118
1IntroductionLarge language models (LLMs)  are revolutionizing the AI industry and providing a high-level intelligence that previous machine learning (ML)  models could not achieve.
To speed inference, caching key-value states (KV-Cache)  in memory is a simple yet effective technique. However, the size of the KV-Cache can grow rapidly, straining memory capacity and bandwidth; further, the memory-bounded characteristics of the decoding stage also limit inference speed when loading KV-Cache data. Therefore, KV-Cache compression has become a central research topic for building efficient LLMs.Figure 1:Palu’slow-rank projection methodfor KV-Cache reduction. A weight matrixWof linear projection is decomposed into two low-rank matrices. InputXis down-projected to latent representationH, which is cached.Yis reconstructed fromHon-the-fly using the up-projection matrixB.Research on KV-Cache compression falls into two categories: quantization and token eviction. Quantization methods aim to use less bit-width to represent each datum; token eviction techniques seek ways to retain only a partial set of KV-Cache. However, both categories fail to explore the hidden dimensions of KV-Cache where high redundancy often occurs.This work examines the problem from a fundamentally different perspective, i.e., ways to reduce the hidden dimensions. We introducePalu, a KV-Cache compression framework that reduces the size of the KV-Cache throughlow-rank projection. Although low-rank projection has been shown to improve LLM efficiency, e.g., by reducing model sizeor minimizing memory footprint for finetuning, it has not been studied for compressing the KV-Cache, especially in post-training.A naive way to compress the KV-Cache with low-rank projection is by directly mapping cached matrices into low-rank space. However, this approach imposes an unacceptably heavy overhead of computing the decomposition matrices during runtime, after the prefill stage. To avoid this,Palustatically decomposes the key and value-projection weight matricesandcaches the latent representations of the low-rank decomposition(Fig.1) , instead of directly decomposing the KV-Cache. This innovative design enablesPaluto reduce memory while substantially mitigating runtime overhead.Despite its promising memory reduction, we found that naively decomposing the key and value projection matrices along the head dimension causes a substantial loss in accuracy. Although we found that decomposing the entire matrix at one time can maintain accuracy, it significantly increases reconstruction overhead.
To find a balance,Paluproposes amedium-grained, group-head low-rank decomposition optimizationthat is both accurate and less costly in terms of reconstruction.For LLMs, each linear projection module has a different sensitivity to compression. To exploit the sensitivity and improve accuracy, we design an efficientrank search algorithmbased on Fisher information. Our algorithm automatically assigns a larger rank for important matrices and smaller ranks for less critical ones, boosting accuracy at the same compression rate.In addition to low-rank decomposition, we integrate quantization intoPalu. We found that low-rank decomposition often induces severe outliers in the latent representation. Inspired by recent LLM quantization methods, we use Hadamard transformation matrices with alow-rank-aware quantization algorithmto eliminate outliers and increase quantization accuracy. To avoid runtime overhead,Palufuses the transformation matrices into the forward and backward decomposed matrices, concealing the overhead completely.To enhance inference latency, we integrate the decomposed matrix into the subsequent linear layer, thereby mitigating the online reconstruction overhead.
In scenarios where RoPE is applied and matrix fusion is infeasible, we developed an optimized GPU kernel to accelerate the reconstruction.We evaluatePaluon popular LLMs and benchmarks. Our experiments show thatPalumaintains strong zero-shot accuracy and perplexity with up to 50% low-rank compression. Combining low-rank compression and quantization,Palucompresses KV-Cache byover 91.25% (11.4compression) while maintaining perplexity. Compared to a state-of-the-art KV-Cache quantizationwith 87.5% compression,Paluachieves asignificantly lower perplexity of 1.19. Additionally, with 50% compression without quantization,Paludemonstrates a1.61end-to-end speedup, highlightingPalu’s ability to reduce memory usage and enhance speed for LLMs.Our key contributions include:•Palu, a new post-training KV-Cache compression framework that cacheslow-rank latent representations.•Group-head low-rank decomposition (G-LRD) , an optimization for balancing accuracy and efficiency.•Anautomated rank searchalgorithmfor adaptively assigning ranks to each decomposed matrix, given a target compression rate.•Alow-rank-aware quantization algorithmthat eliminates low-rank-induced outliers and imposes zero runtime overhead.
introduction
Palu: Compressing KV-Cache with Low-Rank Projection
1Introduction
2407.21118
2Background2.1Multi-Head Attention MechanismThe multi-head attention (MHA)  mechanismis a core component of the transformer architecture. Given a new input token, an MHA withheads projects the input into multiple queries, keys, and values using weight matrices,, and, respectively, for each head, as shown by(1) Here,andrepresent the key and value at time stepfor head. We can then compute the attention score for each headand the corresponding attention output as(2) whereanddenote the concatenation of current and all previous keys and values corresponding to the-th head.
The final MHA output is obtained by concatenating the outputs of all heads and then applying the out-projection layer, as shown by(3) whererepresents the submatrices of the out-projection matrix for each head.2.2Singular Value Decomposition (SVD) SVDis a commonly used technique for computing the low-rank approximation for a given matrix.
We now introduce our specific use of SVD forPalu’s default low-rank decomposition method.Given a weight matrix, SVD decomposesinto three matrices:. Here,andare orthogonal matrices containing the left and right singular vectors, respectively. The matrixis a diagonal matrix that consists of singular values.
We describe the decomposition aswhere,,.is a diagonal matrix containing the largestsingular values, and,are corresponding singular vectors truncated fromand. This truncation and subsequent matrix formation let us approximate weight matrixwith two low-rank matricesand, thereby reducing the storage by.Figure 2:Paluuses low-rank decompositionto project the key (or value)  to a lower-dimensional latent representation, thereby reducing the size of the KV-Cache.
The original keyis reconstructed on-the-fly with, andis fused intoto avoid reconstruction overhead. The fusion also reduces the computational burden for output projection.
body
Palu: Compressing KV-Cache with Low-Rank Projection
2Background
2407.21118
3The Palu Framework3.1Compressing the KV-Cache via Low-Rank ProjectionAs noted, a naive approach to compress the KV-Cache with
low-rank projection is to apply SVD directly on the KV-Cache and store the top-singular vectors.
However, this approach poses significant computational challenges during runtime that make it impractical for deployments (see Section1) .To apply low-rank projection more efficiently,Paluuses SVD to decompose the key and value projection weight matrices.
This approach is based on the observation that low-rank decomposition rewrites the linear projection layer frominto.
Here,is the low-rank projection matrix, andis the reconstruction matrix derived by SVD.
The forward process firstdown-projectsthe input tokeninto a low-dimensional latent spaceand thenup-projectsit back to the original space:(4) This two-step process letsPalu(1)  store the lower dimension latent representation instead of the origin keys and values states, and (2)  reconstruct them during decoding.3.1.1Integration with the Attention Mechanism and Offline Matrix FusionWe now describe howPaludecomposes the key and value linear layers for the attention mechanism.
For each attention head,Paluapplies SVD and maps the key-projection matrixand value-projection matrixintoand.Based on the formula of attention output in equation2,Paluabsorbs the reconstruction matrixinto the output projection matrixoffline:(5) Such fusion letsPaluskip the explicit reconstruction of the full value vectors, reduce the number of matrix multiplications, and improve efficiency.
A similar approach applies for calculating attention scores.
Matrixcan be fused into the query projection matrixoffline, as shown by(6) Here,and, so the fused matrixhas size. This fusion boosts computational efficiency by reducing the matrix dimension during attention score calculation.3.1.2Mechanism with Positional EmbeddingRecent LLMs, such as the Llama family, apply positional embedding (i.e.,RoPE)  onto the query and key states before multiplication.
The non-linear operations of adding positional embedding prevent the fusion for the attention score, as described in equation6.
In this case,Palureconstructs the keys from latent representations on-the-fly during decoding.
We improve reconstruction efficiency by designing an optimized kernel, which we present in Sec.3.5.Note that for some positional embedding methods, such as ALiBi, or new attention mechanisms, such as MLA, positional embedding is not directly applied on the key states; therefore, the fusion of equation6remains valid.
We expect the end-to-end speedup to be larger on these models.
Fig.2presents a detailed workflow ofPalubased on Llama’s attention with RoPE.3.2Decomposition Granularity3.2.1Multi-Head Low-Rank DecompositionWe name the per-head decomposition scheme in Sec.3.1.1asmulti-head low-rank decomposition (M-LRD) .
We found M-LRD often causes a non-negligible accuracy degradation (discussed further in Sec.4.1) , possibly because SVD fails to capture the common information shared across heads.
Therefore, alternative approaches are needed for preserving model accuracy.3.2.2Joint-Head Low-Rank DecompositionAn alternative approach is to jointly decompose weight matrices for all heads.
By considering the combined weight matrix,
we can perform a single low-rank decomposition,
whereand.
We call this schemejoint-head low-rank decomposition (J-LRD) .J-LRD has the advantage of preserving the common principal components shared among different heads.
This occurs because SVD is particularly effective at capturing the dominant components when applied to a larger, combined matrix, resulting in a more accurate approximation.For J-LRD, the joint latent representation shared among all heads can be computed with.
During decoding, the original states for each head can be reconstructed viaHigh Inference Overhead.Despite better preserving model information, J-LRD introducessignificant computational and memory overheadduring decoding. Specifically, the total number of floating point operations (FLOPs)  to reconstruct the key or values state of one head now becomes. Assuming the same size as the total low-rank latent representations (i.e.,) ,
the total reconstruction cost istimes higher than M-LRD, whose total FLOPs is.
When considering the matrix fusion in Sec.3.1.1,
the fused matrix size of J-LRD has a size of, which is alsotimes larger than M-LRD, leading to a substantial higher memory consumption.Figure 3:Performing decomposition at different granularities. Jointly decomposing multiple heads can achieve higher accuracy. Assuming the same total size of the latent representations (i.e.,) , the FLOPs for reconstruction overhead in joint-head decomposition schemes are 4 times larger than those in multi-head ones.3.2.3Group-Head Low-Rank DecompositionTo balance the trade-off between accuracy and reconstruction cost, we proposegroup-head low-rank decomposition(G-LRD) .
G-LRD decomposes the matrices for a group of heads together.
With combined weight matrices, it captures shared information within each group while limiting computational overhead and preserving accuracy.To illustrate the G-LRD process, consider the weight matrices for a group ofheads,, where.
We low-rank decompose,
whereand.
The latent representation shared among attention heads in the same group can be computed as.
During decoding, the original key or value for each head can be reconstructed viaThe FLOPs for reconstructing the keys and values for each head in G-LRD is,
whereis the number of groups.
Comparing the cost to J-LRD and assuming the same total rank size () , G-LRD reduces the reconstruction cost by.
Similarly, G-LRD also reduces the fused matrix size by.
To sum up, G-LRD offers a middle ground between computation overhead and approximation accuracy.
We illustrate M-LRD, J-LRD and G-LRD in Fig.3.For further discussion on decomposed weight matrix size, please refer to AppendixB, where more detailed information is provided.3.3Automatic Rank AllocationTo allocate an ideal rank size to the decomposition target, it is crucial to accurately estimate the importance of the target matrix (e.g.,grouped weights) . InPalu, we identifyFisher informationas an accurate approximator since it can quantify the amount of information for each parameter.
We then employ the sum of Fisher information to estimate the importance of the weight matrix of each linear layer.Assuming that the compression sensitivity is proportional to Fisher information, we determine the rank for each weight matrix by computing the ratio of its Fisher information to the total Fisher information across all decomposition targets.
We use this ratio to allocate the compression rate (i.e.,rank level) , ensuring that more important layers retain higher rank levels.For a detailed ablation study on our Automatic Rank Allocation method, please refer to AppendixH.3, where we analyze the impact of our strategy on model performance.3.4Low-Rank-Aware QuantizationFigure 4:Activation distribution of the low-rank compressed key caches at theLlama-2 attention layer.We integrate quantization intoPaluto further compress the KV-Cache. However, we observe that low-rank compressed latent representations have severe outliers, which limit quantization applicability inPalu. Unlike natural outliers described in previous KV-Cache quantization literature, these outliers are induced by SVD-based low-rank factorization.Fig.4(a)  shows the distribution of low-rank compressed key states from a layer of Llama-2 with G-LRD. Repeating outlier patterns appear at the beginning of each decomposed group because SVD arranges larger eigenvalues in the initial rows or columns, resulting in rapidly descending values in the latent representation. This pattern stretches the data distribution and hurts quantization accuracy.Inspired by recent LLM quantization literature, we apply the Walsh-Hadamard transform (WHT,)  to eliminate outliers (Fig.4(b) ) ,
enabling a high quantization accuracy.
However, this transformation introduces an extra matrix multiplication with associated runtime overhead.
Unlike earlier methodsthat must apply online WHT when quantizing KV-Cache, we optimize this process by integrating the Hadamard matrix into low-rank decomposed weights with no additional overhead, as described by(7) whereis the Hadamard matrix.
This optimization allowsPalucan integrate the proposed low-rank compression technique with low-bit quantization.
Our experiments show that, on top of the low-rank compression, our quantization method onlynegligibly increases perplexity, even at extreme levels such as3-bit or 2-bit(see Sec.4.2) .3.5Efficient Inference ImplementationSupport for RoPE-based Attention.To support RoPE-based attention, we fuse the reconstruction of original keys, applying positional embedding (i.e., RoPE)  and the final matrix-vector multiplication of query and reconstructed keys by implementing customized GPU kernels in Triton. We leverage the tiling techniques akin to FlashAttentionto minimize the memory reads/writes of GPU high bandwidth memory.Table 1:Perplexity and zero-shot accuracy ofPalu, with different decomposition strategies at 50% compression rate.ModelMethodPerplexityZero-Shot Accuracy (%) Wiki2C4OBQAHellaPIQAARC-eARC-cWinoAvg.Llama2-7BBaseline5.477.2644.2076.0078.0776.3046.4269.3065.05J-LRD5.627.7545.4075.5777.4875.9745.3169.2264.82G-LRD6.019.8243.6073.3976.3373.0242.5766.7762.61M-LRD6.7512.0139.6065.3574.7667.1735.2464.6457.79Llama2-13BBaseline4.886.7045.2079.3979.1179.4249.0672.3867.43J-LRD4.976.9246.4079.4878.6279.2949.9170.5667.38G-LRD5.317.7645.6077.2977.4276.0545.9972.4565.80M-LRD5.658.3443.2074.3477.5375.7645.3968.9864.20
body
Palu: Compressing KV-Cache with Low-Rank Projection
3The Palu Framework
2407.21118
4ExperimentsTable 2:Accuracy ofPaluwith 30% and 50% compression rate on 8 tasks in LongBench.ModelMethodSAMSumLCCM-NewsQasperTRECQmSumR-BenchTriQAAvg.Vicuna-7BBaseline40.7349.2027.9825.5769.0022.5443.5086.4245.62Palu-30%40.4249.0026.9618.5465.5021.7845.7785.2244.15Palu-50%38.3634.5924.7819.7763.5021.4435.4279.7439.70Mistral-7BBaseline41.7166.5920.268.1367.0019.9959.1089.9046.59Palu-30%43.2364.2323.5010.2172.5021.1656.8089.8247.68Palu-50%36.4350.2219.237.5467.5019.5644.7685.3741.33Models and Tasks.We evaluatePaluon three LLM families, Llama2, Mistraland Vicuna.
For accuracy evaluation, we measure perplexity on
the WikiText-2and C4datasets and use LM-Evaluation-Harnessto measure zero-shot accuracy on six common sense tasks.
We also evaluate long context accuracy on 8 tasks in LongBench.
See AppendixIfor further details on the dataset and settings.Compression Settings.We implementedPaluusing the Huggingface library. Decomposition of the key and value projection layers was performed using the truncation-aware SVD method proposed by SVD-LLM. Unless otherwise specified, results forPaluare reported using G-LRD with a group size (gs)  of 4 with equal rank size for each group. To calculate Fisher information in rank searching, we used 2048 samples from Wikitext-2, each with a sequence length of 1024. For quantization, we aligned our settings with Atom, and use a simple per-token, asymmetric quantization to quantize the low-rank latent representation of both key and value states.4.1Main ResultsPerplexity Evaluation.We first present perplexity evaluation results on Wikitext-2 and C4 in Table1. For the Llama2-7B model, we observe that the M-LRD method fails to maintain low perplexity at a 50% compression rate. In contrast, the J-LRD method performs significantly better given the same compression rate, achieving perplexity values of 5.62 on Wikitext-2. The G-LRD method, with a group size of 4, achieves an optimal balance between perplexity and computational cost, yielding perplexity values of 6.01 on Wikitext-2 at a 50% compression rate. The same trend is consistent across the Llama2-13B model.Zero-shot Evaluation Results.We further evaluatePaluon six common-sense zero-shot tasks, as shown in Table1. Similar to the perplexity evaluation, the J-LRD method demonstrates the best performance on Llama2-7B, with only a 0.23% average accuracy degradation, while the M-LRD method results in the lowest average performance, with a 7.26% drop in accuracy compared to the baseline. In comparison, G-LRD experiences only a 2.4% average accuracy decline.LongBench Results.To assess whetherPalucan effectively compress the KV-Cache for long-context scenarios, we present the performance ofPaluacross eight tasks from LongBench.
We select Mistral-7B and Vicuna-7Bas our evaluation targets, which support sequences up to 8k and 16k, respectively. Table2reports experiment results. We find that at a 50% compression level, it is relatively difficult to fully preserve accuracy.
However, when compressing KV-Cache for 30%,Paluonly has a 1.47% averaged accuracy degradation for Vicuna-7B, and even achieves a 1.09% enhancement for Mistral-7B. These results evidence ofPalu’s ability to shrink the KV-Cache while maintaining performance.For detailed information on the integration ofPaluwith quantization and the performance measurements on LongBench, please refer to AppendixE.4.2IntegratingPaluwith QuantizationTable3shows the results of integrating quantization withPalu. With 3-bit quantization,Paluexhibits only a negligible perplexity increase of 0.08 at a 30% compression rate and a minimal 0.23 at a 50% compression rate, both compared toPaluat 16-bit. This indicates a minimal trade-off for substantial compression of integrating quantization.
Furthermore, at 2-bit quantization,Palusignificantly outperforms KVQuant, which is a SOTA quantization-based method.Paludemonstrates a compelling reduction in perplexity by 1.19 and 0.54 points, while consuming 30% and 50% less memory, respectively. These results underscore the effectiveness ofPalu, establishing it as a superior KV-Cache compression strategy.4.3Latency EvaluationWe now provide the latency evaluation forPalu. In our experiment, latency is measured on an A6000 GPU, and the averaged results over 100 runs are reported.
For baseline, we use the default implementation from HuggingFace in PyTorch.
We do not apply FlashAttentionfor an apple-to-apple comparison, as our current implementation does not fuse the Softmax operation.
We leave the integration with FlashAttention as the future work.GEMV with Online Reconstruction.We evaluate the performance of our fused reconstruction kernel (Sec.3.5)  for computing RoPE-based attention score.
Specifically, we measure the latency frompre-RoPE query vector to the post-GEMV attention score(refer to Fig.2for details) .
Here, we set the compression rate for our kernel as 75%.
It’s because we observe the key states often achieve such compression rate when setting a 50% overall compression rate (see AppendixH.3for detailed distribution) .We comparePalu’s runtime against the baseline, which is simply a GEMV, and report the speedup in Fig.5(a) . For group size 32 (J-LRD) , although it’s the most accurate decomposition, the high reconstruction leads to a significant 7.09slowdown at sequence length 64K .
For group size 1 (M-LRD) , our fused kernel achieves up to a 2.48speedup at a sequence length 16k, showcasing a high speedup when more accuracy loss is acceptable.
For group size 2 and 4 (G-LRD) , our fused kernel demonstrates 1.75and 1.16speedup at sequence length 16K.
These results highlight the importance of exploring trade-off between decomposition granularity, reconstruction cost, and accuracy.
Furthermore, we observe our kernel speedup decreases for sequence lengths over 16k, due to the increasing reconstruction cost shifts the attention operation from memory- to compute-bounded.Table 3:Perplexity and KV-Cache size comparison ofPaluand other quantization-based methods for Llama2-7B on WikiText-2. For perplexity, sequence length is set to 4096, aligning with baselines. KV-Cache size is demonstrated for a sequence length of 128K.MethodBitPPLKV-CacheSize (GB) Comp.RateBaseline165.1264.0-Palu-30%165.2544.830%Palu-50%165.6332.050%Atom36.1512.680.32%KVQuant35.3512.081.25%Palu-30%35.338.486.87%Palu-50%35.776.090.63%Atom2117.888.686.56%KVQuant26.958.087.50%Palu-30%25.765.691.25%Palu-50%26.414.093.75%End-to-end attention module.We measure the end-to-end latency of the entire RoPE-based attention module, with the integration of our fused reconstruction kernel for key states (Sec.3.5)  and offline matrix fusion for value states (Eq.5) .
Here, we set the compression rate for key and value as 75% and 25%, for an overall 50% compression rate.
The detailed forward pass operations are presented in Fig.2.
We measure the decoding runtime for sequence length from 4K to 64K, and report the runtime ofPaluand baseline in Fig.5(b) .
As Fig.5(b) ,Paluwith smaller group sizes consistently delivers more than 1.43speedup across all sequence lengths.
For group size 1 (M-LRD) ,Paluachieves a 1.98, close to optimal (for 50% compression rate) , speedup for sequence length 16K.
For group size 4 (G-LRD) , which is the sweet spot for maintaining strong accuracy and limiting reconstruction cost,Paluachieves up to1.61end-to-end speedupat sequence length 64K.We observe increasing speedup as sequence length increases from 4k to 64k due to continuously reducing latency from compressing value states. However, speedup is expected to converge or slightly decrease with further length increases due to higher reconstruction costs and the compute-bound nature of reconstruction kernels.
We expect the speedup ofPaluwill be larger for non-RoPE-based attention, as the reconstruction overhead can be entirely eliminated by applying Eq.6.For the end-to-end model latency, please refer to AppendixD, where more detailed information and data are provided.Figure 5:Normalized speedup for (a)  reconstruction kernel at 75% compression and (b)  RoPE-based attention module with an overall 50% compression under different group sizes.
body
Palu: Compressing KV-Cache with Low-Rank Projection
4Experiments
2407.21118
5Related WorkSVD for LLM Compression.There are several related works applying SVD for LLM compression.
An early worksimply compresses the weight matrices using standard SVD, resulting in high compression errors.
To address this, FWSVDincorporates Fisher information to prioritize parameters. Additionally, ASVDtaken the activation outlier phenomenon in LLM into consideration, and propose and scaling techniques to mitigate the compression error. SVD-LLMfurther advances this work by understanding the compression loss from each singular value, improving compression accuracy. Unlike previous related SVD works that focus on compressing model weights,Palufocus on using SVD to reduce KV-Cache size.KV-Cache Quantization.Quantization appears to be an effective compression technique to compress KV-Cache.
Atomuses a simple per-token quantization scheme to compress KV-Cache.
WKVQuantproposes a two-level quantization scheme to improve quantization accuracy.
KIVIutilizes different quantization schemes, per-channel and per-token, for key and value states respectively.
KVQuantfollow a similar setting but adopts non-uniform quantization and sparse matrix for preserving outliers.
InPalu, we eliminate outliers with WHT to enhance quantization friendliness and, therefore, only need a simple per-token quantization and achieve a remarkable accuracy.
body
Palu: Compressing KV-Cache with Low-Rank Projection
5Related Work
2407.21118
6ConclusionWe introducePalu, a novel KV-Cache compression framework that decomposes key and value linear projection weight matrices and caches the compressed latent representations. We propose various optimizations, including group-head low-rank decomposition, rank search algorithm, low-rank-aware quantization, matrix fusion, and kernel fusion. With these optimizations,Palucan maintain accuracy while achieving significant memory reduction and faster inference. Experiments show thatPalucan accelerate the end-to-end attention module by up to 1.61with 50% KV-Cache compression; when combined with 2-bit quantization,Paluachieves a remarkable 91.25% compression rate, both maintaining strong accuracy.
body
Palu: Compressing KV-Cache with Low-Rank Projection
6Conclusion
2407.21118
Limitations and Future WorkWe introduce a novel KV-Cache compression framework based on low-rank projection. However, our experiments cover model variants with up to only 13B parameters due to limited computational resources. Future work needs to establish howPaluperforms for models with larger sizes (e.g., 70B or beyond) . Also, we leave for future work the evaluation of end-to-end speedup when combining low-rank compression and quantization. It would also be interesting to investigate how to combinePaluwith other efficient LLM methods, such as token eviction or weight quantization, to achieve the highest possible acceleration.
body
Palu: Compressing KV-Cache with Low-Rank Projection
Limitations and Future Work
2407.21118
AcknowledgementWe would like to thank Sandy Kaplan for her valuable feedback and support in refining the writing of this paper. Her insights and suggestions significantly contributed to the clarity and quality of the final manuscript.
conclusion
Palu: Compressing KV-Cache with Low-Rank Projection
Acknowledgement