<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2405.12981] Reducing Transformer Key-Value Cache Size with Cross-Layer Attention</title><meta property="og:description" content="Key-value (KV) caching plays an essential role in accelerating decoding for transformer-based autoregressive large language models (LLMs). However, the amount of memory required to store the KV cache can become prohibi…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Reducing Transformer Key-Value Cache Size with Cross-Layer Attention">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Reducing Transformer Key-Value Cache Size with Cross-Layer Attention">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2405.12981">

<!--Generated on Wed Jun  5 13:37:24 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">Reducing Transformer Key-Value Cache Size with Cross-Layer Attention</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">William Brandon 
<br class="ltx_break">MIT CSAIL
<br class="ltx_break"><span id="id3.2.id1" class="ltx_text ltx_font_typewriter">wbrandon@csail.mit.edu</span> 
<br class="ltx_break">&amp;Mayank Mishra<sup id="id4.3.id2" class="ltx_sup">∗</sup> 
<br class="ltx_break">MIT-IBM Watson AI Lab 
<br class="ltx_break">&amp;Aniruddha Nrusimha 
<br class="ltx_break">MIT CSAIL 
<br class="ltx_break">&amp;Rameswar Panda 
<br class="ltx_break">MIT-IBM Watson AI Lab 
<br class="ltx_break">&amp;Jonathan Ragan-Kelley 
<br class="ltx_break">MIT CSAIL
</span><span class="ltx_author_notes">Equal Contribution</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id2.1" class="ltx_p">Key-value (KV) caching plays an essential role in accelerating decoding for transformer-based autoregressive large language models (LLMs). However, the amount of memory required to store the KV cache can become prohibitive at long sequence lengths and large batch sizes. Since the invention of the transformer, two of the most effective interventions discovered for reducing the size of the KV cache have been Multi-Query Attention (MQA) and its generalization, Grouped-Query Attention (GQA). MQA and GQA both modify the design of the attention block so that multiple query heads can share a single key/value head, reducing the number of distinct key/value heads by a large factor while only minimally degrading accuracy. In this paper, we show that it is possible to take Multi-Query Attention a step further by also sharing key and value heads between adjacent layers, yielding a new attention design we call Cross-Layer Attention (CLA). With CLA, we find that it is possible to reduce the size of the KV cache by another <math id="id2.1.m1.1" class="ltx_math_unparsed" alttext="2\times" display="inline"><semantics id="id2.1.m1.1a"><mrow id="id2.1.m1.1b"><mn id="id2.1.m1.1.1">2</mn><mo lspace="0.222em" id="id2.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="id2.1.m1.1c">2\times</annotation></semantics></math> while maintaining nearly the same accuracy as unmodified MQA. In experiments training 1B- and 3B-parameter models from scratch, we demonstrate that CLA provides a Pareto improvement over the memory/accuracy tradeoffs which are possible with traditional MQA, enabling inference with longer sequence lengths and larger batch sizes than would otherwise be possible.</p>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">The memory footprint of the key-value (KV) cache can be a bottleneck when serving large language models (LLMs). Because the size of the KV cache scales proportionally with both sequence length and batch size, the memory overhead of KV cache storage can limit batch sizes when operating on long sequence lengths <cite class="ltx_cite ltx_citemacro_citep">(Chowdhery et al., <a href="#bib.bib7" title="" class="ltx_ref">2022</a>)</cite>, and can require employing costly techniques like offloading when on-device memory is scarce <cite class="ltx_cite ltx_citemacro_citep">(Sheng et al., <a href="#bib.bib36" title="" class="ltx_ref">2023</a>)</cite>. It is also desirable to be able to persist KV caches over long periods of time in order to minimize redundant computations <cite class="ltx_cite ltx_citemacro_citep">(Gao et al., <a href="#bib.bib12" title="" class="ltx_ref">2024</a>; Google, <a href="#bib.bib15" title="" class="ltx_ref">2024</a>)</cite>. However, the size of the KV cache directly determines the cost of storing and retrieving such persistent caches. As new applications of LLMs emerge which demand ever-longer sequence lengths, the memory footprint of the KV cache is becoming an increasingly important consideration in the design of efficient transformer-based language models.</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">Existing work has proposed a variety of methods for decreasing the memory footprint of the KV cache, including storing KV activations in low precision <cite class="ltx_cite ltx_citemacro_citep">(Hooper et al., <a href="#bib.bib18" title="" class="ltx_ref">2024</a>; Zhang et al., <a href="#bib.bib47" title="" class="ltx_ref">2024</a>)</cite>, evicting unimportant KV cache entries <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al., <a href="#bib.bib48" title="" class="ltx_ref">2023</a>; Liu et al., <a href="#bib.bib24" title="" class="ltx_ref">2023</a>)</cite>, and sharing keys and values across query heads in the attention mechanism <cite class="ltx_cite ltx_citemacro_citep">(Shazeer, <a href="#bib.bib34" title="" class="ltx_ref">2019</a>; Ainslie et al., <a href="#bib.bib2" title="" class="ltx_ref">2023b</a>)</cite>.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">In this paper, we introduce a method for reducing the size of the KV cache along a dimension different than those explored in prior work: namely, reducing the number of unique <em id="S1.p3.1.1" class="ltx_emph ltx_font_italic">layers</em> in the KV cache. Our contributions are as follows:</p>
<ol id="S1.I1" class="ltx_enumerate">
<li id="S1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S1.I1.i1.p1" class="ltx_para">
<p id="S1.I1.i1.p1.1" class="ltx_p">We propose <em id="S1.I1.i1.p1.1.1" class="ltx_emph ltx_font_italic">Cross-Layer Attention</em> (CLA), a modification to the transformer architecture which reduces the size of the KV cache by <em id="S1.I1.i1.p1.1.2" class="ltx_emph ltx_font_italic">sharing KV activations across layers</em>.</p>
</div>
</li>
<li id="S1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S1.I1.i2.p1" class="ltx_para">
<p id="S1.I1.i2.p1.1" class="ltx_p">We conduct extensive pretraining experiments to characterize the effect of different CLA configurations on accuracy and memory usage across a range of architectural hyperparameters, learning rates and model sizes.</p>
</div>
</li>
<li id="S1.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">3.</span> 
<div id="S1.I1.i3.p1" class="ltx_para">
<p id="S1.I1.i3.p1.1" class="ltx_p">We demonstrate that CLA enables accuracy/memory Pareto improvements relative to existing Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) architectures.</p>
</div>
</li>
<li id="S1.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">4.</span> 
<div id="S1.I1.i4.p1" class="ltx_para">
<p id="S1.I1.i4.p1.1" class="ltx_p">In particular, we demonstrate at the 1B- and 3B-parameter scales that combining CLA with MQA can achieve a <math id="S1.I1.i4.p1.1.m1.1" class="ltx_math_unparsed" alttext="2\times" display="inline"><semantics id="S1.I1.i4.p1.1.m1.1a"><mrow id="S1.I1.i4.p1.1.m1.1b"><mn id="S1.I1.i4.p1.1.m1.1.1">2</mn><mo lspace="0.222em" id="S1.I1.i4.p1.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S1.I1.i4.p1.1.m1.1c">2\times</annotation></semantics></math> reduction in KV cache size versus a plain MQA baseline, with minimal degradation in perplexity.</p>
</div>
</li>
<li id="S1.I1.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">5.</span> 
<div id="S1.I1.i5.p1" class="ltx_para">
<p id="S1.I1.i5.p1.1" class="ltx_p">We offer guidance on which CLA configurations perform best based on our experiments, finding that CLA should be used between pairs of consecutive layers, and that CLA appears to deliver the most robust benefits when used in conjunction with MQA.</p>
</div>
</li>
</ol>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Cross-Layer Attention</h2>

<div id="S2.p1" class="ltx_para">
<p id="S2.p1.1" class="ltx_p">In this section we describe our Cross-Layer Attention (CLA) technique, and its relationship to the KV-sharing mechanisms employed by the existing Multi-Query and Grouped-Query attention architectures (MQA and GQA).</p>
</div>
<figure id="S2.F1" class="ltx_figure"><img src="/html/2405.12981/assets/x1.png" id="S2.F1.g1" class="ltx_graphics ltx_centering ltx_img_square" width="213" height="231" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>Schematic of two consecutive layers in a transformer using a traditional attention design (left) and in a transformer using Cross-Layer Attention (right).
When using traditional attention, each layer computes its own separate <math id="S2.F1.7.m1.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S2.F1.7.m1.1b"><mi id="S2.F1.7.m1.1.1" xref="S2.F1.7.m1.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S2.F1.7.m1.1c"><ci id="S2.F1.7.m1.1.1.cmml" xref="S2.F1.7.m1.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.F1.7.m1.1d">K</annotation></semantics></math> and <math id="S2.F1.8.m2.1" class="ltx_Math" alttext="V" display="inline"><semantics id="S2.F1.8.m2.1b"><mi id="S2.F1.8.m2.1.1" xref="S2.F1.8.m2.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="S2.F1.8.m2.1c"><ci id="S2.F1.8.m2.1.1.cmml" xref="S2.F1.8.m2.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.F1.8.m2.1d">V</annotation></semantics></math> activations, which must be cached on a per-layer basis during autoregressive decoding.
When using Cross-Layer Attention, some layers compute their own fresh <math id="S2.F1.9.m3.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S2.F1.9.m3.1b"><mi id="S2.F1.9.m3.1.1" xref="S2.F1.9.m3.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S2.F1.9.m3.1c"><ci id="S2.F1.9.m3.1.1.cmml" xref="S2.F1.9.m3.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.F1.9.m3.1d">K</annotation></semantics></math> and <math id="S2.F1.10.m4.1" class="ltx_Math" alttext="V" display="inline"><semantics id="S2.F1.10.m4.1b"><mi id="S2.F1.10.m4.1.1" xref="S2.F1.10.m4.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="S2.F1.10.m4.1c"><ci id="S2.F1.10.m4.1.1.cmml" xref="S2.F1.10.m4.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.F1.10.m4.1d">V</annotation></semantics></math> activations, while other layers reuse the <math id="S2.F1.11.m5.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S2.F1.11.m5.1b"><mi id="S2.F1.11.m5.1.1" xref="S2.F1.11.m5.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S2.F1.11.m5.1c"><ci id="S2.F1.11.m5.1.1.cmml" xref="S2.F1.11.m5.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.F1.11.m5.1d">K</annotation></semantics></math> and <math id="S2.F1.12.m6.1" class="ltx_Math" alttext="V" display="inline"><semantics id="S2.F1.12.m6.1b"><mi id="S2.F1.12.m6.1.1" xref="S2.F1.12.m6.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="S2.F1.12.m6.1c"><ci id="S2.F1.12.m6.1.1.cmml" xref="S2.F1.12.m6.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.F1.12.m6.1d">V</annotation></semantics></math> activations of earlier layers.
</figcaption>
</figure>
<figure id="S2.F2" class="ltx_figure"><img src="/html/2405.12981/assets/x2.png" id="S2.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="300" height="159" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>Schematic of KV cache structures under different attention configurations in a 10-layer transformer. Using traditional attention, each layer has its own KV cache. Using Cross-Layer Attention with a sharing factor of <math id="S2.F2.5.m1.1" class="ltx_Math" alttext="2" display="inline"><semantics id="S2.F2.5.m1.1b"><mn id="S2.F2.5.m1.1.1" xref="S2.F2.5.m1.1.1.cmml">2</mn><annotation-xml encoding="MathML-Content" id="S2.F2.5.m1.1c"><cn type="integer" id="S2.F2.5.m1.1.1.cmml" xref="S2.F2.5.m1.1.1">2</cn></annotation-xml><annotation encoding="application/x-tex" id="S2.F2.5.m1.1d">2</annotation></semantics></math> (CLA2), every group of <math id="S2.F2.6.m2.1" class="ltx_Math" alttext="2" display="inline"><semantics id="S2.F2.6.m2.1b"><mn id="S2.F2.6.m2.1.1" xref="S2.F2.6.m2.1.1.cmml">2</mn><annotation-xml encoding="MathML-Content" id="S2.F2.6.m2.1c"><cn type="integer" id="S2.F2.6.m2.1.1.cmml" xref="S2.F2.6.m2.1.1">2</cn></annotation-xml><annotation encoding="application/x-tex" id="S2.F2.6.m2.1d">2</annotation></semantics></math> consecutive layers shares a single KV cache. Using Cross-Layer Attention with a sharing factor of <math id="S2.F2.7.m3.1" class="ltx_Math" alttext="3" display="inline"><semantics id="S2.F2.7.m3.1b"><mn id="S2.F2.7.m3.1.1" xref="S2.F2.7.m3.1.1.cmml">3</mn><annotation-xml encoding="MathML-Content" id="S2.F2.7.m3.1c"><cn type="integer" id="S2.F2.7.m3.1.1.cmml" xref="S2.F2.7.m3.1.1">3</cn></annotation-xml><annotation encoding="application/x-tex" id="S2.F2.7.m3.1d">3</annotation></semantics></math> (CLA3), every group of <math id="S2.F2.8.m4.1" class="ltx_Math" alttext="3" display="inline"><semantics id="S2.F2.8.m4.1b"><mn id="S2.F2.8.m4.1.1" xref="S2.F2.8.m4.1.1.cmml">3</mn><annotation-xml encoding="MathML-Content" id="S2.F2.8.m4.1c"><cn type="integer" id="S2.F2.8.m4.1.1.cmml" xref="S2.F2.8.m4.1.1">3</cn></annotation-xml><annotation encoding="application/x-tex" id="S2.F2.8.m4.1d">3</annotation></semantics></math> consecutive layers shares a single KV cache. When the sharing factor does not evenly divide the number of layers, as in the CLA3 example, some KV caches must be shared over fewer layers than others; in this CLA3 configuration, we arbitrarily select the layer 0 KV cache to be used only in layer 0.</figcaption>
</figure>
<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>Background: Multi-Query Attention and Grouped-Query Attention</h3>

<div id="S2.SS1.p1" class="ltx_para">
<p id="S2.SS1.p1.3" class="ltx_p">The original transformer architecture employed Multi-Head Attention (MHA) <cite class="ltx_cite ltx_citemacro_citep">(Vaswani et al., <a href="#bib.bib42" title="" class="ltx_ref">2017</a>)</cite>, in which each query head attends over the keys and values produced by a distinct key/value head. In MHA, the KV activations of each key/value head must be stored separately in the KV cache, resulting in a storage overhead of <math id="S2.SS1.p1.1.m1.1" class="ltx_Math" alttext="2\cdot n_{\text{query}}\cdot d_{\text{head}}" display="inline"><semantics id="S2.SS1.p1.1.m1.1a"><mrow id="S2.SS1.p1.1.m1.1.1" xref="S2.SS1.p1.1.m1.1.1.cmml"><mn id="S2.SS1.p1.1.m1.1.1.2" xref="S2.SS1.p1.1.m1.1.1.2.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="S2.SS1.p1.1.m1.1.1.1" xref="S2.SS1.p1.1.m1.1.1.1.cmml">⋅</mo><msub id="S2.SS1.p1.1.m1.1.1.3" xref="S2.SS1.p1.1.m1.1.1.3.cmml"><mi id="S2.SS1.p1.1.m1.1.1.3.2" xref="S2.SS1.p1.1.m1.1.1.3.2.cmml">n</mi><mtext id="S2.SS1.p1.1.m1.1.1.3.3" xref="S2.SS1.p1.1.m1.1.1.3.3a.cmml">query</mtext></msub><mo lspace="0.222em" rspace="0.222em" id="S2.SS1.p1.1.m1.1.1.1a" xref="S2.SS1.p1.1.m1.1.1.1.cmml">⋅</mo><msub id="S2.SS1.p1.1.m1.1.1.4" xref="S2.SS1.p1.1.m1.1.1.4.cmml"><mi id="S2.SS1.p1.1.m1.1.1.4.2" xref="S2.SS1.p1.1.m1.1.1.4.2.cmml">d</mi><mtext id="S2.SS1.p1.1.m1.1.1.4.3" xref="S2.SS1.p1.1.m1.1.1.4.3a.cmml">head</mtext></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.1.m1.1b"><apply id="S2.SS1.p1.1.m1.1.1.cmml" xref="S2.SS1.p1.1.m1.1.1"><ci id="S2.SS1.p1.1.m1.1.1.1.cmml" xref="S2.SS1.p1.1.m1.1.1.1">⋅</ci><cn type="integer" id="S2.SS1.p1.1.m1.1.1.2.cmml" xref="S2.SS1.p1.1.m1.1.1.2">2</cn><apply id="S2.SS1.p1.1.m1.1.1.3.cmml" xref="S2.SS1.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S2.SS1.p1.1.m1.1.1.3.1.cmml" xref="S2.SS1.p1.1.m1.1.1.3">subscript</csymbol><ci id="S2.SS1.p1.1.m1.1.1.3.2.cmml" xref="S2.SS1.p1.1.m1.1.1.3.2">𝑛</ci><ci id="S2.SS1.p1.1.m1.1.1.3.3a.cmml" xref="S2.SS1.p1.1.m1.1.1.3.3"><mtext mathsize="70%" id="S2.SS1.p1.1.m1.1.1.3.3.cmml" xref="S2.SS1.p1.1.m1.1.1.3.3">query</mtext></ci></apply><apply id="S2.SS1.p1.1.m1.1.1.4.cmml" xref="S2.SS1.p1.1.m1.1.1.4"><csymbol cd="ambiguous" id="S2.SS1.p1.1.m1.1.1.4.1.cmml" xref="S2.SS1.p1.1.m1.1.1.4">subscript</csymbol><ci id="S2.SS1.p1.1.m1.1.1.4.2.cmml" xref="S2.SS1.p1.1.m1.1.1.4.2">𝑑</ci><ci id="S2.SS1.p1.1.m1.1.1.4.3a.cmml" xref="S2.SS1.p1.1.m1.1.1.4.3"><mtext mathsize="70%" id="S2.SS1.p1.1.m1.1.1.4.3.cmml" xref="S2.SS1.p1.1.m1.1.1.4.3">head</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.1.m1.1c">2\cdot n_{\text{query}}\cdot d_{\text{head}}</annotation></semantics></math> elements per token, where <math id="S2.SS1.p1.2.m2.1" class="ltx_Math" alttext="n_{\text{query}}" display="inline"><semantics id="S2.SS1.p1.2.m2.1a"><msub id="S2.SS1.p1.2.m2.1.1" xref="S2.SS1.p1.2.m2.1.1.cmml"><mi id="S2.SS1.p1.2.m2.1.1.2" xref="S2.SS1.p1.2.m2.1.1.2.cmml">n</mi><mtext id="S2.SS1.p1.2.m2.1.1.3" xref="S2.SS1.p1.2.m2.1.1.3a.cmml">query</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.2.m2.1b"><apply id="S2.SS1.p1.2.m2.1.1.cmml" xref="S2.SS1.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.2.m2.1.1.1.cmml" xref="S2.SS1.p1.2.m2.1.1">subscript</csymbol><ci id="S2.SS1.p1.2.m2.1.1.2.cmml" xref="S2.SS1.p1.2.m2.1.1.2">𝑛</ci><ci id="S2.SS1.p1.2.m2.1.1.3a.cmml" xref="S2.SS1.p1.2.m2.1.1.3"><mtext mathsize="70%" id="S2.SS1.p1.2.m2.1.1.3.cmml" xref="S2.SS1.p1.2.m2.1.1.3">query</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.2.m2.1c">n_{\text{query}}</annotation></semantics></math> is the number of query heads and <math id="S2.SS1.p1.3.m3.1" class="ltx_Math" alttext="d_{\text{head}}" display="inline"><semantics id="S2.SS1.p1.3.m3.1a"><msub id="S2.SS1.p1.3.m3.1.1" xref="S2.SS1.p1.3.m3.1.1.cmml"><mi id="S2.SS1.p1.3.m3.1.1.2" xref="S2.SS1.p1.3.m3.1.1.2.cmml">d</mi><mtext id="S2.SS1.p1.3.m3.1.1.3" xref="S2.SS1.p1.3.m3.1.1.3a.cmml">head</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.3.m3.1b"><apply id="S2.SS1.p1.3.m3.1.1.cmml" xref="S2.SS1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.3.m3.1.1.1.cmml" xref="S2.SS1.p1.3.m3.1.1">subscript</csymbol><ci id="S2.SS1.p1.3.m3.1.1.2.cmml" xref="S2.SS1.p1.3.m3.1.1.2">𝑑</ci><ci id="S2.SS1.p1.3.m3.1.1.3a.cmml" xref="S2.SS1.p1.3.m3.1.1.3"><mtext mathsize="70%" id="S2.SS1.p1.3.m3.1.1.3.cmml" xref="S2.SS1.p1.3.m3.1.1.3">head</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.3.m3.1c">d_{\text{head}}</annotation></semantics></math> is the embedding dimension of each head.</p>
</div>
<div id="S2.SS1.p2" class="ltx_para">
<p id="S2.SS1.p2.4" class="ltx_p">To reduce the overhead associated with storing and accessing the KV cache during transformer decoding, <cite class="ltx_cite ltx_citemacro_cite">Shazeer (<a href="#bib.bib34" title="" class="ltx_ref">2019</a>)</cite> proposed <em id="S2.SS1.p2.4.1" class="ltx_emph ltx_font_italic">Multi-Query Attention</em> (MQA), which <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib2" title="" class="ltx_ref">Ainslie et al.</a></cite> later generalized to <em id="S2.SS1.p2.4.2" class="ltx_emph ltx_font_italic">Grouped-Query Attention</em> (GQA). Grouped-Query Attention modifies the transformer architecture by organizing the query heads of each attention layer into groups, where each group of query heads shares a single key/value head. Because the size of the KV cache scales only with the number of distinct key/value heads, not the number of query heads, GQA reduces the storage overhead of the KV cache to <math id="S2.SS1.p2.1.m1.1" class="ltx_Math" alttext="2\cdot n_{\text{group}}\cdot d_{\text{head}}" display="inline"><semantics id="S2.SS1.p2.1.m1.1a"><mrow id="S2.SS1.p2.1.m1.1.1" xref="S2.SS1.p2.1.m1.1.1.cmml"><mn id="S2.SS1.p2.1.m1.1.1.2" xref="S2.SS1.p2.1.m1.1.1.2.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="S2.SS1.p2.1.m1.1.1.1" xref="S2.SS1.p2.1.m1.1.1.1.cmml">⋅</mo><msub id="S2.SS1.p2.1.m1.1.1.3" xref="S2.SS1.p2.1.m1.1.1.3.cmml"><mi id="S2.SS1.p2.1.m1.1.1.3.2" xref="S2.SS1.p2.1.m1.1.1.3.2.cmml">n</mi><mtext id="S2.SS1.p2.1.m1.1.1.3.3" xref="S2.SS1.p2.1.m1.1.1.3.3a.cmml">group</mtext></msub><mo lspace="0.222em" rspace="0.222em" id="S2.SS1.p2.1.m1.1.1.1a" xref="S2.SS1.p2.1.m1.1.1.1.cmml">⋅</mo><msub id="S2.SS1.p2.1.m1.1.1.4" xref="S2.SS1.p2.1.m1.1.1.4.cmml"><mi id="S2.SS1.p2.1.m1.1.1.4.2" xref="S2.SS1.p2.1.m1.1.1.4.2.cmml">d</mi><mtext id="S2.SS1.p2.1.m1.1.1.4.3" xref="S2.SS1.p2.1.m1.1.1.4.3a.cmml">head</mtext></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.1.m1.1b"><apply id="S2.SS1.p2.1.m1.1.1.cmml" xref="S2.SS1.p2.1.m1.1.1"><ci id="S2.SS1.p2.1.m1.1.1.1.cmml" xref="S2.SS1.p2.1.m1.1.1.1">⋅</ci><cn type="integer" id="S2.SS1.p2.1.m1.1.1.2.cmml" xref="S2.SS1.p2.1.m1.1.1.2">2</cn><apply id="S2.SS1.p2.1.m1.1.1.3.cmml" xref="S2.SS1.p2.1.m1.1.1.3"><csymbol cd="ambiguous" id="S2.SS1.p2.1.m1.1.1.3.1.cmml" xref="S2.SS1.p2.1.m1.1.1.3">subscript</csymbol><ci id="S2.SS1.p2.1.m1.1.1.3.2.cmml" xref="S2.SS1.p2.1.m1.1.1.3.2">𝑛</ci><ci id="S2.SS1.p2.1.m1.1.1.3.3a.cmml" xref="S2.SS1.p2.1.m1.1.1.3.3"><mtext mathsize="70%" id="S2.SS1.p2.1.m1.1.1.3.3.cmml" xref="S2.SS1.p2.1.m1.1.1.3.3">group</mtext></ci></apply><apply id="S2.SS1.p2.1.m1.1.1.4.cmml" xref="S2.SS1.p2.1.m1.1.1.4"><csymbol cd="ambiguous" id="S2.SS1.p2.1.m1.1.1.4.1.cmml" xref="S2.SS1.p2.1.m1.1.1.4">subscript</csymbol><ci id="S2.SS1.p2.1.m1.1.1.4.2.cmml" xref="S2.SS1.p2.1.m1.1.1.4.2">𝑑</ci><ci id="S2.SS1.p2.1.m1.1.1.4.3a.cmml" xref="S2.SS1.p2.1.m1.1.1.4.3"><mtext mathsize="70%" id="S2.SS1.p2.1.m1.1.1.4.3.cmml" xref="S2.SS1.p2.1.m1.1.1.4.3">head</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.1.m1.1c">2\cdot n_{\text{group}}\cdot d_{\text{head}}</annotation></semantics></math>, where <math id="S2.SS1.p2.2.m2.1" class="ltx_Math" alttext="n_{\text{group}}" display="inline"><semantics id="S2.SS1.p2.2.m2.1a"><msub id="S2.SS1.p2.2.m2.1.1" xref="S2.SS1.p2.2.m2.1.1.cmml"><mi id="S2.SS1.p2.2.m2.1.1.2" xref="S2.SS1.p2.2.m2.1.1.2.cmml">n</mi><mtext id="S2.SS1.p2.2.m2.1.1.3" xref="S2.SS1.p2.2.m2.1.1.3a.cmml">group</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.2.m2.1b"><apply id="S2.SS1.p2.2.m2.1.1.cmml" xref="S2.SS1.p2.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS1.p2.2.m2.1.1.1.cmml" xref="S2.SS1.p2.2.m2.1.1">subscript</csymbol><ci id="S2.SS1.p2.2.m2.1.1.2.cmml" xref="S2.SS1.p2.2.m2.1.1.2">𝑛</ci><ci id="S2.SS1.p2.2.m2.1.1.3a.cmml" xref="S2.SS1.p2.2.m2.1.1.3"><mtext mathsize="70%" id="S2.SS1.p2.2.m2.1.1.3.cmml" xref="S2.SS1.p2.2.m2.1.1.3">group</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.2.m2.1c">n_{\text{group}}</annotation></semantics></math> denotes the number of groups for GQA and <math id="S2.SS1.p2.3.m3.1" class="ltx_Math" alttext="n_{\text{group}}&lt;n_{\text{query}}" display="inline"><semantics id="S2.SS1.p2.3.m3.1a"><mrow id="S2.SS1.p2.3.m3.1.1" xref="S2.SS1.p2.3.m3.1.1.cmml"><msub id="S2.SS1.p2.3.m3.1.1.2" xref="S2.SS1.p2.3.m3.1.1.2.cmml"><mi id="S2.SS1.p2.3.m3.1.1.2.2" xref="S2.SS1.p2.3.m3.1.1.2.2.cmml">n</mi><mtext id="S2.SS1.p2.3.m3.1.1.2.3" xref="S2.SS1.p2.3.m3.1.1.2.3a.cmml">group</mtext></msub><mo id="S2.SS1.p2.3.m3.1.1.1" xref="S2.SS1.p2.3.m3.1.1.1.cmml">&lt;</mo><msub id="S2.SS1.p2.3.m3.1.1.3" xref="S2.SS1.p2.3.m3.1.1.3.cmml"><mi id="S2.SS1.p2.3.m3.1.1.3.2" xref="S2.SS1.p2.3.m3.1.1.3.2.cmml">n</mi><mtext id="S2.SS1.p2.3.m3.1.1.3.3" xref="S2.SS1.p2.3.m3.1.1.3.3a.cmml">query</mtext></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.3.m3.1b"><apply id="S2.SS1.p2.3.m3.1.1.cmml" xref="S2.SS1.p2.3.m3.1.1"><lt id="S2.SS1.p2.3.m3.1.1.1.cmml" xref="S2.SS1.p2.3.m3.1.1.1"></lt><apply id="S2.SS1.p2.3.m3.1.1.2.cmml" xref="S2.SS1.p2.3.m3.1.1.2"><csymbol cd="ambiguous" id="S2.SS1.p2.3.m3.1.1.2.1.cmml" xref="S2.SS1.p2.3.m3.1.1.2">subscript</csymbol><ci id="S2.SS1.p2.3.m3.1.1.2.2.cmml" xref="S2.SS1.p2.3.m3.1.1.2.2">𝑛</ci><ci id="S2.SS1.p2.3.m3.1.1.2.3a.cmml" xref="S2.SS1.p2.3.m3.1.1.2.3"><mtext mathsize="70%" id="S2.SS1.p2.3.m3.1.1.2.3.cmml" xref="S2.SS1.p2.3.m3.1.1.2.3">group</mtext></ci></apply><apply id="S2.SS1.p2.3.m3.1.1.3.cmml" xref="S2.SS1.p2.3.m3.1.1.3"><csymbol cd="ambiguous" id="S2.SS1.p2.3.m3.1.1.3.1.cmml" xref="S2.SS1.p2.3.m3.1.1.3">subscript</csymbol><ci id="S2.SS1.p2.3.m3.1.1.3.2.cmml" xref="S2.SS1.p2.3.m3.1.1.3.2">𝑛</ci><ci id="S2.SS1.p2.3.m3.1.1.3.3a.cmml" xref="S2.SS1.p2.3.m3.1.1.3.3"><mtext mathsize="70%" id="S2.SS1.p2.3.m3.1.1.3.3.cmml" xref="S2.SS1.p2.3.m3.1.1.3.3">query</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.3.m3.1c">n_{\text{group}}&lt;n_{\text{query}}</annotation></semantics></math>. MQA can be seen as the special case of GQA in which <math id="S2.SS1.p2.4.m4.1" class="ltx_Math" alttext="n_{\text{group}}=1" display="inline"><semantics id="S2.SS1.p2.4.m4.1a"><mrow id="S2.SS1.p2.4.m4.1.1" xref="S2.SS1.p2.4.m4.1.1.cmml"><msub id="S2.SS1.p2.4.m4.1.1.2" xref="S2.SS1.p2.4.m4.1.1.2.cmml"><mi id="S2.SS1.p2.4.m4.1.1.2.2" xref="S2.SS1.p2.4.m4.1.1.2.2.cmml">n</mi><mtext id="S2.SS1.p2.4.m4.1.1.2.3" xref="S2.SS1.p2.4.m4.1.1.2.3a.cmml">group</mtext></msub><mo id="S2.SS1.p2.4.m4.1.1.1" xref="S2.SS1.p2.4.m4.1.1.1.cmml">=</mo><mn id="S2.SS1.p2.4.m4.1.1.3" xref="S2.SS1.p2.4.m4.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.4.m4.1b"><apply id="S2.SS1.p2.4.m4.1.1.cmml" xref="S2.SS1.p2.4.m4.1.1"><eq id="S2.SS1.p2.4.m4.1.1.1.cmml" xref="S2.SS1.p2.4.m4.1.1.1"></eq><apply id="S2.SS1.p2.4.m4.1.1.2.cmml" xref="S2.SS1.p2.4.m4.1.1.2"><csymbol cd="ambiguous" id="S2.SS1.p2.4.m4.1.1.2.1.cmml" xref="S2.SS1.p2.4.m4.1.1.2">subscript</csymbol><ci id="S2.SS1.p2.4.m4.1.1.2.2.cmml" xref="S2.SS1.p2.4.m4.1.1.2.2">𝑛</ci><ci id="S2.SS1.p2.4.m4.1.1.2.3a.cmml" xref="S2.SS1.p2.4.m4.1.1.2.3"><mtext mathsize="70%" id="S2.SS1.p2.4.m4.1.1.2.3.cmml" xref="S2.SS1.p2.4.m4.1.1.2.3">group</mtext></ci></apply><cn type="integer" id="S2.SS1.p2.4.m4.1.1.3.cmml" xref="S2.SS1.p2.4.m4.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.4.m4.1c">n_{\text{group}}=1</annotation></semantics></math>.</p>
</div>
<div id="S2.SS1.p3" class="ltx_para">
<p id="S2.SS1.p3.1" class="ltx_p"><cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib34" title="" class="ltx_ref">Shazeer</a></cite> and <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib2" title="" class="ltx_ref">Ainslie et al.</a></cite> find that MQA and GQA enable significant reductions in KV cache size and decoding latency while incurring only a small degradation in accuracy compared to MHA architectures with the same head dimension. The family of attention architectures enabled by using MQA and GQA defines an <em id="S2.SS1.p3.1.1" class="ltx_emph ltx_font_italic">accuracy/memory tradeoff space</em> in which model designers can choose how they want to balance the expressive power and KV cache overhead of their attention mechanism. MQA and GQA stake out different positions in this tradeoff space, and neither is necessarily preferable to the other for all use cases.</p>
</div>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>Sharing KV Activations Across Layers</h3>

<div id="S2.SS2.p1" class="ltx_para">
<p id="S2.SS2.p1.1" class="ltx_p">Inspired by the success of MQA and GQA, which share key/value heads across query heads within a single layer, we propose also sharing key/value heads <em id="S2.SS2.p1.1.1" class="ltx_emph ltx_font_italic">across layers</em>. We refer to such an attention architecture as <em id="S2.SS2.p1.1.2" class="ltx_emph ltx_font_italic">Cross-Layer Attention</em> (CLA), and present a diagrammatic view of it in Figure <a href="#S2.F1" title="Figure 1 ‣ 2 Cross-Layer Attention ‣ Reducing Transformer Key-Value Cache Size with Cross-Layer Attention" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>. CLA computes key/value projections for only a subset of layers in the model; the attention blocks in layers without key/value projections reuse the KV activations of previous layers. Only the subset of layers with key/value projections contribute to the KV cache, allowing a reduction in memory footprint relative to traditional architectures which apply a separate key/value projection in each layer.</p>
</div>
<div id="S2.SS2.p2" class="ltx_para">
<p id="S2.SS2.p2.2" class="ltx_p">CLA is orthogonal to MQA/GQA/MHA, and can be combined with any of them. Moreover, in the same way that GQA allows varying <math id="S2.SS2.p2.1.m1.1" class="ltx_Math" alttext="n_{\text{group}}" display="inline"><semantics id="S2.SS2.p2.1.m1.1a"><msub id="S2.SS2.p2.1.m1.1.1" xref="S2.SS2.p2.1.m1.1.1.cmml"><mi id="S2.SS2.p2.1.m1.1.1.2" xref="S2.SS2.p2.1.m1.1.1.2.cmml">n</mi><mtext id="S2.SS2.p2.1.m1.1.1.3" xref="S2.SS2.p2.1.m1.1.1.3a.cmml">group</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.1.m1.1b"><apply id="S2.SS2.p2.1.m1.1.1.cmml" xref="S2.SS2.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p2.1.m1.1.1.1.cmml" xref="S2.SS2.p2.1.m1.1.1">subscript</csymbol><ci id="S2.SS2.p2.1.m1.1.1.2.cmml" xref="S2.SS2.p2.1.m1.1.1.2">𝑛</ci><ci id="S2.SS2.p2.1.m1.1.1.3a.cmml" xref="S2.SS2.p2.1.m1.1.1.3"><mtext mathsize="70%" id="S2.SS2.p2.1.m1.1.1.3.cmml" xref="S2.SS2.p2.1.m1.1.1.3">group</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.1.m1.1c">n_{\text{group}}</annotation></semantics></math> to access a family of different attention configurations, CLA allows varying the number of layers which share the output of each KV projection, which we refer to as the <em id="S2.SS2.p2.2.1" class="ltx_emph ltx_font_italic">sharing factor</em>. We refer to different configurations of CLA by their sharing factors, giving rise to CLA2, which shares each KV projection among a pair of adjacent layers, CLA3, which shares each KV projection among a group of <math id="S2.SS2.p2.2.m2.1" class="ltx_Math" alttext="3" display="inline"><semantics id="S2.SS2.p2.2.m2.1a"><mn id="S2.SS2.p2.2.m2.1.1" xref="S2.SS2.p2.2.m2.1.1.cmml">3</mn><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.2.m2.1b"><cn type="integer" id="S2.SS2.p2.2.m2.1.1.cmml" xref="S2.SS2.p2.2.m2.1.1">3</cn></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.2.m2.1c">3</annotation></semantics></math> layers, and so on. We present a visualization of different attention configurations possible with CLA in Figure <a href="#S2.F2" title="Figure 2 ‣ 2 Cross-Layer Attention ‣ Reducing Transformer Key-Value Cache Size with Cross-Layer Attention" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.</p>
</div>
</section>
<section id="S2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.3 </span>Implications for System Design</h3>

<div id="S2.SS3.p1" class="ltx_para">
<p id="S2.SS3.p1.1" class="ltx_p">CLA is primarily an intervention to reduce the memory footprint of the KV cache, and only has minor effects on other resources consumed by the model during training and inference. Here, we summarize the effect of CLA on key metrics relevant from a systems engineering perspective, assuming all other architectural hyperparameters are held constant:</p>
<ul id="S2.I1" class="ltx_itemize">
<li id="S2.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I1.i1.p1" class="ltx_para">
<p id="S2.I1.i1.p1.1" class="ltx_p"><span id="S2.I1.i1.p1.1.1" class="ltx_text ltx_font_bold">KV Cache Memory</span>: CLA significantly reduces KV cache memory footprint, shrinking it by a factor equal to the sharing factor, or slightly less if the sharing factor does not evenly divide the number of layers.</p>
</div>
</li>
<li id="S2.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I1.i2.p1" class="ltx_para">
<p id="S2.I1.i2.p1.1" class="ltx_p"><span id="S2.I1.i2.p1.1.1" class="ltx_text ltx_font_bold">Training Memory Footprint</span>: CLA reduces the memory footprint of intermediate KV activation tensors materialized during training, although for GQA and MQA models such KV tensors are typically small compared to the model’s hidden states and MLP activations.</p>
</div>
</li>
<li id="S2.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I1.i3.p1" class="ltx_para">
<p id="S2.I1.i3.p1.1" class="ltx_p"><span id="S2.I1.i3.p1.1.1" class="ltx_text ltx_font_bold">Model Parallelism</span>: CLA is fully compatible with standard tensor parallelism techniques <cite class="ltx_cite ltx_citemacro_citep">(Shoeybi et al., <a href="#bib.bib37" title="" class="ltx_ref">2020</a>)</cite> for sharding model weights across multiple accelerators. In the presence of pipeline parallelism <cite class="ltx_cite ltx_citemacro_citep">(Huang et al., <a href="#bib.bib19" title="" class="ltx_ref">2019</a>)</cite>, either different layers which share a KV cache must be kept in the same pipeline stage, or else KV activations must be communicated between pipeline stages.</p>
</div>
</li>
<li id="S2.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I1.i4.p1" class="ltx_para">
<p id="S2.I1.i4.p1.1" class="ltx_p"><span id="S2.I1.i4.p1.1.1" class="ltx_text ltx_font_bold">Parameters and FLOPs</span>: Because CLA reduces the total number of key/value projection blocks in the model, CLA slightly reduces the number of parameters in the model and the number of FLOPs required during a forward or backward pass.</p>
</div>
</li>
<li id="S2.I1.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I1.i5.p1" class="ltx_para">
<p id="S2.I1.i5.p1.1" class="ltx_p"><span id="S2.I1.i5.p1.1.1" class="ltx_text ltx_font_bold">Decoding Latency</span>: In the context of a full LLM serving stack, CLA can enable larger batch sizes and longer KV cache persistence times than would otherwise be possible, which have the potential to improve inference latency.</p>
</div>
</li>
<li id="S2.I1.i6" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I1.i6.p1" class="ltx_para">
<p id="S2.I1.i6.p1.1" class="ltx_p"><span id="S2.I1.i6.p1.1.1" class="ltx_text ltx_font_bold">Core Attention Latency</span>: Unlike MQA and GQA, CLA has no direct effect on the memory bandwidth consumed by the attention mechanism in each decoding step, because even shared KV cache layers must be separately re-read from main memory in each attention layer. CLA therefore has no direct effect on the latency of the core attention computation during decoding.</p>
</div>
</li>
</ul>
</div>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Pretraining Experiments</h2>

<figure id="S3.F3" class="ltx_figure"><img src="/html/2405.12981/assets/x3.png" id="S3.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="305" height="222" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>The accuracy/memory Pareto frontier discovered in our 1B-scale design space exploration, for models with CLA (red) and without CLA (blue). Lower is better on both axes.</figcaption>
</figure>
<figure id="S3.T1" class="ltx_table">
<table id="S3.T1.122" class="ltx_tabular ltx_centering ltx_align_middle">
<thead class="ltx_thead">
<tr id="S3.T1.1.1" class="ltx_tr">
<th id="S3.T1.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">
<span id="S3.T1.1.1.2.1" class="ltx_text"></span> <span id="S3.T1.1.1.2.2" class="ltx_text">
<span id="S3.T1.1.1.2.2.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T1.1.1.2.2.1.1" class="ltx_tr">
<span id="S3.T1.1.1.2.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T1.1.1.2.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Model</span></span></span>
</span></span><span id="S3.T1.1.1.2.3" class="ltx_text"></span></th>
<th id="S3.T1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">
<span id="S3.T1.1.1.1.2" class="ltx_text"></span> <span id="S3.T1.1.1.1.1" class="ltx_text">
<span id="S3.T1.1.1.1.1.1.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T1.1.1.1.1.1.1.1" class="ltx_tr">
<span id="S3.T1.1.1.1.1.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><math id="S3.T1.1.1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="{\boldsymbol{d}_{\textbf{head}}}" display="inline"><semantics id="S3.T1.1.1.1.1.1.1.1.1.m1.1a"><msub id="S3.T1.1.1.1.1.1.1.1.1.m1.1.1" xref="S3.T1.1.1.1.1.1.1.1.1.m1.1.1.cmml"><mi mathsize="80%" id="S3.T1.1.1.1.1.1.1.1.1.m1.1.1.2" xref="S3.T1.1.1.1.1.1.1.1.1.m1.1.1.2.cmml">𝒅</mi><mtext class="ltx_mathvariant_bold" mathsize="80%" id="S3.T1.1.1.1.1.1.1.1.1.m1.1.1.3" xref="S3.T1.1.1.1.1.1.1.1.1.m1.1.1.3a.cmml">head</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.T1.1.1.1.1.1.1.1.1.m1.1b"><apply id="S3.T1.1.1.1.1.1.1.1.1.m1.1.1.cmml" xref="S3.T1.1.1.1.1.1.1.1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.T1.1.1.1.1.1.1.1.1.m1.1.1.1.cmml" xref="S3.T1.1.1.1.1.1.1.1.1.m1.1.1">subscript</csymbol><ci id="S3.T1.1.1.1.1.1.1.1.1.m1.1.1.2.cmml" xref="S3.T1.1.1.1.1.1.1.1.1.m1.1.1.2">𝒅</ci><ci id="S3.T1.1.1.1.1.1.1.1.1.m1.1.1.3a.cmml" xref="S3.T1.1.1.1.1.1.1.1.1.m1.1.1.3"><mtext class="ltx_mathvariant_bold" mathsize="56%" id="S3.T1.1.1.1.1.1.1.1.1.m1.1.1.3.cmml" xref="S3.T1.1.1.1.1.1.1.1.1.m1.1.1.3">head</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.1.1.1.1.1.1.1.1.m1.1c">{\boldsymbol{d}_{\textbf{head}}}</annotation></semantics></math></span></span>
</span></span><span id="S3.T1.1.1.1.3" class="ltx_text"></span>
</th>
<th id="S3.T1.1.1.3" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">
<span id="S3.T1.1.1.3.1" class="ltx_text"></span> <span id="S3.T1.1.1.3.2" class="ltx_text">
<span id="S3.T1.1.1.3.2.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T1.1.1.3.2.1.1" class="ltx_tr">
<span id="S3.T1.1.1.3.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T1.1.1.3.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Query</span></span></span>
<span id="S3.T1.1.1.3.2.1.2" class="ltx_tr">
<span id="S3.T1.1.1.3.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T1.1.1.3.2.1.2.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Heads</span></span></span>
</span></span><span id="S3.T1.1.1.3.3" class="ltx_text"></span>
</th>
<th id="S3.T1.1.1.4" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">
<span id="S3.T1.1.1.4.1" class="ltx_text"></span> <span id="S3.T1.1.1.4.2" class="ltx_text">
<span id="S3.T1.1.1.4.2.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T1.1.1.4.2.1.1" class="ltx_tr">
<span id="S3.T1.1.1.4.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T1.1.1.4.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">KV</span></span></span>
<span id="S3.T1.1.1.4.2.1.2" class="ltx_tr">
<span id="S3.T1.1.1.4.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T1.1.1.4.2.1.2.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Heads</span></span></span>
</span></span><span id="S3.T1.1.1.4.3" class="ltx_text"></span>
</th>
<th id="S3.T1.1.1.5" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">
<span id="S3.T1.1.1.5.1" class="ltx_text"></span> <span id="S3.T1.1.1.5.2" class="ltx_text">
<span id="S3.T1.1.1.5.2.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T1.1.1.5.2.1.1" class="ltx_tr">
<span id="S3.T1.1.1.5.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T1.1.1.5.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">KV</span></span></span>
<span id="S3.T1.1.1.5.2.1.2" class="ltx_tr">
<span id="S3.T1.1.1.5.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T1.1.1.5.2.1.2.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Layers</span></span></span>
</span></span><span id="S3.T1.1.1.5.3" class="ltx_text"></span>
</th>
<th id="S3.T1.1.1.6" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">
<span id="S3.T1.1.1.6.1" class="ltx_text"></span> <span id="S3.T1.1.1.6.2" class="ltx_text">
<span id="S3.T1.1.1.6.2.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T1.1.1.6.2.1.1" class="ltx_tr">
<span id="S3.T1.1.1.6.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T1.1.1.6.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">KV Bytes Per</span></span></span>
<span id="S3.T1.1.1.6.2.1.2" class="ltx_tr">
<span id="S3.T1.1.1.6.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T1.1.1.6.2.1.2.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Token (16-Bit)</span></span></span>
</span></span><span id="S3.T1.1.1.6.3" class="ltx_text"></span>
</th>
<th id="S3.T1.1.1.7" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">
<span id="S3.T1.1.1.7.1" class="ltx_text"></span> <span id="S3.T1.1.1.7.2" class="ltx_text">
<span id="S3.T1.1.1.7.2.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T1.1.1.7.2.1.1" class="ltx_tr">
<span id="S3.T1.1.1.7.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T1.1.1.7.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Validation</span></span></span>
<span id="S3.T1.1.1.7.2.1.2" class="ltx_tr">
<span id="S3.T1.1.1.7.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T1.1.1.7.2.1.2.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Perplexity</span></span></span>
</span></span><span id="S3.T1.1.1.7.3" class="ltx_text"></span>
</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S3.T1.122.123.1" class="ltx_tr">
<td id="S3.T1.122.123.1.1" class="ltx_td ltx_align_left ltx_border_t" colspan="2"><span id="S3.T1.122.123.1.1.1" class="ltx_text ltx_font_bold">Non-CLA Baselines</span></td>
<td id="S3.T1.122.123.1.2" class="ltx_td ltx_border_t"></td>
<td id="S3.T1.122.123.1.3" class="ltx_td ltx_border_t"></td>
<td id="S3.T1.122.123.1.4" class="ltx_td ltx_border_t"></td>
<td id="S3.T1.122.123.1.5" class="ltx_td ltx_border_t"></td>
<td id="S3.T1.122.123.1.6" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="S3.T1.7.7" class="ltx_tr">
<td id="S3.T1.7.7.7" class="ltx_td ltx_align_left ltx_border_t">H128-MHA</td>
<td id="S3.T1.2.2.1" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.2.2.1.m1.1" class="ltx_Math" alttext="128" display="inline"><semantics id="S3.T1.2.2.1.m1.1a"><mn id="S3.T1.2.2.1.m1.1.1" xref="S3.T1.2.2.1.m1.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="S3.T1.2.2.1.m1.1b"><cn type="integer" id="S3.T1.2.2.1.m1.1.1.cmml" xref="S3.T1.2.2.1.m1.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.2.2.1.m1.1c">128</annotation></semantics></math></td>
<td id="S3.T1.3.3.2" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.3.3.2.m1.1" class="ltx_Math" alttext="16" display="inline"><semantics id="S3.T1.3.3.2.m1.1a"><mn id="S3.T1.3.3.2.m1.1.1" xref="S3.T1.3.3.2.m1.1.1.cmml">16</mn><annotation-xml encoding="MathML-Content" id="S3.T1.3.3.2.m1.1b"><cn type="integer" id="S3.T1.3.3.2.m1.1.1.cmml" xref="S3.T1.3.3.2.m1.1.1">16</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.3.3.2.m1.1c">16</annotation></semantics></math></td>
<td id="S3.T1.4.4.3" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.4.4.3.m1.1" class="ltx_Math" alttext="16" display="inline"><semantics id="S3.T1.4.4.3.m1.1a"><mn id="S3.T1.4.4.3.m1.1.1" xref="S3.T1.4.4.3.m1.1.1.cmml">16</mn><annotation-xml encoding="MathML-Content" id="S3.T1.4.4.3.m1.1b"><cn type="integer" id="S3.T1.4.4.3.m1.1.1.cmml" xref="S3.T1.4.4.3.m1.1.1">16</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.4.4.3.m1.1c">16</annotation></semantics></math></td>
<td id="S3.T1.5.5.4" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.5.5.4.m1.1" class="ltx_Math" alttext="20" display="inline"><semantics id="S3.T1.5.5.4.m1.1a"><mn id="S3.T1.5.5.4.m1.1.1" xref="S3.T1.5.5.4.m1.1.1.cmml">20</mn><annotation-xml encoding="MathML-Content" id="S3.T1.5.5.4.m1.1b"><cn type="integer" id="S3.T1.5.5.4.m1.1.1.cmml" xref="S3.T1.5.5.4.m1.1.1">20</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.5.5.4.m1.1c">20</annotation></semantics></math></td>
<td id="S3.T1.6.6.5" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.6.6.5.m1.1" class="ltx_Math" alttext="163\,840" display="inline"><semantics id="S3.T1.6.6.5.m1.1a"><mn id="S3.T1.6.6.5.m1.1.1" xref="S3.T1.6.6.5.m1.1.1.cmml">163 840</mn><annotation-xml encoding="MathML-Content" id="S3.T1.6.6.5.m1.1b"><cn type="integer" id="S3.T1.6.6.5.m1.1.1.cmml" xref="S3.T1.6.6.5.m1.1.1">163840</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.6.6.5.m1.1c">163\,840</annotation></semantics></math></td>
<td id="S3.T1.7.7.6" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.7.7.6.m1.1" class="ltx_Math" alttext="13.15" display="inline"><semantics id="S3.T1.7.7.6.m1.1a"><mn id="S3.T1.7.7.6.m1.1.1" xref="S3.T1.7.7.6.m1.1.1.cmml">13.15</mn><annotation-xml encoding="MathML-Content" id="S3.T1.7.7.6.m1.1b"><cn type="float" id="S3.T1.7.7.6.m1.1.1.cmml" xref="S3.T1.7.7.6.m1.1.1">13.15</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.7.7.6.m1.1c">13.15</annotation></semantics></math></td>
</tr>
<tr id="S3.T1.13.13" class="ltx_tr">
<td id="S3.T1.13.13.7" class="ltx_td ltx_align_left">H128-GQA4</td>
<td id="S3.T1.8.8.1" class="ltx_td ltx_align_left"><math id="S3.T1.8.8.1.m1.1" class="ltx_Math" alttext="128" display="inline"><semantics id="S3.T1.8.8.1.m1.1a"><mn id="S3.T1.8.8.1.m1.1.1" xref="S3.T1.8.8.1.m1.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="S3.T1.8.8.1.m1.1b"><cn type="integer" id="S3.T1.8.8.1.m1.1.1.cmml" xref="S3.T1.8.8.1.m1.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.8.8.1.m1.1c">128</annotation></semantics></math></td>
<td id="S3.T1.9.9.2" class="ltx_td ltx_align_left"><math id="S3.T1.9.9.2.m1.1" class="ltx_Math" alttext="16" display="inline"><semantics id="S3.T1.9.9.2.m1.1a"><mn id="S3.T1.9.9.2.m1.1.1" xref="S3.T1.9.9.2.m1.1.1.cmml">16</mn><annotation-xml encoding="MathML-Content" id="S3.T1.9.9.2.m1.1b"><cn type="integer" id="S3.T1.9.9.2.m1.1.1.cmml" xref="S3.T1.9.9.2.m1.1.1">16</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.9.9.2.m1.1c">16</annotation></semantics></math></td>
<td id="S3.T1.10.10.3" class="ltx_td ltx_align_left"><math id="S3.T1.10.10.3.m1.1" class="ltx_Math" alttext="4" display="inline"><semantics id="S3.T1.10.10.3.m1.1a"><mn id="S3.T1.10.10.3.m1.1.1" xref="S3.T1.10.10.3.m1.1.1.cmml">4</mn><annotation-xml encoding="MathML-Content" id="S3.T1.10.10.3.m1.1b"><cn type="integer" id="S3.T1.10.10.3.m1.1.1.cmml" xref="S3.T1.10.10.3.m1.1.1">4</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.10.10.3.m1.1c">4</annotation></semantics></math></td>
<td id="S3.T1.11.11.4" class="ltx_td ltx_align_left"><math id="S3.T1.11.11.4.m1.1" class="ltx_Math" alttext="20" display="inline"><semantics id="S3.T1.11.11.4.m1.1a"><mn id="S3.T1.11.11.4.m1.1.1" xref="S3.T1.11.11.4.m1.1.1.cmml">20</mn><annotation-xml encoding="MathML-Content" id="S3.T1.11.11.4.m1.1b"><cn type="integer" id="S3.T1.11.11.4.m1.1.1.cmml" xref="S3.T1.11.11.4.m1.1.1">20</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.11.11.4.m1.1c">20</annotation></semantics></math></td>
<td id="S3.T1.12.12.5" class="ltx_td ltx_align_left"><math id="S3.T1.12.12.5.m1.1" class="ltx_Math" alttext="40\,960" display="inline"><semantics id="S3.T1.12.12.5.m1.1a"><mn id="S3.T1.12.12.5.m1.1.1" xref="S3.T1.12.12.5.m1.1.1.cmml">40 960</mn><annotation-xml encoding="MathML-Content" id="S3.T1.12.12.5.m1.1b"><cn type="integer" id="S3.T1.12.12.5.m1.1.1.cmml" xref="S3.T1.12.12.5.m1.1.1">40960</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.12.12.5.m1.1c">40\,960</annotation></semantics></math></td>
<td id="S3.T1.13.13.6" class="ltx_td ltx_align_left"><math id="S3.T1.13.13.6.m1.1" class="ltx_Math" alttext="13.36" display="inline"><semantics id="S3.T1.13.13.6.m1.1a"><mn id="S3.T1.13.13.6.m1.1.1" xref="S3.T1.13.13.6.m1.1.1.cmml">13.36</mn><annotation-xml encoding="MathML-Content" id="S3.T1.13.13.6.m1.1b"><cn type="float" id="S3.T1.13.13.6.m1.1.1.cmml" xref="S3.T1.13.13.6.m1.1.1">13.36</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.13.13.6.m1.1c">13.36</annotation></semantics></math></td>
</tr>
<tr id="S3.T1.19.19" class="ltx_tr">
<td id="S3.T1.19.19.7" class="ltx_td ltx_align_left">H128-GQA2</td>
<td id="S3.T1.14.14.1" class="ltx_td ltx_align_left"><math id="S3.T1.14.14.1.m1.1" class="ltx_Math" alttext="128" display="inline"><semantics id="S3.T1.14.14.1.m1.1a"><mn id="S3.T1.14.14.1.m1.1.1" xref="S3.T1.14.14.1.m1.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="S3.T1.14.14.1.m1.1b"><cn type="integer" id="S3.T1.14.14.1.m1.1.1.cmml" xref="S3.T1.14.14.1.m1.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.14.14.1.m1.1c">128</annotation></semantics></math></td>
<td id="S3.T1.15.15.2" class="ltx_td ltx_align_left"><math id="S3.T1.15.15.2.m1.1" class="ltx_Math" alttext="16" display="inline"><semantics id="S3.T1.15.15.2.m1.1a"><mn id="S3.T1.15.15.2.m1.1.1" xref="S3.T1.15.15.2.m1.1.1.cmml">16</mn><annotation-xml encoding="MathML-Content" id="S3.T1.15.15.2.m1.1b"><cn type="integer" id="S3.T1.15.15.2.m1.1.1.cmml" xref="S3.T1.15.15.2.m1.1.1">16</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.15.15.2.m1.1c">16</annotation></semantics></math></td>
<td id="S3.T1.16.16.3" class="ltx_td ltx_align_left"><math id="S3.T1.16.16.3.m1.1" class="ltx_Math" alttext="2" display="inline"><semantics id="S3.T1.16.16.3.m1.1a"><mn id="S3.T1.16.16.3.m1.1.1" xref="S3.T1.16.16.3.m1.1.1.cmml">2</mn><annotation-xml encoding="MathML-Content" id="S3.T1.16.16.3.m1.1b"><cn type="integer" id="S3.T1.16.16.3.m1.1.1.cmml" xref="S3.T1.16.16.3.m1.1.1">2</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.16.16.3.m1.1c">2</annotation></semantics></math></td>
<td id="S3.T1.17.17.4" class="ltx_td ltx_align_left"><math id="S3.T1.17.17.4.m1.1" class="ltx_Math" alttext="20" display="inline"><semantics id="S3.T1.17.17.4.m1.1a"><mn id="S3.T1.17.17.4.m1.1.1" xref="S3.T1.17.17.4.m1.1.1.cmml">20</mn><annotation-xml encoding="MathML-Content" id="S3.T1.17.17.4.m1.1b"><cn type="integer" id="S3.T1.17.17.4.m1.1.1.cmml" xref="S3.T1.17.17.4.m1.1.1">20</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.17.17.4.m1.1c">20</annotation></semantics></math></td>
<td id="S3.T1.18.18.5" class="ltx_td ltx_align_left"><math id="S3.T1.18.18.5.m1.1" class="ltx_Math" alttext="20\,480" display="inline"><semantics id="S3.T1.18.18.5.m1.1a"><mn id="S3.T1.18.18.5.m1.1.1" xref="S3.T1.18.18.5.m1.1.1.cmml">20 480</mn><annotation-xml encoding="MathML-Content" id="S3.T1.18.18.5.m1.1b"><cn type="integer" id="S3.T1.18.18.5.m1.1.1.cmml" xref="S3.T1.18.18.5.m1.1.1">20480</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.18.18.5.m1.1c">20\,480</annotation></semantics></math></td>
<td id="S3.T1.19.19.6" class="ltx_td ltx_align_left"><math id="S3.T1.19.19.6.m1.1" class="ltx_Math" alttext="13.52" display="inline"><semantics id="S3.T1.19.19.6.m1.1a"><mn id="S3.T1.19.19.6.m1.1.1" xref="S3.T1.19.19.6.m1.1.1.cmml">13.52</mn><annotation-xml encoding="MathML-Content" id="S3.T1.19.19.6.m1.1b"><cn type="float" id="S3.T1.19.19.6.m1.1.1.cmml" xref="S3.T1.19.19.6.m1.1.1">13.52</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.19.19.6.m1.1c">13.52</annotation></semantics></math></td>
</tr>
<tr id="S3.T1.25.25" class="ltx_tr">
<td id="S3.T1.25.25.7" class="ltx_td ltx_align_left">H128-MQA</td>
<td id="S3.T1.20.20.1" class="ltx_td ltx_align_left"><math id="S3.T1.20.20.1.m1.1" class="ltx_Math" alttext="128" display="inline"><semantics id="S3.T1.20.20.1.m1.1a"><mn id="S3.T1.20.20.1.m1.1.1" xref="S3.T1.20.20.1.m1.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="S3.T1.20.20.1.m1.1b"><cn type="integer" id="S3.T1.20.20.1.m1.1.1.cmml" xref="S3.T1.20.20.1.m1.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.20.20.1.m1.1c">128</annotation></semantics></math></td>
<td id="S3.T1.21.21.2" class="ltx_td ltx_align_left"><math id="S3.T1.21.21.2.m1.1" class="ltx_Math" alttext="16" display="inline"><semantics id="S3.T1.21.21.2.m1.1a"><mn id="S3.T1.21.21.2.m1.1.1" xref="S3.T1.21.21.2.m1.1.1.cmml">16</mn><annotation-xml encoding="MathML-Content" id="S3.T1.21.21.2.m1.1b"><cn type="integer" id="S3.T1.21.21.2.m1.1.1.cmml" xref="S3.T1.21.21.2.m1.1.1">16</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.21.21.2.m1.1c">16</annotation></semantics></math></td>
<td id="S3.T1.22.22.3" class="ltx_td ltx_align_left"><math id="S3.T1.22.22.3.m1.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S3.T1.22.22.3.m1.1a"><mn id="S3.T1.22.22.3.m1.1.1" xref="S3.T1.22.22.3.m1.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S3.T1.22.22.3.m1.1b"><cn type="integer" id="S3.T1.22.22.3.m1.1.1.cmml" xref="S3.T1.22.22.3.m1.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.22.22.3.m1.1c">1</annotation></semantics></math></td>
<td id="S3.T1.23.23.4" class="ltx_td ltx_align_left"><math id="S3.T1.23.23.4.m1.1" class="ltx_Math" alttext="20" display="inline"><semantics id="S3.T1.23.23.4.m1.1a"><mn id="S3.T1.23.23.4.m1.1.1" xref="S3.T1.23.23.4.m1.1.1.cmml">20</mn><annotation-xml encoding="MathML-Content" id="S3.T1.23.23.4.m1.1b"><cn type="integer" id="S3.T1.23.23.4.m1.1.1.cmml" xref="S3.T1.23.23.4.m1.1.1">20</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.23.23.4.m1.1c">20</annotation></semantics></math></td>
<td id="S3.T1.24.24.5" class="ltx_td ltx_align_left"><math id="S3.T1.24.24.5.m1.1" class="ltx_Math" alttext="10\,240" display="inline"><semantics id="S3.T1.24.24.5.m1.1a"><mn id="S3.T1.24.24.5.m1.1.1" xref="S3.T1.24.24.5.m1.1.1.cmml">10 240</mn><annotation-xml encoding="MathML-Content" id="S3.T1.24.24.5.m1.1b"><cn type="integer" id="S3.T1.24.24.5.m1.1.1.cmml" xref="S3.T1.24.24.5.m1.1.1">10240</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.24.24.5.m1.1c">10\,240</annotation></semantics></math></td>
<td id="S3.T1.25.25.6" class="ltx_td ltx_align_left"><math id="S3.T1.25.25.6.m1.1" class="ltx_Math" alttext="13.54" display="inline"><semantics id="S3.T1.25.25.6.m1.1a"><mn id="S3.T1.25.25.6.m1.1.1" xref="S3.T1.25.25.6.m1.1.1.cmml">13.54</mn><annotation-xml encoding="MathML-Content" id="S3.T1.25.25.6.m1.1b"><cn type="float" id="S3.T1.25.25.6.m1.1.1.cmml" xref="S3.T1.25.25.6.m1.1.1">13.54</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.25.25.6.m1.1c">13.54</annotation></semantics></math></td>
</tr>
<tr id="S3.T1.31.31" class="ltx_tr">
<td id="S3.T1.31.31.7" class="ltx_td ltx_align_left">H64-MQA</td>
<td id="S3.T1.26.26.1" class="ltx_td ltx_align_left"><math id="S3.T1.26.26.1.m1.1" class="ltx_Math" alttext="64" display="inline"><semantics id="S3.T1.26.26.1.m1.1a"><mn id="S3.T1.26.26.1.m1.1.1" xref="S3.T1.26.26.1.m1.1.1.cmml">64</mn><annotation-xml encoding="MathML-Content" id="S3.T1.26.26.1.m1.1b"><cn type="integer" id="S3.T1.26.26.1.m1.1.1.cmml" xref="S3.T1.26.26.1.m1.1.1">64</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.26.26.1.m1.1c">64</annotation></semantics></math></td>
<td id="S3.T1.27.27.2" class="ltx_td ltx_align_left"><math id="S3.T1.27.27.2.m1.1" class="ltx_Math" alttext="32" display="inline"><semantics id="S3.T1.27.27.2.m1.1a"><mn id="S3.T1.27.27.2.m1.1.1" xref="S3.T1.27.27.2.m1.1.1.cmml">32</mn><annotation-xml encoding="MathML-Content" id="S3.T1.27.27.2.m1.1b"><cn type="integer" id="S3.T1.27.27.2.m1.1.1.cmml" xref="S3.T1.27.27.2.m1.1.1">32</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.27.27.2.m1.1c">32</annotation></semantics></math></td>
<td id="S3.T1.28.28.3" class="ltx_td ltx_align_left"><math id="S3.T1.28.28.3.m1.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S3.T1.28.28.3.m1.1a"><mn id="S3.T1.28.28.3.m1.1.1" xref="S3.T1.28.28.3.m1.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S3.T1.28.28.3.m1.1b"><cn type="integer" id="S3.T1.28.28.3.m1.1.1.cmml" xref="S3.T1.28.28.3.m1.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.28.28.3.m1.1c">1</annotation></semantics></math></td>
<td id="S3.T1.29.29.4" class="ltx_td ltx_align_left"><math id="S3.T1.29.29.4.m1.1" class="ltx_Math" alttext="20" display="inline"><semantics id="S3.T1.29.29.4.m1.1a"><mn id="S3.T1.29.29.4.m1.1.1" xref="S3.T1.29.29.4.m1.1.1.cmml">20</mn><annotation-xml encoding="MathML-Content" id="S3.T1.29.29.4.m1.1b"><cn type="integer" id="S3.T1.29.29.4.m1.1.1.cmml" xref="S3.T1.29.29.4.m1.1.1">20</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.29.29.4.m1.1c">20</annotation></semantics></math></td>
<td id="S3.T1.30.30.5" class="ltx_td ltx_align_left"><math id="S3.T1.30.30.5.m1.1" class="ltx_Math" alttext="5120" display="inline"><semantics id="S3.T1.30.30.5.m1.1a"><mn id="S3.T1.30.30.5.m1.1.1" xref="S3.T1.30.30.5.m1.1.1.cmml">5120</mn><annotation-xml encoding="MathML-Content" id="S3.T1.30.30.5.m1.1b"><cn type="integer" id="S3.T1.30.30.5.m1.1.1.cmml" xref="S3.T1.30.30.5.m1.1.1">5120</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.30.30.5.m1.1c">5120</annotation></semantics></math></td>
<td id="S3.T1.31.31.6" class="ltx_td ltx_align_left"><math id="S3.T1.31.31.6.m1.1" class="ltx_Math" alttext="13.81" display="inline"><semantics id="S3.T1.31.31.6.m1.1a"><mn id="S3.T1.31.31.6.m1.1.1" xref="S3.T1.31.31.6.m1.1.1.cmml">13.81</mn><annotation-xml encoding="MathML-Content" id="S3.T1.31.31.6.m1.1b"><cn type="float" id="S3.T1.31.31.6.m1.1.1.cmml" xref="S3.T1.31.31.6.m1.1.1">13.81</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.31.31.6.m1.1c">13.81</annotation></semantics></math></td>
</tr>
<tr id="S3.T1.37.37" class="ltx_tr">
<td id="S3.T1.37.37.7" class="ltx_td ltx_align_left">H46-MQA</td>
<td id="S3.T1.32.32.1" class="ltx_td ltx_align_left"><math id="S3.T1.32.32.1.m1.1" class="ltx_Math" alttext="46" display="inline"><semantics id="S3.T1.32.32.1.m1.1a"><mn id="S3.T1.32.32.1.m1.1.1" xref="S3.T1.32.32.1.m1.1.1.cmml">46</mn><annotation-xml encoding="MathML-Content" id="S3.T1.32.32.1.m1.1b"><cn type="integer" id="S3.T1.32.32.1.m1.1.1.cmml" xref="S3.T1.32.32.1.m1.1.1">46</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.32.32.1.m1.1c">46</annotation></semantics></math></td>
<td id="S3.T1.33.33.2" class="ltx_td ltx_align_left"><math id="S3.T1.33.33.2.m1.1" class="ltx_Math" alttext="45" display="inline"><semantics id="S3.T1.33.33.2.m1.1a"><mn id="S3.T1.33.33.2.m1.1.1" xref="S3.T1.33.33.2.m1.1.1.cmml">45</mn><annotation-xml encoding="MathML-Content" id="S3.T1.33.33.2.m1.1b"><cn type="integer" id="S3.T1.33.33.2.m1.1.1.cmml" xref="S3.T1.33.33.2.m1.1.1">45</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.33.33.2.m1.1c">45</annotation></semantics></math></td>
<td id="S3.T1.34.34.3" class="ltx_td ltx_align_left"><math id="S3.T1.34.34.3.m1.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S3.T1.34.34.3.m1.1a"><mn id="S3.T1.34.34.3.m1.1.1" xref="S3.T1.34.34.3.m1.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S3.T1.34.34.3.m1.1b"><cn type="integer" id="S3.T1.34.34.3.m1.1.1.cmml" xref="S3.T1.34.34.3.m1.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.34.34.3.m1.1c">1</annotation></semantics></math></td>
<td id="S3.T1.35.35.4" class="ltx_td ltx_align_left"><math id="S3.T1.35.35.4.m1.1" class="ltx_Math" alttext="20" display="inline"><semantics id="S3.T1.35.35.4.m1.1a"><mn id="S3.T1.35.35.4.m1.1.1" xref="S3.T1.35.35.4.m1.1.1.cmml">20</mn><annotation-xml encoding="MathML-Content" id="S3.T1.35.35.4.m1.1b"><cn type="integer" id="S3.T1.35.35.4.m1.1.1.cmml" xref="S3.T1.35.35.4.m1.1.1">20</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.35.35.4.m1.1c">20</annotation></semantics></math></td>
<td id="S3.T1.36.36.5" class="ltx_td ltx_align_left"><math id="S3.T1.36.36.5.m1.1" class="ltx_Math" alttext="3680" display="inline"><semantics id="S3.T1.36.36.5.m1.1a"><mn id="S3.T1.36.36.5.m1.1.1" xref="S3.T1.36.36.5.m1.1.1.cmml">3680</mn><annotation-xml encoding="MathML-Content" id="S3.T1.36.36.5.m1.1b"><cn type="integer" id="S3.T1.36.36.5.m1.1.1.cmml" xref="S3.T1.36.36.5.m1.1.1">3680</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.36.36.5.m1.1c">3680</annotation></semantics></math></td>
<td id="S3.T1.37.37.6" class="ltx_td ltx_align_left"><math id="S3.T1.37.37.6.m1.1" class="ltx_Math" alttext="13.96" display="inline"><semantics id="S3.T1.37.37.6.m1.1a"><mn id="S3.T1.37.37.6.m1.1.1" xref="S3.T1.37.37.6.m1.1.1.cmml">13.96</mn><annotation-xml encoding="MathML-Content" id="S3.T1.37.37.6.m1.1b"><cn type="float" id="S3.T1.37.37.6.m1.1.1.cmml" xref="S3.T1.37.37.6.m1.1.1">13.96</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.37.37.6.m1.1c">13.96</annotation></semantics></math></td>
</tr>
<tr id="S3.T1.43.43" class="ltx_tr">
<td id="S3.T1.43.43.7" class="ltx_td ltx_align_left">H32-MQA</td>
<td id="S3.T1.38.38.1" class="ltx_td ltx_align_left"><math id="S3.T1.38.38.1.m1.1" class="ltx_Math" alttext="32" display="inline"><semantics id="S3.T1.38.38.1.m1.1a"><mn id="S3.T1.38.38.1.m1.1.1" xref="S3.T1.38.38.1.m1.1.1.cmml">32</mn><annotation-xml encoding="MathML-Content" id="S3.T1.38.38.1.m1.1b"><cn type="integer" id="S3.T1.38.38.1.m1.1.1.cmml" xref="S3.T1.38.38.1.m1.1.1">32</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.38.38.1.m1.1c">32</annotation></semantics></math></td>
<td id="S3.T1.39.39.2" class="ltx_td ltx_align_left"><math id="S3.T1.39.39.2.m1.1" class="ltx_Math" alttext="64" display="inline"><semantics id="S3.T1.39.39.2.m1.1a"><mn id="S3.T1.39.39.2.m1.1.1" xref="S3.T1.39.39.2.m1.1.1.cmml">64</mn><annotation-xml encoding="MathML-Content" id="S3.T1.39.39.2.m1.1b"><cn type="integer" id="S3.T1.39.39.2.m1.1.1.cmml" xref="S3.T1.39.39.2.m1.1.1">64</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.39.39.2.m1.1c">64</annotation></semantics></math></td>
<td id="S3.T1.40.40.3" class="ltx_td ltx_align_left"><math id="S3.T1.40.40.3.m1.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S3.T1.40.40.3.m1.1a"><mn id="S3.T1.40.40.3.m1.1.1" xref="S3.T1.40.40.3.m1.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S3.T1.40.40.3.m1.1b"><cn type="integer" id="S3.T1.40.40.3.m1.1.1.cmml" xref="S3.T1.40.40.3.m1.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.40.40.3.m1.1c">1</annotation></semantics></math></td>
<td id="S3.T1.41.41.4" class="ltx_td ltx_align_left"><math id="S3.T1.41.41.4.m1.1" class="ltx_Math" alttext="20" display="inline"><semantics id="S3.T1.41.41.4.m1.1a"><mn id="S3.T1.41.41.4.m1.1.1" xref="S3.T1.41.41.4.m1.1.1.cmml">20</mn><annotation-xml encoding="MathML-Content" id="S3.T1.41.41.4.m1.1b"><cn type="integer" id="S3.T1.41.41.4.m1.1.1.cmml" xref="S3.T1.41.41.4.m1.1.1">20</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.41.41.4.m1.1c">20</annotation></semantics></math></td>
<td id="S3.T1.42.42.5" class="ltx_td ltx_align_left"><math id="S3.T1.42.42.5.m1.1" class="ltx_Math" alttext="2560" display="inline"><semantics id="S3.T1.42.42.5.m1.1a"><mn id="S3.T1.42.42.5.m1.1.1" xref="S3.T1.42.42.5.m1.1.1.cmml">2560</mn><annotation-xml encoding="MathML-Content" id="S3.T1.42.42.5.m1.1b"><cn type="integer" id="S3.T1.42.42.5.m1.1.1.cmml" xref="S3.T1.42.42.5.m1.1.1">2560</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.42.42.5.m1.1c">2560</annotation></semantics></math></td>
<td id="S3.T1.43.43.6" class="ltx_td ltx_align_left"><math id="S3.T1.43.43.6.m1.1" class="ltx_Math" alttext="14.37" display="inline"><semantics id="S3.T1.43.43.6.m1.1a"><mn id="S3.T1.43.43.6.m1.1.1" xref="S3.T1.43.43.6.m1.1.1.cmml">14.37</mn><annotation-xml encoding="MathML-Content" id="S3.T1.43.43.6.m1.1b"><cn type="float" id="S3.T1.43.43.6.m1.1.1.cmml" xref="S3.T1.43.43.6.m1.1.1">14.37</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.43.43.6.m1.1c">14.37</annotation></semantics></math></td>
</tr>
<tr id="S3.T1.122.124.2" class="ltx_tr">
<td id="S3.T1.122.124.2.1" class="ltx_td ltx_align_left ltx_border_t" colspan="2"><span id="S3.T1.122.124.2.1.1" class="ltx_text ltx_font_bold">MQA + CLA2 Models</span></td>
<td id="S3.T1.122.124.2.2" class="ltx_td ltx_border_t"></td>
<td id="S3.T1.122.124.2.3" class="ltx_td ltx_border_t"></td>
<td id="S3.T1.122.124.2.4" class="ltx_td ltx_border_t"></td>
<td id="S3.T1.122.124.2.5" class="ltx_td ltx_border_t"></td>
<td id="S3.T1.122.124.2.6" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="S3.T1.49.49" class="ltx_tr">
<td id="S3.T1.49.49.7" class="ltx_td ltx_align_left ltx_border_t">H512-MQA-CLA2</td>
<td id="S3.T1.44.44.1" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.44.44.1.m1.1" class="ltx_Math" alttext="512" display="inline"><semantics id="S3.T1.44.44.1.m1.1a"><mn id="S3.T1.44.44.1.m1.1.1" xref="S3.T1.44.44.1.m1.1.1.cmml">512</mn><annotation-xml encoding="MathML-Content" id="S3.T1.44.44.1.m1.1b"><cn type="integer" id="S3.T1.44.44.1.m1.1.1.cmml" xref="S3.T1.44.44.1.m1.1.1">512</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.44.44.1.m1.1c">512</annotation></semantics></math></td>
<td id="S3.T1.45.45.2" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.45.45.2.m1.1" class="ltx_Math" alttext="4" display="inline"><semantics id="S3.T1.45.45.2.m1.1a"><mn id="S3.T1.45.45.2.m1.1.1" xref="S3.T1.45.45.2.m1.1.1.cmml">4</mn><annotation-xml encoding="MathML-Content" id="S3.T1.45.45.2.m1.1b"><cn type="integer" id="S3.T1.45.45.2.m1.1.1.cmml" xref="S3.T1.45.45.2.m1.1.1">4</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.45.45.2.m1.1c">4</annotation></semantics></math></td>
<td id="S3.T1.46.46.3" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.46.46.3.m1.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S3.T1.46.46.3.m1.1a"><mn id="S3.T1.46.46.3.m1.1.1" xref="S3.T1.46.46.3.m1.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S3.T1.46.46.3.m1.1b"><cn type="integer" id="S3.T1.46.46.3.m1.1.1.cmml" xref="S3.T1.46.46.3.m1.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.46.46.3.m1.1c">1</annotation></semantics></math></td>
<td id="S3.T1.47.47.4" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.47.47.4.m1.1" class="ltx_Math" alttext="10" display="inline"><semantics id="S3.T1.47.47.4.m1.1a"><mn id="S3.T1.47.47.4.m1.1.1" xref="S3.T1.47.47.4.m1.1.1.cmml">10</mn><annotation-xml encoding="MathML-Content" id="S3.T1.47.47.4.m1.1b"><cn type="integer" id="S3.T1.47.47.4.m1.1.1.cmml" xref="S3.T1.47.47.4.m1.1.1">10</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.47.47.4.m1.1c">10</annotation></semantics></math></td>
<td id="S3.T1.48.48.5" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.48.48.5.m1.1" class="ltx_Math" alttext="20\,480" display="inline"><semantics id="S3.T1.48.48.5.m1.1a"><mn id="S3.T1.48.48.5.m1.1.1" xref="S3.T1.48.48.5.m1.1.1.cmml">20 480</mn><annotation-xml encoding="MathML-Content" id="S3.T1.48.48.5.m1.1b"><cn type="integer" id="S3.T1.48.48.5.m1.1.1.cmml" xref="S3.T1.48.48.5.m1.1.1">20480</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.48.48.5.m1.1c">20\,480</annotation></semantics></math></td>
<td id="S3.T1.49.49.6" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.49.49.6.m1.1" class="ltx_Math" alttext="13.49" display="inline"><semantics id="S3.T1.49.49.6.m1.1a"><mn id="S3.T1.49.49.6.m1.1.1" xref="S3.T1.49.49.6.m1.1.1.cmml">13.49</mn><annotation-xml encoding="MathML-Content" id="S3.T1.49.49.6.m1.1b"><cn type="float" id="S3.T1.49.49.6.m1.1.1.cmml" xref="S3.T1.49.49.6.m1.1.1">13.49</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.49.49.6.m1.1c">13.49</annotation></semantics></math></td>
</tr>
<tr id="S3.T1.55.55" class="ltx_tr">
<td id="S3.T1.55.55.7" class="ltx_td ltx_align_left">H256-MQA-CLA2</td>
<td id="S3.T1.50.50.1" class="ltx_td ltx_align_left"><math id="S3.T1.50.50.1.m1.1" class="ltx_Math" alttext="256" display="inline"><semantics id="S3.T1.50.50.1.m1.1a"><mn id="S3.T1.50.50.1.m1.1.1" xref="S3.T1.50.50.1.m1.1.1.cmml">256</mn><annotation-xml encoding="MathML-Content" id="S3.T1.50.50.1.m1.1b"><cn type="integer" id="S3.T1.50.50.1.m1.1.1.cmml" xref="S3.T1.50.50.1.m1.1.1">256</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.50.50.1.m1.1c">256</annotation></semantics></math></td>
<td id="S3.T1.51.51.2" class="ltx_td ltx_align_left"><math id="S3.T1.51.51.2.m1.1" class="ltx_Math" alttext="8" display="inline"><semantics id="S3.T1.51.51.2.m1.1a"><mn id="S3.T1.51.51.2.m1.1.1" xref="S3.T1.51.51.2.m1.1.1.cmml">8</mn><annotation-xml encoding="MathML-Content" id="S3.T1.51.51.2.m1.1b"><cn type="integer" id="S3.T1.51.51.2.m1.1.1.cmml" xref="S3.T1.51.51.2.m1.1.1">8</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.51.51.2.m1.1c">8</annotation></semantics></math></td>
<td id="S3.T1.52.52.3" class="ltx_td ltx_align_left"><math id="S3.T1.52.52.3.m1.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S3.T1.52.52.3.m1.1a"><mn id="S3.T1.52.52.3.m1.1.1" xref="S3.T1.52.52.3.m1.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S3.T1.52.52.3.m1.1b"><cn type="integer" id="S3.T1.52.52.3.m1.1.1.cmml" xref="S3.T1.52.52.3.m1.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.52.52.3.m1.1c">1</annotation></semantics></math></td>
<td id="S3.T1.53.53.4" class="ltx_td ltx_align_left"><math id="S3.T1.53.53.4.m1.1" class="ltx_Math" alttext="10" display="inline"><semantics id="S3.T1.53.53.4.m1.1a"><mn id="S3.T1.53.53.4.m1.1.1" xref="S3.T1.53.53.4.m1.1.1.cmml">10</mn><annotation-xml encoding="MathML-Content" id="S3.T1.53.53.4.m1.1b"><cn type="integer" id="S3.T1.53.53.4.m1.1.1.cmml" xref="S3.T1.53.53.4.m1.1.1">10</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.53.53.4.m1.1c">10</annotation></semantics></math></td>
<td id="S3.T1.54.54.5" class="ltx_td ltx_align_left"><math id="S3.T1.54.54.5.m1.1" class="ltx_Math" alttext="10\,240" display="inline"><semantics id="S3.T1.54.54.5.m1.1a"><mn id="S3.T1.54.54.5.m1.1.1" xref="S3.T1.54.54.5.m1.1.1.cmml">10 240</mn><annotation-xml encoding="MathML-Content" id="S3.T1.54.54.5.m1.1b"><cn type="integer" id="S3.T1.54.54.5.m1.1.1.cmml" xref="S3.T1.54.54.5.m1.1.1">10240</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.54.54.5.m1.1c">10\,240</annotation></semantics></math></td>
<td id="S3.T1.55.55.6" class="ltx_td ltx_align_left"><math id="S3.T1.55.55.6.m1.1" class="ltx_Math" alttext="13.51" display="inline"><semantics id="S3.T1.55.55.6.m1.1a"><mn id="S3.T1.55.55.6.m1.1.1" xref="S3.T1.55.55.6.m1.1.1.cmml">13.51</mn><annotation-xml encoding="MathML-Content" id="S3.T1.55.55.6.m1.1b"><cn type="float" id="S3.T1.55.55.6.m1.1.1.cmml" xref="S3.T1.55.55.6.m1.1.1">13.51</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.55.55.6.m1.1c">13.51</annotation></semantics></math></td>
</tr>
<tr id="S3.T1.61.61" class="ltx_tr">
<td id="S3.T1.61.61.7" class="ltx_td ltx_align_left">H128-MQA-CLA2</td>
<td id="S3.T1.56.56.1" class="ltx_td ltx_align_left"><math id="S3.T1.56.56.1.m1.1" class="ltx_Math" alttext="128" display="inline"><semantics id="S3.T1.56.56.1.m1.1a"><mn id="S3.T1.56.56.1.m1.1.1" xref="S3.T1.56.56.1.m1.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="S3.T1.56.56.1.m1.1b"><cn type="integer" id="S3.T1.56.56.1.m1.1.1.cmml" xref="S3.T1.56.56.1.m1.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.56.56.1.m1.1c">128</annotation></semantics></math></td>
<td id="S3.T1.57.57.2" class="ltx_td ltx_align_left"><math id="S3.T1.57.57.2.m1.1" class="ltx_Math" alttext="16" display="inline"><semantics id="S3.T1.57.57.2.m1.1a"><mn id="S3.T1.57.57.2.m1.1.1" xref="S3.T1.57.57.2.m1.1.1.cmml">16</mn><annotation-xml encoding="MathML-Content" id="S3.T1.57.57.2.m1.1b"><cn type="integer" id="S3.T1.57.57.2.m1.1.1.cmml" xref="S3.T1.57.57.2.m1.1.1">16</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.57.57.2.m1.1c">16</annotation></semantics></math></td>
<td id="S3.T1.58.58.3" class="ltx_td ltx_align_left"><math id="S3.T1.58.58.3.m1.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S3.T1.58.58.3.m1.1a"><mn id="S3.T1.58.58.3.m1.1.1" xref="S3.T1.58.58.3.m1.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S3.T1.58.58.3.m1.1b"><cn type="integer" id="S3.T1.58.58.3.m1.1.1.cmml" xref="S3.T1.58.58.3.m1.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.58.58.3.m1.1c">1</annotation></semantics></math></td>
<td id="S3.T1.59.59.4" class="ltx_td ltx_align_left"><math id="S3.T1.59.59.4.m1.1" class="ltx_Math" alttext="10" display="inline"><semantics id="S3.T1.59.59.4.m1.1a"><mn id="S3.T1.59.59.4.m1.1.1" xref="S3.T1.59.59.4.m1.1.1.cmml">10</mn><annotation-xml encoding="MathML-Content" id="S3.T1.59.59.4.m1.1b"><cn type="integer" id="S3.T1.59.59.4.m1.1.1.cmml" xref="S3.T1.59.59.4.m1.1.1">10</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.59.59.4.m1.1c">10</annotation></semantics></math></td>
<td id="S3.T1.60.60.5" class="ltx_td ltx_align_left"><math id="S3.T1.60.60.5.m1.1" class="ltx_Math" alttext="5120" display="inline"><semantics id="S3.T1.60.60.5.m1.1a"><mn id="S3.T1.60.60.5.m1.1.1" xref="S3.T1.60.60.5.m1.1.1.cmml">5120</mn><annotation-xml encoding="MathML-Content" id="S3.T1.60.60.5.m1.1b"><cn type="integer" id="S3.T1.60.60.5.m1.1.1.cmml" xref="S3.T1.60.60.5.m1.1.1">5120</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.60.60.5.m1.1c">5120</annotation></semantics></math></td>
<td id="S3.T1.61.61.6" class="ltx_td ltx_align_left"><math id="S3.T1.61.61.6.m1.1" class="ltx_Math" alttext="13.60" display="inline"><semantics id="S3.T1.61.61.6.m1.1a"><mn id="S3.T1.61.61.6.m1.1.1" xref="S3.T1.61.61.6.m1.1.1.cmml">13.60</mn><annotation-xml encoding="MathML-Content" id="S3.T1.61.61.6.m1.1b"><cn type="float" id="S3.T1.61.61.6.m1.1.1.cmml" xref="S3.T1.61.61.6.m1.1.1">13.60</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.61.61.6.m1.1c">13.60</annotation></semantics></math></td>
</tr>
<tr id="S3.T1.67.67" class="ltx_tr">
<td id="S3.T1.67.67.7" class="ltx_td ltx_align_left">H90-MQA-CLA2</td>
<td id="S3.T1.62.62.1" class="ltx_td ltx_align_left"><math id="S3.T1.62.62.1.m1.1" class="ltx_Math" alttext="90" display="inline"><semantics id="S3.T1.62.62.1.m1.1a"><mn id="S3.T1.62.62.1.m1.1.1" xref="S3.T1.62.62.1.m1.1.1.cmml">90</mn><annotation-xml encoding="MathML-Content" id="S3.T1.62.62.1.m1.1b"><cn type="integer" id="S3.T1.62.62.1.m1.1.1.cmml" xref="S3.T1.62.62.1.m1.1.1">90</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.62.62.1.m1.1c">90</annotation></semantics></math></td>
<td id="S3.T1.63.63.2" class="ltx_td ltx_align_left"><math id="S3.T1.63.63.2.m1.1" class="ltx_Math" alttext="22" display="inline"><semantics id="S3.T1.63.63.2.m1.1a"><mn id="S3.T1.63.63.2.m1.1.1" xref="S3.T1.63.63.2.m1.1.1.cmml">22</mn><annotation-xml encoding="MathML-Content" id="S3.T1.63.63.2.m1.1b"><cn type="integer" id="S3.T1.63.63.2.m1.1.1.cmml" xref="S3.T1.63.63.2.m1.1.1">22</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.63.63.2.m1.1c">22</annotation></semantics></math></td>
<td id="S3.T1.64.64.3" class="ltx_td ltx_align_left"><math id="S3.T1.64.64.3.m1.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S3.T1.64.64.3.m1.1a"><mn id="S3.T1.64.64.3.m1.1.1" xref="S3.T1.64.64.3.m1.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S3.T1.64.64.3.m1.1b"><cn type="integer" id="S3.T1.64.64.3.m1.1.1.cmml" xref="S3.T1.64.64.3.m1.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.64.64.3.m1.1c">1</annotation></semantics></math></td>
<td id="S3.T1.65.65.4" class="ltx_td ltx_align_left"><math id="S3.T1.65.65.4.m1.1" class="ltx_Math" alttext="10" display="inline"><semantics id="S3.T1.65.65.4.m1.1a"><mn id="S3.T1.65.65.4.m1.1.1" xref="S3.T1.65.65.4.m1.1.1.cmml">10</mn><annotation-xml encoding="MathML-Content" id="S3.T1.65.65.4.m1.1b"><cn type="integer" id="S3.T1.65.65.4.m1.1.1.cmml" xref="S3.T1.65.65.4.m1.1.1">10</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.65.65.4.m1.1c">10</annotation></semantics></math></td>
<td id="S3.T1.66.66.5" class="ltx_td ltx_align_left"><math id="S3.T1.66.66.5.m1.1" class="ltx_Math" alttext="3600" display="inline"><semantics id="S3.T1.66.66.5.m1.1a"><mn id="S3.T1.66.66.5.m1.1.1" xref="S3.T1.66.66.5.m1.1.1.cmml">3600</mn><annotation-xml encoding="MathML-Content" id="S3.T1.66.66.5.m1.1b"><cn type="integer" id="S3.T1.66.66.5.m1.1.1.cmml" xref="S3.T1.66.66.5.m1.1.1">3600</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.66.66.5.m1.1c">3600</annotation></semantics></math></td>
<td id="S3.T1.67.67.6" class="ltx_td ltx_align_left"><math id="S3.T1.67.67.6.m1.1" class="ltx_Math" alttext="13.73" display="inline"><semantics id="S3.T1.67.67.6.m1.1a"><mn id="S3.T1.67.67.6.m1.1.1" xref="S3.T1.67.67.6.m1.1.1.cmml">13.73</mn><annotation-xml encoding="MathML-Content" id="S3.T1.67.67.6.m1.1b"><cn type="float" id="S3.T1.67.67.6.m1.1.1.cmml" xref="S3.T1.67.67.6.m1.1.1">13.73</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.67.67.6.m1.1c">13.73</annotation></semantics></math></td>
</tr>
<tr id="S3.T1.73.73" class="ltx_tr">
<td id="S3.T1.73.73.7" class="ltx_td ltx_align_left">H64-MQA-CLA2</td>
<td id="S3.T1.68.68.1" class="ltx_td ltx_align_left"><math id="S3.T1.68.68.1.m1.1" class="ltx_Math" alttext="64" display="inline"><semantics id="S3.T1.68.68.1.m1.1a"><mn id="S3.T1.68.68.1.m1.1.1" xref="S3.T1.68.68.1.m1.1.1.cmml">64</mn><annotation-xml encoding="MathML-Content" id="S3.T1.68.68.1.m1.1b"><cn type="integer" id="S3.T1.68.68.1.m1.1.1.cmml" xref="S3.T1.68.68.1.m1.1.1">64</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.68.68.1.m1.1c">64</annotation></semantics></math></td>
<td id="S3.T1.69.69.2" class="ltx_td ltx_align_left"><math id="S3.T1.69.69.2.m1.1" class="ltx_Math" alttext="32" display="inline"><semantics id="S3.T1.69.69.2.m1.1a"><mn id="S3.T1.69.69.2.m1.1.1" xref="S3.T1.69.69.2.m1.1.1.cmml">32</mn><annotation-xml encoding="MathML-Content" id="S3.T1.69.69.2.m1.1b"><cn type="integer" id="S3.T1.69.69.2.m1.1.1.cmml" xref="S3.T1.69.69.2.m1.1.1">32</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.69.69.2.m1.1c">32</annotation></semantics></math></td>
<td id="S3.T1.70.70.3" class="ltx_td ltx_align_left"><math id="S3.T1.70.70.3.m1.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S3.T1.70.70.3.m1.1a"><mn id="S3.T1.70.70.3.m1.1.1" xref="S3.T1.70.70.3.m1.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S3.T1.70.70.3.m1.1b"><cn type="integer" id="S3.T1.70.70.3.m1.1.1.cmml" xref="S3.T1.70.70.3.m1.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.70.70.3.m1.1c">1</annotation></semantics></math></td>
<td id="S3.T1.71.71.4" class="ltx_td ltx_align_left"><math id="S3.T1.71.71.4.m1.1" class="ltx_Math" alttext="10" display="inline"><semantics id="S3.T1.71.71.4.m1.1a"><mn id="S3.T1.71.71.4.m1.1.1" xref="S3.T1.71.71.4.m1.1.1.cmml">10</mn><annotation-xml encoding="MathML-Content" id="S3.T1.71.71.4.m1.1b"><cn type="integer" id="S3.T1.71.71.4.m1.1.1.cmml" xref="S3.T1.71.71.4.m1.1.1">10</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.71.71.4.m1.1c">10</annotation></semantics></math></td>
<td id="S3.T1.72.72.5" class="ltx_td ltx_align_left"><math id="S3.T1.72.72.5.m1.1" class="ltx_Math" alttext="2560" display="inline"><semantics id="S3.T1.72.72.5.m1.1a"><mn id="S3.T1.72.72.5.m1.1.1" xref="S3.T1.72.72.5.m1.1.1.cmml">2560</mn><annotation-xml encoding="MathML-Content" id="S3.T1.72.72.5.m1.1b"><cn type="integer" id="S3.T1.72.72.5.m1.1.1.cmml" xref="S3.T1.72.72.5.m1.1.1">2560</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.72.72.5.m1.1c">2560</annotation></semantics></math></td>
<td id="S3.T1.73.73.6" class="ltx_td ltx_align_left"><math id="S3.T1.73.73.6.m1.1" class="ltx_Math" alttext="13.89" display="inline"><semantics id="S3.T1.73.73.6.m1.1a"><mn id="S3.T1.73.73.6.m1.1.1" xref="S3.T1.73.73.6.m1.1.1.cmml">13.89</mn><annotation-xml encoding="MathML-Content" id="S3.T1.73.73.6.m1.1b"><cn type="float" id="S3.T1.73.73.6.m1.1.1.cmml" xref="S3.T1.73.73.6.m1.1.1">13.89</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.73.73.6.m1.1c">13.89</annotation></semantics></math></td>
</tr>
<tr id="S3.T1.122.125.3" class="ltx_tr">
<td id="S3.T1.122.125.3.1" class="ltx_td ltx_align_left ltx_border_t" colspan="2"><span id="S3.T1.122.125.3.1.1" class="ltx_text ltx_font_bold">GQA + CLA2 Models</span></td>
<td id="S3.T1.122.125.3.2" class="ltx_td ltx_border_t"></td>
<td id="S3.T1.122.125.3.3" class="ltx_td ltx_border_t"></td>
<td id="S3.T1.122.125.3.4" class="ltx_td ltx_border_t"></td>
<td id="S3.T1.122.125.3.5" class="ltx_td ltx_border_t"></td>
<td id="S3.T1.122.125.3.6" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="S3.T1.79.79" class="ltx_tr">
<td id="S3.T1.79.79.7" class="ltx_td ltx_align_left ltx_border_t">H256-GQA4-CLA2</td>
<td id="S3.T1.74.74.1" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.74.74.1.m1.1" class="ltx_Math" alttext="256" display="inline"><semantics id="S3.T1.74.74.1.m1.1a"><mn id="S3.T1.74.74.1.m1.1.1" xref="S3.T1.74.74.1.m1.1.1.cmml">256</mn><annotation-xml encoding="MathML-Content" id="S3.T1.74.74.1.m1.1b"><cn type="integer" id="S3.T1.74.74.1.m1.1.1.cmml" xref="S3.T1.74.74.1.m1.1.1">256</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.74.74.1.m1.1c">256</annotation></semantics></math></td>
<td id="S3.T1.75.75.2" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.75.75.2.m1.1" class="ltx_Math" alttext="8" display="inline"><semantics id="S3.T1.75.75.2.m1.1a"><mn id="S3.T1.75.75.2.m1.1.1" xref="S3.T1.75.75.2.m1.1.1.cmml">8</mn><annotation-xml encoding="MathML-Content" id="S3.T1.75.75.2.m1.1b"><cn type="integer" id="S3.T1.75.75.2.m1.1.1.cmml" xref="S3.T1.75.75.2.m1.1.1">8</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.75.75.2.m1.1c">8</annotation></semantics></math></td>
<td id="S3.T1.76.76.3" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.76.76.3.m1.1" class="ltx_Math" alttext="4" display="inline"><semantics id="S3.T1.76.76.3.m1.1a"><mn id="S3.T1.76.76.3.m1.1.1" xref="S3.T1.76.76.3.m1.1.1.cmml">4</mn><annotation-xml encoding="MathML-Content" id="S3.T1.76.76.3.m1.1b"><cn type="integer" id="S3.T1.76.76.3.m1.1.1.cmml" xref="S3.T1.76.76.3.m1.1.1">4</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.76.76.3.m1.1c">4</annotation></semantics></math></td>
<td id="S3.T1.77.77.4" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.77.77.4.m1.1" class="ltx_Math" alttext="10" display="inline"><semantics id="S3.T1.77.77.4.m1.1a"><mn id="S3.T1.77.77.4.m1.1.1" xref="S3.T1.77.77.4.m1.1.1.cmml">10</mn><annotation-xml encoding="MathML-Content" id="S3.T1.77.77.4.m1.1b"><cn type="integer" id="S3.T1.77.77.4.m1.1.1.cmml" xref="S3.T1.77.77.4.m1.1.1">10</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.77.77.4.m1.1c">10</annotation></semantics></math></td>
<td id="S3.T1.78.78.5" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.78.78.5.m1.1" class="ltx_Math" alttext="40\,960" display="inline"><semantics id="S3.T1.78.78.5.m1.1a"><mn id="S3.T1.78.78.5.m1.1.1" xref="S3.T1.78.78.5.m1.1.1.cmml">40 960</mn><annotation-xml encoding="MathML-Content" id="S3.T1.78.78.5.m1.1b"><cn type="integer" id="S3.T1.78.78.5.m1.1.1.cmml" xref="S3.T1.78.78.5.m1.1.1">40960</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.78.78.5.m1.1c">40\,960</annotation></semantics></math></td>
<td id="S3.T1.79.79.6" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.79.79.6.m1.1" class="ltx_Math" alttext="13.38" display="inline"><semantics id="S3.T1.79.79.6.m1.1a"><mn id="S3.T1.79.79.6.m1.1.1" xref="S3.T1.79.79.6.m1.1.1.cmml">13.38</mn><annotation-xml encoding="MathML-Content" id="S3.T1.79.79.6.m1.1b"><cn type="float" id="S3.T1.79.79.6.m1.1.1.cmml" xref="S3.T1.79.79.6.m1.1.1">13.38</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.79.79.6.m1.1c">13.38</annotation></semantics></math></td>
</tr>
<tr id="S3.T1.85.85" class="ltx_tr">
<td id="S3.T1.85.85.7" class="ltx_td ltx_align_left">H128-GQA4-CLA2</td>
<td id="S3.T1.80.80.1" class="ltx_td ltx_align_left"><math id="S3.T1.80.80.1.m1.1" class="ltx_Math" alttext="128" display="inline"><semantics id="S3.T1.80.80.1.m1.1a"><mn id="S3.T1.80.80.1.m1.1.1" xref="S3.T1.80.80.1.m1.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="S3.T1.80.80.1.m1.1b"><cn type="integer" id="S3.T1.80.80.1.m1.1.1.cmml" xref="S3.T1.80.80.1.m1.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.80.80.1.m1.1c">128</annotation></semantics></math></td>
<td id="S3.T1.81.81.2" class="ltx_td ltx_align_left"><math id="S3.T1.81.81.2.m1.1" class="ltx_Math" alttext="16" display="inline"><semantics id="S3.T1.81.81.2.m1.1a"><mn id="S3.T1.81.81.2.m1.1.1" xref="S3.T1.81.81.2.m1.1.1.cmml">16</mn><annotation-xml encoding="MathML-Content" id="S3.T1.81.81.2.m1.1b"><cn type="integer" id="S3.T1.81.81.2.m1.1.1.cmml" xref="S3.T1.81.81.2.m1.1.1">16</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.81.81.2.m1.1c">16</annotation></semantics></math></td>
<td id="S3.T1.82.82.3" class="ltx_td ltx_align_left"><math id="S3.T1.82.82.3.m1.1" class="ltx_Math" alttext="4" display="inline"><semantics id="S3.T1.82.82.3.m1.1a"><mn id="S3.T1.82.82.3.m1.1.1" xref="S3.T1.82.82.3.m1.1.1.cmml">4</mn><annotation-xml encoding="MathML-Content" id="S3.T1.82.82.3.m1.1b"><cn type="integer" id="S3.T1.82.82.3.m1.1.1.cmml" xref="S3.T1.82.82.3.m1.1.1">4</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.82.82.3.m1.1c">4</annotation></semantics></math></td>
<td id="S3.T1.83.83.4" class="ltx_td ltx_align_left"><math id="S3.T1.83.83.4.m1.1" class="ltx_Math" alttext="10" display="inline"><semantics id="S3.T1.83.83.4.m1.1a"><mn id="S3.T1.83.83.4.m1.1.1" xref="S3.T1.83.83.4.m1.1.1.cmml">10</mn><annotation-xml encoding="MathML-Content" id="S3.T1.83.83.4.m1.1b"><cn type="integer" id="S3.T1.83.83.4.m1.1.1.cmml" xref="S3.T1.83.83.4.m1.1.1">10</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.83.83.4.m1.1c">10</annotation></semantics></math></td>
<td id="S3.T1.84.84.5" class="ltx_td ltx_align_left"><math id="S3.T1.84.84.5.m1.1" class="ltx_Math" alttext="20\,480" display="inline"><semantics id="S3.T1.84.84.5.m1.1a"><mn id="S3.T1.84.84.5.m1.1.1" xref="S3.T1.84.84.5.m1.1.1.cmml">20 480</mn><annotation-xml encoding="MathML-Content" id="S3.T1.84.84.5.m1.1b"><cn type="integer" id="S3.T1.84.84.5.m1.1.1.cmml" xref="S3.T1.84.84.5.m1.1.1">20480</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.84.84.5.m1.1c">20\,480</annotation></semantics></math></td>
<td id="S3.T1.85.85.6" class="ltx_td ltx_align_left"><math id="S3.T1.85.85.6.m1.1" class="ltx_Math" alttext="13.48" display="inline"><semantics id="S3.T1.85.85.6.m1.1a"><mn id="S3.T1.85.85.6.m1.1.1" xref="S3.T1.85.85.6.m1.1.1.cmml">13.48</mn><annotation-xml encoding="MathML-Content" id="S3.T1.85.85.6.m1.1b"><cn type="float" id="S3.T1.85.85.6.m1.1.1.cmml" xref="S3.T1.85.85.6.m1.1.1">13.48</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.85.85.6.m1.1c">13.48</annotation></semantics></math></td>
</tr>
<tr id="S3.T1.91.91" class="ltx_tr">
<td id="S3.T1.91.91.7" class="ltx_td ltx_align_left">H128-GQA2-CLA2</td>
<td id="S3.T1.86.86.1" class="ltx_td ltx_align_left"><math id="S3.T1.86.86.1.m1.1" class="ltx_Math" alttext="128" display="inline"><semantics id="S3.T1.86.86.1.m1.1a"><mn id="S3.T1.86.86.1.m1.1.1" xref="S3.T1.86.86.1.m1.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="S3.T1.86.86.1.m1.1b"><cn type="integer" id="S3.T1.86.86.1.m1.1.1.cmml" xref="S3.T1.86.86.1.m1.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.86.86.1.m1.1c">128</annotation></semantics></math></td>
<td id="S3.T1.87.87.2" class="ltx_td ltx_align_left"><math id="S3.T1.87.87.2.m1.1" class="ltx_Math" alttext="16" display="inline"><semantics id="S3.T1.87.87.2.m1.1a"><mn id="S3.T1.87.87.2.m1.1.1" xref="S3.T1.87.87.2.m1.1.1.cmml">16</mn><annotation-xml encoding="MathML-Content" id="S3.T1.87.87.2.m1.1b"><cn type="integer" id="S3.T1.87.87.2.m1.1.1.cmml" xref="S3.T1.87.87.2.m1.1.1">16</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.87.87.2.m1.1c">16</annotation></semantics></math></td>
<td id="S3.T1.88.88.3" class="ltx_td ltx_align_left"><math id="S3.T1.88.88.3.m1.1" class="ltx_Math" alttext="2" display="inline"><semantics id="S3.T1.88.88.3.m1.1a"><mn id="S3.T1.88.88.3.m1.1.1" xref="S3.T1.88.88.3.m1.1.1.cmml">2</mn><annotation-xml encoding="MathML-Content" id="S3.T1.88.88.3.m1.1b"><cn type="integer" id="S3.T1.88.88.3.m1.1.1.cmml" xref="S3.T1.88.88.3.m1.1.1">2</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.88.88.3.m1.1c">2</annotation></semantics></math></td>
<td id="S3.T1.89.89.4" class="ltx_td ltx_align_left"><math id="S3.T1.89.89.4.m1.1" class="ltx_Math" alttext="10" display="inline"><semantics id="S3.T1.89.89.4.m1.1a"><mn id="S3.T1.89.89.4.m1.1.1" xref="S3.T1.89.89.4.m1.1.1.cmml">10</mn><annotation-xml encoding="MathML-Content" id="S3.T1.89.89.4.m1.1b"><cn type="integer" id="S3.T1.89.89.4.m1.1.1.cmml" xref="S3.T1.89.89.4.m1.1.1">10</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.89.89.4.m1.1c">10</annotation></semantics></math></td>
<td id="S3.T1.90.90.5" class="ltx_td ltx_align_left"><math id="S3.T1.90.90.5.m1.1" class="ltx_Math" alttext="10\,240" display="inline"><semantics id="S3.T1.90.90.5.m1.1a"><mn id="S3.T1.90.90.5.m1.1.1" xref="S3.T1.90.90.5.m1.1.1.cmml">10 240</mn><annotation-xml encoding="MathML-Content" id="S3.T1.90.90.5.m1.1b"><cn type="integer" id="S3.T1.90.90.5.m1.1.1.cmml" xref="S3.T1.90.90.5.m1.1.1">10240</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.90.90.5.m1.1c">10\,240</annotation></semantics></math></td>
<td id="S3.T1.91.91.6" class="ltx_td ltx_align_left"><math id="S3.T1.91.91.6.m1.1" class="ltx_Math" alttext="13.59" display="inline"><semantics id="S3.T1.91.91.6.m1.1a"><mn id="S3.T1.91.91.6.m1.1.1" xref="S3.T1.91.91.6.m1.1.1.cmml">13.59</mn><annotation-xml encoding="MathML-Content" id="S3.T1.91.91.6.m1.1b"><cn type="float" id="S3.T1.91.91.6.m1.1.1.cmml" xref="S3.T1.91.91.6.m1.1.1">13.59</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.91.91.6.m1.1c">13.59</annotation></semantics></math></td>
</tr>
<tr id="S3.T1.92.92" class="ltx_tr">
<td id="S3.T1.92.92.1" class="ltx_td ltx_align_left ltx_border_t" colspan="2"><span id="S3.T1.92.92.1.1" class="ltx_text ltx_font_bold">MQA + CLA <math id="S3.T1.92.92.1.1.m1.1" class="ltx_Math" alttext="\mathbf{&gt;2}" display="inline"><semantics id="S3.T1.92.92.1.1.m1.1a"><mrow id="S3.T1.92.92.1.1.m1.1.1" xref="S3.T1.92.92.1.1.m1.1.1.cmml"><mi id="S3.T1.92.92.1.1.m1.1.1.2" xref="S3.T1.92.92.1.1.m1.1.1.2.cmml"></mi><mo id="S3.T1.92.92.1.1.m1.1.1.1" xref="S3.T1.92.92.1.1.m1.1.1.1.cmml">&gt;</mo><mn id="S3.T1.92.92.1.1.m1.1.1.3" xref="S3.T1.92.92.1.1.m1.1.1.3.cmml">𝟐</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.T1.92.92.1.1.m1.1b"><apply id="S3.T1.92.92.1.1.m1.1.1.cmml" xref="S3.T1.92.92.1.1.m1.1.1"><gt id="S3.T1.92.92.1.1.m1.1.1.1.cmml" xref="S3.T1.92.92.1.1.m1.1.1.1"></gt><csymbol cd="latexml" id="S3.T1.92.92.1.1.m1.1.1.2.cmml" xref="S3.T1.92.92.1.1.m1.1.1.2">absent</csymbol><cn type="integer" id="S3.T1.92.92.1.1.m1.1.1.3.cmml" xref="S3.T1.92.92.1.1.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.92.92.1.1.m1.1c">\mathbf{&gt;2}</annotation></semantics></math> Models</span></td>
<td id="S3.T1.92.92.2" class="ltx_td ltx_border_t"></td>
<td id="S3.T1.92.92.3" class="ltx_td ltx_border_t"></td>
<td id="S3.T1.92.92.4" class="ltx_td ltx_border_t"></td>
<td id="S3.T1.92.92.5" class="ltx_td ltx_border_t"></td>
<td id="S3.T1.92.92.6" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="S3.T1.98.98" class="ltx_tr">
<td id="S3.T1.98.98.7" class="ltx_td ltx_align_left ltx_border_t">H128-MQA-CLA3</td>
<td id="S3.T1.93.93.1" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.93.93.1.m1.1" class="ltx_Math" alttext="128" display="inline"><semantics id="S3.T1.93.93.1.m1.1a"><mn id="S3.T1.93.93.1.m1.1.1" xref="S3.T1.93.93.1.m1.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="S3.T1.93.93.1.m1.1b"><cn type="integer" id="S3.T1.93.93.1.m1.1.1.cmml" xref="S3.T1.93.93.1.m1.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.93.93.1.m1.1c">128</annotation></semantics></math></td>
<td id="S3.T1.94.94.2" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.94.94.2.m1.1" class="ltx_Math" alttext="16" display="inline"><semantics id="S3.T1.94.94.2.m1.1a"><mn id="S3.T1.94.94.2.m1.1.1" xref="S3.T1.94.94.2.m1.1.1.cmml">16</mn><annotation-xml encoding="MathML-Content" id="S3.T1.94.94.2.m1.1b"><cn type="integer" id="S3.T1.94.94.2.m1.1.1.cmml" xref="S3.T1.94.94.2.m1.1.1">16</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.94.94.2.m1.1c">16</annotation></semantics></math></td>
<td id="S3.T1.95.95.3" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.95.95.3.m1.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S3.T1.95.95.3.m1.1a"><mn id="S3.T1.95.95.3.m1.1.1" xref="S3.T1.95.95.3.m1.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S3.T1.95.95.3.m1.1b"><cn type="integer" id="S3.T1.95.95.3.m1.1.1.cmml" xref="S3.T1.95.95.3.m1.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.95.95.3.m1.1c">1</annotation></semantics></math></td>
<td id="S3.T1.96.96.4" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.96.96.4.m1.1" class="ltx_Math" alttext="7" display="inline"><semantics id="S3.T1.96.96.4.m1.1a"><mn id="S3.T1.96.96.4.m1.1.1" xref="S3.T1.96.96.4.m1.1.1.cmml">7</mn><annotation-xml encoding="MathML-Content" id="S3.T1.96.96.4.m1.1b"><cn type="integer" id="S3.T1.96.96.4.m1.1.1.cmml" xref="S3.T1.96.96.4.m1.1.1">7</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.96.96.4.m1.1c">7</annotation></semantics></math></td>
<td id="S3.T1.97.97.5" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.97.97.5.m1.1" class="ltx_Math" alttext="3584" display="inline"><semantics id="S3.T1.97.97.5.m1.1a"><mn id="S3.T1.97.97.5.m1.1.1" xref="S3.T1.97.97.5.m1.1.1.cmml">3584</mn><annotation-xml encoding="MathML-Content" id="S3.T1.97.97.5.m1.1b"><cn type="integer" id="S3.T1.97.97.5.m1.1.1.cmml" xref="S3.T1.97.97.5.m1.1.1">3584</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.97.97.5.m1.1c">3584</annotation></semantics></math></td>
<td id="S3.T1.98.98.6" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.98.98.6.m1.1" class="ltx_Math" alttext="13.77" display="inline"><semantics id="S3.T1.98.98.6.m1.1a"><mn id="S3.T1.98.98.6.m1.1.1" xref="S3.T1.98.98.6.m1.1.1.cmml">13.77</mn><annotation-xml encoding="MathML-Content" id="S3.T1.98.98.6.m1.1b"><cn type="float" id="S3.T1.98.98.6.m1.1.1.cmml" xref="S3.T1.98.98.6.m1.1.1">13.77</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.98.98.6.m1.1c">13.77</annotation></semantics></math></td>
</tr>
<tr id="S3.T1.104.104" class="ltx_tr">
<td id="S3.T1.104.104.7" class="ltx_td ltx_align_left">H128-MQA-CLA4</td>
<td id="S3.T1.99.99.1" class="ltx_td ltx_align_left"><math id="S3.T1.99.99.1.m1.1" class="ltx_Math" alttext="128" display="inline"><semantics id="S3.T1.99.99.1.m1.1a"><mn id="S3.T1.99.99.1.m1.1.1" xref="S3.T1.99.99.1.m1.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="S3.T1.99.99.1.m1.1b"><cn type="integer" id="S3.T1.99.99.1.m1.1.1.cmml" xref="S3.T1.99.99.1.m1.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.99.99.1.m1.1c">128</annotation></semantics></math></td>
<td id="S3.T1.100.100.2" class="ltx_td ltx_align_left"><math id="S3.T1.100.100.2.m1.1" class="ltx_Math" alttext="16" display="inline"><semantics id="S3.T1.100.100.2.m1.1a"><mn id="S3.T1.100.100.2.m1.1.1" xref="S3.T1.100.100.2.m1.1.1.cmml">16</mn><annotation-xml encoding="MathML-Content" id="S3.T1.100.100.2.m1.1b"><cn type="integer" id="S3.T1.100.100.2.m1.1.1.cmml" xref="S3.T1.100.100.2.m1.1.1">16</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.100.100.2.m1.1c">16</annotation></semantics></math></td>
<td id="S3.T1.101.101.3" class="ltx_td ltx_align_left"><math id="S3.T1.101.101.3.m1.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S3.T1.101.101.3.m1.1a"><mn id="S3.T1.101.101.3.m1.1.1" xref="S3.T1.101.101.3.m1.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S3.T1.101.101.3.m1.1b"><cn type="integer" id="S3.T1.101.101.3.m1.1.1.cmml" xref="S3.T1.101.101.3.m1.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.101.101.3.m1.1c">1</annotation></semantics></math></td>
<td id="S3.T1.102.102.4" class="ltx_td ltx_align_left"><math id="S3.T1.102.102.4.m1.1" class="ltx_Math" alttext="5" display="inline"><semantics id="S3.T1.102.102.4.m1.1a"><mn id="S3.T1.102.102.4.m1.1.1" xref="S3.T1.102.102.4.m1.1.1.cmml">5</mn><annotation-xml encoding="MathML-Content" id="S3.T1.102.102.4.m1.1b"><cn type="integer" id="S3.T1.102.102.4.m1.1.1.cmml" xref="S3.T1.102.102.4.m1.1.1">5</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.102.102.4.m1.1c">5</annotation></semantics></math></td>
<td id="S3.T1.103.103.5" class="ltx_td ltx_align_left"><math id="S3.T1.103.103.5.m1.1" class="ltx_Math" alttext="2560" display="inline"><semantics id="S3.T1.103.103.5.m1.1a"><mn id="S3.T1.103.103.5.m1.1.1" xref="S3.T1.103.103.5.m1.1.1.cmml">2560</mn><annotation-xml encoding="MathML-Content" id="S3.T1.103.103.5.m1.1b"><cn type="integer" id="S3.T1.103.103.5.m1.1.1.cmml" xref="S3.T1.103.103.5.m1.1.1">2560</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.103.103.5.m1.1c">2560</annotation></semantics></math></td>
<td id="S3.T1.104.104.6" class="ltx_td ltx_align_left"><math id="S3.T1.104.104.6.m1.1" class="ltx_Math" alttext="13.95" display="inline"><semantics id="S3.T1.104.104.6.m1.1a"><mn id="S3.T1.104.104.6.m1.1.1" xref="S3.T1.104.104.6.m1.1.1.cmml">13.95</mn><annotation-xml encoding="MathML-Content" id="S3.T1.104.104.6.m1.1b"><cn type="float" id="S3.T1.104.104.6.m1.1.1.cmml" xref="S3.T1.104.104.6.m1.1.1">13.95</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.104.104.6.m1.1c">13.95</annotation></semantics></math></td>
</tr>
<tr id="S3.T1.122.126.4" class="ltx_tr">
<td id="S3.T1.122.126.4.1" class="ltx_td ltx_align_left ltx_border_t" colspan="2"><span id="S3.T1.122.126.4.1.1" class="ltx_text ltx_font_bold">MQA + CLA2, Non-Uniform Sharing</span></td>
<td id="S3.T1.122.126.4.2" class="ltx_td ltx_border_t"></td>
<td id="S3.T1.122.126.4.3" class="ltx_td ltx_border_t"></td>
<td id="S3.T1.122.126.4.4" class="ltx_td ltx_border_t"></td>
<td id="S3.T1.122.126.4.5" class="ltx_td ltx_border_t"></td>
<td id="S3.T1.122.126.4.6" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="S3.T1.110.110" class="ltx_tr">
<td id="S3.T1.110.110.7" class="ltx_td ltx_align_left ltx_border_t">H128-MQA-CLA2-KeepEnds</td>
<td id="S3.T1.105.105.1" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.105.105.1.m1.1" class="ltx_Math" alttext="128" display="inline"><semantics id="S3.T1.105.105.1.m1.1a"><mn id="S3.T1.105.105.1.m1.1.1" xref="S3.T1.105.105.1.m1.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="S3.T1.105.105.1.m1.1b"><cn type="integer" id="S3.T1.105.105.1.m1.1.1.cmml" xref="S3.T1.105.105.1.m1.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.105.105.1.m1.1c">128</annotation></semantics></math></td>
<td id="S3.T1.106.106.2" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.106.106.2.m1.1" class="ltx_Math" alttext="16" display="inline"><semantics id="S3.T1.106.106.2.m1.1a"><mn id="S3.T1.106.106.2.m1.1.1" xref="S3.T1.106.106.2.m1.1.1.cmml">16</mn><annotation-xml encoding="MathML-Content" id="S3.T1.106.106.2.m1.1b"><cn type="integer" id="S3.T1.106.106.2.m1.1.1.cmml" xref="S3.T1.106.106.2.m1.1.1">16</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.106.106.2.m1.1c">16</annotation></semantics></math></td>
<td id="S3.T1.107.107.3" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.107.107.3.m1.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S3.T1.107.107.3.m1.1a"><mn id="S3.T1.107.107.3.m1.1.1" xref="S3.T1.107.107.3.m1.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S3.T1.107.107.3.m1.1b"><cn type="integer" id="S3.T1.107.107.3.m1.1.1.cmml" xref="S3.T1.107.107.3.m1.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.107.107.3.m1.1c">1</annotation></semantics></math></td>
<td id="S3.T1.108.108.4" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.108.108.4.m1.1" class="ltx_Math" alttext="11" display="inline"><semantics id="S3.T1.108.108.4.m1.1a"><mn id="S3.T1.108.108.4.m1.1.1" xref="S3.T1.108.108.4.m1.1.1.cmml">11</mn><annotation-xml encoding="MathML-Content" id="S3.T1.108.108.4.m1.1b"><cn type="integer" id="S3.T1.108.108.4.m1.1.1.cmml" xref="S3.T1.108.108.4.m1.1.1">11</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.108.108.4.m1.1c">11</annotation></semantics></math></td>
<td id="S3.T1.109.109.5" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.109.109.5.m1.1" class="ltx_Math" alttext="5632" display="inline"><semantics id="S3.T1.109.109.5.m1.1a"><mn id="S3.T1.109.109.5.m1.1.1" xref="S3.T1.109.109.5.m1.1.1.cmml">5632</mn><annotation-xml encoding="MathML-Content" id="S3.T1.109.109.5.m1.1b"><cn type="integer" id="S3.T1.109.109.5.m1.1.1.cmml" xref="S3.T1.109.109.5.m1.1.1">5632</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.109.109.5.m1.1c">5632</annotation></semantics></math></td>
<td id="S3.T1.110.110.6" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T1.110.110.6.m1.1" class="ltx_Math" alttext="13.62" display="inline"><semantics id="S3.T1.110.110.6.m1.1a"><mn id="S3.T1.110.110.6.m1.1.1" xref="S3.T1.110.110.6.m1.1.1.cmml">13.62</mn><annotation-xml encoding="MathML-Content" id="S3.T1.110.110.6.m1.1b"><cn type="float" id="S3.T1.110.110.6.m1.1.1.cmml" xref="S3.T1.110.110.6.m1.1.1">13.62</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.110.110.6.m1.1c">13.62</annotation></semantics></math></td>
</tr>
<tr id="S3.T1.116.116" class="ltx_tr">
<td id="S3.T1.116.116.7" class="ltx_td ltx_align_left">H128-MQA-CLA2-DenseFront</td>
<td id="S3.T1.111.111.1" class="ltx_td ltx_align_left"><math id="S3.T1.111.111.1.m1.1" class="ltx_Math" alttext="128" display="inline"><semantics id="S3.T1.111.111.1.m1.1a"><mn id="S3.T1.111.111.1.m1.1.1" xref="S3.T1.111.111.1.m1.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="S3.T1.111.111.1.m1.1b"><cn type="integer" id="S3.T1.111.111.1.m1.1.1.cmml" xref="S3.T1.111.111.1.m1.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.111.111.1.m1.1c">128</annotation></semantics></math></td>
<td id="S3.T1.112.112.2" class="ltx_td ltx_align_left"><math id="S3.T1.112.112.2.m1.1" class="ltx_Math" alttext="16" display="inline"><semantics id="S3.T1.112.112.2.m1.1a"><mn id="S3.T1.112.112.2.m1.1.1" xref="S3.T1.112.112.2.m1.1.1.cmml">16</mn><annotation-xml encoding="MathML-Content" id="S3.T1.112.112.2.m1.1b"><cn type="integer" id="S3.T1.112.112.2.m1.1.1.cmml" xref="S3.T1.112.112.2.m1.1.1">16</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.112.112.2.m1.1c">16</annotation></semantics></math></td>
<td id="S3.T1.113.113.3" class="ltx_td ltx_align_left"><math id="S3.T1.113.113.3.m1.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S3.T1.113.113.3.m1.1a"><mn id="S3.T1.113.113.3.m1.1.1" xref="S3.T1.113.113.3.m1.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S3.T1.113.113.3.m1.1b"><cn type="integer" id="S3.T1.113.113.3.m1.1.1.cmml" xref="S3.T1.113.113.3.m1.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.113.113.3.m1.1c">1</annotation></semantics></math></td>
<td id="S3.T1.114.114.4" class="ltx_td ltx_align_left"><math id="S3.T1.114.114.4.m1.1" class="ltx_Math" alttext="11" display="inline"><semantics id="S3.T1.114.114.4.m1.1a"><mn id="S3.T1.114.114.4.m1.1.1" xref="S3.T1.114.114.4.m1.1.1.cmml">11</mn><annotation-xml encoding="MathML-Content" id="S3.T1.114.114.4.m1.1b"><cn type="integer" id="S3.T1.114.114.4.m1.1.1.cmml" xref="S3.T1.114.114.4.m1.1.1">11</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.114.114.4.m1.1c">11</annotation></semantics></math></td>
<td id="S3.T1.115.115.5" class="ltx_td ltx_align_left"><math id="S3.T1.115.115.5.m1.1" class="ltx_Math" alttext="5632" display="inline"><semantics id="S3.T1.115.115.5.m1.1a"><mn id="S3.T1.115.115.5.m1.1.1" xref="S3.T1.115.115.5.m1.1.1.cmml">5632</mn><annotation-xml encoding="MathML-Content" id="S3.T1.115.115.5.m1.1b"><cn type="integer" id="S3.T1.115.115.5.m1.1.1.cmml" xref="S3.T1.115.115.5.m1.1.1">5632</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.115.115.5.m1.1c">5632</annotation></semantics></math></td>
<td id="S3.T1.116.116.6" class="ltx_td ltx_align_left"><math id="S3.T1.116.116.6.m1.1" class="ltx_Math" alttext="13.75" display="inline"><semantics id="S3.T1.116.116.6.m1.1a"><mn id="S3.T1.116.116.6.m1.1.1" xref="S3.T1.116.116.6.m1.1.1.cmml">13.75</mn><annotation-xml encoding="MathML-Content" id="S3.T1.116.116.6.m1.1b"><cn type="float" id="S3.T1.116.116.6.m1.1.1.cmml" xref="S3.T1.116.116.6.m1.1.1">13.75</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.116.116.6.m1.1c">13.75</annotation></semantics></math></td>
</tr>
<tr id="S3.T1.122.122" class="ltx_tr">
<td id="S3.T1.122.122.7" class="ltx_td ltx_align_left ltx_border_bb">H128-MQA-CLA2-DenseBack</td>
<td id="S3.T1.117.117.1" class="ltx_td ltx_align_left ltx_border_bb"><math id="S3.T1.117.117.1.m1.1" class="ltx_Math" alttext="128" display="inline"><semantics id="S3.T1.117.117.1.m1.1a"><mn id="S3.T1.117.117.1.m1.1.1" xref="S3.T1.117.117.1.m1.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="S3.T1.117.117.1.m1.1b"><cn type="integer" id="S3.T1.117.117.1.m1.1.1.cmml" xref="S3.T1.117.117.1.m1.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.117.117.1.m1.1c">128</annotation></semantics></math></td>
<td id="S3.T1.118.118.2" class="ltx_td ltx_align_left ltx_border_bb"><math id="S3.T1.118.118.2.m1.1" class="ltx_Math" alttext="16" display="inline"><semantics id="S3.T1.118.118.2.m1.1a"><mn id="S3.T1.118.118.2.m1.1.1" xref="S3.T1.118.118.2.m1.1.1.cmml">16</mn><annotation-xml encoding="MathML-Content" id="S3.T1.118.118.2.m1.1b"><cn type="integer" id="S3.T1.118.118.2.m1.1.1.cmml" xref="S3.T1.118.118.2.m1.1.1">16</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.118.118.2.m1.1c">16</annotation></semantics></math></td>
<td id="S3.T1.119.119.3" class="ltx_td ltx_align_left ltx_border_bb"><math id="S3.T1.119.119.3.m1.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S3.T1.119.119.3.m1.1a"><mn id="S3.T1.119.119.3.m1.1.1" xref="S3.T1.119.119.3.m1.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S3.T1.119.119.3.m1.1b"><cn type="integer" id="S3.T1.119.119.3.m1.1.1.cmml" xref="S3.T1.119.119.3.m1.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.119.119.3.m1.1c">1</annotation></semantics></math></td>
<td id="S3.T1.120.120.4" class="ltx_td ltx_align_left ltx_border_bb"><math id="S3.T1.120.120.4.m1.1" class="ltx_Math" alttext="11" display="inline"><semantics id="S3.T1.120.120.4.m1.1a"><mn id="S3.T1.120.120.4.m1.1.1" xref="S3.T1.120.120.4.m1.1.1.cmml">11</mn><annotation-xml encoding="MathML-Content" id="S3.T1.120.120.4.m1.1b"><cn type="integer" id="S3.T1.120.120.4.m1.1.1.cmml" xref="S3.T1.120.120.4.m1.1.1">11</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.120.120.4.m1.1c">11</annotation></semantics></math></td>
<td id="S3.T1.121.121.5" class="ltx_td ltx_align_left ltx_border_bb"><math id="S3.T1.121.121.5.m1.1" class="ltx_Math" alttext="5632" display="inline"><semantics id="S3.T1.121.121.5.m1.1a"><mn id="S3.T1.121.121.5.m1.1.1" xref="S3.T1.121.121.5.m1.1.1.cmml">5632</mn><annotation-xml encoding="MathML-Content" id="S3.T1.121.121.5.m1.1b"><cn type="integer" id="S3.T1.121.121.5.m1.1.1.cmml" xref="S3.T1.121.121.5.m1.1.1">5632</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.121.121.5.m1.1c">5632</annotation></semantics></math></td>
<td id="S3.T1.122.122.6" class="ltx_td ltx_align_left ltx_border_bb"><math id="S3.T1.122.122.6.m1.1" class="ltx_Math" alttext="14.03" display="inline"><semantics id="S3.T1.122.122.6.m1.1a"><mn id="S3.T1.122.122.6.m1.1.1" xref="S3.T1.122.122.6.m1.1.1.cmml">14.03</mn><annotation-xml encoding="MathML-Content" id="S3.T1.122.122.6.m1.1b"><cn type="float" id="S3.T1.122.122.6.m1.1.1.cmml" xref="S3.T1.122.122.6.m1.1.1">14.03</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T1.122.122.6.m1.1c">14.03</annotation></semantics></math></td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>Results of our 1B-scale design space exploration.</figcaption>
</figure>
<div id="S3.p1" class="ltx_para">
<p id="S3.p1.2" class="ltx_p">To determine the effect of Cross-Layer Attention on language modeling accuracy, we trained a collection of transformer-based language models from scratch at the <math id="S3.p1.1.m1.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S3.p1.1.m1.1a"><mn id="S3.p1.1.m1.1.1" xref="S3.p1.1.m1.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S3.p1.1.m1.1b"><cn type="integer" id="S3.p1.1.m1.1.1.cmml" xref="S3.p1.1.m1.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.1.m1.1c">1</annotation></semantics></math> billion and <math id="S3.p1.2.m2.1" class="ltx_Math" alttext="3" display="inline"><semantics id="S3.p1.2.m2.1a"><mn id="S3.p1.2.m2.1.1" xref="S3.p1.2.m2.1.1.cmml">3</mn><annotation-xml encoding="MathML-Content" id="S3.p1.2.m2.1b"><cn type="integer" id="S3.p1.2.m2.1.1.cmml" xref="S3.p1.2.m2.1.1">3</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.2.m2.1c">3</annotation></semantics></math> billion parameter scales.
While running these experiments, we sought to answer the following questions:</p>
<ol id="S3.I1" class="ltx_enumerate">
<li id="S3.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S3.I1.i1.p1" class="ltx_para">
<p id="S3.I1.i1.p1.1" class="ltx_p">What accuracy/memory tradeoffs are possible using CLA?</p>
</div>
</li>
<li id="S3.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S3.I1.i2.p1" class="ltx_para">
<p id="S3.I1.i2.p1.1" class="ltx_p">How does using CLA compare to using plain GQA or MQA?</p>
</div>
</li>
<li id="S3.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">3.</span> 
<div id="S3.I1.i3.p1" class="ltx_para">
<p id="S3.I1.i3.p1.1" class="ltx_p">How does CLA interact with GQA and MQA?</p>
</div>
</li>
<li id="S3.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">4.</span> 
<div id="S3.I1.i4.p1" class="ltx_para">
<p id="S3.I1.i4.p1.1" class="ltx_p">What CLA configurations perform best given a fixed memory budget?</p>
</div>
</li>
<li id="S3.I1.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">5.</span> 
<div id="S3.I1.i5.p1" class="ltx_para">
<p id="S3.I1.i5.p1.1" class="ltx_p">Are the effects of CLA consistent across scales?</p>
</div>
</li>
</ol>
<p id="S3.p1.3" class="ltx_p">We found that CLA enables favorable accuracy/memory tradeoffs compared to what is possible using plain GQA or MQA.
Moreover, we found that in our experimental regime, a sharing factor of <math id="S3.p1.3.m1.1" class="ltx_Math" alttext="2" display="inline"><semantics id="S3.p1.3.m1.1a"><mn id="S3.p1.3.m1.1.1" xref="S3.p1.3.m1.1.1.cmml">2</mn><annotation-xml encoding="MathML-Content" id="S3.p1.3.m1.1b"><cn type="integer" id="S3.p1.3.m1.1.1.cmml" xref="S3.p1.3.m1.1.1">2</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.3.m1.1c">2</annotation></semantics></math> is more effective than other sharing factors, and that CLA is consistently effective when combined with MQA when trying to decrease KV cache storage.
We also found preliminary evidence to suggest that CLA models benefit from training with higher learning rates than comparable non-CLA models.
Finally, we found that CLA confers benefits at both 1B- and 3B-parameter scales.
In the rest of this section, we present our experimental setup and results in more detail.</p>
</div>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Common Experimental Parameters</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.3" class="ltx_p">In all our experiments, we train our models from scratch on data from the SlimPajama <cite class="ltx_cite ltx_citemacro_citep">(Soboleva et al., <a href="#bib.bib38" title="" class="ltx_ref">2023</a>)</cite> dataset, tokenized with the GPT-NeoX tokenizer <cite class="ltx_cite ltx_citemacro_citep">(Black et al., <a href="#bib.bib4" title="" class="ltx_ref">2022</a>)</cite> which uses Byte-Pair Encoding (BPE) <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib43" title="" class="ltx_ref">2019</a>)</cite>.
We adopt a Llama-like <cite class="ltx_cite ltx_citemacro_citep">(Touvron et al., <a href="#bib.bib41" title="" class="ltx_ref">2023</a>)</cite> architecture with pre-normalization, SwiGLU activations <cite class="ltx_cite ltx_citemacro_citep">(Shazeer, <a href="#bib.bib35" title="" class="ltx_ref">2020</a>; Ramachandran et al., <a href="#bib.bib33" title="" class="ltx_ref">2017</a>)</cite>, and rotary position embeddings <cite class="ltx_cite ltx_citemacro_citep">(Su et al., <a href="#bib.bib39" title="" class="ltx_ref">2023</a>)</cite>.
We do not use dropout for any of our models. Our models include learnable elementwise affine parameters for layer-norm, and our CLA models use separately-learnable affine layer-norm parameters for the KV projection blocks and Q projection blocks in attention. Unless otherwise stated, we always set the number of query heads <math id="S3.SS1.p1.1.m1.1" class="ltx_Math" alttext="n_{\text{query}}" display="inline"><semantics id="S3.SS1.p1.1.m1.1a"><msub id="S3.SS1.p1.1.m1.1.1" xref="S3.SS1.p1.1.m1.1.1.cmml"><mi id="S3.SS1.p1.1.m1.1.1.2" xref="S3.SS1.p1.1.m1.1.1.2.cmml">n</mi><mtext id="S3.SS1.p1.1.m1.1.1.3" xref="S3.SS1.p1.1.m1.1.1.3a.cmml">query</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.1.m1.1b"><apply id="S3.SS1.p1.1.m1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.1.m1.1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1">subscript</csymbol><ci id="S3.SS1.p1.1.m1.1.1.2.cmml" xref="S3.SS1.p1.1.m1.1.1.2">𝑛</ci><ci id="S3.SS1.p1.1.m1.1.1.3a.cmml" xref="S3.SS1.p1.1.m1.1.1.3"><mtext mathsize="70%" id="S3.SS1.p1.1.m1.1.1.3.cmml" xref="S3.SS1.p1.1.m1.1.1.3">query</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.1.m1.1c">n_{\text{query}}</annotation></semantics></math> such that <math id="S3.SS1.p1.2.m2.1" class="ltx_Math" alttext="n_{\text{query}}\cdot d_{\text{head}}" display="inline"><semantics id="S3.SS1.p1.2.m2.1a"><mrow id="S3.SS1.p1.2.m2.1.1" xref="S3.SS1.p1.2.m2.1.1.cmml"><msub id="S3.SS1.p1.2.m2.1.1.2" xref="S3.SS1.p1.2.m2.1.1.2.cmml"><mi id="S3.SS1.p1.2.m2.1.1.2.2" xref="S3.SS1.p1.2.m2.1.1.2.2.cmml">n</mi><mtext id="S3.SS1.p1.2.m2.1.1.2.3" xref="S3.SS1.p1.2.m2.1.1.2.3a.cmml">query</mtext></msub><mo lspace="0.222em" rspace="0.222em" id="S3.SS1.p1.2.m2.1.1.1" xref="S3.SS1.p1.2.m2.1.1.1.cmml">⋅</mo><msub id="S3.SS1.p1.2.m2.1.1.3" xref="S3.SS1.p1.2.m2.1.1.3.cmml"><mi id="S3.SS1.p1.2.m2.1.1.3.2" xref="S3.SS1.p1.2.m2.1.1.3.2.cmml">d</mi><mtext id="S3.SS1.p1.2.m2.1.1.3.3" xref="S3.SS1.p1.2.m2.1.1.3.3a.cmml">head</mtext></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.2.m2.1b"><apply id="S3.SS1.p1.2.m2.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1"><ci id="S3.SS1.p1.2.m2.1.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1.1">⋅</ci><apply id="S3.SS1.p1.2.m2.1.1.2.cmml" xref="S3.SS1.p1.2.m2.1.1.2"><csymbol cd="ambiguous" id="S3.SS1.p1.2.m2.1.1.2.1.cmml" xref="S3.SS1.p1.2.m2.1.1.2">subscript</csymbol><ci id="S3.SS1.p1.2.m2.1.1.2.2.cmml" xref="S3.SS1.p1.2.m2.1.1.2.2">𝑛</ci><ci id="S3.SS1.p1.2.m2.1.1.2.3a.cmml" xref="S3.SS1.p1.2.m2.1.1.2.3"><mtext mathsize="70%" id="S3.SS1.p1.2.m2.1.1.2.3.cmml" xref="S3.SS1.p1.2.m2.1.1.2.3">query</mtext></ci></apply><apply id="S3.SS1.p1.2.m2.1.1.3.cmml" xref="S3.SS1.p1.2.m2.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p1.2.m2.1.1.3.1.cmml" xref="S3.SS1.p1.2.m2.1.1.3">subscript</csymbol><ci id="S3.SS1.p1.2.m2.1.1.3.2.cmml" xref="S3.SS1.p1.2.m2.1.1.3.2">𝑑</ci><ci id="S3.SS1.p1.2.m2.1.1.3.3a.cmml" xref="S3.SS1.p1.2.m2.1.1.3.3"><mtext mathsize="70%" id="S3.SS1.p1.2.m2.1.1.3.3.cmml" xref="S3.SS1.p1.2.m2.1.1.3.3">head</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.2.m2.1c">n_{\text{query}}\cdot d_{\text{head}}</annotation></semantics></math> is equal to the hidden size <math id="S3.SS1.p1.3.m3.1" class="ltx_Math" alttext="d_{\text{model}}" display="inline"><semantics id="S3.SS1.p1.3.m3.1a"><msub id="S3.SS1.p1.3.m3.1.1" xref="S3.SS1.p1.3.m3.1.1.cmml"><mi id="S3.SS1.p1.3.m3.1.1.2" xref="S3.SS1.p1.3.m3.1.1.2.cmml">d</mi><mtext id="S3.SS1.p1.3.m3.1.1.3" xref="S3.SS1.p1.3.m3.1.1.3a.cmml">model</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.3.m3.1b"><apply id="S3.SS1.p1.3.m3.1.1.cmml" xref="S3.SS1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.3.m3.1.1.1.cmml" xref="S3.SS1.p1.3.m3.1.1">subscript</csymbol><ci id="S3.SS1.p1.3.m3.1.1.2.cmml" xref="S3.SS1.p1.3.m3.1.1.2">𝑑</ci><ci id="S3.SS1.p1.3.m3.1.1.3a.cmml" xref="S3.SS1.p1.3.m3.1.1.3"><mtext mathsize="70%" id="S3.SS1.p1.3.m3.1.1.3.cmml" xref="S3.SS1.p1.3.m3.1.1.3">model</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.3.m3.1c">d_{\text{model}}</annotation></semantics></math>.</p>
</div>
<div id="S3.SS1.p2" class="ltx_para">
<p id="S3.SS1.p2.10" class="ltx_p">We train all models using the AdamW optimizer <cite class="ltx_cite ltx_citemacro_citep">(Loshchilov and Hutter, <a href="#bib.bib26" title="" class="ltx_ref">2019</a>)</cite> with gradient clipping, using <math id="S3.SS1.p2.1.m1.1" class="ltx_Math" alttext="\beta_{1}=0.9" display="inline"><semantics id="S3.SS1.p2.1.m1.1a"><mrow id="S3.SS1.p2.1.m1.1.1" xref="S3.SS1.p2.1.m1.1.1.cmml"><msub id="S3.SS1.p2.1.m1.1.1.2" xref="S3.SS1.p2.1.m1.1.1.2.cmml"><mi id="S3.SS1.p2.1.m1.1.1.2.2" xref="S3.SS1.p2.1.m1.1.1.2.2.cmml">β</mi><mn id="S3.SS1.p2.1.m1.1.1.2.3" xref="S3.SS1.p2.1.m1.1.1.2.3.cmml">1</mn></msub><mo id="S3.SS1.p2.1.m1.1.1.1" xref="S3.SS1.p2.1.m1.1.1.1.cmml">=</mo><mn id="S3.SS1.p2.1.m1.1.1.3" xref="S3.SS1.p2.1.m1.1.1.3.cmml">0.9</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.1.m1.1b"><apply id="S3.SS1.p2.1.m1.1.1.cmml" xref="S3.SS1.p2.1.m1.1.1"><eq id="S3.SS1.p2.1.m1.1.1.1.cmml" xref="S3.SS1.p2.1.m1.1.1.1"></eq><apply id="S3.SS1.p2.1.m1.1.1.2.cmml" xref="S3.SS1.p2.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.SS1.p2.1.m1.1.1.2.1.cmml" xref="S3.SS1.p2.1.m1.1.1.2">subscript</csymbol><ci id="S3.SS1.p2.1.m1.1.1.2.2.cmml" xref="S3.SS1.p2.1.m1.1.1.2.2">𝛽</ci><cn type="integer" id="S3.SS1.p2.1.m1.1.1.2.3.cmml" xref="S3.SS1.p2.1.m1.1.1.2.3">1</cn></apply><cn type="float" id="S3.SS1.p2.1.m1.1.1.3.cmml" xref="S3.SS1.p2.1.m1.1.1.3">0.9</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.1.m1.1c">\beta_{1}=0.9</annotation></semantics></math>, <math id="S3.SS1.p2.2.m2.1" class="ltx_Math" alttext="\beta_{2}=0.95" display="inline"><semantics id="S3.SS1.p2.2.m2.1a"><mrow id="S3.SS1.p2.2.m2.1.1" xref="S3.SS1.p2.2.m2.1.1.cmml"><msub id="S3.SS1.p2.2.m2.1.1.2" xref="S3.SS1.p2.2.m2.1.1.2.cmml"><mi id="S3.SS1.p2.2.m2.1.1.2.2" xref="S3.SS1.p2.2.m2.1.1.2.2.cmml">β</mi><mn id="S3.SS1.p2.2.m2.1.1.2.3" xref="S3.SS1.p2.2.m2.1.1.2.3.cmml">2</mn></msub><mo id="S3.SS1.p2.2.m2.1.1.1" xref="S3.SS1.p2.2.m2.1.1.1.cmml">=</mo><mn id="S3.SS1.p2.2.m2.1.1.3" xref="S3.SS1.p2.2.m2.1.1.3.cmml">0.95</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.2.m2.1b"><apply id="S3.SS1.p2.2.m2.1.1.cmml" xref="S3.SS1.p2.2.m2.1.1"><eq id="S3.SS1.p2.2.m2.1.1.1.cmml" xref="S3.SS1.p2.2.m2.1.1.1"></eq><apply id="S3.SS1.p2.2.m2.1.1.2.cmml" xref="S3.SS1.p2.2.m2.1.1.2"><csymbol cd="ambiguous" id="S3.SS1.p2.2.m2.1.1.2.1.cmml" xref="S3.SS1.p2.2.m2.1.1.2">subscript</csymbol><ci id="S3.SS1.p2.2.m2.1.1.2.2.cmml" xref="S3.SS1.p2.2.m2.1.1.2.2">𝛽</ci><cn type="integer" id="S3.SS1.p2.2.m2.1.1.2.3.cmml" xref="S3.SS1.p2.2.m2.1.1.2.3">2</cn></apply><cn type="float" id="S3.SS1.p2.2.m2.1.1.3.cmml" xref="S3.SS1.p2.2.m2.1.1.3">0.95</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.2.m2.1c">\beta_{2}=0.95</annotation></semantics></math>, a weight decay factor of <math id="S3.SS1.p2.3.m3.1" class="ltx_Math" alttext="0.1" display="inline"><semantics id="S3.SS1.p2.3.m3.1a"><mn id="S3.SS1.p2.3.m3.1.1" xref="S3.SS1.p2.3.m3.1.1.cmml">0.1</mn><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.3.m3.1b"><cn type="float" id="S3.SS1.p2.3.m3.1.1.cmml" xref="S3.SS1.p2.3.m3.1.1">0.1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.3.m3.1c">0.1</annotation></semantics></math>, and a clipping norm of <math id="S3.SS1.p2.4.m4.1" class="ltx_Math" alttext="1.0" display="inline"><semantics id="S3.SS1.p2.4.m4.1a"><mn id="S3.SS1.p2.4.m4.1.1" xref="S3.SS1.p2.4.m4.1.1.cmml">1.0</mn><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.4.m4.1b"><cn type="float" id="S3.SS1.p2.4.m4.1.1.cmml" xref="S3.SS1.p2.4.m4.1.1">1.0</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.4.m4.1c">1.0</annotation></semantics></math>.
We use a linear learning rate warmup for the first <math id="S3.SS1.p2.5.m5.1" class="ltx_Math" alttext="5\%" display="inline"><semantics id="S3.SS1.p2.5.m5.1a"><mrow id="S3.SS1.p2.5.m5.1.1" xref="S3.SS1.p2.5.m5.1.1.cmml"><mn id="S3.SS1.p2.5.m5.1.1.2" xref="S3.SS1.p2.5.m5.1.1.2.cmml">5</mn><mo id="S3.SS1.p2.5.m5.1.1.1" xref="S3.SS1.p2.5.m5.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.5.m5.1b"><apply id="S3.SS1.p2.5.m5.1.1.cmml" xref="S3.SS1.p2.5.m5.1.1"><csymbol cd="latexml" id="S3.SS1.p2.5.m5.1.1.1.cmml" xref="S3.SS1.p2.5.m5.1.1.1">percent</csymbol><cn type="integer" id="S3.SS1.p2.5.m5.1.1.2.cmml" xref="S3.SS1.p2.5.m5.1.1.2">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.5.m5.1c">5\%</annotation></semantics></math> of training examples and a cosine learning rate schedule <cite class="ltx_cite ltx_citemacro_cite">Loshchilov and Hutter (<a href="#bib.bib25" title="" class="ltx_ref">2017</a>)</cite> decaying to <math id="S3.SS1.p2.6.m6.1" class="ltx_Math" alttext="10\%" display="inline"><semantics id="S3.SS1.p2.6.m6.1a"><mrow id="S3.SS1.p2.6.m6.1.1" xref="S3.SS1.p2.6.m6.1.1.cmml"><mn id="S3.SS1.p2.6.m6.1.1.2" xref="S3.SS1.p2.6.m6.1.1.2.cmml">10</mn><mo id="S3.SS1.p2.6.m6.1.1.1" xref="S3.SS1.p2.6.m6.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.6.m6.1b"><apply id="S3.SS1.p2.6.m6.1.1.cmml" xref="S3.SS1.p2.6.m6.1.1"><csymbol cd="latexml" id="S3.SS1.p2.6.m6.1.1.1.cmml" xref="S3.SS1.p2.6.m6.1.1.1">percent</csymbol><cn type="integer" id="S3.SS1.p2.6.m6.1.1.2.cmml" xref="S3.SS1.p2.6.m6.1.1.2">10</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.6.m6.1c">10\%</annotation></semantics></math> of the peak learning rate over the remainder of training.
We set the sequence length to <math id="S3.SS1.p2.7.m7.1" class="ltx_Math" alttext="2048" display="inline"><semantics id="S3.SS1.p2.7.m7.1a"><mn id="S3.SS1.p2.7.m7.1.1" xref="S3.SS1.p2.7.m7.1.1.cmml">2048</mn><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.7.m7.1b"><cn type="integer" id="S3.SS1.p2.7.m7.1.1.cmml" xref="S3.SS1.p2.7.m7.1.1">2048</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.7.m7.1c">2048</annotation></semantics></math> tokens and the batch size to <math id="S3.SS1.p2.8.m8.1" class="ltx_Math" alttext="2048" display="inline"><semantics id="S3.SS1.p2.8.m8.1a"><mn id="S3.SS1.p2.8.m8.1.1" xref="S3.SS1.p2.8.m8.1.1.cmml">2048</mn><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.8.m8.1b"><cn type="integer" id="S3.SS1.p2.8.m8.1.1.cmml" xref="S3.SS1.p2.8.m8.1.1">2048</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.8.m8.1c">2048</annotation></semantics></math> sequences, for a total of <math id="S3.SS1.p2.9.m9.1" class="ltx_Math" alttext="\approx 4M" display="inline"><semantics id="S3.SS1.p2.9.m9.1a"><mrow id="S3.SS1.p2.9.m9.1.1" xref="S3.SS1.p2.9.m9.1.1.cmml"><mi id="S3.SS1.p2.9.m9.1.1.2" xref="S3.SS1.p2.9.m9.1.1.2.cmml"></mi><mo id="S3.SS1.p2.9.m9.1.1.1" xref="S3.SS1.p2.9.m9.1.1.1.cmml">≈</mo><mrow id="S3.SS1.p2.9.m9.1.1.3" xref="S3.SS1.p2.9.m9.1.1.3.cmml"><mn id="S3.SS1.p2.9.m9.1.1.3.2" xref="S3.SS1.p2.9.m9.1.1.3.2.cmml">4</mn><mo lspace="0em" rspace="0em" id="S3.SS1.p2.9.m9.1.1.3.1" xref="S3.SS1.p2.9.m9.1.1.3.1.cmml">​</mo><mi id="S3.SS1.p2.9.m9.1.1.3.3" xref="S3.SS1.p2.9.m9.1.1.3.3.cmml">M</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.9.m9.1b"><apply id="S3.SS1.p2.9.m9.1.1.cmml" xref="S3.SS1.p2.9.m9.1.1"><approx id="S3.SS1.p2.9.m9.1.1.1.cmml" xref="S3.SS1.p2.9.m9.1.1.1"></approx><csymbol cd="latexml" id="S3.SS1.p2.9.m9.1.1.2.cmml" xref="S3.SS1.p2.9.m9.1.1.2">absent</csymbol><apply id="S3.SS1.p2.9.m9.1.1.3.cmml" xref="S3.SS1.p2.9.m9.1.1.3"><times id="S3.SS1.p2.9.m9.1.1.3.1.cmml" xref="S3.SS1.p2.9.m9.1.1.3.1"></times><cn type="integer" id="S3.SS1.p2.9.m9.1.1.3.2.cmml" xref="S3.SS1.p2.9.m9.1.1.3.2">4</cn><ci id="S3.SS1.p2.9.m9.1.1.3.3.cmml" xref="S3.SS1.p2.9.m9.1.1.3.3">𝑀</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.9.m9.1c">\approx 4M</annotation></semantics></math> tokens per training step.
All our experiments initialize the weights of linear layers from a normal distribution with mean zero and standard deviation <math id="S3.SS1.p2.10.m10.1" class="ltx_Math" alttext="0.01275" display="inline"><semantics id="S3.SS1.p2.10.m10.1a"><mn id="S3.SS1.p2.10.m10.1.1" xref="S3.SS1.p2.10.m10.1.1.cmml">0.01275</mn><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.10.m10.1b"><cn type="float" id="S3.SS1.p2.10.m10.1.1.cmml" xref="S3.SS1.p2.10.m10.1.1">0.01275</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.10.m10.1c">0.01275</annotation></semantics></math>.</p>
</div>
<div id="S3.SS1.p3" class="ltx_para">
<p id="S3.SS1.p3.1" class="ltx_p">We perform all experiments on NVIDIA H100 GPUs using PyTorch <cite class="ltx_cite ltx_citemacro_citep">(Paszke et al., <a href="#bib.bib31" title="" class="ltx_ref">2019</a>; Ansel et al., <a href="#bib.bib3" title="" class="ltx_ref">2024</a>)</cite>. We use mixed precision training <cite class="ltx_cite ltx_citemacro_citep">(Micikevicius et al., <a href="#bib.bib28" title="" class="ltx_ref">2018</a>)</cite> in BF16 <cite class="ltx_cite ltx_citemacro_citep">(Kalamkar et al., <a href="#bib.bib21" title="" class="ltx_ref">2019</a>)</cite> with gradient all-reduce and gradient accumulation in FP32 for training stability.</p>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Experiments at 1B-Parameter Scale</h3>

<figure id="S3.T2" class="ltx_table">
<div class="ltx_flex_figure ltx_flex_table">
<div class="ltx_flex_cell ltx_flex_size_1">
<table id="S3.T2.10" class="ltx_tabular ltx_centering ltx_figure_panel ltx_align_middle">
<tr id="S3.T2.10.11" class="ltx_tr">
<td id="S3.T2.10.11.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_tt"><span id="S3.T2.10.11.1.1" class="ltx_text ltx_font_bold">Model Family</span></td>
<td id="S3.T2.10.11.2" class="ltx_td ltx_align_left ltx_border_tt"><span id="S3.T2.10.11.2.1" class="ltx_text ltx_font_bold">Hidden Size</span></td>
<td id="S3.T2.10.11.3" class="ltx_td ltx_align_left ltx_border_tt"><span id="S3.T2.10.11.3.1" class="ltx_text ltx_font_bold">FFN Size</span></td>
<td id="S3.T2.10.11.4" class="ltx_td ltx_align_left ltx_border_tt"><span id="S3.T2.10.11.4.1" class="ltx_text ltx_font_bold">Layers</span></td>
<td id="S3.T2.10.11.5" class="ltx_td ltx_align_left ltx_border_tt"><span id="S3.T2.10.11.5.1" class="ltx_text ltx_font_bold">Sequence Length</span></td>
<td id="S3.T2.10.11.6" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt"><span id="S3.T2.10.11.6.1" class="ltx_text ltx_font_bold">Training Tokens</span></td>
</tr>
<tr id="S3.T2.5.5" class="ltx_tr">
<td id="S3.T2.5.5.6" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">1B Models</td>
<td id="S3.T2.1.1.1" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T2.1.1.1.m1.1" class="ltx_Math" alttext="2048" display="inline"><semantics id="S3.T2.1.1.1.m1.1a"><mn id="S3.T2.1.1.1.m1.1.1" xref="S3.T2.1.1.1.m1.1.1.cmml">2048</mn><annotation-xml encoding="MathML-Content" id="S3.T2.1.1.1.m1.1b"><cn type="integer" id="S3.T2.1.1.1.m1.1.1.cmml" xref="S3.T2.1.1.1.m1.1.1">2048</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.1.1.1.m1.1c">2048</annotation></semantics></math></td>
<td id="S3.T2.2.2.2" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T2.2.2.2.m1.1" class="ltx_Math" alttext="5472" display="inline"><semantics id="S3.T2.2.2.2.m1.1a"><mn id="S3.T2.2.2.2.m1.1.1" xref="S3.T2.2.2.2.m1.1.1.cmml">5472</mn><annotation-xml encoding="MathML-Content" id="S3.T2.2.2.2.m1.1b"><cn type="integer" id="S3.T2.2.2.2.m1.1.1.cmml" xref="S3.T2.2.2.2.m1.1.1">5472</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.2.2.2.m1.1c">5472</annotation></semantics></math></td>
<td id="S3.T2.3.3.3" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T2.3.3.3.m1.1" class="ltx_Math" alttext="20" display="inline"><semantics id="S3.T2.3.3.3.m1.1a"><mn id="S3.T2.3.3.3.m1.1.1" xref="S3.T2.3.3.3.m1.1.1.cmml">20</mn><annotation-xml encoding="MathML-Content" id="S3.T2.3.3.3.m1.1b"><cn type="integer" id="S3.T2.3.3.3.m1.1.1.cmml" xref="S3.T2.3.3.3.m1.1.1">20</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.3.3.3.m1.1c">20</annotation></semantics></math></td>
<td id="S3.T2.4.4.4" class="ltx_td ltx_align_left ltx_border_t"><math id="S3.T2.4.4.4.m1.1" class="ltx_Math" alttext="2048" display="inline"><semantics id="S3.T2.4.4.4.m1.1a"><mn id="S3.T2.4.4.4.m1.1.1" xref="S3.T2.4.4.4.m1.1.1.cmml">2048</mn><annotation-xml encoding="MathML-Content" id="S3.T2.4.4.4.m1.1b"><cn type="integer" id="S3.T2.4.4.4.m1.1.1.cmml" xref="S3.T2.4.4.4.m1.1.1">2048</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.4.4.4.m1.1c">2048</annotation></semantics></math></td>
<td id="S3.T2.5.5.5" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t"><math id="S3.T2.5.5.5.m1.3" class="ltx_Math" alttext="30\text{\times}{10}^{9}" display="inline"><semantics id="S3.T2.5.5.5.m1.3a"><mrow id="S3.T2.5.5.5.m1.3.3.3" xref="S3.T2.5.5.5.m1.3.3.3.cmml"><mn id="S3.T2.5.5.5.m1.1.1.1.1.1.1.1" xref="S3.T2.5.5.5.m1.3.3.3.cmml">30</mn><mtext id="S3.T2.5.5.5.m1.2.2.2.2.2.2.2" xref="S3.T2.5.5.5.m1.3.3.3.cmml">×</mtext><msup id="S3.T2.5.5.5.m1.3.3.3.3.3.3.3" xref="S3.T2.5.5.5.m1.3.3.3.cmml"><mn id="S3.T2.5.5.5.m1.3.3.3.3.3.3.3.2" xref="S3.T2.5.5.5.m1.3.3.3.cmml">10</mn><mn id="S3.T2.5.5.5.m1.3.3.3.3.3.3.3.3" xref="S3.T2.5.5.5.m1.3.3.3.cmml">9</mn></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.T2.5.5.5.m1.3b"><csymbol cd="latexml" id="S3.T2.5.5.5.m1.3.3.3.cmml" xref="S3.T2.5.5.5.m1.3.3.3">30E9</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.5.5.5.m1.3c">30\text{\times}{10}^{9}</annotation></semantics></math></td>
</tr>
<tr id="S3.T2.10.10" class="ltx_tr">
<td id="S3.T2.10.10.6" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r">3B Models</td>
<td id="S3.T2.6.6.1" class="ltx_td ltx_align_left ltx_border_bb"><math id="S3.T2.6.6.1.m1.1" class="ltx_Math" alttext="3072" display="inline"><semantics id="S3.T2.6.6.1.m1.1a"><mn id="S3.T2.6.6.1.m1.1.1" xref="S3.T2.6.6.1.m1.1.1.cmml">3072</mn><annotation-xml encoding="MathML-Content" id="S3.T2.6.6.1.m1.1b"><cn type="integer" id="S3.T2.6.6.1.m1.1.1.cmml" xref="S3.T2.6.6.1.m1.1.1">3072</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.6.6.1.m1.1c">3072</annotation></semantics></math></td>
<td id="S3.T2.7.7.2" class="ltx_td ltx_align_left ltx_border_bb"><math id="S3.T2.7.7.2.m1.1" class="ltx_Math" alttext="8192" display="inline"><semantics id="S3.T2.7.7.2.m1.1a"><mn id="S3.T2.7.7.2.m1.1.1" xref="S3.T2.7.7.2.m1.1.1.cmml">8192</mn><annotation-xml encoding="MathML-Content" id="S3.T2.7.7.2.m1.1b"><cn type="integer" id="S3.T2.7.7.2.m1.1.1.cmml" xref="S3.T2.7.7.2.m1.1.1">8192</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.7.7.2.m1.1c">8192</annotation></semantics></math></td>
<td id="S3.T2.8.8.3" class="ltx_td ltx_align_left ltx_border_bb"><math id="S3.T2.8.8.3.m1.1" class="ltx_Math" alttext="32" display="inline"><semantics id="S3.T2.8.8.3.m1.1a"><mn id="S3.T2.8.8.3.m1.1.1" xref="S3.T2.8.8.3.m1.1.1.cmml">32</mn><annotation-xml encoding="MathML-Content" id="S3.T2.8.8.3.m1.1b"><cn type="integer" id="S3.T2.8.8.3.m1.1.1.cmml" xref="S3.T2.8.8.3.m1.1.1">32</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.8.8.3.m1.1c">32</annotation></semantics></math></td>
<td id="S3.T2.9.9.4" class="ltx_td ltx_align_left ltx_border_bb"><math id="S3.T2.9.9.4.m1.1" class="ltx_Math" alttext="2048" display="inline"><semantics id="S3.T2.9.9.4.m1.1a"><mn id="S3.T2.9.9.4.m1.1.1" xref="S3.T2.9.9.4.m1.1.1.cmml">2048</mn><annotation-xml encoding="MathML-Content" id="S3.T2.9.9.4.m1.1b"><cn type="integer" id="S3.T2.9.9.4.m1.1.1.cmml" xref="S3.T2.9.9.4.m1.1.1">2048</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.9.9.4.m1.1c">2048</annotation></semantics></math></td>
<td id="S3.T2.10.10.5" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb"><math id="S3.T2.10.10.5.m1.3" class="ltx_Math" alttext="100\text{\times}{10}^{9}" display="inline"><semantics id="S3.T2.10.10.5.m1.3a"><mrow id="S3.T2.10.10.5.m1.3.3.3" xref="S3.T2.10.10.5.m1.3.3.3.cmml"><mn id="S3.T2.10.10.5.m1.1.1.1.1.1.1.1" xref="S3.T2.10.10.5.m1.3.3.3.cmml">100</mn><mtext id="S3.T2.10.10.5.m1.2.2.2.2.2.2.2" xref="S3.T2.10.10.5.m1.3.3.3.cmml">×</mtext><msup id="S3.T2.10.10.5.m1.3.3.3.3.3.3.3" xref="S3.T2.10.10.5.m1.3.3.3.cmml"><mn id="S3.T2.10.10.5.m1.3.3.3.3.3.3.3.2" xref="S3.T2.10.10.5.m1.3.3.3.cmml">10</mn><mn id="S3.T2.10.10.5.m1.3.3.3.3.3.3.3.3" xref="S3.T2.10.10.5.m1.3.3.3.cmml">9</mn></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.T2.10.10.5.m1.3b"><csymbol cd="latexml" id="S3.T2.10.10.5.m1.3.3.3.cmml" xref="S3.T2.10.10.5.m1.3.3.3">100E9</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.10.10.5.m1.3c">100\text{\times}{10}^{9}</annotation></semantics></math></td>
</tr>
</table>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="S3.T2.11" class="ltx_p ltx_figure_panel ltx_align_center">Hello</p>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>Architectural and training hyperparameters shared across our pretraining experiments.</figcaption>
</figure>
<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.1" class="ltx_p">We trained all our 1B-scale models on 30 billion tokens using a consistent data order, and, other than varying the attention mechanism, used the same architectural hyperparameters across all 1B-scale models.
This means that all our 1B models were all trained using approximately the same number of FLOPs and approximately the same number of GPU-hours, with CLA models requiring slightly fewer FLOPs to train than their non-CLA counterparts due to the reduced number of key/value projections.
The common hyperparameters shared across our 1B-scale experiments can be found in Table <a href="#S3.T2" title="Table 2 ‣ 3.2 Experiments at 1B-Parameter Scale ‣ 3 Pretraining Experiments ‣ Reducing Transformer Key-Value Cache Size with Cross-Layer Attention" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.</p>
</div>
<div id="S3.SS2.p2" class="ltx_para">
<p id="S3.SS2.p2.1" class="ltx_p">We ran two main sets of experiments at the 1B-parameter scale. First, we trained a diverse set of CLA and non-CLA models to characterize the range of accuracy/memory tradeoffs achievable with and without CLA, and to determine which CLA configurations are most effective; we refer to these as our <em id="S3.SS2.p2.1.1" class="ltx_emph ltx_font_italic">design space exploration</em> experiments, and describe them in more detail in Section <a href="#S3.SS2.SSS1" title="3.2.1 Design Space Exploration ‣ 3.2 Experiments at 1B-Parameter Scale ‣ 3 Pretraining Experiments ‣ Reducing Transformer Key-Value Cache Size with Cross-Layer Attention" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2.1</span></a>.
Second, we conducted a learning rate sweep on a subset of models from our design space exploration to verify that our results continue to hold even against a strong non-CLA baseline with a well-tuned learning rate.
We describe these learning rate tuning experiments in Appendix <a href="#A1" title="Appendix A Learning rate sweeps ‣ Reducing Transformer Key-Value Cache Size with Cross-Layer Attention" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a>.</p>
</div>
<section id="S3.SS2.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.2.1 </span>Design Space Exploration</h4>

<div id="S3.SS2.SSS1.p1" class="ltx_para">
<p id="S3.SS2.SSS1.p1.1" class="ltx_p">The primary goal of our 1B-parameter-scale design space exploration was to characterize the Pareto frontier of accuracy/memory tradeoffs achievable with and without CLA, and to determine which CLA configurations achieve the best accuracy on a fixed KV cache memory budget.
We train all models in our design space exploration using a learning rate of <math id="S3.SS2.SSS1.p1.1.m1.1" class="ltx_Math" alttext="\text{LR}=3\times 10^{-4}" display="inline"><semantics id="S3.SS2.SSS1.p1.1.m1.1a"><mrow id="S3.SS2.SSS1.p1.1.m1.1.1" xref="S3.SS2.SSS1.p1.1.m1.1.1.cmml"><mtext id="S3.SS2.SSS1.p1.1.m1.1.1.2" xref="S3.SS2.SSS1.p1.1.m1.1.1.2a.cmml">LR</mtext><mo id="S3.SS2.SSS1.p1.1.m1.1.1.1" xref="S3.SS2.SSS1.p1.1.m1.1.1.1.cmml">=</mo><mrow id="S3.SS2.SSS1.p1.1.m1.1.1.3" xref="S3.SS2.SSS1.p1.1.m1.1.1.3.cmml"><mn id="S3.SS2.SSS1.p1.1.m1.1.1.3.2" xref="S3.SS2.SSS1.p1.1.m1.1.1.3.2.cmml">3</mn><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.SSS1.p1.1.m1.1.1.3.1" xref="S3.SS2.SSS1.p1.1.m1.1.1.3.1.cmml">×</mo><msup id="S3.SS2.SSS1.p1.1.m1.1.1.3.3" xref="S3.SS2.SSS1.p1.1.m1.1.1.3.3.cmml"><mn id="S3.SS2.SSS1.p1.1.m1.1.1.3.3.2" xref="S3.SS2.SSS1.p1.1.m1.1.1.3.3.2.cmml">10</mn><mrow id="S3.SS2.SSS1.p1.1.m1.1.1.3.3.3" xref="S3.SS2.SSS1.p1.1.m1.1.1.3.3.3.cmml"><mo id="S3.SS2.SSS1.p1.1.m1.1.1.3.3.3a" xref="S3.SS2.SSS1.p1.1.m1.1.1.3.3.3.cmml">−</mo><mn id="S3.SS2.SSS1.p1.1.m1.1.1.3.3.3.2" xref="S3.SS2.SSS1.p1.1.m1.1.1.3.3.3.2.cmml">4</mn></mrow></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.p1.1.m1.1b"><apply id="S3.SS2.SSS1.p1.1.m1.1.1.cmml" xref="S3.SS2.SSS1.p1.1.m1.1.1"><eq id="S3.SS2.SSS1.p1.1.m1.1.1.1.cmml" xref="S3.SS2.SSS1.p1.1.m1.1.1.1"></eq><ci id="S3.SS2.SSS1.p1.1.m1.1.1.2a.cmml" xref="S3.SS2.SSS1.p1.1.m1.1.1.2"><mtext id="S3.SS2.SSS1.p1.1.m1.1.1.2.cmml" xref="S3.SS2.SSS1.p1.1.m1.1.1.2">LR</mtext></ci><apply id="S3.SS2.SSS1.p1.1.m1.1.1.3.cmml" xref="S3.SS2.SSS1.p1.1.m1.1.1.3"><times id="S3.SS2.SSS1.p1.1.m1.1.1.3.1.cmml" xref="S3.SS2.SSS1.p1.1.m1.1.1.3.1"></times><cn type="integer" id="S3.SS2.SSS1.p1.1.m1.1.1.3.2.cmml" xref="S3.SS2.SSS1.p1.1.m1.1.1.3.2">3</cn><apply id="S3.SS2.SSS1.p1.1.m1.1.1.3.3.cmml" xref="S3.SS2.SSS1.p1.1.m1.1.1.3.3"><csymbol cd="ambiguous" id="S3.SS2.SSS1.p1.1.m1.1.1.3.3.1.cmml" xref="S3.SS2.SSS1.p1.1.m1.1.1.3.3">superscript</csymbol><cn type="integer" id="S3.SS2.SSS1.p1.1.m1.1.1.3.3.2.cmml" xref="S3.SS2.SSS1.p1.1.m1.1.1.3.3.2">10</cn><apply id="S3.SS2.SSS1.p1.1.m1.1.1.3.3.3.cmml" xref="S3.SS2.SSS1.p1.1.m1.1.1.3.3.3"><minus id="S3.SS2.SSS1.p1.1.m1.1.1.3.3.3.1.cmml" xref="S3.SS2.SSS1.p1.1.m1.1.1.3.3.3"></minus><cn type="integer" id="S3.SS2.SSS1.p1.1.m1.1.1.3.3.3.2.cmml" xref="S3.SS2.SSS1.p1.1.m1.1.1.3.3.3.2">4</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.p1.1.m1.1c">\text{LR}=3\times 10^{-4}</annotation></semantics></math>, which we determined to be conservative; we explore the effect of the learning rate on accuracy in more detail in Section <a href="#S3.SS2.SSS2" title="3.2.2 Robustness to Learning Rate Tuning ‣ 3.2 Experiments at 1B-Parameter Scale ‣ 3 Pretraining Experiments ‣ Reducing Transformer Key-Value Cache Size with Cross-Layer Attention" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2.2</span></a>.</p>
</div>
<div id="S3.SS2.SSS1.p2" class="ltx_para">
<p id="S3.SS2.SSS1.p2.4" class="ltx_p">For our design space exploration, we first trained a collection of seven non-CLA baseline models along the MHA-GQA-MQA spectrum, exhibiting a range of KV cache memory requirements spanning two orders of magnitude.
Our baseline model with the largest KV cache memory footprint is an MHA model with a head embedding dimension of <math id="S3.SS2.SSS1.p2.1.m1.1" class="ltx_Math" alttext="d_{\text{head}}=128" display="inline"><semantics id="S3.SS2.SSS1.p2.1.m1.1a"><mrow id="S3.SS2.SSS1.p2.1.m1.1.1" xref="S3.SS2.SSS1.p2.1.m1.1.1.cmml"><msub id="S3.SS2.SSS1.p2.1.m1.1.1.2" xref="S3.SS2.SSS1.p2.1.m1.1.1.2.cmml"><mi id="S3.SS2.SSS1.p2.1.m1.1.1.2.2" xref="S3.SS2.SSS1.p2.1.m1.1.1.2.2.cmml">d</mi><mtext id="S3.SS2.SSS1.p2.1.m1.1.1.2.3" xref="S3.SS2.SSS1.p2.1.m1.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS2.SSS1.p2.1.m1.1.1.1" xref="S3.SS2.SSS1.p2.1.m1.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS1.p2.1.m1.1.1.3" xref="S3.SS2.SSS1.p2.1.m1.1.1.3.cmml">128</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.p2.1.m1.1b"><apply id="S3.SS2.SSS1.p2.1.m1.1.1.cmml" xref="S3.SS2.SSS1.p2.1.m1.1.1"><eq id="S3.SS2.SSS1.p2.1.m1.1.1.1.cmml" xref="S3.SS2.SSS1.p2.1.m1.1.1.1"></eq><apply id="S3.SS2.SSS1.p2.1.m1.1.1.2.cmml" xref="S3.SS2.SSS1.p2.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS1.p2.1.m1.1.1.2.1.cmml" xref="S3.SS2.SSS1.p2.1.m1.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS1.p2.1.m1.1.1.2.2.cmml" xref="S3.SS2.SSS1.p2.1.m1.1.1.2.2">𝑑</ci><ci id="S3.SS2.SSS1.p2.1.m1.1.1.2.3a.cmml" xref="S3.SS2.SSS1.p2.1.m1.1.1.2.3"><mtext mathsize="70%" id="S3.SS2.SSS1.p2.1.m1.1.1.2.3.cmml" xref="S3.SS2.SSS1.p2.1.m1.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS2.SSS1.p2.1.m1.1.1.3.cmml" xref="S3.SS2.SSS1.p2.1.m1.1.1.3">128</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.p2.1.m1.1c">d_{\text{head}}=128</annotation></semantics></math> (<math id="S3.SS2.SSS1.p2.2.m2.1" class="ltx_Math" alttext="163840" display="inline"><semantics id="S3.SS2.SSS1.p2.2.m2.1a"><mn id="S3.SS2.SSS1.p2.2.m2.1.1" xref="S3.SS2.SSS1.p2.2.m2.1.1.cmml">163840</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.p2.2.m2.1b"><cn type="integer" id="S3.SS2.SSS1.p2.2.m2.1.1.cmml" xref="S3.SS2.SSS1.p2.2.m2.1.1">163840</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.p2.2.m2.1c">163840</annotation></semantics></math> bytes per token at 16-bit precision), and our baseline with the smallest footprint is an MQA model with head dimension <math id="S3.SS2.SSS1.p2.3.m3.1" class="ltx_Math" alttext="d_{\text{head}}=32" display="inline"><semantics id="S3.SS2.SSS1.p2.3.m3.1a"><mrow id="S3.SS2.SSS1.p2.3.m3.1.1" xref="S3.SS2.SSS1.p2.3.m3.1.1.cmml"><msub id="S3.SS2.SSS1.p2.3.m3.1.1.2" xref="S3.SS2.SSS1.p2.3.m3.1.1.2.cmml"><mi id="S3.SS2.SSS1.p2.3.m3.1.1.2.2" xref="S3.SS2.SSS1.p2.3.m3.1.1.2.2.cmml">d</mi><mtext id="S3.SS2.SSS1.p2.3.m3.1.1.2.3" xref="S3.SS2.SSS1.p2.3.m3.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS2.SSS1.p2.3.m3.1.1.1" xref="S3.SS2.SSS1.p2.3.m3.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS1.p2.3.m3.1.1.3" xref="S3.SS2.SSS1.p2.3.m3.1.1.3.cmml">32</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.p2.3.m3.1b"><apply id="S3.SS2.SSS1.p2.3.m3.1.1.cmml" xref="S3.SS2.SSS1.p2.3.m3.1.1"><eq id="S3.SS2.SSS1.p2.3.m3.1.1.1.cmml" xref="S3.SS2.SSS1.p2.3.m3.1.1.1"></eq><apply id="S3.SS2.SSS1.p2.3.m3.1.1.2.cmml" xref="S3.SS2.SSS1.p2.3.m3.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS1.p2.3.m3.1.1.2.1.cmml" xref="S3.SS2.SSS1.p2.3.m3.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS1.p2.3.m3.1.1.2.2.cmml" xref="S3.SS2.SSS1.p2.3.m3.1.1.2.2">𝑑</ci><ci id="S3.SS2.SSS1.p2.3.m3.1.1.2.3a.cmml" xref="S3.SS2.SSS1.p2.3.m3.1.1.2.3"><mtext mathsize="70%" id="S3.SS2.SSS1.p2.3.m3.1.1.2.3.cmml" xref="S3.SS2.SSS1.p2.3.m3.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS2.SSS1.p2.3.m3.1.1.3.cmml" xref="S3.SS2.SSS1.p2.3.m3.1.1.3">32</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.p2.3.m3.1c">d_{\text{head}}=32</annotation></semantics></math> (<math id="S3.SS2.SSS1.p2.4.m4.1" class="ltx_Math" alttext="2560" display="inline"><semantics id="S3.SS2.SSS1.p2.4.m4.1a"><mn id="S3.SS2.SSS1.p2.4.m4.1.1" xref="S3.SS2.SSS1.p2.4.m4.1.1.cmml">2560</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.p2.4.m4.1b"><cn type="integer" id="S3.SS2.SSS1.p2.4.m4.1.1.cmml" xref="S3.SS2.SSS1.p2.4.m4.1.1">2560</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.p2.4.m4.1c">2560</annotation></semantics></math> bytes per token).</p>
</div>
<div id="S3.SS2.SSS1.p3" class="ltx_para">
<p id="S3.SS2.SSS1.p3.6" class="ltx_p">We quantify the accuracy of models in our design space exploration using perplexity on a held-out validation set of <math id="S3.SS2.SSS1.p3.1.m1.1" class="ltx_Math" alttext="\approx 4M" display="inline"><semantics id="S3.SS2.SSS1.p3.1.m1.1a"><mrow id="S3.SS2.SSS1.p3.1.m1.1.1" xref="S3.SS2.SSS1.p3.1.m1.1.1.cmml"><mi id="S3.SS2.SSS1.p3.1.m1.1.1.2" xref="S3.SS2.SSS1.p3.1.m1.1.1.2.cmml"></mi><mo id="S3.SS2.SSS1.p3.1.m1.1.1.1" xref="S3.SS2.SSS1.p3.1.m1.1.1.1.cmml">≈</mo><mrow id="S3.SS2.SSS1.p3.1.m1.1.1.3" xref="S3.SS2.SSS1.p3.1.m1.1.1.3.cmml"><mn id="S3.SS2.SSS1.p3.1.m1.1.1.3.2" xref="S3.SS2.SSS1.p3.1.m1.1.1.3.2.cmml">4</mn><mo lspace="0em" rspace="0em" id="S3.SS2.SSS1.p3.1.m1.1.1.3.1" xref="S3.SS2.SSS1.p3.1.m1.1.1.3.1.cmml">​</mo><mi id="S3.SS2.SSS1.p3.1.m1.1.1.3.3" xref="S3.SS2.SSS1.p3.1.m1.1.1.3.3.cmml">M</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.p3.1.m1.1b"><apply id="S3.SS2.SSS1.p3.1.m1.1.1.cmml" xref="S3.SS2.SSS1.p3.1.m1.1.1"><approx id="S3.SS2.SSS1.p3.1.m1.1.1.1.cmml" xref="S3.SS2.SSS1.p3.1.m1.1.1.1"></approx><csymbol cd="latexml" id="S3.SS2.SSS1.p3.1.m1.1.1.2.cmml" xref="S3.SS2.SSS1.p3.1.m1.1.1.2">absent</csymbol><apply id="S3.SS2.SSS1.p3.1.m1.1.1.3.cmml" xref="S3.SS2.SSS1.p3.1.m1.1.1.3"><times id="S3.SS2.SSS1.p3.1.m1.1.1.3.1.cmml" xref="S3.SS2.SSS1.p3.1.m1.1.1.3.1"></times><cn type="integer" id="S3.SS2.SSS1.p3.1.m1.1.1.3.2.cmml" xref="S3.SS2.SSS1.p3.1.m1.1.1.3.2">4</cn><ci id="S3.SS2.SSS1.p3.1.m1.1.1.3.3.cmml" xref="S3.SS2.SSS1.p3.1.m1.1.1.3.3">𝑀</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.p3.1.m1.1c">\approx 4M</annotation></semantics></math> tokens drawn from our SlimPajama corpus.
A summary of results for the models in our design space exploration, including our baseline models, can be found in Table <a href="#S3.T1" title="Table 1 ‣ 3 Pretraining Experiments ‣ Reducing Transformer Key-Value Cache Size with Cross-Layer Attention" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>. We adopt the naming scheme “H<math id="S3.SS2.SSS1.p3.2.m2.1" class="ltx_Math" alttext="\langle d_{\text{head}}\rangle" display="inline"><semantics id="S3.SS2.SSS1.p3.2.m2.1a"><mrow id="S3.SS2.SSS1.p3.2.m2.1.1.1" xref="S3.SS2.SSS1.p3.2.m2.1.1.2.cmml"><mo stretchy="false" id="S3.SS2.SSS1.p3.2.m2.1.1.1.2" xref="S3.SS2.SSS1.p3.2.m2.1.1.2.1.cmml">⟨</mo><msub id="S3.SS2.SSS1.p3.2.m2.1.1.1.1" xref="S3.SS2.SSS1.p3.2.m2.1.1.1.1.cmml"><mi id="S3.SS2.SSS1.p3.2.m2.1.1.1.1.2" xref="S3.SS2.SSS1.p3.2.m2.1.1.1.1.2.cmml">d</mi><mtext id="S3.SS2.SSS1.p3.2.m2.1.1.1.1.3" xref="S3.SS2.SSS1.p3.2.m2.1.1.1.1.3a.cmml">head</mtext></msub><mo stretchy="false" id="S3.SS2.SSS1.p3.2.m2.1.1.1.3" xref="S3.SS2.SSS1.p3.2.m2.1.1.2.1.cmml">⟩</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.p3.2.m2.1b"><apply id="S3.SS2.SSS1.p3.2.m2.1.1.2.cmml" xref="S3.SS2.SSS1.p3.2.m2.1.1.1"><csymbol cd="latexml" id="S3.SS2.SSS1.p3.2.m2.1.1.2.1.cmml" xref="S3.SS2.SSS1.p3.2.m2.1.1.1.2">delimited-⟨⟩</csymbol><apply id="S3.SS2.SSS1.p3.2.m2.1.1.1.1.cmml" xref="S3.SS2.SSS1.p3.2.m2.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS1.p3.2.m2.1.1.1.1.1.cmml" xref="S3.SS2.SSS1.p3.2.m2.1.1.1.1">subscript</csymbol><ci id="S3.SS2.SSS1.p3.2.m2.1.1.1.1.2.cmml" xref="S3.SS2.SSS1.p3.2.m2.1.1.1.1.2">𝑑</ci><ci id="S3.SS2.SSS1.p3.2.m2.1.1.1.1.3a.cmml" xref="S3.SS2.SSS1.p3.2.m2.1.1.1.1.3"><mtext mathsize="70%" id="S3.SS2.SSS1.p3.2.m2.1.1.1.1.3.cmml" xref="S3.SS2.SSS1.p3.2.m2.1.1.1.1.3">head</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.p3.2.m2.1c">\langle d_{\text{head}}\rangle</annotation></semantics></math>-<math id="S3.SS2.SSS1.p3.3.m3.1" class="ltx_Math" alttext="\langle\text{attention mechanism}\rangle" display="inline"><semantics id="S3.SS2.SSS1.p3.3.m3.1a"><mrow id="S3.SS2.SSS1.p3.3.m3.1.2.2" xref="S3.SS2.SSS1.p3.3.m3.1.2.1.cmml"><mo stretchy="false" id="S3.SS2.SSS1.p3.3.m3.1.2.2.1" xref="S3.SS2.SSS1.p3.3.m3.1.2.1.1.cmml">⟨</mo><mtext id="S3.SS2.SSS1.p3.3.m3.1.1" xref="S3.SS2.SSS1.p3.3.m3.1.1a.cmml">attention mechanism</mtext><mo stretchy="false" id="S3.SS2.SSS1.p3.3.m3.1.2.2.2" xref="S3.SS2.SSS1.p3.3.m3.1.2.1.1.cmml">⟩</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.p3.3.m3.1b"><apply id="S3.SS2.SSS1.p3.3.m3.1.2.1.cmml" xref="S3.SS2.SSS1.p3.3.m3.1.2.2"><csymbol cd="latexml" id="S3.SS2.SSS1.p3.3.m3.1.2.1.1.cmml" xref="S3.SS2.SSS1.p3.3.m3.1.2.2.1">delimited-⟨⟩</csymbol><ci id="S3.SS2.SSS1.p3.3.m3.1.1a.cmml" xref="S3.SS2.SSS1.p3.3.m3.1.1"><mtext id="S3.SS2.SSS1.p3.3.m3.1.1.cmml" xref="S3.SS2.SSS1.p3.3.m3.1.1">attention mechanism</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.p3.3.m3.1c">\langle\text{attention mechanism}\rangle</annotation></semantics></math>” for all models in our experiments, so that, for example, a model employing MQA with a head dimension of <math id="S3.SS2.SSS1.p3.4.m4.1" class="ltx_Math" alttext="d_{\text{head}}=64" display="inline"><semantics id="S3.SS2.SSS1.p3.4.m4.1a"><mrow id="S3.SS2.SSS1.p3.4.m4.1.1" xref="S3.SS2.SSS1.p3.4.m4.1.1.cmml"><msub id="S3.SS2.SSS1.p3.4.m4.1.1.2" xref="S3.SS2.SSS1.p3.4.m4.1.1.2.cmml"><mi id="S3.SS2.SSS1.p3.4.m4.1.1.2.2" xref="S3.SS2.SSS1.p3.4.m4.1.1.2.2.cmml">d</mi><mtext id="S3.SS2.SSS1.p3.4.m4.1.1.2.3" xref="S3.SS2.SSS1.p3.4.m4.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS2.SSS1.p3.4.m4.1.1.1" xref="S3.SS2.SSS1.p3.4.m4.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS1.p3.4.m4.1.1.3" xref="S3.SS2.SSS1.p3.4.m4.1.1.3.cmml">64</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.p3.4.m4.1b"><apply id="S3.SS2.SSS1.p3.4.m4.1.1.cmml" xref="S3.SS2.SSS1.p3.4.m4.1.1"><eq id="S3.SS2.SSS1.p3.4.m4.1.1.1.cmml" xref="S3.SS2.SSS1.p3.4.m4.1.1.1"></eq><apply id="S3.SS2.SSS1.p3.4.m4.1.1.2.cmml" xref="S3.SS2.SSS1.p3.4.m4.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS1.p3.4.m4.1.1.2.1.cmml" xref="S3.SS2.SSS1.p3.4.m4.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS1.p3.4.m4.1.1.2.2.cmml" xref="S3.SS2.SSS1.p3.4.m4.1.1.2.2">𝑑</ci><ci id="S3.SS2.SSS1.p3.4.m4.1.1.2.3a.cmml" xref="S3.SS2.SSS1.p3.4.m4.1.1.2.3"><mtext mathsize="70%" id="S3.SS2.SSS1.p3.4.m4.1.1.2.3.cmml" xref="S3.SS2.SSS1.p3.4.m4.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS2.SSS1.p3.4.m4.1.1.3.cmml" xref="S3.SS2.SSS1.p3.4.m4.1.1.3">64</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.p3.4.m4.1c">d_{\text{head}}=64</annotation></semantics></math> would be named “H64-MQA.”
For our baseline models, we observe that validation perplexity increases monotonically as we reduce the memory capacity of the KV cache, ranging from a perplexity of <math id="S3.SS2.SSS1.p3.5.m5.1" class="ltx_Math" alttext="13.15" display="inline"><semantics id="S3.SS2.SSS1.p3.5.m5.1a"><mn id="S3.SS2.SSS1.p3.5.m5.1.1" xref="S3.SS2.SSS1.p3.5.m5.1.1.cmml">13.15</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.p3.5.m5.1b"><cn type="float" id="S3.SS2.SSS1.p3.5.m5.1.1.cmml" xref="S3.SS2.SSS1.p3.5.m5.1.1">13.15</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.p3.5.m5.1c">13.15</annotation></semantics></math> for our H128-MHA baseline to <math id="S3.SS2.SSS1.p3.6.m6.1" class="ltx_Math" alttext="14.37" display="inline"><semantics id="S3.SS2.SSS1.p3.6.m6.1a"><mn id="S3.SS2.SSS1.p3.6.m6.1.1" xref="S3.SS2.SSS1.p3.6.m6.1.1.cmml">14.37</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.p3.6.m6.1b"><cn type="float" id="S3.SS2.SSS1.p3.6.m6.1.1.cmml" xref="S3.SS2.SSS1.p3.6.m6.1.1">14.37</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.p3.6.m6.1c">14.37</annotation></semantics></math> for our H32-MQA baseline.</p>
</div>
<div id="S3.SS2.SSS1.p4" class="ltx_para">
<p id="S3.SS2.SSS1.p4.1" class="ltx_p">In the rest of this section, we present results for the CLA models we trained during our design space exploration.</p>
</div>
<section id="S3.SS2.SSS1.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Best Performance: MQA + CLA2.</h5>

<div id="S3.SS2.SSS1.Px1.p1" class="ltx_para">
<p id="S3.SS2.SSS1.Px1.p1.2" class="ltx_p">We trained a family of five models combining MQA with CLA2.
We varied the head dimension for our MQA-CLA2 models from <math id="S3.SS2.SSS1.Px1.p1.1.m1.1" class="ltx_Math" alttext="d_{\text{head}}=512" display="inline"><semantics id="S3.SS2.SSS1.Px1.p1.1.m1.1a"><mrow id="S3.SS2.SSS1.Px1.p1.1.m1.1.1" xref="S3.SS2.SSS1.Px1.p1.1.m1.1.1.cmml"><msub id="S3.SS2.SSS1.Px1.p1.1.m1.1.1.2" xref="S3.SS2.SSS1.Px1.p1.1.m1.1.1.2.cmml"><mi id="S3.SS2.SSS1.Px1.p1.1.m1.1.1.2.2" xref="S3.SS2.SSS1.Px1.p1.1.m1.1.1.2.2.cmml">d</mi><mtext id="S3.SS2.SSS1.Px1.p1.1.m1.1.1.2.3" xref="S3.SS2.SSS1.Px1.p1.1.m1.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS2.SSS1.Px1.p1.1.m1.1.1.1" xref="S3.SS2.SSS1.Px1.p1.1.m1.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS1.Px1.p1.1.m1.1.1.3" xref="S3.SS2.SSS1.Px1.p1.1.m1.1.1.3.cmml">512</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px1.p1.1.m1.1b"><apply id="S3.SS2.SSS1.Px1.p1.1.m1.1.1.cmml" xref="S3.SS2.SSS1.Px1.p1.1.m1.1.1"><eq id="S3.SS2.SSS1.Px1.p1.1.m1.1.1.1.cmml" xref="S3.SS2.SSS1.Px1.p1.1.m1.1.1.1"></eq><apply id="S3.SS2.SSS1.Px1.p1.1.m1.1.1.2.cmml" xref="S3.SS2.SSS1.Px1.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS1.Px1.p1.1.m1.1.1.2.1.cmml" xref="S3.SS2.SSS1.Px1.p1.1.m1.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS1.Px1.p1.1.m1.1.1.2.2.cmml" xref="S3.SS2.SSS1.Px1.p1.1.m1.1.1.2.2">𝑑</ci><ci id="S3.SS2.SSS1.Px1.p1.1.m1.1.1.2.3a.cmml" xref="S3.SS2.SSS1.Px1.p1.1.m1.1.1.2.3"><mtext mathsize="70%" id="S3.SS2.SSS1.Px1.p1.1.m1.1.1.2.3.cmml" xref="S3.SS2.SSS1.Px1.p1.1.m1.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS2.SSS1.Px1.p1.1.m1.1.1.3.cmml" xref="S3.SS2.SSS1.Px1.p1.1.m1.1.1.3">512</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px1.p1.1.m1.1c">d_{\text{head}}=512</annotation></semantics></math> down to <math id="S3.SS2.SSS1.Px1.p1.2.m2.1" class="ltx_Math" alttext="d_{\text{head}}=64" display="inline"><semantics id="S3.SS2.SSS1.Px1.p1.2.m2.1a"><mrow id="S3.SS2.SSS1.Px1.p1.2.m2.1.1" xref="S3.SS2.SSS1.Px1.p1.2.m2.1.1.cmml"><msub id="S3.SS2.SSS1.Px1.p1.2.m2.1.1.2" xref="S3.SS2.SSS1.Px1.p1.2.m2.1.1.2.cmml"><mi id="S3.SS2.SSS1.Px1.p1.2.m2.1.1.2.2" xref="S3.SS2.SSS1.Px1.p1.2.m2.1.1.2.2.cmml">d</mi><mtext id="S3.SS2.SSS1.Px1.p1.2.m2.1.1.2.3" xref="S3.SS2.SSS1.Px1.p1.2.m2.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS2.SSS1.Px1.p1.2.m2.1.1.1" xref="S3.SS2.SSS1.Px1.p1.2.m2.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS1.Px1.p1.2.m2.1.1.3" xref="S3.SS2.SSS1.Px1.p1.2.m2.1.1.3.cmml">64</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px1.p1.2.m2.1b"><apply id="S3.SS2.SSS1.Px1.p1.2.m2.1.1.cmml" xref="S3.SS2.SSS1.Px1.p1.2.m2.1.1"><eq id="S3.SS2.SSS1.Px1.p1.2.m2.1.1.1.cmml" xref="S3.SS2.SSS1.Px1.p1.2.m2.1.1.1"></eq><apply id="S3.SS2.SSS1.Px1.p1.2.m2.1.1.2.cmml" xref="S3.SS2.SSS1.Px1.p1.2.m2.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS1.Px1.p1.2.m2.1.1.2.1.cmml" xref="S3.SS2.SSS1.Px1.p1.2.m2.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS1.Px1.p1.2.m2.1.1.2.2.cmml" xref="S3.SS2.SSS1.Px1.p1.2.m2.1.1.2.2">𝑑</ci><ci id="S3.SS2.SSS1.Px1.p1.2.m2.1.1.2.3a.cmml" xref="S3.SS2.SSS1.Px1.p1.2.m2.1.1.2.3"><mtext mathsize="70%" id="S3.SS2.SSS1.Px1.p1.2.m2.1.1.2.3.cmml" xref="S3.SS2.SSS1.Px1.p1.2.m2.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS2.SSS1.Px1.p1.2.m2.1.1.3.cmml" xref="S3.SS2.SSS1.Px1.p1.2.m2.1.1.3">64</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px1.p1.2.m2.1c">d_{\text{head}}=64</annotation></semantics></math>, allowing us to compare to a range of non-CLA baseline models with varying KV cache capacities.</p>
</div>
<div id="S3.SS2.SSS1.Px1.p2" class="ltx_para">
<p id="S3.SS2.SSS1.Px1.p2.7" class="ltx_p">We found that our MQA-CLA2 models are able to achieve better perplexities than baseline models requiring the same amount of KV cache memory, advancing the accuracy/memory Pareto frontier.
We present a plot of the accuracy/memory Pareto frontier with and without CLA in Figure <a href="#S3.F3" title="Figure 3 ‣ 3 Pretraining Experiments ‣ Reducing Transformer Key-Value Cache Size with Cross-Layer Attention" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>. Our MQA-CLA2 models with head dimensions <math id="S3.SS2.SSS1.Px1.p2.1.m1.3" class="ltx_Math" alttext="d_{\text{head}}\in\{64,90,128\}" display="inline"><semantics id="S3.SS2.SSS1.Px1.p2.1.m1.3a"><mrow id="S3.SS2.SSS1.Px1.p2.1.m1.3.4" xref="S3.SS2.SSS1.Px1.p2.1.m1.3.4.cmml"><msub id="S3.SS2.SSS1.Px1.p2.1.m1.3.4.2" xref="S3.SS2.SSS1.Px1.p2.1.m1.3.4.2.cmml"><mi id="S3.SS2.SSS1.Px1.p2.1.m1.3.4.2.2" xref="S3.SS2.SSS1.Px1.p2.1.m1.3.4.2.2.cmml">d</mi><mtext id="S3.SS2.SSS1.Px1.p2.1.m1.3.4.2.3" xref="S3.SS2.SSS1.Px1.p2.1.m1.3.4.2.3a.cmml">head</mtext></msub><mo id="S3.SS2.SSS1.Px1.p2.1.m1.3.4.1" xref="S3.SS2.SSS1.Px1.p2.1.m1.3.4.1.cmml">∈</mo><mrow id="S3.SS2.SSS1.Px1.p2.1.m1.3.4.3.2" xref="S3.SS2.SSS1.Px1.p2.1.m1.3.4.3.1.cmml"><mo stretchy="false" id="S3.SS2.SSS1.Px1.p2.1.m1.3.4.3.2.1" xref="S3.SS2.SSS1.Px1.p2.1.m1.3.4.3.1.cmml">{</mo><mn id="S3.SS2.SSS1.Px1.p2.1.m1.1.1" xref="S3.SS2.SSS1.Px1.p2.1.m1.1.1.cmml">64</mn><mo id="S3.SS2.SSS1.Px1.p2.1.m1.3.4.3.2.2" xref="S3.SS2.SSS1.Px1.p2.1.m1.3.4.3.1.cmml">,</mo><mn id="S3.SS2.SSS1.Px1.p2.1.m1.2.2" xref="S3.SS2.SSS1.Px1.p2.1.m1.2.2.cmml">90</mn><mo id="S3.SS2.SSS1.Px1.p2.1.m1.3.4.3.2.3" xref="S3.SS2.SSS1.Px1.p2.1.m1.3.4.3.1.cmml">,</mo><mn id="S3.SS2.SSS1.Px1.p2.1.m1.3.3" xref="S3.SS2.SSS1.Px1.p2.1.m1.3.3.cmml">128</mn><mo stretchy="false" id="S3.SS2.SSS1.Px1.p2.1.m1.3.4.3.2.4" xref="S3.SS2.SSS1.Px1.p2.1.m1.3.4.3.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px1.p2.1.m1.3b"><apply id="S3.SS2.SSS1.Px1.p2.1.m1.3.4.cmml" xref="S3.SS2.SSS1.Px1.p2.1.m1.3.4"><in id="S3.SS2.SSS1.Px1.p2.1.m1.3.4.1.cmml" xref="S3.SS2.SSS1.Px1.p2.1.m1.3.4.1"></in><apply id="S3.SS2.SSS1.Px1.p2.1.m1.3.4.2.cmml" xref="S3.SS2.SSS1.Px1.p2.1.m1.3.4.2"><csymbol cd="ambiguous" id="S3.SS2.SSS1.Px1.p2.1.m1.3.4.2.1.cmml" xref="S3.SS2.SSS1.Px1.p2.1.m1.3.4.2">subscript</csymbol><ci id="S3.SS2.SSS1.Px1.p2.1.m1.3.4.2.2.cmml" xref="S3.SS2.SSS1.Px1.p2.1.m1.3.4.2.2">𝑑</ci><ci id="S3.SS2.SSS1.Px1.p2.1.m1.3.4.2.3a.cmml" xref="S3.SS2.SSS1.Px1.p2.1.m1.3.4.2.3"><mtext mathsize="70%" id="S3.SS2.SSS1.Px1.p2.1.m1.3.4.2.3.cmml" xref="S3.SS2.SSS1.Px1.p2.1.m1.3.4.2.3">head</mtext></ci></apply><set id="S3.SS2.SSS1.Px1.p2.1.m1.3.4.3.1.cmml" xref="S3.SS2.SSS1.Px1.p2.1.m1.3.4.3.2"><cn type="integer" id="S3.SS2.SSS1.Px1.p2.1.m1.1.1.cmml" xref="S3.SS2.SSS1.Px1.p2.1.m1.1.1">64</cn><cn type="integer" id="S3.SS2.SSS1.Px1.p2.1.m1.2.2.cmml" xref="S3.SS2.SSS1.Px1.p2.1.m1.2.2">90</cn><cn type="integer" id="S3.SS2.SSS1.Px1.p2.1.m1.3.3.cmml" xref="S3.SS2.SSS1.Px1.p2.1.m1.3.3">128</cn></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px1.p2.1.m1.3c">d_{\text{head}}\in\{64,90,128\}</annotation></semantics></math> are able to match the KV cache memory footprint of baseline MQA models with head dimensions <math id="S3.SS2.SSS1.Px1.p2.2.m2.3" class="ltx_Math" alttext="d_{\text{head}}\in\{32,46,64\}" display="inline"><semantics id="S3.SS2.SSS1.Px1.p2.2.m2.3a"><mrow id="S3.SS2.SSS1.Px1.p2.2.m2.3.4" xref="S3.SS2.SSS1.Px1.p2.2.m2.3.4.cmml"><msub id="S3.SS2.SSS1.Px1.p2.2.m2.3.4.2" xref="S3.SS2.SSS1.Px1.p2.2.m2.3.4.2.cmml"><mi id="S3.SS2.SSS1.Px1.p2.2.m2.3.4.2.2" xref="S3.SS2.SSS1.Px1.p2.2.m2.3.4.2.2.cmml">d</mi><mtext id="S3.SS2.SSS1.Px1.p2.2.m2.3.4.2.3" xref="S3.SS2.SSS1.Px1.p2.2.m2.3.4.2.3a.cmml">head</mtext></msub><mo id="S3.SS2.SSS1.Px1.p2.2.m2.3.4.1" xref="S3.SS2.SSS1.Px1.p2.2.m2.3.4.1.cmml">∈</mo><mrow id="S3.SS2.SSS1.Px1.p2.2.m2.3.4.3.2" xref="S3.SS2.SSS1.Px1.p2.2.m2.3.4.3.1.cmml"><mo stretchy="false" id="S3.SS2.SSS1.Px1.p2.2.m2.3.4.3.2.1" xref="S3.SS2.SSS1.Px1.p2.2.m2.3.4.3.1.cmml">{</mo><mn id="S3.SS2.SSS1.Px1.p2.2.m2.1.1" xref="S3.SS2.SSS1.Px1.p2.2.m2.1.1.cmml">32</mn><mo id="S3.SS2.SSS1.Px1.p2.2.m2.3.4.3.2.2" xref="S3.SS2.SSS1.Px1.p2.2.m2.3.4.3.1.cmml">,</mo><mn id="S3.SS2.SSS1.Px1.p2.2.m2.2.2" xref="S3.SS2.SSS1.Px1.p2.2.m2.2.2.cmml">46</mn><mo id="S3.SS2.SSS1.Px1.p2.2.m2.3.4.3.2.3" xref="S3.SS2.SSS1.Px1.p2.2.m2.3.4.3.1.cmml">,</mo><mn id="S3.SS2.SSS1.Px1.p2.2.m2.3.3" xref="S3.SS2.SSS1.Px1.p2.2.m2.3.3.cmml">64</mn><mo stretchy="false" id="S3.SS2.SSS1.Px1.p2.2.m2.3.4.3.2.4" xref="S3.SS2.SSS1.Px1.p2.2.m2.3.4.3.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px1.p2.2.m2.3b"><apply id="S3.SS2.SSS1.Px1.p2.2.m2.3.4.cmml" xref="S3.SS2.SSS1.Px1.p2.2.m2.3.4"><in id="S3.SS2.SSS1.Px1.p2.2.m2.3.4.1.cmml" xref="S3.SS2.SSS1.Px1.p2.2.m2.3.4.1"></in><apply id="S3.SS2.SSS1.Px1.p2.2.m2.3.4.2.cmml" xref="S3.SS2.SSS1.Px1.p2.2.m2.3.4.2"><csymbol cd="ambiguous" id="S3.SS2.SSS1.Px1.p2.2.m2.3.4.2.1.cmml" xref="S3.SS2.SSS1.Px1.p2.2.m2.3.4.2">subscript</csymbol><ci id="S3.SS2.SSS1.Px1.p2.2.m2.3.4.2.2.cmml" xref="S3.SS2.SSS1.Px1.p2.2.m2.3.4.2.2">𝑑</ci><ci id="S3.SS2.SSS1.Px1.p2.2.m2.3.4.2.3a.cmml" xref="S3.SS2.SSS1.Px1.p2.2.m2.3.4.2.3"><mtext mathsize="70%" id="S3.SS2.SSS1.Px1.p2.2.m2.3.4.2.3.cmml" xref="S3.SS2.SSS1.Px1.p2.2.m2.3.4.2.3">head</mtext></ci></apply><set id="S3.SS2.SSS1.Px1.p2.2.m2.3.4.3.1.cmml" xref="S3.SS2.SSS1.Px1.p2.2.m2.3.4.3.2"><cn type="integer" id="S3.SS2.SSS1.Px1.p2.2.m2.1.1.cmml" xref="S3.SS2.SSS1.Px1.p2.2.m2.1.1">32</cn><cn type="integer" id="S3.SS2.SSS1.Px1.p2.2.m2.2.2.cmml" xref="S3.SS2.SSS1.Px1.p2.2.m2.2.2">46</cn><cn type="integer" id="S3.SS2.SSS1.Px1.p2.2.m2.3.3.cmml" xref="S3.SS2.SSS1.Px1.p2.2.m2.3.3">64</cn></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px1.p2.2.m2.3c">d_{\text{head}}\in\{32,46,64\}</annotation></semantics></math> while achieving substantial perplexity improvements in the range of <math id="S3.SS2.SSS1.Px1.p2.3.m3.1" class="ltx_Math" alttext="0.21" display="inline"><semantics id="S3.SS2.SSS1.Px1.p2.3.m3.1a"><mn id="S3.SS2.SSS1.Px1.p2.3.m3.1.1" xref="S3.SS2.SSS1.Px1.p2.3.m3.1.1.cmml">0.21</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px1.p2.3.m3.1b"><cn type="float" id="S3.SS2.SSS1.Px1.p2.3.m3.1.1.cmml" xref="S3.SS2.SSS1.Px1.p2.3.m3.1.1">0.21</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px1.p2.3.m3.1c">0.21</annotation></semantics></math>–<math id="S3.SS2.SSS1.Px1.p2.4.m4.1" class="ltx_Math" alttext="0.48" display="inline"><semantics id="S3.SS2.SSS1.Px1.p2.4.m4.1a"><mn id="S3.SS2.SSS1.Px1.p2.4.m4.1.1" xref="S3.SS2.SSS1.Px1.p2.4.m4.1.1.cmml">0.48</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px1.p2.4.m4.1b"><cn type="float" id="S3.SS2.SSS1.Px1.p2.4.m4.1.1.cmml" xref="S3.SS2.SSS1.Px1.p2.4.m4.1.1">0.48</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px1.p2.4.m4.1c">0.48</annotation></semantics></math> points. Additionally, our MQA-CLA2 models with large head sizes of <math id="S3.SS2.SSS1.Px1.p2.5.m5.2" class="ltx_Math" alttext="d_{\text{head}}\in\{256,512\}" display="inline"><semantics id="S3.SS2.SSS1.Px1.p2.5.m5.2a"><mrow id="S3.SS2.SSS1.Px1.p2.5.m5.2.3" xref="S3.SS2.SSS1.Px1.p2.5.m5.2.3.cmml"><msub id="S3.SS2.SSS1.Px1.p2.5.m5.2.3.2" xref="S3.SS2.SSS1.Px1.p2.5.m5.2.3.2.cmml"><mi id="S3.SS2.SSS1.Px1.p2.5.m5.2.3.2.2" xref="S3.SS2.SSS1.Px1.p2.5.m5.2.3.2.2.cmml">d</mi><mtext id="S3.SS2.SSS1.Px1.p2.5.m5.2.3.2.3" xref="S3.SS2.SSS1.Px1.p2.5.m5.2.3.2.3a.cmml">head</mtext></msub><mo id="S3.SS2.SSS1.Px1.p2.5.m5.2.3.1" xref="S3.SS2.SSS1.Px1.p2.5.m5.2.3.1.cmml">∈</mo><mrow id="S3.SS2.SSS1.Px1.p2.5.m5.2.3.3.2" xref="S3.SS2.SSS1.Px1.p2.5.m5.2.3.3.1.cmml"><mo stretchy="false" id="S3.SS2.SSS1.Px1.p2.5.m5.2.3.3.2.1" xref="S3.SS2.SSS1.Px1.p2.5.m5.2.3.3.1.cmml">{</mo><mn id="S3.SS2.SSS1.Px1.p2.5.m5.1.1" xref="S3.SS2.SSS1.Px1.p2.5.m5.1.1.cmml">256</mn><mo id="S3.SS2.SSS1.Px1.p2.5.m5.2.3.3.2.2" xref="S3.SS2.SSS1.Px1.p2.5.m5.2.3.3.1.cmml">,</mo><mn id="S3.SS2.SSS1.Px1.p2.5.m5.2.2" xref="S3.SS2.SSS1.Px1.p2.5.m5.2.2.cmml">512</mn><mo stretchy="false" id="S3.SS2.SSS1.Px1.p2.5.m5.2.3.3.2.3" xref="S3.SS2.SSS1.Px1.p2.5.m5.2.3.3.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px1.p2.5.m5.2b"><apply id="S3.SS2.SSS1.Px1.p2.5.m5.2.3.cmml" xref="S3.SS2.SSS1.Px1.p2.5.m5.2.3"><in id="S3.SS2.SSS1.Px1.p2.5.m5.2.3.1.cmml" xref="S3.SS2.SSS1.Px1.p2.5.m5.2.3.1"></in><apply id="S3.SS2.SSS1.Px1.p2.5.m5.2.3.2.cmml" xref="S3.SS2.SSS1.Px1.p2.5.m5.2.3.2"><csymbol cd="ambiguous" id="S3.SS2.SSS1.Px1.p2.5.m5.2.3.2.1.cmml" xref="S3.SS2.SSS1.Px1.p2.5.m5.2.3.2">subscript</csymbol><ci id="S3.SS2.SSS1.Px1.p2.5.m5.2.3.2.2.cmml" xref="S3.SS2.SSS1.Px1.p2.5.m5.2.3.2.2">𝑑</ci><ci id="S3.SS2.SSS1.Px1.p2.5.m5.2.3.2.3a.cmml" xref="S3.SS2.SSS1.Px1.p2.5.m5.2.3.2.3"><mtext mathsize="70%" id="S3.SS2.SSS1.Px1.p2.5.m5.2.3.2.3.cmml" xref="S3.SS2.SSS1.Px1.p2.5.m5.2.3.2.3">head</mtext></ci></apply><set id="S3.SS2.SSS1.Px1.p2.5.m5.2.3.3.1.cmml" xref="S3.SS2.SSS1.Px1.p2.5.m5.2.3.3.2"><cn type="integer" id="S3.SS2.SSS1.Px1.p2.5.m5.1.1.cmml" xref="S3.SS2.SSS1.Px1.p2.5.m5.1.1">256</cn><cn type="integer" id="S3.SS2.SSS1.Px1.p2.5.m5.2.2.cmml" xref="S3.SS2.SSS1.Px1.p2.5.m5.2.2">512</cn></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px1.p2.5.m5.2c">d_{\text{head}}\in\{256,512\}</annotation></semantics></math> are able to match the KV cache footprint of our MQA and GQA2 baselines with <math id="S3.SS2.SSS1.Px1.p2.6.m6.1" class="ltx_Math" alttext="d_{\text{head}}=128" display="inline"><semantics id="S3.SS2.SSS1.Px1.p2.6.m6.1a"><mrow id="S3.SS2.SSS1.Px1.p2.6.m6.1.1" xref="S3.SS2.SSS1.Px1.p2.6.m6.1.1.cmml"><msub id="S3.SS2.SSS1.Px1.p2.6.m6.1.1.2" xref="S3.SS2.SSS1.Px1.p2.6.m6.1.1.2.cmml"><mi id="S3.SS2.SSS1.Px1.p2.6.m6.1.1.2.2" xref="S3.SS2.SSS1.Px1.p2.6.m6.1.1.2.2.cmml">d</mi><mtext id="S3.SS2.SSS1.Px1.p2.6.m6.1.1.2.3" xref="S3.SS2.SSS1.Px1.p2.6.m6.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS2.SSS1.Px1.p2.6.m6.1.1.1" xref="S3.SS2.SSS1.Px1.p2.6.m6.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS1.Px1.p2.6.m6.1.1.3" xref="S3.SS2.SSS1.Px1.p2.6.m6.1.1.3.cmml">128</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px1.p2.6.m6.1b"><apply id="S3.SS2.SSS1.Px1.p2.6.m6.1.1.cmml" xref="S3.SS2.SSS1.Px1.p2.6.m6.1.1"><eq id="S3.SS2.SSS1.Px1.p2.6.m6.1.1.1.cmml" xref="S3.SS2.SSS1.Px1.p2.6.m6.1.1.1"></eq><apply id="S3.SS2.SSS1.Px1.p2.6.m6.1.1.2.cmml" xref="S3.SS2.SSS1.Px1.p2.6.m6.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS1.Px1.p2.6.m6.1.1.2.1.cmml" xref="S3.SS2.SSS1.Px1.p2.6.m6.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS1.Px1.p2.6.m6.1.1.2.2.cmml" xref="S3.SS2.SSS1.Px1.p2.6.m6.1.1.2.2">𝑑</ci><ci id="S3.SS2.SSS1.Px1.p2.6.m6.1.1.2.3a.cmml" xref="S3.SS2.SSS1.Px1.p2.6.m6.1.1.2.3"><mtext mathsize="70%" id="S3.SS2.SSS1.Px1.p2.6.m6.1.1.2.3.cmml" xref="S3.SS2.SSS1.Px1.p2.6.m6.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS2.SSS1.Px1.p2.6.m6.1.1.3.cmml" xref="S3.SS2.SSS1.Px1.p2.6.m6.1.1.3">128</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px1.p2.6.m6.1c">d_{\text{head}}=128</annotation></semantics></math> while achieving a small perplexity improvement of <math id="S3.SS2.SSS1.Px1.p2.7.m7.1" class="ltx_Math" alttext="0.03" display="inline"><semantics id="S3.SS2.SSS1.Px1.p2.7.m7.1a"><mn id="S3.SS2.SSS1.Px1.p2.7.m7.1.1" xref="S3.SS2.SSS1.Px1.p2.7.m7.1.1.cmml">0.03</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px1.p2.7.m7.1b"><cn type="float" id="S3.SS2.SSS1.Px1.p2.7.m7.1.1.cmml" xref="S3.SS2.SSS1.Px1.p2.7.m7.1.1">0.03</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px1.p2.7.m7.1c">0.03</annotation></semantics></math> points.</p>
</div>
<div id="S3.SS2.SSS1.Px1.p3" class="ltx_para">
<p id="S3.SS2.SSS1.Px1.p3.1" class="ltx_p">We found that our MQA-CLA2 models achieved the best accuracy/memory tradeoffs among all CLA configurations we tested in our design space exploration. In the rest of this section, we briefly describe the ablations we conducted to explore alternate CLA configurations.</p>
</div>
</section>
<section id="S3.SS2.SSS1.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Ablation: GQA + CLA2.</h5>

<div id="S3.SS2.SSS1.Px2.p1" class="ltx_para">
<p id="S3.SS2.SSS1.Px2.p1.3" class="ltx_p">We trained three models to explore combining GQA with CLA2.
We chose GQA4-CLA2 with <math id="S3.SS2.SSS1.Px2.p1.1.m1.1" class="ltx_Math" alttext="d_{\text{head}}=128" display="inline"><semantics id="S3.SS2.SSS1.Px2.p1.1.m1.1a"><mrow id="S3.SS2.SSS1.Px2.p1.1.m1.1.1" xref="S3.SS2.SSS1.Px2.p1.1.m1.1.1.cmml"><msub id="S3.SS2.SSS1.Px2.p1.1.m1.1.1.2" xref="S3.SS2.SSS1.Px2.p1.1.m1.1.1.2.cmml"><mi id="S3.SS2.SSS1.Px2.p1.1.m1.1.1.2.2" xref="S3.SS2.SSS1.Px2.p1.1.m1.1.1.2.2.cmml">d</mi><mtext id="S3.SS2.SSS1.Px2.p1.1.m1.1.1.2.3" xref="S3.SS2.SSS1.Px2.p1.1.m1.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS2.SSS1.Px2.p1.1.m1.1.1.1" xref="S3.SS2.SSS1.Px2.p1.1.m1.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS1.Px2.p1.1.m1.1.1.3" xref="S3.SS2.SSS1.Px2.p1.1.m1.1.1.3.cmml">128</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px2.p1.1.m1.1b"><apply id="S3.SS2.SSS1.Px2.p1.1.m1.1.1.cmml" xref="S3.SS2.SSS1.Px2.p1.1.m1.1.1"><eq id="S3.SS2.SSS1.Px2.p1.1.m1.1.1.1.cmml" xref="S3.SS2.SSS1.Px2.p1.1.m1.1.1.1"></eq><apply id="S3.SS2.SSS1.Px2.p1.1.m1.1.1.2.cmml" xref="S3.SS2.SSS1.Px2.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS1.Px2.p1.1.m1.1.1.2.1.cmml" xref="S3.SS2.SSS1.Px2.p1.1.m1.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS1.Px2.p1.1.m1.1.1.2.2.cmml" xref="S3.SS2.SSS1.Px2.p1.1.m1.1.1.2.2">𝑑</ci><ci id="S3.SS2.SSS1.Px2.p1.1.m1.1.1.2.3a.cmml" xref="S3.SS2.SSS1.Px2.p1.1.m1.1.1.2.3"><mtext mathsize="70%" id="S3.SS2.SSS1.Px2.p1.1.m1.1.1.2.3.cmml" xref="S3.SS2.SSS1.Px2.p1.1.m1.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS2.SSS1.Px2.p1.1.m1.1.1.3.cmml" xref="S3.SS2.SSS1.Px2.p1.1.m1.1.1.3">128</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px2.p1.1.m1.1c">d_{\text{head}}=128</annotation></semantics></math> as our starting point, as GQA4 represents an attention configuration intermediate between our MQA and MHA baselines. We then explored expanding the head dimension of our GQA4-CLA2 model to <math id="S3.SS2.SSS1.Px2.p1.2.m2.1" class="ltx_Math" alttext="d_{\text{head}}=256" display="inline"><semantics id="S3.SS2.SSS1.Px2.p1.2.m2.1a"><mrow id="S3.SS2.SSS1.Px2.p1.2.m2.1.1" xref="S3.SS2.SSS1.Px2.p1.2.m2.1.1.cmml"><msub id="S3.SS2.SSS1.Px2.p1.2.m2.1.1.2" xref="S3.SS2.SSS1.Px2.p1.2.m2.1.1.2.cmml"><mi id="S3.SS2.SSS1.Px2.p1.2.m2.1.1.2.2" xref="S3.SS2.SSS1.Px2.p1.2.m2.1.1.2.2.cmml">d</mi><mtext id="S3.SS2.SSS1.Px2.p1.2.m2.1.1.2.3" xref="S3.SS2.SSS1.Px2.p1.2.m2.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS2.SSS1.Px2.p1.2.m2.1.1.1" xref="S3.SS2.SSS1.Px2.p1.2.m2.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS1.Px2.p1.2.m2.1.1.3" xref="S3.SS2.SSS1.Px2.p1.2.m2.1.1.3.cmml">256</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px2.p1.2.m2.1b"><apply id="S3.SS2.SSS1.Px2.p1.2.m2.1.1.cmml" xref="S3.SS2.SSS1.Px2.p1.2.m2.1.1"><eq id="S3.SS2.SSS1.Px2.p1.2.m2.1.1.1.cmml" xref="S3.SS2.SSS1.Px2.p1.2.m2.1.1.1"></eq><apply id="S3.SS2.SSS1.Px2.p1.2.m2.1.1.2.cmml" xref="S3.SS2.SSS1.Px2.p1.2.m2.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS1.Px2.p1.2.m2.1.1.2.1.cmml" xref="S3.SS2.SSS1.Px2.p1.2.m2.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS1.Px2.p1.2.m2.1.1.2.2.cmml" xref="S3.SS2.SSS1.Px2.p1.2.m2.1.1.2.2">𝑑</ci><ci id="S3.SS2.SSS1.Px2.p1.2.m2.1.1.2.3a.cmml" xref="S3.SS2.SSS1.Px2.p1.2.m2.1.1.2.3"><mtext mathsize="70%" id="S3.SS2.SSS1.Px2.p1.2.m2.1.1.2.3.cmml" xref="S3.SS2.SSS1.Px2.p1.2.m2.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS2.SSS1.Px2.p1.2.m2.1.1.3.cmml" xref="S3.SS2.SSS1.Px2.p1.2.m2.1.1.3">256</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px2.p1.2.m2.1c">d_{\text{head}}=256</annotation></semantics></math>, as well as reducing the GQA factor to GQA2. We found that only the GQA2-CLA2 configuration was able to achieve a perplexity better than the corresponding baseline model with the same KV cache footprint, and that this perplexity was the same (within <math id="S3.SS2.SSS1.Px2.p1.3.m3.1" class="ltx_Math" alttext="0.01" display="inline"><semantics id="S3.SS2.SSS1.Px2.p1.3.m3.1a"><mn id="S3.SS2.SSS1.Px2.p1.3.m3.1.1" xref="S3.SS2.SSS1.Px2.p1.3.m3.1.1.cmml">0.01</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px2.p1.3.m3.1b"><cn type="float" id="S3.SS2.SSS1.Px2.p1.3.m3.1.1.cmml" xref="S3.SS2.SSS1.Px2.p1.3.m3.1.1">0.01</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px2.p1.3.m3.1c">0.01</annotation></semantics></math> points) as our MQA-CLA2 model with the same footprint.</p>
</div>
</section>
<section id="S3.SS2.SSS1.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Ablation: MQA + CLA with Sharing Factor <math id="S3.SS2.SSS1.Px3.1.m1.1" class="ltx_Math" alttext="\mathbf{&gt;2}" display="inline"><semantics id="S3.SS2.SSS1.Px3.1.m1.1b"><mrow id="S3.SS2.SSS1.Px3.1.m1.1.1" xref="S3.SS2.SSS1.Px3.1.m1.1.1.cmml"><mi id="S3.SS2.SSS1.Px3.1.m1.1.1.2" xref="S3.SS2.SSS1.Px3.1.m1.1.1.2.cmml"></mi><mo id="S3.SS2.SSS1.Px3.1.m1.1.1.1" xref="S3.SS2.SSS1.Px3.1.m1.1.1.1.cmml">&gt;</mo><mn id="S3.SS2.SSS1.Px3.1.m1.1.1.3" xref="S3.SS2.SSS1.Px3.1.m1.1.1.3.cmml">𝟐</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px3.1.m1.1c"><apply id="S3.SS2.SSS1.Px3.1.m1.1.1.cmml" xref="S3.SS2.SSS1.Px3.1.m1.1.1"><gt id="S3.SS2.SSS1.Px3.1.m1.1.1.1.cmml" xref="S3.SS2.SSS1.Px3.1.m1.1.1.1"></gt><csymbol cd="latexml" id="S3.SS2.SSS1.Px3.1.m1.1.1.2.cmml" xref="S3.SS2.SSS1.Px3.1.m1.1.1.2">absent</csymbol><cn type="integer" id="S3.SS2.SSS1.Px3.1.m1.1.1.3.cmml" xref="S3.SS2.SSS1.Px3.1.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px3.1.m1.1d">\mathbf{&gt;2}</annotation></semantics></math>.</h5>

<div id="S3.SS2.SSS1.Px3.p1" class="ltx_para">
<p id="S3.SS2.SSS1.Px3.p1.3" class="ltx_p">To explore the effect of using CLA sharing factors <math id="S3.SS2.SSS1.Px3.p1.1.m1.1" class="ltx_Math" alttext="&gt;2" display="inline"><semantics id="S3.SS2.SSS1.Px3.p1.1.m1.1a"><mrow id="S3.SS2.SSS1.Px3.p1.1.m1.1.1" xref="S3.SS2.SSS1.Px3.p1.1.m1.1.1.cmml"><mi id="S3.SS2.SSS1.Px3.p1.1.m1.1.1.2" xref="S3.SS2.SSS1.Px3.p1.1.m1.1.1.2.cmml"></mi><mo id="S3.SS2.SSS1.Px3.p1.1.m1.1.1.1" xref="S3.SS2.SSS1.Px3.p1.1.m1.1.1.1.cmml">&gt;</mo><mn id="S3.SS2.SSS1.Px3.p1.1.m1.1.1.3" xref="S3.SS2.SSS1.Px3.p1.1.m1.1.1.3.cmml">2</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px3.p1.1.m1.1b"><apply id="S3.SS2.SSS1.Px3.p1.1.m1.1.1.cmml" xref="S3.SS2.SSS1.Px3.p1.1.m1.1.1"><gt id="S3.SS2.SSS1.Px3.p1.1.m1.1.1.1.cmml" xref="S3.SS2.SSS1.Px3.p1.1.m1.1.1.1"></gt><csymbol cd="latexml" id="S3.SS2.SSS1.Px3.p1.1.m1.1.1.2.cmml" xref="S3.SS2.SSS1.Px3.p1.1.m1.1.1.2">absent</csymbol><cn type="integer" id="S3.SS2.SSS1.Px3.p1.1.m1.1.1.3.cmml" xref="S3.SS2.SSS1.Px3.p1.1.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px3.p1.1.m1.1c">&gt;2</annotation></semantics></math>, we trained MQA-CLA3 and MQA-CLA4 models with head dimension <math id="S3.SS2.SSS1.Px3.p1.2.m2.1" class="ltx_Math" alttext="d_{\text{head}}=128" display="inline"><semantics id="S3.SS2.SSS1.Px3.p1.2.m2.1a"><mrow id="S3.SS2.SSS1.Px3.p1.2.m2.1.1" xref="S3.SS2.SSS1.Px3.p1.2.m2.1.1.cmml"><msub id="S3.SS2.SSS1.Px3.p1.2.m2.1.1.2" xref="S3.SS2.SSS1.Px3.p1.2.m2.1.1.2.cmml"><mi id="S3.SS2.SSS1.Px3.p1.2.m2.1.1.2.2" xref="S3.SS2.SSS1.Px3.p1.2.m2.1.1.2.2.cmml">d</mi><mtext id="S3.SS2.SSS1.Px3.p1.2.m2.1.1.2.3" xref="S3.SS2.SSS1.Px3.p1.2.m2.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS2.SSS1.Px3.p1.2.m2.1.1.1" xref="S3.SS2.SSS1.Px3.p1.2.m2.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS1.Px3.p1.2.m2.1.1.3" xref="S3.SS2.SSS1.Px3.p1.2.m2.1.1.3.cmml">128</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px3.p1.2.m2.1b"><apply id="S3.SS2.SSS1.Px3.p1.2.m2.1.1.cmml" xref="S3.SS2.SSS1.Px3.p1.2.m2.1.1"><eq id="S3.SS2.SSS1.Px3.p1.2.m2.1.1.1.cmml" xref="S3.SS2.SSS1.Px3.p1.2.m2.1.1.1"></eq><apply id="S3.SS2.SSS1.Px3.p1.2.m2.1.1.2.cmml" xref="S3.SS2.SSS1.Px3.p1.2.m2.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS1.Px3.p1.2.m2.1.1.2.1.cmml" xref="S3.SS2.SSS1.Px3.p1.2.m2.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS1.Px3.p1.2.m2.1.1.2.2.cmml" xref="S3.SS2.SSS1.Px3.p1.2.m2.1.1.2.2">𝑑</ci><ci id="S3.SS2.SSS1.Px3.p1.2.m2.1.1.2.3a.cmml" xref="S3.SS2.SSS1.Px3.p1.2.m2.1.1.2.3"><mtext mathsize="70%" id="S3.SS2.SSS1.Px3.p1.2.m2.1.1.2.3.cmml" xref="S3.SS2.SSS1.Px3.p1.2.m2.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS2.SSS1.Px3.p1.2.m2.1.1.3.cmml" xref="S3.SS2.SSS1.Px3.p1.2.m2.1.1.3">128</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px3.p1.2.m2.1c">d_{\text{head}}=128</annotation></semantics></math>. We found that these CLA3 and CLA4 models achieved a Pareto improvement over our plain MQA baselines, matching the KV cache footprint of our baseline MQA models with head dimensions of <math id="S3.SS2.SSS1.Px3.p1.3.m3.2" class="ltx_Math" alttext="d_{\text{head}}\in\{32,46\}" display="inline"><semantics id="S3.SS2.SSS1.Px3.p1.3.m3.2a"><mrow id="S3.SS2.SSS1.Px3.p1.3.m3.2.3" xref="S3.SS2.SSS1.Px3.p1.3.m3.2.3.cmml"><msub id="S3.SS2.SSS1.Px3.p1.3.m3.2.3.2" xref="S3.SS2.SSS1.Px3.p1.3.m3.2.3.2.cmml"><mi id="S3.SS2.SSS1.Px3.p1.3.m3.2.3.2.2" xref="S3.SS2.SSS1.Px3.p1.3.m3.2.3.2.2.cmml">d</mi><mtext id="S3.SS2.SSS1.Px3.p1.3.m3.2.3.2.3" xref="S3.SS2.SSS1.Px3.p1.3.m3.2.3.2.3a.cmml">head</mtext></msub><mo id="S3.SS2.SSS1.Px3.p1.3.m3.2.3.1" xref="S3.SS2.SSS1.Px3.p1.3.m3.2.3.1.cmml">∈</mo><mrow id="S3.SS2.SSS1.Px3.p1.3.m3.2.3.3.2" xref="S3.SS2.SSS1.Px3.p1.3.m3.2.3.3.1.cmml"><mo stretchy="false" id="S3.SS2.SSS1.Px3.p1.3.m3.2.3.3.2.1" xref="S3.SS2.SSS1.Px3.p1.3.m3.2.3.3.1.cmml">{</mo><mn id="S3.SS2.SSS1.Px3.p1.3.m3.1.1" xref="S3.SS2.SSS1.Px3.p1.3.m3.1.1.cmml">32</mn><mo id="S3.SS2.SSS1.Px3.p1.3.m3.2.3.3.2.2" xref="S3.SS2.SSS1.Px3.p1.3.m3.2.3.3.1.cmml">,</mo><mn id="S3.SS2.SSS1.Px3.p1.3.m3.2.2" xref="S3.SS2.SSS1.Px3.p1.3.m3.2.2.cmml">46</mn><mo stretchy="false" id="S3.SS2.SSS1.Px3.p1.3.m3.2.3.3.2.3" xref="S3.SS2.SSS1.Px3.p1.3.m3.2.3.3.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px3.p1.3.m3.2b"><apply id="S3.SS2.SSS1.Px3.p1.3.m3.2.3.cmml" xref="S3.SS2.SSS1.Px3.p1.3.m3.2.3"><in id="S3.SS2.SSS1.Px3.p1.3.m3.2.3.1.cmml" xref="S3.SS2.SSS1.Px3.p1.3.m3.2.3.1"></in><apply id="S3.SS2.SSS1.Px3.p1.3.m3.2.3.2.cmml" xref="S3.SS2.SSS1.Px3.p1.3.m3.2.3.2"><csymbol cd="ambiguous" id="S3.SS2.SSS1.Px3.p1.3.m3.2.3.2.1.cmml" xref="S3.SS2.SSS1.Px3.p1.3.m3.2.3.2">subscript</csymbol><ci id="S3.SS2.SSS1.Px3.p1.3.m3.2.3.2.2.cmml" xref="S3.SS2.SSS1.Px3.p1.3.m3.2.3.2.2">𝑑</ci><ci id="S3.SS2.SSS1.Px3.p1.3.m3.2.3.2.3a.cmml" xref="S3.SS2.SSS1.Px3.p1.3.m3.2.3.2.3"><mtext mathsize="70%" id="S3.SS2.SSS1.Px3.p1.3.m3.2.3.2.3.cmml" xref="S3.SS2.SSS1.Px3.p1.3.m3.2.3.2.3">head</mtext></ci></apply><set id="S3.SS2.SSS1.Px3.p1.3.m3.2.3.3.1.cmml" xref="S3.SS2.SSS1.Px3.p1.3.m3.2.3.3.2"><cn type="integer" id="S3.SS2.SSS1.Px3.p1.3.m3.1.1.cmml" xref="S3.SS2.SSS1.Px3.p1.3.m3.1.1">32</cn><cn type="integer" id="S3.SS2.SSS1.Px3.p1.3.m3.2.2.cmml" xref="S3.SS2.SSS1.Px3.p1.3.m3.2.2">46</cn></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px3.p1.3.m3.2c">d_{\text{head}}\in\{32,46\}</annotation></semantics></math> while achieving better perplexities. However, we found that they achieved worse perplexities than our MQA-CLA2 models at the same KV cache footprint.</p>
</div>
</section>
<section id="S3.SS2.SSS1.Px4" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Ablation: MQA + CLA2 with Non-Uniform Sharing Patterns.</h5>

<div id="S3.SS2.SSS1.Px4.p1" class="ltx_para">
<p id="S3.SS2.SSS1.Px4.p1.1" class="ltx_p">Finally, we explored using different patterns of KV activation sharing in our MQA-CLA2 models.</p>
</div>
<div id="S3.SS2.SSS1.Px4.p2" class="ltx_para">
<p id="S3.SS2.SSS1.Px4.p2.5" class="ltx_p">On the hypothesis that the first and last layers in the model might benefit from special treatment, we trained a model “H128-MQA-CLA2-KeepEnds” which does not share the layer <math id="S3.SS2.SSS1.Px4.p2.1.m1.1" class="ltx_Math" alttext="0" display="inline"><semantics id="S3.SS2.SSS1.Px4.p2.1.m1.1a"><mn id="S3.SS2.SSS1.Px4.p2.1.m1.1.1" xref="S3.SS2.SSS1.Px4.p2.1.m1.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px4.p2.1.m1.1b"><cn type="integer" id="S3.SS2.SSS1.Px4.p2.1.m1.1.1.cmml" xref="S3.SS2.SSS1.Px4.p2.1.m1.1.1">0</cn></annotation-xml></semantics></math> KV cache with any other layers, and instead groups layer <math id="S3.SS2.SSS1.Px4.p2.2.m2.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S3.SS2.SSS1.Px4.p2.2.m2.1a"><mn id="S3.SS2.SSS1.Px4.p2.2.m2.1.1" xref="S3.SS2.SSS1.Px4.p2.2.m2.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px4.p2.2.m2.1b"><cn type="integer" id="S3.SS2.SSS1.Px4.p2.2.m2.1.1.cmml" xref="S3.SS2.SSS1.Px4.p2.2.m2.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px4.p2.2.m2.1c">1</annotation></semantics></math> with layer <math id="S3.SS2.SSS1.Px4.p2.3.m3.1" class="ltx_Math" alttext="2" display="inline"><semantics id="S3.SS2.SSS1.Px4.p2.3.m3.1a"><mn id="S3.SS2.SSS1.Px4.p2.3.m3.1.1" xref="S3.SS2.SSS1.Px4.p2.3.m3.1.1.cmml">2</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px4.p2.3.m3.1b"><cn type="integer" id="S3.SS2.SSS1.Px4.p2.3.m3.1.1.cmml" xref="S3.SS2.SSS1.Px4.p2.3.m3.1.1">2</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px4.p2.3.m3.1c">2</annotation></semantics></math>, groups layer <math id="S3.SS2.SSS1.Px4.p2.4.m4.1" class="ltx_Math" alttext="3" display="inline"><semantics id="S3.SS2.SSS1.Px4.p2.4.m4.1a"><mn id="S3.SS2.SSS1.Px4.p2.4.m4.1.1" xref="S3.SS2.SSS1.Px4.p2.4.m4.1.1.cmml">3</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px4.p2.4.m4.1b"><cn type="integer" id="S3.SS2.SSS1.Px4.p2.4.m4.1.1.cmml" xref="S3.SS2.SSS1.Px4.p2.4.m4.1.1">3</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px4.p2.4.m4.1c">3</annotation></semantics></math> with layer <math id="S3.SS2.SSS1.Px4.p2.5.m5.1" class="ltx_Math" alttext="4" display="inline"><semantics id="S3.SS2.SSS1.Px4.p2.5.m5.1a"><mn id="S3.SS2.SSS1.Px4.p2.5.m5.1.1" xref="S3.SS2.SSS1.Px4.p2.5.m5.1.1.cmml">4</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px4.p2.5.m5.1b"><cn type="integer" id="S3.SS2.SSS1.Px4.p2.5.m5.1.1.cmml" xref="S3.SS2.SSS1.Px4.p2.5.m5.1.1">4</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px4.p2.5.m5.1c">4</annotation></semantics></math>, and so on. This also has the effect of giving the final layer its own KV cache separate from all other layers.</p>
</div>
<div id="S3.SS2.SSS1.Px4.p3" class="ltx_para">
<p id="S3.SS2.SSS1.Px4.p3.7" class="ltx_p">We also explored imbalanced configurations with all the KV-cache-producing layers concentrated at either the beginning or end of the model. We trained a model “H128-MQA-CLA2-DenseFront” consisting of <math id="S3.SS2.SSS1.Px4.p3.1.m1.1" class="ltx_Math" alttext="10" display="inline"><semantics id="S3.SS2.SSS1.Px4.p3.1.m1.1a"><mn id="S3.SS2.SSS1.Px4.p3.1.m1.1.1" xref="S3.SS2.SSS1.Px4.p3.1.m1.1.1.cmml">10</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px4.p3.1.m1.1b"><cn type="integer" id="S3.SS2.SSS1.Px4.p3.1.m1.1.1.cmml" xref="S3.SS2.SSS1.Px4.p3.1.m1.1.1">10</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px4.p3.1.m1.1c">10</annotation></semantics></math> non-CLA layers, followed by <math id="S3.SS2.SSS1.Px4.p3.2.m2.1" class="ltx_Math" alttext="9" display="inline"><semantics id="S3.SS2.SSS1.Px4.p3.2.m2.1a"><mn id="S3.SS2.SSS1.Px4.p3.2.m2.1.1" xref="S3.SS2.SSS1.Px4.p3.2.m2.1.1.cmml">9</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px4.p3.2.m2.1b"><cn type="integer" id="S3.SS2.SSS1.Px4.p3.2.m2.1.1.cmml" xref="S3.SS2.SSS1.Px4.p3.2.m2.1.1">9</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px4.p3.2.m2.1c">9</annotation></semantics></math> CLA layers all using the KV activations of layer <math id="S3.SS2.SSS1.Px4.p3.3.m3.1" class="ltx_Math" alttext="9" display="inline"><semantics id="S3.SS2.SSS1.Px4.p3.3.m3.1a"><mn id="S3.SS2.SSS1.Px4.p3.3.m3.1.1" xref="S3.SS2.SSS1.Px4.p3.3.m3.1.1.cmml">9</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px4.p3.3.m3.1b"><cn type="integer" id="S3.SS2.SSS1.Px4.p3.3.m3.1.1.cmml" xref="S3.SS2.SSS1.Px4.p3.3.m3.1.1">9</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px4.p3.3.m3.1c">9</annotation></semantics></math>, and a final layer with its own KV cache. Similarly, we trained a model “H128-MQA-CLA2-DenseBack” consisting of <math id="S3.SS2.SSS1.Px4.p3.4.m4.1" class="ltx_Math" alttext="2" display="inline"><semantics id="S3.SS2.SSS1.Px4.p3.4.m4.1a"><mn id="S3.SS2.SSS1.Px4.p3.4.m4.1.1" xref="S3.SS2.SSS1.Px4.p3.4.m4.1.1.cmml">2</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px4.p3.4.m4.1b"><cn type="integer" id="S3.SS2.SSS1.Px4.p3.4.m4.1.1.cmml" xref="S3.SS2.SSS1.Px4.p3.4.m4.1.1">2</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px4.p3.4.m4.1c">2</annotation></semantics></math> non-CLA layers, followed by a run of <math id="S3.SS2.SSS1.Px4.p3.5.m5.1" class="ltx_Math" alttext="10" display="inline"><semantics id="S3.SS2.SSS1.Px4.p3.5.m5.1a"><mn id="S3.SS2.SSS1.Px4.p3.5.m5.1.1" xref="S3.SS2.SSS1.Px4.p3.5.m5.1.1.cmml">10</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px4.p3.5.m5.1b"><cn type="integer" id="S3.SS2.SSS1.Px4.p3.5.m5.1.1.cmml" xref="S3.SS2.SSS1.Px4.p3.5.m5.1.1">10</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px4.p3.5.m5.1c">10</annotation></semantics></math> CLA layers all using the KV activations of layer <math id="S3.SS2.SSS1.Px4.p3.6.m6.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S3.SS2.SSS1.Px4.p3.6.m6.1a"><mn id="S3.SS2.SSS1.Px4.p3.6.m6.1.1" xref="S3.SS2.SSS1.Px4.p3.6.m6.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px4.p3.6.m6.1b"><cn type="integer" id="S3.SS2.SSS1.Px4.p3.6.m6.1.1.cmml" xref="S3.SS2.SSS1.Px4.p3.6.m6.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px4.p3.6.m6.1c">1</annotation></semantics></math>, and finally <math id="S3.SS2.SSS1.Px4.p3.7.m7.1" class="ltx_Math" alttext="9" display="inline"><semantics id="S3.SS2.SSS1.Px4.p3.7.m7.1a"><mn id="S3.SS2.SSS1.Px4.p3.7.m7.1.1" xref="S3.SS2.SSS1.Px4.p3.7.m7.1.1.cmml">9</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px4.p3.7.m7.1b"><cn type="integer" id="S3.SS2.SSS1.Px4.p3.7.m7.1.1.cmml" xref="S3.SS2.SSS1.Px4.p3.7.m7.1.1">9</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px4.p3.7.m7.1c">9</annotation></semantics></math> non-CLA layers.</p>
</div>
<div id="S3.SS2.SSS1.Px4.p4" class="ltx_para">
<p id="S3.SS2.SSS1.Px4.p4.1" class="ltx_p">We found that all of these alternative CLA sharing patterns achieve worse perplexities than the corresponding MQA-CLA2 model with a uniform sharing pattern, while also requiring slightly more KV cache memory.</p>
</div>
</section>
</section>
<section id="S3.SS2.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.2.2 </span>Robustness to Learning Rate Tuning</h4>

<div id="S3.SS2.SSS2.p1" class="ltx_para">
<p id="S3.SS2.SSS2.p1.1" class="ltx_p">The relative performance of different model architectures can change depending on the learning rates at which they are evaluated. To account for the effects of the learning rate on our results, we conducted learning rate tuning experiments on three models of interest from our initial 1B-scale design space exploration. These learning rate tuning experiments help us verify that CLA continues to provide benefits even when compared to baselines trained at their optimal learning rates.</p>
</div>
<div id="S3.SS2.SSS2.p2" class="ltx_para">
<p id="S3.SS2.SSS2.p2.5" class="ltx_p">We chose to tune the learning rate for the baseline models H128-MQA and H64-MQA, as well as the CLA model H128-MQA-CLA2. In our initial design space exploration, our results for these models indicated that CLA makes it possible to shrink the KV cache footprint of an MQA model with <math id="S3.SS2.SSS2.p2.1.m1.1" class="ltx_Math" alttext="d_{\text{head}}=128" display="inline"><semantics id="S3.SS2.SSS2.p2.1.m1.1a"><mrow id="S3.SS2.SSS2.p2.1.m1.1.1" xref="S3.SS2.SSS2.p2.1.m1.1.1.cmml"><msub id="S3.SS2.SSS2.p2.1.m1.1.1.2" xref="S3.SS2.SSS2.p2.1.m1.1.1.2.cmml"><mi id="S3.SS2.SSS2.p2.1.m1.1.1.2.2" xref="S3.SS2.SSS2.p2.1.m1.1.1.2.2.cmml">d</mi><mtext id="S3.SS2.SSS2.p2.1.m1.1.1.2.3" xref="S3.SS2.SSS2.p2.1.m1.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS2.SSS2.p2.1.m1.1.1.1" xref="S3.SS2.SSS2.p2.1.m1.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS2.p2.1.m1.1.1.3" xref="S3.SS2.SSS2.p2.1.m1.1.1.3.cmml">128</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p2.1.m1.1b"><apply id="S3.SS2.SSS2.p2.1.m1.1.1.cmml" xref="S3.SS2.SSS2.p2.1.m1.1.1"><eq id="S3.SS2.SSS2.p2.1.m1.1.1.1.cmml" xref="S3.SS2.SSS2.p2.1.m1.1.1.1"></eq><apply id="S3.SS2.SSS2.p2.1.m1.1.1.2.cmml" xref="S3.SS2.SSS2.p2.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS2.p2.1.m1.1.1.2.1.cmml" xref="S3.SS2.SSS2.p2.1.m1.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS2.p2.1.m1.1.1.2.2.cmml" xref="S3.SS2.SSS2.p2.1.m1.1.1.2.2">𝑑</ci><ci id="S3.SS2.SSS2.p2.1.m1.1.1.2.3a.cmml" xref="S3.SS2.SSS2.p2.1.m1.1.1.2.3"><mtext mathsize="70%" id="S3.SS2.SSS2.p2.1.m1.1.1.2.3.cmml" xref="S3.SS2.SSS2.p2.1.m1.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS2.SSS2.p2.1.m1.1.1.3.cmml" xref="S3.SS2.SSS2.p2.1.m1.1.1.3">128</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p2.1.m1.1c">d_{\text{head}}=128</annotation></semantics></math> by a factor of <math id="S3.SS2.SSS2.p2.2.m2.1" class="ltx_math_unparsed" alttext="2\times" display="inline"><semantics id="S3.SS2.SSS2.p2.2.m2.1a"><mrow id="S3.SS2.SSS2.p2.2.m2.1b"><mn id="S3.SS2.SSS2.p2.2.m2.1.1">2</mn><mo lspace="0.222em" id="S3.SS2.SSS2.p2.2.m2.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p2.2.m2.1c">2\times</annotation></semantics></math> while incurring only a small (<math id="S3.SS2.SSS2.p2.3.m3.1" class="ltx_Math" alttext="0.06" display="inline"><semantics id="S3.SS2.SSS2.p2.3.m3.1a"><mn id="S3.SS2.SSS2.p2.3.m3.1.1" xref="S3.SS2.SSS2.p2.3.m3.1.1.cmml">0.06</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p2.3.m3.1b"><cn type="float" id="S3.SS2.SSS2.p2.3.m3.1.1.cmml" xref="S3.SS2.SSS2.p2.3.m3.1.1">0.06</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p2.3.m3.1c">0.06</annotation></semantics></math> point) degradation in perplexity, or to create a model with the same KV cache footprint as an MQA model with <math id="S3.SS2.SSS2.p2.4.m4.1" class="ltx_Math" alttext="d_{\text{head}}=64" display="inline"><semantics id="S3.SS2.SSS2.p2.4.m4.1a"><mrow id="S3.SS2.SSS2.p2.4.m4.1.1" xref="S3.SS2.SSS2.p2.4.m4.1.1.cmml"><msub id="S3.SS2.SSS2.p2.4.m4.1.1.2" xref="S3.SS2.SSS2.p2.4.m4.1.1.2.cmml"><mi id="S3.SS2.SSS2.p2.4.m4.1.1.2.2" xref="S3.SS2.SSS2.p2.4.m4.1.1.2.2.cmml">d</mi><mtext id="S3.SS2.SSS2.p2.4.m4.1.1.2.3" xref="S3.SS2.SSS2.p2.4.m4.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS2.SSS2.p2.4.m4.1.1.1" xref="S3.SS2.SSS2.p2.4.m4.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS2.p2.4.m4.1.1.3" xref="S3.SS2.SSS2.p2.4.m4.1.1.3.cmml">64</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p2.4.m4.1b"><apply id="S3.SS2.SSS2.p2.4.m4.1.1.cmml" xref="S3.SS2.SSS2.p2.4.m4.1.1"><eq id="S3.SS2.SSS2.p2.4.m4.1.1.1.cmml" xref="S3.SS2.SSS2.p2.4.m4.1.1.1"></eq><apply id="S3.SS2.SSS2.p2.4.m4.1.1.2.cmml" xref="S3.SS2.SSS2.p2.4.m4.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS2.p2.4.m4.1.1.2.1.cmml" xref="S3.SS2.SSS2.p2.4.m4.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS2.p2.4.m4.1.1.2.2.cmml" xref="S3.SS2.SSS2.p2.4.m4.1.1.2.2">𝑑</ci><ci id="S3.SS2.SSS2.p2.4.m4.1.1.2.3a.cmml" xref="S3.SS2.SSS2.p2.4.m4.1.1.2.3"><mtext mathsize="70%" id="S3.SS2.SSS2.p2.4.m4.1.1.2.3.cmml" xref="S3.SS2.SSS2.p2.4.m4.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS2.SSS2.p2.4.m4.1.1.3.cmml" xref="S3.SS2.SSS2.p2.4.m4.1.1.3">64</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p2.4.m4.1c">d_{\text{head}}=64</annotation></semantics></math> while enjoying a substantial (<math id="S3.SS2.SSS2.p2.5.m5.1" class="ltx_Math" alttext="0.21" display="inline"><semantics id="S3.SS2.SSS2.p2.5.m5.1a"><mn id="S3.SS2.SSS2.p2.5.m5.1.1" xref="S3.SS2.SSS2.p2.5.m5.1.1.cmml">0.21</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p2.5.m5.1b"><cn type="float" id="S3.SS2.SSS2.p2.5.m5.1.1.cmml" xref="S3.SS2.SSS2.p2.5.m5.1.1">0.21</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p2.5.m5.1c">0.21</annotation></semantics></math> point) improvement in perplexity. We wanted to verify that this qualitative pattern continues to hold when all models are trained with well-tuned learning rates.</p>
</div>
<section id="S3.SS2.SSS2.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Learning Rate Tuning Strategy.</h5>

<div id="S3.SS2.SSS2.Px1.p1" class="ltx_para">
<p id="S3.SS2.SSS2.Px1.p1.2" class="ltx_p">For each of our three model configurations, we swept the learning rate upwards from an initial value of <math id="S3.SS2.SSS2.Px1.p1.1.m1.1" class="ltx_Math" alttext="3\times 10^{-4}" display="inline"><semantics id="S3.SS2.SSS2.Px1.p1.1.m1.1a"><mrow id="S3.SS2.SSS2.Px1.p1.1.m1.1.1" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.cmml"><mn id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.2" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.2.cmml">3</mn><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.1" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.1.cmml">×</mo><msup id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.cmml"><mn id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.2" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.2.cmml">10</mn><mrow id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.3" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.3.cmml"><mo id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.3a" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.3.cmml">−</mo><mn id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.3.2" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.3.2.cmml">4</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.Px1.p1.1.m1.1b"><apply id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.cmml" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1"><times id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.1.cmml" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.1"></times><cn type="integer" id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.2.cmml" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.2">3</cn><apply id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.cmml" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.1.cmml" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.2.cmml" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.2">10</cn><apply id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.3.cmml" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.3"><minus id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.3.1.cmml" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.3"></minus><cn type="integer" id="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.3.2.cmml" xref="S3.SS2.SSS2.Px1.p1.1.m1.1.1.3.3.2">4</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.Px1.p1.1.m1.1c">3\times 10^{-4}</annotation></semantics></math> in multiplicative increments of <math id="S3.SS2.SSS2.Px1.p1.2.m2.1" class="ltx_math_unparsed" alttext="1.5\times" display="inline"><semantics id="S3.SS2.SSS2.Px1.p1.2.m2.1a"><mrow id="S3.SS2.SSS2.Px1.p1.2.m2.1b"><mn id="S3.SS2.SSS2.Px1.p1.2.m2.1.1">1.5</mn><mo lspace="0.222em" id="S3.SS2.SSS2.Px1.p1.2.m2.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S3.SS2.SSS2.Px1.p1.2.m2.1c">1.5\times</annotation></semantics></math>. We ended our sweep for each model at the point where validation perplexity stopped improving. We treat the learning rate which achieved the lowest validation perplexity for each model as an approximation of that model’s optimal learning rate.</p>
</div>
</section>
<section id="S3.SS2.SSS2.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Results.</h5>

<div id="S3.SS2.SSS2.Px2.p1" class="ltx_para">
<p id="S3.SS2.SSS2.Px2.p1.2" class="ltx_p">We found an optimal learning rate of <math id="S3.SS2.SSS2.Px2.p1.1.m1.1" class="ltx_Math" alttext="\text{LR}=1.5\times 10^{-3}" display="inline"><semantics id="S3.SS2.SSS2.Px2.p1.1.m1.1a"><mrow id="S3.SS2.SSS2.Px2.p1.1.m1.1.1" xref="S3.SS2.SSS2.Px2.p1.1.m1.1.1.cmml"><mtext id="S3.SS2.SSS2.Px2.p1.1.m1.1.1.2" xref="S3.SS2.SSS2.Px2.p1.1.m1.1.1.2a.cmml">LR</mtext><mo id="S3.SS2.SSS2.Px2.p1.1.m1.1.1.1" xref="S3.SS2.SSS2.Px2.p1.1.m1.1.1.1.cmml">=</mo><mrow id="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3" xref="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.cmml"><mn id="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.2" xref="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.2.cmml">1.5</mn><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.1" xref="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.1.cmml">×</mo><msup id="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.3" xref="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.3.cmml"><mn id="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.3.2" xref="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.3.2.cmml">10</mn><mrow id="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.3.3" xref="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.3.3.cmml"><mo id="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.3.3a" xref="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.3.3.cmml">−</mo><mn id="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.3.3.2" xref="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.3.3.2.cmml">3</mn></mrow></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.Px2.p1.1.m1.1b"><apply id="S3.SS2.SSS2.Px2.p1.1.m1.1.1.cmml" xref="S3.SS2.SSS2.Px2.p1.1.m1.1.1"><eq id="S3.SS2.SSS2.Px2.p1.1.m1.1.1.1.cmml" xref="S3.SS2.SSS2.Px2.p1.1.m1.1.1.1"></eq><ci id="S3.SS2.SSS2.Px2.p1.1.m1.1.1.2a.cmml" xref="S3.SS2.SSS2.Px2.p1.1.m1.1.1.2"><mtext id="S3.SS2.SSS2.Px2.p1.1.m1.1.1.2.cmml" xref="S3.SS2.SSS2.Px2.p1.1.m1.1.1.2">LR</mtext></ci><apply id="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.cmml" xref="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3"><times id="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.1.cmml" xref="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.1"></times><cn type="float" id="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.2.cmml" xref="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.2">1.5</cn><apply id="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.3.cmml" xref="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.3"><csymbol cd="ambiguous" id="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.3.1.cmml" xref="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.3">superscript</csymbol><cn type="integer" id="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.3.2.cmml" xref="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.3.2">10</cn><apply id="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.3.3.cmml" xref="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.3.3"><minus id="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.3.3.1.cmml" xref="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.3.3"></minus><cn type="integer" id="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.3.3.2.cmml" xref="S3.SS2.SSS2.Px2.p1.1.m1.1.1.3.3.3.2">3</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.Px2.p1.1.m1.1c">\text{LR}=1.5\times 10^{-3}</annotation></semantics></math> for our H128-MQA baseline, and a higher optimal learning rate of <math id="S3.SS2.SSS2.Px2.p1.2.m2.1" class="ltx_Math" alttext="\text{LR}=2.25\times 10^{-3}" display="inline"><semantics id="S3.SS2.SSS2.Px2.p1.2.m2.1a"><mrow id="S3.SS2.SSS2.Px2.p1.2.m2.1.1" xref="S3.SS2.SSS2.Px2.p1.2.m2.1.1.cmml"><mtext id="S3.SS2.SSS2.Px2.p1.2.m2.1.1.2" xref="S3.SS2.SSS2.Px2.p1.2.m2.1.1.2a.cmml">LR</mtext><mo id="S3.SS2.SSS2.Px2.p1.2.m2.1.1.1" xref="S3.SS2.SSS2.Px2.p1.2.m2.1.1.1.cmml">=</mo><mrow id="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3" xref="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.cmml"><mn id="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.2" xref="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.2.cmml">2.25</mn><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.1" xref="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.1.cmml">×</mo><msup id="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.3" xref="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.3.cmml"><mn id="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.3.2" xref="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.3.2.cmml">10</mn><mrow id="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.3.3" xref="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.3.3.cmml"><mo id="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.3.3a" xref="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.3.3.cmml">−</mo><mn id="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.3.3.2" xref="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.3.3.2.cmml">3</mn></mrow></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.Px2.p1.2.m2.1b"><apply id="S3.SS2.SSS2.Px2.p1.2.m2.1.1.cmml" xref="S3.SS2.SSS2.Px2.p1.2.m2.1.1"><eq id="S3.SS2.SSS2.Px2.p1.2.m2.1.1.1.cmml" xref="S3.SS2.SSS2.Px2.p1.2.m2.1.1.1"></eq><ci id="S3.SS2.SSS2.Px2.p1.2.m2.1.1.2a.cmml" xref="S3.SS2.SSS2.Px2.p1.2.m2.1.1.2"><mtext id="S3.SS2.SSS2.Px2.p1.2.m2.1.1.2.cmml" xref="S3.SS2.SSS2.Px2.p1.2.m2.1.1.2">LR</mtext></ci><apply id="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.cmml" xref="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3"><times id="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.1.cmml" xref="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.1"></times><cn type="float" id="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.2.cmml" xref="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.2">2.25</cn><apply id="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.3.cmml" xref="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.3"><csymbol cd="ambiguous" id="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.3.1.cmml" xref="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.3">superscript</csymbol><cn type="integer" id="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.3.2.cmml" xref="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.3.2">10</cn><apply id="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.3.3.cmml" xref="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.3.3"><minus id="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.3.3.1.cmml" xref="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.3.3"></minus><cn type="integer" id="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.3.3.2.cmml" xref="S3.SS2.SSS2.Px2.p1.2.m2.1.1.3.3.3.2">3</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.Px2.p1.2.m2.1c">\text{LR}=2.25\times 10^{-3}</annotation></semantics></math> for both our H64-MQA baseline and our H128-MQA-CLA2 model.</p>
</div>
<div id="S3.SS2.SSS2.Px2.p2" class="ltx_para">
<p id="S3.SS2.SSS2.Px2.p2.5" class="ltx_p">The validation perplexity results from our 1B-scale learning rate tuning experiments can be found in Table <a href="#S3.T3" title="Table 3 ‣ Results. ‣ 3.2.2 Robustness to Learning Rate Tuning ‣ 3.2 Experiments at 1B-Parameter Scale ‣ 3 Pretraining Experiments ‣ Reducing Transformer Key-Value Cache Size with Cross-Layer Attention" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>. When comparing all three models at their best learning rates, we found that the qualitative result from our design space exploration continues to hold: our CLA2 model incurs only a small (<math id="S3.SS2.SSS2.Px2.p2.1.m1.1" class="ltx_Math" alttext="0.04" display="inline"><semantics id="S3.SS2.SSS2.Px2.p2.1.m1.1a"><mn id="S3.SS2.SSS2.Px2.p2.1.m1.1.1" xref="S3.SS2.SSS2.Px2.p2.1.m1.1.1.cmml">0.04</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.Px2.p2.1.m1.1b"><cn type="float" id="S3.SS2.SSS2.Px2.p2.1.m1.1.1.cmml" xref="S3.SS2.SSS2.Px2.p2.1.m1.1.1">0.04</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.Px2.p2.1.m1.1c">0.04</annotation></semantics></math> point) validation perplexity degradation relative to our <math id="S3.SS2.SSS2.Px2.p2.2.m2.1" class="ltx_Math" alttext="d_{\text{head}}=128" display="inline"><semantics id="S3.SS2.SSS2.Px2.p2.2.m2.1a"><mrow id="S3.SS2.SSS2.Px2.p2.2.m2.1.1" xref="S3.SS2.SSS2.Px2.p2.2.m2.1.1.cmml"><msub id="S3.SS2.SSS2.Px2.p2.2.m2.1.1.2" xref="S3.SS2.SSS2.Px2.p2.2.m2.1.1.2.cmml"><mi id="S3.SS2.SSS2.Px2.p2.2.m2.1.1.2.2" xref="S3.SS2.SSS2.Px2.p2.2.m2.1.1.2.2.cmml">d</mi><mtext id="S3.SS2.SSS2.Px2.p2.2.m2.1.1.2.3" xref="S3.SS2.SSS2.Px2.p2.2.m2.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS2.SSS2.Px2.p2.2.m2.1.1.1" xref="S3.SS2.SSS2.Px2.p2.2.m2.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS2.Px2.p2.2.m2.1.1.3" xref="S3.SS2.SSS2.Px2.p2.2.m2.1.1.3.cmml">128</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.Px2.p2.2.m2.1b"><apply id="S3.SS2.SSS2.Px2.p2.2.m2.1.1.cmml" xref="S3.SS2.SSS2.Px2.p2.2.m2.1.1"><eq id="S3.SS2.SSS2.Px2.p2.2.m2.1.1.1.cmml" xref="S3.SS2.SSS2.Px2.p2.2.m2.1.1.1"></eq><apply id="S3.SS2.SSS2.Px2.p2.2.m2.1.1.2.cmml" xref="S3.SS2.SSS2.Px2.p2.2.m2.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS2.Px2.p2.2.m2.1.1.2.1.cmml" xref="S3.SS2.SSS2.Px2.p2.2.m2.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS2.Px2.p2.2.m2.1.1.2.2.cmml" xref="S3.SS2.SSS2.Px2.p2.2.m2.1.1.2.2">𝑑</ci><ci id="S3.SS2.SSS2.Px2.p2.2.m2.1.1.2.3a.cmml" xref="S3.SS2.SSS2.Px2.p2.2.m2.1.1.2.3"><mtext mathsize="70%" id="S3.SS2.SSS2.Px2.p2.2.m2.1.1.2.3.cmml" xref="S3.SS2.SSS2.Px2.p2.2.m2.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS2.SSS2.Px2.p2.2.m2.1.1.3.cmml" xref="S3.SS2.SSS2.Px2.p2.2.m2.1.1.3">128</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.Px2.p2.2.m2.1c">d_{\text{head}}=128</annotation></semantics></math> baseline while enjoying a <math id="S3.SS2.SSS2.Px2.p2.3.m3.1" class="ltx_math_unparsed" alttext="2\times" display="inline"><semantics id="S3.SS2.SSS2.Px2.p2.3.m3.1a"><mrow id="S3.SS2.SSS2.Px2.p2.3.m3.1b"><mn id="S3.SS2.SSS2.Px2.p2.3.m3.1.1">2</mn><mo lspace="0.222em" id="S3.SS2.SSS2.Px2.p2.3.m3.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S3.SS2.SSS2.Px2.p2.3.m3.1c">2\times</annotation></semantics></math> smaller KV cache footprint, and achieves a substantial (<math id="S3.SS2.SSS2.Px2.p2.4.m4.1" class="ltx_Math" alttext="0.31" display="inline"><semantics id="S3.SS2.SSS2.Px2.p2.4.m4.1a"><mn id="S3.SS2.SSS2.Px2.p2.4.m4.1.1" xref="S3.SS2.SSS2.Px2.p2.4.m4.1.1.cmml">0.31</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.Px2.p2.4.m4.1b"><cn type="float" id="S3.SS2.SSS2.Px2.p2.4.m4.1.1.cmml" xref="S3.SS2.SSS2.Px2.p2.4.m4.1.1">0.31</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.Px2.p2.4.m4.1c">0.31</annotation></semantics></math> point) validation perplexity improvement compared to our <math id="S3.SS2.SSS2.Px2.p2.5.m5.1" class="ltx_Math" alttext="d_{\text{head}}=64" display="inline"><semantics id="S3.SS2.SSS2.Px2.p2.5.m5.1a"><mrow id="S3.SS2.SSS2.Px2.p2.5.m5.1.1" xref="S3.SS2.SSS2.Px2.p2.5.m5.1.1.cmml"><msub id="S3.SS2.SSS2.Px2.p2.5.m5.1.1.2" xref="S3.SS2.SSS2.Px2.p2.5.m5.1.1.2.cmml"><mi id="S3.SS2.SSS2.Px2.p2.5.m5.1.1.2.2" xref="S3.SS2.SSS2.Px2.p2.5.m5.1.1.2.2.cmml">d</mi><mtext id="S3.SS2.SSS2.Px2.p2.5.m5.1.1.2.3" xref="S3.SS2.SSS2.Px2.p2.5.m5.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS2.SSS2.Px2.p2.5.m5.1.1.1" xref="S3.SS2.SSS2.Px2.p2.5.m5.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS2.Px2.p2.5.m5.1.1.3" xref="S3.SS2.SSS2.Px2.p2.5.m5.1.1.3.cmml">64</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.Px2.p2.5.m5.1b"><apply id="S3.SS2.SSS2.Px2.p2.5.m5.1.1.cmml" xref="S3.SS2.SSS2.Px2.p2.5.m5.1.1"><eq id="S3.SS2.SSS2.Px2.p2.5.m5.1.1.1.cmml" xref="S3.SS2.SSS2.Px2.p2.5.m5.1.1.1"></eq><apply id="S3.SS2.SSS2.Px2.p2.5.m5.1.1.2.cmml" xref="S3.SS2.SSS2.Px2.p2.5.m5.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS2.Px2.p2.5.m5.1.1.2.1.cmml" xref="S3.SS2.SSS2.Px2.p2.5.m5.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS2.Px2.p2.5.m5.1.1.2.2.cmml" xref="S3.SS2.SSS2.Px2.p2.5.m5.1.1.2.2">𝑑</ci><ci id="S3.SS2.SSS2.Px2.p2.5.m5.1.1.2.3a.cmml" xref="S3.SS2.SSS2.Px2.p2.5.m5.1.1.2.3"><mtext mathsize="70%" id="S3.SS2.SSS2.Px2.p2.5.m5.1.1.2.3.cmml" xref="S3.SS2.SSS2.Px2.p2.5.m5.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS2.SSS2.Px2.p2.5.m5.1.1.3.cmml" xref="S3.SS2.SSS2.Px2.p2.5.m5.1.1.3">64</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.Px2.p2.5.m5.1c">d_{\text{head}}=64</annotation></semantics></math> baseline while using the same amount of KV cache memory.</p>
</div>
<div id="S3.SS2.SSS2.Px2.p3" class="ltx_para">
<p id="S3.SS2.SSS2.Px2.p3.6" class="ltx_p">To further validate our results, we also evaluate our three learning-rate-tuned 1B-scale models under EleutherAI’s LM Eval Harness <cite class="ltx_cite ltx_citemacro_citep">(Gao et al., <a href="#bib.bib13" title="" class="ltx_ref">2023</a>)</cite> on Wikitext <cite class="ltx_cite ltx_citemacro_citep">(Merity et al., <a href="#bib.bib27" title="" class="ltx_ref">2016</a>)</cite> perplexity and seven standard downstream benchmarks. We report the results of these evaluation in tables <a href="#S3.T3" title="Table 3 ‣ Results. ‣ 3.2.2 Robustness to Learning Rate Tuning ‣ 3.2 Experiments at 1B-Parameter Scale ‣ 3 Pretraining Experiments ‣ Reducing Transformer Key-Value Cache Size with Cross-Layer Attention" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> and <a href="#S3.T4" title="Table 4 ‣ Results. ‣ 3.2.2 Robustness to Learning Rate Tuning ‣ 3.2 Experiments at 1B-Parameter Scale ‣ 3 Pretraining Experiments ‣ Reducing Transformer Key-Value Cache Size with Cross-Layer Attention" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>. On Wikitext perplexity, we observe a similar pattern as with validation perplexity, with our tuned CLA2 model achieving approximately the same (<math id="S3.SS2.SSS2.Px2.p3.1.m1.1" class="ltx_Math" alttext="0.01" display="inline"><semantics id="S3.SS2.SSS2.Px2.p3.1.m1.1a"><mn id="S3.SS2.SSS2.Px2.p3.1.m1.1.1" xref="S3.SS2.SSS2.Px2.p3.1.m1.1.1.cmml">0.01</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.Px2.p3.1.m1.1b"><cn type="float" id="S3.SS2.SSS2.Px2.p3.1.m1.1.1.cmml" xref="S3.SS2.SSS2.Px2.p3.1.m1.1.1">0.01</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.Px2.p3.1.m1.1c">0.01</annotation></semantics></math> points better) perplexity as our <math id="S3.SS2.SSS2.Px2.p3.2.m2.1" class="ltx_Math" alttext="d_{\text{head}}=128" display="inline"><semantics id="S3.SS2.SSS2.Px2.p3.2.m2.1a"><mrow id="S3.SS2.SSS2.Px2.p3.2.m2.1.1" xref="S3.SS2.SSS2.Px2.p3.2.m2.1.1.cmml"><msub id="S3.SS2.SSS2.Px2.p3.2.m2.1.1.2" xref="S3.SS2.SSS2.Px2.p3.2.m2.1.1.2.cmml"><mi id="S3.SS2.SSS2.Px2.p3.2.m2.1.1.2.2" xref="S3.SS2.SSS2.Px2.p3.2.m2.1.1.2.2.cmml">d</mi><mtext id="S3.SS2.SSS2.Px2.p3.2.m2.1.1.2.3" xref="S3.SS2.SSS2.Px2.p3.2.m2.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS2.SSS2.Px2.p3.2.m2.1.1.1" xref="S3.SS2.SSS2.Px2.p3.2.m2.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS2.Px2.p3.2.m2.1.1.3" xref="S3.SS2.SSS2.Px2.p3.2.m2.1.1.3.cmml">128</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.Px2.p3.2.m2.1b"><apply id="S3.SS2.SSS2.Px2.p3.2.m2.1.1.cmml" xref="S3.SS2.SSS2.Px2.p3.2.m2.1.1"><eq id="S3.SS2.SSS2.Px2.p3.2.m2.1.1.1.cmml" xref="S3.SS2.SSS2.Px2.p3.2.m2.1.1.1"></eq><apply id="S3.SS2.SSS2.Px2.p3.2.m2.1.1.2.cmml" xref="S3.SS2.SSS2.Px2.p3.2.m2.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS2.Px2.p3.2.m2.1.1.2.1.cmml" xref="S3.SS2.SSS2.Px2.p3.2.m2.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS2.Px2.p3.2.m2.1.1.2.2.cmml" xref="S3.SS2.SSS2.Px2.p3.2.m2.1.1.2.2">𝑑</ci><ci id="S3.SS2.SSS2.Px2.p3.2.m2.1.1.2.3a.cmml" xref="S3.SS2.SSS2.Px2.p3.2.m2.1.1.2.3"><mtext mathsize="70%" id="S3.SS2.SSS2.Px2.p3.2.m2.1.1.2.3.cmml" xref="S3.SS2.SSS2.Px2.p3.2.m2.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS2.SSS2.Px2.p3.2.m2.1.1.3.cmml" xref="S3.SS2.SSS2.Px2.p3.2.m2.1.1.3">128</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.Px2.p3.2.m2.1c">d_{\text{head}}=128</annotation></semantics></math> baseline, and substantially (<math id="S3.SS2.SSS2.Px2.p3.3.m3.1" class="ltx_Math" alttext="0.71" display="inline"><semantics id="S3.SS2.SSS2.Px2.p3.3.m3.1a"><mn id="S3.SS2.SSS2.Px2.p3.3.m3.1.1" xref="S3.SS2.SSS2.Px2.p3.3.m3.1.1.cmml">0.71</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.Px2.p3.3.m3.1b"><cn type="float" id="S3.SS2.SSS2.Px2.p3.3.m3.1.1.cmml" xref="S3.SS2.SSS2.Px2.p3.3.m3.1.1">0.71</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.Px2.p3.3.m3.1c">0.71</annotation></semantics></math> points) better perplexity than our <math id="S3.SS2.SSS2.Px2.p3.4.m4.1" class="ltx_Math" alttext="d_{\text{head}}=64" display="inline"><semantics id="S3.SS2.SSS2.Px2.p3.4.m4.1a"><mrow id="S3.SS2.SSS2.Px2.p3.4.m4.1.1" xref="S3.SS2.SSS2.Px2.p3.4.m4.1.1.cmml"><msub id="S3.SS2.SSS2.Px2.p3.4.m4.1.1.2" xref="S3.SS2.SSS2.Px2.p3.4.m4.1.1.2.cmml"><mi id="S3.SS2.SSS2.Px2.p3.4.m4.1.1.2.2" xref="S3.SS2.SSS2.Px2.p3.4.m4.1.1.2.2.cmml">d</mi><mtext id="S3.SS2.SSS2.Px2.p3.4.m4.1.1.2.3" xref="S3.SS2.SSS2.Px2.p3.4.m4.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS2.SSS2.Px2.p3.4.m4.1.1.1" xref="S3.SS2.SSS2.Px2.p3.4.m4.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS2.Px2.p3.4.m4.1.1.3" xref="S3.SS2.SSS2.Px2.p3.4.m4.1.1.3.cmml">64</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.Px2.p3.4.m4.1b"><apply id="S3.SS2.SSS2.Px2.p3.4.m4.1.1.cmml" xref="S3.SS2.SSS2.Px2.p3.4.m4.1.1"><eq id="S3.SS2.SSS2.Px2.p3.4.m4.1.1.1.cmml" xref="S3.SS2.SSS2.Px2.p3.4.m4.1.1.1"></eq><apply id="S3.SS2.SSS2.Px2.p3.4.m4.1.1.2.cmml" xref="S3.SS2.SSS2.Px2.p3.4.m4.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS2.Px2.p3.4.m4.1.1.2.1.cmml" xref="S3.SS2.SSS2.Px2.p3.4.m4.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS2.Px2.p3.4.m4.1.1.2.2.cmml" xref="S3.SS2.SSS2.Px2.p3.4.m4.1.1.2.2">𝑑</ci><ci id="S3.SS2.SSS2.Px2.p3.4.m4.1.1.2.3a.cmml" xref="S3.SS2.SSS2.Px2.p3.4.m4.1.1.2.3"><mtext mathsize="70%" id="S3.SS2.SSS2.Px2.p3.4.m4.1.1.2.3.cmml" xref="S3.SS2.SSS2.Px2.p3.4.m4.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS2.SSS2.Px2.p3.4.m4.1.1.3.cmml" xref="S3.SS2.SSS2.Px2.p3.4.m4.1.1.3">64</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.Px2.p3.4.m4.1c">d_{\text{head}}=64</annotation></semantics></math> baseline. On the downstream evaluations, we found that none of our three models model consistently wins or loses across different benchmarks, and that all three models are consistently within <math id="S3.SS2.SSS2.Px2.p3.5.m5.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S3.SS2.SSS2.Px2.p3.5.m5.1a"><mn id="S3.SS2.SSS2.Px2.p3.5.m5.1.1" xref="S3.SS2.SSS2.Px2.p3.5.m5.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.Px2.p3.5.m5.1b"><cn type="integer" id="S3.SS2.SSS2.Px2.p3.5.m5.1.1.cmml" xref="S3.SS2.SSS2.Px2.p3.5.m5.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.Px2.p3.5.m5.1c">1</annotation></semantics></math>–<math id="S3.SS2.SSS2.Px2.p3.6.m6.1" class="ltx_Math" alttext="5" display="inline"><semantics id="S3.SS2.SSS2.Px2.p3.6.m6.1a"><mn id="S3.SS2.SSS2.Px2.p3.6.m6.1.1" xref="S3.SS2.SSS2.Px2.p3.6.m6.1.1.cmml">5</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.Px2.p3.6.m6.1b"><cn type="integer" id="S3.SS2.SSS2.Px2.p3.6.m6.1.1.cmml" xref="S3.SS2.SSS2.Px2.p3.6.m6.1.1">5</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.Px2.p3.6.m6.1c">5</annotation></semantics></math> percentage points of each other.</p>
</div>
<figure id="S3.T3" class="ltx_table">
<table id="S3.T3.9" class="ltx_tabular ltx_centering ltx_align_middle">
<thead class="ltx_thead">
<tr id="S3.T3.9.10.1" class="ltx_tr">
<th id="S3.T3.9.10.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_t">
<span id="S3.T3.9.10.1.1.1" class="ltx_text"></span> <span id="S3.T3.9.10.1.1.2" class="ltx_text">
<span id="S3.T3.9.10.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T3.9.10.1.1.2.1.1" class="ltx_tr">
<span id="S3.T3.9.10.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T3.9.10.1.1.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Model</span></span></span>
</span></span><span id="S3.T3.9.10.1.1.3" class="ltx_text"></span></th>
<th id="S3.T3.9.10.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_t">
<span id="S3.T3.9.10.1.2.1" class="ltx_text"></span> <span id="S3.T3.9.10.1.2.2" class="ltx_text">
<span id="S3.T3.9.10.1.2.2.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T3.9.10.1.2.2.1.1" class="ltx_tr">
<span id="S3.T3.9.10.1.2.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T3.9.10.1.2.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">KV Bytes Per</span></span></span>
<span id="S3.T3.9.10.1.2.2.1.2" class="ltx_tr">
<span id="S3.T3.9.10.1.2.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T3.9.10.1.2.2.1.2.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Token (16-bit)</span></span></span>
</span></span><span id="S3.T3.9.10.1.2.3" class="ltx_text"></span></th>
<th id="S3.T3.9.10.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_t">
<span id="S3.T3.9.10.1.3.1" class="ltx_text"></span> <span id="S3.T3.9.10.1.3.2" class="ltx_text">
<span id="S3.T3.9.10.1.3.2.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T3.9.10.1.3.2.1.1" class="ltx_tr">
<span id="S3.T3.9.10.1.3.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T3.9.10.1.3.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Best LR</span></span></span>
</span></span><span id="S3.T3.9.10.1.3.3" class="ltx_text"></span></th>
<th id="S3.T3.9.10.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_t">
<span id="S3.T3.9.10.1.4.1" class="ltx_text"></span> <span id="S3.T3.9.10.1.4.2" class="ltx_text">
<span id="S3.T3.9.10.1.4.2.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T3.9.10.1.4.2.1.1" class="ltx_tr">
<span id="S3.T3.9.10.1.4.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T3.9.10.1.4.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Validation</span></span></span>
<span id="S3.T3.9.10.1.4.2.1.2" class="ltx_tr">
<span id="S3.T3.9.10.1.4.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T3.9.10.1.4.2.1.2.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Perplexity</span></span></span>
</span></span><span id="S3.T3.9.10.1.4.3" class="ltx_text"></span></th>
<th id="S3.T3.9.10.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_t">
<span id="S3.T3.9.10.1.5.1" class="ltx_text"></span> <span id="S3.T3.9.10.1.5.2" class="ltx_text">
<span id="S3.T3.9.10.1.5.2.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T3.9.10.1.5.2.1.1" class="ltx_tr">
<span id="S3.T3.9.10.1.5.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T3.9.10.1.5.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Wikitext</span></span></span>
<span id="S3.T3.9.10.1.5.2.1.2" class="ltx_tr">
<span id="S3.T3.9.10.1.5.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T3.9.10.1.5.2.1.2.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Perplexity</span></span></span>
</span></span><span id="S3.T3.9.10.1.5.3" class="ltx_text"></span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S3.T3.3.3" class="ltx_tr">
<td id="S3.T3.3.3.4" class="ltx_td ltx_align_left ltx_border_t">H128-MQA</td>
<td id="S3.T3.3.3.5" class="ltx_td ltx_align_center ltx_border_t">10240</td>
<td id="S3.T3.1.1.1" class="ltx_td ltx_align_center ltx_border_t"><math id="S3.T3.1.1.1.m1.1" class="ltx_Math" alttext="1.5\phantom{0}\times 10^{-3}" display="inline"><semantics id="S3.T3.1.1.1.m1.1a"><mrow id="S3.T3.1.1.1.m1.1.1" xref="S3.T3.1.1.1.m1.1.1.cmml"><mn id="S3.T3.1.1.1.m1.1.1.2" xref="S3.T3.1.1.1.m1.1.1.2.cmml">1.5</mn><mo lspace="0.722em" rspace="0.222em" id="S3.T3.1.1.1.m1.1.1.1" xref="S3.T3.1.1.1.m1.1.1.1.cmml">×</mo><msup id="S3.T3.1.1.1.m1.1.1.3" xref="S3.T3.1.1.1.m1.1.1.3.cmml"><mn id="S3.T3.1.1.1.m1.1.1.3.2" xref="S3.T3.1.1.1.m1.1.1.3.2.cmml">10</mn><mrow id="S3.T3.1.1.1.m1.1.1.3.3" xref="S3.T3.1.1.1.m1.1.1.3.3.cmml"><mo id="S3.T3.1.1.1.m1.1.1.3.3a" xref="S3.T3.1.1.1.m1.1.1.3.3.cmml">−</mo><mn id="S3.T3.1.1.1.m1.1.1.3.3.2" xref="S3.T3.1.1.1.m1.1.1.3.3.2.cmml">3</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.T3.1.1.1.m1.1b"><apply id="S3.T3.1.1.1.m1.1.1.cmml" xref="S3.T3.1.1.1.m1.1.1"><times id="S3.T3.1.1.1.m1.1.1.1.cmml" xref="S3.T3.1.1.1.m1.1.1.1"></times><cn type="float" id="S3.T3.1.1.1.m1.1.1.2.cmml" xref="S3.T3.1.1.1.m1.1.1.2">1.5</cn><apply id="S3.T3.1.1.1.m1.1.1.3.cmml" xref="S3.T3.1.1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.T3.1.1.1.m1.1.1.3.1.cmml" xref="S3.T3.1.1.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="S3.T3.1.1.1.m1.1.1.3.2.cmml" xref="S3.T3.1.1.1.m1.1.1.3.2">10</cn><apply id="S3.T3.1.1.1.m1.1.1.3.3.cmml" xref="S3.T3.1.1.1.m1.1.1.3.3"><minus id="S3.T3.1.1.1.m1.1.1.3.3.1.cmml" xref="S3.T3.1.1.1.m1.1.1.3.3"></minus><cn type="integer" id="S3.T3.1.1.1.m1.1.1.3.3.2.cmml" xref="S3.T3.1.1.1.m1.1.1.3.3.2">3</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.T3.1.1.1.m1.1c">1.5\phantom{0}\times 10^{-3}</annotation></semantics></math></td>
<td id="S3.T3.2.2.2" class="ltx_td ltx_align_center ltx_border_t"><math id="S3.T3.2.2.2.m1.1" class="ltx_Math" alttext="\mathbf{12.39}" display="inline"><semantics id="S3.T3.2.2.2.m1.1a"><mn class="ltx_mathvariant_bold" mathvariant="bold" id="S3.T3.2.2.2.m1.1.1" xref="S3.T3.2.2.2.m1.1.1.cmml">12.39</mn><annotation-xml encoding="MathML-Content" id="S3.T3.2.2.2.m1.1b"><cn type="float" id="S3.T3.2.2.2.m1.1.1.cmml" xref="S3.T3.2.2.2.m1.1.1">12.39</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T3.2.2.2.m1.1c">\mathbf{12.39}</annotation></semantics></math></td>
<td id="S3.T3.3.3.3" class="ltx_td ltx_align_center ltx_border_t"><math id="S3.T3.3.3.3.m1.1" class="ltx_Math" alttext="19.30" display="inline"><semantics id="S3.T3.3.3.3.m1.1a"><mn id="S3.T3.3.3.3.m1.1.1" xref="S3.T3.3.3.3.m1.1.1.cmml">19.30</mn><annotation-xml encoding="MathML-Content" id="S3.T3.3.3.3.m1.1b"><cn type="float" id="S3.T3.3.3.3.m1.1.1.cmml" xref="S3.T3.3.3.3.m1.1.1">19.30</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T3.3.3.3.m1.1c">19.30</annotation></semantics></math></td>
</tr>
<tr id="S3.T3.6.6" class="ltx_tr">
<td id="S3.T3.6.6.4" class="ltx_td ltx_align_left">H128-MQA-CLA2</td>
<td id="S3.T3.6.6.5" class="ltx_td ltx_align_center">
<span id="S3.T3.6.6.5.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>5120</td>
<td id="S3.T3.4.4.1" class="ltx_td ltx_align_center"><math id="S3.T3.4.4.1.m1.1" class="ltx_Math" alttext="2.25\times 10^{-3}" display="inline"><semantics id="S3.T3.4.4.1.m1.1a"><mrow id="S3.T3.4.4.1.m1.1.1" xref="S3.T3.4.4.1.m1.1.1.cmml"><mn id="S3.T3.4.4.1.m1.1.1.2" xref="S3.T3.4.4.1.m1.1.1.2.cmml">2.25</mn><mo lspace="0.222em" rspace="0.222em" id="S3.T3.4.4.1.m1.1.1.1" xref="S3.T3.4.4.1.m1.1.1.1.cmml">×</mo><msup id="S3.T3.4.4.1.m1.1.1.3" xref="S3.T3.4.4.1.m1.1.1.3.cmml"><mn id="S3.T3.4.4.1.m1.1.1.3.2" xref="S3.T3.4.4.1.m1.1.1.3.2.cmml">10</mn><mrow id="S3.T3.4.4.1.m1.1.1.3.3" xref="S3.T3.4.4.1.m1.1.1.3.3.cmml"><mo id="S3.T3.4.4.1.m1.1.1.3.3a" xref="S3.T3.4.4.1.m1.1.1.3.3.cmml">−</mo><mn id="S3.T3.4.4.1.m1.1.1.3.3.2" xref="S3.T3.4.4.1.m1.1.1.3.3.2.cmml">3</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.T3.4.4.1.m1.1b"><apply id="S3.T3.4.4.1.m1.1.1.cmml" xref="S3.T3.4.4.1.m1.1.1"><times id="S3.T3.4.4.1.m1.1.1.1.cmml" xref="S3.T3.4.4.1.m1.1.1.1"></times><cn type="float" id="S3.T3.4.4.1.m1.1.1.2.cmml" xref="S3.T3.4.4.1.m1.1.1.2">2.25</cn><apply id="S3.T3.4.4.1.m1.1.1.3.cmml" xref="S3.T3.4.4.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.T3.4.4.1.m1.1.1.3.1.cmml" xref="S3.T3.4.4.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="S3.T3.4.4.1.m1.1.1.3.2.cmml" xref="S3.T3.4.4.1.m1.1.1.3.2">10</cn><apply id="S3.T3.4.4.1.m1.1.1.3.3.cmml" xref="S3.T3.4.4.1.m1.1.1.3.3"><minus id="S3.T3.4.4.1.m1.1.1.3.3.1.cmml" xref="S3.T3.4.4.1.m1.1.1.3.3"></minus><cn type="integer" id="S3.T3.4.4.1.m1.1.1.3.3.2.cmml" xref="S3.T3.4.4.1.m1.1.1.3.3.2">3</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.T3.4.4.1.m1.1c">2.25\times 10^{-3}</annotation></semantics></math></td>
<td id="S3.T3.5.5.2" class="ltx_td ltx_align_center"><math id="S3.T3.5.5.2.m1.1" class="ltx_Math" alttext="12.43" display="inline"><semantics id="S3.T3.5.5.2.m1.1a"><mn id="S3.T3.5.5.2.m1.1.1" xref="S3.T3.5.5.2.m1.1.1.cmml">12.43</mn><annotation-xml encoding="MathML-Content" id="S3.T3.5.5.2.m1.1b"><cn type="float" id="S3.T3.5.5.2.m1.1.1.cmml" xref="S3.T3.5.5.2.m1.1.1">12.43</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T3.5.5.2.m1.1c">12.43</annotation></semantics></math></td>
<td id="S3.T3.6.6.3" class="ltx_td ltx_align_center"><math id="S3.T3.6.6.3.m1.1" class="ltx_Math" alttext="\mathbf{19.29}" display="inline"><semantics id="S3.T3.6.6.3.m1.1a"><mn class="ltx_mathvariant_bold" mathvariant="bold" id="S3.T3.6.6.3.m1.1.1" xref="S3.T3.6.6.3.m1.1.1.cmml">19.29</mn><annotation-xml encoding="MathML-Content" id="S3.T3.6.6.3.m1.1b"><cn type="float" id="S3.T3.6.6.3.m1.1.1.cmml" xref="S3.T3.6.6.3.m1.1.1">19.29</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T3.6.6.3.m1.1c">\mathbf{19.29}</annotation></semantics></math></td>
</tr>
<tr id="S3.T3.9.9" class="ltx_tr">
<td id="S3.T3.9.9.4" class="ltx_td ltx_align_left ltx_border_b">H64-MQA</td>
<td id="S3.T3.9.9.5" class="ltx_td ltx_align_center ltx_border_b">
<span id="S3.T3.9.9.5.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>5120</td>
<td id="S3.T3.7.7.1" class="ltx_td ltx_align_center ltx_border_b"><math id="S3.T3.7.7.1.m1.1" class="ltx_Math" alttext="2.25\times 10^{-3}" display="inline"><semantics id="S3.T3.7.7.1.m1.1a"><mrow id="S3.T3.7.7.1.m1.1.1" xref="S3.T3.7.7.1.m1.1.1.cmml"><mn id="S3.T3.7.7.1.m1.1.1.2" xref="S3.T3.7.7.1.m1.1.1.2.cmml">2.25</mn><mo lspace="0.222em" rspace="0.222em" id="S3.T3.7.7.1.m1.1.1.1" xref="S3.T3.7.7.1.m1.1.1.1.cmml">×</mo><msup id="S3.T3.7.7.1.m1.1.1.3" xref="S3.T3.7.7.1.m1.1.1.3.cmml"><mn id="S3.T3.7.7.1.m1.1.1.3.2" xref="S3.T3.7.7.1.m1.1.1.3.2.cmml">10</mn><mrow id="S3.T3.7.7.1.m1.1.1.3.3" xref="S3.T3.7.7.1.m1.1.1.3.3.cmml"><mo id="S3.T3.7.7.1.m1.1.1.3.3a" xref="S3.T3.7.7.1.m1.1.1.3.3.cmml">−</mo><mn id="S3.T3.7.7.1.m1.1.1.3.3.2" xref="S3.T3.7.7.1.m1.1.1.3.3.2.cmml">3</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.T3.7.7.1.m1.1b"><apply id="S3.T3.7.7.1.m1.1.1.cmml" xref="S3.T3.7.7.1.m1.1.1"><times id="S3.T3.7.7.1.m1.1.1.1.cmml" xref="S3.T3.7.7.1.m1.1.1.1"></times><cn type="float" id="S3.T3.7.7.1.m1.1.1.2.cmml" xref="S3.T3.7.7.1.m1.1.1.2">2.25</cn><apply id="S3.T3.7.7.1.m1.1.1.3.cmml" xref="S3.T3.7.7.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.T3.7.7.1.m1.1.1.3.1.cmml" xref="S3.T3.7.7.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="S3.T3.7.7.1.m1.1.1.3.2.cmml" xref="S3.T3.7.7.1.m1.1.1.3.2">10</cn><apply id="S3.T3.7.7.1.m1.1.1.3.3.cmml" xref="S3.T3.7.7.1.m1.1.1.3.3"><minus id="S3.T3.7.7.1.m1.1.1.3.3.1.cmml" xref="S3.T3.7.7.1.m1.1.1.3.3"></minus><cn type="integer" id="S3.T3.7.7.1.m1.1.1.3.3.2.cmml" xref="S3.T3.7.7.1.m1.1.1.3.3.2">3</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.T3.7.7.1.m1.1c">2.25\times 10^{-3}</annotation></semantics></math></td>
<td id="S3.T3.8.8.2" class="ltx_td ltx_align_center ltx_border_b"><math id="S3.T3.8.8.2.m1.1" class="ltx_Math" alttext="12.74" display="inline"><semantics id="S3.T3.8.8.2.m1.1a"><mn id="S3.T3.8.8.2.m1.1.1" xref="S3.T3.8.8.2.m1.1.1.cmml">12.74</mn><annotation-xml encoding="MathML-Content" id="S3.T3.8.8.2.m1.1b"><cn type="float" id="S3.T3.8.8.2.m1.1.1.cmml" xref="S3.T3.8.8.2.m1.1.1">12.74</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T3.8.8.2.m1.1c">12.74</annotation></semantics></math></td>
<td id="S3.T3.9.9.3" class="ltx_td ltx_align_center ltx_border_b"><math id="S3.T3.9.9.3.m1.1" class="ltx_Math" alttext="20.00" display="inline"><semantics id="S3.T3.9.9.3.m1.1a"><mn id="S3.T3.9.9.3.m1.1.1" xref="S3.T3.9.9.3.m1.1.1.cmml">20.00</mn><annotation-xml encoding="MathML-Content" id="S3.T3.9.9.3.m1.1b"><cn type="float" id="S3.T3.9.9.3.m1.1.1.cmml" xref="S3.T3.9.9.3.m1.1.1">20.00</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T3.9.9.3.m1.1c">20.00</annotation></semantics></math></td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 3: </span>Perplexity results from learning rate tuning experiments at 1B-parameter scale.</figcaption>
</figure>
<figure id="S3.T4" class="ltx_table">
<table id="S3.T4.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S3.T4.1.1" class="ltx_tr">
<td id="S3.T4.1.1.1" class="ltx_td ltx_align_left ltx_border_t"><span id="S3.T4.1.1.1.1" class="ltx_text ltx_font_bold">Model (Best LR)</span></td>
<td id="S3.T4.1.1.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T4.1.1.2.1" class="ltx_text ltx_font_bold">Hellaswag</span></td>
<td id="S3.T4.1.1.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T4.1.1.3.1" class="ltx_text ltx_font_bold">PIQA</span></td>
<td id="S3.T4.1.1.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T4.1.1.4.1" class="ltx_text ltx_font_bold">WG</span></td>
<td id="S3.T4.1.1.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T4.1.1.5.1" class="ltx_text ltx_font_bold">SciQ</span></td>
<td id="S3.T4.1.1.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T4.1.1.6.1" class="ltx_text ltx_font_bold">OBQA</span></td>
<td id="S3.T4.1.1.7" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T4.1.1.7.1" class="ltx_text ltx_font_bold">BoolQ</span></td>
<td id="S3.T4.1.1.8" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T4.1.1.8.1" class="ltx_text ltx_font_bold">ARC-E</span></td>
</tr>
<tr id="S3.T4.1.2" class="ltx_tr">
<td id="S3.T4.1.2.1" class="ltx_td ltx_align_left ltx_border_t">H128-MQA</td>
<td id="S3.T4.1.2.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T4.1.2.2.1" class="ltx_text ltx_font_bold">36.24</span></td>
<td id="S3.T4.1.2.3" class="ltx_td ltx_align_center ltx_border_t">69.15</td>
<td id="S3.T4.1.2.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T4.1.2.4.1" class="ltx_text ltx_font_bold">52.96</span></td>
<td id="S3.T4.1.2.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T4.1.2.5.1" class="ltx_text ltx_font_bold">82.9</span></td>
<td id="S3.T4.1.2.6" class="ltx_td ltx_align_center ltx_border_t">19.0</td>
<td id="S3.T4.1.2.7" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T4.1.2.7.1" class="ltx_text ltx_font_bold">57.40</span></td>
<td id="S3.T4.1.2.8" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T4.1.2.8.1" class="ltx_text ltx_font_bold">55.43</span></td>
</tr>
<tr id="S3.T4.1.3" class="ltx_tr">
<td id="S3.T4.1.3.1" class="ltx_td ltx_align_left">H128-MQA-CLA2</td>
<td id="S3.T4.1.3.2" class="ltx_td ltx_align_center">36.01</td>
<td id="S3.T4.1.3.3" class="ltx_td ltx_align_center">69.15</td>
<td id="S3.T4.1.3.4" class="ltx_td ltx_align_center">51.93</td>
<td id="S3.T4.1.3.5" class="ltx_td ltx_align_center">82.6</td>
<td id="S3.T4.1.3.6" class="ltx_td ltx_align_center"><span id="S3.T4.1.3.6.1" class="ltx_text ltx_font_bold">21.4</span></td>
<td id="S3.T4.1.3.7" class="ltx_td ltx_align_center">53.21</td>
<td id="S3.T4.1.3.8" class="ltx_td ltx_align_center">53.87</td>
</tr>
<tr id="S3.T4.1.4" class="ltx_tr">
<td id="S3.T4.1.4.1" class="ltx_td ltx_align_left ltx_border_b">H64-MQA</td>
<td id="S3.T4.1.4.2" class="ltx_td ltx_align_center ltx_border_b">35.22</td>
<td id="S3.T4.1.4.3" class="ltx_td ltx_align_center ltx_border_b"><span id="S3.T4.1.4.3.1" class="ltx_text ltx_font_bold">69.21</span></td>
<td id="S3.T4.1.4.4" class="ltx_td ltx_align_center ltx_border_b">50.75</td>
<td id="S3.T4.1.4.5" class="ltx_td ltx_align_center ltx_border_b">78.5</td>
<td id="S3.T4.1.4.6" class="ltx_td ltx_align_center ltx_border_b">19.4</td>
<td id="S3.T4.1.4.7" class="ltx_td ltx_align_center ltx_border_b">55.81</td>
<td id="S3.T4.1.4.8" class="ltx_td ltx_align_center ltx_border_b">51.68</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 4: </span>Downstream benchmark results for 1B-scale models with tuned learning rates. The columns “WG” and “OBQA” denote “WinoGrande” and “OpenBookQA”, respectively.</figcaption>
</figure>
</section>
</section>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Experiments at 3B-Parameter Scale</h3>

<div id="S3.SS3.p1" class="ltx_para">
<p id="S3.SS3.p1.1" class="ltx_p">To determine how CLA performs when applied to larger models, we trained a collection of models at the 3B-parameter scale both with and without CLA. We trained each of our 3B-scale model from scratch on 100B tokens from our SlimPajama corpus. The common architectural hyperparameters for our 3B-scale models can be found in Table <a href="#S3.T2" title="Table 2 ‣ 3.2 Experiments at 1B-Parameter Scale ‣ 3 Pretraining Experiments ‣ Reducing Transformer Key-Value Cache Size with Cross-Layer Attention" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.</p>
</div>
<section id="S3.SS3.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Experiments at Head Dimension <math id="S3.SS3.SSS0.Px1.1.m1.1" class="ltx_Math" alttext="\boldsymbol{d}_{\textbf{head}}=\mathbf{128}" display="inline"><semantics id="S3.SS3.SSS0.Px1.1.m1.1b"><mrow id="S3.SS3.SSS0.Px1.1.m1.1.1" xref="S3.SS3.SSS0.Px1.1.m1.1.1.cmml"><msub id="S3.SS3.SSS0.Px1.1.m1.1.1.2" xref="S3.SS3.SSS0.Px1.1.m1.1.1.2.cmml"><mi id="S3.SS3.SSS0.Px1.1.m1.1.1.2.2" xref="S3.SS3.SSS0.Px1.1.m1.1.1.2.2.cmml">𝒅</mi><mtext class="ltx_mathvariant_bold" id="S3.SS3.SSS0.Px1.1.m1.1.1.2.3" xref="S3.SS3.SSS0.Px1.1.m1.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS3.SSS0.Px1.1.m1.1.1.1" xref="S3.SS3.SSS0.Px1.1.m1.1.1.1.cmml">=</mo><mn id="S3.SS3.SSS0.Px1.1.m1.1.1.3" xref="S3.SS3.SSS0.Px1.1.m1.1.1.3.cmml">𝟏𝟐𝟖</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.1.m1.1c"><apply id="S3.SS3.SSS0.Px1.1.m1.1.1.cmml" xref="S3.SS3.SSS0.Px1.1.m1.1.1"><eq id="S3.SS3.SSS0.Px1.1.m1.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.1.m1.1.1.1"></eq><apply id="S3.SS3.SSS0.Px1.1.m1.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.1.m1.1.1.2.1.cmml" xref="S3.SS3.SSS0.Px1.1.m1.1.1.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px1.1.m1.1.1.2.2.cmml" xref="S3.SS3.SSS0.Px1.1.m1.1.1.2.2">𝒅</ci><ci id="S3.SS3.SSS0.Px1.1.m1.1.1.2.3a.cmml" xref="S3.SS3.SSS0.Px1.1.m1.1.1.2.3"><mtext class="ltx_mathvariant_bold" mathsize="70%" id="S3.SS3.SSS0.Px1.1.m1.1.1.2.3.cmml" xref="S3.SS3.SSS0.Px1.1.m1.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS3.SSS0.Px1.1.m1.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.1.m1.1.1.3">128</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.1.m1.1d">\boldsymbol{d}_{\textbf{head}}=\mathbf{128}</annotation></semantics></math>.</h5>

<div id="S3.SS3.SSS0.Px1.p1" class="ltx_para">
<p id="S3.SS3.SSS0.Px1.p1.5" class="ltx_p">We initially ran experiments to compare three 3B-scale models analogous to the models we selected for our learning rate tuning experiments at the 1B-parameter scale. Specifically, we compared a model using MQA-CLA2 and <math id="S3.SS3.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="d_{\text{head}}=128" display="inline"><semantics id="S3.SS3.SSS0.Px1.p1.1.m1.1a"><mrow id="S3.SS3.SSS0.Px1.p1.1.m1.1.1" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.cmml"><msub id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.cmml"><mi id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.2" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.2.cmml">d</mi><mtext id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.3" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.1" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.1.cmml">=</mo><mn id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.cmml">128</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p1.1.m1.1b"><apply id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1"><eq id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.1"></eq><apply id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.1.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.2.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.2">𝑑</ci><ci id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.3a.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.3"><mtext mathsize="70%" id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.3.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3">128</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p1.1.m1.1c">d_{\text{head}}=128</annotation></semantics></math> to an MQA model with the same head dimension (and hence <math id="S3.SS3.SSS0.Px1.p1.2.m2.1" class="ltx_math_unparsed" alttext="2\times" display="inline"><semantics id="S3.SS3.SSS0.Px1.p1.2.m2.1a"><mrow id="S3.SS3.SSS0.Px1.p1.2.m2.1b"><mn id="S3.SS3.SSS0.Px1.p1.2.m2.1.1">2</mn><mo lspace="0.222em" id="S3.SS3.SSS0.Px1.p1.2.m2.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p1.2.m2.1c">2\times</annotation></semantics></math> the KV cache footprint), and to an MQA model with a head dimension of <math id="S3.SS3.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="d_{\text{head}}=64" display="inline"><semantics id="S3.SS3.SSS0.Px1.p1.3.m3.1a"><mrow id="S3.SS3.SSS0.Px1.p1.3.m3.1.1" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1.cmml"><msub id="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2.cmml"><mi id="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2.2" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2.2.cmml">d</mi><mtext id="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2.3" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS3.SSS0.Px1.p1.3.m3.1.1.1" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1.1.cmml">=</mo><mn id="S3.SS3.SSS0.Px1.p1.3.m3.1.1.3" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1.3.cmml">64</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p1.3.m3.1b"><apply id="S3.SS3.SSS0.Px1.p1.3.m3.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1"><eq id="S3.SS3.SSS0.Px1.p1.3.m3.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1.1"></eq><apply id="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2.1.cmml" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2.2.cmml" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2.2">𝑑</ci><ci id="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2.3a.cmml" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2.3"><mtext mathsize="70%" id="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2.3.cmml" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS3.SSS0.Px1.p1.3.m3.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1.3">64</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p1.3.m3.1c">d_{\text{head}}=64</annotation></semantics></math> (and hence the same KV cache footprint). Based on our 1B-scale experiments, we expected that our MQA-CLA2 and MQA models with <math id="S3.SS3.SSS0.Px1.p1.4.m4.1" class="ltx_Math" alttext="d_{\text{head}}=128" display="inline"><semantics id="S3.SS3.SSS0.Px1.p1.4.m4.1a"><mrow id="S3.SS3.SSS0.Px1.p1.4.m4.1.1" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.cmml"><msub id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.2" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.2.cmml"><mi id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.2.2" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.2.2.cmml">d</mi><mtext id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.2.3" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.cmml">=</mo><mn id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.3" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.3.cmml">128</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p1.4.m4.1b"><apply id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1"><eq id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1"></eq><apply id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.2.1.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.2.2.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.2.2">𝑑</ci><ci id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.2.3a.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.2.3"><mtext mathsize="70%" id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.2.3.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.3">128</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p1.4.m4.1c">d_{\text{head}}=128</annotation></semantics></math> would achieve similar perplexities to each other, and that both would outperform the <math id="S3.SS3.SSS0.Px1.p1.5.m5.1" class="ltx_Math" alttext="d_{\text{head}}=64" display="inline"><semantics id="S3.SS3.SSS0.Px1.p1.5.m5.1a"><mrow id="S3.SS3.SSS0.Px1.p1.5.m5.1.1" xref="S3.SS3.SSS0.Px1.p1.5.m5.1.1.cmml"><msub id="S3.SS3.SSS0.Px1.p1.5.m5.1.1.2" xref="S3.SS3.SSS0.Px1.p1.5.m5.1.1.2.cmml"><mi id="S3.SS3.SSS0.Px1.p1.5.m5.1.1.2.2" xref="S3.SS3.SSS0.Px1.p1.5.m5.1.1.2.2.cmml">d</mi><mtext id="S3.SS3.SSS0.Px1.p1.5.m5.1.1.2.3" xref="S3.SS3.SSS0.Px1.p1.5.m5.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS3.SSS0.Px1.p1.5.m5.1.1.1" xref="S3.SS3.SSS0.Px1.p1.5.m5.1.1.1.cmml">=</mo><mn id="S3.SS3.SSS0.Px1.p1.5.m5.1.1.3" xref="S3.SS3.SSS0.Px1.p1.5.m5.1.1.3.cmml">64</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p1.5.m5.1b"><apply id="S3.SS3.SSS0.Px1.p1.5.m5.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.5.m5.1.1"><eq id="S3.SS3.SSS0.Px1.p1.5.m5.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.5.m5.1.1.1"></eq><apply id="S3.SS3.SSS0.Px1.p1.5.m5.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p1.5.m5.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p1.5.m5.1.1.2.1.cmml" xref="S3.SS3.SSS0.Px1.p1.5.m5.1.1.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px1.p1.5.m5.1.1.2.2.cmml" xref="S3.SS3.SSS0.Px1.p1.5.m5.1.1.2.2">𝑑</ci><ci id="S3.SS3.SSS0.Px1.p1.5.m5.1.1.2.3a.cmml" xref="S3.SS3.SSS0.Px1.p1.5.m5.1.1.2.3"><mtext mathsize="70%" id="S3.SS3.SSS0.Px1.p1.5.m5.1.1.2.3.cmml" xref="S3.SS3.SSS0.Px1.p1.5.m5.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS3.SSS0.Px1.p1.5.m5.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p1.5.m5.1.1.3">64</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p1.5.m5.1c">d_{\text{head}}=64</annotation></semantics></math> model.</p>
</div>
<div id="S3.SS3.SSS0.Px1.p2" class="ltx_para">
<p id="S3.SS3.SSS0.Px1.p2.5" class="ltx_p">We tuned the learning rates for these models according to the same learning rate tuning protocol we used at the 1B-parameter scale. After tuning the learning rates for each model, we observed a result different than we had expected: at 3B scale, our MQA-CLA2 model achieves substantially better perplexities than both our <math id="S3.SS3.SSS0.Px1.p2.1.m1.1" class="ltx_Math" alttext="d_{\text{head}}=128" display="inline"><semantics id="S3.SS3.SSS0.Px1.p2.1.m1.1a"><mrow id="S3.SS3.SSS0.Px1.p2.1.m1.1.1" xref="S3.SS3.SSS0.Px1.p2.1.m1.1.1.cmml"><msub id="S3.SS3.SSS0.Px1.p2.1.m1.1.1.2" xref="S3.SS3.SSS0.Px1.p2.1.m1.1.1.2.cmml"><mi id="S3.SS3.SSS0.Px1.p2.1.m1.1.1.2.2" xref="S3.SS3.SSS0.Px1.p2.1.m1.1.1.2.2.cmml">d</mi><mtext id="S3.SS3.SSS0.Px1.p2.1.m1.1.1.2.3" xref="S3.SS3.SSS0.Px1.p2.1.m1.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS3.SSS0.Px1.p2.1.m1.1.1.1" xref="S3.SS3.SSS0.Px1.p2.1.m1.1.1.1.cmml">=</mo><mn id="S3.SS3.SSS0.Px1.p2.1.m1.1.1.3" xref="S3.SS3.SSS0.Px1.p2.1.m1.1.1.3.cmml">128</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p2.1.m1.1b"><apply id="S3.SS3.SSS0.Px1.p2.1.m1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p2.1.m1.1.1"><eq id="S3.SS3.SSS0.Px1.p2.1.m1.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p2.1.m1.1.1.1"></eq><apply id="S3.SS3.SSS0.Px1.p2.1.m1.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p2.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p2.1.m1.1.1.2.1.cmml" xref="S3.SS3.SSS0.Px1.p2.1.m1.1.1.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px1.p2.1.m1.1.1.2.2.cmml" xref="S3.SS3.SSS0.Px1.p2.1.m1.1.1.2.2">𝑑</ci><ci id="S3.SS3.SSS0.Px1.p2.1.m1.1.1.2.3a.cmml" xref="S3.SS3.SSS0.Px1.p2.1.m1.1.1.2.3"><mtext mathsize="70%" id="S3.SS3.SSS0.Px1.p2.1.m1.1.1.2.3.cmml" xref="S3.SS3.SSS0.Px1.p2.1.m1.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS3.SSS0.Px1.p2.1.m1.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p2.1.m1.1.1.3">128</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p2.1.m1.1c">d_{\text{head}}=128</annotation></semantics></math> and <math id="S3.SS3.SSS0.Px1.p2.2.m2.1" class="ltx_Math" alttext="d_{\text{head}}=64" display="inline"><semantics id="S3.SS3.SSS0.Px1.p2.2.m2.1a"><mrow id="S3.SS3.SSS0.Px1.p2.2.m2.1.1" xref="S3.SS3.SSS0.Px1.p2.2.m2.1.1.cmml"><msub id="S3.SS3.SSS0.Px1.p2.2.m2.1.1.2" xref="S3.SS3.SSS0.Px1.p2.2.m2.1.1.2.cmml"><mi id="S3.SS3.SSS0.Px1.p2.2.m2.1.1.2.2" xref="S3.SS3.SSS0.Px1.p2.2.m2.1.1.2.2.cmml">d</mi><mtext id="S3.SS3.SSS0.Px1.p2.2.m2.1.1.2.3" xref="S3.SS3.SSS0.Px1.p2.2.m2.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS3.SSS0.Px1.p2.2.m2.1.1.1" xref="S3.SS3.SSS0.Px1.p2.2.m2.1.1.1.cmml">=</mo><mn id="S3.SS3.SSS0.Px1.p2.2.m2.1.1.3" xref="S3.SS3.SSS0.Px1.p2.2.m2.1.1.3.cmml">64</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p2.2.m2.1b"><apply id="S3.SS3.SSS0.Px1.p2.2.m2.1.1.cmml" xref="S3.SS3.SSS0.Px1.p2.2.m2.1.1"><eq id="S3.SS3.SSS0.Px1.p2.2.m2.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p2.2.m2.1.1.1"></eq><apply id="S3.SS3.SSS0.Px1.p2.2.m2.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p2.2.m2.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p2.2.m2.1.1.2.1.cmml" xref="S3.SS3.SSS0.Px1.p2.2.m2.1.1.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px1.p2.2.m2.1.1.2.2.cmml" xref="S3.SS3.SSS0.Px1.p2.2.m2.1.1.2.2">𝑑</ci><ci id="S3.SS3.SSS0.Px1.p2.2.m2.1.1.2.3a.cmml" xref="S3.SS3.SSS0.Px1.p2.2.m2.1.1.2.3"><mtext mathsize="70%" id="S3.SS3.SSS0.Px1.p2.2.m2.1.1.2.3.cmml" xref="S3.SS3.SSS0.Px1.p2.2.m2.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS3.SSS0.Px1.p2.2.m2.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p2.2.m2.1.1.3">64</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p2.2.m2.1c">d_{\text{head}}=64</annotation></semantics></math> MQA baselines. Moreover, our <math id="S3.SS3.SSS0.Px1.p2.3.m3.1" class="ltx_Math" alttext="d_{\text{head}}=64" display="inline"><semantics id="S3.SS3.SSS0.Px1.p2.3.m3.1a"><mrow id="S3.SS3.SSS0.Px1.p2.3.m3.1.1" xref="S3.SS3.SSS0.Px1.p2.3.m3.1.1.cmml"><msub id="S3.SS3.SSS0.Px1.p2.3.m3.1.1.2" xref="S3.SS3.SSS0.Px1.p2.3.m3.1.1.2.cmml"><mi id="S3.SS3.SSS0.Px1.p2.3.m3.1.1.2.2" xref="S3.SS3.SSS0.Px1.p2.3.m3.1.1.2.2.cmml">d</mi><mtext id="S3.SS3.SSS0.Px1.p2.3.m3.1.1.2.3" xref="S3.SS3.SSS0.Px1.p2.3.m3.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS3.SSS0.Px1.p2.3.m3.1.1.1" xref="S3.SS3.SSS0.Px1.p2.3.m3.1.1.1.cmml">=</mo><mn id="S3.SS3.SSS0.Px1.p2.3.m3.1.1.3" xref="S3.SS3.SSS0.Px1.p2.3.m3.1.1.3.cmml">64</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p2.3.m3.1b"><apply id="S3.SS3.SSS0.Px1.p2.3.m3.1.1.cmml" xref="S3.SS3.SSS0.Px1.p2.3.m3.1.1"><eq id="S3.SS3.SSS0.Px1.p2.3.m3.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p2.3.m3.1.1.1"></eq><apply id="S3.SS3.SSS0.Px1.p2.3.m3.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p2.3.m3.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p2.3.m3.1.1.2.1.cmml" xref="S3.SS3.SSS0.Px1.p2.3.m3.1.1.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px1.p2.3.m3.1.1.2.2.cmml" xref="S3.SS3.SSS0.Px1.p2.3.m3.1.1.2.2">𝑑</ci><ci id="S3.SS3.SSS0.Px1.p2.3.m3.1.1.2.3a.cmml" xref="S3.SS3.SSS0.Px1.p2.3.m3.1.1.2.3"><mtext mathsize="70%" id="S3.SS3.SSS0.Px1.p2.3.m3.1.1.2.3.cmml" xref="S3.SS3.SSS0.Px1.p2.3.m3.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS3.SSS0.Px1.p2.3.m3.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p2.3.m3.1.1.3">64</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p2.3.m3.1c">d_{\text{head}}=64</annotation></semantics></math> MQA baseline model achieves better perplexities than our tuned <math id="S3.SS3.SSS0.Px1.p2.4.m4.1" class="ltx_Math" alttext="d_{\text{head}}=128" display="inline"><semantics id="S3.SS3.SSS0.Px1.p2.4.m4.1a"><mrow id="S3.SS3.SSS0.Px1.p2.4.m4.1.1" xref="S3.SS3.SSS0.Px1.p2.4.m4.1.1.cmml"><msub id="S3.SS3.SSS0.Px1.p2.4.m4.1.1.2" xref="S3.SS3.SSS0.Px1.p2.4.m4.1.1.2.cmml"><mi id="S3.SS3.SSS0.Px1.p2.4.m4.1.1.2.2" xref="S3.SS3.SSS0.Px1.p2.4.m4.1.1.2.2.cmml">d</mi><mtext id="S3.SS3.SSS0.Px1.p2.4.m4.1.1.2.3" xref="S3.SS3.SSS0.Px1.p2.4.m4.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS3.SSS0.Px1.p2.4.m4.1.1.1" xref="S3.SS3.SSS0.Px1.p2.4.m4.1.1.1.cmml">=</mo><mn id="S3.SS3.SSS0.Px1.p2.4.m4.1.1.3" xref="S3.SS3.SSS0.Px1.p2.4.m4.1.1.3.cmml">128</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p2.4.m4.1b"><apply id="S3.SS3.SSS0.Px1.p2.4.m4.1.1.cmml" xref="S3.SS3.SSS0.Px1.p2.4.m4.1.1"><eq id="S3.SS3.SSS0.Px1.p2.4.m4.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p2.4.m4.1.1.1"></eq><apply id="S3.SS3.SSS0.Px1.p2.4.m4.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p2.4.m4.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p2.4.m4.1.1.2.1.cmml" xref="S3.SS3.SSS0.Px1.p2.4.m4.1.1.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px1.p2.4.m4.1.1.2.2.cmml" xref="S3.SS3.SSS0.Px1.p2.4.m4.1.1.2.2">𝑑</ci><ci id="S3.SS3.SSS0.Px1.p2.4.m4.1.1.2.3a.cmml" xref="S3.SS3.SSS0.Px1.p2.4.m4.1.1.2.3"><mtext mathsize="70%" id="S3.SS3.SSS0.Px1.p2.4.m4.1.1.2.3.cmml" xref="S3.SS3.SSS0.Px1.p2.4.m4.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS3.SSS0.Px1.p2.4.m4.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p2.4.m4.1.1.3">128</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p2.4.m4.1c">d_{\text{head}}=128</annotation></semantics></math> MQA baseline, despite having only <math id="S3.SS3.SSS0.Px1.p2.5.m5.1" class="ltx_Math" alttext="\nicefrac{{1}}{{2}}" display="inline"><semantics id="S3.SS3.SSS0.Px1.p2.5.m5.1a"><mrow id="S3.SS3.SSS0.Px1.p2.5.m5.1.1" xref="S3.SS3.SSS0.Px1.p2.5.m5.1.1.cmml"><mpadded voffset="0.3em" id="S3.SS3.SSS0.Px1.p2.5.m5.1.1.2" xref="S3.SS3.SSS0.Px1.p2.5.m5.1.1.2.cmml"><mn mathsize="70%" id="S3.SS3.SSS0.Px1.p2.5.m5.1.1.2a" xref="S3.SS3.SSS0.Px1.p2.5.m5.1.1.2.cmml">1</mn></mpadded><mpadded lspace="-0.1em" width="-0.15em" id="S3.SS3.SSS0.Px1.p2.5.m5.1.1.1" xref="S3.SS3.SSS0.Px1.p2.5.m5.1.1.1.cmml"><mo stretchy="true" symmetric="true" id="S3.SS3.SSS0.Px1.p2.5.m5.1.1.1a" xref="S3.SS3.SSS0.Px1.p2.5.m5.1.1.1.cmml">/</mo></mpadded><mn mathsize="70%" id="S3.SS3.SSS0.Px1.p2.5.m5.1.1.3" xref="S3.SS3.SSS0.Px1.p2.5.m5.1.1.3.cmml">2</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p2.5.m5.1b"><apply id="S3.SS3.SSS0.Px1.p2.5.m5.1.1.cmml" xref="S3.SS3.SSS0.Px1.p2.5.m5.1.1"><divide id="S3.SS3.SSS0.Px1.p2.5.m5.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p2.5.m5.1.1.1"></divide><cn type="integer" id="S3.SS3.SSS0.Px1.p2.5.m5.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p2.5.m5.1.1.2">1</cn><cn type="integer" id="S3.SS3.SSS0.Px1.p2.5.m5.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p2.5.m5.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p2.5.m5.1c">\nicefrac{{1}}{{2}}</annotation></semantics></math> as much KV cache capacity. We report the optimal learning rates and perplexities for these three models in Table <a href="#S3.T5" title='Table 5 ‣ Experiments at Head Dimension 𝒅_"head"=𝟏𝟐𝟖. ‣ 3.3 Experiments at 3B-Parameter Scale ‣ 3 Pretraining Experiments ‣ Reducing Transformer Key-Value Cache Size with Cross-Layer Attention' class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>.</p>
</div>
<div id="S3.SS3.SSS0.Px1.p3" class="ltx_para">
<p id="S3.SS3.SSS0.Px1.p3.1" class="ltx_p">As with our 1B-scale learning rate tuning experiments, we evaluate these models on downstream benchmarks. We report the results of these evaluations in Table <a href="#S3.T6" title='Table 6 ‣ Experiments at Head Dimension 𝒅_"head"=𝟏𝟐𝟖. ‣ 3.3 Experiments at 3B-Parameter Scale ‣ 3 Pretraining Experiments ‣ Reducing Transformer Key-Value Cache Size with Cross-Layer Attention' class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>. As with our 1B-scale experiments, we do not find that any model consistently wins or loses in these downstream evaluations.</p>
</div>
<figure id="S3.T5" class="ltx_table">
<table id="S3.T5.9" class="ltx_tabular ltx_centering ltx_align_middle">
<thead class="ltx_thead">
<tr id="S3.T5.9.10.1" class="ltx_tr">
<th id="S3.T5.9.10.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_t">
<span id="S3.T5.9.10.1.1.1" class="ltx_text"></span> <span id="S3.T5.9.10.1.1.2" class="ltx_text">
<span id="S3.T5.9.10.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T5.9.10.1.1.2.1.1" class="ltx_tr">
<span id="S3.T5.9.10.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T5.9.10.1.1.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Model</span></span></span>
</span></span><span id="S3.T5.9.10.1.1.3" class="ltx_text"></span></th>
<th id="S3.T5.9.10.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_t">
<span id="S3.T5.9.10.1.2.1" class="ltx_text"></span> <span id="S3.T5.9.10.1.2.2" class="ltx_text">
<span id="S3.T5.9.10.1.2.2.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T5.9.10.1.2.2.1.1" class="ltx_tr">
<span id="S3.T5.9.10.1.2.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T5.9.10.1.2.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">KV Bytes Per</span></span></span>
<span id="S3.T5.9.10.1.2.2.1.2" class="ltx_tr">
<span id="S3.T5.9.10.1.2.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T5.9.10.1.2.2.1.2.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Token (16-bit)</span></span></span>
</span></span><span id="S3.T5.9.10.1.2.3" class="ltx_text"></span></th>
<th id="S3.T5.9.10.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_t">
<span id="S3.T5.9.10.1.3.1" class="ltx_text"></span> <span id="S3.T5.9.10.1.3.2" class="ltx_text">
<span id="S3.T5.9.10.1.3.2.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T5.9.10.1.3.2.1.1" class="ltx_tr">
<span id="S3.T5.9.10.1.3.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T5.9.10.1.3.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Best LR</span></span></span>
</span></span><span id="S3.T5.9.10.1.3.3" class="ltx_text"></span></th>
<th id="S3.T5.9.10.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_t">
<span id="S3.T5.9.10.1.4.1" class="ltx_text"></span> <span id="S3.T5.9.10.1.4.2" class="ltx_text">
<span id="S3.T5.9.10.1.4.2.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T5.9.10.1.4.2.1.1" class="ltx_tr">
<span id="S3.T5.9.10.1.4.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T5.9.10.1.4.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Validation</span></span></span>
<span id="S3.T5.9.10.1.4.2.1.2" class="ltx_tr">
<span id="S3.T5.9.10.1.4.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T5.9.10.1.4.2.1.2.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Perplexity</span></span></span>
</span></span><span id="S3.T5.9.10.1.4.3" class="ltx_text"></span></th>
<th id="S3.T5.9.10.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_t">
<span id="S3.T5.9.10.1.5.1" class="ltx_text"></span> <span id="S3.T5.9.10.1.5.2" class="ltx_text">
<span id="S3.T5.9.10.1.5.2.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T5.9.10.1.5.2.1.1" class="ltx_tr">
<span id="S3.T5.9.10.1.5.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T5.9.10.1.5.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Wikitext</span></span></span>
<span id="S3.T5.9.10.1.5.2.1.2" class="ltx_tr">
<span id="S3.T5.9.10.1.5.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T5.9.10.1.5.2.1.2.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Perplexity</span></span></span>
</span></span><span id="S3.T5.9.10.1.5.3" class="ltx_text"></span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S3.T5.3.3" class="ltx_tr">
<td id="S3.T5.3.3.4" class="ltx_td ltx_align_left ltx_border_t">H128-MQA</td>
<td id="S3.T5.3.3.5" class="ltx_td ltx_align_center ltx_border_t">16384</td>
<td id="S3.T5.1.1.1" class="ltx_td ltx_align_center ltx_border_t"><math id="S3.T5.1.1.1.m1.1" class="ltx_Math" alttext="6.75\times 10^{-4}" display="inline"><semantics id="S3.T5.1.1.1.m1.1a"><mrow id="S3.T5.1.1.1.m1.1.1" xref="S3.T5.1.1.1.m1.1.1.cmml"><mn id="S3.T5.1.1.1.m1.1.1.2" xref="S3.T5.1.1.1.m1.1.1.2.cmml">6.75</mn><mo lspace="0.222em" rspace="0.222em" id="S3.T5.1.1.1.m1.1.1.1" xref="S3.T5.1.1.1.m1.1.1.1.cmml">×</mo><msup id="S3.T5.1.1.1.m1.1.1.3" xref="S3.T5.1.1.1.m1.1.1.3.cmml"><mn id="S3.T5.1.1.1.m1.1.1.3.2" xref="S3.T5.1.1.1.m1.1.1.3.2.cmml">10</mn><mrow id="S3.T5.1.1.1.m1.1.1.3.3" xref="S3.T5.1.1.1.m1.1.1.3.3.cmml"><mo id="S3.T5.1.1.1.m1.1.1.3.3a" xref="S3.T5.1.1.1.m1.1.1.3.3.cmml">−</mo><mn id="S3.T5.1.1.1.m1.1.1.3.3.2" xref="S3.T5.1.1.1.m1.1.1.3.3.2.cmml">4</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.T5.1.1.1.m1.1b"><apply id="S3.T5.1.1.1.m1.1.1.cmml" xref="S3.T5.1.1.1.m1.1.1"><times id="S3.T5.1.1.1.m1.1.1.1.cmml" xref="S3.T5.1.1.1.m1.1.1.1"></times><cn type="float" id="S3.T5.1.1.1.m1.1.1.2.cmml" xref="S3.T5.1.1.1.m1.1.1.2">6.75</cn><apply id="S3.T5.1.1.1.m1.1.1.3.cmml" xref="S3.T5.1.1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.T5.1.1.1.m1.1.1.3.1.cmml" xref="S3.T5.1.1.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="S3.T5.1.1.1.m1.1.1.3.2.cmml" xref="S3.T5.1.1.1.m1.1.1.3.2">10</cn><apply id="S3.T5.1.1.1.m1.1.1.3.3.cmml" xref="S3.T5.1.1.1.m1.1.1.3.3"><minus id="S3.T5.1.1.1.m1.1.1.3.3.1.cmml" xref="S3.T5.1.1.1.m1.1.1.3.3"></minus><cn type="integer" id="S3.T5.1.1.1.m1.1.1.3.3.2.cmml" xref="S3.T5.1.1.1.m1.1.1.3.3.2">4</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.T5.1.1.1.m1.1c">6.75\times 10^{-4}</annotation></semantics></math></td>
<td id="S3.T5.2.2.2" class="ltx_td ltx_align_center ltx_border_t"><math id="S3.T5.2.2.2.m1.1" class="ltx_Math" alttext="9.52" display="inline"><semantics id="S3.T5.2.2.2.m1.1a"><mn id="S3.T5.2.2.2.m1.1.1" xref="S3.T5.2.2.2.m1.1.1.cmml">9.52</mn><annotation-xml encoding="MathML-Content" id="S3.T5.2.2.2.m1.1b"><cn type="float" id="S3.T5.2.2.2.m1.1.1.cmml" xref="S3.T5.2.2.2.m1.1.1">9.52</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T5.2.2.2.m1.1c">9.52</annotation></semantics></math></td>
<td id="S3.T5.3.3.3" class="ltx_td ltx_align_center ltx_border_t"><math id="S3.T5.3.3.3.m1.1" class="ltx_Math" alttext="13.63" display="inline"><semantics id="S3.T5.3.3.3.m1.1a"><mn id="S3.T5.3.3.3.m1.1.1" xref="S3.T5.3.3.3.m1.1.1.cmml">13.63</mn><annotation-xml encoding="MathML-Content" id="S3.T5.3.3.3.m1.1b"><cn type="float" id="S3.T5.3.3.3.m1.1.1.cmml" xref="S3.T5.3.3.3.m1.1.1">13.63</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T5.3.3.3.m1.1c">13.63</annotation></semantics></math></td>
</tr>
<tr id="S3.T5.6.6" class="ltx_tr">
<td id="S3.T5.6.6.4" class="ltx_td ltx_align_left">H128-MQA-CLA2</td>
<td id="S3.T5.6.6.5" class="ltx_td ltx_align_center">
<span id="S3.T5.6.6.5.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>8192</td>
<td id="S3.T5.4.4.1" class="ltx_td ltx_align_center"><math id="S3.T5.4.4.1.m1.1" class="ltx_Math" alttext="2.25\times 10^{-3}" display="inline"><semantics id="S3.T5.4.4.1.m1.1a"><mrow id="S3.T5.4.4.1.m1.1.1" xref="S3.T5.4.4.1.m1.1.1.cmml"><mn id="S3.T5.4.4.1.m1.1.1.2" xref="S3.T5.4.4.1.m1.1.1.2.cmml">2.25</mn><mo lspace="0.222em" rspace="0.222em" id="S3.T5.4.4.1.m1.1.1.1" xref="S3.T5.4.4.1.m1.1.1.1.cmml">×</mo><msup id="S3.T5.4.4.1.m1.1.1.3" xref="S3.T5.4.4.1.m1.1.1.3.cmml"><mn id="S3.T5.4.4.1.m1.1.1.3.2" xref="S3.T5.4.4.1.m1.1.1.3.2.cmml">10</mn><mrow id="S3.T5.4.4.1.m1.1.1.3.3" xref="S3.T5.4.4.1.m1.1.1.3.3.cmml"><mo id="S3.T5.4.4.1.m1.1.1.3.3a" xref="S3.T5.4.4.1.m1.1.1.3.3.cmml">−</mo><mn id="S3.T5.4.4.1.m1.1.1.3.3.2" xref="S3.T5.4.4.1.m1.1.1.3.3.2.cmml">3</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.T5.4.4.1.m1.1b"><apply id="S3.T5.4.4.1.m1.1.1.cmml" xref="S3.T5.4.4.1.m1.1.1"><times id="S3.T5.4.4.1.m1.1.1.1.cmml" xref="S3.T5.4.4.1.m1.1.1.1"></times><cn type="float" id="S3.T5.4.4.1.m1.1.1.2.cmml" xref="S3.T5.4.4.1.m1.1.1.2">2.25</cn><apply id="S3.T5.4.4.1.m1.1.1.3.cmml" xref="S3.T5.4.4.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.T5.4.4.1.m1.1.1.3.1.cmml" xref="S3.T5.4.4.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="S3.T5.4.4.1.m1.1.1.3.2.cmml" xref="S3.T5.4.4.1.m1.1.1.3.2">10</cn><apply id="S3.T5.4.4.1.m1.1.1.3.3.cmml" xref="S3.T5.4.4.1.m1.1.1.3.3"><minus id="S3.T5.4.4.1.m1.1.1.3.3.1.cmml" xref="S3.T5.4.4.1.m1.1.1.3.3"></minus><cn type="integer" id="S3.T5.4.4.1.m1.1.1.3.3.2.cmml" xref="S3.T5.4.4.1.m1.1.1.3.3.2">3</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.T5.4.4.1.m1.1c">2.25\times 10^{-3}</annotation></semantics></math></td>
<td id="S3.T5.5.5.2" class="ltx_td ltx_align_center"><math id="S3.T5.5.5.2.m1.1" class="ltx_Math" alttext="\mathbf{9.34}" display="inline"><semantics id="S3.T5.5.5.2.m1.1a"><mn class="ltx_mathvariant_bold" mathvariant="bold" id="S3.T5.5.5.2.m1.1.1" xref="S3.T5.5.5.2.m1.1.1.cmml">9.34</mn><annotation-xml encoding="MathML-Content" id="S3.T5.5.5.2.m1.1b"><cn type="float" id="S3.T5.5.5.2.m1.1.1.cmml" xref="S3.T5.5.5.2.m1.1.1">9.34</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T5.5.5.2.m1.1c">\mathbf{9.34}</annotation></semantics></math></td>
<td id="S3.T5.6.6.3" class="ltx_td ltx_align_center"><span id="S3.T5.6.6.3.1" class="ltx_text ltx_markedasmath ltx_font_bold">13.25</span></td>
</tr>
<tr id="S3.T5.9.9" class="ltx_tr">
<td id="S3.T5.9.9.4" class="ltx_td ltx_align_left ltx_border_b">H64-MQA</td>
<td id="S3.T5.9.9.5" class="ltx_td ltx_align_center ltx_border_b">
<span id="S3.T5.9.9.5.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>8192</td>
<td id="S3.T5.7.7.1" class="ltx_td ltx_align_center ltx_border_b"><math id="S3.T5.7.7.1.m1.1" class="ltx_Math" alttext="1.00\times 10^{-3}" display="inline"><semantics id="S3.T5.7.7.1.m1.1a"><mrow id="S3.T5.7.7.1.m1.1.1" xref="S3.T5.7.7.1.m1.1.1.cmml"><mn id="S3.T5.7.7.1.m1.1.1.2" xref="S3.T5.7.7.1.m1.1.1.2.cmml">1.00</mn><mo lspace="0.222em" rspace="0.222em" id="S3.T5.7.7.1.m1.1.1.1" xref="S3.T5.7.7.1.m1.1.1.1.cmml">×</mo><msup id="S3.T5.7.7.1.m1.1.1.3" xref="S3.T5.7.7.1.m1.1.1.3.cmml"><mn id="S3.T5.7.7.1.m1.1.1.3.2" xref="S3.T5.7.7.1.m1.1.1.3.2.cmml">10</mn><mrow id="S3.T5.7.7.1.m1.1.1.3.3" xref="S3.T5.7.7.1.m1.1.1.3.3.cmml"><mo id="S3.T5.7.7.1.m1.1.1.3.3a" xref="S3.T5.7.7.1.m1.1.1.3.3.cmml">−</mo><mn id="S3.T5.7.7.1.m1.1.1.3.3.2" xref="S3.T5.7.7.1.m1.1.1.3.3.2.cmml">3</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.T5.7.7.1.m1.1b"><apply id="S3.T5.7.7.1.m1.1.1.cmml" xref="S3.T5.7.7.1.m1.1.1"><times id="S3.T5.7.7.1.m1.1.1.1.cmml" xref="S3.T5.7.7.1.m1.1.1.1"></times><cn type="float" id="S3.T5.7.7.1.m1.1.1.2.cmml" xref="S3.T5.7.7.1.m1.1.1.2">1.00</cn><apply id="S3.T5.7.7.1.m1.1.1.3.cmml" xref="S3.T5.7.7.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.T5.7.7.1.m1.1.1.3.1.cmml" xref="S3.T5.7.7.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="S3.T5.7.7.1.m1.1.1.3.2.cmml" xref="S3.T5.7.7.1.m1.1.1.3.2">10</cn><apply id="S3.T5.7.7.1.m1.1.1.3.3.cmml" xref="S3.T5.7.7.1.m1.1.1.3.3"><minus id="S3.T5.7.7.1.m1.1.1.3.3.1.cmml" xref="S3.T5.7.7.1.m1.1.1.3.3"></minus><cn type="integer" id="S3.T5.7.7.1.m1.1.1.3.3.2.cmml" xref="S3.T5.7.7.1.m1.1.1.3.3.2">3</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.T5.7.7.1.m1.1c">1.00\times 10^{-3}</annotation></semantics></math></td>
<td id="S3.T5.8.8.2" class="ltx_td ltx_align_center ltx_border_b"><math id="S3.T5.8.8.2.m1.1" class="ltx_Math" alttext="9.48" display="inline"><semantics id="S3.T5.8.8.2.m1.1a"><mn id="S3.T5.8.8.2.m1.1.1" xref="S3.T5.8.8.2.m1.1.1.cmml">9.48</mn><annotation-xml encoding="MathML-Content" id="S3.T5.8.8.2.m1.1b"><cn type="float" id="S3.T5.8.8.2.m1.1.1.cmml" xref="S3.T5.8.8.2.m1.1.1">9.48</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T5.8.8.2.m1.1c">9.48</annotation></semantics></math></td>
<td id="S3.T5.9.9.3" class="ltx_td ltx_align_center ltx_border_b"><math id="S3.T5.9.9.3.m1.1" class="ltx_Math" alttext="13.49" display="inline"><semantics id="S3.T5.9.9.3.m1.1a"><mn id="S3.T5.9.9.3.m1.1.1" xref="S3.T5.9.9.3.m1.1.1.cmml">13.49</mn><annotation-xml encoding="MathML-Content" id="S3.T5.9.9.3.m1.1b"><cn type="float" id="S3.T5.9.9.3.m1.1.1.cmml" xref="S3.T5.9.9.3.m1.1.1">13.49</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T5.9.9.3.m1.1c">13.49</annotation></semantics></math></td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 5: </span>Optimal learning rate and perplexity results for our first set of 3B-scale experiments.</figcaption>
</figure>
<figure id="S3.T6" class="ltx_table">
<table id="S3.T6.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S3.T6.1.1" class="ltx_tr">
<td id="S3.T6.1.1.1" class="ltx_td ltx_align_left ltx_border_t"><span id="S3.T6.1.1.1.1" class="ltx_text ltx_font_bold">Model (Best LR)</span></td>
<td id="S3.T6.1.1.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T6.1.1.2.1" class="ltx_text ltx_font_bold">Hellaswag</span></td>
<td id="S3.T6.1.1.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T6.1.1.3.1" class="ltx_text ltx_font_bold">PIQA</span></td>
<td id="S3.T6.1.1.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T6.1.1.4.1" class="ltx_text ltx_font_bold">WG</span></td>
<td id="S3.T6.1.1.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T6.1.1.5.1" class="ltx_text ltx_font_bold">SciQ</span></td>
<td id="S3.T6.1.1.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T6.1.1.6.1" class="ltx_text ltx_font_bold">OBQA</span></td>
<td id="S3.T6.1.1.7" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T6.1.1.7.1" class="ltx_text ltx_font_bold">BoolQ</span></td>
<td id="S3.T6.1.1.8" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T6.1.1.8.1" class="ltx_text ltx_font_bold">ARC-E</span></td>
</tr>
<tr id="S3.T6.1.2" class="ltx_tr">
<td id="S3.T6.1.2.1" class="ltx_td ltx_align_left ltx_border_t">H128-MQA</td>
<td id="S3.T6.1.2.2" class="ltx_td ltx_align_center ltx_border_t">45.73</td>
<td id="S3.T6.1.2.3" class="ltx_td ltx_align_center ltx_border_t">73.07</td>
<td id="S3.T6.1.2.4" class="ltx_td ltx_align_center ltx_border_t">60.46</td>
<td id="S3.T6.1.2.5" class="ltx_td ltx_align_center ltx_border_t">88.1</td>
<td id="S3.T6.1.2.6" class="ltx_td ltx_align_center ltx_border_t">25.4</td>
<td id="S3.T6.1.2.7" class="ltx_td ltx_align_center ltx_border_t">59.30</td>
<td id="S3.T6.1.2.8" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T6.1.2.8.1" class="ltx_text ltx_font_bold">64.90</span></td>
</tr>
<tr id="S3.T6.1.3" class="ltx_tr">
<td id="S3.T6.1.3.1" class="ltx_td ltx_align_left">H128-MQA-CLA2</td>
<td id="S3.T6.1.3.2" class="ltx_td ltx_align_center"><span id="S3.T6.1.3.2.1" class="ltx_text ltx_font_bold">47.12</span></td>
<td id="S3.T6.1.3.3" class="ltx_td ltx_align_center"><span id="S3.T6.1.3.3.1" class="ltx_text ltx_font_bold">74.32</span></td>
<td id="S3.T6.1.3.4" class="ltx_td ltx_align_center"><span id="S3.T6.1.3.4.1" class="ltx_text ltx_font_bold">60.69</span></td>
<td id="S3.T6.1.3.5" class="ltx_td ltx_align_center"><span id="S3.T6.1.3.5.1" class="ltx_text ltx_font_bold">89.2</span></td>
<td id="S3.T6.1.3.6" class="ltx_td ltx_align_center">25.2</td>
<td id="S3.T6.1.3.7" class="ltx_td ltx_align_center">58.62</td>
<td id="S3.T6.1.3.8" class="ltx_td ltx_align_center">64.73</td>
</tr>
<tr id="S3.T6.1.4" class="ltx_tr">
<td id="S3.T6.1.4.1" class="ltx_td ltx_align_left ltx_border_b">H64-MQA</td>
<td id="S3.T6.1.4.2" class="ltx_td ltx_align_center ltx_border_b">46.42</td>
<td id="S3.T6.1.4.3" class="ltx_td ltx_align_center ltx_border_b">74.05</td>
<td id="S3.T6.1.4.4" class="ltx_td ltx_align_center ltx_border_b">57.85</td>
<td id="S3.T6.1.4.5" class="ltx_td ltx_align_center ltx_border_b">88.1</td>
<td id="S3.T6.1.4.6" class="ltx_td ltx_align_center ltx_border_b"><span id="S3.T6.1.4.6.1" class="ltx_text ltx_font_bold">25.6</span></td>
<td id="S3.T6.1.4.7" class="ltx_td ltx_align_center ltx_border_b"><span id="S3.T6.1.4.7.1" class="ltx_text ltx_font_bold">59.88</span></td>
<td id="S3.T6.1.4.8" class="ltx_td ltx_align_center ltx_border_b">65.57</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 6: </span>Downstream evaluation results for our first set of 3B-scale experiments.</figcaption>
</figure>
</section>
<section id="S3.SS3.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Experiments at Head Dimension <math id="S3.SS3.SSS0.Px2.1.m1.1" class="ltx_Math" alttext="\boldsymbol{d}_{\textbf{head}}=\mathbf{64}" display="inline"><semantics id="S3.SS3.SSS0.Px2.1.m1.1b"><mrow id="S3.SS3.SSS0.Px2.1.m1.1.1" xref="S3.SS3.SSS0.Px2.1.m1.1.1.cmml"><msub id="S3.SS3.SSS0.Px2.1.m1.1.1.2" xref="S3.SS3.SSS0.Px2.1.m1.1.1.2.cmml"><mi id="S3.SS3.SSS0.Px2.1.m1.1.1.2.2" xref="S3.SS3.SSS0.Px2.1.m1.1.1.2.2.cmml">𝒅</mi><mtext class="ltx_mathvariant_bold" id="S3.SS3.SSS0.Px2.1.m1.1.1.2.3" xref="S3.SS3.SSS0.Px2.1.m1.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS3.SSS0.Px2.1.m1.1.1.1" xref="S3.SS3.SSS0.Px2.1.m1.1.1.1.cmml">=</mo><mn id="S3.SS3.SSS0.Px2.1.m1.1.1.3" xref="S3.SS3.SSS0.Px2.1.m1.1.1.3.cmml">𝟔𝟒</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.1.m1.1c"><apply id="S3.SS3.SSS0.Px2.1.m1.1.1.cmml" xref="S3.SS3.SSS0.Px2.1.m1.1.1"><eq id="S3.SS3.SSS0.Px2.1.m1.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.1.m1.1.1.1"></eq><apply id="S3.SS3.SSS0.Px2.1.m1.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.1.m1.1.1.2.1.cmml" xref="S3.SS3.SSS0.Px2.1.m1.1.1.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.1.m1.1.1.2.2.cmml" xref="S3.SS3.SSS0.Px2.1.m1.1.1.2.2">𝒅</ci><ci id="S3.SS3.SSS0.Px2.1.m1.1.1.2.3a.cmml" xref="S3.SS3.SSS0.Px2.1.m1.1.1.2.3"><mtext class="ltx_mathvariant_bold" mathsize="70%" id="S3.SS3.SSS0.Px2.1.m1.1.1.2.3.cmml" xref="S3.SS3.SSS0.Px2.1.m1.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS3.SSS0.Px2.1.m1.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.1.m1.1.1.3">64</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.1.m1.1d">\boldsymbol{d}_{\textbf{head}}=\mathbf{64}</annotation></semantics></math>.</h5>

<div id="S3.SS3.SSS0.Px2.p1" class="ltx_para">
<p id="S3.SS3.SSS0.Px2.p1.5" class="ltx_p">Because in our initial 3B-scale experiments we found that our <math id="S3.SS3.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="d_{\text{head}}=64" display="inline"><semantics id="S3.SS3.SSS0.Px2.p1.1.m1.1a"><mrow id="S3.SS3.SSS0.Px2.p1.1.m1.1.1" xref="S3.SS3.SSS0.Px2.p1.1.m1.1.1.cmml"><msub id="S3.SS3.SSS0.Px2.p1.1.m1.1.1.2" xref="S3.SS3.SSS0.Px2.p1.1.m1.1.1.2.cmml"><mi id="S3.SS3.SSS0.Px2.p1.1.m1.1.1.2.2" xref="S3.SS3.SSS0.Px2.p1.1.m1.1.1.2.2.cmml">d</mi><mtext id="S3.SS3.SSS0.Px2.p1.1.m1.1.1.2.3" xref="S3.SS3.SSS0.Px2.p1.1.m1.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS3.SSS0.Px2.p1.1.m1.1.1.1" xref="S3.SS3.SSS0.Px2.p1.1.m1.1.1.1.cmml">=</mo><mn id="S3.SS3.SSS0.Px2.p1.1.m1.1.1.3" xref="S3.SS3.SSS0.Px2.p1.1.m1.1.1.3.cmml">64</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p1.1.m1.1b"><apply id="S3.SS3.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p1.1.m1.1.1"><eq id="S3.SS3.SSS0.Px2.p1.1.m1.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p1.1.m1.1.1.1"></eq><apply id="S3.SS3.SSS0.Px2.p1.1.m1.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p1.1.m1.1.1.2.1.cmml" xref="S3.SS3.SSS0.Px2.p1.1.m1.1.1.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p1.1.m1.1.1.2.2.cmml" xref="S3.SS3.SSS0.Px2.p1.1.m1.1.1.2.2">𝑑</ci><ci id="S3.SS3.SSS0.Px2.p1.1.m1.1.1.2.3a.cmml" xref="S3.SS3.SSS0.Px2.p1.1.m1.1.1.2.3"><mtext mathsize="70%" id="S3.SS3.SSS0.Px2.p1.1.m1.1.1.2.3.cmml" xref="S3.SS3.SSS0.Px2.p1.1.m1.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS3.SSS0.Px2.p1.1.m1.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p1.1.m1.1.1.3">64</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p1.1.m1.1c">d_{\text{head}}=64</annotation></semantics></math> MQA model represents a stronger baseline than our <math id="S3.SS3.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="d_{\text{head}}=128" display="inline"><semantics id="S3.SS3.SSS0.Px2.p1.2.m2.1a"><mrow id="S3.SS3.SSS0.Px2.p1.2.m2.1.1" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1.cmml"><msub id="S3.SS3.SSS0.Px2.p1.2.m2.1.1.2" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1.2.cmml"><mi id="S3.SS3.SSS0.Px2.p1.2.m2.1.1.2.2" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1.2.2.cmml">d</mi><mtext id="S3.SS3.SSS0.Px2.p1.2.m2.1.1.2.3" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS3.SSS0.Px2.p1.2.m2.1.1.1" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1.1.cmml">=</mo><mn id="S3.SS3.SSS0.Px2.p1.2.m2.1.1.3" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1.3.cmml">128</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p1.2.m2.1b"><apply id="S3.SS3.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1"><eq id="S3.SS3.SSS0.Px2.p1.2.m2.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1.1"></eq><apply id="S3.SS3.SSS0.Px2.p1.2.m2.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p1.2.m2.1.1.2.1.cmml" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p1.2.m2.1.1.2.2.cmml" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1.2.2">𝑑</ci><ci id="S3.SS3.SSS0.Px2.p1.2.m2.1.1.2.3a.cmml" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1.2.3"><mtext mathsize="70%" id="S3.SS3.SSS0.Px2.p1.2.m2.1.1.2.3.cmml" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS3.SSS0.Px2.p1.2.m2.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1.3">128</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p1.2.m2.1c">d_{\text{head}}=128</annotation></semantics></math> MQA model, we ran a second set of experiments with adjusted head sizes. Specifically, we chose to compare an MQA-CLA2 model with <math id="S3.SS3.SSS0.Px2.p1.3.m3.1" class="ltx_Math" alttext="d_{\text{head}}=64" display="inline"><semantics id="S3.SS3.SSS0.Px2.p1.3.m3.1a"><mrow id="S3.SS3.SSS0.Px2.p1.3.m3.1.1" xref="S3.SS3.SSS0.Px2.p1.3.m3.1.1.cmml"><msub id="S3.SS3.SSS0.Px2.p1.3.m3.1.1.2" xref="S3.SS3.SSS0.Px2.p1.3.m3.1.1.2.cmml"><mi id="S3.SS3.SSS0.Px2.p1.3.m3.1.1.2.2" xref="S3.SS3.SSS0.Px2.p1.3.m3.1.1.2.2.cmml">d</mi><mtext id="S3.SS3.SSS0.Px2.p1.3.m3.1.1.2.3" xref="S3.SS3.SSS0.Px2.p1.3.m3.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS3.SSS0.Px2.p1.3.m3.1.1.1" xref="S3.SS3.SSS0.Px2.p1.3.m3.1.1.1.cmml">=</mo><mn id="S3.SS3.SSS0.Px2.p1.3.m3.1.1.3" xref="S3.SS3.SSS0.Px2.p1.3.m3.1.1.3.cmml">64</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p1.3.m3.1b"><apply id="S3.SS3.SSS0.Px2.p1.3.m3.1.1.cmml" xref="S3.SS3.SSS0.Px2.p1.3.m3.1.1"><eq id="S3.SS3.SSS0.Px2.p1.3.m3.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p1.3.m3.1.1.1"></eq><apply id="S3.SS3.SSS0.Px2.p1.3.m3.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p1.3.m3.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p1.3.m3.1.1.2.1.cmml" xref="S3.SS3.SSS0.Px2.p1.3.m3.1.1.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p1.3.m3.1.1.2.2.cmml" xref="S3.SS3.SSS0.Px2.p1.3.m3.1.1.2.2">𝑑</ci><ci id="S3.SS3.SSS0.Px2.p1.3.m3.1.1.2.3a.cmml" xref="S3.SS3.SSS0.Px2.p1.3.m3.1.1.2.3"><mtext mathsize="70%" id="S3.SS3.SSS0.Px2.p1.3.m3.1.1.2.3.cmml" xref="S3.SS3.SSS0.Px2.p1.3.m3.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS3.SSS0.Px2.p1.3.m3.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p1.3.m3.1.1.3">64</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p1.3.m3.1c">d_{\text{head}}=64</annotation></semantics></math> to a plain MQA model with <math id="S3.SS3.SSS0.Px2.p1.4.m4.1" class="ltx_Math" alttext="d_{\text{head}}=64" display="inline"><semantics id="S3.SS3.SSS0.Px2.p1.4.m4.1a"><mrow id="S3.SS3.SSS0.Px2.p1.4.m4.1.1" xref="S3.SS3.SSS0.Px2.p1.4.m4.1.1.cmml"><msub id="S3.SS3.SSS0.Px2.p1.4.m4.1.1.2" xref="S3.SS3.SSS0.Px2.p1.4.m4.1.1.2.cmml"><mi id="S3.SS3.SSS0.Px2.p1.4.m4.1.1.2.2" xref="S3.SS3.SSS0.Px2.p1.4.m4.1.1.2.2.cmml">d</mi><mtext id="S3.SS3.SSS0.Px2.p1.4.m4.1.1.2.3" xref="S3.SS3.SSS0.Px2.p1.4.m4.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS3.SSS0.Px2.p1.4.m4.1.1.1" xref="S3.SS3.SSS0.Px2.p1.4.m4.1.1.1.cmml">=</mo><mn id="S3.SS3.SSS0.Px2.p1.4.m4.1.1.3" xref="S3.SS3.SSS0.Px2.p1.4.m4.1.1.3.cmml">64</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p1.4.m4.1b"><apply id="S3.SS3.SSS0.Px2.p1.4.m4.1.1.cmml" xref="S3.SS3.SSS0.Px2.p1.4.m4.1.1"><eq id="S3.SS3.SSS0.Px2.p1.4.m4.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p1.4.m4.1.1.1"></eq><apply id="S3.SS3.SSS0.Px2.p1.4.m4.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p1.4.m4.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p1.4.m4.1.1.2.1.cmml" xref="S3.SS3.SSS0.Px2.p1.4.m4.1.1.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p1.4.m4.1.1.2.2.cmml" xref="S3.SS3.SSS0.Px2.p1.4.m4.1.1.2.2">𝑑</ci><ci id="S3.SS3.SSS0.Px2.p1.4.m4.1.1.2.3a.cmml" xref="S3.SS3.SSS0.Px2.p1.4.m4.1.1.2.3"><mtext mathsize="70%" id="S3.SS3.SSS0.Px2.p1.4.m4.1.1.2.3.cmml" xref="S3.SS3.SSS0.Px2.p1.4.m4.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS3.SSS0.Px2.p1.4.m4.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p1.4.m4.1.1.3">64</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p1.4.m4.1c">d_{\text{head}}=64</annotation></semantics></math>, and to a plain MQA model with <math id="S3.SS3.SSS0.Px2.p1.5.m5.1" class="ltx_Math" alttext="d_{\text{head}}=32" display="inline"><semantics id="S3.SS3.SSS0.Px2.p1.5.m5.1a"><mrow id="S3.SS3.SSS0.Px2.p1.5.m5.1.1" xref="S3.SS3.SSS0.Px2.p1.5.m5.1.1.cmml"><msub id="S3.SS3.SSS0.Px2.p1.5.m5.1.1.2" xref="S3.SS3.SSS0.Px2.p1.5.m5.1.1.2.cmml"><mi id="S3.SS3.SSS0.Px2.p1.5.m5.1.1.2.2" xref="S3.SS3.SSS0.Px2.p1.5.m5.1.1.2.2.cmml">d</mi><mtext id="S3.SS3.SSS0.Px2.p1.5.m5.1.1.2.3" xref="S3.SS3.SSS0.Px2.p1.5.m5.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS3.SSS0.Px2.p1.5.m5.1.1.1" xref="S3.SS3.SSS0.Px2.p1.5.m5.1.1.1.cmml">=</mo><mn id="S3.SS3.SSS0.Px2.p1.5.m5.1.1.3" xref="S3.SS3.SSS0.Px2.p1.5.m5.1.1.3.cmml">32</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p1.5.m5.1b"><apply id="S3.SS3.SSS0.Px2.p1.5.m5.1.1.cmml" xref="S3.SS3.SSS0.Px2.p1.5.m5.1.1"><eq id="S3.SS3.SSS0.Px2.p1.5.m5.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p1.5.m5.1.1.1"></eq><apply id="S3.SS3.SSS0.Px2.p1.5.m5.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p1.5.m5.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p1.5.m5.1.1.2.1.cmml" xref="S3.SS3.SSS0.Px2.p1.5.m5.1.1.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p1.5.m5.1.1.2.2.cmml" xref="S3.SS3.SSS0.Px2.p1.5.m5.1.1.2.2">𝑑</ci><ci id="S3.SS3.SSS0.Px2.p1.5.m5.1.1.2.3a.cmml" xref="S3.SS3.SSS0.Px2.p1.5.m5.1.1.2.3"><mtext mathsize="70%" id="S3.SS3.SSS0.Px2.p1.5.m5.1.1.2.3.cmml" xref="S3.SS3.SSS0.Px2.p1.5.m5.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS3.SSS0.Px2.p1.5.m5.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p1.5.m5.1.1.3">32</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p1.5.m5.1c">d_{\text{head}}=32</annotation></semantics></math>.</p>
</div>
<div id="S3.SS3.SSS0.Px2.p2" class="ltx_para">
<p id="S3.SS3.SSS0.Px2.p2.1" class="ltx_p">Due to logistical constraints we trained all models in this second set of 3B-scale experiments on a different training cluster using a different training software stack and data order. This included retraining a new version of our H64-MQA-CLA2 baseline in order to control for differences in the new training environment.</p>
</div>
<div id="S3.SS3.SSS0.Px2.p3" class="ltx_para">
<p id="S3.SS3.SSS0.Px2.p3.6" class="ltx_p">We trained all models in this second set of experiments with a learning rate of <math id="S3.SS3.SSS0.Px2.p3.1.m1.1" class="ltx_Math" alttext="\text{LR}=10^{-3}" display="inline"><semantics id="S3.SS3.SSS0.Px2.p3.1.m1.1a"><mrow id="S3.SS3.SSS0.Px2.p3.1.m1.1.1" xref="S3.SS3.SSS0.Px2.p3.1.m1.1.1.cmml"><mtext id="S3.SS3.SSS0.Px2.p3.1.m1.1.1.2" xref="S3.SS3.SSS0.Px2.p3.1.m1.1.1.2a.cmml">LR</mtext><mo id="S3.SS3.SSS0.Px2.p3.1.m1.1.1.1" xref="S3.SS3.SSS0.Px2.p3.1.m1.1.1.1.cmml">=</mo><msup id="S3.SS3.SSS0.Px2.p3.1.m1.1.1.3" xref="S3.SS3.SSS0.Px2.p3.1.m1.1.1.3.cmml"><mn id="S3.SS3.SSS0.Px2.p3.1.m1.1.1.3.2" xref="S3.SS3.SSS0.Px2.p3.1.m1.1.1.3.2.cmml">10</mn><mrow id="S3.SS3.SSS0.Px2.p3.1.m1.1.1.3.3" xref="S3.SS3.SSS0.Px2.p3.1.m1.1.1.3.3.cmml"><mo id="S3.SS3.SSS0.Px2.p3.1.m1.1.1.3.3a" xref="S3.SS3.SSS0.Px2.p3.1.m1.1.1.3.3.cmml">−</mo><mn id="S3.SS3.SSS0.Px2.p3.1.m1.1.1.3.3.2" xref="S3.SS3.SSS0.Px2.p3.1.m1.1.1.3.3.2.cmml">3</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p3.1.m1.1b"><apply id="S3.SS3.SSS0.Px2.p3.1.m1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p3.1.m1.1.1"><eq id="S3.SS3.SSS0.Px2.p3.1.m1.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p3.1.m1.1.1.1"></eq><ci id="S3.SS3.SSS0.Px2.p3.1.m1.1.1.2a.cmml" xref="S3.SS3.SSS0.Px2.p3.1.m1.1.1.2"><mtext id="S3.SS3.SSS0.Px2.p3.1.m1.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p3.1.m1.1.1.2">LR</mtext></ci><apply id="S3.SS3.SSS0.Px2.p3.1.m1.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p3.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p3.1.m1.1.1.3.1.cmml" xref="S3.SS3.SSS0.Px2.p3.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="S3.SS3.SSS0.Px2.p3.1.m1.1.1.3.2.cmml" xref="S3.SS3.SSS0.Px2.p3.1.m1.1.1.3.2">10</cn><apply id="S3.SS3.SSS0.Px2.p3.1.m1.1.1.3.3.cmml" xref="S3.SS3.SSS0.Px2.p3.1.m1.1.1.3.3"><minus id="S3.SS3.SSS0.Px2.p3.1.m1.1.1.3.3.1.cmml" xref="S3.SS3.SSS0.Px2.p3.1.m1.1.1.3.3"></minus><cn type="integer" id="S3.SS3.SSS0.Px2.p3.1.m1.1.1.3.3.2.cmml" xref="S3.SS3.SSS0.Px2.p3.1.m1.1.1.3.3.2">3</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p3.1.m1.1c">\text{LR}=10^{-3}</annotation></semantics></math>, which we had found to be the best learning rate for our <math id="S3.SS3.SSS0.Px2.p3.2.m2.1" class="ltx_Math" alttext="d_{\text{head}}=64" display="inline"><semantics id="S3.SS3.SSS0.Px2.p3.2.m2.1a"><mrow id="S3.SS3.SSS0.Px2.p3.2.m2.1.1" xref="S3.SS3.SSS0.Px2.p3.2.m2.1.1.cmml"><msub id="S3.SS3.SSS0.Px2.p3.2.m2.1.1.2" xref="S3.SS3.SSS0.Px2.p3.2.m2.1.1.2.cmml"><mi id="S3.SS3.SSS0.Px2.p3.2.m2.1.1.2.2" xref="S3.SS3.SSS0.Px2.p3.2.m2.1.1.2.2.cmml">d</mi><mtext id="S3.SS3.SSS0.Px2.p3.2.m2.1.1.2.3" xref="S3.SS3.SSS0.Px2.p3.2.m2.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS3.SSS0.Px2.p3.2.m2.1.1.1" xref="S3.SS3.SSS0.Px2.p3.2.m2.1.1.1.cmml">=</mo><mn id="S3.SS3.SSS0.Px2.p3.2.m2.1.1.3" xref="S3.SS3.SSS0.Px2.p3.2.m2.1.1.3.cmml">64</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p3.2.m2.1b"><apply id="S3.SS3.SSS0.Px2.p3.2.m2.1.1.cmml" xref="S3.SS3.SSS0.Px2.p3.2.m2.1.1"><eq id="S3.SS3.SSS0.Px2.p3.2.m2.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p3.2.m2.1.1.1"></eq><apply id="S3.SS3.SSS0.Px2.p3.2.m2.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p3.2.m2.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p3.2.m2.1.1.2.1.cmml" xref="S3.SS3.SSS0.Px2.p3.2.m2.1.1.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p3.2.m2.1.1.2.2.cmml" xref="S3.SS3.SSS0.Px2.p3.2.m2.1.1.2.2">𝑑</ci><ci id="S3.SS3.SSS0.Px2.p3.2.m2.1.1.2.3a.cmml" xref="S3.SS3.SSS0.Px2.p3.2.m2.1.1.2.3"><mtext mathsize="70%" id="S3.SS3.SSS0.Px2.p3.2.m2.1.1.2.3.cmml" xref="S3.SS3.SSS0.Px2.p3.2.m2.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS3.SSS0.Px2.p3.2.m2.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p3.2.m2.1.1.3">64</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p3.2.m2.1c">d_{\text{head}}=64</annotation></semantics></math> MQA baseline model in our first set of 3B-scale experiments. For our <math id="S3.SS3.SSS0.Px2.p3.3.m3.1" class="ltx_Math" alttext="d_{\text{head}}=64" display="inline"><semantics id="S3.SS3.SSS0.Px2.p3.3.m3.1a"><mrow id="S3.SS3.SSS0.Px2.p3.3.m3.1.1" xref="S3.SS3.SSS0.Px2.p3.3.m3.1.1.cmml"><msub id="S3.SS3.SSS0.Px2.p3.3.m3.1.1.2" xref="S3.SS3.SSS0.Px2.p3.3.m3.1.1.2.cmml"><mi id="S3.SS3.SSS0.Px2.p3.3.m3.1.1.2.2" xref="S3.SS3.SSS0.Px2.p3.3.m3.1.1.2.2.cmml">d</mi><mtext id="S3.SS3.SSS0.Px2.p3.3.m3.1.1.2.3" xref="S3.SS3.SSS0.Px2.p3.3.m3.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS3.SSS0.Px2.p3.3.m3.1.1.1" xref="S3.SS3.SSS0.Px2.p3.3.m3.1.1.1.cmml">=</mo><mn id="S3.SS3.SSS0.Px2.p3.3.m3.1.1.3" xref="S3.SS3.SSS0.Px2.p3.3.m3.1.1.3.cmml">64</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p3.3.m3.1b"><apply id="S3.SS3.SSS0.Px2.p3.3.m3.1.1.cmml" xref="S3.SS3.SSS0.Px2.p3.3.m3.1.1"><eq id="S3.SS3.SSS0.Px2.p3.3.m3.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p3.3.m3.1.1.1"></eq><apply id="S3.SS3.SSS0.Px2.p3.3.m3.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p3.3.m3.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p3.3.m3.1.1.2.1.cmml" xref="S3.SS3.SSS0.Px2.p3.3.m3.1.1.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p3.3.m3.1.1.2.2.cmml" xref="S3.SS3.SSS0.Px2.p3.3.m3.1.1.2.2">𝑑</ci><ci id="S3.SS3.SSS0.Px2.p3.3.m3.1.1.2.3a.cmml" xref="S3.SS3.SSS0.Px2.p3.3.m3.1.1.2.3"><mtext mathsize="70%" id="S3.SS3.SSS0.Px2.p3.3.m3.1.1.2.3.cmml" xref="S3.SS3.SSS0.Px2.p3.3.m3.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS3.SSS0.Px2.p3.3.m3.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p3.3.m3.1.1.3">64</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p3.3.m3.1c">d_{\text{head}}=64</annotation></semantics></math> MQA-CLA2 model and our <math id="S3.SS3.SSS0.Px2.p3.4.m4.1" class="ltx_Math" alttext="d_{\text{head}}=32" display="inline"><semantics id="S3.SS3.SSS0.Px2.p3.4.m4.1a"><mrow id="S3.SS3.SSS0.Px2.p3.4.m4.1.1" xref="S3.SS3.SSS0.Px2.p3.4.m4.1.1.cmml"><msub id="S3.SS3.SSS0.Px2.p3.4.m4.1.1.2" xref="S3.SS3.SSS0.Px2.p3.4.m4.1.1.2.cmml"><mi id="S3.SS3.SSS0.Px2.p3.4.m4.1.1.2.2" xref="S3.SS3.SSS0.Px2.p3.4.m4.1.1.2.2.cmml">d</mi><mtext id="S3.SS3.SSS0.Px2.p3.4.m4.1.1.2.3" xref="S3.SS3.SSS0.Px2.p3.4.m4.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS3.SSS0.Px2.p3.4.m4.1.1.1" xref="S3.SS3.SSS0.Px2.p3.4.m4.1.1.1.cmml">=</mo><mn id="S3.SS3.SSS0.Px2.p3.4.m4.1.1.3" xref="S3.SS3.SSS0.Px2.p3.4.m4.1.1.3.cmml">32</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p3.4.m4.1b"><apply id="S3.SS3.SSS0.Px2.p3.4.m4.1.1.cmml" xref="S3.SS3.SSS0.Px2.p3.4.m4.1.1"><eq id="S3.SS3.SSS0.Px2.p3.4.m4.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p3.4.m4.1.1.1"></eq><apply id="S3.SS3.SSS0.Px2.p3.4.m4.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p3.4.m4.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p3.4.m4.1.1.2.1.cmml" xref="S3.SS3.SSS0.Px2.p3.4.m4.1.1.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p3.4.m4.1.1.2.2.cmml" xref="S3.SS3.SSS0.Px2.p3.4.m4.1.1.2.2">𝑑</ci><ci id="S3.SS3.SSS0.Px2.p3.4.m4.1.1.2.3a.cmml" xref="S3.SS3.SSS0.Px2.p3.4.m4.1.1.2.3"><mtext mathsize="70%" id="S3.SS3.SSS0.Px2.p3.4.m4.1.1.2.3.cmml" xref="S3.SS3.SSS0.Px2.p3.4.m4.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS3.SSS0.Px2.p3.4.m4.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p3.4.m4.1.1.3">32</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p3.4.m4.1c">d_{\text{head}}=32</annotation></semantics></math> MQA baseline model, we also experimented with learning rates of <math id="S3.SS3.SSS0.Px2.p3.5.m5.2" class="ltx_Math" alttext="\text{LR}\in\{6.75\times 10^{-4},1.5\times 10^{-3}\}" display="inline"><semantics id="S3.SS3.SSS0.Px2.p3.5.m5.2a"><mrow id="S3.SS3.SSS0.Px2.p3.5.m5.2.2" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2.cmml"><mtext id="S3.SS3.SSS0.Px2.p3.5.m5.2.2.4" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2.4a.cmml">LR</mtext><mo id="S3.SS3.SSS0.Px2.p3.5.m5.2.2.3" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2.3.cmml">∈</mo><mrow id="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.3.cmml"><mo stretchy="false" id="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.3" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.3.cmml">{</mo><mrow id="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1" xref="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.cmml"><mn id="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.2" xref="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.2.cmml">6.75</mn><mo lspace="0.222em" rspace="0.222em" id="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.1" xref="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.1.cmml">×</mo><msup id="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.3" xref="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.3.cmml"><mn id="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.3.2" xref="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.3.2.cmml">10</mn><mrow id="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.3.3" xref="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.3.3.cmml"><mo id="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.3.3a" xref="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.3.3.cmml">−</mo><mn id="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.3.3.2" xref="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.3.3.2.cmml">4</mn></mrow></msup></mrow><mo id="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.4" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.3.cmml">,</mo><mrow id="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.cmml"><mn id="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.2" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.2.cmml">1.5</mn><mo lspace="0.222em" rspace="0.222em" id="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.1" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.1.cmml">×</mo><msup id="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.3" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.3.cmml"><mn id="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.3.2" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.3.2.cmml">10</mn><mrow id="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.3.3" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.3.3.cmml"><mo id="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.3.3a" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.3.3.cmml">−</mo><mn id="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.3.3.2" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.3.3.2.cmml">3</mn></mrow></msup></mrow><mo stretchy="false" id="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.5" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.3.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p3.5.m5.2b"><apply id="S3.SS3.SSS0.Px2.p3.5.m5.2.2.cmml" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2"><in id="S3.SS3.SSS0.Px2.p3.5.m5.2.2.3.cmml" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2.3"></in><ci id="S3.SS3.SSS0.Px2.p3.5.m5.2.2.4a.cmml" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2.4"><mtext id="S3.SS3.SSS0.Px2.p3.5.m5.2.2.4.cmml" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2.4">LR</mtext></ci><set id="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.3.cmml" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2"><apply id="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1"><times id="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.1"></times><cn type="float" id="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.2">6.75</cn><apply id="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.3.1.cmml" xref="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.3">superscript</csymbol><cn type="integer" id="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.3.2.cmml" xref="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.3.2">10</cn><apply id="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.3.3.cmml" xref="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.3.3"><minus id="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.3.3.1.cmml" xref="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.3.3"></minus><cn type="integer" id="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.3.3.2.cmml" xref="S3.SS3.SSS0.Px2.p3.5.m5.1.1.1.1.1.3.3.2">4</cn></apply></apply></apply><apply id="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.cmml" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2"><times id="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.1.cmml" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.1"></times><cn type="float" id="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.2.cmml" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.2">1.5</cn><apply id="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.3.cmml" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.3"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.3.1.cmml" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.3">superscript</csymbol><cn type="integer" id="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.3.2.cmml" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.3.2">10</cn><apply id="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.3.3.cmml" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.3.3"><minus id="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.3.3.1.cmml" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.3.3"></minus><cn type="integer" id="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.3.3.2.cmml" xref="S3.SS3.SSS0.Px2.p3.5.m5.2.2.2.2.2.3.3.2">3</cn></apply></apply></apply></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p3.5.m5.2c">\text{LR}\in\{6.75\times 10^{-4},1.5\times 10^{-3}\}</annotation></semantics></math>, but found these achieved worse perplexities than our initial value of <math id="S3.SS3.SSS0.Px2.p3.6.m6.1" class="ltx_Math" alttext="\text{LR}=10^{-3}" display="inline"><semantics id="S3.SS3.SSS0.Px2.p3.6.m6.1a"><mrow id="S3.SS3.SSS0.Px2.p3.6.m6.1.1" xref="S3.SS3.SSS0.Px2.p3.6.m6.1.1.cmml"><mtext id="S3.SS3.SSS0.Px2.p3.6.m6.1.1.2" xref="S3.SS3.SSS0.Px2.p3.6.m6.1.1.2a.cmml">LR</mtext><mo id="S3.SS3.SSS0.Px2.p3.6.m6.1.1.1" xref="S3.SS3.SSS0.Px2.p3.6.m6.1.1.1.cmml">=</mo><msup id="S3.SS3.SSS0.Px2.p3.6.m6.1.1.3" xref="S3.SS3.SSS0.Px2.p3.6.m6.1.1.3.cmml"><mn id="S3.SS3.SSS0.Px2.p3.6.m6.1.1.3.2" xref="S3.SS3.SSS0.Px2.p3.6.m6.1.1.3.2.cmml">10</mn><mrow id="S3.SS3.SSS0.Px2.p3.6.m6.1.1.3.3" xref="S3.SS3.SSS0.Px2.p3.6.m6.1.1.3.3.cmml"><mo id="S3.SS3.SSS0.Px2.p3.6.m6.1.1.3.3a" xref="S3.SS3.SSS0.Px2.p3.6.m6.1.1.3.3.cmml">−</mo><mn id="S3.SS3.SSS0.Px2.p3.6.m6.1.1.3.3.2" xref="S3.SS3.SSS0.Px2.p3.6.m6.1.1.3.3.2.cmml">3</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p3.6.m6.1b"><apply id="S3.SS3.SSS0.Px2.p3.6.m6.1.1.cmml" xref="S3.SS3.SSS0.Px2.p3.6.m6.1.1"><eq id="S3.SS3.SSS0.Px2.p3.6.m6.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p3.6.m6.1.1.1"></eq><ci id="S3.SS3.SSS0.Px2.p3.6.m6.1.1.2a.cmml" xref="S3.SS3.SSS0.Px2.p3.6.m6.1.1.2"><mtext id="S3.SS3.SSS0.Px2.p3.6.m6.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p3.6.m6.1.1.2">LR</mtext></ci><apply id="S3.SS3.SSS0.Px2.p3.6.m6.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p3.6.m6.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p3.6.m6.1.1.3.1.cmml" xref="S3.SS3.SSS0.Px2.p3.6.m6.1.1.3">superscript</csymbol><cn type="integer" id="S3.SS3.SSS0.Px2.p3.6.m6.1.1.3.2.cmml" xref="S3.SS3.SSS0.Px2.p3.6.m6.1.1.3.2">10</cn><apply id="S3.SS3.SSS0.Px2.p3.6.m6.1.1.3.3.cmml" xref="S3.SS3.SSS0.Px2.p3.6.m6.1.1.3.3"><minus id="S3.SS3.SSS0.Px2.p3.6.m6.1.1.3.3.1.cmml" xref="S3.SS3.SSS0.Px2.p3.6.m6.1.1.3.3"></minus><cn type="integer" id="S3.SS3.SSS0.Px2.p3.6.m6.1.1.3.3.2.cmml" xref="S3.SS3.SSS0.Px2.p3.6.m6.1.1.3.3.2">3</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p3.6.m6.1c">\text{LR}=10^{-3}</annotation></semantics></math>.</p>
</div>
<div id="S3.SS3.SSS0.Px2.p4" class="ltx_para">
<p id="S3.SS3.SSS0.Px2.p4.6" class="ltx_p">We report perplexity results for this second set of experiments in Table <a href="#S3.T7" title='Table 7 ‣ Experiments at Head Dimension 𝒅_"head"=𝟔𝟒. ‣ 3.3 Experiments at 3B-Parameter Scale ‣ 3 Pretraining Experiments ‣ Reducing Transformer Key-Value Cache Size with Cross-Layer Attention' class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>, and results for downstream benchmarks in Table <a href="#S3.T8" title='Table 8 ‣ Experiments at Head Dimension 𝒅_"head"=𝟔𝟒. ‣ 3.3 Experiments at 3B-Parameter Scale ‣ 3 Pretraining Experiments ‣ Reducing Transformer Key-Value Cache Size with Cross-Layer Attention' class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a>. In the Wikitext perplexity results for this set of experiments, we find agreement with the pattern observed at the 1B scale. Our MQA-CLA2 model with <math id="S3.SS3.SSS0.Px2.p4.1.m1.1" class="ltx_Math" alttext="d_{\text{head}}=64" display="inline"><semantics id="S3.SS3.SSS0.Px2.p4.1.m1.1a"><mrow id="S3.SS3.SSS0.Px2.p4.1.m1.1.1" xref="S3.SS3.SSS0.Px2.p4.1.m1.1.1.cmml"><msub id="S3.SS3.SSS0.Px2.p4.1.m1.1.1.2" xref="S3.SS3.SSS0.Px2.p4.1.m1.1.1.2.cmml"><mi id="S3.SS3.SSS0.Px2.p4.1.m1.1.1.2.2" xref="S3.SS3.SSS0.Px2.p4.1.m1.1.1.2.2.cmml">d</mi><mtext id="S3.SS3.SSS0.Px2.p4.1.m1.1.1.2.3" xref="S3.SS3.SSS0.Px2.p4.1.m1.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS3.SSS0.Px2.p4.1.m1.1.1.1" xref="S3.SS3.SSS0.Px2.p4.1.m1.1.1.1.cmml">=</mo><mn id="S3.SS3.SSS0.Px2.p4.1.m1.1.1.3" xref="S3.SS3.SSS0.Px2.p4.1.m1.1.1.3.cmml">64</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p4.1.m1.1b"><apply id="S3.SS3.SSS0.Px2.p4.1.m1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p4.1.m1.1.1"><eq id="S3.SS3.SSS0.Px2.p4.1.m1.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p4.1.m1.1.1.1"></eq><apply id="S3.SS3.SSS0.Px2.p4.1.m1.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p4.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p4.1.m1.1.1.2.1.cmml" xref="S3.SS3.SSS0.Px2.p4.1.m1.1.1.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p4.1.m1.1.1.2.2.cmml" xref="S3.SS3.SSS0.Px2.p4.1.m1.1.1.2.2">𝑑</ci><ci id="S3.SS3.SSS0.Px2.p4.1.m1.1.1.2.3a.cmml" xref="S3.SS3.SSS0.Px2.p4.1.m1.1.1.2.3"><mtext mathsize="70%" id="S3.SS3.SSS0.Px2.p4.1.m1.1.1.2.3.cmml" xref="S3.SS3.SSS0.Px2.p4.1.m1.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS3.SSS0.Px2.p4.1.m1.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p4.1.m1.1.1.3">64</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p4.1.m1.1c">d_{\text{head}}=64</annotation></semantics></math> incurs only a small (<math id="S3.SS3.SSS0.Px2.p4.2.m2.1" class="ltx_Math" alttext="0.05" display="inline"><semantics id="S3.SS3.SSS0.Px2.p4.2.m2.1a"><mn id="S3.SS3.SSS0.Px2.p4.2.m2.1.1" xref="S3.SS3.SSS0.Px2.p4.2.m2.1.1.cmml">0.05</mn><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p4.2.m2.1b"><cn type="float" id="S3.SS3.SSS0.Px2.p4.2.m2.1.1.cmml" xref="S3.SS3.SSS0.Px2.p4.2.m2.1.1">0.05</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p4.2.m2.1c">0.05</annotation></semantics></math> point) degradation in perplexity compared to our <math id="S3.SS3.SSS0.Px2.p4.3.m3.1" class="ltx_Math" alttext="d_{\text{head}}=64" display="inline"><semantics id="S3.SS3.SSS0.Px2.p4.3.m3.1a"><mrow id="S3.SS3.SSS0.Px2.p4.3.m3.1.1" xref="S3.SS3.SSS0.Px2.p4.3.m3.1.1.cmml"><msub id="S3.SS3.SSS0.Px2.p4.3.m3.1.1.2" xref="S3.SS3.SSS0.Px2.p4.3.m3.1.1.2.cmml"><mi id="S3.SS3.SSS0.Px2.p4.3.m3.1.1.2.2" xref="S3.SS3.SSS0.Px2.p4.3.m3.1.1.2.2.cmml">d</mi><mtext id="S3.SS3.SSS0.Px2.p4.3.m3.1.1.2.3" xref="S3.SS3.SSS0.Px2.p4.3.m3.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS3.SSS0.Px2.p4.3.m3.1.1.1" xref="S3.SS3.SSS0.Px2.p4.3.m3.1.1.1.cmml">=</mo><mn id="S3.SS3.SSS0.Px2.p4.3.m3.1.1.3" xref="S3.SS3.SSS0.Px2.p4.3.m3.1.1.3.cmml">64</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p4.3.m3.1b"><apply id="S3.SS3.SSS0.Px2.p4.3.m3.1.1.cmml" xref="S3.SS3.SSS0.Px2.p4.3.m3.1.1"><eq id="S3.SS3.SSS0.Px2.p4.3.m3.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p4.3.m3.1.1.1"></eq><apply id="S3.SS3.SSS0.Px2.p4.3.m3.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p4.3.m3.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p4.3.m3.1.1.2.1.cmml" xref="S3.SS3.SSS0.Px2.p4.3.m3.1.1.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p4.3.m3.1.1.2.2.cmml" xref="S3.SS3.SSS0.Px2.p4.3.m3.1.1.2.2">𝑑</ci><ci id="S3.SS3.SSS0.Px2.p4.3.m3.1.1.2.3a.cmml" xref="S3.SS3.SSS0.Px2.p4.3.m3.1.1.2.3"><mtext mathsize="70%" id="S3.SS3.SSS0.Px2.p4.3.m3.1.1.2.3.cmml" xref="S3.SS3.SSS0.Px2.p4.3.m3.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS3.SSS0.Px2.p4.3.m3.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p4.3.m3.1.1.3">64</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p4.3.m3.1c">d_{\text{head}}=64</annotation></semantics></math> baseline while enjoying a <math id="S3.SS3.SSS0.Px2.p4.4.m4.1" class="ltx_math_unparsed" alttext="2\times" display="inline"><semantics id="S3.SS3.SSS0.Px2.p4.4.m4.1a"><mrow id="S3.SS3.SSS0.Px2.p4.4.m4.1b"><mn id="S3.SS3.SSS0.Px2.p4.4.m4.1.1">2</mn><mo lspace="0.222em" id="S3.SS3.SSS0.Px2.p4.4.m4.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p4.4.m4.1c">2\times</annotation></semantics></math> smaller KV cache footprint, and achieves a substantial (<math id="S3.SS3.SSS0.Px2.p4.5.m5.1" class="ltx_Math" alttext="0.35" display="inline"><semantics id="S3.SS3.SSS0.Px2.p4.5.m5.1a"><mn id="S3.SS3.SSS0.Px2.p4.5.m5.1.1" xref="S3.SS3.SSS0.Px2.p4.5.m5.1.1.cmml">0.35</mn><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p4.5.m5.1b"><cn type="float" id="S3.SS3.SSS0.Px2.p4.5.m5.1.1.cmml" xref="S3.SS3.SSS0.Px2.p4.5.m5.1.1">0.35</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p4.5.m5.1c">0.35</annotation></semantics></math> point) improvement in perplexity compared to our <math id="S3.SS3.SSS0.Px2.p4.6.m6.1" class="ltx_Math" alttext="d_{\text{head}}=32" display="inline"><semantics id="S3.SS3.SSS0.Px2.p4.6.m6.1a"><mrow id="S3.SS3.SSS0.Px2.p4.6.m6.1.1" xref="S3.SS3.SSS0.Px2.p4.6.m6.1.1.cmml"><msub id="S3.SS3.SSS0.Px2.p4.6.m6.1.1.2" xref="S3.SS3.SSS0.Px2.p4.6.m6.1.1.2.cmml"><mi id="S3.SS3.SSS0.Px2.p4.6.m6.1.1.2.2" xref="S3.SS3.SSS0.Px2.p4.6.m6.1.1.2.2.cmml">d</mi><mtext id="S3.SS3.SSS0.Px2.p4.6.m6.1.1.2.3" xref="S3.SS3.SSS0.Px2.p4.6.m6.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS3.SSS0.Px2.p4.6.m6.1.1.1" xref="S3.SS3.SSS0.Px2.p4.6.m6.1.1.1.cmml">=</mo><mn id="S3.SS3.SSS0.Px2.p4.6.m6.1.1.3" xref="S3.SS3.SSS0.Px2.p4.6.m6.1.1.3.cmml">32</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p4.6.m6.1b"><apply id="S3.SS3.SSS0.Px2.p4.6.m6.1.1.cmml" xref="S3.SS3.SSS0.Px2.p4.6.m6.1.1"><eq id="S3.SS3.SSS0.Px2.p4.6.m6.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p4.6.m6.1.1.1"></eq><apply id="S3.SS3.SSS0.Px2.p4.6.m6.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p4.6.m6.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p4.6.m6.1.1.2.1.cmml" xref="S3.SS3.SSS0.Px2.p4.6.m6.1.1.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p4.6.m6.1.1.2.2.cmml" xref="S3.SS3.SSS0.Px2.p4.6.m6.1.1.2.2">𝑑</ci><ci id="S3.SS3.SSS0.Px2.p4.6.m6.1.1.2.3a.cmml" xref="S3.SS3.SSS0.Px2.p4.6.m6.1.1.2.3"><mtext mathsize="70%" id="S3.SS3.SSS0.Px2.p4.6.m6.1.1.2.3.cmml" xref="S3.SS3.SSS0.Px2.p4.6.m6.1.1.2.3">head</mtext></ci></apply><cn type="integer" id="S3.SS3.SSS0.Px2.p4.6.m6.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p4.6.m6.1.1.3">32</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p4.6.m6.1c">d_{\text{head}}=32</annotation></semantics></math> baseline while using the same amount of KV cache memory.</p>
</div>
<div id="S3.SS3.SSS0.Px2.p5" class="ltx_para">
<p id="S3.SS3.SSS0.Px2.p5.1" class="ltx_p">We also evaluate these three models on downstream benchmarks, and report the results in Table <a href="#S3.T8" title='Table 8 ‣ Experiments at Head Dimension 𝒅_"head"=𝟔𝟒. ‣ 3.3 Experiments at 3B-Parameter Scale ‣ 3 Pretraining Experiments ‣ Reducing Transformer Key-Value Cache Size with Cross-Layer Attention' class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a>. As with our downstream benchmark evaluations for our other experiments, we find that all models perform similarly.</p>
</div>
<figure id="S3.T7" class="ltx_table">
<table id="S3.T7.6" class="ltx_tabular ltx_centering ltx_align_middle">
<thead class="ltx_thead">
<tr id="S3.T7.6.7.1" class="ltx_tr">
<th id="S3.T7.6.7.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_t">
<span id="S3.T7.6.7.1.1.1" class="ltx_text"></span> <span id="S3.T7.6.7.1.1.2" class="ltx_text">
<span id="S3.T7.6.7.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T7.6.7.1.1.2.1.1" class="ltx_tr">
<span id="S3.T7.6.7.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T7.6.7.1.1.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Model</span></span></span>
</span></span><span id="S3.T7.6.7.1.1.3" class="ltx_text"></span></th>
<th id="S3.T7.6.7.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_t">
<span id="S3.T7.6.7.1.2.1" class="ltx_text"></span> <span id="S3.T7.6.7.1.2.2" class="ltx_text">
<span id="S3.T7.6.7.1.2.2.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T7.6.7.1.2.2.1.1" class="ltx_tr">
<span id="S3.T7.6.7.1.2.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T7.6.7.1.2.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">KV Bytes Per</span></span></span>
<span id="S3.T7.6.7.1.2.2.1.2" class="ltx_tr">
<span id="S3.T7.6.7.1.2.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T7.6.7.1.2.2.1.2.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Token (16-bit)</span></span></span>
</span></span><span id="S3.T7.6.7.1.2.3" class="ltx_text"></span></th>
<th id="S3.T7.6.7.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_t">
<span id="S3.T7.6.7.1.3.1" class="ltx_text"></span> <span id="S3.T7.6.7.1.3.2" class="ltx_text">
<span id="S3.T7.6.7.1.3.2.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T7.6.7.1.3.2.1.1" class="ltx_tr">
<span id="S3.T7.6.7.1.3.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T7.6.7.1.3.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Best LR</span></span></span>
</span></span><span id="S3.T7.6.7.1.3.3" class="ltx_text"></span></th>
<th id="S3.T7.6.7.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_t">
<span id="S3.T7.6.7.1.4.1" class="ltx_text"></span> <span id="S3.T7.6.7.1.4.2" class="ltx_text">
<span id="S3.T7.6.7.1.4.2.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T7.6.7.1.4.2.1.1" class="ltx_tr">
<span id="S3.T7.6.7.1.4.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T7.6.7.1.4.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:80%;">Wikitext Perplexity</span></span></span>
</span></span><span id="S3.T7.6.7.1.4.3" class="ltx_text"></span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S3.T7.2.2" class="ltx_tr">
<td id="S3.T7.2.2.3" class="ltx_td ltx_align_left ltx_border_t">H64-MQA</td>
<td id="S3.T7.2.2.4" class="ltx_td ltx_align_center ltx_border_t">8192</td>
<td id="S3.T7.1.1.1" class="ltx_td ltx_align_center ltx_border_t"><math id="S3.T7.1.1.1.m1.1" class="ltx_Math" alttext="1.0\times 10^{-3}" display="inline"><semantics id="S3.T7.1.1.1.m1.1a"><mrow id="S3.T7.1.1.1.m1.1.1" xref="S3.T7.1.1.1.m1.1.1.cmml"><mn id="S3.T7.1.1.1.m1.1.1.2" xref="S3.T7.1.1.1.m1.1.1.2.cmml">1.0</mn><mo lspace="0.222em" rspace="0.222em" id="S3.T7.1.1.1.m1.1.1.1" xref="S3.T7.1.1.1.m1.1.1.1.cmml">×</mo><msup id="S3.T7.1.1.1.m1.1.1.3" xref="S3.T7.1.1.1.m1.1.1.3.cmml"><mn id="S3.T7.1.1.1.m1.1.1.3.2" xref="S3.T7.1.1.1.m1.1.1.3.2.cmml">10</mn><mrow id="S3.T7.1.1.1.m1.1.1.3.3" xref="S3.T7.1.1.1.m1.1.1.3.3.cmml"><mo id="S3.T7.1.1.1.m1.1.1.3.3a" xref="S3.T7.1.1.1.m1.1.1.3.3.cmml">−</mo><mn id="S3.T7.1.1.1.m1.1.1.3.3.2" xref="S3.T7.1.1.1.m1.1.1.3.3.2.cmml">3</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.T7.1.1.1.m1.1b"><apply id="S3.T7.1.1.1.m1.1.1.cmml" xref="S3.T7.1.1.1.m1.1.1"><times id="S3.T7.1.1.1.m1.1.1.1.cmml" xref="S3.T7.1.1.1.m1.1.1.1"></times><cn type="float" id="S3.T7.1.1.1.m1.1.1.2.cmml" xref="S3.T7.1.1.1.m1.1.1.2">1.0</cn><apply id="S3.T7.1.1.1.m1.1.1.3.cmml" xref="S3.T7.1.1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.T7.1.1.1.m1.1.1.3.1.cmml" xref="S3.T7.1.1.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="S3.T7.1.1.1.m1.1.1.3.2.cmml" xref="S3.T7.1.1.1.m1.1.1.3.2">10</cn><apply id="S3.T7.1.1.1.m1.1.1.3.3.cmml" xref="S3.T7.1.1.1.m1.1.1.3.3"><minus id="S3.T7.1.1.1.m1.1.1.3.3.1.cmml" xref="S3.T7.1.1.1.m1.1.1.3.3"></minus><cn type="integer" id="S3.T7.1.1.1.m1.1.1.3.3.2.cmml" xref="S3.T7.1.1.1.m1.1.1.3.3.2">3</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.T7.1.1.1.m1.1c">1.0\times 10^{-3}</annotation></semantics></math></td>
<td id="S3.T7.2.2.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T7.2.2.2.1" class="ltx_text ltx_markedasmath ltx_font_bold">12.94</span></td>
</tr>
<tr id="S3.T7.4.4" class="ltx_tr">
<td id="S3.T7.4.4.3" class="ltx_td ltx_align_left">H64-MQA-CLA2</td>
<td id="S3.T7.4.4.4" class="ltx_td ltx_align_center">4096</td>
<td id="S3.T7.3.3.1" class="ltx_td ltx_align_center"><math id="S3.T7.3.3.1.m1.1" class="ltx_Math" alttext="1.0\times 10^{-3}" display="inline"><semantics id="S3.T7.3.3.1.m1.1a"><mrow id="S3.T7.3.3.1.m1.1.1" xref="S3.T7.3.3.1.m1.1.1.cmml"><mn id="S3.T7.3.3.1.m1.1.1.2" xref="S3.T7.3.3.1.m1.1.1.2.cmml">1.0</mn><mo lspace="0.222em" rspace="0.222em" id="S3.T7.3.3.1.m1.1.1.1" xref="S3.T7.3.3.1.m1.1.1.1.cmml">×</mo><msup id="S3.T7.3.3.1.m1.1.1.3" xref="S3.T7.3.3.1.m1.1.1.3.cmml"><mn id="S3.T7.3.3.1.m1.1.1.3.2" xref="S3.T7.3.3.1.m1.1.1.3.2.cmml">10</mn><mrow id="S3.T7.3.3.1.m1.1.1.3.3" xref="S3.T7.3.3.1.m1.1.1.3.3.cmml"><mo id="S3.T7.3.3.1.m1.1.1.3.3a" xref="S3.T7.3.3.1.m1.1.1.3.3.cmml">−</mo><mn id="S3.T7.3.3.1.m1.1.1.3.3.2" xref="S3.T7.3.3.1.m1.1.1.3.3.2.cmml">3</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.T7.3.3.1.m1.1b"><apply id="S3.T7.3.3.1.m1.1.1.cmml" xref="S3.T7.3.3.1.m1.1.1"><times id="S3.T7.3.3.1.m1.1.1.1.cmml" xref="S3.T7.3.3.1.m1.1.1.1"></times><cn type="float" id="S3.T7.3.3.1.m1.1.1.2.cmml" xref="S3.T7.3.3.1.m1.1.1.2">1.0</cn><apply id="S3.T7.3.3.1.m1.1.1.3.cmml" xref="S3.T7.3.3.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.T7.3.3.1.m1.1.1.3.1.cmml" xref="S3.T7.3.3.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="S3.T7.3.3.1.m1.1.1.3.2.cmml" xref="S3.T7.3.3.1.m1.1.1.3.2">10</cn><apply id="S3.T7.3.3.1.m1.1.1.3.3.cmml" xref="S3.T7.3.3.1.m1.1.1.3.3"><minus id="S3.T7.3.3.1.m1.1.1.3.3.1.cmml" xref="S3.T7.3.3.1.m1.1.1.3.3"></minus><cn type="integer" id="S3.T7.3.3.1.m1.1.1.3.3.2.cmml" xref="S3.T7.3.3.1.m1.1.1.3.3.2">3</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.T7.3.3.1.m1.1c">1.0\times 10^{-3}</annotation></semantics></math></td>
<td id="S3.T7.4.4.2" class="ltx_td ltx_align_center"><math id="S3.T7.4.4.2.m1.1" class="ltx_Math" alttext="12.99" display="inline"><semantics id="S3.T7.4.4.2.m1.1a"><mn id="S3.T7.4.4.2.m1.1.1" xref="S3.T7.4.4.2.m1.1.1.cmml">12.99</mn><annotation-xml encoding="MathML-Content" id="S3.T7.4.4.2.m1.1b"><cn type="float" id="S3.T7.4.4.2.m1.1.1.cmml" xref="S3.T7.4.4.2.m1.1.1">12.99</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T7.4.4.2.m1.1c">12.99</annotation></semantics></math></td>
</tr>
<tr id="S3.T7.6.6" class="ltx_tr">
<td id="S3.T7.6.6.3" class="ltx_td ltx_align_left ltx_border_b">H32-MQA</td>
<td id="S3.T7.6.6.4" class="ltx_td ltx_align_center ltx_border_b">4096</td>
<td id="S3.T7.5.5.1" class="ltx_td ltx_align_center ltx_border_b"><math id="S3.T7.5.5.1.m1.1" class="ltx_Math" alttext="1.0\times 10^{-3}" display="inline"><semantics id="S3.T7.5.5.1.m1.1a"><mrow id="S3.T7.5.5.1.m1.1.1" xref="S3.T7.5.5.1.m1.1.1.cmml"><mn id="S3.T7.5.5.1.m1.1.1.2" xref="S3.T7.5.5.1.m1.1.1.2.cmml">1.0</mn><mo lspace="0.222em" rspace="0.222em" id="S3.T7.5.5.1.m1.1.1.1" xref="S3.T7.5.5.1.m1.1.1.1.cmml">×</mo><msup id="S3.T7.5.5.1.m1.1.1.3" xref="S3.T7.5.5.1.m1.1.1.3.cmml"><mn id="S3.T7.5.5.1.m1.1.1.3.2" xref="S3.T7.5.5.1.m1.1.1.3.2.cmml">10</mn><mrow id="S3.T7.5.5.1.m1.1.1.3.3" xref="S3.T7.5.5.1.m1.1.1.3.3.cmml"><mo id="S3.T7.5.5.1.m1.1.1.3.3a" xref="S3.T7.5.5.1.m1.1.1.3.3.cmml">−</mo><mn id="S3.T7.5.5.1.m1.1.1.3.3.2" xref="S3.T7.5.5.1.m1.1.1.3.3.2.cmml">3</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.T7.5.5.1.m1.1b"><apply id="S3.T7.5.5.1.m1.1.1.cmml" xref="S3.T7.5.5.1.m1.1.1"><times id="S3.T7.5.5.1.m1.1.1.1.cmml" xref="S3.T7.5.5.1.m1.1.1.1"></times><cn type="float" id="S3.T7.5.5.1.m1.1.1.2.cmml" xref="S3.T7.5.5.1.m1.1.1.2">1.0</cn><apply id="S3.T7.5.5.1.m1.1.1.3.cmml" xref="S3.T7.5.5.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.T7.5.5.1.m1.1.1.3.1.cmml" xref="S3.T7.5.5.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="S3.T7.5.5.1.m1.1.1.3.2.cmml" xref="S3.T7.5.5.1.m1.1.1.3.2">10</cn><apply id="S3.T7.5.5.1.m1.1.1.3.3.cmml" xref="S3.T7.5.5.1.m1.1.1.3.3"><minus id="S3.T7.5.5.1.m1.1.1.3.3.1.cmml" xref="S3.T7.5.5.1.m1.1.1.3.3"></minus><cn type="integer" id="S3.T7.5.5.1.m1.1.1.3.3.2.cmml" xref="S3.T7.5.5.1.m1.1.1.3.3.2">3</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.T7.5.5.1.m1.1c">1.0\times 10^{-3}</annotation></semantics></math></td>
<td id="S3.T7.6.6.2" class="ltx_td ltx_align_center ltx_border_b"><math id="S3.T7.6.6.2.m1.1" class="ltx_Math" alttext="13.34" display="inline"><semantics id="S3.T7.6.6.2.m1.1a"><mn id="S3.T7.6.6.2.m1.1.1" xref="S3.T7.6.6.2.m1.1.1.cmml">13.34</mn><annotation-xml encoding="MathML-Content" id="S3.T7.6.6.2.m1.1b"><cn type="float" id="S3.T7.6.6.2.m1.1.1.cmml" xref="S3.T7.6.6.2.m1.1.1">13.34</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T7.6.6.2.m1.1c">13.34</annotation></semantics></math></td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 7: </span>Optimal learning rate and perplexity results for our second set of 3B-scale experiments.</figcaption>
</figure>
<figure id="S3.T8" class="ltx_table">
<table id="S3.T8.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S3.T8.1.1" class="ltx_tr">
<td id="S3.T8.1.1.1" class="ltx_td ltx_align_left ltx_border_t"><span id="S3.T8.1.1.1.1" class="ltx_text ltx_font_bold">Model (Best LR)</span></td>
<td id="S3.T8.1.1.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T8.1.1.2.1" class="ltx_text ltx_font_bold">Hellaswag</span></td>
<td id="S3.T8.1.1.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T8.1.1.3.1" class="ltx_text ltx_font_bold">PIQA</span></td>
<td id="S3.T8.1.1.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T8.1.1.4.1" class="ltx_text ltx_font_bold">WG</span></td>
<td id="S3.T8.1.1.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T8.1.1.5.1" class="ltx_text ltx_font_bold">SciQ</span></td>
<td id="S3.T8.1.1.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T8.1.1.6.1" class="ltx_text ltx_font_bold">OBQA</span></td>
<td id="S3.T8.1.1.7" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T8.1.1.7.1" class="ltx_text ltx_font_bold">BoolQ</span></td>
<td id="S3.T8.1.1.8" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T8.1.1.8.1" class="ltx_text ltx_font_bold">ARC-E</span></td>
</tr>
<tr id="S3.T8.1.2" class="ltx_tr">
<td id="S3.T8.1.2.1" class="ltx_td ltx_align_left ltx_border_t">H64-MQA</td>
<td id="S3.T8.1.2.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T8.1.2.2.1" class="ltx_text ltx_font_bold">47.34</span></td>
<td id="S3.T8.1.2.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T8.1.2.3.1" class="ltx_text ltx_font_bold">74.54</span></td>
<td id="S3.T8.1.2.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T8.1.2.4.1" class="ltx_text ltx_font_bold">60.46</span></td>
<td id="S3.T8.1.2.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T8.1.2.5.1" class="ltx_text ltx_font_bold">88.9</span></td>
<td id="S3.T8.1.2.6" class="ltx_td ltx_align_center ltx_border_t">24.2</td>
<td id="S3.T8.1.2.7" class="ltx_td ltx_align_center ltx_border_t">57.25</td>
<td id="S3.T8.1.2.8" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T8.1.2.8.1" class="ltx_text ltx_font_bold">66.92</span></td>
</tr>
<tr id="S3.T8.1.3" class="ltx_tr">
<td id="S3.T8.1.3.1" class="ltx_td ltx_align_left">H64-MQA-CLA2</td>
<td id="S3.T8.1.3.2" class="ltx_td ltx_align_center">47.32</td>
<td id="S3.T8.1.3.3" class="ltx_td ltx_align_center"><span id="S3.T8.1.3.3.1" class="ltx_text ltx_font_bold">74.54</span></td>
<td id="S3.T8.1.3.4" class="ltx_td ltx_align_center">57.46</td>
<td id="S3.T8.1.3.5" class="ltx_td ltx_align_center">87.9</td>
<td id="S3.T8.1.3.6" class="ltx_td ltx_align_center">25.2</td>
<td id="S3.T8.1.3.7" class="ltx_td ltx_align_center">61.62</td>
<td id="S3.T8.1.3.8" class="ltx_td ltx_align_center">65.53</td>
</tr>
<tr id="S3.T8.1.4" class="ltx_tr">
<td id="S3.T8.1.4.1" class="ltx_td ltx_align_left ltx_border_b">H32-MQA</td>
<td id="S3.T8.1.4.2" class="ltx_td ltx_align_center ltx_border_b">46.05</td>
<td id="S3.T8.1.4.3" class="ltx_td ltx_align_center ltx_border_b">73.83</td>
<td id="S3.T8.1.4.4" class="ltx_td ltx_align_center ltx_border_b">60.06</td>
<td id="S3.T8.1.4.5" class="ltx_td ltx_align_center ltx_border_b">88.6</td>
<td id="S3.T8.1.4.6" class="ltx_td ltx_align_center ltx_border_b"><span id="S3.T8.1.4.6.1" class="ltx_text ltx_font_bold">25.6</span></td>
<td id="S3.T8.1.4.7" class="ltx_td ltx_align_center ltx_border_b"><span id="S3.T8.1.4.7.1" class="ltx_text ltx_font_bold">61.87</span></td>
<td id="S3.T8.1.4.8" class="ltx_td ltx_align_center ltx_border_b">65.24</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 8: </span>Downstream evaluation results for our second set of 3B-scale experiments.</figcaption>
</figure>
</section>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Discussion &amp; Future Work</h2>

<section id="S4.SS0.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Takeaways and Implications.</h5>

<div id="S4.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS0.SSS0.Px1.p1.2" class="ltx_p">In the regimes where we tested it, we find that MQA-CLA2 consistently achieves the lowest validation perplexity (within <math id="S4.SS0.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="0.01" display="inline"><semantics id="S4.SS0.SSS0.Px1.p1.1.m1.1a"><mn id="S4.SS0.SSS0.Px1.p1.1.m1.1.1" xref="S4.SS0.SSS0.Px1.p1.1.m1.1.1.cmml">0.01</mn><annotation-xml encoding="MathML-Content" id="S4.SS0.SSS0.Px1.p1.1.m1.1b"><cn type="float" id="S4.SS0.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S4.SS0.SSS0.Px1.p1.1.m1.1.1">0.01</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS0.SSS0.Px1.p1.1.m1.1c">0.01</annotation></semantics></math> points) for a given KV cache memory budget and model size. In our ablations, we find that using sharing factors greater than <math id="S4.SS0.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="2" display="inline"><semantics id="S4.SS0.SSS0.Px1.p1.2.m2.1a"><mn id="S4.SS0.SSS0.Px1.p1.2.m2.1.1" xref="S4.SS0.SSS0.Px1.p1.2.m2.1.1.cmml">2</mn><annotation-xml encoding="MathML-Content" id="S4.SS0.SSS0.Px1.p1.2.m2.1b"><cn type="integer" id="S4.SS0.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S4.SS0.SSS0.Px1.p1.2.m2.1.1">2</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS0.SSS0.Px1.p1.2.m2.1c">2</annotation></semantics></math> (CLA3 and above) achieves slightly worse accuracy/memory tradeoffs than using CLA2 and varying the head dimension, although still Pareto-dominates the tradeoffs possible with plain MQA alone.</p>
</div>
<div id="S4.SS0.SSS0.Px1.p2" class="ltx_para">
<p id="S4.SS0.SSS0.Px1.p2.4" class="ltx_p">At both 1B and 3B scale, we find that for MQA models with typical head sizes of <math id="S4.SS0.SSS0.Px1.p2.1.m1.1" class="ltx_Math" alttext="64" display="inline"><semantics id="S4.SS0.SSS0.Px1.p2.1.m1.1a"><mn id="S4.SS0.SSS0.Px1.p2.1.m1.1.1" xref="S4.SS0.SSS0.Px1.p2.1.m1.1.1.cmml">64</mn><annotation-xml encoding="MathML-Content" id="S4.SS0.SSS0.Px1.p2.1.m1.1b"><cn type="integer" id="S4.SS0.SSS0.Px1.p2.1.m1.1.1.cmml" xref="S4.SS0.SSS0.Px1.p2.1.m1.1.1">64</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS0.SSS0.Px1.p2.1.m1.1c">64</annotation></semantics></math> and <math id="S4.SS0.SSS0.Px1.p2.2.m2.1" class="ltx_Math" alttext="128" display="inline"><semantics id="S4.SS0.SSS0.Px1.p2.2.m2.1a"><mn id="S4.SS0.SSS0.Px1.p2.2.m2.1.1" xref="S4.SS0.SSS0.Px1.p2.2.m2.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="S4.SS0.SSS0.Px1.p2.2.m2.1b"><cn type="integer" id="S4.SS0.SSS0.Px1.p2.2.m2.1.1.cmml" xref="S4.SS0.SSS0.Px1.p2.2.m2.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS0.SSS0.Px1.p2.2.m2.1c">128</annotation></semantics></math>, applying CLA2 yields a <math id="S4.SS0.SSS0.Px1.p2.3.m3.1" class="ltx_math_unparsed" alttext="2\times" display="inline"><semantics id="S4.SS0.SSS0.Px1.p2.3.m3.1a"><mrow id="S4.SS0.SSS0.Px1.p2.3.m3.1b"><mn id="S4.SS0.SSS0.Px1.p2.3.m3.1.1">2</mn><mo lspace="0.222em" id="S4.SS0.SSS0.Px1.p2.3.m3.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S4.SS0.SSS0.Px1.p2.3.m3.1c">2\times</annotation></semantics></math> KV cache reduction while incurring at worst a very modest (less than <math id="S4.SS0.SSS0.Px1.p2.4.m4.1" class="ltx_Math" alttext="1\%" display="inline"><semantics id="S4.SS0.SSS0.Px1.p2.4.m4.1a"><mrow id="S4.SS0.SSS0.Px1.p2.4.m4.1.1" xref="S4.SS0.SSS0.Px1.p2.4.m4.1.1.cmml"><mn id="S4.SS0.SSS0.Px1.p2.4.m4.1.1.2" xref="S4.SS0.SSS0.Px1.p2.4.m4.1.1.2.cmml">1</mn><mo id="S4.SS0.SSS0.Px1.p2.4.m4.1.1.1" xref="S4.SS0.SSS0.Px1.p2.4.m4.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS0.SSS0.Px1.p2.4.m4.1b"><apply id="S4.SS0.SSS0.Px1.p2.4.m4.1.1.cmml" xref="S4.SS0.SSS0.Px1.p2.4.m4.1.1"><csymbol cd="latexml" id="S4.SS0.SSS0.Px1.p2.4.m4.1.1.1.cmml" xref="S4.SS0.SSS0.Px1.p2.4.m4.1.1.1">percent</csymbol><cn type="integer" id="S4.SS0.SSS0.Px1.p2.4.m4.1.1.2.cmml" xref="S4.SS0.SSS0.Px1.p2.4.m4.1.1.2">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS0.SSS0.Px1.p2.4.m4.1c">1\%</annotation></semantics></math> change) degradation in perplexity, and in some cases improving perplexity. We recommend this recipe to practitioners as a conservative change to existing MQA architectures which delivers substantial memory overhead reductions with relatively little risk.</p>
</div>
</section>
<section id="S4.SS0.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Future Work.</h5>

<div id="S4.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S4.SS0.SSS0.Px2.p1.1" class="ltx_p">One natural question that rises from any memory efficient LLM alternative is its efficiency improvement when serving through longer sequences and greater batching.
We leave end-to-end inference efficiency evaluations of large, long-context models employing CLA as an interesting problem for future work.
We suspect that the types of LLMs that will stand to gain the most are the ones which have extremely long sequences, such as models that have long term memory or use methods like Landmark Attention <cite class="ltx_cite ltx_citemacro_citep">(Mohtashami and Jaggi, <a href="#bib.bib29" title="" class="ltx_ref">2023</a>)</cite> which render attention over long contexts more feasible.</p>
</div>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Related Work</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">Transformer memory efficiency can refer to many potential objectives.
It can refer to decreasing memory storage or bandwidth requirements, it can be targeted at training or inference, and finally it apply either within a single pass of the model or between passes.
While this work targets decreasing the size of the inference KV cache that persists between passes, notable works such as
Flash Attention <cite class="ltx_cite ltx_citemacro_citep">(Dao et al., <a href="#bib.bib10" title="" class="ltx_ref">2022</a>; Dao, <a href="#bib.bib9" title="" class="ltx_ref">2023</a>)</cite> have decreased the memory bandwidth necessary for a single pass,
and works like FlexGen <cite class="ltx_cite ltx_citemacro_citep">(Sheng et al., <a href="#bib.bib36" title="" class="ltx_ref">2023</a>)</cite> achieved low memory storage during a single forward pass by partial offloading to disk.
Here we discuss related work that improves the memory efficiency of attention, specifically the memory storage of the KV cache.</p>
</div>
<section id="S5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.1 </span>Decreasing KV cache size Post training</h3>

<div id="S5.SS1.p1" class="ltx_para">
<p id="S5.SS1.p1.1" class="ltx_p">Much work has focused on decreasing the size of the KV cache for models that have already been trained.</p>
</div>
<section id="S5.SS1.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">KV cache compression</h5>

<div id="S5.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="S5.SS1.SSS0.Px1.p1.1" class="ltx_p">As many works have tried to compress LLMs through pruning, quantization, and sparsity,
(see <cite class="ltx_cite ltx_citemacro_citet">Zhu et al. (<a href="#bib.bib49" title="" class="ltx_ref">2023</a>)</cite> for a survey)
a subset directly focus on the problem of KV cache compression.
For quantization,
KVQuant <cite class="ltx_cite ltx_citemacro_citep">(Hooper et al., <a href="#bib.bib18" title="" class="ltx_ref">2024</a>)</cite>
and
Coupled Quantization <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al., <a href="#bib.bib47" title="" class="ltx_ref">2024</a>)</cite>
perform targeted transformations of the keys and values along with non uniform encodings to compress the KV cache to one to two bits.
Sparsifying the KV cache done by works such as
H2O <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al., <a href="#bib.bib48" title="" class="ltx_ref">2023</a>)</cite>
Scissorhands
<cite class="ltx_cite ltx_citemacro_citep">(Liu et al., <a href="#bib.bib24" title="" class="ltx_ref">2023</a>)</cite>
and FastGen <cite class="ltx_cite ltx_citemacro_citep">(Ge et al., <a href="#bib.bib14" title="" class="ltx_ref">2024</a>)</cite>
only store a subset of the KV cache during generation.
They do so by storing only tokens that are near to the generating token or important across the sequence, with the heuristic for importance varying between papers.
Finally, Cachegen <cite class="ltx_cite ltx_citemacro_citep">(Liu et al., <a href="#bib.bib23" title="" class="ltx_ref">2024</a>)</cite> directly compresses the KV cache with a tensor encoder.</p>
</div>
</section>
</section>
<section id="S5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.2 </span>Architectural Changes that decrease KV cache size</h3>

<div id="S5.SS2.p1" class="ltx_para">
<p id="S5.SS2.p1.1" class="ltx_p">Most relevant to our work are methods that change the architecture of the model in order to decrease the size of the KV cache.
These methods can roughly grouped into three categories:
those that try to reduce the number of tokens attended to,
those that replace softmax attention with another operation that requires less memory storage,
and those that, like our work, decrease the unique KV values compared to each query.</p>
</div>
<section id="S5.SS2.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Decreasing effective sequence length</h5>

<div id="S5.SS2.SSS0.Px1.p1" class="ltx_para">
<p id="S5.SS2.SSS0.Px1.p1.1" class="ltx_p">Models that decrease the effective sequence length of the model have existed almost as transformers themselves.
Two notable early works are
Transformer XL <cite class="ltx_cite ltx_citemacro_citep">(Dai et al., <a href="#bib.bib8" title="" class="ltx_ref">2019</a>)</cite> and
Sparse Attention <cite class="ltx_cite ltx_citemacro_citep">(Child et al., <a href="#bib.bib6" title="" class="ltx_ref">2019</a>)</cite>.
Both of them performed attention in local windows of smaller sizes instead of across the whole sequence,
and differed in how they incorporated information from prior tokens.
This line of work was used in many models of note such as GPT3, and is commonly known as sliding window attention.
This line of work has been further developed using methods like
Infini attention <cite class="ltx_cite ltx_citemacro_citep">(Munkhdalai et al., <a href="#bib.bib30" title="" class="ltx_ref">2024</a>)</cite>, which compresses prior tokens using a linear attention mechanism.</p>
</div>
<div id="S5.SS2.SSS0.Px1.p2" class="ltx_para">
<p id="S5.SS2.SSS0.Px1.p2.1" class="ltx_p">An alternative approach is to perform a lookup over prior tokens, such as in
Landmark attention <cite class="ltx_cite ltx_citemacro_citep">(Mohtashami and Jaggi, <a href="#bib.bib29" title="" class="ltx_ref">2023</a>)</cite>
Memorizing transformers <cite class="ltx_cite ltx_citemacro_citep">(Wu et al., <a href="#bib.bib45" title="" class="ltx_ref">2022</a>)</cite>
or to do a lookup over an external datastore, which is commonly known as retrieval <cite class="ltx_cite ltx_citemacro_citep">(Guu et al., <a href="#bib.bib17" title="" class="ltx_ref">2020</a>; Izacard et al., <a href="#bib.bib20" title="" class="ltx_ref">2024</a>; Borgeaud et al., <a href="#bib.bib5" title="" class="ltx_ref">2022</a>)</cite>.
However, while these methods reduce computation, they do not reduce storage unless the KVs for lookup are offloaded from GPU memory.</p>
</div>
</section>
<section id="S5.SS2.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Removing Softmax Attention</h5>

<div id="S5.SS2.SSS0.Px2.p1" class="ltx_para">
<p id="S5.SS2.SSS0.Px2.p1.1" class="ltx_p">Replacements to Softmax Attention are often referred to as SSMs or linear attention computations.
Regardless of name, most replace the attention component with an alternative which has constant space complexity during token generation w.r.t. the number of tokens generated.
This also reduces the time complexity of generation to be linear w.r.t. the number of tokens, instead of quadratic.
Various methods differ in how they parameterize their state.
<cite class="ltx_cite ltx_citemacro_citet">Katharopoulos et al. (<a href="#bib.bib22" title="" class="ltx_ref">2020</a>); Wang et al. (<a href="#bib.bib44" title="" class="ltx_ref">2020</a>)</cite>
proposed initial versions of linear attention.
Recent work has focused on using data dependent mechanisms to improve the state, such as in
GLA <cite class="ltx_cite ltx_citemacro_citep">(Yang et al., <a href="#bib.bib46" title="" class="ltx_ref">2024</a>)</cite> Mamba<cite class="ltx_cite ltx_citemacro_citep">(Gu and Dao, <a href="#bib.bib16" title="" class="ltx_ref">2023</a>)</cite>
and RWKV v6 <cite class="ltx_cite ltx_citemacro_citep">(Peng et al., <a href="#bib.bib32" title="" class="ltx_ref">2024</a>)</cite>.</p>
</div>
</section>
<section id="S5.SS2.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Groupings of Attention</h5>

<div id="S5.SS2.SSS0.Px3.p1" class="ltx_para">
<p id="S5.SS2.SSS0.Px3.p1.1" class="ltx_p">Most related are methods that use softmax attention but attempt to use a single KV pair for multiple queries.
GQA <cite class="ltx_cite ltx_citemacro_citep">(Ainslie et al., <a href="#bib.bib1" title="" class="ltx_ref">2023a</a>)</cite> and MQA <cite class="ltx_cite ltx_citemacro_citep">(Shazeer, <a href="#bib.bib34" title="" class="ltx_ref">2019</a>)</cite> do this by grouping keys and values across heads.</p>
</div>
<div id="S5.SS2.SSS0.Px3.p2" class="ltx_para">
<p id="S5.SS2.SSS0.Px3.p2.1" class="ltx_p">Concurrent work tries other strategies to share values between layers.
Deepseek-V2 <cite class="ltx_cite ltx_citemacro_citep">(DeepSeek-AI, <a href="#bib.bib11" title="" class="ltx_ref">2024</a>)</cite> proposes Multi Latent Attention, which uses a low rank projection of the keys and values.
You Only Cache Once <cite class="ltx_cite ltx_citemacro_citep">(Sun et al., <a href="#bib.bib40" title="" class="ltx_ref">2024</a>)</cite> splits the model into two halves. The first half performs local attention and then generates a set of keys and values that are used for global attention across all layers in the second half.</p>
</div>
</section>
</section>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Conclusion</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">Cross-Layer Attention is an effective method for reducing the KV cache memory storage footprint of transformer models by a factor of <math id="S6.p1.1.m1.1" class="ltx_math_unparsed" alttext="2\times" display="inline"><semantics id="S6.p1.1.m1.1a"><mrow id="S6.p1.1.m1.1b"><mn id="S6.p1.1.m1.1.1">2</mn><mo lspace="0.222em" id="S6.p1.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S6.p1.1.m1.1c">2\times</annotation></semantics></math> with roughly equal perplexity.
Based on extensive experimental evaluation against well-tuned baselines at both the 1B- and 3B-parameter scales, we find that CLA advances the Pareto frontier for memory-efficient transformers.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ainslie et al. [2023a]</span>
<span class="ltx_bibblock">
J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebron, and S. Sanghai.

</span>
<span class="ltx_bibblock">GQA: Training generalized multi-query transformer models from multi-head checkpoints.

</span>
<span class="ltx_bibblock">In H. Bouamor, J. Pino, and K. Bali, editors, <em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing</em>, pages 4895–4901, Singapore, Dec. 2023a. Association for Computational Linguistics.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2023.emnlp-main.298</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/2023.emnlp-main.298" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/2023.emnlp-main.298</a>.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ainslie et al. [2023b]</span>
<span class="ltx_bibblock">
J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai.

</span>
<span class="ltx_bibblock">Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023b.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ansel et al. [2024]</span>
<span class="ltx_bibblock">
J. Ansel, E. Yang, H. He, N. Gimelshein, A. Jain, M. Voznesensky, B. Bao, P. Bell, D. Berard, E. Burovski, G. Chauhan, A. Chourdia, W. Constable, A. Desmaison, Z. DeVito, E. Ellison, W. Feng, J. Gong, M. Gschwind, B. Hirsh, S. Huang, K. Kalambarkar, L. Kirsch, M. Lazos, M. Lezcano, Y. Liang, J. Liang, Y. Lu, C. Luk, B. Maher, Y. Pan, C. Puhrsch, M. Reso, M. Saroufim, M. Y. Siraichi, H. Suk, M. Suo, P. Tillet, E. Wang, X. Wang, W. Wen, S. Zhang, X. Zhao, K. Zhou, R. Zou, A. Mathews, G. Chanan, P. Wu, and S. Chintala.

</span>
<span class="ltx_bibblock">PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation.

</span>
<span class="ltx_bibblock">In <em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS ’24)</em>. ACM, Apr. 2024.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1145/3620665.3640366</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://pytorch.org/assets/pytorch2-2.pdf" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://pytorch.org/assets/pytorch2-2.pdf</a>.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Black et al. [2022]</span>
<span class="ltx_bibblock">
S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, J. Phang, M. Pieler, U. S. Prashanth, S. Purohit, L. Reynolds, J. Tow, B. Wang, and S. Weinbach.

</span>
<span class="ltx_bibblock">Gpt-neox-20b: An open-source autoregressive language model, 2022.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Borgeaud et al. [2022]</span>
<span class="ltx_bibblock">
S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. Van Den Driessche, J.-B. Lespiau, B. Damoc, A. Clark, D. De Las Casas, A. Guy, J. Menick, R. Ring, T. Hennigan, S. Huang, L. Maggiore, C. Jones, A. Cassirer, A. Brock, M. Paganini, G. Irving, O. Vinyals, S. Osindero, K. Simonyan, J. Rae, E. Elsen, and L. Sifre.

</span>
<span class="ltx_bibblock">Improving language models by retrieving from trillions of tokens.

</span>
<span class="ltx_bibblock">In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, <em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 39th International Conference on Machine Learning</em>, volume 162 of <em id="bib.bib5.2.2" class="ltx_emph ltx_font_italic">Proceedings of Machine Learning Research</em>, pages 2206–2240. PMLR, 17–23 Jul 2022.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://proceedings.mlr.press/v162/borgeaud22a.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.mlr.press/v162/borgeaud22a.html</a>.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Child et al. [2019]</span>
<span class="ltx_bibblock">
R. Child, S. Gray, A. Radford, and I. Sutskever.

</span>
<span class="ltx_bibblock">Generating long sequences with sparse transformers.

</span>
<span class="ltx_bibblock"><em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/1904.10509, 2019.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://arxiv.org/abs/1904.10509" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://arxiv.org/abs/1904.10509</a>.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chowdhery et al. [2022]</span>
<span class="ltx_bibblock">
A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur-Ari, P. Yin, T. Duke, A. Levskaya, S. Ghemawat, S. Dev, H. Michalewski, X. Garcia, V. Misra, K. Robinson, L. Fedus, D. Zhou, D. Ippolito, D. Luan, H. Lim, B. Zoph, A. Spiridonov, R. Sepassi, D. Dohan, S. Agrawal, M. Omernick, A. M. Dai, T. S. Pillai, M. Pellat, A. Lewkowycz, E. Moreira, R. Child, O. Polozov, K. Lee, Z. Zhou, X. Wang, B. Saeta, M. Diaz, O. Firat, M. Catasta, J. Wei, K. Meier-Hellstern, D. Eck, J. Dean, S. Petrov, and N. Fiedel.

</span>
<span class="ltx_bibblock">Palm: Scaling language modeling with pathways, 2022.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dai et al. [2019]</span>
<span class="ltx_bibblock">
Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. Le, and R. Salakhutdinov.

</span>
<span class="ltx_bibblock">Transformer-XL: Attentive language models beyond a fixed-length context.

</span>
<span class="ltx_bibblock">In A. Korhonen, D. Traum, and L. Màrquez, editors, <em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics</em>, pages 2978–2988, Florence, Italy, July 2019. Association for Computational Linguistics.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/P19-1285</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/P19-1285" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/P19-1285</a>.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dao [2023]</span>
<span class="ltx_bibblock">
T. Dao.

</span>
<span class="ltx_bibblock">Flashattention-2: Faster attention with better parallelism and work partitioning, 2023.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dao et al. [2022]</span>
<span class="ltx_bibblock">
T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré.

</span>
<span class="ltx_bibblock">FlashAttention: Fast and memory-efficient exact attention with IO-awareness.

</span>
<span class="ltx_bibblock">In <em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 2022.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">DeepSeek-AI [2024]</span>
<span class="ltx_bibblock">
DeepSeek-AI.

</span>
<span class="ltx_bibblock">Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gao et al. [2024]</span>
<span class="ltx_bibblock">
B. Gao, Z. He, P. Sharma, Q. Kang, D. Jevdjic, J. Deng, X. Yang, Z. Yu, and P. Zuo.

</span>
<span class="ltx_bibblock">Attentionstore: Cost-effective attention reuse across multi-turn conversations in large language model serving, 2024.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gao et al. [2023]</span>
<span class="ltx_bibblock">
L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou.

</span>
<span class="ltx_bibblock">A framework for few-shot language model evaluation, 12 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://zenodo.org/records/10256836" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://zenodo.org/records/10256836</a>.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ge et al. [2024]</span>
<span class="ltx_bibblock">
S. Ge, Y. Zhang, L. Liu, M. Zhang, J. Han, and J. Gao.

</span>
<span class="ltx_bibblock">Model tells you what to discard: Adaptive kv cache compression for llms, 2024.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Google [2024]</span>
<span class="ltx_bibblock">
Google.

</span>
<span class="ltx_bibblock">Context caching guide.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://ai.google.dev/gemini-api/docs/caching" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://ai.google.dev/gemini-api/docs/caching</a>, 2024.

</span>
<span class="ltx_bibblock">Accessed: 2024-05-20.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gu and Dao [2023]</span>
<span class="ltx_bibblock">
A. Gu and T. Dao.

</span>
<span class="ltx_bibblock">Mamba: Linear-time sequence modeling with selective state spaces, 2023.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Guu et al. [2020]</span>
<span class="ltx_bibblock">
K. Guu, K. Lee, Z. Tung, P. Pasupat, and M.-W. Chang.

</span>
<span class="ltx_bibblock">Realm: retrieval-augmented language model pre-training.

</span>
<span class="ltx_bibblock">In <em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 37th International Conference on Machine Learning</em>, ICML’20. JMLR.org, 2020.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hooper et al. [2024]</span>
<span class="ltx_bibblock">
C. Hooper, S. Kim, H. Mohammadzadeh, M. W. Mahoney, Y. S. Shao, K. Keutzer, and A. Gholami.

</span>
<span class="ltx_bibblock">Kvquant: Towards 10 million context length llm inference with kv cache quantization, 2024.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Huang et al. [2019]</span>
<span class="ltx_bibblock">
Y. Huang, Y. Cheng, A. Bapna, O. Firat, M. X. Chen, D. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wu, and Z. Chen.

</span>
<span class="ltx_bibblock"><em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">GPipe: efficient training of giant neural networks using pipeline parallelism</em>.

</span>
<span class="ltx_bibblock">Curran Associates Inc., Red Hook, NY, USA, 2019.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Izacard et al. [2024]</span>
<span class="ltx_bibblock">
G. Izacard, P. Lewis, M. Lomeli, L. Hosseini, F. Petroni, T. Schick, J. Dwivedi-Yu, A. Joulin, S. Riedel, and E. Grave.

</span>
<span class="ltx_bibblock">Atlas: few-shot learning with retrieval augmented language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">J. Mach. Learn. Res.</em>, 24(1), mar 2024.

</span>
<span class="ltx_bibblock">ISSN 1532-4435.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kalamkar et al. [2019]</span>
<span class="ltx_bibblock">
D. Kalamkar, D. Mudigere, N. Mellempudi, D. Das, K. Banerjee, S. Avancha, D. T. Vooturi, N. Jammalamadaka, J. Huang, H. Yuen, J. Yang, J. Park, A. Heinecke, E. Georganas, S. Srinivasan, A. Kundu, M. Smelyanskiy, B. Kaul, and P. Dubey.

</span>
<span class="ltx_bibblock">A study of bfloat16 for deep learning training, 2019.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Katharopoulos et al. [2020]</span>
<span class="ltx_bibblock">
A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret.

</span>
<span class="ltx_bibblock">Transformers are rnns: fast autoregressive transformers with linear attention.

</span>
<span class="ltx_bibblock">In <em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 37th International Conference on Machine Learning</em>, ICML’20. JMLR.org, 2020.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. [2024]</span>
<span class="ltx_bibblock">
Y. Liu, H. Li, Y. Cheng, S. Ray, Y. Huang, Q. Zhang, K. Du, J. Yao, S. Lu, G. Ananthanarayanan, M. Maire, H. Hoffmann, A. Holtzman, and J. Jiang.

</span>
<span class="ltx_bibblock">Cachegen: Kv cache compression and streaming for fast language model serving, 2024.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. [2023]</span>
<span class="ltx_bibblock">
Z. Liu, A. Desai, F. Liao, W. Wang, V. Xie, Z. Xu, A. Kyrillidis, and A. Shrivastava.

</span>
<span class="ltx_bibblock">Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time, 2023.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Loshchilov and Hutter [2017]</span>
<span class="ltx_bibblock">
I. Loshchilov and F. Hutter.

</span>
<span class="ltx_bibblock">SGDR: Stochastic gradient descent with warm restarts.

</span>
<span class="ltx_bibblock">In <em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>, 2017.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/forum?id=Skq89Scxx" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=Skq89Scxx</a>.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Loshchilov and Hutter [2019]</span>
<span class="ltx_bibblock">
I. Loshchilov and F. Hutter.

</span>
<span class="ltx_bibblock">Decoupled weight decay regularization.

</span>
<span class="ltx_bibblock">In <em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>, 2019.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/forum?id=Bkg6RiCqY7" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=Bkg6RiCqY7</a>.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Merity et al. [2016]</span>
<span class="ltx_bibblock">
S. Merity, C. Xiong, J. Bradbury, and R. Socher.

</span>
<span class="ltx_bibblock">Pointer sentinel mixture models, 2016.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Micikevicius et al. [2018]</span>
<span class="ltx_bibblock">
P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu.

</span>
<span class="ltx_bibblock">Mixed precision training, 2018.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mohtashami and Jaggi [2023]</span>
<span class="ltx_bibblock">
A. Mohtashami and M. Jaggi.

</span>
<span class="ltx_bibblock">Random-access infinite context length for transformers.

</span>
<span class="ltx_bibblock">In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, <em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, volume 36, pages 54567–54585. Curran Associates, Inc., 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://proceedings.neurips.cc/paper_files/paper/2023/file/ab05dc8bf36a9f66edbff6992ec86f56-Paper-Conference.pdf" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.neurips.cc/paper_files/paper/2023/file/ab05dc8bf36a9f66edbff6992ec86f56-Paper-Conference.pdf</a>.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Munkhdalai et al. [2024]</span>
<span class="ltx_bibblock">
T. Munkhdalai, M. Faruqui, and S. Gopal.

</span>
<span class="ltx_bibblock">Leave no context behind: Efficient infinite context transformers with infini-attention, 2024.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Paszke et al. [2019]</span>
<span class="ltx_bibblock">
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala.

</span>
<span class="ltx_bibblock">Pytorch: An imperative style, high-performance deep learning library, 2019.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Peng et al. [2024]</span>
<span class="ltx_bibblock">
B. Peng, D. Goldstein, Q. Anthony, A. Albalak, E. Alcaide, S. Biderman, E. Cheah, X. Du, T. Ferdinan, H. Hou, P. Kazienko, K. K. GV, J. Kocoń, B. Koptyra, S. Krishna, R. M. J. au2, N. Muennighoff, F. Obeid, A. Saito, G. Song, H. Tu, S. Woźniak, R. Zhang, B. Zhao, Q. Zhao, P. Zhou, J. Zhu, and R.-J. Zhu.

</span>
<span class="ltx_bibblock">Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence, 2024.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ramachandran et al. [2017]</span>
<span class="ltx_bibblock">
P. Ramachandran, B. Zoph, and Q. V. Le.

</span>
<span class="ltx_bibblock">Searching for activation functions, 2017.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shazeer [2019]</span>
<span class="ltx_bibblock">
N. Shazeer.

</span>
<span class="ltx_bibblock">Fast transformer decoding: One write-head is all you need, 2019.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shazeer [2020]</span>
<span class="ltx_bibblock">
N. Shazeer.

</span>
<span class="ltx_bibblock">Glu variants improve transformer, 2020.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sheng et al. [2023]</span>
<span class="ltx_bibblock">
Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. Ré, I. Stoica, and C. Zhang.

</span>
<span class="ltx_bibblock">Flexgen: high-throughput generative inference of large language models with a single gpu.

</span>
<span class="ltx_bibblock">In <em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 40th International Conference on Machine Learning</em>, ICML’23. JMLR.org, 2023.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shoeybi et al. [2020]</span>
<span class="ltx_bibblock">
M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro.

</span>
<span class="ltx_bibblock">Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Soboleva et al. [2023]</span>
<span class="ltx_bibblock">
D. Soboleva, F. Al-Khateeb, R. Myers, J. R. Steeves, J. Hestness, and N. Dey.

</span>
<span class="ltx_bibblock">SlimPajama: A 627B token cleaned and deduplicated version of RedPajama.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama</a>, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://huggingface.co/datasets/cerebras/SlimPajama-627B" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/datasets/cerebras/SlimPajama-627B</a>.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Su et al. [2023]</span>
<span class="ltx_bibblock">
J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, and Y. Liu.

</span>
<span class="ltx_bibblock">Roformer: Enhanced transformer with rotary position embedding, 2023.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sun et al. [2024]</span>
<span class="ltx_bibblock">
Y. Sun, L. Dong, Y. Zhu, S. Huang, W. Wang, S. Ma, Q. Zhang, J. Wang, and F. Wei.

</span>
<span class="ltx_bibblock">You only cache once: Decoder-decoder architectures for language models, 2024.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. [2023]</span>
<span class="ltx_bibblock">
H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom.

</span>
<span class="ltx_bibblock">Llama 2: Open foundation and fine-tuned chat models, 2023.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Vaswani et al. [2017]</span>
<span class="ltx_bibblock">
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin.

</span>
<span class="ltx_bibblock">Attention is all you need.

</span>
<span class="ltx_bibblock">In <em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, volume 30. Curran Associates, Inc., 2017.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf</a>.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. [2019]</span>
<span class="ltx_bibblock">
C. Wang, K. Cho, and J. Gu.

</span>
<span class="ltx_bibblock">Neural machine translation with byte-level subwords, 2019.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. [2020]</span>
<span class="ltx_bibblock">
S. Wang, B. Z. Li, M. Khabsa, H. Fang, and H. Ma.

</span>
<span class="ltx_bibblock">Linformer: Self-attention with linear complexity.

</span>
<span class="ltx_bibblock"><em id="bib.bib44.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2006.04768, 2020.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2006.04768" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2006.04768</a>.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wu et al. [2022]</span>
<span class="ltx_bibblock">
Y. Wu, M. N. Rabe, D. Hutchins, and C. Szegedy.

</span>
<span class="ltx_bibblock">Memorizing transformers.

</span>
<span class="ltx_bibblock">In <em id="bib.bib45.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>, 2022.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/forum?id=TrjbxzRcnf-" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=TrjbxzRcnf-</a>.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yang et al. [2024]</span>
<span class="ltx_bibblock">
S. Yang, B. Wang, Y. Shen, R. Panda, and Y. Kim.

</span>
<span class="ltx_bibblock">Gated linear attention transformers with hardware-efficient training, 2024.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. [2024]</span>
<span class="ltx_bibblock">
T. Zhang, J. Yi, Z. Xu, and A. Shrivastava.

</span>
<span class="ltx_bibblock">Kv cache is 1 bit per channel: Efficient large language model inference with coupled quantization, 2024.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. [2023]</span>
<span class="ltx_bibblock">
Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett, Z. A. Wang, and B. Chen.

</span>
<span class="ltx_bibblock">H2o: Heavy-hitter oracle for efficient generative inference of large language models.

</span>
<span class="ltx_bibblock">In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, <em id="bib.bib48.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, volume 36, pages 34661–34710. Curran Associates, Inc., 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://proceedings.neurips.cc/paper_files/paper/2023/file/6ceefa7b15572587b78ecfcebb2827f8-Paper-Conference.pdf" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.neurips.cc/paper_files/paper/2023/file/6ceefa7b15572587b78ecfcebb2827f8-Paper-Conference.pdf</a>.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhu et al. [2023]</span>
<span class="ltx_bibblock">
X. Zhu, J. Li, Y. Liu, C. Ma, and W. Wang.

</span>
<span class="ltx_bibblock">A survey on model compression for large language models, 2023.

</span>
</li>
</ul>
</section>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Learning rate sweeps</h2>

<div id="A1.p1" class="ltx_para">
<p id="A1.p1.1" class="ltx_p">Here we present the results of our learning rate sweeps at the 1B and 3B parameter scales:</p>
</div>
<figure id="A1.F4" class="ltx_figure"><img src="/html/2405.12981/assets/x4.png" id="A1.F4.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="260" height="188" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>3B Learning Rate Sweep</figcaption>
</figure>
<figure id="A1.F5" class="ltx_figure"><img src="/html/2405.12981/assets/x5.png" id="A1.F5.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="260" height="188" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 5: </span>1B Learning Rate Sweep</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2405.12980" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2405.12981" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2405.12981">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2405.12981" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2405.12982" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Wed Jun  5 13:37:24 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2405.12981
Key-value (KV)  caching plays an essential role in accelerating decoding for transformer-based autoregressive large language models (LLMs) . However, the amount of memory required to store the KV cache can become prohibitive at long sequence lengths and large batch sizes. Since the invention of the transformer, two of the most effective interventions discovered for reducing the size of the KV cache have been Multi-Query Attention (MQA)  and its generalization, Grouped-Query Attention (GQA) . MQA and GQA both modify the design of the attention block so that multiple query heads can share a single key/value head, reducing the number of distinct key/value heads by a large factor while only minimally degrading accuracy. In this paper, we show that it is possible to take Multi-Query Attention a step further by also sharing key and value heads between adjacent layers, yielding a new attention design we call Cross-Layer Attention (CLA) . With CLA, we find that it is possible to reduce the size of the KV cache by anotherwhile maintaining nearly the same accuracy as unmodified MQA. In experiments training 1B- and 3B-parameter models from scratch, we demonstrate that CLA provides a Pareto improvement over the memory/accuracy tradeoffs which are possible with traditional MQA, enabling inference with longer sequence lengths and larger batch sizes than would otherwise be possible.
abstract
Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
Abstract
2405.12981
1IntroductionThe memory footprint of the key-value (KV)  cache can be a bottleneck when serving large language models (LLMs) . Because the size of the KV cache scales proportionally with both sequence length and batch size, the memory overhead of KV cache storage can limit batch sizes when operating on long sequence lengths, and can require employing costly techniques like offloading when on-device memory is scarce. It is also desirable to be able to persist KV caches over long periods of time in order to minimize redundant computations. However, the size of the KV cache directly determines the cost of storing and retrieving such persistent caches. As new applications of LLMs emerge which demand ever-longer sequence lengths, the memory footprint of the KV cache is becoming an increasingly important consideration in the design of efficient transformer-based language models.Existing work has proposed a variety of methods for decreasing the memory footprint of the KV cache, including storing KV activations in low precision, evicting unimportant KV cache entries, and sharing keys and values across query heads in the attention mechanism.In this paper, we introduce a method for reducing the size of the KV cache along a dimension different than those explored in prior work: namely, reducing the number of uniquelayersin the KV cache. Our contributions are as follows:1.We proposeCross-Layer Attention(CLA) , a modification to the transformer architecture which reduces the size of the KV cache bysharing KV activations across layers.2.We conduct extensive pretraining experiments to characterize the effect of different CLA configurations on accuracy and memory usage across a range of architectural hyperparameters, learning rates and model sizes.3.We demonstrate that CLA enables accuracy/memory Pareto improvements relative to existing Multi-Query Attention (MQA)  and Grouped-Query Attention (GQA)  architectures.4.In particular, we demonstrate at the 1B- and 3B-parameter scales that combining CLA with MQA can achieve areduction in KV cache size versus a plain MQA baseline, with minimal degradation in perplexity.5.We offer guidance on which CLA configurations perform best based on our experiments, finding that CLA should be used between pairs of consecutive layers, and that CLA appears to deliver the most robust benefits when used in conjunction with MQA.
introduction
Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
1Introduction
2405.12981
2Cross-Layer AttentionIn this section we describe our Cross-Layer Attention (CLA)  technique, and its relationship to the KV-sharing mechanisms employed by the existing Multi-Query and Grouped-Query attention architectures (MQA and GQA) .Figure 1:Schematic of two consecutive layers in a transformer using a traditional attention design (left)  and in a transformer using Cross-Layer Attention (right) .
When using traditional attention, each layer computes its own separateandactivations, which must be cached on a per-layer basis during autoregressive decoding.
When using Cross-Layer Attention, some layers compute their own freshandactivations, while other layers reuse theandactivations of earlier layers.Figure 2:Schematic of KV cache structures under different attention configurations in a 10-layer transformer. Using traditional attention, each layer has its own KV cache. Using Cross-Layer Attention with a sharing factor of(CLA2) , every group ofconsecutive layers shares a single KV cache. Using Cross-Layer Attention with a sharing factor of(CLA3) , every group ofconsecutive layers shares a single KV cache. When the sharing factor does not evenly divide the number of layers, as in the CLA3 example, some KV caches must be shared over fewer layers than others; in this CLA3 configuration, we arbitrarily select the layer 0 KV cache to be used only in layer 0.2.1Background: Multi-Query Attention and Grouped-Query AttentionThe original transformer architecture employed Multi-Head Attention (MHA) , in which each query head attends over the keys and values produced by a distinct key/value head. In MHA, the KV activations of each key/value head must be stored separately in the KV cache, resulting in a storage overhead ofelements per token, whereis the number of query heads andis the embedding dimension of each head.To reduce the overhead associated with storing and accessing the KV cache during transformer decoding,proposedMulti-Query Attention(MQA) , whichlater generalized toGrouped-Query Attention(GQA) . Grouped-Query Attention modifies the transformer architecture by organizing the query heads of each attention layer into groups, where each group of query heads shares a single key/value head. Because the size of the KV cache scales only with the number of distinct key/value heads, not the number of query heads, GQA reduces the storage overhead of the KV cache to, wheredenotes the number of groups for GQA and. MQA can be seen as the special case of GQA in which.andfind that MQA and GQA enable significant reductions in KV cache size and decoding latency while incurring only a small degradation in accuracy compared to MHA architectures with the same head dimension. The family of attention architectures enabled by using MQA and GQA defines anaccuracy/memory tradeoff spacein which model designers can choose how they want to balance the expressive power and KV cache overhead of their attention mechanism. MQA and GQA stake out different positions in this tradeoff space, and neither is necessarily preferable to the other for all use cases.2.2Sharing KV Activations Across LayersInspired by the success of MQA and GQA, which share key/value heads across query heads within a single layer, we propose also sharing key/value headsacross layers. We refer to such an attention architecture asCross-Layer Attention(CLA) , and present a diagrammatic view of it in Figure1. CLA computes key/value projections for only a subset of layers in the model; the attention blocks in layers without key/value projections reuse the KV activations of previous layers. Only the subset of layers with key/value projections contribute to the KV cache, allowing a reduction in memory footprint relative to traditional architectures which apply a separate key/value projection in each layer.CLA is orthogonal to MQA/GQA/MHA, and can be combined with any of them. Moreover, in the same way that GQA allows varyingto access a family of different attention configurations, CLA allows varying the number of layers which share the output of each KV projection, which we refer to as thesharing factor. We refer to different configurations of CLA by their sharing factors, giving rise to CLA2, which shares each KV projection among a pair of adjacent layers, CLA3, which shares each KV projection among a group oflayers, and so on. We present a visualization of different attention configurations possible with CLA in Figure2.2.3Implications for System DesignCLA is primarily an intervention to reduce the memory footprint of the KV cache, and only has minor effects on other resources consumed by the model during training and inference. Here, we summarize the effect of CLA on key metrics relevant from a systems engineering perspective, assuming all other architectural hyperparameters are held constant:•KV Cache Memory: CLA significantly reduces KV cache memory footprint, shrinking it by a factor equal to the sharing factor, or slightly less if the sharing factor does not evenly divide the number of layers.•Training Memory Footprint: CLA reduces the memory footprint of intermediate KV activation tensors materialized during training, although for GQA and MQA models such KV tensors are typically small compared to the model’s hidden states and MLP activations.•Model Parallelism: CLA is fully compatible with standard tensor parallelism techniquesfor sharding model weights across multiple accelerators. In the presence of pipeline parallelism, either different layers which share a KV cache must be kept in the same pipeline stage, or else KV activations must be communicated between pipeline stages.•Parameters and FLOPs: Because CLA reduces the total number of key/value projection blocks in the model, CLA slightly reduces the number of parameters in the model and the number of FLOPs required during a forward or backward pass.•Decoding Latency: In the context of a full LLM serving stack, CLA can enable larger batch sizes and longer KV cache persistence times than would otherwise be possible, which have the potential to improve inference latency.•Core Attention Latency: Unlike MQA and GQA, CLA has no direct effect on the memory bandwidth consumed by the attention mechanism in each decoding step, because even shared KV cache layers must be separately re-read from main memory in each attention layer. CLA therefore has no direct effect on the latency of the core attention computation during decoding.
body
Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
2Cross-Layer Attention
2405.12981
3Pretraining ExperimentsFigure 3:The accuracy/memory Pareto frontier discovered in our 1B-scale design space exploration, for models with CLA (red)  and without CLA (blue) . Lower is better on both axes.ModelQueryHeadsKVHeadsKVLayersKV Bytes PerToken (16-Bit) ValidationPerplexityNon-CLA BaselinesH128-MHAH128-GQA4H128-GQA2H128-MQAH64-MQAH46-MQAH32-MQAMQA + CLA2 ModelsH512-MQA-CLA2H256-MQA-CLA2H128-MQA-CLA2H90-MQA-CLA2H64-MQA-CLA2GQA + CLA2 ModelsH256-GQA4-CLA2H128-GQA4-CLA2H128-GQA2-CLA2MQA + CLAModelsH128-MQA-CLA3H128-MQA-CLA4MQA + CLA2, Non-Uniform SharingH128-MQA-CLA2-KeepEndsH128-MQA-CLA2-DenseFrontH128-MQA-CLA2-DenseBackTable 1:Results of our 1B-scale design space exploration.To determine the effect of Cross-Layer Attention on language modeling accuracy, we trained a collection of transformer-based language models from scratch at thebillion andbillion parameter scales.
While running these experiments, we sought to answer the following questions:1.What accuracy/memory tradeoffs are possible using CLA?2.How does using CLA compare to using plain GQA or MQA?3.How does CLA interact with GQA and MQA?4.What CLA configurations perform best given a fixed memory budget?5.Are the effects of CLA consistent across scales?We found that CLA enables favorable accuracy/memory tradeoffs compared to what is possible using plain GQA or MQA.
Moreover, we found that in our experimental regime, a sharing factor ofis more effective than other sharing factors, and that CLA is consistently effective when combined with MQA when trying to decrease KV cache storage.
We also found preliminary evidence to suggest that CLA models benefit from training with higher learning rates than comparable non-CLA models.
Finally, we found that CLA confers benefits at both 1B- and 3B-parameter scales.
In the rest of this section, we present our experimental setup and results in more detail.3.1Common Experimental ParametersIn all our experiments, we train our models from scratch on data from the SlimPajamadataset, tokenized with the GPT-NeoX tokenizerwhich uses Byte-Pair Encoding (BPE) .
We adopt a Llama-likearchitecture with pre-normalization, SwiGLU activations, and rotary position embeddings.
We do not use dropout for any of our models. Our models include learnable elementwise affine parameters for layer-norm, and our CLA models use separately-learnable affine layer-norm parameters for the KV projection blocks and Q projection blocks in attention. Unless otherwise stated, we always set the number of query headssuch thatis equal to the hidden size.We train all models using the AdamW optimizerwith gradient clipping, using,, a weight decay factor of, and a clipping norm of.
We use a linear learning rate warmup for the firstof training examples and a cosine learning rate scheduledecaying toof the peak learning rate over the remainder of training.
We set the sequence length totokens and the batch size tosequences, for a total oftokens per training step.
All our experiments initialize the weights of linear layers from a normal distribution with mean zero and standard deviation.We perform all experiments on NVIDIA H100 GPUs using PyTorch. We use mixed precision trainingin BF16with gradient all-reduce and gradient accumulation in FP32 for training stability.3.2Experiments at 1B-Parameter ScaleModel FamilyHidden SizeFFN SizeLayersSequence LengthTraining Tokens1B Models3B ModelsHelloTable 2:Architectural and training hyperparameters shared across our pretraining experiments.We trained all our 1B-scale models on 30 billion tokens using a consistent data order, and, other than varying the attention mechanism, used the same architectural hyperparameters across all 1B-scale models.
This means that all our 1B models were all trained using approximately the same number of FLOPs and approximately the same number of GPU-hours, with CLA models requiring slightly fewer FLOPs to train than their non-CLA counterparts due to the reduced number of key/value projections.
The common hyperparameters shared across our 1B-scale experiments can be found in Table2.We ran two main sets of experiments at the 1B-parameter scale. First, we trained a diverse set of CLA and non-CLA models to characterize the range of accuracy/memory tradeoffs achievable with and without CLA, and to determine which CLA configurations are most effective; we refer to these as ourdesign space explorationexperiments, and describe them in more detail in Section3.2.1.
Second, we conducted a learning rate sweep on a subset of models from our design space exploration to verify that our results continue to hold even against a strong non-CLA baseline with a well-tuned learning rate.
We describe these learning rate tuning experiments in AppendixA.3.2.1Design Space ExplorationThe primary goal of our 1B-parameter-scale design space exploration was to characterize the Pareto frontier of accuracy/memory tradeoffs achievable with and without CLA, and to determine which CLA configurations achieve the best accuracy on a fixed KV cache memory budget.
We train all models in our design space exploration using a learning rate of, which we determined to be conservative; we explore the effect of the learning rate on accuracy in more detail in Section3.2.2.For our design space exploration, we first trained a collection of seven non-CLA baseline models along the MHA-GQA-MQA spectrum, exhibiting a range of KV cache memory requirements spanning two orders of magnitude.
Our baseline model with the largest KV cache memory footprint is an MHA model with a head embedding dimension of(bytes per token at 16-bit precision) , and our baseline with the smallest footprint is an MQA model with head dimension(bytes per token) .We quantify the accuracy of models in our design space exploration using perplexity on a held-out validation set oftokens drawn from our SlimPajama corpus.
A summary of results for the models in our design space exploration, including our baseline models, can be found in Table1. We adopt the naming scheme “H-” for all models in our experiments, so that, for example, a model employing MQA with a head dimension ofwould be named “H64-MQA.”
For our baseline models, we observe that validation perplexity increases monotonically as we reduce the memory capacity of the KV cache, ranging from a perplexity offor our H128-MHA baseline tofor our H32-MQA baseline.In the rest of this section, we present results for the CLA models we trained during our design space exploration.Best Performance: MQA + CLA2.We trained a family of five models combining MQA with CLA2.
We varied the head dimension for our MQA-CLA2 models fromdown to, allowing us to compare to a range of non-CLA baseline models with varying KV cache capacities.We found that our MQA-CLA2 models are able to achieve better perplexities than baseline models requiring the same amount of KV cache memory, advancing the accuracy/memory Pareto frontier.
We present a plot of the accuracy/memory Pareto frontier with and without CLA in Figure3. Our MQA-CLA2 models with head dimensionsare able to match the KV cache memory footprint of baseline MQA models with head dimensionswhile achieving substantial perplexity improvements in the range of–points. Additionally, our MQA-CLA2 models with large head sizes ofare able to match the KV cache footprint of our MQA and GQA2 baselines withwhile achieving a small perplexity improvement ofpoints.We found that our MQA-CLA2 models achieved the best accuracy/memory tradeoffs among all CLA configurations we tested in our design space exploration. In the rest of this section, we briefly describe the ablations we conducted to explore alternate CLA configurations.Ablation: GQA + CLA2.We trained three models to explore combining GQA with CLA2.
We chose GQA4-CLA2 withas our starting point, as GQA4 represents an attention configuration intermediate between our MQA and MHA baselines. We then explored expanding the head dimension of our GQA4-CLA2 model to, as well as reducing the GQA factor to GQA2. We found that only the GQA2-CLA2 configuration was able to achieve a perplexity better than the corresponding baseline model with the same KV cache footprint, and that this perplexity was the same (withinpoints)  as our MQA-CLA2 model with the same footprint.Ablation: MQA + CLA with Sharing Factor.To explore the effect of using CLA sharing factors, we trained MQA-CLA3 and MQA-CLA4 models with head dimension. We found that these CLA3 and CLA4 models achieved a Pareto improvement over our plain MQA baselines, matching the KV cache footprint of our baseline MQA models with head dimensions ofwhile achieving better perplexities. However, we found that they achieved worse perplexities than our MQA-CLA2 models at the same KV cache footprint.Ablation: MQA + CLA2 with Non-Uniform Sharing Patterns.Finally, we explored using different patterns of KV activation sharing in our MQA-CLA2 models.On the hypothesis that the first and last layers in the model might benefit from special treatment, we trained a model “H128-MQA-CLA2-KeepEnds” which does not share the layerKV cache with any other layers, and instead groups layerwith layer, groups layerwith layer, and so on. This also has the effect of giving the final layer its own KV cache separate from all other layers.We also explored imbalanced configurations with all the KV-cache-producing layers concentrated at either the beginning or end of the model. We trained a model “H128-MQA-CLA2-DenseFront” consisting ofnon-CLA layers, followed byCLA layers all using the KV activations of layer, and a final layer with its own KV cache. Similarly, we trained a model “H128-MQA-CLA2-DenseBack” consisting ofnon-CLA layers, followed by a run ofCLA layers all using the KV activations of layer, and finallynon-CLA layers.We found that all of these alternative CLA sharing patterns achieve worse perplexities than the corresponding MQA-CLA2 model with a uniform sharing pattern, while also requiring slightly more KV cache memory.3.2.2Robustness to Learning Rate TuningThe relative performance of different model architectures can change depending on the learning rates at which they are evaluated. To account for the effects of the learning rate on our results, we conducted learning rate tuning experiments on three models of interest from our initial 1B-scale design space exploration. These learning rate tuning experiments help us verify that CLA continues to provide benefits even when compared to baselines trained at their optimal learning rates.We chose to tune the learning rate for the baseline models H128-MQA and H64-MQA, as well as the CLA model H128-MQA-CLA2. In our initial design space exploration, our results for these models indicated that CLA makes it possible to shrink the KV cache footprint of an MQA model withby a factor ofwhile incurring only a small (point)  degradation in perplexity, or to create a model with the same KV cache footprint as an MQA model withwhile enjoying a substantial (point)  improvement in perplexity. We wanted to verify that this qualitative pattern continues to hold when all models are trained with well-tuned learning rates.Learning Rate Tuning Strategy.For each of our three model configurations, we swept the learning rate upwards from an initial value ofin multiplicative increments of. We ended our sweep for each model at the point where validation perplexity stopped improving. We treat the learning rate which achieved the lowest validation perplexity for each model as an approximation of that model’s optimal learning rate.Results.We found an optimal learning rate offor our H128-MQA baseline, and a higher optimal learning rate offor both our H64-MQA baseline and our H128-MQA-CLA2 model.The validation perplexity results from our 1B-scale learning rate tuning experiments can be found in Table3. When comparing all three models at their best learning rates, we found that the qualitative result from our design space exploration continues to hold: our CLA2 model incurs only a small (point)  validation perplexity degradation relative to ourbaseline while enjoying asmaller KV cache footprint, and achieves a substantial (point)  validation perplexity improvement compared to ourbaseline while using the same amount of KV cache memory.To further validate our results, we also evaluate our three learning-rate-tuned 1B-scale models under EleutherAI’s LM Eval Harnesson Wikitextperplexity and seven standard downstream benchmarks. We report the results of these evaluation in tables3and4. On Wikitext perplexity, we observe a similar pattern as with validation perplexity, with our tuned CLA2 model achieving approximately the same (points better)  perplexity as ourbaseline, and substantially (points)  better perplexity than ourbaseline. On the downstream evaluations, we found that none of our three models model consistently wins or loses across different benchmarks, and that all three models are consistently within–percentage points of each other.ModelKV Bytes PerToken (16-bit) Best LRValidationPerplexityWikitextPerplexityH128-MQA10240H128-MQA-CLA205120H64-MQA05120Table 3:Perplexity results from learning rate tuning experiments at 1B-parameter scale.Model (Best LR) HellaswagPIQAWGSciQOBQABoolQARC-EH128-MQA36.2469.1552.9682.919.057.4055.43H128-MQA-CLA236.0169.1551.9382.621.453.2153.87H64-MQA35.2269.2150.7578.519.455.8151.68Table 4:Downstream benchmark results for 1B-scale models with tuned learning rates. The columns “WG” and “OBQA” denote “WinoGrande” and “OpenBookQA”, respectively.3.3Experiments at 3B-Parameter ScaleTo determine how CLA performs when applied to larger models, we trained a collection of models at the 3B-parameter scale both with and without CLA. We trained each of our 3B-scale model from scratch on 100B tokens from our SlimPajama corpus. The common architectural hyperparameters for our 3B-scale models can be found in Table2.Experiments at Head Dimension.We initially ran experiments to compare three 3B-scale models analogous to the models we selected for our learning rate tuning experiments at the 1B-parameter scale. Specifically, we compared a model using MQA-CLA2 andto an MQA model with the same head dimension (and hencethe KV cache footprint) , and to an MQA model with a head dimension of(and hence the same KV cache footprint) . Based on our 1B-scale experiments, we expected that our MQA-CLA2 and MQA models withwould achieve similar perplexities to each other, and that both would outperform themodel.We tuned the learning rates for these models according to the same learning rate tuning protocol we used at the 1B-parameter scale. After tuning the learning rates for each model, we observed a result different than we had expected: at 3B scale, our MQA-CLA2 model achieves substantially better perplexities than both ourandMQA baselines. Moreover, ourMQA baseline model achieves better perplexities than our tunedMQA baseline, despite having onlyas much KV cache capacity. We report the optimal learning rates and perplexities for these three models in Table5.As with our 1B-scale learning rate tuning experiments, we evaluate these models on downstream benchmarks. We report the results of these evaluations in Table6. As with our 1B-scale experiments, we do not find that any model consistently wins or loses in these downstream evaluations.ModelKV Bytes PerToken (16-bit) Best LRValidationPerplexityWikitextPerplexityH128-MQA16384H128-MQA-CLA20819213.25H64-MQA08192Table 5:Optimal learning rate and perplexity results for our first set of 3B-scale experiments.Model (Best LR) HellaswagPIQAWGSciQOBQABoolQARC-EH128-MQA45.7373.0760.4688.125.459.3064.90H128-MQA-CLA247.1274.3260.6989.225.258.6264.73H64-MQA46.4274.0557.8588.125.659.8865.57Table 6:Downstream evaluation results for our first set of 3B-scale experiments.Experiments at Head Dimension.Because in our initial 3B-scale experiments we found that ourMQA model represents a stronger baseline than ourMQA model, we ran a second set of experiments with adjusted head sizes. Specifically, we chose to compare an MQA-CLA2 model withto a plain MQA model with, and to a plain MQA model with.Due to logistical constraints we trained all models in this second set of 3B-scale experiments on a different training cluster using a different training software stack and data order. This included retraining a new version of our H64-MQA-CLA2 baseline in order to control for differences in the new training environment.We trained all models in this second set of experiments with a learning rate of, which we had found to be the best learning rate for ourMQA baseline model in our first set of 3B-scale experiments. For ourMQA-CLA2 model and ourMQA baseline model, we also experimented with learning rates of, but found these achieved worse perplexities than our initial value of.We report perplexity results for this second set of experiments in Table7, and results for downstream benchmarks in Table8. In the Wikitext perplexity results for this set of experiments, we find agreement with the pattern observed at the 1B scale. Our MQA-CLA2 model withincurs only a small (point)  degradation in perplexity compared to ourbaseline while enjoying asmaller KV cache footprint, and achieves a substantial (point)  improvement in perplexity compared to ourbaseline while using the same amount of KV cache memory.We also evaluate these three models on downstream benchmarks, and report the results in Table8. As with our downstream benchmark evaluations for our other experiments, we find that all models perform similarly.ModelKV Bytes PerToken (16-bit) Best LRWikitext PerplexityH64-MQA819212.94H64-MQA-CLA24096H32-MQA4096Table 7:Optimal learning rate and perplexity results for our second set of 3B-scale experiments.Model (Best LR) HellaswagPIQAWGSciQOBQABoolQARC-EH64-MQA47.3474.5460.4688.924.257.2566.92H64-MQA-CLA247.3274.5457.4687.925.261.6265.53H32-MQA46.0573.8360.0688.625.661.8765.24Table 8:Downstream evaluation results for our second set of 3B-scale experiments.
body
Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
3Pretraining Experiments
2405.12981
4Discussion & Future WorkTakeaways and Implications.In the regimes where we tested it, we find that MQA-CLA2 consistently achieves the lowest validation perplexity (withinpoints)  for a given KV cache memory budget and model size. In our ablations, we find that using sharing factors greater than(CLA3 and above)  achieves slightly worse accuracy/memory tradeoffs than using CLA2 and varying the head dimension, although still Pareto-dominates the tradeoffs possible with plain MQA alone.At both 1B and 3B scale, we find that for MQA models with typical head sizes ofand, applying CLA2 yields aKV cache reduction while incurring at worst a very modest (less thanchange)  degradation in perplexity, and in some cases improving perplexity. We recommend this recipe to practitioners as a conservative change to existing MQA architectures which delivers substantial memory overhead reductions with relatively little risk.Future Work.One natural question that rises from any memory efficient LLM alternative is its efficiency improvement when serving through longer sequences and greater batching.
We leave end-to-end inference efficiency evaluations of large, long-context models employing CLA as an interesting problem for future work.
We suspect that the types of LLMs that will stand to gain the most are the ones which have extremely long sequences, such as models that have long term memory or use methods like Landmark Attentionwhich render attention over long contexts more feasible.
body
Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
4Discussion & Future Work
2405.12981
5Related WorkTransformer memory efficiency can refer to many potential objectives.
It can refer to decreasing memory storage or bandwidth requirements, it can be targeted at training or inference, and finally it apply either within a single pass of the model or between passes.
While this work targets decreasing the size of the inference KV cache that persists between passes, notable works such as
Flash Attentionhave decreased the memory bandwidth necessary for a single pass,
and works like FlexGenachieved low memory storage during a single forward pass by partial offloading to disk.
Here we discuss related work that improves the memory efficiency of attention, specifically the memory storage of the KV cache.5.1Decreasing KV cache size Post trainingMuch work has focused on decreasing the size of the KV cache for models that have already been trained.KV cache compressionAs many works have tried to compress LLMs through pruning, quantization, and sparsity,
(seefor a survey) 
a subset directly focus on the problem of KV cache compression.
For quantization,
KVQuantand
Coupled Quantizationperform targeted transformations of the keys and values along with non uniform encodings to compress the KV cache to one to two bits.
Sparsifying the KV cache done by works such as
H2OScissorhandsand FastGenonly store a subset of the KV cache during generation.
They do so by storing only tokens that are near to the generating token or important across the sequence, with the heuristic for importance varying between papers.
Finally, Cachegendirectly compresses the KV cache with a tensor encoder.5.2Architectural Changes that decrease KV cache sizeMost relevant to our work are methods that change the architecture of the model in order to decrease the size of the KV cache.
These methods can roughly grouped into three categories:
those that try to reduce the number of tokens attended to,
those that replace softmax attention with another operation that requires less memory storage,
and those that, like our work, decrease the unique KV values compared to each query.Decreasing effective sequence lengthModels that decrease the effective sequence length of the model have existed almost as transformers themselves.
Two notable early works are
Transformer XLand
Sparse Attention.
Both of them performed attention in local windows of smaller sizes instead of across the whole sequence,
and differed in how they incorporated information from prior tokens.
This line of work was used in many models of note such as GPT3, and is commonly known as sliding window attention.
This line of work has been further developed using methods like
Infini attention, which compresses prior tokens using a linear attention mechanism.An alternative approach is to perform a lookup over prior tokens, such as in
Landmark attentionMemorizing transformersor to do a lookup over an external datastore, which is commonly known as retrieval.
However, while these methods reduce computation, they do not reduce storage unless the KVs for lookup are offloaded from GPU memory.Removing Softmax AttentionReplacements to Softmax Attention are often referred to as SSMs or linear attention computations.
Regardless of name, most replace the attention component with an alternative which has constant space complexity during token generation w.r.t. the number of tokens generated.
This also reduces the time complexity of generation to be linear w.r.t. the number of tokens, instead of quadratic.
Various methods differ in how they parameterize their state.proposed initial versions of linear attention.
Recent work has focused on using data dependent mechanisms to improve the state, such as in
GLAMambaand RWKV v6.Groupings of AttentionMost related are methods that use softmax attention but attempt to use a single KV pair for multiple queries.
GQAand MQAdo this by grouping keys and values across heads.Concurrent work tries other strategies to share values between layers.
Deepseek-V2proposes Multi Latent Attention, which uses a low rank projection of the keys and values.
You Only Cache Oncesplits the model into two halves. The first half performs local attention and then generates a set of keys and values that are used for global attention across all layers in the second half.
body
Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
5Related Work
2405.12981
6ConclusionCross-Layer Attention is an effective method for reducing the KV cache memory storage footprint of transformer models by a factor ofwith roughly equal perplexity.
Based on extensive experimental evaluation against well-tuned baselines at both the 1B- and 3B-parameter scales, we find that CLA advances the Pareto frontier for memory-efficient transformers.
conclusion
Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
6Conclusion